[X86][BF16] Enable __bf16 for x86 targets.

[FreddyYe]

Summary

X86 psABI has updated to support __bf16 type, the ABI of which is the
same as FP16. See https://discourse.llvm.org/t/patch-add-optional-bfloat16-support/63149

[pengfei]

Add to ReleaseNotes.rst as well.

[rjmccall]

How are you actually implementing __bf16 on these targets? There isn't even hardware support for conversions.

[bkramer]

In D130964#3694473, @rjmccall wrote:

How are you actually implementing __bf16 on these targets? There isn't even hardware support for conversions.

bf16 -> float is really just a bit shift. The other direction gets lowered to a libcall, compiler-rt has a conversion function with proper rounding. I added some support to make the backend promote all other arithmetic to float, but I think that's only enabled on x86 so far.

[pengfei]

In D130964#3694473, @rjmccall wrote:

How are you actually implementing __bf16 on these targets? There isn't even hardware support for conversions.

We support float -> bf16 in AVX512BF16. https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#avx512techs=AVX512_BF16
And we found some problems in how to represent bf16 types in intrinsics. For example, we currently defined __bfloat16 as unsigned short. We cannot stop user e.g., adding 2 __bfloat16 in C code and getting the wrong result. So we want to introduce the type on X86. For more information, please see the discussions in D120395,

[FreddyYe]

In D130964#3694540, @bkramer wrote:

In D130964#3694473, @rjmccall wrote:

How are you actually implementing __bf16 on these targets? There isn't even hardware support for conversions.

bf16 -> float is really just a bit shift. The other direction gets lowered to a libcall, compiler-rt has a conversion function with proper rounding. I added some support to make the backend promote all other arithmetic to float, but I think that's only enabled on x86 so far.

Yes, we can view x86 backend has been dealing with __bf16. And with https://reviews.llvm.org/D130832, it will complete follow psABI. About hardware support, x86 actually has supported bf16 since AVX512BF16 (https://reviews.llvm.org/D60552), which has vector conversion support between float and bf16. However, at that time we chose a typedef short as C type. In the future, we can support backend lowering for those instructions: VCVTNE2PS2BF16, VCVTNEPS2BF16 and DPBF16PS

[rjmccall]

In D130964#3695408, @FreddyYe wrote:

In D130964#3694540, @bkramer wrote:

In D130964#3694473, @rjmccall wrote:

How are you actually implementing __bf16 on these targets? There isn't even hardware support for conversions.

bf16 -> float is really just a bit shift. The other direction gets lowered to a libcall, compiler-rt has a conversion function with proper rounding. I added some support to make the backend promote all other arithmetic to float, but I think that's only enabled on x86 so far.

About hardware support, x86 actually has supported bf16 since AVX512BF16 (https://reviews.llvm.org/D60552), which has vector conversion support between float and bf16.

Right, but this patch is adding x86 support whenever SSE2 is available. AVX512BF16 is available on a *very* small slice of processors. In contrast, e.g. F16C is relatively broadly available, although I understand that we formally support _Float16 all the way back to SSE2 and thus on some processors that lack F16C.

But okay, pure intrinsic support is fine if that's what we're doing.

I think the patch looks fine.

[FreddyYe]

Right, but this patch is adding x86 support whenever SSE2 is available. AVX512BF16 is available on a *very* small slice of processors. In contrast, e.g. F16C is relatively broadly available, although I understand that we formally support _Float16 all the way back to SSE2 and thus on some processors that lack F16C.

But okay, pure intrinsic support is fine if that's what we're doing.

I think the patch looks fine.

Yes. This type is for pure intrinsic support. Thanks for your review. Let's wait for the backend patch to land first.

[pengfei]

LGTM.

clang/docs/LanguageExtensions.rst

 Otherwise, ``_Float16`` arithmetic is performed by promoting to ``float``,
 performing the operation, and then truncating to ``_Float16``.
 
 ``_Float16`` will be supported on more targets as they define ABIs for it.
 
 ``__bf16`` is purely a storage format; it is currently only supported on the following targets:
 * 32-bit ARM
 * 64-bit ARM (AArch64)
+* X86 (see below)
+
+On X86 targets, ``__bf16`` is supported as long as SSE2 is available, which
+includes all 64-bit and all recent 32-bit processors.
 
 ``__fp16`` is a storage and interchange format only.  This means that values of
 ``__fp16`` are immediately promoted to (at least) ``float`` when used in arithmetic
 operations, so that e.g. the result of adding two ``__fp16`` values has type ``float``.
 The behavior of ``__fp16`` is specified by the ARM C Language Extensions (`ACLE <http://infocenter.arm.com/help/topic/com.arm.doc.ihi0053d/IHI0053D_acle_2_1.pdf>`_).
 Clang uses the ``binary16`` format from IEEE 754-2008 for ``__fp16``, not the ARM
 alternative format.
 

clang/lib/Basic/Targets/X86.h

 protected:
   llvm::X86::CPUKind CPU = llvm::X86::CK_None;
 
   enum FPMathKind { FP_Default, FP_SSE, FP_387 } FPMath = FP_Default;
 
 public:
   X86TargetInfo(const llvm::Triple &Triple, const TargetOptions &)
       : TargetInfo(Triple) {
+    BFloat16Width = BFloat16Align = 16;
+    BFloat16Format = &llvm::APFloat::BFloat();
     LongDoubleFormat = &llvm::APFloat::x87DoubleExtended();
     AddrSpaceMap = &X86AddrSpaceMap;
     HasStrictFP = true;
 
     bool IsWinCOFF =
         getTriple().isOSWindows() && getTriple().isOSBinFormatCOFF();
     if (IsWinCOFF)
       MaxVectorAlign = MaxTLSAlign = 8192u * getCharWidth();
uint64_t getPointerWidthV(unsigned AddrSpace) const override {
     if (AddrSpace == ptr64)
       return 64;
     return PointerWidth;
   }
 
   uint64_t getPointerAlignV(unsigned AddrSpace) const override {
     return getPointerWidthV(AddrSpace);
   }
+
+  const char *getBFloat16Mangling() const override { return "u6__bf16"; };
 };
 
 // X86-32 generic target
 class LLVM_LIBRARY_VISIBILITY X86_32TargetInfo : public X86TargetInfo {
 public:
   X86_32TargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts)
       : X86TargetInfo(Triple, Opts) {
     DoubleAlign = LongLongAlign = 32;

clang/lib/Basic/Targets/X86.cpp

X86SSEEnum Level = llvm::StringSwitch<X86SSEEnum>(Feature)
                            .Case("+sse3", SSE3)
                            .Case("+sse2", SSE2)
                            .Case("+sse", SSE1)
                            .Default(NoSSE);
     SSELevel = std::max(SSELevel, Level);
 
     HasFloat16 = SSELevel >= SSE2;
 
+    HasBFloat16 = SSELevel >= SSE2;
+
     MMX3DNowEnum ThreeDNowLevel = llvm::StringSwitch<MMX3DNowEnum>(Feature)
                                       .Case("+3dnowa", AMD3DNowAthlon)
                                       .Case("+3dnow", AMD3DNow)
                                       .Case("+mmx", MMX)
                                       .Default(NoMMX3DNow);
     MMX3DNowLevel = std::max(MMX3DNowLevel, ThreeDNowLevel);
 
     XOPEnum XLevel = llvm::StringSwitch<XOPEnum>(Feature)

clang/lib/CodeGen/TargetInfo.cpp

if (const BuiltinType *BT = Ty->getAs<BuiltinType>()) {
     if (k == BuiltinType::Void) {
       Current = NoClass;
     } else if (k == BuiltinType::Int128 || k == BuiltinType::UInt128) {
       Lo = Integer;
       Hi = Integer;
     } else if (k >= BuiltinType::Bool && k <= BuiltinType::LongLong) {
       Current = Integer;
     } else if (k == BuiltinType::Float || k == BuiltinType::Double ||
-               k == BuiltinType::Float16) {
+               k == BuiltinType::Float16 || k == BuiltinType::BFloat16) {
       Current = SSE;
     } else if (k == BuiltinType::LongDouble) {
       const llvm::fltSemantics *LDF = &getTarget().getLongDoubleFormat();
       if (LDF == &llvm::APFloat::IEEEquad()) {
         Lo = SSE;
         Hi = SSEUp;
       } else if (LDF == &llvm::APFloat::x87DoubleExtended()) {
         Lo = X87;
if (const ComplexType *CT = Ty->getAs<ComplexType>()) {
     QualType ET = getContext().getCanonicalType(CT->getElementType());
 
     uint64_t Size = getContext().getTypeSize(Ty);
     if (ET->isIntegralOrEnumerationType()) {
       if (Size <= 64)
         Current = Integer;
       else if (Size <= 128)
         Lo = Hi = Integer;
-    } else if (ET->isFloat16Type() || ET == getContext().FloatTy) {
+    } else if (ET->isFloat16Type() || ET == getContext().FloatTy ||
+               ET->isBFloat16Type()) {
       Current = SSE;
     } else if (ET == getContext().DoubleTy) {
       Lo = Hi = SSE;
     } else if (ET == getContext().LongDoubleTy) {
       const llvm::fltSemantics *LDF = &getTarget().getLongDoubleFormat();
       if (LDF == &llvm::APFloat::IEEEquad())
         Current = Memory;
       else if (LDF == &llvm::APFloat::x87DoubleExtended())
if (!T0 || T0->isDoubleTy())
     return llvm::Type::getDoubleTy(getVMContext());
 
   // Get the adjacent FP type.
   llvm::Type *T1 = nullptr;
   unsigned T0Size = TD.getTypeAllocSize(T0);
   if (SourceSize > T0Size)
       T1 = getFPTypeAtOffset(IRType, IROffset + T0Size, TD);
   if (T1 == nullptr) {
-    // Check if IRType is a half + float. float type will be in IROffset+4 due
+    // Check if IRType is a half/bfloat + float. float type will be in IROffset+4 due
     // to its alignment.
-    if (T0->isHalfTy() && SourceSize > 4)
+    if (T0->is16bitFPTy() && SourceSize > 4)
       T1 = getFPTypeAtOffset(IRType, IROffset + 4, TD);
     // If we can't get a second FP type, return a simple half or float.
     // avx512fp16-abi.c:pr51813_2 shows it works to return float for
     // {float, i8} too.
     if (T1 == nullptr)
       return T0;
   }
 
   if (T0->isFloatTy() && T1->isFloatTy())
     return llvm::FixedVectorType::get(T0, 2);
 
-  if (T0->isHalfTy() && T1->isHalfTy()) {
+  if (T0->is16bitFPTy() && T1->is16bitFPTy()) {
     llvm::Type *T2 = nullptr;
     if (SourceSize > 4)
       T2 = getFPTypeAtOffset(IRType, IROffset + 4, TD);
     if (T2 == nullptr)
       return llvm::FixedVectorType::get(T0, 2);
     return llvm::FixedVectorType::get(T0, 4);
   }
 
-  if (T0->isHalfTy() || T1->isHalfTy())
+  if (T0->is16bitFPTy() || T1->is16bitFPTy())
     return llvm::FixedVectorType::get(llvm::Type::getHalfTy(getVMContext()), 4);
 
   return llvm::Type::getDoubleTy(getVMContext());
 }
 
 
 /// GetINTEGERTypeAtOffset - The ABI specifies that a value should be passed in
 /// an 8-byte GPR.  This means that we either have a scalar or we are talking

clang/test/CodeGen/X86/bfloat-abi.c

+// RUN: %clang_cc1 -triple x86_64-linux -emit-llvm  -target-feature +sse2 < %s | FileCheck %s --check-prefixes=CHECK
+
+struct bfloat1 {
+  __bf16 a;
+};
+
+struct bfloat1 h1(__bf16 a) {
+  // CHECK: define{{.*}}bfloat @
+  struct bfloat1 x;
+  x.a = a;
+  return x;
+}
+
+struct bfloat2 {
+  __bf16 a;
+  __bf16 b;
+};
+
+struct bfloat2 h2(__bf16 a, __bf16 b) {
+  // CHECK: define{{.*}}<2 x bfloat> @
+  struct bfloat2 x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat3 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+};
+
+struct bfloat3 h3(__bf16 a, __bf16 b, __bf16 c) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloat4 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+struct bfloat4 h4(__bf16 a, __bf16 b, __bf16 c, __bf16 d) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat4 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct floatbfloat {
+  float a;
+  __bf16 b;
+};
+
+struct floatbfloat fh(float a, __bf16 b) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct floatbfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct floatbfloat2 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+};
+
+struct floatbfloat2 fh2(float a, __bf16 b, __bf16 c) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct floatbfloat2 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloatfloat {
+  __bf16 a;
+  float b;
+};
+
+struct bfloatfloat hf(__bf16 a, float b) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct bfloatfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat2float {
+  __bf16 a;
+  __bf16 b;
+  float c;
+};
+
+struct bfloat2float h2f(__bf16 a, __bf16 b, float c) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat2float x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct floatbfloat3 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+struct floatbfloat3 fh3(float a, __bf16 b, __bf16 c, __bf16 d) {
+  // CHECK: define{{.*}}{ <4 x half>, bfloat } @
+  struct floatbfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct bfloat5 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+  __bf16 e;
+};
+
+struct bfloat5 h5(__bf16 a, __bf16 b, __bf16 c, __bf16 d, __bf16 e) {
+  // CHECK: define{{.*}}{ <4 x bfloat>, bfloat } @
+  struct bfloat5 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  x.e = e;
+  return x;
+}

clang/test/CodeGen/X86/bfloat-half-abi.c

+// RUN: %clang_cc1 -triple x86_64-linux -emit-llvm  -target-feature +sse2 < %s | FileCheck %s --check-prefixes=CHECK
+
+struct bfloat1 {
+  __bf16 a;
+};
+
+struct bfloat1 h1(__bf16 a) {
+  // CHECK: define{{.*}}bfloat @
+  struct bfloat1 x;
+  x.a = a;
+  return x;
+}
+
+struct bfloat2 {
+  __bf16 a;
+  __bf16 b;
+};
+
+struct bfloat2 h2(__bf16 a, __bf16 b) {
+  // CHECK: define{{.*}}<2 x bfloat> @
+  struct bfloat2 x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat3 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+};
+
+struct bfloat3 h3(__bf16 a, __bf16 b, __bf16 c) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloat4 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+struct bfloat4 h4(__bf16 a, __bf16 b, __bf16 c, __bf16 d) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat4 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct floatbfloat {
+  float a;
+  __bf16 b;
+};
+
+struct floatbfloat fh(float a, __bf16 b) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct floatbfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct floatbfloat2 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+};
+
+struct floatbfloat2 fh2(float a, __bf16 b, __bf16 c) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct floatbfloat2 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct bfloatfloat {
+  __bf16 a;
+  float b;
+};
+
+struct bfloatfloat hf(__bf16 a, float b) {
+  // CHECK: define{{.*}}<4 x half> @
+  struct bfloatfloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct bfloat2float {
+  __bf16 a;
+  __bf16 b;
+  float c;
+};
+
+struct bfloat2float h2f(__bf16 a, __bf16 b, float c) {
+  // CHECK: define{{.*}}<4 x bfloat> @
+  struct bfloat2float x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct floatbfloat3 {
+  float a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+};
+
+struct floatbfloat3 fh3(float a, __bf16 b, __bf16 c, __bf16 d) {
+  // CHECK: define{{.*}}{ <4 x half>, bfloat } @
+  struct floatbfloat3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct bfloat5 {
+  __bf16 a;
+  __bf16 b;
+  __bf16 c;
+  __bf16 d;
+  __bf16 e;
+};
+
+struct bfloat5 h5(__bf16 a, __bf16 b, __bf16 c, __bf16 d, __bf16 e) {
+  // CHECK: define{{.*}}{ <4 x bfloat>, bfloat } @
+  struct bfloat5 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  x.e = e;
+  return x;
+}

clang/test/CodeGen/X86/bfloat-mangle.cpp

+// RUN: %clang_cc1 -triple i386-unknown-unknown -target-feature +sse2 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -target-feature +sse2 -emit-llvm -o - %s | FileCheck %s
+
+// CHECK: define {{.*}}void @_Z3foou6__bf16(bfloat noundef %b)
+void foo(__bf16 b) {}

clang/test/Sema/vector-decl-crash.c

-// RUN: %clang_cc1 %s -fsyntax-only -verify -triple x86_64-unknown-unknown
+// RUN: %clang_cc1 %s -fsyntax-only -verify -triple riscv64-unknown-unknown
 
 // GH50171
 // This would previously crash when __bf16 was not a supported type.
 __bf16 v64bf __attribute__((vector_size(128))); // expected-error {{__bf16 is not supported on this target}} \
                                                    expected-error {{vector size not an integral multiple of component size}}
 

llvm/include/llvm/IR/Type.h

public:
   bool isVoidTy() const { return getTypeID() == VoidTyID; }
 
   /// Return true if this is 'half', a 16-bit IEEE fp type.
   bool isHalfTy() const { return getTypeID() == HalfTyID; }
 
   /// Return true if this is 'bfloat', a 16-bit bfloat type.
   bool isBFloatTy() const { return getTypeID() == BFloatTyID; }
 
+  /// Return true if this is a 16-bit float type.
+  bool is16bitFPTy() const {
+    return getTypeID() == BFloatTyID || getTypeID() == HalfTyID;
+  }
+
   /// Return true if this is 'float', a 32-bit IEEE fp type.
   bool isFloatTy() const { return getTypeID() == FloatTyID; }
 
   /// Return true if this is 'double', a 64-bit IEEE fp type.
   bool isDoubleTy() const { return getTypeID() == DoubleTyID; }
 
   /// Return true if this is x86 long double.
   bool isX86_FP80Ty() const { return getTypeID() == X86_FP80TyID; }