[AArch64] Always allow the __bf16 type

[dmgreen]

Summary

We would like to make the ACLE NEON and SVE intrinsics more useable by gating them on the target, not by ifdef preprocessor macros. See #56480. In order to do this the types they use need to be available. This patches makes __bf16 always available under AArch64, bringing it in-line with GCC https://godbolt.org/z/sfWWv74Mj. In subsequent patches the NEON bfloat16x8_t and SVE svbfloat16_t types (along with bfloat16_t used in arm_sve.h) will be made unconditional too.

[simon_tatham]

What actually happens if you try this? Normally I'd expect that if a thing is forbidden in the front end it's because otherwise it would lead to a codegen crash or something similar. But this patch is only removing the restriction, and not adding any support. Is that part already done, and bf16 operations will get automatically converted into some libcall that already exists, in the absence of any better approach?

[dmgreen]

It can only be used to pass values around. The tests in arm-bfloat.cpp shows that you can declare variables but not perform any of the operations in that test. arm-bf16-forbidden-ops.cpp shows it can't be cast. arm-bf16-params-returns.c shows it can be passed through args and rets. And this shows that working as expected in the backend: https://godbolt.org/z/oxzh1PWe9. The idea is that you can't do a huge amount with them, and what you can do can be handled without any special instructions.

(I do know that there are a few areas where they currently do not work but should, like ternary operations: https://godbolt.org/z/v8Mx3EPW7. I hope to fix that but it's an orthogonal issue as it happens with the feature).

That leaves the intrinsics, which are currently part of arm_neon.h. The arm_neon.h file is more difficult to deal with, I'm starting with arm_sve.h. The bfloat16 intrinsics in arm_neon.h are currently behind an #ifdef guard, so cannot be used. Once that changes it will work more like this GCC example, where they are defined but using them in a function without bf16 available will give an error: https://godbolt.org/z/13vfecjx1

[simon_tatham]

Ah, yes, I misread what the test changes were about – I didn't notice that you'd only just introduced the scalar error context, and thought you were enabling all those operations in at least one situation, rather than changing the form of the check that they were unconditionally disabled.

Right, so the effect is that you can talk about the type, put it in struct fields, query its size and alignment, and if you really must, declare variables of the type that you never do anything with. That all sounds safe enough :-)

In that case, LGTM.

clang/docs/LanguageExtensions.rst

 performing the operation, and then truncating to ``_Float16``.
 
 ``_Float16`` will be supported on more targets as they define ABIs for it.
 
 ``__bf16`` is purely a storage format; it is currently only supported on the following targets:
 * 32-bit ARM
 * 64-bit ARM (AArch64)
 
-The ``__bf16`` type is only available when supported in hardware.
-
 ``__fp16`` is a storage and interchange format only.  This means that values of
 ``__fp16`` are immediately promoted to (at least) ``float`` when used in arithmetic
 operations, so that e.g. the result of adding two ``__fp16`` values has type ``float``.
 The behavior of ``__fp16`` is specified by the ARM C Language Extensions (`ACLE <http://infocenter.arm.com/help/topic/com.arm.doc.ihi0053d/IHI0053D_acle_2_1.pdf>`_).
 Clang uses the ``binary16`` format from IEEE 754-2008 for ``__fp16``, not the ARM
 alternative format.
 
 ``_Float16`` is an interchange floating-point type.  This means that, just like arithmetic on

clang/lib/Basic/Targets/AArch64.h

public:
 
   Optional<std::pair<unsigned, unsigned>>
   getVScaleRange(const LangOptions &LangOpts) const override;
 
   bool hasFeature(StringRef Feature) const override;
   bool handleTargetFeatures(std::vector<std::string> &Features,
                             DiagnosticsEngine &Diags) override;
 
+  bool hasBFloat16Type() const override;
+
   CallingConvCheckResult checkCallingConvention(CallingConv CC) const override;
 
   bool isCLZForZeroUndef() const override;
 
   BuiltinVaListKind getBuiltinVaListKind() const override;
 
   ArrayRef<const char *> getGCCRegNames() const override;
   ArrayRef<TargetInfo::GCCRegAlias> getGCCRegAliases() const override;

clang/lib/Basic/Targets/AArch64.cpp

if (Feature == "+mops")
       HasMOPS = true;
   }
 
   setDataLayout();
 
   return true;
 }
 
+bool AArch64TargetInfo::hasBFloat16Type() const {
+  return true;
+}
+
 TargetInfo::CallingConvCheckResult
 AArch64TargetInfo::checkCallingConvention(CallingConv CC) const {
   switch (CC) {
   case CC_C:
   case CC_Swift:
   case CC_SwiftAsync:
   case CC_PreserveMost:
   case CC_PreserveAll:

clang/test/CodeGen/arm-bf16-params-returns.c

 // RUN: %clang_cc1 -triple armv8.6a-arm-none-eabi -target-abi aapcs -mfloat-abi hard -target-feature +bf16 -target-feature +neon -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s --check-prefix=CHECK32-HARD
 // RUN: %clang_cc1 -triple armv8.6a-arm-none-eabi -target-abi aapcs -mfloat-abi softfp -target-feature +bf16 -target-feature +neon -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s --check-prefix=CHECK32-SOFTFP
-// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-abi aapcs -target-feature +bf16 -target-feature +neon -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s --check-prefix=CHECK64
+// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-abi aapcs -target-feature +bf16 -target-feature +neon -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s --check-prefixes=CHECK64,CHECK64NEON
+// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-abi aapcs -target-feature -bf16 -target-feature +neon -DNONEON -emit-llvm -O2 -o - %s | opt -S -mem2reg -sroa | FileCheck %s --check-prefix=CHECK64
 
 // REQUIRES: aarch64-registered-target || arm-registered-target
 
 #include <arm_neon.h>
 
 // function return types
 __bf16 test_ret_bf16(__bf16 v) {
   return v;
 }
 // CHECK32-HARD: define{{.*}} arm_aapcs_vfpcc bfloat @test_ret_bf16(bfloat noundef returned %v) {{.*}} {
 // CHECK32-HARD: ret bfloat %v
 // CHECK32-SOFTFP: define{{.*}} bfloat @test_ret_bf16(bfloat noundef returned %v) {{.*}} {
 // CHECK32-SOFTFP: ret bfloat %v
 // CHECK64: define{{.*}} bfloat @test_ret_bf16(bfloat noundef returned %v) {{.*}} {
 // CHECK64: ret bfloat %v
 
+#ifndef NONEON
+
 bfloat16x4_t test_ret_bf16x4_t(bfloat16x4_t v) {
   return v;
 }
 // CHECK32-HARD: define{{.*}} arm_aapcs_vfpcc <4 x bfloat> @test_ret_bf16x4_t(<4 x bfloat> noundef returned %v) {{.*}} {
 // CHECK32-HARD: ret <4 x bfloat> %v
 // CHECK32-SOFTFP: define{{.*}} <2 x i32> @test_ret_bf16x4_t(<2 x i32> [[V0:.*]]) {{.*}} {
 // CHECK32-SOFTFP: ret <2 x i32> %v
-// CHECK64: define{{.*}} <4 x bfloat> @test_ret_bf16x4_t(<4 x bfloat> noundef returned %v) {{.*}} {
-// CHECK64: ret <4 x bfloat> %v
+// CHECK64NEON: define{{.*}} <4 x bfloat> @test_ret_bf16x4_t(<4 x bfloat> noundef returned %v) {{.*}} {
+// CHECK64NEON: ret <4 x bfloat> %v
+
+#endif
 No newline at end of file

clang/test/CodeGen/arm-mangle-bf16.cpp

 // RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-feature +bf16 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-arm-none-eabi -target-feature -bf16 -emit-llvm -o - %s | FileCheck %s
 // RUN: %clang_cc1 -triple arm-arm-none-eabi -target-feature +bf16 -mfloat-abi hard -emit-llvm -o - %s | FileCheck %s
 // RUN: %clang_cc1 -triple arm-arm-none-eabi -target-feature +bf16 -mfloat-abi softfp -emit-llvm -o - %s | FileCheck %s
 
 // CHECK: define {{.*}}void @_Z3foou6__bf16(bfloat noundef %b)
 void foo(__bf16 b) {}

clang/test/Sema/arm-bf16-forbidden-ops.c

 // RUN: %clang_cc1 -fsyntax-only -verify -triple aarch64 -target-feature +bf16 %s
+// RUN: %clang_cc1 -fsyntax-only -verify -triple aarch64 -target-feature -bf16 %s
 
 __bf16 test_cast_from_float(float in) {
   return (__bf16)in; // expected-error {{cannot type-cast to __bf16}}
 }
 
 __bf16 test_cast_from_float_literal(void) {
   return (__bf16)1.0f; // expected-error {{cannot type-cast to __bf16}}
 }

clang/test/Sema/arm-bf16-forbidden-ops.cpp

 // RUN: %clang_cc1 -fsyntax-only -verify -triple aarch64 -target-feature +bf16 %s
+// RUN: %clang_cc1 -fsyntax-only -verify -triple aarch64 -target-feature -bf16 %s
 
 __bf16 test_static_cast_from_float(float in) {
   return static_cast<__bf16>(in); // expected-error {{static_cast from 'float' to '__bf16' is not allowed}}
 }
 
 __bf16 test_static_cast_from_float_literal(void) {
   return static_cast<__bf16>(1.0f); // expected-error {{static_cast from 'float' to '__bf16' is not allowed}}
 }

clang/test/Sema/arm-bfloat.cpp

-// RUN: %clang_cc1 -fsyntax-only -verify -std=c++11 \
-// RUN: -triple aarch64-arm-none-eabi -target-cpu cortex-a75 \
-// RUN: -target-feature +bf16 -target-feature +neon %s
-// RUN: %clang_cc1 -fsyntax-only -verify -std=c++11 \
-// RUN: -triple arm-arm-none-eabi -target-cpu cortex-a53 \
-// RUN: -target-feature +bf16 -target-feature +neon %s
+// RUN: %clang_cc1 -fsyntax-only -verify=scalar,neon -std=c++11 \
+// RUN:   -triple aarch64-arm-none-eabi -target-cpu cortex-a75 \
+// RUN:   -target-feature +bf16 -target-feature +neon %s
+// RUN: %clang_cc1 -fsyntax-only -verify=scalar,neon -std=c++11 \
+// RUN:   -triple arm-arm-none-eabi -target-cpu cortex-a53 \
+// RUN:   -target-feature +bf16 -target-feature +neon %s
 
+// The types should be available under AArch64 even without the bf16 feature
+// RUN: %clang_cc1 -fsyntax-only -verify=scalar -DNONEON -std=c++11 \
+// RUN:   -triple aarch64-arm-none-eabi -target-cpu cortex-a75 \
+// RUN:   -target-feature -bf16 -target-feature +neon %s
+
 // REQUIRES: aarch64-registered-target || arm-registered-target
 
 void test(bool b) {
   __bf16 bf16;
 
-  bf16 + bf16; // expected-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
-  bf16 - bf16; // expected-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
-  bf16 * bf16; // expected-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
-  bf16 / bf16; // expected-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
+  bf16 + bf16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
+  bf16 - bf16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
+  bf16 * bf16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
+  bf16 / bf16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__bf16')}}
 
   __fp16 fp16;
 
-  bf16 + fp16; // expected-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
-  fp16 + bf16; // expected-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
-  bf16 - fp16; // expected-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
-  fp16 - bf16; // expected-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
-  bf16 * fp16; // expected-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
-  fp16 * bf16; // expected-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
-  bf16 / fp16; // expected-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
-  fp16 / bf16; // expected-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
-  bf16 = fp16; // expected-error {{assigning to '__bf16' from incompatible type '__fp16'}}
-  fp16 = bf16; // expected-error {{assigning to '__fp16' from incompatible type '__bf16'}}
-  bf16 + (b ? fp16 : bf16); // expected-error {{incompatible operand types ('__fp16' and '__bf16')}}
+  bf16 + fp16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
+  fp16 + bf16; // scalar-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
+  bf16 - fp16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
+  fp16 - bf16; // scalar-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
+  bf16 * fp16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
+  fp16 * bf16; // scalar-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
+  bf16 / fp16; // scalar-error {{invalid operands to binary expression ('__bf16' and '__fp16')}}
+  fp16 / bf16; // scalar-error {{invalid operands to binary expression ('__fp16' and '__bf16')}}
+  bf16 = fp16; // scalar-error {{assigning to '__bf16' from incompatible type '__fp16'}}
+  fp16 = bf16; // scalar-error {{assigning to '__fp16' from incompatible type '__bf16'}}
+  bf16 + (b ? fp16 : bf16); // scalar-error {{incompatible operand types ('__fp16' and '__bf16')}}
 }
 
+#ifndef NONEON
+
 #include <arm_neon.h>
 
 void test_vector(bfloat16x4_t a, bfloat16x4_t b, float16x4_t c) {
-  a + b; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
-  a - b; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
-  a * b; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
-  a / b; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
-
-  a + c; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
-  a - c; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
-  a * c; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
-  a / c; // expected-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
-  c + b; // expected-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
-  c - b; // expected-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
-  c * b; // expected-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
-  c / b; // expected-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
+  a + b; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
+  a - b; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
+  a * b; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
+  a / b; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'bfloat16x4_t')}}
+
+  a + c; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
+  a - c; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
+  a * c; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
+  a / c; // neon-error {{invalid operands to binary expression ('bfloat16x4_t' (vector of 4 'bfloat16_t' values) and 'float16x4_t' (vector of 4 'float16_t' values))}}
+  c + b; // neon-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
+  c - b; // neon-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
+  c * b; // neon-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
+  c / b; // neon-error {{invalid operands to binary expression ('float16x4_t' (vector of 4 'float16_t' values) and 'bfloat16x4_t' (vector of 4 'bfloat16_t' values))}}
 }
+#endif
 No newline at end of file