[clang][BFloat] add NEON emitter for bfloat

[stuij]

Summary

This patch adds the bfloat16_t struct typedefs (e.g. bfloat16x8x2_t) to
arm_neon.h

This patch is part of a series implementing the Bfloat16 extension of the
Armv8.6-a architecture, as detailed here:

https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/arm-architecture-developments-armv8-6-a

The bfloat type, and its properties are specified in the Arm Architecture
Reference Manual:

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile

The following people contributed to this patch:

• Luke Cheeseman
• Simon Tatham
• Ties Stuij

[fpetrogalli]

Hi @stuij ,

thank you for working on this.

Would it make sense to add a test that includes the header file you have created?

Regards,

Francesco

clang/include/clang/Basic/arm_neon_incl.td

293

What is this for?

clang/utils/TableGen/NeonEmitter.cpp

628

Remove me.

2198

Is this related to the changes for bfloat? Or is it a just a refactoring that it is nice to have? If the latter, please consider submitting it as a separate patch. If both refactoring and BF16 related, at the moment it is not possible to see clearly which changes are BF16 specific, so please do submit the refactoring first.

2617

Missing \n

[SjoerdMeijer]

clang/include/clang/Basic/arm_bf16.td

2

typo: fp16 - > bf16?
Here, and a few more places, or is it intentional? If so, I guess that can be a bit confusing?

clang/utils/TableGen/NeonEmitter.cpp

2411

I am a bit confused here, we already have a runFP16, I am missing something?

2416

I can't remember the outcome, but I had a discussion with @sdesmalen about this license, if this should be the new or old copyright notice. I believe, but am not certain, that this should be the new one.

[stuij]

adhere to patch attribution conventions

[stuij]

clang/include/clang/Basic/arm_bf16.td

2

Thanks for that. I went through the patch and I only found the mistake in this file.

clang/include/clang/Basic/arm_neon_incl.td

293

Thanks, that is actually part of another patch. I will remove.

clang/utils/TableGen/NeonEmitter.cpp

2198

Yes, related to bfloat. We're emitting that code twice now.

I can make a new patch, but I'm not sure if the effort justifies the in my mind small amount of gain in clarity. It's basically just pasting the removed part on the left into this function. If you disagree, tell me, and I will create the extra patch.

2411

The fairly similar runBF16 fn was added. The way git interprets this is probably a bit confusing. There's still just one runFP16 function.

2416

I'll change it, and will ping @sdesmalen to be sure.

[fpetrogalli]

clang/utils/TableGen/NeonEmitter.cpp

2198

Thank you. I didn't notice you were invoking this twice. Make sense to have it in a separate function, in this patch.

[stuij]

addressed review comments, most of all changed license header on the generated bfloat file

[stuij]

clang/utils/TableGen/NeonEmitter.cpp

2416

Just to be clear: I've changed the BF16 text, not the FP16 text, which should stay the same.

[stuij]

replanted arg passing test from other patch

[stuij]

add bfloat header test

[fpetrogalli]

LGTM! Might be worth waiting an extra day or two before submitting to make sure the people who provided extra feedback are happy.

[SjoerdMeijer]

No objections.
Some nits inlined, which you can ignore if you think they are not correct.

clang/include/clang/Basic/arm_neon_incl.td

218

nit: perhaps bfloat16?

240

nit: perhaps BFloat16?

clang/test/Preprocessor/aarch64-target-features.c

378

nit: BFloat16

clang/test/Preprocessor/arm-target-features.c

855

same

clang/utils/TableGen/NeonEmitter.cpp

747

same?

[stuij]

addressed remaining nits

clang/include/clang/Basic/arm_bf16.td

+//===--- arm_fp16.td - ARM FP16 compiler interface ------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file defines the TableGen definitions from which the ARM FP16 header
+//  file will be generated.
+//
+//===----------------------------------------------------------------------===//
+
+include "arm_neon_incl.td"

clang/include/clang/Basic/arm_neon_incl.td

 // c: char
 // s: short
 // i: int
 // l: long
 // k: 128-bit long
 // f: float
 // h: half-float
 // d: double
+// b: bfloat
 //
 // Typespec modifiers
 // ------------------
 // S: scalar, only used for function mangling.
 // U: unsigned
 // Q: 128b
 // H: 128b without mangling 'q'
 // P: polynomial
 //
 // Prototype modifiers
 // -------------------
 // prototype: return (arg, arg, ...)
 //
 // Each type modifier is either a single character, or a group surrounded by
 // parentheses.
 //
 // .: default
 // v: change to void category.
 // S: change to signed integer category.
 // U: change to unsigned integer category.
 // F: change to floating category.
+// B: change to BFloat
 // P: change to polynomial category.
 // p: change polynomial to equivalent integer category. Otherwise nop.
 //
 // >: double element width (vector size unchanged).
 // <: half element width (vector size unchanged).
 //
 // 1: change to scalar.
 // 2: change to struct of two vectors.
class Inst <string n, string p, string t, Operation o> {
   string InstName = "";
 
   // Certain intrinsics even though they are not a WOpInst or LOpInst,
   // generate a WOpInst/LOpInst instruction (see below for definition
   // of a WOpInst/LOpInst). For testing purposes we need to know
   // this. Ex: vset_lane which outputs vmov instructions.
   bit isHiddenWInst = 0;
   bit isHiddenLInst = 0;
+
+  string CartesianProductWith = "";
 }
 
 // The following instruction classes are implemented via builtins.
 // These declarations are used to generate Builtins.def:
 //
 // SInst: Instruction with signed/unsigned suffix (e.g., "s8", "u8", "p8")
 // IInst: Instruction with generic integer suffix (e.g., "i8")
 // WInst: Instruction with only bit size suffix (e.g., "8")

clang/lib/Basic/Targets/AArch64.cpp

 //===--- AArch64.cpp - Implement AArch64 target feature support -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (HasMTE)
     Builder.defineMacro("__ARM_FEATURE_MEMORY_TAGGING", "1");
 
   if (HasTME)
     Builder.defineMacro("__ARM_FEATURE_TME", "1");
 
   if (HasMatMul)
     Builder.defineMacro("__ARM_FEATURE_MATMUL_INT8", "1");
 
+  if (HasBFloat16) {
+    Builder.defineMacro("__ARM_FEATURE_BF16", "1");
+    Builder.defineMacro("__ARM_FEATURE_BF16_VECTOR_ARITHMETIC", "1");
+    Builder.defineMacro("__ARM_BF16_FORMAT_ALTERNATIVE", "1");
+  }
+
   if ((FPU & NeonMode) && HasFP16FML)
     Builder.defineMacro("__ARM_FEATURE_FP16FML", "1");
 
   if (Opts.hasSignReturnAddress()) {
     // Bitmask:
     // 0: Protection using the A key
     // 1: Protection using the B key
     // 2: Protection including leaf functions

clang/lib/Basic/Targets/ARM.cpp

 //===--- ARM.cpp - Implement ARM target feature support -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
void ARMTargetInfo::getTargetDefines(const LangOptions &Opts,
 
   // Armv8.2-A dot product intrinsics
   if (DotProd)
     Builder.defineMacro("__ARM_FEATURE_DOTPROD", "1");
 
   if (HasMatMul)
     Builder.defineMacro("__ARM_FEATURE_MATMUL_INT8", "1");
 
+  if (HasBFloat16) {
+    Builder.defineMacro("__ARM_FEATURE_BF16", "1");
+    Builder.defineMacro("__ARM_FEATURE_BF16_VECTOR_ARITHMETIC", "1");
+    Builder.defineMacro("__ARM_BF16_FORMAT_ALTERNATIVE", "1");
+  }
+
   switch (ArchKind) {
   default:
     break;
   case llvm::ARM::ArchKind::ARMV8_1A:
     getTargetDefinesARMV81A(Opts, Builder);
     break;
   case llvm::ARM::ArchKind::ARMV8_2A:
     getTargetDefinesARMV82A(Opts, Builder);

clang/lib/Headers/CMakeLists.txt

 
 # Generate header files and copy them to the build directory
 # Generate arm_neon.h
 clang_generate_header(-gen-arm-neon arm_neon.td arm_neon.h)
 # Generate arm_fp16.h
 clang_generate_header(-gen-arm-fp16 arm_fp16.td arm_fp16.h)
 # Generate arm_sve.h
 clang_generate_header(-gen-arm-sve-header arm_sve.td arm_sve.h)
+# Generate arm_bf16.h
+clang_generate_header(-gen-arm-bf16 arm_bf16.td arm_bf16.h)
 # Generate arm_mve.h
 clang_generate_header(-gen-arm-mve-header arm_mve.td arm_mve.h)
 # Generate arm_cde.h
 clang_generate_header(-gen-arm-cde-header arm_cde.td arm_cde.h)
 
 add_custom_target(clang-resource-headers ALL DEPENDS ${out_files})
 set_target_properties(clang-resource-headers PROPERTIES
   FOLDER "Misc"

clang/test/Preprocessor/aarch64-target-features.c

 // CHECK-NOT: __ARM_PCS 1
 // CHECK-NOT: __ARM_PCS_VFP 1
 // CHECK-NOT: __ARM_SIZEOF_MINIMAL_ENUM 1
 // CHECK-NOT: __ARM_SIZEOF_WCHAR_T 2
 // CHECK-NOT: __ARM_FEATURE_SVE
 // CHECK-NOT: __ARM_FEATURE_DOTPROD
 // CHECK-NOT: __ARM_FEATURE_PAC_DEFAULT
 // CHECK-NOT: __ARM_FEATURE_BTI_DEFAULT
+// CHECK-NOT: __ARM_BF16_FORMAT_ALTERNATIVE 1
+// CHECK-NOT: __ARM_FEATURE_BF16 1
+// CHECK-NOT: __ARM_FEATURE_BF16_VECTOR_ARITHMETIC 1
+
+// RUN: %clang -target aarch64_be-eabi -x c -E -dM %s -o - | FileCheck %s -check-prefix CHECK-BIGENDIAN
+// CHECK-BIGENDIAN: __ARM_BIG_ENDIAN 1
 
 // RUN: %clang -target aarch64-none-linux-gnu -march=armv8-a+crypto -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-CRYPTO %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a+crypto -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-CRYPTO %s
 // CHECK-CRYPTO: __ARM_FEATURE_CRYPTO 1
 
 // RUN: %clang -target aarch64-none-linux-gnu -mcrc -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-CRC32 %s
 // RUN: %clang -target arm64-none-linux-gnu -mcrc -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-CRC32 %s
 // RUN: %clang -target aarch64-none-linux-gnu -march=armv8-a+crc -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-CRC32 %s

 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=pac-ret+leaf -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI-OFF %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=pac-ret+b-key -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI-OFF %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=pac-ret+leaf+b-key -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI-OFF %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=standard -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=bti -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI %s
 // RUN: %clang -target arm64-none-linux-gnu -march=armv8-a -mbranch-protection=pac-ret+bti -x c -E -dM %s -o - | FileCheck -check-prefix=CHECK-BTI %s
 // CHECK-BTI-OFF-NOT: __ARM_FEATURE_BTI_DEFAULT
 // CHECK-BTI:         #define __ARM_FEATURE_BTI_DEFAULT 1
+
+// ================== Check BFloat Extensions.
+// RUN: %clang -target aarch64-arm-none-eabi -march=armv8.6-a+bf16 -x c -E -dM %s -o - 2>&1 | FileCheck -check-prefix=CHECK-BFLOAT %s
+// CHECK-BFLOAT: __ARM_BF16_FORMAT_ALTERNATIVE 1
+// CHECK-BFLOAT: __ARM_FEATURE_BF16 1
+// CHECK-BFLOAT: __ARM_FEATURE_BF16_VECTOR_ARITHMETIC 1
+

clang/test/Preprocessor/arm-target-features.c

 // RUN: %clang -target armv8a-none-linux-gnu -x c -E -dM %s -o - | FileCheck -match-full-lines --check-prefix=CHECK-V8A %s
 // CHECK-V8A: #define __ARMEL__ 1
 // CHECK-V8A: #define __ARM_ARCH 8
 // CHECK-V8A: #define __ARM_ARCH_8A__ 1
 // CHECK-V8A: #define __ARM_FEATURE_CRC32 1
 // CHECK-V8A: #define __ARM_FEATURE_DIRECTED_ROUNDING 1
 // CHECK-V8A: #define __ARM_FEATURE_NUMERIC_MAXMIN 1
 // CHECK-V8A-NOT: #define __ARM_FP 0x
 // CHECK-V8A-NOT: #define __ARM_FEATURE_DOTPROD
+// CHECK-V8A-NOT: #define __ARM_BF16_FORMAT_ALTERNATIVE
+// CHECK-V8A-NOT: #define __ARM_FEATURE_BF16
+// CHECK-V8A-NOT: #define __ARM_FEATURE_BF16_VECTOR_ARITHMETIC
 
 // RUN: %clang -target armv8a-none-linux-gnueabi -x c -E -dM %s -o - | FileCheck -match-full-lines --check-prefix=CHECK-V8A-ALLOW-FP-INSTR %s
 // RUN: %clang -target armv8a-none-linux-gnueabihf -x c -E -dM %s -o - | FileCheck -match-full-lines --check-prefix=CHECK-V8A-ALLOW-FP-INSTR %s
 // CHECK-V8A-ALLOW-FP-INSTR: #define __ARMEL__ 1
 // CHECK-V8A-ALLOW-FP-INSTR: #define __ARM_ARCH 8
 // CHECK-V8A-ALLOW-FP-INSTR: #define __ARM_ARCH_8A__ 1
 // CHECK-V8A-ALLOW-FP-INSTR: #define __ARM_FEATURE_CRC32 1
 // CHECK-V8A-ALLOW-FP-INSTR: #define __ARM_FEATURE_DIRECTED_ROUNDING 1

 
 // RUN: %clang -target armv8.6a-none-none-eabi -x c -E -dM %s -o - | FileCheck -match-full-lines --check-prefix=CHECK-V86A %s
 // CHECK-V86A: #define __ARM_ARCH 8
 // CHECK-V86A: #define __ARM_ARCH_8_6A__ 1
 // CHECK-V86A: #define __ARM_ARCH_PROFILE 'A'
 
 // RUN: %clang -target arm-none-none-eabi -march=armv7-m -mfpu=softvfp -x c -E -dM %s -o - | FileCheck --check-prefix=CHECK-SOFTVFP %s
 // CHECK-SOFTVFP-NOT: #define __ARM_FP 0x
+
+// ================== Check BFloat Extensions.
+// RUN: %clang -target arm-arm-none-eabi -march=armv8.6-a+bf16 -x c -E -dM %s -o - 2>&1 | FileCheck -check-prefix=CHECK-BFLOAT %s
+// CHECK-BFLOAT: #define __ARM_BF16_FORMAT_ALTERNATIVE 1
+// CHECK-BFLOAT: #define __ARM_FEATURE_BF16 1
+// CHECK-BFLOAT: #define __ARM_FEATURE_BF16_VECTOR_ARITHMETIC 1

clang/utils/TableGen/NeonEmitter.cpp

 //===- NeonEmitter.cpp - Generate arm_neon.h for use with clang -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum EltType {
   Int32,
   Int64,
   Poly8,
   Poly16,
   Poly64,
   Poly128,
   Float16,
   Float32,
-  Float64
+  Float64,
+  BFloat16
 };
 
 } // end namespace NeonTypeFlags
 
 class NeonEmitter;
 
 //===----------------------------------------------------------------------===//
 // TypeSpec
private:
   TypeSpec TS;
 
   enum TypeKind {
     Void,
     Float,
     SInt,
     UInt,
     Poly,
+    BFloat16,
   };
   TypeKind Kind;
   bool Immediate, Constant, Pointer;
   // ScalarForMangling and NoManglingQ are really not suited to live here as
   // they are not related to the type. But they live in the TypeSpec (not the
   // prototype), so this is really the only place to store them.
   bool ScalarForMangling, NoManglingQ;
   unsigned Bitwidth, ElementBitwidth, NumVectors;
public:
   bool isFloat() const { return isFloating() && ElementBitwidth == 32; }
   bool isDouble() const { return isFloating() && ElementBitwidth == 64; }
   bool isHalf() const { return isFloating() && ElementBitwidth == 16; }
   bool isChar() const { return ElementBitwidth == 8; }
   bool isShort() const { return isInteger() && ElementBitwidth == 16; }
   bool isInt() const { return isInteger() && ElementBitwidth == 32; }
   bool isLong() const { return isInteger() && ElementBitwidth == 64; }
   bool isVoid() const { return Kind == Void; }
+  bool isBFloat16() const { return Kind == BFloat16; }
   unsigned getNumElements() const { return Bitwidth / ElementBitwidth; }
   unsigned getSizeInBits() const { return Bitwidth; }
   unsigned getElementSizeInBits() const { return ElementBitwidth; }
   unsigned getNumVectors() const { return NumVectors; }
 
   //
   // Mutator functions
   //
public:
   }
 
   // run - Emit arm_neon.h.inc
   void run(raw_ostream &o);
 
   // runFP16 - Emit arm_fp16.h.inc
   void runFP16(raw_ostream &o);
 
-  // runHeader - Emit all the __builtin prototypes used in arm_neon.h
-  // and arm_fp16.h
+  // runBF16 - Emit arm_bf16.h.inc
+  void runBF16(raw_ostream &o);
+
+  // runHeader - Emit all the __builtin prototypes used in arm_neon.h,
+  // arm_fp16.h and arm_bf16.h
   void runHeader(raw_ostream &o);
 
   // runTests - Emit tests for all the Neon intrinsics.
   void runTests(raw_ostream &o);
 };
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 // Type implementation
 //===----------------------------------------------------------------------===//
 
 std::string Type::str() const {
   if (isVoid())
     return "void";
   std::string S;
 
   if (isInteger() && !isSigned())
     S += "u";
 
   if (isPoly())
     S += "poly";
   else if (isFloating())
     S += "float";
+  else if (isBFloat16())
+    S += "bfloat";
   else
     S += "int";
 
   S += utostr(ElementBitwidth);
   if (isVector())
     S += "x" + utostr(getNumElements());
+
   if (NumVectors > 1)
     S += "x" + utostr(NumVectors);
   S += "_t";
 
   if (Constant)
     S += " const";
   if (Pointer)
     S += " *";
if (isPointer()) {
     switch (ElementBitwidth) {
     case 8: S += "c"; break;
     case 16: S += "s"; break;
     case 32: S += "i"; break;
     case 64: S += "Wi"; break;
     case 128: S += "LLLi"; break;
     default: llvm_unreachable("Unhandled case!");
     }
-  else
+  else if (isBFloat16()) {
+    assert(ElementBitwidth == 16 && "BFloat16 can only be 16 bits");
+    S += "y";
+  } else
     switch (ElementBitwidth) {
     case 16: S += "h"; break;
     case 32: S += "f"; break;
     case 64: S += "d"; break;
     default: llvm_unreachable("Unhandled case!");
     }
 
   // FIXME: NECESSARY???????????????????????????????????????????????????????????????????????
if (Addend >= 2)
       --Addend;
     Base = (unsigned)NeonTypeFlags::Poly8 + Addend;
   }
   if (isFloating()) {
     assert(Addend != 0 && "Float8 doesn't exist!");
     Base = (unsigned)NeonTypeFlags::Float16 + (Addend - 1);
   }
 
+  if (isBFloat16()) {
+    assert(Addend == 1 && "BFloat16 is only 16 bit");
+    Base = (unsigned)NeonTypeFlags::BFloat16;
+  }
+
   if (Bitwidth == 128)
     Base |= (unsigned)NeonTypeFlags::QuadFlag;
   if (isInteger() && !isSigned())
     Base |= (unsigned)NeonTypeFlags::UnsignedFlag;
 
   return Base;
 }
 
 Type Type::fromTypedefName(StringRef Name) {
   Type T;
   T.Kind = SInt;
 
   if (Name.front() == 'u') {
     T.Kind = UInt;
     Name = Name.drop_front();
   }
 
   if (Name.startswith("float")) {
     T.Kind = Float;
     Name = Name.drop_front(5);
   } else if (Name.startswith("poly")) {
     T.Kind = Poly;
     Name = Name.drop_front(4);
+  } else if (Name.startswith("bfloat")) {
+    T.Kind = BFloat16;
+    Name = Name.drop_front(6);
   } else {
     assert(Name.startswith("int"));
     Name = Name.drop_front(3);
   }
 
   unsigned I = 0;
   for (I = 0; I < Name.size(); ++I) {
     if (!isdigit(Name[I]))
case 'l':
       ElementBitwidth = 64;
       break;
     case 'k':
       ElementBitwidth = 128;
       // Poly doesn't have a 128x1 type.
       if (isPoly())
         NumVectors = 0;
       break;
+    case 'b':
+      Kind = BFloat16;
+      ElementBitwidth = 16;
+      break;
     default:
       llvm_unreachable("Unhandled type code!");
     }
   }
   assert(ElementBitwidth != ~0U && "Bad element bitwidth!");
 
   Bitwidth = Quad ? 128 : 64;
 }
case 'v':
       Kind = Void;
       break;
     case 'S':
       Kind = SInt;
       break;
     case 'U':
       Kind = UInt;
       break;
+    case 'B':
+      Kind = BFloat16;
+      ElementBitwidth = 16;
+      break;
     case 'F':
       Kind = Float;
       break;
     case 'P':
       Kind = Poly;
       break;
     case '>':
       assert(ElementBitwidth < 128);

 
 std::string Intrinsic::getInstTypeCode(Type T, ClassKind CK) const {
   char typeCode = '\0';
   bool printNumber = true;
 
   if (CK == ClassB)
     return "";
 
+  if (T.isBFloat16())
+    return "bf16";
+
   if (T.isPoly())
     typeCode = 'p';
   else if (T.isInteger())
     typeCode = T.isSigned() ? 's' : 'u';
   else
     typeCode = 'f';
 
   if (CK == ClassI) {

 }
 
 std::string Intrinsic::getBuiltinTypeStr() {
   ClassKind LocalCK = getClassKind(true);
   std::string S;
 
   Type RetT = getReturnType();
   if ((LocalCK == ClassI || LocalCK == ClassW) && RetT.isScalar() &&
-      !RetT.isFloating())
+      !RetT.isFloating() && !RetT.isBFloat16())
     RetT.makeInteger(RetT.getElementSizeInBits(), false);
 
   // Since the return value must be one type, return a vector type of the
   // appropriate width which we will bitcast.  An exception is made for
   // returning structs of 2, 3, or 4 vectors which are returned in a sret-like
   // fashion, storing them to a pointer arg.
   if (RetT.getNumVectors() > 1) {
     S += "vv*"; // void result with void* first argument
void NeonEmitter::runHeader(raw_ostream &OS) {
 
   // Generate ARM overloaded type checking code for SemaChecking.cpp
   genOverloadTypeCheckCode(OS, Defs);
 
   // Generate ARM range checking code for shift/lane immediates.
   genIntrinsicRangeCheckCode(OS, Defs);
 }
 
+static void emitNeonTypeDefs(const std::string& types, raw_ostream &OS) {
+  std::string TypedefTypes(types);
+  std::vector<TypeSpec> TDTypeVec = TypeSpec::fromTypeSpecs(TypedefTypes);
+
+  // Emit vector typedefs.
+  bool InIfdef = false;
+  for (auto &TS : TDTypeVec) {
+    bool IsA64 = false;
+    Type T(TS, ".");
+    if (T.isDouble())
+      IsA64 = true;
+
+    if (InIfdef && !IsA64) {
+      OS << "#endif\n";
+      InIfdef = false;
+    }
+    if (!InIfdef && IsA64) {
+      OS << "#ifdef __aarch64__\n";
+      InIfdef = true;
+    }
+
+    if (T.isPoly())
+      OS << "typedef __attribute__((neon_polyvector_type(";
+    else
+      OS << "typedef __attribute__((neon_vector_type(";
+
+    Type T2 = T;
+    T2.makeScalar();
+    OS << T.getNumElements() << "))) ";
+    OS << T2.str();
+    OS << " " << T.str() << ";\n";
+  }
+  if (InIfdef)
+    OS << "#endif\n";
+  OS << "\n";
+
+  // Emit struct typedefs.
+  InIfdef = false;
+  for (unsigned NumMembers = 2; NumMembers <= 4; ++NumMembers) {
+    for (auto &TS : TDTypeVec) {
+      bool IsA64 = false;
+      Type T(TS, ".");
+      if (T.isDouble())
+        IsA64 = true;
+
+      if (InIfdef && !IsA64) {
+        OS << "#endif\n";
+        InIfdef = false;
+      }
+      if (!InIfdef && IsA64) {
+        OS << "#ifdef __aarch64__\n";
+        InIfdef = true;
+      }
+
+      const char Mods[] = { static_cast<char>('2' + (NumMembers - 2)), 0};
+      Type VT(TS, Mods);
+      OS << "typedef struct " << VT.str() << " {\n";
+      OS << "  " << T.str() << " val";
+      OS << "[" << NumMembers << "]";
+      OS << ";\n} ";
+      OS << VT.str() << ";\n";
+      OS << "\n";
+    }
+  }
+  if (InIfdef)
+    OS << "#endif\n";
+}
+
 /// run - Read the records in arm_neon.td and output arm_neon.h.  arm_neon.h
 /// is comprised of type definitions and function declarations.
 void NeonEmitter::run(raw_ostream &OS) {
   OS << "/*===---- arm_neon.h - ARM Neon intrinsics "
         "------------------------------"
         "---===\n"
         " *\n"
         " * Permission is hereby granted, free of charge, to any person "
void NeonEmitter::run(raw_ostream &OS) {
   OS << "#define __ARM_NEON_H\n\n";
 
   OS << "#if !defined(__ARM_NEON)\n";
   OS << "#error \"NEON support not enabled\"\n";
   OS << "#endif\n\n";
 
   OS << "#include <stdint.h>\n\n";
 
+  OS << "#ifdef __ARM_FEATURE_BF16\n";
+  OS << "#include <arm_bf16.h>\n";
+  OS << "typedef __bf16 bfloat16_t;\n";
+  OS << "#endif\n\n";
+
   // Emit NEON-specific scalar typedefs.
   OS << "typedef float float32_t;\n";
   OS << "typedef __fp16 float16_t;\n";
 
   OS << "#ifdef __aarch64__\n";
   OS << "typedef double float64_t;\n";
   OS << "#endif\n\n";
 
   // For now, signedness of polynomial types depends on target
   OS << "#ifdef __aarch64__\n";
   OS << "typedef uint8_t poly8_t;\n";
   OS << "typedef uint16_t poly16_t;\n";
   OS << "typedef uint64_t poly64_t;\n";
   OS << "typedef __uint128_t poly128_t;\n";
   OS << "#else\n";
   OS << "typedef int8_t poly8_t;\n";
   OS << "typedef int16_t poly16_t;\n";
   OS << "typedef int64_t poly64_t;\n";
   OS << "#endif\n";
 
-  // Emit Neon vector typedefs.
-  std::string TypedefTypes(
-      "cQcsQsiQilQlUcQUcUsQUsUiQUiUlQUlhQhfQfdQdPcQPcPsQPsPlQPl");
-  std::vector<TypeSpec> TDTypeVec = TypeSpec::fromTypeSpecs(TypedefTypes);
+  emitNeonTypeDefs("cQcsQsiQilQlUcQUcUsQUsUiQUiUlQUlhQhfQfdQdPcQPcPsQPsPlQPl", OS);
 
-  // Emit vector typedefs.
-  bool InIfdef = false;
-  for (auto &TS : TDTypeVec) {
-    bool IsA64 = false;
-    Type T(TS, ".");
-    if (T.isDouble())
-      IsA64 = true;
+  OS << "#ifdef __ARM_FEATURE_BF16\n";
+  emitNeonTypeDefs("bQb", OS);
+  OS << "#endif\n\n";
 
-    if (InIfdef && !IsA64) {
-      OS << "#endif\n";
-      InIfdef = false;
-    }
-    if (!InIfdef && IsA64) {
-      OS << "#ifdef __aarch64__\n";
-      InIfdef = true;
-    }
+  OS << "#define __ai static __inline__ __attribute__((__always_inline__, "
+        "__nodebug__))\n\n";
 
-    if (T.isPoly())
-      OS << "typedef __attribute__((neon_polyvector_type(";
-    else
-      OS << "typedef __attribute__((neon_vector_type(";
+  SmallVector<Intrinsic *, 128> Defs;
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  for (auto *R : RV)
+    createIntrinsic(R, Defs);
 
-    Type T2 = T;
-    T2.makeScalar();
-    OS << T.getNumElements() << "))) ";
-    OS << T2.str();
-    OS << " " << T.str() << ";\n";
-  }
-  if (InIfdef)
-    OS << "#endif\n";
-  OS << "\n";
+  for (auto *I : Defs)
+    I->indexBody();
 
-  // Emit struct typedefs.
-  InIfdef = false;
-  for (unsigned NumMembers = 2; NumMembers <= 4; ++NumMembers) {
-    for (auto &TS : TDTypeVec) {
-      bool IsA64 = false;
-      Type T(TS, ".");
-      if (T.isDouble())
-        IsA64 = true;
+  llvm::stable_sort(Defs, llvm::deref<std::less<>>());
 
-      if (InIfdef && !IsA64) {
-        OS << "#endif\n";
-        InIfdef = false;
+  // Only emit a def when its requirements have been met.
+  // FIXME: This loop could be made faster, but it's fast enough for now.
+  bool MadeProgress = true;
+  std::string InGuard;
+  while (!Defs.empty() && MadeProgress) {
+    MadeProgress = false;
+
+    for (SmallVector<Intrinsic *, 128>::iterator I = Defs.begin();
+         I != Defs.end(); /*No step*/) {
+      bool DependenciesSatisfied = true;
+      for (auto *II : (*I)->getDependencies()) {
+        if (llvm::is_contained(Defs, II))
+          DependenciesSatisfied = false;
       }
-      if (!InIfdef && IsA64) {
-        OS << "#ifdef __aarch64__\n";
-        InIfdef = true;
+      if (!DependenciesSatisfied) {
+        // Try the next one.
+        ++I;
+        continue;
       }
 
-      const char Mods[] = { static_cast<char>('2' + (NumMembers - 2)), 0};
-      Type VT(TS, Mods);
-      OS << "typedef struct " << VT.str() << " {\n";
-      OS << "  " << T.str() << " val";
-      OS << "[" << NumMembers << "]";
-      OS << ";\n} ";
-      OS << VT.str() << ";\n";
-      OS << "\n";
+      // Emit #endif/#if pair if needed.
+      if ((*I)->getGuard() != InGuard) {
+        if (!InGuard.empty())
+          OS << "#endif\n";
+        InGuard = (*I)->getGuard();
+        if (!InGuard.empty())
+          OS << "#if " << InGuard << "\n";
+      }
+
+      // Actually generate the intrinsic code.
+      OS << (*I)->generate();
+
+      MadeProgress = true;
+      I = Defs.erase(I);
     }
   }
-  if (InIfdef)
+  assert(Defs.empty() && "Some requirements were not satisfied!");
+  if (!InGuard.empty())
     OS << "#endif\n";
+
   OS << "\n";
+  OS << "#undef __ai\n\n";
+  OS << "#endif /* __ARM_NEON_H */\n";
+}
+
+/// run - Read the records in arm_fp16.td and output arm_fp16.h.  arm_fp16.h
+/// is comprised of type definitions and function declarations.
+void NeonEmitter::runFP16(raw_ostream &OS) {
+  OS << "/*===---- arm_fp16.h - ARM FP16 intrinsics "
+        "------------------------------"
+        "---===\n"
+        " *\n"
+        " * Permission is hereby granted, free of charge, to any person "
+        "obtaining a copy\n"
+        " * of this software and associated documentation files (the "
+        "\"Software\"), to deal\n"
+        " * in the Software without restriction, including without limitation "
+        "the rights\n"
+        " * to use, copy, modify, merge, publish, distribute, sublicense, "
+        "and/or sell\n"
+        " * copies of the Software, and to permit persons to whom the Software "
+        "is\n"
+        " * furnished to do so, subject to the following conditions:\n"
+        " *\n"
+        " * The above copyright notice and this permission notice shall be "
+        "included in\n"
+        " * all copies or substantial portions of the Software.\n"
+        " *\n"
+        " * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, "
+        "EXPRESS OR\n"
+        " * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF "
+        "MERCHANTABILITY,\n"
+        " * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT "
+        "SHALL THE\n"
+        " * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR "
+        "OTHER\n"
+        " * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, "
+        "ARISING FROM,\n"
+        " * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER "
+        "DEALINGS IN\n"
+        " * THE SOFTWARE.\n"
+        " *\n"
+        " *===-----------------------------------------------------------------"
+        "---"
+        "---===\n"
+        " */\n\n";
+
+  OS << "#ifndef __ARM_FP16_H\n";
+  OS << "#define __ARM_FP16_H\n\n";
+
+  OS << "#include <stdint.h>\n\n";
+
+  OS << "typedef __fp16 float16_t;\n";
 
   OS << "#define __ai static __inline__ __attribute__((__always_inline__, "
         "__nodebug__))\n\n";
 
   SmallVector<Intrinsic *, 128> Defs;
   std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
   for (auto *R : RV)
     createIntrinsic(R, Defs);
while (!Defs.empty() && MadeProgress) {
     }
   }
   assert(Defs.empty() && "Some requirements were not satisfied!");
   if (!InGuard.empty())
     OS << "#endif\n";
 
   OS << "\n";
   OS << "#undef __ai\n\n";
-  OS << "#endif /* __ARM_NEON_H */\n";
+  OS << "#endif /* __ARM_FP16_H */\n";
 }
 
-/// run - Read the records in arm_fp16.td and output arm_fp16.h.  arm_fp16.h
-/// is comprised of type definitions and function declarations.
-void NeonEmitter::runFP16(raw_ostream &OS) {
-  OS << "/*===---- arm_fp16.h - ARM FP16 intrinsics "
+void NeonEmitter::runBF16(raw_ostream &OS) {
+  OS << "/*===---- arm_bf16.h - ARM BF16 intrinsics "
         "------------------------------"
         "---===\n"
         " *\n"
         " * Permission is hereby granted, free of charge, to any person "
         "obtaining a copy\n"
         " * of this software and associated documentation files (the "
         "\"Software\"), to deal\n"
         " * in the Software without restriction, including without limitation "
         "the rights\n"
         " * to use, copy, modify, merge, publish, distribute, sublicense, "
         "and/or sell\n"
         " * copies of the Software, and to permit persons to whom the Software "
         "is\n"
         " * furnished to do so, subject to the following conditions:\n"
         " *\n"
         " * The above copyright notice and this permission notice shall be "
         "included in\n"
         " * all copies or substantial portions of the Software.\n"
         " *\n"
         " * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, "
         "EXPRESS OR\n"
OS << "/*===---- arm_bf16.h - ARM BF16 intrinsics "
         "DEALINGS IN\n"
         " * THE SOFTWARE.\n"
         " *\n"
         " *===-----------------------------------------------------------------"
         "---"
         "---===\n"
         " */\n\n";
 
-  OS << "#ifndef __ARM_FP16_H\n";
-  OS << "#define __ARM_FP16_H\n\n";
+  OS << "#ifndef __ARM_BF16_H\n";
+  OS << "#define __ARM_BF16_H\n\n";
 
-  OS << "#include <stdint.h>\n\n";
-
-  OS << "typedef __fp16 float16_t;\n";
+  OS << "typedef __bf16 bfloat16_t;\n";
 
   OS << "#define __ai static __inline__ __attribute__((__always_inline__, "
         "__nodebug__))\n\n";
 
   SmallVector<Intrinsic *, 128> Defs;
   std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
   for (auto *R : RV)
     createIntrinsic(R, Defs);
while (!Defs.empty() && MadeProgress) {
     }
   }
   assert(Defs.empty() && "Some requirements were not satisfied!");
   if (!InGuard.empty())
     OS << "#endif\n";
 
   OS << "\n";
   OS << "#undef __ai\n\n";
-  OS << "#endif /* __ARM_FP16_H */\n";
+
+  OS << "#endif";
 }
 
 void clang::EmitNeon(RecordKeeper &Records, raw_ostream &OS) {
   NeonEmitter(Records).run(OS);
 }
 
 void clang::EmitFP16(RecordKeeper &Records, raw_ostream &OS) {
   NeonEmitter(Records).runFP16(OS);
 }
 
+void clang::EmitBF16(RecordKeeper &Records, raw_ostream &OS) {
+  NeonEmitter(Records).runBF16(OS);
+}
+
 void clang::EmitNeonSema(RecordKeeper &Records, raw_ostream &OS) {
   NeonEmitter(Records).runHeader(OS);
 }
 
 void clang::EmitNeonTest(RecordKeeper &Records, raw_ostream &OS) {
   llvm_unreachable("Neon test generation no longer implemented!");
 }

clang/utils/TableGen/TableGen.cpp

 //===- TableGen.cpp - Top-Level TableGen implementation for Clang ---------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum ActionType {
   GenClangCommentHTMLTags,
   GenClangCommentHTMLTagsProperties,
   GenClangCommentHTMLNamedCharacterReferences,
   GenClangCommentCommandInfo,
   GenClangCommentCommandList,
   GenClangOpenCLBuiltins,
   GenArmNeon,
   GenArmFP16,
+  GenArmBF16,
   GenArmNeonSema,
   GenArmNeonTest,
   GenArmMveHeader,
   GenArmMveBuiltinDef,
   GenArmMveBuiltinSema,
   GenArmMveBuiltinCG,
   GenArmMveBuiltinAliases,
   GenArmSveHeader,
cl::values(
                    "are used in documentation comments"),
         clEnumValN(GenClangCommentCommandList, "gen-clang-comment-command-list",
                    "Generate list of commands that are used in "
                    "documentation comments"),
         clEnumValN(GenClangOpenCLBuiltins, "gen-clang-opencl-builtins",
                    "Generate OpenCL builtin declaration handlers"),
         clEnumValN(GenArmNeon, "gen-arm-neon", "Generate arm_neon.h for clang"),
         clEnumValN(GenArmFP16, "gen-arm-fp16", "Generate arm_fp16.h for clang"),
+        clEnumValN(GenArmBF16, "gen-arm-bf16", "Generate arm_bf16.h for clang"),
         clEnumValN(GenArmNeonSema, "gen-arm-neon-sema",
                    "Generate ARM NEON sema support for clang"),
         clEnumValN(GenArmNeonTest, "gen-arm-neon-test",
                    "Generate ARM NEON tests for clang"),
         clEnumValN(GenArmSveHeader, "gen-arm-sve-header",
                    "Generate arm_sve.h for clang"),
         clEnumValN(GenArmSveBuiltins, "gen-arm-sve-builtins",
                    "Generate arm_sve_builtins.inc for clang"),
case GenClangOpenCLBuiltins:
     EmitClangOpenCLBuiltins(Records, OS);
     break;
   case GenArmNeon:
     EmitNeon(Records, OS);
     break;
   case GenArmFP16:
     EmitFP16(Records, OS);
     break;
+  case GenArmBF16:
+    EmitBF16(Records, OS);
+    break;
   case GenArmNeonSema:
     EmitNeonSema(Records, OS);
     break;
   case GenArmNeonTest:
     EmitNeonTest(Records, OS);
     break;
   case GenArmMveHeader:
     EmitMveHeader(Records, OS);

clang/utils/TableGen/TableGenBackends.h

 //===- TableGenBackends.h - Declarations for Clang TableGen Backends ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 void EmitClangCommentCommandInfo(llvm::RecordKeeper &Records,
                                  llvm::raw_ostream &OS);
 void EmitClangCommentCommandList(llvm::RecordKeeper &Records,
                                  llvm::raw_ostream &OS);
 void EmitClangOpcodes(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitNeon(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitFP16(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+void EmitBF16(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonSema(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonTest(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeon2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonSema2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonTest2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitSveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);