[PATCH 3/4][Sema][AArch64] Add codegen for arm_sve_vector_bits attribute

[c-rhodes]

Summary

This patch implements codegen for the 'arm_sve_vector_bits' type
attribute, defined by the Arm C Language Extensions (ACLE) for SVE [1].
The purpose of this attribute is to define fixed-length (VLST) versions
of existing sizeless types (VLAT).

Implemented in this patch is the lowering of VLSTs to valid types.
VLSTs (unlike VLATs) can be used in globals, members of structs
and unions, and arrays. To support this in this patch we lower VLSTs to
arrays. For example, in the following C code:

#if __ARM_FEATURE_SVE_BITS==512
typedef svint32_t fixed_svint32_t __attribute__((arm_sve_vector_bits(512)));
struct struct_int32 {
  fixed_int32_t x;
} struct_int32;
#endif

the struct is lowered to:

%struct.struct_int32 = type { [16 x i32] }

where the member 'x' is a fixed-length variant of 'svint32_t' that
contains exactly 512 bits.

When loading from a VLST to a VLAT, or when storing a VLAT to a VLST,
the address is bitcasted, e.g.

bitcast [N x i8]* %addr.ptr to <vscale x 16 x i8>*

Patch contains changes by Cullen Rhodes and Sander de Smalen.

[1] https://developer.arm.com/documentation/100987/latest

[c-rhodes]

Changes:

• Use fixed-length instead of fixed-width in naming.

[efriedma]

What's the tradeoff of representing these in IR as vscale'ed vector types, as opposed to fixed-wdith vector types?

[sdesmalen]

In D83553#2145227, @efriedma wrote:

What's the tradeoff of representing these in IR as vscale'ed vector types, as opposed to fixed-wdith vector types?

If you mean alloca's for single vectors, then that's partly to do with better test coverage of the stackframe layout with scalable vectors until we can start testing that with auto-vectorized code. Also, currently LLVM only implements the VL-scaled addressing modes for the scalable IR type and would otherwise always use base addressing mode if the type is fixed-width (basereg = sp/fp + byteoffset; ld1 dstreg, [basereg, #0 mul VL]), so until we add those smarts, code quality will probably be better.

clang/lib/CodeGen/CGRecordLayoutBuilder.cpp

135

Can you add comments for the false and true parameters, e.g. /*ForBitField*/ false, /*EnforceFixedLengthSVEAttribute*/ true

clang/lib/CodeGen/CodeGenModule.cpp

3731

same here.

clang/lib/CodeGen/CodeGenTypes.cpp

81

nit: s/getFixedSVETypeForMemory/getFixedLengthSVETypeForMemory/

94

Can you add a comment explaining why SveBool gets an i8 element type for it's memory type?

clang/lib/CodeGen/CodeGenTypes.h

137

Can you add a comment here to explain what EnforceFixedLengthSVEAttribute does?

[efriedma]

If you mean alloca's for single vectors

I was really referring to the IR values themselves, not the memory representation. Since the width of the vectors is known, you could emit IR without any mention of scalable types at all (assuming the backend was extended to handle the intrinsics).

The choice of vscale'ed types for variables is also interesting, though. Thanks for the explanation.

[sdesmalen]

In D83553#2148429, @efriedma wrote:

If you mean alloca's for single vectors

I was really referring to the IR values themselves, not the memory representation. Since the width of the vectors is known, you could emit IR without any mention of scalable types at all (assuming the backend was extended to handle the intrinsics).

That's right, the reason is because codegen of the intrinsics currently only works on scalable types. By casting the pointer to a vscale-pointer, all IR values are always scalable so we don't need to worry about doing things like reinterpet_cast from a scalable to fixed-width vector, or vice versa.

[efriedma]

In D83553#2151591, @sdesmalen wrote:

In D83553#2148429, @efriedma wrote:

If you mean alloca's for single vectors

I was really referring to the IR values themselves, not the memory representation. Since the width of the vectors is known, you could emit IR without any mention of scalable types at all (assuming the backend was extended to handle the intrinsics).

That's right, the reason is because codegen of the intrinsics currently only works on scalable types. By casting the pointer to a vscale-pointer, all IR values are always scalable so we don't need to worry about doing things like reinterpet_cast from a scalable to fixed-width vector, or vice versa.

I guess that's reasonable. I suspect we're eventually going to end up with that functionality anyway, but maybe not right now.

clang/lib/CodeGen/CodeGenTypes.h

138

The default for EnforceFixedLengthSVEAttribute seems backwards; I would expect that almost everywhere that calls ConvertTypeForMem actually wants the fixed-length type. The scalable type only exists in registers.

[c-rhodes]

Changes:

• Rebased.
• Added comments for args in calls to ConvertTypeForMem when EnforceFixedLengthSVEAttribute is set and documented EnforceFixedLengthSVEAttribute.
• s/getFixedSVETypeForMemory/getFixedLengthSVETypeForMemory/
• Documented memory representation for fixed-length predicates.

[c-rhodes]

clang/lib/CodeGen/CodeGenTypes.h

138

The default for EnforceFixedLengthSVEAttribute seems backwards; I would expect that almost everywhere that calls ConvertTypeForMem actually wants the fixed-length type. The scalable type only exists in registers.

It has no effect unless T->isVLST() so I think it makes sense.

[efriedma]

clang/lib/CodeGen/CodeGenTypes.h

138

My question is "why is the current default for EnforceFixedLengthSVEAttribute correct?" You answer for that is "because VLST types are rare"? I'm not sure how that's related.

Essentially, the issue is that ConvertTypeForMem means "I'm allocating something in memory; what is its type?". Except for a few places where we've specifically added handling to make it work, the code assumes scalable types don't exist. So in most places, we want the fixed version. With the current default, I'm afraid we're going to end up with weird failures with various constructs you haven't tested.

I guess if there's some large number of places where the current default is actually beneficial, the current patch wouldn't make it obvious, but my intuition is that are few places like that.

[c-rhodes]

Change the default for EnforceFixedLengthSVEAttribute.

[c-rhodes]

clang/lib/CodeGen/CodeGenTypes.h

138

My question is "why is the current default for EnforceFixedLengthSVEAttribute correct?" You answer for that is "because VLST types are rare"? I'm not sure how that's related.

Essentially, the issue is that ConvertTypeForMem means "I'm allocating something in memory; what is its type?". Except for a few places where we've specifically added handling to make it work, the code assumes scalable types don't exist. So in most places, we want the fixed version. With the current default, I'm afraid we're going to end up with weird failures with various constructs you haven't tested.

Sorry I misunderstood what you meant. I think you're right that does make sense, I guess the benefit of defaulting to false is (hopefully) those failures would have come to our attention and we could explicitly add test cases for those, although I suspect the same applies with your suggestion with the added benefit of us supporting constructs we haven't explicitly tested as you say. Anyhow, I've made the change, cheers!

[efriedma]

clang/lib/CodeGen/CGExpr.cpp

153

Do we need to bitcast the result of CreateTempAlloca to a pointer to the array type? I'm concerned that we might miss a bitcast if the source code uses the address of the variable.

clang/lib/CodeGen/CodeGenModule.cpp

3985

EmitNullConstant should just do the right thing, I think, now that we've changed the default behavior of ConvertTypeForMem.

clang/lib/CodeGen/CodeGenTypes.cpp

147

I think the default handling for constant arrays should do the right thing, now that we've changed the default behavior of ConvertTypeForMem.

[c-rhodes]

clang/lib/CodeGen/CGExpr.cpp

153

Do we need to bitcast the result of CreateTempAlloca to a pointer to the array type? I'm concerned that we might miss a bitcast if the source code uses the address of the variable.

You were right, I've spent some time investigating this. The current implementation crashes on:

fixed_int32_t global;
fixed_int32_t address_of_global() {
  fixed_int32_t *global_ptr;
  global_ptr = &global;
  return *global_ptr;
}

the reason being global is represented as an ArrayType whereas the pointer global_ptr is scalable:

@global = global [4 x i32] zeroinitializer, align 16
%global_ptr = alloca <vscale x 4 x i32>*, align 8

so when storing the address of global to global_ptr the store it tries to create causes a crash:

store [4 x i32]* @global, <vscale x 4 x i32>** %global_ptr, align 8

I tried your suggestion to bitcast to alloca to the array type in CreateMemTemp but found for that example it isn't called, it's created by a call to CreateTempAlloca in CGDecl.cpp (EmitAutoVarAlloca). CreateTempAlloca takes an llvm::Type *Ty so it's not as straightforward as doing a bitcast there, although I found it could be done in EmitAutoVarAlloca but it means having to handle this is two places I'm aware of and potentially others I haven't hit. In this case as well it also required looking through the pointer to see if the pointee was a VLST then doing a bitcast.

I've also experimented with representing allocas as fixed-length arrays to see if that will make it any easier and it does simplify the patch a little. It does require handling PointerType in ConvertTypeForMem however as we do for ConstantArray, same issue I mentioned in response to your other comment about removing that.

I planning to update the patch with that implementation but I've just found another issue:

fixed_int32_t arr[3];
fixed_int32_t *z() {
  fixed_int32_t *array_ptr;
  array_ptr = &arr[0];
  return array_ptr;
}

trying to create a store:
store [4 x i32]* %0, <vscale x 4 x i32>** %retval, align 8

although this is done in CGStmt.cpp as it's for a retval so it looks like a bitcast could also be required there.

clang/lib/CodeGen/CodeGenModule.cpp

3985

EmitNullConstant should just do the right thing, I think, now that we've changed the default behavior of ConvertTypeForMem.

Good spot, these changes can be removed

clang/lib/CodeGen/CodeGenTypes.cpp

147

I think the default handling for constant arrays should do the right thing, now that we've changed the default behavior of ConvertTypeForMem.

ConvertType looks at the canonical type so the type attribute is lost.

[efriedma]

clang/lib/CodeGen/CGExpr.cpp

153

I think a fixed_int32_t * needs to be converted to [4 x i32]*, for the sake of consistency... but see also my other comment.

clang/lib/CodeGen/CodeGenTypes.cpp

147

That sounds like a bug in the AST: since isVLST() affects the semantics of the type, it needs to be part of the canonical type. Otherwise you're going to be finding bugs all over in both Sema and CodeGen.

[c-rhodes]

I've posted a prototype D85128 with an alternative implementation, given it's quite different to this patch I've posted it as a separate patch and am abandoning this one. See new patch for more details, cheers

clang/lib/CodeGen/CGExpr.cpp

 
 Address CodeGenFunction::CreateMemTemp(QualType Ty, CharUnits Align,
                                        const Twine &Name, Address *Alloca) {
   Address Result = CreateTempAlloca(ConvertTypeForMem(Ty), Align, Name,
                                     /*ArraySize=*/nullptr, Alloca);
 
   if (Ty->isConstantMatrixType()) {
     auto *ArrayTy = cast<llvm::ArrayType>(Result.getType()->getElementType());
     auto *VectorTy = llvm::FixedVectorType::get(ArrayTy->getElementType(),
                                                 ArrayTy->getNumElements());
 
     Result = Address(
         Builder.CreateBitCast(Result.getPointer(), VectorTy->getPointerTo()),
         Result.getAlignment());
   }
   return Result;
 }
if (Ty->isVectorType()) {
         // Shuffle vector to get vec3.
         V = Builder.CreateShuffleVector(V, llvm::UndefValue::get(vec4Ty),
                                         ArrayRef<int>{0, 1, 2}, "extractVec");
         return EmitFromMemory(V, Ty);
       }
     }
   }
 
+  // If we're loading from a fixed-length address to a scalable vector, bitcast
+  // the pointer, e.g. bitcast [N x i8]* %addr.ptr to <vscale x 16 x i8>*
+  if (Ty->isVLST()) {
+    llvm::Type *VecTy = ConvertType(Ty);
+    Addr = Builder.CreateElementBitCast(Addr, VecTy, "cast.to.scalable");
+  }
+
   // Atomic operations have to be done on integral types.
   LValue AtomicLValue =
       LValue::MakeAddr(Addr, Ty, getContext(), BaseInfo, TBAAInfo);
   if (Ty->isAtomicType() || LValueIsSuitableForInlineAtomic(AtomicLValue)) {
     return EmitAtomicLoad(AtomicLValue, Loc).getScalarVal();
   }
 
   llvm::LoadInst *Load = Builder.CreateLoad(Addr, Volatile);
if (Ty->isVectorType()) {
         SrcTy = llvm::FixedVectorType::get(VecTy->getElementType(), 4);
       }
       if (Addr.getElementType() != SrcTy) {
         Addr = Builder.CreateElementBitCast(Addr, SrcTy, "storetmp");
       }
     }
   }
 
+  // If we're storing a scalable vector to a fixed-length address, bitcast the
+  // pointer, e.g. bitcast [N x i8]* %addr.ptr to <vscale x 16 x i8>*
+  if (Ty->isVLST()) {
+    llvm::Type *VecTy = ConvertType(Ty);
+    Addr = Builder.CreateElementBitCast(Addr, VecTy, "cast.to.scalable");
+  }
+
   Value = EmitToMemory(Value, Ty);
 
   LValue AtomicLValue =
       LValue::MakeAddr(Addr, Ty, getContext(), BaseInfo, TBAAInfo);
   if (Ty->isAtomicType() ||
       (!isInit && LValueIsSuitableForInlineAtomic(AtomicLValue))) {
     EmitAtomicStore(RValue::get(Value), AtomicLValue, isInit);
     return;

clang/lib/CodeGen/CGRecordLayoutBuilder.cpp

llvm::Type *getByteArrayType(CharUnits NumBytes) {
     assert(!NumBytes.isZero() && "Empty byte arrays aren't allowed.");
     llvm::Type *Type = llvm::Type::getInt8Ty(Types.getLLVMContext());
     return NumBytes == CharUnits::One() ? Type :
         (llvm::Type *)llvm::ArrayType::get(Type, NumBytes.getQuantity());
   }
   /// Gets the storage type for a field decl and handles storage
   /// for itanium bitfields that are smaller than their declared type.
   llvm::Type *getStorageType(const FieldDecl *FD) {
-    llvm::Type *Type = Types.ConvertTypeForMem(FD->getType());
+    llvm::Type *Type = Types.ConvertTypeForMem(FD->getType(), false, true);
     if (!FD->isBitField()) return Type;
     if (isDiscreteBitFieldABI()) return Type;
     return getIntNType(std::min(FD->getBitWidthValue(Context),
                              (unsigned)Context.toBits(getSize(Type))));
   }
   /// Gets the llvm Basesubobject type from a CXXRecordDecl.
   llvm::Type *getStorageType(const CXXRecordDecl *RD) {
     return Types.getCGRecordLayout(RD).getBaseSubobjectLLVMType();

clang/lib/CodeGen/CodeGenModule.cpp

 /// that an actual global with type Ty will be returned, not conversion of a
 /// variable with the same mangled name but some other type.
 llvm::Constant *CodeGenModule::GetAddrOfGlobalVar(const VarDecl *D,
                                                   llvm::Type *Ty,
                                            ForDefinition_t IsForDefinition) {
   assert(D->hasGlobalStorage() && "Not a global variable");
   QualType ASTTy = D->getType();
   if (!Ty)
-    Ty = getTypes().ConvertTypeForMem(ASTTy);
+    Ty = getTypes().ConvertTypeForMem(ASTTy, false, true);
 
   llvm::PointerType *PTy =
     llvm::PointerType::get(Ty, getContext().getTargetAddressSpace(ASTTy));
 
   StringRef MangledName = getMangledName(D);
   return GetOrCreateLLVMGlobal(MangledName, PTy, D, IsForDefinition);
 }
 
else if (!InitExpr) {
     //
     // Note that tentative definitions are only emitted at the end of
     // a translation unit, so they should never have incomplete
     // type. In addition, EmitTentativeDefinition makes sure that we
     // never attempt to emit a tentative definition if a real one
     // exists. A use may still exists, however, so we still may need
     // to do a RAUW.
     assert(!ASTTy->isIncompleteType() && "Unexpected incomplete type");
+    // Lower global scalable vectors to fixed-length vectors.
+    if (auto MemTy = getTypes().getFixedSVETypeForMemory(ASTTy.getTypePtr()))
+      Init = llvm::Constant::getNullValue(*MemTy);
+    else
       Init = EmitNullConstant(D->getType());
   } else {
     initializedGlobalDecl = GlobalDecl(D);
     emitter.emplace(*this);
     Init = emitter->tryEmitForInitializer(*InitDecl);
 
     if (!Init) {
       QualType T = InitExpr->getType();
       if (D->getType()->isReferenceType())
         T = D->getType();

clang/lib/CodeGen/CodeGenTypes.h

public:
 
   /// ConvertType - Convert type T into a llvm::Type.
   llvm::Type *ConvertType(QualType T);
 
   /// ConvertTypeForMem - Convert type T into a llvm::Type.  This differs from
   /// ConvertType in that it is used to convert to the memory representation for
   /// a type.  For example, the scalar representation for _Bool is i1, but the
   /// memory representation is usually i8 or i32, depending on the target.
-  llvm::Type *ConvertTypeForMem(QualType T, bool ForBitField = false);
+  llvm::Type *ConvertTypeForMem(QualType T, bool ForBitField = false,
+                                bool EnforceFixedLengthSVEAttribute = false);
 
   /// GetFunctionType - Get the LLVM function type for \arg Info.
   llvm::FunctionType *GetFunctionType(const CGFunctionInfo &Info);
 
   llvm::FunctionType *GetFunctionType(GlobalDecl GD);
 
   /// isFuncTypeConvertible - Utility to check whether a function type can
   /// be converted to an LLVM type (i.e. doesn't depend on an incomplete tag
public:  // These are internal details of CGT that shouldn't be used externally.
   /// ConvertRecordDeclType - Lay out a tagged decl type like struct or union.
   llvm::StructType *ConvertRecordDeclType(const RecordDecl *TD);
 
   /// getExpandedTypes - Expand the type \arg Ty into the LLVM
   /// argument types it would be passed as. See ABIArgInfo::Expand.
   void getExpandedTypes(QualType Ty,
                         SmallVectorImpl<llvm::Type *>::iterator &TI);
 
+  /// Returns the fixed-length type for an SVE ACLE scalable vector attributed
+  /// with 'arm_sve_vector_bits' that can be used in certain places where
+  /// size is really needed, e.g. members of structs or arrays or globals.
+  llvm::Optional<llvm::Type *> getFixedSVETypeForMemory(const Type *T);
+
   /// IsZeroInitializable - Return whether a type can be
   /// zero-initialized (in the C++ sense) with an LLVM zeroinitializer.
   bool isZeroInitializable(QualType T);
 
   /// Check if the pointer type can be zero-initialized (in the C++ sense)
   /// with an LLVM zeroinitializer.
   bool isPointerZeroInitializable(QualType T);
 

clang/lib/CodeGen/CodeGenTypes.cpp

if (RD->getIdentifier()) {
     OS << "anon";
 
   if (!suffix.empty())
     OS << suffix;
 
   Ty->setName(OS.str());
 }
 
+llvm::Optional<llvm::Type *>
+CodeGenTypes::getFixedSVETypeForMemory(const Type *T) {
+  unsigned VectorSize;
+  if (!Context.getArmSveVectorBits(T, VectorSize))
+    return {};
+
+  llvm::LLVMContext &Context = getLLVMContext();
+
+  llvm::Type *MemEltTy = nullptr;
+  switch (T->castAs<BuiltinType>()->getKind()) {
+  default:
+    llvm_unreachable("unhandled type!");
+  case BuiltinType::SveInt8:
+  case BuiltinType::SveUint8:
+  case BuiltinType::SveBool:
+    MemEltTy = llvm::Type::getInt8Ty(Context);
+    break;
+  case BuiltinType::SveInt16:
+  case BuiltinType::SveUint16:
+    MemEltTy = llvm::Type::getInt16Ty(Context);
+    break;
+  case BuiltinType::SveInt32:
+  case BuiltinType::SveUint32:
+    MemEltTy = llvm::Type::getInt32Ty(Context);
+    break;
+  case BuiltinType::SveInt64:
+  case BuiltinType::SveUint64:
+    MemEltTy = llvm::Type::getInt64Ty(Context);
+    break;
+  case BuiltinType::SveFloat16:
+    MemEltTy = llvm::Type::getHalfTy(Context);
+    break;
+  case BuiltinType::SveFloat32:
+    MemEltTy = llvm::Type::getFloatTy(Context);
+    break;
+  case BuiltinType::SveFloat64:
+    MemEltTy = llvm::Type::getDoubleTy(Context);
+    break;
+  case BuiltinType::SveBFloat16:
+    MemEltTy = llvm::Type::getBFloatTy(Context);
+    break;
+  }
+
+  return {llvm::ArrayType::get(
+      MemEltTy, VectorSize / MemEltTy->getPrimitiveSizeInBits())};
+}
+
 /// ConvertTypeForMem - Convert type T into a llvm::Type.  This differs from
 /// ConvertType in that it is used to convert to the memory representation for
 /// a type.  For example, the scalar representation for _Bool is i1, but the
 /// memory representation is usually i8 or i32, depending on the target.
-llvm::Type *CodeGenTypes::ConvertTypeForMem(QualType T, bool ForBitField) {
+llvm::Type *
+CodeGenTypes::ConvertTypeForMem(QualType T, bool ForBitField,
+                                bool EnforceFixedLengthSVEAttribute) {
   if (T->isConstantMatrixType()) {
     const Type *Ty = Context.getCanonicalType(T).getTypePtr();
     const ConstantMatrixType *MT = cast<ConstantMatrixType>(Ty);
     return llvm::ArrayType::get(ConvertType(MT->getElementType()),
                                 MT->getNumRows() * MT->getNumColumns());
   }
 
+  if (T->isConstantArrayType()) {
+    const ConstantArrayType *A = Context.getAsConstantArrayType(T);
+    const QualType EltTy = A->getElementType();
+
+    if (auto MemTy = getFixedSVETypeForMemory(EltTy.getTypePtr()))
+      return llvm::ArrayType::get(*MemTy, A->getSize().getZExtValue());
+  }
+
+  if (EnforceFixedLengthSVEAttribute) {
+    if (auto MemTy = getFixedSVETypeForMemory(T.getTypePtr()))
+      return *MemTy;
+  }
+
   llvm::Type *R = ConvertType(T);
 
   // If this is a bool type, or an ExtIntType in a bitfield representation,
   // map this integer to the target-specified size.
   if ((ForBitField && T->isExtIntType()) || R->isIntegerTy(1))
     return llvm::IntegerType::get(getLLVMContext(),
                                   (unsigned)Context.getTypeSize(T));
 

clang/test/Sema/attr-arm-sve-vector-bits-bitcast.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=128 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-128
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=256 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-256
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-512
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint64_t fixed_int64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svfloat64_t fixed_float64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbfloat16_t fixed_bfloat16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+#define DEFINE_STRUCT(ty)   \
+  struct struct_##ty {      \
+    fixed_##ty##_t x, y[3]; \
+  } struct_##ty;
+
+DEFINE_STRUCT(int64)
+DEFINE_STRUCT(float64)
+DEFINE_STRUCT(bfloat16)
+DEFINE_STRUCT(bool)
+
+//===----------------------------------------------------------------------===//
+// int64
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @read_int64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x i64>, <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-128-NEXT:    ret <vscale x 2 x i64> [[TMP0]]
+//
+// CHECK-256-LABEL: @read_int64(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-256-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x i64>, <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-256-NEXT:    ret <vscale x 2 x i64> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_int64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x i64>, <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-512-NEXT:    ret <vscale x 2 x i64> [[TMP0]]
+//
+svint64_t read_int64(struct struct_int64 *s) {
+  return s->y[0];
+}
+
+// CHECK-128-LABEL: @write_int64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-128-NEXT:    store <vscale x 2 x i64> [[X:%.*]], <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-256-LABEL: @write_int64(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-256-NEXT:    store <vscale x 2 x i64> [[X:%.*]], <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-256-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_int64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_INT64:%.*]], %struct.struct_int64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x i64]* [[ARRAYIDX]] to <vscale x 2 x i64>*
+// CHECK-512-NEXT:    store <vscale x 2 x i64> [[X:%.*]], <vscale x 2 x i64>* [[CAST_TO_SCALABLE]], align 16, !tbaa !2
+// CHECK-512-NEXT:    ret void
+//
+void write_int64(struct struct_int64 *s, svint64_t x) {
+  s->y[0] = x;
+}
+
+//===----------------------------------------------------------------------===//
+// float64
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @read_float64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x double>, <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-128-NEXT:    ret <vscale x 2 x double> [[TMP0]]
+//
+// CHECK-256-LABEL: @read_float64(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-256-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x double>, <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-256-NEXT:    ret <vscale x 2 x double> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_float64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x double>, <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-512-NEXT:    ret <vscale x 2 x double> [[TMP0]]
+//
+svfloat64_t read_float64(struct struct_float64 *s) {
+  return s->y[0];
+}
+
+// CHECK-128-LABEL: @write_float64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-128-NEXT:    store <vscale x 2 x double> [[X:%.*]], <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-256-LABEL: @write_float64(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-256-NEXT:    store <vscale x 2 x double> [[X:%.*]], <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-256-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_float64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_FLOAT64:%.*]], %struct.struct_float64* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x double]* [[ARRAYIDX]] to <vscale x 2 x double>*
+// CHECK-512-NEXT:    store <vscale x 2 x double> [[X:%.*]], <vscale x 2 x double>* [[CAST_TO_SCALABLE]], align 16, !tbaa !6
+// CHECK-512-NEXT:    ret void
+//
+void write_float64(struct struct_float64 *s, svfloat64_t x) {
+  s->y[0] = x;
+}
+
+//===----------------------------------------------------------------------===//
+// bfloat16
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @read_bfloat16(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 8 x bfloat>, <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-128-NEXT:    ret <vscale x 8 x bfloat> [[TMP0]]
+//
+// CHECK-256-LABEL: @read_bfloat16(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [16 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-256-NEXT:    [[TMP0:%.*]] = load <vscale x 8 x bfloat>, <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-256-NEXT:    ret <vscale x 8 x bfloat> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_bfloat16(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [32 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 8 x bfloat>, <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-512-NEXT:    ret <vscale x 8 x bfloat> [[TMP0]]
+//
+svbfloat16_t read_bfloat16(struct struct_bfloat16 *s) {
+  return s->y[0];
+}
+
+// CHECK-128-LABEL: @write_bfloat16(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-128-NEXT:    store <vscale x 8 x bfloat> [[X:%.*]], <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-256-LABEL: @write_bfloat16(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [16 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-256-NEXT:    store <vscale x 8 x bfloat> [[X:%.*]], <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-256-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_bfloat16(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BFLOAT16:%.*]], %struct.struct_bfloat16* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [32 x bfloat]* [[ARRAYIDX]] to <vscale x 8 x bfloat>*
+// CHECK-512-NEXT:    store <vscale x 8 x bfloat> [[X:%.*]], <vscale x 8 x bfloat>* [[CAST_TO_SCALABLE]], align 16, !tbaa !8
+// CHECK-512-NEXT:    ret void
+//
+void write_bfloat16(struct struct_bfloat16 *s, svbfloat16_t x) {
+  s->y[0] = x;
+}
+
+//===----------------------------------------------------------------------===//
+// bool
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @read_bool(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-128-NEXT:    ret <vscale x 16 x i1> [[TMP0]]
+//
+// CHECK-256-LABEL: @read_bool(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-256-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-256-NEXT:    ret <vscale x 16 x i1> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_bool(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-512-NEXT:    ret <vscale x 16 x i1> [[TMP0]]
+//
+svbool_t read_bool(struct struct_bool *s) {
+  return s->y[0];
+}
+
+// CHECK-128-LABEL: @write_bool(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-128-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [2 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-128-NEXT:    store <vscale x 16 x i1> [[X:%.*]], <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-256-LABEL: @write_bool(
+// CHECK-256-NEXT:  entry:
+// CHECK-256-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-256-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [4 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-256-NEXT:    store <vscale x 16 x i1> [[X:%.*]], <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-256-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_bool(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_STRUCT_BOOL:%.*]], %struct.struct_bool* [[S:%.*]], i64 0, i32 1, i64 0
+// CHECK-512-NEXT:    [[CAST_TO_SCALABLE:%.*]] = bitcast [8 x i8]* [[ARRAYIDX]] to <vscale x 16 x i1>*
+// CHECK-512-NEXT:    store <vscale x 16 x i1> [[X:%.*]], <vscale x 16 x i1>* [[CAST_TO_SCALABLE]], align 2, !tbaa !10
+// CHECK-512-NEXT:    ret void
+//
+void write_bool(struct struct_bool *s, svbool_t x) {
+  s->y[0] = x;
+}

clang/test/Sema/attr-arm-sve-vector-bits-call.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=128 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=256 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=1024 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=2048 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint64_t fixed_int64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svfloat64_t fixed_float64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbfloat16_t fixed_bfloat16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+#define CALL_FIXED_FIXED(ty)                                                \
+  fixed_##ty##_t                                                            \
+      call_##ty##_ff(svbool_t pg, fixed_##ty##_t op1, fixed_##ty##_t op2) { \
+    return svsel(pg, op1, op2);                                             \
+  }
+
+#define CALL_FIXED_SCALABLE(ty)                                         \
+  fixed_##ty##_t                                                        \
+      call_##ty##_fs(svbool_t pg, fixed_##ty##_t op1, sv##ty##_t op2) { \
+    return svsel(pg, op1, op2);                                         \
+  }
+
+#define CALL_SCALABLE_FIXED(ty)                                         \
+  fixed_##ty##_t                                                        \
+      call_##ty##_sf(svbool_t pg, sv##ty##_t op1, fixed_##ty##_t op2) { \
+    return svsel(pg, op1, op2);                                         \
+  }
+
+CALL_FIXED_FIXED(int64);
+CALL_FIXED_FIXED(float64);
+CALL_FIXED_FIXED(bfloat16);
+CALL_FIXED_FIXED(bool);
+
+CALL_FIXED_SCALABLE(int64);
+CALL_FIXED_SCALABLE(float64);
+CALL_FIXED_SCALABLE(bfloat16);
+CALL_FIXED_SCALABLE(bool);
+
+CALL_SCALABLE_FIXED(int64);
+CALL_SCALABLE_FIXED(float64);
+CALL_SCALABLE_FIXED(bfloat16);
+CALL_SCALABLE_FIXED(bool);
+
+// CHECK-LABEL: call_int64_ff
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x i64> @llvm.aarch64.sve.sel.nxv2i64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x i64> %op1, <vscale x 2 x i64> %op2)
+// CHECK: ret <vscale x 2 x i64> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_float64_ff
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x double> @llvm.aarch64.sve.sel.nxv2f64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x double> %op1, <vscale x 2 x double> %op2)
+// CHECK: ret <vscale x 2 x double> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bfloat16_ff
+// CHECK: %[[PG:.*]] = call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 8 x bfloat> @llvm.aarch64.sve.sel.nxv8bf16(<vscale x 8 x i1> %[[PG]], <vscale x 8 x bfloat> %op1, <vscale x 8 x bfloat> %op2)
+// CHECK: ret <vscale x 8 x bfloat> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bool_ff
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 16 x i1> @llvm.aarch64.sve.sel.nxv16i1(<vscale x 16 x i1> %pg, <vscale x 16 x i1> %op1, <vscale x 16 x i1> %op2)
+// CHECK: ret <vscale x 16 x i1> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_int64_fs
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x i64> @llvm.aarch64.sve.sel.nxv2i64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x i64> %op1, <vscale x 2 x i64> %op2)
+// CHECK: ret <vscale x 2 x i64> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_float64_fs
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x double> @llvm.aarch64.sve.sel.nxv2f64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x double> %op1, <vscale x 2 x double> %op2)
+// CHECK: ret <vscale x 2 x double> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bfloat16_fs
+// CHECK: %[[PG:.*]] = call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 8 x bfloat> @llvm.aarch64.sve.sel.nxv8bf16(<vscale x 8 x i1> %[[PG]], <vscale x 8 x bfloat> %op1, <vscale x 8 x bfloat> %op2)
+// CHECK: ret <vscale x 8 x bfloat> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bool_fs
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 16 x i1> @llvm.aarch64.sve.sel.nxv16i1(<vscale x 16 x i1> %pg, <vscale x 16 x i1> %op1, <vscale x 16 x i1> %op2)
+// CHECK: ret <vscale x 16 x i1> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_int64_sf
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x i64> @llvm.aarch64.sve.sel.nxv2i64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x i64> %op1, <vscale x 2 x i64> %op2)
+// CHECK: ret <vscale x 2 x i64> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_float64_sf
+// CHECK: %[[PG:.*]] = call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 2 x double> @llvm.aarch64.sve.sel.nxv2f64(<vscale x 2 x i1> %[[PG]], <vscale x 2 x double> %op1, <vscale x 2 x double> %op2)
+// CHECK: ret <vscale x 2 x double> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bfloat16_sf
+// CHECK: %[[PG:.*]] = call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> %pg)
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 8 x bfloat> @llvm.aarch64.sve.sel.nxv8bf16(<vscale x 8 x i1> %[[PG]], <vscale x 8 x bfloat> %op1, <vscale x 8 x bfloat> %op2)
+// CHECK: ret <vscale x 8 x bfloat> %[[INTRINSIC]]
+
+// CHECK-LABEL: call_bool_sf
+// CHECK: %[[INTRINSIC:.*]] = call <vscale x 16 x i1> @llvm.aarch64.sve.sel.nxv16i1(<vscale x 16 x i1> %pg, <vscale x 16 x i1> %op1, <vscale x 16 x i1> %op2)
+// CHECK: ret <vscale x 16 x i1> %[[INTRINSIC]]

clang/test/Sema/attr-arm-sve-vector-bits-cast.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=128 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=256 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=1024 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=2048 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint64_t fixed_int64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svfloat64_t fixed_float64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbfloat16_t fixed_bfloat16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+#define CAST(TYPE)                                        \
+  sv##TYPE##_t to_sv##TYPE##_t(fixed_##TYPE##_t type) {   \
+    return type;                                          \
+  }                                                       \
+                                                          \
+  fixed_##TYPE##_t from_sv##TYPE##_t(sv##TYPE##_t type) { \
+    return type;                                          \
+  }
+
+CAST(int64)
+CAST(float64)
+CAST(bfloat16)
+CAST(bool)
+
+// CHECK-LABEL: to_svint64_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 2 x i64> %type
+
+// CHECK-LABEL: from_svint64_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 2 x i64> %type
+
+// CHECK-LABEL: to_svfloat64_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 2 x double> %type
+
+// CHECK-LABEL: from_svfloat64_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 2 x double> %type
+
+// CHECK-LABEL: to_svbfloat16_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 8 x bfloat> %type
+
+// CHECK-LABEL: from_svbfloat16_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 8 x bfloat> %type
+
+// CHECK-LABEL: to_svbool_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 16 x i1> %type
+
+// CHECK-LABEL: from_svbool_t
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:  ret <vscale x 16 x i1> %type

clang/test/Sema/attr-arm-sve-vector-bits-codegen.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint32_t fixed_int32_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+fixed_bool_t global_pred;
+fixed_int32_t global_vec;
+
+// CHECK-LABEL: @foo(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* bitcast ([8 x i8]* @global_pred to <vscale x 16 x i1>*), align 2, !tbaa !2
+// CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 16 x i1> @llvm.aarch64.sve.and.z.nxv16i1(<vscale x 16 x i1> [[PRED:%.*]], <vscale x 16 x i1> [[TMP0]], <vscale x 16 x i1> [[TMP0]])
+// CHECK-NEXT:    [[TMP2:%.*]] = load <vscale x 4 x i32>, <vscale x 4 x i32>* bitcast ([16 x i32]* @global_vec to <vscale x 4 x i32>*), align 16, !tbaa !6
+// CHECK-NEXT:    [[TMP3:%.*]] = call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[TMP1]])
+// CHECK-NEXT:    [[TMP4:%.*]] = call <vscale x 4 x i32> @llvm.aarch64.sve.add.nxv4i32(<vscale x 4 x i1> [[TMP3]], <vscale x 4 x i32> [[TMP2]], <vscale x 4 x i32> [[VEC:%.*]])
+// CHECK-NEXT:    ret <vscale x 4 x i32> [[TMP4]]
+//
+fixed_int32_t foo(svbool_t pred, svint32_t vec) {
+  svbool_t pg = svand_z(pred, global_pred, global_pred);
+  return svadd_m(pg, global_vec, vec);
+}

clang/test/Sema/attr-arm-sve-vector-bits-globals.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=128 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-128
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -O1 -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-512
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint64_t fixed_int64_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbfloat16_t fixed_bfloat16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+fixed_int64_t global_i64;
+fixed_bfloat16_t global_bf16;
+fixed_bool_t global_bool;
+
+//===----------------------------------------------------------------------===//
+// WRITES
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @write_global_i64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    store <vscale x 2 x i64> [[V:%.*]], <vscale x 2 x i64>* bitcast ([2 x i64]* @global_i64 to <vscale x 2 x i64>*), align 16, !tbaa !2
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_global_i64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    store <vscale x 2 x i64> [[V:%.*]], <vscale x 2 x i64>* bitcast ([8 x i64]* @global_i64 to <vscale x 2 x i64>*), align 16, !tbaa !2
+// CHECK-512-NEXT:    ret void
+//
+void write_global_i64(svint64_t v) { global_i64 = v; }
+
+// CHECK-128-LABEL: @write_global_bf16(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    store <vscale x 8 x bfloat> [[V:%.*]], <vscale x 8 x bfloat>* bitcast ([8 x bfloat]* @global_bf16 to <vscale x 8 x bfloat>*), align 16, !tbaa !6
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_global_bf16(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    store <vscale x 8 x bfloat> [[V:%.*]], <vscale x 8 x bfloat>* bitcast ([32 x bfloat]* @global_bf16 to <vscale x 8 x bfloat>*), align 16, !tbaa !6
+// CHECK-512-NEXT:    ret void
+//
+void write_global_bf16(svbfloat16_t v) { global_bf16 = v; }
+
+// CHECK-128-LABEL: @write_global_bool(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    store <vscale x 16 x i1> [[V:%.*]], <vscale x 16 x i1>* bitcast ([2 x i8]* @global_bool to <vscale x 16 x i1>*), align 2, !tbaa !8
+// CHECK-128-NEXT:    ret void
+//
+// CHECK-512-LABEL: @write_global_bool(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    store <vscale x 16 x i1> [[V:%.*]], <vscale x 16 x i1>* bitcast ([8 x i8]* @global_bool to <vscale x 16 x i1>*), align 2, !tbaa !8
+// CHECK-512-NEXT:    ret void
+//
+void write_global_bool(svbool_t v) { global_bool = v; }
+
+//===----------------------------------------------------------------------===//
+// READS
+//===----------------------------------------------------------------------===//
+
+// CHECK-128-LABEL: @read_global_i64(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x i64>, <vscale x 2 x i64>* bitcast ([2 x i64]* @global_i64 to <vscale x 2 x i64>*), align 16, !tbaa !2
+// CHECK-128-NEXT:    ret <vscale x 2 x i64> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_global_i64(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 2 x i64>, <vscale x 2 x i64>* bitcast ([8 x i64]* @global_i64 to <vscale x 2 x i64>*), align 16, !tbaa !2
+// CHECK-512-NEXT:    ret <vscale x 2 x i64> [[TMP0]]
+//
+svint64_t read_global_i64() { return global_i64; }
+
+// CHECK-128-LABEL: @read_global_bf16(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 8 x bfloat>, <vscale x 8 x bfloat>* bitcast ([8 x bfloat]* @global_bf16 to <vscale x 8 x bfloat>*), align 16, !tbaa !6
+// CHECK-128-NEXT:    ret <vscale x 8 x bfloat> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_global_bf16(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 8 x bfloat>, <vscale x 8 x bfloat>* bitcast ([32 x bfloat]* @global_bf16 to <vscale x 8 x bfloat>*), align 16, !tbaa !6
+// CHECK-512-NEXT:    ret <vscale x 8 x bfloat> [[TMP0]]
+//
+svbfloat16_t read_global_bf16() { return global_bf16; }
+
+// CHECK-128-LABEL: @read_global_bool(
+// CHECK-128-NEXT:  entry:
+// CHECK-128-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* bitcast ([2 x i8]* @global_bool to <vscale x 16 x i1>*), align 2, !tbaa !8
+// CHECK-128-NEXT:    ret <vscale x 16 x i1> [[TMP0]]
+//
+// CHECK-512-LABEL: @read_global_bool(
+// CHECK-512-NEXT:  entry:
+// CHECK-512-NEXT:    [[TMP0:%.*]] = load <vscale x 16 x i1>, <vscale x 16 x i1>* bitcast ([8 x i8]* @global_bool to <vscale x 16 x i1>*), align 2, !tbaa !8
+// CHECK-512-NEXT:    ret <vscale x 16 x i1> [[TMP0]]
+//
+svbool_t read_global_bool() { return global_bool; }

clang/test/Sema/attr-arm-sve-vector-bits-types.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=128 -fallow-half-arguments-and-returns -S -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-128
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=256 -fallow-half-arguments-and-returns -S -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-256
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=512 -fallow-half-arguments-and-returns -S -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-512
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=1024 -fallow-half-arguments-and-returns -S -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-1024
+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -target-feature +bf16 -D__ARM_FEATURE_SVE_BITS=2048 -fallow-half-arguments-and-returns -S -emit-llvm -o - %s | FileCheck %s --check-prefix=CHECK-2048
+
+#include <arm_sve.h>
+
+#define N __ARM_FEATURE_SVE_BITS
+
+typedef svint8_t fixed_int8_t __attribute__((arm_sve_vector_bits(N)));
+typedef svint16_t fixed_int16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svint32_t fixed_int32_t __attribute__((arm_sve_vector_bits(N)));
+typedef svint64_t fixed_int64_t __attribute__((arm_sve_vector_bits(N)));
+
+typedef svuint8_t fixed_uint8_t __attribute__((arm_sve_vector_bits(N)));
+typedef svuint16_t fixed_uint16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svuint32_t fixed_uint32_t __attribute__((arm_sve_vector_bits(N)));
+typedef svuint64_t fixed_uint64_t __attribute__((arm_sve_vector_bits(N)));
+
+typedef svfloat16_t fixed_float16_t __attribute__((arm_sve_vector_bits(N)));
+typedef svfloat32_t fixed_float32_t __attribute__((arm_sve_vector_bits(N)));
+typedef svfloat64_t fixed_float64_t __attribute__((arm_sve_vector_bits(N)));
+
+typedef svbfloat16_t fixed_bfloat16_t __attribute__((arm_sve_vector_bits(N)));
+
+typedef svbool_t fixed_bool_t __attribute__((arm_sve_vector_bits(N)));
+
+//===----------------------------------------------------------------------===//
+// Structs and unions
+//===----------------------------------------------------------------------===//
+#define DEFINE_STRUCT(ty) \
+  struct struct_##ty {    \
+    fixed_##ty##_t x;     \
+  } struct_##ty;
+
+#define DEFINE_UNION(ty) \
+  union union_##ty {     \
+    fixed_##ty##_t x;    \
+  } union_##ty;
+
+DEFINE_STRUCT(int8)
+DEFINE_STRUCT(int16)
+DEFINE_STRUCT(int32)
+DEFINE_STRUCT(int64)
+DEFINE_STRUCT(uint8)
+DEFINE_STRUCT(uint16)
+DEFINE_STRUCT(uint32)
+DEFINE_STRUCT(uint64)
+DEFINE_STRUCT(float16)
+DEFINE_STRUCT(float32)
+DEFINE_STRUCT(float64)
+DEFINE_STRUCT(bfloat16)
+DEFINE_STRUCT(bool)
+
+DEFINE_UNION(int8)
+DEFINE_UNION(int16)
+DEFINE_UNION(int32)
+DEFINE_UNION(int64)
+DEFINE_UNION(uint8)
+DEFINE_UNION(uint16)
+DEFINE_UNION(uint32)
+DEFINE_UNION(uint64)
+DEFINE_UNION(float16)
+DEFINE_UNION(float32)
+DEFINE_UNION(float64)
+DEFINE_UNION(bfloat16)
+DEFINE_UNION(bool)
+
+//===----------------------------------------------------------------------===//
+// Global variables
+//===----------------------------------------------------------------------===//
+fixed_int8_t global_i8;
+fixed_int16_t global_i16;
+fixed_int32_t global_i32;
+fixed_int64_t global_i64;
+
+fixed_uint8_t global_u8;
+fixed_uint16_t global_u16;
+fixed_uint32_t global_u32;
+fixed_uint64_t global_u64;
+
+fixed_float16_t global_f16;
+fixed_float32_t global_f32;
+fixed_float64_t global_f64;
+
+fixed_bfloat16_t global_bf16;
+
+fixed_bool_t global_bool;
+
+//===----------------------------------------------------------------------===//
+// Global arrays
+//===----------------------------------------------------------------------===//
+fixed_int8_t global_arr_i8[3];
+fixed_int16_t global_arr_i16[3];
+fixed_int32_t global_arr_i32[3];
+fixed_int64_t global_arr_i64[3];
+
+fixed_uint8_t global_arr_u8[3];
+fixed_uint16_t global_arr_u16[3];
+fixed_uint32_t global_arr_u32[3];
+fixed_uint64_t global_arr_u64[3];
+
+fixed_float16_t global_arr_f16[3];
+fixed_float32_t global_arr_f32[3];
+fixed_float64_t global_arr_f64[3];
+
+fixed_bfloat16_t global_arr_bf16[3];
+
+fixed_bool_t global_arr_bool[3];
+
+//===----------------------------------------------------------------------===//
+// Locals
+//===----------------------------------------------------------------------===//
+void f() {
+  // Variables
+  fixed_int8_t local_i8;
+  fixed_int16_t local_i16;
+  fixed_int32_t local_i32;
+  fixed_int64_t local_i64;
+  fixed_uint8_t local_u8;
+  fixed_uint16_t local_u16;
+  fixed_uint32_t local_u32;
+  fixed_uint64_t local_u64;
+  fixed_float16_t local_f16;
+  fixed_float32_t local_f32;
+  fixed_float64_t local_f64;
+  fixed_bfloat16_t local_bf16;
+  fixed_bool_t local_bool;
+
+  // Arrays
+  fixed_int8_t local_arr_i8[3];
+  fixed_int16_t local_arr_i16[3];
+  fixed_int32_t local_arr_i32[3];
+  fixed_int64_t local_arr_i64[3];
+  fixed_uint8_t local_arr_u8[3];
+  fixed_uint16_t local_arr_u16[3];
+  fixed_uint32_t local_arr_u32[3];
+  fixed_uint64_t local_arr_u64[3];
+  fixed_float16_t local_arr_f16[3];
+  fixed_float32_t local_arr_f32[3];
+  fixed_float64_t local_arr_f64[3];
+  fixed_bfloat16_t local_arr_bf16[3];
+  fixed_bool_t local_arr_bool[3];
+}
+
+//===----------------------------------------------------------------------===//
+// Structs and unions
+//===----------------------------------------------------------------------===//
+// CHECK-128:      %struct.struct_int8 = type { [16 x i8] }
+// CHECK-128-NEXT: %struct.struct_int16 = type { [8 x i16] }
+// CHECK-128-NEXT: %struct.struct_int32 = type { [4 x i32] }
+// CHECK-128-NEXT: %struct.struct_int64 = type { [2 x i64] }
+// CHECK-128-NEXT: %struct.struct_uint8 = type { [16 x i8] }
+// CHECK-128-NEXT: %struct.struct_uint16 = type { [8 x i16] }
+// CHECK-128-NEXT: %struct.struct_uint32 = type { [4 x i32] }
+// CHECK-128-NEXT: %struct.struct_uint64 = type { [2 x i64] }
+// CHECK-128-NEXT: %struct.struct_float16 = type { [8 x half] }
+// CHECK-128-NEXT: %struct.struct_float32 = type { [4 x float] }
+// CHECK-128-NEXT: %struct.struct_float64 = type { [2 x double] }
+// CHECK-128-NEXT: %struct.struct_bfloat16 = type { [8 x bfloat] }
+// CHECK-128-NEXT: %struct.struct_bool = type { [2 x i8] }
+
+// CHECK-256:      %struct.struct_int8 = type { [32 x i8] }
+// CHECK-256-NEXT: %struct.struct_int16 = type { [16 x i16] }
+// CHECK-256-NEXT: %struct.struct_int32 = type { [8 x i32] }
+// CHECK-256-NEXT: %struct.struct_int64 = type { [4 x i64] }
+// CHECK-256-NEXT: %struct.struct_uint8 = type { [32 x i8] }
+// CHECK-256-NEXT: %struct.struct_uint16 = type { [16 x i16] }
+// CHECK-256-NEXT: %struct.struct_uint32 = type { [8 x i32] }
+// CHECK-256-NEXT: %struct.struct_uint64 = type { [4 x i64] }
+// CHECK-256-NEXT: %struct.struct_float16 = type { [16 x half] }
+// CHECK-256-NEXT: %struct.struct_float32 = type { [8 x float] }
+// CHECK-256-NEXT: %struct.struct_float64 = type { [4 x double] }
+// CHECK-256-NEXT: %struct.struct_bfloat16 = type { [16 x bfloat] }
+// CHECK-256-NEXT: %struct.struct_bool = type { [4 x i8] }
+
+// CHECK-512:      %struct.struct_int8 = type { [64 x i8] }
+// CHECK-512-NEXT: %struct.struct_int16 = type { [32 x i16] }
+// CHECK-512-NEXT: %struct.struct_int32 = type { [16 x i32] }
+// CHECK-512-NEXT: %struct.struct_int64 = type { [8 x i64] }
+// CHECK-512-NEXT: %struct.struct_uint8 = type { [64 x i8] }
+// CHECK-512-NEXT: %struct.struct_uint16 = type { [32 x i16] }
+// CHECK-512-NEXT: %struct.struct_uint32 = type { [16 x i32] }
+// CHECK-512-NEXT: %struct.struct_uint64 = type { [8 x i64] }
+// CHECK-512-NEXT: %struct.struct_float16 = type { [32 x half] }
+// CHECK-512-NEXT: %struct.struct_float32 = type { [16 x float] }
+// CHECK-512-NEXT: %struct.struct_float64 = type { [8 x double] }
+// CHECK-512-NEXT: %struct.struct_bfloat16 = type { [32 x bfloat] }
+// CHECK-512-NEXT: %struct.struct_bool = type { [8 x i8] }
+
+// CHECK-1024:      %struct.struct_int8 = type { [128 x i8] }
+// CHECK-1024-NEXT: %struct.struct_int16 = type { [64 x i16] }
+// CHECK-1024-NEXT: %struct.struct_int32 = type { [32 x i32] }
+// CHECK-1024-NEXT: %struct.struct_int64 = type { [16 x i64] }
+// CHECK-1024-NEXT: %struct.struct_uint8 = type { [128 x i8] }
+// CHECK-1024-NEXT: %struct.struct_uint16 = type { [64 x i16] }
+// CHECK-1024-NEXT: %struct.struct_uint32 = type { [32 x i32] }
+// CHECK-1024-NEXT: %struct.struct_uint64 = type { [16 x i64] }
+// CHECK-1024-NEXT: %struct.struct_float16 = type { [64 x half] }
+// CHECK-1024-NEXT: %struct.struct_float32 = type { [32 x float] }
+// CHECK-1024-NEXT: %struct.struct_float64 = type { [16 x double] }
+// CHECK-1024-NEXT: %struct.struct_bfloat16 = type { [64 x bfloat] }
+// CHECK-1024-NEXT: %struct.struct_bool = type { [16 x i8] }
+
+// CHECK-2048:      %struct.struct_int8 = type { [256 x i8] }
+// CHECK-2048-NEXT: %struct.struct_int16 = type { [128 x i16] }
+// CHECK-2048-NEXT: %struct.struct_int32 = type { [64 x i32] }
+// CHECK-2048-NEXT: %struct.struct_int64 = type { [32 x i64] }
+// CHECK-2048-NEXT: %struct.struct_uint8 = type { [256 x i8] }
+// CHECK-2048-NEXT: %struct.struct_uint16 = type { [128 x i16] }
+// CHECK-2048-NEXT: %struct.struct_uint32 = type { [64 x i32] }
+// CHECK-2048-NEXT: %struct.struct_uint64 = type { [32 x i64] }
+// CHECK-2048-NEXT: %struct.struct_float16 = type { [128 x half] }
+// CHECK-2048-NEXT: %struct.struct_float32 = type { [64 x float] }
+// CHECK-2048-NEXT: %struct.struct_float64 = type { [32 x double] }
+// CHECK-2048-NEXT: %struct.struct_bfloat16 = type { [128 x bfloat] }
+// CHECK-2048-NEXT: %struct.struct_bool = type { [32 x i8] }
+
+// CHECK-128:      %union.union_int8 = type { [16 x i8] }
+// CHECK-128-NEXT: %union.union_int16 = type { [8 x i16] }
+// CHECK-128-NEXT: %union.union_int32 = type { [4 x i32] }
+// CHECK-128-NEXT: %union.union_int64 = type { [2 x i64] }
+// CHECK-128-NEXT: %union.union_uint8 = type { [16 x i8] }
+// CHECK-128-NEXT: %union.union_uint16 = type { [8 x i16] }
+// CHECK-128-NEXT: %union.union_uint32 = type { [4 x i32] }
+// CHECK-128-NEXT: %union.union_uint64 = type { [2 x i64] }
+// CHECK-128-NEXT: %union.union_float16 = type { [8 x half] }
+// CHECK-128-NEXT: %union.union_float32 = type { [4 x float] }
+// CHECK-128-NEXT: %union.union_float64 = type { [2 x double] }
+// CHECK-128-NEXT: %union.union_bfloat16 = type { [8 x bfloat] }
+// CHECK-128-NEXT: %union.union_bool = type { [2 x i8] }
+
+// CHECK-256:      %union.union_int8 = type { [32 x i8] }
+// CHECK-256-NEXT: %union.union_int16 = type { [16 x i16] }
+// CHECK-256-NEXT: %union.union_int32 = type { [8 x i32] }
+// CHECK-256-NEXT: %union.union_int64 = type { [4 x i64] }
+// CHECK-256-NEXT: %union.union_uint8 = type { [32 x i8] }
+// CHECK-256-NEXT: %union.union_uint16 = type { [16 x i16] }
+// CHECK-256-NEXT: %union.union_uint32 = type { [8 x i32] }
+// CHECK-256-NEXT: %union.union_uint64 = type { [4 x i64] }
+// CHECK-256-NEXT: %union.union_float16 = type { [16 x half] }
+// CHECK-256-NEXT: %union.union_float32 = type { [8 x float] }
+// CHECK-256-NEXT: %union.union_float64 = type { [4 x double] }
+// CHECK-256-NEXT: %union.union_bfloat16 = type { [16 x bfloat] }
+// CHECK-256-NEXT: %union.union_bool = type { [4 x i8] }
+
+// CHECK-512:      %union.union_int8 = type { [64 x i8] }
+// CHECK-512-NEXT: %union.union_int16 = type { [32 x i16] }
+// CHECK-512-NEXT: %union.union_int32 = type { [16 x i32] }
+// CHECK-512-NEXT: %union.union_int64 = type { [8 x i64] }
+// CHECK-512-NEXT: %union.union_uint8 = type { [64 x i8] }
+// CHECK-512-NEXT: %union.union_uint16 = type { [32 x i16] }
+// CHECK-512-NEXT: %union.union_uint32 = type { [16 x i32] }
+// CHECK-512-NEXT: %union.union_uint64 = type { [8 x i64] }
+// CHECK-512-NEXT: %union.union_float16 = type { [32 x half] }
+// CHECK-512-NEXT: %union.union_float32 = type { [16 x float] }
+// CHECK-512-NEXT: %union.union_float64 = type { [8 x double] }
+// CHECK-512-NEXT: %union.union_bfloat16 = type { [32 x bfloat] }
+// CHECK-512-NEXT: %union.union_bool = type { [8 x i8] }
+
+// CHECK-1024:      %union.union_int8 = type { [128 x i8] }
+// CHECK-1024-NEXT: %union.union_int16 = type { [64 x i16] }
+// CHECK-1024-NEXT: %union.union_int32 = type { [32 x i32] }
+// CHECK-1024-NEXT: %union.union_int64 = type { [16 x i64] }
+// CHECK-1024-NEXT: %union.union_uint8 = type { [128 x i8] }
+// CHECK-1024-NEXT: %union.union_uint16 = type { [64 x i16] }
+// CHECK-1024-NEXT: %union.union_uint32 = type { [32 x i32] }
+// CHECK-1024-NEXT: %union.union_uint64 = type { [16 x i64] }
+// CHECK-1024-NEXT: %union.union_float16 = type { [64 x half] }
+// CHECK-1024-NEXT: %union.union_float32 = type { [32 x float] }
+// CHECK-1024-NEXT: %union.union_float64 = type { [16 x double] }
+// CHECK-1024-NEXT: %union.union_bfloat16 = type { [64 x bfloat] }
+// CHECK-1024-NEXT: %union.union_bool = type { [16 x i8] }
+
+// CHECK-2048:      %union.union_int8 = type { [256 x i8] }
+// CHECK-2048-NEXT: %union.union_int16 = type { [128 x i16] }
+// CHECK-2048-NEXT: %union.union_int32 = type { [64 x i32] }
+// CHECK-2048-NEXT: %union.union_int64 = type { [32 x i64] }
+// CHECK-2048-NEXT: %union.union_uint8 = type { [256 x i8] }
+// CHECK-2048-NEXT: %union.union_uint16 = type { [128 x i16] }
+// CHECK-2048-NEXT: %union.union_uint32 = type { [64 x i32] }
+// CHECK-2048-NEXT: %union.union_uint64 = type { [32 x i64] }
+// CHECK-2048-NEXT: %union.union_float16 = type { [128 x half] }
+// CHECK-2048-NEXT: %union.union_float32 = type { [64 x float] }
+// CHECK-2048-NEXT: %union.union_float64 = type { [32 x double] }
+// CHECK-2048-NEXT: %union.union_bfloat16 = type { [128 x bfloat] }
+// CHECK-2048-NEXT: %union.union_bool = type { [32 x i8] }
+
+//===----------------------------------------------------------------------===//
+// Global variables
+//===----------------------------------------------------------------------===//
+// CHECK-128:      @global_i8 = global [16 x i8] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_i16 = global [8 x i16] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_i32 = global [4 x i32] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_i64 = global [2 x i64] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_u8 = global [16 x i8] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_u16 = global [8 x i16] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_u32 = global [4 x i32] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_u64 = global [2 x i64] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_f16 = global [8 x half] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_f32 = global [4 x float] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_f64 = global [2 x double] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_bf16 = global [8 x bfloat] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_bool = global [2 x i8] zeroinitializer, align 2
+
+// CHECK-256:      @global_i8 = global [32 x i8] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_i16 = global [16 x i16] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_i32 = global [8 x i32] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_i64 = global [4 x i64] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_u8 = global [32 x i8] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_u16 = global [16 x i16] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_u32 = global [8 x i32] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_u64 = global [4 x i64] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_f16 = global [16 x half] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_f32 = global [8 x float] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_f64 = global [4 x double] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_bf16 = global [16 x bfloat] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_bool = global [4 x i8] zeroinitializer, align 2
+
+// CHECK-512:      @global_i8 = global [64 x i8] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_i16 = global [32 x i16] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_i32 = global [16 x i32] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_i64 = global [8 x i64] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_u8 = global [64 x i8] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_u16 = global [32 x i16] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_u32 = global [16 x i32] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_u64 = global [8 x i64] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_f16 = global [32 x half] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_f32 = global [16 x float] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_f64 = global [8 x double] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_bf16 = global [32 x bfloat] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_bool = global [8 x i8] zeroinitializer, align 2
+
+// CHECK-1024:      @global_i8 = global [128 x i8] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_i16 = global [64 x i16] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_i32 = global [32 x i32] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_i64 = global [16 x i64] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_u8 = global [128 x i8] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_u16 = global [64 x i16] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_u32 = global [32 x i32] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_u64 = global [16 x i64] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_f16 = global [64 x half] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_f32 = global [32 x float] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_f64 = global [16 x double] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_bf16 = global [64 x bfloat] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_bool = global [16 x i8] zeroinitializer, align 2
+
+// CHECK-2048:      @global_i8 = global [256 x i8] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_i16 = global [128 x i16] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_i32 = global [64 x i32] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_i64 = global [32 x i64] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_u8 = global [256 x i8] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_u16 = global [128 x i16] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_u32 = global [64 x i32] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_u64 = global [32 x i64] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_f16 = global [128 x half] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_f32 = global [64 x float] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_f64 = global [32 x double] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_bf16 = global [128 x bfloat] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_bool = global [32 x i8] zeroinitializer, align 2
+
+//===----------------------------------------------------------------------===//
+// Global arrays
+//===----------------------------------------------------------------------===//
+// CHECK-128:      @global_arr_i8 = global [3 x [16 x i8]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_i16 = global [3 x [8 x i16]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_i32 = global [3 x [4 x i32]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_i64 = global [3 x [2 x i64]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_u8 = global [3 x [16 x i8]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_u16 = global [3 x [8 x i16]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_u32 = global [3 x [4 x i32]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_u64 = global [3 x [2 x i64]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_f16 = global [3 x [8 x half]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_f32 = global [3 x [4 x float]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_f64 = global [3 x [2 x double]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_bf16 = global [3 x [8 x bfloat]] zeroinitializer, align 16
+// CHECK-128-NEXT: @global_arr_bool = global [3 x [2 x i8]] zeroinitializer, align 2
+
+// CHECK-256:      @global_arr_i8 = global [3 x [32 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_i16 = global [3 x [16 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_i32 = global [3 x [8 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_i64 = global [3 x [4 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_u8 = global [3 x [32 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_u16 = global [3 x [16 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_u32 = global [3 x [8 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_u64 = global [3 x [4 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_f16 = global [3 x [16 x half]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_f32 = global [3 x [8 x float]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_f64 = global [3 x [4 x double]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_bf16 = global [3 x [16 x bfloat]] zeroinitializer, align 16
+// CHECK-NEXT-256: @global_arr_bool = global [3 x [4 x i8]] zeroinitializer, align 2
+
+// CHECK-512:      @global_arr_i8 = global [3 x [64 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_i16 = global [3 x [32 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_i32 = global [3 x [16 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_i64 = global [3 x [8 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_u8 = global [3 x [64 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_u16 = global [3 x [32 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_u32 = global [3 x [16 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_u64 = global [3 x [8 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_f16 = global [3 x [32 x half]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_f32 = global [3 x [16 x float]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_f64 = global [3 x [8 x double]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_bf16 = global [3 x [32 x bfloat]] zeroinitializer, align 16
+// CHECK-NEXT-512: @global_arr_bool = global [3 x [8 x i8]] zeroinitializer, align 2
+
+// CHECK-1024:      @global_arr_i8 = global [3 x [128 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_i16 = global [3 x [64 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_i32 = global [3 x [32 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_i64 = global [3 x [16 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_u8 = global [3 x [128 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_u16 = global [3 x [64 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_u32 = global [3 x [32 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_u64 = global [3 x [16 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_f16 = global [3 x [64 x half]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_f32 = global [3 x [32 x float]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_f64 = global [3 x [16 x double]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_bf16 = global [3 x [64 x bfloat]] zeroinitializer, align 16
+// CHECK-NEXT-1024: @global_arr_bool = global [3 x [16 x i8]] zeroinitializer, align 2
+
+// CHECK-2048:      @global_arr_i8 = global [3 x [256 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_i16 = global [3 x [128 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_i32 = global [3 x [64 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_i64 = global [3 x [32 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_u8 = global [3 x [256 x i8]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_u16 = global [3 x [128 x i16]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_u32 = global [3 x [64 x i32]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_u64 = global [3 x [32 x i64]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_f16 = global [3 x [128 x half]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_f32 = global [3 x [64 x float]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_f64 = global [3 x [32 x double]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_bf16 = global [3 x [128 x bfloat]] zeroinitializer, align 16
+// CHECK-NEXT-2048: @global_arr_bool = global [3 x [32 x i8]] zeroinitializer, align 2
+
+//===----------------------------------------------------------------------===//
+// Local variables
+//===----------------------------------------------------------------------===//
+// CHECK:      %local_i8 = alloca <vscale x 16 x i8>, align 16
+// CHECK-NEXT: %local_i16 = alloca <vscale x 8 x i16>, align 16
+// CHECK-NEXT: %local_i32 = alloca <vscale x 4 x i32>, align 16
+// CHECK-NEXT: %local_i64 = alloca <vscale x 2 x i64>, align 16
+// CHECK-NEXT: %local_u8 = alloca <vscale x 16 x i8>, align 16
+// CHECK-NEXT: %local_u16 = alloca <vscale x 8 x i16>, align 16
+// CHECK-NEXT: %local_u32 = alloca <vscale x 4 x i32>, align 16
+// CHECK-NEXT: %local_u64 = alloca <vscale x 2 x i64>, align 16
+// CHECK-NEXT: %local_f16 = alloca <vscale x 8 x half>, align 16
+// CHECK-NEXT: %local_f32 = alloca <vscale x 4 x float>, align 16
+// CHECK-NEXT: %local_f64 = alloca <vscale x 2 x double>, align 16
+// CHECK-NEXT: %local_bf16 = alloca <vscale x 8 x bfloat>, align 16
+// CHECK-NEXT: %local_bool = alloca <vscale x 16 x i1>, align 2
+
+//===----------------------------------------------------------------------===//
+// Local arrays
+//===----------------------------------------------------------------------===//
+// CHECK-128:      %local_arr_i8 = alloca [3 x [16 x i8]], align 16
+// CHECK-128-NEXT: %local_arr_i16 = alloca [3 x [8 x i16]], align 16
+// CHECK-128-NEXT: %local_arr_i32 = alloca [3 x [4 x i32]], align 16
+// CHECK-128-NEXT: %local_arr_i64 = alloca [3 x [2 x i64]], align 16
+// CHECK-128-NEXT: %local_arr_u8 = alloca [3 x [16 x i8]], align 16
+// CHECK-128-NEXT: %local_arr_u16 = alloca [3 x [8 x i16]], align 16
+// CHECK-128-NEXT: %local_arr_u32 = alloca [3 x [4 x i32]], align 16
+// CHECK-128-NEXT: %local_arr_u64 = alloca [3 x [2 x i64]], align 16
+// CHECK-128-NEXT: %local_arr_f16 = alloca [3 x [8 x half]], align 16
+// CHECK-128-NEXT: %local_arr_f32 = alloca [3 x [4 x float]], align 16
+// CHECK-128-NEXT: %local_arr_f64 = alloca [3 x [2 x double]], align 16
+// CHECK-128-NEXT: %local_arr_bf16 = alloca [3 x [8 x bfloat]], align 16
+// CHECK-128-NEXT: %local_arr_bool = alloca [3 x [2 x i8]], align 2
+
+// CHECK-256:      %local_arr_i8 = alloca [3 x [32 x i8]], align 16
+// CHECK-256-NEXT: %local_arr_i16 = alloca [3 x [16 x i16]], align 16
+// CHECK-256-NEXT: %local_arr_i32 = alloca [3 x [8 x i32]], align 16
+// CHECK-256-NEXT: %local_arr_i64 = alloca [3 x [4 x i64]], align 16
+// CHECK-256-NEXT: %local_arr_u8 = alloca [3 x [32 x i8]], align 16
+// CHECK-256-NEXT: %local_arr_u16 = alloca [3 x [16 x i16]], align 16
+// CHECK-256-NEXT: %local_arr_u32 = alloca [3 x [8 x i32]], align 16
+// CHECK-256-NEXT: %local_arr_u64 = alloca [3 x [4 x i64]], align 16
+// CHECK-256-NEXT: %local_arr_f16 = alloca [3 x [16 x half]], align 16
+// CHECK-256-NEXT: %local_arr_f32 = alloca [3 x [8 x float]], align 16
+// CHECK-256-NEXT: %local_arr_f64 = alloca [3 x [4 x double]], align 16
+// CHECK-256-NEXT: %local_arr_bf16 = alloca [3 x [16 x bfloat]], align 16
+// CHECK-256-NEXT: %local_arr_bool = alloca [3 x [4 x i8]], align 2
+
+// CHECK-512:      %local_arr_i8 = alloca [3 x [64 x i8]], align 16
+// CHECK-512-NEXT: %local_arr_i16 = alloca [3 x [32 x i16]], align 16
+// CHECK-512-NEXT: %local_arr_i32 = alloca [3 x [16 x i32]], align 16
+// CHECK-512-NEXT: %local_arr_i64 = alloca [3 x [8 x i64]], align 16
+// CHECK-512-NEXT: %local_arr_u8 = alloca [3 x [64 x i8]], align 16
+// CHECK-512-NEXT: %local_arr_u16 = alloca [3 x [32 x i16]], align 16
+// CHECK-512-NEXT: %local_arr_u32 = alloca [3 x [16 x i32]], align 16
+// CHECK-512-NEXT: %local_arr_u64 = alloca [3 x [8 x i64]], align 16
+// CHECK-512-NEXT: %local_arr_f16 = alloca [3 x [32 x half]], align 16
+// CHECK-512-NEXT: %local_arr_f32 = alloca [3 x [16 x float]], align 16
+// CHECK-512-NEXT: %local_arr_f64 = alloca [3 x [8 x double]], align 16
+// CHECK-512-NEXT: %local_arr_bf16 = alloca [3 x [32 x bfloat]], align 16
+// CHECK-512-NEXT: %local_arr_bool = alloca [3 x [8 x i8]], align 2
+
+// CHECK-1024:       %local_arr_i8 = alloca [3 x [128 x i8]], align 16
+// CHECK-1024-NEXT:  %local_arr_i16 = alloca [3 x [64 x i16]], align 16
+// CHECK-1024-NEXT:  %local_arr_i32 = alloca [3 x [32 x i32]], align 16
+// CHECK-1024-NEXT:  %local_arr_i64 = alloca [3 x [16 x i64]], align 16
+// CHECK-1024-NEXT:  %local_arr_u8 = alloca [3 x [128 x i8]], align 16
+// CHECK-1024-NEXT:  %local_arr_u16 = alloca [3 x [64 x i16]], align 16
+// CHECK-1024-NEXT:  %local_arr_u32 = alloca [3 x [32 x i32]], align 16
+// CHECK-1024-NEXT:  %local_arr_u64 = alloca [3 x [16 x i64]], align 16
+// CHECK-1024-NEXT:  %local_arr_f16 = alloca [3 x [64 x half]], align 16
+// CHECK-1024-NEXT:  %local_arr_f32 = alloca [3 x [32 x float]], align 16
+// CHECK-1024-NEXT:  %local_arr_f64 = alloca [3 x [16 x double]], align 16
+// CHECK-1024-NEXT:  %local_arr_bf16 = alloca [3 x [64 x bfloat]], align 16
+// CHECK-1024-NEXT:  %local_arr_bool = alloca [3 x [16 x i8]], align 2
+
+// CHECK-2048:       %local_arr_i8 = alloca [3 x [256 x i8]], align 16
+// CHECK-2048-NEXT:  %local_arr_i16 = alloca [3 x [128 x i16]], align 16
+// CHECK-2048-NEXT:  %local_arr_i32 = alloca [3 x [64 x i32]], align 16
+// CHECK-2048-NEXT:  %local_arr_i64 = alloca [3 x [32 x i64]], align 16
+// CHECK-2048-NEXT:  %local_arr_u8 = alloca [3 x [256 x i8]], align 16
+// CHECK-2048-NEXT:  %local_arr_u16 = alloca [3 x [128 x i16]], align 16
+// CHECK-2048-NEXT:  %local_arr_u32 = alloca [3 x [64 x i32]], align 16
+// CHECK-2048-NEXT:  %local_arr_u64 = alloca [3 x [32 x i64]], align 16
+// CHECK-2048-NEXT:  %local_arr_f16 = alloca [3 x [128 x half]], align 16
+// CHECK-2048-NEXT:  %local_arr_f32 = alloca [3 x [64 x float]], align 16
+// CHECK-2048-NEXT:  %local_arr_f64 = alloca [3 x [32 x double]], align 16
+// CHECK-2048-NEXT:  %local_arr_bf16 = alloca [3 x [128 x bfloat]], align 16
+// CHECK-2048-NEXT:  %local_arr_bool = alloca [3 x [32 x i8]], align 2