[Clang][SVE] Parse builtin type string for scalable vectors

[sdesmalen]

Summary

This patch adds 'q' to mean 'scalable vector' in the builtin
type string, and for SVE will return the matching builtin
type as defined in the C/C++ language extensions for SVE.

This patch also adds some scaffolding to generate the arm_sve.h
header file, and some builtin definitions (+CodeGen) to be able
to implement some simple masked load intrinsics that use the
ACLE types, such as:

svint8_t test_svld1_s8(svbool_t pg, const int8_t *base) {

return svld1_s8(pg, base);

}

[efriedma]

Please update the documentation comment at the top of Builtins.def .

Can we avoid a gigantic, expensive to parse arm_sve.h somehow? Given that the compiler has to know the signatures for all these buitins anyway, can we just make including arm_sve.h set some flag in the compiler to make it recognize all the SVE intrinsics? This is different from the way we handle intrinsics on other targets, but SVE seems like a good opportunity to try a new approach.

clang/lib/CodeGen/CodeGenFunction.cpp

507

I'm not sure that we'll ever want to take advantage of min-legal-vector-width on targets with scalable vectors... but if we did, we'd probably want to use getKnownMinSize() here

[sdesmalen]

• Added comment describing q for scalable vectors to Builtins.def
• Use getKnownMinSize() for calculating LargestVectorWidth

[sdesmalen]

In D75298#1898770, @efriedma wrote:

Can we avoid a gigantic, expensive to parse arm_sve.h somehow? Given that the compiler has to know the signatures for all these buitins anyway, can we just make including arm_sve.h set some flag in the compiler to make it recognize all the SVE intrinsics? This is different from the way we handle intrinsics on other targets, but SVE seems like a good opportunity to try a new approach.

We actually tried this and already simplified the header file as much as possible. The header file now only contains preprocessor macros and function declarations, but no definitions, so should be relatively easy to parse. All code-gen itself is handled in CGBuiltin.
The main reason we couldn't remove the header file is because of the short-forms/overloaded intrinsics. I wouldn't know of a way to create overloaded intrinsics without having to manually do the usual arithmetic conversions similar to how this is currently done for some overloaded builtins like SemaBuiltinAtomicOverloaded in SemaChecking.cpp.

I also wonder if this problem isn't better solved using pre-compiled modules at some point?

Please update the documentation comment at the top of Builtins.def .

Good spot, fixed it!

[efriedma]

The main reason we couldn't remove the header file is because of the short-forms/overloaded intrinsics

How are you planning to handle them with your current approach?

Anyway, that isn't really relevant to the question of why we need to #define svld1_u8(...) __builtin_sve_svld1_u8(__VA_ARGS__), as opposed to just making svld1_u8 an intrinsic.

I wouldn't know of a way to create overloaded intrinsics without having to manually do the usual arithmetic conversions similar to how this is currently done for some overloaded builtins like SemaBuiltinAtomicOverloaded in SemaChecking.cpp.

The current Builtins.def infrastructure isn't really designed around it, but it would be possible to add overloaded intrinsics. Currently, when a builtin name is looked up, we create one FunctionDecl. But we could change it to make/return a list of FunctionDecls, instead, and pass them to standard overload resolution.

I also wonder if this problem isn't better solved using pre-compiled modules at some point?

Sort of? I mean, yes, loading a precompiled module is somewhat cheaper than parsing C. But it still isn't free, it doesn't really change the fact that the indirection isn't doing anything useful, and we're still making preprocessing/diagnostics more complicated by using defines.

[sdesmalen]

In D75298#1901578, @efriedma wrote:

How are you planning to handle them with your current approach?

Each prototype will get a arm_sve_alias attribute that specifies which builtin it relates to. For example:

__attribute__((__clang_arm_sve_alias(__builtin_sve_svld1_u8)))
svuint8_t svld1(svbool_t, uint8_t const *);

__attribute__((__clang_arm_sve_alias(__builtin_sve_svld1_u32)))
svuint32_t svld1(svbool_t, uint32_t const *);

This is similar to how this was done for MVE.

Anyway, that isn't really relevant to the question of why we need to #define svld1_u8(...) __builtin_sve_svld1_u8(__VA_ARGS__), as opposed to just making svld1_u8 an intrinsic.

This is because many of the builtins are protected by an architecture guard like #ifdef __ARM_FEATURE_SVE or #ifdef __ARM_FEATURE_SVE2.

I wouldn't know of a way to create overloaded intrinsics without having to manually do the usual arithmetic conversions similar to how this is currently done for some overloaded builtins like SemaBuiltinAtomicOverloaded in SemaChecking.cpp.

The current Builtins.def infrastructure isn't really designed around it, but it would be possible to add overloaded intrinsics. Currently, when a builtin name is looked up, we create one FunctionDecl. But we could change it to make/return a list of FunctionDecls, instead, and pass them to standard overload resolution.

Do you happen to know which method in Sema does this? I had a look before, but couldn't find where we could do something like this.

Sort of? I mean, yes, loading a precompiled module is somewhat cheaper than parsing C. But it still isn't free, it doesn't really change the fact that the indirection isn't doing anything useful, and we're still making preprocessing/diagnostics more complicated by using defines.

The reason for using #defines for the non-overloaded intrinsics was mainly performance, the preprocessor was much faster than having to parse all declarations, with the only downside being the redirection in the diagnostics.

[efriedma]

This is because many of the builtins are protected by an architecture guard like #ifdef __ARM_FEATURE_SVE or #ifdef __ARM_FEATURE_SVE2.

I think TARGET_HEADER_BUILTIN has support for requiring target features. Granted, we don't use it much at the moment, so there might be some non-obvious issue with this.

Do you happen to know which method in Sema does this? I had a look before, but couldn't find where we could do something like this.

See Sema::LazilyCreateBuiltin, and its caller Sema::LookupBuiltin.

The reason for using #defines for the non-overloaded intrinsics was mainly performance, the preprocessor was much faster than having to parse all declarations, with the only downside being the redirection in the diagnostics.

Oh, that makes sense.

[sdesmalen]

In D75298#1904561, @efriedma wrote:

Do you happen to know which method in Sema does this? I had a look before, but couldn't find where we could do something like this.

See Sema::LazilyCreateBuiltin, and its caller Sema::LookupBuiltin.

Thanks for those pointers! I spoke with @simon_tatham today, who is also very interested in this in order to simplify the MVE header file.

I would like to suggest first trying to land the current implementation (using the header file with all the declarations), so that we can use the regression tests added as part of this effort to test a different implementation that does not rely as much on the header file. What do you think?

[efriedma]

Changing the way we expose the builtins isn't going to affect most of the code related to the SVE intrinsics. I'm fine sticking with a known working approach, and trying to address that issue later.

[sdesmalen]

In D75298#1918489, @efriedma wrote:

Changing the way we expose the builtins isn't going to affect most of the code related to the SVE intrinsics. I'm fine sticking with a known working approach, and trying to address that issue later.

Okay, thanks! In that case, are you happy with the patch as it is now?

[sdesmalen]

• Updated license header for the arm_sve.h file to use the LLVM license instead of MIT.

[efriedma]

clang/utils/TableGen/SveEmitter.cpp

87

I'd prefer to avoid depending on stdbool if it isn't necessary.

93

The ACLE documentation doesn't say anything about float16_t etc.?

[sdesmalen]

clang/utils/TableGen/SveEmitter.cpp

87

Section 3.3 of the ACLE actually mentions this explicitly:

arm_sve.h includes stdint.h and so provides standard types such as uint32_t. When included from C code the header also includes stdbool.h and so provides the bool type.

93

In that same section, it says:

In addition, the header file defines the following scalar data types:
  float16_t equivalent to __fp16

[efriedma]

LGTM

clang/utils/TableGen/SveEmitter.cpp

87

Oh, sorry, I was accidentally looking at an old version of the ACLE spec.

clang/include/clang/AST/ASTContext.h

public:
 
   /// Return a type for a constant array for a string literal of the
   /// specified element type and length.
   QualType getStringLiteralArrayType(QualType EltTy, unsigned Length) const;
 
   /// Returns a vla type where known sizes are replaced with [*].
   QualType getVariableArrayDecayedType(QualType Ty) const;
 
+  /// Return the unique reference to a scalable vector type of the specified
+  /// element type and scalable number of elements.
+  ///
+  /// \pre \p EltTy must be a built-in type.
+  QualType getScalableVectorType(QualType EltTy, unsigned NumElts) const;
+
   /// Return the unique reference to a vector type of the specified
   /// element type and size.
   ///
   /// \pre \p VectorType must be a built-in type.
   QualType getVectorType(QualType VectorType, unsigned NumElts,
                          VectorType::VectorKind VecKind) const;
   /// Return the unique reference to the type for a dependently sized vector of
   /// the specified element type.

clang/include/clang/Basic/AArch64SVEACLETypes.def

 //  - IsSigned is true for vectors of signed integer elements and
 //    for vectors of floating-point elements.
 //
 //  - IsFP is true for vectors of floating-point elements.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SVE_VECTOR_TYPE
-#define SVE_VECTOR_TYPE(Name, Id, SingletonId, ElKind, ElBits, IsSigned, IsFP)\
+#define SVE_VECTOR_TYPE(Name, Id, SingletonId, NumEls, ElBits, IsSigned, IsFP) \
   SVE_TYPE(Name, Id, SingletonId)
 #endif
 
 #ifndef SVE_PREDICATE_TYPE
-#define SVE_PREDICATE_TYPE(Name, Id, SingletonId, ElKind)\
+#define SVE_PREDICATE_TYPE(Name, Id, SingletonId, NumEls)\
   SVE_TYPE(Name, Id, SingletonId)
 #endif
 
 //===- Vector point types -----------------------------------------------===//
 
-SVE_VECTOR_TYPE("__SVInt8_t",  SveInt8, SveInt8Ty, SveElSInt8, 8, true, false)
-SVE_VECTOR_TYPE("__SVInt16_t", SveInt16, SveInt16Ty, SveElSInt16, 16, true, false)
-SVE_VECTOR_TYPE("__SVInt32_t", SveInt32, SveInt32Ty, SveElSInt32, 32, true, false)
-SVE_VECTOR_TYPE("__SVInt64_t", SveInt64, SveInt64Ty, SveElSInt64, 64, true, false)
-
-SVE_VECTOR_TYPE("__SVUint8_t",  SveUint8, SveUint8Ty, SveElUInt8, 8, false, false)
-SVE_VECTOR_TYPE("__SVUint16_t", SveUint16, SveUint16Ty, SveElUInt16, 16, false, false)
-SVE_VECTOR_TYPE("__SVUint32_t", SveUint32, SveUint32Ty, SveElUInt32, 32, false, false)
-SVE_VECTOR_TYPE("__SVUint64_t", SveUint64, SveUint64Ty, SveElUInt64, 64, false, false)
-
-SVE_VECTOR_TYPE("__SVFloat16_t", SveFloat16, SveFloat16Ty, SveElHalf, 16, true, true)
-SVE_VECTOR_TYPE("__SVFloat32_t", SveFloat32, SveFloat32Ty, SveElFloat, 32, true, true)
-SVE_VECTOR_TYPE("__SVFloat64_t", SveFloat64, SveFloat64Ty, SveElDouble, 64, true, true)
+SVE_VECTOR_TYPE("__SVInt8_t",  SveInt8, SveInt8Ty, 16, 8, true, false)
+SVE_VECTOR_TYPE("__SVInt16_t", SveInt16, SveInt16Ty, 8, 16, true, false)
+SVE_VECTOR_TYPE("__SVInt32_t", SveInt32, SveInt32Ty, 4, 32, true, false)
+SVE_VECTOR_TYPE("__SVInt64_t", SveInt64, SveInt64Ty, 2, 64, true, false)
+
+SVE_VECTOR_TYPE("__SVUint8_t",  SveUint8, SveUint8Ty, 16, 8, false, false)
+SVE_VECTOR_TYPE("__SVUint16_t", SveUint16, SveUint16Ty, 8, 16, false, false)
+SVE_VECTOR_TYPE("__SVUint32_t", SveUint32, SveUint32Ty, 4, 32, false, false)
+SVE_VECTOR_TYPE("__SVUint64_t", SveUint64, SveUint64Ty, 2, 64, false, false)
+
+SVE_VECTOR_TYPE("__SVFloat16_t", SveFloat16, SveFloat16Ty, 8, 16, true, true)
+SVE_VECTOR_TYPE("__SVFloat32_t", SveFloat32, SveFloat32Ty, 4, 32, true, true)
+SVE_VECTOR_TYPE("__SVFloat64_t", SveFloat64, SveFloat64Ty, 2, 64, true, true)
 
-SVE_PREDICATE_TYPE("__SVBool_t", SveBool, SveBoolTy, SveElBool)
+SVE_PREDICATE_TYPE("__SVBool_t", SveBool, SveBoolTy, 16)
 
 #undef SVE_VECTOR_TYPE
 #undef SVE_PREDICATE_TYPE
 #undef SVE_TYPE

clang/include/clang/Basic/Builtins.def

 //  w -> wchar_t
 //  F -> constant CFString
 //  G -> id
 //  H -> SEL
 //  M -> struct objc_super
 //  a -> __builtin_va_list
 //  A -> "reference" to __builtin_va_list
 //  V -> Vector, followed by the number of elements and the base type.
+//  q -> Scalable vector, followed by the number of elements and the base type.
 //  E -> ext_vector, followed by the number of elements and the base type.
 //  X -> _Complex, followed by the base type.
 //  Y -> ptrdiff_t
 //  P -> FILE
 //  J -> jmp_buf
 //  SJ -> sigjmp_buf
 //  K -> ucontext_t
 //  p -> pid_t

clang/include/clang/Basic/BuiltinsAArch64.def

 BUILTIN(__builtin_sponentry, "v*", "c")
 
 // Transactional Memory Extension
 BUILTIN(__builtin_arm_tstart, "WUi", "nj")
 BUILTIN(__builtin_arm_tcommit, "v", "n")
 BUILTIN(__builtin_arm_tcancel, "vWUIi", "n")
 BUILTIN(__builtin_arm_ttest, "WUi", "nc")
 
+// SVE
+BUILTIN(__builtin_sve_svld1_s16, "q8sq16bSsC*", "n")
+BUILTIN(__builtin_sve_svld1_s32, "q4iq16bSiC*", "n")
+BUILTIN(__builtin_sve_svld1_s64, "q2Wiq16bSWiC*", "n")
+BUILTIN(__builtin_sve_svld1_s8, "q16Scq16bScC*", "n")
+BUILTIN(__builtin_sve_svld1_u16, "q8Usq16bUsC*", "n")
+BUILTIN(__builtin_sve_svld1_u32, "q4Uiq16bUiC*", "n")
+BUILTIN(__builtin_sve_svld1_u64, "q2UWiq16bUWiC*", "n")
+BUILTIN(__builtin_sve_svld1_u8, "q16Ucq16bUcC*", "n")
+BUILTIN(__builtin_sve_svld1_f64, "q2dq16bdC*", "n")
+BUILTIN(__builtin_sve_svld1_f32, "q4fq16bfC*", "n")
+BUILTIN(__builtin_sve_svld1_f16, "q8hq16bhC*", "n")
+
 TARGET_HEADER_BUILTIN(_BitScanForward, "UcUNi*UNi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanReverse, "UcUNi*UNi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanForward64, "UcUNi*ULLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanReverse64, "UcUNi*ULLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 
 TARGET_HEADER_BUILTIN(_InterlockedAdd,           "NiNiD*Ni",    "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedAnd64,         "LLiLLiD*LLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement64,   "LLiLLiD*",    "nh", "intrin.h", ALL_MS_LANGUAGES, "")

clang/include/clang/Basic/arm_sve.td

+//===--- arm_sve.td - ARM SVE compiler interface ------------------------===//
+//
+//  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//  See https://llvm.org/LICENSE.txt for license information.
+//  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file defines the TableGen definitions from which the ARM SVE header
+//  file will be generated.  See:
+//
+//      https://developer.arm.com/architectures/system-architectures/software-standards/acle
+//
+//===----------------------------------------------------------------------===//

clang/lib/AST/ASTContext.cpp

#include "clang/Basic/OpenCLExtensionTypes.def"
     // The SVE types are effectively target-specific.  The length of an
     // SVE_VECTOR_TYPE is only known at runtime, but it is always a multiple
     // of 128 bits.  There is one predicate bit for each vector byte, so the
     // length of an SVE_PREDICATE_TYPE is always a multiple of 16 bits.
     //
     // Because the length is only known at runtime, we use a dummy value
     // of 0 for the static length.  The alignment values are those defined
     // by the Procedure Call Standard for the Arm Architecture.
-#define SVE_VECTOR_TYPE(Name, Id, SingletonId, ElKind, ElBits, IsSigned, IsFP)\
-    case BuiltinType::Id: \
-      Width = 0; \
-      Align = 128; \
-      break;
-#define SVE_PREDICATE_TYPE(Name, Id, SingletonId, ElKind) \
-    case BuiltinType::Id: \
-      Width = 0; \
-      Align = 16; \
-      break;
+#define SVE_VECTOR_TYPE(Name, Id, SingletonId, NumEls, ElBits, IsSigned, IsFP) \
+  case BuiltinType::Id:                                                        \
+    Width = 0;                                                                 \
+    Align = 128;                                                               \
+#define SVE_PREDICATE_TYPE(Name, Id, SingletonId, NumEls)                      \
+  case BuiltinType::Id:                                                        \
+    Width = 0;                                                                 \
+    Align = 16;                                                                \
 #include "clang/Basic/AArch64SVEACLETypes.def"
     }
     break;
   case Type::ObjCObjectPointer:
     Width = Target->getPointerWidth(0);
     Align = Target->getPointerAlign(0);
     break;
   case Type::BlockPointer:
QualType ASTContext::getIncompleteArrayType(QualType elementType,
   auto *newType = new (*this, TypeAlignment)
     IncompleteArrayType(elementType, canon, ASM, elementTypeQuals);
 
   IncompleteArrayTypes.InsertNode(newType, insertPos);
   Types.push_back(newType);
   return QualType(newType, 0);
 }
 
+/// getScalableVectorType - Return the unique reference to a scalable vector
+/// type of the specified element type and size. VectorType must be a built-in
+/// type.
+QualType ASTContext::getScalableVectorType(QualType EltTy,
+                                           unsigned NumElts) const {
+  if (Target->hasAArch64SVETypes()) {
+    uint64_t EltTySize = getTypeSize(EltTy);
+#define SVE_VECTOR_TYPE(Name, Id, SingletonId, NumEls, ElBits, IsSigned, IsFP) \
+  if (!EltTy->isBooleanType() &&                                               \
+      ((EltTy->hasIntegerRepresentation() &&                                   \
+        EltTy->hasSignedIntegerRepresentation() == IsSigned) ||                \
+       (EltTy->hasFloatingRepresentation() && IsFP)) &&                        \
+      EltTySize == ElBits && NumElts == NumEls)                                \
+    return SingletonId;
+#define SVE_PREDICATE_TYPE(Name, Id, SingletonId, NumEls)                      \
+  if (EltTy->isBooleanType() && NumElts == NumEls)                             \
+    return SingletonId;
+#include "clang/Basic/AArch64SVEACLETypes.def"
+  }
+  return QualType();
+}
+
 /// getVectorType - Return the unique reference to a vector type of
 /// the specified element type and size. VectorType must be a built-in type.
 QualType ASTContext::getVectorType(QualType vecType, unsigned NumElts,
                                    VectorType::VectorKind VecKind) const {
   assert(vecType->isBuiltinType());
 
   // Check if we've already instantiated a vector of this type.
   llvm::FoldingSetNodeID ID;
case 'A':
     // it to be a __va_list_tag*.
     Type = Context.getBuiltinVaListType();
     assert(!Type.isNull() && "builtin va list type not initialized!");
     if (Type->isArrayType())
       Type = Context.getArrayDecayedType(Type);
     else
       Type = Context.getLValueReferenceType(Type);
     break;
+  case 'q': {
+    char *End;
+    unsigned NumElements = strtoul(Str, &End, 10);
+    assert(End != Str && "Missing vector size");
+    Str = End;
+
+    QualType ElementType = DecodeTypeFromStr(Str, Context, Error,
+                                             RequiresICE, false);
+    assert(!RequiresICE && "Can't require vector ICE");
+
+    Type = Context.getScalableVectorType(ElementType, NumElements);
+    break;
+  }
   case 'V': {
     char *End;
     unsigned NumElements = strtoul(Str, &End, 10);
     assert(End != Str && "Missing vector size");
     Str = End;
 
     QualType ElementType = DecodeTypeFromStr(Str, Context, Error,
                                              RequiresICE, false);

clang/lib/CodeGen/CGBuiltin.cpp

Value *CodeGenFunction::vectorWrapScalar16(Value *Op) {
   llvm::Type *VTy = llvm::VectorType::get(Int16Ty, 4);
   Op = Builder.CreateBitCast(Op, Int16Ty);
   Value *V = UndefValue::get(VTy);
   llvm::Constant *CI = ConstantInt::get(SizeTy, 0);
   Op = Builder.CreateInsertElement(V, Op, CI);
   return Op;
 }
 
+// Reinterpret the input predicate so that it can be used to correctly isolate
+// the elements of the specified datatype.
+Value *CodeGenFunction::EmitSVEPredicateCast(Value *Pred,
+                                             llvm::VectorType *VTy) {
+  llvm::VectorType *RTy = llvm::VectorType::get(
+      IntegerType::get(getLLVMContext(), 1), VTy->getElementCount());
+  if (Pred->getType() == RTy)
+    return Pred;
+
+  unsigned IntID;
+  llvm::Type *IntrinsicTy;
+  switch (VTy->getNumElements()) {
+  default:
+    llvm_unreachable("unsupported element count!");
+  case 2:
+  case 4:
+  case 8:
+    IntID = Intrinsic::aarch64_sve_convert_from_svbool;
+    IntrinsicTy = RTy;
+    break;
+  case 16:
+    IntID = Intrinsic::aarch64_sve_convert_to_svbool;
+    IntrinsicTy = Pred->getType();
+    break;
+  }
+
+  Function *F = CGM.getIntrinsic(IntID, IntrinsicTy);
+  Value *C = Builder.CreateCall(F, Pred);
+  assert(C->getType() == RTy && "Unexpected return type!");
+  return C;
+}
+
+Value *CodeGenFunction::EmitSVEMaskedLoad(llvm::Type *ReturnTy,
+                                          SmallVectorImpl<Value *> &Ops) {
+  llvm::PointerType *PTy = cast<llvm::PointerType>(Ops[1]->getType());
+  llvm::Type *MemEltTy = PTy->getPointerElementType();
+
+  // The vector type that is returned may be different from the
+  // eventual type loaded from memory.
+  auto VectorTy = cast<llvm::VectorType>(ReturnTy);
+  auto MemoryTy =
+      llvm::VectorType::get(MemEltTy, VectorTy->getVectorElementCount());
+
+  Value *Offset = Builder.getInt32(0);
+  Value *Predicate = EmitSVEPredicateCast(Ops[0], MemoryTy);
+  Value *BasePtr = Builder.CreateBitCast(Ops[1], MemoryTy->getPointerTo());
+  BasePtr = Builder.CreateGEP(MemoryTy, BasePtr, Offset);
+
+  Value *Splat0 = Constant::getNullValue(MemoryTy);
+  return Builder.CreateMaskedLoad(BasePtr, Align(1), Predicate, Splat0);
+}
+
 Value *CodeGenFunction::EmitAArch64BuiltinExpr(unsigned BuiltinID,
                                                const CallExpr *E,
                                                llvm::Triple::ArchType Arch) {
   unsigned HintID = static_cast<unsigned>(-1);
   switch (BuiltinID) {
   default: break;
   case AArch64::BI__builtin_arm_nop:
     HintID = 0;
case AArch64::BI__sevl:
     break;
   }
 
   if (HintID != static_cast<unsigned>(-1)) {
     Function *F = CGM.getIntrinsic(Intrinsic::aarch64_hint);
     return Builder.CreateCall(F, llvm::ConstantInt::get(Int32Ty, HintID));
   }
 
+  switch (BuiltinID) {
+  case AArch64::BI__builtin_sve_svld1_u8:
+  case AArch64::BI__builtin_sve_svld1_u16:
+  case AArch64::BI__builtin_sve_svld1_u32:
+  case AArch64::BI__builtin_sve_svld1_u64:
+  case AArch64::BI__builtin_sve_svld1_s8:
+  case AArch64::BI__builtin_sve_svld1_s16:
+  case AArch64::BI__builtin_sve_svld1_s32:
+  case AArch64::BI__builtin_sve_svld1_s64:
+  case AArch64::BI__builtin_sve_svld1_f16:
+  case AArch64::BI__builtin_sve_svld1_f32:
+  case AArch64::BI__builtin_sve_svld1_f64: {
+    llvm::SmallVector<Value *, 4> Ops = {EmitScalarExpr(E->getArg(0)),
+                                         EmitScalarExpr(E->getArg(1))};
+    llvm::Type *Ty = ConvertType(E->getType());
+    return EmitSVEMaskedLoad(Ty, Ops);
+  }
+  default:
+    break;
+  }
+
   if (BuiltinID == AArch64::BI__builtin_arm_prefetch) {
     Value *Address         = EmitScalarExpr(E->getArg(0));
     Value *RW              = EmitScalarExpr(E->getArg(1));
     Value *CacheLevel      = EmitScalarExpr(E->getArg(2));
     Value *RetentionPolicy = EmitScalarExpr(E->getArg(3));
     Value *IsData          = EmitScalarExpr(E->getArg(4));
 
     Value *Locality = nullptr;

clang/lib/CodeGen/CodeGenFunction.h

llvm::Value *EmitNeonCall(llvm::Function *F,
                             const char *name,
                             unsigned shift = 0, bool rightshift = false);
   llvm::Value *EmitNeonSplat(llvm::Value *V, llvm::Constant *Idx);
   llvm::Value *EmitNeonShiftVector(llvm::Value *V, llvm::Type *Ty,
                                    bool negateForRightShift);
   llvm::Value *EmitNeonRShiftImm(llvm::Value *Vec, llvm::Value *Amt,
                                  llvm::Type *Ty, bool usgn, const char *name);
   llvm::Value *vectorWrapScalar16(llvm::Value *Op);
+
+  llvm::Value *EmitSVEPredicateCast(llvm::Value *Pred, llvm::VectorType *VTy);
+  llvm::Value *EmitSVEMaskedLoad(llvm::Type *ReturnTy,
+                                 SmallVectorImpl<llvm::Value *> &Ops);
+
   llvm::Value *EmitAArch64BuiltinExpr(unsigned BuiltinID, const CallExpr *E,
                                       llvm::Triple::ArchType Arch);
   llvm::Value *EmitBPFBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
 
   llvm::Value *BuildVector(ArrayRef<llvm::Value*> Ops);
   llvm::Value *EmitX86BuiltinExpr(unsigned BuiltinID, const CallExpr *E);
   llvm::Value *EmitPPCBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
   llvm::Value *EmitAMDGPUBuiltinExpr(unsigned BuiltinID, const CallExpr *E);

clang/lib/CodeGen/CodeGenFunction.cpp

if (NormalCleanupDest.isValid() && isCoroutine()) {
     llvm::PromoteMemToReg(
         cast<llvm::AllocaInst>(NormalCleanupDest.getPointer()), DT);
     NormalCleanupDest = Address::invalid();
   }
 
   // Scan function arguments for vector width.
   for (llvm::Argument &A : CurFn->args())
     if (auto *VT = dyn_cast<llvm::VectorType>(A.getType()))
-      LargestVectorWidth = std::max((uint64_t)LargestVectorWidth,
-                                   VT->getPrimitiveSizeInBits().getFixedSize());
+      LargestVectorWidth =
+          std::max((uint64_t)LargestVectorWidth,
+                   VT->getPrimitiveSizeInBits().getKnownMinSize());
 
   // Update vector width based on return type.
   if (auto *VT = dyn_cast<llvm::VectorType>(CurFn->getReturnType()))
-    LargestVectorWidth = std::max((uint64_t)LargestVectorWidth,
-                                  VT->getPrimitiveSizeInBits().getFixedSize());
+    LargestVectorWidth =
+        std::max((uint64_t)LargestVectorWidth,
+                 VT->getPrimitiveSizeInBits().getKnownMinSize());
 
   // Add the required-vector-width attribute. This contains the max width from:
   // 1. min-vector-width attribute used in the source program.
   // 2. Any builtins used that have a vector width specified.
   // 3. Values passed in and out of inline assembly.
   // 4. Width of vector arguments and return types for this function.
   // 5. Width of vector aguments and return types for functions called by this
   //    function.
   CurFn->addFnAttr("min-legal-vector-width", llvm::utostr(LargestVectorWidth));

clang/lib/Headers/CMakeLists.txt

foreach( f ${files} ${cuda_wrapper_files} ${ppc_wrapper_files} ${openmp_wrapper_files})
   copy_header_to_output_dir(${CMAKE_CURRENT_SOURCE_DIR} ${f})
 endforeach( f )
 
 # Generate header files and copy them to the build directory
 # Generate arm_neon.h
 clang_generate_header(-gen-arm-neon arm_neon.td arm_neon.h)
 # Generate arm_fp16.h
 clang_generate_header(-gen-arm-fp16 arm_fp16.td arm_fp16.h)
+# Generate arm_sve.h
+clang_generate_header(-gen-arm-sve-header arm_sve.td arm_sve.h)
 # Generate arm_mve.h
 clang_generate_header(-gen-arm-mve-header arm_mve.td arm_mve.h)
 # Generate arm_cde.h
 clang_generate_header(-gen-arm-cde-header arm_cde.td arm_cde.h)
 
 add_custom_target(clang-resource-headers ALL DEPENDS ${out_files})
 set_target_properties(clang-resource-headers PROPERTIES
   FOLDER "Misc"

clang/lib/Headers/module.modulemap

explicit module arm {
     }
 
     explicit module neon {
       requires neon
       header "arm_neon.h"
       header "arm_fp16.h"
       export *
     }
+
+    explicit module sve {
+      requires sve
+      header "arm_sve.h"
+      export *
+    }
   }
 
   explicit module intel {
     requires x86
     export *
 
     header "immintrin.h"
     textual header "f16cintrin.h"

clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_ld1.c

+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -emit-llvm -o - %s -D__ARM_FEATURE_SVE | FileCheck %s
+
+#include <arm_sve.h>
+//
+// ld1
+//
+
+svint8_t test_svld1_s8(svbool_t pg, const int8_t *base)
+{
+  // CHECK-LABEL: test_svld1_s8
+  // CHECK: <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* %{{.*}}, i32 1, <vscale x 16 x i1> %{{.*}}, <vscale x 16 x i8> zeroinitializer)
+  return svld1_s8(pg, base);
+}
+
+svint16_t test_svld1_s16(svbool_t pg, const int16_t *base)
+{
+  // CHECK-LABEL: test_svld1_s16
+  // CHECK: <vscale x 8 x i16> @llvm.masked.load.nxv8i16.p0nxv8i16(<vscale x 8 x i16>* %{{.*}}, i32 1, <vscale x 8 x i1> %{{.*}}, <vscale x 8 x i16> zeroinitializer)
+  return svld1_s16(pg, base);
+}
+
+svint32_t test_svld1_s32(svbool_t pg, const int32_t *base)
+{
+  // CHECK-LABEL: test_svld1_s32
+  // CHECK: <vscale x 4 x i32> @llvm.masked.load.nxv4i32.p0nxv4i32(<vscale x 4 x i32>* %{{.*}}, i32 1, <vscale x 4 x i1> %{{.*}}, <vscale x 4 x i32> zeroinitializer)
+  return svld1_s32(pg, base);
+}
+
+svint64_t test_svld1_s64(svbool_t pg, const int64_t *base)
+{
+  // CHECK-LABEL: test_svld1_s64
+  // CHECK: <vscale x 2 x i64> @llvm.masked.load.nxv2i64.p0nxv2i64(<vscale x 2 x i64>* %{{.*}}, i32 1, <vscale x 2 x i1> %{{.*}}, <vscale x 2 x i64> zeroinitializer)
+  return svld1_s64(pg, base);
+}
+
+svuint8_t test_svld1_u8(svbool_t pg, const uint8_t *base)
+{
+  // CHECK-LABEL: test_svld1_u8
+  // CHECK: <vscale x 16 x i8> @llvm.masked.load.nxv16i8.p0nxv16i8(<vscale x 16 x i8>* %{{.*}}, i32 1, <vscale x 16 x i1> %{{.*}}, <vscale x 16 x i8> zeroinitializer)
+  return svld1_u8(pg, base);
+}
+
+svuint16_t test_svld1_u16(svbool_t pg, const uint16_t *base)
+{
+  // CHECK-LABEL: test_svld1_u16
+  // CHECK: <vscale x 8 x i16> @llvm.masked.load.nxv8i16.p0nxv8i16(<vscale x 8 x i16>* %{{.*}}, i32 1, <vscale x 8 x i1> %{{.*}}, <vscale x 8 x i16> zeroinitializer)
+  return svld1_u16(pg, base);
+}
+
+svuint32_t test_svld1_u32(svbool_t pg, const uint32_t *base)
+{
+  // CHECK-LABEL: test_svld1_u32
+  // CHECK: <vscale x 4 x i32> @llvm.masked.load.nxv4i32.p0nxv4i32(<vscale x 4 x i32>* %{{.*}}, i32 1, <vscale x 4 x i1> %{{.*}}, <vscale x 4 x i32> zeroinitializer)
+  return svld1_u32(pg, base);
+}
+
+svuint64_t test_svld1_u64(svbool_t pg, const uint64_t *base)
+{
+  // CHECK-LABEL: test_svld1_u64
+  // CHECK: <vscale x 2 x i64> @llvm.masked.load.nxv2i64.p0nxv2i64(<vscale x 2 x i64>* %{{.*}}, i32 1, <vscale x 2 x i1> %{{.*}}, <vscale x 2 x i64> zeroinitializer)
+  return svld1_u64(pg, base);
+}
+
+svfloat16_t test_svld1_f16(svbool_t pg, const float16_t *base)
+{
+  // CHECK-LABEL: test_svld1_f16
+  // CHECK: <vscale x 8 x half> @llvm.masked.load.nxv8f16.p0nxv8f16(<vscale x 8 x half>* %{{.*}}, i32 1, <vscale x 8 x i1> %{{.*}}, <vscale x 8 x half> zeroinitializer)
+  return svld1_f16(pg, base);
+}
+
+svfloat32_t test_svld1_f32(svbool_t pg, const float32_t *base)
+{
+  // CHECK-LABEL: test_svld1_f32
+  // CHECK: <vscale x 4 x float> @llvm.masked.load.nxv4f32.p0nxv4f32(<vscale x 4 x float>* %{{.*}}, i32 1, <vscale x 4 x i1> %{{.*}}, <vscale x 4 x float> zeroinitializer)
+  return svld1_f32(pg, base);
+}
+
+svfloat64_t test_svld1_f64(svbool_t pg, const float64_t *base)
+{
+  // CHECK-LABEL: test_svld1_f64
+  // CHECK: <vscale x 2 x double> @llvm.masked.load.nxv2f64.p0nxv2f64(<vscale x 2 x double>* %{{.*}}, i32 1, <vscale x 2 x i1> %{{.*}}, <vscale x 2 x double> zeroinitializer)
+  return svld1_f64(pg, base);
+}

clang/utils/TableGen/CMakeLists.txt

add_tablegen(clang-tblgen CLANG
   ClangDiagnosticsEmitter.cpp
   ClangOpcodesEmitter.cpp
   ClangOpenCLBuiltinEmitter.cpp
   ClangOptionDocEmitter.cpp
   ClangSACheckersEmitter.cpp
   ClangTypeNodesEmitter.cpp
   MveEmitter.cpp
   NeonEmitter.cpp
+  SveEmitter.cpp
   TableGen.cpp
   )
 set_target_properties(clang-tblgen PROPERTIES FOLDER "Clang tablegenning")

clang/utils/TableGen/SveEmitter.cpp

+//===- SveEmitter.cpp - Generate arm_sve.h for use with clang -*- C++ -*-===//
+//
+//  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//  See https://llvm.org/LICENSE.txt for license information.
+//  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This tablegen backend is responsible for emitting arm_sve.h, which includes
+// a declaration and definition of each function specified by the ARM C/C++
+// Language Extensions (ACLE).
+//
+// For details, visit:
+//  https://developer.arm.com/architectures/system-architectures/software-standards/acle
+//
+// Each SVE instruction is implemented in terms of 1 or more functions which
+// are suffixed with the element type of the input vectors.  Functions may be
+// implemented in terms of generic vector operations such as +, *, -, etc. or
+// by calling a __builtin_-prefixed function which will be handled by clang's
+// CodeGen library.
+//
+// See also the documentation in include/clang/Basic/arm_sve.td.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/TableGen/Record.h"
+#include "llvm/TableGen/Error.h"
+#include <string>
+#include <sstream>
+#include <set>
+#include <cctype>
+
+using namespace llvm;
+
+//===----------------------------------------------------------------------===//
+// SVEEmitter
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+class SVEEmitter {
+private:
+  RecordKeeper &Records;
+
+public:
+  SVEEmitter(RecordKeeper &R) : Records(R) {}
+
+  // run - Emit arm_sve.h
+  void run(raw_ostream &o);
+};
+
+} // end anonymous namespace
+
+
+//===----------------------------------------------------------------------===//
+// SVEEmitter implementation
+//===----------------------------------------------------------------------===//
+
+void SVEEmitter::run(raw_ostream &OS) {
+  OS << "/*===---- arm_sve.h - ARM SVE intrinsics "
+        "-----------------------------------===\n"
+        " *\n"
+        " *\n"
+        " * Part of the LLVM Project, under the Apache License v2.0 with LLVM "
+        "Exceptions.\n"
+        " * See https://llvm.org/LICENSE.txt for license information.\n"
+        " * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception\n"
+        " *\n"
+        " *===-----------------------------------------------------------------"
+        "------===\n"
+        " */\n\n";
+
+  OS << "#ifndef __ARM_SVE_H\n";
+  OS << "#define __ARM_SVE_H\n\n";
+
+  OS << "#if !defined(__ARM_FEATURE_SVE)\n";
+  OS << "#error \"SVE support not enabled\"\n";
+  OS << "#else\n\n";
+
+  OS << "#include <stdint.h>\n\n";
+  OS << "#ifndef  __cplusplus\n";
+  OS << "#include <stdbool.h>\n";
+  OS << "#endif\n\n";
+
+  OS << "typedef __fp16 float16_t;\n";
+  OS << "typedef float float32_t;\n";
+  OS << "typedef double float64_t;\n";
+  OS << "typedef bool bool_t;\n\n";
+
+  OS << "typedef __SVInt8_t svint8_t;\n";
+  OS << "typedef __SVInt16_t svint16_t;\n";
+  OS << "typedef __SVInt32_t svint32_t;\n";
+  OS << "typedef __SVInt64_t svint64_t;\n";
+  OS << "typedef __SVUint8_t svuint8_t;\n";
+  OS << "typedef __SVUint16_t svuint16_t;\n";
+  OS << "typedef __SVUint32_t svuint32_t;\n";
+  OS << "typedef __SVUint64_t svuint64_t;\n";
+  OS << "typedef __SVFloat16_t svfloat16_t;\n";
+  OS << "typedef __SVFloat32_t svfloat32_t;\n";
+  OS << "typedef __SVFloat64_t svfloat64_t;\n";
+  OS << "typedef __SVBool_t  svbool_t;\n\n";
+
+  OS << "#define svld1_u8(...) __builtin_sve_svld1_u8(__VA_ARGS__)\n";
+  OS << "#define svld1_u16(...) __builtin_sve_svld1_u16(__VA_ARGS__)\n";
+  OS << "#define svld1_u32(...) __builtin_sve_svld1_u32(__VA_ARGS__)\n";
+  OS << "#define svld1_u64(...) __builtin_sve_svld1_u64(__VA_ARGS__)\n";
+  OS << "#define svld1_s8(...) __builtin_sve_svld1_s8(__VA_ARGS__)\n";
+  OS << "#define svld1_s16(...) __builtin_sve_svld1_s16(__VA_ARGS__)\n";
+  OS << "#define svld1_s32(...) __builtin_sve_svld1_s32(__VA_ARGS__)\n";
+  OS << "#define svld1_s64(...) __builtin_sve_svld1_s64(__VA_ARGS__)\n";
+  OS << "#define svld1_f16(...) __builtin_sve_svld1_f16(__VA_ARGS__)\n";
+  OS << "#define svld1_f32(...) __builtin_sve_svld1_f32(__VA_ARGS__)\n";
+  OS << "#define svld1_f64(...) __builtin_sve_svld1_f64(__VA_ARGS__)\n";
+
+  OS << "#endif /*__ARM_FEATURE_SVE */\n";
+  OS << "#endif /* __ARM_SVE_H */\n";
+}
+
+namespace clang {
+void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).run(OS);
+}
+
+} // End namespace clang

clang/utils/TableGen/TableGen.cpp

enum ActionType {
   GenArmFP16,
   GenArmNeonSema,
   GenArmNeonTest,
   GenArmMveHeader,
   GenArmMveBuiltinDef,
   GenArmMveBuiltinSema,
   GenArmMveBuiltinCG,
   GenArmMveBuiltinAliases,
+  GenArmSveHeader,
   GenArmCdeHeader,
   GenArmCdeBuiltinDef,
   GenArmCdeBuiltinSema,
   GenArmCdeBuiltinCG,
   GenArmCdeBuiltinAliases,
   GenAttrDocs,
   GenDiagDocs,
   GenOptDocs,
cl::values(
         clEnumValN(GenClangOpenCLBuiltins, "gen-clang-opencl-builtins",
                    "Generate OpenCL builtin declaration handlers"),
         clEnumValN(GenArmNeon, "gen-arm-neon", "Generate arm_neon.h for clang"),
         clEnumValN(GenArmFP16, "gen-arm-fp16", "Generate arm_fp16.h for clang"),
         clEnumValN(GenArmNeonSema, "gen-arm-neon-sema",
                    "Generate ARM NEON sema support for clang"),
         clEnumValN(GenArmNeonTest, "gen-arm-neon-test",
                    "Generate ARM NEON tests for clang"),
+        clEnumValN(GenArmSveHeader, "gen-arm-sve-header",
+                   "Generate arm_sve.h for clang"),
         clEnumValN(GenArmMveHeader, "gen-arm-mve-header",
                    "Generate arm_mve.h for clang"),
         clEnumValN(GenArmMveBuiltinDef, "gen-arm-mve-builtin-def",
                    "Generate ARM MVE builtin definitions for clang"),
         clEnumValN(GenArmMveBuiltinSema, "gen-arm-mve-builtin-sema",
                    "Generate ARM MVE builtin sema checks for clang"),
         clEnumValN(GenArmMveBuiltinCG, "gen-arm-mve-builtin-codegen",
                    "Generate ARM MVE builtin code-generator for clang"),
case GenArmMveBuiltinSema:
     EmitMveBuiltinSema(Records, OS);
     break;
   case GenArmMveBuiltinCG:
     EmitMveBuiltinCG(Records, OS);
     break;
   case GenArmMveBuiltinAliases:
     EmitMveBuiltinAliases(Records, OS);
     break;
+  case GenArmSveHeader:
+    EmitSveHeader(Records, OS);
+    break;
   case GenArmCdeHeader:
     EmitCdeHeader(Records, OS);
     break;
   case GenArmCdeBuiltinDef:
     EmitCdeBuiltinDef(Records, OS);
     break;
   case GenArmCdeBuiltinSema:
     EmitCdeBuiltinSema(Records, OS);

clang/utils/TableGen/TableGenBackends.h

 void EmitNeon(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitFP16(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonSema(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonTest(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeon2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonSema2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitNeonTest2(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
+void EmitSveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+
 void EmitMveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinDef(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinSema(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinAliases(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitCdeHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitCdeBuiltinDef(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);