[SveEmitter] Fix encoding/decoding of SVETypeFlags

[sdesmalen]

Summary

This issue was introduced when reworking D75861. The bug isn't
actually hit with current unit tests because the contiguous loads/stores
infer the EltType and the MemEltType from the pointer and result, rather
than using the flags. But it will be needed for other intrinsics, such as
gather/scatter.

[andwar]

Cheers for the fix @sdesmalen !

clang/include/clang/Basic/TargetBuiltins.h

188–197

EltTypeShift and MemEltTypeShift are initialised twice in one constructor.

If removing them from the member initialisation list throws a warning, perhaps you could use in-class initialisers?

[SjoerdMeijer]

Patches with functional changes but without tests are a bit "suspicious". In this case, I see it might be tricky. You could argue that it should be incorporated in the patch that includes the tests, so we can see what's happening. But perhaps separating this out is cleaner, if the other patch is big. But can you at least make the next patch dependent on this one, so we know where this is used?

[sdesmalen]

In D76617#1938787, @SjoerdMeijer wrote:

Patches with functional changes but without tests are a bit "suspicious". In this case, I see it might be tricky. You could argue that it should be incorporated in the patch that includes the tests, so we can see what's happening. But perhaps separating this out is cleaner, if the other patch is big. But can you at least make the next patch dependent on this one, so we know where this is used?

Yes you're right, the patch that I've made dependent needs this one to work properly.

[SjoerdMeijer]

Yes you're right, the patch that I've made dependent needs this one to work properly.

Ah ok, I may have missed that, will have a look

[sdesmalen]

• Made Flags uint64_t instead of unsigned.
• Simplified encoding of flags in SveEmitter and moved encoding of properties into the Flags variable to the constructor of Intrinsic.
• Added (encoding of) MergeType as well.

[SjoerdMeijer]

clang/utils/TableGen/SveEmitter.cpp

229

Should V now be an uint64_t?

[sdesmalen]

• Updated encode functions to take/return uint64_t instead of unsigned.

[sdesmalen]

clang/utils/TableGen/SveEmitter.cpp

229

Yes, good spot!

[SjoerdMeijer]

Cheers, LGTM

[andwar]

clang/utils/TableGen/SveEmitter.cpp

240

Shouldn't MT be uint64_t instead of unsigned?

[sdesmalen]

Changed unsigned -> uint64_t for encodeMemoryElementType and encodeMergeType.

[andwar]

LGTM

This patch now implements a bit more than the original commit message would suggest. Could you please update? Thanks for working on this!

clang/include/clang/Basic/TargetBuiltins.h

 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_BASIC_TARGETBUILTINS_H
 #define LLVM_CLANG_BASIC_TARGETBUILTINS_H
 
 #include <stdint.h>
 #include "clang/Basic/Builtins.h"
+#include "llvm/Support/MathExtras.h"
 #undef PPC
 
 namespace clang {
 
   namespace NEON {
   enum {
     LastTIBuiltin = clang::Builtin::FirstTSBuiltin - 1,
 #define BUILTIN(ID, TYPE, ATTRS) BI##ID,
public:
     }
     bool isUnsigned() const { return (Flags & UnsignedFlag) != 0; }
     bool isQuad() const { return (Flags & QuadFlag) != 0; }
   };
 
   /// Flags to identify the types for overloaded SVE builtins.
   class SVETypeFlags {
     uint64_t Flags;
+    unsigned EltTypeShift;
+    unsigned MemEltTypeShift;
+    unsigned MergeTypeShift;
 
   public:
 #define LLVM_GET_SVE_TYPEFLAGS
 #include "clang/Basic/arm_sve_typeflags.inc"
 #undef LLVM_GET_SVE_TYPEFLAGS
 
     enum EltType {
 #define LLVM_GET_SVE_ELTTYPES
 #include "clang/Basic/arm_sve_typeflags.inc"
 #undef LLVM_GET_SVE_ELTTYPES
     };
 
     enum MemEltType {
 #define LLVM_GET_SVE_MEMELTTYPES
 #include "clang/Basic/arm_sve_typeflags.inc"
 #undef LLVM_GET_SVE_MEMELTTYPES
     };
 
-    SVETypeFlags(uint64_t F) : Flags(F) {}
-    SVETypeFlags(EltType ET, bool IsUnsigned) : Flags(ET) {}
+    enum MergeType {
+#define LLVM_GET_SVE_MERGETYPES
+#include "clang/Basic/arm_sve_typeflags.inc"
+#undef LLVM_GET_SVE_MERGETYPES
+    };
+    SVETypeFlags(uint64_t F) : Flags(F) {
+      EltTypeShift = llvm::countTrailingZeros(EltTypeMask);
+      MemEltTypeShift = llvm::countTrailingZeros(MemEltTypeMask);
+      MergeTypeShift = llvm::countTrailingZeros(MergeTypeMask);
+    }
 
     EltType getEltType() const {
-      return (EltType)((Flags & EltTypeMask) - FirstEltType);
+      return (EltType)((Flags & EltTypeMask) >> EltTypeShift);
     }
 
     MemEltType getMemEltType() const {
-      return (MemEltType)((Flags & MemEltTypeMask) - FirstMemEltType);
+      return (MemEltType)((Flags & MemEltTypeMask) >> MemEltTypeShift);
+    }
+
+    MergeType getMergeType() const {
+      return (MergeType)((Flags & MergeTypeMask) >> MergeTypeShift);
     }
 
     bool isLoad() const { return Flags & IsLoad; }
     bool isStore() const { return Flags & IsStore; }
     bool isGatherLoad() const { return Flags & IsGatherLoad; }
     bool isScatterStore() const { return Flags & IsScatterStore; }
     bool isStructLoad() const { return Flags & IsStructLoad; }
     bool isStructStore() const { return Flags & IsStructStore; }

clang/include/clang/Basic/arm_sve.td

 // U: const pointer to int32_t
 // V: const pointer to int64_t
 //
 // W: const pointer to uint8_t
 // X: const pointer to uint16_t
 // Y: const pointer to uint32_t
 // Z: const pointer to uint64_t
 
-class MergeType<int val> {
+class MergeType<int val, string suffix=""> {
   int Value = val;
+  string Suffix = suffix;
 }
 def MergeNone    : MergeType<0>;
-def MergeAny     : MergeType<1>;
-def MergeOp1     : MergeType<2>;
-def MergeZero    : MergeType<3>;
-def MergeAnyExp  : MergeType<4>; // Use merged builtin with explicit
-def MergeZeroExp : MergeType<5>; // generation of its inactive argument.
+def MergeAny     : MergeType<1, "_x">;
+def MergeOp1     : MergeType<2, "_m">;
+def MergeZero    : MergeType<3, "_z">;
+def MergeAnyExp  : MergeType<4, "_x">; // Use merged builtin with explicit
+def MergeZeroExp : MergeType<5, "_z">; // generation of its inactive argument.
 
 class EltType<int val> {
   int Value = val;
 }
 def EltTyInvalid : EltType<0>;
 def EltTyInt8    : EltType<1>;
 def EltTyInt16   : EltType<2>;
 def EltTyInt32   : EltType<3>;

 def FirstEltType              : FlagType<0x00000001>;
 //      :                                     :
 //      :                                     :
 def EltTypeMask               : FlagType<0x0000000f>;
 def FirstMemEltType           : FlagType<0x00000010>;
 //      :                                     :
 //      :                                     :
 def MemEltTypeMask            : FlagType<0x00000070>;
-def IsLoad                    : FlagType<0x00000080>;
-def IsStore                   : FlagType<0x00000100>;
-def IsGatherLoad              : FlagType<0x00000200>;
-def IsScatterStore            : FlagType<0x00000400>;
-def IsStructLoad              : FlagType<0x00000800>;
-def IsStructStore             : FlagType<0x00001000>;
-def IsZExtReturn              : FlagType<0x00002000>; // Return value is sign-extend by default
+def FirstMergeTypeMask        : FlagType<0x00000080>;
+//      :                                     :
+//      :                                     :
+def MergeTypeMask             : FlagType<0x00000380>;
+def IsLoad                    : FlagType<0x00002000>;
+def IsStore                   : FlagType<0x00004000>;
+def IsGatherLoad              : FlagType<0x00008000>;
+def IsScatterStore            : FlagType<0x00010000>;
+def IsStructLoad              : FlagType<0x00020000>;
+def IsStructStore             : FlagType<0x00040000>;
+def IsZExtReturn              : FlagType<0x00080000>; // Return value is sign-extend by default
 
 // Every intrinsic subclasses Inst.
 class Inst<string n, string p, string t, MergeType mt, string i,
            list<FlagType> ft, MemEltType met> {
   string Name = n;
   string Prototype = p;
   string Types = t;
   string ArchGuard = "";
   int Merge = mt.Value;
+  string MergeSuffix = mt.Suffix;
   string LLVMIntrinsic = i;
   list<FlagType> Flags = ft;
   int MemEltType = met.Value;
 }
 
 // MInst: Instructions which access memory
 class MInst<string n, string p, string t, list<FlagType> f,
             MemEltType met=MemEltTyDefault, string i="">

clang/utils/TableGen/SveEmitter.cpp

SVEType(TypeSpec TS, char CharMod)
         Constant(false), Pointer(false), DefaultType(false), IsScalable(true),
         Predicate(false), PredicatePattern(false), PrefetchOp(false),
         Bitwidth(128), ElementBitwidth(~0U), NumVectors(1) {
     if (!TS.empty())
       applyTypespec();
     applyModifier(CharMod);
   }
 
-  /// Return the value in SVETypeFlags for this type.
-  unsigned getTypeFlags() const;
-
   bool isPointer() const { return Pointer; }
   bool isVoidPointer() const { return Pointer && Void; }
   bool isSigned() const { return Signed; }
   bool isImmediate() const { return Immediate; }
   bool isScalar() const { return NumVectors == 0; }
   bool isVector() const { return NumVectors > 0; }
   bool isScalableVector() const { return isVector() && IsScalable; }
   bool isChar() const { return ElementBitwidth == 8; }
class Intrinsic {
   /// The base class kind. Most intrinsics use ClassS, which has full type
   /// info for integers (_s32/_u32), or ClassG which is used for overloaded
   /// intrinsics.
   ClassKind Class;
 
   /// The architectural #ifdef guard.
   std::string Guard;
 
+  // The merge suffix such as _m, _x or _z.
+  std::string MergeSuffix;
+
   /// The types of return value [0] and parameters [1..].
   std::vector<SVEType> Types;
 
   /// The "base type", which is VarType('d', BaseTypeSpec).
   SVEType BaseType;
 
-  unsigned Flags;
+  uint64_t Flags;
 
 public:
-  /// The type of predication.
-  enum MergeType {
-    MergeNone,
-    MergeAny,
-    MergeOp1,
-    MergeZero,
-    MergeAnyExp,
-    MergeZeroExp,
-    MergeInvalid
-  } Merge;
-
-  Intrinsic(StringRef Name, StringRef Proto, int64_t MT, StringRef LLVMName,
-            unsigned Flags, TypeSpec BT, ClassKind Class, SVEEmitter &Emitter,
-            StringRef Guard)
-      : Name(Name.str()), LLVMName(LLVMName), Proto(Proto.str()),
-        BaseTypeSpec(BT), Class(Class), Guard(Guard.str()), BaseType(BT, 'd'),
-        Flags(Flags), Merge(MergeType(MT)) {
-    // Types[0] is the return value.
-    for (unsigned I = 0; I < Proto.size(); ++I)
-      Types.emplace_back(BaseTypeSpec, Proto[I]);
-  }
+  Intrinsic(StringRef Name, StringRef Proto, uint64_t MergeTy,
+            StringRef MergeSuffix, uint64_t MemoryElementTy, StringRef LLVMName,
+            uint64_t Flags, TypeSpec BT, ClassKind Class, SVEEmitter &Emitter,
+            StringRef Guard);
 
   ~Intrinsic()=default;
 
   std::string getName() const { return Name; }
   std::string getLLVMName() const { return LLVMName; }
   std::string getProto() const { return Proto; }
   TypeSpec getBaseTypeSpec() const { return BaseTypeSpec; }
   SVEType getBaseType() const { return BaseType; }
 
   StringRef getGuard() const { return Guard; }
   ClassKind getClassKind() const { return Class; }
-  MergeType getMergeType() const { return Merge; }
 
   SVEType getReturnType() const { return Types[0]; }
   ArrayRef<SVEType> getTypes() const { return Types; }
   SVEType getParamType(unsigned I) const { return Types[I + 1]; }
   unsigned getNumParams() const { return Proto.size() - 1; }
 
-  unsigned getFlags() const { return Flags; }
+  uint64_t getFlags() const { return Flags; }
   bool isFlagSet(uint64_t Flag) const { return Flags & Flag;}
 
   /// Return the type string for a BUILTIN() macro in Builtins.def.
   std::string getBuiltinTypeStr();
 
   /// Return the name, mangled with type information. The name is mangled for
   /// ClassS, so will add type suffixes such as _u32/_s32.
   std::string getMangledName() const { return mangleName(ClassS); }
 
   /// Returns true if the intrinsic is overloaded, in that it should also generate
   /// a short form without the type-specifiers, e.g. 'svld1(..)' instead of
   /// 'svld1_u32(..)'.
   static bool isOverloadedIntrinsic(StringRef Name) {
     auto BrOpen = Name.find("[");
     auto BrClose = Name.find(']');
     return BrOpen != std::string::npos && BrClose != std::string::npos;
   }
 
   /// Emits the intrinsic declaration to the ostream.
   void emitIntrinsic(raw_ostream &OS) const;
 
 private:
-  std::string getMergeSuffix() const;
+  std::string getMergeSuffix() const { return MergeSuffix; }
   std::string mangleName(ClassKind LocalCK) const;
   std::string replaceTemplatedArgs(std::string Name, TypeSpec TS,
                                    std::string Proto) const;
 };
 
 class SVEEmitter {
 private:
   RecordKeeper &Records;
   llvm::StringMap<uint64_t> EltTypes;
   llvm::StringMap<uint64_t> MemEltTypes;
   llvm::StringMap<uint64_t> FlagTypes;
+  llvm::StringMap<uint64_t> MergeTypes;
 
-  unsigned getTypeFlags(const SVEType &T);
 public:
   SVEEmitter(RecordKeeper &R) : Records(R) {
     for (auto *RV : Records.getAllDerivedDefinitions("EltType"))
       EltTypes[RV->getNameInitAsString()] = RV->getValueAsInt("Value");
     for (auto *RV : Records.getAllDerivedDefinitions("MemEltType"))
       MemEltTypes[RV->getNameInitAsString()] = RV->getValueAsInt("Value");
     for (auto *RV : Records.getAllDerivedDefinitions("FlagType"))
       FlagTypes[RV->getNameInitAsString()] = RV->getValueAsInt("Value");
+    for (auto *RV : Records.getAllDerivedDefinitions("MergeType"))
+      MergeTypes[RV->getNameInitAsString()] = RV->getValueAsInt("Value");
+  }
+
+  // Returns the SVETypeFlags for a given value and mask.
+  uint64_t encodeFlag(uint64_t V, StringRef MaskName) const {
+    auto It = FlagTypes.find(MaskName);
+    if (It != FlagTypes.end()) {
+      uint64_t Mask = It->getValue();
+      unsigned Shift = llvm::countTrailingZeros(Mask);
+      return (V << Shift) & Mask;
+    }
+    llvm_unreachable("Unsupported flag");
   }
 
+  // Returns the SVETypeFlags for the given element type.
+  uint64_t encodeEltType(StringRef EltName) {
+    auto It = EltTypes.find(EltName);
+    if (It != EltTypes.end())
+      return encodeFlag(It->getValue(), "EltTypeMask");
+    llvm_unreachable("Unsupported EltType");
+  }
+
+  // Returns the SVETypeFlags for the given memory element type.
+  uint64_t encodeMemoryElementType(uint64_t MT) {
+    return encodeFlag(MT, "MemEltTypeMask");
+  }
+
+  // Returns the SVETypeFlags for the given merge type.
+  uint64_t encodeMergeType(uint64_t MT) {
+    return encodeFlag(MT, "MergeTypeMask");
+  }
+
+  // Returns the SVETypeFlags value for the given SVEType.
+  uint64_t encodeTypeFlags(const SVEType &T);
+
   /// Emit arm_sve.h.
   void createHeader(raw_ostream &o);
 
   /// Emit all the __builtin prototypes and code needed by Sema.
   void createBuiltins(raw_ostream &o);
 
   /// Emit all the information needed to map builtin -> LLVM IR intrinsic.
   void createCodeGenMap(raw_ostream &o);
 
   /// Create the SVETypeFlags used in CGBuiltins
   void createTypeFlags(raw_ostream &o);
 
   /// Create intrinsic and add it to \p Out
   void createIntrinsic(Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out);
 };
 
 } // end anonymous namespace
 
 
 //===----------------------------------------------------------------------===//
 // Type implementation
 //===----------------------------------------------------------------------===//
 
-unsigned SVEEmitter::getTypeFlags(const SVEType &T) {
-  unsigned FirstEltType = EltTypes["FirstEltType"];
-  if (T.isFloat()) {
-    switch (T.getElementSizeInBits()) {
-    case 16: return FirstEltType + EltTypes["EltTyFloat16"];
-    case 32: return FirstEltType + EltTypes["EltTyFloat32"];
-    case 64: return FirstEltType + EltTypes["EltTyFloat64"];
-    default: llvm_unreachable("Unhandled float element bitwidth!");
-    }
-  }
-
-  if (T.isPredicateVector()) {
-    switch (T.getElementSizeInBits()) {
-    case 8:  return FirstEltType + EltTypes["EltTyBool8"];
-    case 16: return FirstEltType + EltTypes["EltTyBool16"];
-    case 32: return FirstEltType + EltTypes["EltTyBool32"];
-    case 64: return FirstEltType + EltTypes["EltTyBool64"];
-    default: llvm_unreachable("Unhandled predicate element bitwidth!");
-    }
-  }
-
-  switch (T.getElementSizeInBits()) {
-  case 8:  return FirstEltType + EltTypes["EltTyInt8"];
-  case 16: return FirstEltType + EltTypes["EltTyInt16"];
-  case 32: return FirstEltType + EltTypes["EltTyInt32"];
-  case 64: return FirstEltType + EltTypes["EltTyInt64"];
-  default: llvm_unreachable("Unhandled integer element bitwidth!");
-  }
-}
-
 std::string SVEType::builtin_str() const {
   std::string S;
   if (isVoid())
     return "v";
 
   if (isVoidPointer())
     S += "v";
   else if (!Float)
void SVEType::applyModifier(char Mod) {
   }
 }
 
 
 //===----------------------------------------------------------------------===//
 // Intrinsic implementation
 //===----------------------------------------------------------------------===//
 
+Intrinsic::Intrinsic(StringRef Name, StringRef Proto, uint64_t MergeTy,
+                     StringRef MergeSuffix, uint64_t MemoryElementTy,
+                     StringRef LLVMName, uint64_t Flags, TypeSpec BT,
+                     ClassKind Class, SVEEmitter &Emitter, StringRef Guard)
+    : Name(Name.str()), LLVMName(LLVMName), Proto(Proto.str()),
+      BaseTypeSpec(BT), Class(Class), Guard(Guard.str()),
+      MergeSuffix(MergeSuffix.str()), BaseType(BT, 'd'), Flags(Flags) {
+
+  // Types[0] is the return value.
+  for (unsigned I = 0; I < Proto.size(); ++I) {
+    SVEType T(BaseTypeSpec, Proto[I]);
+    Types.push_back(T);
+  }
+
+  // Set flags based on properties
+  this->Flags |= Emitter.encodeTypeFlags(BaseType);
+  this->Flags |= Emitter.encodeMemoryElementType(MemoryElementTy);
+  this->Flags |= Emitter.encodeMergeType(MergeTy);
+}
+
 std::string Intrinsic::getBuiltinTypeStr() {
   std::string S;
 
   SVEType RetT = getReturnType();
   // Since the return value must be one type, return a vector type of the
   // appropriate width which we will bitcast.  An exception is made for
   // returning structs of 2, 3, or 4 vectors which are returned in a sret-like
   // fashion, storing them to a pointer arg.
while (Ret.find('{') != std::string::npos) {
     else
       TypeCode = 'f';
     Ret.replace(Pos, NumChars, TypeCode + utostr(T.getElementSizeInBits()));
   }
 
   return Ret;
 }
 
-// ACLE function names have a merge style postfix.
-std::string Intrinsic::getMergeSuffix() const {
-  switch (getMergeType()) {
-    default:
-      llvm_unreachable("Unknown predication specifier");
-    case MergeNone:    return "";
-    case MergeAny:
-    case MergeAnyExp:  return "_x";
-    case MergeOp1:     return "_m";
-    case MergeZero:
-    case MergeZeroExp: return "_z";
-  }
-}
-
 std::string Intrinsic::mangleName(ClassKind LocalCK) const {
   std::string S = getName();
 
   if (LocalCK == ClassG) {
     // Remove the square brackets and everything in between.
     while (S.find("[") != std::string::npos) {
       auto Start = S.find("[");
       auto End = S.find(']');
if (getClassKind() != ClassG || getProto().size() <= 1) {
     }
     OS << ");\n";
   }
 }
 
 //===----------------------------------------------------------------------===//
 // SVEEmitter implementation
 //===----------------------------------------------------------------------===//
+uint64_t SVEEmitter::encodeTypeFlags(const SVEType &T) {
+  if (T.isFloat()) {
+    switch (T.getElementSizeInBits()) {
+    case 16:
+      return encodeEltType("EltTyFloat16");
+    case 32:
+      return encodeEltType("EltTyFloat32");
+    case 64:
+      return encodeEltType("EltTyFloat64");
+    default:
+      llvm_unreachable("Unhandled float element bitwidth!");
+    }
+  }
+
+  if (T.isPredicateVector()) {
+    switch (T.getElementSizeInBits()) {
+    case 8:
+      return encodeEltType("EltTyBool8");
+    case 16:
+      return encodeEltType("EltTyBool16");
+    case 32:
+      return encodeEltType("EltTyBool32");
+    case 64:
+      return encodeEltType("EltTyBool64");
+    default:
+      llvm_unreachable("Unhandled predicate element bitwidth!");
+    }
+  }
+
+  switch (T.getElementSizeInBits()) {
+  case 8:
+    return encodeEltType("EltTyInt8");
+  case 16:
+    return encodeEltType("EltTyInt16");
+  case 32:
+    return encodeEltType("EltTyInt32");
+  case 64:
+    return encodeEltType("EltTyInt64");
+  default:
+    llvm_unreachable("Unhandled integer element bitwidth!");
+  }
+}
+
 void SVEEmitter::createIntrinsic(
     Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out) {
   StringRef Name = R->getValueAsString("Name");
   StringRef Proto = R->getValueAsString("Prototype");
   StringRef Types = R->getValueAsString("Types");
   StringRef Guard = R->getValueAsString("ArchGuard");
   StringRef LLVMName = R->getValueAsString("LLVMIntrinsic");
-  int64_t Merge = R->getValueAsInt("Merge");
+  uint64_t Merge = R->getValueAsInt("Merge");
+  StringRef MergeSuffix = R->getValueAsString("MergeSuffix");
+  uint64_t MemEltType = R->getValueAsInt("MemEltType");
   std::vector<Record*> FlagsList = R->getValueAsListOfDefs("Flags");
 
   int64_t Flags = 0;
   for (auto FlagRec : FlagsList)
     Flags |= FlagRec->getValueAsInt("Value");
-  Flags |= R->getValueAsInt("MemEltType") + MemEltTypes["FirstMemEltType"];
 
   // Extract type specs from string
   SmallVector<TypeSpec, 8> TypeSpecs;
   TypeSpec Acc;
   for (char I : Types) {
     Acc.push_back(I);
     if (islower(I)) {
       TypeSpecs.push_back(TypeSpec(Acc));
       Acc.clear();
     }
   }
 
   // Remove duplicate type specs.
   std::sort(TypeSpecs.begin(), TypeSpecs.end());
   TypeSpecs.erase(std::unique(TypeSpecs.begin(), TypeSpecs.end()),
                   TypeSpecs.end());
 
   // Create an Intrinsic for each type spec.
   for (auto TS : TypeSpecs) {
-    Out.push_back(std::make_unique<Intrinsic>(Name, Proto, Merge,
-                                              LLVMName, Flags, TS, ClassS,
-                                              *this, Guard));
+    Out.push_back(std::make_unique<Intrinsic>(Name, Proto, Merge, MergeSuffix,
+                                              MemEltType, LLVMName, Flags, TS,
+                                              ClassS, *this, Guard));
 
     // Also generate the short-form (e.g. svadd_m) for the given type-spec.
     if (Intrinsic::isOverloadedIntrinsic(Name))
-      Out.push_back(std::make_unique<Intrinsic>(
-          Name, Proto, Merge, LLVMName, Flags, TS, ClassG, *this, Guard));
+      Out.push_back(std::make_unique<Intrinsic>(Name, Proto, Merge, MergeSuffix,
+                                                MemEltType, LLVMName, Flags, TS,
+                                                ClassG, *this, Guard));
   }
 }
 
 void SVEEmitter::createHeader(raw_ostream &OS) {
   OS << "/*===---- arm_sve.h - ARM SVE intrinsics "
         "-----------------------------------===\n"
         " *\n"
         " *\n"
void SVEEmitter::createCodeGenMap(raw_ostream &OS) {
 
   OS << "#ifdef GET_SVE_LLVM_INTRINSIC_MAP\n";
   for (auto &Def : Defs) {
     // Builtins only exist for non-overloaded intrinsics, overloaded
     // declarations only live in the header file.
     if (Def->getClassKind() == ClassG)
       continue;
 
-    uint64_t Flags = Def->getFlags() | getTypeFlags(Def->getBaseType());
+    uint64_t Flags = Def->getFlags();
     auto FlagString = std::to_string(Flags);
 
     std::string LLVMName = Def->getLLVMName();
     std::string Builtin = Def->getMangledName();
     if (!LLVMName.empty())
       OS << "SVEMAP1(" << Builtin << ", " << LLVMName << ", " << FlagString
          << "),\n";
     else
void SVEEmitter::createTypeFlags(raw_ostream &OS) {
   for (auto &KV : EltTypes)
     OS << "  " << KV.getKey() << " = " << KV.getValue() << ",\n";
   OS << "#endif\n\n";
 
   OS << "#ifdef LLVM_GET_SVE_MEMELTTYPES\n";
   for (auto &KV : MemEltTypes)
     OS << "  " << KV.getKey() << " = " << KV.getValue() << ",\n";
   OS << "#endif\n\n";
+
+  OS << "#ifdef LLVM_GET_SVE_MERGETYPES\n";
+  for (auto &KV : MergeTypes)
+    OS << "  " << KV.getKey() << " = " << KV.getValue() << ",\n";
+  OS << "#endif\n\n";
 }
 
 namespace clang {
 void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createHeader(OS);
 }
 
 void EmitSveBuiltins(RecordKeeper &Records, raw_ostream &OS) {