[IntrisicEmitter] Allow intrinsic types to be dependent on the data layout

[arichardson]

Summary

For certain intrinsics (e.g. stack/code related ones), the address space
always has to be the alloca address space defined in the DataLayout and
there will not be another version of that intrinsic using other address
spaces in the same module.
This change adds new llvm_{prog,alloca,global}_ptr_ty to Intrinsics.td.
It only adds the infrastructure, follow-up commits will use it for e.g.
va_start,va_copy,va_end,returnaddress, etc.

The motivating use case here are the downstream CHERI targets
that use address space 200 for alloca/program/globals. So far CHERI LLVM
overloaded all those intrinsics but this caused a lot of downstream merge
conflicts in test files, so this approach may be more maintainable.

The alternative would be to overload all the remaining intrinsics but that
causes a lot of test churn (especially for llvm.va_*/llvm.stack{save,restore}*)
and should not be necessary since those intrinsics don't really need to
be overloaded, the types just happen to depend on the data layout string.

[arichardson]

try to fix mlir build

[arichardson]

fix style

[arichardson]

update

[arichardson]

llvm/lib/IR/Function.cpp

1314

'G' and 'P' are the wrong way around.

[arichardson]

fix inverted switch statement

[arsenm]

I have mixed feelings about this patch. It feels useful to support the typed matchers, mostly in the context of fixing iPTR (really we should have a more generally parameterized iPTR if we can't fully get rid of it)

The alternative would be to overload all the remaining intrinsics but that
causes a lot of test churn (especially for llvm.va_*/llvm.stack{save,restore}*)
and should not be necessary since those intrinsics don't really need to
be overloaded, the types just happen to depend on the data layout string.

I think this is the correct solution; I think llvm_ptr_ty should be removed entirely and all intrinsics should be mangled on the pointer address space. It's not composable and we've just been migrating an intrinsic at a time as needed. We've already migrated some intrinsics for the stack use case (e.g. int_frameaddress and lifetime markers) and this switches strategies for a different set so it introduces inconsistency. This is a roadblock to fully supporting per-alloca address spaces, which has come up before as potentially useful. The datalayout address spaces are really for situations where some generic code needs to create a new value without any context, which is not the case with an intrinsic call site.

This is also going to make these intrinsics syntactically behave differently based on the datalayout. We tried something similar with program address spaces, such that the assembler currently interprets an undecorated function's address space differently based on the datalayout and I'm currently hating it. It's inconsistent and confusing and I'll likely post a patch to undo it, such that functions are always printed with an explicit addrspace(N) if it's non-0.

Assuming we do want to go with this approach, this is missing various assembler and verifier tests for all the impacted intrinsics.

llvm/include/llvm/IR/Intrinsics.td

189

Just AddrSpace sounds like it's expecting the raw number, like in LLVMQualPointerType

272–274

This part feels useful

llvm/utils/TableGen/IntrinsicEmitter.cpp

387

This is a separate diagnostic improvement that should be split out

[arichardson]

In D138563#3950912, @arsenm wrote:

I have mixed feelings about this patch. It feels useful to support the typed matchers, mostly in the context of fixing iPTR (really we should have a more generally parameterized iPTR if we can't fully get rid of it)

The alternative would be to overload all the remaining intrinsics but that
causes a lot of test churn (especially for llvm.va_*/llvm.stack{save,restore}*)
and should not be necessary since those intrinsics don't really need to
be overloaded, the types just happen to depend on the data layout string.

I think this is the correct solution; I think llvm_ptr_ty should be removed entirely and all intrinsics should be mangled on the pointer address space. It's not composable and we've just been migrating an intrinsic at a time as needed. We've already migrated some intrinsics for the stack use case (e.g. int_frameaddress and lifetime markers) and this switches strategies for a different set so it introduces inconsistency. This is a roadblock to fully supporting per-alloca address spaces, which has come up before as potentially useful. The datalayout address spaces are really for situations where some generic code needs to create a new value without any context, which is not the case with an intrinsic call site.

This is also going to make these intrinsics syntactically behave differently based on the datalayout. We tried something similar with program address spaces, such that the assembler currently interprets an undecorated function's address space differently based on the datalayout and I'm currently hating it. It's inconsistent and confusing and I'll likely post a patch to undo it, such that functions are always printed with an explicit addrspace(N) if it's non-0.

Assuming we do want to go with this approach, this is missing various assembler and verifier tests for all the impacted intrinsics.

I am also happy to go down the overload all intrinsics route, I just posted this as a possible approach that avoids output changes for the common "(almost) everything in AS0" targets.
If others also believe the overloading approach is better, I'll close this and post (large) patches to overload the other intrinsics that we overloaded for CHERI over the past 10 years and failed to upstream. I opened a test PR for CHERI LLVM to see how much this reduces the diff (https://github.com/CTSRD-CHERI/llvm-project/pull/663) and it shows that around 200 test files will have to be updated for overloaded llvm.va*+llvm.stacksave/stackrestore.

Regarding the address space of functions that is probably my fault, but I thought it was always printed? I believe the only thing that I allowed was to allow IR that has calls without an explicit address space to not error if the target global is a function in the program address space - but if you feed it into llvm-as | llvm-dis it should be printed again?
That being said it might also be that global function declarations are implicitly being parsed as being in the program address space, but I'd have to double-check. This is rather useful for some of the downstream CHERI tests, but it can also be fixed using sed for the handful of tests that are affected. IMO a nicer solution for this would be to allow symbolic constants in the textual IR such as addrspace(P), addrspace(G).

[nikic]

I agree with @arsenm, it would be better to overload the relevant intrinsics. I don't think the churn is particularly problematic.

[theraven]

I generally prefer the approach here. Generality is good when necessary but a source of bugs when unnecessary. For things like stack save and stack restore, the only valid address space is the address space used for the stack. Allowing overloading adds more burden to the verifier (it must check that the address space matches) for no benefit that I can perceive.

I could see a benefit if we permitted allocas to exist in different address spaces, for architectures with more than one stack, but that would be a very invasive change to the IR and one that would require rethinking a lot of assumptions.

I'm less convinced on the {prog,global} versions, because we do currently permit globals and functions to exist in specific address spaces and I can imagine a system with multiple ROM banks with different addressing making use of that.

[nikic]

FWIW, we already allow allocas to have an address space that differs from the default alloca address space. IIRC it was added for some wasm use case.

[arichardson]

In D138563#3973909, @nikic wrote:

FWIW, we already allow allocas to have an address space that differs from the default alloca address space. IIRC it was added for some wasm use case.

I did not realize that allocas were allowed in multiple address spaces. In that case the only intrinsics that benefit from this are the returnaddress ones and those could conceivable also be wanted with multiple address spaces (if there are functions in different address spaces).
Will upload reviews to overload the intrinisics instead.

llvm/include/llvm-c/Core.h

  *
  * @see llvm::Intrinsic::getDeclaration()
  */
 LLVMValueRef LLVMGetIntrinsicDeclaration(LLVMModuleRef Mod,
                                          unsigned ID,
                                          LLVMTypeRef *ParamTypes,
                                          size_t ParamCount);
 
+LLVM_ATTRIBUTE_C_DEPRECATED(LLVMTypeRef LLVMIntrinsicGetType(
+                                LLVMContextRef Ctx, unsigned ID,
+                                LLVMTypeRef *ParamTypes, size_t ParamCount),
+                            "Use LLVMIntrinsicGetType2 instead to support "
+                            "datalayout-dependent intrinsics");
 /**
  * Retrieves the type of an intrinsic.  For overloaded intrinsics, parameter
  * types must be provided to uniquely identify an overload.
  *
  * @see llvm::Intrinsic::getType()
  */
-LLVMTypeRef LLVMIntrinsicGetType(LLVMContextRef Ctx, unsigned ID,
+LLVMTypeRef LLVMIntrinsicGetType2(LLVMModuleRef Mod, unsigned ID,
                                   LLVMTypeRef *ParamTypes, size_t ParamCount);
 
 /**
  * Retrieves the name of an intrinsic.
  *
  * @see llvm::Intrinsic::getName()
  */
 const char *LLVMIntrinsicGetName(unsigned ID, size_t *NameLength);
 

llvm/include/llvm/IR/Intrinsics.h

 namespace llvm {
 
 class Type;
 class FunctionType;
 class Function;
 class LLVMContext;
 class Module;
 class AttributeList;
+class DataLayout;
 
 /// This namespace contains an enum with a value for every intrinsic/builtin
 /// function known by LLVM. The enum values are returned by
 /// Function::getIntrinsicID().
 namespace Intrinsic {
   // Abstraction for the arguments of the noalias intrinsics
   static const int NoAliasScopeDeclScopeArg = 0;
 
std::string getName(ID Id, ArrayRef<Type *> Tys, Module *M,
                       FunctionType *FT = nullptr);
 
   /// Return the LLVM name for an intrinsic. This is a special version only to
   /// be used by LLVMIntrinsicCopyOverloadedName. It only supports overloads
   /// based on named types.
   std::string getNameNoUnnamedTypes(ID Id, ArrayRef<Type *> Tys);
 
   /// Return the function type for an intrinsic.
-  FunctionType *getType(LLVMContext &Context, ID id,
-                        ArrayRef<Type*> Tys = None);
+  FunctionType *getType(LLVMContext &Context, ID id, const DataLayout &DL,
+                        ArrayRef<Type *> Tys = None);
 
   /// Returns true if the intrinsic can be overloaded.
   bool isOverloaded(ID id);
 
   /// Returns true if the intrinsic is a leaf, i.e. it does not make any calls
   /// itself.  Most intrinsics are leafs, the exceptions being the patchpoint
   /// and statepoint intrinsics. These call (or invoke) their "target" argument.
   bool isLeaf(ID id);
enum IITDescriptorKind {
       VecOfAnyPtrsToElt,
       VecElementArgument,
       Subdivide2Argument,
       Subdivide4Argument,
       VecOfBitcastsToInt,
       AMX,
       PPCQuad,
       AnyPtrToElt,
+      DataLayoutQualPointer,
     } Kind;
 
     union {
       unsigned Integer_Width;
       unsigned Float_Width;
       unsigned Pointer_AddressSpace;
       unsigned Struct_NumElements;
       unsigned Argument_Info;
       ElementCount Vector_Width;
+      char DataLayout_Pointer_AddressSpace;
     };
 
     enum ArgKind {
       AK_Any,
       AK_AnyInteger,
       AK_AnyFloat,
       AK_AnyVector,
       AK_AnyPointer,
static IITDescriptor get(IITDescriptorKind K, unsigned short Hi,
       return Result;
     }
 
     static IITDescriptor getVector(unsigned Width, bool IsScalable) {
       IITDescriptor Result = {Vector, {0}};
       Result.Vector_Width = ElementCount::get(Width, IsScalable);
       return Result;
     }
+    unsigned getPointerAddressSpace(const DataLayout &DL) const;
   };
 
   /// Return the IIT table descriptor for the specified intrinsic into an array
   /// of IITDescriptors.
   void getIntrinsicInfoTableEntries(ID id, SmallVectorImpl<IITDescriptor> &T);
 
   enum MatchIntrinsicTypesResult {
     MatchIntrinsicTypes_Match = 0,
     MatchIntrinsicTypes_NoMatchRet = 1,
     MatchIntrinsicTypes_NoMatchArg = 2,
   };
 
   /// Match the specified function type with the type constraints specified by
   /// the .td file. If the given type is an overloaded type it is pushed to the
   /// ArgTys vector.
   ///
   /// Returns false if the given type matches with the constraints, true
   /// otherwise.
   MatchIntrinsicTypesResult
   matchIntrinsicSignature(FunctionType *FTy, ArrayRef<IITDescriptor> &Infos,
+                          const DataLayout &DL,
                           SmallVectorImpl<Type *> &ArgTys);
 
   /// Verify if the intrinsic has variable arguments. This method is intended to
   /// be called after all the fixed arguments have been matched first.
   ///
   /// This method returns true on error.
   bool matchIntrinsicVarArg(bool isVarArg, ArrayRef<IITDescriptor> &Infos);
 
   /// Gets the type arguments of an intrinsic call by matching type contraints
   /// specified by the .td file. The overloaded types are pushed into the
   /// AgTys vector.
   ///
   /// Returns false if the given function is not a valid intrinsic call.
-  bool getIntrinsicSignature(Function *F, SmallVectorImpl<Type *> &ArgTys);
+  bool getIntrinsicSignature(Function *F, const DataLayout &DL,
+                             SmallVectorImpl<Type *> &ArgTys);
 
   // Checks if the intrinsic name matches with its signature and if not
   // returns the declaration with the same signature and remangled name.
   // An existing GlobalValue with the wanted name but with a wrong prototype
   // or of the wrong kind will be renamed by adding ".renamed" to the name.
   llvm::Optional<Function*> remangleIntrinsicFunction(Function *F);
 
 } // End Intrinsic namespace
 
 } // End llvm namespace
 
 #endif

llvm/include/llvm/IR/Intrinsics.td

 }
 
 class LLVMQualPointerType<LLVMType elty, int addrspace>
   : LLVMType<iPTR>{
   LLVMType ElTy = elty;
   int AddrSpace = addrspace;
 }
 
+class LLVMDataLayoutPointerType<LLVMType elty, string addrspace>
+  : LLVMType<iPTR>{
+  LLVMType ElTy = elty;
+  // Must be one of 'A', 'G', 'P' (the DataLayout-defined address spaces)
+  string AddrSpace = addrspace;
+}
+
+
 class LLVMPointerType<LLVMType elty>
   : LLVMQualPointerType<elty, 0>;
 
 class LLVMAnyPointerType<LLVMType elty>
   : LLVMType<iPTRAny>{
   LLVMType ElTy = elty;
 
   let isAny = true;

 def llvm_float_ty      : LLVMType<f32>;
 def llvm_double_ty     : LLVMType<f64>;
 def llvm_f80_ty        : LLVMType<f80>;
 def llvm_f128_ty       : LLVMType<f128>;
 def llvm_ppcf128_ty    : LLVMType<ppcf128>;
 def llvm_ptr_ty        : LLVMPointerType<llvm_i8_ty>;             // i8*
 def llvm_ptrptr_ty     : LLVMPointerType<llvm_ptr_ty>;            // i8**
 def llvm_anyptr_ty     : LLVMAnyPointerType<llvm_i8_ty>;          // (space)i8*
+def llvm_prog_ptr_ty   : LLVMDataLayoutPointerType<llvm_i8_ty, "P">; // (prog)i8*
+def llvm_alloca_ptr_ty : LLVMDataLayoutPointerType<llvm_i8_ty, "A">; // (alloca)i8*
+def llvm_global_ptr_ty : LLVMDataLayoutPointerType<llvm_i8_ty, "G">; // (globals)i8*
 def llvm_empty_ty      : LLVMType<OtherVT>;                       // { }
 def llvm_descriptor_ty : LLVMPointerType<llvm_empty_ty>;          // { }*
 def llvm_metadata_ty   : LLVMType<MetadataVT>;                    // !{...}
 def llvm_token_ty      : LLVMType<token>;                         // token
 
 def llvm_x86mmx_ty     : LLVMType<x86mmx>;
 def llvm_ptrx86mmx_ty  : LLVMPointerType<llvm_x86mmx_ty>;         // <1 x i64>*
 

llvm/lib/IR/Core.cpp

const char *LLVMIntrinsicGetName(unsigned ID, size_t *NameLength) {
   *NameLength = Str.size();
   return Str.data();
 }
 
 LLVMTypeRef LLVMIntrinsicGetType(LLVMContextRef Ctx, unsigned ID,
                                  LLVMTypeRef *ParamTypes, size_t ParamCount) {
   auto IID = llvm_map_to_intrinsic_id(ID);
   ArrayRef<Type*> Tys(unwrap(ParamTypes), ParamCount);
-  return wrap(llvm::Intrinsic::getType(*unwrap(Ctx), IID, Tys));
+  DataLayout DefaultDL("");
+  return wrap(llvm::Intrinsic::getType(*unwrap(Ctx), IID, DefaultDL, Tys));
+}
+
+LLVMTypeRef LLVMIntrinsicGetType2(LLVMModuleRef Mod, unsigned ID,
+                                  LLVMTypeRef *ParamTypes, size_t ParamCount) {
+  auto IID = llvm_map_to_intrinsic_id(ID);
+  ArrayRef<Type *> Tys(unwrap(ParamTypes), ParamCount);
+  Module *M = unwrap(Mod);
+  return wrap(
+      llvm::Intrinsic::getType(M->getContext(), IID, M->getDataLayout(), Tys));
 }
 
 const char *LLVMIntrinsicCopyOverloadedName(unsigned ID,
                                             LLVMTypeRef *ParamTypes,
                                             size_t ParamCount,
                                             size_t *NameLength) {
   auto IID = llvm_map_to_intrinsic_id(ID);
   ArrayRef<Type*> Tys(unwrap(ParamTypes), ParamCount);

llvm/lib/IR/Function.cpp

assert((!EarlyModuleCheck || M ||
          "Intrinsic overloading on pointer types need to provide a Module");
   bool HasUnnamedType = false;
   std::string Result(Intrinsic::getBaseName(Id));
   for (Type *Ty : Tys)
     Result += "." + getMangledTypeStr(Ty, HasUnnamedType);
   if (HasUnnamedType) {
     assert(M && "unnamed types need a module");
     if (!FT)
-      FT = Intrinsic::getType(M->getContext(), Id, Tys);
+      FT = Intrinsic::getType(M->getContext(), Id, M->getDataLayout(), Tys);
     else
-      assert((FT == Intrinsic::getType(M->getContext(), Id, Tys)) &&
+      assert((FT == Intrinsic::getType(M->getContext(), Id, M->getDataLayout(),
+                                       Tys)) &&
              "Provided FunctionType must match arguments");
     return M->getUniqueIntrinsicName(Result, Id, FT);
   }
   return Result;
 }
 
 std::string Intrinsic::getName(ID Id, ArrayRef<Type *> Tys, Module *M,
                                FunctionType *FT) {
enum IIT_Info {
   IIT_AMX = 51,
   IIT_PPCF128 = 52,
   IIT_V3 = 53,
   IIT_EXTERNREF = 54,
   IIT_FUNCREF = 55,
   IIT_ANYPTR_TO_ELT = 56,
   IIT_I2 = 57,
   IIT_I4 = 58,
+  IIT_DATALAYOUTPTR = 59,
 };
 
 static void DecodeIITType(unsigned &NextElt, ArrayRef<unsigned char> Infos,
                       IIT_Info LastInfo,
                       SmallVectorImpl<Intrinsic::IITDescriptor> &OutputTable) {
   using namespace Intrinsic;
 
   bool IsScalableVector = (LastInfo == IIT_SCALABLE_VEC);
case IIT_PTR:
     DecodeIITType(NextElt, Infos, Info, OutputTable);
     return;
   case IIT_ANYPTR: {  // [ANYPTR addrspace, subtype]
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Pointer,
                                              Infos[NextElt++]));
     DecodeIITType(NextElt, Infos, Info, OutputTable);
     return;
   }
+  case IIT_DATALAYOUTPTR: { // [DATALAYOUTPTR addrspace-character, subtype]
+    OutputTable.push_back(IITDescriptor::get(
+        IITDescriptor::DataLayoutQualPointer, Infos[NextElt++]));
+    DecodeIITType(NextElt, Infos, Info, OutputTable);
+    return;
+  }
   case IIT_ARG: {
     unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Argument, ArgInfo));
     return;
   }
   case IIT_EXTEND_ARG: {
     unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::ExtendArgument,
static void DecodeIITType(unsigned &NextElt, ArrayRef<unsigned char> Infos,
   }
   llvm_unreachable("unhandled");
 }
 
 #define GET_INTRINSIC_GENERATOR_GLOBAL
 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_INTRINSIC_GENERATOR_GLOBAL
 
+unsigned
+Intrinsic::IITDescriptor::getPointerAddressSpace(const DataLayout &DL) const {
+  if (Kind == DataLayoutQualPointer) {
+    switch (DataLayout_Pointer_AddressSpace) {
+    case 'A':
+      return DL.getAllocaAddrSpace();
+    case 'G':
+      return DL.getDefaultGlobalsAddressSpace();
+    case 'P':
+      return DL.getProgramAddressSpace();
+    default:
+      llvm_unreachable("Invalid pointer address space character");
+    }
+  }
+  assert(Kind == Pointer);
+  return Pointer_AddressSpace;
+}
+
 void Intrinsic::getIntrinsicInfoTableEntries(ID id,
                                              SmallVectorImpl<IITDescriptor> &T){
   // Check to see if the intrinsic's type was expressible by the table.
   unsigned TableVal = IIT_Table[id-1];
 
   // Decode the TableVal into an array of IITValues.
   SmallVector<unsigned char, 8> IITValues;
   ArrayRef<unsigned char> IITEntries;
void Intrinsic::getIntrinsicInfoTableEntries(ID id,
 
   // Okay, decode the table into the output vector of IITDescriptors.
   DecodeIITType(NextElt, IITEntries, IIT_Done, T);
   while (NextElt != IITEntries.size() && IITEntries[NextElt] != 0)
     DecodeIITType(NextElt, IITEntries, IIT_Done, T);
 }
 
 static Type *DecodeFixedType(ArrayRef<Intrinsic::IITDescriptor> &Infos,
-                             ArrayRef<Type*> Tys, LLVMContext &Context) {
+                             ArrayRef<Type *> Tys, LLVMContext &Context,
+                             const DataLayout &DL) {
   using namespace Intrinsic;
 
   IITDescriptor D = Infos.front();
   Infos = Infos.slice(1);
 
   switch (D.Kind) {
   case IITDescriptor::Void: return Type::getVoidTy(Context);
   case IITDescriptor::VarArg: return Type::getVoidTy(Context);
   case IITDescriptor::MMX: return Type::getX86_MMXTy(Context);
   case IITDescriptor::AMX: return Type::getX86_AMXTy(Context);
   case IITDescriptor::Token: return Type::getTokenTy(Context);
   case IITDescriptor::Metadata: return Type::getMetadataTy(Context);
   case IITDescriptor::Half: return Type::getHalfTy(Context);
   case IITDescriptor::BFloat: return Type::getBFloatTy(Context);
   case IITDescriptor::Float: return Type::getFloatTy(Context);
   case IITDescriptor::Double: return Type::getDoubleTy(Context);
   case IITDescriptor::Quad: return Type::getFP128Ty(Context);
   case IITDescriptor::PPCQuad: return Type::getPPC_FP128Ty(Context);
 
   case IITDescriptor::Integer:
     return IntegerType::get(Context, D.Integer_Width);
   case IITDescriptor::Vector:
-    return VectorType::get(DecodeFixedType(Infos, Tys, Context),
+    return VectorType::get(DecodeFixedType(Infos, Tys, Context, DL),
                            D.Vector_Width);
-  case IITDescriptor::Pointer:
-    return PointerType::get(DecodeFixedType(Infos, Tys, Context),
-                            D.Pointer_AddressSpace);
+  case IITDescriptor::DataLayoutQualPointer:
+  case IITDescriptor::Pointer: {
+    unsigned AS = D.getPointerAddressSpace(DL);
+    return PointerType::get(DecodeFixedType(Infos, Tys, Context, DL), AS);
+  }
   case IITDescriptor::Struct: {
     SmallVector<Type *, 8> Elts;
     for (unsigned i = 0, e = D.Struct_NumElements; i != e; ++i)
-      Elts.push_back(DecodeFixedType(Infos, Tys, Context));
+      Elts.push_back(DecodeFixedType(Infos, Tys, Context, DL));
     return StructType::get(Context, Elts);
   }
   case IITDescriptor::Argument:
     return Tys[D.getArgumentNumber()];
   case IITDescriptor::ExtendArgument: {
     Type *Ty = Tys[D.getArgumentNumber()];
     if (VectorType *VTy = dyn_cast<VectorType>(Ty))
       return VectorType::getExtendedElementVectorType(VTy);
case IITDescriptor::Subdivide4Argument: {
     assert(VTy && "Expected an argument of Vector Type");
     int SubDivs = D.Kind == IITDescriptor::Subdivide2Argument ? 1 : 2;
     return VectorType::getSubdividedVectorType(VTy, SubDivs);
   }
   case IITDescriptor::HalfVecArgument:
     return VectorType::getHalfElementsVectorType(cast<VectorType>(
                                                   Tys[D.getArgumentNumber()]));
   case IITDescriptor::SameVecWidthArgument: {
-    Type *EltTy = DecodeFixedType(Infos, Tys, Context);
+    Type *EltTy = DecodeFixedType(Infos, Tys, Context, DL);
     Type *Ty = Tys[D.getArgumentNumber()];
     if (auto *VTy = dyn_cast<VectorType>(Ty))
       return VectorType::get(EltTy, VTy->getElementCount());
     return EltTy;
   }
   case IITDescriptor::PtrToArgument: {
     Type *Ty = Tys[D.getArgumentNumber()];
     return PointerType::getUnqual(Ty);
case IITDescriptor::VecOfAnyPtrsToElt:
     return Tys[D.getOverloadArgNumber()];
   case IITDescriptor::AnyPtrToElt:
     // Return the overloaded type (which determines the pointers address space)
     return Tys[D.getOverloadArgNumber()];
   }
   llvm_unreachable("unhandled");
 }
 
-FunctionType *Intrinsic::getType(LLVMContext &Context,
-                                 ID id, ArrayRef<Type*> Tys) {
+FunctionType *Intrinsic::getType(LLVMContext &Context, ID id,
+                                 const DataLayout &DL, ArrayRef<Type *> Tys) {
   SmallVector<IITDescriptor, 8> Table;
   getIntrinsicInfoTableEntries(id, Table);
 
   ArrayRef<IITDescriptor> TableRef = Table;
-  Type *ResultTy = DecodeFixedType(TableRef, Tys, Context);
+  Type *ResultTy = DecodeFixedType(TableRef, Tys, Context, DL);
 
   SmallVector<Type*, 8> ArgTys;
   while (!TableRef.empty())
-    ArgTys.push_back(DecodeFixedType(TableRef, Tys, Context));
+    ArgTys.push_back(DecodeFixedType(TableRef, Tys, Context, DL));
 
   // DecodeFixedType returns Void for IITDescriptor::Void and IITDescriptor::VarArg
   // If we see void type as the type of the last argument, it is vararg intrinsic
   if (!ArgTys.empty() && ArgTys.back()->isVoidTy()) {
     ArgTys.pop_back();
     return FunctionType::get(ResultTy, ArgTys, true);
   }
   return FunctionType::get(ResultTy, ArgTys, false);

 /// This defines the "Intrinsic::getAttributes(ID id)" method.
 #define GET_INTRINSIC_ATTRIBUTES
 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_INTRINSIC_ATTRIBUTES
 
 Function *Intrinsic::getDeclaration(Module *M, ID id, ArrayRef<Type*> Tys) {
   // There can never be multiple globals with the same name of different types,
   // because intrinsics must be a specific type.
-  auto *FT = getType(M->getContext(), id, Tys);
+  auto *FT = getType(M->getContext(), id, M->getDataLayout(), Tys);
   return cast<Function>(
       M->getOrInsertFunction(
            Tys.empty() ? getName(id) : getName(id, Tys, M, FT), FT)
           .getCallee());
 }
 
 // This defines the "Intrinsic::getIntrinsicForClangBuiltin()" method.
 #define GET_LLVM_INTRINSIC_FOR_CLANG_BUILTIN
 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_LLVM_INTRINSIC_FOR_CLANG_BUILTIN
 
 // This defines the "Intrinsic::getIntrinsicForMSBuiltin()" method.
 #define GET_LLVM_INTRINSIC_FOR_MS_BUILTIN
 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_LLVM_INTRINSIC_FOR_MS_BUILTIN
 
 using DeferredIntrinsicMatchPair =
     std::pair<Type *, ArrayRef<Intrinsic::IITDescriptor>>;
 
-static bool matchIntrinsicType(
-    Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
-    SmallVectorImpl<Type *> &ArgTys,
+static bool
+matchIntrinsicType(Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
+                   const DataLayout &DL, SmallVectorImpl<Type *> &ArgTys,
                    SmallVectorImpl<DeferredIntrinsicMatchPair> &DeferredChecks,
                    bool IsDeferredCheck) {
   using namespace Intrinsic;
 
   // If we ran out of descriptors, there are too many arguments.
   if (Infos.empty()) return true;
 
   // Do this before slicing off the 'front' part
   auto InfosRef = Infos;
   auto DeferCheck = [&DeferredChecks, &InfosRef](Type *T) {
switch (D.Kind) {
     case IITDescriptor::Float: return !Ty->isFloatTy();
     case IITDescriptor::Double: return !Ty->isDoubleTy();
     case IITDescriptor::Quad: return !Ty->isFP128Ty();
     case IITDescriptor::PPCQuad: return !Ty->isPPC_FP128Ty();
     case IITDescriptor::Integer: return !Ty->isIntegerTy(D.Integer_Width);
     case IITDescriptor::Vector: {
       VectorType *VT = dyn_cast<VectorType>(Ty);
       return !VT || VT->getElementCount() != D.Vector_Width ||
-             matchIntrinsicType(VT->getElementType(), Infos, ArgTys,
+             matchIntrinsicType(VT->getElementType(), Infos, DL, ArgTys,
                                 DeferredChecks, IsDeferredCheck);
     }
+    case IITDescriptor::DataLayoutQualPointer:
     case IITDescriptor::Pointer: {
       PointerType *PT = dyn_cast<PointerType>(Ty);
-      if (!PT || PT->getAddressSpace() != D.Pointer_AddressSpace)
-        return true;
+      if (!PT || PT->getAddressSpace() != D.getPointerAddressSpace(DL))
       if (!PT->isOpaque()) {
         /* Manually consume a pointer to empty struct descriptor, which is
          * used for externref. We don't want to enforce that the struct is
          * anonymous in this case. (This renders externref intrinsics
          * non-unique, but this will go away with opaque pointers anyway.) */
         if (Infos.front().Kind == IITDescriptor::Struct &&
             Infos.front().Struct_NumElements == 0) {
           Infos = Infos.slice(1);
           return false;
         }
         return matchIntrinsicType(PT->getNonOpaquePointerElementType(), Infos,
-                                  ArgTys, DeferredChecks, IsDeferredCheck);
+                                  DL, ArgTys, DeferredChecks, IsDeferredCheck);
       }
       // Consume IIT descriptors relating to the pointer element type.
       // FIXME: Intrinsic type matching of nested single value types or even
       // aggregates doesn't work properly with opaque pointers but hopefully
       // doesn't happen in practice.
       while (Infos.front().Kind == IITDescriptor::Pointer ||
              Infos.front().Kind == IITDescriptor::Vector)
         Infos = Infos.slice(1);
       assert((Infos.front().Kind != IITDescriptor::Argument ||
               Infos.front().getArgumentKind() == IITDescriptor::AK_MatchType) &&
              "Unsupported polymorphic pointer type with opaque pointer");
       Infos = Infos.slice(1);
       return false;
     }
 
     case IITDescriptor::Struct: {
       StructType *ST = dyn_cast<StructType>(Ty);
       if (!ST || !ST->isLiteral() || ST->isPacked() ||
           ST->getNumElements() != D.Struct_NumElements)
         return true;
 
       for (unsigned i = 0, e = D.Struct_NumElements; i != e; ++i)
-        if (matchIntrinsicType(ST->getElementType(i), Infos, ArgTys,
+        if (matchIntrinsicType(ST->getElementType(i), Infos, DL, ArgTys,
                                DeferredChecks, IsDeferredCheck))
           return true;
       return false;
     }
 
     case IITDescriptor::Argument:
       // If this is the second occurrence of an argument,
       // verify that the later instance matches the previous instance.
case IITDescriptor::SameVecWidthArgument: {
         return true;
       Type *EltTy = Ty;
       if (ThisArgType) {
         if (ReferenceType->getElementCount() !=
             ThisArgType->getElementCount())
           return true;
         EltTy = ThisArgType->getElementType();
       }
-      return matchIntrinsicType(EltTy, Infos, ArgTys, DeferredChecks,
+      return matchIntrinsicType(EltTy, Infos, DL, ArgTys, DeferredChecks,
                                 IsDeferredCheck);
     }
     case IITDescriptor::PtrToArgument: {
       if (D.getArgumentNumber() >= ArgTys.size())
         return IsDeferredCheck || DeferCheck(Ty);
       Type * ReferenceType = ArgTys[D.getArgumentNumber()];
       PointerType *ThisArgType = dyn_cast<PointerType>(Ty);
       return (!ThisArgType ||
case IITDescriptor::VecOfBitcastsToInt: {
       if (!ThisArgVecTy || !ReferenceType)
         return true;
       return ThisArgVecTy != VectorType::getInteger(ReferenceType);
     }
   }
   llvm_unreachable("unhandled");
 }
 
-Intrinsic::MatchIntrinsicTypesResult
-Intrinsic::matchIntrinsicSignature(FunctionType *FTy,
-                                   ArrayRef<Intrinsic::IITDescriptor> &Infos,
-                                   SmallVectorImpl<Type *> &ArgTys) {
+Intrinsic::MatchIntrinsicTypesResult Intrinsic::matchIntrinsicSignature(
+    FunctionType *FTy, ArrayRef<Intrinsic::IITDescriptor> &Infos,
+    const DataLayout &DL, SmallVectorImpl<Type *> &ArgTys) {
   SmallVector<DeferredIntrinsicMatchPair, 2> DeferredChecks;
-  if (matchIntrinsicType(FTy->getReturnType(), Infos, ArgTys, DeferredChecks,
-                         false))
+  if (matchIntrinsicType(FTy->getReturnType(), Infos, DL, ArgTys,
+                         DeferredChecks, false))
       return MatchIntrinsicTypes_NoMatchRet;
 
   unsigned NumDeferredReturnChecks = DeferredChecks.size();
 
   for (auto *Ty : FTy->params())
-    if (matchIntrinsicType(Ty, Infos, ArgTys, DeferredChecks, false))
+      if (matchIntrinsicType(Ty, Infos, DL, ArgTys, DeferredChecks, false))
         return MatchIntrinsicTypes_NoMatchArg;
 
   for (unsigned I = 0, E = DeferredChecks.size(); I != E; ++I) {
     DeferredIntrinsicMatchPair &Check = DeferredChecks[I];
-    if (matchIntrinsicType(Check.first, Check.second, ArgTys, DeferredChecks,
-                           true))
+    if (matchIntrinsicType(Check.first, Check.second, DL, ArgTys,
+                           DeferredChecks, true))
         return I < NumDeferredReturnChecks ? MatchIntrinsicTypes_NoMatchRet
                                            : MatchIntrinsicTypes_NoMatchArg;
   }
 
   return MatchIntrinsicTypes_Match;
 }
 
 bool
 Intrinsic::matchIntrinsicVarArg(bool isVarArg,
                                 ArrayRef<Intrinsic::IITDescriptor> &Infos) {
Intrinsic::matchIntrinsicVarArg(bool isVarArg,
   IITDescriptor D = Infos.front();
   Infos = Infos.slice(1);
   if (D.Kind == IITDescriptor::VarArg)
     return !isVarArg;
 
   return true;
 }
 
-bool Intrinsic::getIntrinsicSignature(Function *F,
+bool Intrinsic::getIntrinsicSignature(Function *F, const DataLayout &DL,
                                       SmallVectorImpl<Type *> &ArgTys) {
   Intrinsic::ID ID = F->getIntrinsicID();
   if (!ID)
     return false;
 
   SmallVector<Intrinsic::IITDescriptor, 8> Table;
   getIntrinsicInfoTableEntries(ID, Table);
   ArrayRef<Intrinsic::IITDescriptor> TableRef = Table;
 
-  if (Intrinsic::matchIntrinsicSignature(F->getFunctionType(), TableRef,
+  if (Intrinsic::matchIntrinsicSignature(F->getFunctionType(), TableRef, DL,
                                          ArgTys) !=
       Intrinsic::MatchIntrinsicTypesResult::MatchIntrinsicTypes_Match) {
     return false;
   }
   if (Intrinsic::matchIntrinsicVarArg(F->getFunctionType()->isVarArg(),
                                       TableRef))
     return false;
   return true;
 }
 
 Optional<Function *> Intrinsic::remangleIntrinsicFunction(Function *F) {
   SmallVector<Type *, 4> ArgTys;
-  if (!getIntrinsicSignature(F, ArgTys))
+  if (!getIntrinsicSignature(F, F->getParent()->getDataLayout(), ArgTys))
     return None;
 
   Intrinsic::ID ID = F->getIntrinsicID();
   StringRef Name = F->getName();
   std::string WantedName =
       Intrinsic::getName(ID, ArgTys, F->getParent(), F->getFunctionType());
   if (Name == WantedName)
     return None;

llvm/lib/IR/IRBuilder.cpp

CallInst *IRBuilderBase::CreateIntrinsic(Type *RetTy, Intrinsic::ID ID,
 
   SmallVector<Type *> ArgTys;
   ArgTys.reserve(Args.size());
   for (auto &I : Args)
     ArgTys.push_back(I->getType());
   FunctionType *FTy = FunctionType::get(RetTy, ArgTys, false);
   SmallVector<Type *> OverloadTys;
   Intrinsic::MatchIntrinsicTypesResult Res =
-      matchIntrinsicSignature(FTy, TableRef, OverloadTys);
+      matchIntrinsicSignature(FTy, TableRef, M->getDataLayout(), OverloadTys);
   (void)Res;
   assert(Res == Intrinsic::MatchIntrinsicTypes_Match && TableRef.empty() &&
          "Wrong types for intrinsic!");
   // TODO: Handle varargs intrinsics.
 
   Function *Fn = Intrinsic::getDeclaration(M, ID, OverloadTys);
   return createCallHelper(Fn, Args, Name, FMFSource);
 }

llvm/lib/IR/Verifier.cpp

void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
 
   SmallVector<Intrinsic::IITDescriptor, 8> Table;
   getIntrinsicInfoTableEntries(ID, Table);
   ArrayRef<Intrinsic::IITDescriptor> TableRef = Table;
 
   // Walk the descriptors to extract overloaded types.
   SmallVector<Type *, 4> ArgTys;
   Intrinsic::MatchIntrinsicTypesResult Res =
-      Intrinsic::matchIntrinsicSignature(IFTy, TableRef, ArgTys);
+      Intrinsic::matchIntrinsicSignature(IFTy, TableRef, DL, ArgTys);
   Check(Res != Intrinsic::MatchIntrinsicTypes_NoMatchRet,
         "Intrinsic has incorrect return type!", IF);
   Check(Res != Intrinsic::MatchIntrinsicTypes_NoMatchArg,
         "Intrinsic has incorrect argument type!", IF);
 
   // Verify if the intrinsic call matches the vararg property.
   if (IsVarArg)
     Check(!Intrinsic::matchIntrinsicVarArg(IsVarArg, TableRef),

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

 /// modified arguments (based on OldIntr) and replaces InstToReplace with
 /// this newly created intrinsic call.
 static Optional<Instruction *> modifyIntrinsicCall(
     IntrinsicInst &OldIntr, Instruction &InstToReplace, unsigned NewIntr,
     InstCombiner &IC,
     std::function<void(SmallVectorImpl<Value *> &, SmallVectorImpl<Type *> &)>
         Func) {
   SmallVector<Type *, 4> ArgTys;
-  if (!Intrinsic::getIntrinsicSignature(OldIntr.getCalledFunction(), ArgTys))
+  if (!Intrinsic::getIntrinsicSignature(OldIntr.getCalledFunction(),
+                                        IC.getDataLayout(), ArgTys))
     return None;
 
   SmallVector<Value *, 8> Args(OldIntr.args());
 
   // Modify arguments and types
   Func(Args, ArgTys);
 
   Function *I = Intrinsic::getDeclaration(OldIntr.getModule(), NewIntr, ArgTys);
if (NewNumElts >= VWidth && DemandedElts.isMask()) {
     if (DMaskIdx >= 0)
       II.setArgOperand(DMaskIdx, Args[DMaskIdx]);
     return nullptr;
   }
 
   // Validate function argument and return types, extracting overloaded types
   // along the way.
   SmallVector<Type *, 6> OverloadTys;
-  if (!Intrinsic::getIntrinsicSignature(II.getCalledFunction(), OverloadTys))
+  if (!Intrinsic::getIntrinsicSignature(II.getCalledFunction(),
+                                        IC.getDataLayout(), OverloadTys))
     return nullptr;
 
   Module *M = II.getParent()->getParent()->getParent();
   Type *EltTy = IIVTy->getElementType();
   Type *NewTy =
       (NewNumElts == 1) ? EltTy : FixedVectorType::get(EltTy, NewNumElts);
 
   OverloadTys[0] = NewTy;

llvm/utils/TableGen/IntrinsicEmitter.cpp

enum IIT_Info {
   IIT_AMX = 51,
   IIT_PPCF128 = 52,
   IIT_V3 = 53,
   IIT_EXTERNREF = 54,
   IIT_FUNCREF = 55,
   IIT_ANYPTR_TO_ELT = 56,
   IIT_I2 = 57,
   IIT_I4 = 58,
+  IIT_DATALAYOUTPTR = 59,
 };
 
 static void EncodeFixedValueType(MVT::SimpleValueType VT,
                                  std::vector<unsigned char> &Sig) {
   // clang-format off
   if (MVT(VT).isInteger()) {
     unsigned BitWidth = MVT(VT).getFixedSizeInBits();
     switch (BitWidth) {
assert(NextArgCode < ArgCodes.size() && ArgCodes[NextArgCode] == Tmp &&
            "Invalid or no ArgCode associated with overloaded VT!");
     unsigned ArgNo = NextArgCode++;
 
     // Encode what sort of argument it must be in the low 3 bits of the ArgNo.
     return Sig.push_back((ArgNo << 3) | Tmp);
   }
 
   case MVT::iPTR: {
+    if (R->isSubClassOf("LLVMDataLayoutPointerType")) {
+      StringRef AddrSpaceChar = R->getValueAsString("AddrSpace");
+      if (AddrSpaceChar.size() != 1 ||
+          !StringRef("AGP").contains(AddrSpaceChar))
+        PrintFatalError(R,
+                        "Invalid address space value: '" + AddrSpaceChar + "'");
+      Sig.push_back(IIT_DATALAYOUTPTR);
+      Sig.push_back(AddrSpaceChar[0]);
+    } else {
       unsigned AddrSpace = 0;
       if (R->isSubClassOf("LLVMQualPointerType")) {
         AddrSpace = R->getValueAsInt("AddrSpace");
-      assert(AddrSpace < 256 && "Address space exceeds 255");
+        if (!isUInt<8>(AddrSpace))
+          PrintFatalError(R,
+                          "Address space " + Twine(AddrSpace) + " exceeds 255");
       }
       if (AddrSpace) {
         Sig.push_back(IIT_ANYPTR);
         Sig.push_back(AddrSpace);
       } else {
         Sig.push_back(IIT_PTR);
       }
+    }
     return EncodeFixedType(R->getValueAsDef("ElTy"), ArgCodes, NextArgCode, Sig,
                            Mapping);
   }
   }
 
   if (MVT(VT).isVector()) {
     MVT VVT = VT;
     if (VVT.isScalableVector())

mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp

getOverloadedDeclaration(CallIntrinsicOp &op, llvm::Intrinsic::ID id,
   // ATM we do not support variadic intrinsics.
   llvm::FunctionType *ft = llvm::FunctionType::get(resTy, allArgTys, false);
 
   SmallVector<llvm::Intrinsic::IITDescriptor, 8> table;
   getIntrinsicInfoTableEntries(id, table);
   ArrayRef<llvm::Intrinsic::IITDescriptor> tableRef = table;
 
   SmallVector<llvm::Type *, 8> overloadedArgTys;
-  if (llvm::Intrinsic::matchIntrinsicSignature(ft, tableRef,
-                                               overloadedArgTys) !=
+  if (llvm::Intrinsic::matchIntrinsicSignature(
+          ft, tableRef, module->getDataLayout(), overloadedArgTys) !=
       llvm::Intrinsic::MatchIntrinsicTypesResult::MatchIntrinsicTypes_Match) {
     return op.emitOpError("intrinsic type is not a match");
   }
 
   ArrayRef<llvm::Type *> overloadedArgTysRef = overloadedArgTys;
   return llvm::Intrinsic::getDeclaration(module, id, overloadedArgTysRef);
 }