[IntrinsicEmitter] Extend argument overloading with forward references.

[sdesmalen]

Summary

Extend the mechanism to overload intrinsic arguments by using either
backward or forward references to the overloadable arguments.

In for example:

def int_something : Intrinsic<[LLVMPointerToElt<0>],
                              [llvm_anyvector_ty], []>;

LLVMPointerToElt<0> is a forward reference to the overloadable operand
of type 'llvm_anyvector_ty' and would allow intrinsics such as:

declare i32* @llvm.something.v4i32(<4 x i32>);
declare i64* @llvm.something.v2i64(<2 x i64>);

where the result pointer type is deduced from the element type of the
first argument.

If the returned pointer is not a pointer to the element type, LLVM will
give an error:

Intrinsic has incorrect return type!
i64* (<4 x i32>)* @llvm.something.v4i32

[arsenm]

include/llvm/IR/Intrinsics.td

1185–1202

These should just go to a TableGen test file instead of defining a real intrinsic

lib/IR/Function.cpp

1195

IsDeferredCheck || DeferCheck(Ty)

[sdesmalen]

include/llvm/IR/Intrinsics.td

1185–1202

I agree these shouldn't be here, but I don't really know any other way to test the changes to matchIntrinsicType in lib/IR/Function.cpp. If we move these these to a separate TableGen test, it will only test the changes to TableGen.

[arsenm]

include/llvm/IR/Intrinsics.td

1185–1202

Can you just add some cases with the real intrinsics after the use patch is committed?

[sdesmalen]

include/llvm/IR/Intrinsics.td

1185–1202

Sure, no problem!

From these tests int_experimental_dummy_forward_struct_ret can be tested by changing AdvSIMD_3Vec_Load_Intrinsic in IntrinsicsAArch64.td:

class AdvSIMD_3Vec_Load_Intrinsic
  : Intrinsic<[llvm_anyvector_ty, LLVMMatchType<0>, LLVMMatchType<0>],
              [LLVMAnyPointerType<LLVMMatchType<0>>],
              [IntrReadMem, IntrArgMemOnly]>;

to

class AdvSIMD_3Vec_Load_Intrinsic
  : Intrinsic<[LLVMMatchType<0>, LLVMMatchType<0>, llvm_anyvector_ty],
              [LLVMAnyPointerType<LLVMMatchType<0>>],
              [IntrReadMem, IntrArgMemOnly]>;

The _forward_ test that uses LLVMVectorOfAnyPointersToElt is a bit trickier. The masked.gather is the only intrinsic that uses this LLVMMatchTy, but if we change:

def int_masked_gather: Intrinsic<[llvm_anyvector_ty],
                                 [LLVMVectorOfAnyPointersToElt<0>, llvm_i32_ty,
                                  LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                  LLVMMatchType<0>],
                                 [IntrReadMem, ImmArg<1>]>;

into:

def int_masked_gather: Intrinsic<[LLVMMatchType<0>],
                                 [LLVMVectorOfAnyPointersToElt<0>, llvm_i32_ty,
                                  LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                  llvm_anyvector_ty],
                                  [IntrReadMem, ImmArg<1>]>;

Then:

declare <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*>, i32 immarg, <4 x i1>, <4 x float>)

Would change into:

declare <4 x float> @llvm.masked.gather.v4p0f32.v4f32(<4 x float*>, i32 immarg, <4 x i1>, <4 x float>)

And IRBuilder would need to be updated to change the type-list of the function, with the pointer type <4 x float*> before the vector type <4 x float>. I guess this is something we don't want to change? (probably also because we don't want to overload based on the passthu value).

For SVE ACLE intrinsics we rely more heavily on these forward references and we'll be adding more intrinsics that test this behaviour in the near future. (note that I've tested this implementation downstream with our SVE ACLE intrinsics).

In any case I'll remove the "dummy" tests added in this revision, and create a new patch that changes AdvSIMD_3Vec_Load_Intrinsic along with a corresponding test similar to the one in test/Verifier/intrinsic-arg-overloading-struct-ret.ll.

[sdesmalen]

• Changed: return IsDeferredCheck ? true : DeferCheck(Ty); => return IsDeferredCheck || DeferCheck(Ty);
• Use DeferredChecks.size() to determine what MatchIntrinsicTypesResult to return, instead of incorrectly relying on DeferredChecks.end() iterator.
• Removed 'dummy' intrinsics. I will post a separate patch that tests the behaviour tested with llvm.experimental.dummy.forward.struct.ret on existing intrinsics.

[arsenm]

LGTM with nit

utils/TableGen/IntrinsicEmitter.cpp

386

Extra space before (

[sdesmalen]

In D62995#1539852, @arsenm wrote:

LGTM with nit

Thanks for the review!

include/llvm/IR/Intrinsics.h

union {
       unsigned Argument_Info;
     };
 
     enum ArgKind {
       AK_Any,
       AK_AnyInteger,
       AK_AnyFloat,
       AK_AnyVector,
-      AK_AnyPointer
+      AK_AnyPointer,
+      AK_MatchType = 7
     };
 
     unsigned getArgumentNumber() const {
       assert(Kind == Argument || Kind == ExtendArgument ||
              Kind == TruncArgument || Kind == HalfVecArgument ||
              Kind == SameVecWidthArgument || Kind == PtrToArgument ||
              Kind == PtrToElt);
       return Argument_Info >> 3;
static IITDescriptor get(IITDescriptorKind K, unsigned short Hi,
       return Result;
     }
   };
 
   /// Return the IIT table descriptor for the specified intrinsic into an array
   /// of IITDescriptors.
   void getIntrinsicInfoTableEntries(ID id, SmallVectorImpl<IITDescriptor> &T);
 
-  /// Match the specified type (which comes from an intrinsic argument or return
-  /// value) with the type constraints specified by the .td file. If the given
-  /// type is an overloaded type it is pushed to the ArgTys vector.
+  enum MatchIntrinsicTypesResult {
+    MatchIntrinsicTypes_Match = 0,
+    MatchIntrinsicTypes_NoMatchRet = 1,
+    MatchIntrinsicTypes_NoMatchArg = 2,
+  };
+
+  /// Match the specified function type with the type constraints specified by
+  /// the .td file. If the given type is an overloaded type it is pushed to the
+  /// ArgTys vector.
   ///
   /// Returns false if the given type matches with the constraints, true
   /// otherwise.
-  bool matchIntrinsicType(Type *Ty, ArrayRef<IITDescriptor> &Infos,
-                          SmallVectorImpl<Type*> &ArgTys);
+  MatchIntrinsicTypesResult
+  matchIntrinsicSignature(FunctionType *FTy, ArrayRef<IITDescriptor> &Infos,
+                          SmallVectorImpl<Type *> &ArgTys);
 
   /// Verify if the intrinsic has variable arguments. This method is intended to
   /// be called after all the fixed arguments have been matched first.
   ///
   /// This method returns true on error.
   bool matchIntrinsicVarArg(bool isVarArg, ArrayRef<IITDescriptor> &Infos);
 
   // Checks if the intrinsic name matches with its signature and if not
   // returns the declaration with the same signature and remangled name.
   llvm::Optional<Function*> remangleIntrinsicFunction(Function *F);
 
 } // End Intrinsic namespace
 
 } // End llvm namespace
 
 #endif

include/llvm/IR/Intrinsics.td

def int_experimental_vector_reduce_umin : Intrinsic<[llvm_anyint_ty],
                                                     [IntrNoMem]>;
 def int_experimental_vector_reduce_fmax : Intrinsic<[llvm_anyfloat_ty],
                                                     [llvm_anyvector_ty],
                                                     [IntrNoMem]>;
 def int_experimental_vector_reduce_fmin : Intrinsic<[llvm_anyfloat_ty],
                                                     [llvm_anyvector_ty],
                                                     [IntrNoMem]>;
 
+def int_experimental_dummy_backward : Intrinsic<[llvm_anyvector_ty],
+                                                [llvm_anyvector_ty,
+                                                 LLVMVectorOfAnyPointersToElt<0>,
+                                                 LLVMVectorOfAnyPointersToElt<1>,
+                                                 LLVMVectorElementType<0>], []>;
+
+def int_experimental_dummy_forward : Intrinsic<[LLVMVectorOfAnyPointersToElt<0>],
+                                               [LLVMVectorOfAnyPointersToElt<1>,
+                                                llvm_anyvector_ty,
+                                                llvm_anyvector_ty,
+                                                LLVMVectorElementType<0>], []>;
+
+def int_experimental_dummy_forward_struct_ret
+                                   : Intrinsic<[LLVMMatchType<0>,
+                                                LLVMMatchType<0>,
+                                                llvm_anyint_ty],
+                                               [llvm_anyint_ty], []>;
+
 //===----- Intrinsics that are used to provide predicate information -----===//
 
 def int_ssa_copy : Intrinsic<[llvm_any_ty], [LLVMMatchType<0>],
                              [IntrNoMem, Returned<0>]>;
 //===----------------------------------------------------------------------===//
 // Target-specific intrinsics
 //===----------------------------------------------------------------------===//
 

lib/IR/Function.cpp

 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_LLVM_INTRINSIC_FOR_GCC_BUILTIN
 
 // This defines the "Intrinsic::getIntrinsicForMSBuiltin()" method.
 #define GET_LLVM_INTRINSIC_FOR_MS_BUILTIN
 #include "llvm/IR/IntrinsicImpl.inc"
 #undef GET_LLVM_INTRINSIC_FOR_MS_BUILTIN
 
-bool Intrinsic::matchIntrinsicType(Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
-                                   SmallVectorImpl<Type*> &ArgTys) {
+using DeferredIntrinsicMatchPair =
+    std::pair<Type *, ArrayRef<Intrinsic::IITDescriptor>>;
+
+static bool matchIntrinsicType(
+    Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
+    SmallVectorImpl<Type *> &ArgTys,
+    SmallVectorImpl<DeferredIntrinsicMatchPair> &DeferredChecks,
+    bool IsDeferredCheck) {
   using namespace Intrinsic;
 
   // If we ran out of descriptors, there are too many arguments.
   if (Infos.empty()) return true;
+
+  // Do this before slicing off the 'front' part
+  auto InfosRef = Infos;
+  auto DeferCheck = [&DeferredChecks, &InfosRef](Type *T) {
+    DeferredChecks.emplace_back(T, InfosRef);
+    return false;
+  };
+
   IITDescriptor D = Infos.front();
   Infos = Infos.slice(1);
 
   switch (D.Kind) {
     case IITDescriptor::Void: return !Ty->isVoidTy();
     case IITDescriptor::VarArg: return true;
     case IITDescriptor::MMX:  return !Ty->isX86_MMXTy();
     case IITDescriptor::Token: return !Ty->isTokenTy();
     case IITDescriptor::Metadata: return !Ty->isMetadataTy();
     case IITDescriptor::Half: return !Ty->isHalfTy();
     case IITDescriptor::Float: return !Ty->isFloatTy();
     case IITDescriptor::Double: return !Ty->isDoubleTy();
     case IITDescriptor::Quad: return !Ty->isFP128Ty();
     case IITDescriptor::Integer: return !Ty->isIntegerTy(D.Integer_Width);
     case IITDescriptor::Vector: {
       VectorType *VT = dyn_cast<VectorType>(Ty);
       return !VT || VT->getNumElements() != D.Vector_Width ||
-             matchIntrinsicType(VT->getElementType(), Infos, ArgTys);
+             matchIntrinsicType(VT->getElementType(), Infos, ArgTys,
+                                DeferredChecks, IsDeferredCheck);
     }
     case IITDescriptor::Pointer: {
       PointerType *PT = dyn_cast<PointerType>(Ty);
       return !PT || PT->getAddressSpace() != D.Pointer_AddressSpace ||
-             matchIntrinsicType(PT->getElementType(), Infos, ArgTys);
+             matchIntrinsicType(PT->getElementType(), Infos, ArgTys,
+                                DeferredChecks, IsDeferredCheck);
     }
 
     case IITDescriptor::Struct: {
       StructType *ST = dyn_cast<StructType>(Ty);
       if (!ST || ST->getNumElements() != D.Struct_NumElements)
         return true;
 
       for (unsigned i = 0, e = D.Struct_NumElements; i != e; ++i)
-        if (matchIntrinsicType(ST->getElementType(i), Infos, ArgTys))
+        if (matchIntrinsicType(ST->getElementType(i), Infos, ArgTys,
+                               DeferredChecks, IsDeferredCheck))
           return true;
       return false;
     }
 
     case IITDescriptor::Argument:
-      // Two cases here - If this is the second occurrence of an argument, verify
-      // that the later instance matches the previous instance.
+      // If this is the second occurrence of an argument,
+      // verify that the later instance matches the previous instance.
       if (D.getArgumentNumber() < ArgTys.size())
         return Ty != ArgTys[D.getArgumentNumber()];
 
-      // Otherwise, if this is the first instance of an argument, record it and
-      // verify the "Any" kind.
-      assert(D.getArgumentNumber() == ArgTys.size() && "Table consistency error");
+      if (D.getArgumentNumber() > ArgTys.size() ||
+          D.getArgumentKind() == IITDescriptor::AK_MatchType)
+        return IsDeferredCheck ? true : DeferCheck(Ty);
+
+      assert(D.getArgumentNumber() == ArgTys.size() && !IsDeferredCheck &&
+             "Table consistency error");
       ArgTys.push_back(Ty);
 
       switch (D.getArgumentKind()) {
         case IITDescriptor::AK_Any:        return false; // Success
         case IITDescriptor::AK_AnyInteger: return !Ty->isIntOrIntVectorTy();
         case IITDescriptor::AK_AnyFloat:   return !Ty->isFPOrFPVectorTy();
         case IITDescriptor::AK_AnyVector:  return !isa<VectorType>(Ty);
         case IITDescriptor::AK_AnyPointer: return !isa<PointerType>(Ty);
+        default:                           break;
       }
       llvm_unreachable("all argument kinds not covered");
 
     case IITDescriptor::ExtendArgument: {
-      // This may only be used when referring to a previous vector argument.
+      // If this is a forward reference, defer the check for later.
       if (D.getArgumentNumber() >= ArgTys.size())
-        return true;
+        return IsDeferredCheck ? true : DeferCheck(Ty);
 
       Type *NewTy = ArgTys[D.getArgumentNumber()];
       if (VectorType *VTy = dyn_cast<VectorType>(NewTy))
         NewTy = VectorType::getExtendedElementVectorType(VTy);
       else if (IntegerType *ITy = dyn_cast<IntegerType>(NewTy))
         NewTy = IntegerType::get(ITy->getContext(), 2 * ITy->getBitWidth());
       else
         return true;
 
       return Ty != NewTy;
     }
     case IITDescriptor::TruncArgument: {
-      // This may only be used when referring to a previous vector argument.
+      // If this is a forward reference, defer the check for later.
       if (D.getArgumentNumber() >= ArgTys.size())
-        return true;
+        return IsDeferredCheck ? true : DeferCheck(Ty);
 
       Type *NewTy = ArgTys[D.getArgumentNumber()];
       if (VectorType *VTy = dyn_cast<VectorType>(NewTy))
         NewTy = VectorType::getTruncatedElementVectorType(VTy);
       else if (IntegerType *ITy = dyn_cast<IntegerType>(NewTy))
         NewTy = IntegerType::get(ITy->getContext(), ITy->getBitWidth() / 2);
       else
         return true;
 
       return Ty != NewTy;
     }
     case IITDescriptor::HalfVecArgument:
-      // This may only be used when referring to a previous vector argument.
+      // If this is a forward reference, defer the check for later.
       return D.getArgumentNumber() >= ArgTys.size() ||
              !isa<VectorType>(ArgTys[D.getArgumentNumber()]) ||
              VectorType::getHalfElementsVectorType(
                      cast<VectorType>(ArgTys[D.getArgumentNumber()])) != Ty;
     case IITDescriptor::SameVecWidthArgument: {
-      if (D.getArgumentNumber() >= ArgTys.size())
-        return true;
+      if (D.getArgumentNumber() >= ArgTys.size()) {
+        // Defer check and subsequent check for the vector element type.
+        Infos = Infos.slice(1);
+        return IsDeferredCheck ? true : DeferCheck(Ty);
+      }
       auto *ReferenceType = dyn_cast<VectorType>(ArgTys[D.getArgumentNumber()]);
       auto *ThisArgType = dyn_cast<VectorType>(Ty);
       // Both must be vectors of the same number of elements or neither.
       if ((ReferenceType != nullptr) != (ThisArgType != nullptr))
         return true;
       Type *EltTy = Ty;
       if (ThisArgType) {
         if (ReferenceType->getVectorNumElements() !=
             ThisArgType->getVectorNumElements())
           return true;
         EltTy = ThisArgType->getVectorElementType();
       }
-      return matchIntrinsicType(EltTy, Infos, ArgTys);
+      return matchIntrinsicType(EltTy, Infos, ArgTys, DeferredChecks,
+                                IsDeferredCheck);
     }
     case IITDescriptor::PtrToArgument: {
       if (D.getArgumentNumber() >= ArgTys.size())
-        return true;
+        return IsDeferredCheck ? true : DeferCheck(Ty);
       Type * ReferenceType = ArgTys[D.getArgumentNumber()];
       PointerType *ThisArgType = dyn_cast<PointerType>(Ty);
       return (!ThisArgType || ThisArgType->getElementType() != ReferenceType);
     }
     case IITDescriptor::PtrToElt: {
       if (D.getArgumentNumber() >= ArgTys.size())
-        return true;
+        return IsDeferredCheck ? true : DeferCheck(Ty);
       VectorType * ReferenceType =
         dyn_cast<VectorType> (ArgTys[D.getArgumentNumber()]);
       PointerType *ThisArgType = dyn_cast<PointerType>(Ty);
 
       return (!ThisArgType || !ReferenceType ||
               ThisArgType->getElementType() != ReferenceType->getElementType());
     }
     case IITDescriptor::VecOfAnyPtrsToElt: {
       unsigned RefArgNumber = D.getRefArgNumber();
-
-      // This may only be used when referring to a previous argument.
-      if (RefArgNumber >= ArgTys.size())
+      if (RefArgNumber >= ArgTys.size()) {
+        if (IsDeferredCheck)
           return true;
+        // If forward referencing, already add the pointer-vector type and
+        // defer the checks for later.
+        ArgTys.push_back(Ty);
+        return DeferCheck(Ty);
+      }
 
-      // Record the overloaded type
+      if (!IsDeferredCheck){
         assert(D.getOverloadArgNumber() == ArgTys.size() &&
                "Table consistency error");
         ArgTys.push_back(Ty);
+      }
 
       // Verify the overloaded type "matches" the Ref type.
       // i.e. Ty is a vector with the same width as Ref.
       // Composed of pointers to the same element type as Ref.
       VectorType *ReferenceType = dyn_cast<VectorType>(ArgTys[RefArgNumber]);
       VectorType *ThisArgVecTy = dyn_cast<VectorType>(Ty);
       if (!ThisArgVecTy || !ReferenceType ||
           (ReferenceType->getVectorNumElements() !=
            ThisArgVecTy->getVectorNumElements()))
         return true;
       PointerType *ThisArgEltTy =
               dyn_cast<PointerType>(ThisArgVecTy->getVectorElementType());
       if (!ThisArgEltTy)
         return true;
       return ThisArgEltTy->getElementType() !=
              ReferenceType->getVectorElementType();
     }
   }
   llvm_unreachable("unhandled");
 }
 
+Intrinsic::MatchIntrinsicTypesResult
+Intrinsic::matchIntrinsicSignature(FunctionType *FTy,
+                                   ArrayRef<Intrinsic::IITDescriptor> &Infos,
+                                   SmallVectorImpl<Type *> &ArgTys) {
+  SmallVector<DeferredIntrinsicMatchPair, 2> DeferredChecks;
+  if (matchIntrinsicType(FTy->getReturnType(), Infos, ArgTys, DeferredChecks,
+                         false))
+    return MatchIntrinsicTypes_NoMatchRet;
+
+  auto DeferredReturnEnd = DeferredChecks.end();
+
+  for (auto Ty : FTy->params())
+    if (matchIntrinsicType(Ty, Infos, ArgTys, DeferredChecks, false))
+      return MatchIntrinsicTypes_NoMatchArg;
+
+  for (auto I = DeferredChecks.begin(), E = DeferredChecks.end(); I != E; ++I) {
+    DeferredIntrinsicMatchPair &Check = *I;
+    if (matchIntrinsicType(Check.first, Check.second, ArgTys, DeferredChecks,
+                           true))
+      return I < DeferredReturnEnd ? MatchIntrinsicTypes_NoMatchRet
+                                   : MatchIntrinsicTypes_NoMatchArg;
+  }
+
+  return MatchIntrinsicTypes_Match;
+}
+
 bool
 Intrinsic::matchIntrinsicVarArg(bool isVarArg,
                                 ArrayRef<Intrinsic::IITDescriptor> &Infos) {
   // If there are no descriptors left, then it can't be a vararg.
   if (Infos.empty())
     return isVarArg;
 
   // There should be only one descriptor remaining at this point.
Optional<Function*> Intrinsic::remangleIntrinsicFunction(Function *F) {
   FunctionType *FTy = F->getFunctionType();
   // Accumulate an array of overloaded types for the given intrinsic
   SmallVector<Type *, 4> ArgTys;
   {
     SmallVector<Intrinsic::IITDescriptor, 8> Table;
     getIntrinsicInfoTableEntries(ID, Table);
     ArrayRef<Intrinsic::IITDescriptor> TableRef = Table;
 
-    // If we encounter any problems matching the signature with the descriptor
-    // just give up remangling. It's up to verifier to report the discrepancy.
-    if (Intrinsic::matchIntrinsicType(FTy->getReturnType(), TableRef, ArgTys))
-      return None;
-    for (auto Ty : FTy->params())
-      if (Intrinsic::matchIntrinsicType(Ty, TableRef, ArgTys))
-        return None;
+    if (Intrinsic::matchIntrinsicSignature(FTy, TableRef, ArgTys))
     if (Intrinsic::matchIntrinsicVarArg(FTy->isVarArg(), TableRef))
       return None;
   }
 
   StringRef Name = F->getName();
   if (Name == Intrinsic::getName(ID, ArgTys))
     return None;
 

lib/IR/Verifier.cpp

void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
   // describe.
   FunctionType *IFTy = IF->getFunctionType();
   bool IsVarArg = IFTy->isVarArg();
 
   SmallVector<Intrinsic::IITDescriptor, 8> Table;
   getIntrinsicInfoTableEntries(ID, Table);
   ArrayRef<Intrinsic::IITDescriptor> TableRef = Table;
 
+  // Walk the descriptors to extract overloaded types.
   SmallVector<Type *, 4> ArgTys;
-  Assert(!Intrinsic::matchIntrinsicType(IFTy->getReturnType(),
-                                        TableRef, ArgTys),
+  Intrinsic::MatchIntrinsicTypesResult Res =
+      Intrinsic::matchIntrinsicSignature(IFTy, TableRef, ArgTys);
+  Assert(Res != Intrinsic::MatchIntrinsicTypes_NoMatchRet,
          "Intrinsic has incorrect return type!", IF);
-  for (unsigned i = 0, e = IFTy->getNumParams(); i != e; ++i)
-    Assert(!Intrinsic::matchIntrinsicType(IFTy->getParamType(i),
-                                          TableRef, ArgTys),
-           "Intrinsic has incorrect argument type!", IF);
+  Assert(Res != Intrinsic::MatchIntrinsicTypes_NoMatchArg,
 
   // Verify if the intrinsic call matches the vararg property.
   if (IsVarArg)
     Assert(!Intrinsic::matchIntrinsicVarArg(IsVarArg, TableRef),
            "Intrinsic was not defined with variable arguments!", IF);
   else
     Assert(!Intrinsic::matchIntrinsicVarArg(IsVarArg, TableRef),
            "Callsite was not defined with variable arguments!", IF);

lib/Transforms/InstCombine/InstCombineSimplifyDemanded.cpp

Value *InstCombiner::simplifyAMDGCNMemoryIntrinsicDemanded(IntrinsicInst *II,
   }
 
   // Determine the overload types of the original intrinsic.
   auto IID = II->getIntrinsicID();
   SmallVector<Intrinsic::IITDescriptor, 16> Table;
   getIntrinsicInfoTableEntries(IID, Table);
   ArrayRef<Intrinsic::IITDescriptor> TableRef = Table;
 
+  // Validate function argument and return types, extracting overloaded types
+  // along the way.
   FunctionType *FTy = II->getCalledFunction()->getFunctionType();
   SmallVector<Type *, 6> OverloadTys;
-  Intrinsic::matchIntrinsicType(FTy->getReturnType(), TableRef, OverloadTys);
-  for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i)
-    Intrinsic::matchIntrinsicType(FTy->getParamType(i), TableRef, OverloadTys);
+  Intrinsic::matchIntrinsicSignature(FTy, TableRef, OverloadTys);
 
-  // Get the new return type overload of the intrinsic.
   Module *M = II->getParent()->getParent()->getParent();
   Type *EltTy = II->getType()->getVectorElementType();
   Type *NewTy = (NewNumElts == 1) ? EltTy : VectorType::get(EltTy, NewNumElts);
 
   OverloadTys[0] = NewTy;
   Function *NewIntrin = Intrinsic::getDeclaration(M, IID, OverloadTys);
 
   SmallVector<Value *, 16> Args;

test/Verifier/intrinsic-arg-overloading-struct-ret.ll

+; RUN: not opt -verify -S < %s 2>&1 | FileCheck %s
+
+; CHECK: Intrinsic has incorrect return type
+define { i32, i64, i32 } @test(i8 %a) {
+  %res = call { i32, i64, i32 } @llvm.experimental.dummy.forward.struct.ret.i32.i8(i8 %a)
+  ret { i32, i64, i32 } %res
+}
+declare { i32, i64, i32 } @llvm.experimental.dummy.forward.struct.ret.i32.i8(i8);

test/Verifier/intrinsic-arg-overloading.ll

+; RUN: opt -verify -S < %s | FileCheck %s
+
+; CHECK: @test_vector_of_pointers_backward
+; CHECK: call <4 x float> @llvm.experimental.dummy.backward.v4f32.v4f32.v4p1f32.v4p2f32(<4 x float> %a, <4 x float addrspace(1)*> %b, <4 x float addrspace(2)*> %c, float %d)
+define <4 x float> @test_vector_of_pointers_backward(<4 x float> %a, <4 x float addrspace(1)*> %b, <4 x float addrspace(2)*> %c, float %d) {
+  %res = call <4 x float> @llvm.experimental.dummy.backward.v4f32.v4f32.v4p1f32.v4p2f32(<4 x float> %a, <4 x float addrspace(1)*> %b, <4 x float addrspace(2)*> %c, float %d)
+  ret <4 x float> %res
+}
+declare <4 x float> @llvm.experimental.dummy.backward.v4f32.v4f32.v4p1f32.v4p2f32(<4 x float>, <4 x float addrspace(1)*>, <4 x float addrspace(2)*>, float)
+
+
+; CHECK: @test_vector_of_pointers_forward
+; CHECK: call <4 x float addrspace(1)*> @llvm.experimental.dummy.forward.v4p1f32.v4p2f32.v4f32.v4f32(<4 x float addrspace(2)*> %a, <4 x float> %b, <4 x float> %c, float %d)
+define <4 x float addrspace(1)*> @test_vector_of_pointers_forward(<4 x float addrspace(2)*> %a, <4 x float> %b, <4 x float> %c, float %d) {
+  %res = call <4 x float addrspace(1)*> @llvm.experimental.dummy.forward.v4p1f32.v4p2f32.v4f32.v4f32(<4 x float addrspace(2)*> %a, <4 x float> %b, <4 x float> %c, float %d)
+  ret <4 x float addrspace(1)*> %res
+}
+declare <4 x float addrspace(1)*> @llvm.experimental.dummy.forward.v4p1f32.v4p2f32.v4f32.v4f32(<4 x float addrspace(2)*>, <4 x float>, <4 x float>, float)
+
+; CHECK: @test_struct_ret_forward
+; CHECK: call { i32, i32, i32 } @llvm.experimental.dummy.forward.struct.ret.i32.i8(i8 %a)
+define { i32, i32, i32 } @test_struct_ret_forward(i8 %a) {
+  %res = call { i32, i32, i32 } @llvm.experimental.dummy.forward.struct.ret.i32.i8(i8 %a)
+  ret { i32, i32, i32 } %res
+}
+declare { i32, i32, i32 } @llvm.experimental.dummy.forward.struct.ret.i32.i8(i8);

utils/TableGen/CodeGenTarget.cpp

if (!TargetPrefix.empty()) {
     if (Name.size() < 6+TargetPrefix.size() ||
         std::string(Name.begin() + 5, Name.begin() + 6 + TargetPrefix.size())
         != (TargetPrefix + "."))
       PrintFatalError(DefLoc, "Intrinsic '" + DefName +
                                   "' does not start with 'llvm." +
                                   TargetPrefix + ".'!");
   }
 
-  // Parse the list of return types.
+  ListInit *RetTypes = R->getValueAsListInit("RetTypes");
+  ListInit *ParamTypes = R->getValueAsListInit("ParamTypes");
+
+  // First collate a list of overloaded types.
   std::vector<MVT::SimpleValueType> OverloadedVTs;
-  ListInit *TypeList = R->getValueAsListInit("RetTypes");
+  for (ListInit *TypeList : {RetTypes, ParamTypes}) {
+    for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
+      Record *TyEl = TypeList->getElementAsRecord(i);
+      assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
+
+      if (TyEl->isSubClassOf("LLVMMatchType"))
+        continue;
+
+      MVT::SimpleValueType VT = getValueType(TyEl->getValueAsDef("VT"));
+      if (MVT(VT).isOverloaded()) {
+        OverloadedVTs.push_back(VT);
+        isOverloaded = true;
+      }
+    }
+  }
+
+  // Parse the list of return types.
+  ListInit *TypeList = RetTypes;
   for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
     Record *TyEl = TypeList->getElementAsRecord(i);
     assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
     MVT::SimpleValueType VT;
     if (TyEl->isSubClassOf("LLVMMatchType")) {
       unsigned MatchTy = TyEl->getValueAsInt("Number");
       assert(MatchTy < OverloadedVTs.size() &&
              "Invalid matching number!");
       VT = OverloadedVTs[MatchTy];
       // It only makes sense to use the extended and truncated vector element
       // variants with iAny types; otherwise, if the intrinsic is not
       // overloaded, all the types can be specified directly.
       assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
                !TyEl->isSubClassOf("LLVMTruncatedType")) ||
               VT == MVT::iAny || VT == MVT::vAny) &&
              "Expected iAny or vAny type");
     } else {
       VT = getValueType(TyEl->getValueAsDef("VT"));
     }
-    if (MVT(VT).isOverloaded()) {
-      OverloadedVTs.push_back(VT);
-      isOverloaded = true;
-    }
 
     // Reject invalid types.
     if (VT == MVT::isVoid)
       PrintFatalError(DefLoc, "Intrinsic '" + DefName +
                                   " has void in result type list!");
 
     IS.RetVTs.push_back(VT);
     IS.RetTypeDefs.push_back(TyEl);
   }
 
   // Parse the list of parameter types.
-  TypeList = R->getValueAsListInit("ParamTypes");
+  TypeList = ParamTypes;
   for (unsigned i = 0, e = TypeList->size(); i != e; ++i) {
     Record *TyEl = TypeList->getElementAsRecord(i);
     assert(TyEl->isSubClassOf("LLVMType") && "Expected a type!");
     MVT::SimpleValueType VT;
     if (TyEl->isSubClassOf("LLVMMatchType")) {
       unsigned MatchTy = TyEl->getValueAsInt("Number");
       if (MatchTy >= OverloadedVTs.size()) {
         PrintError(R->getLoc(),
if (TyEl->isSubClassOf("LLVMMatchType")) {
       assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&
                !TyEl->isSubClassOf("LLVMTruncatedType") &&
                !TyEl->isSubClassOf("LLVMScalarOrSameVectorWidth")) ||
               VT == MVT::iAny || VT == MVT::vAny) &&
              "Expected iAny or vAny type");
     } else
       VT = getValueType(TyEl->getValueAsDef("VT"));
 
-    if (MVT(VT).isOverloaded()) {
-      OverloadedVTs.push_back(VT);
-      isOverloaded = true;
-    }
-
     // Reject invalid types.
     if (VT == MVT::isVoid && i != e-1 /*void at end means varargs*/)
       PrintFatalError(DefLoc, "Intrinsic '" + DefName +
                                   " has void in result type list!");
 
     IS.ParamVTs.push_back(VT);
     IS.ParamTypeDefs.push_back(TyEl);
   }

utils/TableGen/IntrinsicEmitter.cpp

static void EncodeFixedValueType(MVT::SimpleValueType VT,
   }
 }
 
 #if defined(_MSC_VER) && !defined(__clang__)
 #pragma optimize("",off) // MSVC 2015 optimizer can't deal with this function.
 #endif
 
 static void EncodeFixedType(Record *R, std::vector<unsigned char> &ArgCodes,
-                            std::vector<unsigned char> &Sig) {
+                            unsigned &NextArgCode,
+                            std::vector<unsigned char> &Sig,
+                            ArrayRef<unsigned char> Mapping) {
 
   if (R->isSubClassOf("LLVMMatchType")) {
-    unsigned Number = R->getValueAsInt("Number");
+    unsigned Number = Mapping[R->getValueAsInt("Number")];
     assert(Number < ArgCodes.size() && "Invalid matching number!");
     if (R->isSubClassOf("LLVMExtendedType"))
       Sig.push_back(IIT_EXTEND_ARG);
     else if (R->isSubClassOf("LLVMTruncatedType"))
       Sig.push_back(IIT_TRUNC_ARG);
     else if (R->isSubClassOf("LLVMHalfElementsVectorType"))
       Sig.push_back(IIT_HALF_VEC_ARG);
     else if (R->isSubClassOf("LLVMScalarOrSameVectorWidth")) {
       Sig.push_back(IIT_SAME_VEC_WIDTH_ARG);
       Sig.push_back((Number << 3) | ArgCodes[Number]);
       MVT::SimpleValueType VT = getValueType(R->getValueAsDef("ElTy"));
       EncodeFixedValueType(VT, Sig);
       return;
     }
     else if (R->isSubClassOf("LLVMPointerTo"))
       Sig.push_back(IIT_PTR_TO_ARG);
     else if (R->isSubClassOf("LLVMVectorOfAnyPointersToElt")) {
       Sig.push_back(IIT_VEC_OF_ANYPTRS_TO_ELT);
-      unsigned ArgNo = ArgCodes.size();
-      ArgCodes.push_back(3 /*vAny*/);
       // Encode overloaded ArgNo
-      Sig.push_back(ArgNo);
+      Sig.push_back(NextArgCode++);
       // Encode LLVMMatchType<Number> ArgNo
       Sig.push_back(Number);
       return;
     } else if (R->isSubClassOf("LLVMPointerToElt"))
       Sig.push_back(IIT_PTR_TO_ELT);
     else
       Sig.push_back(IIT_ARG);
-    return Sig.push_back((Number << 3) | ArgCodes[Number]);
+    return Sig.push_back((Number << 3) | 7 /*IITDescriptor::AK_MatchType*/);
   }
 
   MVT::SimpleValueType VT = getValueType(R->getValueAsDef("VT"));
 
   unsigned Tmp = 0;
   switch (VT) {
   default: break;
   case MVT::iPTRAny: ++Tmp; LLVM_FALLTHROUGH;
   case MVT::vAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::fAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::iAny: ++Tmp;    LLVM_FALLTHROUGH;
   case MVT::Any: {
     // If this is an "any" valuetype, then the type is the type of the next
     // type in the list specified to getIntrinsic().
     Sig.push_back(IIT_ARG);
 
     // Figure out what arg # this is consuming, and remember what kind it was.
-    unsigned ArgNo = ArgCodes.size();
-    ArgCodes.push_back(Tmp);
+    assert(NextArgCode < ArgCodes.size() && ArgCodes[NextArgCode] == Tmp &&
+           "Invalid or no ArgCode associated with overloaded VT!");
+    unsigned ArgNo = NextArgCode++;
 
     // Encode what sort of argument it must be in the low 3 bits of the ArgNo.
     return Sig.push_back((ArgNo << 3) | Tmp);
   }
 
   case MVT::iPTR: {
     unsigned AddrSpace = 0;
     if (R->isSubClassOf("LLVMQualPointerType")) {
       AddrSpace = R->getValueAsInt("AddrSpace");
       assert(AddrSpace < 256 && "Address space exceeds 255");
     }
     if (AddrSpace) {
       Sig.push_back(IIT_ANYPTR);
       Sig.push_back(AddrSpace);
     } else {
       Sig.push_back(IIT_PTR);
     }
-    return EncodeFixedType(R->getValueAsDef("ElTy"), ArgCodes, Sig);
+    return EncodeFixedType(R->getValueAsDef("ElTy"), ArgCodes, NextArgCode, Sig,
+                           Mapping);
   }
   }
 
   if (MVT(VT).isVector()) {
     MVT VVT = VT;
     switch (VVT.getVectorNumElements()) {
     default: PrintFatalError("unhandled vector type width in intrinsic!");
     case 1: Sig.push_back(IIT_V1); break;
     case 2: Sig.push_back(IIT_V2); break;
     case 4: Sig.push_back(IIT_V4); break;
     case 8: Sig.push_back(IIT_V8); break;
     case 16: Sig.push_back(IIT_V16); break;
     case 32: Sig.push_back(IIT_V32); break;
     case 64: Sig.push_back(IIT_V64); break;
     case 512: Sig.push_back(IIT_V512); break;
     case 1024: Sig.push_back(IIT_V1024); break;
     }
 
     return EncodeFixedValueType(VVT.getVectorElementType().SimpleTy, Sig);
   }
 
   EncodeFixedValueType(VT, Sig);
 }
 
+static void UpdateArgCodes(Record *R, std::vector<unsigned char> &ArgCodes,
+                           unsigned int &NumInserted,
+                           SmallVectorImpl<unsigned char> &Mapping) {
+  if (R->isSubClassOf("LLVMMatchType")) {
+    if (R->isSubClassOf("LLVMVectorOfAnyPointersToElt")) {
+      ArgCodes.push_back(3 /*vAny*/);
+      ++NumInserted;
+    }
+    return;
+  }
+
+  unsigned Tmp = 0;
+  switch (getValueType(R->getValueAsDef("VT"))) {
+  default: break;
+  case MVT::iPTRAny:
+    ++Tmp;
+    LLVM_FALLTHROUGH;
+  case MVT::vAny:
+    ++Tmp;
+    LLVM_FALLTHROUGH;
+  case MVT::fAny:
+    ++Tmp;
+    LLVM_FALLTHROUGH;
+  case MVT::iAny:
+    ++Tmp;
+    LLVM_FALLTHROUGH;
+  case MVT::Any:
+    unsigned OriginalIdx = ArgCodes.size() - NumInserted;
+    assert (OriginalIdx >= Mapping.size());
+    Mapping.resize(OriginalIdx+1);
+    Mapping[OriginalIdx] = ArgCodes.size();
+    ArgCodes.push_back(Tmp);
+    break;
+  }
+}
+
 #if defined(_MSC_VER) && !defined(__clang__)
 #pragma optimize("",on)
 #endif
 
 /// ComputeFixedEncoding - If we can encode the type signature for this
 /// intrinsic into 32 bits, return it.  If not, return ~0U.
 static void ComputeFixedEncoding(const CodeGenIntrinsic &Int,
                                  std::vector<unsigned char> &TypeSig) {
   std::vector<unsigned char> ArgCodes;
 
+  // Add codes for any overloaded result VTs.
+  unsigned int NumInserted = 0;
+  SmallVector<unsigned char, 8> ArgMapping;
+  for (unsigned i = 0, e = Int.IS.RetVTs.size(); i != e; ++i)
+    UpdateArgCodes(Int.IS.RetTypeDefs[i], ArgCodes, NumInserted, ArgMapping);
+
+  // Add codes for any overloaded operand VTs.
+  for (unsigned i = 0, e = Int.IS.ParamTypeDefs.size(); i != e; ++i)
+    UpdateArgCodes(Int.IS.ParamTypeDefs[i], ArgCodes, NumInserted, ArgMapping);
+
+  unsigned NextArgCode = 0;
   if (Int.IS.RetVTs.empty())
     TypeSig.push_back(IIT_Done);
   else if (Int.IS.RetVTs.size() == 1 &&
            Int.IS.RetVTs[0] == MVT::isVoid)
     TypeSig.push_back(IIT_Done);
   else {
     switch (Int.IS.RetVTs.size()) {
       case 1: break;
       case 2: TypeSig.push_back(IIT_STRUCT2); break;
       case 3: TypeSig.push_back(IIT_STRUCT3); break;
       case 4: TypeSig.push_back(IIT_STRUCT4); break;
       case 5: TypeSig.push_back(IIT_STRUCT5); break;
       case 6: TypeSig.push_back(IIT_STRUCT6); break;
       case 7: TypeSig.push_back(IIT_STRUCT7); break;
       case 8: TypeSig.push_back(IIT_STRUCT8); break;
       default: llvm_unreachable("Unhandled case in struct");
     }
 
     for (unsigned i = 0, e = Int.IS.RetVTs.size(); i != e; ++i)
-      EncodeFixedType(Int.IS.RetTypeDefs[i], ArgCodes, TypeSig);
+      EncodeFixedType(Int.IS.RetTypeDefs[i], ArgCodes, NextArgCode, TypeSig,
+                      ArgMapping);
   }
 
   for (unsigned i = 0, e = Int.IS.ParamTypeDefs.size(); i != e; ++i)
-    EncodeFixedType(Int.IS.ParamTypeDefs[i], ArgCodes, TypeSig);
+    EncodeFixedType(Int.IS.ParamTypeDefs[i], ArgCodes, NextArgCode, TypeSig,
+                    ArgMapping);
 }
 
 static void printIITEntry(raw_ostream &OS, unsigned char X) {
   OS << (unsigned)X;
 }
 
 void IntrinsicEmitter::EmitGenerator(const CodeGenIntrinsicTable &Ints,
                                      raw_ostream &OS) {