POC: Add `elementtype` attribute requirement on atomic memory intrinsics.

[anna]

Summary

LLVM officially supports Garbage Collection environments.
Such environments must be able to detect type of accessed object.
With opaque pointers the only possible way for memory intrinsics is
elementtype argument attribute. This patch adds the support in the
form of LangRef, verifier and pass updates.

[nikic]

Not familiar with GC requirements, but this at least sounds plausible to me (maybe @reames can confirm?) However, I think that if we go down this line, we should probably go all in: Drop the element_size argument from the intrinsics and make elementtype required by the IR verifier, and adjust lowering to perform copies in the correct type.

[reames]

I don't understand the motivation here. Clearly, such an environment could tag the operations this way, and probably should. However, why does that need to be in the LangRef as a requirement? Is the idea to disallow some particular transform in tree that can't meet this?

[apilipenko]

The optimizer can insert new atomic memcpy/memmove calls, e.g loop-idiom-recognize would do that. Currently, when loop-idiom-recognize inserts such a call it doesn't tag it with the element type, which makes it an incorrect transform for us.

If we document the requirement in the langref we can a) add a verification rule to catch these situations, b) fix such transforms upstream. I think we can restrict this requirement to functions that have a GC strategy specified.

[dantrushin]

Idea is being able to push something like these though upstream:
https://reviews.llvm.org/D125690
https://reviews.llvm.org/D127892
From our downstream point of view they look reasonable, but we've encountered upstream resistance on first (so most likely will see it on second as well if we do it unconditionally)

[anna]

Thanks @dantrushin for the patch. We have now had nasty miscompiles in practice (lowering memory intrinsics to regular loops and miss adding barriers for GC) and would like to make progress with this patch.

As a first step, we plan to document in the LangRef that functions having GCStrategy specified needs to have elementtype attribute set for the src and dest arguments for the atomic intrinsics. Also, introducing the API to record the elementtype (https://reviews.llvm.org/D125690).
The next step would be to have the verification rule for this and to teach passes/utilities about this: LoopIdiomRecognize and LowerMemIntrinsics.cpp (utility API for atomic mem intrinsics).

Once both of these have baked in, we can then remove the elementsize argument altogether from the atomic mem intrinsics since they can be inferred from the elementtype and remove the GCStrategy requirement as well. This also allows non-GC users of atomic memory intrinsics enough time to react to the change.

This perhaps deserves an RFC on the mailing list.

[anna]

This is a POC of most of the changes for the RFC to follow. The change will be split into LangRef + API as one change, Verifier + pass updates as another change.

llvm/docs/LangRef.rst

 Overview:
 """""""""
 
 The '``llvm.memcpy.element.unordered.atomic.*``' intrinsic is a specialization of the
 '``llvm.memcpy.*``' intrinsic. It differs in that the ``dest`` and ``src`` are treated
 as arrays with elements that are exactly ``element_size`` bytes, and the copy between
 buffers uses a sequence of :ref:`unordered atomic <ordering>` load/store operations
 that are a positive integer multiple of the ``element_size`` in size.
+If this intrinsic is called in a function that requires garbage collection
+:ref:`gc`, then the ``dest`` and ``src`` arguments should have ``elementtype``
+attribute set. The element type information is needed to choose the correct lowering for
+the intrinsic.
 
 Arguments:
 """"""""""
 
 The first three arguments are the same as they are in the :ref:`@llvm.memcpy <int_memcpy>`
 intrinsic, with the added constraint that ``len`` is required to be a positive integer
 multiple of the ``element_size``. If ``len`` is not a positive integer multiple of
 ``element_size``, then the behaviour of the intrinsic is undefined.

 """""""""
 
 The '``llvm.memmove.element.unordered.atomic.*``' intrinsic is a specialization
 of the '``llvm.memmove.*``' intrinsic. It differs in that the ``dest`` and
 ``src`` are treated as arrays with elements that are exactly ``element_size``
 bytes, and the copy between buffers uses a sequence of
 :ref:`unordered atomic <ordering>` load/store operations that are a positive
 integer multiple of the ``element_size`` in size.
+If this intrinsic is called in a function that requires garbage collection
+:ref:`gc`, then the ``dest`` and ``src`` arguments should have ``elementtype``
+attribute set. The element type information is needed to choose the correct lowering for
+the intrinsic.
 
 Arguments:
 """"""""""
 
 The first three arguments are the same as they are in the
 :ref:`@llvm.memmove <int_memmove>` intrinsic, with the added constraint that
 ``len`` is required to be a positive integer multiple of the ``element_size``.
 If ``len`` is not a positive integer multiple of ``element_size``, then the

llvm/include/llvm/IR/IRBuilder.h

#endif
   ///
   /// If the pointers aren't i8*, they will be converted.  If a TBAA tag is
   /// specified, it will be added to the instruction. Likewise with alias.scope
   /// and noalias tags.
   CallInst *CreateElementUnorderedAtomicMemCpy(
       Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size,
       uint32_t ElementSize, MDNode *TBAATag = nullptr,
       MDNode *TBAAStructTag = nullptr, MDNode *ScopeTag = nullptr,
-      MDNode *NoAliasTag = nullptr);
+      MDNode *NoAliasTag = nullptr,
+      std::optional<Type *> ElementType = std::nullopt);
 
   CallInst *CreateMemMove(Value *Dst, MaybeAlign DstAlign, Value *Src,
                           MaybeAlign SrcAlign, uint64_t Size,
                           bool isVolatile = false, MDNode *TBAATag = nullptr,
                           MDNode *ScopeTag = nullptr,
                           MDNode *NoAliasTag = nullptr) {
     return CreateMemMove(Dst, DstAlign, Src, SrcAlign, getInt64(Size),
                          isVolatile, TBAATag, ScopeTag, NoAliasTag);
#endif
   ///
   /// If the pointers aren't i8*, they will be converted.  If a TBAA tag is
   /// specified, it will be added to the instruction. Likewise with alias.scope
   /// and noalias tags.
   CallInst *CreateElementUnorderedAtomicMemMove(
       Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size,
       uint32_t ElementSize, MDNode *TBAATag = nullptr,
       MDNode *TBAAStructTag = nullptr, MDNode *ScopeTag = nullptr,
-      MDNode *NoAliasTag = nullptr);
+      MDNode *NoAliasTag = nullptr,
+      std::optional<Type *> ElementTy = std::nullopt);
 
 private:
   CallInst *getReductionIntrinsic(Intrinsic::ID ID, Value *Src);
 
 public:
   /// Create a sequential vector fadd reduction intrinsic of the source vector.
   /// The first parameter is a scalar accumulator value. An unordered reduction
   /// can be created by adding the reassoc fast-math flag to the resulting

llvm/include/llvm/IR/IntrinsicInst.h

public:
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 // This class wraps the atomic memcpy/memmove intrinsics
 // i.e. llvm.element.unordered.atomic.memcpy/memmove
 class AtomicMemTransferInst : public MemTransferBase<AtomicMemIntrinsic> {
+private:
+  enum { ARG_DEST = 0, ARG_SOURCE = 1 };
+
 public:
   static bool classof(const IntrinsicInst *I) {
     switch (I->getIntrinsicID()) {
     case Intrinsic::memcpy_element_unordered_atomic:
     case Intrinsic::memmove_element_unordered_atomic:
       return true;
     default:
       return false;
     }
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
+  // These API returns a type only if the AtomicMemTransferInst is
+  // called from a function that has a GC strategy specified. 
+  Type *getSourceElementType() const { return getParamElementType(ARG_SOURCE); }
+  Type *getDestElementType() const { return getParamElementType(ARG_DEST); }
 };
 
 /// This class represents the atomic memcpy intrinsic
 /// i.e. llvm.element.unordered.atomic.memcpy
 class AtomicMemCpyInst : public AtomicMemTransferInst {
 public:
   static bool classof(const IntrinsicInst *I) {
     return I->getIntrinsicID() == Intrinsic::memcpy_element_unordered_atomic;

llvm/lib/IR/IRBuilder.cpp

if (NoAliasTag)
     CI->setMetadata(LLVMContext::MD_noalias, NoAliasTag);
 
   return CI;
 }
 
 CallInst *IRBuilderBase::CreateElementUnorderedAtomicMemCpy(
     Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size,
     uint32_t ElementSize, MDNode *TBAATag, MDNode *TBAAStructTag,
-    MDNode *ScopeTag, MDNode *NoAliasTag) {
+    MDNode *ScopeTag, MDNode *NoAliasTag, std::optional<Type *> ElementTy) {
   assert(DstAlign >= ElementSize &&
          "Pointer alignment must be at least element size");
   assert(SrcAlign >= ElementSize &&
          "Pointer alignment must be at least element size");
   Value *Ops[] = {Dst, Src, Size, getInt32(ElementSize)};
   Type *Tys[] = {Dst->getType(), Src->getType(), Size->getType()};
   Module *M = BB->getParent()->getParent();
   Function *TheFn = Intrinsic::getDeclaration(
if (TBAAStructTag)
     CI->setMetadata(LLVMContext::MD_tbaa_struct, TBAAStructTag);
 
   if (ScopeTag)
     CI->setMetadata(LLVMContext::MD_alias_scope, ScopeTag);
 
   if (NoAliasTag)
     CI->setMetadata(LLVMContext::MD_noalias, NoAliasTag);
 
+  if (ElementTy.has_value()) {
+    CI->addParamAttr(0, Attribute::get(CI->getContext(), Attribute::ElementType,
+                                       *ElementTy));
+    CI->addParamAttr(1, Attribute::get(CI->getContext(), Attribute::ElementType,
+                                       *ElementTy));
+  }
+
   return CI;
 }
 
 CallInst *IRBuilderBase::CreateElementUnorderedAtomicMemMove(
     Value *Dst, Align DstAlign, Value *Src, Align SrcAlign, Value *Size,
     uint32_t ElementSize, MDNode *TBAATag, MDNode *TBAAStructTag,
-    MDNode *ScopeTag, MDNode *NoAliasTag) {
+    MDNode *ScopeTag, MDNode *NoAliasTag, std::optional<Type *> ElementTy) {
   assert(DstAlign >= ElementSize &&
          "Pointer alignment must be at least element size");
   assert(SrcAlign >= ElementSize &&
          "Pointer alignment must be at least element size");
   Value *Ops[] = {Dst, Src, Size, getInt32(ElementSize)};
   Type *Tys[] = {Dst->getType(), Src->getType(), Size->getType()};
   Module *M = BB->getParent()->getParent();
   Function *TheFn = Intrinsic::getDeclaration(
if (TBAAStructTag)
     CI->setMetadata(LLVMContext::MD_tbaa_struct, TBAAStructTag);
 
   if (ScopeTag)
     CI->setMetadata(LLVMContext::MD_alias_scope, ScopeTag);
 
   if (NoAliasTag)
     CI->setMetadata(LLVMContext::MD_noalias, NoAliasTag);
 
+  if (ElementTy.has_value()) {
+    CI->addParamAttr(0, Attribute::get(CI->getContext(), Attribute::ElementType,
+                                       *ElementTy));
+    CI->addParamAttr(1, Attribute::get(CI->getContext(), Attribute::ElementType,
+                                       *ElementTy));
+  }
+
   return CI;
 }
 
 CallInst *IRBuilderBase::getReductionIntrinsic(Intrinsic::ID ID, Value *Src) {
   Module *M = GetInsertBlock()->getParent()->getParent();
   Value *Ops[] = {Src};
   Type *Tys[] = { Src->getType() };
   auto Decl = Intrinsic::getDeclaration(M, ID, Tys);

llvm/lib/IR/Verifier.cpp

#include "llvm/IR/ConstrainedOps.def"
   case Intrinsic::memcpy_inline:
   case Intrinsic::memmove:
   case Intrinsic::memset:
   case Intrinsic::memset_inline: {
     break;
   }
   case Intrinsic::memcpy_element_unordered_atomic:
   case Intrinsic::memmove_element_unordered_atomic:
+                                 {
+    const auto *AMTI = cast<AtomicMemTransferInst>(&Call);
+    if (Call.getParent()->getParent()->hasGC()) {
+      Check(AMTI->getSourceElementType() != nullptr,
+            "elementtype required for atomic memory intrinsic with GC support",
+            Call);
+      Check(AMTI->getSourceElementType() == AMTI->getDestElementType(),
+            "expected source and dest element type to be same", Call);
+    }
+                                 }
+    [[fallthrough]];
   case Intrinsic::memset_element_unordered_atomic: {
     const auto *AMI = cast<AtomicMemIntrinsic>(&Call);
 
     ConstantInt *ElementSizeCI =
         cast<ConstantInt>(AMI->getRawElementSizeInBytes());
     const APInt &ElementSizeVal = ElementSizeCI->getValue();
     Check(ElementSizeVal.isPowerOf2(),
           "element size of the element-wise atomic memory intrinsic "

llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp

if (!TheStore->isAtomic() && !TheLoad->isAtomic()) {
 
     // If the element.atomic memcpy is not lowered into explicit
     // loads/stores later, then it will be lowered into an element-size
     // specific lib call. If the lib call doesn't exist for our store size, then
     // we shouldn't generate the memcpy.
     if (StoreSize > TTI->getAtomicMemIntrinsicMaxElementSize())
       return Changed;
 
+    std::optional<Type *> ElementType = std::nullopt;
+    // Record ElementType for correct lowering of atomic memcpy if the function
+    // requires GC support.
+    if (TheLoad->getFunction()->hasGC())
+      ElementType = TheLoad->getType();
+
     // Create the call.
     // Note that unordered atomic loads/stores are *required* by the spec to
     // have an alignment but non-atomic loads/stores may not.
     NewCall = Builder.CreateElementUnorderedAtomicMemCpy(
         StoreBasePtr, *StoreAlign, LoadBasePtr, *LoadAlign, NumBytes, StoreSize,
-        AATags.TBAA, AATags.TBAAStruct, AATags.Scope, AATags.NoAlias);
+        AATags.TBAA, AATags.TBAAStruct, AATags.Scope, AATags.NoAlias,
+        ElementType);
   }
   NewCall->setDebugLoc(TheStore->getDebugLoc());
 
   if (MSSAU) {
     MemoryAccess *NewMemAcc = MSSAU->createMemoryAccessInBB(
         NewCall, nullptr, NewCall->getParent(), MemorySSA::BeforeTerminator);
     MSSAU->insertDef(cast<MemoryDef>(NewMemAcc), true);
   }

llvm/test/Transforms/LoopIdiom/X86/unordered-atomic-memcpy.ll

for.body:                                         ; preds = %bb.nph, %for.body
   %indvar.next = add i64 %indvar, 1
   %exitcond = icmp eq i64 %indvar.next, %Size
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
+
+define void @test1_gc(i64 %Size) nounwind ssp gc "statepoint-example" {
+; CHECK-LABEL: @test1_gc(
+; CHECK-NEXT:  bb.nph:
+; CHECK-NEXT:    [[BASE:%.*]] = alloca i8, i32 10000, align 1
+; CHECK-NEXT:    [[DEST:%.*]] = alloca i8, i32 10000, align 1
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p0.p0.i64(ptr elementtype(i8) align 1 [[DEST]], ptr elementtype(i8) align 1 [[BASE]], i64 [[SIZE:%.*]], i32 1)
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[INDVAR:%.*]] = phi i64 [ 0, [[BB_NPH:%.*]] ], [ [[INDVAR_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[I_0_014:%.*]] = getelementptr i8, ptr [[BASE]], i64 [[INDVAR]]
+; CHECK-NEXT:    [[DESTI:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[INDVAR]]
+; CHECK-NEXT:    [[V:%.*]] = load atomic i8, ptr [[I_0_014]] unordered, align 1
+; CHECK-NEXT:    [[INDVAR_NEXT]] = add i64 [[INDVAR]], 1
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[INDVAR_NEXT]], [[SIZE]]
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_END:%.*]], label [[FOR_BODY]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+bb.nph:
+  %Base = alloca i8, i32 10000
+  %Dest = alloca i8, i32 10000
+  br label %for.body
+
+for.body:                                         ; preds = %bb.nph, %for.body
+  %indvar = phi i64 [ 0, %bb.nph ], [ %indvar.next, %for.body ]
+  %I.0.014 = getelementptr i8, ptr %Base, i64 %indvar
+  %DestI = getelementptr i8, ptr %Dest, i64 %indvar
+  %V = load atomic i8, ptr %I.0.014 unordered, align 1
+  store atomic i8 %V, ptr %DestI unordered, align 1
+  %indvar.next = add i64 %indvar, 1
+  %exitcond = icmp eq i64 %indvar.next, %Size
+  br i1 %exitcond, label %for.end, label %for.body
+
+for.end:                                          ; preds = %for.body, %entry
+  ret void
+}
+
 ;; memcpy.atomic formation (atomic store, normal load)
 define void @test2(i64 %Size) nounwind ssp {
 ; CHECK-LABEL: @test2(
 ; CHECK-NEXT:  bb.nph:
 ; CHECK-NEXT:    [[BASE:%.*]] = alloca i8, i32 10000, align 1
 ; CHECK-NEXT:    [[DEST:%.*]] = alloca i8, i32 10000, align 1
 ; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p0.p0.i64(ptr align 1 [[DEST]], ptr align 1 [[BASE]], i64 [[SIZE:%.*]], i32 1)
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]

llvm/test/Transforms/RewriteStatepointsForGC/unordered-atomic-memcpy-no-deopt.ll

 ; RUN: opt -passes=rewrite-statepoints-for-gc -rs4gc-allow-statepoint-with-no-deopt-info=0 -S < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-REQUIRE-DEOPT
 ; RUN: opt -passes=rewrite-statepoints-for-gc -rs4gc-allow-statepoint-with-no-deopt-info=1 -S < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-NO-REQUIRE-DEOPT
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 declare void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1), ptr addrspace(1), i32, i32 immarg)
 declare void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1), ptr addrspace(1), i32, i32 immarg)
 
 define void @test_memcpy_no_deopt(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: @test_memcpy_no_deopt
 ; CHECK-REQUIRE-DEOPT-NOT: @llvm.experimental.gc.statepoint
 ; CHECK-NO-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1)
   ret void
 }
 
 define void @test_memmove_no_deopt(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: @test_memmove_no_deopt
 ; CHECK-REQUIRE-DEOPT-NOT: @llvm.experimental.gc.statepoint
 ; CHECK-NO-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1)
   ret void
 }
 
 define void @test_memcpy_with_deopt(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: @test_memcpy_with_deopt
 ; CHECK-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 ; CHECK-NO-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1) [ "deopt"(i32 0) ]
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1) [ "deopt"(i32 0) ]
   ret void
 }
 
 define void @test_memmove_with_deopt(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: @test_memmove_with_deopt
 ; CHECK-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 ; CHECK-NO-REQUIRE-DEOPT: @llvm.experimental.gc.statepoint
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1) [ "deopt"(i32 0) ]
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1) [ "deopt"(i32 0) ]
   ret void
 }

llvm/test/Transforms/RewriteStatepointsForGC/unordered-atomic-memcpy.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature
 ; Use instcombine to cleanup offset computation.
 ; RUN: opt -passes=rewrite-statepoints-for-gc,instcombine -S < %s | FileCheck %s
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128-p1:64:64"
 target triple = "x86_64-apple-macosx10.11.0"
 
 declare void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1), ptr addrspace(1), i32, i32 immarg)
 declare void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1), ptr addrspace(1), i32, i32 immarg)
 
 define void @test_memcpy_gc_leaf_function(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_gc_leaf_function
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[SRC_DERIVED:%.*]] = getelementptr inbounds i8, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]]
 ; CHECK-NEXT:    [[DEST_DERIVED:%.*]] = getelementptr inbounds i8, ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]]
-; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 1) #[[ATTR2:[0-9]+]]
-; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 2) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 4) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 8) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 16) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 1) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 2) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 4) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 8) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 16) #[[ATTR2]]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
 
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1) "gc-leaf-function"
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 2) "gc-leaf-function"
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 4) "gc-leaf-function"
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 8) "gc-leaf-function"
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 16) "gc-leaf-function"
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1) "gc-leaf-function"
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 2) "gc-leaf-function"
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 4) "gc-leaf-function"
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 8) "gc-leaf-function"
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 16) "gc-leaf-function"
   ret void
 }
 
 define void @test_memcpy_element_atomic_1(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_element_atomic_1
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memcpy_element_unordered_atomic_safepoint_1, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1)
   ret void
 }
 
 define void @test_memcpy_element_atomic_2(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_element_atomic_2
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memcpy_element_unordered_atomic_safepoint_2, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 2)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 2)
   ret void
 }
 
 define void @test_memcpy_element_atomic_4(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_element_atomic_4
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memcpy_element_unordered_atomic_safepoint_4, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 4)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 4)
   ret void
 }
 
 define void @test_memcpy_element_atomic_8(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_element_atomic_8
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memcpy_element_unordered_atomic_safepoint_8, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 8)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 8)
   ret void
 }
 
 define void @test_memcpy_element_atomic_16(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memcpy_element_atomic_16
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memcpy_element_unordered_atomic_safepoint_16, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 16)
+  call void @llvm.memcpy.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 16)
   ret void
 }
 
 define void @test_memmove_gc_leaf_function(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_gc_leaf_function
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[SRC_DERIVED:%.*]] = getelementptr inbounds i8, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]]
 ; CHECK-NEXT:    [[DEST_DERIVED:%.*]] = getelementptr inbounds i8, ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]]
-; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 1) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 2) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 4) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 8) #[[ATTR2]]
-; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 [[SRC_DERIVED]], ptr addrspace(1) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 16) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 1) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 2) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 4) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 8) #[[ATTR2]]
+; CHECK-NEXT:    call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[SRC_DERIVED]], ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 [[DEST_DERIVED]], i32 [[LEN]], i32 16) #[[ATTR2]]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
 
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1) "gc-leaf-function"
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 2) "gc-leaf-function"
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 4) "gc-leaf-function"
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 8) "gc-leaf-function"
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 16) "gc-leaf-function"
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1) "gc-leaf-function"
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 2) "gc-leaf-function"
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 4) "gc-leaf-function"
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 8) "gc-leaf-function"
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 16) "gc-leaf-function"
   ret void
 }
 
 define void @test_memmove_element_atomic_1(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_element_atomic_1
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memmove_element_unordered_atomic_safepoint_1, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 1)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 1)
   ret void
 }
 
 define void @test_memmove_element_atomic_2(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_element_atomic_2
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memmove_element_unordered_atomic_safepoint_2, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 2)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 2)
   ret void
 }
 
 define void @test_memmove_element_atomic_4(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_element_atomic_4
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memmove_element_unordered_atomic_safepoint_4, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 4)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 4)
   ret void
 }
 
 define void @test_memmove_element_atomic_8(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_element_atomic_8
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memmove_element_unordered_atomic_safepoint_8, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 8)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 8)
   ret void
 }
 
 define void @test_memmove_element_atomic_16(ptr addrspace(1) %src, i64 %src_offset, ptr addrspace(1) %dest, i64 %dest_offset, i32 %len) gc "statepoint-example" {
 ; CHECK-LABEL: define {{[^@]+}}@test_memmove_element_atomic_16
 ; CHECK-SAME: (ptr addrspace(1) [[SRC:%.*]], i64 [[SRC_OFFSET:%.*]], ptr addrspace(1) [[DEST:%.*]], i64 [[DEST_OFFSET:%.*]], i32 [[LEN:%.*]]) gc "statepoint-example" {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[STATEPOINT_TOKEN:%.*]] = call token (i64, i32, ptr, i32, i32, ...) @llvm.experimental.gc.statepoint.p0(i64 2882400000, i32 0, ptr nonnull elementtype(void (ptr addrspace(1), i64, ptr addrspace(1), i64, i32)) @__llvm_memmove_element_unordered_atomic_safepoint_16, i32 5, i32 0, ptr addrspace(1) [[SRC]], i64 [[SRC_OFFSET]], ptr addrspace(1) [[DEST]], i64 [[DEST_OFFSET]], i32 [[LEN]], i32 0, i32 0) [ "gc-live"() ]
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %src_derived = getelementptr inbounds i8, ptr addrspace(1) %src, i64 %src_offset
   %dest_derived = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %dest_offset
-  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) align 16 %src_derived, ptr addrspace(1) align 16 %dest_derived, i32 %len, i32 16)
+  call void @llvm.memmove.element.unordered.atomic.p1.p1.i32(ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %src_derived, ptr addrspace(1) elementtype(ptr addrspace(1)) align 16 %dest_derived, i32 %len, i32 16)
   ret void
 }