[instrinsics] Add @llvm.memcpy.inline instrinsics

[gchatelet]

Summary

This is a follow up on D61634. It adds an LLVM IR intrinsic to allow better implementation of memcpy from C++.
A follow up CL will add the intrinsics in Clang.

[jdoerfert]

This looks pretty reasonable to me. Test coverage is low though:
I think we need to add it to the read/write intrinsic tests (that I thought existed). Also some tests that verify we handle memcpy in a certain way should be copied for this intrinsic.

llvm/include/llvm/IR/Intrinsics.td

512

Nit: I'd move this below just after the non-inline version.

[gchatelet]

• address comments

[gchatelet]

In D71710#1791589, @jdoerfert wrote:

This looks pretty reasonable to me. Test coverage is low though:
I think we need to add it to the read/write intrinsic tests (that I thought existed). Also some tests that verify we handle memcpy in a certain way should be copied for this intrinsic.

Thx for the review @jdoerfert. I tried to add the relevant tests, do you have any other tests in mind?

llvm/include/llvm/IR/Intrinsics.td

512

Ok although it's not part of the 'Standard C Library Intrinsics' per say.

[gchatelet]

• - Add a Lint test for overlapping buffers

[jdoerfert]

This looks fine to me, I wait and hope someone else accepts.

[gchatelet]

Can someone else have a look and approve this patch?

[tejohnson]

In D71710#1805803, @gchatelet wrote:

Can someone else have a look and approve this patch?

Looks like @efriedma suggested this approach - perhaps he can review?

[efriedma]

Needs a LangRef update.

It would be nice to do a bit more work so various optimizations handle llvm.memcpy.inline in a way that isn't completely conservative (for example, in alias analysis), but I guess that doesn't need to be in the initial patch.

llvm/include/llvm/IR/IntrinsicInst.h

659

Maybe hide the base class getLength() with a version that returns a ConstantInt instead? You could use it, for example, in Lint.cpp.

llvm/include/llvm/IR/Intrinsics.td

515

Maybe clarify what "inlined" means. In this context, it should mean specifically that it doesn't call any external function, including memcpy. (It would be difficult to reliably guarantee that it doesn't contain any call instructions; for example, we might outline a memcpy sequence, and that's probably fine.)

llvm/test/CodeGen/X86/memcpy-inline.ll

13

It would be a good idea to have a testcase for a larger size that wouldn't normally be inlined. (Also, maybe a test on another target that doesn't have a "memcpy" instruction.)

[gchatelet]

• Rebase and address comments

[gchatelet]

llvm/include/llvm/IR/IntrinsicInst.h

659

For Lint.cpp it has to go through findValue which erases the type, not sure what you prefer, make findValue a template or cast (as it's currently done)

llvm/test/CodeGen/X86/memcpy-inline.ll

13

Also, maybe a test on another target that doesn't have a "memcpy" instruction.

I'm not sure what you want to test with this.

[gchatelet]

In D71710#1806641, @efriedma wrote:

It would be nice to do a bit more work so various optimizations handle llvm.memcpy.inline in a way that isn't completely conservative (for example, in alias analysis), but I guess that doesn't need to be in the initial patch.

I'm not too familiar with AA but my understanding is that the definition in llvm/include/llvm/IR/Intrinsics.td already describes the arguments with enough precision to be helpful for the AA pass.

def int_memcpy_inline
    : Intrinsic<[],
      [ llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty, llvm_i1_ty ],
      [ IntrArgMemOnly, NoCapture<0>, NoCapture<1>, WriteOnly<0>, ReadOnly<1>,
      ImmArg<2>, ImmArg<3> ]>;

Did you have something specific in mind? Or am I missing something completely?

[efriedma]

In D71710#1821546, @gchatelet wrote:

In D71710#1806641, @efriedma wrote:

It would be nice to do a bit more work so various optimizations handle llvm.memcpy.inline in a way that isn't completely conservative (for example, in alias analysis), but I guess that doesn't need to be in the initial patch.

I'm not too familiar with AA but my understanding is that the definition in llvm/include/llvm/IR/Intrinsics.td already describes the arguments with enough precision to be helpful for the AA pass.

def int_memcpy_inline
    : Intrinsic<[],
      [ llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty, llvm_i1_ty ],
      [ IntrArgMemOnly, NoCapture<0>, NoCapture<1>, WriteOnly<0>, ReadOnly<1>,
      ImmArg<2>, ImmArg<3> ]>;

Did you have something specific in mind? Or am I missing something completely?

The attributes are helpful, but not enough to figure out how many bytes are modified.

llvm/include/llvm/IR/IntrinsicInst.h

659

You don't need to go through findValue to find a value that's already a ConstantInt.

llvm/test/CodeGen/X86/memcpy-inline.ll

13

Probably not that important, given the tests you have already; nevermind.

[gchatelet]

In D71710#1822800, @efriedma wrote:

In D71710#1821546, @gchatelet wrote:

In D71710#1806641, @efriedma wrote:

It would be nice to do a bit more work so various optimizations handle llvm.memcpy.inline in a way that isn't completely conservative (for example, in alias analysis), but I guess that doesn't need to be in the initial patch.

I'm not too familiar with AA but my understanding is that the definition in llvm/include/llvm/IR/Intrinsics.td already describes the arguments with enough precision to be helpful for the AA pass.

def int_memcpy_inline
    : Intrinsic<[],
      [ llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty, llvm_i1_ty ],
      [ IntrArgMemOnly, NoCapture<0>, NoCapture<1>, WriteOnly<0>, ReadOnly<1>,
      ImmArg<2>, ImmArg<3> ]>;

Did you have something specific in mind? Or am I missing something completely?

The attributes are helpful, but not enough to figure out how many bytes are modified.

I see, I looked it up a bit but it will take some time for me to understand the framework and where to add this information.
Can you provide me with pointers?
I can do this as a follow up patch also or someone more knowledgeable than me maybe?

[gchatelet]

• Simplify code

[gchatelet]

@efriedma does it look good enough as is? Any other suggestions?

[efriedma]

A couple more minor issues.

I see, I looked it up a bit but it will take some time for me to understand the framework and where to add this information.
Can you provide me with pointers?

There's code in BasicAAResult::getModRefInfo, which calls MemoryLocation::getForSource(). I guess you get the implementation of getForSource() for free, by inheriting from MemTransferInst, but I'm not sure we actually call it with your current patch, because you didn't modify classof for AnyMemCpyInst.

There might be a few other places where we could check for llvm.memcpy.inline, but nothing really high-priority, I think.

llvm/include/llvm/IR/IntrinsicInst.h

669

Should memcpy_inline count as an AnyMemIntrinsic/AnyMemTransferInst/AnyMemCpyInst?

llvm/lib/Analysis/Lint.cpp

351

isIntN(64)? Your LangRef documentation explicitly says it doesn't have to be an i64.

You can just drop the assertion, and use getLimitedValue() instead of getZExtValue().

[gchatelet]

• address comments
• - Add a Lint test for overlapping buffers
• Address comments
• Simplify code
• - Address comments, fix test

[gchatelet]

• rebasing

[gchatelet]

Looks good to you @efriedma?

llvm/include/llvm/IR/IntrinsicInst.h

669

Yes I believe it should.

[xbolva00]

llvm/include/llvm/IR/Intrinsics.td

522

NoAlias?

[gchatelet]

• Add NoAlias

llvm/include/llvm/IR/Intrinsics.td

522

Good catch.

[xbolva00]

llvm/include/llvm/IR/Intrinsics.td

522

IntrWillReturn too? (Memcpy has it too)

[gchatelet]

• Fixed test - addressed comments

[gchatelet]

• rebasing

[gchatelet]

A gentle ping

[efriedma]

LGTM

[gchatelet]

Thx everyone for the review !

[lebedev.ri]

llvm/docs/LangRef.rst

11732–11733

This is kinda too late, but i'm failing to find where it's explained why the size is restricted to being an immediate?

[gchatelet]

llvm/docs/LangRef.rst

11732–11733

The discussion happened on the POC patch

[lebedev.ri]

llvm/docs/LangRef.rst

11732–11733

I see, efficient lowering problems. Thanks.

llvm/docs/LangRef.rst

 source location to the destination location, which are not allowed to
 overlap. It copies "len" bytes of memory over. If the argument is known
 to be aligned to some boundary, this can be specified as an attribute on
 the argument.
 
 If "len" is 0, the pointers may be NULL or dangling. However, they must still
 be appropriately aligned.
 
+.. _int_memcpy_inline:
+
+'``llvm.memcpy.inline``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+This is an overloaded intrinsic. You can use ``llvm.memcpy.inline`` on any
+integer bit width and for different address spaces. Not all targets
+support all bit widths however.
+
+::
+
+      declare void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* <dest>, i8* <src>,
+                                                     i32 <len>, i1 <isvolatile>)
+      declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* <dest>, i8* <src>,
+                                                     i64 <len>, i1 <isvolatile>)
+
+Overview:
+"""""""""
+
+The '``llvm.memcpy.inline.*``' intrinsics copy a block of memory from the
+source location to the destination location and guarantees that no external
+functions are called.
+
+Note that, unlike the standard libc function, the ``llvm.memcpy.inline.*``
+intrinsics do not return a value, takes extra isvolatile
+arguments and the pointers can be in specified address spaces.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the destination, the second is a
+pointer to the source. The third argument is a constant integer argument
+specifying the number of bytes to copy, and the fourth is a
+boolean indicating a volatile access.
+
+The :ref:`align <attr_align>` parameter attribute can be provided
+for the first and second arguments.
+
+If the ``isvolatile`` parameter is ``true``, the ``llvm.memcpy.inline`` call is
+a :ref:`volatile operation <volatile>`. The detailed access behavior is not
+very cleanly specified and it is unwise to depend on it.
+
+Semantics:
+""""""""""
+
+The '``llvm.memcpy.inline.*``' intrinsics copy a block of memory from the
+source location to the destination location, which are not allowed to
+overlap. It copies "len" bytes of memory over. If the argument is known
+to be aligned to some boundary, this can be specified as an attribute on
+the argument.
+
+If "len" is 0, the pointers may be NULL or dangling. However, they must still
+be appropriately aligned.
+
+The generated code is guaranteed not to call any external functions.
+
 .. _int_memmove:
 
 '``llvm.memmove``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 

llvm/include/llvm/IR/IntrinsicInst.h

public:
     void setVolatile(Constant *V) { setArgOperand(ARG_VOLATILE, V); }
 
     // Methods for support type inquiry through isa, cast, and dyn_cast:
     static bool classof(const IntrinsicInst *I) {
       switch (I->getIntrinsicID()) {
       case Intrinsic::memcpy:
       case Intrinsic::memmove:
       case Intrinsic::memset:
+      case Intrinsic::memcpy_inline:
         return true;
       default: return false;
       }
     }
     static bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };
public:
     }
   };
 
   /// This class wraps the llvm.memcpy/memmove intrinsics.
   class MemTransferInst : public MemTransferBase<MemIntrinsic> {
   public:
     // Methods for support type inquiry through isa, cast, and dyn_cast:
     static bool classof(const IntrinsicInst *I) {
-      return I->getIntrinsicID() == Intrinsic::memcpy ||
-             I->getIntrinsicID() == Intrinsic::memmove;
+      switch (I->getIntrinsicID()) {
+      case Intrinsic::memcpy:
+      case Intrinsic::memmove:
+      case Intrinsic::memcpy_inline:
+        return true;
+      default:
+        return false;
+      }
     }
     static bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };
 
   /// This class wraps the llvm.memcpy intrinsic.
   class MemCpyInst : public MemTransferInst {
public:
     static bool classof(const IntrinsicInst *I) {
       return I->getIntrinsicID() == Intrinsic::memmove;
     }
     static bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };
 
+  /// This class wraps the llvm.memcpy.inline intrinsic.
+  class MemCpyInlineInst : public MemTransferInst {
+  public:
+    ConstantInt *getLength() const {
+      return cast<ConstantInt>(MemTransferInst::getLength());
+    }
+    // Methods for support type inquiry through isa, cast, and dyn_cast:
+    static bool classof(const IntrinsicInst *I) {
+      return I->getIntrinsicID() == Intrinsic::memcpy_inline;
+    }
+    static bool classof(const Value *V) {
+      return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
+    }
+  };
+
   // The common base class for any memset/memmove/memcpy intrinsics;
   // whether they be atomic or non-atomic.
   // i.e. llvm.element.unordered.atomic.memset/memcpy/memmove
   //  and llvm.memset/memcpy/memmove
   class AnyMemIntrinsic : public MemIntrinsicBase<AnyMemIntrinsic> {
   public:
     bool isVolatile() const {
       // Only the non-atomic intrinsics can be volatile
       if (auto *MI = dyn_cast<MemIntrinsic>(this))
         return MI->isVolatile();
       return false;
     }
 

llvm/include/llvm/IR/Intrinsics.td

def int_instrprof_value_profile : Intrinsic<[],
                                             []>;
 
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                              [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                               llvm_i1_ty],
                             [IntrArgMemOnly, IntrWillReturn, NoCapture<0>, NoCapture<1>,
                              NoAlias<0>, NoAlias<1>, WriteOnly<0>, ReadOnly<1>, ImmArg<3>]>;
+
+// Memcpy semantic that is guaranteed to be inlined.
+// In particular this means that the generated code is not allowed to call any
+// external function.
+// The third argument (specifying the size) must be a constant.
+def int_memcpy_inline
+    : Intrinsic<[],
+      [ llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty, llvm_i1_ty ],
+      [ IntrArgMemOnly, NoCapture<0>, NoCapture<1>, WriteOnly<0>, ReadOnly<1>,
+      ImmArg<2>, ImmArg<3> ]>;
+
 def int_memmove : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                              llvm_i1_ty],
                             [IntrArgMemOnly, IntrWillReturn, NoCapture<0>, NoCapture<1>,
                              ReadOnly<1>, ImmArg<3>]>;
 def int_memset  : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_i8_ty, llvm_anyint_ty,
                              llvm_i1_ty],

llvm/lib/Analysis/Lint.cpp

case Intrinsic::memcpy: {
                                               /*OffsetOk=*/false)))
         if (Len->getValue().isIntN(32))
           Size = LocationSize::precise(Len->getValue().getZExtValue());
       Assert(AA->alias(MCI->getSource(), Size, MCI->getDest(), Size) !=
                  MustAlias,
              "Undefined behavior: memcpy source and destination overlap", &I);
       break;
     }
+    case Intrinsic::memcpy_inline: {
+      MemCpyInlineInst *MCII = cast<MemCpyInlineInst>(&I);
+      const ConstantInt *Len = cast<ConstantInt>(findValue(MCII->getLength(),
+                                                           /*OffsetOk=*/false));
+      const APInt &APLen = Len->getValue();
+      assert(APLen.isIntN(64) && "@llvm.memcpy.inline size must be uint64");
+      const uint64_t Size = APLen.getZExtValue();
+      visitMemoryReference(I, MCII->getDest(), Size, MCII->getDestAlignment(),
+                           nullptr, MemRef::Write);
+      visitMemoryReference(I, MCII->getSource(), Size,
+                           MCII->getSourceAlignment(), nullptr, MemRef::Read);
+
+      // Check that the memcpy arguments don't overlap. The AliasAnalysis API
+      // isn't expressive enough for what we really want to do. Known partial
+      // overlap is not distinguished from the case where nothing is known.
+      const LocationSize LS = LocationSize::precise(Size);
+      Assert(AA->alias(MCII->getSource(), LS, MCII->getDest(), LS) != MustAlias,
+             "Undefined behavior: memcpy source and destination overlap", &I);
+      break;
+    }
     case Intrinsic::memmove: {
       MemMoveInst *MMI = cast<MemMoveInst>(&I);
       // TODO: If the size is known, use it.
       visitMemoryReference(I, MMI->getDest(), MemoryLocation::UnknownSize,
                            MMI->getDestAlignment(), nullptr, MemRef::Write);
       visitMemoryReference(I, MMI->getSource(), MemoryLocation::UnknownSize,
                            MMI->getSourceAlignment(), nullptr, MemRef::Read);
       break;

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::memcpy: {
     unsigned SrcAlign = std::max<unsigned>(MCI.getSourceAlignment(), 1);
     unsigned Align = MinAlign(DstAlign, SrcAlign);
     bool isVol = MCI.isVolatile();
     bool isTC = I.isTailCall() && isInTailCallPosition(&I, DAG.getTarget());
     // FIXME: Support passing different dest/src alignments to the memcpy DAG
     // node.
     SDValue Root = isVol ? getRoot() : getMemoryRoot();
     SDValue MC = DAG.getMemcpy(Root, sdl, Op1, Op2, Op3, Align, isVol,
-                               false, isTC,
+                               /* AlwaysInline */ false, isTC,
                                MachinePointerInfo(I.getArgOperand(0)),
                                MachinePointerInfo(I.getArgOperand(1)));
     updateDAGForMaybeTailCall(MC);
     return;
   }
+  case Intrinsic::memcpy_inline: {
+    const auto &MCI = cast<MemCpyInlineInst>(I);
+    SDValue Dst = getValue(I.getArgOperand(0));
+    SDValue Src = getValue(I.getArgOperand(1));
+    SDValue Size = getValue(I.getArgOperand(2));
+    assert(isa<ConstantSDNode>(Size) && "memcpy_inline needs constant size");
+    // @llvm.memcpy.inline defines 0 and 1 to both mean no alignment.
+    Align DstAlign = MCI.getDestAlign().valueOrOne();
+    Align SrcAlign = MCI.getSourceAlign().valueOrOne();
+    Align Alignment = commonAlignment(DstAlign, SrcAlign);
+    bool isVol = MCI.isVolatile();
+    bool isTC = I.isTailCall() && isInTailCallPosition(&I, DAG.getTarget());
+    // FIXME: Support passing different dest/src alignments to the memcpy DAG
+    // node.
+    SDValue MC = DAG.getMemcpy(
+        getRoot(), sdl, Dst, Src, Size, Alignment.value(), isVol,
+        /* AlwaysInline */ true, isTC, MachinePointerInfo(I.getArgOperand(0)),
+        MachinePointerInfo(I.getArgOperand(1)));
+    updateDAGForMaybeTailCall(MC);
+    return;
+  }
   case Intrinsic::memset: {
     const auto &MSI = cast<MemSetInst>(I);
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     // @llvm.memset defines 0 and 1 to both mean no alignment.
     unsigned Align = std::max<unsigned>(MSI.getDestAlignment(), 1);
     bool isVol = MSI.isVolatile();

llvm/lib/IR/Verifier.cpp

case Intrinsic::dbg_addr: // llvm.dbg.addr
     break;
   case Intrinsic::dbg_value: // llvm.dbg.value
     visitDbgIntrinsic("value", cast<DbgVariableIntrinsic>(Call));
     break;
   case Intrinsic::dbg_label: // llvm.dbg.label
     visitDbgLabelIntrinsic("label", cast<DbgLabelInst>(Call));
     break;
   case Intrinsic::memcpy:
+  case Intrinsic::memcpy_inline:
   case Intrinsic::memmove:
   case Intrinsic::memset: {
     const auto *MI = cast<MemIntrinsic>(&Call);
     auto IsValidAlignment = [&](unsigned Alignment) -> bool {
       return Alignment == 0 || isPowerOf2_32(Alignment);
     };
     Assert(IsValidAlignment(MI->getDestAlignment()),
            "alignment of arg 0 of memory intrinsic must be 0 or a power of 2",

llvm/test/CodeGen/X86/memcpy-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=core2 | FileCheck %s
+
+; NOTE: This fails to compile on platform without memcpy
+; XFAIL: llc < %s -mtriple=r600-unknown-linux-gnu
+
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
+declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
+
+define void @test1(i8* %a, i8* %b) nounwind {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq (%rsi), %rax
+; CHECK-NEXT:    movq %rax, (%rdi)
+; CHECK-NEXT:    retq
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* %a, i8* %b, i64 8, i1 0 )
+  ret void
+}
+
+define void @regular_memcpy_calls_external_function(i8* %a, i8* %b) nounwind {
+; CHECK-LABEL: regular_memcpy_calls_external_function:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl $128, %edx
+; CHECK-NEXT:    jmp memcpy # TAILCALL
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %a, i8* %b, i64 128, i1 0 )
+  ret void
+}
+
+define void @inlined_copy_doesnt_call_external_function(i8* %a, i8* %b) nounwind {
+; CHECK-LABEL: inlined_copy_doesnt_call_external_function:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl $128, %ecx
+; CHECK-NEXT:    rep;movsb (%rsi), %es:(%rdi)
+; CHECK-NEXT:    retq
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* %a, i8* %b, i64 128, i1 0 )
+  ret void
+}

llvm/test/Other/lint.ll

 ; RUN: opt -basicaa -lint -disable-output < %s 2>&1 | FileCheck %s
 target datalayout = "e-p:64:64:64"
 
 declare fastcc void @bar()
 declare void @llvm.stackrestore(i8*)
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
+declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
 declare void @has_sret(i8* sret %p)
 declare void @has_noaliases(i32* noalias %p, i32* %q)
 declare void @one_arg(i32)
 
 @CG = constant i32 7
 @CG2 = constant i32 7
 @E = external global i8
 
; CHECK: Call argument count mismatches callee argument count
   call void (i32, i32) bitcast (void (i32)* @one_arg to void (i32, i32)*)(i32 0, i32 0)
 ; CHECK: Call argument count mismatches callee argument count
   call void () bitcast (void (i32)* @one_arg to void ()*)()
 ; CHECK: Call argument type mismatches callee parameter type
   call void (float) bitcast (void (i32)* @one_arg to void (float)*)(float 0.0)
 
 ; CHECK: Write to read-only memory
 call void @llvm.memcpy.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG2 to i8*), i64 1, i1 0)
+; CHECK: Write to read-only memory
+call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG2 to i8*), i64 1, i1 0)
 ; CHECK: Unusual: noalias argument aliases another argument
 call void @llvm.memcpy.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG to i8*), i64 1, i1 0)
 
 ; CHECK: Undefined behavior: Buffer overflow
   %wider = bitcast i8* %buf to i16*
   store i16 0, i16* %wider
 ; CHECK: Undefined behavior: Buffer overflow
   %inner = getelementptr {i8, i8}, {i8, i8}* %buf2, i32 0, i32 1

 declare i32 @nonstruct_callee() nounwind
 define void @struct_caller() nounwind {
 entry:
   call %struct bitcast (i32 ()* @foo to %struct ()*)()
 
   ; CHECK: Undefined behavior: indirectbr with no destinations
   indirectbr i8* null, []
 }
+
+define i32 @memcpy_inline_same_address() noreturn {
+  %buf = alloca i64, align 1
+  %ptr = bitcast i64* %buf to i8*
+  ; CHECK: Undefined behavior: memcpy source and destination overlap
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* %ptr, i8* %ptr, i64 1, i1 false)
+  unreachable
+}

llvm/test/Verifier/intrinsic-immarg.ll

 define void @memcpy(i8* %dest, i8* %src, i1 %is.volatile) {
   ; CHECK: immarg operand has non-immediate parameter
   ; CHECK-NEXT: i1 %is.volatile
   ; CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
   ret void
 }
 
+declare void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* nocapture, i8* nocapture, i32, i1)
+define void @memcpy_inline_is_volatile(i8* %dest, i8* %src, i1 %is.volatile) {
+  ; CHECK: immarg operand has non-immediate parameter
+  ; CHECK-NEXT: i1 %is.volatile
+  ; CHECK-NEXT: call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
+  ret void
+}
+
+define void @memcpy_inline_variable_size(i8* %dest, i8* %src, i32 %size) {
+  ; CHECK: immarg operand has non-immediate parameter
+  ; CHECK-NEXT: i32 %size
+  ; CHECK-NEXT: call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 %size, i1 true)
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 %size, i1 true)
+  ret void
+}
+
 declare void @llvm.memmove.p0i8.p0i8.i32(i8* nocapture, i8* nocapture, i32, i1)
 define void @memmove(i8* %dest, i8* %src, i1 %is.volatile) {
   ; CHECK: immarg operand has non-immediate parameter
   ; CHECK-NEXT: i1 %is.volatile
   ; CHECK-NEXT: call void @llvm.memmove.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
   call void @llvm.memmove.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 8, i1 %is.volatile)
   ret void
 }

llvm/test/Verifier/memcpy-inline.ll

+; RUN: not opt -verify < %s 2>&1 | FileCheck %s
+
+; CHECK: alignment is not a power of two 
+
+define void @foo(i8* %P, i8* %Q) {
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* align 3 %P, i8* %Q, i32 4, i1 false)
+  ret void
+}
+declare void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* nocapture, i8* nocapture, i32, i1) nounwind