Specify the access behaviour of the memcpy, memmove and memset intrinsics

[aadg]

Summary

memcpy, memmove : set volatile on the source or destination operand, not on the intrinsic itself.

This means that memcpy and memove will now have 2 parameters : isSrcVolatile and isDestVolatile in place of isVolatile. For now, the old 'isVolatile' parameter is duplicated to form the is(Src|Dest)Volatile parameters in order to
preserve the actual behaviour.

The isVolatile and setVolatile method are still there for compatibility reasons, but should disappear as soon as all users have moved to the new interface.

Those changes requires some clang tests to be updated accordingly (see http://llvm-reviews.chandlerc.com/D376)

The update of all IR tests to use the new format is moved to a separate patch, as it is quite big and mechanical.

No functional change. The code in LLVM or Clang using the isVolatile part will be updated in subsequent patches. It is expected to bring some improvement as the old behaviour prevented some optimizations to kick-in.

[aadg]

Rebase + add a fix in AutoUpgrade : memcpy and memove do not return values, so they can not be named

[aadg]

Rebase to tip of the day.

[chandlerc]

Sorry for the long delays, and thanks for reminding me about this. Comments below.

include/llvm/IR/IntrinsicInst.h

135–144

I would just sink this into the specific intrinsic classes so that we don't have to get the intrinsic ID and switch over it here.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

4472–4474

Update to use the new APIs?

4510–4512

Here as well.

lib/Target/ARM/ARMFastISel.cpp

2292

This seems like a moderately horrible interface. I would much prefer to use the types to tell what to do than to count the volatile flags.

[aadg]

I am fixing the patch right now and will resubmit it soon.

include/llvm/IR/IntrinsicInst.h

135–144

I aggree if we consider this method will stay --- but I think this method should just disappear as it has a dubious utility given the new interface. It is just kept with minimal changes for not breaking existing code and will be removed in a subsequent patch : all users of isVolatile have to take proper actions if the volatile attribute is set on src or dst.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

4472–4474

I adapted to the surrounding style, which does not use interface methods. But using them is definitely clearer. And cleaner. Will fix the patch.

4510–4512

Will do.

lib/Target/ARM/ARMFastISel.cpp

2292

Will fix it.

[aadg]

Rebase + address Chandler's comments

The isVolatile method is not as clean as it could be, but knowing that it will disappear in the next commits, I do not think it is worth modifying the different classes to provide a nicer transient interface.

[aadg]

Fix ARMFastISel/SelectCall calls.

I just found out I was not building the ARM target and did not caught this trivial error :(.

[chandlerc]

Sorry this got dropped.

If you're still interested in getting it in, please update the patch and re-add me as a reviewer and I'll take a look.

docs/LangRef.rst

 
 This is an overloaded intrinsic. You can use ``llvm.memcpy`` on any
 integer bit width and for different address spaces. Not all targets
 support all bit widths however.
 
 ::
 
       declare void @llvm.memcpy.p0i8.p0i8.i32(i8* <dest>, i8* <src>,
-                                              i32 <len>, i32 <align>, i1 <isvolatile>)
+                                              i32 <len>, i32 <align>,
+                                              i1 <isdestvolatile>, i1 <issrcvolatile>)
       declare void @llvm.memcpy.p0i8.p0i8.i64(i8* <dest>, i8* <src>,
-                                              i64 <len>, i32 <align>, i1 <isvolatile>)
+                                              i64 <len>, i32 <align>,
+                                              i1 <isdestvolatile>, i1 <issrcvolatile>)
 
 Overview:
 """""""""
 
 The '``llvm.memcpy.*``' intrinsics copy a block of memory from the
 source location to the destination location.
 
 Note that, unlike the standard libc function, the ``llvm.memcpy.*``
-intrinsics do not return a value, takes extra alignment/isvolatile
-arguments and the pointers can be in specified address spaces.
+intrinsics do not return a value, takes extra
+alignment/isdestvolatile/issrcvolatile arguments and the pointers can be
+in specified address spaces.
 
 Arguments:
 """"""""""
 
 The first argument is a pointer to the destination, the second is a
 pointer to the source. The third argument is an integer argument
 specifying the number of bytes to copy, the fourth argument is the
 alignment of the source and destination locations, and the fifth is a
 boolean indicating a volatile access.
 
 If the call to this intrinsic has an alignment value that is not 0 or 1,
 then the caller guarantees that both the source and destination pointers
 are aligned to that boundary.
 
-If the ``isvolatile`` parameter is ``true``, the ``llvm.memcpy`` call is
-a :ref:`volatile operation <volatile>`. The detailed access behavior is not
-very cleanly specified and it is unwise to depend on it.
+If the ``isdestvolatile`` parameter is ``true``, the '``llvm.memcpy.*``' call
+will perform :ref:`volatile accesses <volatile>` to the destination location.
+If the ``issrcvolatile`` parameter is ``true``, the '``llvm.memcpy.*``' call
+will perform :ref:`volatile accesses <volatile>` to the source location.
 
 Semantics:
 """"""""""
 
 The '``llvm.memcpy.*``' intrinsics copy a block of memory from the
 source location to the destination location, which are not allowed to
 overlap. It copies "len" bytes of memory over. If the argument is known
 to be aligned to some boundary, this can be specified as the fourth
 argument, otherwise it should be set to 0 or 1.
 
+In case of volatile accesses, it is guaranteed that each byte loaded is loaded
+exactly once when ``issrcvolatile`` is ``true`` and that each byte stored is
+stored exactly once when ``isdestvolatile`` is ``true``, but these operations
+are free to be batched into 4-byte, 8-byte, or any other width of loads and
+stores. There is no guarantee about the interleaving of these loads and stores
+even when both are volatile.
+
 '``llvm.memmove``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 This is an overloaded intrinsic. You can use llvm.memmove on any integer
 bit width and for different address space. Not all targets support all
 bit widths however.
 
 ::
 
       declare void @llvm.memmove.p0i8.p0i8.i32(i8* <dest>, i8* <src>,
-                                               i32 <len>, i32 <align>, i1 <isvolatile>)
+                                               i32 <len>, i32 <align>,
+                                               i1 <isdestvolatile>, i1 <issrcvolatile>)
       declare void @llvm.memmove.p0i8.p0i8.i64(i8* <dest>, i8* <src>,
-                                               i64 <len>, i32 <align>, i1 <isvolatile>)
+                                               i64 <len>, i32 <align>,
+                                               i1 <isvolatile>, i1 <issrcvolatile>)
 
 Overview:
 """""""""
 
 The '``llvm.memmove.*``' intrinsics move a block of memory from the
 source location to the destination location. It is similar to the
 '``llvm.memcpy``' intrinsic but allows the two memory locations to
 overlap.
 
 Note that, unlike the standard libc function, the ``llvm.memmove.*``
-intrinsics do not return a value, takes extra alignment/isvolatile
-arguments and the pointers can be in specified address spaces.
+intrinsics do not return a value, takes extra
+alignment/isdestvolatile/issrcvolatile arguments and the pointers can be
+in specified address spaces.
 
 Arguments:
 """"""""""
 
 The first argument is a pointer to the destination, the second is a
 pointer to the source. The third argument is an integer argument
 specifying the number of bytes to copy, the fourth argument is the
 alignment of the source and destination locations, and the fifth is a
 boolean indicating a volatile access.
 
 If the call to this intrinsic has an alignment value that is not 0 or 1,
 then the caller guarantees that the source and destination pointers are
 aligned to that boundary.
 
-If the ``isvolatile`` parameter is ``true``, the ``llvm.memmove`` call
-is a :ref:`volatile operation <volatile>`. The detailed access behavior is
-not very cleanly specified and it is unwise to depend on it.
+If the ``isdestvolatile`` parameter is ``true``, the '``llvm.memmove.*``' call
+will perform :ref:`volatile accesses <volatile>` to the destination location.
+If the ``issrcvolatile`` parameter is ``true``, the '``llvm.memmove.*``' call
+will perform :ref:`volatile accesses <volatile>` to the source location.
 
 Semantics:
 """"""""""
 
 The '``llvm.memmove.*``' intrinsics copy a block of memory from the
 source location to the destination location, which may overlap. It
 copies "len" bytes of memory over. If the argument is known to be
 aligned to some boundary, this can be specified as the fourth argument,
 otherwise it should be set to 0 or 1.
 
+In case of volatile accesses, it is guaranteed that each byte loaded is loaded
+exactly once when ``issrcvolatile`` is ``true`` and that each byte stored is
+stored exactly once when ``isdestvolatile`` is ``true``, but these operations
+are free to be batched into 4-byte, 8-byte, or any other width of loads and
+stores. There is no guarantee about the interleaving of these loads and stores
+even when both are volatile.
+
 '``llvm.memset.*``' Intrinsics
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 This is an overloaded intrinsic. You can use llvm.memset on any integer
 bit width and for different address spaces. However, not all targets
 support all bit widths.
 
 ::
 
       declare void @llvm.memset.p0i8.i32(i8* <dest>, i8 <val>,
-                                         i32 <len>, i32 <align>, i1 <isvolatile>)
+                                         i32 <len>, i32 <align>, i1 <isdestvolatile>)
       declare void @llvm.memset.p0i8.i64(i8* <dest>, i8 <val>,
-                                         i64 <len>, i32 <align>, i1 <isvolatile>)
+                                         i64 <len>, i32 <align>, i1 <isdestvolatile>)
 
 Overview:
 """""""""
 
 The '``llvm.memset.*``' intrinsics fill a block of memory with a
 particular byte value.
 
 Note that, unlike the standard libc function, the ``llvm.memset``
 intrinsic does not return a value and takes extra alignment/volatile
 arguments. Also, the destination can be in an arbitrary address space.
 
 Arguments:
 """"""""""
 
 The first argument is a pointer to the destination to fill, the second
 is the byte value with which to fill it, the third argument is an
 integer argument specifying the number of bytes to fill, and the fourth
 argument is the known alignment of the destination location.
 
 If the call to this intrinsic has an alignment value that is not 0 or 1,
 then the caller guarantees that the destination pointer is aligned to
 that boundary.
 
-If the ``isvolatile`` parameter is ``true``, the ``llvm.memset`` call is
-a :ref:`volatile operation <volatile>`. The detailed access behavior is not
-very cleanly specified and it is unwise to depend on it.
+If the ``isdestvolatile`` parameter is ``true``, the ``llvm.memset`` call is
+a :ref:`volatile operation <volatile>`.
 
 Semantics:
 """"""""""
 
 The '``llvm.memset.*``' intrinsics fill "len" bytes of memory starting
 at the destination location. If the argument is known to be aligned to
 some boundary, this can be specified as the fourth argument, otherwise
 it should be set to 0 or 1.
 
+In case of volatile accesses, it is guaranteed that each byte stored is stored
+exactly once, and these stores are free to be batched into 4-byte, 8-byte,
+or any other width of stores.
+
 '``llvm.sqrt.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 This is an overloaded intrinsic. You can use ``llvm.sqrt`` on any
 floating point or vector of floating point type. Not all targets support

include/llvm/IR/IntrinsicInst.h

public:
     ConstantInt *getAlignmentCst() const {
       return cast<ConstantInt>(const_cast<Value*>(getArgOperand(3)));
     }
 
     unsigned getAlignment() const {
       return getAlignmentCst()->getZExtValue();
     }
 
-    ConstantInt *getVolatileCst() const {
-      return cast<ConstantInt>(const_cast<Value*>(getArgOperand(4)));
+    bool isDestVolatile() const {
+      return !cast<ConstantInt>(const_cast<Value*>(getArgOperand(4)))->isZero();
     }
+
     bool isVolatile() const {
-      return !getVolatileCst()->isZero();
+      switch (getIntrinsicID()) {
+        case Intrinsic::memcpy:
+        case Intrinsic::memmove:
+          return isDestVolatile() ||
+            !cast<ConstantInt>(const_cast<Value*>(getArgOperand(5)))->isZero();
+        default:
+          return isDestVolatile();
+      }
     }
 
     unsigned getDestAddressSpace() const {
       return cast<PointerType>(getRawDest()->getType())->getAddressSpace();
     }
 
     /// getDest - This is just like getRawDest, but it strips off any cast
     /// instructions that feed it, giving the original input.  The returned
     /// value is guaranteed to be a pointer.
void setLength(Value *L) {
              "setLength called with value of wrong type!");
       setArgOperand(2, L);
     }
 
     void setAlignment(Constant* A) {
       setArgOperand(3, A);
     }
 
-    void setVolatile(Constant* V) {
+    void setDestVolatile(Constant* V) {
       setArgOperand(4, V);
     }
 
+    void setVolatile(Constant* V) {
+      setDestVolatile(V);
+      if (getIntrinsicID()==Intrinsic::memcpy ||
+          getIntrinsicID()==Intrinsic::memmove) {
+        setArgOperand(5, V);
+      }
+    }
+
     Type *getAlignmentType() const {
       return getArgOperand(3)->getType();
     }
 
     // Methods for support type inquiry through isa, cast, and dyn_cast:
     static inline bool classof(const IntrinsicInst *I) {
       switch (I->getIntrinsicID()) {
       case Intrinsic::memcpy:
public:
     static inline bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };
 
   /// MemTransferInst - This class wraps the llvm.memcpy/memmove intrinsics.
   ///
   class MemTransferInst : public MemIntrinsic {
+
   public:
     /// get* - Return the arguments to the instruction.
     ///
     Value *getRawSource() const { return const_cast<Value*>(getArgOperand(1)); }
 
     /// getSource - This is just like getRawSource, but it strips off any cast
     /// instructions that feed it, giving the original input.  The returned
     /// value is guaranteed to be a pointer.
     Value *getSource() const { return getRawSource()->stripPointerCasts(); }
 
     unsigned getSourceAddressSpace() const {
       return cast<PointerType>(getRawSource()->getType())->getAddressSpace();
     }
 
     void setSource(Value *Ptr) {
       assert(getRawSource()->getType() == Ptr->getType() &&
              "setSource called with pointer of wrong type!");
       setArgOperand(1, Ptr);
     }
 
+    bool isSourceVolatile() const {
+      return !cast<ConstantInt>(const_cast<Value*>(getArgOperand(5)))->isZero();
+    }
+
+    void setSourceVolatile(Constant* V) {
+      setArgOperand(5, V);
+    }
+
     // Methods for support type inquiry through isa, cast, and dyn_cast:
     static inline bool classof(const IntrinsicInst *I) {
       return I->getIntrinsicID() == Intrinsic::memcpy ||
              I->getIntrinsicID() == Intrinsic::memmove;
     }
     static inline bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }

include/llvm/IR/Intrinsics.td

 // guard to the correct place on the stack frame.
 def int_stackprotector : Intrinsic<[], [llvm_ptr_ty, llvm_ptrptr_ty], []>;
 
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                              [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
-                              llvm_i32_ty, llvm_i1_ty],
+                              llvm_i32_ty, llvm_i1_ty, llvm_i1_ty],
                             [IntrReadWriteArgMem, NoCapture<0>, NoCapture<1>]>;
 def int_memmove : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
-                             llvm_i32_ty, llvm_i1_ty],
+                             llvm_i32_ty, llvm_i1_ty, llvm_i1_ty],
                             [IntrReadWriteArgMem, NoCapture<0>, NoCapture<1>]>;
 def int_memset  : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_i8_ty, llvm_anyint_ty,
                              llvm_i32_ty, llvm_i1_ty],
                             [IntrReadWriteArgMem, NoCapture<0>]>;
 
 // These functions do not actually read memory, but they are sensitive to the
 // rounding mode.  This needs to be modelled separately; in the meantime

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

assert(cast<PointerType>(I.getArgOperand(0)->getType())->getAddressSpace()
            < 256 &&
            "Unknown address space");
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     unsigned Align = cast<ConstantInt>(I.getArgOperand(3))->getZExtValue();
     if (!Align)
       Align = 1; // @llvm.memcpy defines 0 and 1 to both mean no alignment.
-    bool isVol = cast<ConstantInt>(I.getArgOperand(4))->getZExtValue();
+    const MemCpyInst &mcpy = cast<MemCpyInst>(I);
+    bool isVol = mcpy.isSourceVolatile() || mcpy.isDestVolatile();
     DAG.setRoot(DAG.getMemcpy(getRoot(), dl, Op1, Op2, Op3, Align, isVol, false,
                               MachinePointerInfo(I.getArgOperand(0)),
                               MachinePointerInfo(I.getArgOperand(1))));
     return 0;
   }
   case Intrinsic::memset: {
     // Assert for address < 256 since we support only user defined address
     // spaces.
     assert(cast<PointerType>(I.getArgOperand(0)->getType())->getAddressSpace()
            < 256 &&
            "Unknown address space");
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     unsigned Align = cast<ConstantInt>(I.getArgOperand(3))->getZExtValue();
     if (!Align)
       Align = 1; // @llvm.memset defines 0 and 1 to both mean no alignment.
-    bool isVol = cast<ConstantInt>(I.getArgOperand(4))->getZExtValue();
+    bool isVol = cast<MemSetInst>(I).isDestVolatile();
     DAG.setRoot(DAG.getMemset(getRoot(), dl, Op1, Op2, Op3, Align, isVol,
                               MachinePointerInfo(I.getArgOperand(0))));
     return 0;
   }
   case Intrinsic::memmove: {
     // Assert for address < 256 since we support only user defined address
     // spaces.
     assert(cast<PointerType>(I.getArgOperand(0)->getType())->getAddressSpace()
            < 256 &&
            cast<PointerType>(I.getArgOperand(1)->getType())->getAddressSpace()
            < 256 &&
            "Unknown address space");
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     unsigned Align = cast<ConstantInt>(I.getArgOperand(3))->getZExtValue();
     if (!Align)
       Align = 1; // @llvm.memmove defines 0 and 1 to both mean no alignment.
-    bool isVol = cast<ConstantInt>(I.getArgOperand(4))->getZExtValue();
+    const MemMoveInst &mm = cast<MemMoveInst>(I);
+    bool isVol = mm.isSourceVolatile() || mm.isDestVolatile();
     DAG.setRoot(DAG.getMemmove(getRoot(), dl, Op1, Op2, Op3, Align, isVol,
                                MachinePointerInfo(I.getArgOperand(0)),
                                MachinePointerInfo(I.getArgOperand(1))));
     return 0;
   }
   case Intrinsic::dbg_declare: {
     const DbgDeclareInst &DI = cast<DbgDeclareInst>(I);
     MDNode *Variable = DI.getVariable();
     const Value *Address = DI.getAddress();

lib/IR/AutoUpgrade.cpp

case 'c': {
     if (Name.startswith("cttz.") && F->arg_size() == 1) {
       F->setName(Name + ".old");
       NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::cttz,
                                         F->arg_begin()->getType());
       return true;
     }
     break;
   }
+  case 'm': {
+    if (Name.startswith("memcpy.") && F->arg_size() == 5) {
+      F->setName(Name + ".old");
+      const FunctionType* FT = F->getFunctionType();
+      Type *Tys[] = { FT->getParamType(0), FT->getParamType(1), FT->getParamType(2) };
+      NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::memcpy, Tys);
+      return true;
+    }
+    if (Name.startswith("memmove.") && F->arg_size() == 5) {
+      F->setName(Name + ".old");
+      const FunctionType* FT = F->getFunctionType();
+      Type *Tys[] = { FT->getParamType(0), FT->getParamType(1), FT->getParamType(2) };
+      NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::memmove, Tys);
+      return true;
+    }
+    break;
+  }
   case 'x': {
     if (Name.startswith("x86.sse2.pcmpeq.") ||
         Name.startswith("x86.sse2.pcmpgt.") ||
         Name.startswith("x86.avx2.pcmpeq.") ||
         Name.startswith("x86.avx2.pcmpgt.") ||
         Name.startswith("x86.avx.vpermil.") ||
         Name == "x86.avx.movnt.dq.256" ||
         Name == "x86.avx.movnt.pd.256" ||
if (!NewFn) {
     }
 
     CI->replaceAllUsesWith(Rep);
     CI->eraseFromParent();
     return;
   }
 
   std::string Name = CI->getName().str();
+  if (!CI->getType()->isVoidTy())
     CI->setName(Name + ".old");
 
   switch (NewFn->getIntrinsicID()) {
   default:
     llvm_unreachable("Unknown function for CallInst upgrade.");
 
   case Intrinsic::ctlz:
   case Intrinsic::cttz:
     assert(CI->getNumArgOperands() == 1 &&
            "Mismatch between function args and call args");
     CI->replaceAllUsesWith(Builder.CreateCall2(NewFn, CI->getArgOperand(0),
                                                Builder.getFalse(), Name));
     CI->eraseFromParent();
     return;
 
+  case Intrinsic::memcpy:
+  case Intrinsic::memmove:
+    if (CI->getNumArgOperands() == 5) {
+      Value *Args[] = {
+        CI->getArgOperand(0),
+        CI->getArgOperand(1),
+        CI->getArgOperand(2),
+        CI->getArgOperand(3),
+        CI->getArgOperand(4), // isDestVolatile
+        CI->getArgOperand(4)  // isSrcVolatile
+      };
+
+      CallInst *NCI = Builder.CreateCall(NewFn, Args, Name);
+
+      NCI->setTailCall( CI->isTailCall());
+
+      if (CI->hasMetadata()) {
+        SmallVector<std::pair<unsigned, MDNode *>, 4> MDs;
+        CI->getAllMetadata(MDs);
+        for (SmallVectorImpl<std::pair<unsigned, MDNode *> >::iterator
+             MI = MDs.begin(), ME = MDs.end(); MI != ME; ++MI)
+          NCI->setMetadata(MI->first, MI->second);
+      }
+
+      CI->replaceAllUsesWith(NCI);
+      CI->eraseFromParent();
+    }
+    return;
+
   case Intrinsic::arm_neon_vclz: {
     // Change name from llvm.arm.neon.vclz.* to llvm.ctlz.*
     CI->replaceAllUsesWith(Builder.CreateCall2(NewFn, CI->getArgOperand(0),
                                                Builder.getFalse(),
                                                "llvm.ctlz." + Name.substr(14)));
     CI->eraseFromParent();
     return;
   }

lib/IR/IRBuilder.cpp

 }
 
 CallInst *IRBuilderBase::
 CreateMemCpy(Value *Dst, Value *Src, Value *Size, unsigned Align,
              bool isVolatile, MDNode *TBAATag, MDNode *TBAAStructTag) {
   Dst = getCastedInt8PtrValue(Dst);
   Src = getCastedInt8PtrValue(Src);
 
-  Value *Ops[] = { Dst, Src, Size, getInt32(Align), getInt1(isVolatile) };
+  Value *Ops[] = { Dst, Src, Size, getInt32(Align),
+                   getInt1(isVolatile), getInt1(isVolatile) };
   Type *Tys[] = { Dst->getType(), Src->getType(), Size->getType() };
   Module *M = BB->getParent()->getParent();
   Value *TheFn = Intrinsic::getDeclaration(M, Intrinsic::memcpy, Tys);
   
   CallInst *CI = createCallHelper(TheFn, Ops, this);
   
   // Set the TBAA info if present.
   if (TBAATag)
     CI->setMetadata(LLVMContext::MD_tbaa, TBAATag);
 
   // Set the TBAA Struct info if present.
   if (TBAAStructTag)
     CI->setMetadata(LLVMContext::MD_tbaa_struct, TBAAStructTag);
   
   return CI;  
 }
 
 CallInst *IRBuilderBase::
 CreateMemMove(Value *Dst, Value *Src, Value *Size, unsigned Align,
               bool isVolatile, MDNode *TBAATag) {
   Dst = getCastedInt8PtrValue(Dst);
   Src = getCastedInt8PtrValue(Src);
   
-  Value *Ops[] = { Dst, Src, Size, getInt32(Align), getInt1(isVolatile) };
+  Value *Ops[] = { Dst, Src, Size, getInt32(Align),
+                   getInt1(isVolatile), getInt1(isVolatile) };
   Type *Tys[] = { Dst->getType(), Src->getType(), Size->getType() };
   Module *M = BB->getParent()->getParent();
   Value *TheFn = Intrinsic::getDeclaration(M, Intrinsic::memmove, Tys);
   
   CallInst *CI = createCallHelper(TheFn, Ops, this);
   
   // Set the TBAA info if present.
   if (TBAATag)

lib/IR/Verifier.cpp

void Verifier::visitIntrinsicFunctionCall(Intrinsic::ID ID, CallInst &CI) {
   } break;
   case Intrinsic::memcpy:
   case Intrinsic::memmove:
   case Intrinsic::memset:
     Assert1(isa<ConstantInt>(CI.getArgOperand(3)),
             "alignment argument of memory intrinsics must be a constant int",
             &CI);
     Assert1(isa<ConstantInt>(CI.getArgOperand(4)),
-            "isvolatile argument of memory intrinsics must be a constant int",
+            "isdestvolatile argument of memory intrinsics must be a constant int",
+            &CI);
+    if (ID!=Intrinsic::memset)
+      Assert1( isa<ConstantInt>(CI.getArgOperand(5)),
+              "issrcvolatile argument of memory intrinsics must be a constant int",
               &CI);
     break;
   case Intrinsic::gcroot:
   case Intrinsic::gcwrite:
   case Intrinsic::gcread:
     if (ID == Intrinsic::gcroot) {
       AllocaInst *AI =
         dyn_cast<AllocaInst>(CI.getArgOperand(0)->stripPointerCasts());
       Assert1(AI, "llvm.gcroot parameter #1 must be an alloca.", &CI);

lib/Target/ARM/ARMFastISel.cpp

bool ARMFastISel::ARMEmitLibcall(const Instruction *I, RTLIB::Libcall Call) {
   // Set all unused physreg defs as dead.
   static_cast<MachineInstr *>(MIB)->setPhysRegsDeadExcept(UsedRegs, TRI);
 
   return true;
 }
 
 bool ARMFastISel::SelectCall(const Instruction *I,
                              const char *IntrMemName = 0) {
   const CallInst *CI = cast<CallInst>(I);
   const Value *Callee = CI->getCalledValue();
 
   // Can't handle inline asm.
   if (isa<InlineAsm>(Callee)) return false;
 
   // Allow SelectionDAG isel to handle tail calls.
   if (CI->isTailCall()) return false;
 
bool ARMFastISel::SelectCall(const Instruction *I,
   unsigned arg_size = CS.arg_size();
   Args.reserve(arg_size);
   ArgRegs.reserve(arg_size);
   ArgVTs.reserve(arg_size);
   ArgFlags.reserve(arg_size);
   for (ImmutableCallSite::arg_iterator i = CS.arg_begin(), e = CS.arg_end();
        i != e; ++i) {
     // If we're lowering a memory intrinsic instead of a regular call, skip the
-    // last two arguments, which shouldn't be passed to the underlying function.
-    if (IntrMemName && e-i <= 2)
+    // last arguments, which shouldn't be passed to the underlying function.
+    if (IntrMemName) {
+      const unsigned numVolatileSpec = isa<MemTransferInst>(I) ? 2 : 1;
+      if ( e-i <= 1+numVolatileSpec)
         break;
+    }
 
     ISD::ArgFlagsTy Flags;
     unsigned AttrInd = i - CS.arg_begin() + 1;
     if (CS.paramHasAttr(AttrInd, Attribute::SExt))
       Flags.setSExt();
     if (CS.paramHasAttr(AttrInd, Attribute::ZExt))
       Flags.setZExt();
 

lib/Transforms/Utils/InlineFunction.cpp

Size = ConstantInt::get(Type::getInt64Ty(Context),
                             IFI.TD->getTypeStoreSize(AggTy));
   
   // Always generate a memcpy of alignment 1 here because we don't know
   // the alignment of the src pointer.  Other optimizations can infer
   // better alignment.
   Value *CallArgs[] = {
     DestCast, SrcCast, Size,
     ConstantInt::get(Type::getInt32Ty(Context), 1),
-    ConstantInt::getFalse(Context) // isVolatile
+    ConstantInt::getFalse(Context), // isDestVolatile
+    ConstantInt::getFalse(Context)  // isSrcVolatile
   };
   IRBuilder<>(TheCall).CreateCall(MemCpyFn, CallArgs);
   
   // Uses of the argument in the function should use our new alloca
   // instead.
   return NewAlloca;
 }
 

test/Analysis/TypeBasedAliasAnalysis/memcpyopt.ll

 ; RUN: opt -S -tbaa -basicaa -memcpyopt -instcombine < %s | FileCheck %s
 
 target datalayout = "e-p:64:64:64"
 
 ; The second memcpy is redundant and can be deleted. There's an intervening store, but
 ; it has a TBAA tag which declares that it is unrelated.
 
 ; CHECK: @foo
-; CHECK-NEXT: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %p, i8* %q, i64 16, i32 1, i1 false), !tbaa !0
+; CHECK-NEXT: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %p, i8* %q, i64 16, i32 1, i1 false, i1 false), !tbaa !0
 ; CHECK-NEXT: store i8 2, i8* %s, align 1, !tbaa !2
 ; CHECK-NEXT: ret void
 define void @foo(i8* nocapture %p, i8* nocapture %q, i8* nocapture %s) nounwind {
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %p, i8* %q, i64 16, i32 1, i1 false), !tbaa !2
   store i8 2, i8* %s, align 1, !tbaa !1
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %q, i8* %p, i64 16, i32 1, i1 false), !tbaa !2
   ret void
 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i32, i1) nounwind
 
 !0 = metadata !{metadata !"tbaa root", null}
 !1 = metadata !{metadata !"A", metadata !0}
 !2 = metadata !{metadata !"B", metadata !0}

test/Assembler/auto_upgrade_intrinsics.ll

entry:
   call i16 @llvm.cttz.i16(i16 %b)
   ; CHECK: call i32 @llvm.cttz.i32(i32 %c, i1 false)
   call i32 @llvm.cttz.i32(i32 %c)
   ; CHECK: call i42 @llvm.cttz.i42(i42 %d, i1 false)
   call i42 @llvm.cttz.i42(i42 %d)
 
   ret void
 }
+
+define void @test.memcpy(i8* %dst, i8* %src, i64 %len) {
+; CHECK: @test.memcpy
+
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64( i8* %dst, i8* %src, i64 %len, i32 1, i1 false)
+  ; CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %len, i32 1, i1 false, i1 false)
+
+  ret void
+}
+
+define void @test.memmove(i8* %dst, i8* %src, i64 %len) {
+; CHECK: @test.memmove
+
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64( i8* %dst, i8* %src, i64 %len, i32 1, i1 false)
+  ; CHECK: call void @llvm.memmove.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %len, i32 1, i1 false, i1 false)
+
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i64( i8* %dst, i8* %src, i64 %len, i32, i1)
+; CHECK: declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i32, i1, i1)
+
+declare void @llvm.memmove.p0i8.p0i8.i64( i8* %dst, i8* %src, i64 %len, i32, i1)
+; CHECK: declare void @llvm.memmove.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i32, i1, i1)

test/Transforms/DeadStoreElimination/OverwriteStoreEnd.ll

; CHECK: call void @llvm.memset.p0i8.i64(i8* %p3, i8 0, i64 32, i32 16, i1 false)
   store i32 1, i32* %arrayidx1, align 4
   ret void
 }
 
 define void @write32to36(%struct.vec2plusi* nocapture %p) nounwind uwtable ssp {
 ; CHECK: @write32to36
 entry:
   %0 = bitcast %struct.vec2plusi* %p to i8*
-; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2plusi* @glob2 to i8*), i64 32, i32 16, i1 false)
+; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2plusi* @glob2 to i8*), i64 32, i32 16, i1 false, i1 false)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2plusi* @glob2 to i8*), i64 36, i32 16, i1 false)
   %c = getelementptr inbounds %struct.vec2plusi* %p, i64 0, i32 2
   store i32 1, i32* %c, align 4
   ret void
 }
 
 define void @write16to32(%struct.vec2* nocapture %p) nounwind uwtable ssp {
 ; CHECK: @write16to32
 entry:
   %0 = bitcast %struct.vec2* %p to i8*
-; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 16, i32 16, i1 false)
+; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 16, i32 16, i1 false, i1 false)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 32, i32 16, i1 false)
   %c = getelementptr inbounds %struct.vec2* %p, i64 0, i32 1
   store <4 x i32> <i32 1, i32 2, i32 3, i32 4>, <4 x i32>* %c, align 4
   ret void
 }
 
 define void @dontwrite28to32memcpy(%struct.vec2* nocapture %p) nounwind uwtable ssp {
 ; CHECK: @dontwrite28to32memcpy
 entry:
   %0 = bitcast %struct.vec2* %p to i8*
-; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 32, i32 16, i1 false)
+; CHECK: tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 32, i32 16, i1 false, i1 false)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* bitcast (%struct.vec2* @glob1 to i8*), i64 32, i32 16, i1 false)
   %arrayidx1 = getelementptr inbounds %struct.vec2* %p, i64 0, i32 0, i64 7
   store i32 1, i32* %arrayidx1, align 4
   ret void
 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i32, i1) nounwind
 declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i32, i1) nounwind
 
 %struct.trapframe = type { i64, i64, i64 }
 
 ; bugzilla 11455 - make sure negative GEP's don't break this optimisation
 ; CHECK: @cpu_lwp_fork
 define void @cpu_lwp_fork(%struct.trapframe* %md_regs, i64 %pcb_rsp0) nounwind uwtable noinline ssp {
 entry:
   %0 = inttoptr i64 %pcb_rsp0 to %struct.trapframe*
   %add.ptr = getelementptr inbounds %struct.trapframe* %0, i64 -1
   %1 = bitcast %struct.trapframe* %add.ptr to i8*
   %2 = bitcast %struct.trapframe* %md_regs to i8*
-; CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* %1, i8* %2, i64 24, i32 1, i1 false)
+; CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* %1, i8* %2, i64 24, i32 1, i1 false, i1 false)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %1, i8* %2, i64 24, i32 1, i1 false)
   %tf_trapno = getelementptr inbounds %struct.trapframe* %0, i64 -1, i32 1
   store i64 3, i64* %tf_trapno, align 8
   ret void
 }

test/Transforms/InstCombine/objsize.ll

 
 define i8* @test5(i32 %n) nounwind ssp {
 ; CHECK: @test5
 entry:
   %0 = tail call noalias i8* @malloc(i32 20) nounwind
   %1 = tail call i32 @llvm.objectsize.i32(i8* %0, i1 false)
   %2 = load i8** @s, align 8
 ; CHECK-NOT: @llvm.objectsize
-; CHECK: @llvm.memcpy.p0i8.p0i8.i32(i8* %0, i8* %1, i32 10, i32 1, i1 false)
+; CHECK: @llvm.memcpy.p0i8.p0i8.i32(i8* %0, i8* %1, i32 10, i32 1, i1 false, i1 false)
   %3 = tail call i8* @__memcpy_chk(i8* %0, i8* %2, i32 10, i32 %1) nounwind
   ret i8* %0
 }
 
 define void @test6(i32 %n) nounwind ssp {
 ; CHECK: @test6
 entry:
   %0 = tail call noalias i8* @malloc(i32 20) nounwind

test/Transforms/InstCombine/sprintf-1.ll

 declare i32 @sprintf(i8*, i8*, ...)
 
 ; Check sprintf(dst, fmt) -> llvm.memcpy(str, fmt, strlen(fmt) + 1, 1).
 
 define void @test_simplify1(i8* %dst) {
 ; CHECK: @test_simplify1
   %fmt = getelementptr [13 x i8]* @hello_world, i32 0, i32 0
   call i32 (i8*, i8*, ...)* @sprintf(i8* %dst, i8* %fmt)
-; CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* getelementptr inbounds ([13 x i8]* @hello_world, i32 0, i32 0), i32 13, i32 1, i1 false)
+; CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* getelementptr inbounds ([13 x i8]* @hello_world, i32 0, i32 0), i32 13, i32 1, i1 false, i1 false)
   ret void
 ; CHECK-NEXT: ret void
 }
 
 define void @test_simplify2(i8* %dst) {
 ; CHECK: @test_simplify2
   %fmt = getelementptr [1 x i8]* @null, i32 0, i32 0
   call i32 (i8*, i8*, ...)* @sprintf(i8* %dst, i8* %fmt)

 ; Check sprintf(dst, "%s", str) -> llvm.memcpy(dest, str, strlen(str) + 1, 1).
 
 define void @test_simplify5(i8* %dst, i8* %str) {
 ; CHECK: @test_simplify5
   %fmt = getelementptr [3 x i8]* @percent_s, i32 0, i32 0
   call i32 (i8*, i8*, ...)* @sprintf(i8* %dst, i8* %fmt, i8* %str)
 ; CHECK-NEXT: [[STRLEN:%[a-z0-9]+]] = call i32 @strlen(i8* %str)
 ; CHECK-NEXT: [[LENINC:%[a-z0-9]+]] = add i32 [[STRLEN]], 1
-; CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %str, i32 [[LENINC]], i32 1, i1 false)
+; CHECK-NEXT: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %str, i32 [[LENINC]], i32 1, i1 false, i1 false)
   ret void
 ; CHECK-NEXT: ret void
 }
 
 ; Check sprintf(dst, format, ...) -> siprintf(str, format, ...) if no floating.
 
 define void @test_simplify6(i8* %dst) {
 ; CHECK-IPRINTF: @test_simplify6

test/Transforms/LoopIdiom/basic.ll

for.body:                                         ; preds = %bb.nph, %for.body
   store i8 %V, i8* %DestI, align 1
   %indvar.next = add i64 %indvar, 1
   %exitcond = icmp eq i64 %indvar.next, %Size
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 ; CHECK: @test6
-; CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* %Dest, i8* %Base, i64 %Size, i32 1, i1 false)
+; CHECK: call void @llvm.memcpy.p0i8.p0i8.i64(i8* %Dest, i8* %Base, i64 %Size, i32 1, i1 false, i1 false)
 ; CHECK-NOT: store
 ; CHECK: ret void
 }
 
 
 ; This is a loop that was rotated but where the blocks weren't merged.  This
 ; shouldn't perturb us.
 define void @test7(i8* %Base, i64 %Size) nounwind ssp {

test/Transforms/MemCpyOpt/align.ll

; CHECK: call void @llvm.memset.p0i8.i64(i8* {{.*}}, i8 0, i64 16, i32 4, i1 false)
   ret void
 }
 
 ; Replacing %a8 with %a4 in the memset requires boosting the alignment of %a4.
 
 define void @bar() {
 ; CHECK: @bar
 ; CHECK: %a4 = alloca i32, align 8
-; CHECK-NOT: memcpy
+; CHECK-NOT: call void @llvm.memcpy
   %a4 = alloca i32, align 4
   %a8 = alloca i32, align 8
   %a8.cast = bitcast i32* %a8 to i8*
   %a4.cast = bitcast i32* %a4 to i8*
   call void @llvm.memset.p0i8.i64(i8* %a8.cast, i8 0, i64 4, i32 8, i1 false)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %a4.cast, i8* %a8.cast, i64 4, i32 4, i1 false)
   ret void
 }

test/Transforms/MemCpyOpt/smaller.ll

 ; RUN: opt -memcpyopt -S < %s | FileCheck %s
 ; rdar://8875553
 
 ; Memcpyopt shouldn't optimize the second memcpy using the first
 ; because the first has a smaller size.
 
-; CHECK: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %tmp, i8* getelementptr inbounds (%struct.s* @cell, i32 0, i32 0, i32 0), i32 16, i32 4, i1 false)
+; CHECK: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %tmp, i8* getelementptr inbounds (%struct.s* @cell, i32 0, i32 0, i32 0), i32 16, i32 4, i1 false, i1 false)
 
 target datalayout = "e-p:32:32:32"
 
 %struct.s = type { [11 x i8], i32 }
 
 @.str = private constant [11 x i8] c"0123456789\00"
 @cell = external global %struct.s
 

test/Verifier/2006-12-12-IntrinsicDefine.ll

 ; RUN: not llvm-as < %s 2>&1 | FileCheck %s
 ; CHECK: llvm intrinsics cannot be defined
 ; PR1047
 
-define void @llvm.memcpy.p0i8.p0i8.i32(i8*, i8*, i32, i32, i1) {
+define void @llvm.memcpy.p0i8.p0i8.i32(i8*, i8*, i32, i32, i1, i1) {
 entry:
   ret void
 }