[GISel] Support llvm.memcpy.inline

[jroelofs]

llvm/test/MachineVerifier/test_g_memcpy.mir

16

delete me

[jroelofs]

• clang-format the patch
• Delete stray declaration
• Add IRTranslator test

[paquette]

llvm/include/llvm/IR/IntrinsicInst.h

895

Looks like this is used in a few passes too (e.g. the MemCpyOptimizer pass). Does this change impact those passes at all?

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Probably worth a comment for folks not familiar with the code.

It'd probably clearer to be able to call a inlineMemCpy function here and just return true, but I'm not sure if that's easy right now.

llvm/lib/CodeGen/MachineVerifier.cpp

1510

I think we only expect 0 or 1 for these values, so maybe we should check the value too?

[jroelofs]

llvm/include/llvm/IR/IntrinsicInst.h

895

Yes. Maybe I ought to put that in its own diff.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Good point. I'll give that a try.

llvm/lib/CodeGen/MachineVerifier.cpp

1510

Good call!

[aemerson]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39

If only memcpy has an inline variant then we shouldn't be generating G_MEMSETs with this extra argument?

[jroelofs]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39

I was on the fence about whether to let them all have the same number of arguments or not, since there are several places where code is shared between different opcodes and having them have different operand counts would lead to lots of special case logic.

To strike a balance, maybe it should only be memcpy+memmove instead?

[aemerson]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39

Why add to memmove though?

Another way to implement this is to just introduce a new G_MEMCPY_INLINE opcode, then we don't have to add another argument anywhere.

[arsenm]

llvm/include/llvm/IR/IntrinsicInst.h

895

Yes, this should be its own patch. I started doing this a few months ago but forgot about it

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Probably should use uint64_t?

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39

I also think this should be a separate opcode.

There's also the difference that memcpy.inline uses an immarg size argument (which I think is entirely pointless and should be removed, but that's a different battle)

[jroelofs]

Make a new opcode: G_MEMCPY_INLINE, instead of adding extra arguments to existing opcodes.

[aemerson]

Why change the existing memcpy test? I would have thought the logical way to test this was to use a G_MEMCPY_INLINE with a length much larger than the heuristics would normally allow for G_MEMCPY expansion, rather than change existing test coverage.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1347

Dynamically sized memcpy.inline isn't legal according to the langref, so we should probably assert here.

llvm/test/CodeGen/AArch64/GlobalISel/inline-memcpy-opt.mir

51 ↗

(On Diff #355417)

This IR section is probably unnecessary.

[jroelofs]

I'll rename the two tests to make it clear the existing one wasn't changed, but what happened is (roughly):

cp inline-memcpy.mir inline-memcpy-opt.mir
sed -e 's/G_MEMCPY/G_MEMCPY_INLINE/' -e 's/memcpy\./memcpy.inline/g' inline-memcpy.mir
update_mir_test_checks.py inline-memcpy.mir

[paquette]

Can you add an entry on G_MEMCPY_INLINE to docs/GlobalISel/GenericOpcode.rst?

[aemerson]

LGTM.

[dyung]

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

632 ↗

(On Diff #355663)

In the future, please use a regex to match any number instead of hard coding a number in here as it causes the test to fail if a downstream copy has private changes which affect which number is assigned to this opcode.

I've fixed this up in f737d9794a40c066c9ccffb9ac277c1b70442ede.

[jroelofs]

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

632 ↗

(On Diff #355663)

Sorry about that. Thank you!

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
 private:
   // Memcpy family optimization helpers.
   bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
                       unsigned KnownLen, Align DstAlign, Align SrcAlign,
-                      bool IsVolatile);
+                      bool IsVolatile, bool IsInlined);
   bool optimizeMemmove(MachineInstr &MI, Register Dst, Register Src,
                        unsigned KnownLen, Align DstAlign, Align SrcAlign,
                        bool IsVolatile);
   bool optimizeMemset(MachineInstr &MI, Register Dst, Register Val,
                       unsigned KnownLen, Align DstAlign, bool IsVolatile);
 
   /// Given a non-indexed load or store instruction \p MI, find an offset that
   /// can be usefully and legally folded into it as a post-indexing operation.

llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

   }
 
   /// Build and insert G_MEMCPY or G_MEMMOVE
   MachineInstrBuilder buildMemTransferInst(unsigned Opcode, const SrcOp &DstPtr,
                                            const SrcOp &SrcPtr,
                                            const SrcOp &Size,
                                            MachineMemOperand &DstMMO,
                                            MachineMemOperand &SrcMMO) {
-    auto MIB = buildInstr(
-        Opcode, {}, {DstPtr, SrcPtr, Size, SrcOp(INT64_C(0) /*isTailCall*/)});
+    auto MIB =
+        buildInstr(Opcode, {},
+                   {DstPtr, SrcPtr, Size, SrcOp(INT64_C(0) /*isTailCall*/),
+                    SrcOp(INT64_C(0) /*isInlined*/)});
     MIB.addMemOperand(&DstMMO);
     MIB.addMemOperand(&SrcMMO);
     return MIB;
   }
 
   MachineInstrBuilder buildMemCpy(const SrcOp &DstPtr, const SrcOp &SrcPtr,
                                   const SrcOp &Size, MachineMemOperand &DstMMO,
                                   MachineMemOperand &SrcMMO) {

llvm/include/llvm/IR/IntrinsicInst.h

default:
       return false;
     }
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
-/// This class wraps the llvm.memcpy intrinsic.
+/// This class wraps the llvm.memcpy and llvm.memcpy.inline intrinsics.
 class MemCpyInst : public MemTransferInst {
 public:
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
-    return I->getIntrinsicID() == Intrinsic::memcpy;
+    return I->getIntrinsicID() == Intrinsic::memcpy ||
+           I->getIntrinsicID() == Intrinsic::memcpy_inline;
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 /// This class wraps the llvm.memmove intrinsic.
 class MemMoveInst : public MemTransferInst {
 public:
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
     return I->getIntrinsicID() == Intrinsic::memmove;
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 /// This class wraps the llvm.memcpy.inline intrinsic.
-class MemCpyInlineInst : public MemTransferInst {
+class MemCpyInlineInst : public MemCpyInst {
 public:
   ConstantInt *getLength() const {
-    return cast<ConstantInt>(MemTransferInst::getLength());
+    return cast<ConstantInt>(MemCpyInst::getLength());
   }
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
     return I->getIntrinsicID() == Intrinsic::memcpy_inline;
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }

llvm/include/llvm/Target/GenericOpcodes.td

 def G_STRICT_FSQRT : ConstrainedIntruction<G_FSQRT>;
 
 //------------------------------------------------------------------------------
 // Memory intrinsics
 //------------------------------------------------------------------------------
 
 def G_MEMCPY : GenericInstruction {
   let OutOperandList = (outs);
-  let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall);
+  let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall, untyped_imm_0:$inlined);
   let hasSideEffects = false;
   let mayLoad = true;
   let mayStore = true;
 }
 
 def G_MEMMOVE : GenericInstruction {
   let OutOperandList = (outs);
-  let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall);
+  let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall, untyped_imm_0:$inlined);
   let hasSideEffects = false;
   let mayLoad = true;
   let mayStore = true;
 }
 
 def G_MEMSET : GenericInstruction {
   let OutOperandList = (outs);
-  let InOperandList = (ins ptype0:$dst_addr, type1:$value, type2:$size, untyped_imm_0:$tailcall);
+  let InOperandList = (ins ptype0:$dst_addr, type1:$value, type2:$size, untyped_imm_0:$tailcall, untyped_imm_0:$inlined);
   let hasSideEffects = false;
   let mayStore = true;
 }
 
 def G_BZERO : GenericInstruction {
   let OutOperandList = (outs);
-  let InOperandList = (ins ptype0:$dst_addr, type1:$size, untyped_imm_0:$tailcall);
+  let InOperandList = (ins ptype0:$dst_addr, type1:$size, untyped_imm_0:$tailcall, untyped_imm_0:$inlined);
   let hasSideEffects = false;
   let mayStore = true;
 }
 
 //------------------------------------------------------------------------------
 // Bitfield extraction.
 //------------------------------------------------------------------------------
 

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

 
   MI.eraseFromParent();
   return true;
 }
 
 bool CombinerHelper::optimizeMemcpy(MachineInstr &MI, Register Dst,
                                     Register Src, unsigned KnownLen,
                                     Align DstAlign, Align SrcAlign,
-                                    bool IsVolatile) {
+                                    bool IsVolatile, bool IsInlined) {
   auto &MF = *MI.getParent()->getParent();
   const auto &TLI = *MF.getSubtarget().getTargetLowering();
   auto &DL = MF.getDataLayout();
   LLVMContext &C = MF.getFunction().getContext();
 
   assert(KnownLen != 0 && "Have a zero length memcpy length!");
 
   bool DstAlignCanChange = false;
   MachineFrameInfo &MFI = MF.getFrameInfo();
   bool OptSize = shouldLowerMemFuncForSize(MF);
   Align Alignment = commonAlignment(DstAlign, SrcAlign);
 
   MachineInstr *FIDef = getOpcodeDef(TargetOpcode::G_FRAME_INDEX, Dst, MRI);
   if (FIDef && !MFI.isFixedObjectIndex(FIDef->getOperand(1).getIndex()))
     DstAlignCanChange = true;
 
   // FIXME: infer better src pointer alignment like SelectionDAG does here.
   // FIXME: also use the equivalent of isMemSrcFromConstant and alwaysinlining
   // if the memcpy is in a tail call position.
 
-  unsigned Limit = TLI.getMaxStoresPerMemcpy(OptSize);
+  unsigned Limit = IsInlined ? std::numeric_limits<unsigned>::max()
+                             : TLI.getMaxStoresPerMemcpy(OptSize);
   std::vector<LLT> MemOps;
 
   const auto &DstMMO = **MI.memoperands_begin();
   const auto &SrcMMO = **std::next(MI.memoperands_begin());
   MachinePointerInfo DstPtrInfo = DstMMO.getPointerInfo();
   MachinePointerInfo SrcPtrInfo = SrcMMO.getPointerInfo();
 
   if (!findGISelOptimalMemOpLowering(

   if (IsVolatile)
     return false;
 
   Align DstAlign = MemOp->getBaseAlign();
   Align SrcAlign;
   Register Dst = MI.getOperand(0).getReg();
   Register Src = MI.getOperand(1).getReg();
   Register Len = MI.getOperand(2).getReg();
+  bool IsInlined = MI.getOperand(4).getImm();
+  assert((!IsInlined || Opc == TargetOpcode::G_MEMCPY) &&
+         "currently, only memcpy supports inlining");
 
   if (Opc != TargetOpcode::G_MEMSET) {
     assert(MMOIt != MI.memoperands_end() && "Expected a second MMO on MI");
     MemOp = *(++MMOIt);
     SrcAlign = MemOp->getBaseAlign();
   }
 
   // See if this is a constant length copy
   auto LenVRegAndVal = getConstantVRegValWithLookThrough(Len, MRI);
   if (!LenVRegAndVal)
     return false; // Leave it to the legalizer to lower it to a libcall.
   unsigned KnownLen = LenVRegAndVal->Value.getZExtValue();
 
   if (KnownLen == 0) {
     MI.eraseFromParent();
     return true;
   }
 
-  if (MaxLen && KnownLen > MaxLen)
+  if (!IsInlined && MaxLen && KnownLen > MaxLen)
     return false;
 
   if (Opc == TargetOpcode::G_MEMCPY)
-    return optimizeMemcpy(MI, Dst, Src, KnownLen, DstAlign, SrcAlign, IsVolatile);
+    return optimizeMemcpy(MI, Dst, Src, KnownLen, DstAlign, SrcAlign,
+                          IsVolatile, IsInlined);
   if (Opc == TargetOpcode::G_MEMMOVE)
     return optimizeMemmove(MI, Dst, Src, KnownLen, DstAlign, SrcAlign, IsVolatile);
   if (Opc == TargetOpcode::G_MEMSET)
     return optimizeMemset(MI, Dst, Src, KnownLen, DstAlign, IsVolatile);
   return false;
 }
 
 static Optional<APFloat> constantFoldFpUnary(unsigned Opcode, LLT DstTy,

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

     DstAlign = MSI->getDestAlign().valueOrOne();
   }
 
   // We need to propagate the tail call flag from the IR inst as an argument.
   // Otherwise, we have to pessimize and assume later that we cannot tail call
   // any memory intrinsics.
   ICall.addImm(CI.isTailCall() ? 1 : 0);
 
+  // The "inline" part of llvm.memcpy.inline is also propagated as an argument.
+  ICall.addImm(isa<MemCpyInlineInst>(&CI) ? 1 : 0);
+
   // Create mem operands to store the alignment and volatile info.
   auto VolFlag = IsVol ? MachineMemOperand::MOVolatile : MachineMemOperand::MONone;
   ICall.addMemOperand(MF->getMachineMemOperand(
       MachinePointerInfo(CI.getArgOperand(0)),
       MachineMemOperand::MOStore | VolFlag, 1, DstAlign));
   if (Opcode != TargetOpcode::G_MEMSET)
     ICall.addMemOperand(MF->getMachineMemOperand(
         MachinePointerInfo(CI.getArgOperand(1)),

                           MachineInstr::copyFlagsFromInstruction(CI));
     return true;
   case Intrinsic::convert_to_fp16:
     // FIXME: This intrinsic should probably be removed from the IR.
     MIRBuilder.buildFPTrunc(getOrCreateVReg(CI),
                             getOrCreateVReg(*CI.getArgOperand(0)),
                             MachineInstr::copyFlagsFromInstruction(CI));
     return true;
+  case Intrinsic::memcpy_inline:
   case Intrinsic::memcpy:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMCPY);
   case Intrinsic::memmove:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMMOVE);
   case Intrinsic::memset:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMSET);
   case Intrinsic::eh_typeid_for: {
     GlobalValue *GV = ExtractTypeInfo(CI.getArgOperand(0));

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

 }
 
 LegalizerHelper::LegalizeResult
 llvm::createMemLibcall(MachineIRBuilder &MIRBuilder, MachineRegisterInfo &MRI,
                        MachineInstr &MI, LostDebugLocObserver &LocObserver) {
   auto &Ctx = MIRBuilder.getMF().getFunction().getContext();
 
   SmallVector<CallLowering::ArgInfo, 3> Args;
-  // Add all the args, except for the last which is an imm denoting 'tail'.
-  for (unsigned i = 0; i < MI.getNumOperands() - 1; ++i) {
+  // Add all the args, except for the last which is an imm denoting 'tail'
+  // and 'inlined' respectively.
+  for (unsigned i = 0; i < MI.getNumOperands() - 2; ++i) {
     Register Reg = MI.getOperand(i).getReg();
 
     // Need derive an IR type for call lowering.
     LLT OpLLT = MRI.getType(Reg);
     Type *OpTy = nullptr;
     if (OpLLT.isPointer())
       OpTy = Type::getInt8PtrTy(Ctx, OpLLT.getAddressSpace());
     else
       OpTy = IntegerType::get(Ctx, OpLLT.getSizeInBits());
     Args.push_back({Reg, OpTy});
LLVM_DEBUG(dbgs() << ".. .. Could not find libcall name for "
                       << MIRBuilder.getTII().getName(Opc) << "\n");
     return LegalizerHelper::UnableToLegalize;
   }
 
   CallLowering::CallLoweringInfo Info;
   Info.CallConv = TLI.getLibcallCallingConv(RTLibcall);
   Info.Callee = MachineOperand::CreateES(Name);
   Info.OrigRet = CallLowering::ArgInfo({0}, Type::getVoidTy(Ctx));
-  Info.IsTailCall = MI.getOperand(MI.getNumOperands() - 1).getImm() &&
+  Info.IsTailCall = MI.getOperand(MI.getNumOperands() - 2).getImm() &&
                     isLibCallInTailPosition(MIRBuilder.getTII(), MI);
+#ifndef NDEBUG
+  bool Inlined = MI.getOperand(MI.getNumOperands() - 1).getImm();
+  assert(!Inlined && "inlined memory op should already have been lowered");
+#endif
 
   std::copy(Args.begin(), Args.end(), std::back_inserter(Info.OrigArgs));
   if (!CLI.lowerCall(MIRBuilder, Info))
     return LegalizerHelper::UnableToLegalize;
 
 
   if (Info.LoweredTailCall) {
     assert(Info.IsTailCall && "Lowered tail call when it wasn't a tail call?");
case TargetOpcode::G_UITOFP: {
     if (Status != Legalized)
       return Status;
     break;
   }
   case TargetOpcode::G_BZERO:
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
+    assert((MI.getOpcode() != TargetOpcode::G_MEMCPY ||
+            MI.getOperand(4).getImm() == 0) &&
+           "llvm.memcpy.inline should already have been expanded before "
+           "legalization");
     LegalizeResult Result =
         createMemLibcall(MIRBuilder, *MIRBuilder.getMRI(), MI, LocObserver);
     if (Result != Legalized)
       return Result;
     MI.eraseFromParent();
     return Result;
   }
   }

llvm/lib/CodeGen/MachineVerifier.cpp

       break;
     }
 
     if (DstPtrTy.getAddressSpace() != MMOs[0]->getAddrSpace())
       report("inconsistent store address space", MI);
     if (SrcPtrTy.getAddressSpace() != MMOs[1]->getAddrSpace())
       report("inconsistent load address space", MI);
 
+    if (!MI->getOperand(3).isImm())
+      report("'tail' flag (operand 3) must be an immediate type", MI);
+
+    if (!MI->getOperand(4).isImm())
+      report("'inlined' flag (operand 4) must be an immediate type", MI);
+
     break;
   }
   case TargetOpcode::G_BZERO:
   case TargetOpcode::G_MEMSET: {
     ArrayRef<MachineMemOperand *> MMOs = MI->memoperands();
     std::string Name = Opc == TargetOpcode::G_MEMSET ? "memset" : "bzero";
     if (MMOs.size() != 1) {
       report(Twine(Name, " must have 1 memory operand"), MI);

     if (!DstPtrTy.isPointer()) {
       report(Twine(Name, " operand must be a pointer"), MI);
       break;
     }
 
     if (DstPtrTy.getAddressSpace() != MMOs[0]->getAddrSpace())
       report("inconsistent " + Twine(Name, " address space"), MI);
 
+    if (!MI->getOperand(MI->getNumOperands() - 2).isImm())
+      report("'tail' flag (second to last operand) must be an immediate type",
+             MI);
+
+    if (!MI->getOperand(MI->getNumOperands() - 1).isImm())
+      report("'inlined' flag (last operand) must be an immediate type", MI);
+
     break;
   }
   case TargetOpcode::G_VECREDUCE_SEQ_FADD:
   case TargetOpcode::G_VECREDUCE_SEQ_FMUL: {
     LLT DstTy = MRI->getType(MI->getOperand(0).getReg());
     LLT Src1Ty = MRI->getType(MI->getOperand(1).getReg());
     LLT Src2Ty = MRI->getType(MI->getOperand(2).getReg());
     if (!DstTy.isScalar())

llvm/lib/Target/AArch64/GISel/AArch64GlobalISelUtils.cpp

if (!MinSize) {
     }
   }
 
   MIRBuilder.setInstrAndDebugLoc(MI);
   MIRBuilder
       .buildInstr(TargetOpcode::G_BZERO, {},
                   {MI.getOperand(0), MI.getOperand(2)})
       .addImm(MI.getOperand(3).getImm())
+      .addImm(MI.getOperand(4).getImm())
       .addMemOperand(*MI.memoperands_begin());
   MI.eraseFromParent();
   return true;
 }
 
 void AArch64GISelUtils::changeFCMPPredToAArch64CC(
     const CmpInst::Predicate P, AArch64CC::CondCode &CondCode,
     AArch64CC::CondCode &CondCode2) {

llvm/lib/Target/AArch64/GISel/AArch64PreLegalizerCombiner.cpp

bool AArch64PreLegalizerCombinerInfo::combine(GISelChangeObserver &Observer,
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
     // If we're at -O0 set a maxlen of 32 to inline, otherwise let the other
-    // heuristics decide.
+    // heuristics decide (unless it is an llvm.memcpy.inline, of course).
     unsigned MaxLen = EnableOpt ? 0 : 32;
     // Try to inline memcpy type calls if optimizations are enabled.
     if (Helper.tryCombineMemCpyFamily(MI, MaxLen))
       return true;
     if (Opc == TargetOpcode::G_MEMSET)
       return llvm::AArch64GISelUtils::tryEmitBZero(MI, B, EnableMinSize);
     return false;
   }

llvm/test/CodeGen/AArch64/GlobalISel/arm64-irtranslator.ll

 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8*, i8*, i64, i1)
 define void @test_memcpy(i8* %dst, i8* %src, i64 %size) {
 ; CHECK-LABEL: name: test_memcpy
 ; CHECK: [[DST:%[0-9]+]]:_(p0) = COPY $x0
 ; CHECK: [[SRC:%[0-9]+]]:_(p0) = COPY $x1
 ; CHECK: [[SIZE:%[0-9]+]]:_(s64) = COPY $x2
-; CHECK: G_MEMCPY [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+; CHECK: G_MEMCPY [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 0, 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %size, i1 0)
   ret void
 }
 
 define void @test_memcpy_tail(i8* %dst, i8* %src, i64 %size) {
 ; CHECK-LABEL: name: test_memcpy_tail
 ; CHECK: [[DST:%[0-9]+]]:_(p0) = COPY $x0
 ; CHECK: [[SRC:%[0-9]+]]:_(p0) = COPY $x1
 ; CHECK: [[SIZE:%[0-9]+]]:_(s64) = COPY $x2
-; CHECK: G_MEMCPY [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 1 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+; CHECK: G_MEMCPY [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 1, 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %size, i1 0)
   ret void
 }
 
 declare void @llvm.memcpy.p1i8.p1i8.i64(i8 addrspace(1)*, i8 addrspace(1)*, i64, i1)
 define void @test_memcpy_nonzero_as(i8 addrspace(1)* %dst, i8 addrspace(1) * %src, i64 %size) {
 ; CHECK-LABEL: name: test_memcpy_nonzero_as
 ; CHECK: [[DST:%[0-9]+]]:_(p1) = COPY $x0
 ; CHECK: [[SRC:%[0-9]+]]:_(p1) = COPY $x1
 ; CHECK: [[SIZE:%[0-9]+]]:_(s64) = COPY $x2
-; CHECK: G_MEMCPY [[DST]](p1), [[SRC]](p1), [[SIZE]](s64), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 1)
+; CHECK: G_MEMCPY [[DST]](p1), [[SRC]](p1), [[SIZE]](s64), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 1)
   call void @llvm.memcpy.p1i8.p1i8.i64(i8 addrspace(1)* %dst, i8 addrspace(1)* %src, i64 %size, i1 0)
   ret void
 }
 
 declare void @llvm.memmove.p0i8.p0i8.i64(i8*, i8*, i64, i1)
 define void @test_memmove(i8* %dst, i8* %src, i64 %size) {
 ; CHECK-LABEL: name: test_memmove
 ; CHECK: [[DST:%[0-9]+]]:_(p0) = COPY $x0
 ; CHECK: [[SRC:%[0-9]+]]:_(p0) = COPY $x1
 ; CHECK: [[SIZE:%[0-9]+]]:_(s64) = COPY $x2
-; CHECK: G_MEMMOVE [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+; CHECK: G_MEMMOVE [[DST]](p0), [[SRC]](p0), [[SIZE]](s64), 0, 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
   call void @llvm.memmove.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %size, i1 0)
   ret void
 }
 
 declare void @llvm.memset.p0i8.i64(i8*, i8, i64, i1)
 define void @test_memset(i8* %dst, i8 %val, i64 %size) {
 ; CHECK-LABEL: name: test_memset
 ; CHECK: [[DST:%[0-9]+]]:_(p0) = COPY $x0
 ; CHECK: [[SRC_C:%[0-9]+]]:_(s32) = COPY $w1
 ; CHECK: [[SRC:%[0-9]+]]:_(s8) = G_TRUNC [[SRC_C]]
 ; CHECK: [[SIZE:%[0-9]+]]:_(s64) = COPY $x2
-; CHECK: G_MEMSET [[DST]](p0), [[SRC]](s8), [[SIZE]](s64), 0 :: (store 1 into %ir.dst)
+; CHECK: G_MEMSET [[DST]](p0), [[SRC]](s8), [[SIZE]](s64), 0, 0 :: (store 1 into %ir.dst)
   call void @llvm.memset.p0i8.i64(i8* %dst, i8 %val, i64 %size, i1 0)
   ret void
 }
 
 define void @test_large_const(i128* %addr) {
 ; CHECK-LABEL: name: test_large_const
 ; CHECK: [[ADDR:%[0-9]+]]:_(p0) = COPY $x0
 ; CHECK: [[VAL:%[0-9]+]]:_(s128) = G_CONSTANT i128 42

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

 name:            snork
 alignment:       4
 tracksRegLiveness: true
 body:             |
   bb.0:
     %0:_(p0) = G_IMPLICIT_DEF debug-location !DILocation(line: 0, scope: !6)
     %1:_(s8) = G_CONSTANT i8 0
     %2:_(s64) = G_IMPLICIT_DEF debug-location !DILocation(line: 0, scope: !6)
-    G_MEMSET %0(p0), %1(s8), %2(s64), 1, debug-location !11 :: (store 1)
+    G_MEMSET %0(p0), %1(s8), %2(s64), 1, 0, debug-location !11 :: (store 1)
     DBG_VALUE 0, 0, !9, !DIExpression(), debug-location !12
     RET_ReallyLR debug-location !12
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/inline-memcpy.mir

body:             |
   bb.1.entry:
     liveins: $x0, $x1, $x2
 
     ; CHECK-LABEL: name: test_memcpy1
     ; CHECK: liveins: $x0, $x1, $x2
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[COPY2:%[0-9]+]]:_(s64) = COPY $x2
-    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[COPY2]](s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[COPY2]](s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = COPY $x2
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memcpy2_const
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_memcpy2_const
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
     ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
-    ; CHECK: [[GEP:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
-    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[GEP]](p0) :: (load 16 from %ir.1 + 16, align 4)
-    ; CHECK: [[GEP1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
-    ; CHECK: G_STORE [[LOAD1]](s128), [[GEP1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
-    ; CHECK: [[GEP2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
-    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[GEP2]](p0) :: (load 16 from %ir.1 + 32, align 4)
-    ; CHECK: [[GEP3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
-    ; CHECK: G_STORE [[LOAD2]](s128), [[GEP3]](p0) :: (store 16 into %ir.0 + 32, align 4)
+    ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
+    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD2]](p0) :: (load 16 from %ir.1 + 32, align 4)
+    ; CHECK: [[PTR_ADD3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
+    ; CHECK: G_STORE [[LOAD2]](s128), [[PTR_ADD3]](p0) :: (store 16 into %ir.0 + 32, align 4)
     ; CHECK: [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
-    ; CHECK: [[GEP4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
-    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[GEP4]](p0) :: (load 16 from %ir.1 + 48, align 4)
-    ; CHECK: [[GEP5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
-    ; CHECK: G_STORE [[LOAD3]](s128), [[GEP5]](p0) :: (store 16 into %ir.0 + 48, align 4)
+    ; CHECK: [[PTR_ADD4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
+    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD4]](p0) :: (load 16 from %ir.1 + 48, align 4)
+    ; CHECK: [[PTR_ADD5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
+    ; CHECK: G_STORE [[LOAD3]](s128), [[PTR_ADD5]](p0) :: (store 16 into %ir.0 + 48, align 4)
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
-    ; CHECK: [[GEP6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
-    ; CHECK: [[LOAD4:%[0-9]+]]:_(s64) = G_LOAD [[GEP6]](p0) :: (load 8 from %ir.1 + 64, align 4)
-    ; CHECK: [[GEP7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
-    ; CHECK: G_STORE [[LOAD4]](s64), [[GEP7]](p0) :: (store 8 into %ir.0 + 64, align 4)
+    ; CHECK: [[PTR_ADD6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
+    ; CHECK: [[LOAD4:%[0-9]+]]:_(s64) = G_LOAD [[PTR_ADD6]](p0) :: (load 8 from %ir.1 + 64, align 4)
+    ; CHECK: [[PTR_ADD7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
+    ; CHECK: G_STORE [[LOAD4]](s64), [[PTR_ADD7]](p0) :: (store 8 into %ir.0 + 64, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 72
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memcpy2_const_optsize
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_memcpy2_const_optsize
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
     ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
-    ; CHECK: [[GEP:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
-    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[GEP]](p0) :: (load 16 from %ir.1 + 16, align 4)
-    ; CHECK: [[GEP1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
-    ; CHECK: G_STORE [[LOAD1]](s128), [[GEP1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
-    ; CHECK: [[GEP2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
-    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[GEP2]](p0) :: (load 16 from %ir.1 + 32, align 4)
-    ; CHECK: [[GEP3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
-    ; CHECK: G_STORE [[LOAD2]](s128), [[GEP3]](p0) :: (store 16 into %ir.0 + 32, align 4)
+    ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
+    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD2]](p0) :: (load 16 from %ir.1 + 32, align 4)
+    ; CHECK: [[PTR_ADD3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
+    ; CHECK: G_STORE [[LOAD2]](s128), [[PTR_ADD3]](p0) :: (store 16 into %ir.0 + 32, align 4)
     ; CHECK: [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
-    ; CHECK: [[GEP4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
-    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[GEP4]](p0) :: (load 16 from %ir.1 + 48, align 4)
-    ; CHECK: [[GEP5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
-    ; CHECK: G_STORE [[LOAD3]](s128), [[GEP5]](p0) :: (store 16 into %ir.0 + 48, align 4)
+    ; CHECK: [[PTR_ADD4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
+    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD4]](p0) :: (load 16 from %ir.1 + 48, align 4)
+    ; CHECK: [[PTR_ADD5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
+    ; CHECK: G_STORE [[LOAD3]](s128), [[PTR_ADD5]](p0) :: (store 16 into %ir.0 + 48, align 4)
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
-    ; CHECK: [[GEP6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
-    ; CHECK: [[LOAD4:%[0-9]+]]:_(s64) = G_LOAD [[GEP6]](p0) :: (load 8 from %ir.1 + 64, align 4)
-    ; CHECK: [[GEP7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
-    ; CHECK: G_STORE [[LOAD4]](s64), [[GEP7]](p0) :: (store 8 into %ir.0 + 64, align 4)
+    ; CHECK: [[PTR_ADD6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
+    ; CHECK: [[LOAD4:%[0-9]+]]:_(s64) = G_LOAD [[PTR_ADD6]](p0) :: (load 8 from %ir.1 + 64, align 4)
+    ; CHECK: [[PTR_ADD7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
+    ; CHECK: G_STORE [[LOAD4]](s64), [[PTR_ADD7]](p0) :: (store 8 into %ir.0 + 64, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 72
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memcpy2_const_minsize
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_memcpy2_const_minsize
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 72
-    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 72
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
-...
 ---
 name:            test_memcpy3_const_arrays_unaligned
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_memcpy3_const_arrays_unaligned
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
     ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
-    ; CHECK: [[GEP:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
-    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[GEP]](p0) :: (load 16 from %ir.1 + 16, align 4)
-    ; CHECK: [[GEP1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
-    ; CHECK: G_STORE [[LOAD1]](s128), [[GEP1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
-    ; CHECK: [[GEP2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
-    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[GEP2]](p0) :: (load 16 from %ir.1 + 32, align 4)
-    ; CHECK: [[GEP3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
-    ; CHECK: G_STORE [[LOAD2]](s128), [[GEP3]](p0) :: (store 16 into %ir.0 + 32, align 4)
+    ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
+    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD2]](p0) :: (load 16 from %ir.1 + 32, align 4)
+    ; CHECK: [[PTR_ADD3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
+    ; CHECK: G_STORE [[LOAD2]](s128), [[PTR_ADD3]](p0) :: (store 16 into %ir.0 + 32, align 4)
     ; CHECK: [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
-    ; CHECK: [[GEP4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
-    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[GEP4]](p0) :: (load 16 from %ir.1 + 48, align 4)
-    ; CHECK: [[GEP5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
-    ; CHECK: G_STORE [[LOAD3]](s128), [[GEP5]](p0) :: (store 16 into %ir.0 + 48, align 4)
+    ; CHECK: [[PTR_ADD4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
+    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD4]](p0) :: (load 16 from %ir.1 + 48, align 4)
+    ; CHECK: [[PTR_ADD5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
+    ; CHECK: G_STORE [[LOAD3]](s128), [[PTR_ADD5]](p0) :: (store 16 into %ir.0 + 48, align 4)
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
-    ; CHECK: [[GEP6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
-    ; CHECK: [[LOAD4:%[0-9]+]]:_(s128) = G_LOAD [[GEP6]](p0) :: (load 16 from %ir.1 + 64, align 4)
-    ; CHECK: [[GEP7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
-    ; CHECK: G_STORE [[LOAD4]](s128), [[GEP7]](p0) :: (store 16 into %ir.0 + 64, align 4)
+    ; CHECK: [[PTR_ADD6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
+    ; CHECK: [[LOAD4:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD6]](p0) :: (load 16 from %ir.1 + 64, align 4)
+    ; CHECK: [[PTR_ADD7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
+    ; CHECK: G_STORE [[LOAD4]](s128), [[PTR_ADD7]](p0) :: (store 16 into %ir.0 + 64, align 4)
     ; CHECK: [[C4:%[0-9]+]]:_(s64) = G_CONSTANT i64 80
-    ; CHECK: [[GEP8:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C4]](s64)
-    ; CHECK: [[LOAD5:%[0-9]+]]:_(s128) = G_LOAD [[GEP8]](p0) :: (load 16 from %ir.1 + 80, align 4)
-    ; CHECK: [[GEP9:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C4]](s64)
-    ; CHECK: G_STORE [[LOAD5]](s128), [[GEP9]](p0) :: (store 16 into %ir.0 + 80, align 4)
+    ; CHECK: [[PTR_ADD8:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C4]](s64)
+    ; CHECK: [[LOAD5:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD8]](p0) :: (load 16 from %ir.1 + 80, align 4)
+    ; CHECK: [[PTR_ADD9:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C4]](s64)
+    ; CHECK: G_STORE [[LOAD5]](s128), [[PTR_ADD9]](p0) :: (store 16 into %ir.0 + 80, align 4)
     ; CHECK: [[C5:%[0-9]+]]:_(s64) = G_CONSTANT i64 96
-    ; CHECK: [[GEP10:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C5]](s64)
-    ; CHECK: [[LOAD6:%[0-9]+]]:_(s128) = G_LOAD [[GEP10]](p0) :: (load 16 from %ir.1 + 96, align 4)
-    ; CHECK: [[GEP11:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C5]](s64)
-    ; CHECK: G_STORE [[LOAD6]](s128), [[GEP11]](p0) :: (store 16 into %ir.0 + 96, align 4)
+    ; CHECK: [[PTR_ADD10:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C5]](s64)
+    ; CHECK: [[LOAD6:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD10]](p0) :: (load 16 from %ir.1 + 96, align 4)
+    ; CHECK: [[PTR_ADD11:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C5]](s64)
+    ; CHECK: G_STORE [[LOAD6]](s128), [[PTR_ADD11]](p0) :: (store 16 into %ir.0 + 96, align 4)
     ; CHECK: [[C6:%[0-9]+]]:_(s64) = G_CONSTANT i64 112
-    ; CHECK: [[GEP12:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C6]](s64)
-    ; CHECK: [[LOAD7:%[0-9]+]]:_(s128) = G_LOAD [[GEP12]](p0) :: (load 16 from %ir.1 + 112, align 4)
-    ; CHECK: [[GEP13:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C6]](s64)
-    ; CHECK: G_STORE [[LOAD7]](s128), [[GEP13]](p0) :: (store 16 into %ir.0 + 112, align 4)
+    ; CHECK: [[PTR_ADD12:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C6]](s64)
+    ; CHECK: [[LOAD7:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD12]](p0) :: (load 16 from %ir.1 + 112, align 4)
+    ; CHECK: [[PTR_ADD13:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C6]](s64)
+    ; CHECK: G_STORE [[LOAD7]](s128), [[PTR_ADD13]](p0) :: (store 16 into %ir.0 + 112, align 4)
     ; CHECK: [[C7:%[0-9]+]]:_(s64) = G_CONSTANT i64 127
-    ; CHECK: [[GEP14:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C7]](s64)
-    ; CHECK: [[LOAD8:%[0-9]+]]:_(s128) = G_LOAD [[GEP14]](p0) :: (load 16 from %ir.1 + 127, align 1, basealign 4)
-    ; CHECK: [[GEP15:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C7]](s64)
-    ; CHECK: G_STORE [[LOAD8]](s128), [[GEP15]](p0) :: (store 16 into %ir.0 + 127, align 1, basealign 4)
+    ; CHECK: [[PTR_ADD14:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C7]](s64)
+    ; CHECK: [[LOAD8:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD14]](p0) :: (load 16 from %ir.1 + 127, align 1, basealign 4)
+    ; CHECK: [[PTR_ADD15:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C7]](s64)
+    ; CHECK: G_STORE [[LOAD8]](s128), [[PTR_ADD15]](p0) :: (store 16 into %ir.0 + 127, align 1, basealign 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 143
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/inline-memmove.mir

body:             |
   bb.1.entry:
     liveins: $x0, $x1, $x2
 
     ; CHECK-LABEL: name: test_memmove1
     ; CHECK: liveins: $x0, $x1, $x2
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[COPY2:%[0-9]+]]:_(s64) = COPY $x2
-    ; CHECK: G_MEMMOVE [[COPY]](p0), [[COPY1]](p0), [[COPY2]](s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: G_MEMMOVE [[COPY]](p0), [[COPY1]](p0), [[COPY2]](s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = COPY $x2
-    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memmove2_const
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: G_STORE [[LOAD1]](s128), [[GEP2]](p0) :: (store 16 into %ir.0 + 16, align 4)
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
     ; CHECK: [[GEP3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
     ; CHECK: G_STORE [[LOAD2]](s128), [[GEP3]](p0) :: (store 16 into %ir.0 + 32, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 48
-    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memmove3_const_toolarge
 alignment:       4
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_memmove3_const_toolarge
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 96
-    ; CHECK: G_MEMMOVE [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: G_MEMMOVE [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 96
-    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memmove4_const_unaligned
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: G_STORE [[LOAD2]](s128), [[GEP4]](p0) :: (store 16 into %ir.0 + 32, align 4)
     ; CHECK: [[C5:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
     ; CHECK: [[GEP5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C5]](s64)
     ; CHECK: G_STORE [[LOAD3]](s32), [[GEP5]](p0) :: (store 4 into %ir.0 + 48)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 52
-    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMMOVE %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/inline-memset.mir

bb.1.entry:
 
     ; CHECK-LABEL: name: test_ms1
     ; CHECK: liveins: $w1, $w2, $x0
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $w1
     ; CHECK: [[COPY2:%[0-9]+]]:_(s32) = COPY $w2
     ; CHECK: [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY1]](s32)
     ; CHECK: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[COPY2]](s32)
-    ; CHECK: G_MEMSET [[COPY]](p0), [[TRUNC]](s8), [[ZEXT]](s64), 1 :: (store 1 into %ir.dst)
+    ; CHECK: G_MEMSET [[COPY]](p0), [[TRUNC]](s8), [[ZEXT]](s64), 1, 0 :: (store 1 into %ir.dst)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %2:_(s32) = COPY $w2
     %3:_(s8) = G_TRUNC %1(s32)
     %4:_(s64) = G_ZEXT %2(s32)
-    G_MEMSET %0(p0), %3(s8), %4(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %3(s8), %4(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 ---
 name:            test_ms2_const
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
     ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
     ; CHECK: G_STORE [[MUL]](s64), [[PTR_ADD]](p0) :: (store 8 into %ir.dst + 8, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %3:_(s64) = G_CONSTANT i64 16
     %2:_(s8) = G_TRUNC %1(s32)
-    G_MEMSET %0(p0), %2(s8), %3(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %2(s8), %3(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 ---
 name:            test_zero_const
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
     ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
     ; CHECK: G_STORE [[BUILD_VECTOR]](<2 x s64>), [[PTR_ADD2]](p0) :: (store 16 into %ir.dst + 48, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = G_CONSTANT i32 0
     %3:_(s64) = G_CONSTANT i64 64
     %2:_(s8) = G_TRUNC %1(s32)
-    G_MEMSET %0(p0), %2(s8), %3(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %2(s8), %3(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 
 ---
 name:            test_ms3_const_both
 alignment:       4
 tracksRegLiveness: true
 body:             |
   bb.1.entry:
     liveins: $x0
 
     ; CHECK-LABEL: name: test_ms3_const_both
     ; CHECK: liveins: $x0
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 4629771061636907072
     ; CHECK: G_STORE [[C]](s64), [[COPY]](p0) :: (store 8 into %ir.dst, align 1)
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
     ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
     ; CHECK: G_STORE [[C]](s64), [[PTR_ADD]](p0) :: (store 8 into %ir.dst + 8, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s8) = G_CONSTANT i8 64
     %2:_(s64) = G_CONSTANT i64 16
-    G_MEMSET %0(p0), %1(s8), %2(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %1(s8), %2(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 ---
 name:            test_ms_vector
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 44
     ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
     ; CHECK: G_STORE [[BUILD_VECTOR]](<2 x s64>), [[PTR_ADD2]](p0) :: (store 16 into %ir.dst + 44, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %3:_(s64) = G_CONSTANT i64 60
     %2:_(s8) = G_TRUNC %1(s32)
-    G_MEMSET %0(p0), %2(s8), %3(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %2(s8), %3(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 ---
 name:            test_ms4_const_both_unaligned
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.1.entry:
     ; CHECK: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[C]](s64)
     ; CHECK: [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
     ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
     ; CHECK: G_STORE [[TRUNC]](s16), [[PTR_ADD1]](p0) :: (store 2 into %ir.dst + 16, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s8) = G_CONSTANT i8 64
     %2:_(s64) = G_CONSTANT i64 18
-    G_MEMSET %0(p0), %1(s8), %2(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %1(s8), %2(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR
 
 ...
 ---
 name:            minsize
 alignment:       4
 tracksRegLiveness: true
 body:             |
bb.0:
     ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
     ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
     ; CHECK: G_STORE [[MUL]](s64), [[PTR_ADD]](p0) :: (store 8 into %ir.dst + 8, align 1)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %3:_(s64) = G_CONSTANT i64 16
     %2:_(s8) = G_TRUNC %1(s32)
-    G_MEMSET %0(p0), %2(s8), %3(s64), 1 :: (store 1 into %ir.dst)
+    G_MEMSET %0(p0), %2(s8), %3(s64), 1, 0 :: (store 1 into %ir.dst)
     RET_ReallyLR

llvm/test/CodeGen/AArch64/GlobalISel/inline-small-memcpy.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -march=aarch64 -run-pass=aarch64-prelegalizer-combiner -O0 -verify-machineinstrs %s -o - | FileCheck %s
 --- |
   target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
   target triple = "arm64-apple-darwin"
 
   declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg) #1
+  declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg) #1
 
   define void @test_small_memcpy(i32* nocapture %dst, i32* nocapture readonly %src) {
   entry:
     %0 = bitcast i32* %dst to i8*
     %1 = bitcast i32* %src to i8*
     tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 32, i1 false)
     ret void
   }
 
   define void @test_large_memcpy(i32* nocapture %dst, i32* nocapture readonly %src) {
   entry:
     %0 = bitcast i32* %dst to i8*
     %1 = bitcast i32* %src to i8*
     tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 36, i1 false)
     ret void
   }
 
+  define void @test_large_memcpy_inline(i32* nocapture %dst, i32* nocapture readonly %src) {
+  entry:
+    %0 = bitcast i32* %dst to i8*
+    %1 = bitcast i32* %src to i8*
+    tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 36, i1 false)
+    ret void
+  }
+
   attributes #1 = { argmemonly nounwind }
 
 ...
 ---
 name:            test_small_memcpy
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_small_memcpy
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
     ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
-    ; CHECK: [[GEP:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
-    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[GEP]](p0) :: (load 16 from %ir.1 + 16, align 4)
-    ; CHECK: [[GEP1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
-    ; CHECK: G_STORE [[LOAD1]](s128), [[GEP1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 32
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...
 ---
 name:            test_large_memcpy
 alignment:       4
 tracksRegLiveness: true
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
 machineFunctionInfo: {}
 body:             |
   bb.1.entry:
     liveins: $x0, $x1
 
     ; CHECK-LABEL: name: test_large_memcpy
     ; CHECK: liveins: $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 36
-    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[C]](s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    ; CHECK: RET_ReallyLR
+    %0:_(p0) = COPY $x0
+    %1:_(p0) = COPY $x1
+    %2:_(s64) = G_CONSTANT i64 36
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    RET_ReallyLR
+
+...
+---
+name:            test_large_memcpy_inline
+alignment:       4
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: _ }
+machineFunctionInfo: {}
+body:             |
+  bb.1.entry:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: test_large_memcpy_inline
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+    ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
+    ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
+    ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
+    ; CHECK: [[LOAD2:%[0-9]+]]:_(s32) = G_LOAD [[PTR_ADD2]](p0) :: (load 4 from %ir.1 + 32)
+    ; CHECK: [[PTR_ADD3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
+    ; CHECK: G_STORE [[LOAD2]](s32), [[PTR_ADD3]](p0) :: (store 4 into %ir.0 + 32)
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s64) = G_CONSTANT i64 36
-    G_MEMCPY %0(p0), %1(p0), %2(s64), 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    G_MEMCPY %0(p0), %1(p0), %2(s64), 1, 1 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
     RET_ReallyLR
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/irtranslator-memcpy-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-unknown -global-isel -global-isel-abort=1 -verify-machineinstrs -stop-after=irtranslator %s -o - | FileCheck %s
+
+define void @copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0, 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @inline_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: inline_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0, 1 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0, 0 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @inline_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: inline_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0, 1 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @tail_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1, 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @tail_inline_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_inline_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1, 1 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @tail_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1, 0 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @tail_inline_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_inline_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1, 1 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i32(i8* nocapture writeonly, i8* nocapture readonly, i32, i1) nounwind
+declare void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* nocapture writeonly, i8* nocapture readonly, i32, i1) nounwind

llvm/test/CodeGen/AArch64/GlobalISel/legalize-bzero.mir

bb.0:
     ; CHECK: ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: $x0 = COPY %ptr(p0)
     ; CHECK: $x1 = COPY %width(s64)
     ; CHECK: BL &bzero, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $x1
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %width:_(s64) = COPY $x1
-    G_BZERO %ptr(p0), %width(s64), 0 :: (store 4)
+    G_BZERO %ptr(p0), %width(s64), 0, 0 :: (store 4)
     RET_ReallyLR
 
 ...
 ---
 name:            bzero_tail_call
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x0, $x1
     ; CHECK-LABEL: name: bzero_tail_call
     ; CHECK: liveins: $x0, $x1
     ; CHECK: %ptr:_(p0) = COPY $x0
     ; CHECK: %width:_(s64) = COPY $x1
     ; CHECK: $x0 = COPY %ptr(p0)
     ; CHECK: $x1 = COPY %width(s64)
     ; CHECK: TCRETURNdi &bzero, 0, csr_darwin_aarch64_aapcs, implicit $sp, implicit $x0, implicit $x1
     %ptr:_(p0) = COPY $x0
     %width:_(s64) = COPY $x1
-    G_BZERO %ptr(p0), %width(s64), 1 :: (store 4)
+    G_BZERO %ptr(p0), %width(s64), 1, 0 :: (store 4)
     RET_ReallyLR

llvm/test/CodeGen/AArch64/GlobalISel/legalize-memcpy-et-al.mir

bb.1:
     ; CHECK: $x2 = COPY [[ZEXT]](s64)
     ; CHECK: BL &memcpy, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $x1, implicit $x2
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s32) = COPY $w2
     %3:_(s64) = G_ZEXT %2(s32)
-    G_MEMCPY %0(p0), %1(p0), %3(s64), 0 :: (store unknown-size), (load unknown-size)
+    G_MEMCPY %0(p0), %1(p0), %3(s64), 0, 0 :: (store unknown-size), (load unknown-size)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memcpy_tail
 tracksRegLiveness: true
 body:             |
   bb.1:
     liveins: $w2, $x0, $x1
 
     ; CHECK-LABEL: name: test_memcpy_tail
     ; CHECK: liveins: $w2, $x0, $x1
     ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
     ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
     ; CHECK: [[COPY2:%[0-9]+]]:_(s32) = COPY $w2
     ; CHECK: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[COPY2]](s32)
     ; CHECK: $x0 = COPY [[COPY]](p0)
     ; CHECK: $x1 = COPY [[COPY1]](p0)
     ; CHECK: $x2 = COPY [[ZEXT]](s64)
     ; CHECK: TCRETURNdi &memcpy, 0, csr_aarch64_aapcs, implicit $sp, implicit $x0, implicit $x1, implicit $x2
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s32) = COPY $w2
     %3:_(s64) = G_ZEXT %2(s32)
-    G_MEMCPY %0(p0), %1(p0), %3(s64), 1 :: (store unknown-size), (load unknown-size)
+    G_MEMCPY %0(p0), %1(p0), %3(s64), 1, 0 :: (store unknown-size), (load unknown-size)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memmove
 tracksRegLiveness: true
 body:             |
   bb.1:
bb.1:
     ; CHECK: $x2 = COPY [[ZEXT]](s64)
     ; CHECK: BL &memmove, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $x1, implicit $x2
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s32) = COPY $w2
     %3:_(s64) = G_ZEXT %2(s32)
-    G_MEMMOVE %0(p0), %1(p0), %3(s64), 0 :: (store unknown-size), (load unknown-size)
+    G_MEMMOVE %0(p0), %1(p0), %3(s64), 0, 0 :: (store unknown-size), (load unknown-size)
     RET_ReallyLR
 
 ...
 ---
 name:            test_memset
 tracksRegLiveness: true
 body:             |
   bb.1:
bb.1:
     ; CHECK: BL &memset, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $w1, implicit $x2
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: RET_ReallyLR
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %2:_(s32) = COPY $w2
     %3:_(s8) = G_TRUNC %1(s32)
     %4:_(s64) = G_ZEXT %2(s32)
-    G_MEMSET %0(p0), %3(s8), %4(s64), 0 :: (store unknown-size)
+    G_MEMSET %0(p0), %3(s8), %4(s64), 0, 0 :: (store unknown-size)
     RET_ReallyLR
 
 ...
 ---
 name:            no_tail_call
 tracksRegLiveness: true
 body:             |
   bb.1:
bb.1:
     ; CHECK: BL &memcpy, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $x1, implicit $x2
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: $x0 = COPY [[ZEXT]](s64)
     ; CHECK: RET_ReallyLR implicit $x0
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s32) = COPY $w2
     %3:_(s64) = G_ZEXT %2(s32)
-    G_MEMCPY %0(p0), %1(p0), %3(s64), 1 :: (store unknown-size), (load unknown-size)
+    G_MEMCPY %0(p0), %1(p0), %3(s64), 1, 0 :: (store unknown-size), (load unknown-size)
     $x0 = COPY %3
     RET_ReallyLR implicit $x0
 
 ...
 ---
 name:            dont_tc_twice
 tracksRegLiveness: true
 body:             |
bb.1:
     ; CHECK: BL &memcpy, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $x1, implicit $x2
     ; CHECK: ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
     ; CHECK: TCRETURNdi &memset, 0, csr_aarch64_aapcs, implicit $sp
     %0:_(p0) = COPY $x0
     %1:_(p0) = COPY $x1
     %2:_(s32) = COPY $w2
     %4:_(s1) = G_CONSTANT i1 false
     %3:_(s64) = G_ZEXT %2(s32)
-    G_MEMCPY %0(p0), %1(p0), %3(s64), 1 :: (store unknown-size), (load unknown-size)
+    G_MEMCPY %0(p0), %1(p0), %3(s64), 1, 0 :: (store unknown-size), (load unknown-size)
     TCRETURNdi &memset, 0, csr_aarch64_aapcs, implicit $sp

llvm/test/CodeGen/AArch64/GlobalISel/legalize-memcpy-with-debug-info.mir

bb.0:
     ; CHECK: $x0 = COPY [[COPY]](p0), debug-location !DILocation(line: 6, column: 1, scope: !5)
     ; CHECK: $x1 = COPY [[COPY1]](p0), debug-location !DILocation(line: 6, column: 1, scope: !5)
     ; CHECK: $x2 = COPY [[ZEXT]](s64), debug-location !DILocation(line: 6, column: 1, scope: !5)
     ; CHECK: TCRETURNdi &memcpy, 0, csr_aarch64_aapcs, implicit $sp, implicit $x0, implicit $x1, implicit $x2, debug-location !DILocation(line: 6, column: 1, scope: !5)
     %0:_(p0) = COPY $x0, debug-location !14
     %1:_(p0) = COPY $x1, debug-location !DILocation(line: 3, column: 1, scope: !11)
     %2:_(s32) = COPY $w2, debug-location !DILocation(line: 4, column: 1, scope: !11)
     %3:_(s64) = G_ZEXT %2(s32), debug-location !DILocation(line: 5, column: 1, scope: !11)
-    G_MEMCPY %0(p0), %1(p0), %3(s64), 1, debug-location !DILocation(line: 6, column: 1, scope: !11) :: (store unknown-size), (load unknown-size)
+    G_MEMCPY %0(p0), %1(p0), %3(s64), 1, 0, debug-location !DILocation(line: 6, column: 1, scope: !11) :: (store unknown-size), (load unknown-size)
     DBG_VALUE 0, $noreg, !13, !DIExpression(), debug-location !DILocation(line: 6, column: 1, scope: !11)
     DBG_VALUE 0, $noreg, !13, !DIExpression(), debug-location !DILocation(line: 6, column: 1, scope: !11)
     RET_ReallyLR debug-location !DILocation(line: 7, column: 1, scope: !11)
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/legalize-memlib-debug-loc.mir

bb.1.entry:
     ; CHECK-LABEL: name: test_memset_debug
     ; CHECK: BL &memset, csr_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0, implicit $w1, implicit $x2, debug-location !11
     ; CHECK: RET_ReallyLR debug-location !12
     %0:_(p0) = COPY $x0
     %1:_(s32) = COPY $w1
     %2:_(s32) = COPY $w2
     %3:_(s64) = G_ZEXT %2(s32), debug-location !11
     %4:_(s8) = G_TRUNC %1(s32), debug-location !11
-    G_MEMSET %0(p0), %4(s8), %3(s64), 0, debug-location !11 :: (store 1 into %ir.ptr)
+    G_MEMSET %0(p0), %4(s8), %3(s64), 0, 0, debug-location !11 :: (store 1 into %ir.ptr)
     RET_ReallyLR debug-location !12
 
 ...

llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-bzero.mir

body:             |
   bb.0:
     liveins: $x0, $x1
     ; Always use G_BZERO when the memset width is unknown on Darwin.
 
     ; DARWIN-LABEL: name: bzero_unknown_width
     ; DARWIN: liveins: $x0, $x1
     ; DARWIN: %ptr:_(p0) = COPY $x0
     ; DARWIN: %width:_(s64) = COPY $x1
-    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0 :: (store 4)
+    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0, 0 :: (store 4)
     ; DARWIN: RET_ReallyLR
     ; UNKNOWN-LABEL: name: bzero_unknown_width
     ; UNKNOWN: liveins: $x0, $x1
     ; UNKNOWN: %ptr:_(p0) = COPY $x0
     ; UNKNOWN: %zero:_(s8) = G_CONSTANT i8 0
     ; UNKNOWN: %width:_(s64) = COPY $x1
-    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     ; UNKNOWN: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %zero:_(s8) = G_CONSTANT i8 0
     %width:_(s64) = COPY $x1
-    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     RET_ReallyLR
 ...
 ---
 name:            bzero_tail_unknown_width
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x0, $x1
     ; DARWIN-LABEL: name: bzero_tail_unknown_width
     ; DARWIN: liveins: $x0, $x1
     ; DARWIN: %ptr:_(p0) = COPY $x0
     ; DARWIN: %width:_(s64) = COPY $x1
-    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 1 :: (store 4)
+    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 1, 0 :: (store 4)
     ; DARWIN: RET_ReallyLR
     ; UNKNOWN-LABEL: name: bzero_tail_unknown_width
     ; UNKNOWN: liveins: $x0, $x1
     ; UNKNOWN: %ptr:_(p0) = COPY $x0
     ; UNKNOWN: %zero:_(s8) = G_CONSTANT i8 0
     ; UNKNOWN: %width:_(s64) = COPY $x1
-    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 1 :: (store 4)
+    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 1, 0 :: (store 4)
     ; UNKNOWN: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %zero:_(s8) = G_CONSTANT i8 0
     %width:_(s64) = COPY $x1
-    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 1 :: (store 4)
+    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 1, 0 :: (store 4)
     RET_ReallyLR
 ...
 ---
 name:            bzero_constant_width
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x0, $x1
     ; For values >256, we should use G_BZERO on Darwin.
 
     ; DARWIN-LABEL: name: bzero_constant_width
     ; DARWIN: liveins: $x0, $x1
     ; DARWIN: %ptr:_(p0) = COPY $x0
     ; DARWIN: %width:_(s64) = G_CONSTANT i64 1024
-    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0 :: (store 4)
+    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0, 0 :: (store 4)
     ; DARWIN: RET_ReallyLR
     ; UNKNOWN-LABEL: name: bzero_constant_width
     ; UNKNOWN: liveins: $x0, $x1
     ; UNKNOWN: %ptr:_(p0) = COPY $x0
     ; UNKNOWN: %zero:_(s8) = G_CONSTANT i8 0
     ; UNKNOWN: %width:_(s64) = G_CONSTANT i64 1024
-    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     ; UNKNOWN: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %zero:_(s8) = G_CONSTANT i8 0
     %width:_(s64) = G_CONSTANT i64 1024
-    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     RET_ReallyLR
 ...
 ---
 name:            bzero_constant_width_minsize
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x0, $x1
     ; With minsize, we should always use G_BZERO to avoid a copy from wzr.
 
     ; DARWIN-LABEL: name: bzero_constant_width_minsize
     ; DARWIN: liveins: $x0, $x1
     ; DARWIN: %ptr:_(p0) = COPY $x0
     ; DARWIN: %width:_(s64) = G_CONSTANT i64 256
-    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0 :: (store 4)
+    ; DARWIN: G_BZERO %ptr(p0), %width(s64), 0, 0 :: (store 4)
     ; DARWIN: RET_ReallyLR
     ; UNKNOWN-LABEL: name: bzero_constant_width_minsize
     ; UNKNOWN: liveins: $x0, $x1
     ; UNKNOWN: %ptr:_(p0) = COPY $x0
     ; UNKNOWN: %zero:_(s8) = G_CONSTANT i8 0
     ; UNKNOWN: %width:_(s64) = G_CONSTANT i64 256
-    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    ; UNKNOWN: G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     ; UNKNOWN: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %zero:_(s8) = G_CONSTANT i8 0
     %width:_(s64) = G_CONSTANT i64 256
-    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0 :: (store 4)
+    G_MEMSET %ptr(p0), %zero(s8), %width(s64), 0, 0 :: (store 4)
     RET_ReallyLR
 ...
 ---
 name:            not_zero
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $x0, $x1
     ; When the value isn't 0, don't create G_BZERO.
 
     ; DARWIN-LABEL: name: not_zero
     ; DARWIN: liveins: $x0, $x1
     ; DARWIN: %ptr:_(p0) = COPY $x0
     ; DARWIN: %not_zero:_(s8) = G_CONSTANT i8 1
     ; DARWIN: %width:_(s64) = G_CONSTANT i64 256
-    ; DARWIN: G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0 :: (store 4)
+    ; DARWIN: G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0, 0 :: (store 4)
     ; DARWIN: RET_ReallyLR
     ; UNKNOWN-LABEL: name: not_zero
     ; UNKNOWN: liveins: $x0, $x1
     ; UNKNOWN: %ptr:_(p0) = COPY $x0
     ; UNKNOWN: %not_zero:_(s8) = G_CONSTANT i8 1
     ; UNKNOWN: %width:_(s64) = G_CONSTANT i64 256
-    ; UNKNOWN: G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0 :: (store 4)
+    ; UNKNOWN: G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0, 0 :: (store 4)
     ; UNKNOWN: RET_ReallyLR
     %ptr:_(p0) = COPY $x0
     %not_zero:_(s8) = G_CONSTANT i8 1
     %width:_(s64) = G_CONSTANT i64 256
-    G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0 :: (store 4)
+    G_MEMSET %ptr(p0), %not_zero(s8), %width(s64), 0, 0 :: (store 4)
     RET_ReallyLR

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-call.ll

   ; CHECK:   [[COPY19:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; CHECK:   [[C5:%[0-9]+]]:_(s32) = G_CONSTANT i32 20
   ; CHECK:   [[SHL1:%[0-9]+]]:_(s32) = G_SHL [[COPY19]], [[C5]](s32)
   ; CHECK:   [[OR1:%[0-9]+]]:_(s32) = G_OR [[OR]], [[SHL1]]
   ; CHECK:   [[COPY20:%[0-9]+]]:_(p5) = COPY $sp_reg
   ; CHECK:   [[C6:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
   ; CHECK:   [[PTR_ADD2:%[0-9]+]]:_(p5) = G_PTR_ADD [[COPY20]], [[C6]](s32)
   ; CHECK:   [[C7:%[0-9]+]]:_(s32) = G_CONSTANT i32 8
-  ; CHECK:   G_MEMCPY [[PTR_ADD2]](p5), [[FRAME_INDEX]](p5), [[C7]](s32), 0 :: (dereferenceable store 8 into stack, align 4, addrspace 5), (dereferenceable load 8 from %ir.val, align 4, addrspace 5)
+  ; CHECK:   G_MEMCPY [[PTR_ADD2]](p5), [[FRAME_INDEX]](p5), [[C7]](s32), 0, 0 :: (dereferenceable store 8 into stack, align 4, addrspace 5), (dereferenceable load 8 from %ir.val, align 4, addrspace 5)
   ; CHECK:   [[COPY21:%[0-9]+]]:_(<4 x s32>) = COPY $private_rsrc_reg
   ; CHECK:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY21]](<4 x s32>)
   ; CHECK:   $sgpr4_sgpr5 = COPY [[COPY10]](p4)
   ; CHECK:   $sgpr6_sgpr7 = COPY [[COPY11]](p4)
   ; CHECK:   $sgpr8_sgpr9 = COPY [[PTR_ADD1]](p4)
   ; CHECK:   $sgpr10_sgpr11 = COPY [[COPY13]](s64)
   ; CHECK:   $sgpr12 = COPY [[COPY14]](s32)
   ; CHECK:   $sgpr13 = COPY [[COPY15]](s32)

   ; CHECK:   [[COPY15:%[0-9]+]]:_(s32) = COPY [[COPY3]]
   ; CHECK:   [[COPY16:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; CHECK:   [[COPY17:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; CHECK:   [[COPY18:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; CHECK:   [[COPY19:%[0-9]+]]:_(p5) = COPY $sgpr32
   ; CHECK:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
   ; CHECK:   [[PTR_ADD:%[0-9]+]]:_(p5) = G_PTR_ADD [[COPY19]], [[C1]](s32)
   ; CHECK:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 12
-  ; CHECK:   G_MEMCPY [[PTR_ADD]](p5), [[COPY8]](p5), [[C2]](s32), 0 :: (dereferenceable store 12 into stack, align 4, addrspace 5), (dereferenceable load 12 from %ir.incoming0, align 4, addrspace 5)
+  ; CHECK:   G_MEMCPY [[PTR_ADD]](p5), [[COPY8]](p5), [[C2]](s32), 0, 0 :: (dereferenceable store 12 into stack, align 4, addrspace 5), (dereferenceable load 12 from %ir.incoming0, align 4, addrspace 5)
   ; CHECK:   [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
   ; CHECK:   [[PTR_ADD1:%[0-9]+]]:_(p5) = G_PTR_ADD [[COPY19]], [[C3]](s32)
   ; CHECK:   [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
-  ; CHECK:   G_MEMCPY [[PTR_ADD1]](p5), [[COPY9]](p5), [[C4]](s32), 0 :: (dereferenceable store 1 into stack + 32, align 32, addrspace 5), (dereferenceable load 1 from %ir.incoming1, align 32, addrspace 5)
+  ; CHECK:   G_MEMCPY [[PTR_ADD1]](p5), [[COPY9]](p5), [[C4]](s32), 0, 0 :: (dereferenceable store 1 into stack + 32, align 32, addrspace 5), (dereferenceable load 1 from %ir.incoming1, align 32, addrspace 5)
   ; CHECK:   $vgpr0 = COPY [[C]](s32)
   ; CHECK:   [[COPY20:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
   ; CHECK:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY20]](<4 x s32>)
   ; CHECK:   $sgpr4_sgpr5 = COPY [[COPY11]](p4)
   ; CHECK:   $sgpr6_sgpr7 = COPY [[COPY12]](p4)
   ; CHECK:   $sgpr8_sgpr9 = COPY [[COPY13]](p4)
   ; CHECK:   $sgpr10_sgpr11 = COPY [[COPY14]](s64)
   ; CHECK:   $sgpr12 = COPY [[COPY15]](s32)

   ; CHECK:   [[COPY14:%[0-9]+]]:_(s32) = COPY [[COPY3]]
   ; CHECK:   [[COPY15:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; CHECK:   [[COPY16:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; CHECK:   [[COPY17:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; CHECK:   [[COPY18:%[0-9]+]]:_(p5) = COPY $sgpr32
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
   ; CHECK:   [[PTR_ADD:%[0-9]+]]:_(p5) = G_PTR_ADD [[COPY18]], [[C]](s32)
   ; CHECK:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
-  ; CHECK:   G_MEMCPY [[PTR_ADD]](p5), [[COPY8]](p5), [[C1]](s32), 0 :: (dereferenceable store 32 into stack, align 4, addrspace 5), (dereferenceable load 32 from %ir.incoming_high_align, align 256, addrspace 5)
+  ; CHECK:   G_MEMCPY [[PTR_ADD]](p5), [[COPY8]](p5), [[C1]](s32), 0, 0 :: (dereferenceable store 32 into stack, align 4, addrspace 5), (dereferenceable load 32 from %ir.incoming_high_align, align 256, addrspace 5)
   ; CHECK:   [[COPY19:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
   ; CHECK:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY19]](<4 x s32>)
   ; CHECK:   $sgpr4_sgpr5 = COPY [[COPY10]](p4)
   ; CHECK:   $sgpr6_sgpr7 = COPY [[COPY11]](p4)
   ; CHECK:   $sgpr8_sgpr9 = COPY [[COPY12]](p4)
   ; CHECK:   $sgpr10_sgpr11 = COPY [[COPY13]](s64)
   ; CHECK:   $sgpr12 = COPY [[COPY14]](s32)
   ; CHECK:   $sgpr13 = COPY [[COPY15]](s32)

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-memory-intrinsics.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -global-isel -march=amdgcn -O0 -stop-after=irtranslator -verify-machineinstrs %s -o - | FileCheck %s
 
 ; Size operand should be the minimum of the two pointer sizes.
 
 define void @test_memcpy_p1_p3_i64(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p1_p3_i64
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 256
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s32) = G_TRUNC [[C]](s64)
-  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[TRUNC]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[TRUNC]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p1i8.p3i8.i64(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i64 256, i1 false)
   ret void
 }
 
 define void @test_memcpy_p1_p3_i32(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p1_p3_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 256
-  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[C]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[C]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p1i8.p3i8.i32(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i32 256, i1 false)
   ret void
 }
 
 define void @test_memcpy_p1_p3_i16(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p1_p3_i16
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s32) = G_ZEXT [[C]](s16)
-  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[ZEXT]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMCPY [[MV]](p1), [[COPY2]](p3), [[ZEXT]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p1i8.p3i8.i16(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i16 256, i1 false)
   ret void
 }
 
 define void @test_memcpy_p3_p1_i64(i8 addrspace(3)* %dst, i8 addrspace(1)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p3_p1_i64
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY1]](s32), [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 256
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s32) = G_TRUNC [[C]](s64)
-  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[TRUNC]](s32), 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
+  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[TRUNC]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p3i8.p1i8.i64(i8 addrspace(3)* %dst, i8 addrspace(1)* %src, i64 256, i1 false)
   ret void
 }
 
 define void @test_memcpy_p3_p1_i32(i8 addrspace(3)* %dst, i8 addrspace(1)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p3_p1_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY1]](s32), [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 256
-  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[C]](s32), 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
+  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[C]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p3i8.p1i8.i32(i8 addrspace(3)* %dst, i8 addrspace(1)* %src, i32 256, i1 false)
   ret void
 }
 
 define void @test_memcpy_p3_p1_i16(i8 addrspace(3)* %dst, i8 addrspace(1)* %src) {
   ; CHECK-LABEL: name: test_memcpy_p3_p1_i16
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY1]](s32), [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s32) = G_ZEXT [[C]](s16)
-  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[ZEXT]](s32), 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
+  ; CHECK:   G_MEMCPY [[COPY]](p3), [[MV]](p1), [[ZEXT]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3), (load 1 from %ir.src, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memcpy.p3i8.p1i8.i16(i8 addrspace(3)* %dst, i8 addrspace(1)* %src, i16 256, i1 false)
   ret void
 }
 
 define void @test_memmove_p1_p3_i64(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memmove_p1_p3_i64
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 256
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s32) = G_TRUNC [[C]](s64)
-  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[TRUNC]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[TRUNC]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memmove.p1i8.p3i8.i64(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i64 256, i1 false)
   ret void
 }
 
 define void @test_memmove_p1_p3_i32(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memmove_p1_p3_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 256
-  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[C]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[C]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memmove.p1i8.p3i8.i32(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i32 256, i1 false)
   ret void
 }
 
 define void @test_memmove_p1_p3_i16(i8 addrspace(1)* %dst, i8 addrspace(3)* %src) {
   ; CHECK-LABEL: name: test_memmove_p1_p3_i16
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(p3) = COPY $vgpr2
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s32) = G_ZEXT [[C]](s16)
-  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[ZEXT]](s32), 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
+  ; CHECK:   G_MEMMOVE [[MV]](p1), [[COPY2]](p3), [[ZEXT]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 1), (load 1 from %ir.src, addrspace 3)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memmove.p1i8.p3i8.i16(i8 addrspace(1)* %dst, i8 addrspace(3)* %src, i16 256, i1 false)
   ret void
 }
 
 define void @test_memset_p1_i64(i8 addrspace(1)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p1_i64
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 256
-  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[C]](s64), 0 :: (store 1 into %ir.dst, addrspace 1)
+  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[C]](s64), 0, 0 :: (store 1 into %ir.dst, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memset.p1i8.i64(i8 addrspace(1)* %dst, i8 %val, i64 256, i1 false)
   ret void
 }
 
 define void @test_memset_p1_i32(i8 addrspace(1)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p1_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
-  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[ZEXT]](s64), 0 :: (store 1 into %ir.dst, addrspace 1)
+  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[ZEXT]](s64), 0, 0 :: (store 1 into %ir.dst, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memset.p1i8.i32(i8 addrspace(1)* %dst, i8 %val, i32 256, i1 false)
   ret void
 }
 
 define void @test_memset_p1_i16(i8 addrspace(1)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p1_i16
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $vgpr2, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[MV:%[0-9]+]]:_(p1) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY2]](s32)
   ; CHECK:   [[COPY3:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s16)
-  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[ZEXT]](s64), 0 :: (store 1 into %ir.dst, addrspace 1)
+  ; CHECK:   G_MEMSET [[MV]](p1), [[TRUNC]](s8), [[ZEXT]](s64), 0, 0 :: (store 1 into %ir.dst, addrspace 1)
   ; CHECK:   [[COPY4:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY3]]
   ; CHECK:   S_SETPC_B64_return [[COPY4]]
   call void @llvm.memset.p1i8.i16(i8 addrspace(1)* %dst, i8 %val, i16 256, i1 false)
   ret void
 }
 
 define void @test_memset_p3_i64(i8 addrspace(3)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p3_i64
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 256
   ; CHECK:   [[TRUNC1:%[0-9]+]]:_(s32) = G_TRUNC [[C]](s64)
-  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[TRUNC1]](s32), 0 :: (store 1 into %ir.dst, addrspace 3)
+  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[TRUNC1]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3)
   ; CHECK:   [[COPY3:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY2]]
   ; CHECK:   S_SETPC_B64_return [[COPY3]]
   call void @llvm.memset.p3i8.i64(i8 addrspace(3)* %dst, i8 %val, i64 256, i1 false)
   ret void
 }
 
 define void @test_memset_p3_i32(i8 addrspace(3)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p3_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 256
-  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[C]](s32), 0 :: (store 1 into %ir.dst, addrspace 3)
+  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[C]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3)
   ; CHECK:   [[COPY3:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY2]]
   ; CHECK:   S_SETPC_B64_return [[COPY3]]
   call void @llvm.memset.p3i8.i32(i8 addrspace(3)* %dst, i8 %val, i32 256, i1 false)
   ret void
 }
 
 define void @test_memset_p3_i16(i8 addrspace(3)* %dst, i8 %val) {
   ; CHECK-LABEL: name: test_memset_p3_i16
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK:   liveins: $vgpr0, $vgpr1, $sgpr30_sgpr31
   ; CHECK:   [[COPY:%[0-9]+]]:_(p3) = COPY $vgpr0
   ; CHECK:   [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; CHECK:   [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[COPY1]](s32)
   ; CHECK:   [[COPY2:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
   ; CHECK:   [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 256
   ; CHECK:   [[ZEXT:%[0-9]+]]:_(s32) = G_ZEXT [[C]](s16)
-  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[ZEXT]](s32), 0 :: (store 1 into %ir.dst, addrspace 3)
+  ; CHECK:   G_MEMSET [[COPY]](p3), [[TRUNC]](s8), [[ZEXT]](s32), 0, 0 :: (store 1 into %ir.dst, addrspace 3)
   ; CHECK:   [[COPY3:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY2]]
   ; CHECK:   S_SETPC_B64_return [[COPY3]]
   call void @llvm.memset.p3i8.i16(i8 addrspace(3)* %dst, i8 %val, i16 256, i1 false)
   ret void
 }
 
 declare void @llvm.memcpy.p1i8.p3i8.i64(i8 addrspace(1)* noalias nocapture writeonly, i8 addrspace(3)* noalias nocapture readonly, i64, i1 immarg) #0
 declare void @llvm.memcpy.p1i8.p3i8.i32(i8 addrspace(1)* noalias nocapture writeonly, i8 addrspace(3)* noalias nocapture readonly, i32, i1 immarg) #0

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-sibling-call.ll

define fastcc i32 @sibling_call_i32_fastcc_i32_byval_i32_byval_parent(i32 %a, i32 addrspace(5)* byval(i32) %b.byval, i32 %c) #1 {
   ; GCN:   [[COPY17:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; GCN:   [[COPY18:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; GCN:   [[COPY19:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; GCN:   $vgpr0 = COPY [[COPY8]](s32)
   ; GCN:   [[COPY20:%[0-9]+]]:_(p5) = COPY $sgpr32
   ; GCN:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
   ; GCN:   [[PTR_ADD:%[0-9]+]]:_(p5) = G_PTR_ADD [[COPY20]], [[C]](s32)
   ; GCN:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
-  ; GCN:   G_MEMCPY [[PTR_ADD]](p5), [[COPY9]](p5), [[C1]](s32), 0 :: (dereferenceable store 4 into stack, addrspace 5), (dereferenceable load 4 from %ir.b.byval, addrspace 5)
+  ; GCN:   G_MEMCPY [[PTR_ADD]](p5), [[COPY9]](p5), [[C1]](s32), 0, 0 :: (dereferenceable store 4 into stack, addrspace 5), (dereferenceable load 4 from %ir.b.byval, addrspace 5)
   ; GCN:   [[COPY21:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
   ; GCN:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY21]](<4 x s32>)
   ; GCN:   $sgpr4_sgpr5 = COPY [[COPY12]](p4)
   ; GCN:   $sgpr6_sgpr7 = COPY [[COPY13]](p4)
   ; GCN:   $sgpr8_sgpr9 = COPY [[COPY14]](p4)
   ; GCN:   $sgpr10_sgpr11 = COPY [[COPY15]](s64)
   ; GCN:   $sgpr12 = COPY [[COPY16]](s32)
   ; GCN:   $sgpr13 = COPY [[COPY17]](s32)
define fastcc i32 @sibling_call_i32_fastcc_i32_byval_i32(i32 %a, [32 x i32] %large) #1 {
   ; GCN:   [[COPY43:%[0-9]+]]:_(s64) = COPY [[COPY4]]
   ; GCN:   [[COPY44:%[0-9]+]]:_(s32) = COPY [[COPY3]]
   ; GCN:   [[COPY45:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; GCN:   [[COPY46:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; GCN:   [[COPY47:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; GCN:   $vgpr0 = COPY [[COPY8]](s32)
   ; GCN:   [[FRAME_INDEX2:%[0-9]+]]:_(p5) = G_FRAME_INDEX %fixed-stack.0
   ; GCN:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
-  ; GCN:   G_MEMCPY [[FRAME_INDEX2]](p5), [[INTTOPTR]](p5), [[C1]](s32), 0 :: (dereferenceable store 4 into %fixed-stack.0, align 16, addrspace 5), (dereferenceable load 4 from `i32 addrspace(5)* inttoptr (i32 16 to i32 addrspace(5)*)`, align 16, addrspace 5)
+  ; GCN:   G_MEMCPY [[FRAME_INDEX2]](p5), [[INTTOPTR]](p5), [[C1]](s32), 0, 0 :: (dereferenceable store 4 into %fixed-stack.0, align 16, addrspace 5), (dereferenceable load 4 from `i32 addrspace(5)* inttoptr (i32 16 to i32 addrspace(5)*)`, align 16, addrspace 5)
   ; GCN:   [[COPY48:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
   ; GCN:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY48]](<4 x s32>)
   ; GCN:   $sgpr4_sgpr5 = COPY [[COPY40]](p4)
   ; GCN:   $sgpr6_sgpr7 = COPY [[COPY41]](p4)
   ; GCN:   $sgpr8_sgpr9 = COPY [[COPY42]](p4)
   ; GCN:   $sgpr10_sgpr11 = COPY [[COPY43]](s64)
   ; GCN:   $sgpr12 = COPY [[COPY44]](s32)
   ; GCN:   $sgpr13 = COPY [[COPY45]](s32)
define fastcc void @sibling_call_fastcc_multi_byval(i32 %a, [64 x i32]) #1 {
   ; GCN:   [[COPY43:%[0-9]+]]:_(s64) = COPY [[COPY4]]
   ; GCN:   [[COPY44:%[0-9]+]]:_(s32) = COPY [[COPY3]]
   ; GCN:   [[COPY45:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; GCN:   [[COPY46:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; GCN:   [[COPY47:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; GCN:   $vgpr0 = COPY [[COPY8]](s32)
   ; GCN:   [[FRAME_INDEX36:%[0-9]+]]:_(p5) = G_FRAME_INDEX %fixed-stack.1
   ; GCN:   [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 12
-  ; GCN:   G_MEMCPY [[FRAME_INDEX36]](p5), [[FRAME_INDEX34]](p5), [[C4]](s32), 0 :: (dereferenceable store 12 into %fixed-stack.1, align 16, addrspace 5), (dereferenceable load 12 from %ir.alloca0, align 16, addrspace 5)
+  ; GCN:   G_MEMCPY [[FRAME_INDEX36]](p5), [[FRAME_INDEX34]](p5), [[C4]](s32), 0, 0 :: (dereferenceable store 12 into %fixed-stack.1, align 16, addrspace 5), (dereferenceable load 12 from %ir.alloca0, align 16, addrspace 5)
   ; GCN:   [[FRAME_INDEX37:%[0-9]+]]:_(p5) = G_FRAME_INDEX %fixed-stack.0
   ; GCN:   [[C5:%[0-9]+]]:_(s32) = G_CONSTANT i32 16
-  ; GCN:   G_MEMCPY [[FRAME_INDEX37]](p5), [[FRAME_INDEX35]](p5), [[C5]](s32), 0 :: (dereferenceable store 16 into %fixed-stack.0, addrspace 5), (dereferenceable load 16 from %ir.alloca1, align 8, addrspace 5)
+  ; GCN:   G_MEMCPY [[FRAME_INDEX37]](p5), [[FRAME_INDEX35]](p5), [[C5]](s32), 0, 0 :: (dereferenceable store 16 into %fixed-stack.0, addrspace 5), (dereferenceable load 16 from %ir.alloca1, align 8, addrspace 5)
   ; GCN:   [[COPY48:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
   ; GCN:   $sgpr0_sgpr1_sgpr2_sgpr3 = COPY [[COPY48]](<4 x s32>)
   ; GCN:   $sgpr4_sgpr5 = COPY [[COPY40]](p4)
   ; GCN:   $sgpr6_sgpr7 = COPY [[COPY41]](p4)
   ; GCN:   $sgpr8_sgpr9 = COPY [[COPY42]](p4)
   ; GCN:   $sgpr10_sgpr11 = COPY [[COPY43]](s64)
   ; GCN:   $sgpr12 = COPY [[COPY44]](s32)
   ; GCN:   $sgpr13 = COPY [[COPY45]](s32)
define fastcc void @sibling_call_byval_and_stack_passed(i32 %stack.out.arg, [64 x i32]) #1 {
   ; GCN:   [[COPY42:%[0-9]+]]:_(p4) = COPY [[COPY5]]
   ; GCN:   [[COPY43:%[0-9]+]]:_(s64) = COPY [[COPY4]]
   ; GCN:   [[COPY44:%[0-9]+]]:_(s32) = COPY [[COPY3]]
   ; GCN:   [[COPY45:%[0-9]+]]:_(s32) = COPY [[COPY2]]
   ; GCN:   [[COPY46:%[0-9]+]]:_(s32) = COPY [[COPY1]]
   ; GCN:   [[COPY47:%[0-9]+]]:_(s32) = COPY [[COPY]](s32)
   ; GCN:   [[FRAME_INDEX35:%[0-9]+]]:_(p5) = G_FRAME_INDEX %fixed-stack.2
   ; GCN:   [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 12
-  ; GCN:   G_MEMCPY [[FRAME_INDEX35]](p5), [[FRAME_INDEX34]](p5), [[C4]](s32), 0 :: (dereferenceable store 12 into %fixed-stack.2, align 16, addrspace 5), (dereferenceable load 12 from %ir.alloca, align 16, addrspace 5)
+  ; GCN:   G_MEMCPY [[FRAME_INDEX35]](p5), [[FRAME_INDEX34]](p5), [[C4]](s32), 0, 0 :: (dereferenceable store 12 into %fixed-stack.2, align 16, addrspace 5), (dereferenceable load 12 from %ir.alloca, align 16, addrspace 5)
   ; GCN:   $vgpr0 = COPY [[C1]](s32)
   ; GCN:   $vgpr1 = COPY [[C1]](s32)
   ; GCN:   $vgpr2 = COPY [[C1]](s32)
   ; GCN:   $vgpr3 = COPY [[C1]](s32)
   ; GCN:   $vgpr4 = COPY [[C1]](s32)
   ; GCN:   $vgpr5 = COPY [[C1]](s32)
   ; GCN:   $vgpr6 = COPY [[C1]](s32)
   ; GCN:   $vgpr7 = COPY [[C1]](s32)

llvm/test/CodeGen/Mips/GlobalISel/irtranslator/call.ll

 
 define void @call_symbol(i8* nocapture readonly %src, i8* nocapture %dest, i32 signext %length) {
   ; MIPS32-LABEL: name: call_symbol
   ; MIPS32: bb.1.entry:
   ; MIPS32:   liveins: $a0, $a1, $a2
   ; MIPS32:   [[COPY:%[0-9]+]]:_(p0) = COPY $a0
   ; MIPS32:   [[COPY1:%[0-9]+]]:_(p0) = COPY $a1
   ; MIPS32:   [[COPY2:%[0-9]+]]:_(s32) = COPY $a2
-  ; MIPS32:   G_MEMCPY [[COPY1]](p0), [[COPY]](p0), [[COPY2]](s32), 0 :: (store 1 into %ir.dest), (load 1 from %ir.src)
+  ; MIPS32:   G_MEMCPY [[COPY1]](p0), [[COPY]](p0), [[COPY2]](s32), 0, 0 :: (store 1 into %ir.dest), (load 1 from %ir.src)
   ; MIPS32:   RetRA
   ; MIPS32_PIC-LABEL: name: call_symbol
   ; MIPS32_PIC: bb.1.entry:
   ; MIPS32_PIC:   liveins: $a0, $a1, $a2
   ; MIPS32_PIC:   [[COPY:%[0-9]+]]:_(p0) = COPY $a0
   ; MIPS32_PIC:   [[COPY1:%[0-9]+]]:_(p0) = COPY $a1
   ; MIPS32_PIC:   [[COPY2:%[0-9]+]]:_(s32) = COPY $a2
-  ; MIPS32_PIC:   G_MEMCPY [[COPY1]](p0), [[COPY]](p0), [[COPY2]](s32), 0 :: (store 1 into %ir.dest), (load 1 from %ir.src)
+  ; MIPS32_PIC:   G_MEMCPY [[COPY1]](p0), [[COPY]](p0), [[COPY2]](s32), 0, 0 :: (store 1 into %ir.dest), (load 1 from %ir.src)
   ; MIPS32_PIC:   RetRA
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 1 %dest, i8* align 1 %src, i32 %length, i1 false)
   ret void
 }
 
 declare void @f_with_void_ret();
 

llvm/test/CodeGen/Mips/GlobalISel/legalizer/dyn_stackalloc.mir

bb.1.entry:
     %13:_(s8) = G_CONSTANT i8 0
     %4:_(s32) = G_ADD %1, %3
     %5:_(s32) = G_MUL %4, %3
     %6:_(s32) = G_CONSTANT i32 7
     %7:_(s32) = nuw G_ADD %5, %6
     %8:_(s32) = G_CONSTANT i32 -8
     %9:_(s32) = G_AND %7, %8
     %10:_(p0) = G_DYN_STACKALLOC %9(s32), 0
-    G_MEMSET %10(p0), %0(s8), %1(s32), 0 :: (store 1 into %ir.vla)
+    G_MEMSET %10(p0), %0(s8), %1(s32), 0, 0 :: (store 1 into %ir.vla)
     %11:_(p0) = G_PTR_ADD %10, %1(s32)
     %12:_(p0) = COPY %11(p0)
     G_STORE %13(s8), %12(p0) :: (store 1 into %ir.arrayidx)
     ADJCALLSTACKDOWN 16, 0, implicit-def $sp, implicit $sp
     $a0 = COPY %10(p0)
     JAL @puts, csr_o32, implicit-def $ra, implicit-def $sp, implicit $a0, implicit-def $v0
     ADJCALLSTACKUP 16, 0, implicit-def $sp, implicit $sp
     RetRA
 
 ...

llvm/test/CodeGen/X86/GlobalISel/x86_64-irtranslator-struct-return.ll

define float @test_return_f1(float %f.coerce) {
   ; ALL-LABEL: name: test_return_f1
   ; ALL: bb.1.entry:
   ; ALL:   liveins: $xmm0
   ; ALL:   [[COPY:%[0-9]+]]:_(s32) = COPY $xmm0
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 4
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.f
   ; ALL:   G_STORE [[COPY]](s32), [[FRAME_INDEX1]](p0) :: (store 4 into %ir.coerce.dive2)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s32) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 4 from %ir.coerce.dive13)
   ; ALL:   $xmm0 = COPY [[LOAD]](s32)
   ; ALL:   RET 0, implicit $xmm0
 entry:
   %retval = alloca %struct.f1, align 4
   %f = alloca %struct.f1, align 4
   %coerce.dive = getelementptr inbounds %struct.f1, %struct.f1* %f, i32 0, i32 0
   store float %f.coerce, float* %coerce.dive, align 4
define double @test_return_d1(double %d.coerce) {
   ; ALL-LABEL: name: test_return_d1
   ; ALL: bb.1.entry:
   ; ALL:   liveins: $xmm0
   ; ALL:   [[COPY:%[0-9]+]]:_(s64) = COPY $xmm0
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.d
   ; ALL:   G_STORE [[COPY]](s64), [[FRAME_INDEX1]](p0) :: (store 8 into %ir.coerce.dive2)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.0, align 8), (load 1 from %ir.1, align 8)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.0, align 8), (load 1 from %ir.1, align 8)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 8 from %ir.coerce.dive13)
   ; ALL:   $xmm0 = COPY [[LOAD]](s64)
   ; ALL:   RET 0, implicit $xmm0
 entry:
   %retval = alloca %struct.d1, align 8
   %d = alloca %struct.d1, align 8
   %coerce.dive = getelementptr inbounds %struct.d1, %struct.d1* %d, i32 0, i32 0
   store double %d.coerce, double* %coerce.dive, align 8
define { double, double } @test_return_d2(double %d.coerce0, double %d.coerce1) {
   ; ALL:   [[COPY1:%[0-9]+]]:_(s64) = COPY $xmm1
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.d
   ; ALL:   G_STORE [[COPY]](s64), [[FRAME_INDEX1]](p0) :: (store 8 into %ir.1)
   ; ALL:   [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; ALL:   [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX1]], [[C1]](s64)
   ; ALL:   G_STORE [[COPY1]](s64), [[PTR_ADD]](p0) :: (store 8 into %ir.2)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.3, align 8), (load 1 from %ir.4, align 8)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.3, align 8), (load 1 from %ir.4, align 8)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 8 from %ir.5)
   ; ALL:   [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX]], [[C1]](s64)
   ; ALL:   [[LOAD1:%[0-9]+]]:_(s64) = G_LOAD [[PTR_ADD1]](p0) :: (dereferenceable load 8 from %ir.5 + 8)
   ; ALL:   $xmm0 = COPY [[LOAD]](s64)
   ; ALL:   $xmm1 = COPY [[LOAD1]](s64)
   ; ALL:   RET 0, implicit $xmm0, implicit $xmm1
 entry:
   %retval = alloca %struct.d2, align 8
define i32 @test_return_i1(i32 %i.coerce) {
   ; ALL-LABEL: name: test_return_i1
   ; ALL: bb.1.entry:
   ; ALL:   liveins: $edi
   ; ALL:   [[COPY:%[0-9]+]]:_(s32) = COPY $edi
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 4
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.i
   ; ALL:   G_STORE [[COPY]](s32), [[FRAME_INDEX1]](p0) :: (store 4 into %ir.coerce.dive2)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s32) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 4 from %ir.coerce.dive13)
   ; ALL:   $eax = COPY [[LOAD]](s32)
   ; ALL:   RET 0, implicit $eax
 entry:
   %retval = alloca %struct.i1, align 4
   %i = alloca %struct.i1, align 4
   %coerce.dive = getelementptr inbounds %struct.i1, %struct.i1* %i, i32 0, i32 0
   store i32 %i.coerce, i32* %coerce.dive, align 4
define i64 @test_return_i2(i64 %i.coerce) {
   ; ALL-LABEL: name: test_return_i2
   ; ALL: bb.1.entry:
   ; ALL:   liveins: $rdi
   ; ALL:   [[COPY:%[0-9]+]]:_(s64) = COPY $rdi
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.i
   ; ALL:   G_STORE [[COPY]](s64), [[FRAME_INDEX1]](p0) :: (store 8 into %ir.0, align 4)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.1, align 4), (load 1 from %ir.2, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.1, align 4), (load 1 from %ir.2, align 4)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 8 from %ir.3, align 4)
   ; ALL:   $rax = COPY [[LOAD]](s64)
   ; ALL:   RET 0, implicit $rax
 entry:
   %retval = alloca %struct.i2, align 4
   %i = alloca %struct.i2, align 4
   %0 = bitcast %struct.i2* %i to i64*
   store i64 %i.coerce, i64* %0, align 4
define { i64, i32 } @test_return_i3(i64 %i.coerce0, i32 %i.coerce1) {
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.i
   ; ALL:   [[FRAME_INDEX2:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.2.coerce
   ; ALL:   [[FRAME_INDEX3:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.3.tmp
   ; ALL:   G_STORE [[COPY]](s64), [[FRAME_INDEX2]](p0) :: (store 8 into %ir.0, align 4)
   ; ALL:   [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; ALL:   [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX2]], [[C1]](s64)
   ; ALL:   G_STORE [[COPY1]](s32), [[PTR_ADD]](p0) :: (store 4 into %ir.1)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX1]](p0), [[FRAME_INDEX2]](p0), [[C]](s64), 0 :: (store 1 into %ir.2, align 4), (load 1 from %ir.3, align 4)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.4, align 4), (load 1 from %ir.5, align 4)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX3]](p0), [[FRAME_INDEX]](p0), [[C]](s64), 0 :: (store 1 into %ir.6, align 8), (load 1 from %ir.7, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX1]](p0), [[FRAME_INDEX2]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.2, align 4), (load 1 from %ir.3, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.4, align 4), (load 1 from %ir.5, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX3]](p0), [[FRAME_INDEX]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.6, align 8), (load 1 from %ir.7, align 4)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[FRAME_INDEX3]](p0) :: (dereferenceable load 8 from %ir.tmp)
   ; ALL:   [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX3]], [[C1]](s64)
   ; ALL:   [[LOAD1:%[0-9]+]]:_(s32) = G_LOAD [[PTR_ADD1]](p0) :: (dereferenceable load 4 from %ir.tmp + 8, align 8)
   ; ALL:   $rax = COPY [[LOAD]](s64)
   ; ALL:   $edx = COPY [[LOAD1]](s32)
   ; ALL:   RET 0, implicit $rax, implicit $edx
 entry:
   %retval = alloca %struct.i3, align 4
define { i64, i64 } @test_return_i4(i64 %i.coerce0, i64 %i.coerce1) {
   ; ALL:   [[COPY1:%[0-9]+]]:_(s64) = COPY $rsi
   ; ALL:   [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
   ; ALL:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.retval
   ; ALL:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.i
   ; ALL:   G_STORE [[COPY]](s64), [[FRAME_INDEX1]](p0) :: (store 8 into %ir.1, align 4)
   ; ALL:   [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; ALL:   [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX1]], [[C1]](s64)
   ; ALL:   G_STORE [[COPY1]](s64), [[PTR_ADD]](p0) :: (store 8 into %ir.2, align 4)
-  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0 :: (store 1 into %ir.3, align 4), (load 1 from %ir.4, align 4)
+  ; ALL:   G_MEMCPY [[FRAME_INDEX]](p0), [[FRAME_INDEX1]](p0), [[C]](s64), 0, 0 :: (store 1 into %ir.3, align 4), (load 1 from %ir.4, align 4)
   ; ALL:   [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[FRAME_INDEX]](p0) :: (dereferenceable load 8 from %ir.5, align 4)
   ; ALL:   [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[FRAME_INDEX]], [[C1]](s64)
   ; ALL:   [[LOAD1:%[0-9]+]]:_(s64) = G_LOAD [[PTR_ADD1]](p0) :: (dereferenceable load 8 from %ir.5 + 8, align 4)
   ; ALL:   $rax = COPY [[LOAD]](s64)
   ; ALL:   $rdx = COPY [[LOAD1]](s64)
   ; ALL:   RET 0, implicit $rax, implicit $rdx
 entry:
   %retval = alloca %struct.i4, align 4

llvm/test/MachineVerifier/test_g_bzero.mir

 body:             |
   bb.0:
 
     %ptr:_(p0) = G_IMPLICIT_DEF
     %cst1:_(s64) = G_CONSTANT i64 4
     %cst2:_(s8) = G_CONSTANT i8 7
 
     ; CHECK: *** Bad machine code: bzero must have 1 memory operand ***
-    G_BZERO %ptr, %cst2, 0
+    G_BZERO %ptr, %cst2, 0, 0
 
     ; CHECK: *** Bad machine code: bzero memory operand must be a store ***
-    G_BZERO %ptr, %cst2, 0 :: (load 4)
+    G_BZERO %ptr, %cst2, 0, 0 :: (load 4)
 
     ; CHECK: *** Bad machine code: Missing mayLoad flag ***
     ; CHECK: *** Bad machine code: bzero memory operand must be a store ***
-    G_BZERO %ptr, %cst2, 0 :: (load store 4)
+    G_BZERO %ptr, %cst2, 0, 0 :: (load store 4)
 
     ; CHECK: *** Bad machine code: inconsistent bzero address space ***
-    G_BZERO %ptr, %cst2, 0 :: (store 4, addrspace 1)
+    G_BZERO %ptr, %cst2, 0, 0 :: (store 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: bzero operand must be a pointer ***
-    G_BZERO %cst1, %cst2, 0 :: (store 4)
+    G_BZERO %cst1, %cst2, 0, 0 :: (store 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (second to last operand) must be an immediate type ***
+    G_BZERO %ptr, %cst2, %cst1, 0 :: (store 4)
+
+    ; CHECK: *** Bad machine code: 'inlined' flag (last operand) must be an immediate type ***
+    G_BZERO %ptr, %cst2, 0, %cst1 :: (store 4)
 ...

llvm/test/MachineVerifier/test_g_memcpy.mir

 #RUN: not --crash llc -o - -march=arm64 -run-pass=none -verify-machineinstrs %s 2>&1 | FileCheck %s
 # REQUIRES: aarch64-registered-target
 ---
 name:            test_memcpy
 legalized:       true
 regBankSelected: false
 selected:        false
 tracksRegLiveness: true
 liveins:
 body:             |
   bb.0:
 
     %0:_(p0) = G_CONSTANT i64 0
     %1:_(p0) = G_CONSTANT i64 4
     %2:_(s64) = G_CONSTANT i64 4
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0
+    G_MEMCPY %0, %1, %2, 0, 0
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (load 4)
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store 4)
 
     ; CHECK: *** Bad machine code: wrong memory operand types ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4), (store 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (load 4), (store 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store 8), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store unknown-size), (load 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store unknown-size), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load unknown-size)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store 8), (load unknown-size)
 
     ; CHECK: *** Bad machine code: inconsistent store address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4, addrspace 1), (load 4)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store 4, addrspace 1), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent load address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4), (load 4, addrspace 1)
+    G_MEMCPY %0, %1, %2, 0, 0 :: (store 4), (load 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %2, %0, %2, 0 :: (store 4), (load 4)
+    G_MEMCPY %2, %0, %2, 0, 0 :: (store 4), (load 4)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %0, %2, %2, 0 :: (store 4), (load 4)
+    G_MEMCPY %0, %2, %2, 0, 0 :: (store 4), (load 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (operand 3) must be an immediate type ***
+    G_MEMCPY %0, %0, %2, %0, 0 :: (store 4), (load 4)
+
+    ; CHECK: *** Bad machine code: 'inlined' flag (operand 4) must be an immediate type ***
+    G_MEMCPY %0, %0, %2, 0, %0 :: (store 4), (load 4)
 ...

llvm/test/MachineVerifier/test_g_memset.mir

 body:             |
   bb.0:
 
     %0:_(p0) = G_CONSTANT i64 0
     %1:_(s64) = G_CONSTANT i64 4
     %2:_(s8) = G_CONSTANT i8 7
 
     ; CHECK: *** Bad machine code: memset must have 1 memory operand ***
-    G_MEMSET %0, %1, %2, 0
+    G_MEMSET %0, %1, %2, 0, 0
 
     ; CHECK: *** Bad machine code: memset memory operand must be a store ***
-    G_MEMSET %0, %1, %2, 0 :: (load 4)
+    G_MEMSET %0, %1, %2, 0, 0 :: (load 4)
 
     ; CHECK: *** Bad machine code: Missing mayLoad flag ***
     ; CHECK: *** Bad machine code: memset memory operand must be a store ***
-    G_MEMSET %0, %1, %2, 0 :: (load store 4)
+    G_MEMSET %0, %1, %2, 0, 0 :: (load store 4)
 
     ; CHECK: *** Bad machine code: inconsistent memset address space ***
-    G_MEMSET %0, %1, %2, 0 :: (store 4, addrspace 1)
+    G_MEMSET %0, %1, %2, 0, 0 :: (store 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: memset operand must be a pointer ***
-    G_MEMSET %1, %1, %2, 0 :: (store 4)
+    G_MEMSET %1, %1, %2, 0, 0 :: (store 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (second to last operand) must be an immediate type ***
+    G_MEMSET %0, %1, %2, %2, 0 :: (store 4)
+
+    ; CHECK: *** Bad machine code: 'inlined' flag (last operand) must be an immediate type ***
+    G_MEMSET %0, %1, %2, 0, %2 :: (store 4)
 ...