[GISel] Support llvm.memcpy.inline

[jroelofs]

llvm/test/MachineVerifier/test_g_memcpy.mir

16

delete me

[jroelofs]

• clang-format the patch
• Delete stray declaration
• Add IRTranslator test

[paquette]

llvm/include/llvm/IR/IntrinsicInst.h

895 ↗

(On Diff #355232)

Looks like this is used in a few passes too (e.g. the MemCpyOptimizer pass). Does this change impact those passes at all?

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Probably worth a comment for folks not familiar with the code.

It'd probably clearer to be able to call a inlineMemCpy function here and just return true, but I'm not sure if that's easy right now.

llvm/lib/CodeGen/MachineVerifier.cpp

1510

I think we only expect 0 or 1 for these values, so maybe we should check the value too?

[jroelofs]

llvm/include/llvm/IR/IntrinsicInst.h

895 ↗

(On Diff #355232)

Yes. Maybe I ought to put that in its own diff.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Good point. I'll give that a try.

llvm/lib/CodeGen/MachineVerifier.cpp

1510

Good call!

[aemerson]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39 ↗

(On Diff #355232)

If only memcpy has an inline variant then we shouldn't be generating G_MEMSETs with this extra argument?

[jroelofs]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39 ↗

(On Diff #355232)

I was on the fence about whether to let them all have the same number of arguments or not, since there are several places where code is shared between different opcodes and having them have different operand counts would lead to lots of special case logic.

To strike a balance, maybe it should only be memcpy+memmove instead?

[aemerson]

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39 ↗

(On Diff #355232)

Why add to memmove though?

Another way to implement this is to just introduce a new G_MEMCPY_INLINE opcode, then we don't have to add another argument anywhere.

[arsenm]

llvm/include/llvm/IR/IntrinsicInst.h

895 ↗

(On Diff #355232)

Yes, this should be its own patch. I started doing this a few months ago but forgot about it

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1357

Probably should use uint64_t?

llvm/test/CodeGen/AArch64/GlobalISel/debug-loc-legalize-tail-call.mir

39 ↗

(On Diff #355232)

I also think this should be a separate opcode.

There's also the difference that memcpy.inline uses an immarg size argument (which I think is entirely pointless and should be removed, but that's a different battle)

[jroelofs]

Make a new opcode: G_MEMCPY_INLINE, instead of adding extra arguments to existing opcodes.

[aemerson]

Why change the existing memcpy test? I would have thought the logical way to test this was to use a G_MEMCPY_INLINE with a length much larger than the heuristics would normally allow for G_MEMCPY expansion, rather than change existing test coverage.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1347

Dynamically sized memcpy.inline isn't legal according to the langref, so we should probably assert here.

llvm/test/CodeGen/AArch64/GlobalISel/inline-memcpy-opt.mir

51 ↗

(On Diff #355417)

This IR section is probably unnecessary.

[jroelofs]

I'll rename the two tests to make it clear the existing one wasn't changed, but what happened is (roughly):

cp inline-memcpy.mir inline-memcpy-opt.mir
sed -e 's/G_MEMCPY/G_MEMCPY_INLINE/' -e 's/memcpy\./memcpy.inline/g' inline-memcpy.mir
update_mir_test_checks.py inline-memcpy.mir

[paquette]

Can you add an entry on G_MEMCPY_INLINE to docs/GlobalISel/GenericOpcode.rst?

[aemerson]

LGTM.

[dyung]

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

632

In the future, please use a regex to match any number instead of hard coding a number in here as it causes the test to fail if a downstream copy has private changes which affect which number is assigned to this opcode.

I've fixed this up in f737d9794a40c066c9ccffb9ac277c1b70442ede.

[jroelofs]

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

632

Sorry about that. Thank you!

llvm/docs/GlobalISel/GenericOpcode.rst

 
 G_FENCE
 ^^^^^^^
 
 .. caution::
 
   I couldn't find any documentation on this at the time of writing.
 
+G_MEMCPY
+^^^^^^^^
+
+Generic memcpy. Expects two MachineMemOperands covering the store and load
+respectively, in addition to explicit operands.
+
+G_MEMCPY_INLINE
+^^^^^^^^^^^^^^^
+
+Generic inlined memcpy. Like G_MEMCPY, but it is guaranteed that this version
+will not be lowered as a call to an external function. Currently the size
+operand is required to evaluate as a constant (not an immediate), though that is
+expected to change when llvm.memcpy.inline is taught to support dynamic sizes.
+
+G_MEMMOVE
+^^^^^^^^^
+
+Generic memmove. Similar to G_MEMCPY, but the source and destination memory
+ranges are allowed to overlap.
+
+G_MEMSET
+^^^^^^^^
+
+Generic memset. Expects a MachineMemOperand in addition to explicit operands.
+
+G_BZERO
+^^^^^^^
+
+Generic bzero. Expects a MachineMemOperand in addition to explicit operands.
+
 Control Flow
 ------------
 
 G_PHI
 ^^^^^
 
 Implement the φ node in the SSA graph representing the function.
 

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
   /// Match: and (lshr x, cst), mask -> ubfx x, cst, width
   bool matchBitfieldExtractFromAnd(
       MachineInstr &MI, std::function<void(MachineIRBuilder &)> &MatchInfo);
 
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
+  /// Emit loads and stores that perform the given memcpy.
+  /// Assumes \p MI is a G_MEMCPY_INLINE
+  /// TODO: implement dynamically sized inline memcpy,
+  ///       and rename: s/bool tryEmit/void emit/
+  bool tryEmitMemcpyInline(MachineInstr &MI);
+
 private:
   // Memcpy family optimization helpers.
-  bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
-                      unsigned KnownLen, Align DstAlign, Align SrcAlign,
+  bool tryEmitMemcpyInline(MachineInstr &MI, Register Dst, Register Src,
+                           uint64_t KnownLen, Align DstAlign, Align SrcAlign,
                            bool IsVolatile);
+  bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
+                      uint64_t KnownLen, uint64_t Limit, Align DstAlign,
+                      Align SrcAlign, bool IsVolatile);
   bool optimizeMemmove(MachineInstr &MI, Register Dst, Register Src,
-                       unsigned KnownLen, Align DstAlign, Align SrcAlign,
+                       uint64_t KnownLen, Align DstAlign, Align SrcAlign,
                        bool IsVolatile);
   bool optimizeMemset(MachineInstr &MI, Register Dst, Register Val,
-                      unsigned KnownLen, Align DstAlign, bool IsVolatile);
+                      uint64_t KnownLen, Align DstAlign, bool IsVolatile);
 
   /// Given a non-indexed load or store instruction \p MI, find an offset that
   /// can be usefully and legally folded into it as a post-indexing operation.
   ///
   /// \returns true if a candidate is found.
   bool findPostIndexCandidate(MachineInstr &MI, Register &Addr, Register &Base,
                               Register &Offset);
 

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(G_READ_REGISTER)
 
 /// write_register intrinsic
 HANDLE_TARGET_OPCODE(G_WRITE_REGISTER)
 
 /// llvm.memcpy intrinsic
 HANDLE_TARGET_OPCODE(G_MEMCPY)
 
+/// llvm.memcpy.inline intrinsic
+HANDLE_TARGET_OPCODE(G_MEMCPY_INLINE)
+
 /// llvm.memmove intrinsic
 HANDLE_TARGET_OPCODE(G_MEMMOVE)
 
 /// llvm.memset intrinsic
 HANDLE_TARGET_OPCODE(G_MEMSET)
 HANDLE_TARGET_OPCODE(G_BZERO)
 
 /// Vector reductions

llvm/include/llvm/Target/GenericOpcodes.td

 def G_MEMCPY : GenericInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall);
   let hasSideEffects = false;
   let mayLoad = true;
   let mayStore = true;
 }
 
+def G_MEMCPY_INLINE : GenericInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size);
+  let hasSideEffects = false;
+  let mayLoad = true;
+  let mayStore = true;
+}
+
 def G_MEMMOVE : GenericInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins ptype0:$dst_addr, ptype1:$src_addr, type2:$size, untyped_imm_0:$tailcall);
   let hasSideEffects = false;
   let mayLoad = true;
   let mayStore = true;
 }
 

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

static Register getMemsetValue(Register Val, LLT Ty, MachineIRBuilder &MIB) {
   // For vector types create a G_BUILD_VECTOR.
   if (Ty.isVector())
     Val = MIB.buildSplatVector(Ty, Val).getReg(0);
 
   return Val;
 }
 
 bool CombinerHelper::optimizeMemset(MachineInstr &MI, Register Dst,
-                                    Register Val, unsigned KnownLen,
+                                    Register Val, uint64_t KnownLen,
                                     Align Alignment, bool IsVolatile) {
   auto &MF = *MI.getParent()->getParent();
   const auto &TLI = *MF.getSubtarget().getTargetLowering();
   auto &DL = MF.getDataLayout();
   LLVMContext &C = MF.getFunction().getContext();
 
   assert(KnownLen != 0 && "Have a zero length memset length!");
 
for (unsigned I = 0; I < MemOps.size(); I++) {
     DstOff += Ty.getSizeInBytes();
     Size -= TySize;
   }
 
   MI.eraseFromParent();
   return true;
 }
 
-bool CombinerHelper::optimizeMemcpy(MachineInstr &MI, Register Dst,
-                                    Register Src, unsigned KnownLen,
+bool CombinerHelper::tryEmitMemcpyInline(MachineInstr &MI) {
+  assert(MI.getOpcode() == TargetOpcode::G_MEMCPY_INLINE);
+
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src = MI.getOperand(1).getReg();
+  Register Len = MI.getOperand(2).getReg();
+
+  const auto *MMOIt = MI.memoperands_begin();
+  const MachineMemOperand *MemOp = *MMOIt;
+  bool IsVolatile = MemOp->isVolatile();
+
+  // See if this is a constant length copy
+  auto LenVRegAndVal = getConstantVRegValWithLookThrough(Len, MRI);
+  // FIXME: support dynamically sized G_MEMCPY_INLINE
+  assert(LenVRegAndVal.hasValue() &&
+         "inline memcpy with dynamic size is not yet supported");
+  uint64_t KnownLen = LenVRegAndVal->Value.getZExtValue();
+  if (KnownLen == 0) {
+    MI.eraseFromParent();
+    return true;
+  }
+
+  const auto &DstMMO = **MI.memoperands_begin();
+  const auto &SrcMMO = **std::next(MI.memoperands_begin());
+  Align DstAlign = DstMMO.getBaseAlign();
+  Align SrcAlign = SrcMMO.getBaseAlign();
+
+  return tryEmitMemcpyInline(MI, Dst, Src, KnownLen, DstAlign, SrcAlign,
+                             IsVolatile);
+}
+
+bool CombinerHelper::tryEmitMemcpyInline(MachineInstr &MI, Register Dst,
+                                         Register Src, uint64_t KnownLen,
                                          Align DstAlign, Align SrcAlign,
                                          bool IsVolatile) {
+  assert(MI.getOpcode() == TargetOpcode::G_MEMCPY_INLINE);
+  return optimizeMemcpy(MI, Dst, Src, KnownLen,
+                        std::numeric_limits<uint64_t>::max(), DstAlign,
+                        SrcAlign, IsVolatile);
+}
+
+bool CombinerHelper::optimizeMemcpy(MachineInstr &MI, Register Dst,
+                                    Register Src, uint64_t KnownLen,
+                                    uint64_t Limit, Align DstAlign,
+                                    Align SrcAlign, bool IsVolatile) {
   auto &MF = *MI.getParent()->getParent();
   const auto &TLI = *MF.getSubtarget().getTargetLowering();
   auto &DL = MF.getDataLayout();
   LLVMContext &C = MF.getFunction().getContext();
 
   assert(KnownLen != 0 && "Have a zero length memcpy length!");
 
   bool DstAlignCanChange = false;
   MachineFrameInfo &MFI = MF.getFrameInfo();
-  bool OptSize = shouldLowerMemFuncForSize(MF);
   Align Alignment = commonAlignment(DstAlign, SrcAlign);
 
   MachineInstr *FIDef = getOpcodeDef(TargetOpcode::G_FRAME_INDEX, Dst, MRI);
   if (FIDef && !MFI.isFixedObjectIndex(FIDef->getOperand(1).getIndex()))
     DstAlignCanChange = true;
 
   // FIXME: infer better src pointer alignment like SelectionDAG does here.
   // FIXME: also use the equivalent of isMemSrcFromConstant and alwaysinlining
   // if the memcpy is in a tail call position.
 
-  unsigned Limit = TLI.getMaxStoresPerMemcpy(OptSize);
   std::vector<LLT> MemOps;
 
   const auto &DstMMO = **MI.memoperands_begin();
   const auto &SrcMMO = **std::next(MI.memoperands_begin());
   MachinePointerInfo DstPtrInfo = DstMMO.getPointerInfo();
   MachinePointerInfo SrcPtrInfo = SrcMMO.getPointerInfo();
 
   if (!findGISelOptimalMemOpLowering(
for (auto CopyTy : MemOps) {
     Size -= CopyTy.getSizeInBytes();
   }
 
   MI.eraseFromParent();
   return true;
 }
 
 bool CombinerHelper::optimizeMemmove(MachineInstr &MI, Register Dst,
-                                     Register Src, unsigned KnownLen,
+                                     Register Src, uint64_t KnownLen,
                                      Align DstAlign, Align SrcAlign,
                                      bool IsVolatile) {
   auto &MF = *MI.getParent()->getParent();
   const auto &TLI = *MF.getSubtarget().getTargetLowering();
   auto &DL = MF.getDataLayout();
   LLVMContext &C = MF.getFunction().getContext();
 
   assert(KnownLen != 0 && "Have a zero length memmove length!");
bool CombinerHelper::tryCombineMemCpyFamily(MachineInstr &MI, unsigned MaxLen) {
   const unsigned Opc = MI.getOpcode();
   // This combine is fairly complex so it's not written with a separate
   // matcher function.
   assert((Opc == TargetOpcode::G_MEMCPY || Opc == TargetOpcode::G_MEMMOVE ||
           Opc == TargetOpcode::G_MEMSET) && "Expected memcpy like instruction");
 
   auto MMOIt = MI.memoperands_begin();
   const MachineMemOperand *MemOp = *MMOIt;
-  bool IsVolatile = MemOp->isVolatile();
-  // Don't try to optimize volatile.
-  if (IsVolatile)
-    return false;
 
   Align DstAlign = MemOp->getBaseAlign();
   Align SrcAlign;
   Register Dst = MI.getOperand(0).getReg();
   Register Src = MI.getOperand(1).getReg();
   Register Len = MI.getOperand(2).getReg();
 
   if (Opc != TargetOpcode::G_MEMSET) {
     assert(MMOIt != MI.memoperands_end() && "Expected a second MMO on MI");
     MemOp = *(++MMOIt);
     SrcAlign = MemOp->getBaseAlign();
   }
 
   // See if this is a constant length copy
   auto LenVRegAndVal = getConstantVRegValWithLookThrough(Len, MRI);
   if (!LenVRegAndVal)
     return false; // Leave it to the legalizer to lower it to a libcall.
-  unsigned KnownLen = LenVRegAndVal->Value.getZExtValue();
+  uint64_t KnownLen = LenVRegAndVal->Value.getZExtValue();
 
   if (KnownLen == 0) {
     MI.eraseFromParent();
     return true;
   }
 
+  bool IsVolatile = MemOp->isVolatile();
+  if (Opc == TargetOpcode::G_MEMCPY_INLINE)
+    return tryEmitMemcpyInline(MI, Dst, Src, KnownLen, DstAlign, SrcAlign,
+                               IsVolatile);
+
+  // Don't try to optimize volatile.
+  if (IsVolatile)
+    return false;
+
   if (MaxLen && KnownLen > MaxLen)
     return false;
 
-  if (Opc == TargetOpcode::G_MEMCPY)
-    return optimizeMemcpy(MI, Dst, Src, KnownLen, DstAlign, SrcAlign, IsVolatile);
+  if (Opc == TargetOpcode::G_MEMCPY) {
+    auto &MF = *MI.getParent()->getParent();
+    const auto &TLI = *MF.getSubtarget().getTargetLowering();
+    bool OptSize = shouldLowerMemFuncForSize(MF);
+    uint64_t Limit = TLI.getMaxStoresPerMemcpy(OptSize);
+    return optimizeMemcpy(MI, Dst, Src, KnownLen, Limit, DstAlign, SrcAlign,
+                          IsVolatile);
+  }
   if (Opc == TargetOpcode::G_MEMMOVE)
     return optimizeMemmove(MI, Dst, Src, KnownLen, DstAlign, SrcAlign, IsVolatile);
   if (Opc == TargetOpcode::G_MEMSET)
     return optimizeMemset(MI, Dst, Src, KnownLen, DstAlign, IsVolatile);
   return false;
 }
 
 static Optional<APFloat> constantFoldFpUnary(unsigned Opcode, LLT DstTy,

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateMemFunc(const CallInst &CI,
   Align SrcAlign;
   unsigned IsVol =
       cast<ConstantInt>(CI.getArgOperand(CI.getNumArgOperands() - 1))
           ->getZExtValue();
 
   if (auto *MCI = dyn_cast<MemCpyInst>(&CI)) {
     DstAlign = MCI->getDestAlign().valueOrOne();
     SrcAlign = MCI->getSourceAlign().valueOrOne();
+  } else if (auto *MCI = dyn_cast<MemCpyInlineInst>(&CI)) {
+    DstAlign = MCI->getDestAlign().valueOrOne();
+    SrcAlign = MCI->getSourceAlign().valueOrOne();
   } else if (auto *MMI = dyn_cast<MemMoveInst>(&CI)) {
     DstAlign = MMI->getDestAlign().valueOrOne();
     SrcAlign = MMI->getSourceAlign().valueOrOne();
   } else {
     auto *MSI = cast<MemSetInst>(&CI);
     DstAlign = MSI->getDestAlign().valueOrOne();
   }
 
+  if (Opcode != TargetOpcode::G_MEMCPY_INLINE) {
     // We need to propagate the tail call flag from the IR inst as an argument.
     // Otherwise, we have to pessimize and assume later that we cannot tail call
     // any memory intrinsics.
     ICall.addImm(CI.isTailCall() ? 1 : 0);
+  }
 
   // Create mem operands to store the alignment and volatile info.
   auto VolFlag = IsVol ? MachineMemOperand::MOVolatile : MachineMemOperand::MONone;
   ICall.addMemOperand(MF->getMachineMemOperand(
       MachinePointerInfo(CI.getArgOperand(0)),
       MachineMemOperand::MOStore | VolFlag, 1, DstAlign));
   if (Opcode != TargetOpcode::G_MEMSET)
     ICall.addMemOperand(MF->getMachineMemOperand(
MIRBuilder.buildFPExt(getOrCreateVReg(CI),
                           MachineInstr::copyFlagsFromInstruction(CI));
     return true;
   case Intrinsic::convert_to_fp16:
     // FIXME: This intrinsic should probably be removed from the IR.
     MIRBuilder.buildFPTrunc(getOrCreateVReg(CI),
                             getOrCreateVReg(*CI.getArgOperand(0)),
                             MachineInstr::copyFlagsFromInstruction(CI));
     return true;
+  case Intrinsic::memcpy_inline:
+    return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMCPY_INLINE);
   case Intrinsic::memcpy:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMCPY);
   case Intrinsic::memmove:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMMOVE);
   case Intrinsic::memset:
     return translateMemFunc(CI, MIRBuilder, TargetOpcode::G_MEMSET);
   case Intrinsic::eh_typeid_for: {
     GlobalValue *GV = ExtractTypeInfo(CI.getArgOperand(0));

llvm/lib/CodeGen/MachineVerifier.cpp

case TargetOpcode::G_DYN_STACKALLOC: {
     }
 
     if (!AlignOp.isImm()) {
       report("src operand 2 must be an immediate type", MI);
       break;
     }
     break;
   }
+  case TargetOpcode::G_MEMCPY_INLINE:
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE: {
     ArrayRef<MachineMemOperand *> MMOs = MI->memoperands();
     if (MMOs.size() != 2) {
       report("memcpy/memmove must have 2 memory operands", MI);
       break;
     }
 
if (!DstPtrTy.isPointer() || !SrcPtrTy.isPointer()) {
       report("memory instruction operand must be a pointer", MI);
       break;
     }
 
     if (DstPtrTy.getAddressSpace() != MMOs[0]->getAddrSpace())
       report("inconsistent store address space", MI);
     if (SrcPtrTy.getAddressSpace() != MMOs[1]->getAddrSpace())
       report("inconsistent load address space", MI);
 
+    if (Opc != TargetOpcode::G_MEMCPY_INLINE)
+      if (!MI->getOperand(3).isImm() || (MI->getOperand(3).getImm() & ~1LL))
+        report("'tail' flag (operand 3) must be an immediate 0 or 1", MI);
+
     break;
   }
   case TargetOpcode::G_BZERO:
   case TargetOpcode::G_MEMSET: {
     ArrayRef<MachineMemOperand *> MMOs = MI->memoperands();
     std::string Name = Opc == TargetOpcode::G_MEMSET ? "memset" : "bzero";
     if (MMOs.size() != 1) {
       report(Twine(Name, " must have 1 memory operand"), MI);
case TargetOpcode::G_MEMSET: {
     if (!DstPtrTy.isPointer()) {
       report(Twine(Name, " operand must be a pointer"), MI);
       break;
     }
 
     if (DstPtrTy.getAddressSpace() != MMOs[0]->getAddrSpace())
       report("inconsistent " + Twine(Name, " address space"), MI);
 
+    if (!MI->getOperand(MI->getNumOperands() - 1).isImm() ||
+        (MI->getOperand(MI->getNumOperands() - 1).getImm() & ~1LL))
+      report("'tail' flag (last operand) must be an immediate 0 or 1", MI);
+
     break;
   }
   case TargetOpcode::G_VECREDUCE_SEQ_FADD:
   case TargetOpcode::G_VECREDUCE_SEQ_FMUL: {
     LLT DstTy = MRI->getType(MI->getOperand(0).getReg());
     LLT Src1Ty = MRI->getType(MI->getOperand(1).getReg());
     LLT Src2Ty = MRI->getType(MI->getOperand(2).getReg());
     if (!DstTy.isScalar())

llvm/lib/Target/AArch64/GISel/AArch64O0PreLegalizerCombiner.cpp

if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
+  case TargetOpcode::G_MEMCPY_INLINE:
+    return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
     // At -O0 set a maxlen of 32 to inline;
     unsigned MaxLen = 32;
     // Try to inline memcpy type calls if optimizations are enabled.
     if (Helper.tryCombineMemCpyFamily(MI, MaxLen))
       return true;

llvm/lib/Target/AArch64/GISel/AArch64PreLegalizerCombiner.cpp

if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
+  case TargetOpcode::G_MEMCPY_INLINE:
+    return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
     // If we're at -O0 set a maxlen of 32 to inline, otherwise let the other
     // heuristics decide.
     unsigned MaxLen = EnableOpt ? 0 : 32;
     // Try to inline memcpy type calls if optimizations are enabled.
     if (Helper.tryCombineMemCpyFamily(MI, MaxLen))

llvm/lib/Target/AMDGPU/AMDGPUPreLegalizerCombiner.cpp

bool AMDGPUPreLegalizerCombinerInfo::combine(GISelChangeObserver &Observer,
   AMDGPUPreLegalizerCombinerHelper PreLegalizerHelper(B, Helper);
   AMDGPUGenPreLegalizerCombinerHelper Generated(GeneratedRuleCfg, Helper,
                                                 PreLegalizerHelper);
 
   if (Generated.tryCombineAll(Observer, MI, B, Helper))
     return true;
 
   switch (MI.getOpcode()) {
+  case TargetOpcode::G_MEMCPY_INLINE:
+    return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
   }
 
   return false;
 }

llvm/lib/Target/Mips/MipsPreLegalizerCombiner.cpp

 bool MipsPreLegalizerCombinerInfo::combine(GISelChangeObserver &Observer,
                                            MachineInstr &MI,
                                            MachineIRBuilder &B) const {
   CombinerHelper Helper(Observer, B);
 
   switch (MI.getOpcode()) {
   default:
     return false;
+  case TargetOpcode::G_MEMCPY_INLINE:
+    return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_LOAD:
   case TargetOpcode::G_SEXTLOAD:
   case TargetOpcode::G_ZEXTLOAD: {
     // Don't attempt to combine non power of 2 loads or unaligned loads when
     // subtarget doesn't support them.
     auto MMO = *MI.memoperands_begin();
     const MipsSubtarget &STI =
         static_cast<const MipsSubtarget &>(MI.getMF()->getSubtarget());

llvm/test/CodeGen/AArch64/GlobalISel/inline-memcpy-forced.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=aarch64 -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs %s -o - | FileCheck %s
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "arm64-apple-darwin"
+
+  declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg) #0
+
+  define void @test_memcpy_inline(i32* nocapture %dst, i32* nocapture readonly %src) local_unnamed_addr {
+  entry:
+    %0 = bitcast i32* %dst to i8*
+    %1 = bitcast i32* %src to i8*
+    tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 143, i1 false)
+    ret void
+  }
+
+  attributes #0 = { argmemonly nounwind }
+
+...
+---
+name:            test_memcpy_inline
+alignment:       4
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: _ }
+machineFunctionInfo: {}
+body:             |
+  bb.1.entry:
+    liveins: $x0, $x1
+
+    ; CHECK-LABEL: name: test_memcpy_inline
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+    ; CHECK: [[LOAD:%[0-9]+]]:_(s128) = G_LOAD [[COPY1]](p0) :: (load 16 from %ir.1, align 4)
+    ; CHECK: G_STORE [[LOAD]](s128), [[COPY]](p0) :: (store 16 into %ir.0, align 4)
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD]](p0) :: (load 16 from %ir.1 + 16, align 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s128), [[PTR_ADD1]](p0) :: (store 16 into %ir.0 + 16, align 4)
+    ; CHECK: [[C1:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
+    ; CHECK: [[PTR_ADD2:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C1]](s64)
+    ; CHECK: [[LOAD2:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD2]](p0) :: (load 16 from %ir.1 + 32, align 4)
+    ; CHECK: [[PTR_ADD3:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C1]](s64)
+    ; CHECK: G_STORE [[LOAD2]](s128), [[PTR_ADD3]](p0) :: (store 16 into %ir.0 + 32, align 4)
+    ; CHECK: [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 48
+    ; CHECK: [[PTR_ADD4:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C2]](s64)
+    ; CHECK: [[LOAD3:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD4]](p0) :: (load 16 from %ir.1 + 48, align 4)
+    ; CHECK: [[PTR_ADD5:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C2]](s64)
+    ; CHECK: G_STORE [[LOAD3]](s128), [[PTR_ADD5]](p0) :: (store 16 into %ir.0 + 48, align 4)
+    ; CHECK: [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
+    ; CHECK: [[PTR_ADD6:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C3]](s64)
+    ; CHECK: [[LOAD4:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD6]](p0) :: (load 16 from %ir.1 + 64, align 4)
+    ; CHECK: [[PTR_ADD7:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
+    ; CHECK: G_STORE [[LOAD4]](s128), [[PTR_ADD7]](p0) :: (store 16 into %ir.0 + 64, align 4)
+    ; CHECK: [[C4:%[0-9]+]]:_(s64) = G_CONSTANT i64 80
+    ; CHECK: [[PTR_ADD8:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C4]](s64)
+    ; CHECK: [[LOAD5:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD8]](p0) :: (load 16 from %ir.1 + 80, align 4)
+    ; CHECK: [[PTR_ADD9:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C4]](s64)
+    ; CHECK: G_STORE [[LOAD5]](s128), [[PTR_ADD9]](p0) :: (store 16 into %ir.0 + 80, align 4)
+    ; CHECK: [[C5:%[0-9]+]]:_(s64) = G_CONSTANT i64 96
+    ; CHECK: [[PTR_ADD10:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C5]](s64)
+    ; CHECK: [[LOAD6:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD10]](p0) :: (load 16 from %ir.1 + 96, align 4)
+    ; CHECK: [[PTR_ADD11:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C5]](s64)
+    ; CHECK: G_STORE [[LOAD6]](s128), [[PTR_ADD11]](p0) :: (store 16 into %ir.0 + 96, align 4)
+    ; CHECK: [[C6:%[0-9]+]]:_(s64) = G_CONSTANT i64 112
+    ; CHECK: [[PTR_ADD12:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C6]](s64)
+    ; CHECK: [[LOAD7:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD12]](p0) :: (load 16 from %ir.1 + 112, align 4)
+    ; CHECK: [[PTR_ADD13:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C6]](s64)
+    ; CHECK: G_STORE [[LOAD7]](s128), [[PTR_ADD13]](p0) :: (store 16 into %ir.0 + 112, align 4)
+    ; CHECK: [[C7:%[0-9]+]]:_(s64) = G_CONSTANT i64 127
+    ; CHECK: [[PTR_ADD14:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C7]](s64)
+    ; CHECK: [[LOAD8:%[0-9]+]]:_(s128) = G_LOAD [[PTR_ADD14]](p0) :: (load 16 from %ir.1 + 127, align 1, basealign 4)
+    ; CHECK: [[PTR_ADD15:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C7]](s64)
+    ; CHECK: G_STORE [[LOAD8]](s128), [[PTR_ADD15]](p0) :: (store 16 into %ir.0 + 127, align 1, basealign 4)
+    ; CHECK: RET_ReallyLR
+    %0:_(p0) = COPY $x0
+    %1:_(p0) = COPY $x1
+    %2:_(s64) = G_CONSTANT i64 143
+    G_MEMCPY_INLINE %0(p0), %1(p0), %2(s64) :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    RET_ReallyLR
+
+...

llvm/test/CodeGen/AArch64/GlobalISel/irtranslator-memcpy-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-unknown -global-isel -global-isel-abort=1 -verify-machineinstrs -stop-after=irtranslator %s -o - | FileCheck %s
+
+define void @copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @inline_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: inline_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY_INLINE [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64) :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 0 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @inline_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: inline_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY_INLINE [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64) :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @tail_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1 :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @tail_inline_copy(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_inline_copy
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY_INLINE [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64) :: (store 1 into %ir.dst), (load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 false)
+  ret void
+}
+
+define void @tail_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64), 1 :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+define void @tail_inline_copy_volatile(i8* %dst, i8* %src) {
+  ; CHECK-LABEL: name: tail_inline_copy_volatile
+  ; CHECK: bb.1.entry:
+  ; CHECK:   liveins: $x0, $x1
+  ; CHECK:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x1
+  ; CHECK:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 4
+  ; CHECK:   [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[C]](s32)
+  ; CHECK:   G_MEMCPY_INLINE [[COPY]](p0), [[COPY1]](p0), [[ZEXT]](s64) :: (volatile store 1 into %ir.dst), (volatile load 1 from %ir.src)
+  ; CHECK:   RET_ReallyLR
+entry:
+  tail call void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* %dst, i8* %src, i32 4, i1 true)
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i32(i8* nocapture writeonly, i8* nocapture readonly, i32, i1) nounwind
+declare void @llvm.memcpy.inline.p0i8.p0i8.i32(i8* nocapture writeonly, i8* nocapture readonly, i32, i1) nounwind

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_WRITE_REGISTER (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_MEMCPY (opcode {{[0-9]+}}): 3 type indices, 1 imm index
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
+# DEBUG-NEXT: G_MEMCPY_INLINE (opcode 219): 3 type indices, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_MEMMOVE (opcode {{[0-9]+}}): 3 type indices, 1 imm index
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: G_MEMSET (opcode {{[0-9]+}}): 3 type indices, 1 imm index
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}
 # DEBUG-NEXT: .. type index coverage check SKIPPED: user-defined predicate detected
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: user-defined predicate detected

llvm/test/CodeGen/AMDGPU/GlobalISel/prelegalizer-combiner-memcpy-inline.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -run-pass=amdgpu-prelegalizer-combiner -verify-machineinstrs -o - %s | FileCheck %s
+--- |
+  target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7"
+  target triple = "amdgcn-amd-amdhsa"
+
+  declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64 immarg, i1 immarg) #0
+
+  define void @test_memcpy_inline(i32* nocapture %dst, i32* nocapture readonly %src) local_unnamed_addr #1 {
+  entry:
+    %0 = bitcast i32* %dst to i8*
+    %1 = bitcast i32* %src to i8*
+    tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 13, i1 false)
+    ret void
+  }
+
+  attributes #0 = { argmemonly nofree nounwind willreturn "target-cpu"="gfx900" }
+  attributes #1 = { "target-cpu"="gfx900" }
+
+...
+---
+name:            test_memcpy_inline
+alignment:       1
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: sgpr_64 }
+  - { id: 3, class: _ }
+  - { id: 4, class: _ }
+  - { id: 5, class: _ }
+  - { id: 6, class: _ }
+  - { id: 7, class: _ }
+  - { id: 8, class: ccr_sgpr_64 }
+liveins:
+  - { reg: '$sgpr30_sgpr31', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo:
+  maxKernArgAlign: 1
+  scratchRSrcReg:  '$sgpr0_sgpr1_sgpr2_sgpr3'
+  frameOffsetReg:  '$sgpr33'
+  stackPtrOffsetReg: '$sgpr32'
+  argumentInfo:
+    privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' }
+  occupancy:       10
+body:             |
+  bb.1.entry:
+    liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $sgpr30_sgpr31
+
+    ; CHECK-LABEL: name: test_memcpy_inline
+    ; CHECK: liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $sgpr30_sgpr31
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $vgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $vgpr1
+    ; CHECK: [[MV:%[0-9]+]]:_(p0) = G_MERGE_VALUES [[COPY]](s32), [[COPY1]](s32)
+    ; CHECK: [[COPY2:%[0-9]+]]:_(s32) = COPY $vgpr2
+    ; CHECK: [[COPY3:%[0-9]+]]:_(s32) = COPY $vgpr3
+    ; CHECK: [[MV1:%[0-9]+]]:_(p0) = G_MERGE_VALUES [[COPY2]](s32), [[COPY3]](s32)
+    ; CHECK: [[COPY4:%[0-9]+]]:sgpr_64 = COPY $sgpr30_sgpr31
+    ; CHECK: [[LOAD:%[0-9]+]]:_(s64) = G_LOAD [[MV1]](p0) :: (load 8 from %ir.1, align 4)
+    ; CHECK: G_STORE [[LOAD]](s64), [[MV]](p0) :: (store 8 into %ir.0, align 4)
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 5
+    ; CHECK: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[MV1]], [[C]](s64)
+    ; CHECK: [[LOAD1:%[0-9]+]]:_(s64) = G_LOAD [[PTR_ADD]](p0) :: (load 8 from %ir.1 + 5, align 1, basealign 4)
+    ; CHECK: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[MV]], [[C]](s64)
+    ; CHECK: G_STORE [[LOAD1]](s64), [[PTR_ADD1]](p0) :: (store 8 into %ir.0 + 5, align 1, basealign 4)
+    ; CHECK: [[COPY5:%[0-9]+]]:ccr_sgpr_64 = COPY [[COPY4]]
+    ; CHECK: S_SETPC_B64_return [[COPY5]]
+    %3:_(s32) = COPY $vgpr0
+    %4:_(s32) = COPY $vgpr1
+    %0:_(p0) = G_MERGE_VALUES %3(s32), %4(s32)
+    %5:_(s32) = COPY $vgpr2
+    %6:_(s32) = COPY $vgpr3
+    %1:_(p0) = G_MERGE_VALUES %5(s32), %6(s32)
+    %2:sgpr_64 = COPY $sgpr30_sgpr31
+    %7:_(s64) = G_CONSTANT i64 13
+    G_MEMCPY_INLINE %0(p0), %1(p0), %7(s64) :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    %8:ccr_sgpr_64 = COPY %2
+    S_SETPC_B64_return %8
+
+...

llvm/test/CodeGen/Mips/GlobalISel/mips-prelegalizer-combiner/inline-memcpy.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -O0 -mtriple=mipsel-linux-gnu -run-pass=mips-prelegalizer-combiner -verify-machineinstrs %s -o - | FileCheck %s -check-prefixes=MIPS32
+--- |
+  ; ModuleID = '../llvm/test/CodeGen/Mips/GlobalISel/mips-prelegalizer-combiner/inline-memcpy.ll'
+  source_filename = "../llvm/test/CodeGen/Mips/GlobalISel/mips-prelegalizer-combiner/inline-memcpy.ll"
+  target datalayout = "e-m:m-p:32:32-i8:8:32-i16:16:32-i64:64-n32-S64"
+  target triple = "mipsel-pc-linux-gnu"
+
+  declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64 immarg, i1 immarg) #0
+
+  define void @test_memcpy_inline(i32* nocapture %dst, i32* nocapture readonly %src) local_unnamed_addr {
+  entry:
+    %0 = bitcast i32* %dst to i8*
+    %1 = bitcast i32* %src to i8*
+    tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 4 %0, i8* align 4 %1, i64 2, i1 false)
+    ret void
+  }
+
+  attributes #0 = { argmemonly nofree nounwind willreturn }
+
+...
+---
+name:            test_memcpy_inline
+alignment:       4
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: _ }
+  - { id: 3, class: _ }
+liveins:
+  - { reg: '$a0' }
+  - { reg: '$a1' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.1.entry:
+    liveins: $a0, $a1
+
+    ; MIPS32-LABEL: name: test_memcpy_inline
+    ; MIPS32: liveins: $a0, $a1
+    ; MIPS32: [[COPY:%[0-9]+]]:_(p0) = COPY $a0
+    ; MIPS32: [[COPY1:%[0-9]+]]:_(p0) = COPY $a1
+    ; MIPS32: [[LOAD:%[0-9]+]]:_(s8) = G_LOAD [[COPY1]](p0) :: (load 1 from %ir.1, align 4)
+    ; MIPS32: G_STORE [[LOAD]](s8), [[COPY]](p0) :: (store 1 into %ir.0, align 4)
+    ; MIPS32: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
+    ; MIPS32: [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY1]], [[C]](s32)
+    ; MIPS32: [[LOAD1:%[0-9]+]]:_(s8) = G_LOAD [[PTR_ADD]](p0) :: (load 1 from %ir.1 + 1, basealign 4)
+    ; MIPS32: [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C]](s32)
+    ; MIPS32: G_STORE [[LOAD1]](s8), [[PTR_ADD1]](p0) :: (store 1 into %ir.0 + 1, basealign 4)
+    ; MIPS32: RetRA
+    %0:_(p0) = COPY $a0
+    %1:_(p0) = COPY $a1
+    %2:_(s64) = G_CONSTANT i64 2
+    %3:_(s32) = G_TRUNC %2(s64)
+    G_MEMCPY_INLINE %0(p0), %1(p0), %3(s32) :: (store 1 into %ir.0, align 4), (load 1 from %ir.1, align 4)
+    RetRA
+
+...

llvm/test/MachineVerifier/test_g_bzero.mir

bb.0:
 
     ; CHECK: *** Bad machine code: Missing mayLoad flag ***
     ; CHECK: *** Bad machine code: bzero memory operand must be a store ***
     G_BZERO %ptr, %cst2, 0 :: (load store 4)
 
     ; CHECK: *** Bad machine code: inconsistent bzero address space ***
     G_BZERO %ptr, %cst2, 0 :: (store 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: bzero operand must be a pointer ***
     G_BZERO %cst1, %cst2, 0 :: (store 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (last operand) must be an immediate 0 or 1 ***
+    G_BZERO %ptr, %cst2, 2 :: (store 4)
 ...

llvm/test/MachineVerifier/test_g_memcpy.mir

 #RUN: not --crash llc -o - -march=arm64 -run-pass=none -verify-machineinstrs %s 2>&1 | FileCheck %s
 # REQUIRES: aarch64-registered-target
 ---
 name:            test_memcpy
 legalized:       true
 regBankSelected: false
 selected:        false
 tracksRegLiveness: true
 liveins:
 body:             |
   bb.0:
 
     %0:_(p0) = G_CONSTANT i64 0
     %1:_(p0) = G_CONSTANT i64 4
     %2:_(s64) = G_CONSTANT i64 4
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
     G_MEMCPY %0, %1, %2, 0
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
     G_MEMCPY %0, %1, %2, 0 :: (load 4)
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
     G_MEMCPY %0, %1, %2, 0 :: (store 4)
bb.0:
     G_MEMCPY %0, %1, %2, 0 :: (store 4), (load 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
     G_MEMCPY %2, %0, %2, 0 :: (store 4), (load 4)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
     G_MEMCPY %0, %2, %2, 0 :: (store 4), (load 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (operand 3) must be an immediate 0 or 1 ***
+    G_MEMCPY %0, %0, %2, %0 :: (store 4), (load 4)
+
+    ; CHECK: *** Bad machine code: 'tail' flag (operand 3) must be an immediate 0 or 1 ***
+    G_MEMCPY %0, %0, %2, 2 :: (store 4), (load 4)
 ...

llvm/test/MachineVerifier/test_g_memcpy_inline.mir

 #RUN: not --crash llc -o - -march=arm64 -run-pass=none -verify-machineinstrs %s 2>&1 | FileCheck %s
 # REQUIRES: aarch64-registered-target
 ---
-name:            test_memcpy
+name:            test_memcpy_inline
 legalized:       true
 regBankSelected: false
 selected:        false
 tracksRegLiveness: true
 liveins:
 body:             |
   bb.0:
 
     %0:_(p0) = G_CONSTANT i64 0
     %1:_(p0) = G_CONSTANT i64 4
     %2:_(s64) = G_CONSTANT i64 4
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0
+    G_MEMCPY_INLINE %0, %1, %2
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (load 4)
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store 4)
 
     ; CHECK: *** Bad machine code: wrong memory operand types ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4), (store 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (load 4), (store 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store 8), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store unknown-size), (load 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store unknown-size), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load unknown-size)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store 8), (load unknown-size)
 
     ; CHECK: *** Bad machine code: inconsistent store address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4, addrspace 1), (load 4)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store 4, addrspace 1), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent load address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4), (load 4, addrspace 1)
+    G_MEMCPY_INLINE %0, %1, %2 :: (store 4), (load 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %2, %0, %2, 0 :: (store 4), (load 4)
+    G_MEMCPY_INLINE %2, %0, %2 :: (store 4), (load 4)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %0, %2, %2, 0 :: (store 4), (load 4)
-
+    G_MEMCPY_INLINE %0, %2, %2 :: (store 4), (load 4)
 ...

llvm/test/MachineVerifier/test_g_memmove.mir

 #RUN: not --crash llc -o - -march=arm64 -run-pass=none -verify-machineinstrs %s 2>&1 | FileCheck %s
 # REQUIRES: aarch64-registered-target
 ---
-name:            test_memcpy
+name:            test_memmove
 legalized:       true
 regBankSelected: false
 selected:        false
 tracksRegLiveness: true
 liveins:
 body:             |
   bb.0:
 
     %0:_(p0) = G_CONSTANT i64 0
     %1:_(p0) = G_CONSTANT i64 4
     %2:_(s64) = G_CONSTANT i64 4
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0
+    G_MEMMOVE %0, %1, %2, 0
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (load 4)
 
     ; CHECK: *** Bad machine code: memcpy/memmove must have 2 memory operands ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (store 4)
 
     ; CHECK: *** Bad machine code: wrong memory operand types ***
-    G_MEMCPY %0, %1, %2, 0 :: (load 4), (store 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (load 4), (store 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (store 8), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store unknown-size), (load 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (store unknown-size), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent memory operand sizes ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 8), (load unknown-size)
+    G_MEMMOVE %0, %1, %2, 0 :: (store 8), (load unknown-size)
 
     ; CHECK: *** Bad machine code: inconsistent store address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4, addrspace 1), (load 4)
+    G_MEMMOVE %0, %1, %2, 0 :: (store 4, addrspace 1), (load 4)
 
     ; CHECK: *** Bad machine code: inconsistent load address space ***
-    G_MEMCPY %0, %1, %2, 0 :: (store 4), (load 4, addrspace 1)
+    G_MEMMOVE %0, %1, %2, 0 :: (store 4), (load 4, addrspace 1)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %2, %0, %2, 0 :: (store 4), (load 4)
+    G_MEMMOVE %2, %0, %2, 0 :: (store 4), (load 4)
 
     ; CHECK: *** Bad machine code: memory instruction operand must be a pointer ***
-    G_MEMCPY %0, %2, %2, 0 :: (store 4), (load 4)
+    G_MEMMOVE %0, %2, %2, 0 :: (store 4), (load 4)
 
+    ; CHECK: *** Bad machine code: 'tail' flag (operand 3) must be an immediate 0 or 1 ***
+    G_MEMMOVE %0, %0, %2, %0 :: (store 4), (load 4)
+
+    ; CHECK: *** Bad machine code: 'tail' flag (operand 3) must be an immediate 0 or 1 ***
+    G_MEMMOVE %0, %0, %2, 2 :: (store 4), (load 4)
 ...