[AArch64][GlobalISel] CodeGen for Armv8.8/9.3 MOPS

[tyb0807]

Summary

This implements codegen for Armv8.8/9.3 Memory Operations extension (MOPS).
Any memcpy/memset/memmov intrinsics will always be emitted as a series
of three consecutive instructions P, M and E which perform the
operation. The SelectionDAG implementation is split into a separate
patch.

AArch64LegalizerInfo will now consider the following generic opcodes
if +mops is available, instead of legalising by expanding them to
libcalls: G_BZERO, G_MEMCPY_INLINE, G_MEMCPY, G_MEMMOVE, G_MEMSET
The s8 value of memset is legalised to s64 to match the pseudos.

AArch64O0PreLegalizerCombinerInfo will still be able to combine
G_MEMCPY_INLINE even if +mops is present, as it is unclear whether it is
better to generate fixed length copies or MOPS instructions for the
inline code of small or zero-sized memory operations, so we choose to be
conservative for now.

AArch64InstructionSelector will select the above as new pseudo
instructions: AArch64::MOPSMemory{Copy/Move/Set/SetTagging} These are
each expanded to a series of three instructions (e.g. SETP/SETM/SETE)
which must be emitted together during code emission to avoid scheduler
reordering.

This is part 3/4 of a series of patches split from
https://reviews.llvm.org/D117405 to facilitate reviewing.

Patch by Tomas Matheson and Son Tuan Vu

[dmgreen]

llvm/lib/Target/AArch64/GISel/AArch64O0PreLegalizerCombiner.cpp

90

Should these still be returning false, or are fixed length copies better left for the inline code? The quality of the code matter less at O0, but we are likely better off being conservative for the time being, like we are for ISel.

llvm/unittests/Target/AArch64/InstSizes.cpp

167

Nice test. Can you format it to limit the length of the line?

[tyb0807]

Be conservative on whether it is better to use MOPS instructions for the inline
code of small or zero-sized memory operations, i.e. we only emit MOPS
instructions for any time we used to emit an inline call to memcopy.

[tyb0807]

Update commit message to reflect latest changes

[tyb0807]

Update commit message

[tyb0807]

clang-format tests

[dmgreen]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

3429

Why does this have a case for G_BZERO if selectMOPS asserts that opcode != G_BZERO?

llvm/lib/Target/AArch64/GISel/AArch64O0PreLegalizerCombiner.cpp

94

This one too? It may want to prevent turning it into a bzero in this case though.

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

3

Can you add a run line without -O0, as they run through different code. Using --check-prefixes can help if they have identical output.

llvm/unittests/Target/AArch64/InstSizes.cpp

25

All of this seems unrelated?

[tyb0807]

Be conservative on when to inline mem* functions. Remove dead code. Add more tests

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

3429

Good point. Thanks for catching this. I'm also removing the assert, as there's no other way expandMOPS is gonna be called.

llvm/unittests/Target/AArch64/InstSizes.cpp

25

This is the result of clang-format -i, I guess this is more correct than the older format

[dmgreen]

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

945

I think this assert is still valid, isn't it?

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

14

These instructions are invalid without MTE. The test should keep +mte

llvm/test/CodeGen/AArch64/aarch64-mops.ll

4–5

perhaps use --check-prefixes=GISel-WITHOUT-MOPS;GISel-WITHOUT-MOPS-O0 and --check-prefixes=GISel-WITHOUT-MOPS;GISel-WITHOUT-MOPS-O3. Same for the GISel-MOPS below. The tests with identical output can share the check lines then, which keeps things a little cleaner.

[tyb0807]

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

945

No, please see Sjoerd's comment https://reviews.llvm.org/D117753#inline-1126864, backed by the specification https://github.com/ARM-software/acle/blob/next-release/main/acle.md#memcpy-family-of-memory-operations-standarization-instructions---mops

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

14

See reply above for removing hasMTE assertion

[dmgreen]

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

945

This can't just generate instructions for an architecture that is not enabled. That is an error (even if it doesn't currently give an error - there should be a verifier check for it, but it only works in certain situations at the moment). It doesn't matter a lot if the assert is there or not, but it should be invalid to get here without the architecture extension (from clang).

The spec sounds wrong to me - it doesn't make much sense to define an intrinsic that can't be used under the current architecture.

[tyb0807]

Refactor tests and update accordingly to change from ACLE specification: __builtin_arm_mops_memset_tag requires _both_ MOPS and MTE features

llvm/unittests/Target/AArch64/InstSizes.cpp

25

Reverted these formatting, according to https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access (point 2). Thanks to @lenary for pointing this out

[tyb0807]

Remove unused declaration

[dmgreen]

Thanks. I'm not the biggest Global-ISel expert, but from what I can tell this looks good to me.

llvm/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h

 //===- llvm/CodeGen/GlobalISel/LegalizerInfo.h ------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
assert(TypeIdx <=
                (MCOI::OPERAND_LAST_GENERIC - MCOI::OPERAND_FIRST_GENERIC) &&
            "Type Index is out of bounds");
 #ifndef NDEBUG
     TypeIdxsCovered.set(TypeIdx);
 #endif
     return TypeIdx;
   }
 
-  unsigned immIdx(unsigned ImmIdx) {
-    assert(ImmIdx <= (MCOI::OPERAND_LAST_GENERIC_IMM -
-                      MCOI::OPERAND_FIRST_GENERIC_IMM) &&
-           "Imm Index is out of bounds");
-#ifndef NDEBUG
-    ImmIdxsCovered.set(ImmIdx);
-#endif
-    return ImmIdx;
-  }
-
   void markAllIdxsAsCovered() {
 #ifndef NDEBUG
     TypeIdxsCovered.set();
     ImmIdxsCovered.set();
 #endif
   }
 
   void add(const LegalizeRule &Rule) {
public:
   void aliasTo(unsigned Opcode) {
     assert((AliasOf == 0 || AliasOf == Opcode) &&
            "Opcode is already aliased to another opcode");
     assert(Rules.empty() && "Aliasing will discard rules");
     AliasOf = Opcode;
   }
   unsigned getAlias() const { return AliasOf; }
 
+  unsigned immIdx(unsigned ImmIdx) {
+    assert(ImmIdx <= (MCOI::OPERAND_LAST_GENERIC_IMM -
+                      MCOI::OPERAND_FIRST_GENERIC_IMM) &&
+           "Imm Index is out of bounds");
+#ifndef NDEBUG
+    ImmIdxsCovered.set(ImmIdx);
+#endif
+    return ImmIdx;
+  }
+
   /// The instruction is legal if predicate is true.
   LegalizeRuleSet &legalIf(LegalityPredicate Predicate) {
     // We have no choice but conservatively assume that the free-form
     // user-provided Predicate properly handles all type indices:
     markAllIdxsAsCovered();
     return actionIf(LegalizeAction::Legal, Predicate);
   }
   /// The instruction is legal when type index 0 is any type in the given list.
#endif
   /// given list.
   LegalizeRuleSet &customFor(std::initializer_list<std::pair<LLT, LLT>> Types) {
     return actionFor(LegalizeAction::Custom, Types);
   }
 
   LegalizeRuleSet &customForCartesianProduct(std::initializer_list<LLT> Types) {
     return actionForCartesianProduct(LegalizeAction::Custom, Types);
   }
+  /// The instruction is custom when type indexes 0 and 1 are both in their
+  /// respective lists.
   LegalizeRuleSet &
   customForCartesianProduct(std::initializer_list<LLT> Types0,
                             std::initializer_list<LLT> Types1) {
     return actionForCartesianProduct(LegalizeAction::Custom, Types0, Types1);
   }
+  /// The instruction is custom when when type indexes 0, 1, and 2 are all in
+  /// their respective lists.
+  LegalizeRuleSet &
+  customForCartesianProduct(std::initializer_list<LLT> Types0,
+                            std::initializer_list<LLT> Types1,
+                            std::initializer_list<LLT> Types2) {
+    return actionForCartesianProduct(LegalizeAction::Custom, Types0, Types1,
+                                     Types2);
+  }
 
   /// Unconditionally custom lower.
   LegalizeRuleSet &custom() {
     return customIf(always);
   }
 
   /// Widen the scalar to the next power of two that is at least MinSize.
   /// No effect if the type is not a scalar or is a power of two.

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

 //===- AArch64AsmPrinter.cpp - AArch64 LLVM assembly writer ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
public:
 
   void emitStartOfAsmFile(Module &M) override;
   void emitJumpTableInfo() override;
 
   void emitFunctionEntryLabel() override;
 
   void LowerJumpTableDest(MCStreamer &OutStreamer, const MachineInstr &MI);
 
+  void LowerMOPS(MCStreamer &OutStreamer, const MachineInstr &MI);
+
   void LowerSTACKMAP(MCStreamer &OutStreamer, StackMaps &SM,
                      const MachineInstr &MI);
   void LowerPATCHPOINT(MCStreamer &OutStreamer, StackMaps &SM,
                        const MachineInstr &MI);
   void LowerSTATEPOINT(MCStreamer &OutStreamer, StackMaps &SM,
                        const MachineInstr &MI);
   void LowerFAULTING_OP(const MachineInstr &MI);
 
void AArch64AsmPrinter::LowerJumpTableDest(llvm::MCStreamer &OutStreamer,
   // compressed.
   EmitToStreamer(OutStreamer, MCInstBuilder(AArch64::ADDXrs)
                                   .addReg(DestReg)
                                   .addReg(DestReg)
                                   .addReg(ScratchReg)
                                   .addImm(Size == 4 ? 0 : 2));
 }
 
+void AArch64AsmPrinter::LowerMOPS(llvm::MCStreamer &OutStreamer,
+                                  const llvm::MachineInstr &MI) {
+  unsigned Opcode = MI.getOpcode();
+  assert(STI->hasMOPS());
+  assert(STI->hasMTE() || Opcode != AArch64::MOPSMemorySetTaggingPseudo);
+
+  const auto Ops = [Opcode]() -> std::array<unsigned, 3> {
+    if (Opcode == AArch64::MOPSMemoryCopyPseudo)
+      return {AArch64::CPYFP, AArch64::CPYFM, AArch64::CPYFE};
+    if (Opcode == AArch64::MOPSMemoryMovePseudo)
+      return {AArch64::CPYP, AArch64::CPYM, AArch64::CPYE};
+    if (Opcode == AArch64::MOPSMemorySetPseudo)
+      return {AArch64::SETP, AArch64::SETM, AArch64::SETE};
+    if (Opcode == AArch64::MOPSMemorySetTaggingPseudo)
+      return {AArch64::SETGP, AArch64::SETGM, AArch64::MOPSSETGE};
+    llvm_unreachable("Unhandled memory operation pseudo");
+  }();
+  const bool IsSet = Opcode == AArch64::MOPSMemorySetPseudo ||
+                     Opcode == AArch64::MOPSMemorySetTaggingPseudo;
+
+  for (auto Op : Ops) {
+    int i = 0;
+    auto MCIB = MCInstBuilder(Op);
+    // Destination registers
+    MCIB.addReg(MI.getOperand(i++).getReg());
+    MCIB.addReg(MI.getOperand(i++).getReg());
+    if (!IsSet)
+      MCIB.addReg(MI.getOperand(i++).getReg());
+    // Input registers
+    MCIB.addReg(MI.getOperand(i++).getReg());
+    MCIB.addReg(MI.getOperand(i++).getReg());
+    MCIB.addReg(MI.getOperand(i++).getReg());
+
+    EmitToStreamer(OutStreamer, MCIB);
+  }
+}
+
 void AArch64AsmPrinter::LowerSTACKMAP(MCStreamer &OutStreamer, StackMaps &SM,
                                       const MachineInstr &MI) {
   unsigned NumNOPBytes = StackMapOpers(&MI).getNumPatchBytes();
 
   auto &Ctx = OutStreamer.getContext();
   MCSymbol *MILabel = Ctx.createTempSymbol();
   OutStreamer.emitLabel(MILabel);
 
case AArch64::JumpTableDest8:
     return;
 
   case AArch64::FMOVH0:
   case AArch64::FMOVS0:
   case AArch64::FMOVD0:
     emitFMov0(*MI);
     return;
 
+  case AArch64::MOPSMemoryCopyPseudo:
+  case AArch64::MOPSMemoryMovePseudo:
+  case AArch64::MOPSMemorySetPseudo:
+  case AArch64::MOPSMemorySetTaggingPseudo:
+    LowerMOPS(*OutStreamer, *MI);
+    return;
+
   case TargetOpcode::STACKMAP:
     return LowerSTACKMAP(*OutStreamer, SM, *MI);
 
   case TargetOpcode::PATCHPOINT:
     return LowerPATCHPOINT(*OutStreamer, SM, *MI);
 
   case TargetOpcode::STATEPOINT:
     return LowerSTATEPOINT(*OutStreamer, SM, *MI);

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

 //===- AArch64InstrInfo.cpp - AArch64 Instruction Information -------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
case AArch64::SpeculationBarrierISBDSBEndBB:
     break;
   case AArch64::SpeculationBarrierSBEndBB:
     // This gets lowered to 1 4-byte instructions.
     NumBytes = 4;
     break;
   case AArch64::JumpTableDest32:
   case AArch64::JumpTableDest16:
   case AArch64::JumpTableDest8:
+  case AArch64::MOPSMemoryCopyPseudo:
+  case AArch64::MOPSMemoryMovePseudo:
+  case AArch64::MOPSMemorySetPseudo:
+  case AArch64::MOPSMemorySetTaggingPseudo:
     NumBytes = 12;
     break;
   case AArch64::SPACE:
     NumBytes = MI.getOperand(1).getImm();
     break;
   case AArch64::StoreSwiftAsyncContext:
     NumBytes = 20;
     break;

llvm/lib/Target/AArch64/AArch64InstrInfo.td

let Predicates = [HasMOPS, HasMTE] in {
   }
   let Uses = [NZCV] in {
     defm SETGM     : MOPSMemorySetTaggingInsns<0b01, "setgm">;
     // Can't use SETGE because it's a reserved name in TargetSelectionDAG.td
     defm MOPSSETGE : MOPSMemorySetTaggingInsns<0b10, "setge">;
   }
 }
 
+let Predicates = [HasMOPS], Defs = [NZCV], Size = 12, mayStore = 1 in {
+  let mayLoad = 1 in {
+    def MOPSMemoryCopyPseudo : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
+                                      (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
+                                      [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;
+    def MOPSMemoryMovePseudo : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
+                                      (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
+                                      [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;
+  }
+  let mayLoad = 0 in {
+    def MOPSMemorySetPseudo  : Pseudo<(outs GPR64common:$Rd_wb, GPR64:$Rn_wb),
+                                      (ins GPR64common:$Rd, GPR64:$Rn, GPR64:$Rm),
+                                      [], "$Rd = $Rd_wb,$Rn = $Rn_wb">, Sched<[]>;
+  }
+}
+let Predicates = [HasMOPS, HasMTE], Defs = [NZCV], Size = 12, mayLoad = 0, mayStore = 1 in {
+  def MOPSMemorySetTaggingPseudo : Pseudo<(outs GPR64common:$Rd_wb, GPR64:$Rn_wb),
+                                          (ins GPR64common:$Rd, GPR64:$Rn, GPR64:$Rm),
+                                          [], "$Rd = $Rd_wb,$Rn = $Rn_wb">, Sched<[]>;
+}
+
 let Defs = [X16, X17], mayStore = 1, isCodeGenOnly = 1 in
 def StoreSwiftAsyncContext
       : Pseudo<(outs), (ins GPR64:$ctx, GPR64sp:$base, simm9:$offset),
                []>, Sched<[]>;
 
 def AArch64AssertZExtBool : SDNode<"AArch64ISD::ASSERT_ZEXT_BOOL", SDT_assert>;
 def : Pat<(AArch64AssertZExtBool GPR32:$op),
           (i32 GPR32:$op)>;
 
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"
 include "AArch64SMEInstrInfo.td"
 include "AArch64InstrGISel.td"

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

 //===- AArch64InstructionSelector.cpp ----------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
private:
   bool selectIntrinsic(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectVectorICmp(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectIntrinsicTrunc(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectIntrinsicRound(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectJumpTable(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectBrJT(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectTLSGlobalValue(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectReduction(MachineInstr &I, MachineRegisterInfo &MRI);
+  bool selectMOPS(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectUSMovFromExtend(MachineInstr &I, MachineRegisterInfo &MRI);
 
   unsigned emitConstantPoolEntry(const Constant *CPVal,
                                  MachineFunction &MF) const;
   MachineInstr *emitLoadFromConstantPool(const Constant *CPVal,
                                          MachineIRBuilder &MIRBuilder) const;
 
   // Emit a vector concat operation.
case TargetOpcode::G_INSERT_VECTOR_ELT:
     return selectInsertElt(I, MRI);
   case TargetOpcode::G_CONCAT_VECTORS:
     return selectConcatVectors(I, MRI);
   case TargetOpcode::G_JUMP_TABLE:
     return selectJumpTable(I, MRI);
   case TargetOpcode::G_VECREDUCE_FADD:
   case TargetOpcode::G_VECREDUCE_ADD:
     return selectReduction(I, MRI);
+  case TargetOpcode::G_BZERO:
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMCPY_INLINE:
+  case TargetOpcode::G_MEMMOVE:
+  case TargetOpcode::G_MEMSET:
+    assert(STI.hasMOPS() && "Shouldn't get here without +mops feature");
+    return selectMOPS(I, MRI);
   }
 
   return false;
 }
 
 bool AArch64InstructionSelector::selectReduction(MachineInstr &I,
                                                  MachineRegisterInfo &MRI) {
   Register VecReg = I.getOperand(1).getReg();
else {
       return false;
     }
     I.setDesc(TII.get(Opc));
     return constrainSelectedInstRegOperands(I, TII, TRI, RBI);
   }
   return false;
 }
 
+bool AArch64InstructionSelector::selectMOPS(MachineInstr &GI,
+                                            MachineRegisterInfo &MRI) {
+  assert(GI.getOpcode() != TargetOpcode::G_BZERO &&
+         "There is no point combining to G_BZERO only to re-materialize the "
+         "zero.");
+
+  unsigned Mopcode;
+  switch (GI.getOpcode()) {
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMCPY_INLINE:
+    Mopcode = AArch64::MOPSMemoryCopyPseudo;
+    break;
+  case TargetOpcode::G_MEMMOVE:
+    Mopcode = AArch64::MOPSMemoryMovePseudo;
+    break;
+  case TargetOpcode::G_MEMSET:
+    // For tagged memset see llvm.aarch64.mops.memset.tag
+    Mopcode = AArch64::MOPSMemorySetPseudo;
+    break;
+  }
+
+  auto &DstPtr = GI.getOperand(0);
+  auto &SrcOrVal = GI.getOperand(1);
+  auto &Size = GI.getOperand(2);
+
+  // Create copies of the registers that can be clobbered.
+  const Register DstPtrCopy = MRI.cloneVirtualRegister(DstPtr.getReg());
+  const Register SrcValCopy = MRI.cloneVirtualRegister(SrcOrVal.getReg());
+  const Register SizeCopy = MRI.cloneVirtualRegister(Size.getReg());
+
+  const bool IsSet = Mopcode == AArch64::MOPSMemorySetPseudo;
+  const auto &SrcValRegClass =
+      IsSet ? AArch64::GPR64RegClass : AArch64::GPR64commonRegClass;
+
+  // Constrain to specific registers
+  RBI.constrainGenericRegister(DstPtrCopy, AArch64::GPR64commonRegClass, MRI);
+  RBI.constrainGenericRegister(SrcValCopy, SrcValRegClass, MRI);
+  RBI.constrainGenericRegister(SizeCopy, AArch64::GPR64RegClass, MRI);
+
+  MIB.buildCopy(DstPtrCopy, DstPtr);
+  MIB.buildCopy(SrcValCopy, SrcOrVal);
+  MIB.buildCopy(SizeCopy, Size);
+
+  // New instruction uses the copied registers because it must update them.
+  // The defs are not used since they don't exist in G_MEM*. They are still
+  // tied.
+  // Note: order of operands is different from G_MEMSET, G_MEMCPY, G_MEMMOVE
+  Register DefDstPtr = MRI.createVirtualRegister(&AArch64::GPR64commonRegClass);
+  Register DefSize = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+  if (IsSet) {
+    MIB.buildInstr(Mopcode, {DefDstPtr, DefSize},
+                   {DstPtrCopy, SizeCopy, SrcValCopy});
+  } else {
+    Register DefSrcPtr = MRI.createVirtualRegister(&SrcValRegClass);
+    MIB.buildInstr(Mopcode, {DefDstPtr, DefSrcPtr, DefSize},
+                   {DstPtrCopy, SrcValCopy, SizeCopy});
+  }
+
+  GI.eraseFromParent();
+  return true;
+}
+
 bool AArch64InstructionSelector::selectBrJT(MachineInstr &I,
                                             MachineRegisterInfo &MRI) {
   assert(I.getOpcode() == TargetOpcode::G_BRJT && "Expected G_BRJT");
   Register JTAddr = I.getOperand(0).getReg();
   unsigned JTI = I.getOperand(1).getIndex();
   Register Index = I.getOperand(2).getReg();
 
   Register TargetReg = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
case Intrinsic::aarch64_neon_st2: {
     SmallVector<Register, 2> Regs = {Src1, Src2};
     Register Tuple = Ty.getSizeInBits() == 128 ? createQTuple(Regs, MIB)
                                                : createDTuple(Regs, MIB);
     auto Store = MIB.buildInstr(Opc, {}, {Tuple, Ptr});
     Store.cloneMemRefs(I);
     constrainSelectedInstRegOperands(*Store, TII, TRI, RBI);
     break;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    // Transform
+    //    %dst:gpr(p0) = \
+    //      G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.aarch64.mops.memset.tag),
+    //      \ %dst:gpr(p0), %val:gpr(s64), %n:gpr(s64)
+    // where %dst is updated, into
+    //    %Rd:GPR64common, %Rn:GPR64) = \
+    //      MOPSMemorySetTaggingPseudo \
+    //      %Rd:GPR64common, %Rn:GPR64, %Rm:GPR64
+    // where Rd and Rn are tied.
+    // It is expected that %val has been extended to s64 in legalization.
+    // Note that the order of the size/value operands are swapped.
+
+    Register DstDef = I.getOperand(0).getReg();
+    // I.getOperand(1) is the intrinsic function
+    Register DstUse = I.getOperand(2).getReg();
+    Register ValUse = I.getOperand(3).getReg();
+    Register SizeUse = I.getOperand(4).getReg();
+
+    // MOPSMemorySetTaggingPseudo has two defs; the intrinsic call has only one.
+    // Therefore an additional virtual register is requried for the updated size
+    // operand. This value is not accessible via the semantics of the intrinsic.
+    Register SizeDef = MRI.createGenericVirtualRegister(LLT::scalar(64));
+
+    auto Memset = MIB.buildInstr(AArch64::MOPSMemorySetTaggingPseudo,
+                                 {DstDef, SizeDef}, {DstUse, SizeUse, ValUse});
+    Memset.cloneMemRefs(I);
+    constrainSelectedInstRegOperands(*Memset, TII, TRI, RBI);
+    break;
+  }
   }
 
   I.eraseFromParent();
   return true;
 }
 
 bool AArch64InstructionSelector::selectIntrinsic(MachineInstr &I,
                                                  MachineRegisterInfo &MRI) {

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.h

 //===- AArch64LegalizerInfo --------------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
bool legalizeBitfieldExtract(MachineInstr &MI, MachineRegisterInfo &MRI,
                                LegalizerHelper &Helper) const;
   bool legalizeRotate(MachineInstr &MI, MachineRegisterInfo &MRI,
                       LegalizerHelper &Helper) const;
   bool legalizeCTPOP(MachineInstr &MI, MachineRegisterInfo &MRI,
                      LegalizerHelper &Helper) const;
   bool legalizeAtomicCmpxchg128(MachineInstr &MI, MachineRegisterInfo &MRI,
                                 LegalizerHelper &Helper) const;
   bool legalizeCTTZ(MachineInstr &MI, LegalizerHelper &Helper) const;
+  bool legalizeMemOps(MachineInstr &MI, LegalizerHelper &Helper) const;
   const AArch64Subtarget *ST;
 };
 } // End llvm namespace.
 #endif

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

 //===- AArch64LegalizerInfo.cpp ----------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST)
   getActionDefinitionsBuilder(G_JUMP_TABLE).legalFor({{p0}, {s64}});
 
   getActionDefinitionsBuilder(G_BRJT).legalIf([=](const LegalityQuery &Query) {
     return Query.Types[0] == p0 && Query.Types[1] == s64;
   });
 
   getActionDefinitionsBuilder(G_DYN_STACKALLOC).lower();
 
+  if (ST.hasMOPS()) {
+    // G_BZERO is not supported. Currently it is only emitted by
+    // PreLegalizerCombiner for G_MEMSET with zero constant.
+    getActionDefinitionsBuilder(G_BZERO).unsupported();
+
+    getActionDefinitionsBuilder(G_MEMSET)
+        .legalForCartesianProduct({p0}, {s64}, {s64})
+        .customForCartesianProduct({p0}, {s8}, {s64})
+        .immIdx(0); // Inform verifier imm idx 0 is handled.
+
+    getActionDefinitionsBuilder({G_MEMCPY, G_MEMMOVE})
+        .legalForCartesianProduct({p0}, {p0}, {s64})
+        .immIdx(0); // Inform verifier imm idx 0 is handled.
+
+    // G_MEMCPY_INLINE does not have a tailcall immediate
+    getActionDefinitionsBuilder(G_MEMCPY_INLINE)
+        .legalForCartesianProduct({p0}, {p0}, {s64});
+
+  } else {
     getActionDefinitionsBuilder({G_BZERO, G_MEMCPY, G_MEMMOVE, G_MEMSET})
         .libcall();
+  }
 
   // FIXME: Legal types are only legal with NEON.
   getActionDefinitionsBuilder(G_ABS)
       .lowerIf(isScalar(0))
       .legalFor(PackedVectorAllTypeList);
 
   getActionDefinitionsBuilder(G_VECREDUCE_FADD)
       // We only have FADDP to do reduction-like operations. Lower the rest.
bool AArch64LegalizerInfo::legalizeCustom(LegalizerHelper &Helper,
   case TargetOpcode::G_ROTR:
     return legalizeRotate(MI, MRI, Helper);
   case TargetOpcode::G_CTPOP:
     return legalizeCTPOP(MI, MRI, Helper);
   case TargetOpcode::G_ATOMIC_CMPXCHG:
     return legalizeAtomicCmpxchg128(MI, MRI, Helper);
   case TargetOpcode::G_CTTZ:
     return legalizeCTTZ(MI, Helper);
+  case TargetOpcode::G_BZERO:
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMMOVE:
+  case TargetOpcode::G_MEMSET:
+    return legalizeMemOps(MI, Helper);
   }
 
   llvm_unreachable("expected switch to return");
 }
 
 bool AArch64LegalizerInfo::legalizeRotate(MachineInstr &MI,
                                           MachineRegisterInfo &MRI,
                                           LegalizerHelper &Helper) const {
case Intrinsic::vacopy: {
     return true;
   }
   case Intrinsic::get_dynamic_area_offset: {
     MachineIRBuilder &MIB = Helper.MIRBuilder;
     MIB.buildConstant(MI.getOperand(0).getReg(), 0);
     MI.eraseFromParent();
     return true;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    assert(MI.getOpcode() == TargetOpcode::G_INTRINSIC_W_SIDE_EFFECTS);
+    // Zext the value to 64 bit
+    MachineIRBuilder MIB(MI);
+    auto &Value = MI.getOperand(3);
+    Register ZExtValueReg = MIB.buildAnyExt(LLT::scalar(64), Value).getReg(0);
+    Value.setReg(ZExtValueReg);
+    return true;
+  }
   }
 
   return true;
 }
 
 bool AArch64LegalizerInfo::legalizeShlAshrLshr(
     MachineInstr &MI, MachineRegisterInfo &MRI, MachineIRBuilder &MIRBuilder,
     GISelChangeObserver &Observer) const {
bool AArch64LegalizerInfo::legalizeCTTZ(MachineInstr &MI,
   MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
   MachineRegisterInfo &MRI = *MIRBuilder.getMRI();
   LLT Ty = MRI.getType(MI.getOperand(1).getReg());
   auto BitReverse = MIRBuilder.buildBitReverse(Ty, MI.getOperand(1));
   MIRBuilder.buildCTLZ(MI.getOperand(0).getReg(), BitReverse);
   MI.eraseFromParent();
   return true;
 }
+
+bool AArch64LegalizerInfo::legalizeMemOps(MachineInstr &MI,
+                                          LegalizerHelper &Helper) const {
+  MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
+
+  // Tagged version MOPSMemorySetTagged is legalised in legalizeIntrinsic
+  if (MI.getOpcode() == TargetOpcode::G_MEMSET) {
+    // Zext the value operand to 64 bit
+    auto &Value = MI.getOperand(1);
+    Register ZExtValueReg =
+        MIRBuilder.buildAnyExt(LLT::scalar(64), Value).getReg(0);
+    Value.setReg(ZExtValueReg);
+    return true;
+  }
+
+  return false;
+}

llvm/lib/Target/AArch64/GISel/AArch64O0PreLegalizerCombiner.cpp

 //=== lib/CodeGen/GlobalISel/AArch64O0PreLegalizerCombiner.cpp ------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
virtual bool combine(GISelChangeObserver &Observer, MachineInstr &MI,
                        MachineIRBuilder &B) const override;
 };
 
 bool AArch64O0PreLegalizerCombinerInfo::combine(GISelChangeObserver &Observer,
                                                 MachineInstr &MI,
                                                 MachineIRBuilder &B) const {
   CombinerHelper Helper(Observer, B, KB, MDT);
   AArch64GenO0PreLegalizerCombinerHelper Generated(GeneratedRuleCfg, Helper);
+  auto &ST = static_cast<const AArch64Subtarget &>(B.getMF().getSubtarget());
 
   if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
   case TargetOpcode::G_MEMCPY_INLINE:
     return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
+    if (ST.hasMOPS())
+      return false;
+
     // At -O0 set a maxlen of 32 to inline;
     unsigned MaxLen = 32;
     // Try to inline memcpy type calls if optimizations are enabled.
     if (Helper.tryCombineMemCpyFamily(MI, MaxLen))
       return true;
     if (Opc == TargetOpcode::G_MEMSET)
       return llvm::AArch64GISelUtils::tryEmitBZero(MI, B, EnableMinSize);
     return false;

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops,+mte  | FileCheck %s --check-prefix=GISel
+
+; Function Attrs: mustprogress nofree nosync nounwind willreturn writeonly
+declare i8* @llvm.aarch64.mops.memset.tag(i8*, i8, i64)
+
+define i8* @memset_tagged_0_zeroval(i8* %dst, i64 %size) {
+; GISel-LABEL: memset_tagged_0_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x8!, x8
+; GISel-NEXT:    setgm [x0]!, x8!, x8
+; GISel-NEXT:    setge [x0]!, x8!, x8
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 0)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_1_zeroval(i8* %dst, i64 %size) {
+; GISel-LABEL: memset_tagged_1_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #1
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 1)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10_zeroval(i8* %dst, i64 %size) {
+; GISel-LABEL: memset_tagged_10_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #10
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10000_zeroval(i8* %dst, i64 %size) {
+; GISel-LABEL: memset_tagged_10000_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #10000
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10000)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_size_zeroval(i8* %dst, i64 %size) {
+; GISel-LABEL: memset_tagged_size_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x1!, x8
+; GISel-NEXT:    setgm [x0]!, x1!, x8
+; GISel-NEXT:    setge [x0]!, x1!, x8
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 %size)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_0(i8* %dst, i64 %size, i32 %value) {
+; GISel-LABEL: memset_tagged_0:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 0)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_1(i8* %dst, i64 %size, i32 %value) {
+; GISel-LABEL: memset_tagged_1:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #1
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 1)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10(i8* %dst, i64 %size, i32 %value) {
+; GISel-LABEL: memset_tagged_10:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #10
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10000(i8* %dst, i64 %size, i32 %value) {
+; GISel-LABEL: memset_tagged_10000:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #10000
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10000)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_size(i8* %dst, i64 %size, i32 %value) {
+; GISel-LABEL: memset_tagged_size:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x8
+; GISel-NEXT:    mov w8, w2
+; GISel-NEXT:    setgp [x0]!, x1!, x8
+; GISel-NEXT:    setgm [x0]!, x1!, x8
+; GISel-NEXT:    setge [x0]!, x1!, x8
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 %size)
+  ret i8* %r
+}

llvm/test/CodeGen/AArch64/aarch64-mops.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1                    | FileCheck %s --check-prefix=O0-GISel-WITHOUT-MOPS
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops       | FileCheck %s --check-prefix=O0-GISel-MOPS
+
+; Function Attrs: argmemonly nofree nounwind willreturn writeonly
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memmove.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+define void @memset_0_zeroval(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
+  ret void
+}
+
+define void @memset_0_zeroval_volatile(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
+  ret void
+}
+
+define void @memset_10_zeroval(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 false)
+  ret void
+}
+
+define void @memset_10_zeroval_volatile(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 true)
+  ret void
+}
+
+define void @memset_10000_zeroval(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 false)
+  ret void
+}
+
+define void @memset_10000_zeroval_volatile(i8* %dst) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 true)
+  ret void
+}
+
+define void @memset_size_zeroval(i8* %dst, i64 %size) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 false)
+  ret void
+}
+
+define void @memset_size_zeroval_volatile(i8* %dst, i64 %size) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 true)
+  ret void
+}
+
+define void @memset_0(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
+  ret void
+}
+
+define void @memset_0_volatile(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
+  ret void
+}
+
+define void @memset_10(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, w1
+; O0-GISel-WITHOUT-MOPS-NEXT:    and x8, x8, #0xff
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x9, #72340172838076673
+; O0-GISel-WITHOUT-MOPS-NEXT:    mul x8, x8, x9
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    // kill: def $w8 killed $w8 killed $x8
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 false)
+  ret void
+}
+
+define void @memset_10_volatile(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 true)
+  ret void
+}
+
+define void @memset_10000(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 false)
+  ret void
+}
+
+define void @memset_10000_volatile(i8* %dst, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 true)
+  ret void
+}
+
+define void @memset_size(i8* %dst, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x2, [sp, #8] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    add sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-MOPS-NEXT:    mov w8, w2
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 false)
+  ret void
+}
+
+define void @memset_size_volatile(i8* %dst, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x2, [sp, #8] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    add sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-MOPS-NEXT:    mov w8, w2
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 true)
+  ret void
+}
+
+define void @memcpy_0(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memcpy_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memcpy_10(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memcpy_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+define void @memcpy_1000(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_1000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
+  ret void
+}
+
+define void @memcpy_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_1000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
+  ret void
+}
+
+define void @memcpy_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_n:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
+  ret void
+}
+
+define void @memcpy_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_n_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 true)
+  ret void
+}
+
+define void @memcpy_inline_0(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memcpy_inline_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memcpy_inline_10(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memcpy_inline_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+define void @memmove_0(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memmove_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memmove_10(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memmove_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+define void @memmove_1000(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_1000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
+  ret void
+}
+
+define void @memmove_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_1000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
+  ret void
+}
+
+define void @memmove_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_n:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
+  ret void
+}
+
+define void @memmove_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_n_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 true)
+  ret void
+}

llvm/unittests/Target/AArch64/InstSizes.cpp

 #include "AArch64Subtarget.h"
 #include "AArch64TargetMachine.h"
 #include "llvm/CodeGen/MIRParser/MIRParser.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/TargetSelect.h"
 
 #include "gtest/gtest.h"
 
 using namespace llvm;
 
 namespace {
 std::unique_ptr<LLVMTargetMachine> createTargetMachine() {
   auto TT(Triple::normalize("aarch64--"));
   std::string CPU("generic");
   std::string FS("");
 
   LLVMInitializeAArch64TargetInfo();
   LLVMInitializeAArch64Target();
   LLVMInitializeAArch64TargetMC();
 
   std::string Error;
   const Target *TheTarget = TargetRegistry::lookupTarget(TT, Error);
 
-  return std::unique_ptr<LLVMTargetMachine>(static_cast<LLVMTargetMachine*>(
-      TheTarget->createTargetMachine(TT, CPU, FS, TargetOptions(), None, None,
-                                     CodeGenOpt::Default)));
+  return std::unique_ptr<LLVMTargetMachine>(
+      static_cast<LLVMTargetMachine *>(TheTarget->createTargetMachine(
+          TT, CPU, FS, TargetOptions(), None, None, CodeGenOpt::Default)));
 }
 
 std::unique_ptr<AArch64InstrInfo> createInstrInfo(TargetMachine *TM) {
   AArch64Subtarget ST(TM->getTargetTriple(), std::string(TM->getTargetCPU()),
                       std::string(TM->getTargetCPU()),
                       std::string(TM->getTargetFeatureString()), *TM,
                       /* isLittle */ false);
   return std::make_unique<AArch64InstrInfo>(ST);
 }
 
 /// The \p InputIRSnippet is only needed for things that can't be expressed in
 /// the \p InputMIRSnippet (global variables etc)
 /// TODO: Some of this might be useful for other architectures as well - extract
 ///       the platform-independent parts somewhere they can be reused.
 void runChecks(
     LLVMTargetMachine *TM, AArch64InstrInfo *II, const StringRef InputIRSnippet,
     const StringRef InputMIRSnippet,
     std::function<void(AArch64InstrInfo &, MachineFunction &)> Checks) {
   LLVMContext Context;
 
-  auto MIRString =
-    "--- |\n"
-    "  declare void @sizes()\n"
-    + InputIRSnippet.str() +
+  auto MIRString = "--- |\n"
+                   "  declare void @sizes()\n" +
+                   InputIRSnippet.str() +
                    "...\n"
                    "---\n"
                    "name: sizes\n"
                    "body: |\n"
-    "  bb.0:\n"
-    + InputMIRSnippet.str();
+                   "  bb.0:\n" +
+                   InputMIRSnippet.str();
 
   std::unique_ptr<MemoryBuffer> MBuffer = MemoryBuffer::getMemBuffer(MIRString);
   std::unique_ptr<MIRParser> MParser =
       createMIRParser(std::move(MBuffer), Context);
   ASSERT_TRUE(MParser);
 
   std::unique_ptr<Module> M = MParser->parseIRModule();
   ASSERT_TRUE(M);
TEST(InstSizes, Authenticated) {
 
   runChecks(TM.get(), II.get(), "",
             "    \n"
             "    RETAB implicit $lr, implicit $sp, implicit killed $x0\n",
             isAuthInst);
 
   runChecks(TM.get(), II.get(), "",
             "    \n"
-            "    frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp\n",
+            "    frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, "
+            "implicit $sp\n",
             isAuthInst);
 
   runChecks(TM.get(), II.get(), "",
             "    \n"
-            "    frame-destroy AUTIBSP implicit-def $lr, implicit killed $lr, implicit $sp\n",
+            "    frame-destroy AUTIBSP implicit-def $lr, implicit killed $lr, "
+            "implicit $sp\n",
             isAuthInst);
 }
 
 TEST(InstSizes, STACKMAP) {
   std::unique_ptr<LLVMTargetMachine> TM = createTargetMachine();
   ASSERT_TRUE(TM);
   std::unique_ptr<AArch64InstrInfo> II = createInstrInfo(TM.get());
 
-  runChecks(TM.get(), II.get(), "", "    STACKMAP 0, 16\n"
-                                    "    STACKMAP 1, 32\n",
+  runChecks(TM.get(), II.get(), "",
+            "    STACKMAP 0, 16\n"
             [](AArch64InstrInfo &II, MachineFunction &MF) {
               auto I = MF.begin()->begin();
               EXPECT_EQ(16u, II.getInstSizeInBytes(*I));
               ++I;
               EXPECT_EQ(32u, II.getInstSizeInBytes(*I));
             });
 }
 
runChecks(
       TM.get(), II.get(),
       "  @ThreadLocalGlobal = external thread_local global i32, align 8\n",
       "    TLSDESC_CALLSEQ target-flags(aarch64-tls) @ThreadLocalGlobal\n",
       [](AArch64InstrInfo &II, MachineFunction &MF) {
         auto I = MF.begin()->begin();
         EXPECT_EQ(16u, II.getInstSizeInBytes(*I));
       });
 }
+
+TEST(InstSizes, MOPSMemorySetTaggingPseudo) {
+  std::unique_ptr<LLVMTargetMachine> TM = createTargetMachine();
+  std::unique_ptr<AArch64InstrInfo> II = createInstrInfo(TM.get());
+
+  runChecks(TM.get(), II.get(), "",
+            "  renamable $x0, dead renamable $x1 = MOPSMemorySetTaggingPseudo "
+            "killed renamable $x0, killed renamable $x1, killed renamable $x2, "
+            "implicit-def dead $nzcv\n",
+            [](AArch64InstrInfo &II, MachineFunction &MF) {
+              auto I = MF.begin()->begin();
+              EXPECT_EQ(12u, II.getInstSizeInBytes(*I));
+            });
+}