[AArch64][SelectionDAG] CodeGen for Armv8.8/9.3 MOPS

[tyb0807]

Summary

New target SDNodes are added: AArch64ISD::MOPS_MEMSET, etc.
Each intrinsic is translated to one of these in SelectionDAGBuilder
via EmitTargetCodeForMOPS.

A custom lowering routine for INTRINSIC_W_CHAIN is added to handle
llvm.aarch64.mops.memset.tag. This takes a separate path from the common
intrinsics but ultimately ends up in the same EmitMOPS().

This is part 4/4 of a series of patches split from
https://reviews.llvm.org/D117405 to facilitate reviewing.

Patch by Tomas Matheson, Lucas Prates and Son Tuan Vu.

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

946–947

I think that if the values in both sides of the if are currently the same - it's probably worth removing the if. We can always re-add it again if/when we alter these in the future.

4093

-> "So hide SizeWb using MERGE_VALUES."

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

26

What does "We don't use it" mean here?

63

Maybe use != MVT::i8, or add an assert. I guess it will always be an i8 or a i64? Will this only be for Set?

In llvm, singled statements don't need brackets around them and can be dropped.

llvm/test/CodeGen/AArch64/aarch64-mops-consecutive.ll

11

You can usually remove "hidden" "dso_local" and "local_unnamed_addr", to clean up the test a little. "Function Attrs: " too.

[tyb0807]

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

26

Not sure. I propose to remove it

63

I'm not sure. For Intrinsic::memcpy and Intrinsic::memmove, from Intrinsics.td, it seems that the type for SrcOrValue is llvm_anyptr_ty. Am I missing something here?

[tyb0807]

Fix typos and cleanup tests.

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

34

Clang format should put these in order

940

I'm not sure this is worth adding - if it turns out that the existing costs are fine then it will be left as a FIXME for all time without adding much useful information.

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

63

The condition doesn't need getSimpleVT().
I think this should only happen with IsSet, so it's probably worth sinking into the if below. The it might as well be if if (SrcOrValue.getValueType() != MVT::i64)

[dmgreen]

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

70

This is also creating Machine nodes very early. It may be better to create an ISD node for it instead. (Although I'm not sure how much it will matter - if there are no dag optimizations that happen for the memcpy nodes).

[tyb0807]

Remove FIXME and minor code refactoring

[tyb0807]

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

70

Since we are not aware of any DAGCombine pattern allowing to optimize memcpy nodes, I propose to create Machine nodes here instead of defining new ISD nodes for these instructions, then do the ISD nodes -> Machine nodes somewhere else. What do you think?

[dmgreen]

Thanks. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

939

You can leave this comment in.

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

70

It can cause issues here and there. But if we need it, we can change it later.

[tyb0807]

Re-add comment explaining that llvm.mem* intrinsics can be either lowered to
ldr/str sequences or pseudo-instructions

[tyb0807]

Update accordingly to change from ACLE specification: __builtin_arm_mops_memset_tag requires _both_ MOPS and MTE features

[GMNGeoffrey]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11898

FYI getElementType was deprecated in https://github.com/llvm/llvm-project/commit/184591a. I've got a build with -Wdeprecated -Werror that is complaining about this. Setting aside that that's probably a silly thing to have, it would be good to update this, I think

[tyb0807]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11898

This has been temporarily fixed in https://reviews.llvm.org/rG51ed14d22430bed0d191e01f7efeae59b1aee5e0.
This will correctly be fixed in https://reviews.llvm.org/D118681

[GMNGeoffrey]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11898

Thanks!

llvm/lib/Target/AArch64/AArch64ISelLowering.h

 //==-- AArch64ISelLowering.h - AArch64 DAG Lowering Interface ----*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum NodeType : unsigned {
   STG,
   STZG,
   ST2G,
   STZ2G,
 
   LDP,
   STP,
   STNP,
+
+  // Memory Operations
+  MOPS_MEMSET,
+  MOPS_MEMSET_TAGGING,
+  MOPS_MEMCOPY,
+  MOPS_MEMMOVE,
 };
 
 } // end namespace AArch64ISD
 
 namespace {
 
 // Any instruction that defines a 32-bit result zeros out the high half of the
 // register. Truncate can be lowered to EXTRACT_SUBREG. CopyFromReg may
private:
   SDValue LowerStore128(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerABS(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue LowerMGATHER(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerMSCATTER(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue LowerMLOAD(SDValue Op, SelectionDAG &DAG) const;
 
+  SDValue LowerINTRINSIC_W_CHAIN(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINTRINSIC_WO_CHAIN(SDValue Op, SelectionDAG &DAG) const;
 
   bool isEligibleForTailCallOptimization(
       SDValue Callee, CallingConv::ID CalleeCC, bool isVarArg,
       const SmallVectorImpl<ISD::OutputArg> &Outs,
       const SmallVectorImpl<SDValue> &OutVals,
       const SmallVectorImpl<ISD::InputArg> &Ins, SelectionDAG &DAG) const;
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 //===-- AArch64ISelLowering.cpp - AArch64 DAG Lowering Implementation  ----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/ObjCARCUtil.h"
 #include "llvm/Analysis/VectorUtils.h"
+#include "llvm/Analysis/MemoryLocation.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
#undef LCALLNAME5
   setTargetDAGCombine(ISD::EXTRACT_VECTOR_ELT);
   setTargetDAGCombine(ISD::VECREDUCE_ADD);
   setTargetDAGCombine(ISD::STEP_VECTOR);
 
   setTargetDAGCombine(ISD::FP_EXTEND);
 
   setTargetDAGCombine(ISD::GlobalAddress);
 
-  // In case of strict alignment, avoid an excessive number of byte wide stores.
+  // FIXME: right now we are being conservative on when it is best to use
+  // MOPS instructions instead of ldr/str sequence(s), i.e. we are emitting
+  // the MOPS instructions for any time we used to emit a call to mem*
+  // functions. These threshold values should be updated once we have further
+  // evaluated the performance differences.
   MaxStoresPerMemsetOptSize = 8;
-  MaxStoresPerMemset = Subtarget->requiresStrictAlign()
-                       ? MaxStoresPerMemsetOptSize : 32;
+  MaxStoresPerMemset =
+      Subtarget->requiresStrictAlign() ? MaxStoresPerMemsetOptSize : 32;
 
   MaxGluedStoresPerMemcpy = 4;
   MaxStoresPerMemcpyOptSize = 4;
-  MaxStoresPerMemcpy = Subtarget->requiresStrictAlign()
-                       ? MaxStoresPerMemcpyOptSize : 16;
+  MaxStoresPerMemcpy =
+      Subtarget->requiresStrictAlign() ? MaxStoresPerMemcpyOptSize : 16;
 
-  MaxStoresPerMemmoveOptSize = MaxStoresPerMemmove = 4;
+  MaxStoresPerMemmoveOptSize = 4;
+  MaxStoresPerMemmove = 4;
 
   MaxLoadsPerMemcmpOptSize = 4;
-  MaxLoadsPerMemcmp = Subtarget->requiresStrictAlign()
-                      ? MaxLoadsPerMemcmpOptSize : 8;
+  MaxLoadsPerMemcmp =
+      Subtarget->requiresStrictAlign() ? MaxLoadsPerMemcmpOptSize : 8;
 
   setStackPointerRegisterToSaveRestore(AArch64::SP);
 
   setSchedulingPreference(Sched::Hybrid);
 
   EnableExtLdPromotion = true;
 
   // Set required alignment.
if (Subtarget->hasSVE()) {
     }
 
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv2i1, MVT::nxv2i64);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv4i1, MVT::nxv4i32);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv8i1, MVT::nxv8i16);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv16i1, MVT::nxv16i8);
   }
 
+  if (Subtarget->hasMOPS() && Subtarget->hasMTE()) {
+    // Only required for llvm.aarch64.mops.memset.tag
+    setOperationAction(ISD::INTRINSIC_W_CHAIN, MVT::i8, Custom);
+  }
+
   PredictableSelectIsExpensive = Subtarget->predictableSelectIsExpensive();
 }
 
 void AArch64TargetLowering::addTypeForNEON(MVT VT) {
   assert(VT.isVector() && "VT should be a vector type");
 
   if (VT.isFloatingPoint()) {
     MVT PromoteTo = EVT(VT).changeVectorElementTypeToInteger().getSimpleVT();
case AArch64ISD::FIRST_NUMBER:
     MAKE_CASE(AArch64ISD::REVW_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTLZ_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTPOP_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::DUP_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::INDEX_VECTOR)
     MAKE_CASE(AArch64ISD::UADDLP)
     MAKE_CASE(AArch64ISD::CALL_RVMARKER)
     MAKE_CASE(AArch64ISD::ASSERT_ZEXT_BOOL)
+    MAKE_CASE(AArch64ISD::MOPS_MEMSET)
+    MAKE_CASE(AArch64ISD::MOPS_MEMSET_TAGGING)
+    MAKE_CASE(AArch64ISD::MOPS_MEMCOPY)
+    MAKE_CASE(AArch64ISD::MOPS_MEMMOVE)
   }
 #undef MAKE_CASE
   return nullptr;
 }
 
 MachineBasicBlock *
 AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const {
static SDValue lowerConvertToSVBool(SDValue Op, SelectionDAG &DAG) {
 
   // Otherwise, zero the newly introduced lanes.
   SDValue Mask = getPTrue(DAG, DL, InVT, AArch64SVEPredPattern::all);
   SDValue MaskReinterpret =
       DAG.getNode(AArch64ISD::REINTERPRET_CAST, DL, OutVT, Mask);
   return DAG.getNode(ISD::AND, DL, OutVT, Reinterpret, MaskReinterpret);
 }
 
+SDValue AArch64TargetLowering::LowerINTRINSIC_W_CHAIN(SDValue Op,
+                                                      SelectionDAG &DAG) const {
+  unsigned IntNo = Op.getConstantOperandVal(1);
+  switch (IntNo) {
+  default:
+    return SDValue(); // Don't custom lower most intrinsics.
+  case Intrinsic::aarch64_mops_memset_tag: {
+    auto Node = cast<MemIntrinsicSDNode>(Op.getNode());
+    SDLoc DL(Op);
+    SDValue Chain = Node->getChain();
+    SDValue Dst = Op.getOperand(2);
+    SDValue Val = Op.getOperand(3);
+    Val = DAG.getAnyExtOrTrunc(Val, DL, MVT::i64);
+    SDValue Size = Op.getOperand(4);
+    auto Alignment = Node->getMemOperand()->getAlign();
+    bool IsVol = Node->isVolatile();
+    auto DstPtrInfo = Node->getPointerInfo();
+
+    const auto &SDI =
+        static_cast<const AArch64SelectionDAGInfo &>(DAG.getSelectionDAGInfo());
+    SDValue MS =
+        SDI.EmitMOPS(AArch64ISD::MOPS_MEMSET_TAGGING, DAG, DL, Chain, Dst, Val,
+                     Size, Alignment, IsVol, DstPtrInfo, MachinePointerInfo{});
+
+    // MOPS_MEMSET_TAGGING has 3 results (DstWb, SizeWb, Chain) whereas the
+    // intrinsic has 2. So hide SizeWb using MERGE_VALUES. Otherwise
+    // LowerOperationWrapper will complain that the number of results has
+    // changed.
+    return DAG.getMergeValues({MS.getValue(0), MS.getValue(2)}, DL);
+  }
+  }
+}
+
 SDValue AArch64TargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
                                                      SelectionDAG &DAG) const {
   unsigned IntNo = cast<ConstantSDNode>(Op.getOperand(0))->getZExtValue();
   SDLoc dl(Op);
   switch (IntNo) {
   default: return SDValue();    // Don't custom lower most intrinsics.
   case Intrinsic::thread_pointer: {
     EVT PtrVT = getPointerTy(DAG.getDataLayout());
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::MUL:
     return LowerMUL(Op, DAG);
   case ISD::MULHS:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHS_PRED,
                                /*OverrideNEON=*/true);
   case ISD::MULHU:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHU_PRED,
                                /*OverrideNEON=*/true);
+  case ISD::INTRINSIC_W_CHAIN:
+    return LowerINTRINSIC_W_CHAIN(Op, DAG);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
   case ISD::ATOMIC_STORE:
     if (cast<MemSDNode>(Op)->getMemoryVT() == MVT::i128) {
       assert(Subtarget->hasLSE2());
       return LowerStore128(Op, DAG);
     }
     return SDValue();
case Intrinsic::aarch64_sve_stnt1: {
     Info.opc = ISD::INTRINSIC_W_CHAIN;
     Info.memVT = MVT::getVT(I.getOperand(0)->getType());
     Info.ptrVal = I.getArgOperand(2);
     Info.offset = 0;
     Info.align = DL.getABITypeAlign(PtrTy->getElementType());
     Info.flags = MachineMemOperand::MOStore | MachineMemOperand::MONonTemporal;
     return true;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    Value *Dst = I.getArgOperand(0);
+    Value *Val = I.getArgOperand(1);
+    PointerType *PtrTy = cast<PointerType>(Dst->getType());
+    Info.opc = ISD::INTRINSIC_W_CHAIN;
+    Info.memVT = MVT::getVT(Val->getType());
+    Info.ptrVal = Dst;
+    Info.offset = 0;
+    Info.align = DL.getABITypeAlign(PtrTy->getElementType());
+    Info.flags = MachineMemOperand::MOStore;
+    // The size of the memory being operated on is unknown at this point
+    Info.size = MemoryLocation::UnknownSize;
+    return true;
+  }
   default:
     break;
   }
 
   return false;
 }
 
 bool AArch64TargetLowering::shouldReduceLoadWidth(SDNode *Load,

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 }
 let Predicates = [HasMOPS, HasMTE] in {
   defm SETGP     : MOPSMemorySetTaggingInsns<0b00, "setgp">;
   defm SETGM     : MOPSMemorySetTaggingInsns<0b01, "setgm">;
   // Can't use SETGE because it's a reserved name in TargetSelectionDAG.td
   defm MOPSSETGE : MOPSMemorySetTaggingInsns<0b10, "setge">;
 }
 
+// MOPS Node operands: 0: Dst, 1: Src or Value, 2: Size, 3: Chain
+// MOPS Node results: 0: Dst writeback, 1: Size writeback, 2: Chain
+def SDT_AArch64mops : SDTypeProfile<2, 3, [ SDTCisInt<0>, SDTCisInt<1>, SDTCisInt<2> ]>;
+def AArch64mops_memset : SDNode<"AArch64ISD::MOPS_MEMSET", SDT_AArch64mops>;
+def AArch64mops_memset_tagging : SDNode<"AArch64ISD::MOPS_MEMSET_TAGGING", SDT_AArch64mops>;
+def AArch64mops_memcopy : SDNode<"AArch64ISD::MOPS_MEMCOPY", SDT_AArch64mops>;
+def AArch64mops_memmove : SDNode<"AArch64ISD::MOPS_MEMMOVE", SDT_AArch64mops>;
+
 let Predicates = [HasMOPS], Defs = [NZCV], Size = 12, mayStore = 1 in {
   let mayLoad = 1 in {
     def MOPSMemoryCopyPseudo : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
                                       (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
                                       [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;
     def MOPSMemoryMovePseudo : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
                                       (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
                                       [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

 //===-- AArch64SelectionDAGInfo.h - AArch64 SelectionDAG Info ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the AArch64 subclass for SelectionDAGTargetInfo.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_AARCH64_AARCH64SELECTIONDAGINFO_H
 #define LLVM_LIB_TARGET_AARCH64_AARCH64SELECTIONDAGINFO_H
 
 #include "llvm/CodeGen/SelectionDAGTargetInfo.h"
 
 namespace llvm {
 
 class AArch64SelectionDAGInfo : public SelectionDAGTargetInfo {
 public:
+  SDValue EmitMOPS(AArch64ISD::NodeType SDOpcode, SelectionDAG &DAG,
+                   const SDLoc &DL, SDValue Chain, SDValue Dst,
+                   SDValue SrcOrValue, SDValue Size, Align Alignment,
+                   bool isVolatile, MachinePointerInfo DstPtrInfo,
+                   MachinePointerInfo SrcPtrInfo) const;
+
+  SDValue EmitTargetCodeForMemcpy(SelectionDAG &DAG, const SDLoc &dl,
+                                  SDValue Chain, SDValue Dst, SDValue Src,
+                                  SDValue Size, Align Alignment,
+                                  bool isVolatile, bool AlwaysInline,
+                                  MachinePointerInfo DstPtrInfo,
+                                  MachinePointerInfo SrcPtrInfo) const override;
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
                                   bool isVolatile,
                                   MachinePointerInfo DstPtrInfo) const override;
+  SDValue
+  EmitTargetCodeForMemmove(SelectionDAG &DAG, const SDLoc &dl, SDValue Chain,
+                           SDValue Dst, SDValue Src, SDValue Size,
+                           Align Alignment, bool isVolatile,
+                           MachinePointerInfo DstPtrInfo,
+                           MachinePointerInfo SrcPtrInfo) const override;
+
   SDValue EmitTargetCodeForSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   MachinePointerInfo DstPtrInfo,
                                   bool ZeroData) const override;
 };
 }
 
 #endif

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

 //===-- AArch64SelectionDAGInfo.cpp - AArch64 SelectionDAG Info -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the AArch64SelectionDAGInfo class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64TargetMachine.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-selectiondag-info"
 
+SDValue AArch64SelectionDAGInfo::EmitMOPS(AArch64ISD::NodeType SDOpcode,
+                                          SelectionDAG &DAG, const SDLoc &DL,
+                                          SDValue Chain, SDValue Dst,
+                                          SDValue SrcOrValue, SDValue Size,
+                                          Align Alignment, bool isVolatile,
+                                          MachinePointerInfo DstPtrInfo,
+                                          MachinePointerInfo SrcPtrInfo) const {
+
+  // Get the constant size of the copy/set.
+  uint64_t ConstSize = 0;
+  if (auto *C = dyn_cast<ConstantSDNode>(Size))
+    ConstSize = C->getZExtValue();
+
+  const bool IsSet = SDOpcode == AArch64ISD::MOPS_MEMSET ||
+                     SDOpcode == AArch64ISD::MOPS_MEMSET_TAGGING;
+
+  const auto MachineOpcode = [&]() {
+    switch (SDOpcode) {
+    case AArch64ISD::MOPS_MEMSET:
+      return AArch64::MOPSMemorySetPseudo;
+    case AArch64ISD::MOPS_MEMSET_TAGGING:
+      return AArch64::MOPSMemorySetTaggingPseudo;
+    case AArch64ISD::MOPS_MEMCOPY:
+      return AArch64::MOPSMemoryCopyPseudo;
+    case AArch64ISD::MOPS_MEMMOVE:
+      return AArch64::MOPSMemoryMovePseudo;
+    default:
+      llvm_unreachable("Unhandled MOPS ISD Opcode");
+    }
+  }();
+
+  MachineMemOperand::Flags Flags = MachineMemOperand::MOStore;
+  if (isVolatile)
+    Flags |= MachineMemOperand::MOVolatile;
+  if (!IsSet)
+    Flags |= MachineMemOperand::MOLoad;
+
+  MachineFunction &MF = DAG.getMachineFunction();
+
+  auto *DstOp =
+      MF.getMachineMemOperand(DstPtrInfo, Flags, ConstSize, Alignment);
+  auto *SrcOp =
+      MF.getMachineMemOperand(SrcPtrInfo, Flags, ConstSize, Alignment);
+
+  // Extend i8 value to i64 if required
+  if (SrcOrValue.getValueType().getSimpleVT() == MVT::i8) {
+    SrcOrValue = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, SrcOrValue);
+  }
+
+  if (IsSet) {
+    SDValue Ops[] = {Dst, Size, SrcOrValue, Chain};
+    const EVT ResultTys[] = {MVT::i64, MVT::i64, MVT::Other};
+    MachineSDNode *Node = DAG.getMachineNode(MachineOpcode, DL, ResultTys, Ops);
+    DAG.setNodeMemRefs(Node, {DstOp});
+    return SDValue(Node, 2);
+  } else {
+    SDValue Ops[] = {Dst, SrcOrValue, Size, Chain};
+    const EVT ResultTys[] = {MVT::i64, MVT::i64, MVT::i64, MVT::Other};
+    MachineSDNode *Node = DAG.getMachineNode(MachineOpcode, DL, ResultTys, Ops);
+    DAG.setNodeMemRefs(Node, {DstOp, SrcOp});
+    return SDValue(Node, 3);
+  }
+}
+
+SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemcpy(
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst, SDValue Src,
+    SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
+    MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+  if (STI.hasMOPS())
+    return EmitMOPS(AArch64ISD::MOPS_MEMCOPY, DAG, DL, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, SrcPtrInfo);
+  return SDValue();
+}
+
 SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, Align Alignment, bool isVolatile,
     MachinePointerInfo DstPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+
+  if (STI.hasMOPS()) {
+    return EmitMOPS(AArch64ISD::MOPS_MEMSET, DAG, dl, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, MachinePointerInfo{});
+  }
+
   // Check to see if there is a specialized entry-point for memory zeroing.
   ConstantSDNode *V = dyn_cast<ConstantSDNode>(Src);
   ConstantSDNode *SizeValue = dyn_cast<ConstantSDNode>(Size);
-  const AArch64Subtarget &STI =
-      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
   const char *bzeroName =
       (V && V->isZero())
           ? DAG.getTargetLoweringInfo().getLibcallName(RTLIB::BZERO)
           : nullptr;
   // For small size (< 256), it is not beneficial to use bzero
   // instead of memset.
   if (bzeroName && (!SizeValue || SizeValue->getZExtValue() > 256)) {
     const AArch64TargetLowering &TLI = *STI.getTargetLowering();
CLI.setDebugLoc(dl)
                       std::move(Args))
         .setDiscardResult();
     std::pair<SDValue, SDValue> CallResult = TLI.LowerCallTo(CLI);
     return CallResult.second;
   }
   return SDValue();
 }
 
+SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemmove(
+    SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
+    SDValue Size, Align Alignment, bool isVolatile,
+    MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+  if (STI.hasMOPS()) {
+    return EmitMOPS(AArch64ISD::MOPS_MEMMOVE, DAG, dl, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, SrcPtrInfo);
+  }
+  return SDValue();
+}
+
 static const int kSetTagLoopThreshold = 176;
 
 static SDValue EmitUnrolledSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Ptr, uint64_t ObjSize,
                                   const MachineMemOperand *BaseMemOperand,
                                   bool ZeroData) {
   MachineFunction &MF = DAG.getMachineFunction();
   unsigned ObjSizeScaled = ObjSize / 16;

llvm/test/CodeGen/AArch64/aarch64-mops-consecutive.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops       | FileCheck %s --check-prefix=CHECK-MOPS
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+
+declare void @fn(i8*, i8*)
+
+define void @consecutive() {
+; CHECK-MOPS-LABEL: consecutive:
+; CHECK-MOPS:       // %bb.0: // %entry
+; CHECK-MOPS-NEXT:    stp x29, x30, [sp, #-16]!           // 16-byte Folded Spill
+; CHECK-MOPS-NEXT:    sub sp, sp, #2016
+; CHECK-MOPS-NEXT:    .cfi_def_cfa_offset 2032
+; CHECK-MOPS-NEXT:    .cfi_offset w30, -8
+; CHECK-MOPS-NEXT:    .cfi_offset w29, -16
+; CHECK-MOPS-NEXT:    mov w8, #1000
+; CHECK-MOPS-NEXT:    add x9, sp, #8
+; CHECK-MOPS-NEXT:    adrp  x10, .LCPI0_0
+; CHECK-MOPS-NEXT:    adrp  x11, .LCPI0_1
+; CHECK-MOPS-NEXT:    mov w12, #6424
+; CHECK-MOPS-NEXT:    mov w13, #7452
+; CHECK-MOPS-NEXT:    setp  [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    setm  [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    sete  [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    movk  w12, #6938, lsl #16
+; CHECK-MOPS-NEXT:    ldr q0, [x10, :lo12:.LCPI0_0]
+; CHECK-MOPS-NEXT:    mov w8, #30
+; CHECK-MOPS-NEXT:    ldr d1, [x11, :lo12:.LCPI0_1]
+; CHECK-MOPS-NEXT:    add x0, sp, #1008
+; CHECK-MOPS-NEXT:    add x1, sp, #8
+; CHECK-MOPS-NEXT:    str w12, [sp, #1032]
+; CHECK-MOPS-NEXT:    strh  w13, [sp, #1036]
+; CHECK-MOPS-NEXT:    str q0, [sp, #1008]
+; CHECK-MOPS-NEXT:    str d1, [sp, #1024]
+; CHECK-MOPS-NEXT:    strb  w8, [sp, #1038]
+; CHECK-MOPS-NEXT:    bl  fn
+; CHECK-MOPS-NEXT:    add sp, sp, #2016
+; CHECK-MOPS-NEXT:    ldp x29, x30, [sp], #16             // 16-byte Folded Reload
+; CHECK-MOPS-NEXT:    ret
+entry:
+  %buf_from = alloca [1000 x i8], align 16
+  %buf_to = alloca [1000 x i8], align 1
+  %0 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 0
+  %1 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_to, i64 0, i64 0
+  call void @llvm.memset.p0i8.i64(i8* noundef nonnull align 1 dereferenceable(1000) %1, i8 0, i64 1000, i1 false)
+  %2 = bitcast [1000 x i8]* %buf_from to <16 x i8>*
+  store <16 x i8> <i8 0, i8 1, i8 2, i8 3, i8 4, i8 5, i8 6, i8 7, i8 8, i8 9, i8 10, i8 11, i8 12, i8 13, i8 14, i8 15>, <16 x i8>* %2, align 16
+  %arrayidx.16 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 16
+  %3 = bitcast i8* %arrayidx.16 to <8 x i8>*
+  store <8 x i8> <i8 16, i8 17, i8 18, i8 19, i8 20, i8 21, i8 22, i8 23>, <8 x i8>* %3, align 16
+  %arrayidx.24 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 24
+  store i8 24, i8* %arrayidx.24, align 8
+  %arrayidx.25 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 25
+  store i8 25, i8* %arrayidx.25, align 1
+  %arrayidx.26 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 26
+  store i8 26, i8* %arrayidx.26, align 2
+  %arrayidx.27 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 27
+  store i8 27, i8* %arrayidx.27, align 1
+  %arrayidx.28 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 28
+  store i8 28, i8* %arrayidx.28, align 4
+  %arrayidx.29 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 29
+  store i8 29, i8* %arrayidx.29, align 1
+  %arrayidx.30 = getelementptr inbounds [1000 x i8], [1000 x i8]* %buf_from, i64 0, i64 30
+  store i8 30, i8* %arrayidx.30, align 2
+  call void @fn(i8* nonnull %0, i8* nonnull %1)
+  ret void
+}

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops,+mte  | FileCheck %s --check-prefix=SDAG
 ; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops,+mte  | FileCheck %s --check-prefix=GISel
 
-; Function Attrs: mustprogress nofree nosync nounwind willreturn writeonly
 declare i8* @llvm.aarch64.mops.memset.tag(i8*, i8, i64)
 
 define i8* @memset_tagged_0_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_0_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov x8, xzr
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_0_zeroval:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    mov x8, xzr
 ; GISel-NEXT:    setgp [x0]!, x8!, x8
 ; GISel-NEXT:    setgm [x0]!, x8!, x8
 ; GISel-NEXT:    setge [x0]!, x8!, x8
 ; GISel-NEXT:    ret
 entry:
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 0)
   ret i8* %r
 }
 
 define i8* @memset_tagged_1_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_1_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #1
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_1_zeroval:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    mov x9, xzr
 ; GISel-NEXT:    mov w8, #1
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 1)
   ret i8* %r
 }
 
 define i8* @memset_tagged_10_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_10_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_10_zeroval:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    mov x9, xzr
 ; GISel-NEXT:    mov w8, #10
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10)
   ret i8* %r
 }
 
 define i8* @memset_tagged_10000_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_10000_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10000
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_10000_zeroval:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    mov x9, xzr
 ; GISel-NEXT:    mov w8, #10000
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10000)
   ret i8* %r
 }
 
 define i8* @memset_tagged_size_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_size_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    setgp [x0]!, x1!, xzr
+; SDAG-NEXT:    setgm [x0]!, x1!, xzr
+; SDAG-NEXT:    setge [x0]!, x1!, xzr
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_size_zeroval:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    mov x8, xzr
 ; GISel-NEXT:    setgp [x0]!, x1!, x8
 ; GISel-NEXT:    setgm [x0]!, x1!, x8
 ; GISel-NEXT:    setge [x0]!, x1!, x8
 ; GISel-NEXT:    ret
 entry:
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 %size)
   ret i8* %r
 }
 
 define i8* @memset_tagged_0(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_0:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov x8, xzr
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_0:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    // implicit-def: $x9
 ; GISel-NEXT:    mov w9, w2
 ; GISel-NEXT:    mov x8, xzr
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 0)
   ret i8* %r
 }
 
 define i8* @memset_tagged_1(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_1:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #1
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_1:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    // implicit-def: $x9
 ; GISel-NEXT:    mov w9, w2
 ; GISel-NEXT:    mov w8, #1
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 1)
   ret i8* %r
 }
 
 define i8* @memset_tagged_10(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_10:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_10:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    // implicit-def: $x9
 ; GISel-NEXT:    mov w9, w2
 ; GISel-NEXT:    mov w8, #10
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10)
   ret i8* %r
 }
 
 define i8* @memset_tagged_10000(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_10000:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10000
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_10000:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    // implicit-def: $x9
 ; GISel-NEXT:    mov w9, w2
 ; GISel-NEXT:    mov w8, #10000
 ; GISel-NEXT:    // kill: def $x8 killed $w8
 ; GISel-NEXT:    setgp [x0]!, x8!, x9
 ; GISel-NEXT:    setgm [x0]!, x8!, x9
 ; GISel-NEXT:    setge [x0]!, x8!, x9
 ; GISel-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10000)
   ret i8* %r
 }
 
 define i8* @memset_tagged_size(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_size:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x1!, x2
+; SDAG-NEXT:    setgm [x0]!, x1!, x2
+; SDAG-NEXT:    setge [x0]!, x1!, x2
+; SDAG-NEXT:    ret
+;
 ; GISel-LABEL: memset_tagged_size:
 ; GISel:       // %bb.0: // %entry
 ; GISel-NEXT:    // implicit-def: $x8
 ; GISel-NEXT:    mov w8, w2
 ; GISel-NEXT:    setgp [x0]!, x1!, x8
 ; GISel-NEXT:    setgm [x0]!, x1!, x8
 ; GISel-NEXT:    setge [x0]!, x1!, x8
 ; GISel-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 %size)
   ret i8* %r
 }

llvm/test/CodeGen/AArch64/aarch64-mops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2                    | FileCheck %s --check-prefix=O2-SDAG-WITHOUT-MOPS
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops       | FileCheck %s --check-prefix=O2-SDAG-MOPS
+
 ; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1                    | FileCheck %s --check-prefix=O0-GISel-WITHOUT-MOPS
 ; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops       | FileCheck %s --check-prefix=O0-GISel-MOPS
 
-; Function Attrs: argmemonly nofree nounwind willreturn writeonly
 declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
 
-; Function Attrs: argmemonly nofree nounwind willreturn
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
 
-; Function Attrs: argmemonly nofree nounwind willreturn
 declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
 
-; Function Attrs: argmemonly nofree nounwind willreturn
 declare void @llvm.memmove.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
 
 define void @memset_0_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_0_zeroval:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    mov x9, x8
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
   ret void
 }
 
 define void @memset_0_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_0_zeroval_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    mov x9, x8
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
   ret void
 }
 
 define void @memset_10_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    strh  wzr, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str xzr, [x0]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_10_zeroval:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x9, xzr
 ; O0-GISel-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 false)
   ret void
 }
 
 define void @memset_10_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    strh  wzr, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8

 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 true)
   ret void
 }
 
 define void @memset_10000_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8

 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 false)
   ret void
 }
 
 define void @memset_10000_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8

 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 true)
   ret void
 }
 
 define void @memset_size_zeroval(i8* %dst, i64 %size) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_size_zeroval:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 false)
   ret void
 }
 
 define void @memset_size_zeroval_volatile(i8* %dst, i64 %size) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_size_zeroval_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 true)
   ret void
 }
 
+
 define void @memset_0(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_0:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_0:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    mov x9, x8
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
   ret void
 }
 
 define void @memset_0_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memset_0_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    mov x9, x8
 ; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
   ret void
 }
 
 define void @memset_10(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-WITHOUT-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-MOPS-NEXT:    strh  w8, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    // implicit-def: $x8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, w1
 ; O0-GISel-WITHOUT-MOPS-NEXT:    and x8, x8, #0xff
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov x9, #72340172838076673
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mul x8, x8, x9
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]

 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 false)
   ret void
 }
 
 define void @memset_10_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-WITHOUT-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-MOPS-NEXT:    strh  w8, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset

 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 true)
   ret void
 }
 
 define void @memset_10000(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset

 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 false)
   ret void
 }
 
 define void @memset_10000_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset

 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 true)
   ret void
 }
 
 define void @memset_size(i8* %dst, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w2 killed $w2 def $x2
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_size:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2

 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 false)
   ret void
 }
 
 define void @memset_size_volatile(i8* %dst, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w2 killed $w2 def $x2
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2

 ; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   %value_trunc = trunc i32 %value to i8
   call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 true)
   ret void
 }
 
+
 define void @memcpy_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_0:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
   ret void
 }
 
 define void @memcpy_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_0_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
   ret void
 }
 
 define void @memcpy_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldrh  w8, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-MOPS-NEXT:    strh  w8, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_10:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
 ; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
   ret void
 }
 
 define void @memcpy_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy

 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
   ret void
 }
 
 define void @memcpy_1000(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_1000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_1000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy

 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
   ret void
 }
 
 define void @memcpy_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_1000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_1000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy

 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
   ret void
 }
 
 define void @memcpy_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_n:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_n:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_n:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
   ret void
 }
 
 define void @memcpy_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_n_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_n_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_n_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 true)
   ret void
 }
 
+
 define void @memcpy_inline_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_inline_0:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
   ret void
 }
 
 define void @memcpy_inline_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_inline_0_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
   ret void
 }
 
 define void @memcpy_inline_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldrh  w8, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-MOPS-NEXT:    strh  w8, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_inline_10:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    ldr x8, [x1]
 ; O0-GISel-MOPS-NEXT:    str x8, [x0]
 ; O0-GISel-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
   ret void
 }
 
 define void @memcpy_inline_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memcpy_inline_10_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    ldr x8, [x1]
 ; O0-GISel-MOPS-NEXT:    str x8, [x0]
 ; O0-GISel-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
   ret void
 }
 
+
 define void @memmove_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memmove_0:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
   ret void
 }
 
 define void @memmove_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memmove_0_volatile:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov x8, xzr
 ; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
   ret void
 }
 
 define void @memmove_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x9, [x0]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memmove_10:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
 ; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
   ret void
 }
 
 define void @memmove_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove

 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
   ret void
 }
 
 define void @memmove_1000(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_1000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_1000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove

 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
   ret void
 }
 
 define void @memmove_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_1000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_1000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
 ; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove

 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
   ret void
 }
 
 define void @memmove_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_n:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_n:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret
 ;
 ; O0-GISel-MOPS-LABEL: memmove_n:
 ; O0-GISel-MOPS:       // %bb.0: // %entry
 ; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
 ; O0-GISel-MOPS-NEXT:    ret
 entry:
   call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
   ret void
 }
 
 define void @memmove_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_n_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_n_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
 ; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n_volatile:
 ; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
 ; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
 ; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; O0-GISel-WITHOUT-MOPS-NEXT:    ret