[AArch64] CodeGen for Armv8.8/9.3 MOPS

[tyb0807]

Summary

This implements codegen for Armv8.8/9.3 Memory Operations extension
(MOPS). Any memcpy/memset/memmov intrinsics will be always be emitted
as a series of three instructions which perform the operation.

In addition, this introduces a new ACLE intrinsic for memset tagged (see
https://github.com/ARM-software/acle/pull/38).

void *__builtin_arm_mops_memset_tag(void *, int, size_t)

A corresponding LLVM intrinsic is introduced:

i8* llvm.aarch64.mops.memset.tag(i8*, i8, i64)

The types match llvm.memset but the return type is not void.

1. SelectionDAG:
   • New target SDNodes are added: AArch64ISD::MOPS_MEMSET, etc. Each intrinsic is translated to one of these in SelectionDAGBuilder via EmitTargetCodeForMOPS.
   • A custom lowering routine for INTRINSIC_W_CHAIN is added to handle llvm.aarch64.mops.memset.tag. This takes a separate path from the common intrinsics but ultimately ends up in the same EmitMOPS().

2. GlobalIsel:
   • AArch64LegalizerInfo will now consider the following generic opcodes if +mops is available, instead of legalising by expanding them to libcalls: G_BZERO, G_MEMCPY_INLINE, G_MEMCPY, G_MEMMOVE, G_MEMSET The s8 value of memset is legalised to s64 to match the pseudos.
   • AArch64O0PreLegalizerCombinerInfo will not combine any of the generic opcodes. This means that small or zero sized memory operations will not be optimised out if +mops is present.
   • AArch64InstructionSelector will select the above as new pseudo instructions: AArch64::MOPSMemory{Copy/Move/Set/SetTagging} These are each expanded in the usual place to a series of three instructions (e.g. SETP/SETM/SETE) which must be emitted together. To avoid the scheduler moving unrelated instructions between parts of MOPS sequences, the sequences are placed inside Machine Instruction Bundles, making sure that late scheduler passes handle it as a single unit.
   • Furthermore, this adds the following additions to the LegalizerInfo API:
     
     + Add a 3-type version of customForCartesianProduct.
     
     + Expose immIdx() so that immediates can be marked checked. This is required for G_MEMCPY etc which have an immediate operand $tailcall, and debug builds of LLVM check that the immediates have all been handled by the legalizer.

Patch by Tomas Matheson and Lucas Prates.

[dmgreen]

This looks like a large patch. It may be better split up into a base/isel part, a clang part and a global isel part (which may require different reviewers).

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

836

It is probably better to expand the instruction later than to create a bundle.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

940

Are you sure we should _always_ be using them? I would expect a ldr+str to be better than a mov #4; cpyfp; cpyfm; cpyfe, for example.

11875

Can this safely set memVT to the size of a single element? That would need to be multiplied by the size of the memory being set, and if this is being used for aliasing info will be too small.

llvm/lib/Target/AArch64/AArch64InstrInfo.td

8350

These need to be marked as 96bit pseudos. They should also clobber NZCV (as should the real instructions above, but that doesn't look like it was ever changed after D116157)

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

28

LLVM doesn't need brackets around a single statement.

48

llvm_unreachable is far more common than llvm_unreachable_internal. It can be moved into the default case too, then it doesn't need a return after it.

53–54

There is commented out code here.

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

3

You likely don't need the -O2 and -O0 for llc run commands.

[tyb0807]

Following Dave's comments above, this patch is now split to 4 different patches
https://reviews.llvm.org/D117753
https://reviews.llvm.org/D117757
https://reviews.llvm.org/D117763
https://reviews.llvm.org/D117764
and may now be abandoned.

[tyb0807]

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

836

Please see https://reviews.llvm.org/D117763, the instruction is now expanded as late as possible in the pipeline (i.e. during code emission)

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

940

Please see https://reviews.llvm.org/D117764, we are now conservative on when to use MOPS instructions instead of ldr+str

11875

Please see https://reviews.llvm.org/D117764, the size is set to unknown.

llvm/lib/Target/AArch64/AArch64InstrInfo.td

8350

Changes to make real instructions NZCV clobbering here https://reviews.llvm.org/D117757

Changes to make pseudos NZCV clobbering (and specifying their size) here https://reviews.llvm.org/D117764

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

3

Agreed, we can do this without. However, this will change the generated code (thus the strings that these tests are looking for), hence I'm leaving those options. Please lmk if this is _really_ unwanted and I'll try to fix these tests

clang/include/clang/Basic/BuiltinsAArch64.def

 // Memory Tagging Extensions (MTE)
 BUILTIN(__builtin_arm_irg, "v*v*Ui", "t")
 BUILTIN(__builtin_arm_addg, "v*v*Ui", "t")
 BUILTIN(__builtin_arm_gmi, "Uiv*Ui", "t")
 BUILTIN(__builtin_arm_ldg, "v*v*", "t")
 BUILTIN(__builtin_arm_stg, "vv*", "t")
 BUILTIN(__builtin_arm_subp, "Uiv*v*", "t")
 
+// Memory Operations
+BUILTIN(__builtin_arm_mops_memset_tag, "v*v*iz", "")
+
 // Memory barrier
 BUILTIN(__builtin_arm_dmb, "vUi", "nc")
 BUILTIN(__builtin_arm_dsb, "vUi", "nc")
 BUILTIN(__builtin_arm_isb, "vUi", "nc")
 
 BUILTIN(__builtin_arm_jcvt, "Zid", "nc")
 
 // Prefetch

clang/lib/CodeGen/CGBuiltin.cpp

 //===---- CGBuiltin.cpp - Emit LLVM Code for builtins ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (CRCIntrinsicID != Intrinsic::not_intrinsic) {
     Function *F = CGM.getIntrinsic(CRCIntrinsicID);
 
     llvm::Type *DataTy = F->getFunctionType()->getParamType(1);
     Arg1 = Builder.CreateZExtOrBitCast(Arg1, DataTy);
 
     return Builder.CreateCall(F, {Arg0, Arg1});
   }
 
+  // Memory Operations (MOPS)
+  if (BuiltinID == AArch64::BI__builtin_arm_mops_memset_tag) {
+    Value *Dst = EmitScalarExpr(E->getArg(0));
+    Value *Val = EmitScalarExpr(E->getArg(1));
+    Value *Size = EmitScalarExpr(E->getArg(2));
+    Dst = Builder.CreatePointerCast(Dst, Int8PtrTy);
+    Val = Builder.CreateTrunc(Val, Int8Ty);
+    Size = Builder.CreateIntCast(Size, Int64Ty, false);
+    return Builder.CreateCall(
+        CGM.getIntrinsic(Intrinsic::aarch64_mops_memset_tag), {Dst, Val, Size});
+  }
+
   // Memory Tagging Extensions (MTE) Intrinsics
   Intrinsic::ID MTEIntrinsicID = Intrinsic::not_intrinsic;
   switch (BuiltinID) {
   case AArch64::BI__builtin_arm_irg:
     MTEIntrinsicID = Intrinsic::aarch64_irg; break;
   case  AArch64::BI__builtin_arm_addg:
     MTEIntrinsicID = Intrinsic::aarch64_addg; break;
   case  AArch64::BI__builtin_arm_gmi:

clang/lib/Headers/arm_acle.h

 /*===---- arm_acle.h - ARM Non-Neon intrinsics -----------------------------===
  *
  * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
  * See https://llvm.org/LICENSE.txt for license information.
  * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
  *
  *===-----------------------------------------------------------------------===
  */

 #define __arm_mte_create_random_tag(__ptr, __mask)  __builtin_arm_irg(__ptr, __mask)
 #define __arm_mte_increment_tag(__ptr, __tag_offset)  __builtin_arm_addg(__ptr, __tag_offset)
 #define __arm_mte_exclude_tag(__ptr, __excluded)  __builtin_arm_gmi(__ptr, __excluded)
 #define __arm_mte_get_tag(__ptr) __builtin_arm_ldg(__ptr)
 #define __arm_mte_set_tag(__ptr) __builtin_arm_stg(__ptr)
 #define __arm_mte_ptrdiff(__ptra, __ptrb) __builtin_arm_subp(__ptra, __ptrb)
 #endif
 
+/* Memory Operations Intrinsics */
+#if __ARM_FEATURE_MOPS && __ARM_FEATURE_MEMORY_TAGGING
+#define __arm_mops_memset_tag(tagged_address, value, size)                     \
+  __builtin_arm_mops_memset_tag(tagged_address, value, size)
+#endif
+
 /* Transactional Memory Extension (TME) Intrinsics */
 #if __ARM_FEATURE_TME
 
 #define _TMFAILURE_REASON  0x00007fffu
 #define _TMFAILURE_RTRY    0x00008000u
 #define _TMFAILURE_CNCL    0x00010000u
 #define _TMFAILURE_MEM     0x00020000u
 #define _TMFAILURE_IMP     0x00040000u

clang/test/CodeGen/aarch64-mops.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// RUN: %clang_cc1 -triple aarch64-arm-unknown-eabi -target-feature +mops -target-feature +mte -O0 -S -emit-llvm -o - %s  | FileCheck %s
+
+#define __ARM_FEATURE_MOPS 1
+#include <arm_acle.h>
+#include <stddef.h>
+
+// CHECK-LABEL: @bzero_0(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 0, i64 0)
+// CHECK-NEXT:    ret i8* [[TMP1]]
+//
+void *bzero_0(void *dst) {
+  return __arm_mops_memset_tag(dst, 0, 0);
+}
+
+// CHECK-LABEL: @bzero_1(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 0, i64 1)
+// CHECK-NEXT:    ret i8* [[TMP1]]
+//
+void *bzero_1(void *dst) {
+  return __arm_mops_memset_tag(dst, 0, 1);
+}
+
+// CHECK-LABEL: @bzero_10(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 0, i64 10)
+// CHECK-NEXT:    ret i8* [[TMP1]]
+//
+void *bzero_10(void *dst) {
+  return __arm_mops_memset_tag(dst, 0, 10);
+}
+
+// CHECK-LABEL: @bzero_10000(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 0, i64 10000)
+// CHECK-NEXT:    ret i8* [[TMP1]]
+//
+void *bzero_10000(void *dst) {
+  return __arm_mops_memset_tag(dst, 0, 10000);
+}
+
+// CHECK-LABEL: @bzero_n(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[SIZE_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i64 [[SIZE:%.*]], i64* [[SIZE_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i64, i64* [[SIZE_ADDR]], align 8
+// CHECK-NEXT:    [[TMP2:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 0, i64 [[TMP1]])
+// CHECK-NEXT:    ret i8* [[TMP2]]
+//
+void *bzero_n(void *dst, size_t size) {
+  return __arm_mops_memset_tag(dst, 0, size);
+}
+
+// CHECK-LABEL: @memset_0(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[VALUE_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[VALUE:%.*]], i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = trunc i32 [[TMP1]] to i8
+// CHECK-NEXT:    [[TMP3:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 [[TMP2]], i64 0)
+// CHECK-NEXT:    ret i8* [[TMP3]]
+//
+void *memset_0(void *dst, int value) {
+  return __arm_mops_memset_tag(dst, value, 0);
+}
+
+// CHECK-LABEL: @memset_1(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[VALUE_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[VALUE:%.*]], i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = trunc i32 [[TMP1]] to i8
+// CHECK-NEXT:    [[TMP3:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 [[TMP2]], i64 1)
+// CHECK-NEXT:    ret i8* [[TMP3]]
+//
+void *memset_1(void *dst, int value) {
+  return __arm_mops_memset_tag(dst, value, 1);
+}
+
+// CHECK-LABEL: @memset_10(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[VALUE_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[VALUE:%.*]], i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = trunc i32 [[TMP1]] to i8
+// CHECK-NEXT:    [[TMP3:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 [[TMP2]], i64 10)
+// CHECK-NEXT:    ret i8* [[TMP3]]
+//
+void *memset_10(void *dst, int value) {
+  return __arm_mops_memset_tag(dst, value, 10);
+}
+
+// CHECK-LABEL: @memset_10000(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[VALUE_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[VALUE:%.*]], i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = trunc i32 [[TMP1]] to i8
+// CHECK-NEXT:    [[TMP3:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 [[TMP2]], i64 10000)
+// CHECK-NEXT:    ret i8* [[TMP3]]
+//
+void *memset_10000(void *dst, int value) {
+  return __arm_mops_memset_tag(dst, value, 10000);
+}
+
+// CHECK-LABEL: @memset_n(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[DST_ADDR:%.*]] = alloca i8*, align 8
+// CHECK-NEXT:    [[VALUE_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[SIZE_ADDR:%.*]] = alloca i64, align 8
+// CHECK-NEXT:    store i8* [[DST:%.*]], i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    store i32 [[VALUE:%.*]], i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    store i64 [[SIZE:%.*]], i64* [[SIZE_ADDR]], align 8
+// CHECK-NEXT:    [[TMP0:%.*]] = load i8*, i8** [[DST_ADDR]], align 8
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[VALUE_ADDR]], align 4
+// CHECK-NEXT:    [[TMP2:%.*]] = load i64, i64* [[SIZE_ADDR]], align 8
+// CHECK-NEXT:    [[TMP3:%.*]] = trunc i32 [[TMP1]] to i8
+// CHECK-NEXT:    [[TMP4:%.*]] = call i8* @llvm.aarch64.mops.memset.tag(i8* [[TMP0]], i8 [[TMP3]], i64 [[TMP2]])
+// CHECK-NEXT:    ret i8* [[TMP4]]
+//
+void *memset_n(void *dst, int value, size_t size) {
+  return __arm_mops_memset_tag(dst, value, size);
+}

llvm/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h

 //===- llvm/CodeGen/GlobalISel/LegalizerInfo.h ------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
assert(TypeIdx <=
                (MCOI::OPERAND_LAST_GENERIC - MCOI::OPERAND_FIRST_GENERIC) &&
            "Type Index is out of bounds");
 #ifndef NDEBUG
     TypeIdxsCovered.set(TypeIdx);
 #endif
     return TypeIdx;
   }
 
-  unsigned immIdx(unsigned ImmIdx) {
-    assert(ImmIdx <= (MCOI::OPERAND_LAST_GENERIC_IMM -
-                      MCOI::OPERAND_FIRST_GENERIC_IMM) &&
-           "Imm Index is out of bounds");
-#ifndef NDEBUG
-    ImmIdxsCovered.set(ImmIdx);
-#endif
-    return ImmIdx;
-  }
-
   void markAllIdxsAsCovered() {
 #ifndef NDEBUG
     TypeIdxsCovered.set();
     ImmIdxsCovered.set();
 #endif
   }
 
   void add(const LegalizeRule &Rule) {
public:
   void aliasTo(unsigned Opcode) {
     assert((AliasOf == 0 || AliasOf == Opcode) &&
            "Opcode is already aliased to another opcode");
     assert(Rules.empty() && "Aliasing will discard rules");
     AliasOf = Opcode;
   }
   unsigned getAlias() const { return AliasOf; }
 
+  unsigned immIdx(unsigned ImmIdx) {
+    assert(ImmIdx <= (MCOI::OPERAND_LAST_GENERIC_IMM -
+                      MCOI::OPERAND_FIRST_GENERIC_IMM) &&
+           "Imm Index is out of bounds");
+#ifndef NDEBUG
+    ImmIdxsCovered.set(ImmIdx);
+#endif
+    return ImmIdx;
+  }
+
   /// The instruction is legal if predicate is true.
   LegalizeRuleSet &legalIf(LegalityPredicate Predicate) {
     // We have no choice but conservatively assume that the free-form
     // user-provided Predicate properly handles all type indices:
     markAllIdxsAsCovered();
     return actionIf(LegalizeAction::Legal, Predicate);
   }
   /// The instruction is legal when type index 0 is any type in the given list.
#endif
   /// given list.
   LegalizeRuleSet &customFor(std::initializer_list<std::pair<LLT, LLT>> Types) {
     return actionFor(LegalizeAction::Custom, Types);
   }
 
   LegalizeRuleSet &customForCartesianProduct(std::initializer_list<LLT> Types) {
     return actionForCartesianProduct(LegalizeAction::Custom, Types);
   }
+  /// The instruction is custom when type indexes 0 and 1 are both in their
+  /// respective lists.
   LegalizeRuleSet &
   customForCartesianProduct(std::initializer_list<LLT> Types0,
                             std::initializer_list<LLT> Types1) {
     return actionForCartesianProduct(LegalizeAction::Custom, Types0, Types1);
   }
+  /// The instruction is custom when when type indexes 0, 1, and 2 are all in
+  /// their respective lists.
+  LegalizeRuleSet &
+  customForCartesianProduct(std::initializer_list<LLT> Types0,
+                            std::initializer_list<LLT> Types1,
+                            std::initializer_list<LLT> Types2) {
+    return actionForCartesianProduct(LegalizeAction::Custom, Types0, Types1,
+                                     Types2);
+  }
 
   /// Unconditionally custom lower.
   LegalizeRuleSet &custom() {
     return customIf(always);
   }
 
   /// Widen the scalar to the next power of two that is at least MinSize.
   /// No effect if the type is not a scalar or is a power of two.

llvm/include/llvm/IR/IntrinsicsAArch64.td

 def int_aarch64_settag_zero  : DefaultAttrsIntrinsic<[], [llvm_ptr_ty, llvm_i64_ty],
     [IntrWriteMem, IntrArgMemOnly, NoCapture<ArgIndex<0>>, WriteOnly<ArgIndex<0>>]>;
 
 // Update allocation tags for 16-aligned, 16-sized memory region, and store a pair 8-byte values.
 def int_aarch64_stgp  : DefaultAttrsIntrinsic<[], [llvm_ptr_ty, llvm_i64_ty, llvm_i64_ty],
     [IntrWriteMem, IntrArgMemOnly, NoCapture<ArgIndex<0>>, WriteOnly<ArgIndex<0>>]>;
 }
 
+//===----------------------------------------------------------------------===//
+// Memory Operations (MOPS) Intrinsics
+let TargetPrefix = "aarch64" in {
+  // Sizes are chosen to correspond to the llvm.memset intrinsic: ptr, i8, i64
+  def int_aarch64_mops_memset_tag : DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_i8_ty, llvm_i64_ty],
+      [IntrWriteMem, IntrArgMemOnly, NoCapture<ArgIndex<0>>, WriteOnly<ArgIndex<0>>]>;
+}
+
 // Transactional Memory Extension (TME) Intrinsics
 let TargetPrefix = "aarch64" in {
 def int_aarch64_tstart  : GCCBuiltin<"__builtin_arm_tstart">,
                          Intrinsic<[llvm_i64_ty], [], [IntrWillReturn]>;
 
 def int_aarch64_tcommit : GCCBuiltin<"__builtin_arm_tcommit">, Intrinsic<[], [], [IntrWillReturn]>;
 
 def int_aarch64_tcancel : GCCBuiltin<"__builtin_arm_tcancel">,

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

 //===- AArch64ExpandPseudoInsts.cpp - Expand pseudo instructions ----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/LivePhysRegs.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineInstrBundle.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/MC/MCInstrDesc.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetMachine.h"
bool expandSetTagLoop(MachineBasicBlock &MBB,
                         MachineBasicBlock::iterator &NextMBBI);
   bool expandSVESpillFill(MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MBBI, unsigned Opc,
                           unsigned N);
   bool expandCALL_RVMARKER(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI);
   bool expandStoreSwiftAsyncContext(MachineBasicBlock &MBB,
                                     MachineBasicBlock::iterator MBBI);
+  bool expandMOPS(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+                  unsigned Opc);
 };
 
 } // end anonymous namespace
 
 char AArch64ExpandPseudo::ID = 0;
 
 INITIALIZE_PASS(AArch64ExpandPseudo, "aarch64-expand-pseudo",
                 AARCH64_EXPAND_PSEUDO_NAME, false, false)
BuildMI(MBB, MBBI, DL, TII->get(AArch64::STRXui))
       .addUse(BaseReg)
       .addImm(Offset / 8)
       .setMIFlag(MachineInstr::FrameSetup);
 
   MBBI->eraseFromParent();
   return true;
 }
 
+bool AArch64ExpandPseudo::expandMOPS(MachineBasicBlock &MBB,
+                                     MachineBasicBlock::iterator MBBI,
+                                     unsigned Pseudo) {
+  auto &STI = MBB.getParent()->getSubtarget<AArch64Subtarget>();
+  assert(STI.hasMOPS());
+  assert(STI.hasMTE() || Pseudo != AArch64::MOPSMemorySetTagging);
+
+  const auto Ops = [Pseudo]() -> std::array<unsigned, 3> {
+    if (Pseudo == AArch64::MOPSMemoryCopy)
+      return {AArch64::CPYFP, AArch64::CPYFM, AArch64::CPYFE};
+    if (Pseudo == AArch64::MOPSMemoryMove)
+      return {AArch64::CPYP, AArch64::CPYM, AArch64::CPYE};
+    if (Pseudo == AArch64::MOPSMemorySet)
+      return {AArch64::SETP, AArch64::SETM, AArch64::SETE};
+    if (Pseudo == AArch64::MOPSMemorySetTagging)
+      return {AArch64::SETGP, AArch64::SETGM, AArch64::MOPSSETGE};
+    llvm_unreachable("Unhandled memory operation pseudo");
+  }();
+  const bool IsSet = Pseudo == AArch64::MOPSMemorySet ||
+                     Pseudo == AArch64::MOPSMemorySetTagging;
+
+  // MOPS requires consecutive instructions in its sequences, so pack them
+  // inside a bundle to prevent other passes to moving things in between.
+  MIBundleBuilder Bundler(MBB, MBBI);
+  auto &MF = *MBB.getParent();
+  for (auto Op : Ops) {
+    auto B = BuildMI(MF, MBBI->getDebugLoc(), TII->get(Op));
+    int i = 0;
+    // Destination registers
+    B.addDef(MBBI->getOperand(i++).getReg());
+    B.addDef(MBBI->getOperand(i++).getReg());
+    if (!IsSet)
+      B.addDef(MBBI->getOperand(i++).getReg());
+    // Input registers
+    B.addUse(MBBI->getOperand(i++).getReg());
+    B.addUse(MBBI->getOperand(i++).getReg());
+    B.addUse(MBBI->getOperand(i++).getReg());
+    Bundler.append(B);
+  }
+  finalizeBundle(MBB, Bundler.begin(), Bundler.end());
+
+  MBBI->eraseFromParent();
+  return true;
+}
+
 /// If MBBI references a pseudo instruction that should be expanded here,
 /// do the expansion and return true.  Otherwise return false.
 bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator MBBI,
                                    MachineBasicBlock::iterator &NextMBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
 
bool AArch64ExpandPseudo::expandMI(MachineBasicBlock &MBB,
    case AArch64::LDR_ZZZXI:
      return expandSVESpillFill(MBB, MBBI, AArch64::LDR_ZXI, 3);
    case AArch64::LDR_ZZXI:
      return expandSVESpillFill(MBB, MBBI, AArch64::LDR_ZXI, 2);
    case AArch64::BLR_RVMARKER:
      return expandCALL_RVMARKER(MBB, MBBI);
    case AArch64::StoreSwiftAsyncContext:
      return expandStoreSwiftAsyncContext(MBB, MBBI);
+   case AArch64::MOPSMemoryCopy:
+   case AArch64::MOPSMemoryMove:
+   case AArch64::MOPSMemorySet:
+   case AArch64::MOPSMemorySetTagging:
+     return expandMOPS(MBB, MBBI, Opcode);
   }
   return false;
 }
 
 /// Iterate over the instructions in basic block MBB and expand any
 /// pseudo instructions.  Return true if anything was modified.
 bool AArch64ExpandPseudo::expandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;

llvm/lib/Target/AArch64/AArch64ISelLowering.h

 //==-- AArch64ISelLowering.h - AArch64 DAG Lowering Interface ----*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum NodeType : unsigned {
   STG,
   STZG,
   ST2G,
   STZ2G,
 
   LDP,
   STP,
   STNP,
+
+  // Memory Operations
+  MOPS_MEMSET,
+  MOPS_MEMSET_TAGGING,
+  MOPS_MEMCOPY,
+  MOPS_MEMMOVE,
 };
 
 } // end namespace AArch64ISD
 
 namespace {
 
 // Any instruction that defines a 32-bit result zeros out the high half of the
 // register. Truncate can be lowered to EXTRACT_SUBREG. CopyFromReg may
private:
   SDValue LowerStore128(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerABS(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue LowerMGATHER(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerMSCATTER(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue LowerMLOAD(SDValue Op, SelectionDAG &DAG) const;
 
+  SDValue LowerINTRINSIC_W_CHAIN(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINTRINSIC_WO_CHAIN(SDValue Op, SelectionDAG &DAG) const;
 
   bool isEligibleForTailCallOptimization(
       SDValue Callee, CallingConv::ID CalleeCC, bool isVarArg,
       const SmallVectorImpl<ISD::OutputArg> &Outs,
       const SmallVectorImpl<SDValue> &OutVals,
       const SmallVectorImpl<ISD::InputArg> &Ins, SelectionDAG &DAG) const;
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 //===-- AArch64ISelLowering.cpp - AArch64 DAG Lowering Implementation  ----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
#undef LCALLNAME5
   setTargetDAGCombine(ISD::EXTRACT_VECTOR_ELT);
   setTargetDAGCombine(ISD::VECREDUCE_ADD);
   setTargetDAGCombine(ISD::STEP_VECTOR);
 
   setTargetDAGCombine(ISD::FP_EXTEND);
 
   setTargetDAGCombine(ISD::GlobalAddress);
 
+  if (Subtarget->hasMOPS()) {
+    // If we have MOPS, always use them
+    MaxStoresPerMemsetOptSize = 0;
+    MaxStoresPerMemset = 0;
+    MaxGluedStoresPerMemcpy = 0;
+    MaxStoresPerMemcpyOptSize = 0;
+    MaxStoresPerMemcpy = 0;
+    MaxStoresPerMemmoveOptSize = 0;
+    MaxStoresPerMemmove = 0;
+  } else {
     // In case of strict alignment, avoid an excessive number of byte wide stores.
     MaxStoresPerMemsetOptSize = 8;
     MaxStoresPerMemset = Subtarget->requiresStrictAlign()
                         ? MaxStoresPerMemsetOptSize : 32;
 
     MaxGluedStoresPerMemcpy = 4;
     MaxStoresPerMemcpyOptSize = 4;
     MaxStoresPerMemcpy = Subtarget->requiresStrictAlign()
                         ? MaxStoresPerMemcpyOptSize : 16;
 
     MaxStoresPerMemmoveOptSize = MaxStoresPerMemmove = 4;
+  }
 
   MaxLoadsPerMemcmpOptSize = 4;
   MaxLoadsPerMemcmp = Subtarget->requiresStrictAlign()
                       ? MaxLoadsPerMemcmpOptSize : 8;
 
   setStackPointerRegisterToSaveRestore(AArch64::SP);
 
   setSchedulingPreference(Sched::Hybrid);
if (Subtarget->hasSVE()) {
     }
 
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv2i1, MVT::nxv2i64);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv4i1, MVT::nxv4i32);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv8i1, MVT::nxv8i16);
     setOperationPromotedToType(ISD::VECTOR_SPLICE, MVT::nxv16i1, MVT::nxv16i8);
   }
 
+  if (Subtarget->hasMOPS() && Subtarget->hasMTE()) {
+    // Only required for llvm.aarch64.mops.memset.tag
+    setOperationAction(ISD::INTRINSIC_W_CHAIN, MVT::i8, Custom);
+  }
+
   PredictableSelectIsExpensive = Subtarget->predictableSelectIsExpensive();
 }
 
 void AArch64TargetLowering::addTypeForNEON(MVT VT) {
   assert(VT.isVector() && "VT should be a vector type");
 
   if (VT.isFloatingPoint()) {
     MVT PromoteTo = EVT(VT).changeVectorElementTypeToInteger().getSimpleVT();
case AArch64ISD::FIRST_NUMBER:
     MAKE_CASE(AArch64ISD::REVW_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTLZ_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::CTPOP_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::DUP_MERGE_PASSTHRU)
     MAKE_CASE(AArch64ISD::INDEX_VECTOR)
     MAKE_CASE(AArch64ISD::UADDLP)
     MAKE_CASE(AArch64ISD::CALL_RVMARKER)
     MAKE_CASE(AArch64ISD::ASSERT_ZEXT_BOOL)
+    MAKE_CASE(AArch64ISD::MOPS_MEMSET)
+    MAKE_CASE(AArch64ISD::MOPS_MEMSET_TAGGING)
+    MAKE_CASE(AArch64ISD::MOPS_MEMCOPY)
+    MAKE_CASE(AArch64ISD::MOPS_MEMMOVE)
   }
 #undef MAKE_CASE
   return nullptr;
 }
 
 MachineBasicBlock *
 AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const {
static SDValue lowerConvertToSVBool(SDValue Op, SelectionDAG &DAG) {
 
   // Otherwise, zero the newly introduced lanes.
   SDValue Mask = getPTrue(DAG, DL, InVT, AArch64SVEPredPattern::all);
   SDValue MaskReinterpret =
       DAG.getNode(AArch64ISD::REINTERPRET_CAST, DL, OutVT, Mask);
   return DAG.getNode(ISD::AND, DL, OutVT, Reinterpret, MaskReinterpret);
 }
 
+SDValue AArch64TargetLowering::LowerINTRINSIC_W_CHAIN(SDValue Op,
+                                                      SelectionDAG &DAG) const {
+  unsigned IntNo = Op.getConstantOperandVal(1);
+  switch (IntNo) {
+  default:
+    return SDValue(); // Don't custom lower most intrinsics.
+  case Intrinsic::aarch64_mops_memset_tag: {
+    auto Node = cast<MemIntrinsicSDNode>(Op.getNode());
+    SDLoc DL(Op);
+    SDValue Chain = Node->getChain();
+    SDValue Dst = Op.getOperand(2);
+    SDValue Val = Op.getOperand(3);
+    Val = DAG.getAnyExtOrTrunc(Val, DL, MVT::i64);
+    SDValue Size = Op.getOperand(4);
+    auto Alignment = Node->getMemOperand()->getAlign();
+    bool IsVol = Node->isVolatile();
+    auto DstPtrInfo = Node->getPointerInfo();
+
+    const auto &SDI =
+        static_cast<const AArch64SelectionDAGInfo &>(DAG.getSelectionDAGInfo());
+    SDValue MS =
+        SDI.EmitMOPS(AArch64ISD::MOPS_MEMSET_TAGGING, DAG, DL, Chain, Dst, Val,
+                     Size, Alignment, IsVol, DstPtrInfo, MachinePointerInfo{});
+
+    // MOPS_MEMSET_TAGGING has 3 results (DstWb, SizeWb, Chain) whereas the
+    // intrinsic has 2. So hide SizeWb it using MERGE_VALUES. Otherwise
+    // LowerOperationWrapper will complain that the number of results has
+    // changed.
+    return DAG.getMergeValues({MS.getValue(0), MS.getValue(2)}, DL);
+  }
+  }
+}
+
 SDValue AArch64TargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
                                                      SelectionDAG &DAG) const {
   unsigned IntNo = cast<ConstantSDNode>(Op.getOperand(0))->getZExtValue();
   SDLoc dl(Op);
   switch (IntNo) {
   default: return SDValue();    // Don't custom lower most intrinsics.
   case Intrinsic::thread_pointer: {
     EVT PtrVT = getPointerTy(DAG.getDataLayout());
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::MUL:
     return LowerMUL(Op, DAG);
   case ISD::MULHS:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHS_PRED,
                                /*OverrideNEON=*/true);
   case ISD::MULHU:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHU_PRED,
                                /*OverrideNEON=*/true);
+  case ISD::INTRINSIC_W_CHAIN:
+    return LowerINTRINSIC_W_CHAIN(Op, DAG);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
   case ISD::ATOMIC_STORE:
     if (cast<MemSDNode>(Op)->getMemoryVT() == MVT::i128) {
       assert(Subtarget->hasLSE2());
       return LowerStore128(Op, DAG);
     }
     return SDValue();
case Intrinsic::aarch64_sve_stnt1: {
     Info.opc = ISD::INTRINSIC_W_CHAIN;
     Info.memVT = MVT::getVT(I.getOperand(0)->getType());
     Info.ptrVal = I.getArgOperand(2);
     Info.offset = 0;
     Info.align = DL.getABITypeAlign(PtrTy->getElementType());
     Info.flags = MachineMemOperand::MOStore | MachineMemOperand::MONonTemporal;
     return true;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    Value *Dst = I.getArgOperand(0);
+    Value *Val = I.getArgOperand(1);
+    PointerType *PtrTy = cast<PointerType>(Dst->getType());
+    Info.opc = ISD::INTRINSIC_W_CHAIN;
+    Info.memVT = MVT::getVT(Val->getType());
+    Info.ptrVal = Dst;
+    Info.offset = 0;
+    Info.align = DL.getABITypeAlign(PtrTy->getElementType());
+    Info.flags = MachineMemOperand::MOStore;
+    return true;
+  }
   default:
     break;
   }
 
   return false;
 }
 
 bool AArch64TargetLowering::shouldReduceLoadWidth(SDNode *Load,

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 }
 let Predicates = [HasMOPS, HasMTE] in {
   defm SETGP     : MOPSMemorySetTaggingInsns<0b00, "setgp">;
   defm SETGM     : MOPSMemorySetTaggingInsns<0b01, "setgm">;
   // Can't use SETGE because it's a reserved name in TargetSelectionDAG.td
   defm MOPSSETGE : MOPSMemorySetTaggingInsns<0b10, "setge">;
 }
 
+// MOPS Node operands: 0: Dst, 1: Src or Value, 2: Size, 3: Chain
+// MOPS Node results: 0: Dst writeback, 1: Size writeback, 2: Chain
+def SDT_AArch64mops : SDTypeProfile<2, 3, [ SDTCisInt<0>, SDTCisInt<1>, SDTCisInt<2> ]>;
+def AArch64mops_memset : SDNode<"AArch64ISD::MOPS_MEMSET", SDT_AArch64mops>;
+def AArch64mops_memset_tagging : SDNode<"AArch64ISD::MOPS_MEMSET_TAGGING", SDT_AArch64mops>;
+def AArch64mops_memcopy : SDNode<"AArch64ISD::MOPS_MEMCOPY", SDT_AArch64mops>;
+def AArch64mops_memmove : SDNode<"AArch64ISD::MOPS_MEMMOVE", SDT_AArch64mops>;
+
+let Predicates = [HasMOPS], mayStore = 1 in {
+  let mayLoad = 1 in {
+    def MOPSMemoryCopy : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
+                                (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
+                                [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;
+    def MOPSMemoryMove : Pseudo<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
+                                (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
+                                [], "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb">, Sched<[]>;
+  }
+  let mayLoad = 0 in {
+    def MOPSMemorySet  : Pseudo<(outs GPR64common:$Rd_wb, GPR64:$Rn_wb),
+                                (ins GPR64common:$Rd, GPR64:$Rn, GPR64:$Rm),
+                                [], "$Rd = $Rd_wb,$Rn = $Rn_wb">, Sched<[]>;
+  }
+}
+let Predicates = [HasMOPS, HasMTE], mayLoad = 0, mayStore = 1 in {
+  def MOPSMemorySetTagging : Pseudo<(outs GPR64common:$Rd_wb, GPR64:$Rn_wb),
+                                    (ins GPR64common:$Rd, GPR64:$Rn, GPR64:$Rm),
+                                    [], "$Rd = $Rd_wb,$Rn = $Rn_wb">, Sched<[]>;
+}
+
 let Defs = [X16, X17], mayStore = 1, isCodeGenOnly = 1 in
 def StoreSwiftAsyncContext
       : Pseudo<(outs), (ins GPR64:$ctx, GPR64sp:$base, simm9:$offset),
                []>, Sched<[]>;
 
 def AArch64AssertZExtBool : SDNode<"AArch64ISD::ASSERT_ZEXT_BOOL", SDT_assert>;
 def : Pat<(AArch64AssertZExtBool GPR32:$op),
           (i32 GPR32:$op)>;
 
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"
 include "AArch64SMEInstrInfo.td"
 include "AArch64InstrGISel.td"

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

 //===-- AArch64SelectionDAGInfo.h - AArch64 SelectionDAG Info ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the AArch64 subclass for SelectionDAGTargetInfo.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_AARCH64_AARCH64SELECTIONDAGINFO_H
 #define LLVM_LIB_TARGET_AARCH64_AARCH64SELECTIONDAGINFO_H
 
 #include "llvm/CodeGen/SelectionDAGTargetInfo.h"
 
 namespace llvm {
 
 class AArch64SelectionDAGInfo : public SelectionDAGTargetInfo {
 public:
+  SDValue EmitMOPS(AArch64ISD::NodeType SDOpcode, SelectionDAG &DAG,
+                   const SDLoc &DL, SDValue Chain, SDValue Dst,
+                   SDValue SrcOrValue, SDValue Size, Align Alignment,
+                   bool isVolatile, MachinePointerInfo DstPtrInfo,
+                   MachinePointerInfo SrcPtrInfo) const;
+
+  SDValue EmitTargetCodeForMemcpy(SelectionDAG &DAG, const SDLoc &dl,
+                                  SDValue Chain, SDValue Dst, SDValue Src,
+                                  SDValue Size, Align Alignment,
+                                  bool isVolatile, bool AlwaysInline,
+                                  MachinePointerInfo DstPtrInfo,
+                                  MachinePointerInfo SrcPtrInfo) const override;
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
                                   bool isVolatile,
                                   MachinePointerInfo DstPtrInfo) const override;
+  SDValue
+  EmitTargetCodeForMemmove(SelectionDAG &DAG, const SDLoc &dl, SDValue Chain,
+                           SDValue Dst, SDValue Src, SDValue Size,
+                           Align Alignment, bool isVolatile,
+                           MachinePointerInfo DstPtrInfo,
+                           MachinePointerInfo SrcPtrInfo) const override;
+
   SDValue EmitTargetCodeForSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   MachinePointerInfo DstPtrInfo,
                                   bool ZeroData) const override;
 };
 }
 
 #endif

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

 //===-- AArch64SelectionDAGInfo.cpp - AArch64 SelectionDAG Info -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the AArch64SelectionDAGInfo class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64TargetMachine.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-selectiondag-info"
 
+SDValue AArch64SelectionDAGInfo::EmitMOPS(AArch64ISD::NodeType SDOpcode,
+                                          SelectionDAG &DAG, const SDLoc &DL,
+                                          SDValue Chain, SDValue Dst,
+                                          SDValue SrcOrValue, SDValue Size,
+                                          Align Alignment, bool isVolatile,
+                                          MachinePointerInfo DstPtrInfo,
+                                          MachinePointerInfo SrcPtrInfo) const {
+
+  // Get the constant size of the copy/set. We don't use it.
+  uint64_t ConstSize = 0;
+  if (auto *C = dyn_cast<ConstantSDNode>(Size)) {
+    ConstSize = cast<ConstantSDNode>(Size)->getZExtValue();
+  }
+
+  const bool IsSet = SDOpcode == AArch64ISD::MOPS_MEMSET ||
+                     SDOpcode == AArch64ISD::MOPS_MEMSET_TAGGING;
+
+  const auto MachineOpcode = [&]() {
+    switch (SDOpcode) {
+    case AArch64ISD::MOPS_MEMSET:
+      return AArch64::MOPSMemorySet;
+    case AArch64ISD::MOPS_MEMSET_TAGGING:
+      return AArch64::MOPSMemorySetTagging;
+    case AArch64ISD::MOPS_MEMCOPY:
+      return AArch64::MOPSMemoryCopy;
+    case AArch64ISD::MOPS_MEMMOVE:
+      return AArch64::MOPSMemoryMove;
+    default:
+      break;
+    }
+    llvm_unreachable_internal("Unhandled MOPS ISD Opcode");
+    return AArch64::INSTRUCTION_LIST_END;
+  }();
+
+  MachineMemOperand::Flags Flags = MachineMemOperand::MOStore;
+  // if (!Temporal)
+  //   Flags |= MachineMemOperand::MONonTemporal;
+  if (isVolatile)
+    Flags |= MachineMemOperand::MOVolatile;
+  if (!IsSet)
+    Flags |= MachineMemOperand::MOLoad;
+
+  MachineFunction &MF = DAG.getMachineFunction();
+
+  auto *DstOp =
+      MF.getMachineMemOperand(DstPtrInfo, Flags, ConstSize, Alignment);
+  auto *SrcOp =
+      MF.getMachineMemOperand(SrcPtrInfo, Flags, ConstSize, Alignment);
+
+  // Extend i8 value to i64 if required
+  if (SrcOrValue.getValueType().getSimpleVT() == MVT::i8) {
+    SrcOrValue = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, SrcOrValue);
+  }
+
+  if (IsSet) {
+    SDValue Ops[] = {Dst, Size, SrcOrValue, Chain};
+    const EVT ResultTys[] = {MVT::i64, MVT::i64, MVT::Other};
+    MachineSDNode *Node = DAG.getMachineNode(MachineOpcode, DL, ResultTys, Ops);
+    DAG.setNodeMemRefs(Node, {DstOp});
+    return SDValue(Node, 2);
+  } else {
+    SDValue Ops[] = {Dst, SrcOrValue, Size, Chain};
+    const EVT ResultTys[] = {MVT::i64, MVT::i64, MVT::i64, MVT::Other};
+    MachineSDNode *Node = DAG.getMachineNode(MachineOpcode, DL, ResultTys, Ops);
+    DAG.setNodeMemRefs(Node, {DstOp, SrcOp});
+    return SDValue(Node, 3);
+  }
+}
+
+SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemcpy(
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst, SDValue Src,
+    SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
+    MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+  if (STI.hasMOPS())
+    return EmitMOPS(AArch64ISD::MOPS_MEMCOPY, DAG, DL, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, SrcPtrInfo);
+  return SDValue();
+}
+
 SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, Align Alignment, bool isVolatile,
     MachinePointerInfo DstPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+
+  if (STI.hasMOPS()) {
+    return EmitMOPS(AArch64ISD::MOPS_MEMSET, DAG, dl, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, MachinePointerInfo{});
+  }
+
   // Check to see if there is a specialized entry-point for memory zeroing.
   ConstantSDNode *V = dyn_cast<ConstantSDNode>(Src);
   ConstantSDNode *SizeValue = dyn_cast<ConstantSDNode>(Size);
-  const AArch64Subtarget &STI =
-      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
   const char *bzeroName =
       (V && V->isZero())
           ? DAG.getTargetLoweringInfo().getLibcallName(RTLIB::BZERO)
           : nullptr;
   // For small size (< 256), it is not beneficial to use bzero
   // instead of memset.
   if (bzeroName && (!SizeValue || SizeValue->getZExtValue() > 256)) {
     const AArch64TargetLowering &TLI = *STI.getTargetLowering();
CLI.setDebugLoc(dl)
                       std::move(Args))
         .setDiscardResult();
     std::pair<SDValue, SDValue> CallResult = TLI.LowerCallTo(CLI);
     return CallResult.second;
   }
   return SDValue();
 }
 
+SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemmove(
+    SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
+    SDValue Size, Align Alignment, bool isVolatile,
+    MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
+  const AArch64Subtarget &STI =
+      DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
+  if (STI.hasMOPS()) {
+    return EmitMOPS(AArch64ISD::MOPS_MEMMOVE, DAG, dl, Chain, Dst, Src, Size,
+                    Alignment, isVolatile, DstPtrInfo, SrcPtrInfo);
+  }
+  return SDValue();
+}
+
 static const int kSetTagLoopThreshold = 176;
 
 static SDValue EmitUnrolledSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Ptr, uint64_t ObjSize,
                                   const MachineMemOperand *BaseMemOperand,
                                   bool ZeroData) {
   MachineFunction &MF = DAG.getMachineFunction();
   unsigned ObjSizeScaled = ObjSize / 16;

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

 //===- AArch64InstructionSelector.cpp ----------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
private:
   bool selectIntrinsic(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectVectorICmp(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectIntrinsicTrunc(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectIntrinsicRound(MachineInstr &I, MachineRegisterInfo &MRI) const;
   bool selectJumpTable(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectBrJT(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectTLSGlobalValue(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectReduction(MachineInstr &I, MachineRegisterInfo &MRI);
+  bool selectMOPS(MachineInstr &I, MachineRegisterInfo &MRI);
   bool selectUSMovFromExtend(MachineInstr &I, MachineRegisterInfo &MRI);
 
   unsigned emitConstantPoolEntry(const Constant *CPVal,
                                  MachineFunction &MF) const;
   MachineInstr *emitLoadFromConstantPool(const Constant *CPVal,
                                          MachineIRBuilder &MIRBuilder) const;
 
   // Emit a vector concat operation.
case TargetOpcode::G_INSERT_VECTOR_ELT:
     return selectInsertElt(I, MRI);
   case TargetOpcode::G_CONCAT_VECTORS:
     return selectConcatVectors(I, MRI);
   case TargetOpcode::G_JUMP_TABLE:
     return selectJumpTable(I, MRI);
   case TargetOpcode::G_VECREDUCE_FADD:
   case TargetOpcode::G_VECREDUCE_ADD:
     return selectReduction(I, MRI);
+  case TargetOpcode::G_BZERO:
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMCPY_INLINE:
+  case TargetOpcode::G_MEMMOVE:
+  case TargetOpcode::G_MEMSET:
+    assert(STI.hasMOPS() && "Shouldn't get here without +mops feature");
+    return selectMOPS(I, MRI);
   }
 
   return false;
 }
 
 bool AArch64InstructionSelector::selectReduction(MachineInstr &I,
                                                  MachineRegisterInfo &MRI) {
   Register VecReg = I.getOperand(1).getReg();
else {
       return false;
     }
     I.setDesc(TII.get(Opc));
     return constrainSelectedInstRegOperands(I, TII, TRI, RBI);
   }
   return false;
 }
 
+bool AArch64InstructionSelector::selectMOPS(MachineInstr &GI,
+                                            MachineRegisterInfo &MRI) {
+  assert(GI.getOpcode() != TargetOpcode::G_BZERO &&
+         "There is no point combining to G_BZERO only to re-materialize the "
+         "zero.");
+
+  unsigned Mopcode;
+  switch (GI.getOpcode()) {
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMCPY_INLINE:
+    Mopcode = AArch64::MOPSMemoryCopy;
+    break;
+  case TargetOpcode::G_MEMMOVE:
+    Mopcode = AArch64::MOPSMemoryMove;
+    break;
+  case TargetOpcode::G_MEMSET:
+    // For tagged memset see llvm.aarch64.mops.memset.tag
+    Mopcode = AArch64::MOPSMemorySet;
+    break;
+  }
+
+  auto &DstPtr = GI.getOperand(0);
+  auto &SrcOrVal = GI.getOperand(1);
+  auto &Size = GI.getOperand(2);
+
+  // Create copies of the registers that can be clobbered.
+  const Register DstPtrCopy = MRI.cloneVirtualRegister(DstPtr.getReg());
+  const Register SrcValCopy = MRI.cloneVirtualRegister(SrcOrVal.getReg());
+  const Register SizeCopy = MRI.cloneVirtualRegister(Size.getReg());
+
+  const bool IsSet = Mopcode == AArch64::MOPSMemorySet;
+  const auto &SrcValRegClass =
+      IsSet ? AArch64::GPR64RegClass : AArch64::GPR64commonRegClass;
+
+  // Constrain to specific registers
+  RBI.constrainGenericRegister(DstPtrCopy, AArch64::GPR64commonRegClass, MRI);
+  RBI.constrainGenericRegister(SrcValCopy, SrcValRegClass, MRI);
+  RBI.constrainGenericRegister(SizeCopy, AArch64::GPR64RegClass, MRI);
+
+  MIB.buildCopy(DstPtrCopy, DstPtr);
+  MIB.buildCopy(SrcValCopy, SrcOrVal);
+  MIB.buildCopy(SizeCopy, Size);
+
+  // New instruction uses the copied registers because it must update them.
+  // The defs are not used since they don't exist in G_MEM*. They are still
+  // tied.
+  // Note: order of operands is different from G_MEMSET, G_MEMCPY, G_MEMMOVE
+  Register DefDstPtr = MRI.createVirtualRegister(&AArch64::GPR64commonRegClass);
+  Register DefSize = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
+  if (IsSet) {
+    MIB.buildInstr(Mopcode, {DefDstPtr, DefSize},
+                   {DstPtrCopy, SizeCopy, SrcValCopy});
+  } else {
+    Register DefSrcPtr = MRI.createVirtualRegister(&SrcValRegClass);
+    MIB.buildInstr(Mopcode, {DefDstPtr, DefSrcPtr, DefSize},
+                   {DstPtrCopy, SrcValCopy, SizeCopy});
+  }
+
+  GI.eraseFromParent();
+  return true;
+}
+
 bool AArch64InstructionSelector::selectBrJT(MachineInstr &I,
                                             MachineRegisterInfo &MRI) {
   assert(I.getOpcode() == TargetOpcode::G_BRJT && "Expected G_BRJT");
   Register JTAddr = I.getOperand(0).getReg();
   unsigned JTI = I.getOperand(1).getIndex();
   Register Index = I.getOperand(2).getReg();
 
   Register TargetReg = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
case Intrinsic::aarch64_neon_st2: {
     SmallVector<Register, 2> Regs = {Src1, Src2};
     Register Tuple = Ty.getSizeInBits() == 128 ? createQTuple(Regs, MIB)
                                                : createDTuple(Regs, MIB);
     auto Store = MIB.buildInstr(Opc, {}, {Tuple, Ptr});
     Store.cloneMemRefs(I);
     constrainSelectedInstRegOperands(*Store, TII, TRI, RBI);
     break;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    // Transform
+    //    %dst:gpr(p0) = \
+    //      G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.aarch64.mops.memset.tag),
+    //      \ %dst:gpr(p0), %val:gpr(s64), %n:gpr(s64)
+    // where %dst is updated, into
+    //    %Rd:GPR64common, %Rn:GPR64) = \
+    //      MOPSMemorySetTagging \
+    //      %Rd:GPR64common, %Rn:GPR64, %Rm:GPR64
+    // where Rd and Rn are tied.
+    // It is expected that %val has been extended to s64 in legalization.
+    // Note that the order of the size/value operands are swapped.
+
+    Register DstDef = I.getOperand(0).getReg();
+    // I.getOperand(1) is the intrinsic function
+    Register DstUse = I.getOperand(2).getReg();
+    Register ValUse = I.getOperand(3).getReg();
+    Register SizeUse = I.getOperand(4).getReg();
+
+    // MOPSMemorySetTagging has two defs; the intrinsic call has only one.
+    // Therefore an additional virtual register is requried for the updated size
+    // operand. This value is not accessible via the semantics of the intrinsic.
+    Register SizeDef = MRI.createGenericVirtualRegister(LLT::scalar(64));
+
+    auto Memset = MIB.buildInstr(AArch64::MOPSMemorySetTagging,
+                                 {DstDef, SizeDef}, {DstUse, SizeUse, ValUse});
+    Memset.cloneMemRefs(I);
+    constrainSelectedInstRegOperands(*Memset, TII, TRI, RBI);
+    break;
+  }
   }
 
   I.eraseFromParent();
   return true;
 }
 
 bool AArch64InstructionSelector::selectIntrinsic(MachineInstr &I,
                                                  MachineRegisterInfo &MRI) {

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.h

 //===- AArch64LegalizerInfo --------------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
bool legalizeBitfieldExtract(MachineInstr &MI, MachineRegisterInfo &MRI,
                                LegalizerHelper &Helper) const;
   bool legalizeRotate(MachineInstr &MI, MachineRegisterInfo &MRI,
                       LegalizerHelper &Helper) const;
   bool legalizeCTPOP(MachineInstr &MI, MachineRegisterInfo &MRI,
                      LegalizerHelper &Helper) const;
   bool legalizeAtomicCmpxchg128(MachineInstr &MI, MachineRegisterInfo &MRI,
                                 LegalizerHelper &Helper) const;
   bool legalizeCTTZ(MachineInstr &MI, LegalizerHelper &Helper) const;
+  bool legalizeMemOps(MachineInstr &MI, LegalizerHelper &Helper) const;
   const AArch64Subtarget *ST;
 };
 } // End llvm namespace.
 #endif

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

 //===- AArch64LegalizerInfo.cpp ----------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST)
   getActionDefinitionsBuilder(G_JUMP_TABLE).legalFor({{p0}, {s64}});
 
   getActionDefinitionsBuilder(G_BRJT).legalIf([=](const LegalityQuery &Query) {
     return Query.Types[0] == p0 && Query.Types[1] == s64;
   });
 
   getActionDefinitionsBuilder(G_DYN_STACKALLOC).lower();
 
+  if (ST.hasMOPS()) {
+    // G_BZERO is not supported. Currently it is only emitted by
+    // PreLegalizerCombiner for G_MEMSET with zero constant.
+    getActionDefinitionsBuilder(G_BZERO).unsupported();
+
+    getActionDefinitionsBuilder(G_MEMSET)
+        .legalForCartesianProduct({p0}, {s64}, {s64})
+        .customForCartesianProduct({p0}, {s8}, {s64})
+        .immIdx(0); // Inform verifier imm idx 0 is handled.
+
+    getActionDefinitionsBuilder({G_MEMCPY, G_MEMMOVE})
+        .legalForCartesianProduct({p0}, {p0}, {s64})
+        .immIdx(0); // Inform verifier imm idx 0 is handled.
+
+    // G_MEMCPY_INLINE does not have a tailcall immediate
+    getActionDefinitionsBuilder(G_MEMCPY_INLINE)
+        .legalForCartesianProduct({p0}, {p0}, {s64});
+
+  } else {
     getActionDefinitionsBuilder({G_BZERO, G_MEMCPY, G_MEMMOVE, G_MEMSET})
         .libcall();
+  }
 
   // FIXME: Legal types are only legal with NEON.
   getActionDefinitionsBuilder(G_ABS)
       .lowerIf(isScalar(0))
       .legalFor(PackedVectorAllTypeList);
 
   getActionDefinitionsBuilder(G_VECREDUCE_FADD)
       // We only have FADDP to do reduction-like operations. Lower the rest.
bool AArch64LegalizerInfo::legalizeCustom(LegalizerHelper &Helper,
   case TargetOpcode::G_ROTR:
     return legalizeRotate(MI, MRI, Helper);
   case TargetOpcode::G_CTPOP:
     return legalizeCTPOP(MI, MRI, Helper);
   case TargetOpcode::G_ATOMIC_CMPXCHG:
     return legalizeAtomicCmpxchg128(MI, MRI, Helper);
   case TargetOpcode::G_CTTZ:
     return legalizeCTTZ(MI, Helper);
+  case TargetOpcode::G_BZERO:
+  case TargetOpcode::G_MEMCPY:
+  case TargetOpcode::G_MEMMOVE:
+  case TargetOpcode::G_MEMSET:
+    return legalizeMemOps(MI, Helper);
   }
 
   llvm_unreachable("expected switch to return");
 }
 
 bool AArch64LegalizerInfo::legalizeRotate(MachineInstr &MI,
                                           MachineRegisterInfo &MRI,
                                           LegalizerHelper &Helper) const {
case Intrinsic::vacopy: {
     return true;
   }
   case Intrinsic::get_dynamic_area_offset: {
     MachineIRBuilder &MIB = Helper.MIRBuilder;
     MIB.buildConstant(MI.getOperand(0).getReg(), 0);
     MI.eraseFromParent();
     return true;
   }
+  case Intrinsic::aarch64_mops_memset_tag: {
+    assert(MI.getOpcode() == TargetOpcode::G_INTRINSIC_W_SIDE_EFFECTS);
+    // Zext the value to 64 bit
+    MachineIRBuilder MIB(MI);
+    auto &Value = MI.getOperand(3);
+    Register ZExtValueReg = MIB.buildAnyExt(LLT::scalar(64), Value).getReg(0);
+    Value.setReg(ZExtValueReg);
+    return true;
+  }
   }
 
   return true;
 }
 
 bool AArch64LegalizerInfo::legalizeShlAshrLshr(
     MachineInstr &MI, MachineRegisterInfo &MRI, MachineIRBuilder &MIRBuilder,
     GISelChangeObserver &Observer) const {
bool AArch64LegalizerInfo::legalizeCTTZ(MachineInstr &MI,
   MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
   MachineRegisterInfo &MRI = *MIRBuilder.getMRI();
   LLT Ty = MRI.getType(MI.getOperand(1).getReg());
   auto BitReverse = MIRBuilder.buildBitReverse(Ty, MI.getOperand(1));
   MIRBuilder.buildCTLZ(MI.getOperand(0).getReg(), BitReverse);
   MI.eraseFromParent();
   return true;
 }
+
+bool AArch64LegalizerInfo::legalizeMemOps(MachineInstr &MI,
+                                          LegalizerHelper &Helper) const {
+  MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
+
+  // Tagged version MOPSMemorySetTagged is legalised in legalizeIntrinsic
+  if (MI.getOpcode() == TargetOpcode::G_MEMSET) {
+    // Zext the value operand to 64 bit
+    auto &Value = MI.getOperand(1);
+    Register ZExtValueReg =
+        MIRBuilder.buildAnyExt(LLT::scalar(64), Value).getReg(0);
+    Value.setReg(ZExtValueReg);
+    return true;
+  }
+
+  return false;
+}

llvm/lib/Target/AArch64/GISel/AArch64O0PreLegalizerCombiner.cpp

 //=== lib/CodeGen/GlobalISel/AArch64O0PreLegalizerCombiner.cpp ------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
virtual bool combine(GISelChangeObserver &Observer, MachineInstr &MI,
                        MachineIRBuilder &B) const override;
 };
 
 bool AArch64O0PreLegalizerCombinerInfo::combine(GISelChangeObserver &Observer,
                                                 MachineInstr &MI,
                                                 MachineIRBuilder &B) const {
   CombinerHelper Helper(Observer, B, KB, MDT);
   AArch64GenO0PreLegalizerCombinerHelper Generated(GeneratedRuleCfg, Helper);
+  auto &ST = static_cast<const AArch64Subtarget &>(B.getMF().getSubtarget());
 
   if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
   case TargetOpcode::G_CONCAT_VECTORS:
     return Helper.tryCombineConcatVectors(MI);
   case TargetOpcode::G_SHUFFLE_VECTOR:
     return Helper.tryCombineShuffleVector(MI);
   case TargetOpcode::G_MEMCPY_INLINE:
+    if (ST.hasMOPS())
+      return false;
     return Helper.tryEmitMemcpyInline(MI);
   case TargetOpcode::G_MEMCPY:
   case TargetOpcode::G_MEMMOVE:
   case TargetOpcode::G_MEMSET: {
+    if (ST.hasMOPS())
+      return false;
+
     // At -O0 set a maxlen of 32 to inline;
     unsigned MaxLen = 32;
     // Try to inline memcpy type calls if optimizations are enabled.
     if (Helper.tryCombineMemCpyFamily(MI, MaxLen))
       return true;
     if (Opc == TargetOpcode::G_MEMSET)
       return llvm::AArch64GISelUtils::tryEmitBZero(MI, B, EnableMinSize);
     return false;

llvm/test/CodeGen/AArch64/aarch64-mops-consecutive.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops       | FileCheck %s --check-prefix=CHECK-MOPS
+
+; Function Attrs: argmemonly mustprogress nofree nounwind willreturn writeonly
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+
+declare dso_local void @fn(i8*, i8*) local_unnamed_addr
+
+define hidden void @consecutive() local_unnamed_addr {
+; CHECK-MOPS-LABEL: consecutive:
+; CHECK-MOPS:       // %bb.0: // %entry
+; CHECK-MOPS-NEXT:    sub sp, sp, #80
+; CHECK-MOPS-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
+; CHECK-MOPS-NEXT:    .cfi_def_cfa_offset 80
+; CHECK-MOPS-NEXT:    .cfi_offset w30, -16
+; CHECK-MOPS-NEXT:    adrp x10, .LCPI0_0
+; CHECK-MOPS-NEXT:    adrp x11, .LCPI0_1
+; CHECK-MOPS-NEXT:    mov w8, #31
+; CHECK-MOPS-NEXT:    mov x9, sp
+; CHECK-MOPS-NEXT:    mov w12, #6424
+; CHECK-MOPS-NEXT:    setp [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    setm [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    sete [x9]!, x8!, xzr
+; CHECK-MOPS-NEXT:    movk w12, #6938, lsl #16
+; CHECK-MOPS-NEXT:    mov w13, #7452
+; CHECK-MOPS-NEXT:    ldr q0, [x10, :lo12:.LCPI0_0]
+; CHECK-MOPS-NEXT:    mov w8, #30
+; CHECK-MOPS-NEXT:    ldr d1, [x11, :lo12:.LCPI0_1]
+; CHECK-MOPS-NEXT:    add x0, sp, #32
+; CHECK-MOPS-NEXT:    mov x1, sp
+; CHECK-MOPS-NEXT:    str w12, [sp, #56]
+; CHECK-MOPS-NEXT:    strh w13, [sp, #60]
+; CHECK-MOPS-NEXT:    str q0, [sp, #32]
+; CHECK-MOPS-NEXT:    str d1, [sp, #48]
+; CHECK-MOPS-NEXT:    strb w8, [sp, #62]
+; CHECK-MOPS-NEXT:    bl fn
+; CHECK-MOPS-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
+; CHECK-MOPS-NEXT:    add sp, sp, #80
+; CHECK-MOPS-NEXT:    ret
+entry:
+  %buf_from = alloca [31 x i8], align 16
+  %buf_to = alloca [31 x i8], align 1
+  %0 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 0
+  %1 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_to, i64 0, i64 0
+  call void @llvm.memset.p0i8.i64(i8* noundef nonnull align 1 dereferenceable(31) %1, i8 0, i64 31, i1 false)
+  %2 = bitcast [31 x i8]* %buf_from to <16 x i8>*
+  store <16 x i8> <i8 0, i8 1, i8 2, i8 3, i8 4, i8 5, i8 6, i8 7, i8 8, i8 9, i8 10, i8 11, i8 12, i8 13, i8 14, i8 15>, <16 x i8>* %2, align 16
+  %arrayidx.16 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 16
+  %3 = bitcast i8* %arrayidx.16 to <8 x i8>*
+  store <8 x i8> <i8 16, i8 17, i8 18, i8 19, i8 20, i8 21, i8 22, i8 23>, <8 x i8>* %3, align 16
+  %arrayidx.24 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 24
+  store i8 24, i8* %arrayidx.24, align 8
+  %arrayidx.25 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 25
+  store i8 25, i8* %arrayidx.25, align 1
+  %arrayidx.26 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 26
+  store i8 26, i8* %arrayidx.26, align 2
+  %arrayidx.27 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 27
+  store i8 27, i8* %arrayidx.27, align 1
+  %arrayidx.28 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 28
+  store i8 28, i8* %arrayidx.28, align 4
+  %arrayidx.29 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 29
+  store i8 29, i8* %arrayidx.29, align 1
+  %arrayidx.30 = getelementptr inbounds [31 x i8], [31 x i8]* %buf_from, i64 0, i64 30
+  store i8 30, i8* %arrayidx.30, align 2
+  call void @fn(i8* nonnull %0, i8* nonnull %1)
+  ret void
+}

llvm/test/CodeGen/AArch64/aarch64-mops-mte.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops,+mte  | FileCheck %s --check-prefix=SDAG
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops,+mte  | FileCheck %s --check-prefix=GISel
+
+; Function Attrs: mustprogress nofree nosync nounwind willreturn writeonly
+declare i8* @llvm.aarch64.mops.memset.tag(i8*, i8, i64)
+
+define i8* @memset_tagged_0_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_0_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov x8, xzr
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_0_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x8!, x8
+; GISel-NEXT:    setgm [x0]!, x8!, x8
+; GISel-NEXT:    setge [x0]!, x8!, x8
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 0)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_1_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_1_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #1
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_1_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #1
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 1)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_10_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_10_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #10
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10000_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_10000_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10000
+; SDAG-NEXT:    setgp [x0]!, x8!, xzr
+; SDAG-NEXT:    setgm [x0]!, x8!, xzr
+; SDAG-NEXT:    setge [x0]!, x8!, xzr
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_10000_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x9, xzr
+; GISel-NEXT:    mov w8, #10000
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 10000)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_size_zeroval(i8* %dst, i64 %size) {
+; SDAG-LABEL: memset_tagged_size_zeroval:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    setgp [x0]!, x1!, xzr
+; SDAG-NEXT:    setgm [x0]!, x1!, xzr
+; SDAG-NEXT:    setge [x0]!, x1!, xzr
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_size_zeroval:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x1!, x8
+; GISel-NEXT:    setgm [x0]!, x1!, x8
+; GISel-NEXT:    setge [x0]!, x1!, x8
+; GISel-NEXT:    ret
+entry:
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 0, i64 %size)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_0(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_0:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov x8, xzr
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_0:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov x8, xzr
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 0)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_1(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_1:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #1
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_1:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #1
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 1)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_10:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_10:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #10
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_10000(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_10000:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    mov w8, #10000
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x8!, x2
+; SDAG-NEXT:    setgm [x0]!, x8!, x2
+; SDAG-NEXT:    setge [x0]!, x8!, x2
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_10000:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x9
+; GISel-NEXT:    mov w9, w2
+; GISel-NEXT:    mov w8, #10000
+; GISel-NEXT:    // kill: def $x8 killed $w8
+; GISel-NEXT:    setgp [x0]!, x8!, x9
+; GISel-NEXT:    setgm [x0]!, x8!, x9
+; GISel-NEXT:    setge [x0]!, x8!, x9
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 10000)
+  ret i8* %r
+}
+
+define i8* @memset_tagged_size(i8* %dst, i64 %size, i32 %value) {
+; SDAG-LABEL: memset_tagged_size:
+; SDAG:       // %bb.0: // %entry
+; SDAG-NEXT:    // kill: def $w2 killed $w2 def $x2
+; SDAG-NEXT:    setgp [x0]!, x1!, x2
+; SDAG-NEXT:    setgm [x0]!, x1!, x2
+; SDAG-NEXT:    setge [x0]!, x1!, x2
+; SDAG-NEXT:    ret
+;
+; GISel-LABEL: memset_tagged_size:
+; GISel:       // %bb.0: // %entry
+; GISel-NEXT:    // implicit-def: $x8
+; GISel-NEXT:    mov w8, w2
+; GISel-NEXT:    setgp [x0]!, x1!, x8
+; GISel-NEXT:    setgm [x0]!, x1!, x8
+; GISel-NEXT:    setge [x0]!, x1!, x8
+; GISel-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  %r = tail call i8* @llvm.aarch64.mops.memset.tag(i8* %dst, i8 %value_trunc, i64 %size)
+  ret i8* %r
+}

llvm/test/CodeGen/AArch64/aarch64-mops.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2                    | FileCheck %s --check-prefix=O2-SDAG-WITHOUT-MOPS
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O2 -mattr=+mops       | FileCheck %s --check-prefix=O2-SDAG-MOPS
+
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1                    | FileCheck %s --check-prefix=O0-GISel-WITHOUT-MOPS
+; RUN: llc %s -o - -mtriple=aarch64-arm-none-eabi -O0 -global-isel=1 -global-isel-abort=1 -mattr=+mops       | FileCheck %s --check-prefix=O0-GISel-MOPS
+
+; Function Attrs: argmemonly nofree nounwind willreturn writeonly
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+; Function Attrs: argmemonly nofree nounwind willreturn
+declare void @llvm.memmove.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i1 immarg)
+
+
+define void @memset_0_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
+  ret void
+}
+
+define void @memset_0_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
+  ret void
+}
+
+define void @memset_10_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 false)
+  ret void
+}
+
+define void @memset_10_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh wzr, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str xzr, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10, i1 true)
+  ret void
+}
+
+define void @memset_10000_zeroval(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 false)
+  ret void
+}
+
+define void @memset_10000_zeroval_volatile(i8* %dst) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x9, xzr
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 10000, i1 true)
+  ret void
+}
+
+define void @memset_size_zeroval(i8* %dst, i64 %size) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_zeroval:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_zeroval:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_zeroval:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 false)
+  ret void
+}
+
+define void @memset_size_zeroval_volatile(i8* %dst, i64 %size) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_zeroval_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_zeroval_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, xzr
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_zeroval_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x2, x1
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, wzr
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_zeroval_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 %size, i1 true)
+  ret void
+}
+
+
+define void @memset_0(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 false)
+  ret void
+}
+
+define void @memset_0_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    mov x9, x8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 0, i64 0, i1 true)
+  ret void
+}
+
+define void @memset_10(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-WITHOUT-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, w1
+; O0-GISel-WITHOUT-MOPS-NEXT:    and x8, x8, #0xff
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov x9, #72340172838076673
+; O0-GISel-WITHOUT-MOPS-NEXT:    mul x8, x8, x9
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    // kill: def $w8 killed $w8 killed $x8
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 false)
+  ret void
+}
+
+define void @memset_10_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, #72340172838076673
+; O2-SDAG-WITHOUT-MOPS-NEXT:    and x9, x1, #0xff
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mul x8, x9, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10, i1 true)
+  ret void
+}
+
+define void @memset_10000(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 false)
+  ret void
+}
+
+define void @memset_10000_volatile(i8* %dst, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_10000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #10000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_10000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10000
+; O2-SDAG-MOPS-NEXT:    // kill: def $w1 killed $w1 def $x1
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x8!, x1
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_10000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_10000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x9
+; O0-GISel-MOPS-NEXT:    mov w9, w1
+; O0-GISel-MOPS-NEXT:    mov w8, #10000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x8!, x9
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 10000, i1 true)
+  ret void
+}
+
+define void @memset_size(i8* %dst, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w2 killed $w2 def $x2
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x2, [sp, #8] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    add sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-MOPS-NEXT:    mov w8, w2
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 false)
+  ret void
+}
+
+define void @memset_size_volatile(i8* %dst, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memset_size_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x8, x1
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov x2, x8
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memset
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memset_size_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    // kill: def $w2 killed $w2 def $x2
+; O2-SDAG-MOPS-NEXT:    setp [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    setm [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    sete [x0]!, x1!, x2
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memset_size_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    sub sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #16] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 32
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x1, [sp, #8] // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w1, w2
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x2, [sp, #8] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memset
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp, #16] // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    add sp, sp, #32
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memset_size_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    // implicit-def: $x8
+; O0-GISel-MOPS-NEXT:    mov w8, w2
+; O0-GISel-MOPS-NEXT:    setp [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    setm [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    sete [x0]!, x1!, x8
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  %value_trunc = trunc i32 %value to i8
+  call void @llvm.memset.p0i8.i64(i8* align 1 %dst, i8 %value_trunc, i64 %size, i1 true)
+  ret void
+}
+
+
+define void @memcpy_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memcpy_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memcpy_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memcpy_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+define void @memcpy_1000(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_1000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_1000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_1000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
+  ret void
+}
+
+define void @memcpy_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_1000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_1000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_1000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_1000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
+  ret void
+}
+
+define void @memcpy_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_n:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_n:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_n:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
+  ret void
+}
+
+define void @memcpy_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_n_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memcpy
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_n_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_n_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memcpy
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_n_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 true)
+  ret void
+}
+
+
+define void @memcpy_inline_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memcpy_inline_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memcpy_inline_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memcpy_inline_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memcpy_inline_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memcpy_inline_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memcpy_inline_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memcpy_inline_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyfp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfm [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpyfe [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+
+define void @memmove_0(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_0:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_0:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_0:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 false)
+  ret void
+}
+
+define void @memmove_0_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_0_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_0_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_0_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_0_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov x8, xzr
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 0, i1 true)
+  ret void
+}
+
+define void @memmove_10(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_10:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_10:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x9, [x1]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldrh w8, [x1, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x9, [x0]
+; O0-GISel-WITHOUT-MOPS-NEXT:    strh w8, [x0, #8]
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_10:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 false)
+  ret void
+}
+
+define void @memmove_10_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_10_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x8, [x1]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldrh w9, [x1, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    strh w9, [x0, #8]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x8, [x0]
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_10_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #10
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_10_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #10
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_10_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #10
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 10, i1 true)
+  ret void
+}
+
+define void @memmove_1000(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_1000:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_1000:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_1000:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 false)
+  ret void
+}
+
+define void @memmove_1000_volatile(i8* %dst, i8* %src, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_1000_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    mov w2, #1000
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_1000_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    mov w8, #1000
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_1000_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-WITHOUT-MOPS-NEXT:    mov w2, w8
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_1000_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    mov w8, #1000
+; O0-GISel-MOPS-NEXT:    // kill: def $x8 killed $w8
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x8!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 1000, i1 true)
+  ret void
+}
+
+define void @memmove_n(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_n:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_n:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_n:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 false)
+  ret void
+}
+
+define void @memmove_n_volatile(i8* %dst, i8* %src, i64 %size, i32 %value) {
+; O2-SDAG-WITHOUT-MOPS-LABEL: memmove_n_volatile:
+; O2-SDAG-WITHOUT-MOPS:       // %bb.0: // %entry
+; O2-SDAG-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O2-SDAG-WITHOUT-MOPS-NEXT:    bl memmove
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O2-SDAG-WITHOUT-MOPS-NEXT:    ret
+;
+; O2-SDAG-MOPS-LABEL: memmove_n_volatile:
+; O2-SDAG-MOPS:       // %bb.0: // %entry
+; O2-SDAG-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O2-SDAG-MOPS-NEXT:    ret
+;
+; O0-GISel-WITHOUT-MOPS-LABEL: memmove_n_volatile:
+; O0-GISel-WITHOUT-MOPS:       // %bb.0: // %entry
+; O0-GISel-WITHOUT-MOPS-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_def_cfa_offset 16
+; O0-GISel-WITHOUT-MOPS-NEXT:    .cfi_offset w30, -16
+; O0-GISel-WITHOUT-MOPS-NEXT:    bl memmove
+; O0-GISel-WITHOUT-MOPS-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; O0-GISel-WITHOUT-MOPS-NEXT:    ret
+;
+; O0-GISel-MOPS-LABEL: memmove_n_volatile:
+; O0-GISel-MOPS:       // %bb.0: // %entry
+; O0-GISel-MOPS-NEXT:    cpyp [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpym [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    cpye [x0]!, [x1]!, x2!
+; O0-GISel-MOPS-NEXT:    ret
+entry:
+  call void @llvm.memmove.p0i8.p0i8.i64(i8* align 1 %dst, i8* align 1 %src, i64 %size, i1 true)
+  ret void
+}