[AArch64] Add a target feature for AArch64StorePairSuppress

[dmgreen]

Summary

The AArch64StorePairSuppress pass prevents the creation of STP under some heuristics. Unfortunately it often prevents the creation of STP in cases where it is obviously beneficial, and it doesn't match my understanding of scheduling/cpu pipelining to prevent the creation of STP. From some benchmarking, even on an in-order cpu where the scheduling is most important I don't see it giving better results. In general the lower instruction count for STP would be expected to give a slightly better cycle count.

As the pass specifically mentions the cyclone cpu, this patch adds a target feature for FeatureStorePairSuppress, enabled for all the non-Arm cpus. This has the effect of disabling it for all Arm cpus.

[Allen]

llvm/lib/Target/AArch64/AArch64.td

1242

TSV110 target is also expected to suppress pair stores

[dmgreen]

llvm/lib/Target/AArch64/AArch64.td

1242

OK good to know, I can change it. I am a little surprised it expected to be beneficial though. Do you have an example where it helped? Perhaps I am just running the wrong code to show where it is useful..

[Allen]

llvm/lib/Target/AArch64/AArch64.td

1242

I don't have small case, but according the history record, this will benefical to 649.fotonik3d_s in spec2006.

[dmgreen]

llvm/lib/Target/AArch64/AArch64.td

1242

Hmm. 649.fotonik3d_s seems to be a fortran benchmark in spec2017. I would not usually measure those. I will see if I can measure some numbers.

[dmgreen]

OK - I've managed to run the fortran benchmarks on a range of micro-architectures, and only saw the same behaviour as before. The STP being less instructions gives slightly less time for the benchmark overall, and I don't see a reason to keep the store-pair-suppress option for the cpus I tried.

I've changed this to keep the feature for non-Arm cpus. I would be surprised if any of them really benefit from it, but this makes it Arm-only (and cpu=generic).

[Allen]

In D134646#3832896, @dmgreen wrote:

OK - I've managed to run the fortran benchmarks on a range of micro-architectures, and only saw the same behaviour as before. The STP being less instructions gives slightly less time for the benchmark overall, and I don't see a reason to keep the store-pair-suppress option for the cpus I tried.

I've changed this to keep the feature for non-Arm cpus. I would be surprised if any of them really benefit from it, but this makes it Arm-only (and cpu=generic).

Thanks for the check. I also recheck the performance and find this change doesn't affect performance in our newest downstream branch, so LGTM for tsv110 now.

[samtebbs]

Nice, LGTM

llvm/lib/Target/AArch64/AArch64.td

 def FeatureFuseAddSub2RegAndConstOne : SubtargetFeature<
    "fuse-addsub-2reg-const1", "HasFuseAddSub2RegAndConstOne", "true",
    "CPU fuses (a + b + 1) and (a - b - 1)">;
 
 def FeatureDisableLatencySchedHeuristic : SubtargetFeature<
     "disable-latency-sched-heuristic", "DisableLatencySchedHeuristic", "true",
     "Disable latency scheduling heuristic">;
 
+def FeatureStorePairSuppress : SubtargetFeature<
+    "store-pair-suppress", "EnableStorePairSuppress", "true",
+    "Enable Store Pair Suppression heuristics">;
+
 def FeatureForce32BitJumpTables
    : SubtargetFeature<"force-32bit-jump-tables", "Force32BitJumpTables", "true",
                       "Force jump table entries to be 32-bits wide except at MinSize">;
 
 def FeatureRCPC : SubtargetFeature<"rcpc", "HasRCPC", "true",
                                    "Enable support for RCPC extension (FEAT_LRCPC)">;
 
 def FeatureUseRSqrt : SubtargetFeature<
def TuneX3 : SubtargetFeature<"cortex-x3", "ARMProcFamily", "CortexX3",
                                FeatureEnableSelectOptimize,
                                FeaturePredictableSelectIsExpensive]>;
 
 def TuneA64FX : SubtargetFeature<"a64fx", "ARMProcFamily", "A64FX",
                                  "Fujitsu A64FX processors", [
                                  FeaturePostRAScheduler,
                                  FeatureAggressiveFMA,
                                  FeatureArithmeticBccFusion,
-                                 FeaturePredictableSelectIsExpensive
-                                 ]>;
+                                 FeatureStorePairSuppress,
+                                 FeaturePredictableSelectIsExpensive]>;
 
 def TuneCarmel : SubtargetFeature<"carmel", "ARMProcFamily", "Carmel",
                                   "Nvidia Carmel processors">;
 
 // Note that cyclone does not fuse AES instructions, but newer apple chips do
 // perform the fusion and cyclone is used by default when targetting apple OSes.
 def TuneAppleA7  : SubtargetFeature<"apple-a7", "ARMProcFamily", "AppleA7",
                                     "Apple A7 (the CPU formerly known as Cyclone)", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAES, FeatureFuseCryptoEOR,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
                                     FeatureZCZeroing,
-                                    FeatureZCZeroingFPWorkaround]
-                                    >;
+                                    FeatureZCZeroingFPWorkaround]>;
 
 def TuneAppleA10 : SubtargetFeature<"apple-a10", "ARMProcFamily", "AppleA10",
                                     "Apple A10", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAES,
                                     FeatureFuseCryptoEOR,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing]
-                                    >;
+                                    FeatureZCZeroing]>;
 
 def TuneAppleA11 : SubtargetFeature<"apple-a11", "ARMProcFamily", "AppleA11",
                                     "Apple A11", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAES,
                                     FeatureFuseCryptoEOR,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing]
-                                    >;
+                                    FeatureZCZeroing]>;
 
 def TuneAppleA12 : SubtargetFeature<"apple-a12", "ARMProcFamily", "AppleA12",
                                     "Apple A12", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAES,
                                     FeatureFuseCryptoEOR,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing]
-                                    >;
+                                    FeatureZCZeroing]>;
 
 def TuneAppleA13 : SubtargetFeature<"apple-a13", "ARMProcFamily", "AppleA13",
                                     "Apple A13", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAES,
                                     FeatureFuseCryptoEOR,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing]
-                                    >;
+                                    FeatureZCZeroing]>;
 
 def TuneAppleA14 : SubtargetFeature<"apple-a14", "ARMProcFamily", "AppleA14",
                                     "Apple A14", [
                                     FeatureAggressiveFMA,
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAddress,
                                     FeatureFuseAES,
                                     FeatureFuseArithmeticLogic,
                                     FeatureFuseCCSelect,
                                     FeatureFuseCryptoEOR,
                                     FeatureFuseAdrpAdd,
                                     FeatureFuseLiterals,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
                                     FeatureZCZeroing]>;
 
 def TuneAppleA15 : SubtargetFeature<"apple-a15", "ARMProcFamily", "AppleA15",
                                     "Apple A15", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAddress,
                                     FeatureFuseAES,
                                     FeatureFuseArithmeticLogic,
                                     FeatureFuseCCSelect,
                                     FeatureFuseCryptoEOR,
                                     FeatureFuseLiterals,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing
-                                    ]>;
+                                    FeatureZCZeroing]>;
 
 def TuneAppleA16 : SubtargetFeature<"apple-a16", "ARMProcFamily", "AppleA16",
                                     "Apple A16", [
                                     FeatureAlternateSExtLoadCVTF32Pattern,
                                     FeatureArithmeticBccFusion,
                                     FeatureArithmeticCbzFusion,
                                     FeatureDisableLatencySchedHeuristic,
                                     FeatureFuseAddress,
                                     FeatureFuseAES,
                                     FeatureFuseArithmeticLogic,
                                     FeatureFuseCCSelect,
                                     FeatureFuseCryptoEOR,
                                     FeatureFuseLiterals,
+                                    FeatureStorePairSuppress,
                                     FeatureZCRegMove,
-                                    FeatureZCZeroing
-                                    ]>;
+                                    FeatureZCZeroing]>;
 
 def TuneExynosM3 : SubtargetFeature<"exynosm3", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M3 processors",
                                     [FeatureExynosCheapAsMoveHandling,
                                      FeatureForce32BitJumpTables,
                                      FeatureFuseAddress,
                                      FeatureFuseAES,
                                      FeatureFuseCCSelect,
                                      FeatureFuseAdrpAdd,
                                      FeatureFuseLiterals,
+                                     FeatureStorePairSuppress,
                                      FeatureAddrLSLFast,
                                      FeatureALULSLFast,
                                      FeaturePostRAScheduler,
                                      FeaturePredictableSelectIsExpensive]>;
 
 // Re-uses some scheduling and tunings from the ExynosM3 proc family.
 def TuneExynosM4 : SubtargetFeature<"exynosm4", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M4 processors",
                                     [FeatureArithmeticBccFusion,
                                      FeatureArithmeticCbzFusion,
                                      FeatureExynosCheapAsMoveHandling,
                                      FeatureForce32BitJumpTables,
                                      FeatureFuseAddress,
                                      FeatureFuseAES,
                                      FeatureFuseArithmeticLogic,
                                      FeatureFuseCCSelect,
                                      FeatureFuseAdrpAdd,
                                      FeatureFuseLiterals,
+                                     FeatureStorePairSuppress,
                                      FeatureAddrLSLFast,
                                      FeatureALULSLFast,
                                      FeaturePostRAScheduler,
                                      FeatureZCZeroing]>;
 
 def TuneKryo    : SubtargetFeature<"kryo", "ARMProcFamily", "Kryo",
                                    "Qualcomm Kryo processors", [
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
                                    FeatureAddrLSLFast,
-                                   FeatureALULSLFast]
-                                   >;
+                                   FeatureALULSLFast,
+                                   FeatureStorePairSuppress]>;
 
 def TuneFalkor  : SubtargetFeature<"falkor", "ARMProcFamily", "Falkor",
                                    "Qualcomm Falkor processors", [
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
+                                   FeatureStorePairSuppress,
                                    FeatureAddrLSLFast,
                                    FeatureALULSLFast,
-                                   FeatureSlowSTRQro
-                                   ]>;
+                                   FeatureSlowSTRQro]>;
 
 def TuneNeoverseE1 : SubtargetFeature<"neoversee1", "ARMProcFamily", "NeoverseE1",
                                       "Neoverse E1 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
                                       FeaturePostRAScheduler]>;
 
 def TuneNeoverseN1 : SubtargetFeature<"neoversen1", "ARMProcFamily", "NeoverseN1",
def TuneNeoverseV2 : SubtargetFeature<"neoversev2", "ARMProcFamily", "NeoverseV2",
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;
 
 def TuneSaphira  : SubtargetFeature<"saphira", "ARMProcFamily", "Saphira",
                                    "Qualcomm Saphira processors", [
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
+                                   FeatureStorePairSuppress,
                                    FeatureAddrLSLFast,
                                    FeatureALULSLFast]>;
 
 def TuneThunderX2T99  : SubtargetFeature<"thunderx2t99", "ARMProcFamily", "ThunderX2T99",
                                          "Cavium ThunderX2 processors", [
                                           FeatureAggressiveFMA,
                                           FeatureArithmeticBccFusion,
                                           FeaturePostRAScheduler,
+                                          FeatureStorePairSuppress,
                                           FeaturePredictableSelectIsExpensive]>;
 
 def TuneThunderX3T110  : SubtargetFeature<"thunderx3t110", "ARMProcFamily",
                                           "ThunderX3T110",
                                           "Marvell ThunderX3 processors", [
                                            FeatureAggressiveFMA,
                                            FeatureArithmeticBccFusion,
                                            FeaturePostRAScheduler,
                                            FeaturePredictableSelectIsExpensive,
                                            FeatureBalanceFPOps,
+                                           FeatureStorePairSuppress,
                                            FeatureStrictAlign]>;
 
 def TuneThunderX : SubtargetFeature<"thunderx", "ARMProcFamily", "ThunderX",
                                     "Cavium ThunderX processors", [
                                     FeaturePostRAScheduler,
+                                    FeatureStorePairSuppress,
                                     FeaturePredictableSelectIsExpensive]>;
 
 def TuneThunderXT88 : SubtargetFeature<"thunderxt88", "ARMProcFamily",
                                        "ThunderXT88",
                                        "Cavium ThunderX processors", [
                                        FeaturePostRAScheduler,
+                                       FeatureStorePairSuppress,
                                        FeaturePredictableSelectIsExpensive]>;
 
 def TuneThunderXT81 : SubtargetFeature<"thunderxt81", "ARMProcFamily",
                                        "ThunderXT81",
                                        "Cavium ThunderX processors", [
                                        FeaturePostRAScheduler,
+                                       FeatureStorePairSuppress,
                                        FeaturePredictableSelectIsExpensive]>;
 
 def TuneThunderXT83 : SubtargetFeature<"thunderxt83", "ARMProcFamily",
                                        "ThunderXT83",
                                        "Cavium ThunderX processors", [
                                        FeaturePostRAScheduler,
+                                       FeatureStorePairSuppress,
                                        FeaturePredictableSelectIsExpensive]>;
 
 def TuneTSV110 : SubtargetFeature<"tsv110", "ARMProcFamily", "TSV110",
                                   "HiSilicon TS-V110 processors", [
                                   FeatureFuseAES,
+                                  FeatureStorePairSuppress,
                                   FeaturePostRAScheduler]>;
 
 def TuneAmpere1 : SubtargetFeature<"ampere1", "ARMProcFamily", "Ampere1",
                                    "Ampere Computing Ampere-1 processors", [
                                    FeaturePostRAScheduler,
                                    FeatureFuseAES,
                                    FeatureAddrLSLFast,
                                    FeatureALULSLFast,
                                    FeatureAggressiveFMA,
                                    FeatureArithmeticBccFusion,
                                    FeatureCmpBccFusion,
                                    FeatureFuseAddress,
                                    FeatureFuseLiterals,
-                 FeatureLdpAlignedOnly,
+                                   FeatureStorePairSuppress,
+                                   FeatureLdpAlignedOnly,
                                    FeatureStpAlignedOnly]>;
 
 def TuneAmpere1A : SubtargetFeature<"ampere1a", "ARMProcFamily", "Ampere1A",
                                     "Ampere Computing Ampere-1A processors", [
                                     FeaturePostRAScheduler,
                                     FeatureFuseAES,
                                     FeatureAddrLSLFast,
                                     FeatureALULSLFast,
                                     FeatureAggressiveFMA,
                                     FeatureArithmeticBccFusion,
                                     FeatureCmpBccFusion,
                                     FeatureFuseAddress,
                                     FeatureFuseLiterals,
                                     FeatureFuseLiterals,
+                                    FeatureStorePairSuppress,
                                     FeatureLdpAlignedOnly,
                                     FeatureStpAlignedOnly]>;
 
 def ProcessorFeatures {
   list<SubtargetFeature> A53  = [HasV8_0aOps, FeatureCRC, FeatureCrypto,
                                  FeatureFPARMv8, FeatureNEON, FeaturePerfMon];
   list<SubtargetFeature> A55  = [HasV8_2aOps, FeatureCrypto, FeatureFPARMv8,
                                  FeatureNEON, FeatureFullFP16, FeatureDotProd,

llvm/lib/Target/AArch64/AArch64StorePairSuppress.cpp

 //===--- AArch64StorePairSuppress.cpp --- Suppress store pair formation ---===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This pass identifies floating point stores that should not be combined into
 // store pairs. Later we may do the same for floating point loads.
 // ===---------------------------------------------------------------------===//
 
 #include "AArch64InstrInfo.h"
+#include "AArch64Subtarget.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineTraceMetrics.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetSchedule.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
case AArch64::STURDi:
     return true;
   }
 }
 
 bool AArch64StorePairSuppress::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()) || MF.getFunction().hasOptSize())
     return false;
 
-  const TargetSubtargetInfo &ST = MF.getSubtarget();
+  const AArch64Subtarget &ST = MF.getSubtarget<AArch64Subtarget>();
+  if (!ST.enableStorePairSuppress())
+    return false;
+
   TII = static_cast<const AArch64InstrInfo *>(ST.getInstrInfo());
   TRI = ST.getRegisterInfo();
   MRI = &MF.getRegInfo();
   SchedModel.init(&ST);
   Traces = &getAnalysis<MachineTraceMetrics>();
   MinInstr = nullptr;
 
   LLVM_DEBUG(dbgs() << "*** " << getPassName() << ": " << MF.getName() << '\n');

llvm/test/CodeGen/AArch64/aarch64-a57-fp-load-balancing.ll

-; RUN: llc < %s -mcpu=cortex-a57 -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-BALFP --check-prefix CHECK-EVEN
-; RUN: llc < %s -mcpu=cortex-a57 -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-BALFP --check-prefix CHECK-ODD
-; RUN: llc < %s -mcpu=cortex-a53 -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-A53 --check-prefix CHECK-EVEN
-; RUN: llc < %s -mcpu=cortex-a53 -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-A53 --check-prefix CHECK-ODD
+; RUN: llc < %s -mcpu=cortex-a57 -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-EVEN
+; RUN: llc < %s -mcpu=cortex-a57 -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-ODD
+; RUN: llc < %s -mcpu=cortex-a53 -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-EVEN
+; RUN: llc < %s -mcpu=cortex-a53 -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-ODD
 
 ; The following tests use the balance-fp-ops feature, and should be independent of
 ; the target cpu.
 
-; RUN: llc < %s -mtriple=aarch64-linux-gnueabi -mattr=+balance-fp-ops -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-EVEN --check-prefix CHECK-BALFP
-; RUN: llc < %s -mtriple=aarch64-linux-gnueabi -mattr=+balance-fp-ops  -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-ODD --check-prefix CHECK-BALFP
+; RUN: llc < %s -mtriple=aarch64-linux-gnueabi -mattr=+balance-fp-ops -aarch64-a57-fp-load-balancing-override=1 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-EVEN
+; RUN: llc < %s -mtriple=aarch64-linux-gnueabi -mattr=+balance-fp-ops  -aarch64-a57-fp-load-balancing-override=2 -aarch64-a57-fp-load-balancing-force-all -enable-misched=false -enable-post-misched=false | FileCheck %s --check-prefix CHECK --check-prefix CHECK-ODD
 
 ; Test the AArch64A57FPLoadBalancing pass. This pass relies heavily on register allocation, so
 ; our test strategy is to:
 ;   * Force the pass to always perform register swapping even if the dest register is of the
 ;     correct color already (-force-all)
 ;   * Force the pass to ignore all hints it obtained from regalloc (-deterministic-balance),
 ;     and run it twice, once where it always hints odd, and once where it always hints even.
 ;

 ; CHECK-EVEN: fmul [[y:d[0-9]*[13579]]]
 ; CHECK-ODD: fmadd [[x:d[0-9]*[13579]]]
 ; CHECK-ODD: fmul [[y:d[0-9]*[02468]]]
 ; CHECK: fmadd [[x]]
 ; CHECK: fmadd [[y]]
 ; CHECK: fmsub [[x]]
 ; CHECK: fmadd [[y]]
 ; CHECK: fmadd [[x]]
-; CHECK-BALFP: stp [[x]], [[y]]
-; CHECK-A53-DAG: str [[x]]
-; CHECK-A53-DAG: str [[y]]
+; CHECK: stp [[x]], [[y]]
 
 define void @f2(ptr nocapture readonly %p, ptr nocapture %q) #0 {
 entry:
   %0 = load double, ptr %p, align 8
   %arrayidx1 = getelementptr inbounds double, ptr %p, i64 1
   %1 = load double, ptr %arrayidx1, align 8
   %arrayidx2 = getelementptr inbounds double, ptr %p, i64 2
   %2 = load double, ptr %arrayidx2, align 8

 ; CHECK-EVEN: fmul [[y:s[0-9]*[13579]]]
 ; CHECK-ODD: fmadd [[x:s[0-9]*[13579]]]
 ; CHECK-ODD: fmul [[y:s[0-9]*[02468]]]
 ; CHECK: fmadd [[x]]
 ; CHECK: fmadd [[y]]
 ; CHECK: fmsub [[x]]
 ; CHECK: fmadd [[y]]
 ; CHECK: fmadd [[x]]
-; CHECK-BALFP: stp [[x]], [[y]]
-; CHECK-A53-DAG: str [[x]]
-; CHECK-A53-DAG: str [[y]]
+; CHECK: stp [[x]], [[y]]
 
 define void @f4(ptr nocapture readonly %p, ptr nocapture %q) #0 {
 entry:
   %0 = load float, ptr %p, align 4
   %arrayidx1 = getelementptr inbounds float, ptr %p, i64 1
   %1 = load float, ptr %arrayidx1, align 4
   %arrayidx2 = getelementptr inbounds float, ptr %p, i64 2
   %2 = load float, ptr %arrayidx2, align 4

llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll

 ; CHECK-LABEL: caller_in_block:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    bl return_in_block
 ; CHECK-NEXT:    adrp x8, in_block_store
 ; CHECK-NEXT:    add x8, x8, :lo12:in_block_store
-; CHECK-NEXT:    str d0, [x8]
-; CHECK-NEXT:    str d1, [x8, #8]
-; CHECK-NEXT:    str d2, [x8, #16]
-; CHECK-NEXT:    str d3, [x8, #24]
-; CHECK-NEXT:    str d4, [x8, #32]
-; CHECK-NEXT:    str d5, [x8, #40]
+; CHECK-NEXT:    stp d0, d1, [x8]
+; CHECK-NEXT:    stp d2, d3, [x8, #16]
+; CHECK-NEXT:    stp d4, d5, [x8, #32]
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %1 = call %T_IN_BLOCK @return_in_block()
   store %T_IN_BLOCK %1, ptr @in_block_store
   ret void
 }
 
 define void @callee_in_block(%T_IN_BLOCK %a) {
 ; CHECK-LABEL: callee_in_block:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, in_block_store
 ; CHECK-NEXT:    add x8, x8, :lo12:in_block_store
-; CHECK-NEXT:    str d5, [x8, #40]
-; CHECK-NEXT:    str d4, [x8, #32]
-; CHECK-NEXT:    str d3, [x8, #24]
-; CHECK-NEXT:    str d2, [x8, #16]
-; CHECK-NEXT:    str d1, [x8, #8]
-; CHECK-NEXT:    str d0, [x8]
+; CHECK-NEXT:    stp d4, d5, [x8, #32]
+; CHECK-NEXT:    stp d2, d3, [x8, #16]
+; CHECK-NEXT:    stp d0, d1, [x8]
 ; CHECK-NEXT:    ret
   store %T_IN_BLOCK %a, ptr @in_block_store
   ret void
 }
 
 define void @argument_in_block() {
 ; CHECK-LABEL: argument_in_block:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/arm64-windows-calls.ll

 ; CHECK-NEXT:    .seh_save_reg_x x19, 16
 ; CHECK-NEXT:    str x30, [sp, #8] // 8-byte Folded Spill
 ; CHECK-NEXT:    .seh_save_reg x30, 8
 ; CHECK-NEXT:    .seh_endprologue
 ; CHECK-NEXT:    adrp x19, Pod
 ; CHECK-NEXT:    add x19, x19, :lo12:Pod
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    bl copy_pod
-; CHECK-NEXT:    str d0, [x19]
-; CHECK-NEXT:    str d1, [x19, #8]
+; CHECK-NEXT:    stp d0, d1, [x19]
 ; CHECK-NEXT:    .seh_startepilogue
 ; CHECK-NEXT:    ldr x30, [sp, #8] // 8-byte Folded Reload
 ; CHECK-NEXT:    .seh_save_reg x30, 8
 ; CHECK-NEXT:    ldr x19, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    .seh_save_reg_x x19, 16
 ; CHECK-NEXT:    .seh_endepilogue
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:    .seh_endfunclet

llvm/test/CodeGen/AArch64/ldrpre-ldr-merge.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 3
 # RUN: llc -o - %s -mtriple=aarch64 -mcpu=cortex-a55 -lsr-preferred-addressing-mode=preindexed -stop-after=aarch64-ldst-opt | FileCheck %s
 
 ---
 name:            1-ldrwpre-ldrwui-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$w0' }
   - { reg: '$w1' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $w0, $w1, $x1
     ; CHECK-LABEL: name: 1-ldrwpre-ldrwui-merge
     ; CHECK: liveins: $w0, $w1, $x1
-    ; CHECK: early-clobber $x1, renamable $w0, renamable $w2 = LDPWpre renamable $x1, 5 :: (load (s32))
-    ; CHECK: STPWi renamable $w0, renamable $w2, renamable $x1, 0 :: (store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $w0, renamable $w2 = LDPWpre renamable $x1, 5 :: (load (s32))
+    ; CHECK-NEXT: STPWi renamable $w0, renamable $w2, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $w0 = LDRWpre killed renamable $x1, 20 :: (load (s32))
     renamable $w2 = LDRWui renamable $x1, 1 :: (load (s32))
     STRWui killed renamable $w0, renamable $x1, 0 :: (store (s32))
     STRWui killed renamable $w2, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
 
 ---
 name:            2-ldrxpre-ldrxui-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$x2' }
   - { reg: '$x3' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x2, $x3, $x1
     ; CHECK-LABEL: name: 2-ldrxpre-ldrxui-merge
     ; CHECK: liveins: $x1, $x2, $x3
-    ; CHECK: early-clobber $x1, renamable $x2, renamable $x3 = LDPXpre renamable $x1, 3 :: (load (s64))
-    ; CHECK: STPXi renamable $x2, renamable $x3, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $x2, renamable $x3 = LDPXpre renamable $x1, 3 :: (load (s64))
+    ; CHECK-NEXT: STPXi renamable $x2, renamable $x3, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $x2 = LDRXpre killed renamable $x1, 24 :: (load (s64))
     renamable $x3 = LDRXui renamable $x1, 1 :: (load (s64))
     STRXui killed renamable $x2, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x3, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            3-ldrspre-ldrsui-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$s0' }
   - { reg: '$s1' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $s0, $s1, $x1
     ; CHECK-LABEL: name: 3-ldrspre-ldrsui-merge
     ; CHECK: liveins: $s0, $s1, $x1
-    ; CHECK: early-clobber $x1, renamable $s0, renamable $s1 = LDPSpre renamable $x1, 3 :: (load (s32))
-    ; CHECK: STRSui renamable $s0, renamable $x1, 0 :: (store (s32))
-    ; CHECK: STRSui renamable $s1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $s0, renamable $s1 = LDPSpre renamable $x1, 3 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 12 :: (load (s32))
     renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
 
 ---
 name:            4-ldrqdre-ldrdui-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$d0' }
   - { reg: '$d1' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $d0, $d1, $x1
     ; CHECK-LABEL: name: 4-ldrqdre-ldrdui-merge
     ; CHECK: liveins: $d0, $d1, $x1
-    ; CHECK: early-clobber $x1, renamable $d0, renamable $d1 = LDPDpre renamable $x1, 16 :: (load (s64))
-    ; CHECK: STRDui renamable $d0, renamable $x1, 0 :: (store (s64))
-    ; CHECK: STRDui renamable $d1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $d0, renamable $d1 = LDPDpre renamable $x1, 16 :: (load (s64))
+    ; CHECK-NEXT: STPDi renamable $d0, renamable $d1, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $d0 = LDRDpre killed renamable $x1, 128 :: (load (s64))
     renamable $d1 = LDRDui renamable $x1, 1 :: (load (s64))
     STRDui killed renamable $d0, renamable $x1, 0 :: (store (s64))
     STRDui killed renamable $d1, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 5-ldrqpre-ldrqui-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 3 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 3 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 48 :: (load (s128))
     renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 6-ldrqui-ldrqpre-no-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 48, implicit $w1 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 48, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 48 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 7-ldrqpre-ldrqui-max-offset-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 15 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 15 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 240 :: (load (s128))
     renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 8-ldrqpre-ldrqui-min-offset-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, -16 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, -16 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, -256 :: (load (s128))
     renamable $q1 = LDRQui renamable $x1, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $s0, $s1, $x0, $x1
     ; CHECK-LABEL: name: 9-ldrspre-ldrsui-mod-base-reg-no-merge
     ; CHECK: liveins: $s0, $s1, $x0, $x1
-    ; CHECK: dead early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $x1 = LDRXui renamable $x0, 1 :: (load (s64))
-    ; CHECK: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
-    ; CHECK: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: dead early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $x1 = LDRXui renamable $x0, 1 :: (load (s64))
+    ; CHECK-NEXT: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 12 :: (load (s32))
     renamable $x1 = LDRXui renamable $x0, 1 :: (load (s64))
     renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $s0, $s1, $x0, $x1
     ; CHECK-LABEL: name: 10-ldrspre-ldrsui-used-base-reg-no-merge
     ; CHECK: liveins: $s0, $s1, $x0, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $x0 = LDRXui renamable $x1, 1 :: (load (s64))
-    ; CHECK: STRXui renamable $x0, renamable $x0, 1 :: (store (s64))
-    ; CHECK: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
-    ; CHECK: STRSui renamable $s0, renamable $x1, 0 :: (store (s32))
-    ; CHECK: STRSui renamable $s1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $x0 = LDRXui renamable $x1, 1 :: (load (s64))
+    ; CHECK-NEXT: STRXui renamable $x0, renamable $x0, 1 :: (store (s64))
+    ; CHECK-NEXT: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 12 :: (load (s32))
     renamable $x0 = LDRXui renamable $x1, 1 :: (load (s64))
     STRXui killed renamable $x0, renamable $x0, 1 :: (store (s64))
     renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 11-ldrqpre-ldrqpre-no-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 48, implicit $w1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, dead renamable $q1 = LDRQpre renamable $x1, 1, implicit $w1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, dead renamable $q1 = LDRQpre renamable $x1, 12, implicit $w1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
-    ; CHECK: early-clobber renamable $x1, renamable $q1 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 48, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $q1 = LDRQpre renamable $x1, 1, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $q1 = LDRQpre renamable $x1, 12, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q1 = LDRQpre renamable $x1, 16, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 48 :: (load (s128))
     early-clobber renamable $x1, renamable $q1 = LDRQpre killed renamable $x1, 1  :: (load (s128))
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 16 :: (load (s128))
     early-clobber renamable $x1, renamable $q1 = LDRQpre killed renamable $x1, 12  :: (load (s128))
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 16 :: (load (s128))
     early-clobber renamable $x1, renamable $q1 = LDRQpre killed renamable $x1, 16  :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
body:             |
   bb.0:
     liveins: $s0, $s1, $x1
 
     ; The offset of the second load is not equal to the
     ; size of the destination register,  and hence can’t be merged.
 
     ; CHECK-LABEL: name: 12-ldrspre-ldrsui-no-merge
     ; CHECK: liveins: $s0, $s1, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $s1 = LDRSui renamable $x1, 2 :: (load (s32))
-    ; CHECK: STRSui renamable $s0, renamable $x1, 0 :: (store (s32))
-    ; CHECK: STRSui renamable $s1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 12, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $s1 = LDRSui renamable $x1, 2 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 12 :: (load (s32))
     renamable $s1 = LDRSui renamable $x1, 2 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $d1, $x1
     ; CHECK-LABEL: name: 13-ldrqpre-ldrdui-no-merge
     ; CHECK: liveins: $d1, $q0, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
-    ; CHECK: renamable $d1 = LDRDui renamable $x1, 1 :: (load (s64))
-    ; CHECK: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
-    ; CHECK: STRDui renamable $d1, renamable $x1, 1 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: renamable $d1 = LDRDui renamable $x1, 1 :: (load (s64))
+    ; CHECK-NEXT: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: STRDui renamable $d1, renamable $x1, 1 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     renamable $d1 = LDRDui renamable $x1, 1 :: (load (s64))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRDui killed renamable $d1, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 14-ldrqpre-strqui-no-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
-    ; CHECK: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     RET undef $lr
 ...
 
 
 ---
 name:            15-ldrqpre-ldrqui-same-dst-reg-no-merge
 alignment:       4
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$q0' }
   - { reg: '$q1' }
 frameInfo:
   maxAlignment:    1
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $x1
     ; CHECK-LABEL: name: 15-ldrqpre-ldrqui-same-dst-reg-no-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
-    ; CHECK: renamable $q0 = LDRQui renamable $x1, 1 :: (load (s128))
-    ; CHECK: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: renamable $q0 = LDRQui renamable $x1, 1 :: (load (s128))
+    ; CHECK-NEXT: STRQui renamable $q0, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     renamable $q0 = LDRQui renamable $x1, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     RET undef $lr
 ...
 
 
 ---
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1, $x2
     ; CHECK-LABEL: name: 16-ldrqpre-ldrqui-diff-base-reg-no-merge
     ; CHECK: liveins: $q0, $q1, $x1, $x2
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
-    ; CHECK: renamable $q1 = LDRQui renamable $x2, 1 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: renamable $q1 = LDRQui renamable $x2, 1 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     renamable $q1 = LDRQui renamable $x2, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 17-ldrqpre-ldurqi-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 2 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $q0, renamable $q1 = LDPQpre renamable $x1, 2 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     renamable $q1 = LDURQi renamable $x1, 16 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
frameInfo:
   maxCallFrameSize: 0
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $q0, $q1, $x1
     ; CHECK-LABEL: name: 18-ldrqpre-ldurqi-no-merge
     ; CHECK: liveins: $q0, $q1, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
-    ; CHECK: renamable $q1 = LDURQi renamable $x1, 1 :: (load (s128))
-    ; CHECK: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $q0 = LDRQpre renamable $x1, 32, implicit $w1 :: (load (s128))
+    ; CHECK-NEXT: renamable $q1 = LDURQi renamable $x1, 1 :: (load (s128))
+    ; CHECK-NEXT: STPQi renamable $q0, renamable $q1, renamable $x1, 0 :: (store (s128))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $q0 = LDRQpre killed renamable $x1, 32 :: (load (s128))
     renamable $q1 = LDURQi renamable $x1, 1 :: (load (s128))
     STRQui killed renamable $q0, renamable $x1, 0 :: (store (s128))
     STRQui killed renamable $q1, renamable $x1, 1 :: (store (s128))
     RET undef $lr
 ...
 
 
 ---
 name:            19-ldrspre-ldrsui-max-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$s0' }
   - { reg: '$s1' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $s0, $s1, $x1
     ; CHECK-LABEL: name: 19-ldrspre-ldrsui-max-merge
     ; CHECK: liveins: $s0, $s1, $x1
-    ; CHECK: early-clobber $x1, renamable $s0, renamable $s1 = LDPSpre renamable $x1, 63 :: (load (s32))
-    ; CHECK: STRSui renamable $s0, renamable $x1, 0 :: (store (s32))
-    ; CHECK: STRSui renamable $s1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $s0, renamable $s1 = LDPSpre renamable $x1, 63 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 252 :: (load (s32))
     renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
 
 ---
 name:            20-ldrspre-ldrsui-unaligned-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x1' }
   - { reg: '$s0' }
   - { reg: '$s1' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $s0, $s1, $x1
     ; CHECK-LABEL: name: 20-ldrspre-ldrsui-unaligned-no-merge
     ; CHECK: liveins: $s0, $s1, $x1
-    ; CHECK: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 251, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
-    ; CHECK: STRSui renamable $s0, renamable $x1, 0 :: (store (s32))
-    ; CHECK: STRSui renamable $s1, renamable $x1, 1 :: ("aarch64-suppress-pair" store (s32))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $s0 = LDRSpre renamable $x1, 251, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
+    ; CHECK-NEXT: STPSi renamable $s0, renamable $s1, renamable $x1, 0 :: (store (s32))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $s0 = LDRSpre killed renamable $x1, 251 :: (load (s32))
     renamable $s1 = LDRSui renamable $x1, 1 :: (load (s32))
     STRSui killed renamable $s0, renamable $x1, 0 :: (store (s32))
     STRSui killed renamable $s1, renamable $x1, 1 :: (store (s32))
     RET undef $lr
 ...
 
 
 ---
 name:            21-ldrswpre-ldrswui-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 21-ldrswpre-ldrswui-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: early-clobber $x1, renamable $x0, renamable $x2 = LDPSWpre renamable $x1, 10 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $x0, renamable $x2 = LDPSWpre renamable $x1, 10 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 40 :: (load (s32))
     renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            22-ldrswpre-ldurswi-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 22-ldrswpre-ldurswi-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: early-clobber $x1, renamable $x0, renamable $x2 = LDPSWpre renamable $x1, 10 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber $x1, renamable $x0, renamable $x2 = LDPSWpre renamable $x1, 10 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 40 :: (load (s32))
     renamable $x2 = LDURSWi renamable $x1, 4 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            23-ldrswui-ldrswpre-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 23-ldrswui-ldrswpre-no-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 40 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            24-ldurswi-ldrswpre-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 24-ldurswi-ldrswpre-no-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: renamable $x2 = LDURSWi renamable $x1, 4 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $x2 = LDURSWi renamable $x1, 4 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     renamable $x2 = LDURSWi renamable $x1, 4 :: (load (s32))
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 40 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            25-ldrswpre-ldrswpre-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 25-ldrswpre-ldrswpre-no-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: early-clobber renamable $x1, dead renamable $x0 = LDRSWpre renamable $x1, 48, implicit $w1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, dead renamable $x2 = LDRSWpre renamable $x1, 1, implicit $w1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, dead renamable $x0 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, dead renamable $x2 = LDRSWpre renamable $x1, 12, implicit $w1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
-    ; CHECK: early-clobber renamable $x1, renamable $x2 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $x0 = LDRSWpre renamable $x1, 48, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $x2 = LDRSWpre renamable $x1, 1, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $x0 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, dead renamable $x2 = LDRSWpre renamable $x1, 12, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $x2 = LDRSWpre renamable $x1, 16, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 48 :: (load (s32))
     early-clobber renamable $x1, renamable $x2 = LDRSWpre killed renamable $x1, 1 :: (load (s32))
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 16 :: (load (s32))
     early-clobber renamable $x1, renamable $x2 = LDRSWpre killed renamable $x1, 12 :: (load (s32))
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 16 :: (load (s32))
     early-clobber renamable $x1, renamable $x2 = LDRSWpre killed renamable $x1, 16 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
liveins:
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 26-ldrswpre-ldrwui-no-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $w2 = LDRWui renamable $x1, 1, implicit-def $x2 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $x0 = LDRSWpre renamable $x1, 40, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $w2 = LDRWui renamable $x1, 1, implicit-def $x2 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $x0 = LDRSWpre killed renamable $x1, 40 :: (load (s32))
     renamable $w2 = LDRWui renamable $x1, 1 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            27-ldrwpre-ldrswui-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x0' }
   - { reg: '$x1' }
   - { reg: '$x2' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x0, $x1, $x2
     ; CHECK-LABEL: name: 27-ldrwpre-ldrswui-no-merge
     ; CHECK: liveins: $x0, $x1, $x2
-    ; CHECK: early-clobber renamable $x1, renamable $w0 = LDRWpre renamable $x1, 40, implicit $w1 :: (load (s32))
-    ; CHECK: renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
-    ; CHECK: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x1, renamable $w0 = LDRWpre renamable $x1, 40, implicit $w1 :: (load (s32))
+    ; CHECK-NEXT: renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
+    ; CHECK-NEXT: STPXi renamable $x0, renamable $x2, renamable $x1, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x1, renamable $w0 = LDRWpre killed renamable $x1, 40 :: (load (s32))
     renamable $x2 = LDRSWui renamable $x1, 1 :: (load (s32))
     STRXui killed renamable $x0, renamable $x1, 0 :: (store (s64))
     STRXui killed renamable $x2, renamable $x1, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            28-ldrswpre-ldrwpre-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x11' }
   - { reg: '$x13' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x11, $x13
     ; CHECK-LABEL: name: 28-ldrswpre-ldrwpre-no-merge
     ; CHECK: liveins: $x11, $x13
-    ; CHECK: early-clobber renamable $x11, dead renamable $x10 = LDRSWpre renamable $x11, 8, implicit $w11 :: (load (s32), align 8)
-    ; CHECK: $x14 = EORXrs renamable $x11, renamable $x13, 0
-    ; CHECK: early-clobber renamable $x11, dead renamable $w12 = LDRWpre renamable $x11, 4, implicit $w11 :: (load (s32))
-    ; CHECK: $x13 = EORXrs renamable $x11, renamable $x13, 0
-    ; CHECK: STPXi renamable $x13, renamable $x14, renamable $x11, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x11, dead renamable $x10 = LDRSWpre renamable $x11, 8, implicit $w11 :: (load (s32), align 8)
+    ; CHECK-NEXT: $x14 = EORXrs renamable $x11, renamable $x13, 0
+    ; CHECK-NEXT: early-clobber renamable $x11, dead renamable $w12 = LDRWpre renamable $x11, 4, implicit $w11 :: (load (s32))
+    ; CHECK-NEXT: $x13 = EORXrs renamable $x11, renamable $x13, 0
+    ; CHECK-NEXT: STPXi renamable $x13, renamable $x14, renamable $x11, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x11, renamable $x10 = LDRSWpre killed renamable $x11, 8 :: (load (s32), align 8)
     $x14 = EORXrs renamable $x11, renamable $x13, 0
     early-clobber renamable $x11, renamable $w12 = LDRWpre killed renamable $x11, 4 :: (load (s32))
     $x13 = EORXrs renamable $x11, killed renamable $x13, 0
     STRXui killed renamable $x13, renamable $x11, 0 :: (store (s64))
     STRXui killed renamable $x14, renamable $x11, 1 :: (store (s64))
     RET undef $lr
 ...
 
 
 ---
 name:            29-ldrwpre-ldrswpre-no-merge
 tracksRegLiveness: true
 liveins:
   - { reg: '$x11' }
   - { reg: '$x13' }
 machineFunctionInfo:
   hasRedZone:      false
 body:             |
   bb.0:
     liveins: $x11, $x13
     ; CHECK-LABEL: name: 29-ldrwpre-ldrswpre-no-merge
     ; CHECK: liveins: $x11, $x13
-    ; CHECK: early-clobber renamable $x11, dead renamable $w12 = LDRWpre renamable $x11, 8, implicit $w11 :: (load (s32))
-    ; CHECK: $x14 = EORXrs renamable $x11, renamable $x13, 0
-    ; CHECK: early-clobber renamable $x11, dead renamable $x10 = LDRSWpre renamable $x11, 4, implicit $w11 :: (load (s32), align 8)
-    ; CHECK: $x13 = EORXrs renamable $x11, renamable $x13, 0
-    ; CHECK: STPXi renamable $x13, renamable $x14, renamable $x11, 0 :: (store (s64))
-    ; CHECK: RET undef $lr
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: early-clobber renamable $x11, dead renamable $w12 = LDRWpre renamable $x11, 8, implicit $w11 :: (load (s32))
+    ; CHECK-NEXT: $x14 = EORXrs renamable $x11, renamable $x13, 0
+    ; CHECK-NEXT: early-clobber renamable $x11, dead renamable $x10 = LDRSWpre renamable $x11, 4, implicit $w11 :: (load (s32), align 8)
+    ; CHECK-NEXT: $x13 = EORXrs renamable $x11, renamable $x13, 0
+    ; CHECK-NEXT: STPXi renamable $x13, renamable $x14, renamable $x11, 0 :: (store (s64))
+    ; CHECK-NEXT: RET undef $lr
     early-clobber renamable $x11, renamable $w12 = LDRWpre killed renamable $x11, 8 :: (load (s32))
     $x14 = EORXrs renamable $x11, renamable $x13, 0
     early-clobber renamable $x11, renamable $x10 = LDRSWpre killed renamable $x11, 4 :: (load (s32), align 8)
     $x13 = EORXrs renamable $x11, killed renamable $x13, 0
     STRXui killed renamable $x13, renamable $x11, 0 :: (store (s64))
     STRXui killed renamable $x14, renamable $x11, 1 :: (store (s64))
     RET undef $lr
 ...

llvm/test/CodeGen/AArch64/no-sve-no-neon.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=-neon < %s | FileCheck %s
 
 target triple = "aarch64-unknown-linux-gnu-elf"
 
 define <16 x float> @foo(<16 x i64> %a) {
 ; CHECK-LABEL: foo:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldp x10, x9, [sp, #48]
 ; CHECK-NEXT:    ucvtf s0, x9
 ; CHECK-NEXT:    ldp x11, x9, [sp, #32]
 ; CHECK-NEXT:    ucvtf s1, x10
 ; CHECK-NEXT:    ucvtf s2, x9
 ; CHECK-NEXT:    ldp x10, x9, [sp, #16]
 ; CHECK-NEXT:    ucvtf s3, x11
 ; CHECK-NEXT:    str s0, [x8, #60]
-; CHECK-NEXT:    str s1, [x8, #56]
 ; CHECK-NEXT:    ucvtf s4, x9
-; CHECK-NEXT:    ucvtf s0, x10
 ; CHECK-NEXT:    ldp x11, x9, [sp]
-; CHECK-NEXT:    str s2, [x8, #52]
-; CHECK-NEXT:    str s3, [x8, #48]
-; CHECK-NEXT:    ucvtf s3, x7
+; CHECK-NEXT:    ucvtf s0, x10
+; CHECK-NEXT:    stp s2, s1, [x8, #52]
 ; CHECK-NEXT:    ucvtf s1, x9
 ; CHECK-NEXT:    ucvtf s2, x11
-; CHECK-NEXT:    str s4, [x8, #44]
+; CHECK-NEXT:    stp s4, s3, [x8, #44]
+; CHECK-NEXT:    ucvtf s3, x7
 ; CHECK-NEXT:    ucvtf s4, x6
-; CHECK-NEXT:    str s0, [x8, #40]
+; CHECK-NEXT:    stp s1, s0, [x8, #36]
 ; CHECK-NEXT:    ucvtf s0, x5
-; CHECK-NEXT:    str s3, [x8, #28]
-; CHECK-NEXT:    ucvtf s3, x2
-; CHECK-NEXT:    str s1, [x8, #36]
 ; CHECK-NEXT:    ucvtf s1, x4
-; CHECK-NEXT:    str s2, [x8, #32]
+; CHECK-NEXT:    stp s3, s2, [x8, #28]
 ; CHECK-NEXT:    ucvtf s2, x3
-; CHECK-NEXT:    str s4, [x8, #24]
+; CHECK-NEXT:    ucvtf s3, x2
+; CHECK-NEXT:    stp s0, s4, [x8, #20]
 ; CHECK-NEXT:    ucvtf s4, x1
-; CHECK-NEXT:    str s0, [x8, #20]
 ; CHECK-NEXT:    ucvtf s0, x0
-; CHECK-NEXT:    str s1, [x8, #16]
-; CHECK-NEXT:    str s2, [x8, #12]
-; CHECK-NEXT:    str s3, [x8, #8]
-; CHECK-NEXT:    str s4, [x8, #4]
+; CHECK-NEXT:    stp s2, s1, [x8, #12]
+; CHECK-NEXT:    stp s4, s3, [x8, #4]
 ; CHECK-NEXT:    str s0, [x8]
 ; CHECK-NEXT:    ret
   %conv1 = uitofp <16 x i64> %a to <16 x float>
   ret <16 x float> %conv1
 }

llvm/test/CodeGen/AArch64/storepairsuppress_minsize.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64 -mcpu=cortex-a55 -o - %s | FileCheck %s
+; RUN: llc -mtriple=aarch64 -mcpu=cortex-a55 -mattr=+store-pair-suppress -o - %s | FileCheck %s
 
 ; Check that stp are not suppressed at minsize.
 
 %T_IN_BLOCK = type [ 2 x { double, { double, double } } ]
 declare %T_IN_BLOCK @return_in_block()
 @in_block_store = dso_local global %T_IN_BLOCK zeroinitializer, align 8
 
 define void @test_default() uwtable {