[AArch64] Order STP Q's by ascending address

[avieira]

Summary

Hi,

This patch adds an AArch64 specific PostRA MachineScheduler to try to schedule STP Q's to the same base-address in ascending order of offsets. We have found this to improve performance on Neoverse N1 and should not hurt other AArch64 cores.

[dmgreen]

In order sounds sensible to me. We may need a subtarget feature for this, it depends on what the Apple folks think, but we can always add one later if needed.

llvm/lib/Target/AArch64/AArch64MachineScheduler.cpp

19

Can you add a comment explaining what this is doing and why.

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

488

Should this either return DAG, or create DAG inside the if.

[avieira]

llvm/lib/Target/AArch64/AArch64MachineScheduler.cpp

19

That sounds like a good idea yeah ;)

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

488

Ah yeah good catch! Forgot to change that when I moved the DSG creation out of the Fusion check.

[fhahn]

In D125377#3508288, @dmgreen wrote:

In order sounds sensible to me. We may need a subtarget feature for this, it depends on what the Apple folks think, but we can always add one later if needed.

I *think* it should be fine, but I can double-check.

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

484

Might make sense to move the code from AArch64MacroFusion.cpp to the new "AArch64MachineScheduler.h"

[avieira]

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

484

I don't have strong feelings about this, but other targets seem to keep mutations and MachineScheduler definitions separately. Could put them all in the same file if you prefer or leave it as is... I do assume you meant merge AArch64MacroFusion.{h,cpp} with respectively AArch64MachineScheduler.{h,cpp}.

[dmgreen]

So long as there is no issues from @fhahn, this LGTM.

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

479

Can you make sure you clang-format the patch? The formatting is a bit off here.

[fhahn]

LGTM, I think that should be fine. ordering by ascending address is likely also slightly more readable for our users.

llvm/lib/Target/AArch64/AArch64MachineScheduler.cpp

19

might be good to reduce an early exit here if the candidate isn't valid to reduce the nesting level below.

23

nit: IMO it would be easier to follow the code if there would be a comment upfront describing what the whole block does, rather than interleaving the comments.

llvm/lib/Target/AArch64/AArch64MachineScheduler.h

34

nit: Stray newline

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

484

I do assume you meant merge AArch64MacroFusion.{h,cpp} with respectively AArch64MachineScheduler.{h,cpp}.

Yeah, moving the stuff from AArch64MacroFusion to AArch64MachineScheduler. In any case that's something that should be done one as follow-up.

[Allen]

hi @avieira , Does Neoverse N1 Out of order execution, why does the order of instruction launch significantly affect the performance?

[avieira]

Hi @Allen ,

I believe that 'scheduling doesn't matter for Out of Order cores' is a long-standing myth that we've seen not to be correct. Yes, scheduling is definitely different for out of order cores, the problem shifts towards thinking about a sliding window of instructions that go into specific pipelines and dispatch queues and the likes. And you now find yourself trying to optimize the utilisation of pipelines, avoiding bubbles, rather than looking for the 'perfect sequence'. Out of order execution has some other limits and in some cases it helps if the compiler can lend the core a hand. In this case the Neoverse N1 prefers ascending STP Q's and an updated Neoverse N1 Software Optimization Guide will be reflecting this.

[Allen]

Thanks @avieira for your detailed explanation, and according your idea, I extend more stores in D126700, can you help me review, thanks.

llvm/lib/Target/AArch64/AArch64MachineScheduler.h

+//===- AArch64MachineScheduler.h - Custom AArch64 MI scheduler --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Custom AArch64 MI scheduler.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AARCH64_AARCH64MACHINESCHEDULER_H
+#define LLVM_LIB_TARGET_AARCH64_AARCH64MACHINESCHEDULER_H
+
+#include "llvm/CodeGen/MachineScheduler.h"
+
+namespace llvm {
+
+/// A MachineSchedStrategy implementation for AArch64 post RA scheduling.
+class AArch64PostRASchedStrategy : public PostGenericScheduler {
+public:
+  AArch64PostRASchedStrategy(const MachineSchedContext *C) :
+    PostGenericScheduler(C) {}
+
+protected:
+  bool tryCandidate(SchedCandidate &Cand, SchedCandidate &TryCand) override;
+};
+
+} // end namespace llvm
+
+#endif
+

llvm/lib/Target/AArch64/AArch64MachineScheduler.cpp

+//===- AArch64MachineScheduler.cpp - MI Scheduler for AArch64 -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "AArch64MachineScheduler.h"
+#include "MCTargetDesc/AArch64MCTargetDesc.h"
+
+using namespace llvm;
+
+bool AArch64PostRASchedStrategy::tryCandidate(SchedCandidate &Cand,
+                                              SchedCandidate &TryCand) {
+  bool OriginalResult = PostGenericScheduler::tryCandidate(Cand, TryCand);
+
+  if (Cand.isValid()) {
+    MachineInstr *Instr0 = TryCand.SU->getInstr();
+    MachineInstr *Instr1 = Cand.SU->getInstr();
+    // When dealing with two STPqi's.
+    if (Instr0 && Instr1 && Instr0->getOpcode() == Instr1->getOpcode () &&
+        Instr0->getOpcode() == AArch64::STPQi)
+    {
+      MachineOperand &Base0 = Instr0->getOperand(2);
+      MachineOperand &Base1 = Instr1->getOperand(2);
+      int64_t Off0 = Instr0->getOperand(3).getImm();
+      int64_t Off1 = Instr1->getOperand(3).getImm();
+      // With the same base address and non-overlapping writes.
+      if (Base0.isIdenticalTo(Base1) && llabs (Off0 - Off1) >= 2) {
+        TryCand.Reason = NodeOrder;
+        // Order them by ascending offsets.
+        return Off0 < Off1;
+      }
+    }
+  }
+
+  return OriginalResult;
+}

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

 //===-- AArch64TargetMachine.cpp - Define TargetMachine for AArch64 -------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64TargetMachine.h"
 #include "AArch64.h"
 #include "AArch64MachineFunctionInfo.h"
+#include "AArch64MachineScheduler.h"
 #include "AArch64MacroFusion.h"
 #include "AArch64Subtarget.h"
 #include "AArch64TargetObjectFile.h"
 #include "AArch64TargetTransformInfo.h"
 #include "MCTargetDesc/AArch64MCTargetDesc.h"
 #include "TargetInfo/AArch64TargetInfo.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Triple.h"
createMachineScheduler(MachineSchedContext *C) const override {
     if (ST.hasFusion())
       DAG->addMutation(createAArch64MacroFusionDAGMutation());
     return DAG;
   }
 
   ScheduleDAGInstrs *
   createPostMachineScheduler(MachineSchedContext *C) const override {
     const AArch64Subtarget &ST = C->MF->getSubtarget<AArch64Subtarget>();
+    ScheduleDAGMI *DAG =
+        new ScheduleDAGMI(C, std::make_unique<AArch64PostRASchedStrategy>(C),
+                          /* RemoveKillFlags=*/true);
     if (ST.hasFusion()) {
       // Run the Macro Fusion after RA again since literals are expanded from
       // pseudos then (v. addPreSched2()).
-      ScheduleDAGMI *DAG = createGenericSchedPostRA(C);
       DAG->addMutation(createAArch64MacroFusionDAGMutation());
       return DAG;
     }
 
-    return nullptr;
+    return DAG;
   }
 
   void addIRPasses()  override;
   bool addPreISel() override;
   void addCodeGenPrepare() override;
   bool addInstSelector() override;
   bool addIRTranslator() override;
   void addPreLegalizeMachineIR() override;

llvm/lib/Target/AArch64/CMakeLists.txt

add_llvm_target(AArch64CodeGen
   AArch64ConditionOptimizer.cpp
   AArch64RedundantCopyElimination.cpp
   AArch64ISelDAGToDAG.cpp
   AArch64ISelLowering.cpp
   AArch64InstrInfo.cpp
   AArch64LoadStoreOptimizer.cpp
   AArch64LowerHomogeneousPrologEpilog.cpp
   AArch64MachineFunctionInfo.cpp
+  AArch64MachineScheduler.cpp
   AArch64MacroFusion.cpp
   AArch64MIPeepholeOpt.cpp
   AArch64MCInstLower.cpp
   AArch64PromoteConstant.cpp
   AArch64PBQPRegAlloc.cpp
   AArch64RegisterInfo.cpp
   AArch64SLSHardening.cpp
   AArch64SelectionDAGInfo.cpp

llvm/test/CodeGen/AArch64/GlobalISel/call-translator-variadic-musttail.ll

 ; CHECK-NEXT:    add x0, x0, _asdf@PAGEOFF
 ; CHECK-NEXT:    mov x20, x1
 ; CHECK-NEXT:    mov x21, x2
 ; CHECK-NEXT:    mov x22, x3
 ; CHECK-NEXT:    mov x23, x4
 ; CHECK-NEXT:    mov x24, x5
 ; CHECK-NEXT:    mov x25, x6
 ; CHECK-NEXT:    mov x26, x7
-; CHECK-NEXT:    stp q1, q0, [sp, #96] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q7, q6, [sp] ; 32-byte Folded Spill
 ; CHECK-NEXT:    mov x27, x8
-; CHECK-NEXT:    stp q3, q2, [sp, #64] ; 32-byte Folded Spill
 ; CHECK-NEXT:    stp q5, q4, [sp, #32] ; 32-byte Folded Spill
-; CHECK-NEXT:    stp q7, q6, [sp] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q3, q2, [sp, #64] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q1, q0, [sp, #96] ; 32-byte Folded Spill
 ; CHECK-NEXT:    bl _puts
 ; CHECK-NEXT:    ldp q1, q0, [sp, #96] ; 32-byte Folded Reload
 ; CHECK-NEXT:    mov w0, w19
 ; CHECK-NEXT:    mov x1, x20
 ; CHECK-NEXT:    mov x2, x21
 ; CHECK-NEXT:    mov x3, x22
 ; CHECK-NEXT:    mov x4, x23
 ; CHECK-NEXT:    mov x5, x24

 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    mov x20, x1
 ; CHECK-NEXT:    mov x21, x2
 ; CHECK-NEXT:    mov x22, x3
 ; CHECK-NEXT:    mov x23, x4
 ; CHECK-NEXT:    mov x24, x5
 ; CHECK-NEXT:    mov x25, x6
 ; CHECK-NEXT:    mov x26, x7
-; CHECK-NEXT:    stp q1, q0, [sp, #96] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q7, q6, [sp] ; 32-byte Folded Spill
 ; CHECK-NEXT:    mov x27, x8
-; CHECK-NEXT:    stp q3, q2, [sp, #64] ; 32-byte Folded Spill
 ; CHECK-NEXT:    stp q5, q4, [sp, #32] ; 32-byte Folded Spill
-; CHECK-NEXT:    stp q7, q6, [sp] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q3, q2, [sp, #64] ; 32-byte Folded Spill
+; CHECK-NEXT:    stp q1, q0, [sp, #96] ; 32-byte Folded Spill
 ; CHECK-NEXT:    str x10, [x9]
 ; CHECK-NEXT:    bl _get_f
 ; CHECK-NEXT:    ldp q1, q0, [sp, #96] ; 32-byte Folded Reload
 ; CHECK-NEXT:    mov x9, x0
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    mov x1, x20
 ; CHECK-NEXT:    mov x2, x21
 ; CHECK-NEXT:    mov x3, x22

llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll

 
 ;; > 8 items goes on the stack
 
 define [ 9 x double ] @array_9() {
 ; CHECK-LABEL: array_9:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    str xzr, [x8, #64]
-; CHECK-NEXT:    stp q0, q0, [x8, #32]
 ; CHECK-NEXT:    stp q0, q0, [x8]
+; CHECK-NEXT:    stp q0, q0, [x8, #32]
 ; CHECK-NEXT:    ret
   ret [ 9 x double ] zeroinitializer
 }
 
 ;; Won't use any registers, just checking for assumptions.
 %T_STRUCT_0M = type { }
 
 define %T_STRUCT_0M @struct_zero_fields() {

 }
 
 ;; 5x2 fields = 10 so it is returned in memory.
 
 define [ 5 x %T_STRUCT_SAMEM ] @array_of_struct_in_memory() {
 ; CHECK-LABEL: array_of_struct_in_memory:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
-; CHECK-NEXT:    stp q0, q0, [x8, #48]
 ; CHECK-NEXT:    stp q0, q0, [x8, #16]
+; CHECK-NEXT:    stp q0, q0, [x8, #48]
 ; CHECK-NEXT:    str q0, [x8]
 ; CHECK-NEXT:    ret
   ret [ 5 x %T_STRUCT_SAMEM ] zeroinitializer
 }
 
 ;; A struct whose field is an array.
 %T_STRUCT_ARRAYM = type { [ 2 x double ]};
 
; CHECK-NEXT:    ret
   ret [ 1 x %T_NESTED_STRUCT_SAMEM ] zeroinitializer
 }
 
 ;; 2 x (1 + (2 x 2)) = 10 so this is returned in memory
 define [ 2 x %T_NESTED_STRUCT_SAMEM ] @array_of_struct_nested_same_field_types_2() {
 ; CHECK-LABEL: array_of_struct_nested_same_field_types_2:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
-; CHECK-NEXT:    stp q0, q0, [x8, #48]
 ; CHECK-NEXT:    stp q0, q0, [x8, #16]
+; CHECK-NEXT:    stp q0, q0, [x8, #48]
 ; CHECK-NEXT:    str q0, [x8]
 ; CHECK-NEXT:    ret
   ret [ 2 x %T_NESTED_STRUCT_SAMEM ] zeroinitializer
 }
 
 ;; Check combinations of call, return and argument passing
 
 %T_IN_BLOCK = type [ 2 x { double, { double, double } } ]

 
 %T_IN_MEMORY = type [ 3 x { double, { double, double } } ]
 
 define %T_IN_MEMORY @return_in_memory() {
 ; CHECK-LABEL: return_in_memory:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    str xzr, [x8, #64]
-; CHECK-NEXT:    stp q0, q0, [x8, #32]
 ; CHECK-NEXT:    stp q0, q0, [x8]
+; CHECK-NEXT:    stp q0, q0, [x8, #32]
 ; CHECK-NEXT:    ret
   ret %T_IN_MEMORY zeroinitializer
 }
 
 @in_memory_store = dso_local global %T_IN_MEMORY zeroinitializer, align 8
 
 define void @caller_in_memory() {
 ; CHECK-LABEL: caller_in_memory:

llvm/test/CodeGen/AArch64/arm64-memset-inline.ll

; CHECK-NEXT:    ret
   call void @llvm.memset.p0i8.i64(i8* align 8 %c, i8 0, i64 32, i1 false)
   ret void
 }
 
 define void @bzero_64_heap(i8* nocapture %c) {
 ; CHECK-LABEL: bzero_64_heap:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
-; CHECK-NEXT:    stp q0, q0, [x0, #32]
 ; CHECK-NEXT:    stp q0, q0, [x0]
+; CHECK-NEXT:    stp q0, q0, [x0, #32]
 ; CHECK-NEXT:    ret
   call void @llvm.memset.p0i8.i64(i8* align 8 %c, i8 0, i64 64, i1 false)
   ret void
 }
 
 define void @bzero_4_stack() {
 ; CHECK-LABEL: bzero_4_stack:
 ; CHECK:       // %bb.0:

 ; CHECK-LABEL: bzero_64_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #80
 ; CHECK-NEXT:    .cfi_def_cfa_offset 80
 ; CHECK-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #80
 ; CHECK-NEXT:    ret
   %buf = alloca [64 x i8], align 1
   %cast = bitcast [64 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 0, i32 64, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @bzero_72_stack() {
 ; CHECK-LABEL: bzero_72_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #96
 ; CHECK-NEXT:    .cfi_def_cfa_offset 96
 ; CHECK-NEXT:    str x30, [sp, #80] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x0, sp
 ; CHECK-NEXT:    str xzr, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #80] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #96
 ; CHECK-NEXT:    ret
   %buf = alloca [72 x i8], align 1
   %cast = bitcast [72 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 0, i32 72, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @bzero_128_stack() {
 ; CHECK-LABEL: bzero_128_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #144
 ; CHECK-NEXT:    .cfi_def_cfa_offset 144
 ; CHECK-NEXT:    str x30, [sp, #128] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #96]
-; CHECK-NEXT:    stp q0, q0, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
+; CHECK-NEXT:    stp q0, q0, [sp, #64]
+; CHECK-NEXT:    stp q0, q0, [sp, #96]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #128] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #144
 ; CHECK-NEXT:    ret
   %buf = alloca [128 x i8], align 1
   %cast = bitcast [128 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 0, i32 128, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @bzero_256_stack() {
 ; CHECK-LABEL: bzero_256_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #272
 ; CHECK-NEXT:    .cfi_def_cfa_offset 272
 ; CHECK-NEXT:    stp x29, x30, [sp, #256] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #224]
-; CHECK-NEXT:    stp q0, q0, [sp, #192]
-; CHECK-NEXT:    stp q0, q0, [sp, #160]
-; CHECK-NEXT:    stp q0, q0, [sp, #128]
-; CHECK-NEXT:    stp q0, q0, [sp, #96]
-; CHECK-NEXT:    stp q0, q0, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
+; CHECK-NEXT:    stp q0, q0, [sp, #64]
+; CHECK-NEXT:    stp q0, q0, [sp, #96]
+; CHECK-NEXT:    stp q0, q0, [sp, #128]
+; CHECK-NEXT:    stp q0, q0, [sp, #160]
+; CHECK-NEXT:    stp q0, q0, [sp, #192]
+; CHECK-NEXT:    stp q0, q0, [sp, #224]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldp x29, x30, [sp, #256] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #272
 ; CHECK-NEXT:    ret
   %buf = alloca [256 x i8], align 1
   %cast = bitcast [256 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 0, i32 256, i1 false)
   call void @something(i8* %cast)

 ; CHECK-LABEL: memset_64_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #80
 ; CHECK-NEXT:    .cfi_def_cfa_offset 80
 ; CHECK-NEXT:    str x30, [sp, #64] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.16b, #170
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #64] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #80
 ; CHECK-NEXT:    ret
   %buf = alloca [64 x i8], align 1
   %cast = bitcast [64 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 -86, i32 64, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @memset_72_stack() {
 ; CHECK-LABEL: memset_72_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #96
 ; CHECK-NEXT:    .cfi_def_cfa_offset 96
 ; CHECK-NEXT:    str x30, [sp, #80] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.16b, #170
 ; CHECK-NEXT:    mov x8, #-6148914691236517206
 ; CHECK-NEXT:    mov x0, sp
 ; CHECK-NEXT:    str x8, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #80] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #96
 ; CHECK-NEXT:    ret
   %buf = alloca [72 x i8], align 1
   %cast = bitcast [72 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 -86, i32 72, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @memset_128_stack() {
 ; CHECK-LABEL: memset_128_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #144
 ; CHECK-NEXT:    .cfi_def_cfa_offset 144
 ; CHECK-NEXT:    str x30, [sp, #128] // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    movi v0.16b, #170
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #96]
-; CHECK-NEXT:    stp q0, q0, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
+; CHECK-NEXT:    stp q0, q0, [sp, #64]
+; CHECK-NEXT:    stp q0, q0, [sp, #96]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldr x30, [sp, #128] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #144
 ; CHECK-NEXT:    ret
   %buf = alloca [128 x i8], align 1
   %cast = bitcast [128 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 -86, i32 128, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 define void @memset_256_stack() {
 ; CHECK-LABEL: memset_256_stack:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #272
 ; CHECK-NEXT:    .cfi_def_cfa_offset 272
 ; CHECK-NEXT:    stp x29, x30, [sp, #256] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
 ; CHECK-NEXT:    movi v0.16b, #170
 ; CHECK-NEXT:    mov x0, sp
-; CHECK-NEXT:    stp q0, q0, [sp, #224]
-; CHECK-NEXT:    stp q0, q0, [sp, #192]
-; CHECK-NEXT:    stp q0, q0, [sp, #160]
-; CHECK-NEXT:    stp q0, q0, [sp, #128]
-; CHECK-NEXT:    stp q0, q0, [sp, #96]
-; CHECK-NEXT:    stp q0, q0, [sp, #64]
-; CHECK-NEXT:    stp q0, q0, [sp, #32]
 ; CHECK-NEXT:    stp q0, q0, [sp]
+; CHECK-NEXT:    stp q0, q0, [sp, #32]
+; CHECK-NEXT:    stp q0, q0, [sp, #64]
+; CHECK-NEXT:    stp q0, q0, [sp, #96]
+; CHECK-NEXT:    stp q0, q0, [sp, #128]
+; CHECK-NEXT:    stp q0, q0, [sp, #160]
+; CHECK-NEXT:    stp q0, q0, [sp, #192]
+; CHECK-NEXT:    stp q0, q0, [sp, #224]
 ; CHECK-NEXT:    bl something
 ; CHECK-NEXT:    ldp x29, x30, [sp, #256] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #272
 ; CHECK-NEXT:    ret
   %buf = alloca [256 x i8], align 1
   %cast = bitcast [256 x i8]* %buf to i8*
   call void @llvm.memset.p0i8.i32(i8* %cast, i8 -86, i32 256, i1 false)
   call void @something(i8* %cast)
   ret void
 }
 
 declare void @something(i8*)
 declare void @llvm.memset.p0i8.i32(i8* nocapture, i8, i32, i1) nounwind
 declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind

llvm/test/CodeGen/AArch64/ragreedy-local-interval-cost.ll

 ; CHECK-NEXT:    add v2.2d, v2.2d, v0.2d
 ; CHECK-NEXT:    b.ne .LBB0_1
 ; CHECK-NEXT:  // %bb.2: // %for.cond.cleanup
 ; CHECK-NEXT:    adrp x8, C
 ; CHECK-NEXT:    add x8, x8, :lo12:C
 ; CHECK-NEXT:    stp q13, q12, [x8]
 ; CHECK-NEXT:    stp q11, q10, [x8, #32]
 ; CHECK-NEXT:    stp q9, q8, [x8, #64]
-; CHECK-NEXT:    stp q4, q15, [x8, #432]
-; CHECK-NEXT:    stp q14, q3, [x8, #464]
-; CHECK-NEXT:    ldp d9, d8, [sp, #80] // 16-byte Folded Reload
-; CHECK-NEXT:    stp q31, q30, [x8, #96]
 ; CHECK-NEXT:    ldp d11, d10, [sp, #64] // 16-byte Folded Reload
-; CHECK-NEXT:    stp q29, q28, [x8, #144]
+; CHECK-NEXT:    stp q31, q30, [x8, #96]
 ; CHECK-NEXT:    ldp d13, d12, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    stp q29, q28, [x8, #144]
 ; CHECK-NEXT:    stp q27, q26, [x8, #176]
-; CHECK-NEXT:    ldp d15, d14, [sp, #32] // 16-byte Folded Reload
 ; CHECK-NEXT:    str q25, [x8, #208]
 ; CHECK-NEXT:    stp q24, q23, [x8, #240]
 ; CHECK-NEXT:    stp q22, q21, [x8, #272]
 ; CHECK-NEXT:    stp q20, q19, [x8, #304]
 ; CHECK-NEXT:    stp q18, q17, [x8, #336]
 ; CHECK-NEXT:    stp q16, q7, [x8, #368]
 ; CHECK-NEXT:    stp q6, q5, [x8, #400]
+; CHECK-NEXT:    stp q4, q15, [x8, #432]
+; CHECK-NEXT:    stp q14, q3, [x8, #464]
+; CHECK-NEXT:    ldp d9, d8, [sp, #80] // 16-byte Folded Reload
 ; CHECK-NEXT:    str q2, [x8, #496]
+; CHECK-NEXT:    ldp d15, d14, [sp, #32] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #96
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore b8
 ; CHECK-NEXT:    .cfi_restore b9
 ; CHECK-NEXT:    .cfi_restore b10
 ; CHECK-NEXT:    .cfi_restore b11
 ; CHECK-NEXT:    .cfi_restore b12
 ; CHECK-NEXT:    .cfi_restore b13

llvm/test/CodeGen/AArch64/sve-fixed-length-fp-select.ll

 ; NO_SVE-NEXT:    ldr q10, [x1, #96]
 ; NO_SVE-NEXT:    bif v4.16b, v28.16b, v8.16b
 ; NO_SVE-NEXT:    ldr q28, [x1, #80]
 ; NO_SVE-NEXT:    ldr q24, [x1, #64]
 ; NO_SVE-NEXT:    ldr q25, [x1, #48]
 ; NO_SVE-NEXT:    ldr q26, [x1, #32]
 ; NO_SVE-NEXT:    ldr q27, [x1, #16]
 ; NO_SVE-NEXT:    ldr q11, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    stp q1, q0, [x0, #224]
 ; NO_SVE-NEXT:    mov v0.16b, v8.16b
-; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    mov v1.16b, v8.16b
 ; NO_SVE-NEXT:    mov v2.16b, v8.16b
 ; NO_SVE-NEXT:    bsl v0.16b, v5.16b, v29.16b
 ; NO_SVE-NEXT:    bsl v1.16b, v6.16b, v30.16b
 ; NO_SVE-NEXT:    bsl v2.16b, v7.16b, v31.16b
 ; NO_SVE-NEXT:    mov v3.16b, v8.16b
 ; NO_SVE-NEXT:    stp q0, q4, [x0, #160]
 ; NO_SVE-NEXT:    mov v4.16b, v8.16b

 ; NO_SVE-NEXT:    ldr q10, [x1, #96]
 ; NO_SVE-NEXT:    bif v4.16b, v28.16b, v8.16b
 ; NO_SVE-NEXT:    ldr q28, [x1, #80]
 ; NO_SVE-NEXT:    ldr q24, [x1, #64]
 ; NO_SVE-NEXT:    ldr q25, [x1, #48]
 ; NO_SVE-NEXT:    ldr q26, [x1, #32]
 ; NO_SVE-NEXT:    ldr q27, [x1, #16]
 ; NO_SVE-NEXT:    ldr q11, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    stp q1, q0, [x0, #224]
 ; NO_SVE-NEXT:    mov v0.16b, v8.16b
-; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    mov v1.16b, v8.16b
 ; NO_SVE-NEXT:    mov v2.16b, v8.16b
 ; NO_SVE-NEXT:    bsl v0.16b, v5.16b, v29.16b
 ; NO_SVE-NEXT:    bsl v1.16b, v6.16b, v30.16b
 ; NO_SVE-NEXT:    bsl v2.16b, v7.16b, v31.16b
 ; NO_SVE-NEXT:    mov v3.16b, v8.16b
 ; NO_SVE-NEXT:    stp q0, q4, [x0, #160]
 ; NO_SVE-NEXT:    mov v4.16b, v8.16b

 ; NO_SVE-NEXT:    ldr q10, [x1, #96]
 ; NO_SVE-NEXT:    bif v4.16b, v28.16b, v8.16b
 ; NO_SVE-NEXT:    ldr q28, [x1, #80]
 ; NO_SVE-NEXT:    ldr q24, [x1, #64]
 ; NO_SVE-NEXT:    ldr q25, [x1, #48]
 ; NO_SVE-NEXT:    ldr q26, [x1, #32]
 ; NO_SVE-NEXT:    ldr q27, [x1, #16]
 ; NO_SVE-NEXT:    ldr q11, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    stp q1, q0, [x0, #224]
 ; NO_SVE-NEXT:    mov v0.16b, v8.16b
-; NO_SVE-NEXT:    stp q3, q2, [x0, #192]
 ; NO_SVE-NEXT:    mov v1.16b, v8.16b
 ; NO_SVE-NEXT:    mov v2.16b, v8.16b
 ; NO_SVE-NEXT:    bsl v0.16b, v5.16b, v29.16b
 ; NO_SVE-NEXT:    bsl v1.16b, v6.16b, v30.16b
 ; NO_SVE-NEXT:    bsl v2.16b, v7.16b, v31.16b
 ; NO_SVE-NEXT:    mov v3.16b, v8.16b
 ; NO_SVE-NEXT:    stp q0, q4, [x0, #160]
 ; NO_SVE-NEXT:    mov v4.16b, v8.16b

llvm/utils/gn/secondary/llvm/lib/Target/AArch64/BUILD.gn

sources = [
     "AArch64ISelDAGToDAG.cpp",
     "AArch64ISelLowering.cpp",
     "AArch64InstrInfo.cpp",
     "AArch64LoadStoreOptimizer.cpp",
     "AArch64LowerHomogeneousPrologEpilog.cpp",
     "AArch64MCInstLower.cpp",
     "AArch64MIPeepholeOpt.cpp",
     "AArch64MachineFunctionInfo.cpp",
+    "AArch64MachineScheduler.cpp",
     "AArch64MacroFusion.cpp",
     "AArch64PBQPRegAlloc.cpp",
     "AArch64PromoteConstant.cpp",
     "AArch64RedundantCopyElimination.cpp",
     "AArch64RegisterInfo.cpp",
     "AArch64SIMDInstrOpt.cpp",
     "AArch64SLSHardening.cpp",
     "AArch64SelectionDAGInfo.cpp",