[AArch64LoadStoreOptimizer] Generate more STPs by renaming registers earlier

[MeeraN]

Summary

Our initial motivating case was memcpy's with alignments > 16. The loads/stores, to which small memcpy's expand, are kept together in several places so that we get a sequence like this for a 64 bit copy:
LD w0
LD w1
ST w0
ST w1
The load/store optimiser can generate a LDP/STP w0, w1 from this because the registers read/written are consecutive. In our case however, the sequence is optimised during ISel, resulting in:
LD w0
ST w0
LD w0
ST w0
This instruction reordering allows reuse of registers. Since the registers are no longer consecutive (i.e. they are the same), it inhibits LDP/STP creation. The approach here is to perform renaming:
LD w0
ST w0
LD w1
ST w1
to enable the folding of the stores into a STP. We do not yet generate the LDP due to a limitation in the renaming implementation, but plan to look at that in a follow-up so that we fully support this case. While this was initially motivated by certain memcpy's, this is a general approach and thus is beneficial for other cases too, as can be seen in some test changes.

[SjoerdMeijer]

Hi Meera,

Thanks for working on this.

Correct me if I am wrong, but I think the problem here is that the LDP/STP creation is very fragile. The load/store optimiser relies on the pairs to be in order, otherwise it won't recognise them. For loads/stores with a bigger alignment (than usual), there is an optimisation in ISel that optimises the chain, and removes the TokenFactor that normally glues these things together, which is sensible optimisation. There have been some attempts to fix to keep the loads/stores for memcpy together in different places, but that is part of the fragile story. Your approach here is to perform renaming earlier for a sequence like this:

LDR w1
STR w1
LDR w1
STR w1

So that we get:

LDR w1
STR w1
LDR w2
STR w2

and can create a LDP w1, w2 and STP w1, w2.

I think it is worth clarifying some of these things, both in the description of the patch and a code comment.

Ideally we want to generate a LDP and STP for the test cases that you added. Why do we not yet get the LDP?

Speaking about the tests, I think they should be MIR tests. Then we see better what's going on, like the order of instructions etc.

[MeeraN]

Hi Sjoerd,

I can definitely add a comment in and some more information to the description as well in order to explain in more detail what is going on. I will also add a MIR test as well.

In D103597#2795765, @SjoerdMeijer wrote:

Ideally we want to generate a LDP and STP for the test cases that you added. Why do we not yet get the LDP?

We don't get the LDP since it fails at the very start of this function (called on line 1529) since FirstMI is a load instruction:

 static bool
 canRenameUpToDef(MachineInstr &FirstMI, LiveRegUnits &UsedInBetween,
                  SmallPtrSetImpl<const TargetRegisterClass *> &RequiredClasses,
                  const TargetRegisterInfo *TRI) {
   if (!FirstMI.mayStore())
     return false;

   ...
}

Because this returns false, the renaming of registers isn't attempted for the loads and so they don't get recognised as a pair. It is something I still need to look into in more detail to see if there is something else we can try.

[SjoerdMeijer]

In D103597#2795810, @MeeraN wrote:

In D103597#2795765, @SjoerdMeijer wrote:

Ideally we want to generate a LDP and STP for the test cases that you added. Why do we not yet get the LDP?

We don't get the LDP since it fails at the very start of this function (called on line 1529) since FirstMI is a load instruction:

 static bool
 canRenameUpToDef(MachineInstr &FirstMI, LiveRegUnits &UsedInBetween,
                  SmallPtrSetImpl<const TargetRegisterClass *> &RequiredClasses,
                  const TargetRegisterInfo *TRI) {
   if (!FirstMI.mayStore())
     return false;

   ...
}

Because this returns false, the renaming of registers isn't attempted for the loads and so they don't get recognised as a pair. It is something I still need to look into in more detail to see if there is something else we can try.

Okay, I see. I am not that familiar with that helper function, but this is a minor code reshuffle and a codegen improvement, so I would be okay to address the LDPs in a follow up if that is more convenient.

[MeeraN]

Added the inline comment to explain the changes, also added an MIR test as well as changing the existing test to show the difference when aarch64-load-store-renaming is enabled.

[SjoerdMeijer]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1712

Hmmmm, this hard coded 16 here....

1722

... and here, makes this look like a not very general approach.

I was hoping that we wouldn't need these alignment checks. We would possibly optimise other cases too, but that would be good. Or is there something else going on?

llvm/test/CodeGen/AArch64/memcpy.ll

1 ↗

(On Diff #349590)

Now that we have the MIR tests, I don't think we need these llc tests, it doesn't add anything, so you can just remove this file.

[MeeraN]

Removed llc test and also removed alignment check for values greater than 16. Updated the tests that failed because of this change with the new assembly now that the STP instruction is being generated.

[fhahn]

This does not seem memcpy specific. If that's the case, could you please update the title/description of the patch? Also the test cases in llvm/test/CodeGen/AArch64/memcpy.mir seem to specifically test additional renaming cases, so it might be good to put them into the existing llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir or into a file named along those lines. it would also be good to include a brief comment with the tests what is different to the existing re-naming tests.

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1521

The comment and the name seem to indicate that this function checks whether a register has been renamed or not, but I don't think that's the case.

The function checks if ice can rename from FirstMI to the beginning of the block and if it can, then it updates Flags with a register that can be used for renaming, if such a register has been found.

[MeeraN]

Changed patch title, moved tests to stp-opt-with-renaming.mir and corrected the function name and comment for the helper function in AArch64LoadStoreOptimizer.cpp

[SjoerdMeijer]

LGTM

Perhaps you can elaborate a little bit in the commit message:

When the order of loads and stores has been optimized during ISel, it prevents LDPs/STPs from being generated, so we attempt to rename registers at an earlier stage so that the instructions can be recognised as a pair in this order.

It's not clear which loads/stores. Perhaps you can expand a little bit on this, and say something along the lines of:

Our motivating case initially were memcpy's with an alignment > 16. The loads/stores to which small memcpy's expand to, are tried to be kept together in several places so that we get a sequence like this for a 64 bits copy:

LD w0
LD w1
ST w0
ST w1

The load/store optimiser can fold this in a LDP/STP w0, w1 because the registers read/written are consecutive. In our case however, the loads and stores chains has been optimised during ISel so that we end up with a sequence like this:

LD w0
ST w0
LD w0
ST w0

This instruction reordering/scheduling allows reuse of registers, and since the registers of the loads/stores are no longer consecutive (i.e. they are the same), it inhibits LDP/STP creation. The approach here is to perform renaming:

LD w0
ST w0
LD w1
ST w1

enabling the folding of the stores into a STP. We do not yet generate the LDP due to a limitation in the renaming implementation, but plan to look at that in a follow up so that we fully support this case. And while this was initially motivated by certain memcpy's, this is a general approach and thus beneficial for other cases too as can be seen in some test changes.

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1706

Nit: yet -> yet.

[fhahn]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

49

Is there a reason to do the whole check for rename registers twice, or could we determine whether re-naming is needed once for all cases? In particular, by doing it at this state, we may miss out on some legality checks, e.g. the one at line 1734 to check if the base register has been modified?

Can you add a test case where the base is modified?

50

nit: I don't think this matches LLVM's coding style (should be camel case with upper case start, https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly)

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

548 ↗

(On Diff #350850)

IIUC the *real* difference between the existing tests in the file and the new ones is that in the new ones, the merge-able STR instructions have the same register as stored operand? If that's the case, it would be good to clarify the comment, as in the other cases things could also originate from re-ordering.

554 ↗

(On Diff #350850)

can you also add a negative test, like one where there is no register available for renaming? I think you can do that by adding a couple of live-ins.

564 ↗

(On Diff #350850)

please reduce the values here, most are not needed (see the tests above).

[MeeraN]

@fhahn in response to your inline comment about performing the renaming registers check twice, if we don't have it we won't try to rename the registers at all, since the check on line 1764 is not reached in our case as we never enter the if statement that contains it. So we brought it earlier to make sure that it tries to rename the registers, if it can. I'll follow-up the rest of the comments in a new diff.

[mstorsjo]

FYI I've bisected down a misoptimization to this particular commit

[haowei]

We are seeing stage 2 clang build crashes on arm64 platform and confirmed it is caused by this change after bisecting. Could you revert the change please?

[mstorsjo]

The problem I'm seeing can be reproduced with this source file: https://martin.st/temp/cdf-preproc.c

To reproduce:

$ clang -target aarch64-w64-mingw32 -S -O3 cdf-preproc.c

One difference in the generated asm looks like this, where I think the error is visible:

        ldrh    w10, [x2, #6032]
        strh    wzr, [x1, #6034]
        strh    w10, [x1, #6032]
-       ldr     q0, [x2, #16]
-       str     q0, [x1, #16]
-       ldr     q0, [x2]
+       ldp     q0, q1, [x2]
        strh    wzr, [x1, #24]
-       str     q0, [x1]
-       ldr     q0, [x2, #48]
-       str     q0, [x1, #48]
-       ldr     q0, [x2, #32]
+       stp     q0, q1, [x1]
+       ldp     q0, q1, [x2, #32]

Previously the strh wzr, [x1, #24] was done after the str q0, [x1, #16] (so it would overwrite one part of the written qword with zeros), but now the strh remains intact and is done before the new stp q0, q1, [x1].

[SjoerdMeijer]

Many thanks for the reproducer!
We are going to have a look at that.

[fhahn]

In D103597#2808045, @MeeraN wrote:

@fhahn in response to your inline comment about performing the renaming registers check twice, if we don't have it we won't try to rename the registers at all, since the check on line 1764 is not reached in our case as we never enter the if statement that contains it. So we brought it earlier to make sure that it tries to rename the registers, if it can. I'll follow-up the rest of the comments in a new diff.

As I mentioned inline, this seems to be skipping a legality check (around line 1734), which is probably the source of the mis-compiles (please add a test for that).

I understand the motivation about trying to find a rename register earlier, but I was wondering if we can detect the need for renaming at a single (possible new) place. It may not be possible, because the register to check for renaming depends on whether we merge forward or backwards.

[thakis]

We were also seeing several downstream test failures on arm64 after this. Thanks for the speedy revert!

[MeeraN]

Added checks to ensure that STPs are only generated when the base register is not in use or has not been modified. Also added test cases when the base register has been modified and when there are no registers available for renaming.

[SjoerdMeijer]

Besides some nits/observations inline, one question about testing. The reproducer that was attached does not get miscompiled anymore? And it is exactly one of these cases that was added, i.e. the modified base register? Just asking to see if there's not another case or test that we are missing.

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1526

Style nit: can we return early here to reduce indentation:

if (!DebugCounter::shouldExecute(RegRenamingCounter))
  return false;

1697

Diffing with previous revision does not seem to work, so just double checking this.
You have added here the check if the base reg has been modified, and is available here, similarly like that is done from line 1726, right?

That seems okay to be me now.

llvm/test/CodeGen/AArch64/consthoist-gep.ll

3

Unrelated nit: perhaps align the assembly of this block for readability, indentation is all over the place

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

666 ↗

(On Diff #352084)

And here you have added a lot of registers as live-in, which results in no registers being available for renaming. Nice one, that looks good.

701 ↗

(On Diff #352084)

Yep, and here the base register is modified. Looks good.

[MeeraN]

Fixed the style points raised in the helper function updateFlagsWithRenameReg and in the test consthoist-gep.ll

[MeeraN]

@SjoerdMeijer, in response to your questions, yes I added the check to see if a base register is used/modified on line 1698/1699 so that we still perform that legality check as we won't reach the one on 1726 if the renaming of registers is successful. Also, the modified base register test is taken from the reproducer and so it should no longer generate extra, unwanted STPs.

[fhahn]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1746

Would it just be possible to check for renaming here, similar to like it is done below?

[MeeraN]

@fhahn I don't think that would be possible, at least not in our case, since it won't actually reach that line.

[SjoerdMeijer]

LGTM, but please wait a day with committing this in case there are more comments.

[mstorsjo]

In D103597#2809810, @mstorsjo wrote:

The problem I'm seeing can be reproduced with this source file: https://martin.st/temp/cdf-preproc.c

To reproduce:

$ clang -target aarch64-w64-mingw32 -S -O3 cdf-preproc.c

There still are issues that are reproducible with this particular sample. (I'm also seeing miscompilations still in some other samples that I haven't narrowed down yet, hoping that they just are more cases of this same issue.) The diff of the assembly output of that particular sample looks like this:

--- good.s      2021-06-23 09:42:24.381499762 +0300
+++ bad.s       2021-06-23 09:42:40.521154128 +0300
@@ -1955,17 +1955,14 @@
        ldrh    w10, [x2, #5756]
        strh    wzr, [x1, #5758]
        strh    w10, [x1, #5756]
-       ldr     q0, [x2, #976]
+       ldr     q1, [x2, #976]
        add     x10, x1, #960                   // =960
-       str     q0, [x1, #976]
        ldr     q0, [x2, #960]
        strh    wzr, [x10, #30]
-       str     q0, [x1, #960]
-       ldr     q0, [x2, #1008]
-       str     q0, [x1, #1008]
-       ldr     q0, [x2, #992]
+       stp     q0, q1, [x1, #960]
+       ldp     q0, q2, [x2, #992]
        strh    wzr, [x10, #62]
-       str     q0, [x1, #992]
+       stp     q0, q2, [x1, #992]
        ldr     q0, [x2, #1040]
        str     q0, [x1, #1040]
        ldr     q0, [x2, #1024]

Note that x10 = x1 + 960. The strh wzr, [x10, #30] clears a halfword at [x1, #990] (i.e. bytes x1[990-991]); this used to be done after str q0, [x1, #976] (which writes x1[976-991]), but now this write is done before the stp q0, q1, [x1, #960] (which writes x1[960-991]).

[fhahn]

In D103597#2824734, @MeeraN wrote:

@fhahn I don't think that would be possible, at least not in our case, since it won't actually reach that line.

The question is *why* does it not reach that code? I think we need to understand this first and provide reasons why skipping the other legality checks is fine. As the mis-compiles indicate, skipping the legality checks does not work well. Would it be possible to adjust the places that need & can legally do renaming in a more targeted approach?

[MeeraN]

In D103597#2835448, @fhahn wrote:

The question is *why* does it not reach that code? I think we need to understand this first and provide reasons why skipping the other legality checks is fine. As the mis-compiles indicate, skipping the legality checks does not work well. Would it be possible to adjust the places that need & can legally do renaming in a more targeted approach?

After taking a look into it, it doesn't reach that line because !mayAlias(MI, MemInsns, AA) will return false. I think that is the only check that is missing. But I'm not sure how we could adjust the place where renaming originally takes place as !mayAlias will return false for the store instructions we are trying to pair.

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <iterator>
 #include <limits>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-ldst-opt"
 
 STATISTIC(NumPairCreated, "Number of load/store pair instructions generated");
 STATISTIC(NumPostFolded, "Number of post-index updates folded");
 STATISTIC(NumPreFolded, "Number of pre-index updates folded");
 STATISTIC(NumUnscaledPairCreated,
           "Number of load/store from unscaled generated");
 STATISTIC(NumZeroStoresPromoted, "Number of narrow zero stores promoted");
 STATISTIC(NumLoadsFromStoresPromoted, "Number of loads from stores promoted");
 
if (DefinedInBB.available(PR) && UsedInBetween.available(PR) &&
       return {PR};
     }
   }
   LLVM_DEBUG(dbgs() << "No rename register found from "
                     << TRI->getRegClassName(RegClass) << "\n");
   return None;
 }
 
+// Returns a boolean that represents whether a register has been
+// renamed or not
+static bool
+regHasBeenRenamed(Optional<bool> MaybeCanRename, LdStPairFlags &Flags,
+                  MachineInstr &FirstMI, MachineInstr &MI,
+                  LiveRegUnits &DefinedInBB, LiveRegUnits &UsedInBetween,
+                  SmallPtrSetImpl<const TargetRegisterClass *> &RequiredClasses,
+                  const TargetRegisterInfo *TRI) {
+  if (DebugCounter::shouldExecute(RegRenamingCounter)) {
+    if (!MaybeCanRename)
+      MaybeCanRename = {
+          canRenameUpToDef(FirstMI, UsedInBetween, RequiredClasses, TRI)};
+
+    if (*MaybeCanRename) {
+      Optional<MCPhysReg> MaybeRenameReg = tryToFindRegisterToRename(
+          FirstMI, MI, DefinedInBB, UsedInBetween, RequiredClasses, TRI);
+      if (MaybeRenameReg) {
+        Flags.setRenameReg(*MaybeRenameReg);
+        Flags.setMergeForward(true);
+        return true;
+      }
+    }
+  }
+  return false;
+}
+
 /// Scan the instructions looking for a load/store that can be combined with the
 /// current instruction into a wider equivalent or a load/store pair.
 MachineBasicBlock::iterator
 AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
                                       LdStPairFlags &Flags, unsigned Limit,
                                       bool FindNarrowMerge) {
   MachineBasicBlock::iterator E = I->getParent()->end();
   MachineBasicBlock::iterator MBBI = I;
if (areCandidatesToMergeOrPair(FirstMI, MI, Flags, TII) &&
           // looking.
           if (IsUnscaled && (alignTo(MinOffset, OffsetStride) != MinOffset)) {
             LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits,
                                               UsedRegUnits, TRI);
             MemInsns.push_back(&MI);
             continue;
           }
         }
+        // If the load/store pattern has been optimized and reordered
+        // into the following:
+        //    ldr q0, [x1, #16]
+        //    str q0, [x0, #16]
+        //    ldr q0, [x1]
+        //    str q0, [x0]
+        // and the destination register of the load/store instruction is
+        // the same register as or a sub/super register of the other
+        // load/store, it will not generate an LDP/STP, so we attempt to
+        // rename the register so that it can be recognised as a pair.
+        // TODO: This is currently supported for STPs, LDPs are not
+        // being generated yet
+        if (TRI->isSuperOrSubRegisterEq(Reg, getLdStRegOp(MI).getReg())) {
+          bool renamedReg =
+              regHasBeenRenamed(MaybeCanRename, Flags, FirstMI, MI, DefinedInBB,
+                                UsedInBetween, RequiredClasses, TRI);
+          if (renamedReg) {
+            MBBIWithRenameReg = MBBI;
+            continue;
+          }
+        }
         // If the destination register of one load is the same register or a
         // sub/super register of the other load, bail and keep looking. A
         // load-pair instruction with both destination registers the same is
         // UNPREDICTABLE and will result in an exception.
         if (MayLoad &&
             TRI->isSuperOrSubRegisterEq(Reg, getLdStRegOp(MI).getReg())) {
           LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits, UsedRegUnits,
                                             TRI);
           MemInsns.push_back(&MI);
           continue;
         }
 
         // If the BaseReg has been modified, then we cannot do the optimization.
         // For example, in the following pattern
         //   ldr x1 [x2]
         //   ldr x2 [x3]
         //   ldr x4 [x2, #8],
         // the first and third ldr cannot be converted to ldp x1, x4, [x2]
         if (!ModifiedRegUnits.available(BaseReg))
           return E;
 
         // If the Rt of the second instruction was not modified or used between
         // the two instructions and none of the instructions between the second
         // and first alias with the second, we can combine the second into the
         // first.
         if (ModifiedRegUnits.available(getLdStRegOp(MI).getReg()) &&
             !(MI.mayLoad() &&
               !UsedRegUnits.available(getLdStRegOp(MI).getReg())) &&
             !mayAlias(MI, MemInsns, AA)) {
 
           Flags.setMergeForward(false);
           Flags.clearRenameReg();
           return MBBI;
         }
 
         // Likewise, if the Rt of the first instruction is not modified or used
         // between the two instructions and none of the instructions between the
         // first and the second alias with the first, we can combine the first
         // into the second.
         if (!(MayLoad &&
               !UsedRegUnits.available(getLdStRegOp(FirstMI).getReg())) &&
             !mayAlias(FirstMI, MemInsns, AA)) {
 
           if (ModifiedRegUnits.available(getLdStRegOp(FirstMI).getReg())) {
             Flags.setMergeForward(true);
             Flags.clearRenameReg();
             return MBBI;
           }
-
-          if (DebugCounter::shouldExecute(RegRenamingCounter)) {
-            if (!MaybeCanRename)
-              MaybeCanRename = {canRenameUpToDef(FirstMI, UsedInBetween,
-                                                 RequiredClasses, TRI)};
-
-            if (*MaybeCanRename) {
-              Optional<MCPhysReg> MaybeRenameReg = tryToFindRegisterToRename(
-                  FirstMI, MI, DefinedInBB, UsedInBetween, RequiredClasses,
-                  TRI);
-              if (MaybeRenameReg) {
-                Flags.setRenameReg(*MaybeRenameReg);
-                Flags.setMergeForward(true);
-                MBBIWithRenameReg = MBBI;
-              }
-            }
-          }
-        }
+          bool renamedReg =
+              regHasBeenRenamed(MaybeCanRename, Flags, FirstMI, MI, DefinedInBB,
+                                UsedInBetween, RequiredClasses, TRI);
+          if (renamedReg) {
         // Unable to combine these instructions due to interference in between.
         // Keep looking.
       }
     }
 
     if (Flags.getRenameReg())
       return MBBIWithRenameReg;
 

llvm/test/CodeGen/AArch64/GlobalISel/byval-call.ll

 ; CHECK-NEXT:    sub sp, sp, #288 // =288
 ; CHECK-NEXT:    stp x29, x30, [sp, #256] // 16-byte Folded Spill
 ; CHECK-NEXT:    str x28, [sp, #272] // 8-byte Folded Spill
 ; CHECK-NEXT:    add x29, sp, #256 // =256
 ; CHECK-NEXT:    .cfi_def_cfa w29, 32
 ; CHECK-NEXT:    .cfi_offset w28, -16
 ; CHECK-NEXT:    .cfi_offset w30, -24
 ; CHECK-NEXT:    .cfi_offset w29, -32
-; CHECK-NEXT:    ldr q0, [x0]
-; CHECK-NEXT:    str q0, [sp]
+; CHECK-NEXT:    ldr q1, [x0]
 ; CHECK-NEXT:    ldr q0, [x0, #16]
-; CHECK-NEXT:    str q0, [sp, #16]
-; CHECK-NEXT:    ldr q0, [x0, #32]
-; CHECK-NEXT:    str q0, [sp, #32]
+; CHECK-NEXT:    stp q1, q0, [sp]
+; CHECK-NEXT:    ldr q1, [x0, #32]
 ; CHECK-NEXT:    ldr q0, [x0, #48]
-; CHECK-NEXT:    str q0, [sp, #48]
-; CHECK-NEXT:    ldr q0, [x0, #64]
-; CHECK-NEXT:    str q0, [sp, #64]
+; CHECK-NEXT:    stp q1, q0, [sp, #32]
+; CHECK-NEXT:    ldr q1, [x0, #64]
 ; CHECK-NEXT:    ldr q0, [x0, #80]
-; CHECK-NEXT:    str q0, [sp, #80]
-; CHECK-NEXT:    ldr q0, [x0, #96]
-; CHECK-NEXT:    str q0, [sp, #96]
+; CHECK-NEXT:    stp q1, q0, [sp, #64]
+; CHECK-NEXT:    ldr q1, [x0, #96]
 ; CHECK-NEXT:    ldr q0, [x0, #112]
-; CHECK-NEXT:    str q0, [sp, #112]
-; CHECK-NEXT:    ldr q0, [x0, #128]
-; CHECK-NEXT:    str q0, [sp, #128]
+; CHECK-NEXT:    stp q1, q0, [sp, #96]
+; CHECK-NEXT:    ldr q1, [x0, #128]
 ; CHECK-NEXT:    ldr q0, [x0, #144]
-; CHECK-NEXT:    str q0, [sp, #144]
-; CHECK-NEXT:    ldr q0, [x0, #160]
-; CHECK-NEXT:    str q0, [sp, #160]
+; CHECK-NEXT:    stp q1, q0, [sp, #128]
+; CHECK-NEXT:    ldr q1, [x0, #160]
 ; CHECK-NEXT:    ldr q0, [x0, #176]
-; CHECK-NEXT:    str q0, [sp, #176]
-; CHECK-NEXT:    ldr q0, [x0, #192]
-; CHECK-NEXT:    str q0, [sp, #192]
+; CHECK-NEXT:    stp q1, q0, [sp, #160]
+; CHECK-NEXT:    ldr q1, [x0, #192]
 ; CHECK-NEXT:    ldr q0, [x0, #208]
-; CHECK-NEXT:    str q0, [sp, #208]
-; CHECK-NEXT:    ldr q0, [x0, #224]
-; CHECK-NEXT:    str q0, [sp, #224]
+; CHECK-NEXT:    stp q1, q0, [sp, #192]
+; CHECK-NEXT:    ldr q1, [x0, #224]
 ; CHECK-NEXT:    ldr q0, [x0, #240]
-; CHECK-NEXT:    str q0, [sp, #240]
+; CHECK-NEXT:    stp q1, q0, [sp, #224]
 ; CHECK-NEXT:    bl byval_a64i32
 ; CHECK-NEXT:    ldr x28, [sp, #272] // 8-byte Folded Reload
 ; CHECK-NEXT:    ldp x29, x30, [sp, #256] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #288 // =288
 ; CHECK-NEXT:    ret
   call void @byval_a64i32([64 x i32]* byval([64 x i32]) %incoming)
   ret void
 }

llvm/test/CodeGen/AArch64/consthoist-gep.ll

 ; RUN: llc -mtriple=aarch64-none-unknown-linuxeabi -consthoist-gep %s -o - | FileCheck %s
 
 ; CHECK-NOT: adrp    x10, global+332
 ; CHECK-NOT: add     x10, x10, :lo12:global+332
 ; CHECK: adrp    x10, global+528
+; CHECK-NEXT:  and w12, w8, #0xffffff
+; CHECK-NEXT:  ldr w8, [x11]
 ; CHECK-NEXT: add     x10, x10, :lo12:global+528
 
 %struct.blam = type { %struct.bar, %struct.bar.0, %struct.wobble, %struct.wombat, i8, i16, %struct.snork.2, %struct.foo, %struct.snork.3, %struct.wobble.4, %struct.quux, [9 x i16], %struct.spam, %struct.zot }
 %struct.bar = type { i8, i8, %struct.snork }
 %struct.snork = type { i16, i8, i8 }
 %struct.bar.0 = type { i8, i8, i16, i8, i8, %struct.barney }
 %struct.barney = type { i8, i8, i8, i8 }
 %struct.wobble = type { i8, i8, %struct.eggs, %struct.bar.1 }

llvm/test/CodeGen/AArch64/ldst-opt.ll

 ;  ->
 ; (ldr|str) X, [x20], #-16
 ;
 ; with X being either w0, x0, s0, d0 or q0.
 
 define void @post-indexed-sub-word(i32* %a, i32* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-word
 ; CHECK: ldr w{{[0-9]+}}, [x{{[0-9]+}}], #-8
-; CHECK: str w{{[0-9]+}}, [x{{[0-9]+}}], #-8
+; CHECK: stp w{{[0-9]+}}, w{{[0-9]+}}, [x0, #-4]
   br label %for.body
 for.body:
   %phi1 = phi i32* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi i32* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr i32, i32* %phi1, i64 -1
   %load1 = load i32, i32* %gep1
   %gep2 = getelementptr i32, i32* %phi2, i64 -1
   store i32 %load1, i32* %gep2
   %load2 = load i32, i32* %phi1
   store i32 %load2, i32* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr i32, i32* %phi2, i64 -2
   %gep4 = getelementptr i32, i32* %phi1, i64 -2
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-sub-doubleword(i64* %a, i64* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-doubleword
 ; CHECK: ldr x{{[0-9]+}}, [x{{[0-9]+}}], #-16
-; CHECK: str x{{[0-9]+}}, [x{{[0-9]+}}], #-16
+; CHECK: stp x{{[0-9]+}}, x{{[0-9]+}}, [x0, #-8]
   br label %for.body
 for.body:
   %phi1 = phi i64* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi i64* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr i64, i64* %phi1, i64 -1
   %load1 = load i64, i64* %gep1
   %gep2 = getelementptr i64, i64* %phi2, i64 -1
   store i64 %load1, i64* %gep2
   %load2 = load i64, i64* %phi1
   store i64 %load2, i64* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr i64, i64* %phi2, i64 -2
   %gep4 = getelementptr i64, i64* %phi1, i64 -2
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-sub-quadword(<2 x i64>* %a, <2 x i64>* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-quadword
 ; CHECK: ldr q{{[0-9]+}}, [x{{[0-9]+}}], #-32
-; CHECK: str q{{[0-9]+}}, [x{{[0-9]+}}], #-32
+; CHECK: stp q{{[0-9]+}}, q{{[0-9]+}}, [x0, #-16]
   br label %for.body
 for.body:
   %phi1 = phi <2 x i64>* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi <2 x i64>* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr <2 x i64>, <2 x i64>* %phi1, i64 -1
   %load1 = load <2 x i64>, <2 x i64>* %gep1
   %gep2 = getelementptr <2 x i64>, <2 x i64>* %phi2, i64 -1
   store <2 x i64> %load1, <2 x i64>* %gep2
   %load2 = load <2 x i64>, <2 x i64>* %phi1
   store <2 x i64> %load2, <2 x i64>* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr <2 x i64>, <2 x i64>* %phi2, i64 -2
   %gep4 = getelementptr <2 x i64>, <2 x i64>* %phi1, i64 -2
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-sub-float(float* %a, float* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-float
 ; CHECK: ldr s{{[0-9]+}}, [x{{[0-9]+}}], #-8
-; CHECK: str s{{[0-9]+}}, [x{{[0-9]+}}], #-8
+; CHECK: stp s{{[0-9]+}}, s{{[0-9]+}}, [x0, #-4]
   br label %for.body
 for.body:
   %phi1 = phi float* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi float* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr float, float* %phi1, i64 -1
   %load1 = load float, float* %gep1
   %gep2 = getelementptr float, float* %phi2, i64 -1
   store float %load1, float* %gep2
   %load2 = load float, float* %phi1
   store float %load2, float* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr float, float* %phi2, i64 -2
   %gep4 = getelementptr float, float* %phi1, i64 -2
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-sub-double(double* %a, double* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-double
 ; CHECK: ldr d{{[0-9]+}}, [x{{[0-9]+}}], #-16
-; CHECK: str d{{[0-9]+}}, [x{{[0-9]+}}], #-16
+; CHECK: stp d{{[0-9]+}}, d{{[0-9]+}}, [x0, #-8]
   br label %for.body
 for.body:
   %phi1 = phi double* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi double* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr double, double* %phi1, i64 -1
   %load1 = load double, double* %gep1
   %gep2 = getelementptr double, double* %phi2, i64 -1
   store double %load1, double* %gep2
   %load2 = load double, double* %phi1
   store double %load2, double* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr double, double* %phi2, i64 -2
   %gep4 = getelementptr double, double* %phi1, i64 -2
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-sub-doubleword-offset-min(i64* %a, i64* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-sub-doubleword-offset-min
 ; CHECK: ldr x{{[0-9]+}}, [x{{[0-9]+}}], #-256
-; CHECK: str x{{[0-9]+}}, [x{{[0-9]+}}], #-256
+; CHECK: stp x{{[0-9]+}}, x{{[0-9]+}}, [x0], #-256
   br label %for.body
 for.body:
   %phi1 = phi i64* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi i64* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr i64, i64* %phi1, i64 1
   %load1 = load i64, i64* %gep1
   %gep2 = getelementptr i64, i64* %phi2, i64 1
   store i64 %load1, i64* %gep2
   %load2 = load i64, i64* %phi1
   store i64 %load2, i64* %phi2
   %dec.i = add nsw i64 %i, -1
   %gep3 = getelementptr i64, i64* %phi2, i64 -32
   %gep4 = getelementptr i64, i64* %phi1, i64 -32
   %cond = icmp sgt i64 %dec.i, 0
   br i1 %cond, label %for.body, label %end
 end:
   ret void
 }
 
 define void @post-indexed-doubleword-offset-out-of-range(i64* %a, i64* %b, i64 %count) nounwind {
 ; CHECK-LABEL: post-indexed-doubleword-offset-out-of-range
 ; CHECK: ldr x{{[0-9]+}}, [x{{[0-9]+}}]
 ; CHECK: add x{{[0-9]+}}, x{{[0-9]+}}, #256
-; CHECK: str x{{[0-9]+}}, [x{{[0-9]+}}]
-; CHECK: add x{{[0-9]+}}, x{{[0-9]+}}, #256
+; CHECK: stp x{{[0-9]+}}, x{{[0-9]+}}, [x0], #256
 
   br label %for.body
 for.body:
   %phi1 = phi i64* [ %gep4, %for.body ], [ %b, %0 ]
   %phi2 = phi i64* [ %gep3, %for.body ], [ %a, %0 ]
   %i = phi i64 [ %dec.i, %for.body], [ %count, %0 ]
   %gep1 = getelementptr i64, i64* %phi1, i64 1
   %load1 = load i64, i64* %gep1

llvm/test/CodeGen/AArch64/memcpy.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64-linux-gnu -run-pass=aarch64-ldst-opt %s -o - | FileCheck %s
+
+...
+---
+name:            memcpy32
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$x0', virtual-reg: '' }
+  - { reg: '$x1', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  bb.0.entry:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: memcpy32
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: $q1 = LDRQui renamable $x1, 1 :: (load 16)
+    ; CHECK: renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 32)
+    ; CHECK: STPQi killed renamable $q0, killed $q1, killed renamable $x0, 0 :: (store 16, align 32)
+    ; CHECK: RET undef $lr
+    renamable $q0 = LDRQui renamable $x1, 1 :: (load 16)
+    STRQui killed renamable $q0, renamable $x0, 1 :: (store 16, basealign 32)
+    renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 32)
+    STRQui killed renamable $q0, killed renamable $x0, 0 :: (store 16, align 32)
+    RET undef $lr
+
+...
+---
+name:            memcpy64
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$x0', virtual-reg: '' }
+  - { reg: '$x1', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  bb.0.entry:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: memcpy64
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: $q1 = LDRQui renamable $x1, 1 :: (load 16)
+    ; CHECK: renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 64)
+    ; CHECK: STPQi killed renamable $q0, killed $q1, killed renamable $x0, 0 :: (store 16, align 64)
+    ; CHECK: RET undef $lr
+    renamable $q0 = LDRQui renamable $x1, 1 :: (load 16)
+    STRQui killed renamable $q0, renamable $x0, 1 :: (store 16, basealign 64)
+    renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 64)
+    STRQui killed renamable $q0, killed renamable $x0, 0 :: (store 16, align 64)
+    RET undef $lr
+
+...
+---
+name:            memcpy128
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$x0', virtual-reg: '' }
+  - { reg: '$x1', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  bb.0.entry:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: memcpy128
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: $q1 = LDRQui renamable $x1, 1 :: (load 16)
+    ; CHECK: renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 128)
+    ; CHECK: STPQi killed renamable $q0, killed $q1, killed renamable $x0, 0 :: (store 16, align 128)
+    ; CHECK: RET undef $lr
+    renamable $q0 = LDRQui renamable $x1, 1 :: (load 16)
+    STRQui killed renamable $q0, renamable $x0, 1 :: (store 16, basealign 128)
+    renamable $q0 = LDRQui killed renamable $x1, 0 :: (load 16, align 128)
+    STRQui killed renamable $q0, killed renamable $x0, 0 :: (store 16, align 128)
+    RET undef $lr
+
+...