[AArch64] Teach Load/Store optimizier to rename store operands for pairing.

[fhahn]

Summary

In some cases, we can rename a store operand, in order to enable pairing
of stores. For store pairs, that cannot be merged because the first
tored register is defined in between the second store, we try to find
suitable rename register.

First, we check if we can rename the given register:

1. The first store register must be killed at the store, which means we do not have to rename instructions after the first store.
2. We scan backwards from the first store, to find the definition of the stored register and check all uses in between are renamable. Along they way, we collect the minimal register classes of the uses for overlapping (sub/super)registers.

Second, we try to find an available register from the minimal physical
register class of the original register. A suitable register must not be

1. defined before FirstMI
2. between the previous definition of the register to rename
3. a callee saved register.

We use KILL flags to clear defined registers while scanning from the
beginning to the end of the block.

This triggers quite often, here are the top changes for MultiSource,
SPEC2000, SPEC2006 compiled with -O3 for iOS:

Metric: aarch64-ldst-opt.NumPairCreated

Program base patch diff
test-suite...nch/fourinarow/fourinarow.test 2.00 39.00 1850.0%
test-suite...s/ASC_Sequoia/IRSmk/IRSmk.test 46.00 80.00 73.9%
test-suite...chmarks/Olden/power/power.test 70.00 96.00 37.1%
test-suite...cations/hexxagon/hexxagon.test 29.00 39.00 34.5%
test-suite...nchmarks/McCat/05-eks/eks.test 100.00 132.00 32.0%
test-suite.../Trimaran/enc-rc4/enc-rc4.test 46.00 59.00 28.3%
test-suite...T2006/473.astar/473.astar.test 160.00 200.00 25.0%
test-suite.../Trimaran/enc-md5/enc-md5.test 8.00 10.00 25.0%
test-suite...telecomm-gsm/telecomm-gsm.test 113.00 139.00 23.0%
test-suite...ediabench/gsm/toast/toast.test 113.00 139.00 23.0%
test-suite...Source/Benchmarks/sim/sim.test 91.00 111.00 22.0%
test-suite...C/CFP2000/179.art/179.art.test 41.00 49.00 19.5%
test-suite...peg2/mpeg2dec/mpeg2decode.test 245.00 279.00 13.9%
test-suite...marks/Olden/health/health.test 16.00 18.00 12.5%
test-suite...ks/Prolangs-C/cdecl/cdecl.test 90.00 101.00 12.2%
test-suite...fice-ispell/office-ispell.test 91.00 100.00 9.9%
test-suite...oxyApps-C/miniGMG/miniGMG.test 430.00 465.00 8.1%
test-suite...lowfish/security-blowfish.test 39.00 42.00 7.7%
test-suite.../Applications/spiff/spiff.test 42.00 45.00 7.1%
test-suite...arks/mafft/pairlocalalign.test 2473.00 2646.00 7.0%
test-suite.../VersaBench/ecbdes/ecbdes.test 29.00 31.00 6.9%
test-suite...nch/beamformer/beamformer.test 220.00 235.00 6.8%
test-suite...CFP2000/177.mesa/177.mesa.test 2110.00 2252.00 6.7%
test-suite...ve-susan/automotive-susan.test 109.00 116.00 6.4%
test-suite...s-C/unix-smail/unix-smail.test 65.00 69.00 6.2%
test-suite...CI_Purple/SMG2000/smg2000.test 1194.00 1265.00 5.9%
test-suite.../Benchmarks/nbench/nbench.test 472.00 500.00 5.9%
test-suite...oxyApps-C/miniAMR/miniAMR.test 248.00 262.00 5.6%
test-suite...quoia/CrystalMk/CrystalMk.test 18.00 19.00 5.6%
test-suite...rks/tramp3d-v4/tramp3d-v4.test 7331.00 7710.00 5.2%
test-suite.../Benchmarks/Bullet/bullet.test 5651.00 5938.00 5.1%
test-suite...ternal/HMMER/hmmcalibrate.test 750.00 788.00 5.1%
test-suite...T2006/456.hmmer/456.hmmer.test 764.00 802.00 5.0%
test-suite...ications/JM/ldecod/ldecod.test 1028.00 1079.00 5.0%
test-suite...CFP2006/444.namd/444.namd.test 1368.00 1434.00 4.8%
test-suite...marks/7zip/7zip-benchmark.test 4471.00 4685.00 4.8%
test-suite...6/464.h264ref/464.h264ref.test 3122.00 3271.00 4.8%
test-suite...pplications/oggenc/oggenc.test 1497.00 1565.00 4.5%
test-suite...T2000/300.twolf/300.twolf.test 742.00 774.00 4.3%
test-suite.../Prolangs-C/loader/loader.test 24.00 25.00 4.2%
test-suite...0.perlbench/400.perlbench.test 1983.00 2058.00 3.8%
test-suite...ications/JM/lencod/lencod.test 4612.00 4785.00 3.8%
test-suite...yApps-C++/PENNANT/PENNANT.test 995.00 1032.00 3.7%
test-suite...arks/VersaBench/dbms/dbms.test 54.00 56.00 3.7%

[fhahn]

I am not entirely happy with the code structure, please let me know if there are any helpful utilities I am missing. Especially when it comes to dealing with sub/super registers. I might try to improve things a bit by adding iterator ranges for sub/super registers as a follow up.

[fhahn]

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

11

it seems like we need the IR references for the test (AA?). Or is there a trick to get rid of them?

[paquette]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

87

Is it necessary to initialize this?

598–608

Do you specifically only want the unscaled stores here?

If not, it might make sense to use AArch64InstrInfo::isPairableLdStInst along with checking mayStore. Then we can avoid duplicating code.

808

Would it be possible to structure this loop differently so we can avoid while(true)?

(Since you're looping backwards, maybe using a reverse_iterator would work?)

815–817

Should probably check that Cur is valid at some point in this loop?

822–833

This looks very similar to the code in LiveRegUnits::stepBackward and LiveRegUnits::accumulate.

I think there's a nice opportunity to share some code there. Maybe it would make sense to add a physreg operand iterator or something and use that instead?

835

Is this comment accurate?

Maybe I'm misreading the function here, but it looks like you don't necessarily apply the function to all of the super registers. In the first loop, you can return before hitting the second loop, no?

850–851

Would it make sense to move this to LiveRegUnits?

869

Do you have to use this->TRI in these functions? Can you just pass TRI in the capture instead?

921–927

Instead of using a bool, how about

if (any_of(...))
  break;

1343

This function is pretty long, but if I'm counting braces right, it always returns false when this if is false.

I think that something like this might be a bit clearer, and would let you reduce the level of indentation:

if (!isStore(FirstMI))
  return false;

MachineFunction &MF = ...
if (!MF.getRegInfo().tracksLiveness())
  return false;

auto *RegClass...
if (!RegClass)
  return false;

...

1366

IsDef?

1396

Do you actually need the else here? I think you should always return in the if branch.

1398

You can probably reduce the level of indentation here:

if (!MOP.isReg() || !TRI->regsOverlap(...))
  continue;

if (!canRenameMOP(MOP)) {
 ...
}

RequiredClasses.insert(...)

2047

This doesn't seem to be used anywhere else?

2092–2095

Might be worth a comment?

2103–2105

remove braces?

2148

remove

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

2

Probably better to use update_mir_test_checks.py for this one?

3

You'll probably want -verify-machineinstrs here

Also I think you can just use -run-pass instead of -start-before and -start-after here.

11

Usually it's because the MIR references some specific bit of the IR.

For example, in this test there's something like this:

(store 4 into %ir.gxf_entry_el1)

There are likely more things than this in the test, but I think (if memory serves me well), to deal with *this* specific issue you would just have to replace this MIR line with (store 4). Then you can remove the associated IR because it is no longer referenced.

So theoretically, you can just try deleting all of the IR, see what fails, and then just add back what you need.

[fhahn]

Thanks Jessica, your comments should be addressed!

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

87

I do not think so, but I thought it's worth keeping things explicit.

598–608

yep, mayStore is enough :)

822–833

Yep, an iterator would be the nicest solution here.

835

Yep, we exit when the result of Fn evaluates to true. This is a bit weird, but was required to be usable in both cases..... An iterator would be nicer here as well.

850–851

I am not entirely sure.

921–927

after another look, this is whole loop is not required. I've replaced it with an assert, making sure the register we use for renaming is not trashed.

2047

Nope!

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

2

Ah yes! I'll drop the line, I've added the checks manually with additional comments.

11

Ah yes, I've adjusted the tests to not rely on IR for aliasing.

[fhahn]

Updated to use iterators proposed in dependent patches, remove some callback functions.

[fhahn]

Drop unnecessary include.

[fhahn]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

835

I've added sub_and_superregs_inclusive and phys_regs_and_masks iterators in a dependent patches.

[fhahn]

ping

@paquette are the refactors in line with what you envisioned?

[paquette]

LGTM!

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

808

Can this fit on one line if you use auto instead of MachineBasicBlock::reverse_iterator?

[phosek]

We're seeing an assertion error in Clang when compiling compiler-rt builtins for aarch64-linux-gnu with this change:

FAILED: CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o 
/b/s/w/ir/k/recipe_cleanup/clanga0lNQB/llvm_build_dir/./bin/clang --target=aarch64-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-arm64 -DVISIBILITY_HIDDEN  -O2 -g -DNDEBUG    -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o -MF CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o.d -o CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o   -c /b/s/w/ir/k/llvm-project/compiler-rt/lib/builtins/multc3.c
clang-10: /b/s/w/ir/k/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:677: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo *): Assertion `Reg && "Null register has no regunits"' failed.
clang-10: error: unable to execute command: Aborted
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)

[fhahn]

In D70450#1779756, @phosek wrote:

We're seeing an assertion error in Clang when compiling compiler-rt builtins for aarch64-linux-gnu with this change:

FAILED: CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o 
/b/s/w/ir/k/recipe_cleanup/clanga0lNQB/llvm_build_dir/./bin/clang --target=aarch64-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-arm64 -DVISIBILITY_HIDDEN  -O2 -g -DNDEBUG    -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o -MF CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o.d -o CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o   -c /b/s/w/ir/k/llvm-project/compiler-rt/lib/builtins/multc3.c
clang-10: /b/s/w/ir/k/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:677: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo *): Assertion `Reg && "Null register has no regunits"' failed.
clang-10: error: unable to execute command: Aborted
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)

Looks like the patch missed to skip debug operands in an assertion. I will push a fix shortly.

[fhahn]

In D70450#1779886, @fhahn wrote:

In D70450#1779756, @phosek wrote:

We're seeing an assertion error in Clang when compiling compiler-rt builtins for aarch64-linux-gnu with this change:

FAILED: CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o 
/b/s/w/ir/k/recipe_cleanup/clanga0lNQB/llvm_build_dir/./bin/clang --target=aarch64-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-arm64 -DVISIBILITY_HIDDEN  -O2 -g -DNDEBUG    -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o -MF CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o.d -o CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o   -c /b/s/w/ir/k/llvm-project/compiler-rt/lib/builtins/multc3.c
clang-10: /b/s/w/ir/k/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:677: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo *): Assertion `Reg && "Null register has no regunits"' failed.
clang-10: error: unable to execute command: Aborted
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)

Looks like the patch missed to skip debug operands in an assertion. I will push a fix shortly.

Should be fixed by rG4fe92abceb9a

[mstorsjo]

In D70450#1779903, @fhahn wrote:

In D70450#1779886, @fhahn wrote:

In D70450#1779756, @phosek wrote:

We're seeing an assertion error in Clang when compiling compiler-rt builtins for aarch64-linux-gnu with this change:

FAILED: CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o 
/b/s/w/ir/k/recipe_cleanup/clanga0lNQB/llvm_build_dir/./bin/clang --target=aarch64-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-arm64 -DVISIBILITY_HIDDEN  -O2 -g -DNDEBUG    -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o -MF CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o.d -o CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o   -c /b/s/w/ir/k/llvm-project/compiler-rt/lib/builtins/multc3.c
clang-10: /b/s/w/ir/k/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:677: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo *): Assertion `Reg && "Null register has no regunits"' failed.
clang-10: error: unable to execute command: Aborted
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)

Looks like the patch missed to skip debug operands in an assertion. I will push a fix shortly.

Should be fixed by rG4fe92abceb9a

Still broken for me; with https://martin.st/temp/mingw_pformatw.c, built with clang -target aarch64-w64-mingw32 -c -O2 -g, I'm still getting the same error.

[fhahn]

In D70450#1780195, @mstorsjo wrote:

In D70450#1779903, @fhahn wrote:

In D70450#1779886, @fhahn wrote:

In D70450#1779756, @phosek wrote:

We're seeing an assertion error in Clang when compiling compiler-rt builtins for aarch64-linux-gnu with this change:

FAILED: CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o 
/b/s/w/ir/k/recipe_cleanup/clanga0lNQB/llvm_build_dir/./bin/clang --target=aarch64-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-arm64 -DVISIBILITY_HIDDEN  -O2 -g -DNDEBUG    -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o -MF CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o.d -o CMakeFiles/clang_rt.builtins-aarch64.dir/multc3.c.o   -c /b/s/w/ir/k/llvm-project/compiler-rt/lib/builtins/multc3.c
clang-10: /b/s/w/ir/k/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:677: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo *): Assertion `Reg && "Null register has no regunits"' failed.
clang-10: error: unable to execute command: Aborted
clang-10: error: clang frontend command failed due to signal (use -v to see invocation)

Looks like the patch missed to skip debug operands in an assertion. I will push a fix shortly.

Should be fixed by rG4fe92abceb9a

Still broken for me; with https://martin.st/temp/mingw_pformatw.c, built with clang -target aarch64-w64-mingw32 -c -O2 -g, I'm still getting the same error.

Yeah, there were some cases with non-debug $noreg operands. Fixed by rG2675a3c8806a

[mstorsjo]

In D70450#1780211, @fhahn wrote:

Yeah, there were some cases with non-debug $noreg operands. Fixed by rG2675a3c8806a

Thanks! Now it seems to work fine (so far) for me too.

[dancgr]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

897

This creates a warning when compiling in debug mode for me.

Did you mean to use this?

assert(MOP.isImplicit() ||
                       (MOP.isRenamable() && !MOP.isEarlyClobber()) &&
                           "Need renamable operands");

It looks like this code is missing brackets wrapping the logical condition, like the ones from from line 886 assertion.

[fhahn]

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

897

thanks for the note, it looks like someone just fixed that in commit dc03b960d0236c18ed4382cea7bb3eb49329ed04

[phosek]

This change seems to have broken -ffixed-xX, we've noticed that our kernel started crashing with tip-of-tree Clang. After debugging, we found out that compiler would generate stores to x15 even though we compile with -ffixed-x15 which breaks the kernel because x15 is used to store per-cpu state. Would it be possible to revert or fix this issue ASAP please?

[phosek]

In D70450#1793699, @phosek wrote:

This change seems to have broken -ffixed-xX, we've noticed that our kernel started crashing with tip-of-tree Clang. After debugging, we found out that compiler would generate stores to x15 even though we compile with -ffixed-x15 which breaks the kernel because x15 is used to store per-cpu state. Would it be possible to revert or fix this issue ASAP please?

I've filed PR44358 to track this and I've also included a reproducer in that bug.

[hans]

In D70450#1793700, @phosek wrote:

In D70450#1793699, @phosek wrote:

This change seems to have broken -ffixed-xX, we've noticed that our kernel started crashing with tip-of-tree Clang. After debugging, we found out that compiler would generate stores to x15 even though we compile with -ffixed-x15 which breaks the kernel because x15 is used to store per-cpu state. Would it be possible to revert or fix this issue ASAP please?

I've filed PR44358 to track this and I've also included a reproducer in that bug.

In addition to that bug, which was fixed in d269255b95151dcd232c8856206b7e79d70eda29, we've bisected another miscompile in Chromium (on Android) to this revision. We don't have a reproducer yet, but figured we should send a heads up in case anyone else was seeing problems too.

The Chromium bug is here: https://bugs.chromium.org/p/chromium/issues/detail?id=1037912

[fhahn]

In D70450#1831948, @hans wrote:

In D70450#1793700, @phosek wrote:

In D70450#1793699, @phosek wrote:

This change seems to have broken -ffixed-xX, we've noticed that our kernel started crashing with tip-of-tree Clang. After debugging, we found out that compiler would generate stores to x15 even though we compile with -ffixed-x15 which breaks the kernel because x15 is used to store per-cpu state. Would it be possible to revert or fix this issue ASAP please?

I've filed PR44358 to track this and I've also included a reproducer in that bug.

In addition to that bug, which was fixed in d269255b95151dcd232c8856206b7e79d70eda29, we've bisected another miscompile in Chromium (on Android) to this revision. We don't have a reproducer yet, but figured we should send a heads up in case anyone else was seeing problems too.

The Chromium bug is here: https://bugs.chromium.org/p/chromium/issues/detail?id=1037912

Thanks for sharing that (and sorry for any inconvenience caused). I'm looking forward to the reproducer. Please let me know if there's anything I can do to help.

[hans]

Thanks for sharing that (and sorry for any inconvenience caused). I'm looking forward to the reproducer. Please let me know if there's anything I can do to help.

We now have a diff of the code before and after this patch. I'm not that familiar with aarch64. Could you take a look and see if this makes sense: https://bugs.chromium.org/p/chromium/issues/detail?id=1037912#c70 (the full Machine IR dumps are attached to the comment before)

[fhahn]

In D70450#1832697, @hans wrote:

Thanks for sharing that (and sorry for any inconvenience caused). I'm looking forward to the reproducer. Please let me know if there's anything I can do to help.

We now have a diff of the code before and after this patch. I'm not that familiar with aarch64. Could you take a look and see if this makes sense: https://bugs.chromium.org/p/chromium/issues/detail?id=1037912#c70 (the full Machine IR dumps are attached to the comment before)

That's great thanks! I think I know what's going on (I'll post here as I don't have a sign-in for bugs.chromium.org).

In the problematic case it looks like we turn

renamable $w8 = KILL killed renamable $w8, implicit-def $x8
STURXi killed renamable $x8, $fp, -40 :: (store 8 into %stack.5)
$w8 = ORRWrs $wzr, killed $w25, 0, implicit-def $x8
STURXi killed renamable $x8, $fp, -32 :: (store 8 into %stack.4)

into

$w9 = KILL killed renamable $w8, implicit-def $x9
$w8 = ORRWrs $wzr, killed $w25, 0, implicit-def $x8
STPXi killed $x9, killed renamable $x8, $fp, -5 :: (store 8 into %stack.5), (store 8 into %stack.4)

On the MIR level, that seems fine, but the problem is that KILL is a noop, so it won't get lowered to any instructions, so $w9 will not contain the correct value. (The changed -5 offset should be fine, as the 64 bit variant of STP uses multiples of 8)

IIUC there are multiple ways to fix this:

1. Change the operand flags on KILL. It seems like even though $w8 is marked as renamable we cannot freely rename the result operand. Unless I am missing something, all operands should probably tied.
2. Add an additional copy to ensure we actually do the renaming.

If marking the operands of KILL as tied makes sense, I think that's what we should go for, as it is more general.

[hans]

IIUC there are multiple ways to fix this:

1. Change the operand flags on KILL. It seems like even though $w8 is marked as renamable we cannot freely rename the result operand. Unless I am missing something, all operands should probably tied.
2. Add an additional copy to ensure we actually do the renaming.

If marking the operands of KILL as tied makes sense, I think that's what we should go for, as it is more general.

Do you think this fix can happen quickly? Otherwise we'd like to revert this to unbreak the compiler as soon as possible.

[fhahn]

In D70450#1834132, @hans wrote:

IIUC there are multiple ways to fix this:

1. Change the operand flags on KILL. It seems like even though $w8 is marked as renamable we cannot freely rename the result operand. Unless I am missing something, all operands should probably tied.
2. Add an additional copy to ensure we actually do the renaming.

If marking the operands of KILL as tied makes sense, I think that's what we should go for, as it is more general.

Do you think this fix can happen quickly? Otherwise we'd like to revert this to unbreak the compiler as soon as possible.

I think both approaches mentioned above will require additional discussion/feedback. I've updated canRenameUpToDef to bail out on any defs that are pseudo instructions for now in 300997c41a00b705ca10264c15910dd8d691ab75 . That should fix the issue.

[fhahn]

@hans it would be great if you could confirm this fixes the issues you are seeing. If that's the case, I'll create an issue to back-port this to the 10.0 release branch.

[hans]

In D70450#1834238, @fhahn wrote:

@hans it would be great if you could confirm this fixes the issues you are seeing. If that's the case, I'll create an issue to back-port this to the 10.0 release branch.

Verified, thanks! I'll go ahead and cherry-pick it.

[hans]

In D70450#1834319, @hans wrote:

In D70450#1834238, @fhahn wrote:

@hans it would be great if you could confirm this fixes the issues you are seeing. If that's the case, I'll create an issue to back-port this to the 10.0 release branch.

Verified, thanks! I'll go ahead and cherry-pick it.

Pushed to the 10.x branch in 050e1a3c2688f2daf03456cf94dd3ed79e8ebe7f

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/DebugCounter.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <cstdint>
+#include <functional>
 #include <iterator>
 #include <limits>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-ldst-opt"
 
 STATISTIC(NumPairCreated, "Number of load/store pair instructions generated");
 STATISTIC(NumPostFolded, "Number of post-index updates folded");
 STATISTIC(NumPreFolded, "Number of pre-index updates folded");
 STATISTIC(NumUnscaledPairCreated,
           "Number of load/store from unscaled generated");
 STATISTIC(NumZeroStoresPromoted, "Number of narrow zero stores promoted");
 STATISTIC(NumLoadsFromStoresPromoted, "Number of loads from stores promoted");
 
+DEBUG_COUNTER(RegRenamingCounter, DEBUG_TYPE "-reg-renaming",
+              "Controls which pairs are considered for renaming");
+
 // The LdStLimit limits how far we search for load/store pairs.
 static cl::opt<unsigned> LdStLimit("aarch64-load-store-scan-limit",
                                    cl::init(20), cl::Hidden);
 
 // The UpdateLimit limits how far we search for update instructions when we form
 // pre-/post-index instructions.
 static cl::opt<unsigned> UpdateLimit("aarch64-update-scan-limit", cl::init(100),
                                      cl::Hidden);
using LdStPairFlags = struct LdStPairFlags {
   bool MergeForward = false;
 
   // SExtIdx gives the index of the result of the load pair that must be
   // extended. The value of SExtIdx assumes that the paired load produces the
   // value in this order: (I, returned iterator), i.e., -1 means no value has
   // to be extended, 0 means I, and 1 means the returned iterator.
   int SExtIdx = -1;
 
+  // If not none, RenameReg can be used to rename the result register of the
+  // first store in a pair. Currently this only works when merging stores
+  // forward.
+  Optional<MCPhysReg> RenameReg = None;
+
   LdStPairFlags() = default;
 
   void setMergeForward(bool V = true) { MergeForward = V; }
   bool getMergeForward() const { return MergeForward; }
 
   void setSExtIdx(int V) { SExtIdx = V; }
   int getSExtIdx() const { return SExtIdx; }
+
+  void setRenameReg(MCPhysReg R) { RenameReg = R; }
+  void clearRenameReg() { RenameReg = None; }
+  Optional<MCPhysReg> getRenameReg() const { return RenameReg; }
 };
 
 struct AArch64LoadStoreOpt : public MachineFunctionPass {
   static char ID;
 
   AArch64LoadStoreOpt() : MachineFunctionPass(ID) {
     initializeAArch64LoadStoreOptPass(*PassRegistry::getPassRegistry());
   }
 
   AliasAnalysis *AA;
   const AArch64InstrInfo *TII;
   const TargetRegisterInfo *TRI;
   const AArch64Subtarget *Subtarget;
 
   // Track which register units have been modified and used.
   LiveRegUnits ModifiedRegUnits, UsedRegUnits;
+  LiveRegUnits DefinedInBB;
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AAResultsWrapperPass>();
     MachineFunctionPass::getAnalysisUsage(AU);
   }
 
   // Scan the instructions looking for a load/store that can be combined
   // with the current instruction into a load/store pair.
static bool isPairedLdSt(const MachineInstr &MI) {
   case AArch64::STPQi:
   case AArch64::STPWi:
   case AArch64::STPXi:
   case AArch64::STGPi:
     return true;
   }
 }
 
 // Returns the scale and offset range of pre/post indexed variants of MI.
 static void getPrePostIndexedMemOpInfo(const MachineInstr &MI, int &Scale,
                                        int &MinOffset, int &MaxOffset) {
   bool IsPaired = isPairedLdSt(MI);
   bool IsTagStore = isTagStore(MI);
   // ST*G and all paired ldst have the same scale in pre/post-indexed variants
   // as in the "unsigned offset" variant.
   // All other pre/post indexed ldst instructions are unscaled.
   Scale = (IsTagStore || IsPaired) ? getMemScale(MI) : 1;
 
   if (IsPaired) {
     MinOffset = -64;
     MaxOffset = 63;
   } else {
     MinOffset = -256;
     MaxOffset = 255;
   }
 }
 
-static const MachineOperand &getLdStRegOp(const MachineInstr &MI,
-                                          unsigned PairedRegOp = 0) {
+static MachineOperand &getLdStRegOp(MachineInstr &MI,
   assert(PairedRegOp < 2 && "Unexpected register operand idx.");
   unsigned Idx = isPairedLdSt(MI) ? PairedRegOp : 0;
   return MI.getOperand(Idx);
 }
 
 static const MachineOperand &getLdStBaseOp(const MachineInstr &MI) {
   unsigned Idx = isPairedLdSt(MI) ? 2 : 1;
   return MI.getOperand(Idx);
AArch64LoadStoreOpt::mergeNarrowZeroStores(MachineBasicBlock::iterator I,
   LLVM_DEBUG(dbgs() << "\n");
 
   // Erase the old instructions.
   I->eraseFromParent();
   MergeMI->eraseFromParent();
   return NextI;
 }
 
+// Apply Fn to all instructions between MI and the beginning of the block, until
+// a def for DefReg is reached. Returns true, iff Fn returns true for all
+// visited instructions. Stop after visiting Limit iterations.
+static bool forAllMIsUntilDef(MachineInstr &MI, MCPhysReg DefReg,
+                              const TargetRegisterInfo *TRI, unsigned Limit,
+                              std::function<bool(MachineInstr &, bool)> &Fn) {
+  auto MBB = MI.getParent();
+  for (MachineBasicBlock::reverse_iterator I = MI.getReverseIterator(),
+                                           E = MBB->rend();
+       I != E; I++) {
+    if (!Limit)
+      return false;
+    --Limit;
+
+    bool isDef = any_of(I->operands(), [DefReg, TRI](MachineOperand &MOP) {
+      return MOP.isReg() && MOP.isDef() &&
+             TRI->regsOverlap(MOP.getReg(), DefReg);
+    });
+    if (!Fn(*I, isDef))
+      return false;
+    if (isDef)
+      break;
+  }
+  return true;
+}
+
+static void updateDefinedRegisters(MachineInstr &MI, LiveRegUnits &Units,
+                                   const TargetRegisterInfo *TRI) {
+
+  for (const MachineOperand &MOP : phys_regs_and_masks(MI))
+    if (MOP.isReg() && MOP.isKill())
+      Units.removeReg(MOP.getReg());
+
+  for (const MachineOperand &MOP : phys_regs_and_masks(MI))
+    if (MOP.isReg() && !MOP.isKill())
+      Units.addReg(MOP.getReg());
+}
+
 MachineBasicBlock::iterator
 AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
                                       MachineBasicBlock::iterator Paired,
                                       const LdStPairFlags &Flags) {
   MachineBasicBlock::iterator NextI = I;
   ++NextI;
   // If NextI is the second of the two instructions to be merged, we need
   // to skip one further. Either way we merge will invalidate the iterator,
   // and we don't need to scan the new instruction, as it's a pairwise
   // instruction, which we're not considering for further action anyway.
   if (NextI == Paired)
     ++NextI;
 
   int SExtIdx = Flags.getSExtIdx();
   unsigned Opc =
       SExtIdx == -1 ? I->getOpcode() : getMatchingNonSExtOpcode(I->getOpcode());
   bool IsUnscaled = TII->isUnscaledLdSt(Opc);
   int OffsetStride = IsUnscaled ? getMemScale(*I) : 1;
 
   bool MergeForward = Flags.getMergeForward();
+
+  Optional<MCPhysReg> RenameReg = Flags.getRenameReg();
+  if (MergeForward && RenameReg) {
+    MCRegister RegToRename = getLdStRegOp(*I).getReg();
+    DefinedInBB.addReg(*RenameReg);
+
+    // Return the sub/super register for RenameReg, matching the size of
+    // OriginalReg.
+    auto GetMatchingSubReg = [this,
+                              RenameReg](MCPhysReg OriginalReg) -> MCPhysReg {
+      for (MCPhysReg SubOrSuper : TRI->sub_and_superregs_inclusive(*RenameReg))
+        if (TRI->getMinimalPhysRegClass(OriginalReg) ==
+            TRI->getMinimalPhysRegClass(SubOrSuper))
+          return SubOrSuper;
+      llvm_unreachable("Should have found matching sub or super register!");
+    };
+
+    std::function<bool(MachineInstr &, bool)> UpdateMIs =
+        [this, RegToRename, GetMatchingSubReg](MachineInstr &MI, bool IsDef) {
+          if (IsDef) {
+            bool SeenDef = false;
+            for (auto &MOP : MI.operands()) {
+              // Rename the first explicit definition and all implicit
+              // definitions matching RegToRename.
+              if (MOP.isReg() &&
+                  (!SeenDef || (MOP.isDef() && MOP.isImplicit())) &&
+                  TRI->regsOverlap(MOP.getReg(), RegToRename)) {
+                assert((MOP.isImplicit() ||
+                        (MOP.isRenamable() && !MOP.isEarlyClobber())) &&
+                       "Need renamable operands");
+                MOP.setReg(GetMatchingSubReg(MOP.getReg()));
+                SeenDef = true;
+              }
+            }
+          } else {
+            for (auto &MOP : MI.operands()) {
+              if (MOP.isReg() && TRI->regsOverlap(MOP.getReg(), RegToRename)) {
+                assert(MOP.isImplicit() ||
+                       (MOP.isRenamable() && !MOP.isEarlyClobber()) &&
+                           "Need renamable operands");
+                MOP.setReg(GetMatchingSubReg(MOP.getReg()));
+              }
+            }
+          }
+          LLVM_DEBUG(dbgs() << "Renamed " << MI << "\n");
+          return true;
+        };
+    forAllMIsUntilDef(*I, RegToRename, TRI, LdStLimit, UpdateMIs);
+
+    // Make sure the register used for renaming is not used between the paired
+    // instructions. That would trash the content before the new paired
+    // instruction.
+    for (auto &MI :
+         iterator_range<MachineInstrBundleIterator<llvm::MachineInstr>>(
+             std::next(I), std::next(Paired)))
+      assert(all_of(MI.operands(),
+                    [this, &RenameReg](const MachineOperand &MOP) {
+                      return !MOP.isReg() ||
+                             !TRI->regsOverlap(MOP.getReg(), *RenameReg);
+                    }) &&
+             "Rename register used between paired instruction, trashing the "
+             "content");
+  }
+
   // Insert our new paired instruction after whichever of the paired
   // instructions MergeForward indicates.
   MachineBasicBlock::iterator InsertionPoint = MergeForward ? Paired : I;
   // Also based on MergeForward is from where we copy the base register operand
   // so we get the flags compatible with the input code.
   const MachineOperand &BaseRegOp =
       MergeForward ? getLdStBaseOp(*Paired) : getLdStBaseOp(*I);
 
   int Offset = getLdStOffsetOp(*I).getImm();
   int PairedOffset = getLdStOffsetOp(*Paired).getImm();
   bool PairedIsUnscaled = TII->isUnscaledLdSt(Paired->getOpcode());
   if (IsUnscaled != PairedIsUnscaled) {
     // We're trying to pair instructions that differ in how they are scaled.  If
if (SExtIdx != -1) {
     (void)MIBSXTW;
     LLVM_DEBUG(dbgs() << "  Extend operand:\n    ");
     LLVM_DEBUG(((MachineInstr *)MIBSXTW)->print(dbgs()));
   } else {
     LLVM_DEBUG(((MachineInstr *)MIB)->print(dbgs()));
   }
   LLVM_DEBUG(dbgs() << "\n");
 
+  if (MergeForward)
+    for (const MachineOperand &MOP : phys_regs_and_masks(*I))
+      if (MOP.isReg() && MOP.isKill())
+        DefinedInBB.addReg(MOP.getReg());
+
   // Erase the old instructions.
   I->eraseFromParent();
   Paired->eraseFromParent();
 
   return NextI;
 }
 
 MachineBasicBlock::iterator
static bool areCandidatesToMergeOrPair(MachineInstr &FirstMI, MachineInstr &MI,
 
   // Try to match an unscaled load/store with a scaled load/store.
   return TII->isUnscaledLdSt(OpcA) != TII->isUnscaledLdSt(OpcB) &&
          getMatchingPairOpcode(OpcA) == getMatchingPairOpcode(OpcB);
 
   // FIXME: Can we also match a mixed sext/zext unscaled/scaled pair?
 }
 
+static bool
+canRenameUpToDef(MachineInstr &FirstMI, LiveRegUnits &UsedInBetween,
+                 SmallPtrSetImpl<const TargetRegisterClass *> &RequiredClasses,
+                 const TargetRegisterInfo *TRI) {
+  if (!FirstMI.mayStore())
+    return false;
+
+  // Check if we can find an unused register which we can use to rename
+  // the register used by the first load/store.
+  auto *RegClass = TRI->getMinimalPhysRegClass(getLdStRegOp(FirstMI).getReg());
+  MachineFunction &MF = *FirstMI.getParent()->getParent();
+  if (!RegClass || !MF.getRegInfo().tracksLiveness())
+    return false;
+
+  auto RegToRename = getLdStRegOp(FirstMI).getReg();
+  // For now, we only rename if the store operand gets killed at the store.
+  if (!getLdStRegOp(FirstMI).isKill() &&
+      !any_of(FirstMI.operands(),
+              [TRI, RegToRename](const MachineOperand &MOP) {
+                return MOP.isReg() && MOP.isImplicit() && MOP.isKill() &&
+                       TRI->regsOverlap(RegToRename, MOP.getReg());
+              })) {
+    LLVM_DEBUG(dbgs() << "  Operand not killed at " << FirstMI << "\n");
+    return false;
+  }
+  auto canRenameMOP = [](const MachineOperand &MOP) {
+    return MOP.isImplicit() ||
+           (MOP.isRenamable() && !MOP.isEarlyClobber() && !MOP.isTied());
+  };
+
+  bool FoundDef = false;
+
+  // For each instruction between FirstMI and the previous def for RegToRename,
+  // we
+  // * check if we can rename RegToRename in this instruction
+  // * collect the registers used and required register classes for RegToRename.
+  std::function<bool(MachineInstr &, bool)> CheckMIs = [&](MachineInstr &MI,
+                                                           bool IsDef) {
+    LLVM_DEBUG(dbgs() << "Checking " << MI << "\n");
+    // Currently we do not try to rename across frame-setup instructions.
+    if (MI.getFlag(MachineInstr::FrameSetup)) {
+      LLVM_DEBUG(dbgs() << "  Cannot rename framesetup instructions currently ("
+                        << MI << ")\n");
+      return false;
+    }
+
+    UsedInBetween.accumulate(MI);
+
+    // For a definition, check that we can rename the definition and exit the
+    // loop.
+    FoundDef = IsDef;
+
+    // For defs, check if we can rename the first def of RegToRename.
+    if (FoundDef) {
+      for (auto &MOP : MI.operands()) {
+        if (!MOP.isReg() || !MOP.isDef() ||
+            !TRI->regsOverlap(MOP.getReg(), RegToRename))
+          continue;
+        if (!canRenameMOP(MOP)) {
+          LLVM_DEBUG(dbgs()
+                     << "  Cannot rename " << MOP << " in " << MI << "\n");
+          return false;
+        }
+        RequiredClasses.insert(TRI->getMinimalPhysRegClass(MOP.getReg()));
+      }
+      return true;
+    } else {
+      for (auto &MOP : MI.operands()) {
+        if (!MOP.isReg() || !TRI->regsOverlap(MOP.getReg(), RegToRename))
+          continue;
+
+        if (!canRenameMOP(MOP)) {
+          LLVM_DEBUG(dbgs()
+                     << "  Cannot rename " << MOP << " in " << MI << "\n");
+          return false;
+        }
+        RequiredClasses.insert(TRI->getMinimalPhysRegClass(MOP.getReg()));
+      }
+    }
+    return true;
+  };
+
+  if (!forAllMIsUntilDef(FirstMI, RegToRename, TRI, LdStLimit, CheckMIs))
+    return false;
+
+  if (!FoundDef) {
+    LLVM_DEBUG(dbgs() << "  Did not find definition for register in BB\n");
+    return false;
+  }
+  return true;
+}
+
+// Check if we can find a physical register for renaming. This register must:
+// * not be defined up to FirstMI (checking DefinedInBB)
+// * not used between the MI and the defining instruction of the register to
+//   rename (checked using UsedInBetween).
+// * is available in all used register classes (checked using RequiredClasses).
+static Optional<MCPhysReg> tryToFindRegisterToRename(
+    MachineInstr &FirstMI, MachineInstr &MI, LiveRegUnits &DefinedInBB,
+    LiveRegUnits &UsedInBetween,
+    SmallPtrSetImpl<const TargetRegisterClass *> &RequiredClasses,
+    const TargetRegisterInfo *TRI) {
+  auto &MF = *FirstMI.getParent()->getParent();
+
+  // Checks if any sub- or super-register of PR is callee saved.
+  auto AnySubOrSuperRegCalleePreserved = [&MF, TRI](MCPhysReg PR) {
+    return any_of(TRI->sub_and_superregs_inclusive(PR),
+                  [&MF, TRI](MCPhysReg SubOrSuper) {
+                    return TRI->isCalleeSavedPhysReg(SubOrSuper, MF);
+                  });
+  };
+
+  // Check if PR or one of its sub- or super-registers can be used for all
+  // required register classes.
+  auto CanBeUsedForAllClasses = [&RequiredClasses, TRI](MCPhysReg PR) {
+    return all_of(RequiredClasses, [PR, TRI](const TargetRegisterClass *C) {
+      return any_of(TRI->sub_and_superregs_inclusive(PR),
+                    [C, TRI](MCPhysReg SubOrSuper) {
+                      return C == TRI->getMinimalPhysRegClass(SubOrSuper);
+                    });
+    });
+  };
+
+  auto *RegClass = TRI->getMinimalPhysRegClass(getLdStRegOp(FirstMI).getReg());
+  for (const MCPhysReg &PR : *RegClass) {
+    if (DefinedInBB.available(PR) && UsedInBetween.available(PR) &&
+        !AnySubOrSuperRegCalleePreserved(PR) && CanBeUsedForAllClasses(PR)) {
+      DefinedInBB.addReg(PR);
+      LLVM_DEBUG(dbgs() << "Found rename register " << printReg(PR, TRI)
+                        << "\n");
+      return {PR};
+    }
+  }
+  LLVM_DEBUG(dbgs() << "No rename register found from "
+                    << TRI->getRegClassName(RegClass) << "\n");
+  return None;
+}
+
 /// Scan the instructions looking for a load/store that can be combined with the
 /// current instruction into a wider equivalent or a load/store pair.
 MachineBasicBlock::iterator
 AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
                                       LdStPairFlags &Flags, unsigned Limit,
                                       bool FindNarrowMerge) {
   MachineBasicBlock::iterator E = I->getParent()->end();
   MachineBasicBlock::iterator MBBI = I;
+  MachineBasicBlock::iterator MBBIWithRenameReg;
   MachineInstr &FirstMI = *I;
   ++MBBI;
 
   bool MayLoad = FirstMI.mayLoad();
   bool IsUnscaled = TII->isUnscaledLdSt(FirstMI);
   Register Reg = getLdStRegOp(FirstMI).getReg();
   Register BaseReg = getLdStBaseOp(FirstMI).getReg();
   int Offset = getLdStOffsetOp(FirstMI).getImm();
   int OffsetStride = IsUnscaled ? getMemScale(FirstMI) : 1;
   bool IsPromotableZeroStore = isPromotableZeroStoreInst(FirstMI);
 
+  Optional<bool> MaybeCanRename = None;
+  SmallPtrSet<const TargetRegisterClass *, 5> RequiredClasses;
+  LiveRegUnits UsedInBetween;
+  UsedInBetween.init(*TRI);
+
+  Flags.clearRenameReg();
+
   // Track which register units have been modified and used between the first
   // insn (inclusive) and the second insn.
   ModifiedRegUnits.clear();
   UsedRegUnits.clear();
 
   // Remember any instructions that read/write memory between FirstMI and MI.
   SmallVector<MachineInstr *, 4> MemInsns;
 
   for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
     MachineInstr &MI = *MBBI;
 
+    UsedInBetween.accumulate(MI);
+
     // Don't count transient instructions towards the search limit since there
     // may be different numbers of them if e.g. debug information is present.
     if (!MI.isTransient())
       ++Count;
 
     Flags.setSExtIdx(-1);
     if (areCandidatesToMergeOrPair(FirstMI, MI, Flags, TII) &&
         getLdStOffsetOp(MI).isImm()) {
if (areCandidatesToMergeOrPair(FirstMI, MI, Flags, TII) &&
         // If the Rt of the second instruction was not modified or used between
         // the two instructions and none of the instructions between the second
         // and first alias with the second, we can combine the second into the
         // first.
         if (ModifiedRegUnits.available(getLdStRegOp(MI).getReg()) &&
             !(MI.mayLoad() &&
               !UsedRegUnits.available(getLdStRegOp(MI).getReg())) &&
             !mayAlias(MI, MemInsns, AA)) {
+
           Flags.setMergeForward(false);
+          Flags.clearRenameReg();
           return MBBI;
         }
 
         // Likewise, if the Rt of the first instruction is not modified or used
         // between the two instructions and none of the instructions between the
         // first and the second alias with the first, we can combine the first
         // into the second.
-        if (ModifiedRegUnits.available(getLdStRegOp(FirstMI).getReg()) &&
-            !(MayLoad &&
+        if (!(MayLoad &&
               !UsedRegUnits.available(getLdStRegOp(FirstMI).getReg())) &&
             !mayAlias(FirstMI, MemInsns, AA)) {
+
+          if (ModifiedRegUnits.available(getLdStRegOp(FirstMI).getReg())) {
             Flags.setMergeForward(true);
+            Flags.clearRenameReg();
             return MBBI;
           }
+
+          if (DebugCounter::shouldExecute(RegRenamingCounter)) {
+            if (!MaybeCanRename)
+              MaybeCanRename = {canRenameUpToDef(FirstMI, UsedInBetween,
+                                                 RequiredClasses, TRI)};
+
+            if (*MaybeCanRename) {
+              Optional<MCPhysReg> MaybeRenameReg = tryToFindRegisterToRename(
+                  FirstMI, MI, DefinedInBB, UsedInBetween, RequiredClasses,
+                  TRI);
+              if (MaybeRenameReg) {
+                Flags.setRenameReg(*MaybeRenameReg);
+                Flags.setMergeForward(true);
+                MBBIWithRenameReg = MBBI;
+              }
+            }
+          }
+        }
         // Unable to combine these instructions due to interference in between.
         // Keep looking.
       }
     }
 
+    if (Flags.getRenameReg())
+      return MBBIWithRenameReg;
+
     // If the instruction wasn't a matching load or store.  Stop searching if we
     // encounter a call instruction that might modify memory.
     if (MI.isCall())
       return E;
 
     // Update modified / uses register units.
     LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits, UsedRegUnits, TRI);
 
bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
   MachineBasicBlock::iterator Paired =
       findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ false);
   if (Paired != E) {
     ++NumPairCreated;
     if (TII->isUnscaledLdSt(MI))
       ++NumUnscaledPairCreated;
     // Keeping the iterator straight is a pain, so we let the merge routine tell
     // us what the next instruction is after it's done mucking about.
+    auto Prev = std::prev(MBBI);
     MBBI = mergePairedInsns(MBBI, Paired, Flags);
+    // Collect liveness info for instructions between Prev and the new position
+    // MBBI.
+    for (auto I = std::next(Prev); I != MBBI; I++)
+      updateDefinedRegisters(*I, DefinedInBB, TRI);
+
     return true;
   }
   return false;
 }
 
 bool AArch64LoadStoreOpt::tryToMergeLdStUpdate
     (MachineBasicBlock::iterator &MBBI) {
   MachineInstr &MI = *MBBI;
if (Update != E) {
     return true;
   }
 
   return false;
 }
 
 bool AArch64LoadStoreOpt::optimizeBlock(MachineBasicBlock &MBB,
                                         bool EnableNarrowZeroStOpt) {
+
   bool Modified = false;
   // Four tranformations to do here:
   // 1) Find loads that directly read from stores and promote them by
   //    replacing with mov instructions. If the store is wider than the load,
   //    the load will be replaced with a bitfield extract.
   //      e.g.,
   //        str w1, [x0, #4]
   //        ldrh w2, [x0, #6]
   //        ; becomes
if (EnableNarrowZeroStOpt)
     }
   // 3) Find loads and stores that can be merged into a single load or store
   //    pair instruction.
   //      e.g.,
   //        ldr x0, [x2]
   //        ldr x1, [x2, #8]
   //        ; becomes
   //        ldp x0, x1, [x2]
+
+  if (MBB.getParent()->getRegInfo().tracksLiveness()) {
+    DefinedInBB.clear();
+    DefinedInBB.addLiveIns(MBB);
+  }
+
   for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
        MBBI != E;) {
+    // Track currently live registers up to this point, to help with
+    // searching for a rename register on demand.
+    updateDefinedRegisters(*MBBI, DefinedInBB, TRI);
     if (TII->isPairableLdStInst(*MBBI) && tryToPairLdStInst(MBBI))
       Modified = true;
     else
       ++MBBI;
   }
   // 4) Find base register updates that can be merged into the load or store
   //    as a base-reg writeback.
   //      e.g.,
   //        ldr x0, [x2]
   //        add x2, x2, #4
   //        ; becomes
   //        ldr x0, [x2], #4
bool AArch64LoadStoreOpt::runOnMachineFunction(MachineFunction &Fn) {
   TRI = Subtarget->getRegisterInfo();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
 
   // Resize the modified and used register unit trackers.  We do this once
   // per function and then clear the register units each time we optimize a load
   // or store.
   ModifiedRegUnits.init(*TRI);
   UsedRegUnits.init(*TRI);
+  DefinedInBB.init(*TRI);
 
   bool Modified = false;
   bool enableNarrowZeroStOpt = !Subtarget->requiresStrictAlign();
-  for (auto &MBB : Fn)
-    Modified |= optimizeBlock(MBB, enableNarrowZeroStOpt);
+  for (auto &MBB : Fn) {
+    auto M = optimizeBlock(MBB, enableNarrowZeroStOpt);
+    Modified |= M;
+  }
 
   return Modified;
 }
 
 // FIXME: Do we need/want a pre-alloc pass like ARM has to try to keep loads and
 // stores near one another?  Note: The pre-RA instruction scheduler already has
 // hooks to try and schedule pairable loads/stores together to improve pairing
 // opportunities.  Thus, pre-RA pairing pass may not be worth the effort.
 
 // FIXME: When pairing store instructions it's very possible for this pass to

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=arm64-apple-ios7.0.0 -mcpu=cyclone -enable-misched=false | FileCheck %s
 
 ; rdar://13625505
 ; Here we have 9 fixed integer arguments the 9th argument in on stack, the
 ; varargs start right after at 8-byte alignment.
 define void @fn9(i32* %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6, i32 %a7, i32 %a8, i32 %a9, ...) nounwind noinline ssp {
 ; CHECK-LABEL: fn9:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #64 ; =64
 ; CHECK-NEXT:    ldr w8, [sp, #64]
 ; CHECK-NEXT:    stp w2, w1, [sp, #52]
 ; CHECK-NEXT:    stp w4, w3, [sp, #44]
 ; CHECK-NEXT:    stp w6, w5, [sp, #36]
 ; CHECK-NEXT:    str w7, [sp, #32]
 ; CHECK-NEXT:    str w8, [x0]
-; CHECK-NEXT:    ldr w8, [sp, #72]
-; CHECK-NEXT:    str w8, [sp, #20]
+; CHECK-NEXT:    ldr w9, [sp, #72]
 ; CHECK-NEXT:    ldr w8, [sp, #80]
-; CHECK-NEXT:    str w8, [sp, #16]
+; CHECK-NEXT:    stp w8, w9, [sp, #16]
 ; CHECK-NEXT:    add x8, sp, #72 ; =72
 ; CHECK-NEXT:    add x8, x8, #24 ; =24
 ; CHECK-NEXT:    str x8, [sp, #24]
 ; CHECK-NEXT:    ldr w8, [sp, #88]
 ; CHECK-NEXT:    str w8, [sp, #12]
 ; CHECK-NEXT:    add sp, sp, #64 ; =64
 ; CHECK-NEXT:    ret
   %1 = alloca i32, align 4

 
 declare void @llvm.va_start(i8*) nounwind
 
 define i32 @main() nounwind ssp {
 ; CHECK-LABEL: main:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #96 ; =96
 ; CHECK-NEXT:    stp x29, x30, [sp, #80] ; 16-byte Folded Spill
-; CHECK-NEXT:    mov w8, #1
-; CHECK-NEXT:    str w8, [sp, #76]
+; CHECK-NEXT:    mov w9, #1
 ; CHECK-NEXT:    mov w8, #2
-; CHECK-NEXT:    str w8, [sp, #72]
-; CHECK-NEXT:    mov w8, #3
-; CHECK-NEXT:    str w8, [sp, #68]
+; CHECK-NEXT:    stp w8, w9, [sp, #72]
+; CHECK-NEXT:    mov w9, #3
 ; CHECK-NEXT:    mov w8, #4
-; CHECK-NEXT:    str w8, [sp, #64]
-; CHECK-NEXT:    mov w8, #5
-; CHECK-NEXT:    str w8, [sp, #60]
+; CHECK-NEXT:    stp w8, w9, [sp, #64]
+; CHECK-NEXT:    mov w9, #5
 ; CHECK-NEXT:    mov w8, #6
-; CHECK-NEXT:    str w8, [sp, #56]
-; CHECK-NEXT:    mov w8, #7
-; CHECK-NEXT:    str w8, [sp, #52]
+; CHECK-NEXT:    stp w8, w9, [sp, #56]
+; CHECK-NEXT:    mov w9, #7
 ; CHECK-NEXT:    mov w8, #8
-; CHECK-NEXT:    str w8, [sp, #48]
+; CHECK-NEXT:    stp w8, w9, [sp, #48]
 ; CHECK-NEXT:    mov w8, #9
 ; CHECK-NEXT:    mov w9, #10
 ; CHECK-NEXT:    stp w9, w8, [sp, #40]
 ; CHECK-NEXT:    mov w10, #11
 ; CHECK-NEXT:    mov w11, #12
 ; CHECK-NEXT:    stp w11, w10, [sp, #32]
 ; CHECK-NEXT:    stp x10, x11, [sp, #16]
 ; CHECK-NEXT:    str x9, [sp, #8]

llvm/test/CodeGen/AArch64/arm64-abi_align.ll

; FAST: add {{w[0-9]+}}, w[[C]], w[[B]]
   %add4 = add i32 %add3, %conv
   %add7 = add i32 %add4, %conv6
   ret i32 %add7
 }
 
 define i32 @caller43() #3 {
 entry:
 ; CHECK-LABEL: caller43
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #48]
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #32]
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #16]
-; CHECK-DAG: str {{q[0-9]+}}, [sp]
+; CHECK-DAG: stp q1, q0, [sp, #32]
+; CHECK-DAG: stp q1, q0, [sp]
 ; CHECK: add x1, sp, #32
 ; CHECK: mov x2, sp
 ; Space for s1 is allocated at sp+32
 ; Space for s2 is allocated at sp
 
 ; FAST-LABEL: caller43
 ; FAST: add x29, sp, #64
 ; Space for s1 is allocated at sp+32
declare i32 @f43_stack(i32 %i, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6,
                        i32 %i7, i32 %i8, i32 %i9, %struct.s43* nocapture %s1,
                        %struct.s43* nocapture %s2) #2
 
 define i32 @caller43_stack() #3 {
 entry:
 ; CHECK-LABEL: caller43_stack
 ; CHECK: sub sp, sp, #112
 ; CHECK: add x29, sp, #96
-; CHECK-DAG: stur {{q[0-9]+}}, [x29, #-16]
-; CHECK-DAG: stur {{q[0-9]+}}, [x29, #-32]
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #48]
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #32]
+; CHECK-DAG: stp q1, q0, [x29, #-32]
+; CHECK-DAG: stp q1, q0, [sp, #32]
 ; Space for s1 is allocated at x29-32 = sp+64
 ; Space for s2 is allocated at sp+32
 ; CHECK: add x[[B:[0-9]+]], sp, #32
 ; CHECK: str x[[B]], [sp, #16]
 ; CHECK: sub x[[A:[0-9]+]], x29, #32
 ; Address of s1 is passed on stack at sp+8
 ; CHECK: str x[[A]], [sp, #8]
 ; CHECK: mov w[[C:[0-9]+]], #9

llvm/test/CodeGen/AArch64/arm64-variadic-aapcs.ll

 ; CHECK-DAG: stp q0, q1, [sp]
 ; ... omit middle ones ...
 ; CHECK-DAG: stp q6, q7, [sp, #
 
 ; CHECK: str [[STACK_TOP]], [x[[VA_LIST]]]
 
 ; CHECK: add [[GR_TOPTMP:x[0-9]+]], sp, #[[GR_BASE]]
 ; CHECK: add [[GR_TOP:x[0-9]+]], [[GR_TOPTMP]], #56
-; CHECK: str [[GR_TOP]], [x[[VA_LIST]], #8]
+
 
 ; CHECK: mov [[VR_TOPTMP:x[0-9]+]], sp
 ; CHECK: add [[VR_TOP:x[0-9]+]], [[VR_TOPTMP]], #128
-; CHECK: str [[VR_TOP]], [x[[VA_LIST]], #16]
+; CHECK: stp [[GR_TOP]], [[VR_TOP]], [x[[VA_LIST]], #8]
 
 ; CHECK: mov     [[GRVR:x[0-9]+]], #-56
 ; CHECK: movk    [[GRVR]], #65408, lsl #32
 ; CHECK: str     [[GRVR]], [x[[VA_LIST]], #24]
 
   %addr = bitcast %va_list* @var to i8*
   call void @llvm.va_start(i8* %addr)
 

 ; CHECK-DAG: stp q6, q7, [sp, #80]
 ; ... omit middle ones ...
 ; CHECK-DAG: str q1, [sp]
 
 ; CHECK: str [[STACK_TOP]], [x[[VA_LIST]]]
 
 ; CHECK: add [[GR_TOPTMP:x[0-9]+]], sp, #[[GR_BASE]]
 ; CHECK: add [[GR_TOP:x[0-9]+]], [[GR_TOPTMP]], #40
-; CHECK: str [[GR_TOP]], [x[[VA_LIST]], #8]
 
 ; CHECK: mov [[VR_TOPTMP:x[0-9]+]], sp
 ; CHECK: add [[VR_TOP:x[0-9]+]], [[VR_TOPTMP]], #112
-; CHECK: str [[VR_TOP]], [x[[VA_LIST]], #16]
+; CHECK: stp [[GR_TOP]], [[VR_TOP]], [x[[VA_LIST]], #8]
 
 ; CHECK: mov  [[GRVR_OFFS:x[0-9]+]], #-40
 ; CHECK: movk [[GRVR_OFFS]], #65424, lsl #32
 ; CHECK: str  [[GRVR_OFFS]], [x[[VA_LIST]], #24]
 
   %addr = bitcast %va_list* @var to i8*
   call void @llvm.va_start(i8* %addr)
 

llvm/test/CodeGen/AArch64/machine-outliner-remarks.ll

 ; RUN: llc %s -enable-machine-outliner -mtriple=aarch64-unknown-unknown -pass-remarks=machine-outliner -pass-remarks-missed=machine-outliner -o /dev/null 2>&1 | FileCheck %s
 ; CHECK: <unknown>:0:0:
 ; CHECK-SAME: Did not outline 2 instructions from 2 locations.
 ; CHECK-SAME: Bytes from outlining all occurrences (16) >=
 ; CHECK-SAME: Unoutlined instruction bytes (16)
 ; CHECK-SAME: (Also found at: <UNKNOWN LOCATION>)
-; CHECK: remark: <unknown>:0:0: Saved 48 bytes by outlining 14 instructions
+; CHECK: remark: <unknown>:0:0: Saved 36 bytes by outlining 11 instructions
 ; CHECK-SAME: from 2 locations. (Found at: <UNKNOWN LOCATION>,
 ; CHECK-SAME: <UNKNOWN LOCATION>)
 ; RUN: llc %s -enable-machine-outliner -mtriple=aarch64-unknown-unknown -o /dev/null -pass-remarks-missed=machine-outliner -pass-remarks-output=%t.yaml
 ; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
 
 ; For the YAML case, the function we pick depends on the order of the candidate
 ; list.
 ; YAML: --- !Missed

 ; YAML-NEXT:   - OtherStartLoc1:  '<UNKNOWN LOCATION>'
 ; YAML-NEXT:   - String:          ')'
 ; YAML: --- !Passed
 ; YAML-NEXT: Pass:            machine-outliner
 ; YAML-NEXT: Name:            OutlinedFunction
 ; YAML-NEXT: Function:        OUTLINED_FUNCTION_0
 ; YAML-NEXT: Args:
 ; YAML-NEXT:   - String:          'Saved '
-; YAML-NEXT:   - OutliningBenefit: '48'
+; YAML-NEXT:   - OutliningBenefit: '36'
 ; YAML-NEXT:   - String:          ' bytes by '
 ; YAML-NEXT:   - String:          'outlining '
-; YAML-NEXT:   - Length:          '14'
+; YAML-NEXT:   - Length:          '11'
 ; YAML-NEXT:   - String:          ' instructions '
 ; YAML-NEXT:   - String:          'from '
 ; YAML-NEXT:   - NumOccurrences:  '2'
 ; YAML-NEXT:   - String:          ' locations. '
 ; YAML-NEXT:   - String:          '(Found at: '
 ; YAML-NEXT:   - StartLoc0:       '<UNKNOWN LOCATION>'
 ; YAML-NEXT:   - String:          ', '
 ; YAML-NEXT:   - StartLoc1:       '<UNKNOWN LOCATION>'

llvm/test/CodeGen/AArch64/machine-outliner.ll

define void @dog() #0 {
   store i32 5, i32* %5, align 4
   store i32 6, i32* %6, align 4
   ret void
 }
 
 ; ODR: [[OUTLINED]]:
 ; CHECK: .p2align 2
 ; CHECK-NEXT: [[OUTLINED]]:
-; CHECK: mov     w8, #1
-; CHECK-NEXT: str     w8, [sp, #28]
-; CHECK-NEXT: mov     w8, #2
-; CHECK-NEXT: str     w8, [sp, #24]
-; CHECK-NEXT: mov     w8, #3
-; CHECK-NEXT: str     w8, [sp, #20]
-; CHECK-NEXT: mov     w8, #4
-; CHECK-NEXT: str     w8, [sp, #16]
-; CHECK-NEXT: mov     w8, #5
-; CHECK-NEXT: str     w8, [sp, #12]
-; CHECK-NEXT: mov     w8, #6
-; CHECK-NEXT: str     w8, [sp, #8]
-; CHECK-NEXT: add     sp, sp, #32
-; CHECK-NEXT: ret
+; CHECK:      mov     w9, #1
+; CHECK-DAG: mov     w8, #2
+; CHECK-DAG: stp     w8, w9, [sp, #24]
+; CHECK-DAG: mov     w9, #3
+; CHECK-DAG: mov     w8, #4
+; CHECK-DAG: stp     w8, w9, [sp, #16]
+; CHECK-DAG: mov     w9, #5
+; CHECK-DAG: mov     w8, #6
+; CHECK-DAG: stp     w8, w9, [sp, #8]
+; CHECK-DAG: add     sp, sp, #32
+; CHECK-DAG: ret
 
 attributes #0 = { noredzone "target-cpu"="cyclone" "target-features"="+sse" }

llvm/test/CodeGen/AArch64/stp-opt-with-renaming.mir

+# RUN: llc -run-pass=aarch64-ldst-opt -mtriple=arm64-apple-iphoneos -verify-machineinstrs -o - %s | FileCheck %s
+
+---
+# CHECK-LABEL: name: test1
+# CHECK: bb.0:
+# CHECK-NEXT: liveins: $x0, $x1
+# CHECK:       $x10, renamable $x8 = LDPXi renamable $x0, 0 :: (load 8)
+# CHECK-NEXT:  renamable $x9 = LDRXui renamable $x0, 1 :: (load 8)
+# CHECK-NEXT:  STRXui renamable $x9, renamable $x0, 100 :: (store 8, align 4)
+# CHECK-NEXT:  renamable $x8 = ADDXrr $x8, $x8
+# CHECK-NEXT:  STPXi renamable $x8, killed $x10, renamable $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:  RET undef $lr
+
+name:            test1
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    renamable $x9, renamable $x8 = LDPXi renamable $x0, 0 :: (load 8)
+    STRXui renamable killed $x9, renamable $x0, 11 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 1 :: (load 8)
+    STRXui renamable $x9, renamable $x0, 100 :: (store 8, align 4)
+    renamable $x8 = ADDXrr $x8, $x8
+    STRXui renamable $x8, renamable $x0, 10 :: (store 8, align 4)
+    RET undef $lr
+
+...
+---
+# CHECK-LABEL: name: test2
+# CHECK-LABEL: bb.0:
+# CHECK-NEXT:    liveins: $x0, $x9, $x1
+
+# CHECK:         $x10, renamable $x8 = LDPXi renamable $x9, 0 :: (load 8)
+# CHECK-NEXT:    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+# CHECK-NEXT:    STRXui renamable $x9, renamable $x0, 100 :: (store 8, align 4)
+# CHECK-NEXT:    renamable $x8 = ADDXrr $x8, $x8
+# CHECK-NEXT:    STPXi renamable $x8, killed $x10, renamable $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test2
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x9' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x9, $x1
+    renamable $x9, renamable $x8 = LDPXi renamable $x9, 0 :: (load 8)
+    STRXui renamable killed $x9, renamable $x0, 11 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+    STRXui renamable $x9, renamable $x0, 100 :: (store 8, align 4)
+    renamable $x8 = ADDXrr $x8, $x8
+    STRXui renamable $x8, renamable $x0, 10 :: (store 8, align 4)
+    RET undef $lr
+
+...
+---
+# MOVK has a tied operand and we currently do not rename across tied defs.
+# CHECK-LABEL: bb.0:
+# CHECK-NEXT:    liveins: $x0
+#
+# CHECK:         renamable $x8 = MRS 58880
+# CHECK-NEXT:    renamable $x8 = MOVZXi 15309, 0
+# CHECK-NEXT:    renamable $x8 = MOVKXi renamable $x8, 26239, 16
+# CHECK-NEXT:    STRXui renamable $x8, renamable $x0, 0, implicit killed $x8 :: (store 8)
+# CHECK-NEXT:    renamable $x8 = MRS 55840
+# CHECK-NEXT:    STRXui killed renamable $x8, killed renamable $x0, 1, implicit killed $x8 :: (store 8)
+# CHECK-NEXT:    RET undef $lr
+#
+name:            test3
+alignment:       2
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+frameInfo:
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+
+    renamable $x8 = MRS 58880
+    renamable $x8 = MOVZXi 15309, 0
+    renamable $x8 = MOVKXi renamable $x8, 26239, 16
+    STRXui renamable $x8, renamable $x0, 0, implicit killed $x8 :: (store 8)
+    renamable $x8 = MRS 55840
+    STRXui killed renamable  $x8, renamable killed $x0, 1, implicit killed $x8 :: (store 8)
+    RET undef $lr
+
+...
+---
+# CHECK-LABEL: name: test4
+# CHECK-LABEL:  bb.0:
+# CHECK-NEXT:    liveins: $x0, $x1
+
+# CHECK:         $x9 = MRS 58880
+# CHECK-NEXT:    renamable $x8 = MRS 55840
+# CHECK-NEXT:    STPXi $x9, killed renamable $x8, killed renamable $x0, 0 :: (store 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test4
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+
+    renamable $x8 = MRS 58880
+    STRXui renamable $x8, renamable $x0, 0, implicit killed $x8 :: (store 4)
+    renamable $x8 = MRS 55840
+    STRXui killed renamable  $x8, renamable killed $x0, 1, implicit killed $x8 :: (store 4)
+    RET undef $lr
+
+...
+---
+# CHECK-LABEL: name: test5
+# CHECK-LABEL:  bb.0:
+# CHECK-NEXT:    liveins: $x0, $x1
+
+# CHECK:         $x9 = MRS 58880
+# CHECK-NEXT:    renamable $x8 = MRS 55840
+# CHECK-NEXT:    STPWi $w9, killed renamable $w8, killed renamable $x0, 0 :: (store 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test5
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+
+    renamable $x8 = MRS 58880
+    STRWui renamable $w8, renamable $x0, 0, implicit killed $x8 :: (store 4)
+    renamable $x8 = MRS 55840
+    STRWui killed renamable $w8, renamable killed $x0, 1, implicit killed $x8 :: (store 4)
+    RET undef $lr
+
+...
+---
+# CHECK-LABEL: name: test6
+# CHECK-LABEL  bb.0:
+# CHECK:    liveins: $x0, $x1, $q3
+
+# CHECK:         renamable $q9 = LDRQui $x0, 0 :: (load 16)
+# CHECK-NEXT:    renamable $q9 = XTNv8i16 renamable $q9, killed renamable $q3
+# CHECK-NEXT:    STRQui renamable $q9, renamable $x0, 11 :: (store 16, align 4)
+# CHECK-NEXT:    renamable $q9 = FADDv2f64 renamable $q9, renamable $q9
+# CHECK-NEXT:    STRQui renamable $q9, renamable $x0, 10 :: (store 16, align 4)
+# CHECK-NEXT:    RET undef $lr
+
+# XTN has a tied use-def.
+name:            test6
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+  - { reg: '$q3' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1, $q3
+    renamable $q9 = LDRQui $x0, 0 :: (load 16)
+    renamable $q9 = XTNv8i16 renamable $q9, killed renamable $q3
+    STRQui renamable $q9, renamable $x0, 11 :: (store 16, align 4)
+    renamable $q9 = FADDv2f64 renamable $q9, renamable $q9
+    STRQui renamable $q9, renamable $x0, 10 :: (store 16, align 4)
+    RET undef $lr
+
+...
+---
+# Currently we do not rename across frame-setup instructions.
+# CHECK-LABEL: name: test7
+# CHECK-LABEL: bb.0:
+# CHECK-NEXT:    liveins: $x0, $x1
+
+# CHECK:         $sp = frame-setup SUBXri $sp, 64, 0
+# CHECK-NEXT:    renamable $x9 = frame-setup LDRXui renamable $x0, 0 :: (load 8)
+# CHECK-NEXT:    STRXui renamable $x9, $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:    renamable $x9 = LDRXui renamable $x0, 1 :: (load 8)
+# CHECK-NEXT:    STRXui renamable $x9, $x0, 11 :: (store 8, align 4)
+# CHECK-NEXT:    RET undef $lr
+#
+name:            test7
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  stackSize:       64
+  maxAlignment:    16
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+stack:
+  - { id: 0, type: spill-slot, offset: -48, size: 16, alignment: 16 }
+  - { id: 1, type: spill-slot, offset: -64, size: 16, alignment: 16 }
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    $sp = frame-setup SUBXri $sp, 64, 0
+    renamable $x9 = frame-setup LDRXui renamable $x0, 0 :: (load 8)
+    STRXui renamable $x9, $x0, 10 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 1 :: (load 8)
+    STRXui renamable $x9, $x0, 11 :: (store 8, align 4)
+    RET undef $lr
+...
+---
+# CHECK-LABEL: name: test8
+# CHECK-LABEL:  bb.0:
+# CHECK-NEXT:    liveins: $x0, $x1
+
+# CHECK:         renamable $x8 = MRS 58880
+# CHECK-NEXT:    $w9 = ORRWrs $wzr, killed renamable $w8, 0, implicit-def $x9
+# CHECK-NEXT:    renamable $x8 = MRS 55840
+# CHECK-NEXT:    STPWi $w9, killed renamable $w8, killed renamable $x0, 0 :: (store 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test8
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+
+    renamable $x8 = MRS 58880
+    renamable $w8 = ORRWrs $wzr, killed renamable $w8, 0, implicit-def $x8
+    STRWui renamable $w8, renamable $x0, 0, implicit killed $x8 :: (store 4)
+    renamable $x8 = MRS 55840
+    STRWui killed renamable $w8, renamable killed $x0, 1, implicit killed $x8 :: (store 4)
+    RET undef $lr
+
+...
+---
+# The reg class returned for $q9 contains only the first 16 Q registers.
+# TODO: Can we check that all instructions that require renaming also support
+#       the second 16 Q registers?
+# CHECK-LABEL: name: test9
+# CHECK-LABEL  bb.0:
+# CHECK:    liveins: $x0, $x1, $q0, $q1, $q2, $q3, $q4, $q5, $q6, $q7
+
+# CHECK:         renamable $q9 = LDRQui $x0, 0 :: (load 16)
+# CHECK-NEXT:    STRQui killed renamable $q9, renamable $x0, 10 :: (store 16, align 4)
+# CHECK:         renamable $q9 = LDRQui $x0, 1 :: (load 16)
+# CHECK-NEXT:    STRQui renamable $q9, renamable $x0, 11 :: (store 16, align 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test9
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+  - { reg: '$q3' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1, $q0, $q1, $q2, $q3, $q4, $q5, $q6, $q7
+    renamable $q9 = LDRQui $x0, 0 :: (load 16)
+    STRQui renamable killed $q9, renamable $x0, 10 :: (store 16, align 4)
+    renamable $q9 = LDRQui $x0, 1 :: (load 16)
+    STRQui renamable $q9, renamable $x0, 11 :: (store 16, align 4)
+    RET undef $lr
+
+...
+---
+# The livein $q7 is killed early, so we can re-use it for renaming.
+# CHECK-LABEL: name: test10
+# CHECK-LABEL  bb.0:
+# CHECK:    liveins: $x0, $x1, $q0, $q1, $q2, $q3, $q4, $q5, $q6, $q7
+
+# CHECK:         renamable $q7 = FADDv2f64 renamable $q7, renamable $q7
+# CHECK-NEXT:    STRQui killed renamable $q7, renamable $x0, 100 :: (store 16, align 4)
+# CHECK-NEXT:    $q7 = LDRQui $x0, 0 :: (load 16)
+# CHECK-NEXT:    renamable $q9 = LDRQui $x0, 1 :: (load 16)
+# CHECK-NEXT:    STPQi killed renamable $q9, killed $q7, renamable $x0, 10 :: (store 16, align 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test10
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+  - { reg: '$q3' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1, $q0, $q1, $q2, $q3, $q4, $q5, $q6, $q7
+    renamable $q7 = FADDv2f64 renamable $q7, renamable $q7
+    STRQui renamable killed $q7, renamable $x0, 100 :: (store 16, align 4)
+    renamable $q9 = LDRQui $x0, 0 :: (load 16)
+    STRQui renamable killed $q9, renamable $x0, 11 :: (store 16, align 4)
+    renamable $q9 = LDRQui $x0, 1 :: (load 16)
+    STRQui renamable killed $q9, renamable $x0, 10 :: (store 16, align 4)
+    RET undef $lr
+
+...
+---
+# Make sure we do not use any registers that are defined between paired candidates
+# ($x14 in this example)
+# CHECK-LABEL: name: test11
+# CHECK: bb.0:
+# CHECK-NEXT: liveins: $x0, $x1, $x11, $x12, $x13
+
+# CHECK:         renamable $w10 = LDRWui renamable $x0, 0 :: (load 8)
+# CHECK-NEXT:    $x15, renamable $x8 = LDPXi renamable $x0, 1 :: (load 8)
+# CHECK-NEXT:    renamable $x9 = LDRXui renamable $x0, 3 :: (load 8)
+# CHECK-NEXT:    renamable $x14 = LDRXui renamable $x0, 5 :: (load 8)
+# CHECK-NEXT:    STPXi renamable $x9, killed $x15, renamable $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:    STRXui killed renamable $x14, renamable $x0, 200 :: (store 8, align 4)
+# CHECK-NEXT:    renamable $w8 = ADDWrr $w10, $w10
+# CHECK-NEXT:    STRWui renamable $w8, renamable $x0, 100 :: (store 8, align 4)
+# CHECK-NEXT:    RET undef $lr
+#
+name:            test11
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1, $x11, $x12, $x13
+    renamable $w10 = LDRWui renamable $x0, 0 :: (load 8)
+    renamable $x9, renamable $x8 = LDPXi renamable $x0, 1 :: (load 8)
+    STRXui renamable killed $x9, renamable $x0, 11 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 3 :: (load 8)
+    renamable $x14 = LDRXui renamable $x0, 5 :: (load 8)
+    STRXui renamable $x9, renamable $x0, 10 :: (store 8, align 4)
+    STRXui renamable killed $x14, renamable $x0, 200 :: (store 8, align 4)
+    renamable $w8 = ADDWrr $w10, $w10
+    STRWui renamable $w8, renamable $x0, 100 :: (store 8, align 4)
+    RET undef $lr
+
+...
+---
+# Check that we correctly deal with killed registers in stores that get merged forward,
+# which extends the live range of the first store operand.
+# CHECK-LABEL: name: test12
+# CHECK: bb.0:
+# CHECK-NEXT: liveins: $x0, $x1
+#
+# CHECK:         renamable $x10 = LDRXui renamable $x0, 0 :: (load 8)
+# CHECK-NEXT:    $x11, renamable $x8 = LDPXi renamable $x0, 3 :: (load 8)
+# CHECK-NEXT:    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+# CHECK-NEXT:    renamable $x8 = ADDXrr $x8, $x8
+# CHECK-NEXT:    STPXi renamable $x8, killed $x11, renamable $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:    STPXi killed renamable $x10, renamable $x9, renamable $x0, 20 :: (store 8, align 4)
+# CHECK-NEXT:    RET undef $lr
+
+name:            test12
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    renamable $x10 = LDRXui renamable $x0, 0 :: (load 8)
+    STRXui renamable killed $x10, renamable $x0, 20 :: (store 8, align 4)
+    renamable $x9, renamable $x8 = LDPXi renamable $x0, 3 :: (load 8)
+    STRXui renamable killed $x9, renamable $x0, 11 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+    renamable $x8 = ADDXrr $x8, $x8
+    STRXui renamable $x8, renamable $x0, 10 :: (store 8, align 4)
+    STRXui renamable $x9, renamable $x0, 21 :: (store 8, align 4)
+    RET undef $lr
+
+...
+---
+# Make sure we do not use any registers that are defined between def to rename and the first
+# paired store. ($x14 in this example)
+# CHECK-LABEL: name: test13
+# CHECK: bb.0:
+# CHECK-NEXT: liveins: $x0, $x1, $x10, $x11, $x12, $x13
+# CHECK:        $x15, renamable $x8 = LDPXi renamable $x0, 0 :: (load 8)
+# CHECK-NEXT:    renamable $x14 = LDRXui renamable $x0, 4 :: (load 8)
+# CHECK-NEXT:    STRXui killed renamable $x14, renamable $x0, 100 :: (store 8, align 4)
+# CHECK-NEXT:    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+# CHECK-NEXT:    STPXi renamable $x9, killed $x15, renamable $x0, 10 :: (store 8, align 4)
+# CHECK-NEXT:    RET undef $lr
+#
+name:            test13
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$x0' }
+  - { reg: '$x1' }
+  - { reg: '$x8' }
+frameInfo:
+  maxAlignment:    1
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1, $x10, $x11, $x12, $x13
+    renamable $x9, renamable $x8 = LDPXi renamable $x0, 0 :: (load 8)
+    renamable $x14 = LDRXui renamable $x0, 4 :: (load 8)
+    STRXui renamable killed $x14, renamable $x0, 100 :: (store 8, align 4)
+    STRXui renamable killed $x9, renamable $x0, 11 :: (store 8, align 4)
+    renamable $x9 = LDRXui renamable $x0, 2 :: (load 8)
+    STRXui renamable $x9, renamable $x0, 10 :: (store 8)
+    RET undef $lr
+
+...