[AArch64] Change location of frame-record within callee-save area.

[sdesmalen]

Summary

This patch changes the location of the frame-record (FP, LR) to the
bottom of the callee-saved area. According to the AAPCS the location of
the frame-record within the stackframe is unspecified (section 5.2.3 The
Frame Pointer), so the compiler should be free to choose a different
location.

The reason for changing the location of the frame-record is to prepare
the frame for allocating an SVE area below the callee-saves. This way the
compiler can use the VL-scaled addressing modes to directly access SVE
objects from the frame-pointer.

          :                :   
      | stack |        | stack |
      |  args |        |  args |
      +-------+        +-------+
      |  x30  |        |  x19  |
      |  x29  |        |  x20  |
FP -> |- - - -|        |  x21  |
      |  x19  |   ==>  |  x22  |
      |  x20  |        |- - - -|
      |  x21  |        |  x30  |
      |  x22  |        |  x29  |
      +-------+        +-------+ <- FP
      |///////|        |///////|         // realignment gap 
      |- - - -|        |- - - -|
      |spills/|        |spills/|
      | locals|        | locals|
SP -> +-------+        +-------+ <- SP

Things to point out:

• The CFI instructions emit the FP/LR registers first (in the order as specified in the CallingConvention.td file) in order to support the Compact Unwinding format.

• The algorith to find a paired register should be prevented from accidentally pairing some callee-saved register with LR that is not FP, since they should always be paired together when the frame has a frame-record.

[greened]

I can't really comment on the correctness of this but other than the one comment I'd like to see added, LGTM.

lib/Target/AArch64/AArch64FrameLowering.cpp

1706 ↗

(On Diff #213021)

Add a comment about what this does.

[efriedma]

I can't think of any issue this would cause, in general; following frame records should still work the same way, and DWARF unwinding should still work.

test/CodeGen/AArch64/wineh-try-catch-realign.ll

13 ↗

(On Diff #213021)

I don't think this is legal. On Windows, paired stores in the prologue must be in the form "stp xN, xN+1, [...]"; otherwise, there is no way to represent the store in the unwind format. (See https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling?view=vs-2019#unwind-codes )

[sdesmalen]

test/CodeGen/AArch64/wineh-try-catch-realign.ll

13 ↗

(On Diff #213021)

The merging of two separate STR instructions into one STP seems to done by the LoadStoreOptimizer, not PEI. The unwind information generated seems correct.

Running with stop-after=prologepilog:

frame-setup STRXui killed $x28, $sp, 2 :: (store 8 into %stack.2)
frame-setup SEH_SaveReg 28, 16
frame-setup STRXui killed $x19, $sp, 3 :: (store 8 into %stack.1)
frame-setup SEH_SaveReg 19, 24

Running readobj --unwind on the binary produced by this test gives:

Prologue [
  0xd003              ; str x19, [sp, #24]
  0xd242              ; str x28, [sp, #16]
  0x83                ; stp x29, x30, [sp, #-32]!
  0xe4                ; end
]

[kristof.beyls]

lib/Target/AArch64/AArch64FrameLowering.cpp

88 ↗

(On Diff #213021)

I wonder if it would be a good idea to record the rationale for (some of the) order in which specific blocks are put in the frame record?
I think it would be useful to extend this comment, stating that the frame record is placed specifically to be able to make use of VL-scaled addressing modes to directly access SVE objects from the frame-pointer.
I feel the frame layout code is pretty complex, and therefore it helps to spell out the higher-level design rationales at the beginning of this file in a comment.

[efriedma]

test/CodeGen/AArch64/wineh-try-catch-realign.ll

13 ↗

(On Diff #213021)

Windows unwinding determines whether an instruction is part of the prologue by computing the distance in bytes between the current pc and the entry point. So if the real prologue is shorter than the encoded prologue, the exception unwinder could be tricked into skipping important steps in unwinding.

Even if your patch isn't causing this issue, it's more likely to come up with your patch, I think.

[sdesmalen]

• Added comments describing rationale of the callee-save layout.
• Added comments to invalidateRegisterPairing
• Rebased patch onto D65817 which addresses the invalid pairing of callee save/restore instructions in the LoadStoreOptimizer.

[sdesmalen]

lib/Target/AArch64/AArch64FrameLowering.cpp

88 ↗

(On Diff #213021)

Good point. I have added some words to describe the rationale.

test/CodeGen/AArch64/wineh-try-catch-realign.ll

13 ↗

(On Diff #213021)

Thanks for pointing this out! I've created D65817 to address that issue.

[t.p.northover]

Unfortunately I don't think this is viable for Darwin platforms, at least not yet. Our compact unwind encoding just has a bitmask for which registers are saved rather than saying where relative to fp.

Longer term there's room for a new encoding to handle this, but that's not something we can do quickly. It looks like there are at least 5 projects that would need to be updated.

lib/Target/AArch64/AArch64FrameLowering.cpp

355–356 ↗

(On Diff #213654)

If changing just the order of the CFI directives convinces the AsmBackend to produce compact unwind for this new scheme then it doesn't have enough checks (understandable, the frame layout is normally very fixed).

[sdesmalen]

• Updated the patch to retain the current location of the frame-record in the callee-save area for Darwin.
• Removed the code that tried to emit the CFI information for FP and LR before the other callee-saves, as this was only needed on Darwin for the compact unwind encoding.

[sdesmalen]

In D65653#1618484, @t.p.northover wrote:

Unfortunately I don't think this is viable for Darwin platforms, at least not yet. Our compact unwind encoding just has a bitmask for which registers are saved rather than saying where relative to fp.

Longer term there's room for a new encoding to handle this, but that's not something we can do quickly. It looks like there are at least 5 projects that would need to be updated.

Thanks for pointing out. I have updated the patch to retain the current callee-save area layout for Darwin (regardless of whether compact unwind encoding is required).
The changes aren't that significant, but it would be nice to use the same layout at some point.

[thegameg]

lib/Target/AArch64/AArch64FrameLowering.cpp

47 ↗

(On Diff #213892)

Maybe it's worth adding a note pointing out that on Darwin this is different?

[sdesmalen]

Added an extra comment describing the swapped order of the frame-record location on Darwin.

lib/Target/AArch64/AArch64FrameLowering.cpp

47 ↗

(On Diff #213892)

Good spot, thanks!

[kristof.beyls]

lib/Target/AArch64/AArch64FrameLowering.cpp

91–95 ↗

(On Diff #213978)

Mostly looks good to me.
I wonder if it'd be useful to explicitly state that the size of "SVE stack objects" aren't known at compile time, and hence the offset that needs to be calculated needs to be multiplied by an unknown-at-compile time factor?
This is of course all very obvious when working already for some time on SVE; but some readers of this comment may be unfamiliar with SVE?

[sdesmalen]

Clarified comment.

[sdesmalen]

lib/Target/AArch64/AArch64FrameLowering.cpp

91–95 ↗

(On Diff #213978)

Agreed. I've added some more words to describe that.

[sdesmalen]

Thanks for all the feedback so far. I think I've addressed all comments and suggestions, are we happy to move forward with this patch?

[greened]

In D65653#1626123, @sdesmalen wrote:

Thanks for all the feedback so far. I think I've addressed all comments and suggestions, are we happy to move forward with this patch?

The comments and changes are great. I'm happy to move forward but I don't think I should be the one to accept, as I only have experience with Linux/AArch64 and not other systems like Windows and Darwin.

[efriedma]

lib/Target/AArch64/AArch64CallingConvention.td

342 ↗

(On Diff #214081)

Is the order of the registers in this list important? If it is, should we explain it in a comment somewhere?

test/CodeGen/AArch64/aarch64-dynamic-stack-layout.ll

583 ↗

(On Diff #214081)

I don't understand what's happening here; are we reducing register pressure somhow?

[sdesmalen]

Added comment to clarify that the order of the callee-saves as specified in AArch64CallingConvention.td actually affects the layout of the callee-saves in the generated stackframe.

[sdesmalen]

lib/Target/AArch64/AArch64CallingConvention.td

342 ↗

(On Diff #214081)

This is indeed important since various parts in AArch64FrameLowering depend on this order (see also my other comment). I've added a comment to describe this, thanks!

test/CodeGen/AArch64/aarch64-dynamic-stack-layout.ll

583 ↗

(On Diff #214081)

Before it did an extra spill of x28 to use as scratch register, but now it uses x30 for that purpose.

This is due to a quirk in determineCalleeSaves, which iterates through the list of callee-saves and then spills the last unused register as a scratch register. With the previous order in which the callee-saves were specified in the .td file, this last register was x28. With the new ordering, the last unused register is x30. This register already conveniently stored as part of the frame-record since we're using a frame-pointer, so there is no need to spill an additional register.

[efriedma]

LGTM

[leonardchan]

I have a suspicion that this might be causing the assertion error we see in https://bugs.llvm.org/show_bug.cgi?id=43015. Checking this now. Just wanted to raise awareness in the meantime.

[sdesmalen]

In D65653#1632508, @leonardchan wrote:

I have a suspicion that this might be causing the assertion error we see in https://bugs.llvm.org/show_bug.cgi?id=43015. Checking this now. Just wanted to raise awareness in the meantime.

Sorry for that, there was an obvious condition missing in that assert. I have relanded the patch in rGf28e1128d9ef .

[leonardchan]

Hi again. After some bisecting, we found that the relanding of this (rGf28e1128d9e) was causing a segfault on our aarch64 asan bots.

Stack trace when attempting to launch Fuchsia on QEMU:

[318.389] 01525.01742> <== exception: process boot/test/sys/thread-state-test[1099746] thread initial-thread[1099761]
[318.389] 01525.01742> <== sw breakpoint, PC at 0xf48c53187fc0
[318.389] 01525.01742>  x0  0xee726573756d65ee x1                   0 x2                   0 x3                0x34
[318.389] 01525.01742>  x4      0xacbe942db2e0 x5      0xa5edc685d51c x6  0x2820202020202020 x7  0x777320676e697375
[318.389] 01525.01742>  x8  0xee726573756d65ee x9  0x988573be33452e9e x10                0x7 x11     0xa5edc685a078
[318.389] 01525.01742>  x12     0x14bdb8d0b40f x13     0x14bdb8d0b40d x14 0xf8f8f8f8f8f8f8f8 x15                  0
[318.390] 01525.01742>  x16               0x1b x17     0xe196795d09a8 x18     0xe287a07e5000 x19         0xfffffff1
[318.390] 01525.01742>  x20     0xf48c5317f8e0 x21     0xacbe942db8f0 x22          0x5743993 x23     0xacbe942db800
[318.390] 01525.01742>  x24     0xacbe942db810 x25     0x193d9a9c80d0 x26     0xacbe942db7e0 x27     0x1597d285b6fc
[318.390] 01525.01742>  x28     0x1597d285b702 x29     0xacbe942db7a0 lr      0xf48c53187fac sp      0xacbe942db7a0
[318.390] 01525.01742>  pc      0xf48c53187fc0 psr         0x60000000
[318.390] 01525.01742> arch: aarch64
[318.391] 01525.01742> dso: id=3ab1b3612220919e base=0xf48c5317c000 name=app:boot/test/sys/thread-state-test
[318.391] 01525.01742> dso: id=6dd3318a7516cc20 base=0xe79ffff8b000 name=libc++.so.2
[318.391] 01525.01742> dso: id=4e7ba3b6018e3afe base=0xe196795ca000 name=<vDSO>
[318.391] 01525.01742> dso: id=1038afe56f0ed1b5 base=0xd0a88b63e000 name=libc++abi.so.1
[318.391] 01525.01742> dso: id=781087240e810874 base=0xc9ecd4e35000 name=libunittest.so
[318.391] 01525.01742> dso: id=8f25bd2b8480d32c base=0xb078a64d3000 name=libfdio.so
[318.391] 01525.01742> dso: id=7f401cc813f2cf9c base=0xa5edc6730000 name=libc.so
[318.391] 01525.01742> dso: id=62ede582d82f7da3 base=0x9535580de000 name=libclang_rt.asan.so
[318.391] 01525.01742> dso: id=738a2fbe70bb8f31 base=0x9460793bf000 name=libunwind.so.1
[318.391] 01525.01742> dso: id=6428e355aa5298dd base=0x55e1360d9000 name=liblaunchpad.so
[318.392] 01525.01742>    #1.1  0x0000f48c53187fc0 in backtrace_request() ../../out/default.zircon/../../zircon/system/ulib/backtrace-request/include/lib/backtrace-request/backtrace-request.h:30 <<VMO#1099739=thread-state-test>>+0xbfc0 sp 0xacbe942db7a0
[318.392] 01525.01742>    #1    0x0000f48c53187fc0 in tu_fatal ../../out/default.zircon/../../zircon/system/ulib/test-utils/test-utils.cc:82 <<VMO#1099739=thread-state-test>>+0xbfc0 sp 0xacbe942db7a0
[318.394] 01525.01742>    #2    0x0000f48c53188488 in tu_channel_read ../../out/default.zircon/../../zircon/system/ulib/test-utils/test-utils.cc:165 <<VMO#1099739=thread-state-test>>+0xc488 sp 0xacbe942db7c0
[318.395] 01525.01742>    #3    0x0000f48c53187004 in recv_msg(unsigned int, Message*) ../../out/default.zircon/../../zircon/system/utest/thread-state/thread-state.cc:94 <<VMO#1099739=thread-state-test>>+0xb004 sp 0xacbe942db7e0
[318.396] 01525.01742>    #4    0x0000f48c53186ab0 in get_child_thread(unsigned int, unsigned int*) ../../out/default.zircon/../../zircon/system/utest/thread-state/thread-state.cc:410 <<VMO#1099739=thread-state-test>>+0xaab0 sp 0xacbe942db8c0
[318.397] 01525.01742>    #5.4  0x0000f48c53182758 in (anonymous namespace)::FidlDecoder::SetError(char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:200 <<VMO#1099739=thread-state-test>>+0x6758 sp 0xacbe942db9c0
[318.397] 01525.01742>    #5.3  0x0000f48c53182758 in (anonymous namespace)::FidlDecoder::OnError(char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:179 <<VMO#1099739=thread-state-test>>+0x6758 sp 0xacbe942db9c0
[318.397] 01525.01742>    #5.2  0x0000f48c53182758 in fidl::internal::Walker<(anonymous namespace)::FidlDecoder>::Walk((anonymous namespace)::FidlDecoder&) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:706 <<VMO#1099739=thread-state-test>>+0x6758 sp 0xacbe942db9c0
[318.397] 01525.01742>    #5.1  0x0000f48c53182758 in void fidl::Walk<(anonymous namespace)::FidlDecoder>((anonymous namespace)::FidlDecoder&, fidl_type const*, (anonymous namespace)::FidlDecoder::StartingPoint) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:893 <<VMO#1099739=thread-state-test>>+0x6758 sp 0xacbe942db9c0
[318.397] 01525.01742>    #5    0x0000f48c53182758 in channel_test() ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:268 <<VMO#1099739=thread-state-test>>+0x6758 sp 0xacbe942db9c0
[318.399] 01525.01742>    #6.3  0x0000c9ecd4e3b40c in unittest_run_test(char const*, bool (*)(), test_info**, bool*) ../../out/default.zircon/../../zircon/system/ulib/unittest/unittest.cc:268 <<VMO#1099829=asan/libunittest.s>+0x640c sp 0xacbe942dbae0
[318.399] 01525.01742>    #6.2  0x0000c9ecd4e3b40c in unittest_run_named_test::$_1::operator()() const ../../out/default.zircon/../../zircon/system/ulib/unittest/unittest.cc:296 <<VMO#1099829=asan/libunittest.s>+0x640c sp 0xacbe942dbae0
[318.399] 01525.01742>    #6.1  0x0000c9ecd4e3b40c in void run_with_watchdog<unittest_run_named_test::$_1>(test_type, char const*, unittest_run_named_test::$_1) ../../out/default.zircon/../../zircon/system/ulib/unittest/unittest.cc:285 <<VMO#1099829=asan/libunittest.s>+0x640c sp 0xacbe942dbae0
[318.399] 01525.01742>    #6    0x0000c9ecd4e3b40c in unittest_run_named_test ../../out/default.zircon/../../zircon/system/ulib/unittest/unittest.cc:295 <<VMO#1099829=asan/libunittest.s>+0x640c sp 0xacbe942dbae0
[318.399] 01525.01742>    #7.4  0x0000f48c53181260 in (anonymous namespace)::FidlHandleCloser::SetError(char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/handle_closing.cc:93 <<VMO#1099739=thread-state-test>>+0x5260 sp 0xacbe942dbbe0
[318.399] 01525.01742>    #7.3  0x0000f48c53181260 in (anonymous namespace)::FidlHandleCloser::OnError(char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/handle_closing.cc:84 <<VMO#1099739=thread-state-test>>+0x5260 sp 0xacbe942dbbe0
[318.399] 01525.01742>    #7.2  0x0000f48c53181260 in fidl::internal::Walker<(anonymous namespace)::FidlHandleCloser>::Walk((anonymous namespace)::FidlHandleCloser&) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:648 <<VMO#1099739=thread-state-test>>+0x5260 sp 0xacbe942dbbe0
[318.399] 01525.01742>    #7.1  0x0000f48c53181260 in void fidl::Walk<(anonymous namespace)::FidlHandleCloser>((anonymous namespace)::FidlHandleCloser&, fidl_type const*, (anonymous namespace)::FidlHandleCloser::StartingPoint) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:893 <<VMO#1099739=thread-state-test>>+0x5260 sp 0xacbe942dbbe0
[318.399] 01525.01742>    #7    0x0000f48c53181260 in thread_state_tests(bool, char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/handle_closing.cc:120 <<VMO#1099739=thread-state-test>>+0x5260 sp 0xacbe942dbbe0
[318.401] 01525.01742>    #8.1  0x0000c9ecd4e3a44c in unittest_run_all_tests_etc(char const*, test_type, char const*, char const*, bool) ../../out/default.zircon/../../zircon/system/ulib/unittest/all-tests.cc:46 <<VMO#1099829=asan/libunittest.s>+0x544c sp 0xacbe942dbcc0
[318.401] 01525.01742>    #8    0x0000c9ecd4e3a44c in unittest_run_all_tests ../../out/default.zircon/../../zircon/system/ulib/unittest/all-tests.cc:207 <<VMO#1099829=asan/libunittest.s>+0x544c sp 0xacbe942dbcc0
[318.401] 01525.01742>    #9.4  0x0000f48c53184bbc in (anonymous namespace)::FidlDecoder::SetError(char const*) ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:197 <<VMO#1099739=thread-state-test>>+0x8bbc sp 0xacbe942dbdc0
[318.401] 01525.01742>    #9.3  0x0000f48c53184bbc in (anonymous namespace)::FidlDecoder::VisitPointer((anonymous namespace)::Position, void**, unsigned int, (anonymous namespace)::Position*) ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:76 <<VMO#1099739=thread-state-test>>+0x8bbc sp 0xacbe942dbdc0
[318.401] 01525.01742>    #9.2  0x0000f48c53184bbc in fidl::internal::Walker<(anonymous namespace)::FidlDecoder>::Walk((anonymous namespace)::FidlDecoder&) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:680 <<VMO#1099739=thread-state-test>>+0x8bbc sp 0xacbe942dbdc0
[318.401] 01525.01742>    #9.1  0x0000f48c53184bbc in void fidl::Walk<(anonymous namespace)::FidlDecoder>((anonymous namespace)::FidlDecoder&, fidl_type const*, (anonymous namespace)::FidlDecoder::StartingPoint) ../../out/default.zircon/../../zircon/system/ulib/fidl/include/lib/fidl/walker.h:893 <<VMO#1099739=thread-state-test>>+0x8bbc sp 0xacbe942dbdc0
[318.401] 01525.01742>    #9    0x0000f48c53184bbc in main ../../out/default.zircon/../../zircon/system/ulib/fidl/decoding.cc:268 <<VMO#1099739=thread-state-test>>+0x8bbc sp 0xacbe942dbdc0
[318.403] 01525.01742>    #10   0x0000a5edc67bd134 in start_main ../../out/default.zircon/../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:93 <<VMO#1099752=asan/ld.so.1>>+0x8d134 sp 0xacbe942dbe00
[318.403] 01525.01742>    #11   0x0000a5edc67bd864 in __libc_start_main ../../out/default.zircon/../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:179 <<VMO#1099752=asan/ld.so.1>>+0x8d864 sp 0xacbe942dc000
[318.403] 01525.01742>    #12   0x0000f48c53187f5c in _start ../../out/default.zircon/../../zircon/system/ulib/c/Scrt1.cc:7 <<VMO#1099739=thread-state-test>>+0xbf5c sp 0xa637d5b2fff0

Could you look into this again? Thanks.

[leonardchan]

Could you temporarily revert this in the meantime? This is blocking our clang roll that we need to fix another Clang issue we're hitting.

[sdesmalen]

In D65653#1636408, @leonardchan wrote:

Could you look into this again? Thanks.

Happy to look into this, but the stacktrace you pasted doesn't give enough information for me to reproduce. Can you give some more details?

Could you temporarily revert this in the meantime? This is blocking our clang roll that we need to fix another Clang issue we're hitting.

From the information you provided it is not clear to me if the issue is actually caused by this patch or whether this is an issue in Fuchsia itself. Do you know if Fuchsia makes any implicit assumptions on the layout of the callee-save area beyond what is specified in the AAPCS? If so, we may be able to temporarily disable this for Fuchsia until that issue is resolved, but I'm hesitant to just make this change without understanding what the issue is first.

[omjavaid]

Hi

This rev breaks LLDB for AArch64. functionalities/thread/concurrent_events/* tests are failing after r368987.

The problem appears to be coming out of signal handling where linux might be assuming stack with previous layout of FP and LR pair.

Although AArch64 AAPCS does allow FP/LR to be implementation specific but this change might be breaking implementations which have adapted to previous layout.

I was wondering if this layout change is absolutely necessary for SVE accommodation?

AArch64 LLDB has a staging buildbot here: http://lab.llvm.org:8014/builders/lldb-aarch64-ubuntu

[sdesmalen]

In D65653#1645767, @omjavaid wrote:

This rev breaks LLDB for AArch64. functionalities/thread/concurrent_events/* tests are failing after r368987.

Thanks for bringing this to my attention @omjavaid and sorry about breaking these tests! Tbh, I was not aware anything was broken, because I normally get spammed by buildbot when something breaks. Is that functionality not working at the moment?

The problem appears to be coming out of signal handling where linux might be assuming stack with previous layout of FP and LR pair.

It appears this patch exposes an existing issue that the callee-save stack size is calculated twice (with different results). This leads to incorrect debug information for locals.
I've addressed this in D66935!

Although AArch64 AAPCS does allow FP/LR to be implementation specific but this change might be breaking implementations which have adapted to previous layout.

I double checked with some folks who have experience working on the Linux kernel (AArch64), who told me that neither Linux nor userspace signal handlers should care about the precise frame layout used by the compiler. They see no obvious reasons why the proposed change to the frame layout would break anything.

[pcc]

So I believe that after this change, we are no longer guaranteed that FP + 16 = SP at exit, correct? I think this would be a requirement for a "fast" unwinder that works by following frame pointers (rather than reading unwind info) and is capable of decrypting return addresses of functions built with -msign-return-addresses (because the AUT instruction would need to know SP at exit). If this is the case, would it be feasible to let a platform opt out of this behaviour? (I realise that Darwin is opted out, but I don't know about the SVE support status there.) I'm not very familiar with SVE, but I would imagine that you could use a different register for this purpose on opted out platforms.

[efriedma]

So I believe that after this change, we are no longer guaranteed that FP + 16 = SP at exit, correct?

Yes.

I think this would be a requirement for a "fast" unwinder that works by following frame pointers (rather than reading unwind info) and is capable of decrypting return addresses of functions built with -msign-return-addresses (because the AUT instruction would need to know SP at exit).

Exception handling has to use a DWARF unwinder, not a "fast" unwinder. So the "fast" unwinder can't actually do anything with the address other than print it out. Given that, I can't think of any security benefit to verifying the signature. (And if you don't care about security, you can "decrypt" an address by just clearing the signing bits.)

I'm not very familiar with SVE, but I would imagine that you could use a different register for this purpose on opted out platforms.

The point of the layout change is so the frame pointer points at the beginning of the scalable section of the stack frame. This isn't strictly necessary; stack lowering should do the right thing anyway. It just makes address computation a little faster (because we can just compute "fp+vscale*scaledoffset", instead of "fp+fixedoffset+vscale*scaledoffset").

[pcc]

In D65653#1825141, @efriedma wrote:

Exception handling has to use a DWARF unwinder, not a "fast" unwinder. So the "fast" unwinder can't actually do anything with the address other than print it out.

Yes, that's essentially all that it needs to do.

(And if you don't care about security, you can "decrypt" an address by just clearing the signing bits.)

Ah, that was the detail that I missed. I don't see any problem then. Thanks.

llvm/trunk/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_GHC : CallingConv<[
 
   // Promote i8/i16/i32 arguments to i64.
   CCIfType<[i8, i16, i32], CCPromoteToType<i64>>,
 
   // Pass in STG registers: Base, Sp, Hp, R1, R2, R3, R4, R5, R6, SpLim
   CCIfType<[i64], CCAssignToReg<[X19, X20, X21, X22, X23, X24, X25, X26, X27, X28]>>
 ]>;
 
+// The order of the callee-saves in this file is important, because the
+// FrameLowering code will use this order to determine the layout the
+// callee-save area in the stack frame. As can be observed below, Darwin
+// requires the frame-record (LR, FP) to be at the top the callee-save area,
+// whereas for other platforms they are at the bottom.
+
 // FIXME: LR is only callee-saved in the sense that *we* preserve it and are
 // presumably a callee to someone. External functions may not do so, but this
 // is currently safe since BL has LR as an implicit-def and what happens after a
 // tail call doesn't matter.
 //
 // It would be better to model its preservation semantics properly (create a
 // vreg on entry, use it in RET & tail call generation; make that vreg def if we
 // end up saving LR as part of a call frame). Watch this space...
-def CSR_AArch64_AAPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21, X22,
+def CSR_AArch64_AAPCS : CalleeSavedRegs<(add X19, X20, X21, X22, X23, X24,
+                                           X25, X26, X27, X28, LR, FP,
+                                           D8,  D9,  D10, D11,
+                                           D12, D13, D14, D15)>;
+
+// Darwin puts the frame-record at the top of the callee-save area.
+def CSR_Darwin_AArch64_AAPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21, X22,
                                            X23, X24, X25, X26, X27, X28,
                                            D8,  D9,  D10, D11,
                                            D12, D13, D14, D15)>;
 
 // Win64 has unwinding codes for an (FP,LR) pair, save_fplr and save_fplr_x.
 // We put FP before LR, so that frame lowering logic generates (FP,LR) pairs,
 // and not (LR,FP) pairs.
-def CSR_Win_AArch64_AAPCS : CalleeSavedRegs<(add FP, LR, X19, X20, X21, X22,
-                                               X23, X24, X25, X26, X27, X28,
+def CSR_Win_AArch64_AAPCS : CalleeSavedRegs<(add X19, X20, X21, X22, X23, X24,
+                                               X25, X26, X27, X28, FP, LR,
                                                D8, D9, D10, D11,
                                                D12, D13, D14, D15)>;
 
 // AArch64 PCS for vector functions (VPCS)
 // must (additionally) preserve full Q8-Q23 registers
-def CSR_AArch64_AAVPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21, X22,
-                                          X23, X24, X25, X26, X27, X28,
+def CSR_AArch64_AAVPCS : CalleeSavedRegs<(add X19, X20, X21, X22, X23, X24,
+                                          X25, X26, X27, X28, LR, FP,
                                           (sequence "Q%u", 8, 23))>;
 
 // Functions taking SVE arguments or returning an SVE type
 // must (additionally) preserve full Z8-Z23 and predicate registers P4-P15
-def CSR_AArch64_SVE_AAPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21, X22,
-                                               X23, X24, X25, X26, X27, X28,
+def CSR_AArch64_SVE_AAPCS : CalleeSavedRegs<(add X19, X20, X21, X22, X23, X24,
+                                               X25, X26, X27, X28, LR, FP,
                                                (sequence "Z%u", 8, 23),
                                                (sequence "P%u", 4, 15))>;
 
 // Constructors and destructors return 'this' in the iOS 64-bit C++ ABI; since
 // 'this' and the pointer return value are both passed in X0 in these cases,
 // this can be partially modelled by treating X0 as a callee-saved register;
 // only the resulting RegMask is used; the SaveList is ignored
 //
 // (For generic ARM 64-bit ABI code, clang will not generate constructors or
 // destructors with 'this' returns, so this RegMask will not be used in that
 // case)
 def CSR_AArch64_AAPCS_ThisReturn : CalleeSavedRegs<(add CSR_AArch64_AAPCS, X0)>;
 
 def CSR_AArch64_AAPCS_SwiftError
-    : CalleeSavedRegs<(sub CSR_AArch64_AAPCS, X21)>;
+    : CalleeSavedRegs<(sub CSR_Darwin_AArch64_AAPCS, X21)>;
 
 // The function used by Darwin to obtain the address of a thread-local variable
 // guarantees more than a normal AAPCS function. x16 and x17 are used on the
 // fast path for calculation, but other registers except X0 (argument/return)
 // and LR (it is a call, after all) are preserved.
 def CSR_AArch64_TLS_Darwin
     : CalleeSavedRegs<(add (sub (sequence "X%u", 1, 28), X16, X17),
                            FP,
                            (sequence "Q%u", 0, 31))>;
 
 // We can only handle a register pair with adjacent registers, the register pair
 // should belong to the same class as well. Since the access function on the
 // fast path calls a function that follows CSR_AArch64_TLS_Darwin,
 // CSR_AArch64_CXX_TLS_Darwin should be a subset of CSR_AArch64_TLS_Darwin.
 def CSR_AArch64_CXX_TLS_Darwin
-    : CalleeSavedRegs<(add CSR_AArch64_AAPCS,
+    : CalleeSavedRegs<(add CSR_Darwin_AArch64_AAPCS,
                            (sub (sequence "X%u", 1, 28), X15, X16, X17, X18),
                            (sequence "D%u", 0, 31))>;
 
 // CSRs that are handled by prologue, epilogue.
 def CSR_AArch64_CXX_TLS_Darwin_PE
     : CalleeSavedRegs<(add LR, FP)>;
 
 // CSRs that are handled explicitly via copies.

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.cpp

 // | arguments passed on the stack     |
 // |                                   |
 // |-----------------------------------|
 // |                                   |
 // | (Win64 only) varargs from reg     |
 // |                                   |
 // |-----------------------------------|
 // |                                   |
-// | prev_fp, prev_lr                  |
+// | callee-saved gpr registers        | <--.
+// |                                   |    | On Darwin platforms these
+// |- - - - - - - - - - - - - - - - - -|    | callee saves are swapped,
+// |                                   |    | (frame record first)
+// | prev_fp, prev_lr                  | <--'
 // | (a.k.a. "frame record")           |
 // |-----------------------------------| <- fp(=x29)
 // |                                   |
-// | other callee-saved registers      |
+// | callee-saved fp/simd/SVE regs     |
 // |                                   |
 // |-----------------------------------|
 // |.empty.space.to.make.part.below....|
 // |.aligned.in.case.it.needs.more.than| (size of this area is unknown at
 // |.the.standard.16-byte.alignment....|  compile time; if present)
 // |-----------------------------------|
 // |                                   |
 // | local variables of fixed size     |

 //
 // For most functions, some of the frame areas are empty. For those functions,
 // it may not be necessary to set up fp or bp:
 // * A base pointer is definitely needed when there are both VLAs and local
 //   variables with more-than-default alignment requirements.
 // * A frame pointer is definitely needed when there are local variables with
 //   more-than-default alignment requirements.
 //
+// For Darwin platforms the frame-record (fp, lr) is stored at the top of the
+// callee-saved area, since the unwind encoding does not allow for encoding
+// this dynamically and existing tools depend on this layout. For other
+// platforms, the frame-record is stored at the bottom of the (gpr) callee-saved
+// area to allow SVE stack objects (allocated directly below the callee-saves,
+// if available) to be accessed directly from the framepointer.
+// The SVE spill/fill instructions have VL-scaled addressing modes such
+// as:
+//    ldr z8, [fp, #-7 mul vl]
+// For SVE the size of the vector length (VL) is not known at compile-time, so
+// '#-7 mul vl' is an offset that can only be evaluated at runtime. With this
+// layout, we don't need to add an unscaled offset to the framepointer before
+// accessing the SVE object in the frame.
+//
 // In some cases when a base pointer is not strictly needed, it is generated
 // anyway when offsets from the frame pointer to access local variables become
 // so large that the offset can't be encoded in the immediate fields of loads
 // or stores.
 //
 // FIXME: also explain the redzone concept.
 // FIXME: also explain the concept of reserved call frames.
 //

 }
 
 static bool needsWinCFI(const MachineFunction &MF) {
   const Function &F = MF.getFunction();
   return MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
          F.needsUnwindTableEntry();
 }
 
+static bool isTargetDarwin(const MachineFunction &MF) {
+  return MF.getSubtarget<AArch64Subtarget>().isTargetDarwin();
+}
+
 void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.begin();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const Function &F = MF.getFunction();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
if (F.hasPersonalityFn()) {
         MBB.addLiveIn(AArch64::X1);
       }
     }
 
     return;
   }
 
   if (HasFP) {
-    // Only set up FP if we actually need to. Frame pointer is fp =
-    // sp - fixedobject - 16.
-    int FPOffset = AFI->getCalleeSavedStackSize() - 16;
+    // Only set up FP if we actually need to.
+    int FPOffset = isTargetDarwin(MF) ? (AFI->getCalleeSavedStackSize() - 16) : 0;
+
     if (CombineSPBump)
       FPOffset += AFI->getLocalStackSize();
 
     // Issue    sub fp, sp, FPOffset or
     //          mov fp,sp          when FPOffset is zero.
     // Note: All stores of callee-saved registers are marked as "FrameSetup".
     // This code marks the instruction(s) that set the FP also.
     emitFrameOffset(MBB, MBBI, DL, AArch64::FP, AArch64::SP,
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   // SEH opcode indicating the prologue end.
   if (NeedsWinCFI && HasWinCFI) {
     BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_PrologEnd))
         .setMIFlag(MachineInstr::FrameSetup);
   }
 
   if (needsFrameMoves) {
     const DataLayout &TD = MF.getDataLayout();
-    const int StackGrowth = -TD.getPointerSize(0);
+    const int StackGrowth = isTargetDarwin(MF)
+                                ? (2 * -TD.getPointerSize(0))
+                                : -AFI->getCalleeSavedStackSize();
     Register FramePtr = RegInfo->getFrameRegister(MF);
     // An example of the prologue:
     //
     //     .globl __foo
     //     .align 2
     //  __foo:
     // Ltmp0:
     //     .cfi_startproc
if (needsFrameMoves) {
     //     .cfi_offset w27, -24
     //  Ltmp5:
     //     .cfi_offset w28, -32
 
     if (HasFP) {
       // Define the current CFA rule to use the provided FP.
       unsigned Reg = RegInfo->getDwarfRegNum(FramePtr, true);
       unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createDefCfa(
-          nullptr, Reg, 2 * StackGrowth - FixedObject));
+          nullptr, Reg, StackGrowth - FixedObject));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex)
           .setMIFlags(MachineInstr::FrameSetup);
     } else {
       // Encode the stack size of the leaf function.
       unsigned CFIIndex = MF.addFrameInst(
           MCCFIInstruction::createDefCfaOffset(nullptr, -MFI.getStackSize()));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
if (!hasFP(MF)) {
 
     NumBytes = 0;
   }
 
   // Restore the original stack pointer.
   // FIXME: Rather than doing the math here, we should instead just use
   // non-post-indexed loads for the restores if we aren't actually going to
   // be able to save any instructions.
-  if (!IsFunclet && (MFI.hasVarSizedObjects() || AFI->isStackRealigned()))
+  if (!IsFunclet && (MFI.hasVarSizedObjects() || AFI->isStackRealigned())) {
+    int64_t OffsetToFrameRecord =
+        isTargetDarwin(MF) ? (-(int64_t)AFI->getCalleeSavedStackSize() + 16) : 0;
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::FP,
-                    {-(int64_t)AFI->getCalleeSavedStackSize() + 16, MVT::i8},
+                    {OffsetToFrameRecord, MVT::i8},
                     TII, MachineInstr::FrameDestroy, false, NeedsWinCFI);
-  else if (NumBytes)
+  } else if (NumBytes)
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP,
                     {NumBytes, MVT::i8}, TII, MachineInstr::FrameDestroy, false,
                     NeedsWinCFI);
 
   // This must be placed after the callee-save restore code because that code
   // assumes the SP is at the same location as it was after the callee-save save
   // code in the prologue.
   if (AfterCSRPopSize) {

 }
 
 static StackOffset getFPOffset(const MachineFunction &MF, int ObjectOffset) {
   const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
   const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
-  return {ObjectOffset + FixedObject + 16, MVT::i8};
+  unsigned FPAdjust = isTargetDarwin(MF) ? 16 : AFI->getCalleeSavedStackSize();
+  return {ObjectOffset + FixedObject + FPAdjust, MVT::i8};
 }
 
 static StackOffset getStackOffset(const MachineFunction &MF, int ObjectOffset) {
   const auto &MFI = MF.getFrameInfo();
   return {ObjectOffset + (int)MFI.getStackSize(), MVT::i8};
 }
 
 int AArch64FrameLowering::getSEHFrameIndexOffset(const MachineFunction &MF,
static bool invalidateWindowsRegisterPairing(unsigned Reg1, unsigned Reg2,
   // the MCLayer.  We need to add support for the save_lrpair unwind code.
   if (!NeedsWinCFI)
     return false;
   if (Reg2 == Reg1 + 1)
     return false;
   return true;
 }
 
+/// Returns true if Reg1 and Reg2 cannot be paired using a ldp/stp instruction.
+/// WindowsCFI requires that only consecutive registers can be paired.
+/// LR and FP need to be allocated together when the frame needs to save
+/// the frame-record. This means any other register pairing with LR is invalid.
+static bool invalidateRegisterPairing(unsigned Reg1, unsigned Reg2,
+                                      bool NeedsWinCFI, bool NeedsFrameRecord) {
+  if (NeedsWinCFI)
+    return invalidateWindowsRegisterPairing(Reg1, Reg2, true);
+
+  // If we need to store the frame record, don't pair any register
+  // with LR other than FP.
+  if (NeedsFrameRecord)
+    return Reg2 == AArch64::LR;
+
+  return false;
+}
+
 namespace {
 
 struct RegPairInfo {
   unsigned Reg1 = AArch64::NoRegister;
   unsigned Reg2 = AArch64::NoRegister;
   int FrameIdx;
   int Offset;
   enum RegType { GPR, FPR64, FPR128 } Type;
 
   RegPairInfo() = default;
 
   bool isPaired() const { return Reg2 != AArch64::NoRegister; }
 };
 
 } // end anonymous namespace
 
 static void computeCalleeSaveRegisterPairs(
     MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI, SmallVectorImpl<RegPairInfo> &RegPairs,
-    bool &NeedShadowCallStackProlog) {
+    bool &NeedShadowCallStackProlog, bool NeedsFrameRecord) {
 
   if (CSI.empty())
     return;
 
   bool NeedsWinCFI = needsWinCFI(MF);
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   CallingConv::ID CC = MF.getFunction().getCallingConv();
else
       llvm_unreachable("Unsupported register class.");
 
     // Add the next reg to the pair if it is in the same register class.
     if (i + 1 < Count) {
       unsigned NextReg = CSI[i + 1].getReg();
       switch (RPI.Type) {
       case RegPairInfo::GPR:
         if (AArch64::GPR64RegClass.contains(NextReg) &&
-            !invalidateWindowsRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI))
+            !invalidateRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI,
+                                       NeedsFrameRecord))
           RPI.Reg2 = NextReg;
         break;
       case RegPairInfo::FPR64:
         if (AArch64::FPR64RegClass.contains(NextReg) &&
             !invalidateWindowsRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI))
           RPI.Reg2 = NextReg;
         break;
       case RegPairInfo::FPR128:
for (unsigned i = 0; i < Count; ++i) {
     // pair instructions directly. Assert if we see anything otherwise.
     //
     // The order of the registers in the list is controlled by
     // getCalleeSavedRegs(), so they will always be in-order, as well.
     assert((!RPI.isPaired() ||
             (CSI[i].getFrameIdx() + 1 == CSI[i + 1].getFrameIdx())) &&
            "Out of order callee saved regs!");
 
+    assert((!RPI.isPaired() || RPI.Reg2 != AArch64::FP ||
+            RPI.Reg1 == AArch64::LR) &&
+           "FrameRecord must be allocated together with LR");
+
     // MachO's compact unwind format relies on all registers being stored in
     // adjacent register pairs.
     assert((!produceCompactUnwindFrame(MF) ||
             CC == CallingConv::PreserveMost ||
             (RPI.isPaired() &&
              ((RPI.Reg1 == AArch64::LR && RPI.Reg2 == AArch64::FP) ||
               RPI.Reg1 + 1 == RPI.Reg2))) &&
            "Callee-save registers not saved as adjacent register pair!");
bool AArch64FrameLowering::spillCalleeSavedRegisters(
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   bool NeedsWinCFI = needsWinCFI(MF);
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
-                                 NeedShadowCallStackProlog);
+                                 NeedShadowCallStackProlog, hasFP(MF));
   const MachineRegisterInfo &MRI = MF.getRegInfo();
 
   if (NeedShadowCallStackProlog) {
     // Shadow call stack prolog: str x30, [x18], #8
     BuildMI(MBB, MI, DL, TII.get(AArch64::STRXpost))
         .addReg(AArch64::X18, RegState::Define)
         .addReg(AArch64::LR)
         .addReg(AArch64::X18)
bool AArch64FrameLowering::restoreCalleeSavedRegisters(
   SmallVector<RegPairInfo, 8> RegPairs;
   bool NeedsWinCFI = needsWinCFI(MF);
 
   if (MI != MBB.end())
     DL = MI->getDebugLoc();
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
-                                 NeedShadowCallStackProlog);
+                                 NeedShadowCallStackProlog, hasFP(MF));
 
   auto EmitMI = [&](const RegPairInfo &RPI) {
     unsigned Reg1 = RPI.Reg1;
     unsigned Reg2 = RPI.Reg2;
 
     // Issue sequence of restores for cs regs. The last restore may be converted
     // to a post-increment load later by emitEpilogue if the callee-save stack
     // area allocation can't be combined with the local stack area allocation.

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.cpp

return MF->getInfo<AArch64FunctionInfo>()->isSplitCSR() ?
            CSR_AArch64_CXX_TLS_Darwin_SaveList;
   if (MF->getSubtarget<AArch64Subtarget>().getTargetLowering()
           ->supportSwiftError() &&
       MF->getFunction().getAttributes().hasAttrSomewhere(
           Attribute::SwiftError))
     return CSR_AArch64_AAPCS_SwiftError_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::PreserveMost)
     return CSR_AArch64_RT_MostRegs_SaveList;
-  else
-    return CSR_AArch64_AAPCS_SaveList;
+  if (MF->getSubtarget<AArch64Subtarget>().isTargetDarwin())
+    return CSR_Darwin_AArch64_AAPCS_SaveList;
 }
 
 const MCPhysReg *AArch64RegisterInfo::getCalleeSavedRegsViaCopy(
     const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
   if (MF->getFunction().getCallingConv() == CallingConv::CXX_FAST_TLS &&
       MF->getInfo<AArch64FunctionInfo>()->isSplitCSR())
     return CSR_AArch64_CXX_TLS_Darwin_ViaCopy_SaveList;

llvm/trunk/test/CodeGen/AArch64/aarch64-dynamic-stack-layout.ll

entry:
   %call = tail call i32 @g()
   %add2 = add nsw i32 %add1, %call
   ret i32 %add2
 }
 ; CHECK-LABEL: novla_nodynamicrealign_call
 ; CHECK: .cfi_startproc
 ;   Check that used callee-saved registers are saved
 ; CHECK: sub  sp, sp, #32
-; CHECK: stp  x19, x30, [sp, #16]
+; CHECK: stp  x30, x19, [sp, #16]
 ;   Check correctness of cfi pseudo-instructions
 ; CHECK: .cfi_def_cfa_offset 32
-; CHECK: .cfi_offset w30, -8
-; CHECK: .cfi_offset w19, -16
+; CHECK: .cfi_offset w19, -8
+; CHECK: .cfi_offset w30, -16
 ;   Check correct access to arguments passed on the stack, through stack pointer
 ; CHECK: ldr  d[[DARG:[0-9]+]], [sp, #56]
 ; CHECK: ldr  w[[IARG:[0-9]+]], [sp, #40]
 ;   Check correct access to local variable on the stack, through stack pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [sp, #12]
 ;   Check epilogue:
-; CHECK: ldp  x19, x30, [sp, #16]
+; CHECK: ldp  x30, x19, [sp, #16]
 ; CHECK: ret
 ; CHECK: .cfi_endproc
 
 ; CHECK-MACHO-LABEL: _novla_nodynamicrealign_call:
 ; CHECK-MACHO: .cfi_startproc
 ;   Check that used callee-saved registers are saved
 ; CHECK-MACHO: sub  sp, sp, #48
 ; CHECK-MACHO: stp  x20, x19, [sp, #16]
entry:
   %call = tail call i32 @g()
   %add2 = add nsw i32 %add1, %call
   ret i32 %add2
 }
 
 ; CHECK-LABEL: novla_dynamicrealign_call
 ; CHECK: .cfi_startproc
 ;   Check that used callee-saved registers are saved
-; CHECK: str  x19, [sp, #-32]!
+; CHECK: stp  x29, x30, [sp, #-32]!
 ;   Check that the frame pointer is created:
-; CHECK: stp  x29, x30, [sp, #16]
-; CHECK: add  x29, sp, #16
+; CHECK: str  x19, [sp, #16]
+; CHECK: mov  x29, sp
 ;   Check the dynamic realignment of the stack pointer to a 128-byte boundary
 ; CHECK: sub  x9, sp, #96
 ; CHECK: and  sp, x9, #0xffffffffffffff80
 ;   Check correctness of cfi pseudo-instructions
-; CHECK: .cfi_def_cfa w29, 16
-; CHECK: .cfi_offset w30, -8
-; CHECK: .cfi_offset w29, -16
-; CHECK: .cfi_offset w19, -32
+; CHECK: .cfi_def_cfa w29, 32
+; CHECK: .cfi_offset w19, -16
+; CHECK: .cfi_offset w30, -24
+; CHECK: .cfi_offset w29, -32
 ;   Check correct access to arguments passed on the stack, through frame pointer
-; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
-; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #56]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #40]
 ;   Check correct access to local variable on the stack, through re-aligned stack pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [sp]
 ;   Check epilogue:
 ;     Check that stack pointer get restored from frame pointer.
-; CHECK: sub  sp, x29, #16            // =16
-; CHECK: ldp  x29, x30, [sp, #16]
-; CHECK: ldr  x19, [sp], #32
+; CHECK: mov  sp, x29
+; CHECK: ldr  x19, [sp, #16]
+; CHECK: ldp  x29, x30, [sp], #32
 ; CHECK: ret
 ; CHECK: .cfi_endproc
 
 ; CHECK-MACHO-LABEL: _novla_dynamicrealign_call:
 ; CHECK-MACHO: .cfi_startproc
 ;   Check that used callee-saved registers are saved
 ; CHECK-MACHO: stp  x20, x19, [sp, #-32]!
 ;   Check that the frame pointer is created:
entry:
   %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
   %add3 = add nsw i32 %add2, %1
   ret i32 %add3
 }
 
 ; CHECK-LABEL: vla_nodynamicrealign_call
 ; CHECK: .cfi_startproc
 ;   Check that used callee-saved registers are saved
-; CHECK: stp  x20, x19, [sp, #-32]!
+; CHECK: stp  x29, x30, [sp, #-32]!
 ;   Check that the frame pointer is created:
-; CHECK: stp  x29, x30, [sp, #16]
-; CHECK: add  x29, sp, #16
+; CHECK: stp  x20, x19, [sp, #16]
+; CHECK: mov  x29, sp
 ;   Check that space is reserved on the stack for the local variable,
 ;   rounded up to a multiple of 16 to keep the stack pointer 16-byte aligned.
 ; CHECK: sub  sp, sp, #16
 ;   Check correctness of cfi pseudo-instructions
-; CHECK: .cfi_def_cfa w29, 16
-; CHECK: .cfi_offset w30, -8
-; CHECK: .cfi_offset w29, -16
-; CHECK: .cfi_offset w19, -24
-; CHECK: .cfi_offset w20, -32
+; CHECK: .cfi_def_cfa w29, 32
+; CHECK: .cfi_offset w19, -8
+; CHECK: .cfi_offset w20, -16
+; CHECK: .cfi_offset w30, -24
+; CHECK: .cfi_offset w29, -32
 ;   Check correct access to arguments passed on the stack, through frame pointer
-; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
-; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #56]
 ;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
 ; CHECK: mov  w9, w0
 ; CHECK: mov   x10, sp
 ; CHECK: lsl  x9, x9, #2
 ; CHECK: add  x9, x9, #15
 ; CHECK: and  x9, x9, #0x7fffffff0
 ; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
 ; CHECK: mov   sp, x[[VLASPTMP]]
 ;   Check correct access to local variable, through frame pointer
-; CHECK: ldur w[[ILOC:[0-9]+]], [x29, #-20]
+; CHECK: ldur w[[ILOC:[0-9]+]], [x29, #-4]
 ;   Check correct accessing of the VLA variable through the base pointer
 ; CHECK: ldr  w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
 ;   Check epilogue:
 ;     Check that stack pointer get restored from frame pointer.
-; CHECK: sub  sp, x29, #16            // =16
-; CHECK: ldp  x29, x30, [sp, #16]
-; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: mov  sp, x29
+; CHECK: ldp  x20, x19, [sp, #16]
+; CHECK: ldp  x29, x30, [sp], #32
 ; CHECK: ret
 ; CHECK: .cfi_endproc
 
 
 ; Function Attrs: nounwind
 define i32 @vla_nodynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
 entry:
   %l1 = alloca i32, align 4
entry:
   %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
   %add3 = add nsw i32 %add2, %1
   ret i32 %add3
 }
 
 ; CHECK-LABEL: vla_dynamicrealign_call
 ; CHECK: .cfi_startproc
 ;   Check that used callee-saved registers are saved
-; CHECK: str  x21, [sp, #-48]!
-; CHECK: stp  x20, x19, [sp, #16]
+; CHECK: stp  x29, x30, [sp, #-48]!
+; CHECK: str  x21, [sp, #16]
+; CHECK: stp  x20, x19, [sp, #32]
 ;   Check that the frame pointer is created:
-; CHECK: stp  x29, x30, [sp, #32]
-; CHECK: add  x29, sp, #32
+; CHECK: mov  x29, sp
 ;   Check that the stack pointer gets re-aligned to 128
 ;   bytes & the base pointer (x19) gets initialized to
 ;   this 128-byte aligned area for local variables &
 ;   spill slots
 ; CHECK: sub  x9, sp, #80            // =80
 ; CHECK: and  sp, x9, #0xffffffffffffff80
 ; CHECK: mov    x19, sp
 ;   Check correctness of cfi pseudo-instructions
-; CHECK: .cfi_def_cfa w29, 16
-; CHECK: .cfi_offset w30, -8
-; CHECK: .cfi_offset w29, -16
-; CHECK: .cfi_offset w19, -24
-; CHECK: .cfi_offset w20, -32
-; CHECK: .cfi_offset w21, -48
+; CHECK: .cfi_def_cfa w29, 48
+; CHECK: .cfi_offset w19, -8
+; CHECK: .cfi_offset w20, -16
+; CHECK: .cfi_offset w21, -32
+; CHECK: .cfi_offset w30, -40
+; CHECK: .cfi_offset w29, -48
 ;   Check correct access to arguments passed on the stack, through frame pointer
-; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
-; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #56]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #72]
 ;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
 ;   and set-up of base pointer (x19).
 ; CHECK: mov  w9, w0
 ; CHECK: mov   x10, sp
 ; CHECK: lsl  x9, x9, #2
 ; CHECK: add  x9, x9, #15
 ; CHECK: and  x9, x9, #0x7fffffff0
 ; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
 ; CHECK: mov   sp, x[[VLASPTMP]]
 ;   Check correct access to local variable, through base pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
 ; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
 ;   Check epilogue:
 ;     Check that stack pointer get restored from frame pointer.
-; CHECK: sub  sp, x29, #32
-; CHECK: ldp  x29, x30, [sp, #32]
-; CHECK: ldp  x20, x19, [sp, #16]
-; CHECK: ldr  x21, [sp], #48
+; CHECK: mov  sp, x29
+; CHECK: ldp  x20, x19, [sp, #32]
+; CHECK: ldr  x21, [sp, #16]
+; CHECK: ldp  x29, x30, [sp], #48
 ; CHECK: ret
 ; CHECK: .cfi_endproc
 
 ; CHECK-MACHO-LABEL: _vla_dynamicrealign_call:
 ; CHECK-MACHO: .cfi_startproc
 ;   Check that used callee-saved registers are saved
 ; CHECK-MACHO: stp  x22, x21, [sp, #-48]!
 ; CHECK-MACHO: stp  x20, x19, [sp, #16]
entry:
   %add1 = add nsw i32 %add, %l1.0.l1.0.
   %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
   %add2 = add nsw i32 %add1, %1
   ret i32 %add2
 }
 
 ; CHECK-LABEL: vla_dynamicrealign_nocall
 ;   Check that used callee-saved registers are saved
-; CHECK: str  x19, [sp, #-32]!
+; CHECK: stp  x29, x30, [sp, #-32]!
+; CHECK: str  x19, [sp, #16]
 ;   Check that the frame pointer is created:
-; CHECK: stp  x29, x30, [sp, #16]
-; CHECK: add  x29, sp, #16
+; CHECK: mov  x29, sp
 ;   Check that the stack pointer gets re-aligned to 128
 ;   bytes & the base pointer (x19) gets initialized to
 ;   this 128-byte aligned area for local variables &
 ;   spill slots
 ; CHECK: sub  x9, sp, #96
 ; CHECK: and  sp, x9, #0xffffffffffffff80
 ; CHECK: mov    x19, sp
 ;   Check correct access to arguments passed on the stack, through frame pointer
-; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
-; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #56]
 ;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
 ;   and set-up of base pointer (x19).
 ; CHECK: mov  w9, w0
 ; CHECK: mov   x10, sp
 ; CHECK: lsl  x9, x9, #2
 ; CHECK: add  x9, x9, #15
 ; CHECK: and  x9, x9, #0x7fffffff0
 ; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
 ; CHECK: mov   sp, x[[VLASPTMP]]
 ;   Check correct access to local variable, through base pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
 ; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
 ;   Check epilogue:
 ;     Check that stack pointer get restored from frame pointer.
-; CHECK: sub  sp, x29, #16
-; CHECK: ldp  x29, x30, [sp, #16]
-; CHECK: ldr  x19, [sp], #32
+; CHECK: mov  sp, x29
+; CHECK: ldr  x19, [sp, #16]
+; CHECK: ldp  x29, x30, [sp], #32
 ; CHECK: ret
 
 ; CHECK-MACHO-LABEL: _vla_dynamicrealign_nocall:
 ;   Check that used callee-saved registers are saved
 ; CHECK-MACHO: stp  x20, x19, [sp, #-32]!
 ;   Check that the frame pointer is created:
 ; CHECK-MACHO: stp  x29, x30, [sp, #16]
 ; CHECK-MACHO: add  x29, sp, #16
entry:
   %add1 = add nsw i32 %add, %l1.0.l1.0.
   %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
   %add2 = add nsw i32 %add1, %1
   ret i32 %add2
 }
 
 ; CHECK-LABEL: vla_dynamicrealign_nocall_large_align
 ;   Check that used callee-saved registers are saved
-; CHECK: stp  x28, x19, [sp, #-32]!
+; CHECK: stp  x29, x30, [sp, #-32]!
+; CHECK: str  x19, [sp, #16]
 ;   Check that the frame pointer is created:
-; CHECK: stp  x29, x30, [sp, #16]
-; CHECK: add  x29, sp, #16
+; CHECK: mov  x29, sp
 ;   Check that the stack pointer gets re-aligned to 128
 ;   bytes & the base pointer (x19) gets initialized to
 ;   this 128-byte aligned area for local variables &
 ;   spill slots
 ; CHECK: sub  x9, sp, #7, lsl #12
 ; CHECK: and  sp, x9, #0xffffffffffff8000
 ; CHECK: mov    x19, sp
 ;   Check correct access to arguments passed on the stack, through frame pointer
-; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
-; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #56]
 ;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
 ;   and set-up of base pointer (x19).
 ; CHECK: mov  w9, w0
 ; CHECK: mov   x10, sp
 ; CHECK: lsl  x9, x9, #2
 ; CHECK: add  x9, x9, #15
 ; CHECK: and  x9, x9, #0x7fffffff0
 ; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
 ; CHECK: mov   sp, x[[VLASPTMP]]
 ;   Check correct access to local variable, through base pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
 ; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
 ;   Check epilogue:
 ;     Check that stack pointer get restored from frame pointer.
-; CHECK: sub  sp, x29, #16
-; CHECK: ldp  x29, x30, [sp, #16]
-; CHECK: ldp  x28, x19, [sp], #32
+; CHECK: mov  sp, x29
+; CHECK: ldr  x19, [sp, #16]
+; CHECK: ldp  x29, x30, [sp], #32
 ; CHECK: ret
 
 ; CHECK-MACHO-LABEL: _vla_dynamicrealign_nocall_large_align:
 ;   Check that used callee-saved registers are saved
 ; CHECK-MACHO: stp  x20, x19, [sp, #-32]!
 ;   Check that the frame pointer is created:
 ; CHECK-MACHO: stp  x29, x30, [sp, #16]
 ; CHECK-MACHO: add  x29, sp, #16

llvm/trunk/test/CodeGen/AArch64/aarch64-vector-pcs.mir

body:             |
   ; CHECK-NEXT: frame-setup STPQi killed $q23, killed $q22, $sp, 1 :: (store 16 into %stack.{{[0-9]+}}), (store 16 into %stack.{{[0-9]+}})
   ; CHECK-NEXT: frame-setup STPQi killed $q21, killed $q20, $sp, 3
   ; CHECK-NEXT: frame-setup STPQi killed $q19, killed $q18, $sp, 5
   ; CHECK-NEXT: frame-setup STPQi killed $q17, killed $q16, $sp, 7
   ; CHECK-NEXT: frame-setup STPQi killed $q15, killed $q14, $sp, 9
   ; CHECK-NEXT: frame-setup STPQi killed $q13, killed $q12, $sp, 11
   ; CHECK-NEXT: frame-setup STPQi killed $q11, killed $q10, $sp, 13
   ; CHECK-NEXT: frame-setup STPQi killed $q9, killed $q8, $sp, 15
-  ; CHECK-NEXT: frame-setup STPXi killed $x28, killed $x27, $sp, 34 :: (store 8 into %stack.{{[0-9]+}}), (store 8 into %stack.{{[0-9]+}})
-  ; CHECK-NEXT: frame-setup STPXi killed $x26, killed $x25, $sp, 36
-  ; CHECK-NEXT: frame-setup STPXi killed $x24, killed $x23, $sp, 38
-  ; CHECK-NEXT: frame-setup STPXi killed $x22, killed $x21, $sp, 40
-  ; CHECK-NEXT: frame-setup STPXi killed $x20, killed $x19, $sp, 42
-  ; CHECK-NEXT: frame-setup STPXi killed $fp, killed $lr, $sp, 44
+  ; CHECK-NEXT: frame-setup STPXi killed $fp, killed $lr, $sp, 34 :: (store 8 into %stack.{{[0-9]+}}), (store 8 into %stack.{{[0-9]+}})
+  ; CHECK-NEXT: frame-setup STPXi killed $x28, killed $x27, $sp, 36
+  ; CHECK-NEXT: frame-setup STPXi killed $x26, killed $x25, $sp, 38
+  ; CHECK-NEXT: frame-setup STPXi killed $x24, killed $x23, $sp, 40
+  ; CHECK-NEXT: frame-setup STPXi killed $x22, killed $x21, $sp, 42
+  ; CHECK-NEXT: frame-setup STPXi killed $x20, killed $x19, $sp, 44
 
 ...
 ---
 name:            test_q8_to_q23_x19_to_x30_preinc
 tracksRegLiveness: true
 stack:
   - { id: 0, size: 160, alignment: 4, local-offset: 0 }
 constants:
body:             |
   ; CHECK: $sp = frame-setup STPQpre killed $q23, killed $q22, $sp, -22 :: (store 16 into %stack.{{[0-9]+}}), (store 16 into %stack.{{[0-9]+}})
   ; CHECK-NEXT: frame-setup STPQi killed $q21, killed $q20, $sp, 2
   ; CHECK-NEXT: frame-setup STPQi killed $q19, killed $q18, $sp, 4
   ; CHECK-NEXT: frame-setup STPQi killed $q17, killed $q16, $sp, 6
   ; CHECK-NEXT: frame-setup STPQi killed $q15, killed $q14, $sp, 8
   ; CHECK-NEXT: frame-setup STPQi killed $q13, killed $q12, $sp, 10
   ; CHECK-NEXT: frame-setup STPQi killed $q11, killed $q10, $sp, 12
   ; CHECK-NEXT: frame-setup STPQi killed $q9, killed $q8, $sp, 14
-  ; CHECK-NEXT: frame-setup STPXi killed $x28, killed $x27, $sp, 32 :: (store 8 into %stack.{{[0-9]+}}), (store 8 into %stack.{{[0-9]+}})
-  ; CHECK-NEXT: frame-setup STPXi killed $x26, killed $x25, $sp, 34
-  ; CHECK-NEXT: frame-setup STPXi killed $x24, killed $x23, $sp, 36
-  ; CHECK-NEXT: frame-setup STPXi killed $x22, killed $x21, $sp, 38
-  ; CHECK-NEXT: frame-setup STPXi killed $x20, killed $x19, $sp, 40
-  ; CHECK-NEXT: frame-setup STPXi killed $fp, killed $lr, $sp, 42
+  ; CHECK-NEXT: frame-setup STPXi killed $fp, killed $lr, $sp, 32 :: (store 8 into %stack.{{[0-9]+}}), (store 8 into %stack.{{[0-9]+}})
+  ; CHECK-NEXT: frame-setup STPXi killed $x28, killed $x27, $sp, 34
+  ; CHECK-NEXT: frame-setup STPXi killed $x26, killed $x25, $sp, 36
+  ; CHECK-NEXT: frame-setup STPXi killed $x24, killed $x23, $sp, 38
+  ; CHECK-NEXT: frame-setup STPXi killed $x22, killed $x21, $sp, 40
+  ; CHECK-NEXT: frame-setup STPXi killed $x20, killed $x19, $sp, 42
   ; CHECK-NEXT: $sp = frame-setup SUBXri $sp, 176, 0
 
 ...

llvm/trunk/test/CodeGen/AArch64/addsub-constant-folding.ll

; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @add_const_add_const_extrause(i32 %arg) {
 ; CHECK-LABEL: add_const_add_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    add w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    add w0, w19, #10 // =10
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_add_const_add_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @add_const_sub_const_extrause(i32 %arg) {
 ; CHECK-LABEL: add_const_sub_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    add w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    add w0, w19, #6 // =6
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_add_const_sub_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define i32 @add_const_const_sub_extrause(i32 %arg) {
 ; CHECK-LABEL: add_const_const_sub_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    add w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    mov w8, #-6
 ; CHECK-NEXT:    sub w0, w8, w19
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = add i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define <4 x i32> @vec_add_const_const_sub(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @sub_const_add_const_extrause(i32 %arg) {
 ; CHECK-LABEL: sub_const_add_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    sub w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    sub w0, w19, #6 // =6
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_sub_const_add_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @sub_const_sub_const_extrause(i32 %arg) {
 ; CHECK-LABEL: sub_const_sub_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    sub w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    sub w0, w19, #10 // =10
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_sub_const_sub_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define i32 @sub_const_const_sub_extrause(i32 %arg) {
 ; CHECK-LABEL: sub_const_const_sub_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    sub w0, w0, #8 // =8
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    mov w8, #10
 ; CHECK-NEXT:    sub w0, w8, w19
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 %arg, 8
   call void @use(i32 %t0)
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define <4 x i32> @vec_sub_const_const_sub(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @const_sub_add_const_extrause(i32 %arg) {
 ; CHECK-LABEL: const_sub_add_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w8, #8
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    sub w0, w8, w0
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    mov w8, #10
 ; CHECK-NEXT:    sub w0, w8, w19
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   call void @use(i32 %t0)
   %t1 = add i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_const_sub_add_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define i32 @const_sub_sub_const_extrause(i32 %arg) {
 ; CHECK-LABEL: const_sub_sub_const_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w8, #8
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    sub w0, w8, w0
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    mov w8, #6
 ; CHECK-NEXT:    sub w0, w8, w19
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   call void @use(i32 %t0)
   %t1 = sub i32 %t0, 2
   ret i32 %t1
 }
 
 define <4 x i32> @vec_const_sub_sub_const(<4 x i32> %arg) {
; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define i32 @const_sub_const_sub_extrause(i32 %arg) {
 ; CHECK-LABEL: const_sub_const_sub_extrause:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    stp x19, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w19, -16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov w8, #8
 ; CHECK-NEXT:    sub w19, w8, w0
 ; CHECK-NEXT:    mov w0, w19
 ; CHECK-NEXT:    bl use
 ; CHECK-NEXT:    mov w8, #2
 ; CHECK-NEXT:    sub w0, w8, w19
-; CHECK-NEXT:    ldp x19, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    ret
   %t0 = sub i32 8, %arg
   call void @use(i32 %t0)
   %t1 = sub i32 2, %t0
   ret i32 %t1
 }
 
 define <4 x i32> @vec_const_sub_const_sub(<4 x i32> %arg) {

llvm/trunk/test/CodeGen/AArch64/alloca.ll

 ; CHECK-NOFP-ARM64: ldp x29, x30, [sp], #16
 }
 
 define void @test_alloca_large_frame(i64 %n) {
 ; CHECK-LABEL: test_alloca_large_frame:
 ; CHECK-MACHO-LABEL: test_alloca_large_frame:
 
 
-; CHECK: stp     x28, x19, [sp, #-32]!
-; CHECK: stp     x29, x30, [sp, #16]
-; CHECK: add     x29, sp, #16
+; CHECK: stp     x29, x30, [sp, #-32]!
+; CHECK: stp     x28, x19, [sp, #16]
+; CHECK: mov     x29, sp
 ; CHECK: sub     sp, sp, #1953, lsl #12
 ; CHECK: sub     sp, sp, #512
 
 ; CHECK-MACHO: stp     x20, x19, [sp, #-32]!
 ; CHECK-MACHO: stp     x29, x30, [sp, #16]
 ; CHECK-MACHO: add     x29, sp, #16
 ; CHECK-MACHO: sub     sp, sp, #1953, lsl #12
 ; CHECK-MACHO: sub     sp, sp, #512
 
   %addr1 = alloca i8, i64 %n
   %addr2 = alloca i64, i64 1000000
 
   call void @use_addr_loc(i8* %addr1, i64* %addr2)
 
   ret void
 
-; CHECK: sub     sp, x29, #16
-; CHECK: ldp     x29, x30, [sp, #16]
-; CHECK: ldp     x28, x19, [sp], #32
+; CHECK: mov     sp, x29
+; CHECK: ldp     x28, x19, [sp, #16]
+; CHECK: ldp     x29, x30, [sp], #32
 
 ; CHECK-MACHO: sub     sp, x29, #16
 ; CHECK-MACHO: ldp     x29, x30, [sp, #16]
 ; CHECK-MACHO: ldp     x20, x19, [sp], #32
 }
 
 declare i8* @llvm.stacksave()
 declare void @llvm.stackrestore(i8*)
 
 define void @test_scoped_alloca(i64 %n) {
 ; CHECK-LABEL: test_scoped_alloca:
 
   %sp = call i8* @llvm.stacksave()
+; CHECK: mov x29, sp
 ; CHECK: mov [[SAVED_SP:x[0-9]+]], sp
 ; CHECK: mov [[OLDSP:x[0-9]+]], sp
 
   %addr = alloca i8, i64 %n
 ; CHECK: and [[SPDELTA:x[0-9]+]], {{x[0-9]+}}, #0xfffffffffffffff0
 ; CHECK-DAG: sub [[NEWSP:x[0-9]+]], [[OLDSP]], [[SPDELTA]]
 ; CHECK: mov sp, [[NEWSP]]
 
   call void @use_addr(i8* %addr)
 ; CHECK: bl use_addr
 
   call void @llvm.stackrestore(i8* %sp)
 ; CHECK: mov sp, [[SAVED_SP]]
 
   ret void
 }

llvm/trunk/test/CodeGen/AArch64/arm64-alloca-frame-pointer-offset.ll

 ; RUN: llc -mtriple=arm64-eabi -mcpu=cyclone < %s | FileCheck %s
 
 ; CHECK: foo
-; CHECK-DAG: stur w[[REG0:[0-9]+]], [x29, #-24]
-; CHECK-DAG: stur w[[REG0]], [x29, #-20]
+; CHECK-DAG: str w[[REG0:[0-9]+]], [x29, #24]
+; CHECK-DAG: str w[[REG0]], [x29, #28]
 define i32 @foo(i32 %a) nounwind {
   %retval = alloca i32, align 4
   %a.addr = alloca i32, align 4
   %arr = alloca [32 x i32], align 4
   %i = alloca i32, align 4
   %arr2 = alloca [32 x i32], align 4
   %j = alloca i32, align 4
   store i32 %a, i32* %a.addr, align 4

llvm/trunk/test/CodeGen/AArch64/cgp-usubo.ll

 end:
   %ov = icmp ult i64 %x, %y
   ret i1 %ov
 }
 
 define i1 @usubo_ult_cmp_dominates_i64(i64 %x, i64 %y, i64* %p, i1 %cond) nounwind {
 ; CHECK-LABEL: usubo_ult_cmp_dominates_i64:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    str x22, [sp, #-48]! // 8-byte Folded Spill
-; CHECK-NEXT:    stp x21, x20, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-48]! // 8-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #32] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w20, w3
-; CHECK-NEXT:    stp x19, x30, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x22, x21, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    tbz w3, #0, .LBB8_3
 ; CHECK-NEXT:  // %bb.1: // %t
 ; CHECK-NEXT:    cmp x0, x1
 ; CHECK-NEXT:    mov x22, x0
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    mov x19, x2
 ; CHECK-NEXT:    mov x21, x1
 ; CHECK-NEXT:    bl call
 ; CHECK-NEXT:    subs x8, x22, x21
 ; CHECK-NEXT:    b.hs .LBB8_3
 ; CHECK-NEXT:  // %bb.2: // %end
 ; CHECK-NEXT:    cset w0, lo
 ; CHECK-NEXT:    str x8, [x19]
 ; CHECK-NEXT:    b .LBB8_4
 ; CHECK-NEXT:  .LBB8_3: // %f
 ; CHECK-NEXT:    and w0, w20, #0x1
 ; CHECK-NEXT:  .LBB8_4: // %f
-; CHECK-NEXT:    ldp x19, x30, [sp, #32] // 16-byte Folded Reload
-; CHECK-NEXT:    ldp x21, x20, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x22, [sp], #48 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    ldp x22, x21, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #48 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
 entry:
   br i1 %cond, label %t, label %f
 
 t:
   %ov = icmp ult i64 %x, %y
   call void @call(i1 %ov)
   br i1 %ov, label %end, label %f

llvm/trunk/test/CodeGen/AArch64/fast-isel-sp-adjust.ll

 ; RUN: llc -O0 -fast-isel -mtriple=aarch64-apple-ios -o - %s | FileCheck %s
 ; RUN: not llc -O0 -mtriple=aarch64-apple-ios -o /dev/null -fast-isel -fast-isel-abort=3 %s 2> %t
 ; RUN: FileCheck %s --check-prefix=CHECK-ERRORS < %t
 
 ; The issue here is that FastISel cannot emit an ADDrr where one of the inputs
 ; is SP. This only ever crops up with function calls, and then only if the
 ; argument is at an offset > 2^12 * size from SP.
 
 ; If FastISel ever starts coping with this and emits an "add xD, sp, xM" it's
 ; critical to check the encoding as well as the textual assembly. An ADDXrs with
 ; SP as an operand will still print with SP, but will actually mean XZR.
 
 ; CHECK-ERRORS: LLVM ERROR: FastISel missed call
 
 ; CHECK-LABEL: foo:
+; CHECK: sub
 ; CHECK-DAG: mov x[[SP:[0-9]+]], sp
 ; CHECK-DAG: mov [[TMP:w[0-9]+]], #4104
 ; CHECK: mov w[[OFFSET:[0-9]+]], [[TMP]]
 ; CHECK: strb w0, [x[[SP]], x[[OFFSET]]]
 
 define void @foo(i8 %in) {
   call void @bar(i64 undef, i64 undef, i64 undef, i64 undef,
                  i64 undef, i64 undef, i64 undef, i64 undef, ; All regs gone.

llvm/trunk/test/CodeGen/AArch64/irg_sp_tagp.ll

; CHECK:      bl use2
   ret void
 }
 
 ; Realigned stack frame. IRG uses value of SP after realignment,
 ; ADDG for the first stack allocation has offset 0.
 define void @realign() {
 entry:
 ; CHECK-LABEL: realign:
-; CHECK:      add  x29, sp, #16
+; CHECK:      mov  x29, sp
 ; CHECK:      and  sp, x{{[0-9]*}}, #0xffffffffffffffc0
 ; CHECK:      irg  [[R:x[0-9]+]], sp{{$}}
 ; CHECK:      addg x0, [[R]], #0, #1
 ; CHECK:      bl use
   %a = alloca i8, i64 4096, align 64
   %base = call i8* @llvm.aarch64.irg.sp(i64 0)
   %a_t = call i8* @llvm.aarch64.tagp.p0i8(i8* %a, i8* %base, i64 1)
   call void @use(i8* %a_t)

llvm/trunk/test/CodeGen/AArch64/regress-w29-reserved-with-fp.ll

 ; RUN: llc -mtriple=aarch64-none-linux-gnu -frame-pointer=all < %s | FileCheck %s
 @var = global i32 0
 
 declare void @bar()
 
 define void @test_w29_reserved() {
 ; CHECK-LABEL: test_w29_reserved:
-; CHECK: add x29, sp, #{{[0-9]+}}
+; CHECK: mov x29, sp
 
   %val1 = load volatile i32, i32* @var
   %val2 = load volatile i32, i32* @var
   %val3 = load volatile i32, i32* @var
   %val4 = load volatile i32, i32* @var
   %val5 = load volatile i32, i32* @var
   %val6 = load volatile i32, i32* @var
   %val7 = load volatile i32, i32* @var

llvm/trunk/test/CodeGen/AArch64/reverse-csr-restore-seq.mir

bb.0:
     $x21 = IMPLICIT_DEF
 
     ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
     BL @foo, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit $x0, implicit-def $sp, implicit-def $x0
     ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
     B %bb.1
 
   bb.1:
-   ; CHECK: $x20, $lr = frame-destroy LDPXi $sp, 2
-   ; BEFORELDSTOPT-NEXT: $x21 = frame-destroy LDRXui $sp, 0
+   ; CHECK: $x21, $x20 = frame-destroy LDPXi $sp, 2
+   ; BEFORELDSTOPT-NEXT: $lr = frame-destroy LDRXui $sp, 0
    ; BEFORELDSTOPT-NEXT: $sp = frame-destroy ADDXri $sp, 32, 0
 
-   ; AFTERLDSTOPT-NEXT: early-clobber $sp, $x21 = frame-destroy LDRXpost $sp, 32
+   ; AFTERLDSTOPT-NEXT: early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 32
     RET_ReallyLR
 ...

llvm/trunk/test/CodeGen/AArch64/seh-finally.ll

; CHECK: bl      foo
   call void @foo(i32 %1)
   ret void
 }
 
 ; Test SEH when stack realignment is needed in case highly aligned stack objects are present.
 define void @stack_realign() #0 personality i8* bitcast (i32 (...)* @__C_specific_handler to i8*) {
 entry:
 ; CHECK-LABEL: stack_realign
-; CHECK: add     x29, sp, #16
+; CHECK: mov     x29, sp
 ; CHECK: sub     x9, sp, #64
 ; CHECK: and     sp, x9, #0xffffffffffffffe0
 ; CHECK: mov     x19, sp
 ; CHECK: mov     x0, #-2
 ; CHECK: stur    x0, [x19, #16]
 ; CHECK: .set .Lstack_realign$frame_escape_0, 32
 ; CHECK: ldr     w0, [x19, #32]
 ; CHECK: bl      foo
; CHECK: bl      foo
   call void @foo(i32 %1)
   ret void
 }
 
 ; Test when both vla's and highly aligned objects are present on stack.
 define void @vla_and_realign(i32 %n) #0 personality i8* bitcast (i32 (...)* @__C_specific_handler to i8*) {
 entry:
 ; CHECK-LABEL: vla_and_realign
-; CHECK: add     x29, sp, #16
+; CHECK: mov     x29, sp
 ; CHECK: sub     x9, sp, #64
 ; CHECK: and     sp, x9, #0xffffffffffffffe0
 ; CHECK: mov     x19, sp
 ; CHECK: mov     x1, #-2
 ; CHECK: stur    x1, [x19]
 ; CHECK: .set .Lvla_and_realign$frame_escape_0, 32
-; CHECK: stur    w0, [x29, #-4]
-; CHECK: ldur    w8, [x29, #-4]
+; CHECK: str     w0, [x29, #28]
+; CHECK: ldr     w8, [x29, #28]
 ; CHECK: mov     x9, sp
 ; CHECK: str     x9, [x19, #24]
 ; CHECK: str     x8, [x19, #16]
 ; CHECK: ldr     w0, [x19, #32]
 ; CHECK: bl      foo
 
   %n.addr = alloca i32, align 4
   %o = alloca %struct.S, align 32

llvm/trunk/test/CodeGen/AArch64/shadow-call-stack.ll

define i32 @f4() shadowcallstack {
   ; CHECK: f4:
   %res1 = call i32 @bar()
   %res2 = call i32 @bar()
   %res3 = call i32 @bar()
   %res4 = call i32 @bar()
   %res12 = add i32 %res1, %res2
   %res34 = add i32 %res3, %res4
   %res1234 = add i32 %res12, %res34
-  ; CHECK: ldp {{.*}}x30, [sp
+  ; CHECK: ldp x30,{{.*}}, [sp
   ; CHECK: ldr x30, [x18, #-8]!
   ; CHECK: ret
   ret i32 %res1234
 }
 
 define i32 @f5() shadowcallstack nounwind {
   ; CHECK: f5:
   ; CHECK-NOT: .cfi_escape
   %res = call i32 @bar()
   %res1 = add i32 %res, 1
   ret i32 %res
 }

llvm/trunk/test/CodeGen/AArch64/sink-copy-for-shrink-wrap.ll

 ; RUN: llc -mtriple=aarch64-linux-gnu -o - %s | FileCheck %s
 
 ; CHECK-LABEL: %bb.0:
 ; CHECK-NOT: stp
 ; CHECK-NOT: mov w{{[0-9]+}}, w0
 ; CHECK-LABEL: %bb.1:
-; CHECK: stp x19
+; CHECK: stp {{.*}}x19
 ; CHECK: mov w{{[0-9]+}}, w0
 
 define i32 @shrinkwrapme(i32 %paramAcrossCall, i32 %paramNotAcrossCall) {
 entry:
   %cmp5 = icmp sgt i32 %paramNotAcrossCall, 0
   br i1 %cmp5, label %CallBB, label %Exit
 CallBB:
   %call = call i32 @fun()
   %add = add i32 %call, %paramAcrossCall
   ret i32 %add
 Exit:
   ret i32 0
 }
 
 declare i32 @fun()

llvm/trunk/test/CodeGen/AArch64/spill-stack-realignment.mir

 # RUN: llc -mtriple=aarch64-none-linux-gnu -run-pass=prologepilog %s -o - | FileCheck %s
 
 # Ensure references to scavenged stack slots in the CSR area use the
 # FP as a base when the stack pointer must be aligned to something
 # larger than required by the target. This is necessary because the
 # alignment padding area is between the CSR area and the SP, so the SP
 # cannot be used to reference the CSR area.
 name:            test
 tracksRegLiveness: true
 frameInfo:
   maxAlignment:    64
 # CHECK:      stack:
 # CHECK:        id: 0, name: '', type: default, offset: -64, size: 4, alignment: 64
 # CHECK-NEXT:     stack-id: default
 # CHECK-NEXT:     local-offset: -64
-# CHECK:        id: 1, name: '', type: default, offset: -20, size: 4, alignment: 4
+# CHECK:        id: 1, name: '', type: default, offset: -4, size: 4, alignment: 4
 # CHECK-NEXT:     stack-id: default
 # CHECK-NEXT:     local-offset: -68
 stack:
   - { id: 0, size: 4, alignment: 64, local-offset: -64 }
   - { id: 1, size: 4, alignment: 4, local-offset: -68 }
 
 # CHECK: body:
 # CHECK:   $sp = ANDXri killed ${{x[0-9]+}}, 7865
 # CHECK:   STRSui $s0, $sp, 0
-# CHECK:   STURSi $s0, $fp, -4
+# CHECK:   STRSui $s0, $fp, 7
 body:             |
   bb.0.entry:
     liveins: $s0
 
     STRSui $s0, %stack.0, 0
     STRSui $s0, %stack.1, 0
     ; Force preserve a CSR to create a hole in the CSR stack region.
     $x28 = IMPLICIT_DEF
     RET_ReallyLR

llvm/trunk/test/CodeGen/AArch64/sponentry.ll

define dso_local void @foo([24 x i64]*) {
   %8 = bitcast i64* %7 to i8*
   %9 = call i32 @_setjmpex(i8* %8, i8* %5)
   store i32 %9, i32* %3, align 4
   ret void
 }
 
 ; CHECK: foo:
 ; CHECK: sub     sp, sp, #448
-; CHECK: add     x29, sp, #432
-; CHECK: add     x1, x29, #16
+; CHECK: add     x29, sp, #416
+; CHECK: add     x1, x29, #32
 ; CHECK: bl      _setjmpex
 
 ; NOFP: sub     sp, sp, #432
 ; NOFP: add     x1, sp, #432
 
 define dso_local void @var_args(i8*, ...) {
   %2 = alloca i8*, align 8
   %3 = alloca i8*, align 8

llvm/trunk/test/CodeGen/AArch64/stack-guard-reassign.ll

 ; RUN: llc -O0 --frame-pointer=all -mtriple=aarch64-- -o - %S/../Inputs/stack-guard-reassign.ll | FileCheck %s
 
 ; Verify that the offset assigned to the stack protector is at the top of the
 ; frame, covering the locals.
 ; CHECK-LABEL: fn:
 ; CHECK:      adrp x8, __stack_chk_guard
 ; CHECK-NEXT: ldr x8, [x8, :lo12:__stack_chk_guard]
-; CHECK-NEXT: stur x8, [x29, #-24]
+; CHECK-NEXT: stur x8, [x29, #-8]

llvm/trunk/test/CodeGen/AArch64/stack-guard-vaarg.ll

 ; RUN: llc --frame-pointer=all -mtriple=aarch64-- < %s | FileCheck %s
 
 ; PR25610: -fstack-protector places the canary in the wrong place on arm64 with
 ;          va_args
 
 %struct.__va_list = type { i8*, i8*, i8*, i32, i32 }
 
 ; CHECK-LABEL: test
 ; CHECK: ldr [[GUARD:x[0-9]+]]{{.*}}:lo12:__stack_chk_guard]
 ; Make sure the canary is placed relative to the frame pointer, not
 ; the stack pointer.
-; CHECK: stur [[GUARD]], [x29, #-24]
+; CHECK: stur [[GUARD]], [x29, #-8]
 define void @test(i8* %i, ...) #0 {
 entry:
   %buf = alloca [10 x i8], align 1
   %ap = alloca %struct.__va_list, align 8
   %tmp = alloca %struct.__va_list, align 8
   %0 = getelementptr inbounds [10 x i8], [10 x i8]* %buf, i64 0, i64 0
   call void @llvm.lifetime.start(i64 10, i8* %0)
   %1 = bitcast %struct.__va_list* %ap to i8*

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-constmask-innerouter.ll

 
 ; Multi-use tests.
 
 declare void @use32(i32) nounwind
 
 define i32 @in_multiuse_A_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_A_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w8, w0, w1
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    and w20, w8, #0xffff00
 ; CHECK-NEXT:    mov w0, w20
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 16776960
   call void @use32(i32 %n1)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 
 define i32 @in_multiuse_B_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_B_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w0, w0, w1
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    and w20, w0, #0xffff00
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 16776960
   call void @use32(i32 %n0)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-constmask-interleavedbits.ll

 
 ; Multi-use tests.
 
 declare void @use32(i32) nounwind
 
 define i32 @in_multiuse_A_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_A_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w8, w0, w1
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    and w20, w8, #0x55555555
 ; CHECK-NEXT:    mov w0, w20
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 1431655765
   call void @use32(i32 %n1)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 
 define i32 @in_multiuse_B_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_B_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w0, w0, w1
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    and w20, w0, #0x55555555
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 1431655765
   call void @use32(i32 %n0)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-constmask-interleavedbytehalves.ll

 
 ; Multi-use tests.
 
 declare void @use32(i32) nounwind
 
 define i32 @in_multiuse_A_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_A_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w8, w0, w1
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    and w20, w8, #0xf0f0f0f
 ; CHECK-NEXT:    mov w0, w20
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 252645135
   call void @use32(i32 %n1)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 
 define i32 @in_multiuse_B_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_B_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w0, w0, w1
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    and w20, w0, #0xf0f0f0f
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 252645135
   call void @use32(i32 %n0)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-constmask-lowhigh.ll

 
 ; Multi-use tests.
 
 declare void @use32(i32) nounwind
 
 define i32 @in_multiuse_A_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_A_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w8, w0, w1
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    and w20, w8, #0xffff
 ; CHECK-NEXT:    mov w0, w20
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 65535
   call void @use32(i32 %n1)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 
 define i32 @in_multiuse_B_constmask(i32 %x, i32 %y, i32 %z) nounwind {
 ; CHECK-LABEL: in_multiuse_B_constmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w0, w0, w1
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    and w20, w0, #0xffff
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, 65535
   call void @use32(i32 %n0)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-variablemask.ll

 ; ============================================================================ ;
 ; Negative tests. Should not be folded.
 ; ============================================================================ ;
 ; Multi-use tests.
 declare void @use32(i32) nounwind
 define i32 @in_multiuse_A(i32 %x, i32 %y, i32 %z, i32 %mask) nounwind {
 ; CHECK-LABEL: in_multiuse_A:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w8, w0, w1
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    and w20, w8, w3
 ; CHECK-NEXT:    mov w0, w20
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, %mask
   call void @use32(i32 %n1)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 define i32 @in_multiuse_B(i32 %x, i32 %y, i32 %z, i32 %mask) nounwind {
 ; CHECK-LABEL: in_multiuse_B:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    str x20, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    str x30, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    eor w0, w0, w1
-; CHECK-NEXT:    stp x19, x30, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    mov w19, w1
 ; CHECK-NEXT:    and w20, w0, w3
 ; CHECK-NEXT:    bl use32
 ; CHECK-NEXT:    eor w0, w20, w19
-; CHECK-NEXT:    ldp x19, x30, [sp, #16] // 16-byte Folded Reload
-; CHECK-NEXT:    ldr x20, [sp], #32 // 8-byte Folded Reload
+; CHECK-NEXT:    ldp x20, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    ldr x30, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, %y
   %n1 = and i32 %n0, %mask
   call void @use32(i32 %n0)
   %r = xor i32 %n1, %y
   ret i32 %r
 }
 ; Various bad variants

llvm/trunk/test/CodeGen/AArch64/unreachable-emergency-spill-slot.mir

 #RUN: llc -mtriple=aarch64-- -run-pass prologepilog %s -o - | FileCheck %s
 # Check that we use the frame pointer to address the emergency spill slot.
 # Using the base pointer will result in an assert with "Emergency spill slot is
 # out of reach".
 ---
 name: hugeStack
 # CHECK-LABEL: name: hugeStack
 tracksRegLiveness: true
 frameInfo:
   localFrameSize:  256
 stack:
   - { id: 0, type: variable-sized,  alignment: 1 }
   - { id: 1, name: '', size: 32761, alignment: 8 }
 body: |
   bb.0:
     STRXui undef $x8, %stack.0, 0
-    ; CHECK: STURXi undef $x8, $fp, -24
+    ; CHECK: STRXui undef $x8, $fp, 0
     B %bb.1
   bb.1:
-    liveins: $x0, $x1, $x2, $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12, $x13, $x14, $x15, $x16, $x17, $x18, $x19, $x20, $x21, $x22, $x23, $x24, $x25, $x26, $x27, $x28, $fp, $lr
-    RET_ReallyLR implicit $x21, implicit $x22, implicit $x23, implicit $x24, implicit $x25, implicit $x26, implicit $x27, implicit $x28
+    liveins: $x0, $x1, $x2, $x3, $x4, $x5, $x6, $x7, $x8, $x9, $x10, $x11, $x12, $x13, $x14, $x15, $x16, $x17, $x18, $x19, $x20, $x21, $x22, $x23, $x24, $x25, $x26, $x27, $x28
+    RET_ReallyLR implicit $x19, implicit $x20, implicit $x21, implicit $x22, implicit $x23, implicit $x24, implicit $x25, implicit $x26, implicit $x27, implicit $x28
 ...
 ---
 name: fpDoesNotFit
 # CHECK-LABEL: name: fpDoesNotFit
 tracksRegLiveness: true
 frameInfo:
   hasStackMap: true
 # set to true to force hasFP to true.

llvm/trunk/test/CodeGen/AArch64/win64_vararg.ll

 declare void @llvm.va_end(i8*)
 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture) #1
 declare void @llvm.lifetime.end.p0i8(i64, i8* nocapture) #1
 
 declare i32 @__stdio_common_vsprintf(i64, i8*, i64, i8*, i8*, i8*) local_unnamed_addr #3
 declare i64* @__local_stdio_printf_options() local_unnamed_addr #4
 
 ; CHECK-LABEL: fp
-; CHECK: str     x21, [sp, #-96]!
-; CHECK: stp     x19, x20, [sp, #16]
-; CHECK: stp     x29, x30, [sp, #32]
-; CHECK: add     x29, sp, #32
-; CHECK: add     x8, x29, #24
+; CHECK: stp     x29, x30, [sp, #-96]
+; CHECK: str     x21,      [sp, #16]
+; CHECK: stp     x19, x20, [sp, #32]
+; CHECK: mov     x29, sp
+; CHECK: add     x8, x29, #56
 ; CHECK: mov     x19, x2
 ; CHECK: mov     x20, x1
 ; CHECK: mov     x21, x0
-; CHECK: stp     x3, x4, [x29, #24]
-; CHECK: stp     x5, x6, [x29, #40]
-; CHECK: str     x7, [x29, #56]
-; CHECK: str     x8, [sp, #8]
+; CHECK: stp     x3, x4, [x29, #56]
+; CHECK: stp     x5, x6, [x29, #72]
+; CHECK: str     x7, [x29, #88]
+; CHECK: str     x8, [x29, #24]
 ; CHECK: bl      __local_stdio_printf_options
 ; CHECK: ldr     x8, [x0]
-; CHECK: add     x5, x29, #24
+; CHECK: add     x5, x29, #56
 ; CHECK: mov     x1, x21
 ; CHECK: mov     x2, x20
 ; CHECK: orr     x0, x8, #0x2
 ; CHECK: mov     x3, x19
 ; CHECK: mov     x4, xzr
 ; CHECK: bl      __stdio_common_vsprintf
 ; CHECK: cmp     w0, #0
 ; CHECK: csinv   w0, w0, wzr, ge
-; CHECK: ldp     x29, x30, [sp, #32]
-; CHECK: ldp     x19, x20, [sp, #16]
-; CHECK: ldr     x21, [sp], #96
+; CHECK: ldp     x19, x20, [sp, #32]
+; CHECK: ldr     x21,      [sp, #16]
+; CHECK: ldp     x29, x30, [sp], #96
 ; CHECK: ret
 define i32 @fp(i8*, i64, i8*, ...) local_unnamed_addr #6 {
   %4 = alloca i8*, align 8
   %5 = bitcast i8** %4 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %5) #2
   call void @llvm.va_start(i8* nonnull %5)
   %6 = load i8*, i8** %4, align 8
   %7 = call i64* @__local_stdio_printf_options() #2
   %8 = load i64, i64* %7, align 8
   %9 = or i64 %8, 2
   %10 = call i32 @__stdio_common_vsprintf(i64 %9, i8* %0, i64 %1, i8* %2, i8* null, i8* %6) #2
   %11 = icmp sgt i32 %10, -1
   %12 = select i1 %11, i32 %10, i32 -1
   call void @llvm.va_end(i8* nonnull %5)
   call void @llvm.lifetime.end.p0i8(i64 8, i8* nonnull %5) #2
   ret i32 %12
 }
 
 attributes #6 = { "no-frame-pointer-elim"="true" }
 
 ; CHECK-LABEL: vla
-; CHECK: str     x23, [sp, #-112]!
-; CHECK: stp     x21, x22, [sp, #16]
-; CHECK: stp     x19, x20, [sp, #32]
-; CHECK: stp     x29, x30, [sp, #48]
-; CHECK: add     x29, sp, #48
-; CHECK: add     x8, x29, #16
-; CHECK: stur    x8, [x29, #-40]
+; CHECK: stp     x29, x30, [sp, #-112]!
+; CHECK: str     x23, [sp, #16]
+; CHECK: stp     x21, x22, [sp, #32]
+; CHECK: stp     x19, x20, [sp, #48]
+; CHECK: mov     x29, sp
+; CHECK: add     x8, x29, #64
+; CHECK: str     x8, [x29, #24]
 ; CHECK: mov     w8, w0
 ; CHECK: add     x8, x8, #15
 ; CHECK: lsr     x15, x8, #4
 ; CHECK: mov     x19, x1
 ; CHECK: mov     [[REG2:x[0-9]+]], sp
-; CHECK: stp     x2, x3, [x29, #16]
-; CHECK: stp     x4, x5, [x29, #32]
-; CHECK: stp     x6, x7, [x29, #48]
+; CHECK: stp     x2, x3, [x29, #64]
+; CHECK: stp     x4, x5, [x29, #80]
+; CHECK: stp     x6, x7, [x29, #96]
 ; CHECK: bl      __chkstk
 ; CHECK: mov     x8, sp
 ; CHECK: sub     [[REG:x[0-9]+]], x8, x15, lsl #4
 ; CHECK: mov     sp, [[REG]]
-; CHECK: ldur    [[REG3:x[0-9]+]], [x29, #-40]
+; CHECK: ldr     [[REG3:x[0-9]+]], [x29, #24]
 ; CHECK: sxtw    [[REG4:x[0-9]+]], w0
 ; CHECK: bl      __local_stdio_printf_options
 ; CHECK: ldr     x8, [x0]
 ; CHECK: mov     x1, [[REG]]
 ; CHECK: mov     x2, [[REG4]]
 ; CHECK: mov     x3, x19
 ; CHECK: orr     x0, x8, #0x2
 ; CHECK: mov     x4, xzr
 ; CHECK: mov     x5, [[REG3]]
 ; CHECK: bl      __stdio_common_vsprintf
 ; CHECK: mov     sp, [[REG2]]
-; CHECK: sub     sp, x29, #48
-; CHECK: ldp     x29, x30, [sp, #48]
-; CHECK: ldp     x19, x20, [sp, #32]
-; CHECK: ldp     x21, x22, [sp, #16]
-; CHECK: ldr     x23, [sp], #112
+; CHECK: mov     sp, x29
+; CHECK: ldp     x19, x20, [sp, #48]
+; CHECK: ldp     x21, x22, [sp, #32]
+; CHECK: ldr     x23, [sp, #16]
+; CHECK: ldp     x29, x30, [sp], #112
 ; CHECK: ret
 define void @vla(i32, i8*, ...) local_unnamed_addr {
   %3 = alloca i8*, align 8
   %4 = bitcast i8** %3 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %4) #5
   call void @llvm.va_start(i8* nonnull %4)
   %5 = zext i32 %0 to i64
   %6 = call i8* @llvm.stacksave()
define void @vla(i32, i8*, ...) local_unnamed_addr {
   ret void
 }
 
 declare i8* @llvm.stacksave()
 declare void @llvm.stackrestore(i8*)
 
 ; CHECK-LABEL: snprintf
 ; CHECK-DAG: sub     sp,  sp, #96
-; CHECK-DAG: str     x21, [sp, #16]
-; CHECK-DAG: stp     x19, x20, [sp, #24]
-; CHECK-DAG: str     x30, [sp, #40]
+; CHECK-DAG: str     x30, [sp, #16]
+; CHECK-DAG: str     x21, [sp, #24]
+; CHECK-DAG: stp     x19, x20, [sp, #32]
 ; CHECK-DAG: add     x8, sp, #56
 ; CHECK-DAG: mov     x19, x2
 ; CHECK-DAG: mov     x20, x1
 ; CHECK-DAG: mov     x21, x0
 ; CHECK-DAG: stp     x3, x4, [sp, #56]
 ; CHECK-DAG: stp     x5, x6, [sp, #72]
 ; CHECK-DAG: str     x7, [sp, #88]
 ; CHECK-DAG: str     x8, [sp, #8]
 ; CHECK-DAG: bl      __local_stdio_printf_options
 ; CHECK-DAG: ldr     x8, [x0]
 ; CHECK-DAG: add     x5, sp, #56
 ; CHECK-DAG: mov     x1, x21
 ; CHECK-DAG: mov     x2, x20
 ; CHECK-DAG: orr     x0, x8, #0x2
 ; CHECK-DAG: mov     x3, x19
 ; CHECK-DAG: mov     x4, xzr
 ; CHECK-DAG: bl      __stdio_common_vsprintf
-; CHECK-DAG: ldr     x30, [sp, #40]
-; CHECK-DAG: ldp     x19, x20, [sp, #24]
-; CHECK-DAG: ldr     x21, [sp, #16]
+; CHECK-DAG: ldr     x30, [sp, #16]
+; CHECK-DAG: ldr     x21, [sp, #24]
+; CHECK-DAG: ldp     x19, x20, [sp, #32]
 ; CHECK-DAG: cmp     w0, #0
 ; CHECK-DAG: csinv   w0, w0, wzr, ge
 ; CHECK-DAG: add     sp, sp, #96
 ; CHECK-DAG: ret
 define i32 @snprintf(i8*, i64, i8*, ...) local_unnamed_addr #5 {
   %4 = alloca i8*, align 8
   %5 = bitcast i8** %4 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %5) #2

llvm/trunk/test/CodeGen/AArch64/wineh-frame5.mir

 # RUN: llc -o - %s -mtriple=aarch64-windows -start-before=prologepilog \
 # RUN:   -stop-after=prologepilog | FileCheck %s
 # Check multiple epilogues, save_reg, save_reg_x.
 
 # CHECK-LABEL:   bb.0.entry:
-# CHECK:         early-clobber $sp = frame-setup STRXpre killed $x28, $sp, -32
-# CHECK-NEXT:    frame-setup SEH_SaveReg_X 28, -32
-# CHECK-NEXT:    frame-setup STRXui killed $x19, $sp, 1
-# CHECK-NEXT:    frame-setup SEH_SaveReg 19, 8
-# CHECK-NEXT:    frame-setup STRXui killed $lr, $sp, 2
-# CHECK-NEXT:    frame-setup SEH_SaveReg 30, 16
+# CHECK:         early-clobber $sp = frame-setup STPXpre killed $fp, killed $lr, $sp, -4
+# CHECK-NEXT:    frame-setup SEH_SaveFPLR_X -32
+# CHECK-NEXT:    frame-setup STRXui killed $x19, $sp, 2
+# CHECK-NEXT:    frame-setup SEH_SaveReg 19, 16
 # CHECK-NEXT:    $sp = frame-setup SUBXri $sp, 496, 0
 # CHECK-NEXT:    frame-setup SEH_StackAlloc 496
 # CHECK-NEXT:    frame-setup SEH_PrologEnd
 
 # CHECK-LABEL:   bb.1.if.then:
 # CHECK:         frame-destroy SEH_EpilogStart
 # CHECK-NEXT:    $sp = frame-destroy ADDXri $sp, 496, 0
 # CHECK-NEXT:    frame-destroy SEH_StackAlloc 496
-# CHECK-NEXT:    $lr = frame-destroy LDRXui $sp, 2
-# CHECK-NEXT:    frame-destroy SEH_SaveReg 30, 16
-# CHECK-NEXT:    $x19 = frame-destroy LDRXui $sp, 1
-# CHECK-NEXT:    frame-destroy SEH_SaveReg 19, 8
-# CHECK-NEXT:    early-clobber $sp, $x28 = frame-destroy LDRXpost $sp, 32
-# CHECK-NEXT:    frame-destroy SEH_SaveReg_X 28, -32
+# CHECK-NEXT:    $x19 = frame-destroy LDRXui $sp, 2
+# CHECK-NEXT:    frame-destroy SEH_SaveReg 19, 16
+# CHECK-NEXT:    early-clobber $sp, $fp, $lr = frame-destroy LDPXpost $sp, 4
+# CHECK-NEXT:    frame-destroy SEH_SaveFPLR_X -32
 # CHECK-NEXT:    frame-destroy SEH_EpilogEnd
 # CHECK-NEXT:    TCRETURNdi @"?func2@@YAHXZ", 0, csr_aarch64_aapcs, implicit $sp
 
 
 --- |
   target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64-unknown-windows-msvc19.11.0"
 

llvm/trunk/test/CodeGen/AArch64/wineh-frame7.mir

 # RUN: llc -o - %s -mtriple=aarch64-windows -start-before=prologepilog \
 # RUN:   -stop-after=prologepilog | FileCheck %s
 # Test that stack probe results in Nop unwind codes in the prologue.  Test
 # save_fplr, save_reg_x and stack_alloc with multiple updates.
 
-# CHECK:      early-clobber $sp = frame-setup STRXpre killed $x28, $sp, -32
-# CHECK-NEXT: frame-setup SEH_SaveReg_X 28, -32
-# CHECK-NEXT: frame-setup STPXi killed $fp, killed $lr, $sp, 2
-# CHECK-NEXT: frame-setup SEH_SaveFPLR 16
+# CHECK:      early-clobber $sp = frame-setup STPXpre killed $fp, killed $lr, $sp, -2
+# CHECK-NEXT: frame-setup SEH_SaveFPLR_X -16
 # CHECK-NEXT: $x15 = frame-setup MOVZXi 56009, 0
 # CHECK-NEXT: frame-setup SEH_Nop
 # CHECK-NEXT: $x15 = frame-setup MOVKXi $x15, 2, 16
 # CHECK-NEXT: frame-setup SEH_Nop
 # CHECK-NEXT: frame-setup BL &__chkstk, implicit-def $lr, implicit $sp, implicit $x15
 # CHECK-NEXT: frame-setup SEH_Nop
 # CHECK-NEXT: $sp = frame-setup SUBXrx64 killed $sp, killed $x15, 28
 # CHECK-NEXT: frame-setup SEH_StackAlloc 2993296
 # CHECK-NEXT: frame-setup SEH_PrologEnd
 # CHECK:      frame-destroy SEH_EpilogStart
 # CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 730, 12
 # CHECK-NEXT: frame-destroy SEH_StackAlloc 2990080
 # CHECK-NEXT: $sp = frame-destroy ADDXri $sp, 3216, 0
 # CHECK-NEXT: frame-destroy SEH_StackAlloc 3216
-# CHECK-NEXT: $fp, $lr = frame-destroy LDPXi $sp, 2
-# CHECK-NEXT: frame-destroy SEH_SaveFPLR 16
-# CHECK-NEXT: early-clobber $sp, $x28 = frame-destroy LDRXpost $sp, 32
-# CHECK-NEXT: frame-destroy SEH_SaveReg_X 28, -32
+# CHECK-NEXT: early-clobber $sp, $fp, $lr = frame-destroy LDPXpost $sp, 2
+# CHECK-NEXT: frame-destroy SEH_SaveFPLR_X -16
 # CHECK-NEXT: frame-destroy SEH_EpilogEnd
 # CHECK-NEXT: RET_ReallyLR implicit killed $w0
 --- |
   target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64-unknown-windows-msvc19.11.0"
 
   ; Function Attrs: noinline optnone
   define dso_local i32 @"?func@@YAHH@Z"(i32 %i) #0 {

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch-realign.ll

 ; RUN: llc -o - %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s
 
 ; Make sure we have a base pointer.
 ; CHECK-LABEL: "?a@@YAXXZ":
 ; CHECK: and     sp, x9, #0xffffffffffffffc0
 ; CHECK: mov     x19, sp
 
 ; Make sure the funclet prologue/epilogue are correct: specifically,
 ; it shouldn't access the parent's frame via sp, and the prologue and
 ; epilogue should be symmetrical.
 ; CHECK-LABEL: "?catch$2@?0??a@@YAXXZ@4HA":
-; CHECK:      str     x28, [sp, #-32]!
-; CHECK-NEXT: str     x19, [sp, #8]
-; CHECK-NEXT: stp     x29, x30, [sp, #16]
+; CHECK:      stp     x29, x30, [sp, #-32]!
+; CHECK-NEXT: str     x28, [sp, #16]
+; CHECK-NEXT: str     x19, [sp, #24]
 ; CHECK-NEXT: add     x0, x19, #64
 ; CHECK-NEXT: mov     w1, wzr
 ; CHECK-NEXT: bl      "?bb@@YAXPEAHH@Z"
 ; CHECK-NEXT: adrp    x0, .LBB0_1
 ; CHECK-NEXT: add     x0, x0, .LBB0_1
-; CHECK-NEXT: ldp     x29, x30, [sp, #16]
-; CHECK-NEXT: ldr     x19, [sp, #8]
-; CHECK-NEXT: ldr     x28, [sp], #32
+; CHECK-NEXT: ldr     x19, [sp, #24]
+; CHECK-NEXT: ldr     x28, [sp, #16]
+; CHECK-NEXT: ldp     x29, x30, [sp], #32
 ; CHECK-NEXT: ret
 
 
 target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-windows-msvc19.11.0"
 
 define dso_local void @"?a@@YAXXZ"() personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
 entry:

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch.ll

 ; 3) That the locals are accessed using the frame pointer in both the funclet
 ;    and the parent function.
 
 ; The following checks that the unwind help object has -2 stored into it at
 ; fp - 400 - 256 = fp - 656, which is on-entry sp - 48 + 32 - 656 =
 ; on-entry sp - 672.  We check this offset in the table later on.
 
 ; CHECK-LABEL: "?func@@YAHXZ":
-; CHECK:       str     x28, [sp, #-48]!
-; CHECK:       str     x21, [sp, #8]
-; CHECK:       stp     x19, x20, [sp, #16]
-; CHECK:       stp     x29, x30, [sp, #32]
-; CHECK:       add     x29, sp, #32
+; CHECK:       stp     x29, x30, [sp, #-48]!
+; CHECK:       str     x28, [sp, #16]
+; CHECK:       str     x21, [sp, #24]
+; CHECK:       stp     x19, x20, [sp, #32]
+; CHECK:       mov     x29, sp
 ; CHECK:       sub     sp, sp, #624
 ; CHECK:       mov     x19, sp
 ; CHECK:       mov     x0, #-2
 ; CHECK:       stur    x0, [x19]
 
 ; Now check that x is stored at fp - 20.  We check that this is the same
 ; location accessed from the funclet to retrieve x.
 ; CHECK:       mov     w8, #1

 ; CHECK:       bl      "?func2@@YAHXZ
 
 ; CHECK:        [[CATCHRETDEST:.LBB0_[0-9]+]]:      ; %catchret.dest
 
 ; Check the catch funclet.
 ; CHECK-LABEL: "?catch$2@?0??func@@YAHXZ@4HA":
 
 ; Check that the stack space is allocated only for the callee saved registers.
-; CHECK:       str     x28, [sp, #-48]!
-; CHECK:       str     x21, [sp, #8]
-; CHECK:       stp     x19, x20, [sp, #16]
-; CHECK:       stp     x29, x30, [sp, #32]
+; CHECK:       stp     x29, x30, [sp, #-48]!
+; CHECK:       str     x28, [sp, #16]
+; CHECK:       str     x21, [sp, #24]
+; CHECK:       stp     x19, x20, [sp, #32]
 ; CHECK:       add     x20, x19, #12
 
 ; Check that there are no further stack updates.
 ; CHECK-NOT:   sub     sp, sp
 
 ; Check that the stack address passed to init2 is off the frame pointer, and
 ; that it matches the address of B in the parent function.
 ; CHECK:       sub     x0, x29, [[B_OFFSET]]

 ; CHECK-NEXT:         .word   4                       ; IPMapEntries
 ; CHECK-NEXT:         .word   ("$ip2state$?func@@YAHXZ")@IMGREL ; IPToStateXData
 ; CHECK-NEXT:         .word   -672                    ; UnwindHelp
 
 ; UNWIND: Function: ?func@@YAHXZ (0x0)
 ; UNWIND: Prologue [
 ; UNWIND-NEXT: ; nop
 ; UNWIND-NEXT: ; sub sp, #624
-; UNWIND-NEXT: ; add fp, sp, #32
-; UNWIND-NEXT: ; stp x29, x30, [sp, #32]
-; UNWIND-NEXT: ; stp x19, x20, [sp, #16]
-; UNWIND-NEXT: ; str x21, [sp, #8]
-; UNWIND-NEXT: ; str x28, [sp, #48]!
+; UNWIND-NEXT: ; mov fp, sp
+; UNWIND-NEXT: ; stp x19, x20, [sp, #32]
+; UNWIND-NEXT: ; str x21, [sp, #24]
+; UNWIND-NEXT: ; str x28, [sp, #16]
+; UNWIND-NEXT: ; stp x29, x30, [sp, #-48]!
 ; UNWIND-NEXT: ; end
 ; UNWIND: Function: ?catch$2@?0??func@@YAHXZ@4HA
 ; UNWIND: Prologue [
-; UNWIND-NEXT: ; stp x29, x30, [sp, #32]
-; UNWIND-NEXT: ; stp x19, x20, [sp, #16]
-; UNWIND-NEXT: ; str x21, [sp, #8]
-; UNWIND-NEXT: ; str x28, [sp, #48]!
+; UNWIND-NEXT: ; stp x19, x20, [sp, #32]
+; UNWIND-NEXT: ; str x21, [sp, #24]
+; UNWIND-NEXT: ; str x28, [sp, #16]
+; UNWIND-NEXT: ; stp x29, x30, [sp, #-48]!
 ; UNWIND-NEXT: ; end
 
 target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-unknown-windows-msvc19.11.0"
 
 %rtti.TypeDescriptor2 = type { i8**, i8*, [3 x i8] }
 %eh.CatchableType = type { i32, i32, i32, i32, i32, i32, i32 }
 %eh.CatchableTypeArray.1 = type { i32, [1 x i32] }

llvm/trunk/test/CodeGen/AArch64/wineh_shrinkwrap.mir

 # RUN: llc -O2 -o - %s -mtriple=aarch64-windows -start-before=shrink-wrap \
 # RUN:   -stop-after=prologepilog | FileCheck %s --check-prefix=WIN64
 # RUN: llc -O2 -o - %s -mtriple=aarch64-linux -start-before=shrink-wrap \
 # RUN:   -stop-after=prologepilog | FileCheck %s --check-prefix=LINUX
 
 # This tests checks that shrink wrapping bails out on Windows AMR64 due to the
 # use of Windows CFI.  We don't currently support fragments for WIndows EH on
 # ARM64.
 # The same test gets shrink wrapped on Linux ARM64.
 
 # WIN64-LABEL: bb.0.entry:
-# WIN64: early-clobber $sp = frame-setup STRXpre killed $x28, $sp, -32
+# WIN64: early-clobber $sp = frame-setup STPXpre killed $fp, killed $lr, $sp, -4
 # WIN64-LABEL: bb.1:
 # WIN64-LABEL: bb.2.if.then:
 
 # LINUX-LABEL: bb.0.entry:
 # LINUX-LABEL: bb.1:
 # LINUX-LABEL: bb.2.if.then:
-# LINUX: early-clobber $sp = frame-setup STRXpre killed $x28, $sp, -32
+# LINUX: early-clobber $sp = frame-setup STRXpre killed $fp, $sp, -32
 --- |
   ; ModuleID = 'shrink.cpp'
   target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64-unknown-windows-msvc19.11.0"
 
   define dso_local i32 @"?func@@YAHHH@Z"(i32 %a, i32 %b) local_unnamed_addr #0 {
   entry:
     %A = alloca [1000 x i32], align 4