[CodeGen] Async unwind - add a pass to fix CFI information

[chill]

Summary

This pass inserts the necessary CFI instructions to compensate for the
inconsistency of the call-frame information caused by linear (non-CGA
aware) nature of the unwind tables.

Unlike the CFIInstrInserer pass, this one almost always emits only
.cfi_remember_state/.cfi_restore_state, which results in smaller
unwind tables and also transparently handles custom unwind info
extensions like CFA offset adjustement and save locations of SVE
registers.

This pass takes advantage of the constraints taht LLVM imposes on the
placement of save/restore points (cf. ShrinkWrap.cpp):

• there is a single basic block, containing the function prologue
• possibly multiple epilogue blocks, where each epilogue block is complete and self-contained, i.e. CSR restore instructions (and the corresponding CFI instructions are not split across two or more blocks.
• prologue and epilogue blocks are outside of any loops

Thus, during execution, at the beginning and at the end of each basic
block the function can be in one of two states:

• "has a call frame", if the function has executed the prologue, or has not executed any epilogue
• "does not have a call frame", if the function has not executed the prologue, or has executed an epilogue

These properties can be computed for each basic block by a single RPO
traversal.

From the point of view of the unwind tables, the "has/does not have
call frame" state at beginning of each block is determined by the
state at the end of the previous block, in layout order.

Where these states differ, we insert compensating CFI instructions,
which come in two flavours:

• CFI instructions, which reset the unwind table state to the initial one. This is done by a target specific hook and is expected to be trivial to implement, for example it could be:

.cfi_def_cfa <sp>, 0
.cfi_same_value <rN>
.cfi_same_value <rN-1>
...

where <rN> are the callee-saved registers.

• CFI instructions, which reset the unwind table state to the one created by the function prologue. These are the sequence:

.cfi_restore_state
.cfi_remember_state

In this case we also insert a .cfi_remember_state after the
last CFI instruction in the function prologue.

[jrtc27]

This seems like it should be a generic pass conditionally added (when supported, until all backends support it) by TargetPassConfig with TRI/TLI/etc hooks, not something each target needs to subclass and manually add to the pass order

[jrtc27]

llvm/lib/CodeGen/CFIFixup.cpp

146

Having both even in the case when you don't need it is a bit weird. You should be able to just keep track of the last MBB seen that had a frame and insert the remember at the end of it when you discover you need a restore in the current MBB. That also removes the need to special-case the prologue.

(This code is also confusing due to using MBB.begin() twice rather than an interator that gets advanced, as the insertion order ends up being backwards so the code looks the wrong way round at first glance)

[chill]

llvm/lib/CodeGen/CFIFixup.cpp

146

Indeed, thanks for the suggestion!

[jrtc27]

llvm/lib/CodeGen/CFIFixup.cpp

190

Ok but doing that here doesn't make sense, now you restore back to the prologue but other things may have happened? I was imagining updating the last MBB with a frame whenever Info.HasFrameOnExit. Then you also don't need InsertPt, just insert at the end of the that MBB. That'll keep the remember/restore tight around just the MBBs that clobber the CFI state.

[jrtc27]

llvm/lib/CodeGen/CFIFixup.cpp

190

Although that doesn't work when you have multiple epilogues next to each other. You can instead keep InsertPt lying around, and updated it to CurrBB->end() whenever HasFrameOnExit is true, and in this case here set it to what you currently do just in case you don't see a non-epilogue frame before the next restore is needed.

[chill]

In D114545#3157219, @jrtc27 wrote:

This seems like it should be a generic pass conditionally added (when supported, until all backends support it) by TargetPassConfig with TRI/TLI/etc hooks, not something each target needs to subclass and manually add to the pass order

If that target specific functionality was needed by multiple separate passes, I would consider using hooks. While technically possible in this case, I fail to see
any advantage in doing so.

[jrtc27]

In D114545#3179495, @chill wrote:

In D114545#3157219, @jrtc27 wrote:

This seems like it should be a generic pass conditionally added (when supported, until all backends support it) by TargetPassConfig with TRI/TLI/etc hooks, not something each target needs to subclass and manually add to the pass order

If that target specific functionality was needed by multiple separate passes, I would consider using hooks. While technically possible in this case, I fail to see
any advantage in doing so.

I'm not aware of any passes that apply to all targets and follow the model of requiring you to subclass them rather than using TLI/TRI/TFI hooks? This pass is very similar in its model to PrologEpilogInserter, which uses TFI hooks, not subclassing. I would expect this to follow the same approach. It's certainly much easier to use from a backend author's perspective, less boilerplate needed, you just implement the hooks in your TFI and, at least for now, opt in via a bool or bool-returning method in your TFI, and that's it. No new files, no messing with TM to add your pass, no target-specific name for the pass that gets annoying when running tests/using -debug-only/etc, ...

[chill]

llvm/lib/CodeGen/CFIFixup.cpp

190

I'm not sure I understand.

Conceptually, we do two operations: reset to initial state and reset to after-prologue state.
The pair .cfi_restore_state/.cfi_remember_state is a workaround for the lack of suitable CFI directive that sets the current state without popping the stack, in other words, these make sense to go together.

As an optimisation from this base case, as you suggested, we can omit the last .cfi_remember_state, saving a bit of space in the unwind tables.

Keeping directives adjacent to one another reduces the size of the unwind tables, as you don't need intervening DW_CFA_advance_loc instructions.

Do you suggest the instead of inserting .cfi_remember_state just after the last .cfi_restore_state we could insert it at the end of the last block, known to have a stack frame?
That would separate the directive from the previous .cfi_restore_state.

Also, I think I had an earlier variant that inserted at the ->end() of blocks, which caused failures in the verifier. Didn't investigate it further, but that would mean the directive has to go before the terminator, which would separate it from a following .cfi_restore_state, too.

Apologies if I misunderstood you.

[chill]

In D114545#3157219, @jrtc27 wrote:

This seems like it should be a generic pass conditionally added (when supported, until all backends support it) by TargetPassConfig with TRI/TLI/etc hooks, not something each target needs to subclass and manually add to the pass order

I reworked it like this^, I think, in the last update.

[xgupta]

Sorry to ping reviewers (i am not confident to review it myself) to review, as this patch's .cfi_remeber_state & .cfi_remember_state support is a prerequisite to fix/complete the RISCV CFI directive support. Just a question: the patch is adding .cfi_same_value support but no test case shows that functionality?

[chill]

In D114545#3201865, @xgupta wrote:

Sorry to ping reviewers (i am not confident to review it myself) to review, as this patch's .cfi_remeber_state & .cfi_remember_state support is a prerequisite to fix/complete the RISCV CFI directive support. Just a question: the patch is adding .cfi_same_value support but no test case shows that functionality?

cfi-fixup.mir has some .cfi_same_values

[danielkiss]

Could you please rebase the series? it generated a lot of conflict for me.
the generated directives in the tests are looking good.

llvm/lib/CodeGen/CFIFixup.cpp

104

NIT, I'm not sure the bitfield here brings anything in terms of performance vs used memory.

122

NIT HasPrologue maybe better to call AfterPrologue since recalculated during the traversal but the function could has a prolog while HasPrologue false.
HasEpilogue -> AfterEpilog?

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

624–625 ↗

(On Diff #399016)

later the for loop can handle the empt list.

[chill]

llvm/lib/CodeGen/CFIFixup.cpp

104

It brings 200% reduction in the size of an auxiliary data structure (from 3 to 1 byte) and
we have one entry of these per basic block. At the same time it's extremely cheap
as coding effort (6 key presses :D) and does not impact readability in any way: if you
delete the bitfields, nothing of the rest of the code needs to change.

Is it a measurable difference? Maybe yes, maybe no, but if no, I will claim the rest of the compiler
is just written in a sloppy and lazy manner :D

122

It's ok, these two variables pertain only to the block examined in the current iteration of the loop.

[danielkiss]

LGMT

llvm/include/llvm/Target/TargetOptions.h

351 ↗

(On Diff #409194)

llvm/lib/CodeGen/CFIFixup.cpp

104

I'm happy with the bitfield just was interested in :D

[chill]

Brought forward, emitting epilogue would no be correct without this patch. A little bit of tests moved to the epilogues patch, no other changes.

[chill]

Updated on top of latest main, will commit it the following days, based on previous
acceptance, as it contains no non-trivial changes.

[MaskRay]

llvm/lib/CodeGen/CMakeLists.txt

15

Move before CFIInstrInserter.cpp to keep alphabetical order.

llvm/lib/CodeGen/TargetPassConfig.cpp

144 ↗

(On Diff #417923)

cl::init(false), can be deleted

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

29

"strong no call frame on entry"

The "on entry" seems important because a block with a prologue may have the StrongNoFrameOnEntry flag set.

88

llvm::reverse(MBB)

Though using reverse iteration may not improve much.

114

115

115

137

139

&& !HasEpilogue seems unneeded.

146

... Every block inherits the frame state (as recorded in the unwind tables) of the previous block. If the intended frame state is different, insert compensating CFI instructions.

160

delete braces

for (MachineInstr &MI : *PrologueBlock) is more common.

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

182

Super nit: if (!HasFrame && Info.HasFrameOnEntry && !Info.StrongNoFrameOnEntry) {

HasFrame represents the previous state and Info.HasFrameOnEntry represents the current state. The order implies a state transition.

---

The term "Reset" seems to imply transiting to the initial state.

"Transit to" may be better. To describe !HasFrame as well, the comment is better changed to something like Transit from ... state to ... state.

200

Similarly, HasFrame && !Info.HasFrameOnEntry

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

29

Does the Info.StrongNoFrameOnEntry semantic drift from the description here? In addition, removing !Info.StrongNoFrameOnEntry && from the .cfi_remember_state code path's condition below doesn't break any test, which indicates missing coverage.

[MaskRay]

there is a single basic block, containing the function prologue

There is the current limitation of LLVM's shrink wrapping.
I think GCC can place a prologue in multiple basic blocks.
It seems that the algorithm can process multiple prologues with some conditions.

If we consider (a) a prologue, (b) an epilogue, or (c) a region delimitered by a pair of prologue/epilogue, as a vertex in a graph,
the algorithm can handle any linear basic block layout which is a pre-order traversal.
(This condition is sufficient but not necessary: if we require that all prologues are the same, more traversal orders can be handled)

Conceptually, the basic blocks can be partitioned into multiple segments where each segment describes a prologue and its associated epilogues/inner regions.
For each segment, the prologue must precede all its associated epilogues.

"prologue and epilogue blocks are outside of any loops"

I think this is naturally satisfied and does not need to be emphasized here.
I think the algorithm can handle prologue and epilogue in a loop (a prologue vertex cannot be recursively entered)

"has a call frame", if the function has executed the prologue, or has not executed any epilogue

or => "... and has not executed any epilogue"

From the point of view of the unwind tables, the "has/does not have call frame" state at beginning of each block is determined by the state at the end of the previous block, in layout order.

"layout order" seems to suggest the linear placement order.
Here we seem to suggest the order related to reachability, but I do not know the appropriate term.

---

In a function using SP as the CFA, if a call needs to temporarily adjust the stack size (e.g. x86 pushl/pushq; in x86-32 pushl is particular common because of the calling convention), llvm may produce something like .cfi_adjust_cfa_offset {4,8}. Can AArch64 codegen produce something like this? Whose responsibility is to emit a negative .cfi_adjust_cfa_offset to cancel the effect? CFIFixup cannot handle such a case while CFIInstrInserter can handle it.

See test/CodeGen/X86/segmented-stacks-dynamic.ll: rbx may be used as a base register and CFIInstrInserter can insert .cfi_restore %rbx. CFIFixup doesn't support this.

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

182

Actually, I think Info.StrongNoFrameOnEntry can just be removed.

Info.HasFrameOnEntry implies that this is a block in "after prologue" state and Info.StrongNoFrameOnEntry must be false.

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

99

MF.getNumBlockIDs() may be larger than MF.size()

[chill]

In D114545#3409903, @MaskRay wrote:

"layout order" seems to suggest the linear placement order.

That's right, it's the order of increasing instruction addresses, corresponding to the basic block layout and the order of instructions in the basic block.

In a function using SP as the CFA, if a call needs to temporarily adjust the stack size (e.g. x86 pushl/pushq; in x86-32 pushl is particular common
because of the calling convention), llvm may produce something like .cfi_adjust_cfa_offset {4,8}. Can AArch64 codegen produce something like this?

I don't think AArch64 ever generates .cfi_adjust_cfa_offset.

Whose responsibility is to emit a negative .cfi_adjust_cfa_offset to cancel the effect? CFIFixup cannot handle such a case while CFIInstrInserter can handle it.

Indeed, if a backend uses SP as a frame pointer and allocates/deallocates outgoing arguments space in different basic blocks, CFIFixup won't fix it.

See test/CodeGen/X86/segmented-stacks-dynamic.ll: rbx may be used as a base register and CFIInstrInserter can insert .cfi_restore %rbx. CFIFixup doesn't support this.

As far as I can understand this testcase, the blocks containing .cfi_restore %rbx would have StrongNoFrameOnEntry set, which would make CFIFixup
fall into the "reset to initial state".

FWIW, for the epilogue blocks the test, the PEI should have already inserted .cfi_restore for the pops of the CSRs.

[MaskRay]

In D114545#3414572, @chill wrote:

In D114545#3409903, @MaskRay wrote:

"layout order" seems to suggest the linear placement order.

That's right, it's the order of increasing instruction addresses, corresponding to the basic block layout and the order of instructions in the basic block.

OK

In a function using SP as the CFA, if a call needs to temporarily adjust the stack size (e.g. x86 pushl/pushq; in x86-32 pushl is particular common
because of the calling convention), llvm may produce something like .cfi_adjust_cfa_offset {4,8}. Can AArch64 codegen produce something like this?

I don't think AArch64 ever generates .cfi_adjust_cfa_offset.

OK. Agreed according to the call sites of emitCFIAdjustCfaOffset.

Whose responsibility is to emit a negative .cfi_adjust_cfa_offset to cancel the effect? CFIFixup cannot handle such a case while CFIInstrInserter can handle it.

Indeed, if a backend uses SP as a frame pointer and allocates/deallocates outgoing arguments space in different basic blocks, CFIFixup won't fix it.

Agree.

See test/CodeGen/X86/segmented-stacks-dynamic.ll: rbx may be used as a base register and CFIInstrInserter can insert .cfi_restore %rbx. CFIFixup doesn't support this.

As far as I can understand this testcase, the blocks containing .cfi_restore %rbx would have StrongNoFrameOnEntry set, which would make CFIFixup
fall into the "reset to initial state".

FWIW, for the epilogue blocks the test, the PEI should have already inserted .cfi_restore for the pops of the CSRs.

Perhaps StrongNoFrameOnEntry has not clearly been explained:) hence my confusion.

[chill]

llvm/lib/CodeGen/CFIFixup.cpp

29

There was a bug in the conditions, that I fixed and it triggers in a few of the tests (those with .cfi_same_value appearing)

182

My mental model is that both represent the "opinion" of the unwind tables on one hand (HasFrame), and the "opinion" of the CFG, on the other hand (Info.HasFrameOnEntry), about the current state at entry of the block. CFG is right (as far as we can tell form the available CFI instructions) and we fix the unwind tables.

182

If there are missing unwind instructions in the epilogue, a block can have HasFrameOnEntrySet, even though it's reachable from function entry without passing through a prologue. In this case StroingNoFrameOnEntry will be true. Example of this is llvm/test/CodeGen/AArch64/fast-isel-branch-cond-split.ll

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

10

"This pass inserts the necessary .cfi_remember_state/.cfi_remember_state and calls resetCFIToInitialState"

resetCFIToInitialState may insert CFI instructions other than .cfi_remember_state/.cfi_remember_state

28

I think this comment should be moved before StrongNoFrameOnEntry is declared in the code below.

29

Thanks:) I think I fully understand it now.

167

Delete the TODO: I believe this is not a TODO, but an assumption. The stack based remember/restore approach just cannot handle epilogue before prologue in theory.

This is what I wanted to express but probably not in a clear way:
"If we consider (a) a prologue, (b) an epilogue, or (c) a region delimitered by a pair of prologue/epilogue, as a vertex in a graph,
the algorithm can handle any linear basic block layout which is a pre-order traversal.
(This condition is sufficient but not necessary: if we require that all prologues are the same, more traversal orders can be handled)"

llvm/test/CodeGen/AArch64/arm64-shrink-wrapping.ll

29 ↗

(On Diff #419186)

Thanks for the update to foo. It makes me understand the purpose of StrongNoFrameOnEntry. Right now there is no epilogue CFI, so BB1's HasFrameOnExit is (incorrectly) false. We rely on StrongNoFrameOnEntry being true to generate .cfi_def_cfa wsp, 0 and .cfi_same_value *.

However, using MF.getFrameInfo().getCalleeSavedInfo(); has the problem that we may add redundant .cfi_same_value. As this test shows, none of w18/w30/w29 is updated in previous blocks but we add them: (a) it unnecessarily increases unwind table size (b) it may confuse a reader.

We may need to check whether the registers have actually been modified by previous CFI instructions, before emitting .cfi_same_value.

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

196

It would be nice to have a test that a cfi_remember_state is inserted after a cfi_restore_state.
Right now if I delete ++InsertPt I don't break any test.

[chill]

llvm/lib/CodeGen/CFIFixup.cpp

28

I'd prefer to have an overall description in one place at the start of the file. It easy to relate the name StrongNoFrameOnEntry to the description.

167

I've moved it to the start of the file. If an actual need appears, I think it's implementable
and the pass need not restrict itself to remember/restore state.

196

cfi-fixup.mir tests this now.

llvm/test/CodeGen/AArch64/arm64-shrink-wrapping.ll

29 ↗

(On Diff #419186)

These (except for x18) are not redundant. At LBB0_2 the unwind info says the CSR are on the stack, but they aren't there.

x18 is because of it's potential use as a shadow call stack pointer, I've fixed the code to emit .cfi_same_value x18 only if the function indeed uses shadow stack.

[MaskRay]

Looks great! Thanks again for the hard work! And sorry again for being so slow to get to this important pass :-(

llvm/include/llvm/CodeGen/CFIFixup.h

2

"C++" is missing. Hyphens on the left are too long. See https://llvm.org/docs/CodingStandards.html#file-headers

llvm/lib/CodeGen/CFIFixup.cpp

167

If an epilogue proceeds the associated prologue, the pass needs to copy the prologue state and remember/restore cannot be used. This is a case I'd call an assumption instead of a TODO since implementing it seems has nearly no advantage, at the cost of large complexity. OK, you make the call whether the TODO should remain here:)

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

301 ↗

(On Diff #419432)

Optional: wonder whether this can be avoided by moving AArch64FrameLowering::resetCFIToInitialState after the body of needsShadowCallStackPrologueEpilogue

llvm/test/CodeGen/AArch64/arm64-shrink-wrapping.ll

29 ↗

(On Diff #419186)

Thanks. I did not notice .cfi_offset w30 yesterday. My bad. Seems that there is some mechanism to not emit .cfi_same_value for unreferenced registers. Sounds good.

[chill]

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

301 ↗

(On Diff #419432)

I'd prefer various functions, which deal with CFI emission to be closer together.

[chill]

In D114545#3421297, @MaskRay wrote:

Looks great! Thanks again for the hard work! And sorry again for being so slow to get to this important pass :-(

Thanks, much appreciated!

[omjavaid]

Hi I have temporarily reverted this change as it has caused LLVM buildbot failures with failing fuzzer-threaded.test with clang crashing over an assertion failure in llvm/lib/CodeGen/CFIFixup.cpp. Kindly look into it. Thanks
.

[chill]

In D114545#3428728, @omjavaid wrote:

Hi I have temporarily reverted this change as it has caused LLVM buildbot failures with failing fuzzer-threaded.test with clang crashing over an assertion failure in llvm/lib/CodeGen/CFIFixup.cpp. Kindly look into it. Thanks
.

Thanks. From what I can tell the reason is the test ends up with unreachable block(s) for which the invariants we maintain do not hold. I'll make modification to ignore those blocks.
The block(s) have their address taken (recorded in the "sancov_pcs" section), but no predecessors.

[chill]

Updated to exclude unreachable blocks.

[MaskRay]

llvm/lib/CodeGen/CFIFixup.cpp

180

llvm/include/llvm/CodeGen/CFIFixup.h

+//===------ CFIFixup.h - Insert CFI remember/restore instructions ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Contains definition of the base CFIFixup pass.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_CFIFIXUP_H
+#define LLVM_CODEGEN_CFIFIXUP_H
+
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/InitializePasses.h"
+
+namespace llvm {
+class CFIFixup : public MachineFunctionPass {
+public:
+  CFIFixup(char &ID) : MachineFunctionPass(ID) {}
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesAll();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+protected:
+  virtual bool skipFunction(MachineFunction &MF) const;
+  virtual void resetToInitialState(MachineBasicBlock &MBB) const = 0;
+};
+} // namespace llvm
+
+#endif // LLVM_CODEGEN_CFIFIXUP_H

llvm/lib/CodeGen/AsmPrinter/AsmPrinterDwarf.cpp

case MCCFIInstruction::OpEscape:
     OutStreamer->emitCFIEscape(Inst.getValues());
     break;
   case MCCFIInstruction::OpRestore:
     OutStreamer->emitCFIRestore(Inst.getRegister());
     break;
   case MCCFIInstruction::OpUndefined:
     OutStreamer->emitCFIUndefined(Inst.getRegister());
     break;
+  case MCCFIInstruction::OpRememberState:
+    OutStreamer->emitCFIRememberState();
+    break;
+  case MCCFIInstruction::OpRestoreState:
+    OutStreamer->emitCFIRestoreState();
+    break;
   }
 }
 
 void AsmPrinter::emitDwarfDIE(const DIE &Die) const {
   // Emit the code (index) for the abbreviation.
   if (isVerbose())
     OutStreamer->AddComment("Abbrev [" + Twine(Die.getAbbrevNumber()) + "] 0x" +
                             Twine::utohexstr(Die.getOffset()) + ":0x" +

llvm/lib/CodeGen/CFIFixup.cpp

+//===------ CFIFixup.cpp - Insert CFI remember/restore instructions -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass inserts the necessary .cfi_remember_state and
+// .cfi_restore_state CFI instructions to adjust for the inconsistency of
+// the call-frame information caused by final machine basic block layout.
+// This pass relies in constraints LLVM imposes on the placement of
+// save/restore points (cf. ShrinkWrap):
+// * there is a single basic block, containing the function prologue
+// * possibly multiple epilogue blocks, where each epilogue block is
+//   complete and self-contained, i.e. CSR restore instructions (and the
+//   corresponding CFI instructions are not split across two or more blocks.
+// * prologue and epilogue blocks are outside of any loops
+// Thus, during execution, at the beginning and at the end of each basic block
+// the function can be in one of two states:
+//  - "has a call frame", if the function has executed the prologue, or
+//    has not executed any epilogue
+//  - "does not have a call frame", if the function has not executed the
+//    prologue, or has executed an epilogue
+// which can be computed by a single RPO traversal.
+
+// In order to acommodate backends, which do not generate unwind info in
+// epilogues we compute an additional property "strong no call frame", which
+// is set for the entry point of the function and for every block reachable
+// from the entry along a path that does not execute the prologue. If this
+// property holds, it takes precedence over the "has a call frame" property.
+
+// From the point of view of the unwind tables, the "has/does not have call
+// frame" state at beginning of each block is determined by the state at the end
+// of the previous block, in layout order. Where these states differ, we insert
+// compensating CFI instructions, which come in two flavours:
+
+//   - CFI instructions, which reset the unwind table state to the initial one.
+//     This is done by a target specific hook and is expected to be trivial
+//     to implement, for example it could be:
+//       .cfi_def_cfa <sp>, 0
+//       .cfi_same_value <rN>
+//       .cfi_same_value <rN-1>
+//       ...
+//     where <rN> are the callee-saved registers.
+//   - CFI instructions, which reset the unwind table state to the one
+//     created by the function prologue. These are
+//       .cfi_restore_state
+//       .cfi_remember_state
+//     In this case we also insert a `.cfi_remember_state` after the last CFI
+//     instruction in the function prologue.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/CFIFixup.h"
+
+#include "llvm/ADT/PostOrderIterator.h"
+#include "llvm/ADT/SmallBitVector.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/MC/MCAsmInfo.h"
+#include "llvm/MC/MCDwarf.h"
+#include "llvm/Target/TargetMachine.h"
+
+using namespace llvm;
+
+bool CFIFixup::skipFunction(MachineFunction &MF) const {
+  return !MF.needsFrameMoves() ||
+         MF.getTarget().getMCAsmInfo()->usesWindowsCFI() ||
+         MF.getFunction().getUWTableKind() != UWTableKind::Async;
+}
+
+static bool isPrologueCFIInstruction(const MachineInstr &MI) {
+  return MI.getOpcode() == TargetOpcode::CFI_INSTRUCTION &&
+         MI.getFlag(MachineInstr::FrameSetup);
+}
+
+static bool containsPrologue(const MachineBasicBlock &MBB) {
+  return llvm::any_of(MBB.instrs(), isPrologueCFIInstruction);
+}
+
+static bool containsEpilogue(const MachineBasicBlock &MBB) {
+  return llvm::any_of(make_range(MBB.rbegin(), MBB.rend()), [](const auto &MI) {
+    return MI.getOpcode() == TargetOpcode::CFI_INSTRUCTION &&
+           MI.getFlag(MachineInstr::FrameDestroy);
+  });
+}
+
+bool CFIFixup::runOnMachineFunction(MachineFunction &MF) {
+  if (skipFunction(MF))
+    return false;
+  unsigned NumBlocks = MF.size();
+  if (NumBlocks < 2)
+    return false;
+
+  struct BlockFlags {
+    bool StrongNoFrameOnEntry : 1;
+    bool HasFrameOnEntry : 1;
+    bool HasFrameOnExit : 1;
+  };
+  SmallVector<BlockFlags, 32> BlockInfo(NumBlocks, {false, false, false});
+  BlockInfo[0].StrongNoFrameOnEntry = true;
+
+  // Compute the presence/absence of frame at each basic block.
+  MachineBasicBlock *PrologueBlock = nullptr;
+  ReversePostOrderTraversal<MachineBasicBlock *> RPOT(&*MF.begin());
+  for (auto *MBB : RPOT) {
+    auto &Info = BlockInfo[MBB->getNumber()];
+
+    bool HasPrologue = false;
+    bool HasEpilogue = false;
+
+    if (!PrologueBlock && !Info.HasFrameOnEntry && containsPrologue(*MBB)) {
+      PrologueBlock = MBB;
+      HasPrologue = true;
+    }
+
+    if (Info.HasFrameOnEntry || HasPrologue)
+      HasEpilogue = containsEpilogue(*MBB);
+
+    // If the function does not have a call frame at the entry of a block and
+    // the block contains the prologue, then the function has a call frame at
+    // the exit of the block, unless the block also contained the epilogue.
+    Info.HasFrameOnExit = (Info.HasFrameOnEntry || HasPrologue) && !HasEpilogue;
+
+    // Set the successors' state on entry.
+    for (auto *Succ : MBB->successors()) {
+      auto &SuccInfo = BlockInfo[Succ->getNumber()];
+      SuccInfo.StrongNoFrameOnEntry |=
+          Info.StrongNoFrameOnEntry && !HasPrologue && !HasEpilogue;
+      SuccInfo.HasFrameOnEntry = Info.HasFrameOnExit;
+    }
+  }
+
+  if (!PrologueBlock)
+    return false;
+
+  // Walk the blocks of the function in "physical" order. After each block, the
+  // current frame state (as recorded in the unwind tables) is the same as
+  // at the end of that block.
+  // At each block, where the current frame state differs from what it
+  // should be at the beginning of that block, insert compensating CFI
+  // instructions.
+  const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+  bool Change = false;
+  // `InsertPt` always points to the point in a preceding block where we have to
+  // insert a `.cfi_remember_state`, in the case that the current block needs a
+  // `.cfi_restore_state`.
+  MachineBasicBlock *InsertMBB = PrologueBlock;
+  MachineBasicBlock::iterator InsertPt = PrologueBlock->end();
+  for (MachineBasicBlock::iterator I = PrologueBlock->begin(),
+                                   E = PrologueBlock->end();
+       I != E; ++I) {
+    if (isPrologueCFIInstruction(*I))
+      InsertPt = std::next(I);
+  }
+  assert(InsertPt != PrologueBlock->end() &&
+         "Inconsistent notion of \"prologue block\"");
+
+  // No point starting before the prologue block.
+  // TODO: the unwind tables will still be incorrect if an epilogue physcally
+  // preceeds the prologue.
+  MachineFunction::iterator CurrBB = std::next(PrologueBlock->getIterator());
+  bool HasFrame = BlockInfo[PrologueBlock->getNumber()].HasFrameOnExit;
+  while (CurrBB != MF.end()) {
+    auto &Info = BlockInfo[CurrBB->getNumber()];
+#ifndef NDEBUG
+    for (auto *Pred : CurrBB->predecessors())
+      assert(Info.HasFrameOnEntry ==
+                 BlockInfo[Pred->getNumber()].HasFrameOnExit &&
+             "Inconsistent call frame state");
+#endif
+    if (!Info.StrongNoFrameOnEntry && Info.HasFrameOnEntry && !HasFrame) {
+      // Reset to the "after prologue" state.
+
+      // Insert a `.cfi_remember_state` into the last block known to have a
+      // stack frame.
+      unsigned CFIIndex =
+          MF.addFrameInst(MCCFIInstruction::createRememberState(nullptr));
+      BuildMI(*InsertMBB, InsertPt, DebugLoc(),
+              TII.get(TargetOpcode::CFI_INSTRUCTION))
+          .addCFIIndex(CFIIndex);
+      // Insert a `.cfi_restore_state` at the beginning of the current block.
+      CFIIndex = MF.addFrameInst(MCCFIInstruction::createRestoreState(nullptr));
+      InsertPt = BuildMI(*CurrBB, CurrBB->begin(), DebugLoc(),
+                         TII.get(TargetOpcode::CFI_INSTRUCTION))
+                     .addCFIIndex(CFIIndex);
+      ++InsertPt;
+      InsertMBB = &*CurrBB;
+      Change = true;
+    } else if (!Info.HasFrameOnEntry && HasFrame) {
+      // Reset to the state upon function entry.
+      resetToInitialState(*CurrBB);
+      Change = true;
+    }
+
+    HasFrame = Info.HasFrameOnExit;
+    ++CurrBB;
+  }
+
+  return Change;
+}

llvm/lib/CodeGen/CMakeLists.txt

 add_llvm_component_library(LLVMCodeGen
   AggressiveAntiDepBreaker.cpp
   AllocationOrder.cpp
   Analysis.cpp
   AtomicExpandPass.cpp
   BasicTargetTransformInfo.cpp
   BranchFolding.cpp
   BranchRelaxation.cpp
   BreakFalseDeps.cpp
   BasicBlockSections.cpp
   CalcSpillWeights.cpp
   CallingConvLower.cpp
   CFGuardLongjmp.cpp
   CFIInstrInserter.cpp
+  CFIFixup.cpp
   CodeGen.cpp
   CodeGenCommonISel.cpp
   CodeGenPassBuilder.cpp
   CodeGenPrepare.cpp
   CommandFlags.cpp
   CriticalAntiDepBreaker.cpp
   DeadMachineInstructionElim.cpp
   DetectDeadLanes.cpp

llvm/lib/CodeGen/TailDuplicator.cpp

 /// the source operands due to earlier PHI translation.
 void TailDuplicator::duplicateInstruction(
     MachineInstr *MI, MachineBasicBlock *TailBB, MachineBasicBlock *PredBB,
     DenseMap<Register, RegSubRegPair> &LocalVRMap,
     const DenseSet<Register> &UsedByPhi) {
   // Allow duplication of CFI instructions.
   if (MI->isCFIInstruction()) {
     BuildMI(*PredBB, PredBB->end(), PredBB->findDebugLoc(PredBB->begin()),
-      TII->get(TargetOpcode::CFI_INSTRUCTION)).addCFIIndex(
-      MI->getOperand(0).getCFIIndex());
+            TII->get(TargetOpcode::CFI_INSTRUCTION))
+        .addCFIIndex(MI->getOperand(0).getCFIIndex())
+        .setMIFlags(MI->getFlags());
     return;
   }
   MachineInstr &NewMI = TII->duplicate(*PredBB, PredBB->end(), *MI);
   if (PreRegAlloc) {
     for (unsigned i = 0, e = NewMI.getNumOperands(); i != e; ++i) {
       MachineOperand &MO = NewMI.getOperand(i);
       if (!MO.isReg())
         continue;

llvm/lib/Target/AArch64/AArch64.h

 FunctionPass *createAArch64PostSelectOptimize();
 FunctionPass *createAArch64StackTaggingPass(bool IsOptNone);
 FunctionPass *createAArch64StackTaggingPreRAPass();
 
 void initializeAArch64A53Fix835769Pass(PassRegistry&);
 void initializeAArch64A57FPLoadBalancingPass(PassRegistry&);
 void initializeAArch64AdvSIMDScalarPass(PassRegistry&);
 void initializeAArch64BranchTargetsPass(PassRegistry&);
+void initializeAArch64CFIFixupPass(PassRegistry&);
 void initializeAArch64CollectLOHPass(PassRegistry&);
 void initializeAArch64CondBrTuningPass(PassRegistry &);
 void initializeAArch64CompressJumpTablesPass(PassRegistry&);
 void initializeAArch64ConditionalComparesPass(PassRegistry&);
 void initializeAArch64ConditionOptimizerPass(PassRegistry&);
 void initializeAArch64DeadRegisterDefinitionsPass(PassRegistry&);
 void initializeAArch64ExpandPseudoPass(PassRegistry&);
 void initializeAArch64SLSHardeningPass(PassRegistry&);

llvm/lib/Target/AArch64/AArch64CFIFixup.h

+//===------ AArch64CFIFixup.h - Insert CFI remember/restore instructions --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// AArch64 specific implementation for the CFIFixup pass.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AARCH64_AARCH64CFIFIXUP_H
+#define LLVM_LIB_TARGET_AARCH64_AARCH64CFIFIXUP_H
+
+#include "llvm/CodeGen/CFIFixup.h"
+#include "AArch64.h"
+
+namespace llvm {
+
+class AArch64CFIFixup : public CFIFixup {
+public:
+  static char ID;
+
+  AArch64CFIFixup() : CFIFixup(ID) {
+    initializeAArch64CFIFixupPass(*PassRegistry::getPassRegistry());
+  }
+
+protected:
+  bool skipFunction(MachineFunction &MF) const override;
+  void resetToInitialState(MachineBasicBlock &MBB) const override;
+};
+
+} // namespace llvm
+
+#endif // LLVM_LIB_TARGET_AARCH64_AARCH64CFIFIXUP_H
 No newline at end of file

llvm/lib/Target/AArch64/AArch64CFIFixup.cpp

+//===- AArch64CFIFixup - AArch64 CFI remember/testore insertion pass ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Contains AArch64 specific parts of the base CFIFixup pass.
+///
+//===----------------------------------------------------------------------===//
+#include "AArch64CFIFixup.h"
+#include "AArch64MachineFunctionInfo.h"
+#include "AArch64Subtarget.h"
+#include "llvm/MC/MCDwarf.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "aarch64-cfi-fixup"
+
+char AArch64CFIFixup::ID = 0;
+
+INITIALIZE_PASS(AArch64CFIFixup, "aarch64-cfi-fixup",
+                "Insert CFI remember/restore state instructions", false, false)
+
+bool AArch64CFIFixup::skipFunction(MachineFunction &MF) const {
+  return !MF.getInfo<AArch64FunctionInfo>()->needsAsyncDwarfUnwindInfo() ||
+         CFIFixup::skipFunction(MF);
+}
+
+static void insertCFISameValue(const MCInstrDesc &Desc, MachineFunction &MF,
+                               MachineBasicBlock &MBB,
+                               MachineBasicBlock::iterator InsertPt,
+                               unsigned DwarfReg) {
+  unsigned CFIIndex =
+      MF.addFrameInst(MCCFIInstruction::createSameValue(nullptr, DwarfReg));
+  BuildMI(MBB, InsertPt, DebugLoc(), Desc).addCFIIndex(CFIIndex);
+}
+
+void AArch64CFIFixup::resetToInitialState(MachineBasicBlock &MBB) const {
+  MachineFunction &MF = *MBB.getParent();
+  const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+  const TargetInstrInfo &TII = *Subtarget.getInstrInfo();
+  const auto &TRI =
+      static_cast<const AArch64RegisterInfo &>(*Subtarget.getRegisterInfo());
+  const auto &MFI = *MF.getInfo<AArch64FunctionInfo>();
+
+  const MCInstrDesc &CFIDesc = TII.get(TargetOpcode::CFI_INSTRUCTION);
+  DebugLoc DL;
+
+  // Reset the CFA to `SP + 0`.
+  MachineBasicBlock::iterator InsertPt = MBB.begin();
+  unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::cfiDefCfa(
+      nullptr, TRI.getDwarfRegNum(AArch64::SP, true), 0));
+  BuildMI(MBB, InsertPt, DL, CFIDesc).addCFIIndex(CFIIndex);
+
+  // Flip the RA sign state.
+  if (MFI.shouldSignReturnAddress()) {
+    CFIIndex = MF.addFrameInst(MCCFIInstruction::createNegateRAState(nullptr));
+    BuildMI(MBB, InsertPt, DL, CFIDesc).addCFIIndex(CFIIndex);
+  }
+
+  // X18 could have been used for the shadow call stack, reset it.
+  if (Subtarget.isXRegisterReserved(18))
+    insertCFISameValue(CFIDesc, MF, MBB, InsertPt,
+                       TRI.getDwarfRegNum(AArch64::X18, true));
+
+  // Emit .cfi_same_value for callee-saved registers.
+  const std::vector<CalleeSavedInfo> &CSI =
+      MF.getFrameInfo().getCalleeSavedInfo();
+  if (CSI.empty())
+    return;
+  for (const auto &Info : CSI) {
+    unsigned Reg = Info.getReg();
+    if (!TRI.regNeedsCFI(Reg, Reg))
+      continue;
+    insertCFISameValue(CFIDesc, MF, MBB, InsertPt,
+                       TRI.getDwarfRegNum(Reg, true));
+  }
+}
 No newline at end of file

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

 //===-- AArch64TargetMachine.cpp - Define TargetMachine for AArch64 -------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64TargetMachine.h"
 #include "AArch64.h"
+#include "AArch64CFIFixup.h"
 #include "AArch64MachineFunctionInfo.h"
 #include "AArch64MacroFusion.h"
 #include "AArch64Subtarget.h"
 #include "AArch64TargetObjectFile.h"
 #include "AArch64TargetTransformInfo.h"
 #include "MCTargetDesc/AArch64MCTargetDesc.h"
 #include "TargetInfo/AArch64TargetInfo.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/CSEConfigBase.h"
 #include "llvm/CodeGen/GlobalISel/IRTranslator.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelect.h"
 #include "llvm/CodeGen/GlobalISel/Legalizer.h"
 #include "llvm/CodeGen/GlobalISel/LoadStoreOpt.h"
 #include "llvm/CodeGen/GlobalISel/Localizer.h"
 #include "llvm/CodeGen/GlobalISel/RegBankSelect.h"
 #include "llvm/CodeGen/MIRParser/MIParser.h"
 #include "llvm/CodeGen/MachineScheduler.h"
 #include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/CFIFixup.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Pass.h"
extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64Target() {
   RegisterTargetMachine<AArch64leTargetMachine> W(getTheARM64_32Target());
   RegisterTargetMachine<AArch64leTargetMachine> V(getTheAArch64_32Target());
   auto PR = PassRegistry::getPassRegistry();
   initializeGlobalISel(*PR);
   initializeAArch64A53Fix835769Pass(*PR);
   initializeAArch64A57FPLoadBalancingPass(*PR);
   initializeAArch64AdvSIMDScalarPass(*PR);
   initializeAArch64BranchTargetsPass(*PR);
+  initializeAArch64CFIFixupPass(*PR);
   initializeAArch64CollectLOHPass(*PR);
   initializeAArch64CompressJumpTablesPass(*PR);
   initializeAArch64ConditionalComparesPass(*PR);
   initializeAArch64ConditionOptimizerPass(*PR);
   initializeAArch64DeadRegisterDefinitionsPass(*PR);
   initializeAArch64ExpandPseudoPass(*PR);
   initializeAArch64LoadStoreOptPass(*PR);
   initializeAArch64MIPeepholeOptPass(*PR);
if (TM->getOptLevel() != CodeGenOpt::None && EnableCollectLOH &&
       TM->getTargetTriple().isOSBinFormatMachO())
     addPass(createAArch64CollectLOHPass());
 }
 
 void AArch64PassConfig::addPreEmitPass2() {
   // SVE bundles move prefixes with destructive operations. BLR_RVMARKER pseudo
   // instructions are lowered to bundles as well.
   addPass(createUnpackMachineBundles(nullptr));
+
+  if (!TM->getMCAsmInfo()->usesWindowsCFI() &&
+      !TM->getTargetTriple().isOSBinFormatMachO())
+    addPass(new AArch64CFIFixup());
 }
 
 yaml::MachineFunctionInfo *
 AArch64TargetMachine::createDefaultFuncInfoYAML() const {
   return new yaml::AArch64FunctionInfo();
 }
 
 yaml::MachineFunctionInfo *

llvm/lib/Target/AArch64/CMakeLists.txt

add_llvm_target(AArch64CodeGen
   GISel/AArch64PostLegalizerLowering.cpp
   GISel/AArch64PostSelectOptimize.cpp
   GISel/AArch64RegisterBankInfo.cpp
   AArch64A57FPLoadBalancing.cpp
   AArch64AdvSIMDScalarPass.cpp
   AArch64AsmPrinter.cpp
   AArch64BranchTargets.cpp
   AArch64CallingConvention.cpp
+  AArch64CFIFixup.cpp
   AArch64CleanupLocalDynamicTLSPass.cpp
   AArch64CollectLOH.cpp
   AArch64CondBrTuning.cpp
   AArch64ConditionalCompares.cpp
   AArch64DeadRegisterDefinitionsPass.cpp
   AArch64ExpandImm.cpp
   AArch64ExpandPseudoInsts.cpp
   AArch64FalkorHWPFFix.cpp

llvm/test/CodeGen/AArch64/O0-pipeline.ll

 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       AArch64 Branch Targets
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Unpack machine instruction bundles
+; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       AArch64 Compress Jump Tables
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Unpack machine instruction bundles
+; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 ; CHECK-NEXT: Pass Arguments:  -domtree
 ; CHECK-NEXT:   FunctionPass Manager
 ; CHECK-NEXT:     Dominator Tree Construction
 ; CHECK-NEXT: Pass Arguments:  -assumption-cache-tracker -targetlibinfo -domtree -loops -scalar-evolution -stack-safety-local

llvm/test/CodeGen/AArch64/arm64-fp128.ll

; CHECK-NEXT:    ret
   %rhs = load fp128, fp128* @rhs, align 16
 
   %val = fcmp ueq fp128 %lhs, %rhs
 
   ret i1 %val
 }
 
 
-define dso_local i32 @test_br_cc() {
+define dso_local i32 @test_br_cc() uwtable {
 ; CHECK-LABEL: test_br_cc:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    adrp x8, lhs
 ; CHECK-NEXT:    ldr q0, [x8, :lo12:lhs]
 ; CHECK-NEXT:    adrp x8, rhs
 ; CHECK-NEXT:    ldr q1, [x8, :lo12:rhs]
 ; CHECK-NEXT:    bl __lttf2
 ; CHECK-NEXT:    cmp w0, #0
 ; CHECK-NEXT:    b.ge .LBB11_2
 ; CHECK-NEXT:  // %bb.1: // %iftrue
 ; CHECK-NEXT:    mov w0, #42
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB11_2: // %iffalse
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    mov w0, #29
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 
   %lhs = load fp128, fp128* @lhs, align 16
   %rhs = load fp128, fp128* @rhs, align 16
 
   ; olt == !uge, which LLVM optimizes this to.
   %cond = fcmp olt fp128 %lhs, %rhs
   br i1 %cond, label %iftrue, label %iffalse

llvm/test/CodeGen/AArch64/call-rv-marker.ll

 entry:
   %tobool.not = icmp eq i32 %c, 0
   %.sink = select i1 %tobool.not, i32 2, i32 1
   %call1 = call i8* @foo0(i32 %.sink) [ "clang.arc.attachedcall"() ]
   tail call void @foo2(i8* %call1)
   ret void
 }
 
-define dso_local void @rv_marker_3() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+define dso_local void @rv_marker_3() uwtable personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 ; CHECK-LABEL: rv_marker_3
 ; CHECK:         .cfi_offset w30, -32
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    bl  foo1
 ; SELDAG-NEXT:   mov x29, x29
 ;
+; CHECK:      .cfi_restore w30
+; CHECK-NEXT: b objc_release
+; CHECK-NEXT:.LBB2_2: // %lpad
+; CHECK-NEXT: .cfi_restore_state
 entry:
   %call = call i8* @foo1() [ "clang.arc.attachedcall"() ]
   invoke void @objc_object(i8* %call) #5
           to label %invoke.cont unwind label %lpad
 
 invoke.cont:                                      ; preds = %entry
   tail call void @llvm.objc.release(i8* %call)
   ret void

llvm/test/CodeGen/AArch64/cfi-fixup.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux %s -o - | FileCheck %s
+
+define i32 @f0(i32 %x) #0 {
+; CHECK-LABEL: f0:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
+; CHECK-NEXT:    cbz w0, .LBB0_4
+; CHECK-NEXT:  // %bb.1: // %entry
+; CHECK-NEXT:    cmp w0, #2
+; CHECK-NEXT:    b.eq .LBB0_5
+; CHECK-NEXT:  // %bb.2: // %entry
+; CHECK-NEXT:    cmp w0, #1
+; CHECK-NEXT:    b.ne .LBB0_6
+; CHECK-NEXT:  // %bb.3: // %if.then2
+; CHECK-NEXT:    bl g1
+; CHECK-NEXT:    add w0, w0, #1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_4:
+; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    .cfi_remember_state
+; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_5: // %if.then5
+; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    .cfi_remember_state
+; CHECK-NEXT:    bl g0
+; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    sub w0, w8, w0
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_6: // %if.end7
+; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    mov w0, wzr
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+entry:
+  switch i32 %x, label %if.end7 [
+    i32 0, label %return
+    i32 1, label %if.then2
+    i32 2, label %if.then5
+  ]
+
+if.then2:
+  %call = tail call i32 @g1(i32 1)
+  %add = add nsw i32 %call, 1
+  br label %return
+
+if.then5:
+  %call6 = tail call i32 @g0(i32 2)
+  %sub = sub nsw i32 1, %call6
+  br label %return
+
+if.end7:
+  br label %return
+
+return:
+  %retval.0 = phi i32 [ %add, %if.then2 ], [ %sub, %if.then5 ], [ 0, %if.end7 ], [ 1, %entry ]
+  ret i32 %retval.0
+}
+
+declare i32 @g1(i32)
+
+declare i32 @g0(i32)
+
+define i32 @f1(i32 %x) #0 {
+; CHECK-LABEL: f1:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    cbz w0, .LBB1_2
+; CHECK-NEXT:  // %bb.1: // %if.end
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    bl g0
+; CHECK-NEXT:    add w0, w0, #1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:  .LBB1_2: // %return
+; CHECK-NEXT:    ret
+entry:
+  %cmp = icmp eq i32 %x, 0
+  br i1 %cmp, label %return, label %if.end
+
+if.end:
+  %call = tail call i32 @g0(i32 %x)
+  %add = add nsw i32 %call, 1
+  br label %return
+
+return:
+  %retval.0 = phi i32 [ %add, %if.end ], [ 0, %entry ]
+  ret i32 %retval.0
+}
+
+define i32 @f2(i32 %x) #0 {
+; CHECK-LABEL: f2:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
+; CHECK-NEXT:    cbz w0, .LBB2_2
+; CHECK-NEXT:  // %bb.1: // %if.end
+; CHECK-NEXT:    bl g1
+; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    sub w0, w8, w0
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB2_2: // %if.then
+; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    bl g0
+; CHECK-NEXT:    add w0, w0, #1
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    ret
+entry:
+  %cmp = icmp eq i32 %x, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:
+  %call = tail call i32 @g0(i32 0)
+  %add = add nsw i32 %call, 1
+  br label %return
+
+if.end:
+  %call1 = tail call i32 @g1(i32 %x)
+  %sub = sub nsw i32 1, %call1
+  br label %return
+
+return:
+  %retval.0 = phi i32 [ %add, %if.then ], [ %sub, %if.end ]
+  ret i32 %retval.0
+}
+
+attributes #0 = { uwtable }

llvm/test/CodeGen/AArch64/cfi-fixup.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64 -run-pass=aarch64-cfi-fixup %s -o - | FileCheck %s
+--- |
+  define i32 @f0(i32 %x) #0 {
+  entry: br label %return
+  if.end: br label %return
+  if.then2: br label %return
+  if.else: br label %return
+  return:
+    ret i32 0
+  }
+
+  define i32 @f1(i32 %x) #0 {
+  entry: br label %return
+  if.end: br label %return
+  if.then2: br label %return
+  if.else: br label %return
+  return:
+    ret i32 0
+  }
+
+  define i32 @f2(i32 %x) #0 {
+  entry: br label %return
+  if.end: br label %return
+  if.then2: br label %return
+  if.else: br label %return
+  return:
+    ret i32 0
+  }
+
+  declare i32 @g(i32)
+
+  attributes #0 = { nounwind shadowcallstack uwtable "sign-return-address"="non-leaf" "target-features"="+reserve-x18" }
+
+...
+---
+name:            f0
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+failsVerification: false
+registers:       []
+liveins:
+  - { reg: '$w0', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       16
+  offsetAdjustment: 0
+  maxAlignment:    16
+  adjustsStack:    true
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -16, size: 8, alignment: 16,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  ; CHECK-LABEL: name: f0
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.4(0x30000000), %bb.1(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CBZW renamable $w0, %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.if.end:
+  ; CHECK-NEXT:   successors: %bb.3(0x30000000), %bb.2(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+  ; CHECK-NEXT:   frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION def_cfa_offset 16
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION offset $w30, -16
+  ; CHECK-NEXT:   CFI_INSTRUCTION remember_state
+  ; CHECK-NEXT:   TBNZW renamable $w0, 31, %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.if.else:
+  ; CHECK-NEXT:   successors: %bb.5(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w8 = MOVZWi 1, 0
+  ; CHECK-NEXT:   $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+  ; CHECK-NEXT:   B %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.if.then2:
+  ; CHECK-NEXT:   successors: %bb.5(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   B %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CFI_INSTRUCTION def_cfa $wsp, 0
+  ; CHECK-NEXT:   CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   CFI_INSTRUCTION same_value $w18
+  ; CHECK-NEXT:   CFI_INSTRUCTION same_value $w30
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CFI_INSTRUCTION restore_state
+  ; CHECK-NEXT:   early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION def_cfa_offset 0
+  ; CHECK-NEXT:   frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION restore $w18
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION restore $w30
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  bb.0.entry:
+    successors: %bb.4(0x30000000), %bb.1(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    CBZW renamable $w0, %bb.4
+
+  bb.1.if.end:
+    successors: %bb.3(0x30000000), %bb.2(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+    frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+    frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    frame-setup CFI_INSTRUCTION negate_ra_sign_state
+    early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    frame-setup CFI_INSTRUCTION offset $w30, -16
+    TBNZW renamable $w0, 31, %bb.3
+
+  bb.2.if.else:
+    successors: %bb.5(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w8 = MOVZWi 1, 0
+    $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+    B %bb.5
+
+  bb.3.if.then2:
+    successors: %bb.5(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+    B %bb.5
+
+  bb.4.return:
+    liveins: $w0
+    RET undef $lr, implicit killed $w0
+
+  bb.5.return:
+    liveins: $w0
+
+    early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 0
+    frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    frame-destroy CFI_INSTRUCTION negate_ra_sign_state
+    early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+    frame-destroy CFI_INSTRUCTION restore $w18
+    frame-destroy CFI_INSTRUCTION restore $w30
+    RET undef $lr, implicit killed $w0
+
+...
+---
+name:            f1
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+failsVerification: false
+registers:       []
+liveins:
+  - { reg: '$w0', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       16
+  offsetAdjustment: 0
+  maxAlignment:    16
+  adjustsStack:    true
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -16, size: 8, alignment: 16,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  ; CHECK-LABEL: name: f1
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.5(0x30000000), %bb.1(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CBZW renamable $w0, %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.if.end:
+  ; CHECK-NEXT:   successors: %bb.3(0x30000000), %bb.2(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+  ; CHECK-NEXT:   frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION def_cfa_offset 16
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION offset $w30, -16
+  ; CHECK-NEXT:   TBNZW renamable $w0, 31, %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.if.else:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w8 = MOVZWi 1, 0
+  ; CHECK-NEXT:   $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.if.then2:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION def_cfa_offset 0
+  ; CHECK-NEXT:   frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION restore $w18
+  ; CHECK-NEXT:   frame-destroy CFI_INSTRUCTION restore $w30
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  bb.0.entry:
+    successors: %bb.5(0x30000000), %bb.1(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    CBZW renamable $w0, %bb.5
+
+  bb.1.if.end:
+    successors: %bb.3(0x30000000), %bb.2(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+    frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+    frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    frame-setup CFI_INSTRUCTION negate_ra_sign_state
+    early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    frame-setup CFI_INSTRUCTION offset $w30, -16
+    TBNZW renamable $w0, 31, %bb.3
+
+  bb.2.if.else:
+    successors: %bb.4(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w8 = MOVZWi 1, 0
+    $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+    B %bb.4
+
+  bb.3.if.then2:
+    successors: %bb.4(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+    B %bb.4
+
+  bb.4.return:
+    liveins: $w0
+
+    early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+    frame-destroy CFI_INSTRUCTION def_cfa_offset 0
+    frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    frame-destroy CFI_INSTRUCTION negate_ra_sign_state
+    early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+    frame-destroy CFI_INSTRUCTION restore $w18
+    frame-destroy CFI_INSTRUCTION restore $w30
+    RET undef $lr, implicit killed $w0
+
+  bb.5.return:
+    liveins: $w0
+    RET undef $lr, implicit killed $w0
+
+...
+---
+name:            f2
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+failsVerification: false
+registers:       []
+liveins:
+  - { reg: '$w0', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       16
+  offsetAdjustment: 0
+  maxAlignment:    16
+  adjustsStack:    true
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -16, size: 8, alignment: 16,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo:
+  hasRedZone:      false
+body:             |
+  ; CHECK-LABEL: name: f2
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.5(0x30000000), %bb.1(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CBZW renamable $w0, %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.if.end:
+  ; CHECK-NEXT:   successors: %bb.3(0x30000000), %bb.2(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0, $lr, $x18
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+  ; CHECK-NEXT:   frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION def_cfa_offset 16
+  ; CHECK-NEXT:   frame-setup CFI_INSTRUCTION offset $w30, -16
+  ; CHECK-NEXT:   TBNZW renamable $w0, 31, %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.if.else:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w8 = MOVZWi 1, 0
+  ; CHECK-NEXT:   $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.if.then2:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+  ; CHECK-NEXT:   frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+  ; CHECK-NEXT:   early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.return:
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CFI_INSTRUCTION def_cfa $wsp, 0
+  ; CHECK-NEXT:   CFI_INSTRUCTION negate_ra_sign_state
+  ; CHECK-NEXT:   CFI_INSTRUCTION same_value $w18
+  ; CHECK-NEXT:   CFI_INSTRUCTION same_value $w30
+  ; CHECK-NEXT:   RET undef $lr, implicit killed $w0
+  bb.0.entry:
+    successors: %bb.5(0x30000000), %bb.1(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    CBZW renamable $w0, %bb.5
+
+  bb.1.if.end:
+    successors: %bb.3(0x30000000), %bb.2(0x50000000)
+    liveins: $w0, $lr, $x18
+
+    early-clobber $x18 = frame-setup STRXpost $lr, $x18, 8
+    frame-setup CFI_INSTRUCTION escape 0x16, 0x12, 0x02, 0x82, 0x78
+    frame-setup PACIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    frame-setup CFI_INSTRUCTION negate_ra_sign_state
+    early-clobber $sp = frame-setup STRXpre killed $lr, $sp, -16 :: (store (s64) into %stack.0)
+    frame-setup CFI_INSTRUCTION def_cfa_offset 16
+    frame-setup CFI_INSTRUCTION offset $w30, -16
+    TBNZW renamable $w0, 31, %bb.3
+
+  bb.2.if.else:
+    successors: %bb.4(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nuw nsw ADDWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w8 = MOVZWi 1, 0
+    $w0 = SUBWrs killed renamable $w8, killed renamable $w0, 0
+    B %bb.4
+
+  bb.3.if.then2:
+    successors: %bb.4(0x80000000)
+    liveins: $w0
+
+    renamable $w0 = nsw SUBWri killed renamable $w0, 1, 0
+    BL @g, csr_aarch64_aapcs_scs, implicit-def dead $lr, implicit $sp, implicit killed $w0, implicit-def $sp, implicit-def $w0
+    renamable $w0 = nsw ADDWri killed renamable $w0, 1, 0
+    B %bb.4
+
+  bb.4.return:
+    liveins: $w0
+
+    early-clobber $sp, $lr = frame-destroy LDRXpost $sp, 16 :: (load (s64) from %stack.0)
+    frame-destroy AUTIASP implicit-def $lr, implicit killed $lr, implicit $sp
+    early-clobber $x18, $lr = frame-destroy LDRXpre $x18, -8
+    RET undef $lr, implicit killed $w0
+
+  bb.5.return:
+    liveins: $w0
+    RET undef $lr, implicit killed $w0
+
+...

llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll

 
 define i32 @do_nothing_if_compares_can_not_be_adjusted_to_each_other() #0 {
 ; CHECK-LABEL: do_nothing_if_compares_can_not_be_adjusted_to_each_other:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    adrp x8, :got:a
 ; CHECK-NEXT:    ldr x8, [x8, :got_lo12:a]
 ; CHECK-NEXT:    ldr w8, [x8]
 ; CHECK-NEXT:    cmp w8, #0
 ; CHECK-NEXT:    b.gt .LBB8_3
 ; CHECK-NEXT:  // %bb.1: // %while.body.preheader
 ; CHECK-NEXT:    sub w19, w8, #1
 ; CHECK-NEXT:  .LBB8_2: // %while.body

 ; CHECK-NEXT:  // %bb.5:
 ; CHECK-NEXT:    mov w0, #123
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB8_6: // %if.end
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 entry:
   %0 = load i32, i32* @a, align 4

 
 declare double @yoo(i32)
 
 declare i32 @xoo(i32, i32)
 
 declare i32 @woo(double, double)
 
 attributes #0 = { uwtable }
 No newline at end of file

llvm/test/CodeGen/AArch64/cond-br-tuning.ll

 ; Don't transform since the call will clobber the NZCV bits.
 define void @test_call_clobber(i32 %unused, i32 %a) uwtable {
 ; CHECK-LABEL: test_call_clobber:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    and w19, w1, #0x6
 ; CHECK-NEXT:    mov w0, w19
 ; CHECK-NEXT:    bl bar
 ; CHECK-NEXT:    cbnz w19, .LBB9_2
 ; CHECK-NEXT:  // %bb.1: // %if.end
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB9_2: // %if.then
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    bl foo
 entry:
   %c = and i32 %a, 6
   call void @bar(i32 %c)
   %tobool = icmp eq i32 %c, 0
   br i1 %tobool, label %if.end, label %if.then
 
 if.then:
   tail call void @foo()
   unreachable
 
 if.end:
   ret void
 }

llvm/test/CodeGen/AArch64/csr-split.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=aarch64-unknown-linux < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mtriple=arm64-apple-ios < %s | FileCheck %s --check-prefix=CHECK-APPLE
 
 ; Check CSR split can work properly for tests below.
 
 @a = common dso_local local_unnamed_addr global i32 0, align 4
 
 define dso_local signext i32 @test1(i32* %b) local_unnamed_addr uwtable  {
 ; CHECK-LABEL: test1:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    adrp x8, a
 ; CHECK-NEXT:    ldrsw x8, [x8, :lo12:a]
 ; CHECK-NEXT:    cmp x8, x0
 ; CHECK-NEXT:    b.eq .LBB0_2
 ; CHECK-NEXT:  // %bb.1: // %if.end
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %if.then
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    bl callVoid
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    b callNonVoid

 
 define dso_local signext i32 @test2(i32* %p1) local_unnamed_addr uwtable  {
 ; CHECK-LABEL: test2:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    cbz x0, .LBB1_3
 ; CHECK-NEXT:  // %bb.1: // %entry
 ; CHECK-NEXT:    adrp x8, a
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    ldrsw x8, [x8, :lo12:a]
 ; CHECK-NEXT:    cmp x8, x0
 ; CHECK-NEXT:    b.ne .LBB1_3
 ; CHECK-NEXT:  // %bb.2: // %if.then2
 ; CHECK-NEXT:    bl callVoid
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    b callNonVoid
 ; CHECK-NEXT:  .LBB1_3: // %return
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ;
 ; CHECK-APPLE-LABEL: test2:

llvm/test/CodeGen/AArch64/merge-store-dependency.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mcpu cortex-a53 -mtriple=aarch64-eabi | FileCheck %s --check-prefix=A53
 
 ; PR26827 - Merge stores causes wrong dependency.
 %struct1 = type { %struct1*, %struct1*, i32, i32, i16, i16, void (i32, i32, i8*)*, i8* }
 @gv0 = internal unnamed_addr global i32 0, align 4
 @gv1 = internal unnamed_addr global %struct1** null, align 8
 
 define void @test(%struct1* %fde, i32 %fd, void (i32, i32, i8*)* %func, i8* %arg) uwtable {
 ;CHECK-LABEL: test
 ; A53-LABEL: test:
 ; A53:       // %bb.0: // %entry
 ; A53-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
 ; A53-NEXT:    .cfi_def_cfa_offset 16
 ; A53-NEXT:    .cfi_offset w19, -8
 ; A53-NEXT:    .cfi_offset w30, -16
+; A53-NEXT:    .cfi_remember_state
 ; A53-NEXT:    movi v0.2d, #0000000000000000
 ; A53-NEXT:    mov x8, x0
 ; A53-NEXT:    mov x19, x8
 ; A53-NEXT:    mov w0, w1
 ; A53-NEXT:    mov w9, #256
 ; A53-NEXT:    stp x2, x3, [x8, #32]
 ; A53-NEXT:    mov x2, x8
 ; A53-NEXT:    str q0, [x19, #16]!

 ; A53-NEXT:    str x0, [x8, :lo12:gv1]
 ; A53-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; A53-NEXT:    .cfi_def_cfa_offset 0
 ; A53-NEXT:    .cfi_restore w19
 ; A53-NEXT:    .cfi_restore w30
 ; A53-NEXT:    ret
 ; A53-NEXT:  .LBB0_4: // %while.body.i.split
 ; A53-NEXT:    // =>This Inner Loop Header: Depth=1
+; A53-NEXT:    .cfi_restore_state
 ; A53-NEXT:    b .LBB0_4
 entry:
   %0 = bitcast %struct1* %fde to i8*
   tail call void @llvm.memset.p0i8.i64(i8* align 8 %0, i8 0, i64 40, i1 false)
   %state = getelementptr inbounds %struct1, %struct1* %fde, i64 0, i32 4
   store i16 256, i16* %state, align 8
   %fd1 = getelementptr inbounds %struct1, %struct1* %fde, i64 0, i32 2
   store i32 %fd, i32* %fd1, align 8

llvm/test/CodeGen/AArch64/nomerge.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64 -o - | FileCheck %s
 
-define void @foo(i32 %i) {
+define void @foo(i32 %i) uwtable {
+; CHECK-LABEL: foo:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
+; CHECK-NEXT:    cmp w0, #7
+; CHECK-NEXT:    b.eq .LBB0_3
+; CHECK-NEXT:  // %bb.1: // %entry
+; CHECK-NEXT:    cmp w0, #5
+; CHECK-NEXT:    b.ne .LBB0_4
+; CHECK-NEXT:  // %bb.2: // %if.then
+; CHECK-NEXT:    bl bar
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    b bar
+; CHECK-NEXT:  .LBB0_3: // %if.then2
+; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    bl bar
+; CHECK-NEXT:  .LBB0_4: // %if.end3
+; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:    .cfi_def_cfa_offset 0
+; CHECK-NEXT:    .cfi_restore w30
+; CHECK-NEXT:    b bar
 entry:
   switch i32 %i, label %if.end3 [
     i32 5, label %if.then
     i32 7, label %if.then2
   ]
 
 if.then:
   tail call void @bar() #0
   br label %if.end3
 
 if.then2:
   tail call void @bar() #0
   br label %if.end3
 
 if.end3:
   tail call void @bar() #0
   ret void
 }
 
 declare void @bar()
 
 attributes #0 = { nomerge }
-
-; CHECK-LABEL: foo:
-; CHECK: // %bb.0: // %entry
-; CHECK: // %bb.1: // %entry
-; CHECK: // %bb.2: // %if.then
-; CHECK-NEXT: bl bar
-; CHECK: b bar
-; CHECK: .LBB0_3: // %if.then2
-; CHECK-NEXT: bl bar
-; CHECK: .LBB0_4: // %if.end3
-; CHECK: b bar

llvm/test/CodeGen/AArch64/optimize-cond-branch.ll

 ; CHECK-LABEL: func:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #1
 ; CHECK-NEXT:    cbnz w8, .LBB0_3
 ; CHECK-NEXT:  // %bb.1: // %b1
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    cbz wzr, .LBB0_4
 ; CHECK-NEXT:  // %bb.2: // %b3
 ; CHECK-NEXT:    ldr w8, [x8]
 ; CHECK-NEXT:    and w0, w8, #0x100
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    cbz w0, .LBB0_5
 ; CHECK-NEXT:  .LBB0_3: // %common.ret.sink.split
 ; CHECK-NEXT:    b extfunc
 ; CHECK-NEXT:  .LBB0_4: // %b2
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    bl extfunc
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    cbnz w0, .LBB0_3
 ; CHECK-NEXT:  .LBB0_5: // %common.ret
 ; CHECK-NEXT:    ret
   %c0 = icmp sgt i64 0, 0

llvm/test/CodeGen/AArch64/stack-guard-sysreg.ll

 ; CHECK-LABEL:   foo:
 ; CHECK:         // %bb.0: // %entry
 ; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    mov x29, sp
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    sub sp, sp, #16
 ; CHECK-NEXT:    mrs x8, SP_EL0
 ; CHECK-NEXT:    lsl x9, x0, #2
 ; CHECK-NO-OFFSET:       ldr x8, [x8]
 ; CHECK-POSITIVE-OFFSET: ldr x8, [x8, #8]
 ; CHECK-NEGATIVE-OFFSET: ldur x8, [x8, #-8]
 ; CHECK-NPOT-OFFSET:     ldur x8, [x8, #1]
 ; CHECK-NPOT-NEG-OFFSET: ldur x8, [x8, #-1]

 ; CHECK-NEXT:    mov sp, x29
 ; CHECK-NEXT:    .cfi_def_cfa wsp, 16
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:   .cfi_def_cfa_offset 0
 ; CHECK-NEXT:   .cfi_restore w30
 ; CHECK-NEXT:   .cfi_restore w29
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %entry
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:    bl __stack_chk_fail
 ; CHECK-NOT: __stack_chk_guard
 entry:
   %vla = alloca i32, i64 %t, align 4
   call void @baz(i32* nonnull %vla)
   ret void
 }
 

llvm/test/CodeGen/AArch64/unwind-preserved.ll

 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 16 - 24 * VG
 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 16 - 32 * VG
 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 16 - 40 * VG
 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 16 - 48 * VG
 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 16 - 56 * VG
 ; CHECK-NEXT:    .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 16 - 64 * VG
 ; CHECK-NEXT:    addvl sp, sp, #-2
 ; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0xa0, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 160 * VG
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    str z0, [sp] // 16-byte Folded Spill
 ; CHECK-NEXT:  .Ltmp0:
 ; CHECK-NEXT:    bl may_throw_sve
 ; CHECK-NEXT:  .Ltmp1:
 ; CHECK-NEXT:    str z0, [sp, #1, mul vl] // 16-byte Folded Spill
 ; CHECK-NEXT:    b .LBB0_1
 ; CHECK-NEXT:  .LBB0_1: // %.Lcontinue
 ; CHECK-NEXT:    ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload

 ; CHECK-NEXT:    .cfi_restore z14
 ; CHECK-NEXT:    .cfi_restore z15
 ; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    .cfi_restore w29
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %.Lunwind
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:  .Ltmp2:
 ; CHECK-NEXT:    ldr z0, [sp] // 16-byte Folded Reload
 ; CHECK-NEXT:    addvl sp, sp, #2
 ; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0x90, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 144 * VG
 ; CHECK-NEXT:    ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
 ; CHECK-NEXT:    ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
 ; CHECK-NEXT:    ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
 ; CHECK-NEXT:    ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload

 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4a, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x68, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d10 @ cfa - 16 - 24 * VG
 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4b, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x60, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d11 @ cfa - 16 - 32 * VG
 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4c, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x58, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d12 @ cfa - 16 - 40 * VG
 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4d, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x50, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d13 @ cfa - 16 - 48 * VG
 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4e, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x48, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d14 @ cfa - 16 - 56 * VG
 ; GISEL-NEXT:    .cfi_escape 0x10, 0x4f, 0x0a, 0x11, 0x70, 0x22, 0x11, 0x40, 0x92, 0x2e, 0x00, 0x1e, 0x22 // $d15 @ cfa - 16 - 64 * VG
 ; GISEL-NEXT:    addvl sp, sp, #-2
 ; GISEL-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0xa0, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 160 * VG
+; GISEL-NEXT:    .cfi_remember_state
 ; GISEL-NEXT:    str z0, [sp] // 16-byte Folded Spill
 ; GISEL-NEXT:  .Ltmp0:
 ; GISEL-NEXT:    bl may_throw_sve
 ; GISEL-NEXT:  .Ltmp1:
 ; GISEL-NEXT:    str z0, [sp, #1, mul vl] // 16-byte Folded Spill
 ; GISEL-NEXT:    b .LBB0_1
 ; GISEL-NEXT:  .LBB0_1: // %.Lcontinue
 ; GISEL-NEXT:    ldr z0, [sp, #1, mul vl] // 16-byte Folded Reload

 ; GISEL-NEXT:    .cfi_restore z14
 ; GISEL-NEXT:    .cfi_restore z15
 ; GISEL-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
 ; GISEL-NEXT:    .cfi_def_cfa_offset 0
 ; GISEL-NEXT:    .cfi_restore w30
 ; GISEL-NEXT:    .cfi_restore w29
 ; GISEL-NEXT:    ret
 ; GISEL-NEXT:  .LBB0_2: // %.Lunwind
+; GISEL-NEXT:    .cfi_restore_state
 ; GISEL-NEXT:  .Ltmp2:
 ; GISEL-NEXT:    ldr z0, [sp] // 16-byte Folded Reload
 ; GISEL-NEXT:    addvl sp, sp, #2
 ; GISEL-NEXT:    .cfi_escape 0x0f, 0x0d, 0x8f, 0x00, 0x11, 0x10, 0x22, 0x11, 0x90, 0x01, 0x92, 0x2e, 0x00, 0x1e, 0x22 // sp + 16 + 144 * VG
 ; GISEL-NEXT:    ldr p15, [sp, #4, mul vl] // 2-byte Folded Reload
 ; GISEL-NEXT:    ldr p14, [sp, #5, mul vl] // 2-byte Folded Reload
 ; GISEL-NEXT:    ldr p13, [sp, #6, mul vl] // 2-byte Folded Reload
 ; GISEL-NEXT:    ldr p12, [sp, #7, mul vl] // 2-byte Folded Reload

 ; CHECK-NEXT:    .cfi_offset b16, -160
 ; CHECK-NEXT:    .cfi_offset b17, -176
 ; CHECK-NEXT:    .cfi_offset b18, -192
 ; CHECK-NEXT:    .cfi_offset b19, -208
 ; CHECK-NEXT:    .cfi_offset b20, -224
 ; CHECK-NEXT:    .cfi_offset b21, -240
 ; CHECK-NEXT:    .cfi_offset b22, -256
 ; CHECK-NEXT:    .cfi_offset b23, -272
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    str q0, [sp] // 16-byte Folded Spill
 ; CHECK-NEXT:  .Ltmp3:
 ; CHECK-NEXT:    bl may_throw_neon
 ; CHECK-NEXT:  .Ltmp4:
 ; CHECK-NEXT:    str q0, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    b .LBB1_1
 ; CHECK-NEXT:  .LBB1_1: // %.Lcontinue
 ; CHECK-NEXT:    ldr q0, [sp, #16] // 16-byte Folded Reload

 ; CHECK-NEXT:    .cfi_restore b18
 ; CHECK-NEXT:    .cfi_restore b19
 ; CHECK-NEXT:    .cfi_restore b20
 ; CHECK-NEXT:    .cfi_restore b21
 ; CHECK-NEXT:    .cfi_restore b22
 ; CHECK-NEXT:    .cfi_restore b23
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB1_2: // %.Lunwind
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:  .Ltmp5:
 ; CHECK-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
 ; CHECK-NEXT:    ldp x29, x30, [sp, #288] // 16-byte Folded Reload
 ; CHECK-NEXT:    ldp q9, q8, [sp, #256] // 32-byte Folded Reload
 ; CHECK-NEXT:    ldp q11, q10, [sp, #224] // 32-byte Folded Reload
 ; CHECK-NEXT:    ldp q13, q12, [sp, #192] // 32-byte Folded Reload
 ; CHECK-NEXT:    ldp q15, q14, [sp, #160] // 32-byte Folded Reload
 ; CHECK-NEXT:    ldp q17, q16, [sp, #128] // 32-byte Folded Reload

 ; GISEL-NEXT:    .cfi_offset b16, -160
 ; GISEL-NEXT:    .cfi_offset b17, -176
 ; GISEL-NEXT:    .cfi_offset b18, -192
 ; GISEL-NEXT:    .cfi_offset b19, -208
 ; GISEL-NEXT:    .cfi_offset b20, -224
 ; GISEL-NEXT:    .cfi_offset b21, -240
 ; GISEL-NEXT:    .cfi_offset b22, -256
 ; GISEL-NEXT:    .cfi_offset b23, -272
+; GISEL-NEXT:    .cfi_remember_state
 ; GISEL-NEXT:    str q0, [sp] // 16-byte Folded Spill
 ; GISEL-NEXT:  .Ltmp3:
 ; GISEL-NEXT:    bl may_throw_neon
 ; GISEL-NEXT:    str q0, [sp, #16] // 16-byte Folded Spill
 ; GISEL-NEXT:  .Ltmp4:
 ; GISEL-NEXT:    b .LBB1_1
 ; GISEL-NEXT:  .LBB1_1: // %.Lcontinue
 ; GISEL-NEXT:    ldr q0, [sp, #16] // 16-byte Folded Reload

 ; GISEL-NEXT:    .cfi_restore b18
 ; GISEL-NEXT:    .cfi_restore b19
 ; GISEL-NEXT:    .cfi_restore b20
 ; GISEL-NEXT:    .cfi_restore b21
 ; GISEL-NEXT:    .cfi_restore b22
 ; GISEL-NEXT:    .cfi_restore b23
 ; GISEL-NEXT:    ret
 ; GISEL-NEXT:  .LBB1_2: // %.Lunwind
+; GISEL-NEXT:    .cfi_restore_state
 ; GISEL-NEXT:  .Ltmp5:
 ; GISEL-NEXT:    ldr q0, [sp] // 16-byte Folded Reload
 ; GISEL-NEXT:    ldp x29, x30, [sp, #288] // 16-byte Folded Reload
 ; GISEL-NEXT:    ldp q9, q8, [sp, #256] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q11, q10, [sp, #224] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q13, q12, [sp, #192] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q15, q14, [sp, #160] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q17, q16, [sp, #128] // 32-byte Folded Reload

llvm/test/Transforms/CodeGenPrepare/AArch64/large-offset-gep.ll

 ; CHECK-NEXT:  // %bb.0: // %entry
 ; CHECK-NEXT:    stp x30, x21, [sp, #-32]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    stp x20, x19, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w19, -8
 ; CHECK-NEXT:    .cfi_offset w20, -16
 ; CHECK-NEXT:    .cfi_offset w21, -24
 ; CHECK-NEXT:    .cfi_offset w30, -32
+; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    mov w19, w0
 ; CHECK-NEXT:    mov w20, wzr
 ; CHECK-NEXT:    mov w21, #40000
 ; CHECK-NEXT:  .LBB3_1: // %while_cond
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:  .Ltmp0:
 ; CHECK-NEXT:    bl foo
 ; CHECK-NEXT:  .Ltmp1:

 ; CHECK-NEXT:    ldp x30, x21, [sp], #32 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w20
 ; CHECK-NEXT:    .cfi_restore w21
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB3_5: // %cleanup
+; CHECK-NEXT:    .cfi_restore_state
 ; CHECK-NEXT:  .Ltmp2:
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    bl foo2
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    bl _Unwind_Resume
 entry:
   br label %while_cond