[CodeGen][Shrink-wrap]split restore point

[sushgokh]

Summary

This change split a restore point to allow it to only post-dominate blocks reachable by use or def of CSRs/FI.

This is a WIP and I'm posting it to continue the discussion about :

Bugzilla : https://bugs.llvm.org/show_bug.cgi?id=33868

I will be happy to hear any high level comment/suggestion.

This change itself increase 15% more shrink-wrapping in spec2000/2006/2017 benchmarks. I observed 160% more shrink-wrapping in spec2000/2006/2017 benchmarks if we apply the copy forwarding (D41835), PostRASink (D41463), and this change all together.

[thegameg]

Thanks for working on this! I think the idea is worth pursuing. This kinda reminds me of Post Register Allocation Spill Code Optimization, Christopher Lupo, Kent D. Wilken. A similar issue is what they describe in §4.8. I think if you extract the blocks C, D, E, F from the example, that's your function.

I tried looking into implementing that a while ago, and I mostly gave up because our SESE region infrastructure is not used / tested much, especially during CodeGen. I think your implementation has a big advantage because it doesn't require to build such (expensive, IIRC) constructions and might be much faster since SESE regions are usually not available, and dominators are.

Few thoughts: (please correct me if I'm wrong):

• We don't want to add extra branches. By that I mean that we want to guarantee that we always (or when we know it's not worth having extra branches) have a fall through from NMBB to MBB. In your example this is perfectly fine, and I see that NMBB is always inserted before MBB, which I think should be always fine.
• We want to be sure that, in the end, if the points are not interesting, we don't insert NMBB.
• I would run a verifier or add some statistics / remarks to see if any of the previous points are happening and if it causes any regressions.
• I think doing this kind of simulations on the post-dominator tree itself as @qcolombet suggested sounds interesting. Correct me if I'm wrong, here we're looking to introduce a common post-dominator of all the "dirty" blocks, right?

I observed 160% more shrink-wrapping in spec2000/2006/2017 benchmarks if we apply the copy forwarding (D41835), PostRASink (D41463), and this change all together.

Just to be sure, on AArch64, right? Do you see any performance improvement with this change? Any regressions?

Since you marked this as [WIP] I'll skip any comments on the code itself for now.

[qcolombet]

Just two remarks:

• So far shrink-wrapping was solely an analysis pass, i.e., it didn't modify the input code
• Ideally I'd like we postpone creating new blocks until we decided where we are going to insert the code, otherwise we'll end up with blocks to clean-up later on

Regarding the SESE approach, I think it wouldn't be enough because what you want and not given by SESE is *new* common (post)dominator. E.g., in the motivating example we wanted to merge (A->C) (B->C) into (A->common) (B->common) (common->C).

[thegameg]

In D42600#992723, @qcolombet wrote:

Regarding the SESE approach, I think it wouldn't be enough because what you want and not given by SESE is *new* common (post)dominator. E.g., in the motivating example we wanted to merge (A->C) (B->C) into (A->common) (B->common) (common->C).

Yes, but if I understand the paper correctly, in Figure 4 (left) they basically create an SESE out of what they call "Set 1" in Figure 3 (depending on what they call "execution count cost model"). I read that part a while ago but I think at that point they are deciding whether to use "Region 1" as a save/restore boundary or to create a new SESE for that.

[junbuml]

Based on the comments from Francis and Quentin, I tried to split the restore block only when we know that the block spilt is used as restore point. After performing current shrink wrapping, I ran post shrinking to see if we can shrink save point further by splitting restore point. Currently, this update is still WIP because I do this in shrink wrapping pass. If this approach is generally reasonable and we need to keep the shrink-wrap pass as an analysis pass, then I will move this in a new pass between shrink-wrap and PEI.

[junbuml]

I observed 160% more shrink-wrapping in spec2000/2006/2017 benchmarks if we apply the copy forwarding (D41835), PostRASink (D41463), and this change all together.

Just to be sure, on AArch64, right? Do you see any performance improvement with this change? Any regressions?

Yes it was on AArch64. I applied this change on top of PostRASink (D41463), and I ran performance test for spec :

spec2000/mesa           -1.09 % 
spec2017/perlbench   +1.33 %
spec2017/xalancbmk  +3.04%
spec2017/povray        +3.59 %
spec2006/povray        +4.61%

[junbuml]

There are up/down in the scores of these benchmarks, but in povray many extra shrink wrapping happened in functions shown in profile and dynamic instruction count was decreased with this change: -4.80% (2006/povray) and -4.91% (2017/povray). Further shrink-wrapping happened in one of the hot function in xalanbmk, but the dynamic instruction count wasn't decreased in xalancbmk.

[junbuml]

Hi Francis and Quentin,
Does any of you has any comment about it? I will be happy to hear any high level comment about it.
Thanks,
Jun

[junbuml]

Rebased. In this change, I perform post shrinking where we try to further shrink the save point by splitting the restore point after finishing the regular shrink wrapping. This change is WIP for now because I do this in shrink wrapping pass. If overall approach is acceptable and we want to keep shrink-wrap pass as an analysis pas , I will move it as a new pass. Please take a look and let me know any comment.

[junbuml]

Rebased and minor changes in comments. Please let me know if the approach I'm using here make sense.

[sushgokh]

@junbuml Is it possible for you to rebase this patch and upstream it? With current LLVM trunk, its giving around 7% uplift on SPEC2017 povray

[sushgokh]

@junbuml I will take over this work next week.

Let me know if there are any issues with taking over.

[sushgokh]

This is reland attempt.

The only change to the patch is introducing 'updateTerminator' function. Rest all is inherited from previous revision.

[nickdesaulniers]

Initial pass at basic code style review.

llvm/lib/CodeGen/ShrinkWrap.cpp

388–390

return !TII.analyzeBranch(...

416

MF is unused

429

Use ArrayRef from ADT for parameters.

444

Please add a comment above the definition describing what this function is doing.

454

Please add a comment above the definition describing what this function is doing.

467

remove auto

470

remove {} from for

471

remove auto

474

remove {} from for

479

replace auto with RegisterMaskPair&

486

remove auto

488

remove for {}

497

What's NMBB? Maybe a more descriptive identifier might be nice?

507

remove auto

510

remove for {}

511

remove auto

514

remove for {}

517

remove auto

519

remove for {}

524

remove auto

528

Please add a comment above the definition describing what this function is doing.

538

remove auto

584

remove auto

607

dominates

940–941

should we still return false here if if Changed == true?

Am I reading this right? Did the previous version of ShrinkWrap::runOnMachineFunction never return true?

[thegameg]

llvm/lib/CodeGen/ShrinkWrap.cpp

940–941

Correct, the previous version doesn't change the code, it just tells PrologEpilogInserter where to place the prologue/epilogue through MachineFrameInfo.

I guess here we should return true if we do any edge-splitting.

[sushgokh]

llvm/lib/CodeGen/ShrinkWrap.cpp

940–941

@nickdesaulniers I think no need of returning true if Changed == true because dominance/post-dominance relations are updated after post-shrinking. Is there any other thing that needs updation ?

[sushgokh]

llvm/lib/CodeGen/ShrinkWrap.cpp

497

Will add a comment to indicate that NMBB is short for new restore point.

[qcolombet]

Hey,

I only had a cursory look but the overall direction seems good.

I'll look closer after you add comments that goes into more details of what each function is doing and lay off the nomenclature around dirty, clean, and splittable.

Cheers,
-Quentin

llvm/lib/CodeGen/ShrinkWrap.cpp

104

How about a more descriptive name like enable-shrink-wrap-region-split?

192

Typo: chance => change

199

Here and everywhere: use /// to enable doxygen comments.

200

Please document the method and all the parameters.
In particular what it means to be a clean/dirty predecessor and what we check here.
I see that you have some more comments in the cpp, but I believe it doesn't answer these questions.

Also we should document what we mean by splittable with respect to what because a block should always be splittable in the more general term, but that may not help us.
I.e., we should always be able to do something like

BB:
   <...>

>

BB:
   // fall through
NewBB:
   <...>

That's useless but there is nothing preventing us to do that so technically blocks are splittable :).

391

Both arguments can be const

391

Document what dirty means.

919

The comment starting at " If MF is irreducible, a block may be in a loop without" is duplicated here.

[qcolombet]

llvm/lib/CodeGen/ShrinkWrap.cpp

940–941

If we changed the CFG (or anything) we need to return true. There may be other things that we are destroying without realizing (e.g., live intervals, but that's not applicable here. The point is we don't know which analysis rely on what.)

[sushgokh]

@qcolombet @nickdesaulniers Have addressed all the suggestions

[nickdesaulniers]

Please mark comments as "Done" in phabricator so that it's clear to reviewers which comment threads are still outstanding.

llvm/lib/CodeGen/ShrinkWrap.cpp

210–211

I think it's more appropriate to pass a SmallVectorImpl<MachineBasicBlock*>& rather than SmallVector<MachineBasicBlock *, 2>& (having the parameter specialization of 2 is...overly specific). It would be preferable to use ArrayRef, but you can't push_back an ArrayRef, so the SmallVectorImp<>& is the way to go.

472

Seems like it can be undone though, IIUC?

481

ArrayRef

524

ArrayRef

552–553

SmallVectorImpl<MachineBasicBlock *>&

559

remove auto

604

remove auto

940–941

tryToSplitRestore is modifying the CFG, right? If that happens, we MUST return true.

Should we perhaps skip calling postShrinkWrapping unless HasCandidate is true?

[sushgokh]

llvm/lib/CodeGen/ShrinkWrap.cpp

472

yes, right

[sushgokh]

ping.

[nickdesaulniers]

llvm/lib/CodeGen/ShrinkWrap.cpp

471

ReachableByDirty was removed in the latest diff. Please clean up the comment block, too.

[nickdesaulniers]

@sushgokh , @nathanchance is reporting issues with this patch breaking a few Linux kernel builds https://github.com/ClangBuiltLinux/linux/issues/1833. Can you please revert, and we'll work on getting you a reproducer?

[sushgokh]

@nickdesaulniers yes, will revert it

[sushgokh]

@nickdesaulniers Does this look good ?

--- a/llvm/lib/CodeGen/ShrinkWrap.cpp
+++ b/llvm/lib/CodeGen/ShrinkWrap.cpp
@@ -601,6 +601,16 @@ bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
                                 CleanPreds, TII, RS))
     return false;

+  // FIXME: Currently, we bail out of optimisation if any of the DirtyPreds has
+  // INLINEASM_BR instruction. Also, if restore pt block address is taken, we
+  // need to make suitable changes post restore point split
+  for (MachineBasicBlock *BB : DirtyPreds)
+    for (const MachineInstr &MI : *BB)
+      if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+        for (const MachineOperand &MO : MI.operands())
+          if (MO.isMBB() && MO.getMBB() == InitRestore)
+            return false;
+
   // Trying to reach out to the new save point which dominates all dirty blocks.
   MachineBasicBlock *NewSave =
       FindIDom<>(**DirtyPreds.begin(), DirtyPreds, *MDT, false);

[nathanchance]

@sushgokh I will let Nick comment on the correctness of that diff but I can confirm that applying that diff on top of 6530bd3030d3 (the parent of the revert of this change, bb5befefc6e7) makes both issues that I reported go away.

[nickdesaulniers]

In D42600#4267396, @sushgokh wrote:

@nickdesaulniers Does this look good ?

--- a/llvm/lib/CodeGen/ShrinkWrap.cpp
+++ b/llvm/lib/CodeGen/ShrinkWrap.cpp
@@ -601,6 +601,16 @@ bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
                                 CleanPreds, TII, RS))
     return false;

+  // FIXME: Currently, we bail out of optimisation if any of the DirtyPreds has
+  // INLINEASM_BR instruction. Also, if restore pt block address is taken, we
+  // need to make suitable changes post restore point split
+  for (MachineBasicBlock *BB : DirtyPreds)

Rather than scan every instruction in the block, I think you can use
MachineBasicBlock::isInlineAsmBrIndirectTarget()
Perhaps we can do that before even adding such an MBB to DirtyPreds?

+ for (const MachineInstr &MI : *BB)
+ if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+ for (const MachineOperand &MO : MI.operands())
+ if (MO.isMBB() && MO.getMBB() == InitRestore)
+ return false;
+

// Trying to reach out to the new save point which dominates all dirty blocks.
MachineBasicBlock *NewSave =
    FindIDom<>(**DirtyPreds.begin(), DirtyPreds, *MDT, false);

[alexfh]

In our internal testing we've noticed that this commit causes a ~10-15% regression of the Shootout-ackermann benchmark (https://github.com/llvm/llvm-test-suite/blob/main/SingleSource/Benchmarks/Shootout/ackermann.c) on x86. The benchmark is somewhat noisy, thus, I'm not 100% sure it's a real regression. But maybe you want to check if there's something obviously wrong started happening on that code.

[sushgokh]

@nickdesaulniers Thanks. Will make the necessary changes.

@alexfh Will surely check for the regression and update you if there's any.

[sushgokh]

@alexfh The regression exists. I tried on AArch64. However, its not due this patch itself but there is some significant block reordering taking place at "block-placement" pass.

@nickdesaulniers would it be good to go ahead and commit the patch and later track this issue with a bug raised for it? This is just to avoid major changes ticking in the patch if its delayed.

[nickdesaulniers]

In D42600#4274129, @sushgokh wrote:

@nickdesaulniers would it be good to go ahead and commit the patch and later track this issue with a bug raised for it? This is just to avoid major changes ticking in the patch if its delayed.

No! Our CI will notice this immediately again. Do not break our builds!

[fhahn]

In D42600#4269504, @alexfh wrote:

In our internal testing we've noticed that this commit causes a ~10-15% regression of the Shootout-ackermann benchmark (https://github.com/llvm/llvm-test-suite/blob/main/SingleSource/Benchmarks/Shootout/ackermann.c) on x86. The benchmark is somewhat noisy, thus, I'm not 100% sure it's a real regression. But maybe you want to check if there's something obviously wrong started happening on that code.

We've also seen a ~8% performance regression on internal benchmarks on ARM64.

[sushgokh]

@alexfh @fhahn Thanks for the inputs. Could you please confirm if the regression goes away by applying this?

@@ -471,7 +471,9 @@ tryToSplitRestore(MachineBasicBlock *MBB,
       MBBFallthrough.insert(BB);   MachineBasicBlock *NMBB = MF->CreateMachineBasicBlock();
-  MF->insert(MachineFunction::iterator(MBB), NMBB);
+  // Insert this block at the end of the function. Inserting in between may
+  // interfere with control flow optimizer decisions.
+  MF->insert(MF->end(), NMBB);   
   for (const MachineBasicBlock::RegisterMaskPair &LI : MBB->liveins())
     NMBB->addLiveIn(LI.PhysReg);

[sushgokh]

@alexfh @fhahn Any update ?

[sushgokh]

@nickdesaulniers Need your approval to go ahead with relanding with 2 changes viz. 1. INLINEASM_BR changes 2.Change in block location

Have tested both against SPEC17 and llvm test-suite and figures look good. No major diversions were observed for test suite

[nathanchance]

@sushgokh Has a new revision with the changes that you have mentioned been pushed here for review? I can pull it down and verify that the two configurations that had boot issues are fixed, as well as making sure that no other configurations regress.

[sushgokh]

@nathanchance Thanks for the initiative. Could you please apply the below patch on top of existing patch and check?

--- a/llvm/lib/CodeGen/ShrinkWrap.cpp
+++ b/llvm/lib/CodeGen/ShrinkWrap.cpp
@@ -471,7 +471,9 @@ tryToSplitRestore(MachineBasicBlock *MBB,
       MBBFallthrough.insert(BB);

   MachineBasicBlock *NMBB = MF->CreateMachineBasicBlock();
-  MF->insert(MachineFunction::iterator(MBB), NMBB);
+  // Insert this block at the end of the function. Inserting in between may
+  // interfere with control flow optimizer decisions.
+  MF->insert(MF->end(), NMBB);

   for (const MachineBasicBlock::RegisterMaskPair &LI : MBB->liveins())
     NMBB->addLiveIn(LI.PhysReg);
@@ -577,6 +579,12 @@ bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
       !MPDT->dominates(InitRestore, InitSave))
     return false;

+  // Bail out of the optimization if any of the basic block is target of
+  // INLINEASM_BR instruction
+  for (MachineBasicBlock &MBB : MF)
+    if (MBB.isInlineAsmBrIndirectTarget())
+      return false;
+
   DenseSet<const MachineBasicBlock *> DirtyBBs;
   for (MachineBasicBlock &MBB : MF) {
     if (MBB.isEHPad()) {

[nathanchance]

@sushgokh I tested that diff on top of the revert and I no longer see the boot issues I initially reported or any new regressions. I do not feel like I am knowledgeable enough to approve the reland, hopefully someone else can chime in on that but this should not be blocked on the Linux kernel regression any longer.

[sushgokh]

@nathanchance Thanks for the update :)

@nickdesaulniers Requesting approval for relanding

[nickdesaulniers]

In D42600#4304778, @sushgokh wrote:

@nathanchance Thanks for the update :)

@nickdesaulniers Requesting approval for relanding

Can you please push the updated patch to this phab review (click reopen in the bottom left "Add Action..." menu) or a new one? It's hard for me to look at the diff without the surrounding context, which phab would allow me to view.

In particular, I'm concerned if the checks for isInlineAsmBrIndirectTarget in ShrinkWrap::performShrinkWrapping are still relevant after this change?

I would also like to see a callbr based test added.

[sushgokh]

@nickdesaulniers test for inlineasm_br added as 'noshrink_bb_as_inlineasmbr_target()' in 'shrinkwrap-split-restore-point.mir'

[nickdesaulniers]

LGTM; though for the two tests you converted to use the update_*_test_checks.py, you should do those conversions as separate parent commits. I'm not going to block this commit on that, but it's nice to do in the future.

llvm/test/CodeGen/PowerPC/shrink-wrap.ll

1–4

Consider pre-committing the conversion to update_llc_test_checks.py so that it's more obvious if this change to LLVM changes anything of interest in this test.

llvm/test/CodeGen/PowerPC/shrink-wrap.mir

2

Consider pre-committing the conversion to update_mir_test_checks.py so that it's more obvious if this change to LLVM changes anything of interest in this test.

[efriedma]

It looks like for 32-bit ARM, shrink-wrapping is blocking the formation of pop {pc} instructions to return from a function (so we instead pop {lr}; bx lr). Is this new, or a side-effect of shrink-wrapping more cases? Any idea what's causing it? Is there an issue tracking it?

[ayzhao]

This is causing lld to crash on Chrome builds when PGO and ThinLTO are enabled: https://crbug.com/1443635

[sushgokh]

In D42600#4327559, @efriedma wrote:

It looks like for 32-bit ARM, shrink-wrapping is blocking the formation of pop {pc} instructions to return from a function (so we instead pop {lr}; bx lr). Is this new, or a side-effect of shrink-wrapping more cases? Any idea what's causing it? Is there an issue tracking it?

This should be side effect of doing more shrink wrapping and caused by later passes as this patch just changes the save/restore point and nothing beyond that. Currently, we dont have any issue tracking it but if this is leading to performance regression, then could you please raise a bug for it when I reland it ?

[sushgokh]

In D42600#4327970, @ayzhao wrote:

This is causing lld to crash on Chrome builds when PGO and ThinLTO are enabled: https://crbug.com/1443635

@ayzhao I request you to please provide me following information:

1. Reproducer of the function causing the assert failure
2. llc -print-before=shrink-wrap for the function
3. llc -debug-only=shrink-wrap for function

Also, I request to check if the following patch works since I suspect that PGO is changing the block frequencies:

--- a/llvm/lib/CodeGen/ShrinkWrap.cpp
+++ b/llvm/lib/CodeGen/ShrinkWrap.cpp
@@ -632,7 +632,8 @@ bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
       FindIDom<>(**DirtyPreds.begin(), DirtyPreds, *MDT, false);

   while (NewSave && (hasDirtyPred(ReachableByDirty, *NewSave) ||
-                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency()))
+                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency() ||
+                     MLI->getLoopFor(NewSave)))
     NewSave = FindIDom<>(**NewSave->pred_begin(), NewSave->predecessors(), *MDT,
                          false);

[ayzhao]

In D42600#4329528, @sushgokh wrote:

In D42600#4327970, @ayzhao wrote:

This is causing lld to crash on Chrome builds when PGO and ThinLTO are enabled: https://crbug.com/1443635

@ayzhao I request you to please provide me following information:

1. Reproducer of the function causing the assert failure

patternprops.o.5.precodegen.ll189 KBDownload

~/src/llvm-project/build/bin/llc patternprops.o.5.precodegen.ll
llc: /usr/local/google/home/ayzhao/src/llvm-project/llvm/lib/CodeGen/ShrinkWrap.cpp:666: bool (anonymous namespace)::ShrinkWrap::postShrinkWrapping(bool, llvm::MachineFunction &, llvm::RegScavenger *): Assertion `(!MLI->getLoopFor(Save) && !MLI->getLoopFor(Restore)) && "Unexpected save or restore point in a loop"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc patternprops.o.5.precodegen.ll
1.      Running pass 'Function Pass Manager' on module 'patternprops.o.5.precodegen.ll'.
2.      Running pass 'Shrink Wrapping analysis' on function '@_ZN6icu_7212PatternProps14trimWhiteSpaceEPKDsRi'
 #0 0x0000563786318bd7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x1e44bd7)
 #1 0x0000563786316a8e llvm::sys::RunSignalHandlers() (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x1e42a8e)
 #2 0x00005637863193ba SignalHandler(int) Signals.cpp:0:0
 #3 0x00007fafd965af90 (/lib/x86_64-linux-gnu/libc.so.6+0x3bf90)
 #4 0x00007fafd96a9ccc __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007fafd965aef2 raise ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x00007fafd9645472 abort ./stdlib/abort.c:81:7
 #7 0x00007fafd9645395 _nl_load_domain ./intl/loadmsgcat.c:1177:9
 #8 0x00007fafd9653df2 (/lib/x86_64-linux-gnu/libc.so.6+0x34df2)
 #9 0x000056378598fbbf (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x14bbbbf)
#10 0x00005637857517bc llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x127d7bc)
#11 0x0000563785c6bcb3 llvm::FPPassManager::runOnFunction(llvm::Function&) (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x1797cb3)
#12 0x0000563785c73e11 llvm::FPPassManager::runOnModule(llvm::Module&) (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x179fe11)
#13 0x0000563785c6c6cc llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x17986cc)
#14 0x0000563784a8ad89 main (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x5b6d89)
#15 0x00007fafd964618a __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#16 0x00007fafd9646245 call_init ./csu/../csu/libc-start.c:128:20
#17 0x00007fafd9646245 __libc_start_main ./csu/../csu/libc-start.c:368:5
#18 0x0000563784a857b1 _start (/usr/local/google/home/ayzhao/src/llvm-project/build/bin/llc+0x5b17b1)
[1]    2041024 IOT instruction  ~/src/llvm-project/build/bin/llc patternprops.o.5.precodegen.ll

2. llc -print-before=shrink-wrap for the function

print-before-shrink-wrap-stderr.txt59 KBDownload

print-before-shrink-wrap-stdout.txt24 KBDownload

3. llc -debug-only=shrink-wrap for function

debug-only-shrink-wrap-stderr.txt4 KBDownload

debug-only-shrink-wrap-stdout.txt24 KBDownload

Also, I request to check if the following patch works since I suspect that PGO is changing the block frequencies:

--- a/llvm/lib/CodeGen/ShrinkWrap.cpp
+++ b/llvm/lib/CodeGen/ShrinkWrap.cpp
@@ -632,7 +632,8 @@ bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
       FindIDom<>(**DirtyPreds.begin(), DirtyPreds, *MDT, false);

   while (NewSave && (hasDirtyPred(ReachableByDirty, *NewSave) ||
-                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency()))
+                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency() ||
+                     MLI->getLoopFor(NewSave)))
     NewSave = FindIDom<>(**NewSave->pred_begin(), NewSave->predecessors(), *MDT,
                          false);

That seems to work:

$ ~/src/llvm-project/build/bin/llc patternprops.o.5.precodegen.ll

$ echo $?
0

[sushgokh]

@nickdesaulniers I have landed the patch with optimisation off by default so that future changes to it and relanding will become easy.

@ayzhao Thanks for the inputs.

[sushgokh]

@nickdesaulniers could you please check if its good to reland?

[nickdesaulniers]

In D42600#4363694, @sushgokh wrote:

@nickdesaulniers could you please check if its good to reland?

Are @ayzhao 's and @efriedma 's comments addressed?

[sushgokh]

@ayzhao issue addressed.
@efriedma's issue needs to addressed separately in other pass(if need be) but I don't think this issue will result in performance regression.

[nickdesaulniers]

llvm/test/CodeGen/AArch64/dont-shrink-wrap-stack-mayloadorstore.s

1 ↗

(On Diff #522416)

There's a presubmit failure that this newly added test is missing a RUN line.

[sushgokh]

I submitted the patch without checking that the patch needs revision. I got confused by back to back accept followed by revision request. Apology.

But I have removed the file accidently added

llvm/test/CodeGen/AArch64/dont-shrink-wrap-stack-mayloadorstore.s

1 ↗

(On Diff #522416)

I had added that by mistake(git add -A). Apologies.

llvm/lib/CodeGen/ShrinkWrap.cpp

 STATISTIC(NumCandidates, "Number of shrink-wrapping candidates");
 STATISTIC(NumCandidatesDropped,
           "Number of shrink-wrapping candidates dropped because of frequency");
 
 static cl::opt<cl::boolOrDefault>
 EnableShrinkWrapOpt("enable-shrink-wrap", cl::Hidden,
                     cl::desc("enable the shrink-wrapping pass"));
 static cl::opt<bool> EnablePostShrinkWrapOpt(
-    "enable-shrink-wrap-region-split", cl::init(false), cl::Hidden,
+    "enable-shrink-wrap-region-split", cl::init(true), cl::Hidden,
     cl::desc("enable splitting of the restore block if possible"));
 
 namespace {
 
 /// Class to determine where the safe point to insert the
 /// prologue and epilogue are.
 /// Unlike the paper from Fred C. Chow, PLDI'88, that introduces the
 /// shrink-wrapping term for prologue/epilogue placement, this pass
 /// does not rely on expensive data-flow analysis. Instead we use the
 /// dominance properties and loop information to decide which point
class ShrinkWrap : public MachineFunctionPass {
   /// Update the Save and Restore points such that \p MBB is in
   /// the region that is dominated by Save and post-dominated by Restore
   /// and Save and Restore still match the safe point definition.
   /// Such point may not exist and Save and/or Restore may be null after
   /// this call.
   void updateSaveRestorePoints(MachineBasicBlock &MBB, RegScavenger *RS);
 
   // Try to find safe point based on dominance and block frequency without
   // any change in IR.
   bool performShrinkWrapping(MachineFunction &MF, RegScavenger *RS);
 
   /// This function tries to split the restore point if doing so can shrink the
   /// save point further. \return True if restore point is split.
   bool postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
                           RegScavenger *RS);
 
   /// This function analyzes if the restore point can split to create a new
   /// restore point. This function collects
   /// 1. Any preds of current restore that are reachable by callee save/FI
   /// blocks
   /// - indicated by DirtyPreds
   /// 2. Any preds of current restore that are not DirtyPreds - indicated by
   /// CleanPreds
   /// Both sets should be non-empty for considering restore point split.
   bool checkIfRestoreSplittable(
       const MachineBasicBlock *CurRestore,
       const DenseSet<const MachineBasicBlock *> &ReachableByDirty,
       SmallVectorImpl<MachineBasicBlock *> &DirtyPreds,
       SmallVectorImpl<MachineBasicBlock *> &CleanPreds,
       const TargetInstrInfo *TII, RegScavenger *RS);
 
   /// Initialize the pass for \p MF.
   void init(MachineFunction &MF) {
     RCI.runOnMachineFunction(MF);
     MDT = &getAnalysis<MachineDominatorTree>();
     MPDT = &getAnalysis<MachinePostDominatorTree>();
static MachineBasicBlock *FindIDom(MachineBasicBlock &Block, ListOfBBs BBs,
   return IDom;
 }
 
 static bool isAnalyzableBB(const TargetInstrInfo &TII,
                            MachineBasicBlock &Entry) {
   // Check if the block is analyzable.
   MachineBasicBlock *TBB = nullptr, *FBB = nullptr;
   SmallVector<MachineOperand, 4> Cond;
   return !TII.analyzeBranch(Entry, TBB, FBB, Cond);
 }
 
 /// Determines if any predecessor of MBB is on the path from block that has use
 /// or def of CSRs/FI to MBB.
 /// ReachableByDirty: All blocks reachable from block that has use or def of
 /// CSR/FI.
 static bool
 hasDirtyPred(const DenseSet<const MachineBasicBlock *> &ReachableByDirty,
              const MachineBasicBlock &MBB) {
   for (const MachineBasicBlock *PredBB : MBB.predecessors())
     if (ReachableByDirty.count(PredBB))
       return true;
   return false;
 }
 
 /// Derives the list of all the basic blocks reachable from MBB.
 static void markAllReachable(DenseSet<const MachineBasicBlock *> &Visited,
                              const MachineBasicBlock &MBB) {
   SmallVector<MachineBasicBlock *, 4> Worklist(MBB.succ_begin(),
                                                MBB.succ_end());
   Visited.insert(&MBB);
   while (!Worklist.empty()) {
     MachineBasicBlock *SuccMBB = Worklist.pop_back_val();
     if (!Visited.insert(SuccMBB).second)
       continue;
     Worklist.append(SuccMBB->succ_begin(), SuccMBB->succ_end());
   }
 }
 
 /// Collect blocks reachable by use or def of CSRs/FI.
 static void collectBlocksReachableByDirty(
     const DenseSet<const MachineBasicBlock *> &DirtyBBs,
     DenseSet<const MachineBasicBlock *> &ReachableByDirty) {
   for (const MachineBasicBlock *MBB : DirtyBBs) {
     if (ReachableByDirty.count(MBB))
       continue;
     // Mark all offsprings as reachable.
     markAllReachable(ReachableByDirty, *MBB);
   }
 }
 
 /// \return true if there is a clean path from SavePoint to the original
 /// Restore.
 static bool
 isSaveReachableThroughClean(const MachineBasicBlock *SavePoint,
                             ArrayRef<MachineBasicBlock *> CleanPreds) {
   DenseSet<const MachineBasicBlock *> Visited;
   SmallVector<MachineBasicBlock *, 4> Worklist(CleanPreds.begin(),
                                                CleanPreds.end());
   while (!Worklist.empty()) {
     MachineBasicBlock *CleanBB = Worklist.pop_back_val();
     if (CleanBB == SavePoint)
       return true;
     if (!Visited.insert(CleanBB).second || !CleanBB->pred_size())
       continue;
     Worklist.append(CleanBB->pred_begin(), CleanBB->pred_end());
   }
   return false;
 }
 
 /// This function updates the branches post restore point split.
 ///
 /// Restore point has been split.
 /// Old restore point: MBB
 /// New restore point: NMBB
 /// Any basic block(say BBToUpdate) which had a fallthrough to MBB
 /// previously should
 /// 1. Fallthrough to NMBB iff NMBB is inserted immediately above MBB in the
 /// block layout OR
 /// 2. Branch unconditionally to NMBB iff NMBB is inserted at any other place.
 static void updateTerminator(MachineBasicBlock *BBToUpdate,
                              MachineBasicBlock *NMBB,
                              const TargetInstrInfo *TII) {
   DebugLoc DL = BBToUpdate->findBranchDebugLoc();
   // if NMBB isn't the new layout successor for BBToUpdate, insert unconditional
   // branch to it
   if (!BBToUpdate->isLayoutSuccessor(NMBB))
     TII->insertUnconditionalBranch(*BBToUpdate, NMBB, DL);
 }
 
 /// This function splits the restore point and returns new restore point/BB.
 ///
 /// DirtyPreds: Predessors of \p MBB that are ReachableByDirty
 ///
 /// Decision has been made to split the restore point.
 /// old restore point: \p MBB
 /// new restore point: \p NMBB
 /// This function makes the necessary block layout changes so that
 /// 1. \p NMBB points to \p MBB unconditionally
 /// 2. All dirtyPreds that previously pointed to \p MBB point to \p NMBB
 static MachineBasicBlock *
 tryToSplitRestore(MachineBasicBlock *MBB,
                   ArrayRef<MachineBasicBlock *> DirtyPreds,
                   const TargetInstrInfo *TII) {
   MachineFunction *MF = MBB->getParent();
 
   // get the list of DirtyPreds who have a fallthrough to MBB
   // before the block layout change. This is just to ensure that if the NMBB is
   // inserted after MBB, then we create unconditional branch from
   // DirtyPred/CleanPred to NMBB
   SmallPtrSet<MachineBasicBlock *, 8> MBBFallthrough;
   for (MachineBasicBlock *BB : DirtyPreds)
     if (BB->getFallThrough(false) == MBB)
       MBBFallthrough.insert(BB);
 
   MachineBasicBlock *NMBB = MF->CreateMachineBasicBlock();
   // Insert this block at the end of the function. Inserting in between may
   // interfere with control flow optimizer decisions.
   MF->insert(MF->end(), NMBB);
 
   for (const MachineBasicBlock::RegisterMaskPair &LI : MBB->liveins())
     NMBB->addLiveIn(LI.PhysReg);
 
   TII->insertUnconditionalBranch(*NMBB, MBB, DebugLoc());
 
   // After splitting, all predecessors of the restore point should be dirty
   // blocks.
   for (MachineBasicBlock *SuccBB : DirtyPreds)
     SuccBB->ReplaceUsesOfBlockWith(MBB, NMBB);
 
   NMBB->addSuccessor(MBB);
 
   for (MachineBasicBlock *BBToUpdate : MBBFallthrough)
     updateTerminator(BBToUpdate, NMBB, TII);
 
   return NMBB;
 }
 
 /// This function undoes the restore point split done earlier.
 ///
 /// DirtyPreds: All predecessors of \p NMBB that are ReachableByDirty.
 ///
 /// Restore point was split and the change needs to be unrolled. Make necessary
 /// changes to reset restore point from \p NMBB to \p MBB.
 static void rollbackRestoreSplit(MachineFunction &MF, MachineBasicBlock *NMBB,
                                  MachineBasicBlock *MBB,
                                  ArrayRef<MachineBasicBlock *> DirtyPreds,
                                  const TargetInstrInfo *TII) {
   // For a BB, if NMBB is fallthrough in the current layout, then in the new
   // layout a. BB should fallthrough to MBB OR b. BB should undconditionally
   // branch to MBB
   SmallPtrSet<MachineBasicBlock *, 8> NMBBFallthrough;
   for (MachineBasicBlock *BB : DirtyPreds)
     if (BB->getFallThrough(false) == NMBB)
       NMBBFallthrough.insert(BB);
 
   NMBB->removeSuccessor(MBB);
   for (MachineBasicBlock *SuccBB : DirtyPreds)
     SuccBB->ReplaceUsesOfBlockWith(NMBB, MBB);
 
   NMBB->erase(NMBB->begin(), NMBB->end());
   NMBB->eraseFromParent();
 
   for (MachineBasicBlock *BBToUpdate : NMBBFallthrough)
     updateTerminator(BBToUpdate, MBB, TII);
 }
 
 // A block is deemed fit for restore point split iff there exist
 // 1. DirtyPreds - preds of CurRestore reachable from use or def of CSR/FI
 // 2. CleanPreds - preds of CurRestore that arent DirtyPreds
 bool ShrinkWrap::checkIfRestoreSplittable(
     const MachineBasicBlock *CurRestore,
     const DenseSet<const MachineBasicBlock *> &ReachableByDirty,
     SmallVectorImpl<MachineBasicBlock *> &DirtyPreds,
     SmallVectorImpl<MachineBasicBlock *> &CleanPreds,
     const TargetInstrInfo *TII, RegScavenger *RS) {
   for (const MachineInstr &MI : *CurRestore)
     if (useOrDefCSROrFI(MI, RS))
       return false;
 
   for (MachineBasicBlock *PredBB : CurRestore->predecessors()) {
     if (!isAnalyzableBB(*TII, *PredBB))
       return false;
 
     if (ReachableByDirty.count(PredBB))
       DirtyPreds.push_back(PredBB);
     else
       CleanPreds.push_back(PredBB);
   }
 
   return !(CleanPreds.empty() || DirtyPreds.empty());
 }
 
 bool ShrinkWrap::postShrinkWrapping(bool HasCandidate, MachineFunction &MF,
                                     RegScavenger *RS) {
   if (!EnablePostShrinkWrapOpt)
     return false;
 
   MachineBasicBlock *InitSave = nullptr;
   MachineBasicBlock *InitRestore = nullptr;
 
   if (HasCandidate) {
     InitSave = Save;
     InitRestore = Restore;
   } else {
     InitRestore = nullptr;
     InitSave = &MF.front();
     for (MachineBasicBlock &MBB : MF) {
       if (MBB.isEHFuncletEntry())
         return false;
       if (MBB.isReturnBlock()) {
         // Do not support multiple restore points.
         if (InitRestore)
           return false;
         InitRestore = &MBB;
       }
     }
   }
 
   if (!InitSave || !InitRestore || InitRestore == InitSave ||
       !MDT->dominates(InitSave, InitRestore) ||
       !MPDT->dominates(InitRestore, InitSave))
     return false;
 
   // Bail out of the optimization if any of the basic block is target of
   // INLINEASM_BR instruction
   for (MachineBasicBlock &MBB : MF)
     if (MBB.isInlineAsmBrIndirectTarget())
       return false;
 
   DenseSet<const MachineBasicBlock *> DirtyBBs;
   for (MachineBasicBlock &MBB : MF) {
     if (MBB.isEHPad()) {
       DirtyBBs.insert(&MBB);
       continue;
     }
     for (const MachineInstr &MI : MBB)
if (!checkIfRestoreSplittable(InitRestore, ReachableByDirty, DirtyPreds,
                                 CleanPreds, TII, RS))
     return false;
 
   // Trying to reach out to the new save point which dominates all dirty blocks.
   MachineBasicBlock *NewSave =
       FindIDom<>(**DirtyPreds.begin(), DirtyPreds, *MDT, false);
 
   while (NewSave && (hasDirtyPred(ReachableByDirty, *NewSave) ||
-                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency()))
+                     EntryFreq < MBFI->getBlockFreq(NewSave).getFrequency() ||
+                     /*Entry freq has been observed more than a loop block in
+                        some cases*/
+                     MLI->getLoopFor(NewSave)))
     NewSave = FindIDom<>(**NewSave->pred_begin(), NewSave->predecessors(), *MDT,
                          false);
 
   const TargetFrameLowering *TFI = MF.getSubtarget().getFrameLowering();
   if (!NewSave || NewSave == InitSave ||
       isSaveReachableThroughClean(NewSave, CleanPreds) ||
       !TFI->canUseAsPrologue(*NewSave))
     return false;
bool ShrinkWrap::runOnMachineFunction(MachineFunction &MF) {
   ReversePostOrderTraversal<MachineBasicBlock *> RPOT(&*MF.begin());
   if (containsIrreducibleCFG<MachineBasicBlock *>(RPOT, *MLI)) {
     // If MF is irreducible, a block may be in a loop without
     // MachineLoopInfo reporting it. I.e., we may use the
     // post-dominance property in loops, which lead to incorrect
     // results. Moreover, we may miss that the prologue and
     // epilogue are not in the same loop, leading to unbalanced
     // construction/deconstruction of the stack frame.
     return giveUpWithRemarks(ORE, "UnsupportedIrreducibleCFG",
                              "Irreducible CFGs are not supported yet.",
                              MF.getFunction().getSubprogram(), &MF.front());
   }
 
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   std::unique_ptr<RegScavenger> RS(
       TRI->requiresRegisterScavenging(MF) ? new RegScavenger() : nullptr);
 
   bool Changed = false;
 
   bool HasCandidate = performShrinkWrapping(MF, RS.get());
   Changed = postShrinkWrapping(HasCandidate, MF, RS.get());
   if (!HasCandidate && !Changed)
     return false;
   if (!ArePointsInteresting())
     return Changed;
 
   LLVM_DEBUG(dbgs() << "Final shrink wrap candidates:\nSave: "
                     << Save->getNumber() << ' ' << Save->getName()
                     << "\nRestore: " << Restore->getNumber() << ' '
                     << Restore->getName() << '\n');
 
   MachineFrameInfo &MFI = MF.getFrameInfo();
   MFI.setSavePoint(Save);
   MFI.setRestorePoint(Restore);
   ++NumCandidates;
   return Changed;
 }
 
 bool ShrinkWrap::isShrinkWrapEnabled(const MachineFunction &MF) {

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

 ; CHECK-NEXT:    mov w10, w2
 ; CHECK-NEXT:    cmp w2, #15
 ; CHECK-NEXT:    b.hi .LBB5_4
 ; CHECK-NEXT:  // %bb.2:
 ; CHECK-NEXT:    mov x11, xzr
 ; CHECK-NEXT:    mov w8, wzr
 ; CHECK-NEXT:    b .LBB5_7
 ; CHECK-NEXT:  .LBB5_3:
-; CHECK-NEXT:    mov w0, wzr
-; CHECK-NEXT:    ret
+; CHECK-NEXT:    mov w8, wzr
+; CHECK-NEXT:    b .LBB5_9
 ; CHECK-NEXT:  .LBB5_4: // %vector.ph
 ; CHECK-NEXT:    and x11, x10, #0xfffffff0
 ; CHECK-NEXT:    add x8, x0, #8
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x12, x11
 ; CHECK-NEXT:    movi v1.2d, #0000000000000000
 ; CHECK-NEXT:    dup v2.8h, w9
 ; CHECK-NEXT:  .LBB5_5: // %vector.body

llvm/test/CodeGen/AArch64/dont-shrink-wrap-stack-mayloadorstore.mir

 --- |
  ; pr37472
  ; These test verify that shrink-wrap does not set the restore point
  ; to a position where the stack might still be accessed by a load or store
  
  ; RUN: llc -x=mir -simplify-mir -run-pass=shrink-wrap -o - %s | FileCheck %s
  ; CHECK:      name:            compiler_pop_stack
  ; CHECK:      frameInfo:
- ; CHECK-NOT:  savePoint:
- ; CHECK-NOT:  restorePoint:
+ ; CHECK:      savePoint:       '%bb.1'
+ ; CHECK:      restorePoint:    '%bb.7'
  ; CHECK:      name:            compiler_pop_stack_no_memoperands
  ; CHECK:      frameInfo:
- ; CHECK-NOT:  savePoint:
- ; CHECK-NOT:  restorePoint:
- ; CHECK:      stack:
+ ; CHECK:      savePoint:       '%bb.1'
+ ; CHECK:      restorePoint:    '%bb.7'
  ; CHECK:      name:            f
  ; CHECK:      frameInfo:
  ; CHECK:      savePoint:       '%bb.2'
  ; CHECK-NEXT: restorePoint:    '%bb.4'
  ; CHECK-NEXT: stack:
 
   target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64-arm-none-eabi"

llvm/test/CodeGen/AArch64/ragreedy-csr.ll

 %struct.E_list_struct = type { ptr, ptr }
 %struct.domain_struct = type { ptr, i32, ptr, i32, i32, ptr, ptr }
 %struct.d_tree_leaf_struct = type { ptr, i32, ptr }
 @_DefaultRuneLocale = external global %struct._RuneLocale
 declare i32 @__maskrune(i32, i64) #7
 define fastcc i32 @prune_match(ptr nocapture readonly %a, ptr nocapture readonly %b) #9 {
 ; CHECK-LABEL: prune_match:
 ; CHECK:       ; %bb.0: ; %entry
+; CHECK-NEXT:    ldrh w8, [x0]
+; CHECK-NEXT:    ldrh w9, [x1]
+; CHECK-NEXT:    cmp w8, w9
+; CHECK-NEXT:    b.ne LBB0_47
+; CHECK-NEXT:  ; %bb.1: ; %if.end
 ; CHECK-NEXT:    sub sp, sp, #64
 ; CHECK-NEXT:    .cfi_def_cfa_offset 64
 ; CHECK-NEXT:    stp x29, x30, [sp, #48] ; 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    ldrh w8, [x0]
-; CHECK-NEXT:    ldrh w9, [x1]
-; CHECK-NEXT:    cmp w8, w9
-; CHECK-NEXT:    b.ne LBB0_42
-; CHECK-NEXT:  ; %bb.1: ; %if.end
 ; CHECK-NEXT:  Lloh0:
 ; CHECK-NEXT:    adrp x14, __DefaultRuneLocale@GOTPAGE
 ; CHECK-NEXT:    mov x9, xzr
 ; CHECK-NEXT:    ldrb w12, [x0, #4]
 ; CHECK-NEXT:    ldrb w13, [x1, #4]
 ; CHECK-NEXT:    ldr x10, [x0, #16]
 ; CHECK-NEXT:    ldr x11, [x1, #16]
 ; CHECK-NEXT:  Lloh1:

 ; CHECK-NEXT:    cmp w13, #42
 ; CHECK-NEXT:    b.eq LBB0_37
 ; CHECK-NEXT:  ; %bb.41: ; %while.body139
 ; CHECK-NEXT:    ; in Loop: Header=BB0_38 Depth=1
 ; CHECK-NEXT:    cmp w8, #94
 ; CHECK-NEXT:    b.eq LBB0_37
 ; CHECK-NEXT:  LBB0_42:
 ; CHECK-NEXT:    mov w0, wzr
-; CHECK-NEXT:  LBB0_43: ; %return
+; CHECK-NEXT:  LBB0_43:
 ; CHECK-NEXT:    ldp x29, x30, [sp, #48] ; 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #64
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  LBB0_44: ; %lor.lhs.false47
 ; CHECK-NEXT:    cmp x12, #2
 ; CHECK-NEXT:    b.ne LBB0_11
 ; CHECK-NEXT:  ; %bb.45: ; %land.lhs.true52
 ; CHECK-NEXT:    add x12, x10, x9
 ; CHECK-NEXT:    mov w0, #1 ; =0x1
 ; CHECK-NEXT:    ldurb w12, [x12, #-1]
 ; CHECK-NEXT:    cmp w12, #73
 ; CHECK-NEXT:    b.eq LBB0_43
 ; CHECK-NEXT:  ; %bb.46: ; %land.lhs.true52
 ; CHECK-NEXT:    cbz w8, LBB0_43
 ; CHECK-NEXT:    b LBB0_12
+; CHECK-NEXT:  LBB0_47:
+; CHECK-NEXT:    .cfi_def_cfa wsp, 0
+; CHECK-NEXT:    .cfi_same_value w30
+; CHECK-NEXT:    .cfi_same_value w29
+; CHECK-NEXT:    mov w0, wzr
+; CHECK-NEXT:    ret
 ; CHECK-NEXT:    .loh AdrpLdrGot Lloh0, Lloh1
 ; CHECK-NEXT:    .loh AdrpLdrGot Lloh2, Lloh3
 ; CHECK-NEXT:    .loh AdrpLdrGot Lloh4, Lloh5
 entry:
   %0 = load i16, ptr %a, align 2
   %1 = load i16, ptr %b, align 2
   %cmp = icmp eq i16 %0, %1
   br i1 %cmp, label %if.end, label %return, !prof !988

llvm/test/CodeGen/AArch64/shrinkwrap-split-restore-point.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=aarch64 -run-pass=shrink-wrap -o - %s | FileCheck %s
+
+--- |
+  define void @shrink_test1(i32 %a) {
+  entry:
+    %cmp5 = icmp sgt i32 %a, 0
+    br i1 %cmp5, label %BB0, label %exit
+
+  BB0:                                              ; preds = %entry
+    %call = call i32 @fun()
+    %c = icmp eq i32 %call, 0
+    br i1 %c, label %BB1, label %exit
+
+  BB1:                                              ; preds = %BB0
+    %call2 = call i32 @fun()
+    br label %exit
+
+  exit:                                             ; preds = %BB1, %BB0, %entry
+    ret void
+  }
+
+  define void @shrink_test2(i32 %a, ptr %P1, ptr %P2) {
+  BB00:
+    %cmp5 = icmp sgt i32 %a, 0
+    br i1 %cmp5, label %BB01, label %exit
+
+  BB01:                                             ; preds = %BB00
+    store i32 %a, ptr %P1, align 4
+    %c1 = icmp sgt i32 %a, 1
+    br i1 %c1, label %BB02, label %BB03
+
+  BB02:                                             ; preds = %BB01
+    store i32 %a, ptr %P2, align 4
+    br label %BB03
+
+  BB03:                                             ; preds = %BB02, %BB01
+    %call03 = call i32 @fun()
+    %c03 = icmp eq i32 %call03, 0
+    br i1 %c03, label %BB04, label %BB05
+
+  BB04:                                             ; preds = %BB03
+    %call04 = call i32 @fun()
+    br label %BB05
+
+  BB05:                                             ; preds = %BB04, %BB03
+    %call05 = call i32 @fun()
+    %c05 = icmp eq i32 %call05, 0
+    br i1 %c05, label %BB06, label %BB07
+
+  BB06:                                             ; preds = %BB05
+    %call06 = call i32 @fun()
+    br label %exit
+
+  BB07:                                             ; preds = %BB05
+    %call07 = call i32 @fun2()
+    br label %exit
+
+  exit:                                             ; preds = %BB07, %BB06, %BB00
+    ret void
+  }
+
+  define void @noshrink_test1(i32 %a, i32 %v, i32 %v2) {
+  entry:
+    %cmp5 = icmp sgt i32 %a, 0
+    br i1 %cmp5, label %BB0, label %exit
+
+  BB0:                                              ; preds = %entry
+    %c = icmp eq i32 %a, 10
+    %c1 = icmp eq i32 %v, 10
+    %or.cond = select i1 %c, i1 %c1, i1 false
+    br i1 %or.cond, label %BB3, label %BB2
+
+  BB2:                                              ; preds = %BB0
+    %c2 = icmp eq i32 %v2, 10
+    br i1 %c2, label %BB4, label %exit
+
+  BB3:                                              ; preds = %BB0
+    %call3 = call i32 @fun()
+    br label %exit
+
+  BB4:                                              ; preds = %BB2
+    %call4 = call i32 @fun2()
+    br label %exit
+
+  exit:                                             ; preds = %BB4, %BB3, %BB2, %entry
+    ret void
+  }
+
+  define void @noshrink_test2(i32 %a) {
+  BB00:
+    %cmp5 = icmp sgt i32 %a, 0
+    br i1 %cmp5, label %BB01, label %InfLoop.preheader
+
+  InfLoop.preheader:                                ; preds = %BB00
+    br label %InfLoop
+
+  BB01:                                             ; preds = %BB00
+    %call = call i32 @fun()
+    %c = icmp eq i32 %call, 0
+    br i1 %c, label %BB02, label %exit
+
+  BB02:                                             ; preds = %BB01
+    %call2 = call i32 @fun()
+    br label %exit
+
+  InfLoop:                                          ; preds = %InfLoop.preheader, %InfLoop
+    %call3 = call i32 @fun()
+    br label %InfLoop
+
+  exit:                                             ; preds = %BB02, %BB01
+    ret void
+  }
+
+  define void @noshrink_test3(i32 %a) {
+  BB00:
+    %cmp5 = icmp sgt i32 %a, 0
+    %call02 = call i32 @fun()
+    br i1 %cmp5, label %BB02, label %BB01
+
+  BB01:                                             ; preds = %BB00
+    %0 = icmp eq i32 %call02, 0
+    br i1 %0, label %BB01.1, label %exit
+
+  BB01.1:                                           ; preds = %BB01
+    call void @abort() #0
+    unreachable
+
+  BB02:                                             ; preds = %BB00
+    %1 = icmp eq i32 %call02, 0
+    br i1 %1, label %BB03, label %BB04
+
+  BB03:                                             ; preds = %BB02
+    %call03 = call i32 @fun()
+    %c03 = icmp eq i32 %call03, 0
+    br i1 %c03, label %BB04, label %exit
+
+  BB04:                                             ; preds = %BB03, %BB02
+    %call04 = call i32 @fun()
+    br label %exit
+
+  exit:                                             ; preds = %BB04, %BB03, %BB01
+    ret void
+  }
+
+  define void @noshrink_bb_as_inlineasmbr_target(i1 %cond) {
+  entry:
+    br i1 %cond, label %0, label %exit
+
+  0:                                                ; preds = %entry
+    callbr void asm sideeffect "", "!i,~{flags}"()
+      to label %1 [label %exit]
+
+  1:                                                ; preds = %0
+    call void @dosomething()
+    br label %exit
+
+  exit:                                             ; preds = %1, %0, %entry
+    ret void
+  }
+
+  declare i32 @fun()
+  declare i32 @fun2()
+  declare void @abort()
+  declare void @dosomething()
+...
+---
+name:            shrink_test1
+alignment:       4
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$w0' }
+frameInfo:
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: shrink_test1
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.1(0x50000000), %bb.3(0x30000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w0, 1, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 11, %bb.3, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.BB0:
+  ; CHECK-NEXT:   successors: %bb.2(0x30000000), %bb.4(0x50000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.4
+  ; CHECK-NEXT:   B %bb.2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.BB1:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.exit:
+  ; CHECK-NEXT:   RET_ReallyLR
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4:
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   B %bb.3
+  bb.0.entry:
+    successors: %bb.1(0x50000000), %bb.3(0x30000000)
+    liveins: $w0
+
+    dead $wzr = SUBSWri killed renamable $w0, 1, 0, implicit-def $nzcv
+    Bcc 11, %bb.3, implicit killed $nzcv
+    B %bb.1
+
+  bb.1.BB0:
+    successors: %bb.2(0x30000000), %bb.3(0x50000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    CBNZW killed renamable $w0, %bb.3
+    B %bb.2
+
+  bb.2.BB1:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.3.exit:
+    RET_ReallyLR
+
+...
+---
+name:            shrink_test2
+alignment:       4
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$w0' }
+  - { reg: '$x1' }
+  - { reg: '$x2' }
+frameInfo:
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: shrink_test2
+  ; CHECK: bb.0.BB00:
+  ; CHECK-NEXT:   successors: %bb.1(0x50000000), %bb.8(0x30000000)
+  ; CHECK-NEXT:   liveins: $w0, $x1, $x2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri renamable $w0, 1, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 11, %bb.8, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.BB01:
+  ; CHECK-NEXT:   successors: %bb.2(0x40000000), %bb.3(0x40000000)
+  ; CHECK-NEXT:   liveins: $w0, $x1, $x2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri renamable $w0, 2, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   STRWui renamable $w0, killed renamable $x1, 0 :: (store (s32) into %ir.P1)
+  ; CHECK-NEXT:   Bcc 11, %bb.3, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.BB02:
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT:   liveins: $w0, $x2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   STRWui killed renamable $w0, killed renamable $x2, 0 :: (store (s32) into %ir.P2)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.BB03:
+  ; CHECK-NEXT:   successors: %bb.4(0x30000000), %bb.5(0x50000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.5
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.BB04:
+  ; CHECK-NEXT:   successors: %bb.5(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.BB05:
+  ; CHECK-NEXT:   successors: %bb.6(0x30000000), %bb.7(0x50000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.7
+  ; CHECK-NEXT:   B %bb.6
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.6.BB06:
+  ; CHECK-NEXT:   successors: %bb.9(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.9
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.7.BB07:
+  ; CHECK-NEXT:   successors: %bb.9(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun2, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.9
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.8.exit:
+  ; CHECK-NEXT:   RET_ReallyLR
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.9:
+  ; CHECK-NEXT:   successors: %bb.8(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   B %bb.8
+  bb.0.BB00:
+    successors: %bb.1(0x50000000), %bb.8(0x30000000)
+    liveins: $w0, $x1, $x2
+
+    dead $wzr = SUBSWri renamable $w0, 1, 0, implicit-def $nzcv
+    Bcc 11, %bb.8, implicit killed $nzcv
+    B %bb.1
+
+  bb.1.BB01:
+    successors: %bb.2, %bb.3
+    liveins: $w0, $x1, $x2
+
+    dead $wzr = SUBSWri renamable $w0, 2, 0, implicit-def $nzcv
+    STRWui renamable $w0, killed renamable $x1, 0 :: (store (s32) into %ir.P1)
+    Bcc 11, %bb.3, implicit killed $nzcv
+    B %bb.2
+
+  bb.2.BB02:
+    liveins: $w0, $x2
+
+    STRWui killed renamable $w0, killed renamable $x2, 0 :: (store (s32) into %ir.P2)
+
+  bb.3.BB03:
+    successors: %bb.4(0x30000000), %bb.5(0x50000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    CBNZW killed renamable $w0, %bb.5
+    B %bb.4
+
+  bb.4.BB04:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.5.BB05:
+    successors: %bb.6(0x30000000), %bb.7(0x50000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    CBNZW killed renamable $w0, %bb.7
+    B %bb.6
+
+  bb.6.BB06:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    B %bb.8
+
+  bb.7.BB07:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun2, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.8.exit:
+    RET_ReallyLR
+
+...
+---
+name:            noshrink_test1
+alignment:       4
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$w0' }
+  - { reg: '$w1' }
+  - { reg: '$w2' }
+frameInfo:
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: noshrink_test1
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.1(0x50000000), %bb.6(0x30000000)
+  ; CHECK-NEXT:   liveins: $w0, $w1, $w2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri renamable $w0, 1, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 11, %bb.6, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.BB0:
+  ; CHECK-NEXT:   successors: %bb.2(0x60000000), %bb.3(0x20000000)
+  ; CHECK-NEXT:   liveins: $w0, $w1, $w2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w0, 10, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 1, %bb.3, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.BB0:
+  ; CHECK-NEXT:   successors: %bb.4(0x55555555), %bb.3(0x2aaaaaab)
+  ; CHECK-NEXT:   liveins: $w1, $w2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w1, 10, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 0, %bb.4, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.BB2:
+  ; CHECK-NEXT:   successors: %bb.5(0x40000000), %bb.6(0x40000000)
+  ; CHECK-NEXT:   liveins: $w2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w2, 10, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 0, %bb.5, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.6
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.BB3:
+  ; CHECK-NEXT:   successors: %bb.6(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.6
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.BB4:
+  ; CHECK-NEXT:   successors: %bb.6(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun2, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.6.exit:
+  ; CHECK-NEXT:   RET_ReallyLR
+  bb.0.entry:
+    successors: %bb.1(0x50000000), %bb.6(0x30000000)
+    liveins: $w0, $w1, $w2
+
+    dead $wzr = SUBSWri renamable $w0, 1, 0, implicit-def $nzcv
+    Bcc 11, %bb.6, implicit killed $nzcv
+    B %bb.1
+
+  bb.1.BB0:
+    successors: %bb.2(0x60000000), %bb.3(0x20000000)
+    liveins: $w0, $w1, $w2
+
+    dead $wzr = SUBSWri killed renamable $w0, 10, 0, implicit-def $nzcv
+    Bcc 1, %bb.3, implicit killed $nzcv
+    B %bb.2
+
+  bb.2.BB0:
+    successors: %bb.4(0x55555555), %bb.3(0x2aaaaaab)
+    liveins: $w1, $w2
+
+    dead $wzr = SUBSWri killed renamable $w1, 10, 0, implicit-def $nzcv
+    Bcc 0, %bb.4, implicit killed $nzcv
+    B %bb.3
+
+  bb.3.BB2:
+    liveins: $w2
+
+    dead $wzr = SUBSWri killed renamable $w2, 10, 0, implicit-def $nzcv
+    Bcc 0, %bb.5, implicit killed $nzcv
+    B %bb.6
+
+  bb.4.BB3:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    B %bb.6
+
+  bb.5.BB4:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun2, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.6.exit:
+    RET_ReallyLR
+
+...
+---
+name:            noshrink_test2
+alignment:       4
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$w0' }
+frameInfo:
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: noshrink_test2
+  ; CHECK: bb.0.BB00:
+  ; CHECK-NEXT:   successors: %bb.2(0x50000000), %bb.1(0x30000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w0, 0, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 12, %bb.2, implicit killed $nzcv
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.BB01:
+  ; CHECK-NEXT:   successors: %bb.3(0x30000000), %bb.5(0x50000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.5
+  ; CHECK-NEXT:   B %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.BB02:
+  ; CHECK-NEXT:   successors: %bb.5(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.InfLoop:
+  ; CHECK-NEXT:   successors: %bb.4(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.exit:
+  ; CHECK-NEXT:   RET_ReallyLR
+  bb.0.BB00:
+    successors: %bb.2(0x50000000), %bb.1(0x30000000)
+    liveins: $w0
+
+    dead $wzr = SUBSWri killed renamable $w0, 0, 0, implicit-def $nzcv
+    Bcc 12, %bb.2, implicit killed $nzcv
+
+  bb.1:
+    B %bb.4
+
+  bb.2.BB01:
+    successors: %bb.3(0x30000000), %bb.5(0x50000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    CBNZW killed renamable $w0, %bb.5
+    B %bb.3
+
+  bb.3.BB02:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    B %bb.5
+
+  bb.4.InfLoop:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    B %bb.4
+
+  bb.5.exit:
+    RET_ReallyLR
+
+...
+---
+name:            noshrink_test3
+alignment:       4
+tracksRegLiveness: true
+tracksDebugUserValues: true
+liveins:
+  - { reg: '$w0' }
+frameInfo:
+  maxAlignment:    1
+  adjustsStack:    true
+  hasCalls:        true
+  maxCallFrameSize: 0
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: noshrink_test3
+  ; CHECK: bb.0.BB00:
+  ; CHECK-NEXT:   successors: %bb.3(0x50000000), %bb.1(0x30000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   renamable $w19 = COPY $w0
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   dead $wzr = SUBSWri killed renamable $w19, 0, 0, implicit-def $nzcv
+  ; CHECK-NEXT:   Bcc 12, %bb.3, implicit killed $nzcv
+  ; CHECK-NEXT:   B %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.BB01:
+  ; CHECK-NEXT:   successors: %bb.2(0x00000800), %bb.6(0x7ffff800)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.6
+  ; CHECK-NEXT:   B %bb.2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.BB01.1:
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @abort, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.BB02:
+  ; CHECK-NEXT:   successors: %bb.4(0x30000000), %bb.5(0x50000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.5
+  ; CHECK-NEXT:   B %bb.4
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.4.BB03:
+  ; CHECK-NEXT:   successors: %bb.5(0x30000000), %bb.6(0x50000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   CBNZW killed renamable $w0, %bb.6
+  ; CHECK-NEXT:   B %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5.BB04:
+  ; CHECK-NEXT:   successors: %bb.6(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.6.exit:
+  ; CHECK-NEXT:   RET_ReallyLR
+  bb.0.BB00:
+    successors: %bb.3(0x50000000), %bb.1(0x30000000)
+    liveins: $w0
+
+    renamable $w19 = COPY $w0
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    dead $wzr = SUBSWri killed renamable $w19, 0, 0, implicit-def $nzcv
+    Bcc 12, %bb.3, implicit killed $nzcv
+    B %bb.1
+
+  bb.1.BB01:
+    successors: %bb.2(0x00000800), %bb.6(0x7ffff800)
+    liveins: $w0
+
+    CBNZW killed renamable $w0, %bb.6
+    B %bb.2
+
+  bb.2.BB01.1:
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @abort, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.3.BB02:
+    successors: %bb.4(0x30000000), %bb.5(0x50000000)
+    liveins: $w0
+
+    CBNZW killed renamable $w0, %bb.5
+    B %bb.4
+
+  bb.4.BB03:
+    successors: %bb.5(0x30000000), %bb.6(0x50000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+    CBNZW killed renamable $w0, %bb.6
+    B %bb.5
+
+  bb.5.BB04:
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @fun, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, implicit-def dead $w0
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.6.exit:
+    RET_ReallyLR
+
+...
+---
+name:            noshrink_bb_as_inlineasmbr_target
+registers:       []
+liveins:
+  - { reg: '$w0', virtual-reg: '' }
+frameInfo:
+  savePoint:       ''
+  restorePoint:    ''
+body:             |
+  ; CHECK-LABEL: name: noshrink_bb_as_inlineasmbr_target
+  ; CHECK: bb.0.entry:
+  ; CHECK-NEXT:   successors: %bb.1(0x40000000), %bb.3(0x40000000)
+  ; CHECK-NEXT:   liveins: $w0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   TBZW killed renamable $w0, 0, %bb.3
+  ; CHECK-NEXT:   B %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1 (%ir-block.0):
+  ; CHECK-NEXT:   successors: %bb.2(0x80000000), %bb.3(0x00000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   INLINEASM_BR &"", 1 /* sideeffect attdialect */, 13 /* imm */, %bb.3
+  ; CHECK-NEXT:   B %bb.2
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2 (%ir-block.1):
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT:   BL @dosomething, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.exit (machine-block-address-taken, inlineasm-br-indirect-target):
+  ; CHECK-NEXT:   RET_ReallyLR
+  bb.0.entry:
+    successors: %bb.1(0x40000000), %bb.3(0x40000000)
+    liveins: $w0
+
+    TBZW killed renamable $w0, 0, %bb.3
+    B %bb.1
+
+  bb.1 (%ir-block.0):
+    successors: %bb.2(0x80000000), %bb.3(0x00000000)
+
+    INLINEASM_BR &"", 1 /* sideeffect attdialect */, 13 /* imm */, %bb.3
+    B %bb.2
+
+  bb.2 (%ir-block.1):
+    successors: %bb.3(0x80000000)
+
+    ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp
+    BL @dosomething, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+    ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp
+
+  bb.3.exit (machine-block-address-taken, inlineasm-br-indirect-target):
+    RET_ReallyLR
+
+...

llvm/test/CodeGen/AArch64/taildup-cfi.ll

 entry.if.end_crit_edge:                           ; preds = %entry
   %.pre = load i32, ptr @f, align 4, !tbaa !2
   br label %if.end
 
 if.then:                                          ; preds = %entry
   store i32 0, ptr @f, align 4, !tbaa !2
   br label %if.end
 
-; DARWIN-NOT:           Merging into block
+; DARWIN:             Merging into block
 ; LINUX:            Merging into block
 
 if.end:                                           ; preds = %entry.if.end_crit_edge, %if.then
   %1 = phi i32 [ %.pre, %entry.if.end_crit_edge ], [ 0, %if.then ]
   %cmp6 = icmp slt i32 %1, %o
   br i1 %cmp6, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %if.end

llvm/test/CodeGen/ARM/ParallelDSP/multi-use-loads.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -O3 -mtriple=armv8m.main-none-none-eabi -mattr=+dsp < %s | FileCheck %s --check-prefixes=CHECK-LE
 ; RUN: llc -O3 -mtriple=armv8m.maineb-none-none-eabi -mattr=+dsp < %s | FileCheck %s --check-prefixes=CHECK-BE
 
 define i32 @add_user(i32 %arg, ptr nocapture readnone %arg1, ptr nocapture readonly %arg2, ptr nocapture readonly %arg3) {
 ; CHECK-LE-LABEL: add_user:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    .save {r4, lr}
-; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    cmp r0, #1
 ; CHECK-LE-NEXT:    blt .LBB0_4
 ; CHECK-LE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-LE-NEXT:    .save {r4, lr}
+; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    sub.w lr, r3, #2
 ; CHECK-LE-NEXT:    subs r2, #2
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:  .LBB0_2: @ %for.body
 ; CHECK-LE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-LE-NEXT:    ldr r3, [lr, #2]!
 ; CHECK-LE-NEXT:    subs r0, #1
 ; CHECK-LE-NEXT:    ldr r4, [r2, #2]!
 ; CHECK-LE-NEXT:    sxtah r1, r1, r3
 ; CHECK-LE-NEXT:    smlad r12, r4, r3, r12
 ; CHECK-LE-NEXT:    bne .LBB0_2
-; CHECK-LE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-LE-NEXT:  @ %bb.3:
+; CHECK-LE-NEXT:    pop.w {r4, lr}
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:  .LBB0_4:
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: add_user:
 ; CHECK-BE:       @ %bb.0: @ %entry
-; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    cmp r0, #1
 ; CHECK-BE-NEXT:    blt .LBB0_4
 ; CHECK-BE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    subs r3, #2
 ; CHECK-BE-NEXT:    subs r2, #2
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:  .LBB0_2: @ %for.body
 ; CHECK-BE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    ldrsh lr, [r3, #2]!
 ; CHECK-BE-NEXT:    subs r0, #1
 ; CHECK-BE-NEXT:    ldrsh r4, [r2, #2]!
 ; CHECK-BE-NEXT:    add r1, lr
 ; CHECK-BE-NEXT:    ldrsh.w r5, [r2, #2]
 ; CHECK-BE-NEXT:    smlabb r12, r4, lr, r12
 ; CHECK-BE-NEXT:    ldrsh.w r4, [r3, #2]
 ; CHECK-BE-NEXT:    smlabb r12, r5, r4, r12
 ; CHECK-BE-NEXT:    bne .LBB0_2
-; CHECK-BE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-BE-NEXT:  @ %bb.3:
+; CHECK-BE-NEXT:    pop.w {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:  .LBB0_4:
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 entry:
   %cmp24 = icmp sgt i32 %arg, 0
   br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:
   %.pre = load i16, ptr %arg3, align 2
   %.pre27 = load i16, ptr %arg2, align 2
   br label %for.body
for.body:
   %add11 = add i32 %mul9, %add10
   %exitcond = icmp ne i32 %add, %arg
   br i1 %exitcond, label %for.body, label %for.cond.cleanup
 }
 
 define i32 @mul_bottom_user(i32 %arg, ptr nocapture readnone %arg1, ptr nocapture readonly %arg2, ptr nocapture readonly %arg3) {
 ; CHECK-LE-LABEL: mul_bottom_user:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    .save {r4, lr}
-; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    cmp r0, #1
 ; CHECK-LE-NEXT:    blt .LBB1_4
 ; CHECK-LE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-LE-NEXT:    .save {r4, lr}
+; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    sub.w lr, r3, #2
 ; CHECK-LE-NEXT:    subs r2, #2
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:  .LBB1_2: @ %for.body
 ; CHECK-LE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-LE-NEXT:    ldr r3, [lr, #2]!
 ; CHECK-LE-NEXT:    subs r0, #1
 ; CHECK-LE-NEXT:    ldr r4, [r2, #2]!
 ; CHECK-LE-NEXT:    smlad r12, r4, r3, r12
 ; CHECK-LE-NEXT:    sxth r3, r3
 ; CHECK-LE-NEXT:    mul r1, r3, r1
 ; CHECK-LE-NEXT:    bne .LBB1_2
-; CHECK-LE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-LE-NEXT:  @ %bb.3:
+; CHECK-LE-NEXT:    pop.w {r4, lr}
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:  .LBB1_4:
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: mul_bottom_user:
 ; CHECK-BE:       @ %bb.0: @ %entry
-; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    cmp r0, #1
 ; CHECK-BE-NEXT:    blt .LBB1_4
 ; CHECK-BE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    subs r3, #2
 ; CHECK-BE-NEXT:    subs r2, #2
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:  .LBB1_2: @ %for.body
 ; CHECK-BE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    ldrsh lr, [r3, #2]!
 ; CHECK-BE-NEXT:    subs r0, #1
 ; CHECK-BE-NEXT:    ldrsh r4, [r2, #2]!
 ; CHECK-BE-NEXT:    ldrsh.w r5, [r2, #2]
 ; CHECK-BE-NEXT:    mul r1, lr, r1
 ; CHECK-BE-NEXT:    smlabb r12, r4, lr, r12
 ; CHECK-BE-NEXT:    ldrsh.w r4, [r3, #2]
 ; CHECK-BE-NEXT:    smlabb r12, r5, r4, r12
 ; CHECK-BE-NEXT:    bne .LBB1_2
-; CHECK-BE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-BE-NEXT:  @ %bb.3:
+; CHECK-BE-NEXT:    pop.w {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:  .LBB1_4:
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 entry:
   %cmp24 = icmp sgt i32 %arg, 0
   br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:
   %.pre = load i16, ptr %arg3, align 2
   %.pre27 = load i16, ptr %arg2, align 2
   br label %for.body
for.body:
   %count.next = mul i32 %conv4, %count
   %exitcond = icmp ne i32 %add, %arg
   br i1 %exitcond, label %for.body, label %for.cond.cleanup
 }
 
 define i32 @mul_top_user(i32 %arg, ptr nocapture readnone %arg1, ptr nocapture readonly %arg2, ptr nocapture readonly %arg3) {
 ; CHECK-LE-LABEL: mul_top_user:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    .save {r4, lr}
-; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    cmp r0, #1
 ; CHECK-LE-NEXT:    blt .LBB2_4
 ; CHECK-LE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-LE-NEXT:    .save {r4, lr}
+; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    subs r3, #2
 ; CHECK-LE-NEXT:    subs r2, #2
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:  .LBB2_2: @ %for.body
 ; CHECK-LE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-LE-NEXT:    ldr lr, [r3, #2]!
 ; CHECK-LE-NEXT:    subs r0, #1
 ; CHECK-LE-NEXT:    ldr r4, [r2, #2]!
 ; CHECK-LE-NEXT:    smlad r12, r4, lr, r12
 ; CHECK-LE-NEXT:    asr.w r4, r4, #16
 ; CHECK-LE-NEXT:    mul r1, r4, r1
 ; CHECK-LE-NEXT:    bne .LBB2_2
-; CHECK-LE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-LE-NEXT:  @ %bb.3:
+; CHECK-LE-NEXT:    pop.w {r4, lr}
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:  .LBB2_4:
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: mul_top_user:
 ; CHECK-BE:       @ %bb.0: @ %entry
-; CHECK-BE-NEXT:    .save {r4, lr}
-; CHECK-BE-NEXT:    push {r4, lr}
 ; CHECK-BE-NEXT:    cmp r0, #1
 ; CHECK-BE-NEXT:    blt .LBB2_4
 ; CHECK-BE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-BE-NEXT:    .save {r4, lr}
+; CHECK-BE-NEXT:    push {r4, lr}
 ; CHECK-BE-NEXT:    subs r3, #2
 ; CHECK-BE-NEXT:    subs r2, #2
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:  .LBB2_2: @ %for.body
 ; CHECK-BE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    ldrsh lr, [r3, #2]!
 ; CHECK-BE-NEXT:    subs r0, #1
 ; CHECK-BE-NEXT:    ldrsh r4, [r2, #2]!
 ; CHECK-BE-NEXT:    smlabb r12, r4, lr, r12
 ; CHECK-BE-NEXT:    ldrsh.w r4, [r2, #2]
 ; CHECK-BE-NEXT:    ldrsh.w lr, [r3, #2]
 ; CHECK-BE-NEXT:    mul r1, r4, r1
 ; CHECK-BE-NEXT:    smlabb r12, r4, lr, r12
 ; CHECK-BE-NEXT:    bne .LBB2_2
-; CHECK-BE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-BE-NEXT:  @ %bb.3:
+; CHECK-BE-NEXT:    pop.w {r4, lr}
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, pc}
+; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:  .LBB2_4:
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, pc}
+; CHECK-BE-NEXT:    bx lr
 entry:
   %cmp24 = icmp sgt i32 %arg, 0
   br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:
   %.pre = load i16, ptr %arg3, align 2
   %.pre27 = load i16, ptr %arg2, align 2
   br label %for.body
for.body:
   %count.next = mul i32 %conv7, %count
   %exitcond = icmp ne i32 %add, %arg
   br i1 %exitcond, label %for.body, label %for.cond.cleanup
 }
 
 define i32 @and_user(i32 %arg, ptr nocapture readnone %arg1, ptr nocapture readonly %arg2, ptr nocapture readonly %arg3) {
 ; CHECK-LE-LABEL: and_user:
 ; CHECK-LE:       @ %bb.0: @ %entry
-; CHECK-LE-NEXT:    .save {r4, lr}
-; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    cmp r0, #1
 ; CHECK-LE-NEXT:    blt .LBB3_4
 ; CHECK-LE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-LE-NEXT:    .save {r4, lr}
+; CHECK-LE-NEXT:    push {r4, lr}
 ; CHECK-LE-NEXT:    sub.w lr, r3, #2
 ; CHECK-LE-NEXT:    subs r2, #2
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:  .LBB3_2: @ %for.body
 ; CHECK-LE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-LE-NEXT:    ldr r3, [lr, #2]!
 ; CHECK-LE-NEXT:    subs r0, #1
 ; CHECK-LE-NEXT:    ldr r4, [r2, #2]!
 ; CHECK-LE-NEXT:    smlad r12, r4, r3, r12
 ; CHECK-LE-NEXT:    uxth r3, r3
 ; CHECK-LE-NEXT:    mul r1, r3, r1
 ; CHECK-LE-NEXT:    bne .LBB3_2
-; CHECK-LE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-LE-NEXT:  @ %bb.3:
+; CHECK-LE-NEXT:    pop.w {r4, lr}
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ; CHECK-LE-NEXT:  .LBB3_4:
 ; CHECK-LE-NEXT:    mov.w r12, #0
 ; CHECK-LE-NEXT:    movs r1, #0
 ; CHECK-LE-NEXT:    add.w r0, r12, r1
-; CHECK-LE-NEXT:    pop {r4, pc}
+; CHECK-LE-NEXT:    bx lr
 ;
 ; CHECK-BE-LABEL: and_user:
 ; CHECK-BE:       @ %bb.0: @ %entry
-; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    cmp r0, #1
 ; CHECK-BE-NEXT:    blt .LBB3_4
 ; CHECK-BE-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-BE-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-BE-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    subs r3, #2
 ; CHECK-BE-NEXT:    subs r2, #2
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:  .LBB3_2: @ %for.body
 ; CHECK-BE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    ldrh lr, [r3, #2]!
 ; CHECK-BE-NEXT:    subs r0, #1
 ; CHECK-BE-NEXT:    ldrsh r4, [r2, #2]!
 ; CHECK-BE-NEXT:    ldrsh.w r5, [r2, #2]
 ; CHECK-BE-NEXT:    mul r1, lr, r1
 ; CHECK-BE-NEXT:    smlabb r12, r4, lr, r12
 ; CHECK-BE-NEXT:    ldrsh.w r4, [r3, #2]
 ; CHECK-BE-NEXT:    smlabb r12, r5, r4, r12
 ; CHECK-BE-NEXT:    bne .LBB3_2
-; CHECK-BE-NEXT:  @ %bb.3: @ %for.cond.cleanup
+; CHECK-BE-NEXT:  @ %bb.3:
+; CHECK-BE-NEXT:    pop.w {r4, r5, r7, lr}
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 ; CHECK-BE-NEXT:  .LBB3_4:
 ; CHECK-BE-NEXT:    mov.w r12, #0
 ; CHECK-BE-NEXT:    movs r1, #0
 ; CHECK-BE-NEXT:    add.w r0, r12, r1
-; CHECK-BE-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-BE-NEXT:    bx lr
 entry:
   %cmp24 = icmp sgt i32 %arg, 0
   br i1 %cmp24, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:
   %.pre = load i16, ptr %arg3, align 2
   %.pre27 = load i16, ptr %arg2, align 2
   br label %for.body

llvm/test/CodeGen/ARM/code-placement.ll

 ; RUN: llc < %s -mtriple=armv7-apple-darwin | FileCheck %s
 ; PHI elimination shouldn't break backedge.
 
 %struct.list_data_s = type { i16, i16 }
 %struct.list_head = type { %struct.list_head*, %struct.list_data_s* }
 
 define arm_apcscc %struct.list_head* @t1(%struct.list_head* %list) nounwind {
 entry:
 ; CHECK-LABEL: t1:
   %0 = icmp eq %struct.list_head* %list, null
   br i1 %0, label %bb2, label %bb
 
 bb:
-; CHECK: LBB0_1:
 ; CHECK: LBB0_[[LABEL:[0-9]]]:
 ; CHECK: bne LBB0_[[LABEL]]
 ; CHECK-NOT: b LBB0_[[LABEL]]
 ; CHECK: bx lr
   %list_addr.05 = phi %struct.list_head* [ %2, %bb ], [ %list, %entry ]
   %next.04 = phi %struct.list_head* [ %list_addr.05, %bb ], [ null, %entry ]
   %1 = getelementptr inbounds %struct.list_head, %struct.list_head* %list_addr.05, i32 0, i32 0
   %2 = load %struct.list_head*, %struct.list_head** %1, align 4

llvm/test/CodeGen/ARM/mbp.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc < %s | FileCheck %s
 target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
 target triple = "thumbv7-unknown-linux-gnueabihf"
 
 %Target = type { ptr, ptr }
 %List = type { i32, ptr }
 
 ; The entry block should be the first block of the function.
-; CHECK-LABEL: foo
-; CHECK:       %entry
-; CHECK:       %for.body
-; CHECK:       %for.inc
-; CHECK:       %if.then
-; CHECK:       %for.cond.i
-; CHECK:       %for.body.i
-; CHECK:       %return
 
 define i1 @foo(ptr %ha, i32 %he) !prof !39 {
+; CHECK-LABEL: foo:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    ldr r2, [r0]
+; CHECK-NEXT:    cmp r2, #0
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB0_1: @ %for.body.preheader
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
+; CHECK-NEXT:    b .LBB0_3
+; CHECK-NEXT:  .LBB0_2: @ %for.inc
+; CHECK-NEXT:    @ in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT:    ldr r2, [r2]
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    cmp r2, #0
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    popeq {r7, pc}
+; CHECK-NEXT:  .LBB0_3: @ %for.body
+; CHECK-NEXT:    @ =>This Loop Header: Depth=1
+; CHECK-NEXT:    @ Child Loop BB0_5 Depth 2
+; CHECK-NEXT:    ldr r0, [r2, #4]
+; CHECK-NEXT:    cmp r0, #0
+; CHECK-NEXT:    beq .LBB0_2
+; CHECK-NEXT:  @ %bb.4: @ %if.then
+; CHECK-NEXT:    @ in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT:    ldrd r3, r0, [r0]
+; CHECK-NEXT:    sub.w r12, r0, #4
+; CHECK-NEXT:  .LBB0_5: @ %for.cond.i
+; CHECK-NEXT:    @ Parent Loop BB0_3 Depth=1
+; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
+; CHECK-NEXT:    cmp r3, #1
+; CHECK-NEXT:    blt .LBB0_2
+; CHECK-NEXT:  @ %bb.6: @ %for.body.i
+; CHECK-NEXT:    @ in Loop: Header=BB0_5 Depth=2
+; CHECK-NEXT:    ldr.w lr, [r12, r3, lsl #2]
+; CHECK-NEXT:    subs r3, #1
+; CHECK-NEXT:    movs r0, #1
+; CHECK-NEXT:    cmp lr, r1
+; CHECK-NEXT:    bne .LBB0_5
+; CHECK-NEXT:  @ %bb.7:
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %TargetPtr = load ptr, ptr %ha, align 4
   %cmp1 = icmp eq ptr %TargetPtr, null
   br i1 %cmp1, label %return, label %for.body, !prof !50
 
 for.body:
   %TargetPhi = phi ptr [ %NextPtr, %for.inc ], [ %TargetPtr, %entry ]
   %ListAddr = getelementptr inbounds %Target, ptr %TargetPhi, i32 0, i32 1

llvm/test/CodeGen/ARM/ssat-unroll-loops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=armv6t2-eabi %s -o - | FileCheck %s --check-prefix=CHECK
 
 ; Checks SSAT is still generated when loop unrolling is on
 
 define void @ssat_unroll(ptr %pSrcA, ptr %pSrcB, ptr %pDst, i32 %blockSize) {
 ; CHECK-LABEL: ssat_unroll:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #0
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB0_1: @ %while.body.preheader
 ; CHECK-NEXT:    .save {r11, lr}
 ; CHECK-NEXT:    push {r11, lr}
-; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB0_5
-; CHECK-NEXT:  @ %bb.1: @ %while.body.preheader
 ; CHECK-NEXT:    sub r12, r3, #1
 ; CHECK-NEXT:    tst r3, #1
 ; CHECK-NEXT:    beq .LBB0_3
 ; CHECK-NEXT:  @ %bb.2: @ %while.body.prol.preheader
 ; CHECK-NEXT:    ldrsh lr, [r0], #2
 ; CHECK-NEXT:    ldrsh r3, [r1], #2
 ; CHECK-NEXT:    smulbb r3, r3, lr
 ; CHECK-NEXT:    ssat r3, #16, r3, asr #14
 ; CHECK-NEXT:    strh r3, [r2], #2
 ; CHECK-NEXT:    mov r3, r12
 ; CHECK-NEXT:  .LBB0_3: @ %while.body.prol.loopexit
 ; CHECK-NEXT:    cmp r12, #0
-; CHECK-NEXT:    popeq {r11, pc}
+; CHECK-NEXT:    beq .LBB0_5
 ; CHECK-NEXT:  .LBB0_4: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrsh r12, [r0]
 ; CHECK-NEXT:    subs r3, r3, #2
 ; CHECK-NEXT:    ldrsh lr, [r1]
 ; CHECK-NEXT:    smulbb r12, lr, r12
 ; CHECK-NEXT:    ssat r12, #16, r12, asr #14
 ; CHECK-NEXT:    strh r12, [r2]
 ; CHECK-NEXT:    ldrsh r12, [r0, #2]
 ; CHECK-NEXT:    add r0, r0, #4
 ; CHECK-NEXT:    ldrsh lr, [r1, #2]
 ; CHECK-NEXT:    add r1, r1, #4
 ; CHECK-NEXT:    smulbb r12, lr, r12
 ; CHECK-NEXT:    ssat r12, #16, r12, asr #14
 ; CHECK-NEXT:    strh r12, [r2, #2]
 ; CHECK-NEXT:    add r2, r2, #4
 ; CHECK-NEXT:    bne .LBB0_4
-; CHECK-NEXT:  .LBB0_5: @ %while.end
-; CHECK-NEXT:    pop {r11, pc}
+; CHECK-NEXT:  .LBB0_5:
+; CHECK-NEXT:    pop {r11, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp.not7 = icmp eq i32 %blockSize, 0
   br i1 %cmp.not7, label %while.end, label %while.body.preheader
 
 while.body.preheader:                             ; preds = %entry
   %0 = add i32 %blockSize, -1
   %xtraiter = and i32 %blockSize, 1
   %lcmp.mod.not = icmp eq i32 %xtraiter, 0

 
 while.end:                                        ; preds = %while.body, %while.body.prol.loopexit, %entry
   ret void
 }
 
 define void @ssat_unroll_minmax(ptr nocapture readonly %pSrcA, ptr nocapture readonly %pSrcB, ptr nocapture writeonly %pDst, i32 %blockSize) {
 ; CHECK-LABEL: ssat_unroll_minmax:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #0
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB1_1: @ %while.body.preheader
 ; CHECK-NEXT:    .save {r11, lr}
 ; CHECK-NEXT:    push {r11, lr}
-; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB1_5
-; CHECK-NEXT:  @ %bb.1: @ %while.body.preheader
 ; CHECK-NEXT:    sub r12, r3, #1
 ; CHECK-NEXT:    tst r3, #1
 ; CHECK-NEXT:    beq .LBB1_3
 ; CHECK-NEXT:  @ %bb.2: @ %while.body.prol.preheader
 ; CHECK-NEXT:    ldrsh lr, [r0], #2
 ; CHECK-NEXT:    ldrsh r3, [r1], #2
 ; CHECK-NEXT:    smulbb r3, r3, lr
 ; CHECK-NEXT:    ssat r3, #16, r3, asr #14
 ; CHECK-NEXT:    strh r3, [r2], #2
 ; CHECK-NEXT:    mov r3, r12
 ; CHECK-NEXT:  .LBB1_3: @ %while.body.prol.loopexit
 ; CHECK-NEXT:    cmp r12, #0
-; CHECK-NEXT:    popeq {r11, pc}
+; CHECK-NEXT:    beq .LBB1_5
 ; CHECK-NEXT:  .LBB1_4: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrsh r12, [r0]
 ; CHECK-NEXT:    subs r3, r3, #2
 ; CHECK-NEXT:    ldrsh lr, [r1]
 ; CHECK-NEXT:    smulbb r12, lr, r12
 ; CHECK-NEXT:    ssat r12, #16, r12, asr #14
 ; CHECK-NEXT:    strh r12, [r2]
 ; CHECK-NEXT:    ldrsh r12, [r0, #2]
 ; CHECK-NEXT:    add r0, r0, #4
 ; CHECK-NEXT:    ldrsh lr, [r1, #2]
 ; CHECK-NEXT:    add r1, r1, #4
 ; CHECK-NEXT:    smulbb r12, lr, r12
 ; CHECK-NEXT:    ssat r12, #16, r12, asr #14
 ; CHECK-NEXT:    strh r12, [r2, #2]
 ; CHECK-NEXT:    add r2, r2, #4
 ; CHECK-NEXT:    bne .LBB1_4
-; CHECK-NEXT:  .LBB1_5: @ %while.end
-; CHECK-NEXT:    pop {r11, pc}
+; CHECK-NEXT:  .LBB1_5:
+; CHECK-NEXT:    pop {r11, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp.not7 = icmp eq i32 %blockSize, 0
   br i1 %cmp.not7, label %while.end, label %while.body.preheader
 
 while.body.preheader:                             ; preds = %entry
   %0 = add i32 %blockSize, -1
   %xtraiter = and i32 %blockSize, 1
   %lcmp.mod.not = icmp eq i32 %xtraiter, 0

llvm/test/CodeGen/PowerPC/common-chain-aix32.ll

 ;   return sum;
 ; }
 ;
 define i64 @two_chain_same_offset_succ_i32(ptr %p, i32 %offset, i32 %base1, i64 %n) {
 ; CHECK-LABEL: two_chain_same_offset_succ_i32:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    cmplwi r6, 0
 ; CHECK-NEXT:    cmpwi cr1, r6, 0
-; CHECK-NEXT:    stw r30, -8(r1) # 4-byte Folded Spill
-; CHECK-NEXT:    stw r31, -4(r1) # 4-byte Folded Spill
 ; CHECK-NEXT:    crandc 4*cr5+lt, 4*cr1+lt, eq
 ; CHECK-NEXT:    cmpwi cr1, r7, 0
-; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_5
+; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_6
 ; CHECK-NEXT:  # %bb.1: # %entry
 ; CHECK-NEXT:    crand 4*cr5+lt, eq, 4*cr1+eq
-; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_5
+; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_6
 ; CHECK-NEXT:  # %bb.2: # %for.body.preheader
 ; CHECK-NEXT:    slwi r8, r4, 1
 ; CHECK-NEXT:    li r10, 0
 ; CHECK-NEXT:    li r11, 0
+; CHECK-NEXT:    stw r30, -8(r1) # 4-byte Folded Spill
 ; CHECK-NEXT:    add r8, r4, r8
+; CHECK-NEXT:    stw r31, -4(r1) # 4-byte Folded Spill
 ; CHECK-NEXT:    add r9, r5, r8
 ; CHECK-NEXT:    add r5, r5, r4
 ; CHECK-NEXT:    add r8, r3, r5
 ; CHECK-NEXT:    add r9, r3, r9
 ; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    li r5, 0
 ; CHECK-NEXT:    .align 4
 ; CHECK-NEXT:  L..BB0_3: # %for.body

 ; CHECK-NEXT:    cmpw cr1, r10, r6
 ; CHECK-NEXT:    crandc 4*cr5+lt, 4*cr1+lt, eq
 ; CHECK-NEXT:    cmplw cr1, r11, r7
 ; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_3
 ; CHECK-NEXT:  # %bb.4: # %for.body
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    crand 4*cr5+lt, eq, 4*cr1+lt
 ; CHECK-NEXT:    bc 12, 4*cr5+lt, L..BB0_3
-; CHECK-NEXT:    b L..BB0_6
-; CHECK-NEXT:  L..BB0_5:
-; CHECK-NEXT:    li r3, 0
-; CHECK-NEXT:    li r5, 0
-; CHECK-NEXT:  L..BB0_6: # %for.cond.cleanup
+; CHECK-NEXT:  # %bb.5:
 ; CHECK-NEXT:    lwz r31, -4(r1) # 4-byte Folded Reload
 ; CHECK-NEXT:    lwz r30, -8(r1) # 4-byte Folded Reload
 ; CHECK-NEXT:    mr r4, r5
 ; CHECK-NEXT:    blr
+; CHECK-NEXT:  L..BB0_6:
+; CHECK-NEXT:    li r3, 0
+; CHECK-NEXT:    li r4, 0
+; CHECK-NEXT:    blr
 entry:
   %add = add nsw i32 %base1, %offset
   %mul = shl nsw i32 %offset, 1
   %add1 = add nsw i32 %mul, %base1
   %mul2 = mul nsw i32 %offset, 3
   %add3 = add nsw i32 %mul2, %base1
   %mul4 = shl nsw i32 %offset, 2
   %add5 = add nsw i32 %mul4, %base1

llvm/test/CodeGen/PowerPC/common-chain.ll

 ;   }
 ;   return sum;
 ; }
 ;
 define i64 @not_perfect_chain_all_same_offset_fail(ptr %p, i64 %offset, i64 %base1, i64 %n) {
 ; CHECK-LABEL: not_perfect_chain_all_same_offset_fail:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    cmpdi r6, 0
-; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    ble cr0, .LBB1_4
 ; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    sldi r7, r4, 1
-; CHECK-NEXT:    sldi r9, r4, 2
 ; CHECK-NEXT:    add r5, r3, r5
 ; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    add r8, r4, r7
+; CHECK-NEXT:    sldi r9, r4, 2
 ; CHECK-NEXT:    mtctr r6
 ; CHECK-NEXT:    add r10, r4, r9
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB1_2: # %for.body
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    ldx r6, r5, r4
 ; CHECK-NEXT:    ldx r11, r5, r7
 ; CHECK-NEXT:    ldx r12, r5, r8
 ; CHECK-NEXT:    ldx r0, r5, r9
 ; CHECK-NEXT:    mulld r6, r11, r6
 ; CHECK-NEXT:    ldx r30, r5, r10
 ; CHECK-NEXT:    addi r5, r5, 1
 ; CHECK-NEXT:    mulld r6, r6, r12
 ; CHECK-NEXT:    mulld r6, r6, r0
 ; CHECK-NEXT:    maddld r3, r6, r30, r3
 ; CHECK-NEXT:    bdnz .LBB1_2
-; CHECK-NEXT:  # %bb.3: # %for.cond.cleanup
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    blr
 ; CHECK-NEXT:  .LBB1_4:
 ; CHECK-NEXT:    li r3, 0
-; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    blr
 entry:
   %mul = shl nsw i64 %offset, 1
   %mul2 = mul nsw i64 %offset, 3
   %mul4 = shl nsw i64 %offset, 2
   %mul6 = mul nsw i64 %offset, 5
   %cmp58 = icmp sgt i64 %n, 0
   br i1 %cmp58, label %for.body, label %for.cond.cleanup

 ;   }
 ;   return sum;
 ; }
 ;
 define i64 @not_same_offset_fail(ptr %p, i64 %offset, i64 %base1, i64 %n) {
 ; CHECK-LABEL: not_same_offset_fail:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    cmpdi r6, 0
+; CHECK-NEXT:    ble cr0, .LBB4_4
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
 ; CHECK-NEXT:    std r28, -32(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std r29, -24(r1) # 8-byte Folded Spill
+; CHECK-NEXT:    add r5, r3, r5
+; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    ble cr0, .LBB4_3
-; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    mtctr r6
 ; CHECK-NEXT:    mulli r11, r4, 10
 ; CHECK-NEXT:    sldi r8, r4, 2
-; CHECK-NEXT:    add r5, r3, r5
-; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    add r8, r4, r8
 ; CHECK-NEXT:    sldi r9, r4, 3
-; CHECK-NEXT:    mtctr r6
-; CHECK-NEXT:    sldi r7, r4, 1
 ; CHECK-NEXT:    sub r10, r9, r4
+; CHECK-NEXT:    sldi r7, r4, 1
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB4_2: # %for.body
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    ldx r6, r5, r4
 ; CHECK-NEXT:    ldx r12, r5, r7
 ; CHECK-NEXT:    ldx r0, r5, r8
 ; CHECK-NEXT:    ldx r30, r5, r10
 ; CHECK-NEXT:    mulld r6, r12, r6
 ; CHECK-NEXT:    ldx r29, r5, r9
 ; CHECK-NEXT:    ldx r28, r5, r11
 ; CHECK-NEXT:    addi r5, r5, 1
 ; CHECK-NEXT:    mulld r6, r6, r0
 ; CHECK-NEXT:    mulld r6, r6, r30
 ; CHECK-NEXT:    mulld r6, r6, r29
 ; CHECK-NEXT:    maddld r3, r6, r28, r3
 ; CHECK-NEXT:    bdnz .LBB4_2
-; CHECK-NEXT:    b .LBB4_4
-; CHECK-NEXT:  .LBB4_3:
-; CHECK-NEXT:    li r3, 0
-; CHECK-NEXT:  .LBB4_4: # %for.cond.cleanup
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r29, -24(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r28, -32(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    blr
+; CHECK-NEXT:  .LBB4_4:
+; CHECK-NEXT:    li r3, 0
+; CHECK-NEXT:    blr
 entry:
   %mul = shl nsw i64 %offset, 1
   %mul2 = mul nsw i64 %offset, 5
   %mul4 = mul nsw i64 %offset, 7
   %mul6 = shl nsw i64 %offset, 3
   %mul8 = mul nsw i64 %offset, 10
   %cmp70 = icmp sgt i64 %n, 0
   br i1 %cmp70, label %for.body, label %for.cond.cleanup

llvm/test/CodeGen/PowerPC/loop-instr-form-prepare.ll

 ;  }
 ;  return res + count;
 ;}
 
 define i64 @test_max_number_reminder(ptr %arg, i32 signext %arg1) {
 ; CHECK-LABEL: test_max_number_reminder:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    cmplwi r4, 0
-; CHECK-NEXT:    std r25, -56(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r26, -48(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r27, -40(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r28, -32(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r29, -24(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    beq cr0, .LBB2_3
+; CHECK-NEXT:    beq cr0, .LBB2_4
 ; CHECK-NEXT:  # %bb.1: # %bb3.preheader
 ; CHECK-NEXT:    cmpldi r4, 1
 ; CHECK-NEXT:    li r5, 1
 ; CHECK-NEXT:    addi r9, r3, 4002
+; CHECK-NEXT:    std r25, -56(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    li r6, -1
+; CHECK-NEXT:    std r26, -48(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    li r7, 3
 ; CHECK-NEXT:    li r8, 5
 ; CHECK-NEXT:    li r10, 9
+; CHECK-NEXT:    std r27, -40(r1) # 8-byte Folded Spill
+; CHECK-NEXT:    std r28, -32(r1) # 8-byte Folded Spill
+; CHECK-NEXT:    std r29, -24(r1) # 8-byte Folded Spill
+; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    iselgt r3, r4, r5
 ; CHECK-NEXT:    mtctr r3
 ; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB2_2: # %bb3
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    ldx r11, r9, r6
 ; CHECK-NEXT:    ld r12, 0(r9)

 ; CHECK-NEXT:    mulld r11, r11, r0
 ; CHECK-NEXT:    mulld r11, r11, r30
 ; CHECK-NEXT:    mulld r11, r11, r29
 ; CHECK-NEXT:    mulld r11, r11, r28
 ; CHECK-NEXT:    mulld r11, r11, r27
 ; CHECK-NEXT:    mulld r11, r11, r26
 ; CHECK-NEXT:    maddld r3, r11, r25, r3
 ; CHECK-NEXT:    bdnz .LBB2_2
-; CHECK-NEXT:    b .LBB2_4
-; CHECK-NEXT:  .LBB2_3:
-; CHECK-NEXT:    li r3, 0
-; CHECK-NEXT:  .LBB2_4: # %bb45
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r29, -24(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r28, -32(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r27, -40(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    add r3, r3, r4
 ; CHECK-NEXT:    ld r26, -48(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r25, -56(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    blr
+; CHECK-NEXT:  .LBB2_4:
+; CHECK-NEXT:    addi r3, r4, 0
+; CHECK-NEXT:    blr
 bb:
   %i = sext i32 %arg1 to i64
   %i2 = icmp eq i32 %arg1, 0
   br i1 %i2, label %bb45, label %bb3
 
 bb3:                                              ; preds = %bb3, %bb
   %i4 = phi i64 [ %i43, %bb3 ], [ 0, %bb ]
   %i5 = phi i64 [ %i42, %bb3 ], [ 0, %bb ]

 ;   }
 ;   return res + count;
 ; }
 
 define dso_local i64 @test_ds_multiple_chains(ptr %arg, ptr %arg1, i32 signext %arg2) {
 ; CHECK-LABEL: test_ds_multiple_chains:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    cmplwi r5, 0
-; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    beq cr0, .LBB5_3
+; CHECK-NEXT:    beq cr0, .LBB5_4
 ; CHECK-NEXT:  # %bb.1: # %bb4.preheader
 ; CHECK-NEXT:    cmpldi r5, 1
 ; CHECK-NEXT:    li r6, 1
+; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    addi r3, r3, 4001
 ; CHECK-NEXT:    addi r4, r4, 4001
 ; CHECK-NEXT:    li r7, 9
 ; CHECK-NEXT:    iselgt r6, r5, r6
 ; CHECK-NEXT:    mtctr r6
 ; CHECK-NEXT:    li r6, 0
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB5_2: # %bb4

 ; CHECK-NEXT:    addi r4, r4, 1
 ; CHECK-NEXT:    mulld r8, r8, r10
 ; CHECK-NEXT:    mulld r8, r8, r11
 ; CHECK-NEXT:    mulld r8, r8, r12
 ; CHECK-NEXT:    mulld r8, r8, r0
 ; CHECK-NEXT:    mulld r8, r8, r30
 ; CHECK-NEXT:    maddld r6, r8, r9, r6
 ; CHECK-NEXT:    bdnz .LBB5_2
-; CHECK-NEXT:    b .LBB5_4
-; CHECK-NEXT:  .LBB5_3:
-; CHECK-NEXT:    li r6, 0
-; CHECK-NEXT:  .LBB5_4: # %bb43
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    add r3, r6, r5
 ; CHECK-NEXT:    blr
+; CHECK-NEXT:  .LBB5_4:
+; CHECK-NEXT:    addi r3, r5, 0
+; CHECK-NEXT:    blr
 bb:
   %i = sext i32 %arg2 to i64
   %i3 = icmp eq i32 %arg2, 0
   br i1 %i3, label %bb43, label %bb4
 
 bb4:                                              ; preds = %bb4, %bb
   %i5 = phi i64 [ %i41, %bb4 ], [ 0, %bb ]
   %i6 = phi i64 [ %i40, %bb4 ], [ 0, %bb ]

 ;}
 
 @arr = external local_unnamed_addr global ptr, align 8
 
 define i64 @test_ds_cross_basic_blocks(ptr %arg, i32 signext %arg1) {
 ; CHECK-LABEL: test_ds_cross_basic_blocks:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    cmplwi r4, 0
-; CHECK-NEXT:    std r28, -32(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r29, -24(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
-; CHECK-NEXT:    beq cr0, .LBB6_8
+; CHECK-NEXT:    beq cr0, .LBB6_9
 ; CHECK-NEXT:  # %bb.1: # %bb3
 ; CHECK-NEXT:    addis r5, r2, .LC0@toc@ha
 ; CHECK-NEXT:    cmpldi r4, 1
 ; CHECK-NEXT:    li r7, 1
 ; CHECK-NEXT:    addi r6, r3, 4009
+; CHECK-NEXT:    std r28, -32(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    ld r5, .LC0@toc@l(r5)
 ; CHECK-NEXT:    iselgt r3, r4, r7
+; CHECK-NEXT:    std r29, -24(r1) # 8-byte Folded Spill
+; CHECK-NEXT:    std r30, -16(r1) # 8-byte Folded Spill
 ; CHECK-NEXT:    li r4, -7
 ; CHECK-NEXT:    li r8, -6
 ; CHECK-NEXT:    li r9, 1
 ; CHECK-NEXT:    li r10, 1
 ; CHECK-NEXT:    li r11, 1
 ; CHECK-NEXT:    li r12, 1
 ; CHECK-NEXT:    li r30, 1
 ; CHECK-NEXT:    ld r5, 0(r5)

 ; CHECK-NEXT:  .LBB6_3: # %bb49
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    mulld r0, r12, r30
 ; CHECK-NEXT:    addi r6, r6, 1
 ; CHECK-NEXT:    mulld r0, r0, r11
 ; CHECK-NEXT:    mulld r0, r0, r10
 ; CHECK-NEXT:    mulld r0, r0, r9
 ; CHECK-NEXT:    maddld r3, r0, r7, r3
-; CHECK-NEXT:    bdz .LBB6_9
+; CHECK-NEXT:    bdz .LBB6_8
 ; CHECK-NEXT:  .LBB6_4: # %bb5
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    lbzu r0, 1(r5)
 ; CHECK-NEXT:    mulli r29, r0, 171
 ; CHECK-NEXT:    rlwinm r28, r29, 24, 8, 30
 ; CHECK-NEXT:    srwi r29, r29, 9
 ; CHECK-NEXT:    add r29, r29, r28
 ; CHECK-NEXT:    sub r0, r0, r29

 ; CHECK-NEXT:  .LBB6_7: # %bb40
 ; CHECK-NEXT:    #
 ; CHECK-NEXT:    ldx r0, r6, r8
 ; CHECK-NEXT:    add r9, r0, r9
 ; CHECK-NEXT:    ld r0, 0(r6)
 ; CHECK-NEXT:    add r7, r0, r7
 ; CHECK-NEXT:    b .LBB6_3
 ; CHECK-NEXT:  .LBB6_8:
-; CHECK-NEXT:    li r3, 0
-; CHECK-NEXT:  .LBB6_9: # %bb64
 ; CHECK-NEXT:    ld r30, -16(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r29, -24(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    ld r28, -32(r1) # 8-byte Folded Reload
 ; CHECK-NEXT:    blr
+; CHECK-NEXT:  .LBB6_9:
+; CHECK-NEXT:    li r3, 0
+; CHECK-NEXT:    blr
 bb:
   %i = sext i32 %arg1 to i64
   %i2 = icmp eq i32 %arg1, 0
   br i1 %i2, label %bb64, label %bb3
 
 bb3:                                              ; preds = %bb
   %i4 = load ptr, ptr @arr, align 8
   br label %bb5

llvm/test/CodeGen/PowerPC/lsr-profitable-chain.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
 ; RUN:   -mcpu=pwr9 < %s | FileCheck %s
 
 define void @foo(ptr readonly %0, ptr %1, i64 %2, i64 %3, i64 %4, i64 %5, i64 %6, i64 %7) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    cmpd 5, 7
-; CHECK-NEXT:    std 22, -80(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 23, -72(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 24, -64(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 25, -56(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 26, -48(1) # 8-byte Folded Spill
+; CHECK-NEXT:    bgelr 0
+; CHECK-NEXT:  # %bb.1: # %.preheader
 ; CHECK-NEXT:    std 27, -40(1) # 8-byte Folded Spill
+; CHECK-NEXT:    addi 27, 5, 2
 ; CHECK-NEXT:    std 28, -32(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 29, -24(1) # 8-byte Folded Spill
+; CHECK-NEXT:    addi 28, 5, 3
 ; CHECK-NEXT:    std 30, -16(1) # 8-byte Folded Spill
-; CHECK-NEXT:    bge 0, .LBB0_6
-; CHECK-NEXT:  # %bb.1: # %.preheader
 ; CHECK-NEXT:    addi 30, 5, 1
-; CHECK-NEXT:    addi 28, 5, 3
-; CHECK-NEXT:    addi 27, 5, 2
 ; CHECK-NEXT:    mulld 12, 8, 5
-; CHECK-NEXT:    addi 29, 3, 16
 ; CHECK-NEXT:    mulld 0, 9, 8
+; CHECK-NEXT:    std 29, -24(1) # 8-byte Folded Spill
+; CHECK-NEXT:    addi 29, 3, 16
 ; CHECK-NEXT:    sldi 11, 10, 3
+; CHECK-NEXT:    std 22, -80(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 23, -72(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 24, -64(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 25, -56(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 26, -48(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    mulld 30, 8, 30
 ; CHECK-NEXT:    mulld 28, 8, 28
 ; CHECK-NEXT:    mulld 8, 8, 27
 ; CHECK-NEXT:    b .LBB0_3
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB0_2:
 ; CHECK-NEXT:    add 5, 5, 9
 ; CHECK-NEXT:    add 12, 12, 0

llvm/test/CodeGen/PowerPC/shrink-wrap.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64le-unknown-unknown -mcpu=pwr9 | FileCheck  %s --check-prefixes=POWERPC64
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc-ibm-aix-xcoff -mcpu=pwr9 -mattr=-altivec | FileCheck  %s --check-prefixes=POWERPC32-AIX
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64-ibm-aix-xcoff -mcpu=pwr9 -mattr=-altivec | FileCheck  %s --check-prefixes=POWERPC64-AIX
 
 define signext i32 @shrinkwrapme(i32 signext %a, i32 signext %lim) {
 ; POWERPC64-LABEL: shrinkwrapme:
 ; POWERPC64:       # %bb.0: # %entry
 ; POWERPC64-NEXT:    cmpwi 4, 0
+; POWERPC64-NEXT:    ble 0, .LBB0_4
+; POWERPC64-NEXT:  # %bb.1: # %for.body.preheader
+; POWERPC64-NEXT:    addi 4, 4, -1
 ; POWERPC64-NEXT:    std 14, -144(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 15, -136(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 16, -128(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 17, -120(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 18, -112(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 19, -104(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 20, -96(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 21, -88(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 22, -80(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 23, -72(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 24, -64(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 25, -56(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 26, -48(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 27, -40(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 28, -32(1) # 8-byte Folded Spill
+; POWERPC64-NEXT:    clrldi 4, 4, 32
+; POWERPC64-NEXT:    addi 4, 4, 1
 ; POWERPC64-NEXT:    std 29, -24(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 30, -16(1) # 8-byte Folded Spill
 ; POWERPC64-NEXT:    std 31, -8(1) # 8-byte Folded Spill
-; POWERPC64-NEXT:    ble 0, .LBB0_3
-; POWERPC64-NEXT:  # %bb.1: # %for.body.preheader
-; POWERPC64-NEXT:    addi 4, 4, -1
-; POWERPC64-NEXT:    clrldi 4, 4, 32
-; POWERPC64-NEXT:    addi 4, 4, 1
 ; POWERPC64-NEXT:    mtctr 4
 ; POWERPC64-NEXT:    li 4, 0
 ; POWERPC64-NEXT:    .p2align 4
 ; POWERPC64-NEXT:  .LBB0_2: # %for.body
 ; POWERPC64-NEXT:    #
 ; POWERPC64-NEXT:    #APP
 ; POWERPC64-NEXT:    add 4, 3, 4
 ; POWERPC64-NEXT:    #NO_APP
 ; POWERPC64-NEXT:    bdnz .LBB0_2
-; POWERPC64-NEXT:    b .LBB0_4
-; POWERPC64-NEXT:  .LBB0_3:
-; POWERPC64-NEXT:    li 4, 0
-; POWERPC64-NEXT:  .LBB0_4: # %for.cond.cleanup
+; POWERPC64-NEXT:  # %bb.3:
 ; POWERPC64-NEXT:    ld 31, -8(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 30, -16(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 29, -24(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 28, -32(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    extsw 3, 4
 ; POWERPC64-NEXT:    ld 27, -40(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 26, -48(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 25, -56(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 24, -64(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 23, -72(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 22, -80(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 21, -88(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 20, -96(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 19, -104(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 18, -112(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 17, -120(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 16, -128(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 15, -136(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    ld 14, -144(1) # 8-byte Folded Reload
 ; POWERPC64-NEXT:    blr
+; POWERPC64-NEXT:  .LBB0_4:
+; POWERPC64-NEXT:    li 4, 0
+; POWERPC64-NEXT:    extsw 3, 4
+; POWERPC64-NEXT:    blr
 ;
 ; POWERPC32-AIX-LABEL: shrinkwrapme:
 ; POWERPC32-AIX:       # %bb.0: # %entry
 ; POWERPC32-AIX-NEXT:    cmpwi 4, 0
+; POWERPC32-AIX-NEXT:    ble 0, L..BB0_4
+; POWERPC32-AIX-NEXT:  # %bb.1: # %for.body.preheader
 ; POWERPC32-AIX-NEXT:    stw 14, -72(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 15, -68(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 16, -64(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 17, -60(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 18, -56(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 19, -52(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 20, -48(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 21, -44(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 22, -40(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 23, -36(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 24, -32(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 25, -28(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 26, -24(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 27, -20(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 28, -16(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 29, -12(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 30, -8(1) # 4-byte Folded Spill
 ; POWERPC32-AIX-NEXT:    stw 31, -4(1) # 4-byte Folded Spill
-; POWERPC32-AIX-NEXT:    ble 0, L..BB0_3
-; POWERPC32-AIX-NEXT:  # %bb.1: # %for.body.preheader
 ; POWERPC32-AIX-NEXT:    mtctr 4
 ; POWERPC32-AIX-NEXT:    li 4, 0
 ; POWERPC32-AIX-NEXT:    .align 4
 ; POWERPC32-AIX-NEXT:  L..BB0_2: # %for.body
 ; POWERPC32-AIX-NEXT:    #
 ; POWERPC32-AIX-NEXT:    #APP
 ; POWERPC32-AIX-NEXT:    add 4, 3, 4
 ; POWERPC32-AIX-NEXT:    #NO_APP
 ; POWERPC32-AIX-NEXT:    bdnz L..BB0_2
-; POWERPC32-AIX-NEXT:    b L..BB0_4
-; POWERPC32-AIX-NEXT:  L..BB0_3:
-; POWERPC32-AIX-NEXT:    li 4, 0
-; POWERPC32-AIX-NEXT:  L..BB0_4: # %for.cond.cleanup
+; POWERPC32-AIX-NEXT:  # %bb.3:
 ; POWERPC32-AIX-NEXT:    lwz 31, -4(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 30, -8(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 29, -12(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 28, -16(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    mr 3, 4
 ; POWERPC32-AIX-NEXT:    lwz 27, -20(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 26, -24(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 25, -28(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 24, -32(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 23, -36(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 22, -40(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 21, -44(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 20, -48(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 19, -52(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 18, -56(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 17, -60(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 16, -64(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 15, -68(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    lwz 14, -72(1) # 4-byte Folded Reload
 ; POWERPC32-AIX-NEXT:    blr
+; POWERPC32-AIX-NEXT:  L..BB0_4:
+; POWERPC32-AIX-NEXT:    li 3, 0
+; POWERPC32-AIX-NEXT:    blr
 ;
 ; POWERPC64-AIX-LABEL: shrinkwrapme:
 ; POWERPC64-AIX:       # %bb.0: # %entry
 ; POWERPC64-AIX-NEXT:    cmpwi 4, 1
+; POWERPC64-AIX-NEXT:    blt 0, L..BB0_4
+; POWERPC64-AIX-NEXT:  # %bb.1: # %for.body.preheader
+; POWERPC64-AIX-NEXT:    addi 4, 4, -1
 ; POWERPC64-AIX-NEXT:    std 14, -144(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 15, -136(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 16, -128(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 17, -120(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 18, -112(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 19, -104(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 20, -96(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 21, -88(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 22, -80(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 23, -72(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 24, -64(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 25, -56(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 26, -48(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 27, -40(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 28, -32(1) # 8-byte Folded Spill
+; POWERPC64-AIX-NEXT:    clrldi 4, 4, 32
+; POWERPC64-AIX-NEXT:    addi 4, 4, 1
 ; POWERPC64-AIX-NEXT:    std 29, -24(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 30, -16(1) # 8-byte Folded Spill
 ; POWERPC64-AIX-NEXT:    std 31, -8(1) # 8-byte Folded Spill
-; POWERPC64-AIX-NEXT:    blt 0, L..BB0_3
-; POWERPC64-AIX-NEXT:  # %bb.1: # %for.body.preheader
-; POWERPC64-AIX-NEXT:    addi 4, 4, -1
-; POWERPC64-AIX-NEXT:    clrldi 4, 4, 32
-; POWERPC64-AIX-NEXT:    addi 4, 4, 1
 ; POWERPC64-AIX-NEXT:    mtctr 4
 ; POWERPC64-AIX-NEXT:    li 4, 0
 ; POWERPC64-AIX-NEXT:    .align 4
 ; POWERPC64-AIX-NEXT:  L..BB0_2: # %for.body
 ; POWERPC64-AIX-NEXT:    #
 ; POWERPC64-AIX-NEXT:    #APP
 ; POWERPC64-AIX-NEXT:    add 4, 3, 4
 ; POWERPC64-AIX-NEXT:    #NO_APP
 ; POWERPC64-AIX-NEXT:    bdnz L..BB0_2
-; POWERPC64-AIX-NEXT:    b L..BB0_4
-; POWERPC64-AIX-NEXT:  L..BB0_3:
-; POWERPC64-AIX-NEXT:    li 4, 0
-; POWERPC64-AIX-NEXT:  L..BB0_4: # %for.cond.cleanup
+; POWERPC64-AIX-NEXT:  # %bb.3:
 ; POWERPC64-AIX-NEXT:    ld 31, -8(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 30, -16(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 29, -24(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 28, -32(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    extsw 3, 4
 ; POWERPC64-AIX-NEXT:    ld 27, -40(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 26, -48(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 25, -56(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 24, -64(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 23, -72(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 22, -80(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 21, -88(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 20, -96(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 19, -104(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 18, -112(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 17, -120(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 16, -128(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 15, -136(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    ld 14, -144(1) # 8-byte Folded Reload
 ; POWERPC64-AIX-NEXT:    blr
+; POWERPC64-AIX-NEXT:  L..BB0_4:
+; POWERPC64-AIX-NEXT:    li 4, 0
+; POWERPC64-AIX-NEXT:    extsw 3, 4
+; POWERPC64-AIX-NEXT:    blr
 entry:
   %cmp5 = icmp sgt i32 %lim, 0
   br i1 %cmp5, label %for.body.preheader, label %for.cond.cleanup
 
  for.body.preheader:                               ; preds = %entry
   br label %for.body
 
  for.cond.cleanup.loopexit:                        ; preds = %for.body

llvm/test/CodeGen/PowerPC/shrink-wrap.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
 # RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple powerpc64le-unknown-linux-gnu \
 # RUN:   -run-pass=shrink-wrap -o - %s | FileCheck %s
 # RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple powerpc-ibm-aix-xcoff \
 # RUN:   -run-pass=shrink-wrap -mattr=-altivec -o - %s | FileCheck %s
 # RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple powerpc64-ibm-aix-xcoff \
 # RUN:   -run-pass=shrink-wrap -mattr=-altivec -o - %s | FileCheck %s
 --- |
   ; ModuleID = 'test.ll'
   source_filename = "test.ll"
--- |
   declare void @llvm.stackprotector(ptr, ptr) #1
 
   attributes #0 = { noduplicate nounwind }
   attributes #1 = { nounwind }
 
 ...
 ---
 name:            shrinkwrapme
-alignment:       16
-exposesReturnsTwice: false
-legalized:       false
-regBankSelected: false
-selected:        false
-failedISel:      false
 tracksRegLiveness: true
-hasWinCFI:       false
-registers:       []
-liveins:
-  - { reg: '$x3', virtual-reg: '' }
-  - { reg: '$x4', virtual-reg: '' }
-frameInfo:
-  isFrameAddressTaken: false
-  isReturnAddressTaken: false
-  hasStackMap:     false
-  hasPatchPoint:   false
-  stackSize:       0
-  offsetAdjustment: 0
-  maxAlignment:    0
-  adjustsStack:    false
-  hasCalls:        false
-  stackProtector:  ''
-  maxCallFrameSize: 4294967295
-  cvBytesOfCalleeSavedRegisters: 0
-  hasOpaqueSPAdjustment: false
-  hasVAStart:      false
-  hasMustTailInVarArgFunc: false
-  localFrameSize:  0
-  savePoint:       ''
-  restorePoint:    ''
-fixedStack:      []
-stack:           []
-callSites:       []
-constants:       []
-machineFunctionInfo: {}
 body:             |
   ; CHECK-LABEL: name: shrinkwrapme
   ; CHECK: bb.0.entry:
   ; CHECK-NEXT:   successors: %bb.2(0x50000000), %bb.1(0x30000000)
   ; CHECK-NEXT:   liveins: $x3, $x4
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   renamable $cr0 = CMPWI renamable $r4, 1
   ; CHECK-NEXT:   BCC 4, killed renamable $cr0, %bb.2
body:             |
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.3.for.cond.cleanup:
   ; CHECK-NEXT:   liveins: $r4
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   renamable $x3 = EXTSW_32_64 killed renamable $r4
   ; CHECK-NEXT:   BLR8 implicit $lr8, implicit $rm, implicit $x3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.4.for.body:
-  ; CHECK-NEXT:   successors: %bb.4(0x7c000000), %bb.3(0x04000000)
+  ; CHECK-NEXT:   successors: %bb.4(0x7c000000), %bb.5(0x04000000)
   ; CHECK-NEXT:   liveins: $r4, $x3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   INLINEASM &"add $0, $1, $2", 0 /* attdialect */, 131082 /* regdef:GPRC */, def renamable $r4, 131081 /* reguse:GPRC */, renamable $r3, 131081 /* reguse:GPRC */, killed renamable $r4, 12 /* clobber */, implicit-def dead early-clobber $r14, 12 /* clobber */, implicit-def dead early-clobber $r15, 12 /* clobber */, implicit-def dead early-clobber $r16, 12 /* clobber */, implicit-def dead early-clobber $r17, 12 /* clobber */, implicit-def dead early-clobber $r18, 12 /* clobber */, implicit-def dead early-clobber $r19, 12 /* clobber */, implicit-def dead early-clobber $r20, 12 /* clobber */, implicit-def dead early-clobber $r21, 12 /* clobber */, implicit-def dead early-clobber $r22, 12 /* clobber */, implicit-def dead early-clobber $r23, 12 /* clobber */, implicit-def dead early-clobber $r24, 12 /* clobber */, implicit-def dead early-clobber $r25, 12 /* clobber */, implicit-def dead early-clobber $r26, 12 /* clobber */, implicit-def dead early-clobber $r27, 12 /* clobber */, implicit-def dead early-clobber $r28, 12 /* clobber */, implicit-def dead early-clobber $r29, 12 /* clobber */, implicit-def dead early-clobber $r30, 12 /* clobber */, implicit-def dead early-clobber $r31
   ; CHECK-NEXT:   BDNZ8 %bb.4, implicit-def dead $ctr8, implicit $ctr8
+  ; CHECK-NEXT:   B %bb.5
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.5:
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT:   liveins: $r4
+  ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   B %bb.3
   bb.0.entry:
     successors: %bb.2(0x50000000), %bb.1(0x30000000)
     liveins: $x3, $x4
 
     renamable $cr0 = CMPWI renamable $r4, 1
     BCC 4, killed renamable $cr0, %bb.2
 

llvm/test/CodeGen/RISCV/aext-to-sext.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \
 ; RUN:   | FileCheck %s -check-prefix=RV64I
 
 ; Make sure we don't generate an addi in the loop in
 ; addition to the addiw. Previously we type legalize the
 ; setcc use using signext and the phi use using anyext.
 ; We now detect when it would be beneficial to replace
 ; anyext with signext.
 
 define void @quux(i32 signext %arg, i32 signext %arg1) nounwind {
 ; RV64I-LABEL: quux:
 ; RV64I:       # %bb.0: # %bb
+; RV64I-NEXT:    beq a0, a1, .LBB0_4
+; RV64I-NEXT:  # %bb.1: # %bb2.preheader
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
-; RV64I-NEXT:    beq a0, a1, .LBB0_3
-; RV64I-NEXT:  # %bb.1: # %bb2.preheader
 ; RV64I-NEXT:    subw s0, a1, a0
 ; RV64I-NEXT:  .LBB0_2: # %bb2
 ; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV64I-NEXT:    call hoge@plt
 ; RV64I-NEXT:    addiw s0, s0, -1
 ; RV64I-NEXT:    bnez s0, .LBB0_2
-; RV64I-NEXT:  .LBB0_3: # %bb6
+; RV64I-NEXT:  # %bb.3:
 ; RV64I-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld s0, 0(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 16
+; RV64I-NEXT:  .LBB0_4: # %bb6
 ; RV64I-NEXT:    ret
 bb:
   %tmp = icmp eq i32 %arg, %arg1
   br i1 %tmp, label %bb6, label %bb2
 
 bb2:                                              ; preds = %bb2, %bb
   %tmp3 = phi i32 [ %tmp4, %bb2 ], [ %arg, %bb ]
   tail call void @hoge()

llvm/test/CodeGen/RISCV/fli-licm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc < %s -mtriple=riscv32 -target-abi=ilp32f -mattr=+experimental-zfa \
 ; RUN:   | FileCheck %s --check-prefix=RV32
 ; RUN: llc < %s -mtriple=riscv64 -target-abi=lp64f -mattr=+experimental-zfa \
 ; RUN:   | FileCheck %s --check-prefix=RV64
 
 ; The purpose of this test is to check that an FLI instruction that
 ; materializes an immediate is not MachineLICM'd out of a loop.
 
 %struct.Node = type { ptr, i8* }
 
 define void @process_nodes(ptr %0) nounwind {
 ; RV32-LABEL: process_nodes:
 ; RV32:       # %bb.0: # %entry
+; RV32-NEXT:    beqz a0, .LBB0_4
+; RV32-NEXT:  # %bb.1: # %loop.preheader
 ; RV32-NEXT:    addi sp, sp, -16
 ; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
-; RV32-NEXT:    beqz a0, .LBB0_3
-; RV32-NEXT:  # %bb.1: # %loop.preheader
 ; RV32-NEXT:    mv s0, a0
 ; RV32-NEXT:  .LBB0_2: # %loop
 ; RV32-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV32-NEXT:    fli.s fa0, 1.0
 ; RV32-NEXT:    mv a0, s0
 ; RV32-NEXT:    call do_it@plt
 ; RV32-NEXT:    lw s0, 0(s0)
 ; RV32-NEXT:    bnez s0, .LBB0_2
-; RV32-NEXT:  .LBB0_3: # %exit
+; RV32-NEXT:  # %bb.3:
 ; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    addi sp, sp, 16
+; RV32-NEXT:  .LBB0_4: # %exit
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: process_nodes:
 ; RV64:       # %bb.0: # %entry
+; RV64-NEXT:    beqz a0, .LBB0_4
+; RV64-NEXT:  # %bb.1: # %loop.preheader
 ; RV64-NEXT:    addi sp, sp, -16
 ; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    sd s0, 0(sp) # 8-byte Folded Spill
-; RV64-NEXT:    beqz a0, .LBB0_3
-; RV64-NEXT:  # %bb.1: # %loop.preheader
 ; RV64-NEXT:    mv s0, a0
 ; RV64-NEXT:  .LBB0_2: # %loop
 ; RV64-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV64-NEXT:    fli.s fa0, 1.0
 ; RV64-NEXT:    mv a0, s0
 ; RV64-NEXT:    call do_it@plt
 ; RV64-NEXT:    ld s0, 0(s0)
 ; RV64-NEXT:    bnez s0, .LBB0_2
-; RV64-NEXT:  .LBB0_3: # %exit
+; RV64-NEXT:  # %bb.3:
 ; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    ld s0, 0(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    addi sp, sp, 16
+; RV64-NEXT:  .LBB0_4: # %exit
 ; RV64-NEXT:    ret
 entry:
   %1 = icmp eq ptr %0, null
   br i1 %1, label %exit, label %loop
 
 loop:
   %2 = phi %struct.Node* [ %4, %loop ], [ %0, %entry ]
   tail call void @do_it(float 1.000000e+00, ptr nonnull %2)

llvm/test/CodeGen/Thumb2/LowOverheadLoops/inlineasm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs -o - %s | FileCheck %s
 
 define i32 @test(ptr nocapture readonly %x, ptr nocapture readonly %y, i32 %n) {
 ; CHECK-LABEL: test:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    itt lt
+; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB0_1: @ %for.body.preheader
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB0_4
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    mov lr, r0
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:  .LBB0_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrh r3, [r1], #2
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    ldrh r12, [lr], #2
 ; CHECK-NEXT:    @APP
 ; CHECK-NEXT:    add r3, r12
 ; CHECK-NEXT:    @NO_APP
 ; CHECK-NEXT:    add r0, r3
 ; CHECK-NEXT:    bne .LBB0_2
-; CHECK-NEXT:  @ %bb.3: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r7, pc}
-; CHECK-NEXT:  .LBB0_4:
-; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp9 = icmp sgt i32 %n, 0
   br i1 %cmp9, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   %s.0.lcssa = phi i32 [ 0, %entry ], [ %add, %for.body ]
   ret i32 %s.0.lcssa
for.body:                                         ; preds = %entry, %for.body
   %inc = add nuw nsw i32 %i.010, 1
   %exitcond.not = icmp eq i32 %inc, %n
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
 define i32 @testlr(ptr nocapture readonly %x, ptr nocapture readonly %y, i32 %n) {
 ; CHECK-LABEL: testlr:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    itt lt
+; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB1_1: @ %for.body.preheader
 ; CHECK-NEXT:    .save {r4, lr}
 ; CHECK-NEXT:    push {r4, lr}
-; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB1_4
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    mov r3, r0
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:  .LBB1_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrh r4, [r1], #2
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    ldrh r12, [r3], #2
 ; CHECK-NEXT:    @APP
 ; CHECK-NEXT:    add r4, r12
 ; CHECK-NEXT:    @NO_APP
 ; CHECK-NEXT:    add r0, r4
 ; CHECK-NEXT:    bne .LBB1_2
-; CHECK-NEXT:  @ %bb.3: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, pc}
-; CHECK-NEXT:  .LBB1_4:
-; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    pop {r4, pc}
 entry:
   %cmp9 = icmp sgt i32 %n, 0
   br i1 %cmp9, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   %s.0.lcssa = phi i32 [ 0, %entry ], [ %add, %for.body ]
   ret i32 %s.0.lcssa

llvm/test/CodeGen/Thumb2/LowOverheadLoops/memcall.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --arm-memtransfer-tploop=allow -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -verify-machineinstrs -tail-predication=enabled -o - %s | FileCheck %s
 
 define void @test_memcpy(ptr nocapture %x, ptr nocapture readonly %y, i32 %n, i32 %m) {
 ; CHECK-LABEL: test_memcpy:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB0_1: @ %for.body.preheader
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB0_5
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    lsl.w r12, r3, #2
 ; CHECK-NEXT:    movs r7, #0
 ; CHECK-NEXT:    b .LBB0_2
 ; CHECK-NEXT:  .LBB0_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB0_4 Depth 2
 ; CHECK-NEXT:    adds r4, r1, r7
 ; CHECK-NEXT:    adds r5, r0, r7
 ; CHECK-NEXT:    wlstp.8 lr, r3, .LBB0_3
 ; CHECK-NEXT:    b .LBB0_4
 ; CHECK-NEXT:  .LBB0_3: @ %for.body
 ; CHECK-NEXT:    @ in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    add r7, r12
 ; CHECK-NEXT:    subs r2, #1
 ; CHECK-NEXT:    beq .LBB0_5
 ; CHECK-NEXT:    b .LBB0_2
 ; CHECK-NEXT:  .LBB0_4: @ Parent Loop BB0_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vldrb.u8 q0, [r4], #16
 ; CHECK-NEXT:    vstrb.8 q0, [r5], #16
 ; CHECK-NEXT:    letp lr, .LBB0_4
 ; CHECK-NEXT:    b .LBB0_3
-; CHECK-NEXT:  .LBB0_5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:  .LBB0_5:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp8 = icmp sgt i32 %n, 0
   br i1 %cmp8, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   ret void
 
 for.body:                                         ; preds = %entry, %for.body
   %i.011 = phi i32 [ %inc, %for.body ], [ 0, %entry ]
   %x.addr.010 = phi ptr [ %add.ptr, %for.body ], [ %x, %entry ]
   %y.addr.09 = phi ptr [ %add.ptr1, %for.body ], [ %y, %entry ]
   tail call void @llvm.memcpy.p0.p0.i32(ptr align 4 %x.addr.010, ptr align 4 %y.addr.09, i32 %m, i1 false)
   %add.ptr = getelementptr inbounds i32, ptr %x.addr.010, i32 %m
   %add.ptr1 = getelementptr inbounds i32, ptr %y.addr.09, i32 %m
   %inc = add nuw nsw i32 %i.011, 1
   %exitcond.not = icmp eq i32 %inc, %n
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
 define void @test_memset(ptr nocapture %x, i32 %n, i32 %m) {
 ; CHECK-LABEL: test_memset:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r7, pc}
+; CHECK-NEXT:    bxlt lr
 ; CHECK-NEXT:  .LBB1_1:
+; CHECK-NEXT:    .save {r7, lr}
+; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    b .LBB1_2
 ; CHECK-NEXT:  .LBB1_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB1_4 Depth 2
 ; CHECK-NEXT:    mov r12, r0
 ; CHECK-NEXT:    wlstp.8 lr, r2, .LBB1_3
 ; CHECK-NEXT:    b .LBB1_4
 ; CHECK-NEXT:  .LBB1_3: @ %for.body
 ; CHECK-NEXT:    @ in Loop: Header=BB1_2 Depth=1
 ; CHECK-NEXT:    add.w r0, r0, r2, lsl #2
 ; CHECK-NEXT:    subs r1, #1
 ; CHECK-NEXT:    beq .LBB1_5
 ; CHECK-NEXT:    b .LBB1_2
 ; CHECK-NEXT:  .LBB1_4: @ Parent Loop BB1_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vstrb.8 q0, [r12], #16
 ; CHECK-NEXT:    letp lr, .LBB1_4
 ; CHECK-NEXT:    b .LBB1_3
-; CHECK-NEXT:  .LBB1_5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:  .LBB1_5:
+; CHECK-NEXT:    pop.w {r7, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp5 = icmp sgt i32 %n, 0
   br i1 %cmp5, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   ret void
 
 for.body:                                         ; preds = %entry, %for.body
   %i.07 = phi i32 [ %inc, %for.body ], [ 0, %entry ]
   %x.addr.06 = phi ptr [ %add.ptr, %for.body ], [ %x, %entry ]
   tail call void @llvm.memset.p0.i32(ptr align 4 %x.addr.06, i8 0, i32 %m, i1 false)
   %add.ptr = getelementptr inbounds i32, ptr %x.addr.06, i32 %m
   %inc = add nuw nsw i32 %i.07, 1
   %exitcond.not = icmp eq i32 %inc, %n
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
 define void @test_memmove(ptr nocapture %x, ptr nocapture readonly %y, i32 %n, i32 %m) {
 ; CHECK-LABEL: test_memmove:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB2_1: @ %for.body.preheader
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB2_3
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    mov r8, r3
 ; CHECK-NEXT:    mov r5, r2
 ; CHECK-NEXT:    mov r9, r1
 ; CHECK-NEXT:    mov r7, r0
 ; CHECK-NEXT:    lsls r4, r3, #2
 ; CHECK-NEXT:    movs r6, #0
 ; CHECK-NEXT:  .LBB2_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    adds r0, r7, r6
 ; CHECK-NEXT:    add.w r1, r9, r6
 ; CHECK-NEXT:    mov r2, r8
 ; CHECK-NEXT:    bl __aeabi_memmove4
 ; CHECK-NEXT:    add r6, r4
 ; CHECK-NEXT:    subs r5, #1
 ; CHECK-NEXT:    bne .LBB2_2
-; CHECK-NEXT:  .LBB2_3: @ %for.cond.cleanup
+; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp8 = icmp sgt i32 %n, 0
   br i1 %cmp8, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:                                 ; preds = %for.body, %entry
   ret void
 
 for.body:                                         ; preds = %entry, %for.body

llvm/test/CodeGen/Thumb2/LowOverheadLoops/mve-float-loops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve.fp,+fp-armv8d16sp,+fp16,+fullfp16 %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc void @float_float_mul(ptr nocapture readonly %a, ptr nocapture readonly %b, ptr nocapture %c, i32 %N) {
 ; CHECK-LABEL: float_float_mul:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB0_10
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB0_1: @ %for.body.preheader
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #3
 ; CHECK-NEXT:    bhi .LBB0_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    b .LBB0_4
 ; CHECK-NEXT:  .LBB0_3: @ %vector.memcheck
 ; CHECK-NEXT:    add.w r7, r1, r3, lsl #2
 ; CHECK-NEXT:    add.w r6, r2, r3, lsl #2

 ; CHECK-NEXT:    vldr s2, [r6, #8]
 ; CHECK-NEXT:    vmul.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #8]
 ; CHECK-NEXT:    vldr s0, [r7, #12]
 ; CHECK-NEXT:    vldr s2, [r6, #12]
 ; CHECK-NEXT:    vmul.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #12]
 ; CHECK-NEXT:    bne .LBB0_9
-; CHECK-NEXT:  .LBB0_10: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  .LBB0_10:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:  .LBB0_11: @ %vector.ph
 ; CHECK-NEXT:    bic r12, r3, #3
 ; CHECK-NEXT:    movs r6, #1
 ; CHECK-NEXT:    sub.w r7, r12, #4
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
 ; CHECK-NEXT:    mov r6, r2
for.body:                                         ; preds = %for.body.prol.loopexit, %for.body
   %inc.3 = add nuw i32 %i.09, 4
   %exitcond.3 = icmp eq i32 %inc.3, %N
   br i1 %exitcond.3, label %for.cond.cleanup, label %for.body
 }
 
 define arm_aapcs_vfpcc void @float_float_add(ptr nocapture readonly %a, ptr nocapture readonly %b, ptr nocapture %c, i32 %N) {
 ; CHECK-LABEL: float_float_add:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB1_10
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB1_1: @ %for.body.preheader
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #3
 ; CHECK-NEXT:    bhi .LBB1_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    b .LBB1_4
 ; CHECK-NEXT:  .LBB1_3: @ %vector.memcheck
 ; CHECK-NEXT:    add.w r7, r1, r3, lsl #2
 ; CHECK-NEXT:    add.w r6, r2, r3, lsl #2

 ; CHECK-NEXT:    vldr s2, [r6, #8]
 ; CHECK-NEXT:    vadd.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #8]
 ; CHECK-NEXT:    vldr s0, [r7, #12]
 ; CHECK-NEXT:    vldr s2, [r6, #12]
 ; CHECK-NEXT:    vadd.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #12]
 ; CHECK-NEXT:    bne .LBB1_9
-; CHECK-NEXT:  .LBB1_10: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  .LBB1_10:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:  .LBB1_11: @ %vector.ph
 ; CHECK-NEXT:    bic r12, r3, #3
 ; CHECK-NEXT:    movs r6, #1
 ; CHECK-NEXT:    sub.w r7, r12, #4
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
 ; CHECK-NEXT:    mov r6, r2
for.body:                                         ; preds = %for.body.prol.loopexit, %for.body
   %inc.3 = add nuw i32 %i.09, 4
   %exitcond.3 = icmp eq i32 %inc.3, %N
   br i1 %exitcond.3, label %for.cond.cleanup, label %for.body
 }
 
 define arm_aapcs_vfpcc void @float_float_sub(ptr nocapture readonly %a, ptr nocapture readonly %b, ptr nocapture %c, i32 %N) {
 ; CHECK-LABEL: float_float_sub:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq .LBB2_10
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB2_1: @ %for.body.preheader
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #3
 ; CHECK-NEXT:    bhi .LBB2_3
 ; CHECK-NEXT:  @ %bb.2:
 ; CHECK-NEXT:    mov.w r12, #0
 ; CHECK-NEXT:    b .LBB2_4
 ; CHECK-NEXT:  .LBB2_3: @ %vector.memcheck
 ; CHECK-NEXT:    add.w r7, r1, r3, lsl #2
 ; CHECK-NEXT:    add.w r6, r2, r3, lsl #2

 ; CHECK-NEXT:    vldr s2, [r6, #8]
 ; CHECK-NEXT:    vsub.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #8]
 ; CHECK-NEXT:    vldr s0, [r7, #12]
 ; CHECK-NEXT:    vldr s2, [r6, #12]
 ; CHECK-NEXT:    vsub.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r5, #12]
 ; CHECK-NEXT:    bne .LBB2_9
-; CHECK-NEXT:  .LBB2_10: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  .LBB2_10:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:  .LBB2_11: @ %vector.ph
 ; CHECK-NEXT:    bic r12, r3, #3
 ; CHECK-NEXT:    movs r6, #1
 ; CHECK-NEXT:    sub.w r7, r12, #4
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    add.w lr, r6, r7, lsr #2
 ; CHECK-NEXT:    mov r6, r2
for.body:                                         ; preds = %for.body.prol.loopexit, %for.body
   %inc.3 = add nuw i32 %i.09, 4
   %exitcond.3 = icmp eq i32 %inc.3, %N
   br i1 %exitcond.3, label %for.cond.cleanup, label %for.body
 }
 
 define arm_aapcs_vfpcc void @float_int_mul(ptr nocapture readonly %a, ptr nocapture readonly %b, ptr nocapture %c, i32 %N) {
 ; CHECK-LABEL: float_int_mul:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #0
-; CHECK-NEXT:    beq.w .LBB3_13
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB3_1: @ %for.body.preheader
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #3
 ; CHECK-NEXT:    bls .LBB3_6
 ; CHECK-NEXT:  @ %bb.2: @ %vector.memcheck
 ; CHECK-NEXT:    add.w r7, r0, r3, lsl #2
 ; CHECK-NEXT:    cmp r7, r2
 ; CHECK-NEXT:    itt hi
 ; CHECK-NEXT:    addhi.w r7, r2, r3, lsl #2
 ; CHECK-NEXT:    cmphi r7, r0

 ; CHECK-NEXT:    vstr s0, [r6, #8]
 ; CHECK-NEXT:    vldr s0, [r1, #4]
 ; CHECK-NEXT:    add.w r1, r1, #16
 ; CHECK-NEXT:    vldr s2, [r7, #12]
 ; CHECK-NEXT:    vcvt.f32.s32 s0, s0
 ; CHECK-NEXT:    vmul.f32 s0, s2, s0
 ; CHECK-NEXT:    vstr s0, [r6, #12]
 ; CHECK-NEXT:    bne .LBB3_12
-; CHECK-NEXT:  .LBB3_13: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  .LBB3_13:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp8 = icmp eq i32 %N, 0
   br i1 %cmp8, label %for.cond.cleanup, label %for.body.preheader
 
 for.body.preheader:                               ; preds = %entry
   %min.iters.check = icmp ult i32 %N, 4
   br i1 %min.iters.check, label %for.body.preheader16, label %vector.memcheck
 

llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions.ll

 for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i16 [ 0, %entry ], [ %i10, %middle.block ]
   ret i16 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc i32 @two_loops_mul_add_v4i32(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: two_loops_mul_add_v4i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    beq .LBB6_8
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
+; CHECK-NEXT:    itt eq
+; CHECK-NEXT:    moveq r0, #0
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB6_1: @ %vector.ph
+; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #3
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #3
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    subs r7, r3, #4
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    add.w r6, r3, r7, lsr #2

 ; CHECK-NEXT:    vldrbt.u32 q2, [r1], #4
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    vmul.i32 q0, q2, q0
 ; CHECK-NEXT:    vadd.i32 q0, q0, q1
 ; CHECK-NEXT:    le lr, .LBB6_5
 ; CHECK-NEXT:  @ %bb.6: @ %middle.block44
 ; CHECK-NEXT:    vpsel q0, q0, q1
 ; CHECK-NEXT:    vaddv.u32 r12, q0
-; CHECK-NEXT:  .LBB6_7: @ %for.cond.cleanup7
+; CHECK-NEXT:  .LBB6_7:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    mov r0, r12
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
-; CHECK-NEXT:  .LBB6_8:
-; CHECK-NEXT:    movs r0, #0
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp35 = icmp eq i32 %N, 0
   br i1 %cmp35, label %for.cond.cleanup7, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 3
   %n.vec = and i32 %n.rnd.up, -4
   br label %vector.body

llvm/test/CodeGen/Thumb2/LowOverheadLoops/sibling-loops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+lob --verify-machineinstrs %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc void @test(ptr noalias nocapture readonly %off, ptr noalias nocapture %data, ptr noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: test:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    cmp r3, #1
-; CHECK-NEXT:    blt .LBB0_7
-; CHECK-NEXT:  @ %bb.1: @ %for.cond1.preheader.us.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB0_1: @ %for.cond1.preheader.us.preheader
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    mov r8, r3
 ; CHECK-NEXT:    lsl.w r12, r3, #1
 ; CHECK-NEXT:    movs r3, #0
 ; CHECK-NEXT:    mov r4, r1
 ; CHECK-NEXT:  .LBB0_2: @ %for.cond1.preheader.us
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB0_3 Depth 2
 ; CHECK-NEXT:    @ Child Loop BB0_5 Depth 2

 ; CHECK-NEXT:    le lr, .LBB0_5
 ; CHECK-NEXT:  @ %bb.6: @ %for.cond.cleanup14.us
 ; CHECK-NEXT:    @ in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    adds r3, #1
 ; CHECK-NEXT:    add r2, r12
 ; CHECK-NEXT:    add r4, r12
 ; CHECK-NEXT:    cmp r3, r8
 ; CHECK-NEXT:    bne .LBB0_2
-; CHECK-NEXT:  .LBB0_7: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  @ %bb.7:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp252 = icmp sgt i32 %n, 0
   br i1 %cmp252, label %for.cond1.preheader.us, label %for.cond.cleanup
 
 for.cond1.preheader.us: ; preds = %entry, %for.cond.cleanup14.us
   %i.057.us = phi i32 [ %inc29.us, %for.cond.cleanup14.us ], [ 0, %entry ]
   %mul.us = mul i32 %i.057.us, %n
   br label %for.body4.us

llvm/test/CodeGen/Thumb2/LowOverheadLoops/spillingmove.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve --verify-machineinstrs %s -o - | FileCheck %s
 
 %struct.arm_2d_size_t = type { i16, i16 }
 define void @__arm_2d_impl_rgb16_colour_filling_with_alpha(ptr noalias nocapture %phwTargetBase, i16 signext %iTargetStride, ptr noalias nocapture readonly %ptCopySize, i16 zeroext %hwColour, i32 %chRatio) {
 ; CHECK-LABEL: __arm_2d_impl_rgb16_colour_filling_with_alpha:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    ldrsh.w r12, [r2, #2]
+; CHECK-NEXT:    cmp.w r12, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB0_1: @ %for.cond3.preheader.lr.ph
 ; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    sub sp, #64
-; CHECK-NEXT:    ldrsh.w r12, [r2, #2]
-; CHECK-NEXT:    cmp.w r12, #1
-; CHECK-NEXT:    itt ge
-; CHECK-NEXT:    ldrshge.w r7, [r2]
-; CHECK-NEXT:    cmpge r7, #1
-; CHECK-NEXT:    blt.w .LBB0_5
-; CHECK-NEXT:  @ %bb.1: @ %for.cond3.preheader.us.preheader
+; CHECK-NEXT:    ldrsh.w r7, [r2]
+; CHECK-NEXT:    cmp r7, #1
+; CHECK-NEXT:    blt.w .LBB0_6
+; CHECK-NEXT:  @ %bb.2: @ %for.cond3.preheader.us.preheader
 ; CHECK-NEXT:    movs r2, #252
 ; CHECK-NEXT:    ldr r4, [sp, #152]
 ; CHECK-NEXT:    and.w r6, r2, r3, lsr #3
 ; CHECK-NEXT:    movs r2, #120
 ; CHECK-NEXT:    and.w r5, r2, r3, lsr #9
 ; CHECK-NEXT:    lsls r3, r3, #3
 ; CHECK-NEXT:    uxtb r3, r3
 ; CHECK-NEXT:    muls r6, r4, r6

 ; CHECK-NEXT:    add.w r3, r4, r3, lsr #3
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #48] @ 16-byte Spill
 ; CHECK-NEXT:    vmov.i16 q0, #0xf800
 ; CHECK-NEXT:    movs r4, #0
 ; CHECK-NEXT:    vmov.i16 q7, #0x78
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill
 ; CHECK-NEXT:    vstrw.32 q2, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vstrw.32 q3, [sp, #16] @ 16-byte Spill
-; CHECK-NEXT:  .LBB0_2: @ %vector.ph
+; CHECK-NEXT:  .LBB0_3: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
-; CHECK-NEXT:    @ Child Loop BB0_3 Depth 2
+; CHECK-NEXT:    @ Child Loop BB0_4 Depth 2
 ; CHECK-NEXT:    mov r5, r0
 ; CHECK-NEXT:    mov r6, r7
 ; CHECK-NEXT:    dls lr, r3
-; CHECK-NEXT:  .LBB0_3: @ %vector.body
-; CHECK-NEXT:    @ Parent Loop BB0_2 Depth=1
+; CHECK-NEXT:  .LBB0_4: @ %vector.body
+; CHECK-NEXT:    @ Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vctp.16 r6
 ; CHECK-NEXT:    subs r6, #8
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vldrht.u16 q0, [r5]
 ; CHECK-NEXT:    vshr.u16 q1, q0, #3
 ; CHECK-NEXT:    vand q1, q1, q2
 ; CHECK-NEXT:    vmov q2, q4

 ; CHECK-NEXT:    vand q0, q0, q7
 ; CHECK-NEXT:    vmla.i16 q2, q0, r2
 ; CHECK-NEXT:    vldrw.u32 q0, [sp] @ 16-byte Reload
 ; CHECK-NEXT:    vand q0, q2, q0
 ; CHECK-NEXT:    vldrw.u32 q2, [sp, #32] @ 16-byte Reload
 ; CHECK-NEXT:    vorr q0, q1, q0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrht.16 q0, [r5], #16
-; CHECK-NEXT:    le lr, .LBB0_3
-; CHECK-NEXT:  @ %bb.4: @ %for.cond3.for.cond.cleanup7_crit_edge.us
-; CHECK-NEXT:    @ in Loop: Header=BB0_2 Depth=1
+; CHECK-NEXT:    le lr, .LBB0_4
+; CHECK-NEXT:  @ %bb.5: @ %for.cond3.for.cond.cleanup7_crit_edge.us
+; CHECK-NEXT:    @ in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    adds r4, #1
 ; CHECK-NEXT:    add.w r0, r0, r1, lsl #1
 ; CHECK-NEXT:    cmp r4, r12
-; CHECK-NEXT:    bne .LBB0_2
-; CHECK-NEXT:  .LBB0_5: @ %for.cond.cleanup
+; CHECK-NEXT:    bne .LBB0_3
+; CHECK-NEXT:  .LBB0_6:
 ; CHECK-NEXT:    add sp, #64
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %iHeight = getelementptr inbounds %struct.arm_2d_size_t, ptr %ptCopySize, i32 0, i32 1
   %0 = load i16, ptr %iHeight, align 2
   %conv1 = sext i16 %0 to i32
   %and.i = shl i16 %hwColour, 3
   %shl.i = and i16 %and.i, 248
   %1 = lshr i16 %hwColour, 9
   %shl4.i = and i16 %1, 120
for.cond3.for.cond.cleanup7_crit_edge.us:         ; preds = %vector.body
   br i1 %exitcond66.not, label %for.cond.cleanup, label %vector.ph
 
 for.cond.cleanup:                                 ; preds = %for.cond3.for.cond.cleanup7_crit_edge.us, %for.cond3.preheader.lr.ph, %entry
   ret void
 }
 define void @__arm_2d_impl_rgb16_colour_filling_with_alpha_sched(ptr noalias nocapture %phwTargetBase, i16 signext %iTargetStride, ptr noalias nocapture readonly %ptCopySize, i16 zeroext %hwColour, i32 %chRatio) "target-cpu"="cortex-m55" {
 ; CHECK-LABEL: __arm_2d_impl_rgb16_colour_filling_with_alpha_sched:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    sub sp, #80
 ; CHECK-NEXT:    ldrsh.w r12, [r2, #2]
 ; CHECK-NEXT:    cmp.w r12, #1
-; CHECK-NEXT:    blt.w .LBB1_6
+; CHECK-NEXT:    blt.w .LBB1_7
 ; CHECK-NEXT:  @ %bb.1: @ %for.cond3.preheader.lr.ph
 ; CHECK-NEXT:    ldrsh.w r2, [r2]
 ; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB1_6
-; CHECK-NEXT:  @ %bb.2: @ %for.cond3.preheader.us.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB1_2: @ %for.cond3.preheader.us.preheader
+; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
+; CHECK-NEXT:    sub sp, #4
+; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
+; CHECK-NEXT:    sub sp, #80
 ; CHECK-NEXT:    ldr r7, [sp, #168]
 ; CHECK-NEXT:    movs r5, #120
 ; CHECK-NEXT:    lsls r6, r3, #3
 ; CHECK-NEXT:    movs r4, #252
 ; CHECK-NEXT:    and.w r5, r5, r3, lsr #9
 ; CHECK-NEXT:    uxtb r6, r6
 ; CHECK-NEXT:    and.w r3, r4, r3, lsr #3
 ; CHECK-NEXT:    muls r6, r7, r6

 ; CHECK-NEXT:    vmov.f64 d5, d7
 ; CHECK-NEXT:    letp lr, .LBB1_4
 ; CHECK-NEXT:  @ %bb.5: @ %for.cond3.for.cond.cleanup7_crit_edge.us
 ; CHECK-NEXT:    @ in Loop: Header=BB1_3 Depth=1
 ; CHECK-NEXT:    add.w r0, r0, r1, lsl #1
 ; CHECK-NEXT:    adds r4, #1
 ; CHECK-NEXT:    cmp r4, r12
 ; CHECK-NEXT:    bne .LBB1_3
-; CHECK-NEXT:  .LBB1_6: @ %for.cond.cleanup
+; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:    add sp, #80
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, lr}
+; CHECK-NEXT:  .LBB1_7: @ %for.cond.cleanup
+; CHECK-NEXT:    bx lr
 entry:
   %iHeight = getelementptr inbounds %struct.arm_2d_size_t, ptr %ptCopySize, i32 0, i32 1
   %0 = load i16, ptr %iHeight, align 2
   %conv1 = sext i16 %0 to i32
   %and.i = shl i16 %hwColour, 3
   %shl.i = and i16 %and.i, 248
   %1 = lshr i16 %hwColour, 9
   %shl4.i = and i16 %1, 120

llvm/test/CodeGen/Thumb2/LowOverheadLoops/while-loops.ll

if.end:                                           ; preds = %do.body, %entry
   store i32 %s.1, ptr %z, align 4
   ret void
 }
 
 ; Tail predicated so we use DLSTP
 define void @nested(ptr nocapture readonly %x, ptr nocapture readnone %y, ptr nocapture %z, i32 %m, i32 %n) {
 ; CHECK-LABEL: nested:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #0
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB1_1: @ %for.body.preheader
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, lr}
-; CHECK-NEXT:    cbz r3, .LBB1_8
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
 ; CHECK-NEXT:    ldr.w r12, [sp, #24]
 ; CHECK-NEXT:    movs r1, #0
 ; CHECK-NEXT:    b .LBB1_4
 ; CHECK-NEXT:  .LBB1_2: @ in Loop: Header=BB1_4 Depth=1
 ; CHECK-NEXT:    mov r4, r3
 ; CHECK-NEXT:  .LBB1_3: @ %if.end
 ; CHECK-NEXT:    @ in Loop: Header=BB1_4 Depth=1
 ; CHECK-NEXT:    str.w r4, [r2, r1, lsl #2]

 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vaddva.s32 r4, q0
 ; CHECK-NEXT:    letp lr, .LBB1_6
 ; CHECK-NEXT:  @ %bb.7: @ %if.end.loopexit
 ; CHECK-NEXT:    @ in Loop: Header=BB1_4 Depth=1
 ; CHECK-NEXT:    sub.w r12, r12, r5
 ; CHECK-NEXT:    mov r0, r8
 ; CHECK-NEXT:    b .LBB1_3
-; CHECK-NEXT:  .LBB1_8: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, pc}
+; CHECK-NEXT:  .LBB1_8:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp20.not = icmp eq i32 %m, 0
   br i1 %cmp20.not, label %for.cond.cleanup, label %for.body
 
 for.cond.cleanup:                                 ; preds = %if.end, %entry
   ret void
 
 for.body:                                         ; preds = %entry, %if.end

llvm/test/CodeGen/Thumb2/mve-float32regloops.ll

 if.end61:                                         ; preds = %if.then59, %while.end55
   ret void
 }
 
 
 define void @fir(ptr nocapture readonly %S, ptr nocapture readonly %pSrc, ptr nocapture %pDst, i32 %blockSize) {
 ; CHECK-LABEL: fir:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #8
+; CHECK-NEXT:    blo.w .LBB16_13
+; CHECK-NEXT:  @ %bb.1: @ %if.then
+; CHECK-NEXT:    lsrs.w r12, r3, #2
+; CHECK-NEXT:    it eq
+; CHECK-NEXT:    bxeq lr
+; CHECK-NEXT:  .LBB16_2: @ %while.body.lr.ph
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    .pad #32
 ; CHECK-NEXT:    sub sp, #32
-; CHECK-NEXT:    cmp r3, #8
-; CHECK-NEXT:    blo.w .LBB16_12
-; CHECK-NEXT:  @ %bb.1: @ %if.then
-; CHECK-NEXT:    lsrs.w r12, r3, #2
-; CHECK-NEXT:    beq.w .LBB16_12
-; CHECK-NEXT:  @ %bb.2: @ %while.body.lr.ph
 ; CHECK-NEXT:    ldrh r6, [r0]
 ; CHECK-NEXT:    movs r5, #1
 ; CHECK-NEXT:    ldrd r4, r10, [r0, #4]
 ; CHECK-NEXT:    sub.w r0, r6, #8
 ; CHECK-NEXT:    add.w r3, r0, r0, lsr #29
 ; CHECK-NEXT:    and r0, r0, #7
 ; CHECK-NEXT:    asrs r7, r3, #3
 ; CHECK-NEXT:    cmp r7, #1

 ; CHECK-NEXT:    vldrw.u32 q1, [r3], #4
 ; CHECK-NEXT:    vfma.f32 q0, q1, r0
 ; CHECK-NEXT:    le lr, .LBB16_10
 ; CHECK-NEXT:  @ %bb.11: @ %while.end.loopexit
 ; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
 ; CHECK-NEXT:    add.w r4, r4, r0, lsl #2
 ; CHECK-NEXT:    b .LBB16_4
-; CHECK-NEXT:  .LBB16_12: @ %if.end
+; CHECK-NEXT:  .LBB16_12:
 ; CHECK-NEXT:    add sp, #32
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    add sp, #4
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
+; CHECK-NEXT:  .LBB16_13: @ %if.end
+; CHECK-NEXT:    bx lr
 entry:
   %pState1 = getelementptr inbounds %struct.arm_fir_instance_f32, ptr %S, i32 0, i32 1
   %i = load ptr, ptr %pState1, align 4
   %pCoeffs2 = getelementptr inbounds %struct.arm_fir_instance_f32, ptr %S, i32 0, i32 2
   %i1 = load ptr, ptr %pCoeffs2, align 4
   %numTaps3 = getelementptr inbounds %struct.arm_fir_instance_f32, ptr %S, i32 0, i32 0
   %i2 = load i16, ptr %numTaps3, align 4
   %conv = zext i16 %i2 to i32

llvm/test/CodeGen/Thumb2/mve-gather-increment.ll

 
 end:
   ret void;
 }
 
 define arm_aapcs_vfpcc void @gather_inc_v4i32_simple(ptr noalias nocapture readonly %data, ptr noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: gather_inc_v4i32_simple:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r4, pc}
+; CHECK-NEXT:    bxlt lr
 ; CHECK-NEXT:  .LBB8_1: @ %vector.ph.preheader
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    bic r12, r2, #3
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    sub.w lr, r12, #4
 ; CHECK-NEXT:    add.w r4, r3, lr, lsr #2
 ; CHECK-NEXT:    adr r3, .LCPI8_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
 ; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB8_2: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB8_3 Depth 2
 ; CHECK-NEXT:    dls lr, r4
 ; CHECK-NEXT:    mov r0, r1
 ; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:  .LBB8_3: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB8_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vldrw.u32 q2, [q1, #16]!
 ; CHECK-NEXT:    vstrb.8 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB8_2 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB8_2
-; CHECK-NEXT:  @ %bb.5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:  @ %bb.5:
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI8_0:
 ; CHECK-NEXT:    .long 4294967280 @ 0xfffffff0
 ; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
 ; CHECK-NEXT:    .long 4294967288 @ 0xfffffff8
 ; CHECK-NEXT:    .long 4294967292 @ 0xfffffffc
 entry:

 
 for.cond.cleanup:                                 ; preds = %for.body, %middle.block, %entry
   ret void
 }
 
 define arm_aapcs_vfpcc void @gather_inc_v4i32_complex(ptr noalias nocapture readonly %data, ptr noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: gather_inc_v4i32_complex:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r2, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB9_1: @ %vector.ph.preheader
 ; CHECK-NEXT:    .save {r4, r5, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB9_5
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
 ; CHECK-NEXT:    bic r12, r2, #3
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    sub.w lr, r12, #4
 ; CHECK-NEXT:    adr r4, .LCPI9_1
 ; CHECK-NEXT:    adr r5, .LCPI9_2
 ; CHECK-NEXT:    vldrw.u32 q1, [r4]
 ; CHECK-NEXT:    add.w r3, r3, lr, lsr #2
 ; CHECK-NEXT:    adr.w lr, .LCPI9_0

 ; CHECK-NEXT:    vldrw.u32 q7, [q4, #48]!
 ; CHECK-NEXT:    vadd.i32 q6, q6, q7
 ; CHECK-NEXT:    vstrb.8 q6, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB9_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB9_2 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB9_2
-; CHECK-NEXT:  .LBB9_5: @ %for.cond.cleanup
+; CHECK-NEXT:  @ %bb.5:
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    pop.w {r4, r5, r7, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI9_0:
 ; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
 ; CHECK-NEXT:    .long 4294967260 @ 0xffffffdc
 ; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
 ; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
 ; CHECK-NEXT:  .LCPI9_1:

 
 for.cond.cleanup:                                 ; preds = %for.body, %middle.block, %entry
   ret void
 }
 
 define arm_aapcs_vfpcc void @gather_inc_v4i32_large(ptr noalias nocapture readonly %data, ptr noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: gather_inc_v4i32_large:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r4, pc}
+; CHECK-NEXT:    bxlt lr
 ; CHECK-NEXT:  .LBB10_1: @ %vector.ph.preheader
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    bic r12, r2, #3
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    sub.w lr, r12, #4
 ; CHECK-NEXT:    add.w r4, r3, lr, lsr #2
 ; CHECK-NEXT:    adr r3, .LCPI10_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
 ; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB10_2: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB10_3 Depth 2
 ; CHECK-NEXT:    dls lr, r4
 ; CHECK-NEXT:    mov r0, r1
 ; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:  .LBB10_3: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB10_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vldrw.u32 q2, [q1, #508]!
 ; CHECK-NEXT:    vstrb.8 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB10_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB10_2 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB10_2
-; CHECK-NEXT:  @ %bb.5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:  @ %bb.5:
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI10_0:
 ; CHECK-NEXT:    .long 4294966788 @ 0xfffffe04
 ; CHECK-NEXT:    .long 4294966792 @ 0xfffffe08
 ; CHECK-NEXT:    .long 4294966796 @ 0xfffffe0c
 ; CHECK-NEXT:    .long 4294966800 @ 0xfffffe10
 entry:

llvm/test/CodeGen/Thumb2/mve-gather-tailpred.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -enable-arm-maskedldst -tail-predication=force-enabled %s -o - -opaque-pointers | FileCheck %s
 
 define arm_aapcs_vfpcc void @gather_inc_v4i32_simple(ptr noalias nocapture readonly %data, ptr noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: gather_inc_v4i32_simple:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r4, pc}
+; CHECK-NEXT:    bxlt lr
 ; CHECK-NEXT:  .LBB0_1: @ %vector.ph.preheader
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    bic r12, r2, #3
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    sub.w lr, r12, #4
 ; CHECK-NEXT:    add.w r4, r3, lr, lsr #2
 ; CHECK-NEXT:    adr r3, .LCPI0_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
 ; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB0_2: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB0_3 Depth 2
 ; CHECK-NEXT:    dls lr, r4
 ; CHECK-NEXT:    mov r0, r1
 ; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:  .LBB0_3: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB0_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vldrw.u32 q2, [q1, #16]!
 ; CHECK-NEXT:    vstrb.8 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB0_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB0_2
-; CHECK-NEXT:  @ %bb.5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:  @ %bb.5:
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 4294967280 @ 0xfffffff0
 ; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
 ; CHECK-NEXT:    .long 4294967288 @ 0xfffffff8
 ; CHECK-NEXT:    .long 4294967292 @ 0xfffffffc
 entry:

llvm/test/CodeGen/Thumb2/mve-memtp-loop.ll

 entry:
   call void @llvm.memcpy.p0.p0.i32(ptr align 4 %X, ptr align 4 %Y, i32 %n, i1 false)
   ret void
 }
 
 define void @test11(ptr nocapture %x, ptr nocapture %y, i32 %n) {
 ; CHECK-LABEL: test11:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp.w r2, #-1
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    popgt {r4, pc}
+; CHECK-NEXT:    bxgt lr
 ; CHECK-NEXT:  .LBB10_1: @ %prehead
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    mov r12, r1
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    wlstp.8 lr, r2, .LBB10_3
 ; CHECK-NEXT:  .LBB10_2: @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q0, [r12], #16
 ; CHECK-NEXT:    vstrb.8 q0, [r4], #16
 ; CHECK-NEXT:    letp lr, .LBB10_2
 ; CHECK-NEXT:  .LBB10_3: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r3, [r0], #1
 ; CHECK-NEXT:    subs r2, #2
 ; CHECK-NEXT:    strb r3, [r1], #1
 ; CHECK-NEXT:    bne .LBB10_3
-; CHECK-NEXT:  @ %bb.4: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:  @ %bb.4:
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp6 = icmp slt i32 %n, 0
   br i1 %cmp6, label %prehead, label %for.cond.cleanup
 
 prehead:                                          ; preds = %entry
   call void @llvm.memcpy.p0.p0.i32(ptr align 4 %x, ptr align 4 %y, i32 %n, i1 false)
   br label %for.body
 

 }
 
 declare void @other()
 
 @arr_56 = external dso_local local_unnamed_addr global [21 x [16 x [11 x i8]]], align 1
 define void @multilooped_exit(i32 %b) {
 ; CHECK-LABEL: multilooped_exit:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r0, #1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    poplt {r4, pc}
+; CHECK-NEXT:    bxlt lr
 ; CHECK-NEXT:  .LBB18_1: @ %loop.preheader
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    mov.w r4, #-1
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    b .LBB18_3
 ; CHECK-NEXT:  .LBB18_2: @ %loop
 ; CHECK-NEXT:    @ in Loop: Header=BB18_3 Depth=1
 ; CHECK-NEXT:    adds r4, #1
 ; CHECK-NEXT:    cmp.w r4, #1024
 ; CHECK-NEXT:    bge .LBB18_12

 ; CHECK-NEXT:  @ %bb.10: @ %loop
 ; CHECK-NEXT:    @ in Loop: Header=BB18_3 Depth=1
 ; CHECK-NEXT:    dlstp.8 lr, r0
 ; CHECK-NEXT:  .LBB18_11: @ Parent Loop BB18_3 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    vstrb.8 q0, [r3], #16
 ; CHECK-NEXT:    letp lr, .LBB18_11
 ; CHECK-NEXT:    b .LBB18_2
-; CHECK-NEXT:  .LBB18_12: @ %exit
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:  .LBB18_12:
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %cmp8 = icmp sgt i32 %b, 0
   br i1 %cmp8, label %loop, label %exit
 
 loop:
   %p = phi i32 [ 0, %entry ], [ %inc, %loop ]
   call void @llvm.memset.p0.i32(ptr align 1 getelementptr ([21 x [16 x [11 x i8]]], ptr @arr_56, i32 0, i32 0, i32 undef, i32 0), i8 0, i32 %b, i1 false)
   call void @llvm.memset.p0.i32(ptr align 1 getelementptr ([21 x [16 x [11 x i8]]], ptr @arr_56, i32 0, i32 0, i32 undef, i32 0), i8 0, i32 %b, i1 false)

llvm/test/CodeGen/Thumb2/mve-postinc-dct.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp %s -o - | FileCheck %s
 
 %struct.DCT_InstanceTypeDef = type { ptr, i32, i32 }
 
 define void @DCT_mve1(ptr nocapture readonly %S, ptr nocapture readonly %pIn, ptr nocapture %pOut) {
 ; CHECK-LABEL: DCT_mve1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, lr}
-; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    ldr r3, [r0, #4]
 ; CHECK-NEXT:    sub.w r12, r3, #1
 ; CHECK-NEXT:    cmp.w r12, #2
-; CHECK-NEXT:    blo .LBB0_5
-; CHECK-NEXT:  @ %bb.1: @ %for.body.preheader
+; CHECK-NEXT:    it lo
+; CHECK-NEXT:    bxlo lr
+; CHECK-NEXT:  .LBB0_1: @ %for.body.preheader
+; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, lr}
+; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    ldr r5, [r0, #8]
 ; CHECK-NEXT:    ldr r3, [r0]
 ; CHECK-NEXT:    add.w r3, r3, r5, lsl #2
 ; CHECK-NEXT:    movs r0, #1
 ; CHECK-NEXT:    lsl.w r9, r5, #2
 ; CHECK-NEXT:  .LBB0_2: @ %for.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB0_3 Depth 2

 ; CHECK-NEXT:    add.w r7, r2, r0, lsl #2
 ; CHECK-NEXT:    vadd.f32 s0, s0, s1
 ; CHECK-NEXT:    adds r0, #1
 ; CHECK-NEXT:    add r3, r9
 ; CHECK-NEXT:    cmp r0, r12
 ; CHECK-NEXT:    vadd.f32 s0, s0, s2
 ; CHECK-NEXT:    vstr s0, [r7]
 ; CHECK-NEXT:    bne .LBB0_2
-; CHECK-NEXT:  .LBB0_5: @ %for.cond.cleanup
-; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
+; CHECK-NEXT:  @ %bb.5:
+; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %NumInputs = getelementptr inbounds %struct.DCT_InstanceTypeDef, ptr %S, i32 0, i32 2
   %i = load i32, ptr %NumInputs, align 4
   %NumFilters = getelementptr inbounds %struct.DCT_InstanceTypeDef, ptr %S, i32 0, i32 1
   %i1 = load i32, ptr %NumFilters, align 4
   %pDCTCoefs = getelementptr inbounds %struct.DCT_InstanceTypeDef, ptr %S, i32 0, i32 0
   %i2 = load ptr, ptr %pDCTCoefs, align 4
   %cmp = icmp ugt i32 %i, 1

llvm/test/CodeGen/Thumb2/mve-scatter-increment.ll

; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
   %2 = getelementptr inbounds i8, ptr %dst, <16 x i32> %1
   call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %data, <16 x ptr> %2, i32 2, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
   ret void
 }
 
 define arm_aapcs_vfpcc void @scatter_inc_v4i32_complex(<4 x i32> %data1, <4 x i32> %data2, <4 x i32> %data3, ptr %dst, i32 %n) {
 ; CHECK-LABEL: scatter_inc_v4i32_complex:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r1, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB3_1: @ %vector.ph.preheader
 ; CHECK-NEXT:    .save {r4, lr}
 ; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    .pad #16
 ; CHECK-NEXT:    sub sp, #16
-; CHECK-NEXT:    cmp r1, #1
-; CHECK-NEXT:    blt .LBB3_5
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
 ; CHECK-NEXT:    adr r4, .LCPI3_2
 ; CHECK-NEXT:    bic r2, r1, #3
 ; CHECK-NEXT:    vldrw.u32 q3, [r4]
 ; CHECK-NEXT:    sub.w r12, r2, #4
 ; CHECK-NEXT:    adr.w lr, .LCPI3_1
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    vadd.i32 q3, q3, r0
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #2

 ; CHECK-NEXT:    vstrw.32 q0, [q5, #48]!
 ; CHECK-NEXT:    vstrw.32 q1, [q6, #48]!
 ; CHECK-NEXT:    vstrw.32 q2, [q7, #48]!
 ; CHECK-NEXT:    le lr, .LBB3_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB3_2 Depth=1
 ; CHECK-NEXT:    cmp r2, r1
 ; CHECK-NEXT:    bne .LBB3_2
-; CHECK-NEXT:  .LBB3_5: @ %for.cond.cleanup
+; CHECK-NEXT:  @ %bb.5:
 ; CHECK-NEXT:    add sp, #16
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI3_0:
 ; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
 ; CHECK-NEXT:    .long 4294967260 @ 0xffffffdc
 ; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
 ; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
 ; CHECK-NEXT:  .LCPI3_1:

llvm/test/CodeGen/Thumb2/mve-tailpred-nonzerostart.ll

 for.cond.cleanup:                                 ; preds = %vector.body, %entry
   ret void
 }
 
 
 define arm_aapcs_vfpcc void @start11(ptr nocapture readonly %x, ptr nocapture readonly %y, ptr noalias nocapture %z, float %a, i32 %n) {
 ; CHECK-LABEL: start11:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB1_1: @ %vector.ph
 ; CHECK-NEXT:    .save {r4, r5, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    cmp r3, #1
-; CHECK-NEXT:    blt .LBB1_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.ph
 ; CHECK-NEXT:    vmov r12, s0
 ; CHECK-NEXT:    adds r4, r3, #3
 ; CHECK-NEXT:    adr r5, .LCPI1_0
 ; CHECK-NEXT:    bic lr, r4, #3
 ; CHECK-NEXT:    adds r0, #44
 ; CHECK-NEXT:    adds r1, #44
 ; CHECK-NEXT:    adds r2, #44
 ; CHECK-NEXT:    vldrw.u32 q0, [r5]
 ; CHECK-NEXT:    movs r4, #11
 ; CHECK-NEXT:    vdup.32 q1, r3
 ; CHECK-NEXT:  .LBB1_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vqadd.u32 q2, q0, r4
 ; CHECK-NEXT:    adds r4, #4
 ; CHECK-NEXT:    cmp lr, r4
 ; CHECK-NEXT:    vptt.u32 hi, q1, q2
 ; CHECK-NEXT:    vldrwt.u32 q2, [r1], #16
 ; CHECK-NEXT:    vldrwt.u32 q3, [r0], #16
 ; CHECK-NEXT:    vfmas.f32 q3, q2, r12
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q3, [r2], #16
 ; CHECK-NEXT:    bne .LBB1_2
-; CHECK-NEXT:  .LBB1_3: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:  @ %bb.3:
+; CHECK-NEXT:    pop.w {r4, r5, r7, lr}
+; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.4:
 ; CHECK-NEXT:  .LCPI1_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 1 @ 0x1
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 3 @ 0x3
 entry:

llvm/test/CodeGen/Thumb2/mve-vmull-loop.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=CHECK
 
 define arm_aapcs_vfpcc void @test32(ptr noalias nocapture readonly %x, ptr noalias nocapture readonly %y, ptr nocapture %z, i32 %n) {
 ; CHECK-LABEL: test32:
 ; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    cmp r3, #1
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:  .LBB0_1: @ %vector.body.preheader
 ; CHECK-NEXT:    .save {r4, r5, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    cmp r3, #1
-; CHECK-NEXT:    blt .LBB0_2
-; CHECK-NEXT:  .LBB0_1: @ %vector.body
+; CHECK-NEXT:  .LBB0_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs r3, #4
 ; CHECK-NEXT:    vmullb.s32 q2, q1, q0
 ; CHECK-NEXT:    vmullt.s32 q3, q1, q0
 ; CHECK-NEXT:    vmov r12, r5, d5
 ; CHECK-NEXT:    lsrl r12, r5, #31
 ; CHECK-NEXT:    vmov r4, r5, d4
 ; CHECK-NEXT:    lsrl r4, r5, #31
 ; CHECK-NEXT:    vmov q2[2], q2[0], r4, r12
 ; CHECK-NEXT:    vmov r12, r5, d7
 ; CHECK-NEXT:    lsrl r12, r5, #31
 ; CHECK-NEXT:    vmov r4, r5, d6
 ; CHECK-NEXT:    lsrl r4, r5, #31
 ; CHECK-NEXT:    vmov q2[3], q2[1], r4, r12
 ; CHECK-NEXT:    vstrb.8 q2, [r2], #16
-; CHECK-NEXT:    bne .LBB0_1
-; CHECK-NEXT:  .LBB0_2: @ %for.cond.cleanup
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    bne .LBB0_2
+; CHECK-NEXT:  @ %bb.3:
+; CHECK-NEXT:    pop.w {r4, r5, r7, lr}
+; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 3
   %cmp = icmp eq i32 %0, 0
   %cmp113 = icmp sgt i32 %n, 0
   br i1 %cmp113, label %vector.body, label %for.cond.cleanup
 
 vector.body:                                      ; preds = %vector.body, %entry
   %index = phi i32 [ %index.next, %vector.body ], [ 0, %entry ]

llvm/test/CodeGen/X86/fold-call-3.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-apple-darwin | FileCheck %s
 ; rdar://6522427
 ; This command line used to crash due to dangling nodes left after PreprocessISelDAG
 ; RUN: llc < %s -mtriple=x86_64-apple-darwin -pre-RA-sched=linearize | FileCheck %s --check-prefix=pre-RA
 
   %"struct.clang::Action" = type { %"struct.clang::ActionBase" }
   %"struct.clang::ActionBase" = type { ptr }
   %"struct.clang::ActionBase::ActionResult<0u>" = type { ptr, i8 }
 @NumTrials = internal global i32 10000000   ; <ptr> [#uses=2]
 @llvm.used = appending global [1 x ptr] [ ptr @_Z25RawPointerPerformanceTestPvRN5clang6ActionE ], section "llvm.metadata"    ; <ptr> [#uses=0]
 
 define void @_Z25RawPointerPerformanceTestPvRN5clang6ActionE(ptr %Val, ptr %Actions) nounwind {
 ; CHECK-LABEL: _Z25RawPointerPerformanceTestPvRN5clang6ActionE:
 ; CHECK:       ## %bb.0: ## %entry
+; CHECK-NEXT:    cmpl $0, _NumTrials(%rip)
+; CHECK-NEXT:    je LBB0_4
+; CHECK-NEXT:  ## %bb.1: ## %bb.nph
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    subq $24, %rsp
-; CHECK-NEXT:    cmpl $0, _NumTrials(%rip)
-; CHECK-NEXT:    je LBB0_3
-; CHECK-NEXT:  ## %bb.1: ## %bb.nph
 ; CHECK-NEXT:    movq %rsi, %rbx
 ; CHECK-NEXT:    movq %rdi, %rax
 ; CHECK-NEXT:    xorl %ebp, %ebp
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  LBB0_2: ## %bb
 ; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    movq (%rbx), %rcx
 ; CHECK-NEXT:    movq %rbx, %rdi
 ; CHECK-NEXT:    movq %rax, %rsi
 ; CHECK-NEXT:    callq *560(%rcx)
 ; CHECK-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    incl %ebp
 ; CHECK-NEXT:    cmpl _NumTrials(%rip), %ebp
 ; CHECK-NEXT:    jb LBB0_2
-; CHECK-NEXT:  LBB0_3: ## %return
+; CHECK-NEXT:  ## %bb.3:
 ; CHECK-NEXT:    addq $24, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:  LBB0_4: ## %return
 ; CHECK-NEXT:    retq
 ;
 ; pre-RA-LABEL: _Z25RawPointerPerformanceTestPvRN5clang6ActionE:
 ; pre-RA:       ## %bb.0: ## %entry
+; pre-RA-NEXT:    cmpl $0, _NumTrials(%rip)
+; pre-RA-NEXT:    je LBB0_4
+; pre-RA-NEXT:  ## %bb.1: ## %bb.nph
 ; pre-RA-NEXT:    pushq %rbp
 ; pre-RA-NEXT:    pushq %rbx
 ; pre-RA-NEXT:    subq $24, %rsp
-; pre-RA-NEXT:    cmpl $0, _NumTrials(%rip)
-; pre-RA-NEXT:    je LBB0_3
-; pre-RA-NEXT:  ## %bb.1: ## %bb.nph
 ; pre-RA-NEXT:    movq %rsi, %rbx
 ; pre-RA-NEXT:    movq %rdi, %rax
 ; pre-RA-NEXT:    xorl %ebp, %ebp
 ; pre-RA-NEXT:    .p2align 4, 0x90
 ; pre-RA-NEXT:  LBB0_2: ## %bb
 ; pre-RA-NEXT:    ## =>This Inner Loop Header: Depth=1
 ; pre-RA-NEXT:    movq (%rbx), %rcx
 ; pre-RA-NEXT:    movq %rbx, %rdi
 ; pre-RA-NEXT:    movq %rax, %rsi
 ; pre-RA-NEXT:    callq *560(%rcx)
 ; pre-RA-NEXT:    incl %ebp
 ; pre-RA-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; pre-RA-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
 ; pre-RA-NEXT:    cmpl _NumTrials(%rip), %ebp
 ; pre-RA-NEXT:    jb LBB0_2
-; pre-RA-NEXT:  LBB0_3: ## %return
+; pre-RA-NEXT:  ## %bb.3:
 ; pre-RA-NEXT:    addq $24, %rsp
 ; pre-RA-NEXT:    popq %rbx
 ; pre-RA-NEXT:    popq %rbp
+; pre-RA-NEXT:  LBB0_4: ## %return
 ; pre-RA-NEXT:    retq
 entry:
   %i = alloca %"struct.clang::ActionBase::ActionResult<0u>", align 8
   %i1 = load i32, ptr @NumTrials, align 4
   %i2 = icmp eq i32 %i1, 0
   br i1 %i2, label %return, label %bb.nph
 
 bb.nph:                                           ; preds = %entry

llvm/test/CodeGen/X86/negative-stride-fptosi-user.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
 
 ; LSR previously eliminated the sitofp by introducing an induction
 ; variable which stepped by a bogus ((double)UINT32_C(-1)). It's theoretically
 ; possible to eliminate the sitofp using a proper -1.0 step though; this
 ; test should be changed if that is done.
 
 define void @foo(i32 %N) nounwind {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    testl %edi, %edi
+; CHECK-NEXT:    js .LBB0_1
+; CHECK-NEXT:  # %bb.4: # %return
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  .LBB0_1: # %bb.preheader
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    pushq %rax
-; CHECK-NEXT:    testl %edi, %edi
-; CHECK-NEXT:    jns .LBB0_3
-; CHECK-NEXT:  # %bb.1: # %bb.preheader
 ; CHECK-NEXT:    movl %edi, %ebx
 ; CHECK-NEXT:    xorl %ebp, %ebp
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_2: # %bb
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    xorps %xmm0, %xmm0
 ; CHECK-NEXT:    cvtsi2sd %ebp, %xmm0
 ; CHECK-NEXT:    callq bar@PLT
 ; CHECK-NEXT:    decl %ebp
 ; CHECK-NEXT:    cmpl %ebp, %ebx
 ; CHECK-NEXT:    jne .LBB0_2
-; CHECK-NEXT:  .LBB0_3: # %return
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    addq $8, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    retq
 entry:
   %0 = icmp slt i32 %N, 0                         ; <i1> [#uses=1]
   br i1 %0, label %bb, label %return
 

llvm/test/CodeGen/X86/pr44412.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ;RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 define void @bar(i32 %0, i32 %1) nounwind {
 ; CHECK-LABEL: bar:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    testl %edi, %edi
-; CHECK-NEXT:    je .LBB0_3
+; CHECK-NEXT:    je .LBB0_4
 ; CHECK-NEXT:  # %bb.1: # %.preheader
+; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    movl %edi, %ebx
 ; CHECK-NEXT:    decl %ebx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_2: # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    movl %ebx, %edi
 ; CHECK-NEXT:    callq foo@PLT
 ; CHECK-NEXT:    addl $-1, %ebx
 ; CHECK-NEXT:    jb .LBB0_2
-; CHECK-NEXT:  .LBB0_3:
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    retq
   %3 = icmp eq i32 %0, 0
   br i1 %3, label %8, label %4
 
 4:                                                ; preds = %2, %4
   %5 = phi i32 [ %6, %4 ], [ %0, %2 ]
   %6 = add nsw i32 %5, -1
   tail call void @foo(i32 %6)
   %7 = icmp eq i32 %6, 0
   br i1 %7, label %8, label %4
 
 8:                                                ; preds = %4, %2
   ret void
 }
 
 define void @baz(i32 %0, i32 %1) nounwind {
 ; CHECK-LABEL: baz:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    testl %edi, %edi
-; CHECK-NEXT:    je .LBB1_3
+; CHECK-NEXT:    je .LBB1_4
 ; CHECK-NEXT:  # %bb.1: # %.preheader
+; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    movl %edi, %ebx
 ; CHECK-NEXT:    decl %ebx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB1_2: # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    movl %ebx, %edi
 ; CHECK-NEXT:    callq foo@PLT
 ; CHECK-NEXT:    addl $-1, %ebx
 ; CHECK-NEXT:    jae .LBB1_2
-; CHECK-NEXT:  .LBB1_3:
+; CHECK-NEXT:  # %bb.3:
 ; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:  .LBB1_4:
 ; CHECK-NEXT:    retq
   %3 = icmp eq i32 %0, 0
   br i1 %3, label %8, label %4
 
 4:                                                ; preds = %2, %4
   %5 = phi i32 [ %6, %4 ], [ %0, %2 ]
   %6 = add nsw i32 %5, -1
   tail call void @foo(i32 %6)
   %7 = icmp ne i32 %6, 0
   br i1 %7, label %8, label %4
 
 8:                                                ; preds = %4, %2
   ret void
 }
 
 declare void @foo(i32)

llvm/test/CodeGen/X86/x86-shrink-wrapping.ll

 %struct.rtx_def = type { i32, [1 x %union.rtunion_def] }
 %union.rtunion_def = type { i64 }
 
 declare hidden fastcc ptr @find_temp_slot_from_address(ptr readonly)
 
 define void @useLEA(ptr readonly %x) {
 ; ENABLE-LABEL: useLEA:
 ; ENABLE:       ## %bb.0: ## %entry
-; ENABLE-NEXT:    pushq %rax
-; ENABLE-NEXT:    .cfi_def_cfa_offset 16
 ; ENABLE-NEXT:    testq %rdi, %rdi
-; ENABLE-NEXT:    je LBB8_7
+; ENABLE-NEXT:    je LBB8_9
 ; ENABLE-NEXT:  ## %bb.1: ## %if.end
 ; ENABLE-NEXT:    cmpw $66, (%rdi)
-; ENABLE-NEXT:    jne LBB8_7
+; ENABLE-NEXT:    jne LBB8_9
 ; ENABLE-NEXT:  ## %bb.2: ## %lor.lhs.false
+; ENABLE-NEXT:    pushq %rax
+; ENABLE-NEXT:    .cfi_def_cfa_offset 16
 ; ENABLE-NEXT:    movq 8(%rdi), %rdi
 ; ENABLE-NEXT:    movzwl (%rdi), %eax
 ; ENABLE-NEXT:    leal -54(%rax), %ecx
 ; ENABLE-NEXT:    cmpl $14, %ecx
 ; ENABLE-NEXT:    ja LBB8_3
-; ENABLE-NEXT:  ## %bb.8: ## %lor.lhs.false
+; ENABLE-NEXT:  ## %bb.7: ## %lor.lhs.false
 ; ENABLE-NEXT:    movl $24599, %edx ## imm = 0x6017
 ; ENABLE-NEXT:    btl %ecx, %edx
 ; ENABLE-NEXT:    jae LBB8_3
-; ENABLE-NEXT:  LBB8_7: ## %cleanup
-; ENABLE-NEXT:    popq %rax
+; ENABLE-NEXT:  LBB8_8:
+; ENABLE-NEXT:    addq $8, %rsp
+; ENABLE-NEXT:  LBB8_9: ## %cleanup
 ; ENABLE-NEXT:    retq
 ; ENABLE-NEXT:  LBB8_3: ## %lor.lhs.false
 ; ENABLE-NEXT:    cmpl $134, %eax
-; ENABLE-NEXT:    je LBB8_7
+; ENABLE-NEXT:    je LBB8_8
 ; ENABLE-NEXT:  ## %bb.4: ## %lor.lhs.false
 ; ENABLE-NEXT:    cmpl $140, %eax
-; ENABLE-NEXT:    je LBB8_7
+; ENABLE-NEXT:    je LBB8_8
 ; ENABLE-NEXT:  ## %bb.5: ## %if.end.55
 ; ENABLE-NEXT:    callq _find_temp_slot_from_address
 ; ENABLE-NEXT:    testq %rax, %rax
-; ENABLE-NEXT:    je LBB8_7
+; ENABLE-NEXT:    je LBB8_8
 ; ENABLE-NEXT:  ## %bb.6: ## %if.then.60
 ; ENABLE-NEXT:    movb $1, 57(%rax)
-; ENABLE-NEXT:    popq %rax
-; ENABLE-NEXT:    retq
+; ENABLE-NEXT:    jmp LBB8_8
 ;
 ; DISABLE-LABEL: useLEA:
 ; DISABLE:       ## %bb.0: ## %entry
 ; DISABLE-NEXT:    pushq %rax
 ; DISABLE-NEXT:    .cfi_def_cfa_offset 16
 ; DISABLE-NEXT:    testq %rdi, %rdi
 ; DISABLE-NEXT:    je LBB8_7
 ; DISABLE-NEXT:  ## %bb.1: ## %if.end

llvm/test/Transforms/LoopStrengthReduce/AArch64/pr53625.ll

 ; CHECK-NEXT:  // %bb.3: // %for.cond
 ; CHECK-NEXT:    // in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    subs x8, x8, #1
 ; CHECK-NEXT:    b.ne .LBB0_2
 ; CHECK-NEXT:  .LBB0_4:
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_5:
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    mov w0, #1 // =0x1
 ; CHECK-NEXT:    ret
 entry:
   %cmp13 = icmp sgt i32 %c, 0
   br i1 %cmp13, label %for.body.preheader, label %return
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %c to i64
   br label %for.body

 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    cmp w0, #1
 ; CHECK-NEXT:    b.lt .LBB1_5
 ; CHECK-NEXT:  // %bb.1: // %for.body.preheader
 ; CHECK-NEXT:    mov x8, xzr
 ; CHECK-NEXT:    mov w9, w0
 ; CHECK-NEXT:    add x10, x1, #4
 ; CHECK-NEXT:    add x11, x2, #8
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    mov w0, #1 // =0x1
 ; CHECK-NEXT:  .LBB1_2: // %for.body
 ; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr w12, [x10, x8, lsl #2]
 ; CHECK-NEXT:    cbnz w12, .LBB1_7
 ; CHECK-NEXT:  // %bb.3: // %if.then
 ; CHECK-NEXT:    // in Loop: Header=BB1_2 Depth=1
 ; CHECK-NEXT:    ldr w12, [x11, x8, lsl #2]
 ; CHECK-NEXT:    cbnz w12, .LBB1_6

 ; CHECK-NEXT:    // in Loop: Header=BB2_2 Depth=1
 ; CHECK-NEXT:    add x1, x1, #4
 ; CHECK-NEXT:    subs x8, x8, #1
 ; CHECK-NEXT:    b.ne .LBB2_2
 ; CHECK-NEXT:  .LBB2_4:
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB2_5:
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    mov w0, #1 // =0x1
 ; CHECK-NEXT:    ret
 entry:
   %cmp13 = icmp sgt i32 %c, 0
   br i1 %cmp13, label %for.body.preheader, label %return
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %c to i64
   br label %for.body

llvm/test/Transforms/LoopStrengthReduce/X86/ivchain-X86.ll

 ; strange increment expressions like this:
 ; IV + ((sext i32 (2 * %s) to i64) + (-1 * (sext i32 %s to i64)))
 ;
 ; For x32, no spills in the preheader, no complex address modes, no reloads.
 
 define void @extrastride(i8* nocapture %main, i32 %main_stride, i32* nocapture %res, i32 %x, i32 %y, i32 %z) nounwind {
 ; X64-LABEL: extrastride:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rbx
 ; X64-NEXT:    # kill: def $ecx killed $ecx def $rcx
 ; X64-NEXT:    # kill: def $esi killed $esi def $rsi
 ; X64-NEXT:    testl %r9d, %r9d
-; X64-NEXT:    je .LBB2_3
+; X64-NEXT:    je .LBB2_4
 ; X64-NEXT:  # %bb.1: # %for.body.lr.ph
+; X64-NEXT:    pushq %rbx
 ; X64-NEXT:    leal (%rsi,%rsi), %r10d
 ; X64-NEXT:    leal (%rsi,%rsi,2), %r11d
 ; X64-NEXT:    addl %esi, %ecx
 ; X64-NEXT:    leal (,%rsi,4), %eax
 ; X64-NEXT:    leal (%rcx,%rsi,4), %ebx
 ; X64-NEXT:    cltq
 ; X64-NEXT:    movslq %r11d, %rcx
 ; X64-NEXT:    movslq %r10d, %r10

 ; X64-NEXT:    addl (%rdi,%r10), %ebx
 ; X64-NEXT:    addl (%rdi,%rcx), %ebx
 ; X64-NEXT:    addl (%rdi,%rax), %ebx
 ; X64-NEXT:    movl %ebx, (%rdx)
 ; X64-NEXT:    addq %r11, %rdi
 ; X64-NEXT:    addq %r8, %rdx
 ; X64-NEXT:    decl %r9d
 ; X64-NEXT:    jne .LBB2_2
-; X64-NEXT:  .LBB2_3: # %for.end
+; X64-NEXT:  # %bb.3:
 ; X64-NEXT:    popq %rbx
+; X64-NEXT:  .LBB2_4: # %for.end
 ; X64-NEXT:    retq
 ;
 ; X32-LABEL: extrastride:
 ; X32:       # %bb.0: # %entry
 ; X32-NEXT:    pushl %ebp
 ; X32-NEXT:    pushl %ebx
 ; X32-NEXT:    pushl %edi
 ; X32-NEXT:    pushl %esi