[ShrinkWrap] Use underlying object to rule out stack access.

[fhahn]

Summary

Allow shrink-wrapping past memory accesses that only access globals or
function arguments. This patch uses getUnderlyingObject to try to
identify the accessed object by a given memory operand. If it is a
global or an argument, it does not access the stack of the current
function and should not block shrink wrapping.

Note that the caller's stack may get accessed when passing an argument
via the stack, but not the stack of the current function.

This addresses part of the TODO from D63152.

[thegameg]

lgtm thanks

llvm/lib/CodeGen/ShrinkWrap.cpp

271

Unused?

[efriedma]

So the idea here is that things that are allocated before entry to the function can't point into the stack of the current function? The basic idea makes sense.

I'm a little concerned about the interaction with byval/preallocated/inalloca arguments, which are, depending on the calling convention, arguably part of the current function's stack. Any thoughts there?

[fhahn]

Update remaining failing tests, remove dead Objs variable.

In D149668#4313246, @efriedma wrote:

So the idea here is that things that are allocated before entry to the function can't point into the stack of the current function? The basic idea makes sense.

I'm a little concerned about the interaction with byval/preallocated/inalloca arguments, which are, depending on the calling convention, arguably part of the current function's stack. Any thoughts there?

Thanks, I added a !Arg->hasPassPointeeByValueCopyAttr() check to catch those. Tests for with those attributes have been added in bc1c95d973b80d7ec7a15d4c5c5248952fc39153.

[fhahn]

llvm/lib/CodeGen/ShrinkWrap.cpp

271

Yep, removed, thanks!

[jpenix-quic]

llvm/lib/CodeGen/ShrinkWrap.cpp

286

Hi @fhahn, does this work as expected when MI.memoperands() is empty? If I understand correctly, all_of will return true in that case which I don't think is what we want (my understanding is we need to be conservative if memoperands() is empty). That said, I struggled to find a test where this causes an issue here, so I'm not sure if this is actually an issue in practice or if any possible situations are covered by the other checks, etc.

Do you have thoughts on this? If it is an issue, I imagine we could just add an extra check for the empty case or something similar.

[fhahn]

llvm/lib/CodeGen/ShrinkWrap.cpp

286

@jpenix-quic I had a closer look and added a case where missing memoperands cause incorrect results. It's a bit artificial as memory operands are dropped from instructions that usually have them, but it illustrates the problem: ac9e9594af37

I also pushed a fix to treat instructions without memoperands conservatively: d0718ff410cc

[jpenix-quic]

llvm/lib/CodeGen/ShrinkWrap.cpp

286

Great, thank you for looking into and handling this!

llvm/lib/CodeGen/ShrinkWrap.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/PostOrderIterator.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/CFG.h"
+#include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"

 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTree)
 INITIALIZE_PASS_DEPENDENCY(MachinePostDominatorTree)
 INITIALIZE_PASS_DEPENDENCY(MachineLoopInfo)
 INITIALIZE_PASS_DEPENDENCY(MachineOptimizationRemarkEmitterPass)
 INITIALIZE_PASS_END(ShrinkWrap, DEBUG_TYPE, "Shrink Wrap Pass", false, false)
 
 bool ShrinkWrap::useOrDefCSROrFI(const MachineInstr &MI,
                                  RegScavenger *RS) const {
+  /// Check if \p Op is known to access an address not on the function's stack .
+  /// At the moment, accesses where the underlying object is a global or a
+  /// function argument are considered non-stack accesses. Note that the
+  /// caller's stack may get accessed when passing an argument via the stack,
+  /// but not the stack of the current function.
+  ///
+  auto IsKnownNonStackPtr = [](MachineMemOperand *Op) {
+    if (Op->getValue()) {
+      const Value *UO = getUnderlyingObject(Op->getValue());
+      if (!UO)
+        return false;
+      if (auto *Arg = dyn_cast<Argument>(UO))
+        return !Arg->hasPassPointeeByValueCopyAttr();
+      return isa<GlobalValue>(UO);
+    }
+    return false;
+  };
   // This prevents premature stack popping when occurs a indirect stack
-  // access. It is overly aggressive for the moment.
-  // TODO: - Obvious non-stack loads and store, such as global values,
-  //         are known to not access the stack.
+  // access.  It is overly aggressive for the moment.
+  // TODO:
   //       - Further, data dependency and alias analysis can validate
   //         that load and stores never derive from the stack pointer.
-  if (MI.mayLoadOrStore())
+  if (MI.mayLoadOrStore() && (MI.isCall() || MI.hasUnmodeledSideEffects() ||
+                              !all_of(MI.memoperands(), IsKnownNonStackPtr)))
     return true;
 
   if (MI.getOpcode() == FrameSetupOpcode ||
       MI.getOpcode() == FrameDestroyOpcode) {
     LLVM_DEBUG(dbgs() << "Frame instruction: " << MI << '\n');
     return true;
   }
   const MachineFunction *MF = MI.getParent()->getParent();

llvm/test/CodeGen/AArch64/addsub.ll

 @jiffies = dso_local local_unnamed_addr global i32 0, align 4
 @primary_crng = dso_local local_unnamed_addr global i32 0, align 4
 @input_pool = dso_local global i32 0, align 4
 declare dso_local i32 @crng_reseed(...) local_unnamed_addr
 ; Function Attrs: nounwind uwtable
 define dso_local i32 @_extract_crng_crng() {
 ; CHECK-LABEL: _extract_crng_crng:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    adrp x8, _extract_crng_crng
 ; CHECK-NEXT:    add x8, x8, :lo12:_extract_crng_crng
 ; CHECK-NEXT:    tbnz x8, #63, .LBB36_2
 ; CHECK-NEXT:  // %bb.1: // %lor.lhs.false
 ; CHECK-NEXT:    adrp x9, jiffies
 ; CHECK-NEXT:    ldrsw x9, [x9, :lo12:jiffies]
 ; CHECK-NEXT:    sub x8, x8, x9
 ; CHECK-NEXT:    add x8, x8, #18, lsl #12 // =73728
 ; CHECK-NEXT:    cmn x8, #1272
 ; CHECK-NEXT:    b.pl .LBB36_3
 ; CHECK-NEXT:  .LBB36_2: // %if.then
+; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    adrp x8, primary_crng
 ; CHECK-NEXT:    adrp x9, input_pool
 ; CHECK-NEXT:    add x9, x9, :lo12:input_pool
 ; CHECK-NEXT:    ldr w8, [x8, :lo12:primary_crng]
 ; CHECK-NEXT:    cmp w8, #0
 ; CHECK-NEXT:    csel x0, xzr, x9, eq
 ; CHECK-NEXT:    bl crng_reseed
-; CHECK-NEXT:  .LBB36_3: // %if.end
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
+; CHECK-NEXT:  .LBB36_3: // %if.end
+; CHECK-NEXT:    .cfi_def_cfa wsp, 0
+; CHECK-NEXT:    .cfi_same_value w30
 ; CHECK-NEXT:    ret
 entry:
   br i1 icmp slt (ptr @_extract_crng_crng, ptr null), label %if.then, label %lor.lhs.false
 lor.lhs.false:                                    ; preds = %entry
   %0 = load i32, ptr @jiffies, align 4
   %idx.ext = sext i32 %0 to i64
   %idx.neg = sub nsw i64 0, %idx.ext
   %add.ptr = getelementptr i8, ptr getelementptr (i8, ptr @_extract_crng_crng, i64 75000), i64 %idx.neg

llvm/test/CodeGen/AArch64/arm64-shrink-wrapping.ll

end:
   ret void
 }
 
 ; Re-aligned stack pointer.  See bug 26642.  Avoid clobbering live
 ; values in the prologue when re-aligning the stack pointer.
 define i32 @stack_realign(i32 %a, i32 %b, ptr %ptr1, ptr %ptr2) {
 ; ENABLE-LABEL: stack_realign:
 ; ENABLE:       ; %bb.0:
+; ENABLE-NEXT:    lsl w8, w0, w1
+; ENABLE-NEXT:    lsl w9, w1, w0
+; ENABLE-NEXT:    cmp w0, w1
+; ENABLE-NEXT:    b.ge LBB13_2
+; ENABLE-NEXT:  ; %bb.1: ; %true
 ; ENABLE-NEXT:    stp x29, x30, [sp, #-16]! ; 16-byte Folded Spill
 ; ENABLE-NEXT:    .cfi_def_cfa_offset 16
 ; ENABLE-NEXT:    mov x29, sp
 ; ENABLE-NEXT:    .cfi_def_cfa w29, 16
 ; ENABLE-NEXT:    .cfi_offset w30, -8
 ; ENABLE-NEXT:    .cfi_offset w29, -16
-; ENABLE-NEXT:    sub x9, sp, #16
-; ENABLE-NEXT:    and sp, x9, #0xffffffffffffffe0
-; ENABLE-NEXT:    lsl w8, w0, w1
-; ENABLE-NEXT:    lsl w9, w1, w0
-; ENABLE-NEXT:    cmp w0, w1
-; ENABLE-NEXT:    b.ge LBB13_2
-; ENABLE-NEXT:  ; %bb.1: ; %true
+; ENABLE-NEXT:    sub x1, sp, #16
+; ENABLE-NEXT:    and sp, x1, #0xffffffffffffffe0
 ; ENABLE-NEXT:    str w0, [sp]
+; ENABLE-NEXT:    mov sp, x29
+; ENABLE-NEXT:    ldp x29, x30, [sp], #16 ; 16-byte Folded Reload
 ; ENABLE-NEXT:  LBB13_2: ; %false
+; ENABLE-NEXT:    .cfi_def_cfa wsp, 0
+; ENABLE-NEXT:    .cfi_same_value w30
+; ENABLE-NEXT:    .cfi_same_value w29
 ; ENABLE-NEXT:    str w8, [x2]
 ; ENABLE-NEXT:    str w9, [x3]
-; ENABLE-NEXT:    mov sp, x29
-; ENABLE-NEXT:    ldp x29, x30, [sp], #16 ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ret
 ;
 ; DISABLE-LABEL: stack_realign:
 ; DISABLE:       ; %bb.0:
 ; DISABLE-NEXT:    stp x29, x30, [sp, #-16]! ; 16-byte Folded Spill
 ; DISABLE-NEXT:    .cfi_def_cfa_offset 16
 ; DISABLE-NEXT:    mov x29, sp
 ; DISABLE-NEXT:    .cfi_def_cfa w29, 16

llvm/test/CodeGen/AArch64/combine-comparisons-by-cse.ll

 ; cinc w0, w19, gt
 ; ...
 ; fcmp d8, #0.0
 ; b.gt .LBB0_5
 
 define i32 @fcmpri(i32 %argc, ptr nocapture readonly %argv) #0 {
 ; CHECK-LABEL: fcmpri:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    str d8, [sp, #-32]! // 8-byte Folded Spill
-; CHECK-NEXT:    .cfi_def_cfa_offset 32
-; CHECK-NEXT:    stp x30, x19, [sp, #16] // 16-byte Folded Spill
-; CHECK-NEXT:    .cfi_offset w19, -8
-; CHECK-NEXT:    .cfi_offset w30, -16
-; CHECK-NEXT:    .cfi_offset b8, -32
 ; CHECK-NEXT:    cmp w0, #2
 ; CHECK-NEXT:    b.lt .LBB9_3
 ; CHECK-NEXT:  // %bb.1: // %land.lhs.true
 ; CHECK-NEXT:    ldr x8, [x1, #8]
 ; CHECK-NEXT:    cbz x8, .LBB9_3
 ; CHECK-NEXT:  // %bb.2:
-; CHECK-NEXT:    mov w0, #3
-; CHECK-NEXT:    b .LBB9_4
+; CHECK-NEXT:    mov w0, #3 // =0x3
+; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB9_3: // %if.end
-; CHECK-NEXT:    mov w0, #1
+; CHECK-NEXT:    str d8, [sp, #-32]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    stp x30, x19, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
+; CHECK-NEXT:    .cfi_offset b8, -32
+; CHECK-NEXT:    mov w0, #1 // =0x1
 ; CHECK-NEXT:    bl zoo
 ; CHECK-NEXT:    mov w19, w0
-; CHECK-NEXT:    mov w0, #-1
+; CHECK-NEXT:    mov w0, #-1 // =0xffffffff
 ; CHECK-NEXT:    bl yoo
 ; CHECK-NEXT:    cmp w19, #0
-; CHECK-NEXT:    mov w1, #2
+; CHECK-NEXT:    mov w1, #2 // =0x2
 ; CHECK-NEXT:    cinc w0, w19, gt
 ; CHECK-NEXT:    fmov d8, d0
 ; CHECK-NEXT:    bl xoo
 ; CHECK-NEXT:    fmov d0, #-1.00000000
 ; CHECK-NEXT:    fcmp d8, #0.0
 ; CHECK-NEXT:    fmov d1, #-2.00000000
 ; CHECK-NEXT:    fadd d0, d8, d0
 ; CHECK-NEXT:    fcsel d0, d8, d0, gt
 ; CHECK-NEXT:    bl woo
-; CHECK-NEXT:    mov w0, #4
-; CHECK-NEXT:  .LBB9_4: // %return
 ; CHECK-NEXT:    ldp x30, x19, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    mov w0, #4 // =0x4
 ; CHECK-NEXT:    ldr d8, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    .cfi_restore b8
 ; CHECK-NEXT:    ret
 
 ; CHECK-LABEL-DAG: .LBB9_3

llvm/test/CodeGen/AArch64/csr-split.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=aarch64-unknown-linux < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mtriple=arm64-apple-ios < %s | FileCheck %s --check-prefix=CHECK-APPLE
 
 ; Check CSR split can work properly for tests below.
 
 @a = common dso_local local_unnamed_addr global i32 0, align 4
 
 define dso_local signext i32 @test1(ptr %b) local_unnamed_addr uwtable  {
 ; CHECK-LABEL: test1:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset w19, -8
-; CHECK-NEXT:    .cfi_offset w30, -16
-; CHECK-NEXT:    .cfi_remember_state
 ; CHECK-NEXT:    adrp x8, a
 ; CHECK-NEXT:    ldrsw x8, [x8, :lo12:a]
 ; CHECK-NEXT:    cmp x8, x0
 ; CHECK-NEXT:    b.eq .LBB0_2
 ; CHECK-NEXT:  // %bb.1: // %if.end
-; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
-; CHECK-NEXT:    .cfi_def_cfa_offset 0
-; CHECK-NEXT:    .cfi_restore w19
-; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB0_2: // %if.then
-; CHECK-NEXT:    .cfi_restore_state
+; CHECK-NEXT:    stp x30, x19, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w30, -16
 ; CHECK-NEXT:    mov x19, x0
 ; CHECK-NEXT:    bl callVoid
 ; CHECK-NEXT:    mov x0, x19
 ; CHECK-NEXT:    ldp x30, x19, [sp], #16 // 16-byte Folded Reload
 ; CHECK-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-NEXT:    .cfi_restore w19
 ; CHECK-NEXT:    .cfi_restore w30
 ; CHECK-NEXT:    b callNonVoid
 ;
 ; CHECK-APPLE-LABEL: test1:
 ; CHECK-APPLE:       ; %bb.0: ; %entry
-; CHECK-APPLE-NEXT:    stp x20, x19, [sp, #-32]! ; 16-byte Folded Spill
-; CHECK-APPLE-NEXT:    .cfi_def_cfa_offset 32
-; CHECK-APPLE-NEXT:    stp x29, x30, [sp, #16] ; 16-byte Folded Spill
-; CHECK-APPLE-NEXT:    .cfi_offset w30, -8
-; CHECK-APPLE-NEXT:    .cfi_offset w29, -16
-; CHECK-APPLE-NEXT:    .cfi_offset w19, -24
-; CHECK-APPLE-NEXT:    .cfi_offset w20, -32
-; CHECK-APPLE-NEXT:    .cfi_remember_state
 ; CHECK-APPLE-NEXT:  Lloh0:
 ; CHECK-APPLE-NEXT:    adrp x8, _a@PAGE
 ; CHECK-APPLE-NEXT:  Lloh1:
 ; CHECK-APPLE-NEXT:    ldrsw x8, [x8, _a@PAGEOFF]
 ; CHECK-APPLE-NEXT:    cmp x8, x0
 ; CHECK-APPLE-NEXT:    b.eq LBB0_2
 ; CHECK-APPLE-NEXT:  ; %bb.1: ; %if.end
-; CHECK-APPLE-NEXT:    ldp x29, x30, [sp, #16] ; 16-byte Folded Reload
-; CHECK-APPLE-NEXT:    ldp x20, x19, [sp], #32 ; 16-byte Folded Reload
-; CHECK-APPLE-NEXT:    .cfi_def_cfa_offset 0
-; CHECK-APPLE-NEXT:    .cfi_restore w30
-; CHECK-APPLE-NEXT:    .cfi_restore w29
-; CHECK-APPLE-NEXT:    .cfi_restore w19
-; CHECK-APPLE-NEXT:    .cfi_restore w20
 ; CHECK-APPLE-NEXT:    ret
 ; CHECK-APPLE-NEXT:  LBB0_2: ; %if.then
-; CHECK-APPLE-NEXT:    .cfi_restore_state
+; CHECK-APPLE-NEXT:    stp x20, x19, [sp, #-32]! ; 16-byte Folded Spill
+; CHECK-APPLE-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-APPLE-NEXT:    stp x29, x30, [sp, #16] ; 16-byte Folded Spill
+; CHECK-APPLE-NEXT:    .cfi_offset w30, -8
+; CHECK-APPLE-NEXT:    .cfi_offset w29, -16
+; CHECK-APPLE-NEXT:    .cfi_offset w19, -24
+; CHECK-APPLE-NEXT:    .cfi_offset w20, -32
 ; CHECK-APPLE-NEXT:    mov x19, x0
 ; CHECK-APPLE-NEXT:    bl _callVoid
 ; CHECK-APPLE-NEXT:    ldp x29, x30, [sp, #16] ; 16-byte Folded Reload
 ; CHECK-APPLE-NEXT:    mov x0, x19
 ; CHECK-APPLE-NEXT:    ldp x20, x19, [sp], #32 ; 16-byte Folded Reload
 ; CHECK-APPLE-NEXT:    .cfi_def_cfa_offset 0
 ; CHECK-APPLE-NEXT:    .cfi_restore w30
 ; CHECK-APPLE-NEXT:    .cfi_restore w29

llvm/test/CodeGen/AArch64/shrink-wrap-byval-inalloca-preallocated.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -o - %s | FileCheck %s
 
 target triple = "arm64-apple-ios"
 
 %struct.s = type {double, double }
 
 declare void @fn(ptr, ptr)
 
 ; %l.a and %l.b read memory allocated in the caller and should not block
 ; shrink-wrapping.
 define void @test_regular_pointers(ptr %a, ptr %b) {
 ; CHECK-LABEL: test_regular_pointers:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    stp x20, x19, [sp, #-32]! ; 16-byte Folded Spill
-; CHECK-NEXT:    .cfi_def_cfa_offset 32
-; CHECK-NEXT:    stp x29, x30, [sp, #16] ; 16-byte Folded Spill
-; CHECK-NEXT:    .cfi_offset w30, -8
-; CHECK-NEXT:    .cfi_offset w29, -16
-; CHECK-NEXT:    .cfi_offset w19, -24
-; CHECK-NEXT:    .cfi_offset w20, -32
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    mov x8, #1 ; =0x1
 ; CHECK-NEXT:    ldr d1, [x1, #8]
 ; CHECK-NEXT:    movk x8, #2047, lsl #16
 ; CHECK-NEXT:    fadd d0, d0, d1
 ; CHECK-NEXT:    fmov d1, x8
 ; CHECK-NEXT:    fcmp d0, d1
 ; CHECK-NEXT:    str d0, [x1]
 ; CHECK-NEXT:    b.mi LBB0_2
 ; CHECK-NEXT:    b.gt LBB0_2
 ; CHECK-NEXT:  ; %bb.1: ; %then
+; CHECK-NEXT:    stp x20, x19, [sp, #-32]! ; 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    stp x29, x30, [sp, #16] ; 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_offset w30, -8
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    .cfi_offset w19, -24
+; CHECK-NEXT:    .cfi_offset w20, -32
 ; CHECK-NEXT:    mov x19, x1
 ; CHECK-NEXT:    bl _fn
-; CHECK-NEXT:    str xzr, [x19]
-; CHECK-NEXT:  LBB0_2: ; %exit
 ; CHECK-NEXT:    ldp x29, x30, [sp, #16] ; 16-byte Folded Reload
+; CHECK-NEXT:    str xzr, [x19]
 ; CHECK-NEXT:    ldp x20, x19, [sp], #32 ; 16-byte Folded Reload
+; CHECK-NEXT:  LBB0_2: ; %exit
+; CHECK-NEXT:    .cfi_def_cfa wsp, 0
+; CHECK-NEXT:    .cfi_same_value w30
+; CHECK-NEXT:    .cfi_same_value w29
+; CHECK-NEXT:    .cfi_same_value w19
+; CHECK-NEXT:    .cfi_same_value w20
 ; CHECK-NEXT:    ret
 entry:
   %l.a = load double, ptr %a, align 8
   %gep.b = getelementptr inbounds %struct.s, ptr %b, i64 0, i32 1
   %l.b = load double, ptr %gep.b, align 8
   %add = fadd double %l.a, %l.b
   store double %add, ptr %b, align 8
   %c = fcmp ueq double %add, 0x7FF0001

llvm/test/CodeGen/AArch64/taildup-cfi.ll

 entry.if.end_crit_edge:                           ; preds = %entry
   %.pre = load i32, ptr @f, align 4, !tbaa !2
   br label %if.end
 
 if.then:                                          ; preds = %entry
   store i32 0, ptr @f, align 4, !tbaa !2
   br label %if.end
 
-; DARWIN:           Merging into block
+; DARWIN-NOT:           Merging into block
 ; LINUX:            Merging into block
 
 if.end:                                           ; preds = %entry.if.end_crit_edge, %if.then
   %1 = phi i32 [ %.pre, %entry.if.end_crit_edge ], [ 0, %if.then ]
   %cmp6 = icmp slt i32 %1, %o
   br i1 %cmp6, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %if.end

llvm/test/CodeGen/PowerPC/branch-on-store-cond.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu \
 ; RUN:   -mcpu=pwr8 < %s | FileCheck %s --check-prefix=CHECK
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-aix \
 ; RUN:   -mcpu=pwr8 < %s | FileCheck %s --check-prefix=CHECK-AIX
 
 define dso_local zeroext i8 @test1(ptr noundef %addr, i8 noundef zeroext %newval) local_unnamed_addr #0 {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    stbcx. 4, 0, 3
+; CHECK-NEXT:    bne 0, .LBB0_2
+; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    mflr 0
 ; CHECK-NEXT:    stdu 1, -32(1)
 ; CHECK-NEXT:    std 0, 48(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset lr, 16
-; CHECK-NEXT:    stbcx. 4, 0, 3
-; CHECK-NEXT:    bne 0, .LBB0_2
-; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    bl dummy
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  .LBB0_2: # %if.end
-; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    addi 1, 1, 32
 ; CHECK-NEXT:    ld 0, 16(1)
 ; CHECK-NEXT:    mtlr 0
+; CHECK-NEXT:  .LBB0_2: # %if.end
+; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-AIX-LABEL: test1:
 ; CHECK-AIX:       # %bb.0: # %entry
-; CHECK-AIX-NEXT:    mflr 0
-; CHECK-AIX-NEXT:    stdu 1, -112(1)
-; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    stbcx. 4, 0, 3
 ; CHECK-AIX-NEXT:    bne 0, L..BB0_2
 ; CHECK-AIX-NEXT:  # %bb.1: # %if.then
+; CHECK-AIX-NEXT:    mflr 0
+; CHECK-AIX-NEXT:    stdu 1, -112(1)
+; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    bl .dummy[PR]
 ; CHECK-AIX-NEXT:    nop
-; CHECK-AIX-NEXT:  L..BB0_2: # %if.end
-; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    addi 1, 1, 112
 ; CHECK-AIX-NEXT:    ld 0, 16(1)
 ; CHECK-AIX-NEXT:    mtlr 0
+; CHECK-AIX-NEXT:  L..BB0_2: # %if.end
+; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    blr
 entry:
   %conv = zext i8 %newval to i32
   %0 = tail call i32 @llvm.ppc.stbcx(ptr %addr, i32 %conv)
   %tobool.not = icmp eq i32 %0, 0
   br i1 %tobool.not, label %if.end, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void @dummy() #3
   br label %if.end
 
 if.end:                                           ; preds = %if.then, %entry
   ret i8 55
 }
 
 define dso_local signext i16 @test2(ptr noundef %addr, i16 noundef signext %newval) local_unnamed_addr #0 {
 ; CHECK-LABEL: test2:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    sthcx. 4, 0, 3
+; CHECK-NEXT:    bne 0, .LBB1_2
+; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    mflr 0
 ; CHECK-NEXT:    stdu 1, -32(1)
 ; CHECK-NEXT:    std 0, 48(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset lr, 16
-; CHECK-NEXT:    sthcx. 4, 0, 3
-; CHECK-NEXT:    bne 0, .LBB1_2
-; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    bl dummy
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  .LBB1_2: # %if.end
-; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    addi 1, 1, 32
 ; CHECK-NEXT:    ld 0, 16(1)
 ; CHECK-NEXT:    mtlr 0
+; CHECK-NEXT:  .LBB1_2: # %if.end
+; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-AIX-LABEL: test2:
 ; CHECK-AIX:       # %bb.0: # %entry
-; CHECK-AIX-NEXT:    mflr 0
-; CHECK-AIX-NEXT:    stdu 1, -112(1)
-; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    sthcx. 4, 0, 3
 ; CHECK-AIX-NEXT:    bne 0, L..BB1_2
 ; CHECK-AIX-NEXT:  # %bb.1: # %if.then
+; CHECK-AIX-NEXT:    mflr 0
+; CHECK-AIX-NEXT:    stdu 1, -112(1)
+; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    bl .dummy[PR]
 ; CHECK-AIX-NEXT:    nop
-; CHECK-AIX-NEXT:  L..BB1_2: # %if.end
-; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    addi 1, 1, 112
 ; CHECK-AIX-NEXT:    ld 0, 16(1)
 ; CHECK-AIX-NEXT:    mtlr 0
+; CHECK-AIX-NEXT:  L..BB1_2: # %if.end
+; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    blr
 entry:
   %0 = sext i16 %newval to i32
   %1 = tail call i32 @llvm.ppc.sthcx(ptr %addr, i32 %0)
   %tobool.not = icmp eq i32 %1, 0
   br i1 %tobool.not, label %if.end, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void @dummy() #3
   br label %if.end
 
 if.end:                                           ; preds = %if.then, %entry
   ret i16 55
 }
 
 define dso_local signext i32 @test3(ptr noundef %addr, i32 noundef signext %newval) local_unnamed_addr #0 {
 ; CHECK-LABEL: test3:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    stwcx. 4, 0, 3
+; CHECK-NEXT:    bne 0, .LBB2_2
+; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    mflr 0
 ; CHECK-NEXT:    stdu 1, -32(1)
 ; CHECK-NEXT:    std 0, 48(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset lr, 16
-; CHECK-NEXT:    stwcx. 4, 0, 3
-; CHECK-NEXT:    bne 0, .LBB2_2
-; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    bl dummy
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  .LBB2_2: # %if.end
-; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    addi 1, 1, 32
 ; CHECK-NEXT:    ld 0, 16(1)
 ; CHECK-NEXT:    mtlr 0
+; CHECK-NEXT:  .LBB2_2: # %if.end
+; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-AIX-LABEL: test3:
 ; CHECK-AIX:       # %bb.0: # %entry
-; CHECK-AIX-NEXT:    mflr 0
-; CHECK-AIX-NEXT:    stdu 1, -112(1)
-; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    stwcx. 4, 0, 3
 ; CHECK-AIX-NEXT:    bne 0, L..BB2_2
 ; CHECK-AIX-NEXT:  # %bb.1: # %if.then
+; CHECK-AIX-NEXT:    mflr 0
+; CHECK-AIX-NEXT:    stdu 1, -112(1)
+; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    bl .dummy[PR]
 ; CHECK-AIX-NEXT:    nop
-; CHECK-AIX-NEXT:  L..BB2_2: # %if.end
-; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    addi 1, 1, 112
 ; CHECK-AIX-NEXT:    ld 0, 16(1)
 ; CHECK-AIX-NEXT:    mtlr 0
+; CHECK-AIX-NEXT:  L..BB2_2: # %if.end
+; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    blr
 entry:
   %0 = tail call i32 @llvm.ppc.stwcx(ptr %addr, i32 %newval)
   %tobool.not = icmp eq i32 %0, 0
   br i1 %tobool.not, label %if.end, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void @dummy() #3
   br label %if.end
 
 if.end:                                           ; preds = %if.then, %entry
   ret i32 55
 }
 
 define dso_local i64 @test4(ptr noundef %addr, i64 noundef %newval) local_unnamed_addr #0 {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    stdcx. 4, 0, 3
+; CHECK-NEXT:    bne 0, .LBB3_2
+; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    mflr 0
 ; CHECK-NEXT:    stdu 1, -32(1)
 ; CHECK-NEXT:    std 0, 48(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    .cfi_offset lr, 16
-; CHECK-NEXT:    stdcx. 4, 0, 3
-; CHECK-NEXT:    bne 0, .LBB3_2
-; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    bl dummy
 ; CHECK-NEXT:    nop
-; CHECK-NEXT:  .LBB3_2: # %if.end
-; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    addi 1, 1, 32
 ; CHECK-NEXT:    ld 0, 16(1)
 ; CHECK-NEXT:    mtlr 0
+; CHECK-NEXT:  .LBB3_2: # %if.end
+; CHECK-NEXT:    li 3, 55
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-AIX-LABEL: test4:
 ; CHECK-AIX:       # %bb.0: # %entry
-; CHECK-AIX-NEXT:    mflr 0
-; CHECK-AIX-NEXT:    stdu 1, -112(1)
-; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    stdcx. 4, 0, 3
 ; CHECK-AIX-NEXT:    bne 0, L..BB3_2
 ; CHECK-AIX-NEXT:  # %bb.1: # %if.then
+; CHECK-AIX-NEXT:    mflr 0
+; CHECK-AIX-NEXT:    stdu 1, -112(1)
+; CHECK-AIX-NEXT:    std 0, 128(1)
 ; CHECK-AIX-NEXT:    bl .dummy[PR]
 ; CHECK-AIX-NEXT:    nop
-; CHECK-AIX-NEXT:  L..BB3_2: # %if.end
-; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    addi 1, 1, 112
 ; CHECK-AIX-NEXT:    ld 0, 16(1)
 ; CHECK-AIX-NEXT:    mtlr 0
+; CHECK-AIX-NEXT:  L..BB3_2: # %if.end
+; CHECK-AIX-NEXT:    li 3, 55
 ; CHECK-AIX-NEXT:    blr
 entry:
   %0 = tail call i32 @llvm.ppc.stdcx(ptr %addr, i64 %newval)
   %tobool.not = icmp eq i32 %0, 0
   br i1 %tobool.not, label %if.end, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void @dummy() #3

llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mcpu=pwr10 < %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-p:64:64-n32:64-v256:256:256-v512:512:512"
 target triple = "powerpc64le-unknown-linux-gnu"
 
 %_elem_type_of_a = type <{ double }>
 %_elem_type_of_x = type <{ double }>
 
 define void @foo(ptr %.m, ptr %.n, ptr %.a, ptr %.x, ptr %.l, ptr %.vy01, ptr %.vy02, ptr %.vy03, ptr %.vy04, ptr %.vy05, ptr %.vy06, ptr %.vy07, ptr %.vy08, ptr %.vy09, ptr %.vy0a, ptr %.vy0b, ptr %.vy0c, ptr %.vy21, ptr %.vy22, ptr %.vy23, ptr %.vy24, ptr %.vy25, ptr %.vy26, ptr %.vy27, ptr %.vy28, ptr %.vy29, ptr %.vy2a, ptr %.vy2b, ptr %.vy2c) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    lwz 4, 0(4)
+; CHECK-NEXT:    cmpwi 4, 1
+; CHECK-NEXT:    bltlr 0
+; CHECK-NEXT:  # %bb.1: # %_loop_1_do_.lr.ph
+; CHECK-NEXT:    lwz 3, 0(3)
+; CHECK-NEXT:    cmpwi 3, 1
+; CHECK-NEXT:    bltlr 0
+; CHECK-NEXT:  # %bb.2: # %_loop_1_do_.preheader
 ; CHECK-NEXT:    stdu 1, -592(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 592
 ; CHECK-NEXT:    .cfi_offset r14, -192
 ; CHECK-NEXT:    .cfi_offset r15, -184
 ; CHECK-NEXT:    .cfi_offset r16, -176
 ; CHECK-NEXT:    .cfi_offset r17, -168
 ; CHECK-NEXT:    .cfi_offset r18, -160
 ; CHECK-NEXT:    .cfi_offset r19, -152

 ; CHECK-NEXT:    .cfi_offset v24, -320
 ; CHECK-NEXT:    .cfi_offset v25, -304
 ; CHECK-NEXT:    .cfi_offset v26, -288
 ; CHECK-NEXT:    .cfi_offset v27, -272
 ; CHECK-NEXT:    .cfi_offset v28, -256
 ; CHECK-NEXT:    .cfi_offset v29, -240
 ; CHECK-NEXT:    .cfi_offset v30, -224
 ; CHECK-NEXT:    .cfi_offset v31, -208
-; CHECK-NEXT:    lwz 4, 0(4)
-; CHECK-NEXT:    std 14, 400(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 15, 408(1) # 8-byte Folded Spill
-; CHECK-NEXT:    cmpwi 4, 1
-; CHECK-NEXT:    std 16, 416(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 17, 424(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 22, 464(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 23, 472(1) # 8-byte Folded Spill
+; CHECK-NEXT:    mr 22, 5
+; CHECK-NEXT:    ld 5, 848(1)
+; CHECK-NEXT:    mr 11, 7
+; CHECK-NEXT:    ld 23, 688(1)
+; CHECK-NEXT:    addi 3, 3, 1
+; CHECK-NEXT:    ld 2, 760(1)
+; CHECK-NEXT:    std 28, 512(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 29, 520(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 28, 824(1)
+; CHECK-NEXT:    ld 7, 728(1)
 ; CHECK-NEXT:    std 18, 432(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 19, 440(1) # 8-byte Folded Spill
+; CHECK-NEXT:    mr 18, 6
+; CHECK-NEXT:    ld 6, 712(1)
+; CHECK-NEXT:    cmpldi 3, 9
+; CHECK-NEXT:    ld 19, 768(1)
+; CHECK-NEXT:    std 10, 64(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 6, 72(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 5, 200(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 5, 840(1)
+; CHECK-NEXT:    lxv 33, 0(6)
+; CHECK-NEXT:    std 14, 400(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 15, 408(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 30, 528(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 31, 536(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 20, 448(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 21, 456(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 22, 464(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 23, 472(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 21, 784(1)
+; CHECK-NEXT:    ld 20, 776(1)
+; CHECK-NEXT:    lxv 10, 0(19)
+; CHECK-NEXT:    lxv 7, 0(21)
+; CHECK-NEXT:    ld 15, 736(1)
+; CHECK-NEXT:    ld 29, 704(1)
+; CHECK-NEXT:    ld 30, 720(1)
+; CHECK-NEXT:    std 2, 112(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 19, 120(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 42, 0(9)
+; CHECK-NEXT:    lxv 37, 0(7)
+; CHECK-NEXT:    std 20, 128(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 21, 136(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 43, 0(8)
+; CHECK-NEXT:    lxv 41, 0(10)
+; CHECK-NEXT:    std 16, 416(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 17, 424(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 17, 752(1)
+; CHECK-NEXT:    ld 16, 744(1)
 ; CHECK-NEXT:    std 24, 480(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 25, 488(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 25, 800(1)
+; CHECK-NEXT:    ld 24, 792(1)
 ; CHECK-NEXT:    std 26, 496(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 27, 504(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 28, 512(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 29, 520(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 30, 528(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 31, 536(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 27, 816(1)
+; CHECK-NEXT:    ld 26, 808(1)
+; CHECK-NEXT:    std 8, 48(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 9, 56(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 40, 0(23)
+; CHECK-NEXT:    lxv 38, 0(29)
+; CHECK-NEXT:    std 7, 80(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 15, 88(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 32, 0(30)
+; CHECK-NEXT:    lxv 36, 0(15)
+; CHECK-NEXT:    mr 8, 29
+; CHECK-NEXT:    mr 10, 30
+; CHECK-NEXT:    std 26, 160(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 27, 168(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 13, 0(16)
+; CHECK-NEXT:    lxv 12, 0(17)
 ; CHECK-NEXT:    stfd 26, 544(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    stfd 27, 552(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 11, 0(2)
+; CHECK-NEXT:    lxv 9, 0(20)
 ; CHECK-NEXT:    stfd 28, 560(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    stfd 29, 568(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 5, 0(24)
+; CHECK-NEXT:    lxv 4, 0(25)
 ; CHECK-NEXT:    stfd 30, 576(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    stfd 31, 584(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 2, 0(26)
+; CHECK-NEXT:    lxv 0, 0(27)
+; CHECK-NEXT:    li 27, 0
 ; CHECK-NEXT:    stxv 52, 208(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 53, 224(1) # 16-byte Folded Spill
+; CHECK-NEXT:    lxv 1, 0(28)
 ; CHECK-NEXT:    stxv 54, 240(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 55, 256(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 56, 272(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 57, 288(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 58, 304(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 59, 320(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 60, 336(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 61, 352(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 62, 368(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    stxv 63, 384(1) # 16-byte Folded Spill
-; CHECK-NEXT:    blt 0, .LBB0_7
-; CHECK-NEXT:  # %bb.1: # %_loop_1_do_.lr.ph
-; CHECK-NEXT:    lwz 3, 0(3)
-; CHECK-NEXT:    cmpwi 3, 1
-; CHECK-NEXT:    blt 0, .LBB0_7
-; CHECK-NEXT:  # %bb.2: # %_loop_1_do_.preheader
-; CHECK-NEXT:    mr 22, 5
-; CHECK-NEXT:    ld 5, 848(1)
-; CHECK-NEXT:    ld 28, 824(1)
-; CHECK-NEXT:    mr 11, 7
-; CHECK-NEXT:    mr 18, 6
-; CHECK-NEXT:    addi 3, 3, 1
-; CHECK-NEXT:    std 8, 48(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 9, 56(1) # 8-byte Folded Spill
-; CHECK-NEXT:    ld 6, 712(1)
-; CHECK-NEXT:    ld 23, 688(1)
-; CHECK-NEXT:    std 10, 64(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 6, 72(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 5, 200(1) # 8-byte Folded Spill
-; CHECK-NEXT:    ld 5, 840(1)
-; CHECK-NEXT:    cmpldi 3, 9
-; CHECK-NEXT:    ld 19, 768(1)
-; CHECK-NEXT:    ld 2, 760(1)
-; CHECK-NEXT:    lxv 33, 0(6)
-; CHECK-NEXT:    ld 21, 784(1)
-; CHECK-NEXT:    ld 20, 776(1)
-; CHECK-NEXT:    std 2, 112(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 19, 120(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 10, 0(19)
-; CHECK-NEXT:    ld 7, 728(1)
-; CHECK-NEXT:    std 20, 128(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 21, 136(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 7, 0(21)
-; CHECK-NEXT:    ld 27, 816(1)
-; CHECK-NEXT:    std 27, 168(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 5, 192(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    ld 5, 832(1)
-; CHECK-NEXT:    ld 26, 808(1)
-; CHECK-NEXT:    ld 25, 800(1)
-; CHECK-NEXT:    ld 24, 792(1)
-; CHECK-NEXT:    ld 17, 752(1)
-; CHECK-NEXT:    ld 16, 744(1)
-; CHECK-NEXT:    ld 15, 736(1)
-; CHECK-NEXT:    ld 29, 704(1)
-; CHECK-NEXT:    ld 30, 720(1)
-; CHECK-NEXT:    lxv 42, 0(9)
-; CHECK-NEXT:    std 7, 80(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 15, 88(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 37, 0(7)
-; CHECK-NEXT:    lxv 43, 0(8)
-; CHECK-NEXT:    mr 8, 29
-; CHECK-NEXT:    std 26, 160(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 16, 96(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 17, 104(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 24, 144(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 25, 152(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 5, 184(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 28, 176(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    ld 5, 696(1)
-; CHECK-NEXT:    lxv 41, 0(10)
-; CHECK-NEXT:    lxv 40, 0(23)
-; CHECK-NEXT:    lxv 38, 0(29)
-; CHECK-NEXT:    lxv 32, 0(30)
-; CHECK-NEXT:    lxv 36, 0(15)
-; CHECK-NEXT:    lxv 13, 0(16)
-; CHECK-NEXT:    lxv 12, 0(17)
-; CHECK-NEXT:    lxv 11, 0(2)
-; CHECK-NEXT:    lxv 9, 0(20)
-; CHECK-NEXT:    lxv 5, 0(24)
-; CHECK-NEXT:    lxv 4, 0(25)
-; CHECK-NEXT:    mr 10, 30
+; CHECK-NEXT:    li 28, 1
+; CHECK-NEXT:    ld 7, 184(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lxv 3, 0(7)
 ; CHECK-NEXT:    std 5, 32(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    std 23, 40(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    lwa 5, 0(11)
 ; CHECK-NEXT:    li 11, 9
-; CHECK-NEXT:    ld 7, 184(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lxv 2, 0(26)
-; CHECK-NEXT:    lxv 0, 0(27)
-; CHECK-NEXT:    lxv 1, 0(28)
-; CHECK-NEXT:    li 28, 1
-; CHECK-NEXT:    li 27, 0
 ; CHECK-NEXT:    ld 9, 32(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lxv 3, 0(7)
 ; CHECK-NEXT:    iselgt 3, 3, 11
 ; CHECK-NEXT:    addi 3, 3, -2
 ; CHECK-NEXT:    mulli 6, 5, 40
 ; CHECK-NEXT:    sldi 0, 5, 4
 ; CHECK-NEXT:    extswsli 14, 5, 3
 ; CHECK-NEXT:    rldicl 12, 3, 61, 3
 ; CHECK-NEXT:    lxv 39, 0(9)
 ; CHECK-NEXT:    add 31, 14, 22

 ; CHECK-NEXT:    add 3, 3, 12
 ; CHECK-NEXT:    add 20, 20, 12
 ; CHECK-NEXT:    add 21, 21, 12
 ; CHECK-NEXT:    addi 27, 27, 1
 ; CHECK-NEXT:    cmpld 28, 4
 ; CHECK-NEXT:    ble 0, .LBB0_3
 ; CHECK-NEXT:  # %bb.6: # %_loop_1_loopHeader_._return_bb_crit_edge.loopexit
 ; CHECK-NEXT:    ld 3, 48(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lxv 63, 384(1) # 16-byte Folded Reload
 ; CHECK-NEXT:    stxv 43, 0(3)
 ; CHECK-NEXT:    ld 3, 56(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lxv 62, 368(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 61, 352(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 60, 336(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 59, 320(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 58, 304(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 57, 288(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 56, 272(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 55, 256(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 54, 240(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 53, 224(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 52, 208(1) # 16-byte Folded Reload
 ; CHECK-NEXT:    stxv 42, 0(3)
 ; CHECK-NEXT:    ld 3, 64(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 31, 584(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 30, 576(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 29, 568(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 28, 560(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 27, 552(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 26, 544(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 31, 536(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 30, 528(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 29, 520(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 28, 512(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 27, 504(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 41, 0(3)
 ; CHECK-NEXT:    ld 3, 40(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 26, 496(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 25, 488(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 24, 480(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 23, 472(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 22, 464(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 21, 456(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 20, 448(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 19, 440(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 18, 432(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 17, 424(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 16, 416(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 40, 0(3)
 ; CHECK-NEXT:    ld 3, 72(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 39, 0(9)
 ; CHECK-NEXT:    stxv 38, 0(8)
+; CHECK-NEXT:    ld 15, 408(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 14, 400(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 33, 0(3)
 ; CHECK-NEXT:    ld 3, 80(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 32, 0(10)
 ; CHECK-NEXT:    stxv 37, 0(3)
 ; CHECK-NEXT:    ld 3, 88(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 36, 0(3)
 ; CHECK-NEXT:    ld 3, 96(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 13, 0(3)

 ; CHECK-NEXT:    ld 3, 176(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 1, 0(3)
 ; CHECK-NEXT:    ld 3, 184(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 3, 0(3)
 ; CHECK-NEXT:    ld 3, 192(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 8, 0(3)
 ; CHECK-NEXT:    ld 3, 200(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 6, 0(3)
-; CHECK-NEXT:  .LBB0_7: # %_return_bb
-; CHECK-NEXT:    lxv 63, 384(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 62, 368(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 61, 352(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 60, 336(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 59, 320(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 58, 304(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 57, 288(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 56, 272(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 55, 256(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 54, 240(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 53, 224(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 52, 208(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lfd 31, 584(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 30, 576(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 29, 568(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 28, 560(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 27, 552(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 26, 544(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 31, 536(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 30, 528(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 29, 520(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 28, 512(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 27, 504(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 26, 496(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 25, 488(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 24, 480(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 23, 472(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 22, 464(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 21, 456(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 20, 448(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 19, 440(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 18, 432(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 17, 424(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 16, 416(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 15, 408(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 14, 400(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    addi 1, 1, 592
 ; CHECK-NEXT:    blr
 entry:
   %_val_l_ = load i32, ptr %.l, align 4
   %_conv = sext i32 %_val_l_ to i64
   %_mult_tmp = shl nsw i64 %_conv, 3
   %_sub_tmp4 = sub nuw nsw i64 -8, %_mult_tmp
   %_val_n_ = load i32, ptr %.n, align 4

llvm/test/CodeGen/PowerPC/ppc64-sibcall-shrinkwrap.ll

 ; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=false | FileCheck %s -check-prefix=CHECK-SCO-ONLY
-; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=true | FileCheck %s -check-prefix=CHECK-SCO-ONLY
+; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=true | FileCheck %s -check-prefix=CHECK-SCO-SR
 ; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64le-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=false | FileCheck %s -check-prefix=CHECK-SCO-ONLY
-; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64le-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=true | FileCheck %s -check-prefix=CHECK-SCO-ONLY
+; RUN: llc -relocation-model=static -verify-machineinstrs < %s -mtriple=powerpc64le-unknown-linux-gnu -disable-ppc-sco=false --enable-shrink-wrap=true | FileCheck %s -check-prefix=CHECK-SCO-SR
 ; RUN: not --crash llc -relocation-model=pic -verify-machineinstrs < %s -mtriple=powerpc64-ibm-aix-xcoff -tailcallopt -disable-ppc-sco=false --enable-shrink-wrap=true 2>&1 | FileCheck %s -check-prefix=CHECK-AIX
 ;; The above RUN command is expected to fail on AIX since tail calling is not implemented ATM
-;; Edit: D63152 prevents stack popping before loads and stores, so shrink-wrap does nothing here
 %"class.clang::NamedDecl" = type { i32 }
 declare void @__assert_fail();
 
 define dso_local i8 @_ZNK5clang9NamedDecl23getLinkageAndVisibilityEv(
     ptr %this) {
 entry:
   %tobool = icmp eq ptr %this, null
   br i1 %tobool, label %cond.false, label %exit
exit:
   ret i8 %call.i
 
 ; CHECK-SCO-ONLY-LABEL: _ZNK5clang9NamedDecl23getLinkageAndVisibilityEv:
 ; CHECK-SCO-ONLY: stdu 1, -{{[0-9]+}}(1)
 ; CHECK-SCO-ONLY: b LVComputationKind
 ; CHECK-SCO-ONLY: #TC_RETURNd8
 ; CHECK-SCO-ONLY: bl __assert_fail
 ;
+; CHECK-SCO-SR-LABEL: _ZNK5clang9NamedDecl23getLinkageAndVisibilityEv:
+; CHECK-SCO-SR: b LVComputationKind
+; CHECK-SCO-SR: #TC_RETURNd8
+; CHECK-SCO-SR: stdu 1, -{{[0-9]+}}(1)
+; CHECK-SCO-SR: bl __assert_fail
+
 ; CHECK-AIX: LLVM ERROR: Tail call support is unimplemented on AIX.
 }
 
 define dso_local fastcc i8 @LVComputationKind(
     ptr %D,
     i32 %computation) {
   ret i8 0
 }

llvm/test/CodeGen/RISCV/hoist-global-addr-base.ll

if.then:                                          ; preds = %entry
   ret ptr getelementptr inbounds (%struct.S, ptr @s, i32 0, i32 1)
 if.end:                                           ; preds = %if.then, %entry
   ret ptr getelementptr inbounds (%struct.S, ptr @s, i32 0, i32 3)
 }
 
 define dso_local i32 @load_half() nounwind {
 ; RV32-LABEL: load_half:
 ; RV32:       # %bb.0: # %entry
-; RV32-NEXT:    addi sp, sp, -16
-; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    lui a0, %hi(foo+8)
 ; RV32-NEXT:    lhu a0, %lo(foo+8)(a0)
 ; RV32-NEXT:    li a1, 140
 ; RV32-NEXT:    bne a0, a1, .LBB8_2
 ; RV32-NEXT:  # %bb.1: # %if.end
 ; RV32-NEXT:    li a0, 0
-; RV32-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
-; RV32-NEXT:    addi sp, sp, 16
 ; RV32-NEXT:    ret
 ; RV32-NEXT:  .LBB8_2: # %if.then
+; RV32-NEXT:    addi sp, sp, -16
+; RV32-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    call abort@plt
 ;
 ; RV64-LABEL: load_half:
 ; RV64:       # %bb.0: # %entry
-; RV64-NEXT:    addi sp, sp, -16
-; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    lui a0, %hi(foo+8)
 ; RV64-NEXT:    lhu a0, %lo(foo+8)(a0)
 ; RV64-NEXT:    li a1, 140
 ; RV64-NEXT:    bne a0, a1, .LBB8_2
 ; RV64-NEXT:  # %bb.1: # %if.end
 ; RV64-NEXT:    li a0, 0
-; RV64-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
-; RV64-NEXT:    addi sp, sp, 16
 ; RV64-NEXT:    ret
 ; RV64-NEXT:  .LBB8_2: # %if.then
+; RV64-NEXT:    addi sp, sp, -16
+; RV64-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    call abort@plt
 entry:
   %0 = load i16, ptr getelementptr inbounds ([6 x i16], ptr @foo, i32 0, i32 4), align 2
   %cmp = icmp eq i16 %0, 140
   br i1 %cmp, label %if.end, label %if.then
 
 if.then:
   tail call void @abort()

llvm/test/CodeGen/RISCV/pr51206.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ;RUN: llc < %s -mtriple=riscv64-unknown-linux-gnu -mattr=+m | FileCheck %s
 
 ; This test used to cause an infinite loop.
 
 @global = global i8 0, align 1
 @global.1 = global i32 0, align 4
 @global.2 = global i8 0, align 1
 @global.3 = global i32 0, align 4
 
 define signext i32 @wobble() nounwind {
 ; CHECK-LABEL: wobble:
 ; CHECK:       # %bb.0: # %bb
-; CHECK-NEXT:    addi sp, sp, -16
-; CHECK-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    lui a0, %hi(global)
 ; CHECK-NEXT:    lbu a0, %lo(global)(a0)
 ; CHECK-NEXT:    lui a1, %hi(global.2)
 ; CHECK-NEXT:    lbu a1, %lo(global.2)(a1)
 ; CHECK-NEXT:    addi a0, a0, 1
 ; CHECK-NEXT:    lui a2, %hi(global.1)
 ; CHECK-NEXT:    sw a0, %lo(global.1)(a2)
 ; CHECK-NEXT:    mul a0, a0, a1
 ; CHECK-NEXT:    slli a1, a0, 48
 ; CHECK-NEXT:    lui a2, 52429
 ; CHECK-NEXT:    slli a2, a2, 4
 ; CHECK-NEXT:    mulhu a1, a1, a2
 ; CHECK-NEXT:    srli a1, a1, 18
 ; CHECK-NEXT:    lui a2, %hi(global.3)
 ; CHECK-NEXT:    li a3, 5
 ; CHECK-NEXT:    sw a1, %lo(global.3)(a2)
 ; CHECK-NEXT:    bltu a0, a3, .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %bb10
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; CHECK-NEXT:    call quux@plt
-; CHECK-NEXT:  .LBB0_2: # %bb12
-; CHECK-NEXT:    li a0, 0
 ; CHECK-NEXT:    ld ra, 8(sp) # 8-byte Folded Reload
 ; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:  .LBB0_2: # %bb12
+; CHECK-NEXT:    li a0, 0
 ; CHECK-NEXT:    ret
 bb:
   %tmp = load i8, ptr @global, align 1
   %tmp1 = zext i8 %tmp to i32
   %tmp2 = add nuw nsw i32 %tmp1, 1
   store i32 %tmp2, ptr @global.1, align 4
   %tmp3 = load i8, ptr @global.2, align 1
   %tmp4 = zext i8 %tmp3 to i32

llvm/test/CodeGen/Thumb/thumb-shrink-wrapping.ll

 if.end:
   %value.addr.0 = phi i32 [ %div, %if.then ], [ %mul, %if.else ]
   ret i32 %value.addr.0
 }
 
 define i1 @beq_to_bx(ptr %y, i32 %head) {
 ; ENABLE-V4T-LABEL: beq_to_bx:
 ; ENABLE-V4T:       @ %bb.0: @ %entry
-; ENABLE-V4T-NEXT:    push {r4, lr}
-; ENABLE-V4T-NEXT:    .cfi_def_cfa_offset 8
-; ENABLE-V4T-NEXT:    .cfi_offset lr, -4
-; ENABLE-V4T-NEXT:    .cfi_offset r4, -8
 ; ENABLE-V4T-NEXT:    movs r2, r0
 ; ENABLE-V4T-NEXT:    movs r0, #1
 ; ENABLE-V4T-NEXT:    cmp r2, #0
 ; ENABLE-V4T-NEXT:    beq LBB11_3
 ; ENABLE-V4T-NEXT:  @ %bb.1: @ %if.end
+; ENABLE-V4T-NEXT:    push {r4, lr}
+; ENABLE-V4T-NEXT:    .cfi_def_cfa_offset 8
+; ENABLE-V4T-NEXT:    .cfi_offset lr, -4
+; ENABLE-V4T-NEXT:    .cfi_offset r4, -8
 ; ENABLE-V4T-NEXT:    ldr r3, [r2]
 ; ENABLE-V4T-NEXT:    lsls r4, r3, #30
+; ENABLE-V4T-NEXT:    ldr r4, [sp, #4]
+; ENABLE-V4T-NEXT:    mov lr, r4
+; ENABLE-V4T-NEXT:    pop {r4}
+; ENABLE-V4T-NEXT:    add sp, #4
 ; ENABLE-V4T-NEXT:    bpl LBB11_3
 ; ENABLE-V4T-NEXT:  @ %bb.2: @ %if.end4
 ; ENABLE-V4T-NEXT:    str r1, [r2]
 ; ENABLE-V4T-NEXT:    str r3, [r2]
 ; ENABLE-V4T-NEXT:    movs r0, #0
 ; ENABLE-V4T-NEXT:  LBB11_3: @ %cleanup
-; ENABLE-V4T-NEXT:    pop {r4}
-; ENABLE-V4T-NEXT:    pop {r1}
-; ENABLE-V4T-NEXT:    bx r1
+; ENABLE-V4T-NEXT:    bx lr
 ;
 ; ENABLE-V5T-LABEL: beq_to_bx:
 ; ENABLE-V5T:       @ %bb.0: @ %entry
-; ENABLE-V5T-NEXT:    push {r4, lr}
-; ENABLE-V5T-NEXT:    .cfi_def_cfa_offset 8
-; ENABLE-V5T-NEXT:    .cfi_offset lr, -4
-; ENABLE-V5T-NEXT:    .cfi_offset r4, -8
 ; ENABLE-V5T-NEXT:    movs r2, r0
 ; ENABLE-V5T-NEXT:    movs r0, #1
 ; ENABLE-V5T-NEXT:    cmp r2, #0
 ; ENABLE-V5T-NEXT:    beq LBB11_3
 ; ENABLE-V5T-NEXT:  @ %bb.1: @ %if.end
+; ENABLE-V5T-NEXT:    push {r4, lr}
+; ENABLE-V5T-NEXT:    .cfi_def_cfa_offset 8
+; ENABLE-V5T-NEXT:    .cfi_offset lr, -4
+; ENABLE-V5T-NEXT:    .cfi_offset r4, -8
 ; ENABLE-V5T-NEXT:    ldr r3, [r2]
 ; ENABLE-V5T-NEXT:    lsls r4, r3, #30
+; ENABLE-V5T-NEXT:    ldr r4, [sp, #4]
+; ENABLE-V5T-NEXT:    mov lr, r4
+; ENABLE-V5T-NEXT:    pop {r4}
+; ENABLE-V5T-NEXT:    add sp, #4
 ; ENABLE-V5T-NEXT:    bpl LBB11_3
 ; ENABLE-V5T-NEXT:  @ %bb.2: @ %if.end4
 ; ENABLE-V5T-NEXT:    str r1, [r2]
 ; ENABLE-V5T-NEXT:    str r3, [r2]
 ; ENABLE-V5T-NEXT:    movs r0, #0
 ; ENABLE-V5T-NEXT:  LBB11_3: @ %cleanup
-; ENABLE-V5T-NEXT:    pop {r4, pc}
+; ENABLE-V5T-NEXT:    bx lr
 ;
 ; DISABLE-V4T-LABEL: beq_to_bx:
 ; DISABLE-V4T:       @ %bb.0: @ %entry
 ; DISABLE-V4T-NEXT:    push {r4, lr}
 ; DISABLE-V4T-NEXT:    .cfi_def_cfa_offset 8
 ; DISABLE-V4T-NEXT:    .cfi_offset lr, -4
 ; DISABLE-V4T-NEXT:    .cfi_offset r4, -8
 ; DISABLE-V4T-NEXT:    movs r2, r0

llvm/test/CodeGen/X86/2010-08-04-MaskedSignedCompare.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 ; PR7814
 
 @g_16 = dso_local global i64 -3738643449681751625, align 8
 @g_38 = dso_local global i32 0, align 4
 @.str = private constant [4 x i8] c"%d\0A\00"
 
 define dso_local i32 @main() nounwind {
 ; CHECK-LABEL: main:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    cmpq g_16(%rip), %rax
 ; CHECK-NEXT:    sbbl %eax, %eax
 ; CHECK-NEXT:    testb $-106, %al
 ; CHECK-NEXT:    jle .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %if.then
 ; CHECK-NEXT:    movl $1, g_38(%rip)
 ; CHECK-NEXT:    movl $1, %esi
 ; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:  .LBB0_1: # %entry.if.end_crit_edge
 ; CHECK-NEXT:    movl g_38(%rip), %esi
 ; CHECK-NEXT:  .LBB0_3: # %if.end
+; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    movl $.L.str, %edi
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    callq printf@PLT
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    popq %rcx
 ; CHECK-NEXT:    retq
 entry:
   %tmp = load i64, ptr @g_16

llvm/test/CodeGen/X86/cmov.ll

 @g_3 = external dso_local global i8
 @g_96 = external dso_local global i8
 @g_100 = external dso_local global i8
 @_2E_str = external dso_local constant [15 x i8], align 1
 
 define i1 @test4() nounwind {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    movsbl g_3(%rip), %edx
 ; CHECK-NEXT:    movzbl %dl, %ecx
 ; CHECK-NEXT:    shrl $7, %ecx
 ; CHECK-NEXT:    xorb $1, %cl
 ; CHECK-NEXT:    # kill: def $cl killed $cl killed $ecx
 ; CHECK-NEXT:    sarl %cl, %edx
 ; CHECK-NEXT:    movzbl g_96(%rip), %eax
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je .LBB3_2
 ; CHECK-NEXT:  # %bb.1: # %bb.i.i.i
 ; CHECK-NEXT:    movzbl g_100(%rip), %ecx
 ; CHECK-NEXT:  .LBB3_2: # %func_4.exit.i
+; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    xorl %esi, %esi
 ; CHECK-NEXT:    testb %dl, %dl
 ; CHECK-NEXT:    setne %bl
 ; CHECK-NEXT:    movzbl %al, %ecx
 ; CHECK-NEXT:    cmovnel %esi, %ecx
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je .LBB3_5
 ; CHECK-NEXT:  # %bb.3: # %func_4.exit.i

llvm/test/CodeGen/X86/copy-eflags.ll

 ; X32-NEXT:    calll external
 ; X32-NEXT:    addl $4, %esp
 ; X32-NEXT:  .LBB0_2: # %if.end
 ; X32-NEXT:    xorl %eax, %eax
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test1:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    movzbl b(%rip), %ecx
 ; X64-NEXT:    leal 1(%rcx), %eax
 ; X64-NEXT:    movb %al, b(%rip)
 ; X64-NEXT:    incl c(%rip)
 ; X64-NEXT:    sete %dl
 ; X64-NEXT:    movzbl a(%rip), %esi
 ; X64-NEXT:    leal 1(%rsi), %edi
 ; X64-NEXT:    cmpb %cl, %sil
 ; X64-NEXT:    sete d(%rip)
 ; X64-NEXT:    movb %dil, a(%rip)
 ; X64-NEXT:    testb %dl, %dl
 ; X64-NEXT:    jne .LBB0_2
 ; X64-NEXT:  # %bb.1: # %if.then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    movsbl %al, %edi
 ; X64-NEXT:    callq external
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB0_2: # %if.end
 ; X64-NEXT:    xorl %eax, %eax
-; X64-NEXT:    popq %rcx
 ; X64-NEXT:    retq
 entry:
   %bval = load i8, ptr @b
   %inc = add i8 %bval, 1
   store volatile i8 %inc, ptr @b
   %cval = load volatile i32, ptr @c
   %inc1 = add nsw i32 %cval, 1
   store volatile i32 %inc1, ptr @c

llvm/test/CodeGen/X86/csr-split.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=x86_64-unknown-linux < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mtriple=i386-unknown-linux < %s | FileCheck %s --check-prefix=CHECK-32BIT
 
 ; Check CSR split can work properly for tests below.
 
 @a = common dso_local local_unnamed_addr global i32 0, align 4
 
 define dso_local signext i32 @test1(ptr %b) local_unnamed_addr  {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset %rbx, -16
 ; CHECK-NEXT:    movslq a(%rip), %rax
 ; CHECK-NEXT:    cmpq %rdi, %rax
 ; CHECK-NEXT:    je .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %if.end
-; CHECK-NEXT:    popq %rbx
-; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  .LBB0_2: # %if.then
+; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbx, -16
 ; CHECK-NEXT:    movq %rdi, %rbx
 ; CHECK-NEXT:    callq callVoid@PLT
 ; CHECK-NEXT:    movq %rbx, %rdi
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    jmp callNonVoid@PLT # TAILCALL
 ;
 ; CHECK-32BIT-LABEL: test1:

llvm/test/CodeGen/X86/i686-win-shrink-wrapping.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc %s -o - -enable-shrink-wrap=true | FileCheck --check-prefix=SHRINK-WRAP %s
 ; RUN: llc %s -o - -enable-shrink-wrap=false | FileCheck --check-prefix=NO-SHRINK-WRAP %s
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-pc-windows-msvc18.0.0"
 
 %struct.S = type { i32 }
 
 ; Check that we do not use a basic block that has EFLAGS as live-in
 ; if we need to realign the stack.
 define x86_thiscallcc void @stackRealignment(ptr %this) {
 ; SHRINK-WRAP-LABEL: stackRealignment:
 ; SHRINK-WRAP:       # %bb.0: # %entry
-; SHRINK-WRAP-NEXT:    pushl %ebp
-; SHRINK-WRAP-NEXT:    movl %esp, %ebp
-; SHRINK-WRAP-NEXT:    andl $-8, %esp
-; SHRINK-WRAP-NEXT:    subl $16, %esp
 ; SHRINK-WRAP-NEXT:    movl (%ecx), %eax
 ; SHRINK-WRAP-NEXT:    cmpl $33, %eax
 ; SHRINK-WRAP-NEXT:    movl $42, %edx
 ; SHRINK-WRAP-NEXT:    jge LBB0_2
 ; SHRINK-WRAP-NEXT:  # %bb.1: # %entry
 ; SHRINK-WRAP-NEXT:    movl $128, %edx
 ; SHRINK-WRAP-NEXT:  LBB0_2: # %entry
+; SHRINK-WRAP-NEXT:    pushl %ebp
+; SHRINK-WRAP-NEXT:    movl %esp, %ebp
+; SHRINK-WRAP-NEXT:    andl $-8, %esp
+; SHRINK-WRAP-NEXT:    subl $16, %esp
 ; SHRINK-WRAP-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; SHRINK-WRAP-NEXT:    cmpl $32, %eax
 ; SHRINK-WRAP-NEXT:    jl LBB0_4
 ; SHRINK-WRAP-NEXT:  # %bb.3: # %if.end
 ; SHRINK-WRAP-NEXT:    leal {{[0-9]+}}(%esp), %eax
 ; SHRINK-WRAP-NEXT:    leal {{[0-9]+}}(%esp), %edx
 ; SHRINK-WRAP-NEXT:    pushl %eax
 ; SHRINK-WRAP-NEXT:    pushl %edx

llvm/test/CodeGen/X86/inline-asm-flag-output.ll

 ; X32-NEXT:    jbe .LBB28_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB28_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_cca_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jbe .LBB28_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB28_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@cca},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jb .LBB29_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB29_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccae_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jb .LBB29_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB29_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccae},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jae .LBB30_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB30_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccb_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jae .LBB30_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB30_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccb},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    ja .LBB31_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB31_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccbe_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    ja .LBB31_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB31_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccbe},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jae .LBB32_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB32_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccc_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jae .LBB32_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB32_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccc},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jne .LBB33_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB33_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_cce_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jne .LBB33_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB33_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@cce},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jne .LBB34_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB34_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccz_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jne .LBB34_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB34_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccz},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jle .LBB35_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB35_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccg_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jle .LBB35_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB35_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccg},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jl .LBB36_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB36_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccge_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jl .LBB36_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB36_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccge},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jge .LBB37_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB37_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccl_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jge .LBB37_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB37_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccl},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jg .LBB38_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB38_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccle_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jg .LBB38_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB38_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccle},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    ja .LBB39_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB39_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccna_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    ja .LBB39_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB39_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccna},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jae .LBB40_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB40_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnae_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jae .LBB40_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB40_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnae},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jb .LBB41_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB41_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnb_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jb .LBB41_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB41_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnb},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jbe .LBB42_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB42_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnbe_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jbe .LBB42_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB42_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnbe},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jb .LBB43_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB43_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnc_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jb .LBB43_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB43_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnc},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    je .LBB44_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB44_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccne_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    je .LBB44_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB44_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccne},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    je .LBB45_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB45_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnz_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    je .LBB45_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB45_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnz},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jg .LBB46_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB46_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccng_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jg .LBB46_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB46_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccng},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jge .LBB47_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB47_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnge_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jge .LBB47_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB47_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnge},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jl .LBB48_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB48_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnl_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jl .LBB48_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB48_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnl},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jle .LBB49_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB49_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnle_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jle .LBB49_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB49_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnle},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jo .LBB50_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB50_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccno_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jo .LBB50_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB50_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccno},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jp .LBB51_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB51_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccnp_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jp .LBB51_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB51_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccnp},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    js .LBB52_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB52_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccns_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    js .LBB52_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB52_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccns},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jno .LBB53_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB53_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_cco_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jno .LBB53_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB53_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@cco},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jnp .LBB54_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB54_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccp_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jnp .LBB54_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB54_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccp},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()

 ; X32-NEXT:    jns .LBB55_2
 ; X32-NEXT:  # %bb.1: # %then
 ; X32-NEXT:    calll bar@PLT
 ; X32-NEXT:  .LBB55_2: # %exit
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_ccs_branch:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    #APP
 ; X64-NEXT:    cmp %rdi,(%rsi)
 ; X64-NEXT:    #NO_APP
 ; X64-NEXT:    jns .LBB55_2
 ; X64-NEXT:  # %bb.1: # %then
+; X64-NEXT:    pushq %rax
 ; X64-NEXT:    callq bar@PLT
+; X64-NEXT:    addq $8, %rsp
 ; X64-NEXT:  .LBB55_2: # %exit
-; X64-NEXT:    popq %rax
 ; X64-NEXT:    retq
 entry:
   %cc = tail call i8 asm "cmp $2,$1", "={@ccs},=*m,r,~{cc},~{dirflag},~{fpsr},~{flags}"(ptr elementtype(i64) %addr, i64 %nr) nounwind
   %tobool = icmp ne i8 %cc, 0
   br i1 %tobool, label %then, label %exit
 
 then:
   call void @bar()
   br label %exit
 
 exit:
   ret void
 }

llvm/test/CodeGen/X86/peep-test-5.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -o - %s -mtriple=x86_64-- | FileCheck %s
 ; Example of a decref operation with "immortal" objects.
 ; void decref(long* refcount) {
 ;  long count = *refcount;
 ;  if (count == 1) { free_object() }
 ;  else if (count > 1) { *refcount = count - 1; }
 ;  else { /* immortal */ }
 ; }
 ; Resulting assembly should share flags from single CMP instruction for both
 ; conditions!
 define void @decref(ptr %p) {
 ; CHECK-LABEL: decref:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pushq %rax
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    movl (%rdi), %eax
 ; CHECK-NEXT:    cmpl $1, %eax
 ; CHECK-NEXT:    jne .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %bb_free
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    callq free_object@PLT
-; CHECK-NEXT:  .LBB0_4: # %end
-; CHECK-NEXT:    popq %rax
+; CHECK-NEXT:    addq $8, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:  .LBB0_4: # %end
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  .LBB0_2: # %bb2
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    jle .LBB0_4
 ; CHECK-NEXT:  # %bb.3: # %bb_dec
 ; CHECK-NEXT:    decl %eax
 ; CHECK-NEXT:    movl %eax, (%rdi)
-; CHECK-NEXT:    popq %rax
-; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %count = load i32, ptr %p, align 4
   %cmp0 = icmp eq i32 %count, 1
   br i1 %cmp0, label %bb_free, label %bb2
 
 bb2:
   %cmp1 = icmp sgt i32 %count, 1
   br i1 %cmp1, label %bb_dec, label %end

llvm/test/CodeGen/X86/shrink-wrap-chkstk-x86_64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64-windows-gnu -exception-model=dwarf < %s | FileCheck %s
 
 %struct.A = type { [4096 x i8] }
 
 @a = common global i32 0, align 4
 @b = common global i32 0, align 4
 
 define void @fn1() nounwind uwtable {
 ; CHECK-LABEL: fn1:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl $4136, %eax # imm = 0x1028
-; CHECK-NEXT:    callq ___chkstk_ms
-; CHECK-NEXT:    subq %rax, %rsp
-; CHECK-NEXT:    .cfi_def_cfa_offset 4144
 ; CHECK-NEXT:    movl a(%rip), %eax
 ; CHECK-NEXT:    testl %eax, %eax
 ; CHECK-NEXT:    jne .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %select.true.sink
 ; CHECK-NEXT:    cltq
 ; CHECK-NEXT:    imulq $715827883, %rax, %rax # imm = 0x2AAAAAAB
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $63, %rcx
 ; CHECK-NEXT:    shrq $32, %rax
 ; CHECK-NEXT:    addl %ecx, %eax
 ; CHECK-NEXT:  .LBB0_2: # %select.end
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    movl $4128, %eax # imm = 0x1020
+; CHECK-NEXT:    callq ___chkstk_ms
+; CHECK-NEXT:    subq %rax, %rsp
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 4144
 ; CHECK-NEXT:    movl %eax, b(%rip)
 ; CHECK-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; CHECK-NEXT:    # kill: def $ecx killed $ecx killed $rcx
 ; CHECK-NEXT:    callq fn2
 ; CHECK-NEXT:    addq $4136, %rsp # imm = 0x1028
 ; CHECK-NEXT:    retq
 entry:
   %ctx = alloca %struct.A, align 1

llvm/test/CodeGen/X86/statepoint-vector.ll

 ; - We waste a stack slot for a value that a backend transform pass
 ;   CSEd to another spilled one.
 ; - We don't remove the testb even though it serves no purpose
 ; - We could in principal reuse the argument memory (%rsi) and do away
 ;   with stack slots entirely.
 define <2 x ptr addrspace(1)> @test3(i1 %cnd, ptr %ptr) gc "statepoint-example" {
 ; CHECK-LABEL: test3:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    subq $40, %rsp
-; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    movaps (%rsi), %xmm0
+; CHECK-NEXT:    subq $40, %rsp
+; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    movaps %xmm0, (%rsp)
 ; CHECK-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    callq do_safepoint@PLT
 ; CHECK-NEXT:  .Ltmp2:
 ; CHECK-NEXT:    movaps (%rsp), %xmm0
 ; CHECK-NEXT:    addq $40, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq

llvm/test/DebugInfo/X86/merge_inlined_loc.ll

 ;     wrapper();
 ; }
 ; void f1() { wrapper2(); }
 
 ; Ensure there are two inlined_subroutine (for wrapper and wrapper2).
 
 ; CHECK: .loc 1 2 25 epilogue_begin
 ; CHECK-NEXT: popq %rax
+; CHECK-NEXT: .Ltmp{{.*}}:
 ; CHECK-NEXT: .cfi_def_cfa_offset 8
+; CHECK-NEXT: .loc 1 2 25 is_stmt 0
 ; CHECK-NEXT: jmp _Z6commonv
 ; CHECK-NEXT: [[LABEL:.*]]:
 
 ; CHECK: .section .debug_info
 ; CHECK: DW_TAG_subprogram
 ; CHECK: DW_TAG_subprogram
 ; CHECK-NOT: {{DW_TAG\|End Of Children}}
 ; CHECK:   DW_TAG_inlined_subroutine