[PowerPC] Make sure the first probe is full size or is the last probe when stack is realigned

[lkail]

Summary

When -fstack-clash-protection is enabled and stack has to be realigned, some parts of redzone is written prior the probe, so probe might overwrite content already written in redzone. To avoid it, we have to make sure the first probe is at full probe size or is the last probe so that we can skip redzone.

It also fixes violation of ABI under PPC where r1 isn't updated atomically.

This fixes https://bugs.llvm.org/show_bug.cgi?id=49903.

[lkail]

Current solution hasn't guaranteed back-chain pointer is updated atomically, I have another solution for this to guarantee safety and atomic.

[lkail]

llvm/test/CodeGen/PowerPC/stack-clash-prologue.ll

681

Old implementation doesn't make backchain pointer updated atomically, violating ABI.

[tstellar]

What is the status of this patch? It fixes a blocking bug for the 12.0.1 release.

[nemanjai]

In D100290#2799987, @tstellar wrote:

What is the status of this patch? It fixes a blocking bug for the 12.0.1 release.

@jsji Do you have a bit of time to review this? If not, we might have to mark the bug as a non-blocker for the release and accept that stack clash protection has this issue in 12.0.0 and 12.0.1. I am hoping we can get this done and backported.

[jsji]

This somehow was missed in my radar.. I will have a look .

[lkail]

1. Adjust comments
2. Swap usage of r0 and r12
3. Shortened liverange of FinalStackPtr

[lkail]

llvm/test/CodeGen/PowerPC/stack-clash-prologue.ll

89

Looks like a bug... I'll check it.

[lkail]

Rebased on https://reviews.llvm.org/D103761

[jsji]

Looks mostly good to me, but I would prefer we split this into two fixes, one for the realign bp algorithm change, another for the other.

llvm/lib/Target/PowerPC/PPCFrameLowering.cpp

1284

*that* -> *when*?

1285

Add the comments about atomic update required by ABI.

1285

maybe rename dynamicProbeto something like dynamicProbe_RealignWithBP as this is only called with HasBP && MaxAlign >1?

1289

This is inconsistent of what we generate below: using stdu <backchain>, <negprobesize>(r1)?

1292

If we use stdux above in pseudo code, shouldn't we use stdux here?

1293

Reorg this comment please. HasBP is always true, so this is really the difference for isPPC64 vs isPPC32?

1315

This looks like a new limitation? Can we add a FIXME here?

1321

Do you mean we can use xform if we have an additional idle register?

1325

*materialized* -> *materializable*?

1326

Do we still need this lambda? Seems like only a ternary operator here?

1361

FPReg? Typo? TempReg?

1404

true /*UseDForm*/

1548

Split change related to this into another patch?

[lkail]

Address comments.

[lkail]

llvm/lib/Target/PowerPC/PPCFrameLowering.cpp

1293

Should be HasBP && HasRedzone, I have updated the condition.

1321

Exactly.

1548

Yes, we can do it later.

[jsji]

LGTM , with some comments on comments.

llvm/lib/Target/PowerPC/PPCFrameLowering.cpp

1300

How about?

When HasBP & HasRedzone, back-chain pointer is already saved in BPReg before probe code, we don't need to save it, so we get one additional reg that can be used to materialize the probeside if needed to use xform.
Otherwise, we can NOT materialize probeside, so we can only use Dform for now. 

The allocations are:
if(HasBP & HasRedzone){
r0: materialize the probesize if needed so that we can use xform.
r12: `neg_gap`
}
else{
r0: back-chain pointer
r12: `neg_gap`.
}

1307

probeStackWhenRealigned -> probeRealignedStack

1326

probeStackWhenRealigned is called only under if (HasBP && MaxAlign > 1) { ,
so the HasBP should be expected to be always true in this function,
I think we can safely remove it from the condition? (you can add an assert if you want).

1326

we should have a idle register to materialize -> we can use xform if we have an additional idle register

1360

PROBED_STACKALLOC_64 assumes Operand(1) stores the old sp, copy BPReg to TempReg to satisfy it.

llvm/test/CodeGen/PowerPC/stack-clash-prologue-nounwind.ll

337 ↗

(On Diff #350482)

Looks like this can be ignored if we found the imm is 0. Can be fixed in a follow up patch.

[lkail]

llvm/test/CodeGen/PowerPC/stack-clash-prologue-nounwind.ll

337 ↗

(On Diff #350482)

Good catch.

[lkail]

Try to fix pre-merge build failure on windows.

llvm/lib/Target/PowerPC/PPCFrameLowering.cpp

for (unsigned I = 0, E = CSI.size(); I != E; ++I) {
             .addCFIIndex(CFIIndex);
       }
     }
   }
 }
 
 void PPCFrameLowering::inlineStackProbe(MachineFunction &MF,
                                         MachineBasicBlock &PrologMBB) const {
-  // TODO: Generate CFI instructions.
   bool isPPC64 = Subtarget.isPPC64();
   const PPCTargetLowering &TLI = *Subtarget.getTargetLowering();
   const PPCInstrInfo &TII = *Subtarget.getInstrInfo();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   MachineModuleInfo &MMI = MF.getMMI();
   const MCRegisterInfo *MRI = MMI.getContext().getRegisterInfo();
   // AIX assembler does not support cfi directives.
   const bool needsCFI = MF.needsFrameMoves() && !Subtarget.isAIXABI();
void PPCFrameLowering::inlineStackProbe(MachineFunction &MF,
   // available and r1 is already copied to r30 which is BPReg. So BPReg stores
   // the value of stackptr.
   // First we have to probe tail interval whose size is less than probesize,
   // i.e., [stackptr - (stackptr % align) % probesize, stackptr). At this stage,
   // ScratchReg stores the value of ((stackptr % align) % probesize). Then we
   // probe each block sized probesize until stackptr meets
   // (stackptr - (stackptr % align)). At this stage, ScratchReg is materialized
   // as negprobesize. At both stages, TempReg stores the value of
   // (stackptr - (stackptr % align)).
-  auto dynamicProbe = [&](MachineBasicBlock &MBB,
+  auto ProbeRealignGap = [&](MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MBBI, Register ScratchReg,
                           Register TempReg) {
     assert(HasBP && isPPC64 && "Probe alignment part not available");
     assert(isPowerOf2_64(ProbeSize) && "Probe size should be power of 2");
     // ScratchReg = stackptr % align
     BuildMI(MBB, MBBI, DL, TII.get(PPC::RLDICL), ScratchReg)
         .addReg(BPReg)
         .addImm(0)
         .addImm(64 - Log2(MaxAlign));
+    // When MaxAlign <= Subtarget.getRedZoneSize(), we already have stored BP
+    // in the redzone of the stack which is considered to be a free probe, so
+    // we don't need to perform intentional probe at such situation.
+    const unsigned RedZoneSize = Subtarget.getRedZoneSize();
+    assert(ProbeSize >= RedZoneSize &&
+           "ProbeSize should be larger than size of redzone");
+    if (MaxAlign <= RedZoneSize) {
+      BuildMI(MBB, MBBI, DL, TII.get(PPC::SUBF8), SPReg)
+          .addReg(ScratchReg)
+          .addReg(BPReg);
+      return &MBB;
+    }
     // TempReg = stackptr - (stackptr % align)
-    BuildMI(MBB, MBBI, DL, TII.get(PPC::SUBFC8), TempReg)
+    BuildMI(MBB, MBBI, DL, TII.get(PPC::SUBF8), TempReg)
         .addReg(ScratchReg)
         .addReg(BPReg);
     // ScratchReg = (stackptr % align) % probesize
     BuildMI(MBB, MBBI, DL, TII.get(PPC::RLDICL), ScratchReg)
         .addReg(ScratchReg)
         .addImm(0)
         .addImm(64 - Log2(ProbeSize));
     Register CRReg = PPC::CR0;
-    // If (stackptr % align) % probesize == 0, we should not generate probe
-    // code. Layout of output assembly kinda like:
+    // If (stackptr % align) % probesize < redzonesize, we don't need to
+    // generate probe code.
+    // Layout of output assembly kinda like:
     // bb.0:
     //   ...
-    //   cmpldi $scratchreg, 0
-    //   beq bb.2
-    // bb.1: # Probe tail interval
+    //   cmpldi $scratchreg, <redzonesize>
+    //   bge bb.2
+    // bb.1: # Skip tail interval < redzoneszie.
+    //   sub r1, r1, $scratchreg
+    //   b bb.3
+    // bb.2: # Probe tail interval >= redzonesize.
     //   neg $scratchreg, $scratchreg
     //   stdux $bpreg, r1, $scratchreg
-    // bb.2:
+    // bb.3:
     //   <materialize negprobesize into $scratchreg>
     //   cmpd r1, $tempreg
-    //   beq bb.4
-    // bb.3: # Loop to probe each block
+    //   beq bb.5
+    // bb.4: # Loop to probe each block
     //   stdux $bpreg, r1, $scratchreg
     //   cmpd r1, $tempreg
-    //   bne bb.3
-    // bb.4:
+    //   bne bb.4
+    // bb.5:
     //   ...
     MachineFunction::iterator MBBInsertPoint = std::next(MBB.getIterator());
+    MachineBasicBlock *SkipResidualMBB = MF.CreateMachineBasicBlock(ProbedBB);
+    MF.insert(MBBInsertPoint, SkipResidualMBB);
     MachineBasicBlock *ProbeResidualMBB = MF.CreateMachineBasicBlock(ProbedBB);
     MF.insert(MBBInsertPoint, ProbeResidualMBB);
     MachineBasicBlock *ProbeLoopPreHeaderMBB =
         MF.CreateMachineBasicBlock(ProbedBB);
     MF.insert(MBBInsertPoint, ProbeLoopPreHeaderMBB);
     MachineBasicBlock *ProbeLoopBodyMBB = MF.CreateMachineBasicBlock(ProbedBB);
     MF.insert(MBBInsertPoint, ProbeLoopBodyMBB);
     MachineBasicBlock *ProbeExitMBB = MF.CreateMachineBasicBlock(ProbedBB);
     MF.insert(MBBInsertPoint, ProbeExitMBB);
-    // bb.4
+    // bb.5
     ProbeExitMBB->splice(ProbeExitMBB->end(), &MBB, MBBI, MBB.end());
     ProbeExitMBB->transferSuccessorsAndUpdatePHIs(&MBB);
     // bb.0
-    BuildMI(&MBB, DL, TII.get(PPC::CMPDI), CRReg).addReg(ScratchReg).addImm(0);
+    BuildMI(&MBB, DL, TII.get(PPC::CMPDI), CRReg)
+        .addReg(ScratchReg)
+        .addImm(RedZoneSize);
     BuildMI(&MBB, DL, TII.get(PPC::BCC))
-        .addImm(PPC::PRED_EQ)
+        .addImm(PPC::PRED_GE)
         .addReg(CRReg)
-        .addMBB(ProbeLoopPreHeaderMBB);
+        .addMBB(ProbeResidualMBB);
     MBB.addSuccessor(ProbeResidualMBB);
-    MBB.addSuccessor(ProbeLoopPreHeaderMBB);
+    MBB.addSuccessor(SkipResidualMBB);
     // bb.1
+    BuildMI(SkipResidualMBB, DL, TII.get(PPC::SUBF8), SPReg)
+        .addReg(ScratchReg)
+        .addReg(SPReg);
+    BuildMI(SkipResidualMBB, DL, TII.get(PPC::B)).addMBB(ProbeLoopPreHeaderMBB);
+    SkipResidualMBB->addSuccessor(ProbeLoopPreHeaderMBB);
+    // bb.2
     BuildMI(ProbeResidualMBB, DL, TII.get(PPC::NEG8), ScratchReg)
         .addReg(ScratchReg);
     allocateAndProbe(*ProbeResidualMBB, ProbeResidualMBB->end(), 0, ScratchReg,
                      false, BPReg);
     ProbeResidualMBB->addSuccessor(ProbeLoopPreHeaderMBB);
-    // bb.2
+    // bb.3
     MaterializeImm(*ProbeLoopPreHeaderMBB, ProbeLoopPreHeaderMBB->end(),
                    NegProbeSize, ScratchReg);
     BuildMI(ProbeLoopPreHeaderMBB, DL, TII.get(PPC::CMPD), CRReg)
         .addReg(SPReg)
         .addReg(TempReg);
     BuildMI(ProbeLoopPreHeaderMBB, DL, TII.get(PPC::BCC))
         .addImm(PPC::PRED_EQ)
         .addReg(CRReg)
         .addMBB(ProbeExitMBB);
     ProbeLoopPreHeaderMBB->addSuccessor(ProbeLoopBodyMBB);
     ProbeLoopPreHeaderMBB->addSuccessor(ProbeExitMBB);
-    // bb.3
+    // bb.4
     allocateAndProbe(*ProbeLoopBodyMBB, ProbeLoopBodyMBB->end(), 0, ScratchReg,
                      false, BPReg);
     BuildMI(ProbeLoopBodyMBB, DL, TII.get(PPC::CMPD), CRReg)
         .addReg(SPReg)
         .addReg(TempReg);
     BuildMI(ProbeLoopBodyMBB, DL, TII.get(PPC::BCC))
         .addImm(PPC::PRED_NE)
         .addReg(CRReg)
         .addMBB(ProbeLoopBodyMBB);
     ProbeLoopBodyMBB->addSuccessor(ProbeExitMBB);
     ProbeLoopBodyMBB->addSuccessor(ProbeLoopBodyMBB);
     // Update liveins.
+    recomputeLiveIns(*SkipResidualMBB);
     recomputeLiveIns(*ProbeResidualMBB);
     recomputeLiveIns(*ProbeLoopPreHeaderMBB);
     recomputeLiveIns(*ProbeLoopBodyMBB);
     recomputeLiveIns(*ProbeExitMBB);
     return ProbeExitMBB;
   };
   // For case HasBP && MaxAlign > 1, we have to realign the SP by performing
   // SP = SP - SP % MaxAlign.
   if (HasBP && MaxAlign > 1) {
     // FIXME: Currently only probe the gap [stackptr & alignmask, stackptr) in
     // 64-bit mode.
     if (isPPC64) {
       // Use BPReg to calculate CFA.
       if (needsCFI)
         buildDefCFA(*CurrentMBB, {MI}, BPReg, 0);
-      // Since we have SPReg copied to BPReg at the moment, FPReg can be used as
-      // TempReg.
+      // Since we have SPReg copied to BPReg at the moment, FPReg can be used
+      // as TempReg.
       Register TempReg = FPReg;
-      CurrentMBB = dynamicProbe(*CurrentMBB, {MI}, ScratchReg, TempReg);
+      CurrentMBB = ProbeRealignGap(*CurrentMBB, {MI}, ScratchReg, TempReg);
       // Copy BPReg to FPReg to meet the definition of PROBED_STACKALLOC_64.
       BuildMI(*CurrentMBB, {MI}, DL, CopyInst, FPReg)
           .addReg(BPReg)
           .addReg(BPReg);
     } else {
       // Initialize current frame pointer.
       BuildMI(*CurrentMBB, {MI}, DL, CopyInst, FPReg)
           .addReg(SPReg)
void PPCFrameLowering::emitEpilogue(MachineFunction &MF,
   bool HasRedZone = Subtarget.isPPC64() || !Subtarget.isSVR4ABI();
 
   Register SPReg      = isPPC64 ? PPC::X1  : PPC::R1;
   Register BPReg = RegInfo->getBaseRegister(MF);
   Register FPReg      = isPPC64 ? PPC::X31 : PPC::R31;
   Register ScratchReg;
   Register TempReg     = isPPC64 ? PPC::X12 : PPC::R12; // another scratch reg
   const MCInstrDesc& MTLRInst = TII.get( isPPC64 ? PPC::MTLR8
                                                  : PPC::MTLR );
   const MCInstrDesc& LoadInst = TII.get( isPPC64 ? PPC::LD
                                                  : PPC::LWZ );
   const MCInstrDesc& LoadImmShiftedInst = TII.get( isPPC64 ? PPC::LIS8
                                                            : PPC::LIS );
   const MCInstrDesc& OrInst = TII.get(isPPC64 ? PPC::OR8
                                               : PPC::OR );
   const MCInstrDesc& OrImmInst = TII.get( isPPC64 ? PPC::ORI8
                                                   : PPC::ORI );

llvm/test/CodeGen/PowerPC/pr46759.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -ppc-asm-full-reg-names -verify-machineinstrs \
 ; RUN:   -mtriple=powerpc64le-linux-gnu < %s | FileCheck \
 ; RUN:   -check-prefix=CHECK-LE %s
 
 define void @foo(i32 %vla_size) #0 {
 ; CHECK-LE-LABEL: foo:
 ; CHECK-LE:       # %bb.0: # %entry
 ; CHECK-LE-NEXT:    std r31, -8(r1)
 ; CHECK-LE-NEXT:    std r30, -16(r1)
 ; CHECK-LE-NEXT:    mr r30, r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-LE-NEXT:    clrldi r0, r30, 53
-; CHECK-LE-NEXT:    subc r12, r30, r0
+; CHECK-LE-NEXT:    sub r12, r30, r0
 ; CHECK-LE-NEXT:    clrldi r0, r0, 52
-; CHECK-LE-NEXT:    cmpdi r0, 0
-; CHECK-LE-NEXT:    beq cr0, .LBB0_2
+; CHECK-LE-NEXT:    cmpdi r0, 288
+; CHECK-LE-NEXT:    blt cr0, .LBB0_2
 ; CHECK-LE-NEXT:  # %bb.1: # %entry
 ; CHECK-LE-NEXT:    neg r0, r0
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:    li r0, -4096
+; CHECK-LE-NEXT:    b .LBB0_4
 ; CHECK-LE-NEXT:  .LBB0_2: # %entry
+; CHECK-LE-NEXT:    sub r1, r1, r0
 ; CHECK-LE-NEXT:    li r0, -4096
-; CHECK-LE-NEXT:    cmpd r1, r12
-; CHECK-LE-NEXT:    beq cr0, .LBB0_4
+; CHECK-LE-NEXT:    b .LBB0_4
+; CHECK-LE-NEXT:    .p2align 4
 ; CHECK-LE-NEXT:  .LBB0_3: # %entry
 ; CHECK-LE-NEXT:    #
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:  .LBB0_4: # %entry
+; CHECK-LE-NEXT:    #
 ; CHECK-LE-NEXT:    cmpd r1, r12
 ; CHECK-LE-NEXT:    bne cr0, .LBB0_3
-; CHECK-LE-NEXT:  .LBB0_4: # %entry
+; CHECK-LE-NEXT:  # %bb.5: # %entry
 ; CHECK-LE-NEXT:    mr r12, r30
 ; CHECK-LE-NEXT:    stdu r12, -2048(r1)
 ; CHECK-LE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-LE-NEXT:    .cfi_offset r31, -8
 ; CHECK-LE-NEXT:    .cfi_offset r30, -16
 ; CHECK-LE-NEXT:    clrldi r3, r3, 32
 ; CHECK-LE-NEXT:    li r5, -2048
 ; CHECK-LE-NEXT:    mr r31, r1
 ; CHECK-LE-NEXT:    addi r3, r3, 15
 ; CHECK-LE-NEXT:    rldicl r3, r3, 60, 4
 ; CHECK-LE-NEXT:    rldicl r3, r3, 4, 31
 ; CHECK-LE-NEXT:    neg r4, r3
 ; CHECK-LE-NEXT:    ld r3, 0(r1)
 ; CHECK-LE-NEXT:    and r5, r4, r5
 ; CHECK-LE-NEXT:    mr r4, r5
 ; CHECK-LE-NEXT:    li r5, -4096
 ; CHECK-LE-NEXT:    divd r6, r4, r5
 ; CHECK-LE-NEXT:    mulld r5, r6, r5
 ; CHECK-LE-NEXT:    sub r5, r4, r5
 ; CHECK-LE-NEXT:    add r4, r1, r4
 ; CHECK-LE-NEXT:    stdux r3, r1, r5
 ; CHECK-LE-NEXT:    cmpd r1, r4
-; CHECK-LE-NEXT:    beq cr0, .LBB0_6
-; CHECK-LE-NEXT:  .LBB0_5: # %entry
+; CHECK-LE-NEXT:    beq cr0, .LBB0_7
+; CHECK-LE-NEXT:  .LBB0_6: # %entry
 ; CHECK-LE-NEXT:    #
 ; CHECK-LE-NEXT:    stdu r3, -4096(r1)
 ; CHECK-LE-NEXT:    cmpd r1, r4
-; CHECK-LE-NEXT:    bne cr0, .LBB0_5
-; CHECK-LE-NEXT:  .LBB0_6: # %entry
+; CHECK-LE-NEXT:    bne cr0, .LBB0_6
+; CHECK-LE-NEXT:  .LBB0_7: # %entry
 ; CHECK-LE-NEXT:    addi r3, r1, 2048
 ; CHECK-LE-NEXT:    lbz r3, 0(r3)
 ; CHECK-LE-NEXT:    mr r1, r30
 ; CHECK-LE-NEXT:    ld r31, -8(r1)
 ; CHECK-LE-NEXT:    ld r30, -16(r1)
 ; CHECK-LE-NEXT:    blr
 entry:
   %0 = zext i32 %vla_size to i64
   %vla = alloca i8, i64 %0, align 2048
   %1 = load volatile i8, i8* %vla, align 2048
   ret void
 }
 
 attributes #0 = { "probe-stack"="inline-asm" }

llvm/test/CodeGen/PowerPC/stack-clash-prologue.ll

 ; CHECK-BE-NEXT:    blr
 ;
 ; CHECK-32-LABEL: f1:
 ; CHECK-32:       # %bb.0: # %entry
 ; CHECK-32-NEXT:    mr r12, r1
 ; CHECK-32-NEXT:    .cfi_def_cfa r12, 0
 ; CHECK-32-NEXT:    li r0, 257
 ; CHECK-32-NEXT:    mtctr r0
 ; CHECK-32-NEXT:  .LBB1_1: # %entry
 ; CHECK-32-NEXT:    #
 ; CHECK-32-NEXT:    stwu r12, -16(r1)
 ; CHECK-32-NEXT:    bdnz .LBB1_1
 ; CHECK-32-NEXT:  # %bb.2: # %entry
 ; CHECK-32-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-32-NEXT:    sub r0, r1, r12
 ; CHECK-32-NEXT:    sub r0, r1, r0
 ; CHECK-32-NEXT:    .cfi_def_cfa_offset 4112

 ; alloca > probe_size, align > probe_size
 define i32 @f9(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LE-LABEL: f9:
 ; CHECK-LE:       # %bb.0:
 ; CHECK-LE-NEXT:    std r30, -16(r1)
 ; CHECK-LE-NEXT:    mr r30, r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-LE-NEXT:    clrldi r0, r30, 53
-; CHECK-LE-NEXT:    subc r12, r30, r0
+; CHECK-LE-NEXT:    sub r12, r30, r0
 ; CHECK-LE-NEXT:    clrldi r0, r0, 52
-; CHECK-LE-NEXT:    cmpdi r0, 0
-; CHECK-LE-NEXT:    beq cr0, .LBB9_2
+; CHECK-LE-NEXT:    cmpdi r0, 288
+; CHECK-LE-NEXT:    blt cr0, .LBB9_2
 ; CHECK-LE-NEXT:  # %bb.1:
 ; CHECK-LE-NEXT:    neg r0, r0
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:    li r0, -4096
+; CHECK-LE-NEXT:    b .LBB9_4
 ; CHECK-LE-NEXT:  .LBB9_2:
+; CHECK-LE-NEXT:    sub r1, r1, r0
 ; CHECK-LE-NEXT:    li r0, -4096
-; CHECK-LE-NEXT:    cmpd r1, r12
-; CHECK-LE-NEXT:    beq cr0, .LBB9_4
+; CHECK-LE-NEXT:    b .LBB9_4
+; CHECK-LE-NEXT:    .p2align 4
 ; CHECK-LE-NEXT:  .LBB9_3:
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:  .LBB9_4:
 ; CHECK-LE-NEXT:    cmpd r1, r12
 ; CHECK-LE-NEXT:    bne cr0, .LBB9_3
-; CHECK-LE-NEXT:  .LBB9_4:
+; CHECK-LE-NEXT:  # %bb.5:
 ; CHECK-LE-NEXT:    mr r12, r30
 ; CHECK-LE-NEXT:    stdu r12, -2048(r1)
 ; CHECK-LE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-LE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-LE-NEXT:    .cfi_offset r30, -16
 ; CHECK-LE-NEXT:    addi r4, r1, 2048
 ; CHECK-LE-NEXT:    sldi r3, r3, 2
 ; CHECK-LE-NEXT:    li r5, 1
 ; CHECK-LE-NEXT:    stwx r5, r4, r3
 ; CHECK-LE-NEXT:    lwz r3, 2048(r1)
 ; CHECK-LE-NEXT:    mr r1, r30
 ; CHECK-LE-NEXT:    ld r30, -16(r1)
 ; CHECK-LE-NEXT:    blr
 ;
 ; CHECK-BE-LABEL: f9:
 ; CHECK-BE:       # %bb.0:
 ; CHECK-BE-NEXT:    std r30, -16(r1)
 ; CHECK-BE-NEXT:    mr r30, r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-BE-NEXT:    clrldi r0, r30, 53
-; CHECK-BE-NEXT:    subc r12, r30, r0
+; CHECK-BE-NEXT:    sub r12, r30, r0
 ; CHECK-BE-NEXT:    clrldi r0, r0, 52
-; CHECK-BE-NEXT:    cmpdi r0, 0
-; CHECK-BE-NEXT:    beq cr0, .LBB9_2
+; CHECK-BE-NEXT:    cmpdi r0, 288
+; CHECK-BE-NEXT:    blt cr0, .LBB9_2
 ; CHECK-BE-NEXT:  # %bb.1:
 ; CHECK-BE-NEXT:    neg r0, r0
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:    li r0, -4096
+; CHECK-BE-NEXT:    b .LBB9_4
 ; CHECK-BE-NEXT:  .LBB9_2:
+; CHECK-BE-NEXT:    sub r1, r1, r0
 ; CHECK-BE-NEXT:    li r0, -4096
-; CHECK-BE-NEXT:    cmpd r1, r12
-; CHECK-BE-NEXT:    beq cr0, .LBB9_4
+; CHECK-BE-NEXT:    b .LBB9_4
 ; CHECK-BE-NEXT:  .LBB9_3:
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:  .LBB9_4:
 ; CHECK-BE-NEXT:    cmpd r1, r12
 ; CHECK-BE-NEXT:    bne cr0, .LBB9_3
-; CHECK-BE-NEXT:  .LBB9_4:
+; CHECK-BE-NEXT:  # %bb.5:
 ; CHECK-BE-NEXT:    mr r12, r30
 ; CHECK-BE-NEXT:    stdu r12, -2048(r1)
 ; CHECK-BE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-BE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-BE-NEXT:    .cfi_offset r30, -16
 ; CHECK-BE-NEXT:    addi r4, r1, 2048
 ; CHECK-BE-NEXT:    li r5, 1
 ; CHECK-BE-NEXT:    sldi r3, r3, 2
 ; CHECK-BE-NEXT:    stwx r5, r4, r3
 ; CHECK-BE-NEXT:    lwz r3, 2048(r1)
 ; CHECK-BE-NEXT:    mr r1, r30
 ; CHECK-BE-NEXT:    ld r30, -16(r1)
 ; CHECK-BE-NEXT:    blr
 ;
 ; CHECK-32-LABEL: f9:
 ; CHECK-32:       # %bb.0:
 ; CHECK-32-NEXT:    mr r12, r1
 ; CHECK-32-NEXT:    .cfi_def_cfa r12, 0
 ; CHECK-32-NEXT:    clrlwi r0, r12, 21
 ; CHECK-32-NEXT:    subc r1, r1, r0
 ; CHECK-32-NEXT:    stwu r12, -2048(r1)
 ; CHECK-32-NEXT:    stwu r12, -4096(r1)
 ; CHECK-32-NEXT:    stwu r12, -4096(r1)
 ; CHECK-32-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-32-NEXT:    sub r0, r1, r12
 ; CHECK-32-NEXT:    sub r0, r1, r0
 ; CHECK-32-NEXT:    addic r0, r0, -8
 ; CHECK-32-NEXT:    stwx r30, 0, r0

 ; alloca < probe_size, align < probe_size, alloca + align > probe_size
 define i32 @f10(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LE-LABEL: f10:
 ; CHECK-LE:       # %bb.0:
 ; CHECK-LE-NEXT:    std r30, -16(r1)
 ; CHECK-LE-NEXT:    mr r30, r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-LE-NEXT:    clrldi r0, r30, 54
-; CHECK-LE-NEXT:    subc r12, r30, r0
+; CHECK-LE-NEXT:    sub r12, r30, r0
 ; CHECK-LE-NEXT:    clrldi r0, r0, 52
-; CHECK-LE-NEXT:    cmpdi r0, 0
-; CHECK-LE-NEXT:    beq cr0, .LBB10_2
+; CHECK-LE-NEXT:    cmpdi r0, 288
+; CHECK-LE-NEXT:    blt cr0, .LBB10_2
 ; CHECK-LE-NEXT:  # %bb.1:
 ; CHECK-LE-NEXT:    neg r0, r0
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:    li r0, -4096
+; CHECK-LE-NEXT:    b .LBB10_4
 ; CHECK-LE-NEXT:  .LBB10_2:
+; CHECK-LE-NEXT:    sub r1, r1, r0
 ; CHECK-LE-NEXT:    li r0, -4096
-; CHECK-LE-NEXT:    cmpd r1, r12
-; CHECK-LE-NEXT:    beq cr0, .LBB10_4
+; CHECK-LE-NEXT:    b .LBB10_4
+; CHECK-LE-NEXT:    .p2align 4
 ; CHECK-LE-NEXT:  .LBB10_3:
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:  .LBB10_4:
 ; CHECK-LE-NEXT:    cmpd r1, r12
 ; CHECK-LE-NEXT:    bne cr0, .LBB10_3
-; CHECK-LE-NEXT:  .LBB10_4:
+; CHECK-LE-NEXT:  # %bb.5:
 ; CHECK-LE-NEXT:    mr r12, r30
 ; CHECK-LE-NEXT:    stdu r12, -1024(r1)
 ; CHECK-LE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-LE-NEXT:    .cfi_offset r30, -16
 ; CHECK-LE-NEXT:    addi r4, r1, 1024
 ; CHECK-LE-NEXT:    sldi r3, r3, 2
 ; CHECK-LE-NEXT:    li r5, 1
 ; CHECK-LE-NEXT:    stwx r5, r4, r3
 ; CHECK-LE-NEXT:    lwz r3, 1024(r1)
 ; CHECK-LE-NEXT:    mr r1, r30
 ; CHECK-LE-NEXT:    ld r30, -16(r1)
 ; CHECK-LE-NEXT:    blr
 ;
 ; CHECK-BE-LABEL: f10:
 ; CHECK-BE:       # %bb.0:
 ; CHECK-BE-NEXT:    std r30, -16(r1)
 ; CHECK-BE-NEXT:    mr r30, r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-BE-NEXT:    clrldi r0, r30, 54
-; CHECK-BE-NEXT:    subc r12, r30, r0
+; CHECK-BE-NEXT:    sub r12, r30, r0
 ; CHECK-BE-NEXT:    clrldi r0, r0, 52
-; CHECK-BE-NEXT:    cmpdi r0, 0
-; CHECK-BE-NEXT:    beq cr0, .LBB10_2
+; CHECK-BE-NEXT:    cmpdi r0, 288
+; CHECK-BE-NEXT:    blt cr0, .LBB10_2
 ; CHECK-BE-NEXT:  # %bb.1:
 ; CHECK-BE-NEXT:    neg r0, r0
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:    li r0, -4096
+; CHECK-BE-NEXT:    b .LBB10_4
 ; CHECK-BE-NEXT:  .LBB10_2:
+; CHECK-BE-NEXT:    sub r1, r1, r0
 ; CHECK-BE-NEXT:    li r0, -4096
-; CHECK-BE-NEXT:    cmpd r1, r12
-; CHECK-BE-NEXT:    beq cr0, .LBB10_4
+; CHECK-BE-NEXT:    b .LBB10_4
 ; CHECK-BE-NEXT:  .LBB10_3:
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:  .LBB10_4:
 ; CHECK-BE-NEXT:    cmpd r1, r12
 ; CHECK-BE-NEXT:    bne cr0, .LBB10_3
-; CHECK-BE-NEXT:  .LBB10_4:
+; CHECK-BE-NEXT:  # %bb.5:
 ; CHECK-BE-NEXT:    mr r12, r30
 ; CHECK-BE-NEXT:    stdu r12, -1024(r1)
 ; CHECK-BE-NEXT:    stdu r12, -4096(r1)
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-BE-NEXT:    .cfi_offset r30, -16
 ; CHECK-BE-NEXT:    addi r4, r1, 1024
 ; CHECK-BE-NEXT:    li r5, 1

 define void @f11(i32 %vla_size, i64 %i) #0 {
 ; CHECK-LE-LABEL: f11:
 ; CHECK-LE:       # %bb.0:
 ; CHECK-LE-NEXT:    std r31, -8(r1)
 ; CHECK-LE-NEXT:    std r30, -16(r1)
 ; CHECK-LE-NEXT:    mr r30, r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-LE-NEXT:    clrldi r0, r30, 49
-; CHECK-LE-NEXT:    subc r12, r30, r0
+; CHECK-LE-NEXT:    sub r12, r30, r0
 ; CHECK-LE-NEXT:    clrldi r0, r0, 52
-; CHECK-LE-NEXT:    cmpdi r0, 0
-; CHECK-LE-NEXT:    beq cr0, .LBB11_2
+; CHECK-LE-NEXT:    cmpdi r0, 288
+; CHECK-LE-NEXT:    blt cr0, .LBB11_2
 ; CHECK-LE-NEXT:  # %bb.1:
 ; CHECK-LE-NEXT:    neg r0, r0
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:    li r0, -4096
+; CHECK-LE-NEXT:    b .LBB11_4
 ; CHECK-LE-NEXT:  .LBB11_2:
+; CHECK-LE-NEXT:    sub r1, r1, r0
 ; CHECK-LE-NEXT:    li r0, -4096
-; CHECK-LE-NEXT:    cmpd r1, r12
-; CHECK-LE-NEXT:    beq cr0, .LBB11_4
+; CHECK-LE-NEXT:    b .LBB11_4
+; CHECK-LE-NEXT:    .p2align 4
 ; CHECK-LE-NEXT:  .LBB11_3:
 ; CHECK-LE-NEXT:    stdux r30, r1, r0
+; CHECK-LE-NEXT:  .LBB11_4:
 ; CHECK-LE-NEXT:    cmpd r1, r12
 ; CHECK-LE-NEXT:    bne cr0, .LBB11_3
-; CHECK-LE-NEXT:  .LBB11_4:
+; CHECK-LE-NEXT:  # %bb.5:
 ; CHECK-LE-NEXT:    mr r12, r30
 ; CHECK-LE-NEXT:    li r0, 24
 ; CHECK-LE-NEXT:    mtctr r0
-; CHECK-LE-NEXT:  .LBB11_5:
+; CHECK-LE-NEXT:  .LBB11_6:
 ; CHECK-LE-NEXT:    stdu r12, -4096(r1)
-; CHECK-LE-NEXT:    bdnz .LBB11_5
-; CHECK-LE-NEXT:  # %bb.6:
+; CHECK-LE-NEXT:    bdnz .LBB11_6
+; CHECK-LE-NEXT:  # %bb.7:
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-LE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-LE-NEXT:    .cfi_offset r31, -8
 ; CHECK-LE-NEXT:    .cfi_offset r30, -16
 ; CHECK-LE-NEXT:    clrldi r3, r3, 32
 ; CHECK-LE-NEXT:    lis r5, 1
 ; CHECK-LE-NEXT:    mr r31, r1
 ; CHECK-LE-NEXT:    li r6, 1

 ; CHECK-LE-NEXT:    mr r7, r4
 ; CHECK-LE-NEXT:    li r4, -4096
 ; CHECK-LE-NEXT:    divd r5, r7, r4
 ; CHECK-LE-NEXT:    mulld r4, r5, r4
 ; CHECK-LE-NEXT:    sub r5, r7, r4
 ; CHECK-LE-NEXT:    add r4, r1, r7
 ; CHECK-LE-NEXT:    stdux r3, r1, r5
 ; CHECK-LE-NEXT:    cmpd r1, r4
-; CHECK-LE-NEXT:    beq cr0, .LBB11_8
-; CHECK-LE-NEXT:  .LBB11_7:
+; CHECK-LE-NEXT:    beq cr0, .LBB11_9
+; CHECK-LE-NEXT:  .LBB11_8:
 ; CHECK-LE-NEXT:    stdu r3, -4096(r1)
 ; CHECK-LE-NEXT:    cmpd r1, r4
-; CHECK-LE-NEXT:    bne cr0, .LBB11_7
-; CHECK-LE-NEXT:  .LBB11_8:
+; CHECK-LE-NEXT:    bne cr0, .LBB11_8
+; CHECK-LE-NEXT:  .LBB11_9:
 ; CHECK-LE-NEXT:    addi r3, r1, -32768
 ; CHECK-LE-NEXT:    lbz r3, 0(r3)
 ; CHECK-LE-NEXT:    mr r1, r30
 ; CHECK-LE-NEXT:    ld r31, -8(r1)
 ; CHECK-LE-NEXT:    ld r30, -16(r1)
 ; CHECK-LE-NEXT:    blr
 ;
 ; CHECK-BE-LABEL: f11:
 ; CHECK-BE:       # %bb.0:
 ; CHECK-BE-NEXT:    std r31, -8(r1)
 ; CHECK-BE-NEXT:    std r30, -16(r1)
 ; CHECK-BE-NEXT:    mr r30, r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa r30, 0
 ; CHECK-BE-NEXT:    clrldi r0, r30, 49
-; CHECK-BE-NEXT:    subc r12, r30, r0
+; CHECK-BE-NEXT:    sub r12, r30, r0
 ; CHECK-BE-NEXT:    clrldi r0, r0, 52
-; CHECK-BE-NEXT:    cmpdi r0, 0
-; CHECK-BE-NEXT:    beq cr0, .LBB11_2
+; CHECK-BE-NEXT:    cmpdi r0, 288
+; CHECK-BE-NEXT:    blt cr0, .LBB11_2
 ; CHECK-BE-NEXT:  # %bb.1:
 ; CHECK-BE-NEXT:    neg r0, r0
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:    li r0, -4096
+; CHECK-BE-NEXT:    b .LBB11_4
 ; CHECK-BE-NEXT:  .LBB11_2:
+; CHECK-BE-NEXT:    sub r1, r1, r0
 ; CHECK-BE-NEXT:    li r0, -4096
-; CHECK-BE-NEXT:    cmpd r1, r12
-; CHECK-BE-NEXT:    beq cr0, .LBB11_4
+; CHECK-BE-NEXT:    b .LBB11_4
 ; CHECK-BE-NEXT:  .LBB11_3:
 ; CHECK-BE-NEXT:    stdux r30, r1, r0
+; CHECK-BE-NEXT:  .LBB11_4:
 ; CHECK-BE-NEXT:    cmpd r1, r12
 ; CHECK-BE-NEXT:    bne cr0, .LBB11_3
-; CHECK-BE-NEXT:  .LBB11_4:
+; CHECK-BE-NEXT:  # %bb.5:
 ; CHECK-BE-NEXT:    mr r12, r30
 ; CHECK-BE-NEXT:    li r0, 24
 ; CHECK-BE-NEXT:    mtctr r0
-; CHECK-BE-NEXT:  .LBB11_5:
+; CHECK-BE-NEXT:  .LBB11_6:
 ; CHECK-BE-NEXT:    stdu r12, -4096(r1)
-; CHECK-BE-NEXT:    bdnz .LBB11_5
-; CHECK-BE-NEXT:  # %bb.6:
+; CHECK-BE-NEXT:    bdnz .LBB11_6
+; CHECK-BE-NEXT:  # %bb.7:
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r1
 ; CHECK-BE-NEXT:    .cfi_def_cfa_register r30
 ; CHECK-BE-NEXT:    .cfi_offset r31, -8
 ; CHECK-BE-NEXT:    .cfi_offset r30, -16
 ; CHECK-BE-NEXT:    clrldi r3, r3, 32
 ; CHECK-BE-NEXT:    lis r5, 1
 ; CHECK-BE-NEXT:    addi r3, r3, 15
 ; CHECK-BE-NEXT:    mr r31, r1

 ; CHECK-BE-NEXT:    mr r7, r4
 ; CHECK-BE-NEXT:    li r4, -4096
 ; CHECK-BE-NEXT:    divd r5, r7, r4
 ; CHECK-BE-NEXT:    mulld r4, r5, r4
 ; CHECK-BE-NEXT:    sub r5, r7, r4
 ; CHECK-BE-NEXT:    add r4, r1, r7
 ; CHECK-BE-NEXT:    stdux r3, r1, r5
 ; CHECK-BE-NEXT:    cmpd r1, r4
-; CHECK-BE-NEXT:    beq cr0, .LBB11_8
-; CHECK-BE-NEXT:  .LBB11_7:
+; CHECK-BE-NEXT:    beq cr0, .LBB11_9
+; CHECK-BE-NEXT:  .LBB11_8:
 ; CHECK-BE-NEXT:    stdu r3, -4096(r1)
 ; CHECK-BE-NEXT:    cmpd r1, r4
-; CHECK-BE-NEXT:    bne cr0, .LBB11_7
-; CHECK-BE-NEXT:  .LBB11_8:
+; CHECK-BE-NEXT:    bne cr0, .LBB11_8
+; CHECK-BE-NEXT:  .LBB11_9:
 ; CHECK-BE-NEXT:    addi r3, r1, -32768
 ; CHECK-BE-NEXT:    lbz r3, 0(r3)
 ; CHECK-BE-NEXT:    mr r1, r30
 ; CHECK-BE-NEXT:    ld r31, -8(r1)
 ; CHECK-BE-NEXT:    ld r30, -16(r1)
 ; CHECK-BE-NEXT:    blr
 ;
 ; CHECK-32-LABEL: f11: