[LoopPredication] Allow predication of loop invariant computations

[reames]

Summary

The purpose of this patch is to eliminate a pass ordering dependence between LoopPredication and LICM. To understand the purpose, consider the following snippet of code inside some loop 'L' with IV 'i'
A = _a.length;
guard (i < A)
a = _a[i]
B = _b.length;
guard (i < B);
b = _b[i];
...
Z = _z.length;
guard (i < Z)
z = _z[i]
accum += a + b + ... + z;

Today, we need LICM to hoist the length loads, LoopPredication to make the guards loop invariant, and TrivialUnswitch to eliminate the loop invariant guard to establish must execute for the next length load. Today, if we can't prove speculation safety, we'd have to iterate these three passes 26 times to reduce this example down to the minimal form.

Using the fact that the array lengths are known to be invariant, we can short circuit this iteration. By forming the loop invariant form of all the guards at once, we remove the need for LoopPredication from the iterative cycle. At the moment, we'd still have to iterate LICM and TrivialUnswitch; we'll leave that part for later.

There are two high level design questions on this patch I'd like input on:

1. I realized I'm essentially duplicating most of the loop disposition logic from SCEV, but expressed over IR. The critical missing piece is that SCEV does not consider a SCEVUknown for an invariant load w/invariant operands to be invariant. Would it make sense to implement this in SCEV?
2. Anyone see a better way to handle the adjustment of the insertion point? I want to be inserting in the preheader if possible, and only falling back to in the loop if required. I thought about inserting in the loop and then hoisting, but we may have expressions we can't prove the safety of in IR which SCEV has already established. Wasting that opportunity is undesirable, but mucking around with the insertion point is also quite ugly. Ideas?

[apilipenko]

1. Since we need to fiddle with the insertion point I think we should stop using single IRBuilder which we pass around. It looks like creating a local IRBuilder is a cheap operation, so we can have local IRBuilders when we need to generate instructions. This way the insertion point will be clear from the local context.

2. Invariant expression analysis over IR is a bit non-trivial. It looks like extending SCEV would be a simpler option. Alternatively, we can build up the complexity of the IR analysis incrementally as I suggest in the comment below.

lib/Transforms/Scalar/LoopPredication.cpp

401–409

Nit. Extract SE->isLoopInvariant(LHS, L) && SE->isLoopInvariant(RHS, L) into a local variable? You check it below as well.

413

You can use Preheader cached in the field.

473

Debug printing.

966–977

The complexity to handle cases when the expression is a non-side effecting operation with invariant inputs deserves a test.

Alternatively, you can introduce this complexity as a follow up. In the initial patch you can handle only loads, which should cover most of the cases.

[reames]

In D60093#1453993, @apilipenko wrote:

2. Invariant expression analysis over IR is a bit non-trivial. It looks like extending SCEV would be a simpler option. Alternatively, we can build up the complexity of the IR analysis incrementally as I suggest in the comment below.

I started prototyping the SCEV change, and found one downside. Because SCEV doesn't have access to AA, I can't use the pointsToConstantMemory as I do with the current scheme. I think this is a fairly minor impact since loads from constant memory which can't be proven trivial dereferenceable should be rare, but I'd like a second opinion on that.

(Also, marking as plan changes so that this shows up in my queue of things to do)

[hfinkel]

Because SCEV doesn't have access to AA,

We should change that if useful.

[reames]

Remove the invariant expression complexity, and the SCEV loop disposition case. I'm still hoping to sink that into SCEV, but handling the simple case is straight forward and I'd like to get at least that in. The SCEV patch turns out to be a bit more complex than I'd originally expected, so I need to take a step back and revaluate that approach.

Still working on the builder usage. Likely to update again, but I wanted to keep the snapshot of the progress.

[reames]

In D60093#1453993, @apilipenko wrote:

1. Since we need to fiddle with the insertion point I think we should stop using single IRBuilder which we pass around. It looks like creating a local IRBuilder is a cheap operation, so we can have local IRBuilders when we need to generate instructions. This way the insertion point will be clear from the local context.

This is handled in a separate review ( D60718 ). Once that's landed, I'll rebase.

[reames]

Update after rebasing over previous changes. This is a fairly major rework though, so please review from scratch.

Key thing is that I stumbled across a bug in the previous implementation. I've committed the test to highlight it as neg_invariant_latch_limit. The problem is that SCEV turns out not to be correctly handling dominance within a single block within isSafeToSpeculateAt. The fix for that is included, along with just enough special casing to keep all of the existing tests passing.

Secondly, I realized that we were mixing two different concepts and that's what made the bug possible. Given that, I restructured the change to check invariance separately from insertion safety. This relies on the previous patch which made it clear that insertion at the guard is always valid, and that hoisting out of the loop is purely an optimization. Interestingly, splitting in this manner actually makes the transform *more* powerful, not less. See the (correct) diffs around udiv handling.

[apilipenko]

Looks good overall, a couple of comments inline.

lib/Transforms/Scalar/LoopPredication.cpp

452

Am I right that this is not specific to the change you make and just a fix for an existing bug? If so, can be integrated separately with a test demonstrating the problem.

498–500

Can we assert isSafeToExpandAt for the values we don't check?

498–517

These changes look correct to me, but there are actually a couple of NFC refactorings (replacing CanExpand(RHS) with CanExpand(LatchStart), inlining CanExpand, removing isSafeToExpandAt checks for GuardStart, GuardLimit) mixed with a functional change to use isLoopInvariantValue instead of SE->isLoopInvariant. You might want to split these when integrating to make bisecting easier.

965

Debug printing.

[reames]

lib/Transforms/Scalar/LoopPredication.cpp

452

You are correct, and I'd completely missed that when writing. Will do!

498–500

No we can't. That's the entire point of not checking.

The basic structure here is that isSafeToExpand guarantees expansion is safe, but a false return value can still be safe to expand. In this case, we use information about dominance of the original conditions in the IR to ensure safety of these two.

498–517

I thought about it, but explaining the NFC part without context seemed challenging. So I left them together.

965

Oops, good catch, thanks.

[reames]

lib/Transforms/Scalar/LoopPredication.cpp

452

Actually, no. On further reflection, this is a silent bug without the rest of this change. All of the operands to the expressions we call this on have previously been checked via isSafeToSpeculate. It's only with this change that we weaken that precondition and expose the latent issue. Given that, I'm not going to submit this separately.

[reames]

Immediately after submitting this, I noticed an important piece had gotten lost somewhere in the rebasing. rL358688 fixes an obvious bug with the invariant_load casing.

lib/Transforms/Scalar/LoopPredication.cpp

 //   For uge condition the widened condition is:
 //     guardStart u< guardLimit && latchLimit u> 1.
 //   For sge condition the widened condition is:
 //     guardStart u< guardLimit && latchLimit s> 1.
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/Scalar/LoopPredication.h"
 #include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/GuardUtils.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/LoopPass.h"
 #include "llvm/Analysis/ScalarEvolution.h"
 #include "llvm/Analysis/ScalarEvolutionExpander.h"
 #include "llvm/Analysis/ScalarEvolutionExpressions.h"
 #include "llvm/IR/Function.h"
LoopICmp(ICmpInst::Predicate Pred, const SCEVAddRecExpr *IV,
         : Pred(Pred), IV(IV), Limit(Limit) {}
     LoopICmp() {}
     void dump() {
       dbgs() << "LoopICmp Pred = " << Pred << ", IV = " << *IV
              << ", Limit = " << *Limit << "\n";
     }
   };
 
+  AliasAnalysis *AA;
   ScalarEvolution *SE;
   BranchProbabilityInfo *BPI;
 
   Loop *L;
   const DataLayout *DL;
   BasicBlock *Preheader;
   LoopICmp LatchCheck;
 
   bool isSupportedStep(const SCEV* Step);
   Optional<LoopICmp> parseLoopICmp(ICmpInst *ICI) {
     return parseLoopICmp(ICI->getPredicate(), ICI->getOperand(0),
                          ICI->getOperand(1));
   }
   Optional<LoopICmp> parseLoopICmp(ICmpInst::Predicate Pred, Value *LHS,
                                    Value *RHS);
 
   Optional<LoopICmp> parseLoopLatchICmp();
 
-  bool CanExpand(const SCEV* S);
+  bool CanExpand(const SCEV* S, const Instruction *At);
   Value *expandCheck(SCEVExpander &Expander, IRBuilder<> &Builder,
                      ICmpInst::Predicate Pred, const SCEV *LHS,
                      const SCEV *RHS);
 
   Optional<Value *> widenICmpRangeCheck(ICmpInst *ICI, SCEVExpander &Expander,
                                         IRBuilder<> &Builder);
   Optional<Value *> widenICmpRangeCheckIncrementingLoop(LoopICmp LatchCheck,
                                                         LoopICmp RangeCheck,
class LoopPredication {
   // This function returns true if we can safely represent the IV type in
   // the RangeCheckType without loss of information.
   bool isSafeToTruncateWideIVType(Type *RangeCheckType);
   // Return the loopLatchCheck corresponding to the RangeCheckType if safe to do
   // so.
   Optional<LoopICmp> generateLoopLatchCheck(Type *RangeCheckType);
 
 public:
-  LoopPredication(ScalarEvolution *SE, BranchProbabilityInfo *BPI)
-      : SE(SE), BPI(BPI){};
+  LoopPredication(AliasAnalysis *AA, ScalarEvolution *SE,
+                  BranchProbabilityInfo *BPI)
+    : AA(AA), SE(SE), BPI(BPI){};
   bool runOnLoop(Loop *L);
 };
 
 class LoopPredicationLegacyPass : public LoopPass {
 public:
   static char ID;
   LoopPredicationLegacyPass() : LoopPass(ID) {
     initializeLoopPredicationLegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<BranchProbabilityInfoWrapperPass>();
     getLoopAnalysisUsage(AU);
   }
 
   bool runOnLoop(Loop *L, LPPassManager &LPM) override {
     if (skipLoop(L))
       return false;
     auto *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
     BranchProbabilityInfo &BPI =
-        getAnalysis<BranchProbabilityInfoWrapperPass>().getBPI();
-    LoopPredication LP(SE, &BPI);
+    auto *AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+    LoopPredication LP(AA, SE, &BPI);
     return LP.runOnLoop(L);
   }
 };
 
 char LoopPredicationLegacyPass::ID = 0;
 } // end namespace llvm
 
 INITIALIZE_PASS_BEGIN(LoopPredicationLegacyPass, "loop-predication",

 
 PreservedAnalyses LoopPredicationPass::run(Loop &L, LoopAnalysisManager &AM,
                                            LoopStandardAnalysisResults &AR,
                                            LPMUpdater &U) {
   const auto &FAM =
       AM.getResult<FunctionAnalysisManagerLoopProxy>(L, AR).getManager();
   Function *F = L.getHeader()->getParent();
   auto *BPI = FAM.getCachedResult<BranchProbabilityAnalysis>(*F);
-  LoopPredication LP(&AR.SE, BPI);
+  LoopPredication LP(&AR.AA, &AR.SE, BPI);
   if (!LP.runOnLoop(&L))
     return PreservedAnalyses::all();
 
   return getLoopPassPreservedAnalyses();
 }
 
 Optional<LoopPredication::LoopICmp>
 LoopPredication::parseLoopICmp(ICmpInst::Predicate Pred, Value *LHS,

 
 Value *LoopPredication::expandCheck(SCEVExpander &Expander,
                                     IRBuilder<> &Builder,
                                     ICmpInst::Predicate Pred, const SCEV *LHS,
                                     const SCEV *RHS) {
   Type *Ty = LHS->getType();
   assert(Ty == RHS->getType() && "expandCheck operands have different types?");
 
+  const bool OpsAreLoopInvariant =
+    SE->isLoopInvariant(LHS, L) && SE->isLoopInvariant(RHS, L);
+  if (OpsAreLoopInvariant) {
     if (SE->isLoopEntryGuardedByCond(L, Pred, LHS, RHS))
       return Builder.getTrue();
     if (SE->isLoopEntryGuardedByCond(L, ICmpInst::getInversePredicate(Pred),
                                      LHS, RHS))
       return Builder.getFalse();
+  }
 
-  Instruction *InsertAt = &*Builder.GetInsertPoint();
+  Instruction * const InsertAt = OpsAreLoopInvariant ?
+    Preheader->getTerminator() : &*Builder.GetInsertPoint();
   Value *LHSV = Expander.expandCodeFor(LHS, Ty, InsertAt);
   Value *RHSV = Expander.expandCodeFor(RHS, Ty, InsertAt);
+  IRBuilder<>::InsertPointGuard G(Builder);
+  Builder.SetInsertPoint(InsertAt);
   return Builder.CreateICmp(Pred, LHSV, RHSV);
 }
 
 Optional<LoopPredication::LoopICmp>
 LoopPredication::generateLoopLatchCheck(Type *RangeCheckType) {
 
   auto *LatchType = LatchCheck.IV->getType();
   if (RangeCheckType == LatchType)
LoopPredication::generateLoopLatchCheck(Type *RangeCheckType) {
   LLVM_DEBUG(dbgs() << "LatchCheck.Limit: " << *NewLatchCheck.Limit << "\n");
   return NewLatchCheck;
 }
 
 bool LoopPredication::isSupportedStep(const SCEV* Step) {
   return Step->isOne() || (Step->isAllOnesValue() && EnableCountDownLoop);
 }
 
-bool LoopPredication::CanExpand(const SCEV* S) {
-  return SE->isLoopInvariant(S, L) && isSafeToExpand(S, *SE);
+bool LoopPredication::CanExpand(const SCEV* S, const Instruction *At) {
+  if (SE->isLoopInvariant(S, L) && isSafeToExpand(S, *SE))
+    return true;
+
+  // Handling expressions which produce invariant results, but *haven't* yet
+  // been removed from the loop serves two important purposes.
+  // 1) Most importantly, it resolves a pass ordering cycle which would
+  // otherwise need us to iteration licm, loop-predication, and either
+  // loop-unswitch or loop-peeling to make progress on examples with lots of
+  // predicable range checks in a row.  (Since, in the general case,  we can't
+  // hoist the length checks until the dominating checks have been discharged
+  // as we can't prove doing so is safe.)
+  // 2) As a nice side effect, this exposes the value of peeling or unswitching
+  // much more obviously in the IR.  Otherwise, the cost modeling for other
+  // transforms would end up needing to duplicate all of this logic to model a
+  // check which becomes predictable based on a modeled peel or unswitch.
+  //
+  // The cost of doing so in the worst case is an extra load in the loop to
+  // materialize the loop invariant test value instead of checking against the
+  // original IV which is presumable in a register inside the loop.  Such cases
+  // are presumably rare, and hint at missing oppurtunities for other passes.
+  if (const SCEVUnknown *U = dyn_cast<SCEVUnknown>(S))
+    if (const auto *LI = dyn_cast<LoadInst>(U->getValue())) {
+      if (!LI->isUnordered())
+        return false;
+      if (AA->pointsToConstantMemory(LI->getOperand(0)) ||
+          LI->getMetadata(LLVMContext::MD_invariant_load) != nullptr)
+        return isSafeToExpandAt(S, At, *SE);
+    }
+  return false;
 }
 
 Optional<Value *> LoopPredication::widenICmpRangeCheckIncrementingLoop(
     LoopPredication::LoopICmp LatchCheck, LoopPredication::LoopICmp RangeCheck,
     SCEVExpander &Expander, IRBuilder<> &Builder) {
   auto *Ty = RangeCheck.IV->getType();
   // Generate the widened condition for the forward loop:
   //   guardStart u< guardLimit &&
   //   latchLimit <pred> guardLimit - 1 - guardStart + latchStart
   // where <pred> depends on the latch condition predicate. See the file
   // header comment for the reasoning.
   // guardLimit - guardStart + latchStart - 1
   const SCEV *GuardStart = RangeCheck.IV->getStart();
   const SCEV *GuardLimit = RangeCheck.Limit;
   const SCEV *LatchStart = LatchCheck.IV->getStart();
   const SCEV *LatchLimit = LatchCheck.Limit;
 
   // guardLimit - guardStart + latchStart - 1
   const SCEV *RHS =
       SE->getAddExpr(SE->getMinusSCEV(GuardLimit, GuardStart),
                      SE->getMinusSCEV(LatchStart, SE->getOne(Ty)));
-  if (!CanExpand(GuardStart) || !CanExpand(GuardLimit) ||
-      !CanExpand(LatchLimit) || !CanExpand(RHS)) {
+  Instruction *InsertAt = &*Builder.GetInsertPoint();
+  Instruction *PHTerm = Preheader->getTerminator();
+  if (!CanExpand(GuardStart, PHTerm) || !CanExpand(GuardLimit, InsertAt) ||
+      !CanExpand(LatchLimit, PHTerm) || !CanExpand(RHS, InsertAt)) {
     LLVM_DEBUG(dbgs() << "Can't expand limit check!\n");
     return None;
   }
   auto LimitCheckPred =
       ICmpInst::getFlippedStrictnessPredicate(LatchCheck.Pred);
 
   LLVM_DEBUG(dbgs() << "LHS: " << *LatchLimit << "\n");
   LLVM_DEBUG(dbgs() << "RHS: " << *RHS << "\n");
   LLVM_DEBUG(dbgs() << "Pred: " << LimitCheckPred << "\n");
  
   auto *LimitCheck =
       expandCheck(Expander, Builder, LimitCheckPred, LatchLimit, RHS);
   auto *FirstIterationCheck = expandCheck(Expander, Builder, RangeCheck.Pred,
                                           GuardStart, GuardLimit);
+  if (L->isLoopInvariant(LimitCheck) && L->isLoopInvariant(FirstIterationCheck))
+    InsertAt = Preheader->getTerminator();
+  IRBuilder<>::InsertPointGuard G(Builder);
+  Builder.SetInsertPoint(InsertAt);
   return Builder.CreateAnd(FirstIterationCheck, LimitCheck);
 }
 
 Optional<Value *> LoopPredication::widenICmpRangeCheckDecrementingLoop(
     LoopPredication::LoopICmp LatchCheck, LoopPredication::LoopICmp RangeCheck,
     SCEVExpander &Expander, IRBuilder<> &Builder) {
   auto *Ty = RangeCheck.IV->getType();
   const SCEV *GuardStart = RangeCheck.IV->getStart();
   const SCEV *GuardLimit = RangeCheck.Limit;
   const SCEV *LatchLimit = LatchCheck.Limit;
-  if (!CanExpand(GuardStart) || !CanExpand(GuardLimit) ||
-      !CanExpand(LatchLimit)) {
+  Instruction *InsertAt = &*Builder.GetInsertPoint();
+  if (!CanExpand(GuardStart, InsertAt) || !CanExpand(GuardLimit, InsertAt) ||
+      !CanExpand(LatchLimit, InsertAt)) {
     LLVM_DEBUG(dbgs() << "Can't expand limit check!\n");
     return None;
   }
   // The decrement of the latch check IV should be the same as the
   // rangeCheckIV.
   auto *PostDecLatchCheckIV = LatchCheck.IV->getPostIncExpr(*SE);
   if (RangeCheck.IV != PostDecLatchCheckIV) {
     LLVM_DEBUG(dbgs() << "Not the same. PostDecLatchCheckIV: "
                       << *PostDecLatchCheckIV
                       << "  and RangeCheckIV: " << *RangeCheck.IV << "\n");
     return None;
   }
 
   // Generate the widened condition for CountDownLoop:
   // guardStart u< guardLimit &&
   // latchLimit <pred> 1.
   // See the header comment for reasoning of the checks.
   auto LimitCheckPred =
       ICmpInst::getFlippedStrictnessPredicate(LatchCheck.Pred);
   auto *FirstIterationCheck = expandCheck(Expander, Builder, ICmpInst::ICMP_ULT,
                                           GuardStart, GuardLimit);
   auto *LimitCheck = expandCheck(Expander, Builder, LimitCheckPred, LatchLimit,
                                  SE->getOne(Ty));
+  if (L->isLoopInvariant(LimitCheck) && L->isLoopInvariant(FirstIterationCheck))
+    InsertAt = Preheader->getTerminator();
+  IRBuilder<>::InsertPointGuard G(Builder);
+  Builder.SetInsertPoint(InsertAt);
   return Builder.CreateAnd(FirstIterationCheck, LimitCheck);
 }
 
 /// If ICI can be widened to a loop invariant condition emits the loop
 /// invariant condition in the loop preheader and return it, otherwise
 /// returns None.
 Optional<Value *> LoopPredication::widenICmpRangeCheck(ICmpInst *ICI,
                                                        SCEVExpander &Expander,
return widenICmpRangeCheckDecrementingLoop(CurrLatchCheck, *RangeCheck,
                                                Expander, Builder);
   }
 }
 
 unsigned LoopPredication::collectChecks(SmallVectorImpl<Value *> &Checks,
                                         Value *Condition,
                                         SCEVExpander &Expander,
                                         IRBuilder<> &Builder) {
+  IRBuilder<>::InsertPointGuard G(Builder);
   unsigned NumWidened = 0;
   // The guard condition is expected to be in form of:
   //   cond1 && cond2 && cond3 ...
   // Iterate over subconditions looking for icmp conditions which can be
   // widened across loop iterations. Widening these conditions remember the
   // resulting list of subconditions in Checks vector.
   SmallVector<Value *, 4> Worklist(1, Condition);
   SmallPtrSet<Value *, 4> Visited;
do {
     if (match(Condition,
               m_Intrinsic<Intrinsic::experimental_widenable_condition>())) {
       // Pick any, we don't care which
       WideableCond = Condition;
       continue;
     }
 
     if (ICmpInst *ICI = dyn_cast<ICmpInst>(Condition)) {
+      IRBuilder<>::InsertPointGuard G(Builder);
+      Builder.SetInsertPoint(ICI);
       if (auto NewRangeCheck = widenICmpRangeCheck(ICI, Expander,
                                                    Builder)) {
         Checks.push_back(NewRangeCheck.getValue());
         NumWidened++;
         continue;
       }
     }
 
bool LoopPredication::runOnLoop(Loop *Loop) {
   SCEVExpander Expander(*SE, *DL, "loop-predication");
 
   bool Changed = false;
   for (auto *Guard : Guards)
     Changed |= widenGuardConditions(Guard, Expander);
   for (auto *Guard : GuardsAsWidenableBranches)
     Changed |= widenWidenableBranchGuardConditions(Guard, Expander);
 
   return Changed;
 }

test/Transforms/LoopPredication/invariant_load.ll

 ; CHECK:       loop.preheader:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[LOOP_ACC:%.*]] = phi i32 [ [[LOOP_ACC_NEXT:%.*]], [[LOOP]] ], [ 0, [[LOOP_PREHEADER]] ]
 ; CHECK-NEXT:    [[I:%.*]] = phi i32 [ [[I_NEXT:%.*]], [[LOOP]] ], [ 0, [[LOOP_PREHEADER]] ]
 ; CHECK-NEXT:    [[UNKNOWN:%.*]] = load volatile i1, i1* @UNKNOWN
 ; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[UNKNOWN]]) [ "deopt"() ]
 ; CHECK-NEXT:    [[LEN:%.*]] = load i32, i32* [[LENGTH:%.*]], align 4, !invariant.load !0
-; CHECK-NEXT:    [[WITHIN_BOUNDS:%.*]] = icmp ult i32 [[I]], [[LEN]]
-; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[WITHIN_BOUNDS]], i32 9) [ "deopt"() ]
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp ule i32 [[N]], [[LEN]]
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ult i32 0, [[LEN]]
+; CHECK-NEXT:    [[TMP2:%.*]] = and i1 [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[TMP2]], i32 9) [ "deopt"() ]
 ; CHECK-NEXT:    [[I_I64:%.*]] = zext i32 [[I]] to i64
 ; CHECK-NEXT:    [[ARRAY_I_PTR:%.*]] = getelementptr inbounds i32, i32* [[ARRAY:%.*]], i64 [[I_I64]]
 ; CHECK-NEXT:    [[ARRAY_I:%.*]] = load i32, i32* [[ARRAY_I_PTR]], align 4
 ; CHECK-NEXT:    [[LOOP_ACC_NEXT]] = add i32 [[LOOP_ACC]], [[ARRAY_I]]
 ; CHECK-NEXT:    [[I_NEXT]] = add nuw i32 [[I]], 1
 ; CHECK-NEXT:    [[CONTINUE:%.*]] = icmp ult i32 [[I_NEXT]], [[N]]
 ; CHECK-NEXT:    br i1 [[CONTINUE]], label [[LOOP]], label [[EXIT_LOOPEXIT:%.*]]
 ; CHECK:       exit.loopexit:

 ; CHECK:       loop.preheader:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[LOOP_ACC:%.*]] = phi i32 [ [[LOOP_ACC_NEXT:%.*]], [[LOOP]] ], [ 0, [[LOOP_PREHEADER]] ]
 ; CHECK-NEXT:    [[I:%.*]] = phi i32 [ [[I_NEXT:%.*]], [[LOOP]] ], [ 0, [[LOOP_PREHEADER]] ]
 ; CHECK-NEXT:    [[UNKNOWN:%.*]] = load volatile i1, i1* @UNKNOWN
 ; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[UNKNOWN]]) [ "deopt"() ]
 ; CHECK-NEXT:    [[LEN:%.*]] = load i32, i32* @Length, align 4
-; CHECK-NEXT:    [[WITHIN_BOUNDS:%.*]] = icmp ult i32 [[I]], [[LEN]]
-; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[WITHIN_BOUNDS]], i32 9) [ "deopt"() ]
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp ule i32 [[N]], [[LEN]]
+; CHECK-NEXT:    [[TMP1:%.*]] = icmp ult i32 0, [[LEN]]
+; CHECK-NEXT:    [[TMP2:%.*]] = and i1 [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    call void (i1, ...) @llvm.experimental.guard(i1 [[TMP2]], i32 9) [ "deopt"() ]
 ; CHECK-NEXT:    [[I_I64:%.*]] = zext i32 [[I]] to i64
 ; CHECK-NEXT:    [[ARRAY_I_PTR:%.*]] = getelementptr inbounds i32, i32* [[ARRAY:%.*]], i64 [[I_I64]]
 ; CHECK-NEXT:    [[ARRAY_I:%.*]] = load i32, i32* [[ARRAY_I_PTR]], align 4
 ; CHECK-NEXT:    [[LOOP_ACC_NEXT]] = add i32 [[LOOP_ACC]], [[ARRAY_I]]
 ; CHECK-NEXT:    [[I_NEXT]] = add nuw i32 [[I]], 1
 ; CHECK-NEXT:    [[CONTINUE:%.*]] = icmp ult i32 [[I_NEXT]], [[N]]
 ; CHECK-NEXT:    br i1 [[CONTINUE]], label [[LOOP]], label [[EXIT_LOOPEXIT:%.*]]
 ; CHECK:       exit.loopexit: