[LAA] Support runtime checks for select GEP base pointers.

[fhahn]

Summary

Scaffolding support for generating runtime checks for multiple SCEV expressions
per pointer. The initial version just adds support for looking through
a single select.

The more sophisticated logic for analyzing forks is in D108699

[fhahn]

rebased on top of precommitted tests, simplified code

[fhahn]

Rebase and stripped down logic for translating pointers to bare minimum.

[fhahn]

ping :)

[huntergr]

LGTM

[vitalybuka]

This patch is causing false reports with msan. I am trying to creduce the code. It's with TensorFlow, so far it's quite complicated.

I see that before the patch the pass produced

vector.memcheck:                                  ; preds = %vector.scevcheck
  %scevgep77 = getelementptr i32, i32* %retval.0.i.sroa.sel, i64 1, !dbg !1439
  %scevgep7778 = bitcast i32* %scevgep77 to i8*, !dbg !1439
  %bound0 = icmp ult i8* %scevgep73, %scevgep7778, !dbg !1439
  %bound1 = icmp ult i8* %retval.0.i.sroa.sel76, %scevgep7475, !dbg !1439
  %found.conflict = and i1 %bound0, %bound1, !dbg !1439
  br i1 %found.conflict, label %scalar.ph, label %vector.ph

After the patch it's more complicated:

vector.memcheck:                                  ; preds = %vector.scevcheck
  %scevgep79 = getelementptr %"union.absl::container_internal::map_slot_type", %"union.absl::container_internal::map_slot_type"* %33, i64 0, i32 0, i32 1, i32 1, i32 0, i32 2, !dbg !1439
  %scevgep7980 = bitcast i32* %scevgep79 to i8*, !dbg !1439
  %scevgep81 = getelementptr %"union.absl::container_internal::map_slot_type", %"union.absl::container_internal::map_slot_type"* %33, i64 0, i32 0, i32 1, i32 2, i64 0, !dbg !1439
  %bound0 = icmp ult i8* %scevgep73, %scevgep78, !dbg !1439
  %bound1 = icmp ult i8* %scevgep7677, %scevgep7475, !dbg !1439
  %found.conflict = and i1 %bound0, %bound1, !dbg !1439
  %bound082 = icmp ult i8* %scevgep73, %scevgep81, !dbg !1439
  %bound183 = icmp ult i8* %scevgep7980, %scevgep7475, !dbg !1439
  %found.conflict84 = and i1 %bound082, %bound183, !dbg !1439
  %conflict.rdx = or i1 %found.conflict, %found.conflict84, !dbg !1439
  br i1 %conflict.rdx, label %scalar.ph, label %vector.ph

Then then when Msan instruments the code, it detects branch on uninitialized %conflict.rdx

I will continue to minimize the reproducer.
Should we revert the patch, because it breaks Msan and maybe there is a chance the bounds are calculated incorrectly?

[fhahn]

In D114487#3546674, @vitalybuka wrote:

I will continue to minimize the reproducer.
Should we revert the patch, because it breaks Msan and maybe there is a chance the bounds are calculated incorrectly?

Thanks for the report, I'll take a look. I already have suspicion of what's going wrong. Hopefully I'll be able to write up a test case & fix today. If not, we can revert tomorrow.

[vitalybuka]

In D114487#3547560, @fhahn wrote:

In D114487#3546674, @vitalybuka wrote:

I will continue to minimize the reproducer.
Should we revert the patch, because it breaks Msan and maybe there is a chance the bounds are calculated incorrectly?

Thanks for the report, I'll take a look. I already have suspicion of what's going wrong.

My current reproducer is

#include "third_party/absl/container/flat_hash_map.h"
#include "third_party/tensorflow/core/framework/tensor.h"

int main() {
  absl::flat_hash_map<std::string, int> ids_map;
  std::vector<std::string> ids = {"1405217"};
  int d = 0;
  tensorflow::Tensor tensor(tensorflow::DataTypeToEnum<float>::v(),
                            {1, 20, 50, 1});
  tensor.flat<float>().setConstant(0);

  auto tm = tensor.tensor<float, 4>();
  int i = 0;
  for (const auto& other_agent_id : ids) {
    auto it = ids_map.find(other_agent_id);
    const int& f = it != ids_map.end() ? it->second : d;
    for (int j = 0; j < 50; ++j) {
      tm(0, i, j, 0) = f;
    }
    ++i;
  }
}

Not sure if it's helpful, as it requires TF with dependencies build with msan:
-O2 -fsanitize=memory -fsanitize-memory-param-retval -fsanitize-memory-use-after-dtor

Hopefully I'll be able to write up a test case & fix today. If not, we can revert tomorrow.

Thanks, revert will help, as out internal release is blocked on this patch.

[alexfh]

I've reverted the commit in 7aa8a678826dea86ff3e6c7df9d2a8a6ef868f5d to unblock our internal release.

[vitalybuka]

@fhahn
This is reduced reproducer:

struct c {
  int b;
};
struct e {
  int &operator*() { return ae->b; }
  bool operator!=(e) __attribute__((noinline)) { return 0; }
  c *ae;
};
int a;
float g;
int main() {
  e b, d;
  int &f = d != b ? *d : a;
  for (int h = 0; h < 50; ++h)
    g = f;
}

clang++ -cc1 -triple x86_64-grtev4-linux-gnu -emit-obj -mframe-pointer=non-leaf -relaxed-aliasing -mconstructor-aliases -O2 -fsanitize=memory -Werror -Wreturn-type -fsanitize-memory-use-after-dtor -fsanitize-memory-param-retval -vectorize-loops -o t.o -x c++ t.cc
clang++ t.o -fsanitize=memory -fuse-ld=lld -o t -g0 -s -Wl,--strip-all

./t

==350572==WARNING: MemorySanitizer: use-of-uninitialized-value
SUMMARY: MemorySanitizer: use-of-uninitialized-value

[fhahn]

In D114487#3555041, @vitalybuka wrote:

@fhahn
This is reduced reproducer:

Thanks! I think I was able to isolate the bits that caused problems:

1. Unecessarily looking through selects outside the loop
2. Introducing branches on undef/poison when looking through selects.

I recommitted the change in e9cced27390ba38eac1144aa1240281a1edadec0. I couldn't test with msan though, as it's not available on macOS. Please let me know if you see further issues.

[vitalybuka]

In D114487#3593011, @fhahn wrote:

In D114487#3555041, @vitalybuka wrote:

@fhahn
This is reduced reproducer:

Thanks! I think I was able to isolate the bits that caused problems:

1. Unecessarily looking through selects outside the loop
2. Introducing branches on undef/poison when looking through selects.

I recommitted the change in e9cced27390ba38eac1144aa1240281a1edadec0. I couldn't test with msan though, as it's not available on macOS. Please let me know if you see further issues.

Thanks! Out test works with e9cced27390ba38eac1144aa1240281a1edadec0

llvm/include/llvm/Analysis/LoopAccessAnalysis.h

void reset() {
     Checks.clear();
   }
 
   /// Insert a pointer and calculate the start and end SCEVs.
   /// We need \p PSE in order to compute the SCEV expression of the pointer
   /// according to the assumptions that we've made during the analysis.
   /// The method might also version the pointer stride according to \p Strides,
   /// and add new predicates to \p PSE.
-  void insert(Loop *Lp, Value *Ptr, bool WritePtr, unsigned DepSetId,
-              unsigned ASId, const ValueToValueMap &Strides,
-              PredicatedScalarEvolution &PSE);
+  void insert(Loop *Lp, Value *Ptr, const SCEV *PtrExpr, bool WritePtr,
+              unsigned DepSetId, unsigned ASId, PredicatedScalarEvolution &PSE);
 
   /// No run-time memory checking is necessary.
   bool empty() const { return Pointers.empty(); }
 
   /// Generate the checks and store it.  This also performs the grouping
   /// of pointers to reduce the number of memchecks necessary.
   void generateChecks(MemoryDepChecker::DepCandidates &DepCands,
                       bool UseDependencies);

llvm/lib/Analysis/LoopAccessAnalysis.cpp

 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/IR/PassManager.h"
+#include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Value.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <cstdlib>
 #include <iterator>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
+using namespace llvm::PatternMatch;
 
 #define DEBUG_TYPE "loop-accesses"
 
 static cl::opt<unsigned, true>
 VectorizationFactor("force-vector-width", cl::Hidden,
                     cl::desc("Sets the SIMD width. Zero is autoselect."),
                     cl::location(VectorizerParams::VectorizationFactor));
 unsigned VectorizerParams::VectorizationFactor;

 /// N is a calculated back-edge taken count:
 ///     N = (TripCount > 0) ? RoundDown(TripCount -1 , VF) : 0
 /// Start and End points are calculated in the following way:
 /// Start = UMIN(A, B) ; End = UMAX(A, B) + SizeOfElt,
 /// where SizeOfElt is the size of single memory access in bytes.
 ///
 /// There is no conflict when the intervals are disjoint:
 /// NoConflict = (P2.Start >= P1.End) || (P1.Start >= P2.End)
-void RuntimePointerChecking::insert(Loop *Lp, Value *Ptr, bool WritePtr,
-                                    unsigned DepSetId, unsigned ASId,
-                                    const ValueToValueMap &Strides,
+void RuntimePointerChecking::insert(Loop *Lp, Value *Ptr, const SCEV *PtrExpr,
+                                    bool WritePtr, unsigned DepSetId,
+                                    unsigned ASId,
                                     PredicatedScalarEvolution &PSE) {
-  // Get the stride replaced scev.
-  const SCEV *Sc = replaceSymbolicStrideSCEV(PSE, Strides, Ptr);
+  const SCEV *Sc = PtrExpr;
   ScalarEvolution *SE = PSE.getSE();
 
   const SCEV *ScStart;
   const SCEV *ScEnd;
 
   if (SE->isLoopInvariant(Sc, Lp)) {
     ScStart = ScEnd = Sc;
   } else {
void RuntimePointerChecking::groupChecks(
   if (!UseDependencies) {
     for (unsigned I = 0; I < Pointers.size(); ++I)
       CheckingGroups.push_back(RuntimeCheckingPtrGroup(I, *this));
     return;
   }
 
   unsigned TotalComparisons = 0;
 
-  DenseMap<Value *, unsigned> PositionMap;
-  for (unsigned Index = 0; Index < Pointers.size(); ++Index)
-    PositionMap[Pointers[Index].PointerValue] = Index;
+  DenseMap<Value *, SmallVector<unsigned>> PositionMap;
+  for (unsigned Index = 0; Index < Pointers.size(); ++Index) {
+    auto Iter = PositionMap.insert({Pointers[Index].PointerValue, {}});
+    Iter.first->second.push_back(Index);
+  }
 
   // We need to keep track of what pointers we've already seen so we
   // don't process them twice.
   SmallSet<unsigned, 2> Seen;
 
   // Go through all equivalence classes, get the "pointer check groups"
   // and add them to the overall solution. We use the order in which accesses
   // appear in 'Pointers' to enforce determinism.
for (unsigned I = 0; I < Pointers.size(); ++I) {
     // iteration order within an equivalence class member is only dependent on
     // the order in which unions and insertions are performed on the
     // equivalence class, the iteration order is deterministic.
     for (auto MI = DepCands.member_begin(LeaderI), ME = DepCands.member_end();
          MI != ME; ++MI) {
       auto PointerI = PositionMap.find(MI->getPointer());
       assert(PointerI != PositionMap.end() &&
              "pointer in equivalence class not found in PositionMap");
-      unsigned Pointer = PointerI->second;
+      for (unsigned Pointer : PointerI->second) {
         bool Merged = false;
         // Mark this pointer as seen.
         Seen.insert(Pointer);
 
         // Go through all the existing sets and see if we can find one
         // which can include this pointer.
         for (RuntimeCheckingPtrGroup &Group : Groups) {
           // Don't perform more than a certain amount of comparisons.
           // This should limit the cost of grouping the pointers to something
           // reasonable.  If we do end up hitting this threshold, the algorithm
           // will create separate groups for all remaining pointers.
           if (TotalComparisons > MemoryCheckMergeThreshold)
             break;
 
           TotalComparisons++;
 
           if (Group.addPointer(Pointer, *this)) {
             Merged = true;
             break;
           }
         }
 
         if (!Merged)
           // We couldn't add this pointer to any existing set or the threshold
           // for the number of comparisons has been reached. Create a new group
           // to hold the current pointer.
           Groups.push_back(RuntimeCheckingPtrGroup(Pointer, *this));
       }
+    }
 
     // We've computed the grouped checks for this partition.
     // Save the results and continue with the next one.
     llvm::copy(Groups, std::back_inserter(CheckingGroups));
   }
 }
 
 bool RuntimePointerChecking::arePointersInSamePartition(
private:
   PredicatedScalarEvolution &PSE;
 };
 
 } // end anonymous namespace
 
 /// Check whether a pointer can participate in a runtime bounds check.
 /// If \p Assume, try harder to prove that we can compute the bounds of \p Ptr
 /// by adding run-time checks (overflow checks) if necessary.
-static bool hasComputableBounds(PredicatedScalarEvolution &PSE,
-                                const ValueToValueMap &Strides, Value *Ptr,
-                                Loop *L, bool Assume) {
-  const SCEV *PtrScev = replaceSymbolicStrideSCEV(PSE, Strides, Ptr);
-
+static bool hasComputableBounds(PredicatedScalarEvolution &PSE, Value *Ptr,
+                                const SCEV *PtrScev, Loop *L, bool Assume) {
   // The bounds for loop-invariant pointer is trivial.
   if (PSE.getSE()->isLoopInvariant(PtrScev, L))
     return true;
 
   const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(PtrScev);
 
   if (!AR && Assume)
     AR = PSE.getAsAddRec(Ptr);
bool AccessAnalysis::createCheckForAccess(RuntimePointerChecking &RtCheck,
                                           MemAccessInfo Access,
                                           const ValueToValueMap &StridesMap,
                                           DenseMap<Value *, unsigned> &DepSetId,
                                           Loop *TheLoop, unsigned &RunningDepId,
                                           unsigned ASId, bool ShouldCheckWrap,
                                           bool Assume) {
   Value *Ptr = Access.getPointer();
 
-  if (!hasComputableBounds(PSE, StridesMap, Ptr, TheLoop, Assume))
+  auto TranslatePointers = [&](Value *Ptr) -> SmallVector<const SCEV *> {
+    ScalarEvolution &SE = *PSE.getSE();
+    auto *GEP = dyn_cast<GetElementPtrInst>(Ptr);
+    auto AssumeInBoundsFlags = [&]() {
+      if (!GEP->isInBounds())
+        return false;
+
+      // We'd like to propagate flags from the IR to the corresponding
+      // SCEV nodes, but to do that, we have to ensure that said flag is
+      // valid in the entire defined scope of the SCEV.
+      auto *GEPI = dyn_cast<Instruction>(GEP);
+      // TODO: non-instructions have global scope.  We might be able to
+      // prove some global scope cases
+      return GEPI && programUndefinedIfPoison(GEPI);
+    };
+
+    if (GEP && GEP->getNumOperands() == 2) {
+      if (auto *SI = dyn_cast<SelectInst>(GEP->getOperand(0))) {
+        const SCEV *BaseA = SE.getSCEV(SI->getOperand(1));
+        const SCEV *BaseB = SE.getSCEV(SI->getOperand(2));
+        const SCEV *Offset = SE.getSCEV(GEP->getOperand(1));
+        if (SE.getTypeSizeInBits(Offset->getType()) <
+            SE.getTypeSizeInBits(BaseA->getType()))
+          Offset = SE.getSignExtendExpr(
+              Offset, SE.getEffectiveSCEVType(BaseA->getType()));
+
+        SCEV::NoWrapFlags OffsetWrap =
+            AssumeInBoundsFlags() ? SCEV::FlagNSW : SCEV::FlagAnyWrap;
+
+        Type *IntIdxTy = SE.getEffectiveSCEVType(BaseA->getType());
+        auto *CurTy = GEP->getSourceElementType();
+        const SCEV *ElementSize = SE.getSizeOfExpr(IntIdxTy, CurTy);
+        // Getelementptr indices are signed.
+        Offset = SE.getTruncateOrSignExtend(Offset, IntIdxTy);
+
+        // Multiply the index by the element size to compute the element
+        // offset.
+        Offset = SE.getMulExpr(Offset, ElementSize, OffsetWrap);
+        auto *PtrA = SE.getAddExpr(BaseA, Offset, SCEV::FlagNUW);
+        auto *PtrB = SE.getAddExpr(BaseB, Offset, SCEV::FlagNUW);
+        return {PtrA, PtrB};
+      } else if (match(GEP->getOperand(1),
+                       m_ZExt(m_Select(m_Value(), m_Value(), m_Value())))) {
+        auto *ZExt = cast<ZExtInst>(GEP->getOperand(1));
+        auto *SI = cast<SelectInst>(ZExt->getOperand(0));
+        const SCEV *OffsetA = SE.getSCEV(SI->getOperand(1));
+        const SCEV *OffsetB = SE.getSCEV(SI->getOperand(2));
+
+        SCEV::NoWrapFlags OffsetWrap =
+            AssumeInBoundsFlags() ? SCEV::FlagNSW : SCEV::FlagAnyWrap;
+
+        auto *Base = SE.getSCEV(GEP->getOperand(0));
+        Type *IntIdxTy = SE.getEffectiveSCEVType(Base->getType());
+        auto *CurTy = GEP->getSourceElementType();
+        const SCEV *ElementSize = SE.getSizeOfExpr(IntIdxTy, CurTy);
+        // Getelementptr indices are signed.
+        OffsetA = SE.getTruncateOrSignExtend(OffsetA, IntIdxTy);
+        OffsetB = SE.getTruncateOrSignExtend(OffsetB, IntIdxTy);
+
+        // Multiply the index by the element size to compute the element
+        // offset.
+        OffsetA = SE.getMulExpr(OffsetA, ElementSize, OffsetWrap);
+        OffsetB = SE.getMulExpr(OffsetB, ElementSize, OffsetWrap);
+        auto *PtrA = SE.getAddExpr(Base, OffsetA, SCEV::FlagNUW);
+        auto *PtrB = SE.getAddExpr(Base, OffsetB, SCEV::FlagNUW);
+        return {PtrA, PtrB};
+      }
+    }
+    return {replaceSymbolicStrideSCEV(PSE, StridesMap, Ptr)};
+  };
+
+  SmallVector<const SCEV *> TranslatedPtrs = TranslatePointers(Ptr);
+
+  for (const SCEV *PtrExpr : TranslatedPtrs) {
+    if (!hasComputableBounds(PSE, Ptr, PtrExpr, TheLoop, Assume))
       return false;
 
     // When we run after a failing dependency check we have to make sure
     // we don't have wrapping pointers.
-  if (ShouldCheckWrap && !isNoWrap(PSE, StridesMap, Ptr, TheLoop)) {
+    if (ShouldCheckWrap) {
+      if (TranslatedPtrs.size() > 1) {
+        return false;
+      }
+      if (!isNoWrap(PSE, StridesMap, Ptr, TheLoop)) {
         auto *Expr = PSE.getSCEV(Ptr);
         if (!Assume || !isa<SCEVAddRecExpr>(Expr))
           return false;
         PSE.setNoOverflow(Ptr, SCEVWrapPredicate::IncrementNUSW);
       }
+    }
+    // If there's only one option for Ptr, look it up after bounds and wrap
+    // checking, because assumptions might have been added to PSE.
+    if (TranslatedPtrs.size() == 1)
+      TranslatedPtrs[0] = replaceSymbolicStrideSCEV(PSE, StridesMap, Ptr);
+  }
 
+  for (const SCEV *PtrExpr : TranslatedPtrs) {
     // The id of the dependence set.
     unsigned DepId;
 
     if (isDependencyCheckNeeded()) {
       Value *Leader = DepCands.getLeaderValue(Access).getPointer();
       unsigned &LeaderId = DepSetId[Leader];
       if (!LeaderId)
         LeaderId = RunningDepId++;
       DepId = LeaderId;
     } else
       // Each access has its own dependence set.
       DepId = RunningDepId++;
 
     bool IsWrite = Access.getInt();
-  RtCheck.insert(TheLoop, Ptr, IsWrite, DepId, ASId, StridesMap, PSE);
+    RtCheck.insert(TheLoop, Ptr, PtrExpr, IsWrite, DepId, ASId, PSE);
     LLVM_DEBUG(dbgs() << "LAA: Found a runtime check ptr:" << *Ptr << '\n');
+  }
 
   return true;
  }
 
 bool AccessAnalysis::canCheckPtrAtRT(RuntimePointerChecking &RtCheck,
                                      ScalarEvolution *SE, Loop *TheLoop,
                                      const ValueToValueMap &StridesMap,
                                      bool ShouldCheckWrap) {

llvm/test/Analysis/LoopAccessAnalysis/forked-pointers.ll

 ; RUN: opt -loop-accesses -analyze -enable-new-pm=0 %s 2>&1 | FileCheck %s
 ; RUN: opt -disable-output -passes='require<scalar-evolution>,require<aa>,loop(print-access-info)' %s 2>&1 | FileCheck %s
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 
 ; CHECK-LABEL: function 'forked_ptrs_different_base_same_offset':
 ; CHECK-NEXT:  for.body:
-; CHECK-NEXT:    Report: cannot identify array bounds
-; CHECK-NEXT:    Dependences:
-; CHECK-NEXT:    Run-time memory checks:
-; CHECK-NEXT:    Grouped accesses:
+; CHECK-NEXT:   Memory dependences are safe with run-time checks
+; CHECK-NEXT:   Dependences:
+; CHECK-NEXT:   Run-time memory checks:
+; CHECK-NEXT:   Check 0:
+; CHECK-NEXT:     Comparing group
+; CHECK-NEXT:       %1 = getelementptr inbounds float, float* %Dest, i64 %indvars.iv
+; CHECK-NEXT:     Against group
+; CHECK-NEXT:       %arrayidx = getelementptr inbounds i32, i32* %Preds, i64 %indvars.iv
+; CHECK-NEXT:   Check 1:
+; CHECK-NEXT:     Comparing group
+; CHECK-NEXT:       %1 = getelementptr inbounds float, float* %Dest, i64 %indvars.iv
+; CHECK-NEXT:     Against group
+; CHECK-NEXT:       %.sink.in = getelementptr inbounds float, float* %spec.select, i64 %indvars.iv
+; CHECK-NEXT:   Check 2:
+; CHECK-NEXT:     Comparing group
+; CHECK-NEXT:       %1 = getelementptr inbounds float, float* %Dest, i64 %indvars.iv
+; CHECK-NEXT:     Against group
+; CHECK-NEXT:       %.sink.in = getelementptr inbounds float, float* %spec.select, i64 %indvars.iv
+; CHECK-NEXT:   Grouped accesses:
+; CHECK-NEXT:     Group
+; CHECK-NEXT:       (Low: %Dest High: (400 + %Dest))
+; CHECK-NEXT:         Member: {%Dest,+,4}<nuw><%for.body>
+; CHECK-NEXT:     Group
+; CHECK-NEXT:       (Low: %Preds High: (400 + %Preds))
+; CHECK-NEXT:         Member: {%Preds,+,4}<nuw><%for.body>
+; CHECK-NEXT:     Group
+; CHECK-NEXT:       (Low: %Base2 High: (400 + %Base2))
+; CHECK-NEXT:         Member: {%Base2,+,4}<nuw><%for.body>
+; CHECK-NEXT:     Group
+; CHECK-NEXT:       (Low: %Base1 High: (400 + %Base1))
+; CHECK-NEXT:         Member: {%Base1,+,4}<nuw><%for.body>
 ; CHECK-EMPTY:
 ; CHECK-NEXT:    Non vectorizable stores to invariant address were not found in loop.
 ; CHECK-NEXT:    SCEV assumptions:
 ; CHECK-EMPTY:
 ; CHECK-NEXT:    Expressions re-written:
 
 ;;;; Derived from the following C code
 ;; void forked_ptrs_different_base_same_offset(float *A, float *B, float *C, int *D) {
for.body:
   store float %.sink, float* %1, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond.not = icmp eq i64 %indvars.iv.next, 100
   br i1 %exitcond.not, label %for.cond.cleanup, label %for.body
 }
 
 ; CHECK-LABEL: function 'forked_ptrs_same_base_different_offset':
 ; CHECK-NEXT:   for.body:
-; CHECK-NEXT:     Report: cannot identify array bounds
-; CHECK-NEXT:     Dependences:
-; CHECK-NEXT:     Run-time memory checks:
-; CHECK-NEXT:     Grouped accesses:
+; CHECK-NEXT:   Memory dependences are safe with run-time checks
+; CHECK-NEXT:   Dependences:
+; CHECK-NEXT:   Run-time memory checks:
+; CHECK-NEXT:   Check 0:
+; CHECK-NEXT:     Comparing group ([[GRP1:.+]]):
+; CHECK-NEXT:       %arrayidx5 = getelementptr inbounds float, float* %Dest, i64 %indvars.iv
+; CHECK-NEXT:     Against group ([[GRP2:.+]]):
+; CHECK-NEXT:       %arrayidx = getelementptr inbounds i32, i32* %Preds, i64 %indvars.iv
+; CHECK-NEXT:   Check 1:
+; CHECK-NEXT:     Comparing group ([[GRP1]]):
+; CHECK-NEXT:       %arrayidx5 = getelementptr inbounds float, float* %Dest, i64 %indvars.iv
+; CHECK-NEXT:     Against group ([[GRP3:.+]]):
+; CHECK-NEXT:       %arrayidx3 = getelementptr inbounds float, float* %Base, i64 %idxprom213
+; CHECK-NEXT:       %arrayidx3 = getelementptr inbounds float, float* %Base, i64 %idxprom213
+; CHECK-NEXT:   Grouped accesses:
+; CHECK-NEXT:     Group [[GRP1]]:
+; CHECK-NEXT:       (Low: %Dest High: (400 + %Dest))
+; CHECK-NEXT:         Member: {%Dest,+,4}<nuw><%for.body>
+; CHECK-NEXT:     Group [[GRP2]]:
+; CHECK-NEXT:       (Low: %Preds High: (400 + %Preds))
+; CHECK-NEXT:         Member: {%Preds,+,4}<nuw><%for.body>
+; CHECK-NEXT:     Group [[GRP3]]:
+; CHECK-NEXT:       (Low: %Base High: (404 + %Base))
+; CHECK-NEXT:         Member: {(4 + %Base)<nuw>,+,4}<nuw><%for.body>
+; CHECK-NEXT:         Member: {%Base,+,4}<nuw><%for.body>
 ; CHECK-EMPTY:
-; CHECK-NEXT:     Non vectorizable stores to invariant address were not found in loop.
-; CHECK-NEXT:     SCEV assumptions:
-; CHECK-EMPTY:
-; CHECK-NEXT:     Expressions re-written:
+; CHECK-NEXT:  Non vectorizable stores to invariant address were not found in loop.
 
 ;;;; Derived from the following C code
 ;; void forked_ptrs_same_base_different_offset(float *A, float *B, int *C) {
 ;;   int offset;
 ;;   for (int i = 0; i < 100; i++) {
 ;;     if (C[i] != 0)
 ;;       offset = i;
 ;;     else

llvm/test/Transforms/LoopVectorize/forked-pointers.ll

 ;;       C[i] = B[i];
 ;;     }
 ;;   }
 ;; }
 
 define dso_local void @forked_ptrs_different_base_same_offset(float* nocapture readonly %Base1, float* nocapture readonly %Base2, float* nocapture %Dest, i32* nocapture readonly %Preds) {
 ; CHECK-LABEL: @forked_ptrs_different_base_same_offset(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
+; CHECK:       vector.memcheck:
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr float, float* [[DEST:%.*]], i64 100
+; CHECK-NEXT:    [[SCEVGEP4:%.*]] = getelementptr i32, i32* [[PREDS:%.*]], i64 100
+; CHECK-NEXT:    [[SCEVGEP7:%.*]] = getelementptr float, float* [[BASE2:%.*]], i64 100
+; CHECK-NEXT:    [[SCEVGEP10:%.*]] = getelementptr float, float* [[BASE1:%.*]], i64 100
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SCEVGEP4]] to float*
+; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ugt float* [[TMP0]], [[DEST]]
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast float* [[SCEVGEP]] to i32*
+; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ugt i32* [[TMP1]], [[PREDS]]
+; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
+; CHECK-NEXT:    [[BOUND012:%.*]] = icmp ugt float* [[SCEVGEP7]], [[DEST]]
+; CHECK-NEXT:    [[BOUND113:%.*]] = icmp ugt float* [[SCEVGEP]], [[BASE2]]
+; CHECK-NEXT:    [[FOUND_CONFLICT14:%.*]] = and i1 [[BOUND012]], [[BOUND113]]
+; CHECK-NEXT:    [[CONFLICT_RDX:%.*]] = or i1 [[FOUND_CONFLICT]], [[FOUND_CONFLICT14]]
+; CHECK-NEXT:    [[BOUND015:%.*]] = icmp ugt float* [[SCEVGEP10]], [[DEST]]
+; CHECK-NEXT:    [[BOUND116:%.*]] = icmp ugt float* [[SCEVGEP]], [[BASE1]]
+; CHECK-NEXT:    [[FOUND_CONFLICT17:%.*]] = and i1 [[BOUND015]], [[BOUND116]]
+; CHECK-NEXT:    [[CONFLICT_RDX18:%.*]] = or i1 [[CONFLICT_RDX]], [[FOUND_CONFLICT17]]
+; CHECK-NEXT:    br i1 [[CONFLICT_RDX18]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x float*> poison, float* [[BASE2]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x float*> [[BROADCAST_SPLATINSERT]], <4 x float*> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT19:%.*]] = insertelement <4 x float*> poison, float* [[BASE1]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT20:%.*]] = shufflevector <4 x float*> [[BROADCAST_SPLATINSERT19]], <4 x float*> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = or i64 [[INDEX]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = or i64 [[INDEX]], 2
+; CHECK-NEXT:    [[TMP4:%.*]] = or i64 [[INDEX]], 3
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i32, i32* [[PREDS]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i32* [[TMP5]] to <4 x i32>*
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP6]], align 4, !alias.scope !0
+; CHECK-NEXT:    [[TMP7:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT:    [[TMP8:%.*]] = select <4 x i1> [[TMP7]], <4 x float*> [[BROADCAST_SPLAT]], <4 x float*> [[BROADCAST_SPLAT20]]
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <4 x float*> [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds float, float* [[TMP9]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <4 x float*> [[TMP8]], i32 1
+; CHECK-NEXT:    [[TMP12:%.*]] = getelementptr inbounds float, float* [[TMP11]], i64 [[TMP2]]
+; CHECK-NEXT:    [[TMP13:%.*]] = extractelement <4 x float*> [[TMP8]], i32 2
+; CHECK-NEXT:    [[TMP14:%.*]] = getelementptr inbounds float, float* [[TMP13]], i64 [[TMP3]]
+; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <4 x float*> [[TMP8]], i32 3
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds float, float* [[TMP15]], i64 [[TMP4]]
+; CHECK-NEXT:    [[TMP17:%.*]] = load float, float* [[TMP10]], align 4, !alias.scope !3
+; CHECK-NEXT:    [[TMP18:%.*]] = load float, float* [[TMP12]], align 4, !alias.scope !3
+; CHECK-NEXT:    [[TMP19:%.*]] = load float, float* [[TMP14]], align 4, !alias.scope !3
+; CHECK-NEXT:    [[TMP20:%.*]] = load float, float* [[TMP16]], align 4, !alias.scope !3
+; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <4 x float> poison, float [[TMP17]], i32 0
+; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <4 x float> [[TMP21]], float [[TMP18]], i32 1
+; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <4 x float> [[TMP22]], float [[TMP19]], i32 2
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <4 x float> [[TMP23]], float [[TMP20]], i32 3
+; CHECK-NEXT:    [[TMP25:%.*]] = getelementptr inbounds float, float* [[DEST]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP26:%.*]] = bitcast float* [[TMP25]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP24]], <4 x float>* [[TMP26]], align 4, !alias.scope !5, !noalias !7
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[TMP27:%.*]] = icmp eq i64 [[INDEX_NEXT]], 100
+; CHECK-NEXT:    br i1 [[TMP27]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    br i1 true, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 100, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup:
 ; CHECK-NEXT:    ret void
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[PREDS:%.*]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[CMP1_NOT:%.*]] = icmp eq i32 [[TMP0]], 0
-; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[CMP1_NOT]], float* [[BASE2:%.*]], float* [[BASE1:%.*]]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[PREDS]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[TMP28:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[CMP1_NOT:%.*]] = icmp eq i32 [[TMP28]], 0
+; CHECK-NEXT:    [[SPEC_SELECT:%.*]] = select i1 [[CMP1_NOT]], float* [[BASE2]], float* [[BASE1]]
 ; CHECK-NEXT:    [[DOTSINK_IN:%.*]] = getelementptr inbounds float, float* [[SPEC_SELECT]], i64 [[INDVARS_IV]]
 ; CHECK-NEXT:    [[DOTSINK:%.*]] = load float, float* [[DOTSINK_IN]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds float, float* [[DEST:%.*]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    store float [[DOTSINK]], float* [[TMP1]], align 4
+; CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds float, float* [[DEST]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    store float [[DOTSINK]], float* [[TMP29]], align 4
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], 100
-; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY]]
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_COND_CLEANUP]], label [[FOR_BODY]], !llvm.loop [[LOOP11:![0-9]+]]
 ;
 entry:
   br label %for.body
 
 for.cond.cleanup:
   ret void
 
 for.body:

 ;;       offset = i+1;
 ;;     B[i] = A[offset];
 ;;   }
 ;; }
 
 define dso_local void @forked_ptrs_same_base_different_offset(float* nocapture readonly %Base, float* nocapture %Dest, i32* nocapture readonly %Preds) {
 ; CHECK-LABEL: @forked_ptrs_same_base_different_offset(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
+; CHECK:       vector.memcheck:
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr float, float* [[DEST:%.*]], i64 100
+; CHECK-NEXT:    [[SCEVGEP4:%.*]] = getelementptr i32, i32* [[PREDS:%.*]], i64 100
+; CHECK-NEXT:    [[SCEVGEP7:%.*]] = getelementptr float, float* [[BASE:%.*]], i64 101
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SCEVGEP4]] to float*
+; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ugt float* [[TMP0]], [[DEST]]
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast float* [[SCEVGEP]] to i32*
+; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ugt i32* [[TMP1]], [[PREDS]]
+; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
+; CHECK-NEXT:    [[BOUND09:%.*]] = icmp ugt float* [[SCEVGEP7]], [[DEST]]
+; CHECK-NEXT:    [[BOUND110:%.*]] = icmp ugt float* [[SCEVGEP]], [[BASE]]
+; CHECK-NEXT:    [[FOUND_CONFLICT11:%.*]] = and i1 [[BOUND09]], [[BOUND110]]
+; CHECK-NEXT:    [[CONFLICT_RDX:%.*]] = or i1 [[FOUND_CONFLICT]], [[FOUND_CONFLICT11]]
+; CHECK-NEXT:    br i1 [[CONFLICT_RDX]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND13:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT14:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND15:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT16:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[PREDS]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4, !alias.scope !12
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT:    [[TMP5:%.*]] = add nuw nsw <4 x i32> [[VEC_IND13]], <i32 1, i32 1, i32 1, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = select <4 x i1> [[TMP4]], <4 x i32> [[TMP5]], <4 x i32> [[VEC_IND15]]
+; CHECK-NEXT:    [[TMP7:%.*]] = zext <4 x i32> [[TMP6]] to <4 x i64>
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <4 x i64> [[TMP7]], i32 0
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds float, float* [[BASE]], i64 [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <4 x i64> [[TMP7]], i32 1
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds float, float* [[BASE]], i64 [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = extractelement <4 x i64> [[TMP7]], i32 2
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr inbounds float, float* [[BASE]], i64 [[TMP12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <4 x i64> [[TMP7]], i32 3
+; CHECK-NEXT:    [[TMP15:%.*]] = getelementptr inbounds float, float* [[BASE]], i64 [[TMP14]]
+; CHECK-NEXT:    [[TMP16:%.*]] = load float, float* [[TMP9]], align 4, !alias.scope !15
+; CHECK-NEXT:    [[TMP17:%.*]] = load float, float* [[TMP11]], align 4, !alias.scope !15
+; CHECK-NEXT:    [[TMP18:%.*]] = load float, float* [[TMP13]], align 4, !alias.scope !15
+; CHECK-NEXT:    [[TMP19:%.*]] = load float, float* [[TMP15]], align 4, !alias.scope !15
+; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <4 x float> poison, float [[TMP16]], i32 0
+; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <4 x float> [[TMP20]], float [[TMP17]], i32 1
+; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <4 x float> [[TMP21]], float [[TMP18]], i32 2
+; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <4 x float> [[TMP22]], float [[TMP19]], i32 3
+; CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds float, float* [[DEST]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP25:%.*]] = bitcast float* [[TMP24]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP23]], <4 x float>* [[TMP25]], align 4, !alias.scope !17, !noalias !19
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT:    [[VEC_IND_NEXT14]] = add <4 x i32> [[VEC_IND13]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[VEC_IND_NEXT16]] = add <4 x i32> [[VEC_IND15]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[TMP26:%.*]] = icmp eq i64 [[INDEX_NEXT]], 100
+; CHECK-NEXT:    br i1 [[TMP26]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP20:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    br i1 true, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ 100, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
+; CHECK-NEXT:    [[BC_RESUME_VAL12:%.*]] = phi i32 [ 100, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY]] ], [ 0, [[VECTOR_MEMCHECK]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup:
 ; CHECK-NEXT:    ret void
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[I_014:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[ADD:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[PREDS:%.*]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
-; CHECK-NEXT:    [[CMP1_NOT:%.*]] = icmp eq i32 [[TMP0]], 0
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[I_014:%.*]] = phi i32 [ [[BC_RESUME_VAL12]], [[SCALAR_PH]] ], [ [[ADD:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[PREDS]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[TMP27:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[CMP1_NOT:%.*]] = icmp eq i32 [[TMP27]], 0
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[ADD]] = add nuw nsw i32 [[I_014]], 1
-; CHECK-NEXT:    [[TMP1:%.*]] = trunc i64 [[INDVARS_IV]] to i32
-; CHECK-NEXT:    [[OFFSET_0:%.*]] = select i1 [[CMP1_NOT]], i32 [[ADD]], i32 [[TMP1]]
+; CHECK-NEXT:    [[TMP28:%.*]] = trunc i64 [[INDVARS_IV]] to i32
+; CHECK-NEXT:    [[OFFSET_0:%.*]] = select i1 [[CMP1_NOT]], i32 [[ADD]], i32 [[TMP28]]
 ; CHECK-NEXT:    [[IDXPROM213:%.*]] = zext i32 [[OFFSET_0]] to i64
-; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds float, float* [[BASE:%.*]], i64 [[IDXPROM213]]
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[ARRAYIDX3]], align 4
-; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, float* [[DEST:%.*]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    store float [[TMP2]], float* [[ARRAYIDX5]], align 4
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds float, float* [[BASE]], i64 [[IDXPROM213]]
+; CHECK-NEXT:    [[TMP29:%.*]] = load float, float* [[ARRAYIDX3]], align 4
+; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, float* [[DEST]], i64 [[INDVARS_IV]]
+; CHECK-NEXT:    store float [[TMP29]], float* [[ARRAYIDX5]], align 4
 ; CHECK-NEXT:    [[EXITCOND_NOT:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT]], 100
-; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY]]
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT]], label [[FOR_COND_CLEANUP]], label [[FOR_BODY]], !llvm.loop [[LOOP21:![0-9]+]]
 ;
 entry:
   br label %for.body
 
 for.cond.cleanup:                                 ; preds = %for.body
   ret void
 
 for.body:                                         ; preds = %entry, %for.body