Recommit "[LAA] Support pointer phis in loop by analyzing each incoming pointer."

[fhahn]

Summary

SCEV does not look through non-header PHIs inside the loop. Such phis
can be analyzed by adding separate accesses for each incoming pointer
value.

This results in 2 more loops vectorized in SPEC2000/186.crafty and
avoids regressions when sinking instructions before vectorizing.

Fixes PR50296, PR50288.

[fhahn]

There's a bit of duplication that might be good to clean up separately :)

[uabelho]

Thanks! I've verified that this fixes the crashes I saw.

[Meinersbur]

The merge-bot reports the test to be failing. Can you have a look?

Should the PHINode (in addition to its incoming arguments) also marked as accessed?

What it the incoming values themselves are PHINodes?

[bjope]

We still see these failures. Should we perhaps revert rG1ed7f8ede564c3b11da4fdca30c36ccbff422576 while sorting out remaining issues here?
(that was just an optimization afaict, so maybe better to have a compiler that doesn't crash on certain input, or is the loop-distribution considered as some kind of alpha-feature)

[fhahn]

In D102266#2786427, @bjope wrote:

We still see these failures. Should we perhaps revert rG1ed7f8ede564c3b11da4fdca30c36ccbff422576 while sorting out remaining issues here?
(that was just an optimization afaict, so maybe better to have a compiler that doesn't crash on certain input, or is the loop-distribution considered as some kind of alpha-feature)

Fair point. Unfortunately I did not yet have time to get back to this as quickly as I hoped. Reverted the original commit for now (ec1f6f7e3f92d806d0f7e4d687173a150102ec5d)

[uabelho]

Thanks!

[fhahn]

Folded changes from D101286 (which got reverted) into this revision, remove duplication.

I'd appreciate another loop at the whole patch, which now handles phis in both ::addAccess and MemDepChecker.

[Meinersbur]

Should the PHINode itself also be added to the Accesses list? Or is there a guarantee that it is never queried?

llvm/lib/Analysis/LoopAccessAnalysis.cpp

1275

Should this be recursive, the incoming value might be a PHI inside the InnermostLoop as well?

[fhahn]

In D102266#2983003, @Meinersbur wrote:

Should the PHINode itself also be added to the Accesses list? Or is there a guarantee that it is never queried?

I don't think it should be added, I think it's best to handle both the dependency check and runtime check cases in the same fashion. Otherwise it might get a bit confusing if the tracked pointers diverge. If there are any places that would still query the phi node, those would need to be fixed. Adding the phi node might hide such issues. WDYT?

llvm/lib/Analysis/LoopAccessAnalysis.cpp

1275

Yes, that would allow to catch more cases. I put up D109381

[Meinersbur]

IMHO, this and D109381 only make sense together. Either we do register in-loop PHINodes in accesses or we do not. A PHI value might be used directly as well as indirectly giving us a mix of some PHINodes added to AddPointer while others are not. Always (also) registering the PHINode itself at least would not give use regressions. If we consider a PHINode in the accesses list a bug, could you add an assertion for it?

I'd LGTM this patch if either you promise to commit it together with D109381 or merge them.

[fhahn]

In D102266#2992524, @Meinersbur wrote:

IMHO, this and D109381 only make sense together. Either we do register in-loop PHINodes in accesses or we do not. A PHI value might be used directly as well as indirectly giving us a mix of some PHINodes added to AddPointer while others are not. Always (also) registering the PHINode itself at least would not give use regressions. If we consider a PHINode in the accesses list a bug, could you add an assertion for it?

Sounds good!

folded the changes from D109381 into this patch in the latest version. I also updated the code to keep track of visited values to detect cycles and added a test case with cyclic phi nodes with irreducible control flow in 4c84a0f24c10 (together with the other additonal test).

[Meinersbur]

LGTM

llvm/include/llvm/Analysis/LoopAccessAnalysis.h

public:
   MemoryDepChecker(PredicatedScalarEvolution &PSE, const Loop *L)
       : PSE(PSE), InnermostLoop(L), AccessIdx(0), MaxSafeDepDistBytes(0),
         MaxSafeVectorWidthInBits(-1U),
         FoundNonConstantDistanceDependence(false),
         Status(VectorizationSafetyStatus::Safe), RecordDependences(true) {}
 
   /// Register the location (instructions are given increasing numbers)
   /// of a write access.
-  void addAccess(StoreInst *SI) {
-    Value *Ptr = SI->getPointerOperand();
-    Accesses[MemAccessInfo(Ptr, true)].push_back(AccessIdx);
-    InstMap.push_back(SI);
-    ++AccessIdx;
-  }
+  void addAccess(StoreInst *SI);
 
   /// Register the location (instructions are given increasing numbers)
   /// of a write access.
-  void addAccess(LoadInst *LI) {
-    Value *Ptr = LI->getPointerOperand();
-    Accesses[MemAccessInfo(Ptr, false)].push_back(AccessIdx);
-    InstMap.push_back(LI);
-    ++AccessIdx;
-  }
+  void addAccess(LoadInst *LI);
 
   /// Check whether the dependencies between the accesses are safe.
   ///
   /// Only checks sets with elements in \p CheckDeps.
   bool areDepsSafe(DepCandidates &AccessSets, MemAccessInfoList &CheckDeps,
                    const ValueToValueMap &Strides);
 
   /// No memory dependence was encountered that would inhibit

llvm/lib/Analysis/LoopAccessAnalysis.cpp

if (!PtrA || !PtrB)
     return false;
   Type *ElemTyA = getLoadStoreType(A);
   Type *ElemTyB = getLoadStoreType(B);
   Optional<int> Diff = getPointersDiff(ElemTyA, PtrA, ElemTyB, PtrB, DL, SE,
                                        /*StrictCheck=*/true, CheckType);
   return Diff && *Diff == 1;
 }
 
+static void visitPointers(Value *StartPtr, const Loop &InnermostLoop,
+                          function_ref<void(Value *)> AddPointer) {
+  SmallPtrSet<Value *, 8> Visited;
+  SmallVector<Value *> WorkList;
+  WorkList.push_back(StartPtr);
+
+  while (!WorkList.empty()) {
+    Value *Ptr = WorkList.pop_back_val();
+    if (!Visited.insert(Ptr).second)
+      continue;
+    auto *PN = dyn_cast<PHINode>(Ptr);
+    // SCEV does not look through non-header PHIs inside the loop. Such phis
+    // can be analyzed by adding separate accesses for each incoming pointer
+    // value.
+    if (PN && InnermostLoop.contains(PN->getParent()) &&
+        PN->getParent() != InnermostLoop.getHeader()) {
+      for (const Use &Inc : PN->incoming_values())
+        WorkList.push_back(Inc);
+    } else
+      AddPointer(Ptr);
+  }
+}
+
+void MemoryDepChecker::addAccess(StoreInst *SI) {
+  visitPointers(SI->getPointerOperand(), *InnermostLoop,
+                [this, SI](Value *Ptr) {
+                  Accesses[MemAccessInfo(Ptr, true)].push_back(AccessIdx);
+                  InstMap.push_back(SI);
+                  ++AccessIdx;
+                });
+}
+
+void MemoryDepChecker::addAccess(LoadInst *LI) {
+  visitPointers(LI->getPointerOperand(), *InnermostLoop,
+                [this, LI](Value *Ptr) {
+                  Accesses[MemAccessInfo(Ptr, false)].push_back(AccessIdx);
+                  InstMap.push_back(LI);
+                  ++AccessIdx;
+                });
+}
+
 MemoryDepChecker::VectorizationSafetyStatus
 MemoryDepChecker::Dependence::isSafeForVectorization(DepType Type) {
   switch (Type) {
   case NoDep:
   case Forward:
   case BackwardVectorizable:
     return VectorizationSafetyStatus::Safe;
 
if (Seen.insert(Ptr).second) {
 
       MemoryLocation Loc = MemoryLocation::get(ST);
       // The TBAA metadata could have a control dependency on the predication
       // condition, so we cannot rely on it when determining whether or not we
       // need runtime pointer checks.
       if (blockNeedsPredication(ST->getParent(), TheLoop, DT))
         Loc.AATags.TBAA = nullptr;
 
-      Accesses.addStore(Loc);
+      visitPointers(const_cast<Value *>(Loc.Ptr), *TheLoop,
+                    [&Accesses, Loc](Value *Ptr) {
+                      MemoryLocation NewLoc = Loc.getWithNewPtr(Ptr);
+                      Accesses.addStore(NewLoc);
+                    });
     }
   }
 
   if (IsAnnotatedParallel) {
     LLVM_DEBUG(
         dbgs() << "LAA: A loop annotated parallel, ignore memory dependency "
                << "checks.\n");
     CanVecMem = true;
for (LoadInst *LD : Loads) {
 
     MemoryLocation Loc = MemoryLocation::get(LD);
     // The TBAA metadata could have a control dependency on the predication
     // condition, so we cannot rely on it when determining whether or not we
     // need runtime pointer checks.
     if (blockNeedsPredication(LD->getParent(), TheLoop, DT))
       Loc.AATags.TBAA = nullptr;
 
-    Accesses.addLoad(Loc, IsReadOnlyPtr);
+    visitPointers(const_cast<Value *>(Loc.Ptr), *TheLoop,
+                  [&Accesses, Loc, IsReadOnlyPtr](Value *Ptr) {
+                    MemoryLocation NewLoc = Loc.getWithNewPtr(Ptr);
+                    Accesses.addLoad(NewLoc, IsReadOnlyPtr);
+                  });
   }
 
   // If we write (or read-write) to a single destination and there are no
   // other reads in this loop then is it safe to vectorize.
   if (NumReadWrites == 1 && NumReads == 0) {
     LLVM_DEBUG(dbgs() << "LAA: Found a write-only loop!\n");
     CanVecMem = true;
     return;

llvm/test/Analysis/LoopAccessAnalysis/pointer-phis.ll

 ; RUN: opt -passes='require<scalar-evolution>,require<aa>,loop(print-access-info)' -disable-output  < %s 2>&1 | FileCheck %s
 
 %s1 = type { [32000 x double], [32000 x double], [32000 x double] }
 
 define i32 @load_with_pointer_phi_no_runtime_checks(%s1* %data) {
 ; CHECK-LABEL: load_with_pointer_phi_no_runtime_checks
 ; CHECK-NEXT:  loop.header:
-; CHECK-NEXT:    Report: cannot identify array bounds
+; CHECK-NEXT:    Memory dependences are safe
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999

 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define i32 @store_with_pointer_phi_no_runtime_checks(%s1* %data) {
 ; CHECK-LABEL: 'store_with_pointer_phi_no_runtime_checks'
 ; CHECK-NEXT:  loop.header:
-; CHECK-NEXT:    Report: cannot identify array bounds
+; CHECK-NEXT:    Memory dependences are safe
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999

 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define i32 @store_with_pointer_phi_runtime_checks(double* %A, double* %B, double* %C) {
 ; CHECK-LABEL: 'store_with_pointer_phi_runtime_checks'
 ; CHECK-NEXT:  loop.header:
-; CHECK-NEXT:    Report: cannot identify array bounds
+; CHECK-NEXT:    Memory dependences are safe with run-time checks
+; CHECK:         Run-time memory checks:
+; CHECK-NEXT:    Check 0:
+; CHECK-NEXT:      Comparing group ([[GROUP_B:.+]]):
+; CHECK-NEXT:        %gep.1 = getelementptr inbounds double, double* %B, i64 %iv
+; CHECK-NEXT:      Against group ([[GROUP_C:.+]]):
+; CHECK-NEXT:        %gep.2 = getelementptr inbounds double, double* %C, i64 %iv
+; CHECK-NEXT:    Check 1:
+; CHECK-NEXT:      Comparing group ([[GROUP_B]]):
+; CHECK-NEXT:        %gep.1 = getelementptr inbounds double, double* %B, i64 %iv
+; CHECK-NEXT:      Against group ([[GROUP_A:.+]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
+; CHECK-NEXT:    Check 2:
+; CHECK-NEXT:      Comparing group ([[GROUP_C]]):
+; CHECK-NEXT:        %gep.2 = getelementptr inbounds double, double* %C, i64 %iv
+; CHECK-NEXT:      Against group ([[GROUP_A]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999

 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define i32 @store_with_pointer_phi_incoming_phi(double* %A, double* %B, double* %C, i1 %c.0, i1 %c.1) {
 ; CHECK-LABEL: 'store_with_pointer_phi_incoming_phi'
 ; CHECK-NEXT:  loop.header:
-; CHECK-NEXT:     Report: cannot identify array bounds
-; CHECK-NEXT:     Dependences:
-; CHECK-NEXT:     Run-time memory checks:
-; CHECK-NEXT:     Grouped accesses:
+; CHECK-NEXT:    Report: unsafe dependent memory operations in loop. Use #pragma loop distribute(enable) to allow loop distribution to attempt to isolate the offending operations into a separate loop
+; CHECK-NEXT:    Dependences:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          %v8 = load double, double* %arrayidx, align 8 ->
+; CHECK-NEXT:          store double %mul16, double* %ptr.2, align 8
+; CHECK-EMPTY:
+; CHECK-NEXT:    Run-time memory checks:
+; CHECK-NEXT:    Check 0:
+; CHECK-NEXT:      Comparing group ([[GROUP_C:.+]]):
+; CHECK-NEXT:      double* %C
+; CHECK-NEXT:      Against group ([[GROUP_B:.+]]):
+; CHECK-NEXT:      double* %B
+; CHECK-NEXT:    Check 1:
+; CHECK-NEXT:      Comparing group ([[GROUP_C]]):
+; CHECK-NEXT:      double* %C
+; CHECK-NEXT:      Against group ([[GROUP_A:.+]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
+; CHECK-NEXT:      double* %A
+; CHECK-NEXT:    Check 2:
+; CHECK-NEXT:      Comparing group ([[GROUP_B]]):
+; CHECK-NEXT:      double* %B
+; CHECK-NEXT:      Against group ([[GROUP_A]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
+; CHECK-NEXT:      double* %A
+; CHECK-NEXT:    Grouped accesses:
+; CHECK-NEXT:      Group [[GROUP_C]]:
+; CHECK-NEXT:        (Low: %C High: (8 + %C))
+; CHECK-NEXT:          Member: %C
+; CHECK-NEXT:      Group [[GROUP_B]]:
+; CHECK-NEXT:        (Low: %B High: (8 + %B))
+; CHECK-NEXT:          Member: %B
+; CHECK-NEXT:      Group [[GROUP_A]]:
+; CHECK-NEXT:        (Low: %A High: (256000 + %A))
+; CHECK-NEXT:          Member: {%A,+,8}<nuw><%loop.header>
+; CHECK-NEXT:          Member: %A
 ; CHECK-EMPTY
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %arrayidx = getelementptr inbounds double, double* %A, i64 %iv

 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 ; Test cases with pointer phis forming a cycle.
 define i32 @store_with_pointer_phi_incoming_phi_irreducible_cycle(double* %A, double* %B, double* %C, i1 %c.0, i1 %c.1) {
 ; CHECK-LABEL: 'store_with_pointer_phi_incoming_phi_irreducible_cycle'
 ; CHECK-NEXT:  loop.header:
-; CHECK-NEXT:     Report: cannot identify array bounds
-; CHECK-NEXT:     Dependences:
-; CHECK-NEXT:     Run-time memory checks:
-; CHECK-NEXT:     Grouped accesses:
+; CHECK-NEXT:    Report: unsafe dependent memory operations in loop. Use #pragma loop distribute(enable) to allow loop distribution to attempt to isolate the offending operations into a separate loop
+; CHECK-NEXT:    Dependences:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          %v8 = load double, double* %arrayidx, align 8 ->
+; CHECK-NEXT:          store double %mul16, double* %ptr.3, align 8
+; CHECK-EMPTY:
+; CHECK-NEXT:    Run-time memory checks:
+; CHECK-NEXT:    Check 0:
+; CHECK-NEXT:      Comparing group ([[GROUP_C:.+]]):
+; CHECK-NEXT:      double* %C
+; CHECK-NEXT:      Against group ([[GROUP_B:.+]]):
+; CHECK-NEXT:      double* %B
+; CHECK-NEXT:    Check 1:
+; CHECK-NEXT:      Comparing group ([[GROUP_C]]):
+; CHECK-NEXT:      double* %C
+; CHECK-NEXT:      Against group ([[GROUP_A:.+]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
+; CHECK-NEXT:      double* %A
+; CHECK-NEXT:    Check 2:
+; CHECK-NEXT:      Comparing group ([[GROUP_B]]):
+; CHECK-NEXT:      double* %B
+; CHECK-NEXT:      Against group ([[GROUP_A]]):
+; CHECK-NEXT:        %arrayidx = getelementptr inbounds double, double* %A, i64 %iv
+; CHECK-NEXT:      double* %A
+; CHECK-NEXT:    Grouped accesses:
+; CHECK-NEXT:      Group [[GROUP_C]]
+; CHECK-NEXT:        (Low: %C High: (8 + %C))
+; CHECK-NEXT:          Member: %C
+; CHECK-NEXT:      Group [[GROUP_B]]
+; CHECK-NEXT:        (Low: %B High: (8 + %B))
+; CHECK-NEXT:          Member: %B
+; CHECK-NEXT:      Group [[GROUP_A]]
+; CHECK-NEXT:        (Low: %A High: (256000 + %A))
+; CHECK-NEXT:          Member: {%A,+,8}<nuw><%loop.header>
+; CHECK-NEXT:          Member: %A
 ; CHECK-EMPTY
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %arrayidx = getelementptr inbounds double, double* %A, i64 %iv

 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define void @phi_load_store_memdep_check(i1 %c, i16* %A, i16* %B, i16* %C) {
 ; CHECK-LABEL: Loop access info in function 'phi_load_store_memdep_check':
 ; CHECK-NEXT:   for.body:
-; CHECK-NEXT:     Report: cannot identify array bounds
-; CHECK-NEXT:     Dependences:
-; CHECK-NEXT:     Run-time memory checks:
-; CHECK-NEXT:     Grouped accesses:
+; CHECK-NEXT:    Report: unsafe dependent memory operations in loop. Use #pragma loop distribute(enable) to allow loop distribution to attempt to isolate the offending operations into a separate loop
+; CHECK-NEXT:    Dependences:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          %lv3 = load i16, i16* %c.sink, align 2 ->
+; CHECK-NEXT:          store i16 %add, i16* %c.sink, align 1
+; CHECK-EMPTY:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          %lv3 = load i16, i16* %c.sink, align 2 ->
+; CHECK-NEXT:          store i16 %add, i16* %c.sink, align 1
+; CHECK-EMPTY:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          %lv = load i16, i16* %A, align 1 ->
+; CHECK-NEXT:          store i16 %lv, i16* %A, align 1
+; CHECK-EMPTY:
+; CHECK-NEXT:      Unknown:
+; CHECK-NEXT:          store i16 %lv, i16* %A, align 1 ->
+; CHECK-NEXT:          %lv2 = load i16, i16* %A, align 1
+; CHECK-EMPTY:
+; CHECK-NEXT:    Run-time memory checks:
+; CHECK-NEXT:    Check 0:
+; CHECK-NEXT:      Comparing group ([[GROUP_A:.+]]):
+; CHECK-NEXT:      i16* %A
+; CHECK-NEXT:      i16* %A
+; CHECK-NEXT:      Against group ([[GROUP_C:.+]]):
+; CHECK-NEXT:      i16* %C
+; CHECK-NEXT:      i16* %C
+; CHECK-NEXT:    Check 1:
+; CHECK-NEXT:      Comparing group ([[GROUP_A]]):
+; CHECK-NEXT:      i16* %A
+; CHECK-NEXT:      i16* %A
+; CHECK-NEXT:      Against group ([[GROUP_B:.+]]):
+; CHECK-NEXT:      i16* %B
+; CHECK-NEXT:      i16* %B
+; CHECK-NEXT:    Check 2:
+; CHECK-NEXT:      Comparing group ([[GROUP_C]]):
+; CHECK-NEXT:      i16* %C
+; CHECK-NEXT:      i16* %C
+; CHECK-NEXT:      Against group ([[GROUP_B]]):
+; CHECK-NEXT:      i16* %B
+; CHECK-NEXT:      i16* %B
+; CHECK-NEXT:    Grouped accesses:
+; CHECK-NEXT:      Group [[GROUP_A]]
+; CHECK-NEXT:        (Low: %A High: (2 + %A))
+; CHECK-NEXT:          Member: %A
+; CHECK-NEXT:          Member: %A
+; CHECK-NEXT:      Group [[GROUP_C]]
+; CHECK-NEXT:        (Low: %C High: (2 + %C))
+; CHECK-NEXT:          Member: %C
+; CHECK-NEXT:          Member: %C
+; CHECK-NEXT:      Group [[GROUP_B]]
+; CHECK-NEXT:        (Low: %B High: (2 + %B))
+; CHECK-NEXT:          Member: %B
+; CHECK-NEXT:          Member: %B
 ; CHECK-EMPTY:
 ;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %if.end, %entry
   %iv = phi i16 [ 0, %entry ], [ %iv.next, %if.end ]
   %lv = load i16, i16* %A, align 1

llvm/test/Transforms/LoopDistribute/pointer-phi-in-loop.ll

 
 for.end.loopexit:                                 ; preds = %if.end
   ret void
 }
 
 define void @phi_load_distribute(i1 %c, i16* %A, i16* %B, i16* %C) {
 ; CHECK-LABEL: @phi_load_distribute(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
-; CHECK:       for.body:
-; CHECK-NEXT:    [[IV:%.*]] = phi i16 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[IF_END:%.*]] ]
-; CHECK-NEXT:    [[LV:%.*]] = load i16, i16* [[A:%.*]], align 1
-; CHECK-NEXT:    store i16 [[LV]], i16* [[A]], align 1
-; CHECK-NEXT:    br i1 [[C:%.*]], label [[IF_THEN:%.*]], label [[IF_END]]
-; CHECK:       if.then:
-; CHECK-NEXT:    [[LV2:%.*]] = load i16, i16* [[A]], align 1
-; CHECK-NEXT:    br label [[IF_END]]
-; CHECK:       if.end:
-; CHECK-NEXT:    [[C_SINK:%.*]] = phi i16* [ [[B:%.*]], [[IF_THEN]] ], [ [[C:%.*]], [[FOR_BODY]] ]
-; CHECK-NEXT:    [[LV3:%.*]] = load i16, i16* [[C_SINK]], align 2
-; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i16 [[IV]], 1
-; CHECK-NEXT:    [[TOBOOL_NOT:%.*]] = icmp eq i16 [[IV_NEXT]], 1000
-; CHECK-NEXT:    br i1 [[TOBOOL_NOT]], label [[FOR_END_LOOPEXIT:%.*]], label [[FOR_BODY]]
 ; CHECK:       for.end.loopexit:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %if.end, %entry
   %iv = phi i16 [ 0, %entry ], [ %iv.next, %if.end ]

llvm/test/Transforms/LoopVectorize/vectorize-pointer-phis.ll

 ; RUN: opt -loop-vectorize -force-vector-width=2 -S %s | FileCheck %s
 
 %s1 = type { [32000 x double], [32000 x double], [32000 x double] }
 
 define i32 @load_with_pointer_phi_no_runtime_checks(%s1* %data) {
 ; CHECK-LABEL: @load_with_pointer_phi_no_runtime_checks
-; CHECK-NOT: vector.body
+; CHECK-NOT: memcheck
+; CHECK:     vector.body:
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999
loop.latch:                                          ; preds = %if.else, %if.then
   br i1 %exitcond.not, label %exit, label %loop.header
 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define i32 @store_with_pointer_phi_no_runtime_checks(%s1* %data) {
 ; CHECK-LABEL: @store_with_pointer_phi_no_runtime_checks
-; CHECK-NOT: vector.body
+; CHECK-NOT: memcheck
+; CHECK:     vector.body
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999
loop.latch:                                          ; preds = %if.else, %if.then
   br i1 %exitcond.not, label %exit, label %loop.header
 
 exit:                                             ; preds = %loop.latch
   ret i32 10
 }
 
 define i32 @store_with_pointer_phi_runtime_checks(double* %A, double* %B, double* %C) {
 ; CHECK-LABEL: @store_with_pointer_phi_runtime_checks
-; CHECK-NOT: vector.body
+; CHECK:     memcheck
+; CHECK:     vector.body
 ;
 entry:
   br label %loop.header
 
 loop.header:                                        ; preds = %loop.latch, %entry
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
   %iv.next = add nuw nsw i64 %iv, 1
   %cmp5 = icmp ult i64 %iv, 15999