[ValueTracking] getUnderlyingObject() look through phi.

[caojoshua]

Summary

For a pointer phi, we can strenghten the underlying object if each of
its incoming pointers are the same, or the PHI itself.

Some benefits are:

• Stronger alias analysis
• Folding phis that point to a zero initialized constant array

[caojoshua]

small updates

[caojoshua]

small update

[caojoshua]

I think this analysis of objects is correct. Would appreciate if reviewers let me know whether my understanding is correct or not. This change is on top of a local NFC to make getUnderlyingObject() recursive.

Some caveats:

• With unbounded lookups, this could increase compile times since this is a frequently used API. However, there is a default max lookup of 6. I only see one caller that uses a different value of 0, for infinite max lookups. I think overall compile time should see minimal impact.
• This change unintentionally made a bunch of changes to codegen tests. The tests with small changes look fine to me, but I'm unsure about @wrongUseOfPostDominate. I have not work with LLVM CodeGen before, so I don't want to dive too deep into this test in case the idea in this patch is completely wrong. I'd like to get a first round of feedback before I look into it.

llvm/test/Transforms/LoopVectorize/ARM/pointer_iv.ll

884

Is this transformation legal? I believe it is. I tried putting this in alive2, but compilation is failing without any output.. I tried an example in godbolt which is functionally the same, except we GEP directly from the array base. The generated code in that case eliminates the bounds check, so I think it should be fine here as well.

Although the test goes through multiple passes, the removal of the bounds check is occurring during loop-vectorize, and I'm not familiar with the pass. It's not clear to me how this patch is affecting anything in loop-vectorize.

[goldstein.w.n]

llvm/lib/Analysis/ValueTracking.cpp

5647

How often is the cache hit?

5691

Shouldn't V startout as PHI->getIncomingValue(0)?

[caojoshua]

Logic was not quite right. After updating, a bunch more tests are changed.

[caojoshua]

llvm/lib/Analysis/ValueTracking.cpp

5647

Not much with default MaxLookup = 6.

The reason I add this is because with looking through loop PHI's, we will inifinite loop. I don't actually run into this in the test suite, but if I hardcode MaxLookup to default to 0, we infinite loop in 11 InstCombine tests.

5691

I changed the logic so that we initially set NewUnderlying to getIncomingValue(0). This also requires that we check the phi has >0 arguments, or else some JumpThreading tests fail. Its functionally the same.

llvm/test/CodeGen/AMDGPU/promote-alloca-to-lds-phi.ll

131

I don't understand AMDGPU code, but I believe we solve the FIXME by promoting the alloca.

llvm/test/Transforms/LoopVectorize/X86/gather_scatter.ll

859

Similar to ARM/iv_pointer.ll changes. We remove the bounds checks.

[nikic]

If you upload the base revision I can check, but I'm pretty confident that this is not viable from a compile-time perspective. getUnderlyingObject() is very performance critical, and we rely on the fact that this operation is "essentially free". We have a separate getUnderlyingObjects() function that can recurse though selects and phis.

[caojoshua]

Capture Count variable by reference, and increment it once each time Visit() is called.

This hopefully alleviates some compile-time concerns. When looking through PHIs, each operand will increment the same Count.

[caojoshua]

In D152082#4393701, @nikic wrote:

If you upload the base revision I can check, but I'm pretty confident that this is not viable from a compile-time perspective. getUnderlyingObject() is very performance critical, and we rely on the fact that this operation is "essentially free". We have a separate getUnderlyingObjects() function that can recurse though selects and phis.

Please check if you are able to. I have uploaded to https://github.com/caojoshua/llvm-project/tree/underlyingphi.

My hope is that with the default MaxLookup = 6, the compile time does not see significant impact. I just uploaded some new changes that hopefully makes compile time better.

[nikic]

Results: https://llvm-compile-time-tracker.com/?config=NewPM-O3&stat=instructions%3Au&branch=nikic/perf/underlyingphi

As expected, a major regression. I've left some suggestions for possible improvements.

llvm/lib/Analysis/ValueTracking.cpp

5648

SmallDenseMap

5650

Don't use std::function

5653

Only use Visited on the phi path.

[caojoshua]

In D152082#4393799, @nikic wrote:

Results: https://llvm-compile-time-tracker.com/?config=NewPM-O3&stat=instructions%3Au&branch=nikic/perf/underlyingphi

As expected, a major regression. I've left some suggestions for possible improvements.

The NFC patch itself creates a regression. I don't think the recursion is an issue, since I see on my local build the tail recursion is eliminated. I suspect removing the std::function might be sufficient to eliminate regression.

For the main patch, it looks unrealistic to eliminate compile-time regression. I think this functionality could be useful for some callers, maybe it would be good to create a different code path that does not touch the existing getUnderlyingObject.

[caojoshua]

I tried reducing the compile time in https://github.com/caojoshua/llvm-project/tree/underlyingperf. From local results, I'm able to eliminate regressions in the NFC by moving away from std::function. Regressions are still very significant for the functionality changing patch.

llvm/lib/Analysis/ValueTracking.cpp

static bool isSameUnderlyingObjectInLoop(const PHINode *PN,
   //       ...
   if (auto *Load = dyn_cast<LoadInst>(PrevValue))
     if (!L->isLoopInvariant(Load->getPointerOperand()))
       return false;
   return true;
 }
 
 const Value *llvm::getUnderlyingObject(const Value *V, unsigned MaxLookup) {
-  std::function<const Value *(const Value *, unsigned)> Visit =
-      [&](const Value *Root, unsigned Count) {
-        const Value *V = Root;
-        if (!V->getType()->isPointerTy() ||
-            (MaxLookup != 0 && Count == MaxLookup))
-          return V;
+  unsigned Count = 0;
+  DenseMap<const Value *, const Value *> Visited;
+
+  std::function<const Value *(const Value *)> Visit = [&](const Value *Root) {
+    if (Visited.contains(V))
+      return Visited.at(V);
+    if (!V->getType()->isPointerTy() || (MaxLookup != 0 && Count == MaxLookup))
+    Count += 1;
 
-        if (auto *GEP = dyn_cast<GEPOperator>(V)) {
-          V = GEP->getPointerOperand();
-        } else if (Operator::getOpcode(V) == Instruction::BitCast ||
-                   Operator::getOpcode(V) == Instruction::AddrSpaceCast) {
-          V = cast<Operator>(V)->getOperand(0);
-          if (!V->getType()->isPointerTy())
-            return V;
-        } else if (auto *GA = dyn_cast<GlobalAlias>(V)) {
-          if (GA->isInterposable())
-            return V;
-          V = GA->getAliasee();
-        } else {
-          if (auto *PHI = dyn_cast<PHINode>(V)) {
-            // Look through single-arg phi nodes created by LCSSA.
-            if (PHI->getNumIncomingValues() == 1) {
-              V = PHI->getIncomingValue(0);
-              return Visit(V, Count + 1);
-            }
-          } else if (auto *Call = dyn_cast<CallBase>(V)) {
-            // CaptureTracking can know about special capturing properties of
-            // some intrinsics like launder.invariant.group, that can't be
-            // expressed with the attributes, but have properties like returning
-            // aliasing pointer. Because some analysis may assume that
-            // nocaptured pointer is not returned from some special intrinsic
-            // (because function would have to be marked with returns
-            // attribute), it is crucial to use this function because it should
-            // be in sync with CaptureTracking. Not using it may cause weird
-            // miscompilations where 2 aliasing pointers are assumed to noalias.
-            if (auto *RP = getArgumentAliasingToReturnedPointer(Call, false)) {
-              V = RP;
-              return Visit(RP, Count + 1);
-            }
-          }
-
-          return V;
-        }
-        assert(V->getType()->isPointerTy() && "Unexpected operand type!");
-        return Visit(V, Count + 1);
-      };
+        Visited[V] = V;
+        if (PHI->getNumIncomingValues() == 0)
+          return V;
+
+        // We can look through Phi's if each incoming value has the same
+        // underlying object, or is the phi itself.
+        const Value *NewUnderlying = Visit(PHI->getIncomingValue(0));
+        for (unsigned I = 1; I < PHI->getNumIncomingValues(); ++I) {
+          const Value *IncomingUnderlying = Visit(PHI->getIncomingValue(I));
+          if (IncomingUnderlying == V || IncomingUnderlying == NewUnderlying)
+            continue;
+          if (NewUnderlying == V)
+            // Found a new possible underlying object.
+            NewUnderlying = IncomingUnderlying;
+          else // IncomingUnderlying != NewUnderlying
+            // There are >=2 possible underlying objects. We cannot
+            // determine a new underlying object.
+            return V;
+        V = NewUnderlying;
+          return Visit(RP);
+      Visited[Root] = V;
+    Visited[Root] = V;
+    return Visit(V);
 
-  return Visit(V, 0);
+  return Visit(V);
 }
 
 void llvm::getUnderlyingObjects(const Value *V,
                                 SmallVectorImpl<const Value *> &Objects,
                                 LoopInfo *LI, unsigned MaxLookup) {
   SmallPtrSet<const Value *, 4> Visited;
   SmallVector<const Value *, 4> Worklist;
   Worklist.push_back(V);

llvm/test/Analysis/BasicAA/phi-aa.ll

 
 declare void @inc(ptr)
 
 
 ; When we have a chain of phis in nested loops we should recognise if there's
 ; actually only one underlying value.
 ; FIXME: All of these could be NoAlias.
 ; CHECK-LABEL: loop_phi_chain
-; CHECK: MayAlias: i32* %val1, i32* @Y
+; CHECK: NoAlias: i32* %val1, i32* @Y
 ; CHECK: MayAlias: i32* %val2, i32* @Y
 ; CHECK: MayAlias: i32* %val3, i32* @Y
 define void @loop_phi_chain(i32 %a, i32 %b, i32 %c) {
 entry:
   br label %loop1
 
 loop1:
   %n1 = phi i32 [ 0, %entry ], [ %add1, %loop2 ]

llvm/test/Analysis/BasicAA/recphi.ll

 
 ; CHECK-LABEL: Function: notmust: 6 pointers, 0 call sites
 ; CHECK: MustAlias: i8* %tab, [2 x i32]* %tab
 ; CHECK: PartialAlias (off -4): i32* %arrayidx, [2 x i32]* %tab
 ; CHECK: NoAlias: i32* %arrayidx, i8* %tab
 ; CHECK: MustAlias: i32* %tab, [2 x i32]* %tab
 ; CHECK: MustAlias: i32* %tab, i8* %tab
 ; CHECK: NoAlias: i32* %arrayidx, i32* %tab
-; CHECK: MayAlias:  i32* %incdec.ptr.i, [2 x i32]* %tab
+; CHECK: PartialAlias:  i32* %incdec.ptr.i, [2 x i32]* %tab
 ; CHECK: NoAlias: i32* %incdec.ptr.i, i8* %tab
 ; CHECK: MayAlias:  i32* %arrayidx, i32* %incdec.ptr.i
 ; CHECK: NoAlias: i32* %incdec.ptr.i, i32* %tab
-; CHECK: MayAlias:  i32* %p.addr.05.i, [2 x i32]* %tab
+; CHECK: PartialAlias:  i32* %p.addr.05.i, [2 x i32]* %tab
 ; CHECK: MayAlias:  i32* %p.addr.05.i, i8* %tab
 ; CHECK: MayAlias:  i32* %arrayidx, i32* %p.addr.05.i
 ; CHECK: MayAlias:  i32* %p.addr.05.i, i32* %tab
 ; CHECK: NoAlias: i32* %incdec.ptr.i, i32* %p.addr.05.i
 define i32 @notmust() nounwind {
 entry:
   %tab = alloca [2 x i32], align 4
   %ignore1 = load [2 x i32], ptr %tab

 
 ; CHECK-LABEL: Function: reverse: 6 pointers, 0 call sites
 ; CHECK: MustAlias: i8* %tab, [10 x i32]* %tab
 ; CHECK: MustAlias: i32* %tab, [10 x i32]* %tab
 ; CHECK: MustAlias: i32* %tab, i8* %tab
 ; CHECK: PartialAlias (off -36):  i32* %arrayidx1, [10 x i32]* %tab
 ; CHECK: NoAlias: i32* %arrayidx1, i8* %tab
 ; CHECK: NoAlias: i32* %arrayidx1, i32* %tab
-; CHECK: MayAlias:  i32* %incdec.ptr.i, [10 x i32]* %tab
+; CHECK: PartialAlias:  i32* %incdec.ptr.i, [10 x i32]* %tab
 ; CHECK: MayAlias:  i32* %incdec.ptr.i, i8* %tab
 ; CHECK: MayAlias:  i32* %incdec.ptr.i, i32* %tab
 ; CHECK: MayAlias:  i32* %arrayidx1, i32* %incdec.ptr.i
-; CHECK: MayAlias:  i32* %p.addr.05.i, [10 x i32]* %tab
+; CHECK: PartialAlias:  i32* %p.addr.05.i, [10 x i32]* %tab
 ; CHECK: MayAlias:  i32* %p.addr.05.i, i8* %tab
 ; CHECK: MayAlias:  i32* %p.addr.05.i, i32* %tab
 ; CHECK: MayAlias:  i32* %arrayidx1, i32* %p.addr.05.i
 ; CHECK: NoAlias: i32* %incdec.ptr.i, i32* %p.addr.05.i
 define i32 @reverse() nounwind {
 entry:
   %tab = alloca [10 x i32], align 4
   %ignore1 = load [10 x i32], ptr %tab
if.then: ; preds = %f.exit
   unreachable
 
 if.end: ; preds = %f.exit
   ret i32 0
 }
 
 ; CHECK-LABEL: Function: negative: 5 pointers, 1 call sites
 ; CHECK: PartialAlias (off -4): i16* %_tmp1, [3 x i16]* %int_arr.10
-; CHECK: MayAlias:  [3 x i16]* %int_arr.10, i16* %ls1.9.0
+; CHECK: PartialAlias:  [3 x i16]* %int_arr.10, i16* %ls1.9.0
 ; CHECK: MayAlias:  i16* %_tmp1, i16* %ls1.9.0
-; CHECK: MayAlias:  i16* %_tmp7, [3 x i16]* %int_arr.10
+; CHECK: PartialAlias:  i16* %_tmp7, [3 x i16]* %int_arr.10
 ; CHECK: MayAlias:  i16* %_tmp1, i16* %_tmp7
 ; CHECK: NoAlias: i16* %_tmp7, i16* %ls1.9.0
 ; CHECK: PartialAlias (off -2): i16* %_tmp11, [3 x i16]* %int_arr.10
 ; CHECK: NoAlias: i16* %_tmp1, i16* %_tmp11
 ; CHECK: MayAlias:  i16* %_tmp11, i16* %ls1.9.0
 ; CHECK: MayAlias:  i16* %_tmp11, i16* %_tmp7
 ; CHECK: NoModRef:  Ptr: [3 x i16]* %int_arr.10 <->  %_tmp16 = call i16 @call(i32 %_tmp13)
 ; CHECK: NoModRef:  Ptr: i16* %_tmp1  <->  %_tmp16 = call i16 @call(i32 %_tmp13)
-; CHECK: Both ModRef:  Ptr: i16* %ls1.9.0 <->  %_tmp16 = call i16 @call(i32 %_tmp13)
-; CHECK: Both ModRef:  Ptr: i16* %_tmp7 <->  %_tmp16 = call i16 @call(i32 %_tmp13)
+; CHECK: NoModRef:  Ptr: i16* %ls1.9.0  <->  %_tmp16 = call i16 @call(i32 %_tmp13)
+; CHECK: NoModRef:  Ptr: i16* %_tmp7  <->  %_tmp16 = call i16 @call(i32 %_tmp13)
 ; CHECK: NoModRef:  Ptr: i16* %_tmp11 <->  %_tmp16 = call i16 @call(i32 %_tmp13)
 define i16 @negative(i16 %argc.5.par) {
   %int_arr.10 = alloca [3 x i16], align 1
   load [3 x i16], ptr %int_arr.10
   %_tmp1 = getelementptr inbounds [3 x i16], ptr %int_arr.10, i16 0, i16 2
   load i16, ptr %_tmp1
   br label %bb1
 
exit:
   ret void
 }
 
 ; Same as the previous test case, but avoiding phi of phi.
 ; CHECK-LABEL: Function: nested_loop2
 ; CHECK: NoAlias:  i8* %a, i8* %p.base
 ; CHECK: NoAlias:  i8* %a, i8* %p.outer
 ; CHECK: NoAlias:  i8* %a, i8* %p.outer.next
-; CHECK: MayAlias: i8* %a, i8* %p.inner
+; CHECK: NoAlias:  i8* %a, i8* %p.inner
 ; CHECK: NoAlias:  i8* %a, i8* %p.inner.next
 ; TODO: (a, p.inner) could be NoAlias
 define void @nested_loop2(i1 %c, i1 %c2, ptr noalias %p.base) {
 entry:
   %a = alloca i8
   load i8, ptr %p.base
   load i8, ptr %a
   br label %outer_loop

 exit:
   ret void
 }
 
 ; CHECK-LABEL: Function: sibling_loop
 ; CHECK: NoAlias: i8* %a, i8* %p.base
 ; CHECK: NoAlias: i8* %a, i8* %p1
 ; CHECK: NoAlias: i8* %a, i8* %p1.next
-; CHECK: MayAlias:  i8* %a, i8* %p2
+; CHECK: NoAlias: i8* %a, i8* %p2
 ; CHECK: NoAlias: i8* %a, i8* %p2.next
 ; TODO: %p2 does not alias %a
 define void @sibling_loop(i1 %c, i1 %c2, ptr noalias %p.base) {
 entry:
   %a = alloca i8
   load i8, ptr %p.base
   load i8, ptr %a
   br label %loop1

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

 ; CHECK-NEXT:    mov w10, w2
 ; CHECK-NEXT:    cmp w2, #15
 ; CHECK-NEXT:    b.hi .LBB5_4
 ; CHECK-NEXT:  // %bb.2:
 ; CHECK-NEXT:    mov x11, xzr
 ; CHECK-NEXT:    mov w8, wzr
 ; CHECK-NEXT:    b .LBB5_7
 ; CHECK-NEXT:  .LBB5_3:
-; CHECK-NEXT:    mov w8, wzr
-; CHECK-NEXT:    b .LBB5_9
+; CHECK-NEXT:    mov w0, wzr
+; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB5_4: // %vector.ph
 ; CHECK-NEXT:    and x11, x10, #0xfffffff0
 ; CHECK-NEXT:    add x8, x0, #8
 ; CHECK-NEXT:    movi v0.2d, #0000000000000000
 ; CHECK-NEXT:    mov x12, x11
 ; CHECK-NEXT:    movi v1.2d, #0000000000000000
 ; CHECK-NEXT:    dup v2.8h, w9
 ; CHECK-NEXT:  .LBB5_5: // %vector.body

llvm/test/CodeGen/AMDGPU/promote-alloca-to-lds-phi.ll

 
 ;     #pragma nounroll
 ;     for (int* p = &alloca[2], *e = &alloca[48]; p != e; ++p, ++i)
 ;     {
 ;         *p = i;
 ;     }
 ; }
 
-; FIXME: This should be promotable. We need to use
-; getUnderlyingObjects when looking at the icmp user.
-
 ; CHECK-LABEL: @ptr_induction_var_same_alloca(
-; CHECK: %alloca = alloca [64 x i32], align 4
-; CHECK: phi ptr addrspace(5) [ %arrayidx, %entry ], [ %incdec.ptr, %for.body ]
+; CHECK-NOT: {{.*}} = alloca
+; CHECK: phi ptr addrspace(3) [ %arrayidx, %entry ], [ %incdec.ptr, %for.body ]
 define amdgpu_kernel void @ptr_induction_var_same_alloca() #0 {
 entry:
   %alloca = alloca [64 x i32], align 4, addrspace(5)
   %arrayidx = getelementptr inbounds [64 x i32], ptr addrspace(5) %alloca, i32 0, i32 2
   %arrayidx1 = getelementptr inbounds [64 x i32], ptr addrspace(5) %alloca, i32 0, i32 48
   br label %for.body
 
 for.cond.cleanup:                                 ; preds = %for.body

llvm/test/CodeGen/ARM/arm-shrink-wrapping-linux.ll

 ; This happened because of a bad use of the post-dominance property.
 ; The exit block of the loop happens to also lead to defs/uses of CSRs.
 ; It also post-dominates the loop body and we use to generate invalid
 ; restore sequence. I.e., we restored too early.
 
 define fastcc ptr @wrongUseOfPostDominate(ptr readonly %s, i32 %off, ptr readnone %lim) {
 ; ENABLE-LABEL: wrongUseOfPostDominate:
 ; ENABLE:       @ %bb.0: @ %entry
-; ENABLE-NEXT:    .save {r11, lr}
-; ENABLE-NEXT:    push {r11, lr}
 ; ENABLE-NEXT:    cmn r1, #1
 ; ENABLE-NEXT:    ble .LBB0_7
 ; ENABLE-NEXT:  @ %bb.1: @ %while.cond.preheader
 ; ENABLE-NEXT:    cmp r1, #0
 ; ENABLE-NEXT:    beq .LBB0_6
 ; ENABLE-NEXT:  @ %bb.2: @ %while.cond.preheader
 ; ENABLE-NEXT:    cmp r0, r2
-; ENABLE-NEXT:    pophs {r11, pc}
-; ENABLE-NEXT:  .LBB0_3: @ %while.body.preheader
+; ENABLE-NEXT:    bhs .LBB0_6
+; ENABLE-NEXT:  @ %bb.3: @ %while.body.preheader
 ; ENABLE-NEXT:    movw r12, :lower16:skip
 ; ENABLE-NEXT:    sub r1, r1, #1
 ; ENABLE-NEXT:    movt r12, :upper16:skip
 ; ENABLE-NEXT:  .LBB0_4: @ %while.body
 ; ENABLE-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; ENABLE-NEXT:    ldrb r3, [r0]
 ; ENABLE-NEXT:    ldrb r3, [r12, r3]
 ; ENABLE-NEXT:    add r0, r0, r3
 ; ENABLE-NEXT:    sub r3, r1, #1
 ; ENABLE-NEXT:    cmp r3, r1
-; ENABLE-NEXT:    bhs .LBB0_6
-; ENABLE-NEXT:  @ %bb.5: @ %while.body
+; ENABLE-NEXT:    bxhs lr
+; ENABLE-NEXT:  .LBB0_5: @ %while.body
 ; ENABLE-NEXT:    @ in Loop: Header=BB0_4 Depth=1
 ; ENABLE-NEXT:    cmp r0, r2
 ; ENABLE-NEXT:    mov r1, r3
 ; ENABLE-NEXT:    blo .LBB0_4
 ; ENABLE-NEXT:  .LBB0_6: @ %if.end29
-; ENABLE-NEXT:    pop {r11, pc}
-; ENABLE-NEXT:  .LBB0_7: @ %while.cond2.outer
+; ENABLE-NEXT:    bx lr
+; ENABLE-NEXT:  .LBB0_7:
+; ENABLE-NEXT:    .save {r11, lr}
+; ENABLE-NEXT:    push {r11, lr}
+; ENABLE-NEXT:  .LBB0_8: @ %while.cond2.outer
 ; ENABLE-NEXT:    @ =>This Loop Header: Depth=1
-; ENABLE-NEXT:    @ Child Loop BB0_8 Depth 2
-; ENABLE-NEXT:    @ Child Loop BB0_15 Depth 2
+; ENABLE-NEXT:    @ Child Loop BB0_9 Depth 2
+; ENABLE-NEXT:    @ Child Loop BB0_16 Depth 2
 ; ENABLE-NEXT:    mov r3, r0
-; ENABLE-NEXT:  .LBB0_8: @ %while.cond2
-; ENABLE-NEXT:    @ Parent Loop BB0_7 Depth=1
+; ENABLE-NEXT:  .LBB0_9: @ %while.cond2
+; ENABLE-NEXT:    @ Parent Loop BB0_8 Depth=1
 ; ENABLE-NEXT:    @ => This Inner Loop Header: Depth=2
 ; ENABLE-NEXT:    add r1, r1, #1
 ; ENABLE-NEXT:    cmp r1, #1
-; ENABLE-NEXT:    beq .LBB0_18
-; ENABLE-NEXT:  @ %bb.9: @ %while.body4
-; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=2
+; ENABLE-NEXT:    beq .LBB0_19
+; ENABLE-NEXT:  @ %bb.10: @ %while.body4
+; ENABLE-NEXT:    @ in Loop: Header=BB0_9 Depth=2
 ; ENABLE-NEXT:    cmp r3, r2
-; ENABLE-NEXT:    bls .LBB0_8
-; ENABLE-NEXT:  @ %bb.10: @ %if.then7
-; ENABLE-NEXT:    @ in Loop: Header=BB0_7 Depth=1
+; ENABLE-NEXT:    bls .LBB0_9
+; ENABLE-NEXT:  @ %bb.11: @ %if.then7
+; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=1
 ; ENABLE-NEXT:    mov r0, r3
 ; ENABLE-NEXT:    ldrb r12, [r0, #-1]!
 ; ENABLE-NEXT:    sxtb lr, r12
 ; ENABLE-NEXT:    cmn lr, #1
-; ENABLE-NEXT:    bgt .LBB0_7
-; ENABLE-NEXT:  @ %bb.11: @ %if.then7
-; ENABLE-NEXT:    @ in Loop: Header=BB0_7 Depth=1
+; ENABLE-NEXT:    bgt .LBB0_8
+; ENABLE-NEXT:  @ %bb.12: @ %if.then7
+; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=1
 ; ENABLE-NEXT:    cmp r0, r2
-; ENABLE-NEXT:    bls .LBB0_7
-; ENABLE-NEXT:  @ %bb.12: @ %land.rhs14.preheader
-; ENABLE-NEXT:    @ in Loop: Header=BB0_7 Depth=1
-; ENABLE-NEXT:    cmn lr, #1
-; ENABLE-NEXT:    bgt .LBB0_7
+; ENABLE-NEXT:    bls .LBB0_8
 ; ENABLE-NEXT:  @ %bb.13: @ %land.rhs14.preheader
-; ENABLE-NEXT:    @ in Loop: Header=BB0_7 Depth=1
+; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=1
+; ENABLE-NEXT:    cmn lr, #1
+; ENABLE-NEXT:    bgt .LBB0_8
+; ENABLE-NEXT:  @ %bb.14: @ %land.rhs14.preheader
+; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=1
 ; ENABLE-NEXT:    cmp r12, #191
-; ENABLE-NEXT:    bhi .LBB0_7
-; ENABLE-NEXT:  @ %bb.14: @ %while.body24.preheader
-; ENABLE-NEXT:    @ in Loop: Header=BB0_7 Depth=1
+; ENABLE-NEXT:    bhi .LBB0_8
+; ENABLE-NEXT:  @ %bb.15: @ %while.body24.preheader
+; ENABLE-NEXT:    @ in Loop: Header=BB0_8 Depth=1
 ; ENABLE-NEXT:    sub r3, r3, #2
-; ENABLE-NEXT:  .LBB0_15: @ %while.body24
-; ENABLE-NEXT:    @ Parent Loop BB0_7 Depth=1
+; ENABLE-NEXT:  .LBB0_16: @ %while.body24
+; ENABLE-NEXT:    @ Parent Loop BB0_8 Depth=1
 ; ENABLE-NEXT:    @ => This Inner Loop Header: Depth=2
 ; ENABLE-NEXT:    mov r0, r3
 ; ENABLE-NEXT:    cmp r3, r2
-; ENABLE-NEXT:    bls .LBB0_7
-; ENABLE-NEXT:  @ %bb.16: @ %while.body24.land.rhs14_crit_edge
-; ENABLE-NEXT:    @ in Loop: Header=BB0_15 Depth=2
+; ENABLE-NEXT:    bls .LBB0_8
+; ENABLE-NEXT:  @ %bb.17: @ %while.body24.land.rhs14_crit_edge
+; ENABLE-NEXT:    @ in Loop: Header=BB0_16 Depth=2
 ; ENABLE-NEXT:    mov r3, r0
 ; ENABLE-NEXT:    ldrsb lr, [r3], #-1
 ; ENABLE-NEXT:    cmn lr, #1
 ; ENABLE-NEXT:    uxtb r12, lr
-; ENABLE-NEXT:    bgt .LBB0_7
-; ENABLE-NEXT:  @ %bb.17: @ %while.body24.land.rhs14_crit_edge
-; ENABLE-NEXT:    @ in Loop: Header=BB0_15 Depth=2
+; ENABLE-NEXT:    bgt .LBB0_8
+; ENABLE-NEXT:  @ %bb.18: @ %while.body24.land.rhs14_crit_edge
+; ENABLE-NEXT:    @ in Loop: Header=BB0_16 Depth=2
 ; ENABLE-NEXT:    cmp r12, #192
-; ENABLE-NEXT:    blo .LBB0_15
-; ENABLE-NEXT:    b .LBB0_7
-; ENABLE-NEXT:  .LBB0_18:
+; ENABLE-NEXT:    blo .LBB0_16
+; ENABLE-NEXT:    b .LBB0_8
+; ENABLE-NEXT:  .LBB0_19:
 ; ENABLE-NEXT:    mov r0, r3
 ; ENABLE-NEXT:    pop {r11, pc}
 ;
 ; DISABLE-LABEL: wrongUseOfPostDominate:
 ; DISABLE:       @ %bb.0: @ %entry
 ; DISABLE-NEXT:    .save {r11, lr}
 ; DISABLE-NEXT:    push {r11, lr}
 ; DISABLE-NEXT:    cmn r1, #1

llvm/test/CodeGen/Thumb2/mve-memtp-loop.ll

 ; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    mov r12, r1
 ; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    wlstp.8 lr, r2, .LBB10_3
 ; CHECK-NEXT:  .LBB10_2: @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrb.u8 q0, [r12], #16
 ; CHECK-NEXT:    vstrb.8 q0, [r4], #16
 ; CHECK-NEXT:    letp lr, .LBB10_2
-; CHECK-NEXT:  .LBB10_3: @ %for.body
+; CHECK-NEXT:  .LBB10_3: @ %prehead
+; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:  .LBB10_4: @ %for.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldrb r3, [r0], #1
 ; CHECK-NEXT:    subs r2, #2
 ; CHECK-NEXT:    strb r3, [r1], #1
-; CHECK-NEXT:    bne .LBB10_3
-; CHECK-NEXT:  @ %bb.4:
-; CHECK-NEXT:    pop.w {r4, lr}
+; CHECK-NEXT:    bne .LBB10_4
+; CHECK-NEXT:  @ %bb.5: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %cmp6 = icmp slt i32 %n, 0
   br i1 %cmp6, label %prehead, label %for.cond.cleanup
 
 prehead:                                          ; preds = %entry
   call void @llvm.memcpy.p0.p0.i32(ptr align 4 %x, ptr align 4 %y, i32 %n, i1 false)
   br label %for.body

llvm/test/Transforms/InstCombine/ptr-replace-alloca.ll

 ; CHECK-LABEL: @remove_alloca_use_arg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[COND:%.*]], label [[IF:%.*]], label [[ELSE:%.*]]
 ; CHECK:       if:
 ; CHECK-NEXT:    br label [[SINK:%.*]]
 ; CHECK:       else:
 ; CHECK-NEXT:    br label [[SINK]]
 ; CHECK:       sink:
-; CHECK-NEXT:    [[PTR1:%.*]] = phi ptr [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 2), [[IF]] ], [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 1), [[ELSE]] ]
-; CHECK-NEXT:    [[LOAD:%.*]] = load i8, ptr [[PTR1]], align 1
-; CHECK-NEXT:    ret i8 [[LOAD]]
+; CHECK-NEXT:    ret i8 0
 ;
 entry:
   %alloca = alloca [32 x i8], align 4, addrspace(1)
   call void @llvm.memcpy.p1.p0.i64(ptr addrspace(1) %alloca, ptr @g1, i64 32, i1 false)
   br i1 %cond, label %if, label %else
 
 if:
   %val.if = getelementptr inbounds [32 x i8], ptr addrspace(1) %alloca, i32 0, i32 2
sink:
   ret i8 %load
 }
 
 define i8 @loop_phi_remove_alloca(i1 %cond) {
 ; CHECK-LABEL: @loop_phi_remove_alloca(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[BB_0:%.*]]
 ; CHECK:       bb.0:
-; CHECK-NEXT:    [[PTR1:%.*]] = phi ptr [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 1), [[ENTRY:%.*]] ], [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 2), [[BB_1:%.*]] ]
-; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_1]], label [[EXIT:%.*]]
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_1:%.*]], label [[EXIT:%.*]]
 ; CHECK:       bb.1:
 ; CHECK-NEXT:    br label [[BB_0]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i8, ptr [[PTR1]], align 1
-; CHECK-NEXT:    ret i8 [[LOAD]]
+; CHECK-NEXT:    ret i8 0
 ;
 entry:
   %alloca = alloca [32 x i8], align 4, addrspace(1)
   call void @llvm.memcpy.p1.p0.i64(ptr addrspace(1) %alloca, ptr @g1, i64 32, i1 false)
   %val1 = getelementptr inbounds [32 x i8], ptr addrspace(1) %alloca, i32 0, i32 1
   br label %bb.0
 
 bb.0:
join:
   ret i32 %v
 }
 
 define i8 @loop_phi_late_memtransfer_remove_alloca(i1 %cond) {
 ; CHECK-LABEL: @loop_phi_late_memtransfer_remove_alloca(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[BB_0:%.*]]
 ; CHECK:       bb.0:
-; CHECK-NEXT:    [[PTR1:%.*]] = phi ptr [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 1), [[ENTRY:%.*]] ], [ getelementptr inbounds ([32 x i8], ptr @g1, i64 0, i64 2), [[BB_1:%.*]] ]
-; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_1]], label [[EXIT:%.*]]
+; CHECK-NEXT:    br i1 [[COND:%.*]], label [[BB_1:%.*]], label [[EXIT:%.*]]
 ; CHECK:       bb.1:
 ; CHECK-NEXT:    br label [[BB_0]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[LOAD:%.*]] = load i8, ptr [[PTR1]], align 1
-; CHECK-NEXT:    ret i8 [[LOAD]]
+; CHECK-NEXT:    ret i8 0
 ;
 entry:
   %alloca = alloca [32 x i8], align 4, addrspace(1)
   %val1 = getelementptr inbounds [32 x i8], ptr addrspace(1) %alloca, i32 0, i32 1
   br label %bb.0
 
 bb.0:
   %ptr = phi ptr addrspace(1) [ %val1, %entry ], [ %val2, %bb.1 ]

 
 define i32 @addrspace_diff_remove_alloca(i1 %cond) {
 ; CHECK-LABEL: @addrspace_diff_remove_alloca(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[COND:%.*]], label [[IF:%.*]], label [[JOIN:%.*]]
 ; CHECK:       if:
 ; CHECK-NEXT:    br label [[JOIN]]
 ; CHECK:       join:
-; CHECK-NEXT:    [[PHI1:%.*]] = phi ptr addrspace(1) [ @g2, [[IF]] ], [ getelementptr inbounds ([32 x i8], ptr addrspace(1) @g2, i64 0, i64 2), [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[V:%.*]] = load i32, ptr addrspace(1) [[PHI1]], align 4
-; CHECK-NEXT:    ret i32 [[V]]
+; CHECK-NEXT:    ret i32 0
 ;
 entry:
   %a = alloca [32 x i8]
   call void @llvm.memcpy.p0.p1.i64(ptr %a, ptr addrspace(1) @g2, i64 32, i1 false)
   %gep = getelementptr inbounds [32 x i8], ptr %a, i32 0, i32 2
   br i1 %cond, label %if, label %join
 
 if:
   br label %join
 
 join:
   %phi = phi ptr [ %a, %if ], [ %gep, %entry ]
   %v = load i32, ptr %phi
   ret i32 %v
 }
 
 define i32 @phi_loop(i1 %c) {
 ; CHECK-LABEL: @phi_loop(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[PTR:%.*]] = phi ptr [ @g1, [[ENTRY:%.*]] ], [ [[PTR_NEXT:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[PTR_NEXT]] = getelementptr i8, ptr [[PTR]], i64 4
 ; CHECK-NEXT:    br i1 [[C:%.*]], label [[EXIT:%.*]], label [[LOOP]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[V:%.*]] = load i32, ptr [[PTR]], align 4
-; CHECK-NEXT:    ret i32 [[V]]
+; CHECK-NEXT:    ret i32 0
 ;
 entry:
   %alloca = alloca [32 x i8]
   call void @llvm.memcpy.p0.p0.i64(ptr %alloca, ptr @g1, i64 32, i1 false)
   br label %loop
 
 loop:
   %ptr = phi ptr [ %alloca, %entry ], [ %ptr.next, %loop ]

llvm/test/Transforms/LoopVectorize/ARM/pointer_iv.ll

 ; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[A]], i32 [[TMP0]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = shl i32 [[INDEX]], 2
 ; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i8, ptr [[B]], i32 [[TMP1]]
 ; CHECK-NEXT:    [[WIDE_VEC:%.*]] = load <8 x i32>, ptr [[NEXT_GEP]], align 4
 ; CHECK-NEXT:    [[STRIDED_VEC:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
 ; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> [[STRIDED_VEC]], [[BROADCAST_SPLAT]]
 ; CHECK-NEXT:    store <4 x i32> [[TMP2]], ptr [[NEXT_GEP4]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
-; CHECK-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
+; CHECK-NEXT:    br i1 [[TMP3]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[A_ADDR_09:%.*]] = phi ptr [ [[ADD_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[I_08:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 996, [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[B_ADDR_07:%.*]] = phi ptr [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END2]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[A_ADDR_09]], align 4
 ; CHECK-NEXT:    [[ADD_PTR]] = getelementptr inbounds i32, ptr [[A_ADDR_09]], i32 2
 ; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP4]], [[Y]]
 ; CHECK-NEXT:    store i32 [[ADD]], ptr [[B_ADDR_07]], align 4
for.body:
   %inc = add nuw nsw i32 %i.07, 1
   %exitcond = icmp eq i32 %inc, 10000
   br i1 %exitcond, label %for.cond.cleanup, label %for.body, !llvm.loop !2
 }
 
 define hidden void @mult_ptr_iv(ptr noalias nocapture readonly %x, ptr noalias nocapture %z) {
 ; CHECK-LABEL: @mult_ptr_iv(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[Z:%.*]], i32 3000
-; CHECK-NEXT:    [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[X:%.*]], i32 3000
-; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ugt ptr [[UGLYGEP1]], [[Z]]
-; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ugt ptr [[UGLYGEP]], [[X]]
-; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[FOR_BODY:%.*]], label [[VECTOR_PH:%.*]]
-; CHECK:       vector.ph:
-; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[X]], i32 3000
-; CHECK-NEXT:    [[IND_END2:%.*]] = getelementptr i8, ptr [[Z]], i32 3000
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi ptr [ [[X]], [[VECTOR_PH]] ], [ [[PTR_IND:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[POINTER_PHI5:%.*]] = phi ptr [ [[Z]], [[VECTOR_PH]] ], [ [[PTR_IND6:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi ptr [ [[X:%.*]], [[ENTRY:%.*]] ], [ [[PTR_IND:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[POINTER_PHI4:%.*]] = phi ptr [ [[Z:%.*]], [[ENTRY]] ], [ [[PTR_IND5:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr i8, ptr [[POINTER_PHI]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
-; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i8, ptr [[POINTER_PHI5]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i8, ptr [[POINTER_PHI4]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP0]], i32 1
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP0]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison), !alias.scope !28
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP0]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison)
 ; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP0]], i32 2
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER7:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP2]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison), !alias.scope !28
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER8:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP3]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison), !alias.scope !28
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER6:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP2]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison)
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER7:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0(<4 x ptr> [[TMP3]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> poison)
 ; CHECK-NEXT:    [[TMP4:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], <i8 10, i8 10, i8 10, i8 10>
-; CHECK-NEXT:    [[TMP5:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER7]]
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER8]]
+; CHECK-NEXT:    [[TMP5:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER6]]
+; CHECK-NEXT:    [[TMP6:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER7]]
 ; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP1]], i32 1
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP4]], <4 x ptr> [[TMP1]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !31, !noalias !28
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP4]], <4 x ptr> [[TMP1]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>)
 ; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP1]], i32 2
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP5]], <4 x ptr> [[TMP7]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !31, !noalias !28
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP6]], <4 x ptr> [[TMP8]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !31, !noalias !28
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP5]], <4 x ptr> [[TMP7]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>)
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0(<4 x i8> [[TMP6]], <4 x ptr> [[TMP8]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>)
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
 ; CHECK-NEXT:    [[PTR_IND]] = getelementptr i8, ptr [[POINTER_PHI]], i32 12
-; CHECK-NEXT:    [[PTR_IND6]] = getelementptr i8, ptr [[POINTER_PHI5]], i32 12
+; CHECK-NEXT:    [[PTR_IND5]] = getelementptr i8, ptr [[POINTER_PHI4]], i32 12
 ; CHECK-NEXT:    [[TMP9:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1000
-; CHECK-NEXT:    br i1 [[TMP9]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP33:![0-9]+]]
-; CHECK:       for.body:
-; CHECK-NEXT:    [[X_ADDR_050:%.*]] = phi ptr [ [[INCDEC_PTR2:%.*]], [[FOR_BODY]] ], [ [[X]], [[ENTRY:%.*]] ]
-; CHECK-NEXT:    [[Z_ADDR_049:%.*]] = phi ptr [ [[INCDEC_PTR34:%.*]], [[FOR_BODY]] ], [ [[Z]], [[ENTRY]] ]
-; CHECK-NEXT:    [[I_048:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 0, [[ENTRY]] ]
-; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR_050]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = load i8, ptr [[X_ADDR_050]], align 1
-; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i8, ptr [[X_ADDR_050]], i32 2
-; CHECK-NEXT:    [[TMP11:%.*]] = load i8, ptr [[INCDEC_PTR]], align 1
-; CHECK-NEXT:    [[INCDEC_PTR2]] = getelementptr inbounds i8, ptr [[X_ADDR_050]], i32 3
-; CHECK-NEXT:    [[TMP12:%.*]] = load i8, ptr [[INCDEC_PTR1]], align 1
-; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[TMP10]], 10
-; CHECK-NEXT:    [[MUL1:%.*]] = mul i8 [[TMP10]], [[TMP11]]
-; CHECK-NEXT:    [[MUL2:%.*]] = mul i8 [[TMP10]], [[TMP12]]
-; CHECK-NEXT:    [[INCDEC_PTR32:%.*]] = getelementptr inbounds i8, ptr [[Z_ADDR_049]], i32 1
-; CHECK-NEXT:    store i8 [[MUL]], ptr [[Z_ADDR_049]], align 1
-; CHECK-NEXT:    [[INCDEC_PTR33:%.*]] = getelementptr inbounds i8, ptr [[Z_ADDR_049]], i32 2
-; CHECK-NEXT:    store i8 [[MUL1]], ptr [[INCDEC_PTR32]], align 1
-; CHECK-NEXT:    [[INCDEC_PTR34]] = getelementptr inbounds i8, ptr [[Z_ADDR_049]], i32 3
-; CHECK-NEXT:    store i8 [[MUL2]], ptr [[INCDEC_PTR33]], align 1
-; CHECK-NEXT:    [[INC]] = add nuw i32 [[I_048]], 1
-; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 1000
-; CHECK-NEXT:    br i1 [[EXITCOND]], label [[END]], label [[FOR_BODY]], !llvm.loop [[LOOP34:![0-9]+]]
+; CHECK-NEXT:    br i1 [[TMP9]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP28:![0-9]+]]
 ; CHECK:       end:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br label %for.body
 
 for.body:
   %x.addr.050 = phi ptr [ %incdec.ptr2, %for.body ], [ %x, %entry ]

llvm/test/Transforms/LoopVectorize/X86/gather_scatter.ll

 ; AVX512:       iter.check:
 ; AVX512-NEXT:    [[MUL:%.*]] = sub nsw i32 0, [[D]]
 ; AVX512-NEXT:    [[IDXPROM:%.*]] = sext i32 [[MUL]] to i64
 ; AVX512-NEXT:    [[TMP0:%.*]] = shl nsw i64 [[IDX_EXT]], 2
 ; AVX512-NEXT:    [[TMP1:%.*]] = add nsw i64 [[TMP0]], -4
 ; AVX512-NEXT:    [[TMP2:%.*]] = lshr i64 [[TMP1]], 2
 ; AVX512-NEXT:    [[TMP3:%.*]] = add nuw nsw i64 [[TMP2]], 1
 ; AVX512-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 8
-; AVX512-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[VEC_EPILOG_SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
-; AVX512:       vector.memcheck:
-; AVX512-NEXT:    [[TMP4:%.*]] = shl nsw i64 [[IDX_EXT]], 2
-; AVX512-NEXT:    [[TMP5:%.*]] = add nsw i64 [[TMP4]], -4
-; AVX512-NEXT:    [[TMP6:%.*]] = lshr i64 [[TMP5]], 2
-; AVX512-NEXT:    [[TMP7:%.*]] = shl i64 [[TMP6]], 6
-; AVX512-NEXT:    [[TMP8:%.*]] = add nuw nsw i64 [[TMP7]], 8
-; AVX512-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[DEST:%.*]], i64 [[TMP8]]
-; AVX512-NEXT:    [[TMP9:%.*]] = shl nuw i64 [[TMP6]], 2
-; AVX512-NEXT:    [[TMP10:%.*]] = add i64 [[TMP9]], 4
-; AVX512-NEXT:    [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP10]]
-; AVX512-NEXT:    [[TMP11:%.*]] = mul nsw i64 [[IDX_EXT]], -4
-; AVX512-NEXT:    [[UGLYGEP2:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP11]]
-; AVX512-NEXT:    [[TMP12:%.*]] = sub i64 [[TMP10]], [[TMP4]]
-; AVX512-NEXT:    [[UGLYGEP3:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP12]]
-; AVX512-NEXT:    [[BOUND0:%.*]] = icmp ult ptr [[DEST]], [[UGLYGEP1]]
-; AVX512-NEXT:    [[BOUND1:%.*]] = icmp ult ptr [[PTR]], [[UGLYGEP]]
-; AVX512-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; AVX512-NEXT:    [[BOUND04:%.*]] = icmp ult ptr [[DEST]], [[UGLYGEP3]]
-; AVX512-NEXT:    [[BOUND15:%.*]] = icmp ult ptr [[UGLYGEP2]], [[UGLYGEP]]
-; AVX512-NEXT:    [[FOUND_CONFLICT6:%.*]] = and i1 [[BOUND04]], [[BOUND15]]
-; AVX512-NEXT:    [[CONFLICT_RDX:%.*]] = or i1 [[FOUND_CONFLICT]], [[FOUND_CONFLICT6]]
-; AVX512-NEXT:    br i1 [[CONFLICT_RDX]], label [[VEC_EPILOG_SCALAR_PH]], label [[VECTOR_MAIN_LOOP_ITER_CHECK:%.*]]
+; AVX512-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[VEC_EPILOG_SCALAR_PH:%.*]], label [[VECTOR_MAIN_LOOP_ITER_CHECK:%.*]]
 ; AVX512:       vector.main.loop.iter.check:
-; AVX512-NEXT:    [[MIN_ITERS_CHECK7:%.*]] = icmp ult i64 [[TMP3]], 16
-; AVX512-NEXT:    br i1 [[MIN_ITERS_CHECK7]], label [[VEC_EPILOG_PH:%.*]], label [[VECTOR_PH:%.*]]
+; AVX512-NEXT:    [[MIN_ITERS_CHECK1:%.*]] = icmp ult i64 [[TMP3]], 16
+; AVX512-NEXT:    br i1 [[MIN_ITERS_CHECK1]], label [[VEC_EPILOG_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; AVX512:       vector.ph:
 ; AVX512-NEXT:    [[N_MOD_VF:%.*]] = urem i64 [[TMP3]], 16
 ; AVX512-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[N_MOD_VF]]
-; AVX512-NEXT:    [[TMP13:%.*]] = mul i64 [[N_VEC]], 4
-; AVX512-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP13]]
-; AVX512-NEXT:    [[TMP14:%.*]] = mul i64 [[N_VEC]], 64
-; AVX512-NEXT:    [[IND_END9:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP14]]
+; AVX512-NEXT:    [[TMP4:%.*]] = mul i64 [[N_VEC]], 4
+; AVX512-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP4]]
+; AVX512-NEXT:    [[TMP5:%.*]] = mul i64 [[N_VEC]], 64
+; AVX512-NEXT:    [[IND_END3:%.*]] = getelementptr i8, ptr [[DEST:%.*]], i64 [[TMP5]]
 ; AVX512-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; AVX512:       vector.body:
 ; AVX512-NEXT:    [[POINTER_PHI:%.*]] = phi ptr [ [[DEST]], [[VECTOR_PH]] ], [ [[PTR_IND:%.*]], [[VECTOR_BODY]] ]
 ; AVX512-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; AVX512-NEXT:    [[TMP15:%.*]] = add i64 [[INDEX]], 0
-; AVX512-NEXT:    [[TMP16:%.*]] = mul i64 [[TMP15]], 4
-; AVX512-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP16]]
-; AVX512-NEXT:    [[TMP17:%.*]] = getelementptr i8, ptr [[POINTER_PHI]], <16 x i64> <i64 0, i64 64, i64 128, i64 192, i64 256, i64 320, i64 384, i64 448, i64 512, i64 576, i64 640, i64 704, i64 768, i64 832, i64 896, i64 960>
-; AVX512-NEXT:    [[TMP18:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP]], i64 [[IDXPROM]]
-; AVX512-NEXT:    [[TMP19:%.*]] = getelementptr inbounds float, ptr [[TMP18]], i32 0
-; AVX512-NEXT:    [[WIDE_LOAD:%.*]] = load <16 x float>, ptr [[TMP19]], align 4, !alias.scope !14
-; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> [[WIDE_LOAD]], <16 x ptr> [[TMP17]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !17, !noalias !19
-; AVX512-NEXT:    [[TMP20:%.*]] = getelementptr float, ptr [[NEXT_GEP]], i32 0
-; AVX512-NEXT:    [[WIDE_LOAD8:%.*]] = load <16 x float>, ptr [[TMP20]], align 4, !alias.scope !21
-; AVX512-NEXT:    [[TMP21:%.*]] = getelementptr inbounds float, <16 x ptr> [[TMP17]], i64 1
-; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> [[WIDE_LOAD8]], <16 x ptr> [[TMP21]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !17, !noalias !19
+; AVX512-NEXT:    [[TMP6:%.*]] = add i64 [[INDEX]], 0
+; AVX512-NEXT:    [[TMP7:%.*]] = mul i64 [[TMP6]], 4
+; AVX512-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP7]]
+; AVX512-NEXT:    [[TMP8:%.*]] = getelementptr i8, ptr [[POINTER_PHI]], <16 x i64> <i64 0, i64 64, i64 128, i64 192, i64 256, i64 320, i64 384, i64 448, i64 512, i64 576, i64 640, i64 704, i64 768, i64 832, i64 896, i64 960>
+; AVX512-NEXT:    [[TMP9:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP]], i64 [[IDXPROM]]
+; AVX512-NEXT:    [[TMP10:%.*]] = getelementptr inbounds float, ptr [[TMP9]], i32 0
+; AVX512-NEXT:    [[WIDE_LOAD:%.*]] = load <16 x float>, ptr [[TMP10]], align 4
+; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> [[WIDE_LOAD]], <16 x ptr> [[TMP8]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
+; AVX512-NEXT:    [[TMP11:%.*]] = getelementptr float, ptr [[NEXT_GEP]], i32 0
+; AVX512-NEXT:    [[WIDE_LOAD2:%.*]] = load <16 x float>, ptr [[TMP11]], align 4
+; AVX512-NEXT:    [[TMP12:%.*]] = getelementptr inbounds float, <16 x ptr> [[TMP8]], i64 1
+; AVX512-NEXT:    call void @llvm.masked.scatter.v16f32.v16p0(<16 x float> [[WIDE_LOAD2]], <16 x ptr> [[TMP12]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
 ; AVX512-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 16
 ; AVX512-NEXT:    [[PTR_IND]] = getelementptr i8, ptr [[POINTER_PHI]], i64 1024
-; AVX512-NEXT:    [[TMP22:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; AVX512-NEXT:    br i1 [[TMP22]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP22:![0-9]+]]
+; AVX512-NEXT:    [[TMP13:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; AVX512-NEXT:    br i1 [[TMP13]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP14:![0-9]+]]
 ; AVX512:       middle.block:
 ; AVX512-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]]
 ; AVX512-NEXT:    br i1 [[CMP_N]], label [[FOR_END]], label [[VEC_EPILOG_ITER_CHECK:%.*]]
 ; AVX512:       vec.epilog.iter.check:
-; AVX512-NEXT:    [[TMP23:%.*]] = mul i64 [[N_VEC]], 64
-; AVX512-NEXT:    [[IND_END17:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP23]]
-; AVX512-NEXT:    [[TMP24:%.*]] = mul i64 [[N_VEC]], 4
-; AVX512-NEXT:    [[IND_END14:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP24]]
+; AVX512-NEXT:    [[TMP14:%.*]] = mul i64 [[N_VEC]], 64
+; AVX512-NEXT:    [[IND_END11:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP14]]
+; AVX512-NEXT:    [[TMP15:%.*]] = mul i64 [[N_VEC]], 4
+; AVX512-NEXT:    [[IND_END8:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP15]]
 ; AVX512-NEXT:    [[N_VEC_REMAINING:%.*]] = sub i64 [[TMP3]], [[N_VEC]]
 ; AVX512-NEXT:    [[MIN_EPILOG_ITERS_CHECK:%.*]] = icmp ult i64 [[N_VEC_REMAINING]], 8
 ; AVX512-NEXT:    br i1 [[MIN_EPILOG_ITERS_CHECK]], label [[VEC_EPILOG_SCALAR_PH]], label [[VEC_EPILOG_PH]]
 ; AVX512:       vec.epilog.ph:
 ; AVX512-NEXT:    [[BC_RESUME_VAL:%.*]] = phi ptr [ [[IND_END]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[PTR]], [[VECTOR_MAIN_LOOP_ITER_CHECK]] ]
-; AVX512-NEXT:    [[BC_RESUME_VAL10:%.*]] = phi ptr [ [[IND_END9]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[DEST]], [[VECTOR_MAIN_LOOP_ITER_CHECK]] ]
+; AVX512-NEXT:    [[BC_RESUME_VAL4:%.*]] = phi ptr [ [[IND_END3]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[DEST]], [[VECTOR_MAIN_LOOP_ITER_CHECK]] ]
 ; AVX512-NEXT:    [[VEC_EPILOG_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[VEC_EPILOG_ITER_CHECK]] ], [ 0, [[VECTOR_MAIN_LOOP_ITER_CHECK]] ]
-; AVX512-NEXT:    [[N_MOD_VF11:%.*]] = urem i64 [[TMP3]], 8
-; AVX512-NEXT:    [[N_VEC12:%.*]] = sub i64 [[TMP3]], [[N_MOD_VF11]]
-; AVX512-NEXT:    [[TMP25:%.*]] = mul i64 [[N_VEC12]], 4
-; AVX512-NEXT:    [[IND_END13:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP25]]
-; AVX512-NEXT:    [[TMP26:%.*]] = mul i64 [[N_VEC12]], 64
-; AVX512-NEXT:    [[IND_END16:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP26]]
+; AVX512-NEXT:    [[N_MOD_VF5:%.*]] = urem i64 [[TMP3]], 8
+; AVX512-NEXT:    [[N_VEC6:%.*]] = sub i64 [[TMP3]], [[N_MOD_VF5]]
+; AVX512-NEXT:    [[TMP16:%.*]] = mul i64 [[N_VEC6]], 4
+; AVX512-NEXT:    [[IND_END7:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP16]]
+; AVX512-NEXT:    [[TMP17:%.*]] = mul i64 [[N_VEC6]], 64
+; AVX512-NEXT:    [[IND_END10:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP17]]
 ; AVX512-NEXT:    br label [[VEC_EPILOG_VECTOR_BODY:%.*]]
 ; AVX512:       vec.epilog.vector.body:
-; AVX512-NEXT:    [[POINTER_PHI22:%.*]] = phi ptr [ [[BC_RESUME_VAL10]], [[VEC_EPILOG_PH]] ], [ [[PTR_IND23:%.*]], [[VEC_EPILOG_VECTOR_BODY]] ]
-; AVX512-NEXT:    [[INDEX20:%.*]] = phi i64 [ [[VEC_EPILOG_RESUME_VAL]], [[VEC_EPILOG_PH]] ], [ [[INDEX_NEXT26:%.*]], [[VEC_EPILOG_VECTOR_BODY]] ]
-; AVX512-NEXT:    [[TMP27:%.*]] = add i64 [[INDEX20]], 0
-; AVX512-NEXT:    [[TMP28:%.*]] = mul i64 [[TMP27]], 4
-; AVX512-NEXT:    [[NEXT_GEP21:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP28]]
-; AVX512-NEXT:    [[TMP29:%.*]] = getelementptr i8, ptr [[POINTER_PHI22]], <8 x i64> <i64 0, i64 64, i64 128, i64 192, i64 256, i64 320, i64 384, i64 448>
-; AVX512-NEXT:    [[TMP30:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP21]], i64 [[IDXPROM]]
-; AVX512-NEXT:    [[TMP31:%.*]] = getelementptr inbounds float, ptr [[TMP30]], i32 0
-; AVX512-NEXT:    [[WIDE_LOAD24:%.*]] = load <8 x float>, ptr [[TMP31]], align 4, !alias.scope !23
-; AVX512-NEXT:    call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> [[WIDE_LOAD24]], <8 x ptr> [[TMP29]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !26, !noalias !28
-; AVX512-NEXT:    [[TMP32:%.*]] = getelementptr float, ptr [[NEXT_GEP21]], i32 0
-; AVX512-NEXT:    [[WIDE_LOAD25:%.*]] = load <8 x float>, ptr [[TMP32]], align 4, !alias.scope !30
-; AVX512-NEXT:    [[TMP33:%.*]] = getelementptr inbounds float, <8 x ptr> [[TMP29]], i64 1
-; AVX512-NEXT:    call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> [[WIDE_LOAD25]], <8 x ptr> [[TMP33]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !26, !noalias !28
-; AVX512-NEXT:    [[INDEX_NEXT26]] = add nuw i64 [[INDEX20]], 8
-; AVX512-NEXT:    [[PTR_IND23]] = getelementptr i8, ptr [[POINTER_PHI22]], i64 512
-; AVX512-NEXT:    [[TMP34:%.*]] = icmp eq i64 [[INDEX_NEXT26]], [[N_VEC12]]
-; AVX512-NEXT:    br i1 [[TMP34]], label [[VEC_EPILOG_MIDDLE_BLOCK:%.*]], label [[VEC_EPILOG_VECTOR_BODY]], !llvm.loop [[LOOP31:![0-9]+]]
+; AVX512-NEXT:    [[POINTER_PHI16:%.*]] = phi ptr [ [[BC_RESUME_VAL4]], [[VEC_EPILOG_PH]] ], [ [[PTR_IND17:%.*]], [[VEC_EPILOG_VECTOR_BODY]] ]
+; AVX512-NEXT:    [[INDEX14:%.*]] = phi i64 [ [[VEC_EPILOG_RESUME_VAL]], [[VEC_EPILOG_PH]] ], [ [[INDEX_NEXT20:%.*]], [[VEC_EPILOG_VECTOR_BODY]] ]
+; AVX512-NEXT:    [[TMP18:%.*]] = add i64 [[INDEX14]], 0
+; AVX512-NEXT:    [[TMP19:%.*]] = mul i64 [[TMP18]], 4
+; AVX512-NEXT:    [[NEXT_GEP15:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP19]]
+; AVX512-NEXT:    [[TMP20:%.*]] = getelementptr i8, ptr [[POINTER_PHI16]], <8 x i64> <i64 0, i64 64, i64 128, i64 192, i64 256, i64 320, i64 384, i64 448>
+; AVX512-NEXT:    [[TMP21:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP15]], i64 [[IDXPROM]]
+; AVX512-NEXT:    [[TMP22:%.*]] = getelementptr inbounds float, ptr [[TMP21]], i32 0
+; AVX512-NEXT:    [[WIDE_LOAD18:%.*]] = load <8 x float>, ptr [[TMP22]], align 4
+; AVX512-NEXT:    call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> [[WIDE_LOAD18]], <8 x ptr> [[TMP20]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
+; AVX512-NEXT:    [[TMP23:%.*]] = getelementptr float, ptr [[NEXT_GEP15]], i32 0
+; AVX512-NEXT:    [[WIDE_LOAD19:%.*]] = load <8 x float>, ptr [[TMP23]], align 4
+; AVX512-NEXT:    [[TMP24:%.*]] = getelementptr inbounds float, <8 x ptr> [[TMP20]], i64 1
+; AVX512-NEXT:    call void @llvm.masked.scatter.v8f32.v8p0(<8 x float> [[WIDE_LOAD19]], <8 x ptr> [[TMP24]], i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
+; AVX512-NEXT:    [[INDEX_NEXT20]] = add nuw i64 [[INDEX14]], 8
+; AVX512-NEXT:    [[PTR_IND17]] = getelementptr i8, ptr [[POINTER_PHI16]], i64 512
+; AVX512-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT20]], [[N_VEC6]]
+; AVX512-NEXT:    br i1 [[TMP25]], label [[VEC_EPILOG_MIDDLE_BLOCK:%.*]], label [[VEC_EPILOG_VECTOR_BODY]], !llvm.loop [[LOOP15:![0-9]+]]
 ; AVX512:       vec.epilog.middle.block:
-; AVX512-NEXT:    [[CMP_N19:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC12]]
-; AVX512-NEXT:    br i1 [[CMP_N19]], label [[FOR_END]], label [[VEC_EPILOG_SCALAR_PH]]
+; AVX512-NEXT:    [[CMP_N13:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC6]]
+; AVX512-NEXT:    br i1 [[CMP_N13]], label [[FOR_END]], label [[VEC_EPILOG_SCALAR_PH]]
 ; AVX512:       vec.epilog.scalar.ph:
-; AVX512-NEXT:    [[BC_RESUME_VAL15:%.*]] = phi ptr [ [[IND_END13]], [[VEC_EPILOG_MIDDLE_BLOCK]] ], [ [[IND_END14]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[PTR]], [[VECTOR_MEMCHECK]] ], [ [[PTR]], [[ITER_CHECK]] ]
-; AVX512-NEXT:    [[BC_RESUME_VAL18:%.*]] = phi ptr [ [[IND_END16]], [[VEC_EPILOG_MIDDLE_BLOCK]] ], [ [[IND_END17]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[DEST]], [[VECTOR_MEMCHECK]] ], [ [[DEST]], [[ITER_CHECK]] ]
+; AVX512-NEXT:    [[BC_RESUME_VAL9:%.*]] = phi ptr [ [[IND_END7]], [[VEC_EPILOG_MIDDLE_BLOCK]] ], [ [[IND_END8]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[PTR]], [[ITER_CHECK]] ]
+; AVX512-NEXT:    [[BC_RESUME_VAL12:%.*]] = phi ptr [ [[IND_END10]], [[VEC_EPILOG_MIDDLE_BLOCK]] ], [ [[IND_END11]], [[VEC_EPILOG_ITER_CHECK]] ], [ [[DEST]], [[ITER_CHECK]] ]
 ; AVX512-NEXT:    br label [[FOR_BODY:%.*]]
 ; AVX512:       for.body:
-; AVX512-NEXT:    [[PTR_ADDR_012:%.*]] = phi ptr [ [[BC_RESUME_VAL15]], [[VEC_EPILOG_SCALAR_PH]] ], [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ]
-; AVX512-NEXT:    [[DEST_ADDR_011:%.*]] = phi ptr [ [[BC_RESUME_VAL18]], [[VEC_EPILOG_SCALAR_PH]] ], [ [[ADD_PTR6:%.*]], [[FOR_BODY]] ]
+; AVX512-NEXT:    [[PTR_ADDR_012:%.*]] = phi ptr [ [[BC_RESUME_VAL9]], [[VEC_EPILOG_SCALAR_PH]] ], [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ]
+; AVX512-NEXT:    [[DEST_ADDR_011:%.*]] = phi ptr [ [[BC_RESUME_VAL12]], [[VEC_EPILOG_SCALAR_PH]] ], [ [[ADD_PTR6:%.*]], [[FOR_BODY]] ]
 ; AVX512-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, ptr [[PTR_ADDR_012]], i64 [[IDXPROM]]
-; AVX512-NEXT:    [[TMP35:%.*]] = load float, ptr [[ARRAYIDX]], align 4
-; AVX512-NEXT:    store float [[TMP35]], ptr [[DEST_ADDR_011]], align 4
-; AVX512-NEXT:    [[TMP36:%.*]] = load float, ptr [[PTR_ADDR_012]], align 4
+; AVX512-NEXT:    [[TMP26:%.*]] = load float, ptr [[ARRAYIDX]], align 4
+; AVX512-NEXT:    store float [[TMP26]], ptr [[DEST_ADDR_011]], align 4
+; AVX512-NEXT:    [[TMP27:%.*]] = load float, ptr [[PTR_ADDR_012]], align 4
 ; AVX512-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, ptr [[DEST_ADDR_011]], i64 1
-; AVX512-NEXT:    store float [[TMP36]], ptr [[ARRAYIDX5]], align 4
+; AVX512-NEXT:    store float [[TMP27]], ptr [[ARRAYIDX5]], align 4
 ; AVX512-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds float, ptr [[PTR_ADDR_012]], i64 1
 ; AVX512-NEXT:    [[ADD_PTR6]] = getelementptr inbounds float, ptr [[DEST_ADDR_011]], i64 16
 ; AVX512-NEXT:    [[CMP_NOT:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[ADD_PTR]]
-; AVX512-NEXT:    br i1 [[CMP_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP32:![0-9]+]]
+; AVX512-NEXT:    br i1 [[CMP_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP16:![0-9]+]]
 ; AVX512:       for.end:
 ; AVX512-NEXT:    ret void
 ;
 ; FVW2-LABEL: @test_gather_not_profitable_pr48429(
 ; FVW2-NEXT:  entry:
 ; FVW2-NEXT:    [[IDX_EXT:%.*]] = sext i32 [[D:%.*]] to i64
 ; FVW2-NEXT:    [[ADD_PTR:%.*]] = getelementptr inbounds float, ptr [[PTR:%.*]], i64 [[IDX_EXT]]
 ; FVW2-NEXT:    [[CMP_NOT10:%.*]] = icmp eq i32 [[D]], 0
 ; FVW2-NEXT:    br i1 [[CMP_NOT10]], label [[FOR_END:%.*]], label [[FOR_BODY_LR_PH:%.*]]
 ; FVW2:       for.body.lr.ph:
 ; FVW2-NEXT:    [[MUL:%.*]] = sub nsw i32 0, [[D]]
 ; FVW2-NEXT:    [[IDXPROM:%.*]] = sext i32 [[MUL]] to i64
 ; FVW2-NEXT:    [[TMP0:%.*]] = shl nsw i64 [[IDX_EXT]], 2
 ; FVW2-NEXT:    [[TMP1:%.*]] = add nsw i64 [[TMP0]], -4
 ; FVW2-NEXT:    [[TMP2:%.*]] = lshr i64 [[TMP1]], 2
 ; FVW2-NEXT:    [[TMP3:%.*]] = add nuw nsw i64 [[TMP2]], 1
 ; FVW2-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 2
-; FVW2-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
-; FVW2:       vector.memcheck:
-; FVW2-NEXT:    [[TMP4:%.*]] = shl nsw i64 [[IDX_EXT]], 2
-; FVW2-NEXT:    [[TMP5:%.*]] = add nsw i64 [[TMP4]], -4
-; FVW2-NEXT:    [[TMP6:%.*]] = lshr i64 [[TMP5]], 2
-; FVW2-NEXT:    [[TMP7:%.*]] = shl i64 [[TMP6]], 6
-; FVW2-NEXT:    [[TMP8:%.*]] = add nuw nsw i64 [[TMP7]], 8
-; FVW2-NEXT:    [[UGLYGEP:%.*]] = getelementptr i8, ptr [[DEST:%.*]], i64 [[TMP8]]
-; FVW2-NEXT:    [[TMP9:%.*]] = shl nuw i64 [[TMP6]], 2
-; FVW2-NEXT:    [[TMP10:%.*]] = add i64 [[TMP9]], 4
-; FVW2-NEXT:    [[UGLYGEP1:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP10]]
-; FVW2-NEXT:    [[TMP11:%.*]] = mul nsw i64 [[IDX_EXT]], -4
-; FVW2-NEXT:    [[UGLYGEP2:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP11]]
-; FVW2-NEXT:    [[TMP12:%.*]] = sub i64 [[TMP10]], [[TMP4]]
-; FVW2-NEXT:    [[UGLYGEP3:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP12]]
-; FVW2-NEXT:    [[BOUND0:%.*]] = icmp ult ptr [[DEST]], [[UGLYGEP1]]
-; FVW2-NEXT:    [[BOUND1:%.*]] = icmp ult ptr [[PTR]], [[UGLYGEP]]
-; FVW2-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; FVW2-NEXT:    [[BOUND04:%.*]] = icmp ult ptr [[DEST]], [[UGLYGEP3]]
-; FVW2-NEXT:    [[BOUND15:%.*]] = icmp ult ptr [[UGLYGEP2]], [[UGLYGEP]]
-; FVW2-NEXT:    [[FOUND_CONFLICT6:%.*]] = and i1 [[BOUND04]], [[BOUND15]]
-; FVW2-NEXT:    [[CONFLICT_RDX:%.*]] = or i1 [[FOUND_CONFLICT]], [[FOUND_CONFLICT6]]
-; FVW2-NEXT:    br i1 [[CONFLICT_RDX]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
+; FVW2-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; FVW2:       vector.ph:
 ; FVW2-NEXT:    [[N_MOD_VF:%.*]] = urem i64 [[TMP3]], 2
 ; FVW2-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[N_MOD_VF]]
-; FVW2-NEXT:    [[TMP13:%.*]] = mul i64 [[N_VEC]], 4
-; FVW2-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP13]]
-; FVW2-NEXT:    [[TMP14:%.*]] = mul i64 [[N_VEC]], 64
-; FVW2-NEXT:    [[IND_END7:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP14]]
+; FVW2-NEXT:    [[TMP4:%.*]] = mul i64 [[N_VEC]], 4
+; FVW2-NEXT:    [[IND_END:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP4]]
+; FVW2-NEXT:    [[TMP5:%.*]] = mul i64 [[N_VEC]], 64
+; FVW2-NEXT:    [[IND_END1:%.*]] = getelementptr i8, ptr [[DEST:%.*]], i64 [[TMP5]]
 ; FVW2-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; FVW2:       vector.body:
 ; FVW2-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; FVW2-NEXT:    [[TMP15:%.*]] = add i64 [[INDEX]], 0
-; FVW2-NEXT:    [[TMP16:%.*]] = mul i64 [[TMP15]], 4
-; FVW2-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP16]]
-; FVW2-NEXT:    [[TMP17:%.*]] = add i64 [[INDEX]], 0
-; FVW2-NEXT:    [[TMP18:%.*]] = mul i64 [[TMP17]], 64
-; FVW2-NEXT:    [[NEXT_GEP9:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP18]]
-; FVW2-NEXT:    [[TMP19:%.*]] = add i64 [[INDEX]], 1
-; FVW2-NEXT:    [[TMP20:%.*]] = mul i64 [[TMP19]], 64
-; FVW2-NEXT:    [[NEXT_GEP10:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP20]]
-; FVW2-NEXT:    [[TMP21:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP]], i64 [[IDXPROM]]
-; FVW2-NEXT:    [[TMP22:%.*]] = getelementptr inbounds float, ptr [[TMP21]], i32 0
-; FVW2-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x float>, ptr [[TMP22]], align 4, !alias.scope !14
-; FVW2-NEXT:    [[TMP23:%.*]] = extractelement <2 x float> [[WIDE_LOAD]], i32 0
-; FVW2-NEXT:    store float [[TMP23]], ptr [[NEXT_GEP9]], align 4, !alias.scope !17, !noalias !19
-; FVW2-NEXT:    [[TMP24:%.*]] = extractelement <2 x float> [[WIDE_LOAD]], i32 1
-; FVW2-NEXT:    store float [[TMP24]], ptr [[NEXT_GEP10]], align 4, !alias.scope !17, !noalias !19
-; FVW2-NEXT:    [[TMP25:%.*]] = getelementptr float, ptr [[NEXT_GEP]], i32 0
-; FVW2-NEXT:    [[WIDE_LOAD11:%.*]] = load <2 x float>, ptr [[TMP25]], align 4, !alias.scope !21
-; FVW2-NEXT:    [[TMP26:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP9]], i64 1
-; FVW2-NEXT:    [[TMP27:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP10]], i64 1
-; FVW2-NEXT:    [[TMP28:%.*]] = extractelement <2 x float> [[WIDE_LOAD11]], i32 0
-; FVW2-NEXT:    store float [[TMP28]], ptr [[TMP26]], align 4, !alias.scope !17, !noalias !19
-; FVW2-NEXT:    [[TMP29:%.*]] = extractelement <2 x float> [[WIDE_LOAD11]], i32 1
-; FVW2-NEXT:    store float [[TMP29]], ptr [[TMP27]], align 4, !alias.scope !17, !noalias !19
+; FVW2-NEXT:    [[TMP6:%.*]] = add i64 [[INDEX]], 0
+; FVW2-NEXT:    [[TMP7:%.*]] = mul i64 [[TMP6]], 4
+; FVW2-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, ptr [[PTR]], i64 [[TMP7]]
+; FVW2-NEXT:    [[TMP8:%.*]] = add i64 [[INDEX]], 0
+; FVW2-NEXT:    [[TMP9:%.*]] = mul i64 [[TMP8]], 64
+; FVW2-NEXT:    [[NEXT_GEP3:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP9]]
+; FVW2-NEXT:    [[TMP10:%.*]] = add i64 [[INDEX]], 1
+; FVW2-NEXT:    [[TMP11:%.*]] = mul i64 [[TMP10]], 64
+; FVW2-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i8, ptr [[DEST]], i64 [[TMP11]]
+; FVW2-NEXT:    [[TMP12:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP]], i64 [[IDXPROM]]
+; FVW2-NEXT:    [[TMP13:%.*]] = getelementptr inbounds float, ptr [[TMP12]], i32 0
+; FVW2-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x float>, ptr [[TMP13]], align 4
+; FVW2-NEXT:    [[TMP14:%.*]] = extractelement <2 x float> [[WIDE_LOAD]], i32 0
+; FVW2-NEXT:    store float [[TMP14]], ptr [[NEXT_GEP3]], align 4
+; FVW2-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[WIDE_LOAD]], i32 1
+; FVW2-NEXT:    store float [[TMP15]], ptr [[NEXT_GEP4]], align 4
+; FVW2-NEXT:    [[TMP16:%.*]] = getelementptr float, ptr [[NEXT_GEP]], i32 0
+; FVW2-NEXT:    [[WIDE_LOAD5:%.*]] = load <2 x float>, ptr [[TMP16]], align 4
+; FVW2-NEXT:    [[TMP17:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP3]], i64 1
+; FVW2-NEXT:    [[TMP18:%.*]] = getelementptr inbounds float, ptr [[NEXT_GEP4]], i64 1
+; FVW2-NEXT:    [[TMP19:%.*]] = extractelement <2 x float> [[WIDE_LOAD5]], i32 0
+; FVW2-NEXT:    store float [[TMP19]], ptr [[TMP17]], align 4
+; FVW2-NEXT:    [[TMP20:%.*]] = extractelement <2 x float> [[WIDE_LOAD5]], i32 1
+; FVW2-NEXT:    store float [[TMP20]], ptr [[TMP18]], align 4
 ; FVW2-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
-; FVW2-NEXT:    [[TMP30:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; FVW2-NEXT:    br i1 [[TMP30]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP22:![0-9]+]]
+; FVW2-NEXT:    [[TMP21:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; FVW2-NEXT:    br i1 [[TMP21]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP14:![0-9]+]]
 ; FVW2:       middle.block:
 ; FVW2-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]]
 ; FVW2-NEXT:    br i1 [[CMP_N]], label [[FOR_END]], label [[SCALAR_PH]]
 ; FVW2:       scalar.ph:
-; FVW2-NEXT:    [[BC_RESUME_VAL:%.*]] = phi ptr [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ [[PTR]], [[FOR_BODY_LR_PH]] ], [ [[PTR]], [[VECTOR_MEMCHECK]] ]
-; FVW2-NEXT:    [[BC_RESUME_VAL8:%.*]] = phi ptr [ [[IND_END7]], [[MIDDLE_BLOCK]] ], [ [[DEST]], [[FOR_BODY_LR_PH]] ], [ [[DEST]], [[VECTOR_MEMCHECK]] ]
+; FVW2-NEXT:    [[BC_RESUME_VAL:%.*]] = phi ptr [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ [[PTR]], [[FOR_BODY_LR_PH]] ]
+; FVW2-NEXT:    [[BC_RESUME_VAL2:%.*]] = phi ptr [ [[IND_END1]], [[MIDDLE_BLOCK]] ], [ [[DEST]], [[FOR_BODY_LR_PH]] ]
 ; FVW2-NEXT:    br label [[FOR_BODY:%.*]]
 ; FVW2:       for.body:
 ; FVW2-NEXT:    [[PTR_ADDR_012:%.*]] = phi ptr [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ]
-; FVW2-NEXT:    [[DEST_ADDR_011:%.*]] = phi ptr [ [[BC_RESUME_VAL8]], [[SCALAR_PH]] ], [ [[ADD_PTR6:%.*]], [[FOR_BODY]] ]
+; FVW2-NEXT:    [[DEST_ADDR_011:%.*]] = phi ptr [ [[BC_RESUME_VAL2]], [[SCALAR_PH]] ], [ [[ADD_PTR6:%.*]], [[FOR_BODY]] ]
 ; FVW2-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, ptr [[PTR_ADDR_012]], i64 [[IDXPROM]]
-; FVW2-NEXT:    [[TMP31:%.*]] = load float, ptr [[ARRAYIDX]], align 4
-; FVW2-NEXT:    store float [[TMP31]], ptr [[DEST_ADDR_011]], align 4
-; FVW2-NEXT:    [[TMP32:%.*]] = load float, ptr [[PTR_ADDR_012]], align 4
+; FVW2-NEXT:    [[TMP22:%.*]] = load float, ptr [[ARRAYIDX]], align 4
+; FVW2-NEXT:    store float [[TMP22]], ptr [[DEST_ADDR_011]], align 4
+; FVW2-NEXT:    [[TMP23:%.*]] = load float, ptr [[PTR_ADDR_012]], align 4
 ; FVW2-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds float, ptr [[DEST_ADDR_011]], i64 1
-; FVW2-NEXT:    store float [[TMP32]], ptr [[ARRAYIDX5]], align 4
+; FVW2-NEXT:    store float [[TMP23]], ptr [[ARRAYIDX5]], align 4
 ; FVW2-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds float, ptr [[PTR_ADDR_012]], i64 1
 ; FVW2-NEXT:    [[ADD_PTR6]] = getelementptr inbounds float, ptr [[DEST_ADDR_011]], i64 16
 ; FVW2-NEXT:    [[CMP_NOT:%.*]] = icmp eq ptr [[INCDEC_PTR]], [[ADD_PTR]]
-; FVW2-NEXT:    br i1 [[CMP_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP23:![0-9]+]]
+; FVW2-NEXT:    br i1 [[CMP_NOT]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP15:![0-9]+]]
 ; FVW2:       for.end:
 ; FVW2-NEXT:    ret void
 ;
 entry:
   %idx.ext = sext i32 %d to i64
   %add.ptr = getelementptr inbounds float, ptr %ptr, i64 %idx.ext
   %cmp.not10 = icmp eq i32 %d, 0
   br i1 %cmp.not10, label %for.end, label %for.body.lr.ph