[LV] Enable the LoopVectorizer to create pointer inductions

[anwel]

Summary

Currently, pointer inductions are scalarized by the LoopVectorizer. Individual getelementptrs are built to create an address from the induction variable and are then inserted into vectors to be used by vector loads and stores. This patch enables the LoopVectorizer to build a phi of pointer type and provide the vector loads and stores with vector type getelementptrs built from the pointer induction variable.

[SjoerdMeijer]

The approach looks very reasonable to me. A first round of mainly nits, while in the mean time I go over this a second time and let things sink in.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4192

Ah, look at this....I was actually wondering about this last week, and now I found it :)

4222

nit: unnecessary change?

4222

Nit: can we just return here, and get rid of the "else"? That saves some indentation.

4223

Nit: I don't know if the step is always a constant for a PtrInduction, so don't know if this should an assert or just a return.

4230

Nit: NewOldPhi is a bit of a confusing name.

4246

nit: this comment could be more descriptive.

4535

Nit: this comment is slightly out of date, i.e. the ScalarPtrs part....

4539

.... because we are adding things to Worklist here.
I was getting confused a bit if this should not be ScalarsPtrs, but because it is an induction it makes sense to add it to Worklist?

4574

Nit: missing .

[anwel]

All nitpicks have been addressed.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4222

Yes, but also unnecessary return in the first place, and a change very close to the relevant changes?

4222

Yes, we can.

4223

Using the assert here was inspired by emitTransformedIndex, which was - and, if Cost->isScalarAfterVectorization(P, VF) is true, still is - used for scalarising the induction. It uses the exact same assert for the step of a pointer induction.

4230

Okay, maybe a bit confusing. I renamed it to NewPointerPhi, that's a bit more straightforward.

4246

Comment has been made more descriptive :)

[SjoerdMeijer]

This looks like a good change to me. Please wait a day or 2 with committing this in case @Ayal or @fhahn have additional comments.

[rupprecht]

Hi, I had to revert this in 10c82eecbcb7 due to crashing when building gzip (util.c). Reduced as both C and IR:

$ cat repro.c
void a(char* b) {
  for (char* c = 0; c != b;)
    if (*--c) *c = '_';
}
$ clang -cc1 -emit-obj -target-cpu x86-64 -target-feature +sse4.2 -O2 -Wall -vectorize-loops repro.c
PHI node entries do not match predecessors!
  %pointer.phi = phi i8* [ null, %vector.ph ], [ %2, %vector.body ]
label %vector.body
label %vector.ph
clang: /home/rupprecht/src/llvm-project/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:8029: bool llvm::LoopVectorizePass::processLoop(llvm::Loop *): Assertion `!verifyFunction(*L->getHeader()->getParent(), &dbgs())' failed.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /home/rupprecht/dev/clang -cc1 -emit-obj -target-cpu x86-64 -target-feature +sse4.2 -O2 -Wall -vectorize-loops repro.c 
1.      <eof> parser at end of file
2.      Per-module optimization passes
3.      Running pass 'Function Pass Manager' on module 'repro.c'.
4.      Running pass 'Loop Vectorization' on function '@a'
 #0 0x0000000008912197 llvm::sys::PrintStackTrace(llvm::raw_ostream&) /home/rupprecht/src/llvm-project/llvm/lib/Support/Unix/Signals.inc:564:11
 #1 0x0000000008912339 PrintStackTraceSignalHandler(void*) /home/rupprecht/src/llvm-project/llvm/lib/Support/Unix/Signals.inc:625:1
 #2 0x0000000008910b4b llvm::sys::RunSignalHandlers() /home/rupprecht/src/llvm-project/llvm/lib/Support/Signals.cpp:67:5
 #3 0x0000000008912a95 SignalHandler(int) /home/rupprecht/src/llvm-project/llvm/lib/Support/Unix/Signals.inc:406:1
 #4 0x00007f7c948ff110 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14110)
 #5 0x00007f7c92e7a761 raise /build/glibc-M65Gwz/glibc-2.30/signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #6 0x00007f7c92e6455b abort /build/glibc-M65Gwz/glibc-2.30/stdlib/abort.c:81:7
 #7 0x00007f7c92e6442f get_sysdep_segment_value /build/glibc-M65Gwz/glibc-2.30/intl/loadmsgcat.c:509:8
 #8 0x00007f7c92e6442f _nl_load_domain /build/glibc-M65Gwz/glibc-2.30/intl/loadmsgcat.c:970:34
 #9 0x00007f7c92e73092 (/lib/x86_64-linux-gnu/libc.so.6+0x34092)
#10 0x0000000008adb37b llvm::LoopVectorizePass::processLoop(llvm::Loop*) /home/rupprecht/src/llvm-project/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:8030:3

$ cat repro.ll
; ModuleID = './repro.ll'
source_filename = "repro.c"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

; Function Attrs: noinline nounwind
define void @a(i8* %b) #0 {
entry:
  %b.addr = alloca i8*, align 8
  %c = alloca i8*, align 8
  store i8* %b, i8** %b.addr, align 8
  br label %for.cond

for.cond:                                         ; preds = %if.end, %entry
  %0 = load i8*, i8** %c, align 8
  %1 = load i8*, i8** %b.addr, align 8
  %cmp = icmp ne i8* %0, %1
  br i1 %cmp, label %for.body, label %for.end

for.body:                                         ; preds = %for.cond
  %2 = load i8*, i8** %c, align 8
  %incdec.ptr = getelementptr inbounds i8, i8* %2, i32 -1
  store i8* %incdec.ptr, i8** %c, align 8
  %3 = load i8, i8* %incdec.ptr, align 1
  %tobool = icmp ne i8 %3, 0
  br i1 %tobool, label %if.then, label %if.end

if.then:                                          ; preds = %for.body
  %4 = load i8*, i8** %c, align 8
  store i8 95, i8* %4, align 1
  br label %if.end

if.end:                                           ; preds = %if.then, %for.body
  br label %for.cond

for.end:                                          ; preds = %for.cond
  ret void
}

attributes #0 = { noinline nounwind "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+popcnt,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }

!llvm.module.flags = !{!0}

!0 = !{i32 1, !"wchar_size", i32 4}
$ opt -O2 repro.ll -o repro.o
<same as above>

Reopening as the patch is reverted

[rupprecht]

I guess I should also request changes since there's a crash :)

[anwel]

Hi, there was a bug concerning the location at which the new getelementptr instruction would be generated, thanks for catching this.
I have fixed the bug and derived a LoopVectorize test case from your C example.

[dmgreen]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

1033

Latch might be a better name here.

llvm/test/Transforms/LoopVectorize/pointer-induction.ll

4 ↗

(On Diff #276391)

I think if you use x86 as a target (and needs it for the costing), the test needs to go into test/Transforms/LoopVectorize/X86 in case the target is not compiled in.

7 ↗

(On Diff #276391)

Also some of this might be able to be cleaned up, like the local_unnamed_addr, the metadata and all/most(?) of the attributes.

[fhahn]

llvm/test/Transforms/LoopVectorize/pointer-induction.ll

4 ↗

(On Diff #276391)

It looks like the options above actually force vectorization with a certain factor. In that case, it Is probably best to remove the triple.

I'd also consider just checking the loop-vectorize output (without -dce -instcombine), if it is not too messy, as it makes the test more prone to break when something changes in instcombine. Also, it might be possible to only specifically check the IR related to the generated induction, rather than autogenerating the checks, which include a lot of relatively irrelevant stuff.

[anwel]

Revisited the new test to make it cleaner.

llvm/test/Transforms/LoopVectorize/pointer-induction.ll

4 ↗

(On Diff #276391)

I thought it did need the target information to behave in the right way, but apparently I was mistaken - so no relocation necessary, I removed the target.

4 ↗

(On Diff #276391)

Thanks for the feedback, I don't have much experience writing opt tests so your advice is very welcome.

I have removed the triple and the meta data, after checking that we don't need them, and reduced the checks to vector.ph, vector.body and the loop latch that changes the induction variable.

7 ↗

(On Diff #276391)

Should be a lot cleaner now.

[dmgreen]

Thanks. This LGTM.

Perhaps wait a few days until after the branch is taken though, to be on the safe side.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

   // This PHINode must be an induction variable.
   // Make sure that we know about it.
   assert(Legal->getInductionVars().count(P) && "Not an induction variable");
 
   InductionDescriptor II = Legal->getInductionVars().lookup(P);
   const DataLayout &DL = OrigLoop->getHeader()->getModule()->getDataLayout();
 
   // FIXME: The newly created binary instructions should contain nsw/nuw flags,
   // which can be found from the original scalar operations.
   switch (II.getKind()) {
   case InductionDescriptor::IK_NoInduction:
     llvm_unreachable("Unknown induction");
   case InductionDescriptor::IK_IntInduction:
   case InductionDescriptor::IK_FpInduction:
     llvm_unreachable("Integer/fp induction is handled elsewhere.");
   case InductionDescriptor::IK_PtrInduction: {
     // Handle the pointer induction variable case.
     assert(P->getType()->isPointerTy() && "Unexpected type.");
-    // This is the normalized GEP that starts counting at zero.
-    Value *PtrInd = Induction;
-    PtrInd = Builder.CreateSExtOrTrunc(PtrInd, II.getStep()->getType());
-    // Determine the number of scalars we need to generate for each unroll
-    // iteration. If the instruction is uniform, we only need to generate the
-    // first lane. Otherwise, we generate all VF values.
-    unsigned Lanes = Cost->isUniformAfterVectorization(P, VF) ? 1 : VF;
-    // These are the scalar results. Notice that we don't generate vector GEPs
-    // because scalar GEPs result in better code.
+
+    if (Cost->isScalarAfterVectorization(P, VF)) {
+      // This is the normalized GEP that starts counting at zero.
+      Value *PtrInd =
+          Builder.CreateSExtOrTrunc(Induction, II.getStep()->getType());
+      // Determine the number of scalars we need to generate for each unroll
+      // iteration. If the instruction is uniform, we only need to generate the
+      // first lane. Otherwise, we generate all VF values.
+      unsigned Lanes = Cost->isUniformAfterVectorization(P, VF) ? 1 : VF;
       for (unsigned Part = 0; Part < UF; ++Part) {
         for (unsigned Lane = 0; Lane < Lanes; ++Lane) {
           Constant *Idx = ConstantInt::get(PtrInd->getType(), Lane + Part * VF);
           Value *GlobalIdx = Builder.CreateAdd(PtrInd, Idx);
           Value *SclrGep =
               emitTransformedIndex(Builder, GlobalIdx, PSE.getSE(), DL, II);
           SclrGep->setName("next.gep");
           VectorLoopValueMap.setScalarValue(P, {Part, Lane}, SclrGep);
+        }
       }
+      return;
+    }
+    assert(isa<SCEVConstant>(II.getStep()) &&
+           "Induction step not a SCEV constant!");
+    Type *PhiType = II.getStep()->getType();
+
+    // Build a pointer phi
+    Value *ScalarStartValue = II.getStartValue();
+    Type *ScStValueType = ScalarStartValue->getType();
+    PHINode *NewPointerPhi =
+        PHINode::Create(ScStValueType, 2, "pointer.phi", Induction);
+    NewPointerPhi->addIncoming(ScalarStartValue, LoopVectorPreHeader);
+
+    // A pointer induction, performed by using a gep
+    const SCEV *ScalarStep = II.getStep();
+    SCEVExpander Exp(*PSE.getSE(), DL, "induction");
+    Value *ScalarStepValue =
+        Exp.expandCodeFor(ScalarStep, PhiType, &*Builder.GetInsertPoint());
+    Value *InductionGEP = Builder.CreateGEP(
+        ScStValueType->getPointerElementType(), NewPointerPhi,
+        Builder.CreateMul(ScalarStepValue, ConstantInt::get(PhiType, VF * UF)));
+    NewPointerPhi->addIncoming(InductionGEP,
+                               cast<Instruction>(InductionGEP)->getParent());
+
+    // Create UF many actual address geps that use the pointer
+    // phi as base and a vectorized version of the step value
+    // (<step*0, ..., step*N>) as offset.
+    for (unsigned Part = 0; Part < UF; ++Part) {
+      SmallVector<Constant *, 8> Indices;
+      // Create a vector of consecutive numbers from zero to VF.
+      for (unsigned i = 0; i < VF; ++i)
+        Indices.push_back(ConstantInt::get(PhiType, i + Part * VF));
+      Constant *StartOffset = ConstantVector::get(Indices);
+
+      Value *GEP = Builder.CreateGEP(
+          ScStValueType->getPointerElementType(), NewPointerPhi,
+          Builder.CreateMul(StartOffset,
+                            Builder.CreateVectorSplat(VF, ScalarStepValue),
+                            "vector.gep"));
+      VectorLoopValueMap.setVectorValue(P, Part, GEP);
     }
-    return;
   }
   }
 }
 
 /// A helper function for checking whether an integer division-related
 /// instruction may divide by zero (in which case it must be predicated if
 /// executed conditionally in the scalar code).
 /// TODO: It may be worthwhile to generalize and check isKnownNonZero().
 /// Non-zero divisors that are non compile-time constants will not be

          "This function should not be visited twice for the same VF");
 
   SmallSetVector<Instruction *, 8> Worklist;
 
   // These sets are used to seed the analysis with pointers used by memory
   // accesses that will remain scalar.
   SmallSetVector<Instruction *, 8> ScalarPtrs;
   SmallPtrSet<Instruction *, 8> PossibleNonScalarPtrs;
+  auto *Latch = TheLoop->getLoopLatch();
 
   // A helper that returns true if the use of Ptr by MemAccess will be scalar.
   // The pointer operands of loads and stores will be scalar as long as the
   // memory access is not a gather or scatter operation. The value operand of a
   // store will remain scalar if the store is scalarized.
   auto isScalarUse = [&](Instruction *MemAccess, Value *Ptr) {
     InstWidening WideningDecision = getWideningDecision(MemAccess, VF);
     assert(WideningDecision != CM_Unknown &&

   // A helper that returns true if the given value is a bitcast or
   // getelementptr instruction contained in the loop.
   auto isLoopVaryingBitCastOrGEP = [&](Value *V) {
     return ((isa<BitCastInst>(V) && V->getType()->isPointerTy()) ||
             isa<GetElementPtrInst>(V)) &&
            !TheLoop->isLoopInvariant(V);
   };
 
-  // A helper that evaluates a memory access's use of a pointer. If the use
-  // will be a scalar use, and the pointer is only used by memory accesses, we
-  // place the pointer in ScalarPtrs. Otherwise, the pointer is placed in
-  // PossibleNonScalarPtrs.
+  auto isScalarPtrInduction = [&](Instruction *MemAccess, Value *Ptr) {
+    if (!isa<PHINode>(Ptr) ||
+        !Legal->getInductionVars().count(cast<PHINode>(Ptr)))
+      return false;
+    auto &Induction = Legal->getInductionVars()[cast<PHINode>(Ptr)];
+    if (Induction.getKind() != InductionDescriptor::IK_PtrInduction)
+      return false;
+    return isScalarUse(MemAccess, Ptr);
+  };
+
+  // A helper that evaluates a memory access's use of a pointer. If the
+  // pointer is actually the pointer induction of a loop, it is being
+  // inserted into Worklist. If the use will be a scalar use, and the
+  // pointer is only used by memory accesses, we place the pointer in
+  // ScalarPtrs. Otherwise, the pointer is placed in PossibleNonScalarPtrs.
   auto evaluatePtrUse = [&](Instruction *MemAccess, Value *Ptr) {
+    if (isScalarPtrInduction(MemAccess, Ptr)) {
+      Worklist.insert(cast<Instruction>(Ptr));
+      Instruction *Update = cast<Instruction>(
+          cast<PHINode>(Ptr)->getIncomingValueForBlock(Latch));
+      Worklist.insert(Update);
+      LLVM_DEBUG(dbgs() << "LV: Found new scalar instruction: " << *Ptr
+                        << "\n");
+      LLVM_DEBUG(dbgs() << "LV: Found new scalar instruction: " << *Update
+                        << "\n");
+      return;
+    }
     // We only care about bitcast and getelementptr instructions contained in
     // the loop.
     if (!isLoopVaryingBitCastOrGEP(Ptr))
       return;
 
     // If the pointer has already been identified as scalar (e.g., if it was
     // also identified as uniform), there's nothing to do.
     auto *I = cast<Instruction>(Ptr);
     if (Worklist.count(I))
       return;
 
     // If the use of the pointer will be a scalar use, and all users of the
     // pointer are memory accesses, place the pointer in ScalarPtrs. Otherwise,
     // place the pointer in PossibleNonScalarPtrs.
     if (isScalarUse(MemAccess, Ptr) && llvm::all_of(I->users(), [&](User *U) {
           return isa<LoadInst>(U) || isa<StoreInst>(U);
         }))
       ScalarPtrs.insert(I);
     else
       PossibleNonScalarPtrs.insert(I);
   };
 
   // We seed the scalars analysis with three classes of instructions: (1)
-  // instructions marked uniform-after-vectorization, (2) bitcast and
-  // getelementptr instructions used by memory accesses requiring a scalar use,
-  // and (3) pointer induction variables and their update instructions (we
-  // currently only scalarize these).
+  // instructions marked uniform-after-vectorization and (2) bitcast,
+  // getelementptr and (pointer) phi instructions used by memory accesses
+  // requiring a scalar use.
   //
   // (1) Add to the worklist all instructions that have been identified as
   // uniform-after-vectorization.
   Worklist.insert(Uniforms[VF].begin(), Uniforms[VF].end());
 
   // (2) Add to the worklist all bitcast and getelementptr instructions used by
   // memory accesses requiring a scalar use. The pointer operands of loads and
   // stores will be scalar as long as the memory accesses is not a gather or

       }
     }
   for (auto *I : ScalarPtrs)
     if (!PossibleNonScalarPtrs.count(I)) {
       LLVM_DEBUG(dbgs() << "LV: Found scalar instruction: " << *I << "\n");
       Worklist.insert(I);
     }
 
-  // (3) Add to the worklist all pointer induction variables and their update
-  // instructions.
-  //
-  // TODO: Once we are able to vectorize pointer induction variables we should
-  //       no longer insert them into the worklist here.
-  auto *Latch = TheLoop->getLoopLatch();
-  for (auto &Induction : Legal->getInductionVars()) {
-    auto *Ind = Induction.first;
-    auto *IndUpdate = cast<Instruction>(Ind->getIncomingValueForBlock(Latch));
-    if (Induction.second.getKind() != InductionDescriptor::IK_PtrInduction)
-      continue;
-    Worklist.insert(Ind);
-    Worklist.insert(IndUpdate);
-    LLVM_DEBUG(dbgs() << "LV: Found scalar instruction: " << *Ind << "\n");
-    LLVM_DEBUG(dbgs() << "LV: Found scalar instruction: " << *IndUpdate
-                      << "\n");
-  }
-
   // Insert the forced scalars.
   // FIXME: Currently widenPHIInstruction() often creates a dead vector
   // induction variable when the PHI user is scalarized.
   auto ForcedScalar = ForcedScalars.find(VF);
   if (ForcedScalar != ForcedScalars.end())
     for (auto *I : ForcedScalar->second)
       Worklist.insert(I);
 

   }
 
   // An induction variable will remain scalar if all users of the induction
   // variable and induction variable update remain scalar.
   for (auto &Induction : Legal->getInductionVars()) {
     auto *Ind = Induction.first;
     auto *IndUpdate = cast<Instruction>(Ind->getIncomingValueForBlock(Latch));
 
-    // We already considered pointer induction variables, so there's no reason
-    // to look at their users again.
-    //
-    // TODO: Once we are able to vectorize pointer induction variables we
-    //       should no longer skip over them here.
-    if (Induction.second.getKind() == InductionDescriptor::IK_PtrInduction)
-      continue;
-
     // If tail-folding is applied, the primary induction variable will be used
     // to feed a vector compare.
     if (Ind == Legal->getPrimaryInduction() && foldTailByMasking())
       continue;
 
     // Determine if all users of the induction variable are scalar after
     // vectorization.
     auto ScalarInd = llvm::all_of(Ind->users(), [&](User *U) -> bool {

llvm/test/Transforms/LoopVectorize/ARM/pointer_iv.ll

 ; CHECK-LABEL: @pointer_phi_v4i32_add3(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i32, i32* [[A:%.*]], i32 2988
 ; CHECK-NEXT:    [[IND_END3:%.*]] = getelementptr i32, i32* [[B:%.*]], i32 996
 ; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> undef, i32 [[Y:%.*]], i32 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i32 [[INDEX]], 3
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], 3
-; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP2]]
-; CHECK-NEXT:    [[TMP3:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP4:%.*]] = mul i32 [[TMP3]], 3
-; CHECK-NEXT:    [[NEXT_GEP5:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP4]]
-; CHECK-NEXT:    [[TMP5:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i32 [[TMP5]], 3
-; CHECK-NEXT:    [[NEXT_GEP6:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32*> [[TMP7]], i32* [[NEXT_GEP4]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32*> [[TMP8]], i32* [[NEXT_GEP5]], i32 2
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32*> [[TMP9]], i32* [[NEXT_GEP6]], i32 3
-; CHECK-NEXT:    [[NEXT_GEP7:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP10]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[TMP11:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP12:%.*]] = bitcast i32* [[NEXT_GEP7]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP11]], <4 x i32>* [[TMP12]], align 4
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi i32* [ [[A]], [[ENTRY:%.*]] ], [ [[TMP0:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP0]] = getelementptr i32, i32* [[POINTER_PHI]], i32 12
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP1]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[NEXT_GEP]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP13:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
-; CHECK-NEXT:    br i1 [[TMP13]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !5
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
+; CHECK-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !5
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[A_ADDR_09:%.*]] = phi i32* [ [[ADD_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[I_08:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 996, [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[B_ADDR_07:%.*]] = phi i32* [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END3]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP14:%.*]] = load i32, i32* [[A_ADDR_09]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = load i32, i32* [[A_ADDR_09]], align 4
 ; CHECK-NEXT:    [[ADD_PTR]] = getelementptr inbounds i32, i32* [[A_ADDR_09]], i32 3
-; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP14]], [[Y]]
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP5]], [[Y]]
 ; CHECK-NEXT:    store i32 [[ADD]], i32* [[B_ADDR_07]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds i32, i32* [[B_ADDR_07]], i32 1
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_08]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 1000
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[END:%.*]], label [[FOR_BODY]], !llvm.loop !6
 ; CHECK:       end:
 ; CHECK-NEXT:    ret void
 ;

 ; CHECK-LABEL: @pointer_phi_v4f32_add3(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr float, float* [[A:%.*]], i32 2988
 ; CHECK-NEXT:    [[IND_END3:%.*]] = getelementptr float, float* [[B:%.*]], i32 996
 ; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x float> undef, float [[Y:%.*]], i32 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x float> [[BROADCAST_SPLATINSERT]], <4 x float> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i32 [[INDEX]], 3
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr float, float* [[A]], i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], 3
-; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr float, float* [[A]], i32 [[TMP2]]
-; CHECK-NEXT:    [[TMP3:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP4:%.*]] = mul i32 [[TMP3]], 3
-; CHECK-NEXT:    [[NEXT_GEP5:%.*]] = getelementptr float, float* [[A]], i32 [[TMP4]]
-; CHECK-NEXT:    [[TMP5:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i32 [[TMP5]], 3
-; CHECK-NEXT:    [[NEXT_GEP6:%.*]] = getelementptr float, float* [[A]], i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x float*> undef, float* [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x float*> [[TMP7]], float* [[NEXT_GEP4]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x float*> [[TMP8]], float* [[NEXT_GEP5]], i32 2
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x float*> [[TMP9]], float* [[NEXT_GEP6]], i32 3
-; CHECK-NEXT:    [[NEXT_GEP7:%.*]] = getelementptr float, float* [[B]], i32 [[INDEX]]
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> [[TMP10]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
-; CHECK-NEXT:    [[TMP11:%.*]] = fadd fast <4 x float> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP12:%.*]] = bitcast float* [[NEXT_GEP7]] to <4 x float>*
-; CHECK-NEXT:    store <4 x float> [[TMP11]], <4 x float>* [[TMP12]], align 4
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi float* [ [[A]], [[ENTRY:%.*]] ], [ [[TMP0:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP0]] = getelementptr float, float* [[POINTER_PHI]], i32 12
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr float, float* [[POINTER_PHI]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr float, float* [[B]], i32 [[INDEX]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> [[TMP1]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd fast <4 x float> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[NEXT_GEP]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP2]], <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP13:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
-; CHECK-NEXT:    br i1 [[TMP13]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !17
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[INDEX_NEXT]], 996
+; CHECK-NEXT:    br i1 [[TMP4]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !17
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[A_ADDR_09:%.*]] = phi float* [ [[ADD_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[I_08:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 996, [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[B_ADDR_07:%.*]] = phi float* [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END3]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP14:%.*]] = load float, float* [[A_ADDR_09]], align 4
+; CHECK-NEXT:    [[TMP5:%.*]] = load float, float* [[A_ADDR_09]], align 4
 ; CHECK-NEXT:    [[ADD_PTR]] = getelementptr inbounds float, float* [[A_ADDR_09]], i32 3
-; CHECK-NEXT:    [[ADD:%.*]] = fadd fast float [[TMP14]], [[Y]]
+; CHECK-NEXT:    [[ADD:%.*]] = fadd fast float [[TMP5]], [[Y]]
 ; CHECK-NEXT:    store float [[ADD]], float* [[B_ADDR_07]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds float, float* [[B_ADDR_07]], i32 1
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_08]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 1000
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[END:%.*]], label [[FOR_BODY]], !llvm.loop !18
 ; CHECK:       end:
 ; CHECK-NEXT:    ret void
 ;

 
 define hidden void @pointer_phi_v4i32_uf2(i32* noalias nocapture readonly %A, i32* noalias nocapture %B, i32 %n, i32 %y) {
 ; CHECK-LABEL: @pointer_phi_v4i32_uf2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i32, i32* [[A:%.*]], i32 59952
 ; CHECK-NEXT:    [[IND_END3:%.*]] = getelementptr i32, i32* [[B:%.*]], i32 9992
 ; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> undef, i32 [[Y:%.*]], i32 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> undef, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT14:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT15:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT14]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT6:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT7:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT6]], <4 x i32> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i32 [[INDEX]], 6
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], 6
-; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP2]]
-; CHECK-NEXT:    [[TMP3:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP4:%.*]] = mul i32 [[TMP3]], 6
-; CHECK-NEXT:    [[NEXT_GEP5:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP4]]
-; CHECK-NEXT:    [[TMP5:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i32 [[TMP5]], 6
-; CHECK-NEXT:    [[NEXT_GEP6:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32*> [[TMP7]], i32* [[NEXT_GEP4]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32*> [[TMP8]], i32* [[NEXT_GEP5]], i32 2
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32*> [[TMP9]], i32* [[NEXT_GEP6]], i32 3
-; CHECK-NEXT:    [[TMP11:%.*]] = or i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP12:%.*]] = mul i32 [[TMP11]], 6
-; CHECK-NEXT:    [[NEXT_GEP7:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP12]]
-; CHECK-NEXT:    [[TMP13:%.*]] = or i32 [[INDEX]], 5
-; CHECK-NEXT:    [[TMP14:%.*]] = mul i32 [[TMP13]], 6
-; CHECK-NEXT:    [[NEXT_GEP8:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP14]]
-; CHECK-NEXT:    [[TMP15:%.*]] = or i32 [[INDEX]], 6
-; CHECK-NEXT:    [[TMP16:%.*]] = mul i32 [[TMP15]], 6
-; CHECK-NEXT:    [[NEXT_GEP9:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP16]]
-; CHECK-NEXT:    [[TMP17:%.*]] = or i32 [[INDEX]], 7
-; CHECK-NEXT:    [[TMP18:%.*]] = mul i32 [[TMP17]], 6
-; CHECK-NEXT:    [[NEXT_GEP10:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP18]]
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP7]], i32 0
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <4 x i32*> [[TMP19]], i32* [[NEXT_GEP8]], i32 1
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <4 x i32*> [[TMP20]], i32* [[NEXT_GEP9]], i32 2
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <4 x i32*> [[TMP21]], i32* [[NEXT_GEP10]], i32 3
-; CHECK-NEXT:    [[NEXT_GEP11:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP10]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER13:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP22]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[TMP23:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP24:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER13]], [[BROADCAST_SPLAT15]]
-; CHECK-NEXT:    [[TMP25:%.*]] = bitcast i32* [[NEXT_GEP11]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP23]], <4 x i32>* [[TMP25]], align 4
-; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr i32, i32* [[NEXT_GEP11]], i32 4
-; CHECK-NEXT:    [[TMP27:%.*]] = bitcast i32* [[TMP26]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP24]], <4 x i32>* [[TMP27]], align 4
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi i32* [ [[A]], [[ENTRY:%.*]] ], [ [[TMP0:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP0]] = getelementptr i32, i32* [[POINTER_PHI]], i32 48
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 0, i32 6, i32 12, i32 18>
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 24, i32 30, i32 36, i32 42>
+; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP1]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER5:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP2]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[TMP3:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP4:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER5]], [[BROADCAST_SPLAT7]]
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i32* [[NEXT_GEP]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP3]], <4 x i32>* [[TMP5]], align 4
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr i32, i32* [[NEXT_GEP]], i32 4
+; CHECK-NEXT:    [[TMP7:%.*]] = bitcast i32* [[TMP6]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], <4 x i32>* [[TMP7]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 8
-; CHECK-NEXT:    [[TMP28:%.*]] = icmp eq i32 [[INDEX_NEXT]], 9992
-; CHECK-NEXT:    br i1 [[TMP28]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !22
+; CHECK-NEXT:    [[TMP8:%.*]] = icmp eq i32 [[INDEX_NEXT]], 9992
+; CHECK-NEXT:    br i1 [[TMP8]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !22
 ; CHECK:       for.cond.cleanup:
 ; CHECK-NEXT:    ret void
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[A_ADDR_08:%.*]] = phi i32* [ [[ADD_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[I_07:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 9992, [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[B_ADDR_06:%.*]] = phi i32* [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END3]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP29:%.*]] = load i32, i32* [[A_ADDR_08]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = load i32, i32* [[A_ADDR_08]], align 4
 ; CHECK-NEXT:    [[ADD_PTR]] = getelementptr inbounds i32, i32* [[A_ADDR_08]], i32 6
-; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP29]], [[Y]]
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP9]], [[Y]]
 ; CHECK-NEXT:    store i32 [[ADD]], i32* [[B_ADDR_06]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds i32, i32* [[B_ADDR_06]], i32 1
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_07]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 10000
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY]], !llvm.loop !23
 ;
+
 entry:
   br label %for.body
 
 for.cond.cleanup:
   ret void
 
 for.body:
   %A.addr.08 = phi i32* [ %A, %entry ], [ %add.ptr, %for.body ]

 
 define hidden void @pointer_phi_v4i32_uf4(i32* noalias nocapture readonly %A, i32* noalias nocapture %B, i32 %n, i32 %y) {
 ; CHECK-LABEL: @pointer_phi_v4i32_uf4(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i32, i32* [[A:%.*]], i32 59904
 ; CHECK-NEXT:    [[IND_END3:%.*]] = getelementptr i32, i32* [[B:%.*]], i32 9984
 ; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> undef, i32 [[Y:%.*]], i32 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> undef, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT26:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT27:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT26]], <4 x i32> undef, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT28:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT29:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT28]], <4 x i32> undef, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT30:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT31:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT30]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT10:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT11:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT10]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT12:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT13:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT12]], <4 x i32> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT14:%.*]] = insertelement <4 x i32> undef, i32 [[Y]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT15:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT14]], <4 x i32> undef, <4 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i32 [[INDEX]], 6
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], 6
-; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP2]]
-; CHECK-NEXT:    [[TMP3:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP4:%.*]] = mul i32 [[TMP3]], 6
-; CHECK-NEXT:    [[NEXT_GEP5:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP4]]
-; CHECK-NEXT:    [[TMP5:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i32 [[TMP5]], 6
-; CHECK-NEXT:    [[NEXT_GEP6:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32*> [[TMP7]], i32* [[NEXT_GEP4]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i32*> [[TMP8]], i32* [[NEXT_GEP5]], i32 2
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x i32*> [[TMP9]], i32* [[NEXT_GEP6]], i32 3
-; CHECK-NEXT:    [[TMP11:%.*]] = or i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP12:%.*]] = mul i32 [[TMP11]], 6
-; CHECK-NEXT:    [[NEXT_GEP7:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP12]]
-; CHECK-NEXT:    [[TMP13:%.*]] = or i32 [[INDEX]], 5
-; CHECK-NEXT:    [[TMP14:%.*]] = mul i32 [[TMP13]], 6
-; CHECK-NEXT:    [[NEXT_GEP8:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP14]]
-; CHECK-NEXT:    [[TMP15:%.*]] = or i32 [[INDEX]], 6
-; CHECK-NEXT:    [[TMP16:%.*]] = mul i32 [[TMP15]], 6
-; CHECK-NEXT:    [[NEXT_GEP9:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP16]]
-; CHECK-NEXT:    [[TMP17:%.*]] = or i32 [[INDEX]], 7
-; CHECK-NEXT:    [[TMP18:%.*]] = mul i32 [[TMP17]], 6
-; CHECK-NEXT:    [[NEXT_GEP10:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP18]]
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP7]], i32 0
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <4 x i32*> [[TMP19]], i32* [[NEXT_GEP8]], i32 1
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <4 x i32*> [[TMP20]], i32* [[NEXT_GEP9]], i32 2
-; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <4 x i32*> [[TMP21]], i32* [[NEXT_GEP10]], i32 3
-; CHECK-NEXT:    [[TMP23:%.*]] = or i32 [[INDEX]], 8
-; CHECK-NEXT:    [[TMP24:%.*]] = mul i32 [[TMP23]], 6
-; CHECK-NEXT:    [[NEXT_GEP11:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP24]]
-; CHECK-NEXT:    [[TMP25:%.*]] = or i32 [[INDEX]], 9
-; CHECK-NEXT:    [[TMP26:%.*]] = mul i32 [[TMP25]], 6
-; CHECK-NEXT:    [[NEXT_GEP12:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP26]]
-; CHECK-NEXT:    [[TMP27:%.*]] = or i32 [[INDEX]], 10
-; CHECK-NEXT:    [[TMP28:%.*]] = mul i32 [[TMP27]], 6
-; CHECK-NEXT:    [[NEXT_GEP13:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP28]]
-; CHECK-NEXT:    [[TMP29:%.*]] = or i32 [[INDEX]], 11
-; CHECK-NEXT:    [[TMP30:%.*]] = mul i32 [[TMP29]], 6
-; CHECK-NEXT:    [[NEXT_GEP14:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP30]]
-; CHECK-NEXT:    [[TMP31:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP11]], i32 0
-; CHECK-NEXT:    [[TMP32:%.*]] = insertelement <4 x i32*> [[TMP31]], i32* [[NEXT_GEP12]], i32 1
-; CHECK-NEXT:    [[TMP33:%.*]] = insertelement <4 x i32*> [[TMP32]], i32* [[NEXT_GEP13]], i32 2
-; CHECK-NEXT:    [[TMP34:%.*]] = insertelement <4 x i32*> [[TMP33]], i32* [[NEXT_GEP14]], i32 3
-; CHECK-NEXT:    [[TMP35:%.*]] = or i32 [[INDEX]], 12
-; CHECK-NEXT:    [[TMP36:%.*]] = mul i32 [[TMP35]], 6
-; CHECK-NEXT:    [[NEXT_GEP15:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP36]]
-; CHECK-NEXT:    [[TMP37:%.*]] = or i32 [[INDEX]], 13
-; CHECK-NEXT:    [[TMP38:%.*]] = mul i32 [[TMP37]], 6
-; CHECK-NEXT:    [[NEXT_GEP16:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP38]]
-; CHECK-NEXT:    [[TMP39:%.*]] = or i32 [[INDEX]], 14
-; CHECK-NEXT:    [[TMP40:%.*]] = mul i32 [[TMP39]], 6
-; CHECK-NEXT:    [[NEXT_GEP17:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP40]]
-; CHECK-NEXT:    [[TMP41:%.*]] = or i32 [[INDEX]], 15
-; CHECK-NEXT:    [[TMP42:%.*]] = mul i32 [[TMP41]], 6
-; CHECK-NEXT:    [[NEXT_GEP18:%.*]] = getelementptr i32, i32* [[A]], i32 [[TMP42]]
-; CHECK-NEXT:    [[TMP43:%.*]] = insertelement <4 x i32*> undef, i32* [[NEXT_GEP15]], i32 0
-; CHECK-NEXT:    [[TMP44:%.*]] = insertelement <4 x i32*> [[TMP43]], i32* [[NEXT_GEP16]], i32 1
-; CHECK-NEXT:    [[TMP45:%.*]] = insertelement <4 x i32*> [[TMP44]], i32* [[NEXT_GEP17]], i32 2
-; CHECK-NEXT:    [[TMP46:%.*]] = insertelement <4 x i32*> [[TMP45]], i32* [[NEXT_GEP18]], i32 3
-; CHECK-NEXT:    [[NEXT_GEP19:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP10]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER23:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP22]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER24:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP34]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER25:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP46]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
-; CHECK-NEXT:    [[TMP47:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP48:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER23]], [[BROADCAST_SPLAT27]]
-; CHECK-NEXT:    [[TMP49:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER24]], [[BROADCAST_SPLAT29]]
-; CHECK-NEXT:    [[TMP50:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER25]], [[BROADCAST_SPLAT31]]
-; CHECK-NEXT:    [[TMP51:%.*]] = bitcast i32* [[NEXT_GEP19]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP47]], <4 x i32>* [[TMP51]], align 4
-; CHECK-NEXT:    [[TMP52:%.*]] = getelementptr i32, i32* [[NEXT_GEP19]], i32 4
-; CHECK-NEXT:    [[TMP53:%.*]] = bitcast i32* [[TMP52]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP48]], <4 x i32>* [[TMP53]], align 4
-; CHECK-NEXT:    [[TMP54:%.*]] = getelementptr i32, i32* [[NEXT_GEP19]], i32 8
-; CHECK-NEXT:    [[TMP55:%.*]] = bitcast i32* [[TMP54]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP49]], <4 x i32>* [[TMP55]], align 4
-; CHECK-NEXT:    [[TMP56:%.*]] = getelementptr i32, i32* [[NEXT_GEP19]], i32 12
-; CHECK-NEXT:    [[TMP57:%.*]] = bitcast i32* [[TMP56]] to <4 x i32>*
-; CHECK-NEXT:    store <4 x i32> [[TMP50]], <4 x i32>* [[TMP57]], align 4
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi i32* [ [[A]], [[ENTRY:%.*]] ], [ [[TMP0:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP0]] = getelementptr i32, i32* [[POINTER_PHI]], i32 96
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 0, i32 6, i32 12, i32 18>
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 24, i32 30, i32 36, i32 42>
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 48, i32 54, i32 60, i32 66>
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr i32, i32* [[POINTER_PHI]], <4 x i32> <i32 72, i32 78, i32 84, i32 90>
+; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i32, i32* [[B]], i32 [[INDEX]]
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP1]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER7:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP2]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER8:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP3]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER9:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> [[TMP4]], i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
+; CHECK-NEXT:    [[TMP5:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER7]], [[BROADCAST_SPLAT11]]
+; CHECK-NEXT:    [[TMP7:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER8]], [[BROADCAST_SPLAT13]]
+; CHECK-NEXT:    [[TMP8:%.*]] = add nsw <4 x i32> [[WIDE_MASKED_GATHER9]], [[BROADCAST_SPLAT15]]
+; CHECK-NEXT:    [[TMP9:%.*]] = bitcast i32* [[NEXT_GEP]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP5]], <4 x i32>* [[TMP9]], align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr i32, i32* [[NEXT_GEP]], i32 4
+; CHECK-NEXT:    [[TMP11:%.*]] = bitcast i32* [[TMP10]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP6]], <4 x i32>* [[TMP11]], align 4
+; CHECK-NEXT:    [[TMP12:%.*]] = getelementptr i32, i32* [[NEXT_GEP]], i32 8
+; CHECK-NEXT:    [[TMP13:%.*]] = bitcast i32* [[TMP12]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP7]], <4 x i32>* [[TMP13]], align 4
+; CHECK-NEXT:    [[TMP14:%.*]] = getelementptr i32, i32* [[NEXT_GEP]], i32 12
+; CHECK-NEXT:    [[TMP15:%.*]] = bitcast i32* [[TMP14]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP8]], <4 x i32>* [[TMP15]], align 4
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 16
-; CHECK-NEXT:    [[TMP58:%.*]] = icmp eq i32 [[INDEX_NEXT]], 9984
-; CHECK-NEXT:    br i1 [[TMP58]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !24
+; CHECK-NEXT:    [[TMP16:%.*]] = icmp eq i32 [[INDEX_NEXT]], 9984
+; CHECK-NEXT:    br i1 [[TMP16]], label [[FOR_BODY:%.*]], label [[VECTOR_BODY]], !llvm.loop !24
 ; CHECK:       for.cond.cleanup:
 ; CHECK-NEXT:    ret void
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[A_ADDR_08:%.*]] = phi i32* [ [[ADD_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[I_07:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 9984, [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[B_ADDR_06:%.*]] = phi i32* [ [[INCDEC_PTR:%.*]], [[FOR_BODY]] ], [ [[IND_END3]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP59:%.*]] = load i32, i32* [[A_ADDR_08]], align 4
+; CHECK-NEXT:    [[TMP17:%.*]] = load i32, i32* [[A_ADDR_08]], align 4
 ; CHECK-NEXT:    [[ADD_PTR]] = getelementptr inbounds i32, i32* [[A_ADDR_08]], i32 6
-; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP59]], [[Y]]
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP17]], [[Y]]
 ; CHECK-NEXT:    store i32 [[ADD]], i32* [[B_ADDR_06]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR]] = getelementptr inbounds i32, i32* [[B_ADDR_06]], i32 1
 ; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_07]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 10000
 ; CHECK-NEXT:    br i1 [[EXITCOND]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY]], !llvm.loop !25
 ;
 entry:
   br label %for.body

 ; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ugt i8* [[SCEVGEP]], [[X]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
 ; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[FOR_BODY:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    [[IND_END:%.*]] = getelementptr i8, i8* [[X]], i32 3000
 ; CHECK-NEXT:    [[IND_END3:%.*]] = getelementptr i8, i8* [[Z]], i32 3000
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
+; CHECK-NEXT:    [[POINTER_PHI:%.*]] = phi i8* [ [[X]], [[VECTOR_PH]] ], [ [[TMP0:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[POINTER_PHI5:%.*]] = phi i8* [ [[Z]], [[VECTOR_PH]] ], [ [[TMP2:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP0:%.*]] = mul i32 [[INDEX]], 3
-; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i8, i8* [[X]], i32 [[TMP0]]
-; CHECK-NEXT:    [[TMP1:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], 3
-; CHECK-NEXT:    [[NEXT_GEP5:%.*]] = getelementptr i8, i8* [[X]], i32 [[TMP2]]
-; CHECK-NEXT:    [[TMP3:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP4:%.*]] = mul i32 [[TMP3]], 3
-; CHECK-NEXT:    [[NEXT_GEP6:%.*]] = getelementptr i8, i8* [[X]], i32 [[TMP4]]
-; CHECK-NEXT:    [[TMP5:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP6:%.*]] = mul i32 [[TMP5]], 3
-; CHECK-NEXT:    [[NEXT_GEP7:%.*]] = getelementptr i8, i8* [[X]], i32 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <4 x i8*> undef, i8* [[NEXT_GEP]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <4 x i8*> [[TMP7]], i8* [[NEXT_GEP5]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <4 x i8*> [[TMP8]], i8* [[NEXT_GEP6]], i32 2
-; CHECK-NEXT:    [[TMP10:%.*]] = insertelement <4 x i8*> [[TMP9]], i8* [[NEXT_GEP7]], i32 3
-; CHECK-NEXT:    [[TMP11:%.*]] = mul i32 [[INDEX]], 3
-; CHECK-NEXT:    [[NEXT_GEP8:%.*]] = getelementptr i8, i8* [[Z]], i32 [[TMP11]]
-; CHECK-NEXT:    [[TMP12:%.*]] = or i32 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP13:%.*]] = mul i32 [[TMP12]], 3
-; CHECK-NEXT:    [[NEXT_GEP9:%.*]] = getelementptr i8, i8* [[Z]], i32 [[TMP13]]
-; CHECK-NEXT:    [[TMP14:%.*]] = or i32 [[INDEX]], 2
-; CHECK-NEXT:    [[TMP15:%.*]] = mul i32 [[TMP14]], 3
-; CHECK-NEXT:    [[NEXT_GEP10:%.*]] = getelementptr i8, i8* [[Z]], i32 [[TMP15]]
-; CHECK-NEXT:    [[TMP16:%.*]] = or i32 [[INDEX]], 3
-; CHECK-NEXT:    [[TMP17:%.*]] = mul i32 [[TMP16]], 3
-; CHECK-NEXT:    [[NEXT_GEP11:%.*]] = getelementptr i8, i8* [[Z]], i32 [[TMP17]]
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <4 x i8*> undef, i8* [[NEXT_GEP8]], i32 0
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <4 x i8*> [[TMP18]], i8* [[NEXT_GEP9]], i32 1
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <4 x i8*> [[TMP19]], i8* [[NEXT_GEP10]], i32 2
-; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <4 x i8*> [[TMP20]], i8* [[NEXT_GEP11]], i32 3
-; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP10]], i32 1
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP10]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
-; CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP10]], i32 2
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER12:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP22]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
-; CHECK-NEXT:    [[WIDE_MASKED_GATHER13:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP23]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
-; CHECK-NEXT:    [[TMP24:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], <i8 10, i8 10, i8 10, i8 10>
-; CHECK-NEXT:    [[TMP25:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER12]]
-; CHECK-NEXT:    [[TMP26:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER13]]
-; CHECK-NEXT:    [[TMP27:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP21]], i32 1
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP24]], <4 x i8*> [[TMP21]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
-; CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP21]], i32 2
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP25]], <4 x i8*> [[TMP27]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
-; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP26]], <4 x i8*> [[TMP28]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
+; CHECK-NEXT:    [[TMP0]] = getelementptr i8, i8* [[POINTER_PHI]], i32 12
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr i8, i8* [[POINTER_PHI]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+; CHECK-NEXT:    [[TMP2]] = getelementptr i8, i8* [[POINTER_PHI5]], i32 12
+; CHECK-NEXT:    [[TMP3:%.*]] = getelementptr i8, i8* [[POINTER_PHI5]], <4 x i32> <i32 0, i32 3, i32 6, i32 9>
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP1]], i32 1
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP1]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP1]], i32 2
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER6:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP4]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
+; CHECK-NEXT:    [[WIDE_MASKED_GATHER7:%.*]] = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> [[TMP5]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef), !alias.scope !26
+; CHECK-NEXT:    [[TMP6:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], <i8 10, i8 10, i8 10, i8 10>
+; CHECK-NEXT:    [[TMP7:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER6]]
+; CHECK-NEXT:    [[TMP8:%.*]] = mul <4 x i8> [[WIDE_MASKED_GATHER]], [[WIDE_MASKED_GATHER7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP3]], i32 1
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP6]], <4 x i8*> [[TMP3]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i8, <4 x i8*> [[TMP3]], i32 2
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP7]], <4 x i8*> [[TMP9]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
+; CHECK-NEXT:    call void @llvm.masked.scatter.v4i8.v4p0i8(<4 x i8> [[TMP8]], <4 x i8*> [[TMP10]], i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>), !alias.scope !29, !noalias !26
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i32 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP29:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1000
-; CHECK-NEXT:    br i1 [[TMP29]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop !31
+; CHECK-NEXT:    [[TMP11:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1000
+; CHECK-NEXT:    br i1 [[TMP11]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop !31
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[X_ADDR_050:%.*]] = phi i8* [ [[INCDEC_PTR2:%.*]], [[FOR_BODY]] ], [ [[X]], [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[Z_ADDR_049:%.*]] = phi i8* [ [[INCDEC_PTR34:%.*]], [[FOR_BODY]] ], [ [[Z]], [[ENTRY]] ]
 ; CHECK-NEXT:    [[I_048:%.*]] = phi i32 [ [[INC:%.*]], [[FOR_BODY]] ], [ 0, [[ENTRY]] ]
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i8, i8* [[X_ADDR_050]], i32 1
-; CHECK-NEXT:    [[TMP30:%.*]] = load i8, i8* [[X_ADDR_050]], align 1
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* [[X_ADDR_050]], align 1
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i8, i8* [[X_ADDR_050]], i32 2
-; CHECK-NEXT:    [[TMP31:%.*]] = load i8, i8* [[INCDEC_PTR]], align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = load i8, i8* [[INCDEC_PTR]], align 1
 ; CHECK-NEXT:    [[INCDEC_PTR2]] = getelementptr inbounds i8, i8* [[X_ADDR_050]], i32 3
-; CHECK-NEXT:    [[TMP32:%.*]] = load i8, i8* [[INCDEC_PTR1]], align 1
-; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[TMP30]], 10
-; CHECK-NEXT:    [[MUL1:%.*]] = mul i8 [[TMP30]], [[TMP31]]
-; CHECK-NEXT:    [[MUL2:%.*]] = mul i8 [[TMP30]], [[TMP32]]
+; CHECK-NEXT:    [[TMP14:%.*]] = load i8, i8* [[INCDEC_PTR1]], align 1
+; CHECK-NEXT:    [[MUL:%.*]] = mul i8 [[TMP12]], 10
+; CHECK-NEXT:    [[MUL1:%.*]] = mul i8 [[TMP12]], [[TMP13]]
+; CHECK-NEXT:    [[MUL2:%.*]] = mul i8 [[TMP12]], [[TMP14]]
 ; CHECK-NEXT:    [[INCDEC_PTR32:%.*]] = getelementptr inbounds i8, i8* [[Z_ADDR_049]], i32 1
 ; CHECK-NEXT:    store i8 [[MUL]], i8* [[Z_ADDR_049]], align 1
 ; CHECK-NEXT:    [[INCDEC_PTR33:%.*]] = getelementptr inbounds i8, i8* [[Z_ADDR_049]], i32 2
 ; CHECK-NEXT:    store i8 [[MUL1]], i8* [[INCDEC_PTR32]], align 1
 ; CHECK-NEXT:    [[INCDEC_PTR34]] = getelementptr inbounds i8, i8* [[Z_ADDR_049]], i32 3
 ; CHECK-NEXT:    store i8 [[MUL2]], i8* [[INCDEC_PTR33]], align 1
 ; CHECK-NEXT:    [[INC]] = add nuw i32 [[I_048]], 1
 ; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i32 [[INC]], 1000