[LV] Don't mark pointers used by scalarized memory accesses uniform

[mssimpso]

Summary

Previously, all consecutive pointers were marked uniform after vectorization. However, if a consecutive pointer is used by a memory access that is eventually scalarized, the pointer may not remain uniform after all. An example is predicated stores. Even though a predicated store may be consecutive, it will still be scalarized, making it's pointer operand non-uniform.

This patch updates the logic in collectLoopUniforms to consider the cases where a memory access may be scalarized. If a memory access may be scalarized, its pointer operand is not marked uniform.

[mkuper]

Thanks, Matt!

lib/Transforms/Vectorize/LoopVectorize.cpp

5300

I think this will also return true for "reverse consecutive".
Is that true? If so, is it intentional?

5319

Any chance this and vectorizeMemoryInstruction can share code?
Having this duplicated scares me a bit.

[mssimpso]

lib/Transforms/Vectorize/LoopVectorize.cpp

5300

Good point. I'll check this out and add a test case, regardless.

5319

Sure, we can pull this out into a shared helper function.

5325–5327

Ideally, we could share this check with vectorizeMemoryInstruction as well. But the version in vectorizeMemoryInstruction uses the actual VF. We don't have the VF available to us yet in Legality. And I think the VF selection is to some extent affected by the uniforms. So the check here is essentially the same as the one in vectorizeMemoryInstruction, but with VF = 1. I'm wondering if it makes sense to replace the check in vectorizeMemoryInstruction with this one. Do you know of any cases that we would miss vectorizing?

[mssimpso]

Addressed Michael's comments. Thanks!

• Added a test case for reverse consecutive pointers. Reverse consecutive pointers and the pointers for reverse interleaved groups remain uniform.
• Added a shared helper for predicated stores
• Added a shared helper for irregular types (parameterized by VF)

[mssimpso]

Fixed a typo.

[mkuper]

lib/Transforms/Vectorize/LoopVectorize.cpp

5319

What I meant was that I would have preferred a single shared function that performs all of these checks, so that it's easier to keep memoryInstructionMayBeScalarized and vectorizeMemoryInstruction in sync.
(I didn't notice the VF issue, but you could pass VF as part of the interface to that shared helper).

5325–5327

I think in vectorizeMemoryInstruction you really want VF.

Let's say you have i20, and the ABI alignment for i20 is 1 byte.
Assuming I didn't get the math wrong:
For scalars, you'll have ScalarAllocatedSize == VectorElementSize == 3, so the type is regular.
But for VF=4, you'll DL.getTypeStoreSize(VectorTy) == 10, so VectorElementSize ==2, and the type is irregular.

Which is exactly what we want to happen - storing 4 consecutive i20s writes 12 bytes, not 10, so we can't vectorize the store.

5330

Now that I thought about this a bit more, checking this with "1" is probably not very useful. Maybe 2? (since, as discussed above, we don't have the final VF). Although I'm not 100% sure 2 makes sense either.

[mssimpso]

lib/Transforms/Vectorize/LoopVectorize.cpp

5319

Ah, I see. Yes, I think we can move all of this into a single function.

5325–5327

Oh I see. I think I was misinterpreting the difference between allocated size and store size. But aren't we really just wanting to know if there's padding been successive elements of a given type? That is, in your example if we have a <4 x i20>, we can't just store this type to memory without the 4 bits between each element.

What about DL.getTypeSizeInBits(ScalarType) != DL.getTypeAllocSizeInBits(ScalarType)? Using "type size" instead of "store size" should prevent rounding up to the next byte. For the i20 example, we would have 20 != 24, so the type is irregular.

[mssimpso]

lib/Transforms/Vectorize/LoopVectorize.cpp

5325–5327

What about DL.getTypeSizeInBits(ScalarType) != DL.getTypeAllocSizeInBits(ScalarType)?

After further digging, I think this is indeed equivalent to the existing code in vectorlizeMemoryInstruction. The reason is that DL.getTypeStoreSize(VectorType) is implemented as VF * DL.getTypeSizeInBits(ElementType) in DataLayout.h. So the VF's cancel. I'll update the patch accordingly.

[mssimpso]

Addressed Michael's comments. Thanks!

• Pulled out all scalarization checks from vectorizeMemoryInstruction into shared function.
• Updated irregular type check to remove VF as previously mentioned.
• Added some additional comments.

[mkuper]

This LGTM, except for the type size bit, which I'm still not sure I completely understand.

lib/Transforms/Vectorize/LoopVectorize.cpp

5342

I'm still not 100% convinced.

I mean, I'm fairly sure this is correct, I'm just not sure this doesn't miss any cases we would have caught before, where the sizes in bits don't match, but the rounded sizes do. (And to be honest, I'm not sure what the correct behavior is anymore)

To be more explicit originally we were comparing:
alignTo((getTypeSizeInBits(Ty) + 7) / 8, getABITypeAlignment(Ty))
to
(((VF * getTypeSizeInBits(Ty)) + 7) / 8) / VF

And now we're comparing:
alignTo((getTypeSizeInBits(Ty) + 7) / 8, getABITypeAlignment(Ty)) * 8
to
getTypeSizeInBits(Ty)

Are you absolutely sure these are equivalent (or the second one is "more correct")?

[mssimpso]

lib/Transforms/Vectorize/LoopVectorize.cpp

5342

Michael,

Thanks for digging deeper on this issue. I appreciate it! You've convinced me that these implementations aren't doing the same thing. Which one is "more correct" is unfortunately debatable at this point.

I think it helps somewhat to look at the original patch (rL180196, PR15758). The intent it seems was to ensure an array of type ScalarTy is "bitcast compatible" with a vector of type ScalarTy. It would seem the most straightforward implementation of that concept is:

VF * DL.getTypeAllocSize(ScalarTy) == DL.getTypeStoreSize(VectorTy)  // (1)

(The existing code moved the VF to the other side of the equation, but the floor division looks a little suspicious to me).

The equivalence of the two implementations (and whether we need the VF or not) boils down to whether vector types are packed or not. If we always assume vectors are packed, we can just check to see if there's padding in the array representation, and the VF is not needed. This is what I assumed was the case. However, it doesn't look like we've made a clear decision about the packed question yet (PR1784). The recent discussions on the mailing list about <N x i1> types seem related as well.

So I think the best thing to do here, at least from vectorizeMemoryInstruction, is to keep the type check as it is for now.

But since the check in collectLoopUniforms doesn't yet have the VF, I think we should be conservative and only mark an instruction uniform if we are sure it will be. The most conservative form of the type check is the one where we just check for padding in the array representation. This is because if:

DL.getTypeAllocSizeInBits(ScalarTy) == DL.getTypeSizeInBits(ScalarTy)  // (2)

The other equation must be true as well. So I think it makes sense to use (1) from vectorizeMemoryInstruction and getInstructionCost* and (2) from collectLoopUniforms. *I forgot to replace the scalarization checks in the cost model with the shared function.

I'll make these updates and post a new version of the patch. Thanks!

[mssimpso]

Implemented changes mentioned in my last comment.

• Restored previous type check when VF is known. We use a conservative estimate when VF is not known.
• Updated getInstructionCost to use the shared scalarization function.

[mkuper]

LGTM

[mssimpso]

Thanks, Michael!

lib/Transforms/Vectorize/LoopVectorize.cpp

 static Value *getPointerOperand(Value *I) {
   if (auto *LI = dyn_cast<LoadInst>(I))
     return LI->getPointerOperand();
   if (auto *SI = dyn_cast<StoreInst>(I))
     return SI->getPointerOperand();
   return nullptr;
 }
 
+/// A helper function that returns true if the given type is irregular. The
+/// type is irregular if its allocated size doesn't equal the store size of an
+/// element of the corresponding vector type at the given vectorization factor.
+static bool hasIrregularType(Type *Ty, const DataLayout &DL, unsigned VF) {
+
+  // Determine if an array of VF elements of type Ty is "bitcast compatible"
+  // with a <VF x Ty> vector.
+  if (VF > 1) {
+    auto *VectorTy = VectorType::get(Ty, VF);
+    return VF * DL.getTypeAllocSize(Ty) != DL.getTypeStoreSize(VectorTy);
+  }
+
+  // If the vectorization factor is one, we just check if an array of type Ty
+  // requires padding between elements.
+  return DL.getTypeAllocSizeInBits(Ty) != DL.getTypeSizeInBits(Ty);
+}
+
 /// InnerLoopVectorizer vectorizes loops which contain only one basic
 /// block to a specified vectorization factor (VF).
 /// This class performs the widening of scalars into vectors, or multiple
 /// scalars. This class also implements the following features:
 /// * It inserts an epilogue loop for handling loops that don't have iteration
 ///   counts that are known to be a multiple of the vectorization factor.
 /// * It handles the code generation for reduction variables.
 /// * Scalarization (implementation using scalars) of un-vectorizable
public:
 
   /// Returns true if vector representation of the instruction \p I
   /// requires mask.
   bool isMaskRequired(const Instruction *I) { return (MaskedOp.count(I) != 0); }
   unsigned getNumStores() const { return LAI->getNumStores(); }
   unsigned getNumLoads() const { return LAI->getNumLoads(); }
   unsigned getNumPredStores() const { return NumPredStores; }
 
+  /// Returns true if \p I is a store instruction in a predicated block that
+  /// will be scalarized during vectorization.
+  bool isPredicatedStore(Instruction *I);
+
+  /// Returns true if \p I is a memory instruction that has a consecutive or
+  /// consecutive-like pointer operand. Consecutive-like pointers are pointers
+  /// that are treated like consecutive pointers during vectorization. The
+  /// pointer operands of interleaved accesses are an example.
+  bool hasConsecutiveLikePtrOperand(Instruction *I);
+
+  /// Returns true if \p I is a memory instruction that must be scalarized
+  /// during vectorization.
+  bool memoryInstructionMustBeScalarized(Instruction *I, unsigned VF = 1);
+
 private:
   /// Check if a single basic block loop is vectorizable.
   /// At this point we know that this is a loop with a constant trip count
   /// and we only need to check individual instructions.
   bool canVectorizeInstrs();
 
   /// When we vectorize loops we may change the order in which
   /// we read and write from memory. This method checks if it is
void InnerLoopVectorizer::vectorizeMemoryInstruction(Instruction *Instr) {
   Value *Ptr = getPointerOperand(Instr);
   unsigned Alignment = LI ? LI->getAlignment() : SI->getAlignment();
   // An alignment of 0 means target abi alignment. We need to use the scalar's
   // target abi alignment in such a case.
   const DataLayout &DL = Instr->getModule()->getDataLayout();
   if (!Alignment)
     Alignment = DL.getABITypeAlignment(ScalarDataTy);
   unsigned AddressSpace = Ptr->getType()->getPointerAddressSpace();
-  uint64_t ScalarAllocatedSize = DL.getTypeAllocSize(ScalarDataTy);
-  uint64_t VectorElementSize = DL.getTypeStoreSize(DataTy) / VF;
 
-  if (SI && Legal->blockNeedsPredication(SI->getParent()) &&
-      !Legal->isMaskRequired(SI))
-    return scalarizeInstruction(Instr, true);
-
-  if (ScalarAllocatedSize != VectorElementSize)
-    return scalarizeInstruction(Instr);
-
-  // If the pointer is loop invariant scalarize the load.
-  if (LI && Legal->isUniform(Ptr))
-    return scalarizeInstruction(Instr);
+  // Scalarize the memory instruction if necessary.
+  if (Legal->memoryInstructionMustBeScalarized(Instr, VF))
+    return scalarizeInstruction(Instr, Legal->isPredicatedStore(Instr));
 
-  // If the pointer is non-consecutive and gather/scatter is not supported
-  // scalarize the instruction.
+  // Determine if the pointer operand of the access is either consecutive or
+  // reverse consecutive.
   int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);
   bool Reverse = ConsecutiveStride < 0;
-  bool CreateGatherScatter =
-      !ConsecutiveStride && ((LI && Legal->isLegalMaskedGather(ScalarDataTy)) ||
-                             (SI && Legal->isLegalMaskedScatter(ScalarDataTy)));
 
-  if (!ConsecutiveStride && !CreateGatherScatter)
-    return scalarizeInstruction(Instr);
+  // Determine if either a gather or scatter operation is legal.
+  bool CreateGatherScatter =
+      !ConsecutiveStride && Legal->isLegalGatherOrScatter(Instr);
 
   VectorParts VectorGep;
 
   // Handle consecutive loads/stores.
   GetElementPtrInst *Gep = getGEPInstruction(Ptr);
   if (ConsecutiveStride) {
     if (Gep && Legal->isInductionVariable(Gep->getPointerOperand())) {
       setDebugLocFromInst(Builder, Gep);
if (!ScalarIndUpdate)
       continue;
 
     // The induction variable and its update instruction will remain scalar.
     Scalars.insert(Ind);
     Scalars.insert(IndUpdate);
   }
 }
 
+bool LoopVectorizationLegality::hasConsecutiveLikePtrOperand(Instruction *I) {
+  if (isAccessInterleaved(I))
+    return true;
+  if (auto *Ptr = getPointerOperand(I))
+    return isConsecutivePtr(Ptr);
+  return false;
+}
+
+bool LoopVectorizationLegality::isPredicatedStore(Instruction *I) {
+  auto *SI = dyn_cast<StoreInst>(I);
+  return SI && blockNeedsPredication(SI->getParent()) && !isMaskRequired(SI);
+}
+
+bool LoopVectorizationLegality::memoryInstructionMustBeScalarized(
+    Instruction *I, unsigned VF) {
+
+  // If the memory instruction is in an interleaved group, it will be
+  // vectorized and its pointer will remain uniform.
+  if (isAccessInterleaved(I))
+    return false;
+
+  // Get and ensure we have a valid memory instruction.
+  LoadInst *LI = dyn_cast<LoadInst>(I);
+  StoreInst *SI = dyn_cast<StoreInst>(I);
+  assert((LI || SI) && "Invalid memory instruction");
+
+  // If the pointer operand is uniform (loop invariant), the memory instruction
+  // will be scalarized.
+  auto *Ptr = getPointerOperand(I);
+  if (LI && isUniform(Ptr))
+    return true;
+
+  // If the pointer operand is non-consecutive and neither a gather nor a
+  // scatter operation is legal, the memory instruction will be scalarized.
+  if (!isConsecutivePtr(Ptr) && !isLegalGatherOrScatter(I))
+    return true;
+
+  // If the instruction is a store located in a predicated block, it will be
+  // scalarized.
+  if (isPredicatedStore(I))
+    return true;
+
+  // If the instruction's allocated size doesn't equal it's type size, it
+  // requires padding and will be scalarized.
+  auto &DL = I->getModule()->getDataLayout();
+  auto *ScalarTy = LI ? LI->getType() : SI->getValueOperand()->getType();
+  if (hasIrregularType(ScalarTy, DL, VF))
+    return true;
+
+  // Otherwise, the memory instruction should be vectorized if the rest of the
+  // loop is.
+  return false;
+}
+
 void LoopVectorizationLegality::collectLoopUniforms() {
   // We now know that the loop is vectorizable!
   // Collect instructions inside the loop that will remain uniform after
   // vectorization.
 
   // Global values, params and instructions outside of current loop are out of
   // scope.
   auto isOutOfScope = [&](Value *V) -> bool {
     Instruction *I = dyn_cast<Instruction>(V);
     return (!I || !TheLoop->contains(I));
   };
 
   SetVector<Instruction *> Worklist;
   BasicBlock *Latch = TheLoop->getLoopLatch();
   // Start with the conditional branch.
   if (!isOutOfScope(Latch->getTerminator()->getOperand(0))) {
     Instruction *Cmp = cast<Instruction>(Latch->getTerminator()->getOperand(0));
     Worklist.insert(Cmp);
     DEBUG(dbgs() << "LV: Found uniform instruction: " << *Cmp << "\n");
   }
 
-  // Add all consecutive pointer values; these values will be uniform after
-  // vectorization (and subsequent cleanup). Although non-consecutive, we also
-  // add the pointer operands of interleaved accesses since they are treated
-  // like consecutive pointers during vectorization.
+  // Holds consecutive and consecutive-like pointers. Consecutive-like pointers
+  // are pointers that are treated like consecutive pointers during
+  // vectorization. The pointer operands of interleaved accesses are an
+  // example.
+  SmallPtrSet<Instruction *, 8> ConsecutiveLikePtrs;
+
+  // Holds pointer operands of instructions that are possibly non-uniform.
+  SmallPtrSet<Instruction *, 8> PossibleNonUniformPtrs;
+
+  // Iterate over the instructions in the loop, and collect all
+  // consecutive-like pointer operands in ConsecutiveLikePtrs. If it's possible
+  // that a consecutive-like pointer operand will be scalarized, we collect it
+  // in PossibleNonUniformPtrs instead. We use two sets here because a single
+  // getelementptr instruction can be used by both vectorized and scalarized
+  // memory instructions. For example, if a loop loads and stores from the same
+  // location, but the store is conditional, the store will be scalarized, and
+  // the getelementptr won't remain uniform.
   for (auto *BB : TheLoop->blocks())
     for (auto &I : *BB) {
-      Instruction *Ptr = nullptr;
-      if (I.getType()->isPointerTy() && isConsecutivePtr(&I))
-        Ptr = &I;
-      else if (isAccessInterleaved(&I))
-        Ptr = cast<Instruction>(getPointerOperand(&I));
-      else
+
+      // If the pointer operand is not consecutive-like, there's nothing to do.
+      auto *Ptr = dyn_cast_or_null<Instruction>(getPointerOperand(&I));
+      if (!Ptr || isUniform(Ptr) || !hasConsecutiveLikePtrOperand(&I))
         continue;
-      Worklist.insert(Ptr);
-      DEBUG(dbgs() << "LV: Found uniform instruction: " << *Ptr << "\n");
+
+      // Ensure the memory instruction will not be scalarized, making its
+      // pointer operand non-uniform.
+      if (memoryInstructionMustBeScalarized(&I))
+        PossibleNonUniformPtrs.insert(Ptr);
+
+      // If the memory instruction will be vectorized, its consecutive-like
+      // pointer operand should remain uniform.
+      else
+        ConsecutiveLikePtrs.insert(Ptr);
+    }
+
+  // Add to the Worklist all consecutive and consecutive-like pointers that
+  // aren't also identified as possibly non-uniform.
+  for (auto *V : ConsecutiveLikePtrs)
+    if (!PossibleNonUniformPtrs.count(V)) {
+      DEBUG(dbgs() << "LV: Found uniform instruction: " << *V << "\n");
+      Worklist.insert(V);
     }
 
   // Expand Worklist in topological order: whenever a new instruction
   // is added , its users should be either already inside Worklist, or
   // out of scope. It ensures a uniform instruction will only be used
   // by uniform instructions or out of scope instructions.
   unsigned idx = 0;
   while (idx != Worklist.size()) {
if (Legal->isAccessInterleaved(I)) {
             TTI.getShuffleCost(TargetTransformInfo::SK_Reverse, VectorTy, 0);
 
       // FIXME: The interleaved load group with a huge gap could be even more
       // expensive than scalar operations. Then we could ignore such group and
       // use scalar operations instead.
       return Cost;
     }
 
-    // Scalarized loads/stores.
-    int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);
-    bool UseGatherOrScatter =
-        (ConsecutiveStride == 0) && Legal->isLegalGatherOrScatter(I);
-
-    bool Reverse = ConsecutiveStride < 0;
-    const DataLayout &DL = I->getModule()->getDataLayout();
-    uint64_t ScalarAllocatedSize = DL.getTypeAllocSize(ValTy);
-    uint64_t VectorElementSize = DL.getTypeStoreSize(VectorTy) / VF;
-    if ((!ConsecutiveStride && !UseGatherOrScatter) ||
-        ScalarAllocatedSize != VectorElementSize) {
+    // Check if the memory instruction will be scalarized.
+    if (Legal->memoryInstructionMustBeScalarized(I, VF)) {
       bool IsComplexComputation =
           isLikelyComplexAddressComputation(Ptr, Legal, SE, TheLoop);
       unsigned Cost = 0;
       // The cost of extracting from the value vector and pointer vector.
       Type *PtrTy = ToVectorTy(Ptr->getType(), VF);
       for (unsigned i = 0; i < VF; ++i) {
         //  The cost of extracting the pointer operand.
         Cost += TTI.getVectorInstrCost(Instruction::ExtractElement, PtrTy, i);
         // In case of STORE, the cost of ExtractElement from the vector.
         // In case of LOAD, the cost of InsertElement into the returned
         // vector.
         Cost += TTI.getVectorInstrCost(SI ? Instruction::ExtractElement
                                           : Instruction::InsertElement,
                                        VectorTy, i);
       }
 
       // The cost of the scalar loads/stores.
       Cost += VF * TTI.getAddressComputationCost(PtrTy, IsComplexComputation);
       Cost += VF *
               TTI.getMemoryOpCost(I->getOpcode(), ValTy->getScalarType(),
                                   Alignment, AS);
       return Cost;
     }
 
+    // Determine if the pointer operand of the access is either consecutive or
+    // reverse consecutive.
+    int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);
+    bool Reverse = ConsecutiveStride < 0;
+
+    // Determine if either a gather or scatter operation is legal.
+    bool UseGatherOrScatter =
+        !ConsecutiveStride && Legal->isLegalGatherOrScatter(I);
+
     unsigned Cost = TTI.getAddressComputationCost(VectorTy);
     if (UseGatherOrScatter) {
       assert(ConsecutiveStride == 0 &&
              "Gather/Scatter are not used for consecutive stride");
       return Cost +
              TTI.getGatherScatterOpCost(I->getOpcode(), VectorTy, Ptr,
                                         Legal->isMaskRequired(I), Alignment);
     }

test/Transforms/LoopVectorize/consecutive-ptr-uniforms.ll

+; REQUIRES: asserts
+; RUN: opt < %s -loop-vectorize -force-vector-width=4 -force-vector-interleave=1 -instcombine -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s
+; RUN: opt < %s -loop-vectorize -force-vector-width=4 -force-vector-interleave=1 -enable-interleaved-mem-accesses -enable-cond-stores-vec -instcombine -debug-only=loop-vectorize -disable-output -print-after=instcombine 2>&1 | FileCheck %s --check-prefix=INTER
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+
+%pair = type { i32, i32 }
+
+; CHECK-LABEL: consecutive_ptr_forward
+;
+; Check that a forward consecutive pointer is recognized as uniform and remains
+; uniform after vectorization.
+;
+; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+; CHECK:     vector.body
+; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK-NOT:   getelementptr
+; CHECK:       getelementptr inbounds i32, i32* %a, i64 %index
+; CHECK-NOT:   getelementptr
+; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define i32 @consecutive_ptr_forward(i32* %a, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
+  %tmp0 = phi i32 [ %tmp3, %for.body ], [ 0, %entry ]
+  %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+  %tmp2 = load i32, i32* %tmp1, align 8
+  %tmp3 = add i32 %tmp0, %tmp2
+  %i.next = add nuw nsw i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  %tmp4 = phi i32 [ %tmp3, %for.body ]
+  ret i32 %tmp4
+}
+
+; CHECK-LABEL: consecutive_ptr_reverse
+;
+; Check that a reverse consecutive pointer is recognized as uniform and remains
+; uniform after vectorization.
+;
+; CHECK:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+; CHECK:     vector.body
+; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK:       %offset.idx = sub i64 %n, %index
+; CHECK-NOT:   getelementptr
+; CHECK:       %[[G0:.+]] = getelementptr inbounds i32, i32* %a, i64 %offset.idx
+; CHECK:       getelementptr i32, i32* %[[G0]], i64 -3
+; CHECK-NOT:   getelementptr
+; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define i32 @consecutive_ptr_reverse(i32* %a, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i = phi i64 [ %i.next, %for.body ], [ %n, %entry ]
+  %tmp0 = phi i32 [ %tmp3, %for.body ], [ 0, %entry ]
+  %tmp1 = getelementptr inbounds i32, i32* %a, i64 %i
+  %tmp2 = load i32, i32* %tmp1, align 8
+  %tmp3 = add i32 %tmp0, %tmp2
+  %i.next = add nuw nsw i64 %i, -1
+  %cond = icmp sgt i64 %i.next, 0
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  %tmp4 = phi i32 [ %tmp3, %for.body ]
+  ret i32 %tmp4
+}
+; CHECK-LABEL: interleaved_access_forward
+;
+; Check that a consecutive-like pointer used by a forward interleaved group is
+; recognized as uniform and remains uniform after vectorization. When
+; interleaved memory accesses aren't enabled, the pointer should not be
+; recognized as uniform, and it should not be uniform after vectorization.
+;
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; CHECK:     vector.body
+; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK:       %[[I1:.+]] = or i64 %index, 1
+; CHECK:       %[[I2:.+]] = or i64 %index, 2
+; CHECK:       %[[I3:.+]] = or i64 %index, 3
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 1
+; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; INTER:     vector.body
+; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; INTER-NOT:   getelementptr
+; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
+; INTER-NOT:   getelementptr
+; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define i32 @interleaved_access_forward(%pair* %p, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i = phi i64 [ %i.next, %for.body ], [ 0, %entry ]
+  %tmp0 = phi i32 [ %tmp6, %for.body ], [ 0, %entry ]
+  %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+  %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+  %tmp3 = load i32, i32* %tmp1, align 8
+  %tmp4 = load i32, i32* %tmp2, align 8
+  %tmp5 = add i32 %tmp3, %tmp4
+  %tmp6 = add i32 %tmp0, %tmp5
+  %i.next = add nuw nsw i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  %tmp14 = phi i32 [ %tmp6, %for.body ]
+  ret i32 %tmp14
+}
+
+; CHECK-LABEL: interleaved_access_reverse
+;
+; Check that a consecutive-like pointer used by a reverse interleaved group is
+; recognized as uniform and remains uniform after vectorization. When
+; interleaved memory accesses aren't enabled, the pointer should not be
+; recognized as uniform, and it should not be uniform after vectorization.
+;
+; recognized as uniform, and it should not be uniform after vectorization.
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; CHECK-NOT: LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; CHECK:     vector.body
+; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK:       %offset.idx = sub i64 %n, %index
+; CHECK:       %[[I1:.+]] = add i64 %offset.idx, -1
+; CHECK:       %[[I2:.+]] = add i64 %offset.idx, -2
+; CHECK:       %[[I3:.+]] = add i64 %offset.idx, -3
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 1
+; CHECK:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 1
+; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+; INTER:     LV: Found uniform instruction: %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; INTER:     LV: Found uniform instruction: %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+; INTER:     vector.body
+; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; INTER:       %offset.idx = sub i64 %n, %index
+; INTER-NOT:   getelementptr
+; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, %pair* %p, i64 %offset.idx, i32 0
+; INTER:       getelementptr i32, i32* %[[G0]], i64 -6
+; INTER-NOT:   getelementptr
+; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define i32 @interleaved_access_reverse(%pair* %p, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i = phi i64 [ %i.next, %for.body ], [ %n, %entry ]
+  %tmp0 = phi i32 [ %tmp6, %for.body ], [ 0, %entry ]
+  %tmp1 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+  %tmp2 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 1
+  %tmp3 = load i32, i32* %tmp1, align 8
+  %tmp4 = load i32, i32* %tmp2, align 8
+  %tmp5 = add i32 %tmp3, %tmp4
+  %tmp6 = add i32 %tmp0, %tmp5
+  %i.next = add nuw nsw i64 %i, -1
+  %cond = icmp sgt i64 %i.next, 0
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  %tmp14 = phi i32 [ %tmp6, %for.body ]
+  ret i32 %tmp14
+}
+
+; CHECK-LABEL: predicated_store
+;
+; Check that a consecutive-like pointer used by a forward interleaved group and
+; scalarized store is not recognized as uniform and is not uniform after
+; vectorization. The store is scalarized because it's in a predicated block.
+; Even though the load in this example is vectorized and only uses the pointer
+; as if it were uniform, the store is scalarized, making the pointer
+; non-uniform.
+;
+; INTER-NOT: LV: Found uniform instruction: %tmp0 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+; INTER:     vector.body
+; INTER:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, {{.*}} ]
+; INTER:       %[[I1:.+]] = or i64 %index, 1
+; INTER:       %[[I2:.+]] = or i64 %index, 2
+; INTER:       %[[I3:.+]] = or i64 %index, 3
+; INTER:       %[[G0:.+]] = getelementptr inbounds %pair, %pair* %p, i64 %index, i32 0
+; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I1]], i32 0
+; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I2]], i32 0
+; INTER:       getelementptr inbounds %pair, %pair* %p, i64 %[[I3]], i32 0
+; INTER:       %[[B0:.+]] = bitcast i32* %[[G0]] to <8 x i32>*
+; INTER:       %wide.vec = load <8 x i32>, <8 x i32>* %[[B0]], align 8
+; INTER:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define void @predicated_store(%pair *%p, i32 %x, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i  = phi i64 [ %i.next, %if.merge ], [ 0, %entry ]
+  %tmp0 = getelementptr inbounds %pair, %pair* %p, i64 %i, i32 0
+  %tmp1 = load i32, i32* %tmp0, align 8
+  %tmp2 = icmp eq i32 %tmp1, %x
+  br i1 %tmp2, label %if.then, label %if.merge
+
+if.then:
+  store i32 %tmp1, i32* %tmp0, align 8
+  br label %if.merge
+
+if.merge:
+  %i.next = add nuw nsw i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  ret void
+}
+
+; CHECK-LABEL: irregular_type
+;
+; Check that a consecutive pointer used by a scalarized store is not recognized
+; as uniform and is not uniform after vectorization. The store is scalarized
+; because the stored type may required padding.
+;
+; CHECK-NOT: LV: Found uniform instruction: %tmp1 = getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %i
+; CHECK:     vector.body
+; CHECK:       %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+; CHECK:       %[[I1:.+]] = or i64 %index, 1
+; CHECK:       %[[I2:.+]] = or i64 %index, 2
+; CHECK:       %[[I3:.+]] = or i64 %index, 3
+; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %index
+; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I1]]
+; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I2]]
+; CHECK:       getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %[[I3]]
+; CHECK:       br i1 {{.*}}, label %middle.block, label %vector.body
+;
+define void @irregular_type(x86_fp80* %a, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
+  %tmp0 = sitofp i32 1 to x86_fp80
+  %tmp1 = getelementptr inbounds x86_fp80, x86_fp80* %a, i64 %i
+  store x86_fp80 %tmp0, x86_fp80* %tmp1, align 16
+  %i.next = add i64 %i, 1
+  %cond = icmp slt i64 %i.next, %n
+  br i1 %cond, label %for.body, label %for.end
+
+for.end:
+  ret void
+}