[TTI/CostModel] improve TTI::getGEPCost and use it in CostModel::getInstructionCost

[jingyue]

Summary

This patch updates TargetTransformInfoImplCRTPBase::getGEPCost to consider
addressing modes. It now returns TCC_Free when the GEP can be completely folded
to an addresing mode.

I started this patch as I refactored SLSR. Function isGEPFoldable looks common
and is indeed used by some WIP of mine. So I extracted that logic to getGEPCost.

Furthermore, I noticed getGEPCost wasn't directly tested anywhere. The best
testing bed seems CostModel, but its getInstructionCost method invokes
getAddressComputationCost for GEPs which provides very coarse estimation. So
this patch also makes getInstructionCost call the updated getGEPCost for GEPs.
This change inevitably breaks some tests because the cost model changes, but
nothing looks seriously wrong -- if we believe the new cost model is the right
way to go, these tests should be updated.

This patch is not perfect yet -- the comments in some tests need to be updated.
I want to know whether this is a right approach before fixing those details.

[hfinkel]

include/llvm/Analysis/TargetTransformInfo.h

139

I see no reason to change the interface here. We can localize the const_casts to the implementation, instead of forcing them into all callers that have const references/pointers.

include/llvm/Analysis/TargetTransformInfoImpl.h

392

You should call stripPointerCasts before doing the dyn_cast.

392

Should be dyn_cast_or_null b/c the cost model should not require the base pointer be known when estimating costs.

lib/Analysis/CostModel.cpp

386

You don't need the { } here any longer.

[jingyue]

include/llvm/Analysis/TargetTransformInfoImpl.h

392

Interesting -- when Ptr == nullptr, what is a good estimation? Fall back to using getAddressComputationCost?

[hfinkel]

include/llvm/Analysis/TargetTransformInfoImpl.h

392

No, but there is another issue here. We're currently moving to give pointer an opaque type, and so you'll need to separately require a Type* on this interface. Please add that. Once you have the Type *, you should:

1. assert that the type is equal to the pointer element type of the provided pointer (if it is provided) -- we'll remove this once we actually get rid of pointer element types, but for now we need the asserts
2. use that type in the code below (there is a getGEPReturnType overload that takes the element type instead of the pointer itself)
3. Make some default assumption for HasBaseReg (I think you should assume true).

[jingyue]

Makes sense. Thank you!

[jingyue]

Some changes

1. strip pointer casts before checking whether Ptr is a GlobalValue

2. support Ptr == nullptr

3. getGEPCost still takes ArrayRef<const Value *> as operands

4. in the direction of supporting opaque pointer types, getGEPCost takes the source pointer element type as a parameter.

5. fixed partial.ll. With the new getGEPCost, the loop size in @test1 is 13 and thus the loop can be partially unrolled. Added extra instructions so that the loop size remains 16.

[jingyue]

test/Transforms/LoopVectorize/X86/metadata-enable.ll

63

After the change to getGEPCost, the loop size is reduced. Bump the trip count so that the loops still cannot be fully unrolled.

[hfinkel]

include/llvm/Analysis/TargetTransformInfoImpl.h

395

pointer can have -> pointers have

408

I don't understand why you're doing this. Will gep_type_begin not compile with an ArrayRef<const Value *>? If so, we should just fix that.

411

That's not right. isLegalAddressingMode takes the address space as an optional last parameter. Please propagate it.

[jingyue]

Two changes

1. pass Ptr's address space to isLegalAddressingMode
2. allow gep_type_begin to take ArrayRef<const Value *>

[jingyue]

Some new changes in this patch are due to "git rebase".

[hfinkel]

LGTM.

include/llvm/Analysis/TargetTransformInfoImpl.h

395

a pointer can have -> pointers have

(when we make the change, there will be no 'can' about it, they'll have to have an opaque type).

[jingyue]

updated comments

[hfinkel]

LGTM, thanks!

[haicheng]

Hi Jingyue and Hal,

I believe this patch has a mistake in calling isLegalAddressingMode(). Please check the inlined the comments.

Thank you,

Haicheng Wu

include/llvm/Analysis/TargetTransformInfoImpl.h

430

I think we should use *GTI here rather than PointerType::get(*GTI, AS).

For example, on AArch64, the GEP in the following IRs can be folded

%"class.boost::array.16" = type { [24 x i32] }
%arrayidx.i17 = getelementptr inbounds %"class.boost::array.16", %"class.boost::array.16"* %moves, i64 0, i32 0, i64 %conv7                                                                               
store i32 %add, i32* %arrayidx.i17, align 4, !tbaa !18

The assembly is simple as

str     w24, [x20, x8, lsl #2]

However, isLegalAddressingMode() here returns false because we pass a wrong type.

I checked the implementation of isLegalAddressingMode() in several different backends (including ARM) and several other passes (LoopStrengthReduce and CodeGenPrepare) that call isLegalAddressingMode(). It think I am correct.

[jingyue]

include/llvm/Analysis/TargetTransformInfoImpl.h

430

Thanks! I think you are right. Can you create a patch and add a unit test for this?

[haicheng]

Thank you. I am working on this.

include/llvm/Analysis/TargetTransformInfo.h

public:
   unsigned getOperationCost(unsigned Opcode, Type *Ty,
                             Type *OpTy = nullptr) const;
 
   /// \brief Estimate the cost of a GEP operation when lowered.
   ///
   /// The contract for this function is the same as \c getOperationCost except
   /// that it supports an interface that provides extra information specific to
   /// the GEP operation.
-  unsigned getGEPCost(const Value *Ptr, ArrayRef<const Value *> Operands) const;
+  unsigned getGEPCost(const Value *Ptr, ArrayRef<Value *> Operands) const;
 
   /// \brief Estimate the cost of a function call when lowered.
   ///
   /// The contract for this is the same as \c getOperationCost except that it
   /// supports an interface that provides extra information specific to call
   /// instructions.
   ///
   /// This is the most basic query for estimating call cost: it only knows the
private:
   std::unique_ptr<Concept> TTIImpl;
 };
 
 class TargetTransformInfo::Concept {
 public:
   virtual ~Concept() = 0;
 
   virtual unsigned getOperationCost(unsigned Opcode, Type *Ty, Type *OpTy) = 0;
-  virtual unsigned getGEPCost(const Value *Ptr,
-                              ArrayRef<const Value *> Operands) = 0;
+  virtual unsigned getGEPCost(const Value *Ptr, ArrayRef<Value *> Operands) = 0;
   virtual unsigned getCallCost(FunctionType *FTy, int NumArgs) = 0;
   virtual unsigned getCallCost(const Function *F, int NumArgs) = 0;
   virtual unsigned getCallCost(const Function *F,
                                ArrayRef<const Value *> Arguments) = 0;
   virtual unsigned getIntrinsicCost(Intrinsic::ID IID, Type *RetTy,
                                     ArrayRef<Type *> ParamTys) = 0;
   virtual unsigned getIntrinsicCost(Intrinsic::ID IID, Type *RetTy,
                                     ArrayRef<const Value *> Arguments) = 0;

 
 public:
   Model(T Impl) : Impl(std::move(Impl)) {}
   ~Model() override {}
 
   unsigned getOperationCost(unsigned Opcode, Type *Ty, Type *OpTy) override {
     return Impl.getOperationCost(Opcode, Ty, OpTy);
   }
-  unsigned getGEPCost(const Value *Ptr,
-                      ArrayRef<const Value *> Operands) override {
+  unsigned getGEPCost(const Value *Ptr, ArrayRef<Value *> Operands) override {
     return Impl.getGEPCost(Ptr, Operands);
   }
   unsigned getCallCost(FunctionType *FTy, int NumArgs) override {
     return Impl.getCallCost(FTy, NumArgs);
   }
   unsigned getCallCost(const Function *F, int NumArgs) override {
     return Impl.getCallCost(F, NumArgs);
   }

include/llvm/Analysis/TargetTransformInfoImpl.h

case Instruction::Trunc:
       // shift-right of the same width).
       if (DL && DL->isLegalInteger(DL->getTypeSizeInBits(Ty)))
         return TTI::TCC_Free;
 
       return TTI::TCC_Basic;
     }
   }
 
-  unsigned getGEPCost(const Value *Ptr, ArrayRef<const Value *> Operands) {
+  unsigned getGEPCost(const Value *Ptr, ArrayRef<Value *> Operands) {
     // In the basic model, we just assume that all-constant GEPs will be folded
     // into their uses via addressing modes.
     for (unsigned Idx = 0, Size = Operands.size(); Idx != Size; ++Idx)
       if (!isa<Constant>(Operands[Idx]))
         return TTI::TCC_Basic;
 
     return TTI::TCC_Free;
   }
public:
 
   unsigned getCallCost(const Function *F, ArrayRef<const Value *> Arguments) {
     // Simply delegate to generic handling of the call.
     // FIXME: We should use instsimplify or something else to catch calls which
     // will constant fold with these arguments.
     return static_cast<T *>(this)->getCallCost(F, Arguments.size());
   }
 
+  using BaseT::getGEPCost;
+
+  unsigned getGEPCost(const Value *Ptr, ArrayRef<Value *> Operands) {
+    const GlobalValue *BaseGV = dyn_cast<GlobalValue>(Ptr);
+    bool HasBaseReg = (BaseGV == nullptr);
+    int64_t BaseOffset = 0;
+    int64_t Scale = 0;
+
+    generic_gep_type_iterator<Value *const *> GTI =
+        gep_type_begin(Ptr->getType()->getScalarType(), Operands);
+    for (auto I = Operands.begin(); I != Operands.end(); ++I, ++GTI) {
+      if (isa<SequentialType>(*GTI)) {
+        int64_t ElementSize = DL->getTypeAllocSize(GTI.getIndexedType());
+        if (const ConstantInt *ConstIdx = dyn_cast<ConstantInt>(*I)) {
+          BaseOffset += ConstIdx->getSExtValue() * ElementSize;
+        } else {
+          // Needs scale register.
+          if (Scale != 0) {
+            // No addressing mode takes two scale registers.
+            return TTI::TCC_Basic;
+          }
+          Scale = ElementSize;
+        }
+      } else {
+        StructType *STy = cast<StructType>(*GTI);
+        uint64_t Field = cast<ConstantInt>(*I)->getZExtValue();
+        BaseOffset += DL->getStructLayout(STy)->getElementOffset(Field);
+      }
+    }
+
+    Type *GEPReturnTy =
+        GetElementPtrInst::getGEPReturnType(const_cast<Value *>(Ptr), Operands);
+    if (static_cast<T *>(this)->isLegalAddressingMode(
+            GEPReturnTy, const_cast<GlobalValue *>(BaseGV), BaseOffset,
+            HasBaseReg, Scale)) {
+      return TTI::TCC_Free;
+    }
+    return TTI::TCC_Basic;
+  }
+
   using BaseT::getIntrinsicCost;
 
   unsigned getIntrinsicCost(Intrinsic::ID IID, Type *RetTy,
                             ArrayRef<const Value *> Arguments) {
     // Delegate to the generic intrinsic handling code. This mostly provides an
     // opportunity for targets to (for example) special case the cost of
     // certain intrinsics based on constants used as arguments.
     SmallVector<Type *, 8> ParamTys;
     ParamTys.reserve(Arguments.size());
     for (unsigned Idx = 0, Size = Arguments.size(); Idx != Size; ++Idx)
       ParamTys.push_back(Arguments[Idx]->getType());
     return static_cast<T *>(this)->getIntrinsicCost(IID, RetTy, ParamTys);
   }
 
   unsigned getUserCost(const User *U) {
     if (isa<PHINode>(U))
       return TTI::TCC_Free; // Model all PHI nodes as free.
 
     if (const GEPOperator *GEP = dyn_cast<GEPOperator>(U)) {
-      SmallVector<const Value *, 4> Indices(GEP->idx_begin(), GEP->idx_end());
+      SmallVector<Value *, 4> Indices(GEP->idx_begin(), GEP->idx_end());
       return static_cast<T *>(this)
           ->getGEPCost(GEP->getPointerOperand(), Indices);
     }
 
     if (auto CS = ImmutableCallSite(U)) {
       const Function *F = CS.getCalledFunction();
       if (!F) {
         // Just use the called value type.

lib/Analysis/CostModel.cpp

static bool matchVectorSplittingReduction(const ExtractElementInst *ReduxRoot,
   return true;
 }
 
 unsigned CostModelAnalysis::getInstructionCost(const Instruction *I) const {
   if (!TTI)
     return -1;
 
   switch (I->getOpcode()) {
   case Instruction::GetElementPtr:{
-    Type *ValTy = I->getOperand(0)->getType()->getPointerElementType();
-    return TTI->getAddressComputationCost(ValTy);
+    return TTI->getUserCost(I);
   }
 
   case Instruction::Ret:
   case Instruction::PHI:
   case Instruction::Br: {
     return TTI->getCFInstrCost(I->getOpcode());
   }
   case Instruction::Add:

lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

return (Basis.Ins != C.Ins && // skip the same instruction
           // Basis must dominate C in order to rewrite C with respect to Basis.
           DT->dominates(Basis.Ins->getParent(), C.Ins->getParent()) &&
           // They share the same base, stride, and candidate kind.
           Basis.Base == C.Base &&
           Basis.Stride == C.Stride &&
           Basis.CandidateKind == C.CandidateKind);
 }
 
+// TODO: use TTI->getGEPCost.
 static bool isGEPFoldable(GetElementPtrInst *GEP,
                           const TargetTransformInfo *TTI,
                           const DataLayout *DL) {
   GlobalVariable *BaseGV = nullptr;
   int64_t BaseOffset = 0;
   bool HasBaseReg = false;
   int64_t Scale = 0;
 

test/Analysis/CostModel/ARM/gep.ll

 ; RUN: opt -cost-model -analyze -mtriple=thumbv7-apple-ios6.0.0 -mcpu=swift < %s | FileCheck %s
 
 target datalayout = "e-p:32:32:32-i1:8:32-i8:8:32-i16:16:32-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:32:64-v128:32:128-a0:0:32-n32-S32"
 target triple = "thumbv7-apple-ios6.0.0"
 
-define void @test_geps() {
-  ; Cost of scalar integer geps should be one. We can't always expect it to be
-  ; folded into the instruction addressing mode.
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i8, i8*
+define void @test_geps(i32 %i) {
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i8, i8*
   %a0 = getelementptr inbounds i8, i8* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i16, i16*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i16, i16*
   %a1 = getelementptr inbounds i16, i16* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i32, i32*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i32, i32*
   %a2 = getelementptr inbounds i32, i32* undef, i32 0
-
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i64, i64*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i64, i64*
   %a3 = getelementptr inbounds i64, i64* undef, i32 0
 
   ; Cost of scalar floating point geps should be one. We cannot fold the address
   ; computation.
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds float, float*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds float, float*
   %a4 = getelementptr inbounds float, float* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds double, double*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds double, double*
   %a5 = getelementptr inbounds double, double* undef, i32 0
 
-
   ; Cost of vector geps should be one. We cannot fold the address computation.
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i8>, <4 x i8>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i8>, <4 x i8>*
   %a7 = getelementptr inbounds <4 x i8>, <4 x i8>* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i16>, <4 x i16>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i16>, <4 x i16>*
   %a8 = getelementptr inbounds <4 x i16>, <4 x i16>* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i32>, <4 x i32>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i32>, <4 x i32>*
   %a9 = getelementptr inbounds <4 x i32>, <4 x i32>* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i64>, <4 x i64>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i64>, <4 x i64>*
   %a10 = getelementptr inbounds <4 x i64>, <4 x i64>* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x float>, <4 x float>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x float>, <4 x float>*
   %a11 = getelementptr inbounds <4 x float>, <4 x float>* undef, i32 0
-;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x double>, <4 x double>*
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x double>, <4 x double>*
   %a12 = getelementptr inbounds <4 x double>, <4 x double>* undef, i32 0
 
 
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i8, i8*
+  %b0 = getelementptr inbounds i8, i8* undef, i32 1024
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i16, i16*
+  %b1 = getelementptr inbounds i16, i16* undef, i32 1024
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i32, i32*
+  %b2 = getelementptr inbounds i32, i32* undef, i32 1024
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds i64, i64*
+  %b3 = getelementptr inbounds i64, i64* undef, i32 1024
+
+  ; Cost of scalar floating point geps should be one. We cannot fold the address
+  ; computation.
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds float, float*
+  %b4 = getelementptr inbounds float, float* undef, i32 1024
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds double, double*
+  %b5 = getelementptr inbounds double, double* undef, i32 1024
+
+  ; Cost of vector geps should be one. We cannot fold the address computation.
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i8>, <4 x i8>*
+  %b7 = getelementptr inbounds <4 x i8>, <4 x i8>* undef, i32 1
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i16>, <4 x i16>*
+  %b8 = getelementptr inbounds <4 x i16>, <4 x i16>* undef, i32 1
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i32>, <4 x i32>*
+  %b9 = getelementptr inbounds <4 x i32>, <4 x i32>* undef, i32 1
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i64>, <4 x i64>*
+  %b10 = getelementptr inbounds <4 x i64>, <4 x i64>* undef, i32 1
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x float>, <4 x float>*
+  %b11 = getelementptr inbounds <4 x float>, <4 x float>* undef, i32 1
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x double>, <4 x double>*
+  %b12 = getelementptr inbounds <4 x double>, <4 x double>* undef, i32 1
+
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i8, i8*
+  %c0 = getelementptr inbounds i8, i8* undef, i32 %i
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i16, i16*
+  %c1 = getelementptr inbounds i16, i16* undef, i32 %i
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i32, i32*
+  %c2 = getelementptr inbounds i32, i32* undef, i32 %i
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds i64, i64*
+  %c3 = getelementptr inbounds i64, i64* undef, i32 %i
+
+  ; Cost of scalar floating point geps should be one. We cannot fold the address
+  ; computation.
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds float, float*
+  %c4 = getelementptr inbounds float, float* undef, i32 %i
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds double, double*
+  %c5 = getelementptr inbounds double, double* undef, i32 %i
+
+  ; Cost of vector geps should be one. We cannot fold the address computation.
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i8>, <4 x i8>*
+  %c7 = getelementptr inbounds <4 x i8>, <4 x i8>* undef, i32 %i
+;CHECK: cost of 0 for instruction: {{.*}} getelementptr inbounds <4 x i16>, <4 x i16>*
+  %c8 = getelementptr inbounds <4 x i16>, <4 x i16>* undef, i32 %i
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i32>, <4 x i32>*
+  %c9 = getelementptr inbounds <4 x i32>, <4 x i32>* undef, i32 %i
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x i64>, <4 x i64>*
+  %c10 = getelementptr inbounds <4 x i64>, <4 x i64>* undef, i32 %i
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x float>, <4 x float>*
+  %c11 = getelementptr inbounds <4 x float>, <4 x float>* undef, i32 %i
+;CHECK: cost of 1 for instruction: {{.*}} getelementptr inbounds <4 x double>, <4 x double>*
+  %c12 = getelementptr inbounds <4 x double>, <4 x double>* undef, i32 %i
+
+
   ret void
 }

test/Analysis/CostModel/no_info.ll

 ; RUN: opt < %s -cost-model -analyze | FileCheck %s
 
 ; The cost model does not have any target information so it just makes boring
 ; assumptions.
 
 ; -- No triple in this module --
 
-;CHECK: cost of 1 {{.*}} add
-;CHECK: cost of 1 {{.*}} ret
+; CHECK-LABEL: function 'no_info'
+; CHECK: cost of 1 {{.*}} add
+; CHECK: cost of 1 {{.*}} ret
 define i32 @no_info(i32 %arg) {
   %e = add i32 %arg, %arg
   ret i32 %e
 }
+
+define i8 @addressing_mode_reg_reg(i8* %a, i32 %b) {
+; CHECK-LABEL: function 'addressing_mode_reg_reg'
+  %p = getelementptr i8, i8* %a, i32 %b ; NoTTI accepts reg+reg addressing.
+; CHECK: cost of 0 {{.*}} getelementptr
+  %v = load i8, i8* %p
+  ret i8 %v
+}
+
+; CHECK-LABEL: function 'addressing_mode_scaled_reg'
+define i32 @addressing_mode_scaled_reg(i32* %a, i32 %b) {
+  ; NoTTI rejects reg+scale*reg addressing.
+  %p = getelementptr i32, i32* %a, i32 %b
+; CHECK: cost of 1 {{.*}} getelementptr
+  %v = load i32, i32* %p
+  ret i32 %v
+}

test/Transforms/LoopVectorize/X86/metadata-enable.ll

 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i64 %indvars.iv
   %0 = load i32, i32* %arrayidx, align 4
   %add = add nsw i32 %0, %N
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i64 %indvars.iv
   store i32 %add, i32* %arrayidx2, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 32
+  %exitcond = icmp eq i64 %indvars.iv.next, 64
   br i1 %exitcond, label %for.end, label %for.body, !llvm.loop !0
 
 for.end:                                          ; preds = %for.body
   %1 = load i32, i32* %a, align 4
   ret i32 %1
 }
 
 ; O1-LABEL: @nopragma(

 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i64 %indvars.iv
   %0 = load i32, i32* %arrayidx, align 4
   %add = add nsw i32 %0, %N
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i64 %indvars.iv
   store i32 %add, i32* %arrayidx2, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 32
+  %exitcond = icmp eq i64 %indvars.iv.next, 64
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   %1 = load i32, i32* %a, align 4
   ret i32 %1
 }
 
 ; O1-LABEL: @disabled(

 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i64 %indvars.iv
   %0 = load i32, i32* %arrayidx, align 4
   %add = add nsw i32 %0, %N
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i64 %indvars.iv
   store i32 %add, i32* %arrayidx2, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
-  %exitcond = icmp eq i64 %indvars.iv.next, 32
+  %exitcond = icmp eq i64 %indvars.iv.next, 64
   br i1 %exitcond, label %for.end, label %for.body, !llvm.loop !2
 
 for.end:                                          ; preds = %for.body
   %1 = load i32, i32* %a, align 4
   ret i32 %1
 }
 
 !0 = !{!0, !1}
 !1 = !{!"llvm.loop.vectorize.enable", i1 1}
 !2 = !{!2, !3}
 !3 = !{!"llvm.loop.vectorize.enable", i1 0}