This is an archive of the discontinued LLVM Phabricator instance.

Differential D67510

[LV] Support gaps, overlaps, and inexact sizes in speculation logic
AbandonedPublic

Authored by reames on Sep 12 2019, 11:28 AM.

Details

Reviewers
Ayal
hsaito
Summary

Implement last set of TODOs from rL371452. Three cases:

  1. A loop which loads N out of every M bytes. (i.e. there are gaps)
  2. A loop which loads N bytes every M bytes where N > M. (i.e. the loads overlap, and the alignment must be less than natural alignment)
  3. An exact constant trip count is not known, but an upper bound is and the entire region is dereferenceable.

The SCEV change is needed for (3) above. I'm going to separate that into it's own review (with it's own tests) and then rebase once that's landed.

Diff Detail

Event Timeline

reames created this revision.Sep 12 2019, 11:28 AM
Herald added a project: Restricted Project. · View Herald TranscriptSep 12 2019, 11:28 AM
Herald added subscribers: rkruppe, javed.absar, bollu, mcrosier. · View Herald Transcript
reames marked an inline comment as done.Sep 12 2019, 12:29 PM
reames added inline comments.
lib/Analysis/ScalarEvolution.cpp
5599 ↗(On Diff #219955)

This SCEV change is under review here: https://reviews.llvm.org/D67514

Once landed, I will rebase.

Ayal added inline comments.Sep 12 2019, 2:13 PM
lib/Analysis/Loads.cpp
228

Note that StepC may be negative; LV can vectorize accesses such as A[N-i], and also group together accesses with negative non-unit stride into "reversed" interleave groups.

230–238

try [the] an

232

then > than

246

Note that interleave groups (of loads with positive step currently) with gaps at the end may benefit from checking dereferenceability across AccessSize of TC * StepC. But that warrants a separate patch.

256

tha[n]

reames marked an inline comment as done.Sep 12 2019, 2:43 PM
reames added inline comments.
lib/Analysis/Loads.cpp
246

I don't understand the case you're describing here. What do you mean by a "gap at the end"?

p.s. Is there a definition somewhere in code of what an interleave group is? At the moment, I'm assuming it's an access pattern with periodic gaps, but is there something more specific?

reames updated this revision to Diff 220006.Sep 12 2019, 2:43 PM

Address review comments.

Ayal added inline comments.Sep 14 2019, 9:52 AM
lib/Analysis/Loads.cpp
246

What do you mean by a "gap at the end"?

Vectorizing with "gap at the end" was originally described in D19487, and extended under fold-tail in D53668. Simple examples are

for (int i = 0; i < N; i++)
  B[i] = A[2*i];

and

for (int i = 0; i < N; i++)
  C[i] = D[4*i] + D[4*i+1];

These currently trigger requiresScalarEpilogue(), which is better to avoid if dereferenceability can be proven (for missing elements A[2*(N-1)+1] and D[4*(N-1)+2], D[4*(N-1)+3]).

Is there a definition somewhere in code of what an interleave group is?

Yes, an interleave group is defined in Analysis/VectorUtils.h, above the definition of class InterleaveGroup.
Descriptions how interleave groups are vectorized appear above
InterleavedAccessInfo::analyzeInterleaving() in Analysis/VectorUtils.cpp and above
InnerLoopVectorizer::vectorizeInterleaveGroup() in LoopVectorize.cpp.

At the moment, I'm assuming it's an access pattern with periodic gaps, but is there something more specific?

It's usually several access patterns with the same periodic step that can be optimized together using one common wider unit-stride load/store followed-by/preceded-by shuffles, instead of independent gathers/scatters. Plenty of more specific information related to vectorizing interleave groups is available, e.g., "Automatic Vectorization of Interleaved Data Revisited" TACO 2015, "Exploiting mixed SIMD parallelism by reducing data reorganization overhead" CGO 2016.

Ayal added inline comments.Sep 15 2019, 2:03 AM
lib/Analysis/Loads.cpp
225

St[r]ide of zero should be covered by the loop-invariant case above.

226

Worth adding test with negative step?

254

This "if" condition is both necessary and sufficient ("iff"); i.e., nothing TODO/generalize, right?

reames abandoned this revision.Oct 15 2021, 12:03 PM

Abandoning an old review I'm not going to return to any time soon.

Revision Contents

PathSize
lib/
Analysis/
36 lines
test/
Transforms/
LoopVectorize/
X86/
564 lines

Diff 220006

lib/Analysis/Loads.cpp

Show First 20 Lines • Show All 215 Lines • ▼ Show 20 Linesbool llvm::isDereferenceableAndAlignedInLoop(LoadInst *LI, Loop *L,
// Otherwise, check to see if we have a repeating access pattern where we can // Otherwise, check to see if we have a repeating access pattern where we can
// prove that all accesses are well aligned and dereferenceable. // prove that all accesses are well aligned and dereferenceable.
auto *AddRec = dyn_cast<SCEVAddRecExpr>(SE.getSCEV(Ptr)); auto *AddRec = dyn_cast<SCEVAddRecExpr>(SE.getSCEV(Ptr));
if (!AddRec || AddRec->getLoop() != L || !AddRec->isAffine()) if (!AddRec || AddRec->getLoop() != L || !AddRec->isAffine())
return false; return false;
auto* Step = dyn_cast<SCEVConstant>(AddRec->getStepRecurrence(SE)); auto* Step = dyn_cast<SCEVConstant>(AddRec->getStepRecurrence(SE));
if (!Step) if (!Step)
return false; return false;
// TODO: generalize to access patterns which have gaps APInt StepC = Step->getAPInt();
if (Step->getAPInt() != EltSize) // Stide of zero is a degenerate cases, avoid
AyalUnsubmitted
Not Done
ReplyInline Actions

St[r]ide of zero should be covered by the loop-invariant case above.

Ayal: St[r]ide of zero should be covered by the loop-invariant case above.
// TODO: Negative strides require computing the base to query from
AyalUnsubmitted
Not Done
ReplyInline Actions

Worth adding test with negative step?

Ayal: Worth adding test with negative step?
if (!StepC.isStrictlyPositive())
return false; return false;
AyalUnsubmitted
Not Done
ReplyInline Actions

Note that StepC may be negative; LV can vectorize accesses such as A[N-i], and also group together accesses with negative non-unit stride into "reversed" interleave groups.

Ayal: Note that StepC may be negative; LV can vectorize accesses such as A[N-i], and also group…
// TODO: If the symbolic trip count has a small bound (max count), we might // Use the exact trip count if we can compute one; if we can't, try an
// be able to prove safety. // upper bound (which forces us to prove a wider range is dereferenceable
auto TC = SE.getSmallConstantTripCount(L); // than is actually required if we knew the precise trip count.)
AyalUnsubmitted
Not Done
ReplyInline Actions

then > than

Ayal: then > than
unsigned TC = SE.getSmallConstantTripCount(L);
if (!TC) {
TC = SE.getSmallConstantMaxTripCount(L);
if (!TC) if (!TC)
return false; return false;
}
AyalUnsubmitted
Not Done
ReplyInline Actions

try [the] an

Ayal: try [the] an
const APInt AccessSize = TC * EltSize; // Allow gaps and overlaps in the access pattern. We check the
// dereferenceability of the entire region.
// gap:
// load i32, i32* <0,+,8>, align 4 (i.e load every other 4 byte chunk)
// overlap:
// load i32, i32* <0,+,2>, align 2 (every 2 bytes, load a 4 byte chunk)
const APInt AccessSize = (TC-1) * StepC + EltSize;
AyalUnsubmitted
Not Done
ReplyInline Actions

Note that interleave groups (of loads with positive step currently) with gaps at the end may benefit from checking dereferenceability across AccessSize of TC * StepC. But that warrants a separate patch.

Ayal: Note that interleave groups (of loads with positive step currently) with gaps at the end may…
reamesAuthorUnsubmitted
Done
ReplyInline Actions

I don't understand the case you're describing here. What do you mean by a "gap at the end"?

p.s. Is there a definition somewhere in code of what an interleave group is? At the moment, I'm assuming it's an access pattern with periodic gaps, but is there something more specific?

reames: I don't understand the case you're describing here. What do you mean by a "gap at the end"?
AyalUnsubmitted
Not Done
ReplyInline Actions

What do you mean by a "gap at the end"?

Vectorizing with "gap at the end" was originally described in D19487, and extended under fold-tail in D53668. Simple examples are

for (int i = 0; i < N; i++)
  B[i] = A[2*i];

and

for (int i = 0; i < N; i++)
  C[i] = D[4*i] + D[4*i+1];

These currently trigger requiresScalarEpilogue(), which is better to avoid if dereferenceability can be proven (for missing elements A[2*(N-1)+1] and D[4*(N-1)+2], D[4*(N-1)+3]).

Is there a definition somewhere in code of what an interleave group is?

Yes, an interleave group is defined in Analysis/VectorUtils.h, above the definition of class InterleaveGroup.
Descriptions how interleave groups are vectorized appear above
InterleavedAccessInfo::analyzeInterleaving() in Analysis/VectorUtils.cpp and above
InnerLoopVectorizer::vectorizeInterleaveGroup() in LoopVectorize.cpp.

At the moment, I'm assuming it's an access pattern with periodic gaps, but is there something more specific?

It's usually several access patterns with the same periodic step that can be optimized together using one common wider unit-stride load/store followed-by/preceded-by shuffles, instead of independent gathers/scatters. Plenty of more specific information related to vectorizing interleave groups is available, e.g., "Automatic Vectorization of Interleaved Data Revisited" TACO 2015, "Exploiting mixed SIMD parallelism by reducing data reorganization overhead" CGO 2016.

Ayal: > What do you mean by a "gap at the end"? Vectorizing with "gap at the end" was originally…
auto *StartS = dyn_cast<SCEVUnknown>(AddRec->getStart()); auto *StartS = dyn_cast<SCEVUnknown>(AddRec->getStart());
if (!StartS) if (!StartS)
return false; return false;
assert(SE.isLoopInvariant(StartS, L) && "implied by addrec definition"); assert(SE.isLoopInvariant(StartS, L) && "implied by addrec definition");
Value *Base = StartS->getValue(); Value *Base = StartS->getValue();
// For the moment, restrict ourselves to the case where the access size is a // If the base is aligned, and the step is multiple of the alignment, then
AyalUnsubmitted
Not Done
ReplyInline Actions

This "if" condition is both necessary and sufficient ("iff"); i.e., nothing TODO/generalize, right?

Ayal: This "if" condition is both necessary and sufficient ("iff"); i.e., nothing TODO/generalize…
// multiple of the requested alignment and the base is aligned. // the address at each iteration is aligned. (Note: Alignment may be less
// TODO: generalize if a case found which warrants // than element size.)
AyalUnsubmitted
Not Done
ReplyInline Actions

tha[n]

Ayal: tha[n]
if (EltSize.urem(Align) != 0) if (StepC.urem(Align) != 0)
return false; return false;
return isDereferenceableAndAlignedPointer(Base, Align, AccessSize, return isDereferenceableAndAlignedPointer(Base, Align, AccessSize,
DL, HeaderFirstNonPHI, &DT); DL, HeaderFirstNonPHI, &DT);
} }
/// Check if executing a load of this pointer value cannot trap. /// Check if executing a load of this pointer value cannot trap.
/// ///
/// If DT and ScanFrom are specified this method performs context-sensitive /// If DT and ScanFrom are specified this method performs context-sensitive
▲ Show 20 Lines • Show All 228 Lines • Show Last 20 Lines

test/Transforms/LoopVectorize/X86/load-deref-pred.ll

Show First 20 Lines • Show All 530 Lines • ▼ Show 20 Lines
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[ALLOCA:%.*]] = alloca [4096 x i32] ; CHECK-NEXT: [[ALLOCA:%.*]] = alloca [4096 x i32]
; CHECK-NEXT: [[BASE:%.*]] = bitcast [4096 x i32]* [[ALLOCA]] to i32* ; CHECK-NEXT: [[BASE:%.*]] = bitcast [4096 x i32]* [[ALLOCA]] to i32*
; CHECK-NEXT: call void @init(i32* [[BASE]]) ; CHECK-NEXT: call void @init(i32* [[BASE]])
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]] ; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph: ; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]] ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body: ; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE36:%.*]] ] ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP180:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP148:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI4:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP181:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI4:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP149:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI5:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP182:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI5:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP150:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI6:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP183:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI6:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP151:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i64> undef, i64 [[INDEX]], i32 0 ; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i64> undef, i64 [[INDEX]], i32 0
; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i64> [[BROADCAST_SPLATINSERT]], <4 x i64> undef, <4 x i32> zeroinitializer ; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i64> [[BROADCAST_SPLATINSERT]], <4 x i64> undef, <4 x i32> zeroinitializer
; CHECK-NEXT: [[INDUCTION:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 0, i64 1, i64 2, i64 3> ; CHECK-NEXT: [[INDUCTION:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 0, i64 1, i64 2, i64 3>
; CHECK-NEXT: [[INDUCTION1:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 4, i64 5, i64 6, i64 7> ; CHECK-NEXT: [[INDUCTION1:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 4, i64 5, i64 6, i64 7>
; CHECK-NEXT: [[INDUCTION2:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 8, i64 9, i64 10, i64 11> ; CHECK-NEXT: [[INDUCTION2:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 8, i64 9, i64 10, i64 11>
; CHECK-NEXT: [[INDUCTION3:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 12, i64 13, i64 14, i64 15> ; CHECK-NEXT: [[INDUCTION3:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 12, i64 13, i64 14, i64 15>
; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[INDEX]], 0 ; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[INDEX]], 0
; CHECK-NEXT: [[TMP1:%.*]] = add i64 [[INDEX]], 1 ; CHECK-NEXT: [[TMP1:%.*]] = add i64 [[INDEX]], 1
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines
; CHECK-NEXT: [[TMP56:%.*]] = load i1, i1* [[TMP28]] ; CHECK-NEXT: [[TMP56:%.*]] = load i1, i1* [[TMP28]]
; CHECK-NEXT: [[TMP57:%.*]] = load i1, i1* [[TMP29]] ; CHECK-NEXT: [[TMP57:%.*]] = load i1, i1* [[TMP29]]
; CHECK-NEXT: [[TMP58:%.*]] = load i1, i1* [[TMP30]] ; CHECK-NEXT: [[TMP58:%.*]] = load i1, i1* [[TMP30]]
; CHECK-NEXT: [[TMP59:%.*]] = load i1, i1* [[TMP31]] ; CHECK-NEXT: [[TMP59:%.*]] = load i1, i1* [[TMP31]]
; CHECK-NEXT: [[TMP60:%.*]] = insertelement <4 x i1> undef, i1 [[TMP56]], i32 0 ; CHECK-NEXT: [[TMP60:%.*]] = insertelement <4 x i1> undef, i1 [[TMP56]], i32 0
; CHECK-NEXT: [[TMP61:%.*]] = insertelement <4 x i1> [[TMP60]], i1 [[TMP57]], i32 1 ; CHECK-NEXT: [[TMP61:%.*]] = insertelement <4 x i1> [[TMP60]], i1 [[TMP57]], i32 1
; CHECK-NEXT: [[TMP62:%.*]] = insertelement <4 x i1> [[TMP61]], i1 [[TMP58]], i32 2 ; CHECK-NEXT: [[TMP62:%.*]] = insertelement <4 x i1> [[TMP61]], i1 [[TMP58]], i32 2
; CHECK-NEXT: [[TMP63:%.*]] = insertelement <4 x i1> [[TMP62]], i1 [[TMP59]], i32 3 ; CHECK-NEXT: [[TMP63:%.*]] = insertelement <4 x i1> [[TMP62]], i1 [[TMP59]], i32 3
; CHECK-NEXT: [[TMP64:%.*]] = extractelement <4 x i1> [[TMP39]], i32 0 ; CHECK-NEXT: [[TMP64:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: br i1 [[TMP64]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; CHECK: pred.load.if:
; CHECK-NEXT: [[TMP65:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP65:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP66:%.*]] = getelementptr inbounds i16, i16* [[TMP65]], i64 [[TMP0]] ; CHECK-NEXT: [[TMP66:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP67:%.*]] = bitcast i16* [[TMP66]] to i32* ; CHECK-NEXT: [[TMP67:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP68:%.*]] = load i32, i32* [[TMP67]] ; CHECK-NEXT: [[TMP68:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP69:%.*]] = insertelement <4 x i32> undef, i32 [[TMP68]], i32 0 ; CHECK-NEXT: [[TMP69:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE]] ; CHECK-NEXT: [[TMP70:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK: pred.load.continue: ; CHECK-NEXT: [[TMP71:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP70:%.*]] = phi <4 x i32> [ undef, [[VECTOR_BODY]] ], [ [[TMP69]], [[PRED_LOAD_IF]] ]
; CHECK-NEXT: [[TMP71:%.*]] = extractelement <4 x i1> [[TMP39]], i32 1
; CHECK-NEXT: br i1 [[TMP71]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; CHECK: pred.load.if7:
; CHECK-NEXT: [[TMP72:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP72:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP73:%.*]] = getelementptr inbounds i16, i16* [[TMP72]], i64 [[TMP1]] ; CHECK-NEXT: [[TMP73:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP74:%.*]] = bitcast i16* [[TMP73]] to i32* ; CHECK-NEXT: [[TMP74:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP75:%.*]] = load i32, i32* [[TMP74]] ; CHECK-NEXT: [[TMP75:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP76:%.*]] = insertelement <4 x i32> [[TMP70]], i32 [[TMP75]], i32 1 ; CHECK-NEXT: [[TMP76:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE8]] ; CHECK-NEXT: [[TMP77:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK: pred.load.continue8: ; CHECK-NEXT: [[TMP78:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP77:%.*]] = phi <4 x i32> [ [[TMP70]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP76]], [[PRED_LOAD_IF7]] ]
; CHECK-NEXT: [[TMP78:%.*]] = extractelement <4 x i1> [[TMP39]], i32 2
; CHECK-NEXT: br i1 [[TMP78]], label [[PRED_LOAD_IF9:%.*]], label [[PRED_LOAD_CONTINUE10:%.*]]
; CHECK: pred.load.if9:
; CHECK-NEXT: [[TMP79:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP79:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP80:%.*]] = getelementptr inbounds i16, i16* [[TMP79]], i64 [[TMP2]] ; CHECK-NEXT: [[TMP80:%.*]] = getelementptr inbounds i16, i16* [[TMP64]], i64 [[TMP0]]
; CHECK-NEXT: [[TMP81:%.*]] = bitcast i16* [[TMP80]] to i32* ; CHECK-NEXT: [[TMP81:%.*]] = getelementptr inbounds i16, i16* [[TMP65]], i64 [[TMP1]]
; CHECK-NEXT: [[TMP82:%.*]] = load i32, i32* [[TMP81]] ; CHECK-NEXT: [[TMP82:%.*]] = getelementptr inbounds i16, i16* [[TMP66]], i64 [[TMP2]]
; CHECK-NEXT: [[TMP83:%.*]] = insertelement <4 x i32> [[TMP77]], i32 [[TMP82]], i32 2 ; CHECK-NEXT: [[TMP83:%.*]] = getelementptr inbounds i16, i16* [[TMP67]], i64 [[TMP3]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE10]] ; CHECK-NEXT: [[TMP84:%.*]] = getelementptr inbounds i16, i16* [[TMP68]], i64 [[TMP4]]
; CHECK: pred.load.continue10: ; CHECK-NEXT: [[TMP85:%.*]] = getelementptr inbounds i16, i16* [[TMP69]], i64 [[TMP5]]
; CHECK-NEXT: [[TMP84:%.*]] = phi <4 x i32> [ [[TMP77]], [[PRED_LOAD_CONTINUE8]] ], [ [[TMP83]], [[PRED_LOAD_IF9]] ] ; CHECK-NEXT: [[TMP86:%.*]] = getelementptr inbounds i16, i16* [[TMP70]], i64 [[TMP6]]
; CHECK-NEXT: [[TMP85:%.*]] = extractelement <4 x i1> [[TMP39]], i32 3 ; CHECK-NEXT: [[TMP87:%.*]] = getelementptr inbounds i16, i16* [[TMP71]], i64 [[TMP7]]
; CHECK-NEXT: br i1 [[TMP85]], label [[PRED_LOAD_IF11:%.*]], label [[PRED_LOAD_CONTINUE12:%.*]] ; CHECK-NEXT: [[TMP88:%.*]] = getelementptr inbounds i16, i16* [[TMP72]], i64 [[TMP8]]
; CHECK: pred.load.if11: ; CHECK-NEXT: [[TMP89:%.*]] = getelementptr inbounds i16, i16* [[TMP73]], i64 [[TMP9]]
; CHECK-NEXT: [[TMP86:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP90:%.*]] = getelementptr inbounds i16, i16* [[TMP74]], i64 [[TMP10]]
; CHECK-NEXT: [[TMP87:%.*]] = getelementptr inbounds i16, i16* [[TMP86]], i64 [[TMP3]] ; CHECK-NEXT: [[TMP91:%.*]] = getelementptr inbounds i16, i16* [[TMP75]], i64 [[TMP11]]
; CHECK-NEXT: [[TMP88:%.*]] = bitcast i16* [[TMP87]] to i32* ; CHECK-NEXT: [[TMP92:%.*]] = getelementptr inbounds i16, i16* [[TMP76]], i64 [[TMP12]]
; CHECK-NEXT: [[TMP89:%.*]] = load i32, i32* [[TMP88]] ; CHECK-NEXT: [[TMP93:%.*]] = getelementptr inbounds i16, i16* [[TMP77]], i64 [[TMP13]]
; CHECK-NEXT: [[TMP90:%.*]] = insertelement <4 x i32> [[TMP84]], i32 [[TMP89]], i32 3 ; CHECK-NEXT: [[TMP94:%.*]] = getelementptr inbounds i16, i16* [[TMP78]], i64 [[TMP14]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE12]] ; CHECK-NEXT: [[TMP95:%.*]] = getelementptr inbounds i16, i16* [[TMP79]], i64 [[TMP15]]
; CHECK: pred.load.continue12: ; CHECK-NEXT: [[TMP96:%.*]] = bitcast i16* [[TMP80]] to i32*
; CHECK-NEXT: [[TMP91:%.*]] = phi <4 x i32> [ [[TMP84]], [[PRED_LOAD_CONTINUE10]] ], [ [[TMP90]], [[PRED_LOAD_IF11]] ] ; CHECK-NEXT: [[TMP97:%.*]] = bitcast i16* [[TMP81]] to i32*
; CHECK-NEXT: [[TMP92:%.*]] = extractelement <4 x i1> [[TMP47]], i32 0 ; CHECK-NEXT: [[TMP98:%.*]] = bitcast i16* [[TMP82]] to i32*
; CHECK-NEXT: br i1 [[TMP92]], label [[PRED_LOAD_IF13:%.*]], label [[PRED_LOAD_CONTINUE14:%.*]] ; CHECK-NEXT: [[TMP99:%.*]] = bitcast i16* [[TMP83]] to i32*
; CHECK: pred.load.if13: ; CHECK-NEXT: [[TMP100:%.*]] = bitcast i16* [[TMP84]] to i32*
; CHECK-NEXT: [[TMP93:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP101:%.*]] = bitcast i16* [[TMP85]] to i32*
; CHECK-NEXT: [[TMP94:%.*]] = getelementptr inbounds i16, i16* [[TMP93]], i64 [[TMP4]] ; CHECK-NEXT: [[TMP102:%.*]] = bitcast i16* [[TMP86]] to i32*
; CHECK-NEXT: [[TMP95:%.*]] = bitcast i16* [[TMP94]] to i32* ; CHECK-NEXT: [[TMP103:%.*]] = bitcast i16* [[TMP87]] to i32*
; CHECK-NEXT: [[TMP96:%.*]] = load i32, i32* [[TMP95]] ; CHECK-NEXT: [[TMP104:%.*]] = bitcast i16* [[TMP88]] to i32*
; CHECK-NEXT: [[TMP97:%.*]] = insertelement <4 x i32> undef, i32 [[TMP96]], i32 0 ; CHECK-NEXT: [[TMP105:%.*]] = bitcast i16* [[TMP89]] to i32*
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE14]] ; CHECK-NEXT: [[TMP106:%.*]] = bitcast i16* [[TMP90]] to i32*
; CHECK: pred.load.continue14: ; CHECK-NEXT: [[TMP107:%.*]] = bitcast i16* [[TMP91]] to i32*
; CHECK-NEXT: [[TMP98:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE12]] ], [ [[TMP97]], [[PRED_LOAD_IF13]] ] ; CHECK-NEXT: [[TMP108:%.*]] = bitcast i16* [[TMP92]] to i32*
; CHECK-NEXT: [[TMP99:%.*]] = extractelement <4 x i1> [[TMP47]], i32 1 ; CHECK-NEXT: [[TMP109:%.*]] = bitcast i16* [[TMP93]] to i32*
; CHECK-NEXT: br i1 [[TMP99]], label [[PRED_LOAD_IF15:%.*]], label [[PRED_LOAD_CONTINUE16:%.*]] ; CHECK-NEXT: [[TMP110:%.*]] = bitcast i16* [[TMP94]] to i32*
; CHECK: pred.load.if15: ; CHECK-NEXT: [[TMP111:%.*]] = bitcast i16* [[TMP95]] to i32*
; CHECK-NEXT: [[TMP100:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP112:%.*]] = load i32, i32* [[TMP96]], align 2
; CHECK-NEXT: [[TMP101:%.*]] = getelementptr inbounds i16, i16* [[TMP100]], i64 [[TMP5]] ; CHECK-NEXT: [[TMP113:%.*]] = load i32, i32* [[TMP97]], align 2
; CHECK-NEXT: [[TMP102:%.*]] = bitcast i16* [[TMP101]] to i32* ; CHECK-NEXT: [[TMP114:%.*]] = load i32, i32* [[TMP98]], align 2
; CHECK-NEXT: [[TMP103:%.*]] = load i32, i32* [[TMP102]] ; CHECK-NEXT: [[TMP115:%.*]] = load i32, i32* [[TMP99]], align 2
; CHECK-NEXT: [[TMP104:%.*]] = insertelement <4 x i32> [[TMP98]], i32 [[TMP103]], i32 1 ; CHECK-NEXT: [[TMP116:%.*]] = insertelement <4 x i32> undef, i32 [[TMP112]], i32 0
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE16]] ; CHECK-NEXT: [[TMP117:%.*]] = insertelement <4 x i32> [[TMP116]], i32 [[TMP113]], i32 1
; CHECK: pred.load.continue16: ; CHECK-NEXT: [[TMP118:%.*]] = insertelement <4 x i32> [[TMP117]], i32 [[TMP114]], i32 2
; CHECK-NEXT: [[TMP105:%.*]] = phi <4 x i32> [ [[TMP98]], [[PRED_LOAD_CONTINUE14]] ], [ [[TMP104]], [[PRED_LOAD_IF15]] ] ; CHECK-NEXT: [[TMP119:%.*]] = insertelement <4 x i32> [[TMP118]], i32 [[TMP115]], i32 3
; CHECK-NEXT: [[TMP106:%.*]] = extractelement <4 x i1> [[TMP47]], i32 2 ; CHECK-NEXT: [[TMP120:%.*]] = load i32, i32* [[TMP100]], align 2
; CHECK-NEXT: br i1 [[TMP106]], label [[PRED_LOAD_IF17:%.*]], label [[PRED_LOAD_CONTINUE18:%.*]] ; CHECK-NEXT: [[TMP121:%.*]] = load i32, i32* [[TMP101]], align 2
; CHECK: pred.load.if17: ; CHECK-NEXT: [[TMP122:%.*]] = load i32, i32* [[TMP102]], align 2
; CHECK-NEXT: [[TMP107:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP123:%.*]] = load i32, i32* [[TMP103]], align 2
; CHECK-NEXT: [[TMP108:%.*]] = getelementptr inbounds i16, i16* [[TMP107]], i64 [[TMP6]] ; CHECK-NEXT: [[TMP124:%.*]] = insertelement <4 x i32> undef, i32 [[TMP120]], i32 0
; CHECK-NEXT: [[TMP109:%.*]] = bitcast i16* [[TMP108]] to i32* ; CHECK-NEXT: [[TMP125:%.*]] = insertelement <4 x i32> [[TMP124]], i32 [[TMP121]], i32 1
; CHECK-NEXT: [[TMP110:%.*]] = load i32, i32* [[TMP109]] ; CHECK-NEXT: [[TMP126:%.*]] = insertelement <4 x i32> [[TMP125]], i32 [[TMP122]], i32 2
; CHECK-NEXT: [[TMP111:%.*]] = insertelement <4 x i32> [[TMP105]], i32 [[TMP110]], i32 2 ; CHECK-NEXT: [[TMP127:%.*]] = insertelement <4 x i32> [[TMP126]], i32 [[TMP123]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE18]] ; CHECK-NEXT: [[TMP128:%.*]] = load i32, i32* [[TMP104]], align 2
; CHECK: pred.load.continue18: ; CHECK-NEXT: [[TMP129:%.*]] = load i32, i32* [[TMP105]], align 2
; CHECK-NEXT: [[TMP112:%.*]] = phi <4 x i32> [ [[TMP105]], [[PRED_LOAD_CONTINUE16]] ], [ [[TMP111]], [[PRED_LOAD_IF17]] ] ; CHECK-NEXT: [[TMP130:%.*]] = load i32, i32* [[TMP106]], align 2
; CHECK-NEXT: [[TMP113:%.*]] = extractelement <4 x i1> [[TMP47]], i32 3 ; CHECK-NEXT: [[TMP131:%.*]] = load i32, i32* [[TMP107]], align 2
; CHECK-NEXT: br i1 [[TMP113]], label [[PRED_LOAD_IF19:%.*]], label [[PRED_LOAD_CONTINUE20:%.*]] ; CHECK-NEXT: [[TMP132:%.*]] = insertelement <4 x i32> undef, i32 [[TMP128]], i32 0
; CHECK: pred.load.if19: ; CHECK-NEXT: [[TMP133:%.*]] = insertelement <4 x i32> [[TMP132]], i32 [[TMP129]], i32 1
; CHECK-NEXT: [[TMP114:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP134:%.*]] = insertelement <4 x i32> [[TMP133]], i32 [[TMP130]], i32 2
; CHECK-NEXT: [[TMP115:%.*]] = getelementptr inbounds i16, i16* [[TMP114]], i64 [[TMP7]] ; CHECK-NEXT: [[TMP135:%.*]] = insertelement <4 x i32> [[TMP134]], i32 [[TMP131]], i32 3
; CHECK-NEXT: [[TMP116:%.*]] = bitcast i16* [[TMP115]] to i32* ; CHECK-NEXT: [[TMP136:%.*]] = load i32, i32* [[TMP108]], align 2
; CHECK-NEXT: [[TMP117:%.*]] = load i32, i32* [[TMP116]] ; CHECK-NEXT: [[TMP137:%.*]] = load i32, i32* [[TMP109]], align 2
; CHECK-NEXT: [[TMP118:%.*]] = insertelement <4 x i32> [[TMP112]], i32 [[TMP117]], i32 3 ; CHECK-NEXT: [[TMP138:%.*]] = load i32, i32* [[TMP110]], align 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE20]] ; CHECK-NEXT: [[TMP139:%.*]] = load i32, i32* [[TMP111]], align 2
; CHECK: pred.load.continue20: ; CHECK-NEXT: [[TMP140:%.*]] = insertelement <4 x i32> undef, i32 [[TMP136]], i32 0
; CHECK-NEXT: [[TMP119:%.*]] = phi <4 x i32> [ [[TMP112]], [[PRED_LOAD_CONTINUE18]] ], [ [[TMP118]], [[PRED_LOAD_IF19]] ] ; CHECK-NEXT: [[TMP141:%.*]] = insertelement <4 x i32> [[TMP140]], i32 [[TMP137]], i32 1
; CHECK-NEXT: [[TMP120:%.*]] = extractelement <4 x i1> [[TMP55]], i32 0 ; CHECK-NEXT: [[TMP142:%.*]] = insertelement <4 x i32> [[TMP141]], i32 [[TMP138]], i32 2
; CHECK-NEXT: br i1 [[TMP120]], label [[PRED_LOAD_IF21:%.*]], label [[PRED_LOAD_CONTINUE22:%.*]] ; CHECK-NEXT: [[TMP143:%.*]] = insertelement <4 x i32> [[TMP142]], i32 [[TMP139]], i32 3
; CHECK: pred.load.if21: ; CHECK-NEXT: [[TMP144:%.*]] = xor <4 x i1> [[TMP39]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP121:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[TMP145:%.*]] = xor <4 x i1> [[TMP47]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP122:%.*]] = getelementptr inbounds i16, i16* [[TMP121]], i64 [[TMP8]] ; CHECK-NEXT: [[TMP146:%.*]] = xor <4 x i1> [[TMP55]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP123:%.*]] = bitcast i16* [[TMP122]] to i32* ; CHECK-NEXT: [[TMP147:%.*]] = xor <4 x i1> [[TMP63]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP124:%.*]] = load i32, i32* [[TMP123]] ; CHECK-NEXT: [[PREDPHI:%.*]] = select <4 x i1> [[TMP39]], <4 x i32> [[TMP119]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP125:%.*]] = insertelement <4 x i32> undef, i32 [[TMP124]], i32 0 ; CHECK-NEXT: [[PREDPHI7:%.*]] = select <4 x i1> [[TMP47]], <4 x i32> [[TMP127]], <4 x i32> zeroinitializer
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE22]] ; CHECK-NEXT: [[PREDPHI8:%.*]] = select <4 x i1> [[TMP55]], <4 x i32> [[TMP135]], <4 x i32> zeroinitializer
; CHECK: pred.load.continue22: ; CHECK-NEXT: [[PREDPHI9:%.*]] = select <4 x i1> [[TMP63]], <4 x i32> [[TMP143]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP126:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE20]] ], [ [[TMP125]], [[PRED_LOAD_IF21]] ] ; CHECK-NEXT: [[TMP148]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
; CHECK-NEXT: [[TMP127:%.*]] = extractelement <4 x i1> [[TMP55]], i32 1 ; CHECK-NEXT: [[TMP149]] = add <4 x i32> [[VEC_PHI4]], [[PREDPHI7]]
; CHECK-NEXT: br i1 [[TMP127]], label [[PRED_LOAD_IF23:%.*]], label [[PRED_LOAD_CONTINUE24:%.*]] ; CHECK-NEXT: [[TMP150]] = add <4 x i32> [[VEC_PHI5]], [[PREDPHI8]]
; CHECK: pred.load.if23: ; CHECK-NEXT: [[TMP151]] = add <4 x i32> [[VEC_PHI6]], [[PREDPHI9]]
; CHECK-NEXT: [[TMP128:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP129:%.*]] = getelementptr inbounds i16, i16* [[TMP128]], i64 [[TMP9]]
; CHECK-NEXT: [[TMP130:%.*]] = bitcast i16* [[TMP129]] to i32*
; CHECK-NEXT: [[TMP131:%.*]] = load i32, i32* [[TMP130]]
; CHECK-NEXT: [[TMP132:%.*]] = insertelement <4 x i32> [[TMP126]], i32 [[TMP131]], i32 1
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE24]]
; CHECK: pred.load.continue24:
; CHECK-NEXT: [[TMP133:%.*]] = phi <4 x i32> [ [[TMP126]], [[PRED_LOAD_CONTINUE22]] ], [ [[TMP132]], [[PRED_LOAD_IF23]] ]
; CHECK-NEXT: [[TMP134:%.*]] = extractelement <4 x i1> [[TMP55]], i32 2
; CHECK-NEXT: br i1 [[TMP134]], label [[PRED_LOAD_IF25:%.*]], label [[PRED_LOAD_CONTINUE26:%.*]]
; CHECK: pred.load.if25:
; CHECK-NEXT: [[TMP135:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP136:%.*]] = getelementptr inbounds i16, i16* [[TMP135]], i64 [[TMP10]]
; CHECK-NEXT: [[TMP137:%.*]] = bitcast i16* [[TMP136]] to i32*
; CHECK-NEXT: [[TMP138:%.*]] = load i32, i32* [[TMP137]]
; CHECK-NEXT: [[TMP139:%.*]] = insertelement <4 x i32> [[TMP133]], i32 [[TMP138]], i32 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE26]]
; CHECK: pred.load.continue26:
; CHECK-NEXT: [[TMP140:%.*]] = phi <4 x i32> [ [[TMP133]], [[PRED_LOAD_CONTINUE24]] ], [ [[TMP139]], [[PRED_LOAD_IF25]] ]
; CHECK-NEXT: [[TMP141:%.*]] = extractelement <4 x i1> [[TMP55]], i32 3
; CHECK-NEXT: br i1 [[TMP141]], label [[PRED_LOAD_IF27:%.*]], label [[PRED_LOAD_CONTINUE28:%.*]]
; CHECK: pred.load.if27:
; CHECK-NEXT: [[TMP142:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP143:%.*]] = getelementptr inbounds i16, i16* [[TMP142]], i64 [[TMP11]]
; CHECK-NEXT: [[TMP144:%.*]] = bitcast i16* [[TMP143]] to i32*
; CHECK-NEXT: [[TMP145:%.*]] = load i32, i32* [[TMP144]]
; CHECK-NEXT: [[TMP146:%.*]] = insertelement <4 x i32> [[TMP140]], i32 [[TMP145]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE28]]
; CHECK: pred.load.continue28:
; CHECK-NEXT: [[TMP147:%.*]] = phi <4 x i32> [ [[TMP140]], [[PRED_LOAD_CONTINUE26]] ], [ [[TMP146]], [[PRED_LOAD_IF27]] ]
; CHECK-NEXT: [[TMP148:%.*]] = extractelement <4 x i1> [[TMP63]], i32 0
; CHECK-NEXT: br i1 [[TMP148]], label [[PRED_LOAD_IF29:%.*]], label [[PRED_LOAD_CONTINUE30:%.*]]
; CHECK: pred.load.if29:
; CHECK-NEXT: [[TMP149:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP150:%.*]] = getelementptr inbounds i16, i16* [[TMP149]], i64 [[TMP12]]
; CHECK-NEXT: [[TMP151:%.*]] = bitcast i16* [[TMP150]] to i32*
; CHECK-NEXT: [[TMP152:%.*]] = load i32, i32* [[TMP151]]
; CHECK-NEXT: [[TMP153:%.*]] = insertelement <4 x i32> undef, i32 [[TMP152]], i32 0
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE30]]
; CHECK: pred.load.continue30:
; CHECK-NEXT: [[TMP154:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE28]] ], [ [[TMP153]], [[PRED_LOAD_IF29]] ]
; CHECK-NEXT: [[TMP155:%.*]] = extractelement <4 x i1> [[TMP63]], i32 1
; CHECK-NEXT: br i1 [[TMP155]], label [[PRED_LOAD_IF31:%.*]], label [[PRED_LOAD_CONTINUE32:%.*]]
; CHECK: pred.load.if31:
; CHECK-NEXT: [[TMP156:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP157:%.*]] = getelementptr inbounds i16, i16* [[TMP156]], i64 [[TMP13]]
; CHECK-NEXT: [[TMP158:%.*]] = bitcast i16* [[TMP157]] to i32*
; CHECK-NEXT: [[TMP159:%.*]] = load i32, i32* [[TMP158]]
; CHECK-NEXT: [[TMP160:%.*]] = insertelement <4 x i32> [[TMP154]], i32 [[TMP159]], i32 1
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE32]]
; CHECK: pred.load.continue32:
; CHECK-NEXT: [[TMP161:%.*]] = phi <4 x i32> [ [[TMP154]], [[PRED_LOAD_CONTINUE30]] ], [ [[TMP160]], [[PRED_LOAD_IF31]] ]
; CHECK-NEXT: [[TMP162:%.*]] = extractelement <4 x i1> [[TMP63]], i32 2
; CHECK-NEXT: br i1 [[TMP162]], label [[PRED_LOAD_IF33:%.*]], label [[PRED_LOAD_CONTINUE34:%.*]]
; CHECK: pred.load.if33:
; CHECK-NEXT: [[TMP163:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP164:%.*]] = getelementptr inbounds i16, i16* [[TMP163]], i64 [[TMP14]]
; CHECK-NEXT: [[TMP165:%.*]] = bitcast i16* [[TMP164]] to i32*
; CHECK-NEXT: [[TMP166:%.*]] = load i32, i32* [[TMP165]]
; CHECK-NEXT: [[TMP167:%.*]] = insertelement <4 x i32> [[TMP161]], i32 [[TMP166]], i32 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE34]]
; CHECK: pred.load.continue34:
; CHECK-NEXT: [[TMP168:%.*]] = phi <4 x i32> [ [[TMP161]], [[PRED_LOAD_CONTINUE32]] ], [ [[TMP167]], [[PRED_LOAD_IF33]] ]
; CHECK-NEXT: [[TMP169:%.*]] = extractelement <4 x i1> [[TMP63]], i32 3
; CHECK-NEXT: br i1 [[TMP169]], label [[PRED_LOAD_IF35:%.*]], label [[PRED_LOAD_CONTINUE36]]
; CHECK: pred.load.if35:
; CHECK-NEXT: [[TMP170:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[TMP171:%.*]] = getelementptr inbounds i16, i16* [[TMP170]], i64 [[TMP15]]
; CHECK-NEXT: [[TMP172:%.*]] = bitcast i16* [[TMP171]] to i32*
; CHECK-NEXT: [[TMP173:%.*]] = load i32, i32* [[TMP172]]
; CHECK-NEXT: [[TMP174:%.*]] = insertelement <4 x i32> [[TMP168]], i32 [[TMP173]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE36]]
; CHECK: pred.load.continue36:
; CHECK-NEXT: [[TMP175:%.*]] = phi <4 x i32> [ [[TMP168]], [[PRED_LOAD_CONTINUE34]] ], [ [[TMP174]], [[PRED_LOAD_IF35]] ]
; CHECK-NEXT: [[TMP176:%.*]] = xor <4 x i1> [[TMP39]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP177:%.*]] = xor <4 x i1> [[TMP47]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP178:%.*]] = xor <4 x i1> [[TMP55]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP179:%.*]] = xor <4 x i1> [[TMP63]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[PREDPHI:%.*]] = select <4 x i1> [[TMP39]], <4 x i32> [[TMP91]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI37:%.*]] = select <4 x i1> [[TMP47]], <4 x i32> [[TMP119]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI38:%.*]] = select <4 x i1> [[TMP55]], <4 x i32> [[TMP147]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI39:%.*]] = select <4 x i1> [[TMP63]], <4 x i32> [[TMP175]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP180]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
; CHECK-NEXT: [[TMP181]] = add <4 x i32> [[VEC_PHI4]], [[PREDPHI37]]
; CHECK-NEXT: [[TMP182]] = add <4 x i32> [[VEC_PHI5]], [[PREDPHI38]]
; CHECK-NEXT: [[TMP183]] = add <4 x i32> [[VEC_PHI6]], [[PREDPHI39]]
; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 16 ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 16
; CHECK-NEXT: [[TMP184:%.*]] = icmp eq i64 [[INDEX_NEXT]], 4096 ; CHECK-NEXT: [[TMP152:%.*]] = icmp eq i64 [[INDEX_NEXT]], 4096
; CHECK-NEXT: br i1 [[TMP184]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !8 ; CHECK-NEXT: br i1 [[TMP152]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !8
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[BIN_RDX:%.*]] = add <4 x i32> [[TMP181]], [[TMP180]] ; CHECK-NEXT: [[BIN_RDX:%.*]] = add <4 x i32> [[TMP149]], [[TMP148]]
; CHECK-NEXT: [[BIN_RDX40:%.*]] = add <4 x i32> [[TMP182]], [[BIN_RDX]] ; CHECK-NEXT: [[BIN_RDX10:%.*]] = add <4 x i32> [[TMP150]], [[BIN_RDX]]
; CHECK-NEXT: [[BIN_RDX41:%.*]] = add <4 x i32> [[TMP183]], [[BIN_RDX40]] ; CHECK-NEXT: [[BIN_RDX11:%.*]] = add <4 x i32> [[TMP151]], [[BIN_RDX10]]
; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x i32> [[BIN_RDX41]], <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef> ; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x i32> [[BIN_RDX11]], <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
; CHECK-NEXT: [[BIN_RDX42:%.*]] = add <4 x i32> [[BIN_RDX41]], [[RDX_SHUF]] ; CHECK-NEXT: [[BIN_RDX12:%.*]] = add <4 x i32> [[BIN_RDX11]], [[RDX_SHUF]]
; CHECK-NEXT: [[RDX_SHUF43:%.*]] = shufflevector <4 x i32> [[BIN_RDX42]], <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef> ; CHECK-NEXT: [[RDX_SHUF13:%.*]] = shufflevector <4 x i32> [[BIN_RDX12]], <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
; CHECK-NEXT: [[BIN_RDX44:%.*]] = add <4 x i32> [[BIN_RDX42]], [[RDX_SHUF43]] ; CHECK-NEXT: [[BIN_RDX14:%.*]] = add <4 x i32> [[BIN_RDX12]], [[RDX_SHUF13]]
; CHECK-NEXT: [[TMP185:%.*]] = extractelement <4 x i32> [[BIN_RDX44]], i32 0 ; CHECK-NEXT: [[TMP153:%.*]] = extractelement <4 x i32> [[BIN_RDX14]], i32 0
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 4096, 4096 ; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 4096, 4096
; CHECK-NEXT: br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]] ; CHECK-NEXT: br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 4096, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ] ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 4096, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP185]], [[MIDDLE_BLOCK]] ] ; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP153]], [[MIDDLE_BLOCK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]] ; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop: ; CHECK: loop:
; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ] ; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
; CHECK-NEXT: [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ] ; CHECK-NEXT: [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
; CHECK-NEXT: [[IV_NEXT]] = add i64 [[IV]], 1 ; CHECK-NEXT: [[IV_NEXT]] = add i64 [[IV]], 1
; CHECK-NEXT: [[TEST_ADDR:%.*]] = getelementptr inbounds i1, i1* [[TEST_BASE]], i64 [[IV]] ; CHECK-NEXT: [[TEST_ADDR:%.*]] = getelementptr inbounds i1, i1* [[TEST_BASE]], i64 [[IV]]
; CHECK-NEXT: [[EARLYCND:%.*]] = load i1, i1* [[TEST_ADDR]] ; CHECK-NEXT: [[EARLYCND:%.*]] = load i1, i1* [[TEST_ADDR]]
; CHECK-NEXT: br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]] ; CHECK-NEXT: br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
; CHECK: pred: ; CHECK: pred:
; CHECK-NEXT: [[BASE_I16P:%.*]] = bitcast i32* [[BASE]] to i16* ; CHECK-NEXT: [[BASE_I16P:%.*]] = bitcast i32* [[BASE]] to i16*
; CHECK-NEXT: [[ADDR_I16P:%.*]] = getelementptr inbounds i16, i16* [[BASE_I16P]], i64 [[IV]] ; CHECK-NEXT: [[ADDR_I16P:%.*]] = getelementptr inbounds i16, i16* [[BASE_I16P]], i64 [[IV]]
; CHECK-NEXT: [[ADDR:%.*]] = bitcast i16* [[ADDR_I16P]] to i32* ; CHECK-NEXT: [[ADDR:%.*]] = bitcast i16* [[ADDR_I16P]] to i32*
; CHECK-NEXT: [[VAL:%.*]] = load i32, i32* [[ADDR]] ; CHECK-NEXT: [[VAL:%.*]] = load i32, i32* [[ADDR]], align 2
; CHECK-NEXT: br label [[LATCH]] ; CHECK-NEXT: br label [[LATCH]]
; CHECK: latch: ; CHECK: latch:
; CHECK-NEXT: [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ] ; CHECK-NEXT: [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
; CHECK-NEXT: [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]] ; CHECK-NEXT: [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
; CHECK-NEXT: [[EXIT:%.*]] = icmp ugt i64 [[IV]], 4094 ; CHECK-NEXT: [[EXIT:%.*]] = icmp ugt i64 [[IV]], 4094
; CHECK-NEXT: br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop !9 ; CHECK-NEXT: br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop !9
; CHECK: loop_exit: ; CHECK: loop_exit:
; CHECK-NEXT: [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP185]], [[MIDDLE_BLOCK]] ] ; CHECK-NEXT: [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP153]], [[MIDDLE_BLOCK]] ]
; CHECK-NEXT: ret i32 [[ACCUM_NEXT_LCSSA]] ; CHECK-NEXT: ret i32 [[ACCUM_NEXT_LCSSA]]
; ;
entry: entry:
%alloca = alloca [4096 x i32] %alloca = alloca [4096 x i32]
%base = bitcast [4096 x i32]* %alloca to i32* %base = bitcast [4096 x i32]* %alloca to i32*
call void @init(i32* %base) call void @init(i32* %base)
br label %loop br label %loop
loop: loop:
%iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ] %iv = phi i64 [ 0, %entry ], [ %iv.next, %latch ]
%accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ] %accum = phi i32 [ 0, %entry ], [ %accum.next, %latch ]
%iv.next = add i64 %iv, 1 %iv.next = add i64 %iv, 1
%test_addr = getelementptr inbounds i1, i1* %test_base, i64 %iv %test_addr = getelementptr inbounds i1, i1* %test_base, i64 %iv
%earlycnd = load i1, i1* %test_addr %earlycnd = load i1, i1* %test_addr
br i1 %earlycnd, label %pred, label %latch br i1 %earlycnd, label %pred, label %latch
pred: pred:
%base.i16p = bitcast i32* %base to i16* %base.i16p = bitcast i32* %base to i16*
%addr.i16p = getelementptr inbounds i16, i16* %base.i16p, i64 %iv %addr.i16p = getelementptr inbounds i16, i16* %base.i16p, i64 %iv
%addr = bitcast i16* %addr.i16p to i32* %addr = bitcast i16* %addr.i16p to i32*
%val = load i32, i32* %addr %val = load i32, i32* %addr, align 2
br label %latch br label %latch
latch: latch:
%val.phi = phi i32 [0, %loop], [%val, %pred] %val.phi = phi i32 [0, %loop], [%val, %pred]
%accum.next = add i32 %accum, %val.phi %accum.next = add i32 %accum, %val.phi
%exit = icmp ugt i64 %iv, 4094 %exit = icmp ugt i64 %iv, 4094
br i1 %exit, label %loop_exit, label %loop br i1 %exit, label %loop_exit, label %loop
loop_exit: loop_exit:
▲ Show 20 Lines • Show All 414 Lines • ▼ Show 20 Lineslatch:
%accum.next = add i32 %accum, %val.phi %accum.next = add i32 %accum, %val.phi
%exit = icmp ugt i64 %iv, 4094 %exit = icmp ugt i64 %iv, 4094
br i1 %exit, label %loop_exit, label %loop br i1 %exit, label %loop_exit, label %loop
loop_exit: loop_exit:
ret i32 %accum.next ret i32 %accum.next
} }
;; TODO: handle non-unit strides
define i32 @test_non_unit_stride(i64 %len, i1* %test_base) { define i32 @test_non_unit_stride(i64 %len, i1* %test_base) {
; CHECK-LABEL: @test_non_unit_stride( ; CHECK-LABEL: @test_non_unit_stride(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[ALLOCA:%.*]] = alloca [4096 x i32] ; CHECK-NEXT: [[ALLOCA:%.*]] = alloca [4096 x i32]
; CHECK-NEXT: [[BASE:%.*]] = bitcast [4096 x i32]* [[ALLOCA]] to i32* ; CHECK-NEXT: [[BASE:%.*]] = bitcast [4096 x i32]* [[ALLOCA]] to i32*
; CHECK-NEXT: call void @init(i32* [[BASE]]) ; CHECK-NEXT: call void @init(i32* [[BASE]])
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]] ; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph: ; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]] ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body: ; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE36:%.*]] ] ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP148:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP116:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI4:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP149:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI4:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP117:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI5:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP150:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI5:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP118:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[VEC_PHI6:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP151:%.*]], [[PRED_LOAD_CONTINUE36]] ] ; CHECK-NEXT: [[VEC_PHI6:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP119:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 2 ; CHECK-NEXT: [[OFFSET_IDX:%.*]] = mul i64 [[INDEX]], 2
; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i64> undef, i64 [[OFFSET_IDX]], i32 0 ; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i64> undef, i64 [[OFFSET_IDX]], i32 0
; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i64> [[BROADCAST_SPLATINSERT]], <4 x i64> undef, <4 x i32> zeroinitializer ; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i64> [[BROADCAST_SPLATINSERT]], <4 x i64> undef, <4 x i32> zeroinitializer
; CHECK-NEXT: [[INDUCTION:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 0, i64 2, i64 4, i64 6> ; CHECK-NEXT: [[INDUCTION:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 0, i64 2, i64 4, i64 6>
; CHECK-NEXT: [[INDUCTION1:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 8, i64 10, i64 12, i64 14> ; CHECK-NEXT: [[INDUCTION1:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 8, i64 10, i64 12, i64 14>
; CHECK-NEXT: [[INDUCTION2:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 16, i64 18, i64 20, i64 22> ; CHECK-NEXT: [[INDUCTION2:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 16, i64 18, i64 20, i64 22>
; CHECK-NEXT: [[INDUCTION3:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 24, i64 26, i64 28, i64 30> ; CHECK-NEXT: [[INDUCTION3:%.*]] = add <4 x i64> [[BROADCAST_SPLAT]], <i64 24, i64 26, i64 28, i64 30>
; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0 ; CHECK-NEXT: [[TMP0:%.*]] = add i64 [[OFFSET_IDX]], 0
▲ Show 20 Lines • Show All 55 Lines • ▼ Show 20 Lines
; CHECK-NEXT: [[TMP56:%.*]] = load i1, i1* [[TMP28]] ; CHECK-NEXT: [[TMP56:%.*]] = load i1, i1* [[TMP28]]
; CHECK-NEXT: [[TMP57:%.*]] = load i1, i1* [[TMP29]] ; CHECK-NEXT: [[TMP57:%.*]] = load i1, i1* [[TMP29]]
; CHECK-NEXT: [[TMP58:%.*]] = load i1, i1* [[TMP30]] ; CHECK-NEXT: [[TMP58:%.*]] = load i1, i1* [[TMP30]]
; CHECK-NEXT: [[TMP59:%.*]] = load i1, i1* [[TMP31]] ; CHECK-NEXT: [[TMP59:%.*]] = load i1, i1* [[TMP31]]
; CHECK-NEXT: [[TMP60:%.*]] = insertelement <4 x i1> undef, i1 [[TMP56]], i32 0 ; CHECK-NEXT: [[TMP60:%.*]] = insertelement <4 x i1> undef, i1 [[TMP56]], i32 0
; CHECK-NEXT: [[TMP61:%.*]] = insertelement <4 x i1> [[TMP60]], i1 [[TMP57]], i32 1 ; CHECK-NEXT: [[TMP61:%.*]] = insertelement <4 x i1> [[TMP60]], i1 [[TMP57]], i32 1
; CHECK-NEXT: [[TMP62:%.*]] = insertelement <4 x i1> [[TMP61]], i1 [[TMP58]], i32 2 ; CHECK-NEXT: [[TMP62:%.*]] = insertelement <4 x i1> [[TMP61]], i1 [[TMP58]], i32 2
; CHECK-NEXT: [[TMP63:%.*]] = insertelement <4 x i1> [[TMP62]], i1 [[TMP59]], i32 3 ; CHECK-NEXT: [[TMP63:%.*]] = insertelement <4 x i1> [[TMP62]], i1 [[TMP59]], i32 3
; CHECK-NEXT: [[TMP64:%.*]] = extractelement <4 x i1> [[TMP39]], i32 0 ; CHECK-NEXT: [[TMP64:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP0]]
; CHECK-NEXT: br i1 [[TMP64]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]] ; CHECK-NEXT: [[TMP65:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP1]]
; CHECK: pred.load.if: ; CHECK-NEXT: [[TMP66:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP2]]
; CHECK-NEXT: [[TMP65:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP0]] ; CHECK-NEXT: [[TMP67:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP3]]
; CHECK-NEXT: [[TMP66:%.*]] = load i32, i32* [[TMP65]] ; CHECK-NEXT: [[TMP68:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP4]]
; CHECK-NEXT: [[TMP67:%.*]] = insertelement <4 x i32> undef, i32 [[TMP66]], i32 0 ; CHECK-NEXT: [[TMP69:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP5]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE]] ; CHECK-NEXT: [[TMP70:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP6]]
; CHECK: pred.load.continue: ; CHECK-NEXT: [[TMP71:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP7]]
; CHECK-NEXT: [[TMP68:%.*]] = phi <4 x i32> [ undef, [[VECTOR_BODY]] ], [ [[TMP67]], [[PRED_LOAD_IF]] ] ; CHECK-NEXT: [[TMP72:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP8]]
; CHECK-NEXT: [[TMP69:%.*]] = extractelement <4 x i1> [[TMP39]], i32 1 ; CHECK-NEXT: [[TMP73:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP9]]
; CHECK-NEXT: br i1 [[TMP69]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]] ; CHECK-NEXT: [[TMP74:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP10]]
; CHECK: pred.load.if7: ; CHECK-NEXT: [[TMP75:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP11]]
; CHECK-NEXT: [[TMP70:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP1]] ; CHECK-NEXT: [[TMP76:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP12]]
; CHECK-NEXT: [[TMP71:%.*]] = load i32, i32* [[TMP70]] ; CHECK-NEXT: [[TMP77:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP13]]
; CHECK-NEXT: [[TMP72:%.*]] = insertelement <4 x i32> [[TMP68]], i32 [[TMP71]], i32 1 ; CHECK-NEXT: [[TMP78:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP14]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE8]] ; CHECK-NEXT: [[TMP79:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP15]]
; CHECK: pred.load.continue8: ; CHECK-NEXT: [[TMP80:%.*]] = load i32, i32* [[TMP64]]
; CHECK-NEXT: [[TMP73:%.*]] = phi <4 x i32> [ [[TMP68]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP72]], [[PRED_LOAD_IF7]] ] ; CHECK-NEXT: [[TMP81:%.*]] = load i32, i32* [[TMP65]]
; CHECK-NEXT: [[TMP74:%.*]] = extractelement <4 x i1> [[TMP39]], i32 2 ; CHECK-NEXT: [[TMP82:%.*]] = load i32, i32* [[TMP66]]
; CHECK-NEXT: br i1 [[TMP74]], label [[PRED_LOAD_IF9:%.*]], label [[PRED_LOAD_CONTINUE10:%.*]] ; CHECK-NEXT: [[TMP83:%.*]] = load i32, i32* [[TMP67]]
; CHECK: pred.load.if9: ; CHECK-NEXT: [[TMP84:%.*]] = insertelement <4 x i32> undef, i32 [[TMP80]], i32 0
; CHECK-NEXT: [[TMP75:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP2]] ; CHECK-NEXT: [[TMP85:%.*]] = insertelement <4 x i32> [[TMP84]], i32 [[TMP81]], i32 1
; CHECK-NEXT: [[TMP76:%.*]] = load i32, i32* [[TMP75]] ; CHECK-NEXT: [[TMP86:%.*]] = insertelement <4 x i32> [[TMP85]], i32 [[TMP82]], i32 2
; CHECK-NEXT: [[TMP77:%.*]] = insertelement <4 x i32> [[TMP73]], i32 [[TMP76]], i32 2 ; CHECK-NEXT: [[TMP87:%.*]] = insertelement <4 x i32> [[TMP86]], i32 [[TMP83]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE10]] ; CHECK-NEXT: [[TMP88:%.*]] = load i32, i32* [[TMP68]]
; CHECK: pred.load.continue10: ; CHECK-NEXT: [[TMP89:%.*]] = load i32, i32* [[TMP69]]
; CHECK-NEXT: [[TMP78:%.*]] = phi <4 x i32> [ [[TMP73]], [[PRED_LOAD_CONTINUE8]] ], [ [[TMP77]], [[PRED_LOAD_IF9]] ] ; CHECK-NEXT: [[TMP90:%.*]] = load i32, i32* [[TMP70]]
; CHECK-NEXT: [[TMP79:%.*]] = extractelement <4 x i1> [[TMP39]], i32 3 ; CHECK-NEXT: [[TMP91:%.*]] = load i32, i32* [[TMP71]]
; CHECK-NEXT: br i1 [[TMP79]], label [[PRED_LOAD_IF11:%.*]], label [[PRED_LOAD_CONTINUE12:%.*]] ; CHECK-NEXT: [[TMP92:%.*]] = insertelement <4 x i32> undef, i32 [[TMP88]], i32 0
; CHECK: pred.load.if11: ; CHECK-NEXT: [[TMP93:%.*]] = insertelement <4 x i32> [[TMP92]], i32 [[TMP89]], i32 1
; CHECK-NEXT: [[TMP80:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP3]] ; CHECK-NEXT: [[TMP94:%.*]] = insertelement <4 x i32> [[TMP93]], i32 [[TMP90]], i32 2
; CHECK-NEXT: [[TMP81:%.*]] = load i32, i32* [[TMP80]] ; CHECK-NEXT: [[TMP95:%.*]] = insertelement <4 x i32> [[TMP94]], i32 [[TMP91]], i32 3
; CHECK-NEXT: [[TMP82:%.*]] = insertelement <4 x i32> [[TMP78]], i32 [[TMP81]], i32 3 ; CHECK-NEXT: [[TMP96:%.*]] = load i32, i32* [[TMP72]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE12]] ; CHECK-NEXT: [[TMP97:%.*]] = load i32, i32* [[TMP73]]
; CHECK: pred.load.continue12: ; CHECK-NEXT: [[TMP98:%.*]] = load i32, i32* [[TMP74]]
; CHECK-NEXT: [[TMP83:%.*]] = phi <4 x i32> [ [[TMP78]], [[PRED_LOAD_CONTINUE10]] ], [ [[TMP82]], [[PRED_LOAD_IF11]] ] ; CHECK-NEXT: [[TMP99:%.*]] = load i32, i32* [[TMP75]]
; CHECK-NEXT: [[TMP84:%.*]] = extractelement <4 x i1> [[TMP47]], i32 0 ; CHECK-NEXT: [[TMP100:%.*]] = insertelement <4 x i32> undef, i32 [[TMP96]], i32 0
; CHECK-NEXT: br i1 [[TMP84]], label [[PRED_LOAD_IF13:%.*]], label [[PRED_LOAD_CONTINUE14:%.*]] ; CHECK-NEXT: [[TMP101:%.*]] = insertelement <4 x i32> [[TMP100]], i32 [[TMP97]], i32 1
; CHECK: pred.load.if13: ; CHECK-NEXT: [[TMP102:%.*]] = insertelement <4 x i32> [[TMP101]], i32 [[TMP98]], i32 2
; CHECK-NEXT: [[TMP85:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP4]] ; CHECK-NEXT: [[TMP103:%.*]] = insertelement <4 x i32> [[TMP102]], i32 [[TMP99]], i32 3
; CHECK-NEXT: [[TMP86:%.*]] = load i32, i32* [[TMP85]] ; CHECK-NEXT: [[TMP104:%.*]] = load i32, i32* [[TMP76]]
; CHECK-NEXT: [[TMP87:%.*]] = insertelement <4 x i32> undef, i32 [[TMP86]], i32 0 ; CHECK-NEXT: [[TMP105:%.*]] = load i32, i32* [[TMP77]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE14]] ; CHECK-NEXT: [[TMP106:%.*]] = load i32, i32* [[TMP78]]
; CHECK: pred.load.continue14: ; CHECK-NEXT: [[TMP107:%.*]] = load i32, i32* [[TMP79]]
; CHECK-NEXT: [[TMP88:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE12]] ], [ [[TMP87]], [[PRED_LOAD_IF13]] ] ; CHECK-NEXT: [[TMP108:%.*]] = insertelement <4 x i32> undef, i32 [[TMP104]], i32 0
; CHECK-NEXT: [[TMP89:%.*]] = extractelement <4 x i1> [[TMP47]], i32 1 ; CHECK-NEXT: [[TMP109:%.*]] = insertelement <4 x i32> [[TMP108]], i32 [[TMP105]], i32 1
; CHECK-NEXT: br i1 [[TMP89]], label [[PRED_LOAD_IF15:%.*]], label [[PRED_LOAD_CONTINUE16:%.*]] ; CHECK-NEXT: [[TMP110:%.*]] = insertelement <4 x i32> [[TMP109]], i32 [[TMP106]], i32 2
; CHECK: pred.load.if15: ; CHECK-NEXT: [[TMP111:%.*]] = insertelement <4 x i32> [[TMP110]], i32 [[TMP107]], i32 3
; CHECK-NEXT: [[TMP90:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP5]] ; CHECK-NEXT: [[TMP112:%.*]] = xor <4 x i1> [[TMP39]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP91:%.*]] = load i32, i32* [[TMP90]] ; CHECK-NEXT: [[TMP113:%.*]] = xor <4 x i1> [[TMP47]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP92:%.*]] = insertelement <4 x i32> [[TMP88]], i32 [[TMP91]], i32 1 ; CHECK-NEXT: [[TMP114:%.*]] = xor <4 x i1> [[TMP55]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE16]] ; CHECK-NEXT: [[TMP115:%.*]] = xor <4 x i1> [[TMP63]], <i1 true, i1 true, i1 true, i1 true>
; CHECK: pred.load.continue16: ; CHECK-NEXT: [[PREDPHI:%.*]] = select <4 x i1> [[TMP39]], <4 x i32> [[TMP87]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP93:%.*]] = phi <4 x i32> [ [[TMP88]], [[PRED_LOAD_CONTINUE14]] ], [ [[TMP92]], [[PRED_LOAD_IF15]] ] ; CHECK-NEXT: [[PREDPHI7:%.*]] = select <4 x i1> [[TMP47]], <4 x i32> [[TMP95]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP94:%.*]] = extractelement <4 x i1> [[TMP47]], i32 2 ; CHECK-NEXT: [[PREDPHI8:%.*]] = select <4 x i1> [[TMP55]], <4 x i32> [[TMP103]], <4 x i32> zeroinitializer
; CHECK-NEXT: br i1 [[TMP94]], label [[PRED_LOAD_IF17:%.*]], label [[PRED_LOAD_CONTINUE18:%.*]] ; CHECK-NEXT: [[PREDPHI9:%.*]] = select <4 x i1> [[TMP63]], <4 x i32> [[TMP111]], <4 x i32> zeroinitializer
; CHECK: pred.load.if17: ; CHECK-NEXT: [[TMP116]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
; CHECK-NEXT: [[TMP95:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP6]] ; CHECK-NEXT: [[TMP117]] = add <4 x i32> [[VEC_PHI4]], [[PREDPHI7]]
; CHECK-NEXT: [[TMP96:%.*]] = load i32, i32* [[TMP95]] ; CHECK-NEXT: [[TMP118]] = add <4 x i32> [[VEC_PHI5]], [[PREDPHI8]]
; CHECK-NEXT: [[TMP97:%.*]] = insertelement <4 x i32> [[TMP93]], i32 [[TMP96]], i32 2 ; CHECK-NEXT: [[TMP119]] = add <4 x i32> [[VEC_PHI6]], [[PREDPHI9]]
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE18]]
; CHECK: pred.load.continue18:
; CHECK-NEXT: [[TMP98:%.*]] = phi <4 x i32> [ [[TMP93]], [[PRED_LOAD_CONTINUE16]] ], [ [[TMP97]], [[PRED_LOAD_IF17]] ]
; CHECK-NEXT: [[TMP99:%.*]] = extractelement <4 x i1> [[TMP47]], i32 3
; CHECK-NEXT: br i1 [[TMP99]], label [[PRED_LOAD_IF19:%.*]], label [[PRED_LOAD_CONTINUE20:%.*]]
; CHECK: pred.load.if19:
; CHECK-NEXT: [[TMP100:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP7]]
; CHECK-NEXT: [[TMP101:%.*]] = load i32, i32* [[TMP100]]
; CHECK-NEXT: [[TMP102:%.*]] = insertelement <4 x i32> [[TMP98]], i32 [[TMP101]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE20]]
; CHECK: pred.load.continue20:
; CHECK-NEXT: [[TMP103:%.*]] = phi <4 x i32> [ [[TMP98]], [[PRED_LOAD_CONTINUE18]] ], [ [[TMP102]], [[PRED_LOAD_IF19]] ]
; CHECK-NEXT: [[TMP104:%.*]] = extractelement <4 x i1> [[TMP55]], i32 0
; CHECK-NEXT: br i1 [[TMP104]], label [[PRED_LOAD_IF21:%.*]], label [[PRED_LOAD_CONTINUE22:%.*]]
; CHECK: pred.load.if21:
; CHECK-NEXT: [[TMP105:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP8]]
; CHECK-NEXT: [[TMP106:%.*]] = load i32, i32* [[TMP105]]
; CHECK-NEXT: [[TMP107:%.*]] = insertelement <4 x i32> undef, i32 [[TMP106]], i32 0
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE22]]
; CHECK: pred.load.continue22:
; CHECK-NEXT: [[TMP108:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE20]] ], [ [[TMP107]], [[PRED_LOAD_IF21]] ]
; CHECK-NEXT: [[TMP109:%.*]] = extractelement <4 x i1> [[TMP55]], i32 1
; CHECK-NEXT: br i1 [[TMP109]], label [[PRED_LOAD_IF23:%.*]], label [[PRED_LOAD_CONTINUE24:%.*]]
; CHECK: pred.load.if23:
; CHECK-NEXT: [[TMP110:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP9]]
; CHECK-NEXT: [[TMP111:%.*]] = load i32, i32* [[TMP110]]
; CHECK-NEXT: [[TMP112:%.*]] = insertelement <4 x i32> [[TMP108]], i32 [[TMP111]], i32 1
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE24]]
; CHECK: pred.load.continue24:
; CHECK-NEXT: [[TMP113:%.*]] = phi <4 x i32> [ [[TMP108]], [[PRED_LOAD_CONTINUE22]] ], [ [[TMP112]], [[PRED_LOAD_IF23]] ]
; CHECK-NEXT: [[TMP114:%.*]] = extractelement <4 x i1> [[TMP55]], i32 2
; CHECK-NEXT: br i1 [[TMP114]], label [[PRED_LOAD_IF25:%.*]], label [[PRED_LOAD_CONTINUE26:%.*]]
; CHECK: pred.load.if25:
; CHECK-NEXT: [[TMP115:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP10]]
; CHECK-NEXT: [[TMP116:%.*]] = load i32, i32* [[TMP115]]
; CHECK-NEXT: [[TMP117:%.*]] = insertelement <4 x i32> [[TMP113]], i32 [[TMP116]], i32 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE26]]
; CHECK: pred.load.continue26:
; CHECK-NEXT: [[TMP118:%.*]] = phi <4 x i32> [ [[TMP113]], [[PRED_LOAD_CONTINUE24]] ], [ [[TMP117]], [[PRED_LOAD_IF25]] ]
; CHECK-NEXT: [[TMP119:%.*]] = extractelement <4 x i1> [[TMP55]], i32 3
; CHECK-NEXT: br i1 [[TMP119]], label [[PRED_LOAD_IF27:%.*]], label [[PRED_LOAD_CONTINUE28:%.*]]
; CHECK: pred.load.if27:
; CHECK-NEXT: [[TMP120:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP11]]
; CHECK-NEXT: [[TMP121:%.*]] = load i32, i32* [[TMP120]]
; CHECK-NEXT: [[TMP122:%.*]] = insertelement <4 x i32> [[TMP118]], i32 [[TMP121]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE28]]
; CHECK: pred.load.continue28:
; CHECK-NEXT: [[TMP123:%.*]] = phi <4 x i32> [ [[TMP118]], [[PRED_LOAD_CONTINUE26]] ], [ [[TMP122]], [[PRED_LOAD_IF27]] ]
; CHECK-NEXT: [[TMP124:%.*]] = extractelement <4 x i1> [[TMP63]], i32 0
; CHECK-NEXT: br i1 [[TMP124]], label [[PRED_LOAD_IF29:%.*]], label [[PRED_LOAD_CONTINUE30:%.*]]
; CHECK: pred.load.if29:
; CHECK-NEXT: [[TMP125:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP12]]
; CHECK-NEXT: [[TMP126:%.*]] = load i32, i32* [[TMP125]]
; CHECK-NEXT: [[TMP127:%.*]] = insertelement <4 x i32> undef, i32 [[TMP126]], i32 0
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE30]]
; CHECK: pred.load.continue30:
; CHECK-NEXT: [[TMP128:%.*]] = phi <4 x i32> [ undef, [[PRED_LOAD_CONTINUE28]] ], [ [[TMP127]], [[PRED_LOAD_IF29]] ]
; CHECK-NEXT: [[TMP129:%.*]] = extractelement <4 x i1> [[TMP63]], i32 1
; CHECK-NEXT: br i1 [[TMP129]], label [[PRED_LOAD_IF31:%.*]], label [[PRED_LOAD_CONTINUE32:%.*]]
; CHECK: pred.load.if31:
; CHECK-NEXT: [[TMP130:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP13]]
; CHECK-NEXT: [[TMP131:%.*]] = load i32, i32* [[TMP130]]
; CHECK-NEXT: [[TMP132:%.*]] = insertelement <4 x i32> [[TMP128]], i32 [[TMP131]], i32 1
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE32]]
; CHECK: pred.load.continue32:
; CHECK-NEXT: [[TMP133:%.*]] = phi <4 x i32> [ [[TMP128]], [[PRED_LOAD_CONTINUE30]] ], [ [[TMP132]], [[PRED_LOAD_IF31]] ]
; CHECK-NEXT: [[TMP134:%.*]] = extractelement <4 x i1> [[TMP63]], i32 2
; CHECK-NEXT: br i1 [[TMP134]], label [[PRED_LOAD_IF33:%.*]], label [[PRED_LOAD_CONTINUE34:%.*]]
; CHECK: pred.load.if33:
; CHECK-NEXT: [[TMP135:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP14]]
; CHECK-NEXT: [[TMP136:%.*]] = load i32, i32* [[TMP135]]
; CHECK-NEXT: [[TMP137:%.*]] = insertelement <4 x i32> [[TMP133]], i32 [[TMP136]], i32 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE34]]
; CHECK: pred.load.continue34:
; CHECK-NEXT: [[TMP138:%.*]] = phi <4 x i32> [ [[TMP133]], [[PRED_LOAD_CONTINUE32]] ], [ [[TMP137]], [[PRED_LOAD_IF33]] ]
; CHECK-NEXT: [[TMP139:%.*]] = extractelement <4 x i1> [[TMP63]], i32 3
; CHECK-NEXT: br i1 [[TMP139]], label [[PRED_LOAD_IF35:%.*]], label [[PRED_LOAD_CONTINUE36]]
; CHECK: pred.load.if35:
; CHECK-NEXT: [[TMP140:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[TMP15]]
; CHECK-NEXT: [[TMP141:%.*]] = load i32, i32* [[TMP140]]
; CHECK-NEXT: [[TMP142:%.*]] = insertelement <4 x i32> [[TMP138]], i32 [[TMP141]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE36]]
; CHECK: pred.load.continue36:
; CHECK-NEXT: [[TMP143:%.*]] = phi <4 x i32> [ [[TMP138]], [[PRED_LOAD_CONTINUE34]] ], [ [[TMP142]], [[PRED_LOAD_IF35]] ]
; CHECK-NEXT: [[TMP144:%.*]] = xor <4 x i1> [[TMP39]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP145:%.*]] = xor <4 x i1> [[TMP47]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP146:%.*]] = xor <4 x i1> [[TMP55]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[TMP147:%.*]] = xor <4 x i1> [[TMP63]], <i1 true, i1 true, i1 true, i1 true>
; CHECK-NEXT: [[PREDPHI:%.*]] = select <4 x i1> [[TMP39]], <4 x i32> [[TMP83]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI37:%.*]] = select <4 x i1> [[TMP47]], <4 x i32> [[TMP103]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI38:%.*]] = select <4 x i1> [[TMP55]], <4 x i32> [[TMP123]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[PREDPHI39:%.*]] = select <4 x i1> [[TMP63]], <4 x i32> [[TMP143]], <4 x i32> zeroinitializer
; CHECK-NEXT: [[TMP148]] = add <4 x i32> [[VEC_PHI]], [[PREDPHI]]
; CHECK-NEXT: [[TMP149]] = add <4 x i32> [[VEC_PHI4]], [[PREDPHI37]]
; CHECK-NEXT: [[TMP150]] = add <4 x i32> [[VEC_PHI5]], [[PREDPHI38]]
; CHECK-NEXT: [[TMP151]] = add <4 x i32> [[VEC_PHI6]], [[PREDPHI39]]
; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 16 ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 16
; CHECK-NEXT: [[TMP152:%.*]] = icmp eq i64 [[INDEX_NEXT]], 2048 ; CHECK-NEXT: [[TMP120:%.*]] = icmp eq i64 [[INDEX_NEXT]], 2048
; CHECK-NEXT: br i1 [[TMP152]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !14 ; CHECK-NEXT: br i1 [[TMP120]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !14
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[BIN_RDX:%.*]] = add <4 x i32> [[TMP149]], [[TMP148]] ; CHECK-NEXT: [[BIN_RDX:%.*]] = add <4 x i32> [[TMP117]], [[TMP116]]
; CHECK-NEXT: [[BIN_RDX40:%.*]] = add <4 x i32> [[TMP150]], [[BIN_RDX]] ; CHECK-NEXT: [[BIN_RDX10:%.*]] = add <4 x i32> [[TMP118]], [[BIN_RDX]]
; CHECK-NEXT: [[BIN_RDX41:%.*]] = add <4 x i32> [[TMP151]], [[BIN_RDX40]] ; CHECK-NEXT: [[BIN_RDX11:%.*]] = add <4 x i32> [[TMP119]], [[BIN_RDX10]]
; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x i32> [[BIN_RDX41]], <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef> ; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x i32> [[BIN_RDX11]], <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
; CHECK-NEXT: [[BIN_RDX42:%.*]] = add <4 x i32> [[BIN_RDX41]], [[RDX_SHUF]] ; CHECK-NEXT: [[BIN_RDX12:%.*]] = add <4 x i32> [[BIN_RDX11]], [[RDX_SHUF]]
; CHECK-NEXT: [[RDX_SHUF43:%.*]] = shufflevector <4 x i32> [[BIN_RDX42]], <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef> ; CHECK-NEXT: [[RDX_SHUF13:%.*]] = shufflevector <4 x i32> [[BIN_RDX12]], <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
; CHECK-NEXT: [[BIN_RDX44:%.*]] = add <4 x i32> [[BIN_RDX42]], [[RDX_SHUF43]] ; CHECK-NEXT: [[BIN_RDX14:%.*]] = add <4 x i32> [[BIN_RDX12]], [[RDX_SHUF13]]
; CHECK-NEXT: [[TMP153:%.*]] = extractelement <4 x i32> [[BIN_RDX44]], i32 0 ; CHECK-NEXT: [[TMP121:%.*]] = extractelement <4 x i32> [[BIN_RDX14]], i32 0
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 2048, 2048 ; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 2048, 2048
; CHECK-NEXT: br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]] ; CHECK-NEXT: br i1 [[CMP_N]], label [[LOOP_EXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 4096, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ] ; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 4096, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP153]], [[MIDDLE_BLOCK]] ] ; CHECK-NEXT: [[BC_MERGE_RDX:%.*]] = phi i32 [ 0, [[ENTRY]] ], [ [[TMP121]], [[MIDDLE_BLOCK]] ]
; CHECK-NEXT: br label [[LOOP:%.*]] ; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop: ; CHECK: loop:
; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ] ; CHECK-NEXT: [[IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[IV_NEXT:%.*]], [[LATCH:%.*]] ]
; CHECK-NEXT: [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ] ; CHECK-NEXT: [[ACCUM:%.*]] = phi i32 [ [[BC_MERGE_RDX]], [[SCALAR_PH]] ], [ [[ACCUM_NEXT:%.*]], [[LATCH]] ]
; CHECK-NEXT: [[IV_NEXT]] = add i64 [[IV]], 2 ; CHECK-NEXT: [[IV_NEXT]] = add i64 [[IV]], 2
; CHECK-NEXT: [[TEST_ADDR:%.*]] = getelementptr inbounds i1, i1* [[TEST_BASE]], i64 [[IV]] ; CHECK-NEXT: [[TEST_ADDR:%.*]] = getelementptr inbounds i1, i1* [[TEST_BASE]], i64 [[IV]]
; CHECK-NEXT: [[EARLYCND:%.*]] = load i1, i1* [[TEST_ADDR]] ; CHECK-NEXT: [[EARLYCND:%.*]] = load i1, i1* [[TEST_ADDR]]
; CHECK-NEXT: br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]] ; CHECK-NEXT: br i1 [[EARLYCND]], label [[PRED:%.*]], label [[LATCH]]
; CHECK: pred: ; CHECK: pred:
; CHECK-NEXT: [[ADDR:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[IV]] ; CHECK-NEXT: [[ADDR:%.*]] = getelementptr inbounds i32, i32* [[BASE]], i64 [[IV]]
; CHECK-NEXT: [[VAL:%.*]] = load i32, i32* [[ADDR]] ; CHECK-NEXT: [[VAL:%.*]] = load i32, i32* [[ADDR]]
; CHECK-NEXT: br label [[LATCH]] ; CHECK-NEXT: br label [[LATCH]]
; CHECK: latch: ; CHECK: latch:
; CHECK-NEXT: [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ] ; CHECK-NEXT: [[VAL_PHI:%.*]] = phi i32 [ 0, [[LOOP]] ], [ [[VAL]], [[PRED]] ]
; CHECK-NEXT: [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]] ; CHECK-NEXT: [[ACCUM_NEXT]] = add i32 [[ACCUM]], [[VAL_PHI]]
; CHECK-NEXT: [[EXIT:%.*]] = icmp ugt i64 [[IV]], 4093 ; CHECK-NEXT: [[EXIT:%.*]] = icmp ugt i64 [[IV]], 4093
; CHECK-NEXT: br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop !15 ; CHECK-NEXT: br i1 [[EXIT]], label [[LOOP_EXIT]], label [[LOOP]], !llvm.loop !15
; CHECK: loop_exit: ; CHECK: loop_exit:
; CHECK-NEXT: [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP153]], [[MIDDLE_BLOCK]] ] ; CHECK-NEXT: [[ACCUM_NEXT_LCSSA:%.*]] = phi i32 [ [[ACCUM_NEXT]], [[LATCH]] ], [ [[TMP121]], [[MIDDLE_BLOCK]] ]
; CHECK-NEXT: ret i32 [[ACCUM_NEXT_LCSSA]] ; CHECK-NEXT: ret i32 [[ACCUM_NEXT_LCSSA]]
; ;
entry: entry:
%alloca = alloca [4096 x i32] %alloca = alloca [4096 x i32]
%base = bitcast [4096 x i32]* %alloca to i32* %base = bitcast [4096 x i32]* %alloca to i32*
call void @init(i32* %base) call void @init(i32* %base)
br label %loop br label %loop
loop: loop:
▲ Show 20 Lines • Show All 552 Lines • Show Last 20 Lines