This is an archive of the discontinued LLVM Phabricator instance.

Differential D115712

[LV] Make sure prefer-predicate-over-epilogue works for short TC loops
Needs ReviewPublic

Authored by ebrevnov on Dec 14 2021, 1:37 AM.

Details

Reviewers
fhahn
anna
nikic
lebedev.ri
Summary

Currently "short trip count" heuristic overrides user settings of PreferPredicateOverEpilogue and llvm.loop.vectorize.predicate.enable. That doesn't seem right for several reasons. Typically user settings are expected to override compiler chosen heuristics. Another reason is surprising difference in behavior for "known" short trip count loops and all other cases.

PS: This change is exploited in https://reviews.llvm.org/D115713 to be able to keep testing coverage.

Diff Detail

Event Timeline

ebrevnov created this revision.Dec 14 2021, 1:37 AM
Herald added a subscriber: hiraditya. · View Herald TranscriptDec 14 2021, 1:37 AM
ebrevnov requested review of this revision.Dec 14 2021, 1:37 AM
Herald added a project: Restricted Project. · View Herald TranscriptDec 14 2021, 1:37 AM
Herald added a subscriber: llvm-commits. · View Herald Transcript
Harbormaster completed remote builds in B139184: Diff 394186.Dec 14 2021, 1:38 AM
ebrevnov mentioned this in D115711: [LV][NFC] New test case to check prefer-predicate-over-epilogue for short TC loops.Dec 14 2021, 1:41 AM
ebrevnov added a parent revision: D115711: [LV][NFC] New test case to check prefer-predicate-over-epilogue for short TC loops.
ebrevnov edited the summary of this revision. (Show Details)Dec 14 2021, 1:53 AM
ebrevnov added a reviewer: fhahn.
ebrevnov added a child revision: D115713: [LV] Don't apply "TinyTripCountVectorThreshold" for loops with compile time known TC..
xbolva00 added a subscriber: xbolva00.Jan 12 2022, 1:48 AM
ebrevnov added a comment.Jan 24 2022, 4:59 AM

Ping

ebrevnov added reviewers: anna, nikic, lebedev.ri.Oct 9 2022, 7:24 AM
Herald added a project: Restricted Project. · View Herald TranscriptOct 9 2022, 7:24 AM
Herald added a subscriber: • pcwang-thead. · View Herald Transcript
ebrevnov added a comment.Oct 9 2022, 7:24 AM

ping

ebrevnov updated this revision to Diff 467816.Oct 14 2022, 10:09 AM

rebase

ebrevnov removed a child revision: D115713: [LV] Don't apply "TinyTripCountVectorThreshold" for loops with compile time known TC..Oct 14 2022, 10:13 AM
ebrevnov added a child revision: D135971: [LV][NFC][WIP] Restructure getScalarEpilogueLowering.
Harbormaster completed remote builds in B192209: Diff 467816.Oct 14 2022, 11:06 AM
lebedev.ri resigned from this revision.Jan 12 2023, 5:31 PM

This review may be stuck/dead, consider abandoning if no longer relevant.
Removing myself as reviewer in attempt to clean dashboard.

Herald added a subscriber: StephenFan. · View Herald TranscriptJan 12 2023, 5:31 PM
nikic resigned from this revision.Jan 20 2023, 12:33 PM

(not familiar with LV)

Revision Contents

PathSize
llvm/
lib/
Transforms/
Vectorize/
76 lines
test/
Transforms/
LoopVectorize/
633 lines

Diff 467816

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 9,826 Lines • ▼ Show 20 Lines for (unsigned Part = 0; Part < State.UF; ++Part) {
State.set(getVPSingleValue(), NewLI, Part); State.set(getVPSingleValue(), NewLI, Part);
} }
} }
// Determine how to lower the scalar epilogue, which depends on 1) optimising // Determine how to lower the scalar epilogue, which depends on 1) optimising
// for minimum code-size, 2) predicate compiler options, 3) loop hints forcing // for minimum code-size, 2) predicate compiler options, 3) loop hints forcing
// predication, and 4) a TTI hook that analyses whether the loop is suitable // predication, and 4) a TTI hook that analyses whether the loop is suitable
// for predication. // for predication.
static ScalarEpilogueLowering getScalarEpilogueLowering( static Optional<ScalarEpilogueLowering> getScalarEpilogueLowering(
Function *F, Loop *L, LoopVectorizeHints &Hints, ProfileSummaryInfo *PSI, Function *F, Loop *L, LoopVectorizeHints &Hints, ProfileSummaryInfo *PSI,
BlockFrequencyInfo *BFI, TargetTransformInfo *TTI, TargetLibraryInfo *TLI, BlockFrequencyInfo *BFI, TargetTransformInfo *TTI, TargetLibraryInfo *TLI,
AssumptionCache *AC, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT, AssumptionCache *AC, LoopInfo *LI, ScalarEvolution *SE, DominatorTree *DT,
LoopVectorizationLegality &LVL, InterleavedAccessInfo *IAI) { LoopVectorizationLegality &LVL, InterleavedAccessInfo *IAI,
OptimizationRemarkEmitter *ORE) {
// 1) OptSize takes precedence over all other options, i.e. if this is set, // 1) OptSize takes precedence over all other options, i.e. if this is set,
// don't look at hints or options, and don't request a scalar epilogue. // don't look at hints or options, and don't request a scalar epilogue.
// (For PGSO, as shouldOptimizeForSize isn't currently accessible from // (For PGSO, as shouldOptimizeForSize isn't currently accessible from
// LoopAccessInfo (due to code dependency and not being able to reliably get // LoopAccessInfo (due to code dependency and not being able to reliably get
// PSI/BFI from a loop analysis under NPM), we cannot suppress the collection // PSI/BFI from a loop analysis under NPM), we cannot suppress the collection
// of strides in LoopAccessInfo::analyzeLoop() and vectorize without // of strides in LoopAccessInfo::analyzeLoop() and vectorize without
// versioning when the vectorization is forced, unlike hasOptSize. So revert // versioning when the vectorization is forced, unlike hasOptSize. So revert
// back to the old way and vectorize with versioning when forced. See D81345.) // back to the old way and vectorize with versioning when forced. See D81345.)
Show All 17 Linesstatic Optional<ScalarEpilogueLowering> getScalarEpilogueLowering(
// 3) If set, obey the hints // 3) If set, obey the hints
switch (Hints.getPredicate()) { switch (Hints.getPredicate()) {
case LoopVectorizeHints::FK_Enabled: case LoopVectorizeHints::FK_Enabled:
return CM_ScalarEpilogueNotNeededUsePredicate; return CM_ScalarEpilogueNotNeededUsePredicate;
case LoopVectorizeHints::FK_Disabled: case LoopVectorizeHints::FK_Disabled:
return CM_ScalarEpilogueAllowed; return CM_ScalarEpilogueAllowed;
}; };
// 4) if the TTI hook indicates this is profitable, request predication. // 4) Check the loop for a trip count threshold. Vectorize loops with a tiny
// trip count by optimizing for size, to minimize overheads.
auto ExpectedTC = getSmallBestKnownTC(*SE, L);
if (ExpectedTC && *ExpectedTC < TinyTripCountVectorThreshold) {
LLVM_DEBUG(dbgs() << "LV: Found a loop with a very small trip count. "
<< "This loop is worth vectorizing only if no scalar "
<< "iteration overheads are incurred.");
if (Hints.getForce() == LoopVectorizeHints::FK_Enabled)
LLVM_DEBUG(dbgs() << " But vectorizing was explicitly forced.\n");
else {
if (*ExpectedTC > TTI->getMinTripCountTailFoldingThreshold()) {
return CM_ScalarEpilogueNotAllowedLowTripLoop;
} else {
LLVM_DEBUG(dbgs() << " But the target considers the trip count too "
"small to consider vectorizing.\n");
reportVectorizationFailure(
"The trip count is below the minimal threshold value.",
"loop trip count is too low, avoiding vectorization",
"LowTripCount", ORE, L);
return None;
}
}
}
// 5) if the TTI hook indicates this is profitable, request predication.
if (TTI->preferPredicateOverEpilogue(L, LI, *SE, *AC, TLI, DT, &LVL, IAI)) if (TTI->preferPredicateOverEpilogue(L, LI, *SE, *AC, TLI, DT, &LVL, IAI))
return CM_ScalarEpilogueNotNeededUsePredicate; return CM_ScalarEpilogueNotNeededUsePredicate;
return CM_ScalarEpilogueAllowed; return CM_ScalarEpilogueAllowed;
} }
Value *VPTransformState::get(VPValue *Def, unsigned Part) { Value *VPTransformState::get(VPValue *Def, unsigned Part) {
// If Values have been set for this Def return the one relevant for \p Part. // If Values have been set for this Def return the one relevant for \p Part.
▲ Show 20 Lines • Show All 77 Lines • ▼ Show 20 Linesstatic bool processLoopInVPlanNativePath(
if (isa<SCEVCouldNotCompute>(PSE.getBackedgeTakenCount())) { if (isa<SCEVCouldNotCompute>(PSE.getBackedgeTakenCount())) {
LLVM_DEBUG(dbgs() << "LV: cannot compute the outer-loop trip count\n"); LLVM_DEBUG(dbgs() << "LV: cannot compute the outer-loop trip count\n");
return false; return false;
} }
assert(EnableVPlanNativePath && "VPlan-native path is disabled."); assert(EnableVPlanNativePath && "VPlan-native path is disabled.");
Function *F = L->getHeader()->getParent(); Function *F = L->getHeader()->getParent();
InterleavedAccessInfo IAI(PSE, L, DT, LI, LVL->getLAI()); InterleavedAccessInfo IAI(PSE, L, DT, LI, LVL->getLAI());
ScalarEpilogueLowering SEL = getScalarEpilogueLowering( Optional<ScalarEpilogueLowering> SEL =
F, L, Hints, PSI, BFI, TTI, TLI, AC, LI, PSE.getSE(), DT, *LVL, &IAI); getScalarEpilogueLowering(F, L, Hints, PSI, BFI, TTI, TLI, AC, LI,
PSE.getSE(), DT, *LVL, &IAI, ORE);
LoopVectorizationCostModel CM(SEL, L, PSE, LI, LVL, *TTI, TLI, DB, AC, ORE, F, if (!SEL) {
Hints.emitRemarkWithHints();
return false;
}
LoopVectorizationCostModel CM(*SEL, L, PSE, LI, LVL, *TTI, TLI, DB, AC, ORE, F,
&Hints, IAI); &Hints, IAI);
// Use the planner for outer loop vectorization. // Use the planner for outer loop vectorization.
// TODO: CM is not used at this point inside the planner. Turn CM into an // TODO: CM is not used at this point inside the planner. Turn CM into an
// optional argument if we don't need it in the future. // optional argument if we don't need it in the future.
LoopVectorizationPlanner LVP(L, LI, TLI, TTI, LVL, CM, IAI, PSE, Hints, ORE); LoopVectorizationPlanner LVP(L, LI, TLI, TTI, LVL, CM, IAI, PSE, Hints, ORE);
// Get user vectorization factor. // Get user vectorization factor.
ElementCount UserVF = Hints.getWidth(); ElementCount UserVF = Hints.getWidth();
▲ Show 20 Lines • Show All 249 Lines • ▼ Show 20 Lines if (EnableInterleavedMemAccesses.getNumOccurrences() > 0)
UseInterleaved = EnableInterleavedMemAccesses; UseInterleaved = EnableInterleavedMemAccesses;
// Analyze interleaved memory accesses. // Analyze interleaved memory accesses.
if (UseInterleaved) if (UseInterleaved)
IAI.analyzeInterleaving(useMaskedInterleavedAccesses(*TTI)); IAI.analyzeInterleaving(useMaskedInterleavedAccesses(*TTI));
// Check the function attributes and profiles to find out if this function // Check the function attributes and profiles to find out if this function
// should be optimized for size. // should be optimized for size.
ScalarEpilogueLowering SEL = getScalarEpilogueLowering( Optional<ScalarEpilogueLowering> SEL = getScalarEpilogueLowering(
F, L, Hints, PSI, BFI, TTI, TLI, AC, LI, PSE.getSE(), DT, LVL, &IAI); F, L, Hints, PSI, BFI, TTI, TLI, AC, LI, PSE.getSE(), DT, LVL, &IAI, ORE);
// Check the loop for a trip count threshold: vectorize loops with a tiny trip if (!SEL) {
// count by optimizing for size, to minimize overheads.
auto ExpectedTC = getSmallBestKnownTC(*SE, L);
if (ExpectedTC && *ExpectedTC < TinyTripCountVectorThreshold) {
LLVM_DEBUG(dbgs() << "LV: Found a loop with a very small trip count. "
<< "This loop is worth vectorizing only if no scalar "
<< "iteration overheads are incurred.");
if (Hints.getForce() == LoopVectorizeHints::FK_Enabled)
LLVM_DEBUG(dbgs() << " But vectorizing was explicitly forced.\n");
else {
if (*ExpectedTC > TTI->getMinTripCountTailFoldingThreshold()) {
LLVM_DEBUG(dbgs() << "\n");
SEL = CM_ScalarEpilogueNotAllowedLowTripLoop;
} else {
LLVM_DEBUG(dbgs() << " But the target considers the trip count too "
"small to consider vectorizing.\n");
reportVectorizationFailure(
"The trip count is below the minial threshold value.",
"loop trip count is too low, avoiding vectorization",
"LowTripCount", ORE, L);
Hints.emitRemarkWithHints(); Hints.emitRemarkWithHints();
return false; return false;
} }
}
}
// Check the function attributes to see if implicit floats or vectors are // Check the function attributes to see if implicit floats or vectors are
// allowed. // allowed.
if (F->hasFnAttribute(Attribute::NoImplicitFloat)) { if (F->hasFnAttribute(Attribute::NoImplicitFloat)) {
reportVectorizationFailure( reportVectorizationFailure(
"Can't vectorize when the NoImplicitFloat attribute is used", "Can't vectorize when the NoImplicitFloat attribute is used",
"loop not vectorized due to NoImplicitFloat attribute", "loop not vectorized due to NoImplicitFloat attribute",
"NoImplicitFloat", ORE, L); "NoImplicitFloat", ORE, L);
Show All 32 Lines if (!LVL.canVectorizeFPMath(AllowOrderedReductions)) {
}); });
LLVM_DEBUG(dbgs() << "LV: loop not vectorized: cannot prove it is safe to " LLVM_DEBUG(dbgs() << "LV: loop not vectorized: cannot prove it is safe to "
"reorder floating-point operations\n"); "reorder floating-point operations\n");
Hints.emitRemarkWithHints(); Hints.emitRemarkWithHints();
return false; return false;
} }
// Use the cost model. // Use the cost model.
LoopVectorizationCostModel CM(SEL, L, PSE, LI, &LVL, *TTI, TLI, DB, AC, ORE, LoopVectorizationCostModel CM(*SEL, L, PSE, LI, &LVL, *TTI, TLI, DB, AC, ORE,
F, &Hints, IAI); F, &Hints, IAI);
CM.collectValuesToIgnore(); CM.collectValuesToIgnore();
CM.collectElementTypesForWidening(); CM.collectElementTypesForWidening();
// Use the planner for vectorization. // Use the planner for vectorization.
LoopVectorizationPlanner LVP(L, LI, TLI, TTI, &LVL, CM, IAI, PSE, Hints, ORE); LoopVectorizationPlanner LVP(L, LI, TLI, TTI, &LVL, CM, IAI, PSE, Hints, ORE);
// Get user vectorization factor and interleave count. // Get user vectorization factor and interleave count.
▲ Show 20 Lines • Show All 365 Lines • Show Last 20 Lines

llvm/test/Transforms/LoopVectorize/override-short-tc-heuristic.ll

; NOTE: Assertions have been autogenerated by utils/update_test_checks.py ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt < %s -passes=loop-vectorize -S | FileCheck %s ; RUN: opt < %s -passes=loop-vectorize -S | FileCheck %s
; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=scalar-epilogue -S | FileCheck -check-prefix CHECK-SCALAR %s ; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=scalar-epilogue -S | FileCheck -check-prefix CHECK-SCALAR %s
; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue -S | FileCheck -check-prefix CHECK-PRED-OR-SCALAR %s ; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue -S | FileCheck -check-prefix CHECK-PRED-OR-SCALAR %s
; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=predicate-dont-vectorize -S | FileCheck -check-prefix PRED-OR-DONTVEC %s ; RUN: opt < %s -passes=loop-vectorize -prefer-predicate-over-epilogue=predicate-dont-vectorize -S | FileCheck -check-prefix PRED-OR-DONTVEC %s
;@src = common global [32 x i8] zeroinitializer, align 1 ;@src = common global [32 x i8] zeroinitializer, align 1
;@dst = common global [32 x i8] zeroinitializer, align 1 ;@dst = common global [32 x i8] zeroinitializer, align 1
; Simple loop with small constant trip count. ; Simple loop with small constant trip count.
; TODO: Check that -prefer-predicate-over-epilogue and "llvm.loop.vectorize.predicate.enable=true" ; Check that -prefer-predicate-over-epilogue and "llvm.loop.vectorize.predicate.enable=true"
; hint override "small trip count" heuristic. ; hint override "small trip count" heuristic.
define i32 @const_low_trip_count_hint_pred(i8 *%dst, i8 *%src) { define i32 @const_low_trip_count_hint_pred(i8 *%dst, i8 *%src) {
; CHECK-LABEL: @const_low_trip_count_hint_pred( ; CHECK-LABEL: @const_low_trip_count_hint_pred(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK: vector.memcheck:
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE14:%.*]] ]
; CHECK-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE14]] ]
; CHECK-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-NEXT: [[TMP2:%.*]] = add i32 [[INDEX]], 1
; CHECK-NEXT: [[TMP3:%.*]] = add i32 [[INDEX]], 2
; CHECK-NEXT: [[TMP4:%.*]] = add i32 [[INDEX]], 3
; CHECK-NEXT: [[TMP5:%.*]] = icmp ule <4 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8>
; CHECK-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; CHECK: pred.load.if:
; CHECK-NEXT: [[TMP7:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-NEXT: [[TMP8:%.*]] = load i8, i8* [[TMP7]], align 1
; CHECK-NEXT: [[TMP9:%.*]] = insertelement <4 x i8> poison, i8 [[TMP8]], i32 0
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE]]
; CHECK: pred.load.continue:
; CHECK-NEXT: [[TMP10:%.*]] = phi <4 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
; CHECK-NEXT: [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-NEXT: br i1 [[TMP11]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
; CHECK: pred.load.if3:
; CHECK-NEXT: [[TMP12:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP2]]
; CHECK-NEXT: [[TMP13:%.*]] = load i8, i8* [[TMP12]], align 1
; CHECK-NEXT: [[TMP14:%.*]] = insertelement <4 x i8> [[TMP10]], i8 [[TMP13]], i32 1
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE4]]
; CHECK: pred.load.continue4:
; CHECK-NEXT: [[TMP15:%.*]] = phi <4 x i8> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF3]] ]
; CHECK-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-NEXT: br i1 [[TMP16]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
; CHECK: pred.load.if5:
; CHECK-NEXT: [[TMP17:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP3]]
; CHECK-NEXT: [[TMP18:%.*]] = load i8, i8* [[TMP17]], align 1
; CHECK-NEXT: [[TMP19:%.*]] = insertelement <4 x i8> [[TMP15]], i8 [[TMP18]], i32 2
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE6]]
; CHECK: pred.load.continue6:
; CHECK-NEXT: [[TMP20:%.*]] = phi <4 x i8> [ [[TMP15]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP19]], [[PRED_LOAD_IF5]] ]
; CHECK-NEXT: [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-NEXT: br i1 [[TMP21]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; CHECK: pred.load.if7:
; CHECK-NEXT: [[TMP22:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP4]]
; CHECK-NEXT: [[TMP23:%.*]] = load i8, i8* [[TMP22]], align 1
; CHECK-NEXT: [[TMP24:%.*]] = insertelement <4 x i8> [[TMP20]], i8 [[TMP23]], i32 3
; CHECK-NEXT: br label [[PRED_LOAD_CONTINUE8]]
; CHECK: pred.load.continue8:
; CHECK-NEXT: [[TMP25:%.*]] = phi <4 x i8> [ [[TMP20]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP24]], [[PRED_LOAD_IF7]] ]
; CHECK-NEXT: [[TMP26:%.*]] = icmp eq <4 x i8> [[TMP25]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-NEXT: [[TMP27:%.*]] = select <4 x i1> [[TMP26]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-NEXT: [[TMP28:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-NEXT: br i1 [[TMP28]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
; CHECK: pred.store.if:
; CHECK-NEXT: [[TMP29:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-NEXT: [[TMP30:%.*]] = extractelement <4 x i8> [[TMP27]], i32 0
; CHECK-NEXT: store i8 [[TMP30]], i8* [[TMP29]], align 1
; CHECK-NEXT: br label [[PRED_STORE_CONTINUE]]
; CHECK: pred.store.continue:
; CHECK-NEXT: [[TMP31:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-NEXT: br i1 [[TMP31]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10:%.*]]
; CHECK: pred.store.if9:
; CHECK-NEXT: [[TMP32:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP2]]
; CHECK-NEXT: [[TMP33:%.*]] = extractelement <4 x i8> [[TMP27]], i32 1
; CHECK-NEXT: store i8 [[TMP33]], i8* [[TMP32]], align 1
; CHECK-NEXT: br label [[PRED_STORE_CONTINUE10]]
; CHECK: pred.store.continue10:
; CHECK-NEXT: [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-NEXT: br i1 [[TMP34]], label [[PRED_STORE_IF11:%.*]], label [[PRED_STORE_CONTINUE12:%.*]]
; CHECK: pred.store.if11:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP3]]
; CHECK-NEXT: [[TMP36:%.*]] = extractelement <4 x i8> [[TMP27]], i32 2
; CHECK-NEXT: store i8 [[TMP36]], i8* [[TMP35]], align 1
; CHECK-NEXT: br label [[PRED_STORE_CONTINUE12]]
; CHECK: pred.store.continue12:
; CHECK-NEXT: [[TMP37:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-NEXT: br i1 [[TMP37]], label [[PRED_STORE_IF13:%.*]], label [[PRED_STORE_CONTINUE14]]
; CHECK: pred.store.if13:
; CHECK-NEXT: [[TMP38:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP4]]
; CHECK-NEXT: [[TMP39:%.*]] = extractelement <4 x i8> [[TMP27]], i32 3
; CHECK-NEXT: store i8 [[TMP39]], i8* [[TMP38]], align 1
; CHECK-NEXT: br label [[PRED_STORE_CONTINUE14]]
; CHECK: pred.store.continue14:
; CHECK-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
; CHECK-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; CHECK-NEXT: [[TMP40:%.*]] = icmp eq i32 [[INDEX_NEXT]], 12
; CHECK-NEXT: br i1 [[TMP40]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 12, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
; CHECK: for.body: ; CHECK: for.body:
; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP0:![0-9]+]] ; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
; CHECK: for.end: ; CHECK: for.end:
; CHECK-NEXT: ret i32 0 ; CHECK-NEXT: ret i32 0
; ;
; CHECK-SCALAR-LABEL: @const_low_trip_count_hint_pred( ; CHECK-SCALAR-LABEL: @const_low_trip_count_hint_pred(
; CHECK-SCALAR-NEXT: entry: ; CHECK-SCALAR-NEXT: entry:
; CHECK-SCALAR-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-SCALAR-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-SCALAR-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK-SCALAR: vector.memcheck:
; CHECK-SCALAR-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-SCALAR-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-SCALAR-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK-SCALAR: vector.ph:
; CHECK-SCALAR-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK-SCALAR: vector.body:
; CHECK-SCALAR-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-SCALAR-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-SCALAR-NEXT: [[TMP2:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-SCALAR-NEXT: [[TMP3:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-SCALAR-NEXT: [[TMP4:%.*]] = getelementptr i8, i8* [[TMP2]], i32 0
; CHECK-SCALAR-NEXT: [[TMP5:%.*]] = bitcast i8* [[TMP4]] to <4 x i8>*
; CHECK-SCALAR-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, <4 x i8>* [[TMP5]], align 1
; CHECK-SCALAR-NEXT: [[TMP6:%.*]] = icmp eq <4 x i8> [[WIDE_LOAD]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-SCALAR-NEXT: [[TMP7:%.*]] = select <4 x i1> [[TMP6]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-SCALAR-NEXT: [[TMP8:%.*]] = getelementptr i8, i8* [[TMP3]], i32 0
; CHECK-SCALAR-NEXT: [[TMP9:%.*]] = bitcast i8* [[TMP8]] to <4 x i8>*
; CHECK-SCALAR-NEXT: store <4 x i8> [[TMP7]], <4 x i8>* [[TMP9]], align 1
; CHECK-SCALAR-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
; CHECK-SCALAR-NEXT: [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], 8
; CHECK-SCALAR-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; CHECK-SCALAR: middle.block:
; CHECK-SCALAR-NEXT: [[CMP_N:%.*]] = icmp eq i32 9, 8
; CHECK-SCALAR-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK-SCALAR: scalar.ph:
; CHECK-SCALAR-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 8, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-SCALAR-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-SCALAR-NEXT: br label [[FOR_BODY:%.*]]
; CHECK-SCALAR: for.body: ; CHECK-SCALAR: for.body:
; CHECK-SCALAR-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-SCALAR-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP0:![0-9]+]] ; CHECK-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
; CHECK-SCALAR: for.end: ; CHECK-SCALAR: for.end:
; CHECK-SCALAR-NEXT: ret i32 0 ; CHECK-SCALAR-NEXT: ret i32 0
; ;
; CHECK-PRED-OR-SCALAR-LABEL: @const_low_trip_count_hint_pred( ; CHECK-PRED-OR-SCALAR-LABEL: @const_low_trip_count_hint_pred(
; CHECK-PRED-OR-SCALAR-NEXT: entry: ; CHECK-PRED-OR-SCALAR-NEXT: entry:
; CHECK-PRED-OR-SCALAR-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-PRED-OR-SCALAR-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-PRED-OR-SCALAR-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK-PRED-OR-SCALAR: vector.memcheck:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK-PRED-OR-SCALAR: vector.ph:
; CHECK-PRED-OR-SCALAR-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK-PRED-OR-SCALAR: vector.body:
; CHECK-PRED-OR-SCALAR-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE14:%.*]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE14]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP2:%.*]] = add i32 [[INDEX]], 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP3:%.*]] = add i32 [[INDEX]], 2
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP4:%.*]] = add i32 [[INDEX]], 3
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP5:%.*]] = icmp ule <4 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP7:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP8:%.*]] = load i8, i8* [[TMP7]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP9:%.*]] = insertelement <4 x i8> poison, i8 [[TMP8]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE]]
; CHECK-PRED-OR-SCALAR: pred.load.continue:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP10:%.*]] = phi <4 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP11]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if3:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP12:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP13:%.*]] = load i8, i8* [[TMP12]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP14:%.*]] = insertelement <4 x i8> [[TMP10]], i8 [[TMP13]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE4]]
; CHECK-PRED-OR-SCALAR: pred.load.continue4:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP15:%.*]] = phi <4 x i8> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF3]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP16]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if5:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP17:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP3]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP18:%.*]] = load i8, i8* [[TMP17]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP19:%.*]] = insertelement <4 x i8> [[TMP15]], i8 [[TMP18]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE6]]
; CHECK-PRED-OR-SCALAR: pred.load.continue6:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP20:%.*]] = phi <4 x i8> [ [[TMP15]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP19]], [[PRED_LOAD_IF5]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP21]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if7:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP22:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP4]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP23:%.*]] = load i8, i8* [[TMP22]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP24:%.*]] = insertelement <4 x i8> [[TMP20]], i8 [[TMP23]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE8]]
; CHECK-PRED-OR-SCALAR: pred.load.continue8:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP25:%.*]] = phi <4 x i8> [ [[TMP20]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP24]], [[PRED_LOAD_IF7]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP26:%.*]] = icmp eq <4 x i8> [[TMP25]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP27:%.*]] = select <4 x i1> [[TMP26]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP28:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP28]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP29:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP30:%.*]] = extractelement <4 x i8> [[TMP27]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP30]], i8* [[TMP29]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE]]
; CHECK-PRED-OR-SCALAR: pred.store.continue:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP31:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP31]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if9:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP32:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP33:%.*]] = extractelement <4 x i8> [[TMP27]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP33]], i8* [[TMP32]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE10]]
; CHECK-PRED-OR-SCALAR: pred.store.continue10:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP34]], label [[PRED_STORE_IF11:%.*]], label [[PRED_STORE_CONTINUE12:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if11:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP35:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP3]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP36:%.*]] = extractelement <4 x i8> [[TMP27]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP36]], i8* [[TMP35]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE12]]
; CHECK-PRED-OR-SCALAR: pred.store.continue12:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP37:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP37]], label [[PRED_STORE_IF13:%.*]], label [[PRED_STORE_CONTINUE14]]
; CHECK-PRED-OR-SCALAR: pred.store.if13:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP38:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP4]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP39:%.*]] = extractelement <4 x i8> [[TMP27]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP39]], i8* [[TMP38]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE14]]
; CHECK-PRED-OR-SCALAR: pred.store.continue14:
; CHECK-PRED-OR-SCALAR-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
; CHECK-PRED-OR-SCALAR-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP40:%.*]] = icmp eq i32 [[INDEX_NEXT]], 12
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP40]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; CHECK-PRED-OR-SCALAR: middle.block:
; CHECK-PRED-OR-SCALAR-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK-PRED-OR-SCALAR: scalar.ph:
; CHECK-PRED-OR-SCALAR-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 12, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-PRED-OR-SCALAR-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-PRED-OR-SCALAR-NEXT: br label [[FOR_BODY:%.*]]
; CHECK-PRED-OR-SCALAR: for.body: ; CHECK-PRED-OR-SCALAR: for.body:
; CHECK-PRED-OR-SCALAR-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-PRED-OR-SCALAR-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-PRED-OR-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-PRED-OR-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-PRED-OR-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-PRED-OR-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-PRED-OR-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-PRED-OR-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-PRED-OR-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-PRED-OR-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-PRED-OR-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-PRED-OR-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-PRED-OR-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP0:![0-9]+]] ; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
; CHECK-PRED-OR-SCALAR: for.end: ; CHECK-PRED-OR-SCALAR: for.end:
; CHECK-PRED-OR-SCALAR-NEXT: ret i32 0 ; CHECK-PRED-OR-SCALAR-NEXT: ret i32 0
; ;
; PRED-OR-DONTVEC-LABEL: @const_low_trip_count_hint_pred( ; PRED-OR-DONTVEC-LABEL: @const_low_trip_count_hint_pred(
; PRED-OR-DONTVEC-NEXT: entry: ; PRED-OR-DONTVEC-NEXT: entry:
; PRED-OR-DONTVEC-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; PRED-OR-DONTVEC-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; PRED-OR-DONTVEC-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; PRED-OR-DONTVEC: vector.memcheck:
; PRED-OR-DONTVEC-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; PRED-OR-DONTVEC-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; PRED-OR-DONTVEC-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; PRED-OR-DONTVEC: vector.ph:
; PRED-OR-DONTVEC-NEXT: br label [[VECTOR_BODY:%.*]]
; PRED-OR-DONTVEC: vector.body:
; PRED-OR-DONTVEC-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE14:%.*]] ]
; PRED-OR-DONTVEC-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE14]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; PRED-OR-DONTVEC-NEXT: [[TMP2:%.*]] = add i32 [[INDEX]], 1
; PRED-OR-DONTVEC-NEXT: [[TMP3:%.*]] = add i32 [[INDEX]], 2
; PRED-OR-DONTVEC-NEXT: [[TMP4:%.*]] = add i32 [[INDEX]], 3
; PRED-OR-DONTVEC-NEXT: [[TMP5:%.*]] = icmp ule <4 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8>
; PRED-OR-DONTVEC-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; PRED-OR-DONTVEC: pred.load.if:
; PRED-OR-DONTVEC-NEXT: [[TMP7:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; PRED-OR-DONTVEC-NEXT: [[TMP8:%.*]] = load i8, i8* [[TMP7]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP9:%.*]] = insertelement <4 x i8> poison, i8 [[TMP8]], i32 0
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE]]
; PRED-OR-DONTVEC: pred.load.continue:
; PRED-OR-DONTVEC-NEXT: [[TMP10:%.*]] = phi <4 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP11]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
; PRED-OR-DONTVEC: pred.load.if3:
; PRED-OR-DONTVEC-NEXT: [[TMP12:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP2]]
; PRED-OR-DONTVEC-NEXT: [[TMP13:%.*]] = load i8, i8* [[TMP12]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP14:%.*]] = insertelement <4 x i8> [[TMP10]], i8 [[TMP13]], i32 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE4]]
; PRED-OR-DONTVEC: pred.load.continue4:
; PRED-OR-DONTVEC-NEXT: [[TMP15:%.*]] = phi <4 x i8> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF3]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP16]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
; PRED-OR-DONTVEC: pred.load.if5:
; PRED-OR-DONTVEC-NEXT: [[TMP17:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP3]]
; PRED-OR-DONTVEC-NEXT: [[TMP18:%.*]] = load i8, i8* [[TMP17]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP19:%.*]] = insertelement <4 x i8> [[TMP15]], i8 [[TMP18]], i32 2
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE6]]
; PRED-OR-DONTVEC: pred.load.continue6:
; PRED-OR-DONTVEC-NEXT: [[TMP20:%.*]] = phi <4 x i8> [ [[TMP15]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP19]], [[PRED_LOAD_IF5]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP21]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; PRED-OR-DONTVEC: pred.load.if7:
; PRED-OR-DONTVEC-NEXT: [[TMP22:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP4]]
; PRED-OR-DONTVEC-NEXT: [[TMP23:%.*]] = load i8, i8* [[TMP22]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP24:%.*]] = insertelement <4 x i8> [[TMP20]], i8 [[TMP23]], i32 3
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE8]]
; PRED-OR-DONTVEC: pred.load.continue8:
; PRED-OR-DONTVEC-NEXT: [[TMP25:%.*]] = phi <4 x i8> [ [[TMP20]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP24]], [[PRED_LOAD_IF7]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP26:%.*]] = icmp eq <4 x i8> [[TMP25]], <i8 5, i8 5, i8 5, i8 5>
; PRED-OR-DONTVEC-NEXT: [[TMP27:%.*]] = select <4 x i1> [[TMP26]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; PRED-OR-DONTVEC-NEXT: [[TMP28:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP28]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
; PRED-OR-DONTVEC: pred.store.if:
; PRED-OR-DONTVEC-NEXT: [[TMP29:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; PRED-OR-DONTVEC-NEXT: [[TMP30:%.*]] = extractelement <4 x i8> [[TMP27]], i32 0
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP30]], i8* [[TMP29]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE]]
; PRED-OR-DONTVEC: pred.store.continue:
; PRED-OR-DONTVEC-NEXT: [[TMP31:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP31]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10:%.*]]
; PRED-OR-DONTVEC: pred.store.if9:
; PRED-OR-DONTVEC-NEXT: [[TMP32:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP2]]
; PRED-OR-DONTVEC-NEXT: [[TMP33:%.*]] = extractelement <4 x i8> [[TMP27]], i32 1
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP33]], i8* [[TMP32]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE10]]
; PRED-OR-DONTVEC: pred.store.continue10:
; PRED-OR-DONTVEC-NEXT: [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP34]], label [[PRED_STORE_IF11:%.*]], label [[PRED_STORE_CONTINUE12:%.*]]
; PRED-OR-DONTVEC: pred.store.if11:
; PRED-OR-DONTVEC-NEXT: [[TMP35:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP3]]
; PRED-OR-DONTVEC-NEXT: [[TMP36:%.*]] = extractelement <4 x i8> [[TMP27]], i32 2
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP36]], i8* [[TMP35]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE12]]
; PRED-OR-DONTVEC: pred.store.continue12:
; PRED-OR-DONTVEC-NEXT: [[TMP37:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP37]], label [[PRED_STORE_IF13:%.*]], label [[PRED_STORE_CONTINUE14]]
; PRED-OR-DONTVEC: pred.store.if13:
; PRED-OR-DONTVEC-NEXT: [[TMP38:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP4]]
; PRED-OR-DONTVEC-NEXT: [[TMP39:%.*]] = extractelement <4 x i8> [[TMP27]], i32 3
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP39]], i8* [[TMP38]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE14]]
; PRED-OR-DONTVEC: pred.store.continue14:
; PRED-OR-DONTVEC-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
; PRED-OR-DONTVEC-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; PRED-OR-DONTVEC-NEXT: [[TMP40:%.*]] = icmp eq i32 [[INDEX_NEXT]], 12
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP40]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; PRED-OR-DONTVEC: middle.block:
; PRED-OR-DONTVEC-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
; PRED-OR-DONTVEC: scalar.ph:
; PRED-OR-DONTVEC-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 12, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; PRED-OR-DONTVEC-NEXT: br label [[FOR_BODY:%.*]] ; PRED-OR-DONTVEC-NEXT: br label [[FOR_BODY:%.*]]
; PRED-OR-DONTVEC: for.body: ; PRED-OR-DONTVEC: for.body:
; PRED-OR-DONTVEC-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; PRED-OR-DONTVEC-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; PRED-OR-DONTVEC-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; PRED-OR-DONTVEC-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; PRED-OR-DONTVEC-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; PRED-OR-DONTVEC-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; PRED-OR-DONTVEC-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; PRED-OR-DONTVEC-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; PRED-OR-DONTVEC-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; PRED-OR-DONTVEC-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; PRED-OR-DONTVEC-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; PRED-OR-DONTVEC-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; PRED-OR-DONTVEC-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; PRED-OR-DONTVEC-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; PRED-OR-DONTVEC-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; PRED-OR-DONTVEC-NEXT: [[INC]] = add nsw i32 [[I]], 1
; PRED-OR-DONTVEC-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; PRED-OR-DONTVEC-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; PRED-OR-DONTVEC-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP0:![0-9]+]] ; PRED-OR-DONTVEC-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
; PRED-OR-DONTVEC: for.end: ; PRED-OR-DONTVEC: for.end:
; PRED-OR-DONTVEC-NEXT: ret i32 0 ; PRED-OR-DONTVEC-NEXT: ret i32 0
; ;
entry: entry:
br label %for.body br label %for.body
for.body: ; preds = %for.body, %entry for.body: ; preds = %for.body, %entry
%i = phi i32 [ 0, %entry ], [ %inc, %for.body ] %i = phi i32 [ 0, %entry ], [ %inc, %for.body ]
%ldidx = getelementptr i8, i8* %src, i32 %i %ldidx = getelementptr i8, i8* %src, i32 %i
%stidx = getelementptr i8, i8* %dst, i32 %i %stidx = getelementptr i8, i8* %dst, i32 %i
%ldval = load i8, i8* %ldidx, align 1 %ldval = load i8, i8* %ldidx, align 1
%cmp1 = icmp eq i8 %ldval, 5 %cmp1 = icmp eq i8 %ldval, 5
%val = select i1 %cmp1, i8 1, i8 2 %val = select i1 %cmp1, i8 1, i8 2
store i8 %val, i8* %stidx, align 1 store i8 %val, i8* %stidx, align 1
%inc = add nsw i32 %i, 1 %inc = add nsw i32 %i, 1
%exitcond = icmp slt i32 %i, 8 %exitcond = icmp slt i32 %i, 8
br i1 %exitcond, label %for.body, label %for.end, !llvm.loop !1 br i1 %exitcond, label %for.body, label %for.end, !llvm.loop !1
for.end: ; preds = %for.body for.end: ; preds = %for.body
ret i32 0 ret i32 0
} }
; Simple loop with small constant trip count. ; Simple loop with small constant trip count.
; TODO: Check that -prefer-predicate-over-epilogue and "llvm.loop.vectorize.predicate.enable=false" ; Check that -prefer-predicate-over-epilogue and "llvm.loop.vectorize.predicate.enable=false"
; hint override "small trip count" heuristic. ; hint override "small trip count" heuristic.
define i32 @const_low_trip_count_hint_no_pred(i8 *%dst, i8 *%src) { define i32 @const_low_trip_count_hint_no_pred(i8 *%dst, i8 *%src) {
; CHECK-LABEL: @const_low_trip_count_hint_no_pred( ; CHECK-LABEL: @const_low_trip_count_hint_no_pred(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK: vector.memcheck:
; CHECK-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK: vector.ph:
; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK: vector.body:
; CHECK-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-NEXT: [[TMP2:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-NEXT: [[TMP4:%.*]] = getelementptr i8, i8* [[TMP2]], i32 0
; CHECK-NEXT: [[TMP5:%.*]] = bitcast i8* [[TMP4]] to <4 x i8>*
; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, <4 x i8>* [[TMP5]], align 1
; CHECK-NEXT: [[TMP6:%.*]] = icmp eq <4 x i8> [[WIDE_LOAD]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-NEXT: [[TMP7:%.*]] = select <4 x i1> [[TMP6]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-NEXT: [[TMP8:%.*]] = getelementptr i8, i8* [[TMP3]], i32 0
; CHECK-NEXT: [[TMP9:%.*]] = bitcast i8* [[TMP8]] to <4 x i8>*
; CHECK-NEXT: store <4 x i8> [[TMP7]], <4 x i8>* [[TMP9]], align 1
; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], 8
; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 9, 8
; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph:
; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 8, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
; CHECK: for.body: ; CHECK: for.body:
; CHECK-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP3:![0-9]+]] ; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP4:![0-9]+]]
; CHECK: for.end: ; CHECK: for.end:
; CHECK-NEXT: ret i32 0 ; CHECK-NEXT: ret i32 0
; ;
; CHECK-SCALAR-LABEL: @const_low_trip_count_hint_no_pred( ; CHECK-SCALAR-LABEL: @const_low_trip_count_hint_no_pred(
; CHECK-SCALAR-NEXT: entry: ; CHECK-SCALAR-NEXT: entry:
; CHECK-SCALAR-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-SCALAR-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-SCALAR-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK-SCALAR: vector.memcheck:
; CHECK-SCALAR-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-SCALAR-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-SCALAR-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK-SCALAR: vector.ph:
; CHECK-SCALAR-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK-SCALAR: vector.body:
; CHECK-SCALAR-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; CHECK-SCALAR-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-SCALAR-NEXT: [[TMP2:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-SCALAR-NEXT: [[TMP3:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-SCALAR-NEXT: [[TMP4:%.*]] = getelementptr i8, i8* [[TMP2]], i32 0
; CHECK-SCALAR-NEXT: [[TMP5:%.*]] = bitcast i8* [[TMP4]] to <4 x i8>*
; CHECK-SCALAR-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, <4 x i8>* [[TMP5]], align 1
; CHECK-SCALAR-NEXT: [[TMP6:%.*]] = icmp eq <4 x i8> [[WIDE_LOAD]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-SCALAR-NEXT: [[TMP7:%.*]] = select <4 x i1> [[TMP6]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-SCALAR-NEXT: [[TMP8:%.*]] = getelementptr i8, i8* [[TMP3]], i32 0
; CHECK-SCALAR-NEXT: [[TMP9:%.*]] = bitcast i8* [[TMP8]] to <4 x i8>*
; CHECK-SCALAR-NEXT: store <4 x i8> [[TMP7]], <4 x i8>* [[TMP9]], align 1
; CHECK-SCALAR-NEXT: [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
; CHECK-SCALAR-NEXT: [[TMP10:%.*]] = icmp eq i32 [[INDEX_NEXT]], 8
; CHECK-SCALAR-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; CHECK-SCALAR: middle.block:
; CHECK-SCALAR-NEXT: [[CMP_N:%.*]] = icmp eq i32 9, 8
; CHECK-SCALAR-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK-SCALAR: scalar.ph:
; CHECK-SCALAR-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 8, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-SCALAR-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-SCALAR-NEXT: br label [[FOR_BODY:%.*]]
; CHECK-SCALAR: for.body: ; CHECK-SCALAR: for.body:
; CHECK-SCALAR-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-SCALAR-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP3:![0-9]+]] ; CHECK-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP4:![0-9]+]]
; CHECK-SCALAR: for.end: ; CHECK-SCALAR: for.end:
; CHECK-SCALAR-NEXT: ret i32 0 ; CHECK-SCALAR-NEXT: ret i32 0
; ;
; CHECK-PRED-OR-SCALAR-LABEL: @const_low_trip_count_hint_no_pred( ; CHECK-PRED-OR-SCALAR-LABEL: @const_low_trip_count_hint_no_pred(
; CHECK-PRED-OR-SCALAR-NEXT: entry: ; CHECK-PRED-OR-SCALAR-NEXT: entry:
; CHECK-PRED-OR-SCALAR-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; CHECK-PRED-OR-SCALAR-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; CHECK-PRED-OR-SCALAR-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; CHECK-PRED-OR-SCALAR: vector.memcheck:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; CHECK-PRED-OR-SCALAR: vector.ph:
; CHECK-PRED-OR-SCALAR-NEXT: br label [[VECTOR_BODY:%.*]]
; CHECK-PRED-OR-SCALAR: vector.body:
; CHECK-PRED-OR-SCALAR-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE14:%.*]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE14]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP2:%.*]] = add i32 [[INDEX]], 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP3:%.*]] = add i32 [[INDEX]], 2
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP4:%.*]] = add i32 [[INDEX]], 3
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP5:%.*]] = icmp ule <4 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP7:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP8:%.*]] = load i8, i8* [[TMP7]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP9:%.*]] = insertelement <4 x i8> poison, i8 [[TMP8]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE]]
; CHECK-PRED-OR-SCALAR: pred.load.continue:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP10:%.*]] = phi <4 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP11]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if3:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP12:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP13:%.*]] = load i8, i8* [[TMP12]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP14:%.*]] = insertelement <4 x i8> [[TMP10]], i8 [[TMP13]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE4]]
; CHECK-PRED-OR-SCALAR: pred.load.continue4:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP15:%.*]] = phi <4 x i8> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF3]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP16]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if5:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP17:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP3]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP18:%.*]] = load i8, i8* [[TMP17]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP19:%.*]] = insertelement <4 x i8> [[TMP15]], i8 [[TMP18]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE6]]
; CHECK-PRED-OR-SCALAR: pred.load.continue6:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP20:%.*]] = phi <4 x i8> [ [[TMP15]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP19]], [[PRED_LOAD_IF5]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP21]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; CHECK-PRED-OR-SCALAR: pred.load.if7:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP22:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP4]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP23:%.*]] = load i8, i8* [[TMP22]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP24:%.*]] = insertelement <4 x i8> [[TMP20]], i8 [[TMP23]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_LOAD_CONTINUE8]]
; CHECK-PRED-OR-SCALAR: pred.load.continue8:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP25:%.*]] = phi <4 x i8> [ [[TMP20]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP24]], [[PRED_LOAD_IF7]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP26:%.*]] = icmp eq <4 x i8> [[TMP25]], <i8 5, i8 5, i8 5, i8 5>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP27:%.*]] = select <4 x i1> [[TMP26]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP28:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP28]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP29:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP30:%.*]] = extractelement <4 x i8> [[TMP27]], i32 0
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP30]], i8* [[TMP29]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE]]
; CHECK-PRED-OR-SCALAR: pred.store.continue:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP31:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP31]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if9:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP32:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP2]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP33:%.*]] = extractelement <4 x i8> [[TMP27]], i32 1
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP33]], i8* [[TMP32]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE10]]
; CHECK-PRED-OR-SCALAR: pred.store.continue10:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP34]], label [[PRED_STORE_IF11:%.*]], label [[PRED_STORE_CONTINUE12:%.*]]
; CHECK-PRED-OR-SCALAR: pred.store.if11:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP35:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP3]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP36:%.*]] = extractelement <4 x i8> [[TMP27]], i32 2
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP36]], i8* [[TMP35]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE12]]
; CHECK-PRED-OR-SCALAR: pred.store.continue12:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP37:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP37]], label [[PRED_STORE_IF13:%.*]], label [[PRED_STORE_CONTINUE14]]
; CHECK-PRED-OR-SCALAR: pred.store.if13:
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP38:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP4]]
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP39:%.*]] = extractelement <4 x i8> [[TMP27]], i32 3
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[TMP39]], i8* [[TMP38]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: br label [[PRED_STORE_CONTINUE14]]
; CHECK-PRED-OR-SCALAR: pred.store.continue14:
; CHECK-PRED-OR-SCALAR-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
; CHECK-PRED-OR-SCALAR-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; CHECK-PRED-OR-SCALAR-NEXT: [[TMP40:%.*]] = icmp eq i32 [[INDEX_NEXT]], 12
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[TMP40]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; CHECK-PRED-OR-SCALAR: middle.block:
; CHECK-PRED-OR-SCALAR-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
; CHECK-PRED-OR-SCALAR: scalar.ph:
; CHECK-PRED-OR-SCALAR-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 12, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; CHECK-PRED-OR-SCALAR-NEXT: br label [[FOR_BODY:%.*]] ; CHECK-PRED-OR-SCALAR-NEXT: br label [[FOR_BODY:%.*]]
; CHECK-PRED-OR-SCALAR: for.body: ; CHECK-PRED-OR-SCALAR: for.body:
; CHECK-PRED-OR-SCALAR-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; CHECK-PRED-OR-SCALAR-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; CHECK-PRED-OR-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; CHECK-PRED-OR-SCALAR-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; CHECK-PRED-OR-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; CHECK-PRED-OR-SCALAR-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; CHECK-PRED-OR-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; CHECK-PRED-OR-SCALAR-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; CHECK-PRED-OR-SCALAR-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; CHECK-PRED-OR-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; CHECK-PRED-OR-SCALAR-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; CHECK-PRED-OR-SCALAR-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; CHECK-PRED-OR-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; CHECK-PRED-OR-SCALAR-NEXT: [[INC]] = add nsw i32 [[I]], 1
; CHECK-PRED-OR-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; CHECK-PRED-OR-SCALAR-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP3:![0-9]+]] ; CHECK-PRED-OR-SCALAR-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP4:![0-9]+]]
; CHECK-PRED-OR-SCALAR: for.end: ; CHECK-PRED-OR-SCALAR: for.end:
; CHECK-PRED-OR-SCALAR-NEXT: ret i32 0 ; CHECK-PRED-OR-SCALAR-NEXT: ret i32 0
; ;
; PRED-OR-DONTVEC-LABEL: @const_low_trip_count_hint_no_pred( ; PRED-OR-DONTVEC-LABEL: @const_low_trip_count_hint_no_pred(
; PRED-OR-DONTVEC-NEXT: entry: ; PRED-OR-DONTVEC-NEXT: entry:
; PRED-OR-DONTVEC-NEXT: [[SRC2:%.*]] = ptrtoint i8* [[SRC:%.*]] to i64
; PRED-OR-DONTVEC-NEXT: [[DST1:%.*]] = ptrtoint i8* [[DST:%.*]] to i64
; PRED-OR-DONTVEC-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_MEMCHECK:%.*]]
; PRED-OR-DONTVEC: vector.memcheck:
; PRED-OR-DONTVEC-NEXT: [[TMP0:%.*]] = sub i64 [[DST1]], [[SRC2]]
; PRED-OR-DONTVEC-NEXT: [[DIFF_CHECK:%.*]] = icmp ult i64 [[TMP0]], 4
; PRED-OR-DONTVEC-NEXT: br i1 [[DIFF_CHECK]], label [[SCALAR_PH]], label [[VECTOR_PH:%.*]]
; PRED-OR-DONTVEC: vector.ph:
; PRED-OR-DONTVEC-NEXT: br label [[VECTOR_BODY:%.*]]
; PRED-OR-DONTVEC: vector.body:
; PRED-OR-DONTVEC-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_STORE_CONTINUE14:%.*]] ]
; PRED-OR-DONTVEC-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_STORE_CONTINUE14]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP1:%.*]] = add i32 [[INDEX]], 0
; PRED-OR-DONTVEC-NEXT: [[TMP2:%.*]] = add i32 [[INDEX]], 1
; PRED-OR-DONTVEC-NEXT: [[TMP3:%.*]] = add i32 [[INDEX]], 2
; PRED-OR-DONTVEC-NEXT: [[TMP4:%.*]] = add i32 [[INDEX]], 3
; PRED-OR-DONTVEC-NEXT: [[TMP5:%.*]] = icmp ule <4 x i32> [[VEC_IND]], <i32 8, i32 8, i32 8, i32 8>
; PRED-OR-DONTVEC-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
; PRED-OR-DONTVEC: pred.load.if:
; PRED-OR-DONTVEC-NEXT: [[TMP7:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP1]]
; PRED-OR-DONTVEC-NEXT: [[TMP8:%.*]] = load i8, i8* [[TMP7]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP9:%.*]] = insertelement <4 x i8> poison, i8 [[TMP8]], i32 0
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE]]
; PRED-OR-DONTVEC: pred.load.continue:
; PRED-OR-DONTVEC-NEXT: [[TMP10:%.*]] = phi <4 x i8> [ poison, [[VECTOR_BODY]] ], [ [[TMP9]], [[PRED_LOAD_IF]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP11:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP11]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
; PRED-OR-DONTVEC: pred.load.if3:
; PRED-OR-DONTVEC-NEXT: [[TMP12:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP2]]
; PRED-OR-DONTVEC-NEXT: [[TMP13:%.*]] = load i8, i8* [[TMP12]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP14:%.*]] = insertelement <4 x i8> [[TMP10]], i8 [[TMP13]], i32 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE4]]
; PRED-OR-DONTVEC: pred.load.continue4:
; PRED-OR-DONTVEC-NEXT: [[TMP15:%.*]] = phi <4 x i8> [ [[TMP10]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP14]], [[PRED_LOAD_IF3]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP16]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6:%.*]]
; PRED-OR-DONTVEC: pred.load.if5:
; PRED-OR-DONTVEC-NEXT: [[TMP17:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP3]]
; PRED-OR-DONTVEC-NEXT: [[TMP18:%.*]] = load i8, i8* [[TMP17]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP19:%.*]] = insertelement <4 x i8> [[TMP15]], i8 [[TMP18]], i32 2
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE6]]
; PRED-OR-DONTVEC: pred.load.continue6:
; PRED-OR-DONTVEC-NEXT: [[TMP20:%.*]] = phi <4 x i8> [ [[TMP15]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP19]], [[PRED_LOAD_IF5]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP21:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP21]], label [[PRED_LOAD_IF7:%.*]], label [[PRED_LOAD_CONTINUE8:%.*]]
; PRED-OR-DONTVEC: pred.load.if7:
; PRED-OR-DONTVEC-NEXT: [[TMP22:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[TMP4]]
; PRED-OR-DONTVEC-NEXT: [[TMP23:%.*]] = load i8, i8* [[TMP22]], align 1
; PRED-OR-DONTVEC-NEXT: [[TMP24:%.*]] = insertelement <4 x i8> [[TMP20]], i8 [[TMP23]], i32 3
; PRED-OR-DONTVEC-NEXT: br label [[PRED_LOAD_CONTINUE8]]
; PRED-OR-DONTVEC: pred.load.continue8:
; PRED-OR-DONTVEC-NEXT: [[TMP25:%.*]] = phi <4 x i8> [ [[TMP20]], [[PRED_LOAD_CONTINUE6]] ], [ [[TMP24]], [[PRED_LOAD_IF7]] ]
; PRED-OR-DONTVEC-NEXT: [[TMP26:%.*]] = icmp eq <4 x i8> [[TMP25]], <i8 5, i8 5, i8 5, i8 5>
; PRED-OR-DONTVEC-NEXT: [[TMP27:%.*]] = select <4 x i1> [[TMP26]], <4 x i8> <i8 1, i8 1, i8 1, i8 1>, <4 x i8> <i8 2, i8 2, i8 2, i8 2>
; PRED-OR-DONTVEC-NEXT: [[TMP28:%.*]] = extractelement <4 x i1> [[TMP5]], i32 0
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP28]], label [[PRED_STORE_IF:%.*]], label [[PRED_STORE_CONTINUE:%.*]]
; PRED-OR-DONTVEC: pred.store.if:
; PRED-OR-DONTVEC-NEXT: [[TMP29:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP1]]
; PRED-OR-DONTVEC-NEXT: [[TMP30:%.*]] = extractelement <4 x i8> [[TMP27]], i32 0
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP30]], i8* [[TMP29]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE]]
; PRED-OR-DONTVEC: pred.store.continue:
; PRED-OR-DONTVEC-NEXT: [[TMP31:%.*]] = extractelement <4 x i1> [[TMP5]], i32 1
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP31]], label [[PRED_STORE_IF9:%.*]], label [[PRED_STORE_CONTINUE10:%.*]]
; PRED-OR-DONTVEC: pred.store.if9:
; PRED-OR-DONTVEC-NEXT: [[TMP32:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP2]]
; PRED-OR-DONTVEC-NEXT: [[TMP33:%.*]] = extractelement <4 x i8> [[TMP27]], i32 1
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP33]], i8* [[TMP32]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE10]]
; PRED-OR-DONTVEC: pred.store.continue10:
; PRED-OR-DONTVEC-NEXT: [[TMP34:%.*]] = extractelement <4 x i1> [[TMP5]], i32 2
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP34]], label [[PRED_STORE_IF11:%.*]], label [[PRED_STORE_CONTINUE12:%.*]]
; PRED-OR-DONTVEC: pred.store.if11:
; PRED-OR-DONTVEC-NEXT: [[TMP35:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP3]]
; PRED-OR-DONTVEC-NEXT: [[TMP36:%.*]] = extractelement <4 x i8> [[TMP27]], i32 2
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP36]], i8* [[TMP35]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE12]]
; PRED-OR-DONTVEC: pred.store.continue12:
; PRED-OR-DONTVEC-NEXT: [[TMP37:%.*]] = extractelement <4 x i1> [[TMP5]], i32 3
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP37]], label [[PRED_STORE_IF13:%.*]], label [[PRED_STORE_CONTINUE14]]
; PRED-OR-DONTVEC: pred.store.if13:
; PRED-OR-DONTVEC-NEXT: [[TMP38:%.*]] = getelementptr i8, i8* [[DST]], i32 [[TMP4]]
; PRED-OR-DONTVEC-NEXT: [[TMP39:%.*]] = extractelement <4 x i8> [[TMP27]], i32 3
; PRED-OR-DONTVEC-NEXT: store i8 [[TMP39]], i8* [[TMP38]], align 1
; PRED-OR-DONTVEC-NEXT: br label [[PRED_STORE_CONTINUE14]]
; PRED-OR-DONTVEC: pred.store.continue14:
; PRED-OR-DONTVEC-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
; PRED-OR-DONTVEC-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; PRED-OR-DONTVEC-NEXT: [[TMP40:%.*]] = icmp eq i32 [[INDEX_NEXT]], 12
; PRED-OR-DONTVEC-NEXT: br i1 [[TMP40]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; PRED-OR-DONTVEC: middle.block:
; PRED-OR-DONTVEC-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
; PRED-OR-DONTVEC: scalar.ph:
; PRED-OR-DONTVEC-NEXT: [[BC_RESUME_VAL:%.*]] = phi i32 [ 12, [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ], [ 0, [[VECTOR_MEMCHECK]] ]
; PRED-OR-DONTVEC-NEXT: br label [[FOR_BODY:%.*]] ; PRED-OR-DONTVEC-NEXT: br label [[FOR_BODY:%.*]]
; PRED-OR-DONTVEC: for.body: ; PRED-OR-DONTVEC: for.body:
; PRED-OR-DONTVEC-NEXT: [[I:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ] ; PRED-OR-DONTVEC-NEXT: [[I:%.*]] = phi i32 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
; PRED-OR-DONTVEC-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i32 [[I]] ; PRED-OR-DONTVEC-NEXT: [[LDIDX:%.*]] = getelementptr i8, i8* [[SRC]], i32 [[I]]
; PRED-OR-DONTVEC-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST:%.*]], i32 [[I]] ; PRED-OR-DONTVEC-NEXT: [[STIDX:%.*]] = getelementptr i8, i8* [[DST]], i32 [[I]]
; PRED-OR-DONTVEC-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1 ; PRED-OR-DONTVEC-NEXT: [[LDVAL:%.*]] = load i8, i8* [[LDIDX]], align 1
; PRED-OR-DONTVEC-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5 ; PRED-OR-DONTVEC-NEXT: [[CMP1:%.*]] = icmp eq i8 [[LDVAL]], 5
; PRED-OR-DONTVEC-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2 ; PRED-OR-DONTVEC-NEXT: [[VAL:%.*]] = select i1 [[CMP1]], i8 1, i8 2
; PRED-OR-DONTVEC-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1 ; PRED-OR-DONTVEC-NEXT: store i8 [[VAL]], i8* [[STIDX]], align 1
; PRED-OR-DONTVEC-NEXT: [[INC]] = add nsw i32 [[I]], 1 ; PRED-OR-DONTVEC-NEXT: [[INC]] = add nsw i32 [[I]], 1
; PRED-OR-DONTVEC-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8 ; PRED-OR-DONTVEC-NEXT: [[EXITCOND:%.*]] = icmp slt i32 [[I]], 8
; PRED-OR-DONTVEC-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END:%.*]], !llvm.loop [[LOOP3:![0-9]+]] ; PRED-OR-DONTVEC-NEXT: br i1 [[EXITCOND]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP4:![0-9]+]]
; PRED-OR-DONTVEC: for.end: ; PRED-OR-DONTVEC: for.end:
; PRED-OR-DONTVEC-NEXT: ret i32 0 ; PRED-OR-DONTVEC-NEXT: ret i32 0
; ;
entry: entry:
br label %for.body br label %for.body
for.body: ; preds = %for.body, %entry for.body: ; preds = %for.body, %entry
%i = phi i32 [ 0, %entry ], [ %inc, %for.body ] %i = phi i32 [ 0, %entry ], [ %inc, %for.body ]
Show All 19 Lines