This is an archive of the discontinued LLVM Phabricator instance.

Differential D103255

[LV] Mark increment of main vector loop induction variable as NUW.
ClosedPublic

Authored by fhahn on May 27 2021, 7:56 AM.

Details

Reviewers
reames
Ayal
mkazantsev
Commits
rG23c2f2e6b24d: [LV] Mark increment of main vector loop induction variable as NUW.
Summary

This patch marks the induction increment of the main induction variable
of the vector loop as NUW when not folding the tail.

If the tail is not folded, we know that End - Start >= Step (either
statically or through the minimum iteration checks). We also know that both
Start % Step == 0 and End % Step == 0. We exit the vector loop if %IV +
%Step == %End. Hence we must exit the loop before %IV + %Step unsigned
overflows and we can mark the induction increment as NUW.

This should make SCEV return more precise bounds for the created vector
loops, used by later optimizations, like late unrolling.

At the moment quite a few tests still need to be updated, but before
doing so I'd like to get initial feedback to make sure I am not missing
anything.

Note that this could probably be further improved by using information
from the original IV.

Attempt of modeling of the assumption in Alive2:
https://alive2.llvm.org/ce/z/H_DL_g

Part of a set of fixes required for PR50412.

Diff Detail

Unit TestsFailed

TimeTest
60 msx64 debian > LLVM.Transforms/LoopVectorize::assume.ll
190 msx64 debian > LLVM.Transforms/LoopVectorize::consecutive-ptr-uniforms.ll
40 msx64 debian > LLVM.Transforms/LoopVectorize::dead_instructions.ll
40 msx64 debian > LLVM.Transforms/LoopVectorize::debugloc.ll
40 msx64 debian > LLVM.Transforms/LoopVectorize::dont-fold-tail-for-const-TC.ll
View Full Test Results (107 Failed)

Event Timeline

fhahn created this revision.May 27 2021, 7:56 AM
Herald added subscribers: javed.absar, hiraditya. · View Herald TranscriptMay 27 2021, 7:56 AM
fhahn requested review of this revision.May 27 2021, 7:56 AM
Herald added a project: Restricted Project. · View Herald TranscriptMay 27 2021, 7:56 AM
Harbormaster completed remote builds in B106516: Diff 348272.May 27 2021, 8:37 AM
mkazantsev added inline comments.May 31 2021, 2:28 AM
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
3085

Just to clarify, what exactly is the semantics of End here? Is it last used value of IV or the last computed value (that may be not used)?

fhahn added inline comments.May 31 2021, 4:31 AM
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
3085

End is the trip-count of the original loop, clamped down to the closest value that is divisible by Step. It's compared against against the last computed value of the IV (Next) to determine whether to exit the loop or now (compare at line 3096)

fhahn edited the summary of this revision. (Show Details)May 31 2021, 12:01 PM
mkazantsev accepted this revision.Jun 1 2021, 8:31 PM

Looks legit, thanks!

This revision is now accepted and ready to land.Jun 1 2021, 8:31 PM
reames added a comment.Jun 3 2021, 9:18 AM

LGTM as well. I'm not super familiar with this code, but I skimmed the callers and the necessary conditions seemed to be locally implied.

As an aside, do you have a test case where SCEV can't figure this out? I've got D103118 out for review now, and am interested in other such cases.

This revision was landed with ongoing or failed builds.Jun 7 2021, 2:50 AM
Closed by commit rG23c2f2e6b24d: [LV] Mark increment of main vector loop induction variable as NUW. (authored by fhahn). · Explain Why
This revision was automatically updated to reflect the committed changes.
fhahn added a commit: rG23c2f2e6b24d: [LV] Mark increment of main vector loop induction variable as NUW..
Herald added subscribers: dmgreen, zzheng. · View Herald TranscriptJun 7 2021, 2:50 AM
stuart mentioned this in D103833: [demangler] Fix demangling of 'half'.Jun 7 2021, 1:44 PM
fhahn added a comment.Jun 7 2021, 2:58 PM
In D103255#2796605, @reames wrote:

LGTM as well. I'm not super familiar with this code, but I skimmed the callers and the necessary conditions seemed to be locally implied.

As an aside, do you have a test case where SCEV can't figure this out? I've got D103118 out for review now, and am interested in other such cases.

The motivating case is something like the function below, where nuw improves the results. There are a few other improvements I am planning as follow-up.

define i32 @f(i32 %n) {
entry:
  %cmp5 = icmp ult i32 %n, 13
  tail call void @llvm.assume(i1 %cmp5)
  %cmp16 = icmp ult i32 %n, 1
  br i1 %cmp16, label %middle.block, label %if.end.preheader

if.end.preheader:                                 ; preds = %entry
  %min.iters.check = icmp ult i32 %n, 8
  br i1 %min.iters.check, label %middle.block, label %vector.ph

vector.ph:                                        ; preds = %if.end.preheader
  %n.mod.vf = urem i32 %n, 8
  %n.vec = sub i32 %n, %n.mod.vf
  %ind.end = sub i32 %n, %n.vec
  br label %vector.body

vector.body:                                      ; preds = %vector.body, %vector.ph
  %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body ]
  %index.next = add nuw i32 %index, 8
  %ec = icmp eq i32 %index.next, %n.vec
  br i1 %ec, label %middle.block, label %vector.body

middle.block:
  ret i32 10
}

; Function Attrs: inaccessiblememonly nofree nosync nounwind willreturn mustprogress
declare void @llvm.assume(i1 noundef %0) #1
Ayal added inline comments.Jun 9 2021, 3:03 PM
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
3092

Wrapping of %IV + %Step is related to wrapping of %End = %BackedgeTakenCount + 1, i.e., when %End wraps around to zero. In this case, minimum iteration checks indeed redirects away from the vector loop, unless the tail is folded.
However, it is conceivable to avoid this redirection and vectorize loops whose %End wraps around to zero, i.e., whose %IV + %Step also wraps; see D75746.

An alternative direction to ensure that %IV+%Step of vectorized loops don't wrap, even if their original %End does wrap and also when their tails are folded, is to control the vector loop with a unit Step %IV, whose upper bound is at most half of the original %End. If a vector %IV is also required, one can be constructed in parallel, or computed from %IV * %Step + <0,1,...,%Step-1>; eventually, LSR should know best to optimize them. Sounds reasonable?

Revision Contents

PathSize
llvm/
lib/
Transforms/
Vectorize/
10 lines
test/
Transforms/
LoopVectorize/
X86/
20 lines
2 lines
2 lines

Diff 348272

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 3,075 Lines • ▼ Show 20 LinesPHINode *InnerLoopVectorizer::createInductionVariable(Loop *L, Value *Start,
Instruction *OldInst = getDebugLocFromInstOrOperands(OldInduction); Instruction *OldInst = getDebugLocFromInstOrOperands(OldInduction);
setDebugLocFromInst(Builder, OldInst); setDebugLocFromInst(Builder, OldInst);
auto *Induction = Builder.CreatePHI(Start->getType(), 2, "index"); auto *Induction = Builder.CreatePHI(Start->getType(), 2, "index");
Builder.SetInsertPoint(Latch->getTerminator()); Builder.SetInsertPoint(Latch->getTerminator());
setDebugLocFromInst(Builder, OldInst); setDebugLocFromInst(Builder, OldInst);
// Create i+1 and fill the PHINode. // Create i+1 and fill the PHINode.
Value *Next = Builder.CreateAdd(Induction, Step, "index.next"); //
// If the tail is not folded, we know that End - Start >= Step (either
mkazantsevUnsubmitted
Not Done
ReplyInline Actions

Just to clarify, what exactly is the semantics of End here? Is it last used value of IV or the last computed value (that may be not used)?

mkazantsev: Just to clarify, what exactly is the semantics of `End` here? Is it last used value of IV or…
fhahnAuthorUnsubmitted
Done
ReplyInline Actions

End is the trip-count of the original loop, clamped down to the closest value that is divisible by Step. It's compared against against the last computed value of the IV (Next) to determine whether to exit the loop or now (compare at line 3096)

fhahn: `End` is the trip-count of the original loop, clamped down to the closest value that is…
// statically or through the minimum iteration checks). We also know that both
// Start % Step == 0 and End % Step == 0. We exit the vector loop if %IV +
// %Step == %End. Hence we must exit the loop before %IV + %Step unsigned
// overflows and we can mark the induction increment as NUW.
Value *Next =
Builder.CreateAdd(Induction, Step, "index.next",
/*NUW=*/!Cost->foldTailByMasking(), /*NSW=*/false);
AyalUnsubmitted
Not Done
ReplyInline Actions

Wrapping of %IV + %Step is related to wrapping of %End = %BackedgeTakenCount + 1, i.e., when %End wraps around to zero. In this case, minimum iteration checks indeed redirects away from the vector loop, unless the tail is folded.
However, it is conceivable to avoid this redirection and vectorize loops whose %End wraps around to zero, i.e., whose %IV + %Step also wraps; see D75746.

An alternative direction to ensure that %IV+%Step of vectorized loops don't wrap, even if their original %End does wrap and also when their tails are folded, is to control the vector loop with a unit Step %IV, whose upper bound is at most half of the original %End. If a vector %IV is also required, one can be constructed in parallel, or computed from %IV * %Step + <0,1,...,%Step-1>; eventually, LSR should know best to optimize them. Sounds reasonable?

Ayal: Wrapping of %IV + %Step is related to wrapping of %End = %BackedgeTakenCount + 1, i.e., when…
Induction->addIncoming(Start, L->getLoopPreheader()); Induction->addIncoming(Start, L->getLoopPreheader());
Induction->addIncoming(Next, Latch); Induction->addIncoming(Next, Latch);
// Create the compare. // Create the compare.
Value *ICmp = Builder.CreateICmpEQ(Next, End); Value *ICmp = Builder.CreateICmpEQ(Next, End);
Builder.CreateCondBr(ICmp, L->getUniqueExitBlock(), Header); Builder.CreateCondBr(ICmp, L->getUniqueExitBlock(), Header);
// Now we have two terminators. Remove the old one from the block. // Now we have two terminators. Remove the old one from the block.
Latch->getTerminator()->eraseFromParent(); Latch->getTerminator()->eraseFromParent();
▲ Show 20 Lines • Show All 7,221 Lines • Show Last 20 Lines

llvm/test/Transforms/LoopVectorize/X86/gather_scatter.ll

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 146 Lines • ▼ Show 20 Lines
; FVW2-NEXT: [[TMP35:%.*]] = bitcast float* [[TMP34]] to <2 x float>* ; FVW2-NEXT: [[TMP35:%.*]] = bitcast float* [[TMP34]] to <2 x float>*
; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP29]], <2 x float>* [[TMP35]], i32 4, <2 x i1> [[TMP9]]) ; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP29]], <2 x float>* [[TMP35]], i32 4, <2 x i1> [[TMP9]])
; FVW2-NEXT: [[TMP36:%.*]] = getelementptr inbounds float, float* [[TMP32]], i64 4 ; FVW2-NEXT: [[TMP36:%.*]] = getelementptr inbounds float, float* [[TMP32]], i64 4
; FVW2-NEXT: [[TMP37:%.*]] = bitcast float* [[TMP36]] to <2 x float>* ; FVW2-NEXT: [[TMP37:%.*]] = bitcast float* [[TMP36]] to <2 x float>*
; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP30]], <2 x float>* [[TMP37]], i32 4, <2 x i1> [[TMP10]]) ; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP30]], <2 x float>* [[TMP37]], i32 4, <2 x i1> [[TMP10]])
; FVW2-NEXT: [[TMP38:%.*]] = getelementptr inbounds float, float* [[TMP32]], i64 6 ; FVW2-NEXT: [[TMP38:%.*]] = getelementptr inbounds float, float* [[TMP32]], i64 6
; FVW2-NEXT: [[TMP39:%.*]] = bitcast float* [[TMP38]] to <2 x float>* ; FVW2-NEXT: [[TMP39:%.*]] = bitcast float* [[TMP38]] to <2 x float>*
; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP31]], <2 x float>* [[TMP39]], i32 4, <2 x i1> [[TMP11]]) ; FVW2-NEXT: call void @llvm.masked.store.v2f32.p0v2f32(<2 x float> [[TMP31]], <2 x float>* [[TMP39]], i32 4, <2 x i1> [[TMP11]])
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX6]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX6]], 8
; FVW2-NEXT: [[TMP40:%.*]] = icmp eq i64 [[INDEX_NEXT]], 4096 ; FVW2-NEXT: [[TMP40:%.*]] = icmp eq i64 [[INDEX_NEXT]], 4096
; FVW2-NEXT: br i1 [[TMP40]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP40]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca float*, align 8 %in.addr = alloca float*, align 8
%out.addr = alloca float*, align 8 %out.addr = alloca float*, align 8
▲ Show 20 Lines • Show All 297 Lines • ▼ Show 20 Lines
; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.if29: ; FVW2: pred.store.if29:
; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112 ; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112
; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP44]] ; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP44]]
; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1 ; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1
; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4 ; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4
; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.continue30: ; FVW2: pred.store.continue30:
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX6]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX6]], 8
; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128> ; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128>
; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256 ; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca %struct.In*, align 8 %in.addr = alloca %struct.In*, align 8
▲ Show 20 Lines • Show All 299 Lines • ▼ Show 20 Lines
; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF28:%.*]], label [[PRED_STORE_CONTINUE29]] ; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF28:%.*]], label [[PRED_STORE_CONTINUE29]]
; FVW2: pred.store.if28: ; FVW2: pred.store.if28:
; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112 ; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112
; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds [[STRUCT_OUT]], %struct.Out* [[OUT]], i64 [[TMP44]], i32 1 ; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds [[STRUCT_OUT]], %struct.Out* [[OUT]], i64 [[TMP44]], i32 1
; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1 ; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1
; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4 ; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4
; FVW2-NEXT: br label [[PRED_STORE_CONTINUE29]] ; FVW2-NEXT: br label [[PRED_STORE_CONTINUE29]]
; FVW2: pred.store.continue29: ; FVW2: pred.store.continue29:
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 8
; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128> ; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128>
; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256 ; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca %struct.In*, align 8 %in.addr = alloca %struct.In*, align 8
▲ Show 20 Lines • Show All 285 Lines • ▼ Show 20 Lines
; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.if29: ; FVW2: pred.store.if29:
; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112 ; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112
; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP44]] ; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP44]]
; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1 ; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1
; FVW2-NEXT: store float [[TMP46]], float addrspace(1)* [[TMP45]], align 4 ; FVW2-NEXT: store float [[TMP46]], float addrspace(1)* [[TMP45]], align 4
; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.continue30: ; FVW2: pred.store.continue30:
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX6]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX6]], 8
; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128> ; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128>
; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256 ; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca %struct.In addrspace(1)*, align 8 %in.addr = alloca %struct.In addrspace(1)*, align 8
▲ Show 20 Lines • Show All 285 Lines • ▼ Show 20 Lines
; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.if29: ; FVW2: pred.store.if29:
; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112 ; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112
; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP44]] ; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float* [[OUT]], i64 [[TMP44]]
; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1 ; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1
; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4 ; FVW2-NEXT: store float [[TMP46]], float* [[TMP45]], align 4
; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.continue30: ; FVW2: pred.store.continue30:
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX6]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX6]], 8
; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128> ; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128>
; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256 ; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca %struct.In addrspace(1)*, align 8 %in.addr = alloca %struct.In addrspace(1)*, align 8
▲ Show 20 Lines • Show All 285 Lines • ▼ Show 20 Lines
; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br i1 [[TMP43]], label [[PRED_STORE_IF29:%.*]], label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.if29: ; FVW2: pred.store.if29:
; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112 ; FVW2-NEXT: [[TMP44:%.*]] = or i64 [[OFFSET_IDX]], 112
; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP44]] ; FVW2-NEXT: [[TMP45:%.*]] = getelementptr inbounds float, float addrspace(1)* [[OUT]], i64 [[TMP44]]
; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1 ; FVW2-NEXT: [[TMP46:%.*]] = extractelement <2 x float> [[TMP15]], i32 1
; FVW2-NEXT: store float [[TMP46]], float addrspace(1)* [[TMP45]], align 4 ; FVW2-NEXT: store float [[TMP46]], float addrspace(1)* [[TMP45]], align 4
; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]] ; FVW2-NEXT: br label [[PRED_STORE_CONTINUE30]]
; FVW2: pred.store.continue30: ; FVW2: pred.store.continue30:
; FVW2-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX6]], 8 ; FVW2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX6]], 8
; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128> ; FVW2-NEXT: [[VEC_IND_NEXT]] = add <2 x i64> [[VEC_IND]], <i64 128, i64 128>
; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256 ; FVW2-NEXT: [[TMP47:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]] ; FVW2-NEXT: br i1 [[TMP47]], label [[FOR_END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
; FVW2: for.end: ; FVW2: for.end:
; FVW2-NEXT: ret void ; FVW2-NEXT: ret void
; ;
entry: entry:
%in.addr = alloca %struct.In addrspace(0)*, align 8 %in.addr = alloca %struct.In addrspace(0)*, align 8
▲ Show 20 Lines • Show All 195 Lines • ▼ Show 20 Lines
; AVX512-NEXT: [[TMP53:%.*]] = getelementptr inbounds float, float* [[NEXT_GEP_7]], i64 [[IDXPROM]] ; AVX512-NEXT: [[TMP53:%.*]] = getelementptr inbounds float, float* [[NEXT_GEP_7]], i64 [[IDXPROM]]
; AVX512-NEXT: [[TMP54:%.*]] = bitcast float* [[TMP53]] to <16 x float>* ; AVX512-NEXT: [[TMP54:%.*]] = bitcast float* [[TMP53]] to <16 x float>*
; AVX512-NEXT: [[WIDE_LOAD_7:%.*]] = load <16 x float>, <16 x float>* [[TMP54]], align 4, !alias.scope !2 ; AVX512-NEXT: [[WIDE_LOAD_7:%.*]] = load <16 x float>, <16 x float>* [[TMP54]], align 4, !alias.scope !2
; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD_7]], <16 x float*> [[TMP52]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7 ; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD_7]], <16 x float*> [[TMP52]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7
; AVX512-NEXT: [[TMP55:%.*]] = bitcast float* [[NEXT_GEP_7]] to <16 x float>* ; AVX512-NEXT: [[TMP55:%.*]] = bitcast float* [[NEXT_GEP_7]] to <16 x float>*
; AVX512-NEXT: [[WIDE_LOAD15_7:%.*]] = load <16 x float>, <16 x float>* [[TMP55]], align 4, !alias.scope !9 ; AVX512-NEXT: [[WIDE_LOAD15_7:%.*]] = load <16 x float>, <16 x float>* [[TMP55]], align 4, !alias.scope !9
; AVX512-NEXT: [[TMP56:%.*]] = getelementptr inbounds float, <16 x float*> [[TMP52]], i64 1 ; AVX512-NEXT: [[TMP56:%.*]] = getelementptr inbounds float, <16 x float*> [[TMP52]], i64 1
; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD15_7]], <16 x float*> [[TMP56]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7 ; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD15_7]], <16 x float*> [[TMP56]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7
; AVX512-NEXT: [[INDEX_NEXT_7]] = add i64 [[INDEX]], 128 ; AVX512-NEXT: [[INDEX_NEXT_7]] = add nuw i64 [[INDEX]], 128
; AVX512-NEXT: [[PTR_IND_7]] = getelementptr float, float* [[POINTER_PHI]], i64 2048 ; AVX512-NEXT: [[PTR_IND_7]] = getelementptr float, float* [[POINTER_PHI]], i64 2048
; AVX512-NEXT: [[NITER_NSUB_7]] = add i64 [[NITER]], -8 ; AVX512-NEXT: [[NITER_NSUB_7]] = add i64 [[NITER]], -8
; AVX512-NEXT: [[NITER_NCMP_7:%.*]] = icmp eq i64 [[NITER_NSUB_7]], 0 ; AVX512-NEXT: [[NITER_NCMP_7:%.*]] = icmp eq i64 [[NITER_NSUB_7]], 0
; AVX512-NEXT: br i1 [[NITER_NCMP_7]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]] ; AVX512-NEXT: br i1 [[NITER_NCMP_7]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
; AVX512: middle.block.unr-lcssa: ; AVX512: middle.block.unr-lcssa:
; AVX512-NEXT: [[POINTER_PHI_UNR:%.*]] = phi float* [ [[DEST]], [[VECTOR_PH]] ], [ [[PTR_IND_7]], [[VECTOR_BODY]] ] ; AVX512-NEXT: [[POINTER_PHI_UNR:%.*]] = phi float* [ [[DEST]], [[VECTOR_PH]] ], [ [[PTR_IND_7]], [[VECTOR_BODY]] ]
; AVX512-NEXT: [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_7]], [[VECTOR_BODY]] ] ; AVX512-NEXT: [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_7]], [[VECTOR_BODY]] ]
; AVX512-NEXT: [[LCMP_MOD_NOT:%.*]] = icmp eq i64 [[XTRAITER]], 0 ; AVX512-NEXT: [[LCMP_MOD_NOT:%.*]] = icmp eq i64 [[XTRAITER]], 0
; AVX512-NEXT: br i1 [[LCMP_MOD_NOT]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY_EPIL:%.*]] ; AVX512-NEXT: br i1 [[LCMP_MOD_NOT]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY_EPIL:%.*]]
; AVX512: vector.body.epil: ; AVX512: vector.body.epil:
; AVX512-NEXT: [[POINTER_PHI_EPIL:%.*]] = phi float* [ [[PTR_IND_EPIL:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[POINTER_PHI_UNR]], [[MIDDLE_BLOCK_UNR_LCSSA]] ] ; AVX512-NEXT: [[POINTER_PHI_EPIL:%.*]] = phi float* [ [[PTR_IND_EPIL:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[POINTER_PHI_UNR]], [[MIDDLE_BLOCK_UNR_LCSSA]] ]
; AVX512-NEXT: [[INDEX_EPIL:%.*]] = phi i64 [ [[INDEX_NEXT_EPIL:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[INDEX_UNR]], [[MIDDLE_BLOCK_UNR_LCSSA]] ] ; AVX512-NEXT: [[INDEX_EPIL:%.*]] = phi i64 [ [[INDEX_NEXT_EPIL:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[INDEX_UNR]], [[MIDDLE_BLOCK_UNR_LCSSA]] ]
; AVX512-NEXT: [[EPIL_ITER:%.*]] = phi i64 [ [[EPIL_ITER_SUB:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[XTRAITER]], [[MIDDLE_BLOCK_UNR_LCSSA]] ] ; AVX512-NEXT: [[EPIL_ITER:%.*]] = phi i64 [ [[EPIL_ITER_SUB:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[XTRAITER]], [[MIDDLE_BLOCK_UNR_LCSSA]] ]
; AVX512-NEXT: [[NEXT_GEP_EPIL:%.*]] = getelementptr float, float* [[PTR]], i64 [[INDEX_EPIL]] ; AVX512-NEXT: [[NEXT_GEP_EPIL:%.*]] = getelementptr float, float* [[PTR]], i64 [[INDEX_EPIL]]
; AVX512-NEXT: [[TMP57:%.*]] = getelementptr float, float* [[POINTER_PHI_EPIL]], <16 x i64> <i64 0, i64 16, i64 32, i64 48, i64 64, i64 80, i64 96, i64 112, i64 128, i64 144, i64 160, i64 176, i64 192, i64 208, i64 224, i64 240> ; AVX512-NEXT: [[TMP57:%.*]] = getelementptr float, float* [[POINTER_PHI_EPIL]], <16 x i64> <i64 0, i64 16, i64 32, i64 48, i64 64, i64 80, i64 96, i64 112, i64 128, i64 144, i64 160, i64 176, i64 192, i64 208, i64 224, i64 240>
; AVX512-NEXT: [[TMP58:%.*]] = getelementptr inbounds float, float* [[NEXT_GEP_EPIL]], i64 [[IDXPROM]] ; AVX512-NEXT: [[TMP58:%.*]] = getelementptr inbounds float, float* [[NEXT_GEP_EPIL]], i64 [[IDXPROM]]
; AVX512-NEXT: [[TMP59:%.*]] = bitcast float* [[TMP58]] to <16 x float>* ; AVX512-NEXT: [[TMP59:%.*]] = bitcast float* [[TMP58]] to <16 x float>*
; AVX512-NEXT: [[WIDE_LOAD_EPIL:%.*]] = load <16 x float>, <16 x float>* [[TMP59]], align 4, !alias.scope !2 ; AVX512-NEXT: [[WIDE_LOAD_EPIL:%.*]] = load <16 x float>, <16 x float>* [[TMP59]], align 4, !alias.scope !2
; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD_EPIL]], <16 x float*> [[TMP57]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7 ; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD_EPIL]], <16 x float*> [[TMP57]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7
; AVX512-NEXT: [[TMP60:%.*]] = bitcast float* [[NEXT_GEP_EPIL]] to <16 x float>* ; AVX512-NEXT: [[TMP60:%.*]] = bitcast float* [[NEXT_GEP_EPIL]] to <16 x float>*
; AVX512-NEXT: [[WIDE_LOAD15_EPIL:%.*]] = load <16 x float>, <16 x float>* [[TMP60]], align 4, !alias.scope !9 ; AVX512-NEXT: [[WIDE_LOAD15_EPIL:%.*]] = load <16 x float>, <16 x float>* [[TMP60]], align 4, !alias.scope !9
; AVX512-NEXT: [[TMP61:%.*]] = getelementptr inbounds float, <16 x float*> [[TMP57]], i64 1 ; AVX512-NEXT: [[TMP61:%.*]] = getelementptr inbounds float, <16 x float*> [[TMP57]], i64 1
; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD15_EPIL]], <16 x float*> [[TMP61]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7 ; AVX512-NEXT: call void @llvm.masked.scatter.v16f32.v16p0f32(<16 x float> [[WIDE_LOAD15_EPIL]], <16 x float*> [[TMP61]], i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>), !alias.scope !5, !noalias !7
; AVX512-NEXT: [[INDEX_NEXT_EPIL]] = add i64 [[INDEX_EPIL]], 16 ; AVX512-NEXT: [[INDEX_NEXT_EPIL]] = add nuw i64 [[INDEX_EPIL]], 16
; AVX512-NEXT: [[PTR_IND_EPIL]] = getelementptr float, float* [[POINTER_PHI_EPIL]], i64 256 ; AVX512-NEXT: [[PTR_IND_EPIL]] = getelementptr float, float* [[POINTER_PHI_EPIL]], i64 256
; AVX512-NEXT: [[EPIL_ITER_SUB]] = add i64 [[EPIL_ITER]], -1 ; AVX512-NEXT: [[EPIL_ITER_SUB]] = add i64 [[EPIL_ITER]], -1
; AVX512-NEXT: [[EPIL_ITER_CMP_NOT:%.*]] = icmp eq i64 [[EPIL_ITER_SUB]], 0 ; AVX512-NEXT: [[EPIL_ITER_CMP_NOT:%.*]] = icmp eq i64 [[EPIL_ITER_SUB]], 0
; AVX512-NEXT: br i1 [[EPIL_ITER_CMP_NOT]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], !llvm.loop [[LOOP11:![0-9]+]] ; AVX512-NEXT: br i1 [[EPIL_ITER_CMP_NOT]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], !llvm.loop [[LOOP11:![0-9]+]]
; AVX512: middle.block: ; AVX512: middle.block:
; AVX512-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]] ; AVX512-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]]
; AVX512-NEXT: br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER]] ; AVX512-NEXT: br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER]]
; AVX512: for.body.preheader: ; AVX512: for.body.preheader:
▲ Show 20 Lines • Show All 151 Lines • ▼ Show 20 Lines
; FVW2-NEXT: [[WIDE_LOAD17_3:%.*]] = load <2 x float>, <2 x float>* [[TMP56]], align 4, !alias.scope !14 ; FVW2-NEXT: [[WIDE_LOAD17_3:%.*]] = load <2 x float>, <2 x float>* [[TMP56]], align 4, !alias.scope !14
; FVW2-NEXT: [[TMP57:%.*]] = getelementptr float, float* [[NEXT_GEP_3]], i64 2 ; FVW2-NEXT: [[TMP57:%.*]] = getelementptr float, float* [[NEXT_GEP_3]], i64 2
; FVW2-NEXT: [[TMP58:%.*]] = bitcast float* [[TMP57]] to <2 x float>* ; FVW2-NEXT: [[TMP58:%.*]] = bitcast float* [[TMP57]] to <2 x float>*
; FVW2-NEXT: [[WIDE_LOAD18_3:%.*]] = load <2 x float>, <2 x float>* [[TMP58]], align 4, !alias.scope !14 ; FVW2-NEXT: [[WIDE_LOAD18_3:%.*]] = load <2 x float>, <2 x float>* [[TMP58]], align 4, !alias.scope !14
; FVW2-NEXT: [[TMP59:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP50]], i64 1 ; FVW2-NEXT: [[TMP59:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP50]], i64 1
; FVW2-NEXT: [[TMP60:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP51]], i64 1 ; FVW2-NEXT: [[TMP60:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP51]], i64 1
; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD17_3]], <2 x float*> [[TMP59]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12 ; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD17_3]], <2 x float*> [[TMP59]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12
; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD18_3]], <2 x float*> [[TMP60]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12 ; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD18_3]], <2 x float*> [[TMP60]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12
; FVW2-NEXT: [[INDEX_NEXT_3]] = add i64 [[INDEX]], 16 ; FVW2-NEXT: [[INDEX_NEXT_3]] = add nuw i64 [[INDEX]], 16
; FVW2-NEXT: [[PTR_IND_3]] = getelementptr float, float* [[POINTER_PHI]], i64 256 ; FVW2-NEXT: [[PTR_IND_3]] = getelementptr float, float* [[POINTER_PHI]], i64 256
; FVW2-NEXT: [[NITER_NSUB_3]] = add i64 [[NITER]], -4 ; FVW2-NEXT: [[NITER_NSUB_3]] = add i64 [[NITER]], -4
; FVW2-NEXT: [[NITER_NCMP_3:%.*]] = icmp eq i64 [[NITER_NSUB_3]], 0 ; FVW2-NEXT: [[NITER_NCMP_3:%.*]] = icmp eq i64 [[NITER_NSUB_3]], 0
; FVW2-NEXT: br i1 [[NITER_NCMP_3]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], !llvm.loop [[LOOP15:![0-9]+]] ; FVW2-NEXT: br i1 [[NITER_NCMP_3]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], !llvm.loop [[LOOP15:![0-9]+]]
; FVW2: middle.block.unr-lcssa: ; FVW2: middle.block.unr-lcssa:
; FVW2-NEXT: [[POINTER_PHI_UNR:%.*]] = phi float* [ [[DEST]], [[VECTOR_PH]] ], [ [[PTR_IND_3]], [[VECTOR_BODY]] ] ; FVW2-NEXT: [[POINTER_PHI_UNR:%.*]] = phi float* [ [[DEST]], [[VECTOR_PH]] ], [ [[PTR_IND_3]], [[VECTOR_BODY]] ]
; FVW2-NEXT: [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_3]], [[VECTOR_BODY]] ] ; FVW2-NEXT: [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_3]], [[VECTOR_BODY]] ]
; FVW2-NEXT: [[LCMP_MOD_NOT:%.*]] = icmp eq i64 [[XTRAITER]], 0 ; FVW2-NEXT: [[LCMP_MOD_NOT:%.*]] = icmp eq i64 [[XTRAITER]], 0
Show All 17 Lines
; FVW2-NEXT: [[WIDE_LOAD17_EPIL:%.*]] = load <2 x float>, <2 x float>* [[TMP67]], align 4, !alias.scope !14 ; FVW2-NEXT: [[WIDE_LOAD17_EPIL:%.*]] = load <2 x float>, <2 x float>* [[TMP67]], align 4, !alias.scope !14
; FVW2-NEXT: [[TMP68:%.*]] = getelementptr float, float* [[NEXT_GEP_EPIL]], i64 2 ; FVW2-NEXT: [[TMP68:%.*]] = getelementptr float, float* [[NEXT_GEP_EPIL]], i64 2
; FVW2-NEXT: [[TMP69:%.*]] = bitcast float* [[TMP68]] to <2 x float>* ; FVW2-NEXT: [[TMP69:%.*]] = bitcast float* [[TMP68]] to <2 x float>*
; FVW2-NEXT: [[WIDE_LOAD18_EPIL:%.*]] = load <2 x float>, <2 x float>* [[TMP69]], align 4, !alias.scope !14 ; FVW2-NEXT: [[WIDE_LOAD18_EPIL:%.*]] = load <2 x float>, <2 x float>* [[TMP69]], align 4, !alias.scope !14
; FVW2-NEXT: [[TMP70:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP61]], i64 1 ; FVW2-NEXT: [[TMP70:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP61]], i64 1
; FVW2-NEXT: [[TMP71:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP62]], i64 1 ; FVW2-NEXT: [[TMP71:%.*]] = getelementptr inbounds float, <2 x float*> [[TMP62]], i64 1
; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD17_EPIL]], <2 x float*> [[TMP70]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12 ; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD17_EPIL]], <2 x float*> [[TMP70]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12
; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD18_EPIL]], <2 x float*> [[TMP71]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12 ; FVW2-NEXT: call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> [[WIDE_LOAD18_EPIL]], <2 x float*> [[TMP71]], i32 4, <2 x i1> <i1 true, i1 true>), !alias.scope !10, !noalias !12
; FVW2-NEXT: [[INDEX_NEXT_EPIL]] = add i64 [[INDEX_EPIL]], 4 ; FVW2-NEXT: [[INDEX_NEXT_EPIL]] = add nuw i64 [[INDEX_EPIL]], 4
; FVW2-NEXT: [[PTR_IND_EPIL]] = getelementptr float, float* [[POINTER_PHI_EPIL]], i64 64 ; FVW2-NEXT: [[PTR_IND_EPIL]] = getelementptr float, float* [[POINTER_PHI_EPIL]], i64 64
; FVW2-NEXT: [[EPIL_ITER_SUB]] = add i64 [[EPIL_ITER]], -1 ; FVW2-NEXT: [[EPIL_ITER_SUB]] = add i64 [[EPIL_ITER]], -1
; FVW2-NEXT: [[EPIL_ITER_CMP_NOT:%.*]] = icmp eq i64 [[EPIL_ITER_SUB]], 0 ; FVW2-NEXT: [[EPIL_ITER_CMP_NOT:%.*]] = icmp eq i64 [[EPIL_ITER_SUB]], 0
; FVW2-NEXT: br i1 [[EPIL_ITER_CMP_NOT]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], !llvm.loop [[LOOP16:![0-9]+]] ; FVW2-NEXT: br i1 [[EPIL_ITER_CMP_NOT]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], !llvm.loop [[LOOP16:![0-9]+]]
; FVW2: middle.block: ; FVW2: middle.block:
; FVW2-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]] ; FVW2-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[TMP3]], [[N_VEC]]
; FVW2-NEXT: br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER]] ; FVW2-NEXT: br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER]]
; FVW2: for.body.preheader: ; FVW2: for.body.preheader:
▲ Show 20 Lines • Show All 52 Lines • Show Last 20 Lines

llvm/test/Transforms/LoopVectorize/pr39417-optsize-scevchecks.ll

Show First 20 Lines • Show All 59 Lines • ▼ Show 20 Lines
; CHECK-NEXT: [[TMP17:%.*]] = insertelement <4 x i32> poison, i32 [[TMP13]], i32 0 ; CHECK-NEXT: [[TMP17:%.*]] = insertelement <4 x i32> poison, i32 [[TMP13]], i32 0
; CHECK-NEXT: [[TMP18:%.*]] = insertelement <4 x i32> [[TMP17]], i32 [[TMP14]], i32 1 ; CHECK-NEXT: [[TMP18:%.*]] = insertelement <4 x i32> [[TMP17]], i32 [[TMP14]], i32 1
; CHECK-NEXT: [[TMP19:%.*]] = insertelement <4 x i32> [[TMP18]], i32 [[TMP15]], i32 2 ; CHECK-NEXT: [[TMP19:%.*]] = insertelement <4 x i32> [[TMP18]], i32 [[TMP15]], i32 2
; CHECK-NEXT: [[TMP20:%.*]] = insertelement <4 x i32> [[TMP19]], i32 [[TMP16]], i32 3 ; CHECK-NEXT: [[TMP20:%.*]] = insertelement <4 x i32> [[TMP19]], i32 [[TMP16]], i32 3
; CHECK-NEXT: [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i32 [[TMP0]] ; CHECK-NEXT: [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i32 [[TMP0]]
; CHECK-NEXT: [[TMP22:%.*]] = getelementptr inbounds i32, i32* [[TMP21]], i32 0 ; CHECK-NEXT: [[TMP22:%.*]] = getelementptr inbounds i32, i32* [[TMP21]], i32 0
; CHECK-NEXT: [[TMP23:%.*]] = bitcast i32* [[TMP22]] to <4 x i32>* ; CHECK-NEXT: [[TMP23:%.*]] = bitcast i32* [[TMP22]] to <4 x i32>*
; CHECK-NEXT: store <4 x i32> [[TMP20]], <4 x i32>* [[TMP23]], align 4 ; CHECK-NEXT: store <4 x i32> [[TMP20]], <4 x i32>* [[TMP23]], align 4
; CHECK-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4 ; CHECK-NEXT: [[INDEX_NEXT]] = add i32 nuw [[INDEX]], 4
; CHECK-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4> ; CHECK-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], <i32 4, i32 4, i32 4, i32 4>
; CHECK-NEXT: [[TMP24:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024 ; CHECK-NEXT: [[TMP24:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1024
; CHECK-NEXT: br i1 [[TMP24]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0 ; CHECK-NEXT: br i1 [[TMP24]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
; CHECK: middle.block: ; CHECK: middle.block:
; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 1024, 1024 ; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i32 1024, 1024
; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END_LOOPEXIT:%.*]], label [[SCALAR_PH]] ; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END_LOOPEXIT:%.*]], label [[SCALAR_PH]]
; CHECK: scalar.ph: ; CHECK: scalar.ph:
; CHECK: for.body: ; CHECK: for.body:
Show All 22 Lines

llvm/test/Transforms/LoopVectorize/ptr-induction.ll

; RUN: opt < %s -loop-vectorize -force-vector-width=4 -S | FileCheck %s ; RUN: opt < %s -loop-vectorize -force-vector-width=4 -S | FileCheck %s
target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128" target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
; This testcase causes SCEV to return a pointer-typed exit value. ; This testcase causes SCEV to return a pointer-typed exit value.
; CHECK: @f ; CHECK: @f
; Expect that the pointer indvar has been converted into an integer indvar. ; Expect that the pointer indvar has been converted into an integer indvar.
; CHECK: %index.next = add i64 %index, 4 ; CHECK: %index.next = add nuw i64 %index, 4
define i32 @f(i32* readonly %a, i32* readnone %b) #0 { define i32 @f(i32* readonly %a, i32* readnone %b) #0 {
entry: entry:
%cmp.6 = icmp ult i32* %a, %b %cmp.6 = icmp ult i32* %a, %b
br i1 %cmp.6, label %while.body.preheader, label %while.end br i1 %cmp.6, label %while.body.preheader, label %while.end
while.body.preheader: ; preds = %entry while.body.preheader: ; preds = %entry
br label %while.body br label %while.body
Show All 17 Lines