This is an archive of the discontinued LLVM Phabricator instance.

[VPlan] Expand induction step in VPlan pre-header.
ClosedPublic

Authored by fhahn on Mar 20 2022, 4:22 AM.

Details

Summary

This patch moves SCEV expansion of steps used by
VPWidenIntOrFpInductionRecipes to the pre-header using
VPExpandSCEVRecipe. This ensures that those steps are expanded while the
CFG is in a valid state. Previously, SCEV expansion may happen during
vector body code-generation, during which the CFG may be invalid,
causing issues with SCEV expansion.

Depends on D122095.

Diff Detail

Event Timeline

fhahn created this revision.Mar 20 2022, 4:22 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 20 2022, 4:22 AM
fhahn requested review of this revision.Mar 20 2022, 4:22 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 20 2022, 4:22 AM
Herald added a subscriber: vkmr. · View Herald Transcript
Ayal added inline comments.Mar 28 2022, 3:20 AM
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
8201

"expandSCEVExpr" - rename? This wraps the SCEVExpr in a VPValue (or recipe - introduced in preheader, if not present there already), but does not expand it.

9477

getOperand(1) >> getStepValue()? Analogous to setStartValue().

llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
75

Who's \I ?

llvm/lib/Transforms/Vectorize/VPlan.cpp
1770

Add comment; searching if a recipe has already been created for Expr, and return it if so, to save redundant duplication(?)
Can alternatively cache Expr's that have recipes.

Should we also check if Constant or Unknown Expr's were assigned VPValues, and return them, instead of creating new ones - potentially leading to a leak?

1774

nit: can fold the two if's

llvm/test/Transforms/LoopVectorize/vplan-printing.ll
429

Should this change appear in additional tests?

fhahn updated this revision to Diff 420472.Apr 5 2022, 5:52 AM

Thanks Ayal! I addressed the latest comments and also managed to make the patch independent of the VPlan native patches. To do so, the native VPlan is made compatible with the inner loop vplan after construction and predication.

fhahn added a comment.Apr 5 2022, 5:55 AM

Thanks Ayal! I addressed the latest comments and also managed to make the patch independent of the VPlan native patches. To do so, the native VPlan is made compatible with the inner loop vplan after construction and predication.

Please disregard the comment, it was meant for D121624.

Ayal added a comment.Apr 9 2022, 8:19 AM

Ping :)

Previous comments still await response?

fhahn updated this revision to Diff 421747.Apr 9 2022, 11:33 AM
fhahn marked 6 inline comments as done.

Ping :)

Previous comments still await response?

Oh sorry, I missed that due to accidentially updating with a comment meant for a different revision.

Comments should be addressed in the latest update, thanks!

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
8201

Good point, I changed it to getOrCreateVPValueForSCEVExpr, which also more closely matches the documentation of the function.

9477

Thanks, updated to use getStepValue().

llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
75

Updated to Phi, thanks!

llvm/lib/Transforms/Vectorize/VPlan.cpp
1770

Updated to cache scev expressions.

llvm/test/Transforms/LoopVectorize/vplan-printing.ll
429

I think this is the only place where a expanded scev operand is printed. I updated the other tests that take an ir constant as step when printing.

Ayal added inline comments.Apr 12 2022, 10:48 PM
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
8167–8168

Strictly speaking, this may create more than the WidenInductionRecipe it returns. Perhaps rename, e.g., using plural (Recipes)? Worth documenting.

llvm/lib/Transforms/Vectorize/VPlan.cpp
1300

getStepValue()?

1770

Ah, creating multiple VPValues for same underlying External Def Value should not lead to a leak; VPlan's VPExternalDefs should hold them all, i.e., addExternalDef()'s insert should succeed. But perhaps better to ensure each External Def Value is wrapped by a VPValue only once (see TODO next to addExternalDef()), by having Plan.getOrCreateExternalDef(Value*), potentially indexing VPExternalDefs by their underlying Value?

Perhaps better to simplify initial recipe creation here, and deduplicate (SCEV) recipes by a VPlan2VPlan optimization later (scanning the preheader with a SCEV2Recipe map)?

fhahn updated this revision to Diff 422591.Apr 13 2022, 12:12 PM

Always add VPExpandSCEVRecipe, clean up later.

fhahn marked 2 inline comments as done.Apr 13 2022, 12:13 PM
fhahn added inline comments.
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
8167–8168

Updated and added a comment, thanks!

llvm/lib/Transforms/Vectorize/VPlan.cpp
1300

Updated, thanks!

1770

I put up D123700 to address the TODO.

Perhaps better to simplify initial recipe creation here, and deduplicate (SCEV) recipes by a VPlan2VPlan optimization later (scanning the preheader with a SCEV2Recipe map)?

Update the patch, thanks!

Ayal accepted this revision.Apr 13 2022, 1:08 PM

This looks good to me, thanks for accommodating, ship it!
Adding minor nits.

llvm/lib/Transforms/Vectorize/VPlan.cpp
1761

nit: can return Plan.getOrAddExternalDef(E->getValue()); can eliminate brackets?

1763

ditto.

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
453

nit: perhaps "eliminateCommonExpandSCEVRecipes()" or "removeRedundantExpandSCEVRecipes()" would be a more descriptive name.

This revision is now accepted and ready to land.Apr 13 2022, 1:08 PM
This revision was landed with ongoing or failed builds.Apr 19 2022, 4:07 AM
This revision was automatically updated to reflect the committed changes.
fhahn marked 3 inline comments as done.
fhahn marked 2 inline comments as done.Apr 19 2022, 4:07 AM
fhahn added inline comments.
llvm/lib/Transforms/Vectorize/VPlan.cpp
1761

Thanks, simplified in the committed version.

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
453

Thanks, updated in the committed version!