Page MenuHomePhabricator

[LV] Fold tail by masking - handle reductions
ClosedPublic

Authored by Ayal on Aug 25 2019, 10:33 AM.

Details

Summary

This addresses the "TODO: handle reductions when tail is folded by masking" left by D50480.

When folding the tail (scalar leftover iterations) by introducing an additional (masked) vector iteration at the end, first note that
the header phi of a reduction can remain intact - it takes care of "accumulating the partial sums" of a reduction across all iterations except for the last one. It is only this last iteration which now needs to accumulate these partial sums under a mask, effecting the live-out values. This can be accomplished by introducing a Select instruction, choosing between the last and penultimate (i.e., header phi) values of the partial sums.

Diff Detail

Repository
rL LLVM

Event Timeline

Ayal created this revision.Aug 25 2019, 10:33 AM
Herald added a project: Restricted Project. · View Herald TranscriptAug 25 2019, 10:33 AM
fhahn accepted this revision.Aug 26 2019, 9:54 AM

LGTM, thanks!

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
3690 ↗(On Diff #217047)

I think the assertions here could be a bit stronger, by additionally ensuring we only have a single SelectInst as user and all other users are PHINodes, instead of breaking once we find a SelectInst.

This revision is now accepted and ready to land.Aug 26 2019, 9:54 AM

Thanks, Ayal! LGTM

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
3682 ↗(On Diff #217047)

chosing -> choosing?

This revision was automatically updated to reflect the committed changes.
Ayal marked 2 inline comments as done.Aug 28 2019, 5:49 AM