This patch adds a new abstraction layer to VPlan and leverages it to model masking in VPlan. Masking is essential to the vectorizer's predication process, but is currently an ad-hoc side-effect of the vectorized IR-generation stage. Modeling masking directly in VPlan facilitates moving predication from IR-generation stage to the planning stage.
VPlan models the contents of VPBasicBlocks using Recipes, which do not currently expose their planned instructions nor the Def-Use relations between them. Modeling masking in VPlan requires VPlan to model Def-Use relations, specifically among the planned instructions that will generate, manipulate and consume masks.
The new VPValue and VPUser classes model how data flows into, through and out of a VPlan, forming the vertices of a planned Def-Use graph. The new VPInstruction class is a generic single-instruction Recipe that models a planned instruction along with its opcode, operands and users. See VectorizationPlan.rst for more details.
With this patch, VPlan models as VPInstructions the planned instructions that manipulate masks (AND, OR, NOT), introduced during predication. Their operands are other VPInstructions (e.g., a NOT feeding an AND), or VPValues coming from other Recipes or from live-in values. Their users are other VPInstructions or VPUsers employed to retrieve the mask during Recipes' execution (e.g., for generating calls to masked load/store intrinsics). More concretely:
- createBlockInMask(), createEdgeMask() were moved from ILV to the Planner. Their code is essentially unchanged, except they now generate VPValues instead of Values. They are now called by Planner and passed down to mask-aware Recipes during VPlan construction.
- ILV::vectorizeMemoryInstruction() so far generated the masks on-the-fly using createBlockInMask(). In this patch VPWidenMemoryInstructionRecipe provides the masks as an optional argument.
- VPBlendRecipe takes over non-loop phi's. It generates the same sequence of selects, only based on VPlan masks.
- VPBranchOnMaskRecipe now takes a VPValue mask.
Another aspect to notice is that until VPValues fully replace the existing scalar-IR-based ValueMap mechanism we effectively have two Def-Use graphs co-exist during vectorized code generation. Rewiring these two graphs together is done using
(a) Recipes taking VPValues and writing to ValueMap, and conversely
(b) VPValues that model Values still handled in ValueMap. The ILVCallback provides the bridge from VPlan’s DataState to ValueMap.
This abstraction layer facilitates modeling additional Def-Use relations in VPlan, to support bringing additional transformations to the planning stage.
Ayal & Gil