[VPlan] Turn VPReductionRecipe into a VPValue

[dmgreen]

Summary

This converts the VPReductionRecipe into a VPValue, like other VPRecipe's in preparation for traversing def-use chains.

It doesn't yet change how the VPReductionRecipes are created. It will need to call replaceAllUsesWith from the original recipe they replace, but that is not done yet as VPWidenRecipe need to be created first.

[fhahn]

llvm/lib/Transforms/Vectorize/VPlan.cpp

832

now that this is a VPValue we should probably just reference the VPValue name using printAsOperand(), which will use the underlying value if available.

[dmgreen]

Thanks. I've updated the printing, but not rebased this onto any other patches yet.

[dmgreen]

Rebase and update.

[dmgreen]

Rebase

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

8069

nit: assign in if() and use below?

8093

The recipes that get converted should also be updated to use State directly to map the result VPValue to the generated IR. Could this just be State.set(this, NextInChain, Part)?

llvm/lib/Transforms/Vectorize/VPlan.cpp

838

If it's not too much trouble, it would also be good to add a test to vplan-printing.ll

[dmgreen]

Ooops. I thought this was already using State.set, but uploaded the wrong version. Now with that and the other changes.

[fhahn]

LGTM, thanks!

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7953

Now that both VPWidenREcipe and VPRedcutionRecipe are VPValues, we have to update all users of WidenRecipe with RedRecipe.

7961

We need to update the users of VPWidenRecipe's VPValue or assert that there are none.

[dmgreen]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7953

I think these were happening in the other order, if I am understanding you correctly. The VPWidenRecipe patch is added on top of this, and hopefully fixes up what you are referring to?

7961

Hmm. This is the cmp in a min/max pattern. I am hoping that it would only have one user that we have removed above. At least, I can't seem to get it to vectorize when it has multiple uses. I'll make sure this is an assert in the VPWiden patch.

[fhahn]

small nit for the commit message: might be good to mention that this also turns it into a VPUser.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7953

Yes I think that should be fine after checking the other patch. Looks like I looked at the patches out-of-order for the final round.

7961

Sounds good, Could you perhaps add an assert that the users are empty to D88447?

[fhahn]

llvm/lib/Transforms/Vectorize/VPlan.cpp

79–80

now that VPReductionRecipe is also a VPUser, it should also be added here. (this will go away once all recipes have been converted)

[dmgreen]

llvm/lib/Transforms/Vectorize/VPlan.cpp

79–80

Yeah thanks, I noticed this one as I was getting the patch together for committing. I was trying a sanitizer build to check they are OK, which is taking a rather long time..

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

for (Instruction *R : ReductionOperations) {
           RdxDesc.getReductionFlags(Legal->hasFunNoNaNAttr()));
       auto *CondOp = CM.foldTailByMasking()
                          ? RecipeBuilder.createBlockInMask(R->getParent(), Plan)
                          : nullptr;
       VPReductionRecipe *RedRecipe = new VPReductionRecipe(
           &RdxDesc, R, ChainOp, VecOp, CondOp, Legal->hasFunNoNaNAttr(),
           UseReductionIntrinsic);
       WidenRecipe->getParent()->insert(RedRecipe, WidenRecipe->getIterator());
       WidenRecipe->eraseFromParent();
 
       if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
           Kind == RecurrenceDescriptor::RK_FloatMinMax) {
         VPRecipeBase *CompareRecipe =
             RecipeBuilder.getRecipe(cast<Instruction>(R->getOperand(0)));
         assert(CompareRecipe->getVPRecipeID() == VPRecipeBase::VPWidenSC &&
                "Expected to replace a VPWidenSC");
         CompareRecipe->eraseFromParent();
       }
       Chain = R;
     }
   }
 }
 
 Value* LoopVectorizationPlanner::VPCallbackILV::
 getOrCreateVectorValues(Value *V, unsigned Part) {
void VPInterleaveRecipe::execute(VPTransformState &State) {
   assert(!State.Instance && "Interleave group being replicated.");
   State.ILV->vectorizeInterleaveGroup(IG, State, getAddr(), getMask());
 }
 
 void VPReductionRecipe::execute(VPTransformState &State) {
   assert(!State.Instance && "Reduction being replicated.");
   for (unsigned Part = 0; Part < State.UF; ++Part) {
     RecurrenceDescriptor::RecurrenceKind Kind = RdxDesc->getRecurrenceKind();
-    Value *NewVecOp = State.get(VecOp, Part);
-    if (CondOp) {
-      Value *NewCond = State.get(CondOp, Part);
+    Value *NewVecOp = State.get(getVecOp(), Part);
+    if (getCondOp()) {
+      Value *NewCond = State.get(getCondOp(), Part);
       VectorType *VecTy = cast<VectorType>(NewVecOp->getType());
       Constant *Iden = RecurrenceDescriptor::getRecurrenceIdentity(
           Kind, RdxDesc->getMinMaxRecurrenceKind(), VecTy->getElementType());
       Constant *IdenVec =
           ConstantVector::getSplat(VecTy->getElementCount(), Iden);
       Value *Select = State.Builder.CreateSelect(NewCond, NewVecOp, IdenVec);
       NewVecOp = Select;
     }
     Value *NewRed = createTargetReduction(State.Builder, *RdxDesc, NewVecOp,
                                           UseReductionIntrinsic, NoNaN);
-    Value *PrevInChain = State.get(ChainOp, Part);
+    Value *PrevInChain = State.get(getChainOp(), Part);
     Value *NextInChain;
     if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
         Kind == RecurrenceDescriptor::RK_FloatMinMax) {
       NextInChain =
           createMinMaxOp(State.Builder, RdxDesc->getMinMaxRecurrenceKind(),
                          NewRed, PrevInChain);
     } else {
       NextInChain = State.Builder.CreateBinOp(
-          (Instruction::BinaryOps)I->getOpcode(), NewRed, PrevInChain);
+          (Instruction::BinaryOps)getUnderlyingInstr()->getOpcode(), NewRed,
+          PrevInChain);
     }
-    State.ValueMap.setVectorValue(I, Part, NextInChain);
+    State.ValueMap.setVectorValue(getUnderlyingInstr(), Part, NextInChain);
   }
 }
 
 void VPReplicateRecipe::execute(VPTransformState &State) {
   if (State.Instance) { // Generate a single instance.
     State.ILV->scalarizeInstruction(Ingredient, User, *State.Instance,
                                     IsPredicated, State);
     // Insert scalar instance packing it into a vector.

llvm/lib/Transforms/Vectorize/VPlan.h

public:
   void print(raw_ostream &O, const Twine &Indent,
              VPSlotTracker &SlotTracker) const override;
 
   const InterleaveGroup<Instruction> *getInterleaveGroup() { return IG; }
 };
 
 /// A recipe to represent inloop reduction operations, performing a reduction on
 /// a vector operand into a scalar value, and adding the result to a chain.
-class VPReductionRecipe : public VPRecipeBase {
+/// The Operands are {VecOp, ChainOp, Condition}.
+class VPReductionRecipe : public VPRecipeBase, public VPValue, public VPUser {
   /// The recurrence decriptor for the reduction in question.
   RecurrenceDescriptor *RdxDesc;
-  /// The original instruction being converted to a reduction.
-  Instruction *I;
-  /// The VPValue of the vector value to be reduced.
-  VPValue *VecOp;
-  /// The VPValue of the scalar Chain being accumulated.
-  VPValue *ChainOp;
-  /// The VPValue of the condition for the block.
-  VPValue *CondOp;
   /// Fast math flags to use for the resulting reduction operation.
   bool NoNaN;
   /// Flag for whether to use reduction intrinsics vs shuffle expansions.
   bool UseReductionIntrinsic;
 
 public:
   VPReductionRecipe(RecurrenceDescriptor *R, Instruction *I, VPValue *ChainOp,
                     VPValue *VecOp, VPValue *CondOp, bool NoNaN,
                     bool UseReductionIntrinsic)
-      : VPRecipeBase(VPReductionSC), RdxDesc(R), I(I), VecOp(VecOp),
-        ChainOp(ChainOp), CondOp(CondOp), NoNaN(NoNaN),
-        UseReductionIntrinsic(UseReductionIntrinsic) {}
+      : VPRecipeBase(VPRecipeBase::VPReductionSC),
+        VPValue(VPValue::VPReductionSC, I), VPUser({ChainOp, VecOp}),
+        RdxDesc(R), NoNaN(NoNaN), UseReductionIntrinsic(UseReductionIntrinsic) {
+    if (CondOp)
+      addOperand(CondOp);
+  }
 
   ~VPReductionRecipe() override = default;
 
   /// Method to support type inquiry through isa, cast, and dyn_cast.
+  static inline bool classof(const VPValue *V) {
+    return V->getVPValueID() == VPValue::VPReductionSC;
+  }
   static inline bool classof(const VPRecipeBase *V) {
     return V->getVPRecipeID() == VPRecipeBase::VPReductionSC;
   }
 
   /// Generate the reduction in the loop
   void execute(VPTransformState &State) override;
 
   /// Print the recipe.
   void print(raw_ostream &O, const Twine &Indent,
              VPSlotTracker &SlotTracker) const override;
+
+  /// The VPValue of the scalar Chain being accumulated.
+  VPValue *getChainOp() const { return getOperand(0); }
+  /// The VPValue of the vector value to be reduced.
+  VPValue *getVecOp() const { return getOperand(1); }
+  /// The VPValue of the condition for the block.
+  VPValue *getCondOp() const {
+    return getNumOperands() > 2 ? getOperand(2) : nullptr;
+  }
+
+  /// The original instruction being converted to a reduction.
+  Instruction *getUnderlyingInstr() {
+    return cast<Instruction>(getUnderlyingValue());
+  }
 };
 
 /// VPReplicateRecipe replicates a given instruction producing multiple scalar
 /// copies of the original scalar type, one per lane, instead of producing a
 /// single copy of widened type for all lanes. If the instruction is known to be
 /// uniform only one copy, per lane zero, will be generated.
 class VPReplicateRecipe : public VPRecipeBase {
   /// The instruction being replicated.

llvm/lib/Transforms/Vectorize/VPlan.cpp

void VPValue::dump() const {
   dbgs() << "\n";
 }
 
 void VPRecipeBase::dump() const {
   VPSlotTracker SlotTracker(nullptr);
   print(dbgs(), "", SlotTracker);
   dbgs() << "\n";
 }
 
 // Get the top-most entry block of \p Start. This is the entry block of the
 // containing VPlan. This function is templated to support both const and non-const blocks
 template <typename T> static T *getPlanEntry(T *Start) {
   T *Next = Start;
   T *Current = Start;
   while ((Next = Next->getParent()))
     Current = Next;
 
   SmallSetVector<T *, 8> WorkList;
for (unsigned I = 0, E = getNumIncomingValues(); I < E; ++I) {
       O << "/";
       getMask(I)->printAsOperand(O, SlotTracker);
     }
   }
 }
 
 void VPReductionRecipe::print(raw_ostream &O, const Twine &Indent,
                               VPSlotTracker &SlotTracker) const {
-  O << "\"REDUCE of" << *I << " as ";
-  ChainOp->printAsOperand(O, SlotTracker);
+  O << "\"REDUCE of" << *getUnderlyingValue() << " as ";
+  getChainOp()->printAsOperand(O, SlotTracker);
   O << " + reduce(";
-  VecOp->printAsOperand(O, SlotTracker);
-  if (CondOp) {
+  getVecOp()->printAsOperand(O, SlotTracker);
+  if (getCondOp()) {
     O << ", ";
-    CondOp->printAsOperand(O, SlotTracker);
+    getCondOp()->printAsOperand(O, SlotTracker);
   }
   O << ")";
 }
 
 void VPReplicateRecipe::print(raw_ostream &O, const Twine &Indent,
                               VPSlotTracker &SlotTracker) const {
   O << "\"" << (IsUniform ? "CLONE " : "REPLICATE ")
     << VPlanIngredient(Ingredient);

llvm/lib/Transforms/Vectorize/VPlanValue.h

void setUnderlyingValue(Value *Val) {
     UnderlyingVal = Val;
   }
 
 public:
   /// An enumeration for keeping track of the concrete subclass of VPValue that
   /// are actually instantiated. Values of this enumeration are kept in the
   /// SubclassID field of the VPValue objects. They are used for concrete
   /// type identification.
-  enum { VPValueSC, VPInstructionSC };
+  enum { VPValueSC, VPInstructionSC, VPReductionSC };
 
   VPValue(Value *UV = nullptr) : VPValue(VPValueSC, UV) {}
   VPValue(const VPValue &) = delete;
   VPValue &operator=(const VPValue &) = delete;
 
   /// \return an ID for the concrete type of this object.
   /// This is used to implement the classof checks. This should not be used
   /// for any other purpose, as the values may change as LLVM evolves.