[LV] Split off code to create initial VPlan (NFC).

[fhahn]

Summary

Split up tryToBuildVPlanWithVPRecipes into intial plan creation and
optimizations.

Depends on D154640.

[Ayal]

Thanks for following-up with refactoring the excessive tryToBuildVPlanWithVPRecipes!
This raises several thoughts...

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9121

An alternative refactoring may be to outline the second "Now optimize the initial VPlan" part, rather than the first "build Initial VPlan" part. This could be done perhaps by introducing a new VPlanTransforms::process(VPlan) method which manages all passes, similar to the meta-pass createAndOptimizeReplicateRegions().

9143

Should this mark the completion of an initial VPlan, which now fully represents its scope and can be transformed w/o referring to original IR based relations?

9147

This is currently the only transformation which may bail out of building a VPlan entirely. Marking the optional completion of an initial VPlan here would enable separating it from transformations that only improve it w/o bailing out, as in having the caller do:

if (auto Plan = tryToBuildInitialVPlanWithVPRecipes(SubRange, DeadInstructions)) {
  transformVPlanWithRecipes(*Plan); // Or directly VPlanTransforms::process(*Plan);
  VPlans.push_back(std::move(*Plan));
}

which clearly indicates that DeadInstructions and SubRange are not used nor modified by the latter transforming phase.

9156

Note that creating replicate regions is currently the only (meta) pass that is mandatory. It may be useful to have a minimal "-O0" path consisting of vital transforms only, which checks the functional correctness of the outcome.

[fhahn]

Restructure code as suggested by first only splitting off pure VPlan-to-VPlan optimizations.

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9121

Adjusted, thanks!

9143

Sounds good as a start, updated!

9147

Adjusted, thanks!

9156

This might be good in the future, not sure how we best model the options (would also be good to allow running individual optimizations)

[Ayal]

This looks good to me, thanks! Adding a couple of final nits.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9162

nit: is replacing return std::move(Plan) here with return Plan possible independent of all other changes?

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp

416 ↗

(On Diff #546997)

Note: this search works around setting FindMyCast directly by retrieving it from Value2VPValue[IRCast], because the latter mapping should be avoided at this time. Perhaps Casts should be converted into CastRecipes and recorded in IV recipe after the latter is built and before Value2VPValue is abandoned.

792 ↗

(On Diff #546997)

nit: do the local calls above need the VPlanTransforms:: prefix?

[fhahn]

Address cleanups, thanks! I am planning to land this soon.

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9162

Yep, split off in 39cf2104507a

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp

792 ↗

(On Diff #546997)

Removed the prefix, thanks!

llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h

private:
   /// Legal. This method is only used for the legacy inner loop vectorizer.
   /// \p Range's largest included VF is restricted to the maximum VF the
   /// returned VPlan is valid for. If no VPlan can be built for the input range,
   /// set the largest included VF to the maximum VF for which no plan could be
   /// built.
   std::optional<VPlanPtr> tryToBuildVPlanWithVPRecipes(
       VFRange &Range, SmallPtrSetImpl<Instruction *> &DeadInstructions);
 
+  VPlanPtr buildInitialVPlanWithVPRecipes(
+      VFRange &Range, SmallPtrSetImpl<Instruction *> &DeadInstructions);
+
   /// Build VPlans for power-of-2 VF's between \p MinVF and \p MaxVF inclusive,
   /// according to the information gathered by Legal when it checked if it is
   /// legal to vectorize the loop. This method creates VPlans using VPRecipes.
   void buildVPlansWithVPRecipes(ElementCount MinVF, ElementCount MaxVF);
 
   // Adjust the recipes for reductions. For in-loop reductions the chain of
   // instructions leading from the loop exit instr to the phi need to be
   // converted to reductions, with one operand being vector and the other being

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

static void addUsersInExitBlock(VPBasicBlock *HeaderVPBB,
   for (PHINode &ExitPhi : ExitBB->phis()) {
     Value *IncomingValue =
         ExitPhi.getIncomingValueForBlock(ExitingBB);
     VPValue *V = Plan.getVPValueOrAddLiveIn(IncomingValue);
     Plan.addLiveOut(&ExitPhi, V);
   }
 }
 
-std::optional<VPlanPtr> LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
+VPlanPtr LoopVectorizationPlanner::buildInitialVPlanWithVPRecipes(
     VFRange &Range, SmallPtrSetImpl<Instruction *> &DeadInstructions) {
 
   SmallPtrSet<const InterleaveGroup<Instruction> *, 1> InterleaveGroups;
 
   VPRecipeBuilder RecipeBuilder(OrigLoop, TLI, Legal, CM, PSE, Builder);
 
   // ---------------------------------------------------------------------------
   // Pre-construction: record ingredients whose recipes we'll need to further
for (unsigned i = 0; i < IG->getFactor(); ++i)
         }
         MemberR->eraseFromParent();
       }
   }
 
   for (ElementCount VF : Range)
     Plan->addVF(VF);
   Plan->setName("Initial VPlan");
+  return Plan;
+}
+
+std::optional<VPlanPtr> LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
+    VFRange &Range, SmallPtrSetImpl<Instruction *> &DeadInstructions) {
 
+  VPlanPtr Plan = buildInitialVPlanWithVPRecipes(Range, DeadInstructions);
+
+  // Now optimize the initial VPlan.
+  //
   // Replace VPValues for known constant strides guaranteed by predicate scalar
   // evolution.
   for (auto [_, Stride] : Legal->getLAI()->getSymbolicStrides()) {
     auto *StrideV = cast<SCEVUnknown>(Stride)->getValue();
     auto *ScevStride = dyn_cast<SCEVConstant>(PSE.getSCEV(StrideV));
     // Only handle constant strides for now.
     if (!ScevStride)
       continue;
     Constant *CI = ConstantInt::get(Stride->getType(), ScevStride->getAPInt());
 
     auto *ConstVPV = Plan->getVPValueOrAddLiveIn(CI);
     // The versioned value may not be used in the loop directly, so just add a
     // new live-in in those cases.
     Plan->getVPValueOrAddLiveIn(StrideV)->replaceAllUsesWith(ConstVPV);
   }
 
   // From this point onwards, VPlan-to-VPlan transformations may change the plan
   // in ways that accessing values using original IR values is incorrect.
   Plan->disableValue2VPValue();
 
   // Sink users of fixed-order recurrence past the recipe defining the previous
   // value and introduce FirstOrderRecurrenceSplice VPInstructions.
   if (!VPlanTransforms::adjustFixedOrderRecurrences(*Plan, Builder))
     return std::nullopt;
 
   VPlanTransforms::removeRedundantCanonicalIVs(*Plan);
   VPlanTransforms::removeRedundantInductionCasts(*Plan);
 
   VPlanTransforms::optimizeInductions(*Plan, *PSE.getSE());
   VPlanTransforms::removeDeadRecipes(*Plan);
 
   VPlanTransforms::createAndOptimizeReplicateRegions(*Plan);
 
   VPlanTransforms::removeRedundantExpandSCEVRecipes(*Plan);
   VPlanTransforms::mergeBlocksIntoPredecessors(*Plan);
 
   assert(VPlanVerifier::verifyPlanIsValid(*Plan) && "VPlan is invalid");
   return std::make_optional(std::move(Plan));
 }
 
 VPlanPtr LoopVectorizationPlanner::buildVPlan(VFRange &Range) {
   // Outer loop handling: They may require CFG and instruction level
   // transformations before even evaluating whether vectorization is profitable.
   // Since we cannot modify the incoming IR, we need to build VPlan upfront in
   // the vectorization pipeline.
   assert(!OrigLoop->isInnermost());