[VPlan] Add & use VPValue operands for VPWidenRecipe (NFC).

[fhahn]

Summary

This patch adds VPValue version of the instruction operands to
VPWidenRecipe and uses them during code-generation.

Similar to D76373 this reduces ingredient def-use usage by ILV as
a step towards full VPlan-based def-use relations.

[Ayal]

Thanks for following-up quickly. The part that extends VPWidenRecipe to have a User for holding its VPValue operands looks ok, but could the patch make sure that these operands are indeed used correctly by ILV::widenInstruction(), instead of moving the latter over to its execute()? The former may involve refactoring the case that deals with Calls, to record cost-based UseVectorInstrinsic/NeedToScalarize decisions when constructing the VPlan/recipe rather than during codegen; to record InvariantCond of Selects instead of accessing getSCEV(I.getOperand(0)) during codegen, possibly cleaning up to generate ScalarCond OR Cond but not both. In any case, accesses to I’s operands should all change to access User's operands instead, possibly with their underlying ingredient if needed.

This follows the changes D70865 made to ILV::vectorizeInterleaveGroup() and ILV::vectorizeMemoryInstruction(), focusing only on transforming the def/use relations between recipes, as a first (or in this case second ;-) separate step.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7452

Simpler to pass the User to GetVectorOps() and have it iterate over its Operands?

7615

Instead of accessing CI->getNumArgOperands(), get the number of User’s operands; or simply iterate over them?

[fhahn]

In D76992#1951192, @Ayal wrote:

Thanks for following-up quickly. The part that extends VPWidenRecipe to have a User for holding its VPValue operands looks ok, but could the patch make sure that these operands are indeed used correctly by ILV::widenInstruction(), instead of moving the latter over to its execute()? The former may involve refactoring the case that deals with Calls, to record cost-based UseVectorInstrinsic/NeedToScalarize decisions when constructing the VPlan/recipe rather than during codegen; to record InvariantCond of Selects instead of accessing getSCEV(I.getOperand(0)) during codegen, possibly cleaning up to generate ScalarCond OR Cond but not both. In any case, accesses to I’s operands should all change to access User's operands instead, possibly with their underlying ingredient if needed.

The main reason for moving it into the recipe is so we can use getUnderlyingValue when lowering calls. A cleaner solution to deal with call would be to move them to their own recipe/instruction and record the required info as suggested. Is that along the line you suggest? I.e. something like the following: Move calls to their own recipe (NFC) and then move VPWidenRecipe and VPCallRecipe to VPUser in parallel?

Without changes to the call handling, I am not sure how we would be able to use the existing logic (unless we expose getUnderlyingValue to ILV.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7615

CallInsts have their arguments first, followed by the called function, followed by operand bundles. Currently they will all be added to User. We could check the function type to see how many operands the call passes, but that might not work for variadic calls.

[Ayal]

In D76992#1952322, @fhahn wrote:

In D76992#1951192, @Ayal wrote:

Thanks for following-up quickly. The part that extends VPWidenRecipe to have a User for holding its VPValue operands looks ok, but could the patch make sure that these operands are indeed used correctly by ILV::widenInstruction(), instead of moving the latter over to its execute()? The former may involve refactoring the case that deals with Calls, to record cost-based UseVectorInstrinsic/NeedToScalarize decisions when constructing the VPlan/recipe rather than during codegen; to record InvariantCond of Selects instead of accessing getSCEV(I.getOperand(0)) during codegen, possibly cleaning up to generate ScalarCond OR Cond but not both. In any case, accesses to I’s operands should all change to access User's operands instead, possibly with their underlying ingredient if needed.

The main reason for moving it into the recipe is so we can use getUnderlyingValue when lowering calls. A cleaner solution to deal with call would be to move them to their own recipe/instruction and record the required info as suggested. Is that along the line you suggest? I.e. something like the following: Move calls to their own recipe (NFC) and then move VPWidenRecipe and VPCallRecipe to VPUser in parallel?

Without changes to the call handling, I am not sure how we would be able to use the existing logic (unless we expose getUnderlyingValue to ILV.

Yes, let's go for this cleaner solution. The first phase of VPlan integration using recipes is focused on refactoring ILV logic so that decisions are made and recorded in VPlan rather than during codegen. The latter, taken care of by recipe::execute(), should be as straightforward as possible.

[fhahn]

In D76992#1954253, @Ayal wrote:>

Yes, let's go for this cleaner solution. The first phase of VPlan integration using recipes is focused on refactoring ILV logic so that decisions are made and recorded in VPlan rather than during codegen. The latter, taken care of by recipe::execute(), should be as straightforward as possible.

Great. i'l update the patches accordingly in a bit.

[fhahn]

In D76992#1954253, @Ayal wrote:

In D76992#1952322, @fhahn wrote:

The main reason for moving it into the recipe is so we can use getUnderlyingValue when lowering calls. A cleaner solution to deal with call would be to move them to their own recipe/instruction and record the required info as suggested. Is that along the line you suggest? I.e. something like the following: Move calls to their own recipe (NFC) and then move VPWidenRecipe and VPCallRecipe to VPUser in parallel?

Without changes to the call handling, I am not sure how we would be able to use the existing logic (unless we expose getUnderlyingValue to ILV.

Yes, let's go for this cleaner solution. The first phase of VPlan integration using recipes is focused on refactoring ILV logic so that decisions are made and recorded in VPlan rather than during codegen. The latter, taken care of by recipe::execute(), should be as straightforward as possible.

I've updated the code to just change how operands are accessed during ILV::widenInstruction. Instead of accessing the operands via the instruction, GetUnderlyingValue/GetScalarValue/GetVectorValue are provided, which use the VPUser & VPTransformState. Call handling was moved to a separate recipe D77467.

[fhahn]

Rebased after landing VPWidenCallRecipe in 90be3c24a7162a488f68f7cce159017c10408133.

[fhahn]

Pass VPUser and VPTransformState to widenInstruction, instead of accessor callbacks. A callback is still needed to access the underlying value for checking whether the condition is loop-invariant. But I do not think there's a way around that without additional refactoring.

[fhahn]

Updated to use iterator_range instead of ArrayRef, as in D77655.

[gilr]

llvm/lib/Transforms/Vectorize/VPlanValue.h

47

Make ILV friends with VPValue instead? Would facilitate removing that last lambda.

[fhahn]

llvm/lib/Transforms/Vectorize/VPlanValue.h

47

I think granting access to ILV would mean that the scope where the underlying value can be accessed would be extended a lot and might encourage more uses (which I think is undesired). I think the lambda allows us to grant more targeted access and there is only a single use in widenInstruction, which will hopefully removed in the future. Does that make sense? I would be also happy to make ILV a friend, if you think that's preferable over the lambda in the long run.

[Ayal]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4280–4281

Better for a VPlan recipe to record such information/decision what Select should eventually be generated, rather than for code-gen to check the SCEV of an underlying value of an operand.

Perhaps worth taking the Select case out of ILV::widenInstruction() and into a dedicated ILV::widenSelectInstruction() which will be given InvariantCond as a parameter, and have VPWidenRecipe (or a dedicated VPWidenSelectRecipe) record InvariantCond and feed it?

[gilr]

llvm/lib/Transforms/Vectorize/VPlanValue.h

47

Letting ILV access the underlying value seems to follow the intent of the design principle preceding getUnderlyingValue()'s declaration of limiting its use to the code-gen phase.
Anyway, I agree with Ayal that a better solution would be to take that decision during planning rather than during code-gen.

[fhahn]

Rebased on top of D77869 to get rid of the last callback.

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4280–4281

Better for a VPlan recipe to record such information/decision what Select should eventually be generated, rather than for code-gen to check the SCEV of an underlying value of an operand.

Perhaps worth taking the Select case out of ILV::widenInstruction() and into a dedicated ILV::widenSelectInstruction() which will be given InvariantCond as a parameter, and have VPWidenRecipe (or a dedicated VPWidenSelectRecipe) record InvariantCond and feed it?

Agreed. I originally planned on doing that as a follow-up, but it's probably easier to do up-front: D77869

llvm/lib/Transforms/Vectorize/VPlanValue.h

47

Anyway, I agree with Ayal that a better solution would be to take that decision during planning rather than during code-gen.

+1
I originally planned on doing that as a follow-up, but it's probably easier to do up-front: D77869

[fhahn]

Rebased after recent refactoring patches and VPWidenSelectRecipe landed.

[fhahn]

ping :) All suggested refactors should have landed now

[gilr]

llvm/lib/Transforms/Vectorize/VPlanValue.h

47

Redundant now.

[fhahn]

Remove unnecessary friend class,

[gilr]

llvm/lib/Transforms/Vectorize/VPlan.h

774

Missing member comment.

[gilr]

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp

76

Could the code creating the iterator range be made reusable by becoming a VPlan method taking a User?

[fhahn]

Add VPlan::mapToVPValues helper

[gilr]

LGTM (with that missing comment nit), thanks!

[fhahn]

In D76992#1996936, @gilr wrote:

LGTM (with that missing comment nit), thanks!

Thanks Gil! I missed the comment in the previous update (sorry!), but I'll add it in the commit.

[mehdi_amini]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4289

FYI XLA generated IR triggers a crash here.

I have the IR right before the pass (I can share it, this is the public XLA test-suite) but I can't reproduce with opt -loop-vectorize (I suspect some TTI/TLI settings are affecting this).

I don't really understand this code so I couldn't debug it right now but in gdb:

(gdb) call User.getUnderlyingValue() 
$13 = (llvm::Value *) 0x0
(gdb) call User.getOperand(0)->getUnderlyingValue()->dump()
  %reduce-window.10.clone.invar_address.dim.3.016 = phi i64 [ %bc.resume.val, %scalar.ph ], [ %invar.inc3, %reduce-window.10.clone.inner.loop_exit.window.0 ]
(gdb) call User.getOperand(1)->getUnderlyingValue()->dump()
i64 13
(gdb) p Part
$14 = 0

I'll likely revert while we're trying to extract a minimum reproducer. Let me know if you have any advice for this?

[mehdi_amini]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4289

While I have a gdb session opened:

#0  llvm::Value::getType (this=0x0) at llvm/include/llvm/IR/Value.h:244
#1  0x0000555596796922 in llvm::InnerLoopVectorizer::getOrCreateVectorValue (this=0x7fffee60fae0, V=0x0, Part=0) at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:2020
#2  0x00005555967b1a87 in llvm::LoopVectorizationPlanner::VPCallbackILV::getOrCreateVectorValues (this=0x7fffee60f8a8, V=0x0, Part=0)
    at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:7376
#3  0x00005555967bd8fd in llvm::VPTransformState::get (this=0x7fffee60f770, Def=0x37e73a4572d0, Part=0) at llvm/lib/Transforms/Vectorize/VPlan.h:270
#4  0x00005555967a211e in llvm::InnerLoopVectorizer::widenInstruction (this=0x7fffee60fae0, I=..., User=..., State=...) at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:4292
#5  0x00005555967b1d65 in llvm::VPWidenRecipe::execute (this=0x37e73a450b60, State=...) at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:7412
#6  0x00005555968488a0 in llvm::VPBasicBlock::execute (this=0x37e73aa4c360, State=0x7fffee60f770) at llvm/lib/Transforms/Vectorize/VPlan.cpp:240
#7  0x0000555596849c71 in llvm::VPlan::execute (this=0x37e73cbcc260, State=0x7fffee60f770) at llvm/lib/Transforms/Vectorize/VPlan.cpp:476
#8  0x00005555967ae456 in llvm::LoopVectorizationPlanner::executePlan (this=0x7fffee610080, ILV=..., DT=0x37e73b2db0c0) at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:6612
#9  0x00005555967b4181 in llvm::LoopVectorizePass::processLoop (this=0x37e73c142860, L=0x37e73cc9d2a0) at llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:7919

[bkramer]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4289

https://reviews.llvm.org/P8215 crashes when run through opt -O3. Sadly bugpoint doesn't seem to like it so it's a huge test case.

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4289

Thanks, I'll take a look.

[fedor.sergeev]

@fhahn, we also had a crash downstream on our fuzzer run, if you still need a reproducer I can try reducing our failure.

[fhahn]

In D76992#2003990, @fedor.sergeev wrote:

@fhahn, we also had a crash downstream on our fuzzer run, if you still need a reproducer I can try reducing our failure.

Thanks, I've already have a very reduced reproducer, which I'll add when I recommit this patch. Ayal posted a fix for the actual issue (D78847)

But it would be great if you could share the unreduced reproducer, to verify it is the same issue.

[Ayal]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

4289

This patch exposed a bug in how backedge-taken-count is recorded; uploaded a fix in D78847.
Would be good to confirm if the two patches combined resolve the original issue(s).

llvm/lib/Transforms/Vectorize/VPlan.h

270

Would be good to add an assert of VPValue2Value.count(Def), instead of creating a entry with null Value leading to a crash, as reported.

277

ditto.

[fedor.sergeev]

it would be great if you could share the unreduced reproducer, to verify it is the same issue.

This is our original reproducer:

before-vectorize.ll34 KBDownload

Fails on commit 9245c7ac13480ed48ae339ad0e68cbe680cd0642 like this:

]  bin/opt -loop-vectorize before-vectorize.ll 2>&1 | c++filt | head -n 12
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: bin/opt -loop-vectorize before-vectorize.ll 
1.      Running pass 'Function Pass Manager' on module 'before-vectorize.ll'.
2.      Running pass 'Loop Vectorization' on function '@"virtual void Test.vMeth1(jint)527365"'
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(llvm::sys::PrintStackTrace(llvm::raw_ostream&)+0x1a)[0x7ff9600e0ada]
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(llvm::sys::RunSignalHandlers()+0x34)[0x7ff9600deb44]
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(+0x72ec83)[0x7ff9600dec83]
/lib64/libpthread.so.0(+0xf5d0)[0x7ff95f7a35d0]
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(llvm::InnerLoopVectorizer::getBroadcastInstrs(llvm::Value*)+0x10)[0x7ff9611cddb0]
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(llvm::InnerLoopVectorizer::getOrCreateVectorValue(llvm::Value*, unsigned int)+0xce)[0x7ff9611db3ee]
/ws/llvm-upstream/builds/release-assert/bin/../lib/libLLVM-11git.so(+0x1832361)[0x7ff9611e2361]

[fhahn]

Recommitted with a reduced version of the reproducer. Both the reproducers don't crash after a3c964a278b4.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

InnerLoopVectorizer(Loop *OrigLoop, PredicatedScalarEvolution &PSE,
         VectorLoopValueMap(UnrollFactor, VecWidth), Legal(LVL), Cost(CM) {}
   virtual ~InnerLoopVectorizer() = default;
 
   /// Create a new empty loop. Unlink the old loop and connect the new one.
   /// Return the pre-header block of the new loop.
   BasicBlock *createVectorizedLoopSkeleton();
 
   /// Widen a single instruction within the innermost loop.
-  void widenInstruction(Instruction &I);
+  void widenInstruction(Instruction &I, VPUser &Operands,
+                        VPTransformState &State);
 
   /// Widen a single call instruction within the innermost loop.
   void widenCallInstruction(CallInst &I, VPUser &ArgOperands,
                             VPTransformState &State);
 
   /// Widen a single select instruction within the innermost loop.
   void widenSelectInstruction(SelectInst &I, bool InvariantCond);
 
assert((I.getOpcode() == Instruction::UDiv ||
           I.getOpcode() == Instruction::URem ||
           I.getOpcode() == Instruction::SRem) &&
          "Unexpected instruction");
   Value *Divisor = I.getOperand(1);
   auto *CInt = dyn_cast<ConstantInt>(Divisor);
   return !CInt || CInt->isZero();
 }
 
-void InnerLoopVectorizer::widenInstruction(Instruction &I) {
+void InnerLoopVectorizer::widenInstruction(Instruction &I, VPUser &User,
+                                           VPTransformState &State) {
   switch (I.getOpcode()) {
   case Instruction::Call:
   case Instruction::Br:
   case Instruction::PHI:
   case Instruction::GetElementPtr:
   case Instruction::Select:
     llvm_unreachable("This instruction is handled by a different recipe.");
   case Instruction::UDiv:
void InnerLoopVectorizer::widenInstruction(Instruction &I, VPUser &User,
   case Instruction::And:
   case Instruction::Or:
   case Instruction::Xor: {
     // Just widen unops and binops.
     setDebugLocFromInst(Builder, &I);
 
     for (unsigned Part = 0; Part < UF; ++Part) {
       SmallVector<Value *, 2> Ops;
-      for (Value *Op : I.operands())
-        Ops.push_back(getOrCreateVectorValue(Op, Part));
+      for (VPValue *VPOp : User.operands())
+        Ops.push_back(State.get(VPOp, Part));
 
       Value *V = Builder.CreateNAryOp(I.getOpcode(), Ops);
 
       if (auto *VecOp = dyn_cast<Instruction>(V))
         VecOp->copyIRFlags(&I);
 
       // Use this vector value for all users of the original instruction.
       VectorLoopValueMap.setVectorValue(&I, Part, V);
       addMetadata(V, &I);
     }
 
     break;
   }
   case Instruction::ICmp:
   case Instruction::FCmp: {
     // Widen compares. Generate vector compares.
     bool FCmp = (I.getOpcode() == Instruction::FCmp);
     auto *Cmp = cast<CmpInst>(&I);
     setDebugLocFromInst(Builder, Cmp);
     for (unsigned Part = 0; Part < UF; ++Part) {
-      Value *A = getOrCreateVectorValue(Cmp->getOperand(0), Part);
-      Value *B = getOrCreateVectorValue(Cmp->getOperand(1), Part);
+      Value *A = State.get(User.getOperand(0), Part);
+      Value *B = State.get(User.getOperand(1), Part);
       Value *C = nullptr;
       if (FCmp) {
         // Propagate fast math flags.
         IRBuilder<>::FastMathFlagGuard FMFG(Builder);
         Builder.setFastMathFlags(Cmp->getFastMathFlags());
         C = Builder.CreateFCmp(Cmp->getPredicate(), A, B);
       } else {
         C = Builder.CreateICmp(Cmp->getPredicate(), A, B);
case Instruction::BitCast: {
     auto *CI = cast<CastInst>(&I);
     setDebugLocFromInst(Builder, CI);
 
     /// Vectorize casts.
     Type *DestTy =
         (VF == 1) ? CI->getType() : VectorType::get(CI->getType(), VF);
 
     for (unsigned Part = 0; Part < UF; ++Part) {
-      Value *A = getOrCreateVectorValue(CI->getOperand(0), Part);
+      Value *A = State.get(User.getOperand(0), Part);
       Value *Cast = Builder.CreateCast(CI->getOpcode(), A, DestTy);
       VectorLoopValueMap.setVectorValue(&I, Part, Cast);
       addMetadata(Cast, &I);
     }
     break;
   }
   default:
     // This instruction is not vectorized by simple widening.
VPWidenSelectRecipe *VPRecipeBuilder::tryToWidenSelect(Instruction *I,
 
   auto *SE = PSE.getSE();
   bool InvariantCond =
       SE->isLoopInvariant(PSE.getSCEV(I->getOperand(0)), OrigLoop);
   // Success: widen this instruction.
   return new VPWidenSelectRecipe(*SI, InvariantCond);
 }
 
-VPWidenRecipe *VPRecipeBuilder::tryToWiden(Instruction *I, VFRange &Range) {
+VPWidenRecipe *VPRecipeBuilder::tryToWiden(Instruction *I, VFRange &Range,
+                                           VPlan &Plan) {
   bool IsPredicated = LoopVectorizationPlanner::getDecisionAndClampRange(
       [&](unsigned VF) { return CM.isScalarWithPredication(I, VF); }, Range);
 
   if (IsPredicated)
     return nullptr;
 
   auto IsVectorizableOpcode = [](unsigned Opcode) {
     switch (Opcode) {
auto willWiden = [&](unsigned VF) -> bool {
     }
     return true;
   };
 
   if (!LoopVectorizationPlanner::getDecisionAndClampRange(willWiden, Range))
     return nullptr;
 
   // Success: widen this instruction.
-  return new VPWidenRecipe(*I);
+  // Create VPValue operands.
+  auto VPValues = map_range(
+      I->operands(), [&Plan](Value *Op) { return Plan.getOrAddVPValue(Op); });
+  return new VPWidenRecipe(*I, VPValues);
 }
 
 VPBasicBlock *VPRecipeBuilder::handleReplication(
     Instruction *I, VFRange &Range, VPBasicBlock *VPBB,
     DenseMap<Instruction *, VPReplicateRecipe *> &PredInst2Recipe,
     VPlanPtr &Plan) {
   bool IsUniform = LoopVectorizationPlanner::getDecisionAndClampRange(
       [&](unsigned VF) { return CM.isUniformAfterVectorization(I, VF); },
if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Instr)) {
     VPWidenGEPRecipe *Recipe = new VPWidenGEPRecipe(GEP, OrigLoop);
     setRecipe(Instr, Recipe);
     VPBB->appendRecipe(Recipe);
     return true;
   }
 
   // Check if Instr is to be widened by a general VPWidenRecipe, after
   // having first checked for specific widening recipes.
-  if ((Recipe = tryToWiden(Instr, Range))) {
+  if ((Recipe = tryToWiden(Instr, Range, *Plan))) {
     setRecipe(Instr, Recipe);
     VPBB->appendRecipe(Recipe);
     return true;
   }
 
   return false;
 }
 
void VPWidenCallRecipe::execute(VPTransformState &State) {
   State.ILV->widenCallInstruction(Ingredient, User, State);
 }
 
 void VPWidenSelectRecipe::execute(VPTransformState &State) {
   State.ILV->widenSelectInstruction(Ingredient, InvariantCond);
 }
 
 void VPWidenRecipe::execute(VPTransformState &State) {
-  State.ILV->widenInstruction(Ingredient);
+  State.ILV->widenInstruction(Ingredient, User, State);
 }
 
 void VPWidenGEPRecipe::execute(VPTransformState &State) {
   State.ILV->widenGEP(GEP, State.UF, State.VF, IsPtrLoopInvariant,
                       IsIndexLoopInvariant);
 }
 
 void VPWidenIntOrFpInductionRecipe::execute(VPTransformState &State) {
   assert(!State.Instance && "Int or FP induction being replicated.");
static ScalarEpilogueLowering getScalarEpilogueLowering(
     LoopVectorizationLegality &LVL) {
   bool OptSize =
       F->hasOptSize() || llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI,
                                                      PGSOQueryType::IRPass);
   // 1) OptSize takes precedence over all other options, i.e. if this is set,
   // don't look at hints or options, and don't request a scalar epilogue.
   if (OptSize && Hints.getForce() != LoopVectorizeHints::FK_Enabled)
     return CM_ScalarEpilogueNotAllowedOptSize;
 
   bool PredicateOptDisabled = PreferPredicateOverEpilog.getNumOccurrences() &&
                               !PreferPredicateOverEpilog;
 
   // 2) Next, if disabling predication is requested on the command line, honour
   // this and request a scalar epilogue.
   if (PredicateOptDisabled)
     return CM_ScalarEpilogueAllowed;
 

llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h

VPWidenCallRecipe *tryToWidenCall(Instruction *I, VFRange &Range,
                                     VPlan &Plan);
 
   VPWidenSelectRecipe *tryToWidenSelect(Instruction *I, VFRange &Range);
 
   /// Check if \p I can be widened within the given VF \p Range. If \p I can be
   /// widened for \p Range.Start, build a new VPWidenRecipe and return it.
   /// Range.End may be decreased to ensure same decision from \p Range.Start to
   /// \p Range.End.
-  VPWidenRecipe *tryToWiden(Instruction *I, VFRange &Range);
+  VPWidenRecipe *tryToWiden(Instruction *I, VFRange &Range, VPlan &Plan);
 
   /// Create a replicating region for instruction \p I that requires
   /// predication. \p PredRecipe is a VPReplicateRecipe holding \p I.
   VPRegionBlock *createReplicateRegion(Instruction *I, VPRecipeBase *PredRecipe,
                                        VPlanPtr &Plan);
 
 public:
   VPRecipeBuilder(Loop *OrigLoop, const TargetLibraryInfo *TLI,

llvm/lib/Transforms/Vectorize/VPlan.h

struct VPTransformState {
   /// as some Defs are still created by ILV and managed in its ValueMap, this
   /// method will delegate the call to ILV in such cases in order to provide
   /// callers a consistent API.
   /// \see set.
   Value *get(VPValue *Def, unsigned Part) {
     // If Values have been set for this Def return the one relevant for \p Part.
     if (Data.PerPartOutput.count(Def))
       return Data.PerPartOutput[Def][Part];
     // Def is managed by ILV: bring the Values from ValueMap.
     return Callback.getOrCreateVectorValues(VPValue2Value[Def], Part);
   }
 
   /// Get the generated Value for a given VPValue and given Part and Lane. Note
   /// that as per-lane Defs are still created by ILV and managed in its ValueMap
   /// this method currently just delegates the call to ILV.
   Value *get(VPValue *Def, const VPIteration &Instance) {
     return Callback.getOrCreateScalarValue(VPValue2Value[Def], Instance);
   }
 
   /// Set the generated Value for a given VPValue and a given Part.
   void set(VPValue *Def, Value *V, unsigned Part) {
     if (!Data.PerPartOutput.count(Def)) {
       DataState::PerPartValuesTy Entry(UF);
       Data.PerPartOutput[Def] = Entry;

 /// VPWidenRecipe is a recipe for producing a copy of vector type its
 /// ingredient. This recipe covers most of the traditional vectorization cases
 /// where each ingredient transforms into a vectorized version of itself.
 class VPWidenRecipe : public VPRecipeBase {
 private:
   /// Hold the instruction to be widened.
   Instruction &Ingredient;
 
+  VPUser User;
+
 public:
-  VPWidenRecipe(Instruction &I) : VPRecipeBase(VPWidenSC), Ingredient(I) {}
+  template <typename IterT>
+  VPWidenRecipe(Instruction &I, iterator_range<IterT> Operands)
+      : VPRecipeBase(VPWidenSC), Ingredient(I), User(Operands) {}
 
   ~VPWidenRecipe() override = default;
 
   /// Method to support type inquiry through isa, cast, and dyn_cast.
   static inline bool classof(const VPRecipeBase *V) {
     return V->getVPRecipeID() == VPRecipeBase::VPWidenSC;
   }
 

llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp

for (auto I = VPBB->begin(), E = VPBB->end(); I != E;) {
         InductionDescriptor II = Inductions.lookup(Phi);
         if (II.getKind() == InductionDescriptor::IK_IntInduction ||
             II.getKind() == InductionDescriptor::IK_FpInduction) {
           NewRecipe = new VPWidenIntOrFpInductionRecipe(Phi);
         } else
           NewRecipe = new VPWidenPHIRecipe(Phi);
       } else if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Inst)) {
         NewRecipe = new VPWidenGEPRecipe(GEP, OrigLoop);
-      } else
-        NewRecipe = new VPWidenRecipe(*Inst);
+      } else {
+        // Create VPValue operands.
+        auto VPValues = map_range(Inst->operands(), [&Plan](Value *Op) {
+          return Plan->getOrAddVPValue(Op);
+        });
+        NewRecipe = new VPWidenRecipe(*Inst, VPValues);
+      }
 
       NewRecipe->insertBefore(Ingredient);
       Ingredient->eraseFromParent();
     }
   }
 }

llvm/lib/Transforms/Vectorize/VPlanValue.h

 // coming from the input IR, instructions which VPlan will generate if executed
 // and live-outs which the VPlan will need to fix accordingly.
 class VPValue {
   friend class VPBuilder;
   friend class VPlanTransforms;
   friend class VPBasicBlock;
   friend class VPInterleavedAccessInfo;
   friend class VPSlotTracker;
+  friend class VPWidenRecipe;
 
 private:
   const unsigned char SubclassID; ///< Subclass identifier (for isa/dyn_cast).
 
   SmallVector<VPUser *, 1> Users;
 
 protected:
   // Hold the underlying Value, if any, attached to this VPValue.