[VPlan] Use VPValue def for VPInterleaveRecipe.

[fhahn]

Summary

This patch turns VPInterleaveRecipe into a VPValue and uses it
during VPlan construction and codegeneration instead of the plain IR
reference where possible.

VPInterleaveRecipe produces multiple result values (one for each group
member). This patch introduces a very simple VPMultiUse class.

VPInterleaveRecipe is a VPValue itself, to track all uses of any result
value. For each member, it also has an individual VPMultiUse which is
used for each member result.

This is mostly a straight-forward initial approach to model operations
with multiple results and I am sure this can be much improved.

[fhahn]

Split off multi-value parts to D87752

[dmgreen]

Would this now be based upon D88380 now instead?

I was thinking about interleaving a little. Trying some experiments to do with "de-interleaving" code. (Much like the VPlanSLP stuff I think).

The LLVM-IR equivalent of this would be an extract from a struct? Would it be worth representing this in a similar way? There could be a VPInterleaveRecipe which was a VPValue, with multiple "VPExtractElementRecipe"'s or something similar. Those would have 1 operand - the VPInterleaveRecipe and act as the "real" VPValues. They could have a 0 cost and possible hold the index for which element they are getting out of the interleaving group. Maybe there would be a validate method to make sure they remain as the only users of a VPInterleaveRecipe.

It is essentially equivalent to these other ways of handling it as a new type of VPValue, but isn't as tied into the lower levels of the class. What do you think? There might be something about it that wouldn't work, but it might be a simple way to handle these multiple-result nodes.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7323

Should this be surrounded by some sort of "if it's a load IG" check?

Equally, do we need to be adding the operands for a store IG group?

llvm/lib/Transforms/Vectorize/VPlan.h

1073

This looks left over from something.

[fhahn]

Thanks for taking a look. Addressed comments.

In D84684#2308806, @dmgreen wrote:

Would this now be based upon D88380 now instead?

Yep, let me update the links in Phab.

I was thinking about interleaving a little. Trying some experiments to do with "de-interleaving" code. (Much like the VPlanSLP stuff I think).

The LLVM-IR equivalent of this would be an extract from a struct? Would it be worth representing this in a similar way? There could be a VPInterleaveRecipe which was a VPValue, with multiple "VPExtractElementRecipe"'s or something similar. Those would have 1 operand - the VPInterleaveRecipe and act as the "real" VPValues. They could have a 0 cost and possible hold the index for which element they are getting out of the interleaving group. Maybe there would be a validate method to make sure they remain as the only users of a VPInterleaveRecipe.

It is essentially equivalent to these other ways of handling it as a new type of VPValue, but isn't as tied into the lower levels of the class. What do you think? There might be something about it that wouldn't work, but it might be a simple way to handle these multiple-result nodes.

Agreed, we could also model it so that VPInterleaveRecipe creates a single vector with all loaded values and then use extract instructions for the different parts of the interleave group. This keeps the modeling in the VPValue classes & co a bit simpler. But it has the cost that we need to add additional instructions/recipes to the plan, which need to be considered by other analyses and transformations. Another problem is that we would have to materialize a vector with all loaded values, which is completely artificial.

I think we have the make the following trade-off. Either we add additional complexity to VPValue & co to model recipes that produce multiple VPValues or we add additional instructions/recipes to the plans we generate.

Personally I think the additional complexity in VPValue & co is worth having simpler plans. Also, there might be additional complex recipes that produce multiple values in the future, which would also benefit from dedicated multi-value support.

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7323

Right, we only need to add VPValues for load groups, updated. I should probably also add a check that we do not add VPValues to a plan with underlying instructions/values of type void.

llvm/lib/Transforms/Vectorize/VPlan.h

1073

Indeed, that should be gone now, thanks!

[dmgreen]

Agreed, we could also model it so that VPInterleaveRecipe creates a single vector with all loaded values and then use extract instructions for the different parts of the interleave group. This keeps the modeling in the VPValue classes & co a bit simpler. But it has the cost that we need to add additional instructions/recipes to the plan, which need to be considered by other analyses and transformations. Another problem is that we would have to materialize a vector with all loaded values, which is completely artificial.

I was considering that the VPInterleaveRecipe and the VPExtract's would work in tandem. You would never get the "all loaded values" vector, it would just create the interleaving gorup as it does at the moment, with the VPExtracts representing the values for each item. They would still be special values, VPExtracts would be the only type of recipe that could use a VPInterleave, and the VPExtract would not really contain anything in it's execute method.

You could just consider the VPExtract as a new type of VPUser, not a VPRecipe directly. It's really a convenient way of using the operands to join up the two elements, not having to rely upon UnderlyingOrProducerTy. I guess that's a lot like the VPMultiValue concept. Using a new type of VPUser removes the complexity from VPValue at least!

I think we have the make the following trade-off. Either we add additional complexity to VPValue & co to model recipes that produce multiple VPValues or we add additional instructions/recipes to the plans we generate.

Personally I think the additional complexity in VPValue & co is worth having simpler plans. Also, there might be additional complex recipes that produce multiple values in the future, which would also benefit from dedicated multi-value support.

It feels like we have one type system (VPValueSC/VPInstructionSC/VPWidenSC/etc), and we are adding another system on top of it, for different types of VPValue via UnderlyingOrProducerTy being virtual or subvalues or concrete.

Is there a nice way at the moment of detecting _which_ output you are seeing from a subvalue VPValue? As in which member index it would be from a interleaved group.

llvm/lib/Transforms/Vectorize/VPlan.cpp

980

Is this used at the moment? Should we be blindly replacing all the different uses with the same value?

llvm/lib/Transforms/Vectorize/VPlan.h

1028–1029

-> VPRecipeBase::VPInterleaveSC

1033

Does anything delete these Defs?

[fhahn]

In D84684#2314359, @dmgreen wrote:

Agreed, we could also model it so that VPInterleaveRecipe creates a single vector with all loaded values and then use extract instructions for the different parts of the interleave group. This keeps the modeling in the VPValue classes & co a bit simpler. But it has the cost that we need to add additional instructions/recipes to the plan, which need to be considered by other analyses and transformations. Another problem is that we would have to materialize a vector with all loaded values, which is completely artificial.

I was considering that the VPInterleaveRecipe and the VPExtract's would work in tandem. You would never get the "all loaded values" vector, it would just create the interleaving gorup as it does at the moment, with the VPExtracts representing the values for each item. They would still be special values, VPExtracts would be the only type of recipe that could use a VPInterleave, and the VPExtract would not really contain anything in it's execute method.

You could just consider the VPExtract as a new type of VPUser, not a VPRecipe directly. It's really a convenient way of using the operands to join up the two elements, not having to rely upon UnderlyingOrProducerTy. I guess that's a lot like the VPMultiValue concept. Using a new type of VPUser removes the complexity from VPValue at least!

IICU conceptually the suggested VPExtract and the 'sub-value' are effectively the same thing, just slightly different in terms where exactly they are defined.

I think most of the trade-offs are still similar, with a VPExtract version probably ending up slightly simpler and the sub-value one being a bit more complex but also a bit more explicit in the modeling and more flexible in the future. I think as-is, the 'sub-value' approach is relatively lightweight and does not add too much complexity and the difference between the VPExtract alternative should be relatively small. But if people generally prefer this approach I am happy to update the patches.

I think we have the make the following trade-off. Either we add additional complexity to VPValue & co to model recipes that produce multiple VPValues or we add additional instructions/recipes to the plans we generate.

Personally I think the additional complexity in VPValue & co is worth having simpler plans. Also, there might be additional complex recipes that produce multiple values in the future, which would also benefit from dedicated multi-value support.

It feels like we have one type system (VPValueSC/VPInstructionSC/VPWidenSC/etc), and we are adding another system on top of it, for different types of VPValue via UnderlyingOrProducerTy being virtual or subvalues or concrete.

That definitely can be improved. We should probably just rely on the VPValueID and use that to extract the right value from a union instead of sum type.

Is there a nice way at the moment of detecting _which_ output you are seeing from a subvalue VPValue? As in which member index it would be from a interleaved group.

Currently there's no nice way, but it can easily be done by getting the defining value and then checking which def corresponds to the sub-value. It should be possible to add a convenient interface once there's a need for it.

[fhahn]

Delete sub-value defs in VPInterleaveRecipe, handle virtual values in dropAllReferences, assert that repalceAllUses is not called with virtual VPValues.

[fhahn]

Use VPRecipeBase::VPInterleaveSC.

llvm/lib/Transforms/Vectorize/VPlan.cpp

980

No it's not used (also not in follow-ups). I added an assert.

llvm/lib/Transforms/Vectorize/VPlan.h

1033

They should be freed in the destructor. Done now.

[fhahn]

Superseded by VPDef approach.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

public:
   /// Construct the vector value of a scalarized value \p V one lane at a time.
   void packScalarIntoVectorValue(Value *V, const VPIteration &Instance);
 
   /// Try to vectorize interleaved access group \p Group with the base address
   /// given in \p Addr, optionally masking the vector operations if \p
   /// BlockInMask is non-null. Use \p State to translate given VPValues to IR
   /// values in the vectorized loop.
   void vectorizeInterleaveGroup(const InterleaveGroup<Instruction> *Group,
+                                ArrayRef<VPValue *> VPDefs,
                                 VPTransformState &State, VPValue *Addr,
                                 VPValue *BlockInMask = nullptr);
 
   /// Vectorize Load and Store instructions with the base address given in \p
   /// Addr, optionally masking the vector operations if \p BlockInMask is
   /// non-null. Use \p State to translate given VPValues to IR values in the
   /// vectorized loop.
   void vectorizeMemoryInstruction(Instruction *Instr, VPTransformState &State,

 //   }
 // To:
 //   %R_G.vec = shuffle %R.vec, %G.vec, <0, 1, 2, ..., 7>
 //   %B_U.vec = shuffle %B.vec, undef, <0, 1, 2, 3, u, u, u, u>
 //   %interleaved.vec = shuffle %R_G.vec, %B_U.vec,
 //        <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>    ; Interleave R,G,B elements
 //   store <12 x i32> %interleaved.vec              ; Write 4 tuples of R,G,B
 void InnerLoopVectorizer::vectorizeInterleaveGroup(
-    const InterleaveGroup<Instruction> *Group, VPTransformState &State,
-    VPValue *Addr, VPValue *BlockInMask) {
+    const InterleaveGroup<Instruction> *Group, ArrayRef<VPValue *> VPDefs,
+    VPTransformState &State, VPValue *Addr, VPValue *BlockInMask) {
   Instruction *Instr = Group->getInsertPos();
   const DataLayout &DL = Instr->getModule()->getDataLayout();
 
   // Prepare for the vector type of the interleaved load/store.
   Type *ScalarTy = getMemInstValueType(Instr);
   unsigned InterleaveFactor = Group->getFactor();
   assert(!VF.isScalable() && "scalable vectors not yet supported.");
   auto *VecTy = VectorType::get(ScalarTy, VF * InterleaveFactor);
for (unsigned Part = 0; Part < UF; Part++) {
         NewLoad = Builder.CreateAlignedLoad(VecTy, AddrParts[Part],
                                             Group->getAlign(), "wide.vec");
       Group->addMetadata(NewLoad);
       NewLoads.push_back(NewLoad);
     }
 
     // For each member in the group, shuffle out the appropriate data from the
     // wide loads.
+    unsigned J = 0;
     for (unsigned I = 0; I < InterleaveFactor; ++I) {
       Instruction *Member = Group->getMember(I);
 
       // Skip the gaps in the group.
       if (!Member)
         continue;
 
       assert(!VF.isScalable() && "scalable vectors not yet supported.");
       auto StrideMask =
           createStrideMask(I, InterleaveFactor, VF.getKnownMinValue());
       for (unsigned Part = 0; Part < UF; Part++) {
         Value *StridedVec = Builder.CreateShuffleVector(
             NewLoads[Part], StrideMask, "strided.vec");
 
         // If this member has different type, cast the result type.
         if (Member->getType() != ScalarTy) {
           assert(!VF.isScalable() && "VF is assumed to be non scalable.");
           VectorType *OtherVTy = VectorType::get(Member->getType(), VF);
           StridedVec = createBitOrPointerCast(StridedVec, OtherVTy, DL);
         }
 
         if (Group->isReverse())
           StridedVec = reverseVector(StridedVec);
 
-        VectorLoopValueMap.setVectorValue(Member, Part, StridedVec);
+        State.set(VPDefs[J], Member, StridedVec, Part);
       }
+      ++J;
     }
     return;
   }
 
   // The sub vector type for current instruction.
   assert(!VF.isScalable() && "VF is assumed to be non scalable.");
   auto *SubVT = VectorType::get(ScalarTy, VF);
 
for (auto *Predecessor : predecessors(BB)) {
     }
 
     BlockMask = Builder.createOr(BlockMask, EdgeMask);
   }
 
   return BlockMaskCache[BB] = BlockMask;
 }
 
-VPWidenMemoryInstructionRecipe *
-VPRecipeBuilder::tryToWidenMemory(Instruction *I, VFRange &Range,
+VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(Instruction *I, VFRange &Range,
                                                 VPlanPtr &Plan) {
   assert((isa<LoadInst>(I) || isa<StoreInst>(I)) &&
          "Must be called with either a load or store");
 
   auto willWiden = [&](ElementCount VF) -> bool {
     assert(!VF.isScalable() && "unexpected scalable ElementCount");
     if (VF.isScalar())
       return false;
     LoopVectorizationCostModel::InstWidening Decision =
VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(Instruction *I, VFRange &Range,
   if (!LoopVectorizationPlanner::getDecisionAndClampRange(willWiden, Range))
     return nullptr;
 
   VPValue *Mask = nullptr;
   if (Legal->isMaskRequired(I))
     Mask = createBlockInMask(I->getParent(), Plan);
 
   VPValue *Addr = Plan->getOrAddVPValue(getLoadStorePointerOperand(I));
+  auto II = InsertPtToGroup.find(I);
+  if (II != InsertPtToGroup.end()) {
+    auto *IG = II->second;
+    auto *InterleaveG = new VPInterleaveRecipe(IG, Addr, Mask);
+
+    // If this is a load interleave group, add sub-values for each member.
+    if (isa<LoadInst>(I)) {
+      unsigned j = 0;
+      for (unsigned i = 0; i < IG->getFactor(); i++)
+        if (Instruction *Member = IG->getMember(i)) {
+          Plan->addVPValue(Member, InterleaveG->getResult(j));
+          j++;
+        }
+    }
+    return InterleaveG;
+  }
+
   if (LoadInst *Load = dyn_cast<LoadInst>(I))
     return new VPWidenMemoryInstructionRecipe(*Load, Addr, Mask);
 
   StoreInst *Store = cast<StoreInst>(I);
   VPValue *StoredValue = Plan->getOrAddVPValue(Store->getValueOperand());
   return new VPWidenMemoryInstructionRecipe(*Store, Addr, StoredValue, Mask);
 }
 
VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
     const DenseMap<Instruction *, Instruction *> &SinkAfter) {
 
   // Hold a mapping from predicated instructions to their recipes, in order to
   // fix their AlsoPack behavior if a user is determined to replicate and use a
   // scalar instead of vector value.
   DenseMap<Instruction *, VPReplicateRecipe *> PredInst2Recipe;
 
   SmallPtrSet<const InterleaveGroup<Instruction> *, 1> InterleaveGroups;
+  SmallPtrSet<Instruction *, 8> DeadInterleaveGroupMembers;
 
   VPRecipeBuilder RecipeBuilder(OrigLoop, TLI, Legal, CM, PSE, Builder);
 
   // ---------------------------------------------------------------------------
   // Pre-construction: record ingredients whose recipes we'll need to further
   // process after constructing the initial VPlan.
   // ---------------------------------------------------------------------------
 
for (InterleaveGroup<Instruction> *IG : IAI.getInterleaveGroups()) {
     auto applyIG = [IG, this](ElementCount VF) -> bool {
       return (VF.isVector() && // Query is illegal for VF == 1
               CM.getWideningDecision(IG->getInsertPos(), VF) ==
                   LoopVectorizationCostModel::CM_Interleave);
     };
     if (!getDecisionAndClampRange(applyIG, Range))
       continue;
     InterleaveGroups.insert(IG);
+    RecipeBuilder.recordInterleaveGroup(IG);
     for (unsigned i = 0; i < IG->getFactor(); i++)
-      if (Instruction *Member = IG->getMember(i))
+      if (Instruction *Member = IG->getMember(i)) {
         RecipeBuilder.recordRecipeOf(Member);
+        if (Member != IG->getInsertPos())
+          DeadInterleaveGroupMembers.insert(Member);
+      }
   };
 
+  auto skipDeadInterleaveMembers =
+      [&DeadInterleaveGroupMembers](Instruction *I) {
+        BasicBlock *BB = I->getParent();
+        for (auto &I : make_range(I->getIterator(), BB->end()))
+          if (!DeadInterleaveGroupMembers.contains(&I))
+            return &I;
+        llvm_unreachable("Need to find a valid insert point");
+      };
+  // Mark instructions we'll need to sink later and their targets as
+  // ingredients whose recipe we'll need to record.
+  for (auto &Entry : SinkAfter) {
+    RecipeBuilder.recordRecipeOf(skipDeadInterleaveMembers(Entry.first));
+    RecipeBuilder.recordRecipeOf(skipDeadInterleaveMembers(Entry.second));
+  }
+
   // ---------------------------------------------------------------------------
   // Build initial VPlan: Scan the body of the loop in a topological order to
   // visit each basic block after having visited its predecessor basic blocks.
   // ---------------------------------------------------------------------------
 
   // Create a dummy pre-entry VPBasicBlock to start building the VPlan.
   auto Plan = std::make_unique<VPlan>();
   VPBasicBlock *VPBB = new VPBasicBlock("Pre-Entry");
for (BasicBlock *BB : make_range(DFS.beginRPO(), DFS.endRPO())) {
 
     // Introduce each ingredient into VPlan.
     // TODO: Model and preserve debug instrinsics in VPlan.
     for (Instruction &I : BB->instructionsWithoutDebug()) {
       Instruction *Instr = &I;
 
       // First filter out irrelevant instructions, to ensure no recipes are
       // built for them.
-      if (isa<BranchInst>(Instr) || DeadInstructions.count(Instr))
+      if (isa<BranchInst>(Instr) || DeadInstructions.count(Instr) ||
+          DeadInterleaveGroupMembers.contains(Instr))
         continue;
 
       if (auto Recipe =
               RecipeBuilder.tryToCreateWidenRecipe(Instr, Range, Plan)) {
         // Check if the recipe can be converted to a VPValue. We need the extra
         // down-casting step until VPRecipeBase inherits from VPValue.
         VPValue *MaybeVPValue = Recipe->toVPValue();
         if (!Instr->getType()->isVoidTy() && MaybeVPValue) {
VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
 
   // ---------------------------------------------------------------------------
   // Transform initial VPlan: Apply previously taken decisions, in order, to
   // bring the VPlan to its final state.
   // ---------------------------------------------------------------------------
 
   // Apply Sink-After legal constraints.
   for (auto &Entry : SinkAfter) {
-    VPRecipeBase *Sink = RecipeBuilder.getRecipe(Entry.first);
-    VPRecipeBase *Target = RecipeBuilder.getRecipe(Entry.second);
+    VPRecipeBase *Sink =
+        RecipeBuilder.getRecipe(skipDeadInterleaveMembers(Entry.first));
+    VPRecipeBase *Target =
+        RecipeBuilder.getRecipe(skipDeadInterleaveMembers(Entry.second));
     Sink->moveAfter(Target);
   }
 
-  // Interleave memory: for each Interleave Group we marked earlier as relevant
-  // for this VPlan, replace the Recipes widening its memory instructions with a
-  // single VPInterleaveRecipe at its insertion point.
-  for (auto IG : InterleaveGroups) {
-    auto *Recipe = cast<VPWidenMemoryInstructionRecipe>(
-        RecipeBuilder.getRecipe(IG->getInsertPos()));
-    (new VPInterleaveRecipe(IG, Recipe->getAddr(), Recipe->getMask()))
-        ->insertBefore(Recipe);
-
-    for (unsigned i = 0; i < IG->getFactor(); ++i)
-      if (Instruction *Member = IG->getMember(i)) {
-        VPValue *NewVPV = nullptr;
-        if (!Member->getType()->isVoidTy()) {
-          NewVPV = new VPValue(Member);
-          Plan->getVPValue(Member)->replaceAllUsesWith(NewVPV);
-        }
-        RecipeBuilder.getRecipe(Member)->eraseFromParent();
-        if (NewVPV)
-          Plan->addVPValue(Member, NewVPV);
-      }
-  }
-
   // Adjust the recipes for any inloop reductions.
   if (Range.Start > 1)
     adjustRecipesForInLoopReductions(Plan, RecipeBuilder);
 
   // Finally, if tail is folded by masking, introduce selects between the phi
   // and the live-out instruction of each reduction, at the end of the latch.
   if (CM.foldTailByMasking() && !Legal->getReductionVars().empty()) {
     Builder.setInsertPoint(VPBB);
for (unsigned In = 0; In < NumIncoming; ++In) {
     }
   }
   for (unsigned Part = 0; Part < State.UF; ++Part)
     State.ValueMap.setVectorValue(Phi, Part, Entry[Part]);
 }
 
 void VPInterleaveRecipe::execute(VPTransformState &State) {
   assert(!State.Instance && "Interleave group being replicated.");
-  State.ILV->vectorizeInterleaveGroup(IG, State, getAddr(), getMask());
+  State.ILV->vectorizeInterleaveGroup(IG, getDefs(), State, getAddr(),
+                                      getMask());
 }
 
 void VPReductionRecipe::execute(VPTransformState &State) {
   assert(!State.Instance && "Reduction being replicated.");
   for (unsigned Part = 0; Part < State.UF; ++Part) {
     unsigned Kind = RdxDesc->getRecurrenceKind();
     Value *NewVecOp = State.get(VecOp, Part);
     Value *NewRed =

llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h

class VPRecipeBuilder {
   EdgeMaskCacheTy EdgeMaskCache;
   BlockMaskCacheTy BlockMaskCache;
 
   // VPlan-VPlan transformations support: Hold a mapping from ingredients to
   // their recipe. To save on memory, only do so for selected ingredients,
   // marked by having a nullptr entry in this map.
   DenseMap<Instruction *, VPRecipeBase *> Ingredient2Recipe;
 
+  DenseMap<Instruction *, const InterleaveGroup<Instruction> *> InsertPtToGroup;
+
   /// Check if \p I can be widened at the start of \p Range and possibly
   /// decrease the range such that the returned value holds for the entire \p
   /// Range. The function should not be called for memory instructions or calls.
   bool shouldWiden(Instruction *I, VFRange &Range) const;
 
   /// Check if the load or store instruction \p I should widened for \p
   /// Range.Start and potentially masked. Such instructions are handled by a
   /// recipe that takes an additional VPInstruction for the mask.
-  VPWidenMemoryInstructionRecipe *
-  tryToWidenMemory(Instruction *I, VFRange &Range, VPlanPtr &Plan);
+  VPRecipeBase *tryToWidenMemory(Instruction *I, VFRange &Range,
+                                 VPlanPtr &Plan);
 
   /// Check if an induction recipe should be constructed for \I. If so build and
   /// return it. If not, return null.
   VPWidenIntOrFpInductionRecipe *tryToOptimizeInductionPHI(PHINode *Phi) const;
 
   /// Optimize the special case where the operand of \p I is a constant integer
   /// induction variable.
   VPWidenIntOrFpInductionRecipe *
public:
   /// Mark given ingredient for recording its recipe once one is created for
   /// it.
   void recordRecipeOf(Instruction *I) {
     assert((!Ingredient2Recipe.count(I) || Ingredient2Recipe[I] == nullptr) &&
            "Recipe already set for ingredient");
     Ingredient2Recipe[I] = nullptr;
   }
 
+  void recordInterleaveGroup(const InterleaveGroup<Instruction> *IG) {
+    InsertPtToGroup[IG->getInsertPos()] = IG;
+  }
+
   /// Return the recipe created for given ingredient.
   VPRecipeBase *getRecipe(Instruction *I) {
     assert(Ingredient2Recipe.count(I) &&
            "Recording this ingredients recipe was not requested");
     assert(Ingredient2Recipe[I] != nullptr &&
            "Ingredient doesn't have a recipe");
     return Ingredient2Recipe[I];
   }

llvm/lib/Transforms/Vectorize/VPlan.h

public:
 
   /// Print the recipe.
   void print(raw_ostream &O, const Twine &Indent,
              VPSlotTracker &SlotTracker) const override;
 };
 
 /// VPInterleaveRecipe is a recipe for transforming an interleave group of load
 /// or stores into one wide load/store and shuffles.
-class VPInterleaveRecipe : public VPRecipeBase, public VPUser {
+class VPInterleaveRecipe : public VPRecipeBase, public VPValue, public VPUser {
   const InterleaveGroup<Instruction> *IG;
 
 public:
+  SmallVector<VPValue *, 4> Defs;
+
   VPInterleaveRecipe(const InterleaveGroup<Instruction> *IG, VPValue *Addr,
                      VPValue *Mask)
-      : VPRecipeBase(VPInterleaveSC), VPUser({Addr}), IG(IG) {
+      : VPRecipeBase(VPRecipeBase::VPInterleaveSC),
+        VPValue(VPValue::VPVInterleaveSC), VPUser({Addr}), IG(IG) {
     if (Mask)
       addOperand(Mask);
+    for (unsigned i = 0; i < IG->getNumMembers(); i++)
+      Defs.push_back(new VPValue(this));
+  }
+  ~VPInterleaveRecipe() override {
+    for (auto *Def : Defs)
+      delete Def;
   }
-  ~VPInterleaveRecipe() override = default;
 
   /// Method to support type inquiry through isa, cast, and dyn_cast.
   static inline bool classof(const VPRecipeBase *V) {
     return V->getVPRecipeID() == VPRecipeBase::VPInterleaveSC;
   }
+  static inline bool classof(const VPValue *V) {
+    return V->getVPValueID() == VPValue::VPVInterleaveSC;
+  }
 
   /// Return the address accessed by this recipe.
   VPValue *getAddr() const {
     return getOperand(0); // Address is the 1st, mandatory operand.
   }
 
   /// Return the mask used by this recipe. Note that a full mask is represented
   /// by a nullptr.
   VPValue *getMask() const {
     // Mask is optional and therefore the last, currently 2nd operand.
     return getNumOperands() == 2 ? getOperand(1) : nullptr;
   }
 
   /// Generate the wide load or store, and shuffles.
   void execute(VPTransformState &State) override;
 
   /// Print the recipe.
   void print(raw_ostream &O, const Twine &Indent,
              VPSlotTracker &SlotTracker) const override;
 
   const InterleaveGroup<Instruction> *getInterleaveGroup() { return IG; }
+
+  VPValue *getResult(unsigned Idx) { return Defs[Idx]; }
+  ArrayRef<VPValue *> getDefs() { return Defs; }
 };
 
 /// A recipe to represent inloop reduction operations, performing a reduction on
 /// a vector operand into a scalar value, and adding the result to a chain.
 class VPReductionRecipe : public VPRecipeBase {
   /// The recurrence decriptor for the reduction in question.
   RecurrenceDescriptor *RdxDesc;
   /// The original instruction being converted to a reduction.
   Instruction *I;
   /// The VPValue of the vector value to be reduced.
   VPValue *VecOp;

llvm/lib/Transforms/Vectorize/VPlan.cpp

if (EnableVPlanNativePath && (CBV = getCondBit())) {
     ReplaceInstWithInst(CurrentTerminator, CondBr);
   }
 
   LLVM_DEBUG(dbgs() << "LV: filled BB:" << *NewBB);
 }
 
 void VPBasicBlock::dropAllReferences(VPValue *NewValue) {
   for (VPRecipeBase &R : Recipes) {
-    if (auto *VPV = R.toVPValue())
+    if (auto *InterleaveR = dyn_cast<VPInterleaveRecipe>(&R)) {
+      for (auto *Def : InterleaveR->Defs)
+        Def->replaceAllUsesWith(NewValue);
+    } else {
+      if (auto *VPV = R.toVPValue()) {
+        assert(VPV->isConcrete() && "virtual/sub-values need special handling");
         VPV->replaceAllUsesWith(NewValue);
+      }
+    }
 
     if (auto *User = R.toVPUser())
       for (unsigned I = 0, E = User->getNumOperands(); I != E; I++)
         User->setOperand(I, NewValue);
   }
 }
 
 void VPRegionBlock::execute(VPTransformState *State) {
void VPWidenCanonicalIVRecipe::print(raw_ostream &O, const Twine &Indent,
   O << "\"EMIT ";
   getVPValue()->printAsOperand(O, SlotTracker);
   O << " = WIDEN-CANONICAL-INDUCTION";
 }
 
 template void DomTreeBuilder::Calculate<VPDominatorTree>(VPDominatorTree &DT);
 
 void VPValue::replaceAllUsesWith(VPValue *New) {
+  assert((isConcrete() || isSubValue()) &&
+         "can only replace concrete or sub-values");
   for (unsigned J = 0; J < getNumUsers();) {
     VPUser *User = Users[J];
     unsigned NumUsers = getNumUsers();
     for (unsigned I = 0, E = User->getNumOperands(); I < E; ++I)
       if (User->getOperand(I) == this)
         User->setOperand(I, New);
     // If a user got removed after updating the current user, the next user to
     // update will be moved to the current position, so we only need to

 VPInterleavedAccessInfo::VPInterleavedAccessInfo(VPlan &Plan,
                                                  InterleavedAccessInfo &IAI) {
   Old2NewTy Old2New;
   visitRegion(cast<VPRegionBlock>(Plan.getEntry()), Old2New, IAI);
 }
 
 void VPSlotTracker::assignSlot(const VPValue *V) {
   assert(Slots.find(V) == Slots.end() && "VPValue already has a slot!");
+  if (!V->isSubValue()) {
     const Value *UV = V->getUnderlyingValue();
     if (UV)
       return;
+  }
   const auto *VPI = dyn_cast<VPInstruction>(V);
   if (VPI && !VPI->hasResult())
     return;
 
   Slots[V] = NextSlot++;
 }
 
 void VPSlotTracker::assignSlots(const VPBlockBase *VPBB) {

llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll

 ; CHECK-LABEL: @PR34743(
 ; CHECK: vector.body:
 ; CHECK:   %vector.recur = phi <4 x i16> [ %vector.recur.init, %vector.ph ], [ %[[VSHUF1:.+]], %vector.body ]
 ; CHECK:   %wide.vec = load <8 x i16>
 ; CHECK:   %[[VSHUF0:.+]] = shufflevector <8 x i16> %wide.vec, <8 x i16> undef, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
 ; CHECK:   %[[VSHUF1:.+]] = shufflevector <8 x i16> %wide.vec, <8 x i16> undef, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
 ; CHECK:   %[[VSHUF:.+]] = shufflevector <4 x i16> %vector.recur, <4 x i16> %[[VSHUF1]], <4 x i32> <i32 3, i32 4, i32 5, i32 6>
 ; CHECK:   sext <4 x i16> %[[VSHUF0]] to <4 x i32>
-; CHECK:   sext <4 x i16> %[[VSHUF]] to <4 x i32>
 ; CHECK:   sext <4 x i16> %[[VSHUF1]] to <4 x i32>
+; CHECK:   sext <4 x i16> %[[VSHUF]] to <4 x i32>
 ; CHECK:   mul nsw <4 x i32>
 ; CHECK:   mul nsw <4 x i32>
 
 define void @PR34743(i16* %a, i32* %b, i64 %n) {
 entry:
   %.pre = load i16, i16* %a
   br label %loop