[VPlan] Use isUniformAfterVec in VPReplicateRecipe::execute.

[fhahn]

Summary

I was unable to find a case where this actually changes generated code,
but it enables the bug fix in D144434. It also brings codegen in line
with the handling of stores to uniform addresses in the cost model
(D134460).

[Ayal]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9572–9575

only needs a single only...

9575

This is a bit fragile considering that isUniformAfterVectorization() in general also represents lane-varying values whose first lane need only be used, as in vector stores - see isUniformDecision .

[Ayal]

LGTM, with a couple of nits.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9572–9575

9575

This patch is fine - effectively considering Replicate recipe invariants as "uniform addresses" in addition to current isLiveIns (aka !hasDefiningRecipe).

Above comment (independent of this patch) refers to vputils which introduced "onlyFirstLaneUsed" to handle the "AfterVectorization" part, and its isUniformAfterVectorization() should be renamed isUniform(). In fact, it currently checks isInvariant(), i.e., same value across all loop iterations. This could be improved to "uniform" - same value across lanes (per part) as in D148841, and possibly also to same value across lanes*parts (per vectorized iteration) - but the latter refers to UF.

[Ayal]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9575

Also, regarding isUniform - in addition to recipe-less LiveIns and uniform Replicate recipes, uniform values may also be provided by non-Replicate recipes placed in vec/pre-headers and by WidenGEP recipe (though broadcasted into a vector).

[fhahn]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

9572–9575

Fixed, thanks!

9575

Above comment (independent of this patch) refers to vputils which introduced "onlyFirstLaneUsed" to handle the "AfterVectorization" part, and its isUniformAfterVectorization() should be renamed isUniform(). In fact, it currently checks isInvariant(), i.e., same value across all loop iterations. This could be improved to "uniform" - same value across lanes (per part) as in D148841, and possibly also to same value across lanes*parts (per vectorized iteration) - but the latter refers to UF.

Let me do the renaming as follow-up. I think we shouldn't need any additional work to benefit from D148841 here; uniformity should be propagated to replicate recipes?

Also, regarding isUniform - in addition to recipe-less LiveIns and uniform Replicate recipes, uniform values may also be provided by non-Replicate recipes placed in vec/pre-headers and by WidenGEP recipe (though broadcasted into a vector).

This will be addressed by fixing the TODO in isDefinedOutsideVectorRegions, will do as follow-up once I have a suitable test case.

[fhahn]

Address comments and rebase, plan to land it soon.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

if (IsUniform) {
 
     // Uniform within VL means we need to generate lane 0 only for each
     // unrolled copy.
     for (unsigned Part = 0; Part < State.UF; ++Part)
       State.ILV->scalarizeInstruction(UI, this, VPIteration(Part, 0), State);
     return;
   }
 
-  // A store of a loop varying value to a loop invariant address only
-  // needs only the last copy of the store.
-  if (isa<StoreInst>(UI) && getOperand(1)->isLiveIn()) {
+  // A store of a loop varying value to a uniform address only needs the last
+  // copy of the store.
+  if (isa<StoreInst>(UI) &&
+      vputils::isUniformAfterVectorization(getOperand(1))) {
     auto Lane = VPLane::getLastLaneForVF(State.VF);
     State.ILV->scalarizeInstruction(UI, this, VPIteration(State.UF - 1, Lane),
                                     State);
     return;
   }
 
   // Generate scalar instances for all VF lanes of all UF parts.
   assert(!State.VF.isScalable() && "Can't scalarize a scalable vector");