[LV] Fix PR38786 - consider first order recurrence phis non-uniform

[orivej]

Summary

Patch by Ayal Zaks

[anna]

LGTM. thanks for the fix!

[orivej]

Renamed %i32as64 to %i64next.

[orivej]

Specify in the test comment that this is not about all phis, but only about first order recurrence phis.

[orivej]

test/Transforms/LoopVectorize/X86/uniform-phi.ll

90 ↗

(On Diff #163849)

Note: if I replace %i64next = zext i32 %i32next to i64 with %i64next = add i64 %phi64, 1 it ceases to be a first order recurrence, and the loop vectorizer considers %phi32 and %phi64 uniform and correctly optimizes this.

[anna]

test/Transforms/LoopVectorize/X86/uniform-phi.ll

90 ↗

(On Diff #163849)

yes, that is expected :)

iv64next is a uniform IV user, so both the phis and their increments are considered uniforms. See the processing of uniforms for IV phis in the source code, just after the above fix.

[orivej]

I think this is ready. Would someone like to commit it?

[anna]

In D51639#1223677, @orivej wrote:

I think this is ready. Would someone like to commit it?

Okay, I'll commit it.

llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp

void LoopVectorizationCostModel::collectLoopUniforms(unsigned VF) {
   unsigned idx = 0;
   while (idx != Worklist.size()) {
     Instruction *I = Worklist[idx++];
 
     for (auto OV : I->operand_values()) {
       // isOutOfScope operands cannot be uniform instructions.
       if (isOutOfScope(OV))
         continue;
+      // First order recurrence Phi's should typically be considered
+      // non-uniform.
+      auto *OP = dyn_cast<PHINode>(OV);
+      if (OP && Legal->isFirstOrderRecurrence(OP))
+        continue;
       // If all the users of the operand are uniform, then add the
       // operand into the uniform worklist.
       auto *OI = cast<Instruction>(OV);
       if (llvm::all_of(OI->users(), [&](User *U) -> bool {
             auto *J = cast<Instruction>(U);
             return Worklist.count(J) ||
                    (OI == getLoadStorePointerOperand(J) &&
                     isUniformDecision(J, VF));

llvm/trunk/test/Transforms/LoopVectorize/X86/uniform-phi.ll

for.body:                                         ; preds = %for.body, %entry
   %exitcond = icmp eq i64 %indvars.iv, 1599
   br i1 %exitcond, label %for.end, label %for.body
 
 for.end:                                          ; preds = %for.body
   %retval = add i64 %indvars.iv, %indvars.iv.next
   ret i64 %retval
 }
 
+; CHECK-LABEL: PR38786
+; Check that first order recurrence phis (%phi32 and %phi64) are not uniform.
+; CHECK-NOT: LV: Found uniform instruction:   %phi
+define void @PR38786(double* %y, double* %x, i64 %n) {
+entry:
+  br label %for.body
+
+for.body:
+  %phi32 = phi i32 [ 0, %entry ], [ %i32next, %for.body ]
+  %phi64 = phi i64 [ 0, %entry ], [ %i64next, %for.body ]
+  %i32next = add i32 %phi32, 1
+  %i64next = zext i32 %i32next to i64
+  %xip = getelementptr inbounds double, double* %x, i64 %i64next
+  %yip = getelementptr inbounds double, double* %y, i64 %phi64
+  %xi = load double, double* %xip, align 8
+  store double %xi, double* %yip, align 8
+  %cmp = icmp slt i64 %i64next, %n
+  br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+  ret void
+}