[LV] Fix PR36983. For a given recurrence, fix all phis in exit block

[roman.shirokiy]

Summary

There could be several identical PHIs in exit block using same loop recurrence.

For example, we have a following 2 phis in the loop, both %local_2_23 and %local_4_25 values are not used inside the loop.

loop:
 %local_2_23 = phi i32 [ %iv.rem, %loop ], [ %43, %preheader ]
 %local_4_25 = phi i32 [ %iv.rem, %loop ], [ -25, %preheader ]

LCSSA will produce two corresponding phis in exit block.

loop.exit:
%local_2_23.lcssa = phi i32 [ %local_2_23, %loop ]
%local_4_25.lcssa = phi i32 [ %local_4_25, %loop ]

Further passes may deduce that these depend only on iv value hence we may end up with the identical phi nodes in loop exit block.

 loop:
 %iv.rem185 = phi i32 [ 7, %preheader ], [ %iv.rem, %loop ]
 ...

loop_exit:
 %local_2_23.lcssa = phi i32 [ %iv.rem185, %loop ]
 %local_4_25.lcssa = phi i32 [ %iv.rem185, %loop ]

When LV is fixing recurrence users, only the first phi is handled.

[hsaito]

Given that there can be more than one users, the fix makes sense to me. LGTM.

lib/Transforms/Vectorize/LoopVectorize.cpp

void InnerLoopVectorizer::fixFirstOrderRecurrence(PHINode *Phi) {
   }
 
   Phi->setIncomingValue(Phi->getBasicBlockIndex(LoopScalarPreHeader), Start);
   Phi->setName("scalar.recur");
 
   // Finally, fix users of the recurrence outside the loop. The users will need
   // either the last value of the scalar recurrence or the last value of the
   // vector recurrence we extracted in the middle block. Since the loop is in
-  // LCSSA form, we just need to find the phi node for the original scalar
+  // LCSSA form, we just need to find all the phi nodes for the original scalar
   // recurrence in the exit block, and then add an edge for the middle block.
   for (PHINode &LCSSAPhi : LoopExitBlock->phis()) {
     if (LCSSAPhi.getIncomingValue(0) == Phi) {
       LCSSAPhi.addIncoming(ExtractForPhiUsedOutsideLoop, LoopMiddleBlock);
-      break;
     }
   }
 }
 
 void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
   Constant *Zero = Builder.getInt32(0);
 
   // Get it's reduction variable descriptor.

test/Transforms/LoopVectorize/pr36983.ll

+; RUN: opt < %s -loop-vectorize -S | FileCheck %s
+
+; There could be more than one LCSSA PHIs in loop exit block.
+
+; CHECK-LABEL: bb1.bb3_crit_edge:
+; CHECK: %_tmp133.lcssa1 = phi i16 [ %scalar.recur, %bb2 ], [ %vector.recur.extract.for.phi, %middle.block ]
+; CHECK: %_tmp133.lcssa = phi i16 [ %scalar.recur, %bb2 ], [ %vector.recur.extract.for.phi, %middle.block ]
+
+define void @f1() {
+bb2.lr.ph:
+  br label %bb2
+
+bb2:                                              ; preds = %bb2, %bb2.lr.ph
+  %_tmp132 = phi i16 [ 0, %bb2.lr.ph ], [ %_tmp10, %bb2 ]
+  %_tmp133 = phi i16 [ undef, %bb2.lr.ph ], [ %_tmp10, %bb2 ]
+  %_tmp10 = sub nsw i16 %_tmp132, 1
+  %_tmp15 = icmp ne i16 %_tmp10, 0
+  br i1 %_tmp15, label %bb2, label %bb1.bb3_crit_edge
+
+bb1.bb3_crit_edge:                                ; preds = %bb2
+  %_tmp133.lcssa1 = phi i16 [ %_tmp133, %bb2 ]
+  %_tmp133.lcssa = phi i16 [ %_tmp133, %bb2 ]
+  ret void
+}