[Analysis][LoopVectorize] do not try to form reductions of pointers

[spatel]

Summary

This is a fix for https://llvm.org/PR49215 either before/after we make a verifier enhancement for vector reductions with D96904.

I'm not sure what the current thinking is for pointer math/logic in IR. We allow icmp on pointer values. Therefore, we match min/max patterns, so without this patch, the vectorizer could form a vector reduction from that sequence.

But the LangRef definitions for min/max and vector reduction intrinsics do not allow pointer types:
https://llvm.org/docs/LangRef.html#llvm-smax-intrinsic
https://llvm.org/docs/LangRef.html#llvm-vector-reduce-umax-intrinsic

So we would crash/assert at some point - either in IR verification, in the cost model, or in codegen. If we do want to allow this kind of transform, we will need to update the LangRef and all of those parts of the compiler.

[nikic]

LGTM. While we may want to support pointer min/max in the future (probably not worth it though), we clearly don't now, so bailing out is the right thing to do.

Just one suggestion on placement of the check.

llvm/lib/Analysis/IVDescriptors.cpp

246

Personally, I'd add an else if (RecurrenceType->isIntegerTy()) here and then else return false. This seems like the place that is making the implicit assumption that the type can only be float or int.

llvm/lib/Analysis/IVDescriptors.cpp

bool RecurrenceDescriptor::AddReductionVar(PHINode *Phi, RecurKind Kind,
   // Return early if the recurrence kind does not match the type of Phi. If the
   // recurrence kind is arithmetic, we attempt to look through AND operations
   // resulting from the type promotion performed by InstCombine.  Vector
   // operations are not limited to the legal integer widths, so we may be able
   // to evaluate the reduction in the narrower width.
   if (RecurrenceType->isFloatingPointTy()) {
     if (!isFloatingPointRecurrenceKind(Kind))
       return false;
-  } else {
+  } else if (RecurrenceType->isIntegerTy()) {
     if (!isIntegerRecurrenceKind(Kind))
       return false;
     if (isArithmeticRecurrenceKind(Kind))
       Start = lookThroughAnd(Phi, RecurrenceType, VisitedInsts, CastInsts);
+  } else {
+    // Pointer min/max may exist, but it is not supported as a reduction op.
+    return false;
   }
 
   Worklist.push_back(Start);
   VisitedInsts.insert(Start);
 
   // Start with all flags set because we will intersect this with the reduction
   // flags from all the reduction operations.
   FastMathFlags FMF = FastMathFlags::getFast();

llvm/test/Transforms/LoopVectorize/reduction-ptr.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s  -loop-vectorize -force-vector-width=4 -S | FileCheck %s
+
+target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+
+; Reductions of pointer types are not supported.
+
+define void @PR49215(i32* %p, i32* %q) {
+; CHECK-LABEL: @PR49215(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[G:%.*]] = phi i32* [ [[P:%.*]], [[ENTRY]] ], [ [[UMIN:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[CMP2:%.*]] = icmp ult i32* [[Q:%.*]], [[G]]
+; CHECK-NEXT:    [[UMIN]] = select i1 [[CMP2]], i32* [[Q]], i32* [[G]]
+; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1
+; CHECK-NEXT:    [[EXITCOND:%.*]] = icmp eq i64 [[IV_NEXT]], undef
+; CHECK-NEXT:    br i1 [[EXITCOND]], label [[LOOPEXIT:%.*]], label [[FOR_BODY]]
+; CHECK:       loopexit:
+; CHECK-NEXT:    [[UMIN_LCSSA:%.*]] = phi i32* [ [[UMIN]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[PHI_CAST:%.*]] = ptrtoint i32* [[UMIN_LCSSA]] to i64
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:
+  %iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
+  %g = phi i32* [ %p, %entry ], [ %umin, %for.body ]
+  %cmp2 = icmp ult i32* %q, %g
+  %umin = select i1 %cmp2, i32* %q, i32* %g
+  %iv.next = add nuw nsw i64 %iv, 1
+  %exitcond = icmp eq i64 %iv.next, undef
+  br i1 %exitcond, label %loopexit, label %for.body
+
+loopexit:
+  %phi.cast = ptrtoint i32* %umin to i64
+  ret void
+}