[SCEV] Generalize MatchBinaryAddToConst to support non-add expressions.

[fhahn]

Summary

This patch generalizes MatchBinaryAddToConst to support matching
(A + C1), (A + C2), instead of just matching (A + C1), A.

The existing cases can be handled by treating non-add expressions A as
A + 0.

[mkazantsev]

I think this is wrong.

llvm/lib/Analysis/ScalarEvolution.cpp

10106

Is lack of nsw in (X + C2) intentional? If yes, the counter-example to this is

X = SINT_MIN
C1 = 1
C2 = -1

(X + C1)<nsw> = SINT_MIN + 1 <= (X - 1) = SINT_MAX, but obviously C1 > C2.

[mkazantsev]

llvm/lib/Analysis/ScalarEvolution.cpp

10106

I'm not sure whether two-ways implication is required here, or one way implication is enough. It was two-ways in old code. If I'm wrong here, please let me know.

[fhahn]

Add <nsw> and <nuw> to the sub-expressions where it was missing.

[fhahn]

llvm/lib/Analysis/ScalarEvolution.cpp

10106

The comment is missing <nsw> from the second expression, the actual checks enforce nsw for both (same for nuw below). I updated the comment.

I'm not sure whether two-ways implication is required here, or one way implication is enough. It was two-ways in old code. If I'm wrong here, please let me know.

The new reasoning should handle both previous checks I think , as, X s<= (X + C) treated as X + 0 s<= (X + C) and (X + C) s<= X is treated as (X + C) s<= X + 0

[mkazantsev]

Ok, thanks for clarification. With this clarification looks fine. Please give me some time to run it through my fuzz corps.

[reames]

For an optional follow up change, I believe we can avoid checking the wrap flags for one operand, if C1 and C2 share the same sign.

Consider:
(X + C1) s<= (X + C2)<nsw> if C1 s<= C2 and both are positive
(X + C1)<nsw> s<= (X + C2) if C1 s<= C2 and both are negative

If X + C2 does not signed wrap, and C1 s<= C2, then X+C1 also can not wrap if both are positive. As a result, we don't need to check for the wrap flag on LHS as we've proven it. The negative case is the inverse. The mixed sign case does require both checks as the two could overflow off different ends of the range. (Actually, can they? Is there any single value of X which you can add a value to and both wrap above and wrap below? Might be worth further thought.)

It's also worth noting that we can *disprove* cases here as well.

Consider:
// (X + C1)<nsw> s<= (X + C2)<nsw> == false if C1 s> C2.

This would require a change to the API (probably returning an optional bool), but might be interesting to short circuit later checks. Again, follow up change (if interested).

[mkazantsev]

Ok, looks good!

[lebedev.ri]

Looks good to me

[fhahn]

For an optional follow up change, I believe we can avoid checking the wrap flags for one operand, if C1 and C2 share the same sign.

Thanks, I'll try to look into this as a follow-up.

In D104634#2830649, @mkazantsev wrote:

Ok, thanks for clarification. With this clarification looks fine. Please give me some time to run it through my fuzz corps.

Thanks for the testing ahead of landing! I'll wait a day or so with landing, in case one of my earlier patches surfaces any issues.

[fhahn]

Remove test changes that depend on D102834.

llvm/lib/Analysis/ScalarEvolution.cpp

if (CmpInst::isSigned(Pred))
     return CheckRanges(getSignedRange(LHS), getSignedRange(RHS));
 
   return CheckRanges(getUnsignedRange(LHS), getUnsignedRange(RHS));
 }
 
 bool ScalarEvolution::isKnownPredicateViaNoOverflow(ICmpInst::Predicate Pred,
                                                     const SCEV *LHS,
                                                     const SCEV *RHS) {
-  // Match Result to (X + Y)<ExpectedFlags> where Y is a constant integer.
-  // Return Y via OutY.
-  auto MatchBinaryAddToConst =
-      [this](const SCEV *Result, const SCEV *X, APInt &OutY,
+  // Match X to (A + C1)<ExpectedFlags> and Y to (A + C2)<ExpectedFlags>, where
+  // C1 and C2 are constant integers. If either X or Y are not add expressions,
+  // consider them as X + 0 and Y + 0 respectively. C1 and C2 are returned via
+  // OutC1 and OutC2.
+  auto MatchBinaryAddToConst = [this](const SCEV *X, const SCEV *Y,
+                                       APInt &OutC1, APInt &OutC2,
                                        SCEV::NoWrapFlags ExpectedFlags) {
-    const SCEV *NonConstOp, *ConstOp;
-    SCEV::NoWrapFlags FlagsPresent;
+    const SCEV *XNonConstOp, *XConstOp;
+    const SCEV *YNonConstOp, *YConstOp;
+    SCEV::NoWrapFlags XFlagsPresent;
+    SCEV::NoWrapFlags YFlagsPresent;
 
-    if (!splitBinaryAdd(Result, ConstOp, NonConstOp, FlagsPresent) ||
-        !isa<SCEVConstant>(ConstOp) || NonConstOp != X)
+    if (!splitBinaryAdd(X, XConstOp, XNonConstOp, XFlagsPresent)) {
+      XConstOp = getZero(X->getType());
+      XNonConstOp = X;
+      XFlagsPresent = ExpectedFlags;
+    }
+    if (!isa<SCEVConstant>(XConstOp) ||
+        (XFlagsPresent & ExpectedFlags) != ExpectedFlags)
+      return false;
+
+    if (!splitBinaryAdd(Y, YConstOp, YNonConstOp, YFlagsPresent)) {
+      YConstOp = getZero(Y->getType());
+      YNonConstOp = Y;
+      YFlagsPresent = ExpectedFlags;
+    }
+
+    if (!isa<SCEVConstant>(YConstOp) ||
+        (YFlagsPresent & ExpectedFlags) != ExpectedFlags)
+      return false;
+
+    if (YNonConstOp != XNonConstOp)
       return false;
 
-    OutY = cast<SCEVConstant>(ConstOp)->getAPInt();
-    return (FlagsPresent & ExpectedFlags) == ExpectedFlags;
+    OutC1 = cast<SCEVConstant>(XConstOp)->getAPInt();
+    OutC2 = cast<SCEVConstant>(YConstOp)->getAPInt();
+
+    return true;
   };
 
-  APInt C;
+  APInt C1;
+  APInt C2;
 
   switch (Pred) {
   default:
     break;
 
   case ICmpInst::ICMP_SGE:
     std::swap(LHS, RHS);
     LLVM_FALLTHROUGH;
   case ICmpInst::ICMP_SLE:
-    // X s<= (X + C)<nsw> if C >= 0
-    if (MatchBinaryAddToConst(RHS, LHS, C, SCEV::FlagNSW) && C.isNonNegative())
+    // (X + C1)<nsw> s<= (X + C2) if C1 s<= C2.
+    if (MatchBinaryAddToConst(LHS, RHS, C1, C2, SCEV::FlagNSW) && C1.sle(C2))
       return true;
 
-    // (X + C)<nsw> s<= X if C <= 0
-    if (MatchBinaryAddToConst(LHS, RHS, C, SCEV::FlagNSW) &&
-        !C.isStrictlyPositive())
-      return true;
     break;
 
   case ICmpInst::ICMP_SGT:
     std::swap(LHS, RHS);
     LLVM_FALLTHROUGH;
   case ICmpInst::ICMP_SLT:
-    // X s< (X + C)<nsw> if C > 0
-    if (MatchBinaryAddToConst(RHS, LHS, C, SCEV::FlagNSW) &&
-        C.isStrictlyPositive())
+    // (X + C1) s< (X + C2)<nsw> if C1 s< C2.
+    if (MatchBinaryAddToConst(LHS, RHS, C1, C2, SCEV::FlagNSW) && C1.slt(C2))
       return true;
 
-    // (X + C)<nsw> s< X if C < 0
-    if (MatchBinaryAddToConst(LHS, RHS, C, SCEV::FlagNSW) && C.isNegative())
-      return true;
     break;
 
   case ICmpInst::ICMP_UGE:
     std::swap(LHS, RHS);
     LLVM_FALLTHROUGH;
   case ICmpInst::ICMP_ULE:
-    // X u<= (X + C)<nuw> for any C
-    if (MatchBinaryAddToConst(RHS, LHS, C, SCEV::FlagNUW))
+    // (X + C1) u<= (X + C2)<nuw> for C1 u<= C2.
+    if (MatchBinaryAddToConst(RHS, LHS, C2, C1, SCEV::FlagNUW) && C1.ule(C2))
       return true;
+
     break;
 
   case ICmpInst::ICMP_UGT:
     std::swap(LHS, RHS);
     LLVM_FALLTHROUGH;
   case ICmpInst::ICMP_ULT:
-    // X u< (X + C)<nuw> if C != 0
-    if (MatchBinaryAddToConst(RHS, LHS, C, SCEV::FlagNUW) && !C.isNullValue())
+    // (X + C1 u< (X + C2)<nuw> if C1 u< C2.
+    if (MatchBinaryAddToConst(RHS, LHS, C2, C1, SCEV::FlagNUW) && C1.ult(C2))
       return true;
     break;
   }
 
   return false;
 }
 
 bool ScalarEvolution::isKnownPredicateViaSplitting(ICmpInst::Predicate Pred,

llvm/test/Transforms/SLPVectorizer/AArch64/memory-runtime-checks.ll

entry:
   ret void
 }
 
 define void @needs_versioning_profitable(i32* %dst, i32* %src) {
 ; CHECK-LABEL: @needs_versioning_profitable(
 ; CHECK-NEXT:  entry.slpmemcheck:
 ; CHECK-NEXT:    [[DST16:%.*]] = bitcast i32* [[DST:%.*]] to i8*
 ; CHECK-NEXT:    [[SRC18:%.*]] = bitcast i32* [[SRC:%.*]] to i8*
-; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[DST]], i64 2
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[DST]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP17:%.*]] = bitcast i32* [[SCEVGEP]] to i8*
 ; CHECK-NEXT:    [[SCEVGEP19:%.*]] = getelementptr i32, i32* [[SRC]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP1920:%.*]] = bitcast i32* [[SCEVGEP19]] to i8*
 ; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[DST16]], [[SCEVGEP1920]]
 ; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[SRC18]], [[SCEVGEP17]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
+; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[ENTRY_SCALAR:%.*]], label [[ENTRY_SLPVERSIONED:%.*]]
+; CHECK:       entry.slpversioned:
+; CHECK-NEXT:    [[SRC_GEP_1:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 1
+; CHECK-NEXT:    [[DST_GEP_1:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 1
+; CHECK-NEXT:    [[SRC_GEP_2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
+; CHECK-NEXT:    [[DST_GEP_2:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
+; CHECK-NEXT:    [[SRC_GEP_3:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4, !alias.scope !0, !noalias !3
+; CHECK-NEXT:    [[TMP2:%.*]] = ashr <4 x i32> [[TMP1]], <i32 16, i32 16, i32 16, i32 16>
+; CHECK-NEXT:    [[DST_GEP_3:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    br label [[ENTRY_MERGE:%.*]]
+; CHECK:       entry.merge:
+; CHECK-NEXT:    ret void
+; CHECK:       entry.scalar:
 ; CHECK-NEXT:    [[SRC_02:%.*]] = load i32, i32* [[SRC]], align 4
 ; CHECK-NEXT:    [[R_03:%.*]] = ashr i32 [[SRC_02]], 16
 ; CHECK-NEXT:    store i32 [[R_03]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[SRC_GEP_14:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 1
 ; CHECK-NEXT:    [[SRC_15:%.*]] = load i32, i32* [[SRC_GEP_14]], align 4
 ; CHECK-NEXT:    [[R_16:%.*]] = ashr i32 [[SRC_15]], 16
 ; CHECK-NEXT:    [[DST_GEP_17:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 1
 ; CHECK-NEXT:    store i32 [[R_16]], i32* [[DST_GEP_17]], align 4
 ; CHECK-NEXT:    [[SRC_GEP_28:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
 ; CHECK-NEXT:    [[SRC_29:%.*]] = load i32, i32* [[SRC_GEP_28]], align 4
 ; CHECK-NEXT:    [[R_210:%.*]] = ashr i32 [[SRC_29]], 16
 ; CHECK-NEXT:    [[DST_GEP_211:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
 ; CHECK-NEXT:    store i32 [[R_210]], i32* [[DST_GEP_211]], align 4
 ; CHECK-NEXT:    [[SRC_GEP_312:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
 ; CHECK-NEXT:    [[SRC_313:%.*]] = load i32, i32* [[SRC_GEP_312]], align 4
 ; CHECK-NEXT:    [[R_314:%.*]] = ashr i32 [[SRC_313]], 16
 ; CHECK-NEXT:    [[DST_GEP_315:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
 ; CHECK-NEXT:    store i32 [[R_314]], i32* [[DST_GEP_315]], align 4
-; CHECK-NEXT:    ret void
+; CHECK-NEXT:    br label [[ENTRY_MERGE]]
 ;
 entry:
   %src.0 = load i32, i32* %src, align 4
   %r.0 = ashr i32 %src.0, 16
   store i32 %r.0, i32* %dst, align 4
   %src.gep.1 = getelementptr inbounds i32, i32* %src, i64 1
   %src.1 = load i32, i32* %src.gep.1, align 4
   %r.1 = ashr i32 %src.1, 16
entry:
   ret void
 }
 
 define void @version_multiple(i32* nocapture %out_block, i32* nocapture readonly %counter) {
 ; CHECK-LABEL: @version_multiple(
 ; CHECK-NEXT:  entry.slpmemcheck:
 ; CHECK-NEXT:    [[OUT_BLOCK12:%.*]] = bitcast i32* [[OUT_BLOCK:%.*]] to i8*
 ; CHECK-NEXT:    [[COUNTER14:%.*]] = bitcast i32* [[COUNTER:%.*]] to i8*
-; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[OUT_BLOCK]], i64 2
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[OUT_BLOCK]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP13:%.*]] = bitcast i32* [[SCEVGEP]] to i8*
 ; CHECK-NEXT:    [[SCEVGEP15:%.*]] = getelementptr i32, i32* [[COUNTER]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP1516:%.*]] = bitcast i32* [[SCEVGEP15]] to i8*
 ; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[OUT_BLOCK12]], [[SCEVGEP1516]]
 ; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[COUNTER14]], [[SCEVGEP13]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
 ; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[ENTRY_SCALAR:%.*]], label [[ENTRY_SLPVERSIONED:%.*]]
 ; CHECK:       entry.slpversioned:
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[COUNTER]], align 4, !alias.scope !0, !noalias !3
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[OUT_BLOCK]], align 4, !alias.scope !3, !noalias !0
-; CHECK-NEXT:    [[XOR:%.*]] = xor i32 [[TMP1]], [[TMP0]]
-; CHECK-NEXT:    store i32 [[XOR]], i32* [[OUT_BLOCK]], align 4, !alias.scope !3, !noalias !0
 ; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[ARRAYIDX_1]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_1:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[ARRAYIDX2_1]], align 4
-; CHECK-NEXT:    [[XOR_1:%.*]] = xor i32 [[TMP3]], [[TMP2]]
-; CHECK-NEXT:    store i32 [[XOR_1]], i32* [[ARRAYIDX2_1]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 2
 ; CHECK-NEXT:    [[ARRAYIDX2_2:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 2
 ; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 3
-; CHECK-NEXT:    [[TMP4:%.*]] = bitcast i32* [[ARRAYIDX_2]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP5:%.*]] = load <2 x i32>, <2 x i32>* [[TMP4]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[COUNTER]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4, !alias.scope !5, !noalias !8
 ; CHECK-NEXT:    [[ARRAYIDX2_3:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 3
-; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i32* [[ARRAYIDX2_2]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP7:%.*]] = load <2 x i32>, <2 x i32>* [[TMP6]], align 4
-; CHECK-NEXT:    [[TMP8:%.*]] = xor <2 x i32> [[TMP7]], [[TMP5]]
-; CHECK-NEXT:    [[TMP9:%.*]] = bitcast i32* [[ARRAYIDX2_2]] to <2 x i32>*
-; CHECK-NEXT:    store <2 x i32> [[TMP8]], <2 x i32>* [[TMP9]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i32* [[OUT_BLOCK]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP2]], align 4, !alias.scope !8, !noalias !5
+; CHECK-NEXT:    [[TMP4:%.*]] = xor <4 x i32> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i32* [[OUT_BLOCK]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], <4 x i32>* [[TMP5]], align 4, !alias.scope !8, !noalias !5
 ; CHECK-NEXT:    br label [[ENTRY_MERGE:%.*]]
 ; CHECK:       entry.merge:
 ; CHECK-NEXT:    ret void
 ; CHECK:       entry.scalar:
-; CHECK-NEXT:    [[TMP10:%.*]] = load i32, i32* [[COUNTER]], align 4
-; CHECK-NEXT:    [[TMP11:%.*]] = load i32, i32* [[OUT_BLOCK]], align 4
-; CHECK-NEXT:    [[XOR2:%.*]] = xor i32 [[TMP11]], [[TMP10]]
+; CHECK-NEXT:    [[TMP6:%.*]] = load i32, i32* [[COUNTER]], align 4
+; CHECK-NEXT:    [[TMP7:%.*]] = load i32, i32* [[OUT_BLOCK]], align 4
+; CHECK-NEXT:    [[XOR2:%.*]] = xor i32 [[TMP7]], [[TMP6]]
 ; CHECK-NEXT:    store i32 [[XOR2]], i32* [[OUT_BLOCK]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_13:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 1
-; CHECK-NEXT:    [[TMP12:%.*]] = load i32, i32* [[ARRAYIDX_13]], align 4
+; CHECK-NEXT:    [[TMP8:%.*]] = load i32, i32* [[ARRAYIDX_13]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_14:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 1
-; CHECK-NEXT:    [[TMP13:%.*]] = load i32, i32* [[ARRAYIDX2_14]], align 4
-; CHECK-NEXT:    [[XOR_15:%.*]] = xor i32 [[TMP13]], [[TMP12]]
+; CHECK-NEXT:    [[TMP9:%.*]] = load i32, i32* [[ARRAYIDX2_14]], align 4
+; CHECK-NEXT:    [[XOR_15:%.*]] = xor i32 [[TMP9]], [[TMP8]]
 ; CHECK-NEXT:    store i32 [[XOR_15]], i32* [[ARRAYIDX2_14]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_26:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 2
-; CHECK-NEXT:    [[TMP14:%.*]] = load i32, i32* [[ARRAYIDX_26]], align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = load i32, i32* [[ARRAYIDX_26]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_27:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 2
-; CHECK-NEXT:    [[TMP15:%.*]] = load i32, i32* [[ARRAYIDX2_27]], align 4
-; CHECK-NEXT:    [[XOR_28:%.*]] = xor i32 [[TMP15]], [[TMP14]]
+; CHECK-NEXT:    [[TMP11:%.*]] = load i32, i32* [[ARRAYIDX2_27]], align 4
+; CHECK-NEXT:    [[XOR_28:%.*]] = xor i32 [[TMP11]], [[TMP10]]
 ; CHECK-NEXT:    store i32 [[XOR_28]], i32* [[ARRAYIDX2_27]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_39:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 3
-; CHECK-NEXT:    [[TMP16:%.*]] = load i32, i32* [[ARRAYIDX_39]], align 4
+; CHECK-NEXT:    [[TMP12:%.*]] = load i32, i32* [[ARRAYIDX_39]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_310:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 3
-; CHECK-NEXT:    [[TMP17:%.*]] = load i32, i32* [[ARRAYIDX2_310]], align 4
-; CHECK-NEXT:    [[XOR_311:%.*]] = xor i32 [[TMP17]], [[TMP16]]
+; CHECK-NEXT:    [[TMP13:%.*]] = load i32, i32* [[ARRAYIDX2_310]], align 4
+; CHECK-NEXT:    [[XOR_311:%.*]] = xor i32 [[TMP13]], [[TMP12]]
 ; CHECK-NEXT:    store i32 [[XOR_311]], i32* [[ARRAYIDX2_310]], align 4
 ; CHECK-NEXT:    br label [[ENTRY_MERGE]]
 ;
 entry:
   %0 = load i32, i32* %counter, align 4
   %1 = load i32, i32* %out_block, align 4
   %xor = xor i32 %1, %0
   store i32 %xor, i32* %out_block, align 4

 
 declare void @clobber()
 
 define void @slp_not_beneficial(i32* %A, i32* %B) {
 ; CHECK-LABEL: @slp_not_beneficial(
 ; CHECK-NEXT:  bb.slpmemcheck:
 ; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[A:%.*]], i64 4
 ; CHECK-NEXT:    [[SCEVGEP6:%.*]] = bitcast i32* [[SCEVGEP]] to i8*
-; CHECK-NEXT:    [[SCEVGEP7:%.*]] = getelementptr i32, i32* [[B:%.*]], i64 4
+; CHECK-NEXT:    [[SCEVGEP7:%.*]] = getelementptr i32, i32* [[A]], i64 5
 ; CHECK-NEXT:    [[SCEVGEP78:%.*]] = bitcast i32* [[SCEVGEP7]] to i8*
-; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[SCEVGEP6]], [[SCEVGEP78]]
-; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[SCEVGEP78]], [[SCEVGEP6]]
+; CHECK-NEXT:    [[SCEVGEP9:%.*]] = getelementptr i32, i32* [[B:%.*]], i64 4
+; CHECK-NEXT:    [[SCEVGEP910:%.*]] = bitcast i32* [[SCEVGEP9]] to i8*
+; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[SCEVGEP6]], [[SCEVGEP910]]
+; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[SCEVGEP910]], [[SCEVGEP78]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 4
 ; CHECK-NEXT:    store i32 0, i32* [[TMP2]], align 8
 ; CHECK-NEXT:    [[TMP33:%.*]] = getelementptr inbounds i32, i32* [[A]], i32 5
 ; CHECK-NEXT:    [[TMP44:%.*]] = getelementptr inbounds i32, i32* [[B]], i32 4
 ; CHECK-NEXT:    [[TMP55:%.*]] = load i32, i32* [[TMP44]], align 8
 ; CHECK-NEXT:    store i32 [[TMP55]], i32* [[TMP33]], align 8
 ; CHECK-NEXT:    ret void

llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -scoped-noalias-aa -slp-vectorizer -mtriple=x86_64-apple-darwin -enable-new-pm=false -S %s | FileCheck %s
 
 define void @version_multiple(i32* nocapture %out_block, i32* nocapture readonly %counter) {
 ; CHECK-LABEL: @version_multiple(
 ; CHECK-NEXT:  entry.slpmemcheck:
 ; CHECK-NEXT:    [[OUT_BLOCK12:%.*]] = bitcast i32* [[OUT_BLOCK:%.*]] to i8*
 ; CHECK-NEXT:    [[COUNTER14:%.*]] = bitcast i32* [[COUNTER:%.*]] to i8*
-; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[OUT_BLOCK]], i64 1
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[OUT_BLOCK]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP13:%.*]] = bitcast i32* [[SCEVGEP]] to i8*
-; CHECK-NEXT:    [[SCEVGEP15:%.*]] = getelementptr i32, i32* [[COUNTER]], i64 1
+; CHECK-NEXT:    [[SCEVGEP15:%.*]] = getelementptr i32, i32* [[COUNTER]], i64 3
 ; CHECK-NEXT:    [[SCEVGEP1516:%.*]] = bitcast i32* [[SCEVGEP15]] to i8*
 ; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[OUT_BLOCK12]], [[SCEVGEP1516]]
 ; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[COUNTER14]], [[SCEVGEP13]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[COUNTER]], align 4
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[OUT_BLOCK]], align 4
-; CHECK-NEXT:    [[XOR2:%.*]] = xor i32 [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[ENTRY_SCALAR:%.*]], label [[ENTRY_SLPVERSIONED:%.*]]
+; CHECK:       entry.slpversioned:
+; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX2_1:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 2
+; CHECK-NEXT:    [[ARRAYIDX2_2:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 2
+; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 3
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[COUNTER]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4, !alias.scope !0, !noalias !3
+; CHECK-NEXT:    [[ARRAYIDX2_3:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 3
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i32* [[OUT_BLOCK]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP2]], align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    [[TMP4:%.*]] = xor <4 x i32> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i32* [[OUT_BLOCK]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], <4 x i32>* [[TMP5]], align 4, !alias.scope !3, !noalias !0
+; CHECK-NEXT:    br label [[ENTRY_MERGE:%.*]]
+; CHECK:       entry.merge:
+; CHECK-NEXT:    ret void
+; CHECK:       entry.scalar:
+; CHECK-NEXT:    [[TMP6:%.*]] = load i32, i32* [[COUNTER]], align 4
+; CHECK-NEXT:    [[TMP7:%.*]] = load i32, i32* [[OUT_BLOCK]], align 4
+; CHECK-NEXT:    [[XOR2:%.*]] = xor i32 [[TMP7]], [[TMP6]]
 ; CHECK-NEXT:    store i32 [[XOR2]], i32* [[OUT_BLOCK]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_13:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[ARRAYIDX_13]], align 4
+; CHECK-NEXT:    [[TMP8:%.*]] = load i32, i32* [[ARRAYIDX_13]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_14:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[ARRAYIDX2_14]], align 4
-; CHECK-NEXT:    [[XOR_15:%.*]] = xor i32 [[TMP3]], [[TMP2]]
+; CHECK-NEXT:    [[TMP9:%.*]] = load i32, i32* [[ARRAYIDX2_14]], align 4
+; CHECK-NEXT:    [[XOR_15:%.*]] = xor i32 [[TMP9]], [[TMP8]]
 ; CHECK-NEXT:    store i32 [[XOR_15]], i32* [[ARRAYIDX2_14]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_26:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 2
-; CHECK-NEXT:    [[TMP4:%.*]] = load i32, i32* [[ARRAYIDX_26]], align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = load i32, i32* [[ARRAYIDX_26]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_27:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 2
-; CHECK-NEXT:    [[TMP5:%.*]] = load i32, i32* [[ARRAYIDX2_27]], align 4
-; CHECK-NEXT:    [[XOR_28:%.*]] = xor i32 [[TMP5]], [[TMP4]]
+; CHECK-NEXT:    [[TMP11:%.*]] = load i32, i32* [[ARRAYIDX2_27]], align 4
+; CHECK-NEXT:    [[XOR_28:%.*]] = xor i32 [[TMP11]], [[TMP10]]
 ; CHECK-NEXT:    store i32 [[XOR_28]], i32* [[ARRAYIDX2_27]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX_39:%.*]] = getelementptr inbounds i32, i32* [[COUNTER]], i64 3
-; CHECK-NEXT:    [[TMP6:%.*]] = load i32, i32* [[ARRAYIDX_39]], align 4
+; CHECK-NEXT:    [[TMP12:%.*]] = load i32, i32* [[ARRAYIDX_39]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2_310:%.*]] = getelementptr inbounds i32, i32* [[OUT_BLOCK]], i64 3
-; CHECK-NEXT:    [[TMP7:%.*]] = load i32, i32* [[ARRAYIDX2_310]], align 4
-; CHECK-NEXT:    [[XOR_311:%.*]] = xor i32 [[TMP7]], [[TMP6]]
+; CHECK-NEXT:    [[TMP13:%.*]] = load i32, i32* [[ARRAYIDX2_310]], align 4
+; CHECK-NEXT:    [[XOR_311:%.*]] = xor i32 [[TMP13]], [[TMP12]]
 ; CHECK-NEXT:    store i32 [[XOR_311]], i32* [[ARRAYIDX2_310]], align 4
-; CHECK-NEXT:    ret void
+; CHECK-NEXT:    br label [[ENTRY_MERGE]]
 ;
 entry:
   %0 = load i32, i32* %counter, align 4
   %1 = load i32, i32* %out_block, align 4
   %xor = xor i32 %1, %0
   store i32 %xor, i32* %out_block, align 4
   %arrayidx.1 = getelementptr inbounds i32, i32* %counter, i64 1
   %2 = load i32, i32* %arrayidx.1, align 4

 ; CHECK:       else:
 ; CHECK-NEXT:    [[TMP3:%.*]] = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> [[TMP2]], i32 4, <2 x i1> <i1 true, i1 true>, <2 x float> undef)
 ; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x float> [[TMP3]], <2 x float> poison, <4 x i32> <i32 0, i32 0, i32 1, i32 1>
 ; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[SHUFFLE]], <4 x float> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 undef, i32 undef, i32 3, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[I71:%.*]] = shufflevector <8 x float> undef, <8 x float> [[TMP4]], <8 x i32> <i32 0, i32 1, i32 8, i32 9, i32 10, i32 5, i32 6, i32 13>
 ; CHECK-NEXT:    call void @use(<8 x float> [[I71]])
 ; CHECK-NEXT:    ret void
 ; CHECK:       then.slpmemcheck:
-; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr float, float* [[A:%.*]], i64 8
+; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr float, float* [[A:%.*]], i64 5
 ; CHECK-NEXT:    [[SCEVGEP8:%.*]] = bitcast float* [[SCEVGEP]] to i8*
-; CHECK-NEXT:    [[SCEVGEP9:%.*]] = getelementptr float, float* [[B]], i64 14
+; CHECK-NEXT:    [[SCEVGEP9:%.*]] = getelementptr float, float* [[A]], i64 8
 ; CHECK-NEXT:    [[SCEVGEP910:%.*]] = bitcast float* [[SCEVGEP9]] to i8*
-; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[SCEVGEP8]], [[SCEVGEP910]]
-; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[SCEVGEP910]], [[SCEVGEP8]]
+; CHECK-NEXT:    [[SCEVGEP11:%.*]] = getelementptr float, float* [[B]], i64 14
+; CHECK-NEXT:    [[SCEVGEP1112:%.*]] = bitcast float* [[SCEVGEP11]] to i8*
+; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ult i8* [[SCEVGEP8]], [[SCEVGEP1112]]
+; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ult i8* [[SCEVGEP1112]], [[SCEVGEP910]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
 ; CHECK-NEXT:    [[A_83:%.*]] = getelementptr inbounds float, float* [[A]], i64 8
 ; CHECK-NEXT:    store float 0.000000e+00, float* [[A_83]], align 4
 ; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <2 x float*> [[TMP2]], i32 1
 ; CHECK-NEXT:    [[L64:%.*]] = load float, float* [[TMP5]], align 4
 ; CHECK-NEXT:    [[A_55:%.*]] = getelementptr inbounds float, float* [[A]], i64 5
 ; CHECK-NEXT:    store float [[L64]], float* [[A_55]], align 4
 ; CHECK-NEXT:    [[A_66:%.*]] = getelementptr inbounds float, float* [[A]], i64 6