[KnownBits] Improve `KnownBits::rem(X, Y)` in cases where we can deduce low-bits of output

[goldstein.w.n]

Summary

The first cttz(Y) bits in X are translated 1-1 in the output.

Alive2 Links:

https://alive2.llvm.org/ce/z/Qc47p7
https://alive2.llvm.org/ce/z/19ut5H

[foad]

Looks reasonable overall.

llvm/lib/Support/KnownBits.cpp

552

Missing word after "will"?

554

Use getLowBitsSet?

557

We already checked !LHS.hasConflict() in both callers so this seems unnecessary to me.

573

Just a matter of taste, but I'd prefer to write this as Known |= remGetLowBits(LHS, RHS);.

Also could move this above the RHS.isConstant case - then that case would only need to handle setting the high bits of Known.Zero.

llvm/test/Analysis/ScalarEvolution/merge-add-rec-many-inputs.ll

1 ↗

(On Diff #517805)

Looks like a spurious change?

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

573

Just a matter of taste, but I'd prefer to write this as Known |= remGetLowBits(LHS, RHS);.

I don't think this is right. KnowBits::| does Known.Zero &= remGetLowBits.Zero. We want the opposite. If some of the lowbits of X are zero in this case, we want to forcefully set those bits in Known.

Also could move this above the RHS.isConstant case - then that case would only need to handle setting the high bits of Known.Zero.

sure.

[goldstein.w.n]

Remove spurios change, move before const case, little cleanup

[foad]

llvm/lib/Support/KnownBits.cpp

573

I don't think this is right.

Good point. You'd want some kind of "merge" operation which doesn't exist because I can't think of a good name for it.

[foad]

llvm/lib/Support/KnownBits.cpp

563–566

New suggestion: simplify this to KnownBits Known = remGetLowBits(LHS, RHS);

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

563–566

Any objections to pass Known to remGetLowBits as const-ref? Don't want to force the assumption that remGetLowBits is called before any other bits are set.

[tschuett]

How about:

KnownBits remGetLowBits(const KnownBits &Known, const KnownBits &LHS,
                          const KnownBits &RHS);

Passing with &for mutability is always an odd pattern.

[goldstein.w.n]

In D149421#4306030, @tschuett wrote:

How about:

KnownBits remGetLowBits(const KnownBits &Known, const KnownBits &LHS,
                          const KnownBits &RHS);

Passing with &for mutability is always an odd pattern.

Agreed and done.

[goldstein.w.n]

Remove mutable reference

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

573

I don't think this is right.

Good point. You'd want some kind of "merge" operation which doesn't exist because I can't think of a good name for it.

How about "merge"? XD

[RKSimon]

This comment has been deleted.

[goldstein.w.n]

ping.

[RKSimon]

llvm/lib/Support/KnownBits.cpp

549

Do we need Known?

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Yeah. We don't have a "merge" function (to take known results from A and apply them to B).
There was a bit of conversation below (around L580) about it.

[RKSimon]

llvm/lib/Support/KnownBits.cpp

549

But do we need a merge? All the calls to remGetLowBits are at the start when Known is empty

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Oh yeah. My feeling there was the API should work on arbitrary state, not just init state. Otherwise just seems like a bug waiting to happen.

[nikic]

llvm/lib/Support/KnownBits.cpp

549

As this is a private API, I don't think there's a need to generalize it to a use case that currently doesn't exist...

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Private API to users sure, if in a year if someone add some improvement will it be clear? To me it seems like basically zero cost to passing Known and potential cost in not doing so. If you're not convinced, however, (either of you), ill change. LMK.

[foad]

llvm/lib/Support/KnownBits.cpp

549

I much prefer not passing in Known.

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Okay done :)

[goldstein.w.n]

Remove known as arg

[foad]

llvm/lib/Support/KnownBits.cpp

587–588

Remove this line too.

[nikic]

llvm/lib/Support/KnownBits.cpp

570

You can drop the KnownOut variable here and just use Known.

[RKSimon]

KnownBitsTest has an exhaustive test for srem and urem - so the likelihood of bugs surfacing are pretty low.

llvm/lib/Support/KnownBits.cpp

573

return KnownBits(ZerosMask, OnesMask)

575

return KnownBits(BitWidth)

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

587–588

Why? Isn't it a bit confusing that we only set zeros?

[goldstein.w.n]

In D149421#4318299, @RKSimon wrote:

KnownBitsTest has an exhaustive test for srem and urem - so the likelihood of bugs surfacing are pretty low.

I'm less worried about a correctness failure. Moreso undoing information we could have deduced with unknown state.

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

573

Sure, although this is private so need to put this function in the class then.

[goldstein.w.n]

Cleanup helper now that it returns init state

[RKSimon]

LGTM with a minor

llvm/lib/Support/KnownBits.cpp

567

// NB: Low one bits set in remGetLowBits

587

// NB: Low one bits set in remGetLowBits

[foad]

llvm/lib/Support/KnownBits.cpp

587–588

I'm saying you should remove Known.Zero = LHS.Zero & LowBits, because it has already been done by remGetLowBits. The only extra stuff we need to do in the constant RHS case is to do the high bits.

[goldstein.w.n]

Also use for low-zeros

[foad]

llvm/lib/Support/KnownBits.cpp

573

How about "merge"? XD

Great idea! But here's what I came up with: D150443

llvm/include/llvm/Support/KnownBits.h

public:
   bool operator==(const KnownBits &Other) const {
     return Zero == Other.Zero && One == Other.One;
   }
 
   bool operator!=(const KnownBits &Other) const { return !(*this == Other); }
 
   void print(raw_ostream &OS) const;
   void dump() const;
+
+private:
+  // Internal helper for getting the initial KnownBits for an `srem` or `urem`
+  // operation with the low-bits set.
+  static KnownBits remGetLowBits(const KnownBits &LHS, const KnownBits &RHS);
 };
 
 inline KnownBits operator&(KnownBits LHS, const KnownBits &RHS) {
   LHS &= RHS;
   return LHS;
 }
 
 inline KnownBits operator&(const KnownBits &LHS, KnownBits &&RHS) {

llvm/lib/Support/KnownBits.cpp

KnownBits KnownBits::udiv(const KnownBits &LHS, const KnownBits &RHS) {
 
   if (RHSMaxLeadingZeros != BitWidth)
     LeadZ = std::min(BitWidth, LeadZ + BitWidth - RHSMaxLeadingZeros - 1);
 
   Known.Zero.setHighBits(LeadZ);
   return Known;
 }
 
-KnownBits KnownBits::urem(const KnownBits &LHS, const KnownBits &RHS) {
+KnownBits KnownBits::remGetLowBits(const KnownBits &LHS, const KnownBits &RHS) {
   unsigned BitWidth = LHS.getBitWidth();
+  if (!RHS.isZero() && RHS.Zero[0]) {
+    // rem X, Y where Y[0:N] is zero will preserve X[0:N] in the result.
+    unsigned RHSZeros = RHS.countMinTrailingZeros();
+    APInt Mask = APInt::getLowBitsSet(BitWidth, RHSZeros);
+    APInt OnesMask = LHS.One & Mask;
+    APInt ZerosMask = LHS.Zero & Mask;
+    return KnownBits(ZerosMask, OnesMask);
+  }
+  return KnownBits(BitWidth);
+}
+
+KnownBits KnownBits::urem(const KnownBits &LHS, const KnownBits &RHS) {
   assert(!LHS.hasConflict() && !RHS.hasConflict());
-  KnownBits Known(BitWidth);
 
+  KnownBits Known = remGetLowBits(LHS, RHS);
   if (RHS.isConstant() && RHS.getConstant().isPowerOf2()) {
-    // The upper bits are all zero, the lower ones are unchanged.
-    APInt LowBits = RHS.getConstant() - 1;
-    Known.Zero = LHS.Zero | ~LowBits;
-    Known.One = LHS.One & LowBits;
+    // NB: Low bits set in `remGetLowBits`.
+    APInt HighBits = ~(RHS.getConstant() - 1);
+    Known.Zero |= HighBits;
     return Known;
   }
 
   // Since the result is less than or equal to either operand, any leading
   // zero bits in either operand must also exist in the result.
   uint32_t Leaders =
       std::max(LHS.countMinLeadingZeros(), RHS.countMinLeadingZeros());
   Known.Zero.setHighBits(Leaders);
   return Known;
 }
 
 KnownBits KnownBits::srem(const KnownBits &LHS, const KnownBits &RHS) {
-  unsigned BitWidth = LHS.getBitWidth();
   assert(!LHS.hasConflict() && !RHS.hasConflict());
-  KnownBits Known(BitWidth);
 
+  KnownBits Known = remGetLowBits(LHS, RHS);
   if (RHS.isConstant() && RHS.getConstant().isPowerOf2()) {
-    // The low bits of the first operand are unchanged by the srem.
+    // NB: Low bits are set in `remGetLowBits`.
     APInt LowBits = RHS.getConstant() - 1;
-    Known.Zero = LHS.Zero & LowBits;
-    Known.One = LHS.One & LowBits;
-
     // If the first operand is non-negative or has all low bits zero, then
     // the upper bits are all zero.
     if (LHS.isNonNegative() || LowBits.isSubsetOf(LHS.Zero))
       Known.Zero |= ~LowBits;
 
     // If the first operand is negative and not all low bits are zero, then
     // the upper bits are all one.
     if (LHS.isNegative() && LowBits.intersects(LHS.One))
       Known.One |= ~LowBits;

llvm/test/Analysis/ValueTracking/knownbits-rem-lowbits.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=instcombine -S | FileCheck %s
 
 define i8 @urem_low_bits_know(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @urem_low_bits_know(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 2
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = urem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 2
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %x = or i8 %xx, 2
   %y = and i8 %yy, -4
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @urem_low_bits_know2(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @urem_low_bits_know2(
-; CHECK-NEXT:    [[XO:%.*]] = and i8 [[XX:%.*]], -4
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XO]], 2
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = urem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 3
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %xo = or i8 %xx, 2
   %x = and i8 %xo, 254
   %y = and i8 %yy, -4
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 3
   ret i8 %r
 }
;
   %y = and i8 %yy, -5
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @srem_low_bits_know(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @srem_low_bits_know(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 10
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = srem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 2
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %x = or i8 %xx, 10
   %y = and i8 %yy, -4
   %rem = srem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @srem_low_bits_know2(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @srem_low_bits_know2(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 1
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -2
-; CHECK-NEXT:    [[REM:%.*]] = srem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 1
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 1
 ;
   %x = or i8 %xx, 1
   %y = and i8 %yy, -2
   %rem = srem i8 %x, %y
   %r = and i8 %rem, 1
   ret i8 %r
 }
 

llvm/test/Transforms/LoopVectorize/AArch64/masked-call.ll

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR2:[0-9]+]]

 ; TFNONE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds i64, ptr [[B:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[PREDPHI]], ptr [[TMP8]], align 8
 ; TFNONE-NEXT:    [[TMP9:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP10:%.*]] = mul i64 [[TMP9]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP10]]
 ; TFNONE-NEXT:    [[TMP11:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP11]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[IF_END:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; TFNONE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[TMP12:%.*]] = load i64, ptr [[ARRAYIDX]], align 8
 ; TFNONE-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP12]], 50

 ; TFNONE-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i64, ptr [[B:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[PREDPHI]], ptr [[TMP9]], align 8
 ; TFNONE-NEXT:    [[TMP10:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP11:%.*]] = mul i64 [[TMP10]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP11]]
 ; TFNONE-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[IF_END:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; TFNONE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[TMP13:%.*]] = load i64, ptr [[ARRAYIDX]], align 8
 ; TFNONE-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP13]], 50

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR4:[0-9]+]]

 ; TFFALLBACK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFFALLBACK-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFFALLBACK-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFFALLBACK-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFFALLBACK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFFALLBACK-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFFALLBACK-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; TFFALLBACK:       middle.block:
-; TFFALLBACK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFFALLBACK-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFFALLBACK-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFFALLBACK:       scalar.ph:
 ; TFFALLBACK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFFALLBACK-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFFALLBACK:       for.body:
 ; TFFALLBACK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFFALLBACK-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFFALLBACK-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFFALLBACK-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR6:[0-9]+]]

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR5:[0-9]+]]