[KnownBits] Improve `KnownBits::rem(X, Y)` in cases where we can deduce low-bits of output

[goldstein.w.n]

Summary

The first cttz(Y) bits in X are translated 1-1 in the output.

Alive2 Links:

https://alive2.llvm.org/ce/z/Qc47p7
https://alive2.llvm.org/ce/z/19ut5H

[foad]

Looks reasonable overall.

llvm/lib/Support/KnownBits.cpp

552

Missing word after "will"?

554

Use getLowBitsSet?

557

We already checked !LHS.hasConflict() in both callers so this seems unnecessary to me.

576

Just a matter of taste, but I'd prefer to write this as Known |= remGetLowBits(LHS, RHS);.

Also could move this above the RHS.isConstant case - then that case would only need to handle setting the high bits of Known.Zero.

llvm/test/Analysis/ScalarEvolution/merge-add-rec-many-inputs.ll

1

Looks like a spurious change?

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

576

Just a matter of taste, but I'd prefer to write this as Known |= remGetLowBits(LHS, RHS);.

I don't think this is right. KnowBits::| does Known.Zero &= remGetLowBits.Zero. We want the opposite. If some of the lowbits of X are zero in this case, we want to forcefully set those bits in Known.

Also could move this above the RHS.isConstant case - then that case would only need to handle setting the high bits of Known.Zero.

sure.

[goldstein.w.n]

Remove spurios change, move before const case, little cleanup

[foad]

llvm/lib/Support/KnownBits.cpp

576

I don't think this is right.

Good point. You'd want some kind of "merge" operation which doesn't exist because I can't think of a good name for it.

[foad]

llvm/lib/Support/KnownBits.cpp

566–568

New suggestion: simplify this to KnownBits Known = remGetLowBits(LHS, RHS);

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

566–568

Any objections to pass Known to remGetLowBits as const-ref? Don't want to force the assumption that remGetLowBits is called before any other bits are set.

[tschuett]

How about:

KnownBits remGetLowBits(const KnownBits &Known, const KnownBits &LHS,
                          const KnownBits &RHS);

Passing with &for mutability is always an odd pattern.

[goldstein.w.n]

In D149421#4306030, @tschuett wrote:

How about:

KnownBits remGetLowBits(const KnownBits &Known, const KnownBits &LHS,
                          const KnownBits &RHS);

Passing with &for mutability is always an odd pattern.

Agreed and done.

[goldstein.w.n]

Remove mutable reference

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

576

I don't think this is right.

Good point. You'd want some kind of "merge" operation which doesn't exist because I can't think of a good name for it.

How about "merge"? XD

[RKSimon]

This comment has been deleted.

[goldstein.w.n]

ping.

[RKSimon]

llvm/lib/Support/KnownBits.cpp

549

Do we need Known?

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Yeah. We don't have a "merge" function (to take known results from A and apply them to B).
There was a bit of conversation below (around L580) about it.

[RKSimon]

llvm/lib/Support/KnownBits.cpp

549

But do we need a merge? All the calls to remGetLowBits are at the start when Known is empty

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Oh yeah. My feeling there was the API should work on arbitrary state, not just init state. Otherwise just seems like a bug waiting to happen.

[nikic]

llvm/lib/Support/KnownBits.cpp

549

As this is a private API, I don't think there's a need to generalize it to a use case that currently doesn't exist...

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Private API to users sure, if in a year if someone add some improvement will it be clear? To me it seems like basically zero cost to passing Known and potential cost in not doing so. If you're not convinced, however, (either of you), ill change. LMK.

[foad]

llvm/lib/Support/KnownBits.cpp

549

I much prefer not passing in Known.

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

549

Okay done :)

[goldstein.w.n]

Remove known as arg

[foad]

llvm/lib/Support/KnownBits.cpp

594–595

Remove this line too.

[nikic]

llvm/lib/Support/KnownBits.cpp

573

You can drop the KnownOut variable here and just use Known.

[RKSimon]

KnownBitsTest has an exhaustive test for srem and urem - so the likelihood of bugs surfacing are pretty low.

llvm/lib/Support/KnownBits.cpp

576

return KnownBits(ZerosMask, OnesMask)

578

return KnownBits(BitWidth)

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

594–595

Why? Isn't it a bit confusing that we only set zeros?

[goldstein.w.n]

In D149421#4318299, @RKSimon wrote:

KnownBitsTest has an exhaustive test for srem and urem - so the likelihood of bugs surfacing are pretty low.

I'm less worried about a correctness failure. Moreso undoing information we could have deduced with unknown state.

[goldstein.w.n]

llvm/lib/Support/KnownBits.cpp

576

Sure, although this is private so need to put this function in the class then.

[goldstein.w.n]

Cleanup helper now that it returns init state

[RKSimon]

LGTM with a minor

llvm/lib/Support/KnownBits.cpp

570

// NB: Low one bits set in remGetLowBits

593

// NB: Low one bits set in remGetLowBits

[foad]

llvm/lib/Support/KnownBits.cpp

594–595

I'm saying you should remove Known.Zero = LHS.Zero & LowBits, because it has already been done by remGetLowBits. The only extra stuff we need to do in the constant RHS case is to do the high bits.

[goldstein.w.n]

Also use for low-zeros

[foad]

llvm/lib/Support/KnownBits.cpp

576

How about "merge"? XD

Great idea! But here's what I came up with: D150443

llvm/lib/Support/KnownBits.cpp

KnownBits KnownBits::udiv(const KnownBits &LHS, const KnownBits &RHS) {
 
   if (RHSMaxLeadingZeros != BitWidth)
     LeadZ = std::min(BitWidth, LeadZ + BitWidth - RHSMaxLeadingZeros - 1);
 
   Known.Zero.setHighBits(LeadZ);
   return Known;
 }
 
+static void remGetLowBits(KnownBits &Known, const KnownBits &LHS,
+                          const KnownBits &RHS) {
+  if (!RHS.isZero() && RHS.Zero[0]) {
+    // rem X, Y where Y[0:N] is zero will X[0:N] in the result.
+    unsigned RHSZeros = RHS.countMinTrailingZeros();
+    APInt Mask = APInt::getBitsSet(LHS.getBitWidth(), 0, RHSZeros);
+    APInt OnesMask = LHS.One & Mask;
+    APInt ZerosMask = LHS.Zero & Mask;
+    assert((OnesMask & ZerosMask).isZero());
+    Known.One |= OnesMask;
+    Known.Zero |= ZerosMask;
+  }
+}
+
 KnownBits KnownBits::urem(const KnownBits &LHS, const KnownBits &RHS) {
   unsigned BitWidth = LHS.getBitWidth();
   assert(!LHS.hasConflict() && !RHS.hasConflict());
   KnownBits Known(BitWidth);
 
   if (RHS.isConstant() && RHS.getConstant().isPowerOf2()) {
     // The upper bits are all zero, the lower ones are unchanged.
     APInt LowBits = RHS.getConstant() - 1;
     Known.Zero = LHS.Zero | ~LowBits;
     Known.One = LHS.One & LowBits;
     return Known;
   }
 
+  remGetLowBits(Known, LHS, RHS);
+
   // Since the result is less than or equal to either operand, any leading
   // zero bits in either operand must also exist in the result.
   uint32_t Leaders =
       std::max(LHS.countMinLeadingZeros(), RHS.countMinLeadingZeros());
   Known.Zero.setHighBits(Leaders);
   return Known;
 }
 
 KnownBits KnownBits::srem(const KnownBits &LHS, const KnownBits &RHS) {
   unsigned BitWidth = LHS.getBitWidth();
   assert(!LHS.hasConflict() && !RHS.hasConflict());
   KnownBits Known(BitWidth);
 
   if (RHS.isConstant() && RHS.getConstant().isPowerOf2()) {
     // The low bits of the first operand are unchanged by the srem.
     APInt LowBits = RHS.getConstant() - 1;
     Known.Zero = LHS.Zero & LowBits;
     Known.One = LHS.One & LowBits;
 
     // If the first operand is non-negative or has all low bits zero, then
     // the upper bits are all zero.
     if (LHS.isNonNegative() || LowBits.isSubsetOf(LHS.Zero))
       Known.Zero |= ~LowBits;
 
     // If the first operand is negative and not all low bits are zero, then
     // the upper bits are all one.
     if (LHS.isNegative() && LowBits.intersects(LHS.One))
       Known.One |= ~LowBits;
     return Known;
   }
 
+  remGetLowBits(Known, LHS, RHS);
+
   // The sign bit is the LHS's sign bit, except when the result of the
   // remainder is zero. The magnitude of the result should be less than or
   // equal to the magnitude of the LHS. Therefore any leading zeros that exist
   // in the left hand side must also exist in the result.
   Known.Zero.setHighBits(LHS.countMinLeadingZeros());
   return Known;
 }
 

llvm/test/Analysis/ScalarEvolution/merge-add-rec-many-inputs.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; RUN: opt < %s -disable-output "-passes=print<scalar-evolution>" 2>&1 | FileCheck %s
 
 ; Check that isImpliedViaMerge wouldn't crash when trying to prove
 ; SCEVUnknown and AddRec with phi having many inputs
 ; CHECK: @foo
 
 define void @foo(i1 %cond) {
 osr.type.merge79:

 bci_591:                                          ; preds = %bci_216
   %6 = add i32 1, %local_7_217
   br label %bci_199
 
 bci_221:                                          ; preds = %bci_216
   %7 = srem i32 %local_7_217, 6
   %8 = add i32 %7, 114
   switch i32 %8, label %done405 [
-    i32 114, label %bci_562
-    i32 116, label %bci_304
-    i32 117, label %bci_395
-    i32 118, label %bci_407
-    i32 119, label %bci_419
   ]
 
 bci_419:                                          ; preds = %bci_221
   %9 = sub i32 %local_2_188, %local_11_197
   br label %bci_435
 
 bci_435:                                          ; preds = %not_zero322, %bci_419
   br i1 %cond, label %not_zero265, label %never_deopt
out_of_bounds:                                    ; preds = %not_zero265
   br i1 %cond, label %bci_565, label %not_subtype
 
 bci_565:                                          ; preds = %out_of_bounds
   br label %bci_604
 
 not_subtype:                                      ; preds = %out_of_bounds
   br label %bci_604
 }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK: {{.*}}

llvm/test/Analysis/ValueTracking/knownbits-rem-lowbits.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=instcombine -S | FileCheck %s
 
 define i8 @urem_low_bits_know(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @urem_low_bits_know(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 2
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = urem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 2
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %x = or i8 %xx, 2
   %y = and i8 %yy, -4
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @urem_low_bits_know2(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @urem_low_bits_know2(
-; CHECK-NEXT:    [[XO:%.*]] = and i8 [[XX:%.*]], -4
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XO]], 2
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = urem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 3
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %xo = or i8 %xx, 2
   %x = and i8 %xo, 254
   %y = and i8 %yy, -4
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 3
   ret i8 %r
 }
;
   %y = and i8 %yy, -5
   %rem = urem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @srem_low_bits_know(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @srem_low_bits_know(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 10
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -4
-; CHECK-NEXT:    [[REM:%.*]] = srem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 2
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 2
 ;
   %x = or i8 %xx, 10
   %y = and i8 %yy, -4
   %rem = srem i8 %x, %y
   %r = and i8 %rem, 2
   ret i8 %r
 }
 
 define i8 @srem_low_bits_know2(i8 %xx, i8 %yy) {
 ; CHECK-LABEL: @srem_low_bits_know2(
-; CHECK-NEXT:    [[X:%.*]] = or i8 [[XX:%.*]], 1
-; CHECK-NEXT:    [[Y:%.*]] = and i8 [[YY:%.*]], -2
-; CHECK-NEXT:    [[REM:%.*]] = srem i8 [[X]], [[Y]]
-; CHECK-NEXT:    [[R:%.*]] = and i8 [[REM]], 1
-; CHECK-NEXT:    ret i8 [[R]]
+; CHECK-NEXT:    ret i8 1
 ;
   %x = or i8 %xx, 1
   %y = and i8 %yy, -2
   %rem = srem i8 %x, %y
   %r = and i8 %rem, 1
   ret i8 %r
 }
 

llvm/test/Transforms/LoopVectorize/AArch64/masked-call.ll

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR2:[0-9]+]]

 ; TFNONE-NEXT:    [[TMP8:%.*]] = getelementptr inbounds i64, ptr [[B:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[PREDPHI]], ptr [[TMP8]], align 8
 ; TFNONE-NEXT:    [[TMP9:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP10:%.*]] = mul i64 [[TMP9]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP10]]
 ; TFNONE-NEXT:    [[TMP11:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP11]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[IF_END:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; TFNONE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[TMP12:%.*]] = load i64, ptr [[ARRAYIDX]], align 8
 ; TFNONE-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP12]], 50

 ; TFNONE-NEXT:    [[TMP9:%.*]] = getelementptr inbounds i64, ptr [[B:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[PREDPHI]], ptr [[TMP9]], align 8
 ; TFNONE-NEXT:    [[TMP10:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP11:%.*]] = mul i64 [[TMP10]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP11]]
 ; TFNONE-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[IF_END:%.*]] ], [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ]
 ; TFNONE-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[TMP13:%.*]] = load i64, ptr [[ARRAYIDX]], align 8
 ; TFNONE-NEXT:    [[CMP:%.*]] = icmp ugt i64 [[TMP13]], 50

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR4:[0-9]+]]

 ; TFFALLBACK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFFALLBACK-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFFALLBACK-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFFALLBACK-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFFALLBACK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFFALLBACK-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFFALLBACK-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; TFFALLBACK:       middle.block:
-; TFFALLBACK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFFALLBACK-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFFALLBACK-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFFALLBACK:       scalar.ph:
 ; TFFALLBACK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFFALLBACK-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFFALLBACK:       for.body:
 ; TFFALLBACK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFFALLBACK-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFFALLBACK-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFFALLBACK-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR6:[0-9]+]]

 ; TFNONE-NEXT:    [[TMP6:%.*]] = getelementptr inbounds i64, ptr [[A:%.*]], i64 [[INDEX]]
 ; TFNONE-NEXT:    store <vscale x 2 x i64> [[TMP5]], ptr [[TMP6]], align 4
 ; TFNONE-NEXT:    [[TMP7:%.*]] = call i64 @llvm.vscale.i64()
 ; TFNONE-NEXT:    [[TMP8:%.*]] = mul i64 [[TMP7]], 2
 ; TFNONE-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP8]]
 ; TFNONE-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
 ; TFNONE-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
 ; TFNONE:       middle.block:
-; TFNONE-NEXT:    [[CMP_N:%.*]] = icmp eq i64 1025, [[N_VEC]]
-; TFNONE-NEXT:    br i1 [[CMP_N]], label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
+; TFNONE-NEXT:    br i1 false, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
 ; TFNONE:       scalar.ph:
 ; TFNONE-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
 ; TFNONE-NEXT:    br label [[FOR_BODY:%.*]]
 ; TFNONE:       for.body:
 ; TFNONE-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
 ; TFNONE-NEXT:    [[GEP:%.*]] = getelementptr i64, ptr [[B]], i64 [[INDVARS_IV]]
 ; TFNONE-NEXT:    [[LOAD:%.*]] = load i64, ptr [[GEP]], align 4
 ; TFNONE-NEXT:    [[CALL:%.*]] = call i64 @foo(i64 [[LOAD]]) #[[ATTR5:[0-9]+]]