[InstCombine] Simplify cttz/ctlz + icmp ugt/ult into mask check

[nikic]

Summary

Followup to D55745, this time handling comparisons with ugt and ult predicates (which are the canonical forms for non-equality predicates).

Based on the results, for ctlz we have the nice property that the mask check can always be represented as a ult/ugt comparison instead. I'm wondering if I should just emit it directly in that form and drop the single use check?

[lebedev.ri]

lib/Transforms/InstCombine/InstCombineCompares.cpp

2771–2772 ↗

(On Diff #180370)

I wonder if it would be less ugly to split this into foldICmpIntrinsicWithConstant() and foldICmpEQIntrinsicWithConstant()?

2789–2791 ↗

(On Diff #180370)

Can we instead do this check later on in the code, when we know it is needed?

test/Transforms/InstCombine/cmp-intrinsic.ll

311–401 ↗

(On Diff #180370)

1. Why cram everything into one test file? Why not split into ctlz.ll + cttz.ll + ...
2. These are cttz tests. Using %lz is *very* confusing. Please fix this.

[RKSimon]

vector tests?

[nikic]

Split up functions for eq and non-eq icmps. Handle ctlz separately from cttz by going directly to the form using only a comparison. Rebase over improved tests with multiuse and vector variants.

[lebedev.ri]

Hm, why does this modify the existing instruction, and manually adds it to worklist,
instead of simply creating a new instruction? This is rather unusual for instcombine i think.

lib/Transforms/InstCombine/InstCombineCompares.cpp

2771–2772 ↗

(On Diff #180370)

Please do mark fixed comments as such.

[nikic]

In D56355#1347365, @lebedev.ri wrote:

Hm, why does this modify the existing instruction, and manually adds it to worklist,
instead of simply creating a new instruction? This is rather unusual for instcombine i think.

I assumed that this was to avoid creating a new instruction when we can reuse an existing one. If this is just an anachronism, I can switch the existing code to create new nodes in a separate NFC commit and then adjust it here as well.

[lebedev.ri]

In D56355#1347389, @nikic wrote:

In D56355#1347365, @lebedev.ri wrote:

Hm, why does this modify the existing instruction, and manually adds it to worklist,
instead of simply creating a new instruction? This is rather unusual for instcombine i think.

I assumed that this was to avoid creating a new instruction when we can reuse an existing one.
If this is just an anachronism, I can switch the existing code to create new nodes in a separate NFC commit and then adjust it here as well.

Unless the current approach was specifically requested during previous reviews (then a comment is needed),
IMHO yes please do change this.

[nikic]

Create new compare instructions instead of modifying existing ones and manually managing worklist.

I'll also update the existing code for intrinsic icmps to this style, if it's accepted in this form.

[nikic]

Ping

[RKSimon]

lib/Transforms/InstCombine/InstCombineCompares.cpp

2617 ↗

(On Diff #180378)

(style) Remove unnecessary braces

2836 ↗

(On Diff #180378)

In case of "silly" numbers from fuzzers etc. better to use C.ult(BitWidth) etc.and move C.getLimitedValue() calls inside the ifs.

[nikic]

Remove unnecessary braces. Use getLimitedValue() only after range checks.

[RKSimon]

No more comments from me @spatel ?

[spatel]

LGTM - see inline comment for minor improvement.

lib/Transforms/InstCombine/InstCombineCompares.cpp

2855 ↗

(On Diff #182338)

It would be nice to have a code comment that shows the general transform formula and/or an example of how this works based on the regression tests. Seeing those makes it clear, but it's not obvious to me just looking at the code.

llvm/trunk/lib/Transforms/InstCombine/InstCombineCompares.cpp

if (Instruction *I = foldICmpTruncConstant(Cmp, TI, *C))
       return I;
   }
 
   if (auto *BCI = dyn_cast<BitCastInst>(Cmp.getOperand(0))) {
     if (Instruction *I = foldICmpBitCastConstant(Cmp, BCI, *C))
       return I;
   }
 
-  if (Instruction *I = foldICmpIntrinsicWithConstant(Cmp, *C))
+  if (auto *II = dyn_cast<IntrinsicInst>(Cmp.getOperand(0)))
+    if (Instruction *I = foldICmpIntrinsicWithConstant(Cmp, II, *C))
       return I;
 
   return nullptr;
 }
 
 /// Fold an icmp equality instruction with binary operator LHS and constant RHS:
 /// icmp eq/ne BO, C.
 Instruction *InstCombiner::foldICmpBinOpEqualityWithConstant(ICmpInst &Cmp,
                                                              BinaryOperator *BO,
case Instruction::UDiv:
     }
     break;
   default:
     break;
   }
   return nullptr;
 }
 
-/// Fold an icmp with LLVM intrinsic and constant operand: icmp Pred II, C.
-Instruction *InstCombiner::foldICmpIntrinsicWithConstant(ICmpInst &Cmp,
+/// Fold an equality icmp with LLVM intrinsic and constant operand.
+Instruction *InstCombiner::foldICmpEqIntrinsicWithConstant(ICmpInst &Cmp,
+                                                           IntrinsicInst *II,
                                                            const APInt &C) {
-  IntrinsicInst *II = dyn_cast<IntrinsicInst>(Cmp.getOperand(0));
-  if (!II || !Cmp.isEquality())
-    return nullptr;
-
-  // Handle icmp {eq|ne} <intrinsic>, Constant.
   Type *Ty = II->getType();
   unsigned BitWidth = C.getBitWidth();
   switch (II->getIntrinsicID()) {
   case Intrinsic::bswap:
     Worklist.Add(II);
     Cmp.setOperand(0, II->getArgOperand(0));
     Cmp.setOperand(1, ConstantInt::get(Ty, C.byteSwap()));
     return &Cmp;
Instruction *InstCombiner::foldICmpEqIntrinsicWithConstant(ICmpInst &Cmp,
   }
   default:
     break;
   }
 
   return nullptr;
 }
 
+/// Fold an icmp with LLVM intrinsic and constant operand: icmp Pred II, C.
+Instruction *InstCombiner::foldICmpIntrinsicWithConstant(ICmpInst &Cmp,
+                                                         IntrinsicInst *II,
+                                                         const APInt &C) {
+  if (Cmp.isEquality())
+    return foldICmpEqIntrinsicWithConstant(Cmp, II, C);
+
+  Type *Ty = II->getType();
+  unsigned BitWidth = C.getBitWidth();
+  switch (II->getIntrinsicID()) {
+  case Intrinsic::ctlz: {
+    // ctlz(0bXXXXXXXX) > 3 -> 0bXXXXXXXX < 0b00010000
+    if (Cmp.getPredicate() == ICmpInst::ICMP_UGT && C.ult(BitWidth)) {
+      unsigned Num = C.getLimitedValue();
+      APInt Limit = APInt::getOneBitSet(BitWidth, BitWidth - Num - 1);
+      return CmpInst::Create(Instruction::ICmp, ICmpInst::ICMP_ULT,
+                             II->getArgOperand(0), ConstantInt::get(Ty, Limit));
+    }
+
+    // ctlz(0bXXXXXXXX) < 3 -> 0bXXXXXXXX > 0b00011111
+    if (Cmp.getPredicate() == ICmpInst::ICMP_ULT &&
+        C.uge(1) && C.ule(BitWidth)) {
+      unsigned Num = C.getLimitedValue();
+      APInt Limit = APInt::getLowBitsSet(BitWidth, BitWidth - Num);
+      return CmpInst::Create(Instruction::ICmp, ICmpInst::ICMP_UGT,
+                             II->getArgOperand(0), ConstantInt::get(Ty, Limit));
+    }
+    break;
+  }
+  case Intrinsic::cttz: {
+    // Limit to one use to ensure we don't increase instruction count.
+    if (!II->hasOneUse())
+      return nullptr;
+
+    // cttz(0bXXXXXXXX) > 3 -> 0bXXXXXXXX & 0b00001111 == 0
+    if (Cmp.getPredicate() == ICmpInst::ICMP_UGT && C.ult(BitWidth)) {
+      APInt Mask = APInt::getLowBitsSet(BitWidth, C.getLimitedValue() + 1);
+      return CmpInst::Create(Instruction::ICmp, ICmpInst::ICMP_EQ,
+                             Builder.CreateAnd(II->getArgOperand(0), Mask),
+                             ConstantInt::getNullValue(Ty));
+    }
+
+    // cttz(0bXXXXXXXX) < 3 -> 0bXXXXXXXX & 0b00000111 != 0
+    if (Cmp.getPredicate() == ICmpInst::ICMP_ULT &&
+        C.uge(1) && C.ule(BitWidth)) {
+      APInt Mask = APInt::getLowBitsSet(BitWidth, C.getLimitedValue());
+      return CmpInst::Create(Instruction::ICmp, ICmpInst::ICMP_NE,
+                             Builder.CreateAnd(II->getArgOperand(0), Mask),
+                             ConstantInt::getNullValue(Ty));
+    }
+    break;
+  }
+  default:
+    break;
+  }
+
+  return nullptr;
+}
+
 /// Handle icmp with constant (but not simple integer constant) RHS.
 Instruction *InstCombiner::foldICmpInstWithConstantNotInt(ICmpInst &I) {
   Value *Op0 = I.getOperand(0), *Op1 = I.getOperand(1);
   Constant *RHSC = dyn_cast<Constant>(Op1);
   Instruction *LHSI = dyn_cast<Instruction>(Op0);
   if (!RHSC || !LHSI)
     return nullptr;
 

llvm/trunk/lib/Transforms/InstCombine/InstCombineInternal.h

private:
   Instruction *foldICmpShrConstConst(ICmpInst &I, Value *ShAmt, const APInt &C1,
                                      const APInt &C2);
   Instruction *foldICmpShlConstConst(ICmpInst &I, Value *ShAmt, const APInt &C1,
                                      const APInt &C2);
 
   Instruction *foldICmpBinOpEqualityWithConstant(ICmpInst &Cmp,
                                                  BinaryOperator *BO,
                                                  const APInt &C);
-  Instruction *foldICmpIntrinsicWithConstant(ICmpInst &ICI, const APInt &C);
+  Instruction *foldICmpIntrinsicWithConstant(ICmpInst &ICI, IntrinsicInst *II,
+                                             const APInt &C);
+  Instruction *foldICmpEqIntrinsicWithConstant(ICmpInst &ICI, IntrinsicInst *II,
+                                               const APInt &C);
 
   // Helpers of visitSelectInst().
   Instruction *foldSelectExtConst(SelectInst &Sel);
   Instruction *foldSelectOpOp(SelectInst &SI, Instruction *TI, Instruction *FI);
   Instruction *foldSelectIntoOp(SelectInst &SI, Value *, Value *);
   Instruction *foldSPFofSPF(Instruction *Inner, SelectPatternFlavor SPF1,
                             Value *A, Value *B, Instruction &Outer,
                             SelectPatternFlavor SPF2, Value *C);

llvm/trunk/test/Transforms/InstCombine/cmp-intrinsic.ll

 ;
   %lz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
   %cmp = icmp ugt i32 %lz, 0
   ret i1 %cmp
 }
 
 define i1 @ctlz_ugt_one_i32(i32 %x) {
 ; CHECK-LABEL: @ctlz_ugt_one_i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[X:%.*]], i1 false), !range !0
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[LZ]], 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[X:%.*]], 1073741824
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %lz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
   %cmp = icmp ugt i32 %lz, 1
   ret i1 %cmp
 }
 
 define i1 @ctlz_ugt_other_i32(i32 %x) {
 ; CHECK-LABEL: @ctlz_ugt_other_i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[X:%.*]], i1 false), !range !0
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[LZ]], 16
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[X:%.*]], 32768
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %lz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
   %cmp = icmp ugt i32 %lz, 16
   ret i1 %cmp
 }
 
 define i1 @ctlz_ugt_other_multiuse_i32(i32 %x, i32* %p) {
 ; CHECK-LABEL: @ctlz_ugt_other_multiuse_i32(
 ; CHECK-NEXT:    [[LZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[X:%.*]], i1 false), !range !0
 ; CHECK-NEXT:    store i32 [[LZ]], i32* [[P:%.*]], align 4
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[LZ]], 16
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[X]], 32768
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %lz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
   store i32 %lz, i32* %p
   %cmp = icmp ugt i32 %lz, 16
   ret i1 %cmp
 }
 
 define i1 @ctlz_ugt_bw_minus_one_i32(i32 %x) {
 ; CHECK-LABEL: @ctlz_ugt_bw_minus_one_i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[X:%.*]], i1 false), !range !0
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[LZ]], 31
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %lz = tail call i32 @llvm.ctlz.i32(i32 %x, i1 false)
   %cmp = icmp ugt i32 %lz, 31
   ret i1 %cmp
 }
 
 define <2 x i1> @ctlz_ult_one_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @ctlz_ult_one_v2i32(
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp slt <2 x i32> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %lz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %lz, <i32 1, i32 1>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @ctlz_ult_other_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @ctlz_ult_other_v2i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[LZ]], <i32 16, i32 16>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt <2 x i32> [[X:%.*]], <i32 65535, i32 65535>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %lz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %lz, <i32 16, i32 16>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @ctlz_ult_other_multiuse_v2i32(<2 x i32> %x, <2 x i32>* %p) {
 ; CHECK-LABEL: @ctlz_ult_other_multiuse_v2i32(
 ; CHECK-NEXT:    [[LZ:%.*]] = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> [[X:%.*]], i1 false)
 ; CHECK-NEXT:    store <2 x i32> [[LZ]], <2 x i32>* [[P:%.*]], align 8
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[LZ]], <i32 16, i32 16>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt <2 x i32> [[X]], <i32 65535, i32 65535>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %lz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 false)
   store <2 x i32> %lz, <2 x i32>* %p
   %cmp = icmp ult <2 x i32> %lz, <i32 16, i32 16>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @ctlz_ult_bw_minus_one_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @ctlz_ult_bw_minus_one_v2i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[LZ]], <i32 31, i32 31>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt <2 x i32> [[X:%.*]], <i32 1, i32 1>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %lz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %lz, <i32 31, i32 31>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @ctlz_ult_bitwidth_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @ctlz_ult_bitwidth_v2i32(
-; CHECK-NEXT:    [[LZ:%.*]] = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[LZ]], <i32 32, i32 32>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne <2 x i32> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %lz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %lz, <i32 32, i32 32>
   ret <2 x i1> %cmp
 }
 
 define i1 @cttz_ne_bitwidth_i33(i33 %x) {

 ;
   %tz = tail call i33 @llvm.cttz.i33(i33 %x, i1 false)
   %cmp = icmp ugt i33 %tz, 0
   ret i1 %cmp
 }
 
 define i1 @cttz_ugt_one_i33(i33 %x) {
 ; CHECK-LABEL: @cttz_ugt_one_i33(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call i33 @llvm.cttz.i33(i33 [[X:%.*]], i1 false), !range !1
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i33 [[TZ]], 1
+; CHECK-NEXT:    [[TMP1:%.*]] = and i33 [[X:%.*]], 3
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i33 [[TMP1]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %tz = tail call i33 @llvm.cttz.i33(i33 %x, i1 false)
   %cmp = icmp ugt i33 %tz, 1
   ret i1 %cmp
 }
 
 define i1 @cttz_ugt_other_i33(i33 %x) {
 ; CHECK-LABEL: @cttz_ugt_other_i33(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call i33 @llvm.cttz.i33(i33 [[X:%.*]], i1 false), !range !1
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i33 [[TZ]], 16
+; CHECK-NEXT:    [[TMP1:%.*]] = and i33 [[X:%.*]], 131071
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i33 [[TMP1]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %tz = tail call i33 @llvm.cttz.i33(i33 %x, i1 false)
   %cmp = icmp ugt i33 %tz, 16
   ret i1 %cmp
 }
 
 define i1 @cttz_ugt_other_multiuse_i33(i33 %x, i33* %p) {
 ; CHECK-LABEL: @cttz_ugt_other_multiuse_i33(
 ; CHECK-NEXT:    [[TZ:%.*]] = tail call i33 @llvm.cttz.i33(i33 [[X:%.*]], i1 false), !range !1
 ; CHECK-NEXT:    store i33 [[TZ]], i33* [[P:%.*]], align 4
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i33 [[TZ]], 16
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %tz = tail call i33 @llvm.cttz.i33(i33 %x, i1 false)
   store i33 %tz, i33* %p
   %cmp = icmp ugt i33 %tz, 16
   ret i1 %cmp
 }
 
 define i1 @cttz_ugt_bw_minus_one_i33(i33 %x) {
 ; CHECK-LABEL: @cttz_ugt_bw_minus_one_i33(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call i33 @llvm.cttz.i33(i33 [[X:%.*]], i1 false), !range !1
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i33 [[TZ]], 32
+; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i33 [[X:%.*]], 0
 ; CHECK-NEXT:    ret i1 [[CMP]]
 ;
   %tz = tail call i33 @llvm.cttz.i33(i33 %x, i1 false)
   %cmp = icmp ugt i33 %tz, 32
   ret i1 %cmp
 }
 
 define <2 x i1> @cttz_ult_one_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_ult_one_v2i32(
 ; CHECK-NEXT:    [[CMP:%.*]] = trunc <2 x i32> [[X:%.*]] to <2 x i1>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %tz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %tz, <i32 1, i32 1>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @cttz_ult_other_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_ult_other_v2i32(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[TZ]], <i32 16, i32 16>
+; CHECK-NEXT:    [[TMP1:%.*]] = and <2 x i32> [[X:%.*]], <i32 65535, i32 65535>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne <2 x i32> [[TMP1]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %tz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %tz, <i32 16, i32 16>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @cttz_ult_other_multiuse_v2i32(<2 x i32> %x, <2 x i32>* %p) {
 ; CHECK-LABEL: @cttz_ult_other_multiuse_v2i32(
 ; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 false)
 ; CHECK-NEXT:    store <2 x i32> [[TZ]], <2 x i32>* [[P:%.*]], align 8
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[TZ]], <i32 16, i32 16>
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %tz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %x, i1 false)
   store <2 x i32> %tz, <2 x i32>* %p
   %cmp = icmp ult <2 x i32> %tz, <i32 16, i32 16>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @cttz_ult_bw_minus_one_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_ult_bw_minus_one_v2i32(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[TZ]], <i32 31, i32 31>
+; CHECK-NEXT:    [[TMP1:%.*]] = and <2 x i32> [[X:%.*]], <i32 2147483647, i32 2147483647>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne <2 x i32> [[TMP1]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %tz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %tz, <i32 31, i32 31>
   ret <2 x i1> %cmp
 }
 
 define <2 x i1> @cttz_ult_bitwidth_v2i32(<2 x i32> %x) {
 ; CHECK-LABEL: @cttz_ult_bitwidth_v2i32(
-; CHECK-NEXT:    [[TZ:%.*]] = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[X:%.*]], i1 false)
-; CHECK-NEXT:    [[CMP:%.*]] = icmp ult <2 x i32> [[TZ]], <i32 32, i32 32>
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ne <2 x i32> [[X:%.*]], zeroinitializer
 ; CHECK-NEXT:    ret <2 x i1> [[CMP]]
 ;
   %tz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %x, i1 false)
   %cmp = icmp ult <2 x i32> %tz, <i32 32, i32 32>
   ret <2 x i1> %cmp
 }
 
 define i1 @ctpop_eq_zero_i11(i11 %x) {