[IR] document and update ctlz/cttz intrinsics to optionally return poison rather than undef

[spatel]

Summary

The behavior in Analysis (knownbits) implements poison semantics already, and we expect the transforms (for example, in instcombine) derived from those semantics, so this patch changes the LangRef and remaining code to be consistent. This is one more step in removing "undef" from LLVM.

Without this, I think https://github.com/llvm/llvm-project/issues/53330 has a legitimate complaint because that report wants to allow subsequent code to mask off bits, and that is allowed with undef values. The clang builtins are not actually documented anywhere AFAICT, but we might want to add that to remove more uncertainty.

Hopefully, we are approaching a point where we can stop caring about the legacy behavior (~10 years since better x86 instructions were introduced)...but I'm usually too optimistic about those timeframes. :)

[nikic]

LGTM

[lebedev.ri]

lg
Thanks.

[efriedma]

Seems fine. This is technically not bitcode backwards-compatible, but that seems unlikely to matter.

Hopefully, we are approaching a point where we can stop caring about the legacy behavior

You mean, we might reach a point where we could make clang emit is_zero_poison=false without anyone caring? Don't think we're quite there yet. x86-64-v2 doesn't include lzcnt. And not sure about the current state of the compiler-rt/libgcc intrinsics.

[nlopes]

Great, thank you!

[spatel]

In D117912#3262125, @efriedma wrote:

You mean, we might reach a point where we could make clang emit is_zero_poison=false without anyone caring? Don't think we're quite there yet. x86-64-v2 doesn't include lzcnt. And not sure about the current state of the compiler-rt/libgcc intrinsics.

Yes, I was wondering what level of default x86 tuning was needed to flip the setting...I suppose that's x86-64-v3 ("Close to Haswell").

[craig.topper]

In D117912#3262280, @spatel wrote:

In D117912#3262125, @efriedma wrote:

You mean, we might reach a point where we could make clang emit is_zero_poison=false without anyone caring? Don't think we're quite there yet. x86-64-v2 doesn't include lzcnt. And not sure about the current state of the compiler-rt/libgcc intrinsics.

Yes, I was wondering what level of default x86 tuning was needed to flip the setting...I suppose that's x86-64-v3 ("Close to Haswell").

Is updating isCheapToSpeculateCTTZ/CTLZ to check CMOV instead of BMI/LZCNT enough?

[spatel]

In D117912#3262368, @craig.topper wrote:

In D117912#3262280, @spatel wrote:

In D117912#3262125, @efriedma wrote:

You mean, we might reach a point where we could make clang emit is_zero_poison=false without anyone caring? Don't think we're quite there yet. x86-64-v2 doesn't include lzcnt. And not sure about the current state of the compiler-rt/libgcc intrinsics.

Yes, I was wondering what level of default x86 tuning was needed to flip the setting...I suppose that's x86-64-v3 ("Close to Haswell").

Is updating isCheapToSpeculateCTTZ/CTLZ to check CMOV instead of BMI/LZCNT enough?

That would let this:

int clipped2(std::uint64_t a) { return (a == 0) ? 64 : __builtin_ctzll(a); }

Become:

bsfq	%rdi, %rcx
movl	$64, %eax
cmovneq	%rcx, %rax

Instead of a 'test + je' around the bsf, so that's probably good. But I can't remember all of the corner-case patterns that we're trying to account for with these builtins.

[zielaj]

I'd like to provide more context for https://github.com/llvm/llvm-project/issues/53330. This issue arose while working on performance-critical code in primesieve, which uses the output of __builtin_ctzll to do an array lookup. It turns out that, because of poorly predictable branches, it's faster for us to compute __builtin_ctzll(0), do the lookup, and then ignore it later, than to call __builtin_ctzll(x) conditionally on x != 0. Note that we don't need __builtin_ctzll(0) to be equal to 64 or to any specific value, we just need to be able to bound it somehow so that the array lookup doesn't segfault. This is where the idea of & 0x7f came from. The hope was that when compiled on processors that support TZCNT this would be optimized out to nothing, yet still work on processors that don't, at a cost of one additional instruction (AND).

The fastest workaround we have is computing __builtin_ctzll(x | (1ull << 63)) instead. This adds one more 1-cycle instruction (OR), is always bounded, and is equivalent to __builtin_ctzll(x) for non-zero x. Still, since the calls are in a hot loop, this additional instruction results in measurable slowdown, this is why primesieve currently uses inline assembly to force TZCNT when possible, because we couldn't find a C++-only way to accomplish this.

This comment is not a disagreement with this proposal, it just provides more context. I accept that our need may be too obscure to be addressed by clang, and wanted to say thank you for responding to the issue I raised so quickly and for clarifying how undefined __builtin_ctzll(0) is.

[craig.topper]

In D117912#3262669, @zielaj wrote:

I'd like to provide more context for https://github.com/llvm/llvm-project/issues/53330. This issue arose while working on performance-critical code in primesieve, which uses the output of __builtin_ctzll to do an array lookup. It turns out that, because of poorly predictable branches, it's faster for us to compute __builtin_ctzll(0), do the lookup, and then ignore it later, than to call __builtin_ctzll(x) conditionally on x != 0. Note that we don't need __builtin_ctzll(0) to be equal to 64 or to any specific value, we just need to be able to bound it somehow so that the array lookup doesn't segfault. This is where the idea of & 0x7f came from. The hope was that when compiled on processors that support TZCNT this would be optimized out to nothing, yet still work on processors that don't, at a cost of one additional instruction (AND).

The fastest workaround we have is computing __builtin_ctzll(x | (1ull << 63)) instead. This adds one more 1-cycle instruction (OR), is always bounded, and is equivalent to __builtin_ctzll(x) for non-zero x. Still, since the calls are in a hot loop, this additional instruction results in measurable slowdown, this is why primesieve currently uses inline assembly to force TZCNT when possible, because we couldn't find a C++-only way to accomplish this.

This comment is not a disagreement with this proposal, it just provides more context. I accept that our need may be too obscure to be addressed by clang, and wanted to say thank you for responding to the issue I raised so quickly and for clarifying how undefined __builtin_ctzll(0) is.

Does x != 0 ? 64 : __builtin_ctzll(x) not produce tzcnt when it's available? I would hope you wouldn't need to resort to inline assembly.

[zielaj]

@craig.topper Yes, it does, when compiled for example with -march=skylake. But when compiled with -march=x86-64, which I guess is a default for many linux distributions, it compiles to conditional jumps rather than conditional moves, even with __builtin_unpredictable, which is slow in our case.

On top of this, a widely used version of GCC (11.2), inserts conditional jumps even with -march=skylake (the trunk version of GCC has this fixed).

So, in this case, it's difficult to guarantee optimum generated code case across compilers and architectures.

[xbolva00]

__builtin_unpredictable

Backend (SDAG) ignores this hint sadly, it would be really cool to have it.

[craig.topper]

In D117912#3263773, @xbolva00 wrote:

__builtin_unpredictable

Backend (SDAG) ignores this hint sadly, it would be really cool to have it.

Does the hint even survive through the middle end? I think the pattern will be turned into llvm.cttz.i64(i64 %x, i1 false). Then, CodeGenPrepare turns it back into the control flow version based on isCheapToSpeculateCTTZ.

[xbolva00]

In D117912#3263872, @craig.topper wrote:

In D117912#3263773, @xbolva00 wrote:

__builtin_unpredictable

Backend (SDAG) ignores this hint sadly, it would be really cool to have it.

Does the hint even survive through the middle end? I think the pattern will be turned into llvm.cttz.i64(i64 %x, i1 false). Then, CodeGenPrepare turns it back into the control flow version based on isCheapToSpeculateCTTZ.

Ah, indeed, yes.

[xbolva00]

unpredictable metadata may be attached to any branch or switch instruction.

maybe allow for intrinsics too?

[spatel]

Thanks, @zielaj for explaining the larger example. It's not clear to me exactly what clang/LLVM can do to improve that case, but if you have a suggestion, please do file another issue.

llvm/docs/LangRef.rst

 """""""
 
 This is an overloaded intrinsic. You can use ``llvm.ctlz`` on any
 integer bit width, or any vector whose elements are integers. Not all
 targets support all bit widths or vector types, however.
 
 ::
 
-      declare i8   @llvm.ctlz.i8  (i8   <src>, i1 <is_zero_undef>)
-      declare i16  @llvm.ctlz.i16 (i16  <src>, i1 <is_zero_undef>)
-      declare i32  @llvm.ctlz.i32 (i32  <src>, i1 <is_zero_undef>)
-      declare i64  @llvm.ctlz.i64 (i64  <src>, i1 <is_zero_undef>)
-      declare i256 @llvm.ctlz.i256(i256 <src>, i1 <is_zero_undef>)
-      declare <2 x i32> @llvm.ctlz.v2i32(<2 x i32> <src>, i1 <is_zero_undef>)
+      declare i8   @llvm.ctlz.i8  (i8   <src>, i1 <is_zero_poison>)
+      declare <2 x i37> @llvm.ctlz.v2i37(<2 x i37> <src>, i1 <is_zero_poison>)
 
 Overview:
 """""""""
 
 The '``llvm.ctlz``' family of intrinsic functions counts the number of
 leading zeros in a variable.
 
 Arguments:
 """"""""""
 
 The first argument is the value to be counted. This argument may be of
 any integer type, or a vector with integer element type. The return
 type must match the first argument type.
 
-The second argument must be a constant and is a flag to indicate whether
-the intrinsic should ensure that a zero as the first argument produces a
-defined result. Historically some architectures did not provide a
-defined result for zero values as efficiently, and many algorithms are
-now predicated on avoiding zero-value inputs.
+The second argument is a constant flag that indicates whether the intrinsic
+returns a valid result if the first argument is zero. If the first
+argument is zero and the second argument is true, the result is poison.
+Historically some architectures did not provide a defined result for zero
+values as efficiently, and many algorithms are now predicated on avoiding
+zero-value inputs.
 
 Semantics:
 """"""""""
 
 The '``llvm.ctlz``' intrinsic counts the leading (most significant)
 zeros in a variable, or within each element of the vector. If
 ``src == 0`` then the result is the size in bits of the type of ``src``
-if ``is_zero_undef == 0`` and ``undef`` otherwise. For example,
+if ``is_zero_poison == 0`` and ``poison`` otherwise. For example,
 ``llvm.ctlz(i32 2) = 30``.
 
 '``llvm.cttz.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 This is an overloaded intrinsic. You can use ``llvm.cttz`` on any
 integer bit width, or any vector of integer elements. Not all targets
 support all bit widths or vector types, however.
 
 ::
 
-      declare i8   @llvm.cttz.i8  (i8   <src>, i1 <is_zero_undef>)
-      declare i16  @llvm.cttz.i16 (i16  <src>, i1 <is_zero_undef>)
-      declare i32  @llvm.cttz.i32 (i32  <src>, i1 <is_zero_undef>)
-      declare i64  @llvm.cttz.i64 (i64  <src>, i1 <is_zero_undef>)
-      declare i256 @llvm.cttz.i256(i256 <src>, i1 <is_zero_undef>)
-      declare <2 x i32> @llvm.cttz.v2i32(<2 x i32> <src>, i1 <is_zero_undef>)
+      declare i42   @llvm.cttz.i42  (i42   <src>, i1 <is_zero_poison>)
+      declare <2 x i32> @llvm.cttz.v2i32(<2 x i32> <src>, i1 <is_zero_poison>)
 
 Overview:
 """""""""
 
 The '``llvm.cttz``' family of intrinsic functions counts the number of
 trailing zeros.
 
 Arguments:
 """"""""""
 
 The first argument is the value to be counted. This argument may be of
 any integer type, or a vector with integer element type. The return
 type must match the first argument type.
 
-The second argument must be a constant and is a flag to indicate whether
-the intrinsic should ensure that a zero as the first argument produces a
-defined result. Historically some architectures did not provide a
-defined result for zero values as efficiently, and many algorithms are
-now predicated on avoiding zero-value inputs.
+The second argument is a constant flag that indicates whether the intrinsic
+returns a valid result if the first argument is zero. If the first
+argument is zero and the second argument is true, the result is poison.
+Historically some architectures did not provide a defined result for zero
+values as efficiently, and many algorithms are now predicated on avoiding
+zero-value inputs.
 
 Semantics:
 """"""""""
 
 The '``llvm.cttz``' intrinsic counts the trailing (least significant)
 zeros in a variable, or within each element of a vector. If ``src == 0``
 then the result is the size in bits of the type of ``src`` if
-``is_zero_undef == 0`` and ``undef`` otherwise. For example,
+``is_zero_poison == 0`` and ``poison`` otherwise. For example,
 ``llvm.cttz(2) = 1``.
 
 .. _int_overflow:
 
 '``llvm.fshl.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:

llvm/lib/Analysis/ConstantFolding.cpp

case Intrinsic::ssub_sat:
       if (IntrinsicID == Intrinsic::usub_sat)
         return ConstantInt::get(Ty, C0->usub_sat(*C1));
       else
         return ConstantInt::get(Ty, C0->ssub_sat(*C1));
     case Intrinsic::cttz:
     case Intrinsic::ctlz:
       assert(C1 && "Must be constant int");
 
-      // cttz(0, 1) and ctlz(0, 1) are undef.
+      // cttz(0, 1) and ctlz(0, 1) are poison.
       if (C1->isOne() && (!C0 || C0->isZero()))
-        return UndefValue::get(Ty);
+        return PoisonValue::get(Ty);
       if (!C0)
         return Constant::getNullValue(Ty);
       if (IntrinsicID == Intrinsic::cttz)
         return ConstantInt::get(Ty, C0->countTrailingZeros());
       else
         return ConstantInt::get(Ty, C0->countLeadingZeros());
 
     case Intrinsic::abs:

llvm/lib/Analysis/ValueTracking.cpp

if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(I)) {
         computeKnownBits(I->getOperand(0), DemandedElts, Known2, Depth + 1, Q);
         Known.Zero |= Known2.Zero.byteSwap();
         Known.One |= Known2.One.byteSwap();
         break;
       case Intrinsic::ctlz: {
         computeKnownBits(I->getOperand(0), Known2, Depth + 1, Q);
         // If we have a known 1, its position is our upper bound.
         unsigned PossibleLZ = Known2.countMaxLeadingZeros();
-        // If this call is undefined for 0, the result will be less than 2^n.
+        // If this call is poison for 0 input, the result will be less than 2^n.
         if (II->getArgOperand(1) == ConstantInt::getTrue(II->getContext()))
           PossibleLZ = std::min(PossibleLZ, BitWidth - 1);
         unsigned LowBits = Log2_32(PossibleLZ)+1;
         Known.Zero.setBitsFrom(LowBits);
         break;
       }
       case Intrinsic::cttz: {
         computeKnownBits(I->getOperand(0), Known2, Depth + 1, Q);
         // If we have a known 1, its position is our upper bound.
         unsigned PossibleTZ = Known2.countMaxTrailingZeros();
-        // If this call is undefined for 0, the result will be less than 2^n.
+        // If this call is poison for 0 input, the result will be less than 2^n.
         if (II->getArgOperand(1) == ConstantInt::getTrue(II->getContext()))
           PossibleTZ = std::min(PossibleTZ, BitWidth - 1);
         unsigned LowBits = Log2_32(PossibleTZ)+1;
         Known.Zero.setBitsFrom(LowBits);
         break;
       }
       case Intrinsic::ctpop: {
         computeKnownBits(I->getOperand(0), Known2, Depth + 1, Q);

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

if (match(Op0, m_BitReverse(m_Value(X)))) {
     Function *F = Intrinsic::getDeclaration(II.getModule(), ID, II.getType());
     return CallInst::Create(F, {X, II.getArgOperand(1)});
   }
 
   if (II.getType()->isIntOrIntVectorTy(1)) {
     // ctlz/cttz i1 Op0 --> not Op0
     if (match(Op1, m_Zero()))
       return BinaryOperator::CreateNot(Op0);
-    // If zero is undef, then the input can be assumed to be "true", so the
+    // If zero is poison, then the input can be assumed to be "true", so the
     // instruction simplifies to "false".
     assert(match(Op1, m_One()) && "Expected ctlz/cttz operand to be 0 or 1");
     return IC.replaceInstUsesWith(II, ConstantInt::getNullValue(II.getType()));
   }
 
   // If the operand is a select with constant arm(s), try to hoist ctlz/cttz.
   if (auto *Sel = dyn_cast<SelectInst>(Op0))
     if (Instruction *R = IC.FoldOpIntoSelect(II, Sel))
       return R;
 
   if (IsTZ) {
     // cttz(-x) -> cttz(x)
     if (match(Op0, m_Neg(m_Value(X))))
       return IC.replaceOperand(II, 0, X);
 
     // cttz(sext(x)) -> cttz(zext(x))
     if (match(Op0, m_OneUse(m_SExt(m_Value(X))))) {
       auto *Zext = IC.Builder.CreateZExt(X, II.getType());
       auto *CttzZext =
           IC.Builder.CreateBinaryIntrinsic(Intrinsic::cttz, Zext, Op1);
       return IC.replaceInstUsesWith(II, CttzZext);
     }
 
     // Zext doesn't change the number of trailing zeros, so narrow:
-    // cttz(zext(x)) -> zext(cttz(x)) if the 'ZeroIsUndef' parameter is 'true'.
+    // cttz(zext(x)) -> zext(cttz(x)) if the 'ZeroIsPoison' parameter is 'true'.
     if (match(Op0, m_OneUse(m_ZExt(m_Value(X)))) && match(Op1, m_One())) {
       auto *Cttz = IC.Builder.CreateBinaryIntrinsic(Intrinsic::cttz, X,
                                                     IC.Builder.getTrue());
       auto *ZextCttz = IC.Builder.CreateZExt(Cttz, II.getType());
       return IC.replaceInstUsesWith(II, ZextCttz);
     }
 
     // cttz(abs(x)) -> cttz(x)
static Instruction *foldCttzCtlz(IntrinsicInst &II, InstCombinerImpl &IC) {
   // FIXME: This should be in InstSimplify because we're replacing an
   // instruction with a constant.
   if (PossibleZeros == DefiniteZeros) {
     auto *C = ConstantInt::get(Op0->getType(), DefiniteZeros);
     return IC.replaceInstUsesWith(II, C);
   }
 
   // If the input to cttz/ctlz is known to be non-zero,
-  // then change the 'ZeroIsUndef' parameter to 'true'
+  // then change the 'ZeroIsPoison' parameter to 'true'
   // because we know the zero behavior can't affect the result.
   if (!Known.One.isZero() ||
       isKnownNonZero(Op0, IC.getDataLayout(), 0, &IC.getAssumptionCache(), &II,
                      &IC.getDominatorTree())) {
     if (!match(II.getArgOperand(1), m_One()))
       return IC.replaceOperand(II, 1, IC.Builder.getTrue());
   }
 

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

if (!match(II->getOperand(0), m_c_And(m_Specific(X), m_Neg(m_Specific(X)))))
     return nullptr;
 
   Function *F = Intrinsic::getDeclaration(II->getModule(), Intrinsic::cttz,
                                           II->getType());
   return CallInst::Create(F, {X, II->getArgOperand(1)});
 }
 
 /// Attempt to fold a cttz/ctlz followed by a icmp plus select into a single
-/// call to cttz/ctlz with flag 'is_zero_undef' cleared.
+/// call to cttz/ctlz with flag 'is_zero_poison' cleared.
 ///
 /// For example, we can fold the following code sequence:
 /// \code
 ///   %0 = tail call i32 @llvm.cttz.i32(i32 %x, i1 true)
 ///   %1 = icmp ne i32 %x, 0
 ///   %2 = select i1 %1, i32 %0, i32 32
 /// \code
 ///
if (!match(Count, m_Intrinsic<Intrinsic::cttz>(m_Specific(CmpLHS))) &&
     return nullptr;
 
   IntrinsicInst *II = cast<IntrinsicInst>(Count);
 
   // Check if the value propagated on zero is a constant number equal to the
   // sizeof in bits of 'Count'.
   unsigned SizeOfInBits = Count->getType()->getScalarSizeInBits();
   if (match(ValueOnZero, m_SpecificInt(SizeOfInBits))) {
-    // Explicitly clear the 'undef_on_zero' flag. It's always valid to go from
+    // Explicitly clear the 'is_zero_poison' flag. It's always valid to go from
     // true to false on this flag, so we can replace it for all users.
     II->setArgOperand(1, ConstantInt::getFalse(II->getContext()));
     return SelectArg;
   }
 
   // The ValueOnZero is not the bitwidth. But if the cttz/ctlz (and optional
   // zext/trunc) have one use (ending at the select), the cttz/ctlz result will
-  // not be used if the input is zero. Relax to 'undef_on_zero' for that case.
+  // not be used if the input is zero. Relax to 'zero is poison' for that case.
   if (II->hasOneUse() && SelectArg->hasOneUse() &&
       !match(II->getArgOperand(1), m_One()))
     II->setArgOperand(1, ConstantInt::getTrue(II->getContext()));
 
   return nullptr;
 }
 
 /// Return true if we find and adjust an icmp+select pattern where the compare

llvm/test/Transforms/InstCombine/intrinsic-select.ll

;
   %s = select i1 %b, i32 5, i32 %x
   %c = call i32 @llvm.ctlz.i32(i32 %s, i1 false)
   ret i32 %c
 }
 
 define <3 x i17> @ctlz_sel_const_false(<3 x i1> %b, <3 x i17> %x) {
 ; CHECK-LABEL: @ctlz_sel_const_false(
 ; CHECK-NEXT:    [[TMP1:%.*]] = call <3 x i17> @llvm.ctlz.v3i17(<3 x i17> [[X:%.*]], i1 true)
-; CHECK-NEXT:    [[C:%.*]] = select <3 x i1> [[B:%.*]], <3 x i17> [[TMP1]], <3 x i17> <i17 14, i17 0, i17 undef>
+; CHECK-NEXT:    [[C:%.*]] = select <3 x i1> [[B:%.*]], <3 x i17> [[TMP1]], <3 x i17> <i17 14, i17 0, i17 poison>
 ; CHECK-NEXT:    ret <3 x i17> [[C]]
 ;
   %s = select <3 x i1> %b, <3 x i17> %x, <3 x i17> <i17 7, i17 -1, i17 0>
   %c = call <3 x i17> @llvm.ctlz.v3i17(<3 x i17> %s, i1 true)
   ret <3 x i17> %c
 }
 
 define i32 @ctlz_sel_const_true_false_extra_use(i1 %b) {

llvm/test/Transforms/InstCombine/intrinsics.ll

 ; CHECK-LABEL: @cttz_i1(
 ; CHECK-NEXT:    [[CNT:%.*]] = xor i1 [[ARG:%.*]], true
 ; CHECK-NEXT:    ret i1 [[CNT]]
 ;
   %cnt = call i1 @llvm.cttz.i1(i1 %arg, i1 false) nounwind readnone
   ret i1 %cnt
 }
 
-define i1 @cttz_i1_zero_is_undef(i1 %arg) {
-; CHECK-LABEL: @cttz_i1_zero_is_undef(
+define i1 @cttz_i1_zero_is_poison(i1 %arg) {
+; CHECK-LABEL: @cttz_i1_zero_is_poison(
 ; CHECK-NEXT:    ret i1 false
 ;
   %cnt = call i1 @llvm.cttz.i1(i1 %arg, i1 true) nounwind readnone
   ret i1 %cnt
 }
 
 define <2 x i1> @cttz_v2i1(<2 x i1> %arg) {
 ; CHECK-LABEL: @cttz_v2i1(
 ; CHECK-NEXT:    [[CNT:%.*]] = xor <2 x i1> [[ARG:%.*]], <i1 true, i1 true>
 ; CHECK-NEXT:    ret <2 x i1> [[CNT]]
 ;
   %cnt = call <2 x i1> @llvm.cttz.v2i1(<2 x i1> %arg, i1 false) nounwind readnone
   ret <2 x i1> %cnt
 }
 
-define <2 x i1> @cttz_v2i1_zero_is_undef(<2 x i1> %arg) {
-; CHECK-LABEL: @cttz_v2i1_zero_is_undef(
+define <2 x i1> @cttz_v2i1_zero_is_poison(<2 x i1> %arg) {
+; CHECK-LABEL: @cttz_v2i1_zero_is_poison(
 ; CHECK-NEXT:    ret <2 x i1> zeroinitializer
 ;
   %cnt = call <2 x i1> @llvm.cttz.v2i1(<2 x i1> %arg, i1 true) nounwind readnone
   ret <2 x i1> %cnt
 }
 
 define i1 @cttz_knownbits(i32 %arg) {
 ; CHECK-LABEL: @cttz_knownbits(

 ; CHECK-LABEL: @ctlz_i1(
 ; CHECK-NEXT:    [[CNT:%.*]] = xor i1 [[ARG:%.*]], true
 ; CHECK-NEXT:    ret i1 [[CNT]]
 ;
   %cnt = call i1 @llvm.ctlz.i1(i1 %arg, i1 false) nounwind readnone
   ret i1 %cnt
 }
 
-define i1 @ctlz_i1_zero_is_undef(i1 %arg) {
-; CHECK-LABEL: @ctlz_i1_zero_is_undef(
+define i1 @ctlz_i1_zero_is_poison(i1 %arg) {
+; CHECK-LABEL: @ctlz_i1_zero_is_poison(
 ; CHECK-NEXT:    ret i1 false
 ;
   %cnt = call i1 @llvm.ctlz.i1(i1 %arg, i1 true) nounwind readnone
   ret i1 %cnt
 }
 
 define <2 x i1> @ctlz_v2i1(<2 x i1> %arg) {
 ; CHECK-LABEL: @ctlz_v2i1(
 ; CHECK-NEXT:    [[CNT:%.*]] = xor <2 x i1> [[ARG:%.*]], <i1 true, i1 true>
 ; CHECK-NEXT:    ret <2 x i1> [[CNT]]
 ;
   %cnt = call <2 x i1> @llvm.ctlz.v2i1(<2 x i1> %arg, i1 false) nounwind readnone
   ret <2 x i1> %cnt
 }
 
-define <2 x i1> @ctlz_v2i1_zero_is_undef(<2 x i1> %arg) {
-; CHECK-LABEL: @ctlz_v2i1_zero_is_undef(
+define <2 x i1> @ctlz_v2i1_zero_is_poison(<2 x i1> %arg) {
+; CHECK-LABEL: @ctlz_v2i1_zero_is_poison(
 ; CHECK-NEXT:    ret <2 x i1> zeroinitializer
 ;
   %cnt = call <2 x i1> @llvm.ctlz.v2i1(<2 x i1> %arg, i1 true) nounwind readnone
   ret <2 x i1> %cnt
 }
 
 define i1 @ctlz_knownbits(i8 %arg) {
 ; CHECK-LABEL: @ctlz_knownbits(

 ; CHECK-NEXT:    ret <2 x i1> zeroinitializer
 ;
   %or = or <2 x i8> %arg, <i8 32, i8 32>
   %cnt = call <2 x i8> @llvm.ctlz.v2i8(<2 x i8> %or, i1 true) nounwind readnone
   %res = icmp eq <2 x i8> %cnt, <i8 3, i8 3>
   ret <2 x i1> %res
 }
 
-define i32 @ctlz_undef(i32 %Value) {
-; CHECK-LABEL: @ctlz_undef(
-; CHECK-NEXT:    ret i32 undef
+define i32 @ctlz_poison(i32 %Value) {
+; CHECK-LABEL: @ctlz_poison(
+; CHECK-NEXT:    ret i32 poison
 ;
   %ctlz = call i32 @llvm.ctlz.i32(i32 0, i1 true)
   ret i32 %ctlz
 }
 
-define <2 x i32> @ctlz_undef_vec(<2 x i32> %Value) {
-; CHECK-LABEL: @ctlz_undef_vec(
-; CHECK-NEXT:    ret <2 x i32> undef
+define <2 x i32> @ctlz_poison_vec(<2 x i32> %Value) {
+; CHECK-LABEL: @ctlz_poison_vec(
+; CHECK-NEXT:    ret <2 x i32> poison
 ;
   %ctlz = call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> zeroinitializer, i1 true)
   ret <2 x i32> %ctlz
 }
 
-define i32 @ctlz_make_undef(i32 %a) {
-; CHECK-LABEL: @ctlz_make_undef(
+define i32 @ctlz_no_zero(i32 %a) {
+; CHECK-LABEL: @ctlz_no_zero(
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[A:%.*]], 8
 ; CHECK-NEXT:    [[CTLZ:%.*]] = tail call i32 @llvm.ctlz.i32(i32 [[OR]], i1 true), !range [[RNG2:![0-9]+]]
 ; CHECK-NEXT:    ret i32 [[CTLZ]]
 ;
   %or = or i32 %a, 8
   %ctlz = tail call i32 @llvm.ctlz.i32(i32 %or, i1 false)
   ret i32 %ctlz
 }
 
-define <2 x i32> @ctlz_make_undef_vec(<2 x i32> %a) {
-; CHECK-LABEL: @ctlz_make_undef_vec(
+define <2 x i32> @ctlz_no_zero_vec(<2 x i32> %a) {
+; CHECK-LABEL: @ctlz_no_zero_vec(
 ; CHECK-NEXT:    [[OR:%.*]] = or <2 x i32> [[A:%.*]], <i32 8, i32 8>
 ; CHECK-NEXT:    [[CTLZ:%.*]] = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> [[OR]], i1 true)
 ; CHECK-NEXT:    ret <2 x i32> [[CTLZ]]
 ;
   %or = or <2 x i32> %a, <i32 8, i32 8>
   %ctlz = tail call <2 x i32> @llvm.ctlz.v2i32(<2 x i32> %or, i1 false)
   ret <2 x i32> %ctlz
 }
 
-define i32 @cttz_undef(i32 %Value) nounwind {
-; CHECK-LABEL: @cttz_undef(
-; CHECK-NEXT:    ret i32 undef
+define i32 @cttz_poison(i32 %Value) {
+; CHECK-LABEL: @cttz_poison(
+; CHECK-NEXT:    ret i32 poison
 ;
   %cttz = call i32 @llvm.cttz.i32(i32 0, i1 true)
   ret i32 %cttz
 }
 
-define <2 x i32> @cttz_undef_vec(<2 x i32> %Value) nounwind {
-; CHECK-LABEL: @cttz_undef_vec(
-; CHECK-NEXT:    ret <2 x i32> undef
+define <2 x i32> @cttz_poison_vec(<2 x i32> %Value) {
+; CHECK-LABEL: @cttz_poison_vec(
+; CHECK-NEXT:    ret <2 x i32> poison
 ;
   %cttz = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> zeroinitializer, i1 true)
   ret <2 x i32> %cttz
 }
 
-define i32 @cttz_make_undef(i32 %a) {
-; CHECK-LABEL: @cttz_make_undef(
+define i32 @cttz_no_zero(i32 %a) {
+; CHECK-LABEL: @cttz_no_zero(
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[A:%.*]], 8
 ; CHECK-NEXT:    [[CTTZ:%.*]] = tail call i32 @llvm.cttz.i32(i32 [[OR]], i1 true), !range [[RNG3:![0-9]+]]
 ; CHECK-NEXT:    ret i32 [[CTTZ]]
 ;
   %or = or i32 %a, 8
   %cttz = tail call i32 @llvm.cttz.i32(i32 %or, i1 false)
   ret i32 %cttz
 }
 
-define <2 x i32> @cttz_make_undef_vec(<2 x i32> %a) {
-; CHECK-LABEL: @cttz_make_undef_vec(
+define <2 x i32> @cttz_no_zero_vec(<2 x i32> %a) {
+; CHECK-LABEL: @cttz_no_zero_vec(
 ; CHECK-NEXT:    [[OR:%.*]] = or <2 x i32> [[A:%.*]], <i32 8, i32 8>
 ; CHECK-NEXT:    [[CTTZ:%.*]] = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> [[OR]], i1 true)
 ; CHECK-NEXT:    ret <2 x i32> [[CTTZ]]
 ;
   %or = or <2 x i32> %a, <i32 8, i32 8>
   %cttz = tail call <2 x i32> @llvm.cttz.v2i32(<2 x i32> %or, i1 false)
   ret <2 x i32> %cttz
 }

llvm/test/Transforms/InstSimplify/ConstProp/bitcount.ll

 define i32 @cttz_zero_defined() {
 ; CHECK-LABEL: @cttz_zero_defined(
 ; CHECK-NEXT:    ret i32 32
 ;
   %x = call i32 @llvm.cttz.i32(i32 0, i1 false)
   ret i32 %x
 }
 
-define i32 @cttz_zero_undefined() {
-; CHECK-LABEL: @cttz_zero_undefined(
-; CHECK-NEXT:    ret i32 undef
+define i32 @cttz_zero_is_poison() {
+; CHECK-LABEL: @cttz_zero_is_poison(
+; CHECK-NEXT:    ret i32 poison
 ;
   %x = call i32 @llvm.cttz.i32(i32 0, i1 true)
   ret i32 %x
 }
 
 define i33 @ctlz_zero_defined() {
 ; CHECK-LABEL: @ctlz_zero_defined(
 ; CHECK-NEXT:    ret i33 33
 ;
   %x = call i33 @llvm.ctlz.i33(i33 0, i1 false)
   ret i33 %x
 }
 
-define i33 @ctlz_zero_undefined() {
-; CHECK-LABEL: @ctlz_zero_undefined(
-; CHECK-NEXT:    ret i33 undef
+define i33 @ctlz_zero_is_poison() {
+; CHECK-LABEL: @ctlz_zero_is_poison(
+; CHECK-NEXT:    ret i33 poison
 ;
   %x = call i33 @llvm.ctlz.i33(i33 0, i1 true)
   ret i33 %x
 }
 
 define i31 @ctpop_undef() {
 ; CHECK-LABEL: @ctpop_undef(
 ; CHECK-NEXT:    ret i31 0
 ;
   %x = call i31 @llvm.ctpop.i31(i31 undef)
   ret i31 %x
 }
 
 define i32 @cttz_undef_defined() {
 ; CHECK-LABEL: @cttz_undef_defined(
 ; CHECK-NEXT:    ret i32 0
 ;
   %x = call i32 @llvm.cttz.i32(i32 undef, i1 false)
   ret i32 %x
 }
 
-define i32 @cttz_undef_undefined() {
-; CHECK-LABEL: @cttz_undef_undefined(
-; CHECK-NEXT:    ret i32 undef
+define i32 @cttz_undef_zero_is_poison() {
+; CHECK-LABEL: @cttz_undef_zero_is_poison(
+; CHECK-NEXT:    ret i32 poison
 ;
   %x = call i32 @llvm.cttz.i32(i32 undef, i1 true)
   ret i32 %x
 }
 
 define i33 @ctlz_undef_defined() {
 ; CHECK-LABEL: @ctlz_undef_defined(
 ; CHECK-NEXT:    ret i33 0
 ;
   %x = call i33 @llvm.ctlz.i33(i33 undef, i1 false)
   ret i33 %x
 }
 
-define i33 @ctlz_undef_undefined() {
-; CHECK-LABEL: @ctlz_undef_undefined(
-; CHECK-NEXT:    ret i33 undef
+define i33 @ctlz_undef_zero_is_poison() {
+; CHECK-LABEL: @ctlz_undef_zero_is_poison(
+; CHECK-NEXT:    ret i33 poison
 ;
   %x = call i33 @llvm.ctlz.i33(i33 undef, i1 true)
   ret i33 %x
 }
 
 define <2 x i31> @ctpop_vector() {
 ; CHECK-LABEL: @ctpop_vector(
 ; CHECK-NEXT:    ret <2 x i31> <i31 8, i31 1>

 define <2 x i32> @cttz_vector_undef_defined() {
 ; CHECK-LABEL: @cttz_vector_undef_defined(
 ; CHECK-NEXT:    ret <2 x i32> <i32 32, i32 0>
 ;
   %x = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> <i32 0, i32 undef>, i1 false)
   ret <2 x i32> %x
 }
 
-define <2 x i32> @cttz_vector_undef_undefined() {
-; CHECK-LABEL: @cttz_vector_undef_undefined(
-; CHECK-NEXT:    ret <2 x i32> undef
+define <2 x i32> @cttz_vector_undef_zero_is_poison() {
+; CHECK-LABEL: @cttz_vector_undef_zero_is_poison(
+; CHECK-NEXT:    ret <2 x i32> poison
 ;
   %x = call <2 x i32> @llvm.cttz.v2i32(<2 x i32> <i32 0, i32 undef>, i1 true)
   ret <2 x i32> %x
 }
 
 define <2 x i33> @ctlz_vector() {
 ; CHECK-LABEL: @ctlz_vector(
 ; CHECK-NEXT:    ret <2 x i33> <i33 25, i33 28>
 ;
   %x = call <2 x i33> @llvm.ctlz.v2i33(<2 x i33> <i33 255, i33 16>, i1 true)
   ret <2 x i33> %x
 }
 
 define <2 x i33> @ctlz_vector_undef_defined() {
 ; CHECK-LABEL: @ctlz_vector_undef_defined(
 ; CHECK-NEXT:    ret <2 x i33> <i33 33, i33 0>
 ;
   %x = call <2 x i33> @llvm.ctlz.v2i33(<2 x i33> <i33 0, i33 undef>, i1 false)
   ret <2 x i33> %x
 }
 
-define <2 x i33> @ctlz_vector_undef_undefined() {
-; CHECK-LABEL: @ctlz_vector_undef_undefined(
-; CHECK-NEXT:    ret <2 x i33> undef
+define <2 x i33> @ctlz_vector_undef_zero_is_poison() {
+; CHECK-LABEL: @ctlz_vector_undef_zero_is_poison(
+; CHECK-NEXT:    ret <2 x i33> poison
 ;
   %x = call <2 x i33> @llvm.ctlz.v2i33(<2 x i33> <i33 0, i33 undef>, i1 true)
   ret <2 x i33> %x
 }