Drop the ZeroBehavior parameter from countLeadingZeros and the like (NFC)

[kazu]

Summary

This patch drops the ZeroBehavior parameter from bit counting
functions like countLeadingZeros. ZeroBehavior specifies the behavior
when the input to count{Leading,Trailing}Zeros is zero and when the
input to count{Leading,Trailing}Ones is all ones.

ZeroBehavior was first introduced on May 24, 2013 in commit
eb91eac9fb866ab1243366d2e238b9961895612d. While that patch did not
state the intention, I would guess ZeroBehavior was for performance
reasons. The x86 machines around that time required a conditional
branch to implement countLeadingZero<uint32_t> that returns the 32 on
zero:

test    edi, edi
je      .LBB0_2
bsr     eax, edi
xor     eax, 31

.LBB1_2:

mov     eax, 32

That is, we can remove the conditional branch if we don't care about
the behavior on zero.

IIUC, Intel's Haswell architecture, launched on June 4, 2013,
introduced several bit manipulation instructions, including lzcnt and
tzcnt, which eliminated the need for the conditional branch.

I think it's time to retire ZeroBehavior as its utility is very
limited. If you care about compilation speed, you should build LLVM
with an appropriate -march= to take advantage of lzcnt and tzcnt.
Even if not, modern host compilers should be able to optimize away
quite a few conditional branches because the input is often known to
be nonzero from dominating conditional branches.

[craig.topper]

llvm/include/llvm/Support/MathExtras.h

227

Note, x86 does not have an efficient instruction for find first set with zero returning -1. It will require a cmov to handle zero.

[barannikov88]

It would be nice to have comments reflecting the new behavior in the case of 0 / max value.

[foad]

In D141798#4055050, @barannikov88 wrote:

It would be nice to have comments reflecting the new behavior in the case of 0 / max value.

+1

[kazu]

Updated the patch to leave findFirstSet and findLastSet untouched.

Added a comment about the behavior on 0/max input.

[kazu]

In D141798#4055050, @barannikov88 wrote:

It would be nice to have comments reflecting the new behavior in the case of 0 / max value.

I've added comments to all four functions -- count{Leading,Trailing}{Zeros,Ones}.

llvm/include/llvm/Support/MathExtras.h

227

The new iteration of the patch leaves findFirstSet and findLastSet untouched.

[kazu]

Please take a look. Thanks!

[barannikov88]

LGTM

llvm/include/llvm/Support/MathExtras.h

227

(optional) ZB could be made a template argument and constexpr-ifed.

[arsenm]

If you care about compilation speed, you should build LLVM with an appropriate -march= to take advantage of lzcnt and tzcnt.

I think this is bad reasoning, nobody really uses -march

[jrtc27]

llvm/include/llvm/Support/MathExtras.h

224–225

A bunch of functions still have this documentation, but these are now the only possible values, so this seems redundant

[MaskRay]

ZB_Max is the strange mode that should be dropped, perhaps also ZB_Undefined.

In D141798#4064114, @arsenm wrote:

If you care about compilation speed, you should build LLVM with an appropriate -march= to take advantage of lzcnt and tzcnt.

I think this is bad reasoning, nobody really uses -march

I agree. The reason should be clarified that the lzcnt performance here really doesn't matter.
Note: https://stackoverflow.com/questions/21390165/why-does-breaking-the-output-dependency-of-lzcnt-matter lzcnt/tzcnt have false dependencies on older (pre-Skylake) Intel processors. But this doesn't really matter for LLVM, at least the minor issue does not justify keeping the weird mode ZB_Undefined.

[craig.topper]

In D141798#4064142, @MaskRay wrote:

ZB_Max is the strange mode that should be dropped, perhaps also ZB_Undefined.

In D141798#4064114, @arsenm wrote:

If you care about compilation speed, you should build LLVM with an appropriate -march= to take advantage of lzcnt and tzcnt.

I think this is bad reasoning, nobody really uses -march

I agree. The reason should be clarified that the lzcnt performance here really doesn't matter.
Note: https://stackoverflow.com/questions/21390165/why-does-breaking-the-output-dependency-of-lzcnt-matter lzcnt/tzcnt have false dependencies on older (pre-Skylake) Intel processors. But this doesn't really matter for LLVM, at least the minor issue does not justify keeping the weird mode ZB_Undefined.

I'm not sure the false dependency issue is relevant here. Without -march, compilers will emit BSR/BSF and possibly a cmov to handle the zero behavior. Both BSR/BSF always have a false dependency because the behavior for 0 input is to keep the old value of the output register. Knowing we can use lzcnt/tzcnt is always better than BSR/BSF.

If we know zero doesn't matter, without march, gcc and recent clang will always emit BSF using the encoding for TZCNT. On CPUs without TZCNT this encoding is treated as BSF. This allows us to use TZCNT at runtime if the CPU is modern even without -march being passed. We can't do the same for BSF/LZCNT because their outputs are inverted from each other.

[kazu]

I've adjusted comments for findFirstSet and findLastSet to reflect the
fact that ZeroBehavior now takes only two values -- ZB_Undefined and
ZB_Max.

[kazu]

llvm/include/llvm/Support/MathExtras.h

224–225

I've adjusted the redundant comments on two functions - findFirstSet and findLastSet.

[RKSimon]

@kazu Thanks for dealing with this!

I'd like to build on this and create llvm variants of the C++20 <bit> countl_zero/countr_zero/countl_one/countr_one template functions similar to what I did for popcount in D132407 (and have MathExtras.h reuse it) - I just wanted to ensure you hadn't already started anything similar?

[kazu]

In D141798#4065025, @RKSimon wrote:

@kazu Thanks for dealing with this!

I'd like to build on this and create llvm variants of the C++20 <bit> countl_zero/countr_zero/countl_one/countr_one template functions similar to what I did for popcount in D132407 (and have MathExtras.h reuse it) - I just wanted to ensure you hadn't already started anything similar?

I just uploaded:

https://reviews.llvm.org/D142078

llvm/include/llvm/Support/MathExtras.h

 
 namespace llvm {
 
 /// The behavior an operation has on an input of 0.
 enum ZeroBehavior {
   /// The returned value is undefined.
   ZB_Undefined,
   /// The returned value is numeric_limits<T>::max()
-  ZB_Max,
-  /// The returned value is numeric_limits<T>::digits
-  ZB_Width
+  ZB_Max
 };
 
 /// Mathematical constants.
 namespace numbers {
 // TODO: Track C++20 std::numbers.
 // TODO: Favor using the hexadecimal FP constants (requires C++17).
 constexpr double e          = 2.7182818284590452354, // (0x1.5bf0a8b145749P+1) https://oeis.org/A001113
                  egamma     = .57721566490153286061, // (0x1.2788cfc6fb619P-1) https://oeis.org/A001620
constexpr float ef          = 2.71828183F, // (0x1.5bf0a8P+1) https://oeis.org/A001113
                 inv_sqrt2f  = .707106781F, // (0x1.6a09e6P-1)
                 sqrt3f      = 1.73205081F, // (0x1.bb67aeP+0) https://oeis.org/A002194
                 inv_sqrt3f  = .577350269F, // (0x1.279a74P-1)
                 phif        = 1.61803399F; // (0x1.9e377aP+0) https://oeis.org/A001622
 } // namespace numbers
 
 namespace detail {
 template <typename T, std::size_t SizeOfT> struct TrailingZerosCounter {
-  static unsigned count(T Val, ZeroBehavior) {
+  static unsigned count(T Val) {
     if (!Val)
       return std::numeric_limits<T>::digits;
     if (Val & 0x1)
       return 0;
 
     // Bisection method.
     unsigned ZeroBits = 0;
     T Shift = std::numeric_limits<T>::digits >> 1;
     T Mask = std::numeric_limits<T>::max() >> Shift;
     while (Shift) {
       if ((Val & Mask) == 0) {
         Val >>= Shift;
         ZeroBits |= Shift;
       }
       Shift >>= 1;
       Mask >>= Shift;
     }
     return ZeroBits;
   }
 };
 
 #if defined(__GNUC__) || defined(_MSC_VER)
 template <typename T> struct TrailingZerosCounter<T, 4> {
-  static unsigned count(T Val, ZeroBehavior ZB) {
-    if (ZB != ZB_Undefined && Val == 0)
+  static unsigned count(T Val) {
+    if (Val == 0)
       return 32;
 
 #if __has_builtin(__builtin_ctz) || defined(__GNUC__)
     return __builtin_ctz(Val);
 #elif defined(_MSC_VER)
     unsigned long Index;
     _BitScanForward(&Index, Val);
     return Index;
 #endif
   }
 };
 
 #if !defined(_MSC_VER) || defined(_M_X64)
 template <typename T> struct TrailingZerosCounter<T, 8> {
-  static unsigned count(T Val, ZeroBehavior ZB) {
-    if (ZB != ZB_Undefined && Val == 0)
+  static unsigned count(T Val) {
+    if (Val == 0)
       return 64;
 
 #if __has_builtin(__builtin_ctzll) || defined(__GNUC__)
     return __builtin_ctzll(Val);
 #elif defined(_MSC_VER)
     unsigned long Index;
     _BitScanForward64(&Index, Val);
     return Index;
 #endif
   }
 };
 #endif
 #endif
 } // namespace detail
 
 /// Count number of 0's from the least significant bit to the most
 ///   stopping at the first 1.
 ///
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of 0. Only ZB_Width and ZB_Undefined are
-///   valid arguments.
-template <typename T>
-unsigned countTrailingZeros(T Val, ZeroBehavior ZB = ZB_Width) {
+/// Returns std::numeric_limits<T>::digits on an input of 0.
+template <typename T> unsigned countTrailingZeros(T Val) {
   static_assert(std::is_unsigned_v<T>,
                 "Only unsigned integral types are allowed.");
-  return llvm::detail::TrailingZerosCounter<T, sizeof(T)>::count(Val, ZB);
+  return llvm::detail::TrailingZerosCounter<T, sizeof(T)>::count(Val);
 }
 
 namespace detail {
 template <typename T, std::size_t SizeOfT> struct LeadingZerosCounter {
-  static unsigned count(T Val, ZeroBehavior) {
+  static unsigned count(T Val) {
     if (!Val)
       return std::numeric_limits<T>::digits;
 
     // Bisection method.
     unsigned ZeroBits = 0;
     for (T Shift = std::numeric_limits<T>::digits >> 1; Shift; Shift >>= 1) {
       T Tmp = Val >> Shift;
       if (Tmp)
         Val = Tmp;
       else
         ZeroBits |= Shift;
     }
     return ZeroBits;
   }
 };
 
 #if defined(__GNUC__) || defined(_MSC_VER)
 template <typename T> struct LeadingZerosCounter<T, 4> {
-  static unsigned count(T Val, ZeroBehavior ZB) {
-    if (ZB != ZB_Undefined && Val == 0)
+  static unsigned count(T Val) {
+    if (Val == 0)
       return 32;
 
 #if __has_builtin(__builtin_clz) || defined(__GNUC__)
     return __builtin_clz(Val);
 #elif defined(_MSC_VER)
     unsigned long Index;
     _BitScanReverse(&Index, Val);
     return Index ^ 31;
 #endif
   }
 };
 
 #if !defined(_MSC_VER) || defined(_M_X64)
 template <typename T> struct LeadingZerosCounter<T, 8> {
-  static unsigned count(T Val, ZeroBehavior ZB) {
-    if (ZB != ZB_Undefined && Val == 0)
+  static unsigned count(T Val) {
+    if (Val == 0)
       return 64;
 
 #if __has_builtin(__builtin_clzll) || defined(__GNUC__)
     return __builtin_clzll(Val);
 #elif defined(_MSC_VER)
     unsigned long Index;
     _BitScanReverse64(&Index, Val);
     return Index ^ 63;
 #endif
   }
 };
 #endif
 #endif
 } // namespace detail
 
 /// Count number of 0's from the most significant bit to the least
 ///   stopping at the first 1.
 ///
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of 0. Only ZB_Width and ZB_Undefined are
-///   valid arguments.
-template <typename T>
-unsigned countLeadingZeros(T Val, ZeroBehavior ZB = ZB_Width) {
+/// Returns std::numeric_limits<T>::digits on an input of 0.
+template <typename T> unsigned countLeadingZeros(T Val) {
   static_assert(std::is_unsigned_v<T>,
                 "Only unsigned integral types are allowed.");
-  return llvm::detail::LeadingZerosCounter<T, sizeof(T)>::count(Val, ZB);
+  return llvm::detail::LeadingZerosCounter<T, sizeof(T)>::count(Val);
 }
 
 /// Get the index of the first set bit starting from the least
 ///   significant bit.
 ///
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of 0. Only ZB_Max and ZB_Undefined are
-///   valid arguments.
+/// \param ZB the behavior on an input of 0.
 template <typename T> T findFirstSet(T Val, ZeroBehavior ZB = ZB_Max) {
   if (ZB == ZB_Max && Val == 0)
     return std::numeric_limits<T>::max();
 
-  return countTrailingZeros(Val, ZB_Undefined);
+  return countTrailingZeros(Val);
 }
 
 /// Create a bitmask with the N right-most bits set to 1, and all other
 /// bits set to 0.  Only unsigned types are allowed.
 template <typename T> T maskTrailingOnes(unsigned N) {
   static_assert(std::is_unsigned<T>::value, "Invalid type!");
   const unsigned Bits = CHAR_BIT * sizeof(T);
   assert(N <= Bits && "Invalid bit index");
template <typename T> T maskLeadingZeros(unsigned N) {
   return maskTrailingOnes<T>(CHAR_BIT * sizeof(T) - N);
 }
 
 /// Get the index of the last set bit starting from the least
 ///   significant bit.
 ///
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of 0. Only ZB_Max and ZB_Undefined are
-///   valid arguments.
+/// \param ZB the behavior on an input of 0.
 template <typename T> T findLastSet(T Val, ZeroBehavior ZB = ZB_Max) {
   if (ZB == ZB_Max && Val == 0)
     return std::numeric_limits<T>::max();
 
   // Use ^ instead of - because both gcc and llvm can remove the associated ^
   // in the __builtin_clz intrinsic on x86.
-  return countLeadingZeros(Val, ZB_Undefined) ^
-         (std::numeric_limits<T>::digits - 1);
+  return countLeadingZeros(Val) ^ (std::numeric_limits<T>::digits - 1);
 }
 
 /// Macro compressed bit reversal table for 256 bits.
 ///
 /// http://graphics.stanford.edu/~seander/bithacks.html#BitReverseTable
 static const unsigned char BitReverseTable256[256] = {
 #define R2(n) n, n + 2 * 64, n + 1 * 64, n + 3 * 64
 #define R4(n) R2(n), R2(n + 2 * 16), R2(n + 1 * 16), R2(n + 3 * 16)

 }
 
 /// Count the number of ones from the most significant bit to the first
 /// zero bit.
 ///
 /// Ex. countLeadingOnes(0xFF0FFF00) == 8.
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of all ones. Only ZB_Width and
-/// ZB_Undefined are valid arguments.
-template <typename T>
-unsigned countLeadingOnes(T Value, ZeroBehavior ZB = ZB_Width) {
+/// Returns std::numeric_limits<T>::digits on an input of all ones.
+template <typename T> unsigned countLeadingOnes(T Value) {
   static_assert(std::is_unsigned_v<T>,
                 "Only unsigned integral types are allowed.");
-  return countLeadingZeros<T>(~Value, ZB);
+  return countLeadingZeros<T>(~Value);
 }
 
 /// Count the number of ones from the least significant bit to the first
 /// zero bit.
 ///
 /// Ex. countTrailingOnes(0x00FF00FF) == 8.
 /// Only unsigned integral types are allowed.
 ///
-/// \param ZB the behavior on an input of all ones. Only ZB_Width and
-/// ZB_Undefined are valid arguments.
-template <typename T>
-unsigned countTrailingOnes(T Value, ZeroBehavior ZB = ZB_Width) {
+/// Returns std::numeric_limits<T>::digits on an input of all ones.
+template <typename T> unsigned countTrailingOnes(T Value) {
   static_assert(std::is_unsigned_v<T>,
                 "Only unsigned integral types are allowed.");
-  return countTrailingZeros<T>(~Value, ZB);
+  return countTrailingZeros<T>(~Value);
 }
 
 /// Count the number of set bits in a value.
 /// Ex. countPopulation(0xF000F000) = 8
 /// Returns 0 if the word is zero.
 template <typename T>
 inline unsigned countPopulation(T Value) {
   static_assert(std::is_unsigned_v<T>,
constexpr inline uint64_t NextPowerOf2(uint64_t A) {
   A |= (A >> 32);
   return A + 1;
 }
 
 /// Returns the power of two which is less than or equal to the given value.
 /// Essentially, it is a floor operation across the domain of powers of two.
 inline uint64_t PowerOf2Floor(uint64_t A) {
   if (!A) return 0;
-  return 1ull << (63 - countLeadingZeros(A, ZB_Undefined));
+  return 1ull << (63 - countLeadingZeros(A));
 }
 
 /// Returns the power of two which is greater than or equal to the given value.
 /// Essentially, it is a ceil operation across the domain of powers of two.
 inline uint64_t PowerOf2Ceil(uint64_t A) {
   if (!A)
     return 0;
   return NextPowerOf2(A - 1);

llvm/lib/Target/AMDGPU/AMDGPUCallLowering.cpp

if (Subtarget.isAmdPalOS()) {
       // a bit is set in PSInputAddr but not PSInputEnable is where the frontend
       // set up an input arg for a particular interpolation mode, but nothing
       // uses that input arg. Really we should have an earlier pass that removes
       // such an arg.)
       unsigned PsInputBits = Info->getPSInputAddr() & Info->getPSInputEnable();
       if ((PsInputBits & 0x7F) == 0 ||
           ((PsInputBits & 0xF) == 0 &&
            (PsInputBits >> 11 & 1)))
-        Info->markPSInputEnabled(
-          countTrailingZeros(Info->getPSInputAddr(), ZB_Undefined));
+        Info->markPSInputEnabled(countTrailingZeros(Info->getPSInputAddr()));
     }
   }
 
   const SITargetLowering &TLI = *getTLI<SITargetLowering>();
   CCAssignFn *AssignFn = TLI.CCAssignFnForCall(CC, F.isVarArg());
 
   if (!MBB.empty())
     B.setInstr(*MBB.begin());

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

if (Subtarget->isAmdPalOS()) {
       // workaround to PSInputAddr and PSInputEnable together.  (The case where
       // a bit is set in PSInputAddr but not PSInputEnable is where the
       // frontend set up an input arg for a particular interpolation mode, but
       // nothing uses that input arg. Really we should have an earlier pass
       // that removes such an arg.)
       unsigned PsInputBits = Info->getPSInputAddr() & Info->getPSInputEnable();
       if ((PsInputBits & 0x7F) == 0 ||
           ((PsInputBits & 0xF) == 0 && (PsInputBits >> 11 & 1)))
-        Info->markPSInputEnabled(
-            countTrailingZeros(Info->getPSInputAddr(), ZB_Undefined));
+        Info->markPSInputEnabled(countTrailingZeros(Info->getPSInputAddr()));
     }
   } else if (IsKernel) {
     assert(Info->hasWorkGroupIDX() && Info->hasWorkItemIDX());
   } else {
     Splits.append(Ins.begin(), Ins.end());
   }
 
   if (IsEntryFunc) {

llvm/lib/Target/RISCV/MCTargetDesc/RISCVMatInt.cpp

 InstSeq generateInstSeq(int64_t Val, const FeatureBitset &ActiveFeatures) {
   RISCVMatInt::InstSeq Res;
   generateInstSeqImpl(Val, ActiveFeatures, Res);
 
   // If the low 12 bits are non-zero, the first expansion may end with an ADDI
   // or ADDIW. If there are trailing zeros, try generating a sign extended
   // constant with no trailing zeros and use a final SLLI to restore them.
   if ((Val & 0xfff) != 0 && (Val & 1) == 0 && Res.size() >= 2) {
-    unsigned TrailingZeros = countTrailingZeros((uint64_t)Val, ZB_Undefined);
+    unsigned TrailingZeros = countTrailingZeros((uint64_t)Val);
     int64_t ShiftedVal = Val >> TrailingZeros;
     // If we can use C.LI+C.SLLI instead of LUI+ADDI(W) prefer that since
     // its more compressible. But only if LUI+ADDI(W) isn't fusable.
     // NOTE: We don't check for C extension to minimize differences in generated
     // code.
     bool IsShiftedCompressible =
               isInt<6>(ShiftedVal) && !ActiveFeatures[RISCV::TuneLUIADDIFusion];
     RISCVMatInt::InstSeq TmpSeq;
     generateInstSeqImpl(ShiftedVal, ActiveFeatures, TmpSeq);
     TmpSeq.emplace_back(RISCV::SLLI, TrailingZeros);
 
     // Keep the new sequence if it is an improvement.
     if (TmpSeq.size() < Res.size() || IsShiftedCompressible)
       Res = TmpSeq;
   }
 
   // If the constant is positive we might be able to generate a shifted constant
   // with no leading zeros and use a final SRLI to restore them.
   if (Val > 0 && Res.size() > 2) {
     assert(ActiveFeatures[RISCV::Feature64Bit] &&
            "Expected RV32 to only need 2 instructions");
-    unsigned LeadingZeros = countLeadingZeros((uint64_t)Val, ZB_Undefined);
+    unsigned LeadingZeros = countLeadingZeros((uint64_t)Val);
     uint64_t ShiftedVal = (uint64_t)Val << LeadingZeros;
     // Fill in the bits that will be shifted out with 1s. An example where this
     // helps is trailing one masks with 32 or more ones. This will generate
     // ADDI -1 and an SRLI.
     ShiftedVal |= maskTrailingOnes<uint64_t>(LeadingZeros);
 
     RISCVMatInt::InstSeq TmpSeq;
     generateInstSeqImpl(ShiftedVal, ActiveFeatures, TmpSeq);

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

for (uint64_t &Offset : Offsets) {
     Mask |= Offset;
   }
 
   BitSetInfo BSI;
   BSI.ByteOffset = Min;
 
   BSI.AlignLog2 = 0;
   if (Mask != 0)
-    BSI.AlignLog2 = countTrailingZeros(Mask, ZB_Undefined);
+    BSI.AlignLog2 = countTrailingZeros(Mask);
 
   // Build the compressed bitset while normalizing the offsets against the
   // computed alignment.
   BSI.BitSize = ((Max - Min) >> BSI.AlignLog2) + 1;
   for (uint64_t Offset : Offsets) {
     Offset >>= BSI.AlignLog2;
     BSI.Bits.insert(Offset);
   }

llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp

wholeprogramdevirt::findLowestOffset(ArrayRef<VirtualCallTarget> Targets,
   if (Size == 1) {
     // Find a free bit in each member of Used.
     for (unsigned I = 0;; ++I) {
       uint8_t BitsUsed = 0;
       for (auto &&B : Used)
         if (I < B.size())
           BitsUsed |= B[I];
       if (BitsUsed != 0xff)
-        return (MinByte + I) * 8 +
-               countTrailingZeros(uint8_t(~BitsUsed), ZB_Undefined);
+        return (MinByte + I) * 8 + countTrailingZeros(uint8_t(~BitsUsed));
     }
   } else {
     // Find a free (Size/8) byte region in each member of Used.
     // FIXME: see if alignment helps.
     for (unsigned I = 0;; ++I) {
       for (auto &&B : Used) {
         unsigned Byte = 0;
         while ((I + Byte) < B.size() && Byte < (Size / 8)) {

mlir/lib/Bytecode/Reader/BytecodeReader.cpp

private:
   /// NOTE: This method is marked noinline to avoid pessimizing the common case
   /// of single byte encoding.
   LLVM_ATTRIBUTE_NOINLINE LogicalResult parseMultiByteVarInt(uint64_t &result) {
     // Count the number of trailing zeros in the marker byte, this indicates the
     // number of trailing bytes that are part of the value. We use `uint32_t`
     // here because we only care about the first byte, and so that be actually
     // get ctz intrinsic calls when possible (the `uint8_t` overload uses a loop
     // implementation).
-    uint32_t numBytes =
-        llvm::countTrailingZeros<uint32_t>(result, llvm::ZB_Undefined);
+    uint32_t numBytes = llvm::countTrailingZeros<uint32_t>(result);
     assert(numBytes > 0 && numBytes <= 7 &&
            "unexpected number of trailing zeros in varint encoding");
 
     // Parse in the remaining bytes of the value.
     if (failed(parseBytes(numBytes, reinterpret_cast<uint8_t *>(&result) + 1)))
       return failure();
 
     // Shift out the low-order bits that were used to mark how the value was