[RISCV] Lower vector CTLZ_ZERO_UNDEF/CTTZ_ZERO_UNDEF by converting to FP and extracting the exponent.

[craig.topper]

Summary

If we have a large enough floating point type that can exactly
represent the integer value, we can convert the value to FP and
use the exponent to calculate the leading/trailing zeros.

The exponent will contain log2 of the value plus the exponent bias.
We can then remove the bias and convert from log2 to leading/trailing
zeros.

This doesn't work for zero since the exponent of zero is zero so we
can only do this for CTLZ_ZERO_UNDEF/CTTZ_ZERO_UNDEF. If we need
a value for zero we can use a vmseq and a vmerge to handle it.

We need to be careful to make sure the floating point type is legal.
If it isn't we'll continue using the integer expansion. We could split the vector
and concatenate the results but that needs some additional work and evaluation.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

7085–7086

This is needed to get VSELECT for vectors.

7134–7135

This isn't tested by this patch but makes it consistent with CTLZ.

[craig.topper]

Fix a bad comment. Use Src SDValue instead of calling getOperand. clang-format.

[craig.topper]

Add support for CTTZ_ZERO_UNDEF as well.

[craig.topper]

Rebase after fixing scalarization of ctlz/cttz.

[craig.topper]

Remove splitting if FP type isn't legal. Some cases it generated worse code. Need to re-evaluate.

I had to make an additional change to TargetLowering.cpp to prevent an infinite loop.

[craig.topper]

Fix bad check prefixes

[craig.topper]

Rebase

[craig.topper]

Ping

[frasercrmck]

Sorry for not being very prompt on this one. I'll try and think about it in more detail later.

In the meantime, what is it about this that doesn't/shouldn't apply to scalable vectors?

[craig.topper]

Support scalable vectors.
Check legal types in RISCVTargetLowering constructor. Fixes unneeded vmerges.

[HsiangKai]

It makes sense to me. I'll let @frasercrmck chime in.

[frasercrmck]

LGTM too other than nits. Thanks, @craig.topper!

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

634

Could maybe update this comment to use the actual ISD names for both nodes we're dealing with.

864

Same here as above.

2364

This comment seems stale since we're no longer falling back.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (Node->getOpcode() == ISD::CTLZ_ZERO_UNDEF &&
     return DAG.getNode(ISD::CTLZ, dl, VT, Op);
 
   // If the ZERO_UNDEF version is supported use that and handle the zero case.
   if (isOperationLegalOrCustom(ISD::CTLZ_ZERO_UNDEF, VT)) {
     EVT SetCCVT =
         getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
     SDValue CTLZ = DAG.getNode(ISD::CTLZ_ZERO_UNDEF, dl, VT, Op);
     SDValue Zero = DAG.getConstant(0, dl, VT);
     SDValue SrcIsZero = DAG.getSetCC(dl, SetCCVT, Op, Zero, ISD::SETEQ);
-    return DAG.getNode(ISD::SELECT, dl, VT, SrcIsZero,
+    return DAG.getSelect(dl, VT, SrcIsZero,
                          DAG.getConstant(NumBitsPerElt, dl, VT), CTLZ);
   }
 
   // Only expand vector types if we have the appropriate vector bit operations.
   // This includes the operations needed to expand CTPOP if it isn't supported.
   if (VT.isVector() && (!isPowerOf2_32(NumBitsPerElt) ||
                         (!isOperationLegalOrCustom(ISD::CTPOP, VT) &&
                          !canExpandVectorCTPOP(*this, VT)) ||
                         !isOperationLegalOrCustom(ISD::SRL, VT) ||
if (Node->getOpcode() == ISD::CTTZ_ZERO_UNDEF &&
     return DAG.getNode(ISD::CTTZ, dl, VT, Op);
 
   // If the ZERO_UNDEF version is supported use that and handle the zero case.
   if (isOperationLegalOrCustom(ISD::CTTZ_ZERO_UNDEF, VT)) {
     EVT SetCCVT =
         getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
     SDValue CTTZ = DAG.getNode(ISD::CTTZ_ZERO_UNDEF, dl, VT, Op);
     SDValue Zero = DAG.getConstant(0, dl, VT);
     SDValue SrcIsZero = DAG.getSetCC(dl, SetCCVT, Op, Zero, ISD::SETEQ);
-    return DAG.getNode(ISD::SELECT, dl, VT, SrcIsZero,
+    return DAG.getSelect(dl, VT, SrcIsZero,
                          DAG.getConstant(NumBitsPerElt, dl, VT), CTTZ);
   }
 
   // Only expand vector types if we have the appropriate vector bit operations.
   // This includes the operations needed to expand CTPOP if it isn't supported.
   if (VT.isVector() && (!isPowerOf2_32(NumBitsPerElt) ||
                         (!isOperationLegalOrCustom(ISD::CTPOP, VT) &&
                          !isOperationLegalOrCustom(ISD::CTLZ, VT) &&
                          !canExpandVectorCTPOP(*this, VT)) ||

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

for (MVT VT : IntVecVTs) {
       setOperationAction(ISD::VECTOR_REVERSE, VT, Custom);
 
       for (MVT OtherVT : MVT::integer_scalable_vector_valuetypes()) {
         setTruncStoreAction(VT, OtherVT, Expand);
         setLoadExtAction(ISD::EXTLOAD, OtherVT, VT, Expand);
         setLoadExtAction(ISD::SEXTLOAD, OtherVT, VT, Expand);
         setLoadExtAction(ISD::ZEXTLOAD, OtherVT, VT, Expand);
       }
+
+      // Lower CTLZ_ZERO_UNDEF and CTTZ_ZERO_UNDEF if we have a floating point
+      // type that can represent the value exactly.
+      if (VT.getVectorElementType() != MVT::i64) {
+        MVT FloatEltVT =
+            VT.getVectorElementType() == MVT::i32 ? MVT::f64 : MVT::f32;
+        EVT FloatVT = MVT::getVectorVT(FloatEltVT, VT.getVectorElementCount());
+        if (isTypeLegal(FloatVT)) {
+          setOperationAction(ISD::CTLZ_ZERO_UNDEF, VT, Custom);
+          setOperationAction(ISD::CTTZ_ZERO_UNDEF, VT, Custom);
+        }
+      }
     }
 
     // Expand various CCs to best match the RVV ISA, which natively supports UNE
     // but no other unordered comparisons, and supports all ordered comparisons
     // except ONE. Additionally, we expand GT,OGT,GE,OGE for optimization
     // purposes; they are expanded to their swapped-operand CCs (LT,OLT,LE,OLE),
     // and we pattern-match those back to the "original", swapping operands once
     // more. This way we catch both operations and both "vf" and "fv" forms with
if (Subtarget.useRVVForFixedLengthVectors()) {
         setOperationAction(ISD::VECREDUCE_ADD, VT, Custom);
         setOperationAction(ISD::VECREDUCE_SMAX, VT, Custom);
         setOperationAction(ISD::VECREDUCE_SMIN, VT, Custom);
         setOperationAction(ISD::VECREDUCE_UMAX, VT, Custom);
         setOperationAction(ISD::VECREDUCE_UMIN, VT, Custom);
 
         for (unsigned VPOpc : IntegerVPOps)
           setOperationAction(VPOpc, VT, Custom);
+
+        // Lower CTLZ_ZERO_UNDEF and CTTZ_ZERO_UNDEF if we have a floating point
+        // type that can represent the value exactly.
+        if (VT.getVectorElementType() != MVT::i64) {
+          MVT FloatEltVT =
+              VT.getVectorElementType() == MVT::i32 ? MVT::f64 : MVT::f32;
+          EVT FloatVT =
+              MVT::getVectorVT(FloatEltVT, VT.getVectorElementCount());
+          if (isTypeLegal(FloatVT)) {
+            setOperationAction(ISD::CTLZ_ZERO_UNDEF, VT, Custom);
+            setOperationAction(ISD::CTTZ_ZERO_UNDEF, VT, Custom);
+          }
+        }
       }
 
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes()) {
         if (!useRVVForFixedLengthVectorVT(VT))
           continue;
 
         // By default everything must be expanded.
         for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op)
static SDValue getRVVFPExtendOrRound(SDValue Op, MVT VT, MVT ContainerVT,
   SDValue Mask, VL;
   std::tie(Mask, VL) = getDefaultVLOps(VT, ContainerVT, DL, DAG, Subtarget);
   unsigned RVVOpc = ContainerVT.bitsGT(Op.getSimpleValueType())
                         ? RISCVISD::FP_EXTEND_VL
                         : RISCVISD::FP_ROUND_VL;
   return DAG.getNode(RVVOpc, DL, ContainerVT, Op, Mask, VL);
 }
 
+// Lower CTLZ_ZERO_UNDEF or CTTZ_ZERO_UNDEF by converting to FP and extracting
+// the exponent.
+static SDValue lowerCTLZ_CTTZ_ZERO_UNDEF(SDValue Op, SelectionDAG &DAG) {
+  MVT VT = Op.getSimpleValueType();
+  unsigned EltSize = VT.getScalarSizeInBits();
+  SDValue Src = Op.getOperand(0);
+  SDLoc DL(Op);
+
+  // We need a FP type that can represent the value.
+  // TODO: Use f16 for i8 when possible?
+  MVT FloatEltVT = EltSize == 32 ? MVT::f64 : MVT::f32;
+  MVT FloatVT = MVT::getVectorVT(FloatEltVT, VT.getVectorElementCount());
+
+  // Legal types should have been checked in the RISCVTargetLowering
+  // constructor.
+  // TODO: Splitting may make sense in some cases.
+  assert(DAG.getTargetLoweringInfo().isTypeLegal(FloatVT) &&
+         "Expected legal float type!");
+
+  // For CTTZ_ZERO_UNDEF, we need to extract the lowest set bit using X & -X.
+  // The trailing zero count is equal to log2 of this single bit value.
+  if (Op.getOpcode() == ISD::CTTZ_ZERO_UNDEF) {
+    SDValue Neg =
+        DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(0, DL, VT), Src);
+    Src = DAG.getNode(ISD::AND, DL, VT, Src, Neg);
+  }
+
+  // We have a legal FP type, convert to it.
+  SDValue FloatVal = DAG.getNode(ISD::UINT_TO_FP, DL, FloatVT, Src);
+  // Bitcast to integer and shift the exponent to the LSB.
+  EVT IntVT = FloatVT.changeVectorElementTypeToInteger();
+  SDValue Bitcast = DAG.getBitcast(IntVT, FloatVal);
+  unsigned ShiftAmt = FloatEltVT == MVT::f64 ? 52 : 23;
+  SDValue Shift = DAG.getNode(ISD::SRL, DL, IntVT, Bitcast,
+                              DAG.getConstant(ShiftAmt, DL, IntVT));
+  // Truncate back to original type to allow vnsrl.
+  SDValue Trunc = DAG.getNode(ISD::TRUNCATE, DL, VT, Shift);
+  // The exponent contains log2 of the value in biased form.
+  unsigned ExponentBias = FloatEltVT == MVT::f64 ? 1023 : 127;
+
+  // For trailing zeros, we just need to subtract the bias.
+  if (Op.getOpcode() == ISD::CTTZ_ZERO_UNDEF)
+    return DAG.getNode(ISD::SUB, DL, VT, Trunc,
+                       DAG.getConstant(ExponentBias, DL, VT));
+
+  // For leading zeros, we need to remove the bias and convert from log2 to
+  // leading zeros. We can do this by subtracting from (Bias + (EltSize - 1)).
+  unsigned Adjust = ExponentBias + (EltSize - 1);
+  return DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(Adjust, DL, VT), Trunc);
+}
+
 // While RVV has alignment restrictions, we should always be able to load as a
 // legal equivalently-sized byte-typed vector instead. This method is
 // responsible for re-expressing a ISD::LOAD via a correctly-aligned type. If
 // the load is already correctly-aligned, it returns SDValue().
 SDValue RISCVTargetLowering::expandUnalignedRVVLoad(SDValue Op,
                                                     SelectionDAG &DAG) const {
   auto *Load = cast<LoadSDNode>(Op);
   assert(Load && Load->getMemoryVT().isVector() && "Expected vector load");
SDValue RISCVTargetLowering::LowerOperation(SDValue Op,
   case ISD::UMAX:
     return lowerToScalableOp(Op, DAG, RISCVISD::UMAX_VL);
   case ISD::FMINNUM:
     return lowerToScalableOp(Op, DAG, RISCVISD::FMINNUM_VL);
   case ISD::FMAXNUM:
     return lowerToScalableOp(Op, DAG, RISCVISD::FMAXNUM_VL);
   case ISD::ABS:
     return lowerABS(Op, DAG);
+  case ISD::CTLZ_ZERO_UNDEF:
+  case ISD::CTTZ_ZERO_UNDEF:
+    return lowerCTLZ_CTTZ_ZERO_UNDEF(Op, DAG);
   case ISD::VSELECT:
     return lowerFixedLengthVectorSelectToRVV(Op, DAG);
   case ISD::FCOPYSIGN:
     return lowerFixedLengthVectorFCOPYSIGNToRVV(Op, DAG);
   case ISD::MGATHER:
   case ISD::VP_GATHER:
     return lowerMaskedGather(Op, DAG);
   case ISD::MSCATTER:

llvm/test/CodeGen/RISCV/rvv/ctlz-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=riscv32 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32
-; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32,RV32I
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64,RV64I
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-v,+d -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32,RV32D
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v,+d -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64,RV64D
 
 define <vscale x 1 x i8> @ctlz_nxv1i8(<vscale x 1 x i8> %va) {
-; CHECK-LABEL: ctlz_nxv1i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv1i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv1i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv1i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v9, v9
+; RV32D-NEXT:    vsrl.vi v9, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    vrsub.vx v8, v9, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv1i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v9, v9
+; RV64D-NEXT:    vsrl.vi v9, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    vrsub.vx v8, v9, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i8> @llvm.ctlz.nxv1i8(<vscale x 1 x i8> %va, i1 false)
   ret <vscale x 1 x i8> %a
 }
 declare <vscale x 1 x i8> @llvm.ctlz.nxv1i8(<vscale x 1 x i8>, i1)
 
 define <vscale x 2 x i8> @ctlz_nxv2i8(<vscale x 2 x i8> %va) {
-; CHECK-LABEL: ctlz_nxv2i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv2i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv2i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv2i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v9, v9
+; RV32D-NEXT:    vsrl.vi v9, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    vrsub.vx v8, v9, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv2i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v9, v9
+; RV64D-NEXT:    vsrl.vi v9, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    vrsub.vx v8, v9, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i8> @llvm.ctlz.nxv2i8(<vscale x 2 x i8> %va, i1 false)
   ret <vscale x 2 x i8> %a
 }
 declare <vscale x 2 x i8> @llvm.ctlz.nxv2i8(<vscale x 2 x i8>, i1)
 
 define <vscale x 4 x i8> @ctlz_nxv4i8(<vscale x 4 x i8> %va) {
-; CHECK-LABEL: ctlz_nxv4i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv4i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv4i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv4i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v10, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v10, v10
+; RV32D-NEXT:    vsrl.vi v10, v10, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v10, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    vrsub.vx v8, v9, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv4i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v10, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v10, v10
+; RV64D-NEXT:    vsrl.vi v10, v10, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v10, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    vrsub.vx v8, v9, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i8> @llvm.ctlz.nxv4i8(<vscale x 4 x i8> %va, i1 false)
   ret <vscale x 4 x i8> %a
 }
 declare <vscale x 4 x i8> @llvm.ctlz.nxv4i8(<vscale x 4 x i8>, i1)
 
 define <vscale x 8 x i8> @ctlz_nxv8i8(<vscale x 8 x i8> %va) {
-; CHECK-LABEL: ctlz_nxv8i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv8i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv8i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv8i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vzext.vf4 v12, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v12, v12
+; RV32D-NEXT:    vsrl.vi v12, v12, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v12, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v10, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    vrsub.vx v8, v9, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv8i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vzext.vf4 v12, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v12, v12
+; RV64D-NEXT:    vsrl.vi v12, v12, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v12, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v10, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    vrsub.vx v8, v9, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i8> @llvm.ctlz.nxv8i8(<vscale x 8 x i8> %va, i1 false)
   ret <vscale x 8 x i8> %a
 }
 declare <vscale x 8 x i8> @llvm.ctlz.nxv8i8(<vscale x 8 x i8>, i1)
 
 define <vscale x 16 x i8> @ctlz_nxv16i8(<vscale x 16 x i8> %va) {
-; CHECK-LABEL: ctlz_nxv16i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v10, v10, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v10
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v10, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v10, v8
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v10
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv16i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv16i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv16i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vzext.vf4 v16, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v16, v16
+; RV32D-NEXT:    vsrl.vi v16, v16, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v16, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v12, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    vrsub.vx v8, v10, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv16i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vzext.vf4 v16, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v16, v16
+; RV64D-NEXT:    vsrl.vi v16, v16, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v16, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v12, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    vrsub.vx v8, v10, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i8> @llvm.ctlz.nxv16i8(<vscale x 16 x i8> %va, i1 false)
   ret <vscale x 16 x i8> %a
 }
 declare <vscale x 16 x i8> @llvm.ctlz.nxv16i8(<vscale x 16 x i8>, i1)
 
 define <vscale x 32 x i8> @ctlz_nxv32i8(<vscale x 32 x i8> %va) {
 ; CHECK-LABEL: ctlz_nxv32i8:
 ; CHECK:       # %bb.0:

 ; CHECK-NEXT:    vand.vi v8, v8, 15
 ; CHECK-NEXT:    ret
   %a = call <vscale x 64 x i8> @llvm.ctlz.nxv64i8(<vscale x 64 x i8> %va, i1 false)
   ret <vscale x 64 x i8> %a
 }
 declare <vscale x 64 x i8> @llvm.ctlz.nxv64i8(<vscale x 64 x i8>, i1)
 
 define <vscale x 1 x i16> @ctlz_nxv1i16(<vscale x 1 x i16> %va) {
-; RV32-LABEL: ctlz_nxv1i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv1i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv1i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv1i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv1i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vsrl.vi v9, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv1i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vsrl.vi v9, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v9, v9, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i16> @llvm.ctlz.nxv1i16(<vscale x 1 x i16> %va, i1 false)
   ret <vscale x 1 x i16> %a
 }
 declare <vscale x 1 x i16> @llvm.ctlz.nxv1i16(<vscale x 1 x i16>, i1)
 
 define <vscale x 2 x i16> @ctlz_nxv2i16(<vscale x 2 x i16> %va) {
-; RV32-LABEL: ctlz_nxv2i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv2i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv2i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv2i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv2i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vsrl.vi v9, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv2i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vsrl.vi v9, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v9, v9, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i16> @llvm.ctlz.nxv2i16(<vscale x 2 x i16> %va, i1 false)
   ret <vscale x 2 x i16> %a
 }
 declare <vscale x 2 x i16> @llvm.ctlz.nxv2i16(<vscale x 2 x i16>, i1)
 
 define <vscale x 4 x i16> @ctlz_nxv4i16(<vscale x 4 x i16> %va) {
-; RV32-LABEL: ctlz_nxv4i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv4i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv4i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv4i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv4i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vsrl.vi v10, v10, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v10, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv4i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vsrl.vi v10, v10, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v10, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v9, v9, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i16> @llvm.ctlz.nxv4i16(<vscale x 4 x i16> %va, i1 false)
   ret <vscale x 4 x i16> %a
 }
 declare <vscale x 4 x i16> @llvm.ctlz.nxv4i16(<vscale x 4 x i16>, i1)
 
 define <vscale x 8 x i16> @ctlz_nxv8i16(<vscale x 8 x i16> %va) {
-; RV32-LABEL: ctlz_nxv8i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv8i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv8i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv8i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv8i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vsrl.vi v12, v12, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v12, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v10, v10, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v10, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv8i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vsrl.vi v12, v12, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v12, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v10, v10, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v10, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i16> @llvm.ctlz.nxv8i16(<vscale x 8 x i16> %va, i1 false)
   ret <vscale x 8 x i16> %a
 }
 declare <vscale x 8 x i16> @llvm.ctlz.nxv8i16(<vscale x 8 x i16>, i1)
 
 define <vscale x 16 x i16> @ctlz_nxv16i16(<vscale x 16 x i16> %va) {
-; RV32-LABEL: ctlz_nxv16i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv16i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv16i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv16i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv16i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vsrl.vi v16, v16, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v16, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v12, v12, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v12, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv16i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vsrl.vi v16, v16, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v16, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v12, v12, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v12, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i16> @llvm.ctlz.nxv16i16(<vscale x 16 x i16> %va, i1 false)
   ret <vscale x 16 x i16> %a
 }
 declare <vscale x 16 x i16> @llvm.ctlz.nxv16i16(<vscale x 16 x i16>, i1)
 
 define <vscale x 32 x i16> @ctlz_nxv32i16(<vscale x 32 x i16> %va) {
 ; RV32-LABEL: ctlz_nxv32i16:
 ; RV32:       # %bb.0:

 ; RV64-NEXT:    vsrl.vi v8, v8, 8
 ; RV64-NEXT:    ret
   %a = call <vscale x 32 x i16> @llvm.ctlz.nxv32i16(<vscale x 32 x i16> %va, i1 false)
   ret <vscale x 32 x i16> %a
 }
 declare <vscale x 32 x i16> @llvm.ctlz.nxv32i16(<vscale x 32 x i16>, i1)
 
 define <vscale x 1 x i32> @ctlz_nxv1i32(<vscale x 1 x i32> %va) {
-; RV32-LABEL: ctlz_nxv1i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv1i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv1i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv1i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv1i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV32D-NEXT:    vsrl.vx v9, v9, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv1i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV64D-NEXT:    vsrl.vx v9, v9, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v9, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v9, v9, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i32> @llvm.ctlz.nxv1i32(<vscale x 1 x i32> %va, i1 false)
   ret <vscale x 1 x i32> %a
 }
 declare <vscale x 1 x i32> @llvm.ctlz.nxv1i32(<vscale x 1 x i32>, i1)
 
 define <vscale x 2 x i32> @ctlz_nxv2i32(<vscale x 2 x i32> %va) {
-; RV32-LABEL: ctlz_nxv2i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv2i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv2i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv2i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv2i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32D-NEXT:    vsrl.vx v10, v10, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v10, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv2i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV64D-NEXT:    vsrl.vx v10, v10, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v9, v10, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v9, v9, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i32> @llvm.ctlz.nxv2i32(<vscale x 2 x i32> %va, i1 false)
   ret <vscale x 2 x i32> %a
 }
 declare <vscale x 2 x i32> @llvm.ctlz.nxv2i32(<vscale x 2 x i32>, i1)
 
 define <vscale x 4 x i32> @ctlz_nxv4i32(<vscale x 4 x i32> %va) {
-; RV32-LABEL: ctlz_nxv4i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv4i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv4i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv4i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv4i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV32D-NEXT:    vsrl.vx v12, v12, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v12, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v10, v10, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v10, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv4i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV64D-NEXT:    vsrl.vx v12, v12, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v12, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v10, v10, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v10, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i32> @llvm.ctlz.nxv4i32(<vscale x 4 x i32> %va, i1 false)
   ret <vscale x 4 x i32> %a
 }
 declare <vscale x 4 x i32> @llvm.ctlz.nxv4i32(<vscale x 4 x i32>, i1)
 
 define <vscale x 8 x i32> @ctlz_nxv8i32(<vscale x 8 x i32> %va) {
-; RV32-LABEL: ctlz_nxv8i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_nxv8i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_nxv8i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_nxv8i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_nxv8i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV32D-NEXT:    vsrl.vx v16, v16, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v16, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v12, v12, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v12, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_nxv8i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV64D-NEXT:    vsrl.vx v16, v16, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v16, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v12, v12, a0
+; RV64D-NEXT:    vmseq.vi v0, v8, 0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v12, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i32> @llvm.ctlz.nxv8i32(<vscale x 8 x i32> %va, i1 false)
   ret <vscale x 8 x i32> %a
 }
 declare <vscale x 8 x i32> @llvm.ctlz.nxv8i32(<vscale x 8 x i32>, i1)
 
 define <vscale x 16 x i32> @ctlz_nxv16i32(<vscale x 16 x i32> %va) {
 ; RV32-LABEL: ctlz_nxv16i32:
 ; RV32:       # %bb.0:

 ; RV64-NEXT:    vsrl.vx v8, v8, a0
 ; RV64-NEXT:    ret
   %a = call <vscale x 8 x i64> @llvm.ctlz.nxv8i64(<vscale x 8 x i64> %va, i1 false)
   ret <vscale x 8 x i64> %a
 }
 declare <vscale x 8 x i64> @llvm.ctlz.nxv8i64(<vscale x 8 x i64>, i1)
 
 define <vscale x 1 x i8> @ctlz_zero_undef_nxv1i8(<vscale x 1 x i8> %va) {
-; CHECK-LABEL: ctlz_zero_undef_nxv1i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv1i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv1i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv1i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv1i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i8> @llvm.ctlz.nxv1i8(<vscale x 1 x i8> %va, i1 true)
   ret <vscale x 1 x i8> %a
 }
 
 define <vscale x 2 x i8> @ctlz_zero_undef_nxv2i8(<vscale x 2 x i8> %va) {
-; CHECK-LABEL: ctlz_zero_undef_nxv2i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv2i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv2i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv2i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv2i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i8> @llvm.ctlz.nxv2i8(<vscale x 2 x i8> %va, i1 true)
   ret <vscale x 2 x i8> %a
 }
 
 define <vscale x 4 x i8> @ctlz_zero_undef_nxv4i8(<vscale x 4 x i8> %va) {
-; CHECK-LABEL: ctlz_zero_undef_nxv4i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv4i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv4i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv4i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v10, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v10, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv4i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v10, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v10, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i8> @llvm.ctlz.nxv4i8(<vscale x 4 x i8> %va, i1 true)
   ret <vscale x 4 x i8> %a
 }
 
 define <vscale x 8 x i8> @ctlz_zero_undef_nxv8i8(<vscale x 8 x i8> %va) {
-; CHECK-LABEL: ctlz_zero_undef_nxv8i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v9
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv8i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv8i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv8i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vzext.vf4 v12, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v12, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv8i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vzext.vf4 v12, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v12, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i8> @llvm.ctlz.nxv8i8(<vscale x 8 x i8> %va, i1 true)
   ret <vscale x 8 x i8> %a
 }
 
 define <vscale x 16 x i8> @ctlz_zero_undef_nxv16i8(<vscale x 16 x i8> %va) {
-; CHECK-LABEL: ctlz_zero_undef_nxv16i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 2
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vor.vv v8, v8, v10
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v10, v10, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v10
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v10, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v10, v8
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v10
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv16i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv16i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv16i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vzext.vf4 v16, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v16, 0
+; RV32D-NEXT:    addi a0, zero, 134
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv16i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vzext.vf4 v16, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v16, 0
+; RV64D-NEXT:    addi a0, zero, 134
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i8> @llvm.ctlz.nxv16i8(<vscale x 16 x i8> %va, i1 true)
   ret <vscale x 16 x i8> %a
 }
 
 define <vscale x 32 x i8> @ctlz_zero_undef_nxv32i8(<vscale x 32 x i8> %va) {
 ; CHECK-LABEL: ctlz_zero_undef_nxv32i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vsetvli a0, zero, e8, m4, ta, mu

 ; CHECK-NEXT:    vadd.vv v8, v8, v16
 ; CHECK-NEXT:    vand.vi v8, v8, 15
 ; CHECK-NEXT:    ret
   %a = call <vscale x 64 x i8> @llvm.ctlz.nxv64i8(<vscale x 64 x i8> %va, i1 true)
   ret <vscale x 64 x i8> %a
 }
 
 define <vscale x 1 x i16> @ctlz_zero_undef_nxv1i16(<vscale x 1 x i16> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv1i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv1i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv1i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv1i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv1i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv1i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i16> @llvm.ctlz.nxv1i16(<vscale x 1 x i16> %va, i1 true)
   ret <vscale x 1 x i16> %a
 }
 
 define <vscale x 2 x i16> @ctlz_zero_undef_nxv2i16(<vscale x 2 x i16> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv2i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv2i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv2i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv2i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv2i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv2i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i16> @llvm.ctlz.nxv2i16(<vscale x 2 x i16> %va, i1 true)
   ret <vscale x 2 x i16> %a
 }
 
 define <vscale x 4 x i16> @ctlz_zero_undef_nxv4i16(<vscale x 4 x i16> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv4i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv4i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv4i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv4i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv4i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v10, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v8, v10, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv4i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v10, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v8, v10, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i16> @llvm.ctlz.nxv4i16(<vscale x 4 x i16> %va, i1 true)
   ret <vscale x 4 x i16> %a
 }
 
 define <vscale x 8 x i16> @ctlz_zero_undef_nxv8i16(<vscale x 8 x i16> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv8i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv8i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv8i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv8i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv8i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v12, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v8, v12, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv8i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v12, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v8, v12, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i16> @llvm.ctlz.nxv8i16(<vscale x 8 x i16> %va, i1 true)
   ret <vscale x 8 x i16> %a
 }
 
 define <vscale x 16 x i16> @ctlz_zero_undef_nxv16i16(<vscale x 16 x i16> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv16i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv16i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv16i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv16i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv16i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v16, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    addi a0, zero, 142
+; RV32D-NEXT:    vrsub.vx v8, v16, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv16i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v16, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 142
+; RV64D-NEXT:    vrsub.vx v8, v16, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i16> @llvm.ctlz.nxv16i16(<vscale x 16 x i16> %va, i1 true)
   ret <vscale x 16 x i16> %a
 }
 
 define <vscale x 32 x i16> @ctlz_zero_undef_nxv32i16(<vscale x 32 x i16> %va) {
 ; RV32-LABEL: ctlz_zero_undef_nxv32i16:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vsetvli a0, zero, e16, m8, ta, mu

 ; RV64-NEXT:    vmul.vx v8, v8, a0
 ; RV64-NEXT:    vsrl.vi v8, v8, 8
 ; RV64-NEXT:    ret
   %a = call <vscale x 32 x i16> @llvm.ctlz.nxv32i16(<vscale x 32 x i16> %va, i1 true)
   ret <vscale x 32 x i16> %a
 }
 
 define <vscale x 1 x i32> @ctlz_zero_undef_nxv1i32(<vscale x 1 x i32> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv1i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv1i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv1i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv1i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv1i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v9, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv1i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v9, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i32> @llvm.ctlz.nxv1i32(<vscale x 1 x i32> %va, i1 true)
   ret <vscale x 1 x i32> %a
 }
 
 define <vscale x 2 x i32> @ctlz_zero_undef_nxv2i32(<vscale x 2 x i32> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv2i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v9
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv2i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v9
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv2i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v9
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv2i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v9
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv2i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v10, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v8, v10, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv2i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v10, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v8, v10, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i32> @llvm.ctlz.nxv2i32(<vscale x 2 x i32> %va, i1 true)
   ret <vscale x 2 x i32> %a
 }
 
 define <vscale x 4 x i32> @ctlz_zero_undef_nxv4i32(<vscale x 4 x i32> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv4i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v10
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv4i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v10
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv4i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v10
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv4i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v10
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv4i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v12, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v8, v12, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv4i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v12, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v8, v12, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i32> @llvm.ctlz.nxv4i32(<vscale x 4 x i32> %va, i1 true)
   ret <vscale x 4 x i32> %a
 }
 
 define <vscale x 8 x i32> @ctlz_zero_undef_nxv8i32(<vscale x 8 x i32> %va) {
-; RV32-LABEL: ctlz_zero_undef_nxv8i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 2
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 8
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 16
-; RV32-NEXT:    vor.vv v8, v8, v12
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: ctlz_zero_undef_nxv8i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 2
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 8
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 16
-; RV64-NEXT:    vor.vv v8, v8, v12
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: ctlz_zero_undef_nxv8i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 2
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 8
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 16
+; RV32I-NEXT:    vor.vv v8, v8, v12
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: ctlz_zero_undef_nxv8i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 2
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 8
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 16
+; RV64I-NEXT:    vor.vv v8, v8, v12
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: ctlz_zero_undef_nxv8i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v16, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1054
+; RV32D-NEXT:    vrsub.vx v8, v16, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: ctlz_zero_undef_nxv8i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v16, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1054
+; RV64D-NEXT:    vrsub.vx v8, v16, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i32> @llvm.ctlz.nxv8i32(<vscale x 8 x i32> %va, i1 true)
   ret <vscale x 8 x i32> %a
 }
 
 define <vscale x 16 x i32> @ctlz_zero_undef_nxv16i32(<vscale x 16 x i32> %va) {
 ; RV32-LABEL: ctlz_zero_undef_nxv16i32:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    vsetvli a0, zero, e32, m8, ta, mu

llvm/test/CodeGen/RISCV/rvv/cttz-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=riscv32 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32
-; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32,RV32I
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64,RV64I
+; RUN: llc -mtriple=riscv32 -mattr=+experimental-v,+d -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV32,RV32D
+; RUN: llc -mtriple=riscv64 -mattr=+experimental-v,+d -verify-machineinstrs < %s | FileCheck %s --check-prefixes=CHECK,RV64,RV64D
 
 define <vscale x 1 x i8> @cttz_nxv1i8(<vscale x 1 x i8> %va) {
-; CHECK-LABEL: cttz_nxv1i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_nxv1i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv1i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv1i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv1i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i8> @llvm.cttz.nxv1i8(<vscale x 1 x i8> %va, i1 false)
   ret <vscale x 1 x i8> %a
 }
 declare <vscale x 1 x i8> @llvm.cttz.nxv1i8(<vscale x 1 x i8>, i1)
 
 define <vscale x 2 x i8> @cttz_nxv2i8(<vscale x 2 x i8> %va) {
-; CHECK-LABEL: cttz_nxv2i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_nxv2i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv2i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv2i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv2i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i8> @llvm.cttz.nxv2i8(<vscale x 2 x i8> %va, i1 false)
   ret <vscale x 2 x i8> %a
 }
 declare <vscale x 2 x i8> @llvm.cttz.nxv2i8(<vscale x 2 x i8>, i1)
 
 define <vscale x 4 x i8> @cttz_nxv4i8(<vscale x 4 x i8> %va) {
-; CHECK-LABEL: cttz_nxv4i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_nxv4i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv4i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv4i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v10, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v10, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv4i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v10, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v10, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i8> @llvm.cttz.nxv4i8(<vscale x 4 x i8> %va, i1 false)
   ret <vscale x 4 x i8> %a
 }
 declare <vscale x 4 x i8> @llvm.cttz.nxv4i8(<vscale x 4 x i8>, i1)
 
 define <vscale x 8 x i8> @cttz_nxv8i8(<vscale x 8 x i8> %va) {
-; CHECK-LABEL: cttz_nxv8i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_nxv8i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv8i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv8i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vzext.vf4 v12, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v12, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv8i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vzext.vf4 v12, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v12, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i8> @llvm.cttz.nxv8i8(<vscale x 8 x i8> %va, i1 false)
   ret <vscale x 8 x i8> %a
 }
 declare <vscale x 8 x i8> @llvm.cttz.nxv8i8(<vscale x 8 x i8>, i1)
 
 define <vscale x 16 x i8> @cttz_nxv16i8(<vscale x 16 x i8> %va) {
-; CHECK-LABEL: cttz_nxv16i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
-; CHECK-NEXT:    vsub.vx v10, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v10, v10, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v10
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v10, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v10, v8
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v10
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_nxv16i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv16i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv16i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vmv.v.i v10, 0
+; RV32D-NEXT:    vmseq.vv v0, v10, v8
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v10
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vzext.vf4 v16, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v16, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv16i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vmv.v.i v10, 0
+; RV64D-NEXT:    vmseq.vv v0, v10, v8
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vzext.vf4 v16, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v16, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    vmerge.vim v8, v8, 8, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i8> @llvm.cttz.nxv16i8(<vscale x 16 x i8> %va, i1 false)
   ret <vscale x 16 x i8> %a
 }
 declare <vscale x 16 x i8> @llvm.cttz.nxv16i8(<vscale x 16 x i8>, i1)
 
 define <vscale x 32 x i8> @cttz_nxv32i8(<vscale x 32 x i8> %va) {
 ; CHECK-LABEL: cttz_nxv32i8:
 ; CHECK:       # %bb.0:

 ; CHECK-NEXT:    vand.vi v8, v8, 15
 ; CHECK-NEXT:    ret
   %a = call <vscale x 64 x i8> @llvm.cttz.nxv64i8(<vscale x 64 x i8> %va, i1 false)
   ret <vscale x 64 x i8> %a
 }
 declare <vscale x 64 x i8> @llvm.cttz.nxv64i8(<vscale x 64 x i8>, i1)
 
 define <vscale x 1 x i16> @cttz_nxv1i16(<vscale x 1 x i16> %va) {
-; RV32-LABEL: cttz_nxv1i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv1i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv1i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv1i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv1i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv1i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i16> @llvm.cttz.nxv1i16(<vscale x 1 x i16> %va, i1 false)
   ret <vscale x 1 x i16> %a
 }
 declare <vscale x 1 x i16> @llvm.cttz.nxv1i16(<vscale x 1 x i16>, i1)
 
 define <vscale x 2 x i16> @cttz_nxv2i16(<vscale x 2 x i16> %va) {
-; RV32-LABEL: cttz_nxv2i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv2i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv2i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv2i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv2i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv2i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i16> @llvm.cttz.nxv2i16(<vscale x 2 x i16> %va, i1 false)
   ret <vscale x 2 x i16> %a
 }
 declare <vscale x 2 x i16> @llvm.cttz.nxv2i16(<vscale x 2 x i16>, i1)
 
 define <vscale x 4 x i16> @cttz_nxv4i16(<vscale x 4 x i16> %va) {
-; RV32-LABEL: cttz_nxv4i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv4i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv4i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv4i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv4i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vmv.v.i v9, 0
+; RV32D-NEXT:    vmseq.vv v0, v9, v8
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v10, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v10, a0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv4i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v10, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v10, a0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i16> @llvm.cttz.nxv4i16(<vscale x 4 x i16> %va, i1 false)
   ret <vscale x 4 x i16> %a
 }
 declare <vscale x 4 x i16> @llvm.cttz.nxv4i16(<vscale x 4 x i16>, i1)
 
 define <vscale x 8 x i16> @cttz_nxv8i16(<vscale x 8 x i16> %va) {
-; RV32-LABEL: cttz_nxv8i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
-; RV32-NEXT:    vsub.vx v10, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv8i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
-; RV64-NEXT:    vsub.vx v10, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv8i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv8i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv8i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vmv.v.i v10, 0
+; RV32D-NEXT:    vmseq.vv v0, v10, v8
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v10
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v12, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v12, a0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv8i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vmv.v.i v10, 0
+; RV64D-NEXT:    vmseq.vv v0, v10, v8
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v12, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v12, a0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i16> @llvm.cttz.nxv8i16(<vscale x 8 x i16> %va, i1 false)
   ret <vscale x 8 x i16> %a
 }
 declare <vscale x 8 x i16> @llvm.cttz.nxv8i16(<vscale x 8 x i16>, i1)
 
 define <vscale x 16 x i16> @cttz_nxv16i16(<vscale x 16 x i16> %va) {
-; RV32-LABEL: cttz_nxv16i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
-; RV32-NEXT:    vsub.vx v12, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv16i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
-; RV64-NEXT:    vsub.vx v12, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv16i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
+; RV32I-NEXT:    vsub.vx v12, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv16i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
+; RV64I-NEXT:    vsub.vx v12, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv16i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vmv.v.i v12, 0
+; RV32D-NEXT:    vmseq.vv v0, v12, v8
+; RV32D-NEXT:    vrsub.vi v12, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v12
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v16, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v16, a0
+; RV32D-NEXT:    addi a0, zero, 16
+; RV32D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv16i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vmv.v.i v12, 0
+; RV64D-NEXT:    vmseq.vv v0, v12, v8
+; RV64D-NEXT:    vrsub.vi v12, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v12
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v16, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v16, a0
+; RV64D-NEXT:    addi a0, zero, 16
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i16> @llvm.cttz.nxv16i16(<vscale x 16 x i16> %va, i1 false)
   ret <vscale x 16 x i16> %a
 }
 declare <vscale x 16 x i16> @llvm.cttz.nxv16i16(<vscale x 16 x i16>, i1)
 
 define <vscale x 32 x i16> @cttz_nxv32i16(<vscale x 32 x i16> %va) {
 ; RV32-LABEL: cttz_nxv32i16:
 ; RV32:       # %bb.0:

 ; RV64-NEXT:    vsrl.vi v8, v8, 8
 ; RV64-NEXT:    ret
   %a = call <vscale x 32 x i16> @llvm.cttz.nxv32i16(<vscale x 32 x i16> %va, i1 false)
   ret <vscale x 32 x i16> %a
 }
 declare <vscale x 32 x i16> @llvm.cttz.nxv32i16(<vscale x 32 x i16>, i1)
 
 define <vscale x 1 x i32> @cttz_nxv1i32(<vscale x 1 x i32> %va) {
-; RV32-LABEL: cttz_nxv1i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv1i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv1i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv1i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv1i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v9, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v9
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV32D-NEXT:    vsrl.vx v9, v10, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v9, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv1i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v9, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i32> @llvm.cttz.nxv1i32(<vscale x 1 x i32> %va, i1 false)
   ret <vscale x 1 x i32> %a
 }
 declare <vscale x 1 x i32> @llvm.cttz.nxv1i32(<vscale x 1 x i32>, i1)
 
 define <vscale x 2 x i32> @cttz_nxv2i32(<vscale x 2 x i32> %va) {
-; RV32-LABEL: cttz_nxv2i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv2i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv2i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv2i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv2i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v9, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v9
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32D-NEXT:    vsrl.vx v10, v10, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v9, v10, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v9, v9, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v9, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv2i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vmv.v.i v9, 0
+; RV64D-NEXT:    vmseq.vv v0, v9, v8
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v10, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v10, a0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i32> @llvm.cttz.nxv2i32(<vscale x 2 x i32> %va, i1 false)
   ret <vscale x 2 x i32> %a
 }
 declare <vscale x 2 x i32> @llvm.cttz.nxv2i32(<vscale x 2 x i32>, i1)
 
 define <vscale x 4 x i32> @cttz_nxv4i32(<vscale x 4 x i32> %va) {
-; RV32-LABEL: cttz_nxv4i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
-; RV32-NEXT:    vsub.vx v10, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv4i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
-; RV64-NEXT:    vsub.vx v10, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv4i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv4i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv4i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v10, v8, v10
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v10
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV32D-NEXT:    vsrl.vx v12, v12, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v12, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v10, v10, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v10, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv4i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vmv.v.i v10, 0
+; RV64D-NEXT:    vmseq.vv v0, v10, v8
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v12, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v12, a0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i32> @llvm.cttz.nxv4i32(<vscale x 4 x i32> %va, i1 false)
   ret <vscale x 4 x i32> %a
 }
 declare <vscale x 4 x i32> @llvm.cttz.nxv4i32(<vscale x 4 x i32>, i1)
 
 define <vscale x 8 x i32> @cttz_nxv8i32(<vscale x 8 x i32> %va) {
-; RV32-LABEL: cttz_nxv8i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
-; RV32-NEXT:    vsub.vx v12, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_nxv8i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
-; RV64-NEXT:    vsub.vx v12, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_nxv8i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; RV32I-NEXT:    vsub.vx v12, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_nxv8i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; RV64I-NEXT:    vsub.vx v12, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_nxv8i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vrsub.vi v12, v8, 0
+; RV32D-NEXT:    vand.vv v12, v8, v12
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v12
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV32D-NEXT:    vsrl.vx v16, v16, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v16, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v12, v12, a0
+; RV32D-NEXT:    vmseq.vi v0, v8, 0
+; RV32D-NEXT:    addi a0, zero, 32
+; RV32D-NEXT:    vmerge.vxm v8, v12, a0, v0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_nxv8i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vmv.v.i v12, 0
+; RV64D-NEXT:    vmseq.vv v0, v12, v8
+; RV64D-NEXT:    vrsub.vi v12, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v12
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v16, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v16, a0
+; RV64D-NEXT:    addi a0, zero, 32
+; RV64D-NEXT:    vmerge.vxm v8, v8, a0, v0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i32> @llvm.cttz.nxv8i32(<vscale x 8 x i32> %va, i1 false)
   ret <vscale x 8 x i32> %a
 }
 declare <vscale x 8 x i32> @llvm.cttz.nxv8i32(<vscale x 8 x i32>, i1)
 
 define <vscale x 16 x i32> @cttz_nxv16i32(<vscale x 16 x i32> %va) {
 ; RV32-LABEL: cttz_nxv16i32:
 ; RV32:       # %bb.0:

 ; RV64-NEXT:    vsrl.vx v8, v8, a0
 ; RV64-NEXT:    ret
   %a = call <vscale x 8 x i64> @llvm.cttz.nxv8i64(<vscale x 8 x i64> %va, i1 false)
   ret <vscale x 8 x i64> %a
 }
 declare <vscale x 8 x i64> @llvm.cttz.nxv8i64(<vscale x 8 x i64>, i1)
 
 define <vscale x 1 x i8> @cttz_zero_undef_nxv1i8(<vscale x 1 x i8> %va) {
-; CHECK-LABEL: cttz_zero_undef_nxv1i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv1i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv1i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf8, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv1i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv1i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf8, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i8> @llvm.cttz.nxv1i8(<vscale x 1 x i8> %va, i1 true)
   ret <vscale x 1 x i8> %a
 }
 
 define <vscale x 2 x i8> @cttz_zero_undef_nxv2i8(<vscale x 2 x i8> %va) {
-; CHECK-LABEL: cttz_zero_undef_nxv2i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv2i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv2i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf4, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv2i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vzext.vf4 v9, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv2i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vzext.vf4 v9, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v9
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i8> @llvm.cttz.nxv2i8(<vscale x 2 x i8> %va, i1 true)
   ret <vscale x 2 x i8> %a
 }
 
 define <vscale x 4 x i8> @cttz_zero_undef_nxv4i8(<vscale x 4 x i8> %va) {
-; CHECK-LABEL: cttz_zero_undef_nxv4i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv4i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv4i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv4i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vzext.vf4 v10, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v10, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv4i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vzext.vf4 v10, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v10
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v10, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i8> @llvm.cttz.nxv4i8(<vscale x 4 x i8> %va, i1 true)
   ret <vscale x 4 x i8> %a
 }
 
 define <vscale x 8 x i8> @cttz_zero_undef_nxv8i8(<vscale x 8 x i8> %va) {
-; CHECK-LABEL: cttz_zero_undef_nxv8i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
-; CHECK-NEXT:    vsub.vx v9, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v9
-; CHECK-NEXT:    vsrl.vi v9, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v9, v9, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v9
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v9, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v9, v8
-; CHECK-NEXT:    vsrl.vi v9, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v9
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv8i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv8i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv8i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vzext.vf4 v12, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v12, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv8i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vzext.vf4 v12, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v12
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v12, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i8> @llvm.cttz.nxv8i8(<vscale x 8 x i8> %va, i1 true)
   ret <vscale x 8 x i8> %a
 }
 
 define <vscale x 16 x i8> @cttz_zero_undef_nxv16i8(<vscale x 16 x i8> %va) {
-; CHECK-LABEL: cttz_zero_undef_nxv16i8:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    addi a0, zero, 1
-; CHECK-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
-; CHECK-NEXT:    vsub.vx v10, v8, a0
-; CHECK-NEXT:    vxor.vi v8, v8, -1
-; CHECK-NEXT:    vand.vv v8, v8, v10
-; CHECK-NEXT:    vsrl.vi v10, v8, 1
-; CHECK-NEXT:    addi a0, zero, 85
-; CHECK-NEXT:    vand.vx v10, v10, a0
-; CHECK-NEXT:    vsub.vv v8, v8, v10
-; CHECK-NEXT:    addi a0, zero, 51
-; CHECK-NEXT:    vand.vx v10, v8, a0
-; CHECK-NEXT:    vsrl.vi v8, v8, 2
-; CHECK-NEXT:    vand.vx v8, v8, a0
-; CHECK-NEXT:    vadd.vv v8, v10, v8
-; CHECK-NEXT:    vsrl.vi v10, v8, 4
-; CHECK-NEXT:    vadd.vv v8, v8, v10
-; CHECK-NEXT:    vand.vi v8, v8, 15
-; CHECK-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv16i8:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    addi a0, zero, 85
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    addi a0, zero, 51
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    vand.vi v8, v8, 15
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv16i8:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e8, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    addi a0, zero, 85
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    addi a0, zero, 51
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    vand.vi v8, v8, 15
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv16i8:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v10
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vzext.vf4 v16, v8
+; RV32D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV32D-NEXT:    vsrl.vi v8, v8, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v16, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv16i8:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vzext.vf4 v16, v8
+; RV64D-NEXT:    vfcvt.f.xu.v v8, v16
+; RV64D-NEXT:    vsrl.vi v8, v8, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v16, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i8> @llvm.cttz.nxv16i8(<vscale x 16 x i8> %va, i1 true)
   ret <vscale x 16 x i8> %a
 }
 
 define <vscale x 32 x i8> @cttz_zero_undef_nxv32i8(<vscale x 32 x i8> %va) {
 ; CHECK-LABEL: cttz_zero_undef_nxv32i8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addi a0, zero, 1

 ; CHECK-NEXT:    vadd.vv v8, v8, v16
 ; CHECK-NEXT:    vand.vi v8, v8, 15
 ; CHECK-NEXT:    ret
   %a = call <vscale x 64 x i8> @llvm.cttz.nxv64i8(<vscale x 64 x i8> %va, i1 true)
   ret <vscale x 64 x i8> %a
 }
 
 define <vscale x 1 x i16> @cttz_zero_undef_nxv1i16(<vscale x 1 x i16> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv1i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv1i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv1i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv1i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, mf4, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv1i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv1i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i16> @llvm.cttz.nxv1i16(<vscale x 1 x i16> %va, i1 true)
   ret <vscale x 1 x i16> %a
 }
 
 define <vscale x 2 x i16> @cttz_zero_undef_nxv2i16(<vscale x 2 x i16> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv2i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv2i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv2i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv2i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv2i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v9, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv2i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v9, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i16> @llvm.cttz.nxv2i16(<vscale x 2 x i16> %va, i1 true)
   ret <vscale x 2 x i16> %a
 }
 
 define <vscale x 4 x i16> @cttz_zero_undef_nxv4i16(<vscale x 4 x i16> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv4i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv4i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv4i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv4i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv4i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v10, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v10, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv4i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v10, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v10, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i16> @llvm.cttz.nxv4i16(<vscale x 4 x i16> %va, i1 true)
   ret <vscale x 4 x i16> %a
 }
 
 define <vscale x 8 x i16> @cttz_zero_undef_nxv8i16(<vscale x 8 x i16> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv8i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
-; RV32-NEXT:    vsub.vx v10, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv8i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
-; RV64-NEXT:    vsub.vx v10, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv8i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv8i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv8i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v10
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v12, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v12, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv8i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v12, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v12, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i16> @llvm.cttz.nxv8i16(<vscale x 8 x i16> %va, i1 true)
   ret <vscale x 8 x i16> %a
 }
 
 define <vscale x 16 x i16> @cttz_zero_undef_nxv16i16(<vscale x 16 x i16> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv16i16:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
-; RV32-NEXT:    vsub.vx v12, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 5
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 3
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 1
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    addi a0, zero, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 8
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv16i16:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
-; RV64-NEXT:    vsub.vx v12, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 5
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 3
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 1
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    addi a0, zero, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 8
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv16i16:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
+; RV32I-NEXT:    vsub.vx v12, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 5
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 3
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 1
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    addi a0, zero, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 8
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv16i16:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e16, m4, ta, mu
+; RV64I-NEXT:    vsub.vx v12, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 5
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 3
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 1
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    addi a0, zero, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 8
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv16i16:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vrsub.vi v12, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v12
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV32D-NEXT:    vsrl.vi v8, v16, 23
+; RV32D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    addi a0, zero, 127
+; RV32D-NEXT:    vsub.vx v8, v16, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv16i16:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vrsub.vi v12, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v12
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; RV64D-NEXT:    vsrl.vi v8, v16, 23
+; RV64D-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 127
+; RV64D-NEXT:    vsub.vx v8, v16, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 16 x i16> @llvm.cttz.nxv16i16(<vscale x 16 x i16> %va, i1 true)
   ret <vscale x 16 x i16> %a
 }
 
 define <vscale x 32 x i16> @cttz_zero_undef_nxv32i16(<vscale x 32 x i16> %va) {
 ; RV32-LABEL: cttz_zero_undef_nxv32i16:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    addi a0, zero, 1

 ; RV64-NEXT:    vmul.vx v8, v8, a0
 ; RV64-NEXT:    vsrl.vi v8, v8, 8
 ; RV64-NEXT:    ret
   %a = call <vscale x 32 x i16> @llvm.cttz.nxv32i16(<vscale x 32 x i16> %va, i1 true)
   ret <vscale x 32 x i16> %a
 }
 
 define <vscale x 1 x i32> @cttz_zero_undef_nxv1i32(<vscale x 1 x i32> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv1i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv1i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv1i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv1i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, mf2, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv1i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v9, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v8, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v8, v8, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv1i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v9, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m1, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v9, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, mf2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v8, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v8, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 1 x i32> @llvm.cttz.nxv1i32(<vscale x 1 x i32> %va, i1 true)
   ret <vscale x 1 x i32> %a
 }
 
 define <vscale x 2 x i32> @cttz_zero_undef_nxv2i32(<vscale x 2 x i32> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv2i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
-; RV32-NEXT:    vsub.vx v9, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v9
-; RV32-NEXT:    vsrl.vi v9, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v9, v9, a0
-; RV32-NEXT:    vsub.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v9, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v9, v8
-; RV32-NEXT:    vsrl.vi v9, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v9
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv2i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
-; RV64-NEXT:    vsub.vx v9, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v9
-; RV64-NEXT:    vsrl.vi v9, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v9, v9, a0
-; RV64-NEXT:    vsub.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v9, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v9, v8
-; RV64-NEXT:    vsrl.vi v9, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v9
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv2i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; RV32I-NEXT:    vsub.vx v9, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v9
+; RV32I-NEXT:    vsrl.vi v9, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v9, v9, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v9, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v9, v8
+; RV32I-NEXT:    vsrl.vi v9, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v9
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv2i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m1, ta, mu
+; RV64I-NEXT:    vsub.vx v9, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v9
+; RV64I-NEXT:    vsrl.vi v9, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v9, v9, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v9, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v9, v8
+; RV64I-NEXT:    vsrl.vi v9, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v9
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv2i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vrsub.vi v9, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v9
+; RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v10, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV32D-NEXT:    vnsrl.wi v10, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v8, v10, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv2i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vrsub.vi v9, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v9
+; RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m2, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v10, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m1, ta, mu
+; RV64D-NEXT:    vnsrl.wi v10, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v10, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 2 x i32> @llvm.cttz.nxv2i32(<vscale x 2 x i32> %va, i1 true)
   ret <vscale x 2 x i32> %a
 }
 
 define <vscale x 4 x i32> @cttz_zero_undef_nxv4i32(<vscale x 4 x i32> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv4i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
-; RV32-NEXT:    vsub.vx v10, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v10
-; RV32-NEXT:    vsrl.vi v10, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v10, v10, a0
-; RV32-NEXT:    vsub.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v10, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v10, v8
-; RV32-NEXT:    vsrl.vi v10, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v10
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv4i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
-; RV64-NEXT:    vsub.vx v10, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v10
-; RV64-NEXT:    vsrl.vi v10, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v10, v10, a0
-; RV64-NEXT:    vsub.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v10, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v10, v8
-; RV64-NEXT:    vsrl.vi v10, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v10
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv4i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; RV32I-NEXT:    vsub.vx v10, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v10
+; RV32I-NEXT:    vsrl.vi v10, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v10, v10, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v10, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v10, v8
+; RV32I-NEXT:    vsrl.vi v10, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v10
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv4i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m2, ta, mu
+; RV64I-NEXT:    vsub.vx v10, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v10
+; RV64I-NEXT:    vsrl.vi v10, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v10, v10, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v10, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v10, v8
+; RV64I-NEXT:    vsrl.vi v10, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v10
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv4i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vrsub.vi v10, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v10
+; RV32D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v12, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV32D-NEXT:    vnsrl.wi v12, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v8, v12, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv4i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vrsub.vi v10, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v10
+; RV64D-NEXT:    vfwcvt.f.xu.v v12, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m4, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v12, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m2, ta, mu
+; RV64D-NEXT:    vnsrl.wi v12, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v12, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 4 x i32> @llvm.cttz.nxv4i32(<vscale x 4 x i32> %va, i1 true)
   ret <vscale x 4 x i32> %a
 }
 
 define <vscale x 8 x i32> @cttz_zero_undef_nxv8i32(<vscale x 8 x i32> %va) {
-; RV32-LABEL: cttz_zero_undef_nxv8i32:
-; RV32:       # %bb.0:
-; RV32-NEXT:    addi a0, zero, 1
-; RV32-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
-; RV32-NEXT:    vsub.vx v12, v8, a0
-; RV32-NEXT:    vxor.vi v8, v8, -1
-; RV32-NEXT:    vand.vv v8, v8, v12
-; RV32-NEXT:    vsrl.vi v12, v8, 1
-; RV32-NEXT:    lui a0, 349525
-; RV32-NEXT:    addi a0, a0, 1365
-; RV32-NEXT:    vand.vx v12, v12, a0
-; RV32-NEXT:    vsub.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 209715
-; RV32-NEXT:    addi a0, a0, 819
-; RV32-NEXT:    vand.vx v12, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 2
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    vadd.vv v8, v12, v8
-; RV32-NEXT:    vsrl.vi v12, v8, 4
-; RV32-NEXT:    vadd.vv v8, v8, v12
-; RV32-NEXT:    lui a0, 61681
-; RV32-NEXT:    addi a0, a0, -241
-; RV32-NEXT:    vand.vx v8, v8, a0
-; RV32-NEXT:    lui a0, 4112
-; RV32-NEXT:    addi a0, a0, 257
-; RV32-NEXT:    vmul.vx v8, v8, a0
-; RV32-NEXT:    vsrl.vi v8, v8, 24
-; RV32-NEXT:    ret
-;
-; RV64-LABEL: cttz_zero_undef_nxv8i32:
-; RV64:       # %bb.0:
-; RV64-NEXT:    addi a0, zero, 1
-; RV64-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
-; RV64-NEXT:    vsub.vx v12, v8, a0
-; RV64-NEXT:    vxor.vi v8, v8, -1
-; RV64-NEXT:    vand.vv v8, v8, v12
-; RV64-NEXT:    vsrl.vi v12, v8, 1
-; RV64-NEXT:    lui a0, 349525
-; RV64-NEXT:    addiw a0, a0, 1365
-; RV64-NEXT:    vand.vx v12, v12, a0
-; RV64-NEXT:    vsub.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 209715
-; RV64-NEXT:    addiw a0, a0, 819
-; RV64-NEXT:    vand.vx v12, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 2
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    vadd.vv v8, v12, v8
-; RV64-NEXT:    vsrl.vi v12, v8, 4
-; RV64-NEXT:    vadd.vv v8, v8, v12
-; RV64-NEXT:    lui a0, 61681
-; RV64-NEXT:    addiw a0, a0, -241
-; RV64-NEXT:    vand.vx v8, v8, a0
-; RV64-NEXT:    lui a0, 4112
-; RV64-NEXT:    addiw a0, a0, 257
-; RV64-NEXT:    vmul.vx v8, v8, a0
-; RV64-NEXT:    vsrl.vi v8, v8, 24
-; RV64-NEXT:    ret
+; RV32I-LABEL: cttz_zero_undef_nxv8i32:
+; RV32I:       # %bb.0:
+; RV32I-NEXT:    addi a0, zero, 1
+; RV32I-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; RV32I-NEXT:    vsub.vx v12, v8, a0
+; RV32I-NEXT:    vxor.vi v8, v8, -1
+; RV32I-NEXT:    vand.vv v8, v8, v12
+; RV32I-NEXT:    vsrl.vi v12, v8, 1
+; RV32I-NEXT:    lui a0, 349525
+; RV32I-NEXT:    addi a0, a0, 1365
+; RV32I-NEXT:    vand.vx v12, v12, a0
+; RV32I-NEXT:    vsub.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 209715
+; RV32I-NEXT:    addi a0, a0, 819
+; RV32I-NEXT:    vand.vx v12, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 2
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    vadd.vv v8, v12, v8
+; RV32I-NEXT:    vsrl.vi v12, v8, 4
+; RV32I-NEXT:    vadd.vv v8, v8, v12
+; RV32I-NEXT:    lui a0, 61681
+; RV32I-NEXT:    addi a0, a0, -241
+; RV32I-NEXT:    vand.vx v8, v8, a0
+; RV32I-NEXT:    lui a0, 4112
+; RV32I-NEXT:    addi a0, a0, 257
+; RV32I-NEXT:    vmul.vx v8, v8, a0
+; RV32I-NEXT:    vsrl.vi v8, v8, 24
+; RV32I-NEXT:    ret
+;
+; RV64I-LABEL: cttz_zero_undef_nxv8i32:
+; RV64I:       # %bb.0:
+; RV64I-NEXT:    addi a0, zero, 1
+; RV64I-NEXT:    vsetvli a1, zero, e32, m4, ta, mu
+; RV64I-NEXT:    vsub.vx v12, v8, a0
+; RV64I-NEXT:    vxor.vi v8, v8, -1
+; RV64I-NEXT:    vand.vv v8, v8, v12
+; RV64I-NEXT:    vsrl.vi v12, v8, 1
+; RV64I-NEXT:    lui a0, 349525
+; RV64I-NEXT:    addiw a0, a0, 1365
+; RV64I-NEXT:    vand.vx v12, v12, a0
+; RV64I-NEXT:    vsub.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 209715
+; RV64I-NEXT:    addiw a0, a0, 819
+; RV64I-NEXT:    vand.vx v12, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 2
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    vadd.vv v8, v12, v8
+; RV64I-NEXT:    vsrl.vi v12, v8, 4
+; RV64I-NEXT:    vadd.vv v8, v8, v12
+; RV64I-NEXT:    lui a0, 61681
+; RV64I-NEXT:    addiw a0, a0, -241
+; RV64I-NEXT:    vand.vx v8, v8, a0
+; RV64I-NEXT:    lui a0, 4112
+; RV64I-NEXT:    addiw a0, a0, 257
+; RV64I-NEXT:    vmul.vx v8, v8, a0
+; RV64I-NEXT:    vsrl.vi v8, v8, 24
+; RV64I-NEXT:    ret
+;
+; RV32D-LABEL: cttz_zero_undef_nxv8i32:
+; RV32D:       # %bb.0:
+; RV32D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vrsub.vi v12, v8, 0
+; RV32D-NEXT:    vand.vv v8, v8, v12
+; RV32D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV32D-NEXT:    addi a0, zero, 52
+; RV32D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV32D-NEXT:    vsrl.vx v8, v16, a0
+; RV32D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV32D-NEXT:    vnsrl.wi v16, v8, 0
+; RV32D-NEXT:    addi a0, zero, 1023
+; RV32D-NEXT:    vsub.vx v8, v16, a0
+; RV32D-NEXT:    ret
+;
+; RV64D-LABEL: cttz_zero_undef_nxv8i32:
+; RV64D:       # %bb.0:
+; RV64D-NEXT:    vsetvli a0, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vrsub.vi v12, v8, 0
+; RV64D-NEXT:    vand.vv v8, v8, v12
+; RV64D-NEXT:    vfwcvt.f.xu.v v16, v8
+; RV64D-NEXT:    addi a0, zero, 52
+; RV64D-NEXT:    vsetvli zero, zero, e64, m8, ta, mu
+; RV64D-NEXT:    vsrl.vx v8, v16, a0
+; RV64D-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; RV64D-NEXT:    vnsrl.wi v16, v8, 0
+; RV64D-NEXT:    addi a0, zero, 1023
+; RV64D-NEXT:    vsub.vx v8, v16, a0
+; RV64D-NEXT:    ret
   %a = call <vscale x 8 x i32> @llvm.cttz.nxv8i32(<vscale x 8 x i32> %va, i1 true)
   ret <vscale x 8 x i32> %a
 }
 
 define <vscale x 16 x i32> @cttz_zero_undef_nxv16i32(<vscale x 16 x i32> %va) {
 ; RV32-LABEL: cttz_zero_undef_nxv16i32:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    addi a0, zero, 1

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ctlz.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32
-; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32,LMULMAX2-RV32I
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64,LMULMAX2-RV64I
 ; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV32
 ; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32,LMULMAX2-RV32D
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64,LMULMAX2-RV64D
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV32
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=8 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX8-RV32
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=8 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX8-RV64
 
 define void @ctlz_v16i8(<16 x i8>* %x, <16 x i8>* %y) nounwind {
 ; LMULMAX2-RV32-LABEL: ctlz_v16i8:
 ; LMULMAX2-RV32:       # %bb.0:
 ; LMULMAX2-RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
 ; LMULMAX2-RV32-NEXT:    vle8.v v8, (a0)
 ; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
 ; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 2
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vadd.vv v8, v9, v8
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v8, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v8, v8, v9
 ; LMULMAX1-RV64-NEXT:    vand.vi v8, v8, 15
 ; LMULMAX1-RV64-NEXT:    vse8.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v16i8:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; LMULMAX8-RV32-NEXT:    vzext.vf4 v12, v8
+; LMULMAX8-RV32-NEXT:    vfcvt.f.xu.v v12, v12
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v9, v10, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 134
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    vrsub.vx v8, v9, a1
+; LMULMAX8-RV32-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV32-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v16i8:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; LMULMAX8-RV64-NEXT:    vzext.vf4 v12, v8
+; LMULMAX8-RV64-NEXT:    vfcvt.f.xu.v v12, v12
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v9, v10, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 134
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    vrsub.vx v8, v9, a1
+; LMULMAX8-RV64-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV64-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <16 x i8>, <16 x i8>* %x
   %b = load <16 x i8>, <16 x i8>* %y
   %c = call <16 x i8> @llvm.ctlz.v16i8(<16 x i8> %a, i1 false)
   store <16 x i8> %c, <16 x i8>* %x
   ret void
 }
 declare <16 x i8> @llvm.ctlz.v16i8(<16 x i8>, i1)
 
 define void @ctlz_v8i16(<8 x i16>* %x, <8 x i16>* %y) nounwind {
-; LMULMAX2-RV32-LABEL: ctlz_v8i16:
-; LMULMAX2-RV32:       # %bb.0:
-; LMULMAX2-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
-; LMULMAX2-RV32-NEXT:    vle16.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 2
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 8
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    lui a1, 5
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 1365
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV32-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 3
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 819
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 1
-; LMULMAX2-RV32-NEXT:    addi a1, a1, -241
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    addi a1, zero, 257
-; LMULMAX2-RV32-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 8
-; LMULMAX2-RV32-NEXT:    vse16.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    ret
-;
-; LMULMAX2-RV64-LABEL: ctlz_v8i16:
-; LMULMAX2-RV64:       # %bb.0:
-; LMULMAX2-RV64-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
-; LMULMAX2-RV64-NEXT:    vle16.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 2
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 8
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    lui a1, 5
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 1365
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV64-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 3
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 819
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 1
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, -241
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    addi a1, zero, 257
-; LMULMAX2-RV64-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 8
-; LMULMAX2-RV64-NEXT:    vse16.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    ret
+; LMULMAX2-RV32I-LABEL: ctlz_v8i16:
+; LMULMAX2-RV32I:       # %bb.0:
+; LMULMAX2-RV32I-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV32I-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    lui a1, 5
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 1365
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV32I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 3
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 819
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 1
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, -241
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    addi a1, zero, 257
+; LMULMAX2-RV32I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 8
+; LMULMAX2-RV32I-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    ret
+;
+; LMULMAX2-RV64I-LABEL: ctlz_v8i16:
+; LMULMAX2-RV64I:       # %bb.0:
+; LMULMAX2-RV64I-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV64I-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    lui a1, 5
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 1365
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV64I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 3
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 819
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 1
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, -241
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    addi a1, zero, 257
+; LMULMAX2-RV64I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 8
+; LMULMAX2-RV64I-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    ret
 ;
 ; LMULMAX1-RV32-LABEL: ctlz_v8i16:
 ; LMULMAX1-RV32:       # %bb.0:
 ; LMULMAX1-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
 ; LMULMAX1-RV32-NEXT:    vle16.v v8, (a0)
 ; LMULMAX1-RV32-NEXT:    vsrl.vi v9, v8, 1
 ; LMULMAX1-RV32-NEXT:    vor.vv v8, v8, v9
 ; LMULMAX1-RV32-NEXT:    vsrl.vi v9, v8, 2

 ; LMULMAX1-RV64-NEXT:    lui a1, 1
 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, -241
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    addi a1, zero, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 8
 ; LMULMAX1-RV64-NEXT:    vse16.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX2-RV32D-LABEL: ctlz_v8i16:
+; LMULMAX2-RV32D:       # %bb.0:
+; LMULMAX2-RV32D-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV32D-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX2-RV32D-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 142
+; LMULMAX2-RV32D-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX2-RV32D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 16
+; LMULMAX2-RV32D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV32D-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    ret
+;
+; LMULMAX2-RV64D-LABEL: ctlz_v8i16:
+; LMULMAX2-RV64D:       # %bb.0:
+; LMULMAX2-RV64D-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV64D-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX2-RV64D-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 142
+; LMULMAX2-RV64D-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX2-RV64D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 16
+; LMULMAX2-RV64D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV64D-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v8i16:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 142
+; LMULMAX8-RV32-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV32-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v8i16:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 142
+; LMULMAX8-RV64-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV64-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <8 x i16>, <8 x i16>* %x
   %b = load <8 x i16>, <8 x i16>* %y
   %c = call <8 x i16> @llvm.ctlz.v8i16(<8 x i16> %a, i1 false)
   store <8 x i16> %c, <8 x i16>* %x
   ret void
 }
 declare <8 x i16> @llvm.ctlz.v8i16(<8 x i16>, i1)
 
 define void @ctlz_v4i32(<4 x i32>* %x, <4 x i32>* %y) nounwind {
-; LMULMAX2-RV32-LABEL: ctlz_v4i32:
-; LMULMAX2-RV32:       # %bb.0:
-; LMULMAX2-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
-; LMULMAX2-RV32-NEXT:    vle32.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 2
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 8
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 16
-; LMULMAX2-RV32-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    lui a1, 349525
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 1365
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV32-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 209715
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 819
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 61681
-; LMULMAX2-RV32-NEXT:    addi a1, a1, -241
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    lui a1, 4112
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 257
-; LMULMAX2-RV32-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 24
-; LMULMAX2-RV32-NEXT:    vse32.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    ret
-;
-; LMULMAX2-RV64-LABEL: ctlz_v4i32:
-; LMULMAX2-RV64:       # %bb.0:
-; LMULMAX2-RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
-; LMULMAX2-RV64-NEXT:    vle32.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 2
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 8
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 16
-; LMULMAX2-RV64-NEXT:    vor.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    lui a1, 349525
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 1365
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV64-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 209715
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 819
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 61681
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, -241
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    lui a1, 4112
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 257
-; LMULMAX2-RV64-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 24
-; LMULMAX2-RV64-NEXT:    vse32.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    ret
+; LMULMAX2-RV32I-LABEL: ctlz_v4i32:
+; LMULMAX2-RV32I:       # %bb.0:
+; LMULMAX2-RV32I-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV32I-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 16
+; LMULMAX2-RV32I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    lui a1, 349525
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 1365
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV32I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 209715
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 819
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 61681
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, -241
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    lui a1, 4112
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 257
+; LMULMAX2-RV32I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 24
+; LMULMAX2-RV32I-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    ret
+;
+; LMULMAX2-RV64I-LABEL: ctlz_v4i32:
+; LMULMAX2-RV64I:       # %bb.0:
+; LMULMAX2-RV64I-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV64I-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 16
+; LMULMAX2-RV64I-NEXT:    vor.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    lui a1, 349525
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 1365
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV64I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 209715
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 819
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 61681
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, -241
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    lui a1, 4112
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 257
+; LMULMAX2-RV64I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 24
+; LMULMAX2-RV64I-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    ret
 ;
 ; LMULMAX1-RV32-LABEL: ctlz_v4i32:
 ; LMULMAX1-RV32:       # %bb.0:
 ; LMULMAX1-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
 ; LMULMAX1-RV32-NEXT:    vle32.v v8, (a0)
 ; LMULMAX1-RV32-NEXT:    vsrl.vi v9, v8, 1
 ; LMULMAX1-RV32-NEXT:    vor.vv v8, v8, v9
 ; LMULMAX1-RV32-NEXT:    vsrl.vi v9, v8, 2

 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, -241
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    lui a1, 4112
 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 24
 ; LMULMAX1-RV64-NEXT:    vse32.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX2-RV32D-LABEL: ctlz_v4i32:
+; LMULMAX2-RV32D:       # %bb.0:
+; LMULMAX2-RV32D-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV32D-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 52
+; LMULMAX2-RV32D-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 1054
+; LMULMAX2-RV32D-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX2-RV32D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 32
+; LMULMAX2-RV32D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV32D-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    ret
+;
+; LMULMAX2-RV64D-LABEL: ctlz_v4i32:
+; LMULMAX2-RV64D:       # %bb.0:
+; LMULMAX2-RV64D-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV64D-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 52
+; LMULMAX2-RV64D-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 1054
+; LMULMAX2-RV64D-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX2-RV64D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 32
+; LMULMAX2-RV64D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV64D-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v4i32:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV32-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1054
+; LMULMAX8-RV32-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV32-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v4i32:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v10, v8
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV64-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1054
+; LMULMAX8-RV64-NEXT:    vrsub.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV64-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <4 x i32>, <4 x i32>* %x
   %b = load <4 x i32>, <4 x i32>* %y
   %c = call <4 x i32> @llvm.ctlz.v4i32(<4 x i32> %a, i1 false)
   store <4 x i32> %c, <4 x i32>* %x
   ret void
 }
 declare <4 x i32> @llvm.ctlz.v4i32(<4 x i32>, i1)
 

 ; LMULMAX1-RV64-NEXT:    addi a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    slli a1, a1, 16
 ; LMULMAX1-RV64-NEXT:    addi a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    addi a1, zero, 56
 ; LMULMAX1-RV64-NEXT:    vsrl.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vse64.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v2i64:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 16
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vsrl.vx v9, v8, a1
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.i v9, -1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vxor.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV32-NEXT:    lui a1, 349525
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v9, v9, v10
+; LMULMAX8-RV32-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 209715
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v10, v8
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 61681
+; LMULMAX8-RV32-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 4112
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmul.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV32-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV32-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v2i64:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 2
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 8
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 16
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vsrl.vx v9, v8, a1
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV64-NEXT:    lui a1, 21845
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    vand.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    lui a1, 13107
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    vand.vx v9, v8, a1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    lui a1, 3855
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    lui a1, 4112
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV64-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <2 x i64>, <2 x i64>* %x
   %b = load <2 x i64>, <2 x i64>* %y
   %c = call <2 x i64> @llvm.ctlz.v2i64(<2 x i64> %a, i1 false)
   store <2 x i64> %c, <2 x i64>* %x
   ret void
 }
 declare <2 x i64> @llvm.ctlz.v2i64(<2 x i64>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a3
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v10, v9
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vi v9, v9, 15
 ; LMULMAX1-RV64-NEXT:    vse8.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse8.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v32i8:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; LMULMAX8-RV32-NEXT:    vzext.vf4 v16, v8
+; LMULMAX8-RV32-NEXT:    vfcvt.f.xu.v v16, v16
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v12, v16, 23
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 134
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    vrsub.vx v8, v10, a1
+; LMULMAX8-RV32-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV32-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v32i8:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; LMULMAX8-RV64-NEXT:    vzext.vf4 v16, v8
+; LMULMAX8-RV64-NEXT:    vfcvt.f.xu.v v16, v16
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v12, v16, 23
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 134
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    vrsub.vx v8, v10, a1
+; LMULMAX8-RV64-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV64-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <32 x i8>, <32 x i8>* %x
   %b = load <32 x i8>, <32 x i8>* %y
   %c = call <32 x i8> @llvm.ctlz.v32i8(<32 x i8> %a, i1 false)
   store <32 x i8> %c, <32 x i8>* %x
   ret void
 }
 declare <32 x i8> @llvm.ctlz.v32i8(<32 x i8>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a4
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v9, 8
 ; LMULMAX1-RV64-NEXT:    vse16.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse16.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v16i16:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v12, v8
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 142
+; LMULMAX8-RV32-NEXT:    vrsub.vx v10, v10, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV32-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v16i16:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v12, v8
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 142
+; LMULMAX8-RV64-NEXT:    vrsub.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV64-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <16 x i16>, <16 x i16>* %x
   %b = load <16 x i16>, <16 x i16>* %y
   %c = call <16 x i16> @llvm.ctlz.v16i16(<16 x i16> %a, i1 false)
   store <16 x i16> %c, <16 x i16>* %x
   ret void
 }
 declare <16 x i16> @llvm.ctlz.v16i16(<16 x i16>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a4
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v9, 24
 ; LMULMAX1-RV64-NEXT:    vse32.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse32.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v8i32:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v12, v8
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV32-NEXT:    vnsrl.wx v10, v12, a1
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1054
+; LMULMAX8-RV32-NEXT:    vrsub.vx v10, v10, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV32-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v8i32:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v12, v8
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV64-NEXT:    vnsrl.wx v10, v12, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1054
+; LMULMAX8-RV64-NEXT:    vrsub.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV64-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <8 x i32>, <8 x i32>* %x
   %b = load <8 x i32>, <8 x i32>* %y
   %c = call <8 x i32> @llvm.ctlz.v8i32(<8 x i32> %a, i1 false)
   store <8 x i32> %c, <8 x i32>* %x
   ret void
 }
 declare <8 x i32> @llvm.ctlz.v8i32(<8 x i32>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a2
 ; LMULMAX1-RV64-NEXT:    vsrl.vx v9, v9, a1
 ; LMULMAX1-RV64-NEXT:    vse64.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse64.v v8, (a7)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: ctlz_v4i64:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 2
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 8
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 16
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vsrl.vx v10, v8, a1
+; LMULMAX8-RV32-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.i v10, -1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vxor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV32-NEXT:    lui a1, 349525
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v12, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v10, v12
+; LMULMAX8-RV32-NEXT:    vsub.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 209715
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v12, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v12, v8
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 61681
+; LMULMAX8-RV32-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 4112
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmul.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV32-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV32-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: ctlz_v4i64:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 2
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 8
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 16
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vsrl.vx v10, v8, a1
+; LMULMAX8-RV64-NEXT:    vor.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV64-NEXT:    lui a1, 21845
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    vand.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vsub.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    lui a1, 13107
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    vand.vx v10, v8, a1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v10, v8
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    lui a1, 3855
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    lui a1, 4112
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV64-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <4 x i64>, <4 x i64>* %x
   %b = load <4 x i64>, <4 x i64>* %y
   %c = call <4 x i64> @llvm.ctlz.v4i64(<4 x i64> %a, i1 false)
   store <4 x i64> %c, <4 x i64>* %x
   ret void
 }
 declare <4 x i64> @llvm.ctlz.v4i64(<4 x i64>, i1)

llvm/test/CodeGen/RISCV/rvv/fixed-vectors-cttz.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32
-; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32,LMULMAX2-RV32I
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64,LMULMAX2-RV64I
 ; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV32
 ; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV32,LMULMAX2-RV32D
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=2 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX2-RV64,LMULMAX2-RV64D
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV32
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=1 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX1-RV64
+; RUN: llc -mtriple=riscv32 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=8 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX8-RV32
+; RUN: llc -mtriple=riscv64 -mattr=+m,+experimental-v,+d -riscv-v-vector-bits-min=128 -riscv-v-fixed-length-vector-lmul-max=8 -verify-machineinstrs < %s | FileCheck %s --check-prefixes=LMULMAX8-RV64
 
 define void @cttz_v16i8(<16 x i8>* %x, <16 x i8>* %y) nounwind {
 ; LMULMAX2-RV32-LABEL: cttz_v16i8:
 ; LMULMAX2-RV32:       # %bb.0:
 ; LMULMAX2-RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
 ; LMULMAX2-RV32-NEXT:    vle8.v v8, (a0)
 ; LMULMAX2-RV32-NEXT:    addi a1, zero, 1
 ; LMULMAX2-RV32-NEXT:    vsub.vx v9, v8, a1

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 2
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vadd.vv v8, v9, v8
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v8, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v8, v8, v9
 ; LMULMAX1-RV64-NEXT:    vand.vi v8, v8, 15
 ; LMULMAX1-RV64-NEXT:    vse8.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v16i8:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; LMULMAX8-RV32-NEXT:    vzext.vf4 v12, v9
+; LMULMAX8-RV32-NEXT:    vfcvt.f.xu.v v12, v12
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v9, v10, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    vsub.vx v8, v9, a1
+; LMULMAX8-RV32-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV32-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v16i8:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e32, m4, ta, mu
+; LMULMAX8-RV64-NEXT:    vzext.vf4 v12, v9
+; LMULMAX8-RV64-NEXT:    vfcvt.f.xu.v v12, v12
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e16, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e8, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v9, v10, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    vsub.vx v8, v9, a1
+; LMULMAX8-RV64-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV64-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <16 x i8>, <16 x i8>* %x
   %b = load <16 x i8>, <16 x i8>* %y
   %c = call <16 x i8> @llvm.cttz.v16i8(<16 x i8> %a, i1 false)
   store <16 x i8> %c, <16 x i8>* %x
   ret void
 }
 declare <16 x i8> @llvm.cttz.v16i8(<16 x i8>, i1)
 
 define void @cttz_v8i16(<8 x i16>* %x, <8 x i16>* %y) nounwind {
-; LMULMAX2-RV32-LABEL: cttz_v8i16:
-; LMULMAX2-RV32:       # %bb.0:
-; LMULMAX2-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
-; LMULMAX2-RV32-NEXT:    vle16.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    addi a1, zero, 1
-; LMULMAX2-RV32-NEXT:    vsub.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV32-NEXT:    vand.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    lui a1, 5
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 1365
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV32-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 3
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 819
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 1
-; LMULMAX2-RV32-NEXT:    addi a1, a1, -241
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    addi a1, zero, 257
-; LMULMAX2-RV32-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 8
-; LMULMAX2-RV32-NEXT:    vse16.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    ret
-;
-; LMULMAX2-RV64-LABEL: cttz_v8i16:
-; LMULMAX2-RV64:       # %bb.0:
-; LMULMAX2-RV64-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
-; LMULMAX2-RV64-NEXT:    vle16.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    addi a1, zero, 1
-; LMULMAX2-RV64-NEXT:    vsub.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV64-NEXT:    vand.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    lui a1, 5
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 1365
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV64-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 3
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 819
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 1
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, -241
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    addi a1, zero, 257
-; LMULMAX2-RV64-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 8
-; LMULMAX2-RV64-NEXT:    vse16.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    ret
+; LMULMAX2-RV32I-LABEL: cttz_v8i16:
+; LMULMAX2-RV32I:       # %bb.0:
+; LMULMAX2-RV32I-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV32I-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    addi a1, zero, 1
+; LMULMAX2-RV32I-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV32I-NEXT:    vand.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    lui a1, 5
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 1365
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV32I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 3
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 819
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 1
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, -241
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    addi a1, zero, 257
+; LMULMAX2-RV32I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 8
+; LMULMAX2-RV32I-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    ret
+;
+; LMULMAX2-RV64I-LABEL: cttz_v8i16:
+; LMULMAX2-RV64I:       # %bb.0:
+; LMULMAX2-RV64I-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV64I-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    addi a1, zero, 1
+; LMULMAX2-RV64I-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV64I-NEXT:    vand.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    lui a1, 5
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 1365
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV64I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 3
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 819
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 1
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, -241
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    addi a1, zero, 257
+; LMULMAX2-RV64I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 8
+; LMULMAX2-RV64I-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    ret
 ;
 ; LMULMAX1-RV32-LABEL: cttz_v8i16:
 ; LMULMAX1-RV32:       # %bb.0:
 ; LMULMAX1-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
 ; LMULMAX1-RV32-NEXT:    vle16.v v8, (a0)
 ; LMULMAX1-RV32-NEXT:    addi a1, zero, 1
 ; LMULMAX1-RV32-NEXT:    vsub.vx v9, v8, a1
 ; LMULMAX1-RV32-NEXT:    vxor.vi v8, v8, -1

 ; LMULMAX1-RV64-NEXT:    lui a1, 1
 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, -241
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    addi a1, zero, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 8
 ; LMULMAX1-RV64-NEXT:    vse16.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX2-RV32D-LABEL: cttz_v8i16:
+; LMULMAX2-RV32D:       # %bb.0:
+; LMULMAX2-RV32D-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV32D-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX2-RV32D-NEXT:    vand.vv v9, v8, v9
+; LMULMAX2-RV32D-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX2-RV32D-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 127
+; LMULMAX2-RV32D-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX2-RV32D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 16
+; LMULMAX2-RV32D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV32D-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    ret
+;
+; LMULMAX2-RV64D-LABEL: cttz_v8i16:
+; LMULMAX2-RV64D:       # %bb.0:
+; LMULMAX2-RV64D-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX2-RV64D-NEXT:    vle16.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX2-RV64D-NEXT:    vand.vv v9, v8, v9
+; LMULMAX2-RV64D-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX2-RV64D-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 127
+; LMULMAX2-RV64D-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX2-RV64D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 16
+; LMULMAX2-RV64D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV64D-NEXT:    vse16.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v8i16:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV32-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV32-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v8i16:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 8, e16, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v9, v10, 23
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV64-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV64-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <8 x i16>, <8 x i16>* %x
   %b = load <8 x i16>, <8 x i16>* %y
   %c = call <8 x i16> @llvm.cttz.v8i16(<8 x i16> %a, i1 false)
   store <8 x i16> %c, <8 x i16>* %x
   ret void
 }
 declare <8 x i16> @llvm.cttz.v8i16(<8 x i16>, i1)
 
 define void @cttz_v4i32(<4 x i32>* %x, <4 x i32>* %y) nounwind {
-; LMULMAX2-RV32-LABEL: cttz_v4i32:
-; LMULMAX2-RV32:       # %bb.0:
-; LMULMAX2-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
-; LMULMAX2-RV32-NEXT:    vle32.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    addi a1, zero, 1
-; LMULMAX2-RV32-NEXT:    vsub.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV32-NEXT:    vand.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV32-NEXT:    lui a1, 349525
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 1365
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV32-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 209715
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 819
-; LMULMAX2-RV32-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV32-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV32-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV32-NEXT:    lui a1, 61681
-; LMULMAX2-RV32-NEXT:    addi a1, a1, -241
-; LMULMAX2-RV32-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    lui a1, 4112
-; LMULMAX2-RV32-NEXT:    addi a1, a1, 257
-; LMULMAX2-RV32-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV32-NEXT:    vsrl.vi v8, v8, 24
-; LMULMAX2-RV32-NEXT:    vse32.v v8, (a0)
-; LMULMAX2-RV32-NEXT:    ret
-;
-; LMULMAX2-RV64-LABEL: cttz_v4i32:
-; LMULMAX2-RV64:       # %bb.0:
-; LMULMAX2-RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
-; LMULMAX2-RV64-NEXT:    vle32.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    addi a1, zero, 1
-; LMULMAX2-RV64-NEXT:    vsub.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vxor.vi v8, v8, -1
-; LMULMAX2-RV64-NEXT:    vand.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 1
-; LMULMAX2-RV64-NEXT:    lui a1, 349525
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 1365
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v9, a1
-; LMULMAX2-RV64-NEXT:    vsub.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 209715
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 819
-; LMULMAX2-RV64-NEXT:    vand.vx v9, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 2
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v9, v8
-; LMULMAX2-RV64-NEXT:    vsrl.vi v9, v8, 4
-; LMULMAX2-RV64-NEXT:    vadd.vv v8, v8, v9
-; LMULMAX2-RV64-NEXT:    lui a1, 61681
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, -241
-; LMULMAX2-RV64-NEXT:    vand.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    lui a1, 4112
-; LMULMAX2-RV64-NEXT:    addiw a1, a1, 257
-; LMULMAX2-RV64-NEXT:    vmul.vx v8, v8, a1
-; LMULMAX2-RV64-NEXT:    vsrl.vi v8, v8, 24
-; LMULMAX2-RV64-NEXT:    vse32.v v8, (a0)
-; LMULMAX2-RV64-NEXT:    ret
+; LMULMAX2-RV32I-LABEL: cttz_v4i32:
+; LMULMAX2-RV32I:       # %bb.0:
+; LMULMAX2-RV32I-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV32I-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    addi a1, zero, 1
+; LMULMAX2-RV32I-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV32I-NEXT:    vand.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV32I-NEXT:    lui a1, 349525
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 1365
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV32I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 209715
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 819
+; LMULMAX2-RV32I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV32I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV32I-NEXT:    lui a1, 61681
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, -241
+; LMULMAX2-RV32I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    lui a1, 4112
+; LMULMAX2-RV32I-NEXT:    addi a1, a1, 257
+; LMULMAX2-RV32I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV32I-NEXT:    vsrl.vi v8, v8, 24
+; LMULMAX2-RV32I-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV32I-NEXT:    ret
+;
+; LMULMAX2-RV64I-LABEL: cttz_v4i32:
+; LMULMAX2-RV64I:       # %bb.0:
+; LMULMAX2-RV64I-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV64I-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    addi a1, zero, 1
+; LMULMAX2-RV64I-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX2-RV64I-NEXT:    vand.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX2-RV64I-NEXT:    lui a1, 349525
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 1365
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v9, a1
+; LMULMAX2-RV64I-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 209715
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 819
+; LMULMAX2-RV64I-NEXT:    vand.vx v9, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX2-RV64I-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX2-RV64I-NEXT:    lui a1, 61681
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, -241
+; LMULMAX2-RV64I-NEXT:    vand.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    lui a1, 4112
+; LMULMAX2-RV64I-NEXT:    addiw a1, a1, 257
+; LMULMAX2-RV64I-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX2-RV64I-NEXT:    vsrl.vi v8, v8, 24
+; LMULMAX2-RV64I-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV64I-NEXT:    ret
 ;
 ; LMULMAX1-RV32-LABEL: cttz_v4i32:
 ; LMULMAX1-RV32:       # %bb.0:
 ; LMULMAX1-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
 ; LMULMAX1-RV32-NEXT:    vle32.v v8, (a0)
 ; LMULMAX1-RV32-NEXT:    addi a1, zero, 1
 ; LMULMAX1-RV32-NEXT:    vsub.vx v9, v8, a1
 ; LMULMAX1-RV32-NEXT:    vxor.vi v8, v8, -1

 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, -241
 ; LMULMAX1-RV64-NEXT:    vand.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    lui a1, 4112
 ; LMULMAX1-RV64-NEXT:    addiw a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v8, v8, 24
 ; LMULMAX1-RV64-NEXT:    vse32.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX2-RV32D-LABEL: cttz_v4i32:
+; LMULMAX2-RV32D:       # %bb.0:
+; LMULMAX2-RV32D-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV32D-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX2-RV32D-NEXT:    vand.vv v9, v8, v9
+; LMULMAX2-RV32D-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 52
+; LMULMAX2-RV32D-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 1023
+; LMULMAX2-RV32D-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX2-RV32D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV32D-NEXT:    addi a1, zero, 32
+; LMULMAX2-RV32D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV32D-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV32D-NEXT:    ret
+;
+; LMULMAX2-RV64D-LABEL: cttz_v4i32:
+; LMULMAX2-RV64D:       # %bb.0:
+; LMULMAX2-RV64D-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX2-RV64D-NEXT:    vle32.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX2-RV64D-NEXT:    vand.vv v9, v8, v9
+; LMULMAX2-RV64D-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 52
+; LMULMAX2-RV64D-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 1023
+; LMULMAX2-RV64D-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX2-RV64D-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX2-RV64D-NEXT:    addi a1, zero, 32
+; LMULMAX2-RV64D-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX2-RV64D-NEXT:    vse32.v v8, (a0)
+; LMULMAX2-RV64D-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v4i32:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV32-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1023
+; LMULMAX8-RV32-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV32-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v4i32:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v9, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v9, v8, v9
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v10, v9
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV64-NEXT:    vnsrl.wx v9, v10, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1023
+; LMULMAX8-RV64-NEXT:    vsub.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v9, a1, v0
+; LMULMAX8-RV64-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <4 x i32>, <4 x i32>* %x
   %b = load <4 x i32>, <4 x i32>* %y
   %c = call <4 x i32> @llvm.cttz.v4i32(<4 x i32> %a, i1 false)
   store <4 x i32> %c, <4 x i32>* %x
   ret void
 }
 declare <4 x i32> @llvm.cttz.v4i32(<4 x i32>, i1)
 

 ; LMULMAX1-RV64-NEXT:    addi a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    slli a1, a1, 16
 ; LMULMAX1-RV64-NEXT:    addi a1, a1, 257
 ; LMULMAX1-RV64-NEXT:    vmul.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    addi a1, zero, 56
 ; LMULMAX1-RV64-NEXT:    vsrl.vx v8, v8, a1
 ; LMULMAX1-RV64-NEXT:    vse64.v v8, (a0)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v2i64:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1
+; LMULMAX8-RV32-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.i v10, -1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vxor.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV32-NEXT:    lui a1, 349525
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v9, v9, v10
+; LMULMAX8-RV32-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 209715
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v8, v9
+; LMULMAX8-RV32-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v10, v8
+; LMULMAX8-RV32-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 61681
+; LMULMAX8-RV32-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    lui a1, 4112
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e32, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v9, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV32-NEXT:    vmul.vv v8, v8, v9
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV32-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV32-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v2i64:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 2, e64, m1, ta, mu
+; LMULMAX8-RV64-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1
+; LMULMAX8-RV64-NEXT:    vsub.vx v9, v8, a1
+; LMULMAX8-RV64-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX8-RV64-NEXT:    vand.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 1
+; LMULMAX8-RV64-NEXT:    lui a1, 21845
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    vand.vx v9, v9, a1
+; LMULMAX8-RV64-NEXT:    vsub.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    lui a1, 13107
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    vand.vx v9, v8, a1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v9, v8
+; LMULMAX8-RV64-NEXT:    vsrl.vi v9, v8, 4
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v8, v9
+; LMULMAX8-RV64-NEXT:    lui a1, 3855
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    lui a1, 4112
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV64-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <2 x i64>, <2 x i64>* %x
   %b = load <2 x i64>, <2 x i64>* %y
   %c = call <2 x i64> @llvm.cttz.v2i64(<2 x i64> %a, i1 false)
   store <2 x i64> %c, <2 x i64>* %x
   ret void
 }
 declare <2 x i64> @llvm.cttz.v2i64(<2 x i64>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v10, v9
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vi v9, v9, 15
 ; LMULMAX1-RV64-NEXT:    vse8.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse8.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v32i8:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; LMULMAX8-RV32-NEXT:    vzext.vf4 v16, v10
+; LMULMAX8-RV32-NEXT:    vfcvt.f.xu.v v16, v16
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v12, v16, 23
+; LMULMAX8-RV32-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    vsub.vx v8, v10, a1
+; LMULMAX8-RV32-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV32-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v32i8:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vsetvli zero, a1, e8, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e32, m8, ta, mu
+; LMULMAX8-RV64-NEXT:    vzext.vf4 v16, v10
+; LMULMAX8-RV64-NEXT:    vfcvt.f.xu.v v16, v16
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e16, m4, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v12, v16, 23
+; LMULMAX8-RV64-NEXT:    vsetvli zero, zero, e8, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    vsub.vx v8, v10, a1
+; LMULMAX8-RV64-NEXT:    vmerge.vim v8, v8, 8, v0
+; LMULMAX8-RV64-NEXT:    vse8.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <32 x i8>, <32 x i8>* %x
   %b = load <32 x i8>, <32 x i8>* %y
   %c = call <32 x i8> @llvm.cttz.v32i8(<32 x i8> %a, i1 false)
   store <32 x i8> %c, <32 x i8>* %x
   ret void
 }
 declare <32 x i8> @llvm.cttz.v32i8(<32 x i8>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a2
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v9, 8
 ; LMULMAX1-RV64-NEXT:    vse16.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse16.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v16i16:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v12, v10
+; LMULMAX8-RV32-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV32-NEXT:    vsub.vx v10, v10, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV32-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v16i16:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 16, e16, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v12, v10
+; LMULMAX8-RV64-NEXT:    vnsrl.wi v10, v12, 23
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 127
+; LMULMAX8-RV64-NEXT:    vsub.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 16
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV64-NEXT:    vse16.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <16 x i16>, <16 x i16>* %x
   %b = load <16 x i16>, <16 x i16>* %y
   %c = call <16 x i16> @llvm.cttz.v16i16(<16 x i16> %a, i1 false)
   store <16 x i16> %c, <16 x i16>* %x
   ret void
 }
 declare <16 x i16> @llvm.cttz.v16i16(<16 x i16>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a2
 ; LMULMAX1-RV64-NEXT:    vsrl.vi v9, v9, 24
 ; LMULMAX1-RV64-NEXT:    vse32.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse32.v v8, (a1)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v8i32:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV32-NEXT:    vfwcvt.f.xu.v v12, v10
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV32-NEXT:    vnsrl.wx v10, v12, a1
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1023
+; LMULMAX8-RV32-NEXT:    vsub.vx v10, v10, a1
+; LMULMAX8-RV32-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV32-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV32-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v8i32:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    vrsub.vi v10, v8, 0
+; LMULMAX8-RV64-NEXT:    vand.vv v10, v8, v10
+; LMULMAX8-RV64-NEXT:    vfwcvt.f.xu.v v12, v10
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 52
+; LMULMAX8-RV64-NEXT:    vnsrl.wx v10, v12, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1023
+; LMULMAX8-RV64-NEXT:    vsub.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vmseq.vi v0, v8, 0
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 32
+; LMULMAX8-RV64-NEXT:    vmerge.vxm v8, v10, a1, v0
+; LMULMAX8-RV64-NEXT:    vse32.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <8 x i32>, <8 x i32>* %x
   %b = load <8 x i32>, <8 x i32>* %y
   %c = call <8 x i32> @llvm.cttz.v8i32(<8 x i32> %a, i1 false)
   store <8 x i32> %c, <8 x i32>* %x
   ret void
 }
 declare <8 x i32> @llvm.cttz.v8i32(<8 x i32>, i1)
 

 ; LMULMAX1-RV64-NEXT:    vsrl.vi v10, v9, 4
 ; LMULMAX1-RV64-NEXT:    vadd.vv v9, v9, v10
 ; LMULMAX1-RV64-NEXT:    vand.vx v9, v9, a5
 ; LMULMAX1-RV64-NEXT:    vmul.vx v9, v9, a2
 ; LMULMAX1-RV64-NEXT:    vsrl.vx v9, v9, a1
 ; LMULMAX1-RV64-NEXT:    vse64.v v9, (a0)
 ; LMULMAX1-RV64-NEXT:    vse64.v v8, (a7)
 ; LMULMAX1-RV64-NEXT:    ret
+;
+; LMULMAX8-RV32-LABEL: cttz_v4i64:
+; LMULMAX8-RV32:       # %bb.0:
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 1
+; LMULMAX8-RV32-NEXT:    vsub.vx v10, v8, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.i v12, -1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vxor.vv v8, v8, v12
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV32-NEXT:    lui a1, 349525
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v12, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v10, v10, v12
+; LMULMAX8-RV32-NEXT:    vsub.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 209715
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v12, v8, v10
+; LMULMAX8-RV32-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v12, v8
+; LMULMAX8-RV32-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV32-NEXT:    vadd.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 61681
+; LMULMAX8-RV32-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    lui a1, 4112
+; LMULMAX8-RV32-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 8, e32, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmv.v.x v10, a1
+; LMULMAX8-RV32-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV32-NEXT:    vmul.vv v8, v8, v10
+; LMULMAX8-RV32-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV32-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV32-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV32-NEXT:    ret
+;
+; LMULMAX8-RV64-LABEL: cttz_v4i64:
+; LMULMAX8-RV64:       # %bb.0:
+; LMULMAX8-RV64-NEXT:    vsetivli zero, 4, e64, m2, ta, mu
+; LMULMAX8-RV64-NEXT:    vle64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 1
+; LMULMAX8-RV64-NEXT:    vsub.vx v10, v8, a1
+; LMULMAX8-RV64-NEXT:    vxor.vi v8, v8, -1
+; LMULMAX8-RV64-NEXT:    vand.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 1
+; LMULMAX8-RV64-NEXT:    lui a1, 21845
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 1365
+; LMULMAX8-RV64-NEXT:    vand.vx v10, v10, a1
+; LMULMAX8-RV64-NEXT:    vsub.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    lui a1, 13107
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 819
+; LMULMAX8-RV64-NEXT:    vand.vx v10, v8, a1
+; LMULMAX8-RV64-NEXT:    vsrl.vi v8, v8, 2
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v10, v8
+; LMULMAX8-RV64-NEXT:    vsrl.vi v10, v8, 4
+; LMULMAX8-RV64-NEXT:    vadd.vv v8, v8, v10
+; LMULMAX8-RV64-NEXT:    lui a1, 3855
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 241
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 12
+; LMULMAX8-RV64-NEXT:    addi a1, a1, -241
+; LMULMAX8-RV64-NEXT:    vand.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    lui a1, 4112
+; LMULMAX8-RV64-NEXT:    addiw a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    slli a1, a1, 16
+; LMULMAX8-RV64-NEXT:    addi a1, a1, 257
+; LMULMAX8-RV64-NEXT:    vmul.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    addi a1, zero, 56
+; LMULMAX8-RV64-NEXT:    vsrl.vx v8, v8, a1
+; LMULMAX8-RV64-NEXT:    vse64.v v8, (a0)
+; LMULMAX8-RV64-NEXT:    ret
   %a = load <4 x i64>, <4 x i64>* %x
   %b = load <4 x i64>, <4 x i64>* %y
   %c = call <4 x i64> @llvm.cttz.v4i64(<4 x i64> %a, i1 false)
   store <4 x i64> %c, <4 x i64>* %x
   ret void
 }
 declare <4 x i64> @llvm.cttz.v4i64(<4 x i64>, i1)