Improve WebAssembly vector bitmask, mask reduction, and extending

[calebzulawski]

Summary

This is inspired by a recently filed Rust issue noting poor codegen for vector masks (https://github.com/rust-lang/portable-simd/issues/351). I'm new to wasm, so I simply followed the reported issue and the wasm spec.

I'm not sure who to add to review this, so feel free to add others or remove yourself.

[calebzulawski]

Use sext instead of zext (better codegen for vecreduce, and necessary for bitmask correctness)

[tlively]

This looks really good! It's nice to see the improved codegen on the existing tests.

llvm/test/CodeGen/WebAssembly/simd-vecreduce-bool.ll

27–30

Could we skip this sign extension before the v128.any_true? As long as the high bits of each input lane are zeroed (which I believe they should be), the sign extension doesn't affect the outcome of v128.any_true.

[calebzulawski]

Thanks! By the way, I don't have commit access to the repo, if you are able to commit for me.

llvm/test/CodeGen/WebAssembly/simd-vecreduce-bool.ll

27–30

I originally thought this as well, but I think the shifts are inserted because the other lanes are undef instead of zero, if that's possible. I added the case at the end (test_cmp_v16i8) where all bits are known, and the shifts disappear.

[Sp00ph]

llvm/test/CodeGen/WebAssembly/simd-vecreduce-bool.ll

27–30

The high bits of <N x i1> are unspecified, which also previously led to miscompilations on AArch64, see this issue: https://github.com/llvm/llvm-project/issues/62211

[tlively]

I'll go ahead and land this.

llvm/test/CodeGen/WebAssembly/simd-vecreduce-bool.ll

27–30

Great, thanks for the clarification on that.

[srj]

This change has injected a SIGSEGV failure (apparent null pointer deference) in Halide's WebAssembly codegen. I'm working on providing a repro case for you I can post here and hope to have it soon, but in the meantime, can I ask for a revert, to unbreak downstream code?

[srj]

FYI, building with an asserts-enabled LLVM, I fail with:

llvm-project/17/llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp:2770: llvm::SDValue performBitcastCombine(llvm::SDNode*, llvm::TargetLowering::DAGCombinerInfo&): Assertion `NumElts == 2 || NumElts == 4 || NumElts == 8 || NumElts == 16' failed.

Stand by for a repro.

[srj]

Enclosed is a repro case: if I just run llc breakage.ll I fail in the above manner.

breakage.ll27 KBDownload

[tlively]

Sorry for the breakage! Taking a look now. If a fix looks nontrivial, I'll revert.

[calebzulawski]

In D151782#4406514, @tlively wrote:

Sorry for the breakage! Taking a look now. If a fix looks nontrivial, I'll revert.

I've already identified the issue, the assert is triggering due to vectors of length 1. In that case, the combine step should be skipped.

Also, I noticed only the modified tests made it into the commit, not the new tests I added. Would you like me to open a new review with the changes? Thanks!

[tlively]

Oops, sorry for missing those tests, Caleb. Yes, opening a new review with the fix and tests would be great. Thanks!

[srj]

In D151782#4406550, @tlively wrote:

Oops, sorry for missing those tests, Caleb. Yes, opening a new review with the fix and tests would be great. Thanks!

So if I understand correctly, you are working on a fix-forward rather than a revert? (If so, please cc me on the fix-forward review)

[tlively]

Yes, my understanding is that @calebzulawski is working on a fix. Since I don't know what the expected timeline is, I'll revert in the meantime.

[calebzulawski]

I'm hoping to have it ready today, but a revert is fine with me

[calebzulawski]

Since this has been reverted, I'm just updating the complete patch here.

All I changed was allowing unexpected vector lengths to lower as usual. Codegen for unusual vector lengths is still very bad--there should be an additional step somewhere that breaks up bitcasts to legal vector lengths (the X86 codegen does this before MOVMSK, but I can't figure out how).

[tlively]

Thanks! I'll land this again and make sure to include all the tests this time.

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.h

private:
   SDValue LowerGlobalTLSAddress(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerExternalSymbol(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBR_JT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerJumpTable(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVASTART(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCopyToReg(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerIntrinsic(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSIGN_EXTEND_INREG(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerEXTEND_VECTOR_INREG(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECTOR_SHUFFLE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSETCC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerAccessVectorElement(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShift(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_TO_INT_SAT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerLoad(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerStore(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp

WebAssemblyTargetLowering::WebAssemblyTargetLowering(
 
   if (Subtarget->hasNontrappingFPToInt())
     for (auto Op : {ISD::FP_TO_SINT_SAT, ISD::FP_TO_UINT_SAT})
       for (auto T : {MVT::i32, MVT::i64})
         setOperationAction(Op, T, Custom);
 
   // SIMD-specific configuration
   if (Subtarget->hasSIMD128()) {
+    // Combine vector mask reductions into alltrue/anytrue
+    setTargetDAGCombine(ISD::SETCC);
+
+    // Convert vector to integer bitcasts to bitmask
+    setTargetDAGCombine(ISD::BITCAST);
+
     // Hoist bitcasts out of shuffles
     setTargetDAGCombine(ISD::VECTOR_SHUFFLE);
 
     // Combine extends of extract_subvectors into widening ops
     setTargetDAGCombine({ISD::SIGN_EXTEND, ISD::ZERO_EXTEND});
 
     // Combine int_to_fp or fp_extend of extract_vectors and vice versa into
     // conversions ops
if (Subtarget->hasSIMD128()) {
     for (auto Op :
          {ISD::SINT_TO_FP, ISD::UINT_TO_FP, ISD::FP_TO_SINT, ISD::FP_TO_UINT})
       for (auto T : {MVT::v2i64, MVT::v2f64})
         setOperationAction(Op, T, Expand);
 
     // But saturating fp_to_int converstions are
     for (auto Op : {ISD::FP_TO_SINT_SAT, ISD::FP_TO_UINT_SAT})
       setOperationAction(Op, MVT::v4i32, Custom);
+
+    // Support vector extending
+    for (auto T : MVT::integer_fixedlen_vector_valuetypes()) {
+      setOperationAction(ISD::SIGN_EXTEND_VECTOR_INREG, T, Custom);
+      setOperationAction(ISD::ZERO_EXTEND_VECTOR_INREG, T, Custom);
+    }
   }
 
   // As a special case, these operators use the type to mean the type to
   // sign-extend from.
   setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i1, Expand);
   if (!Subtarget->hasSignExt()) {
     // Sign extends are legal only when extending a vector extract
     auto Action = Subtarget->hasSIMD128() ? Custom : Expand;
void WebAssemblyTargetLowering::ReplaceNodeResults(
     SDNode *N, SmallVectorImpl<SDValue> &Results, SelectionDAG &DAG) const {
   switch (N->getOpcode()) {
   case ISD::SIGN_EXTEND_INREG:
     // Do not add any results, signifying that N should not be custom lowered
     // after all. This happens because simd128 turns on custom lowering for
     // SIGN_EXTEND_INREG, but for non-vector sign extends the result might be an
     // illegal type.
     break;
+  case ISD::SIGN_EXTEND_VECTOR_INREG:
+  case ISD::ZERO_EXTEND_VECTOR_INREG:
+    // Do not add any results, signifying that N should not be custom lowered.
+    // EXTEND_VECTOR_INREG is implemented for some vectors, but not all.
+    break;
   default:
     llvm_unreachable(
         "ReplaceNodeResults not implemented for this op for WebAssembly!");
   }
 }
 
 //===----------------------------------------------------------------------===//
 //  Custom lowering hooks.
SDValue WebAssemblyTargetLowering::LowerOperation(SDValue Op,
   case ISD::INSERT_VECTOR_ELT:
     return LowerAccessVectorElement(Op, DAG);
   case ISD::INTRINSIC_VOID:
   case ISD::INTRINSIC_WO_CHAIN:
   case ISD::INTRINSIC_W_CHAIN:
     return LowerIntrinsic(Op, DAG);
   case ISD::SIGN_EXTEND_INREG:
     return LowerSIGN_EXTEND_INREG(Op, DAG);
+  case ISD::ZERO_EXTEND_VECTOR_INREG:
+  case ISD::SIGN_EXTEND_VECTOR_INREG:
+    return LowerEXTEND_VECTOR_INREG(Op, DAG);
   case ISD::BUILD_VECTOR:
     return LowerBUILD_VECTOR(Op, DAG);
   case ISD::VECTOR_SHUFFLE:
     return LowerVECTOR_SHUFFLE(Op, DAG);
   case ISD::SETCC:
     return LowerSETCC(Op, DAG);
   case ISD::SHL:
   case ISD::SRA:
SDValue NewIndex =
       DAG.getConstant(IndexVal * Scale, DL, Index->getValueType(0));
   SDValue NewExtract = DAG.getNode(
       ISD::EXTRACT_VECTOR_ELT, DL, Extract.getValueType(),
       DAG.getBitcast(ExtractedVecT, Extract.getOperand(0)), NewIndex);
   return DAG.getNode(ISD::SIGN_EXTEND_INREG, DL, Op.getValueType(), NewExtract,
                      Op.getOperand(1));
 }
 
+SDValue
+WebAssemblyTargetLowering::LowerEXTEND_VECTOR_INREG(SDValue Op,
+                                                    SelectionDAG &DAG) const {
+  SDLoc DL(Op);
+  EVT VT = Op.getValueType();
+  SDValue Src = Op.getOperand(0);
+  EVT SrcVT = Src.getValueType();
+
+  if (SrcVT.getVectorElementType() == MVT::i1 ||
+      SrcVT.getVectorElementType() == MVT::i64)
+    return SDValue();
+
+  assert(VT.getScalarSizeInBits() % SrcVT.getScalarSizeInBits() == 0 &&
+         "Unexpected extension factor.");
+  unsigned Scale = VT.getScalarSizeInBits() / SrcVT.getScalarSizeInBits();
+
+  if (Scale != 2 && Scale != 4 && Scale != 8)
+    return SDValue();
+
+  unsigned Ext;
+  switch (Op.getOpcode()) {
+  case ISD::ZERO_EXTEND_VECTOR_INREG:
+    Ext = WebAssemblyISD::EXTEND_LOW_U;
+    break;
+  case ISD::SIGN_EXTEND_VECTOR_INREG:
+    Ext = WebAssemblyISD::EXTEND_LOW_S;
+    break;
+  }
+
+  SDValue Ret = Src;
+  while (Scale != 1) {
+    Ret = DAG.getNode(Ext, DL,
+                      Ret.getValueType()
+                          .widenIntegerVectorElementType(*DAG.getContext())
+                          .getHalfNumVectorElementsVT(*DAG.getContext()),
+                      Ret);
+    Scale /= 2;
+  }
+  assert(Ret.getValueType() == VT);
+  return Ret;
+}
+
 static SDValue LowerConvertLow(SDValue Op, SelectionDAG &DAG) {
   SDLoc DL(Op);
   if (Op.getValueType() != MVT::v2f64)
     return SDValue();
 
   auto GetConvertedLane = [](SDValue Op, unsigned &Opcode, SDValue &SrcVec,
                              unsigned &Index) -> bool {
     switch (Op.getOpcode()) {
static SDValue performTruncateCombine(SDNode *N,
 
   SDLoc DL(N);
   APInt Mask = APInt::getLowBitsSet(InVT.getScalarSizeInBits(),
                                     OutVT.getScalarSizeInBits());
   In = DAG.getNode(ISD::AND, DL, InVT, In, DAG.getConstant(Mask, DL, InVT));
   return truncateVectorWithNARROW(OutVT, In, DL, DAG);
 }
 
+static SDValue performBitcastCombine(SDNode *N,
+                                     TargetLowering::DAGCombinerInfo &DCI) {
+  auto &DAG = DCI.DAG;
+  SDLoc DL(N);
+  SDValue Src = N->getOperand(0);
+  EVT VT = N->getValueType(0);
+  EVT SrcVT = Src.getValueType();
+
+  // bitcast <N x i1> to iN
+  //   ==> bitmask
+  if (DCI.isBeforeLegalize() && VT.isScalarInteger() &&
+      SrcVT.isFixedLengthVector() && SrcVT.getScalarType() == MVT::i1) {
+    unsigned NumElts = SrcVT.getVectorNumElements();
+    assert(NumElts == 2 || NumElts == 4 || NumElts == 8 || NumElts == 16);
+    EVT Width = MVT::getIntegerVT(128 / NumElts);
+    return DAG.getZExtOrTrunc(
+        DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, MVT::i32,
+                    {DAG.getConstant(Intrinsic::wasm_bitmask, DL, MVT::i32),
+                     DAG.getSExtOrTrunc(N->getOperand(0), DL,
+                                        SrcVT.changeVectorElementType(Width))}),
+        DL, VT);
+  }
+
+  return SDValue();
+}
+
+static SDValue performSETCCCombine(SDNode *N,
+                                   TargetLowering::DAGCombinerInfo &DCI) {
+  auto &DAG = DCI.DAG;
+
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  ISD::CondCode Cond = cast<CondCodeSDNode>(N->getOperand(2))->get();
+  SDLoc DL(N);
+  EVT VT = N->getValueType(0);
+
+  // setcc (iN (bitcast (vNi1 X))), 0, ne
+  //   ==> any_true (vNi1 X)
+  // setcc (iN (bitcast (vNi1 X))), 0, eq
+  //   ==> xor (any_true (vNi1 X)), -1
+  // setcc (iN (bitcast (vNi1 X))), -1, eq
+  //   ==> all_true (vNi1 X)
+  // setcc (iN (bitcast (vNi1 X))), -1, ne
+  //   ==> xor (all_true (vNi1 X)), -1
+  if (DCI.isBeforeLegalize() && VT.isScalarInteger() &&
+      (Cond == ISD::SETEQ || Cond == ISD::SETNE) &&
+      (isNullConstant(RHS) || isAllOnesConstant(RHS)) &&
+      LHS->getOpcode() == ISD::BITCAST) {
+    EVT FromVT = LHS->getOperand(0).getValueType();
+    if (FromVT.isFixedLengthVector() &&
+        FromVT.getVectorElementType() == MVT::i1) {
+      int Intrin = isNullConstant(RHS) ? Intrinsic::wasm_anytrue
+                                       : Intrinsic::wasm_alltrue;
+      unsigned NumElts = FromVT.getVectorNumElements();
+      assert(NumElts == 2 || NumElts == 4 || NumElts == 8 || NumElts == 16);
+      EVT Width = MVT::getIntegerVT(128 / NumElts);
+      SDValue Ret = DAG.getZExtOrTrunc(
+          DAG.getNode(
+              ISD::INTRINSIC_WO_CHAIN, DL, MVT::i32,
+              {DAG.getConstant(Intrin, DL, MVT::i32),
+               DAG.getSExtOrTrunc(LHS->getOperand(0), DL,
+                                  FromVT.changeVectorElementType(Width))}),
+          DL, MVT::i1);
+      if ((isNullConstant(RHS) && (Cond == ISD::SETEQ)) ||
+          (isAllOnesConstant(RHS) && (Cond == ISD::SETNE))) {
+        Ret = DAG.getNOT(DL, Ret, MVT::i1);
+      }
+      return DAG.getZExtOrTrunc(Ret, DL, VT);
+    }
+  }
+
+  return SDValue();
+}
+
 SDValue
 WebAssemblyTargetLowering::PerformDAGCombine(SDNode *N,
                                              DAGCombinerInfo &DCI) const {
   switch (N->getOpcode()) {
   default:
     return SDValue();
+  case ISD::BITCAST:
+    return performBitcastCombine(N, DCI);
+  case ISD::SETCC:
+    return performSETCCCombine(N, DCI);
   case ISD::VECTOR_SHUFFLE:
     return performVECTOR_SHUFFLECombine(N, DCI);
   case ISD::SIGN_EXTEND:
   case ISD::ZERO_EXTEND:
     return performVectorExtendCombine(N, DCI);
   case ISD::UINT_TO_FP:
   case ISD::SINT_TO_FP:
     return performVectorExtendToFPCombine(N, DCI);

llvm/test/CodeGen/WebAssembly/simd-bitmask.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mattr=+simd128 | FileCheck %s
+
+;; Test that SIMD bitmask instruction can be selected
+
+target triple = "wasm32-unknown-unknown"
+
+define i16 @bitmask_v16i8(<16 x i8> %v) {
+; CHECK-LABEL: bitmask_v16i8:
+; CHECK:         .functype bitmask_v16i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.eq
+; CHECK-NEXT:    i8x16.bitmask
+; CHECK-NEXT:    # fallthrough-return
+  %cmp = icmp eq <16 x i8> %v, zeroinitializer
+  %bitmask = bitcast <16 x i1> %cmp to i16
+  ret i16 %bitmask
+}
+
+define i8 @bitmask_v8i16(<8 x i16> %v) {
+; CHECK-LABEL: bitmask_v8i16:
+; CHECK:         .functype bitmask_v8i16 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i16x8.eq
+; CHECK-NEXT:    i16x8.bitmask
+; CHECK-NEXT:    # fallthrough-return
+  %cmp = icmp eq <8 x i16> %v, zeroinitializer
+  %bitmask = bitcast <8 x i1> %cmp to i8
+  ret i8 %bitmask
+}
+
+define i8 @bitmask_v4i32(<4 x i32> %v) {
+; CHECK-LABEL: bitmask_v4i32:
+; CHECK:         .functype bitmask_v4i32 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 0, 0, 0, 0
+; CHECK-NEXT:    i32x4.eq
+; CHECK-NEXT:    i32x4.bitmask
+; CHECK-NEXT:    # fallthrough-return
+  %cmp = icmp eq <4 x i32> %v, zeroinitializer
+  %bitmask = bitcast <4 x i1> %cmp to i4
+  %ext = zext i4 %bitmask to i8
+  ret i8 %ext
+}
+
+define i8 @bitmask_v2i64(<2 x i64> %v) {
+; CHECK-LABEL: bitmask_v2i64:
+; CHECK:         .functype bitmask_v2i64 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 0, 0
+; CHECK-NEXT:    i64x2.eq
+; CHECK-NEXT:    i64x2.bitmask
+; CHECK-NEXT:    # fallthrough-return
+  %cmp = icmp eq <2 x i64> %v, zeroinitializer
+  %bitmask = bitcast <2 x i1> %cmp to i2
+  %ext = zext i2 %bitmask to i8
+  ret i8 %ext
+}
+
+;; Test a smaller vector
+
+define i8 @bitmask_v8i8(<8 x i8> %v) {
+; CHECK-LABEL: bitmask_v8i8:
+; CHECK:         .functype bitmask_v8i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.eq
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i16x8.bitmask
+; CHECK-NEXT:    # fallthrough-return
+  %cmp = icmp eq <8 x i8> %v, zeroinitializer
+  %bitmask = bitcast <8 x i1> %cmp to i8
+  ret i8 %bitmask
+}

llvm/test/CodeGen/WebAssembly/simd-extending-convert.ll

; CHECK-NEXT:    # fallthrough-return
   %extended = uitofp <4 x i16> %high to <4 x float>
   ret <4 x float> %extended
 }
 
 define <4 x float> @extend_to_float_low_i8x16_u(<8 x i8> %x) {
 ; CHECK-LABEL: extend_to_float_low_i8x16_u:
 ; CHECK:         .functype extend_to_float_low_i8x16_u (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    f32x4.convert_i32x4_u
 ; CHECK-NEXT:    # fallthrough-return
   %low = shufflevector <8 x i8> %x, <8 x i8> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %extended = uitofp <4 x i8> %low to <4 x float>
   ret <4 x float> %extended
 }
 
 define <4 x float> @extend_to_float_high_i8x16_u(<8 x i8> %x) {
 ; CHECK-LABEL: extend_to_float_high_i8x16_u:
 ; CHECK:         .functype extend_to_float_high_i8x16_u (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
-; CHECK-NEXT:    i8x16.shuffle 4, 17, 18, 19, 5, 21, 22, 23, 6, 25, 26, 27, 7, 29, 30, 31
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i8x16.shuffle 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    f32x4.convert_i32x4_u
 ; CHECK-NEXT:    # fallthrough-return
   %high = shufflevector <8 x i8> %x, <8 x i8> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
   %extended = uitofp <4 x i8> %high to <4 x float>
   ret <4 x float> %extended
 }
 
 define <4 x float> @extend_to_float_low_i16x8_s(<8 x i16> %x) {
; CHECK-NEXT:    # fallthrough-return
   ret <4 x float> %extended
 }
 
 define <4 x float> @extend_to_float_low_i8x16_s(<8 x i8> %x) {
 ; CHECK-LABEL: extend_to_float_low_i8x16_s:
 ; CHECK:         .functype extend_to_float_low_i8x16_s (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    f32x4.convert_i32x4_s
 ; CHECK-NEXT:    # fallthrough-return
   %low = shufflevector <8 x i8> %x, <8 x i8> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %extended = sitofp <4 x i8> %low to <4 x float>
   ret <4 x float> %extended
 }
 
 define <4 x float> @extend_to_float_high_i8x16_s(<8 x i8> %x) {
 ; CHECK-LABEL: extend_to_float_high_i8x16_s:
 ; CHECK:         .functype extend_to_float_high_i8x16_s (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 4, 0, 0, 0, 5, 0, 0, 0, 6, 0, 0, 0, 7, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i8x16.shuffle 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    f32x4.convert_i32x4_s
 ; CHECK-NEXT:    # fallthrough-return
   %high = shufflevector <8 x i8> %x, <8 x i8> undef, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
   %extended = sitofp <4 x i8> %high to <4 x float>
   ret <4 x float> %extended
 }
 
 define <2 x double> @extend_to_double_low_i32x4_u(<4 x i32> %x) {
 ; CHECK-LABEL: extend_to_double_low_i32x4_u:
 ; CHECK:         .functype extend_to_double_low_i32x4_u (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    # fallthrough-return
   %low = shufflevector <4 x i32> %x, <4 x i32> undef, <2 x i32> <i32 0, i32 1>
   %extended = uitofp <2 x i32> %low to <2 x double>
   ret <2 x double> %extended
 }
 
 define <2 x double> @extend_to_double_low_i16x4_u(<4 x i16> %x) {
 ; CHECK-LABEL: extend_to_double_low_i16x4_u:
 ; CHECK:         .functype extend_to_double_low_i16x4_u (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 16, 17, 2, 3, 18, 19, 6, 7, 20, 21, 10, 11, 22, 23, 14, 15
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    # fallthrough-return
   %low = shufflevector <4 x i16> %x, <4 x i16> undef, <2 x i32> <i32 0, i32 1>
   %extended = uitofp <2 x i16> %low to <2 x double>
   ret <2 x double> %extended
 }

llvm/test/CodeGen/WebAssembly/simd-extending.ll

 ;; the low or high half are correctly expanded.
 
 define <8 x i16> @extend_lowish_i8x16_s(<16 x i8> %v) {
 ; CHECK-LABEL: extend_lowish_i8x16_s:
 ; CHECK:         .functype extend_lowish_i8x16_s (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 1, 0, 2, 0, 3, 0, 4, 0, 5, 0, 6, 0, 7, 0, 8, 0
-; CHECK-NEXT:    i32.const 8
-; CHECK-NEXT:    i16x8.shl
-; CHECK-NEXT:    i32.const 8
-; CHECK-NEXT:    i16x8.shr_s
+; CHECK-NEXT:    i8x16.shuffle 1, 2, 3, 4, 5, 6, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
 ; CHECK-NEXT:    # fallthrough-return
   %lowish = shufflevector <16 x i8> %v, <16 x i8> undef,
            <8 x i32> <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8>
   %extended = sext <8 x i8> %lowish to <8 x i16>
   ret <8 x i16> %extended
 }
 
 define <4 x i32> @extend_lowish_i16x8_s(<8 x i16> %v) {
 ; CHECK-LABEL: extend_lowish_i16x8_s:
 ; CHECK:         .functype extend_lowish_i16x8_s (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 2, 3, 0, 1, 4, 5, 0, 1, 6, 7, 0, 1, 8, 9, 0, 1
-; CHECK-NEXT:    i32.const 16
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 16
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i8x16.shuffle 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 0, 1, 0, 1, 0, 1
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %lowish = shufflevector <8 x i16> %v, <8 x i16> undef,
            <4 x i32> <i32 1, i32 2, i32 3, i32 4>
   %extended = sext <4 x i16> %lowish to <4 x i32>
   ret <4 x i32> %extended
 }
+
+;; Also test vectors that aren't full 128 bits, or might require
+;; multiple extensions
+
+define <16 x i8> @extend_i1x16_i8(<16 x i1> %v) {
+; CHECK-LABEL: extend_i1x16_i8:
+; CHECK:         .functype extend_i1x16_i8 (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 7
+; CHECK-NEXT:    i8x16.shl
+; CHECK-NEXT:    i32.const 7
+; CHECK-NEXT:    i8x16.shr_s
+; CHECK-NEXT:    # fallthrough-return
+    %extended = sext <16 x i1> %v to <16 x i8>
+    ret <16 x i8> %extended
+}
+
+define <8 x i8> @extend_i1x8_i8(<8 x i1> %v) {
+; CHECK-LABEL: extend_i1x8_i8:
+; CHECK:         .functype extend_i1x8_i8 (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i8x16.shuffle 0, 2, 4, 6, 8, 10, 12, 14, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i32.const 7
+; CHECK-NEXT:    i8x16.shl
+; CHECK-NEXT:    i32.const 7
+; CHECK-NEXT:    i8x16.shr_s
+; CHECK-NEXT:    # fallthrough-return
+    %extended = sext <8 x i1> %v to <8 x i8>
+    ret <8 x i8> %extended
+}
+
+define <8 x i16> @extend_i1x8_i16(<8 x i1> %v) {
+; CHECK-LABEL: extend_i1x8_i16:
+; CHECK:         .functype extend_i1x8_i16 (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.const 1, 1, 1, 1, 1, 1, 1, 1
+; CHECK-NEXT:    v128.and
+; CHECK-NEXT:    # fallthrough-return
+    %extended = zext <8 x i1> %v to <8 x i16>
+    ret <8 x i16> %extended
+}
+
+define <4 x i32> @extend_i8x4_i32(<4 x i8> %v) {
+; CHECK-LABEL: extend_i8x4_i32:
+; CHECK:         .functype extend_i8x4_i32 (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
+; CHECK-NEXT:    # fallthrough-return
+    %extended = zext <4 x i8> %v to <4 x i32>
+    ret <4 x i32> %extended
+}
+
+define <2 x i64> @extend_i8x2_i64(<2 x i8> %v) {
+; CHECK-LABEL: extend_i8x2_i64:
+; CHECK:         .functype extend_i8x2_i64 (v128) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
+; CHECK-NEXT:    i64x2.extend_low_i32x4_s
+; CHECK-NEXT:    # fallthrough-return
+    %extended = sext <2 x i8> %v to <2 x i64>
+    ret <2 x i64> %extended
+}

llvm/test/CodeGen/WebAssembly/simd-offset.ll

; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %p
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32 (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i8>, ptr %p
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32 (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i8>, ptr %p
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32(ptr %p) {
 ; CHECK-LABEL: load_ext_v4i32:
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_with_folded_offset(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_folded_offset:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_with_folded_offset (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 16
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint ptr %p to i32
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_with_folded_offset(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_folded_offset:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_with_folded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 16
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint ptr %p to i32
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_with_folded_gep_offset(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_folded_gep_offset:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_with_folded_gep_offset (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 4
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i8>, ptr %p, i32 1
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_with_folded_gep_offset(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_folded_gep_offset:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_with_folded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load32_zero 4
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i8>, ptr %p, i32 1
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32_with_folded_gep_offset(ptr %p) {
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -4
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i8>, ptr %p, i32 -1
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -4
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i8>, ptr %p, i32 -1
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32_with_unfolded_gep_negative_offset(ptr %p) {
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_offset(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_offset:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_offset (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint ptr %p to i32
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_offset(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_offset:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint ptr %p to i32
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_gep_offset(ptr %p) {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_gep_offset:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 4
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr <4 x i8>, ptr %p, i32 1
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_gep_offset(ptr %p) {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_gep_offset:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 4
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load32_zero 0
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr <4 x i8>, ptr %p, i32 1
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32_with_unfolded_gep_offset(ptr %p) {
; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, ptr %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_sext_v4i8_to_v4i32_from_numeric_address() {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_from_numeric_address:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_from_numeric_address () -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load32_zero 32
-; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_from_numeric_address() {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_from_numeric_address:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_from_numeric_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load32_zero 32
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to ptr
   %v = load <4 x i8>, ptr %s
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32_from_numeric_address() {
; CHECK-NEXT:    # fallthrough-return
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 @gv_v4i8 = global <4 x i8> <i8 42, i8 42, i8 42, i8 42>
 define <4 x i32> @load_sext_v4i8_to_v4i32_from_global_address() {
 ; CHECK-LABEL: load_sext_v4i8_to_v4i32_from_global_address:
 ; CHECK:         .functype load_sext_v4i8_to_v4i32_from_global_address () -> (v128)
-; CHECK-NEXT:    .local v128
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load32_zero gv_v4i8
-; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shl
-; CHECK-NEXT:    i32.const 24
-; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    i16x8.extend_low_i8x16_s
+; CHECK-NEXT:    i32x4.extend_low_i16x8_s
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i8>, ptr @gv_v4i8
   %v2 = sext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i32> @load_zext_v4i8_to_v4i32_from_global_address() {
 ; CHECK-LABEL: load_zext_v4i8_to_v4i32_from_global_address:
 ; CHECK:         .functype load_zext_v4i8_to_v4i32_from_global_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load32_zero gv_v4i8
-; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    i16x8.extend_low_i8x16_u
+; CHECK-NEXT:    i32x4.extend_low_i16x8_u
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i8>, ptr @gv_v4i8
   %v2 = zext <4 x i8> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
 define <4 x i16> @load_ext_v4i32_from_global_address() {
 ; CHECK-LABEL: load_ext_v4i32_from_global_address:

llvm/test/CodeGen/WebAssembly/simd-vecreduce-bool.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -verify-machineinstrs -disable-wasm-fallthrough-return-opt -wasm-disable-explicit-locals -wasm-keep-registers -mattr=+simd128 | FileCheck %s
+
+; Tests that bool vecreduce produces anytrue and alltrue instructions
+
+target triple = "wasm32-unknown-unknown"
+
+declare i1 @llvm.vector.reduce.or.v2i1(<2 x i1>)
+declare i1 @llvm.vector.reduce.or.v4i1(<4 x i1>)
+declare i1 @llvm.vector.reduce.or.v7i1(<7 x i1>)
+declare i1 @llvm.vector.reduce.or.v8i1(<8 x i1>)
+declare i1 @llvm.vector.reduce.or.v16i1(<16 x i1>)
+declare i1 @llvm.vector.reduce.and.v2i1(<2 x i1>)
+declare i1 @llvm.vector.reduce.and.v4i1(<4 x i1>)
+declare i1 @llvm.vector.reduce.and.v7i1(<7 x i1>)
+declare i1 @llvm.vector.reduce.and.v8i1(<8 x i1>)
+declare i1 @llvm.vector.reduce.and.v16i1(<16 x i1>)
+
+; =====================
+; Regular vectors of i1
+; =====================
+
+define i1 @test_any_v8i1(<8 x i1> %x) {
+; CHECK-LABEL: test_any_v8i1:
+; CHECK:         .functype test_any_v8i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %ret = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> %x)
+  ret i1 %ret
+}
+
+define i1 @test_all_v8i1(<8 x i1> %x) {
+; CHECK-LABEL: test_all_v8i1:
+; CHECK:         .functype test_all_v8i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i16x8.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %ret = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> %x)
+  ret i1 %ret
+}
+
+define i1 @test_none_v8i1(<8 x i1> %x) {
+; CHECK-LABEL: test_none_v8i1:
+; CHECK:         .functype test_none_v8i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push6=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop6
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    i32.const $push4=, 1
+; CHECK-NEXT:    i32.xor $push5=, $pop3, $pop4
+; CHECK-NEXT:    return $pop5
+  %any = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> %x)
+  %none = xor i1 %any, 1
+  ret i1 %none
+}
+
+define i1 @test_not_all_v8i1(<8 x i1> %x) {
+; CHECK-LABEL: test_not_all_v8i1:
+; CHECK:         .functype test_not_all_v8i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push6=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop6
+; CHECK-NEXT:    i16x8.all_true $push3=, $pop2
+; CHECK-NEXT:    i32.const $push4=, 1
+; CHECK-NEXT:    i32.xor $push5=, $pop3, $pop4
+; CHECK-NEXT:    return $pop5
+  %all = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> %x)
+  %notall = xor i1 %all, 1
+  ret i1 %notall
+}
+
+define i1 @test_any_v16i1(<16 x i1> %x) {
+; CHECK-LABEL: test_any_v16i1:
+; CHECK:         .functype test_any_v16i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 7
+; CHECK-NEXT:    i8x16.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 7
+; CHECK-NEXT:    i8x16.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %ret = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> %x)
+  ret i1 %ret
+}
+
+define i1 @test_all_v16i1(<16 x i1> %x) {
+; CHECK-LABEL: test_all_v16i1:
+; CHECK:         .functype test_all_v16i1 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 7
+; CHECK-NEXT:    i8x16.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 7
+; CHECK-NEXT:    i8x16.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i8x16.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %ret = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> %x)
+  ret i1 %ret
+}
+
+; ==================================
+; Regular vectors of larger integers
+; ==================================
+
+define i1 @test_any_v16i8(<16 x i8> %x) {
+; CHECK-LABEL: test_any_v16i8:
+; CHECK:         .functype test_any_v16i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 7
+; CHECK-NEXT:    i8x16.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 7
+; CHECK-NEXT:    i8x16.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <16 x i8> %x to <16 x i1>
+  %ret = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_all_v16i8(<16 x i8> %x) {
+; CHECK-LABEL: test_all_v16i8:
+; CHECK:         .functype test_all_v16i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 7
+; CHECK-NEXT:    i8x16.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 7
+; CHECK-NEXT:    i8x16.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i8x16.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <16 x i8> %x to <16 x i1>
+  %ret = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_any_v8i16(<8 x i16> %x) {
+; CHECK-LABEL: test_any_v8i16:
+; CHECK:         .functype test_any_v8i16 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <8 x i16> %x to <8 x i1>
+  %ret = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_all_v8i16(<8 x i16> %x) {
+; CHECK-LABEL: test_all_v8i16:
+; CHECK:         .functype test_all_v8i16 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 15
+; CHECK-NEXT:    i16x8.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 15
+; CHECK-NEXT:    i16x8.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i16x8.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <8 x i16> %x to <8 x i1>
+  %ret = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_any_v4i32(<4 x i32> %x) {
+; CHECK-LABEL: test_any_v4i32:
+; CHECK:         .functype test_any_v4i32 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 31
+; CHECK-NEXT:    i32x4.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 31
+; CHECK-NEXT:    i32x4.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <4 x i32> %x to <4 x i1>
+  %ret = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_all_v4i32(<4 x i32> %x) {
+; CHECK-LABEL: test_all_v4i32:
+; CHECK:         .functype test_all_v4i32 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 31
+; CHECK-NEXT:    i32x4.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 31
+; CHECK-NEXT:    i32x4.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i32x4.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <4 x i32> %x to <4 x i1>
+  %ret = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_any_v2i64(<2 x i64> %x) {
+; CHECK-LABEL: test_any_v2i64:
+; CHECK:         .functype test_any_v2i64 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 63
+; CHECK-NEXT:    i64x2.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 63
+; CHECK-NEXT:    i64x2.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    v128.any_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <2 x i64> %x to <2 x i1>
+  %ret = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_all_v2i64(<2 x i64> %x) {
+; CHECK-LABEL: test_all_v2i64:
+; CHECK:         .functype test_all_v2i64 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const $push0=, 63
+; CHECK-NEXT:    i64x2.shl $push1=, $0, $pop0
+; CHECK-NEXT:    i32.const $push4=, 63
+; CHECK-NEXT:    i64x2.shr_s $push2=, $pop1, $pop4
+; CHECK-NEXT:    i64x2.all_true $push3=, $pop2
+; CHECK-NEXT:    return $pop3
+  %bits = trunc <2 x i64> %x to <2 x i1>
+  %ret = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> %bits)
+  ret i1 %ret
+}
+
+; ====================
+; Unusual vector sizes
+; ====================
+
+define i1 @test_any_v7i1(<7 x i1> %x) {
+; CHECK-LABEL: test_any_v7i1:
+; CHECK:         .functype test_any_v7i1 (i32, i32, i32, i32, i32, i32, i32) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.or $push0=, $0, $1
+; CHECK-NEXT:    i32.or $push1=, $pop0, $2
+; CHECK-NEXT:    i32.or $push2=, $pop1, $3
+; CHECK-NEXT:    i32.or $push3=, $pop2, $4
+; CHECK-NEXT:    i32.or $push4=, $pop3, $5
+; CHECK-NEXT:    i32.or $push5=, $pop4, $6
+; CHECK-NEXT:    return $pop5
+  %ret = call i1 @llvm.vector.reduce.or.v7i1(<7 x i1> %x)
+  ret i1 %ret
+}
+
+define i1 @test_all_v7i1(<7 x i1> %x) {
+; CHECK-LABEL: test_all_v7i1:
+; CHECK:         .functype test_all_v7i1 (i32, i32, i32, i32, i32, i32, i32) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.and $push0=, $0, $1
+; CHECK-NEXT:    i32.and $push1=, $pop0, $2
+; CHECK-NEXT:    i32.and $push2=, $pop1, $3
+; CHECK-NEXT:    i32.and $push3=, $pop2, $4
+; CHECK-NEXT:    i32.and $push4=, $pop3, $5
+; CHECK-NEXT:    i32.and $push5=, $pop4, $6
+; CHECK-NEXT:    i32.const $push6=, 1
+; CHECK-NEXT:    i32.and $push7=, $pop5, $pop6
+; CHECK-NEXT:    return $pop7
+  %ret = call i1 @llvm.vector.reduce.and.v7i1(<7 x i1> %x)
+  ret i1 %ret
+}
+
+define i1 @test_any_v8i8(<8 x i8> %x) {
+; CHECK-LABEL: test_any_v8i8:
+; CHECK:         .functype test_any_v8i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i8x16.shuffle $push0=, $0, $0, 0, 0, 1, 0, 2, 0, 3, 0, 4, 0, 5, 0, 6, 0, 7, 0
+; CHECK-NEXT:    i32.const $push1=, 15
+; CHECK-NEXT:    i16x8.shl $push2=, $pop0, $pop1
+; CHECK-NEXT:    i32.const $push5=, 15
+; CHECK-NEXT:    i16x8.shr_s $push3=, $pop2, $pop5
+; CHECK-NEXT:    v128.any_true $push4=, $pop3
+; CHECK-NEXT:    return $pop4
+  %bits = trunc <8 x i8> %x to <8 x i1>
+  %ret = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> %bits)
+  ret i1 %ret
+}
+
+define i1 @test_all_v8i8(<8 x i8> %x) {
+; CHECK-LABEL: test_all_v8i8:
+; CHECK:         .functype test_all_v8i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i8x16.shuffle $push0=, $0, $0, 0, 0, 1, 0, 2, 0, 3, 0, 4, 0, 5, 0, 6, 0, 7, 0
+; CHECK-NEXT:    i32.const $push1=, 15
+; CHECK-NEXT:    i16x8.shl $push2=, $pop0, $pop1
+; CHECK-NEXT:    i32.const $push5=, 15
+; CHECK-NEXT:    i16x8.shr_s $push3=, $pop2, $pop5
+; CHECK-NEXT:    i16x8.all_true $push4=, $pop3
+; CHECK-NEXT:    return $pop4
+  %bits = trunc <8 x i8> %x to <8 x i1>
+  %ret = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> %bits)
+  ret i1 %ret
+}
+
+;; =====================
+;; Test reduce after cmp
+;; =====================
+
+define i1 @test_cmp_v16i8(<16 x i8> %x) {
+; CHECK-LABEL: test_cmp_v16i8:
+; CHECK:         .functype test_cmp_v16i8 (v128) -> (i32)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const $push0=, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.eq $push1=, $0, $pop0
+; CHECK-NEXT:    v128.any_true $push2=, $pop1
+; CHECK-NEXT:    return $pop2
+  %zero = icmp eq <16 x i8> %x, zeroinitializer
+  %ret = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> %zero)
+  ret i1 %ret
+}