[WebAssembly] Improve codegen for shuffles with undefined lane indices

[fanchenkong1]

Summary

For undefined lane indices, fill the mask with {0..N} instead of zeros, to allow further reduction to word/dword shuffle at VM.

[fanchenkong1]

minor change

[penzn]

To me this makes since this applies to undefined indices and should not make any difference in case of valid ones.

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp

2325–2328

Nit: maybe expand on why '{0..J}', it might be not immediately obvious for those who didn't look at shuffles recently.

[tlively]

The code looks good, but I don't know much about native SIMD. Is this strictly better for engine-side codegen on all platforms? LGTM if so.

[penzn]

In D133473#3781884, @tlively wrote:

The code looks good, but I don't know much about native SIMD. Is this strictly better for engine-side codegen on all platforms? LGTM if so.

Yes, it is, that's why maybe expanding on description/comments would be good :)

Filling undefined lanes with 0's would effectively turn the shuffle into a byte shuffle, which is more expensive than "coarser" shuffles.

[tlively]

Sounds good. @fanchenkong1, it would be great if you could expand the comment as @penzn suggested. After that, do you have commit access or would you like me to merge this for you? If you would like me to commit it, what author email should I use for you?

[penzn]

A stray thought: we have disabled combining shuffles because it was producing precisely the masks this change seeks to eliminate, I am wondering if we can reenable some of those optimization based on this.

Background:

https://github.com/emscripten-core/emscripten/issues/9340
https://github.com/WebAssembly/simd/issues/196

[fanchenkong1]

handle comments

[fanchenkong1]

minor change

[fanchenkong1]

Thank Thomas and Petr for reviewing this patch! The comment is expanded in the latest change.

Failed test is found on x64 windows, which seems to be related to mlir. Is there any action needed to fix the test?

Hi @tlively, I don't have the commit access. Would you please help land this patch once it is ready? The author name and mail could be "Fanchen Kong <[e-mail]>". Thanks!

[penzn]

In D133473#3785568, @fanchenkong1 wrote:

Failed test is found on x64 windows, which seems to be related to mlir. Is there any action needed to fix the test?

IMO you don't need to worry about those, you can retest on Windows if you want, but your change shouldn't/can't cause this. @tlively what do you think?

[tlively]

Yeah the unrelated test failures can be ignored. Good point about our shuffle combine workaround, @penzn. Would you have time and resources to investigate whether we can remove the workaround now?

[penzn]

In D133473#3787964, @tlively wrote:

Yeah the unrelated test failures can be ignored. Good point about our shuffle combine workaround, @penzn. Would you have time and resources to investigate whether we can remove the workaround now?

Yes, we will have time to investigate.

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp

WebAssemblyTargetLowering::LowerVECTOR_SHUFFLE(SDValue Op,
   SDValue Ops[18];
   size_t OpIdx = 0;
   Ops[OpIdx++] = Op.getOperand(0);
   Ops[OpIdx++] = Op.getOperand(1);
 
   // Expand mask indices to byte indices and materialize them as operands
   for (int M : Mask) {
     for (size_t J = 0; J < LaneBytes; ++J) {
-      // Lower undefs (represented by -1 in mask) to zero
-      uint64_t ByteIndex = M == -1 ? 0 : (uint64_t)M * LaneBytes + J;
+      // Lower undefs (represented by -1 in mask) to {0..J}, which use a
+      // whole lane of vector input, to allow further reduction at VM. E.g.
+      // match an 8x16 byte shuffle to an equivalent cheaper 32x4 shuffle.
+      uint64_t ByteIndex = M == -1 ? J : (uint64_t)M * LaneBytes + J;
       Ops[OpIdx++] = DAG.getConstant(ByteIndex, DL, MVT::i32);
     }
   }
 
   return DAG.getNode(WebAssemblyISD::SHUFFLE, DL, Op.getValueType(), Ops);
 }
 
 SDValue WebAssemblyTargetLowering::LowerSETCC(SDValue Op,

llvm/test/CodeGen/WebAssembly/fpclamptosat_vec.ll

 ; CHECK-NEXT:    local.tee 0
 ; CHECK-NEXT:    v128.const -2147483648, -2147483648
 ; CHECK-NEXT:    local.tee 1
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    i64x2.gt_s
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i64>
   %0 = icmp slt <2 x i64> %conv, <i64 2147483647, i64 2147483647>
   %spec.store.select = select <2 x i1> %0, <2 x i64> %conv, <2 x i64> <i64 2147483647, i64 2147483647>
   %1 = icmp sgt <2 x i64> %spec.store.select, <i64 -2147483648, i64 -2147483648>
   %spec.store.select7 = select <2 x i1> %1, <2 x i64> %spec.store.select, <2 x i64> <i64 -2147483648, i64 -2147483648>
   %conv6 = trunc <2 x i64> %spec.store.select7 to <2 x i32>

 ; CHECK-NEXT:    i64.const 0
 ; CHECK-NEXT:    local.get 2
 ; CHECK-NEXT:    i64.const 4294967295
 ; CHECK-NEXT:    i64.lt_u
 ; CHECK-NEXT:    i64.select
 ; CHECK-NEXT:    i64x2.replace_lane 1
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <2 x double> %x to <2 x i64>
   %0 = icmp ult <2 x i64> %conv, <i64 4294967295, i64 4294967295>
   %spec.store.select = select <2 x i1> %0, <2 x i64> %conv, <2 x i64> <i64 4294967295, i64 4294967295>
   %conv6 = trunc <2 x i64> %spec.store.select to <2 x i32>
   ret <2 x i32> %conv6
 }

 ; CHECK-NEXT:    local.tee 0
 ; CHECK-NEXT:    v128.const 0, 0
 ; CHECK-NEXT:    local.tee 1
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    i64x2.gt_s
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i64>
   %0 = icmp slt <2 x i64> %conv, <i64 4294967295, i64 4294967295>
   %spec.store.select = select <2 x i1> %0, <2 x i64> %conv, <2 x i64> <i64 4294967295, i64 4294967295>
   %1 = icmp sgt <2 x i64> %spec.store.select, zeroinitializer
   %spec.store.select7 = select <2 x i1> %1, <2 x i64> %spec.store.select, <2 x i64> zeroinitializer
   %conv6 = trunc <2 x i64> %spec.store.select7 to <2 x i32>

 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_s
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 32767, 32767, 0, 0
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const -32768, -32768, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i32>
   %0 = icmp slt <2 x i32> %conv, <i32 32767, i32 32767>
   %spec.store.select = select <2 x i1> %0, <2 x i32> %conv, <2 x i32> <i32 32767, i32 32767>
   %1 = icmp sgt <2 x i32> %spec.store.select, <i32 -32768, i32 -32768>
   %spec.store.select7 = select <2 x i1> %1, <2 x i32> %spec.store.select, <2 x i32> <i32 -32768, i32 -32768>
   %conv6 = trunc <2 x i32> %spec.store.select7 to <2 x i16>

 ; CHECK-NEXT:    i32x4.splat
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_u
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 65535, 65535, 0, 0
 ; CHECK-NEXT:    i32x4.min_u
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <2 x double> %x to <2 x i32>
   %0 = icmp ult <2 x i32> %conv, <i32 65535, i32 65535>
   %spec.store.select = select <2 x i1> %0, <2 x i32> %conv, <2 x i32> <i32 65535, i32 65535>
   %conv6 = trunc <2 x i32> %spec.store.select to <2 x i16>
   ret <2 x i16> %conv6
 }

 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_s
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 65535, 65535, 0, 0
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const 0, 0, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i32>
   %0 = icmp slt <2 x i32> %conv, <i32 65535, i32 65535>
   %spec.store.select = select <2 x i1> %0, <2 x i32> %conv, <2 x i32> <i32 65535, i32 65535>
   %1 = icmp sgt <2 x i32> %spec.store.select, zeroinitializer
   %spec.store.select7 = select <2 x i1> %1, <2 x i32> %spec.store.select, <2 x i32> zeroinitializer
   %conv6 = trunc <2 x i32> %spec.store.select7 to <2 x i16>
   ret <2 x i16> %conv6
 }
 
 define <4 x i16> @stest_f32i16(<4 x float> %x) {
 ; CHECK-LABEL: stest_f32i16:
 ; CHECK:         .functype stest_f32i16 (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_s
 ; CHECK-NEXT:    v128.const 32767, 32767, 32767, 32767
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const -32768, -32768, -32768, -32768
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <4 x float> %x to <4 x i32>
   %0 = icmp slt <4 x i32> %conv, <i32 32767, i32 32767, i32 32767, i32 32767>
   %spec.store.select = select <4 x i1> %0, <4 x i32> %conv, <4 x i32> <i32 32767, i32 32767, i32 32767, i32 32767>
   %1 = icmp sgt <4 x i32> %spec.store.select, <i32 -32768, i32 -32768, i32 -32768, i32 -32768>
   %spec.store.select7 = select <4 x i1> %1, <4 x i32> %spec.store.select, <4 x i32> <i32 -32768, i32 -32768, i32 -32768, i32 -32768>
   %conv6 = trunc <4 x i32> %spec.store.select7 to <4 x i16>
   ret <4 x i16> %conv6
 }
 
 define <4 x i16> @utest_f32i16(<4 x float> %x) {
 ; CHECK-LABEL: utest_f32i16:
 ; CHECK:         .functype utest_f32i16 (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_u
 ; CHECK-NEXT:    v128.const 65535, 65535, 65535, 65535
 ; CHECK-NEXT:    i32x4.min_u
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <4 x float> %x to <4 x i32>
   %0 = icmp ult <4 x i32> %conv, <i32 65535, i32 65535, i32 65535, i32 65535>
   %spec.store.select = select <4 x i1> %0, <4 x i32> %conv, <4 x i32> <i32 65535, i32 65535, i32 65535, i32 65535>
   %conv6 = trunc <4 x i32> %spec.store.select to <4 x i16>
   ret <4 x i16> %conv6
 }
 
 define <4 x i16> @ustest_f32i16(<4 x float> %x) {
 ; CHECK-LABEL: ustest_f32i16:
 ; CHECK:         .functype ustest_f32i16 (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_s
 ; CHECK-NEXT:    v128.const 65535, 65535, 65535, 65535
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const 0, 0, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <4 x float> %x to <4 x i32>
   %0 = icmp slt <4 x i32> %conv, <i32 65535, i32 65535, i32 65535, i32 65535>
   %spec.store.select = select <4 x i1> %0, <4 x i32> %conv, <4 x i32> <i32 65535, i32 65535, i32 65535, i32 65535>
   %1 = icmp sgt <4 x i32> %spec.store.select, zeroinitializer
   %spec.store.select7 = select <4 x i1> %1, <4 x i32> %spec.store.select, <4 x i32> zeroinitializer
   %conv6 = trunc <4 x i32> %spec.store.select7 to <4 x i16>

 ; CHECK-NEXT:    local.tee 0
 ; CHECK-NEXT:    v128.const -2147483648, -2147483648
 ; CHECK-NEXT:    local.tee 1
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    i64x2.gt_s
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i64>
   %spec.store.select = call <2 x i64> @llvm.smin.v2i64(<2 x i64> %conv, <2 x i64> <i64 2147483647, i64 2147483647>)
   %spec.store.select7 = call <2 x i64> @llvm.smax.v2i64(<2 x i64> %spec.store.select, <2 x i64> <i64 -2147483648, i64 -2147483648>)
   %conv6 = trunc <2 x i64> %spec.store.select7 to <2 x i32>
   ret <2 x i32> %conv6
 }

 ; CHECK-NEXT:    i64.const 0
 ; CHECK-NEXT:    local.get 2
 ; CHECK-NEXT:    i64.const 4294967295
 ; CHECK-NEXT:    i64.lt_u
 ; CHECK-NEXT:    i64.select
 ; CHECK-NEXT:    i64x2.replace_lane 1
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <2 x double> %x to <2 x i64>
   %spec.store.select = call <2 x i64> @llvm.umin.v2i64(<2 x i64> %conv, <2 x i64> <i64 4294967295, i64 4294967295>)
   %conv6 = trunc <2 x i64> %spec.store.select to <2 x i32>
   ret <2 x i32> %conv6
 }
 

 ; CHECK-NEXT:    local.tee 0
 ; CHECK-NEXT:    v128.const 0, 0
 ; CHECK-NEXT:    local.tee 1
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    i64x2.gt_s
 ; CHECK-NEXT:    v128.bitselect
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i64>
   %spec.store.select = call <2 x i64> @llvm.smin.v2i64(<2 x i64> %conv, <2 x i64> <i64 4294967295, i64 4294967295>)
   %spec.store.select7 = call <2 x i64> @llvm.smax.v2i64(<2 x i64> %spec.store.select, <2 x i64> zeroinitializer)
   %conv6 = trunc <2 x i64> %spec.store.select7 to <2 x i32>
   ret <2 x i32> %conv6
 }

 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_s
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 32767, 32767, 0, 0
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const -32768, -32768, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i32>
   %spec.store.select = call <2 x i32> @llvm.smin.v2i32(<2 x i32> %conv, <2 x i32> <i32 32767, i32 32767>)
   %spec.store.select7 = call <2 x i32> @llvm.smax.v2i32(<2 x i32> %spec.store.select, <2 x i32> <i32 -32768, i32 -32768>)
   %conv6 = trunc <2 x i32> %spec.store.select7 to <2 x i16>
   ret <2 x i16> %conv6
 }
 
 define <2 x i16> @utest_f64i16_mm(<2 x double> %x) {
 ; CHECK-LABEL: utest_f64i16_mm:
 ; CHECK:         .functype utest_f64i16_mm (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    f64x2.extract_lane 0
 ; CHECK-NEXT:    i32.trunc_sat_f64_u
 ; CHECK-NEXT:    i32x4.splat
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_u
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 65535, 65535, 0, 0
 ; CHECK-NEXT:    i32x4.min_u
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <2 x double> %x to <2 x i32>
   %spec.store.select = call <2 x i32> @llvm.umin.v2i32(<2 x i32> %conv, <2 x i32> <i32 65535, i32 65535>)
   %conv6 = trunc <2 x i32> %spec.store.select to <2 x i16>
   ret <2 x i16> %conv6
 }
 

 ; CHECK-NEXT:    f64x2.extract_lane 1
 ; CHECK-NEXT:    i32.trunc_sat_f64_s
 ; CHECK-NEXT:    i32x4.replace_lane 1
 ; CHECK-NEXT:    v128.const 65535, 65535, 0, 0
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const 0, 0, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <2 x double> %x to <2 x i32>
   %spec.store.select = call <2 x i32> @llvm.smin.v2i32(<2 x i32> %conv, <2 x i32> <i32 65535, i32 65535>)
   %spec.store.select7 = call <2 x i32> @llvm.smax.v2i32(<2 x i32> %spec.store.select, <2 x i32> zeroinitializer)
   %conv6 = trunc <2 x i32> %spec.store.select7 to <2 x i16>
   ret <2 x i16> %conv6
 }
 
 define <4 x i16> @stest_f32i16_mm(<4 x float> %x) {
 ; CHECK-LABEL: stest_f32i16_mm:
 ; CHECK:         .functype stest_f32i16_mm (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_s
 ; CHECK-NEXT:    v128.const 32767, 32767, 32767, 32767
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const -32768, -32768, -32768, -32768
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <4 x float> %x to <4 x i32>
   %spec.store.select = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %conv, <4 x i32> <i32 32767, i32 32767, i32 32767, i32 32767>)
   %spec.store.select7 = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %spec.store.select, <4 x i32> <i32 -32768, i32 -32768, i32 -32768, i32 -32768>)
   %conv6 = trunc <4 x i32> %spec.store.select7 to <4 x i16>
   ret <4 x i16> %conv6
 }
 
 define <4 x i16> @utest_f32i16_mm(<4 x float> %x) {
 ; CHECK-LABEL: utest_f32i16_mm:
 ; CHECK:         .functype utest_f32i16_mm (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_u
 ; CHECK-NEXT:    v128.const 65535, 65535, 65535, 65535
 ; CHECK-NEXT:    i32x4.min_u
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptoui <4 x float> %x to <4 x i32>
   %spec.store.select = call <4 x i32> @llvm.umin.v4i32(<4 x i32> %conv, <4 x i32> <i32 65535, i32 65535, i32 65535, i32 65535>)
   %conv6 = trunc <4 x i32> %spec.store.select to <4 x i16>
   ret <4 x i16> %conv6
 }
 
 define <4 x i16> @ustest_f32i16_mm(<4 x float> %x) {
 ; CHECK-LABEL: ustest_f32i16_mm:
 ; CHECK:         .functype ustest_f32i16_mm (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0: # %entry
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.trunc_sat_f32x4_s
 ; CHECK-NEXT:    v128.const 65535, 65535, 65535, 65535
 ; CHECK-NEXT:    i32x4.min_s
 ; CHECK-NEXT:    v128.const 0, 0, 0, 0
 ; CHECK-NEXT:    i32x4.max_s
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 4, 5, 8, 9, 12, 13, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
 entry:
   %conv = fptosi <4 x float> %x to <4 x i32>
   %spec.store.select = call <4 x i32> @llvm.smin.v4i32(<4 x i32> %conv, <4 x i32> <i32 65535, i32 65535, i32 65535, i32 65535>)
   %spec.store.select7 = call <4 x i32> @llvm.smax.v4i32(<4 x i32> %spec.store.select, <4 x i32> zeroinitializer)
   %conv6 = trunc <4 x i32> %spec.store.select7 to <4 x i16>
   ret <4 x i16> %conv6
 }

llvm/test/CodeGen/WebAssembly/simd-build-vector.ll

; CHECK-NEXT:    return $pop6
   %v0 = insertelement <8 x i16> undef, i16 %s0, i32 0
   ret <8 x i16> %v0
 }
 
 define <4 x i32> @half_shuffle_i32x4(<4 x i32> %src) {
 ; CHECK-LABEL: half_shuffle_i32x4:
 ; CHECK:         .functype half_shuffle_i32x4 (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
-; CHECK-NEXT:    i8x16.shuffle $push0=, $0, $0, 0, 0, 0, 0, 8, 9, 10, 11, 0, 1, 2, 3, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle $push0=, $0, $0, 0, 1, 2, 3, 8, 9, 10, 11, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    i32.const $push1=, 0
 ; CHECK-NEXT:    i32x4.replace_lane $push2=, $pop0, 0, $pop1
 ; CHECK-NEXT:    i32.const $push3=, 3
 ; CHECK-NEXT:    i32x4.replace_lane $push4=, $pop2, 3, $pop3
 ; CHECK-NEXT:    return $pop4
   %s0 = extractelement <4 x i32> %src, i32 0
   %s2 = extractelement <4 x i32> %src, i32 2
   %v0 = insertelement <4 x i32> undef, i32 0, i32 0

llvm/test/CodeGen/WebAssembly/simd-concat.ll

 }
 
 define <4 x i16> @concat_v2i16(<2 x i16> %a, <2 x i16> %b) {
 ; CHECK-LABEL: concat_v2i16:
 ; CHECK:         .functype concat_v2i16 (v128, v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 16, 17, 18, 19, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 16, 17, 18, 19, 0, 1, 0, 1, 0, 1, 0, 1
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <2 x i16> %a, <2 x i16> %b, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x i16> %v
 }
 
 define <4 x i32> @concat_v2i32(<2 x i32> %a, <2 x i32> %b) {
 ; CHECK-LABEL: concat_v2i32:
 ; CHECK:         .functype concat_v2i32 (v128, v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    i8x16.shuffle 0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <2 x i32> %a, <2 x i32> %b, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x i32> %v
 }

llvm/test/CodeGen/WebAssembly/simd-conversions.ll

 
 define <4 x double> @convert_low_s_v4f64(<8 x i32> %x) {
 ; CHECK-LABEL: convert_low_s_v4f64:
 ; CHECK:         .functype convert_low_s_v4f64 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_s
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_s
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <8 x i32> %x, <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %a = sitofp <4 x i32> %v to <4 x double>
   ret <4 x double> %a
 }
 
 define <4 x double> @convert_low_u_v4f64(<8 x i32> %x) {
 ; CHECK-LABEL: convert_low_u_v4f64:
 ; CHECK:         .functype convert_low_u_v4f64 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <8 x i32> %x, <8 x i32> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %a = uitofp <4 x i32> %v to <4 x double>
   ret <4 x double> %a
 }
 
 
 define <4 x double> @convert_low_s_v4f64_2(<8 x i32> %x) {
 ; CHECK-LABEL: convert_low_s_v4f64_2:
 ; CHECK:         .functype convert_low_s_v4f64_2 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_s
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_s
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = sitofp <8 x i32> %x to <8 x double>
   %a = shufflevector <8 x double> %v, <8 x double> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x double> %a
 }
 
 define <4 x double> @convert_low_u_v4f64_2(<8 x i32> %x) {
 ; CHECK-LABEL: convert_low_u_v4f64_2:
 ; CHECK:         .functype convert_low_u_v4f64_2 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.convert_low_i32x4_u
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = uitofp <8 x i32> %x to <8 x double>
   %a = shufflevector <8 x double> %v, <8 x double> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x double> %a
 }
 
 define <4 x double> @promote_low_v4f64(<8 x float> %x) {
 ; CHECK-LABEL: promote_low_v4f64:
 ; CHECK:         .functype promote_low_v4f64 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.promote_low_f32x4
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.promote_low_f32x4
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <8 x float> %x, <8 x float> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %a = fpext <4 x float> %v to <4 x double>
   ret <4 x double> %a
 }
 
 define <4 x double> @promote_low_v4f64_2(<8 x float> %x) {
 ; CHECK-LABEL: promote_low_v4f64_2:
 ; CHECK:         .functype promote_low_v4f64_2 (i32, v128, v128) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.promote_low_f32x4
 ; CHECK-NEXT:    v128.store 16
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    f64x2.promote_low_f32x4
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = fpext <8 x float> %x to <8 x double>
   %a = shufflevector <8 x double> %v, <8 x double> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x double> %a
 }
 
 define <2 x double> @promote_mixed_v2f64(<4 x float> %x, <4 x float> %y) {
 ; CHECK-LABEL: promote_mixed_v2f64:
 ; CHECK:         .functype promote_mixed_v2f64 (v128, v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 1
-; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 28, 29, 30, 31, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 8, 9, 10, 11, 28, 29, 30, 31, 0, 1, 2, 3, 0, 1, 2, 3
 ; CHECK-NEXT:    f64x2.promote_low_f32x4
 ; CHECK-NEXT:    # fallthrough-return
   %v = shufflevector <4 x float> %x, <4 x float> %y, <2 x i32> <i32 2, i32 7>
   %a = fpext <2 x float> %v to <2 x double>
   ret <2 x double> %a
 }

llvm/test/CodeGen/WebAssembly/simd-extending.ll

 }
 
 define <4 x i32> @extend_lowish_i16x8_s(<8 x i16> %v) {
 ; CHECK-LABEL: extend_lowish_i16x8_s:
 ; CHECK:         .functype extend_lowish_i16x8_s (v128) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    local.get 0
-; CHECK-NEXT:    i8x16.shuffle 2, 3, 0, 0, 4, 5, 0, 0, 6, 7, 0, 0, 8, 9, 0, 0
+; CHECK-NEXT:    i8x16.shuffle 2, 3, 0, 1, 4, 5, 0, 1, 6, 7, 0, 1, 8, 9, 0, 1
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32x4.shl
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32x4.shr_s
 ; CHECK-NEXT:    # fallthrough-return
   %lowish = shufflevector <8 x i16> %v, <8 x i16> undef,
            <4 x i32> <i32 1, i32 2, i32 3, i32 4>
   %extended = sext <4 x i16> %lowish to <4 x i32>
   ret <4 x i32> %extended
 }

llvm/test/CodeGen/WebAssembly/simd.ll

%res = shufflevector <8 x i16> %x, <8 x i16> %y,
     <8 x i32> <i32 0, i32 9, i32 2, i32 11, i32 4, i32 13, i32 6, i32 15>
   ret <8 x i16> %res
 }
 
 ; CHECK-LABEL: shuffle_undef_v8i16:
 ; NO-SIMD128-NOT: i8x16
 ; SIMD128-NEXT: .functype shuffle_undef_v8i16 (v128, v128) -> (v128){{$}}
 ; SIMD128-NEXT: i8x16.shuffle $push[[R:[0-9]+]]=, $0, $0,
-; SIMD128-SAME: 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0{{$}}
+; SIMD128-SAME: 2, 3, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1{{$}}
 ; SIMD128-NEXT: return $pop[[R]]{{$}}
 define <8 x i16> @shuffle_undef_v8i16(<8 x i16> %x, <8 x i16> %y) {
   %res = shufflevector <8 x i16> %x, <8 x i16> %y,
     <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef,
                i32 undef, i32 undef, i32 undef, i32 undef>
   ret <8 x i16> %res
 }
 
%res = shufflevector <4 x i32> %x, <4 x i32> %y,
     <4 x i32> <i32 0, i32 5, i32 2, i32 7>
   ret <4 x i32> %res
 }
 
 ; CHECK-LABEL: shuffle_undef_v4i32:
 ; NO-SIMD128-NOT: i8x16
 ; SIMD128-NEXT: .functype shuffle_undef_v4i32 (v128, v128) -> (v128){{$}}
 ; SIMD128-NEXT: i8x16.shuffle $push[[R:[0-9]+]]=, $0, $0,
-; SIMD128-SAME: 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0{{$}}
+; SIMD128-SAME: 4, 5, 6, 7, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3{{$}}
 ; SIMD128-NEXT: return $pop[[R]]{{$}}
 define <4 x i32> @shuffle_undef_v4i32(<4 x i32> %x, <4 x i32> %y) {
   %res = shufflevector <4 x i32> %x, <4 x i32> %y,
     <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   ret <4 x i32> %res
 }
 
 ; CHECK-LABEL: build_v4i32:
define <2 x i64> @shuffle_v2i64(<2 x i64> %x, <2 x i64> %y) {
   %res = shufflevector <2 x i64> %x, <2 x i64> %y, <2 x i32> <i32 0, i32 3>
   ret <2 x i64> %res
 }
 
 ; CHECK-LABEL: shuffle_undef_v2i64:
 ; NO-SIMD128-NOT: i8x16
 ; SIMD128-NEXT: .functype shuffle_undef_v2i64 (v128, v128) -> (v128){{$}}
 ; SIMD128-NEXT: i8x16.shuffle $push[[R:[0-9]+]]=, $0, $0,
-; SIMD128-SAME: 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0{{$}}
+; SIMD128-SAME: 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7{{$}}
 ; SIMD128-NEXT: return $pop[[R]]{{$}}
 define <2 x i64> @shuffle_undef_v2i64(<2 x i64> %x, <2 x i64> %y) {
   %res = shufflevector <2 x i64> %x, <2 x i64> %y,
     <2 x i32> <i32 1, i32 undef>
   ret <2 x i64> %res
 }
 
 ; CHECK-LABEL: build_v2i64:
%res = shufflevector <4 x float> %x, <4 x float> %y,
     <4 x i32> <i32 0, i32 5, i32 2, i32 7>
   ret <4 x float> %res
 }
 
 ; CHECK-LABEL: shuffle_undef_v4f32:
 ; NO-SIMD128-NOT: i8x16
 ; SIMD128-NEXT: .functype shuffle_undef_v4f32 (v128, v128) -> (v128){{$}}
 ; SIMD128-NEXT: i8x16.shuffle $push[[R:[0-9]+]]=, $0, $0,
-; SIMD128-SAME: 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0{{$}}
+; SIMD128-SAME: 4, 5, 6, 7, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3{{$}}
 ; SIMD128-NEXT: return $pop[[R]]{{$}}
 define <4 x float> @shuffle_undef_v4f32(<4 x float> %x, <4 x float> %y) {
   %res = shufflevector <4 x float> %x, <4 x float> %y,
     <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   ret <4 x float> %res
 }
 
 ; CHECK-LABEL: build_v4f32:
%res = shufflevector <2 x double> %x, <2 x double> %y,
     <2 x i32> <i32 0, i32 3>
   ret <2 x double> %res
 }
 
 ; CHECK-LABEL: shuffle_undef_v2f64:
 ; NO-SIMD128-NOT: i8x16
 ; SIMD128-NEXT: .functype shuffle_undef_v2f64 (v128, v128) -> (v128){{$}}
 ; SIMD128-NEXT: i8x16.shuffle $push[[R:[0-9]+]]=, $0, $0,
-; SIMD128-SAME: 8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0{{$}}
+; SIMD128-SAME: 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7{{$}}
 ; SIMD128-NEXT: return $pop[[R]]{{$}}
 define <2 x double> @shuffle_undef_v2f64(<2 x double> %x, <2 x double> %y) {
   %res = shufflevector <2 x double> %x, <2 x double> %y,
     <2 x i32> <i32 1, i32 undef>
   ret <2 x double> %res
 }
 
 ; CHECK-LABEL: build_v2f64: