[WebAssembly] Improve codegen for loading scalars from memory to v128

[fanchenkong1]

Summary

Use load32_zero instead of load32_splat to load the low 32 bits from memory to v128. Test cases are added to cover this change.

[fanchenkong1]

minor change

[fanchenkong1]

This change extends an optimization at Legalize vector types by widening on load scalars to 32 bits. Since we have seen i8x4 to i32x4 widen in a hot function of PhotoShop Web.

[tlively]

llvm/test/CodeGen/WebAssembly/simd-offset.ll

1196–1199

It looks like there is some room for improvement here. These shifts aren't necessary, are they? It would be good to at least add a TODO about cleaning them up.

1210

Do you know why the zero vector is materialized here while the previous function uses a local.get to get an implicit zero vector?

[fanchenkong1]

Thanks Thomas for reviewing this change!

llvm/test/CodeGen/WebAssembly/simd-offset.ll

1196–1199

Yes, a TODO can be added if further change is needed. But I'm not sure if I fully understand what to do to remove the shifts. Does it mean by using two sign extend? e.g.

i16x8.extend_low_i8x16_s
i32x4.extend_low_i16x8_s

1210

The difference seems to be introduced by the vector legalizer, where the sign extend is expanded by VectorLegalizer::ExpandSIGN_EXTEND_VECTOR_INREG, followed with VectorLegalizer::ExpandANY_EXTEND_VECTOR_INREG. The later one may introduce an undef node. While the zero extend is expanded by VectorLegalizer::ExpandZERO_EXTEND_VECTOR_INREG, where an explicit zero vector is created.

[tlively]

Thanks! Do you need me to land this?

llvm/test/CodeGen/WebAssembly/simd-offset.ll

1196–1199

Oh, I see, we need the shifts because they implement the sign extend part. Using the sequence of extend_low instructions is also a good idea. How would the native code from that solution compare?

[fanchenkong1]

In D134257#3805936, @tlively wrote:

Thanks! Do you need me to land this?

Yes, would you please help me land this change? The author name and mail could be "Fanchen Kong <[e-mail]>". Thanks!

llvm/test/CodeGen/WebAssembly/simd-offset.ll

1196–1199

On x64, for the shuffle + shifts solution, the native code maybe,

shuffle byte (or packed zero extend if zero vector is detectable)
packed shift left
packed shift right arithmetic

For sequence of extend_low, the expected code can be

packed byte to dword sign extend

The current solution seems not bad if shuffle byte can be reduced at VM. The extend_low sequence seems to be a little better on x64, but I'm not sure if thats the case for all platforms.

[tlively]

llvm/test/CodeGen/WebAssembly/simd-offset.ll

1196–1199

Thanks for the detail. I'll land this as-is because it is an improvement over the status quo, but we should keep that other possibility in mind.

llvm/lib/Target/WebAssembly/WebAssemblyInstrSIMD.td

SIMD_I<(outs V128:$dst),
            name#"\t$off$p2align", simdop>;
   } // mayLoad = 1, UseNamedOperandTable = 1
 }
 
 defm "" : SIMDLoadZero<I32x4, 0x5c>;
 defm "" : SIMDLoadZero<I64x2, 0x5d>;
 
 // Use load_zero to load scalars into vectors as well where possible.
-// TODO: i32, i16, and i8 scalars
-def load_scalar :
-  PatFrag<(ops node:$addr), (scalar_to_vector (i64 (load $addr)))>;
-defm : LoadPatNoOffset<v2i64, load_scalar, "LOAD_ZERO_I64x2">;
-defm : LoadPatImmOff<v2i64, load_scalar, regPlusImm, "LOAD_ZERO_I64x2">;
-defm : LoadPatImmOff<v2i64, load_scalar, or_is_add, "LOAD_ZERO_I64x2">;
-defm : LoadPatOffsetOnly<v2i64, load_scalar, "LOAD_ZERO_I64x2">;
-defm : LoadPatGlobalAddrOffOnly<v2i64, load_scalar, "LOAD_ZERO_I64x2">;
+// TODO: i16, and i8 scalars
+foreach vec = [I32x4, I64x2] in {
+  defvar inst = "LOAD_ZERO_"#vec;
+  defvar pat = PatFrag<(ops node:$addr), (scalar_to_vector (vec.lane_vt (load $addr)))>;
+  defm : LoadPatNoOffset<vec.vt, pat, inst>;
+  defm : LoadPatImmOff<vec.vt, pat, regPlusImm, inst>;
+  defm : LoadPatImmOff<vec.vt, pat, or_is_add, inst>;
+  defm : LoadPatOffsetOnly<vec.vt, pat, inst>;
+  defm : LoadPatGlobalAddrOffOnly<vec.vt, pat, inst>;
+}
 
 // TODO: f32x4 and f64x2 as well
 foreach vec = [I32x4, I64x2] in {
   defvar inst = "LOAD_ZERO_"#vec;
   defvar pat = PatFrag<(ops node:$ptr),
     (vector_insert (vec.splat (vec.lane_vt 0)), (vec.lane_vt (load $ptr)), 0)>;
   defm : LoadPatNoOffset<vec.vt, pat, inst>;
   defm : LoadPatImmOff<vec.vt, pat, regPlusImm, inst>;

llvm/test/CodeGen/WebAssembly/simd-offset.ll

 ; CHECK-NEXT:    v128.load32_splat 0
 ; CHECK-NEXT:    # fallthrough-return
   %e = load i32, i32* %addr, align 4
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32:
-; CHECK:         .functype load_sext_v4i32 (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32:
+; CHECK:         .functype load_sext_v4i16_to_v4i32 (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_s 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* %p
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32:
-; CHECK:         .functype load_zext_v4i32 (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32:
+; CHECK:         .functype load_zext_v4i16_to_v4i32 (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_u 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* %p
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32:
+; CHECK:         .functype load_sext_v4i8_to_v4i32 (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %v = load <4 x i8>, <4 x i8>* %p
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32:
+; CHECK:         .functype load_zext_v4i8_to_v4i32 (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %v = load <4 x i8>, <4 x i8>* %p
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32:
 ; CHECK:         .functype load_ext_v4i32 (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load64_zero 0
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* %p
; CHECK-NEXT:    # fallthrough-return
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to i32*
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_with_folded_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32_with_folded_offset:
-; CHECK:         .functype load_sext_v4i32_with_folded_offset (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_with_folded_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_with_folded_offset:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_with_folded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_s 16
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <4 x i16>* %p to i32
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_with_folded_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32_with_folded_offset:
-; CHECK:         .functype load_zext_v4i32_with_folded_offset (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_with_folded_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_with_folded_offset:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_with_folded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_u 16
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <4 x i16>* %p to i32
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_with_folded_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_folded_offset:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_with_folded_offset (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load32_zero 16
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x i8>* %p to i32
+  %r = add nuw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_with_folded_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_folded_offset:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_with_folded_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load32_zero 16
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x i8>* %p to i32
+  %r = add nuw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_with_folded_offset(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32_with_folded_offset:
 ; CHECK:         .functype load_ext_v4i32_with_folded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load64_zero 16
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <4 x i16>* %p to i32

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds i32, i32* %p, i32 1
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_with_folded_gep_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32_with_folded_gep_offset:
-; CHECK:         .functype load_sext_v4i32_with_folded_gep_offset (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_with_folded_gep_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_with_folded_gep_offset:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_with_folded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_s 8
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i16>, <4 x i16>* %p, i32 1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_with_folded_gep_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32_with_folded_gep_offset:
-; CHECK:         .functype load_zext_v4i32_with_folded_gep_offset (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_with_folded_gep_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_with_folded_gep_offset:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_with_folded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32x4.load16x4_u 8
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i16>, <4 x i16>* %p, i32 1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_with_folded_gep_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_folded_gep_offset:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_with_folded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load32_zero 4
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <4 x i8>, <4 x i8>* %p, i32 1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_with_folded_gep_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_folded_gep_offset:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_with_folded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:  v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:  local.get 0
+; CHECK-NEXT:  v128.load32_zero 4
+; CHECK-NEXT:  i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:  # fallthrough-return
+  %s = getelementptr inbounds <4 x i8>, <4 x i8>* %p, i32 1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_with_folded_gep_offset(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32_with_folded_gep_offset:
 ; CHECK:         .functype load_ext_v4i32_with_folded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load64_zero 8
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i16>, <4 x i16>* %p, i32 1

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds i32, i32* %p, i32 -1
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_with_unfolded_gep_negative_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32_with_unfolded_gep_negative_offset:
-; CHECK:         .functype load_sext_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_with_unfolded_gep_negative_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_s 0
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i16>, <4 x i16>* %p, i32 -1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_with_unfolded_gep_negative_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32_with_unfolded_gep_negative_offset:
-; CHECK:         .functype load_zext_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_with_unfolded_gep_negative_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_u 0
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <4 x i16>, <4 x i16>* %p, i32 -1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const -4
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <4 x i8>, <4 x i8>* %p, i32 -1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const -4
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <4 x i8>, <4 x i8>* %p, i32 -1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_with_unfolded_gep_negative_offset(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32_with_unfolded_gep_negative_offset:
 ; CHECK:         .functype load_ext_v4i32_with_unfolded_gep_negative_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load64_zero 0
; CHECK-NEXT:    # fallthrough-return
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to i32*
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_with_unfolded_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32_with_unfolded_offset:
-; CHECK:         .functype load_sext_v4i32_with_unfolded_offset (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_with_unfolded_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_with_unfolded_offset:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_with_unfolded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_s 0
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <4 x i16>* %p to i32
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_with_unfolded_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32_with_unfolded_offset:
-; CHECK:         .functype load_zext_v4i32_with_unfolded_offset (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_with_unfolded_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_with_unfolded_offset:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_with_unfolded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_u 0
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <4 x i16>* %p to i32
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_offset:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_offset (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x i8>* %p to i32
+  %r = add nsw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_offset:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x i8>* %p to i32
+  %r = add nsw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_with_unfolded_offset(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32_with_unfolded_offset:
 ; CHECK:         .functype load_ext_v4i32_with_unfolded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load64_zero 0

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr i32, i32* %p, i32 1
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_with_unfolded_gep_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_sext_v4i32_with_unfolded_gep_offset:
-; CHECK:         .functype load_sext_v4i32_with_unfolded_gep_offset (i32) -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_with_unfolded_gep_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_with_unfolded_gep_offset:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_s 0
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr <4 x i16>, <4 x i16>* %p, i32 1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_with_unfolded_gep_offset(<4 x i16>* %p) {
-; CHECK-LABEL: load_zext_v4i32_with_unfolded_gep_offset:
-; CHECK:         .functype load_zext_v4i32_with_unfolded_gep_offset (i32) -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_with_unfolded_gep_offset(<4 x i16>* %p) {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_with_unfolded_gep_offset:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    i32x4.load16x4_u 0
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr <4 x i16>, <4 x i16>* %p, i32 1
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_with_unfolded_gep_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_with_unfolded_gep_offset:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 4
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr <4 x i8>, <4 x i8>* %p, i32 1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_with_unfolded_gep_offset(<4 x i8>* %p) {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_with_unfolded_gep_offset:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_with_unfolded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 4
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load32_zero 0
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr <4 x i8>, <4 x i8>* %p, i32 1
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_with_unfolded_gep_offset(<4 x i16>* %p) {
 ; CHECK-LABEL: load_ext_v4i32_with_unfolded_gep_offset:
 ; CHECK:         .functype load_ext_v4i32_with_unfolded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 8
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load64_zero 0

 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to i32*
   %e = load i32, i32* %s
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_sext_v4i32_from_numeric_address() {
-; CHECK-LABEL: load_sext_v4i32_from_numeric_address:
-; CHECK:         .functype load_sext_v4i32_from_numeric_address () -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_from_numeric_address() {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_from_numeric_address:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_from_numeric_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    i32x4.load16x4_s 32
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_from_numeric_address() {
-; CHECK-LABEL: load_zext_v4i32_from_numeric_address:
-; CHECK:         .functype load_zext_v4i32_from_numeric_address () -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_from_numeric_address() {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_from_numeric_address:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_from_numeric_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    i32x4.load16x4_u 32
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to <4 x i16>*
   %v = load <4 x i16>, <4 x i16>* %s
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+define <4 x i32> @load_sext_v4i8_to_v4i32_from_numeric_address() {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_from_numeric_address:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_from_numeric_address () -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const 0
+; CHECK-NEXT:    v128.load32_zero 32
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %s = inttoptr i32 32 to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_from_numeric_address() {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_from_numeric_address:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_from_numeric_address () -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i32.const 0
+; CHECK-NEXT:    v128.load32_zero 32
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %s = inttoptr i32 32 to <4 x i8>*
+  %v = load <4 x i8>, <4 x i8>* %s
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_from_numeric_address() {
 ; CHECK-LABEL: load_ext_v4i32_from_numeric_address:
 ; CHECK:         .functype load_ext_v4i32_from_numeric_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load64_zero 32
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to <4 x i16>*

 ; CHECK-NEXT:    # fallthrough-return
   %e = load i32, i32* @gv_i32
   %v1 = insertelement <4 x i32> undef, i32 %e, i32 0
   %v2 = shufflevector <4 x i32> %v1, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %v2
 }
 
 @gv_v4i16 = global <4 x i16> <i16 42, i16 42, i16 42, i16 42>
-define <4 x i32> @load_sext_v4i32_from_global_address() {
-; CHECK-LABEL: load_sext_v4i32_from_global_address:
-; CHECK:         .functype load_sext_v4i32_from_global_address () -> (v128)
+define <4 x i32> @load_sext_v4i16_to_v4i32_from_global_address() {
+; CHECK-LABEL: load_sext_v4i16_to_v4i32_from_global_address:
+; CHECK:         .functype load_sext_v4i16_to_v4i32_from_global_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    i32x4.load16x4_s gv_v4i16
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* @gv_v4i16
   %v2 = sext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
-define <4 x i32> @load_zext_v4i32_from_global_address() {
-; CHECK-LABEL: load_zext_v4i32_from_global_address:
-; CHECK:         .functype load_zext_v4i32_from_global_address () -> (v128)
+define <4 x i32> @load_zext_v4i16_to_v4i32_from_global_address() {
+; CHECK-LABEL: load_zext_v4i16_to_v4i32_from_global_address:
+; CHECK:         .functype load_zext_v4i16_to_v4i32_from_global_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    i32x4.load16x4_u gv_v4i16
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* @gv_v4i16
   %v2 = zext <4 x i16> %v to <4 x i32>
   ret <4 x i32> %v2
 }
 
+@gv_v4i8 = global <4 x i8> <i8 42, i8 42, i8 42, i8 42>
+define <4 x i32> @load_sext_v4i8_to_v4i32_from_global_address() {
+; CHECK-LABEL: load_sext_v4i8_to_v4i32_from_global_address:
+; CHECK:         .functype load_sext_v4i8_to_v4i32_from_global_address () -> (v128)
+; CHECK-NEXT:    .local v128
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const 0
+; CHECK-NEXT:    v128.load32_zero gv_v4i8
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i8x16.shuffle 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shl
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32x4.shr_s
+; CHECK-NEXT:    # fallthrough-return
+  %v = load <4 x i8>, <4 x i8>* @gv_v4i8
+  %v2 = sext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
+define <4 x i32> @load_zext_v4i8_to_v4i32_from_global_address() {
+; CHECK-LABEL: load_zext_v4i8_to_v4i32_from_global_address:
+; CHECK:         .functype load_zext_v4i8_to_v4i32_from_global_address () -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+; CHECK-NEXT:    i32.const 0
+; CHECK-NEXT:    v128.load32_zero gv_v4i8
+; CHECK-NEXT:    i8x16.shuffle 16, 1, 2, 3, 17, 5, 6, 7, 18, 9, 10, 11, 19, 13, 14, 15
+; CHECK-NEXT:    # fallthrough-return
+  %v = load <4 x i8>, <4 x i8>* @gv_v4i8
+  %v2 = zext <4 x i8> %v to <4 x i32>
+  ret <4 x i32> %v2
+}
+
 define <4 x i16> @load_ext_v4i32_from_global_address() {
 ; CHECK-LABEL: load_ext_v4i32_from_global_address:
 ; CHECK:         .functype load_ext_v4i32_from_global_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load64_zero gv_v4i16
 ; CHECK-NEXT:    # fallthrough-return
   %v = load <4 x i16>, <4 x i16>* @gv_v4i16