[WebAssembly] Lower v2f32 to v2f64 extending loads with promote_low

[tlively]

Summary

Previously extra wide v4f32 to v4f64 extending loads would be legalized to v2f32
to v2f64 extending loads, which would then be scalarized by legalization. (v2f32
to v2f64 extending loads not produced by legalization were already being emitted
correctly.) Instead, mark v2f32 to v2f64 extending loads as legal and explicitly
lower them using promote_low. This regresses the addressing modes supported for
the extloads not produced by legalization, but that's a fine trade off for now.

[srj]

LGTM

[aheejin]

Previously extra wide v4f32 to v4f64 extending loads would be legalized to v2f32
to v2f64 extending loads, which would then be scalarized by legalization. (v2f32
to v2f64 extending loads not produced by legalization were already being emitted
correctly.)

Why v2f32 to v2f64 extending loads not produced by legalization are currently handled fine but not the one produced by legalization? I guess I lack the knowledge of order of transformation within isel..

This regresses the addressing modes supported for
the extloads not produced by legalization, but that's a fine trade off for now.

Can you give an example of this case?

llvm/lib/Target/WebAssembly/WebAssemblyInstrSIMD.td

1282

Any reason for using unindexedload instead of load?

1283

Is this necessary? unindexedload already sets IsLoad to true.

llvm/test/CodeGen/WebAssembly/simd-load-promote-wide.ll

9–11

29

Can we do a similar optimization for integer vectors, such as extending <4 x i32> to <4 x i64>, using v128.load64_zero followed by i64x2.extend_low_i32x4_s/u (and for other integer types too)? Or is it possible to use v128.load to load everything at once and use i64x.2_extend_low_... and i64x2.extend_high_... to extend each part?

I'm just asking for a possibility and of course don't mean we do that in this CL ;)

llvm/test/CodeGen/WebAssembly/simd-offset.ll

2960–2962

[tlively]

In D108496#2958980, @aheejin wrote:

Previously extra wide v4f32 to v4f64 extending loads would be legalized to v2f32
to v2f64 extending loads, which would then be scalarized by legalization. (v2f32
to v2f64 extending loads not produced by legalization were already being emitted
correctly.)

Why v2f32 to v2f64 extending loads not produced by legalization are currently handled fine but not the one produced by legalization? I guess I lack the knowledge of order of transformation within isel..

v2f32 to v2f64 extending loads were previously lowered like this:

SelectionDAG has 7 nodes:
  t0: ch = EntryToken
        t2: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<0>
      t5: v2f32,ch = load<(load (s64) from %ir.p)> t0, t2, undef:i32
    t6: v2f64 = fp_extend t5
  t7: ch = WebAssemblyISD::RETURN t0, t6

...

Optimized type-legalized selection DAG: %bb.0 'load_promote_v2f62:'
SelectionDAG has 15 nodes:
  t0: ch = EntryToken
        t2: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<0>
      t8: i64,ch = load<(load (s64) from %ir.p)> t0, t2, undef:i32
    t9: v2i64 = scalar_to_vector t8
  t10: v4f32 = bitcast t9
        t12: f32 = extract_vector_elt t10, Constant:i32<0>
      t13: f64 = fp_extend t12
        t15: f32 = extract_vector_elt t10, Constant:i32<1>
      t16: f64 = fp_extend t15
    t17: v2f64 = BUILD_VECTOR t13, t16
  t7: ch = WebAssemblyISD::RETURN t0, t17

but v4f32 to v4f64 extending loads were lowered like this:

SelectionDAG has 14 nodes:
  t0: ch = EntryToken
    t2: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<0>
  t6: ch = CopyToReg t0, Register:i32 %0, t2
  t7: i32 = Constant<0>
          t4: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<1>
        t9: v4f32,ch = load<(load (s128) from %ir.p)> t6, t4, undef:i32
      t10: v4f64 = fp_extend t9
      t11: i32,ch = CopyFromReg t0, Register:i32 %0
    t12: ch = store<(store (s256))> t6, t10, t11, undef:i32
  t13: ch = WebAssemblyISD::RETURN t12

...

Optimized type-legalized selection DAG: %bb.0 'load_promote_v4f64:'
SelectionDAG has 19 nodes:
  t0: ch = EntryToken
  t4: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<1>
    t2: i32 = WebAssemblyISD::ARGUMENT TargetConstant:i32<0>
  t6: ch = CopyToReg t0, Register:i32 %0, t2
  t11: i32,ch = CopyFromReg t0, Register:i32 %0
        t17: v2f64,ch = load<(load (s64) from %ir.p, align 16), anyext from v2f32> t6, t4, undef:i32
      t22: ch = store<(store (s128), align 32)> t6, t17, t11, undef:i32
          t19: i32 = add nuw t4, Constant:i32<8>
        t20: v2f64,ch = load<(load (s64) from %ir.p + 8, basealign 16), anyext from v2f32> t6, t19, undef:i32
        t24: i32 = add nuw t11, Constant:i32<16>
      t25: ch = store<(store (s128) into unknown-address + 16, basealign 32)> t6, t20, t24, undef:i32
    t26: ch = TokenFactor t22, t25
  t13: ch = WebAssemblyISD::RETURN t26

So the original loads get split up in both cases, but the reason is different. For the narrower vectors, the v2f32 is not a legal type so the load gets split by type legalization, but for the wider vectors, it's the v4f64 result of the fp_extend that gets split up with the load in the operand position. I guess the different splitting code paths there lead to completely different DAGs later.

This regresses the addressing modes supported for
the extloads not produced by legalization, but that's a fine trade off for now.

Can you give an example of this case?

All of the new load_promote tests in simd-offset.ll have to explicitly calculate the load offset rather than just use the offset immediate on the load. Before this change they would use the offset immediate.

llvm/lib/Target/WebAssembly/WebAssemblyInstrSIMD.td

1282

load sets IsNonExtLoad = true, which is not what we want here. Apparently an "unindexed" load is a normal load and the other kinds of loads also perform address computations and return those results in the same instruction.

1283

I believe so. We are inheriting from PatFrag, not from unindexedload directly, and I think the whole new fragment has to be marked as a load. Looking at the definitions of e.g. extloadf32, though, we could use extload instead of unindexedload and get rid of the IsAnyExtLoad = true line.

llvm/test/CodeGen/WebAssembly/simd-load-promote-wide.ll

29

Yes, there is room to further optimize the integer version of this. Right now <4 x i32> to <4 x i64> extending loads are lowered to a pattern of one v128.load followed by some shuffles and vector shifts. That doesn't seem too bad, but it would be better to use the extend_low and extend_high instructions as you suggest instead.

Even here it might be better to do a v128.load followed by a promote_low, shuffle, and second promote_low rather than doing two 128.load64_zero. I'll check with the SIMD experts on that...

[tlively]

• Address feedback

[tlively]

• Missed feedback

[aheejin]

Thanks for the explanation! Then after this CL, does promote_low part for LowerConvertLow become redundant? Hmm, maybe not completely, because this has a capability of shuffling? (By the way we don't seem to have a test of shuffled promote_low case.. simd-conversions.ll only contains integer vector to v2f64 tests. It would be good to have one for float vectors to v2f64 too) But given that LowerConvertLow function also takes care of integer to float conversions, we cannot remove that function anyway though. (Unless we do a similar optimization for integer vectors)

llvm/lib/Target/WebAssembly/WebAssemblyInstrSIMD.td

1283

That's nice. By the way extload also sets IsLoad to true, so this line will still be unnecessary.

[tlively]

In D108496#2970332, @aheejin wrote:

Thanks for the explanation! Then after this CL, does promote_low part for LowerConvertLow become redundant? Hmm, maybe not completely, because this has a capability of shuffling? (By the way we don't seem to have a test of shuffled promote_low case.. simd-conversions.ll only contains integer vector to v2f64 tests. It would be good to have one for float vectors to v2f64 too) But given that LowerConvertLow function also takes care of integer to float conversions, we cannot remove that function anyway though. (Unless we do a similar optimization for integer vectors)

That lowering is used in cases where there is no load, so I don't think it's redundant. We also already have tests for float to double promote_low and one test that uses the shuffle behavior: https://github.com/llvm/llvm-project/blob/main/llvm/test/CodeGen/WebAssembly/simd-conversions.ll#L431-L443

[tlively]

• Remove unneccesary line

llvm/lib/Target/WebAssembly/WebAssemblyISelLowering.cpp

for (auto T : {MVT::v16i8, MVT::v8i16, MVT::v4i32, MVT::v2i64, MVT::v4f32,
       }
     }
     // But some vector extending loads are legal
     for (auto Ext : {ISD::EXTLOAD, ISD::SEXTLOAD, ISD::ZEXTLOAD}) {
       setLoadExtAction(Ext, MVT::v8i16, MVT::v8i8, Legal);
       setLoadExtAction(Ext, MVT::v4i32, MVT::v4i16, Legal);
       setLoadExtAction(Ext, MVT::v2i64, MVT::v2i32, Legal);
     }
+    setLoadExtAction(ISD::EXTLOAD, MVT::v2f64, MVT::v2f32, Legal);
   }
 
   // Don't do anything clever with build_pairs
   setOperationAction(ISD::BUILD_PAIR, MVT::i64, Expand);
 
   // Trap lowers to wasm unreachable
   setOperationAction(ISD::TRAP, MVT::Other, Legal);
   setOperationAction(ISD::DEBUGTRAP, MVT::Other, Legal);

llvm/lib/Target/WebAssembly/WebAssemblyInstrSIMD.td

 def demote_zero : SDNode<"WebAssemblyISD::DEMOTE_ZERO", demote_t>;
 defm "" : SIMDConvert<F32x4, F64x2, demote_zero,
                       "demote_zero_f64x2", 0x5e>;
 
 def promote_t : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVec<1>]>;
 def promote_low : SDNode<"WebAssemblyISD::PROMOTE_LOW", promote_t>;
 defm "" : SIMDConvert<F64x2, F32x4, promote_low, "promote_low_f32x4", 0x5f>;
 
+// Lower extending loads to load64_zero + promote_low
+def extloadv2f32 : PatFrag<(ops node:$ptr), (unindexedload node:$ptr)> {
+  let IsLoad = true;
+  let IsAnyExtLoad = true;
+  let MemoryVT = v2f32;
+}
+// Adapted from the body of LoadPatNoOffset
+// TODO: other addressing patterns
+def : Pat<(v2f64 (extloadv2f32 (i32 I32:$addr))),
+          (promote_low_F64x2 (LOAD_ZERO_I64x2_A32 0, 0, I32:$addr))>,
+      Requires<[HasAddr32]>;
+def : Pat<(v2f64 (extloadv2f32 (i64 I64:$addr))),
+          (promote_low_F64x2 (LOAD_ZERO_I64x2_A64 0, 0, I64:$addr))>,
+      Requires<[HasAddr64]>;
+
 //===----------------------------------------------------------------------===//
 // Saturating Rounding Q-Format Multiplication
 //===----------------------------------------------------------------------===//
 
 defm Q15MULR_SAT_S :
   SIMDBinary<I16x8, int_wasm_q15mulr_sat_signed, "q15mulr_sat_s", 0x82>;

llvm/test/CodeGen/WebAssembly/simd-load-promote-wide.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -verify-machineinstrs -mattr=+simd128 | FileCheck %s
+
+; Test wide load+promote patterns, which after combines and legalization are
+; represented differently than 128-bit load+promote patterns.
+
+target triple = "wasm32-unknown-unknown"
+
+define <4 x double> @load_promote_v2f62(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f62:
+; CHECK:         .functype load_promote_v2f62 (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %e = load <4 x float>, <4 x float>* %p
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_with_folded_offset(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_folded_offset:
+; CHECK:         .functype load_promote_v2f64_with_folded_offset (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x float>* %p to i32
+  %r = add nuw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x float>*
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_with_folded_gep_offset(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_folded_gep_offset:
+; CHECK:         .functype load_promote_v2f64_with_folded_gep_offset (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <4 x float>, <4 x float>* %p, i32 1
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_with_unfolded_gep_negative_offset(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_gep_negative_offset (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const -16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    local.tee 1
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <4 x float>, <4 x float>* %p, i32 -1
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_with_unfolded_offset(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_offset (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <4 x float>* %p to i32
+  %r = add nsw i32 %q, 16
+  %s = inttoptr i32 %r to <4 x float>*
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_with_unfolded_gep_offset(<4 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_gep_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_gep_offset (i32, i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 24
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    local.get 1
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr <4 x float>, <4 x float>* %p, i32 1
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+define <4 x double> @load_promote_v2f64_from_numeric_address() {
+; CHECK-LABEL: load_promote_v2f64_from_numeric_address:
+; CHECK:         .functype load_promote_v2f64_from_numeric_address (i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 40
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 32
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %s = inttoptr i32 32 to <4 x float>*
+  %e = load <4 x float>, <4 x float>* %s
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}
+
+@gv_v4f32 = global <4 x float> <float 42., float 42., float 42., float 42.>
+define <4 x double> @load_promote_v2f64_from_global_address() {
+; CHECK-LABEL: load_promote_v2f64_from_global_address:
+; CHECK:         .functype load_promote_v2f64_from_global_address (i32) -> ()
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const gv_v4f32
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 16
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const gv_v4f32
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    v128.store 0
+; CHECK-NEXT:    # fallthrough-return
+  %e = load <4 x float>, <4 x float>* @gv_v4f32
+  %v = fpext <4 x float> %e to <4 x double>
+  ret <4 x double> %v
+}

llvm/test/CodeGen/WebAssembly/simd-offset.ll

 ; CHECK-NEXT:    v128.load64_splat 0
 ; CHECK-NEXT:    # fallthrough-return
   %e = load double, double* %p
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f62(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f62:
+; CHECK:         .functype load_promote_v2f62 (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %e = load <2 x float>, <2 x float>* %p
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_with_folded_offset(<2 x double>* %p) {
 ; CHECK-LABEL: load_v2f64_with_folded_offset:
 ; CHECK:         .functype load_v2f64_with_folded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load 16
 ; CHECK-NEXT:    # fallthrough-return
   %q = ptrtoint <2 x double>* %p to i32
; CHECK-NEXT:    # fallthrough-return
   %r = add nuw i32 %q, 16
   %s = inttoptr i32 %r to double*
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_with_folded_offset(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_folded_offset:
+; CHECK:         .functype load_promote_v2f64_with_folded_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <2 x float>* %p to i32
+  %r = add nuw i32 %q, 16
+  %s = inttoptr i32 %r to <2 x float>*
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_with_folded_gep_offset(<2 x double>* %p) {
 ; CHECK-LABEL: load_v2f64_with_folded_gep_offset:
 ; CHECK:         .functype load_v2f64_with_folded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.load 16
 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds <2 x double>, <2 x double>* %p, i32 1

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds double, double* %p, i32 1
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_with_folded_gep_offset(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_folded_gep_offset:
+; CHECK:         .functype load_promote_v2f64_with_folded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <2 x float>, <2 x float>* %p, i32 1
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_with_unfolded_gep_negative_offset(<2 x double>* %p) {
 ; CHECK-LABEL: load_v2f64_with_unfolded_gep_negative_offset:
 ; CHECK:         .functype load_v2f64_with_unfolded_gep_negative_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const -16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load 0

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr inbounds double, double* %p, i32 -1
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_with_unfolded_gep_negative_offset(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_gep_negative_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_gep_negative_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const -8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr inbounds <2 x float>, <2 x float>* %p, i32 -1
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_with_unfolded_offset(<2 x double>* %p) {
 ; CHECK-LABEL: load_v2f64_with_unfolded_offset:
 ; CHECK:         .functype load_v2f64_with_unfolded_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load 0
; CHECK-NEXT:    # fallthrough-return
   %r = add nsw i32 %q, 16
   %s = inttoptr i32 %r to double*
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_with_unfolded_offset(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 16
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %q = ptrtoint <2 x float>* %p to i32
+  %r = add nsw i32 %q, 16
+  %s = inttoptr i32 %r to <2 x float>*
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_with_unfolded_gep_offset(<2 x double>* %p) {
 ; CHECK-LABEL: load_v2f64_with_unfolded_gep_offset:
 ; CHECK:         .functype load_v2f64_with_unfolded_gep_offset (i32) -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    i32.const 16
 ; CHECK-NEXT:    i32.add
 ; CHECK-NEXT:    v128.load 0

 ; CHECK-NEXT:    # fallthrough-return
   %s = getelementptr double, double* %p, i32 1
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_with_unfolded_gep_offset(<2 x float>* %p) {
+; CHECK-LABEL: load_promote_v2f64_with_unfolded_gep_offset:
+; CHECK:         .functype load_promote_v2f64_with_unfolded_gep_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %s = getelementptr <2 x float>, <2 x float>* %p, i32 1
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define <2 x double> @load_v2f64_from_numeric_address() {
 ; CHECK-LABEL: load_v2f64_from_numeric_address:
 ; CHECK:         .functype load_v2f64_from_numeric_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load 32
 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to <2 x double>*

 ; CHECK-NEXT:    # fallthrough-return
   %s = inttoptr i32 32 to double*
   %e = load double, double* %s
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+define <2 x double> @load_promote_v2f64_from_numeric_address() {
+; CHECK-LABEL: load_promote_v2f64_from_numeric_address:
+; CHECK:         .functype load_promote_v2f64_from_numeric_address () -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const 32
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %s = inttoptr i32 32 to <2 x float>*
+  %e = load <2 x float>, <2 x float>* %s
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 @gv_v2f64 = global <2 x double> <double 42., double 42.>
 define <2 x double> @load_v2f64_from_global_address() {
 ; CHECK-LABEL: load_v2f64_from_global_address:
 ; CHECK:         .functype load_v2f64_from_global_address () -> (v128)
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    i32.const 0
 ; CHECK-NEXT:    v128.load gv_v2f64
 ; CHECK-NEXT:    # fallthrough-return

 ; CHECK-NEXT:    v128.load64_splat gv_f64
 ; CHECK-NEXT:    # fallthrough-return
   %e = load double, double* @gv_f64
   %v1 = insertelement <2 x double> undef, double %e, i32 0
   %v2 = shufflevector <2 x double> %v1, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %v2
 }
 
+@gv_v2f32 = global <2 x float> <float 42., float 42.>
+define <2 x double> @load_promote_v2f64_from_global_address() {
+; CHECK-LABEL: load_promote_v2f64_from_global_address:
+; CHECK:         .functype load_promote_v2f64_from_global_address () -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    i32.const gv_v2f32
+; CHECK-NEXT:    v128.load64_zero 0
+; CHECK-NEXT:    f64x2.promote_low_f32x4
+; CHECK-NEXT:    # fallthrough-return
+  %e = load <2 x float>, <2 x float>* @gv_v2f32
+  %v = fpext <2 x float> %e to <2 x double>
+  ret <2 x double> %v
+}
+
 define void @store_v2f64(<2 x double> %v, <2 x double>* %p) {
 ; CHECK-LABEL: store_v2f64:
 ; CHECK:         .functype store_v2f64 (v128, i32) -> ()
 ; CHECK-NEXT:  # %bb.0:
 ; CHECK-NEXT:    local.get 1
 ; CHECK-NEXT:    local.get 0
 ; CHECK-NEXT:    v128.store 0
 ; CHECK-NEXT:    # fallthrough-return