[DAGCombiner] Scalarize vectorized loads that are splatted

[luke]

Summary

This un-vectorizes vector loads into scalar loads whenever they are used in a shuffle_vector that are splats, since only one element from them will be accessed.
This opens up better instruction selection of splatted loads and should address a WebAssembly issue: https://github.com/llvm/llvm-project/issues/59120

[luke]

Fix commit message

[pengfei]

Checked changes in X86 tests are all correct.

[luke]

In D140069#3997198, @pengfei wrote:

Checked changes in X86 tests are all correct.

Thanks! It looks like some other X86 tests are still failing, I'll try to address them also.

On aarch64 some tests in arm64-dup.ll are failing:

define <8 x i8> @vduplane8(<8 x i8>* %A) nounwind {
; CHECK-LABEL: vduplane8:
; CHECK:       // %bb.0:
; CHECK-NEXT:    ldr d0, [x0]
; CHECK-NEXT:    dup.8b v0, v0[1]
; CHECK-NEXT:    ret
	%tmp1 = load <8 x i8>, <8 x i8>* %A
	%tmp2 = shufflevector <8 x i8> %tmp1, <8 x i8> undef, <8 x i32> < i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1 >
	ret <8 x i8> %tmp2
}

Since they're now being selected as a broadcasted load:

vduplane8:                              // @vduplane8
// %bb.0:
	add	x8, x0, #1
	ld1r.8b	{ v0 }, [x8]
	ret

I'm not familiar with aarch64, but is it not possible to fold the offset in like this?

vduplane8:                              // @vduplane8
// %bb.0:
	ld1r.8b	{ v0 }, [x0], #1
	ret

[luke]

Update tests

[luke]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

138–142 ↗

(On Diff #483182)

I presume this a regression since even though it's loading smaller sizes, it has to do more twiddling.

llvm/test/CodeGen/PowerPC/canonical-merge-shuffles.ll

1149–1162 ↗

(On Diff #483182)

These extra lines replace the old P8-AIX prefixed checks that must have been left behind

llvm/test/CodeGen/X86/half.ll

1342–1344 ↗

(On Diff #483182)

@pengfei This looks like a regression, the scalarized load t18 gets selected as VPINSRWrm

t0: ch,glue = EntryToken
                  t2: i64,ch = CopyFromReg t0, Register:i64 %0
                t17: i64 = add t2, Constant:i64<8>
              t18: f16,ch = load<(load (s16) from %ir.p + 8, align 8)> t0, t17, undef:i64
            t21: v8f16 = scalar_to_vector t18
          t23: v8i16 = bitcast t21
        t28: v8i16 = X86ISD::PSHUFLW t23, TargetConstant:i8<0>
      t29: v4i32 = bitcast t28
    t30: v4i32 = X86ISD::PSHUFD t29, TargetConstant:i8<0>
  t36: v8f16 = bitcast t30
t10: ch,glue = CopyToReg t0, Register:v8f16 $xmm0, t36
t11: ch = X86ISD::RET_FLAG t10, TargetConstant:i32<0>, Register:v8f16 $xmm0, t10:1

[luke]

Update aarch64 tests

[luke]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1102–1106 ↗

(On Diff #483185)

Regression: Can the add be folded in as an immediate offset to ld1r.8h { v0 }, [x8]?
Same applies for the cases below

[pengfei]

llvm/test/CodeGen/X86/half.ll

1342–1344 ↗

(On Diff #483182)

Right. I think this is a special case. We don't have native scalar instructions to load/store half or bfloat in old targets. Instead, we have to use the more expensive pinsrw/pextrw to emulate. Which makes scalar load/store operations are suboptimal to vector ones.
Have you noticed if other targets have a similar problem. It's better if we can find a way to avoid the regression, otherwise, I think we can add FIXME at the moment.

[luke]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1102–1106 ↗

(On Diff #483185)

No, since the offset would actually increment the register operand.
Should we instead check if the target is able to perform an indexed load, and bail otherwise when the offset != 0?
There's a target lowering hook called isIndexingLegal that seems like it could be used to check this, but no targets currently implement it, and it was added for GlobalISel: https://reviews.llvm.org/D66287
@t.p.northover would you have any thoughts on this?

[dmgreen]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1102–1106 ↗

(On Diff #483185)

Hello. This can actually be worse, but I don't think that's an issue with this patch. We tend to prefer ld1r over the dup in the mul instruction, but I believe the opposite can be quicker. That is a general issue though, this patch is just exposing it in a few extra places.

None of these tests need to load data, and I think it would be better to remove that part. If they just use vectors parameters directly then they will be less susceptible to optimizations on the load altering what the test is intended to check. I can put a patch together to clean this up, and if you rebase on top most of these changes should hopefully disappear.

[dmgreen]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1102–1106 ↗

(On Diff #483185)

Hopefully if you rebase over rG752819e813d1, most of these changes will go away.

[luke]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1102–1106 ↗

(On Diff #483185)

Hello, thanks a million for that patch. Will rebase.

[luke]

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1227–1230 ↗

(On Diff #483185)

@dmgreen This looks like a regression to me but I'm not familiar enough with aarch64 to really know for certain. I presume the cost of the additional add instruction outweighs any gains from a smaller load, is that correct?

(Hope I'm not bombarding you with too many questions, let me know if there's someone else I can ask!)

[luke]

Rebase

[dmgreen]

Thanks. The remaining Arm and AArch64 test cases look OK to me.

llvm/test/CodeGen/AArch64/arm64-vmul.ll

1227–1230 ↗

(On Diff #483185)

Yeah it's the same as the other cases. I heard that it can be a little worse than if the dup could be part of the mul/fmulx/etc, but that's a separate issue from this patch. In practice many cases will already be a splat of a scalar value, so will already run into the same issue.

[luke]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

@foad @ruiling Apologies if I'm pinging the wrong people here, just wanted to get some AMDGPU eyes over this.
From what I understand this looks like a regression since the two loads aren't dispatched in tandem anymore, there's separate waits. Are there any suggestions as to how to avoid this/are there any target info hooks that might be relevant here?

[foad]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

Yes it looks like a regression but I'm not sure how serious it is.

The original code did two 4-byte loads even though we only want the upper two bytes of each value. Now we've turned the second one into a 2-byte load that overwrites part of the result of the first load, hence the WAW dependency. Why can't we also turn the first load into a 2-byte load?

Also @rampitec @arsenm

[ruiling]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

An extra wait usually means serious regression. But I did not see why we need the s_waitcnt here. The global_loads should return the value in order, so there is no WAW dependency here, right? @foad

[arsenm]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

The waitcnt insertion pass probably doesn't try to understand the tied operands of the d16 loads

[foad]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

Right, at the MIR level it looks like a RAW dependency because the d16 load has a tied read representing the parts of the destination register that are not overwritten. So I guess we *could* fix this in the waitcnt insertion pass. (It sounds similar to the special case for writelane in AMDGPUInsertDelayAlu.)

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

22844

Cleaner to use Shuf->getValueType(0) - result/operands will have matching types in DAG.

22849

Please check the operand numbers -
LHS (Op0) = 0 <= Idx < NumElts
RHS (Op1) = NumElts <= Idx < 2*NumElts

22860

Pull out repeated Shuf->getSplatIndex() calls

[luke]

Address review comments

[foad]

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

D140537 should fix this when it lands.

[RKSimon]

llvm/test/CodeGen/X86/avx-vbroadcast.ll

367 ↗

(On Diff #487360)

; Pointer adjusted broadcasts

[luke]

Rebase and update comments

[luke]

Update AMDGPU tests with D140537

llvm/test/CodeGen/AMDGPU/vector_shuffle.packed.ll

1905–1914 ↗

(On Diff #484287)

Thanks!

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

22841

Similar to D140811 - we might want to limit this to cases where the shuffle is not just a single non-undef mask element

[luke]

Limit to cases where the shuffle is not just a single non-undef mask element

[luke]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

22841

Done: Is there another optimisation that occurs on a splat w/ a single non-undef mask element?

[luke]

Update affected tests

[RKSimon]

llvm/test/CodeGen/PowerPC/canonical-merge-shuffles.ll

1149–1162 ↗

(On Diff #483182)

Aren't the P8-AIX/P8-AIX-32/P8-AIX-64 checks still used? There's more likely an issue with aix-32 having a legal <2 x i64> type but no the i64 type

llvm/test/CodeGen/WebAssembly/simd-vectorized-load-splat.ll

6

please can you pre-commit this test file to trunk with trunk's current codegen and then rebase to show the codegen diffs

[luke]

Split out webassembly test into pre-commit test

[luke]

llvm/test/CodeGen/PowerPC/canonical-merge-shuffles.ll

1149–1162 ↗

(On Diff #483182)

Sorry you're right, not sure why I believed that. So if I'm understanding this correctly now, P8-AIX-NEXT checks are generated whenever aix-32 and aix-64 have the same lines.
Will take a look

[luke]

Only scalarize if scalar type is legal

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

static SDValue formSplatFromShuffles(ShuffleVectorSDNode *OuterShuf,
   assert(VT == InnerShuf->getValueType(0) && "Expected matching shuffle types");
   if (!DAG.getTargetLoweringInfo().isShuffleMaskLegal(CombinedMask, VT))
     return SDValue();
 
   return DAG.getVectorShuffle(VT, SDLoc(OuterShuf), InnerShuf->getOperand(0),
                               InnerShuf->getOperand(1), CombinedMask);
 }
 
+/// Given a vectorized load used in a splat, scalarize the load to only load the
+/// element required for splatting.
+static SDValue scalarizeLoadIntoSplat(ShuffleVectorSDNode *Shuf,
+                                      SelectionDAG &DAG) {
+  if (!Shuf->isSplat())
+    return SDValue();
+
+  EVT VecVT = Shuf->getOperand(0).getValueType();
+  SDValue SplattedOp;
+  if ((unsigned)Shuf->getSplatIndex() >= VecVT.getVectorNumElements())
+    SplattedOp = Shuf->getOperand(0);
+  else
+    SplattedOp = Shuf->getOperand(1);
+
+  LoadSDNode *Load = dyn_cast<LoadSDNode>(Shuf->getOperand(0).getNode());
+  if (!Load)
+    return SDValue();
+
+  if (!(Load->isSimple() && Load->hasOneUse() && VecVT.isVector()))
+    return SDValue();
+
+  auto &TLI = DAG.getTargetLoweringInfo();
+  SDValue SplatIdx =
+      DAG.getConstant(Shuf->getSplatIndex(), SDLoc(Shuf), MVT::i32);
+  SDValue NewPtr =
+      TLI.getVectorElementPointer(DAG, Load->getBasePtr(), VecVT, SplatIdx);
+
+  EVT VecEltVT = VecVT.getVectorElementType();
+  unsigned PtrOff = VecEltVT.getSizeInBits() * Shuf->getSplatIndex() / 8;
+  MachinePointerInfo MPI = Load->getPointerInfo().getWithOffset(PtrOff);
+  Align Alignment = commonAlignment(Load->getAlign(), PtrOff);
+
+  auto NewLoad = DAG.getLoad(VecEltVT, SDLoc(Load), Load->getChain(), NewPtr,
+                             MPI, Alignment, Load->getMemOperand()->getFlags(),
+                             Load->getAAInfo());
+  DAG.makeEquivalentMemoryOrdering(Load, NewLoad);
+
+  return DAG.getSplatBuildVector(Shuf->getValueType(0), SDLoc(Shuf), NewLoad);
+}
+
 /// If the shuffle mask is taking exactly one element from the first vector
 /// operand and passing through all other elements from the second vector
 /// operand, return the index of the mask element that is choosing an element
 /// from the first operand. Otherwise, return -1.
 static int getShuffleMaskIndexOfOneElementFromOp0IntoOp1(ArrayRef<int> Mask) {
   int MaskSize = Mask.size();
   int EltFromOp0 = -1;
   // TODO: This does not match if there are undef elements in the shuffle mask.
SDValue DAGCombiner::visitVECTOR_SHUFFLE(SDNode *N) {
 
   // A shuffle of a single vector that is a splatted value can always be folded.
   if (SDValue V = combineShuffleOfSplatVal(SVN, DAG))
     return V;
 
   if (SDValue V = formSplatFromShuffles(SVN, DAG))
     return V;
 
+  if (SDValue V = scalarizeLoadIntoSplat(SVN, DAG))
+    return V;
+
   // If it is a splat, check if the argument vector is another splat or a
   // build_vector.
   if (SVN->isSplat() && SVN->getSplatIndex() < (int)NumElts) {
     int SplatIndex = SVN->getSplatIndex();
     if (N0.hasOneUse() && TLI.isExtractVecEltCheap(VT, SplatIndex) &&
         TLI.isBinOp(N0.getOpcode()) && N0->getNumValues() == 1) {
       // splat (vector_bo L, R), Index -->
       // splat (scalar_bo (extelt L, Index), (extelt R, Index))

llvm/test/CodeGen/AArch64/sve-fixed-length-splat-vector.ll

; CHECK-NEXT:    ret
   store <8 x double> %splat, ptr %a
   ret void
 }
 
 define <8 x float> @load_splat_v8f32(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <8 x float>, ptr %p
   %splat = shufflevector <8 x float> %v, <8 x float> undef, <8 x i32> zeroinitializer
   ret <8 x float> %splat
 }
 
 define <4 x double> @load_splat_v4f64(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.d, d0
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <4 x double>, ptr %p
   %splat = shufflevector <4 x double> %v, <4 x double> undef, <4 x i32> zeroinitializer
   ret <4 x double> %splat
 }
 
 define <32 x i8> @load_splat_v32i8(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v32i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.b
-; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.b, b0
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
 ; CHECK-NEXT:    st1b { z0.b }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <32 x i8>, ptr %p
   %splat = shufflevector <32 x i8> %v, <32 x i8> undef, <32 x i32> zeroinitializer
   ret <32 x i8> %splat
 }
 
 define <16 x i16> @load_splat_v16i16(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v16i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <16 x i16>, ptr %p
   %splat = shufflevector <16 x i16> %v, <16 x i16> undef, <16 x i32> zeroinitializer
   ret <16 x i16> %splat
 }
 
 define <8 x i32> @load_splat_v8i32(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v8i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <8 x i32>, ptr %p
   %splat = shufflevector <8 x i32> %v, <8 x i32> undef, <8 x i32> zeroinitializer
   ret <8 x i32> %splat
 }
 
 define <4 x i64> @load_splat_v4i64(ptr %p) vscale_range(2,2) #0 {
 ; CHECK-LABEL: load_splat_v4i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    mov z0.d, d0
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x8]
 ; CHECK-NEXT:    ret
   %v = load <4 x i64>, ptr %p
   %splat = shufflevector <4 x i64> %v, <4 x i64> undef, <4 x i32> zeroinitializer
   ret <4 x i64> %splat
 }
 
 attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/WebAssembly/simd-vectorized-load-splat.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -verify-machineinstrs -mattr=+simd128 | FileCheck %s
+
+; Ensures that vectorized loads that are really just splatted loads, are indeed
+; selected as splatted loads
+
+target triple = "wasm32-unknown-unknown"
+
+define <4 x i32> @load_splat_shuhffle_lhs(ptr %p) {
+; CHECK-LABEL: load_splat_shuhffle_lhs:
+; CHECK:         .functype load_splat_shuhffle_lhs (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load64_splat 0
+; CHECK-NEXT:    # fallthrough-return
+  %a = load <2 x i64>, ptr %p
+  %b = shufflevector <2 x i64> %a, <2 x i64> poison, <2 x i32> <i32 0, i32 0>
+  %c = bitcast <2 x i64> %b to <4 x i32>
+  %d = shufflevector <4 x i32> %c, <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+  ret <4 x i32> %d
+}
+
+define <4 x i32> @load_splat_shuffle_lhs_with_offset(ptr %p) {
+; CHECK-LABEL: load_splat_shuffle_lhs_with_offset:
+; CHECK:         .functype load_splat_shuffle_lhs_with_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_splat 0
+; CHECK-NEXT:    # fallthrough-return
+  %a = load <2 x i64>, ptr %p
+  %b = shufflevector <2 x i64> %a, <2 x i64> poison, <2 x i32> <i32 1, i32 undef>
+  %c = bitcast <2 x i64> %b to <4 x i32>
+  %d = shufflevector <4 x i32> %c, <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+  ret <4 x i32> %d
+}
+
+define <4 x i32> @load_splat_shuffle_rhs(ptr %p) {
+; CHECK-LABEL: load_splat_shuffle_rhs:
+; CHECK:         .functype load_splat_shuffle_rhs (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    v128.load64_splat 0
+; CHECK-NEXT:    # fallthrough-return
+  %a = load <2 x i64>, ptr %p
+  %b = shufflevector <2 x i64> poison, <2 x i64> %a, <2 x i32> <i32 2, i32 undef>
+  %c = bitcast <2 x i64> %b to <4 x i32>
+  %d = shufflevector <4 x i32> %c, <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+  ret <4 x i32> %d
+}
+
+define <4 x i32> @load_splat_shuffle_rhs_with_offset(ptr %p) {
+; CHECK-LABEL: load_splat_shuffle_rhs_with_offset:
+; CHECK:         .functype load_splat_shuffle_rhs_with_offset (i32) -> (v128)
+; CHECK-NEXT:  # %bb.0:
+; CHECK-NEXT:    local.get 0
+; CHECK-NEXT:    i32.const 8
+; CHECK-NEXT:    i32.add
+; CHECK-NEXT:    v128.load64_splat 0
+; CHECK-NEXT:    # fallthrough-return
+  %a = load <2 x i64>, ptr %p
+  %b = shufflevector <2 x i64> poison, <2 x i64> %a, <2 x i32> <i32 3, i32 undef>
+  %c = bitcast <2 x i64> %b to <4 x i32>
+  %d = shufflevector <4 x i32> %c, <4 x i32> poison, <4 x i32> <i32 0, i32 1, i32 0, i32 1>
+  ret <4 x i32> %d
+}

llvm/test/CodeGen/X86/extractelement-load.ll

 ; Case where a load is unary shuffled, then bitcast (to a type with the same
 ; number of elements) before extractelement.
 ; This is testing for an assertion - the extraction was assuming that the undef
 ; second shuffle operand was a post-bitcast type instead of a pre-bitcast type.
 define i64 @t4(ptr %a) {
 ; X32-SSE2-LABEL: t4:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE2-NEXT:    movdqa (%eax), %xmm0
+; X32-SSE2-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
 ; X32-SSE2-NEXT:    movd %xmm0, %eax
 ; X32-SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[1,1,1,1]
 ; X32-SSE2-NEXT:    movd %xmm0, %edx
 ; X32-SSE2-NEXT:    retl
 ;
 ; X64-LABEL: t4:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq (%rdi), %rax

llvm/test/CodeGen/X86/sse3.ll

entry:
   %tmp9 = shufflevector <16 x i8> %tmp8, <16 x i8> %T0,  <16 x i32> < i32 0, i32 1, i32 2, i32 17,  i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef , i32 undef >
   ret <16 x i8> %tmp9
 }
 
 ; rdar://8520311
 define <4 x i32> @t17() nounwind {
 ; X86-LABEL: t17:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    pshufd {{.*#+}} xmm0 = mem[0,1,0,1]
-; X86-NEXT:    pand {{\.?LCPI[0-9]+_[0-9]+}}, %xmm0
+; X86-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; X86-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,0,1]
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: t17:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pshufd {{.*#+}} xmm0 = mem[0,1,0,1]
-; X64-NEXT:    pand {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; X64-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,0,1]
 ; X64-NEXT:    retq
 entry:
   %tmp1 = load <4 x float>, ptr undef, align 16
   %tmp2 = shufflevector <4 x float> %tmp1, <4 x float> undef, <4 x i32> <i32 4, i32 1, i32 2, i32 3>
   %tmp3 = load <4 x float>, ptr undef, align 16
   %tmp4 = shufflevector <4 x float> %tmp2, <4 x float> undef, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
   %tmp5 = bitcast <4 x float> %tmp3 to <4 x i32>
   %tmp6 = shufflevector <4 x i32> %tmp5, <4 x i32> undef, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
   %tmp7 = and <4 x i32> %tmp6, <i32 undef, i32 undef, i32 -1, i32 0>
   ret <4 x i32> %tmp7
 }

llvm/test/CodeGen/X86/sse41.ll

; X64-AVX512-NEXT:    retq ## encoding: [0xc3]
   %7 = tail call <4 x float> @llvm.x86.sse41.insertps(<4 x float> %a, <4 x float> %6, i32 48)
   ret <4 x float> %7
 }
 
 define <4 x float> @insertps_from_broadcast_loadv4f32(<4 x float> %a, ptr nocapture readonly %b) {
 ; X86-SSE-LABEL: insertps_from_broadcast_loadv4f32:
 ; X86-SSE:       ## %bb.0:
 ; X86-SSE-NEXT:    movl {{[0-9]+}}(%esp), %eax ## encoding: [0x8b,0x44,0x24,0x04]
-; X86-SSE-NEXT:    movups (%eax), %xmm1 ## encoding: [0x0f,0x10,0x08]
-; X86-SSE-NEXT:    insertps $48, %xmm1, %xmm0 ## encoding: [0x66,0x0f,0x3a,0x21,0xc1,0x30]
-; X86-SSE-NEXT:    ## xmm0 = xmm0[0,1,2],xmm1[0]
+; X86-SSE-NEXT:    insertps $48, (%eax), %xmm0 ## encoding: [0x66,0x0f,0x3a,0x21,0x00,0x30]
+; X86-SSE-NEXT:    ## xmm0 = xmm0[0,1,2],mem[0]
 ; X86-SSE-NEXT:    retl ## encoding: [0xc3]
 ;
 ; X86-AVX1-LABEL: insertps_from_broadcast_loadv4f32:
 ; X86-AVX1:       ## %bb.0:
 ; X86-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %eax ## encoding: [0x8b,0x44,0x24,0x04]
 ; X86-AVX1-NEXT:    vinsertps $48, (%eax), %xmm0, %xmm0 ## encoding: [0xc4,0xe3,0x79,0x21,0x00,0x30]
 ; X86-AVX1-NEXT:    ## xmm0 = xmm0[0,1,2],mem[0]
 ; X86-AVX1-NEXT:    retl ## encoding: [0xc3]
 ;
 ; X86-AVX512-LABEL: insertps_from_broadcast_loadv4f32:
 ; X86-AVX512:       ## %bb.0:
 ; X86-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax ## encoding: [0x8b,0x44,0x24,0x04]
 ; X86-AVX512-NEXT:    vinsertps $48, (%eax), %xmm0, %xmm0 ## EVEX TO VEX Compression encoding: [0xc4,0xe3,0x79,0x21,0x00,0x30]
 ; X86-AVX512-NEXT:    ## xmm0 = xmm0[0,1,2],mem[0]
 ; X86-AVX512-NEXT:    retl ## encoding: [0xc3]
 ;
 ; X64-SSE-LABEL: insertps_from_broadcast_loadv4f32:
 ; X64-SSE:       ## %bb.0:
-; X64-SSE-NEXT:    movups (%rdi), %xmm1 ## encoding: [0x0f,0x10,0x0f]
-; X64-SSE-NEXT:    insertps $48, %xmm1, %xmm0 ## encoding: [0x66,0x0f,0x3a,0x21,0xc1,0x30]
-; X64-SSE-NEXT:    ## xmm0 = xmm0[0,1,2],xmm1[0]
+; X64-SSE-NEXT:    insertps $48, (%rdi), %xmm0 ## encoding: [0x66,0x0f,0x3a,0x21,0x07,0x30]
+; X64-SSE-NEXT:    ## xmm0 = xmm0[0,1,2],mem[0]
 ; X64-SSE-NEXT:    retq ## encoding: [0xc3]
 ;
 ; X64-AVX1-LABEL: insertps_from_broadcast_loadv4f32:
 ; X64-AVX1:       ## %bb.0:
 ; X64-AVX1-NEXT:    vinsertps $48, (%rdi), %xmm0, %xmm0 ## encoding: [0xc4,0xe3,0x79,0x21,0x07,0x30]
 ; X64-AVX1-NEXT:    ## xmm0 = xmm0[0,1,2],mem[0]
 ; X64-AVX1-NEXT:    retq ## encoding: [0xc3]
 ;

llvm/test/CodeGen/X86/vector-shuffle-128-v2.ll

; AVX-NEXT:    retq
   %v = insertelement <2 x double> undef, double %a, i32 0
   %shuffle = shufflevector <2 x double> %v, <2 x double> undef, <2 x i32> <i32 0, i32 0>
   ret <2 x double> %shuffle
 }
 
 define <2 x double> @insert_dup_mem128_v2f64(ptr %ptr) nounwind {
 ; SSE2-LABEL: insert_dup_mem128_v2f64:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movaps (%rdi), %xmm0
+; SSE2-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; SSE2-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0,0]
 ; SSE2-NEXT:    retq
 ;
 ; SSE3-LABEL: insert_dup_mem128_v2f64:
 ; SSE3:       # %bb.0:
 ; SSE3-NEXT:    movddup {{.*#+}} xmm0 = mem[0,0]
 ; SSE3-NEXT:    retq
 ;

llvm/test/CodeGen/X86/widened-broadcast.ll

entry:
   %ld = load <32 x i8>, ptr %ptr
   %ret = shufflevector <32 x i8> %ld, <32 x i8> undef, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   ret <32 x i8> %ret
 }
 
 define <4 x float> @load_splat_4f32_8f32_0000(ptr %ptr) nounwind uwtable readnone ssp {
 ; SSE-LABEL: load_splat_4f32_8f32_0000:
 ; SSE:       # %bb.0: # %entry
-; SSE-NEXT:    movaps (%rdi), %xmm0
+; SSE-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; SSE-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,0,0,0]
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: load_splat_4f32_8f32_0000:
 ; AVX:       # %bb.0: # %entry
 ; AVX-NEXT:    vbroadcastss (%rdi), %xmm0
 ; AVX-NEXT:    retq
 entry: