[X86] Reenable store merging post-legalization

[lebedev.ri]

Summary

D62498 disabled store merging post-legalization,
because we do indeed want to split store of concat.

But the current way it's disabled is a way too heavy of a hammer.
We only want to disable the cases that would be problematic,
but e.g. still want to allow store merging of loads.

This requires D141776, otherwise we have conflicting combines.

[lebedev.ri]

(wrong diff)

[lebedev.ri]

Now with a proper diff.

[lebedev.ri]

llvm/lib/Target/X86/X86ISelLowering.h

1074–1075

This is a bit ugly. I'm using StoreSource::Unknown as
"we are post-legalization. do we allow merging *any* sources?".
Is that too ugly?

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19397–19398 ↗

(On Diff #489316)

If I read the patch correctly, this is the real point we care about. Should it be better to just introduce another TLI function to do such work?

[lebedev.ri]

Don't abuse StoreSource::Unknown, wrap it into std::optional<>.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19397–19398 ↗

(On Diff #489316)

I don't see why? We are simply enriching it with more knowledge to give a more precise answer.

[barannikov88]

llvm/include/llvm/CodeGen/TargetLowering.h

606 ↗

(On Diff #489349)

Appears unused with std::optional.

llvm/lib/Target/X86/X86ISelLowering.h

1074–1075

Just always pass "valid" StoreSrc as determined by getStoreSource?

[lebedev.ri]

llvm/include/llvm/CodeGen/TargetLowering.h

606 ↗

(On Diff #489349)

I do not follow. Both the struct and the element are both clearly still used.

llvm/lib/Target/X86/X86ISelLowering.h

1074–1075

The idea is that there is no point in ever trying to guess the StoreSource,
which may be costly to do, if the target will always refuse.

[barannikov88]

llvm/include/llvm/CodeGen/TargetLowering.h

606 ↗

(On Diff #489349)

Yes, my bad. Although mergeStoresAfterLegalization will never receive Unknown.
I'd suggest to stick to either Unknown or std::optional, but not both. Either is fine to me.

[lebedev.ri]

llvm/include/llvm/CodeGen/TargetLowering.h

606 ↗

(On Diff #489349)

Note that i'm simply moving the existing enum, it can't not have all of it's original elements.

[lebedev.ri]

ping

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675 ↗

(On Diff #489349)

I'd agree with @barannikov88. Unknown and optional represent the same meaning. Why don't return std::nullopt here too?

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675 ↗

(On Diff #489349)

The literally next change will return splat here instead.

[lebedev.ri]

Rebased, NFC.
Now that D141776 has landed, this can proceed.

[RKSimon]

CC'ing @TokarIP who recently added the light-avx tuning flag for something similar.

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h

247 ↗

(On Diff #491099)

drop unused StoreSrc?

mergeStoresAfterLegalization(EVT, std::optional<StoreSource>)

llvm/lib/Target/X86/X86ISelLowering.h

1076

should we play safe and treat unknown as a potential extractsubvector?

[lebedev.ri]

llvm/lib/Target/X86/X86ISelLowering.h

1076

Let me just assert that we don't get unknown here.

[lebedev.ri]

@RKSimon thank you for taking a look!
Addressing nits.

llvm/lib/Target/X86/X86ISelLowering.h

public:
     /// Returns true if it's safe to use load / store of the
     /// specified type to expand memcpy / memset inline. This is mostly true
     /// for all types except for some special cases. For example, on X86
     /// targets without SSE2 f64 load / store are done with fldl / fstpl which
     /// also does type conversion. Note the specified type doesn't have to be
     /// legal as the hook is used before type legalization.
     bool isSafeMemOpType(MVT VT) const override;
 
+    bool isMemoryAccessesFast(EVT VT, Align Alignment) const;
+
     /// Returns true if the target allows unaligned memory accesses of the
     /// specified type. Returns whether it is "fast" in the last argument.
     bool allowsMisalignedMemoryAccesses(EVT VT, unsigned AS, Align Alignment,
                                         MachineMemOperand::Flags Flags,
                                         unsigned *Fast) const override;
 
     /// This function returns true if the memory access is aligned or if the
     /// target allows this specific unaligned memory access. If the access is
public:
 
     /// Do not merge vector stores after legalization because that may conflict
     /// with x86-specific store splitting optimizations.
     bool mergeStoresAfterLegalization(EVT MemVT,
                                       StoreSource StoreSrc) const override {
       return StoreSrc == StoreSource::Unknown ||
              StoreSrc != StoreSource::Extract;
     }
 
     bool canMergeStoresTo(unsigned AddressSpace, EVT MemVT,
                           const MachineFunction &MF) const override;
 
     bool isCheapToSpeculateCttz(Type *Ty) const override;
 
     bool isCheapToSpeculateCtlz(Type *Ty) const override;
 
     bool isCtlzFast() const override;
 
     bool hasBitPreservingFPLogic(EVT VT) const override;

llvm/lib/Target/X86/X86ISelLowering.cpp

 bool X86TargetLowering::isSafeMemOpType(MVT VT) const {
   if (VT == MVT::f32)
     return Subtarget.hasSSE1();
   if (VT == MVT::f64)
     return Subtarget.hasSSE2();
   return true;
 }
 
-bool X86TargetLowering::allowsMisalignedMemoryAccesses(
-    EVT VT, unsigned, Align Alignment, MachineMemOperand::Flags Flags,
-    unsigned *Fast) const {
-  if (Fast) {
-    switch (VT.getSizeInBits()) {
-    default:
-      // 8-byte and under are always assumed to be fast.
-      *Fast = 1;
-      break;
-    case 128:
-      *Fast = !Subtarget.isUnalignedMem16Slow();
-      break;
-    case 256:
-      *Fast = !Subtarget.isUnalignedMem32Slow();
-      break;
+bool X86TargetLowering::isMemoryAccessesFast(EVT VT, Align Alignment) const {
+  bool IsMisaligned = (8 * Alignment.value()) % VT.getSizeInBits() != 0;
+  if (!IsMisaligned)
+    return true;
+    return true;
+    return !Subtarget.isUnalignedMem16Slow();
+    return !Subtarget.isUnalignedMem32Slow();
     // TODO: What about AVX-512 (512-bit) accesses?
-    }
-  }
+
+bool X86TargetLowering::allowsMisalignedMemoryAccesses(
+    EVT VT, unsigned, Align Alignment, MachineMemOperand::Flags Flags,
+    unsigned *Fast) const {
+  if (Fast)
+    *Fast = isMemoryAccessesFast(VT, Alignment);
   // NonTemporal vector memory ops must be aligned.
   if (!!(Flags & MachineMemOperand::MONonTemporal) && VT.isVector()) {
     // NT loads can only be vector aligned, so if its less aligned than the
     // minimum vector size (which we can split the vector down to), we might as
     // well use a regular unaligned vector load.
     // We don't have any NT loads pre-SSE41.
     if (!!(Flags & MachineMemOperand::MOLoad))
       return (Alignment < 16 || !Subtarget.hasSSE41());
     return false;
   }
   // Misaligned accesses of any size are always allowed.
   return true;
 }
 
 bool X86TargetLowering::allowsMemoryAccess(LLVMContext &Context,
                                            const DataLayout &DL, EVT VT,
                                            unsigned AddrSpace, Align Alignment,
                                            MachineMemOperand::Flags Flags,
                                            unsigned *Fast) const {
-  return allowsMisalignedMemoryAccesses(VT, AddrSpace, Alignment, Flags, Fast);
+  if (Fast)
+    *Fast = isMemoryAccessesFast(VT, Alignment);
+  if (!!(Flags & MachineMemOperand::MONonTemporal) && VT.isVector()) {
+    if (allowsMisalignedMemoryAccesses(VT, AddrSpace, Alignment, Flags,
+                                       /*Fast=*/nullptr))
+      return true;
+    // NonTemporal vector memory ops are special, and must be aligned.
+    bool IsMisaligned = (8 * Alignment.value()) % VT.getSizeInBits() != 0;
+    if (IsMisaligned)
+      return false;
+    switch (VT.getSizeInBits()) {
+    case 128:
+      if (!!(Flags & MachineMemOperand::MOLoad) && Subtarget.hasSSE41())
+        return true;
+      if (!!(Flags & MachineMemOperand::MOStore) && Subtarget.hasSSE2())
+        return true;
+      return false;
+    case 256:
+      if (!!(Flags & MachineMemOperand::MOLoad) && Subtarget.hasAVX2())
+        return true;
+      if (!!(Flags & MachineMemOperand::MOStore) && Subtarget.hasAVX())
+        return true;
+      return false;
+    case 512:
+      if (Subtarget.hasAVX512())
+        return true;
+      return false;
+    default:
+      return false; // Don't have NonTemporal vector memory ops of this size.
+    }
+  }
+  return true;
 }
 
 /// Return the entry encoding for a jump table in the
 /// current function.  The returned value is a member of the
 /// MachineJumpTableInfo::JTEntryKind enum.
 unsigned X86TargetLowering::getJumpTableEncoding() const {
   // In GOT pic mode, each entry in the jump table is emitted as a @GOTOFF
   // symbol.

llvm/test/CodeGen/X86/avx-arith.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7-avx -mattr=+avx | FileCheck %s
 
 define <4 x double> @addpd256(<4 x double> %y, <4 x double> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: addpd256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vaddpd %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %add.i = fadd <4 x double> %x, %y
   ret <4 x double> %add.i
 }
 
 define <4 x double> @addpd256fold(<4 x double> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: addpd256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.3999999999999999E+0,2.2999999999999998E+0,1.2E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vaddpd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %add.i = fadd <4 x double> %y, <double 4.500000e+00, double 3.400000e+00, double 2.300000e+00, double 1.200000e+00>
   ret <4 x double> %add.i
 }
 
 define <8 x float> @addps256(<8 x float> %y, <8 x float> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: addps256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %add.i = fadd <8 x float> %x, %y
   ret <8 x float> %add.i
 }
 
 define <8 x float> @addps256fold(<8 x float> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: addps256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0,4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vaddps %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vaddps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %add.i = fadd <8 x float> %y, <float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000, float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000>
   ret <8 x float> %add.i
 }
 
 define <4 x double> @subpd256(<4 x double> %y, <4 x double> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: subpd256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vsubpd %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %sub.i = fsub <4 x double> %x, %y
   ret <4 x double> %sub.i
 }
 
 define <4 x double> @subpd256fold(<4 x double> %y, ptr nocapture %x) nounwind uwtable readonly ssp {
 ; CHECK-LABEL: subpd256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps (%rdi), %xmm1
-; CHECK-NEXT:    vinsertf128 $1, 16(%rdi), %ymm1, %ymm1
-; CHECK-NEXT:    vsubpd %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vsubpd (%rdi), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %tmp2 = load <4 x double>, ptr %x, align 32
   %sub.i = fsub <4 x double> %y, %tmp2
   ret <4 x double> %sub.i
 }
 
 define <8 x float> @subps256(<8 x float> %y, <8 x float> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: subps256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vsubps %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %sub.i = fsub <8 x float> %x, %y
   ret <8 x float> %sub.i
 }
 
 define <8 x float> @subps256fold(<8 x float> %y, ptr nocapture %x) nounwind uwtable readonly ssp {
 ; CHECK-LABEL: subps256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps (%rdi), %xmm1
-; CHECK-NEXT:    vinsertf128 $1, 16(%rdi), %ymm1, %ymm1
-; CHECK-NEXT:    vsubps %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vsubps (%rdi), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %tmp2 = load <8 x float>, ptr %x, align 32
   %sub.i = fsub <8 x float> %y, %tmp2
   ret <8 x float> %sub.i
 }
 
 define <4 x double> @mulpd256(<4 x double> %y, <4 x double> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: mulpd256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vmulpd %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %mul.i = fmul <4 x double> %x, %y
   ret <4 x double> %mul.i
 }
 
 define <4 x double> @mulpd256fold(<4 x double> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: mulpd256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.3999999999999999E+0,2.2999999999999998E+0,1.2E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vmulpd %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vmulpd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %mul.i = fmul <4 x double> %y, <double 4.500000e+00, double 3.400000e+00, double 2.300000e+00, double 1.200000e+00>
   ret <4 x double> %mul.i
 }
 
 define <8 x float> @mulps256(<8 x float> %y, <8 x float> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: mulps256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %mul.i = fmul <8 x float> %x, %y
   ret <8 x float> %mul.i
 }
 
 define <8 x float> @mulps256fold(<8 x float> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: mulps256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0,4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vmulps %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %mul.i = fmul <8 x float> %y, <float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000, float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000>
   ret <8 x float> %mul.i
 }
 
 define <4 x double> @divpd256(<4 x double> %y, <4 x double> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: divpd256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vdivpd %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %div.i = fdiv <4 x double> %x, %y
   ret <4 x double> %div.i
 }
 
 define <4 x double> @divpd256fold(<4 x double> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: divpd256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.3999999999999999E+0,2.2999999999999998E+0,1.2E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vdivpd %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vdivpd {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %div.i = fdiv <4 x double> %y, <double 4.500000e+00, double 3.400000e+00, double 2.300000e+00, double 1.200000e+00>
   ret <4 x double> %div.i
 }
 
 define <8 x float> @divps256(<8 x float> %y, <8 x float> %x) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: divps256:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %div.i = fdiv <8 x float> %x, %y
   ret <8 x float> %div.i
 }
 
 define <8 x float> @divps256fold(<8 x float> %y) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: divps256fold:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm1 = [4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0,4.5E+0,3.4000001E+0,2.29999995E+0,1.20000005E+0]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; CHECK-NEXT:    vdivps %ymm1, %ymm0, %ymm0
+; CHECK-NEXT:    vdivps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %div.i = fdiv <8 x float> %y, <float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000, float 4.500000e+00, float 0x400B333340000000, float 0x4002666660000000, float 0x3FF3333340000000>
   ret <8 x float> %div.i
 }
 
 define float @sqrtA(float %a) nounwind uwtable readnone ssp {
 ; CHECK-LABEL: sqrtA:

llvm/test/CodeGen/X86/avx-load-store.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,slow-unaligned-mem-32 | FileCheck %s
 ; RUN: llc -O0 < %s -mtriple=x86_64-unknown-unknown -mattr=avx,slow-unaligned-mem-32 | FileCheck %s -check-prefix=CHECK_O0
 
 define void @test_256_load(ptr nocapture %d, ptr nocapture %f, ptr nocapture %i) nounwind {
 ; CHECK-LABEL: test_256_load:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushq %r15
 ; CHECK-NEXT:    pushq %r14
 ; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    subq $144, %rsp
+; CHECK-NEXT:    subq $96, %rsp
 ; CHECK-NEXT:    movq %rdx, %rbx
 ; CHECK-NEXT:    movq %rsi, %r14
 ; CHECK-NEXT:    movq %rdi, %r15
-; CHECK-NEXT:    vmovaps (%rdi), %xmm1
+; CHECK-NEXT:    vmovaps (%rdi), %ymm0
+; CHECK-NEXT:    vmovups %ymm0, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    vmovaps (%rsi), %ymm1
 ; CHECK-NEXT:    vmovups %ymm1, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
-; CHECK-NEXT:    vmovaps 16(%rdi), %xmm0
-; CHECK-NEXT:    vmovaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
-; CHECK-NEXT:    vmovaps (%rsi), %xmm2
-; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
-; CHECK-NEXT:    vmovaps 16(%rsi), %xmm1
-; CHECK-NEXT:    vmovaps %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
-; CHECK-NEXT:    vmovaps (%rdx), %xmm3
-; CHECK-NEXT:    vmovups %ymm3, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
-; CHECK-NEXT:    vmovaps 16(%rdx), %xmm2
-; CHECK-NEXT:    vmovaps %xmm2, (%rsp) # 16-byte Spill
-; CHECK-NEXT:    vinsertf128 $1, %xmm2, %ymm3, %ymm2
+; CHECK-NEXT:    vmovaps (%rdx), %ymm2
+; CHECK-NEXT:    vmovups %ymm2, (%rsp) # 32-byte Spill
 ; CHECK-NEXT:    callq dummy@PLT
 ; CHECK-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %ymm0 # 32-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, (%r15)
-; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, 16(%r15)
-; CHECK-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %ymm0 # 32-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, (%r14)
-; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, 16(%r14)
+; CHECK-NEXT:    vmovaps %ymm0, (%r15)
 ; CHECK-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %ymm0 # 32-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, (%rbx)
-; CHECK-NEXT:    vmovaps (%rsp), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, 16(%rbx)
-; CHECK-NEXT:    addq $144, %rsp
+; CHECK-NEXT:    vmovaps %ymm0, (%r14)
+; CHECK-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
+; CHECK-NEXT:    vmovaps %ymm0, (%rbx)
+; CHECK-NEXT:    addq $96, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 ;
 ; CHECK_O0-LABEL: test_256_load:
 ; CHECK_O0:       # %bb.0: # %entry
; CHECK_O0-NEXT:    retq
   %val = load double, ptr %ptr
   %i0 = insertelement <4 x double> zeroinitializer, double %val, i32 0
   ret <4 x double> %i0
 }
 
 define void @storev16i16(<16 x i16> %a) nounwind {
 ; CHECK-LABEL: storev16i16:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vextractf128 $1, %ymm0, (%rax)
-; CHECK-NEXT:    vmovaps %xmm0, (%rax)
+; CHECK-NEXT:    vmovaps %ymm0, (%rax)
 ;
 ; CHECK_O0-LABEL: storev16i16:
 ; CHECK_O0:       # %bb.0:
 ; CHECK_O0-NEXT:    # implicit-def: $rax
 ; CHECK_O0-NEXT:    vmovdqa %ymm0, (%rax)
   store <16 x i16> %a, ptr undef, align 32
   unreachable
 }

 ; CHECK_O0-NEXT:    vmovdqu %ymm0, (%rax)
   store <16 x i16> %a, ptr undef, align 4
   unreachable
 }
 
 define void @storev32i8(<32 x i8> %a) nounwind {
 ; CHECK-LABEL: storev32i8:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vextractf128 $1, %ymm0, (%rax)
-; CHECK-NEXT:    vmovaps %xmm0, (%rax)
+; CHECK-NEXT:    vmovaps %ymm0, (%rax)
 ;
 ; CHECK_O0-LABEL: storev32i8:
 ; CHECK_O0:       # %bb.0:
 ; CHECK_O0-NEXT:    # implicit-def: $rax
 ; CHECK_O0-NEXT:    vmovdqa %ymm0, (%rax)
   store <32 x i8> %a, ptr undef, align 32
   unreachable
 }
; CHECK_O0-NEXT:    retq
   %x = add <8 x i32> zeroinitializer, %b
   store <8 x i32> %x, ptr %ret, align 1
   ret void
 }
 
 define void @add4i64a64(ptr %ret, ptr %bp) nounwind {
 ; CHECK-LABEL: add4i64a64:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovaps (%rsi), %xmm0
-; CHECK-NEXT:    vmovaps 16(%rsi), %xmm1
-; CHECK-NEXT:    vmovaps %xmm0, (%rdi)
-; CHECK-NEXT:    vmovaps %xmm1, 16(%rdi)
+; CHECK-NEXT:    vmovaps (%rsi), %ymm0
+; CHECK-NEXT:    vmovaps %ymm0, (%rdi)
+; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 ;
 ; CHECK_O0-LABEL: add4i64a64:
 ; CHECK_O0:       # %bb.0:
-; CHECK_O0-NEXT:    vmovdqa (%rsi), %xmm2
-; CHECK_O0-NEXT:    vmovdqa 16(%rsi), %xmm1
-; CHECK_O0-NEXT:    # implicit-def: $ymm0
-; CHECK_O0-NEXT:    vmovaps %xmm2, %xmm0
-; CHECK_O0-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; CHECK_O0-NEXT:    vmovaps (%rsi), %ymm0
 ; CHECK_O0-NEXT:    vmovdqa %ymm0, (%rdi)
 ; CHECK_O0-NEXT:    vzeroupper
 ; CHECK_O0-NEXT:    retq
   %b = load <4 x i64>, ptr %bp, align 64
   %x = add <4 x i64> zeroinitializer, %b
   store <4 x i64> %x, ptr %ret, align 64
   ret void
 }

llvm/test/CodeGen/X86/avx-varargs-x86_64.ll

 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    andq $-32, %rsp
 ; CHECK-NEXT:    subq $64, %rsp
 ; CHECK-NEXT:    movq _x@GOTPCREL(%rip), %rax
-; CHECK-NEXT:    vmovaps (%rax), %xmm0
-; CHECK-NEXT:    vmovaps 16(%rax), %xmm1
-; CHECK-NEXT:    vmovaps %xmm1, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    vmovaps %xmm0, (%rsp)
+; CHECK-NEXT:    vmovaps (%rax), %ymm0
+; CHECK-NEXT:    vmovaps %ymm0, (%rsp)
 ; CHECK-NEXT:    movl $1, %edi
 ; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    callq _f
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    retq
 entry:
   %0 = load <8 x float>, ptr @x, align 32
   %call = call i32 (i32, ...) @f(i32 1, <8 x float> %0)
   ret void
 }

llvm/test/CodeGen/X86/fold-vector-sext-crash2.ll

 ; X32-NEXT:    movl $0, 8(%eax)
 ; X32-NEXT:    movl $0, 4(%eax)
 ; X32-NEXT:    movl $0, (%eax)
 ; X32-NEXT:    retl $4
 ;
 ; X64-LABEL: test_sext1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
+; X64-NEXT:    movaps %xmm0, (%rdi)
 ; X64-NEXT:    movq $-1, 56(%rdi)
 ; X64-NEXT:    movq $-1, 48(%rdi)
 ; X64-NEXT:    movq $-1, 40(%rdi)
 ; X64-NEXT:    movq $-99, 32(%rdi)
-; X64-NEXT:    movq $0, 24(%rdi)
-; X64-NEXT:    movq $0, 16(%rdi)
-; X64-NEXT:    movq $0, 8(%rdi)
-; X64-NEXT:    movq $0, (%rdi)
 ; X64-NEXT:    retq
   %Se = sext <2 x i8> <i8 -100, i8 -99> to <2 x i256>
   %Shuff = shufflevector <2 x i256> zeroinitializer, <2 x i256> %Se, <2 x i32> <i32 1, i32 3>
   ret <2 x i256> %Shuff
 }
 
 define <2 x i256> @test_sext2() {
 ; X32-LABEL: test_sext2:

 ; X32-NEXT:    movl $0, 8(%eax)
 ; X32-NEXT:    movl $0, 4(%eax)
 ; X32-NEXT:    movl $0, (%eax)
 ; X32-NEXT:    retl $4
 ;
 ; X64-LABEL: test_sext2:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
+; X64-NEXT:    movaps %xmm0, (%rdi)
 ; X64-NEXT:    movq $-1, 56(%rdi)
 ; X64-NEXT:    movq $-1, 48(%rdi)
 ; X64-NEXT:    movq $-1, 40(%rdi)
 ; X64-NEXT:    movq $-1999, 32(%rdi) # imm = 0xF831
-; X64-NEXT:    movq $0, 24(%rdi)
-; X64-NEXT:    movq $0, 16(%rdi)
-; X64-NEXT:    movq $0, 8(%rdi)
-; X64-NEXT:    movq $0, (%rdi)
 ; X64-NEXT:    retq
   %Se = sext <2 x i128> <i128 -2000, i128 -1999> to <2 x i256>
   %Shuff = shufflevector <2 x i256> zeroinitializer, <2 x i256> %Se, <2 x i32> <i32 1, i32 3>
   ret <2 x i256> %Shuff
 }
 
 define <2 x i256> @test_zext1() {
 ; X32-LABEL: test_zext1:

 ; X32-NEXT:    movl $0, 8(%eax)
 ; X32-NEXT:    movl $0, 4(%eax)
 ; X32-NEXT:    movl $0, (%eax)
 ; X32-NEXT:    retl $4
 ;
 ; X64-LABEL: test_zext1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
-; X64-NEXT:    movq $0, 56(%rdi)
-; X64-NEXT:    movq $0, 48(%rdi)
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 48(%rdi)
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
+; X64-NEXT:    movaps %xmm0, (%rdi)
 ; X64-NEXT:    movq $0, 40(%rdi)
 ; X64-NEXT:    movq $254, 32(%rdi)
-; X64-NEXT:    movq $0, 24(%rdi)
-; X64-NEXT:    movq $0, 16(%rdi)
-; X64-NEXT:    movq $0, 8(%rdi)
-; X64-NEXT:    movq $0, (%rdi)
 ; X64-NEXT:    retq
   %Se = zext <2 x i8> <i8 -1, i8 -2> to <2 x i256>
   %Shuff = shufflevector <2 x i256> zeroinitializer, <2 x i256> %Se, <2 x i32> <i32 1, i32 3>
   ret <2 x i256> %Shuff
 }
 
 define <2 x i256> @test_zext2() {
 ; X32-LABEL: test_zext2:

 ; X32-NEXT:    movl $0, 8(%eax)
 ; X32-NEXT:    movl $0, 4(%eax)
 ; X32-NEXT:    movl $0, (%eax)
 ; X32-NEXT:    retl $4
 ;
 ; X64-LABEL: test_zext2:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
-; X64-NEXT:    movq $0, 56(%rdi)
-; X64-NEXT:    movq $0, 48(%rdi)
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 48(%rdi)
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
+; X64-NEXT:    movaps %xmm0, (%rdi)
 ; X64-NEXT:    movq $-1, 40(%rdi)
 ; X64-NEXT:    movq $-2, 32(%rdi)
-; X64-NEXT:    movq $0, 24(%rdi)
-; X64-NEXT:    movq $0, 16(%rdi)
-; X64-NEXT:    movq $0, 8(%rdi)
-; X64-NEXT:    movq $0, (%rdi)
 ; X64-NEXT:    retq
   %Se = zext <2 x i128> <i128 -1, i128 -2> to <2 x i256>
   %Shuff = shufflevector <2 x i256> zeroinitializer, <2 x i256> %Se, <2 x i32> <i32 1, i32 3>
   ret <2 x i256> %Shuff
 }

llvm/test/CodeGen/X86/fp-load-trunc.ll

 ; CHECK-NEXT:    cvtpd2ps 16(%eax), %xmm1
 ; CHECK-NEXT:    cvtpd2ps (%eax), %xmm0
 ; CHECK-NEXT:    unpcklpd {{.*#+}} xmm0 = xmm0[0],xmm1[0]
 ; CHECK-NEXT:    retl
 ;
 ; AVX-LABEL: test3:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; AVX-NEXT:    vmovaps (%eax), %xmm0
-; AVX-NEXT:    vinsertf128 $1, 16(%eax), %ymm0, %ymm0
-; AVX-NEXT:    vcvtpd2ps %ymm0, %xmm0
-; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    vcvtpd2psy (%eax), %xmm0
 ; AVX-NEXT:    retl
   %x = load <4 x double>, ptr %p
   %y = fptrunc <4 x double> %x to <4 x float>
   ret <4 x float> %y
 }
 
 define <8 x float> @test4(ptr %p) nounwind {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    cvtpd2ps 16(%eax), %xmm2
 ; CHECK-NEXT:    cvtpd2ps (%eax), %xmm0
 ; CHECK-NEXT:    cvtpd2ps 48(%eax), %xmm3
 ; CHECK-NEXT:    cvtpd2ps 32(%eax), %xmm1
 ; CHECK-NEXT:    unpcklpd {{.*#+}} xmm0 = xmm0[0],xmm2[0]
 ; CHECK-NEXT:    unpcklpd {{.*#+}} xmm1 = xmm1[0],xmm3[0]
 ; CHECK-NEXT:    retl
 ;
 ; AVX-LABEL: test4:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; AVX-NEXT:    vmovaps (%eax), %xmm0
-; AVX-NEXT:    vmovaps 32(%eax), %xmm1
-; AVX-NEXT:    vinsertf128 $1, 48(%eax), %ymm1, %ymm1
-; AVX-NEXT:    vinsertf128 $1, 16(%eax), %ymm0, %ymm0
-; AVX-NEXT:    vcvtpd2ps %ymm0, %xmm0
-; AVX-NEXT:    vcvtpd2ps %ymm1, %xmm1
+; AVX-NEXT:    vcvtpd2psy (%eax), %xmm0
+; AVX-NEXT:    vcvtpd2psy 32(%eax), %xmm1
 ; AVX-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX-NEXT:    retl
   %x = load <8 x double>, ptr %p
   %y = fptrunc <8 x double> %x to <8 x float>
   ret <8 x float> %y
 }
 
 

llvm/test/CodeGen/X86/i128-fpconv-win64-strict.ll

; WIN64-NEXT:    retq
   %4 = trunc i128 %3 to i64
   ret i64 %4
 }
 
 define double @i128_to_double(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_double:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floattidf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = tail call double @llvm.experimental.constrained.sitofp.f64.i128(i128 %2, metadata !"round.dynamic", metadata !"fpexcept.strict")
   ret double %3
 }
 
 define double @ui128_to_double(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_double:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floatuntidf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = tail call double @llvm.experimental.constrained.uitofp.f64.i128(i128 %2, metadata !"round.dynamic", metadata !"fpexcept.strict")
   ret double %3
 }
 
 define float @i128_to_float(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_float:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floattisf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = tail call float @llvm.experimental.constrained.sitofp.f32.i128(i128 %2, metadata !"round.dynamic", metadata !"fpexcept.strict")
   ret float %3
 }
 
 define float @ui128_to_float(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_float:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floatuntisf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = tail call float @llvm.experimental.constrained.uitofp.f32.i128(i128 %2, metadata !"round.dynamic", metadata !"fpexcept.strict")
   ret float %3
 }
 
 define void @i128_to_longdouble(ptr noalias nocapture sret(x86_fp80) align 16 %agg.result, ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_longdouble:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    pushq %rsi
 ; WIN64-NEXT:    subq $64, %rsp
 ; WIN64-NEXT:    movq %rcx, %rsi
-; WIN64-NEXT:    movq (%rdx), %rax
-; WIN64-NEXT:    movq 8(%rdx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rdx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
 ; WIN64-NEXT:    callq __floattixf
 ; WIN64-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    fstpt (%rsi)
 ; WIN64-NEXT:    movq %rsi, %rax
 ; WIN64-NEXT:    addq $64, %rsp
 ; WIN64-NEXT:    popq %rsi
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = tail call x86_fp80 @llvm.experimental.constrained.sitofp.f80.i128(i128 %2, metadata !"round.dynamic", metadata !"fpexcept.strict")
   store x86_fp80 %3, ptr %agg.result, align 16
   ret void
 }
 
 define void @ui128_to_longdouble(ptr noalias nocapture sret(x86_fp80) align 16 %agg.result, ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_longdouble:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    pushq %rsi
 ; WIN64-NEXT:    subq $64, %rsp
 ; WIN64-NEXT:    movq %rcx, %rsi
-; WIN64-NEXT:    movq (%rdx), %rax
-; WIN64-NEXT:    movq 8(%rdx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rdx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
 ; WIN64-NEXT:    callq __floatuntixf
 ; WIN64-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    fstpt (%rsi)
 ; WIN64-NEXT:    movq %rsi, %rax
 ; WIN64-NEXT:    addq $64, %rsp
 ; WIN64-NEXT:    popq %rsi

llvm/test/CodeGen/X86/i128-fpconv-win64.ll

; WIN64-NEXT:    retq
   %4 = trunc i128 %3 to i64
   ret i64 %4
 }
 
 define double @i128_to_double(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_double:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floattidf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = sitofp i128 %2 to double
   ret double %3
 }
 
 define double @ui128_to_double(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_double:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floatuntidf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = uitofp i128 %2 to double
   ret double %3
 }
 
 define float @i128_to_float(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_float:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floattisf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = sitofp i128 %2 to float
   ret float %3
 }
 
 define float @ui128_to_float(ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_float:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    subq $56, %rsp
-; WIN64-NEXT:    movq (%rcx), %rax
-; WIN64-NEXT:    movq 8(%rcx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rcx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    callq __floatuntisf
 ; WIN64-NEXT:    addq $56, %rsp
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = uitofp i128 %2 to float
   ret float %3
 }
 
 define void @i128_to_longdouble(ptr noalias nocapture sret(x86_fp80) align 16 %agg.result, ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: i128_to_longdouble:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    pushq %rsi
 ; WIN64-NEXT:    subq $64, %rsp
 ; WIN64-NEXT:    movq %rcx, %rsi
-; WIN64-NEXT:    movq (%rdx), %rax
-; WIN64-NEXT:    movq 8(%rdx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rdx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
 ; WIN64-NEXT:    callq __floattixf
 ; WIN64-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    fstpt (%rsi)
 ; WIN64-NEXT:    movq %rsi, %rax
 ; WIN64-NEXT:    addq $64, %rsp
 ; WIN64-NEXT:    popq %rsi
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = sitofp i128 %2 to x86_fp80
   store x86_fp80 %3, ptr %agg.result, align 16
   ret void
 }
 
 define void @ui128_to_longdouble(ptr noalias nocapture sret(x86_fp80) align 16 %agg.result, ptr nocapture readonly %0) nounwind {
 ; WIN64-LABEL: ui128_to_longdouble:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    pushq %rsi
 ; WIN64-NEXT:    subq $64, %rsp
 ; WIN64-NEXT:    movq %rcx, %rsi
-; WIN64-NEXT:    movq (%rdx), %rax
-; WIN64-NEXT:    movq 8(%rdx), %rcx
-; WIN64-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; WIN64-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WIN64-NEXT:    movaps (%rdx), %xmm0
+; WIN64-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rcx
 ; WIN64-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
 ; WIN64-NEXT:    callq __floatuntixf
 ; WIN64-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; WIN64-NEXT:    fstpt (%rsi)
 ; WIN64-NEXT:    movq %rsi, %rax
 ; WIN64-NEXT:    addq $64, %rsp
 ; WIN64-NEXT:    popq %rsi
 ; WIN64-NEXT:    retq
   %2 = load i128, ptr %0, align 16
   %3 = uitofp i128 %2 to x86_fp80
   store x86_fp80 %3, ptr %agg.result, align 16
   ret void
 }

llvm/test/CodeGen/X86/legalize-shl-vec.ll

 ; X64-NEXT:    shldq $2, %r9, %rdi
 ; X64-NEXT:    shlq $63, %rsi
 ; X64-NEXT:    shlq $2, %r9
 ; X64-NEXT:    movq %rcx, 56(%rax)
 ; X64-NEXT:    movq %rdx, 48(%rax)
 ; X64-NEXT:    movq %rdi, 40(%rax)
 ; X64-NEXT:    movq %r9, 32(%rax)
 ; X64-NEXT:    movq %rsi, 24(%rax)
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, (%rax)
 ; X64-NEXT:    movq $0, 16(%rax)
-; X64-NEXT:    movq $0, 8(%rax)
-; X64-NEXT:    movq $0, (%rax)
 ; X64-NEXT:    retq
   %Amt = insertelement <2 x i256> <i256 1, i256 2>, i256 255, i32 0
   %Out = shl <2 x i256> %In, %Amt
   ret <2 x i256> %Out
 }
 
 define <2 x i256> @test_srl(<2 x i256> %In) {
 ; X32-LABEL: test_srl:

 ; X64-NEXT:    shrdq $4, %rcx, %rdx
 ; X64-NEXT:    shrq $63, %r8
 ; X64-NEXT:    shrq $4, %rcx
 ; X64-NEXT:    movq %rcx, 56(%rdi)
 ; X64-NEXT:    movq %rdx, 48(%rdi)
 ; X64-NEXT:    movq %rsi, 40(%rdi)
 ; X64-NEXT:    movq %r9, 32(%rdi)
 ; X64-NEXT:    movq %r8, (%rdi)
-; X64-NEXT:    movq $0, 24(%rdi)
-; X64-NEXT:    movq $0, 16(%rdi)
+; X64-NEXT:    xorps %xmm0, %xmm0
+; X64-NEXT:    movaps %xmm0, 16(%rdi)
 ; X64-NEXT:    movq $0, 8(%rdi)
 ; X64-NEXT:    retq
   %Amt = insertelement <2 x i256> <i256 3, i256 4>, i256 255, i32 0
   %Out = lshr <2 x i256> %In, %Amt
   ret <2 x i256> %Out
 }
 
 define <2 x i256> @test_sra(<2 x i256> %In) {

llvm/test/CodeGen/X86/masked_store.ll

 ; X86-AVX512-NEXT:    vmovlps %xmm0, 88(%eax)
 ; X86-AVX512-NEXT:    vzeroupper
 ; X86-AVX512-NEXT:    retl
   call void @llvm.masked.store.v16i64.p0(<16 x i64> %val, ptr %addr, i32 4, <16 x i1><i1 false, i1 false, i1 false, i1 false, i1 false, i1 false, i1 true, i1 false, i1 false, i1 false, i1 false, i1 true, i1 false, i1 false, i1 false, i1 false>)
   ret void
 }
 
 define void @top_bits_unset_stack() {
-; SSE2-LABEL: top_bits_unset_stack:
-; SSE2:       ## %bb.0: ## %entry
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    movq $0, -{{[0-9]+}}(%rsp)
-; SSE2-NEXT:    retq
-;
-; SSE4-LABEL: top_bits_unset_stack:
-; SSE4:       ## %bb.0: ## %entry
-; SSE4-NEXT:    xorps %xmm0, %xmm0
-; SSE4-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; SSE4-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; SSE4-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; SSE4-NEXT:    retq
+; SSE-LABEL: top_bits_unset_stack:
+; SSE:       ## %bb.0: ## %entry
+; SSE-NEXT:    xorps %xmm0, %xmm0
+; SSE-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; SSE-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; SSE-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; SSE-NEXT:    retq
 ;
 ; AVX1OR2-LABEL: top_bits_unset_stack:
 ; AVX1OR2:       ## %bb.0: ## %entry
 ; AVX1OR2-NEXT:    vxorpd %xmm0, %xmm0, %xmm0
 ; AVX1OR2-NEXT:    vmovapd {{.*#+}} ymm1 = [18446744073709551615,18446744073709551615,0,0]
 ; AVX1OR2-NEXT:    vmaskmovpd %ymm0, %ymm1, -{{[0-9]+}}(%rsp)
 ; AVX1OR2-NEXT:    vmovupd %ymm0, -{{[0-9]+}}(%rsp)
 ; AVX1OR2-NEXT:    vzeroupper

llvm/test/CodeGen/X86/memset-zero.ll

 ; NEHALEM-NEXT:    xorps %xmm0, %xmm0
 ; NEHALEM-NEXT:    movaps %xmm0, 16(%rdi)
 ; NEHALEM-NEXT:    movaps %xmm0, (%rdi)
 ; NEHALEM-NEXT:    retq
 ;
 ; SANDYBRIDGE-LABEL: memset_32_align32:
 ; SANDYBRIDGE:       # %bb.0: # %entry
 ; SANDYBRIDGE-NEXT:    vxorps %xmm0, %xmm0, %xmm0
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, 16(%rdi)
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, (%rdi)
+; SANDYBRIDGE-NEXT:    vmovaps %ymm0, (%rdi)
+; SANDYBRIDGE-NEXT:    vzeroupper
 ; SANDYBRIDGE-NEXT:    retq
 ;
 ; SKYLAKE-LABEL: memset_32_align32:
 ; SKYLAKE:       # %bb.0: # %entry
 ; SKYLAKE-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; SKYLAKE-NEXT:    vmovaps %ymm0, (%rdi)
 ; SKYLAKE-NEXT:    vzeroupper
 ; SKYLAKE-NEXT:    retq

 ; NEHALEM-NEXT:    movaps %xmm0, 32(%rdi)
 ; NEHALEM-NEXT:    movaps %xmm0, 16(%rdi)
 ; NEHALEM-NEXT:    movaps %xmm0, (%rdi)
 ; NEHALEM-NEXT:    retq
 ;
 ; SANDYBRIDGE-LABEL: memset_64_align64:
 ; SANDYBRIDGE:       # %bb.0: # %entry
 ; SANDYBRIDGE-NEXT:    vxorps %xmm0, %xmm0, %xmm0
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, 16(%rdi)
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, (%rdi)
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, 48(%rdi)
-; SANDYBRIDGE-NEXT:    vmovaps %xmm0, 32(%rdi)
+; SANDYBRIDGE-NEXT:    vmovaps %ymm0, 32(%rdi)
+; SANDYBRIDGE-NEXT:    vmovaps %ymm0, (%rdi)
+; SANDYBRIDGE-NEXT:    vzeroupper
 ; SANDYBRIDGE-NEXT:    retq
 ;
 ; SKYLAKE-LABEL: memset_64_align64:
 ; SKYLAKE:       # %bb.0: # %entry
 ; SKYLAKE-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; SKYLAKE-NEXT:    vmovaps %ymm0, 32(%rdi)
 ; SKYLAKE-NEXT:    vmovaps %ymm0, (%rdi)
 ; SKYLAKE-NEXT:    vzeroupper

llvm/test/CodeGen/X86/merge-consecutive-loads-128.ll

 ; AVX-LABEL: merge_4i32_i32_combine:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; AVX-NEXT:    vmovaps %xmm0, (%rdi)
 ; AVX-NEXT:    retq
 ;
 ; X86-SSE1-LABEL: merge_4i32_i32_combine:
 ; X86-SSE1:       # %bb.0:
-; X86-SSE1-NEXT:    pushl %ebp
-; X86-SSE1-NEXT:    .cfi_def_cfa_offset 8
-; X86-SSE1-NEXT:    .cfi_offset %ebp, -8
-; X86-SSE1-NEXT:    movl %esp, %ebp
-; X86-SSE1-NEXT:    .cfi_def_cfa_register %ebp
-; X86-SSE1-NEXT:    pushl %edi
-; X86-SSE1-NEXT:    pushl %esi
-; X86-SSE1-NEXT:    andl $-16, %esp
-; X86-SSE1-NEXT:    subl $16, %esp
-; X86-SSE1-NEXT:    .cfi_offset %esi, -16
-; X86-SSE1-NEXT:    .cfi_offset %edi, -12
-; X86-SSE1-NEXT:    movl 8(%ebp), %eax
-; X86-SSE1-NEXT:    movl 12(%ebp), %ecx
+; X86-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-SSE1-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X86-SSE1-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
 ; X86-SSE1-NEXT:    andps %xmm0, %xmm1
-; X86-SSE1-NEXT:    movaps %xmm1, (%esp)
-; X86-SSE1-NEXT:    movl (%esp), %ecx
-; X86-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X86-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-SSE1-NEXT:    movl %edi, 12(%eax)
-; X86-SSE1-NEXT:    movl %esi, 8(%eax)
-; X86-SSE1-NEXT:    movl %edx, 4(%eax)
-; X86-SSE1-NEXT:    movl %ecx, (%eax)
-; X86-SSE1-NEXT:    leal -8(%ebp), %esp
-; X86-SSE1-NEXT:    popl %esi
-; X86-SSE1-NEXT:    popl %edi
-; X86-SSE1-NEXT:    popl %ebp
-; X86-SSE1-NEXT:    .cfi_def_cfa %esp, 4
+; X86-SSE1-NEXT:    movaps %xmm1, (%eax)
 ; X86-SSE1-NEXT:    retl
 ;
 ; X86-SSE41-LABEL: merge_4i32_i32_combine:
 ; X86-SSE41:       # %bb.0:
 ; X86-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-SSE41-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X86-SSE41-NEXT:    movaps %xmm0, (%eax)

llvm/test/CodeGen/X86/merge-consecutive-stores-nt.ll

 ; X64-SSE41-LABEL: merge_2_v4f32_align32:
 ; X64-SSE41:       # %bb.0:
 ; X64-SSE41-NEXT:    movntdqa (%rdi), %xmm0
 ; X64-SSE41-NEXT:    movntdqa 16(%rdi), %xmm1
 ; X64-SSE41-NEXT:    movntdq %xmm0, (%rsi)
 ; X64-SSE41-NEXT:    movntdq %xmm1, 16(%rsi)
 ; X64-SSE41-NEXT:    retq
 ;
-; X64-AVX-LABEL: merge_2_v4f32_align32:
-; X64-AVX:       # %bb.0:
-; X64-AVX-NEXT:    vmovntdqa (%rdi), %xmm0
-; X64-AVX-NEXT:    vmovntdqa 16(%rdi), %xmm1
-; X64-AVX-NEXT:    vmovntdq %xmm0, (%rsi)
-; X64-AVX-NEXT:    vmovntdq %xmm1, 16(%rsi)
-; X64-AVX-NEXT:    retq
+; X64-AVX1-LABEL: merge_2_v4f32_align32:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; X64-AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm1
+; X64-AVX1-NEXT:    vmovntdq %xmm0, (%rsi)
+; X64-AVX1-NEXT:    vmovntdq %xmm1, 16(%rsi)
+; X64-AVX1-NEXT:    retq
+;
+; X64-AVX2-LABEL: merge_2_v4f32_align32:
+; X64-AVX2:       # %bb.0:
+; X64-AVX2-NEXT:    vmovntdqa (%rdi), %ymm0
+; X64-AVX2-NEXT:    vmovntdq %ymm0, (%rsi)
+; X64-AVX2-NEXT:    vzeroupper
+; X64-AVX2-NEXT:    retq
   %1 = getelementptr inbounds <4 x float>, ptr %a0, i64 1, i64 0
   %2 = load <4 x float>, ptr %a0, align 32, !nontemporal !0
   %3 = load <4 x float>, ptr %1, align 16, !nontemporal !0
   %4 = getelementptr inbounds <4 x float>, ptr %a1, i64 1, i64 0
   store <4 x float> %2, ptr %a1, align 32, !nontemporal !0
   store <4 x float> %3, ptr %4, align 16, !nontemporal !0
   ret void
 }
; X64-AVX-NEXT:    retq
   %3 = load <4 x float>, ptr %1, align 1, !nontemporal !0
   %4 = getelementptr inbounds <4 x float>, ptr %a1, i64 1, i64 0
   store <4 x float> %2, ptr %a1, align 1, !nontemporal !0
   store <4 x float> %3, ptr %4, align 1, !nontemporal !0
   ret void
 }
 
 !0 = !{i32 1}
-;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
-; X64-AVX1: {{.*}}
-; X64-AVX2: {{.*}}

llvm/test/CodeGen/X86/nontemporal-loads-2.ll

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v4f64_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <4 x double>, ptr %src, align 16, !nontemporal !1
   ret <4 x double> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v8f32_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <8 x float>, ptr %src, align 16, !nontemporal !1
   ret <8 x float> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v4i64_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <4 x i64>, ptr %src, align 16, !nontemporal !1
   ret <4 x i64> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v8i32_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <8 x i32>, ptr %src, align 16, !nontemporal !1
   ret <8 x i32> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v16i16_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <16 x i16>, ptr %src, align 16, !nontemporal !1
   ret <16 x i16> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: test_v32i8_align16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
 ; AVX-NEXT:    movq %rsp, %rbp
 ; AVX-NEXT:    andq $-32, %rsp
 ; AVX-NEXT:    subq $64, %rsp
-; AVX-NEXT:    movq 24(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq 16(%rdi), %rax
-; AVX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq (%rdi), %rax
-; AVX-NEXT:    movq 8(%rdi), %rcx
-; AVX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    movq %rax, (%rsp)
+; AVX-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX-NEXT:    movq %rbp, %rsp
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
   %1 = load <32 x i8>, ptr %src, align 16, !nontemporal !1
   ret <32 x i8> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v8f64_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v8f64_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8f64_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <8 x double>, ptr %src, align 16, !nontemporal !1
   ret <8 x double> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v16f32_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v16f32_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v16f32_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <16 x float>, ptr %src, align 16, !nontemporal !1
   ret <16 x float> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v8i64_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v8i64_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8i64_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <8 x i64>, ptr %src, align 16, !nontemporal !1
   ret <8 x i64> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v16i32_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v16i32_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v16i32_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <16 x i32>, ptr %src, align 16, !nontemporal !1
   ret <16 x i32> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v32i16_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v32i16_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v32i16_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <32 x i16>, ptr %src, align 16, !nontemporal !1
   ret <32 x i16> %1
 }
 

 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test_v64i8_align16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    pushq %rbp
 ; AVX1-NEXT:    movq %rsp, %rbp
 ; AVX1-NEXT:    andq $-32, %rsp
 ; AVX1-NEXT:    subq $96, %rsp
-; AVX1-NEXT:    movq 24(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 16(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq (%rdi), %rax
-; AVX1-NEXT:    movq 8(%rdi), %rcx
-; AVX1-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq %rax, (%rsp)
-; AVX1-NEXT:    movq 56(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 48(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 40(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX1-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX1-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v64i8_align16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:    movq %rsp, %rbp
 ; AVX2-NEXT:    andq $-32, %rsp
 ; AVX2-NEXT:    subq $96, %rsp
-; AVX2-NEXT:    movq 24(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 16(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq (%rdi), %rax
-; AVX2-NEXT:    movq 8(%rdi), %rcx
-; AVX2-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq %rax, (%rsp)
-; AVX2-NEXT:    movq 56(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 48(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 40(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX2-NEXT:    movq 32(%rdi), %rax
-; AVX2-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsp)
+; AVX2-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX2-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    vmovaps (%rsp), %ymm0
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %ymm1
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v64i8_align16:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 48(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa 16(%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <64 x i8>, ptr %src, align 16, !nontemporal !1
   ret <64 x i8> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8f64_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <8 x double>, ptr %src, align 32, !nontemporal !1
   ret <8 x double> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v16f32_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <16 x float>, ptr %src, align 32, !nontemporal !1
   ret <16 x float> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8i64_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <8 x i64>, ptr %src, align 32, !nontemporal !1
   ret <8 x i64> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v16i32_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <16 x i32>, ptr %src, align 32, !nontemporal !1
   ret <16 x i32> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v32i16_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <32 x i16>, ptr %src, align 32, !nontemporal !1
   ret <32 x i16> %1
 }
 

 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v64i8_align32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    pushq %rbp
 ; AVX512-NEXT:    movq %rsp, %rbp
 ; AVX512-NEXT:    andq $-64, %rsp
 ; AVX512-NEXT:    subq $128, %rsp
-; AVX512-NEXT:    movq 56(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 48(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 40(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 32(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 24(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq 16(%rdi), %rax
-; AVX512-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq (%rdi), %rax
-; AVX512-NEXT:    movq 8(%rdi), %rcx
-; AVX512-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movq %rax, (%rsp)
+; AVX512-NEXT:    vmovntdqa 32(%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
+; AVX512-NEXT:    vmovntdqa (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsp)
 ; AVX512-NEXT:    vmovaps (%rsp), %zmm0
 ; AVX512-NEXT:    movq %rbp, %rsp
 ; AVX512-NEXT:    popq %rbp
 ; AVX512-NEXT:    retq
   %1 = load <64 x i8>, ptr %src, align 32, !nontemporal !1
   ret <64 x i8> %1
 }
 
 !1 = !{i32 1}

llvm/test/CodeGen/X86/recip-fastmath.ll

 ; BTVER2-LABEL: v8f32_no_estimate:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} ymm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BTVER2-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_no_estimate:
 ; SANDY:       # %bb.0:
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_no_estimate:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; HASWELL-NEXT:    retq

 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_one_step:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm1

 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_two_step:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; SANDY-NEXT:    vaddps %ymm2, %ymm1, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq

 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BTVER2-NEXT:    vdivps %ymm0, %ymm2, %ymm0
 ; BTVER2-NEXT:    vdivps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_no_estimate:
 ; SANDY:       # %bb.0:
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vdivps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vdivps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_no_estimate:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NEXT:    vdivps %ymm0, %ymm2, %ymm0

 ; BTVER2-NEXT:    vaddps %ymm0, %ymm2, %ymm0
 ; BTVER2-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_one_step:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    vrcpps %ymm1, %ymm4
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm4, %ymm1, %ymm1
 ; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm4, %ymm1
 ; SANDY-NEXT:    vaddps %ymm1, %ymm4, %ymm1

 ; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_two_step:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm3
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm4, %ymm4
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; SANDY-NEXT:    vaddps %ymm3, %ymm2, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm4, %ymm0
 ; SANDY-NEXT:    vrcpps %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm2, %ymm0

llvm/test/CodeGen/X86/recip-fastmath2.ll

 ; BTVER2-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_one_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step2:

 ; BTVER2-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm1
 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_one_step_2_divs:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
 ; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm1
+; SANDY-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step_2_divs:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NEXT:    vfmsub231ps {{.*#+}} ymm2 = (ymm1 * ymm0) - ymm2

 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; BTVER2-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_two_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; SANDY-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_two_step2:

 ; SSE-LABEL: v8f32_no_step2:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm0
 ; SSE-NEXT:    mulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    mulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; SSE-NEXT:    retq
 ;
-; AVX-RECIP-LABEL: v8f32_no_step2:
-; AVX-RECIP:       # %bb.0:
-; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; AVX-RECIP-NEXT:    retq
-;
-; FMA-RECIP-LABEL: v8f32_no_step2:
-; FMA-RECIP:       # %bb.0:
-; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm0
-; FMA-RECIP-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; FMA-RECIP-NEXT:    retq
-;
-; BDVER2-LABEL: v8f32_no_step2:
-; BDVER2:       # %bb.0:
-; BDVER2-NEXT:    vrcpps %ymm0, %ymm0
-; BDVER2-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; BDVER2-NEXT:    retq
-;
-; BTVER2-LABEL: v8f32_no_step2:
-; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vrcpps %ymm0, %ymm0
-; BTVER2-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; BTVER2-NEXT:    retq
-;
-; SANDY-LABEL: v8f32_no_step2:
-; SANDY:       # %bb.0:
-; SANDY-NEXT:    vrcpps %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm1, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; SANDY-NEXT:    retq
-;
-; HASWELL-LABEL: v8f32_no_step2:
-; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %ymm0, %ymm0
-; HASWELL-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; HASWELL-NEXT:    retq
-;
-; HASWELL-NO-FMA-LABEL: v8f32_no_step2:
-; HASWELL-NO-FMA:       # %bb.0:
-; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    retq
-;
-; AVX512-LABEL: v8f32_no_step2:
-; AVX512:       # %bb.0:
-; AVX512-NEXT:    vrcpps %ymm0, %ymm0
-; AVX512-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
-; AVX512-NEXT:    retq
+; AVX-LABEL: v8f32_no_step2:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vrcpps %ymm0, %ymm0
+; AVX-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
+; AVX-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, %x
   ret <8 x float> %div
 }
 
 define <16 x float> @v16f32_one_step2(<16 x float> %x) #1 {
 ; SSE-LABEL: v16f32_one_step2:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movaps %xmm1, %xmm4

 ; BTVER2-NEXT:    vsubps %ymm1, %ymm3, %ymm1
 ; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    vaddps %ymm1, %ymm5, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_one_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm4
 ; SANDY-NEXT:    vmulps %ymm4, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
-; SANDY-NEXT:    vrcpps %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
 ; SANDY-NEXT:    vaddps %ymm0, %ymm4, %ymm0
-; SANDY-NEXT:    vmulps %ymm2, %ymm3, %ymm4
+; SANDY-NEXT:    vrcpps %ymm1, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm4
 ; SANDY-NEXT:    vmulps %ymm4, %ymm1, %ymm1
-; SANDY-NEXT:    vsubps %ymm1, %ymm2, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm3, %ymm1
+; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_one_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
 ; HASWELL-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; HASWELL-NEXT:    vmulps %ymm3, %ymm2, %ymm4

 ; BTVER2-NEXT:    vmulps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_one_step_2_divs:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vrcpps %ymm1, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm1
 ; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    vaddps %ymm1, %ymm2, %ymm1
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
-; SANDY-NEXT:    vmulps %ymm3, %ymm0, %ymm3
+; SANDY-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm2
+; SANDY-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_one_step_2_divs:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]

 ; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_two_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm3
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm4, %ymm4
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; SANDY-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm5
 ; SANDY-NEXT:    vmulps %ymm5, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
-; SANDY-NEXT:    vrcpps %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm5, %ymm0
-; SANDY-NEXT:    vmulps %ymm3, %ymm1, %ymm2
-; SANDY-NEXT:    vsubps %ymm2, %ymm4, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm3, %ymm2
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm4 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm4, %ymm4
-; SANDY-NEXT:    vaddps %ymm2, %ymm3, %ymm2
-; SANDY-NEXT:    vmulps %ymm4, %ymm2, %ymm3
-; SANDY-NEXT:    vmulps %ymm3, %ymm1, %ymm1
-; SANDY-NEXT:    vsubps %ymm1, %ymm4, %ymm1
+; SANDY-NEXT:    vrcpps %ymm1, %ymm2
+; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; SANDY-NEXT:    vsubps %ymm3, %ymm4, %ymm3
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm3
+; SANDY-NEXT:    vaddps %ymm3, %ymm2, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; SANDY-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; SANDY-NEXT:    vaddps %ymm1, %ymm3, %ymm1
+; SANDY-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_two_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NEXT:    vmovaps %ymm2, %ymm4
 ; HASWELL-NEXT:    vfmsub213ps {{.*#+}} ymm4 = (ymm0 * ymm4) - ymm3

 ; BTVER2-NEXT:    vrcpps %ymm1, %ymm1
 ; BTVER2-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; BTVER2-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_no_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SANDY-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; SANDY-NEXT:    vrcpps %ymm1, %ymm1
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
-; SANDY-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
-; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; SANDY-NEXT:    vmulps %ymm3, %ymm1, %ymm1
+; SANDY-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_no_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm0
 ; HASWELL-NEXT:    vrcpps %ymm1, %ymm1
 ; HASWELL-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; HASWELL-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm1

llvm/test/CodeGen/X86/sandybridge-loads.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=corei7-avx | FileCheck %s
 
 define void @wideloads(ptr %a, ptr %b, ptr %c) nounwind {
 ; CHECK-LABEL: wideloads:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmovaps (%rdi), %xmm0
 ; CHECK-NEXT:    vinsertf128 $1, 16(%rdi), %ymm0, %ymm0
-; CHECK-NEXT:    vmovaps (%rsi), %xmm1
-; CHECK-NEXT:    vinsertf128 $1, 16(%rsi), %ymm1, %ymm1
+; CHECK-NEXT:    vmovaps (%rsi), %ymm1
 ; CHECK-NEXT:    vcmpltps %ymm0, %ymm1, %ymm1
-; CHECK-NEXT:    vmovaps (%rdx), %xmm2
-; CHECK-NEXT:    vinsertf128 $1, 16(%rdx), %ymm2, %ymm2
+; CHECK-NEXT:    vmovaps (%rdx), %ymm2
 ; CHECK-NEXT:    vcmpltps %ymm0, %ymm2, %ymm0
-; CHECK-NEXT:    vmovaps {{.*#+}} xmm2 = [1,1,1,1,1,1,1,1]
-; CHECK-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
 ; CHECK-NEXT:    vandps %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vandps %ymm2, %ymm0, %ymm0
-; CHECK-NEXT:    vextractf128 $1, %ymm0, (%rax)
-; CHECK-NEXT:    vmovaps %xmm0, (%rax)
+; CHECK-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
+; CHECK-NEXT:    vmovaps %ymm0, (%rax)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
   %v0 = load <8 x float>, ptr %a, align 16  ; <---- unaligned!
   %v1 = load <8 x float>, ptr %b, align 32  ; <---- aligned!
   %m0 = fcmp olt <8 x float> %v1, %v0
   %v2 = load <8 x float>, ptr %c, align 32  ; <---- aligned!
   %m1 = fcmp olt <8 x float> %v2, %v0
   %mand = and <8 x i1> %m1, %m0
   %r = zext <8 x i1> %mand to <8 x i32>
   store <8 x i32> %r, ptr undef, align 32
   ret void
 }
 
 define void @widestores(ptr %a, ptr %b, ptr %c) nounwind {
 ; CHECK-LABEL: widestores:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovaps (%rdi), %xmm0
-; CHECK-NEXT:    vmovaps 16(%rdi), %xmm1
-; CHECK-NEXT:    vmovaps (%rsi), %xmm2
-; CHECK-NEXT:    vmovaps 16(%rsi), %xmm3
-; CHECK-NEXT:    vmovaps %xmm1, 16(%rsi)
-; CHECK-NEXT:    vmovaps %xmm0, (%rsi)
-; CHECK-NEXT:    vmovaps %xmm2, (%rdi)
-; CHECK-NEXT:    vmovaps %xmm3, 16(%rdi)
+; CHECK-NEXT:    vmovaps (%rdi), %ymm0
+; CHECK-NEXT:    vmovaps (%rsi), %ymm1
+; CHECK-NEXT:    vmovaps %ymm0, (%rsi)
+; CHECK-NEXT:    vextractf128 $1, %ymm1, 16(%rdi)
+; CHECK-NEXT:    vmovaps %xmm1, (%rdi)
+; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
   %v0 = load <8 x float>, ptr %a, align 32
   %v1 = load <8 x float>, ptr %b, align 32
   store <8 x float> %v0, ptr %b, align 32 ; <--- aligned
   store <8 x float> %v1, ptr %a, align 16 ; <--- unaligned
   ret void
 }
 
 define void @widestores_unaligned_load(ptr %a, ptr %b, ptr %c) nounwind {
 ; CHECK-LABEL: widestores_unaligned_load:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovaps (%rdi), %xmm0
-; CHECK-NEXT:    vmovaps 16(%rdi), %xmm1
-; CHECK-NEXT:    vmovaps (%rsi), %xmm2
-; CHECK-NEXT:    vmovaps 16(%rsi), %xmm3
-; CHECK-NEXT:    vmovaps %xmm1, 16(%rsi)
-; CHECK-NEXT:    vmovaps %xmm0, (%rsi)
-; CHECK-NEXT:    vmovaps %xmm2, (%rdi)
-; CHECK-NEXT:    vmovaps %xmm3, 16(%rdi)
+; CHECK-NEXT:    vmovaps (%rdi), %ymm0
+; CHECK-NEXT:    vmovaps (%rsi), %xmm1
+; CHECK-NEXT:    vmovaps 16(%rsi), %xmm2
+; CHECK-NEXT:    vmovaps %ymm0, (%rsi)
+; CHECK-NEXT:    vmovaps %xmm2, 16(%rdi)
+; CHECK-NEXT:    vmovaps %xmm1, (%rdi)
+; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
   %v0 = load <8 x float>, ptr %a, align 32 ; <--- aligned
   %v1 = load <8 x float>, ptr %b, align 16 ; <--- unaligned
   store <8 x float> %v0, ptr %b, align 32 ; <--- aligned
   store <8 x float> %v1, ptr %a, align 16 ; <--- unaligned
   ret void
 }

llvm/test/CodeGen/X86/shift-i128.ll

 ; i686-NEXT:    movl $0, 12(%eax)
 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl
 ;
 ; x86_64-LABEL: test_lshr_v2i128_outofrange_sum:
 ; x86_64:       # %bb.0: # %entry
-; x86_64-NEXT:    movq $0, 24(%r8)
-; x86_64-NEXT:    movq $0, 16(%r8)
-; x86_64-NEXT:    movq $0, 8(%r8)
-; x86_64-NEXT:    movq $0, (%r8)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 16(%r8)
+; x86_64-NEXT:    movaps %xmm0, (%r8)
 ; x86_64-NEXT:    retq
 entry:
   %0 = lshr <2 x i128> %x, <i128 -1, i128 -1>
   %1 = lshr <2 x i128> %0, <i128  1, i128  1>
   store <2 x i128> %1, ptr %r, align 16
   ret void
 }
 
 define void @test_ashr_v2i128_outofrange_sum(<2 x i128> %x, ptr nocapture %r) nounwind {
 ; i686-LABEL: test_ashr_v2i128_outofrange_sum:
 ; i686:       # %bb.0: # %entry
 ; i686-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; i686-NEXT:    movl $0, 28(%eax)
 ; i686-NEXT:    movl $0, 24(%eax)
 ; i686-NEXT:    movl $0, 20(%eax)
 ; i686-NEXT:    movl $0, 16(%eax)
 ; i686-NEXT:    movl $0, 12(%eax)
 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl
 ;
 ; x86_64-LABEL: test_ashr_v2i128_outofrange_sum:
 ; x86_64:       # %bb.0: # %entry
-; x86_64-NEXT:    movq $0, 24(%r8)
-; x86_64-NEXT:    movq $0, 16(%r8)
-; x86_64-NEXT:    movq $0, 8(%r8)
-; x86_64-NEXT:    movq $0, (%r8)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 16(%r8)
+; x86_64-NEXT:    movaps %xmm0, (%r8)
 ; x86_64-NEXT:    retq
 entry:
   %0 = ashr <2 x i128> %x, <i128 -1, i128 -1>
   %1 = ashr <2 x i128> %0, <i128  1, i128  1>
   store <2 x i128> %1, ptr %r, align 16
   ret void
 }
 
 define void @test_shl_v2i128_outofrange_sum(<2 x i128> %x, ptr nocapture %r) nounwind {
 ; i686-LABEL: test_shl_v2i128_outofrange_sum:
 ; i686:       # %bb.0: # %entry
 ; i686-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; i686-NEXT:    movl $0, 28(%eax)
 ; i686-NEXT:    movl $0, 24(%eax)
 ; i686-NEXT:    movl $0, 20(%eax)
 ; i686-NEXT:    movl $0, 16(%eax)
 ; i686-NEXT:    movl $0, 12(%eax)
 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl
 ;
 ; x86_64-LABEL: test_shl_v2i128_outofrange_sum:
 ; x86_64:       # %bb.0: # %entry
-; x86_64-NEXT:    movq $0, 24(%r8)
-; x86_64-NEXT:    movq $0, 16(%r8)
-; x86_64-NEXT:    movq $0, 8(%r8)
-; x86_64-NEXT:    movq $0, (%r8)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 16(%r8)
+; x86_64-NEXT:    movaps %xmm0, (%r8)
 ; x86_64-NEXT:    retq
 entry:
   %0 = shl <2 x i128> %x, <i128 -1, i128 -1>
   %1 = shl <2 x i128> %0, <i128  1, i128  1>
   store <2 x i128> %1, ptr %r, align 16
   ret void
 }
 

 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl $4
 ;
 ; x86_64-LABEL: shl_sext_shl_outofrange:
 ; x86_64:       # %bb.0:
 ; x86_64-NEXT:    movq %rdi, %rax
-; x86_64-NEXT:    movq $0, 56(%rdi)
-; x86_64-NEXT:    movq $0, 48(%rdi)
-; x86_64-NEXT:    movq $0, 40(%rdi)
-; x86_64-NEXT:    movq $0, 32(%rdi)
-; x86_64-NEXT:    movq $0, 24(%rdi)
-; x86_64-NEXT:    movq $0, 16(%rdi)
-; x86_64-NEXT:    movq $0, 8(%rdi)
-; x86_64-NEXT:    movq $0, (%rdi)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 48(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 32(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 16(%rdi)
+; x86_64-NEXT:    movaps %xmm0, (%rdi)
 ; x86_64-NEXT:    retq
   %1 = shl <2 x i128> %a0, <i128 -1, i128 -1>
   %2 = sext <2 x i128> %1 to <2 x i256>
   %3 = shl <2 x i256> %2, <i256 128, i256 128>
   ret <2 x i256> %3
 }
 
 define <2 x i256> @shl_zext_shl_outofrange(<2 x i128> %a0) {

 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl $4
 ;
 ; x86_64-LABEL: shl_zext_shl_outofrange:
 ; x86_64:       # %bb.0:
 ; x86_64-NEXT:    movq %rdi, %rax
-; x86_64-NEXT:    movq $0, 56(%rdi)
-; x86_64-NEXT:    movq $0, 48(%rdi)
-; x86_64-NEXT:    movq $0, 40(%rdi)
-; x86_64-NEXT:    movq $0, 32(%rdi)
-; x86_64-NEXT:    movq $0, 24(%rdi)
-; x86_64-NEXT:    movq $0, 16(%rdi)
-; x86_64-NEXT:    movq $0, 8(%rdi)
-; x86_64-NEXT:    movq $0, (%rdi)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 48(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 32(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 16(%rdi)
+; x86_64-NEXT:    movaps %xmm0, (%rdi)
 ; x86_64-NEXT:    retq
   %1 = shl <2 x i128> %a0, <i128 -1, i128 -1>
   %2 = zext <2 x i128> %1 to <2 x i256>
   %3 = shl <2 x i256> %2, <i256 128, i256 128>
   ret <2 x i256> %3
 }
 
 define <2 x i256> @shl_zext_lshr_outofrange(<2 x i128> %a0) {

 ; i686-NEXT:    movl $0, 8(%eax)
 ; i686-NEXT:    movl $0, 4(%eax)
 ; i686-NEXT:    movl $0, (%eax)
 ; i686-NEXT:    retl $4
 ;
 ; x86_64-LABEL: shl_zext_lshr_outofrange:
 ; x86_64:       # %bb.0:
 ; x86_64-NEXT:    movq %rdi, %rax
-; x86_64-NEXT:    movq $0, 56(%rdi)
-; x86_64-NEXT:    movq $0, 48(%rdi)
-; x86_64-NEXT:    movq $0, 40(%rdi)
-; x86_64-NEXT:    movq $0, 32(%rdi)
-; x86_64-NEXT:    movq $0, 24(%rdi)
-; x86_64-NEXT:    movq $0, 16(%rdi)
-; x86_64-NEXT:    movq $0, 8(%rdi)
-; x86_64-NEXT:    movq $0, (%rdi)
+; x86_64-NEXT:    xorps %xmm0, %xmm0
+; x86_64-NEXT:    movaps %xmm0, 48(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 32(%rdi)
+; x86_64-NEXT:    movaps %xmm0, 16(%rdi)
+; x86_64-NEXT:    movaps %xmm0, (%rdi)
 ; x86_64-NEXT:    retq
   %1 = lshr <2 x i128> %a0, <i128 -1, i128 -1>
   %2 = zext <2 x i128> %1 to <2 x i256>
   %3 = shl <2 x i256> %2, <i256 128, i256 128>
   ret <2 x i256> %3
 }
 
 define i128 @lshr_shl_mask(i128 %a0) {

llvm/test/CodeGen/X86/single_elt_vector_memory_operation.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+sse2 | FileCheck %s --check-prefixes=SSE,SSE2,FALLBACK0
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+sse4.2 | FileCheck %s --check-prefixes=SSE,SSE42,FALLBACK1
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx  | FileCheck %s --check-prefixes=AVX,AVX1-ONLY,FALLBACK2
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx2 | FileCheck %s --check-prefixes=AVX2,AVX2-SLOW,FALLBACK3
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx2,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX2,AVX2-FAST-PERLANE,FALLBACK4
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx2,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX2,AVX2-FAST,FALLBACK5
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512vl | FileCheck %s --check-prefixes=AVX512F,AVX512F-SLOW,FALLBACK6
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx512vl,+fast-variable-crosslane-shuffle,+fast-variable-perlane-shuffle | FileCheck %s --check-prefixes=AVX512F,AVX512F-FAST,FALLBACK7
 
 define void @load_single_128bit_elt_vector(ptr %in, ptr %off, ptr %out) nounwind {
 ; SSE2-LABEL: load_single_128bit_elt_vector:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa (%rdi), %xmm0
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[2,3,2,3]
 ; SSE2-NEXT:    movq %xmm0, (%rdx)
+; SSE2-NEXT:    pxor %xmm0, %xmm0
+; SSE2-NEXT:    movdqa %xmm0, 16(%rdx)
 ; SSE2-NEXT:    movq %xmm1, 8(%rdx)
-; SSE2-NEXT:    movq $0, 24(%rdx)
-; SSE2-NEXT:    movq $0, 16(%rdx)
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: load_single_128bit_elt_vector:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa (%rdi), %xmm0
 ; SSE42-NEXT:    pextrq $1, %xmm0, 8(%rdx)
 ; SSE42-NEXT:    movq %xmm0, (%rdx)
 ; SSE42-NEXT:    pxor %xmm0, %xmm0

 
 define void @load_single_256bit_elt_vector(ptr %in, ptr %off, ptr %out) nounwind {
 ; SSE2-LABEL: load_single_256bit_elt_vector:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa (%rdi), %xmm0
 ; SSE2-NEXT:    movdqa 16(%rdi), %xmm1
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[2,3,2,3]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,2,3]
+; SSE2-NEXT:    xorps %xmm4, %xmm4
+; SSE2-NEXT:    movaps %xmm4, 48(%rdx)
+; SSE2-NEXT:    movaps %xmm4, 32(%rdx)
 ; SSE2-NEXT:    movq %xmm1, 16(%rdx)
 ; SSE2-NEXT:    movq %xmm0, (%rdx)
 ; SSE2-NEXT:    movq %xmm3, 24(%rdx)
 ; SSE2-NEXT:    movq %xmm2, 8(%rdx)
-; SSE2-NEXT:    movq $0, 56(%rdx)
-; SSE2-NEXT:    movq $0, 48(%rdx)
-; SSE2-NEXT:    movq $0, 40(%rdx)
-; SSE2-NEXT:    movq $0, 32(%rdx)
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: load_single_256bit_elt_vector:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa (%rdi), %xmm0
 ; SSE42-NEXT:    movdqa 16(%rdi), %xmm1
 ; SSE42-NEXT:    xorps %xmm2, %xmm2
 ; SSE42-NEXT:    movaps %xmm2, 48(%rdx)

 ; SSE2-NEXT:    movdqa (%rdi), %xmm0
 ; SSE2-NEXT:    movdqa 16(%rdi), %xmm1
 ; SSE2-NEXT:    movdqa 32(%rdi), %xmm2
 ; SSE2-NEXT:    movdqa 48(%rdi), %xmm3
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm4 = xmm0[2,3,2,3]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm1[2,3,2,3]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm6 = xmm2[2,3,2,3]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm7 = xmm3[2,3,2,3]
+; SSE2-NEXT:    xorps %xmm8, %xmm8
+; SSE2-NEXT:    movaps %xmm8, 112(%rdx)
+; SSE2-NEXT:    movaps %xmm8, 96(%rdx)
+; SSE2-NEXT:    movaps %xmm8, 80(%rdx)
+; SSE2-NEXT:    movaps %xmm8, 64(%rdx)
 ; SSE2-NEXT:    movq %xmm3, 48(%rdx)
 ; SSE2-NEXT:    movq %xmm2, 32(%rdx)
 ; SSE2-NEXT:    movq %xmm1, 16(%rdx)
 ; SSE2-NEXT:    movq %xmm0, (%rdx)
 ; SSE2-NEXT:    movq %xmm7, 56(%rdx)
 ; SSE2-NEXT:    movq %xmm6, 40(%rdx)
 ; SSE2-NEXT:    movq %xmm5, 24(%rdx)
 ; SSE2-NEXT:    movq %xmm4, 8(%rdx)
-; SSE2-NEXT:    movq $0, 112(%rdx)
-; SSE2-NEXT:    movq $0, 120(%rdx)
-; SSE2-NEXT:    movq $0, 96(%rdx)
-; SSE2-NEXT:    movq $0, 104(%rdx)
-; SSE2-NEXT:    movq $0, 80(%rdx)
-; SSE2-NEXT:    movq $0, 88(%rdx)
-; SSE2-NEXT:    movq $0, 64(%rdx)
-; SSE2-NEXT:    movq $0, 72(%rdx)
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: load_single_512bit_elt_vector:
 ; SSE42:       # %bb.0:
 ; SSE42-NEXT:    movdqa (%rdi), %xmm0
 ; SSE42-NEXT:    movdqa 16(%rdi), %xmm1
 ; SSE42-NEXT:    movdqa 32(%rdi), %xmm2
 ; SSE42-NEXT:    movdqa 48(%rdi), %xmm3

llvm/test/CodeGen/X86/sqrt-fastmath-tune.ll

 ; NHM-NEXT:    movaps %xmm3, %xmm0
 ; NHM-NEXT:    movaps %xmm2, %xmm1
 ; NHM-NEXT:    retq
 ;
 ; SNB-LABEL: v8f32_no_daz:
 ; SNB:       # %bb.0:
 ; SNB-NEXT:    vrsqrtps %ymm0, %ymm1
 ; SNB-NEXT:    vmulps %ymm1, %ymm0, %ymm2
-; SNB-NEXT:    vmovaps {{.*#+}} xmm3 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
-; SNB-NEXT:    vmulps %ymm3, %ymm2, %ymm3
+; SNB-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm2, %ymm3
 ; SNB-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; SNB-NEXT:    vmovaps {{.*#+}} xmm2 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
-; SNB-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; SNB-NEXT:    vmovaps {{.*#+}} xmm2 = [NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
+; SNB-NEXT:    vaddps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm1
 ; SNB-NEXT:    vmulps %ymm1, %ymm3, %ymm1
-; SNB-NEXT:    vmovaps {{.*#+}} xmm3 = [1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
-; SNB-NEXT:    vandps %ymm2, %ymm0, %ymm0
-; SNB-NEXT:    vcmpleps %ymm0, %ymm3, %ymm0
+; SNB-NEXT:    vandps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
+; SNB-NEXT:    vmovaps {{.*#+}} ymm2 = [1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38,1.17549435E-38]
+; SNB-NEXT:    vcmpleps %ymm0, %ymm2, %ymm0
 ; SNB-NEXT:    vandps %ymm1, %ymm0, %ymm0
 ; SNB-NEXT:    retq
 ;
 ; BDW-LABEL: v8f32_no_daz:
 ; BDW:       # %bb.0:
 ; BDW-NEXT:    vrsqrtps %ymm0, %ymm1
 ; BDW-NEXT:    vmulps %ymm1, %ymm0, %ymm2
 ; BDW-NEXT:    vbroadcastss {{.*#+}} ymm3 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]

 ; NHM-NEXT:    mulps %xmm4, %xmm6
 ; NHM-NEXT:    cmpneqps %xmm5, %xmm1
 ; NHM-NEXT:    andps %xmm6, %xmm1
 ; NHM-NEXT:    retq
 ;
 ; SNB-LABEL: v8f32_daz:
 ; SNB:       # %bb.0:
 ; SNB-NEXT:    vrsqrtps %ymm0, %ymm1
-; SNB-NEXT:    vmovaps {{.*#+}} xmm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm2, %ymm2
-; SNB-NEXT:    vmulps %ymm1, %ymm0, %ymm3
-; SNB-NEXT:    vmulps %ymm2, %ymm3, %ymm2
-; SNB-NEXT:    vmulps %ymm1, %ymm3, %ymm1
-; SNB-NEXT:    vmovaps {{.*#+}} xmm3 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]
-; SNB-NEXT:    vinsertf128 $1, {{\.?LCPI[0-9]+_[0-9]+}}+16(%rip), %ymm3, %ymm3
-; SNB-NEXT:    vaddps %ymm3, %ymm1, %ymm1
+; SNB-NEXT:    vmulps %ymm1, %ymm0, %ymm2
+; SNB-NEXT:    vmulps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm2, %ymm3
 ; SNB-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; SNB-NEXT:    vaddps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm1, %ymm1
+; SNB-NEXT:    vmulps %ymm1, %ymm3, %ymm1
 ; SNB-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; SNB-NEXT:    vcmpneqps %ymm2, %ymm0, %ymm0
 ; SNB-NEXT:    vandps %ymm1, %ymm0, %ymm0
 ; SNB-NEXT:    retq
 ;
 ; BDW-LABEL: v8f32_daz:
 ; BDW:       # %bb.0:
 ; BDW-NEXT:    vrsqrtps %ymm0, %ymm1

llvm/test/CodeGen/X86/sse1.ll

 }
 
 ; Don't crash trying to do the impossible: an integer vector comparison doesn't exist, so we must scalarize.
 ; https://llvm.org/bugs/show_bug.cgi?id=30512
 
 define <4 x i32> @PR30512(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; X86-LABEL: PR30512:
 ; X86:       # %bb.0:
-; X86-NEXT:    pushl %ebp
-; X86-NEXT:    movl %esp, %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:    andl $-16, %esp
-; X86-NEXT:    subl $48, %esp
-; X86-NEXT:    movl 8(%ebp), %eax
-; X86-NEXT:    movl 12(%ebp), %ecx
-; X86-NEXT:    movl 16(%ebp), %edx
-; X86-NEXT:    movl 24(%ebp), %esi
+; X86-NEXT:    subl $16, %esp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X86-NEXT:    xorl %ebx, %ebx
-; X86-NEXT:    cmpl 40(%ebp), %esi
-; X86-NEXT:    movl 20(%ebp), %esi
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %edi
 ; X86-NEXT:    sete %bl
 ; X86-NEXT:    negl %ebx
 ; X86-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X86-NEXT:    xorl %ebx, %ebx
-; X86-NEXT:    cmpl 36(%ebp), %esi
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    sete %bl
 ; X86-NEXT:    negl %ebx
 ; X86-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X86-NEXT:    xorl %ebx, %ebx
-; X86-NEXT:    cmpl 32(%ebp), %edx
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    sete %bl
 ; X86-NEXT:    negl %ebx
 ; X86-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X86-NEXT:    xorl %edx, %edx
-; X86-NEXT:    cmpl 28(%ebp), %ecx
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    sete %dl
 ; X86-NEXT:    negl %edx
 ; X86-NEXT:    movl %edx, (%esp)
 ; X86-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X86-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
 ; X86-NEXT:    unpcklps {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
 ; X86-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X86-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; X86-NEXT:    unpcklps {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
 ; X86-NEXT:    movlhps {{.*#+}} xmm2 = xmm2[0],xmm1[0]
 ; X86-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}, %xmm2
-; X86-NEXT:    movaps %xmm2, {{[0-9]+}}(%esp)
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl %edi, 12(%eax)
-; X86-NEXT:    movl %esi, 8(%eax)
-; X86-NEXT:    movl %edx, 4(%eax)
-; X86-NEXT:    movl %ecx, (%eax)
-; X86-NEXT:    leal -12(%ebp), %esp
+; X86-NEXT:    movaps %xmm2, (%eax)
+; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
-; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: PR30512:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    xorl %edi, %edi
 ; X64-NEXT:    cmpl {{[0-9]+}}(%rsp), %r8d
 ; X64-NEXT:    sete %dil

 ; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X64-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
 ; X64-NEXT:    unpcklps {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
 ; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X64-NEXT:    movss {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; X64-NEXT:    unpcklps {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
 ; X64-NEXT:    movlhps {{.*#+}} xmm2 = xmm2[0],xmm1[0]
 ; X64-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm2
-; X64-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
-; X64-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; X64-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; X64-NEXT:    movl %edx, 8(%rax)
-; X64-NEXT:    movl %ecx, (%rax)
-; X64-NEXT:    shrq $32, %rcx
-; X64-NEXT:    shrq $32, %rdx
-; X64-NEXT:    movl %edx, 12(%rax)
-; X64-NEXT:    movl %ecx, 4(%rax)
+; X64-NEXT:    movaps %xmm2, (%rax)
 ; X64-NEXT:    retq
   %cmp = icmp eq <4 x i32> %x, %y
   %zext = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %zext
 }
 
 ; Fragile test warning - we need to induce the generation of a vselect
 ; post-legalization to cause the crash seen in:

llvm/test/CodeGen/X86/unaligned-32-byte-memops.ll

; AVX2-NEXT:    retq
   ret <8 x float> %v3
 }
 
 ; If the first load is 32-byte aligned, then the loads should be merged in all cases.
 
 define <8 x float> @combine_16_byte_loads_aligned(ptr %ptr) {
 ; AVXSLOW-LABEL: combine_16_byte_loads_aligned:
 ; AVXSLOW:       # %bb.0:
-; AVXSLOW-NEXT:    vmovaps 48(%rdi), %xmm0
-; AVXSLOW-NEXT:    vinsertf128 $1, 64(%rdi), %ymm0, %ymm0
+; AVXSLOW-NEXT:    vmovaps 48(%rdi), %ymm0
 ; AVXSLOW-NEXT:    retq
 ;
 ; AVXFAST-LABEL: combine_16_byte_loads_aligned:
 ; AVXFAST:       # %bb.0:
 ; AVXFAST-NEXT:    vmovaps 48(%rdi), %ymm0
 ; AVXFAST-NEXT:    retq
 ;
 ; AVX2-LABEL: combine_16_byte_loads_aligned:

llvm/test/CodeGen/X86/unaligned-load.ll

 return:                                           ; No predecessors!
   ret void
 }
 
 define void @func_aligned() nounwind ssp {
 ; I386-LABEL: func_aligned:
 ; I386:       ## %bb.0: ## %entry
 ; I386-NEXT:    subl $44, %esp
+; I386-NEXT:    movaps {{.*#+}} xmm0 = [1498564676,1313821779,1380982853,1095911247]
 ; I386-NEXT:    .p2align 4, 0x90
 ; I386-NEXT:  LBB1_1: ## %bb
 ; I386-NEXT:    ## =>This Inner Loop Header: Depth=1
+; I386-NEXT:    movaps %xmm0, (%esp)
 ; I386-NEXT:    movl $4673097, {{[0-9]+}}(%esp) ## imm = 0x474E49
 ; I386-NEXT:    movl $1230132307, {{[0-9]+}}(%esp) ## imm = 0x49525453
 ; I386-NEXT:    movl $541347367, {{[0-9]+}}(%esp) ## imm = 0x20444E27
 ; I386-NEXT:    movl $840969293, {{[0-9]+}}(%esp) ## imm = 0x32202C4D
-; I386-NEXT:    movl $1095911247, {{[0-9]+}}(%esp) ## imm = 0x4152474F
-; I386-NEXT:    movl $1380982853, {{[0-9]+}}(%esp) ## imm = 0x52502045
-; I386-NEXT:    movl $1313821779, {{[0-9]+}}(%esp) ## imm = 0x4E4F5453
-; I386-NEXT:    movl $1498564676, (%esp) ## imm = 0x59524844
 ; I386-NEXT:    jmp LBB1_1
 ;
 ; CORE2-LABEL: func_aligned:
 ; CORE2:       ## %bb.0: ## %entry
 ; CORE2-NEXT:    movabsq $20070800167293728, %rax ## imm = 0x474E4952545320
 ; CORE2-NEXT:    movabsq $2325069237881678925, %rcx ## imm = 0x20444E2732202C4D
 ; CORE2-NEXT:    movabsq $4706902966564560965, %rdx ## imm = 0x4152474F52502045
 ; CORE2-NEXT:    movabsq $5642821575076104260, %rsi ## imm = 0x4E4F545359524844

llvm/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

 ; CHECK-SSE1-LABEL: out_constant_varx_mone:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [NaN,NaN,NaN,NaN]
 ; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andps (%rsi), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_mone:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-SSE2-NEXT:    pxor %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    pand (%rdi), %xmm0

 
 define <4 x i32> @in_constant_varx_mone(ptr%px, ptr%py, ptr%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_varx_mone:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm0
 ; CHECK-SSE1-NEXT:    andnps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    xorps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_mone:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movdqa (%rdi), %xmm0
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-SSE2-NEXT:    pandn (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm0

 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_mone_invmask(ptr%px, ptr%py, ptr%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm0
 ; CHECK-SSE1-NEXT:    orps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
 ; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;

 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm0
 ; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [NaN,NaN,NaN,NaN]
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm2
 ; CHECK-SSE1-NEXT:    xorps %xmm1, %xmm2
 ; CHECK-SSE1-NEXT:    andnps %xmm2, %xmm0
 ; CHECK-SSE1-NEXT:    xorps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_mone_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movdqa (%rdi), %xmm0
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm2
 ; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm2

 ; CHECK-SSE1-LABEL: out_constant_varx_42:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_42:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

 ; CHECK-SSE1-LABEL: in_constant_varx_42:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_42:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

 ; CHECK-SSE1-LABEL: out_constant_varx_42_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_42_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

 ; CHECK-SSE1-LABEL: in_constant_varx_42_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_42_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

 }
 
 define <4 x i32> @out_constant_mone_vary(ptr%px, ptr%py, ptr%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm0
 ; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;

 }
 
 define <4 x i32> @in_constant_mone_vary(ptr%px, ptr%py, ptr%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    orps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    orps (%rsi), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;

 ; CHECK-SSE1-LABEL: out_constant_mone_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [NaN,NaN,NaN,NaN]
 ; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_mone_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-SSE2-NEXT:    pxor %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    pand (%rsi), %xmm0

 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_mone_vary_invmask(ptr%px, ptr%py, ptr%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    xorps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm0, %xmm0
 ; CHECK-SSE2-NEXT:    pxor (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    por (%rsi), %xmm0
 ; CHECK-SSE2-NEXT:    retq

 ; CHECK-SSE1-LABEL: out_constant_42_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [5.88545355E-44,5.88545355E-44,5.88545355E-44,5.88545355E-44]
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_42_vary:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps {{.*#+}} xmm1 = [42,42,42,42]
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0

 ; CHECK-SSE1-LABEL: in_constant_42_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
 ; CHECK-SSE1-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_42_vary:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm1
 ; CHECK-SSE2-NEXT:    andps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

 ; CHECK-SSE1-LABEL: out_constant_42_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; CHECK-SSE1-NEXT:    andps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_42_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm1
 ; CHECK-SSE2-NEXT:    andps (%rsi), %xmm0

 ; CHECK-SSE1-LABEL: in_constant_42_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm1
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_42_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0

llvm/test/CodeGen/X86/unfold-masked-merge-vector-variablemask.ll

 ; CHECK-SSE1-LABEL: out_v4i32:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_v4i32:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0

 ; CHECK-SSE1-LABEL: out_v4i32_undef:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_v4i32_undef:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm1
 ; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0

 ; CHECK-SSE1-LABEL: in_v4i32:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
 ; CHECK-SSE1-NEXT:    andps (%rsi), %xmm0
 ; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-SSE1-NEXT:    movq -{{[0-9]+}}(%rsp), %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 8(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, (%rdi)
-; CHECK-SSE1-NEXT:    shrq $32, %rcx
-; CHECK-SSE1-NEXT:    shrq $32, %rdx
-; CHECK-SSE1-NEXT:    movl %edx, 12(%rdi)
-; CHECK-SSE1-NEXT:    movl %ecx, 4(%rdi)
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v4i32:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm1
 ; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0

llvm/test/CodeGen/X86/vector-gep.ll

 
 define <64 x ptr> @AGEP9(ptr %param, <64 x i32> %off) nounwind {
 ; CHECK-LABEL: AGEP9:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pushl %ebp
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    andl $-32, %esp
 ; CHECK-NEXT:    subl $160, %esp
-; CHECK-NEXT:    vpaddd %xmm1, %xmm1, %xmm3
+; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm3
 ; CHECK-NEXT:    vbroadcastss 12(%ebp), %xmm5
 ; CHECK-NEXT:    vpaddd %xmm3, %xmm5, %xmm3
 ; CHECK-NEXT:    vmovdqa %xmm3, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    vextractf128 $1, %ymm1, %xmm1
-; CHECK-NEXT:    vpaddd %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpaddd %xmm1, %xmm5, %xmm1
-; CHECK-NEXT:    vmovdqa %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm1
-; CHECK-NEXT:    vpaddd %xmm1, %xmm5, %xmm1
-; CHECK-NEXT:    vmovdqa %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    vpaddd %xmm1, %xmm1, %xmm0
+; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
+; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    vextractf128 $1, %ymm1, %xmm0
+; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
+; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
+; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    vpaddd %xmm2, %xmm2, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    vextractf128 $1, %ymm2, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    vmovdqa 40(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    vmovdqa 56(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    vmovdqa 104(%ebp), %xmm0
+; CHECK-NEXT:    vmovdqa 72(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
 ; CHECK-NEXT:    vmovdqa %xmm0, (%esp) # 16-byte Spill
-; CHECK-NEXT:    vmovdqa 120(%ebp), %xmm0
+; CHECK-NEXT:    vmovdqa 88(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm2
-; CHECK-NEXT:    vmovdqa 72(%ebp), %xmm0
+; CHECK-NEXT:    vmovdqa 104(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm1
-; CHECK-NEXT:    vmovdqa 88(%ebp), %xmm0
+; CHECK-NEXT:    vmovdqa 120(%ebp), %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vpaddd %xmm0, %xmm5, %xmm0
-; CHECK-NEXT:    vmovdqa 168(%ebp), %xmm6
+; CHECK-NEXT:    vmovdqa 136(%ebp), %xmm6
 ; CHECK-NEXT:    vpaddd %xmm6, %xmm6, %xmm6
 ; CHECK-NEXT:    vpaddd %xmm6, %xmm5, %xmm6
-; CHECK-NEXT:    vmovdqa 184(%ebp), %xmm7
+; CHECK-NEXT:    vmovdqa 152(%ebp), %xmm7
 ; CHECK-NEXT:    vpaddd %xmm7, %xmm7, %xmm7
 ; CHECK-NEXT:    vpaddd %xmm7, %xmm5, %xmm7
-; CHECK-NEXT:    vmovdqa 136(%ebp), %xmm4
+; CHECK-NEXT:    vmovdqa 168(%ebp), %xmm4
 ; CHECK-NEXT:    vpaddd %xmm4, %xmm4, %xmm4
 ; CHECK-NEXT:    vpaddd %xmm4, %xmm5, %xmm4
-; CHECK-NEXT:    vmovdqa 152(%ebp), %xmm3
+; CHECK-NEXT:    vmovdqa 184(%ebp), %xmm3
 ; CHECK-NEXT:    vpaddd %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vpaddd %xmm3, %xmm5, %xmm3
 ; CHECK-NEXT:    movl 8(%ebp), %eax
-; CHECK-NEXT:    vmovdqa %xmm3, 208(%eax)
-; CHECK-NEXT:    vmovdqa %xmm4, 192(%eax)
-; CHECK-NEXT:    vmovdqa %xmm7, 240(%eax)
-; CHECK-NEXT:    vmovdqa %xmm6, 224(%eax)
-; CHECK-NEXT:    vmovdqa %xmm0, 144(%eax)
-; CHECK-NEXT:    vmovdqa %xmm1, 128(%eax)
-; CHECK-NEXT:    vmovdqa %xmm2, 176(%eax)
+; CHECK-NEXT:    vmovdqa %xmm3, 240(%eax)
+; CHECK-NEXT:    vmovdqa %xmm4, 224(%eax)
+; CHECK-NEXT:    vmovdqa %xmm7, 208(%eax)
+; CHECK-NEXT:    vmovdqa %xmm6, 192(%eax)
+; CHECK-NEXT:    vmovdqa %xmm0, 176(%eax)
+; CHECK-NEXT:    vmovdqa %xmm1, 160(%eax)
+; CHECK-NEXT:    vmovdqa %xmm2, 144(%eax)
 ; CHECK-NEXT:    vmovaps (%esp), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, 160(%eax)
+; CHECK-NEXT:    vmovaps %xmm0, 128(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 112(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 96(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 80(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 64(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, 16(%eax)
-; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
-; CHECK-NEXT:    vmovaps %xmm0, (%eax)
-; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 48(%eax)
 ; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
 ; CHECK-NEXT:    vmovaps %xmm0, 32(%eax)
+; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-NEXT:    vmovaps %xmm0, 16(%eax)
+; CHECK-NEXT:    vmovaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; CHECK-NEXT:    vmovaps %xmm0, (%eax)
 ; CHECK-NEXT:    movl %ebp, %esp
 ; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retl $4
   %A = getelementptr i16, ptr %param, <64 x i32> %off
   ret <64 x ptr> %A
 }