[DAGCombiner][X86] `mergeConsecutiveStores()`: support merging splat-stores of the same value

[lebedev.ri]

Summary

getStoreMergeCandidates() is quite obvious for this kind
of source - just look for identical store values.
tryStoreMergeOfConstants() basically already supports it,
we just need to bypass some checks that are constant-specific.

For X86, this does appear to be always better,
as long as we store at least a single full vector register.
But the test changes for other targets were more questionable,
so i didn't enable it elsewhere.

[lebedev.ri]

Drop unused function declaration, NFC.

[pengfei]

obious -> obvious

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

Is this too strong to assume all the other cases are Splat? If this is the fact, why adding it in StoreSource rather than replacing Unknown?

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

Looks like regression here.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

What do you mean by "too strong"?
Originally we'd just return StoreSource::Unknown and bailout
without looking at other stores. If the other stores *don't* store
the same value, then we'll bailout just the same.

There is probably some generalization -- just because we've matched
a more fine-grained StoreSource, doesn't mean that the stores
*aren't* splatting the same value -- but that is not a required part
of this enhancement.

As for Unknown, i don't know, but it seems not useful to just drop it,
since it can be used to signal invalid state.

[lebedev.ri]

Add subvector test coverage.
@spatel should we not split such stores just like we do with stores of CONCAT_VECTOR?

[lebedev.ri]

Actually upload the right diff.

[lebedev.ri]

In D141778#4054600, @lebedev.ri wrote:

Add subvector test coverage.
@spatel should we not split such stores just like we do with stores of CONCAT_VECTOR?

Ah, so we already do that in collectConcatOps(). Okay, then these must be improvements.

[lebedev.ri]

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

We've traded 4 scalar stores to two vector stores + GPR->XMM xfer + two shuffles + shift.
I wouldn't say it's an obvious regression, since we get less contention in CPU store unit,
but it's not really an improvement, yes.

Can you help spot issues in the tests in elementwise-store-of-scalar-splat.ll / subvectorwise-store-of-vector-splat.ll?
If those do not have regressions, then we need to restrict some other fold.

[lebedev.ri]

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

This is an unrelated shuffle combining issue

movq    %r8, %xmm0
pshufd  $68, %xmm0, %xmm0               # xmm0 = xmm0[0,1,0,1]
psrad   $31, %xmm0
pshufd  $245, %xmm0, %xmm0              # xmm0 = xmm0[1,1,3,3]

What we want here, is to splat the sign bit of a 64-bit scalar to the entire XMM.
This should just be: (note the decoded shuffle mask)

movq    %r8, %xmm0
pshufd  $<???>, %xmm0, %xmm0            # xmm0 = xmm0[1,1,1,1]
psrad   $31, %xmm0

[lebedev.ri]

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

https://reviews.llvm.org/D141806 will address this.
Does anyone see any other regressions?

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

Alright, how about to use MayBeSplat?

I got the logic now. There is a getStoreSource before calling getStoreMergeCandidates and there is another getStoreSource in getStoreMergeCandidates. I think this is just a trick to make it works. It introduces uncertainty into the result. People may get unexpected result if they use this function somewhere else.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

Essentially, we want to guess what strategy will succeed
to merge the stores, looking at a single store.
StoreSource don't really describe the source,
but names a strategy, it's not an ideal enum name.

But likewise, i'm not sure why MayBeSplat is a better strategy name?
It either is if the strategy matched, or isn't if it didn't.

I got the logic now. There is a getStoreSource before calling getStoreMergeCandidates and there is another getStoreSource in getStoreMergeCandidates.

Yes.

I think this is just a trick to make it works.

What is?

It introduces uncertainty into the result. People may get unexpected result if they use this function somewhere else.

Sorry, i do not follow.

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

StoreSource don't really describe the source, but names a strategy, it's not an ideal enum name.

I understand it now. Thanks for the explanation!
I had wrong assumption about the function. Please ignore the comments.

19413–19414

This is dead now.

[pengfei]

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

Can you help spot issues in the tests in elementwise-store-of-scalar-splat.ll / subvectorwise-store-of-vector-splat.ll?

I like the AVX512 broadcast version, i.e., GPR->XMM directly. AVX2 and some AVX512F is suboptimal. I cannot tell whether it's good or not to replace splat store with shufle on SSE2. Should a rep stos be better?

[lebedev.ri]

llvm/test/CodeGen/X86/legalize-shl-vec.ll

247–252

Should a rep stos be better?

Almost certainly not.

[lebedev.ri]

Nits.

[lebedev.ri]

ping

[lebedev.ri]

Rebased, NFC.

[RKSimon]

llvm/test/CodeGen/X86/elementwise-store-of-scalar-splat.ll

2

why is this 'SCALAR' given that the triple implicitly assumes SSE2?

[lebedev.ri]

@RKSimon thank you for taking a look!
Addressing nits.

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   virtual bool hasPairedLoad(EVT /*LoadedType*/,
                              Align & /*RequiredAlignment*/) const {
     return false;
   }
 
   /// Return true if the target has a vector blend instruction.
   virtual bool hasVectorBlend() const { return false; }
 
+  /// Return true if the target can efficiently produce a VecVT-typed vector of
+  /// Elt elements, which may be either a scalar, or a vector itself,
+  /// to replace a chain of scalar stores of the Elt with that vector.
+  virtual bool shouldVectorizeScalarElementSplattingStores(SDValue Elt,
+                                                           EVT VecVT) const {
+    return false;
+  }
+
   /// Get the maximum supported factor for interleaved memory accesses.
   /// Default to be the minimum interleave factor: 2.
   virtual unsigned getMaxSupportedInterleaveFactor() const { return 2; }
 
   /// Lower an interleaved load to target specific intrinsics. Return
   /// true on success.
   ///
   /// \p LI is the vector load instruction.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

struct MemOpLink {
       // Offset from the base ptr.
       int64_t OffsetFromBase;
 
       MemOpLink(LSBaseSDNode *N, int64_t Offset)
           : MemNode(N), OffsetFromBase(Offset) {}
     };
 
     // Classify the origin of a stored value.
-    enum class StoreSource { Unknown, Constant, Extract, Load };
+    enum class StoreSource { Unknown, Splat, Constant, Extract, Load };
     StoreSource getStoreSource(SDValue StoreVal) {
       switch (StoreVal.getOpcode()) {
       case ISD::Constant:
       case ISD::ConstantFP:
         return StoreSource::Constant;
       case ISD::EXTRACT_VECTOR_ELT:
       case ISD::EXTRACT_SUBVECTOR:
         return StoreSource::Extract;
       case ISD::LOAD:
         return StoreSource::Load;
       default:
-        return StoreSource::Unknown;
+        return StoreSource::Splat;
       }
     }
 
     /// This is a helper function for visitMUL to check the profitability
     /// of folding (mul (add x, c1), c2) -> (add (mul x, c2), c1*c2).
     /// MulNode is the original multiply, AddNode is (add x, c1),
     /// and ConstNode is c2.
     bool isMulAddWithConstProfitable(SDNode *MulNode, SDValue AddNode,
private:
     /// This is a helper function for mergeConsecutiveStores. Given a list of
     /// store candidates, find the first N that are consecutive in memory.
     /// Returns 0 if there are not at least 2 consecutive stores to try merging.
     unsigned getConsecutiveStores(SmallVectorImpl<MemOpLink> &StoreNodes,
                                   int64_t ElementSizeBytes) const;
 
     /// This is a helper function for mergeConsecutiveStores. It is used for
     /// store chains that are composed entirely of constant values.
-    bool tryStoreMergeOfConstants(SmallVectorImpl<MemOpLink> &StoreNodes,
-                                  unsigned NumConsecutiveStores,
-                                  EVT MemVT, SDNode *Root, bool AllowVectors);
+    bool tryStoreMergeOfConstantsOrEltSplat(
+        SmallVectorImpl<MemOpLink> &StoreNodes, unsigned NumConsecutiveStores,
+        EVT MemVT, SDNode *Root, bool IsConstantSrc, bool AllowVectors);
 
     /// This is a helper function for mergeConsecutiveStores. It is used for
     /// store chains that are composed entirely of extracted vector elements.
     /// When extracting multiple vector elements, try to store them in one
     /// vector store rather than a sequence of scalar stores.
     bool tryStoreMergeOfExtracts(SmallVectorImpl<MemOpLink> &StoreNodes,
                                  unsigned NumConsecutiveStores, EVT MemVT,
                                  SDNode *Root);
case StoreSource::Extract:
       if (Other->isTruncatingStore())
         return false;
       if (!MemVT.bitsEq(OtherBC.getValueType()))
         return false;
       if (OtherBC.getOpcode() != ISD::EXTRACT_VECTOR_ELT &&
           OtherBC.getOpcode() != ISD::EXTRACT_SUBVECTOR)
         return false;
       break;
+    case StoreSource::Splat:
+      if (OtherBC != Val)
+        return false;
+      break;
     default:
       llvm_unreachable("Unhandled store source for merging");
     }
     Ptr = BaseIndexOffset::match(Other, DAG);
     return (BasePtr.equalBaseIndex(Ptr, DAG, Offset));
   };
 
   // Check if the pair of StoreNode and the RootNode already bail out many
if (NumConsecutiveStores > 1)
       return NumConsecutiveStores;
 
     // There are no consecutive stores at the start of the list.
     // Remove the first store and try again.
     StoreNodes.erase(StoreNodes.begin(), StoreNodes.begin() + 1);
   }
 }
 
-bool DAGCombiner::tryStoreMergeOfConstants(
+bool DAGCombiner::tryStoreMergeOfConstantsOrEltSplat(
     SmallVectorImpl<MemOpLink> &StoreNodes, unsigned NumConsecutiveStores,
-    EVT MemVT, SDNode *RootNode, bool AllowVectors) {
+    EVT MemVT, SDNode *RootNode, bool IsConstantSrc, bool AllowVectors) {
   LLVMContext &Context = *DAG.getContext();
   const DataLayout &DL = DAG.getDataLayout();
   int64_t ElementSizeBytes = MemVT.getStoreSize();
   unsigned NumMemElts = MemVT.isVector() ? MemVT.getVectorNumElements() : 1;
   bool MadeChange = false;
 
-  // Store the constants into memory as one consecutive store.
+  // Store the constants/same element into memory as one consecutive store.
   while (NumConsecutiveStores >= 2) {
     LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;
     unsigned FirstStoreAS = FirstInChain->getAddressSpace();
     Align FirstStoreAlign = FirstInChain->getAlign();
     unsigned LastLegalType = 1;
     unsigned LastLegalVectorType = 1;
     bool LastIntegerTrunc = false;
     bool NonZero = false;
     unsigned FirstZeroAfterNonZero = NumConsecutiveStores;
     for (unsigned i = 0; i < NumConsecutiveStores; ++i) {
       StoreSDNode *ST = cast<StoreSDNode>(StoreNodes[i].MemNode);
       SDValue StoredVal = ST->getValue();
+      if (IsConstantSrc) {
         bool IsElementZero = false;
         if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(StoredVal))
           IsElementZero = C->isZero();
         else if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(StoredVal))
           IsElementZero = C->getConstantFPValue()->isNullValue();
         if (IsElementZero) {
           if (NonZero && FirstZeroAfterNonZero == NumConsecutiveStores)
             FirstZeroAfterNonZero = i;
         }
         NonZero |= !IsElementZero;
+      }
 
       // Find a legal type for the constant store.
       unsigned SizeInBits = (i + 1) * ElementSizeBytes * 8;
       EVT StoreTy = EVT::getIntegerVT(Context, SizeInBits);
       unsigned IsFast = 0;
 
       // Break early when size is too large to be legal.
       if (StoreTy.getSizeInBits() > MaximumLegalStoreInBits)
for (unsigned i = 0; i < NumConsecutiveStores; ++i) {
             IsFast) {
           LastIntegerTrunc = true;
           LastLegalType = i + 1;
         }
       }
 
       // We only use vectors if the constant is known to be zero or the
       // target allows it and the function is not marked with the
-      // noimplicitfloat attribute.
-      if ((!NonZero ||
-           TLI.storeOfVectorConstantIsCheap(MemVT, i + 1, FirstStoreAS)) &&
-          AllowVectors) {
+      // noimplicitfloat attribute, or we are merging a splatting stores.
+      if (!IsConstantSrc || ((!NonZero || TLI.storeOfVectorConstantIsCheap(
+                                              MemVT, i + 1, FirstStoreAS)) &&
+                             AllowVectors)) {
         // Find a legal type for the vector store.
         unsigned Elts = (i + 1) * NumMemElts;
         EVT Ty = EVT::getVectorVT(Context, MemVT.getScalarType(), Elts);
         if (TLI.isTypeLegal(Ty) && TLI.isTypeLegal(MemVT) &&
             TLI.canMergeStoresTo(FirstStoreAS, Ty, DAG.getMachineFunction()) &&
             TLI.allowsMemoryAccess(Context, DL, Ty,
                                    *FirstInChain->getMemOperand(), &IsFast) &&
             IsFast)
           LastLegalVectorType = i + 1;
       }
     }
 
-    bool UseVector = (LastLegalVectorType > LastLegalType) && AllowVectors;
+    bool UseVector = !IsConstantSrc ||
+                     ((LastLegalVectorType > LastLegalType) && AllowVectors);
     unsigned NumElem = (UseVector) ? LastLegalVectorType : LastLegalType;
     bool UseTrunc = LastIntegerTrunc && !UseVector;
 
     // Check if we found a legal integer type that creates a meaningful
     // merge.
     if (NumElem < 2) {
       // We know that candidate stores are in order and of correct
       // shape. While there is no mergeable sequence from the
while (NumConsecutiveStores >= 2) {
     // Check that we can merge these candidates without causing a cycle.
     if (!checkMergeStoreCandidatesForDependencies(StoreNodes, NumElem,
                                                   RootNode)) {
       StoreNodes.erase(StoreNodes.begin(), StoreNodes.begin() + NumElem);
       NumConsecutiveStores -= NumElem;
       continue;
     }
 
-    MadeChange |= mergeStoresOfConstantsOrVecElts(StoreNodes, MemVT, NumElem,
-                                                  /*IsConstantSrc*/ true,
-                                                  UseVector, UseTrunc);
+    // If we are producing a splat-store, we want a profitability check.
+    if (!IsConstantSrc) {
+      // Get the type for the merged vector store.
+      unsigned Elts = NumElem * NumMemElts;
+      EVT StoreTy =
+          EVT::getVectorVT(*DAG.getContext(), MemVT.getScalarType(), Elts);
+      // Can we efficiently produce a vector of the value we are splatting?
+      if (!TLI.shouldVectorizeScalarElementSplattingStores(
+              cast<StoreSDNode>(StoreNodes[0].MemNode)->getValue(), StoreTy)) {
+        StoreNodes.erase(StoreNodes.begin(), StoreNodes.begin() + NumElem);
+        NumConsecutiveStores -= NumElem;
+        continue;
+      }
+    }
+
+    MadeChange |= mergeStoresOfConstantsOrVecElts(
+        StoreNodes, MemVT, NumElem, IsConstantSrc, UseVector, UseTrunc);
 
     // Remove merged stores for next iteration.
     StoreNodes.erase(StoreNodes.begin(), StoreNodes.begin() + NumElem);
     NumConsecutiveStores -= NumElem;
   }
   return MadeChange;
 }
 
bool DAGCombiner::mergeConsecutiveStores(StoreSDNode *St) {
   int64_t ElementSizeBytes = MemVT.getStoreSize();
   if (ElementSizeBytes * 8 != (int64_t)MemVT.getSizeInBits())
     return false;
 
   // Do not bother looking at stored values that are not constants, loads, or
   // extracted vector elements.
   SDValue StoredVal = peekThroughBitcasts(St->getValue());
   const StoreSource StoreSrc = getStoreSource(StoredVal);
   if (StoreSrc == StoreSource::Unknown)
     return false;
 
   SmallVector<MemOpLink, 8> StoreNodes;
   SDNode *RootNode;
   // Find potential store merge candidates by searching through chain sub-DAG
   getStoreMergeCandidates(St, StoreNodes, RootNode);
 
   // Check if there is anything to merge.
   if (StoreNodes.size() < 2)
while (StoreNodes.size() > 1) {
     // There are no more stores in the list to examine.
     if (NumConsecutiveStores == 0)
       return MadeChange;
 
     // We have at least 2 consecutive stores. Try to merge them.
     assert(NumConsecutiveStores >= 2 && "Expected at least 2 stores");
     switch (StoreSrc) {
     case StoreSource::Constant:
-      MadeChange |= tryStoreMergeOfConstants(StoreNodes, NumConsecutiveStores,
-                                             MemVT, RootNode, AllowVectors);
+    case StoreSource::Splat:
+      MadeChange |= tryStoreMergeOfConstantsOrEltSplat(
+          StoreNodes, NumConsecutiveStores, MemVT, RootNode,
+          /*IsConstantSrc=*/StoreSrc == StoreSource::Constant, AllowVectors);
       break;
 
     case StoreSource::Extract:
       MadeChange |= tryStoreMergeOfExtracts(StoreNodes, NumConsecutiveStores,
                                             MemVT, RootNode);
       break;
 
     case StoreSource::Load:

llvm/lib/Target/X86/X86ISelLowering.h

public:
     bool hasStackProbeSymbol(const MachineFunction &MF) const override;
     bool hasInlineStackProbe(const MachineFunction &MF) const override;
     StringRef getStackProbeSymbolName(const MachineFunction &MF) const override;
 
     unsigned getStackProbeSize(const MachineFunction &MF) const;
 
     bool hasVectorBlend() const override { return true; }
 
+    bool shouldVectorizeScalarElementSplattingStores(SDValue Elt,
+                                                     EVT VecVT) const override;
+
     unsigned getMaxSupportedInterleaveFactor() const override { return 4; }
 
     bool isInlineAsmTargetBranch(const SmallVectorImpl<StringRef> &AsmStrs,
                                  unsigned OpNo) const override;
 
     /// Lower interleaved load(s) into target specific
     /// instructions/intrinsics.
     bool lowerInterleavedLoad(LoadInst *LI,

llvm/lib/Target/X86/X86ISelLowering.cpp

 }
 
 Align X86TargetLowering::getPrefLoopAlignment(MachineLoop *ML) const {
   if (ML->isInnermost() &&
       ExperimentalPrefInnermostLoopAlignment.getNumOccurrences())
     return Align(1ULL << ExperimentalPrefInnermostLoopAlignment);
   return TargetLowering::getPrefLoopAlignment();
 }
+
+bool X86TargetLowering::shouldVectorizeScalarElementSplattingStores(
+    SDValue Elt, EVT VecVT) const {
+  return Subtarget.hasSSE2() && VecVT.getSizeInBits() >= 128;
+}

llvm/test/CodeGen/X86/MergeConsecutiveStores.ll

 
 }
 
 ; Merging vector stores when sourced from a constant vector is not currently handled.
 define void @merge_vec_stores_of_constants(<4 x i32>* %ptr) {
 ; CHECK-LABEL: merge_vec_stores_of_constants:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    vmovaps %xmm0, 48(%rdi)
-; CHECK-NEXT:    vmovaps %xmm0, 64(%rdi)
+; CHECK-NEXT:    vmovups %ymm0, 48(%rdi)
+; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
   %idx0 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 3
   %idx1 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 4
   store <4 x i32> <i32 0, i32 0, i32 0, i32 0>, <4 x i32>* %idx0, align 16
   store <4 x i32> <i32 0, i32 0, i32 0, i32 0>, <4 x i32>* %idx1, align 16
   ret void
 
 }

llvm/test/CodeGen/X86/elementwise-store-of-scalar-splat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s --check-prefixes=ALL,SCALAR
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=-sse2 | FileCheck %s --check-prefixes=ALL,SCALAR
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+sse2 | FileCheck %s --check-prefixes=ALL,SSE,SSE2,SSE2-ONLY
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+sse3 | FileCheck %s --check-prefixes=ALL,SSE,SSE2,SSE3
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+ssse3 | FileCheck %s --check-prefixes=ALL,SSE,SSSE3,SSSE3-ONLY
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 | FileCheck %s --check-prefixes=ALL,SSE,SSSE3,SSE41
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.2 | FileCheck %s --check-prefixes=ALL,SSE,SSSE3,SSE42
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx | FileCheck %s --check-prefixes=ALL,AVX,AVX1
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 | FileCheck %s --check-prefixes=ALL,AVX,AVX2
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512vl | FileCheck %s --check-prefixes=ALL,AVX512,AVX512F
; ALL-NEXT:    retq
   %out.elt0.ptr = getelementptr float, ptr %out.vec.ptr, i64 0
   store float %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr float, ptr %out.vec.ptr, i64 1
   store float %in.elt, ptr %out.elt1.ptr, align 4
   ret void
 }
 
 define void @vec128_i8(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_i8:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movzbl (%rdi), %eax
-; ALL-NEXT:    notb %al
-; ALL-NEXT:    movb %al, (%rsi)
-; ALL-NEXT:    movb %al, 1(%rsi)
-; ALL-NEXT:    movb %al, 2(%rsi)
-; ALL-NEXT:    movb %al, 3(%rsi)
-; ALL-NEXT:    movb %al, 4(%rsi)
-; ALL-NEXT:    movb %al, 5(%rsi)
-; ALL-NEXT:    movb %al, 6(%rsi)
-; ALL-NEXT:    movb %al, 7(%rsi)
-; ALL-NEXT:    movb %al, 8(%rsi)
-; ALL-NEXT:    movb %al, 9(%rsi)
-; ALL-NEXT:    movb %al, 10(%rsi)
-; ALL-NEXT:    movb %al, 11(%rsi)
-; ALL-NEXT:    movb %al, 12(%rsi)
-; ALL-NEXT:    movb %al, 13(%rsi)
-; ALL-NEXT:    movb %al, 14(%rsi)
-; ALL-NEXT:    movb %al, 15(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_i8:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movzbl (%rdi), %eax
+; SCALAR-NEXT:    notb %al
+; SCALAR-NEXT:    movb %al, (%rsi)
+; SCALAR-NEXT:    movb %al, 1(%rsi)
+; SCALAR-NEXT:    movb %al, 2(%rsi)
+; SCALAR-NEXT:    movb %al, 3(%rsi)
+; SCALAR-NEXT:    movb %al, 4(%rsi)
+; SCALAR-NEXT:    movb %al, 5(%rsi)
+; SCALAR-NEXT:    movb %al, 6(%rsi)
+; SCALAR-NEXT:    movb %al, 7(%rsi)
+; SCALAR-NEXT:    movb %al, 8(%rsi)
+; SCALAR-NEXT:    movb %al, 9(%rsi)
+; SCALAR-NEXT:    movb %al, 10(%rsi)
+; SCALAR-NEXT:    movb %al, 11(%rsi)
+; SCALAR-NEXT:    movb %al, 12(%rsi)
+; SCALAR-NEXT:    movb %al, 13(%rsi)
+; SCALAR-NEXT:    movb %al, 14(%rsi)
+; SCALAR-NEXT:    movb %al, 15(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE2-LABEL: vec128_i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzbl (%rdi), %eax
+; SSE2-NEXT:    notb %al
+; SSE2-NEXT:    movzbl %al, %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, (%rsi)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: vec128_i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzbl (%rdi), %eax
+; SSSE3-NEXT:    notb %al
+; SSSE3-NEXT:    movzbl %al, %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    pxor %xmm1, %xmm1
+; SSSE3-NEXT:    pshufb %xmm1, %xmm0
+; SSSE3-NEXT:    movdqa %xmm0, (%rsi)
+; SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: vec128_i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzbl (%rdi), %eax
+; AVX1-NEXT:    notb %al
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzbl (%rdi), %eax
+; AVX2-NEXT:    notb %al
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastb %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec128_i8:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movzbl (%rdi), %eax
+; AVX512F-NEXT:    notb %al
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastb %xmm0, %xmm0
+; AVX512F-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec128_i8:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movzbl (%rdi), %eax
+; AVX512BW-NEXT:    notb %al
+; AVX512BW-NEXT:    vpbroadcastb %eax, %xmm0
+; AVX512BW-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i8, ptr %in.elt.ptr, align 64
   %in.elt = xor i8 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i8, ptr %out.vec.ptr, i64 0
   store i8 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i8, ptr %out.vec.ptr, i64 1
   store i8 %in.elt, ptr %out.elt1.ptr, align 1
   %out.elt2.ptr = getelementptr i8, ptr %out.vec.ptr, i64 2
   store i8 %in.elt, ptr %out.elt2.ptr, align 2
; AVX512BW-NEXT:    retq
   %out.elt14.ptr = getelementptr i8, ptr %out.vec.ptr, i64 14
   store i8 %in.elt, ptr %out.elt14.ptr, align 2
   %out.elt15.ptr = getelementptr i8, ptr %out.vec.ptr, i64 15
   store i8 %in.elt, ptr %out.elt15.ptr, align 1
   ret void
 }
 
 define void @vec128_i16(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_i16:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movw %ax, (%rsi)
-; ALL-NEXT:    movw %ax, 2(%rsi)
-; ALL-NEXT:    movw %ax, 4(%rsi)
-; ALL-NEXT:    movw %ax, 6(%rsi)
-; ALL-NEXT:    movw %ax, 8(%rsi)
-; ALL-NEXT:    movw %ax, 10(%rsi)
-; ALL-NEXT:    movw %ax, 12(%rsi)
-; ALL-NEXT:    movw %ax, 14(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_i16:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movw %ax, (%rsi)
+; SCALAR-NEXT:    movw %ax, 2(%rsi)
+; SCALAR-NEXT:    movw %ax, 4(%rsi)
+; SCALAR-NEXT:    movw %ax, 6(%rsi)
+; SCALAR-NEXT:    movw %ax, 8(%rsi)
+; SCALAR-NEXT:    movw %ax, 10(%rsi)
+; SCALAR-NEXT:    movw %ax, 12(%rsi)
+; SCALAR-NEXT:    movw %ax, 14(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec128_i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec128_i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastw %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec128_i16:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movl (%rdi), %eax
+; AVX512F-NEXT:    notl %eax
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastw %xmm0, %xmm0
+; AVX512F-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec128_i16:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movl (%rdi), %eax
+; AVX512BW-NEXT:    notl %eax
+; AVX512BW-NEXT:    vpbroadcastw %eax, %xmm0
+; AVX512BW-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i16, ptr %in.elt.ptr, align 64
   %in.elt = xor i16 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i16, ptr %out.vec.ptr, i64 0
   store i16 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i16, ptr %out.vec.ptr, i64 1
   store i16 %in.elt, ptr %out.elt1.ptr, align 2
   %out.elt2.ptr = getelementptr i16, ptr %out.vec.ptr, i64 2
   store i16 %in.elt, ptr %out.elt2.ptr, align 4
   %out.elt3.ptr = getelementptr i16, ptr %out.vec.ptr, i64 3
   store i16 %in.elt, ptr %out.elt3.ptr, align 2
   %out.elt4.ptr = getelementptr i16, ptr %out.vec.ptr, i64 4
   store i16 %in.elt, ptr %out.elt4.ptr, align 8
   %out.elt5.ptr = getelementptr i16, ptr %out.vec.ptr, i64 5
   store i16 %in.elt, ptr %out.elt5.ptr, align 2
   %out.elt6.ptr = getelementptr i16, ptr %out.vec.ptr, i64 6
   store i16 %in.elt, ptr %out.elt6.ptr, align 4
   %out.elt7.ptr = getelementptr i16, ptr %out.vec.ptr, i64 7
   store i16 %in.elt, ptr %out.elt7.ptr, align 2
   ret void
 }
 
 define void @vec128_i32(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_i32:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_i32:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec128_i32:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec128_i32:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_i32:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec128_i32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt = xor i32 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i32, ptr %out.vec.ptr, i64 0
   store i32 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i32, ptr %out.vec.ptr, i64 1
   store i32 %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr i32, ptr %out.vec.ptr, i64 2
   store i32 %in.elt, ptr %out.elt2.ptr, align 8
   %out.elt3.ptr = getelementptr i32, ptr %out.vec.ptr, i64 3
   store i32 %in.elt, ptr %out.elt3.ptr, align 4
   ret void
 }
 
 define void @vec128_float(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_float:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_float:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec128_float:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec128_float:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_float:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec128_float:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i32 %in.elt.not, -1
   %in.elt = bitcast i32 %in.elt.int to float
   %out.elt0.ptr = getelementptr float, ptr %out.vec.ptr, i64 0
   store float %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr float, ptr %out.vec.ptr, i64 1
   store float %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr float, ptr %out.vec.ptr, i64 2
   store float %in.elt, ptr %out.elt2.ptr, align 8
   %out.elt3.ptr = getelementptr float, ptr %out.vec.ptr, i64 3
   store float %in.elt, ptr %out.elt3.ptr, align 4
   ret void
 }
 
 define void @vec128_i64(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_i64:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_i64:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec128_i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec128_i64:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_i64:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec128_i64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt = xor i64 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i64, ptr %out.vec.ptr, i64 0
   store i64 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i64, ptr %out.vec.ptr, i64 1
   store i64 %in.elt, ptr %out.elt1.ptr, align 8
   ret void
 }
 
 define void @vec128_double(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec128_double:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec128_double:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec128_double:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec128_double:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec128_double:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %xmm0
+; AVX2-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec128_double:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i64 %in.elt.not, -1
   %in.elt = bitcast i64 %in.elt.int to double
   %out.elt0.ptr = getelementptr double, ptr %out.vec.ptr, i64 0
   store double %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr double, ptr %out.vec.ptr, i64 1
   store double %in.elt, ptr %out.elt1.ptr, align 8
   ret void
 }
 
 define void @vec256_i8(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_i8:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movzbl (%rdi), %eax
-; ALL-NEXT:    notb %al
-; ALL-NEXT:    movb %al, (%rsi)
-; ALL-NEXT:    movb %al, 1(%rsi)
-; ALL-NEXT:    movb %al, 2(%rsi)
-; ALL-NEXT:    movb %al, 3(%rsi)
-; ALL-NEXT:    movb %al, 4(%rsi)
-; ALL-NEXT:    movb %al, 5(%rsi)
-; ALL-NEXT:    movb %al, 6(%rsi)
-; ALL-NEXT:    movb %al, 7(%rsi)
-; ALL-NEXT:    movb %al, 8(%rsi)
-; ALL-NEXT:    movb %al, 9(%rsi)
-; ALL-NEXT:    movb %al, 10(%rsi)
-; ALL-NEXT:    movb %al, 11(%rsi)
-; ALL-NEXT:    movb %al, 12(%rsi)
-; ALL-NEXT:    movb %al, 13(%rsi)
-; ALL-NEXT:    movb %al, 14(%rsi)
-; ALL-NEXT:    movb %al, 15(%rsi)
-; ALL-NEXT:    movb %al, 16(%rsi)
-; ALL-NEXT:    movb %al, 17(%rsi)
-; ALL-NEXT:    movb %al, 18(%rsi)
-; ALL-NEXT:    movb %al, 19(%rsi)
-; ALL-NEXT:    movb %al, 20(%rsi)
-; ALL-NEXT:    movb %al, 21(%rsi)
-; ALL-NEXT:    movb %al, 22(%rsi)
-; ALL-NEXT:    movb %al, 23(%rsi)
-; ALL-NEXT:    movb %al, 24(%rsi)
-; ALL-NEXT:    movb %al, 25(%rsi)
-; ALL-NEXT:    movb %al, 26(%rsi)
-; ALL-NEXT:    movb %al, 27(%rsi)
-; ALL-NEXT:    movb %al, 28(%rsi)
-; ALL-NEXT:    movb %al, 29(%rsi)
-; ALL-NEXT:    movb %al, 30(%rsi)
-; ALL-NEXT:    movb %al, 31(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_i8:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movzbl (%rdi), %eax
+; SCALAR-NEXT:    notb %al
+; SCALAR-NEXT:    movb %al, (%rsi)
+; SCALAR-NEXT:    movb %al, 1(%rsi)
+; SCALAR-NEXT:    movb %al, 2(%rsi)
+; SCALAR-NEXT:    movb %al, 3(%rsi)
+; SCALAR-NEXT:    movb %al, 4(%rsi)
+; SCALAR-NEXT:    movb %al, 5(%rsi)
+; SCALAR-NEXT:    movb %al, 6(%rsi)
+; SCALAR-NEXT:    movb %al, 7(%rsi)
+; SCALAR-NEXT:    movb %al, 8(%rsi)
+; SCALAR-NEXT:    movb %al, 9(%rsi)
+; SCALAR-NEXT:    movb %al, 10(%rsi)
+; SCALAR-NEXT:    movb %al, 11(%rsi)
+; SCALAR-NEXT:    movb %al, 12(%rsi)
+; SCALAR-NEXT:    movb %al, 13(%rsi)
+; SCALAR-NEXT:    movb %al, 14(%rsi)
+; SCALAR-NEXT:    movb %al, 15(%rsi)
+; SCALAR-NEXT:    movb %al, 16(%rsi)
+; SCALAR-NEXT:    movb %al, 17(%rsi)
+; SCALAR-NEXT:    movb %al, 18(%rsi)
+; SCALAR-NEXT:    movb %al, 19(%rsi)
+; SCALAR-NEXT:    movb %al, 20(%rsi)
+; SCALAR-NEXT:    movb %al, 21(%rsi)
+; SCALAR-NEXT:    movb %al, 22(%rsi)
+; SCALAR-NEXT:    movb %al, 23(%rsi)
+; SCALAR-NEXT:    movb %al, 24(%rsi)
+; SCALAR-NEXT:    movb %al, 25(%rsi)
+; SCALAR-NEXT:    movb %al, 26(%rsi)
+; SCALAR-NEXT:    movb %al, 27(%rsi)
+; SCALAR-NEXT:    movb %al, 28(%rsi)
+; SCALAR-NEXT:    movb %al, 29(%rsi)
+; SCALAR-NEXT:    movb %al, 30(%rsi)
+; SCALAR-NEXT:    movb %al, 31(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE2-LABEL: vec256_i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzbl (%rdi), %eax
+; SSE2-NEXT:    notb %al
+; SSE2-NEXT:    movzbl %al, %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, (%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: vec256_i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzbl (%rdi), %eax
+; SSSE3-NEXT:    notb %al
+; SSSE3-NEXT:    movzbl %al, %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    pxor %xmm1, %xmm1
+; SSSE3-NEXT:    pshufb %xmm1, %xmm0
+; SSSE3-NEXT:    movdqa %xmm0, (%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: vec256_i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzbl (%rdi), %eax
+; AVX1-NEXT:    notb %al
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzbl (%rdi), %eax
+; AVX2-NEXT:    notb %al
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec256_i8:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movzbl (%rdi), %eax
+; AVX512F-NEXT:    notb %al
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX512F-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec256_i8:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movzbl (%rdi), %eax
+; AVX512BW-NEXT:    notb %al
+; AVX512BW-NEXT:    vpbroadcastb %eax, %ymm0
+; AVX512BW-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i8, ptr %in.elt.ptr, align 64
   %in.elt = xor i8 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i8, ptr %out.vec.ptr, i64 0
   store i8 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i8, ptr %out.vec.ptr, i64 1
   store i8 %in.elt, ptr %out.elt1.ptr, align 1
   %out.elt2.ptr = getelementptr i8, ptr %out.vec.ptr, i64 2
   store i8 %in.elt, ptr %out.elt2.ptr, align 2
; AVX512BW-NEXT:    retq
   %out.elt30.ptr = getelementptr i8, ptr %out.vec.ptr, i64 30
   store i8 %in.elt, ptr %out.elt30.ptr, align 2
   %out.elt31.ptr = getelementptr i8, ptr %out.vec.ptr, i64 31
   store i8 %in.elt, ptr %out.elt31.ptr, align 1
   ret void
 }
 
 define void @vec256_i16(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_i16:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movw %ax, (%rsi)
-; ALL-NEXT:    movw %ax, 2(%rsi)
-; ALL-NEXT:    movw %ax, 4(%rsi)
-; ALL-NEXT:    movw %ax, 6(%rsi)
-; ALL-NEXT:    movw %ax, 8(%rsi)
-; ALL-NEXT:    movw %ax, 10(%rsi)
-; ALL-NEXT:    movw %ax, 12(%rsi)
-; ALL-NEXT:    movw %ax, 14(%rsi)
-; ALL-NEXT:    movw %ax, 16(%rsi)
-; ALL-NEXT:    movw %ax, 18(%rsi)
-; ALL-NEXT:    movw %ax, 20(%rsi)
-; ALL-NEXT:    movw %ax, 22(%rsi)
-; ALL-NEXT:    movw %ax, 24(%rsi)
-; ALL-NEXT:    movw %ax, 26(%rsi)
-; ALL-NEXT:    movw %ax, 28(%rsi)
-; ALL-NEXT:    movw %ax, 30(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_i16:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movw %ax, (%rsi)
+; SCALAR-NEXT:    movw %ax, 2(%rsi)
+; SCALAR-NEXT:    movw %ax, 4(%rsi)
+; SCALAR-NEXT:    movw %ax, 6(%rsi)
+; SCALAR-NEXT:    movw %ax, 8(%rsi)
+; SCALAR-NEXT:    movw %ax, 10(%rsi)
+; SCALAR-NEXT:    movw %ax, 12(%rsi)
+; SCALAR-NEXT:    movw %ax, 14(%rsi)
+; SCALAR-NEXT:    movw %ax, 16(%rsi)
+; SCALAR-NEXT:    movw %ax, 18(%rsi)
+; SCALAR-NEXT:    movw %ax, 20(%rsi)
+; SCALAR-NEXT:    movw %ax, 22(%rsi)
+; SCALAR-NEXT:    movw %ax, 24(%rsi)
+; SCALAR-NEXT:    movw %ax, 26(%rsi)
+; SCALAR-NEXT:    movw %ax, 28(%rsi)
+; SCALAR-NEXT:    movw %ax, 30(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec256_i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec256_i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec256_i16:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movl (%rdi), %eax
+; AVX512F-NEXT:    notl %eax
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX512F-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec256_i16:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movl (%rdi), %eax
+; AVX512BW-NEXT:    notl %eax
+; AVX512BW-NEXT:    vpbroadcastw %eax, %ymm0
+; AVX512BW-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i16, ptr %in.elt.ptr, align 64
   %in.elt = xor i16 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i16, ptr %out.vec.ptr, i64 0
   store i16 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i16, ptr %out.vec.ptr, i64 1
   store i16 %in.elt, ptr %out.elt1.ptr, align 2
   %out.elt2.ptr = getelementptr i16, ptr %out.vec.ptr, i64 2
   store i16 %in.elt, ptr %out.elt2.ptr, align 4
; AVX512BW-NEXT:    retq
   %out.elt14.ptr = getelementptr i16, ptr %out.vec.ptr, i64 14
   store i16 %in.elt, ptr %out.elt14.ptr, align 4
   %out.elt15.ptr = getelementptr i16, ptr %out.vec.ptr, i64 15
   store i16 %in.elt, ptr %out.elt15.ptr, align 2
   ret void
 }
 
 define void @vec256_i32(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_i32:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_i32:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec256_i32:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec256_i32:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_i32:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec256_i32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt = xor i32 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i32, ptr %out.vec.ptr, i64 0
   store i32 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i32, ptr %out.vec.ptr, i64 1
   store i32 %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr i32, ptr %out.vec.ptr, i64 2
   store i32 %in.elt, ptr %out.elt2.ptr, align 8
   %out.elt3.ptr = getelementptr i32, ptr %out.vec.ptr, i64 3
   store i32 %in.elt, ptr %out.elt3.ptr, align 4
   %out.elt4.ptr = getelementptr i32, ptr %out.vec.ptr, i64 4
   store i32 %in.elt, ptr %out.elt4.ptr, align 16
   %out.elt5.ptr = getelementptr i32, ptr %out.vec.ptr, i64 5
   store i32 %in.elt, ptr %out.elt5.ptr, align 4
   %out.elt6.ptr = getelementptr i32, ptr %out.vec.ptr, i64 6
   store i32 %in.elt, ptr %out.elt6.ptr, align 8
   %out.elt7.ptr = getelementptr i32, ptr %out.vec.ptr, i64 7
   store i32 %in.elt, ptr %out.elt7.ptr, align 4
   ret void
 }
 
 define void @vec256_float(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_float:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_float:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec256_float:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec256_float:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_float:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec256_float:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i32 %in.elt.not, -1
   %in.elt = bitcast i32 %in.elt.int to float
   %out.elt0.ptr = getelementptr float, ptr %out.vec.ptr, i64 0
   store float %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr float, ptr %out.vec.ptr, i64 1
   store float %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr float, ptr %out.vec.ptr, i64 2
   store float %in.elt, ptr %out.elt2.ptr, align 8
   %out.elt3.ptr = getelementptr float, ptr %out.vec.ptr, i64 3
   store float %in.elt, ptr %out.elt3.ptr, align 4
   %out.elt4.ptr = getelementptr float, ptr %out.vec.ptr, i64 4
   store float %in.elt, ptr %out.elt4.ptr, align 16
   %out.elt5.ptr = getelementptr float, ptr %out.vec.ptr, i64 5
   store float %in.elt, ptr %out.elt5.ptr, align 4
   %out.elt6.ptr = getelementptr float, ptr %out.vec.ptr, i64 6
   store float %in.elt, ptr %out.elt6.ptr, align 8
   %out.elt7.ptr = getelementptr float, ptr %out.vec.ptr, i64 7
   store float %in.elt, ptr %out.elt7.ptr, align 4
   ret void
 }
 
 define void @vec256_i64(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_i64:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_i64:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec256_i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec256_i64:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_i64:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec256_i64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt = xor i64 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i64, ptr %out.vec.ptr, i64 0
   store i64 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i64, ptr %out.vec.ptr, i64 1
   store i64 %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr i64, ptr %out.vec.ptr, i64 2
   store i64 %in.elt, ptr %out.elt2.ptr, align 16
   %out.elt3.ptr = getelementptr i64, ptr %out.vec.ptr, i64 3
   store i64 %in.elt, ptr %out.elt3.ptr, align 8
   ret void
 }
 
 define void @vec256_double(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec256_double:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec256_double:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec256_double:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec256_double:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec256_double:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec256_double:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i64 %in.elt.not, -1
   %in.elt = bitcast i64 %in.elt.int to double
   %out.elt0.ptr = getelementptr double, ptr %out.vec.ptr, i64 0
   store double %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr double, ptr %out.vec.ptr, i64 1
   store double %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr double, ptr %out.vec.ptr, i64 2
; ALL-NEXT:    retq
   %out.elt0.ptr = getelementptr i128, ptr %out.vec.ptr, i64 0
   store i128 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i128, ptr %out.vec.ptr, i64 1
   store i128 %in.elt, ptr %out.elt1.ptr, align 16
   ret void
 }
 
 define void @vec384_i8(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_i8:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movzbl (%rdi), %eax
-; ALL-NEXT:    notb %al
-; ALL-NEXT:    movb %al, (%rsi)
-; ALL-NEXT:    movb %al, 1(%rsi)
-; ALL-NEXT:    movb %al, 2(%rsi)
-; ALL-NEXT:    movb %al, 3(%rsi)
-; ALL-NEXT:    movb %al, 4(%rsi)
-; ALL-NEXT:    movb %al, 5(%rsi)
-; ALL-NEXT:    movb %al, 6(%rsi)
-; ALL-NEXT:    movb %al, 7(%rsi)
-; ALL-NEXT:    movb %al, 8(%rsi)
-; ALL-NEXT:    movb %al, 9(%rsi)
-; ALL-NEXT:    movb %al, 10(%rsi)
-; ALL-NEXT:    movb %al, 11(%rsi)
-; ALL-NEXT:    movb %al, 12(%rsi)
-; ALL-NEXT:    movb %al, 13(%rsi)
-; ALL-NEXT:    movb %al, 14(%rsi)
-; ALL-NEXT:    movb %al, 15(%rsi)
-; ALL-NEXT:    movb %al, 16(%rsi)
-; ALL-NEXT:    movb %al, 17(%rsi)
-; ALL-NEXT:    movb %al, 18(%rsi)
-; ALL-NEXT:    movb %al, 19(%rsi)
-; ALL-NEXT:    movb %al, 20(%rsi)
-; ALL-NEXT:    movb %al, 21(%rsi)
-; ALL-NEXT:    movb %al, 22(%rsi)
-; ALL-NEXT:    movb %al, 23(%rsi)
-; ALL-NEXT:    movb %al, 24(%rsi)
-; ALL-NEXT:    movb %al, 25(%rsi)
-; ALL-NEXT:    movb %al, 26(%rsi)
-; ALL-NEXT:    movb %al, 27(%rsi)
-; ALL-NEXT:    movb %al, 28(%rsi)
-; ALL-NEXT:    movb %al, 29(%rsi)
-; ALL-NEXT:    movb %al, 30(%rsi)
-; ALL-NEXT:    movb %al, 31(%rsi)
-; ALL-NEXT:    movb %al, 32(%rsi)
-; ALL-NEXT:    movb %al, 33(%rsi)
-; ALL-NEXT:    movb %al, 34(%rsi)
-; ALL-NEXT:    movb %al, 35(%rsi)
-; ALL-NEXT:    movb %al, 36(%rsi)
-; ALL-NEXT:    movb %al, 37(%rsi)
-; ALL-NEXT:    movb %al, 38(%rsi)
-; ALL-NEXT:    movb %al, 39(%rsi)
-; ALL-NEXT:    movb %al, 40(%rsi)
-; ALL-NEXT:    movb %al, 41(%rsi)
-; ALL-NEXT:    movb %al, 42(%rsi)
-; ALL-NEXT:    movb %al, 43(%rsi)
-; ALL-NEXT:    movb %al, 44(%rsi)
-; ALL-NEXT:    movb %al, 45(%rsi)
-; ALL-NEXT:    movb %al, 46(%rsi)
-; ALL-NEXT:    movb %al, 47(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_i8:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movzbl (%rdi), %eax
+; SCALAR-NEXT:    notb %al
+; SCALAR-NEXT:    movb %al, (%rsi)
+; SCALAR-NEXT:    movb %al, 1(%rsi)
+; SCALAR-NEXT:    movb %al, 2(%rsi)
+; SCALAR-NEXT:    movb %al, 3(%rsi)
+; SCALAR-NEXT:    movb %al, 4(%rsi)
+; SCALAR-NEXT:    movb %al, 5(%rsi)
+; SCALAR-NEXT:    movb %al, 6(%rsi)
+; SCALAR-NEXT:    movb %al, 7(%rsi)
+; SCALAR-NEXT:    movb %al, 8(%rsi)
+; SCALAR-NEXT:    movb %al, 9(%rsi)
+; SCALAR-NEXT:    movb %al, 10(%rsi)
+; SCALAR-NEXT:    movb %al, 11(%rsi)
+; SCALAR-NEXT:    movb %al, 12(%rsi)
+; SCALAR-NEXT:    movb %al, 13(%rsi)
+; SCALAR-NEXT:    movb %al, 14(%rsi)
+; SCALAR-NEXT:    movb %al, 15(%rsi)
+; SCALAR-NEXT:    movb %al, 16(%rsi)
+; SCALAR-NEXT:    movb %al, 17(%rsi)
+; SCALAR-NEXT:    movb %al, 18(%rsi)
+; SCALAR-NEXT:    movb %al, 19(%rsi)
+; SCALAR-NEXT:    movb %al, 20(%rsi)
+; SCALAR-NEXT:    movb %al, 21(%rsi)
+; SCALAR-NEXT:    movb %al, 22(%rsi)
+; SCALAR-NEXT:    movb %al, 23(%rsi)
+; SCALAR-NEXT:    movb %al, 24(%rsi)
+; SCALAR-NEXT:    movb %al, 25(%rsi)
+; SCALAR-NEXT:    movb %al, 26(%rsi)
+; SCALAR-NEXT:    movb %al, 27(%rsi)
+; SCALAR-NEXT:    movb %al, 28(%rsi)
+; SCALAR-NEXT:    movb %al, 29(%rsi)
+; SCALAR-NEXT:    movb %al, 30(%rsi)
+; SCALAR-NEXT:    movb %al, 31(%rsi)
+; SCALAR-NEXT:    movb %al, 32(%rsi)
+; SCALAR-NEXT:    movb %al, 33(%rsi)
+; SCALAR-NEXT:    movb %al, 34(%rsi)
+; SCALAR-NEXT:    movb %al, 35(%rsi)
+; SCALAR-NEXT:    movb %al, 36(%rsi)
+; SCALAR-NEXT:    movb %al, 37(%rsi)
+; SCALAR-NEXT:    movb %al, 38(%rsi)
+; SCALAR-NEXT:    movb %al, 39(%rsi)
+; SCALAR-NEXT:    movb %al, 40(%rsi)
+; SCALAR-NEXT:    movb %al, 41(%rsi)
+; SCALAR-NEXT:    movb %al, 42(%rsi)
+; SCALAR-NEXT:    movb %al, 43(%rsi)
+; SCALAR-NEXT:    movb %al, 44(%rsi)
+; SCALAR-NEXT:    movb %al, 45(%rsi)
+; SCALAR-NEXT:    movb %al, 46(%rsi)
+; SCALAR-NEXT:    movb %al, 47(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE2-LABEL: vec384_i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzbl (%rdi), %eax
+; SSE2-NEXT:    notb %al
+; SSE2-NEXT:    movzbl %al, %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, (%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: vec384_i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzbl (%rdi), %eax
+; SSSE3-NEXT:    notb %al
+; SSSE3-NEXT:    movzbl %al, %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    pxor %xmm1, %xmm1
+; SSSE3-NEXT:    pshufb %xmm1, %xmm0
+; SSSE3-NEXT:    movdqa %xmm0, (%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: vec384_i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzbl (%rdi), %eax
+; AVX1-NEXT:    notb %al
+; AVX1-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %eax, %xmm1
+; AVX1-NEXT:    vpshufb %xmm0, %xmm1, %xmm0
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzbl (%rdi), %eax
+; AVX2-NEXT:    notb %al
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec384_i8:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movzbl (%rdi), %eax
+; AVX512F-NEXT:    notb %al
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX512F-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512F-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec384_i8:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movzbl (%rdi), %eax
+; AVX512BW-NEXT:    notb %al
+; AVX512BW-NEXT:    vpbroadcastb %eax, %ymm0
+; AVX512BW-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512BW-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i8, ptr %in.elt.ptr, align 64
   %in.elt = xor i8 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i8, ptr %out.vec.ptr, i64 0
   store i8 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i8, ptr %out.vec.ptr, i64 1
   store i8 %in.elt, ptr %out.elt1.ptr, align 1
   %out.elt2.ptr = getelementptr i8, ptr %out.vec.ptr, i64 2
   store i8 %in.elt, ptr %out.elt2.ptr, align 2
; AVX512BW-NEXT:    retq
   %out.elt46.ptr = getelementptr i8, ptr %out.vec.ptr, i64 46
   store i8 %in.elt, ptr %out.elt46.ptr, align 2
   %out.elt47.ptr = getelementptr i8, ptr %out.vec.ptr, i64 47
   store i8 %in.elt, ptr %out.elt47.ptr, align 1
   ret void
 }
 
 define void @vec384_i16(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_i16:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movw %ax, (%rsi)
-; ALL-NEXT:    movw %ax, 2(%rsi)
-; ALL-NEXT:    movw %ax, 4(%rsi)
-; ALL-NEXT:    movw %ax, 6(%rsi)
-; ALL-NEXT:    movw %ax, 8(%rsi)
-; ALL-NEXT:    movw %ax, 10(%rsi)
-; ALL-NEXT:    movw %ax, 12(%rsi)
-; ALL-NEXT:    movw %ax, 14(%rsi)
-; ALL-NEXT:    movw %ax, 16(%rsi)
-; ALL-NEXT:    movw %ax, 18(%rsi)
-; ALL-NEXT:    movw %ax, 20(%rsi)
-; ALL-NEXT:    movw %ax, 22(%rsi)
-; ALL-NEXT:    movw %ax, 24(%rsi)
-; ALL-NEXT:    movw %ax, 26(%rsi)
-; ALL-NEXT:    movw %ax, 28(%rsi)
-; ALL-NEXT:    movw %ax, 30(%rsi)
-; ALL-NEXT:    movw %ax, 32(%rsi)
-; ALL-NEXT:    movw %ax, 34(%rsi)
-; ALL-NEXT:    movw %ax, 36(%rsi)
-; ALL-NEXT:    movw %ax, 38(%rsi)
-; ALL-NEXT:    movw %ax, 40(%rsi)
-; ALL-NEXT:    movw %ax, 42(%rsi)
-; ALL-NEXT:    movw %ax, 44(%rsi)
-; ALL-NEXT:    movw %ax, 46(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_i16:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movw %ax, (%rsi)
+; SCALAR-NEXT:    movw %ax, 2(%rsi)
+; SCALAR-NEXT:    movw %ax, 4(%rsi)
+; SCALAR-NEXT:    movw %ax, 6(%rsi)
+; SCALAR-NEXT:    movw %ax, 8(%rsi)
+; SCALAR-NEXT:    movw %ax, 10(%rsi)
+; SCALAR-NEXT:    movw %ax, 12(%rsi)
+; SCALAR-NEXT:    movw %ax, 14(%rsi)
+; SCALAR-NEXT:    movw %ax, 16(%rsi)
+; SCALAR-NEXT:    movw %ax, 18(%rsi)
+; SCALAR-NEXT:    movw %ax, 20(%rsi)
+; SCALAR-NEXT:    movw %ax, 22(%rsi)
+; SCALAR-NEXT:    movw %ax, 24(%rsi)
+; SCALAR-NEXT:    movw %ax, 26(%rsi)
+; SCALAR-NEXT:    movw %ax, 28(%rsi)
+; SCALAR-NEXT:    movw %ax, 30(%rsi)
+; SCALAR-NEXT:    movw %ax, 32(%rsi)
+; SCALAR-NEXT:    movw %ax, 34(%rsi)
+; SCALAR-NEXT:    movw %ax, 36(%rsi)
+; SCALAR-NEXT:    movw %ax, 38(%rsi)
+; SCALAR-NEXT:    movw %ax, 40(%rsi)
+; SCALAR-NEXT:    movw %ax, 42(%rsi)
+; SCALAR-NEXT:    movw %ax, 44(%rsi)
+; SCALAR-NEXT:    movw %ax, 46(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec384_i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec384_i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm1
+; AVX1-NEXT:    vmovaps %ymm1, (%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec384_i16:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movl (%rdi), %eax
+; AVX512F-NEXT:    notl %eax
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX512F-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512F-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec384_i16:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movl (%rdi), %eax
+; AVX512BW-NEXT:    notl %eax
+; AVX512BW-NEXT:    vpbroadcastw %eax, %ymm0
+; AVX512BW-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512BW-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i16, ptr %in.elt.ptr, align 64
   %in.elt = xor i16 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i16, ptr %out.vec.ptr, i64 0
   store i16 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i16, ptr %out.vec.ptr, i64 1
   store i16 %in.elt, ptr %out.elt1.ptr, align 2
   %out.elt2.ptr = getelementptr i16, ptr %out.vec.ptr, i64 2
   store i16 %in.elt, ptr %out.elt2.ptr, align 4
; AVX512BW-NEXT:    retq
   %out.elt22.ptr = getelementptr i16, ptr %out.vec.ptr, i64 22
   store i16 %in.elt, ptr %out.elt22.ptr, align 4
   %out.elt23.ptr = getelementptr i16, ptr %out.vec.ptr, i64 23
   store i16 %in.elt, ptr %out.elt23.ptr, align 2
   ret void
 }
 
 define void @vec384_i32(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_i32:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    movl %eax, 32(%rsi)
-; ALL-NEXT:    movl %eax, 36(%rsi)
-; ALL-NEXT:    movl %eax, 40(%rsi)
-; ALL-NEXT:    movl %eax, 44(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_i32:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    movl %eax, 32(%rsi)
+; SCALAR-NEXT:    movl %eax, 36(%rsi)
+; SCALAR-NEXT:    movl %eax, 40(%rsi)
+; SCALAR-NEXT:    movl %eax, 44(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec384_i32:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec384_i32:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_i32:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec384_i32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt = xor i32 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i32, ptr %out.vec.ptr, i64 0
   store i32 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i32, ptr %out.vec.ptr, i64 1
   store i32 %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr i32, ptr %out.vec.ptr, i64 2
   store i32 %in.elt, ptr %out.elt2.ptr, align 8
; AVX512-NEXT:    retq
   %out.elt10.ptr = getelementptr i32, ptr %out.vec.ptr, i64 10
   store i32 %in.elt, ptr %out.elt10.ptr, align 8
   %out.elt11.ptr = getelementptr i32, ptr %out.vec.ptr, i64 11
   store i32 %in.elt, ptr %out.elt11.ptr, align 4
   ret void
 }
 
 define void @vec384_float(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_float:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    movl %eax, 32(%rsi)
-; ALL-NEXT:    movl %eax, 36(%rsi)
-; ALL-NEXT:    movl %eax, 40(%rsi)
-; ALL-NEXT:    movl %eax, 44(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_float:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    movl %eax, 32(%rsi)
+; SCALAR-NEXT:    movl %eax, 36(%rsi)
+; SCALAR-NEXT:    movl %eax, 40(%rsi)
+; SCALAR-NEXT:    movl %eax, 44(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec384_float:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec384_float:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vmovdqa %xmm0, 16(%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rsi)
+; AVX1-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_float:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec384_float:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i32 %in.elt.not, -1
   %in.elt = bitcast i32 %in.elt.int to float
   %out.elt0.ptr = getelementptr float, ptr %out.vec.ptr, i64 0
   store float %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr float, ptr %out.vec.ptr, i64 1
   store float %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr float, ptr %out.vec.ptr, i64 2
; AVX512-NEXT:    retq
   %out.elt10.ptr = getelementptr float, ptr %out.vec.ptr, i64 10
   store float %in.elt, ptr %out.elt10.ptr, align 8
   %out.elt11.ptr = getelementptr float, ptr %out.vec.ptr, i64 11
   store float %in.elt, ptr %out.elt11.ptr, align 4
   ret void
 }
 
 define void @vec384_i64(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_i64:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    movq %rax, 32(%rsi)
-; ALL-NEXT:    movq %rax, 40(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_i64:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    movq %rax, 32(%rsi)
+; SCALAR-NEXT:    movq %rax, 40(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec384_i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec384_i64:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %xmm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_i64:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec384_i64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt = xor i64 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i64, ptr %out.vec.ptr, i64 0
   store i64 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i64, ptr %out.vec.ptr, i64 1
   store i64 %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr i64, ptr %out.vec.ptr, i64 2
   store i64 %in.elt, ptr %out.elt2.ptr, align 16
   %out.elt3.ptr = getelementptr i64, ptr %out.vec.ptr, i64 3
   store i64 %in.elt, ptr %out.elt3.ptr, align 8
   %out.elt4.ptr = getelementptr i64, ptr %out.vec.ptr, i64 4
   store i64 %in.elt, ptr %out.elt4.ptr, align 32
   %out.elt5.ptr = getelementptr i64, ptr %out.vec.ptr, i64 5
   store i64 %in.elt, ptr %out.elt5.ptr, align 8
   ret void
 }
 
 define void @vec384_double(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec384_double:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    movq %rax, 32(%rsi)
-; ALL-NEXT:    movq %rax, 40(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec384_double:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    movq %rax, 32(%rsi)
+; SCALAR-NEXT:    movq %rax, 40(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec384_double:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec384_double:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %xmm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec384_double:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec384_double:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX512-NEXT:    vmovdqa %xmm0, 32(%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i64 %in.elt.not, -1
   %in.elt = bitcast i64 %in.elt.int to double
   %out.elt0.ptr = getelementptr double, ptr %out.vec.ptr, i64 0
   store double %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr double, ptr %out.vec.ptr, i64 1
   store double %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr double, ptr %out.vec.ptr, i64 2
; ALL-NEXT:    retq
   %out.elt1.ptr = getelementptr i128, ptr %out.vec.ptr, i64 1
   store i128 %in.elt, ptr %out.elt1.ptr, align 16
   %out.elt2.ptr = getelementptr i128, ptr %out.vec.ptr, i64 2
   store i128 %in.elt, ptr %out.elt2.ptr, align 32
   ret void
 }
 
 define void @vec512_i8(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_i8:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movzbl (%rdi), %eax
-; ALL-NEXT:    notb %al
-; ALL-NEXT:    movb %al, (%rsi)
-; ALL-NEXT:    movb %al, 1(%rsi)
-; ALL-NEXT:    movb %al, 2(%rsi)
-; ALL-NEXT:    movb %al, 3(%rsi)
-; ALL-NEXT:    movb %al, 4(%rsi)
-; ALL-NEXT:    movb %al, 5(%rsi)
-; ALL-NEXT:    movb %al, 6(%rsi)
-; ALL-NEXT:    movb %al, 7(%rsi)
-; ALL-NEXT:    movb %al, 8(%rsi)
-; ALL-NEXT:    movb %al, 9(%rsi)
-; ALL-NEXT:    movb %al, 10(%rsi)
-; ALL-NEXT:    movb %al, 11(%rsi)
-; ALL-NEXT:    movb %al, 12(%rsi)
-; ALL-NEXT:    movb %al, 13(%rsi)
-; ALL-NEXT:    movb %al, 14(%rsi)
-; ALL-NEXT:    movb %al, 15(%rsi)
-; ALL-NEXT:    movb %al, 16(%rsi)
-; ALL-NEXT:    movb %al, 17(%rsi)
-; ALL-NEXT:    movb %al, 18(%rsi)
-; ALL-NEXT:    movb %al, 19(%rsi)
-; ALL-NEXT:    movb %al, 20(%rsi)
-; ALL-NEXT:    movb %al, 21(%rsi)
-; ALL-NEXT:    movb %al, 22(%rsi)
-; ALL-NEXT:    movb %al, 23(%rsi)
-; ALL-NEXT:    movb %al, 24(%rsi)
-; ALL-NEXT:    movb %al, 25(%rsi)
-; ALL-NEXT:    movb %al, 26(%rsi)
-; ALL-NEXT:    movb %al, 27(%rsi)
-; ALL-NEXT:    movb %al, 28(%rsi)
-; ALL-NEXT:    movb %al, 29(%rsi)
-; ALL-NEXT:    movb %al, 30(%rsi)
-; ALL-NEXT:    movb %al, 31(%rsi)
-; ALL-NEXT:    movb %al, 32(%rsi)
-; ALL-NEXT:    movb %al, 33(%rsi)
-; ALL-NEXT:    movb %al, 34(%rsi)
-; ALL-NEXT:    movb %al, 35(%rsi)
-; ALL-NEXT:    movb %al, 36(%rsi)
-; ALL-NEXT:    movb %al, 37(%rsi)
-; ALL-NEXT:    movb %al, 38(%rsi)
-; ALL-NEXT:    movb %al, 39(%rsi)
-; ALL-NEXT:    movb %al, 40(%rsi)
-; ALL-NEXT:    movb %al, 41(%rsi)
-; ALL-NEXT:    movb %al, 42(%rsi)
-; ALL-NEXT:    movb %al, 43(%rsi)
-; ALL-NEXT:    movb %al, 44(%rsi)
-; ALL-NEXT:    movb %al, 45(%rsi)
-; ALL-NEXT:    movb %al, 46(%rsi)
-; ALL-NEXT:    movb %al, 47(%rsi)
-; ALL-NEXT:    movb %al, 48(%rsi)
-; ALL-NEXT:    movb %al, 49(%rsi)
-; ALL-NEXT:    movb %al, 50(%rsi)
-; ALL-NEXT:    movb %al, 51(%rsi)
-; ALL-NEXT:    movb %al, 52(%rsi)
-; ALL-NEXT:    movb %al, 53(%rsi)
-; ALL-NEXT:    movb %al, 54(%rsi)
-; ALL-NEXT:    movb %al, 55(%rsi)
-; ALL-NEXT:    movb %al, 56(%rsi)
-; ALL-NEXT:    movb %al, 57(%rsi)
-; ALL-NEXT:    movb %al, 58(%rsi)
-; ALL-NEXT:    movb %al, 59(%rsi)
-; ALL-NEXT:    movb %al, 60(%rsi)
-; ALL-NEXT:    movb %al, 61(%rsi)
-; ALL-NEXT:    movb %al, 62(%rsi)
-; ALL-NEXT:    movb %al, 63(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_i8:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movzbl (%rdi), %eax
+; SCALAR-NEXT:    notb %al
+; SCALAR-NEXT:    movb %al, (%rsi)
+; SCALAR-NEXT:    movb %al, 1(%rsi)
+; SCALAR-NEXT:    movb %al, 2(%rsi)
+; SCALAR-NEXT:    movb %al, 3(%rsi)
+; SCALAR-NEXT:    movb %al, 4(%rsi)
+; SCALAR-NEXT:    movb %al, 5(%rsi)
+; SCALAR-NEXT:    movb %al, 6(%rsi)
+; SCALAR-NEXT:    movb %al, 7(%rsi)
+; SCALAR-NEXT:    movb %al, 8(%rsi)
+; SCALAR-NEXT:    movb %al, 9(%rsi)
+; SCALAR-NEXT:    movb %al, 10(%rsi)
+; SCALAR-NEXT:    movb %al, 11(%rsi)
+; SCALAR-NEXT:    movb %al, 12(%rsi)
+; SCALAR-NEXT:    movb %al, 13(%rsi)
+; SCALAR-NEXT:    movb %al, 14(%rsi)
+; SCALAR-NEXT:    movb %al, 15(%rsi)
+; SCALAR-NEXT:    movb %al, 16(%rsi)
+; SCALAR-NEXT:    movb %al, 17(%rsi)
+; SCALAR-NEXT:    movb %al, 18(%rsi)
+; SCALAR-NEXT:    movb %al, 19(%rsi)
+; SCALAR-NEXT:    movb %al, 20(%rsi)
+; SCALAR-NEXT:    movb %al, 21(%rsi)
+; SCALAR-NEXT:    movb %al, 22(%rsi)
+; SCALAR-NEXT:    movb %al, 23(%rsi)
+; SCALAR-NEXT:    movb %al, 24(%rsi)
+; SCALAR-NEXT:    movb %al, 25(%rsi)
+; SCALAR-NEXT:    movb %al, 26(%rsi)
+; SCALAR-NEXT:    movb %al, 27(%rsi)
+; SCALAR-NEXT:    movb %al, 28(%rsi)
+; SCALAR-NEXT:    movb %al, 29(%rsi)
+; SCALAR-NEXT:    movb %al, 30(%rsi)
+; SCALAR-NEXT:    movb %al, 31(%rsi)
+; SCALAR-NEXT:    movb %al, 32(%rsi)
+; SCALAR-NEXT:    movb %al, 33(%rsi)
+; SCALAR-NEXT:    movb %al, 34(%rsi)
+; SCALAR-NEXT:    movb %al, 35(%rsi)
+; SCALAR-NEXT:    movb %al, 36(%rsi)
+; SCALAR-NEXT:    movb %al, 37(%rsi)
+; SCALAR-NEXT:    movb %al, 38(%rsi)
+; SCALAR-NEXT:    movb %al, 39(%rsi)
+; SCALAR-NEXT:    movb %al, 40(%rsi)
+; SCALAR-NEXT:    movb %al, 41(%rsi)
+; SCALAR-NEXT:    movb %al, 42(%rsi)
+; SCALAR-NEXT:    movb %al, 43(%rsi)
+; SCALAR-NEXT:    movb %al, 44(%rsi)
+; SCALAR-NEXT:    movb %al, 45(%rsi)
+; SCALAR-NEXT:    movb %al, 46(%rsi)
+; SCALAR-NEXT:    movb %al, 47(%rsi)
+; SCALAR-NEXT:    movb %al, 48(%rsi)
+; SCALAR-NEXT:    movb %al, 49(%rsi)
+; SCALAR-NEXT:    movb %al, 50(%rsi)
+; SCALAR-NEXT:    movb %al, 51(%rsi)
+; SCALAR-NEXT:    movb %al, 52(%rsi)
+; SCALAR-NEXT:    movb %al, 53(%rsi)
+; SCALAR-NEXT:    movb %al, 54(%rsi)
+; SCALAR-NEXT:    movb %al, 55(%rsi)
+; SCALAR-NEXT:    movb %al, 56(%rsi)
+; SCALAR-NEXT:    movb %al, 57(%rsi)
+; SCALAR-NEXT:    movb %al, 58(%rsi)
+; SCALAR-NEXT:    movb %al, 59(%rsi)
+; SCALAR-NEXT:    movb %al, 60(%rsi)
+; SCALAR-NEXT:    movb %al, 61(%rsi)
+; SCALAR-NEXT:    movb %al, 62(%rsi)
+; SCALAR-NEXT:    movb %al, 63(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE2-LABEL: vec512_i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzbl (%rdi), %eax
+; SSE2-NEXT:    notb %al
+; SSE2-NEXT:    movzbl %al, %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, (%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE2-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: vec512_i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzbl (%rdi), %eax
+; SSSE3-NEXT:    notb %al
+; SSSE3-NEXT:    movzbl %al, %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    pxor %xmm1, %xmm1
+; SSSE3-NEXT:    pshufb %xmm1, %xmm0
+; SSSE3-NEXT:    movdqa %xmm0, (%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSSE3-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSSE3-NEXT:    retq
+;
+; AVX1-LABEL: vec512_i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzbl (%rdi), %eax
+; AVX1-NEXT:    notb %al
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX1-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzbl (%rdi), %eax
+; AVX2-NEXT:    notb %al
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec512_i8:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movzbl (%rdi), %eax
+; AVX512F-NEXT:    notb %al
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX512F-NEXT:    vinserti64x4 $1, %ymm0, %zmm0, %zmm0
+; AVX512F-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec512_i8:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movzbl (%rdi), %eax
+; AVX512BW-NEXT:    notb %al
+; AVX512BW-NEXT:    vpbroadcastb %eax, %zmm0
+; AVX512BW-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i8, ptr %in.elt.ptr, align 64
   %in.elt = xor i8 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i8, ptr %out.vec.ptr, i64 0
   store i8 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i8, ptr %out.vec.ptr, i64 1
   store i8 %in.elt, ptr %out.elt1.ptr, align 1
   %out.elt2.ptr = getelementptr i8, ptr %out.vec.ptr, i64 2
   store i8 %in.elt, ptr %out.elt2.ptr, align 2
; AVX512BW-NEXT:    retq
   %out.elt62.ptr = getelementptr i8, ptr %out.vec.ptr, i64 62
   store i8 %in.elt, ptr %out.elt62.ptr, align 2
   %out.elt63.ptr = getelementptr i8, ptr %out.vec.ptr, i64 63
   store i8 %in.elt, ptr %out.elt63.ptr, align 1
   ret void
 }
 
 define void @vec512_i16(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_i16:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movw %ax, (%rsi)
-; ALL-NEXT:    movw %ax, 2(%rsi)
-; ALL-NEXT:    movw %ax, 4(%rsi)
-; ALL-NEXT:    movw %ax, 6(%rsi)
-; ALL-NEXT:    movw %ax, 8(%rsi)
-; ALL-NEXT:    movw %ax, 10(%rsi)
-; ALL-NEXT:    movw %ax, 12(%rsi)
-; ALL-NEXT:    movw %ax, 14(%rsi)
-; ALL-NEXT:    movw %ax, 16(%rsi)
-; ALL-NEXT:    movw %ax, 18(%rsi)
-; ALL-NEXT:    movw %ax, 20(%rsi)
-; ALL-NEXT:    movw %ax, 22(%rsi)
-; ALL-NEXT:    movw %ax, 24(%rsi)
-; ALL-NEXT:    movw %ax, 26(%rsi)
-; ALL-NEXT:    movw %ax, 28(%rsi)
-; ALL-NEXT:    movw %ax, 30(%rsi)
-; ALL-NEXT:    movw %ax, 32(%rsi)
-; ALL-NEXT:    movw %ax, 34(%rsi)
-; ALL-NEXT:    movw %ax, 36(%rsi)
-; ALL-NEXT:    movw %ax, 38(%rsi)
-; ALL-NEXT:    movw %ax, 40(%rsi)
-; ALL-NEXT:    movw %ax, 42(%rsi)
-; ALL-NEXT:    movw %ax, 44(%rsi)
-; ALL-NEXT:    movw %ax, 46(%rsi)
-; ALL-NEXT:    movw %ax, 48(%rsi)
-; ALL-NEXT:    movw %ax, 50(%rsi)
-; ALL-NEXT:    movw %ax, 52(%rsi)
-; ALL-NEXT:    movw %ax, 54(%rsi)
-; ALL-NEXT:    movw %ax, 56(%rsi)
-; ALL-NEXT:    movw %ax, 58(%rsi)
-; ALL-NEXT:    movw %ax, 60(%rsi)
-; ALL-NEXT:    movw %ax, 62(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_i16:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movw %ax, (%rsi)
+; SCALAR-NEXT:    movw %ax, 2(%rsi)
+; SCALAR-NEXT:    movw %ax, 4(%rsi)
+; SCALAR-NEXT:    movw %ax, 6(%rsi)
+; SCALAR-NEXT:    movw %ax, 8(%rsi)
+; SCALAR-NEXT:    movw %ax, 10(%rsi)
+; SCALAR-NEXT:    movw %ax, 12(%rsi)
+; SCALAR-NEXT:    movw %ax, 14(%rsi)
+; SCALAR-NEXT:    movw %ax, 16(%rsi)
+; SCALAR-NEXT:    movw %ax, 18(%rsi)
+; SCALAR-NEXT:    movw %ax, 20(%rsi)
+; SCALAR-NEXT:    movw %ax, 22(%rsi)
+; SCALAR-NEXT:    movw %ax, 24(%rsi)
+; SCALAR-NEXT:    movw %ax, 26(%rsi)
+; SCALAR-NEXT:    movw %ax, 28(%rsi)
+; SCALAR-NEXT:    movw %ax, 30(%rsi)
+; SCALAR-NEXT:    movw %ax, 32(%rsi)
+; SCALAR-NEXT:    movw %ax, 34(%rsi)
+; SCALAR-NEXT:    movw %ax, 36(%rsi)
+; SCALAR-NEXT:    movw %ax, 38(%rsi)
+; SCALAR-NEXT:    movw %ax, 40(%rsi)
+; SCALAR-NEXT:    movw %ax, 42(%rsi)
+; SCALAR-NEXT:    movw %ax, 44(%rsi)
+; SCALAR-NEXT:    movw %ax, 46(%rsi)
+; SCALAR-NEXT:    movw %ax, 48(%rsi)
+; SCALAR-NEXT:    movw %ax, 50(%rsi)
+; SCALAR-NEXT:    movw %ax, 52(%rsi)
+; SCALAR-NEXT:    movw %ax, 54(%rsi)
+; SCALAR-NEXT:    movw %ax, 56(%rsi)
+; SCALAR-NEXT:    movw %ax, 58(%rsi)
+; SCALAR-NEXT:    movw %ax, 60(%rsi)
+; SCALAR-NEXT:    movw %ax, 62(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec512_i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec512_i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512F-LABEL: vec512_i16:
+; AVX512F:       # %bb.0:
+; AVX512F-NEXT:    movl (%rdi), %eax
+; AVX512F-NEXT:    notl %eax
+; AVX512F-NEXT:    vmovd %eax, %xmm0
+; AVX512F-NEXT:    vpbroadcastw %xmm0, %ymm0
+; AVX512F-NEXT:    vinserti64x4 $1, %ymm0, %zmm0, %zmm0
+; AVX512F-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512F-NEXT:    vzeroupper
+; AVX512F-NEXT:    retq
+;
+; AVX512BW-LABEL: vec512_i16:
+; AVX512BW:       # %bb.0:
+; AVX512BW-NEXT:    movl (%rdi), %eax
+; AVX512BW-NEXT:    notl %eax
+; AVX512BW-NEXT:    vpbroadcastw %eax, %zmm0
+; AVX512BW-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512BW-NEXT:    vzeroupper
+; AVX512BW-NEXT:    retq
   %in.elt.not = load i16, ptr %in.elt.ptr, align 64
   %in.elt = xor i16 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i16, ptr %out.vec.ptr, i64 0
   store i16 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i16, ptr %out.vec.ptr, i64 1
   store i16 %in.elt, ptr %out.elt1.ptr, align 2
   %out.elt2.ptr = getelementptr i16, ptr %out.vec.ptr, i64 2
   store i16 %in.elt, ptr %out.elt2.ptr, align 4
; AVX512BW-NEXT:    retq
   %out.elt30.ptr = getelementptr i16, ptr %out.vec.ptr, i64 30
   store i16 %in.elt, ptr %out.elt30.ptr, align 4
   %out.elt31.ptr = getelementptr i16, ptr %out.vec.ptr, i64 31
   store i16 %in.elt, ptr %out.elt31.ptr, align 2
   ret void
 }
 
 define void @vec512_i32(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_i32:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    movl %eax, 32(%rsi)
-; ALL-NEXT:    movl %eax, 36(%rsi)
-; ALL-NEXT:    movl %eax, 40(%rsi)
-; ALL-NEXT:    movl %eax, 44(%rsi)
-; ALL-NEXT:    movl %eax, 48(%rsi)
-; ALL-NEXT:    movl %eax, 52(%rsi)
-; ALL-NEXT:    movl %eax, 56(%rsi)
-; ALL-NEXT:    movl %eax, 60(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_i32:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    movl %eax, 32(%rsi)
+; SCALAR-NEXT:    movl %eax, 36(%rsi)
+; SCALAR-NEXT:    movl %eax, 40(%rsi)
+; SCALAR-NEXT:    movl %eax, 44(%rsi)
+; SCALAR-NEXT:    movl %eax, 48(%rsi)
+; SCALAR-NEXT:    movl %eax, 52(%rsi)
+; SCALAR-NEXT:    movl %eax, 56(%rsi)
+; SCALAR-NEXT:    movl %eax, 60(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec512_i32:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec512_i32:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_i32:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec512_i32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %zmm0
+; AVX512-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt = xor i32 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i32, ptr %out.vec.ptr, i64 0
   store i32 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i32, ptr %out.vec.ptr, i64 1
   store i32 %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr i32, ptr %out.vec.ptr, i64 2
   store i32 %in.elt, ptr %out.elt2.ptr, align 8
; AVX512-NEXT:    retq
   %out.elt14.ptr = getelementptr i32, ptr %out.vec.ptr, i64 14
   store i32 %in.elt, ptr %out.elt14.ptr, align 8
   %out.elt15.ptr = getelementptr i32, ptr %out.vec.ptr, i64 15
   store i32 %in.elt, ptr %out.elt15.ptr, align 4
   ret void
 }
 
 define void @vec512_float(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_float:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movl (%rdi), %eax
-; ALL-NEXT:    notl %eax
-; ALL-NEXT:    movl %eax, (%rsi)
-; ALL-NEXT:    movl %eax, 4(%rsi)
-; ALL-NEXT:    movl %eax, 8(%rsi)
-; ALL-NEXT:    movl %eax, 12(%rsi)
-; ALL-NEXT:    movl %eax, 16(%rsi)
-; ALL-NEXT:    movl %eax, 20(%rsi)
-; ALL-NEXT:    movl %eax, 24(%rsi)
-; ALL-NEXT:    movl %eax, 28(%rsi)
-; ALL-NEXT:    movl %eax, 32(%rsi)
-; ALL-NEXT:    movl %eax, 36(%rsi)
-; ALL-NEXT:    movl %eax, 40(%rsi)
-; ALL-NEXT:    movl %eax, 44(%rsi)
-; ALL-NEXT:    movl %eax, 48(%rsi)
-; ALL-NEXT:    movl %eax, 52(%rsi)
-; ALL-NEXT:    movl %eax, 56(%rsi)
-; ALL-NEXT:    movl %eax, 60(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_float:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movl (%rdi), %eax
+; SCALAR-NEXT:    notl %eax
+; SCALAR-NEXT:    movl %eax, (%rsi)
+; SCALAR-NEXT:    movl %eax, 4(%rsi)
+; SCALAR-NEXT:    movl %eax, 8(%rsi)
+; SCALAR-NEXT:    movl %eax, 12(%rsi)
+; SCALAR-NEXT:    movl %eax, 16(%rsi)
+; SCALAR-NEXT:    movl %eax, 20(%rsi)
+; SCALAR-NEXT:    movl %eax, 24(%rsi)
+; SCALAR-NEXT:    movl %eax, 28(%rsi)
+; SCALAR-NEXT:    movl %eax, 32(%rsi)
+; SCALAR-NEXT:    movl %eax, 36(%rsi)
+; SCALAR-NEXT:    movl %eax, 40(%rsi)
+; SCALAR-NEXT:    movl %eax, 44(%rsi)
+; SCALAR-NEXT:    movl %eax, 48(%rsi)
+; SCALAR-NEXT:    movl %eax, 52(%rsi)
+; SCALAR-NEXT:    movl %eax, 56(%rsi)
+; SCALAR-NEXT:    movl %eax, 60(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec512_float:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movl (%rdi), %eax
+; SSE-NEXT:    notl %eax
+; SSE-NEXT:    movd %eax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec512_float:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movl (%rdi), %eax
+; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_float:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movl (%rdi), %eax
+; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    vpbroadcastd %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec512_float:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movl (%rdi), %eax
+; AVX512-NEXT:    notl %eax
+; AVX512-NEXT:    vpbroadcastd %eax, %zmm0
+; AVX512-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i32, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i32 %in.elt.not, -1
   %in.elt = bitcast i32 %in.elt.int to float
   %out.elt0.ptr = getelementptr float, ptr %out.vec.ptr, i64 0
   store float %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr float, ptr %out.vec.ptr, i64 1
   store float %in.elt, ptr %out.elt1.ptr, align 4
   %out.elt2.ptr = getelementptr float, ptr %out.vec.ptr, i64 2
; AVX512-NEXT:    retq
   %out.elt14.ptr = getelementptr float, ptr %out.vec.ptr, i64 14
   store float %in.elt, ptr %out.elt14.ptr, align 8
   %out.elt15.ptr = getelementptr float, ptr %out.vec.ptr, i64 15
   store float %in.elt, ptr %out.elt15.ptr, align 4
   ret void
 }
 
 define void @vec512_i64(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_i64:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    movq %rax, 32(%rsi)
-; ALL-NEXT:    movq %rax, 40(%rsi)
-; ALL-NEXT:    movq %rax, 48(%rsi)
-; ALL-NEXT:    movq %rax, 56(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_i64:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    movq %rax, 32(%rsi)
+; SCALAR-NEXT:    movq %rax, 40(%rsi)
+; SCALAR-NEXT:    movq %rax, 48(%rsi)
+; SCALAR-NEXT:    movq %rax, 56(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec512_i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec512_i64:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_i64:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec512_i64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %zmm0
+; AVX512-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt = xor i64 %in.elt.not, -1
   %out.elt0.ptr = getelementptr i64, ptr %out.vec.ptr, i64 0
   store i64 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i64, ptr %out.vec.ptr, i64 1
   store i64 %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr i64, ptr %out.vec.ptr, i64 2
   store i64 %in.elt, ptr %out.elt2.ptr, align 16
   %out.elt3.ptr = getelementptr i64, ptr %out.vec.ptr, i64 3
   store i64 %in.elt, ptr %out.elt3.ptr, align 8
   %out.elt4.ptr = getelementptr i64, ptr %out.vec.ptr, i64 4
   store i64 %in.elt, ptr %out.elt4.ptr, align 32
   %out.elt5.ptr = getelementptr i64, ptr %out.vec.ptr, i64 5
   store i64 %in.elt, ptr %out.elt5.ptr, align 8
   %out.elt6.ptr = getelementptr i64, ptr %out.vec.ptr, i64 6
   store i64 %in.elt, ptr %out.elt6.ptr, align 16
   %out.elt7.ptr = getelementptr i64, ptr %out.vec.ptr, i64 7
   store i64 %in.elt, ptr %out.elt7.ptr, align 8
   ret void
 }
 
 define void @vec512_double(ptr %in.elt.ptr, ptr %out.vec.ptr) nounwind {
-; ALL-LABEL: vec512_double:
-; ALL:       # %bb.0:
-; ALL-NEXT:    movq (%rdi), %rax
-; ALL-NEXT:    notq %rax
-; ALL-NEXT:    movq %rax, (%rsi)
-; ALL-NEXT:    movq %rax, 8(%rsi)
-; ALL-NEXT:    movq %rax, 16(%rsi)
-; ALL-NEXT:    movq %rax, 24(%rsi)
-; ALL-NEXT:    movq %rax, 32(%rsi)
-; ALL-NEXT:    movq %rax, 40(%rsi)
-; ALL-NEXT:    movq %rax, 48(%rsi)
-; ALL-NEXT:    movq %rax, 56(%rsi)
-; ALL-NEXT:    retq
+; SCALAR-LABEL: vec512_double:
+; SCALAR:       # %bb.0:
+; SCALAR-NEXT:    movq (%rdi), %rax
+; SCALAR-NEXT:    notq %rax
+; SCALAR-NEXT:    movq %rax, (%rsi)
+; SCALAR-NEXT:    movq %rax, 8(%rsi)
+; SCALAR-NEXT:    movq %rax, 16(%rsi)
+; SCALAR-NEXT:    movq %rax, 24(%rsi)
+; SCALAR-NEXT:    movq %rax, 32(%rsi)
+; SCALAR-NEXT:    movq %rax, 40(%rsi)
+; SCALAR-NEXT:    movq %rax, 48(%rsi)
+; SCALAR-NEXT:    movq %rax, 56(%rsi)
+; SCALAR-NEXT:    retq
+;
+; SSE-LABEL: vec512_double:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq (%rdi), %rax
+; SSE-NEXT:    notq %rax
+; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE-NEXT:    movdqa %xmm0, (%rsi)
+; SSE-NEXT:    movdqa %xmm0, 16(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 32(%rsi)
+; SSE-NEXT:    movdqa %xmm0, 48(%rsi)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: vec512_double:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movq (%rdi), %rax
+; AVX1-NEXT:    notq %rax
+; AVX1-NEXT:    vmovq %rax, %xmm0
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX1-NEXT:    vmovaps %ymm0, (%rsi)
+; AVX1-NEXT:    vmovaps %ymm0, 32(%rsi)
+; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: vec512_double:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movq (%rdi), %rax
+; AVX2-NEXT:    notq %rax
+; AVX2-NEXT:    vmovq %rax, %xmm0
+; AVX2-NEXT:    vpbroadcastq %xmm0, %ymm0
+; AVX2-NEXT:    vmovdqa %ymm0, (%rsi)
+; AVX2-NEXT:    vmovdqa %ymm0, 32(%rsi)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: vec512_double:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movq (%rdi), %rax
+; AVX512-NEXT:    notq %rax
+; AVX512-NEXT:    vpbroadcastq %rax, %zmm0
+; AVX512-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %in.elt.not = load i64, ptr %in.elt.ptr, align 64
   %in.elt.int = xor i64 %in.elt.not, -1
   %in.elt = bitcast i64 %in.elt.int to double
   %out.elt0.ptr = getelementptr double, ptr %out.vec.ptr, i64 0
   store double %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr double, ptr %out.vec.ptr, i64 1
   store double %in.elt, ptr %out.elt1.ptr, align 8
   %out.elt2.ptr = getelementptr double, ptr %out.vec.ptr, i64 2
; ALL-NEXT:    retq
   %out.elt0.ptr = getelementptr i256, ptr %out.vec.ptr, i64 0
   store i256 %in.elt, ptr %out.elt0.ptr, align 64
   %out.elt1.ptr = getelementptr i256, ptr %out.vec.ptr, i64 1
   store i256 %in.elt, ptr %out.elt1.ptr, align 32
   ret void
 }
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
 ; AVX: {{.*}}
-; AVX1: {{.*}}
-; AVX2: {{.*}}
-; AVX512: {{.*}}
-; AVX512BW: {{.*}}
-; AVX512F: {{.*}}
-; SCALAR: {{.*}}
-; SSE: {{.*}}
-; SSE2: {{.*}}
 ; SSE2-ONLY: {{.*}}
 ; SSE3: {{.*}}
 ; SSE41: {{.*}}
 ; SSE42: {{.*}}
-; SSSE3: {{.*}}
 ; SSSE3-ONLY: {{.*}}

llvm/test/CodeGen/X86/legalize-shl-vec.ll

 ; X64-LABEL: test_sra:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
 ; X64-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
 ; X64-NEXT:    shrdq $6, %rsi, %r9
 ; X64-NEXT:    shrdq $6, %rcx, %rsi
-; X64-NEXT:    sarq $63, %r8
 ; X64-NEXT:    shrdq $6, %rdx, %rcx
 ; X64-NEXT:    sarq $6, %rdx
 ; X64-NEXT:    movq %rdx, 56(%rdi)
 ; X64-NEXT:    movq %rcx, 48(%rdi)
 ; X64-NEXT:    movq %rsi, 40(%rdi)
 ; X64-NEXT:    movq %r9, 32(%rdi)
-; X64-NEXT:    movq %r8, 24(%rdi)
-; X64-NEXT:    movq %r8, 16(%rdi)
-; X64-NEXT:    movq %r8, 8(%rdi)
-; X64-NEXT:    movq %r8, (%rdi)
+; X64-NEXT:    movq %r8, %xmm0
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; X64-NEXT:    psrad $31, %xmm0
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
+; X64-NEXT:    movdqa %xmm0, 16(%rdi)
+; X64-NEXT:    movdqa %xmm0, (%rdi)
 ; X64-NEXT:    retq
   %Amt = insertelement <2 x i256> <i256 5, i256 6>, i256 255, i32 0
   %Out = ashr <2 x i256> %In, %Amt
   ret <2 x i256> %Out
 }

llvm/test/CodeGen/X86/wide-scalar-shift-by-byte-multiple-legalization.ll

 ; X32-SSE42-NEXT:    movl 4(%edx), %edi
 ; X32-SSE42-NEXT:    movl 8(%edx), %ebx
 ; X32-SSE42-NEXT:    movl 12(%edx), %edx
 ; X32-SSE42-NEXT:    movzbl (%ecx), %ecx
 ; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %edi, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %esi, (%esp)
-; X32-SSE42-NEXT:    sarl $31, %edx
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movd %edx, %xmm0
+; X32-SSE42-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-SSE42-NEXT:    psrad $31, %xmm0
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    andl $15, %ecx
 ; X32-SSE42-NEXT:    movups (%esp,%ecx), %xmm0
 ; X32-SSE42-NEXT:    movups %xmm0, (%eax)
 ; X32-SSE42-NEXT:    addl $32, %esp
 ; X32-SSE42-NEXT:    popl %esi
 ; X32-SSE42-NEXT:    popl %edi
 ; X32-SSE42-NEXT:    popl %ebx
 ; X32-SSE42-NEXT:    retl
 ;
-; X32-AVX-LABEL: ashr_16bytes:
-; X32-AVX:       # %bb.0:
-; X32-AVX-NEXT:    pushl %ebx
-; X32-AVX-NEXT:    pushl %edi
-; X32-AVX-NEXT:    pushl %esi
-; X32-AVX-NEXT:    subl $32, %esp
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X32-AVX-NEXT:    movl (%edx), %esi
-; X32-AVX-NEXT:    movl 4(%edx), %edi
-; X32-AVX-NEXT:    movl 8(%edx), %ebx
-; X32-AVX-NEXT:    movl 12(%edx), %edx
-; X32-AVX-NEXT:    movzbl (%ecx), %ecx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %esi, (%esp)
-; X32-AVX-NEXT:    sarl $31, %edx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    andl $15, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
-; X32-AVX-NEXT:    addl $32, %esp
-; X32-AVX-NEXT:    popl %esi
-; X32-AVX-NEXT:    popl %edi
-; X32-AVX-NEXT:    popl %ebx
-; X32-AVX-NEXT:    retl
+; X32-AVX1-LABEL: ashr_16bytes:
+; X32-AVX1:       # %bb.0:
+; X32-AVX1-NEXT:    pushl %ebx
+; X32-AVX1-NEXT:    pushl %edi
+; X32-AVX1-NEXT:    pushl %esi
+; X32-AVX1-NEXT:    subl $32, %esp
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX1-NEXT:    movl (%edx), %esi
+; X32-AVX1-NEXT:    movl 4(%edx), %edi
+; X32-AVX1-NEXT:    movl 8(%edx), %ebx
+; X32-AVX1-NEXT:    movl 12(%edx), %edx
+; X32-AVX1-NEXT:    movzbl (%ecx), %ecx
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %esi, (%esp)
+; X32-AVX1-NEXT:    vmovd %edx, %xmm0
+; X32-AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-AVX1-NEXT:    vpsrad $31, %xmm0, %xmm0
+; X32-AVX1-NEXT:    vmovdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    andl $15, %ecx
+; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX1-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX1-NEXT:    addl $32, %esp
+; X32-AVX1-NEXT:    popl %esi
+; X32-AVX1-NEXT:    popl %edi
+; X32-AVX1-NEXT:    popl %ebx
+; X32-AVX1-NEXT:    retl
+;
+; X32-AVX512-LABEL: ashr_16bytes:
+; X32-AVX512:       # %bb.0:
+; X32-AVX512-NEXT:    pushl %ebx
+; X32-AVX512-NEXT:    pushl %edi
+; X32-AVX512-NEXT:    pushl %esi
+; X32-AVX512-NEXT:    subl $32, %esp
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX512-NEXT:    movl (%edx), %esi
+; X32-AVX512-NEXT:    movl 4(%edx), %edi
+; X32-AVX512-NEXT:    movl 8(%edx), %ebx
+; X32-AVX512-NEXT:    movl 12(%edx), %edx
+; X32-AVX512-NEXT:    movzbl (%ecx), %ecx
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %esi, (%esp)
+; X32-AVX512-NEXT:    sarl $31, %edx
+; X32-AVX512-NEXT:    vpbroadcastd %edx, %xmm0
+; X32-AVX512-NEXT:    vmovdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    andl $15, %ecx
+; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX512-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX512-NEXT:    addl $32, %esp
+; X32-AVX512-NEXT:    popl %esi
+; X32-AVX512-NEXT:    popl %edi
+; X32-AVX512-NEXT:    popl %ebx
+; X32-AVX512-NEXT:    retl
   %src = load i128, ptr %src.ptr, align 1
   %byteOff = load i128, ptr %byteOff.ptr, align 1
   %bitOff = shl i128 %byteOff, 3
   %res = ashr i128 %src, %bitOff
   store i128 %res, ptr %dst, align 1
   ret void
 }
 

 ; X64-SSE42:       # %bb.0:
 ; X64-SSE42-NEXT:    movups (%rdi), %xmm0
 ; X64-SSE42-NEXT:    movq 16(%rdi), %rax
 ; X64-SSE42-NEXT:    movq 24(%rdi), %rcx
 ; X64-SSE42-NEXT:    movzbl (%rsi), %esi
 ; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    movups %xmm0, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    sarq $63, %rcx
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movq %rcx, %xmm0
+; X64-SSE42-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; X64-SSE42-NEXT:    pxor %xmm1, %xmm1
+; X64-SSE42-NEXT:    pcmpgtq %xmm0, %xmm1
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    andl $31, %esi
 ; X64-SSE42-NEXT:    movups -64(%rsp,%rsi), %xmm0
 ; X64-SSE42-NEXT:    movups -48(%rsp,%rsi), %xmm1
-; X64-SSE42-NEXT:    movups %xmm1, 16(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm0, (%rdx)
+; X64-SSE42-NEXT:    movups %xmm1, 16(%rdx)
 ; X64-SSE42-NEXT:    retq
 ;
-; X64-AVX-LABEL: ashr_32bytes:
-; X64-AVX:       # %bb.0:
-; X64-AVX-NEXT:    vmovups (%rdi), %xmm0
-; X64-AVX-NEXT:    movq 16(%rdi), %rax
-; X64-AVX-NEXT:    movq 24(%rdi), %rcx
-; X64-AVX-NEXT:    movzbl (%rsi), %esi
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    vmovups %xmm0, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    sarq $63, %rcx
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    andl $31, %esi
-; X64-AVX-NEXT:    vmovups -64(%rsp,%rsi), %xmm0
-; X64-AVX-NEXT:    vmovups -48(%rsp,%rsi), %xmm1
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
-; X64-AVX-NEXT:    retq
+; X64-AVX1-LABEL: ashr_32bytes:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovups (%rdi), %xmm0
+; X64-AVX1-NEXT:    movq 16(%rdi), %rax
+; X64-AVX1-NEXT:    movq 24(%rdi), %rcx
+; X64-AVX1-NEXT:    movzbl (%rsi), %esi
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovups %xmm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovq %rcx, %xmm0
+; X64-AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; X64-AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
+; X64-AVX1-NEXT:    vmovdqu %xmm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovdqu %xmm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    andl $31, %esi
+; X64-AVX1-NEXT:    vmovups -64(%rsp,%rsi), %xmm0
+; X64-AVX1-NEXT:    vmovups -48(%rsp,%rsi), %xmm1
+; X64-AVX1-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX1-NEXT:    vmovups %xmm1, 16(%rdx)
+; X64-AVX1-NEXT:    retq
+;
+; X64-AVX512-LABEL: ashr_32bytes:
+; X64-AVX512:       # %bb.0:
+; X64-AVX512-NEXT:    vmovups (%rdi), %xmm0
+; X64-AVX512-NEXT:    movq 16(%rdi), %rax
+; X64-AVX512-NEXT:    movq 24(%rdi), %rcx
+; X64-AVX512-NEXT:    movzbl (%rsi), %esi
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    vmovups %xmm0, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    sarq $63, %rcx
+; X64-AVX512-NEXT:    vpbroadcastq %rcx, %ymm0
+; X64-AVX512-NEXT:    vmovdqu %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    andl $31, %esi
+; X64-AVX512-NEXT:    vmovups -64(%rsp,%rsi), %xmm0
+; X64-AVX512-NEXT:    vmovups -48(%rsp,%rsi), %xmm1
+; X64-AVX512-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX512-NEXT:    vmovups %xmm1, 16(%rdx)
+; X64-AVX512-NEXT:    vzeroupper
+; X64-AVX512-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: ashr_32bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi
 ; X32-SSE2-NEXT:    pushl %esi
 ; X32-SSE2-NEXT:    subl $72, %esp

 ; X32-SSE42-NEXT:    movl 24(%edx), %ebx
 ; X32-SSE42-NEXT:    movl 28(%edx), %edx
 ; X32-SSE42-NEXT:    movzbl (%ecx), %ecx
 ; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %edi, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %esi, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movups %xmm0, (%esp)
-; X32-SSE42-NEXT:    sarl $31, %edx
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movd %edx, %xmm0
+; X32-SSE42-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-SSE42-NEXT:    psrad $31, %xmm0
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    andl $31, %ecx
 ; X32-SSE42-NEXT:    movups (%esp,%ecx), %xmm0
 ; X32-SSE42-NEXT:    movups 16(%esp,%ecx), %xmm1
-; X32-SSE42-NEXT:    movups %xmm1, 16(%eax)
 ; X32-SSE42-NEXT:    movups %xmm0, (%eax)
+; X32-SSE42-NEXT:    movups %xmm1, 16(%eax)
 ; X32-SSE42-NEXT:    addl $64, %esp
 ; X32-SSE42-NEXT:    popl %esi
 ; X32-SSE42-NEXT:    popl %edi
 ; X32-SSE42-NEXT:    popl %ebx
 ; X32-SSE42-NEXT:    retl
 ;
-; X32-AVX-LABEL: ashr_32bytes:
-; X32-AVX:       # %bb.0:
-; X32-AVX-NEXT:    pushl %ebx
-; X32-AVX-NEXT:    pushl %edi
-; X32-AVX-NEXT:    pushl %esi
-; X32-AVX-NEXT:    subl $64, %esp
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X32-AVX-NEXT:    vmovups (%edx), %xmm0
-; X32-AVX-NEXT:    movl 16(%edx), %esi
-; X32-AVX-NEXT:    movl 20(%edx), %edi
-; X32-AVX-NEXT:    movl 24(%edx), %ebx
-; X32-AVX-NEXT:    movl 28(%edx), %edx
-; X32-AVX-NEXT:    movzbl (%ecx), %ecx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %esi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    vmovups %xmm0, (%esp)
-; X32-AVX-NEXT:    sarl $31, %edx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    andl $31, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
-; X32-AVX-NEXT:    addl $64, %esp
-; X32-AVX-NEXT:    popl %esi
-; X32-AVX-NEXT:    popl %edi
-; X32-AVX-NEXT:    popl %ebx
-; X32-AVX-NEXT:    retl
+; X32-AVX1-LABEL: ashr_32bytes:
+; X32-AVX1:       # %bb.0:
+; X32-AVX1-NEXT:    pushl %ebx
+; X32-AVX1-NEXT:    pushl %edi
+; X32-AVX1-NEXT:    pushl %esi
+; X32-AVX1-NEXT:    subl $64, %esp
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX1-NEXT:    vmovups (%edx), %xmm0
+; X32-AVX1-NEXT:    movl 16(%edx), %esi
+; X32-AVX1-NEXT:    movl 20(%edx), %edi
+; X32-AVX1-NEXT:    movl 24(%edx), %ebx
+; X32-AVX1-NEXT:    movl 28(%edx), %edx
+; X32-AVX1-NEXT:    movzbl (%ecx), %ecx
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %xmm0, (%esp)
+; X32-AVX1-NEXT:    vmovd %edx, %xmm0
+; X32-AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-AVX1-NEXT:    vpsrad $31, %xmm0, %xmm0
+; X32-AVX1-NEXT:    vmovdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    andl $31, %ecx
+; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX1-NEXT:    vmovups 16(%esp,%ecx), %xmm1
+; X32-AVX1-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX1-NEXT:    vmovups %xmm1, 16(%eax)
+; X32-AVX1-NEXT:    addl $64, %esp
+; X32-AVX1-NEXT:    popl %esi
+; X32-AVX1-NEXT:    popl %edi
+; X32-AVX1-NEXT:    popl %ebx
+; X32-AVX1-NEXT:    retl
+;
+; X32-AVX512-LABEL: ashr_32bytes:
+; X32-AVX512:       # %bb.0:
+; X32-AVX512-NEXT:    pushl %ebx
+; X32-AVX512-NEXT:    pushl %edi
+; X32-AVX512-NEXT:    pushl %esi
+; X32-AVX512-NEXT:    subl $64, %esp
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX512-NEXT:    vmovups (%edx), %xmm0
+; X32-AVX512-NEXT:    movl 16(%edx), %esi
+; X32-AVX512-NEXT:    movl 20(%edx), %edi
+; X32-AVX512-NEXT:    movl 24(%edx), %ebx
+; X32-AVX512-NEXT:    movl 28(%edx), %edx
+; X32-AVX512-NEXT:    movzbl (%ecx), %ecx
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    vmovups %xmm0, (%esp)
+; X32-AVX512-NEXT:    sarl $31, %edx
+; X32-AVX512-NEXT:    vpbroadcastd %edx, %ymm0
+; X32-AVX512-NEXT:    vmovdqu %ymm0, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    andl $31, %ecx
+; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX512-NEXT:    vmovups 16(%esp,%ecx), %xmm1
+; X32-AVX512-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX512-NEXT:    vmovups %xmm1, 16(%eax)
+; X32-AVX512-NEXT:    addl $64, %esp
+; X32-AVX512-NEXT:    popl %esi
+; X32-AVX512-NEXT:    popl %edi
+; X32-AVX512-NEXT:    popl %ebx
+; X32-AVX512-NEXT:    vzeroupper
+; X32-AVX512-NEXT:    retl
   %src = load i256, ptr %src.ptr, align 1
   %byteOff = load i256, ptr %byteOff.ptr, align 1
   %bitOff = shl i256 %byteOff, 3
   %res = ashr i256 %src, %bitOff
   store i256 %res, ptr %dst, align 1
   ret void
 }
 

 ; X64-SSE2-NEXT:    movq %rsi, 8(%rdx)
 ; X64-SSE2-NEXT:    popq %rbx
 ; X64-SSE2-NEXT:    popq %r14
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-SSE42-LABEL: ashr_64bytes:
 ; X64-SSE42:       # %bb.0:
 ; X64-SSE42-NEXT:    movups (%rdi), %xmm0
-; X64-SSE42-NEXT:    movups 16(%rdi), %xmm1
+; X64-SSE42-NEXT:    movdqu 16(%rdi), %xmm1
 ; X64-SSE42-NEXT:    movups 32(%rdi), %xmm2
 ; X64-SSE42-NEXT:    movq 48(%rdi), %rax
 ; X64-SSE42-NEXT:    movq 56(%rdi), %rcx
 ; X64-SSE42-NEXT:    movl (%rsi), %esi
 ; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    movups %xmm2, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movups %xmm1, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    movups %xmm0, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    sarq $63, %rcx
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-SSE42-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movq %rcx, %xmm0
+; X64-SSE42-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; X64-SSE42-NEXT:    pxor %xmm1, %xmm1
+; X64-SSE42-NEXT:    pcmpgtq %xmm0, %xmm1
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
+; X64-SSE42-NEXT:    movdqu %xmm1, -{{[0-9]+}}(%rsp)
 ; X64-SSE42-NEXT:    andl $63, %esi
 ; X64-SSE42-NEXT:    movups -128(%rsp,%rsi), %xmm0
 ; X64-SSE42-NEXT:    movups -112(%rsp,%rsi), %xmm1
 ; X64-SSE42-NEXT:    movups -96(%rsp,%rsi), %xmm2
 ; X64-SSE42-NEXT:    movups -80(%rsp,%rsi), %xmm3
-; X64-SSE42-NEXT:    movups %xmm1, 16(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm2, 32(%rdx)
-; X64-SSE42-NEXT:    movups %xmm3, 48(%rdx)
+; X64-SSE42-NEXT:    movups %xmm1, 16(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm0, (%rdx)
+; X64-SSE42-NEXT:    movups %xmm3, 48(%rdx)
 ; X64-SSE42-NEXT:    retq
 ;
-; X64-AVX-LABEL: ashr_64bytes:
-; X64-AVX:       # %bb.0:
-; X64-AVX-NEXT:    vmovups (%rdi), %ymm0
-; X64-AVX-NEXT:    vmovups 32(%rdi), %xmm1
-; X64-AVX-NEXT:    movq 48(%rdi), %rax
-; X64-AVX-NEXT:    movq 56(%rdi), %rcx
-; X64-AVX-NEXT:    movl (%rsi), %esi
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    vmovups %xmm1, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    sarq $63, %rcx
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    andl $63, %esi
-; X64-AVX-NEXT:    vmovups -128(%rsp,%rsi), %xmm0
-; X64-AVX-NEXT:    vmovups -112(%rsp,%rsi), %xmm1
-; X64-AVX-NEXT:    vmovups -96(%rsp,%rsi), %xmm2
-; X64-AVX-NEXT:    vmovups -80(%rsp,%rsi), %xmm3
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
-; X64-AVX-NEXT:    vzeroupper
-; X64-AVX-NEXT:    retq
+; X64-AVX1-LABEL: ashr_64bytes:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX1-NEXT:    vmovdqu 32(%rdi), %xmm1
+; X64-AVX1-NEXT:    movq 48(%rdi), %rax
+; X64-AVX1-NEXT:    movq 56(%rdi), %rcx
+; X64-AVX1-NEXT:    movl (%rsi), %esi
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovdqu %xmm1, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovq %rcx, %xmm0
+; X64-AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; X64-AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpcmpgtq %xmm0, %xmm1, %xmm0
+; X64-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    andl $63, %esi
+; X64-AVX1-NEXT:    vmovups -128(%rsp,%rsi), %xmm0
+; X64-AVX1-NEXT:    vmovups -112(%rsp,%rsi), %xmm1
+; X64-AVX1-NEXT:    vmovups -96(%rsp,%rsi), %xmm2
+; X64-AVX1-NEXT:    vmovups -80(%rsp,%rsi), %xmm3
+; X64-AVX1-NEXT:    vmovups %xmm2, 32(%rdx)
+; X64-AVX1-NEXT:    vmovups %xmm1, 16(%rdx)
+; X64-AVX1-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX1-NEXT:    vmovups %xmm3, 48(%rdx)
+; X64-AVX1-NEXT:    vzeroupper
+; X64-AVX1-NEXT:    retq
+;
+; X64-AVX512-LABEL: ashr_64bytes:
+; X64-AVX512:       # %bb.0:
+; X64-AVX512-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX512-NEXT:    vmovups 32(%rdi), %xmm1
+; X64-AVX512-NEXT:    movq 48(%rdi), %rax
+; X64-AVX512-NEXT:    movq 56(%rdi), %rcx
+; X64-AVX512-NEXT:    movl (%rsi), %esi
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    vmovups %xmm1, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    sarq $63, %rcx
+; X64-AVX512-NEXT:    vpbroadcastq %rcx, %zmm0
+; X64-AVX512-NEXT:    vmovdqu64 %zmm0, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    andl $63, %esi
+; X64-AVX512-NEXT:    vmovups -128(%rsp,%rsi), %xmm0
+; X64-AVX512-NEXT:    vmovups -112(%rsp,%rsi), %xmm1
+; X64-AVX512-NEXT:    vmovups -96(%rsp,%rsi), %xmm2
+; X64-AVX512-NEXT:    vmovups -80(%rsp,%rsi), %xmm3
+; X64-AVX512-NEXT:    vmovups %xmm2, 32(%rdx)
+; X64-AVX512-NEXT:    vmovups %xmm1, 16(%rdx)
+; X64-AVX512-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX512-NEXT:    vmovups %xmm3, 48(%rdx)
+; X64-AVX512-NEXT:    vzeroupper
+; X64-AVX512-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: ashr_64bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi
 ; X32-SSE2-NEXT:    pushl %esi
 ; X32-SSE2-NEXT:    subl $168, %esp

 ; X32-SSE42-NEXT:    movl (%ecx), %ecx
 ; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %edi, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movl %esi, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movups %xmm2, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movups %xmm1, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    movups %xmm0, (%esp)
-; X32-SSE42-NEXT:    sarl $31, %edx
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-SSE42-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movd %edx, %xmm0
+; X32-SSE42-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-SSE42-NEXT:    psrad $31, %xmm0
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
+; X32-SSE42-NEXT:    movdqu %xmm0, {{[0-9]+}}(%esp)
 ; X32-SSE42-NEXT:    andl $63, %ecx
 ; X32-SSE42-NEXT:    movups (%esp,%ecx), %xmm0
 ; X32-SSE42-NEXT:    movups 16(%esp,%ecx), %xmm1
 ; X32-SSE42-NEXT:    movups 32(%esp,%ecx), %xmm2
 ; X32-SSE42-NEXT:    movups 48(%esp,%ecx), %xmm3
 ; X32-SSE42-NEXT:    movups %xmm3, 48(%eax)
 ; X32-SSE42-NEXT:    movups %xmm2, 32(%eax)
 ; X32-SSE42-NEXT:    movups %xmm1, 16(%eax)
 ; X32-SSE42-NEXT:    movups %xmm0, (%eax)
 ; X32-SSE42-NEXT:    addl $128, %esp
 ; X32-SSE42-NEXT:    popl %esi
 ; X32-SSE42-NEXT:    popl %edi
 ; X32-SSE42-NEXT:    popl %ebx
 ; X32-SSE42-NEXT:    retl
 ;
-; X32-AVX-LABEL: ashr_64bytes:
-; X32-AVX:       # %bb.0:
-; X32-AVX-NEXT:    pushl %ebx
-; X32-AVX-NEXT:    pushl %edi
-; X32-AVX-NEXT:    pushl %esi
-; X32-AVX-NEXT:    subl $128, %esp
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X32-AVX-NEXT:    vmovups (%edx), %ymm0
-; X32-AVX-NEXT:    vmovups 32(%edx), %xmm1
-; X32-AVX-NEXT:    movl 48(%edx), %esi
-; X32-AVX-NEXT:    movl 52(%edx), %edi
-; X32-AVX-NEXT:    movl 56(%edx), %ebx
-; X32-AVX-NEXT:    movl 60(%edx), %edx
-; X32-AVX-NEXT:    movl (%ecx), %ecx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %esi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    vmovups %ymm0, (%esp)
-; X32-AVX-NEXT:    sarl $31, %edx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    andl $63, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups 32(%esp,%ecx), %xmm2
-; X32-AVX-NEXT:    vmovups 48(%esp,%ecx), %xmm3
-; X32-AVX-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
-; X32-AVX-NEXT:    addl $128, %esp
-; X32-AVX-NEXT:    popl %esi
-; X32-AVX-NEXT:    popl %edi
-; X32-AVX-NEXT:    popl %ebx
-; X32-AVX-NEXT:    vzeroupper
-; X32-AVX-NEXT:    retl
+; X32-AVX1-LABEL: ashr_64bytes:
+; X32-AVX1:       # %bb.0:
+; X32-AVX1-NEXT:    pushl %ebx
+; X32-AVX1-NEXT:    pushl %edi
+; X32-AVX1-NEXT:    pushl %esi
+; X32-AVX1-NEXT:    subl $128, %esp
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX1-NEXT:    vmovups (%edx), %ymm0
+; X32-AVX1-NEXT:    vmovups 32(%edx), %xmm1
+; X32-AVX1-NEXT:    movl 48(%edx), %esi
+; X32-AVX1-NEXT:    movl 52(%edx), %edi
+; X32-AVX1-NEXT:    movl 56(%edx), %ebx
+; X32-AVX1-NEXT:    movl 60(%edx), %edx
+; X32-AVX1-NEXT:    movl (%ecx), %ecx
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %ymm0, (%esp)
+; X32-AVX1-NEXT:    vmovd %edx, %xmm0
+; X32-AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; X32-AVX1-NEXT:    vpsrad $31, %xmm0, %xmm0
+; X32-AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; X32-AVX1-NEXT:    vmovups %ymm0, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %ymm0, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    andl $63, %ecx
+; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX1-NEXT:    vmovups 16(%esp,%ecx), %xmm1
+; X32-AVX1-NEXT:    vmovups 32(%esp,%ecx), %xmm2
+; X32-AVX1-NEXT:    vmovups 48(%esp,%ecx), %xmm3
+; X32-AVX1-NEXT:    vmovups %xmm3, 48(%eax)
+; X32-AVX1-NEXT:    vmovups %xmm2, 32(%eax)
+; X32-AVX1-NEXT:    vmovups %xmm1, 16(%eax)
+; X32-AVX1-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX1-NEXT:    addl $128, %esp
+; X32-AVX1-NEXT:    popl %esi
+; X32-AVX1-NEXT:    popl %edi
+; X32-AVX1-NEXT:    popl %ebx
+; X32-AVX1-NEXT:    vzeroupper
+; X32-AVX1-NEXT:    retl
+;
+; X32-AVX512-LABEL: ashr_64bytes:
+; X32-AVX512:       # %bb.0:
+; X32-AVX512-NEXT:    pushl %ebx
+; X32-AVX512-NEXT:    pushl %edi
+; X32-AVX512-NEXT:    pushl %esi
+; X32-AVX512-NEXT:    subl $128, %esp
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX512-NEXT:    vmovups (%edx), %ymm0
+; X32-AVX512-NEXT:    vmovups 32(%edx), %xmm1
+; X32-AVX512-NEXT:    movl 48(%edx), %esi
+; X32-AVX512-NEXT:    movl 52(%edx), %edi
+; X32-AVX512-NEXT:    movl 56(%edx), %ebx
+; X32-AVX512-NEXT:    movl 60(%edx), %edx
+; X32-AVX512-NEXT:    movl (%ecx), %ecx
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    vmovups %ymm0, (%esp)
+; X32-AVX512-NEXT:    sarl $31, %edx
+; X32-AVX512-NEXT:    vpbroadcastd %edx, %zmm0
+; X32-AVX512-NEXT:    vmovdqu64 %zmm0, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    andl $63, %ecx
+; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %xmm0
+; X32-AVX512-NEXT:    vmovups 16(%esp,%ecx), %xmm1
+; X32-AVX512-NEXT:    vmovups 32(%esp,%ecx), %xmm2
+; X32-AVX512-NEXT:    vmovups 48(%esp,%ecx), %xmm3
+; X32-AVX512-NEXT:    vmovups %xmm3, 48(%eax)
+; X32-AVX512-NEXT:    vmovups %xmm2, 32(%eax)
+; X32-AVX512-NEXT:    vmovups %xmm1, 16(%eax)
+; X32-AVX512-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX512-NEXT:    addl $128, %esp
+; X32-AVX512-NEXT:    popl %esi
+; X32-AVX512-NEXT:    popl %edi
+; X32-AVX512-NEXT:    popl %ebx
+; X32-AVX512-NEXT:    vzeroupper
+; X32-AVX512-NEXT:    retl
   %src = load i512, ptr %src.ptr, align 1
   %byteOff = load i512, ptr %byteOff.ptr, align 1
   %bitOff = shl i512 %byteOff, 3
   %res = ashr i512 %src, %bitOff
   store i512 %res, ptr %dst, align 1
   ret void
 }
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line: