[X86] Reenable store merging post-legalization

[lebedev.ri]

Summary

D62498 disabled store merging post-legalization,
because we do indeed want to split store of concat.

But the current way it's disabled is a way too heavy of a hammer.
We only want to disable the cases that would be problematic,
but e.g. still want to allow store merging of loads.

This requires D141776, otherwise we have conflicting combines.

[lebedev.ri]

(wrong diff)

[lebedev.ri]

Now with a proper diff.

[lebedev.ri]

llvm/lib/Target/X86/X86ISelLowering.h

1072–1073

This is a bit ugly. I'm using StoreSource::Unknown as
"we are post-legalization. do we allow merging *any* sources?".
Is that too ugly?

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19397–19398

If I read the patch correctly, this is the real point we care about. Should it be better to just introduce another TLI function to do such work?

[lebedev.ri]

Don't abuse StoreSource::Unknown, wrap it into std::optional<>.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19397–19398

I don't see why? We are simply enriching it with more knowledge to give a more precise answer.

[barannikov88]

llvm/include/llvm/CodeGen/TargetLowering.h

606

Appears unused with std::optional.

llvm/lib/Target/X86/X86ISelLowering.h

1072–1073

Just always pass "valid" StoreSrc as determined by getStoreSource?

[lebedev.ri]

llvm/include/llvm/CodeGen/TargetLowering.h

606

I do not follow. Both the struct and the element are both clearly still used.

llvm/lib/Target/X86/X86ISelLowering.h

1072–1073

The idea is that there is no point in ever trying to guess the StoreSource,
which may be costly to do, if the target will always refuse.

[barannikov88]

llvm/include/llvm/CodeGen/TargetLowering.h

606

Yes, my bad. Although mergeStoresAfterLegalization will never receive Unknown.
I'd suggest to stick to either Unknown or std::optional, but not both. Either is fine to me.

[lebedev.ri]

llvm/include/llvm/CodeGen/TargetLowering.h

606

Note that i'm simply moving the existing enum, it can't not have all of it's original elements.

[lebedev.ri]

ping

[pengfei]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

I'd agree with @barannikov88. Unknown and optional represent the same meaning. Why don't return std::nullopt here too?

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

674–675

The literally next change will return splat here instead.

[lebedev.ri]

Rebased, NFC.
Now that D141776 has landed, this can proceed.

[RKSimon]

CC'ing @TokarIP who recently added the light-avx tuning flag for something similar.

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h

247

drop unused StoreSrc?

mergeStoresAfterLegalization(EVT, std::optional<StoreSource>)

llvm/lib/Target/X86/X86ISelLowering.h

1074

should we play safe and treat unknown as a potential extractsubvector?

[lebedev.ri]

llvm/lib/Target/X86/X86ISelLowering.h

1074

Let me just assert that we don't get unknown here.

[lebedev.ri]

@RKSimon thank you for taking a look!
Addressing nits.

llvm/include/llvm/CodeGen/TargetLowering.h

virtual bool storeOfVectorConstantIsCheap(EVT MemVT,
                                             unsigned NumElem,
                                             unsigned AddrSpace) const {
     return false;
   }
 
   /// Allow store merging for the specified type after legalization in addition
   /// to before legalization. This may transform stores that do not exist
   /// earlier (for example, stores created from intrinsics).
-  virtual bool mergeStoresAfterLegalization(EVT MemVT) const {
+  /// The answer may depend on the origin of the store:
+  enum class StoreSource {
+    Unknown,
+    Constant,
+    ExtractVectorElt,
+    ExtractSubvector,
+    Load
+  };
+  virtual bool mergeStoresAfterLegalization(EVT MemVT,
+                                            StoreSource StoreSrc) const {
     return true;
   }
 
   /// Returns if it's reasonable to merge stores to MemVT size.
   virtual bool canMergeStoresTo(unsigned AS, EVT MemVT,
                                 const MachineFunction &MF) const {
     return true;
   }

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

struct MemOpLink {
       // Offset from the base ptr.
       int64_t OffsetFromBase;
 
       MemOpLink(LSBaseSDNode *N, int64_t Offset)
           : MemNode(N), OffsetFromBase(Offset) {}
     };
 
     // Classify the origin of a stored value.
-    enum class StoreSource { Unknown, Constant, Extract, Load };
-    StoreSource getStoreSource(SDValue StoreVal) {
+    TargetLoweringBase::StoreSource getStoreSource(SDValue StoreVal) {
       switch (StoreVal.getOpcode()) {
       case ISD::Constant:
       case ISD::ConstantFP:
-        return StoreSource::Constant;
+        return TargetLoweringBase::StoreSource::Constant;
       case ISD::EXTRACT_VECTOR_ELT:
+        return TargetLoweringBase::StoreSource::ExtractVectorElt;
       case ISD::EXTRACT_SUBVECTOR:
-        return StoreSource::Extract;
+        return TargetLoweringBase::StoreSource::ExtractSubvector;
       case ISD::LOAD:
-        return StoreSource::Load;
+        return TargetLoweringBase::StoreSource::Load;
       default:
-        return StoreSource::Unknown;
+        return TargetLoweringBase::StoreSource::Unknown;
       }
     }
 
     /// This is a helper function for visitMUL to check the profitability
     /// of folding (mul (add x, c1), c2) -> (add (mul x, c2), c1*c2).
     /// MulNode is the original multiply, AddNode is (add x, c1),
     /// and ConstNode is c2.
     bool isMulAddWithConstProfitable(SDNode *MulNode, SDValue AddNode,
void DAGCombiner::getStoreMergeCandidates(
   // This holds the base pointer, index, and the offset in bytes from the base
   // pointer. We must have a base and an offset. Do not handle stores to undef
   // base pointers.
   BaseIndexOffset BasePtr = BaseIndexOffset::match(St, DAG);
   if (!BasePtr.getBase().getNode() || BasePtr.getBase().isUndef())
     return;
 
   SDValue Val = peekThroughBitcasts(St->getValue());
-  StoreSource StoreSrc = getStoreSource(Val);
-  assert(StoreSrc != StoreSource::Unknown && "Expected known source for store");
+  TargetLoweringBase::StoreSource StoreSrc = getStoreSource(Val);
+  assert(StoreSrc != TargetLoweringBase::StoreSource::Unknown &&
+         "Expected known source for store");
 
   // Match on loadbaseptr if relevant.
   EVT MemVT = St->getMemoryVT();
   BaseIndexOffset LBasePtr;
   EVT LoadVT;
-  if (StoreSrc == StoreSource::Load) {
+  if (StoreSrc == TargetLoweringBase::StoreSource::Load) {
     auto *Ld = cast<LoadSDNode>(Val);
     LBasePtr = BaseIndexOffset::match(Ld, DAG);
     LoadVT = Ld->getMemoryVT();
     // Load and store should be the same type.
     if (MemVT != LoadVT)
       return;
     // Loads must only have one use.
     if (!Ld->hasNUsesOfValue(1, 0))
auto CandidateMatch = [&](StoreSDNode *Other, BaseIndexOffset &Ptr,
     // Don't mix temporal stores with non-temporal stores.
     if (St->isNonTemporal() != Other->isNonTemporal())
       return false;
     SDValue OtherBC = peekThroughBitcasts(Other->getValue());
     // Allow merging constants of different types as integers.
     bool NoTypeMatch = (MemVT.isInteger()) ? !MemVT.bitsEq(Other->getMemoryVT())
                                            : Other->getMemoryVT() != MemVT;
     switch (StoreSrc) {
-    case StoreSource::Load: {
+    case TargetLoweringBase::StoreSource::Load: {
       if (NoTypeMatch)
         return false;
       // The Load's Base Ptr must also match.
       auto *OtherLd = dyn_cast<LoadSDNode>(OtherBC);
       if (!OtherLd)
         return false;
       BaseIndexOffset LPtr = BaseIndexOffset::match(OtherLd, DAG);
       if (LoadVT != OtherLd->getMemoryVT())
         return false;
       // Loads must only have one use.
       if (!OtherLd->hasNUsesOfValue(1, 0))
         return false;
       // The memory operands must not be volatile/indexed/atomic.
       // TODO: May be able to relax for unordered atomics (see D66309)
       if (!OtherLd->isSimple() || OtherLd->isIndexed())
         return false;
       // Don't mix temporal loads with non-temporal loads.
       if (cast<LoadSDNode>(Val)->isNonTemporal() != OtherLd->isNonTemporal())
         return false;
       if (!(LBasePtr.equalBaseIndex(LPtr, DAG)))
         return false;
       break;
     }
-    case StoreSource::Constant:
+    case TargetLoweringBase::StoreSource::Constant:
       if (NoTypeMatch)
         return false;
       if (!isIntOrFPConstant(OtherBC))
         return false;
       break;
-    case StoreSource::Extract:
+    case TargetLoweringBase::StoreSource::ExtractVectorElt:
+    case TargetLoweringBase::StoreSource::ExtractSubvector:
       // Do not merge truncated stores here.
       if (Other->isTruncatingStore())
         return false;
       if (!MemVT.bitsEq(OtherBC.getValueType()))
         return false;
       if (OtherBC.getOpcode() != ISD::EXTRACT_VECTOR_ELT &&
           OtherBC.getOpcode() != ISD::EXTRACT_SUBVECTOR)
         return false;
if (!MemVT.isSimple() || MemVT.getSizeInBits() * 2 > MaximumLegalStoreInBits)
     return false;
 
   // This function cannot currently deal with non-byte-sized memory sizes.
   int64_t ElementSizeBytes = MemVT.getStoreSize();
   if (ElementSizeBytes * 8 != (int64_t)MemVT.getSizeInBits())
     return false;
 
   // Do not bother looking at stored values that are not constants, loads, or
-  // extracted vector elements.
+  // extracted vector elements/subvectors.
   SDValue StoredVal = peekThroughBitcasts(St->getValue());
-  const StoreSource StoreSrc = getStoreSource(StoredVal);
-  if (StoreSrc == StoreSource::Unknown)
+  const TargetLoweringBase::StoreSource StoreSrc = getStoreSource(StoredVal);
+  if (StoreSrc == TargetLoweringBase::StoreSource::Unknown)
+    return false;
+
+  if (LegalTypes &&
+      !TLI.mergeStoresAfterLegalization(St->getMemoryVT(), StoreSrc))
     return false;
 
   SmallVector<MemOpLink, 8> StoreNodes;
   SDNode *RootNode;
   // Find potential store merge candidates by searching through chain sub-DAG
   getStoreMergeCandidates(St, StoreNodes, RootNode);
 
   // Check if there is anything to merge.
   if (StoreNodes.size() < 2)
     return false;
 
   // Sort the memory operands according to their distance from the
   // base pointer.
   llvm::sort(StoreNodes, [](MemOpLink LHS, MemOpLink RHS) {
     return LHS.OffsetFromBase < RHS.OffsetFromBase;
   });
 
   bool AllowVectors = !DAG.getMachineFunction().getFunction().hasFnAttribute(
       Attribute::NoImplicitFloat);
   bool IsNonTemporalStore = St->isNonTemporal();
-  bool IsNonTemporalLoad = StoreSrc == StoreSource::Load &&
+  bool IsNonTemporalLoad = StoreSrc == TargetLoweringBase::StoreSource::Load &&
                            cast<LoadSDNode>(StoredVal)->isNonTemporal();
 
   // Store Merge attempts to merge the lowest stores. This generally
   // works out as if successful, as the remaining stores are checked
   // after the first collection of stores is merged. However, in the
   // case that a non-mergeable store is found first, e.g., {p[-2],
   // p[0], p[1], p[2], p[3]}, we would fail and miss the subsequent
   // mergeable cases. To prevent this, we prune such stores from the
   // front of StoreNodes here.
   bool MadeChange = false;
   while (StoreNodes.size() > 1) {
     unsigned NumConsecutiveStores =
         getConsecutiveStores(StoreNodes, ElementSizeBytes);
     // There are no more stores in the list to examine.
     if (NumConsecutiveStores == 0)
       return MadeChange;
 
     // We have at least 2 consecutive stores. Try to merge them.
     assert(NumConsecutiveStores >= 2 && "Expected at least 2 stores");
     switch (StoreSrc) {
-    case StoreSource::Constant:
+    case TargetLoweringBase::StoreSource::Constant:
       MadeChange |= tryStoreMergeOfConstants(StoreNodes, NumConsecutiveStores,
                                              MemVT, RootNode, AllowVectors);
       break;
 
-    case StoreSource::Extract:
+    case TargetLoweringBase::StoreSource::ExtractVectorElt:
+    case TargetLoweringBase::StoreSource::ExtractSubvector:
       MadeChange |= tryStoreMergeOfExtracts(StoreNodes, NumConsecutiveStores,
                                             MemVT, RootNode);
       break;
 
-    case StoreSource::Load:
+    case TargetLoweringBase::StoreSource::Load:
       MadeChange |= tryStoreMergeOfLoads(StoreNodes, NumConsecutiveStores,
                                          MemVT, RootNode, AllowVectors,
                                          IsNonTemporalStore, IsNonTemporalLoad);
       break;
 
     default:
       llvm_unreachable("Unhandled store source type");
     }
if ((Value.getOpcode() == ISD::FP_ROUND ||
                                ST->getMemoryVT(), LegalOperations)) {
     return DAG.getTruncStore(Chain, SDLoc(N), Value.getOperand(0),
                              Ptr, ST->getMemoryVT(), ST->getMemOperand());
   }
 
   // Always perform this optimization before types are legal. If the target
   // prefers, also try this after legalization to catch stores that were created
   // by intrinsics or other nodes.
-  if (!LegalTypes || (TLI.mergeStoresAfterLegalization(ST->getMemoryVT()))) {
+  if (!LegalTypes ||
+      (TLI.mergeStoresAfterLegalization(
+          ST->getMemoryVT(), TargetLoweringBase::StoreSource::Unknown))) {
     while (true) {
       // There can be multiple store sequences on the same chain.
       // Keep trying to merge store sequences until we are unable to do so
       // or until we merge the last store on the chain.
       bool Changed = mergeConsecutiveStores(ST);
       if (!Changed) break;
       // Return N as merge only uses CombineTo and no worklist clean
       // up is necessary.

llvm/lib/Target/AArch64/AArch64ISelLowering.h

public:
   bool fallBackToDAGISel(const Instruction &Inst) const override;
 
   /// SVE code generation for fixed length vectors does not custom lower
   /// BUILD_VECTOR. This makes BUILD_VECTOR legalisation a source of stores to
   /// merge. However, merging them creates a BUILD_VECTOR that is just as
   /// illegal as the original, thus leading to an infinite legalisation loop.
   /// NOTE: Once BUILD_VECTOR is legal or can be custom lowered for all legal
   /// vector types this override can be removed.
-  bool mergeStoresAfterLegalization(EVT VT) const override;
+  bool mergeStoresAfterLegalization(EVT VT,
+                                    StoreSource StoreSrc) const override;
 
   // If the platform/function should have a redzone, return the size in bytes.
   unsigned getRedZoneSize(const Function &F) const {
     if (F.hasFnAttribute(Attribute::NoRedZone))
       return 0;
     return 128;
   }
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

case ISD::WRITE_REGISTER: {
     SDValue Result = DAG.getNode(AArch64ISD::MSRR, DL, MVT::Other, Chain,
                                  SysRegName, PairLo, PairHi);
 
     return Result;
   }
   }
 }
 
-bool AArch64TargetLowering::mergeStoresAfterLegalization(EVT VT) const {
+bool AArch64TargetLowering::mergeStoresAfterLegalization(
+    EVT VT, StoreSource StoreSrc) const {
   return !Subtarget->useSVEForFixedLengthVectors();
 }
 
 bool AArch64TargetLowering::useSVEForFixedLengthVectorVT(
     EVT VT, bool OverrideNEON) const {
   if (!VT.isFixedLengthVector() || !VT.isSimple())
     return false;
 

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h

public:
   // FIXME: Turn off MergeConsecutiveStores() before Instruction Selection for
   // AMDGPU.  Commit r319036,
   // (https://github.com/llvm/llvm-project/commit/db77e57ea86d941a4262ef60261692f4cb6893e6)
   // turned on MergeConsecutiveStores() before Instruction Selection for all
   // targets.  Enough AMDGPU compiles go into an infinite loop (
   // MergeConsecutiveStores() merges two stores; LegalizeStoreOps() un-merges;
   // MergeConsecutiveStores() re-merges, etc. ) to warrant turning it off for
   // now.
-  bool mergeStoresAfterLegalization(EVT) const override { return false; }
+  bool mergeStoresAfterLegalization(EVT, StoreSource StoreSrc) const override {
+    return false;
+  }
 
   bool isFsqrtCheap(SDValue Operand, SelectionDAG &DAG) const override {
     return true;
   }
   SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                            int &RefinementSteps, bool &UseOneConstNR,
                            bool Reciprocal) const override;
   SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,

llvm/lib/Target/RISCV/RISCVISelLowering.h

private:
   MVT getVPExplicitVectorLengthTy() const override;
 
   /// RVV code generation for fixed length vectors does not lower all
   /// BUILD_VECTORs. This makes BUILD_VECTOR legalisation a source of stores to
   /// merge. However, merging them creates a BUILD_VECTOR that is just as
   /// illegal as the original, thus leading to an infinite legalisation loop.
   /// NOTE: Once BUILD_VECTOR can be custom lowered for all legal vector types,
   /// this override can be removed.
-  bool mergeStoresAfterLegalization(EVT VT) const override;
+  bool mergeStoresAfterLegalization(EVT VT,
+                                    StoreSource StoreSrc) const override;
 
   /// Disable normalizing
   /// select(N0&N1, X, Y) => select(N0, select(N1, X, Y), Y) and
   /// select(N0|N1, X, Y) => select(N0, select(N1, X, Y, Y))
   /// RISCV doesn't have flags so it's better to perform the and/or in a GPR.
   bool shouldNormalizeToSelectSequence(LLVMContext &, EVT) const override {
     return false;
   };

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if (VecRegClassID > RCID && SubRegClassID <= RCID) {
       if (IsHi)
         InsertExtractIdx -= VecVT.getVectorElementCount().getKnownMinValue();
     }
   return {SubRegIdx, InsertExtractIdx};
 }
 
 // Permit combining of mask vectors as BUILD_VECTOR never expands to scalar
 // stores for those types.
-bool RISCVTargetLowering::mergeStoresAfterLegalization(EVT VT) const {
+bool RISCVTargetLowering::mergeStoresAfterLegalization(
+    EVT VT, StoreSource StoreSrc) const {
   return !Subtarget.useRVVForFixedLengthVectors() ||
          (VT.isFixedLengthVector() && VT.getVectorElementType() == MVT::i1);
 }
 
 bool RISCVTargetLowering::isLegalElementTypeForRVV(Type *ScalarTy) const {
   if (ScalarTy->isPointerTy())
     return true;
 

llvm/lib/Target/X86/X86ISelLowering.h

public:
 
     MachineBasicBlock *
     EmitInstrWithCustomInserter(MachineInstr &MI,
                                 MachineBasicBlock *MBB) const override;
 
     /// This method returns the name of a target specific DAG node.
     const char *getTargetNodeName(unsigned Opcode) const override;
 
-    /// Do not merge vector stores after legalization because that may conflict
-    /// with x86-specific store splitting optimizations.
-    bool mergeStoresAfterLegalization(EVT MemVT) const override {
-      return !MemVT.isVector();
+    /// Do not merge vector stores of subvector extractions after legalization
+    /// because that may conflict with x86-specific store splitting
+    /// optimizations.
+    bool mergeStoresAfterLegalization(EVT MemVT,
+                                      StoreSource StoreSrc) const override {
+      return StoreSrc == StoreSource::Unknown ||
+             StoreSrc != StoreSource::ExtractSubvector;
     }
 
     bool canMergeStoresTo(unsigned AddressSpace, EVT MemVT,
                           const MachineFunction &MF) const override;
 
     bool isCheapToSpeculateCttz(Type *Ty) const override;
 
     bool isCheapToSpeculateCtlz(Type *Ty) const override;
 

llvm/test/CodeGen/X86/wide-scalar-shift-by-byte-multiple-legalization.ll

 ; X64-AVX-LABEL: lshr_32bytes:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovups (%rdi), %ymm0
 ; X64-AVX-NEXT:    movzbl (%rsi), %eax
 ; X64-AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X64-AVX-NEXT:    vmovups %ymm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    andl $31, %eax
-; X64-AVX-NEXT:    vmovups -64(%rsp,%rax), %xmm0
-; X64-AVX-NEXT:    vmovups -48(%rsp,%rax), %xmm1
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX-NEXT:    vmovups -64(%rsp,%rax), %ymm0
+; X64-AVX-NEXT:    vmovups %ymm0, (%rdx)
 ; X64-AVX-NEXT:    vzeroupper
 ; X64-AVX-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: lshr_32bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi

 ; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-AVX-NEXT:    vmovups (%edx), %ymm0
 ; X32-AVX-NEXT:    movzbl (%ecx), %ecx
 ; X32-AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X32-AVX-NEXT:    vmovups %ymm1, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    vmovups %ymm0, (%esp)
 ; X32-AVX-NEXT:    andl $31, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX-NEXT:    vmovups (%esp,%ecx), %ymm0
+; X32-AVX-NEXT:    vmovups %ymm0, (%eax)
 ; X32-AVX-NEXT:    addl $64, %esp
 ; X32-AVX-NEXT:    vzeroupper
 ; X32-AVX-NEXT:    retl
   %src = load i256, ptr %src.ptr, align 1
   %byteOff = load i256, ptr %byteOff.ptr, align 1
   %bitOff = shl i256 %byteOff, 3
   %res = lshr i256 %src, %bitOff
   store i256 %res, ptr %dst, align 1

 ; X64-AVX-NEXT:    vmovups (%rdi), %ymm0
 ; X64-AVX-NEXT:    movzbl (%rsi), %eax
 ; X64-AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X64-AVX-NEXT:    vmovups %ymm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    andb $31, %al
 ; X64-AVX-NEXT:    negb %al
 ; X64-AVX-NEXT:    movsbq %al, %rax
-; X64-AVX-NEXT:    vmovups -32(%rsp,%rax), %xmm0
-; X64-AVX-NEXT:    vmovups -16(%rsp,%rax), %xmm1
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX-NEXT:    vmovups -32(%rsp,%rax), %ymm0
+; X64-AVX-NEXT:    vmovups %ymm0, (%rdx)
 ; X64-AVX-NEXT:    vzeroupper
 ; X64-AVX-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: shl_32bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi

 ; X32-AVX-NEXT:    vmovups (%edx), %ymm0
 ; X32-AVX-NEXT:    movzbl (%ecx), %ecx
 ; X32-AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X32-AVX-NEXT:    vmovups %ymm1, (%esp)
 ; X32-AVX-NEXT:    vmovups %ymm0, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    andb $31, %cl
 ; X32-AVX-NEXT:    negb %cl
 ; X32-AVX-NEXT:    movsbl %cl, %ecx
-; X32-AVX-NEXT:    vmovups 32(%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 48(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX-NEXT:    vmovups 32(%esp,%ecx), %ymm0
+; X32-AVX-NEXT:    vmovups %ymm0, (%eax)
 ; X32-AVX-NEXT:    addl $64, %esp
 ; X32-AVX-NEXT:    vzeroupper
 ; X32-AVX-NEXT:    retl
   %src = load i256, ptr %src.ptr, align 1
   %byteOff = load i256, ptr %byteOff.ptr, align 1
   %bitOff = shl i256 %byteOff, 3
   %res = shl i256 %src, %bitOff
   store i256 %res, ptr %dst, align 1

 ; X64-AVX-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    vmovups %xmm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    sarq $63, %rcx
 ; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; X64-AVX-NEXT:    andl $31, %esi
-; X64-AVX-NEXT:    vmovups -64(%rsp,%rsi), %xmm0
-; X64-AVX-NEXT:    vmovups -48(%rsp,%rsi), %xmm1
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX-NEXT:    vmovups -64(%rsp,%rsi), %ymm0
+; X64-AVX-NEXT:    vmovups %ymm0, (%rdx)
+; X64-AVX-NEXT:    vzeroupper
 ; X64-AVX-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: ashr_32bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi
 ; X32-SSE2-NEXT:    pushl %esi

 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
 ; X32-AVX-NEXT:    andl $31, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX-NEXT:    vmovups (%esp,%ecx), %ymm0
+; X32-AVX-NEXT:    vmovups %ymm0, (%eax)
 ; X32-AVX-NEXT:    addl $64, %esp
 ; X32-AVX-NEXT:    popl %esi
 ; X32-AVX-NEXT:    popl %edi
 ; X32-AVX-NEXT:    popl %ebx
+; X32-AVX-NEXT:    vzeroupper
 ; X32-AVX-NEXT:    retl
   %src = load i256, ptr %src.ptr, align 1
   %byteOff = load i256, ptr %byteOff.ptr, align 1
   %bitOff = shl i256 %byteOff, 3
   %res = ashr i256 %src, %bitOff
   store i256 %res, ptr %dst, align 1
   ret void
 }

 ; X64-AVX1-NEXT:    vmovups 32(%rdi), %ymm1
 ; X64-AVX1-NEXT:    movl (%rsi), %eax
 ; X64-AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; X64-AVX1-NEXT:    vmovups %ymm2, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm2, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    andl $63, %eax
-; X64-AVX1-NEXT:    vmovups -128(%rsp,%rax), %xmm0
-; X64-AVX1-NEXT:    vmovups -112(%rsp,%rax), %xmm1
-; X64-AVX1-NEXT:    vmovups -96(%rsp,%rax), %xmm2
-; X64-AVX1-NEXT:    vmovups -80(%rsp,%rax), %xmm3
-; X64-AVX1-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX1-NEXT:    vmovups -128(%rsp,%rax), %ymm0
+; X64-AVX1-NEXT:    vmovups -96(%rsp,%rax), %ymm1
+; X64-AVX1-NEXT:    vmovups %ymm1, 32(%rdx)
+; X64-AVX1-NEXT:    vmovups %ymm0, (%rdx)
 ; X64-AVX1-NEXT:    vzeroupper
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX512-LABEL: lshr_64bytes:
 ; X64-AVX512:       # %bb.0:
 ; X64-AVX512-NEXT:    vmovups (%rdi), %zmm0
 ; X64-AVX512-NEXT:    movl (%rsi), %eax
 ; X64-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X64-AVX512-NEXT:    vmovups %zmm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX512-NEXT:    vmovups %zmm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX512-NEXT:    andl $63, %eax
-; X64-AVX512-NEXT:    vmovups -128(%rsp,%rax), %xmm0
-; X64-AVX512-NEXT:    vmovups -112(%rsp,%rax), %xmm1
-; X64-AVX512-NEXT:    vmovups -96(%rsp,%rax), %xmm2
-; X64-AVX512-NEXT:    vmovups -80(%rsp,%rax), %xmm3
-; X64-AVX512-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX512-NEXT:    vmovups -128(%rsp,%rax), %zmm0
+; X64-AVX512-NEXT:    vmovups %zmm0, (%rdx)
 ; X64-AVX512-NEXT:    vzeroupper
 ; X64-AVX512-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: lshr_64bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi

 ; X32-AVX1-NEXT:    vmovups 32(%edx), %ymm1
 ; X32-AVX1-NEXT:    movl (%ecx), %ecx
 ; X32-AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; X32-AVX1-NEXT:    vmovups %ymm2, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm2, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm1, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm0, (%esp)
 ; X32-AVX1-NEXT:    andl $63, %ecx
-; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX1-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX1-NEXT:    vmovups 32(%esp,%ecx), %xmm2
-; X32-AVX1-NEXT:    vmovups 48(%esp,%ecx), %xmm3
-; X32-AVX1-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %ymm0
+; X32-AVX1-NEXT:    vmovups 32(%esp,%ecx), %ymm1
+; X32-AVX1-NEXT:    vmovups %ymm1, 32(%eax)
+; X32-AVX1-NEXT:    vmovups %ymm0, (%eax)
 ; X32-AVX1-NEXT:    addl $128, %esp
 ; X32-AVX1-NEXT:    vzeroupper
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: lshr_64bytes:
 ; X32-AVX512:       # %bb.0:
 ; X32-AVX512-NEXT:    subl $128, %esp
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-AVX512-NEXT:    vmovups (%edx), %zmm0
 ; X32-AVX512-NEXT:    movl (%ecx), %ecx
 ; X32-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X32-AVX512-NEXT:    vmovups %zmm1, {{[0-9]+}}(%esp)
 ; X32-AVX512-NEXT:    vmovups %zmm0, (%esp)
 ; X32-AVX512-NEXT:    andl $63, %ecx
-; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX512-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX512-NEXT:    vmovups 32(%esp,%ecx), %xmm2
-; X32-AVX512-NEXT:    vmovups 48(%esp,%ecx), %xmm3
-; X32-AVX512-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %zmm0
+; X32-AVX512-NEXT:    vmovups %zmm0, (%eax)
 ; X32-AVX512-NEXT:    addl $128, %esp
 ; X32-AVX512-NEXT:    vzeroupper
 ; X32-AVX512-NEXT:    retl
   %src = load i512, ptr %src.ptr, align 1
   %byteOff = load i512, ptr %byteOff.ptr, align 1
   %bitOff = shl i512 %byteOff, 3
   %res = lshr i512 %src, %bitOff
   store i512 %res, ptr %dst, align 1

 ; X64-AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; X64-AVX1-NEXT:    vmovups %ymm2, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm2, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX1-NEXT:    andl $63, %eax
 ; X64-AVX1-NEXT:    negl %eax
 ; X64-AVX1-NEXT:    cltq
-; X64-AVX1-NEXT:    vmovups -64(%rsp,%rax), %xmm0
-; X64-AVX1-NEXT:    vmovups -48(%rsp,%rax), %xmm1
-; X64-AVX1-NEXT:    vmovups -32(%rsp,%rax), %xmm2
-; X64-AVX1-NEXT:    vmovups -16(%rsp,%rax), %xmm3
-; X64-AVX1-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX1-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX1-NEXT:    vmovups -64(%rsp,%rax), %ymm0
+; X64-AVX1-NEXT:    vmovups -32(%rsp,%rax), %ymm1
+; X64-AVX1-NEXT:    vmovups %ymm1, 32(%rdx)
+; X64-AVX1-NEXT:    vmovups %ymm0, (%rdx)
 ; X64-AVX1-NEXT:    vzeroupper
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX512-LABEL: shl_64bytes:
 ; X64-AVX512:       # %bb.0:
 ; X64-AVX512-NEXT:    vmovups (%rdi), %zmm0
 ; X64-AVX512-NEXT:    movl (%rsi), %eax
 ; X64-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X64-AVX512-NEXT:    vmovups %zmm1, -{{[0-9]+}}(%rsp)
 ; X64-AVX512-NEXT:    vmovups %zmm0, -{{[0-9]+}}(%rsp)
 ; X64-AVX512-NEXT:    andl $63, %eax
 ; X64-AVX512-NEXT:    negl %eax
 ; X64-AVX512-NEXT:    cltq
-; X64-AVX512-NEXT:    vmovups -64(%rsp,%rax), %xmm0
-; X64-AVX512-NEXT:    vmovups -48(%rsp,%rax), %xmm1
-; X64-AVX512-NEXT:    vmovups -32(%rsp,%rax), %xmm2
-; X64-AVX512-NEXT:    vmovups -16(%rsp,%rax), %xmm3
-; X64-AVX512-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX512-NEXT:    vmovups %xmm0, (%rdx)
+; X64-AVX512-NEXT:    vmovups -64(%rsp,%rax), %zmm0
+; X64-AVX512-NEXT:    vmovups %zmm0, (%rdx)
 ; X64-AVX512-NEXT:    vzeroupper
 ; X64-AVX512-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: shl_64bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi

 ; X32-AVX1-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; X32-AVX1-NEXT:    vmovups %ymm2, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm2, (%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm1, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    vmovups %ymm0, {{[0-9]+}}(%esp)
 ; X32-AVX1-NEXT:    andl $63, %ecx
 ; X32-AVX1-NEXT:    leal {{[0-9]+}}(%esp), %edx
 ; X32-AVX1-NEXT:    subl %ecx, %edx
-; X32-AVX1-NEXT:    vmovups (%edx), %xmm0
-; X32-AVX1-NEXT:    vmovups 16(%edx), %xmm1
-; X32-AVX1-NEXT:    vmovups 32(%edx), %xmm2
+; X32-AVX1-NEXT:    vmovups (%edx), %ymm0
 ; X32-AVX1-NEXT:    negl %ecx
-; X32-AVX1-NEXT:    vmovups 112(%esp,%ecx), %xmm3
-; X32-AVX1-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX1-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX1-NEXT:    vmovups 96(%esp,%ecx), %ymm1
+; X32-AVX1-NEXT:    vmovups %ymm1, 32(%eax)
+; X32-AVX1-NEXT:    vmovups %ymm0, (%eax)
 ; X32-AVX1-NEXT:    addl $128, %esp
 ; X32-AVX1-NEXT:    vzeroupper
 ; X32-AVX1-NEXT:    retl
 ;
 ; X32-AVX512-LABEL: shl_64bytes:
 ; X32-AVX512:       # %bb.0:
 ; X32-AVX512-NEXT:    subl $128, %esp
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-AVX512-NEXT:    vmovups (%edx), %zmm0
 ; X32-AVX512-NEXT:    movl (%ecx), %ecx
 ; X32-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X32-AVX512-NEXT:    vmovups %zmm1, (%esp)
 ; X32-AVX512-NEXT:    vmovups %zmm0, {{[0-9]+}}(%esp)
 ; X32-AVX512-NEXT:    andl $63, %ecx
-; X32-AVX512-NEXT:    leal {{[0-9]+}}(%esp), %edx
-; X32-AVX512-NEXT:    subl %ecx, %edx
-; X32-AVX512-NEXT:    vmovups (%edx), %xmm0
-; X32-AVX512-NEXT:    vmovups 16(%edx), %xmm1
-; X32-AVX512-NEXT:    vmovups 32(%edx), %xmm2
 ; X32-AVX512-NEXT:    negl %ecx
-; X32-AVX512-NEXT:    vmovups 112(%esp,%ecx), %xmm3
-; X32-AVX512-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX512-NEXT:    vmovups %xmm0, (%eax)
+; X32-AVX512-NEXT:    vmovups 64(%esp,%ecx), %zmm0
+; X32-AVX512-NEXT:    vmovups %zmm0, (%eax)
 ; X32-AVX512-NEXT:    addl $128, %esp
 ; X32-AVX512-NEXT:    vzeroupper
 ; X32-AVX512-NEXT:    retl
   %src = load i512, ptr %src.ptr, align 1
   %byteOff = load i512, ptr %byteOff.ptr, align 1
   %bitOff = shl i512 %byteOff, 3
   %res = shl i512 %src, %bitOff
   store i512 %res, ptr %dst, align 1

 ; X64-SSE42-NEXT:    movups -96(%rsp,%rsi), %xmm2
 ; X64-SSE42-NEXT:    movups -80(%rsp,%rsi), %xmm3
 ; X64-SSE42-NEXT:    movups %xmm1, 16(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm2, 32(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm3, 48(%rdx)
 ; X64-SSE42-NEXT:    movups %xmm0, (%rdx)
 ; X64-SSE42-NEXT:    retq
 ;
-; X64-AVX-LABEL: ashr_64bytes:
-; X64-AVX:       # %bb.0:
-; X64-AVX-NEXT:    vmovups (%rdi), %ymm0
-; X64-AVX-NEXT:    vmovups 32(%rdi), %xmm1
-; X64-AVX-NEXT:    movq 48(%rdi), %rax
-; X64-AVX-NEXT:    movq 56(%rdi), %rcx
-; X64-AVX-NEXT:    movl (%rsi), %esi
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    vmovups %xmm1, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    sarq $63, %rcx
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; X64-AVX-NEXT:    andl $63, %esi
-; X64-AVX-NEXT:    vmovups -128(%rsp,%rsi), %xmm0
-; X64-AVX-NEXT:    vmovups -112(%rsp,%rsi), %xmm1
-; X64-AVX-NEXT:    vmovups -96(%rsp,%rsi), %xmm2
-; X64-AVX-NEXT:    vmovups -80(%rsp,%rsi), %xmm3
-; X64-AVX-NEXT:    vmovups %xmm1, 16(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm2, 32(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm3, 48(%rdx)
-; X64-AVX-NEXT:    vmovups %xmm0, (%rdx)
-; X64-AVX-NEXT:    vzeroupper
-; X64-AVX-NEXT:    retq
+; X64-AVX1-LABEL: ashr_64bytes:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX1-NEXT:    vmovups 32(%rdi), %xmm1
+; X64-AVX1-NEXT:    movq 48(%rdi), %rax
+; X64-AVX1-NEXT:    movq 56(%rdi), %rcx
+; X64-AVX1-NEXT:    movl (%rsi), %esi
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovups %xmm1, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    sarq $63, %rcx
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX1-NEXT:    andl $63, %esi
+; X64-AVX1-NEXT:    vmovups -128(%rsp,%rsi), %ymm0
+; X64-AVX1-NEXT:    vmovups -96(%rsp,%rsi), %ymm1
+; X64-AVX1-NEXT:    vmovups %ymm1, 32(%rdx)
+; X64-AVX1-NEXT:    vmovups %ymm0, (%rdx)
+; X64-AVX1-NEXT:    vzeroupper
+; X64-AVX1-NEXT:    retq
+;
+; X64-AVX512-LABEL: ashr_64bytes:
+; X64-AVX512:       # %bb.0:
+; X64-AVX512-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX512-NEXT:    vmovups 32(%rdi), %xmm1
+; X64-AVX512-NEXT:    movq 48(%rdi), %rax
+; X64-AVX512-NEXT:    movq 56(%rdi), %rcx
+; X64-AVX512-NEXT:    movl (%rsi), %esi
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    vmovups %xmm1, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    vmovups %ymm0, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    sarq $63, %rcx
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; X64-AVX512-NEXT:    andl $63, %esi
+; X64-AVX512-NEXT:    vmovups -128(%rsp,%rsi), %zmm0
+; X64-AVX512-NEXT:    vmovups %zmm0, (%rdx)
+; X64-AVX512-NEXT:    vzeroupper
+; X64-AVX512-NEXT:    retq
 ;
 ; X32-SSE2-LABEL: ashr_64bytes:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    pushl %ebp
 ; X32-SSE2-NEXT:    pushl %ebx
 ; X32-SSE2-NEXT:    pushl %edi
 ; X32-SSE2-NEXT:    pushl %esi
 ; X32-SSE2-NEXT:    subl $168, %esp

 ; X32-SSE42-NEXT:    movups %xmm1, 16(%eax)
 ; X32-SSE42-NEXT:    movups %xmm0, (%eax)
 ; X32-SSE42-NEXT:    addl $128, %esp
 ; X32-SSE42-NEXT:    popl %esi
 ; X32-SSE42-NEXT:    popl %edi
 ; X32-SSE42-NEXT:    popl %ebx
 ; X32-SSE42-NEXT:    retl
 ;
-; X32-AVX-LABEL: ashr_64bytes:
-; X32-AVX:       # %bb.0:
-; X32-AVX-NEXT:    pushl %ebx
-; X32-AVX-NEXT:    pushl %edi
-; X32-AVX-NEXT:    pushl %esi
-; X32-AVX-NEXT:    subl $128, %esp
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-AVX-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X32-AVX-NEXT:    vmovups (%edx), %ymm0
-; X32-AVX-NEXT:    vmovups 32(%edx), %xmm1
-; X32-AVX-NEXT:    movl 48(%edx), %esi
-; X32-AVX-NEXT:    movl 52(%edx), %edi
-; X32-AVX-NEXT:    movl 56(%edx), %ebx
-; X32-AVX-NEXT:    movl 60(%edx), %edx
-; X32-AVX-NEXT:    movl (%ecx), %ecx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %esi, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    vmovups %ymm0, (%esp)
-; X32-AVX-NEXT:    sarl $31, %edx
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    movl %edx, {{[0-9]+}}(%esp)
-; X32-AVX-NEXT:    andl $63, %ecx
-; X32-AVX-NEXT:    vmovups (%esp,%ecx), %xmm0
-; X32-AVX-NEXT:    vmovups 16(%esp,%ecx), %xmm1
-; X32-AVX-NEXT:    vmovups 32(%esp,%ecx), %xmm2
-; X32-AVX-NEXT:    vmovups 48(%esp,%ecx), %xmm3
-; X32-AVX-NEXT:    vmovups %xmm3, 48(%eax)
-; X32-AVX-NEXT:    vmovups %xmm2, 32(%eax)
-; X32-AVX-NEXT:    vmovups %xmm1, 16(%eax)
-; X32-AVX-NEXT:    vmovups %xmm0, (%eax)
-; X32-AVX-NEXT:    addl $128, %esp
-; X32-AVX-NEXT:    popl %esi
-; X32-AVX-NEXT:    popl %edi
-; X32-AVX-NEXT:    popl %ebx
-; X32-AVX-NEXT:    vzeroupper
-; X32-AVX-NEXT:    retl
+; X32-AVX1-LABEL: ashr_64bytes:
+; X32-AVX1:       # %bb.0:
+; X32-AVX1-NEXT:    pushl %ebx
+; X32-AVX1-NEXT:    pushl %edi
+; X32-AVX1-NEXT:    pushl %esi
+; X32-AVX1-NEXT:    subl $128, %esp
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX1-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX1-NEXT:    vmovups (%edx), %ymm0
+; X32-AVX1-NEXT:    vmovups 32(%edx), %xmm1
+; X32-AVX1-NEXT:    movl 48(%edx), %esi
+; X32-AVX1-NEXT:    movl 52(%edx), %edi
+; X32-AVX1-NEXT:    movl 56(%edx), %ebx
+; X32-AVX1-NEXT:    movl 60(%edx), %edx
+; X32-AVX1-NEXT:    movl (%ecx), %ecx
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    vmovups %ymm0, (%esp)
+; X32-AVX1-NEXT:    sarl $31, %edx
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX1-NEXT:    andl $63, %ecx
+; X32-AVX1-NEXT:    vmovups (%esp,%ecx), %ymm0
+; X32-AVX1-NEXT:    vmovups 32(%esp,%ecx), %ymm1
+; X32-AVX1-NEXT:    vmovups %ymm1, 32(%eax)
+; X32-AVX1-NEXT:    vmovups %ymm0, (%eax)
+; X32-AVX1-NEXT:    addl $128, %esp
+; X32-AVX1-NEXT:    popl %esi
+; X32-AVX1-NEXT:    popl %edi
+; X32-AVX1-NEXT:    popl %ebx
+; X32-AVX1-NEXT:    vzeroupper
+; X32-AVX1-NEXT:    retl
+;
+; X32-AVX512-LABEL: ashr_64bytes:
+; X32-AVX512:       # %bb.0:
+; X32-AVX512-NEXT:    pushl %ebx
+; X32-AVX512-NEXT:    pushl %edi
+; X32-AVX512-NEXT:    pushl %esi
+; X32-AVX512-NEXT:    subl $128, %esp
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X32-AVX512-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X32-AVX512-NEXT:    vmovups (%edx), %ymm0
+; X32-AVX512-NEXT:    vmovups 32(%edx), %xmm1
+; X32-AVX512-NEXT:    movl 48(%edx), %esi
+; X32-AVX512-NEXT:    movl 52(%edx), %edi
+; X32-AVX512-NEXT:    movl 56(%edx), %ebx
+; X32-AVX512-NEXT:    movl 60(%edx), %edx
+; X32-AVX512-NEXT:    movl (%ecx), %ecx
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %ebx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %esi, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    vmovups %xmm1, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    vmovups %ymm0, (%esp)
+; X32-AVX512-NEXT:    sarl $31, %edx
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    movl %edx, {{[0-9]+}}(%esp)
+; X32-AVX512-NEXT:    andl $63, %ecx
+; X32-AVX512-NEXT:    vmovups (%esp,%ecx), %zmm0
+; X32-AVX512-NEXT:    vmovups %zmm0, (%eax)
+; X32-AVX512-NEXT:    addl $128, %esp
+; X32-AVX512-NEXT:    popl %esi
+; X32-AVX512-NEXT:    popl %edi
+; X32-AVX512-NEXT:    popl %ebx
+; X32-AVX512-NEXT:    vzeroupper
+; X32-AVX512-NEXT:    retl
   %src = load i512, ptr %src.ptr, align 1
   %byteOff = load i512, ptr %byteOff.ptr, align 1
   %bitOff = shl i512 %byteOff, 3
   %res = ashr i512 %src, %bitOff
   store i512 %res, ptr %dst, align 1
   ret void
 }
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line: