[DAG] Support store merging of vector constant stores

[reames]

Summary

Ran across this when making a change to RISCV memset lowering. Seems very odd that manually merging a store into a vector prevents it from being further merged.

p.s. I'm making this depend on the RISCV mem* lowering, but this could easily be rebased off that change set. If reviewers would prefer to separate, let me know.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

702

I believe this function allows undef elements. Does that case work ok?

[reames]

Add test coverage for case mentioned in review comment.

[reames]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

702

Seems to. I added x86 coverage for these cases, and got reasonable results.

It does seem like maybe we should be consistent in allowing scalar undefs in our constant determination, but that looks like a bigger change as we have to add Undef to the lattice when merging. I guess I could start with always considering undef constant - which is basically what this change does - but either option feels like it should be a separate change.

[craig.topper]

LGTM

[bjope]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19406

Just a heads up. Found some problems with this downstream.

When it crashes it tries to merge these stores:

t218: v4i32 = BUILD_VECTOR Constant:i32<0>, Constant:i32<0>, Constant:i32<0>, Constant:i32<0>
t221: ch = store<(store (s64) into %ir.arrayinit.start, align 1, !tbaa <0x557ab8701858>), trunc to v4i16> t4, t218, t104, undef:i16
t219: v2i32 = BUILD_VECTOR Constant:i32<0>, Constant:i32<0>
t194: ch = store<(store (s64) into %fixed-stack.0 + 15, align 1)> t4, t219, t102, undef:i16

So the first store is a v4i32->v4i16 trunc store.

When we arrive on this line (19406) we got MemVT=v2i32 but Val is t218 from above (so that is a v4i32).
I think the correct thing would be to first truncate to v4i16 and then bitcast to v2i32 (given that we have a trunc store). And maybe the same bug exist for the IsConstantSrc false path as well.

I'll see if I can create an upstream reproducer for this. I only see it downstream in certain configs depending on which vector types that are legal, so not sure how easy it will be.

[bjope]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19406

I find it quite hard to find an in-tree reproducer.

I think I need to fulfil UseVector = (LastLegalVectorType > LastLegalType) && AllowVectors to trigger this path. And then for example TLI.storeOfVectorConstantIsCheap need to return true in order to set LastLegalVectorType (limiting the amount of possible targets a bit). And then it depends on which vector types that are legal. And if I for example use RISCV as target it seems like it folds the truncate of a build vector, rather than introducing a truncating store.

[bjope]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19406

Finally some luck:

; RUN: llc -mtriple=x86_64-unknown-unknown -mcpu=skx -o - %s
define void @foo(ptr %p, ptr %q) {
  %v = insertelement <8 x i64> zeroinitializer, i64 0, i32 1
  %tranc = trunc <8 x i64> %v to <8 x i8>
  %p1 = getelementptr i8, ptr %p, i32 0
  %p2 = getelementptr i8, ptr %p, i32 8
  %p3 = getelementptr i8, ptr %p, i32 128
  store <8 x i8> %tranc, ptr %p1
  store <8 x i8> zeroinitializer, ptr %p2
  ret void
}

[bjope]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

19406

I wrote https://github.com/llvm/llvm-project/issues/64593

[reames]

@bjope I relanded with the fix for your reported issue since it turned out to be fairly simple. Can you confirm that everything works on your original example?

Also, for future reference, feel free to just revert in a case like this. You don't need to wait for me to see the comments. Clear compiler crash or regression with a patch which reverts cleanly - go for it. You're doing the author a favor.

[bjope]

In D156349#4577235, @reames wrote:

@bjope I relanded with the fix for your reported issue since it turned out to be fairly simple. Can you confirm that everything works on your original example?

Also, for future reference, feel free to just revert in a case like this. You don't need to wait for me to see the comments. Clear compiler crash or regression with a patch which reverts cleanly - go for it. You're doing the author a favor.

@reames: Ok, great!, Thanks a lot, and I can confirm that the original test case now pass with the updated patch.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
 
     // Classify the origin of a stored value.
     enum class StoreSource { Unknown, Constant, Extract, Load };
     StoreSource getStoreSource(SDValue StoreVal) {
       switch (StoreVal.getOpcode()) {
       case ISD::Constant:
       case ISD::ConstantFP:
         return StoreSource::Constant;
+      case ISD::BUILD_VECTOR:
+        if (ISD::isBuildVectorOfConstantSDNodes(StoreVal.getNode()) ||
+            ISD::isBuildVectorOfConstantFPSDNodes(StoreVal.getNode()))
+          return StoreSource::Constant;
+        return StoreSource::Unknown;
       case ISD::EXTRACT_VECTOR_ELT:
       case ISD::EXTRACT_SUBVECTOR:
         return StoreSource::Extract;
       case ISD::LOAD:
         return StoreSource::Load;
       default:
         return StoreSource::Unknown;
       }
if (IsConstantSrc) {
 
             if (auto *C = dyn_cast<ConstantSDNode>(Val))
               Val = DAG.getConstant(C->getAPIntValue()
                                         .zextOrTrunc(Val.getValueSizeInBits())
                                         .zextOrTrunc(ElementSizeBits),
                                     SDLoc(C), IntMemVT);
           }
           // Make sure correctly size type is the correct type.
           Val = DAG.getBitcast(MemVT, Val);
         }
         BuildVector.push_back(Val);
       }
       StoredVal = DAG.getNode(MemVT.isVector() ? ISD::CONCAT_VECTORS
                                                : ISD::BUILD_VECTOR,
                               DL, StoreTy, BuildVector);
     } else {
       SmallVector<SDValue, 8> Ops;
for (unsigned i = 0; i < NumStores; ++i) {
       } else if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(Val)) {
         StoreInt |= C->getValueAPF()
                         .bitcastToAPInt()
                         .zextOrTrunc(ElementSizeBits)
                         .zextOrTrunc(SizeInBits);
         // If fp truncation is necessary give up for now.
         if (MemVT.getSizeInBits() != ElementSizeBits)
           return false;
+      } else if (ISD::isBuildVectorOfConstantSDNodes(Val.getNode()) ||
+                 ISD::isBuildVectorOfConstantFPSDNodes(Val.getNode())) {
+        // Not yet handled
+        return false;
       } else {
         llvm_unreachable("Invalid constant element type");
       }
     }
 
     // Create the new Load and Store operations.
     StoredVal = DAG.getConstant(StoreInt, DL, StoreTy);
   }
case StoreSource::Load: {
         return false;
       if (!(LBasePtr.equalBaseIndex(LPtr, DAG)))
         return false;
       break;
     }
     case StoreSource::Constant:
       if (NoTypeMatch)
         return false;
-      if (!isIntOrFPConstant(OtherBC))
+      if (getStoreSource(OtherBC) != StoreSource::Constant)
         return false;
       break;
     case StoreSource::Extract:
       // Do not merge truncated stores here.
       if (Other->isTruncatingStore())
         return false;
       if (!MemVT.bitsEq(OtherBC.getValueType()))
         return false;
while (NumConsecutiveStores >= 2) {
     for (unsigned i = 0; i < NumConsecutiveStores; ++i) {
       StoreSDNode *ST = cast<StoreSDNode>(StoreNodes[i].MemNode);
       SDValue StoredVal = ST->getValue();
       bool IsElementZero = false;
       if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(StoredVal))
         IsElementZero = C->isZero();
       else if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(StoredVal))
         IsElementZero = C->getConstantFPValue()->isNullValue();
+      else if (ISD::isBuildVectorAllZeros(StoredVal.getNode()))
+        IsElementZero = true;
       if (IsElementZero) {
         if (NonZero && FirstZeroAfterNonZero == NumConsecutiveStores)
           FirstZeroAfterNonZero = i;
       }
       NonZero |= !IsElementZero;
 
       // Find a legal type for the constant store.
       unsigned SizeInBits = (i + 1) * ElementSizeBytes * 8;

llvm/test/CodeGen/RISCV/rvv/memset-inline.ll

 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 0, i64 32, i1 0)
   ret void
 }
 
 define void @bzero_64(ptr %a) nounwind {
 ; RV32-LABEL: bzero_64:
 ; RV32:       # %bb.0:
-; RV32-NEXT:    addi a1, a0, 48
-; RV32-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV32-NEXT:    li a1, 64
+; RV32-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; RV32-NEXT:    vmv.v.i v8, 0
-; RV32-NEXT:    vse8.v v8, (a1)
-; RV32-NEXT:    addi a1, a0, 32
-; RV32-NEXT:    vse8.v v8, (a1)
-; RV32-NEXT:    addi a1, a0, 16
-; RV32-NEXT:    vse8.v v8, (a1)
 ; RV32-NEXT:    vse8.v v8, (a0)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: bzero_64:
 ; RV64:       # %bb.0:
-; RV64-NEXT:    addi a1, a0, 48
-; RV64-NEXT:    vsetivli zero, 16, e8, m1, ta, ma
+; RV64-NEXT:    li a1, 64
+; RV64-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
 ; RV64-NEXT:    vmv.v.i v8, 0
-; RV64-NEXT:    vse8.v v8, (a1)
-; RV64-NEXT:    addi a1, a0, 32
-; RV64-NEXT:    vse8.v v8, (a1)
-; RV64-NEXT:    addi a1, a0, 16
-; RV64-NEXT:    vse8.v v8, (a1)
 ; RV64-NEXT:    vse8.v v8, (a0)
 ; RV64-NEXT:    ret
 ;
 ; RV32-FAST-LABEL: bzero_64:
 ; RV32-FAST:       # %bb.0:
-; RV32-FAST-NEXT:    addi a1, a0, 48
-; RV32-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV32-FAST-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV32-FAST-NEXT:    vmv.v.i v8, 0
-; RV32-FAST-NEXT:    vse64.v v8, (a1)
-; RV32-FAST-NEXT:    addi a1, a0, 32
-; RV32-FAST-NEXT:    vse64.v v8, (a1)
-; RV32-FAST-NEXT:    addi a1, a0, 16
-; RV32-FAST-NEXT:    vse64.v v8, (a1)
 ; RV32-FAST-NEXT:    vse64.v v8, (a0)
 ; RV32-FAST-NEXT:    ret
 ;
 ; RV64-FAST-LABEL: bzero_64:
 ; RV64-FAST:       # %bb.0:
-; RV64-FAST-NEXT:    addi a1, a0, 48
-; RV64-FAST-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-FAST-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-FAST-NEXT:    vmv.v.i v8, 0
-; RV64-FAST-NEXT:    vse64.v v8, (a1)
-; RV64-FAST-NEXT:    addi a1, a0, 32
-; RV64-FAST-NEXT:    vse64.v v8, (a1)
-; RV64-FAST-NEXT:    addi a1, a0, 16
-; RV64-FAST-NEXT:    vse64.v v8, (a1)
 ; RV64-FAST-NEXT:    vse64.v v8, (a0)
 ; RV64-FAST-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr %a, i8 0, i64 64, i1 0)
   ret void
 }
 
 ; /////////////////////////////////////////////////////////////////////////////
 

 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 32 %a, i8 0, i64 32, i1 0)
   ret void
 }
 
 define void @aligned_bzero_64(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_64:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    addi a1, a0, 48
-; RV32-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV32-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 32
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 16
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_64:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    addi a1, a0, 48
-; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-BOTH-NEXT:    vmv.v.i v8, 0
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 32
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 16
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 64, i1 0)
   ret void
 }
 
 define void @aligned_bzero_66(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_66:
 ; RV32-BOTH:       # %bb.0:
 ; RV32-BOTH-NEXT:    sh zero, 64(a0)
-; RV32-BOTH-NEXT:    addi a1, a0, 48
-; RV32-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV32-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 32
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 16
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_66:
 ; RV64-BOTH:       # %bb.0:
 ; RV64-BOTH-NEXT:    sh zero, 64(a0)
-; RV64-BOTH-NEXT:    addi a1, a0, 48
-; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
 ; RV64-BOTH-NEXT:    vmv.v.i v8, 0
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 32
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 16
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 66, i1 0)
   ret void
 }
 
 define void @aligned_bzero_96(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_96:
 ; RV32-BOTH:       # %bb.0:
 ; RV32-BOTH-NEXT:    addi a1, a0, 80
 ; RV32-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    addi a1, a0, 64
 ; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 48
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 32
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 16
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
+; RV32-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV32-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_96:
 ; RV64-BOTH:       # %bb.0:
 ; RV64-BOTH-NEXT:    addi a1, a0, 80
 ; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
 ; RV64-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    addi a1, a0, 64
 ; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 48
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 32
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 16
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
+; RV64-BOTH-NEXT:    vsetivli zero, 8, e64, m4, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 96, i1 0)
   ret void
 }
 
 define void @aligned_bzero_128(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_128:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    addi a1, a0, 112
-; RV32-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV32-BOTH-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
 ; RV32-BOTH-NEXT:    vmv.v.i v8, 0
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 96
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 80
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 64
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 48
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 32
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 16
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_128:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    addi a1, a0, 112
-; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
+; RV64-BOTH-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
 ; RV64-BOTH-NEXT:    vmv.v.i v8, 0
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 96
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 80
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 64
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 48
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 32
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 16
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 128, i1 0)
   ret void
 }
 
 define void @aligned_bzero_256(ptr %a) nounwind {
 ; RV32-BOTH-LABEL: aligned_bzero_256:
 ; RV32-BOTH:       # %bb.0:
-; RV32-BOTH-NEXT:    addi a1, a0, 240
-; RV32-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; RV32-BOTH-NEXT:    vmv.v.i v8, 0
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 224
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 208
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 192
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 176
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 160
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 144
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    addi a1, a0, 128
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 112
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 96
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 80
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 64
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 48
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 32
-; RV32-BOTH-NEXT:    vse64.v v8, (a1)
-; RV32-BOTH-NEXT:    addi a1, a0, 16
+; RV32-BOTH-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
+; RV32-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV32-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV32-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV32-BOTH-NEXT:    ret
 ;
 ; RV64-BOTH-LABEL: aligned_bzero_256:
 ; RV64-BOTH:       # %bb.0:
-; RV64-BOTH-NEXT:    addi a1, a0, 240
-; RV64-BOTH-NEXT:    vsetivli zero, 2, e64, m1, ta, ma
-; RV64-BOTH-NEXT:    vmv.v.i v8, 0
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 224
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 208
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 192
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 176
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 160
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 144
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    addi a1, a0, 128
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 112
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 96
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 80
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 64
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 48
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 32
-; RV64-BOTH-NEXT:    vse64.v v8, (a1)
-; RV64-BOTH-NEXT:    addi a1, a0, 16
+; RV64-BOTH-NEXT:    vsetivli zero, 16, e64, m8, ta, ma
+; RV64-BOTH-NEXT:    vmv.v.i v8, 0
 ; RV64-BOTH-NEXT:    vse64.v v8, (a1)
 ; RV64-BOTH-NEXT:    vse64.v v8, (a0)
 ; RV64-BOTH-NEXT:    ret
   tail call void @llvm.memset.inline.p0.i64(ptr align 64 %a, i8 0, i64 256, i1 0)
   ret void
 }

llvm/test/CodeGen/X86/MergeConsecutiveStores.ll

; CHECK-NEXT:    retq
   %store_idx0 = getelementptr inbounds <4 x float>, <4 x float>* %ptr, i64 0
   %store_idx1 = getelementptr inbounds <4 x float>, <4 x float>* %ptr, i64 1
   store <4 x float> %v0, <4 x float>* %store_idx0, align 16
   store <4 x float> %v1, <4 x float>* %store_idx1, align 16
   ret void
 
 }
 
-; Merging vector stores when sourced from a constant vector is not currently handled.
+define void @merge_vec_stores_of_zero(<4 x i32>* %ptr) {
+; CHECK-LABEL: merge_vec_stores_of_zero:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; CHECK-NEXT:    vmovups %ymm0, 48(%rdi)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %idx0 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 3
+  %idx1 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 4
+  store <4 x i32> zeroinitializer, <4 x i32>* %idx0, align 16
+  store <4 x i32> zeroinitializer, <4 x i32>* %idx1, align 16
+  ret void
+}
+
+define void @merge_vec_stores_of_constant_splat(<4 x i32>* %ptr) {
+; CHECK-LABEL: merge_vec_stores_of_constant_splat:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vbroadcastss {{.*#+}} xmm0 = [42,42,42,42]
+; CHECK-NEXT:    vmovaps %xmm0, 48(%rdi)
+; CHECK-NEXT:    vmovaps %xmm0, 64(%rdi)
+; CHECK-NEXT:    retq
+  %idx0 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 3
+  %idx1 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 4
+  store <4 x i32> <i32 42, i32 42, i32 42, i32 42>, <4 x i32>* %idx0, align 16
+  store <4 x i32> <i32 42, i32 42, i32 42, i32 42>, <4 x i32>* %idx1, align 16
+  ret void
+}
+
 define void @merge_vec_stores_of_constants(<4 x i32>* %ptr) {
 ; CHECK-LABEL: merge_vec_stores_of_constants:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; CHECK-NEXT:    vmovaps {{.*#+}} xmm0 = [25,51,45,0]
 ; CHECK-NEXT:    vmovaps %xmm0, 48(%rdi)
+; CHECK-NEXT:    vmovaps {{.*#+}} xmm0 = [0,265,26,0]
 ; CHECK-NEXT:    vmovaps %xmm0, 64(%rdi)
 ; CHECK-NEXT:    retq
   %idx0 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 3
   %idx1 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 4
-  store <4 x i32> <i32 0, i32 0, i32 0, i32 0>, <4 x i32>* %idx0, align 16
-  store <4 x i32> <i32 0, i32 0, i32 0, i32 0>, <4 x i32>* %idx1, align 16
+  store <4 x i32> <i32 25, i32 51, i32 45, i32 0>, <4 x i32>* %idx0, align 16
+  store <4 x i32> <i32 0, i32 265, i32 26, i32 0>, <4 x i32>* %idx1, align 16
   ret void
+}
 
+define void @merge_vec_stores_of_constants_with_undefs(<4 x i32>* %ptr) {
+; CHECK-LABEL: merge_vec_stores_of_constants_with_undefs:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; CHECK-NEXT:    vmovups %ymm0, 48(%rdi)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %idx0 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 3
+  %idx1 = getelementptr inbounds <4 x i32>, <4 x i32>* %ptr, i64 4
+  store <4 x i32> <i32 0, i32 0, i32 0, i32 undef>, <4 x i32>* %idx0, align 16
+  store <4 x i32> <i32 0, i32 undef, i32 0, i32 0>, <4 x i32>* %idx1, align 16
+  ret void
 }
 
 ; This is a minimized test based on real code that was failing.
 ; This should now be merged.
 define void @merge_vec_element_and_scalar_load([6 x i64]* %array) {
 ; CHECK-LABEL: merge_vec_element_and_scalar_load:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmovups (%rdi), %xmm0
; CHECK-NEXT:    retq
   ret void
 }
 
 ; same as @merge_const_store with heterogeneous types.
 define void @merge_const_store_heterogeneous(i32 %count, %struct.C* nocapture %p) nounwind uwtable noinline ssp {
 ; CHECK-LABEL: merge_const_store_heterogeneous:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    testl %edi, %edi
-; CHECK-NEXT:    jle .LBB20_3
+; CHECK-NEXT:    jle .LBB23_3
 ; CHECK-NEXT:  # %bb.1: # %.lr.ph.preheader
 ; CHECK-NEXT:    movabsq $578437695752307201, %rax # imm = 0x807060504030201
 ; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB20_2: # %.lr.ph
+; CHECK-NEXT:  .LBB23_2: # %.lr.ph
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    movq %rax, (%rsi)
 ; CHECK-NEXT:    addq $24, %rsi
 ; CHECK-NEXT:    decl %edi
-; CHECK-NEXT:    jne .LBB20_2
-; CHECK-NEXT:  .LBB20_3: # %._crit_edge
+; CHECK-NEXT:    jne .LBB23_2
+; CHECK-NEXT:  .LBB23_3: # %._crit_edge
 ; CHECK-NEXT:    retq
   %1 = icmp sgt i32 %count, 0
   br i1 %1, label %.lr.ph, label %._crit_edge
 .lr.ph:
   %i.02 = phi i32 [ %7, %.lr.ph ], [ 0, %0 ]
   %.01 = phi %struct.C* [ %8, %.lr.ph ], [ %p, %0 ]
   %2 = getelementptr inbounds %struct.C, %struct.C* %.01, i64 0, i32 0
   store i8 1, i8* %2, align 1

llvm/test/CodeGen/X86/avx512-mask-op.ll

entry:
   %5 = bitcast <8 x i64> %e to <16 x i32>
   %6 = shufflevector <8 x i1> %3, <8 x i1> <i1 false, i1 undef, i1 false, i1 false, i1 false, i1 false, i1 false, i1 false>, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
   %7 = select <16 x i1> %6, <16 x i32> %4, <16 x i32> %5
   %8 = bitcast <16 x i32> %7 to <8 x i64>
   ret <8 x i64> %8
 }
 
 define void @store_v128i1_constant(ptr %R) {
-; CHECK-LABEL: store_v128i1_constant:
-; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    movabsq $-4611686310485172227, %rax ## imm = 0xBFFFFFBBFFFFDFFD
-; CHECK-NEXT:    movq %rax, 8(%rdi)
-; CHECK-NEXT:    movabsq $-2305843576149381123, %rax ## imm = 0xDFFFFF7BFFFFEFFD
-; CHECK-NEXT:    movq %rax, (%rdi)
-; CHECK-NEXT:    retq
+; KNL-LABEL: store_v128i1_constant:
+; KNL:       ## %bb.0: ## %entry
+; KNL-NEXT:    vmovaps {{.*#+}} xmm0 = [61437,65535,65403,57343,57341,65535,65467,49151]
+; KNL-NEXT:    vmovaps %xmm0, (%rdi)
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: store_v128i1_constant:
+; SKX:       ## %bb.0: ## %entry
+; SKX-NEXT:    movabsq $-4611686310485172227, %rax ## imm = 0xBFFFFFBBFFFFDFFD
+; SKX-NEXT:    movq %rax, 8(%rdi)
+; SKX-NEXT:    movabsq $-2305843576149381123, %rax ## imm = 0xDFFFFF7BFFFFEFFD
+; SKX-NEXT:    movq %rax, (%rdi)
+; SKX-NEXT:    retq
+;
+; AVX512BW-LABEL: store_v128i1_constant:
+; AVX512BW:       ## %bb.0: ## %entry
+; AVX512BW-NEXT:    movabsq $-4611686310485172227, %rax ## imm = 0xBFFFFFBBFFFFDFFD
+; AVX512BW-NEXT:    movq %rax, 8(%rdi)
+; AVX512BW-NEXT:    movabsq $-2305843576149381123, %rax ## imm = 0xDFFFFF7BFFFFEFFD
+; AVX512BW-NEXT:    movq %rax, (%rdi)
+; AVX512BW-NEXT:    retq
+;
+; AVX512DQ-LABEL: store_v128i1_constant:
+; AVX512DQ:       ## %bb.0: ## %entry
+; AVX512DQ-NEXT:    vmovaps {{.*#+}} xmm0 = [61437,65535,65403,57343,57341,65535,65467,49151]
+; AVX512DQ-NEXT:    vmovaps %xmm0, (%rdi)
+; AVX512DQ-NEXT:    retq
 ;
 ; X86-LABEL: store_v128i1_constant:
 ; X86:       ## %bb.0: ## %entry
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    vmovaps {{.*#+}} xmm0 = [4294963197,3758096251,4294959101,3221225403]
 ; X86-NEXT:    vmovaps %xmm0, (%eax)
 ; X86-NEXT:    retl
 entry: