[llvm] [CodeGen] Fixed vector halving bug for masked load

[aartbik]

Summary

Given a VL=14 that is enveloped by a proper VL=16, splitting the
masked load using the enveloping halving VL=8/8 should yields
should eventually yield V=8/5. This fixes various assert failures
in getHalfNumVectorElementsVT() and IncrementMemoryAddress().

Note, I suspect similar fixes will be needed for other masked
operations, but for now I send out a fix for masked load only.

Bugzilla issue 45563
https://bugs.llvm.org/show_bug.cgi?id=45563

[craig.topper]

I'm concerned that there might be a way to get HiMemVT to contain 0 elements. I think v17i32 would trigger it. We'd widen to v32i32, then split to v16i32 which will have a memvt of v1i32 and v16i32. Then we'll split the v16i32 pieces. And now we'll try to split that v1i32 memory vt.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

9468

EVTs should be passed by value.

[aartbik]

In D78608#1996059, @craig.topper wrote:

I'm concerned that there might be a way to get HiMemVT to contain 0 elements. I think v17i32 would trigger it. We'd widen to v32i32, then split to v16i32 which will have a memvt of v1i32 and v16i32. Then we'll split the v16i32 pieces. And now we'll try to split that v1i32 memory vt.

But do you agree this solution goes in the right direction?
Do you have any suggestions for the v17i32 case?
Happy to test that tomorrow and think about it some more.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

9468

I mimicked the prototypes of related methods in this file? Are those wrong too?
Happy to change, just checking what best practices are here?

[craig.topper]

In D78608#1996085, @aartbik wrote:

In D78608#1996059, @craig.topper wrote:

I'm concerned that there might be a way to get HiMemVT to contain 0 elements. I think v17i32 would trigger it. We'd widen to v32i32, then split to v16i32 which will have a memvt of v1i32 and v16i32. Then we'll split the v16i32 pieces. And now we'll try to split that v1i32 memory vt.

But do you agree this solution goes in the right direction?
Do you have any suggestions for the v17i32 case?
Happy to test that tomorrow and think about it some more.

I wonder if we should fix this in WidenVecRes_MLOAD by doing something closer to WidenVecRes_LOAD. Finding a legal type and slicing up the vector as we widen it.

[aartbik]

do not generate zero storage size masked loads

[aartbik]

PTAL

You had a very sharp eye that v17f32 would cause a split with an empty hi masked load eventually.
It is somewhat harmless since the load is not used and thus removed from the DAG, but this revision
is even cleaner by not generating the hi masked load at all!

[aartbik]

removed left behind debug code

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

9526

Stray 'k' got added to this line.

[aartbik]

fixed some stray diffs

[dmgreen]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

1596

Could we return PassThruHi directly?

llvm/test/CodeGen/X86/pr45563-2.ll

2

Please use update_llc_test_checks on tests this size

[aartbik]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

1596

I experimented with different values, this seemed the least code change, since it folds directly in the DAG.getNode below, and is removed right after that. Any thing else would require more changes to deal with the 1 vs 2 result values.

llvm/test/CodeGen/X86/pr45563-2.ll

2

Ah, I saw some auto generated messages, but was not aware of this tool. But, forgive my noobness, what will this auto-generate? I am really just counting the number of masked moves that result, I don't want to pin down the full assembly?

[craig.topper]

llvm/test/CodeGen/X86/pr45563-2.ll

2

It will generate checks for the full assembly, but the X86 maintainers are used to that. We'll just rerun the script and look at the diff if it fails for some change in the future.

[aartbik]

used update_llc_test_checks (cool!)

[aartbik]

llvm/test/CodeGen/X86/pr45563-2.ll

2

Nice utility! Having the full assembly makes it a bit sensitive to changes, but I can also see that the X86 maintainers want to be aware of even the slightest of changes and with this utility that becomes a lot simpler. Cool!

[craig.topper]

LGTM

llvm/include/llvm/CodeGen/SelectionDAG.h

if (auto A = InferPtrAlign(Ptr))
       return A->value();
     return 0;
   }
 
   /// Compute the VTs needed for the low/hi parts of a type
   /// which is split (or expanded) into two not necessarily identical pieces.
   std::pair<EVT, EVT> GetSplitDestVTs(const EVT &VT) const;
 
+  /// Compute the VTs needed for the low/hi parts of a type, dependent on an
+  /// enveloping VT that has been split into two identical pieces. Sets the
+  /// HisIsEmpty flag when hi type has zero storage size.
+  std::pair<EVT, EVT> GetDependentSplitDestVTs(const EVT &VT, const EVT &EnvVT,
+                                               bool *HiIsEmpty) const;
+
   /// Split the vector with EXTRACT_SUBVECTOR using the provides
   /// VTs and return the low/high part.
   std::pair<SDValue, SDValue> SplitVector(const SDValue &N, const SDLoc &DL,
                                           const EVT &LoVT, const EVT &HiVT);
 
   /// Split the vector with EXTRACT_SUBVECTOR and return the low/high part.
   std::pair<SDValue, SDValue> SplitVector(const SDValue &N, const SDLoc &DL) {
     EVT LoVT, HiVT;

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

if (Mask.getOpcode() == ISD::SETCC) {
     if (getTypeAction(Mask.getValueType()) == TargetLowering::TypeSplitVector)
       GetSplitVector(Mask, MaskLo, MaskHi);
     else
       std::tie(MaskLo, MaskHi) = DAG.SplitVector(Mask, dl);
   }
 
   EVT MemoryVT = MLD->getMemoryVT();
   EVT LoMemVT, HiMemVT;
-  std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);
+  bool HiIsEmpty = false;
+  std::tie(LoMemVT, HiMemVT) =
+      DAG.GetDependentSplitDestVTs(MemoryVT, LoVT, &HiIsEmpty);
 
   SDValue PassThruLo, PassThruHi;
   if (getTypeAction(PassThru.getValueType()) == TargetLowering::TypeSplitVector)
     GetSplitVector(PassThru, PassThruLo, PassThruHi);
   else
     std::tie(PassThruLo, PassThruHi) = DAG.SplitVector(PassThru, dl);
 
   MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
       MLD->getPointerInfo(), MachineMemOperand::MOLoad, LoMemVT.getStoreSize(),
       Alignment, MLD->getAAInfo(), MLD->getRanges());
 
   Lo = DAG.getMaskedLoad(LoVT, dl, Ch, Ptr, Offset, MaskLo, PassThruLo, LoMemVT,
                          MMO, MLD->getAddressingMode(), ExtType,
                          MLD->isExpandingLoad());
 
+  if (HiIsEmpty) {
+    // The hi masked load has zero storage size. We therefore simply set it to
+    // the low masked load and rely on subsequent removal from the chain.
+    Hi = Lo;
+  } else {
+    // Generate hi masked load.
     Ptr = TLI.IncrementMemoryAddress(Ptr, MaskLo, dl, LoMemVT, DAG,
                                      MLD->isExpandingLoad());
     unsigned HiOffset = LoMemVT.getStoreSize();
 
     MMO = DAG.getMachineFunction().getMachineMemOperand(
-      MLD->getPointerInfo().getWithOffset(HiOffset), MachineMemOperand::MOLoad,
-      HiMemVT.getStoreSize(), Alignment, MLD->getAAInfo(), MLD->getRanges());
+        MLD->getPointerInfo().getWithOffset(HiOffset),
+        MachineMemOperand::MOLoad, HiMemVT.getStoreSize(), Alignment,
+        MLD->getAAInfo(), MLD->getRanges());
 
-  Hi = DAG.getMaskedLoad(HiVT, dl, Ch, Ptr, Offset, MaskHi, PassThruHi, HiMemVT,
-                         MMO, MLD->getAddressingMode(), ExtType,
+    Hi = DAG.getMaskedLoad(HiVT, dl, Ch, Ptr, Offset, MaskHi, PassThruHi,
+                           HiMemVT, MMO, MLD->getAddressingMode(), ExtType,
                            MLD->isExpandingLoad());
+  }
 
   // Build a factor node to remember that this load is independent of the
   // other one.
   Ch = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1),
                    Hi.getValue(1));
 
   // Legalize the chain result - switch anything that used the old chain to
   // use the new one.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

std::pair<EVT, EVT> SelectionDAG::GetSplitDestVTs(const EVT &VT) const {
   if (!VT.isVector())
     LoVT = HiVT = TLI->getTypeToTransformTo(*getContext(), VT);
   else
     LoVT = HiVT = VT.getHalfNumVectorElementsVT(*getContext());
 
   return std::make_pair(LoVT, HiVT);
 }
 
+/// GetDependentSplitDestVTs - Compute the VTs needed for the low/hi parts of a
+/// type, dependent on an enveloping VT that has been split into two identical
+/// pieces. Sets the HiIsEmpty flag when hi type has zero storage size.
+std::pair<EVT, EVT>
+SelectionDAG::GetDependentSplitDestVTs(const EVT &VT, const EVT &EnvVT,
+                                       bool *HiIsEmpty) const {
+  EVT EltTp = VT.getVectorElementType();
+  bool IsScalable = VT.isScalableVector();
+  // Examples:
+  //   custom VL=8  with enveloping VL=8/8 yields 8/0 (hi empty)
+  //   custom VL=9  with enveloping VL=8/8 yields 8/1
+  //   custom VL=10 with enveloping VL=8/8 yields 8/2
+  //   etc.
+  unsigned VTNumElts = VT.getVectorNumElements();
+  unsigned EnvNumElts = EnvVT.getVectorNumElements();
+  EVT LoVT, HiVT;
+  if (VTNumElts > EnvNumElts) {
+    LoVT = EnvVT;
+    HiVT = EVT::getVectorVT(*getContext(), EltTp, VTNumElts - EnvNumElts,
+                            IsScalable);
+    *HiIsEmpty = false;
+  } else {
+    // Flag that hi type has zero storage size, but return split envelop type
+    // (this would be easier if vector types with zero elements were allowed).
+    LoVT = EVT::getVectorVT(*getContext(), EltTp, VTNumElts, IsScalable);
+    HiVT = EnvVT;
+    *HiIsEmpty = true;
+  }
+  return std::make_pair(LoVT, HiVT);
+}
+
 /// SplitVector - Split the vector with EXTRACT_SUBVECTOR and return the
 /// low/high part.
 std::pair<SDValue, SDValue>
 SelectionDAG::SplitVector(const SDValue &N, const SDLoc &DL, const EVT &LoVT,
                           const EVT &HiVT) {
   assert(LoVT.getVectorNumElements() + HiVT.getVectorNumElements() <=
          N.getValueType().getVectorNumElements() &&
          "More vector elements requested than available!");

 }
 
 void SelectionDAG::ExtractVectorElements(SDValue Op,
                                          SmallVectorImpl<SDValue> &Args,
                                          unsigned Start, unsigned Count,
                                          EVT EltVT) {
   EVT VT = Op.getValueType();
   if (Count == 0)
     Count = VT.getVectorNumElements();
   if (EltVT == EVT())
     EltVT = VT.getVectorElementType();
   SDLoc SL(Op);
   for (unsigned i = Start, e = Start + Count; i != e; ++i) {
     Args.push_back(getNode(ISD::EXTRACT_VECTOR_ELT, SL, EltVT, Op,
                            getVectorIdxConstant(i, SL)));
   }
 }

llvm/test/CodeGen/X86/pr45563-2.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O3 -mtriple=x86_64-linux-generic -mattr=avx < %s | FileCheck %s
+
+; Bug 45563:
+; The SplitVecRes_MLOAD method should split a extended value type
+; according to the halving of the enveloping type to avoid all sorts
+; of inconsistencies downstream. For example for a extended value type
+; with VL=14 and enveloping type VL=16 that is split 8/8, the extended
+; type should be split 8/6 and not 7/7. This also accounts for hi masked
+; load that get zero storage size (and are unused).
+
+define <9 x float> @mload_split9(<9 x i1> %mask, <9 x float>* %addr, <9 x float> %dst) {
+; CHECK-LABEL: mload_split9:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1],xmm6[0],xmm4[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1,2],xmm7[0]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],xmm2[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],xmm3[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; CHECK-NEXT:    vmovd %esi, %xmm2
+; CHECK-NEXT:    vpinsrw $1, %edx, %xmm2, %xmm2
+; CHECK-NEXT:    vpinsrw $2, %ecx, %xmm2, %xmm2
+; CHECK-NEXT:    vpinsrw $3, %r8d, %xmm2, %xmm2
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm3 = xmm2[0],zero,xmm2[1],zero,xmm2[2],zero,xmm2[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrw $4, %r9d, %xmm2, %xmm2
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $5, %ecx, %xmm2, %xmm2
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $6, %ecx, %xmm2, %xmm2
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $7, %ecx, %xmm2, %xmm2
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm2 = xmm2[4],xmm0[4],xmm2[5],xmm0[5],xmm2[6],xmm0[6],xmm2[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm2, %xmm2
+; CHECK-NEXT:    vpsrad $31, %xmm2, %xmm2
+; CHECK-NEXT:    vinsertf128 $1, %xmm2, %ymm3, %ymm2
+; CHECK-NEXT:    vmaskmovps (%rdi), %ymm2, %ymm3
+; CHECK-NEXT:    vblendvps %ymm2, %ymm3, %ymm0, %ymm0
+; CHECK-NEXT:    movzbl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vmovd %ecx, %xmm2
+; CHECK-NEXT:    vpslld $31, %xmm2, %xmm2
+; CHECK-NEXT:    vpsrad $31, %xmm2, %xmm2
+; CHECK-NEXT:    vmaskmovps 32(%rdi), %ymm2, %ymm3
+; CHECK-NEXT:    vblendvps %xmm2, %xmm3, %xmm1, %xmm1
+; CHECK-NEXT:    vmovss %xmm1, 32(%rax)
+; CHECK-NEXT:    vmovaps %ymm0, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %res = call <9 x float> @llvm.masked.load.v9f32.p0v9f32(<9 x float>* %addr, i32 4, <9 x i1>%mask, <9 x float> %dst)
+  ret <9 x float> %res
+}
+
+define <13 x float> @mload_split13(<13 x i1> %mask, <13 x float>* %addr, <13 x float> %dst) {
+; CHECK-LABEL: mload_split13:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1],xmm6[0],xmm4[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1,2],xmm7[0]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],xmm2[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],xmm3[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm2
+; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm0[0,1,2],mem[0]
+; CHECK-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; CHECK-NEXT:    vmovd %esi, %xmm3
+; CHECK-NEXT:    vpinsrw $1, %edx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrw $2, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrw $3, %r8d, %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrw $4, %r9d, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $5, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $6, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $7, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm3
+; CHECK-NEXT:    vmaskmovps (%rdi), %ymm3, %ymm4
+; CHECK-NEXT:    vblendvps %ymm3, %ymm4, %ymm2, %ymm2
+; CHECK-NEXT:    movzbl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vmovd %ecx, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $1, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $2, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $3, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $4, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm5
+; CHECK-NEXT:    vmaskmovps 32(%rdi), %ymm5, %ymm5
+; CHECK-NEXT:    vblendvps %xmm4, %xmm5, %xmm1, %xmm1
+; CHECK-NEXT:    vmovaps %xmm1, 32(%rax)
+; CHECK-NEXT:    vextractf128 $1, %ymm5, %xmm1
+; CHECK-NEXT:    vblendvps %xmm3, %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:    vmovss %xmm0, 48(%rax)
+; CHECK-NEXT:    vmovaps %ymm2, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %res = call <13 x float> @llvm.masked.load.v13f32.p0v13f32(<13 x float>* %addr, i32 4, <13 x i1>%mask, <13 x float> %dst)
+  ret <13 x float> %res
+}
+
+define <14 x float> @mload_split14(<14 x i1> %mask, <14 x float>* %addr, <14 x float> %dst) {
+; CHECK-LABEL: mload_split14:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1],xmm6[0],xmm4[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1,2],xmm7[0]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],xmm2[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],xmm3[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm2
+; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],mem[0]
+; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],mem[0],xmm1[2,3]
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; CHECK-NEXT:    vmovd %esi, %xmm3
+; CHECK-NEXT:    vpinsrw $1, %edx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrw $2, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrw $3, %r8d, %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrw $4, %r9d, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $5, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $6, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $7, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm3
+; CHECK-NEXT:    vmaskmovps (%rdi), %ymm3, %ymm4
+; CHECK-NEXT:    vblendvps %ymm3, %ymm4, %ymm2, %ymm2
+; CHECK-NEXT:    movzbl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vmovd %ecx, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $1, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $2, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $3, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $4, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %ecx
+; CHECK-NEXT:    vpinsrw $5, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm5
+; CHECK-NEXT:    vmaskmovps 32(%rdi), %ymm5, %ymm5
+; CHECK-NEXT:    vextractf128 $1, %ymm5, %xmm6
+; CHECK-NEXT:    vblendvps %xmm3, %xmm6, %xmm1, %xmm1
+; CHECK-NEXT:    vmovlps %xmm1, 48(%rax)
+; CHECK-NEXT:    vblendvps %xmm4, %xmm5, %xmm0, %xmm0
+; CHECK-NEXT:    vmovaps %xmm0, 32(%rax)
+; CHECK-NEXT:    vmovaps %ymm2, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %res = call <14 x float> @llvm.masked.load.v14f32.p0v14f32(<14 x float>* %addr, i32 4, <14 x i1>%mask, <14 x float> %dst)
+  ret <14 x float> %res
+}
+
+define <17 x float> @mload_split17(<17 x i1> %mask, <17 x float>* %addr, <17 x float> %dst) {
+; CHECK-LABEL: mload_split17:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1],xmm6[0],xmm4[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1,2],xmm7[0]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],xmm2[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],xmm3[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm2
+; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],mem[0]
+; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],mem[0],xmm1[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],mem[0],xmm1[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],mem[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm1
+; CHECK-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    movzbl {{[0-9]+}}(%rsp), %edi
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %r10
+; CHECK-NEXT:    vmovd %esi, %xmm3
+; CHECK-NEXT:    vpinsrb $2, %edx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $4, %ecx, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $6, %r8d, %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $8, %r9d, %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $10, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $12, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $14, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm3
+; CHECK-NEXT:    vmaskmovps (%r10), %ymm3, %ymm4
+; CHECK-NEXT:    vblendvps %ymm3, %ymm4, %ymm2, %ymm2
+; CHECK-NEXT:    vmovd {{.*#+}} xmm3 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vpinsrb $2, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $4, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $6, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm4 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $8, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $10, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $12, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpinsrb $14, {{[0-9]+}}(%rsp), %xmm3, %xmm3
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm3 = xmm3[4],xmm0[4],xmm3[5],xmm0[5],xmm3[6],xmm0[6],xmm3[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vinsertf128 $1, %xmm3, %ymm4, %ymm3
+; CHECK-NEXT:    vmaskmovps 32(%r10), %ymm3, %ymm4
+; CHECK-NEXT:    vblendvps %ymm3, %ymm4, %ymm1, %ymm1
+; CHECK-NEXT:    vmovd %edi, %xmm3
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm3 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm3, %xmm3
+; CHECK-NEXT:    vpsrad $31, %xmm3, %xmm3
+; CHECK-NEXT:    vmaskmovps 64(%r10), %ymm3, %ymm4
+; CHECK-NEXT:    vblendvps %xmm3, %xmm4, %xmm0, %xmm0
+; CHECK-NEXT:    vmovss %xmm0, 64(%rax)
+; CHECK-NEXT:    vmovaps %ymm1, 32(%rax)
+; CHECK-NEXT:    vmovaps %ymm2, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %res = call <17 x float> @llvm.masked.load.v17f32.p0v17f32(<17 x float>* %addr, i32 4, <17 x i1>%mask, <17 x float> %dst)
+  ret <17 x float> %res
+}
+
+define <23 x float> @mload_split23(<23 x i1> %mask, <23 x float>* %addr, <23 x float> %dst) {
+; CHECK-LABEL: mload_split23:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1],xmm6[0],xmm4[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm4 = xmm4[0,1,2],xmm7[0]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],xmm2[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],xmm3[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm3
+; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],mem[0]
+; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],mem[0],xmm1[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],mem[0],xmm1[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1,2],mem[0]
+; CHECK-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm2
+; CHECK-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1],mem[0],xmm0[3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],mem[0]
+; CHECK-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],mem[0],xmm1[2,3]
+; CHECK-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0,1],mem[0],xmm1[3]
+; CHECK-NEXT:    movzbl {{[0-9]+}}(%rsp), %edi
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %r10
+; CHECK-NEXT:    vmovd %esi, %xmm4
+; CHECK-NEXT:    vpinsrb $2, %edx, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $4, %ecx, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $6, %r8d, %xmm4, %xmm4
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm5 = xmm4[0],zero,xmm4[1],zero,xmm4[2],zero,xmm4[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpsrad $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpinsrb $8, %r9d, %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $10, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $12, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $14, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm4 = xmm4[4],xmm0[4],xmm4[5],xmm0[5],xmm4[6],xmm0[6],xmm4[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
+; CHECK-NEXT:    vmaskmovps (%r10), %ymm4, %ymm5
+; CHECK-NEXT:    vblendvps %ymm4, %ymm5, %ymm3, %ymm3
+; CHECK-NEXT:    vmovd {{.*#+}} xmm4 = mem[0],zero,zero,zero
+; CHECK-NEXT:    vpinsrb $2, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $4, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $6, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm5 = xmm4[0],zero,xmm4[1],zero,xmm4[2],zero,xmm4[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpsrad $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpinsrb $8, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $10, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $12, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $14, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm4 = xmm4[4],xmm0[4],xmm4[5],xmm0[5],xmm4[6],xmm0[6],xmm4[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm4
+; CHECK-NEXT:    vmaskmovps 32(%r10), %ymm4, %ymm5
+; CHECK-NEXT:    vblendvps %ymm4, %ymm5, %ymm2, %ymm2
+; CHECK-NEXT:    vmovd %edi, %xmm4
+; CHECK-NEXT:    vpinsrb $2, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $4, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $6, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpmovzxwd {{.*#+}} xmm5 = xmm4[0],zero,xmm4[1],zero,xmm4[2],zero,xmm4[3],zero
+; CHECK-NEXT:    vpslld $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpsrad $31, %xmm5, %xmm5
+; CHECK-NEXT:    vpinsrb $8, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $10, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpinsrb $12, {{[0-9]+}}(%rsp), %xmm4, %xmm4
+; CHECK-NEXT:    vpunpckhwd {{.*#+}} xmm4 = xmm4[4],xmm0[4],xmm4[5],xmm0[5],xmm4[6],xmm0[6],xmm4[7],xmm0[7]
+; CHECK-NEXT:    vpslld $31, %xmm4, %xmm4
+; CHECK-NEXT:    vpsrad $31, %xmm4, %xmm4
+; CHECK-NEXT:    vinsertf128 $1, %xmm4, %ymm5, %ymm6
+; CHECK-NEXT:    vmaskmovps 64(%r10), %ymm6, %ymm6
+; CHECK-NEXT:    vmovaps %ymm2, 32(%rax)
+; CHECK-NEXT:    vextractf128 $1, %ymm6, %xmm2
+; CHECK-NEXT:    vblendvps %xmm4, %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:    vextractps $2, %xmm1, 88(%rax)
+; CHECK-NEXT:    vmovlps %xmm1, 80(%rax)
+; CHECK-NEXT:    vblendvps %xmm5, %xmm6, %xmm0, %xmm0
+; CHECK-NEXT:    vmovaps %xmm0, 64(%rax)
+; CHECK-NEXT:    vmovaps %ymm3, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %res = call <23 x float> @llvm.masked.load.v23f32.p0v23f32(<23 x float>* %addr, i32 4, <23 x i1>%mask, <23 x float> %dst)
+  ret <23 x float> %res
+}
+
+declare <9 x float> @llvm.masked.load.v9f32.p0v9f32(<9 x float>* %addr, i32 %align, <9 x i1> %mask, <9 x float> %dst)
+declare <13 x float> @llvm.masked.load.v13f32.p0v13f32(<13 x float>* %addr, i32 %align, <13 x i1> %mask, <13 x float> %dst)
+declare <14 x float> @llvm.masked.load.v14f32.p0v14f32(<14 x float>* %addr, i32 %align, <14 x i1> %mask, <14 x float> %dst)
+declare <17 x float> @llvm.masked.load.v17f32.p0v17f32(<17 x float>* %addr, i32 %align, <17 x i1> %mask, <17 x float> %dst)
+declare <23 x float> @llvm.masked.load.v23f32.p0v23f32(<23 x float>* %addr, i32 %align, <23 x i1> %mask, <23 x float> %dst)