[SDAG] Use UnknownSize for masked load/store MMO size

[dmgreen]

Summary

A masked load or store will load a potentially unknown size of bytes from a memory location - that is not generally known at compile time. They do not necessarily load/store the entire vector width, and treating them as such can lead to incorrect aliasing information (for example, if the underlying object is smaller than the size of the vector).

This makes sure that the MMO is given an unknown size to represent this. which is less accurate that "may load/store from up to 16 bytes", but less incorrect that "will load/store from 16 bytes".

[nikic]

Wouldn't LocationSize::upperBound() be a better fit here? https://github.com/llvm/llvm-project/blob/95102b7dc3c1b5b3f1b688221d9aa28cb1e17974/llvm/include/llvm/Analysis/MemoryLocation.h#L115-L119

[dmgreen]

In D113888#3131285, @nikic wrote:

Wouldn't LocationSize::upperBound() be a better fit here? https://github.com/llvm/llvm-project/blob/95102b7dc3c1b5b3f1b688221d9aa28cb1e17974/llvm/include/llvm/Analysis/MemoryLocation.h#L115-L119

That would be good! Unfortunately the size is really stored as a LLT, which means we shouldn't be passing MemoryLocation::UnknownSize but can't use a LocationSize either.
https://github.com/llvm/llvm-project/blob/95102b7dc3c1b5b3f1b688221d9aa28cb1e17974/llvm/include/llvm/CodeGen/MachineFunction.h#L937
This should be creating an invalid LLT, I'll change that. The LLT gets used in the GlobalISel lowering a fair amount it seems.

[dmgreen]

Create an invalid LLT using LLT().

[efriedma]

They do not necessarily load/store the entire vector width, and treating them as such can lead to incorrect aliasing information (for example, if the underlying object is smaller than the size of the vector).

Does this mean any operation that's a mayload/maystore needs to have the size of the operand set to "unknown"? I suspect we need to do significantly more work than just this patch if we want memory operands to work this way. (e.g. type legalization, if conversion.) What happens if we just say that size of a memory location is supposed to represent an upper bound, at least for now?

we shouldn't be passing MemoryLocation::UnknownSize

If that's true, maybe we need to assert this? Apparently we are actually passing in MemoryLocation::UnknownSize in some places.

[dmgreen]

Does this mean any operation that's a mayload/maystore needs to have the size of the operand set to "unknown"? I suspect we need to do significantly more work than just this patch if we want memory operands to work this way. (e.g. type legalization, if conversion.) What happens if we just say that size of a memory location is supposed to represent an upper bound, at least for now?

Why would it be any operation, not just masked loads/stores? The problem is in this case the underlying object only had 10 dereferenceable chars, where as the entire vector width is 16bytes. I don't think that would apply to any normal load/store - it would already be UB at the llvm-ir level.

It's the equivalent of this code, but there is unfortunately no current way of adding MemoryLocations directly, it goes via a LLT size. Maybe we do want to change that, but I didn't want to try and do it here. It looks quite involved.
https://github.com/llvm/llvm-project/blob/913d78c40c37c9c3428285d868ce454b058e40f3/llvm/lib/Analysis/MemoryLocation.cpp#L165

The exact thing going wrong here is that this code in the scheduler is adding chain dependencies:
https://github.com/llvm/llvm-project/blob/913d78c40c37c9c3428285d868ce454b058e40f3/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp#L942
Which checks this AA check, which returns false, I think from basic-aa:
https://github.com/llvm/llvm-project/blob/913d78c40c37c9c3428285d868ce454b058e40f3/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp#L543
So no scheduling dependencies between the instructions and we end up in the wrong order. This started happening after we moved the VPT block pass later so predicated MVE instructions are no long in bundles during the post-ra scheduler.

we shouldn't be passing MemoryLocation::UnknownSize

If that's true, maybe we need to assert this? Apparently we are actually passing in MemoryLocation::UnknownSize in some places.

Sure that sounds good, I can take a look. It might be simpler to make sute the relevant methods work sensibly with MemoryLocation::UnknownSize.

[efriedma]

In D113888#3136093, @dmgreen wrote:

Does this mean any operation that's a mayload/maystore needs to have the size of the operand set to "unknown"? I suspect we need to do significantly more work than just this patch if we want memory operands to work this way. (e.g. type legalization, if conversion.) What happens if we just say that size of a memory location is supposed to represent an upper bound, at least for now?

Why would it be any operation, not just masked loads/stores? The problem is in this case the underlying object only had 10 dereferenceable chars, where as the entire vector width is 16bytes. I don't think that would apply to any normal load/store - it would already be UB at the llvm-ir level.

It's basically "masked" load/store operations. At the IR level, those are rare, sure. At the machine code level, there are operations other than llvm.masked.load that end up masked, though. For example, the ARM instruction "strne". Not sure how many places can end up with instructions like that off the top of my head.

The type legalization for llvm.masked.load also has the same issue as SelectionDAGBuilder.

[dmgreen]

It's basically "masked" load/store operations. At the IR level, those are rare, sure. At the machine code level, there are operations other than llvm.masked.load that end up masked, though. For example, the ARM instruction "strne". Not sure how many places can end up with instructions like that off the top of my head.

I don't believe that is how it works though, not exactly. In this case the backend is constructing a query of the form "does (%1, 16) alias (%arrayidx4, 1)". And because the underlying object of %arrayidx4 is only 10 bytes large, the answer is no as the alternative would be UB. But we are using the aliasing information at the IR level, not what the mir has become. If the mir has been altered to the point that the llvm-lr level aliasing info no longer applies then yes that would be invalid, but I haven't seen that happen anywhere. A predicated store (strne) still uses the llvm-ir level aliasing info which will be valid if the original store was in an if block (or however it was predicated). And so long as that aliasing query remains valid at the ir level, the size of the predicated store's memory location should be OK.

At least - I don't have an example of that going wrong. If you do know of a way to make it act incorrectly, let me know.

The type legalization for llvm.masked.load also has the same issue as SelectionDAGBuilder.

What do you mean by "type legalization" here exactly? I don't think global-isel handles masked loads/stores yet, and most of the illegal masked operation lowering is performed pre-isel. I wasn't sure what other type legalization you were referring to.

[dmgreen]

Updated getMachineMemOperand overload to handle -1 sizes correctly.

[craig.topper]

In D113888#3139874, @dmgreen wrote:

It's basically "masked" load/store operations. At the IR level, those are rare, sure. At the machine code level, there are operations other than llvm.masked.load that end up masked, though. For example, the ARM instruction "strne". Not sure how many places can end up with instructions like that off the top of my head.

I don't believe that is how it works though, not exactly. In this case the backend is constructing a query of the form "does (%1, 16) alias (%arrayidx4, 1)". And because the underlying object of %arrayidx4 is only 10 bytes large, the answer is no as the alternative would be UB. But we are using the aliasing information at the IR level, not what the mir has become. If the mir has been altered to the point that the llvm-lr level aliasing info no longer applies then yes that would be invalid, but I haven't seen that happen anywhere. A predicated store (strne) still uses the llvm-ir level aliasing info which will be valid if the original store was in an if block (or however it was predicated). And so long as that aliasing query remains valid at the ir level, the size of the predicated store's memory location should be OK.

At least - I don't have an example of that going wrong. If you do know of a way to make it act incorrectly, let me know.

The type legalization for llvm.masked.load also has the same issue as SelectionDAGBuilder.

What do you mean by "type legalization" here exactly? I don't think global-isel handles masked loads/stores yet, and most of the illegal masked operation lowering is performed pre-isel. I wasn't sure what other type legalization you were referring to.

I believe Eli was referring to functions like DAGTypeLegalizer::SplitVecRes_MLOAD and DAGTypeLegalizer::SplitVecOp_MSTORE.

[dmgreen]

I believe Eli was referring to functions like DAGTypeLegalizer::SplitVecRes_MLOAD and DAGTypeLegalizer::SplitVecOp_MSTORE.

I see. Because it's recreating a MMO with the new widths of the vector, not using the existing MMO. Thanks.

[dmgreen]

Update DAGTypeLegalizer::SplitVecRes_MLOAD, DAGTypeLegalizer::SplitVecOp_MSTORE and HexagonTargetLowering::SplitHvxMemOp, which now shows some other differences in other tests. The other uses of getMaskedLoad/store looked OK from what I could see.

[efriedma]

LGTM

[dmgreen]

Thanks

[yubing]

Hi, @dmgreen With this patch, the llc command crashed. Would you take a look?
llc -mcpu=core-avx2 main.ll

define void @main() unnamed_addr #0 {
entry:
  %P.i150.i.i = alloca [3 x [3 x double]], align 16
  %0 = bitcast [3 x [3 x double]]* %P.i150.i.i to <8 x double>*
  call void @llvm.masked.store.v8f64.p0v8f64(<8 x double> zeroinitializer, <8 x double>* %0, i32 8, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 false, i1 false>)
  ret void
}

; Function Attrs: argmemonly nofree nosync nounwind willreturn writeonly
declare void @llvm.masked.store.v8f64.p0v8f64(<8 x double>, <8 x double>*, i32 immarg, <8 x i1>)

[dmgreen]

In D113888#3152950, @yubing wrote:

Hi, @dmgreen With this patch, the llc command crashed. Would you take a look?
llc -mcpu=core-avx2 main.ll

define void @main() unnamed_addr #0 {
entry:
  %P.i150.i.i = alloca [3 x [3 x double]], align 16
  %0 = bitcast [3 x [3 x double]]* %P.i150.i.i to <8 x double>*
  call void @llvm.masked.store.v8f64.p0v8f64(<8 x double> zeroinitializer, <8 x double>* %0, i32 8, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 false, i1 false>)
  ret void
}

; Function Attrs: argmemonly nofree nosync nounwind willreturn writeonly
declare void @llvm.masked.store.v8f64.p0v8f64(<8 x double>, <8 x double>*, i32 immarg, <8 x i1>)

Thanks. Seems that a masked store is turned back into an (unmasked) store, still with an unknown size. The assert seems superfluous, but I will try and fix the size when we promote the masked store to a store too. Thanks for the simple reproducer.

llvm/include/llvm/CodeGen/MachineFunction.h

public:
   /// getMachineMemOperand - Allocate a new MachineMemOperand by copying
   /// an existing one, adjusting by an offset and using the given size.
   /// MachineMemOperands are owned by the MachineFunction and need not be
   /// explicitly deallocated.
   MachineMemOperand *getMachineMemOperand(const MachineMemOperand *MMO,
                                           int64_t Offset, LLT Ty);
   MachineMemOperand *getMachineMemOperand(const MachineMemOperand *MMO,
                                           int64_t Offset, uint64_t Size) {
-    return getMachineMemOperand(MMO, Offset, LLT::scalar(8 * Size));
+    return getMachineMemOperand(
+        MMO, Offset, Size == ~UINT64_C(0) ? LLT() : LLT::scalar(8 * Size));
   }
 
   /// getMachineMemOperand - Allocate a new MachineMemOperand by copying
   /// an existing one, replacing only the MachinePointerInfo and size.
   /// MachineMemOperands are owned by the MachineFunction and need not be
   /// explicitly deallocated.
   MachineMemOperand *getMachineMemOperand(const MachineMemOperand *MMO,
                                           const MachinePointerInfo &PtrInfo,

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

std::tie(LoMemVT, HiMemVT) =
       DAG.GetDependentSplitDestVTs(MemoryVT, LoVT, &HiIsEmpty);
 
   SDValue PassThruLo, PassThruHi;
   if (getTypeAction(PassThru.getValueType()) == TargetLowering::TypeSplitVector)
     GetSplitVector(PassThru, PassThruLo, PassThruHi);
   else
     std::tie(PassThruLo, PassThruHi) = DAG.SplitVector(PassThru, dl);
 
-  unsigned LoSize = MemoryLocation::getSizeOrUnknown(LoMemVT.getStoreSize());
   MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
-      MLD->getPointerInfo(), MachineMemOperand::MOLoad, LoSize, Alignment,
-      MLD->getAAInfo(), MLD->getRanges());
+      MLD->getPointerInfo(), MachineMemOperand::MOLoad,
+      MemoryLocation::UnknownSize, Alignment, MLD->getAAInfo(),
+      MLD->getRanges());
 
   Lo = DAG.getMaskedLoad(LoVT, dl, Ch, Ptr, Offset, MaskLo, PassThruLo, LoMemVT,
                          MMO, MLD->getAddressingMode(), ExtType,
                          MLD->isExpandingLoad());
 
   if (HiIsEmpty) {
     // The hi masked load has zero storage size. We therefore simply set it to
     // the low masked load and rely on subsequent removal from the chain.
     Hi = Lo;
   } else {
     // Generate hi masked load.
     Ptr = TLI.IncrementMemoryAddress(Ptr, MaskLo, dl, LoMemVT, DAG,
                                      MLD->isExpandingLoad());
-    unsigned HiSize = MemoryLocation::getSizeOrUnknown(HiMemVT.getStoreSize());
 
     MachinePointerInfo MPI;
     if (LoMemVT.isScalableVector())
       MPI = MachinePointerInfo(MLD->getPointerInfo().getAddrSpace());
     else
       MPI = MLD->getPointerInfo().getWithOffset(
           LoMemVT.getStoreSize().getFixedSize());
 
     MMO = DAG.getMachineFunction().getMachineMemOperand(
-        MPI, MachineMemOperand::MOLoad, HiSize, Alignment, MLD->getAAInfo(),
-        MLD->getRanges());
+        MPI, MachineMemOperand::MOLoad, MemoryLocation::UnknownSize, Alignment,
+        MLD->getAAInfo(), MLD->getRanges());
 
     Hi = DAG.getMaskedLoad(HiVT, dl, Ch, Ptr, Offset, MaskHi, PassThruHi,
                            HiMemVT, MMO, MLD->getAddressingMode(), ExtType,
                            MLD->isExpandingLoad());
   }
 
   // Build a factor node to remember that this load is independent of the
   // other one.
SDValue DAGTypeLegalizer::SplitVecOp_MSTORE(MaskedStoreSDNode *N,
 
   EVT MemoryVT = N->getMemoryVT();
   EVT LoMemVT, HiMemVT;
   bool HiIsEmpty = false;
   std::tie(LoMemVT, HiMemVT) =
       DAG.GetDependentSplitDestVTs(MemoryVT, DataLo.getValueType(), &HiIsEmpty);
 
   SDValue Lo, Hi, Res;
-  unsigned LoSize = MemoryLocation::getSizeOrUnknown(LoMemVT.getStoreSize());
   MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
-      N->getPointerInfo(), MachineMemOperand::MOStore, LoSize, Alignment,
-      N->getAAInfo(), N->getRanges());
+      N->getPointerInfo(), MachineMemOperand::MOStore,
+      MemoryLocation::UnknownSize, Alignment, N->getAAInfo(), N->getRanges());
 
   Lo = DAG.getMaskedStore(Ch, DL, DataLo, Ptr, Offset, MaskLo, LoMemVT, MMO,
                           N->getAddressingMode(), N->isTruncatingStore(),
                           N->isCompressingStore());
 
   if (HiIsEmpty) {
     // The hi masked store has zero storage size.
     // Only the lo masked store is needed.
     Res = Lo;
   } else {
 
     Ptr = TLI.IncrementMemoryAddress(Ptr, MaskLo, DL, LoMemVT, DAG,
                                      N->isCompressingStore());
 
     MachinePointerInfo MPI;
     if (LoMemVT.isScalableVector()) {
       Alignment = commonAlignment(
           Alignment, LoMemVT.getSizeInBits().getKnownMinSize() / 8);
       MPI = MachinePointerInfo(N->getPointerInfo().getAddrSpace());
     } else
       MPI = N->getPointerInfo().getWithOffset(
           LoMemVT.getStoreSize().getFixedSize());
 
-    unsigned HiSize = MemoryLocation::getSizeOrUnknown(HiMemVT.getStoreSize());
     MMO = DAG.getMachineFunction().getMachineMemOperand(
-        MPI, MachineMemOperand::MOStore, HiSize, Alignment, N->getAAInfo(),
-        N->getRanges());
+        MPI, MachineMemOperand::MOStore, MemoryLocation::UnknownSize, Alignment,
+        N->getAAInfo(), N->getRanges());
 
     Hi = DAG.getMaskedStore(Ch, DL, DataHi, Ptr, Offset, MaskHi, HiMemVT, MMO,
                             N->getAddressingMode(), N->isTruncatingStore(),
                             N->isCompressingStore());
 
     // Build a factor node to remember that this store is independent of the
     // other one.
     Res = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo, Hi);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitMaskedStore(const CallInst &I,
   SDValue Offset = DAG.getUNDEF(Ptr.getValueType());
 
   EVT VT = Src0.getValueType();
   if (!Alignment)
     Alignment = DAG.getEVTAlign(VT);
 
   MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
       MachinePointerInfo(PtrOperand), MachineMemOperand::MOStore,
-      // TODO: Make MachineMemOperands aware of scalable
-      // vectors.
-      VT.getStoreSize().getKnownMinSize(), *Alignment, I.getAAMetadata());
+      MemoryLocation::UnknownSize, *Alignment, I.getAAMetadata());
   SDValue StoreNode =
       DAG.getMaskedStore(getMemoryRoot(), sdl, Src0, Ptr, Offset, Mask, VT, MMO,
                          ISD::UNINDEXED, false /* Truncating */, IsCompressing);
   DAG.setRoot(StoreNode);
   setValue(&I, StoreNode);
 }
 
 // Get a uniform base for the Gather/Scatter intrinsic.
void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
   EVT VT = Src0.getValueType();
   if (!Alignment)
     Alignment = DAG.getEVTAlign(VT);
 
   AAMDNodes AAInfo = I.getAAMetadata();
   const MDNode *Ranges = I.getMetadata(LLVMContext::MD_range);
 
   // Do not serialize masked loads of constant memory with anything.
-  MemoryLocation ML;
-  if (VT.isScalableVector())
-    ML = MemoryLocation::getAfter(PtrOperand);
-  else
-    ML = MemoryLocation(PtrOperand, LocationSize::precise(
-                           DAG.getDataLayout().getTypeStoreSize(I.getType())),
-                           AAInfo);
+  MemoryLocation ML = MemoryLocation::getAfter(PtrOperand, AAInfo);
   bool AddToChain = !AA || !AA->pointsToConstantMemory(ML);
 
   SDValue InChain = AddToChain ? DAG.getRoot() : DAG.getEntryNode();
 
   MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
       MachinePointerInfo(PtrOperand), MachineMemOperand::MOLoad,
-      // TODO: Make MachineMemOperands aware of scalable
-      // vectors.
-      VT.getStoreSize().getKnownMinSize(), *Alignment, AAInfo, Ranges);
+      MemoryLocation::UnknownSize, *Alignment, AAInfo, Ranges);
 
   SDValue Load =
       DAG.getMaskedLoad(VT, sdl, InChain, Ptr, Offset, Mask, Src0, VT, MMO,
                         ISD::UNINDEXED, ISD::NON_EXTLOAD, IsExpanding);
   if (AddToChain)
     PendingLoads.push_back(Load.getValue(1));
   setValue(&I, Load);
 }

llvm/lib/Target/Hexagon/HexagonISelLoweringHVX.cpp

 //===-- HexagonISelLoweringHVX.cpp --- Lowering HVX operations ------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "HexagonISelLowering.h"
 #include "HexagonRegisterInfo.h"
 #include "HexagonSubtarget.h"
+#include "llvm/Analysis/MemoryLocation.h"
 #include "llvm/IR/IntrinsicsHexagon.h"
 #include "llvm/Support/CommandLine.h"
 
 using namespace llvm;
 
 static cl::opt<unsigned> HvxWidenThreshold("hexagon-hvx-widen",
   cl::Hidden, cl::init(16),
   cl::desc("Lower threshold (in bytes) for widening to HVX vectors"));
if (!isHvxPairTy(MemTy))
     return Op;
 
   const SDLoc &dl(Op);
   unsigned HwLen = Subtarget.getVectorLength();
   MVT SingleTy = typeSplit(MemTy).first;
   SDValue Chain = MemN->getChain();
   SDValue Base0 = MemN->getBasePtr();
   SDValue Base1 = DAG.getMemBasePlusOffset(Base0, TypeSize::Fixed(HwLen), dl);
+  unsigned MemOpc = MemN->getOpcode();
 
   MachineMemOperand *MOp0 = nullptr, *MOp1 = nullptr;
   if (MachineMemOperand *MMO = MemN->getMemOperand()) {
     MachineFunction &MF = DAG.getMachineFunction();
-    MOp0 = MF.getMachineMemOperand(MMO, 0, HwLen);
-    MOp1 = MF.getMachineMemOperand(MMO, HwLen, HwLen);
+    uint64_t MemSize = (MemOpc == ISD::MLOAD || MemOpc == ISD::MSTORE)
+                           ? MemoryLocation::UnknownSize
+                           : HwLen;
+    MOp0 = MF.getMachineMemOperand(MMO, 0, MemSize);
+    MOp1 = MF.getMachineMemOperand(MMO, HwLen, MemSize);
   }
 
-  unsigned MemOpc = MemN->getOpcode();
-
   if (MemOpc == ISD::LOAD) {
     assert(cast<LoadSDNode>(Op)->isUnindexed());
     SDValue Load0 = DAG.getLoad(SingleTy, dl, Chain, Base0, MOp0);
     SDValue Load1 = DAG.getLoad(SingleTy, dl, Chain, Base1, MOp1);
     return DAG.getMergeValues(
         { DAG.getNode(ISD::CONCAT_VECTORS, dl, MemTy, Load0, Load1),
           DAG.getNode(ISD::TokenFactor, dl, MVT::Other,
                       Load0.getValue(1), Load1.getValue(1)) }, dl);

llvm/test/CodeGen/Thumb2/mve-masked-store-mmo.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve -verify-machineinstrs %s -o - | FileCheck %s
 
 define i32 @incorrectmmo() {
 ; CHECK-LABEL: incorrectmmo:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .pad #12
 ; CHECK-NEXT:    sub sp, #12
 ; CHECK-NEXT:    movw r0, #1023
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    adr r0, .LCPI0_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r0]
 ; CHECK-NEXT:    add.w r0, sp, #2
-; CHECK-NEXT:    ldrb.w r1, [sp, #3]
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrbt.8 q0, [r0]
-; CHECK-NEXT:    ldrb.w r2, [sp, #4]
 ; CHECK-NEXT:    ldrb.w r0, [sp, #2]
-; CHECK-NEXT:    ldrb.w r3, [sp, #10]
+; CHECK-NEXT:    ldrb.w r1, [sp, #3]
+; CHECK-NEXT:    ldrb.w r2, [sp, #4]
 ; CHECK-NEXT:    add r0, r1
-; CHECK-NEXT:    ldrb.w r1, [sp, #11]
+; CHECK-NEXT:    ldrb.w r3, [sp, #10]
 ; CHECK-NEXT:    add r0, r2
+; CHECK-NEXT:    ldrb.w r1, [sp, #11]
 ; CHECK-NEXT:    add r0, r3
 ; CHECK-NEXT:    add r0, r1
 ; CHECK-NEXT:    add sp, #12
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .byte 0 @ 0x0

llvm/test/CodeGen/X86/masked_compressstore.ll

 ; AVX1OR2-NEXT:    vzeroupper
 ; AVX1OR2-NEXT:    retq
 ;
 ; AVX512F-LABEL: compressstore_v16f64_v16i1:
 ; AVX512F:       ## %bb.0:
 ; AVX512F-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; AVX512F-NEXT:    vpslld $31, %zmm2, %zmm2
 ; AVX512F-NEXT:    vptestmd %zmm2, %zmm2, %k1
-; AVX512F-NEXT:    kshiftrw $8, %k1, %k2
-; AVX512F-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512F-NEXT:    kmovw %k1, %eax
 ; AVX512F-NEXT:    movzbl %al, %eax
 ; AVX512F-NEXT:    movl %eax, %ecx
 ; AVX512F-NEXT:    shrl %ecx
 ; AVX512F-NEXT:    andl $-43, %ecx
 ; AVX512F-NEXT:    subl %ecx, %eax
 ; AVX512F-NEXT:    movl %eax, %ecx
 ; AVX512F-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512F-NEXT:    shrl $2, %eax
 ; AVX512F-NEXT:    andl $858993459, %eax ## imm = 0x33333333
 ; AVX512F-NEXT:    addl %ecx, %eax
 ; AVX512F-NEXT:    movl %eax, %ecx
 ; AVX512F-NEXT:    shrl $4, %ecx
 ; AVX512F-NEXT:    addl %eax, %ecx
 ; AVX512F-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
 ; AVX512F-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
 ; AVX512F-NEXT:    shrl $24, %eax
+; AVX512F-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512F-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
+; AVX512F-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512VLDQ-LABEL: compressstore_v16f64_v16i1:
 ; AVX512VLDQ:       ## %bb.0:
 ; AVX512VLDQ-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; AVX512VLDQ-NEXT:    vpslld $31, %zmm2, %zmm2
 ; AVX512VLDQ-NEXT:    vpmovd2m %zmm2, %k1

 ; AVX512VLDQ-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512VLDQ-NEXT:    vzeroupper
 ; AVX512VLDQ-NEXT:    retq
 ;
 ; AVX512VLBW-LABEL: compressstore_v16f64_v16i1:
 ; AVX512VLBW:       ## %bb.0:
 ; AVX512VLBW-NEXT:    vpsllw $7, %xmm2, %xmm2
 ; AVX512VLBW-NEXT:    vpmovb2m %xmm2, %k1
-; AVX512VLBW-NEXT:    kshiftrw $8, %k1, %k2
-; AVX512VLBW-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512VLBW-NEXT:    kmovd %k1, %eax
 ; AVX512VLBW-NEXT:    movzbl %al, %eax
 ; AVX512VLBW-NEXT:    movl %eax, %ecx
 ; AVX512VLBW-NEXT:    shrl %ecx
 ; AVX512VLBW-NEXT:    andl $-43, %ecx
 ; AVX512VLBW-NEXT:    subl %ecx, %eax
 ; AVX512VLBW-NEXT:    movl %eax, %ecx
 ; AVX512VLBW-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512VLBW-NEXT:    shrl $2, %eax
 ; AVX512VLBW-NEXT:    andl $858993459, %eax ## imm = 0x33333333
 ; AVX512VLBW-NEXT:    addl %ecx, %eax
 ; AVX512VLBW-NEXT:    movl %eax, %ecx
 ; AVX512VLBW-NEXT:    shrl $4, %ecx
 ; AVX512VLBW-NEXT:    addl %eax, %ecx
 ; AVX512VLBW-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
 ; AVX512VLBW-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
 ; AVX512VLBW-NEXT:    shrl $24, %eax
+; AVX512VLBW-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512VLBW-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
+; AVX512VLBW-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512VLBW-NEXT:    vzeroupper
 ; AVX512VLBW-NEXT:    retq
   call void @llvm.masked.compressstore.v16f64(<16 x double> %V, double* %base, <16 x i1> %mask)
   ret void
 }
 
 ;
 ; vXf32

llvm/test/CodeGen/X86/vmaskmov-offset.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -mtriple=x86_64-unknown-unknown -mcpu=core-avx2 -stop-after finalize-isel -o - %s | FileCheck %s
 
 declare void @llvm.masked.store.v16f32.p0v16f32(<16 x float>, <16 x float>*, i32, <16 x i1>)
 declare <16 x float> @llvm.masked.load.v16f32.p0v16f32(<16 x float>*, i32, <16 x i1>, <16 x float>)
 
 define void @test_v16f(<16 x i32> %x) {
   ; CHECK-LABEL: name: test_v16f
   ; CHECK: bb.0.bb:
   ; CHECK-NEXT:   liveins: $ymm0, $ymm1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:vr256 = COPY $ymm1
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vr256 = COPY $ymm0
   ; CHECK-NEXT:   [[AVX_SET0_:%[0-9]+]]:vr256 = AVX_SET0
   ; CHECK-NEXT:   [[VPCMPEQDYrr:%[0-9]+]]:vr256 = VPCMPEQDYrr [[COPY]], [[AVX_SET0_]]
   ; CHECK-NEXT:   [[VPCMPEQDYrr1:%[0-9]+]]:vr256 = VPCMPEQDYrr [[COPY1]], [[AVX_SET0_]]
-  ; CHECK-NEXT:   [[VMASKMOVPSYrm:%[0-9]+]]:vr256 = VMASKMOVPSYrm [[VPCMPEQDYrr1]], %stack.0.stack_input_vec, 1, $noreg, 0, $noreg :: (load (s256) from %ir.stack_input_vec, align 4)
-  ; CHECK-NEXT:   [[VMASKMOVPSYrm1:%[0-9]+]]:vr256 = VMASKMOVPSYrm [[VPCMPEQDYrr]], %stack.0.stack_input_vec, 1, $noreg, 32, $noreg :: (load (s256) from %ir.stack_input_vec + 32, align 4)
-  ; CHECK-NEXT:   VMASKMOVPSYmr %stack.1.stack_output_vec, 1, $noreg, 32, $noreg, [[VPCMPEQDYrr]], killed [[VMASKMOVPSYrm1]] :: (store (s256) into %ir.stack_output_vec + 32, align 4)
-  ; CHECK-NEXT:   VMASKMOVPSYmr %stack.1.stack_output_vec, 1, $noreg, 0, $noreg, [[VPCMPEQDYrr1]], killed [[VMASKMOVPSYrm]] :: (store (s256) into %ir.stack_output_vec, align 4)
+  ; CHECK-NEXT:   [[VMASKMOVPSYrm:%[0-9]+]]:vr256 = VMASKMOVPSYrm [[VPCMPEQDYrr1]], %stack.0.stack_input_vec, 1, $noreg, 0, $noreg :: (load unknown-size from %ir.stack_input_vec, align 4)
+  ; CHECK-NEXT:   [[VMASKMOVPSYrm1:%[0-9]+]]:vr256 = VMASKMOVPSYrm [[VPCMPEQDYrr]], %stack.0.stack_input_vec, 1, $noreg, 32, $noreg :: (load unknown-size from %ir.stack_input_vec + 32, align 4)
+  ; CHECK-NEXT:   VMASKMOVPSYmr %stack.1.stack_output_vec, 1, $noreg, 32, $noreg, [[VPCMPEQDYrr]], killed [[VMASKMOVPSYrm1]] :: (store unknown-size into %ir.stack_output_vec + 32, align 4)
+  ; CHECK-NEXT:   VMASKMOVPSYmr %stack.1.stack_output_vec, 1, $noreg, 0, $noreg, [[VPCMPEQDYrr1]], killed [[VMASKMOVPSYrm]] :: (store unknown-size into %ir.stack_output_vec, align 4)
   ; CHECK-NEXT:   RET 0
 bb:
   %stack_input_vec = alloca <16 x float>, align 64
   %stack_output_vec = alloca <16 x float>, align 64
   %mask = icmp eq <16 x i32> %x, zeroinitializer
   %masked_loaded_vec = call <16 x float> @llvm.masked.load.v16f32.p0v16f32(<16 x float>* nonnull %stack_input_vec, i32 4, <16 x i1> %mask, <16 x float> undef)
   call void @llvm.masked.store.v16f32.p0v16f32(<16 x float> %masked_loaded_vec, <16 x float>* nonnull %stack_output_vec, i32 4, <16 x i1> %mask)
   ret void
 }
 
 declare void @llvm.masked.store.v8f64.p0v8f64(<8 x double>, <8 x double>*, i32, <8 x i1>)
 declare <8 x double> @llvm.masked.load.v8f64.p0v8f64(<8 x double>*, i32, <8 x i1>, <8 x double>)
 
 define void @test_v8d(<8 x i64> %x) {
   ; CHECK-LABEL: name: test_v8d
   ; CHECK: bb.0.bb:
   ; CHECK-NEXT:   liveins: $ymm0, $ymm1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:vr256 = COPY $ymm1
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vr256 = COPY $ymm0
   ; CHECK-NEXT:   [[AVX_SET0_:%[0-9]+]]:vr256 = AVX_SET0
   ; CHECK-NEXT:   [[VPCMPEQQYrr:%[0-9]+]]:vr256 = VPCMPEQQYrr [[COPY]], [[AVX_SET0_]]
   ; CHECK-NEXT:   [[VPCMPEQQYrr1:%[0-9]+]]:vr256 = VPCMPEQQYrr [[COPY1]], [[AVX_SET0_]]
-  ; CHECK-NEXT:   [[VMASKMOVPDYrm:%[0-9]+]]:vr256 = VMASKMOVPDYrm [[VPCMPEQQYrr1]], %stack.0.stack_input_vec, 1, $noreg, 0, $noreg :: (load (s256) from %ir.stack_input_vec, align 4)
-  ; CHECK-NEXT:   [[VMASKMOVPDYrm1:%[0-9]+]]:vr256 = VMASKMOVPDYrm [[VPCMPEQQYrr]], %stack.0.stack_input_vec, 1, $noreg, 32, $noreg :: (load (s256) from %ir.stack_input_vec + 32, align 4)
-  ; CHECK-NEXT:   VMASKMOVPDYmr %stack.1.stack_output_vec, 1, $noreg, 32, $noreg, [[VPCMPEQQYrr]], killed [[VMASKMOVPDYrm1]] :: (store (s256) into %ir.stack_output_vec + 32, align 4)
-  ; CHECK-NEXT:   VMASKMOVPDYmr %stack.1.stack_output_vec, 1, $noreg, 0, $noreg, [[VPCMPEQQYrr1]], killed [[VMASKMOVPDYrm]] :: (store (s256) into %ir.stack_output_vec, align 4)
+  ; CHECK-NEXT:   [[VMASKMOVPDYrm:%[0-9]+]]:vr256 = VMASKMOVPDYrm [[VPCMPEQQYrr1]], %stack.0.stack_input_vec, 1, $noreg, 0, $noreg :: (load unknown-size from %ir.stack_input_vec, align 4)
+  ; CHECK-NEXT:   [[VMASKMOVPDYrm1:%[0-9]+]]:vr256 = VMASKMOVPDYrm [[VPCMPEQQYrr]], %stack.0.stack_input_vec, 1, $noreg, 32, $noreg :: (load unknown-size from %ir.stack_input_vec + 32, align 4)
+  ; CHECK-NEXT:   VMASKMOVPDYmr %stack.1.stack_output_vec, 1, $noreg, 32, $noreg, [[VPCMPEQQYrr]], killed [[VMASKMOVPDYrm1]] :: (store unknown-size into %ir.stack_output_vec + 32, align 4)
+  ; CHECK-NEXT:   VMASKMOVPDYmr %stack.1.stack_output_vec, 1, $noreg, 0, $noreg, [[VPCMPEQQYrr1]], killed [[VMASKMOVPDYrm]] :: (store unknown-size into %ir.stack_output_vec, align 4)
   ; CHECK-NEXT:   RET 0
 bb:
   %stack_input_vec = alloca <8 x double>, align 64
   %stack_output_vec = alloca <8 x double>, align 64
   %mask = icmp eq <8 x i64> %x, zeroinitializer
   %masked_loaded_vec = call <8 x double> @llvm.masked.load.v8f64.p0v8f64(<8 x double>* nonnull %stack_input_vec, i32 4, <8 x i1> %mask, <8 x double> undef)
   call void @llvm.masked.store.v8f64.p0v8f64(<8 x double> %masked_loaded_vec, <8 x double>* nonnull %stack_output_vec, i32 4, <8 x i1> %mask)
   ret void