[VP] Implementing expansion pass for VP load and store.

[loralb]

Summary

Added function to the ExpandVectorPredication pass to handle VP loads and stores.

[hussainjk]

Fixing errors that were introduced from rebasing.

[hussainjk]

More fixes to the length-only scalarization, and added tests using the Power8 target for this scalarization.

[frasercrmck]

Adding some extra reviewers to help keep on top of this

[frasercrmck]

This needs "generic" testing e.g. like those in test/CodeGen/Generic/expand-vp.ll

llvm/lib/CodeGen/ExpandVectorPredication.cpp

531

I wonder if the addition of this method should be in a follow-up patch? That is, first get the "basic" support in using masked.* intrinsics in and consider this an enhancement/optimization for certain targets like PowerPC?

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

518

We should pick the expansion strategy based on the target's preferences. Either extend VPLegalizationStrategy to allow for strategy selection or add a new function to TTI that returns the expansion method (enum).

531

I agree. Better to add the default expansion strategy first, then extend TTI to allow for strategy selection and the cascading scheme in a followup.

[bmahjour]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

531

This scheme is meant for any target where masked load/stores are not supported in hardware and all the active lanes are packed on the left (ie EVL only). I have a hard time imagining a target that wouldn't benefit from this scheme. I think that's why Hussain added this as part of the default expansion.

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

531

If there are two expansion schemes, targets should be able to choose between them through TTI. AVX.* supports masked load/store well - PPC may benefit more from the piecewise expansion scheme.

[bmahjour]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

531

The only other expansion scheme is a fail-safe that applies to legalization of masked load/stores in general (not just the EVL cases), so it's less optimal regardless of the architecture. I neither see the need nor strongly object to adding a TTI query for this, but if we add one I think the default should be the cascading scheme.

[bmahjour]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

531

(and again we are talking about targets where masked/load stores are not supported in hardware, so I guess AVX doesn't apply).

[simoll]

I'm still in favor of splitting up the patch into the default expansion (which can be the cascading loads) and a second one for masked.load expansion.

llvm/lib/CodeGen/ExpandVectorPredication.cpp

475

I had overlooked this before. You are checking whether masked.load is supported, so my argument for selecting the expansion scheme with TTI is moot.

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

482

TTI.isLegalMaskedStore?

[simoll]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

518

load/store unfolding does not work for scalable vectors this way. We should default to the memory intrinsic path for scalable types.

[craig.topper]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

396

VPIntrinsic is a subclass of Instruction, we shouldn't need an explicit cast.

398

Don't use auto here.

408

Why not llvm_unreachable?

466

Doesn't IRBuilder's constructor also set the debug location?

469

Don't use auto.

473

Don't use auto

478

Drop else after return

485

Drop else after return

492

Does't IRBuilder's constructor set the debug location?

515

llvm_unreachable

520

Drop else after return and drop the curly braces for the if body

536

Shouldn't be needed

545

Use StringRef

549

llvm_unreachable

559

Capitalize

576

break should be inside the curly braces

582

Same here

595

Drop parentheses around the statement

601

Using StringRef for Prefix should eliminate the need for Twine constructor here

[loralb]

hello @hussainjk @bmahjour, are you still working on this? Otherwise, I may take it over on your behalf.

[bmahjour]

In D109584#3345571, @loralb wrote:

hello @hussainjk @bmahjour, are you still working on this? Otherwise, I may take it over on your behalf.

Thanks @loralb. Please go ahead.

[loralb]

Changelog:

• Remove VP gather/scatter references (to be added in a follow-up patch)
• Following the discussion in the comments, remove expandPredicationInUnfoldedLoadStore() function (to be added in a follow-up patch)
• Apply suggestions in comments
• Add tests

[loralb]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

515

I changed the behaviour of the defalut case to reflect what was done before this patch, but I am not sure which one is the right approach: what do you think is best?

[frasercrmck]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

166

unpredicated seems misleading here, since we're using masked.load and masked.store: those are predicated in a sense.

406

nit, but I don't think you need these braces around the switch statements. NewStore/NewLoad are defined in their own scope.

408

I'd prefer something like /*IsVolatile*/ false

llvm/test/CodeGen/Generic/expand-vp-load-store.ll

16

We should be testing the IsUnmasked path here too.

[loralb]

Changelog:

• Address comments

N.B.: the IsUnmasked == true path in expandPredicationInMemoryIntrinsic() is not reachable, as also shown by the tests. How should this be handled in the code? Do we still handle this case or we add something like assert(!isAllTrueMask(MaskParam)) instead of defining the IsUnmasked boloean?

[frasercrmck]

In D109584#3352203, @loralb wrote:

Changelog:

• Address comments

N.B.: the IsUnmasked == true path in expandPredicationInMemoryIntrinsic() is not reachable, as also shown by the tests. How should this be handled in the code? Do we still handle this case or we add something like assert(!isAllTrueMask(MaskParam)) instead of defining the IsUnmasked boloean?

Is that because you're using scalable vectors and the isAllTrueMask is expecting a ConstantVector? To my mind we should either:

• use a better true-mask check (surely there's one we can reuse instead of implementing our own)
• add tests for fixed vectors
• use constantexpr all-ones masks in the scalable-vector tests

The last is a bit of a hack, but the other two sound reasonable. Adding tests for fixed vectors sounds like a good idea for this patch anyway, and it'd give us coverage of this code path. The first one should be done, but done in a separate patch. Just add a FIXME in the scalable-vector tests for now?

[loralb]

Changelog:

• Improve isAllTrueMask() (for scalable vectors only)
• Add and update tests

I did not manage to find any ready to use function in place of isAllTrueMask(), so I added this PatternMatch approach found in other places of the codebase. Maybe we can unify its behaviour in a later patch?

[loralb]

Changelog:

• Use getSplatValue() instead of PatternMatch in isAllTrueMask()
• Update tests

[frasercrmck]

llvm/lib/CodeGen/ExpandVectorPredication.cpp

90

I don't know if it matters whether this is isOneValue or isAllOnesValue? In practice for i1 masks it's the same, but somehow the latter sounds more appropriate to me.

llvm/test/CodeGen/Generic/expand-vp-load-store.ll

35

Shouldn't we see regular load here?

79

Same here: regular store?

[loralb]

llvm/test/CodeGen/Generic/expand-vp-load-store.ll

35

If I understand this correctly, since the evl value is unknown, using here a regular load means we may try to load elements that we should not, hence why we need the masked version

79

This should work like the load above

[loralb]

Changelog:

• Use isAllOnesValue() instead of isOneValue()

[simoll]

LGTM

[bmahjour]

Following the discussion in the comments, remove expandPredicationInUnfoldedLoadStore() function (to be added in a follow-up patch)

Is there a follow up patch for this yet?

[loralb]

In D109584#3553244, @bmahjour wrote:

Following the discussion in the comments, remove expandPredicationInUnfoldedLoadStore() function (to be added in a follow-up patch)

Is there a follow up patch for this yet?

hello @bmahjour, there is no follow-up patch yet and right now it is not in my short-term plan; I may be able to keep working on it later on. There wouldn't be any problem though if someone is interested on taking this over.

[loralb]

Changelog:

• Rebase

llvm/lib/CodeGen/ExpandVectorPredication.cpp

 
 ///// Helpers {
 
 /// \returns Whether the vector mask \p MaskVal has all lane bits set.
 static bool isAllTrueMask(Value *MaskVal) {
   auto *ConstVec = dyn_cast<ConstantVector>(MaskVal);
   return ConstVec && ConstVec->isAllOnesValue();
 }
 
 /// \returns A non-excepting divisor constant for this type.
 static Constant *getSafeDivisor(Type *DivTy) {
   assert(DivTy->isIntOrIntVectorTy() && "Unsupported divisor type");
   return ConstantInt::get(DivTy, 1u, false);
 }
 
 /// Transfer operation properties from \p OldVPI to \p NewVal.
 static void transferDecorations(Value &NewVal, VPIntrinsic &VPI) {
struct CachingVPExpander {
   Value *expandPredicationInBinaryOperator(IRBuilder<> &Builder,
                                            VPIntrinsic &PI);
 
   /// \brief Lower this VP reduction to a call to an unpredicated reduction
   /// intrinsic.
   Value *expandPredicationInReduction(IRBuilder<> &Builder,
                                       VPReductionIntrinsic &PI);
 
+  /// \brief Lower this VP memory operation to a non-VP intrinsic.
+  Value *expandPredicationInMemoryIntrinsic(IRBuilder<> &Builder,
+                                            VPIntrinsic &VPI);
+
   /// \brief Query TTI and expand the vector predication in \p P accordingly.
   Value *expandPredication(VPIntrinsic &PI);
 
   /// \brief  Determine how and whether the VPIntrinsic \p VPI shall be
   /// expanded. This overrides TTI with the cl::opts listed at the top of this
   /// file.
   VPLegalization getVPLegalizationStrategy(const VPIntrinsic &VPI) const;
   bool UsingTTIOverrides;
case Intrinsic::vp_reduce_fmul:
     Reduction = Builder.CreateFMulReduce(Start, RedOp);
     break;
   }
 
   replaceOperation(*Reduction, VPI);
   return Reduction;
 }
 
+Value *
+CachingVPExpander::expandPredicationInMemoryIntrinsic(IRBuilder<> &Builder,
+                                                      VPIntrinsic &VPI) {
+  assert(VPI.canIgnoreVectorLengthParam());
+
+  Value *MaskParam = VPI.getMaskParam();
+  Value *PtrParam = VPI.getMemoryPointerParam();
+  Value *DataParam = VPI.getMemoryDataParam();
+  bool IsUnmasked = isAllTrueMask(MaskParam);
+
+  MaybeAlign AlignOpt = VPI.getPointerAlignment();
+
+  Value *NewMemoryInst = nullptr;
+  switch (VPI.getIntrinsicID()) {
+  default:
+    llvm_unreachable("Not a VP memory intrinsic");
+  case Intrinsic::vp_store:
+    if (IsUnmasked) {
+      StoreInst *NewStore =
+          Builder.CreateStore(DataParam, PtrParam, /*IsVolatile*/ false);
+      if (AlignOpt.hasValue())
+        NewStore->setAlignment(AlignOpt.getValue());
+      NewMemoryInst = NewStore;
+    } else
+      NewMemoryInst = Builder.CreateMaskedStore(
+          DataParam, PtrParam, AlignOpt.valueOrOne(), MaskParam);
+
+    break;
+  case Intrinsic::vp_load:
+    if (IsUnmasked) {
+      LoadInst *NewLoad =
+          Builder.CreateLoad(VPI.getType(), PtrParam, /*IsVolatile*/ false);
+      if (AlignOpt.hasValue())
+        NewLoad->setAlignment(AlignOpt.getValue());
+      NewMemoryInst = NewLoad;
+    } else
+      NewMemoryInst = Builder.CreateMaskedLoad(
+          VPI.getType(), PtrParam, AlignOpt.valueOrOne(), MaskParam);
+
+    break;
+  }
+
+  assert(NewMemoryInst);
+  replaceOperation(*NewMemoryInst, VPI);
+  return NewMemoryInst;
+}
+
 void CachingVPExpander::discardEVLParameter(VPIntrinsic &VPI) {
   LLVM_DEBUG(dbgs() << "Discard EVL parameter in " << VPI << "\n");
 
   if (VPI.canIgnoreVectorLengthParam())
     return;
 
   Value *EVLParam = VPI.getVectorLengthParam();
   if (!EVLParam)
if (StaticElemCount.isScalable()) {
     MaxEVL = Builder.CreateMul(VScale, FactorConst, "scalable_size",
                                /*NUW*/ true, /*NSW*/ false);
   } else {
     MaxEVL = ConstantInt::get(Int32Ty, StaticElemCount.getFixedValue(), false);
   }
   VPI.setVectorLengthParam(MaxEVL);
 }
 
 Value *CachingVPExpander::foldEVLIntoMask(VPIntrinsic &VPI) {
   LLVM_DEBUG(dbgs() << "Folding vlen for " << VPI << '\n');
 
   IRBuilder<> Builder(&VPI);
 
   // Ineffective %evl parameter and so nothing to do here.
   if (VPI.canIgnoreVectorLengthParam())
     return &VPI;
 
   // Only VP intrinsics can have an %evl parameter.
   Value *OldMaskParam = VPI.getMaskParam();
   Value *OldEVLParam = VPI.getVectorLengthParam();
   assert(OldMaskParam && "no mask param to fold the vl param into");
   assert(OldEVLParam && "no EVL param to fold away");
 
   LLVM_DEBUG(dbgs() << "OLD evl: " << *OldEVLParam << '\n');
   LLVM_DEBUG(dbgs() << "OLD mask: " << *OldMaskParam << '\n');
 
   // Convert the %evl predication into vector mask predication.
   ElementCount ElemCount = VPI.getStaticVectorLength();
   Value *VLMask = convertEVLToMask(Builder, OldEVLParam, ElemCount);
   Value *NewMaskParam = Builder.CreateAnd(VLMask, OldMaskParam);
   VPI.setMaskParam(NewMaskParam);
 
   // Drop the %evl parameter.
   discardEVLParameter(VPI);
   assert(VPI.canIgnoreVectorLengthParam() &&
          "transformation did not render the evl param ineffective!");
 
   // Reassess the modified instruction.
   return &VPI;
 }
 
 Value *CachingVPExpander::expandPredication(VPIntrinsic &VPI) {
   LLVM_DEBUG(dbgs() << "Lowering to unpredicated op: " << VPI << '\n');
 
   IRBuilder<> Builder(&VPI);
 
   // Try lowering to a LLVM instruction first.
   auto OC = VPI.getFunctionalOpcode();
 
   if (OC && Instruction::isBinaryOp(*OC))
     return expandPredicationInBinaryOperator(Builder, VPI);
 
   if (auto *VPRI = dyn_cast<VPReductionIntrinsic>(&VPI))
     return expandPredicationInReduction(Builder, *VPRI);
 
+  switch (VPI.getIntrinsicID()) {
+  default:
+    break;
+  case Intrinsic::vp_load:
+  case Intrinsic::vp_store:
+    return expandPredicationInMemoryIntrinsic(Builder, VPI);
+  }
+
   return &VPI;
 }
 
 //// } CachingVPExpander
 
 struct TransformJob {
   VPIntrinsic *PI;
   TargetTransformInfo::VPLegalization Strategy;
   TransformJob(VPIntrinsic *PI, TargetTransformInfo::VPLegalization InitStrat)
       : PI(PI), Strategy(InitStrat) {}
 
   bool isDone() const { return Strategy.shouldDoNothing(); }
 };
 
 void sanitizeStrategy(Instruction &I, VPLegalization &LegalizeStrat) {
   // Speculatable instructions do not strictly need predication.
   if (isSafeToSpeculativelyExecute(&I)) {
     // Converting a speculatable VP intrinsic means dropping %mask and %evl.
     // No need to expand %evl into the %mask only to ignore that code.
     if (LegalizeStrat.OpStrategy == VPLegalization::Convert)
       LegalizeStrat.EVLParamStrategy = VPLegalization::Discard;
     return;
   }
 
   // We have to preserve the predicating effect of %evl for this
   // non-speculatable VP intrinsic.
   // 1) Never discard %evl.
   // 2) If this VP intrinsic will be expanded to non-VP code, make sure that
   //    %evl gets folded into %mask.
   if ((LegalizeStrat.EVLParamStrategy == VPLegalization::Discard) ||
       (LegalizeStrat.OpStrategy == VPLegalization::Convert)) {
     LegalizeStrat.EVLParamStrategy = VPLegalization::Convert;
   }
 }
 
 VPLegalization
 CachingVPExpander::getVPLegalizationStrategy(const VPIntrinsic &VPI) const {
   auto VPStrat = TTI.getVPLegalizationStrategy(VPI);
   if (LLVM_LIKELY(!UsingTTIOverrides)) {
     // No overrides - we are in production.
     return VPStrat;
   }
 
   // Overrides set - we are in testing, the following does not need to be
   // efficient.
   VPStrat.EVLParamStrategy = parseOverrideOption(EVLTransformOverride);
   VPStrat.OpStrategy = parseOverrideOption(MaskTransformOverride);
   return VPStrat;
 }
 
 /// \brief Expand llvm.vp.* intrinsics as requested by \p TTI.
 bool CachingVPExpander::expandVectorPredication() {
   SmallVector<TransformJob, 16> Worklist;
 
   // Collect all VPIntrinsics that need expansion and determine their expansion
   // strategy.
   for (auto &I : instructions(F)) {
     auto *VPI = dyn_cast<VPIntrinsic>(&I);
     if (!VPI)
       continue;
     auto VPStrat = getVPLegalizationStrategy(*VPI);
     sanitizeStrategy(I, VPStrat);
     if (!VPStrat.shouldDoNothing())
       Worklist.emplace_back(VPI, VPStrat);
   }
   if (Worklist.empty())
     return false;
 
   // Transform all VPIntrinsics on the worklist.
   LLVM_DEBUG(dbgs() << "\n:::: Transforming " << Worklist.size()
                     << " instructions ::::\n");
   for (TransformJob Job : Worklist) {
     // Transform the EVL parameter.
     switch (Job.Strategy.EVLParamStrategy) {
     case VPLegalization::Legal:
       break;
     case VPLegalization::Discard:
       discardEVLParameter(*Job.PI);
       break;
     case VPLegalization::Convert:
       if (foldEVLIntoMask(*Job.PI))
         ++NumFoldedVL;
       break;
     }
     Job.Strategy.EVLParamStrategy = VPLegalization::Legal;
 
     // Replace with a non-predicated operation.
     switch (Job.Strategy.OpStrategy) {
     case VPLegalization::Legal:

llvm/test/CodeGen/Generic/expand-vp-load-store.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt --expandvp -S < %s | FileCheck %s
+; RUN: opt --expandvp --expandvp-override-evl-transform=Legal --expandvp-override-mask-transform=Convert -S < %s | FileCheck %s
+
+declare <vscale x 1 x i64> @llvm.vp.load.nxv1i64.p0nxv1i64(<vscale x 1 x i64>*, <vscale x 1 x i1>, i32)
+
+define <vscale x 1 x i64> @vpload_nxv1i64(<vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: @vpload_nxv1i64(
+; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 1 x i1> @llvm.get.active.lane.mask.nxv1i1.i32(i32 0, i32 [[EVL:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = and <vscale x 1 x i1> [[TMP1]], [[M:%.*]]
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i32 @llvm.vscale.i32()
+; CHECK-NEXT:    [[SCALABLE_SIZE:%.*]] = mul nuw i32 [[VSCALE]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = call <vscale x 1 x i64> @llvm.masked.load.nxv1i64.p0nxv1i64(<vscale x 1 x i64>* [[PTR:%.*]], i32 1, <vscale x 1 x i1> [[TMP2]], <vscale x 1 x i64> undef)
+; CHECK-NEXT:    ret <vscale x 1 x i64> [[TMP3]]
+;
+  %load = call <vscale x 1 x i64> @llvm.vp.load.nxv1i64.p0nxv1i64(<vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %m, i32 %evl)
+  ret <vscale x 1 x i64> %load
+}
+
+define <vscale x 1 x i64> @vpload_nxv1i64_allones_mask(<vscale x 1 x i64>* %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: @vpload_nxv1i64_allones_mask(
+; CHECK-NEXT:    [[ONE:%.*]] = insertelement <vscale x 1 x i1> poison, i1 true, i32 0
+; CHECK-NEXT:    [[ALLONES:%.*]] = shufflevector <vscale x 1 x i1> [[ONE]], <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 1 x i1> @llvm.get.active.lane.mask.nxv1i1.i32(i32 0, i32 [[EVL:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = and <vscale x 1 x i1> [[TMP1]], [[ALLONES]]
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i32 @llvm.vscale.i32()
+; CHECK-NEXT:    [[SCALABLE_SIZE:%.*]] = mul nuw i32 [[VSCALE]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = call <vscale x 1 x i64> @llvm.masked.load.nxv1i64.p0nxv1i64(<vscale x 1 x i64>* [[PTR:%.*]], i32 1, <vscale x 1 x i1> [[TMP2]], <vscale x 1 x i64> undef)
+; CHECK-NEXT:    ret <vscale x 1 x i64> [[TMP3]]
+;
+  %one = insertelement <vscale x 1 x i1> poison, i1 true, i32 0
+  %allones = shufflevector <vscale x 1 x i1> %one, <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer
+  %load = call <vscale x 1 x i64> @llvm.vp.load.nxv1i64.p0nxv1i64(<vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %allones, i32 %evl)
+  ret <vscale x 1 x i64> %load
+}
+
+declare void @llvm.vp.store.nxv1i64.p0nxv1i64(<vscale x 1 x i64>, <vscale x 1 x i64>*, <vscale x 1 x i1>, i32)
+
+define void @vpstore_nxv1i64(<vscale x 1 x i64> %val, <vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: @vpstore_nxv1i64(
+; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 1 x i1> @llvm.get.active.lane.mask.nxv1i1.i32(i32 0, i32 [[EVL:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = and <vscale x 1 x i1> [[TMP1]], [[M:%.*]]
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i32 @llvm.vscale.i32()
+; CHECK-NEXT:    [[SCALABLE_SIZE:%.*]] = mul nuw i32 [[VSCALE]], 1
+; CHECK-NEXT:    call void @llvm.masked.store.nxv1i64.p0nxv1i64(<vscale x 1 x i64> [[VAL:%.*]], <vscale x 1 x i64>* [[PTR:%.*]], i32 1, <vscale x 1 x i1> [[TMP2]])
+; CHECK-NEXT:    ret void
+;
+  call void @llvm.vp.store.nxv1i64.p0nxv1i64(<vscale x 1 x i64> %val, <vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %m, i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv1i64_allones_mask(<vscale x 1 x i64> %val, <vscale x 1 x i64>* %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: @vpstore_nxv1i64_allones_mask(
+; CHECK-NEXT:    [[ONE:%.*]] = insertelement <vscale x 1 x i1> poison, i1 true, i32 0
+; CHECK-NEXT:    [[ALLONES:%.*]] = shufflevector <vscale x 1 x i1> [[ONE]], <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP1:%.*]] = call <vscale x 1 x i1> @llvm.get.active.lane.mask.nxv1i1.i32(i32 0, i32 [[EVL:%.*]])
+; CHECK-NEXT:    [[TMP2:%.*]] = and <vscale x 1 x i1> [[TMP1]], [[ALLONES]]
+; CHECK-NEXT:    [[VSCALE:%.*]] = call i32 @llvm.vscale.i32()
+; CHECK-NEXT:    [[SCALABLE_SIZE:%.*]] = mul nuw i32 [[VSCALE]], 1
+; CHECK-NEXT:    call void @llvm.masked.store.nxv1i64.p0nxv1i64(<vscale x 1 x i64> [[VAL:%.*]], <vscale x 1 x i64>* [[PTR:%.*]], i32 1, <vscale x 1 x i1> [[TMP2]])
+; CHECK-NEXT:    ret void
+;
+  %one = insertelement <vscale x 1 x i1> poison, i1 true, i32 0
+  %allones = shufflevector <vscale x 1 x i1> %one, <vscale x 1 x i1> poison, <vscale x 1 x i32> zeroinitializer
+  call void @llvm.vp.store.nxv1i64.p0nxv1i64(<vscale x 1 x i64> %val, <vscale x 1 x i64>* %ptr, <vscale x 1 x i1> %allones, i32 %evl)
+  ret void
+}