POC patch to demonstrate how new intrinsics for interleaved load/store could be used in LoopVectorize

[mgabka]

Summary

This patch needs to be broken down into smaller chunks.

The purpose of this patch is to demonstrate how new, generic LLVM intrinsics
could be emitted directly from inside LoopVectorize,
and later transformed into a target specific ones.
This POC patch focuses on the SVE side, however the intention is to not limit it
only to SVE. Code generation for NEON can also use this approach.

The motivation for such solution is that at the vectorization stage we already
know if we are handling interleaved memory accesses or not, so we can use
that knowledge to emit dedicated intrinsics for such accesses.

The current LLVM's implementation uses shufflevector to interleave
the data after/before load/store what is not suitable for scalable vectors.

[nlopes]

llvm/lib/IR/IRBuilder.cpp

605

Please use PoisonValue here. We already changed all the other similar intrinsics to have poison as default passthru.

[reames]

Interesting! This is a different approach than I'd been considering.

I had been picturing adding intrinsics for each of the shuffles used during interleave lowering. All three have reasonable lowerings on RISCV - not sure about SVE. Your approach would make matching segmented load/store easier, but is a larger divergence from the way we currently handle fixed length interleaves.

On the other hand, I'd been debating something similar to this approach for strided load/store intrinsics. Would you have some time to chat on a call and debate approaches here?

[paulwalker-arm]

In D134438#3821583, @reames wrote:

Would you have some time to chat on a call and debate approaches here?

A call will be very helpful, thanks. I'll get something set up.

[reames]

FYI, D141924 has landed. This patch took a slightly different approach and introduced intrinsics for interleave and deinterleave respectively (not the combined memory ops). As such, this patch is now stale. There appear to be a few parts of this which are potentially salvageable (tests, vectorizer changes), but the patch either needs a significant rework and rebase or to be abandoned.

[mgabka]

@reames yes I am aware of it, and I will either update this patch or create a new one (in that case will add the same set of reviewers).

[mgabka]

Hi Luke Lau,
Hie is a POC patch I posted initially some time ago, however after landing https://reviews.llvm.org/D141924 it needs to be changed, I am working on a new patch which enables vectorization for interleaved accesses, with interleaving factor of 2. Will add you there as a reviewer when it is posted.

[luke]

In D134438#4157776, @mgabka wrote:

Hi Luke Lau,
Hie is a POC patch I posted initially some time ago, however after landing https://reviews.llvm.org/D141924 it needs to be changed, I am working on a new patch which enables vectorization for interleaved accesses, with interleaving factor of 2. Will add you there as a reviewer when it is posted.

Sounds good, thanks for taking care of this

[mgabka]

I posted a new patch https://reviews.llvm.org/D145163 updated to use the new intrinciscs

llvm/include/llvm/Analysis/TargetTransformInfo.h

struct MemCmpExpansionOptions {
     bool AllowOverlappingLoads = false;
   };
   MemCmpExpansionOptions enableMemCmpExpansion(bool OptSize,
                                                bool IsZeroCmp) const;
 
   /// Enable matching of interleaved access groups.
   bool enableInterleavedAccessVectorization() const;
 
+  bool hasInterleavedLoad(VectorType *VecTy, Value *Addr, uint32_t Factor,
+                          bool IsMasked) const;
+
+  bool hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs, Value *Addr,
+                           uint32_t Factor, bool IsMasked) const;
+
   /// Enable matching of interleaved access groups that contain predicated
   /// accesses or gaps and therefore vectorized using masked
   /// vector loads/stores.
   bool enableMaskedInterleavedAccessVectorization() const;
 
   /// Indicate that it is potentially unsafe to automatically vectorize
   /// floating-point operations because the semantics of vector and scalar
   /// floating-point semantics may differ. For example, ARM NEON v7 SIMD math
getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                    ArrayRef<Type *> Tys) = 0;
   virtual bool supportsEfficientVectorElementLoadStore() = 0;
   virtual bool supportsTailCalls() = 0;
   virtual bool supportsTailCallFor(const CallBase *CB) = 0;
   virtual bool enableAggressiveInterleaving(bool LoopHasReductions) = 0;
   virtual MemCmpExpansionOptions
   enableMemCmpExpansion(bool OptSize, bool IsZeroCmp) const = 0;
   virtual bool enableInterleavedAccessVectorization() = 0;
+  virtual bool hasInterleavedLoad(VectorType *VecTy, Value *Addr,
+                                  uint32_t Factor, bool IsMasked) = 0;
+  virtual bool hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs,
+                                   Value *Addr, uint32_t Factor,
+                                   bool IsMasked) = 0;
   virtual bool enableMaskedInterleavedAccessVectorization() = 0;
   virtual bool isFPVectorizationPotentiallyUnsafe() = 0;
   virtual bool allowsMisalignedMemoryAccesses(LLVMContext &Context,
                                               unsigned BitWidth,
                                               unsigned AddressSpace,
                                               Align Alignment,
                                               bool *Fast) = 0;
   virtual PopcntSupportKind getPopcntSupport(unsigned IntTyWidthInBit) = 0;
public:
   }
   MemCmpExpansionOptions enableMemCmpExpansion(bool OptSize,
                                                bool IsZeroCmp) const override {
     return Impl.enableMemCmpExpansion(OptSize, IsZeroCmp);
   }
   bool enableInterleavedAccessVectorization() override {
     return Impl.enableInterleavedAccessVectorization();
   }
+  bool hasInterleavedLoad(VectorType *VecTy, Value *Addr, uint32_t Factor,
+                          bool IsMasked) override {
+    return Impl.hasInterleavedLoad(VecTy, Addr, Factor, IsMasked);
+  }
+  bool hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs, Value *Addr,
+                           uint32_t Factor, bool IsMasked) override {
+    return Impl.hasInterleavedStore(StoredVecs, Addr, Factor, IsMasked);
+  }
   bool enableMaskedInterleavedAccessVectorization() override {
     return Impl.enableMaskedInterleavedAccessVectorization();
   }
   bool isFPVectorizationPotentiallyUnsafe() override {
     return Impl.isFPVectorizationPotentiallyUnsafe();
   }
   bool allowsMisalignedMemoryAccesses(LLVMContext &Context, unsigned BitWidth,
                                       unsigned AddressSpace, Align Alignment,

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   TTI::MemCmpExpansionOptions enableMemCmpExpansion(bool OptSize,
                                                     bool IsZeroCmp) const {
     return {};
   }
 
   bool enableInterleavedAccessVectorization() const { return false; }
 
+  bool hasInterleavedLoad(VectorType *VecTy, Value *Addr, uint32_t Factor,
+                          bool IsMasked) const {
+    return false;
+  }
+
+  bool hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs, Value *Addr,
+                           uint32_t Factor, bool IsMasked) const {
+    return false;
+  }
+
   bool enableMaskedInterleavedAccessVectorization() const { return false; }
 
   bool isFPVectorizationPotentiallyUnsafe() const { return false; }
 
   bool allowsMisalignedMemoryAccesses(LLVMContext &Context, unsigned BitWidth,
                                       unsigned AddressSpace, Align Alignment,
                                       bool *Fast) const {
     return false;

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   /// \p Factor is the interleave factor.
   virtual bool lowerInterleavedLoad(LoadInst *LI,
                                     ArrayRef<ShuffleVectorInst *> Shuffles,
                                     ArrayRef<unsigned> Indices,
                                     unsigned Factor) const {
     return false;
   }
 
+  /// Lower an interleaved load to target specific intrinsics. Return
+  /// true on success.
+  ///
+  /// \p LI is the vector interleaved load intrinsic.
+  virtual bool lowerInterleavedLoad(IntrinsicInst *LI) const { return false; }
+
   /// Lower an interleaved store to target specific intrinsics. Return
   /// true on success.
   ///
   /// \p SI is the vector store instruction.
   /// \p SVI is the shufflevector to RE-interleave the stored vector.
   /// \p Factor is the interleave factor.
   virtual bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                                      unsigned Factor) const {
     return false;
   }
 
+  /// Lower an interleaved store to target specific intrinsics. Return
+  /// true on success.
+  ///
+  /// \p SI is the vector interleaved store intrinsic.
+  virtual bool lowerInterleavedStore(IntrinsicInst *SI) const { return false; }
+
   /// Return true if zero-extending the specific node Val to type VT2 is free
   /// (either because it's implicitly zero-extended such as ARM ldrb / ldrh or
   /// because it's folded such as X86 zero-extending loads).
   virtual bool isZExtFree(SDValue Val, EVT VT2) const {
     return isZExtFree(Val.getValueType(), VT2);
   }
 
   /// Return true if an fpext operation is free (for instance, because

llvm/include/llvm/IR/IRBuilder.h

public:
   /// Create a call to invariant.start intrinsic.
   ///
   /// If the pointer isn't i8* it will be converted.
   CallInst *CreateInvariantStart(Value *Ptr, ConstantInt *Size = nullptr);
 
   /// Create a call to llvm.threadlocal.address intrinsic.
   CallInst *CreateThreadLocalAddress(Value *Ptr);
 
+  /// Create a call to a Masked Interleaved Load intrinsic.
+  CallInst *CreateMaskedInterleavedLoad(uint32_t Factor, Type *Ty, Value *Ptr,
+                                        Align Alignment, Value *Mask = nullptr,
+                                        Value *PassThru = nullptr,
+                                        const Twine &Name = "");
+
+  /// Create a call to a Masked Interleaved Store intrinsic.
+  CallInst *CreateMaskedInterleavedStore(uint32_t Factor, ArrayRef<Value *> Val,
+                                         Value *Ptr, Align Alignment,
+                                         Value *Mask = nullptr);
+
   /// Create a call to Masked Load intrinsic
   CallInst *CreateMaskedLoad(Type *Ty, Value *Ptr, Align Alignment, Value *Mask,
                              Value *PassThru = nullptr, const Twine &Name = "");
 
   /// Create a call to Masked Store intrinsic
   CallInst *CreateMaskedStore(Value *Val, Value *Ptr, Align Alignment,
                               Value *Mask);
 

llvm/include/llvm/IR/Intrinsics.td

DefaultAttrsIntrinsic<[llvm_anyvector_ty],
             [IntrReadMem, IntrWillReturn, ImmArg<ArgIndex<1>>]>;
 
 def int_masked_scatter:
   DefaultAttrsIntrinsic<[],
             [llvm_anyvector_ty, LLVMVectorOfAnyPointersToElt<0>, llvm_i32_ty,
              LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
             [IntrWriteMem, IntrWillReturn, ImmArg<ArgIndex<2>>]>;
 
+def int_experimental_masked_interleaved2_load:
+  DefaultAttrsIntrinsic<[llvm_anyvector_ty, LLVMMatchType<0>],
+            [LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>, LLVMMatchType<0>],
+            [IntrReadMem, IntrArgMemOnly]>;
+
+def int_experimental_masked_interleaved3_load:
+  DefaultAttrsIntrinsic<[llvm_anyvector_ty, LLVMMatchType<0>, LLVMMatchType<0>],
+            [LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>, LLVMMatchType<0>],
+            [IntrReadMem, IntrArgMemOnly]>;
+
+def int_experimental_masked_interleaved4_load:
+  DefaultAttrsIntrinsic<[llvm_anyvector_ty, LLVMMatchType<0>, LLVMMatchType<0>,
+                         LLVMMatchType<0>],
+            [LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>, LLVMMatchType<0>],
+            [IntrReadMem, IntrArgMemOnly]>;
+
+def int_experimental_masked_interleaved2_store:
+  DefaultAttrsIntrinsic<[],
+            [llvm_anyvector_ty, LLVMMatchType<0>,
+             LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+            [IntrWriteMem, IntrWillReturn, ImmArg<ArgIndex<3>>]>;
+
+def int_experimental_masked_interleaved3_store:
+  DefaultAttrsIntrinsic<[],
+            [llvm_anyvector_ty, LLVMMatchType<0>, LLVMMatchType<0>,
+             LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+            [IntrWriteMem, IntrWillReturn, ImmArg<ArgIndex<4>>]>;
+
+def int_experimental_masked_interleaved4_store:
+  DefaultAttrsIntrinsic<[],
+            [llvm_anyvector_ty, LLVMMatchType<0>, LLVMMatchType<0>,
+             LLVMMatchType<0>,
+             LLVMPointerToElt<0>, llvm_i32_ty, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
+            [IntrWriteMem, IntrWillReturn, ImmArg<ArgIndex<5>>]>;
+
 def int_masked_expandload:
   DefaultAttrsIntrinsic<[llvm_anyvector_ty],
             [LLVMPointerToElt<0>, LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
              LLVMMatchType<0>],
             [IntrReadMem, IntrWillReturn]>;
 
 def int_masked_compressstore:
   DefaultAttrsIntrinsic<[],

llvm/lib/Analysis/TargetTransformInfo.cpp

 TargetTransformInfo::enableMemCmpExpansion(bool OptSize, bool IsZeroCmp) const {
   return TTIImpl->enableMemCmpExpansion(OptSize, IsZeroCmp);
 }
 
 bool TargetTransformInfo::enableInterleavedAccessVectorization() const {
   return TTIImpl->enableInterleavedAccessVectorization();
 }
 
+bool TargetTransformInfo::hasInterleavedLoad(VectorType *VecTy, Value *Addr,
+                                             uint32_t Factor,
+                                             bool IsMasked) const {
+  return TTIImpl->hasInterleavedLoad(VecTy, Addr, Factor, IsMasked);
+}
+
+bool TargetTransformInfo::hasInterleavedStore(
+    SmallVectorImpl<Value *> &StoredVecs, Value *Addr, uint32_t Factor,
+    bool IsMasked) const {
+  return TTIImpl->hasInterleavedStore(StoredVecs, Addr, Factor, IsMasked);
+}
+
 bool TargetTransformInfo::enableMaskedInterleavedAccessVectorization() const {
   return TTIImpl->enableMaskedInterleavedAccessVectorization();
 }
 
 bool TargetTransformInfo::isFPVectorizationPotentiallyUnsafe() const {
   return TTIImpl->isFPVectorizationPotentiallyUnsafe();
 }
 

llvm/lib/CodeGen/InterleavedAccessPass.cpp

 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
private:
   /// Transform an interleaved load into target specific intrinsics.
   bool lowerInterleavedLoad(LoadInst *LI,
                             SmallVector<Instruction *, 32> &DeadInsts);
 
   /// Transform an interleaved store into target specific intrinsics.
   bool lowerInterleavedStore(StoreInst *SI,
                              SmallVector<Instruction *, 32> &DeadInsts);
 
+  /// Transform an interleaved llvm intrinsic into target specific intrinsics.
+  bool lowerInterleavedIntrinsic(IntrinsicInst *SI,
+                                 SmallVector<Instruction *, 32> &DeadInsts);
+
   /// Returns true if the uses of an interleaved load by the
   /// extractelement instructions in \p Extracts can be replaced by uses of the
   /// shufflevector instructions in \p Shuffles instead. If so, the necessary
   /// replacements are also performed.
   bool tryReplaceExtracts(ArrayRef<ExtractElementInst *> Extracts,
                           ArrayRef<ShuffleVectorInst *> Shuffles);
 
   /// Given a number of shuffles of the form shuffle(binop(x,y)), convert them
if (!TLI->lowerInterleavedStore(SI, SVI, Factor))
     return false;
 
   // Already have a new target specific interleaved store. Erase the old store.
   DeadInsts.push_back(SI);
   DeadInsts.push_back(SVI);
   return true;
 }
 
+bool InterleavedAccess::lowerInterleavedIntrinsic(
+    IntrinsicInst *II, SmallVector<Instruction *, 32> &DeadInsts) {
+  switch (II->getIntrinsicID()) {
+  case Intrinsic::experimental_masked_interleaved2_load:
+  case Intrinsic::experimental_masked_interleaved3_load:
+  case Intrinsic::experimental_masked_interleaved4_load:
+    if (!TLI->lowerInterleavedLoad(II))
+      return false;
+    break;
+  case Intrinsic::experimental_masked_interleaved2_store:
+  case Intrinsic::experimental_masked_interleaved3_store:
+  case Intrinsic::experimental_masked_interleaved4_store:
+    if (!TLI->lowerInterleavedStore(II))
+      return false;
+    break;
+  default:
+    return false;
+  }
+  DeadInsts.push_back(II);
+  return true;
+}
+
 bool InterleavedAccess::runOnFunction(Function &F) {
   auto *TPC = getAnalysisIfAvailable<TargetPassConfig>();
   if (!TPC || !LowerInterleavedAccesses)
     return false;
 
   LLVM_DEBUG(dbgs() << "*** " << getPassName() << ": " << F.getName() << "\n");
 
   DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
   auto &TM = TPC->getTM<TargetMachine>();
   TLI = TM.getSubtargetImpl(F)->getTargetLowering();
   MaxFactor = TLI->getMaxSupportedInterleaveFactor();
 
   // Holds dead instructions that will be erased later.
   SmallVector<Instruction *, 32> DeadInsts;
   bool Changed = false;
 
   for (auto &I : instructions(F)) {
     if (auto *LI = dyn_cast<LoadInst>(&I))
       Changed |= lowerInterleavedLoad(LI, DeadInsts);
 
+    if (auto *II = dyn_cast<IntrinsicInst>(&I))
+      Changed |= lowerInterleavedIntrinsic(II, DeadInsts);
+
     if (auto *SI = dyn_cast<StoreInst>(&I))
       Changed |= lowerInterleavedStore(SI, DeadInsts);
   }
 
   for (auto *I : DeadInsts)
     I->eraseFromParent();
 
   return Changed;
 }

llvm/lib/IR/IRBuilder.cpp

 
 Instruction *IRBuilderBase::CreateNoAliasScopeDeclaration(Value *Scope) {
   Module *M = BB->getModule();
   auto *FnIntrinsic = Intrinsic::getDeclaration(
       M, Intrinsic::experimental_noalias_scope_decl, {});
   return CreateCall(FnIntrinsic, {Scope});
 }
 
+/// Create a call to a Masked Interleaved Load intrinsic.
+/// \p Ty        - vector type to load
+/// \p Ptr       - base pointer for the load
+/// \p Alignment - alignment of the source location
+/// \p Mask      - vector of booleans which indicates what vector lanes should
+///                be accessed in memory
+/// \p PassThru  - pass-through value that is used to fill the masked-off lanes
+///                of the result
+/// \p Name      - name of the result variable
+/// \p Factor    - interleaving factor
+CallInst *IRBuilderBase::CreateMaskedInterleavedLoad(
+    uint32_t Factor, Type *Ty, Value *Ptr, Align Alignment, Value *Mask,
+    Value *PassThru, const Twine &Name) {
+  assert(Ty->isVectorTy() && "Type should be vector");
+  auto *PtrTy = cast<PointerType>(Ptr->getType());
+  assert(PtrTy->isOpaqueOrPointeeTypeMatches(Ty) && "Wrong element type");
+
+  auto *VecTy = cast<VectorType>(Ty);
+  ElementCount NumElts = VecTy->getElementCount().divideCoefficientBy(Factor);
+  auto *LDVTy =
+      VectorType::get(VecTy->getElementType(),
+                      VecTy->getElementCount().divideCoefficientBy(Factor));
+  if (!Mask)
+    Mask = Constant::getAllOnesValue(
+        VectorType::get(Type::getInt1Ty(Context), NumElts));
+  if (!PassThru)
+    PassThru = UndefValue::get(LDVTy);
+
+  auto *PtrVecTy = LDVTy->getElementType()->getPointerTo(
+      Ptr->getType()->getPointerAddressSpace());
+  Type *OverloadedTypes[] = {LDVTy};
+  Value *Ops[] = {this->CreateBitCast(Ptr, PtrVecTy),
+                  getInt32(Alignment.value()), Mask, PassThru};
+  switch (Factor) {
+  case 2:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved2_load, Ops, OverloadedTypes,
+        Name);
+  case 3:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved3_load, Ops, OverloadedTypes,
+        Name);
+  case 4:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved4_load, Ops, OverloadedTypes,
+        Name);
+  default:
+    break;
+  }
+  return nullptr;
+}
+
+/// Create a call to a Masked Interleaved Store intrinsic.
+/// \p StoredVals  - data to be stored
+/// \p Ptr         - base pointer for the store
+/// \p Alignment   - alignment of the destination location
+/// \p Mask        - vector of booleans which indicates what vector lanes should
+///                  be accessed in memory
+/// \p Factor      - interleaving factor
+CallInst *IRBuilderBase::CreateMaskedInterleavedStore(
+    uint32_t Factor, ArrayRef<Value *> StoredVals, Value *Ptr, Align Alignment,
+    Value *Mask) {
+  assert(StoredVals.size() == Factor &&
+         "Not enough data to store for given factor");
+  Type *DataTy = (*StoredVals.begin())->getType();
+#ifndef NDEBUG
+  for (auto &Val : StoredVals)
+    assert(Val->getType()->isVectorTy() && "Stored value should be a vector");
+#endif
+  auto *VecTy = cast<VectorType>(DataTy);
+  auto *PtrTy = VecTy->getElementType()->getPointerTo(
+      Ptr->getType()->getPointerAddressSpace());
+  assert(PtrTy->isOpaqueOrPointeeTypeMatches(VecTy->getElementType()) &&
+         "Wrong element type");
+
+  ElementCount NumElts = VecTy->getElementCount();
+  if (!Mask)
+    Mask = Constant::getAllOnesValue(
+        VectorType::get(Type::getInt1Ty(Context), NumElts));
+
+  Type *OverloadedTypes[] = {DataTy};
+  SmallVector<Value *, 8> Ops(StoredVals.begin(), StoredVals.end());
+  Ops.append(
+      {this->CreateBitCast(Ptr, PtrTy), getInt32(Alignment.value()), Mask});
+
+  switch (Factor) {
+  case 2:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved2_store, Ops,
+        OverloadedTypes);
+  case 3:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved3_store, Ops,
+        OverloadedTypes);
+  case 4:
+    return CreateMaskedIntrinsic(
+        Intrinsic::experimental_masked_interleaved4_store, Ops,
+        OverloadedTypes);
+  default:
+    break;
+  }
+  return nullptr;
+}
+
 /// Create a call to a Masked Load intrinsic.
 /// \p Ty        - vector type to load
 /// \p Ptr       - base pointer for the load
 /// \p Alignment - alignment of the source location
 /// \p Mask      - vector of booleans which indicates what vector lanes should
 ///                be accessed in memory
 /// \p PassThru  - pass-through value that is used to fill the masked-off lanes
 ///                of the result

llvm/lib/Target/AArch64/AArch64ISelLowering.h

public:
   bool hasPairedLoad(EVT LoadedType, Align &RequiredAligment) const override;
 
   unsigned getMaxSupportedInterleaveFactor() const override { return 4; }
 
   bool lowerInterleavedLoad(LoadInst *LI,
                             ArrayRef<ShuffleVectorInst *> Shuffles,
                             ArrayRef<unsigned> Indices,
                             unsigned Factor) const override;
+  bool lowerInterleavedLoad(IntrinsicInst *LI) const override;
   bool lowerInterleavedStore(StoreInst *SI, ShuffleVectorInst *SVI,
                              unsigned Factor) const override;
+  bool lowerInterleavedStore(IntrinsicInst *SI) const override;
 
   bool isLegalAddImmediate(int64_t) const override;
   bool isLegalICmpImmediate(int64_t) const override;
 
   bool isMulAddWithConstProfitable(SDValue AddNode,
                                    SDValue ConstNode) const override;
 
   bool shouldConsiderGEPOffsetSplit() const override;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::hasPairedLoad(EVT LoadedType,
   unsigned NumBits = LoadedType.getSizeInBits();
   return NumBits == 32 || NumBits == 64;
 }
 
 /// A helper function for determining the number of interleaved accesses we
 /// will generate when lowering accesses of the given type.
 unsigned AArch64TargetLowering::getNumInterleavedAccesses(
     VectorType *VecTy, const DataLayout &DL, bool UseScalable) const {
+  auto EC = VecTy->getElementCount();
+  unsigned ElSize = DL.getTypeSizeInBits(VecTy->getElementType());
   unsigned VecSize = UseScalable ? Subtarget->getMinSVEVectorSizeInBits() : 128;
-  return std::max<unsigned>(1, (DL.getTypeSizeInBits(VecTy) + 127) / VecSize);
+  return std::max<unsigned>(1,
+                            (EC.getKnownMinValue() * ElSize + 127) / VecSize);
 }
 
 MachineMemOperand::Flags
 AArch64TargetLowering::getTargetMMOFlags(const Instruction &I) const {
   if (Subtarget->getProcFamily() == AArch64Subtarget::Falkor &&
       I.getMetadata(FALKOR_STRIDED_ACCESS_MD) != nullptr)
     return MOStridedAccess;
   return MachineMemOperand::MONone;
 }
 
 bool AArch64TargetLowering::isLegalInterleavedAccessType(
     VectorType *VecTy, const DataLayout &DL, bool &UseScalable) const {
 
-  unsigned VecSize = DL.getTypeSizeInBits(VecTy);
   unsigned ElSize = DL.getTypeSizeInBits(VecTy->getElementType());
-  unsigned NumElements = cast<FixedVectorType>(VecTy)->getNumElements();
+  auto EC = VecTy->getElementCount();
 
   UseScalable = false;
 
   // Ensure the number of vector elements is greater than 1.
-  if (NumElements < 2)
+  if (EC.getKnownMinValue() < 2)
     return false;
 
   // Ensure the element type is legal.
   if (ElSize != 8 && ElSize != 16 && ElSize != 32 && ElSize != 64)
     return false;
 
+  if (EC.isScalable()) {
+    if (EC.getKnownMinValue() * ElSize == 128)
+      return true;
+    return false;
+  }
+
+  unsigned VecSize = DL.getTypeSizeInBits(VecTy);
   if (Subtarget->useSVEForFixedLengthVectors() &&
       (VecSize % Subtarget->getMinSVEVectorSizeInBits() == 0 ||
        (VecSize < Subtarget->getMinSVEVectorSizeInBits() &&
-        isPowerOf2_32(NumElements) && VecSize > 128))) {
+        isPowerOf2_32(EC.getKnownMinValue()) && VecSize > 128))) {
     UseScalable = true;
     return true;
   }
 
   // Ensure the total vector size is 64 or a multiple of 128. Types larger than
   // 128 will be split into multiple interleaved accesses.
   return VecSize == 64 || VecSize % 128 == 0;
 }
for (ShuffleVectorInst *SVI : Shuffles) {
     auto *WideVec =
         SubVec.size() > 1 ? concatenateVectors(Builder, SubVec) : SubVec[0];
     SVI->replaceAllUsesWith(WideVec);
   }
 
   return true;
 }
 
+bool AArch64TargetLowering::lowerInterleavedLoad(IntrinsicInst *LI) const {
+  if (!Subtarget->hasSVE())
+    return false;
+  IRBuilder<> Builder(LI);
+  static const llvm::Intrinsic::ID SVELoadIntrs[3] = {
+      Intrinsic::aarch64_sve_ld2_sret, Intrinsic::aarch64_sve_ld3_sret,
+      Intrinsic::aarch64_sve_ld4_sret};
+  auto *RetTy = dyn_cast<StructType>(LI->getType());
+  if (!RetTy)
+    return false;
+  uint64_t Factor = RetTy->getNumElements();
+  if (Factor > 4 || Factor < 2)
+    return false;
+  auto *LdNFunc = Intrinsic::getDeclaration(
+      LI->getModule(), SVELoadIntrs[Factor - 2], {RetTy->getElementType(0)});
+  auto *AArch64LI =
+      Builder.CreateCall(LdNFunc, {LI->getOperand(2), LI->getOperand(0)});
+  LI->replaceAllUsesWith(AArch64LI);
+  return true;
+}
+
 /// Lower an interleaved store into a stN intrinsic.
 ///
 /// E.g. Lower an interleaved store (Factor = 3):
 ///        %i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1,
 ///                 <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
 ///        store <12 x i32> %i.vec, <12 x i32>* %ptr
 ///
 ///      Into:
if (StoreCount > 0)
                                             BaseAddr, LaneLen * Factor);
 
     Ops.push_back(Builder.CreateBitCast(BaseAddr, PtrTy));
     Builder.CreateCall(StNFunc, Ops);
   }
   return true;
 }
 
+bool AArch64TargetLowering::lowerInterleavedStore(IntrinsicInst *SI) const {
+  if (!Subtarget->hasSVE())
+    return false;
+  IRBuilder<> Builder(SI);
+  static const Intrinsic::ID SVEStoreIntrs[3] = {Intrinsic::aarch64_sve_st2,
+                                                 Intrinsic::aarch64_sve_st3,
+                                                 Intrinsic::aarch64_sve_st4};
+  auto *VecTy = SI->getOperand(0)->getType();
+  if (!VecTy)
+    return false;
+  unsigned Factor = 0;
+  for (auto &Arg : SI->args()) {
+    if (!Arg->getType()->isVectorTy())
+      break;
+    Factor++;
+  }
+
+  if (Factor > 4 || Factor < 2)
+    return false;
+  auto *StNFunc = Intrinsic::getDeclaration(SI->getModule(),
+                                            SVEStoreIntrs[Factor - 2], {VecTy});
+  SmallVector<Value *, 4> Ops(SI->arg_begin(), SI->arg_begin() + Factor);
+  Ops.push_back(SI->getOperand(Factor + 2)); // mask
+  Ops.push_back(SI->getOperand(Factor));     // addr
+  auto *AArch64SI = Builder.CreateCall(StNFunc, Ops);
+  SI->replaceAllUsesWith(AArch64SI);
+  return true;
+}
+
 EVT AArch64TargetLowering::getOptimalMemOpType(
     const MemOp &Op, const AttributeList &FuncAttributes) const {
   bool CanImplicitFloat = !FuncAttributes.hasFnAttr(Attribute::NoImplicitFloat);
   bool CanUseNEON = Subtarget->hasNEON() && CanImplicitFloat;
   bool CanUseFP = Subtarget->hasFPARMv8() && CanImplicitFloat;
   // Only use AdvSIMD to implement memset of 32-byte and above. It would have
   // taken one instruction to materialize the v2i64 zero and one store (with
   // restrictive addressing mode). Just do i64 stores.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

InstructionCost getIntImmCostIntrin(Intrinsic::ID IID, unsigned Idx,
                                       TTI::TargetCostKind CostKind);
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
 
   /// @}
 
   /// \name Vector TTI Implementations
   /// @{
 
-  bool enableInterleavedAccessVectorization() { return true; }
+  bool enableInterleavedAccessVectorization();
+
+  bool hasInterleavedLoad(VectorType *VecTy, Value *Addr, uint32_t Factor,
+                          bool IsMasked);
+
+  bool hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs, Value *Addr,
+                           uint32_t Factor, bool IsMasked);
 
   unsigned getNumberOfRegisters(unsigned ClassID) const {
     bool Vector = (ClassID == 1);
     if (Vector) {
       if (ST->hasNEON())
         return 32;
       return 0;
     }

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

static cl::opt<bool> EnableFalkorHWPFUnrollFix("enable-falkor-hwpf-unroll-fix",
                                                cl::init(true), cl::Hidden);
 
 static cl::opt<unsigned> SVEGatherOverhead("sve-gather-overhead", cl::init(10),
                                            cl::Hidden);
 
 static cl::opt<unsigned> SVEScatterOverhead("sve-scatter-overhead",
                                             cl::init(10), cl::Hidden);
 
+cl::opt<bool> EnableSVEInterleavedMemAccesses(
+    "enable-sve-interleaved-mem-accesses", cl::init(true), cl::Hidden,
+    cl::desc("Enable vectorization on interleaved memory accesses in a loop "
+             "using sve load/store."));
+
+bool AArch64TTIImpl::enableInterleavedAccessVectorization() {
+  return !ST->hasSVE() || EnableSVEInterleavedMemAccesses;
+}
+
+bool AArch64TTIImpl::hasInterleavedLoad(VectorType *VecTy, Value *Addr,
+                                        uint32_t Factor, bool IsMasked) {
+  if (!enableInterleavedAccessVectorization() ||
+      !isa<ScalableVectorType>(VecTy))
+    return false;
+  if (Factor < 1 || Factor > 4)
+    return false;
+
+  return true;
+}
+
+bool AArch64TTIImpl::hasInterleavedStore(SmallVectorImpl<Value *> &StoredVecs,
+                                         Value *Addr, uint32_t Factor,
+                                         bool IsMasked) {
+  return hasInterleavedLoad(
+      dyn_cast<VectorType>((*StoredVecs.begin())->getType()), Addr, Factor,
+      IsMasked);
+}
+
 class TailFoldingKind {
 private:
   uint8_t Bits = 0; // Currently defaults to disabled.
 
 public:
   enum TailFoldingOpts {
     TFDisabled = 0x0,
     TFReductions = 0x01,
InstructionCost AArch64TTIImpl::getMemoryOpCost(unsigned Opcode, Type *Ty,
   return LT.first;
 }
 
 InstructionCost AArch64TTIImpl::getInterleavedMemoryOpCost(
     unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
     Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
     bool UseMaskForCond, bool UseMaskForGaps) {
   assert(Factor >= 2 && "Invalid interleave factor");
-  auto *VecVTy = cast<FixedVectorType>(VecTy);
-
+  auto *VecVTy = cast<VectorType>(VecTy);
   if (!UseMaskForCond && !UseMaskForGaps &&
       Factor <= TLI->getMaxSupportedInterleaveFactor()) {
-    unsigned NumElts = VecVTy->getNumElements();
+    unsigned NumElts = VecVTy->getElementCount().getKnownMinValue();
     auto *SubVecTy =
-        FixedVectorType::get(VecTy->getScalarType(), NumElts / Factor);
+        VectorType::get(VecVTy->getElementType(),
+                        VecVTy->getElementCount().divideCoefficientBy(Factor));
 
     // ldN/stN only support legal vector types of size 64 or 128 in bits.
     // Accesses having vector types that are a multiple of 128 bits can be
     // matched to more than one ldN/stN instruction.
     bool UseScalable;
-    if (NumElts % Factor == 0 &&
-        TLI->isLegalInterleavedAccessType(SubVecTy, DL, UseScalable))
-      return Factor * TLI->getNumInterleavedAccesses(SubVecTy, DL, UseScalable);
+    if (NumElts % Factor == 0) {
+      if (TLI->isLegalInterleavedAccessType(SubVecTy, DL, UseScalable))
+        return Factor *
+               TLI->getNumInterleavedAccesses(SubVecTy, DL, UseScalable);
+    }
   }
+  // Calling the base implementation should only happen
+  // when compiler wants to scalarize the operation.
+  if (isa<ScalableVectorType>(VecTy))
+    return InstructionCost::getInvalid();
 
   return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
                                            Alignment, AddressSpace, CostKind,
                                            UseMaskForCond, UseMaskForGaps);
 }
 
 InstructionCost
 AArch64TTIImpl::getCostOfKeepingLiveOverCall(ArrayRef<Type *> Tys) {

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

void InnerLoopVectorizer::vectorizeInterleaveGroup(
     VPTransformState &State, VPValue *Addr, ArrayRef<VPValue *> StoredValues,
     VPValue *BlockInMask) {
   Instruction *Instr = Group->getInsertPos();
   const DataLayout &DL = Instr->getModule()->getDataLayout();
 
   // Prepare for the vector type of the interleaved load/store.
   Type *ScalarTy = getLoadStoreType(Instr);
   unsigned InterleaveFactor = Group->getFactor();
-  assert(!VF.isScalable() && "scalable vectors not yet supported.");
   auto *VecTy = VectorType::get(ScalarTy, VF * InterleaveFactor);
 
   // Prepare for the new pointers.
   SmallVector<Value *, 2> AddrParts;
   unsigned Index = Group->getIndex(Instr);
 
   // TODO: extend the masked interleaved-group support to reversed access.
   assert((!BlockInMask || !Group->isReverse()) &&
          "Reversed masked interleave-group not supported.");
 
+  Value *Idx;
   // If the group is reverse, adjust the index to refer to the last vector lane
   // instead of the first. We adjust the index from the first vector lane,
   // rather than directly getting the pointer for lane VF - 1, because the
   // pointer operand of the interleaved access is supposed to be uniform. For
   // uniform instructions, we're only required to generate a value for the
   // first vector lane in each unroll iteration.
-  if (Group->isReverse())
-    Index += (VF.getKnownMinValue() - 1) * Group->getFactor();
+  if (Group->isReverse()) {
+    Value *RuntimeVF = getRuntimeVF(Builder, Builder.getInt32Ty(), VF);
+    Idx = Builder.CreateSub(RuntimeVF, Builder.getInt32(1));
+    Idx = Builder.CreateMul(Idx, Builder.getInt32(Group->getFactor()));
+    Idx = Builder.CreateAdd(Idx, Builder.getInt32(Index));
+    Idx = Builder.CreateNeg(Idx);
+  } else
+    Idx = Builder.getInt32(-Index);
 
   for (unsigned Part = 0; Part < UF; Part++) {
     Value *AddrPart = State.get(Addr, VPIteration(Part, 0));
     State.setDebugLocFromInst(AddrPart);
 
     // Notice current instruction could be any index. Need to adjust the address
     // to the member of index 0.
     //
     // E.g.  a = A[i+1];     // Member of index 1 (Current instruction)
     //       b = A[i];       // Member of index 0
     // Current pointer is pointed to A[i+1], adjust it to A[i].
     //
     // E.g.  A[i+1] = a;     // Member of index 1
     //       A[i]   = b;     // Member of index 0
     //       A[i+2] = c;     // Member of index 2 (Current instruction)
     // Current pointer is pointed to A[i+2], adjust it to A[i].
 
     bool InBounds = false;
     if (auto *gep = dyn_cast<GetElementPtrInst>(AddrPart->stripPointerCasts()))
       InBounds = gep->isInBounds();
-    AddrPart = Builder.CreateGEP(ScalarTy, AddrPart, Builder.getInt32(-Index));
+    AddrPart = Builder.CreateGEP(ScalarTy, AddrPart, Idx);
     cast<GetElementPtrInst>(AddrPart)->setIsInBounds(InBounds);
 
     // Cast to the vector pointer type.
     unsigned AddressSpace = AddrPart->getType()->getPointerAddressSpace();
     Type *PtrTy = VecTy->getPointerTo(AddressSpace);
     AddrParts.push_back(Builder.CreateBitCast(AddrPart, PtrTy));
   }
 
for (unsigned Part = 0; Part < UF; Part++) {
           GroupMask = MaskForGaps
                           ? Builder.CreateBinOp(Instruction::And, ShuffledMask,
                                                 MaskForGaps)
                           : ShuffledMask;
         }
         NewLoad =
             Builder.CreateMaskedLoad(VecTy, AddrParts[Part], Group->getAlign(),
                                      GroupMask, PoisonVec, "wide.masked.vec");
-      }
+      } else {
+        // Check if we can create target specific interleaving load.
+        // We can not express alignment so just drop it for now.
+        if (TTI->hasInterleavedLoad(VecTy, AddrParts[Part], Group->getFactor(),
+                                    false))
+          NewLoad = Builder.CreateMaskedInterleavedLoad(
+              Group->getFactor(), VecTy, AddrParts[Part], Group->getAlign());
         else
           NewLoad = Builder.CreateAlignedLoad(VecTy, AddrParts[Part],
                                               Group->getAlign(), "wide.vec");
+      }
       Group->addMetadata(NewLoad);
       NewLoads.push_back(NewLoad);
     }
 
     // For each member in the group, shuffle out the appropriate data from the
     // wide loads.
     unsigned J = 0;
     for (unsigned I = 0; I < InterleaveFactor; ++I) {
       Instruction *Member = Group->getMember(I);
 
       // Skip the gaps in the group.
       if (!Member)
         continue;
 
       auto StrideMask =
           createStrideMask(I, InterleaveFactor, VF.getKnownMinValue());
       for (unsigned Part = 0; Part < UF; Part++) {
-        Value *StridedVec = Builder.CreateShuffleVector(
-            NewLoads[Part], StrideMask, "strided.vec");
+        Value *StridedVec;
+        if (NewLoads[Part]->getType()->isAggregateType())
+          StridedVec = Builder.CreateExtractValue(NewLoads[Part], I);
+        else
+          StridedVec = Builder.CreateShuffleVector(NewLoads[Part], StrideMask,
+                                                   "strided.vec");
 
         // If this member has different type, cast the result type.
         if (Member->getType() != ScalarTy) {
-          assert(!VF.isScalable() && "VF is assumed to be non scalable.");
           VectorType *OtherVTy = VectorType::get(Member->getType(), VF);
           StridedVec = createBitOrPointerCast(StridedVec, OtherVTy, DL);
         }
 
         if (Group->isReverse())
           StridedVec = Builder.CreateVectorReverse(StridedVec, "reverse");
 
         State.set(VPDefs[J], StridedVec, Part);
for (unsigned i = 0; i < InterleaveFactor; i++) {
       // If this member has different type, cast it to a unified type.
 
       if (StoredVec->getType() != SubVT)
         StoredVec = createBitOrPointerCast(StoredVec, SubVT, DL);
 
       StoredVecs.push_back(StoredVec);
     }
 
+    // Check if we can create target specific interleaving store.
+    // We can not express alignment so just drop it for now.
+    if (TTI->hasInterleavedStore(StoredVecs, AddrParts[Part],
+                                 Group->getFactor(), false)) {
+      CallInst *Store = Builder.CreateMaskedInterleavedStore(
+          Group->getFactor(), StoredVecs, AddrParts[Part], Group->getAlign());
+
+      // create interleaved store
+      Group->addMetadata(Store);
+      continue;
+    }
     // Concatenate all vectors into a wide vector.
     Value *WideVec = concatenateVectors(Builder, StoredVecs);
 
     // Interleave the elements in the wide vector.
     Value *IVec = Builder.CreateShuffleVector(
         WideVec, createInterleaveMask(VF.getKnownMinValue(), InterleaveFactor),
         "interleaved.vec");
 
InnerLoopVectorizer::getOrCreateVectorTripCount(BasicBlock *InsertBlock) {
   VectorTripCount = Builder.CreateSub(TC, R, "n.vec");
 
   return VectorTripCount;
 }
 
 Value *InnerLoopVectorizer::createBitOrPointerCast(Value *V, VectorType *DstVTy,
                                                    const DataLayout &DL) {
   // Verify that V is a vector type with same number of elements as DstVTy.
-  auto *DstFVTy = cast<FixedVectorType>(DstVTy);
-  unsigned VF = DstFVTy->getNumElements();
-  auto *SrcVecTy = cast<FixedVectorType>(V->getType());
-  assert((VF == SrcVecTy->getNumElements()) && "Vector dimensions do not match");
+  auto *DstFVTy = cast<VectorType>(DstVTy);
+  auto VF = DstFVTy->getElementCount();
+  auto *SrcVecTy = cast<VectorType>(V->getType());
+  assert((VF == SrcVecTy->getElementCount()) &&
+         "Vector dimensions do not match");
   Type *SrcElemTy = SrcVecTy->getElementType();
   Type *DstElemTy = DstFVTy->getElementType();
   assert((DL.getTypeSizeInBits(SrcElemTy) == DL.getTypeSizeInBits(DstElemTy)) &&
          "Vector elements must have same size");
 
   // Do a direct cast if element types are castable.
   if (CastInst::isBitOrNoopPointerCastable(SrcElemTy, DstElemTy, DL)) {
     return Builder.CreateBitOrPointerCast(V, DstFVTy);
   }
   // V cannot be directly casted to desired vector type.
   // May happen when V is a floating point vector but DstVTy is a vector of
   // pointers or vice-versa. Handle this using a two-step bitcast using an
   // intermediate Integer type for the bitcast i.e. Ptr <-> Int <-> Float.
   assert((DstElemTy->isPointerTy() != SrcElemTy->isPointerTy()) &&
          "Only one type should be a pointer type");
   assert((DstElemTy->isFloatingPointTy() != SrcElemTy->isFloatingPointTy()) &&
          "Only one type should be a floating point type");
   Type *IntTy =
       IntegerType::getIntNTy(V->getContext(), DL.getTypeSizeInBits(SrcElemTy));
-  auto *VecIntTy = FixedVectorType::get(IntTy, VF);
+  auto *VecIntTy = VectorType::get(IntTy, VF);
   Value *CastVal = Builder.CreateBitOrPointerCast(V, VecIntTy);
   return Builder.CreateBitOrPointerCast(CastVal, DstFVTy);
 }
 
 void InnerLoopVectorizer::emitIterationCountCheck(BasicBlock *Bypass) {
   Value *Count = getOrCreateTripCount(LoopVectorPreHeader);
   // Reuse existing vector loop preheader for TC checks.
   // Note that new preheader block is generated for vector loop.
return TTI.getAddressComputationCost(VectorTy) +
          TTI.getGatherScatterOpCost(
              I->getOpcode(), VectorTy, Ptr, Legal->isMaskRequired(I), Alignment,
              TargetTransformInfo::TCK_RecipThroughput, I);
 }
 
 InstructionCost
 LoopVectorizationCostModel::getInterleaveGroupCost(Instruction *I,
                                                    ElementCount VF) {
-  // TODO: Once we have support for interleaving with scalable vectors
-  // we can calculate the cost properly here.
-  if (VF.isScalable())
-    return InstructionCost::getInvalid();
-
   Type *ValTy = getLoadStoreType(I);
   auto *VectorTy = cast<VectorType>(ToVectorTy(ValTy, VF));
   unsigned AS = getLoadStoreAddressSpace(I);
   enum TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput;
 
   auto Group = getInterleavedAccessGroup(I);
   assert(Group && "Fail to get an interleaved access group.");
 

llvm/test/Transforms/InterleavedAccess/AArch64/sve-interleaved-accesses-load.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -interleaved-access -S | FileCheck %s -check-prefix=NEON
+; RUN: opt < %s -mattr=+sve -interleaved-access -S | FileCheck %s -check-prefix=SVE
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64--linux-gnu"
+
+; ld2b
+define { <vscale x 16 x i8>, <vscale x 16 x i8> } @ld2.nxv16i8(<vscale x 16 x i1> %Pg, i8 *%addr) {
+; NEON-LABEL: @ld2.nxv16i8(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved2.load.nxv16i8(i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PG:%.*]], <vscale x 16 x i8> undef)
+; NEON-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv16i8(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.aarch64.sve.ld2.sret.nxv16i8(<vscale x 16 x i1> [[PG:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8> } [[LDN]]
+;
+  %res = call { <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved2.load.nxv16i8(i8 *%addr, i32 0, <vscale x 16 x i1> %Pg, <vscale x 16 x i8> undef)
+  ret { <vscale x 16 x i8>, <vscale x 16 x i8> } %res
+}
+
+; ld2h
+define { <vscale x 8 x i16>, <vscale x 8 x i16> } @ld2.nxv8i16(<vscale x 8 x i1> %Pg, i16 *%addr) {
+; NEON-LABEL: @ld2.nxv8i16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved2.load.nxv8i16(i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x i16> undef)
+; NEON-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv8i16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.aarch64.sve.ld2.sret.nxv8i16(<vscale x 8 x i1> [[PG:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16> } [[LDN]]
+;
+  %res = call { <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved2.load.nxv8i16(i16 *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x i16> undef)
+  ret { <vscale x 8 x i16>, <vscale x 8 x i16> } %res
+}
+
+define { <vscale x 8 x half>, <vscale x 8 x half> } @ld2.nxv8f16(<vscale x 8 x i1> %Pg, half *%addr) {
+; NEON-LABEL: @ld2.nxv8f16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved2.load.nxv8f16(half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x half> undef)
+; NEON-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv8f16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half> } @llvm.aarch64.sve.ld2.sret.nxv8f16(<vscale x 8 x i1> [[PG:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half> } [[LDN]]
+;
+  %res = call { <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved2.load.nxv8f16(half *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x half> undef)
+  ret { <vscale x 8 x half>, <vscale x 8 x half> } %res
+}
+
+define { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @ld2.nxv8bf16(<vscale x 8 x i1> %Pg, bfloat *%addr) #0 {
+; NEON-LABEL: @ld2.nxv8bf16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved2.load.nxv8bf16(bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x bfloat> undef)
+; NEON-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv8bf16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.aarch64.sve.ld2.sret.nxv8bf16(<vscale x 8 x i1> [[PG:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[LDN]]
+;
+  %res = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved2.load.nxv8bf16(bfloat *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x bfloat> undef)
+  ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } %res
+}
+
+; ld2w
+define { <vscale x 4 x i32>, <vscale x 4 x i32> } @ld2.nxv4i32(<vscale x 4 x i1> %Pg, i32 *%addr) {
+; NEON-LABEL: @ld2.nxv4i32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x i32> undef)
+; NEON-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv4i32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.aarch64.sve.ld2.sret.nxv4i32(<vscale x 4 x i1> [[PG:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32> } [[LDN]]
+;
+  %res = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32 *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x i32> undef)
+  ret { <vscale x 4 x i32>, <vscale x 4 x i32> } %res
+}
+
+define { <vscale x 4 x float>, <vscale x 4 x float> } @ld2.nxv4f32(<vscale x 4 x i1> %Pg, float *%addr) {
+; NEON-LABEL: @ld2.nxv4f32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x float> undef)
+; NEON-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv4f32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.aarch64.sve.ld2.sret.nxv4f32(<vscale x 4 x i1> [[PG:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float> } [[LDN]]
+;
+  %res = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(float *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x float> undef)
+  ret { <vscale x 4 x float>, <vscale x 4 x float> } %res
+}
+
+; ld2d
+define { <vscale x 2 x i64>, <vscale x 2 x i64> } @ld2.nxv2i64(<vscale x 2 x i1> %Pg, i64 *%addr) {
+; NEON-LABEL: @ld2.nxv2i64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved2.load.nxv2i64(i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x i64> undef)
+; NEON-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv2i64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.aarch64.sve.ld2.sret.nxv2i64(<vscale x 2 x i1> [[PG:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64> } [[LDN]]
+;
+  %res = call { <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved2.load.nxv2i64(i64 *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x i64> undef)
+  ret { <vscale x 2 x i64>, <vscale x 2 x i64> } %res
+}
+
+define { <vscale x 2 x double>, <vscale x 2 x double> } @ld2.nxv2f64(<vscale x 2 x i1> %Pg, double *%addr) {
+; NEON-LABEL: @ld2.nxv2f64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved2.load.nxv2f64(double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x double> undef)
+; NEON-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double> } [[RES]]
+;
+; SVE-LABEL: @ld2.nxv2f64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double> } @llvm.aarch64.sve.ld2.sret.nxv2f64(<vscale x 2 x i1> [[PG:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double> } [[LDN]]
+;
+  %res = call { <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved2.load.nxv2f64(double *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x double> undef)
+  ret { <vscale x 2 x double>, <vscale x 2 x double> } %res
+}
+
+; ld3b
+define { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @ld3.nxv16i8(<vscale x 16 x i1> %Pg, i8 *%addr) {
+; NEON-LABEL: @ld3.nxv16i8(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved3.load.nxv16i8(i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PG:%.*]], <vscale x 16 x i8> undef)
+; NEON-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv16i8(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.aarch64.sve.ld3.sret.nxv16i8(<vscale x 16 x i1> [[PG:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } [[LDN]]
+;
+  %res = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved3.load.nxv16i8(i8 *%addr, i32 0, <vscale x 16 x i1> %Pg, <vscale x 16 x i8> undef)
+  ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } %res
+}
+
+; ld3h
+define { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @ld3.nxv8i16(<vscale x 8 x i1> %Pg, i16 *%addr) {
+; NEON-LABEL: @ld3.nxv8i16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved3.load.nxv8i16(i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x i16> undef)
+; NEON-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv8i16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.aarch64.sve.ld3.sret.nxv8i16(<vscale x 8 x i1> [[PG:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } [[LDN]]
+;
+  %res = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved3.load.nxv8i16(i16 *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x i16> undef)
+  ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } %res
+}
+
+define { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @ld3.nxv8f16(<vscale x 8 x i1> %Pg, half *%addr) {
+; NEON-LABEL: @ld3.nxv8f16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved3.load.nxv8f16(half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x half> undef)
+; NEON-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv8f16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.aarch64.sve.ld3.sret.nxv8f16(<vscale x 8 x i1> [[PG:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } [[LDN]]
+;
+  %res = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved3.load.nxv8f16(half *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x half> undef)
+  ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } %res
+}
+
+define { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @ld3.nxv8bf16(<vscale x 8 x i1> %Pg, bfloat *%addr) #0 {
+; NEON-LABEL: @ld3.nxv8bf16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved3.load.nxv8bf16(bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x bfloat> undef)
+; NEON-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv8bf16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.aarch64.sve.ld3.sret.nxv8bf16(<vscale x 8 x i1> [[PG:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[LDN]]
+;
+  %res = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved3.load.nxv8bf16(bfloat *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x bfloat> undef)
+  ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } %res
+}
+
+; ld3w
+define { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @ld3.nxv4i32(<vscale x 4 x i1> %Pg, i32 *%addr) {
+; NEON-LABEL: @ld3.nxv4i32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved3.load.nxv4i32(i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x i32> undef)
+; NEON-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv4i32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.aarch64.sve.ld3.sret.nxv4i32(<vscale x 4 x i1> [[PG:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } [[LDN]]
+;
+  %res = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved3.load.nxv4i32(i32 *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x i32> undef)
+  ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } %res
+}
+
+define { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @ld3.nxv4f32(<vscale x 4 x i1> %Pg, float *%addr) {
+; NEON-LABEL: @ld3.nxv4f32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved3.load.nxv4f32(float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x float> undef)
+; NEON-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv4f32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.aarch64.sve.ld3.sret.nxv4f32(<vscale x 4 x i1> [[PG:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } [[LDN]]
+;
+  %res = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved3.load.nxv4f32(float *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x float> undef)
+  ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } %res
+}
+
+; ld3d
+define { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @ld3.nxv2i64(<vscale x 2 x i1> %Pg, i64 *%addr) {
+; NEON-LABEL: @ld3.nxv2i64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved3.load.nxv2i64(i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x i64> undef)
+; NEON-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv2i64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.aarch64.sve.ld3.sret.nxv2i64(<vscale x 2 x i1> [[PG:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } [[LDN]]
+;
+  %res = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved3.load.nxv2i64(i64 *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x i64> undef)
+  ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } %res
+}
+
+define { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @ld3.nxv2f64(<vscale x 2 x i1> %Pg, double *%addr) {
+; NEON-LABEL: @ld3.nxv2f64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved3.load.nxv2f64(double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x double> undef)
+; NEON-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } [[RES]]
+;
+; SVE-LABEL: @ld3.nxv2f64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.aarch64.sve.ld3.sret.nxv2f64(<vscale x 2 x i1> [[PG:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } [[LDN]]
+;
+  %res = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved3.load.nxv2f64(double *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x double> undef)
+  ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } %res
+}
+
+; ld4b
+define { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @ld4.nxv16i8(<vscale x 16 x i1> %Pg, i8 *%addr) {
+; NEON-LABEL: @ld4.nxv16i8(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved4.load.nxv16i8(i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PG:%.*]], <vscale x 16 x i8> undef)
+; NEON-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv16i8(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.aarch64.sve.ld4.sret.nxv16i8(<vscale x 16 x i1> [[PG:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } [[LDN]]
+;
+  %res = call { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved4.load.nxv16i8(i8 *%addr, i32 0, <vscale x 16 x i1> %Pg, <vscale x 16 x i8> undef)
+  ret { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } %res
+}
+
+; ld4h
+define { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @ld4.nxv8i16(<vscale x 8 x i1> %Pg, i16 *%addr) {
+; NEON-LABEL: @ld4.nxv8i16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved4.load.nxv8i16(i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x i16> undef)
+; NEON-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv8i16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.aarch64.sve.ld4.sret.nxv8i16(<vscale x 8 x i1> [[PG:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } [[LDN]]
+;
+  %res = call { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved4.load.nxv8i16(i16 *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x i16> undef)
+  ret { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } %res
+}
+
+define { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @ld4.nxv8f16(<vscale x 8 x i1> %Pg, half *%addr) {
+; NEON-LABEL: @ld4.nxv8f16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved4.load.nxv8f16(half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x half> undef)
+; NEON-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv8f16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.aarch64.sve.ld4.sret.nxv8f16(<vscale x 8 x i1> [[PG:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } [[LDN]]
+;
+  %res = call { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved4.load.nxv8f16(half *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x half> undef)
+  ret { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } %res
+}
+
+define { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @ld4.nxv8bf16(<vscale x 8 x i1> %Pg, bfloat *%addr) #0 {
+; NEON-LABEL: @ld4.nxv8bf16(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved4.load.nxv8bf16(bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PG:%.*]], <vscale x 8 x bfloat> undef)
+; NEON-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv8bf16(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.aarch64.sve.ld4.sret.nxv8bf16(<vscale x 8 x i1> [[PG:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } [[LDN]]
+;
+  %res = call { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved4.load.nxv8bf16(bfloat *%addr, i32 0, <vscale x 8 x i1> %Pg, <vscale x 8 x bfloat> undef)
+  ret { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } %res
+}
+
+; ld4w
+define { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @ld4.nxv4i32(<vscale x 4 x i1> %Pg, i32 *%addr) {
+; NEON-LABEL: @ld4.nxv4i32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved4.load.nxv4i32(i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x i32> undef)
+; NEON-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv4i32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.aarch64.sve.ld4.sret.nxv4i32(<vscale x 4 x i1> [[PG:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } [[LDN]]
+;
+  %res = call { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved4.load.nxv4i32(i32 *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x i32> undef)
+  ret { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } %res
+}
+
+define { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @ld4.nxv4f32(<vscale x 4 x i1> %Pg, float *%addr) {
+; NEON-LABEL: @ld4.nxv4f32(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved4.load.nxv4f32(float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PG:%.*]], <vscale x 4 x float> undef)
+; NEON-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv4f32(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.aarch64.sve.ld4.sret.nxv4f32(<vscale x 4 x i1> [[PG:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } [[LDN]]
+;
+  %res = call { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved4.load.nxv4f32(float *%addr, i32 0, <vscale x 4 x i1> %Pg, <vscale x 4 x float> undef)
+  ret { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } %res
+}
+
+; ld4d
+define { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @ld4.nxv2i64(<vscale x 2 x i1> %Pg, i64 *%addr) {
+; NEON-LABEL: @ld4.nxv2i64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved4.load.nxv2i64(i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x i64> undef)
+; NEON-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv2i64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.aarch64.sve.ld4.sret.nxv2i64(<vscale x 2 x i1> [[PG:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } [[LDN]]
+;
+  %res = call { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved4.load.nxv2i64(i64 *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x i64> undef)
+  ret { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } %res
+}
+
+define { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @ld4.nxv2f64(<vscale x 2 x i1> %Pg, double *%addr) {
+; NEON-LABEL: @ld4.nxv2f64(
+; NEON-NEXT:    [[RES:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved4.load.nxv2f64(double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PG:%.*]], <vscale x 2 x double> undef)
+; NEON-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } [[RES]]
+;
+; SVE-LABEL: @ld4.nxv2f64(
+; SVE-NEXT:    [[LDN:%.*]] = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.aarch64.sve.ld4.sret.nxv2f64(<vscale x 2 x i1> [[PG:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } [[LDN]]
+;
+  %res = call { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved4.load.nxv2f64(double *%addr, i32 0, <vscale x 2 x i1> %Pg, <vscale x 2 x double> undef)
+  ret { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } %res
+}
+
+declare { <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved2.load.nxv16i8(i8*, i32, <vscale x 16 x i1>, <vscale x 16 x i8>)
+declare { <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved2.load.nxv8i16(i16*, i32, <vscale x 8 x i1>, <vscale x 8 x i16>)
+declare { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32*, i32, <vscale x 4 x i1>, <vscale x 4 x i32>)
+declare { <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved2.load.nxv2i64(i64*, i32, <vscale x 2 x i1>, <vscale x 2 x i64>)
+declare { <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved2.load.nxv8f16(half*, i32, <vscale x 8 x i1>, <vscale x 8 x half>)
+declare { <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved2.load.nxv8bf16(bfloat*, i32, <vscale x 8 x i1>, <vscale x 8 x bfloat>)
+declare { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(float*, i32, <vscale x 4 x i1>, <vscale x 4 x float>)
+declare { <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved2.load.nxv2f64(double*, i32, <vscale x 2 x i1>, <vscale x 2 x double>)
+
+declare { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved3.load.nxv16i8(i8*, i32, <vscale x 16 x i1>, <vscale x 16 x i8>)
+declare { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved3.load.nxv8i16(i16*, i32, <vscale x 8 x i1>, <vscale x 8 x i16>)
+declare { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved3.load.nxv4i32(i32*, i32, <vscale x 4 x i1>, <vscale x 4 x i32>)
+declare { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved3.load.nxv2i64(i64*, i32, <vscale x 2 x i1>, <vscale x 2 x i64>)
+declare { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved3.load.nxv8f16(half*, i32, <vscale x 8 x i1>, <vscale x 8 x half>)
+declare { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved3.load.nxv8bf16(bfloat*, i32, <vscale x 8 x i1>, <vscale x 8 x bfloat>)
+declare { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved3.load.nxv4f32(float*, i32, <vscale x 4 x i1>, <vscale x 4 x float>)
+declare { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved3.load.nxv2f64(double*, i32, <vscale x 2 x i1>, <vscale x 2 x double>)
+
+declare { <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8> } @llvm.experimental.masked.interleaved4.load.nxv16i8(i8*, i32, <vscale x 16 x i1>, <vscale x 16 x i8>)
+declare { <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16> } @llvm.experimental.masked.interleaved4.load.nxv8i16(i16*, i32, <vscale x 8 x i1>, <vscale x 8 x i16>)
+declare { <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved4.load.nxv4i32(i32*, i32, <vscale x 4 x i1>, <vscale x 4 x i32>)
+declare { <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64>, <vscale x 2 x i64> } @llvm.experimental.masked.interleaved4.load.nxv2i64(i64*, i32, <vscale x 2 x i1>, <vscale x 2 x i64>)
+declare { <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half>, <vscale x 8 x half> } @llvm.experimental.masked.interleaved4.load.nxv8f16(half*, i32, <vscale x 8 x i1>, <vscale x 8 x half>)
+declare { <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat>, <vscale x 8 x bfloat> } @llvm.experimental.masked.interleaved4.load.nxv8bf16(bfloat*, i32, <vscale x 8 x i1>, <vscale x 8 x bfloat>)
+declare { <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved4.load.nxv4f32(float*, i32, <vscale x 4 x i1>, <vscale x 4 x float>)
+declare { <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double>, <vscale x 2 x double> } @llvm.experimental.masked.interleaved4.load.nxv2f64(double*, i32, <vscale x 2 x i1>, <vscale x 2 x double>)
+
+
+; +bf16 is required for the bfloat version.
+attributes #0 = { "target-features"="+bf16" }

llvm/test/Transforms/InterleavedAccess/AArch64/sve-interleaved-accesses-store.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -interleaved-access -S | FileCheck %s -check-prefix=NEON
+; RUN: opt < %s -mattr=+sve -interleaved-access -S | FileCheck %s -check-prefix=SVE
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64--linux-gnu"
+
+;
+; ST2B
+;
+
+define void @st2.nxv16i8(<vscale x 16 x i8> %v0, <vscale x 16 x i8> %v1, <vscale x 16 x i1> %pred, i8* %addr) {
+; NEON-LABEL: @st2.nxv16i8(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2.nxv16i8(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], <vscale x 16 x i1> [[PRED:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv16i8(<vscale x 16 x i8> %v0,
+  <vscale x 16 x i8> %v1,
+  i8* %addr, i32 0,
+  <vscale x 16 x i1> %pred)
+  ret void
+}
+
+;
+; ST2H
+;
+
+define void @st2.nxv8i16(<vscale x 8 x i16> %v0, <vscale x 8 x i16> %v1, <vscale x 8 x i1> %pred, i16* %addr) {
+; NEON-LABEL: @st2.nxv8i16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2.nxv8i16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], <vscale x 8 x i1> [[PRED:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv8i16(<vscale x 8 x i16> %v0,
+  <vscale x 8 x i16> %v1,
+  i16* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st2.nxv8f16(<vscale x 8 x half> %v0, <vscale x 8 x half> %v1, <vscale x 8 x i1> %pred, half* %addr) {
+; NEON-LABEL: @st2.nxv8f16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2.nxv8f16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], <vscale x 8 x i1> [[PRED:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv8f16(<vscale x 8 x half> %v0,
+  <vscale x 8 x half> %v1,
+  half* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st2.nxv8bf16(<vscale x 8 x bfloat> %v0, <vscale x 8 x bfloat> %v1, <vscale x 8 x i1> %pred, bfloat* %addr) #0 {
+; NEON-LABEL: @st2.nxv8bf16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2.nxv8bf16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], <vscale x 8 x i1> [[PRED:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv8bf16(<vscale x 8 x bfloat> %v0,
+  <vscale x 8 x bfloat> %v1,
+  bfloat* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+;
+; ST2W
+;
+
+define void @st2_nxv4i32(<vscale x 4 x i32> %v0, <vscale x 4 x i32> %v1, <vscale x 4 x i1> %pred, i32* %addr) {
+; NEON-LABEL: @st2_nxv4i32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2_nxv4i32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], <vscale x 4 x i1> [[PRED:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32> %v0,
+  <vscale x 4 x i32> %v1,
+  i32* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+define void @st2_nx4f32(<vscale x 4 x float> %v0, <vscale x 4 x float> %v1, <vscale x 4 x i1> %pred, float* %addr) {
+; NEON-LABEL: @st2_nx4f32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2_nx4f32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], <vscale x 4 x i1> [[PRED:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv4f32(<vscale x 4 x float> %v0,
+  <vscale x 4 x float> %v1,
+  float* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+;
+; ST2D
+;
+
+define void @st2_nx2i64(<vscale x 2 x i64> %v0, <vscale x 2 x i64> %v1, <vscale x 2 x i1> %pred, i64* %addr) {
+; NEON-LABEL: @st2_nx2i64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2_nx2i64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], <vscale x 2 x i1> [[PRED:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv2i64(<vscale x 2 x i64> %v0,
+  <vscale x 2 x i64> %v1,
+  i64* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+define void @st2_nx2f64(<vscale x 2 x double> %v0, <vscale x 2 x double> %v1, <vscale x 2 x i1> %pred, double* %addr) {
+; NEON-LABEL: @st2_nx2f64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st2_nx2f64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st2.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], <vscale x 2 x i1> [[PRED:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved2.store.nxv2f64(<vscale x 2 x double> %v0,
+  <vscale x 2 x double> %v1,
+  double* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+;
+; ST3B
+;
+
+define void @st3_nx16i8(<vscale x 16 x i8> %v0, <vscale x 16 x i8> %v1, <vscale x 16 x i8> %v2, <vscale x 16 x i1> %pred, i8* %addr) {
+; NEON-LABEL: @st3_nx16i8(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], <vscale x 16 x i8> [[V2:%.*]], i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx16i8(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], <vscale x 16 x i8> [[V2:%.*]], <vscale x 16 x i1> [[PRED:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv16i8(<vscale x 16 x i8> %v0,
+  <vscale x 16 x i8> %v1,
+  <vscale x 16 x i8> %v2,
+  i8* %addr, i32 0,
+  <vscale x 16 x i1> %pred)
+  ret void
+}
+
+;
+; ST3H
+;
+
+define void @st3_nx8i16(<vscale x 8 x i16> %v0, <vscale x 8 x i16> %v1, <vscale x 8 x i16> %v2, <vscale x 8 x i1> %pred, i16* %addr) {
+; NEON-LABEL: @st3_nx8i16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], <vscale x 8 x i16> [[V2:%.*]], i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx8i16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], <vscale x 8 x i16> [[V2:%.*]], <vscale x 8 x i1> [[PRED:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv8i16(<vscale x 8 x i16> %v0,
+  <vscale x 8 x i16> %v1,
+  <vscale x 8 x i16> %v2,
+  i16* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st3_nx8f16(<vscale x 8 x half> %v0, <vscale x 8 x half> %v1, <vscale x 8 x half> %v2, <vscale x 8 x i1> %pred, half* %addr) {
+; NEON-LABEL: @st3_nx8f16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], <vscale x 8 x half> [[V2:%.*]], half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx8f16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], <vscale x 8 x half> [[V2:%.*]], <vscale x 8 x i1> [[PRED:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv8f16(<vscale x 8 x half> %v0,
+  <vscale x 8 x half> %v1,
+  <vscale x 8 x half> %v2,
+  half* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st3_nx8bf16(<vscale x 8 x bfloat> %v0, <vscale x 8 x bfloat> %v1, <vscale x 8 x bfloat> %v2, <vscale x 8 x i1> %pred, bfloat* %addr) #0 {
+; NEON-LABEL: @st3_nx8bf16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], <vscale x 8 x bfloat> [[V2:%.*]], bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx8bf16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], <vscale x 8 x bfloat> [[V2:%.*]], <vscale x 8 x i1> [[PRED:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv8bf16(<vscale x 8 x bfloat> %v0,
+  <vscale x 8 x bfloat> %v1,
+  <vscale x 8 x bfloat> %v2,
+  bfloat* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+;
+; ST3W
+;
+
+define void @st3_nx4i32(<vscale x 4 x i32> %v0, <vscale x 4 x i32> %v1, <vscale x 4 x i32> %v2, <vscale x 4 x i1> %pred, i32* %addr) {
+; NEON-LABEL: @st3_nx4i32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], <vscale x 4 x i32> [[V2:%.*]], i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx4i32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], <vscale x 4 x i32> [[V2:%.*]], <vscale x 4 x i1> [[PRED:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv4i32(<vscale x 4 x i32> %v0,
+  <vscale x 4 x i32> %v1,
+  <vscale x 4 x i32> %v2,
+  i32* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+define void @st3_nx4f32(<vscale x 4 x float> %v0, <vscale x 4 x float> %v1, <vscale x 4 x float> %v2, <vscale x 4 x i1> %pred, float* %addr) {
+; NEON-LABEL: @st3_nx4f32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], <vscale x 4 x float> [[V2:%.*]], float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx4f32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], <vscale x 4 x float> [[V2:%.*]], <vscale x 4 x i1> [[PRED:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv4f32(<vscale x 4 x float> %v0,
+  <vscale x 4 x float> %v1,
+  <vscale x 4 x float> %v2,
+  float* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+;
+; ST3D
+;
+
+define void @st3_nx2i64(<vscale x 2 x i64> %v0, <vscale x 2 x i64> %v1, <vscale x 2 x i64> %v2, <vscale x 2 x i1> %pred, i64* %addr) {
+; NEON-LABEL: @st3_nx2i64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], <vscale x 2 x i64> [[V2:%.*]], i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx2i64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], <vscale x 2 x i64> [[V2:%.*]], <vscale x 2 x i1> [[PRED:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv2i64(<vscale x 2 x i64> %v0,
+  <vscale x 2 x i64> %v1,
+  <vscale x 2 x i64> %v2,
+  i64* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+define void @st3_nx2f64(<vscale x 2 x double> %v0, <vscale x 2 x double> %v1, <vscale x 2 x double> %v2, <vscale x 2 x i1> %pred, double* %addr) {
+; NEON-LABEL: @st3_nx2f64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved3.store.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], <vscale x 2 x double> [[V2:%.*]], double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st3_nx2f64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st3.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], <vscale x 2 x double> [[V2:%.*]], <vscale x 2 x i1> [[PRED:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved3.store.nxv2f64(<vscale x 2 x double> %v0,
+  <vscale x 2 x double> %v1,
+  <vscale x 2 x double> %v2,
+  double* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+;
+; ST4B
+;
+
+define void @st4_nx16i8(<vscale x 16 x i8> %v0, <vscale x 16 x i8> %v1, <vscale x 16 x i8> %v2, <vscale x 16 x i8> %v3, <vscale x 16 x i1> %pred, i8* %addr) {
+; NEON-LABEL: @st4_nx16i8(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], <vscale x 16 x i8> [[V2:%.*]], <vscale x 16 x i8> [[V3:%.*]], i8* [[ADDR:%.*]], i32 0, <vscale x 16 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx16i8(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv16i8(<vscale x 16 x i8> [[V0:%.*]], <vscale x 16 x i8> [[V1:%.*]], <vscale x 16 x i8> [[V2:%.*]], <vscale x 16 x i8> [[V3:%.*]], <vscale x 16 x i1> [[PRED:%.*]], i8* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv16i8(<vscale x 16 x i8> %v0,
+  <vscale x 16 x i8> %v1,
+  <vscale x 16 x i8> %v2,
+  <vscale x 16 x i8> %v3,
+  i8* %addr, i32 0,
+  <vscale x 16 x i1> %pred)
+  ret void
+}
+
+;
+; ST4H
+;
+
+define void @st4_nx8i16(<vscale x 8 x i16> %v0, <vscale x 8 x i16> %v1, <vscale x 8 x i16> %v2, <vscale x 8 x i16> %v3, <vscale x 8 x i1> %pred, i16* %addr) {
+; NEON-LABEL: @st4_nx8i16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], <vscale x 8 x i16> [[V2:%.*]], <vscale x 8 x i16> [[V3:%.*]], i16* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx8i16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv8i16(<vscale x 8 x i16> [[V0:%.*]], <vscale x 8 x i16> [[V1:%.*]], <vscale x 8 x i16> [[V2:%.*]], <vscale x 8 x i16> [[V3:%.*]], <vscale x 8 x i1> [[PRED:%.*]], i16* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv8i16(<vscale x 8 x i16> %v0,
+  <vscale x 8 x i16> %v1,
+  <vscale x 8 x i16> %v2,
+  <vscale x 8 x i16> %v3,
+  i16* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st4_nx8f16(<vscale x 8 x half> %v0, <vscale x 8 x half> %v1, <vscale x 8 x half> %v2, <vscale x 8 x half> %v3, <vscale x 8 x i1> %pred, half* %addr) {
+; NEON-LABEL: @st4_nx8f16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], <vscale x 8 x half> [[V2:%.*]], <vscale x 8 x half> [[V3:%.*]], half* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx8f16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv8f16(<vscale x 8 x half> [[V0:%.*]], <vscale x 8 x half> [[V1:%.*]], <vscale x 8 x half> [[V2:%.*]], <vscale x 8 x half> [[V3:%.*]], <vscale x 8 x i1> [[PRED:%.*]], half* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv8f16(<vscale x 8 x half> %v0,
+  <vscale x 8 x half> %v1,
+  <vscale x 8 x half> %v2,
+  <vscale x 8 x half> %v3,
+  half* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+define void @st4_nx8bf16(<vscale x 8 x bfloat> %v0, <vscale x 8 x bfloat> %v1, <vscale x 8 x bfloat> %v2, <vscale x 8 x bfloat> %v3, <vscale x 8 x i1> %pred, bfloat* %addr) #0 {
+; NEON-LABEL: @st4_nx8bf16(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], <vscale x 8 x bfloat> [[V2:%.*]], <vscale x 8 x bfloat> [[V3:%.*]], bfloat* [[ADDR:%.*]], i32 0, <vscale x 8 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx8bf16(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv8bf16(<vscale x 8 x bfloat> [[V0:%.*]], <vscale x 8 x bfloat> [[V1:%.*]], <vscale x 8 x bfloat> [[V2:%.*]], <vscale x 8 x bfloat> [[V3:%.*]], <vscale x 8 x i1> [[PRED:%.*]], bfloat* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv8bf16(<vscale x 8 x bfloat> %v0,
+  <vscale x 8 x bfloat> %v1,
+  <vscale x 8 x bfloat> %v2,
+  <vscale x 8 x bfloat> %v3,
+  bfloat* %addr, i32 0,
+  <vscale x 8 x i1> %pred)
+  ret void
+}
+
+;
+; ST4W
+;
+
+define void @st4_nx4i32(<vscale x 4 x i32> %v0, <vscale x 4 x i32> %v1, <vscale x 4 x i32> %v2, <vscale x 4 x i32> %v3, <vscale x 4 x i1> %pred, i32* %addr) {
+; NEON-LABEL: @st4_nx4i32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], <vscale x 4 x i32> [[V2:%.*]], <vscale x 4 x i32> [[V3:%.*]], i32* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx4i32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv4i32(<vscale x 4 x i32> [[V0:%.*]], <vscale x 4 x i32> [[V1:%.*]], <vscale x 4 x i32> [[V2:%.*]], <vscale x 4 x i32> [[V3:%.*]], <vscale x 4 x i1> [[PRED:%.*]], i32* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv4i32(<vscale x 4 x i32> %v0,
+  <vscale x 4 x i32> %v1,
+  <vscale x 4 x i32> %v2,
+  <vscale x 4 x i32> %v3,
+  i32* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+define void @st4_nx4f32(<vscale x 4 x float> %v0, <vscale x 4 x float> %v1, <vscale x 4 x float> %v2, <vscale x 4 x float> %v3, <vscale x 4 x i1> %pred, float* %addr) {
+; NEON-LABEL: @st4_nx4f32(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], <vscale x 4 x float> [[V2:%.*]], <vscale x 4 x float> [[V3:%.*]], float* [[ADDR:%.*]], i32 0, <vscale x 4 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx4f32(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv4f32(<vscale x 4 x float> [[V0:%.*]], <vscale x 4 x float> [[V1:%.*]], <vscale x 4 x float> [[V2:%.*]], <vscale x 4 x float> [[V3:%.*]], <vscale x 4 x i1> [[PRED:%.*]], float* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv4f32(<vscale x 4 x float> %v0,
+  <vscale x 4 x float> %v1,
+  <vscale x 4 x float> %v2,
+  <vscale x 4 x float> %v3,
+  float* %addr, i32 0,
+  <vscale x 4 x i1> %pred)
+  ret void
+}
+
+;
+; ST4D
+;
+
+define void @st4_nx2i64(<vscale x 2 x i64> %v0, <vscale x 2 x i64> %v1, <vscale x 2 x i64> %v2, <vscale x 2 x i64> %v3, <vscale x 2 x i1> %pred, i64* %addr) {
+; NEON-LABEL: @st4_nx2i64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], <vscale x 2 x i64> [[V2:%.*]], <vscale x 2 x i64> [[V3:%.*]], i64* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx2i64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv2i64(<vscale x 2 x i64> [[V0:%.*]], <vscale x 2 x i64> [[V1:%.*]], <vscale x 2 x i64> [[V2:%.*]], <vscale x 2 x i64> [[V3:%.*]], <vscale x 2 x i1> [[PRED:%.*]], i64* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv2i64(<vscale x 2 x i64> %v0,
+  <vscale x 2 x i64> %v1,
+  <vscale x 2 x i64> %v2,
+  <vscale x 2 x i64> %v3,
+  i64* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+define void @st4_nx2f64(<vscale x 2 x double> %v0, <vscale x 2 x double> %v1, <vscale x 2 x double> %v2, <vscale x 2 x double> %v3, <vscale x 2 x i1> %pred, double* %addr) {
+; NEON-LABEL: @st4_nx2f64(
+; NEON-NEXT:    call void @llvm.experimental.masked.interleaved4.store.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], <vscale x 2 x double> [[V2:%.*]], <vscale x 2 x double> [[V3:%.*]], double* [[ADDR:%.*]], i32 0, <vscale x 2 x i1> [[PRED:%.*]])
+; NEON-NEXT:    ret void
+;
+; SVE-LABEL: @st4_nx2f64(
+; SVE-NEXT:    call void @llvm.aarch64.sve.st4.nxv2f64(<vscale x 2 x double> [[V0:%.*]], <vscale x 2 x double> [[V1:%.*]], <vscale x 2 x double> [[V2:%.*]], <vscale x 2 x double> [[V3:%.*]], <vscale x 2 x i1> [[PRED:%.*]], double* [[ADDR:%.*]])
+; SVE-NEXT:    ret void
+;
+  call void @llvm.experimental.masked.interleaved4.store.nxv2f64(<vscale x 2 x double> %v0,
+  <vscale x 2 x double> %v1,
+  <vscale x 2 x double> %v2,
+  <vscale x 2 x double> %v3,
+  double* %addr, i32 0,
+  <vscale x 2 x i1> %pred)
+  ret void
+}
+
+attributes #0 = { "target-features"="+bf16" }
+
+declare void @llvm.experimental.masked.interleaved2.store.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>, i8*, i32, <vscale x 16 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>, i16*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>, i32*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>, i64*, i32, <vscale x 2 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>, half*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv8bf16(<vscale x 8 x bfloat>, <vscale x 8 x bfloat>, bfloat*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>, float*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved2.store.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>, double*, i32, <vscale x 2 x i1>)
+
+declare void @llvm.experimental.masked.interleaved3.store.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, i8*, i32, <vscale x 16 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, i16*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>,<vscale x 4 x i32>, i32*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>,<vscale x 2 x i64>, i64*, i32, <vscale x 2 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>,<vscale x 8 x half>, half*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv8bf16(<vscale x 8 x bfloat>, <vscale x 8 x bfloat>,<vscale x 8 x bfloat>, bfloat*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>,<vscale x 4 x float>, float*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved3.store.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>,<vscale x 2 x double>, double*, i32, <vscale x 2 x i1>)
+
+declare void @llvm.experimental.masked.interleaved4.store.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, <vscale x 16 x i8>, i8*, i32, <vscale x 16 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, <vscale x 8 x i16>, i16*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i32>,<vscale x 4 x i32>, <vscale x 4 x i32>, i32*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>,<vscale x 2 x i64>, <vscale x 2 x i64>, i64*, i32, <vscale x 2 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv8f16(<vscale x 8 x half>, <vscale x 8 x half>,<vscale x 8 x half>, <vscale x 8 x half>, half*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv8bf16(<vscale x 8 x bfloat>, <vscale x 8 x bfloat>,<vscale x 8 x bfloat>, <vscale x 8 x bfloat>, bfloat*, i32, <vscale x 8 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv4f32(<vscale x 4 x float>, <vscale x 4 x float>,<vscale x 4 x float>, <vscale x 4 x float>, float*, i32, <vscale x 4 x i1>)
+declare void @llvm.experimental.masked.interleaved4.store.nxv2f64(<vscale x 2 x double>, <vscale x 2 x double>,<vscale x 2 x double>, <vscale x 2 x double>, double*, i32, <vscale x 2 x i1>)

llvm/test/Transforms/LoopVectorize/AArch64/scalable-strict-fadd.ll

 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=scalar-epilogue \
-; RUN:   -force-ordered-reductions=false -hints-allow-reordering=false -S | FileCheck %s --check-prefix=CHECK-NOT-VECTORIZED
+; RUN:   -force-ordered-reductions=false -hints-allow-reordering=false -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-NOT-VECTORIZED
 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=scalar-epilogue \
-; RUN:   -force-ordered-reductions=false -hints-allow-reordering=true  -S | FileCheck %s --check-prefix=CHECK-UNORDERED
+; RUN:   -force-ordered-reductions=false -hints-allow-reordering=true -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-UNORDERED
 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=scalar-epilogue \
-; RUN:   -force-ordered-reductions=true  -hints-allow-reordering=false -S | FileCheck %s --check-prefix=CHECK-ORDERED
+; RUN:   -force-ordered-reductions=true  -hints-allow-reordering=false -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-ORDERED
 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=scalar-epilogue \
-; RUN:   -force-ordered-reductions=true  -hints-allow-reordering=true  -S | FileCheck %s --check-prefix=CHECK-UNORDERED
+; RUN:   -force-ordered-reductions=true  -hints-allow-reordering=true -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-UNORDERED
 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=scalar-epilogue \
-; RUN:   -hints-allow-reordering=false -S | FileCheck %s --check-prefix=CHECK-ORDERED
+; RUN:   -hints-allow-reordering=false -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-ORDERED
 ; RUN: opt < %s -loop-vectorize -mtriple aarch64-unknown-linux-gnu -mattr=+sve -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue \
-; RUN:   -hints-allow-reordering=false -S | FileCheck %s --check-prefix=CHECK-ORDERED-TF
+; RUN:   -hints-allow-reordering=false -enable-sve-interleaved-mem-accesses=false -S | FileCheck %s --check-prefix=CHECK-ORDERED-TF
 
 define float @fadd_strict(float* noalias nocapture readonly %a, i64 %n) #0 {
 ; CHECK-ORDERED-LABEL: @fadd_strict
 ; CHECK-ORDERED: vector.body:
 ; CHECK-ORDERED: %[[VEC_PHI:.*]] = phi float [ 0.000000e+00, %vector.ph ], [ %[[RDX:.*]], %vector.body ]
 ; CHECK-ORDERED: %[[LOAD:.*]] = load <vscale x 8 x float>, <vscale x 8 x float>*
 ; CHECK-ORDERED: %[[RDX]] = call float @llvm.vector.reduce.fadd.nxv8f32(float %[[VEC_PHI]], <vscale x 8 x float> %[[LOAD]])
 ; CHECK-ORDERED: for.end

llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll

 
 for.end:                                          ; preds = %for.body
   ret void
 }
 
 define void @interleave(float* noalias %dst, float* noalias %src, i64 %n) #0 {
 ; CHECK-NOTF-LABEL: @interleave(
 ; CHECK-NOTF:       vector.body:
-; CHECK-NOTF:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
-; CHECK-NOTF:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
-; CHECK-NOTF:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-NOTF:         %[[LOAD:.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(
 
 ; CHECK-TF-LABEL: @interleave(
 ; CHECK-TF:       vector.body:
-; CHECK-TF:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
-; CHECK-TF:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
-; CHECK-TF:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-TF:         %[[LOAD:.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(
 
 ; CHECK-TF-NORED-LABEL: @interleave(
 ; CHECK-TF-NORED:       vector.body:
-; CHECK-TF-NORED:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
-; CHECK-TF-NORED:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
-; CHECK-TF-NORED:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-TF-NORED:         %[[LOAD:.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(
 
 ; CHECK-TF-NOREC-LABEL: @interleave(
 ; CHECK-TF-NOREC:       vector.body:
-; CHECK-TF-NOREC:         %[[LOAD:.*]] = load <8 x float>, <8 x float>
-; CHECK-TF-NOREC:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
-; CHECK-TF-NOREC:         %{{.*}} = shufflevector <8 x float> %[[LOAD]], <8 x float> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7>
+; CHECK-TF-NOREC:         %[[LOAD:.*]] = call { <vscale x 4 x float>, <vscale x 4 x float> } @llvm.experimental.masked.interleaved2.load.nxv4f32(
 
 entry:
   br label %for.body
 
 for.body:                                         ; preds = %entry, %for.body
   %i.021 = phi i64 [ %inc, %for.body ], [ 0, %entry ]
   %mul = shl nuw nsw i64 %i.021, 1
   %arrayidx = getelementptr inbounds float, float* %src, i64 %mul

llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -mtriple aarch64-linux-gnu -mattr=+sve -loop-vectorize -dce -instcombine -S \
-; RUN:   -prefer-predicate-over-epilogue=scalar-epilogue < %s | FileCheck %s
+; RUN:   -prefer-predicate-over-epilogue=scalar-epilogue -enable-sve-interleaved-mem-accesses=false < %s | FileCheck %s
 
 ; Ensure that we can vectorize loops such as:
 ;   int *ptr = c;
 ;   for (long long i = 0; i < n; i++) {
 ;     int X1 = *ptr++;
 ;     int X2 = *ptr++;
 ;     a[i] = X1 + 1;
 ;     b[i] = X2 + 1;

llvm/test/Transforms/LoopVectorize/sve-interleaved-accesses-ic-2.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-none-linux-gnu -S -loop-vectorize -instcombine -force-vector-width=4 -force-vector-interleave=2 -enable-interleaved-mem-accesses=true -enable-sve-interleaved-mem-accesses=true -mattr=+sve -scalable-vectorization=on -runtime-memory-check-threshold=24 < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+
+; Check vectorization on an interleaved load group of factor 2 and an interleaved
+; store group of factor 2.
+
+; int AB[1024];
+; int CD[1024];
+;  void test_array_load2_store2(int C, int D) {
+;   for (int i = 0; i < 1024; i+=2) {
+;     int A = AB[i];
+;     int B = AB[i+1];
+;     CD[i] = A + C;
+;     CD[i+1] = B * D;
+;   }
+; }
+
+
+@AB = common global [1024 x i32] zeroinitializer, align 4
+@CD = common global [1024 x i32] zeroinitializer, align 4
+
+define void @test_array_load2_store2(i32 %C, i32 %D) #1 {
+; CHECK-LABEL: @test_array_load2_store2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 3
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i64 512, [[TMP1]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub nuw nsw i64 512, [[N_MOD_VF]]
+; CHECK-NEXT:    [[IND_END:%.*]] = shl nuw nsw i64 [[N_VEC]], 1
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[C:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT1:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[C]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT2:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT1]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT3:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[D:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT4:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT3]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT5:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[D]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT6:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT5]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX]], 1
+; CHECK-NEXT:    [[TMP2:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP3:%.*]] = shl nuw nsw i64 [[TMP2]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = add i64 [[OFFSET_IDX]], [[TMP3]]
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[TMP4]]
+; CHECK-NEXT:    [[TMP7:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32* nonnull [[TMP5]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer), <vscale x 4 x i32> undef)
+; CHECK-NEXT:    [[TMP8:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32* nonnull [[TMP6]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer), <vscale x 4 x i32> undef)
+; CHECK-NEXT:    [[TMP9:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP7]], 0
+; CHECK-NEXT:    [[TMP10:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP8]], 0
+; CHECK-NEXT:    [[TMP11:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP7]], 1
+; CHECK-NEXT:    [[TMP12:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP8]], 1
+; CHECK-NEXT:    [[TMP13:%.*]] = or i64 [[OFFSET_IDX]], 1
+; CHECK-NEXT:    [[TMP14:%.*]] = or i64 [[TMP4]], 1
+; CHECK-NEXT:    [[TMP15:%.*]] = add nsw <vscale x 4 x i32> [[TMP9]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP16:%.*]] = add nsw <vscale x 4 x i32> [[TMP10]], [[BROADCAST_SPLAT2]]
+; CHECK-NEXT:    [[TMP17:%.*]] = mul nsw <vscale x 4 x i32> [[TMP11]], [[BROADCAST_SPLAT4]]
+; CHECK-NEXT:    [[TMP18:%.*]] = mul nsw <vscale x 4 x i32> [[TMP12]], [[BROADCAST_SPLAT6]]
+; CHECK-NEXT:    [[TMP19:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[TMP13]]
+; CHECK-NEXT:    [[TMP20:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[TMP14]]
+; CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 -1
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds i32, i32* [[TMP20]], i64 -1
+; CHECK-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32> [[TMP15]], <vscale x 4 x i32> [[TMP17]], i32* nonnull [[TMP21]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer))
+; CHECK-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32> [[TMP16]], <vscale x 4 x i32> [[TMP18]], i32* nonnull [[TMP22]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer))
+; CHECK-NEXT:    [[TMP23:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP24:%.*]] = shl nuw nsw i64 [[TMP23]], 3
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP24]]
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP25]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[ARRAYIDX0:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[TMP:%.*]] = load i32, i32* [[ARRAYIDX0]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = or i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[ARRAYIDX1]], align 4
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP]], [[C]]
+; CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[TMP2]], [[D]]
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[INDVARS_IV]]
+; CHECK-NEXT:    store i32 [[ADD]], i32* [[ARRAYIDX2]], align 4
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[TMP1]]
+; CHECK-NEXT:    store i32 [[MUL]], i32* [[ARRAYIDX3]], align 4
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 2
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i64 [[INDVARS_IV]], 1022
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %arrayidx0 = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 %indvars.iv
+  %tmp = load i32, i32* %arrayidx0, align 4
+  %tmp1 = or i64 %indvars.iv, 1
+  %arrayidx1 = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 %tmp1
+  %tmp2 = load i32, i32* %arrayidx1, align 4
+  %add = add nsw i32 %tmp, %C
+  %mul = mul nsw i32 %tmp2, %D
+  %arrayidx2 = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 %indvars.iv
+  store i32 %add, i32* %arrayidx2, align 4
+  %arrayidx3 = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 %tmp1
+  store i32 %mul, i32* %arrayidx3, align 4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 2
+  %cmp = icmp slt i64 %indvars.iv.next, 1024
+  br i1 %cmp, label %for.body, label %for.end
+
+for.end:                                          ; preds = %for.body
+  ret void
+}
+
+
+attributes #1 = { "target-features"="+sve" vscale_range(1, 16) }

llvm/test/Transforms/LoopVectorize/sve-interleaved-accesses.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-none-linux-gnu -S -loop-vectorize -instcombine -force-vector-width=4 -force-vector-interleave=1 -enable-interleaved-mem-accesses=true -enable-sve-interleaved-mem-accesses=true -mattr=+sve -scalable-vectorization=on -runtime-memory-check-threshold=24 < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+
+; Check vectorization on an interleaved load group of factor 2 and an interleaved
+; store group of factor 2.
+
+; int AB[1024];
+; int CD[1024];
+;  void test_array_load2_store2(int C, int D) {
+;   for (int i = 0; i < 1024; i+=2) {
+;     int A = AB[i];
+;     int B = AB[i+1];
+;     CD[i] = A + C;
+;     CD[i+1] = B * D;
+;   }
+; }
+
+
+@AB = common global [1024 x i32] zeroinitializer, align 4
+@CD = common global [1024 x i32] zeroinitializer, align 4
+
+define void @test_array_load2_store2(i32 %C, i32 %D) #1 {
+; CHECK-LABEL: @test_array_load2_store2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 2
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i64 512, [[TMP1]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub nuw nsw i64 512, [[N_MOD_VF]]
+; CHECK-NEXT:    [[IND_END:%.*]] = shl nuw nsw i64 [[N_VEC]], 1
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[C:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT1:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[D:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT2:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT1]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
+; CHECK:       vector.body:
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = shl i64 [[INDEX]], 1
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call { <vscale x 4 x i32>, <vscale x 4 x i32> } @llvm.experimental.masked.interleaved2.load.nxv4i32(i32* nonnull [[TMP2]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer), <vscale x 4 x i32> undef)
+; CHECK-NEXT:    [[TMP4:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP3]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = extractvalue { <vscale x 4 x i32>, <vscale x 4 x i32> } [[TMP3]], 1
+; CHECK-NEXT:    [[TMP6:%.*]] = or i64 [[OFFSET_IDX]], 1
+; CHECK-NEXT:    [[TMP7:%.*]] = add nsw <vscale x 4 x i32> [[TMP4]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP8:%.*]] = mul nsw <vscale x 4 x i32> [[TMP5]], [[BROADCAST_SPLAT2]]
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[TMP6]]
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i32, i32* [[TMP9]], i64 -1
+; CHECK-NEXT:    call void @llvm.experimental.masked.interleaved2.store.nxv4i32(<vscale x 4 x i32> [[TMP7]], <vscale x 4 x i32> [[TMP8]], i32* nonnull [[TMP10]], i32 4, <vscale x 4 x i1> shufflevector (<vscale x 4 x i1> insertelement (<vscale x 4 x i1> poison, i1 true, i32 0), <vscale x 4 x i1> poison, <vscale x 4 x i32> zeroinitializer))
+; CHECK-NEXT:    [[TMP11:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP12:%.*]] = shl nuw nsw i64 [[TMP11]], 2
+; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], [[TMP12]]
+; CHECK-NEXT:    [[TMP13:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP13]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK:       middle.block:
+; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK:       scalar.ph:
+; CHECK-NEXT:    [[BC_RESUME_VAL:%.*]] = phi i64 [ [[IND_END]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[ARRAYIDX0:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[INDVARS_IV]]
+; CHECK-NEXT:    [[TMP:%.*]] = load i32, i32* [[ARRAYIDX0]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = or i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 [[TMP1]]
+; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[ARRAYIDX1]], align 4
+; CHECK-NEXT:    [[ADD:%.*]] = add nsw i32 [[TMP]], [[C]]
+; CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[TMP2]], [[D]]
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[INDVARS_IV]]
+; CHECK-NEXT:    store i32 [[ADD]], i32* [[ARRAYIDX2]], align 4
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 [[TMP1]]
+; CHECK-NEXT:    store i32 [[MUL]], i32* [[ARRAYIDX3]], align 4
+; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add nuw nsw i64 [[INDVARS_IV]], 2
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i64 [[INDVARS_IV]], 1022
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[FOR_END]], !llvm.loop [[LOOP2:![0-9]+]]
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %arrayidx0 = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 %indvars.iv
+  %tmp = load i32, i32* %arrayidx0, align 4
+  %tmp1 = or i64 %indvars.iv, 1
+  %arrayidx1 = getelementptr inbounds [1024 x i32], [1024 x i32]* @AB, i64 0, i64 %tmp1
+  %tmp2 = load i32, i32* %arrayidx1, align 4
+  %add = add nsw i32 %tmp, %C
+  %mul = mul nsw i32 %tmp2, %D
+  %arrayidx2 = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 %indvars.iv
+  store i32 %add, i32* %arrayidx2, align 4
+  %arrayidx3 = getelementptr inbounds [1024 x i32], [1024 x i32]* @CD, i64 0, i64 %tmp1
+  store i32 %mul, i32* %arrayidx3, align 4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 2
+  %cmp = icmp slt i64 %indvars.iv.next, 1024
+  br i1 %cmp, label %for.body, label %for.end
+
+for.end:                                          ; preds = %for.body
+  ret void
+}
+
+; Check vectorization on an interleaved load group of factor 2 with narrower types  and an interleaved
+; store group of factor 2.
+
+; short AB[1024];
+; int CD[1024];
+;  void test_array_load2_store2(int C, int D) {
+;   for (int i = 0; i < 1024; i+=2) {
+;     short A = AB[i];
+;     short B = AB[i+1];
+;     CD[i] = A + C;
+;     CD[i+1] = B * D;
+;   }
+; }
+
+
+@AB_i16 = common global [1024 x i16] zeroinitializer, align 4
+
+define void @test_array_load2_i16_store2(i32 %C, i32 %D) #1 {
+; CHECK-LABEL: @test_array_load2_i16_store2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK:       vector.ph:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP1:%.*]] = shl nuw nsw i64 [[TMP0]], 2
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = urem i64 512, [[TMP1]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub nuw nsw i64 512, [[N_MOD_VF]]
+; CHECK-NEXT:    [[IND_END:%.*]] = shl nuw nsw i64 [[N_VEC]], 1
+; CHECK-NEXT:    [[TMP2:%.*]] = call <vscale x 4 x i64> @llvm.experimental.stepvector.nxv4i64()
+; CHECK-NEXT:    [[TMP3:%.*]] = shl <vscale x 4 x i64> [[TMP2]], shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> poison, i64 1, i32 0), <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer)
+; CHECK-NEXT:    [[TMP4:%.*]] = call i64 @llvm.vscale.i64()
+; CHECK-NEXT:    [[TMP5:%.*]] = shl nuw nsw i64 [[TMP4]], 3
+; CHECK-NEXT:    [[DOTSPLATINSERT:%.*]] = insertelement <vscale x 4 x i64> poison, i64 [[TMP5]], i64 0
+; CHECK-NEXT:    [[DOTSPLAT:%.*]] = shufflevector <vscale x 4 x i64> [[DOTSPLATINSERT]], <vscale x 4 x i64> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[C:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT2:%.*]] = insertelement <vscale x 4 x i32> poison, i32 [[D:%.*]], i64 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT3:%.*]] = shufflevector <vscale x 4 x i32> [[BROADCAST_SPLATINSERT2]], <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer