[AArch64-SVE]: Force generating code compatible to streaming mode.

[hassnaa-arm]

Summary

When streaming mode is enabled, lower some operations and disable some code paths to force generating code compatible to streaming mode.
Add testing files for shifts, build_vector, concat, extract_subvector, extract_vector_elt, and shuffle.

[hassnaa-arm]

Add additional test cases

[hassnaa-arm]

get latest changes of parent revision

[hassnaa-arm]

update by rebasing parent revision

[hassnaa-arm]

Remove unrelated changes

[hassnaa-arm]

Restore some changes removed by mistake

[hassnaa-arm]

Update by parent branch

[hassnaa-arm]

Update by parent patch

[SjoerdMeijer]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

12099

Drive by comment:

we don't need to pass Subtarget->forceStreamingCompatibleSVE() but can just query that inside useSVEForFixedLengthVectorVT?

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

12099

I think we still need to pass it, because useSVEForFixedLengthVectorVT is used in many places, some of them don't need the flag to be enabled.
The flag is used for enabling lowering specific nodes that cause generating invalid code in streaming mode.

[hassnaa-arm]

Update by changes of parent patch.
While lowering ISD::load remove enabling LowerFixedLengthVectorLoadToSVE,
no need for it, as zero_Extend is custom-lowered.
Previously, LowerLOAD() creates zero_Extend node, which cause invalid generated code,
but now, the zero_extend node is custom-lowered, which cause valid generated code.

[hassnaa-arm]

Update by parent patch

[sdesmalen]

Hi @hassnaa-arm, I think you have the tests the wrong way around. The tests from D136147 should be part of this patch, because this is the patch where you're implementing the lowering of the operations you're testing in D136147.
After this patch, you get the masked/truncating/extending load/store operations "for free", so the tests for those operations could be moved to a separate test-only patch like D136147.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1394

It should be able to use standard scalar instructions for v1i64 in streaming-compatible mode, so this one can be removed from the list.

1397

Most scalar FP operations are valid in streaming mode, so we probably don't need to do anything custom for this type.

1608–1610

nit: Perhaps it doesn't lead to an error, but these operations only operate on integers, so should be guarded by:

if (VT.isInteger()) { ... }

12270

nit: rather than wrapping this in another condition, can you just add it to the existing condition with && !Subtarget->forceStreamingCompatibleSVE() ?

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3049–3050

nit: Can you change this into:

let Predicates = [NotInStreamingSVEMode], AddedComplexity = 1 in {
def : Pat<...>
..
}

let Predicates = [NotInStreamingSVEMode] in {
def : Pat<..>
...
}

Rather than indenting?

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-store.ll

313

Can you remove all tests that are larger than "twice the size" of a 128bit vector (v32f32 is 8x the size, I'm not sure what value that adds for the testing of this functionality)

[hassnaa-arm]

Remove all tests that are larger than "twice the size" of a 128bit vector.

[hassnaa-arm]

Remove masked/truncating/extending load/store, to be added in a test-only patch.

[hassnaa-arm]

Lower And operation, and disable replacing 'and' by 'bic' while combining step.

[hassnaa-arm]

Update by parent patch

[hassnaa-arm]

Revert changes added by mistake

[hassnaa-arm]

Add testing files for shifts, build_vector, concat, extract_subvector, extract_vector_elt, and shuffle.

[david-arm]

Hi @hassnaa-arm, I think it's almost there! Most of the tests look good to me. I just had a few minor comments ...

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11372

I don't think you need to pass in the Subtarget here. In the code below you can just do

if (VT.isFixedLengthVector() &&
    DAG.getSubtarget<AArch64Subtarget>().forceStreamingCompatibleSVE()
  return SDValue();

11418

Same comment as above for tryAdvSIMDModImm32

13986

nit: The comment can probably be formatted better I think so that you use up 80 chars, i.e.:

// Skip if streaming compatible SVE is enabled, because it generates invalid
// code in streaming mode when SVE length is not specified.

15719

Again, you can avoid passing in the subtarget here if you make the changes to tryAdvSIMDModImm32 and tryAdvSIMDModImm16.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3044

When we guard something by a predicate we normally add a comment on the final brace '}' to make it easy to see, i.e. something like:

} // End NotInStreamingSVEMode

3058

} // End NotInStreamingSVEMode

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-concat.ll

10 ↗

(On Diff #470094)

I don't think we need to have vscale_range(2,0) on these tests, right? We want streaming SVE to work for vector lengths.

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-extract-vector-elt.ll

17 ↗

(On Diff #470094)

nit: I think in all these tests there should only be 2 spaces at the start of the IR, i.e.

%r = extractelement <2 x half> %op1, i64 1

etc.

[hassnaa-arm]

Remove vscale_range from concat.ll test file.

[hassnaa-arm]

Fix identation

[david-arm]

LGTM! Thanks for making the changes @hassnaa-arm.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

22379–22380

As with the above change can this be V.getValueType().isFixedLengthVector() && isTypeLegal(V.getValueType()) &&?

llvm/test/CodeGen/AArch64/sve-streaming-fixed-length-int-shifts.ll

1–5 ↗

(On Diff #471171)

Please rename this file sve-streaming-mode-fixed-length-int-shifts.ll to match the same format as the others.

[hassnaa-arm]

Rename sve-streaming-fixed-length-int-shifts.ll to sve-streaming-mode-fixed-length-int-shifts.ll

[sdesmalen]

LGTM as well (please address nit before submitting)

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

22379

nit: Does this cross the 80-character limit? (please use clang-format to be sure)

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1614

nit: I only just spot this now in one of your other patches, but ISD::AND should also be guarded by VT.isInteger().

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1614

Although not wrong it doesn't really matter as legalisation is smart enough to not care about the operation action for types that make no sense. We rely on this in addTypeForFixedLengthSVE where the type is only considered when handling extend-loads/truncating-store plus the odd compare.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1614

In that case it's probably better to remove the condition entirely.

llvm/lib/Target/AArch64/AArch64ISelLowering.h

 private:
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   void addTypeForNEON(MVT VT);
+  void addTypeForStreamingSVE(MVT VT);
   void addTypeForFixedLengthSVE(MVT VT);
   void addDRTypeForNEON(MVT VT);
   void addQRTypeForNEON(MVT VT);
 
   unsigned allocateLazySaveBuffer(SDValue &Chain, const SDLoc &DL,
                                   SelectionDAG &DAG, Register &Reg) const;
 
   SDValue LowerFormalArguments(SDValue Chain, CallingConv::ID CallConv,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->hasSVE()) {
     // NEON doesn't support 64-bit vector integer muls, but SVE does.
     setOperationAction(ISD::MUL, MVT::v1i64, Custom);
     setOperationAction(ISD::MUL, MVT::v2i64, Custom);
 
     // NEON doesn't support across-vector reductions, but SVE does.
     for (auto VT : {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v2f64})
       setOperationAction(ISD::VECREDUCE_SEQ_FADD, VT, Custom);
 
+    if (Subtarget->forceStreamingCompatibleSVE()) {
+      for (MVT VT : {MVT::v8i8, MVT::v16i8, MVT::v4i16, MVT::v8i16, MVT::v2i32,
+                     MVT::v4i32, MVT::v2i64})
+        addTypeForStreamingSVE(VT);
+
+      for (MVT VT : {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v2f64})
+        addTypeForStreamingSVE(VT);
+    }
+
     // NOTE: Currently this has to happen after computeRegisterProperties rather
     // than the preferred option of combining it with the addRegisterClass call.
     if (Subtarget->useSVEForFixedLengthVectors()) {
       for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
           addTypeForFixedLengthSVE(VT);
       for (MVT VT : MVT::fp_fixedlen_vector_valuetypes())
         if (useSVEForFixedLengthVectorVT(VT))
bool AArch64TargetLowering::shouldExpandGetActiveLaneMask(EVT ResVT,
 
   // The whilelo instruction only works with i32 or i64 scalar inputs.
   if (OpVT != MVT::i32 && OpVT != MVT::i64)
     return true;
 
   return false;
 }
 
+void AArch64TargetLowering::addTypeForStreamingSVE(MVT VT) {
+  if(VT.isInteger()) {
+    setOperationAction(ISD::ANY_EXTEND, VT, Custom);
+    setOperationAction(ISD::ZERO_EXTEND, VT, Custom);
+    setOperationAction(ISD::SIGN_EXTEND, VT, Custom);
+  }
+  setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
+}
+
 void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   assert(VT.isFixedLengthVector() && "Expected fixed length vector type!");
 
   // By default everything must be expanded.
   for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op)
     setOperationAction(Op, VT, Expand);
 
   // We use EXTRACT_SUBVECTOR to "cast" a scalable vector to a fixed length one.
case ISD::SIGN_EXTEND_INREG: {
     return LowerToPredicatedOp(Op, DAG,
                                AArch64ISD::SIGN_EXTEND_INREG_MERGE_PASSTHRU);
   }
   case ISD::TRUNCATE:
     return LowerTRUNCATE(Op, DAG);
   case ISD::MLOAD:
     return LowerMLOAD(Op, DAG);
   case ISD::LOAD:
-    if (useSVEForFixedLengthVectorVT(Op.getValueType(),
-                                     Subtarget->forceStreamingCompatibleSVE()))
+    if (useSVEForFixedLengthVectorVT(Op.getValueType()))
       return LowerFixedLengthVectorLoadToSVE(Op, DAG);
     return LowerLOAD(Op, DAG);
   case ISD::ADD:
   case ISD::AND:
   case ISD::SUB:
     return LowerToScalableOp(Op, DAG);
   case ISD::FMAXIMUM:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::FMAX_PRED);

 }
 
 // Try 32-bit splatted SIMD immediate.
 static SDValue tryAdvSIMDModImm32(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
                                   const APInt &Bits,
                                   const SDValue *LHS = nullptr) {
   if (Bits.getHiBits(64) == Bits.getLoBits(64)) {
     uint64_t Value = Bits.zextOrTrunc(64).getZExtValue();
     EVT VT = Op.getValueType();
     MVT MovTy = (VT.getSizeInBits() == 128) ? MVT::v4i32 : MVT::v2i32;
     bool isAdvSIMDModImm = false;
     uint64_t Shift;
 
     if ((isAdvSIMDModImm = AArch64_AM::isAdvSIMDModImmType1(Value))) {
       Value = AArch64_AM::encodeAdvSIMDModImmType1(Value);
       Shift = 0;
     }
static SDValue tryAdvSIMDModImm32(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
 
   return SDValue();
 }
 
 // Try 16-bit splatted SIMD immediate.
 static SDValue tryAdvSIMDModImm16(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
                                   const APInt &Bits,
                                   const SDValue *LHS = nullptr) {
   if (Bits.getHiBits(64) == Bits.getLoBits(64)) {
     uint64_t Value = Bits.zextOrTrunc(64).getZExtValue();
     EVT VT = Op.getValueType();
     MVT MovTy = (VT.getSizeInBits() == 128) ? MVT::v8i16 : MVT::v4i16;
     bool isAdvSIMDModImm = false;
     uint64_t Shift;
 
     if ((isAdvSIMDModImm = AArch64_AM::isAdvSIMDModImmType5(Value))) {
       Value = AArch64_AM::encodeAdvSIMDModImmType5(Value);
SDValue AArch64TargetLowering::LowerBUILD_VECTOR(SDValue Op,
   LLVM_DEBUG(
       dbgs() << "LowerBUILD_VECTOR: use default expansion, failed to find "
                 "better alternative\n");
   return SDValue();
 }
 
 SDValue AArch64TargetLowering::LowerCONCAT_VECTORS(SDValue Op,
                                                    SelectionDAG &DAG) const {
-  if (useSVEForFixedLengthVectorVT(Op.getValueType()))
+  if (useSVEForFixedLengthVectorVT(Op.getValueType(),
+                                   Subtarget->forceStreamingCompatibleSVE()))
     return LowerFixedLengthConcatVectorsToSVE(Op, DAG);
 
   assert(Op.getValueType().isScalableVector() &&
          isTypeLegal(Op.getValueType()) &&
          "Expected legal scalable vector type!");
 
   if (isTypeLegal(Op.getOperand(0).getValueType())) {
     unsigned NumOperands = Op->getNumOperands();
if (VT.getScalarType() == MVT::i1) {
     SDValue Extend =
         DAG.getNode(ISD::ANY_EXTEND, DL, VectorVT, Op.getOperand(0));
     MVT ExtractTy = VectorVT == MVT::nxv2i64 ? MVT::i64 : MVT::i32;
     SDValue Extract = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, ExtractTy,
                                   Extend, Op.getOperand(1));
     return DAG.getAnyExtOrTrunc(Extract, DL, Op.getValueType());
   }
 
-  if (useSVEForFixedLengthVectorVT(VT))
+  if (useSVEForFixedLengthVectorVT(VT,
+                                   Subtarget->forceStreamingCompatibleSVE()))
     return LowerFixedLengthExtractVectorElt(Op, DAG);
 
   // Check for non-constant or out of range lane.
   ConstantSDNode *CI = dyn_cast<ConstantSDNode>(Op.getOperand(1));
   if (!CI || CI->getZExtValue() >= VT.getVectorNumElements())
     return SDValue();
 
   // Insertion/extraction are legal for V128 types.
SDValue AArch64TargetLowering::LowerEXTRACT_SUBVECTOR(SDValue Op,
 
   // This will get lowered to an appropriate EXTRACT_SUBREG in ISel.
   if (Idx == 0 && InVT.getSizeInBits() <= 128)
     return Op;
 
   // If this is extracting the upper 64-bits of a 128-bit vector, we match
   // that directly.
   if (Size == 64 && Idx * InVT.getScalarSizeInBits() == 64 &&
-      InVT.getSizeInBits() == 128)
+      InVT.getSizeInBits() == 128 &&
+      !Subtarget->forceStreamingCompatibleSVE())
     return Op;
 
-  if (useSVEForFixedLengthVectorVT(InVT)) {
+  if (useSVEForFixedLengthVectorVT(InVT,
+                                   Subtarget->forceStreamingCompatibleSVE())) {
     SDLoc DL(Op);
 
     EVT ContainerVT = getContainerForFixedLengthVector(DAG, InVT);
     SDValue NewInVec =
         convertToScalableVector(DAG, ContainerVT, Op.getOperand(0));
 
     SDValue Splice = DAG.getNode(ISD::VECTOR_SPLICE, DL, ContainerVT, NewInVec,
                                  NewInVec, DAG.getConstant(Idx, DL, MVT::i64));
SDValue AArch64TargetLowering::LowerVectorSRA_SRL_SHL(SDValue Op,
   int64_t Cnt;
 
   if (!Op.getOperand(1).getValueType().isVector())
     return Op;
   unsigned EltSize = VT.getScalarSizeInBits();
 
   switch (Op.getOpcode()) {
   case ISD::SHL:
-    if (VT.isScalableVector() || useSVEForFixedLengthVectorVT(VT))
+    if (VT.isScalableVector() ||
+        useSVEForFixedLengthVectorVT(VT,
+                                     Subtarget->forceStreamingCompatibleSVE()))
       return LowerToPredicatedOp(Op, DAG, AArch64ISD::SHL_PRED);
 
     if (isVShiftLImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize)
       return DAG.getNode(AArch64ISD::VSHL, DL, VT, Op.getOperand(0),
                          DAG.getConstant(Cnt, DL, MVT::i32));
     return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
                        DAG.getConstant(Intrinsic::aarch64_neon_ushl, DL,
                                        MVT::i32),
                        Op.getOperand(0), Op.getOperand(1));
   case ISD::SRA:
   case ISD::SRL:
-    if (VT.isScalableVector() || useSVEForFixedLengthVectorVT(VT)) {
+    if (VT.isScalableVector() ||
+        useSVEForFixedLengthVectorVT(
+            VT, Subtarget->forceStreamingCompatibleSVE())) {
       unsigned Opc = Op.getOpcode() == ISD::SRA ? AArch64ISD::SRA_PRED
                                                 : AArch64ISD::SRL_PRED;
       return LowerToPredicatedOp(Op, DAG, Opc);
     }
 
     // Right shift immediate
     if (isVShiftRImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize) {
       unsigned Opc =

 ///      Into:
 ///        %sub.v0 = shuffle <32 x i32> %v0, <32 x i32> v1, <4, 5, 6, 7>
 ///        %sub.v1 = shuffle <32 x i32> %v0, <32 x i32> v1, <32, 33, 34, 35>
 ///        %sub.v2 = shuffle <32 x i32> %v0, <32 x i32> v1, <16, 17, 18, 19>
 ///        call void llvm.aarch64.neon.st3(%sub.v0, %sub.v1, %sub.v2, %ptr)
 bool AArch64TargetLowering::lowerInterleavedStore(StoreInst *SI,
                                                   ShuffleVectorInst *SVI,
                                                   unsigned Factor) const {
   assert(Factor >= 2 && Factor <= getMaxSupportedInterleaveFactor() &&
          "Invalid interleave factor");
 
   auto *VecTy = cast<FixedVectorType>(SVI->getType());
   assert(VecTy->getNumElements() % Factor == 0 && "Invalid interleaved store");
 
   unsigned LaneLen = VecTy->getNumElements() / Factor;
   Type *EltTy = VecTy->getElementType();
   auto *SubVecTy = FixedVectorType::get(EltTy, LaneLen);
static SDValue performSVEAndCombine(SDNode *N,
   if (isConstantSplatVectorMaskForType(Mask.getNode(), MemVT))
     return Src;
 
   return SDValue();
 }
 
 static SDValue performANDCombine(SDNode *N,
                                  TargetLowering::DAGCombinerInfo &DCI) {
   SelectionDAG &DAG = DCI.DAG;
   SDValue LHS = N->getOperand(0);
   SDValue RHS = N->getOperand(1);
   EVT VT = N->getValueType(0);
 
   if (SDValue R = performANDORCSELCombine(N, DAG))
     return R;
 
   if (!DAG.getTargetLoweringInfo().isTypeLegal(VT))
for (const SDValue &V : Op->op_values()) {
 
     // Pass through non-vector operands.
     if (!V.getValueType().isVector()) {
       Ops.push_back(V);
       continue;
     }
 
     // "cast" fixed length vector to a scalable vector.
     assert(useSVEForFixedLengthVectorVT(V.getValueType()) &&
            "Only fixed length vectors are supported!");
     Ops.push_back(convertToScalableVector(DAG, ContainerVT, V));
   }
 
   auto ScalableRes = DAG.getNode(Op.getOpcode(), SDLoc(Op), ContainerVT, Ops);
   return convertFromScalableVector(DAG, VT, ScalableRes);
 }
 
 SDValue AArch64TargetLowering::LowerVECREDUCE_SEQ_FADD(SDValue ScalarOp,

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVEorSME] in {
   def : Pat<(f32 (vector_extract (nxv4f32 ZPR:$vec), sve_elm_idx_extdup_s:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_S ZPR:$vec, sve_elm_idx_extdup_s:$index), ssub)>;
   def : Pat<(f32 (vector_extract (nxv2f32 ZPR:$vec), sve_elm_idx_extdup_d:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_D ZPR:$vec, sve_elm_idx_extdup_d:$index), ssub)>;
   def : Pat<(f64 (vector_extract (nxv2f64 ZPR:$vec), sve_elm_idx_extdup_d:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_D ZPR:$vec, sve_elm_idx_extdup_d:$index), dsub)>;
 
   // Extract element from vector with immediate index that's within the bottom 128-bits.
-  let AddedComplexity = 1 in {
+  let Predicates = [NotInStreamingSVEMode], AddedComplexity = 1 in {
   def : Pat<(i32 (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index)),
             (i32 (UMOVvi8 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
   def : Pat<(i32 (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index)),
             (i32 (UMOVvi16 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
   def : Pat<(i32 (vector_extract (nxv4i32 ZPR:$vec), VectorIndexS:$index)),
             (i32 (UMOVvi32 (v4i32 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexS:$index))>;
   def : Pat<(i64 (vector_extract (nxv2i64 ZPR:$vec), VectorIndexD:$index)),
             (i64 (UMOVvi64 (v2i64 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexD:$index))>;
   }
-
+  let Predicates = [NotInStreamingSVEMode] in {
   def : Pat<(sext_inreg (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index), i8),
             (i32 (SMOVvi8to32 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
   def : Pat<(sext_inreg (anyext (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index)), i8),
             (i64 (SMOVvi8to64 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
 
   def : Pat<(sext_inreg (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index), i16),
             (i32 (SMOVvi16to32 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
   def : Pat<(sext_inreg (anyext (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index)), i16),
             (i64 (SMOVvi16to64 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
 
   def : Pat<(sext (vector_extract (nxv4i32 ZPR:$vec), VectorIndexS:$index)),
             (i64 (SMOVvi32to64 (v4i32 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexS:$index))>;
-
+  }
   // Extract first element from vector.
   let AddedComplexity = 2 in {
   def : Pat<(vector_extract (nxv16i8 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv8i16 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv4i32 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ext-loads.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+define <8 x i16> @load_zext_v8i8i16(<8 x i8>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v8i8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldp s1, s0, [x0]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    uunpklo z2.h, z0.b
+; CHECK-NEXT:    uunpklo z0.h, z1.b
+; CHECK-NEXT:    splice z0.h, p0, z0.h, z2.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <8 x i8>, <8 x i8>* %ap
+  %val = zext <8 x i8> %a to <8 x i16>
+  ret <8 x i16> %val
+}
+
+define <4 x i32> @load_zext_v4i16i32(<4 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v4i16i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <4 x i16>, <4 x i16>* %ap
+  %val = zext <4 x i16> %a to <4 x i32>
+  ret <4 x i32> %val
+}
+
+define <2 x i64> @load_zext_v2i32i64(<2 x i32>* %ap) #0 {
+; CHECK-LABEL: load_zext_v2i32i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <2 x i32>, <2 x i32>* %ap
+  %val = zext <2 x i32> %a to <2 x i64>
+  ret <2 x i64> %val
+}
+
+define <2 x i256> @load_zext_v2i64i256(<2 x i64>* %ap) #0 {
+; CHECK-LABEL: load_zext_v2i64i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI3_0
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    mov x1, xzr
+; CHECK-NEXT:    mov x5, xzr
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI3_0]
+; CHECK-NEXT:    mov z2.d, z0.d[1]
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    fmov x4, d2
+; CHECK-NEXT:    mov z0.d, z1.d[1]
+; CHECK-NEXT:    fmov x2, d1
+; CHECK-NEXT:    fmov x3, d0
+; CHECK-NEXT:    mov x6, x2
+; CHECK-NEXT:    mov x7, x3
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = zext <2 x i64> %a to <2 x i256>
+  ret <2 x i256> %val
+}
+
+define <16 x i32> @load_sext_v16i8i32(<16 x i8>* %ap)  #0 {
+; CHECK-LABEL: load_sext_v16i8i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q1, [x0]
+; CHECK-NEXT:    sunpklo z3.h, z1.b
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    sunpklo z4.h, z1.b
+; CHECK-NEXT:    sunpklo z0.s, z3.h
+; CHECK-NEXT:    ext z3.b, z3.b, z3.b, #8
+; CHECK-NEXT:    sunpklo z2.s, z4.h
+; CHECK-NEXT:    ext z4.b, z4.b, z4.b, #8
+; CHECK-NEXT:    sunpklo z1.s, z3.h
+; CHECK-NEXT:    sunpklo z3.s, z4.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    // kill: def $q2 killed $q2 killed $z2
+; CHECK-NEXT:    // kill: def $q3 killed $q3 killed $z3
+; CHECK-NEXT:    ret
+  %a = load <16 x i8>, <16 x i8>* %ap
+  %val = sext <16 x i8> %a to <16 x i32>
+  ret <16 x i32> %val
+}
+
+define <8 x i32> @load_sext_v8i16i32(<8 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_sext_v8i16i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q1, [x0]
+; CHECK-NEXT:    sunpklo z0.s, z1.h
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    sunpklo z1.s, z1.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %val = sext <8 x i16> %a to <8 x i32>
+  ret <8 x i32> %val
+}
+
+define <4 x i256> @load_sext_v4i32i256(<4 x i32>* %ap) #0 {
+; CHECK-LABEL: load_sext_v4i32i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    sunpklo z1.d, z0.s
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    fmov x9, d1
+; CHECK-NEXT:    sunpklo z0.d, z0.s
+; CHECK-NEXT:    fmov x11, d0
+; CHECK-NEXT:    mov z0.d, z0.d[1]
+; CHECK-NEXT:    asr x10, x9, #63
+; CHECK-NEXT:    asr x12, x11, #63
+; CHECK-NEXT:    stp x9, x10, [x8]
+; CHECK-NEXT:    fmov x9, d0
+; CHECK-NEXT:    mov z0.d, z1.d[1]
+; CHECK-NEXT:    stp x11, x12, [x8, #64]
+; CHECK-NEXT:    fmov x11, d0
+; CHECK-NEXT:    stp x10, x10, [x8, #16]
+; CHECK-NEXT:    stp x12, x12, [x8, #80]
+; CHECK-NEXT:    asr x10, x9, #63
+; CHECK-NEXT:    asr x12, x11, #63
+; CHECK-NEXT:    stp x10, x10, [x8, #112]
+; CHECK-NEXT:    stp x9, x10, [x8, #96]
+; CHECK-NEXT:    stp x12, x12, [x8, #48]
+; CHECK-NEXT:    stp x11, x12, [x8, #32]
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = sext <4 x i32> %a to <4 x i256>
+  ret <4 x i256> %val
+}
+
+define <2 x i256> @load_sext_v2i64i256(<2 x i64>* %ap) #0 {
+; CHECK-LABEL: load_sext_v2i64i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    mov z1.d, z0.d[1]
+; CHECK-NEXT:    fmov x8, d0
+; CHECK-NEXT:    fmov x9, d1
+; CHECK-NEXT:    asr x8, x8, #63
+; CHECK-NEXT:    asr x9, x9, #63
+; CHECK-NEXT:    fmov d2, x8
+; CHECK-NEXT:    fmov d3, x9
+; CHECK-NEXT:    zip1 z0.d, z0.d, z2.d
+; CHECK-NEXT:    zip1 z2.d, z2.d, z2.d
+; CHECK-NEXT:    zip1 z1.d, z1.d, z3.d
+; CHECK-NEXT:    mov z4.d, z0.d[1]
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    mov z0.d, z2.d[1]
+; CHECK-NEXT:    fmov x2, d2
+; CHECK-NEXT:    mov z2.d, z1.d[1]
+; CHECK-NEXT:    fmov x4, d1
+; CHECK-NEXT:    zip1 z1.d, z3.d, z3.d
+; CHECK-NEXT:    fmov x3, d0
+; CHECK-NEXT:    mov z0.d, z1.d[1]
+; CHECK-NEXT:    fmov x1, d4
+; CHECK-NEXT:    fmov x6, d1
+; CHECK-NEXT:    fmov x5, d2
+; CHECK-NEXT:    fmov x7, d0
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = sext <2 x i64> %a to <2 x i256>
+  ret <2 x i256> %val
+}
+
+define <16 x i64> @load_zext_v16i16i64(<16 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v16i16i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldp q1, q2, [x0]
+; CHECK-NEXT:    uunpklo z3.s, z1.h
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    uunpklo z7.s, z1.h
+; CHECK-NEXT:    uunpklo z0.d, z3.s
+; CHECK-NEXT:    uunpklo z5.s, z2.h
+; CHECK-NEXT:    ext z2.b, z2.b, z2.b, #8
+; CHECK-NEXT:    uunpklo z16.s, z2.h
+; CHECK-NEXT:    uunpklo z4.d, z5.s
+; CHECK-NEXT:    ext z3.b, z3.b, z3.b, #8
+; CHECK-NEXT:    ext z5.b, z5.b, z5.b, #8
+; CHECK-NEXT:    uunpklo z2.d, z7.s
+; CHECK-NEXT:    uunpklo z6.d, z16.s
+; CHECK-NEXT:    ext z7.b, z7.b, z7.b, #8
+; CHECK-NEXT:    ext z16.b, z16.b, z16.b, #8
+; CHECK-NEXT:    uunpklo z1.d, z3.s
+; CHECK-NEXT:    uunpklo z5.d, z5.s
+; CHECK-NEXT:    uunpklo z3.d, z7.s
+; CHECK-NEXT:    uunpklo z7.d, z16.s
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    // kill: def $q2 killed $q2 killed $z2
+; CHECK-NEXT:    // kill: def $q3 killed $q3 killed $z3
+; CHECK-NEXT:    // kill: def $q4 killed $q4 killed $z4
+; CHECK-NEXT:    // kill: def $q5 killed $q5 killed $z5
+; CHECK-NEXT:    // kill: def $q6 killed $q6 killed $z6
+; CHECK-NEXT:    // kill: def $q7 killed $q7 killed $z7
+; CHECK-NEXT:    ret
+  %a = load <16 x i16>, <16 x i16>* %ap
+  %val = zext <16 x i16> %a to <16 x i64>
+  ret <16 x i64> %val
+}
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-loads.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
 
 target triple = "aarch64-unknown-linux-gnu"
 
 define <4 x i8> @load_v4i8(<4 x i8>* %a) #0 {
 ; CHECK-LABEL: load_v4i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.h, vl4
-; CHECK-NEXT:    ld1b { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ldr s0, [x0]
+; CHECK-NEXT:    uunpklo z0.h, z0.b
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
 ; CHECK-NEXT:    ret
   %load = load <4 x i8>, <4 x i8>* %a
   ret <4 x i8> %load
 }
 
 define <8 x i8> @load_v8i8(<8 x i8>* %a) #0 {
 ; CHECK-LABEL: load_v8i8:

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-load.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+;
+; Masked Load
+;
+
+define <4 x i8> @masked_load_v4i8(<4 x i8>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI0_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_0]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1b { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 8, <4 x i1> %mask, <4 x i8> zeroinitializer)
+  ret <4 x i8> %load
+}
+
+define <8 x i8> @masked_load_v8i8(<8 x i8>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI1_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl8
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI1_0]
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 8, <8 x i1> %mask, <8 x i8> zeroinitializer)
+  ret <8 x i8> %load
+}
+
+define <16 x i8> @masked_load_v16i8(<16 x i8>* %src, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI2_0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI2_0]
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %src, i32 8, <16 x i1> %mask, <16 x i8> zeroinitializer)
+  ret <16 x i8> %load
+}
+
+define <32 x i8> @masked_load_v32i8(<32 x i8>* %src, <32 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #32
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    ldr w8, [sp, #224]
+; CHECK-NEXT:    strb w7, [sp, #6]
+; CHECK-NEXT:    ldr w9, [sp, #216]
+; CHECK-NEXT:    strb w6, [sp, #5]
+; CHECK-NEXT:    ldr w10, [sp, #208]
+; CHECK-NEXT:    strb w5, [sp, #4]
+; CHECK-NEXT:    strb w8, [sp, #31]
+; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    strb w9, [sp, #30]
+; CHECK-NEXT:    ldr w9, [sp, #192]
+; CHECK-NEXT:    strb w10, [sp, #29]
+; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    strb w8, [sp, #28]
+; CHECK-NEXT:    ldr w8, [sp, #176]
+; CHECK-NEXT:    strb w9, [sp, #27]
+; CHECK-NEXT:    ldr w9, [sp, #168]
+; CHECK-NEXT:    strb w10, [sp, #26]
+; CHECK-NEXT:    ldr w10, [sp, #160]
+; CHECK-NEXT:    strb w8, [sp, #25]
+; CHECK-NEXT:    ldr w8, [sp, #152]
+; CHECK-NEXT:    strb w9, [sp, #24]
+; CHECK-NEXT:    ldr w9, [sp, #144]
+; CHECK-NEXT:    strb w10, [sp, #23]
+; CHECK-NEXT:    ldr w10, [sp, #136]
+; CHECK-NEXT:    strb w8, [sp, #22]
+; CHECK-NEXT:    ldr w8, [sp, #128]
+; CHECK-NEXT:    strb w9, [sp, #21]
+; CHECK-NEXT:    ldr w9, [sp, #120]
+; CHECK-NEXT:    strb w10, [sp, #20]
+; CHECK-NEXT:    ldr w10, [sp, #112]
+; CHECK-NEXT:    strb w8, [sp, #19]
+; CHECK-NEXT:    ldr w8, [sp, #104]
+; CHECK-NEXT:    strb w9, [sp, #18]
+; CHECK-NEXT:    ldr w9, [sp, #96]
+; CHECK-NEXT:    strb w10, [sp, #17]
+; CHECK-NEXT:    ldr w10, [sp, #88]
+; CHECK-NEXT:    strb w8, [sp, #16]
+; CHECK-NEXT:    ldr w8, [sp, #80]
+; CHECK-NEXT:    strb w9, [sp, #15]
+; CHECK-NEXT:    ldr w9, [sp, #72]
+; CHECK-NEXT:    strb w10, [sp, #14]
+; CHECK-NEXT:    ldr w10, [sp, #64]
+; CHECK-NEXT:    strb w8, [sp, #13]
+; CHECK-NEXT:    ldr w8, [sp, #56]
+; CHECK-NEXT:    strb w9, [sp, #12]
+; CHECK-NEXT:    ldr w9, [sp, #48]
+; CHECK-NEXT:    strb w10, [sp, #11]
+; CHECK-NEXT:    ldr w10, [sp, #40]
+; CHECK-NEXT:    strb w8, [sp, #10]
+; CHECK-NEXT:    ldr w8, [sp, #32]
+; CHECK-NEXT:    strb w9, [sp, #9]
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    strb w10, [sp, #8]
+; CHECK-NEXT:    strb w8, [sp, #7]
+; CHECK-NEXT:    adrp x8, .LCPI3_0
+; CHECK-NEXT:    strb w4, [sp, #3]
+; CHECK-NEXT:    strb w3, [sp, #2]
+; CHECK-NEXT:    strb w2, [sp, #1]
+; CHECK-NEXT:    ldr q0, [x8, :lo12:.LCPI3_0]
+; CHECK-NEXT:    strb w1, [sp]
+; CHECK-NEXT:    mov w8, #16
+; CHECK-NEXT:    ldp q2, q1, [sp]
+; CHECK-NEXT:    lsl z2.b, p0/m, z2.b, z0.b
+; CHECK-NEXT:    lsl z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    asr z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    asrr z0.b, p0/m, z0.b, z2.b
+; CHECK-NEXT:    cmpne p1.b, p0/z, z1.b, #0
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ld1b { z1.b }, p1/z, [x0, x8]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    add sp, sp, #32
+; CHECK-NEXT:    ret
+  %load = call <32 x i8> @llvm.masked.load.v32i8(<32 x i8>* %src, i32 8, <32 x i1> %mask, <32 x i8> zeroinitializer)
+  ret <32 x i8> %load
+}
+
+define <2 x half> @masked_load_v2f16(<2 x half>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    adrp x8, .LCPI4_0
+; CHECK-NEXT:    str wzr, [sp, #12]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    strh w10, [sp, #10]
+; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI4_0]
+; CHECK-NEXT:    ldr d1, [sp, #8]
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z0.h
+; CHECK-NEXT:    asrr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %load = call <2 x half> @llvm.masked.load.v2f16(<2 x half>* %src, i32 8, <2 x i1> %mask, <2 x half> zeroinitializer)
+  ret <2 x half> %load
+}
+
+define <4 x half> @masked_load_v4f16(<4 x half>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI5_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI5_0]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x half> @llvm.masked.load.v4f16(<4 x half>* %src, i32 8, <4 x i1> %mask, <4 x half> zeroinitializer)
+  ret <4 x half> %load
+}
+
+define <8 x half> @masked_load_v8f16(<8 x half>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI6_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI6_0]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <8 x half> @llvm.masked.load.v8f16(<8 x half>* %src, i32 8, <8 x i1> %mask, <8 x half> zeroinitializer)
+  ret <8 x half> %load
+}
+
+define <16 x half> @masked_load_v16f16(<16 x half>* %src, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v16f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI7_0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    uunpklo z2.h, z0.b
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI7_0]
+; CHECK-NEXT:    mov x8, #8
+; CHECK-NEXT:    lsl z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p1.h, p0/z, z2.h, #0
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p1/z, [x0]
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0, x8, lsl #1]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %load = call <16 x half> @llvm.masked.load.v16f16(<16 x half>* %src, i32 8, <16 x i1> %mask, <16 x half> zeroinitializer)
+  ret <16 x half> %load
+}
+
+define <2 x float> @masked_load_v2f32(<2 x float>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI8_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.s, vl2
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI8_0]
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <2 x float> @llvm.masked.load.v2f32(<2 x float>* %src, i32 8, <2 x i1> %mask, <2 x float> zeroinitializer)
+  ret <2 x float> %load
+}
+
+define <4 x float> @masked_load_v4f32(<4 x float>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI9_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI9_0]
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x float> @llvm.masked.load.v4f32(<4 x float>* %src, i32 8, <4 x i1> %mask, <4 x float> zeroinitializer)
+  ret <4 x float> %load
+}
+
+define <8 x float> @masked_load_v8f32(<8 x float>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    mov z1.b, z0.b[3]
+; CHECK-NEXT:    mov z2.b, z0.b[2]
+; CHECK-NEXT:    adrp x8, .LCPI10_0
+; CHECK-NEXT:    mov z3.b, z0.b[1]
+; CHECK-NEXT:    mov z4.b, z0.b[7]
+; CHECK-NEXT:    mov z5.b, z0.b[6]
+; CHECK-NEXT:    mov z6.b, z0.b[5]
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    mov z0.b, z0.b[4]
+; CHECK-NEXT:    fmov w11, s2
+; CHECK-NEXT:    strh w9, [sp, #-16]!
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    fmov w9, s3
+; CHECK-NEXT:    strh w10, [sp, #6]
+; CHECK-NEXT:    fmov w10, s4
+; CHECK-NEXT:    strh w11, [sp, #4]
+; CHECK-NEXT:    fmov w11, s5
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI10_0]
+; CHECK-NEXT:    strh w9, [sp, #2]
+; CHECK-NEXT:    fmov w9, s6
+; CHECK-NEXT:    strh w10, [sp, #14]
+; CHECK-NEXT:    fmov w10, s0
+; CHECK-NEXT:    strh w11, [sp, #12]
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    strh w9, [sp, #10]
+; CHECK-NEXT:    mov x8, #4
+; CHECK-NEXT:    strh w10, [sp, #8]
+; CHECK-NEXT:    ldp d0, d2, [sp]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p1/z, [x0]
+; CHECK-NEXT:    asrr z1.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x0, x8, lsl #2]
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %load = call <8 x float> @llvm.masked.load.v8f32(<8 x float>* %src, i32 8, <8 x i1> %mask, <8 x float> zeroinitializer)
+  ret <8 x float> %load
+}
+
+define <2 x double> @masked_load_v2f64(<2 x double>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI11_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI11_0]
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <2 x double> @llvm.masked.load.v2f64(<2 x double>* %src, i32 8, <2 x i1> %mask, <2 x double> zeroinitializer)
+  ret <2 x double> %load
+}
+
+define <4 x double> @masked_load_v4f64(<4 x double>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI12_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    uunpklo z2.d, z0.s
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI12_0]
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    mov x8, #2
+; CHECK-NEXT:    lsl z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p1.d, p0/z, z2.d, #0
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    ld1d { z0.d }, p1/z, [x0]
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x0, x8, lsl #3]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %load = call <4 x double> @llvm.masked.load.v4f64(<4 x double>* %src, i32 8, <4 x i1> %mask, <4 x double> zeroinitializer)
+  ret <4 x double> %load
+}
+
+declare <4 x i8> @llvm.masked.load.v4i8(<4 x i8>*, i32, <4 x i1>, <4 x i8>)
+declare <8 x i8> @llvm.masked.load.v8i8(<8 x i8>*, i32, <8 x i1>, <8 x i8>)
+declare <16 x i8> @llvm.masked.load.v16i8(<16 x i8>*, i32, <16 x i1>, <16 x i8>)
+declare <32 x i8> @llvm.masked.load.v32i8(<32 x i8>*, i32, <32 x i1>, <32 x i8>)
+
+declare <2 x half> @llvm.masked.load.v2f16(<2 x half>*, i32, <2 x i1>, <2 x half>)
+declare <4 x half> @llvm.masked.load.v4f16(<4 x half>*, i32, <4 x i1>, <4 x half>)
+declare <8 x half> @llvm.masked.load.v8f16(<8 x half>*, i32, <8 x i1>, <8 x half>)
+declare <16 x half> @llvm.masked.load.v16f16(<16 x half>*, i32, <16 x i1>, <16 x half>)
+
+declare <2 x float> @llvm.masked.load.v2f32(<2 x float>*, i32, <2 x i1>, <2 x float>)
+declare <4 x float> @llvm.masked.load.v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
+declare <8 x float> @llvm.masked.load.v8f32(<8 x float>*, i32, <8 x i1>, <8 x float>)
+
+declare <2 x double> @llvm.masked.load.v2f64(<2 x double>*, i32, <2 x i1>, <2 x double>)
+declare <4 x double> @llvm.masked.load.v4f64(<4 x double>*, i32, <4 x i1>, <4 x double>)
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-store.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+;
+; Masked Store
+;
+
+define void @masked_store_v4i8(<4 x i8>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI0_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_0]
+; CHECK-NEXT:    adrp x8, .LCPI0_1
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI0_1]
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1b { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4i8(<4 x i8> zeroinitializer, <4 x i8>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8i8(<8 x i8>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI1_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl8
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI1_0]
+; CHECK-NEXT:    adrp x8, .LCPI1_1
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI1_1]
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z2.b }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8i8(<8 x i8> zeroinitializer, <8 x i8>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v16i8(<16 x i8>* %dst, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI2_0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI2_0]
+; CHECK-NEXT:    adrp x8, .LCPI2_1
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI2_1]
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z2.b }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v16i8(<16 x i8> zeroinitializer, <16 x i8>* %dst, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v32i8(<32 x i8>* %dst, <32 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #32
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    ldr w8, [sp, #96]
+; CHECK-NEXT:    strb w7, [sp, #6]
+; CHECK-NEXT:    ldr w9, [sp, #88]
+; CHECK-NEXT:    strb w6, [sp, #5]
+; CHECK-NEXT:    ldr w10, [sp, #80]
+; CHECK-NEXT:    strb w5, [sp, #4]
+; CHECK-NEXT:    strb w8, [sp, #15]
+; CHECK-NEXT:    ldr w8, [sp, #72]
+; CHECK-NEXT:    strb w9, [sp, #14]
+; CHECK-NEXT:    ldr w9, [sp, #64]
+; CHECK-NEXT:    strb w10, [sp, #13]
+; CHECK-NEXT:    ldr w10, [sp, #56]
+; CHECK-NEXT:    strb w8, [sp, #12]
+; CHECK-NEXT:    ldr w8, [sp, #48]
+; CHECK-NEXT:    strb w9, [sp, #11]
+; CHECK-NEXT:    ldr w9, [sp, #40]
+; CHECK-NEXT:    strb w10, [sp, #10]
+; CHECK-NEXT:    ldr w10, [sp, #32]
+; CHECK-NEXT:    strb w8, [sp, #9]
+; CHECK-NEXT:    ldr w8, [sp, #224]
+; CHECK-NEXT:    strb w9, [sp, #8]
+; CHECK-NEXT:    ldr w9, [sp, #216]
+; CHECK-NEXT:    strb w10, [sp, #7]
+; CHECK-NEXT:    ldr w10, [sp, #208]
+; CHECK-NEXT:    strb w8, [sp, #31]
+; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    strb w9, [sp, #30]
+; CHECK-NEXT:    ldr w9, [sp, #192]
+; CHECK-NEXT:    strb w10, [sp, #29]
+; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    strb w8, [sp, #28]
+; CHECK-NEXT:    ldr w8, [sp, #176]
+; CHECK-NEXT:    strb w9, [sp, #27]
+; CHECK-NEXT:    ldr w9, [sp, #168]
+; CHECK-NEXT:    strb w10, [sp, #26]
+; CHECK-NEXT:    ldr w10, [sp, #160]
+; CHECK-NEXT:    strb w8, [sp, #25]
+; CHECK-NEXT:    ldr w8, [sp, #152]
+; CHECK-NEXT:    strb w9, [sp, #24]
+; CHECK-NEXT:    ldr w9, [sp, #144]
+; CHECK-NEXT:    strb w10, [sp, #23]
+; CHECK-NEXT:    ldr w10, [sp, #136]
+; CHECK-NEXT:    strb w8, [sp, #22]
+; CHECK-NEXT:    ldr w8, [sp, #128]
+; CHECK-NEXT:    strb w9, [sp, #21]
+; CHECK-NEXT:    ldr w9, [sp, #120]
+; CHECK-NEXT:    strb w10, [sp, #20]
+; CHECK-NEXT:    ldr w10, [sp, #104]
+; CHECK-NEXT:    strb w8, [sp, #19]
+; CHECK-NEXT:    ldr w8, [sp, #112]
+; CHECK-NEXT:    strb w4, [sp, #3]
+; CHECK-NEXT:    adrp x11, .LCPI3_0
+; CHECK-NEXT:    strb w3, [sp, #2]
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    strb w2, [sp, #1]
+; CHECK-NEXT:    strb w1, [sp]
+; CHECK-NEXT:    ldr q0, [x11, :lo12:.LCPI3_0]
+; CHECK-NEXT:    strb w9, [sp, #18]
+; CHECK-NEXT:    strb w8, [sp, #17]
+; CHECK-NEXT:    adrp x8, .LCPI3_1
+; CHECK-NEXT:    strb w10, [sp, #16]
+; CHECK-NEXT:    ldp q1, q2, [sp]
+; CHECK-NEXT:    lsl z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    asr z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    lsl z2.b, p0/m, z2.b, z0.b
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI3_1]
+; CHECK-NEXT:    mov w8, #16
+; CHECK-NEXT:    asrr z0.b, p0/m, z0.b, z2.b
+; CHECK-NEXT:    cmpne p1.b, p0/z, z1.b, #0
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z3.b }, p0, [x0, x8]
+; CHECK-NEXT:    st1b { z3.b }, p1, [x0]
+; CHECK-NEXT:    add sp, sp, #32
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v32i8(<32 x i8> zeroinitializer, <32 x i8>* %dst, i32 8, <32 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v2f16(<2 x half>* %dst, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v2f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    adrp x8, .LCPI4_0
+; CHECK-NEXT:    str wzr, [sp, #12]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    strh w10, [sp, #10]
+; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI4_0]
+; CHECK-NEXT:    ldr d1, [sp, #8]
+; CHECK-NEXT:    adrp x8, .LCPI4_1
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z0.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI4_1]
+; CHECK-NEXT:    asrr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v2f16(<2 x half> zeroinitializer, <2 x half>* %dst, i32 8, <2 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f16(<4 x half>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI5_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI5_0]
+; CHECK-NEXT:    adrp x8, .LCPI5_1
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI5_1]
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f16(<4 x half> zeroinitializer, <4 x half>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8f16(<8 x half>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI6_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI6_0]
+; CHECK-NEXT:    adrp x8, .LCPI6_1
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI6_1]
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8f16(<8 x half> zeroinitializer, <8 x half>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v16f16(<16 x half>* %dst, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v16f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI7_0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    mov z1.d, z0.d
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    uunpklo z1.h, z1.b
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI7_0]
+; CHECK-NEXT:    adrp x8, .LCPI7_1
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI7_1]
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    mov x8, #8
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z2.h
+; CHECK-NEXT:    asr z1.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z2.h
+; CHECK-NEXT:    cmpne p1.h, p0/z, z1.h, #0
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z3.h }, p1, [x0, x8, lsl #1]
+; CHECK-NEXT:    st1h { z3.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v16f16(<16 x half> zeroinitializer, <16 x half>* %dst, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f32(<4 x float>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI8_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI8_0]
+; CHECK-NEXT:    adrp x8, .LCPI8_1
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI8_1]
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z2.s }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f32(<4 x float> zeroinitializer, <4 x float>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8f32(<8 x float>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.b, z0.b[7]
+; CHECK-NEXT:    mov z2.b, z0.b[6]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    mov z1.b, z0.b[5]
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    mov z2.b, z0.b[4]
+; CHECK-NEXT:    fmov w11, s1
+; CHECK-NEXT:    adrp x8, .LCPI9_0
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    fmov w9, s2
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    adrp x10, .LCPI9_1
+; CHECK-NEXT:    strh w11, [sp, #10]
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI9_0]
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    mov x9, #4
+; CHECK-NEXT:    ldr d1, [sp, #8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    ldr q3, [x10, :lo12:.LCPI9_1]
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    mov z4.b, z0.b[3]
+; CHECK-NEXT:    mov z5.b, z0.b[2]
+; CHECK-NEXT:    uunpklo z1.s, z1.h
+; CHECK-NEXT:    mov z6.b, z0.b[1]
+; CHECK-NEXT:    lsl z1.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    movprfx z0, z1
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z2.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    fmov w10, s4
+; CHECK-NEXT:    st1w { z3.s }, p1, [x0, x9, lsl #2]
+; CHECK-NEXT:    fmov w9, s5
+; CHECK-NEXT:    strh w8, [sp]
+; CHECK-NEXT:    fmov w8, s6
+; CHECK-NEXT:    strh w10, [sp, #6]
+; CHECK-NEXT:    strh w9, [sp, #4]
+; CHECK-NEXT:    strh w8, [sp, #2]
+; CHECK-NEXT:    ldr d0, [sp]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z2.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z2.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z3.s }, p0, [x0]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8f32(<8 x float> zeroinitializer, <8 x float>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v2f64(<2 x double>* %dst, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI11_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI11_0]
+; CHECK-NEXT:    adrp x8, .LCPI11_1
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI11_1]
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    st1d { z2.d }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v2f64(<2 x double> zeroinitializer, <2 x double>* %dst, i32 8, <2 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f64(<4 x double>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI12_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    uunpklo z2.d, z0.s
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI12_0]
+; CHECK-NEXT:    adrp x8, .LCPI12_1
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI12_1]
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    mov x8, #2
+; CHECK-NEXT:    lsl z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asrr z1.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
+; CHECK-NEXT:    st1d { z3.d }, p1, [x0, x8, lsl #3]
+; CHECK-NEXT:    st1d { z3.d }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f64(<4 x double> zeroinitializer, <4 x double>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+declare void @llvm.masked.store.v4i8(<4 x i8>, <4 x i8>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8i8(<8 x i8>, <8 x i8>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16i8(<16 x i8>, <16 x i8>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v32i8(<32 x i8>, <32 x i8>*, i32, <32 x i1>)
+declare void @llvm.masked.store.v2f16(<2 x half>, <2 x half>*, i32, <2 x i1>)
+declare void @llvm.masked.store.v4f16(<4 x half>, <4 x half>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8f16(<8 x half>, <8 x half>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16f16(<16 x half>, <16 x half>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v4f32(<4 x float>, <4 x float>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8f32(<8 x float>, <8 x float>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v2f64(<2 x double>, <2 x double>*, i32, <2 x i1>)
+declare void @llvm.masked.store.v4f64(<4 x double>, <4 x double>*, i32, <4 x i1>)
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-stores.ll

; CHECK-NEXT:    ret
   ret void
 }
 
 define void @store_v2f16(<2 x half>* %a) #0 {
 ; CHECK-LABEL: store_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI5_0
 ; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI5_0]
-; CHECK-NEXT:    str s0, [x0]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    str w8, [x0]
 ; CHECK-NEXT:    ret
   store <2 x half> zeroinitializer, <2 x half>* %a
   ret void
 }
 
 define void @store_v4i16(<4 x i16>* %a) #0 {
 ; CHECK-LABEL: store_v4i16:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-trunc-stores.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+define void @store_trunc_v8i16i8(<8 x i16>* %ap, <8 x i8>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v8i16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    mov z1.h, z0.h[7]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    mov z2.h, z0.h[6]
+; CHECK-NEXT:    mov z3.h, z0.h[5]
+; CHECK-NEXT:    mov z4.h, z0.h[4]
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strb w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s3
+; CHECK-NEXT:    strb w9, [sp, #15]
+; CHECK-NEXT:    fmov w9, s4
+; CHECK-NEXT:    mov z5.h, z0.h[3]
+; CHECK-NEXT:    mov z6.h, z0.h[2]
+; CHECK-NEXT:    mov z0.h, z0.h[1]
+; CHECK-NEXT:    strb w10, [sp, #14]
+; CHECK-NEXT:    fmov w10, s5
+; CHECK-NEXT:    strb w8, [sp, #13]
+; CHECK-NEXT:    fmov w8, s6
+; CHECK-NEXT:    strb w9, [sp, #12]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    strb w10, [sp, #11]
+; CHECK-NEXT:    strb w8, [sp, #10]
+; CHECK-NEXT:    strb w9, [sp, #9]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    str d0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %val = trunc <8 x i16> %a to <8 x i8>
+  store <8 x i8> %val, <8 x i8>* %dest
+  ret void
+}
+
+define void @store_trunc_v4i32i8(<4 x i32>* %ap, <4 x i8>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v4i32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    mov z1.s, z0.s[3]
+; CHECK-NEXT:    mov z2.s, z0.s[2]
+; CHECK-NEXT:    mov z0.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    strh w8, [sp, #10]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = trunc <4 x i32> %a to <4 x i8>
+  store <4 x i8> %val, <4 x i8>* %dest
+  ret void
+}
+
+define void @store_trunc_v4i32i16(<4 x i32>* %ap, <4 x i16>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v4i32i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    mov z1.s, z0.s[3]
+; CHECK-NEXT:    mov z2.s, z0.s[2]
+; CHECK-NEXT:    mov z0.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    strh w8, [sp, #10]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    str d0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = trunc <4 x i32> %a to <4 x i16>
+  store <4 x i16> %val, <4 x i16>* %dest
+  ret void
+}
+
+define void @store_trunc_v2i64i8(<2 x i64>* %ap, <2 x i32>* %dest) vscale_range(2,0) #0 {
+; CHECK-LABEL: store_trunc_v2i64i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    st1w { z0.d }, p0, [x1]
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = trunc <2 x i64> %a to <2 x i32>
+  store <2 x i32> %val, <2 x i32>* %dest
+  ret void
+}
+
+define void @store_trunc_v2i256i64(<2 x i256>* %ap, <2 x i64>* %dest) vscale_range(2,0) #0 {
+; CHECK-LABEL: store_trunc_v2i256i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0, #32]
+; CHECK-NEXT:    ptrue p0.d, vl1
+; CHECK-NEXT:    ldr d1, [x0]
+; CHECK-NEXT:    splice z1.d, p0, z1.d, z0.d
+; CHECK-NEXT:    str q1, [x1]
+; CHECK-NEXT:    ret
+  %a = load <2 x i256>, <2 x i256>* %ap
+  %val = trunc <2 x i256> %a to <2 x i64>
+  store <2 x i64> %val, <2 x i64>* %dest
+  ret void
+}
+
+attributes #0 = { "target-features"="+sve" }