[AArch64-SVE]: Force generating code compatible to streaming mode.

[hassnaa-arm]

Summary

When streaming mode is enabled, lower some operations and disable some code paths to force generating code compatible to streaming mode.
Add testing files for shifts, build_vector, concat, extract_subvector, extract_vector_elt, and shuffle.

[hassnaa-arm]

Add additional test cases

[hassnaa-arm]

get latest changes of parent revision

[hassnaa-arm]

update by rebasing parent revision

[hassnaa-arm]

Remove unrelated changes

[hassnaa-arm]

Restore some changes removed by mistake

[hassnaa-arm]

Update by parent branch

[hassnaa-arm]

Update by parent patch

[SjoerdMeijer]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11925

Drive by comment:

we don't need to pass Subtarget->forceStreamingCompatibleSVE() but can just query that inside useSVEForFixedLengthVectorVT?

[hassnaa-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11925

I think we still need to pass it, because useSVEForFixedLengthVectorVT is used in many places, some of them don't need the flag to be enabled.
The flag is used for enabling lowering specific nodes that cause generating invalid code in streaming mode.

[hassnaa-arm]

Update by changes of parent patch.
While lowering ISD::load remove enabling LowerFixedLengthVectorLoadToSVE,
no need for it, as zero_Extend is custom-lowered.
Previously, LowerLOAD() creates zero_Extend node, which cause invalid generated code,
but now, the zero_extend node is custom-lowered, which cause valid generated code.

[hassnaa-arm]

Update by parent patch

[sdesmalen]

Hi @hassnaa-arm, I think you have the tests the wrong way around. The tests from D136147 should be part of this patch, because this is the patch where you're implementing the lowering of the operations you're testing in D136147.
After this patch, you get the masked/truncating/extending load/store operations "for free", so the tests for those operations could be moved to a separate test-only patch like D136147.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1399

It should be able to use standard scalar instructions for v1i64 in streaming-compatible mode, so this one can be removed from the list.

1402

Most scalar FP operations are valid in streaming mode, so we probably don't need to do anything custom for this type.

1614–1616

nit: Perhaps it doesn't lead to an error, but these operations only operate on integers, so should be guarded by:

if (VT.isInteger()) { ... }

12091–12097

nit: rather than wrapping this in another condition, can you just add it to the existing condition with && !Subtarget->forceStreamingCompatibleSVE() ?

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3035–3046

nit: Can you change this into:

let Predicates = [NotInStreamingSVEMode], AddedComplexity = 1 in {
def : Pat<...>
..
}

let Predicates = [NotInStreamingSVEMode] in {
def : Pat<..>
...
}

Rather than indenting?

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-store.ll

313

Can you remove all tests that are larger than "twice the size" of a 128bit vector (v32f32 is 8x the size, I'm not sure what value that adds for the testing of this functionality)

[hassnaa-arm]

Remove all tests that are larger than "twice the size" of a 128bit vector.

[hassnaa-arm]

Remove masked/truncating/extending load/store, to be added in a test-only patch.

[hassnaa-arm]

Lower And operation, and disable replacing 'and' by 'bic' while combining step.

[hassnaa-arm]

Update by parent patch

[hassnaa-arm]

Revert changes added by mistake

[hassnaa-arm]

Add testing files for shifts, build_vector, concat, extract_subvector, extract_vector_elt, and shuffle.

[david-arm]

Hi @hassnaa-arm, I think it's almost there! Most of the tests look good to me. I just had a few minor comments ...

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11198

I don't think you need to pass in the Subtarget here. In the code below you can just do

if (VT.isFixedLengthVector() &&
    DAG.getSubtarget<AArch64Subtarget>().forceStreamingCompatibleSVE()
  return SDValue();

11244

Same comment as above for tryAdvSIMDModImm32

13813

nit: The comment can probably be formatted better I think so that you use up 80 chars, i.e.:

// Skip if streaming compatible SVE is enabled, because it generates invalid
// code in streaming mode when SVE length is not specified.

15528

Again, you can avoid passing in the subtarget here if you make the changes to tryAdvSIMDModImm32 and tryAdvSIMDModImm16.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3034–3035

When we guard something by a predicate we normally add a comment on the final brace '}' to make it easy to see, i.e. something like:

} // End NotInStreamingSVEMode

3056–3059

} // End NotInStreamingSVEMode

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-concat.ll

10 ↗

(On Diff #470094)

I don't think we need to have vscale_range(2,0) on these tests, right? We want streaming SVE to work for vector lengths.

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-extract-vector-elt.ll

17 ↗

(On Diff #470094)

nit: I think in all these tests there should only be 2 spaces at the start of the IR, i.e.

%r = extractelement <2 x half> %op1, i64 1

etc.

[hassnaa-arm]

Remove vscale_range from concat.ll test file.

[hassnaa-arm]

Fix identation

[david-arm]

LGTM! Thanks for making the changes @hassnaa-arm.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

22083–22084

As with the above change can this be V.getValueType().isFixedLengthVector() && isTypeLegal(V.getValueType()) &&?

llvm/test/CodeGen/AArch64/sve-streaming-fixed-length-int-shifts.ll

1–5 ↗

(On Diff #471171)

Please rename this file sve-streaming-mode-fixed-length-int-shifts.ll to match the same format as the others.

[hassnaa-arm]

Rename sve-streaming-fixed-length-int-shifts.ll to sve-streaming-mode-fixed-length-int-shifts.ll

[sdesmalen]

LGTM as well (please address nit before submitting)

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

22083

nit: Does this cross the 80-character limit? (please use clang-format to be sure)

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1620

nit: I only just spot this now in one of your other patches, but ISD::AND should also be guarded by VT.isInteger().

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1620

Although not wrong it doesn't really matter as legalisation is smart enough to not care about the operation action for types that make no sense. We rely on this in addTypeForFixedLengthSVE where the type is only considered when handling extend-loads/truncating-store plus the odd compare.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1620

In that case it's probably better to remove the condition entirely.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->hasSVE()) {
     setOperationAction(ISD::MUL, MVT::v2i64, Custom);
 
     // NEON doesn't support across-vector reductions, but SVE does.
     for (auto VT : {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v2f64})
       setOperationAction(ISD::VECREDUCE_SEQ_FADD, VT, Custom);
 
     if (Subtarget->forceStreamingCompatibleSVE()) {
       for (MVT VT : {MVT::v8i8, MVT::v16i8, MVT::v4i16, MVT::v8i16, MVT::v2i32,
                      MVT::v4i32, MVT::v1i64, MVT::v2i64})
         addTypeForStreamingSVE(VT);
 
       for (MVT VT : {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v1f64,
                      MVT::v2f64})
         addTypeForStreamingSVE(VT);
     }
 
     // NOTE: Currently this has to happen after computeRegisterProperties rather
     // than the preferred option of combining it with the addRegisterClass call.
     if (Subtarget->useSVEForFixedLengthVectors()) {
       for (MVT VT : MVT::integer_fixedlen_vector_valuetypes())
bool AArch64TargetLowering::shouldExpandGetActiveLaneMask(EVT ResVT,
 
   // The whilelo instruction only works with i32 or i64 scalar inputs.
   if (OpVT != MVT::i32 && OpVT != MVT::i64)
     return true;
 
   return false;
 }
 
-void AArch64TargetLowering::addTypeForStreamingSVE(MVT VT) {}
+void AArch64TargetLowering::addTypeForStreamingSVE(MVT VT) {
+  setOperationAction(ISD::ANY_EXTEND, VT, Custom);
+  setOperationAction(ISD::ZERO_EXTEND, VT, Custom);
+  setOperationAction(ISD::SIGN_EXTEND, VT, Custom);
+  setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
+}
 
 void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   assert(VT.isFixedLengthVector() && "Expected fixed length vector type!");
 
   // By default everything must be expanded.
   for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op)
     setOperationAction(Op, VT, Expand);
 
   // We use EXTRACT_SUBVECTOR to "cast" a scalable vector to a fixed length one.
   setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);

 }
 
 // Try 32-bit splatted SIMD immediate.
 static SDValue tryAdvSIMDModImm32(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
                                   const APInt &Bits,
                                   const SDValue *LHS = nullptr) {
   if (Bits.getHiBits(64) == Bits.getLoBits(64)) {
     uint64_t Value = Bits.zextOrTrunc(64).getZExtValue();
     EVT VT = Op.getValueType();
     MVT MovTy = (VT.getSizeInBits() == 128) ? MVT::v4i32 : MVT::v2i32;
     bool isAdvSIMDModImm = false;
     uint64_t Shift;
 
     if ((isAdvSIMDModImm = AArch64_AM::isAdvSIMDModImmType1(Value))) {
       Value = AArch64_AM::encodeAdvSIMDModImmType1(Value);
       Shift = 0;
     }
static SDValue tryAdvSIMDModImm32(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
 
   return SDValue();
 }
 
 // Try 16-bit splatted SIMD immediate.
 static SDValue tryAdvSIMDModImm16(unsigned NewOp, SDValue Op, SelectionDAG &DAG,
                                   const APInt &Bits,
                                   const SDValue *LHS = nullptr) {
   if (Bits.getHiBits(64) == Bits.getLoBits(64)) {
     uint64_t Value = Bits.zextOrTrunc(64).getZExtValue();
     EVT VT = Op.getValueType();
     MVT MovTy = (VT.getSizeInBits() == 128) ? MVT::v8i16 : MVT::v4i16;
     bool isAdvSIMDModImm = false;
     uint64_t Shift;
 
     if ((isAdvSIMDModImm = AArch64_AM::isAdvSIMDModImmType5(Value))) {
       Value = AArch64_AM::encodeAdvSIMDModImmType5(Value);
SDValue AArch64TargetLowering::LowerBUILD_VECTOR(SDValue Op,
   LLVM_DEBUG(
       dbgs() << "LowerBUILD_VECTOR: use default expansion, failed to find "
                 "better alternative\n");
   return SDValue();
 }
 
 SDValue AArch64TargetLowering::LowerCONCAT_VECTORS(SDValue Op,
                                                    SelectionDAG &DAG) const {
-  if (useSVEForFixedLengthVectorVT(Op.getValueType()))
+  if (useSVEForFixedLengthVectorVT(Op.getValueType(),
+                                   Subtarget->forceStreamingCompatibleSVE()))
     return LowerFixedLengthConcatVectorsToSVE(Op, DAG);
 
   assert(Op.getValueType().isScalableVector() &&
          isTypeLegal(Op.getValueType()) &&
          "Expected legal scalable vector type!");
 
   if (isTypeLegal(Op.getOperand(0).getValueType())) {
     unsigned NumOperands = Op->getNumOperands();
if (VT.getScalarType() == MVT::i1) {
     SDValue Extend =
         DAG.getNode(ISD::ANY_EXTEND, DL, VectorVT, Op.getOperand(0));
     MVT ExtractTy = VectorVT == MVT::nxv2i64 ? MVT::i64 : MVT::i32;
     SDValue Extract = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, ExtractTy,
                                   Extend, Op.getOperand(1));
     return DAG.getAnyExtOrTrunc(Extract, DL, Op.getValueType());
   }
 
-  if (useSVEForFixedLengthVectorVT(VT))
+  if (useSVEForFixedLengthVectorVT(VT,
+                                   Subtarget->forceStreamingCompatibleSVE()))
     return LowerFixedLengthExtractVectorElt(Op, DAG);
 
   // Check for non-constant or out of range lane.
   ConstantSDNode *CI = dyn_cast<ConstantSDNode>(Op.getOperand(1));
   if (!CI || CI->getZExtValue() >= VT.getVectorNumElements())
     return SDValue();
 
   // Insertion/extraction are legal for V128 types.
if (InVT.isScalableVector()) {
 
     return SDValue();
   }
 
   // This will get lowered to an appropriate EXTRACT_SUBREG in ISel.
   if (Idx == 0 && InVT.getSizeInBits() <= 128)
     return Op;
 
+  if (!Subtarget->forceStreamingCompatibleSVE()) {
     // If this is extracting the upper 64-bits of a 128-bit vector, we match
     // that directly.
     if (Size == 64 && Idx * InVT.getScalarSizeInBits() == 64 &&
         InVT.getSizeInBits() == 128)
       return Op;
+  }
 
-  if (useSVEForFixedLengthVectorVT(InVT)) {
+  if (useSVEForFixedLengthVectorVT(InVT,
+                                   Subtarget->forceStreamingCompatibleSVE())) {
     SDLoc DL(Op);
 
     EVT ContainerVT = getContainerForFixedLengthVector(DAG, InVT);
     SDValue NewInVec =
         convertToScalableVector(DAG, ContainerVT, Op.getOperand(0));
 
     SDValue Splice = DAG.getNode(ISD::VECTOR_SPLICE, DL, ContainerVT, NewInVec,
                                  NewInVec, DAG.getConstant(Idx, DL, MVT::i64));
SDValue AArch64TargetLowering::LowerVectorSRA_SRL_SHL(SDValue Op,
   int64_t Cnt;
 
   if (!Op.getOperand(1).getValueType().isVector())
     return Op;
   unsigned EltSize = VT.getScalarSizeInBits();
 
   switch (Op.getOpcode()) {
   case ISD::SHL:
-    if (VT.isScalableVector() || useSVEForFixedLengthVectorVT(VT))
+    if (VT.isScalableVector() ||
+        useSVEForFixedLengthVectorVT(VT,
+                                     Subtarget->forceStreamingCompatibleSVE()))
       return LowerToPredicatedOp(Op, DAG, AArch64ISD::SHL_PRED);
 
     if (isVShiftLImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize)
       return DAG.getNode(AArch64ISD::VSHL, DL, VT, Op.getOperand(0),
                          DAG.getConstant(Cnt, DL, MVT::i32));
     return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
                        DAG.getConstant(Intrinsic::aarch64_neon_ushl, DL,
                                        MVT::i32),
                        Op.getOperand(0), Op.getOperand(1));
   case ISD::SRA:
   case ISD::SRL:
-    if (VT.isScalableVector() || useSVEForFixedLengthVectorVT(VT)) {
+    if (VT.isScalableVector() ||
+        useSVEForFixedLengthVectorVT(
+            VT, Subtarget->forceStreamingCompatibleSVE())) {
       unsigned Opc = Op.getOpcode() == ISD::SRA ? AArch64ISD::SRA_PRED
                                                 : AArch64ISD::SRL_PRED;
       return LowerToPredicatedOp(Op, DAG, Opc);
     }
 
     // Right shift immediate
     if (isVShiftRImm(Op.getOperand(1), VT, false, Cnt) && Cnt < EltSize) {
       unsigned Opc =

 ///      Into:
 ///        %sub.v0 = shuffle <32 x i32> %v0, <32 x i32> v1, <4, 5, 6, 7>
 ///        %sub.v1 = shuffle <32 x i32> %v0, <32 x i32> v1, <32, 33, 34, 35>
 ///        %sub.v2 = shuffle <32 x i32> %v0, <32 x i32> v1, <16, 17, 18, 19>
 ///        call void llvm.aarch64.neon.st3(%sub.v0, %sub.v1, %sub.v2, %ptr)
 bool AArch64TargetLowering::lowerInterleavedStore(StoreInst *SI,
                                                   ShuffleVectorInst *SVI,
                                                   unsigned Factor) const {
   assert(Factor >= 2 && Factor <= getMaxSupportedInterleaveFactor() &&
          "Invalid interleave factor");
 
   auto *VecTy = cast<FixedVectorType>(SVI->getType());
   assert(VecTy->getNumElements() % Factor == 0 && "Invalid interleaved store");
 
   unsigned LaneLen = VecTy->getNumElements() / Factor;
   Type *EltTy = VecTy->getElementType();
   auto *SubVecTy = FixedVectorType::get(EltTy, LaneLen);
static SDValue performSVEAndCombine(SDNode *N,
   if (isConstantSplatVectorMaskForType(Mask.getNode(), MemVT))
     return Src;
 
   return SDValue();
 }
 
 static SDValue performANDCombine(SDNode *N,
                                  TargetLowering::DAGCombinerInfo &DCI) {
   SelectionDAG &DAG = DCI.DAG;
   SDValue LHS = N->getOperand(0);
   SDValue RHS = N->getOperand(1);
   EVT VT = N->getValueType(0);
 
   if (SDValue R = performANDORCSELCombine(N, DAG))
     return R;
 
   if (!DAG.getTargetLoweringInfo().isTypeLegal(VT))
for (const SDValue &V : Op->op_values()) {
 
     // Pass through non-vector operands.
     if (!V.getValueType().isVector()) {
       Ops.push_back(V);
       continue;
     }
 
     // "cast" fixed length vector to a scalable vector.
     assert(useSVEForFixedLengthVectorVT(V.getValueType()) &&
            "Only fixed length vectors are supported!");
     Ops.push_back(convertToScalableVector(DAG, ContainerVT, V));
   }
 
   auto ScalableRes = DAG.getNode(Op.getOpcode(), SDLoc(Op), ContainerVT, Ops);
   return convertFromScalableVector(DAG, VT, ScalableRes);
 }
 
 SDValue AArch64TargetLowering::LowerVECREDUCE_SEQ_FADD(SDValue ScalarOp,

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

def : Pat<(f16 (vector_extract (nxv2f16 ZPR:$vec), sve_elm_idx_extdup_d:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_D ZPR:$vec, sve_elm_idx_extdup_d:$index), hsub)>;
   def : Pat<(f32 (vector_extract (nxv4f32 ZPR:$vec), sve_elm_idx_extdup_s:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_S ZPR:$vec, sve_elm_idx_extdup_s:$index), ssub)>;
   def : Pat<(f32 (vector_extract (nxv2f32 ZPR:$vec), sve_elm_idx_extdup_d:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_D ZPR:$vec, sve_elm_idx_extdup_d:$index), ssub)>;
   def : Pat<(f64 (vector_extract (nxv2f64 ZPR:$vec), sve_elm_idx_extdup_d:$index)),
             (EXTRACT_SUBREG (DUP_ZZI_D ZPR:$vec, sve_elm_idx_extdup_d:$index), dsub)>;
 
   // Extract element from vector with immediate index that's within the bottom 128-bits.
+  let Predicates = [IsStreamingCompatibleSVEDisabled] in {
     let AddedComplexity = 1 in {
     def : Pat<(i32 (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index)),
               (i32 (UMOVvi8 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
     def : Pat<(i32 (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index)),
               (i32 (UMOVvi16 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
     def : Pat<(i32 (vector_extract (nxv4i32 ZPR:$vec), VectorIndexS:$index)),
               (i32 (UMOVvi32 (v4i32 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexS:$index))>;
     def : Pat<(i64 (vector_extract (nxv2i64 ZPR:$vec), VectorIndexD:$index)),
               (i64 (UMOVvi64 (v2i64 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexD:$index))>;
     }
 
     def : Pat<(sext_inreg (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index), i8),
               (i32 (SMOVvi8to32 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
     def : Pat<(sext_inreg (anyext (vector_extract (nxv16i8 ZPR:$vec), VectorIndexB:$index)), i8),
               (i64 (SMOVvi8to64 (v16i8 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexB:$index))>;
 
     def : Pat<(sext_inreg (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index), i16),
               (i32 (SMOVvi16to32 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
     def : Pat<(sext_inreg (anyext (vector_extract (nxv8i16 ZPR:$vec), VectorIndexH:$index)), i16),
               (i64 (SMOVvi16to64 (v8i16 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexH:$index))>;
 
     def : Pat<(sext (vector_extract (nxv4i32 ZPR:$vec), VectorIndexS:$index)),
               (i64 (SMOVvi32to64 (v4i32 (EXTRACT_SUBREG ZPR:$vec, zsub)), VectorIndexS:$index))>;
-
+  }
   // Extract first element from vector.
   let AddedComplexity = 2 in {
   def : Pat<(vector_extract (nxv16i8 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv8i16 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv4i32 ZPR:$Zs), (i64 0)),
             (i32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-ext-loads.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+define <8 x i16> @load_zext_v8i8i16(<8 x i8>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v8i8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #4
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ld1b { z1.h }, p0/z, [x0, x8]
+; CHECK-NEXT:    ld1b { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    splice z0.h, p0, z0.h, z1.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <8 x i8>, <8 x i8>* %ap
+  %val = zext <8 x i8> %a to <8 x i16>
+  ret <8 x i16> %val
+}
+
+define <4 x i32> @load_zext_v4i16i32(<4 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v4i16i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <4 x i16>, <4 x i16>* %ap
+  %val = zext <4 x i16> %a to <4 x i32>
+  ret <4 x i32> %val
+}
+
+define <2 x i64> @load_zext_v2i32i64(<2 x i32>* %ap) #0 {
+; CHECK-LABEL: load_zext_v2i32i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %a = load <2 x i32>, <2 x i32>* %ap
+  %val = zext <2 x i32> %a to <2 x i64>
+  ret <2 x i64> %val
+}
+
+define <2 x i256> @load_zext_v2i64i256(<2 x i64>* %ap) #0 {
+; CHECK-LABEL: load_zext_v2i64i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI3_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI3_0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    mov x1, xzr
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    mov x5, xzr
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x8]
+; CHECK-NEXT:    mov z2.d, z0.d[1]
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    mov z0.d, z1.d[1]
+; CHECK-NEXT:    fmov x2, d1
+; CHECK-NEXT:    fmov x3, d0
+; CHECK-NEXT:    fmov x4, d2
+; CHECK-NEXT:    mov x6, x2
+; CHECK-NEXT:    mov x7, x3
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = zext <2 x i64> %a to <2 x i256>
+  ret <2 x i256> %val
+}
+
+define <16 x i32> @load_sext_v16i8i32(<16 x i8>* %ap)  #0 {
+; CHECK-LABEL: load_sext_v16i8i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x0]
+; CHECK-NEXT:    sunpklo z3.h, z1.b
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    sunpklo z4.h, z1.b
+; CHECK-NEXT:    sunpklo z0.s, z3.h
+; CHECK-NEXT:    ext z3.b, z3.b, z3.b, #8
+; CHECK-NEXT:    sunpklo z2.s, z4.h
+; CHECK-NEXT:    ext z4.b, z4.b, z4.b, #8
+; CHECK-NEXT:    sunpklo z1.s, z3.h
+; CHECK-NEXT:    sunpklo z3.s, z4.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    // kill: def $q2 killed $q2 killed $z2
+; CHECK-NEXT:    // kill: def $q3 killed $q3 killed $z3
+; CHECK-NEXT:    ret
+  %a = load <16 x i8>, <16 x i8>* %ap
+  %val = sext <16 x i8> %a to <16 x i32>
+  ret <16 x i32> %val
+}
+
+define <8 x i32> @load_sext_v8i16i32(<8 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_sext_v8i16i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; CHECK-NEXT:    sunpklo z0.s, z1.h
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    sunpklo z1.s, z1.h
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %val = sext <8 x i16> %a to <8 x i32>
+  ret <8 x i32> %val
+}
+
+define <4 x i256> @load_sext_v4i32i256(<4 x i32>* %ap) #0 {
+; CHECK-LABEL: load_sext_v4i32i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    sunpklo z1.d, z0.s
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    fmov x9, d1
+; CHECK-NEXT:    mov z1.d, z1.d[1]
+; CHECK-NEXT:    fmov x11, d1
+; CHECK-NEXT:    sunpklo z0.d, z0.s
+; CHECK-NEXT:    asr x10, x9, #63
+; CHECK-NEXT:    asr x12, x11, #63
+; CHECK-NEXT:    stp x9, x10, [x8]
+; CHECK-NEXT:    fmov x9, d0
+; CHECK-NEXT:    mov z0.d, z0.d[1]
+; CHECK-NEXT:    stp x11, x12, [x8, #32]
+; CHECK-NEXT:    fmov x11, d0
+; CHECK-NEXT:    stp x10, x10, [x8, #16]
+; CHECK-NEXT:    stp x12, x12, [x8, #48]
+; CHECK-NEXT:    asr x10, x9, #63
+; CHECK-NEXT:    asr x12, x11, #63
+; CHECK-NEXT:    stp x10, x10, [x8, #80]
+; CHECK-NEXT:    stp x9, x10, [x8, #64]
+; CHECK-NEXT:    stp x12, x12, [x8, #112]
+; CHECK-NEXT:    stp x11, x12, [x8, #96]
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = sext <4 x i32> %a to <4 x i256>
+  ret <4 x i256> %val
+}
+
+define <2 x i256> @load_sext_v2i64i256(<2 x i64>* %ap) #0 {
+; CHECK-LABEL: load_sext_v2i64i256:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    mov z1.d, z0.d[1]
+; CHECK-NEXT:    fmov x8, d0
+; CHECK-NEXT:    fmov x9, d1
+; CHECK-NEXT:    asr x8, x8, #63
+; CHECK-NEXT:    asr x9, x9, #63
+; CHECK-NEXT:    fmov d2, x8
+; CHECK-NEXT:    fmov d3, x9
+; CHECK-NEXT:    zip1 z0.d, z0.d, z2.d
+; CHECK-NEXT:    zip1 z2.d, z2.d, z2.d
+; CHECK-NEXT:    zip1 z1.d, z1.d, z3.d
+; CHECK-NEXT:    mov z4.d, z0.d[1]
+; CHECK-NEXT:    fmov x0, d0
+; CHECK-NEXT:    mov z0.d, z2.d[1]
+; CHECK-NEXT:    fmov x2, d2
+; CHECK-NEXT:    mov z2.d, z1.d[1]
+; CHECK-NEXT:    fmov x4, d1
+; CHECK-NEXT:    zip1 z1.d, z3.d, z3.d
+; CHECK-NEXT:    fmov x3, d0
+; CHECK-NEXT:    mov z0.d, z1.d[1]
+; CHECK-NEXT:    fmov x1, d4
+; CHECK-NEXT:    fmov x6, d1
+; CHECK-NEXT:    fmov x5, d2
+; CHECK-NEXT:    fmov x7, d0
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = sext <2 x i64> %a to <2 x i256>
+  ret <2 x i256> %val
+}
+
+define <16 x i64> @load_zext_v16i16i64(<16 x i16>* %ap)  #0 {
+; CHECK-LABEL: load_zext_v16i16i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov x8, #8
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; CHECK-NEXT:    uunpklo z2.s, z0.h
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    uunpklo z3.s, z1.h
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #8
+; CHECK-NEXT:    uunpklo z4.d, z2.s
+; CHECK-NEXT:    ext z2.b, z2.b, z2.b, #8
+; CHECK-NEXT:    uunpklo z7.s, z0.h
+; CHECK-NEXT:    uunpklo z16.s, z1.h
+; CHECK-NEXT:    uunpklo z0.d, z3.s
+; CHECK-NEXT:    ext z3.b, z3.b, z3.b, #8
+; CHECK-NEXT:    uunpklo z5.d, z2.s
+; CHECK-NEXT:    uunpklo z6.d, z7.s
+; CHECK-NEXT:    ext z7.b, z7.b, z7.b, #8
+; CHECK-NEXT:    uunpklo z2.d, z16.s
+; CHECK-NEXT:    ext z16.b, z16.b, z16.b, #8
+; CHECK-NEXT:    uunpklo z1.d, z3.s
+; CHECK-NEXT:    uunpklo z7.d, z7.s
+; CHECK-NEXT:    uunpklo z3.d, z16.s
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    // kill: def $q2 killed $q2 killed $z2
+; CHECK-NEXT:    // kill: def $q3 killed $q3 killed $z3
+; CHECK-NEXT:    // kill: def $q4 killed $q4 killed $z4
+; CHECK-NEXT:    // kill: def $q5 killed $q5 killed $z5
+; CHECK-NEXT:    // kill: def $q6 killed $q6 killed $z6
+; CHECK-NEXT:    // kill: def $q7 killed $q7 killed $z7
+; CHECK-NEXT:    ret
+  %a = load <16 x i16>, <16 x i16>* %ap
+  %val = zext <16 x i16> %a to <16 x i64>
+  ret <16 x i64> %val
+}
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-load.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+;
+; Masked Load
+;
+
+define <4 x i8> @masked_load_v4i8(<4 x i8>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI0_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_0]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1b { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x i8> @llvm.masked.load.v4i8(<4 x i8>* %src, i32 8, <4 x i1> %mask, <4 x i8> zeroinitializer)
+  ret <4 x i8> %load
+}
+
+define <8 x i8> @masked_load_v8i8(<8 x i8>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI1_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl8
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI1_0]
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %src, i32 8, <8 x i1> %mask, <8 x i8> zeroinitializer)
+  ret <8 x i8> %load
+}
+
+define <16 x i8> @masked_load_v16i8(<16 x i8>* %src, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI2_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI2_0
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %src, i32 8, <16 x i1> %mask, <16 x i8> zeroinitializer)
+  ret <16 x i8> %load
+}
+
+define <32 x i8> @masked_load_v32i8(<32 x i8>* %src, <32 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #32
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    ldr w8, [sp, #224]
+; CHECK-NEXT:    strb w7, [sp, #6]
+; CHECK-NEXT:    ldr w9, [sp, #216]
+; CHECK-NEXT:    strb w6, [sp, #5]
+; CHECK-NEXT:    ldr w10, [sp, #208]
+; CHECK-NEXT:    strb w5, [sp, #4]
+; CHECK-NEXT:    strb w8, [sp, #31]
+; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    strb w9, [sp, #30]
+; CHECK-NEXT:    ldr w9, [sp, #192]
+; CHECK-NEXT:    strb w10, [sp, #29]
+; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    strb w8, [sp, #28]
+; CHECK-NEXT:    ldr w8, [sp, #176]
+; CHECK-NEXT:    strb w9, [sp, #27]
+; CHECK-NEXT:    ldr w9, [sp, #168]
+; CHECK-NEXT:    strb w10, [sp, #26]
+; CHECK-NEXT:    ldr w10, [sp, #160]
+; CHECK-NEXT:    strb w8, [sp, #25]
+; CHECK-NEXT:    ldr w8, [sp, #152]
+; CHECK-NEXT:    strb w9, [sp, #24]
+; CHECK-NEXT:    ldr w9, [sp, #144]
+; CHECK-NEXT:    strb w10, [sp, #23]
+; CHECK-NEXT:    ldr w10, [sp, #136]
+; CHECK-NEXT:    strb w8, [sp, #22]
+; CHECK-NEXT:    ldr w8, [sp, #128]
+; CHECK-NEXT:    strb w9, [sp, #21]
+; CHECK-NEXT:    ldr w9, [sp, #120]
+; CHECK-NEXT:    strb w10, [sp, #20]
+; CHECK-NEXT:    ldr w10, [sp, #112]
+; CHECK-NEXT:    strb w8, [sp, #19]
+; CHECK-NEXT:    ldr w8, [sp, #104]
+; CHECK-NEXT:    strb w9, [sp, #18]
+; CHECK-NEXT:    ldr w9, [sp, #96]
+; CHECK-NEXT:    strb w10, [sp, #17]
+; CHECK-NEXT:    ldr w10, [sp, #88]
+; CHECK-NEXT:    strb w8, [sp, #16]
+; CHECK-NEXT:    ldr w8, [sp, #80]
+; CHECK-NEXT:    strb w9, [sp, #15]
+; CHECK-NEXT:    ldr w9, [sp, #72]
+; CHECK-NEXT:    strb w10, [sp, #14]
+; CHECK-NEXT:    ldr w10, [sp, #64]
+; CHECK-NEXT:    strb w8, [sp, #13]
+; CHECK-NEXT:    ldr w8, [sp, #56]
+; CHECK-NEXT:    strb w9, [sp, #12]
+; CHECK-NEXT:    ldr w9, [sp, #48]
+; CHECK-NEXT:    strb w10, [sp, #11]
+; CHECK-NEXT:    ldr w10, [sp, #40]
+; CHECK-NEXT:    strb w8, [sp, #10]
+; CHECK-NEXT:    ldr w8, [sp, #32]
+; CHECK-NEXT:    strb w9, [sp, #9]
+; CHECK-NEXT:    adrp x9, .LCPI3_0
+; CHECK-NEXT:    add x9, x9, :lo12:.LCPI3_0
+; CHECK-NEXT:    strb w10, [sp, #8]
+; CHECK-NEXT:    strb w8, [sp, #7]
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    strb w4, [sp, #3]
+; CHECK-NEXT:    mov x8, sp
+; CHECK-NEXT:    strb w3, [sp, #2]
+; CHECK-NEXT:    strb w2, [sp, #1]
+; CHECK-NEXT:    strb w1, [sp]
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x9]
+; CHECK-NEXT:    add x9, sp, #16
+; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x9]
+; CHECK-NEXT:    ld1b { z2.b }, p0/z, [x8]
+; CHECK-NEXT:    mov w8, #16
+; CHECK-NEXT:    lsl z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    lsl z2.b, p0/m, z2.b, z0.b
+; CHECK-NEXT:    asr z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    asrr z0.b, p0/m, z0.b, z2.b
+; CHECK-NEXT:    cmpne p1.b, p0/z, z1.b, #0
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ld1b { z1.b }, p1/z, [x0, x8]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    add sp, sp, #32
+; CHECK-NEXT:    ret
+  %load = call <32 x i8> @llvm.masked.load.v32i8(<32 x i8>* %src, i32 8, <32 x i1> %mask, <32 x i8> zeroinitializer)
+  ret <32 x i8> %load
+}
+
+define <2 x half> @masked_load_v2f16(<2 x half>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    adrp x8, .LCPI4_0
+; CHECK-NEXT:    str wzr, [sp, #12]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    strh w10, [sp, #10]
+; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI4_0]
+; CHECK-NEXT:    ldr d1, [sp, #8]
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z0.h
+; CHECK-NEXT:    asrr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %load = call <2 x half> @llvm.masked.load.v2f16(<2 x half>* %src, i32 8, <2 x i1> %mask, <2 x half> zeroinitializer)
+  ret <2 x half> %load
+}
+
+define <4 x half> @masked_load_v4f16(<4 x half>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI5_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI5_0]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x half> @llvm.masked.load.v4f16(<4 x half>* %src, i32 8, <4 x i1> %mask, <4 x half> zeroinitializer)
+  ret <4 x half> %load
+}
+
+define <8 x half> @masked_load_v8f16(<8 x half>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI6_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI6_0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <8 x half> @llvm.masked.load.v8f16(<8 x half>* %src, i32 8, <8 x i1> %mask, <8 x half> zeroinitializer)
+  ret <8 x half> %load
+}
+
+define <16 x half> @masked_load_v16f16(<16 x half>* %src, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v16f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI7_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI7_0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    uunpklo z2.h, z0.b
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x8]
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    mov x8, #8
+; CHECK-NEXT:    lsl z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p1.h, p0/z, z2.h, #0
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    ld1h { z0.h }, p1/z, [x0]
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x0, x8, lsl #1]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %load = call <16 x half> @llvm.masked.load.v16f16(<16 x half>* %src, i32 8, <16 x i1> %mask, <16 x half> zeroinitializer)
+  ret <16 x half> %load
+}
+
+define <2 x float> @masked_load_v2f32(<2 x float>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI8_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.s, vl2
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI8_0]
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <2 x float> @llvm.masked.load.v2f32(<2 x float>* %src, i32 8, <2 x i1> %mask, <2 x float> zeroinitializer)
+  ret <2 x float> %load
+}
+
+define <4 x float> @masked_load_v4f32(<4 x float>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI9_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI9_0
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <4 x float> @llvm.masked.load.v4f32(<4 x float>* %src, i32 8, <4 x i1> %mask, <4 x float> zeroinitializer)
+  ret <4 x float> %load
+}
+
+define <8 x float> @masked_load_v8f32(<8 x float>* %src, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v8f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    mov z1.b, z0.b[3]
+; CHECK-NEXT:    mov z2.b, z0.b[2]
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    mov z3.b, z0.b[1]
+; CHECK-NEXT:    mov z4.b, z0.b[7]
+; CHECK-NEXT:    mov z5.b, z0.b[6]
+; CHECK-NEXT:    mov z6.b, z0.b[5]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    mov z0.b, z0.b[4]
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strh w8, [sp, #-16]!
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    fmov w8, s3
+; CHECK-NEXT:    strh w9, [sp, #6]
+; CHECK-NEXT:    fmov w9, s4
+; CHECK-NEXT:    strh w10, [sp, #4]
+; CHECK-NEXT:    fmov w10, s5
+; CHECK-NEXT:    strh w8, [sp, #2]
+; CHECK-NEXT:    fmov w8, s6
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    strh w8, [sp, #10]
+; CHECK-NEXT:    adrp x8, .LCPI10_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI10_0
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x8]
+; CHECK-NEXT:    mov x8, #4
+; CHECK-NEXT:    ldp d0, d2, [sp]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p1/z, [x0]
+; CHECK-NEXT:    asrr z1.s, p0/m, z1.s, z2.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x0, x8, lsl #2]
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %load = call <8 x float> @llvm.masked.load.v8f32(<8 x float>* %src, i32 8, <8 x i1> %mask, <8 x float> zeroinitializer)
+  ret <8 x float> %load
+}
+
+define <2 x double> @masked_load_v2f64(<2 x double>* %src, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI11_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI11_0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    ret
+  %load = call <2 x double> @llvm.masked.load.v2f64(<2 x double>* %src, i32 8, <2 x i1> %mask, <2 x double> zeroinitializer)
+  ret <2 x double> %load
+}
+
+define <4 x double> @masked_load_v4f64(<4 x double>* %src, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_load_v4f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI12_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI12_0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x8]
+; CHECK-NEXT:    uunpklo z2.d, z0.s
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #8
+; CHECK-NEXT:    mov x8, #2
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    lsl z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p1.d, p0/z, z2.d, #0
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    ld1d { z0.d }, p1/z, [x0]
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x0, x8, lsl #3]
+; CHECK-NEXT:    // kill: def $q0 killed $q0 killed $z0
+; CHECK-NEXT:    // kill: def $q1 killed $q1 killed $z1
+; CHECK-NEXT:    ret
+  %load = call <4 x double> @llvm.masked.load.v4f64(<4 x double>* %src, i32 8, <4 x i1> %mask, <4 x double> zeroinitializer)
+  ret <4 x double> %load
+}
+
+declare <4 x i8> @llvm.masked.load.v4i8(<4 x i8>*, i32, <4 x i1>, <4 x i8>)
+declare <8 x i8> @llvm.masked.load.v8i8(<8 x i8>*, i32, <8 x i1>, <8 x i8>)
+declare <16 x i8> @llvm.masked.load.v16i8(<16 x i8>*, i32, <16 x i1>, <16 x i8>)
+declare <32 x i8> @llvm.masked.load.v32i8(<32 x i8>*, i32, <32 x i1>, <32 x i8>)
+
+declare <2 x half> @llvm.masked.load.v2f16(<2 x half>*, i32, <2 x i1>, <2 x half>)
+declare <4 x half> @llvm.masked.load.v4f16(<4 x half>*, i32, <4 x i1>, <4 x half>)
+declare <8 x half> @llvm.masked.load.v8f16(<8 x half>*, i32, <8 x i1>, <8 x half>)
+declare <16 x half> @llvm.masked.load.v16f16(<16 x half>*, i32, <16 x i1>, <16 x half>)
+
+declare <2 x float> @llvm.masked.load.v2f32(<2 x float>*, i32, <2 x i1>, <2 x float>)
+declare <4 x float> @llvm.masked.load.v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
+declare <8 x float> @llvm.masked.load.v8f32(<8 x float>*, i32, <8 x i1>, <8 x float>)
+
+declare <2 x double> @llvm.masked.load.v2f64(<2 x double>*, i32, <2 x i1>, <2 x double>)
+declare <4 x double> @llvm.masked.load.v4f64(<4 x double>*, i32, <4 x i1>, <4 x double>)
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-store.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+;
+; Masked Store
+;
+
+define void @masked_store_v4i8(<4 x i8>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI0_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_0]
+; CHECK-NEXT:    adrp x8, .LCPI0_1
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI0_1]
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1b { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4i8(<4 x i8> zeroinitializer, <4 x i8>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8i8(<8 x i8>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI1_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.b, vl8
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI1_0]
+; CHECK-NEXT:    adrp x8, .LCPI1_1
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI1_1]
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z2.b }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8i8(<8 x i8> zeroinitializer, <8 x i8>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v16i8(<16 x i8>* %dst, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI2_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI2_0
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI2_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI2_1
+; CHECK-NEXT:    ld1b { z2.b }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z2.b }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v16i8(<16 x i8> zeroinitializer, <16 x i8>* %dst, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v32i8(<32 x i8>* %dst, <32 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #32
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    ldr w8, [sp, #96]
+; CHECK-NEXT:    strb w7, [sp, #6]
+; CHECK-NEXT:    ldr w9, [sp, #88]
+; CHECK-NEXT:    strb w6, [sp, #5]
+; CHECK-NEXT:    ldr w10, [sp, #80]
+; CHECK-NEXT:    strb w5, [sp, #4]
+; CHECK-NEXT:    strb w8, [sp, #15]
+; CHECK-NEXT:    ldr w8, [sp, #72]
+; CHECK-NEXT:    strb w9, [sp, #14]
+; CHECK-NEXT:    ldr w9, [sp, #64]
+; CHECK-NEXT:    strb w10, [sp, #13]
+; CHECK-NEXT:    ldr w10, [sp, #56]
+; CHECK-NEXT:    strb w8, [sp, #12]
+; CHECK-NEXT:    ldr w8, [sp, #48]
+; CHECK-NEXT:    strb w9, [sp, #11]
+; CHECK-NEXT:    ldr w9, [sp, #40]
+; CHECK-NEXT:    strb w10, [sp, #10]
+; CHECK-NEXT:    ldr w10, [sp, #32]
+; CHECK-NEXT:    strb w8, [sp, #9]
+; CHECK-NEXT:    ldr w8, [sp, #224]
+; CHECK-NEXT:    strb w9, [sp, #8]
+; CHECK-NEXT:    ldr w9, [sp, #216]
+; CHECK-NEXT:    strb w10, [sp, #7]
+; CHECK-NEXT:    ldr w10, [sp, #208]
+; CHECK-NEXT:    strb w8, [sp, #31]
+; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    strb w9, [sp, #30]
+; CHECK-NEXT:    ldr w9, [sp, #192]
+; CHECK-NEXT:    strb w10, [sp, #29]
+; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    strb w8, [sp, #28]
+; CHECK-NEXT:    ldr w8, [sp, #176]
+; CHECK-NEXT:    strb w9, [sp, #27]
+; CHECK-NEXT:    ldr w9, [sp, #168]
+; CHECK-NEXT:    strb w10, [sp, #26]
+; CHECK-NEXT:    ldr w10, [sp, #160]
+; CHECK-NEXT:    strb w8, [sp, #25]
+; CHECK-NEXT:    ldr w8, [sp, #152]
+; CHECK-NEXT:    strb w9, [sp, #24]
+; CHECK-NEXT:    ldr w9, [sp, #144]
+; CHECK-NEXT:    strb w10, [sp, #23]
+; CHECK-NEXT:    ldr w10, [sp, #136]
+; CHECK-NEXT:    strb w8, [sp, #22]
+; CHECK-NEXT:    ldr w8, [sp, #128]
+; CHECK-NEXT:    strb w9, [sp, #21]
+; CHECK-NEXT:    ldr w9, [sp, #120]
+; CHECK-NEXT:    strb w10, [sp, #20]
+; CHECK-NEXT:    ldr w10, [sp, #112]
+; CHECK-NEXT:    strb w8, [sp, #19]
+; CHECK-NEXT:    ldr w8, [sp, #104]
+; CHECK-NEXT:    strb w4, [sp, #3]
+; CHECK-NEXT:    ptrue p0.b, vl16
+; CHECK-NEXT:    strb w3, [sp, #2]
+; CHECK-NEXT:    strb w8, [sp, #16]
+; CHECK-NEXT:    adrp x8, .LCPI3_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI3_0
+; CHECK-NEXT:    strb w2, [sp, #1]
+; CHECK-NEXT:    strb w1, [sp]
+; CHECK-NEXT:    strb w9, [sp, #18]
+; CHECK-NEXT:    mov x9, sp
+; CHECK-NEXT:    strb w10, [sp, #17]
+; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x8]
+; CHECK-NEXT:    add x8, sp, #16
+; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x9]
+; CHECK-NEXT:    ld1b { z2.b }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI3_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI3_1
+; CHECK-NEXT:    lsl z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    asr z1.b, p0/m, z1.b, z0.b
+; CHECK-NEXT:    ld1b { z3.b }, p0/z, [x8]
+; CHECK-NEXT:    lsl z2.b, p0/m, z2.b, z0.b
+; CHECK-NEXT:    mov w8, #16
+; CHECK-NEXT:    asrr z0.b, p0/m, z0.b, z2.b
+; CHECK-NEXT:    cmpne p1.b, p0/z, z1.b, #0
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    st1b { z3.b }, p0, [x0, x8]
+; CHECK-NEXT:    st1b { z3.b }, p1, [x0]
+; CHECK-NEXT:    add sp, sp, #32
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v32i8(<32 x i8> zeroinitializer, <32 x i8>* %dst, i32 8, <32 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v2f16(<2 x half>* %dst, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v2f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    adrp x8, .LCPI4_0
+; CHECK-NEXT:    str wzr, [sp, #12]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    strh w9, [sp, #8]
+; CHECK-NEXT:    strh w10, [sp, #10]
+; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI4_0]
+; CHECK-NEXT:    ldr d1, [sp, #8]
+; CHECK-NEXT:    adrp x8, .LCPI4_1
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z0.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI4_1]
+; CHECK-NEXT:    asrr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v2f16(<2 x half> zeroinitializer, <2 x half>* %dst, i32 8, <2 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f16(<4 x half>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI5_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI5_0]
+; CHECK-NEXT:    adrp x8, .LCPI5_1
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI5_1]
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f16(<4 x half> zeroinitializer, <4 x half>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8f16(<8 x half>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI6_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI6_0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI6_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI6_1
+; CHECK-NEXT:    ld1h { z2.h }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z2.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8f16(<8 x half> zeroinitializer, <8 x half>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v16f16(<16 x half>* %dst, <16 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v16f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI7_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI7_0
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    // kill: def $q0 killed $q0 def $z0
+; CHECK-NEXT:    mov z1.d, z0.d
+; CHECK-NEXT:    ext z1.b, z1.b, z0.b, #8
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    ld1h { z2.h }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI7_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI7_1
+; CHECK-NEXT:    uunpklo z1.h, z1.b
+; CHECK-NEXT:    ld1h { z3.h }, p0/z, [x8]
+; CHECK-NEXT:    lsl z1.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    mov x8, #8
+; CHECK-NEXT:    lsl z0.h, p0/m, z0.h, z2.h
+; CHECK-NEXT:    asr z1.h, p0/m, z1.h, z2.h
+; CHECK-NEXT:    asr z0.h, p0/m, z0.h, z2.h
+; CHECK-NEXT:    cmpne p1.h, p0/z, z1.h, #0
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    st1h { z3.h }, p1, [x0, x8, lsl #1]
+; CHECK-NEXT:    st1h { z3.h }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v16f16(<16 x half> zeroinitializer, <16 x half>* %dst, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f32(<4 x float>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI8_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI8_0
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI8_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI8_1
+; CHECK-NEXT:    ld1w { z2.s }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z2.s }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f32(<4 x float> zeroinitializer, <4 x float>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8f32(<8 x float>* %dst, <8 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v8f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    mov z1.b, z0.b[7]
+; CHECK-NEXT:    mov z2.b, z0.b[6]
+; CHECK-NEXT:    fmov w8, s1
+; CHECK-NEXT:    fmov w9, s2
+; CHECK-NEXT:    mov z2.b, z0.b[4]
+; CHECK-NEXT:    mov z1.b, z0.b[5]
+; CHECK-NEXT:    fmov w10, s1
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    strh w8, [sp, #14]
+; CHECK-NEXT:    fmov w8, s2
+; CHECK-NEXT:    strh w9, [sp, #12]
+; CHECK-NEXT:    mov x9, #4
+; CHECK-NEXT:    strh w10, [sp, #10]
+; CHECK-NEXT:    mov z3.b, z0.b[3]
+; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    adrp x8, .LCPI9_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI9_0
+; CHECK-NEXT:    mov z5.b, z0.b[2]
+; CHECK-NEXT:    mov z6.b, z0.b[1]
+; CHECK-NEXT:    fmov w10, s3
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI9_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI9_1
+; CHECK-NEXT:    ldr d2, [sp, #8]
+; CHECK-NEXT:    ld1w { z4.s }, p0/z, [x8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT:    movprfx z0, z2
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z4.s }, p1, [x0, x9, lsl #2]
+; CHECK-NEXT:    fmov w9, s5
+; CHECK-NEXT:    strh w8, [sp]
+; CHECK-NEXT:    fmov w8, s6
+; CHECK-NEXT:    strh w10, [sp, #6]
+; CHECK-NEXT:    strh w9, [sp, #4]
+; CHECK-NEXT:    strh w8, [sp, #2]
+; CHECK-NEXT:    ldr d0, [sp]
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    lsl z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z4.s }, p0, [x0]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v8f32(<8 x float> zeroinitializer, <8 x float>* %dst, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v32f32(<32 x float>* %dst, <32 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v32f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #64
+; CHECK-NEXT:    .cfi_def_cfa_offset 64
+; CHECK-NEXT:    ldr w9, [sp, #248]
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    ldr w8, [sp, #256]
+; CHECK-NEXT:    ldr w10, [sp, #240]
+; CHECK-NEXT:    ldr w11, [sp, #232]
+; CHECK-NEXT:    strh w9, [sp, #60]
+; CHECK-NEXT:    adrp x9, .LCPI10_0
+; CHECK-NEXT:    add x9, x9, :lo12:.LCPI10_0
+; CHECK-NEXT:    strh w8, [sp, #62]
+; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    strh w10, [sp, #58]
+; CHECK-NEXT:    adrp x10, .LCPI10_1
+; CHECK-NEXT:    add x10, x10, :lo12:.LCPI10_1
+; CHECK-NEXT:    strh w11, [sp, #56]
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x9]
+; CHECK-NEXT:    ldr d1, [sp, #56]
+; CHECK-NEXT:    ldr w9, [sp, #224]
+; CHECK-NEXT:    ldr w11, [sp, #216]
+; CHECK-NEXT:    uunpklo z2.s, z1.h
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x10]
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    mov x10, #28
+; CHECK-NEXT:    ldr w12, [sp, #208]
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x10, lsl #2]
+; CHECK-NEXT:    strh w9, [sp, #54]
+; CHECK-NEXT:    strh w11, [sp, #52]
+; CHECK-NEXT:    ldr w9, [sp, #192]
+; CHECK-NEXT:    strh w12, [sp, #50]
+; CHECK-NEXT:    mov x10, #24
+; CHECK-NEXT:    strh w8, [sp, #48]
+; CHECK-NEXT:    ldr w8, [sp, #168]
+; CHECK-NEXT:    ldr d2, [sp, #48]
+; CHECK-NEXT:    ldr w11, [sp, #184]
+; CHECK-NEXT:    ldr w12, [sp, #176]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x10, lsl #2]
+; CHECK-NEXT:    strh w9, [sp, #46]
+; CHECK-NEXT:    strh w11, [sp, #44]
+; CHECK-NEXT:    ldr w9, [sp, #160]
+; CHECK-NEXT:    strh w12, [sp, #42]
+; CHECK-NEXT:    mov x10, #20
+; CHECK-NEXT:    strh w8, [sp, #40]
+; CHECK-NEXT:    ldr w8, [sp, #136]
+; CHECK-NEXT:    ldr d2, [sp, #40]
+; CHECK-NEXT:    ldr w11, [sp, #152]
+; CHECK-NEXT:    ldr w12, [sp, #144]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x10, lsl #2]
+; CHECK-NEXT:    strh w9, [sp, #38]
+; CHECK-NEXT:    strh w11, [sp, #36]
+; CHECK-NEXT:    ldr w9, [sp, #128]
+; CHECK-NEXT:    strh w12, [sp, #34]
+; CHECK-NEXT:    mov x10, #16
+; CHECK-NEXT:    strh w8, [sp, #32]
+; CHECK-NEXT:    ldr w8, [sp, #104]
+; CHECK-NEXT:    ldr d2, [sp, #32]
+; CHECK-NEXT:    ldr w11, [sp, #120]
+; CHECK-NEXT:    ldr w12, [sp, #112]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x10, lsl #2]
+; CHECK-NEXT:    strh w9, [sp, #30]
+; CHECK-NEXT:    strh w11, [sp, #28]
+; CHECK-NEXT:    ldr w9, [sp, #96]
+; CHECK-NEXT:    strh w12, [sp, #26]
+; CHECK-NEXT:    mov x10, #12
+; CHECK-NEXT:    strh w8, [sp, #24]
+; CHECK-NEXT:    ldr w8, [sp, #72]
+; CHECK-NEXT:    ldr d2, [sp, #24]
+; CHECK-NEXT:    ldr w11, [sp, #88]
+; CHECK-NEXT:    ldr w12, [sp, #80]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x10, lsl #2]
+; CHECK-NEXT:    strh w9, [sp, #22]
+; CHECK-NEXT:    strh w11, [sp, #20]
+; CHECK-NEXT:    mov x9, #8
+; CHECK-NEXT:    strh w12, [sp, #18]
+; CHECK-NEXT:    strh w8, [sp, #16]
+; CHECK-NEXT:    ldr w8, [sp, #64]
+; CHECK-NEXT:    ldr d2, [sp, #16]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x9, lsl #2]
+; CHECK-NEXT:    strh w8, [sp, #14]
+; CHECK-NEXT:    strh w7, [sp, #12]
+; CHECK-NEXT:    mov x8, #4
+; CHECK-NEXT:    strh w6, [sp, #10]
+; CHECK-NEXT:    strh w5, [sp, #8]
+; CHECK-NEXT:    ldr d2, [sp, #8]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asr z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    cmpne p1.s, p0/z, z2.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p1, [x0, x8, lsl #2]
+; CHECK-NEXT:    strh w4, [sp, #6]
+; CHECK-NEXT:    strh w3, [sp, #4]
+; CHECK-NEXT:    strh w2, [sp, #2]
+; CHECK-NEXT:    strh w1, [sp]
+; CHECK-NEXT:    ldr d2, [sp]
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    lsl z2.s, p0/m, z2.s, z0.s
+; CHECK-NEXT:    asrr z0.s, p0/m, z0.s, z2.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    st1w { z1.s }, p0, [x0]
+; CHECK-NEXT:    add sp, sp, #64
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v32f32(<32 x float> zeroinitializer, <32 x float>* %dst, i32 8, <32 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v2f64(<2 x double>* %dst, <2 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI11_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI11_0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI11_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI11_1
+; CHECK-NEXT:    ld1d { z2.d }, p0/z, [x8]
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    st1d { z2.d }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v2f64(<2 x double> zeroinitializer, <2 x double>* %dst, i32 8, <2 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f64(<4 x double>* %dst, <4 x i1> %mask) #0 {
+; CHECK-LABEL: masked_store_v4f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    adrp x8, .LCPI12_0
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI12_0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    ld1d { z2.d }, p0/z, [x8]
+; CHECK-NEXT:    adrp x8, .LCPI12_1
+; CHECK-NEXT:    add x8, x8, :lo12:.LCPI12_1
+; CHECK-NEXT:    mov z1.d, z0.d
+; CHECK-NEXT:    ext z1.b, z1.b, z0.b, #8
+; CHECK-NEXT:    uunpklo z0.d, z0.s
+; CHECK-NEXT:    uunpklo z1.d, z1.s
+; CHECK-NEXT:    ld1d { z3.d }, p0/z, [x8]
+; CHECK-NEXT:    lsl z1.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    mov x8, #2
+; CHECK-NEXT:    lsl z0.d, p0/m, z0.d, z2.d
+; CHECK-NEXT:    asr z1.d, p0/m, z1.d, z2.d
+; CHECK-NEXT:    asr z0.d, p0/m, z0.d, z2.d
+; CHECK-NEXT:    cmpne p1.d, p0/z, z1.d, #0
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    st1d { z3.d }, p1, [x0, x8, lsl #3]
+; CHECK-NEXT:    st1d { z3.d }, p0, [x0]
+; CHECK-NEXT:    ret
+  call void @llvm.masked.store.v4f64(<4 x double> zeroinitializer, <4 x double>* %dst, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+declare void @llvm.masked.store.v4i8(<4 x i8>, <4 x i8>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8i8(<8 x i8>, <8 x i8>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16i8(<16 x i8>, <16 x i8>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v32i8(<32 x i8>, <32 x i8>*, i32, <32 x i1>)
+declare void @llvm.masked.store.v2f16(<2 x half>, <2 x half>*, i32, <2 x i1>)
+declare void @llvm.masked.store.v4f16(<4 x half>, <4 x half>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8f16(<8 x half>, <8 x half>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16f16(<16 x half>, <16 x half>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v4f32(<4 x float>, <4 x float>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8f32(<8 x float>, <8 x float>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v32f32(<32 x float>, <32 x float>*, i32, <32 x i1>)
+declare void @llvm.masked.store.v2f64(<2 x double>, <2 x double>*, i32, <2 x i1>)
+declare void @llvm.masked.store.v4f64(<4 x double>, <4 x double>*, i32, <4 x i1>)
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-stores.ll

; CHECK-NEXT:    ret
   ret void
 }
 
 define void @store_v2f16(<2 x half>* %a) #0 {
 ; CHECK-LABEL: store_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI5_0
 ; CHECK-NEXT:    ldr d0, [x8, :lo12:.LCPI5_0]
-; CHECK-NEXT:    str s0, [x0]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    str w8, [x0]
 ; CHECK-NEXT:    ret
   store <2 x half> zeroinitializer, <2 x half>* %a
   ret void
 }
 
 define void @store_v4i16(<4 x i16>* %a) #0 {
 ; CHECK-LABEL: store_v4i16:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-trunc-stores.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -force-streaming-compatible-sve < %s | FileCheck %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+define void @store_trunc_v8i16i8(<8 x i16>* %ap, <8 x i8>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v8i16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ptrue p0.h, vl8
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    mov z1.h, z0.h[7]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    mov z2.h, z0.h[6]
+; CHECK-NEXT:    mov z3.h, z0.h[5]
+; CHECK-NEXT:    mov z4.h, z0.h[4]
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strb w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s3
+; CHECK-NEXT:    strb w9, [sp, #15]
+; CHECK-NEXT:    fmov w9, s4
+; CHECK-NEXT:    mov z5.h, z0.h[3]
+; CHECK-NEXT:    mov z6.h, z0.h[2]
+; CHECK-NEXT:    mov z0.h, z0.h[1]
+; CHECK-NEXT:    strb w10, [sp, #14]
+; CHECK-NEXT:    fmov w10, s5
+; CHECK-NEXT:    strb w8, [sp, #13]
+; CHECK-NEXT:    fmov w8, s6
+; CHECK-NEXT:    strb w9, [sp, #12]
+; CHECK-NEXT:    fmov w9, s0
+; CHECK-NEXT:    strb w10, [sp, #11]
+; CHECK-NEXT:    strb w8, [sp, #10]
+; CHECK-NEXT:    strb w9, [sp, #9]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    str d0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %val = trunc <8 x i16> %a to <8 x i8>
+  store <8 x i8> %val, <8 x i8>* %dest
+  ret void
+}
+
+define void @store_trunc_v4i32i8(<4 x i32>* %ap, <4 x i8>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v4i32i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ptrue p0.h, vl4
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    mov z1.s, z0.s[3]
+; CHECK-NEXT:    mov z2.s, z0.s[2]
+; CHECK-NEXT:    mov z0.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    strh w8, [sp, #10]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    st1b { z0.h }, p0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = trunc <4 x i32> %a to <4 x i8>
+  store <4 x i8> %val, <4 x i8>* %dest
+  ret void
+}
+
+define void @store_trunc_v4i32i16(<4 x i32>* %ap, <4 x i16>* %dest) #0 {
+; CHECK-LABEL: store_trunc_v4i32i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    ptrue p0.s, vl4
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    mov z1.s, z0.s[3]
+; CHECK-NEXT:    mov z2.s, z0.s[2]
+; CHECK-NEXT:    mov z0.s, z0.s[1]
+; CHECK-NEXT:    fmov w9, s1
+; CHECK-NEXT:    fmov w10, s2
+; CHECK-NEXT:    strh w8, [sp, #8]
+; CHECK-NEXT:    fmov w8, s0
+; CHECK-NEXT:    strh w9, [sp, #14]
+; CHECK-NEXT:    strh w10, [sp, #12]
+; CHECK-NEXT:    strh w8, [sp, #10]
+; CHECK-NEXT:    ldr d0, [sp, #8]
+; CHECK-NEXT:    str d0, [x1]
+; CHECK-NEXT:    add sp, sp, #16
+; CHECK-NEXT:    ret
+  %a = load <4 x i32>, <4 x i32>* %ap
+  %val = trunc <4 x i32> %a to <4 x i16>
+  store <4 x i16> %val, <4 x i16>* %dest
+  ret void
+}
+
+define void @store_trunc_v2i64i8(<2 x i64>* %ap, <2 x i32>* %dest) vscale_range(2,0) #0 {
+; CHECK-LABEL: store_trunc_v2i64i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d, vl2
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    st1w { z0.d }, p0, [x1]
+; CHECK-NEXT:    ret
+  %a = load <2 x i64>, <2 x i64>* %ap
+  %val = trunc <2 x i64> %a to <2 x i32>
+  store <2 x i32> %val, <2 x i32>* %dest
+  ret void
+}
+
+define void @store_trunc_v2i256i64(<2 x i256>* %ap, <2 x i64>* %dest) vscale_range(2,0) #0 {
+; CHECK-LABEL: store_trunc_v2i256i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0, #32]
+; CHECK-NEXT:    ptrue p0.d, vl1
+; CHECK-NEXT:    ldr d1, [x0]
+; CHECK-NEXT:    splice z1.d, p0, z1.d, z0.d
+; CHECK-NEXT:    str q1, [x1]
+; CHECK-NEXT:    ret
+  %a = load <2 x i256>, <2 x i256>* %ap
+  %val = trunc <2 x i256> %a to <2 x i64>
+  store <2 x i64> %val, <2 x i64>* %dest
+  ret void
+}
+
+attributes #0 = { "target-features"="+sve" }