[llvm][AArch64][SVE] Fold literals into math instructions

[DavidTruby]

Summary

SVE has predicated literal forms of some instructions for specific
literals, which currently are generated correctly when using ACLE
but not when those instructions are generated directly.

This adds the patterns to generate those instructions when
generating from standard LLVM IR instructions.

Co-authored-by: Sander De Smalen <[e-mail]>
Co-authored-by: Cullen Rhodes <[e-mail]>

[david-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2247 ↗

(On Diff #332264)

Hi @DavidTruby, this is quite a lot of very similar looking patterns to maintain here. Is there a possible way to condense them somehow, i.e. the A and B patterns are duplicates, so you could invoke a multiclass for A, then a multiclass for B?

[DavidTruby]

Added extra tests for fmax, fmaxnm, fmin, fminnm

[DavidTruby]

Refactored patterns into helper multiclass

[DavidTruby]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2247 ↗

(On Diff #332264)

@david-arm hopefully this helps a bit, it at least means that each pattern is only mentioned once so only need changing once if they have to be changed in future

[joechrisellis]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

1209–1219 ↗

(On Diff #333860)

nit: really small thing but the naming here is inconsistent with fpimm0. I would expect fpimm1 and fpimm2. I suppose the difficulty here is shoehoring fpimm_half into this naming convention. If you can think of a sensible way of doing this, I would prefer we change this. Otherwise, ignore me. :-)

llvm/test/CodeGen/AArch64/sve-fp-immediates-merging.ll

1 ↗

(On Diff #333860)

General test suggestion -- prefer:

target triple = "aarch64-unknown-linux-gnu"

...

attributes #0 = { "target-features"="+sve" }

... and tag your functions with #0. I personally find this to be a bit more reliable when you're running llc on this file. 😄

Also might be worth adding:

; RUN: ... 2>%t ...
; RUN: FileCheck --check-prefix=WARN --allow-empty %s <%t

2 ↗

(On Diff #333860)

Please can we add:

; If this check fails please read test/CodeGen/AArch64/README for instructions on how to resolve it.
; WARN-NOT: warning

[DavidTruby]

Moved the zeroing predicates behind the UseExperimentalZeroingPseudos feature
Added additional tests for one of the patterns
Addressed misc reivew comments

[DavidTruby]

ping

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

415–422 ↗

(On Diff #335785)

These will need to be split in two because the the UNDEF forms are safe to use by default but the ZERO forms must be protected by UseExperimentalZeroingPseudos. You can see how we've done this for FADD_ZPZZ where there exists two multiclasses, sve_fp_bin_pred_hfd and sve_fp_2op_p_zds_zeroing_hsd.

2247 ↗

(On Diff #332264)

My immediate thought is whether these (or at least some of these) be rolled into the instruction/pseudo multiclasses. By this I mean can sve_fp_2op_i_p_zds be extended with extra parameters so the necessary intrinsic patterns can be added.

Likewise for the multiclasses used to create the UNDEF and ZERO psuedos, can the multiclasses themselves contain the IR patterns plus the vselect variants.

llvm/test/CodeGen/AArch64/sve-fp-immediates-merging.ll

5–6 ↗

(On Diff #335785)

Minor point but attributes are normally placed after the function definitions as that where the IR printer would put them.

14–15 ↗

(On Diff #335785)

Please use update_llc_test_checks.py to generate the CHECK lines.

2 ↗

(On Diff #333860)

Just a note to say this and the above RUN: ... 2>%t request are no longer necessary because llc will now error in these circumstances.

[DavidTruby]

Separated zeroing forms and regenreated check lines

[DavidTruby]

This comment was removed by DavidTruby.

[DavidTruby]

Fixed accidentally overwriting this with a different patch...

[DavidTruby]

Merge multiclasses into SVEInstrFormats.td

[DavidTruby]

@paulwalker-arm sorry for the long lag on this; is this new update more like what you wanted to see?

[bsmith]

Would be good to be more consistent with the existing classes in here, for example the classes with patterns in are mostly near the top of the file with specific naming SVE_ style.

llvm/lib/Target/AArch64/SVEInstrFormats.td

8407 ↗

(On Diff #374180)

This doesn't really need to be a separate mutliclass as it is only instantiated once, similar with the other classes.

[DavidTruby]

Merge additional multiclasses

[paulwalker-arm]

llvm/lib/Target/AArch64/SVEInstrFormats.td

1820–1823 ↗

(On Diff #377223)

Assuming my other comments are valid I believe we won't need a multiclass and can then follow the same idiom as with class SVE_2_Op_Pred... (i.e. add new classes in the SVE pattern match helpers section towards the start of this file.

1824–1827 ↗

(On Diff #377223)

This pattern seems like a better fit for the sve_fp_2op_i_p_zds multiclass where the relevant instructions are defined. Is that possible?

1828–1831 ↗

(On Diff #377223)

Is this pattern necessary? I'm thinking that even for the IR ops you can just pass the predicate through and thus we can reuse the first pattern. Doing this means we'll also take advantage of the immediate forms when lowering VLS operations.

1835 ↗

(On Diff #377223)

There's a bit of clash with naming here. In the past _zx implied zeroing where as we now typically attach element types info for the undef case (i.e. _hfd in this instance) and then zeroing_hfd for the zeroing case.

1859 ↗

(On Diff #377223)

Perhaps you don't need this parameter. Typically you can reference NAME which is the string after the defm when used. So for example you have:

defm FADD_ZPZI    : sve_fp_2op_i_p_zds_zx_zeroing

so NAME will be FADD_ZPZI and thus you can then do:

defm : sve_intrinsic_compact_fp_immediates_zeroing<NAME # "_ZERO_H",

This suggestion likely applies to sve_intrinsic_compact_fp_immediates also, especially if the second pattern can be removed.

[DavidTruby]

Further cleanup of patterns

[DavidTruby]

Hopefully this is going broadly in the direction you wanted @paulwalker-arm I just had a couple of questions about your comments

llvm/lib/Target/AArch64/SVEInstrFormats.td

1828–1831 ↗

(On Diff #377223)

I'm not sure I understand fully; I've managed to reuse this pattern for both the op and ir_op but it definitely seems to still be necessary to have the ir_op version there. I might be wrong about what you meant though?

1835 ↗

(On Diff #377223)

Hopefully the new names are more what you're expecting? Let me know if not!

[DavidTruby]

Further refactor of patterns

[paulwalker-arm]

Structurally this is looking great. There is just a few missing patterns after which I think you're done.

llvm/lib/Target/AArch64/SVEInstrFormats.td

498 ↗

(On Diff #379384)

Does the inst line need the PPR_3b: and ZPR: parts? I ask because the next class you've added seems to work without them.

1878–1883 ↗

(On Diff #379384)

For this class, which deals the the normal IR nodes (i.e. not intrinsics), can you add entries for all the legal floating point types (i.e. add entries for nxv2f16, nxv4f16 & nxv2f32) along with suitable tests.

llvm/test/CodeGen/AArch64/sve-fp-immediates-merging.ll

22–33 ↗

(On Diff #379384)

Sorry to be a pain but can you move the intrinsic tests into a separate file. This is what we've done for some of the other ones, for example there's sve-intrinsics-int-arith.ll and then sve-intrinsics-int-arith-imm.ll. So can you move them into sve-intrinsics-fp-arith-imm.ll leaving this file to just contain the stock IR tests.

[DavidTruby]

Add extra patterns and tests for additional legal types

llvm/lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerSVEStructLoad(unsigned Intrinsic, ArrayRef<SDValue> LoadOps,
                              EVT VT, SelectionDAG &DAG, const SDLoc &DL) const;
 
   SDValue LowerFixedLengthVectorIntDivideToSVE(SDValue Op,
                                                SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorIntExtendToSVE(SDValue Op,
                                                SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorLoadToSVE(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerFixedLengthVectorMLoadToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECREDUCE_SEQ_FADD(SDValue ScalarOp, SelectionDAG &DAG) const;
   SDValue LowerPredReductionToSVE(SDValue ScalarOp, SelectionDAG &DAG) const;
   SDValue LowerReductionToSVE(unsigned Opcode, SDValue ScalarOp,
                               SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorSelectToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorSetccToSVE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorStoreToSVE(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerFixedLengthVectorMStoreToSVE(SDValue Op,
+                                            SelectionDAG &DAG) const;
   SDValue LowerFixedLengthVectorTruncateToSVE(SDValue Op,
                                               SelectionDAG &DAG) const;
   SDValue LowerFixedLengthExtractVectorElt(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFixedLengthInsertVectorElt(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                         SmallVectorImpl<SDNode *> &Created) const override;
   SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->hasNEON()) {
 
     if (Subtarget->hasSVE())
       setOperationAction(ISD::VSCALE, MVT::i32, Custom);
 
     setTruncStoreAction(MVT::v4i16, MVT::v4i8, Custom);
   }
 
   if (Subtarget->hasSVE()) {
-    // FIXME: Add custom lowering of MLOAD to handle different passthrus (not a
-    // splat of 0 or undef) once vector selects supported in SVE codegen. See
-    // D68877 for more details.
     for (auto VT : {MVT::nxv16i8, MVT::nxv8i16, MVT::nxv4i32, MVT::nxv2i64}) {
       setOperationAction(ISD::BITREVERSE, VT, Custom);
       setOperationAction(ISD::BSWAP, VT, Custom);
       setOperationAction(ISD::CTLZ, VT, Custom);
       setOperationAction(ISD::CTPOP, VT, Custom);
       setOperationAction(ISD::CTTZ, VT, Custom);
       setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);
       setOperationAction(ISD::UINT_TO_FP, VT, Custom);
for (auto VT : {MVT::nxv16i1, MVT::nxv8i1, MVT::nxv4i1, MVT::nxv2i1}) {
 
       setOperationAction(ISD::SELECT_CC, VT, Expand);
 
       // There are no legal MVT::nxv16f## based types.
       if (VT != MVT::nxv16i1) {
         setOperationAction(ISD::SINT_TO_FP, VT, Custom);
         setOperationAction(ISD::UINT_TO_FP, VT, Custom);
       }
+
+      // NEON doesn't support masked loads or stores, but SVE does
+      for (auto VT :
+           {MVT::v4f16, MVT::v8f16, MVT::v2f32, MVT::v4f32, MVT::v1f64,
+            MVT::v2f64, MVT::v8i8, MVT::v16i8, MVT::v4i16, MVT::v8i16,
+            MVT::v2i32, MVT::v4i32, MVT::v1i64, MVT::v2i64}) {
+        setOperationAction(ISD::MLOAD, VT, Custom);
+        setOperationAction(ISD::MSTORE, VT, Custom);
+      }
     }
 
     for (auto VT : {MVT::nxv2f16, MVT::nxv4f16, MVT::nxv8f16, MVT::nxv2f32,
                     MVT::nxv4f32, MVT::nxv2f64}) {
       for (auto InnerVT : {MVT::nxv2f16, MVT::nxv4f16, MVT::nxv8f16,
                            MVT::nxv2f32, MVT::nxv4f32, MVT::nxv2f64}) {
         // Avoid marking truncating FP stores as legal to prevent the
         // DAGCombiner from creating unsupported truncating stores.
void AArch64TargetLowering::addTypeForFixedLengthSVE(MVT VT) {
   setOperationAction(ISD::FNEG, VT, Custom);
   setOperationAction(ISD::FRINT, VT, Custom);
   setOperationAction(ISD::FROUND, VT, Custom);
   setOperationAction(ISD::FROUNDEVEN, VT, Custom);
   setOperationAction(ISD::FSQRT, VT, Custom);
   setOperationAction(ISD::FSUB, VT, Custom);
   setOperationAction(ISD::FTRUNC, VT, Custom);
   setOperationAction(ISD::LOAD, VT, Custom);
+  setOperationAction(ISD::MLOAD, VT, Custom);
+  setOperationAction(ISD::MSTORE, VT, Custom);
   setOperationAction(ISD::MUL, VT, Custom);
   setOperationAction(ISD::MULHS, VT, Custom);
   setOperationAction(ISD::MULHU, VT, Custom);
   setOperationAction(ISD::OR, VT, Custom);
   setOperationAction(ISD::SDIV, VT, Custom);
   setOperationAction(ISD::SELECT, VT, Custom);
   setOperationAction(ISD::SETCC, VT, Custom);
   setOperationAction(ISD::SHL, VT, Custom);
case ISD::FRAMEADDR:
     return LowerFRAMEADDR(Op, DAG);
   case ISD::SPONENTRY:
     return LowerSPONENTRY(Op, DAG);
   case ISD::RETURNADDR:
     return LowerRETURNADDR(Op, DAG);
   case ISD::ADDROFRETURNADDR:
     return LowerADDROFRETURNADDR(Op, DAG);
   case ISD::CONCAT_VECTORS:
+
     return LowerCONCAT_VECTORS(Op, DAG);
   case ISD::INSERT_VECTOR_ELT:
     return LowerINSERT_VECTOR_ELT(Op, DAG);
   case ISD::EXTRACT_VECTOR_ELT:
     return LowerEXTRACT_VECTOR_ELT(Op, DAG);
   case ISD::BUILD_VECTOR:
     return LowerBUILD_VECTOR(Op, DAG);
   case ISD::VECTOR_SHUFFLE:
return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHS_PRED,
                                /*OverrideNEON=*/true);
   case ISD::MULHU:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::MULHU_PRED,
                                /*OverrideNEON=*/true);
   case ISD::INTRINSIC_WO_CHAIN:
     return LowerINTRINSIC_WO_CHAIN(Op, DAG);
   case ISD::STORE:
     return LowerSTORE(Op, DAG);
+  case ISD::MSTORE:
+    return LowerFixedLengthVectorMStoreToSVE(Op, DAG);
   case ISD::MGATHER:
     return LowerMGATHER(Op, DAG);
   case ISD::MSCATTER:
     return LowerMSCATTER(Op, DAG);
   case ISD::VECREDUCE_SEQ_FADD:
     return LowerVECREDUCE_SEQ_FADD(Op, DAG);
   case ISD::VECREDUCE_ADD:
   case ISD::VECREDUCE_AND:
if ((ExtraEltVT != MVT::i8) && (ExtraEltVT != MVT::i16) &&
         (ExtraEltVT != MVT::i32) && (ExtraEltVT != MVT::i64))
       return SDValue();
 
     return LowerToPredicatedOp(Op, DAG,
                                AArch64ISD::SIGN_EXTEND_INREG_MERGE_PASSTHRU);
   }
   case ISD::TRUNCATE:
     return LowerTRUNCATE(Op, DAG);
+  case ISD::MLOAD:
+    return LowerFixedLengthVectorMLoadToSVE(Op, DAG);
   case ISD::LOAD:
     if (useSVEForFixedLengthVectorVT(Op.getValueType()))
       return LowerFixedLengthVectorLoadToSVE(Op, DAG);
     llvm_unreachable("Unexpected request to lower ISD::LOAD");
   case ISD::ADD:
     return LowerToPredicatedOp(Op, DAG, AArch64ISD::ADD_PRED);
   case ISD::AND:
     return LowerToScalableOp(Op, DAG);
auto NewLoad = DAG.getMaskedLoad(
       Load->getMemoryVT(), Load->getMemOperand(), Load->getAddressingMode(),
       Load->getExtensionType());
 
   auto Result = convertFromScalableVector(DAG, VT, NewLoad);
   SDValue MergedValues[2] = {Result, Load->getChain()};
   return DAG.getMergeValues(MergedValues, DL);
 }
 
+static SDValue convertFixedMaskToScalableVector(SDValue Mask,
+                                                SelectionDAG &DAG) {
+  SDLoc DL(Mask);
+  EVT InVT = Mask.getValueType();
+  EVT ContainerVT = getContainerForFixedLengthVector(DAG, InVT);
+
+  auto Op1 = convertToScalableVector(DAG, ContainerVT, Mask);
+  auto Op2 = DAG.getConstant(0, DL, ContainerVT);
+  auto Pg = getPredicateForFixedLengthVector(DAG, DL, InVT);
+
+  EVT CmpVT = Pg.getValueType();
+  auto Cmp = DAG.getNode(AArch64ISD::SETCC_MERGE_ZERO, DL, CmpVT,
+                         {Pg, Op1, Op2, DAG.getCondCode(ISD::SETNE)});
+
+  return Cmp;
+}
+
+// Convert all fixed length vector loads larger than NEON to masked_loads.
+SDValue AArch64TargetLowering::LowerFixedLengthVectorMLoadToSVE(
+    SDValue Op, SelectionDAG &DAG) const {
+  auto Load = cast<MaskedLoadSDNode>(Op);
+
+  SDLoc DL(Op);
+  EVT VT = Op.getValueType();
+  EVT ContainerVT = getContainerForFixedLengthVector(DAG, VT);
+
+  SDValue Mask = convertFixedMaskToScalableVector(Load->getMask(), DAG);
+
+  SDValue PassThru;
+  bool IsPassThruZeroOrUndef = false;
+
+  if (Load->getPassThru()->isUndef()) {
+    PassThru = DAG.getUNDEF(ContainerVT);
+    IsPassThruZeroOrUndef = true;
+  } else {
+    if (ContainerVT.isInteger())
+      PassThru = DAG.getConstant(0, DL, ContainerVT);
+    else
+      PassThru = DAG.getConstantFP(0, DL, ContainerVT);
+    if (isZerosVector(Load->getPassThru().getNode()))
+      IsPassThruZeroOrUndef = true;
+  }
+
+  auto NewLoad = DAG.getMaskedLoad(
+      ContainerVT, DL, Load->getChain(), Load->getBasePtr(), Load->getOffset(),
+      Mask, PassThru, Load->getMemoryVT(), Load->getMemOperand(),
+      Load->getAddressingMode(), Load->getExtensionType());
+
+  if (!IsPassThruZeroOrUndef) {
+    SDValue OldPassThru =
+        convertToScalableVector(DAG, ContainerVT, Load->getPassThru());
+    NewLoad = DAG.getSelect(DL, ContainerVT, Mask, NewLoad, OldPassThru);
+  }
+
+  auto Result = convertFromScalableVector(DAG, VT, NewLoad);
+  SDValue MergedValues[2] = {Result, Load->getChain()};
+  return DAG.getMergeValues(MergedValues, DL);
+}
+
 // Convert all fixed length vector stores larger than NEON to masked_stores.
 SDValue AArch64TargetLowering::LowerFixedLengthVectorStoreToSVE(
     SDValue Op, SelectionDAG &DAG) const {
   auto Store = cast<StoreSDNode>(Op);
 
   SDLoc DL(Op);
   EVT VT = Store->getValue().getValueType();
   EVT ContainerVT = getContainerForFixedLengthVector(DAG, VT);
 
   auto NewValue = convertToScalableVector(DAG, ContainerVT, Store->getValue());
   return DAG.getMaskedStore(
       Store->getChain(), DL, NewValue, Store->getBasePtr(), Store->getOffset(),
       getPredicateForFixedLengthVector(DAG, DL, VT), Store->getMemoryVT(),
       Store->getMemOperand(), Store->getAddressingMode(),
       Store->isTruncatingStore());
 }
 
+SDValue AArch64TargetLowering::LowerFixedLengthVectorMStoreToSVE(
+    SDValue Op, SelectionDAG &DAG) const {
+  auto Store = cast<MaskedStoreSDNode>(Op);
+
+  SDLoc DL(Op);
+  EVT VT = Store->getValue().getValueType();
+  EVT ContainerVT = getContainerForFixedLengthVector(DAG, VT);
+
+  auto NewValue = convertToScalableVector(DAG, ContainerVT, Store->getValue());
+  SDValue Mask = convertFixedMaskToScalableVector(Store->getMask(), DAG);
+
+  return DAG.getMaskedStore(
+      Store->getChain(), DL, NewValue, Store->getBasePtr(), Store->getOffset(),
+      Mask, Store->getMemoryVT(), Store->getMemOperand(),
+      Store->getAddressingMode(), Store->isTruncatingStore());
+}
+
 SDValue AArch64TargetLowering::LowerFixedLengthVectorIntDivideToSVE(
     SDValue Op, SelectionDAG &DAG) const {
   SDLoc dl(Op);
   EVT VT = Op.getValueType();
   EVT EltVT = VT.getVectorElementType();
 
   bool Signed = Op.getOpcode() == ISD::SDIV;
   unsigned PredOpcode = Signed ? AArch64ISD::SDIV_PRED : AArch64ISD::UDIV_PRED;

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

bool isLegalElementTypeForSVE(Type *Ty) const {
     if (Ty->isIntegerTy(8) || Ty->isIntegerTy(16) ||
         Ty->isIntegerTy(32) || Ty->isIntegerTy(64))
       return true;
 
     return false;
   }
 
   bool isLegalMaskedLoadStore(Type *DataType, Align Alignment) {
-    if (isa<FixedVectorType>(DataType) || !ST->hasSVE())
+    if (!ST->hasSVE())
       return false;
 
+    // For fixed vectors, aqvoid scalarization if using SVE for them.
+    if (isa<FixedVectorType>(DataType) && !ST->useSVEForFixedLengthVectors())
+      return false; // Fall back to scalarization of masked operations.
+
     return isLegalElementTypeForSVE(DataType->getScalarType());
   }
 
   bool isLegalMaskedLoad(Type *DataType, Align Alignment) {
     return isLegalMaskedLoadStore(DataType, Alignment);
   }
 
   bool isLegalMaskedStore(Type *DataType, Align Alignment) {

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-loads.ll

+; RUN: llc -aarch64-sve-vector-bits-min=128  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=16  -check-prefix=NO_SVE
+; RUN: llc -aarch64-sve-vector-bits-min=256  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=512  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=640  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=768  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=896  -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1024 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1152 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1280 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1408 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1536 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1664 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1792 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1920 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=2048 -asm-verbose=0 < %s | FileCheck %s -D#VBYTES=256 -check-prefixes=CHECK,VBITS_GE_2048,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+
+target triple = "aarch64-unknown-linux-gnu"
+
+; Don't use SVE when its registers are no bigger than NEON.
+; NO_SVE-NOT: ptrue
+
+;
+; Masked Loads
+;
+
+define <2 x float> @masked_load_v2f32(<2 x float>* %ap, <2 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v2f32:
+; CHECK: ptrue [[PG0:p[0-9]+]].s, vl2
+; CHECK: fcmeq v[[N:[0-9]+]].2s, v{{[0-9]+}}.2s, v{{[0-9]+}}.2s
+; CHECK-NEXT: cmpne [[PG1:p[0-9]+]].s, [[PG0]]/z, z[[N]].s, #0
+; CHECK-NEXT: ld1w { z0.s }, [[PG1]]/z, [x0]
+; CHECK-NEXT: ret
+  %a = load <2 x float>, <2 x float>* %ap
+  %b = load <2 x float>, <2 x float>* %bp
+  %mask = fcmp oeq <2 x float> %a, %b
+  %load = call <2 x float> @llvm.masked.load.v2f32(<2 x float>* %ap, i32 8, <2 x i1> %mask, <2 x float> zeroinitializer)
+  ret <2 x float> %load
+}
+
+define <4 x float> @masked_load_v4f32(<4 x float>* %ap, <4 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v4f32:
+; CHECK: ptrue [[PG0:p[0-9]+]].s, vl4
+; CHECK: fcmeq v[[N:[0-9]+]].4s, v{{[0-9]+}}.4s, v{{[0-9]+}}.4s
+; CHECK-NEXT: cmpne [[PG1:p[0-9]+]].s, [[PG0]]/z, z[[N]].s, #0
+; CHECK-NEXT: ld1w { z0.s }, [[PG1]]/z, [x0]
+; CHECK-NEXT: ret
+  %a = load <4 x float>, <4 x float>* %ap
+  %b = load <4 x float>, <4 x float>* %bp
+  %mask = fcmp oeq <4 x float> %a, %b
+  %load = call <4 x float> @llvm.masked.load.v4f32(<4 x float>* %ap, i32 8, <4 x i1> %mask, <4 x float> zeroinitializer)
+  ret <4 x float> %load
+}
+
+define <8 x float> @masked_load_v8f32(<8 x float>* %ap, <8 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v8f32:
+; VBITS_GE_256: ptrue [[PG0:p[0-9]+]].s, vl[[#min(div(VBYTES,4),8)]]
+; VBITS_GE_256: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, [[Z0:z[0-9]+]].s, [[Z1:z[0-9]+]].s
+; VBITS_GE_256-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_256-NEXT: cmpne [[PG1]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_256-NEXT: ld1w { [[Z0]].s }, [[PG1]]/z, [x0]
+; VBITS_GE_256: ret
+  %a = load <8 x float>, <8 x float>* %ap
+  %b = load <8 x float>, <8 x float>* %bp
+  %mask = fcmp oeq <8 x float> %a, %b
+  %load = call <8 x float> @llvm.masked.load.v8f32(<8 x float>* %ap, i32 8, <8 x i1> %mask, <8 x float> zeroinitializer)
+  ret <8 x float> %load
+}
+
+define <16 x float> @masked_load_v16f32(<16 x float>* %ap, <16 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v16f32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].s, vl[[#min(div(VBYTES,4),16)]]
+; VBITS_GE_512: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_512-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_512-NEXT: ld1w { [[Z0]].s }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: ret
+  %a = load <16 x float>, <16 x float>* %ap
+  %b = load <16 x float>, <16 x float>* %bp
+  %mask = fcmp oeq <16 x float> %a, %b
+  %load = call <16 x float> @llvm.masked.load.v16f32(<16 x float>* %ap, i32 8, <16 x i1> %mask, <16 x float> zeroinitializer)
+  ret <16 x float> %load
+}
+
+define <32 x float> @masked_load_v32f32(<32 x float>* %ap, <32 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v32f32:
+; VBITS_GE_1024: ptrue [[PG0:p[0-9]+]].s, vl[[#min(div(VBYTES,4),32)]]
+; VBITS_GE_1024: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_1024-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_1024-NEXT: cmpne [[PG1]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_1024-NEXT: ld1w { [[Z0]].s }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_1024: ret
+  %a = load <32 x float>, <32 x float>* %ap
+  %b = load <32 x float>, <32 x float>* %bp
+  %mask = fcmp oeq <32 x float> %a, %b
+  %load = call <32 x float> @llvm.masked.load.v32f32(<32 x float>* %ap, i32 8, <32 x i1> %mask, <32 x float> zeroinitializer)
+  ret <32 x float> %load
+}
+
+define <64 x float> @masked_load_v64f32(<64 x float>* %ap, <64 x float>* %bp) #0 {
+; CHECK-LABEL: masked_load_v64f32:
+; VBITS_GE_2048: ptrue [[PG0:p[0-9]+]].s, vl[[#min(div(VBYTES,4),64)]]
+; VBITS_GE_2048: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_2048-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_2048-NEXT: cmpne [[PG1]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_2048-NEXT: ld1w { [[Z0]].s }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_2048: ret
+
+  %a = load <64 x float>, <64 x float>* %ap
+  %b = load <64 x float>, <64 x float>* %bp
+  %mask = fcmp oeq <64 x float> %a, %b
+  %load = call <64 x float> @llvm.masked.load.v64f32(<64 x float>* %ap, i32 8, <64 x i1> %mask, <64 x float> zeroinitializer)
+  ret <64 x float> %load
+}
+
+define <64 x i8> @masked_load_v64i8(<64 x i8>* %ap, <64 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_v64i8:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl64
+; VBITS_GE_512: cmpeq [[PG1:p[0-9]+]].b, [[PG0]]/z, [[Z0:z[0-9]+]].b, z{{[0-9]+}}.b
+; VBITS_GE_512-NEXT: mov [[Z0]].b, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].b, [[PG0]]/z, [[Z0]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: ret
+  %a = load <64 x i8>, <64 x i8>* %ap
+  %b = load <64 x i8>, <64 x i8>* %bp
+  %mask = icmp eq <64 x i8> %a, %b
+  %load = call <64 x i8> @llvm.masked.load.v64i8(<64 x i8>* %ap, i32 8, <64 x i1> %mask, <64 x i8> undef)
+  ret <64 x i8> %load
+}
+
+define <32 x i16> @masked_load_v32i16(<32 x i16>* %ap, <32 x i16>* %bp) #0 {
+; CHECK-LABEL: masked_load_v32i16:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].h, vl32
+; VBITS_GE_512: cmpeq [[PG1:p[0-9]+]].h, [[PG0]]/z, [[Z0:z[0-9]+]].h, z{{[0-9]+}}.h
+; VBITS_GE_512-NEXT: mov [[Z0]].h, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].h, [[PG0]]/z, [[Z0]].h, #0
+; VBITS_GE_512-NEXT: ld1h { [[Z0]].h }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: ret
+  %a = load <32 x i16>, <32 x i16>* %ap
+  %b = load <32 x i16>, <32 x i16>* %bp
+  %mask = icmp eq <32 x i16> %a, %b
+  %load = call <32 x i16> @llvm.masked.load.v32i16(<32 x i16>* %ap, i32 8, <32 x i1> %mask, <32 x i16> undef)
+  ret <32 x i16> %load
+}
+
+define <16 x i32> @masked_load_v16i32(<16 x i32>* %ap, <16 x i32>* %bp) #0 {
+; CHECK-LABEL: masked_load_v16i32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].s, vl16
+; VBITS_GE_512: cmpeq [[PG1:p[0-9]+]].s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_512-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_512-NEXT: ld1w { [[Z0]].s }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: ret
+  %a = load <16 x i32>, <16 x i32>* %ap
+  %b = load <16 x i32>, <16 x i32>* %bp
+  %mask = icmp eq <16 x i32> %a, %b
+  %load = call <16 x i32> @llvm.masked.load.v16i32(<16 x i32>* %ap, i32 8, <16 x i1> %mask, <16 x i32> undef)
+  ret <16 x i32> %load
+}
+
+define <8 x i64> @masked_load_v8i64(<8 x i64>* %ap, <8 x i64>* %bp) #0 {
+; CHECK-LABEL: masked_load_v8i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].d, vl8
+; VBITS_GE_512: cmpeq [[PG1:p[0-9]+]].d, [[PG0]]/z, [[Z0:z[0-9]+]].d, z{{[0-9]+}}.d
+; VBITS_GE_512-NEXT: mov [[Z0]].d, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].d, [[PG0]]/z, [[Z0]].d, #0
+; VBITS_GE_512-NEXT: ld1d { [[Z0]].d }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: ret
+  %a = load <8 x i64>, <8 x i64>* %ap
+  %b = load <8 x i64>, <8 x i64>* %bp
+  %mask = icmp eq <8 x i64> %a, %b
+  %load = call <8 x i64> @llvm.masked.load.v8i64(<8 x i64>* %ap, i32 8, <8 x i1> %mask, <8 x i64> undef)
+  ret <8 x i64> %load
+}
+
+define <8 x i64> @masked_load_passthru_v8i64(<8 x i64>* %ap, <8 x i64>* %bp) #0 {
+; CHECK-LABEL: masked_load_passthru_v8i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].d, vl8
+; VBITS_GE_512: cmpeq [[PG1:p[0-9]+]].d, [[PG0]]/z, [[Z0:z[0-9]+]].d, [[Z1:z[0-9]+]].d
+; VBITS_GE_512-NEXT: mov [[Z0]].d, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].d, [[PG0]]/z, [[Z0]].d, #0
+; VBITS_GE_512-NEXT: ld1d { [[Z0]].d }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: sel z{{[0-9]+}}.d, [[PG1]], [[Z0]].d, [[Z1]].d
+; VBITS_GE_512: ret
+  %a = load <8 x i64>, <8 x i64>* %ap
+  %b = load <8 x i64>, <8 x i64>* %bp
+  %mask = icmp eq <8 x i64> %a, %b
+  %load = call <8 x i64> @llvm.masked.load.v8i64(<8 x i64>* %ap, i32 8, <8 x i1> %mask, <8 x i64> %b)
+  ret <8 x i64> %load
+}
+
+define <8 x double> @masked_load_passthru_v8f64(<8 x double>* %ap, <8 x double>* %bp) #0 {
+; CHECK-LABEL: masked_load_passthru_v8f64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].d, vl8
+; VBITS_GE_512: fcmeq [[PG1:p[0-9]+]].d, [[PG0]]/z, [[Z0:z[0-9]+]].d, [[Z1:z[0-9]+]].d
+; VBITS_GE_512-NEXT: mov [[Z0]].d, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG1]].d, [[PG0]]/z, [[Z0]].d, #0
+; VBITS_GE_512-NEXT: ld1d { [[Z0]].d }, [[PG1]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: sel z{{[0-9]+}}.d, [[PG1]], [[Z0]].d, [[Z1]].d
+; VBITS_GE_512: ret
+  %a = load <8 x double>, <8 x double>* %ap
+  %b = load <8 x double>, <8 x double>* %bp
+  %mask = fcmp oeq <8 x double> %a, %b
+  %load = call <8 x double> @llvm.masked.load.v8f64(<8 x double>* %ap, i32 8, <8 x i1> %mask, <8 x double> %b)
+  ret <8 x double> %load
+}
+
+define <32 x i16> @masked_load_sext_v32i8i16(<32 x i8>* %ap, <32 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v32i8i16:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl32
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.b, [[PG0]]/z, [[Z0:z[0-9]+]].b, z{{[0-9]+}}.b
+; VBITS_GE_512-NEXT: mov [[Z0]].b, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].b, [[PG0]]/z, [[Z0]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512: ret
+  %a = load <32 x i8>, <32 x i8>* %ap
+  %b = load <32 x i8>, <32 x i8>* %bp
+  %mask = icmp eq <32 x i8> %a, %b
+  %load = call <32 x i8> @llvm.masked.load.v32i8(<32 x i8>* %ap, i32 8, <32 x i1> %mask, <32 x i8> undef)
+  %ext = sext <32 x i8> %load to <32 x i16>
+  ret <32 x i16> %ext
+}
+
+define <16 x i32> @masked_load_sext_v16i8i32(<16 x i8>* %ap, <16 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v16i8i32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl16
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].16b, v{{[0-9]+}}.16b, v{{[0-9]+}}.16b
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].b, [[PG0]]/z, [[Z0]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512-NEXT: sunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512: ret
+  %a = load <16 x i8>, <16 x i8>* %ap
+  %b = load <16 x i8>, <16 x i8>* %bp
+  %mask = icmp eq <16 x i8> %a, %b
+  %load = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %ap, i32 8, <16 x i1> %mask, <16 x i8> undef)
+  %ext = sext <16 x i8> %load to <16 x i32>
+  ret <16 x i32> %ext
+}
+
+define <8 x i64> @masked_load_sext_v8i8i64(<8 x i8>* %ap, <8 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v8i8i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl8
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].8b, v{{[0-9]+}}.8b, v{{[0-9]+}}.8b
+; VBITS_GE_512-NEXT: cmpne p[[PG:[0-9]+]].b, p0/z, z[[V]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, p[[PG]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512-NEXT: sunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512-NEXT: sunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i8>, <8 x i8>* %ap
+  %b = load <8 x i8>, <8 x i8>* %bp
+  %mask = icmp eq <8 x i8> %a, %b
+  %load = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %ap, i32 8, <8 x i1> %mask, <8 x i8> undef)
+  %ext = sext <8 x i8> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+define <16 x i32> @masked_load_sext_v16i16i32(<16 x i16>* %ap, <16 x i16>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v16i16i32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].h, vl16
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.h, [[PG0]]/z, [[Z0:z[0-9]+]].h, z{{[0-9]+}}.h
+; VBITS_GE_512-NEXT: mov [[Z0]].h, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].h, [[PG0]]/z, [[Z0]].h, #0
+; VBITS_GE_512-NEXT: ld1h { [[Z0]].h }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512: ret
+  %a = load <16 x i16>, <16 x i16>* %ap
+  %b = load <16 x i16>, <16 x i16>* %bp
+  %mask = icmp eq <16 x i16> %a, %b
+  %load = call <16 x i16> @llvm.masked.load.v16i16(<16 x i16>* %ap, i32 8, <16 x i1> %mask, <16 x i16> undef)
+  %ext = sext <16 x i16> %load to <16 x i32>
+  ret <16 x i32> %ext
+}
+
+define <8 x i64> @masked_load_sext_v8i16i64(<8 x i16>* %ap, <8 x i16>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v8i16i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].h, vl8
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].8h, v{{[0-9]+}}.8h, v{{[0-9]+}}.8h
+; VBITS_GE_512-NEXT: cmpne p[[PG:[0-9]+]].h, p0/z, z[[V]].h, #0
+; VBITS_GE_512-NEXT: ld1h { [[Z0]].h }, p[[PG]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512-NEXT: sunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %b = load <8 x i16>, <8 x i16>* %bp
+  %mask = icmp eq <8 x i16> %a, %b
+  %load = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %ap, i32 8, <8 x i1> %mask, <8 x i16> undef)
+  %ext = sext <8 x i16> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+define <8 x i64> @masked_load_sext_v8i32i64(<8 x i32>* %ap, <8 x i32>* %bp) #0 {
+; CHECK-LABEL: masked_load_sext_v8i32i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].s, vl8
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_512-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_512-NEXT: ld1w { [[Z0]].s }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: sunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i32>, <8 x i32>* %ap
+  %b = load <8 x i32>, <8 x i32>* %bp
+  %mask = icmp eq <8 x i32> %a, %b
+  %load = call <8 x i32> @llvm.masked.load.v8i32(<8 x i32>* %ap, i32 8, <8 x i1> %mask, <8 x i32> undef)
+  %ext = sext <8 x i32> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+define <32 x i16> @masked_load_zext_v32i8i16(<32 x i8>* %ap, <32 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v32i8i16:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl32
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.b, [[PG0]]/z, [[Z0:z[0-9]+]].b, z{{[0-9]+}}.b
+; VBITS_GE_512-NEXT: mov [[Z0]].b, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].b, [[PG0]]/z, [[Z0]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512: ret
+  %a = load <32 x i8>, <32 x i8>* %ap
+  %b = load <32 x i8>, <32 x i8>* %bp
+  %mask = icmp eq <32 x i8> %a, %b
+  %load = call <32 x i8> @llvm.masked.load.v32i8(<32 x i8>* %ap, i32 8, <32 x i1> %mask, <32 x i8> undef)
+  %ext = zext <32 x i8> %load to <32 x i16>
+  ret <32 x i16> %ext
+}
+
+define <16 x i32> @masked_load_zext_v16i8i32(<16 x i8>* %ap, <16 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v16i8i32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl16
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].16b, v{{[0-9]+}}.16b, v{{[0-9]+}}.16b
+; VBITS_GE_512-NEXT: cmpne p[[PG:[0-9]+]].b, p0/z, z[[V]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, p[[PG]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512-NEXT: uunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512: ret
+  %a = load <16 x i8>, <16 x i8>* %ap
+  %b = load <16 x i8>, <16 x i8>* %bp
+  %mask = icmp eq <16 x i8> %a, %b
+  %load = call <16 x i8> @llvm.masked.load.v16i8(<16 x i8>* %ap, i32 8, <16 x i1> %mask, <16 x i8> undef)
+  %ext = zext <16 x i8> %load to <16 x i32>
+  ret <16 x i32> %ext
+}
+
+define <8 x i64> @masked_load_zext_v8i8i64(<8 x i8>* %ap, <8 x i8>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v8i8i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].b, vl8
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].8b, v{{[0-9]+}}.8b, v{{[0-9]+}}.8b
+; VBITS_GE_512-NEXT: cmpne p[[PG:[0-9]+]].b, p0/z, z[[V]].b, #0
+; VBITS_GE_512-NEXT: ld1b { [[Z0]].b }, p[[PG]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].h, [[Z0]].b
+; VBITS_GE_512-NEXT: uunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512-NEXT: uunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i8>, <8 x i8>* %ap
+  %b = load <8 x i8>, <8 x i8>* %bp
+  %mask = icmp eq <8 x i8> %a, %b
+  %load = call <8 x i8> @llvm.masked.load.v8i8(<8 x i8>* %ap, i32 8, <8 x i1> %mask, <8 x i8> undef)
+  %ext = zext <8 x i8> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+define <16 x i32> @masked_load_zext_v16i16i32(<16 x i16>* %ap, <16 x i16>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v16i16i32:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].h, vl16
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.h, [[PG0]]/z, [[Z0:z[0-9]+]].h, z{{[0-9]+}}.h
+; VBITS_GE_512-NEXT: mov [[Z0]].h, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].h, [[PG0]]/z, [[Z0]].h, #0
+; VBITS_GE_512-NEXT: ld1h { [[Z0]].h }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512: ret
+  %a = load <16 x i16>, <16 x i16>* %ap
+  %b = load <16 x i16>, <16 x i16>* %bp
+  %mask = icmp eq <16 x i16> %a, %b
+  %load = call <16 x i16> @llvm.masked.load.v16i16(<16 x i16>* %ap, i32 8, <16 x i1> %mask, <16 x i16> undef)
+  %ext = zext <16 x i16> %load to <16 x i32>
+  ret <16 x i32> %ext
+}
+
+define <8 x i64> @masked_load_zext_v8i16i64(<8 x i16>* %ap, <8 x i16>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v8i16i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].h, vl8
+; VBITS_GE_512: cmeq v[[V:[0-9]+]].8h, v{{[0-9]+}}.8h, v{{[0-9]+}}.8h
+; VBITS_GE_512-NEXT: cmpne p[[PG:[0-9]+]].h, p0/z, z[[V]].h, #0
+; VBITS_GE_512-NEXT: ld1h { [[Z0]].h }, p[[PG]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].s, [[Z0]].h
+; VBITS_GE_512-NEXT: uunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i16>, <8 x i16>* %ap
+  %b = load <8 x i16>, <8 x i16>* %bp
+  %mask = icmp eq <8 x i16> %a, %b
+  %load = call <8 x i16> @llvm.masked.load.v8i16(<8 x i16>* %ap, i32 8, <8 x i1> %mask, <8 x i16> undef)
+  %ext = zext <8 x i16> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+define <8 x i64> @masked_load_zext_v8i32i64(<8 x i32>* %ap, <8 x i32>* %bp) #0 {
+; CHECK-LABEL: masked_load_zext_v8i32i64:
+; VBITS_GE_512: ptrue [[PG0:p[0-9]+]].s, vl8
+; VBITS_GE_512: cmpeq p{{[0-9]+}}.s, [[PG0]]/z, [[Z0:z[0-9]+]].s, z{{[0-9]+}}.s
+; VBITS_GE_512-NEXT: mov [[Z0]].s, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z0]].s, #0
+; VBITS_GE_512-NEXT: ld1w { [[Z0]].s }, [[PG2]]/z, [x{{[0-9]+}}]
+; VBITS_GE_512: uunpklo [[Z0]].d, [[Z0]].s
+; VBITS_GE_512: ret
+  %a = load <8 x i32>, <8 x i32>* %ap
+  %b = load <8 x i32>, <8 x i32>* %bp
+  %mask = icmp eq <8 x i32> %a, %b
+  %load = call <8 x i32> @llvm.masked.load.v8i32(<8 x i32>* %ap, i32 8, <8 x i1> %mask, <8 x i32> undef)
+  %ext = zext <8 x i32> %load to <8 x i64>
+  ret <8 x i64> %ext
+}
+
+declare <2 x float> @llvm.masked.load.v2f32(<2 x float>*, i32, <2 x i1>, <2 x float>)
+declare <4 x float> @llvm.masked.load.v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
+declare <8 x float> @llvm.masked.load.v8f32(<8 x float>*, i32, <8 x i1>, <8 x float>)
+declare <16 x float> @llvm.masked.load.v16f32(<16 x float>*, i32, <16 x i1>, <16 x float>)
+declare <32 x float> @llvm.masked.load.v32f32(<32 x float>*, i32, <32 x i1>, <32 x float>)
+declare <64 x float> @llvm.masked.load.v64f32(<64 x float>*, i32, <64 x i1>, <64 x float>)
+
+declare <64 x i8> @llvm.masked.load.v64i8(<64 x i8>*, i32, <64 x i1>, <64 x i8>)
+declare <32 x i8> @llvm.masked.load.v32i8(<32 x i8>*, i32, <32 x i1>, <32 x i8>)
+declare <16 x i8> @llvm.masked.load.v16i8(<16 x i8>*, i32, <16 x i1>, <16 x i8>)
+declare <16 x i16> @llvm.masked.load.v16i16(<16 x i16>*, i32, <16 x i1>, <16 x i16>)
+declare <8 x i8> @llvm.masked.load.v8i8(<8 x i8>*, i32, <8 x i1>, <8 x i8>)
+declare <8 x i16> @llvm.masked.load.v8i16(<8 x i16>*, i32, <8 x i1>, <8 x i16>)
+declare <8 x i32> @llvm.masked.load.v8i32(<8 x i32>*, i32, <8 x i1>, <8 x i32>)
+declare <32 x i16> @llvm.masked.load.v32i16(<32 x i16>*, i32, <32 x i1>, <32 x i16>)
+declare <16 x i32> @llvm.masked.load.v16i32(<16 x i32>*, i32, <16 x i1>, <16 x i32>)
+declare <8 x i64> @llvm.masked.load.v8i64(<8 x i64>*, i32, <8 x i1>, <8 x i64>)
+declare <8 x double> @llvm.masked.load.v8f64(<8 x double>*, i32, <8 x i1>, <8 x double>)
+
+attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-stores.ll

+; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -D#VBYTES=16  -check-prefix=NO_SVE
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -D#VBYTES=256 -check-prefixes=CHECK,VBITS_GE_2048,VBITS_GE_1024,VBITS_GE_512,VBITS_GE_256
+
+target triple = "aarch64-unknown-linux-gnu"
+
+; Don't use SVE when its registers are no bigger than NEON.
+; NO_SVE-NOT: ptrue
+
+;;
+;; Masked Stores
+;;
+define void @masked_store_v2f32(<2 x float>* %ap, <2 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v2f32:
+; CHECK: ptrue p[[P0:[0-9]+]].s, vl2
+; CHECK: fcmeq v[[P1:[0-9]+]].2s, v{{[0-9]+}}.2s, v{{[0-9]+}}.2s
+; CHECK-NEXT: cmpne p[[P2:[0-9]+]].s, p[[P0]]/z, z[[P1]].s, #0
+; CHECK-NEXT: st1w { z0.s }, p[[P2]], [x{{[0-9]+}}]
+; CHECK-NEXT: ret
+  %a = load <2 x float>, <2 x float>* %ap
+  %b = load <2 x float>, <2 x float>* %bp
+  %mask = fcmp oeq <2 x float> %a, %b
+  call void @llvm.masked.store.v2f32(<2 x float> %a, <2 x float>* %bp, i32 8, <2 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v4f32(<4 x float>* %ap, <4 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v4f32:
+; CHECK: ptrue p[[P0:[0-9]+]].s, vl4
+; CHECK: fcmeq v[[P1:[0-9]+]].4s, v{{[0-9]+}}.4s, v{{[0-9]+}}.4s
+; CHECK-NEXT: cmpne p[[P2:[0-9]+]].s, p[[P0]]/z, z[[P1]].s, #0
+; CHECK-NEXT: st1w { z0.s }, p[[P2]], [x{{[0-9]+}}]
+; CHECK-NEXT: ret
+  %a = load <4 x float>, <4 x float>* %ap
+  %b = load <4 x float>, <4 x float>* %bp
+  %mask = fcmp oeq <4 x float> %a, %b
+  call void @llvm.masked.store.v4f32(<4 x float> %a, <4 x float>* %bp, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v8f32(<8 x float>* %ap, <8 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v8f32:
+; VBITS_GE_256: ptrue [[PG0:p[0-9]+]].s, vl[[#min(div(VBYTES,4),8)]]
+; VBITS_GE_256: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, z{{[0-9]+}}.s, z{{[0-9]+}}.s
+; VBITS_GE_256-NEXT: mov [[Z1:z[0-9]+]].s, [[PG1]]/z, #-1
+; VBITS_GE_256-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z1]].s, #0
+; VBITS_GE_256-NEXT: st1w { z0.s }, [[PG2]], [x{{[0-9]+}}]
+; VBITS_GE_256-NEXT: ret
+  %a = load <8 x float>, <8 x float>* %ap
+  %b = load <8 x float>, <8 x float>* %bp
+  %mask = fcmp oeq <8 x float> %a, %b
+  call void @llvm.masked.store.v8f32(<8 x float> %a, <8 x float>* %bp, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v16f32(<16 x float>* %ap, <16 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v16f32:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].s, vl[[#min(div(VBYTES,4),16)]]
+; VBITS_GE_512: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, z{{[0-9]+}}.s, z{{[0-9]+}}.s
+; VBITS_GE_512-NEXT: mov [[Z1:z[0-9]+]].s, [[PG1]]/z, #-1
+; VBITS_GE_512-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z1]].s, #0
+; VBITS_GE_512-NEXT: st1w { z0.s }, [[PG2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <16 x float>, <16 x float>* %ap
+  %b = load <16 x float>, <16 x float>* %bp
+  %mask = fcmp oeq <16 x float> %a, %b
+  call void @llvm.masked.store.v16f32(<16 x float> %a, <16 x float>* %ap, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v32f32(<32 x float>* %ap, <32 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v32f32:
+; VBITS_GE_1024: ptrue p[[P0:[0-9]+]].s, vl[[#min(div(VBYTES,4),32)]]
+; VBITS_GE_1024: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, z{{[0-9]+}}.s, z{{[0-9]+}}.s
+; VBITS_GE_1024-NEXT: mov [[Z1:z[0-9]+]].s, [[PG1]]/z, #-1
+; VBITS_GE_1024-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z1]].s, #0
+; VBITS_GE_1024-NEXT: st1w { z0.s }, [[PG2]], [x{{[0-9]+}}]
+; VBITS_GE_1024-NEXT: ret
+  %a = load <32 x float>, <32 x float>* %ap
+  %b = load <32 x float>, <32 x float>* %bp
+  %mask = fcmp oeq <32 x float> %a, %b
+  call void @llvm.masked.store.v32f32(<32 x float> %a, <32 x float>* %ap, i32 8, <32 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_v64f32(<64 x float>* %ap, <64 x float>* %bp) #0 {
+; CHECK-LABEL: masked_store_v64f32:
+; VBITS_GE_2048: ptrue p[[P0:[0-9]+]].s, vl[[#min(div(VBYTES,4),64)]]
+; VBITS_GE_2048: fcmeq [[PG1:p[0-9]+]].s, [[PG0]]/z, z{{[0-9]+}}.s, z{{[0-9]+}}.s
+; VBITS_GE_2048-NEXT: mov [[Z1:z[0-9]+]].s, [[PG1]]/z, #-1
+; VBITS_GE_2048-NEXT: cmpne [[PG2:p[0-9]+]].s, [[PG0]]/z, [[Z1]].s, #0
+; VBITS_GE_2048-NEXT: st1w { z0.s }, [[PG2]], [x{{[0-9]+}}]
+; VBITS_GE_2048-NEXT: ret
+  %a = load <64 x float>, <64 x float>* %ap
+  %b = load <64 x float>, <64 x float>* %bp
+  %mask = fcmp oeq <64 x float> %a, %b
+  call void @llvm.masked.store.v64f32(<64 x float> %a, <64 x float>* %ap, i32 8, <64 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v8i64i8(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i8>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v8i64i8:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].d, vl8
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0:z[0-9]+]].d, [[Z1:z[0-9]+]].d
+; VBITS_GE_512-DAG: uzp1 [[Z1]].s, [[Z1]].s, [[Z1]].s
+; VBITS_GE_512-DAG: uzp1 [[Z1]].h, [[Z1]].h, [[Z1]].h
+; VBITS_GE_512-DAG: uzp1 [[Z1]].b, [[Z1]].b, [[Z1]].b
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].b, p{{[0-9]+}}/z, [[Z1]].b, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].s, [[Z0]].s, [[Z0]].s
+; VBITS_GE_512-DAG: uzp1 [[Z0]].h, [[Z0]].h, [[Z0]].h
+; VBITS_GE_512-DAG: uzp1 [[Z0]].b, [[Z0]].b, [[Z0]].b
+; VBITS_GE_512: st1b { [[Z0]].b }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <8 x i64>, <8 x i64>* %ap
+  %b = load <8 x i64>, <8 x i64>* %bp
+  %mask = icmp eq <8 x i64> %a, %b
+  %val = trunc <8 x i64> %a to <8 x i8>
+  call void @llvm.masked.store.v8i8(<8 x i8> %val, <8 x i8>* %dest, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v8i64i16(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i16>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v8i64i16:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].d, vl8
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0:z[0-9]+]].d, [[Z1:z[0-9]+]].d
+; VBITS_GE_512-DAG: uzp1 [[Z1]].s, [[Z1]].s, [[Z1]].s
+; VBITS_GE_512-DAG: uzp1 [[Z1]].h, [[Z1]].h, [[Z1]].h
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].h, p{{[0-9]+}}/z, [[Z1]].h, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].s, [[Z0]].s, [[Z0]].s
+; VBITS_GE_512-DAG: uzp1 [[Z0]].h, [[Z0]].h, [[Z0]].h
+; VBITS_GE_512: st1h { [[Z0]].h }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <8 x i64>, <8 x i64>* %ap
+  %b = load <8 x i64>, <8 x i64>* %bp
+  %mask = icmp eq <8 x i64> %a, %b
+  %val = trunc <8 x i64> %a to <8 x i16>
+  call void @llvm.masked.store.v8i16(<8 x i16> %val, <8 x i16>* %dest, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v8i64i32(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i32>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v8i64i32:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].d, vl8
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0:z[0-9]+]].d, [[Z1:z[0-9]+]].d
+; VBITS_GE_512-DAG: uzp1 [[Z1]].s, [[Z1]].s, [[Z1]].s
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].s, p{{[0-9]+}}/z, [[Z1]].s, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].s, [[Z0]].s, [[Z0]].s
+; VBITS_GE_512: st1w { [[Z0]].s }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <8 x i64>, <8 x i64>* %ap
+  %b = load <8 x i64>, <8 x i64>* %bp
+  %mask = icmp eq <8 x i64> %a, %b
+  %val = trunc <8 x i64> %a to <8 x i32>
+  call void @llvm.masked.store.v8i32(<8 x i32> %val, <8 x i32>* %dest, i32 8, <8 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v16i32i8(<16 x i32>* %ap, <16 x i32>* %bp, <16 x i8>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v16i32i8:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].s, vl16
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].s, p[[P0]]/z, [[Z0:z[0-9]+]].s, [[Z1:z[0-9]+]].s
+; VBITS_GE_512-DAG: uzp1 [[Z1]].h, [[Z1]].h, [[Z1]].h
+; VBITS_GE_512-DAG: uzp1 [[Z1]].b, [[Z1]].b, [[Z1]].b
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].b, p{{[0-9]+}}/z, [[Z1]].b, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].h, [[Z0]].h, [[Z0]].h
+; VBITS_GE_512-DAG: uzp1 [[Z0]].b, [[Z0]].b, [[Z0]].b
+; VBITS_GE_512: st1b { [[Z0]].b }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <16 x i32>, <16 x i32>* %ap
+  %b = load <16 x i32>, <16 x i32>* %bp
+  %mask = icmp eq <16 x i32> %a, %b
+  %val = trunc <16 x i32> %a to <16 x i8>
+  call void @llvm.masked.store.v16i8(<16 x i8> %val, <16 x i8>* %dest, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v16i32i16(<16 x i32>* %ap, <16 x i32>* %bp, <16 x i16>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v16i32i16:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].s, vl16
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].s, p[[P0]]/z, [[Z0:z[0-9]+]].s, [[Z1:z[0-9]+]].s
+; VBITS_GE_512-DAG: uzp1 [[Z1]].h, [[Z1]].h, [[Z1]].h
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].h, p{{[0-9]+}}/z, [[Z1]].h, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].h, [[Z0]].h, [[Z0]].h
+; VBITS_GE_512: st1h { [[Z0]].h }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <16 x i32>, <16 x i32>* %ap
+  %b = load <16 x i32>, <16 x i32>* %bp
+  %mask = icmp eq <16 x i32> %a, %b
+  %val = trunc <16 x i32> %a to <16 x i16>
+  call void @llvm.masked.store.v16i16(<16 x i16> %val, <16 x i16>* %dest, i32 8, <16 x i1> %mask)
+  ret void
+}
+
+define void @masked_store_trunc_v32i16i8(<32 x i16>* %ap, <32 x i16>* %bp, <32 x i8>* %dest) #0 {
+; CHECK-LABEL: masked_store_trunc_v32i16i8:
+; VBITS_GE_512: ptrue p[[P0:[0-9]+]].h, vl32
+; VBITS_GE_512: cmpeq p[[P1:[0-9]+]].h, p[[P0]]/z, [[Z0:z[0-9]+]].h, [[Z1:z[0-9]+]].h
+; VBITS_GE_512-DAG: uzp1 [[Z1]].b, [[Z1]].b, [[Z1]].b
+; VBITS_GE_512-DAG: cmpne p[[P2:[0-9]+]].b, p{{[0-9]+}}/z, [[Z1]].b, #0
+; VBITS_GE_512-DAG: uzp1 [[Z0]].b, [[Z0]].b, [[Z0]].b
+; VBITS_GE_512: st1b { [[Z0]].b }, p[[P2]], [x{{[0-9]+}}]
+; VBITS_GE_512-NEXT: ret
+  %a = load <32 x i16>, <32 x i16>* %ap
+  %b = load <32 x i16>, <32 x i16>* %bp
+  %mask = icmp eq <32 x i16> %a, %b
+  %val = trunc <32 x i16> %a to <32 x i8>
+  call void @llvm.masked.store.v32i8(<32 x i8> %val, <32 x i8>* %dest, i32 8, <32 x i1> %mask)
+  ret void
+}
+
+declare void @llvm.masked.store.v2f32(<2 x float>, <2 x float>*, i32, <2 x i1>)
+declare void @llvm.masked.store.v4f32(<4 x float>, <4 x float>*, i32, <4 x i1>)
+declare void @llvm.masked.store.v8f32(<8 x float>, <8 x float>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16f32(<16 x float>, <16 x float>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v32f32(<32 x float>, <32 x float>*, i32, <32 x i1>)
+declare void @llvm.masked.store.v64f32(<64 x float>, <64 x float>*, i32, <64 x i1>)
+
+declare void @llvm.masked.store.v8i8(<8 x i8>, <8 x i8>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v8i16(<8 x i16>, <8 x i16>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v8i32(<8 x i32>, <8 x i32>*, i32, <8 x i1>)
+declare void @llvm.masked.store.v16i8(<16 x i8>, <16 x i8>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v16i16(<16 x i16>, <16 x i16>*, i32, <16 x i1>)
+declare void @llvm.masked.store.v32i8(<32 x i8>, <32 x i8>*, i32, <32 x i1>)
+
+attributes #0 = { "target-features"="+sve" }