[AArch64][SVE] Fold vselect into predicated fmul, fsub and fadd

[MattDevereau]

Summary

[AArch64][SVE] Fold vselect into predicated fmul, fsub and fadd

Fold vselect with an unpredicated fmul/fsub/fadd operand into a predicated fmul/fsub/fadd:

(vselect (p) (op (a) (b)) (a)) => (op -> (p) (a) (b))

[peterwaller-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

You might be able to condense it to something like this suggestion, by introducing another multiclass in InstrFormats SVE_VSelect_3_Op_Pat, analogous to SVE_3_Op_Pat_SelZero. What you have looks reasonable to me, you might want a second opinion from someone else as to whether my proposal here makes sense.

llvm/test/CodeGen/AArch64/sve-fp-vselect.ll

90

Register called %mul for an fsub -- looks like a few register names need updating.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

If possible I prefer the patterns to be added directly to the class that defines the instruction (i.e. sve_fp_2op_p_zds in this case)?

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

Just to extend on this. Perhaps you can create a PatFrags that contains both the intrinsic and vselect DAG patterns? That way you won't need to change the sve_fp_2op_p_zds class, you'll just pass in a different op in place of int_aarch64_sve_fadd for example.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

I don't understand what you mean by the intrinsic. I've not used the int_aarch64_sve_fadd naming so far so I'm not sure which part that refers to, and the only DAG nodes I saw when working on this was AArch64fadd_p. Are you suggesting I use the PatFrags to add the match to defm FADD_ZPmZ on line 454 when it uses sve_fp_2op_p_zds?

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

The sve_fp_2op_p_zds multiclass is used to define the FADD_ZPmZ instruction. It takes an operator that represents the instruction at the DAG level. Currently this is set to int_aarch64_sve_fadd. With this patch you're effectively saying there's another way to represent this instruction within the DAG (i.e. (vselect (p) (a) (fadd (a) (b)))) and so I'm thinking you can create a PatFrags that contains bother operators which is them passed into sve_fp_2op_p_zds in place of the existing int_aarch64_sve_fadd. This way you shouldn't need to create any extra isel patterns.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

Me and Peter had a dive into this, and we assumed you were getting at something like that.
We came up with this PatFrags:

def myfirstpatfrag : PatFrags<(ops node:$Pg, node:$Op1, node:$Op2), [
  (int_aarch64_sve_fmul node:$Pg, node:$Op1, node:$Op2),
  (vselect node:$Pg, node:$Op1, 
    (AArch64fmul_p (nxv2i1 (AArch64ptrue 31)), node:$Op1, node:$Op2)
  ),
]>;

but we haven't managed to find a way to generalise the nxv2i1 part so far.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

515–532

You should be able to drop the (nxv2i1 ) part as the predicate type should be inferable. I just gave this a quick try and it works but you also need to tighten up the definition of the predicated binops (i.e. SDT_AArch64Arith, which is used to define AArch64fmul_p)

I extended it to include SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>,...., which looks to do the trick.

[MattDevereau]

Use patfrags
Update commit message

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

168

Out of interest are the SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3> bits removable? I'm thinking they're redundant given the next line is now very specific about what the types of those operands are.

243

We already have a defined naming scheme for these operations, so these should be named AArch64fadd_m1, AArch64fmul_m1 & AArch64fsub_m1.

[MattDevereau]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

168

Removing them has no effect on the tests added in this patch or anything in the check-llvm target

[peterwaller-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

246

Can the PatFrags be factored into a class with two parameters to avoid stamping it out repeatedly?

251

s/(AArch64ptrue 31)/(SVEAllActive)/g

[MattDevereau]

Moved repeated patfrag into class

[peterwaller-arm]

llvm/lib/Target/AArch64/SVEInstrFormats.td

8512

I don't think this should mimic the form of the other (SVE_...) names, since those are for defining patterns, whereas this is a PatFrags, which is a different kind of thing. So in my view the name should be distinct to try and avoid catching unsuspecting readers into another trap.

How about EitherVSelectOrPassthruPatFrags. I'm thinking along the lines of trying to avoid the error of Or reading as the binary operation by prefacing it with Either.

Also, this doesn't look like the right place for this. I think a better location would be up near this, prehaps right at the end of that block before the getSVEPseudoMap definition. For good measure I might put a comment-block delimiter which separates it from the patterns above, and indicate in the comment that this "matches either an intrinsic, or a predicated operation with an all active predicate".

//===----------------------------------------------------------------------===//
// SVE pattern match helpers.
//===----------------------------------------------------------------------===//

Note also the missing trailing newline phabricator is complaining about.

[peterwaller-arm]

llvm/lib/Target/AArch64/SVEInstrFormats.td

8512

I think better names might be:
op => intrinsic
inst => sdnode

Beware copying the names form the other SVE_... classes, which I think would be confusing. Because in those classes, op is the thing to match for and 'inst' is the replacement, which is not the case for your PatFrags where, both cases are actually things to match.

[MattDevereau]

Updated PatFrags class name and location

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

168–169

I cannot say if it really matters (given SDTCisSameNumEltsAs<0,1> implies 0 must be a vector) but I'd be inclined to keep SDTCisVec<0>.

llvm/test/CodeGen/AArch64/sve-fp-vselect.ll

5–13

Are these tests correct? To me it looks like you've got the true and false operands the wrong way round. This IR is effectively saying "when the predicate it true return op1 else return op1 * op2".

The fact the generate code is what we're after suggests the same mistake exists for the new isel patterns.

[MattDevereau]

llvm/test/CodeGen/AArch64/sve-fp-vselect.ll

5–13

The tests aren't correct, I missed the fact that fcmeq in the sqrt example is creating a predicate of 1s. Is it possible to reverse the operands for specific patterns in the PatFrags list? If not I'll have to split up parts of the logic introduced in this patch

[paulwalker-arm]

I'd focus on one thing at a time. This patch enables the isel required for the way merged arithmetic is modelled by the DAG. Once that is complete you can see what's needed to improve the reciprocal code. For that follow-on work I'm guessing you'll need a DAG combine for the select that inverts the comparison for the cases where this'll be beneficial.

[MattDevereau]

update commit message, flip operands for vselect pattern

[paulwalker-arm]

The IR within the tests looks like it's been doubly indented and there's a typo in the commit message (should be (vselect (p) (op (a) (b)) (a)) => (op -> (p) (a) (b)) but otherwise looks good.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

 def AArch64uminv_p   : SDNode<"AArch64ISD::UMINV_PRED",   SDT_AArch64Reduce>;
 def AArch64orv_p     : SDNode<"AArch64ISD::ORV_PRED",     SDT_AArch64Reduce>;
 def AArch64eorv_p    : SDNode<"AArch64ISD::EORV_PRED",    SDT_AArch64Reduce>;
 def AArch64andv_p    : SDNode<"AArch64ISD::ANDV_PRED",    SDT_AArch64Reduce>;
 def AArch64lasta     : SDNode<"AArch64ISD::LASTA",        SDT_AArch64Reduce>;
 def AArch64lastb     : SDNode<"AArch64ISD::LASTB",        SDT_AArch64Reduce>;
 
 def SDT_AArch64Arith : SDTypeProfile<1, 3, [
-  SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>,
-  SDTCVecEltisVT<1,i1>, SDTCisSameAs<0,2>, SDTCisSameAs<2,3>
+  SDTCisVec<0>, SDTCVecEltisVT<1,i1>, SDTCisSameAs<0,2>,
+  SDTCisSameAs<2,3>, SDTCisSameNumEltsAs<0,1>
 ]>;
 
 def SDT_AArch64FMA : SDTypeProfile<1, 4, [
   SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>, SDTCisVec<4>,
   SDTCVecEltisVT<1,i1>, SDTCisSameAs<0,2>, SDTCisSameAs<2,3>, SDTCisSameAs<3,4>
 ]>;
 
 // Predicated operations with the result of inactive lanes being unspecified.

 def AArch64revb_mt   : SDNode<"AArch64ISD::BSWAP_MERGE_PASSTHRU", SDT_AArch64Arith>;
 def AArch64revh_mt   : SDNode<"AArch64ISD::REVH_MERGE_PASSTHRU", SDT_AArch64Arith>;
 def AArch64revw_mt   : SDNode<"AArch64ISD::REVW_MERGE_PASSTHRU", SDT_AArch64Arith>;
 
 // These are like the above but we don't yet have need for ISD nodes. They allow
 // a single pattern to match intrinsic and ISD operand layouts.
 def AArch64cls_mt  : PatFrags<(ops node:$pg, node:$op, node:$pt), [(int_aarch64_sve_cls  node:$pt, node:$pg, node:$op)]>;
 def AArch64cnot_mt : PatFrags<(ops node:$pg, node:$op, node:$pt), [(int_aarch64_sve_cnot node:$pt, node:$pg, node:$op)]>;
 def AArch64not_mt  : PatFrags<(ops node:$pg, node:$op, node:$pt), [(int_aarch64_sve_not  node:$pt, node:$pg, node:$op)]>;
 
+def AArch64fmul_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_fmul, AArch64fmul_p>;
+def AArch64fadd_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_fadd, AArch64fadd_p>;
+def AArch64fsub_m1 : EitherVSelectOrPassthruPatFrags<int_aarch64_sve_fsub, AArch64fsub_p>;
+
 def SDT_AArch64FCVT : SDTypeProfile<1, 3, [
   SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisVec<3>,
   SDTCVecEltisVT<1,i1>
 ]>;
 
 def SDT_AArch64FCVTR : SDTypeProfile<1, 4, [
   SDTCisVec<0>, SDTCisVec<1>, SDTCisVec<2>, SDTCisInt<3>, SDTCisVec<4>,
   SDTCVecEltisVT<1,i1>
 ]>;
 
 def AArch64fcvtr_mt  : SDNode<"AArch64ISD::FP_ROUND_MERGE_PASSTHRU", SDT_AArch64FCVTR>;
let Predicates = [HasSVE, UseExperimentalZeroingPseudos] in {
     defm FMUL_ZPZI    : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_half_two, fpimm_half, fpimm_two, int_aarch64_sve_fmul>;
     defm FSUBR_ZPZI   : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_half_one, fpimm_half, fpimm_one, int_aarch64_sve_fsubr>;
     defm FMAXNM_ZPZI  : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_zero_one, fpimm0, fpimm_one, int_aarch64_sve_fmaxnm>;
     defm FMINNM_ZPZI  : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_zero_one, fpimm0, fpimm_one, int_aarch64_sve_fminnm>;
     defm FMAX_ZPZI    : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_zero_one, fpimm0, fpimm_one, int_aarch64_sve_fmax>;
     defm FMIN_ZPZI    : sve_fp_2op_i_p_zds_zeroing_hfd<sve_fpimm_zero_one, fpimm0, fpimm_one, int_aarch64_sve_fmin>;
   }
 
-  defm FADD_ZPmZ   : sve_fp_2op_p_zds<0b0000, "fadd", "FADD_ZPZZ", int_aarch64_sve_fadd, DestructiveBinaryComm>;
-  defm FSUB_ZPmZ   : sve_fp_2op_p_zds<0b0001, "fsub", "FSUB_ZPZZ", int_aarch64_sve_fsub, DestructiveBinaryCommWithRev, "FSUBR_ZPmZ">;
-  defm FMUL_ZPmZ   : sve_fp_2op_p_zds<0b0010, "fmul", "FMUL_ZPZZ", int_aarch64_sve_fmul, DestructiveBinaryComm>;
+  defm FADD_ZPmZ   : sve_fp_2op_p_zds<0b0000, "fadd", "FADD_ZPZZ", AArch64fadd_m1, DestructiveBinaryComm>;
+  defm FSUB_ZPmZ   : sve_fp_2op_p_zds<0b0001, "fsub", "FSUB_ZPZZ", AArch64fsub_m1, DestructiveBinaryCommWithRev, "FSUBR_ZPmZ">;
+  defm FMUL_ZPmZ   : sve_fp_2op_p_zds<0b0010, "fmul", "FMUL_ZPZZ", AArch64fmul_m1, DestructiveBinaryComm>;
   defm FSUBR_ZPmZ  : sve_fp_2op_p_zds<0b0011, "fsubr", "FSUBR_ZPZZ", int_aarch64_sve_fsubr, DestructiveBinaryCommWithRev, "FSUB_ZPmZ", /*isReverseInstr*/ 1>;
   defm FMAXNM_ZPmZ : sve_fp_2op_p_zds<0b0100, "fmaxnm", "FMAXNM_ZPZZ", int_aarch64_sve_fmaxnm, DestructiveBinaryComm>;
   defm FMINNM_ZPmZ : sve_fp_2op_p_zds<0b0101, "fminnm", "FMINNM_ZPZZ", int_aarch64_sve_fminnm, DestructiveBinaryComm>;
   defm FMAX_ZPmZ   : sve_fp_2op_p_zds<0b0110, "fmax", "FMAX_ZPZZ", int_aarch64_sve_fmax, DestructiveBinaryComm>;
   defm FMIN_ZPmZ   : sve_fp_2op_p_zds<0b0111, "fmin", "FMIN_ZPZZ", int_aarch64_sve_fmin, DestructiveBinaryComm>;
   defm FABD_ZPmZ   : sve_fp_2op_p_zds<0b1000, "fabd", "FABD_ZPZZ", int_aarch64_sve_fabd, DestructiveBinaryComm>;
   defm FSCALE_ZPmZ : sve_fp_2op_p_zds_fscale<0b1001, "fscale", int_aarch64_sve_fscale>;
   defm FMULX_ZPmZ  : sve_fp_2op_p_zds<0b1010, "fmulx", "FMULX_ZPZZ", int_aarch64_sve_fmulx, DestructiveBinaryComm>;

 
 let Predicates = [HasSVEorStreamingSVE] in {
   defm FADD_ZZZ    : sve_fp_3op_u_zd<0b000, "fadd", fadd, AArch64fadd_p>;
   defm FSUB_ZZZ    : sve_fp_3op_u_zd<0b001, "fsub", fsub, AArch64fsub_p>;
   defm FMUL_ZZZ    : sve_fp_3op_u_zd<0b010, "fmul", fmul, AArch64fmul_p>;
 } // End HasSVEorStreamingSVE
 
 let Predicates = [HasSVE] in {
   defm FTSMUL_ZZZ  : sve_fp_3op_u_zd_ftsmul<0b011, "ftsmul", int_aarch64_sve_ftsmul_x>;
 } // End HasSVE
 
 let Predicates = [HasSVEorStreamingSVE] in {
   defm FRECPS_ZZZ  : sve_fp_3op_u_zd<0b110, "frecps",  AArch64frecps>;
   defm FRSQRTS_ZZZ : sve_fp_3op_u_zd<0b111, "frsqrts", AArch64frsqrts>;
 } // End HasSVEorStreamingSVE
 
 let Predicates = [HasSVE] in {
   defm FTSSEL_ZZZ : sve_int_bin_cons_misc_0_b<"ftssel", int_aarch64_sve_ftssel_x>;
 } // End HasSVE
 
 let Predicates = [HasSVEorStreamingSVE] in {
   defm FCADD_ZPmZ : sve_fp_fcadd<"fcadd", int_aarch64_sve_fcadd>;
   defm FCMLA_ZPmZZ : sve_fp_fcmla<"fcmla", int_aarch64_sve_fcmla>;
 
   defm FMLA_ZPmZZ  : sve_fp_3op_p_zds_a<0b00, "fmla",  "FMLA_ZPZZZ", int_aarch64_sve_fmla, "FMAD_ZPmZZ">;
   defm FMLS_ZPmZZ  : sve_fp_3op_p_zds_a<0b01, "fmls",  "FMLS_ZPZZZ", int_aarch64_sve_fmls, "FMSB_ZPmZZ">;
   defm FNMLA_ZPmZZ : sve_fp_3op_p_zds_a<0b10, "fnmla", "FNMLA_ZPZZZ", int_aarch64_sve_fnmla, "FNMAD_ZPmZZ">;
   defm FNMLS_ZPmZZ : sve_fp_3op_p_zds_a<0b11, "fnmls", "FNMLS_ZPZZZ", int_aarch64_sve_fnmls, "FNMSB_ZPmZZ">;
 
   defm FMAD_ZPmZZ  : sve_fp_3op_p_zds_b<0b00, "fmad",  int_aarch64_sve_fmad, "FMLA_ZPmZZ", /*isReverseInstr*/ 1>;
   defm FMSB_ZPmZZ  : sve_fp_3op_p_zds_b<0b01, "fmsb",  int_aarch64_sve_fmsb, "FMLS_ZPmZZ", /*isReverseInstr*/ 1>;
   defm FNMAD_ZPmZZ : sve_fp_3op_p_zds_b<0b10, "fnmad", int_aarch64_sve_fnmad, "FNMLA_ZPmZZ", /*isReverseInstr*/ 1>;
   defm FNMSB_ZPmZZ : sve_fp_3op_p_zds_b<0b11, "fnmsb", int_aarch64_sve_fnmsb, "FNMLS_ZPmZZ", /*isReverseInstr*/ 1>;
 

llvm/lib/Target/AArch64/SVEInstrFormats.td

 class SVE_2_Op_Fp_Imm_Pat_Zero<ValueType vt, SDPatternOperator op,
                               ValueType pt, ValueType it,
                               FPImmLeaf immL, int imm,
                               Instruction inst>
 : Pat<(vt (op pt:$Pg, (vselect pt:$Pg, vt:$Zs1, (SVEDup0)),
                       (vt (AArch64dup (it immL))))),
       (inst $Pg, $Zs1, imm)>;
 
+//===----------------------------------------------------------------------===//
+// SVE pattern match helpers.
+//===----------------------------------------------------------------------===//
+
+// Matches either an intrinsic, or a predicated operation with an all active predicate
+class EitherVSelectOrPassthruPatFrags<SDPatternOperator intrinsic, SDPatternOperator sdnode>
+: PatFrags<(ops node:$Pg, node:$Op1, node:$Op2), [
+    (intrinsic node:$Pg, node:$Op1, node:$Op2),
+    (vselect node:$Pg, (sdnode (SVEAllActive), node:$Op1, node:$Op2), node:$Op1),
+  ]>;
+
 //
 // Pseudo -> Instruction mappings
 //
 def getSVEPseudoMap : InstrMapping {
   let FilterClass = "SVEPseudo2Instr";
   let RowFields = ["PseudoName"];
   let ColFields = ["IsInstr"];
   let KeyCol = ["0"];
multiclass sve_int_bin_pred_all_active_bhsd<SDPatternOperator op> {
   def _UNDEF_S : PredTwoOpPseudo<NAME # _S, ZPR32, FalseLanesUndef>;
   def _UNDEF_D : PredTwoOpPseudo<NAME # _D, ZPR64, FalseLanesUndef>;
 
   def : SVE_2_Op_Pred_All_Active_Pt<nxv16i8, op, nxv16i1, nxv16i8, nxv16i8, !cast<Pseudo>(NAME # _UNDEF_B)>;
   def : SVE_2_Op_Pred_All_Active_Pt<nxv8i16, op, nxv8i1,  nxv8i16, nxv8i16, !cast<Pseudo>(NAME # _UNDEF_H)>;
   def : SVE_2_Op_Pred_All_Active_Pt<nxv4i32, op, nxv4i1,  nxv4i32, nxv4i32, !cast<Pseudo>(NAME # _UNDEF_S)>;
   def : SVE_2_Op_Pred_All_Active_Pt<nxv2i64, op, nxv2i1,  nxv2i64, nxv2i64, !cast<Pseudo>(NAME # _UNDEF_D)>;
 }
 

llvm/test/CodeGen/AArch64/sve-fp-vselect.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+define <vscale x 8 x half> @vselect_fmul_f16(<vscale x 8 x i1> %p, <vscale x 8 x half> %a, <vscale x 8 x half> %b) {
+; CHECK-LABEL: vselect_fmul_f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %mul = fmul <vscale x 8 x half> %a, %b
+  %sel = select <vscale x 8 x i1> %p, <vscale x 8 x half> %mul, <vscale x 8 x half> %a
+  ret <vscale x 8 x half> %sel
+}
+
+define <vscale x 4 x float> @vselect_fmul_f32(<vscale x 4 x i1> %p, <vscale x 4 x float> %a, <vscale x 4 x float> %b) {
+; CHECK-LABEL: vselect_fmul_f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %mul = fmul <vscale x 4 x float> %a, %b
+  %sel = select <vscale x 4 x i1> %p, <vscale x 4 x float> %mul, <vscale x 4 x float> %a
+  ret <vscale x 4 x float> %sel
+}
+
+define <vscale x 2 x double> @vselect_fmul_f64(<vscale x 2 x i1> %p, <vscale x 2 x double> %a, <vscale x 2 x double> %b) {
+; CHECK-LABEL: vselect_fmul_f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %mul = fmul <vscale x 2 x double> %a, %b
+  %sel = select <vscale x 2 x i1> %p, <vscale x 2 x double> %mul, <vscale x 2 x double> %a
+  ret <vscale x 2 x double> %sel
+}
+
+define <vscale x 8 x half> @vselect_fadd_f16(<vscale x 8 x i1> %p, <vscale x 8 x half> %a, <vscale x 8 x half> %b) {
+; CHECK-LABEL: vselect_fadd_f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fadd z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %add = fadd <vscale x 8 x half> %a, %b
+  %sel = select <vscale x 8 x i1> %p, <vscale x 8 x half> %add, <vscale x 8 x half> %a
+  ret <vscale x 8 x half> %sel
+}
+
+define <vscale x 4 x float> @vselect_fadd_f32(<vscale x 4 x i1> %p, <vscale x 4 x float> %a, <vscale x 4 x float> %b) {
+; CHECK-LABEL: vselect_fadd_f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fadd z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %add = fadd <vscale x 4 x float> %a, %b
+  %sel = select <vscale x 4 x i1> %p, <vscale x 4 x float> %add, <vscale x 4 x float> %a
+  ret <vscale x 4 x float> %sel
+}
+
+define <vscale x 2 x double> @vselect_fadd_f64(<vscale x 2 x i1> %p, <vscale x 2 x double> %a, <vscale x 2 x double> %b) {
+; CHECK-LABEL: vselect_fadd_f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fadd z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %add = fadd <vscale x 2 x double> %a, %b
+  %sel = select <vscale x 2 x i1> %p, <vscale x 2 x double> %add, <vscale x 2 x double> %a
+  ret <vscale x 2 x double> %sel
+}
+
+define <vscale x 8 x half> @vselect_fsub_f16(<vscale x 8 x i1> %p, <vscale x 8 x half> %a, <vscale x 8 x half> %b) {
+; CHECK-LABEL: vselect_fsub_f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fsub z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    ret
+  %sub = fsub <vscale x 8 x half> %a, %b
+  %sel = select <vscale x 8 x i1> %p, <vscale x 8 x half> %sub, <vscale x 8 x half> %a
+  ret <vscale x 8 x half> %sel
+}
+
+define <vscale x 4 x float> @vselect_fsub_f32(<vscale x 4 x i1> %p, <vscale x 4 x float> %a, <vscale x 4 x float> %b) {
+; CHECK-LABEL: vselect_fsub_f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fsub z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    ret
+  %sub = fsub <vscale x 4 x float> %a, %b
+  %sel = select <vscale x 4 x i1> %p, <vscale x 4 x float> %sub, <vscale x 4 x float> %a
+  ret <vscale x 4 x float> %sel
+}
+
+define <vscale x 2 x double> @vselect_fsub_f64(<vscale x 2 x i1> %p, <vscale x 2 x double> %a, <vscale x 2 x double> %b) {
+; CHECK-LABEL: vselect_fsub_f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fsub z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    ret
+  %sub = fsub <vscale x 2 x double> %a, %b
+  %sel = select <vscale x 2 x i1> %p, <vscale x 2 x double> %sub, <vscale x 2 x double> %a
+  ret <vscale x 2 x double> %sel
+}