[AArch64][SVE] Add patterns to generate FMLA/FMLS/FNMLA/FNMLS/FMAD

[bsmith]

Summary

Add tablegen patterns for a variety of FP multiply type instructions.

Adjust generateFMAsInMachineCombiner to return false if SVE is present
in order to combine fmul+fadd into fma. Also add new pseudo instructions
so as to select the most appropriate of FMLA/FMAD depending on register
allocation.

Depends on D96599

[dmgreen]

Hello. Would the machine combiner method not be better, if it can reduce critical path length?

[bsmith]

In D96424#2556165, @dmgreen wrote:

Hello. Would the machine combiner method not be better, if it can reduce critical path length?

In principal it perhaps would yes, however I'm not sure by itself it is up to the task. The main issue here is the fact that there are different instructions available depending on whether you have a predicate or not, (notably the combined instructions (FMLA etc) have only predicated forms). As such, ISelLowering lowers nodes (e.g. FADD + FMUL) to predicated forms to allow instruction selection to either select these predicated combined instructions (FMLA) or to select in such a way that the predicate is removed again if it could not combine instructions.

If this instruction combining was deferred to the machine combiner then either it would have to learn how to add PTRUES in to allow use of predicated forms, or ISelLowering would keep lowering to predicated forms and in the case of no combining happening a later post machine combiner pass would have to remove PTRUES that were no longer needed and revert instructions to unpredicated forms. Essentially doing it this way allows for a lot of this complexity to be taken care of by lowering and instruction selection.

Perhaps in the future this is something that could be addressed, but for now I think this is a reasonable approach.

[dmgreen]

Sure, It would at least make sense to do things in machine combiner at a later date. We need to be sure we don't make anything else worse whilst we are doing it though!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11553–11554

This feels like a separate change. I don't see a reason to not fold to FMA when HasFullFP16 in general. (But that would require it's own tests).

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

56

This needs to be more precise. Enabling SVE should not regress scalar/Neon code.

It looks like it should be possible to add the VT to this definition.

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

30–31

Could we just make it so that AArch64Subtarget->TSInfo is passed the subtarget directly on construction?

llvm/test/CodeGen/AArch64/sve-fp-combine.ll

4

It's better, in the long run, to use fast math flags over the global fp-contract I believe.

[dmgreen]

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

30–31

Alternatively, I'm not sure why this function is here in AArch64SelectionDAGInfo and could possibly just be moved to AArch64TargetLowering?

[bsmith]

Updated patch:

• Add some more FNMLA patterns that are required when things get combined slightly differently
• Only return true in generateFMAsInMachineCombiner when the types in question are scalable, and not always when SVE is present.
• Move generateFMAsInMachineCombiner into AArch64TargetLowering
• Replace use of -fp-contract=fast in tests with fast flag on instructions
• Pull out FP16 combining into a separate patch

[dmgreen]

Thanks. It looks like this could do with more half tests.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2133

Should there be nxv4f16 and nxv2f16 patterns?

It may also make sense to make multipatterns from these. They all just differ in a similar way to how TriOpFrag is used.

And it would probably be best to keep these close to the instructions they create, unless there is a reason have them separate.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2132–2140

I might have misread something but these looks like a duplicate of the patterns you've updated just above, which does support the other f16 vector types. As Dave said, for completeness the other selection blocks will need patterns for nxv4f16 and nxv2f16.

llvm/test/CodeGen/AArch64/sve-fp-combine.ll

5–7

Generally for these types of tests we use update_llc_test_checks.py to generate the CHECK lines as it aids future updates.

194–199

It didn't exist when the tests were written but today I think/hope you can use the fneg instruction and thus simplify the tests a little.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2199–2200

This doesn't effect this patch but just so you know where my head is at, my feeling is that we may end up moving this kind of logic into an AArch64 specific DAG combine. I say this because between the select, fma and fneg we have three predicates in play and so there's likely to be more cases we can perform the match against plus in the future we're quite likely to have to consider the implications of floating point exceptions which will make predicate analysis even more important.

[bsmith]

• Regenerate tests with update_llc_test_checks.py
• Restructure patterns to use a multiclass
• Remove duplicated patterns
• Add missing patterns for more f16 types

[bsmith]

• Reverse order of dependant patches, pulling half tests from D96599
• Add missing half tests

[paulwalker-arm]

Given the extensive testing within sve-fp-combine.ll I do wonder why sve-fma-dagcombine.ll is required, but otherwise the patch looks good.

[bsmith]

In D96424#2566176, @paulwalker-arm wrote:

Given the extensive testing within sve-fp-combine.ll I do wonder why sve-fma-dagcombine.ll is required, but otherwise the patch looks good.

Test removed in commit, as yes, it is totally redundant.

llvm/include/llvm/CodeGen/SelectionDAGTargetInfo.h

public:
   virtual SDValue EmitTargetCodeForSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                           SDValue Chain, SDValue Addr,
                                           SDValue Size,
                                           MachinePointerInfo DstPtrInfo,
                                           bool ZeroData) const {
     return SDValue();
   }
 
-  // Return true when the decision to generate FMA's (or FMS, FMLA etc) rather
-  // than FMUL and ADD is delegated to the machine combiner.
-  virtual bool generateFMAsInMachineCombiner(CodeGenOpt::Level OptLevel) const {
-    return false;
-  }
-
   // Return true if the DAG Combiner should disable generic combines.
   virtual bool disableGenericCombines(CodeGenOpt::Level OptLevel) const {
     return false;
   }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_SELECTIONDAGTARGETINFO_H

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   /// fadd/fsub.
   virtual bool isFMADLegal(const SelectionDAG &DAG, const SDNode *N) const {
     assert((N->getOpcode() == ISD::FADD || N->getOpcode() == ISD::FSUB ||
             N->getOpcode() == ISD::FMUL) &&
            "unexpected node in FMAD forming combine");
     return isOperationLegal(ISD::FMAD, N->getValueType(0));
   }
 
+  // Return true when the decision to generate FMA's (or FMS, FMLA etc) rather
+  // than FMUL and ADD is delegated to the machine combiner.
+  virtual bool generateFMAsInMachineCombiner(EVT VT,
+                                             CodeGenOpt::Level OptLevel) const {
+    return false;
+  }
+
   /// Return true if it's profitable to narrow operations of type VT1 to
   /// VT2. e.g. on x86, it's profitable to narrow from i32 to i8 but not from
   /// i32 to i16.
   virtual bool isNarrowingProfitable(EVT /*VT1*/, EVT /*VT2*/) const {
     return false;
   }
 
   /// Return true if it is beneficial to convert a load of a constant to

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitFADDForFMACombine(SDNode *N) {
   bool CanReassociate =
       Options.UnsafeFPMath || N->getFlags().hasAllowReassociation();
   bool AllowFusionGlobally = (Options.AllowFPOpFusion == FPOpFusion::Fast ||
                               CanFuse || HasFMAD);
   // If the addition is not contractable, do not combine.
   if (!AllowFusionGlobally && !isContractable(N))
     return SDValue();
 
-  if (STI && STI->generateFMAsInMachineCombiner(OptLevel))
+  if (TLI.generateFMAsInMachineCombiner(VT, OptLevel))
     return SDValue();
 
   // Always prefer FMAD to FMA for precision.
   unsigned PreferredFusedOpcode = HasFMAD ? ISD::FMAD : ISD::FMA;
   bool Aggressive = TLI.enableAggressiveFMAFusion(VT);
 
   // Is the node an FMUL and contractable either due to global flags or
   // SDNodeFlags.
SDValue DAGCombiner::visitFSUBForFMACombine(SDNode *N) {
   bool CanFuse = Options.UnsafeFPMath || isContractable(N);
   bool AllowFusionGlobally = (Options.AllowFPOpFusion == FPOpFusion::Fast ||
                               CanFuse || HasFMAD);
 
   // If the subtraction is not contractable, do not combine.
   if (!AllowFusionGlobally && !isContractable(N))
     return SDValue();
 
-  if (STI && STI->generateFMAsInMachineCombiner(OptLevel))
+  if (TLI.generateFMAsInMachineCombiner(VT, OptLevel))
     return SDValue();
 
   // Always prefer FMAD to FMA for precision.
   unsigned PreferredFusedOpcode = HasFMAD ? ISD::FMAD : ISD::FMA;
   bool Aggressive = TLI.enableAggressiveFMAFusion(VT);
   bool NoSignedZero = Options.NoSignedZerosFPMath || Flags.hasNoSignedZeros();
 
   // Is the node an FMUL and contractable either due to global flags or

llvm/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp

bool AArch64ExpandPseudo::expand_DestructiveOp(
 
   unsigned DstReg = MI.getOperand(0).getReg();
   bool DstIsDead = MI.getOperand(0).isDead();
 
   if (DType == AArch64::DestructiveBinary)
     assert(DstReg != MI.getOperand(3).getReg());
 
   bool UseRev = false;
-  unsigned PredIdx, DOPIdx, SrcIdx;
+  unsigned PredIdx, DOPIdx, SrcIdx, Src2Idx;
   switch (DType) {
   case AArch64::DestructiveBinaryComm:
   case AArch64::DestructiveBinaryCommWithRev:
     if (DstReg == MI.getOperand(3).getReg()) {
       // FSUB Zd, Pg, Zs1, Zd  ==> FSUBR   Zd, Pg/m, Zd, Zs1
       std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(1, 3, 2);
       UseRev = true;
       break;
     }
     LLVM_FALLTHROUGH;
   case AArch64::DestructiveBinary:
   case AArch64::DestructiveBinaryImm:
     std::tie(PredIdx, DOPIdx, SrcIdx) = std::make_tuple(1, 2, 3);
     break;
+  case AArch64::DestructiveTernaryCommWithRev:
+    std::tie(PredIdx, DOPIdx, SrcIdx, Src2Idx) = std::make_tuple(1, 2, 3, 4);
+    if (DstReg == MI.getOperand(3).getReg()) {
+      // FMLA Zd, Pg, Za, Zd, Zm ==> FMAD Zdn, Pg, Zm, Za
+      std::tie(PredIdx, DOPIdx, SrcIdx, Src2Idx) = std::make_tuple(1, 3, 4, 2);
+      UseRev = true;
+    } else if (DstReg == MI.getOperand(4).getReg()) {
+      // FMLA Zd, Pg, Za, Zm, Zd ==> FMAD Zdn, Pg, Zm, Za
+      std::tie(PredIdx, DOPIdx, SrcIdx, Src2Idx) = std::make_tuple(1, 4, 3, 2);
+      UseRev = true;
+    }
+    break;
   default:
     llvm_unreachable("Unsupported Destructive Operand type");
   }
 
 #ifndef NDEBUG
   // MOVPRFX can only be used if the destination operand
   // is the destructive operand, not as any other operand,
   // so the Destructive Operand must be unique.
   bool DOPRegIsUnique = false;
   switch (DType) {
   case AArch64::DestructiveBinaryComm:
   case AArch64::DestructiveBinaryCommWithRev:
     DOPRegIsUnique =
       DstReg != MI.getOperand(DOPIdx).getReg() ||
       MI.getOperand(DOPIdx).getReg() != MI.getOperand(SrcIdx).getReg();
     break;
   case AArch64::DestructiveBinaryImm:
     DOPRegIsUnique = true;
     break;
+  case AArch64::DestructiveTernaryCommWithRev:
+    DOPRegIsUnique =
+        DstReg != MI.getOperand(DOPIdx).getReg() ||
+        (MI.getOperand(DOPIdx).getReg() != MI.getOperand(SrcIdx).getReg() &&
+         MI.getOperand(DOPIdx).getReg() != MI.getOperand(Src2Idx).getReg());
+    break;
   }
 #endif
 
   // Resolve the reverse opcode
   if (UseRev) {
     int NewOpcode;
     // e.g. DIV -> DIVR
     if ((NewOpcode = AArch64::getSVERevInstr(Opcode)) != -1)
#endif
   switch (DType) {
   case AArch64::DestructiveBinaryImm:
   case AArch64::DestructiveBinaryComm:
   case AArch64::DestructiveBinaryCommWithRev:
     DOP.add(MI.getOperand(PredIdx))
        .addReg(MI.getOperand(DOPIdx).getReg(), RegState::Kill)
        .add(MI.getOperand(SrcIdx));
     break;
+  case AArch64::DestructiveTernaryCommWithRev:
+    DOP.add(MI.getOperand(PredIdx))
+        .addReg(MI.getOperand(DOPIdx).getReg(), RegState::Kill)
+        .add(MI.getOperand(SrcIdx))
+        .add(MI.getOperand(Src2Idx));
+    break;
   }
 
   if (PRFX) {
     finalizeBundle(MBB, PRFX->getIterator(), MBBI->getIterator());
     transferImpOps(MI, PRFX, DOP);
   } else
     transferImpOps(MI, DOP, DOP);
 

llvm/lib/Target/AArch64/AArch64ISelLowering.h

public:
 
   /// Return true if an FMA operation is faster than a pair of fmul and fadd
   /// instructions. fmuladd intrinsics will be expanded to FMAs when this method
   /// returns true, otherwise fmuladd is expanded to fmul + fadd.
   bool isFMAFasterThanFMulAndFAdd(const MachineFunction &MF,
                                   EVT VT) const override;
   bool isFMAFasterThanFMulAndFAdd(const Function &F, Type *Ty) const override;
 
+  bool generateFMAsInMachineCombiner(EVT VT,
+                                     CodeGenOpt::Level OptLevel) const override;
+
   const MCPhysReg *getScratchRegisters(CallingConv::ID CC) const override;
 
   /// Returns false if N is a bit extraction pattern of (X >> C) & Mask.
   bool isDesirableToCommuteWithShift(const SDNode *N,
                                      CombineLevel Level) const override;
 
   /// Returns true if it is beneficial to convert a load of a constant
   /// to just the constant itself.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 bool AArch64TargetLowering::isFMAFasterThanFMulAndFAdd(
     const MachineFunction &MF, EVT VT) const {
   VT = VT.getScalarType();
 
   if (!VT.isSimple())
     return false;
 
   switch (VT.getSimpleVT().SimpleTy) {
   case MVT::f32:
   case MVT::f64:
     return true;
   default:
     break;
   }
 
   return false;
 }
 
 bool AArch64TargetLowering::isFMAFasterThanFMulAndFAdd(const Function &F,
                                                        Type *Ty) const {
   switch (Ty->getScalarType()->getTypeID()) {
   case Type::FloatTyID:
   case Type::DoubleTyID:
     return true;
   default:
     return false;
   }
 }
 
+bool AArch64TargetLowering::generateFMAsInMachineCombiner(
+    EVT VT, CodeGenOpt::Level OptLevel) const {
+  return (OptLevel >= CodeGenOpt::Aggressive) && !VT.isScalableVector();
+}
+
 const MCPhysReg *
 AArch64TargetLowering::getScratchRegisters(CallingConv::ID) const {
   // LR is a callee-save register, but we must treat it as clobbered by any call
   // site. Hence we include LR in the scratch registers, which are in turn added
   // as implicit-defs for stackmaps and patchpoints.
   static const MCPhysReg ScratchRegs[] = {
     AArch64::X16, AArch64::X17, AArch64::LR, 0
   };

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVE] in {
   defm FRECPS_ZZZ  : sve_fp_3op_u_zd<0b110, "frecps",  int_aarch64_sve_frecps_x>;
   defm FRSQRTS_ZZZ : sve_fp_3op_u_zd<0b111, "frsqrts", int_aarch64_sve_frsqrts_x>;
 
   defm FTSSEL_ZZZ : sve_int_bin_cons_misc_0_b<"ftssel", int_aarch64_sve_ftssel_x>;
 
   defm FCADD_ZPmZ : sve_fp_fcadd<"fcadd", int_aarch64_sve_fcadd>;
   defm FCMLA_ZPmZZ : sve_fp_fcmla<"fcmla", int_aarch64_sve_fcmla>;
 
-  defm FMLA_ZPmZZ  : sve_fp_3op_p_zds_a<0b00, "fmla",  int_aarch64_sve_fmla>;
-  defm FMLS_ZPmZZ  : sve_fp_3op_p_zds_a<0b01, "fmls",  int_aarch64_sve_fmls>;
-  defm FNMLA_ZPmZZ : sve_fp_3op_p_zds_a<0b10, "fnmla", int_aarch64_sve_fnmla>;
-  defm FNMLS_ZPmZZ : sve_fp_3op_p_zds_a<0b11, "fnmls", int_aarch64_sve_fnmls>;
-
-  defm FMAD_ZPmZZ  : sve_fp_3op_p_zds_b<0b00, "fmad",  int_aarch64_sve_fmad>;
-  defm FMSB_ZPmZZ  : sve_fp_3op_p_zds_b<0b01, "fmsb",  int_aarch64_sve_fmsb>;
-  defm FNMAD_ZPmZZ : sve_fp_3op_p_zds_b<0b10, "fnmad", int_aarch64_sve_fnmad>;
-  defm FNMSB_ZPmZZ : sve_fp_3op_p_zds_b<0b11, "fnmsb", int_aarch64_sve_fnmsb>;
+  defm FMLA_ZPmZZ  : sve_fp_3op_p_zds_a<0b00, "fmla",  "FMLA_ZPZZZ", int_aarch64_sve_fmla, "FMAD_ZPmZZ">;
+  defm FMLS_ZPmZZ  : sve_fp_3op_p_zds_a<0b01, "fmls",  "FMLS_ZPZZZ", int_aarch64_sve_fmls, "FMSB_ZPmZZ">;
+  defm FNMLA_ZPmZZ : sve_fp_3op_p_zds_a<0b10, "fnmla", "FNMLA_ZPZZZ", int_aarch64_sve_fnmla, "FNMAD_ZPmZZ">;
+  defm FNMLS_ZPmZZ : sve_fp_3op_p_zds_a<0b11, "fnmls", "FNMLS_ZPZZZ", int_aarch64_sve_fnmls, "FNMSB_ZPmZZ">;
+
+  defm FMAD_ZPmZZ  : sve_fp_3op_p_zds_b<0b00, "fmad",  int_aarch64_sve_fmad, "FMLA_ZPmZZ", /*isReverseInstr*/ 1>;
+  defm FMSB_ZPmZZ  : sve_fp_3op_p_zds_b<0b01, "fmsb",  int_aarch64_sve_fmsb, "FMLS_ZPmZZ", /*isReverseInstr*/ 1>;
+  defm FNMAD_ZPmZZ : sve_fp_3op_p_zds_b<0b10, "fnmad", int_aarch64_sve_fnmad, "FNMLA_ZPmZZ", /*isReverseInstr*/ 1>;
+  defm FNMSB_ZPmZZ : sve_fp_3op_p_zds_b<0b11, "fnmsb", int_aarch64_sve_fnmsb, "FNMLS_ZPmZZ", /*isReverseInstr*/ 1>;
+
+  defm FMLA_ZPZZZ  : sve_fp_3op_p_zds_zx<int_aarch64_sve_fmla, int_aarch64_sve_fmad>;
+  defm FMLS_ZPZZZ  : sve_fp_3op_p_zds_zx<int_aarch64_sve_fmls, int_aarch64_sve_fmsb>;
+  defm FNMLA_ZPZZZ : sve_fp_3op_p_zds_zx<int_aarch64_sve_fnmla, int_aarch64_sve_fnmad>;
+  defm FNMLS_ZPZZZ : sve_fp_3op_p_zds_zx<int_aarch64_sve_fnmls, int_aarch64_sve_fnmsb>;
 
   // Add patterns for FMA where disabled lanes are undef.
-  // FIXME: Implement a pseudo so we can choose a better instruction after
-  // regalloc.
   def : Pat<(nxv8f16 (AArch64fma_p nxv8i1:$P, nxv8f16:$Op1, nxv8f16:$Op2, nxv8f16:$Op3)),
-            (FMLA_ZPmZZ_H $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_H $P, $Op3, $Op1, $Op2)>;
   def : Pat<(nxv4f16 (AArch64fma_p nxv4i1:$P, nxv4f16:$Op1, nxv4f16:$Op2, nxv4f16:$Op3)),
-            (FMLA_ZPmZZ_H $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_H $P, $Op3, $Op1, $Op2)>;
   def : Pat<(nxv2f16 (AArch64fma_p nxv2i1:$P, nxv2f16:$Op1, nxv2f16:$Op2, nxv2f16:$Op3)),
-            (FMLA_ZPmZZ_H $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_H $P, $Op3, $Op1, $Op2)>;
   def : Pat<(nxv4f32 (AArch64fma_p nxv4i1:$P, nxv4f32:$Op1, nxv4f32:$Op2, nxv4f32:$Op3)),
-            (FMLA_ZPmZZ_S $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_S $P, $Op3, $Op1, $Op2)>;
   def : Pat<(nxv2f32 (AArch64fma_p nxv2i1:$P, nxv2f32:$Op1, nxv2f32:$Op2, nxv2f32:$Op3)),
-            (FMLA_ZPmZZ_S $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_S $P, $Op3, $Op1, $Op2)>;
   def : Pat<(nxv2f64 (AArch64fma_p nxv2i1:$P, nxv2f64:$Op1, nxv2f64:$Op2, nxv2f64:$Op3)),
-            (FMLA_ZPmZZ_D $P, $Op3, $Op1, $Op2)>;
+            (FMLA_ZPZZZ_UNDEF_D $P, $Op3, $Op1, $Op2)>;
 
   defm FTMAD_ZZI : sve_fp_ftmad<"ftmad", int_aarch64_sve_ftmad_x>;
 
   defm FMLA_ZZZI : sve_fp_fma_by_indexed_elem<0b0, "fmla", int_aarch64_sve_fmla_lane>;
   defm FMLS_ZZZI : sve_fp_fma_by_indexed_elem<0b1, "fmls", int_aarch64_sve_fmls_lane>;
 
   defm FCMLA_ZZZI : sve_fp_fcmla_by_indexed_elem<"fcmla", int_aarch64_sve_fcmla_lane>;
   defm FMUL_ZZZI   : sve_fp_fmul_by_indexed_elem<"fmul", int_aarch64_sve_fmul_lane>;

 
   // 8-element contiguous store
   defm : st1<ST1B_H, ST1B_H_IMM, nxv8i16, AArch64st1, nxv8i1, nxv8i8,  am_sve_regreg_lsl0>;
   defm : st1<ST1H,   ST1H_IMM,   nxv8i16, AArch64st1, nxv8i1, nxv8i16, am_sve_regreg_lsl1>;
 
   // 16-element contiguous store
   defm : st1<ST1B, ST1B_IMM,   nxv16i8, AArch64st1, nxv16i1, nxv16i8, am_sve_regreg_lsl0>;
 
+  // Zd = Za + Zn * Zm
+  def : Pat<(nxv8f16 (AArch64fma_p nxv8i1:$P, nxv8f16:$Zn, nxv8f16:$Zm, nxv8f16:$Za)),
+            (FMLA_ZPZZZ_UNDEF_H $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv4f32 (AArch64fma_p nxv4i1:$P, nxv4f32:$Zn, nxv4f32:$Zm, nxv4f32:$Za)),
+            (FMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f32 (AArch64fma_p nxv2i1:$P, nxv2f32:$Zn, nxv2f32:$Zm, nxv2f32:$Za)),
+            (FMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f64 (AArch64fma_p nxv2i1:$P, nxv2f64:$Zn, nxv2f64:$Zm, nxv2f64:$Za)),
+            (FMLA_ZPZZZ_UNDEF_D $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zd = Za + -Zn * Zm
+  def : Pat<(nxv8f16 (AArch64fma_p nxv8i1:$P, (AArch64fneg_mt nxv8i1:$P, nxv8f16:$Zn, (nxv8f16 (undef))), nxv8f16:$Zm, nxv8f16:$Za)),
+            (FMLS_ZPZZZ_UNDEF_H $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv4f32 (AArch64fma_p nxv4i1:$P, (AArch64fneg_mt nxv4i1:$P, nxv4f32:$Zn, (nxv4f32 (undef))), nxv4f32:$Zm, nxv4f32:$Za)),
+            (FMLS_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f32 (AArch64fma_p nxv2i1:$P, (AArch64fneg_mt nxv2i1:$P, nxv2f32:$Zn, (nxv2f32 (undef))), nxv2f32:$Zm, nxv2f32:$Za)),
+            (FMLS_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f64 (AArch64fma_p nxv2i1:$P, (AArch64fneg_mt nxv2i1:$P, nxv2f64:$Zn, (nxv2f64 (undef))), nxv2f64:$Zm, nxv2f64:$Za)),
+            (FMLS_ZPZZZ_UNDEF_D $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zd = -Za + Zn * Zm
+  def : Pat<(nxv8f16 (AArch64fma_p nxv8i1:$P, nxv8f16:$Zn, nxv8f16:$Zm, (AArch64fneg_mt nxv8i1:$P, nxv8f16:$Za, (nxv8f16 (undef))))),
+            (FNMLS_ZPZZZ_UNDEF_H $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv4f32 (AArch64fma_p nxv4i1:$P, nxv4f32:$Zn, nxv4f32:$Zm, (AArch64fneg_mt nxv4i1:$P, nxv4f32:$Za, (nxv4f32 (undef))))),
+            (FNMLS_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f32 (AArch64fma_p nxv2i1:$P, nxv2f32:$Zn, nxv2f32:$Zm, (AArch64fneg_mt nxv2i1:$P, nxv2f32:$Za, (nxv2f32 (undef))))),
+            (FNMLS_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f64 (AArch64fma_p nxv2i1:$P, nxv2f64:$Zn, nxv2f64:$Zm, (AArch64fneg_mt nxv2i1:$P, nxv2f64:$Za, (nxv2f64 (undef))))),
+            (FNMLS_ZPZZZ_UNDEF_D $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zd = -Za + -Zn * Zm
+  def : Pat<(nxv8f16 (AArch64fma_p nxv8i1:$P, (AArch64fneg_mt nxv8i1:$P, nxv8f16:$Zn, (nxv8f16 (undef))), nxv8f16:$Zm, (AArch64fneg_mt nxv8i1:$P, nxv8f16:$Za, (nxv8f16 (undef))))),
+            (FNMLA_ZPZZZ_UNDEF_H $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv4f32 (AArch64fma_p nxv4i1:$P, (AArch64fneg_mt nxv4i1:$P, nxv4f32:$Zn, (nxv4f32 (undef))), nxv4f32:$Zm, (AArch64fneg_mt nxv4i1:$P, nxv4f32:$Za, (nxv4f32 (undef))))),
+            (FNMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f32 (AArch64fma_p nxv2i1:$P, (AArch64fneg_mt nxv2i1:$P, nxv2f32:$Zn, (nxv2f32 (undef))), nxv2f32:$Zm, (AArch64fneg_mt nxv2i1:$P, nxv2f32:$Za, (nxv2f32 (undef))))),
+            (FNMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(nxv2f64 (AArch64fma_p nxv2i1:$P, (AArch64fneg_mt nxv2i1:$P, nxv2f64:$Zn, (nxv2f64 (undef))), nxv2f64:$Zm, (AArch64fneg_mt nxv2i1:$P, nxv2f64:$Za, (nxv2f64 (undef))))),
+            (FNMLA_ZPZZZ_UNDEF_D $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zd = -(Za + Zn * Zm)
+  def : Pat<(AArch64fneg_mt nxv8i1:$P, (AArch64fma_p nxv8i1:$P, nxv8f16:$Zn, nxv8f16:$Zm, nxv8f16:$Za), (nxv8f16 (undef))),
+            (FNMLA_ZPZZZ_UNDEF_H $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(AArch64fneg_mt nxv4i1:$P, (AArch64fma_p nxv4i1:$P, nxv4f32:$Zn, nxv4f32:$Zm, nxv4f32:$Za), (nxv4f32 (undef))),
+            (FNMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(AArch64fneg_mt nxv2i1:$P, (AArch64fma_p nxv2i1:$P, nxv2f32:$Zn, nxv2f32:$Zm, nxv2f32:$Za), (nxv2f32 (undef))),
+            (FNMLA_ZPZZZ_UNDEF_S $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(AArch64fneg_mt nxv2i1:$P, (AArch64fma_p nxv2i1:$P, nxv2f64:$Zn, nxv2f64:$Zm, nxv2f64:$Za), (nxv2f64 (undef))),
+            (FNMLA_ZPZZZ_UNDEF_D $P, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zda = Zda + Zn * Zm
+  def : Pat<(vselect (nxv8i1 PPR:$Pg), (nxv8f16 (AArch64fma_p (nxv8i1 (AArch64ptrue 31)), ZPR:$Zn, ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLA_ZPmZZ_H PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv4i1 PPR:$Pg), (nxv4f32 (AArch64fma_p (nxv4i1 (AArch64ptrue 31)), ZPR:$Zn, ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLA_ZPmZZ_S PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv2i1 PPR:$Pg), (nxv2f32 (AArch64fma_p (nxv2i1 (AArch64ptrue 31)), ZPR:$Zn, ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLA_ZPmZZ_S PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv2i1 PPR:$Pg), (nxv2f64 (AArch64fma_p (nxv2i1 (AArch64ptrue 31)), ZPR:$Zn, ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLA_ZPmZZ_D PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
+  // Zda = Zda + -Zn * Zm
+  def : Pat<(vselect (nxv8i1 PPR:$Pg), (nxv8f16 (AArch64fma_p (nxv8i1 (AArch64ptrue 31)), (AArch64fneg_mt (nxv8i1 (AArch64ptrue 31)), nxv8f16:$Zn, (nxv8f16 (undef))), ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLS_ZPmZZ_H PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv4i1 PPR:$Pg), (nxv4f32 (AArch64fma_p (nxv4i1 (AArch64ptrue 31)), (AArch64fneg_mt (nxv4i1 (AArch64ptrue 31)), nxv4f32:$Zn, (nxv4f32 (undef))), ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLS_ZPmZZ_S PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv2i1 PPR:$Pg), (nxv2f32 (AArch64fma_p (nxv2i1 (AArch64ptrue 31)), (AArch64fneg_mt (nxv2i1 (AArch64ptrue 31)), nxv2f32:$Zn, (nxv2f32 (undef))), ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLS_ZPmZZ_S PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+  def : Pat<(vselect (nxv2i1 PPR:$Pg), (nxv2f64 (AArch64fma_p (nxv2i1 (AArch64ptrue 31)), (AArch64fneg_mt (nxv2i1 (AArch64ptrue 31)), nxv2f64:$Zn, (nxv2f64 (undef))), ZPR:$Zm, ZPR:$Za)), ZPR:$Za),
+            (FMLS_ZPmZZ_D PPR:$Pg, ZPR:$Za, ZPR:$Zn, ZPR:$Zm)>;
+
   def : Pat<(nxv16i8 (vector_insert (nxv16i8 (undef)), (i32 FPR32:$src), 0)),
             (INSERT_SUBREG (nxv16i8 (IMPLICIT_DEF)), FPR32:$src, ssub)>;
   def : Pat<(nxv8i16 (vector_insert (nxv8i16 (undef)), (i32 FPR32:$src), 0)),
             (INSERT_SUBREG (nxv8i16 (IMPLICIT_DEF)), FPR32:$src, ssub)>;
   def : Pat<(nxv4i32 (vector_insert (nxv4i32 (undef)), (i32 FPR32:$src), 0)),
             (INSERT_SUBREG (nxv4i32 (IMPLICIT_DEF)), FPR32:$src, ssub)>;
   def : Pat<(nxv2i64 (vector_insert (nxv2i64 (undef)), (i64 FPR64:$src), 0)),
             (INSERT_SUBREG (nxv2i64 (IMPLICIT_DEF)), FPR64:$src, dsub)>;

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

public:
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
                                   bool isVolatile,
                                   MachinePointerInfo DstPtrInfo) const override;
   SDValue EmitTargetCodeForSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   MachinePointerInfo DstPtrInfo,
                                   bool ZeroData) const override;
-  bool generateFMAsInMachineCombiner(CodeGenOpt::Level OptLevel) const override;
 };
 }
 
 #endif

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

CLI.setDebugLoc(dl)
                       DAG.getExternalSymbol(bzeroName, IntPtr),
                       std::move(Args))
         .setDiscardResult();
     std::pair<SDValue, SDValue> CallResult = TLI.LowerCallTo(CLI);
     return CallResult.second;
   }
   return SDValue();
 }
-bool AArch64SelectionDAGInfo::generateFMAsInMachineCombiner(
-    CodeGenOpt::Level OptLevel) const {
-  return OptLevel >= CodeGenOpt::Aggressive;
-}
 
 static const int kSetTagLoopThreshold = 176;
 
 static SDValue EmitUnrolledSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Ptr, uint64_t ObjSize,
                                   const MachineMemOperand *BaseMemOperand,
                                   bool ZeroData) {
   MachineFunction &MF = DAG.getMachineFunction();
   unsigned ObjSizeScaled = ObjSize / 16;
 

llvm/lib/Target/AArch64/SVEInstrFormats.td

let hasNoSchedulingInfo = 1 in {
   }
 
   class PredTwoOpImmPseudo<string name, ZPRRegOp zprty, Operand immty,
                            FalseLanesEnum flags = FalseLanesNone>
   : SVEPseudo2Instr<name, 0>,
     Pseudo<(outs zprty:$Zd), (ins PPR3bAny:$Pg, zprty:$Zs1, immty:$imm), []> {
     let FalseLanes = flags;
   }
+
+  class PredThreeOpPseudo<string name, ZPRRegOp zprty,
+                          FalseLanesEnum flags = FalseLanesNone>
+  : SVEPseudo2Instr<name, 0>,
+    Pseudo<(outs zprty:$Zd), (ins PPR3bAny:$Pg, zprty:$Zs1, zprty:$Zs2, zprty:$Zs3), []> {
+    let FalseLanes = flags;
+  }
 }
 
 //===----------------------------------------------------------------------===//
 // SVE Predicate Misc Group
 //===----------------------------------------------------------------------===//
 
 class sve_int_pfalse<bits<6> opc, string asm>
 : I<(outs PPR8:$Pd), (ins),
: I<(outs zprty:$Zda), (ins PPR3bAny:$Pg, zprty:$_Zda, zprty:$Zn, zprty:$Zm),
   let Inst{20-16} = Zm;
   let Inst{15}    = 0b0;
   let Inst{14-13} = opc;
   let Inst{12-10} = Pg;
   let Inst{9-5}   = Zn;
   let Inst{4-0}   = Zda;
 
   let Constraints = "$Zda = $_Zda";
-  let DestructiveInstType = DestructiveOther;
   let ElementSize = zprty.ElementSize;
 }
 
-multiclass sve_fp_3op_p_zds_a<bits<2> opc, string asm, SDPatternOperator op> {
-  def _H : sve_fp_3op_p_zds_a<0b01, opc, asm, ZPR16>;
-  def _S : sve_fp_3op_p_zds_a<0b10, opc, asm, ZPR32>;
-  def _D : sve_fp_3op_p_zds_a<0b11, opc, asm, ZPR64>;
+multiclass sve_fp_3op_p_zds_a<bits<2> opc, string asm, string Ps,
+                              SDPatternOperator op, string revname,
+                              bit isReverseInstr=0> {
+  let DestructiveInstType = DestructiveTernaryCommWithRev in {
+  def _H : sve_fp_3op_p_zds_a<0b01, opc, asm, ZPR16>,
+           SVEPseudo2Instr<Ps # _H, 1>, SVEInstr2Rev<NAME # _H, revname # _H, isReverseInstr>;
+  def _S : sve_fp_3op_p_zds_a<0b10, opc, asm, ZPR32>,
+           SVEPseudo2Instr<Ps # _S, 1>, SVEInstr2Rev<NAME # _S, revname # _S, isReverseInstr>;
+  def _D : sve_fp_3op_p_zds_a<0b11, opc, asm, ZPR64>,
+           SVEPseudo2Instr<Ps # _D, 1>, SVEInstr2Rev<NAME # _D, revname # _D, isReverseInstr>;
+  }
 
   def : SVE_4_Op_Pat<nxv8f16, op, nxv8i1, nxv8f16, nxv8f16, nxv8f16, !cast<Instruction>(NAME # _H)>;
   def : SVE_4_Op_Pat<nxv4f32, op, nxv4i1, nxv4f32, nxv4f32, nxv4f32, !cast<Instruction>(NAME # _S)>;
   def : SVE_4_Op_Pat<nxv2f64, op, nxv2i1, nxv2f64, nxv2f64, nxv2f64, !cast<Instruction>(NAME # _D)>;
 }
 
 class sve_fp_3op_p_zds_b<bits<2> sz, bits<2> opc, string asm,
                          ZPRRegOp zprty>
: I<(outs zprty:$Zdn), (ins PPR3bAny:$Pg, zprty:$_Zdn, zprty:$Zm, zprty:$Za),
   let Inst{9-5}   = Zm;
   let Inst{4-0}   = Zdn;
 
   let Constraints = "$Zdn = $_Zdn";
   let DestructiveInstType = DestructiveOther;
   let ElementSize = zprty.ElementSize;
 }
 
-multiclass sve_fp_3op_p_zds_b<bits<2> opc, string asm, SDPatternOperator op> {
-  def _H : sve_fp_3op_p_zds_b<0b01, opc, asm, ZPR16>;
-  def _S : sve_fp_3op_p_zds_b<0b10, opc, asm, ZPR32>;
-  def _D : sve_fp_3op_p_zds_b<0b11, opc, asm, ZPR64>;
+multiclass sve_fp_3op_p_zds_b<bits<2> opc, string asm, SDPatternOperator op,
+                              string revname, bit isReverseInstr> {
+  def _H : sve_fp_3op_p_zds_b<0b01, opc, asm, ZPR16>,
+           SVEInstr2Rev<NAME # _H, revname # _H, isReverseInstr>;
+  def _S : sve_fp_3op_p_zds_b<0b10, opc, asm, ZPR32>,
+           SVEInstr2Rev<NAME # _S, revname # _S, isReverseInstr>;
+  def _D : sve_fp_3op_p_zds_b<0b11, opc, asm, ZPR64>,
+           SVEInstr2Rev<NAME # _D, revname # _D, isReverseInstr>;
 
   def : SVE_4_Op_Pat<nxv8f16, op, nxv8i1, nxv8f16, nxv8f16, nxv8f16, !cast<Instruction>(NAME # _H)>;
   def : SVE_4_Op_Pat<nxv4f32, op, nxv4i1, nxv4f32, nxv4f32, nxv4f32, !cast<Instruction>(NAME # _S)>;
   def : SVE_4_Op_Pat<nxv2f64, op, nxv2i1, nxv2f64, nxv2f64, nxv2f64, !cast<Instruction>(NAME # _D)>;
 }
 
+multiclass sve_fp_3op_p_zds_zx<SDPatternOperator op, SDPatternOperator rev_op> {
+  def _UNDEF_H : PredThreeOpPseudo<NAME # _H, ZPR16, FalseLanesUndef>;
+  def _UNDEF_S : PredThreeOpPseudo<NAME # _S, ZPR32, FalseLanesUndef>;
+  def _UNDEF_D : PredThreeOpPseudo<NAME # _D, ZPR64, FalseLanesUndef>;
+}
+
 //===----------------------------------------------------------------------===//
 // SVE Floating Point Multiply-Add - Indexed Group
 //===----------------------------------------------------------------------===//
 
 class sve_fp_fma_by_indexed_elem<bits<2> sz, bit opc, string asm,
                                  ZPRRegOp zprty1,
                                  ZPRRegOp zprty2, Operand itype>
 : I<(outs zprty1:$Zda), (ins zprty1:$_Zda, zprty1:$Zn, zprty2:$Zm, itype:$iop),

llvm/test/CodeGen/AArch64/sve-fixed-length-fp-arith.ll

 }
 
 define void @fma_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x half>* %c) #0 {
 ; CHECK-LABEL: fma_v16f16:
 ; CHECK: ptrue [[PG:p[0-9]+]].h, vl[[#min(div(VBYTES,2),16)]]
 ; CHECK-DAG: ld1h { [[OP1:z[0-9]+]].h }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1h { [[OP2:z[0-9]+]].h }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1h { [[OP3:z[0-9]+]].h }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].h, [[PG]]/m, [[OP1]].h, [[OP2]].h
-; CHECK: st1h { [[OP3]].h }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].h, [[PG]]/m, [[OP2]].h, [[OP3]].h
+; CHECK: st1h { [[OP1]].h }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %op3 = load <16 x half>, <16 x half>* %c
   %res = call <16 x half> @llvm.fma.v16f16(<16 x half> %op1, <16 x half> %op2, <16 x half> %op3)
   store <16 x half> %res, <16 x half>* %a
   ret void
 }
 
 define void @fma_v32f16(<32 x half>* %a, <32 x half>* %b, <32 x half>* %c) #0 {
 ; CHECK-LABEL: fma_v32f16:
 ; CHECK: ptrue [[PG:p[0-9]+]].h, vl[[#min(div(VBYTES,2),32)]]
 ; CHECK-DAG: ld1h { [[OP1:z[0-9]+]].h }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1h { [[OP2:z[0-9]+]].h }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1h { [[OP3:z[0-9]+]].h }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].h, [[PG]]/m, [[OP1]].h, [[OP2]].h
-; CHECK: st1h { [[OP3]].h }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].h, [[PG]]/m, [[OP2]].h, [[OP3]].h
+; CHECK: st1h { [[OP1]].h }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <32 x half>, <32 x half>* %a
   %op2 = load <32 x half>, <32 x half>* %b
   %op3 = load <32 x half>, <32 x half>* %c
   %res = call <32 x half> @llvm.fma.v32f16(<32 x half> %op1, <32 x half> %op2, <32 x half> %op3)
   store <32 x half> %res, <32 x half>* %a
   ret void
 }
 
 define void @fma_v64f16(<64 x half>* %a, <64 x half>* %b, <64 x half>* %c) #0 {
 ; CHECK-LABEL: fma_v64f16:
 ; CHECK: ptrue [[PG:p[0-9]+]].h, vl[[#min(div(VBYTES,2),64)]]
 ; CHECK-DAG: ld1h { [[OP1:z[0-9]+]].h }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1h { [[OP2:z[0-9]+]].h }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1h { [[OP3:z[0-9]+]].h }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].h, [[PG]]/m, [[OP1]].h, [[OP2]].h
-; CHECK: st1h { [[OP3]].h }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].h, [[PG]]/m, [[OP2]].h, [[OP3]].h
+; CHECK: st1h { [[OP1]].h }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <64 x half>, <64 x half>* %a
   %op2 = load <64 x half>, <64 x half>* %b
   %op3 = load <64 x half>, <64 x half>* %c
   %res = call <64 x half> @llvm.fma.v64f16(<64 x half> %op1, <64 x half> %op2, <64 x half> %op3)
   store <64 x half> %res, <64 x half>* %a
   ret void
 }
 
 define void @fma_v128f16(<128 x half>* %a, <128 x half>* %b, <128 x half>* %c) #0 {
 ; CHECK-LABEL: fma_v128f16:
 ; CHECK: ptrue [[PG:p[0-9]+]].h, vl[[#min(div(VBYTES,2),128)]]
 ; CHECK-DAG: ld1h { [[OP1:z[0-9]+]].h }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1h { [[OP2:z[0-9]+]].h }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1h { [[OP3:z[0-9]+]].h }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].h, [[PG]]/m, [[OP1]].h, [[OP2]].h
-; CHECK: st1h { [[OP3]].h }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].h, [[PG]]/m, [[OP2]].h, [[OP3]].h
+; CHECK: st1h { [[OP1]].h }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <128 x half>, <128 x half>* %a
   %op2 = load <128 x half>, <128 x half>* %b
   %op3 = load <128 x half>, <128 x half>* %c
   %res = call <128 x half> @llvm.fma.v128f16(<128 x half> %op1, <128 x half> %op2, <128 x half> %op3)
   store <128 x half> %res, <128 x half>* %a
   ret void
 }

 }
 
 define void @fma_v8f32(<8 x float>* %a, <8 x float>* %b, <8 x float>* %c) #0 {
 ; CHECK-LABEL: fma_v8f32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl[[#min(div(VBYTES,4),8)]]
 ; CHECK-DAG: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1w { [[OP3:z[0-9]+]].s }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
-; CHECK: st1w { [[OP3]].s }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].s, [[PG]]/m, [[OP2]].s, [[OP3]].s
+; CHECK: st1w { [[OP1]].s }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <8 x float>, <8 x float>* %a
   %op2 = load <8 x float>, <8 x float>* %b
   %op3 = load <8 x float>, <8 x float>* %c
   %res = call <8 x float> @llvm.fma.v8f32(<8 x float> %op1, <8 x float> %op2, <8 x float> %op3)
   store <8 x float> %res, <8 x float>* %a
   ret void
 }
 
 define void @fma_v16f32(<16 x float>* %a, <16 x float>* %b, <16 x float>* %c) #0 {
 ; CHECK-LABEL: fma_v16f32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl[[#min(div(VBYTES,4),16)]]
 ; CHECK-DAG: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1w { [[OP3:z[0-9]+]].s }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
-; CHECK: st1w { [[OP3]].s }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].s, [[PG]]/m, [[OP2]].s, [[OP3]].s
+; CHECK: st1w { [[OP1]].s }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <16 x float>, <16 x float>* %a
   %op2 = load <16 x float>, <16 x float>* %b
   %op3 = load <16 x float>, <16 x float>* %c
   %res = call <16 x float> @llvm.fma.v16f32(<16 x float> %op1, <16 x float> %op2, <16 x float> %op3)
   store <16 x float> %res, <16 x float>* %a
   ret void
 }
 
 define void @fma_v32f32(<32 x float>* %a, <32 x float>* %b, <32 x float>* %c) #0 {
 ; CHECK-LABEL: fma_v32f32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl[[#min(div(VBYTES,4),32)]]
 ; CHECK-DAG: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1w { [[OP3:z[0-9]+]].s }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
-; CHECK: st1w { [[OP3]].s }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].s, [[PG]]/m, [[OP2]].s, [[OP3]].s
+; CHECK: st1w { [[OP1]].s }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <32 x float>, <32 x float>* %a
   %op2 = load <32 x float>, <32 x float>* %b
   %op3 = load <32 x float>, <32 x float>* %c
   %res = call <32 x float> @llvm.fma.v32f32(<32 x float> %op1, <32 x float> %op2, <32 x float> %op3)
   store <32 x float> %res, <32 x float>* %a
   ret void
 }
 
 define void @fma_v64f32(<64 x float>* %a, <64 x float>* %b, <64 x float>* %c) #0 {
 ; CHECK-LABEL: fma_v64f32:
 ; CHECK: ptrue [[PG:p[0-9]+]].s, vl[[#min(div(VBYTES,4),64)]]
 ; CHECK-DAG: ld1w { [[OP1:z[0-9]+]].s }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1w { [[OP2:z[0-9]+]].s }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1w { [[OP3:z[0-9]+]].s }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].s, [[PG]]/m, [[OP1]].s, [[OP2]].s
-; CHECK: st1w { [[OP3]].s }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].s, [[PG]]/m, [[OP2]].s, [[OP3]].s
+; CHECK: st1w { [[OP1]].s }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <64 x float>, <64 x float>* %a
   %op2 = load <64 x float>, <64 x float>* %b
   %op3 = load <64 x float>, <64 x float>* %c
   %res = call <64 x float> @llvm.fma.v64f32(<64 x float> %op1, <64 x float> %op2, <64 x float> %op3)
   store <64 x float> %res, <64 x float>* %a
   ret void
 }

 }
 
 define void @fma_v4f64(<4 x double>* %a, <4 x double>* %b, <4 x double>* %c) #0 {
 ; CHECK-LABEL: fma_v4f64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl[[#min(div(VBYTES,8),4)]]
 ; CHECK-DAG: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1d { [[OP3:z[0-9]+]].d }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
-; CHECK: st1d { [[OP3]].d }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].d, [[PG]]/m, [[OP2]].d, [[OP3]].d
+; CHECK: st1d { [[OP1]].d }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <4 x double>, <4 x double>* %a
   %op2 = load <4 x double>, <4 x double>* %b
   %op3 = load <4 x double>, <4 x double>* %c
   %res = call <4 x double> @llvm.fma.v4f64(<4 x double> %op1, <4 x double> %op2, <4 x double> %op3)
   store <4 x double> %res, <4 x double>* %a
   ret void
 }
 
 define void @fma_v8f64(<8 x double>* %a, <8 x double>* %b, <8 x double>* %c) #0 {
 ; CHECK-LABEL: fma_v8f64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl[[#min(div(VBYTES,8),8)]]
 ; CHECK-DAG: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1d { [[OP3:z[0-9]+]].d }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
-; CHECK: st1d { [[OP3]].d }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].d, [[PG]]/m, [[OP2]].d, [[OP3]].d
+; CHECK: st1d { [[OP1]].d }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <8 x double>, <8 x double>* %a
   %op2 = load <8 x double>, <8 x double>* %b
   %op3 = load <8 x double>, <8 x double>* %c
   %res = call <8 x double> @llvm.fma.v8f64(<8 x double> %op1, <8 x double> %op2, <8 x double> %op3)
   store <8 x double> %res, <8 x double>* %a
   ret void
 }
 
 define void @fma_v16f64(<16 x double>* %a, <16 x double>* %b, <16 x double>* %c) #0 {
 ; CHECK-LABEL: fma_v16f64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl[[#min(div(VBYTES,8),16)]]
 ; CHECK-DAG: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1d { [[OP3:z[0-9]+]].d }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
-; CHECK: st1d { [[OP3]].d }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].d, [[PG]]/m, [[OP2]].d, [[OP3]].d
+; CHECK: st1d { [[OP1]].d }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <16 x double>, <16 x double>* %a
   %op2 = load <16 x double>, <16 x double>* %b
   %op3 = load <16 x double>, <16 x double>* %c
   %res = call <16 x double> @llvm.fma.v16f64(<16 x double> %op1, <16 x double> %op2, <16 x double> %op3)
   store <16 x double> %res, <16 x double>* %a
   ret void
 }
 
 define void @fma_v32f64(<32 x double>* %a, <32 x double>* %b, <32 x double>* %c) #0 {
 ; CHECK-LABEL: fma_v32f64:
 ; CHECK: ptrue [[PG:p[0-9]+]].d, vl[[#min(div(VBYTES,8),32)]]
 ; CHECK-DAG: ld1d { [[OP1:z[0-9]+]].d }, [[PG]]/z, [x0]
 ; CHECK-DAG: ld1d { [[OP2:z[0-9]+]].d }, [[PG]]/z, [x1]
 ; CHECK-DAG: ld1d { [[OP3:z[0-9]+]].d }, [[PG]]/z, [x2]
-; CHECK: fmla [[OP3]].d, [[PG]]/m, [[OP1]].d, [[OP2]].d
-; CHECK: st1d { [[OP3]].d }, [[PG]], [x0]
+; CHECK: fmad [[OP1]].d, [[PG]]/m, [[OP2]].d, [[OP3]].d
+; CHECK: st1d { [[OP1]].d }, [[PG]], [x0]
 ; CHECK: ret
   %op1 = load <32 x double>, <32 x double>* %a
   %op2 = load <32 x double>, <32 x double>* %b
   %op3 = load <32 x double>, <32 x double>* %c
   %res = call <32 x double> @llvm.fma.v32f64(<32 x double> %op1, <32 x double> %op2, <32 x double> %op3)
   store <32 x double> %res, <32 x double>* %a
   ret void
 }

llvm/test/CodeGen/AArch64/sve-fma-dagcombine.ll

+; RUN: llc -march=aarch64 -mattr=+sve < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-none--elf"
+
+define <vscale x 4 x float> @use_fma(<vscale x 4 x float> %a, <vscale x 4 x float> %b, <vscale x 4 x float> %c) {
+; CHECK-LABEL: use_fma
+; CHECK: fmad
+  %mul = fmul fast <vscale x 4 x float> %a, %b
+  %res = fadd fast <vscale x 4 x float> %mul, %c
+  ret <vscale x 4 x float> %res
+}

llvm/test/CodeGen/AArch64/sve-fp-combine.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+define <vscale x 4 x float> @fmla_s_sel(<vscale x 4 x i1> %pred, <vscale x 4 x float> %acc,
+                               <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fmla_s_sel:
+; CHECK: fmla z0.s, p0/m, z1.s, z2.s
+; CHECK: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %add = fadd fast <vscale x 4 x float> %acc, %mul
+  %res = select <vscale x 4 x i1> %pred, <vscale x 4 x float> %add, <vscale x 4 x float> %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmla_sx2_sel(<vscale x 2 x i1> %pred, <vscale x 2 x float> %acc,
+                                 <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fmla_sx2_sel:
+; CHECK: fmla z0.s, p0/m, z1.s, z2.s
+; CHECK: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %add = fadd fast <vscale x 2 x float> %acc, %mul
+  %res = select <vscale x 2 x i1> %pred, <vscale x 2 x float> %add, <vscale x 2 x float> %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmla_d_sel(<vscale x 2 x i1> %pred, <vscale x 2 x double> %acc,
+                                <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fmla_d_sel:
+; CHECK: fmla z0.d, p0/m, z1.d, z2.d
+; CHECK: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %add = fadd fast <vscale x 2 x double> %acc, %mul
+  %res = select <vscale x 2 x i1> %pred, <vscale x 2 x double> %add, <vscale x 2 x double> %acc
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fmls_s_sel(<vscale x 4 x i1> %pred, <vscale x 4 x float> %acc,
+                               <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fmls_s_sel:
+; CHECK: fmls z0.s, p0/m, z1.s, z2.s
+; CHECK: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %sub = fsub fast <vscale x 4 x float> %acc, %mul
+  %res = select <vscale x 4 x i1> %pred, <vscale x 4 x float> %sub, <vscale x 4 x float> %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmls_sx2_sel(<vscale x 2 x i1> %pred, <vscale x 2 x float> %acc,
+                                 <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fmls_sx2_sel:
+; CHECK: fmls z0.s, p0/m, z1.s, z2.s
+; CHECK: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %sub = fsub fast <vscale x 2 x float> %acc, %mul
+  %res = select <vscale x 2 x i1> %pred, <vscale x 2 x float> %sub, <vscale x 2 x float> %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmls_d_sel(<vscale x 2 x i1> %pred, <vscale x 2 x double> %acc,
+                                <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fmls_d_sel:
+; CHECK: fmls z0.d, p0/m, z1.d, z2.d
+; CHECK: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %sub = fsub fast <vscale x 2 x double> %acc, %mul
+  %res = select <vscale x 2 x i1> %pred, <vscale x 2 x double> %sub, <vscale x 2 x double> %acc
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fmad_s(<vscale x 4 x float> %m1, <vscale x 4 x float> %m2, <vscale x 4 x float> %acc) {
+; CHECK-LABEL: fmad_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fmad z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fadd fast <vscale x 4 x float> %acc, %mul
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmad_sx2(<vscale x 2 x float> %m1, <vscale x 2 x float> %m2, <vscale x 2 x float> %acc) {
+; CHECK-LABEL: fmad_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmad z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fadd fast <vscale x 2 x float> %acc, %mul
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmad_d(<vscale x 2 x double> %m1, <vscale x 2 x double> %m2, <vscale x 2 x double> %acc) {
+; CHECK-LABEL: fmad_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmad z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fadd fast <vscale x 2 x double> %acc, %mul
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fmla_s(<vscale x 4 x float> %acc, <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fmla_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fadd fast <vscale x 4 x float> %acc, %mul
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmla_sx2(<vscale x 2 x float> %acc, <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fmla_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fadd fast <vscale x 2 x float> %acc, %mul
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmla_d(<vscale x 2 x double> %acc, <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fmla_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmla z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fadd fast <vscale x 2 x double> %acc, %mul
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fmls_s(<vscale x 4 x float> %acc, <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fmls_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fmls z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fsub fast <vscale x 4 x float> %acc, %mul
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmls_sx2(<vscale x 2 x float> %acc, <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fmls_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmls z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fsub fast <vscale x 2 x float> %acc, %mul
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmls_d(<vscale x 2 x double> %acc, <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fmls_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmls z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fsub fast <vscale x 2 x double> %acc, %mul
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fmsb_s(<vscale x 4 x float> %m1, <vscale x 4 x float> %m2, <vscale x 4 x float> %acc) {
+; CHECK-LABEL: fmsb_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fmsb z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fsub fast <vscale x 4 x float> %acc, %mul
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fmsb_sx2(<vscale x 2 x float> %m1, <vscale x 2 x float> %m2, <vscale x 2 x float> %acc) {
+; CHECK-LABEL: fmsb_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmsb z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fsub fast <vscale x 2 x float> %acc, %mul
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fmsb_d(<vscale x 2 x double> %m1, <vscale x 2 x double> %m2, <vscale x 2 x double> %acc) {
+; CHECK-LABEL: fmsb_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fmsb z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fsub fast <vscale x 2 x double> %acc, %mul
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fnmad_s(<vscale x 4 x float> %m1, <vscale x 4 x float> %m2, <vscale x 4 x float> %acc) {
+; CHECK-LABEL: fnmad_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fnmad z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 4 x float>
+               shufflevector (<vscale x 4 x float>
+                 insertelement (<vscale x 4 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 4 x float> undef,
+                 <vscale x 4 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 4 x float> %neg_m1, %m2
+  %res = fsub fast <vscale x 4 x float> %mul, %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fnmad_sx2(<vscale x 2 x float> %m1, <vscale x 2 x float> %m2, <vscale x 2 x float> %acc) {
+; CHECK-LABEL: fnmad_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmad z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 2 x float>
+               shufflevector (<vscale x 2 x float>
+                 insertelement (<vscale x 2 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 2 x float> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 2 x float> %neg_m1, %m2
+  %res = fsub fast <vscale x 2 x float> %mul, %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fnmad_d(<vscale x 2 x double> %m1, <vscale x 2 x double> %m2, <vscale x 2 x double> %acc) {
+; CHECK-LABEL: fnmad_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmad z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 2 x double>
+               shufflevector (<vscale x 2 x double>
+                 insertelement (<vscale x 2 x double> undef, double -0.000000e+00, i32 0),
+                 <vscale x 2 x double> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 2 x double> %neg_m1, %m2
+  %res = fsub fast <vscale x 2 x double> %mul, %acc
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fnmla_s(<vscale x 4 x float> %acc, <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fnmla_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fnmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 4 x float>
+               shufflevector (<vscale x 4 x float>
+                 insertelement (<vscale x 4 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 4 x float> undef,
+                 <vscale x 4 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 4 x float> %neg_m1, %m2
+  %res = fsub fast <vscale x 4 x float> %mul, %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fnmla_sx2(<vscale x 2 x float> %acc, <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fnmla_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 2 x float>
+               shufflevector (<vscale x 2 x float>
+                 insertelement (<vscale x 2 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 2 x float> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 2 x float> %neg_m1, %m2
+  %res = fsub fast <vscale x 2 x float> %mul, %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fnmla_d(<vscale x 2 x double> %acc, <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fnmla_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmla z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %neg_m1 = fsub fast <vscale x 2 x double>
+               shufflevector (<vscale x 2 x double>
+                 insertelement (<vscale x 2 x double> undef, double -0.000000e+00, i32 0),
+                 <vscale x 2 x double> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %m1
+
+  %mul = fmul fast <vscale x 2 x double> %neg_m1, %m2
+  %res = fsub fast <vscale x 2 x double> %mul, %acc
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fnmla_s_reversed(<vscale x 4 x float> %acc, <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fnmla_s_reversed:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fnmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %add = fadd fast <vscale x 4 x float> %mul, %acc
+  %res = fsub fast <vscale x 4 x float>
+               shufflevector (<vscale x 4 x float>
+                 insertelement (<vscale x 4 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 4 x float> undef,
+                 <vscale x 4 x i32> zeroinitializer),
+               %add
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fnmla_sx2_reversed(<vscale x 2 x float> %acc, <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fnmla_sx2_reversed:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmla z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %add = fadd fast <vscale x 2 x float> %mul, %acc
+  %res = fsub fast <vscale x 2 x float>
+               shufflevector (<vscale x 2 x float>
+                 insertelement (<vscale x 2 x float> undef, float -0.000000e+00, i32 0),
+                 <vscale x 2 x float> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %add
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fnmla_d_reversed(<vscale x 2 x double> %acc, <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fnmla_d_reversed:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmla z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %add = fadd fast <vscale x 2 x double> %mul, %acc
+  %res = fsub fast <vscale x 2 x double>
+               shufflevector (<vscale x 2 x double>
+                 insertelement (<vscale x 2 x double> undef, double -0.000000e+00, i32 0),
+                 <vscale x 2 x double> undef,
+                 <vscale x 2 x i32> zeroinitializer),
+               %add
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fnmls_s(<vscale x 4 x float> %acc, <vscale x 4 x float> %m1, <vscale x 4 x float> %m2) {
+; CHECK-LABEL: fnmls_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fnmls z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fsub fast <vscale x 4 x float> %mul, %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fnmls_sx2(<vscale x 2 x float> %acc, <vscale x 2 x float> %m1, <vscale x 2 x float> %m2) {
+; CHECK-LABEL: fnmls_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmls z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fsub fast <vscale x 2 x float> %mul, %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fnmls_d(<vscale x 2 x double> %acc, <vscale x 2 x double> %m1, <vscale x 2 x double> %m2) {
+; CHECK-LABEL: fnmls_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmls z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fsub fast <vscale x 2 x double> %mul, %acc
+  ret <vscale x 2 x double> %res
+}
+
+define <vscale x 4 x float> @fnmsb_s(<vscale x 4 x float> %m1, <vscale x 4 x float> %m2, <vscale x 4 x float> %acc) {
+; CHECK-LABEL: fnmsb_s:
+; CHECK: ptrue [[PG:p[0-9]+]].s
+; CHECK-NEXT: fnmsb z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 4 x float> %m1, %m2
+  %res = fsub fast <vscale x 4 x float> %mul, %acc
+  ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x float> @fnmsb_sx2(<vscale x 2 x float> %m1, <vscale x 2 x float> %m2, <vscale x 2 x float> %acc) {
+; CHECK-LABEL: fnmsb_sx2:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmsb z0.s, [[PG]]/m, z1.s, z2.s
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x float> %m1, %m2
+  %res = fsub fast <vscale x 2 x float> %mul, %acc
+  ret <vscale x 2 x float> %res
+}
+
+define <vscale x 2 x double> @fnmsb_d(<vscale x 2 x double> %m1, <vscale x 2 x double> %m2, <vscale x 2 x double> %acc) {
+; CHECK-LABEL: fnmsb_d:
+; CHECK: ptrue [[PG:p[0-9]+]].d
+; CHECK-NEXT: fnmsb z0.d, [[PG]]/m, z1.d, z2.d
+; CHECK-NEXT: ret
+  %mul = fmul fast <vscale x 2 x double> %m1, %m2
+  %res = fsub fast <vscale x 2 x double> %mul, %acc
+  ret <vscale x 2 x double> %res
+}

llvm/test/CodeGen/AArch64/sve-fp.ll

; CHECK-NEXT:    ret
   %res = fmul <vscale x 2 x double> %a, %b
   ret <vscale x 2 x double> %res
 }
 
 define <vscale x 8 x half> @fma_nxv8f16(<vscale x 8 x half> %a, <vscale x 8 x half> %b, <vscale x 8 x half> %c) {
 ; CHECK-LABEL: fma_nxv8f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    fmla z2.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.h, p0/m, z1.h, z2.h
 ; CHECK-NEXT:    ret
   %r = call <vscale x 8 x half> @llvm.fma.nxv8f16(<vscale x 8 x half> %a, <vscale x 8 x half> %b, <vscale x 8 x half> %c)
   ret <vscale x 8 x half> %r
 }
 
 define <vscale x 4 x half> @fma_nxv4f16(<vscale x 4 x half> %a, <vscale x 4 x half> %b, <vscale x 4 x half> %c) {
 ; CHECK-LABEL: fma_nxv4f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    fmla z2.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.h, p0/m, z1.h, z2.h
 ; CHECK-NEXT:    ret
   %r = call <vscale x 4 x half> @llvm.fma.nxv4f16(<vscale x 4 x half> %a, <vscale x 4 x half> %b, <vscale x 4 x half> %c)
   ret <vscale x 4 x half> %r
 }
 
 define <vscale x 2 x half> @fma_nxv2f16(<vscale x 2 x half> %a, <vscale x 2 x half> %b, <vscale x 2 x half> %c) {
 ; CHECK-LABEL: fma_nxv2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    fmla z2.h, p0/m, z0.h, z1.h
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.h, p0/m, z1.h, z2.h
 ; CHECK-NEXT:    ret
   %r = call <vscale x 2 x half> @llvm.fma.nxv2f16(<vscale x 2 x half> %a, <vscale x 2 x half> %b, <vscale x 2 x half> %c)
   ret <vscale x 2 x half> %r
 }
 
 define <vscale x 4 x float> @fma_nxv4f32(<vscale x 4 x float> %a, <vscale x 4 x float> %b, <vscale x 4 x float> %c) {
 ; CHECK-LABEL: fma_nxv4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    fmla z2.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.s, p0/m, z1.s, z2.s
 ; CHECK-NEXT:    ret
   %r = call <vscale x 4 x float> @llvm.fma.nxv4f32(<vscale x 4 x float> %a, <vscale x 4 x float> %b, <vscale x 4 x float> %c)
   ret <vscale x 4 x float> %r
 }
 
 define <vscale x 2 x float> @fma_nxv2f32(<vscale x 2 x float> %a, <vscale x 2 x float> %b, <vscale x 2 x float> %c) {
 ; CHECK-LABEL: fma_nxv2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    fmla z2.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.s, p0/m, z1.s, z2.s
 ; CHECK-NEXT:    ret
   %r = call <vscale x 2 x float> @llvm.fma.nxv2f32(<vscale x 2 x float> %a, <vscale x 2 x float> %b, <vscale x 2 x float> %c)
   ret <vscale x 2 x float> %r
 }
 
 define <vscale x 2 x double> @fma_nxv2f64_1(<vscale x 2 x double> %a, <vscale x 2 x double> %b, <vscale x 2 x double> %c) {
 ; CHECK-LABEL: fma_nxv2f64_1:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    fmla z2.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.d, p0/m, z1.d, z2.d
 ; CHECK-NEXT:    ret
   %r = call <vscale x 2 x double> @llvm.fma.nxv2f64(<vscale x 2 x double> %a, <vscale x 2 x double> %b, <vscale x 2 x double> %c)
   ret <vscale x 2 x double> %r
 }
 
 define <vscale x 2 x double> @fma_nxv2f64_2(<vscale x 2 x double> %a, <vscale x 2 x double> %b, <vscale x 2 x double> %c) {
 ; CHECK-LABEL: fma_nxv2f64_2:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    fmla z2.d, p0/m, z1.d, z0.d
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    fmad z0.d, p0/m, z1.d, z2.d
 ; CHECK-NEXT:    ret
   %r = call <vscale x 2 x double> @llvm.fma.nxv2f64(<vscale x 2 x double> %b, <vscale x 2 x double> %a, <vscale x 2 x double> %c)
   ret <vscale x 2 x double> %r
 }
 
 define <vscale x 2 x double> @fma_nxv2f64_3(<vscale x 2 x double> %a, <vscale x 2 x double> %b, <vscale x 2 x double> %c) {
 ; CHECK-LABEL: fma_nxv2f64_3:
 ; CHECK:       // %bb.0: