[GISel] Add new combines for G_FMUL

[mkitzan]

Summary

Patch adds eight new GICombineRules for G_FMUL:

• G_FMUL(x, 0.0) -> 0.0
• G_FMUL(x, 1.0) -> x
• G_FMUL(x, -1.0) -> G_FSUB(0.0, x)
• G_FMUL(G_FNEG(x), G_FNEG(y)) -> G_FMUL(x, y)
• G_FMUL(G_FMUL(x, c0), c1) -> G_FMUL(x, c0 * c1) (unsafe fp math only)
• G_FMUL(G_FADD(x, x), c) -> G_FMUL(x, c * 2.0) (unsafe fp math only)
• G_FMUL(x, G_SELECT(G_FCMP(x > 0.0), -1.0, 1.0) -> G_FNEG(G_FABS(x))
• G_FMUL(x, G_SELECT(G_FCMP(x > 0.0), 1.0, -1.0)) -> G_FABS(x)

Patch additionally adds new combine tests for AArch64 target for these new rules, as well as updating AMDGPU GISel tests.

[arsenm]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

358–359

This is incorrect since it replaces a canonicalizing operation with a non-canonicalizing operation

[mkitzan]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2206

Decided against using the new matchOperandIsFPZero here, because it would require matchOperandIsFPZero not to validate that the FConstant operand could replace the dst of its use (which would diverge from the semantics of matchOperandIsZero).

[mkitzan]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

358–359

Worth noting there is an identical DAGCombiner for this though it was committed 13yrs ago. Is this a GISel specific requirement (not replacing a canonicalizing op with a non-canonicalizing op)? Regardless, it can easily be turned into G_FMUL(x, -1.0) -> G_FSUB(0.0, x).

[arsenm]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

358–359

Until recently, fneg and fsub -0, x were considered identical and didn't try to get this right. The DAG is broken here

[mkitzan]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

358–359

I see. I was wondering about the discrepancy between the G_MUL version and this one, but chalked it up floating point stuff. I'll post a fixup changing it to fold into G_FSUB(0.0, x).

[mkitzan]

• Changed G_FMUL(x, -1.0) to fold into G_FSUB(0.0, x)
• Fixed clang-tidy warnings
• Changed some fp constants in comments from {n} to {n}.0

[jurahul]

llvm/include/llvm/Target/GlobalISel/Combine.td

299

Sorry to comment on a change that I know nothing about, but ts this somewhere checking if fast math or equivalent is enabled? Otherwise, X * 0.0 != 0.0 in floating point.

[mkitzan]

llvm/include/llvm/Target/GlobalISel/Combine.td

299

At the moment nothing is checking whether fast math of special flags are enabled, but perhaps it should.

[arsenm]

llvm/include/llvm/Target/GlobalISel/Combine.td

299

This probably shouldn't be generalized to "binop". I think this is just fadd and fsub, and both require nsz (although actually I think you don't need nsz for -0)

[mkitzan]

llvm/include/llvm/Target/GlobalISel/Combine.td

299

I choose to generalize it as a binop because the G_MUL version was implemented in this way (https://reviews.llvm.org/D80094).

[arsenm]

llvm/include/llvm/Target/GlobalISel/Combine.td

299

The integer and FP operations have little to with each other, so I don't think following how they work is necessarily the best example

[arsenm]

reverse ping

[mkitzan]

Renamed binop_right_to_fp_zero to fmul_by_zero

[foad]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

358

I think this should be "G_FSUB(-0.0, x)".

llvm/include/llvm/Target/GlobalISel/Combine.td

491

This is wrong if x is Inf or NaN.

499

"-0.0".

[mkitzan]

I have a fixup ready with the G_FMUL(x, -1.0)->G_FSUB(-0.0, x) change. However, I just want to know in specific terms if there's anything else blocking approval.

[foad]

Well (fmul x, 0.0) -> 0.0 is still wrong if x is Inf or NaN.

[paquette]

I feel like this would be a bit easier to get in if it was broken into 1 patch per combine?

[mkitzan]

Sure thing. I'll split this up into separate patches.

[mkitzan]

Patch has been split up into sub-patches, closing this revision:

• https://reviews.llvm.org/D126008
• https://reviews.llvm.org/D126006
• https://reviews.llvm.org/D126005

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

 //===--------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_COMBINER_HELPER_H
 #define LLVM_CODEGEN_GLOBALISEL_COMBINER_HELPER_H
 
 #include "llvm/ADT/APFloat.h"
 #include "llvm/CodeGen/LowLevelType.h"
 #include "llvm/CodeGen/Register.h"
+#include "llvm/IR/InstrTypes.h"
 #include "llvm/Support/Alignment.h"
 
 namespace llvm {
 
 class GISelChangeObserver;
 class MachineIRBuilder;
 class MachineInstrBuilder;
 class MachineRegisterInfo;
public:
 
   /// Transform trunc (shl x, K) to shl (trunc x),
   /// K => K < VT.getScalarSizeInBits().
   bool matchCombineTruncOfShl(MachineInstr &MI,
                               std::pair<Register, Register> &MatchInfo);
   bool applyCombineTruncOfShl(MachineInstr &MI,
                               std::pair<Register, Register> &MatchInfo);
 
+  /// Transform G_FMUL(G_FNEG(x), G_FNEG(y)) to G_FMUL(x, y).
+  bool matchFMulWithNegatedInputs(MachineInstr &MI,
+                                  std::pair<Register, Register> &MatchInfo);
+  bool applyFMulWithNegatedInputs(MachineInstr &MI,
+                                  std::pair<Register, Register> &MatchInfo);
+
+  /// Transform G_FMUL(G_FMUL(x, c0) c1) to G_FMUL(x, c0 * c1).
+  /// Unsafe floating point math only.
+  bool matchFMulFMulWithConstantOps(
+      MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo);
+  bool applyFMulFMulWithConstantOps(
+      MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo);
+
+  /// Transform G_FMUL(G_FADD(x, x) c) to G_FMUL(x, c * 2.0).
+  /// Unsafe floating point math only.
+  bool
+  matchFMulFAddWithSameReg(MachineInstr &MI,
+                           std::pair<Register, Optional<APFloat>> &MatchInfo);
+  bool
+  applyFMulFAddWithSameReg(MachineInstr &MI,
+                           std::pair<Register, Optional<APFloat>> &MatchInfo);
+
+  /// Transform G_FMUL(x, G_SELECT(G_FCMP(x > 0.0), -1.0, 1.0)) to
+  /// G_FNEG(G_FABS(x)).
+  /// Transform G_FMUL(x, G_SELECT(G_FCMP(x > 0.0), 1.0, -1.0)) to
+  /// G_FABS(x).
+  bool matchForceFPNegOrPos(MachineInstr &MI,
+                            std::tuple<CmpInst::Predicate, Optional<APFloat>,
+                                       Optional<APFloat>> &MatchInfo);
+  bool applyForceFPNegOrPos(MachineInstr &MI,
+                            std::tuple<CmpInst::Predicate, Optional<APFloat>,
+                                       Optional<APFloat>> &MatchInfo);
+
   /// Transform G_MUL(x, -1) to G_SUB(0, x)
   bool applyCombineMulByNegativeOne(MachineInstr &MI);
 
+  /// Transform G_FMUL(x, -1.0) to G_FSUB(0.0, x)
+  bool applyCombineFMulByNegativeOne(MachineInstr &MI);
+
   /// Return true if any explicit use operand on \p MI is defined by a
   /// G_IMPLICIT_DEF.
   bool matchAnyExplicitUseIsUndef(MachineInstr &MI);
 
   /// Return true if all register explicit use operands on \p MI are defined by
   /// a G_IMPLICIT_DEF.
   bool matchAllExplicitUsesAreUndef(MachineInstr &MI);
 
public:
   /// Return true if \p MOP1 and \p MOP2 are register operands are defined by
   /// equivalent instructions.
   bool matchEqualDefs(const MachineOperand &MOP1, const MachineOperand &MOP2);
 
   /// Return true if \p MOP is defined by a G_CONSTANT with a value equal to
   /// \p C.
   bool matchConstantOp(const MachineOperand &MOP, int64_t C);
 
+  /// Return true if \p MOP is defined by a G_FCONSTANT with a value equal to
+  /// \p C.
+  bool matchFConstantOp(const MachineOperand &MOP, double C);
+
   /// Optimize (cond ? x : x) -> x
   bool matchSelectSameVal(MachineInstr &MI);
 
   /// Optimize (x op x) -> x
   bool matchBinOpSameVal(MachineInstr &MI);
 
   /// Check if operand \p OpIdx is zero.
   bool matchOperandIsZero(MachineInstr &MI, unsigned OpIdx);
 
+  /// Check if operand \p OpIdx is floating point zero.
+  bool matchOperandIsFPZero(MachineInstr &MI, unsigned OpIdx);
+
   /// Check if operand \p OpIdx is undef.
   bool matchOperandIsUndef(MachineInstr &MI, unsigned OpIdx);
 
   /// Erase \p MI
   bool eraseInst(MachineInstr &MI);
 
   /// Return true if MI is a G_ADD which can be simplified to a G_SUB.
   bool matchSimplifyAddToSub(MachineInstr &MI,

llvm/include/llvm/Target/GlobalISel/Combine.td

 // Fold x op 1 -> x
 def right_identity_one: GICombineRule<
   (defs root:$root),
   (match (wip_match_opcode G_MUL):$root,
     [{ return Helper.matchConstantOp(${root}->getOperand(2), 1); }]),
   (apply [{ return Helper.replaceSingleDefInstWithOperand(*${root}, 1); }])
 >;
 
+// Fold x op 1.0 -> x
+def right_identity_fp_one: GICombineRule<
+  (defs root:$root),
+  (match (wip_match_opcode G_FMUL):$root,
+    [{ return Helper.matchFConstantOp(${root}->getOperand(2), 1.0); }]),
+  (apply [{ return Helper.replaceSingleDefInstWithOperand(*${root}, 1); }])
+>;
+
 // Fold (x op x) - > x
 def binop_same_val: GICombineRule<
   (defs root:$root),
   (match (wip_match_opcode G_AND, G_OR):$root,
     [{ return Helper.matchBinOpSameVal(*${root}); }]),
   (apply [{ return Helper.replaceSingleDefInstWithOperand(*${root}, 1); }])
 >;
 

 def binop_right_to_zero: GICombineRule<
   (defs root:$root),
   (match (wip_match_opcode G_MUL):$root,
     [{ return Helper.matchOperandIsZero(*${root}, 2); }]),
   (apply [{ return Helper.replaceSingleDefInstWithOperand(*${root}, 2); }])
 >;
 
 // Erase stores of undef values.
 def erase_undef_store : GICombineRule<
   (defs root:$root),
   (match (wip_match_opcode G_STORE):$root,
     [{ return Helper.matchUndefStore(*${root}); }]),
   (apply [{ return Helper.eraseInst(*${root}); }])
 >;
 
 def simplify_add_to_sub_matchinfo: GIDefMatchData<"std::tuple<Register, Register>">;
 def simplify_add_to_sub: GICombineRule <

 // Transform (mul x, -1) -> (sub 0, x)
 def mul_by_neg_one: GICombineRule <
   (defs root:$root),
   (match (wip_match_opcode G_MUL):$root,
          [{ return Helper.matchConstantOp(${root}->getOperand(2), -1); }]),
   (apply [{ return Helper.applyCombineMulByNegativeOne(*${root}); }])
 >;
 
+// Transform (fmul x, 0.0) -> 0.0
+def fmul_by_zero: GICombineRule<
+  (defs root:$root),
+  (match (wip_match_opcode G_FMUL):$root,
+    [{ return Helper.matchOperandIsFPZero(*${root}, 2); }]),
+  (apply [{ return Helper.replaceSingleDefInstWithOperand(*${root}, 2); }])
+>;
+
+// Transform (fmul x, -1.0) -> (fsub 0.0, x)
+def fmul_by_neg_one: GICombineRule <
+  (defs root:$root),
+  (match (wip_match_opcode G_FMUL):$root,
+         [{ return Helper.matchFConstantOp(${root}->getOperand(2), -1.0); }]),
+  (apply [{ return Helper.applyCombineFMulByNegativeOne(*${root}); }])
+>;
+
+// Transform (fmul (fneg x), (fneg y)) -> (fmul x, y)
+def fmul_fneg_fneg_matchinfo : GIDefMatchData<"std::pair<Register, Register>">;
+def fmul_fneg_fneg: GICombineRule <
+  (defs root:$root, fmul_fneg_fneg_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_FMUL):$root,
+         [{ return Helper.matchFMulWithNegatedInputs(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyFMulWithNegatedInputs(*${root}, ${matchinfo}); }])
+>;
+
+// Transform (fmul (fmul x, c0), c1) -> (fmul x, c0 * c1)
+// Unsafe fp math only
+def fmul_fmul_cst_matchinfo : GIDefMatchData<"std::pair<Register, Optional<APFloat>>">;
+def fmul_fmul_cst: GICombineRule<
+  (defs root:$root, fmul_fmul_cst_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_FMUL):$root,
+         [{ return Helper.matchFMulFMulWithConstantOps(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyFMulFMulWithConstantOps(*${root}, ${matchinfo}); }])
+>;
+
+// Transform (fmul (fadd x, x), c) -> (fmul x, c * 2.0)
+// Unsafe fp math only
+def fmul_fadd_cst_matchinfo : GIDefMatchData<"std::pair<Register, Optional<APFloat>>">;
+def fmul_fadd_cst: GICombineRule<
+  (defs root:$root, fmul_fadd_cst_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_FMUL):$root,
+         [{ return Helper.matchFMulFAddWithSameReg(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyFMulFAddWithSameReg(*${root}, ${matchinfo}); }])
+>;
+
+
+// Transform (fmul x, (select (fcmp x > 0.0), -1.0, 1.0)) -> (fneg (fabs X))
+// Transform (fmul x, (select (fcmp x > 0.0), 1.0, -1.0)) -> (fabs X)
+def fp_neg_or_pos_matchinfo : GIDefMatchData<"std::tuple<CmpInst::Predicate, Optional<APFloat>, Optional<APFloat>>">;
+def fp_neg_or_pos: GICombineRule<
+  (defs root:$root, fp_neg_or_pos_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_FMUL):$root,
+         [{ return Helper.matchForceFPNegOrPos(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyForceFPNegOrPos(*${root}, ${matchinfo}); }])
+>;
+
 // FIXME: These should use the custom predicate feature once it lands.
 def undef_combines : GICombineGroup<[undef_to_fp_zero, undef_to_int_zero,
                                      undef_to_negative_one,
                                      binop_left_undef_to_zero,
                                      propagate_undef_any_op,
                                      propagate_undef_all_ops,
                                      propagate_undef_shuffle_mask,
                                      erase_undef_store]>;
 
 def identity_combines : GICombineGroup<[select_same_val, right_identity_zero,
                                         binop_same_val, binop_left_to_zero,
                                         binop_right_to_zero, p2i_to_i2p,
                                         i2p_to_p2i, anyext_trunc_fold,
-                                        fneg_fneg_fold, right_identity_one]>;
+                                        fneg_fneg_fold, right_identity_one,
+                                        right_identity_fp_one, fmul_by_zero]>;
 
 def known_bits_simplifications : GICombineGroup<[
   and_trivial_mask, redundant_sext_inreg]>;
 
 def width_reduction_combines : GICombineGroup<[reduce_shl_of_extend]>;
 
 def select_combines : GICombineGroup<[select_undef_cmp, select_constant_cmp]>;
 
 def trivial_combines : GICombineGroup<[copy_prop, mul_to_shl, add_p2i_to_ptradd,
-                                       mul_by_neg_one]>;
+                                       mul_by_neg_one, fmul_by_neg_one]>;
+
+def unsafe_fp_math_combines: GICombineGroup<[fmul_fmul_cst, fmul_fadd_cst]>;
 
 def all_combines : GICombineGroup<[trivial_combines, ptr_add_immed_chain,
     combines_for_extload, combine_indexed_load_store, undef_combines,
     identity_combines, simplify_add_to_sub,
     hoist_logic_op_with_same_opcode_hands,
     shl_ashr_to_sext_inreg, sext_inreg_of_load,
     width_reduction_combines, select_combines,
     known_bits_simplifications, ext_ext_fold,
     not_cmp_fold, opt_brcond_by_inverting_cond,
     unmerge_merge, fabs_fabs_fold, unmerge_cst, unmerge_dead_to_trunc,
     unmerge_zext_to_zext, trunc_ext_fold, trunc_shl,
-    constant_fp_op]>;
+    constant_fp_op, fmul_fneg_fneg, unsafe_fp_math_combines,
+    fp_neg_or_pos]>;

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

if (MI.getOpcode() == TargetOpcode::G_ANYEXT ||
     Builder.buildInstr(SrcExtOp, {DstReg}, {Reg});
     MI.eraseFromParent();
     return true;
   }
 
   return false;
 }
 
+bool CombinerHelper::matchFMulWithNegatedInputs(
+    MachineInstr &MI, std::pair<Register, Register> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src0;
+  Register Src1;
+
+  if (mi_match(Dst, MRI, m_GFMul(m_GFNeg(m_Reg(Src0)), m_GFNeg(m_Reg(Src1))))) {
+    MatchInfo = std::make_pair(Src0, Src1);
+    return true;
+  }
+  return false;
+}
+
+bool CombinerHelper::applyFMulWithNegatedInputs(
+    MachineInstr &MI, std::pair<Register, Register> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src0 = MatchInfo.first;
+  Register Src1 = MatchInfo.second;
+
+  Builder.setInstrAndDebugLoc(MI);
+  Builder.buildFMul({Dst}, Src0, Src1, MI.getFlags());
+  MI.eraseFromParent();
+  return true;
+}
+
+bool CombinerHelper::matchFMulFMulWithConstantOps(
+    MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  auto Options = MI.getMF()->getTarget().Options;
+  if (!(Options.UnsafeFPMath || MI.getFlag(MachineInstr::MIFlag::FmReassoc)))
+    return false;
+
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src;
+  const ConstantFP *FP0{};
+  const ConstantFP *FP1{};
+  if (mi_match(Dst, MRI,
+               m_GFMul(m_GFMul(m_Reg(Src), m_GFCst(FP0)), m_GFCst(FP1)))) {
+    const fltSemantics &FPSemantic = getFltSemanticForLLT(MRI.getType(Dst));
+    APFloat RHS = FP0->getValueAPF();
+    APFloat LHS = FP1->getValueAPF();
+    bool Unused;
+    RHS.convert(FPSemantic, APFloat::rmNearestTiesToEven, &Unused);
+    LHS.convert(FPSemantic, APFloat::rmNearestTiesToEven, &Unused);
+    MatchInfo = std::make_pair(Src, RHS * LHS);
+    return true;
+  }
+  return false;
+}
+
+bool CombinerHelper::applyFMulFMulWithConstantOps(
+    MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL &&
+         MatchInfo.second.hasValue() &&
+         "Expected a G_FMUL and an non-empty optional");
+  Register Src = MatchInfo.first;
+  APFloat NewFP = MatchInfo.second.getValue();
+  Register Dst = MI.getOperand(0).getReg();
+  LLT DstTy = MRI.getType(Dst);
+
+  Builder.setInstrAndDebugLoc(MI);
+  auto MIBFPNewC = Builder.buildFConstant(
+      DstTy, *ConstantFP::get(MI.getMF()->getFunction().getContext(), NewFP));
+  Builder.buildFMul({Dst}, Src, MIBFPNewC, MI.getFlags());
+  MI.eraseFromParent();
+  return true;
+}
+
+bool CombinerHelper::matchFMulFAddWithSameReg(
+    MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  auto Options = MI.getMF()->getTarget().Options;
+  if (!(Options.UnsafeFPMath || MI.getFlag(MachineInstr::MIFlag::FmReassoc)))
+    return false;
+
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src0;
+  Register Src1;
+  const ConstantFP *FP{};
+  if (mi_match(Dst, MRI,
+               m_GFMul(m_GFAdd(m_Reg(Src0), m_Reg(Src1)), m_GFCst(FP))) &&
+      Src0 == Src1) {
+    APFloat OldFP = FP->getValueAPF();
+    APFloat NewFP = OldFP * APFloat(OldFP.getSemantics(), "2.0");
+    MatchInfo = std::make_pair(Src0, NewFP);
+    return true;
+  }
+  return false;
+}
+
+bool CombinerHelper::applyFMulFAddWithSameReg(
+    MachineInstr &MI, std::pair<Register, Optional<APFloat>> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL &&
+         MatchInfo.second.hasValue() && "Expected a G_FMUL");
+  Register Src = MatchInfo.first;
+  APFloat NewFP = MatchInfo.second.getValue();
+  Register Dst = MI.getOperand(0).getReg();
+  LLT DstTy = MRI.getType(Dst);
+
+  Builder.setInstrAndDebugLoc(MI);
+  auto MIBFPNewC = Builder.buildFConstant(
+      DstTy, *ConstantFP::get(MI.getMF()->getFunction().getContext(), NewFP));
+  Builder.buildFMul({Dst}, Src, MIBFPNewC, MI.getFlags());
+  MI.eraseFromParent();
+  return true;
+}
+
+bool CombinerHelper::matchForceFPNegOrPos(
+    MachineInstr &MI,
+    std::tuple<CmpInst::Predicate, Optional<APFloat>, Optional<APFloat>>
+        &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  // `G_FMUL`s which allow NaNs or consider signed-zero aren't candidates for
+  // this optimization.
+  if (!MI.getFlag(MachineInstr::MIFlag::FmNoNans) ||
+      !MI.getFlag(MachineInstr::MIFlag::FmNsz))
+    return false;
+
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src0;
+  Register Src1;
+  if (mi_match(Dst, MRI, m_GFMul(m_Reg(Src0), m_Reg(Src1)))) {
+    MachineInstr *SrcMI = MRI.getVRegDef(Src1);
+    if (SrcMI->getOpcode() != TargetOpcode::G_SELECT)
+      return false;
+
+    // Verify the conditions that `x` is being compared to `0.0` and that `x` is
+    // the same value as the input to `G_FMUL`.
+    MachineInstr *CmpMI = MRI.getVRegDef(SrcMI->getOperand(1).getReg());
+    if (CmpMI->getOpcode() != TargetOpcode::G_FCMP ||
+        !matchFConstantOp(CmpMI->getOperand(3), 0.0) ||
+        Src0 != CmpMI->getOperand(2).getReg())
+      return false;
+
+    CmpInst::Predicate Pred =
+        static_cast<CmpInst::Predicate>(CmpMI->getOperand(1).getPredicate());
+    const ConstantFP *TrueOp =
+        getConstantFPVRegVal(SrcMI->getOperand(2).getReg(), MRI);
+    const ConstantFP *FalseOp =
+        getConstantFPVRegVal(SrcMI->getOperand(3).getReg(), MRI);
+    if (!TrueOp || !FalseOp)
+      return false;
+
+    MatchInfo =
+        std::make_tuple(Pred, TrueOp->getValueAPF(), FalseOp->getValueAPF());
+    return true;
+  }
+  return false;
+}
+
+bool CombinerHelper::applyForceFPNegOrPos(
+    MachineInstr &MI,
+    std::tuple<CmpInst::Predicate, Optional<APFloat>, Optional<APFloat>>
+        &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL &&
+         std::get<1>(MatchInfo).hasValue() &&
+         std::get<2>(MatchInfo).hasValue() &&
+         "Expected a G_FMUL and optionals with values");
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src = MI.getOperand(1).getReg();
+  LLT DstTy = MRI.getType(Dst);
+  auto Pred = std::get<0>(MatchInfo);
+  auto TrueFP = std::get<1>(MatchInfo).getValue();
+  auto FalseFP = std::get<2>(MatchInfo).getValue();
+
+  switch (Pred) {
+  default:
+    break;
+  case CmpInst::FCMP_OLT:
+  case CmpInst::FCMP_ULT:
+  case CmpInst::FCMP_OLE:
+  case CmpInst::FCMP_ULE:
+    std::swap(TrueFP, FalseFP);
+    LLVM_FALLTHROUGH;
+  case CmpInst::FCMP_OGT:
+  case CmpInst::FCMP_UGT:
+  case CmpInst::FCMP_OGE:
+  case CmpInst::FCMP_UGE: {
+    Builder.setInstrAndDebugLoc(MI);
+    if (TrueFP.isExactlyValue(-1.0) && FalseFP.isExactlyValue(1.0)) {
+      auto FAbsMIB = Builder.buildFAbs({DstTy}, {Src});
+      Builder.buildFNeg({Dst}, {FAbsMIB}, MI.getFlags());
+      MI.eraseFromParent();
+      return true;
+    }
+    if (TrueFP.isExactlyValue(1.0) && FalseFP.isExactlyValue(-1.0)) {
+      Builder.buildFAbs({Dst}, {Src});
+      MI.eraseFromParent();
+      return true;
+    }
+    break;
+  }
+  }
+  return false;
+}
+
 bool CombinerHelper::applyCombineMulByNegativeOne(MachineInstr &MI) {
   assert(MI.getOpcode() == TargetOpcode::G_MUL && "Expected a G_MUL");
   Register DstReg = MI.getOperand(0).getReg();
   Register SrcReg = MI.getOperand(1).getReg();
   LLT DstTy = MRI.getType(DstReg);
 
   Builder.setInstrAndDebugLoc(MI);
   Builder.buildSub(DstReg, Builder.buildConstant(DstTy, 0), SrcReg,
                    MI.getFlags());
   MI.eraseFromParent();
   return true;
 }
 
+bool CombinerHelper::applyCombineFMulByNegativeOne(MachineInstr &MI) {
+  assert(MI.getOpcode() == TargetOpcode::G_FMUL && "Expected a G_FMUL");
+  Register DstReg = MI.getOperand(0).getReg();
+  Register SrcReg = MI.getOperand(1).getReg();
+  LLT DstTy = MRI.getType(DstReg);
+
+  Builder.setInstrAndDebugLoc(MI);
+  Builder.buildFSub(DstReg, Builder.buildFConstant(DstTy, 0.0), SrcReg,
+                    MI.getFlags());
+  MI.eraseFromParent();
+  return true;
+}
+
 bool CombinerHelper::matchCombineFNegOfFNeg(MachineInstr &MI, Register &Reg) {
   assert(MI.getOpcode() == TargetOpcode::G_FNEG && "Expected a G_FNEG");
   Register SrcReg = MI.getOperand(1).getReg();
   return mi_match(SrcReg, MRI, m_GFNeg(m_Reg(Reg)));
 }
 
 bool CombinerHelper::matchCombineFAbsOfFAbs(MachineInstr &MI, Register &Src) {
   assert(MI.getOpcode() == TargetOpcode::G_FABS && "Expected a G_FABS");

 bool CombinerHelper::matchConstantOp(const MachineOperand &MOP, int64_t C) {
   if (!MOP.isReg())
     return false;
   // MIPatternMatch doesn't let us look through G_ZEXT etc.
   auto ValAndVReg = getConstantVRegValWithLookThrough(MOP.getReg(), MRI);
   return ValAndVReg && ValAndVReg->Value == C;
 }
 
+bool CombinerHelper::matchFConstantOp(const MachineOperand &MOP, double C) {
+  if (!MOP.isReg())
+    return false;
+  const ConstantFP *FCst = getConstantFPVRegVal(MOP.getReg(), MRI);
+  return FCst && FCst->getValueAPF().isExactlyValue(C);
+}
+
 bool CombinerHelper::replaceSingleDefInstWithOperand(MachineInstr &MI,
                                                      unsigned OpIdx) {
   assert(MI.getNumExplicitDefs() == 1 && "Expected one explicit def?");
   Register OldReg = MI.getOperand(0).getReg();
   Register Replacement = MI.getOperand(OpIdx).getReg();
   assert(canReplaceReg(OldReg, Replacement, MRI) && "Cannot replace register?");
   MI.eraseFromParent();
   replaceRegWith(MRI, OldReg, Replacement);

 }
 
 bool CombinerHelper::matchOperandIsZero(MachineInstr &MI, unsigned OpIdx) {
   return matchConstantOp(MI.getOperand(OpIdx), 0) &&
          canReplaceReg(MI.getOperand(0).getReg(), MI.getOperand(OpIdx).getReg(),
                        MRI);
 }
 
+bool CombinerHelper::matchOperandIsFPZero(MachineInstr &MI, unsigned OpIdx) {
+  return matchFConstantOp(MI.getOperand(OpIdx), 0.0) &&
+         canReplaceReg(MI.getOperand(0).getReg(), MI.getOperand(OpIdx).getReg(),
+                       MRI);
+}
+
 bool CombinerHelper::matchOperandIsUndef(MachineInstr &MI, unsigned OpIdx) {
   MachineOperand &MO = MI.getOperand(OpIdx);
   return MO.isReg() &&
          getOpcodeDef(TargetOpcode::G_IMPLICIT_DEF, MO.getReg(), MRI);
 }
 
 bool CombinerHelper::replaceInstWithFConstant(MachineInstr &MI, double C) {
   assert(MI.getNumDefs() == 1 && "Expected only one def?");

llvm/test/CodeGen/AArch64/GlobalISel/combine-fmul.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs -mtriple aarch64-unknown-unknown %s -o - | FileCheck %s
+
+---
+name:            fmul_by_zero
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; CHECK-LABEL: name: fmul_by_zero
+    ; CHECK: liveins: $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 0.000000e+00
+    ; CHECK: $x0 = COPY [[C]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FCONSTANT double 0.0
+    %2:_(s64) = G_FMUL %0, %1(s64)
+    $x0 = COPY %2(s64)
+...
+---
+name:            fmul_vector_by_zero
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0
+    ; Currently not implemented.
+    ; CHECK-LABEL: name: fmul_vector_by_zero
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_FCONSTANT float 0.000000e+00
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<4 x s32>) = G_FMUL [[COPY]], [[BUILD_VECTOR]]
+    ; CHECK: $q0 = COPY [[FMUL]](<4 x s32>)
+    %0:_(<4 x s32>) = COPY $q0
+    %1:_(s32) = G_FCONSTANT float 0.0
+    %2:_(<4 x s32>) = G_BUILD_VECTOR %1(s32), %1(s32), %1(s32), %1(s32)
+    %3:_(<4 x s32>) = G_FMUL %0, %2(<4 x s32>)
+    $q0 = COPY %3(<4 x s32>)
+...
+---
+name:            fmul_by_one
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; CHECK-LABEL: name: fmul_by_one
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: $x0 = COPY [[COPY]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FCONSTANT double 1.0
+    %2:_(s64) = G_FMUL %0, %1(s64)
+    $x0 = COPY %2(s64)
+...
+---
+name:            mul_vector_by_one
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0
+    ; Currently not implemented.
+    ; CHECK-LABEL: name: mul_vector_by_one
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_FCONSTANT float 1.000000e+00
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<4 x s32>) = G_FMUL [[COPY]], [[BUILD_VECTOR]]
+    ; CHECK: $q0 = COPY [[FMUL]](<4 x s32>)
+    %0:_(<4 x s32>) = COPY $q0
+    %1:_(s32) = G_FCONSTANT float 1.0
+    %2:_(<4 x s32>) = G_BUILD_VECTOR %1(s32), %1(s32), %1(s32), %1(s32)
+    %3:_(<4 x s32>) = G_FMUL %0, %2(<4 x s32>)
+    $q0 = COPY %3(<4 x s32>)
+...
+---
+name:            fmul_by_neg_one
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; CHECK-LABEL: name: fmul_by_neg_one
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 0.000000e+00
+    ; CHECK: [[FSUB:%[0-9]+]]:_(s64) = G_FSUB [[C]], [[COPY]]
+    ; CHECK: $x0 = COPY [[FSUB]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FCONSTANT double -1.0
+    %2:_(s64) = G_FMUL %0, %1(s64)
+    $x0 = COPY %2(s64)
+...
+---
+name:            fmul_vector_by_neg_one
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0
+    ; Currently not implemented.
+    ; CHECK-LABEL: name: fmul_vector_by_neg_one
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_FCONSTANT float -1.000000e+00
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<4 x s32>) = G_FMUL [[COPY]], [[BUILD_VECTOR]]
+    ; CHECK: $q0 = COPY [[FMUL]](<4 x s32>)
+    %0:_(<4 x s32>) = COPY $q0
+    %1:_(s32) = G_FCONSTANT float -1.0
+    %2:_(<4 x s32>) = G_BUILD_VECTOR %1(s32), %1(s32), %1(s32), %1(s32)
+    %3:_(<4 x s32>) = G_FMUL %0, %2(<4 x s32>)
+    $q0 = COPY %3(<4 x s32>)
+...
+---
+name:            fmul_fneg_fneg
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; CHECK-LABEL: name: fmul_fneg_fneg
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[FMUL:%[0-9]+]]:_(s64) = G_FMUL [[COPY]], [[COPY]]
+    ; CHECK: $x0 = COPY [[FMUL]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FNEG %0(s64)
+    %2:_(s64) = G_FMUL %1(s64), %1(s64)
+    $x0 = COPY %2(s64)
+...
+---
+name:            fmul_vector_fneg_fneg
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0
+    ; Currently not implemented.
+    ; CHECK-LABEL: name: fmul_vector_fneg_fneg
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
+    ; CHECK: [[FMUL:%[0-9]+]]:_(<4 x s32>) = G_FMUL [[COPY]], [[COPY]]
+    ; CHECK: $q0 = COPY [[FMUL]](<4 x s32>)
+    %0:_(<4 x s32>) = COPY $q0
+    %1:_(<4 x s32>) = G_FNEG %0(<4 x s32>)
+    %2:_(<4 x s32>) = G_FMUL %1(<4 x s32>), %1(<4 x s32>)
+    $q0 = COPY %2(<4 x s32>)
+...
+---
+name:            make_fp_negative
+body:             |
+  bb.1:
+    liveins: $w0
+    ; CHECK-LABEL: name: make_fp_negative
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[FABS:%[0-9]+]]:_(s32) = G_FABS [[COPY]]
+    ; CHECK: [[FNEG:%[0-9]+]]:_(s32) = nnan nsz G_FNEG [[FABS]]
+    ; CHECK: $w0 = COPY [[FNEG]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = G_FCONSTANT float 0.0
+    %2:_(s1) = G_FCMP floatpred(ogt), %0, %1
+    %3:_(s32) = G_FCONSTANT float -1.0
+    %4:_(s32) = G_FCONSTANT float 1.0
+    %5:_(s32) = G_SELECT %2(s1), %3, %4
+    %6:_(s32) = nnan nsz G_FMUL %0, %5
+    $w0 = COPY %6(s32)
+...
+---
+name:            make_fp_positive
+body:             |
+  bb.1:
+    liveins: $w0
+    ; CHECK-LABEL: name: make_fp_positive
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[FABS:%[0-9]+]]:_(s32) = G_FABS [[COPY]]
+    ; CHECK: $w0 = COPY [[FABS]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = G_FCONSTANT float 0.0
+    %2:_(s1) = G_FCMP floatpred(ogt), %0, %1
+    %3:_(s32) = G_FCONSTANT float 1.0
+    %4:_(s32) = G_FCONSTANT float -1.0
+    %5:_(s32) = G_SELECT %2(s1), %3, %4
+    %6:_(s32) = nnan nsz G_FMUL %0, %5
+    $w0 = COPY %6(s32)
+...

llvm/test/CodeGen/AArch64/GlobalISel/combine-unsafe-fp-math.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs -mtriple aarch64-unknown-unknown %s -o - | FileCheck %s --check-prefixes=SAFE
+# RUN: llc -enable-unsafe-fp-math -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs -mtriple aarch64-unknown-unknown %s -o - | FileCheck %s --check-prefixes=UNSAFE
+
+---
+name:            fmul_fmul_cst
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; SAFE-LABEL: name: fmul_fmul_cst
+    ; SAFE: liveins: $x0
+    ; SAFE: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; SAFE: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 2.000000e+00
+    ; SAFE: [[FMUL:%[0-9]+]]:_(s64) = G_FMUL [[COPY]], [[C]]
+    ; SAFE: [[FMUL1:%[0-9]+]]:_(s64) = G_FMUL [[FMUL]], [[C]]
+    ; SAFE: $x0 = COPY [[FMUL1]](s64)
+    ; UNSAFE-LABEL: name: fmul_fmul_cst
+    ; UNSAFE: liveins: $x0
+    ; UNSAFE: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; UNSAFE: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 4.000000e+00
+    ; UNSAFE: [[FMUL:%[0-9]+]]:_(s64) = G_FMUL [[COPY]], [[C]]
+    ; UNSAFE: $x0 = COPY [[FMUL]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FCONSTANT double 2.0
+    %2:_(s64) = G_FMUL %0, %1(s64)
+    %3:_(s64) = G_FMUL %2, %1(s64)
+    $x0 = COPY %3(s64)
+...
+---
+name:            fmul_fadd_same_reg
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; SAFE-LABEL: name: fmul_fadd_same_reg
+    ; SAFE: liveins: $x0
+    ; SAFE: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; SAFE: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 4.000000e+00
+    ; SAFE: [[FADD:%[0-9]+]]:_(s64) = G_FADD [[COPY]], [[COPY]]
+    ; SAFE: [[FMUL:%[0-9]+]]:_(s64) = G_FMUL [[FADD]], [[C]]
+    ; SAFE: $x0 = COPY [[FMUL]](s64)
+    ; UNSAFE-LABEL: name: fmul_fadd_same_reg
+    ; UNSAFE: liveins: $x0
+    ; UNSAFE: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; UNSAFE: [[C:%[0-9]+]]:_(s64) = G_FCONSTANT double 8.000000e+00
+    ; UNSAFE: [[FMUL:%[0-9]+]]:_(s64) = G_FMUL [[COPY]], [[C]]
+    ; UNSAFE: $x0 = COPY [[FMUL]](s64)
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_FCONSTANT double 4.0
+    %2:_(s64) = G_FADD %0, %0
+    %3:_(s64) = G_FMUL %2, %1(s64)
+    $x0 = COPY %3(s64)
+...

llvm/test/CodeGen/AMDGPU/GlobalISel/fmul.v2f16.ll

; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %mul = fmul <2 x half> %a, %neg.b
   ret <2 x half> %mul
 }
 
 define <2 x half> @v_fmul_v2f16_fneg_lhs_fneg_rhs(<2 x half> %a, <2 x half> %b) {
 ; GFX9-LABEL: v_fmul_v2f16_fneg_lhs_fneg_rhs:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v1 neg_lo:[1,1] neg_hi:[1,1]
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fmul_v2f16_fneg_lhs_fneg_rhs:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    s_mov_b32 s4, 0x80008000
-; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v0
-; GFX8-NEXT:    v_xor_b32_e32 v1, s4, v1
 ; GFX8-NEXT:    v_mul_f16_e32 v2, v0, v1
 ; GFX8-NEXT:    v_mul_f16_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mov_b32_e32 v1, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %neg.a = fneg <2 x half> %a
   %neg.b = fneg <2 x half> %b
; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %mul = fmul <4 x half> %a, %neg.b
   ret <4 x half> %mul
 }
 
 define <4 x half> @v_fmul_v4f16_fneg_lhs_fneg_rhs(<4 x half> %a, <4 x half> %b) {
 ; GFX9-LABEL: v_fmul_v4f16_fneg_lhs_fneg_rhs:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v2 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v3 neg_lo:[1,1] neg_hi:[1,1]
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v2
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v3
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fmul_v4f16_fneg_lhs_fneg_rhs:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    s_mov_b32 s4, 0x80008000
-; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v0
-; GFX8-NEXT:    v_xor_b32_e32 v2, s4, v2
-; GFX8-NEXT:    v_xor_b32_e32 v1, s4, v1
-; GFX8-NEXT:    v_xor_b32_e32 v3, s4, v3
 ; GFX8-NEXT:    v_mul_f16_e32 v4, v0, v2
 ; GFX8-NEXT:    v_mul_f16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v2, v1, v3
 ; GFX8-NEXT:    v_mul_f16_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mov_b32_e32 v3, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v4, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %mul = fmul <6 x half> %a, %neg.b
   ret <6 x half> %mul
 }
 
 define <6 x half> @v_fmul_v6f16_fneg_lhs_fneg_rhs(<6 x half> %a, <6 x half> %b) {
 ; GFX9-LABEL: v_fmul_v6f16_fneg_lhs_fneg_rhs:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v3 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v4 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v2, v2, v5 neg_lo:[1,1] neg_hi:[1,1]
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v3
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v4
+; GFX9-NEXT:    v_pk_mul_f16 v2, v2, v5
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fmul_v6f16_fneg_lhs_fneg_rhs:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    s_mov_b32 s4, 0x80008000
-; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v0
-; GFX8-NEXT:    v_xor_b32_e32 v3, s4, v3
-; GFX8-NEXT:    v_xor_b32_e32 v1, s4, v1
-; GFX8-NEXT:    v_xor_b32_e32 v2, s4, v2
-; GFX8-NEXT:    v_xor_b32_e32 v4, s4, v4
-; GFX8-NEXT:    v_xor_b32_e32 v5, s4, v5
 ; GFX8-NEXT:    v_mul_f16_e32 v6, v0, v3
 ; GFX8-NEXT:    v_mul_f16_sdwa v0, v0, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v3, v1, v4
 ; GFX8-NEXT:    v_mul_f16_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v4, v2, v5
 ; GFX8-NEXT:    v_mul_f16_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mov_b32_e32 v5, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v5, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %mul = fmul <8 x half> %a, %neg.b
   ret <8 x half> %mul
 }
 
 define <8 x half> @v_fmul_v8f16_fneg_lhs_fneg_rhs(<8 x half> %a, <8 x half> %b) {
 ; GFX9-LABEL: v_fmul_v8f16_fneg_lhs_fneg_rhs:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v4 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v5 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v2, v2, v6 neg_lo:[1,1] neg_hi:[1,1]
-; GFX9-NEXT:    v_pk_mul_f16 v3, v3, v7 neg_lo:[1,1] neg_hi:[1,1]
+; GFX9-NEXT:    v_pk_mul_f16 v0, v0, v4
+; GFX9-NEXT:    v_pk_mul_f16 v1, v1, v5
+; GFX9-NEXT:    v_pk_mul_f16 v2, v2, v6
+; GFX9-NEXT:    v_pk_mul_f16 v3, v3, v7
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fmul_v8f16_fneg_lhs_fneg_rhs:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    s_mov_b32 s4, 0x80008000
-; GFX8-NEXT:    v_xor_b32_e32 v0, s4, v0
-; GFX8-NEXT:    v_xor_b32_e32 v4, s4, v4
-; GFX8-NEXT:    v_xor_b32_e32 v1, s4, v1
-; GFX8-NEXT:    v_xor_b32_e32 v2, s4, v2
-; GFX8-NEXT:    v_xor_b32_e32 v3, s4, v3
-; GFX8-NEXT:    v_xor_b32_e32 v5, s4, v5
-; GFX8-NEXT:    v_xor_b32_e32 v6, s4, v6
-; GFX8-NEXT:    v_xor_b32_e32 v7, s4, v7
 ; GFX8-NEXT:    v_mul_f16_e32 v8, v0, v4
 ; GFX8-NEXT:    v_mul_f16_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v4, v1, v5
 ; GFX8-NEXT:    v_mul_f16_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v5, v2, v6
 ; GFX8-NEXT:    v_mul_f16_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_mul_f16_e32 v6, v3, v7
 ; GFX8-NEXT:    v_mul_f16_sdwa v3, v3, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1