AMDGPU : Custom lowering constrained fps.

[arsenm]

Summary

This patch only shows a way how to custom lowering the constrained fma operation.

What does this patch do:

1. Expand SDNodeFlags APIs to set up SDNodeFlags at the initial DAG build phase when reading the constrained fps metadata data.
2. AMDGPU backend sets up resister modes based on retrieved SDNodeFlags.

[wdng]

Fixed format issue.

[andrew.w.kaylor]

include/llvm/CodeGen/SelectionDAGNodes.h

368

I don't really like the fact that these are separate flags, given that they're mutually exclusive.

Also, I think we're eventually going to need to be able to distinguish between assumed rounding modes (where the instruction encoding isn't expected to include the rounding mode) and forced rounding modes (where the rounding mode will be encoded in the instruction). I don't have a specific vision for how that will need to work, but I know there are instructions that work this way and we'll need to handle at least intrinsics that use them. As I recall someone at AMD mentioned wanting behavior like that for flush-to-zero also.

The currently documented behavior of the constrained FP intrinsics is that the rounding mode tells the optimizer what it may assume about the rounding mode at the intrinsic location. Something else must have been done to set the rounding mode. If you are lowering to instructions that include a rounding mode, how do you handle the RoundDynamic case?

384–386

I don't think all of the rounding modes can default to false. Maybe you need a RoundDefault option. For instance, if SDNodeFlags::isDefined() returns true, but the node doesn't have a constrained rounding mode I'd need to check four different flags and then make an assumption to see that.

510

I think you need some logic here to set the rounding mode to dynamic if the flags being intersected conflict.

515

There needs to be a hierarchy here. For instance, if you merge ExceptIgnore with ExceptStrict it should result in ExceptStrict.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

943

This change will fail on all platforms except AMDGPU as you have this patch written. If you need target-specific behavior here, we'll need a target hook of some kind.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5471

All of the intrinsics above from fadd to fma fall through into this code. That's obviously not what you intended.

5473

What about the other rounding modes?

lib/Target/AMDGPU/SIISelLowering.cpp

3177

If you're going to make the changes in this patch, you need at least reasonable default behavior for all other platforms.

[rampitec]

include/llvm/CodeGen/SelectionDAGNodes.h

368

As these flags only apply to the ISD::STRICT_* opcodes you probably do not need to add fields to a generic SDNodeFlags. STRICT_* opcodes can get an extra operand for rounding mode.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

943

Looks like a good place to check for TLI.getOperationAction on the incoming Opcode and keep it as is if it is custom.

[wdng]

Instead of using SDNodeFlags to store metadata information, this patch directly appends an extra operand for rounding mode during the DAG build phase based on Stats's suggestion. This patch currently implements strict fps for fadd, fsub, fmul, fma, and fsqrt. Thanks a lot for @andrew.w.kaylor and @rampitec comments for this!

Known issues:

1. FDIV case is a special case, a separate patch will be created for it.
2. Currently, we don't take care of the "round.dynamic" rounding mode for the time being.
3. Will create a separate patch for f16 data type.

[rampitec]

lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

206

You have removed it but declaration remains.

783–788

What about f16?

804

Use of "if(cond) ... else .. cond ? :" is weird.

845

f16?

lib/Target/AMDGPU/AMDGPUISelLowering.h

338 ↗

(On Diff #119184)

You do not handle every one of that.

lib/Target/AMDGPU/SIISelLowering.cpp

319

These commented lines not needed.

4918

llvm_unreachable

4948

You are already doing translation, you can remove all the switches translating EqOpc into the same with chain. Just use it here.

4995

OK, you have chained all the nodes which require to reside within two s_setreg statements. How do you prevent any other regular fp operations without a chain to be scheduled in between of them?

[kzhuravl]

lib/Target/AMDGPU/SIDefines.h

461–462 ↗

(On Diff #119184)

These should go to relevant enums (like Id, Offset, WidthMinusOne, etc.).

lib/Target/AMDGPU/SIISelLowering.cpp

4912

Rename WidthBit to "Offset".

4917

Remove. This can go to default?

4928

New line.

4931

Missing ID_SHIFT_.

4933

What is 1? Do not use bare numbers.

[andrew.w.kaylor]

It isn't clear to me how your custom lowering interacts, if at all, with existing table-driven selection patterns. One of the goals in the implementation up to this point has been to have the instruction selection fall back on existing pattern matching as much as possible so that we don't need to duplicate all of the cases that are currently handled. Can you explain to me how this applies in the AMDGPU case?

lib/Target/AMDGPU/SIISelLowering.cpp

4925

You have upward and downward reversed.

4941

I think you're interpreting the rounding mode argument differently than I have, and therefore differently than the documentation in the LLVM Language Reference ("therefore" because I wrote the documentation). My intention was that the rounding mode argument was provided as information to the optimizer. It tells the optimizer what it can assume about rounding mode at the point of the operation. It was not intended to actually set the rounding mode.

I'm approaching this from the perspective of the STDC pragmas related to the FP environment. My understanding of these is that if FENV_ACCESS on is declared, we must assume dynamic (i.e. unknown) rounding mode in those scopes unless we can prove otherwise, but if the user wants to change the rounding mode a specific function call (such as fesetround) will be used.

I'm not sure what sort of front end you are assuming here, so that may explain the difference in your approach.

There are some x86 instructions that can incorporate a rounding mode operand, and it is my understanding that the AMDGPU architecture has similar needs. However, I believe we will need to extend the constrained FP intrinsics (or possibly introduce new intrinsics to handle cases like that.

[arsenm]

Needs to be redone

include/llvm/CodeGen/SelectionDAGNodes.h

private:
   bool UnsafeAlgebra : 1;
   bool NoNaNs : 1;
   bool NoInfs : 1;
   bool NoSignedZeros : 1;
   bool AllowReciprocal : 1;
   bool VectorReduction : 1;
   bool AllowContract : 1;
 
+  bool RoundDynamic : 1;
+  bool RoundTonearest : 1;
+  bool RoundDownward : 1;
+  bool RoundUpward : 1;
+  bool RoundTowardZero : 1;
+
+  bool ExceptIgnore : 1;
+  bool ExceptMayTrap : 1;
+  bool ExceptStrict : 1;
+
 public:
   /// Default constructor turns off all optimization flags.
   SDNodeFlags()
       : AnyDefined(false), NoUnsignedWrap(false), NoSignedWrap(false),
         Exact(false), UnsafeAlgebra(false), NoNaNs(false), NoInfs(false),
         NoSignedZeros(false), AllowReciprocal(false), VectorReduction(false),
-        AllowContract(false) {}
+        AllowContract(false), RoundDynamic(false),  RoundTonearest(false),
+        RoundDownward(false), RoundUpward(false), RoundTowardZero(false),
+        ExceptIgnore(false), ExceptMayTrap(false), ExceptStrict(false) {}
 
   /// Sets the state of the flags to the defined state.
   void setDefined() { AnyDefined = true; }
   /// Returns true if the flags are in a defined state.
   bool isDefined() const { return AnyDefined; }
 
   // These are mutators for each flag.
   void setNoUnsignedWrap(bool b) {
void setVectorReduction(bool b) {
     setDefined();
     VectorReduction = b;
   }
   void setAllowContract(bool b) {
     setDefined();
     AllowContract = b;
   }
 
+  void setRoundDynamic(bool b) {
+  setDefined();
+  RoundDynamic = b;
+  }
+
+  void setRoundTonearest(bool b) {
+    setDefined();
+    RoundTonearest = b;
+  }
+
+  void setRoundDownward(bool b) {
+  setDefined();
+  RoundDownward = b;
+  }
+
+  void setRoundUpward(bool b) {
+  setDefined();
+  RoundUpward = b;
+  }
+
+  void setRoundTowardZero(bool b) {
+  setDefined();
+  RoundTowardZero = b;
+  }
+
+  void setExceptIgnore(bool b) {
+  setDefined();
+  ExceptIgnore = b;
+  }
+
+  void setExceptMayTrap(bool b) {
+  setDefined();
+  ExceptMayTrap = b;
+  }
+
+  void setExceptStrict(bool b) {
+  setDefined();
+  ExceptStrict = b;
+  }
+
   // These are accessors for each flag.
   bool hasNoUnsignedWrap() const { return NoUnsignedWrap; }
   bool hasNoSignedWrap() const { return NoSignedWrap; }
   bool hasExact() const { return Exact; }
   bool hasUnsafeAlgebra() const { return UnsafeAlgebra; }
   bool hasNoNaNs() const { return NoNaNs; }
   bool hasNoInfs() const { return NoInfs; }
   bool hasNoSignedZeros() const { return NoSignedZeros; }
   bool hasAllowReciprocal() const { return AllowReciprocal; }
   bool hasVectorReduction() const { return VectorReduction; }
   bool hasAllowContract() const { return AllowContract; }
+  bool hasRoundDynamic() const { return RoundDynamic; }
+  bool hasRoundTonearest() const { return RoundTonearest; }
+  bool hasRoundDownward() const { return RoundDownward; }
+  bool hasRoundUpward() const { return RoundUpward; }
+  bool hasRoundTowardZero() const { return RoundTowardZero; }
+  bool hasExceptIgnore() const { return ExceptIgnore; }
+  bool hasExceptMayTrap() const { return ExceptMayTrap; }
+  bool hasExceptStrict() const { return ExceptStrict; }
 
   /// Clear any flags in this flag set that aren't also set in Flags.
   /// If the given Flags are undefined then don't do anything.
   void intersectWith(const SDNodeFlags Flags) {
     if (!Flags.isDefined())
       return;
     NoUnsignedWrap &= Flags.NoUnsignedWrap;
     NoSignedWrap &= Flags.NoSignedWrap;
     Exact &= Flags.Exact;
     UnsafeAlgebra &= Flags.UnsafeAlgebra;
     NoNaNs &= Flags.NoNaNs;
     NoInfs &= Flags.NoInfs;
     NoSignedZeros &= Flags.NoSignedZeros;
     AllowReciprocal &= Flags.AllowReciprocal;
     VectorReduction &= Flags.VectorReduction;
     AllowContract &= Flags.AllowContract;
+    RoundDynamic &= Flags.RoundDynamic;
+    RoundTonearest &= Flags.RoundTonearest;
+    RoundDownward &= Flags.RoundDownward;
+    RoundUpward &= Flags.RoundUpward;
+    RoundTowardZero &= Flags.RoundTowardZero;
+    ExceptIgnore &= Flags.ExceptIgnore;
+    ExceptMayTrap &= Flags.ExceptMayTrap;
+    ExceptStrict &= Flags.ExceptStrict;
   }
 };
 
 /// Represents one node in the SelectionDAG.
 ///
 class SDNode : public FoldingSetNode, public ilist_node<SDNode> {
 private:
   /// The operation that this node performs.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

 static TargetLowering::LegalizeAction
 getStrictFPOpcodeAction(const TargetLowering &TLI, unsigned Opcode, EVT VT) {
   unsigned EqOpc;
   switch (Opcode) {
     default: llvm_unreachable("Unexpected FP pseudo-opcode");
     case ISD::STRICT_FSQRT: EqOpc = ISD::FSQRT; break;
     case ISD::STRICT_FPOW: EqOpc = ISD::FPOW; break;
     case ISD::STRICT_FPOWI: EqOpc = ISD::FPOWI; break;
-    case ISD::STRICT_FMA: EqOpc = ISD::FMA; break;
+    case ISD::STRICT_FMA: EqOpc = ISD::STRICT_FMA; break;
     case ISD::STRICT_FSIN: EqOpc = ISD::FSIN; break;
     case ISD::STRICT_FCOS: EqOpc = ISD::FCOS; break;
     case ISD::STRICT_FEXP: EqOpc = ISD::FEXP; break;
     case ISD::STRICT_FEXP2: EqOpc = ISD::FEXP2; break;
     case ISD::STRICT_FLOG: EqOpc = ISD::FLOG; break;
     case ISD::STRICT_FLOG10: EqOpc = ISD::FLOG10; break;
     case ISD::STRICT_FLOG2: EqOpc = ISD::FLOG2; break;
     case ISD::STRICT_FRINT: EqOpc = ISD::FRINT; break;
     case ISD::STRICT_FNEARBYINT: EqOpc = ISD::FNEARBYINT; break;
   }
 
   auto Action = TLI.getOperationAction(EqOpc, VT);
 
+  if (Action == TargetLowering::Custom)
+    return Action;
+
   // We don't currently handle Custom or Promote for strict FP pseudo-ops.
   // For now, we just expand for those cases.
   if (Action != TargetLowering::Legal)
     Action = TargetLowering::Expand;
 
   return Action;
 }
 

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

setValue(&I, DAG.getNode(ISD::FMA, sdl,
                              getValue(I.getArgOperand(1)),
                              getValue(I.getArgOperand(2))));
     return nullptr;
   case Intrinsic::experimental_constrained_fadd:
   case Intrinsic::experimental_constrained_fsub:
   case Intrinsic::experimental_constrained_fmul:
   case Intrinsic::experimental_constrained_fdiv:
   case Intrinsic::experimental_constrained_frem:
-  case Intrinsic::experimental_constrained_fma:
+  case Intrinsic::experimental_constrained_fma: {
+    SDNodeFlags SDFlags;
+    const ConstrainedFPIntrinsic &FPI = cast<ConstrainedFPIntrinsic>(I);
+    if (FPI.getRoundingMode() == llvm::ConstrainedFPIntrinsic::rmDynamic)
+      SDFlags.setRoundDynamic(true);
+
+    EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+    SDValue Res = DAG.getNode(ISD::STRICT_FMA, sdl, VT,
+                              getValue(I.getArgOperand(0)),
+                              getValue(I.getArgOperand(1)),
+                              getValue(I.getArgOperand(2)));
+
+    Res.getNode()->setFlags(SDFlags);
+    setValue(&I, Res);
+    return nullptr;
+  }
   case Intrinsic::experimental_constrained_sqrt:
   case Intrinsic::experimental_constrained_pow:
   case Intrinsic::experimental_constrained_powi:
   case Intrinsic::experimental_constrained_sin:
   case Intrinsic::experimental_constrained_cos:
   case Intrinsic::experimental_constrained_exp:
   case Intrinsic::experimental_constrained_exp2:
   case Intrinsic::experimental_constrained_log:

lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

bool SelectVOP3OpSelMods0(SDValue In, SDValue &Src, SDValue &SrcMods,
                             SDValue &Clamp) const;
   bool SelectVOP3PMadMixModsImpl(SDValue In, SDValue &Src, unsigned &Mods) const;
   bool SelectVOP3PMadMixMods(SDValue In, SDValue &Src, SDValue &SrcMods) const;
 
   void SelectADD_SUB_I64(SDNode *N);
   void SelectUADDO_USUBO(SDNode *N);
   void SelectDIV_SCALE(SDNode *N);
   void SelectFMA_W_CHAIN(SDNode *N);
   void SelectFMUL_W_CHAIN(SDNode *N);
 
   SDNode *getS_BFE(unsigned Opcode, const SDLoc &DL, SDValue Val,
                    uint32_t Offset, uint32_t Width);
   void SelectS_BFEFromShifts(SDNode *N);
   void SelectS_BFE(SDNode *N);
   bool isCBranchSCC(const SDNode *N) const;
   void SelectBRCOND(SDNode *N);
   void SelectFMAD(SDNode *N);
void AMDGPUDAGToDAGISel::SelectFMA_W_CHAIN(SDNode *N) {
   //  src0_modifiers, src0,  src1_modifiers, src1, src2_modifiers, src2, clamp, omod
   SDValue Ops[10];
 
   SelectVOP3Mods0(N->getOperand(1), Ops[1], Ops[0], Ops[6], Ops[7]);
   SelectVOP3Mods(N->getOperand(2), Ops[3], Ops[2]);
   SelectVOP3Mods(N->getOperand(3), Ops[5], Ops[4]);
   Ops[8] = N->getOperand(0);
   Ops[9] = N->getOperand(4);
-
-  CurDAG->SelectNodeTo(N, AMDGPU::V_FMA_F32, N->getVTList(), Ops);
+  assert((N->getValueType(0) == MVT::f32 || N->getValueType(0) == MVT::f64) &&
+         "Incorrent Value Type!");
+  unsigned TargetOpc = N->getValueType(0) == MVT::f32 ?
+                       AMDGPU::V_FMA_F32 :
+                       AMDGPU::V_FMA_F64;
+  CurDAG->SelectNodeTo(N, TargetOpc, N->getVTList(), Ops);
 }
 
 void AMDGPUDAGToDAGISel::SelectFMUL_W_CHAIN(SDNode *N) {
   SDLoc SL(N);
   //  src0_modifiers, src0,  src1_modifiers, src1, clamp, omod
   SDValue Ops[8];
 
   SelectVOP3Mods0(N->getOperand(1), Ops[1], Ops[0], Ops[4], Ops[5]);
   SelectVOP3Mods(N->getOperand(2), Ops[3], Ops[2]);
   Ops[6] = N->getOperand(0);
   Ops[7] = N->getOperand(3);
 
   CurDAG->SelectNodeTo(N, AMDGPU::V_MUL_F32_e64, N->getVTList(), Ops);
 }
 
 // We need to handle this here because tablegen doesn't support matching
 // instructions with multiple outputs.
 void AMDGPUDAGToDAGISel::SelectDIV_SCALE(SDNode *N) {
   SDLoc SL(N);
   EVT VT = N->getValueType(0);
 
   assert(VT == MVT::f32 || VT == MVT::f64);
 
   unsigned Opc
bool AMDGPUDAGToDAGISel::SelectDS1Addr1Offset(SDValue Addr, SDValue &Base,
   if (CurDAG->isBaseWithConstantOffset(Addr)) {
     SDValue N0 = Addr.getOperand(0);
     SDValue N1 = Addr.getOperand(1);
     ConstantSDNode *C1 = cast<ConstantSDNode>(N1);
     if (isDSOffsetLegal(N0, C1->getSExtValue(), 16)) {
       // (add n0, c0)
       Base = N0;
       Offset = CurDAG->getTargetConstant(C1->getZExtValue(), DL, MVT::i16);
       return true;
     }
   } else if (Addr.getOpcode() == ISD::SUB) {
     // sub C, x -> add (sub 0, x), C
     if (const ConstantSDNode *C = dyn_cast<ConstantSDNode>(Addr.getOperand(0))) {
       int64_t ByteOffset = C->getSExtValue();
       if (isUInt<16>(ByteOffset)) {
         SDValue Zero = CurDAG->getTargetConstant(0, DL, MVT::i32);
 

lib/Target/AMDGPU/SIISelLowering.h

class SITargetLowering final : public AMDGPUTargetLowering {
   SDValue LowerLOAD(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerFastUnsafeFDIV(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerFDIV_FAST(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV16(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV32(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV64(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerConstrainedFMA(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINT_TO_FP(SDValue Op, SelectionDAG &DAG, bool Signed) const;
   SDValue LowerSTORE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerTrig(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerATOMIC_CMP_SWAP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBRCOND(SDValue Op, SelectionDAG &DAG) const;
 
   /// \brief Converts \p Op, which must be of floating point type, to the
   /// floating point type \p VT, by either extending or truncating it.

lib/Target/AMDGPU/SIISelLowering.cpp

SITargetLowering::SITargetLowering(const TargetMachine &TM,
 
   setOperationAction(ISD::FFLOOR, MVT::f64, Legal);
 
   setOperationAction(ISD::FSIN, MVT::f32, Custom);
   setOperationAction(ISD::FCOS, MVT::f32, Custom);
   setOperationAction(ISD::FDIV, MVT::f32, Custom);
   setOperationAction(ISD::FDIV, MVT::f64, Custom);
 
+  //setOperationAction(ISD::FMA, MVT::f32, Custom);
+  //setOperationAction(ISD::FMA, MVT::f64, Custom);
+
+  setOperationAction(ISD::STRICT_FMA, MVT::f32, Custom);
+  setOperationAction(ISD::STRICT_FMA, MVT::f64, Custom);
+
   if (Subtarget->has16BitInsts()) {
     setOperationAction(ISD::Constant, MVT::i16, Legal);
 
     setOperationAction(ISD::SMIN, MVT::i16, Legal);
     setOperationAction(ISD::SMAX, MVT::i16, Legal);
 
     setOperationAction(ISD::UMIN, MVT::i16, Legal);
     setOperationAction(ISD::UMAX, MVT::i16, Legal);
case ISD::LOAD: {
     return Result;
   }
 
   case ISD::FSIN:
   case ISD::FCOS:
     return LowerTrig(Op, DAG);
   case ISD::SELECT: return LowerSELECT(Op, DAG);
   case ISD::FDIV: return LowerFDIV(Op, DAG);
+  case ISD::STRICT_FMA: return LowerConstrainedFMA(Op, DAG);
   case ISD::ATOMIC_CMP_SWAP: return LowerATOMIC_CMP_SWAP(Op, DAG);
   case ISD::STORE: return LowerSTORE(Op, DAG);
   case ISD::GlobalAddress: {
     MachineFunction &MF = DAG.getMachineFunction();
     SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();
     return LowerGlobalAddress(MFI, Op, DAG);
   }
   case ISD::INTRINSIC_WO_CHAIN: return LowerINTRINSIC_WO_CHAIN(Op, DAG);

 
 static SDValue getFPTernOp(SelectionDAG &DAG, unsigned Opcode, const SDLoc &SL,
                            EVT VT, SDValue A, SDValue B, SDValue C,
                            SDValue GlueChain) {
   if (GlueChain->getNumValues() <= 1) {
     return DAG.getNode(Opcode, SL, VT, A, B, C);
   }
 
-  assert(GlueChain->getNumValues() == 3);
+  assert(GlueChain->getNumValues() == 3 || GlueChain->getNumValues() == 2);
 
   SDVTList VTList = DAG.getVTList(VT, MVT::Other, MVT::Glue);
   switch (Opcode) {
   default: llvm_unreachable("no chain equivalent for opcode");
   case ISD::FMA:
     Opcode = AMDGPUISD::FMA_W_CHAIN;
     break;
   }
 
+  if (GlueChain->getNumValues() == 3)
     return DAG.getNode(Opcode, SL, VTList, GlueChain.getValue(1), A, B, C,
                        GlueChain.getValue(2));
+  else if (GlueChain->getNumValues() == 2)
+    return DAG.getNode(Opcode, SL, VTList, GlueChain.getValue(0), A, B, C,
+                       GlueChain.getValue(1));
 }
 
 SDValue SITargetLowering::LowerFDIV16(SDValue Op, SelectionDAG &DAG) const {
   if (SDValue FastLowered = lowerFastUnsafeFDIV(Op, DAG))
     return FastLowered;
 
   SDLoc SL(Op);
   SDValue Src0 = Op.getOperand(0);
if (VT == MVT::f64)
     return LowerFDIV64(Op, DAG);
 
   if (VT == MVT::f16)
     return LowerFDIV16(Op, DAG);
 
   llvm_unreachable("Unexpected type for fdiv");
 }
 
+SDValue SITargetLowering::LowerConstrainedFMA(SDValue Op, SelectionDAG &DAG) const {
+  SDLoc SL(Op);
+
+  // Retrieve FP Rouding Mode.
+  bool RoundMode = Op->getFlags().hasRoundDynamic();
+  // TODO: Based on retrieved FP RoundMode to set up register modes.
+  const unsigned Denorm32Reg = AMDGPU::Hwreg::ID_MODE |
+                               (2 << AMDGPU::Hwreg::OFFSET_SHIFT_) |
+                               (1 << AMDGPU::Hwreg::WIDTH_M1_SHIFT_);
+
+  const SDValue BitField = DAG.getTargetConstant(Denorm32Reg, SL, MVT::i16);
+
+  SDVTList BindParamVTs = DAG.getVTList(MVT::Other, MVT::Glue);
+  const SDValue EnableDenormValue = DAG.getConstant(FP_DENORM_FLUSH_NONE,
+                                                      SL, MVT::i32);
+
+  SDValue EnableDenorm = DAG.getNode(AMDGPUISD::SETREG, SL, BindParamVTs,
+                                     DAG.getEntryNode(),
+                                     EnableDenormValue, BitField);
+
+  SDValue FMA = getFPTernOp(DAG, ISD::FMA, SL, MVT::f64, Op.getOperand(0),
+                                                         Op.getOperand(1),
+                                                         Op.getOperand(2),
+                                                         EnableDenorm);
+
+  const SDValue DisableDenormValue = DAG.getConstant(FP_DENORM_FLUSH_NONE,
+                                                     SL, MVT::i32);
+
+  SDValue DisableDenorm = DAG.getNode(AMDGPUISD::SETREG, SL, BindParamVTs,
+                                      FMA.getValue(1),
+                                      DisableDenormValue,
+                                      BitField,
+                                      FMA.getValue(2));
+
+  SDValue OutputChain = DAG.getNode(ISD::TokenFactor, SL, MVT::Other,
+                                    DisableDenorm, DAG.getRoot());
+  DAG.setRoot(OutputChain);
+
+  return FMA;
+
+  llvm_unreachable("Unexpected type for fma");
+}
+
 SDValue SITargetLowering::LowerSTORE(SDValue Op, SelectionDAG &DAG) const {
   SDLoc DL(Op);
   StoreSDNode *Store = cast<StoreSDNode>(Op);
   EVT VT = Store->getMemoryVT();
 
   if (VT == MVT::i1) {
     return DAG.getTruncStore(Store->getChain(), DL,
        DAG.getSExtOrTrunc(Store->getValue(), DL, MVT::i32),
        Store->getBasePtr(), MVT::i1, Store->getMemOperand());
   }
case 8:
         return SplitVectorStore(Op, DAG);
       return SDValue();
     case 16:
       if (NumElements > 4)
         return SplitVectorStore(Op, DAG);
       return SDValue();
     default:
       llvm_unreachable("unsupported private_element_size");
     }
   } else if (AS == AMDGPUASI.LOCAL_ADDRESS) {
     if (NumElements > 2)
       return SplitVectorStore(Op, DAG);
 
     if (NumElements == 2)
       return Op;
 
     // If properly aligned, if we split we might be able to use ds_write_b64.

test/CodeGen/AMDGPU/constrained_fp.ll

+; RUN:  llc -amdgpu-scalarize-global-loads=false  -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=SI -check-prefix=FUNC %s
+
+declare double @llvm.experimental.constrained.fma.f64(double, double, double, metadata, metadata) nounwind readnone
+
+; FUNC-LABEL: {{^}}fma_f64:
+; FUNC: s_setreg_b32
+; FUNC: v_fma_f64
+; FUNC: s_setreg_b32
+define amdgpu_kernel void @fma_f64(double addrspace(1)* %out, double addrspace(1)* %in1,
+                     double addrspace(1)* %in2, double addrspace(1)* %in3) {
+   %r0 = load double, double addrspace(1)* %in1
+   %r1 = load double, double addrspace(1)* %in2
+   %r2 = load double, double addrspace(1)* %in3
+   %r3 = tail call double @llvm.experimental.constrained.fma.f64(double %r0, double %r1, double %r2, metadata !"round.dynamic", metadata !"fpexcept.strict")
+   store double %r3, double addrspace(1)* %out
+   ret void
+}
+
+