Add ‘llvm.experimental.constrained.fma‘ Intrinsic

[wdng]

Summary

Add ‘llvm.experimental.constrained.fma‘ Intrinsic.

[arsenm]

Needs tests

[b-sumner]

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6651 ↗

(On Diff #109797)

This is a ternary operation. Code below assumes unary or binary.

[b-sumner]

An update to docs/LangRef.rst is needed.

[wdng]

Add missing lit tests.

[wdng]

In D36335#832367, @b-sumner wrote:

An update to docs/LangRef.rst is needed.

Sure, will do.

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6651 ↗

(On Diff #109797)

In Intrinsics.td, we have defined fma is a ternary operator. Here it only mutates STRICT_FMA to FMA and IsUnary is false by default. So we may not need to specify whether is unary of binary here?

[b-sumner]

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6651 ↗

(On Diff #109797)

Please take a look at lines 6676 - 6680 below. Do you not need to pass a 3 element list to MorphNodeTo for the FMA case?

[andrew.w.kaylor]

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6651 ↗

(On Diff #109797)

You definitely need to add code to handle the third argument.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6015 ↗

(On Diff #109821)

This code also needs to be updated to handle the case of three value operands.

lib/IR/Verifier.cpp

3985 ↗

(On Diff #109821)

The implementation of this function assumes only 1 or 2 value operands. It will need to be updated.

test/Feature/fp-intrinsics.ll

236 ↗

(On Diff #109821)

If you checked the arguments here it should reveal the problems in the code.

There's also a test at llvm/tests/CodeGen/X86/fp-intrinsics.ll that carries the constrained FP intrinsics all the way through code generation. Can you add a case there for this intrinsic?

[andrew.w.kaylor]

Could you also add a use of this new intrinsic to llvm/test/Verifier/fp-intrinsics.ll?

[wdng]

Address code reviews.

[wdng]

Upload correct diff.

[b-sumner]

docs/LangRef.rst

13035 ↗

(On Diff #110808)

Too much cut and paste from frem

13043 ↗

(On Diff #110808)

rounding only once

lib/Target/X86/X86ISelDAGToDAG.cpp

2017 ↗

(On Diff #110808)

Did you run clang-format?

[wdng]

Code changes based on Brian's comments.

[arsenm]

test/CodeGen/X86/fp-intrinsics.ll

2 ↗

(On Diff #110823)

Missing -check-prefix=CHECK

246 ↗

(On Diff #110823)

You need a separate check-label for the FMAless run line

test/Feature/fp-intrinsics.ll

244 ↗

(On Diff #110823)

Should also test for the other FP types

[andrew.w.kaylor]

docs/LangRef.rst

13035 ↗

(On Diff #110823)

I'm not sure it's clear what the comment "Note that the rounding happens only once here" means in this context. The rounding mode argument provides information to the optimizer and does not have any functional effect. I hope that this is straightforward enough with the other intrinsics that the terse comments there were sufficient.

In the case of the constrained fma intrinsic, it is worth mentioning that any actions the optimizer performs on the intrinsic must be consistent with the rounding behavior of an fma instruction. For instance, the optimizer cannot perform constant folding where a rounded multiply is performed followed a rounded add -- the rounding must be atomic. Perhaps that is what you intended to say here. If so, I believe a more verbose statement is needed.

13037 ↗

(On Diff #110823)

I think it would be a good idea to discuss here the circumstances under which this intrinsic can be formed. Specifically, what is the relationship between rounding mode control and the fp-contract setting. If strict rounding behavior is required within a scope, but fusing is enabled globally within the compilation unit does the rounding requirement override the fp-contract setting? I think it should.

Also, what are the expected exception semantics? If a scope is governed by strict exception behavior, how will the FP status flags be handled if a multiply and an add are fused? I believe what is required is that if either operation would have set an FP status flag then the fused operation must also set that flag, and no flag should be set by the fused operation that would not have been set by either of the two operations separately.

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

Can you explain why this was necessary? I would have expected there to have been handling already in place for ISD::FMA.

[wdng]

Address code reviews.

[wdng]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

No it doesn't, looks like X86 doesn't handle ISD:FMA automatically unless we there is -mattr=+fma option. Without this, CodeGen/X86/fp-intrinsics.ll will fail in instruction selection.

[andrew.w.kaylor]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

I still don't understand. What happens when -mattr=+fma is used?

The CodeGen/X86/fma.ll test uses that option. This case should work in the same way.

[b-sumner]

docs/LangRef.rst

13043 ↗

(On Diff #110808)

How about "The result produced is the product of the first two operands added to the third operand computed with infinite precision, and then rounded to the target precision."

13023 ↗

(On Diff #111268)

How about "...returns the result of a fused-multiply-add operation on its operands."?

[wdng]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

I think I made a mistake when describing the problem in my early comments. Let me rephrase and explain it there.

1. Without -mattr=+fma, a FMA libcall will be generated
2. With -mattr=+fma, we are expecting the corresponding FMA instruction to be generated.

In fma.ll, all fma tests are *not* constrained fp operations, during the during the X86ISelLowering phase, the FMA node has been lowered to X86ISD::FMADD. So there is no ISD::FMA at this phase since it has already been changed to X86ISD::FMADD before the instruction selection starts. Please refer to the following dump.

(gdb) p CurDAG->dump()
SelectionDAG has 12 nodes:
  t0: ch = EntryToken
      t2: f64,ch = CopyFromReg t0, Register:f64 %vreg0
      t4: f64,ch = CopyFromReg t0, Register:f64 %vreg1
      t6: f64,ch = CopyFromReg t0, Register:f64 %vreg2
    t12: f64 = X86ISD::FMADD t2, t4, t6
  t10: ch,glue = CopyToReg t0, Register:f64 %XMM0, t12
  t11: ch = X86ISD::RET_FLAG t10, TargetConstant:i32<0>, Register:f64 %XMM0, t10:1

However, for the constrained fma, we use mutateStrictFPToFP( ) function to mutate constrained_fma to normal fma, namely ISD::FMA before the instruction selction starts. The X86 backend cannot recognize the ISD::FMA, so we have to add codes to convert ISD::FMA to X86ISD::FMADD during the instruction selection.

[wdng]

Update LangRef.rst based on comments.

[wdng]

Update LangRef.rst: put more accurate descriptions into the constrained.fma semantic section.

[b-sumner]

docs/LangRef.rst

13040 ↗

(On Diff #111423)

Extra period

[wdng]

Remove extra period. Thanks!

[wdng]

Ping.

[andrew.w.kaylor]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

I'm still not sure I understand this, but it sounds to me like this should be happening somewhere else.

Are you saying that if -mattr=+fma is not used the ISD::STRICT_FMA will be expanded to a libcall before we reach mutateStrictFpToFP and so this code will never be reached in that case? And if so, are you further saying that when -mattr=+fma is used we will reach this code only after mutateStrictFpToFp() has converted ISD::STRICT_FMA to ISD::FMA?

My concern is that this is adding a generic (not constrained-specific) handler to handle the constrained case. I would much rather figure out a way to get ISD::STRICT_FMA to follow the existing path.

[wdng]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

Are you saying that if -mattr=+fma is not used the ISD::STRICT_FMA will be expanded to a libcall before we reach mutateStrictFpToFP and so this code will never be reached in that case? And if so, are you further saying that when -mattr=+fma is used we will reach this code only after mutateStrictFpToFp() has converted ISD::STRICT_FMA to ISD::FMA?

--> Yes

My concern is that this is adding a generic (not constrained-specific) handler to handle the constrained case. I would much rather figure out a way to get ISD::STRICT_FMA to follow the existing path.

---> I once tried to move the "mutateStrictFPToFP( )" to the LegalizeDAG phase, like the following code shows and I found it works and there is no need to add codes into X86 backend instruction selector:

    switch (Action) {
    case TargetLowering::Legal:
		if (Node->isStrictFPOpcode())
        Node = DAG.mutateStrictFPToFP(Node);
      return;

So once those strict fp operator haven legalized to legal, we can directly mutate them to their corresponding normal fp operator.

However, here comes a problem that non-default FP (or constrained fp operations) exception behaviors are target-specific, which means we have to leave it to each sub-target selectors to handle them. So I would not suggest mutating those instructions at somewhere. What do you think?

[wdng]

Ping.

[andrew.w.kaylor]

lib/Target/X86/X86ISelDAGToDAG.cpp

2015 ↗

(On Diff #110823)

I do think the mutate needs to be done as late as possible. I'm not even entirely certain that we won't need to figure out a way to communicate the FP constraints beyond instruction selection.

Would it be possible to have the mutateStrictFPToFP call (in its current location) call a target-specific hook to get a target-specific mutated node, so that we could convert directly to X86ISD::FMADD there?

Also, have you considered how non-X86 architectures need to handle this case?

[craig.topper]

Do we want to give the target any chance to use FMSUB/FNMADD/FNSUB if any of the arguments are negated?

[andrew.w.kaylor]

In D36335#846018, @craig.topper wrote:

Do we want to give the target any chance to use FMSUB/FNMADD/FNSUB if any of the arguments are negated?

That's exactly the kind of thing I was afraid of missing by not channeling this through the normal path that ISD::FMA takes.

[craig.topper]

I'm wondering if X86 really needs X86ISD::FMADD opcode at all. We definitely need FNMADD, FMSUB, and FNMSUB. But I don't think there's any real difference between X86ISD::FMADD and ISD::FMA.

include/llvm/CodeGen/ISDOpcodes.h

266 ↗

(On Diff #111428)

Please keep the blank line here.

lib/Target/X86/X86ISelDAGToDAG.cpp

2016 ↗

(On Diff #111428)

Add a comment that this is here because STRICT_FMA is turned into FMA after legalization and DAG combine.

[wdng]

Address code reviews. Thanks a lot!

[andrew.w.kaylor]

Can you put this off until the patch that Craig submitted in D36983 either lands or gets rejected? If that change goes through, you should be able to remove your modifications to X86ISelDAGToDAG.cpp.

[wdng]

In D36335#849535, @andrew.w.kaylor wrote:

Can you put this off until the patch that Craig submitted in D36983 either lands or gets rejected? If that change goes through, you should be able to remove your modifications to X86ISelDAGToDAG.cpp.

Sure, thanks!

[wdng]

Patch update after the patch [X86] Remove X86ISD::FMADD in favor ISD::FMA has been upstreamed.

[wdng]

Fixed a format issue.

[andrew.w.kaylor]

test/CodeGen/X86/fp-intrinsics.ll

245 ↗

(On Diff #112408)

These values could be constant folded without rounding, so even though this test case works now it's testing something that we don't necessarily want to be true. At some point, we're going to want to teach optimizations to recognize these intrinsics and fold cases like this. That's why I was using 42.1 in the other tests. It's just an arbitrary value that introduces rounding errors.

[andrew.w.kaylor]

Can you revert the white space changes in the places you aren't otherwise modifying? In general, you shouldn't make formatting changes outside of the parts of the file your patch is modifying. It complicates the version control blame process without adding a lot of benefit.

Also, your latest diffs seem not to have full file context (such as you get with the -x -U99999 switch with diff). This isn't important for the current review, but it is something to keep in mind going forward.

I really appreciate your work on this patch, and I hate to seem like I'm nit-picking a lot. I just want to make sure we do things correctly. Thanks!

[wdng]

In D36335#850574, @andrew.w.kaylor wrote:

Can you revert the white space changes in the places you aren't otherwise modifying? In general, you shouldn't make formatting changes outside of the parts of the file your patch is modifying. It complicates the version control blame process without adding a lot of benefit.

Also, your latest diffs seem not to have full file context (such as you get with the -x -U99999 switch with diff). This isn't important for the current review, but it is something to keep in mind going forward.

I really appreciate your work on this patch, and I hate to seem like I'm nit-picking a lot. I just want to make sure we do things correctly. Thanks!

Hi, Andrew, no problem at all. I will provide an updated full patch for this. Thanks a lot!

[wdng]

Address code reviews. Thanks!

[andrew.w.kaylor]

LGTM

llvm/trunk/docs/LangRef.rst

 
 Semantics:
 """"""""""
 
 The value produced is the floating point remainder from the division of the two
 value operands and has the same type as the operands.  The remainder has the
 same sign as the dividend. 
 
+'``llvm.experimental.constrained.fma``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare <type>
+      @llvm.experimental.constrained.fma(<type> <op1>, <type> <op2>, <type> <op3>,
+                                          metadata <rounding mode>,
+                                          metadata <exception behavior>)
+
+Overview:
+"""""""""
+
+The '``llvm.experimental.constrained.fma``' intrinsic returns the result of a
+fused-multiply-add operation on its operands.
+
+Arguments:
+""""""""""
+
+The first three arguments to the '``llvm.experimental.constrained.fma``'
+intrinsic must be :ref:`floating point <t_floating>` or :ref:`vector
+<t_vector>` of floating point values. All arguments must have identical types.
+
+The fourth and fifth arguments specify the rounding mode and exception behavior
+as described above.
+
+Semantics:
+""""""""""
+
+The result produced is the product of the first two operands added to the third
+operand computed with infinite precision, and then rounded to the target
+precision.
 
 Constrained libm-equivalent Intrinsics
 --------------------------------------
 
 In addition to the basic floating point operations for which constrained
 intrinsics are described above, there are constrained versions of various
 operations which provide equivalent behavior to a corresponding libm function.
 These intrinsics allow the precise behavior of these operations with respect to

llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// Simple binary floating point operators.
     FADD, FSUB, FMUL, FDIV, FREM,
 
     /// Constrained versions of the binary floating point operators.
     /// These will be lowered to the simple operators before final selection.
     /// They are used to limit optimizations while the DAG is being
     /// optimized.
     STRICT_FADD, STRICT_FSUB, STRICT_FMUL, STRICT_FDIV, STRICT_FREM,
+    STRICT_FMA,
 
     /// Constrained versions of libm-equivalent floating point intrinsics.
     /// These will be lowered to the equivalent non-constrained pseudo-op
     /// (or expanded to the equivalent library call) before final selection.
     /// They are used to limit optimizations while the DAG is being optimized.
     STRICT_FSQRT, STRICT_FPOW, STRICT_FPOWI, STRICT_FSIN, STRICT_FCOS,
     STRICT_FEXP, STRICT_FEXP2, STRICT_FLOG, STRICT_FLOG10, STRICT_FLOG2,
     STRICT_FRINT, STRICT_FNEARBYINT,

llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h

bool isMemIntrinsic() const {
     return (NodeType == ISD::INTRINSIC_W_CHAIN ||
             NodeType == ISD::INTRINSIC_VOID) &&
            SDNodeBits.IsMemIntrinsic;
   }
 
   /// Test if this node is a strict floating point pseudo-op.
   bool isStrictFPOpcode() {
     switch (NodeType) {
-      default: 
+      default:
         return false;
       case ISD::STRICT_FADD:
       case ISD::STRICT_FSUB:
       case ISD::STRICT_FMUL:
       case ISD::STRICT_FDIV:
       case ISD::STRICT_FREM:
+      case ISD::STRICT_FMA:
       case ISD::STRICT_FSQRT:
       case ISD::STRICT_FPOW:
       case ISD::STRICT_FPOWI:
       case ISD::STRICT_FSIN:
       case ISD::STRICT_FCOS:
       case ISD::STRICT_FEXP:
       case ISD::STRICT_FEXP2:
       case ISD::STRICT_FLOG:

llvm/trunk/include/llvm/IR/IntrinsicInst.h

public:
     enum ExceptionBehavior {
       ebInvalid,
       ebIgnore,
       ebMayTrap,
       ebStrict
     };
 
     bool isUnaryOp() const;
+    bool isTernaryOp() const;
     RoundingMode getRoundingMode() const;
     ExceptionBehavior getExceptionBehavior() const;
 
     // Methods for support type inquiry through isa, cast, and dyn_cast:
     static bool classof(const IntrinsicInst *I) {
       switch (I->getIntrinsicID()) {
       case Intrinsic::experimental_constrained_fadd:
       case Intrinsic::experimental_constrained_fsub:
       case Intrinsic::experimental_constrained_fmul:
       case Intrinsic::experimental_constrained_fdiv:
       case Intrinsic::experimental_constrained_frem:
+      case Intrinsic::experimental_constrained_fma:
       case Intrinsic::experimental_constrained_sqrt:
       case Intrinsic::experimental_constrained_pow:
       case Intrinsic::experimental_constrained_powi:
       case Intrinsic::experimental_constrained_sin:
       case Intrinsic::experimental_constrained_cos:
       case Intrinsic::experimental_constrained_exp:
       case Intrinsic::experimental_constrained_exp2:
       case Intrinsic::experimental_constrained_log:

llvm/trunk/include/llvm/IR/Intrinsics.td

def int_experimental_constrained_fdiv : Intrinsic<[ llvm_anyfloat_ty ],
                                                       llvm_metadata_ty,
                                                       llvm_metadata_ty ]>;
   def int_experimental_constrained_frem : Intrinsic<[ llvm_anyfloat_ty ],
                                                     [ LLVMMatchType<0>,
                                                       LLVMMatchType<0>,
                                                       llvm_metadata_ty,
                                                       llvm_metadata_ty ]>;
 
+  def int_experimental_constrained_fma : Intrinsic<[ llvm_anyfloat_ty ],
+                                                    [ LLVMMatchType<0>,
+                                                      LLVMMatchType<0>,
+                                                      LLVMMatchType<0>,
+                                                      llvm_metadata_ty,
+                                                      llvm_metadata_ty ]>;
+
   // These intrinsics are sensitive to the rounding mode so we need constrained
   // versions of each of them.  When strict rounding and exception control are
   // not required the non-constrained versions of these intrinsics should be
   // used.
   def int_experimental_constrained_sqrt : Intrinsic<[ llvm_anyfloat_ty ],
                                                     [ LLVMMatchType<0>,
                                                       llvm_metadata_ty,
                                                       llvm_metadata_ty ]>;

llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

 static TargetLowering::LegalizeAction
 getStrictFPOpcodeAction(const TargetLowering &TLI, unsigned Opcode, EVT VT) {
   unsigned EqOpc;
   switch (Opcode) {
     default: llvm_unreachable("Unexpected FP pseudo-opcode");
     case ISD::STRICT_FSQRT: EqOpc = ISD::FSQRT; break;
     case ISD::STRICT_FPOW: EqOpc = ISD::FPOW; break;
     case ISD::STRICT_FPOWI: EqOpc = ISD::FPOWI; break;
+    case ISD::STRICT_FMA: EqOpc = ISD::FMA; break;
     case ISD::STRICT_FSIN: EqOpc = ISD::FSIN; break;
     case ISD::STRICT_FCOS: EqOpc = ISD::FCOS; break;
     case ISD::STRICT_FEXP: EqOpc = ISD::FEXP; break;
     case ISD::STRICT_FEXP2: EqOpc = ISD::FEXP2; break;
     case ISD::STRICT_FLOG: EqOpc = ISD::FLOG; break;
     case ISD::STRICT_FLOG10: EqOpc = ISD::FLOG10; break;
     case ISD::STRICT_FLOG2: EqOpc = ISD::FLOG2; break;
     case ISD::STRICT_FRINT: EqOpc = ISD::FRINT; break;
if (Action == TargetLowering::Expand) {
       NewVal = DAG.getNode(ISD::TRAP, SDLoc(Node), Node->getVTList(),
                            Node->getOperand(0));
       ReplaceNode(Node, NewVal.getNode());
       LegalizeOp(NewVal.getNode());
       return;
     }
     break;
   case ISD::STRICT_FSQRT:
+  case ISD::STRICT_FMA:
   case ISD::STRICT_FPOW:
   case ISD::STRICT_FPOWI:
   case ISD::STRICT_FSIN:
   case ISD::STRICT_FCOS:
   case ISD::STRICT_FEXP:
   case ISD::STRICT_FEXP2:
   case ISD::STRICT_FLOG:
   case ISD::STRICT_FLOG10:
if (StoreSDNode *ST = dyn_cast<StoreSDNode>(User)) {
       // Make sure that nothing else could have stored into the destination of
       // this store.
       if (!ST->getChain().reachesChainWithoutSideEffects(DAG.getEntryNode()))
         continue;
 
       // If the index is dependent on the store we will introduce a cycle when
       // creating the load (the load uses the index, and by replacing the chain
       // we will make the index dependent on the load). Also, the store might be
-      // dependent on the extractelement and introduce a cycle when creating 
+      // dependent on the extractelement and introduce a cycle when creating
       // the load.
       if (SDNode::hasPredecessorHelper(ST, Visited, Worklist) ||
           ST->hasPredecessor(Op.getNode()))
         continue;
 
       StackPtr = ST->getBasePtr();
       Ch = SDValue(ST, 0);
       break;
case ISD::FREM:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::REM_F32, RTLIB::REM_F64,
                                       RTLIB::REM_F80, RTLIB::REM_F128,
                                       RTLIB::REM_PPCF128));
     break;
   case ISD::FMA:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::FMA_F32, RTLIB::FMA_F64,
                                       RTLIB::FMA_F80, RTLIB::FMA_F128,
                                       RTLIB::FMA_PPCF128));
+  case ISD::STRICT_FMA:
+    Results.push_back(ExpandFPLibCall(Node, RTLIB::FMA_F32, RTLIB::FMA_F64,
+                                      RTLIB::FMA_F80, RTLIB::FMA_F128,
+                                      RTLIB::FMA_PPCF128));
     break;
   case ISD::FADD:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::ADD_F32, RTLIB::ADD_F64,
                                       RTLIB::ADD_F80, RTLIB::ADD_F128,
                                       RTLIB::ADD_PPCF128));
     break;
   case ISD::FMUL:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::MUL_F32, RTLIB::MUL_F64,

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

if (IP)
     CSEMap.InsertNode(N, IP);   // Memoize the new node.
   return N;
 }
 
 SDNode* SelectionDAG::mutateStrictFPToFP(SDNode *Node) {
   unsigned OrigOpc = Node->getOpcode();
   unsigned NewOpc;
   bool IsUnary = false;
+  bool IsTernary = false;
   switch (OrigOpc) {
   default:
     llvm_unreachable("mutateStrictFPToFP called with unexpected opcode!");
   case ISD::STRICT_FADD: NewOpc = ISD::FADD; break;
   case ISD::STRICT_FSUB: NewOpc = ISD::FSUB; break;
   case ISD::STRICT_FMUL: NewOpc = ISD::FMUL; break;
   case ISD::STRICT_FDIV: NewOpc = ISD::FDIV; break;
   case ISD::STRICT_FREM: NewOpc = ISD::FREM; break;
+  case ISD::STRICT_FMA: NewOpc = ISD::FMA; IsTernary = true; break;
   case ISD::STRICT_FSQRT: NewOpc = ISD::FSQRT; IsUnary = true; break;
   case ISD::STRICT_FPOW: NewOpc = ISD::FPOW; break;
   case ISD::STRICT_FPOWI: NewOpc = ISD::FPOWI; break;
   case ISD::STRICT_FSIN: NewOpc = ISD::FSIN; IsUnary = true; break;
   case ISD::STRICT_FCOS: NewOpc = ISD::FCOS; IsUnary = true; break;
   case ISD::STRICT_FEXP: NewOpc = ISD::FEXP; IsUnary = true; break;
   case ISD::STRICT_FEXP2: NewOpc = ISD::FEXP2; IsUnary = true; break;
   case ISD::STRICT_FLOG: NewOpc = ISD::FLOG; IsUnary = true; break;
SDNode* SelectionDAG::mutateStrictFPToFP(SDNode *Node) {
   SDValue InputChain = Node->getOperand(0);
   SDValue OutputChain = SDValue(Node, 1);
   ReplaceAllUsesOfValueWith(OutputChain, InputChain);
 
   SDVTList VTs = getVTList(Node->getOperand(1).getValueType());
   SDNode *Res = nullptr;
   if (IsUnary)
     Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1) });
+  else if (IsTernary)
+    Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1),
+                                           Node->getOperand(2),
+                                           Node->getOperand(3)});
   else
     Res = MorphNodeTo(Node, NewOpc, VTs, { Node->getOperand(1),
                                            Node->getOperand(2) });
 
   // MorphNodeTo can operate in two ways: if an existing node with the
   // specified operands exists, it can just return it.  Otherwise, it
   // updates the node in place to have the requested operands.
   if (Res == Node) {

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

setValue(&I, DAG.getNode(ISD::FMA, sdl,
                              getValue(I.getArgOperand(1)),
                              getValue(I.getArgOperand(2))));
     return nullptr;
   case Intrinsic::experimental_constrained_fadd:
   case Intrinsic::experimental_constrained_fsub:
   case Intrinsic::experimental_constrained_fmul:
   case Intrinsic::experimental_constrained_fdiv:
   case Intrinsic::experimental_constrained_frem:
+  case Intrinsic::experimental_constrained_fma:
   case Intrinsic::experimental_constrained_sqrt:
   case Intrinsic::experimental_constrained_pow:
   case Intrinsic::experimental_constrained_powi:
   case Intrinsic::experimental_constrained_sin:
   case Intrinsic::experimental_constrained_cos:
   case Intrinsic::experimental_constrained_exp:
   case Intrinsic::experimental_constrained_exp2:
   case Intrinsic::experimental_constrained_log:
case Intrinsic::experimental_constrained_fmul:
     Opcode = ISD::STRICT_FMUL;
     break;
   case Intrinsic::experimental_constrained_fdiv:
     Opcode = ISD::STRICT_FDIV;
     break;
   case Intrinsic::experimental_constrained_frem:
     Opcode = ISD::STRICT_FREM;
     break;
+  case Intrinsic::experimental_constrained_fma:
+    Opcode = ISD::STRICT_FMA;
+    break;
   case Intrinsic::experimental_constrained_sqrt:
     Opcode = ISD::STRICT_FSQRT;
     break;
   case Intrinsic::experimental_constrained_pow:
     Opcode = ISD::STRICT_FPOW;
     break;
   case Intrinsic::experimental_constrained_powi:
     Opcode = ISD::STRICT_FPOWI;
void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
   SDValue Chain = getRoot();
   SmallVector<EVT, 4> ValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
   ValueVTs.push_back(MVT::Other); // Out chain
 
   SDVTList VTs = DAG.getVTList(ValueVTs);
   SDValue Result;
   if (FPI.isUnaryOp())
-    Result = DAG.getNode(Opcode, sdl, VTs, 
+    Result = DAG.getNode(Opcode, sdl, VTs,
                          { Chain, getValue(FPI.getArgOperand(0)) });
+  else if (FPI.isTernaryOp())
+    Result = DAG.getNode(Opcode, sdl, VTs,
+                         { Chain, getValue(FPI.getArgOperand(0)),
+                                  getValue(FPI.getArgOperand(1)),
+                                  getValue(FPI.getArgOperand(2)) });
   else
-    Result = DAG.getNode(Opcode, sdl, VTs, 
+    Result = DAG.getNode(Opcode, sdl, VTs,
                          { Chain, getValue(FPI.getArgOperand(0)),
                            getValue(FPI.getArgOperand(1))  });
 
   assert(Result.getNode()->getNumValues() == 2);
   SDValue OutChain = Result.getValue(1);
   DAG.setRoot(OutChain);
   SDValue FPResult = Result.getValue(0);
   setValue(&FPI, FPResult);

llvm/trunk/lib/IR/IntrinsicInst.cpp

 //===-- InstrinsicInst.cpp - Intrinsic Instruction Wrappers ---------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements methods that make it really easy to deal with intrinsic
 // functions.
 //
 // All intrinsic function calls are instances of the call instruction, so these
 // are all subclasses of the CallInst class.  Note that none of these classes
 // has state or virtual methods, which is an important part of this gross/neat
 // hack working.
-// 
+//
 // In some cases, arguments to intrinsics need to be generic and are defined as
 // type pointer to empty struct { }*.  To access the real item of interest the
-// cast instruction needs to be stripped away. 
+// cast instruction needs to be stripped away.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Metadata.h"
Value *InstrProfIncrementInst::getStep() const {
   const Module *M = getModule();
   LLVMContext &Context = M->getContext();
   return ConstantInt::get(Type::getInt64Ty(Context), 1);
 }
 
 ConstrainedFPIntrinsic::RoundingMode
 ConstrainedFPIntrinsic::getRoundingMode() const {
   unsigned NumOperands = getNumArgOperands();
-  Metadata *MD = 
+  Metadata *MD =
       dyn_cast<MetadataAsValue>(getArgOperand(NumOperands - 2))->getMetadata();
   if (!MD || !isa<MDString>(MD))
     return rmInvalid;
   StringRef RoundingArg = cast<MDString>(MD)->getString();
 
   // For dynamic rounding mode, we use round to nearest but we will set the
   // 'exact' SDNodeFlag so that the value will not be rounded.
   return StringSwitch<RoundingMode>(RoundingArg)
     .Case("round.dynamic",    rmDynamic)
     .Case("round.tonearest",  rmToNearest)
     .Case("round.downward",   rmDownward)
     .Case("round.upward",     rmUpward)
     .Case("round.towardzero", rmTowardZero)
     .Default(rmInvalid);
 }
 
 ConstrainedFPIntrinsic::ExceptionBehavior
 ConstrainedFPIntrinsic::getExceptionBehavior() const {
   unsigned NumOperands = getNumArgOperands();
-  Metadata *MD = 
+  Metadata *MD =
       dyn_cast<MetadataAsValue>(getArgOperand(NumOperands - 1))->getMetadata();
   if (!MD || !isa<MDString>(MD))
     return ebInvalid;
   StringRef ExceptionArg = cast<MDString>(MD)->getString();
   return StringSwitch<ExceptionBehavior>(ExceptionArg)
     .Case("fpexcept.ignore",  ebIgnore)
     .Case("fpexcept.maytrap", ebMayTrap)
     .Case("fpexcept.strict",  ebStrict)
     .Default(ebInvalid);
 }
 
 bool ConstrainedFPIntrinsic::isUnaryOp() const {
   switch (getIntrinsicID()) {
-    default: 
+    default:
       return false;
     case Intrinsic::experimental_constrained_sqrt:
     case Intrinsic::experimental_constrained_sin:
     case Intrinsic::experimental_constrained_cos:
     case Intrinsic::experimental_constrained_exp:
     case Intrinsic::experimental_constrained_exp2:
     case Intrinsic::experimental_constrained_log:
     case Intrinsic::experimental_constrained_log10:
     case Intrinsic::experimental_constrained_log2:
     case Intrinsic::experimental_constrained_rint:
     case Intrinsic::experimental_constrained_nearbyint:
       return true;
   }
 }
+
+bool ConstrainedFPIntrinsic::isTernaryOp() const {
+  switch (getIntrinsicID()) {
+    default:
+      return false;
+    case Intrinsic::experimental_constrained_fma:
+      return true;
+  }
+}
+

llvm/trunk/lib/IR/Verifier.cpp

Assert(isa<ConstantInt>(CS.getArgOperand(1)),
            "constant int",
            CS);
     break;
   case Intrinsic::experimental_constrained_fadd:
   case Intrinsic::experimental_constrained_fsub:
   case Intrinsic::experimental_constrained_fmul:
   case Intrinsic::experimental_constrained_fdiv:
   case Intrinsic::experimental_constrained_frem:
+  case Intrinsic::experimental_constrained_fma:
   case Intrinsic::experimental_constrained_sqrt:
   case Intrinsic::experimental_constrained_pow:
   case Intrinsic::experimental_constrained_powi:
   case Intrinsic::experimental_constrained_sin:
   case Intrinsic::experimental_constrained_cos:
   case Intrinsic::experimental_constrained_exp:
   case Intrinsic::experimental_constrained_exp2:
   case Intrinsic::experimental_constrained_log:
static DISubprogram *getSubprogram(Metadata *LocalScope) {
 
   // Just return null; broken scope chains are checked elsewhere.
   assert(!isa<DILocalScope>(LocalScope) && "Unknown type of local scope");
   return nullptr;
 }
 
 void Verifier::visitConstrainedFPIntrinsic(ConstrainedFPIntrinsic &FPI) {
   unsigned NumOperands = FPI.getNumArgOperands();
-  Assert(((NumOperands == 3 && FPI.isUnaryOp()) || (NumOperands == 4)),
+  Assert(((NumOperands == 5 && FPI.isTernaryOp()) ||
+          (NumOperands == 3 && FPI.isUnaryOp()) || (NumOperands == 4)),
            "invalid arguments for constrained FP intrinsic", &FPI);
   Assert(isa<MetadataAsValue>(FPI.getArgOperand(NumOperands-1)),
          "invalid exception behavior argument", &FPI);
   Assert(isa<MetadataAsValue>(FPI.getArgOperand(NumOperands-2)),
          "invalid rounding mode argument", &FPI);
   Assert(FPI.getRoundingMode() != ConstrainedFPIntrinsic::rmInvalid,
          "invalid rounding mode argument", &FPI);
   Assert(FPI.getExceptionBehavior() != ConstrainedFPIntrinsic::ebInvalid,
          "invalid exception behavior argument", &FPI);

llvm/trunk/test/CodeGen/X86/fp-intrinsics.ll

-; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s | FileCheck %s
+; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s | FileCheck --check-prefix=COMMON --check-prefix=NO-FMA --check-prefix=FMACALL64 --check-prefix=FMACALL32 %s
+; RUN: llc -O3 -mtriple=x86_64-pc-linux -mattr=+fma < %s | FileCheck -check-prefix=COMMON --check-prefix=HAS-FMA --check-prefix=FMA64 --check-prefix=FMA32 %s
 
 ; Verify that constants aren't folded to inexact results when the rounding mode
 ; is unknown.
 ;
 ; double f1() {
 ;   // Because 0.1 cannot be represented exactly, this shouldn't be folded.
 ;   return 1.0/10.0;
 ; }
 ;
 ; CHECK-LABEL: f1
-; CHECK: divsd
+; COMMON: divsd
 define double @f1() {
 entry:
   %div = call double @llvm.experimental.constrained.fdiv.f64(
                                                double 1.000000e+00,
                                                double 1.000000e+01,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %div
 }
 
 ; Verify that 'a - 0' isn't simplified to 'a' when the rounding mode is unknown.
 ;
 ; double f2(double a) {
 ;   // Because the result of '0 - 0' is negative zero if rounding mode is
 ;   // downward, this shouldn't be simplified.
 ;   return a - 0;
 ; }
 ;
 ; CHECK-LABEL: f2
-; CHECK:  subsd
+; COMMON:  subsd
 define double @f2(double %a) {
 entry:
   %div = call double @llvm.experimental.constrained.fsub.f64(
                                                double %a,
                                                double 0.000000e+00,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %div
 }
 
 ; Verify that '-((-a)*b)' isn't simplified to 'a*b' when the rounding mode is
 ; unknown.
 ;
 ; double f3(double a, double b) {
 ;   // Because the intermediate value involved in this calculation may require
 ;   // rounding, this shouldn't be simplified.
 ;   return -((-a)*b);
 ; }
 ;
 ; CHECK-LABEL: f3:
-; CHECK:  subsd
-; CHECK:  mulsd
-; CHECK:  subsd
+; COMMON:  subsd
+; COMMON:  mulsd
+; COMMON:  subsd
 define double @f3(double %a, double %b) {
 entry:
   %sub = call double @llvm.experimental.constrained.fsub.f64(
                                                double -0.000000e+00, double %a,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   %mul = call double @llvm.experimental.constrained.fmul.f64(
                                                double %sub, double %b,

 ;
 ; double f4(int n, double a) {
 ;   // Because a + 1 may overflow, this should not be simplified.
 ;   if (n > 0)
 ;     return a + 1.0;
 ;   return a;
 ; }
 ;
-; 
+;
 ; CHECK-LABEL: f4:
-; CHECK: testl
-; CHECK: jle
-; CHECK: addsd
+; COMMON: testl
+; COMMON: jle
+; COMMON: addsd
 define double @f4(i32 %n, double %a) {
 entry:
   %cmp = icmp sgt i32 %n, 0
   br i1 %cmp, label %if.then, label %if.end
 
 if.then:
   %add = call double @llvm.experimental.constrained.fadd.f64(
                                                double 1.000000e+00, double %a,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   br label %if.end
 
 if.end:
   %a.0 = phi double [%add, %if.then], [ %a, %entry ]
   ret double %a.0
 }
 
 ; Verify that sqrt(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f5
-; CHECK:  sqrtsd
+; COMMON:  sqrtsd
 define double @f5() {
 entry:
   %result = call double @llvm.experimental.constrained.sqrt.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that pow(42.1, 3.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f6
-; CHECK:  pow
+; COMMON:  pow
 define double @f6() {
 entry:
   %result = call double @llvm.experimental.constrained.pow.f64(double 42.1,
                                                double 3.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that powi(42.1, 3) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f7
-; CHECK:  powi
+; COMMON:  powi
 define double @f7() {
 entry:
   %result = call double @llvm.experimental.constrained.powi.f64(double 42.1,
                                                i32 3,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that sin(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f8
-; CHECK:  sin
+; COMMON:  sin
 define double @f8() {
 entry:
   %result = call double @llvm.experimental.constrained.sin.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that cos(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f9
-; CHECK:  cos
+; COMMON:  cos
 define double @f9() {
 entry:
   %result = call double @llvm.experimental.constrained.cos.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that exp(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f10
-; CHECK:  exp
+; COMMON:  exp
 define double @f10() {
 entry:
   %result = call double @llvm.experimental.constrained.exp.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that exp2(42.1) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f11
-; CHECK:  exp2
+; COMMON:  exp2
 define double @f11() {
 entry:
   %result = call double @llvm.experimental.constrained.exp2.f64(double 42.1,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that log(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f12
-; CHECK:  log
+; COMMON:  log
 define double @f12() {
 entry:
   %result = call double @llvm.experimental.constrained.log.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that log10(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f13
-; CHECK:  log10
+; COMMON:  log10
 define double @f13() {
 entry:
   %result = call double @llvm.experimental.constrained.log10.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that log2(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f14
-; CHECK:  log2
+; COMMON:  log2
 define double @f14() {
 entry:
   %result = call double @llvm.experimental.constrained.log2.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that rint(42.1) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f15
-; CHECK:  rint
+; NO-FMA:  rint
+; HAS-FMA: vroundsd
 define double @f15() {
 entry:
   %result = call double @llvm.experimental.constrained.rint.f64(double 42.1,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
 ; Verify that nearbyint(42.1) isn't simplified when the rounding mode is
 ; unknown.
 ; CHECK-LABEL: f16
-; CHECK:  nearbyint
+; NO-FMA:  nearbyint
+; HAS-FMA: vroundsd
 define double @f16() {
 entry:
   %result = call double @llvm.experimental.constrained.nearbyint.f64(
                                                double 42.1,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
+; Verify that fma(3.5) isn't simplified when the rounding mode is
+; unknown.
+; CHECK-LABEL: f17
+; FMACALL32: jmp fmaf  # TAILCALL
+; FMA32: vfmadd213ss
+define float @f17() {
+entry:
+  %result = call float @llvm.experimental.constrained.fma.f32(
+                                               float 3.5,
+                                               float 3.5,
+                                               float 3.5,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret float %result
+}
+
+; Verify that fma(42.1) isn't simplified when the rounding mode is
+; unknown.
+; CHECK-LABEL: f18
+; FMACALL64: jmp fma  # TAILCALL
+; FMA64: vfmadd213sd
+define double @f18() {
+entry:
+  %result = call double @llvm.experimental.constrained.fma.f64(
+                                               double 42.1,
+                                               double 42.1,
+                                               double 42.1,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret double %result
+}
+
 @llvm.fp.env = thread_local global i8 zeroinitializer, section "llvm.metadata"
 declare double @llvm.experimental.constrained.fdiv.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fmul.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fadd.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fsub.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.sqrt.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.pow.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.powi.f64(double, i32, metadata, metadata)
 declare double @llvm.experimental.constrained.sin.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.cos.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log10.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.rint.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.nearbyint.f64(double, metadata, metadata)
+declare float @llvm.experimental.constrained.fma.f32(float, float, float, metadata, metadata)
+declare double @llvm.experimental.constrained.fma.f64(double, double, double, metadata, metadata)

llvm/trunk/test/Feature/fp-intrinsics.ll

 ;
 ; double f4(int n, double a) {
 ;   // Because a + 1 may overflow, this should not be simplified.
 ;   if (n > 0)
 ;     return a + 1.0;
 ;   return a;
 ; }
 ;
-; 
+;
 ; CHECK-LABEL: @f4
 ; CHECK-NOT: select
 ; CHECK: br i1 %cmp
 define double @f4(i32 %n, double %a) {
 entry:
   %cmp = icmp sgt i32 %n, 0
   br i1 %cmp, label %if.then, label %if.end
 
 if.then:
   %add = call double @llvm.experimental.constrained.fadd.f64(
                                                double 1.000000e+00, double %a,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   br label %if.end
 
 if.end:
   %a.0 = phi double [%add, %if.then], [ %a, %entry ]
   ret double %a.0
 }
 
-
 ; Verify that sqrt(42.0) isn't simplified when the rounding mode is unknown.
 ; CHECK-LABEL: f5
 ; CHECK: call double @llvm.experimental.constrained.sqrt
 define double @f5() {
 entry:
   %result = call double @llvm.experimental.constrained.sqrt.f64(double 42.0,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")

 entry:
   %result = call double @llvm.experimental.constrained.nearbyint.f64(
                                                double 42.1,
                                                metadata !"round.dynamic",
                                                metadata !"fpexcept.strict")
   ret double %result
 }
 
+; Verify that fma(42.1) isn't simplified when the rounding mode is
+; unknown.
+; CHECK-LABEL: f17
+; CHECK: call double @llvm.experimental.constrained.fma
+define double @f17() {
+entry:
+  %result = call double @llvm.experimental.constrained.fma.f64(double 42.1, double 42.1, double 42.1,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret double %result
+}
+
 @llvm.fp.env = thread_local global i8 zeroinitializer, section "llvm.metadata"
 declare double @llvm.experimental.constrained.fdiv.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fmul.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fadd.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.fsub.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.sqrt.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.pow.f64(double, double, metadata, metadata)
 declare double @llvm.experimental.constrained.powi.f64(double, i32, metadata, metadata)
 declare double @llvm.experimental.constrained.sin.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.cos.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.exp2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log10.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.log2.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.rint.f64(double, metadata, metadata)
 declare double @llvm.experimental.constrained.nearbyint.f64(double, metadata, metadata)
+declare double @llvm.experimental.constrained.fma.f64(double, double, double, metadata, metadata)