[FPEnv Core 14/14] Introduce F*_W_CHAIN instrs to prevent reordering

[sdmitrouk]

Summary

When floating-point environment is not examined, reordering instructions doesn't
harm, otherwise library call that checks floating-point environment state can be
moved before the actual computation, producing wrong results (not affected by
side-effects of operations).

[arsenm]

Please drop the AMDGPU changes. I don't see a point in trying to support these at this time, and they can't be meaningfully tested now. While the hardware supports floating point exceptions, they require a lot of additional compiler work to implement beyond the core operations. A lot of SGPRs need to be reserved and initialized, the control registers set to enable them, somehow supporting trap handlers and probably other things.

Other than that, Legal is a bad default for these. These should use the new LibCall action by default set in TargetLoweringBase. There is also a hasFloatingPointExceptions() target hook already, perhaps something could error if these are used and not supported.

include/llvm/CodeGen/ISDOpcodes.h

243

Typo: inary

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1422–1424

You can use a statically sized C array for this

4485

Using the integer size for address space 0 seems like a poor choice. Do targets actually care about this type? Why isn't it an i1 target constant?

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

202–206

A new utility function for isFPOpWithChain would be useful.

208–209

Why is this Op.getOperand(0) instead of just Op? I would also move the chain handling out of the loop and handle separately or have a separate w/chain and wo/chain loop

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2182–2186

A Chain out arguments looks weird here. Why can't UnrollVectorOp's result have the chain result?

lib/CodeGen/SelectionDAG/TargetLowering.cpp

92

Ops.empty()

lib/Target/AMDGPU/SIISelLowering.cpp

1309–1312 ↗

(On Diff #38417)

Why would only the last fmul need to be FMUL_W_CHAIN?

lib/Target/AMDGPU/SIInstructions.td

1450–1454 ↗

(On Diff #38417)

The instruction asm string should not include the _wchain. A better name I think for this would be V_ADD_F32_FPE or something else that doesn't mention chains.

[jholewinski]

lib/Target/NVPTX/NVPTXInstrInfo.td

696

Does it matter if the target does not actually support FP exceptions? This is not currently modeled in PTX.

[sdmitrouk]

lib/Target/NVPTX/NVPTXInstrInfo.td

696

Does it matter if the target does not actually support FP exceptions? This is not currently modeled in PTX.

Not really, this only fixes ordering of FP instructions, everything else (i.e., changes in floating-point state registers) is left for future improvements. I just wanted to ensure that new instructions won't cause selection failures on supported targets, but if you think this is useless for PTX or other targets I can remove these changes.

[jyknight]

Drive-by-comment...

Is there an advantage to duplicating FADD, FSUB, FMUL, FDIV, and FREM into a second W_CHAIN set, instead of having only a single set of ops which have the chain operand? It would naively seem better to have less duplication? (That is, if the compilation flags say to not care about fp ordering, the input chain would still be there, but would always be set to the entry node.)

[sdmitrouk]

In D14079#276541, @jyknight wrote:

Drive-by-comment...

Is there an advantage to duplicating FADD, FSUB, FMUL, FDIV, and FREM into a second W_CHAIN set, instead of having only a single set of ops which have the chain operand? It would naively seem better to have less duplication? (That is, if the compilation flags say to not care about fp ordering, the input chain would still be there, but would always be set to the entry node.)

This was the first implementation, but the issue was that in that case you can't (or I don't know how) use same instruction twice to create one node with side effects and the other one without (and model changes in related registers probably too). Hal actually said that one could add new instructions and for me it also appeared like a safer choice to do not break current floating-point math.

[sdmitrouk]

Dropped AMDGPU. Addressed comments.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

4485

This is a rewrite of code in the case directly above, I just made it aware of the chain.

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

208–209

Thanks, it should be Op.

I would also move the chain handling out of the loop and handle separately or have a separate w/chain and wo/chain loop

It's not the chain of the node we're processing, it's reference to the chain of an operand. Node shouldn't be legalized by reference to its chain, which will be replaced with new value when such node gets legalized through reverence to its value.

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2182–2186

Why can't UnrollVectorOp's result have the chain result?

Because its result is processed is assumed to have type of vector element and it is processed that way.

[mehdi_amini]

Haven't looked at the target specific code, seems mostly mechanical changes.
Some comments on the generic part.

include/llvm/CodeGen/SelectionDAG.h

1228

Remove spurious space change.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

1338

I'm not sure I understand what's going on here, this need to be documented.

8701

It is not clear to me that this will preserve the FP env?
Also "UnsafeFPMath" seems fundamentally not compatible with FP env, so it smells.

lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

1064

spurious change

1070

spurious formatting change?

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

170

Not clear how it relates to the "FP with chain" stuff.

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

114

spurious formatting change

2058

In the rest of the patch, I think you grouped the *_W_CHAIN cases.

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

3928

I'm not convince you maintain correct FPenv semantics here.

3936

Another case of mixing UnsafeFPMath with FPenv, I'm not sure about where this is going. Shouldn't we just drop the chain and turn every XXX_W_CHAIN into XXX when UnsafeFPMath is enabled?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2352

This has changed recently :)

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     SSUBO, USUBO,
 
     /// Same for multiplication.
     SMULO, UMULO,
 
     /// Simple binary floating point operators.
     FADD, FSUB, FMUL, FDIV, FREM,
 
+    /// Simple binary floating point operators with side effects that have token
+    /// chains as their first operand.
+    FADD_W_CHAIN, FSUB_W_CHAIN, FMUL_W_CHAIN, FDIV_W_CHAIN, FREM_W_CHAIN,
+
     /// FMA - Perform a * b + c with no intermediate rounding step.
     FMA,
 
     /// FMAD - Perform a * b + c, while getting the same result as the
     /// separately rounded operations.
     FMAD,
 
     /// FCOPYSIGN(X, Y) - Return the value of X with the sign of Y.  NOTE: This
enum CvtCode {
     CVT_SU,     /// Signed from Unsigned
     CVT_US,     /// Unsigned from Signed
     CVT_UU,     /// Unsigned from Unsigned
     CVT_INVALID /// Marker - Invalid opcode
   };
 
 } // end llvm::ISD namespace
 
+/// Returns true if the opcode is one of floating-operations with chain.
+inline bool isFPOpWithChain(unsigned OpCode) {
+  return OpCode == ISD::FADD_W_CHAIN ||
+         OpCode == ISD::FSUB_W_CHAIN ||
+         OpCode == ISD::FDIV_W_CHAIN ||
+         OpCode == ISD::FREM_W_CHAIN ||
+         OpCode == ISD::FMUL_W_CHAIN;
+}
+
 } // end llvm namespace
 
 #endif

include/llvm/CodeGen/SelectionDAG.h

#endif
   ///
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT,
                   ArrayRef<SDUse> Ops);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT,
                   ArrayRef<SDValue> Ops, const SDNodeFlags *Flags = nullptr);
   SDValue getNode(unsigned Opcode, SDLoc DL, ArrayRef<EVT> ResultTys,
                   ArrayRef<SDValue> Ops);
   SDValue getNode(unsigned Opcode, SDLoc DL, SDVTList VTs,
-                  ArrayRef<SDValue> Ops);
+                  ArrayRef<SDValue> Ops, const SDNodeFlags *Flags = nullptr);
 
   // Specialize based on number of operands.
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1, SDValue N2,
                   const SDNodeFlags *Flags = nullptr);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1, SDValue N2,
                   SDValue N3);
+  SDValue getNode(unsigned Opcode, SDLoc DL, SDVTList VTs, SDValue Chain,
+                  SDValue N1, SDValue N2, const SDNodeFlags *Flags);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1, SDValue N2,
                   SDValue N3, SDValue N4);
   SDValue getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1, SDValue N2,
                   SDValue N3, SDValue N4, SDValue N5);
 
   // Specialize again based on number of operands for nodes with a VTList
   // rather than a single VT.
   SDValue getNode(unsigned Opcode, SDLoc DL, SDVTList VTs);
#endif
 
   /// Return true if A and B have no common bits set. As an example, this can
   /// allow an 'add' to be transformed into an 'or'.
   bool haveNoCommonBitsSet(SDValue A, SDValue B) const;
 
   /// Utility function used by legalize and lowering to
   /// "unroll" a vector operation by splitting out the scalars and operating
   /// on each element individually.  If the ResNE is 0, fully unroll the vector
-  /// op. If ResNE is less than the width of the vector op, unroll up to ResNE.
-  /// If the  ResNE is greater than the width of the vector op, unroll the
-  /// vector op and fill the end of the resulting vector with UNDEFS.
-  SDValue UnrollVectorOp(SDNode *N, unsigned ResNE = 0);
+  /// op.  If ResNE is less than the width of the vector op, unroll up to ResNE.
+  /// If the ResNE is greater than the width of the vector op, unroll the
+  /// vector op and fill the end of the resulting vector with UNDEFS.  Optional
+  /// TokenFactor argument can be set for floating-point instructions that have
+  /// chains.
+  SDValue UnrollVectorOp(SDNode *N, unsigned ResNE = 0,
+                         SDValue *TokenFactor = nullptr);
 
   /// Return true if LD is loading 'Bytes' bytes from a location that is 'Dist'
   /// units away from the location that the 'Base' load is loading from.
   bool isConsecutiveLoad(LoadSDNode *LD, LoadSDNode *Base,
                          unsigned Bytes, int Dist) const;
 
   /// Infer alignment of a load / store address. Return 0 if
   /// it cannot be inferred.
private:
   void DeallocateNode(SDNode *N);
 
   void allnodes_clear();
 
   BinarySDNode *GetBinarySDNode(unsigned Opcode, SDLoc DL, SDVTList VTs,
                                 SDValue N1, SDValue N2,
                                 const SDNodeFlags *Flags = nullptr);
 
+  BinarySDNode *GetBinarySDNode(unsigned Opcode, SDLoc DL, SDVTList VTs,
+                                SDValue Chain, SDValue N1, SDValue N2,
+                                const SDNodeFlags *Flags = nullptr);
+
   /// Look up the node specified by ID in CSEMap.  If it exists, return it.  If
   /// not, return the insertion token that will make insertion faster.  This
   /// overload is for nodes other than Constant or ConstantFP, use the other one
   /// for those.
   SDNode *FindNodeOrInsertPos(const FoldingSetNodeID &ID, void *&InsertPos);
 
   /// Look up the node specified by ID in CSEMap.  If it exists, return it.  If
   /// not, return the insertion token that will make insertion faster.  Performs

include/llvm/CodeGen/SelectionDAGNodes.h

public:
 
   /// get the index which selects a specific result in the SDNode
   unsigned getResNo() const { return ResNo; }
 
   /// get the SDNode which holds the desired result
   SDNode *getNode() const { return Node; }
 
   /// set the SDNode
-  void setNode(SDNode *N) { Node = N; }
+  inline void setNode(SDNode *N);
 
   inline SDNode *operator->() const { return Node; }
 
   bool operator==(const SDValue &O) const {
     return Node == O.Node && ResNo == O.ResNo;
   }
   bool operator!=(const SDValue &O) const {
     return !operator==(O);
DebugLoc getDebugLoc() {
     const SDNode *N = (const SDNode*)(Ptr);
     return N->getDebugLoc();
   }
 };
 
 
 // Define inline functions from the SDValue class.
 
+void SDValue::setNode(SDNode *N) {
+  assert((!N || ResNo < N->getNumValues()) && "Wrong ResNo for new node.");
+  Node = N;
+}
+
 inline SDValue::SDValue(SDNode *node, unsigned resno)
     : Node(node), ResNo(resno) {
   assert((!Node || ResNo < Node->getNumValues()) &&
          "Invalid result number for the given node!");
   assert(ResNo < -2U && "Cannot use result numbers reserved for DenseMaps.");
 }
 
 inline unsigned SDValue::getOpcode() const {
UnarySDNode(unsigned Opc, unsigned Order, DebugLoc dl, SDVTList VTs,
     : SDNode(Opc, Order, dl, VTs) {
     InitOperands(&Op, X);
   }
 };
 
 /// This class is used for two-operand SDNodes.  This is solely
 /// to allow co-allocation of node operands with the node itself.
 class BinarySDNode : public SDNode {
-  SDUse Ops[2];
+  SDUse Ops[3];
 public:
   BinarySDNode(unsigned Opc, unsigned Order, DebugLoc dl, SDVTList VTs,
                SDValue X, SDValue Y)
     : SDNode(Opc, Order, dl, VTs) {
     InitOperands(Ops, X, Y);
   }
+  BinarySDNode(unsigned Opc, unsigned Order, DebugLoc dl, SDVTList VTs,
+               SDValue Chain, SDValue X, SDValue Y)
+    : SDNode(Opc, Order, dl, VTs) {
+    InitOperands(Ops, Chain, X, Y);
+  }
 };
 
 /// Returns true if the opcode is a binary operation with flags.
 static bool isBinOpWithFlags(unsigned Opcode) {
   switch (Opcode) {
   case ISD::SDIV:
   case ISD::UDIV:
   case ISD::SRA:
   case ISD::SRL:
   case ISD::MUL:
   case ISD::ADD:
   case ISD::SUB:
   case ISD::SHL:
   case ISD::FADD:
   case ISD::FDIV:
   case ISD::FMUL:
   case ISD::FREM:
   case ISD::FSUB:
+
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+
     return true;
   default:
     return false;
   }
 }
 
 /// This class is an extension of BinarySDNode
 /// used from those opcodes that have associated extra flags.
 class BinaryWithFlagsSDNode : public BinarySDNode {
 public:
   SDNodeFlags Flags;
   BinaryWithFlagsSDNode(unsigned Opc, unsigned Order, DebugLoc dl, SDVTList VTs,
                         SDValue X, SDValue Y, const SDNodeFlags &NodeFlags)
       : BinarySDNode(Opc, Order, dl, VTs, X, Y), Flags(NodeFlags) {}
+  BinaryWithFlagsSDNode(unsigned Opc, unsigned Order, DebugLoc dl, SDVTList VTs,
+                        SDValue Chain, SDValue X, SDValue Y,
+                        const SDNodeFlags &NodeFlags)
+      : BinarySDNode(Opc, Order, dl, VTs, Chain, X, Y), Flags(NodeFlags) {}
   static bool classof(const SDNode *N) {
     return isBinOpWithFlags(N->getOpcode());
   }
 };
 
 /// This class is used for three-operand SDNodes. This is solely
 /// to allow co-allocation of node operands with the node itself.
 class TernarySDNode : public SDNode {

include/llvm/IR/Instructions.h

 /// Returns true if the ordering is at least as strong as release
 /// (i.e. release, acq_rel or seq_cst)
 inline bool isAtLeastRelease(AtomicOrdering Ord) {
 return (Ord == Release ||
     Ord == AcquireRelease ||
     Ord == SequentiallyConsistent);
 }
 
+/// Returns true if the opcode is one of floating-operations with chain.
+inline bool isFPOpWithChain(unsigned OpCode) {
+  return OpCode == ISD::FADD_W_CHAIN ||
+         OpCode == ISD::FSUB_W_CHAIN ||
+         OpCode == ISD::FDIV_W_CHAIN ||
+         OpCode == ISD::FREM_W_CHAIN ||
+         OpCode == ISD::FMUL_W_CHAIN;
+}
+
 //===----------------------------------------------------------------------===//
 //                                AllocaInst Class
 //===----------------------------------------------------------------------===//
 
 /// AllocaInst - an instruction to allocate memory on the stack
 ///
 class AllocaInst : public UnaryInstruction {
   Type *AllocatedType;

include/llvm/IR/IntrinsicsX86.td

 //===----------------------------------------------------------------------===//
 // SSE1
 
 // Arithmetic ops
 let TargetPrefix = "x86" in {  // All intrinsics start with "llvm.x86.".
   def int_x86_sse_add_ss : GCCBuiltin<"__builtin_ia32_addss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
                          llvm_v4f32_ty], [IntrNoMem]>;
+  def int_x86_sse_addwchain_ss : Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
+                                            llvm_v4f32_ty], [IntrReadMem]>;
   def int_x86_sse_sub_ss : GCCBuiltin<"__builtin_ia32_subss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
                          llvm_v4f32_ty], [IntrNoMem]>;
+  def int_x86_sse_subwchain_ss : Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
+                                            llvm_v4f32_ty], [IntrNoMem]>;
   def int_x86_sse_mul_ss : GCCBuiltin<"__builtin_ia32_mulss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
                          llvm_v4f32_ty], [IntrNoMem]>;
+  def int_x86_sse_mulwchain_ss : Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
+                                            llvm_v4f32_ty], [IntrNoMem]>;
   def int_x86_sse_div_ss : GCCBuiltin<"__builtin_ia32_divss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
                          llvm_v4f32_ty], [IntrNoMem]>;
+  def int_x86_sse_divwchain_ss : Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty,
+                                            llvm_v4f32_ty], [IntrNoMem]>;
   def int_x86_sse_sqrt_ss : GCCBuiltin<"__builtin_ia32_sqrtss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty],
                         [IntrNoMem]>;
   def int_x86_sse_sqrt_ps : GCCBuiltin<"__builtin_ia32_sqrtps">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty],
                         [IntrNoMem]>;
   def int_x86_sse_rcp_ss : GCCBuiltin<"__builtin_ia32_rcpss">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty],

 //===----------------------------------------------------------------------===//
 // SSE2
 
 // FP arithmetic ops
 let TargetPrefix = "x86" in {  // All intrinsics start with "llvm.x86.".
   def int_x86_sse2_add_sd : GCCBuiltin<"__builtin_ia32_addsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
                          llvm_v2f64_ty], [IntrNoMem]>;
+  def int_x86_sse2_addwchain_sd : Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
+                                             llvm_v2f64_ty], [IntrNoMem]>;
   def int_x86_sse2_sub_sd : GCCBuiltin<"__builtin_ia32_subsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
                          llvm_v2f64_ty], [IntrNoMem]>;
+  def int_x86_sse2_subwchain_sd : Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
+                                             llvm_v2f64_ty], [IntrNoMem]>;
   def int_x86_sse2_mul_sd : GCCBuiltin<"__builtin_ia32_mulsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
                          llvm_v2f64_ty], [IntrNoMem]>;
+  def int_x86_sse2_mulwchain_sd : Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
+                                             llvm_v2f64_ty], [IntrNoMem]>;
   def int_x86_sse2_div_sd : GCCBuiltin<"__builtin_ia32_divsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
                          llvm_v2f64_ty], [IntrNoMem]>;
+  def int_x86_sse2_divwchain_sd : Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,
+                                             llvm_v2f64_ty], [IntrNoMem]>;
   def int_x86_sse2_sqrt_sd : GCCBuiltin<"__builtin_ia32_sqrtsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty],
                         [IntrNoMem]>;
   def int_x86_sse2_sqrt_pd : GCCBuiltin<"__builtin_ia32_sqrtpd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty],
                         [IntrNoMem]>;
   def int_x86_sse2_min_sd : GCCBuiltin<"__builtin_ia32_minsd">,
               Intrinsic<[llvm_v2f64_ty], [llvm_v2f64_ty,

include/llvm/Target/TargetSelectionDAG.td

 def extractelt : SDNode<"ISD::EXTRACT_VECTOR_ELT", SDTVecExtract>;
 def insertelt  : SDNode<"ISD::INSERT_VECTOR_ELT", SDTVecInsert>;
 
 def fadd       : SDNode<"ISD::FADD"       , SDTFPBinOp, [SDNPCommutative]>;
 def fsub       : SDNode<"ISD::FSUB"       , SDTFPBinOp>;
 def fmul       : SDNode<"ISD::FMUL"       , SDTFPBinOp, [SDNPCommutative]>;
 def fdiv       : SDNode<"ISD::FDIV"       , SDTFPBinOp>;
 def frem       : SDNode<"ISD::FREM"       , SDTFPBinOp>;
+def faddwchain : SDNode<"ISD::FADD_W_CHAIN", SDTFPBinOp, [SDNPCommutative, SDNPHasChain, SDNPSideEffect]>;
+def fsubwchain : SDNode<"ISD::FSUB_W_CHAIN", SDTFPBinOp, [SDNPHasChain, SDNPSideEffect]>;
+def fmulwchain : SDNode<"ISD::FMUL_W_CHAIN", SDTFPBinOp, [SDNPCommutative, SDNPHasChain, SDNPSideEffect]>;
+def fdivwchain : SDNode<"ISD::FDIV_W_CHAIN", SDTFPBinOp, [SDNPHasChain, SDNPSideEffect]>;
+def fremwchain : SDNode<"ISD::FREM_W_CHAIN", SDTFPBinOp, [SDNPHasChain, SDNPSideEffect]>;
 def fma        : SDNode<"ISD::FMA"        , SDTFPTernaryOp>;
 def fmad       : SDNode<"ISD::FMAD"       , SDTFPTernaryOp>;
 def fabs       : SDNode<"ISD::FABS"       , SDTFPUnaryOp>;
 def fminnum    : SDNode<"ISD::FMINNUM"    , SDTFPBinOp>;
 def fmaxnum    : SDNode<"ISD::FMAXNUM"    , SDTFPBinOp>;
 def fminnan    : SDNode<"ISD::FMINNAN"    , SDTFPBinOp>;
 def fmaxnan    : SDNode<"ISD::FMAXNAN"    , SDTFPBinOp>;
 def fgetsign   : SDNode<"ISD::FGETSIGN"   , SDTFPToIntOp>;

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue visitTRUNCATE(SDNode *N);
     SDValue visitBITCAST(SDNode *N);
     SDValue visitBUILD_PAIR(SDNode *N);
     SDValue visitFADD(SDNode *N);
     SDValue visitFSUB(SDNode *N);
     SDValue visitFMUL(SDNode *N);
     SDValue visitFMA(SDNode *N);
     SDValue visitFDIV(SDNode *N);
+    SDValue visitFDIV_W_CHAIN(SDNode *N);
     SDValue visitFREM(SDNode *N);
     SDValue visitFSQRT(SDNode *N);
     SDValue visitFCOPYSIGN(SDNode *N);
     SDValue visitSINT_TO_FP(SDNode *N);
     SDValue visitUINT_TO_FP(SDNode *N);
     SDValue visitFP_TO_SINT(SDNode *N);
     SDValue visitFP_TO_UINT(SDNode *N);
     SDValue visitFP_ROUND(SDNode *N);
assert(N->getOpcode() != ISD::DELETED_NODE &&
            RV.getNode()->getOpcode() != ISD::DELETED_NODE &&
            "Node was deleted but visit returned new node!");
 
     DEBUG(dbgs() << " ... into: ";
           RV.getNode()->dump(&DAG));
 
     // Transfer debug value.
     DAG.TransferDbgValues(SDValue(N, 0), RV);
-    if (N->getNumValues() == RV.getNode()->getNumValues())
+    if (N->getNumValues() == RV.getNode()->getNumValues()) {
       DAG.ReplaceAllUsesWith(N, RV.getNode());
-    else {
-      assert(N->getValueType(0) == RV.getValueType() &&
-             N->getNumValues() == 1 && "Type mismatch");
+    } else if (N->getNumValues() == 2 && N->getValueType(1) == MVT::Other) {
+      assert(N->getValueType(0) == RV.getValueType() && "Type mismatch");
+      assert(N->getOpcode() == ISD::FDIV_W_CHAIN && "Wrong opcode");
+      SDValue OpV[] = { RV, DAG.getEntryNode() };
+      DAG.ReplaceAllUsesWith(N, OpV);
+    } else {
+      assert(N->getValueType(0) == RV.getValueType() && "Type mismatch");
+      assert(N->getNumValues() == 1 && "Type mismatch");
       SDValue OpV = RV;
       DAG.ReplaceAllUsesWith(N, &OpV);
     }
 
     // Push the new node and any users onto the worklist
     AddToWorklist(RV.getNode());
     AddUsersToWorklist(RV.getNode());
 
SDValue DAGCombiner::visit(SDNode *N) {
   case ISD::TRUNCATE:           return visitTRUNCATE(N);
   case ISD::BITCAST:            return visitBITCAST(N);
   case ISD::BUILD_PAIR:         return visitBUILD_PAIR(N);
   case ISD::FADD:               return visitFADD(N);
   case ISD::FSUB:               return visitFSUB(N);
   case ISD::FMUL:               return visitFMUL(N);
   case ISD::FMA:                return visitFMA(N);
   case ISD::FDIV:               return visitFDIV(N);
+  case ISD::FDIV_W_CHAIN:       return visitFDIV_W_CHAIN(N);
   case ISD::FREM:               return visitFREM(N);
   case ISD::FSQRT:              return visitFSQRT(N);
   case ISD::FCOPYSIGN:          return visitFCOPYSIGN(N);
   case ISD::SINT_TO_FP:         return visitSINT_TO_FP(N);
   case ISD::UINT_TO_FP:         return visitUINT_TO_FP(N);
   case ISD::FP_TO_SINT:         return visitFP_TO_SINT(N);
   case ISD::FP_TO_UINT:         return visitFP_TO_UINT(N);
   case ISD::FP_ROUND:           return visitFP_ROUND(N);
SDValue DAGCombiner::visitFDIV(SDNode *N) {
   }
 
   if (SDValue CombineRepeatedDivisors = combineRepeatedFPDivisors(N))
     return CombineRepeatedDivisors;
 
   return SDValue();
 }
 
+SDValue DAGCombiner::visitFDIV_W_CHAIN(SDNode *N) {
+  SDValue Chain = N->getOperand(0);
+  SDValue N0 = N->getOperand(1);
+  SDValue N1 = N->getOperand(2);
+  EVT VT = N->getValueType(0);
+  SDLoc DL(N);
+  const TargetOptions &Options = DAG.getTarget().Options;
+  SDNodeFlags *Flags = &cast<BinaryWithFlagsSDNode>(N)->Flags;
+
+  if (Options.UnsafeFPMath) {
+    // Fold into a reciprocal estimate and multiply instead of a real divide.
+    if (SDValue RV = BuildReciprocalEstimate(N1, Flags)) {
+      AddToWorklist(RV.getNode());
+      return DAG.getNode(ISD::FMUL_W_CHAIN, DL, DAG.getVTList(VT, MVT::Other),
+                         Chain, N0, RV, Flags);
+    }
+  }
+
+  return SDValue();
+}
+
 SDValue DAGCombiner::visitFREM(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   ConstantFPSDNode *N0CFP = dyn_cast<ConstantFPSDNode>(N0);
   ConstantFPSDNode *N1CFP = dyn_cast<ConstantFPSDNode>(N1);
   EVT VT = N->getValueType(0);
 
   // fold (frem c1, c2) -> fmod(c1,c2)

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

SDValue ExpandLibCall(RTLIB::Libcall LC, EVT RetVT, const SDValue *Ops,
                         unsigned NumOps, bool isSigned, SDLoc dl);
 
   std::pair<SDValue, SDValue> ExpandChainLibCall(RTLIB::Libcall LC,
                                                  SDNode *Node, bool isSigned);
   SDValue ExpandFPLibCall(SDNode *Node, RTLIB::Libcall Call_F32,
                           RTLIB::Libcall Call_F64, RTLIB::Libcall Call_F80,
                           RTLIB::Libcall Call_F128,
                           RTLIB::Libcall Call_PPCF128);
+  std::pair<SDValue, SDValue> ExpandChainFPLibCall(SDNode *Node,
+                                                   RTLIB::Libcall Call_F32,
+                                                   RTLIB::Libcall Call_F64,
+                                                   RTLIB::Libcall Call_F80,
+                                                   RTLIB::Libcall Call_F128,
+                                                   RTLIB::Libcall Call_PPCF128);
   SDValue ExpandIntLibCall(SDNode *Node, bool isSigned,
                            RTLIB::Libcall Call_I8,
                            RTLIB::Libcall Call_I16,
                            RTLIB::Libcall Call_I32,
                            RTLIB::Libcall Call_I64,
                            RTLIB::Libcall Call_I128);
   void ExpandDivRemLibCall(SDNode *Node, SmallVectorImpl<SDValue> &Results);
   void ExpandSinCosLibCall(SDNode *Node, SmallVectorImpl<SDValue> &Results);
case TargetLowering::Custom: {
           return;
 
         if (Node->getNumValues() == 1) {
           // We can just directly replace this node with the lowered value.
           ReplaceNode(SDValue(Node, 0), Res);
           return;
         }
 
+        // Handle selection of a node with a chain into node without one.
+        if (Node->getNumValues() == 2 && Res->getNumValues() == 1 &&
+            Node->getValueType(1) == MVT::Other) {
+          SDValue ResultVals[] = { Res.getValue(0), DAG.getEntryNode() };
+          ReplaceNode(Node, &ResultVals[0]);
+          return;
+        }
+
+        assert(Node->getNumValues() <= Res->getNumValues() &&
+               "Too few values in new node!");
         SmallVector<SDValue, 8> ResultVals;
         for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
           ResultVals.push_back(Res.getValue(i));
         ReplaceNode(Node, ResultVals.data());
         return;
       }
     }
       // FALL THROUGH
CLI.setDebugLoc(SDLoc(Node)).setChain(InChain)
     .setCallee(TLI.getLibcallCallingConv(LC), RetTy, Callee, std::move(Args), 0)
     .setSExtResult(isSigned).setZExtResult(!isSigned);
 
   std::pair<SDValue, SDValue> CallInfo = TLI.LowerCallTo(CLI);
 
   return CallInfo;
 }
 
-SDValue SelectionDAGLegalize::ExpandFPLibCall(SDNode* Node,
-                                              RTLIB::Libcall Call_F32,
-                                              RTLIB::Libcall Call_F64,
-                                              RTLIB::Libcall Call_F80,
-                                              RTLIB::Libcall Call_F128,
-                                              RTLIB::Libcall Call_PPCF128) {
+static RTLIB::Libcall GetFPLibCall(SDNode *Node, RTLIB::Libcall Call_F32,
   RTLIB::Libcall LC;
   switch (Node->getSimpleValueType(0).SimpleTy) {
   default: llvm_unreachable("Unexpected request for libcall!");
   case MVT::f32: LC = Call_F32; break;
   case MVT::f64: LC = Call_F64; break;
   case MVT::f80: LC = Call_F80; break;
   case MVT::f128: LC = Call_F128; break;
   case MVT::ppcf128: LC = Call_PPCF128; break;
   }
+  return LC;
+}
+
+SDValue SelectionDAGLegalize::ExpandFPLibCall(SDNode* Node,
+                                              RTLIB::Libcall Call_F32,
+                                              RTLIB::Libcall Call_F64,
+                                              RTLIB::Libcall Call_F80,
+                                              RTLIB::Libcall Call_F128,
+                                              RTLIB::Libcall Call_PPCF128) {
+  RTLIB::Libcall LC = GetFPLibCall(Node, Call_F32, Call_F64, Call_F80,
+                                   Call_F128, Call_PPCF128);
   return ExpandLibCall(LC, Node, false);
 }
 
+std::pair<SDValue, SDValue> SelectionDAGLegalize::ExpandChainFPLibCall(
+                                                  SDNode* Node,
+                                                  RTLIB::Libcall Call_F32,
+                                                  RTLIB::Libcall Call_F64,
+                                                  RTLIB::Libcall Call_F80,
+                                                  RTLIB::Libcall Call_F128,
+                                                  RTLIB::Libcall Call_PPCF128) {
+  RTLIB::Libcall LC = GetFPLibCall(Node, Call_F32, Call_F64, Call_F80,
+                                   Call_F128, Call_PPCF128);
+  return ExpandChainLibCall(LC, Node, false);
+}
+
 SDValue SelectionDAGLegalize::ExpandIntLibCall(SDNode* Node, bool isSigned,
                                                RTLIB::Libcall Call_I8,
                                                RTLIB::Libcall Call_I16,
                                                RTLIB::Libcall Call_I32,
                                                RTLIB::Libcall Call_I64,
                                                RTLIB::Libcall Call_I128) {
   RTLIB::Libcall LC;
   switch (Node->getSimpleValueType(0).SimpleTy) {
if (Results.empty())
     return false;
 
   ReplaceNode(Node, Results.data());
   return true;
 }
 
 void SelectionDAGLegalize::ConvertNodeToLibcall(SDNode *Node) {
   SmallVector<SDValue, 8> Results;
+  std::pair<SDValue, SDValue> CallInfo;
   SDLoc dl(Node);
   SDValue Tmp1, Tmp2, Tmp3, Tmp4;
   unsigned Opc = Node->getOpcode();
   switch (Opc) {
   case ISD::ATOMIC_FENCE: {
     // If the target didn't lower this, lower it to '__sync_synchronize()' call
     // FIXME: handle "fence singlethread" more efficiently.
     TargetLowering::ArgListTy Args;
Results.push_back(ExpandFPLibCall(Node, RTLIB::POW_F32, RTLIB::POW_F64,
                                       RTLIB::POW_F80, RTLIB::POW_F128,
                                       RTLIB::POW_PPCF128));
     break;
   case ISD::FDIV:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::DIV_F32, RTLIB::DIV_F64,
                                       RTLIB::DIV_F80, RTLIB::DIV_F128,
                                       RTLIB::DIV_PPCF128));
     break;
+  case ISD::FDIV_W_CHAIN:
+    CallInfo = ExpandChainFPLibCall(Node, RTLIB::DIV_F32, RTLIB::DIV_F64,
+                                    RTLIB::DIV_F80, RTLIB::DIV_F128,
+                                    RTLIB::DIV_PPCF128);
+    Results.push_back(CallInfo.first);
+    Results.push_back(CallInfo.second);
+    break;
   case ISD::FREM:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::REM_F32, RTLIB::REM_F64,
                                       RTLIB::REM_F80, RTLIB::REM_F128,
                                       RTLIB::REM_PPCF128));
     break;
+  case ISD::FREM_W_CHAIN:
+    CallInfo = ExpandChainFPLibCall(Node, RTLIB::REM_F32, RTLIB::REM_F64,
+                                    RTLIB::REM_F80, RTLIB::REM_F128,
+                                    RTLIB::REM_PPCF128);
+    Results.push_back(CallInfo.first);
+    Results.push_back(CallInfo.second);
+    break;
   case ISD::FMA:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::FMA_F32, RTLIB::FMA_F64,
                                       RTLIB::FMA_F80, RTLIB::FMA_F128,
                                       RTLIB::FMA_PPCF128));
     break;
   case ISD::FADD:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::ADD_F32, RTLIB::ADD_F64,
                                       RTLIB::ADD_F80, RTLIB::ADD_F128,
                                       RTLIB::ADD_PPCF128));
     break;
+  case ISD::FADD_W_CHAIN:
+    CallInfo = ExpandChainFPLibCall(Node, RTLIB::ADD_F32, RTLIB::ADD_F64,
+                                    RTLIB::ADD_F80, RTLIB::ADD_F128,
+                                    RTLIB::ADD_PPCF128);
+    Results.push_back(CallInfo.first);
+    Results.push_back(CallInfo.second);
+    break;
   case ISD::FMUL:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::MUL_F32, RTLIB::MUL_F64,
                                       RTLIB::MUL_F80, RTLIB::MUL_F128,
                                       RTLIB::MUL_PPCF128));
     break;
+  case ISD::FMUL_W_CHAIN:
+    CallInfo = ExpandChainFPLibCall(Node, RTLIB::MUL_F32, RTLIB::MUL_F64,
+                                    RTLIB::MUL_F80, RTLIB::MUL_F128,
+                                    RTLIB::MUL_PPCF128);
+    Results.push_back(CallInfo.first);
+    Results.push_back(CallInfo.second);
+    break;
   case ISD::FP16_TO_FP:
     if (Node->getValueType(0) == MVT::f32) {
       Results.push_back(ExpandLibCall(RTLIB::FPEXT_F16_F32, Node, false));
     }
     break;
   case ISD::FP_TO_FP16: {
     RTLIB::Libcall LC =
         RTLIB::getFPROUND(Node->getOperand(0).getValueType(), MVT::f16);
     assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unable to expand fp_to_fp16");
     Results.push_back(ExpandLibCall(LC, Node, false));
     break;
   }
   case ISD::FSUB:
     Results.push_back(ExpandFPLibCall(Node, RTLIB::SUB_F32, RTLIB::SUB_F64,
                                       RTLIB::SUB_F80, RTLIB::SUB_F128,
                                       RTLIB::SUB_PPCF128));
     break;
+  case ISD::FSUB_W_CHAIN:
+    CallInfo = ExpandChainFPLibCall(Node, RTLIB::SUB_F32, RTLIB::SUB_F64,
+                                    RTLIB::SUB_F80, RTLIB::SUB_F128,
+                                    RTLIB::SUB_PPCF128);
+    Results.push_back(CallInfo.first);
+    Results.push_back(CallInfo.second);
+    break;
   case ISD::SREM:
     Results.push_back(ExpandIntLibCall(Node, true,
                                        RTLIB::SREM_I8,
                                        RTLIB::SREM_I16, RTLIB::SREM_I32,
                                        RTLIB::SREM_I64, RTLIB::SREM_I128));
     break;
   case ISD::UREM:
     Results.push_back(ExpandIntLibCall(Node, false,
case ISD::FPOW: {
     Tmp1 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(0));
     Tmp2 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(1));
     Tmp3 = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2,
                        Node->getFlags());
     Results.push_back(DAG.getNode(ISD::FP_ROUND, dl, OVT,
                                   Tmp3, DAG.getIntPtrConstant(0, dl)));
     break;
   }
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
+  case ISD::FMUL_W_CHAIN: {
+    Tmp1 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(1));
+    Tmp2 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(2));
+    Tmp3 = DAG.getNode(Node->getOpcode(), dl, DAG.getVTList(NVT, MVT::Other),
+                       Node->getOperand(0), Tmp1, Tmp2, Node->getFlags());
+    Results.push_back(DAG.getNode(ISD::FP_ROUND, dl, OVT,
+                                  Tmp3, DAG.getIntPtrConstant(0, dl)));
+    Results.push_back(Tmp3.getValue(1));
+    break;
+  }
   case ISD::FMA: {
     Tmp1 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(0));
     Tmp2 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(1));
     Tmp3 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(2));
     Results.push_back(
         DAG.getNode(ISD::FP_ROUND, dl, OVT,
                     DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2, Tmp3),
                     DAG.getIntPtrConstant(0, dl)));

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

#endif
     case ISD::BUILD_PAIR:  R = SoftenFloatRes_BUILD_PAIR(N); break;
     case ISD::ConstantFP:  R = SoftenFloatRes_ConstantFP(N, ResNo); break;
     case ISD::EXTRACT_VECTOR_ELT:
       R = SoftenFloatRes_EXTRACT_VECTOR_ELT(N); break;
     case ISD::FABS:        R = SoftenFloatRes_FABS(N, ResNo); break;
     case ISD::FMINNUM:     R = SoftenFloatRes_FMINNUM(N); break;
     case ISD::FMAXNUM:     R = SoftenFloatRes_FMAXNUM(N); break;
     case ISD::FADD:        R = SoftenFloatRes_FADD(N); break;
+    case ISD::FADD_W_CHAIN:
+                           R = SoftenFloatRes_FADD_W_CHAIN(N); break;
     case ISD::FCEIL:       R = SoftenFloatRes_FCEIL(N); break;
     case ISD::FCOPYSIGN:   R = SoftenFloatRes_FCOPYSIGN(N, ResNo); break;
     case ISD::FCOS:        R = SoftenFloatRes_FCOS(N); break;
     case ISD::FDIV:        R = SoftenFloatRes_FDIV(N); break;
+    case ISD::FDIV_W_CHAIN:
+                           R = SoftenFloatRes_FDIV_W_CHAIN(N); break;
     case ISD::FEXP:        R = SoftenFloatRes_FEXP(N); break;
     case ISD::FEXP2:       R = SoftenFloatRes_FEXP2(N); break;
     case ISD::FFLOOR:      R = SoftenFloatRes_FFLOOR(N); break;
     case ISD::FLOG:        R = SoftenFloatRes_FLOG(N); break;
     case ISD::FLOG2:       R = SoftenFloatRes_FLOG2(N); break;
     case ISD::FLOG10:      R = SoftenFloatRes_FLOG10(N); break;
     case ISD::FMA:         R = SoftenFloatRes_FMA(N); break;
     case ISD::FMUL:        R = SoftenFloatRes_FMUL(N); break;
+    case ISD::FMUL_W_CHAIN:
+                           R = SoftenFloatRes_FMUL_W_CHAIN(N); break;
     case ISD::FNEARBYINT:  R = SoftenFloatRes_FNEARBYINT(N); break;
     case ISD::FNEG:        R = SoftenFloatRes_FNEG(N, ResNo); break;
     case ISD::FP_EXTEND:   R = SoftenFloatRes_FP_EXTEND(N); break;
     case ISD::FP_ROUND:    R = SoftenFloatRes_FP_ROUND(N); break;
     case ISD::FP16_TO_FP:  R = SoftenFloatRes_FP16_TO_FP(N); break;
     case ISD::FPOW:        R = SoftenFloatRes_FPOW(N); break;
     case ISD::FPOWI:       R = SoftenFloatRes_FPOWI(N); break;
     case ISD::FREM:        R = SoftenFloatRes_FREM(N); break;
+    case ISD::FREM_W_CHAIN:
+                           R = SoftenFloatRes_FREM_W_CHAIN(N); break;
     case ISD::FRINT:       R = SoftenFloatRes_FRINT(N); break;
     case ISD::FROUND:      R = SoftenFloatRes_FROUND(N); break;
     case ISD::FSIN:        R = SoftenFloatRes_FSIN(N); break;
     case ISD::FSQRT:       R = SoftenFloatRes_FSQRT(N); break;
     case ISD::FSUB:        R = SoftenFloatRes_FSUB(N); break;
+    case ISD::FSUB_W_CHAIN:
+                           R = SoftenFloatRes_FSUB_W_CHAIN(N); break;
     case ISD::FTRUNC:      R = SoftenFloatRes_FTRUNC(N); break;
     case ISD::LOAD:        R = SoftenFloatRes_LOAD(N, ResNo); break;
     case ISD::SELECT:      R = SoftenFloatRes_SELECT(N, ResNo); break;
     case ISD::SELECT_CC:   R = SoftenFloatRes_SELECT_CC(N, ResNo); break;
     case ISD::SINT_TO_FP:
     case ISD::UINT_TO_FP:  R = SoftenFloatRes_XINT_TO_FP(N); break;
     case ISD::UNDEF:       R = SoftenFloatRes_UNDEF(N); break;
     case ISD::VAARG:       R = SoftenFloatRes_VAARG(N); break;
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::ADD_F32,
                                            RTLIB::ADD_F64,
                                            RTLIB::ADD_F80,
                                            RTLIB::ADD_F128,
                                            RTLIB::ADD_PPCF128),
                          NVT, Ops, false, SDLoc(N)).first;
 }
 
+SDValue DAGTypeLegalizer::SoftenFloatRes_FADD_W_CHAIN(SDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Ops[3] = { N->getOperand(0),
+                     GetSoftenedFloat(N->getOperand(1)),
+                     GetSoftenedFloat(N->getOperand(2)) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::ADD_F32,
+                                                        RTLIB::ADD_F64,
+                                                        RTLIB::ADD_F80,
+                                                        RTLIB::ADD_F128,
+                                                        RTLIB::ADD_PPCF128),
+                                      NVT, Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  return Val;
+}
+
 SDValue DAGTypeLegalizer::SoftenFloatRes_FCEIL(SDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   SDValue Op = GetSoftenedFloat(N->getOperand(0));
   return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::CEIL_F32,
                                            RTLIB::CEIL_F64,
                                            RTLIB::CEIL_F80,
                                            RTLIB::CEIL_F128,
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::DIV_F32,
                                            RTLIB::DIV_F64,
                                            RTLIB::DIV_F80,
                                            RTLIB::DIV_F128,
                                            RTLIB::DIV_PPCF128),
                          NVT, Ops, false, SDLoc(N)).first;
 }
 
+SDValue DAGTypeLegalizer::SoftenFloatRes_FDIV_W_CHAIN(SDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Ops[3] = { N->getOperand(0),
+                     GetSoftenedFloat(N->getOperand(1)),
+                     GetSoftenedFloat(N->getOperand(2)) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::DIV_F32,
+                                                        RTLIB::DIV_F64,
+                                                        RTLIB::DIV_F80,
+                                                        RTLIB::DIV_F128,
+                                                        RTLIB::DIV_PPCF128),
+                                      NVT, Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  return Val;
+}
+
 SDValue DAGTypeLegalizer::SoftenFloatRes_FEXP(SDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   SDValue Op = GetSoftenedFloat(N->getOperand(0));
   return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::EXP_F32,
                                            RTLIB::EXP_F64,
                                            RTLIB::EXP_F80,
                                            RTLIB::EXP_F128,
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::MUL_F32,
                                            RTLIB::MUL_F64,
                                            RTLIB::MUL_F80,
                                            RTLIB::MUL_F128,
                                            RTLIB::MUL_PPCF128),
                          NVT, Ops, false, SDLoc(N)).first;
 }
 
+SDValue DAGTypeLegalizer::SoftenFloatRes_FMUL_W_CHAIN(SDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Ops[3] = { N->getOperand(0),
+                     GetSoftenedFloat(N->getOperand(1)),
+                     GetSoftenedFloat(N->getOperand(2)) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::MUL_F32,
+                                                        RTLIB::MUL_F64,
+                                                        RTLIB::MUL_F80,
+                                                        RTLIB::MUL_F128,
+                                                        RTLIB::MUL_PPCF128),
+                                      NVT, Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  return Val;
+}
+
 SDValue DAGTypeLegalizer::SoftenFloatRes_FNEARBYINT(SDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   SDValue Op = GetSoftenedFloat(N->getOperand(0));
   return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::NEARBYINT_F32,
                                            RTLIB::NEARBYINT_F64,
                                            RTLIB::NEARBYINT_F80,
                                            RTLIB::NEARBYINT_F128,
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::REM_F32,
                                            RTLIB::REM_F64,
                                            RTLIB::REM_F80,
                                            RTLIB::REM_F128,
                                            RTLIB::REM_PPCF128),
                          NVT, Ops, false, SDLoc(N)).first;
 }
 
+SDValue DAGTypeLegalizer::SoftenFloatRes_FREM_W_CHAIN(SDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Ops[3] = { N->getOperand(0),
+                     GetSoftenedFloat(N->getOperand(1)),
+                     GetSoftenedFloat(N->getOperand(2)) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::REM_F32,
+                                                        RTLIB::REM_F64,
+                                                        RTLIB::REM_F80,
+                                                        RTLIB::REM_F128,
+                                                        RTLIB::REM_PPCF128),
+                                      NVT, Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  return Val;
+}
+
 SDValue DAGTypeLegalizer::SoftenFloatRes_FRINT(SDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   SDValue Op = GetSoftenedFloat(N->getOperand(0));
   return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::RINT_F32,
                                            RTLIB::RINT_F64,
                                            RTLIB::RINT_F80,
                                            RTLIB::RINT_F128,
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::SUB_F32,
                                            RTLIB::SUB_F64,
                                            RTLIB::SUB_F80,
                                            RTLIB::SUB_F128,
                                            RTLIB::SUB_PPCF128),
                          NVT, Ops, false, SDLoc(N)).first;
 }
 
+SDValue DAGTypeLegalizer::SoftenFloatRes_FSUB_W_CHAIN(SDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Ops[3] = { N->getOperand(0),
+                     GetSoftenedFloat(N->getOperand(1)),
+                     GetSoftenedFloat(N->getOperand(2)) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::SUB_F32,
+                                                        RTLIB::SUB_F64,
+                                                        RTLIB::SUB_F80,
+                                                        RTLIB::SUB_F128,
+                                                        RTLIB::SUB_PPCF128),
+                                      NVT, Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  return Val;
+}
+
 SDValue DAGTypeLegalizer::SoftenFloatRes_FTRUNC(SDNode *N) {
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   if (N->getValueType(0) == MVT::f16)
     return DAG.getNode(ISD::FP_TO_FP16, SDLoc(N), NVT, N->getOperand(0));
 
   SDValue Op = GetSoftenedFloat(N->getOperand(0));
   return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::TRUNC_F32,
#endif
   case ISD::EXTRACT_ELEMENT:    ExpandRes_EXTRACT_ELEMENT(N, Lo, Hi); break;
   case ISD::EXTRACT_VECTOR_ELT: ExpandRes_EXTRACT_VECTOR_ELT(N, Lo, Hi); break;
   case ISD::VAARG:              ExpandRes_VAARG(N, Lo, Hi); break;
 
   case ISD::ConstantFP: ExpandFloatRes_ConstantFP(N, Lo, Hi); break;
   case ISD::FABS:       ExpandFloatRes_FABS(N, Lo, Hi); break;
   case ISD::FMINNUM:    ExpandFloatRes_FMINNUM(N, Lo, Hi); break;
   case ISD::FMAXNUM:    ExpandFloatRes_FMAXNUM(N, Lo, Hi); break;
+  case ISD::FADD_W_CHAIN:
   case ISD::FADD:       ExpandFloatRes_FADD(N, Lo, Hi); break;
   case ISD::FCEIL:      ExpandFloatRes_FCEIL(N, Lo, Hi); break;
   case ISD::FCOPYSIGN:  ExpandFloatRes_FCOPYSIGN(N, Lo, Hi); break;
   case ISD::FCOS:       ExpandFloatRes_FCOS(N, Lo, Hi); break;
   case ISD::FDIV:       ExpandFloatRes_FDIV(N, Lo, Hi); break;
+  case ISD::FDIV_W_CHAIN:
+                        ExpandFloatRes_FDIV_W_CHAIN(N, Lo, Hi); break;
   case ISD::FEXP:       ExpandFloatRes_FEXP(N, Lo, Hi); break;
   case ISD::FEXP2:      ExpandFloatRes_FEXP2(N, Lo, Hi); break;
   case ISD::FFLOOR:     ExpandFloatRes_FFLOOR(N, Lo, Hi); break;
   case ISD::FLOG:       ExpandFloatRes_FLOG(N, Lo, Hi); break;
   case ISD::FLOG2:      ExpandFloatRes_FLOG2(N, Lo, Hi); break;
   case ISD::FLOG10:     ExpandFloatRes_FLOG10(N, Lo, Hi); break;
   case ISD::FMA:        ExpandFloatRes_FMA(N, Lo, Hi); break;
   case ISD::FMUL:       ExpandFloatRes_FMUL(N, Lo, Hi); break;
+  case ISD::FMUL_W_CHAIN:
+                        ExpandFloatRes_FMUL_W_CHAIN(N, Lo, Hi); break;
   case ISD::FNEARBYINT: ExpandFloatRes_FNEARBYINT(N, Lo, Hi); break;
   case ISD::FNEG:       ExpandFloatRes_FNEG(N, Lo, Hi); break;
   case ISD::FP_EXTEND:  ExpandFloatRes_FP_EXTEND(N, Lo, Hi); break;
   case ISD::FPOW:       ExpandFloatRes_FPOW(N, Lo, Hi); break;
   case ISD::FPOWI:      ExpandFloatRes_FPOWI(N, Lo, Hi); break;
   case ISD::FRINT:      ExpandFloatRes_FRINT(N, Lo, Hi); break;
   case ISD::FROUND:     ExpandFloatRes_FROUND(N, Lo, Hi); break;
   case ISD::FSIN:       ExpandFloatRes_FSIN(N, Lo, Hi); break;
   case ISD::FSQRT:      ExpandFloatRes_FSQRT(N, Lo, Hi); break;
   case ISD::FSUB:       ExpandFloatRes_FSUB(N, Lo, Hi); break;
+  case ISD::FSUB_W_CHAIN:
+                        ExpandFloatRes_FSUB_W_CHAIN(N, Lo, Hi); break;
   case ISD::FTRUNC:     ExpandFloatRes_FTRUNC(N, Lo, Hi); break;
   case ISD::LOAD:       ExpandFloatRes_LOAD(N, Lo, Hi); break;
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP: ExpandFloatRes_XINT_TO_FP(N, Lo, Hi); break;
+  case ISD::FREM_W_CHAIN:
   case ISD::FREM:       ExpandFloatRes_FREM(N, Lo, Hi); break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
   if (Lo.getNode())
     SetExpandedFloat(SDValue(N, ResNo), Lo, Hi);
 }
 
SDValue Call = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                                    RTLIB::DIV_F80,
                                                    RTLIB::DIV_F128,
                                                    RTLIB::DIV_PPCF128),
                                  N->getValueType(0), Ops, false,
                                  SDLoc(N)).first;
   GetPairElements(Call, Lo, Hi);
 }
 
+void DAGTypeLegalizer::ExpandFloatRes_FDIV_W_CHAIN(SDNode *N, SDValue &Lo,
+                                                   SDValue &Hi) {
+  SDValue Ops[3] = { N->getOperand(0), N->getOperand(1), N->getOperand(2) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::DIV_F32,
+                                                        RTLIB::DIV_F64,
+                                                        RTLIB::DIV_F80,
+                                                        RTLIB::DIV_F128,
+                                                        RTLIB::DIV_PPCF128),
+                                      N->getValueType(0), Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  GetPairElements(Val, Lo, Hi);
+}
+
 void DAGTypeLegalizer::ExpandFloatRes_FEXP(SDNode *N,
                                            SDValue &Lo, SDValue &Hi) {
   SDValue Call = LibCallify(GetFPLibCall(N->getValueType(0),
                                          RTLIB::EXP_F32, RTLIB::EXP_F64,
                                          RTLIB::EXP_F80, RTLIB::EXP_F128,
                                          RTLIB::EXP_PPCF128),
                             N, false);
   GetPairElements(Call, Lo, Hi);
SDValue Call = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                                    RTLIB::MUL_F80,
                                                    RTLIB::MUL_F128,
                                                    RTLIB::MUL_PPCF128),
                                  N->getValueType(0), Ops, false,
                                  SDLoc(N)).first;
   GetPairElements(Call, Lo, Hi);
 }
 
+void DAGTypeLegalizer::ExpandFloatRes_FMUL_W_CHAIN(SDNode *N, SDValue &Lo,
+                                                   SDValue &Hi) {
+  SDValue Ops[3] = { N->getOperand(0), N->getOperand(1), N->getOperand(2) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::MUL_F32,
+                                                        RTLIB::MUL_F64,
+                                                        RTLIB::MUL_F80,
+                                                        RTLIB::MUL_F128,
+                                                        RTLIB::MUL_PPCF128),
+                                      N->getValueType(0), Ops, false, SDLoc(N));
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  GetPairElements(Val, Lo, Hi);
+}
+
 void DAGTypeLegalizer::ExpandFloatRes_FNEARBYINT(SDNode *N,
                                                  SDValue &Lo, SDValue &Hi) {
   SDValue Call = LibCallify(GetFPLibCall(N->getValueType(0),
                                          RTLIB::NEARBYINT_F32,
                                          RTLIB::NEARBYINT_F64,
                                          RTLIB::NEARBYINT_F80,
                                          RTLIB::NEARBYINT_F128,
                                          RTLIB::NEARBYINT_PPCF128),
SDValue Call = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                                    RTLIB::SUB_F80,
                                                    RTLIB::SUB_F128,
                                                    RTLIB::SUB_PPCF128),
                                  N->getValueType(0), Ops, false,
                                  SDLoc(N)).first;
   GetPairElements(Call, Lo, Hi);
 }
 
+void DAGTypeLegalizer::ExpandFloatRes_FSUB_W_CHAIN(SDNode *N, SDValue &Lo,
+                                                   SDValue &Hi) {
+  SDValue Ops[3] = { N->getOperand(0), N->getOperand(1), N->getOperand(2) };
+  SDValue Val, Ch;
+  std::tie(Val, Ch) = TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
+                                                        RTLIB::SUB_F32,
+                                                        RTLIB::SUB_F64,
+                                                        RTLIB::SUB_F80,
+                                                        RTLIB::SUB_F128,
+                                                        RTLIB::SUB_PPCF128),
+                                      N->getValueType(0), Ops, false, SDLoc(N));
+  ReplaceValueWith(SDValue(N, 1), Ch);
+  GetPairElements(Val, Lo, Hi);
+}
+
 void DAGTypeLegalizer::ExpandFloatRes_FTRUNC(SDNode *N,
                                              SDValue &Lo, SDValue &Hi) {
   SDValue Call = LibCallify(GetFPLibCall(N->getValueType(0),
                                          RTLIB::TRUNC_F32, RTLIB::TRUNC_F64,
                                          RTLIB::TRUNC_F80, RTLIB::TRUNC_F128,
                                          RTLIB::TRUNC_PPCF128),
                             N, false);
   GetPairElements(Call, Lo, Hi);
switch (N->getOpcode()) {
     case ISD::FDIV:
     case ISD::FMAXNUM:
     case ISD::FMINNUM:
     case ISD::FMUL:
     case ISD::FPOW:
     case ISD::FREM:
     case ISD::FSUB:       R = PromoteFloatRes_BinOp(N); break;
 
+    case ISD::FADD_W_CHAIN:
+    case ISD::FDIV_W_CHAIN:
+    case ISD::FMUL_W_CHAIN:
+    case ISD::FREM_W_CHAIN:
+    case ISD::FSUB_W_CHAIN:
+                          R = PromoteFloatRes_BinOpWithChain(N); break;
+
     case ISD::FMA:        // FMA is same as FMAD
     case ISD::FMAD:       R = PromoteFloatRes_FMAD(N); break;
 
     case ISD::FPOWI:      R = PromoteFloatRes_FPOWI(N); break;
 
     case ISD::FP_ROUND:   R = PromoteFloatRes_FP_ROUND(N); break;
     case ISD::LOAD:       R = PromoteFloatRes_LOAD(N); break;
     case ISD::SELECT:     R = PromoteFloatRes_SELECT(N); break;

 SDValue DAGTypeLegalizer::PromoteFloatRes_BinOp(SDNode *N) {
   EVT VT = N->getValueType(0);
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
   SDValue Op0 = GetPromotedFloat(N->getOperand(0));
   SDValue Op1 = GetPromotedFloat(N->getOperand(1));
   return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op0, Op1, N->getFlags());
 }
 
+// Binary operations with a chain where the result and both operands have
+// PromoteFloat type action.  Construct a new SDNode with the promoted float
+// values of the old operands.
+SDValue DAGTypeLegalizer::PromoteFloatRes_BinOpWithChain(SDNode *N) {
+  EVT VT = N->getValueType(0);
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
+  SDValue Chain = N->getOperand(0);
+  SDValue Op0 = GetPromotedFloat(N->getOperand(1));
+  SDValue Op1 = GetPromotedFloat(N->getOperand(2));
+
+  SDValue R = DAG.getNode(N->getOpcode(), SDLoc(N),
+                          DAG.getVTList(NVT, MVT::Other), Chain, Op0, Op1,
+                          N->getFlags());
+
+  // Modified the chain - switch anything that used the old chain to use the
+  // new one.
+  ReplaceValueWith(SDValue(N, 1), R.getValue(1));
+
+  return R.getValue(0);
+}
+
 SDValue DAGTypeLegalizer::PromoteFloatRes_FMAD(SDNode *N) {
   EVT VT = N->getValueType(0);
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
   SDValue Op0 = GetPromotedFloat(N->getOperand(0));
   SDValue Op1 = GetPromotedFloat(N->getOperand(1));
   SDValue Op2 = GetPromotedFloat(N->getOperand(2));
 
   return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op0, Op1, Op2);

lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue SoftenFloatRes_BITCAST(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_BUILD_PAIR(SDNode *N);
   SDValue SoftenFloatRes_ConstantFP(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue SoftenFloatRes_FABS(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_FMINNUM(SDNode *N);
   SDValue SoftenFloatRes_FMAXNUM(SDNode *N);
   SDValue SoftenFloatRes_FADD(SDNode *N);
+  SDValue SoftenFloatRes_FADD_W_CHAIN(SDNode *N);
   SDValue SoftenFloatRes_FCEIL(SDNode *N);
   SDValue SoftenFloatRes_FCOPYSIGN(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_FCOS(SDNode *N);
   SDValue SoftenFloatRes_FDIV(SDNode *N);
+  SDValue SoftenFloatRes_FDIV_W_CHAIN(SDNode *N);
   SDValue SoftenFloatRes_FEXP(SDNode *N);
   SDValue SoftenFloatRes_FEXP2(SDNode *N);
   SDValue SoftenFloatRes_FFLOOR(SDNode *N);
   SDValue SoftenFloatRes_FLOG(SDNode *N);
   SDValue SoftenFloatRes_FLOG2(SDNode *N);
   SDValue SoftenFloatRes_FLOG10(SDNode *N);
   SDValue SoftenFloatRes_FMA(SDNode *N);
   SDValue SoftenFloatRes_FMUL(SDNode *N);
+  SDValue SoftenFloatRes_FMUL_W_CHAIN(SDNode *N);
   SDValue SoftenFloatRes_FNEARBYINT(SDNode *N);
   SDValue SoftenFloatRes_FNEG(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_FP_EXTEND(SDNode *N);
   SDValue SoftenFloatRes_FP16_TO_FP(SDNode *N);
   SDValue SoftenFloatRes_FP_ROUND(SDNode *N);
   SDValue SoftenFloatRes_FPOW(SDNode *N);
   SDValue SoftenFloatRes_FPOWI(SDNode *N);
   SDValue SoftenFloatRes_FREM(SDNode *N);
+  SDValue SoftenFloatRes_FREM_W_CHAIN(SDNode *N);
   SDValue SoftenFloatRes_FRINT(SDNode *N);
   SDValue SoftenFloatRes_FROUND(SDNode *N);
   SDValue SoftenFloatRes_FSIN(SDNode *N);
   SDValue SoftenFloatRes_FSQRT(SDNode *N);
   SDValue SoftenFloatRes_FSUB(SDNode *N);
+  SDValue SoftenFloatRes_FSUB_W_CHAIN(SDNode *N);
   SDValue SoftenFloatRes_FTRUNC(SDNode *N);
   SDValue SoftenFloatRes_LOAD(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_SELECT(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_SELECT_CC(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_UNDEF(SDNode *N);
   SDValue SoftenFloatRes_VAARG(SDNode *N);
   SDValue SoftenFloatRes_XINT_TO_FP(SDNode *N);
 
private:
   /// exactly equal Hi.  For example, if Op is a ppcf128 which was expanded
   /// into two f64's, then this method returns the two f64's, with Lo being
   /// equal to the lower 64 bits of Op, and Hi to the upper 64 bits.
   void GetExpandedFloat(SDValue Op, SDValue &Lo, SDValue &Hi);
   void SetExpandedFloat(SDValue Op, SDValue Lo, SDValue Hi);
 
   // Float Result Expansion.
   void ExpandFloatResult(SDNode *N, unsigned ResNo);
-  void ExpandFloatRes_ConstantFP(SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FABS      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FMINNUM   (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FMAXNUM   (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FADD      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FCEIL     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FCOPYSIGN (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FCOS      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FDIV      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FEXP      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FEXP2     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FFLOOR    (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FLOG      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FLOG2     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FLOG10    (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FMA       (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FMUL      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FNEARBYINT(SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FNEG      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FP_EXTEND (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FPOW      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FPOWI     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FREM      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FRINT     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FROUND    (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FSIN      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FSQRT     (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FSUB      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_FTRUNC    (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_LOAD      (SDNode *N, SDValue &Lo, SDValue &Hi);
-  void ExpandFloatRes_XINT_TO_FP(SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_ConstantFP  (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FABS        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FMINNUM     (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FMAXNUM     (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FADD        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FCEIL       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FCOPYSIGN   (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FCOS        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FDIV        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FDIV_W_CHAIN(SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FEXP        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FEXP2       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FFLOOR      (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FLOG        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FLOG2       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FLOG10      (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FMA         (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FMUL        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FMUL_W_CHAIN(SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FNEARBYINT  (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FNEG        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FP_EXTEND   (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FPOW        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FPOWI       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FREM        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FRINT       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FROUND      (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FSIN        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FSQRT       (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FSUB        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FSUB_W_CHAIN(SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_FTRUNC      (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_LOAD        (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandFloatRes_XINT_TO_FP  (SDNode *N, SDValue &Lo, SDValue &Hi);
 
   // Float Operand Expansion.
   bool ExpandFloatOperand(SDNode *N, unsigned OperandNo);
   SDValue ExpandFloatOp_BR_CC(SDNode *N);
   SDValue ExpandFloatOp_FCOPYSIGN(SDNode *N);
   SDValue ExpandFloatOp_FP_ROUND(SDNode *N);
   SDValue ExpandFloatOp_FP_TO_SINT(SDNode *N);
   SDValue ExpandFloatOp_FP_TO_UINT(SDNode *N);
SDValue GetPromotedFloat(SDValue Op) {
     assert(PromotedOp.getNode() && "Operand wasn't promoted?");
     return PromotedOp;
   }
   void SetPromotedFloat(SDValue Op, SDValue Result);
 
   void PromoteFloatResult(SDNode *N, unsigned ResNo);
   SDValue PromoteFloatRes_BITCAST(SDNode *N);
   SDValue PromoteFloatRes_BinOp(SDNode *N);
+  SDValue PromoteFloatRes_BinOpWithChain(SDNode *N);
   SDValue PromoteFloatRes_ConstantFP(SDNode *N);
   SDValue PromoteFloatRes_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue PromoteFloatRes_FCOPYSIGN(SDNode *N);
   SDValue PromoteFloatRes_FMAD(SDNode *N);
   SDValue PromoteFloatRes_FPOWI(SDNode *N);
   SDValue PromoteFloatRes_FP_ROUND(SDNode *N);
   SDValue PromoteFloatRes_LOAD(SDNode *N);
   SDValue PromoteFloatRes_SELECT(SDNode *N);
SDValue GetScalarizedVector(SDValue Op) {
     return ScalarizedOp;
   }
   void SetScalarizedVector(SDValue Op, SDValue Result);
 
   // Vector Result Scalarization: <1 x ty> -> ty.
   void ScalarizeVectorResult(SDNode *N, unsigned OpNo);
   SDValue ScalarizeVecRes_MERGE_VALUES(SDNode *N, unsigned ResNo);
   SDValue ScalarizeVecRes_BinOp(SDNode *N);
+  SDValue ScalarizeVecRes_BinOpWithChain(SDNode *N);
   SDValue ScalarizeVecRes_TernaryOp(SDNode *N);
   SDValue ScalarizeVecRes_UnaryOp(SDNode *N);
   SDValue ScalarizeVecRes_InregOp(SDNode *N);
 
   SDValue ScalarizeVecRes_BITCAST(SDNode *N);
   SDValue ScalarizeVecRes_BUILD_VECTOR(SDNode *N);
   SDValue ScalarizeVecRes_CONVERT_RNDSAT(SDNode *N);
   SDValue ScalarizeVecRes_EXTRACT_SUBVECTOR(SDNode *N);
private:
   /// this method returns the two v4i32's, with Lo corresponding to the first 4
   /// elements of Op, and Hi to the last 4 elements.
   void GetSplitVector(SDValue Op, SDValue &Lo, SDValue &Hi);
   void SetSplitVector(SDValue Op, SDValue Lo, SDValue Hi);
 
   // Vector Result Splitting: <128 x ty> -> 2 x <64 x ty>.
   void SplitVectorResult(SDNode *N, unsigned OpNo);
   void SplitVecRes_BinOp(SDNode *N, SDValue &Lo, SDValue &Hi);
+  void SplitVecRes_BinOpWithChain(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_TernaryOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_UnaryOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_ExtendOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_InregOp(SDNode *N, SDValue &Lo, SDValue &Hi);
 
   void SplitVecRes_BITCAST(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_BUILD_VECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_CONCAT_VECTORS(SDNode *N, SDValue &Lo, SDValue &Hi);

lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

SDValue DAGTypeLegalizer::JoinIntegers(SDValue Lo, SDValue Hi) {
   return DAG.getNode(ISD::OR, dlHi, NVT, Lo, Hi);
 }
 
 /// LibCallify - Convert the node into a libcall with the same prototype.
 SDValue DAGTypeLegalizer::LibCallify(RTLIB::Libcall LC, SDNode *N,
                                      bool isSigned) {
   unsigned NumOps = N->getNumOperands();
   SDLoc dl(N);
+
   if (NumOps == 0) {
     return TLI.makeLibCall(DAG, LC, N->getValueType(0), None, isSigned,
                            dl).first;
   } else if (NumOps == 1) {
     SDValue Op = N->getOperand(0);
-    return TLI.makeLibCall(DAG, LC, N->getValueType(0), Op, isSigned,
-                           dl).first;
+    return TLI.makeLibCall(DAG, LC, N->getValueType(0), Op, isSigned, dl).first;
   } else if (NumOps == 2) {
     SDValue Ops[2] = { N->getOperand(0), N->getOperand(1) };
     return TLI.makeLibCall(DAG, LC, N->getValueType(0), Ops, isSigned,
                            dl).first;
   }
-  SmallVector<SDValue, 8> Ops(NumOps);
-  for (unsigned i = 0; i < NumOps; ++i)
-    Ops[i] = N->getOperand(i);
 
-  return TLI.makeLibCall(DAG, LC, N->getValueType(0), Ops, isSigned, dl).first;
+  bool HasChain = isFPOpWithChain(N->getOpcode());
+
+  SmallVector<SDValue, 8> Ops(HasChain ? NumOps - 1 : NumOps);
+  for (unsigned i = HasChain ? 1 : 0; i < NumOps; ++i)
+    Ops[HasChain ? i - 1 : i] = N->getOperand(i);
+
+  std::pair<SDValue, SDValue> CallInfo =
+    TLI.makeLibCall(DAG, LC, N->getValueType(0), Ops, isSigned, dl);
+
+  if (HasChain)
+    ReplaceValueWith(SDValue(N, 1), CallInfo.second);
+
+  return CallInfo.first;
 }
 
 // ExpandChainLibCall - Expand a node into a call to a libcall. Similar to
 // ExpandLibCall except that the first operand is the in-chain.
 std::pair<SDValue, SDValue>
 DAGTypeLegalizer::ExpandChainLibCall(RTLIB::Libcall LC,
                                          SDNode *Node,
                                          bool isSigned) {

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(),
     if (HasVectors)
       break;
   }
 
   // If this basic block has no vectors then no need to legalize vectors.
   if (!HasVectors)
     return false;
 
+  SDValue OldRoot = DAG.getRoot();
+
   // The legalize process is inherently a bottom-up recursive process (users
   // legalize their uses before themselves).  Given infinite stack space, we
   // could just start legalizing on the root and traverse the whole graph.  In
   // practice however, this causes us to run out of stack space on large basic
   // blocks.  To avoid this problem, compute an ordering of the nodes where each
   // node is only legalized after all of its operands are legalized.
   DAG.AssignTopologicalOrder();
   for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(),
-       E = std::prev(DAG.allnodes_end()); I != std::next(E); ++I)
-    LegalizeOp(SDValue(&*I, 0));
+       E = std::prev(DAG.allnodes_end()); I != std::next(E); ++I) {
+    // Preserve result number on the root node so that old and new nodes could
+    // be compared correctly (same node, but different result number will make
+    // comparison fail).
+    unsigned ResNo = ((SDNode*)I == OldRoot.getNode() ? OldRoot.getResNo() : 0);
+    LegalizeOp(SDValue(&*I, ResNo));
+  }
 
   // Finally, it's possible the root changed.  Get the new root.
-  SDValue OldRoot = DAG.getRoot();
   assert(LegalizedNodes.count(OldRoot) && "Root didn't get legalized?");
   DAG.setRoot(LegalizedNodes[OldRoot]);
 
   LegalizedNodes.clear();
 
   // Remove dead nodes now.
   DAG.RemoveDeadNodes();
 
SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   // means that we always must cache transformed nodes.
   DenseMap<SDValue, SDValue>::iterator I = LegalizedNodes.find(Op);
   if (I != LegalizedNodes.end()) return I->second;
 
   SDNode* Node = Op.getNode();
 
   // Legalize the operands
   SmallVector<SDValue, 8> Ops;
-  for (const SDValue &Op : Node->op_values())
+  for (const SDValue &Op : Node->op_values()) {
+    if (isFPOpWithChain(Op.getOpcode()) && Op.getValueType() == MVT::Other)
+      // Do not attempt to legalize chain.
+      Ops.push_back(Op);
+    else
       Ops.push_back(LegalizeOp(Op));
+  }
 
-  SDValue Result = SDValue(DAG.UpdateNodeOperands(Op.getNode(), Ops), 0);
+  SDValue Result = SDValue(DAG.UpdateNodeOperands(Op.getNode(), Ops),
+                           Op.getResNo());
 
   bool HasVectorValue = false;
   if (Op.getOpcode() == ISD::LOAD) {
     LoadSDNode *LD = cast<LoadSDNode>(Op.getNode());
     ISD::LoadExtType ExtType = LD->getExtensionType();
     if (LD->getMemoryVT().isVector() && ExtType != ISD::NON_EXTLOAD)
       switch (TLI.getLoadExtAction(LD->getExtensionType(), LD->getValueType(0),
                                    LD->getMemoryVT())) {
SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   case ISD::SIGN_EXTEND_VECTOR_INREG:
   case ISD::ZERO_EXTEND_VECTOR_INREG:
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
   case ISD::UABSDIFF:
   case ISD::SABSDIFF:
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
     QueryType = Node->getValueType(0);
     break;
   case ISD::FP_ROUND_INREG:
     QueryType = cast<VTSDNode>(Node->getOperand(1))->getVT();
     break;
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
     QueryType = Node->getOperand(0).getValueType();
SDValue VectorLegalizer::Promote(SDValue Op) {
   }
 
   // There are currently two cases of vector promotion:
   // 1) Bitcasting a vector of integers to a different type to a vector of the
   //    same overall length. For example, x86 promotes ISD::AND v2i32 to v1i64.
   // 2) Extending a vector of floats to a vector of the same number of larger
   //    floats. For example, AArch64 promotes ISD::FADD on v4f16 to v4f32.
   MVT VT = Op.getSimpleValueType();
-  assert(Op.getNode()->getNumValues() == 1 &&
+  assert(((isFPOpWithChain(Op.getOpcode()) &&
+           Op.getNode()->getNumValues() == 2) ||
+          Op.getNode()->getNumValues() == 1) &&
          "Can't promote a vector with multiple results!");
   MVT NVT = TLI.getTypeToPromoteTo(Op.getOpcode(), VT);
   SDLoc dl(Op);
   SmallVector<SDValue, 4> Operands(Op.getNumOperands());
 
   for (unsigned j = 0; j != Op.getNumOperands(); ++j) {
     if (Op.getOperand(j).getValueType().isVector())
       if (Op.getOperand(j)

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::FADD:
   case ISD::FCOPYSIGN:
   case ISD::FDIV:
   case ISD::FMUL:
   case ISD::FMINNUM:
   case ISD::FMAXNUM:
   case ISD::FMINNAN:
   case ISD::FMAXNAN:
-
   case ISD::FPOW:
   case ISD::FREM:
   case ISD::FSUB:
   case ISD::MUL:
   case ISD::OR:
   case ISD::SDIV:
   case ISD::SREM:
   case ISD::SUB:
   case ISD::UDIV:
   case ISD::UREM:
   case ISD::XOR:
   case ISD::SHL:
   case ISD::SRA:
   case ISD::SRL:
     R = ScalarizeVecRes_BinOp(N);
     break;
+  case ISD::FADD_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+    R = ScalarizeVecRes_BinOpWithChain(N);
+    break;
   case ISD::FMA:
     R = ScalarizeVecRes_TernaryOp(N);
     break;
   }
 
   // If R is null, the sub-method took care of registering the result.
   if (R.getNode())
     SetScalarizedVector(SDValue(N, ResNo), R);
 }
 
 SDValue DAGTypeLegalizer::ScalarizeVecRes_BinOp(SDNode *N) {
   SDValue LHS = GetScalarizedVector(N->getOperand(0));
   SDValue RHS = GetScalarizedVector(N->getOperand(1));
   return DAG.getNode(N->getOpcode(), SDLoc(N),
                      LHS.getValueType(), LHS, RHS, N->getFlags());
 }
 
+SDValue DAGTypeLegalizer::ScalarizeVecRes_BinOpWithChain(SDNode *N) {
+  SDValue LHS = GetScalarizedVector(N->getOperand(1));
+  SDValue RHS = GetScalarizedVector(N->getOperand(2));
+  SDValue Ch = DAG.getNode(N->getOpcode(), SDLoc(N),
+                           DAG.getVTList(LHS.getValueType(), MVT::Other),
+                           N->getOperand(0), LHS, RHS, N->getFlags());
+
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch.getValue(1));
+
+  return Ch.getValue(0);
+}
+
 SDValue DAGTypeLegalizer::ScalarizeVecRes_TernaryOp(SDNode *N) {
   SDValue Op0 = GetScalarizedVector(N->getOperand(0));
   SDValue Op1 = GetScalarizedVector(N->getOperand(1));
   SDValue Op2 = GetScalarizedVector(N->getOperand(2));
   return DAG.getNode(N->getOpcode(), SDLoc(N),
                      Op0.getValueType(), Op0, Op1, Op2);
 }
 
#endif
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
   case ISD::UABSDIFF:
   case ISD::SABSDIFF:
     SplitVecRes_BinOp(N, Lo, Hi);
     break;
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
+    SplitVecRes_BinOpWithChain(N, Lo, Hi);
+    break;
   case ISD::FMA:
     SplitVecRes_TernaryOp(N, Lo, Hi);
     break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
   if (Lo.getNode())
     SetSplitVector(SDValue(N, ResNo), Lo, Hi);
 }
 
 void DAGTypeLegalizer::SplitVecRes_BinOp(SDNode *N, SDValue &Lo,
                                          SDValue &Hi) {
   SDValue LHSLo, LHSHi;
   GetSplitVector(N->getOperand(0), LHSLo, LHSHi);
   SDValue RHSLo, RHSHi;
   GetSplitVector(N->getOperand(1), RHSLo, RHSHi);
   SDLoc dl(N);
 
   const SDNodeFlags *Flags = N->getFlags();
   unsigned Opcode = N->getOpcode();
   Lo = DAG.getNode(Opcode, dl, LHSLo.getValueType(), LHSLo, RHSLo, Flags);
   Hi = DAG.getNode(Opcode, dl, LHSHi.getValueType(), LHSHi, RHSHi, Flags);
 }
 
+void DAGTypeLegalizer::SplitVecRes_BinOpWithChain(SDNode *N, SDValue &Lo,
+                                                  SDValue &Hi) {
+  SDValue LHSLo, LHSHi;
+  GetSplitVector(N->getOperand(1), LHSLo, LHSHi);
+  SDValue RHSLo, RHSHi;
+  GetSplitVector(N->getOperand(2), RHSLo, RHSHi);
+  SDLoc dl(N);
+
+  const SDNodeFlags *Flags = N->getFlags();
+  SDValue Ch = N->getOperand(0);
+
+  Lo = DAG.getNode(N->getOpcode(), dl,
+                   DAG.getVTList(LHSLo.getValueType(), MVT::Other),
+                   Ch, LHSLo, RHSLo, Flags);
+  Hi = DAG.getNode(N->getOpcode(), dl,
+                   DAG.getVTList(LHSHi.getValueType(), MVT::Other),
+                   Ch, LHSHi, RHSHi, Flags);
+
+  Ch = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1),
+                   Hi.getValue(1));
+
+  Lo = Lo.getValue(0);
+  Hi = Hi.getValue(0);
+
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Ch);
+}
+
 void DAGTypeLegalizer::SplitVecRes_TernaryOp(SDNode *N, SDValue &Lo,
                                              SDValue &Hi) {
   SDValue Op0Lo, Op0Hi;
   GetSplitVector(N->getOperand(0), Op0Lo, Op0Hi);
   SDValue Op1Lo, Op1Hi;
   GetSplitVector(N->getOperand(1), Op1Lo, Op1Hi);
   SDValue Op2Lo, Op2Hi;
   GetSplitVector(N->getOperand(2), Op2Lo, Op2Hi);
#endif
   case ISD::FMINNUM:
   case ISD::FMAXNUM:
   case ISD::FMINNAN:
   case ISD::FMAXNAN:
     Res = WidenVecRes_Binary(N);
     break;
 
   case ISD::FADD:
+  case ISD::FADD_W_CHAIN:
   case ISD::FMUL:
+  case ISD::FMUL_W_CHAIN:
   case ISD::FPOW:
   case ISD::FSUB:
+  case ISD::FSUB_W_CHAIN:
   case ISD::FDIV:
+  case ISD::FDIV_W_CHAIN:
   case ISD::FREM:
+  case ISD::FREM_W_CHAIN:
   case ISD::SDIV:
   case ISD::UDIV:
   case ISD::SREM:
   case ISD::UREM:
     Res = WidenVecRes_BinaryCanTrap(N);
     break;
 
   case ISD::FCOPYSIGN:
SDValue DAGTypeLegalizer::WidenVecRes_BinaryCanTrap(SDNode *N) {
   unsigned NumElts =  VT.getVectorNumElements();
   const SDNodeFlags *Flags = N->getFlags();
   while (!TLI.isTypeLegal(VT) && NumElts != 1) {
     NumElts = NumElts / 2;
     VT = EVT::getVectorVT(*DAG.getContext(), WidenEltVT, NumElts);
   }
 
   if (NumElts != 1 && !TLI.canOpTrap(N->getOpcode(), VT)) {
+    if (isFPOpWithChain(N->getOpcode())) {
+      SDValue InOp1 = GetWidenedVector(N->getOperand(1));
+      SDValue InOp2 = GetWidenedVector(N->getOperand(2));
+      SDValue Ch = DAG.getNode(N->getOpcode(), dl,
+                               DAG.getVTList(WidenVT, MVT::Other),
+                               N->getOperand(0), InOp1, InOp2, N->getFlags());
+
+      // Legalized the chain result - switch anything that used the old chain to
+      // use the new one.
+      ReplaceValueWith(SDValue(N, 1), Ch.getValue(1));
+
+      return Ch.getValue(0);
+    }
+
     // Operation doesn't trap so just widen as normal.
     SDValue InOp1 = GetWidenedVector(N->getOperand(0));
     SDValue InOp2 = GetWidenedVector(N->getOperand(1));
     return DAG.getNode(N->getOpcode(), dl, WidenVT, InOp1, InOp2, Flags);
   }
 
   // No legal vector version so unroll the vector operation and then widen.
-  if (NumElts == 1)
-    return DAG.UnrollVectorOp(N, WidenVT.getVectorNumElements());
+  if (NumElts == 1) {
+    SDValue Chain;
+    SDValue Ret = DAG.UnrollVectorOp(N, WidenVT.getVectorNumElements(), &Chain);
+    if (Chain)
+      AnalyzeNewNode(Chain.getNode());
+    return Ret;
+  }
 
   // Since the operation can trap, apply operation on the original vector.
   EVT MaxVT = VT;
   SDValue InOp1 = GetWidenedVector(N->getOperand(0));
   SDValue InOp2 = GetWidenedVector(N->getOperand(1));
   unsigned CurNumElts = N->getValueType(0).getVectorNumElements();
 
   SmallVector<SDValue, 16> ConcatOps(CurNumElts);

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

#ifndef NDEBUG
   NextPersistentId = 0;
 #endif
 }
 
 BinarySDNode *SelectionDAG::GetBinarySDNode(unsigned Opcode, SDLoc DL,
                                             SDVTList VTs, SDValue N1,
                                             SDValue N2,
                                             const SDNodeFlags *Flags) {
+  assert(!isFPOpWithChain(Opcode) &&
+         "F*_W_CHAIN instructions should be constucted with overload that "
+         "takes a chain.");
+
   if (isBinOpWithFlags(Opcode)) {
     // If no flags were passed in, use a default flags object.
     SDNodeFlags F;
     if (Flags == nullptr)
       Flags = &F;
 
     BinaryWithFlagsSDNode *FN = new (NodeAllocator) BinaryWithFlagsSDNode(
         Opcode, DL.getIROrder(), DL.getDebugLoc(), VTs, N1, N2, *Flags);
 
     return FN;
   }
 
   BinarySDNode *N = new (NodeAllocator)
       BinarySDNode(Opcode, DL.getIROrder(), DL.getDebugLoc(), VTs, N1, N2);
   return N;
 }
 
+BinarySDNode *SelectionDAG::GetBinarySDNode(unsigned Opcode, SDLoc DL,
+                                            SDVTList VTs, SDValue Chain,
+                                            SDValue N1, SDValue N2,
+                                            const SDNodeFlags *Flags) {
+  assert(isFPOpWithChain(Opcode) &&
+         "Expected floating-point instruction with a chain.");
+
+  // If no flags were passed in, use a default flags object.
+  SDNodeFlags F;
+  if (Flags == nullptr)
+    Flags = &F;
+
+  BinaryWithFlagsSDNode *FN = new (NodeAllocator) BinaryWithFlagsSDNode(
+      Opcode, DL.getIROrder(), DL.getDebugLoc(), VTs, Chain, N1, N2, *Flags);
+
+  return FN;
+}
+
 SDNode *SelectionDAG::FindNodeOrInsertPos(const FoldingSetNodeID &ID,
                                           void *&InsertPos) {
   SDNode *N = CSEMap.FindNodeOrInsertPos(ID, InsertPos);
   if (N) {
     switch (N->getOpcode()) {
     default: break;
     case ISD::Constant:
     case ISD::ConstantFP:
SDValue SelectionDAG::FoldConstantVectorArithmetic(unsigned Opcode, SDLoc DL,
 
   assert(ScalarResults.size() == NumElts &&
          "Unexpected number of scalar results for BUILD_VECTOR");
   return getNode(ISD::BUILD_VECTOR, DL, VT, ScalarResults);
 }
 
 SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1,
                               SDValue N2, const SDNodeFlags *Flags) {
+  assert(!isFPOpWithChain(Opcode) &&
+         "F*_W_CHAIN instructions require a chain, use another overload.");
+
   ConstantSDNode *N1C = dyn_cast<ConstantSDNode>(N1);
   ConstantSDNode *N2C = dyn_cast<ConstantSDNode>(N2);
   ConstantFPSDNode *N1CFP = dyn_cast<ConstantFPSDNode>(N1);
   ConstantFPSDNode *N2CFP = dyn_cast<ConstantFPSDNode>(N2);
 
   // Canonicalize constant to RHS if commutative.
   if (isCommutativeBinOp(Opcode)) {
     if (N1C && !N2C) {
SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL, EVT VT, SDValue N1,
   } else {
     N = GetBinarySDNode(Opcode, DL, VTs, N1, N2, Flags);
   }
 
   InsertNode(N);
   return SDValue(N, 0);
 }
 
+SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL, SDVTList VTs,
+                              SDValue Chain, SDValue N1, SDValue N2,
+                              const SDNodeFlags *Flags) {
+  assert(isFPOpWithChain(Opcode) &&
+         "Expected floating-point instruction with a chain.");
+
+  if (N1.getOpcode() == ISD::UNDEF) {
+    switch (Opcode) {
+    case ISD::FSUB_W_CHAIN:
+    case ISD::FDIV_W_CHAIN:
+    case ISD::FREM_W_CHAIN:
+      return N1;     // fold op(undef, arg2) -> undef
+    case ISD::FADD_W_CHAIN:
+    case ISD::FMUL_W_CHAIN:
+      std::swap(N1, N2);
+      break;
+    }
+  }
+
+  if (getTarget().Options.UnsafeFPMath && N2.getOpcode() == ISD::UNDEF)
+    return N2;     // fold op(arg1, undef) -> undef
+
+  // Memoize this node if possible.
+  SDValue Ops[] = {Chain, N1, N2};
+  FoldingSetNodeID ID;
+  AddNodeIDNode(ID, Opcode, VTs, Ops);
+  AddNodeIDFlags(ID, Opcode, Flags);
+  void *IP = nullptr;
+  if (SDNode *E = FindNodeOrInsertPos(ID, DL.getDebugLoc(), IP))
+    return SDValue(E, 0);
+
+  BinarySDNode *N = GetBinarySDNode(Opcode, DL, VTs, Chain, N1, N2, Flags);
+
+  CSEMap.InsertNode(N, IP);
+
+  InsertNode(N);
+  return SDValue(N, 0);
+}
+
 SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL, EVT VT,
                               SDValue N1, SDValue N2, SDValue N3) {
   // Perform various simplifications.
   switch (Opcode) {
   case ISD::FMA: {
     ConstantFPSDNode *N1CFP = dyn_cast<ConstantFPSDNode>(N1);
     ConstantFPSDNode *N2CFP = dyn_cast<ConstantFPSDNode>(N2);
     ConstantFPSDNode *N3CFP = dyn_cast<ConstantFPSDNode>(N3);

 }
 
 SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL,
                               ArrayRef<EVT> ResultTys, ArrayRef<SDValue> Ops) {
   return getNode(Opcode, DL, getVTList(ResultTys), Ops);
 }
 
 SDValue SelectionDAG::getNode(unsigned Opcode, SDLoc DL, SDVTList VTList,
-                              ArrayRef<SDValue> Ops) {
+                              ArrayRef<SDValue> Ops, const SDNodeFlags *Flags) {
   if (VTList.NumVTs == 1)
-    return getNode(Opcode, DL, VTList.VTs[0], Ops);
+    return getNode(Opcode, DL, VTList.VTs[0], Ops, Flags);
 
 #if 0
   switch (Opcode) {
   // FIXME: figure out how to safely handle things like
   // int foo(int x) { return 1 << (x & 255); }
   // int bar() { return foo(256); }
   case ISD::SRA_PARTS:
   case ISD::SRL_PARTS:
if (VTList.VTs[VTList.NumVTs-1] != MVT::Glue) {
     void *IP = nullptr;
     if (SDNode *E = FindNodeOrInsertPos(ID, DL.getDebugLoc(), IP))
       return SDValue(E, 0);
 
     if (NumOps == 1) {
       N = new (NodeAllocator) UnarySDNode(Opcode, DL.getIROrder(),
                                           DL.getDebugLoc(), VTList, Ops[0]);
     } else if (NumOps == 2) {
-      N = new (NodeAllocator) BinarySDNode(Opcode, DL.getIROrder(),
-                                           DL.getDebugLoc(), VTList, Ops[0],
-                                           Ops[1]);
+      N = GetBinarySDNode(Opcode, DL, VTList, Ops[0], Ops[1], Flags);
     } else if (NumOps == 3) {
+      if (isFPOpWithChain(Opcode))
+        N = GetBinarySDNode(Opcode, DL, VTList, Ops[0], Ops[1], Ops[2], Flags);
+      else
         N = new (NodeAllocator) TernarySDNode(Opcode, DL.getIROrder(),
-                                            DL.getDebugLoc(), VTList, Ops[0],
-                                            Ops[1], Ops[2]);
+                                              DL.getDebugLoc(), VTList,
+                                              Ops[0], Ops[1], Ops[2]);
     } else {
       N = new (NodeAllocator) SDNode(Opcode, DL.getIROrder(), DL.getDebugLoc(),
                                      VTList, Ops);
     }
     CSEMap.InsertNode(N, IP);
   } else {
     if (NumOps == 1) {
       N = new (NodeAllocator) UnarySDNode(Opcode, DL.getIROrder(),
                                           DL.getDebugLoc(), VTList, Ops[0]);
     } else if (NumOps == 2) {
-      N = new (NodeAllocator) BinarySDNode(Opcode, DL.getIROrder(),
-                                           DL.getDebugLoc(), VTList, Ops[0],
-                                           Ops[1]);
+      N = GetBinarySDNode(Opcode, DL, VTList, Ops[0], Ops[1], Flags);
     } else if (NumOps == 3) {
+      if (isFPOpWithChain(Opcode))
+        N = GetBinarySDNode(Opcode, DL, VTList, Ops[0], Ops[1], Ops[2], Flags);
+      else
         N = new (NodeAllocator) TernarySDNode(Opcode, DL.getIROrder(),
                                               DL.getDebugLoc(), VTList, Ops[0],
                                               Ops[1], Ops[2]);
     } else {
       N = new (NodeAllocator) SDNode(Opcode, DL.getIROrder(), DL.getDebugLoc(),
                                      VTList, Ops);
     }
   }
   InsertNode(N);
   return SDValue(N, 0);
 }

 }
 
 const SDNodeFlags *SDNode::getFlags() const {
   if (auto *FlagsNode = dyn_cast<BinaryWithFlagsSDNode>(this))
     return &FlagsNode->Flags;
   return nullptr;
 }
 
-SDValue SelectionDAG::UnrollVectorOp(SDNode *N, unsigned ResNE) {
-  assert(N->getNumValues() == 1 &&
+SDValue SelectionDAG::UnrollVectorOp(SDNode *N, unsigned ResNE,
+                                     SDValue *TokenFactor) {
+  assert(((isFPOpWithChain(N->getOpcode()) && N->getNumValues() == 2) ||
+          N->getNumValues() == 1) &&
          "Can't unroll a vector with multiple results!");
 
   EVT VT = N->getValueType(0);
+  const SDNodeFlags *Flags = N->getFlags();
   unsigned NE = VT.getVectorNumElements();
   EVT EltVT = VT.getVectorElementType();
   SDLoc dl(N);
 
   SmallVector<SDValue, 8> Scalars;
   SmallVector<SDValue, 4> Operands(N->getNumOperands());
 
   // If ResNE is 0, fully unroll the vector op.
   if (ResNE == 0)
     ResNE = NE;
   else if (NE > ResNE)
     NE = ResNE;
 
+  SmallVector<SDValue, 8> ArgChains;
+
   unsigned i;
   for (i= 0; i != NE; ++i) {
     for (unsigned j = 0, e = N->getNumOperands(); j != e; ++j) {
       SDValue Operand = N->getOperand(j);
       EVT OperandVT = Operand.getValueType();
       if (OperandVT.isVector()) {
         // A vector operand; extract a single element.
         EVT OperandEltVT = OperandVT.getVectorElementType();
         Operands[j] =
             getNode(ISD::EXTRACT_VECTOR_ELT, dl, OperandEltVT, Operand,
                     getConstant(i, dl, TLI->getVectorIdxTy(getDataLayout())));
       } else {
         // A scalar operand; just use it as is.
         Operands[j] = Operand;
       }
     }
 
     switch (N->getOpcode()) {
     default: {
       Scalars.push_back(getNode(N->getOpcode(), dl, EltVT, Operands,
                                 N->getFlags()));
       break;
     }
+    case ISD::FADD_W_CHAIN:
+    case ISD::FSUB_W_CHAIN:
+    case ISD::FMUL_W_CHAIN:
+    case ISD::FDIV_W_CHAIN:
+    case ISD::FREM_W_CHAIN: {
+      SDValue NewNode = getNode(N->getOpcode(), dl,
+                                getVTList(EltVT, MVT::Other), Operands, Flags);
+      Scalars.push_back(NewNode);
+      ArgChains.push_back(NewNode.getValue(1));
+      break;
+    }
     case ISD::VSELECT:
       Scalars.push_back(getNode(ISD::SELECT, dl, EltVT, Operands));
       break;
     case ISD::SHL:
     case ISD::SRA:
     case ISD::SRL:
     case ISD::ROTL:
     case ISD::ROTR:
       Scalars.push_back(getNode(N->getOpcode(), dl, EltVT, Operands[0],
                                getShiftAmountOperand(Operands[0].getValueType(),
                                                      Operands[1])));
       break;
     case ISD::SIGN_EXTEND_INREG:
     case ISD::FP_ROUND_INREG: {
       EVT ExtVT = cast<VTSDNode>(Operands[1])->getVT().getVectorElementType();
       Scalars.push_back(getNode(N->getOpcode(), dl, EltVT,
                                 Operands[0],
                                 getValueType(ExtVT)));
     }
     }
   }
 
+  switch (N->getOpcode()) {
+  default: break;
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN: {
+    SDValue tmp;
+    if (!TokenFactor)
+      TokenFactor = &tmp;
+    *TokenFactor = getNode(ISD::TokenFactor, dl, MVT::Other, ArgChains);
+    ReplaceAllUsesOfValueWith(SDValue(N, 1), *TokenFactor);
+    break;
+  }
+  }
+
   for (; i < ResNE; ++i)
     Scalars.push_back(getUNDEF(EltVT));
 
   return getNode(ISD::BUILD_VECTOR, dl,
                  EVT::getVectorVT(*getContext(), EltVT, ResNE), Scalars);
 }
 
 

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (isa<Constant>(I.getOperand(0)) &&
     setValue(&I, DAG.getNode(ISD::FNEG, getCurSDLoc(),
                              Op2.getValueType(), Op2));
     return;
   }
 
   visitBinary(I, ISD::FSUB);
 }
 
+/// Possibly updates opcode of a floating point operation to one that is
+/// protected against instruction reordering, which can otherwise lead to
+/// differences in observable side effects at run-time.
+static unsigned toFPOpWithChain(unsigned OpCode) {
+  switch (OpCode) {
+  default: return OpCode;
+  case ISD::FADD: return ISD::FADD_W_CHAIN;
+  case ISD::FSUB: return ISD::FSUB_W_CHAIN;
+  case ISD::FMUL: return ISD::FMUL_W_CHAIN;
+  case ISD::FDIV: return ISD::FDIV_W_CHAIN;
+  case ISD::FREM: return ISD::FREM_W_CHAIN;
+  }
+}
+
 void SelectionDAGBuilder::visitBinary(const User &I, unsigned OpCode) {
   SDValue Op1 = getValue(I.getOperand(0));
   SDValue Op2 = getValue(I.getOperand(1));
 
   bool nuw = false;
   bool nsw = false;
   bool exact = false;
   FastMathFlags FMF;
 
   if (const OverflowingBinaryOperator *OFBinOp =
           dyn_cast<const OverflowingBinaryOperator>(&I)) {
     nuw = OFBinOp->hasNoUnsignedWrap();
     nsw = OFBinOp->hasNoSignedWrap();
   }
   if (const PossiblyExactOperator *ExactOp =
           dyn_cast<const PossiblyExactOperator>(&I))
     exact = ExactOp->isExact();
-  if (const FPMathOperator *FPOp = dyn_cast<const FPMathOperator>(&I))
+  if (auto *FPOp = dyn_cast<const FPMathOperator>(&I)) {
     FMF = FPOp->getFastMathFlags();
+  } else if (isa<ConstantExpr>(&I)) {
+    // Constant expressions don't store fast-math flags, so fill those related
+    // to floating-point access from global options.
+    switch (OpCode) {
+    case ISD::FADD:
+    case ISD::FSUB:
+    case ISD::FMUL:
+    case ISD::FDIV:
+    case ISD::FREM:
+      if (!TM.Options.AllowFPExceptAccess)
+        FMF.setNoExceptions();
+      if (!TM.Options.AllowFPRoundAccess)
+        FMF.setNoRounding();
+      break;
+    }
+  }
 
   SDNodeFlags Flags;
   Flags.setExact(exact);
   Flags.setNoSignedWrap(nsw);
   Flags.setNoUnsignedWrap(nuw);
   if (EnableFMFInDAG) {
     Flags.setAllowReciprocal(FMF.allowReciprocal());
     Flags.setNoInfs(FMF.noInfs());
     Flags.setNoNaNs(FMF.noNaNs());
     Flags.setNoSignedZeros(FMF.noSignedZeros());
     Flags.setUnsafeAlgebra(FMF.unsafeAlgebra());
     Flags.setNoExceptions(FMF.noExceptions());
     Flags.setNoRounding(FMF.noRounding());
   }
-  SDValue BinNodeValue = DAG.getNode(OpCode, getCurSDLoc(), Op1.getValueType(),
-                                     Op1, Op2, &Flags);
+
+  if (!FMF.noExceptions() || !FMF.noRounding() ||
+      TM.Options.AllowFPExceptAccess || TM.Options.AllowFPRoundAccess)
+    OpCode = toFPOpWithChain(OpCode);
+
+  SDValue BinNodeValue;
+  switch (OpCode) {
+  default:
+    BinNodeValue = DAG.getNode(OpCode, getCurSDLoc(), Op1.getValueType(),
     setValue(&I, BinNodeValue);
+    break;
+  case ISD::FADD_W_CHAIN:
+  case ISD::FSUB_W_CHAIN:
+  case ISD::FMUL_W_CHAIN:
+  case ISD::FDIV_W_CHAIN:
+  case ISD::FREM_W_CHAIN:
+    BinNodeValue = DAG.getNode(OpCode, getCurSDLoc(),
+                               DAG.getVTList(Op1.getValueType(), MVT::Other),
+                               getRoot(), Op1, Op2, &Flags);
+    setValue(&I, BinNodeValue.getValue(0));
+
+    if (BinNodeValue->getNumValues() == 2)
+      DAG.setRoot(BinNodeValue.getValue(1));
+    break;
+  }
 }
 
 void SelectionDAGBuilder::visitShift(const User &I, unsigned Opcode) {
   SDValue Op1 = getValue(I.getOperand(0));
   SDValue Op2 = getValue(I.getOperand(1));
 
   EVT ShiftTy = DAG.getTargetLoweringInfo().getShiftAmountTy(
       Op2.getValueType(), DAG.getDataLayout());

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::OR:                         return "or";
   case ISD::XOR:                        return "xor";
   case ISD::SHL:                        return "shl";
   case ISD::SRA:                        return "sra";
   case ISD::SRL:                        return "srl";
   case ISD::ROTL:                       return "rotl";
   case ISD::ROTR:                       return "rotr";
   case ISD::FADD:                       return "fadd";
+  case ISD::FADD_W_CHAIN:               return "fadd_w_chain";
   case ISD::FSUB:                       return "fsub";
+  case ISD::FSUB_W_CHAIN:               return "fsub_w_chain";
   case ISD::FMUL:                       return "fmul";
+  case ISD::FMUL_W_CHAIN:               return "fmul_w_chain";
   case ISD::FDIV:                       return "fdiv";
+  case ISD::FDIV_W_CHAIN:               return "fdiv_w_chain";
   case ISD::FMA:                        return "fma";
   case ISD::FMAD:                       return "fmad";
   case ISD::FREM:                       return "frem";
+  case ISD::FREM_W_CHAIN:               return "frem_w_chain";
   case ISD::FCOPYSIGN:                  return "fcopysign";
   case ISD::FGETSIGN:                   return "fgetsign";
   case ISD::FPOW:                       return "fpow";
   case ISD::SMIN:                       return "smin";
   case ISD::SMAX:                       return "smax";
   case ISD::UMIN:                       return "umin";
   case ISD::UMAX:                       return "umax";
 

lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

case OPC_MorphNodeTo: {
              i != e; ++i) {
           SDValue V = NodeToMatch->getOperand(i);
           if (V.getValueType() == MVT::Glue) break;
           Ops.push_back(V);
         }
       }
 
       // If this has chain/glue inputs, add them.
-      if (EmitNodeInfo & OPFL_Chain)
+      if (EmitNodeInfo & OPFL_Chain) {
+        if (InputChain.getNode())
           Ops.push_back(InputChain);
+        else
+          Ops.push_back(CurDAG->getEntryNode());
+      }
       if ((EmitNodeInfo & OPFL_GlueInput) && InputGlue.getNode() != nullptr)
         Ops.push_back(InputGlue);
 
       // Create the node.
       SDNode *Res = nullptr;
       if (Opcode != OPC_MorphNodeTo) {
         // If this is a normal EmitNode command, just create the new node and
         // add the results to the RecordedNodes list.

lib/CodeGen/SelectionDAG/TargetLowering.cpp

 /// result of type RetVT.
 std::pair<SDValue, SDValue>
 TargetLowering::makeLibCall(SelectionDAG &DAG,
                             RTLIB::Libcall LC, EVT RetVT,
                             ArrayRef<SDValue> Ops,
                             bool isSigned, SDLoc dl,
                             bool doesNotReturn,
                             bool isReturnValueUsed) const {
+  bool HasChain = (!Ops.empty() && Ops[0].getValueType() == MVT::Other);
+
   TargetLowering::ArgListTy Args;
   Args.reserve(Ops.size());
 
   TargetLowering::ArgListEntry Entry;
-  for (SDValue Op : Ops) {
-    Entry.Node = Op;
+  for (unsigned i = HasChain ? 1 : 0; i != Ops.size(); ++i) {
+    Entry.Node = Ops[i];
     Entry.Ty = Entry.Node.getValueType().getTypeForEVT(*DAG.getContext());
-    Entry.isSExt = shouldSignExtendTypeInLibCall(Op.getValueType(), isSigned);
-    Entry.isZExt = !shouldSignExtendTypeInLibCall(Op.getValueType(), isSigned);
+    Entry.isSExt = shouldSignExtendTypeInLibCall(Ops[i].getValueType(),
+                                                 isSigned);
+    Entry.isZExt = !shouldSignExtendTypeInLibCall(Ops[i].getValueType(),
+                                                  isSigned);
     Args.push_back(Entry);
   }
 
   markInRegArguments(DAG, Args);
 
   if (LC == RTLIB::UNKNOWN_LIBCALL)
     report_fatal_error("Unsupported library call operation!");
   SDValue Callee = DAG.getExternalSymbol(getLibcallName(LC),
                                          getPointerTy(DAG.getDataLayout()));
 
   Type *RetTy = RetVT.getTypeForEVT(*DAG.getContext());
   TargetLowering::CallLoweringInfo CLI(DAG);
   bool signExtend = shouldSignExtendTypeInLibCall(RetVT, isSigned);
-  CLI.setDebugLoc(dl).setChain(DAG.getEntryNode())
+  CLI.setDebugLoc(dl)
+    .setChain(HasChain ? Ops[0] : DAG.getEntryNode())
     .setCallee(getLibcallCallingConv(LC), RetTy, Callee, std::move(Args), 0)
     .setNoReturn(doesNotReturn).setDiscardResult(!isReturnValueUsed)
     .setSExtResult(signExtend).setZExtResult(!signExtend);
   return LowerCallTo(CLI);
 }
 
 /// SoftenSetCCOperands - Soften the operands of a comparison.  This code is
 /// shared among BR_CC, SELECT_CC, and SETCC handlers.

lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerEXTRACT_SUBVECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorSRA_SRL_SHL(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShiftLeftParts(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShiftRightParts(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVSETCC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCTPOP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerF128Call(SDValue Op, SelectionDAG &DAG,
                         RTLIB::Libcall Call) const;
+  SDValue LowerF128CallWithChain(SDValue Op, SelectionDAG &DAG,
+                                 RTLIB::Libcall Call) const;
   SDValue LowerFCOPYSIGN(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_EXTEND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_ROUND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFP_TO_INT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINT_TO_FP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorAND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCONCAT_VECTORS(SDValue Op, SelectionDAG &DAG) const;

lib/Target/AArch64/AArch64ISelLowering.cpp

AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
 
   setOperationAction(ISD::SHL_PARTS, MVT::i64, Custom);
   setOperationAction(ISD::SRA_PARTS, MVT::i64, Custom);
   setOperationAction(ISD::SRL_PARTS, MVT::i64, Custom);
 
   setOperationAction(ISD::FREM, MVT::f32, Expand);
   setOperationAction(ISD::FREM, MVT::f64, Expand);
   setOperationAction(ISD::FREM, MVT::f80, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f32, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f64, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f80, Expand);
 
   // Custom lowering hooks are needed for XOR
   // to fold it into CSINC/CSINV.
   setOperationAction(ISD::XOR, MVT::i32, Custom);
   setOperationAction(ISD::XOR, MVT::i64, Custom);
 
   // Virtually no operation on f128 is legal, but LLVM can't expand them when
   // there's a valid register class, so we need custom operations in most cases.
   setOperationAction(ISD::FABS, MVT::f128, Expand);
   setOperationAction(ISD::FADD, MVT::f128, Custom);
+  setOperationAction(ISD::FADD_W_CHAIN, MVT::f128, Custom);
   setOperationAction(ISD::FCOPYSIGN, MVT::f128, Expand);
   setOperationAction(ISD::FCOS, MVT::f128, Expand);
   setOperationAction(ISD::FDIV, MVT::f128, Custom);
+  setOperationAction(ISD::FDIV_W_CHAIN, MVT::f128, Custom);
   setOperationAction(ISD::FMA, MVT::f128, Expand);
   setOperationAction(ISD::FMUL, MVT::f128, Custom);
+  setOperationAction(ISD::FMUL_W_CHAIN, MVT::f128, Custom);
   setOperationAction(ISD::FNEG, MVT::f128, Expand);
   setOperationAction(ISD::FPOW, MVT::f128, Expand);
   setOperationAction(ISD::FREM, MVT::f128, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f128, Expand);
   setOperationAction(ISD::FRINT, MVT::f128, Expand);
   setOperationAction(ISD::FSIN, MVT::f128, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f128, Expand);
   setOperationAction(ISD::FSQRT, MVT::f128, Expand);
   setOperationAction(ISD::FSUB, MVT::f128, Custom);
+  setOperationAction(ISD::FSUB_W_CHAIN, MVT::f128, Custom);
   setOperationAction(ISD::FTRUNC, MVT::f128, Expand);
   setOperationAction(ISD::SETCC, MVT::f128, Custom);
   setOperationAction(ISD::BR_CC, MVT::f128, Custom);
   setOperationAction(ISD::SELECT, MVT::f128, Custom);
   setOperationAction(ISD::SELECT_CC, MVT::f128, Custom);
   setOperationAction(ISD::FP_EXTEND, MVT::f128, Custom);
 
   // Lowering for many of the conversions is actually specified by the non-f128
AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
   setOperationAction(ISD::FCOS, MVT::f32, Expand);
   setOperationAction(ISD::FCOS, MVT::f64, Expand);
   setOperationAction(ISD::FPOW, MVT::f32, Expand);
   setOperationAction(ISD::FPOW, MVT::f64, Expand);
   setOperationAction(ISD::FCOPYSIGN, MVT::f64, Custom);
   setOperationAction(ISD::FCOPYSIGN, MVT::f32, Custom);
 
   // f16 is a storage-only type, always promote it to f32.
-  setOperationAction(ISD::SETCC,       MVT::f16,  Promote);
-  setOperationAction(ISD::BR_CC,       MVT::f16,  Promote);
-  setOperationAction(ISD::SELECT_CC,   MVT::f16,  Promote);
-  setOperationAction(ISD::SELECT,      MVT::f16,  Promote);
-  setOperationAction(ISD::FADD,        MVT::f16,  Promote);
-  setOperationAction(ISD::FSUB,        MVT::f16,  Promote);
-  setOperationAction(ISD::FMUL,        MVT::f16,  Promote);
-  setOperationAction(ISD::FDIV,        MVT::f16,  Promote);
-  setOperationAction(ISD::FREM,        MVT::f16,  Promote);
-  setOperationAction(ISD::FMA,         MVT::f16,  Promote);
-  setOperationAction(ISD::FNEG,        MVT::f16,  Promote);
-  setOperationAction(ISD::FABS,        MVT::f16,  Promote);
-  setOperationAction(ISD::FCEIL,       MVT::f16,  Promote);
-  setOperationAction(ISD::FCOPYSIGN,   MVT::f16,  Promote);
-  setOperationAction(ISD::FCOS,        MVT::f16,  Promote);
-  setOperationAction(ISD::FFLOOR,      MVT::f16,  Promote);
-  setOperationAction(ISD::FNEARBYINT,  MVT::f16,  Promote);
-  setOperationAction(ISD::FPOW,        MVT::f16,  Promote);
-  setOperationAction(ISD::FPOWI,       MVT::f16,  Promote);
-  setOperationAction(ISD::FRINT,       MVT::f16,  Promote);
-  setOperationAction(ISD::FSIN,        MVT::f16,  Promote);
-  setOperationAction(ISD::FSINCOS,     MVT::f16,  Promote);
-  setOperationAction(ISD::FSQRT,       MVT::f16,  Promote);
-  setOperationAction(ISD::FEXP,        MVT::f16,  Promote);
-  setOperationAction(ISD::FEXP2,       MVT::f16,  Promote);
-  setOperationAction(ISD::FLOG,        MVT::f16,  Promote);
-  setOperationAction(ISD::FLOG2,       MVT::f16,  Promote);
-  setOperationAction(ISD::FLOG10,      MVT::f16,  Promote);
-  setOperationAction(ISD::FROUND,      MVT::f16,  Promote);
-  setOperationAction(ISD::FTRUNC,      MVT::f16,  Promote);
-  setOperationAction(ISD::FMINNUM,     MVT::f16,  Promote);
-  setOperationAction(ISD::FMAXNUM,     MVT::f16,  Promote);
-  setOperationAction(ISD::FMINNAN,     MVT::f16,  Promote);
-  setOperationAction(ISD::FMAXNAN,     MVT::f16,  Promote);
+  setOperationAction(ISD::SETCC,         MVT::f16,  Promote);
+  setOperationAction(ISD::BR_CC,         MVT::f16,  Promote);
+  setOperationAction(ISD::SELECT_CC,     MVT::f16,  Promote);
+  setOperationAction(ISD::SELECT,        MVT::f16,  Promote);
+  setOperationAction(ISD::FADD,          MVT::f16,  Promote);
+  setOperationAction(ISD::FADD_W_CHAIN,  MVT::f16,  Promote);
+  setOperationAction(ISD::FSUB,          MVT::f16,  Promote);
+  setOperationAction(ISD::FSUB_W_CHAIN,  MVT::f16,  Promote);
+  setOperationAction(ISD::FMUL,          MVT::f16,  Promote);
+  setOperationAction(ISD::FMUL_W_CHAIN,  MVT::f16,  Promote);
+  setOperationAction(ISD::FDIV,          MVT::f16,  Promote);
+  setOperationAction(ISD::FDIV_W_CHAIN,  MVT::f16,  Promote);
+  setOperationAction(ISD::FREM,          MVT::f16,  Promote);
+  setOperationAction(ISD::FREM_W_CHAIN,  MVT::f16,  Promote);
+  setOperationAction(ISD::FMA,           MVT::f16,  Promote);
+  setOperationAction(ISD::FNEG,          MVT::f16,  Promote);
+  setOperationAction(ISD::FABS,          MVT::f16,  Promote);
+  setOperationAction(ISD::FCEIL,         MVT::f16,  Promote);
+  setOperationAction(ISD::FCOPYSIGN,     MVT::f16,  Promote);
+  setOperationAction(ISD::FCOS,          MVT::f16,  Promote);
+  setOperationAction(ISD::FFLOOR,        MVT::f16,  Promote);
+  setOperationAction(ISD::FNEARBYINT,    MVT::f16,  Promote);
+  setOperationAction(ISD::FPOW,          MVT::f16,  Promote);
+  setOperationAction(ISD::FPOWI,         MVT::f16,  Promote);
+  setOperationAction(ISD::FRINT,         MVT::f16,  Promote);
+  setOperationAction(ISD::FSIN,          MVT::f16,  Promote);
+  setOperationAction(ISD::FSINCOS,       MVT::f16,  Promote);
+  setOperationAction(ISD::FSQRT,         MVT::f16,  Promote);
+  setOperationAction(ISD::FEXP,          MVT::f16,  Promote);
+  setOperationAction(ISD::FEXP2,         MVT::f16,  Promote);
+  setOperationAction(ISD::FLOG,          MVT::f16,  Promote);
+  setOperationAction(ISD::FLOG2,         MVT::f16,  Promote);
+  setOperationAction(ISD::FLOG10,        MVT::f16,  Promote);
+  setOperationAction(ISD::FROUND,        MVT::f16,  Promote);
+  setOperationAction(ISD::FTRUNC,        MVT::f16,  Promote);
+  setOperationAction(ISD::FMINNUM,       MVT::f16,  Promote);
+  setOperationAction(ISD::FMAXNUM,       MVT::f16,  Promote);
+  setOperationAction(ISD::FMINNAN,       MVT::f16,  Promote);
+  setOperationAction(ISD::FMAXNAN,       MVT::f16,  Promote);
 
   // v4f16 is also a storage-only type, so promote it to v4f32 when that is
   // known to be safe.
   setOperationAction(ISD::FADD, MVT::v4f16, Promote);
+  setOperationAction(ISD::FADD_W_CHAIN, MVT::v4f16, Promote);
   setOperationAction(ISD::FSUB, MVT::v4f16, Promote);
+  setOperationAction(ISD::FSUB_W_CHAIN, MVT::v4f16, Promote);
   setOperationAction(ISD::FMUL, MVT::v4f16, Promote);
+  setOperationAction(ISD::FMUL_W_CHAIN, MVT::v4f16, Promote);
   setOperationAction(ISD::FDIV, MVT::v4f16, Promote);
+  setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f16, Promote);
   setOperationAction(ISD::FP_EXTEND, MVT::v4f16, Promote);
   setOperationAction(ISD::FP_ROUND, MVT::v4f16, Promote);
   AddPromotedToType(ISD::FADD, MVT::v4f16, MVT::v4f32);
+  AddPromotedToType(ISD::FADD_W_CHAIN, MVT::v4f16, MVT::v4f32);
   AddPromotedToType(ISD::FSUB, MVT::v4f16, MVT::v4f32);
+  AddPromotedToType(ISD::FSUB_W_CHAIN, MVT::v4f16, MVT::v4f32);
   AddPromotedToType(ISD::FMUL, MVT::v4f16, MVT::v4f32);
+  AddPromotedToType(ISD::FMUL_W_CHAIN, MVT::v4f16, MVT::v4f32);
   AddPromotedToType(ISD::FDIV, MVT::v4f16, MVT::v4f32);
+  AddPromotedToType(ISD::FDIV_W_CHAIN, MVT::v4f16, MVT::v4f32);
   AddPromotedToType(ISD::FP_EXTEND, MVT::v4f16, MVT::v4f32);
   AddPromotedToType(ISD::FP_ROUND, MVT::v4f16, MVT::v4f32);
 
   // Expand all other v4f16 operations.
   // FIXME: We could generate better code by promoting some operations to
   // a pair of v4f32s
   setOperationAction(ISD::FABS, MVT::v4f16, Expand);
   setOperationAction(ISD::FCEIL, MVT::v4f16, Expand);
   setOperationAction(ISD::FCOPYSIGN, MVT::v4f16, Expand);
   setOperationAction(ISD::FCOS, MVT::v4f16, Expand);
   setOperationAction(ISD::FFLOOR, MVT::v4f16, Expand);
   setOperationAction(ISD::FMA, MVT::v4f16, Expand);
   setOperationAction(ISD::FNEARBYINT, MVT::v4f16, Expand);
   setOperationAction(ISD::FNEG, MVT::v4f16, Expand);
   setOperationAction(ISD::FPOW, MVT::v4f16, Expand);
   setOperationAction(ISD::FPOWI, MVT::v4f16, Expand);
   setOperationAction(ISD::FREM, MVT::v4f16, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::v4f16, Expand);
   setOperationAction(ISD::FROUND, MVT::v4f16, Expand);
   setOperationAction(ISD::FRINT, MVT::v4f16, Expand);
   setOperationAction(ISD::FSIN, MVT::v4f16, Expand);
   setOperationAction(ISD::FSINCOS, MVT::v4f16, Expand);
   setOperationAction(ISD::FSQRT, MVT::v4f16, Expand);
   setOperationAction(ISD::FTRUNC, MVT::v4f16, Expand);
   setOperationAction(ISD::SETCC, MVT::v4f16, Expand);
   setOperationAction(ISD::BR_CC, MVT::v4f16, Expand);
   setOperationAction(ISD::SELECT, MVT::v4f16, Expand);
   setOperationAction(ISD::SELECT_CC, MVT::v4f16, Expand);
   setOperationAction(ISD::FEXP, MVT::v4f16, Expand);
   setOperationAction(ISD::FEXP2, MVT::v4f16, Expand);
   setOperationAction(ISD::FLOG, MVT::v4f16, Expand);
   setOperationAction(ISD::FLOG2, MVT::v4f16, Expand);
   setOperationAction(ISD::FLOG10, MVT::v4f16, Expand);
 
 
   // v8f16 is also a storage-only type, so expand it.
   setOperationAction(ISD::FABS, MVT::v8f16, Expand);
   setOperationAction(ISD::FADD, MVT::v8f16, Expand);
+  setOperationAction(ISD::FADD_W_CHAIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FCEIL, MVT::v8f16, Expand);
   setOperationAction(ISD::FCOPYSIGN, MVT::v8f16, Expand);
   setOperationAction(ISD::FCOS, MVT::v8f16, Expand);
   setOperationAction(ISD::FDIV, MVT::v8f16, Expand);
+  setOperationAction(ISD::FDIV_W_CHAIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FFLOOR, MVT::v8f16, Expand);
   setOperationAction(ISD::FMA, MVT::v8f16, Expand);
   setOperationAction(ISD::FMUL, MVT::v8f16, Expand);
+  setOperationAction(ISD::FMUL_W_CHAIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FNEARBYINT, MVT::v8f16, Expand);
   setOperationAction(ISD::FNEG, MVT::v8f16, Expand);
   setOperationAction(ISD::FPOW, MVT::v8f16, Expand);
   setOperationAction(ISD::FPOWI, MVT::v8f16, Expand);
   setOperationAction(ISD::FREM, MVT::v8f16, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FROUND, MVT::v8f16, Expand);
   setOperationAction(ISD::FRINT, MVT::v8f16, Expand);
   setOperationAction(ISD::FSIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FSINCOS, MVT::v8f16, Expand);
   setOperationAction(ISD::FSQRT, MVT::v8f16, Expand);
   setOperationAction(ISD::FSUB, MVT::v8f16, Expand);
+  setOperationAction(ISD::FSUB_W_CHAIN, MVT::v8f16, Expand);
   setOperationAction(ISD::FTRUNC, MVT::v8f16, Expand);
   setOperationAction(ISD::SETCC, MVT::v8f16, Expand);
   setOperationAction(ISD::BR_CC, MVT::v8f16, Expand);
   setOperationAction(ISD::SELECT, MVT::v8f16, Expand);
   setOperationAction(ISD::SELECT_CC, MVT::v8f16, Expand);
   setOperationAction(ISD::FP_EXTEND, MVT::v8f16, Expand);
   setOperationAction(ISD::FEXP, MVT::v8f16, Expand);
   setOperationAction(ISD::FEXP2, MVT::v8f16, Expand);
AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
 
   setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::Other, Custom);
 
   if (Subtarget->hasNEON()) {
     // FIXME: v1f64 shouldn't be legal if we can avoid it, because it leads to
     // silliness like this:
     setOperationAction(ISD::FABS, MVT::v1f64, Expand);
     setOperationAction(ISD::FADD, MVT::v1f64, Expand);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FCEIL, MVT::v1f64, Expand);
     setOperationAction(ISD::FCOPYSIGN, MVT::v1f64, Expand);
     setOperationAction(ISD::FCOS, MVT::v1f64, Expand);
     setOperationAction(ISD::FDIV, MVT::v1f64, Expand);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FFLOOR, MVT::v1f64, Expand);
     setOperationAction(ISD::FMA, MVT::v1f64, Expand);
     setOperationAction(ISD::FMUL, MVT::v1f64, Expand);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FNEARBYINT, MVT::v1f64, Expand);
     setOperationAction(ISD::FNEG, MVT::v1f64, Expand);
     setOperationAction(ISD::FPOW, MVT::v1f64, Expand);
     setOperationAction(ISD::FREM, MVT::v1f64, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FROUND, MVT::v1f64, Expand);
     setOperationAction(ISD::FRINT, MVT::v1f64, Expand);
     setOperationAction(ISD::FSIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FSINCOS, MVT::v1f64, Expand);
     setOperationAction(ISD::FSQRT, MVT::v1f64, Expand);
     setOperationAction(ISD::FSUB, MVT::v1f64, Expand);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::v1f64, Expand);
     setOperationAction(ISD::FTRUNC, MVT::v1f64, Expand);
     setOperationAction(ISD::SETCC, MVT::v1f64, Expand);
     setOperationAction(ISD::BR_CC, MVT::v1f64, Expand);
     setOperationAction(ISD::SELECT, MVT::v1f64, Expand);
     setOperationAction(ISD::SELECT_CC, MVT::v1f64, Expand);
     setOperationAction(ISD::FP_EXTEND, MVT::v1f64, Expand);
 
     setOperationAction(ISD::FP_TO_SINT, MVT::v1i64, Expand);
void AArch64TargetLowering::addTypeForNEON(EVT VT, EVT PromotedBitwiseVT) {
   if (VT != MVT::v8i8 && VT != MVT::v16i8)
     setOperationAction(ISD::CTPOP, VT.getSimpleVT(), Expand);
 
   setOperationAction(ISD::UDIV, VT.getSimpleVT(), Expand);
   setOperationAction(ISD::SDIV, VT.getSimpleVT(), Expand);
   setOperationAction(ISD::UREM, VT.getSimpleVT(), Expand);
   setOperationAction(ISD::SREM, VT.getSimpleVT(), Expand);
   setOperationAction(ISD::FREM, VT.getSimpleVT(), Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, VT.getSimpleVT(), Expand);
 
   setOperationAction(ISD::FP_TO_SINT, VT.getSimpleVT(), Custom);
   setOperationAction(ISD::FP_TO_UINT, VT.getSimpleVT(), Custom);
 
   // [SU][MIN|MAX] and [SU]ABSDIFF are available for all NEON types apart from
   // i64.
   if (!VT.isFloatingPoint() &&
       VT.getSimpleVT() != MVT::v2i64 && VT.getSimpleVT() != MVT::v1i64)

 }
 
 SDValue AArch64TargetLowering::LowerF128Call(SDValue Op, SelectionDAG &DAG,
                                              RTLIB::Libcall Call) const {
   SmallVector<SDValue, 2> Ops(Op->op_begin(), Op->op_end());
   return makeLibCall(DAG, Call, MVT::f128, Ops, false, SDLoc(Op)).first;
 }
 
+SDValue AArch64TargetLowering::LowerF128CallWithChain(SDValue Op,
+                                                      SelectionDAG &DAG,
+                                                    RTLIB::Libcall Call) const {
+  SmallVector<SDValue, 2> Ops(Op->op_begin() + 1, Op->op_end());
+  return makeLibCall(DAG, Call, MVT::f128, Ops, false, SDLoc(Op)).first;
+}
+
 static SDValue LowerXOR(SDValue Op, SelectionDAG &DAG) {
   SDValue Sel = Op.getOperand(0);
   SDValue Other = Op.getOperand(1);
 
   // If neither operand is a SELECT_CC, give up.
   if (Sel.getOpcode() != ISD::SELECT_CC)
     std::swap(Sel, Other);
   if (Sel.getOpcode() != ISD::SELECT_CC)
SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
   case ISD::FADD:
     return LowerF128Call(Op, DAG, RTLIB::ADD_F128);
   case ISD::FSUB:
     return LowerF128Call(Op, DAG, RTLIB::SUB_F128);
   case ISD::FMUL:
     return LowerF128Call(Op, DAG, RTLIB::MUL_F128);
   case ISD::FDIV:
     return LowerF128Call(Op, DAG, RTLIB::DIV_F128);
+  case ISD::FADD_W_CHAIN:
+    return LowerF128CallWithChain(Op, DAG, RTLIB::ADD_F128);
+  case ISD::FSUB_W_CHAIN:
+    return LowerF128CallWithChain(Op, DAG, RTLIB::SUB_F128);
+  case ISD::FMUL_W_CHAIN:
+    return LowerF128CallWithChain(Op, DAG, RTLIB::MUL_F128);
+  case ISD::FDIV_W_CHAIN:
+    return LowerF128CallWithChain(Op, DAG, RTLIB::DIV_F128);
   case ISD::FP_ROUND:
     return LowerFP_ROUND(Op, DAG);
   case ISD::FP_EXTEND:
     return LowerFP_EXTEND(Op, DAG);
   case ISD::FRAMEADDR:
     return LowerFRAMEADDR(Op, DAG);
   case ISD::RETURNADDR:
     return LowerRETURNADDR(Op, DAG);

lib/Target/AArch64/AArch64InstrInfo.td

 defm FSQRT  : SingleOperandFPData<0b0011, "fsqrt", fsqrt>;
 }
 
 //===----------------------------------------------------------------------===//
 // Floating point two operand instructions.
 //===----------------------------------------------------------------------===//
 
 defm FADD   : TwoOperandFPData<0b0010, "fadd", fadd>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FADD_FPE : TwoOperandFPData<0b0010, "fadd", faddwchain>;
 let SchedRW = [WriteFDiv] in {
 defm FDIV   : TwoOperandFPData<0b0001, "fdiv", fdiv>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FDIV_FPE : TwoOperandFPData<0b0001, "fdiv", fdivwchain>;
 }
 defm FMAXNM : TwoOperandFPData<0b0110, "fmaxnm", fmaxnum>;
 defm FMAX   : TwoOperandFPData<0b0100, "fmax", fmaxnan>;
 defm FMINNM : TwoOperandFPData<0b0111, "fminnm", fminnum>;
 defm FMIN   : TwoOperandFPData<0b0101, "fmin", fminnan>;
 let SchedRW = [WriteFMul] in {
 defm FMUL   : TwoOperandFPData<0b0000, "fmul", fmul>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FMUL_FPE : TwoOperandFPData<0b0000, "fmul", fmulwchain>;
 defm FNMUL  : TwoOperandFPDataNeg<0b1000, "fnmul", fmul>;
 }
 defm FSUB   : TwoOperandFPData<0b0011, "fsub", fsub>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FSUB_FPE : TwoOperandFPData<0b0011, "fsub", fsubwchain>;
 
 def : Pat<(v1f64 (fmaxnan (v1f64 FPR64:$Rn), (v1f64 FPR64:$Rm))),
           (FMAXDrr FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(v1f64 (fminnan (v1f64 FPR64:$Rn), (v1f64 FPR64:$Rm))),
           (FMINDrr FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(v1f64 (fmaxnum (v1f64 FPR64:$Rn), (v1f64 FPR64:$Rm))),
           (FMAXNMDrr FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(v1f64 (fminnum (v1f64 FPR64:$Rn), (v1f64 FPR64:$Rm))),

 defm CMGT    : SIMDThreeSameVector<0, 0b00110, "cmgt", AArch64cmgt>;
 defm CMHI    : SIMDThreeSameVector<1, 0b00110, "cmhi", AArch64cmhi>;
 defm CMHS    : SIMDThreeSameVector<1, 0b00111, "cmhs", AArch64cmhs>;
 defm CMTST   : SIMDThreeSameVector<0, 0b10001, "cmtst", AArch64cmtst>;
 defm FABD    : SIMDThreeSameVectorFP<1,1,0b11010,"fabd", int_aarch64_neon_fabd>;
 defm FACGE   : SIMDThreeSameVectorFPCmp<1,0,0b11101,"facge",int_aarch64_neon_facge>;
 defm FACGT   : SIMDThreeSameVectorFPCmp<1,1,0b11101,"facgt",int_aarch64_neon_facgt>;
 defm FADDP   : SIMDThreeSameVectorFP<1,0,0b11010,"faddp",int_aarch64_neon_addp>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FADDP_FPE : SIMDThreeSameVectorFP<1,0,0b11010,"faddp",
+                                       int_aarch64_neon_addp>;
 defm FADD    : SIMDThreeSameVectorFP<0,0,0b11010,"fadd", fadd>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FADD_FPE : SIMDThreeSameVectorFP<0,0,0b11010,"fadd", faddwchain>;
 defm FCMEQ   : SIMDThreeSameVectorFPCmp<0, 0, 0b11100, "fcmeq", AArch64fcmeq>;
 defm FCMGE   : SIMDThreeSameVectorFPCmp<1, 0, 0b11100, "fcmge", AArch64fcmge>;
 defm FCMGT   : SIMDThreeSameVectorFPCmp<1, 1, 0b11100, "fcmgt", AArch64fcmgt>;
 defm FDIV    : SIMDThreeSameVectorFP<1,0,0b11111,"fdiv", fdiv>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FDIV_FPE : SIMDThreeSameVectorFP<1,0,0b11111,"fdiv", fdivwchain>;
 defm FMAXNMP : SIMDThreeSameVectorFP<1,0,0b11000,"fmaxnmp", int_aarch64_neon_fmaxnmp>;
 defm FMAXNM  : SIMDThreeSameVectorFP<0,0,0b11000,"fmaxnm", fmaxnum>;
 defm FMAXP   : SIMDThreeSameVectorFP<1,0,0b11110,"fmaxp", int_aarch64_neon_fmaxp>;
 defm FMAX    : SIMDThreeSameVectorFP<0,0,0b11110,"fmax", fmaxnan>;
 defm FMINNMP : SIMDThreeSameVectorFP<1,1,0b11000,"fminnmp", int_aarch64_neon_fminnmp>;
 defm FMINNM  : SIMDThreeSameVectorFP<0,1,0b11000,"fminnm", fminnum>;
 defm FMINP   : SIMDThreeSameVectorFP<1,1,0b11110,"fminp", int_aarch64_neon_fminp>;
 defm FMIN    : SIMDThreeSameVectorFP<0,1,0b11110,"fmin", fminnan>;

 def : Pat<(v4f32 (fma (fneg V128:$Rn), V128:$Rm, V128:$Rd)),
           (FMLSv4f32 V128:$Rd, V128:$Rn, V128:$Rm)>;
 
 def : Pat<(v2f64 (fma (fneg V128:$Rn), V128:$Rm, V128:$Rd)),
           (FMLSv2f64 V128:$Rd, V128:$Rn, V128:$Rm)>;
 
 defm FMULX    : SIMDThreeSameVectorFP<0,0,0b11011,"fmulx", int_aarch64_neon_fmulx>;
 defm FMUL     : SIMDThreeSameVectorFP<1,0,0b11011,"fmul", fmul>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FMUL_FPE : SIMDThreeSameVectorFP<1,0,0b11011,"fmul", fmulwchain>;
 defm FRECPS   : SIMDThreeSameVectorFP<0,0,0b11111,"frecps", int_aarch64_neon_frecps>;
 defm FRSQRTS  : SIMDThreeSameVectorFP<0,1,0b11111,"frsqrts", int_aarch64_neon_frsqrts>;
 defm FSUB     : SIMDThreeSameVectorFP<0,1,0b11010,"fsub", fsub>;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FSUB_FPE : SIMDThreeSameVectorFP<0,1,0b11010,"fsub", fsubwchain>;
 defm MLA      : SIMDThreeSameVectorBHSTied<0, 0b10010, "mla",
                       TriOpFrag<(add node:$LHS, (mul node:$MHS, node:$RHS))> >;
 defm MLS      : SIMDThreeSameVectorBHSTied<1, 0b10010, "mls",
                       TriOpFrag<(sub node:$LHS, (mul node:$MHS, node:$RHS))> >;
 defm MUL      : SIMDThreeSameVectorBHS<0, 0b10011, "mul", mul>;
 defm PMUL     : SIMDThreeSameVectorB<1, 0b10011, "pmul", int_aarch64_neon_pmul>;
 defm SABA     : SIMDThreeSameVectorBHSTied<0, 0b01111, "saba",
       TriOpFrag<(add node:$LHS, (sabsdiff node:$MHS, node:$RHS))> >;

 defm CPY : SIMDScalarCPY<"cpy">;
 
 //----------------------------------------------------------------------------
 // AdvSIMD scalar pairwise instructions
 //----------------------------------------------------------------------------
 
 defm ADDP    : SIMDPairwiseScalarD<0, 0b11011, "addp">;
 defm FADDP   : SIMDFPPairwiseScalar<1, 0, 0b01101, "faddp">;
+let isCodeGenOnly = 1, hasSideEffects = 1 in
+defm FADDP_FPE : SIMDFPPairwiseScalar<1, 0, 0b01101, "faddp">;
 defm FMAXNMP : SIMDFPPairwiseScalar<1, 0, 0b01100, "fmaxnmp">;
 defm FMAXP   : SIMDFPPairwiseScalar<1, 0, 0b01111, "fmaxp">;
 defm FMINNMP : SIMDFPPairwiseScalar<1, 1, 0b01100, "fminnmp">;
 defm FMINP   : SIMDFPPairwiseScalar<1, 1, 0b01111, "fminp">;
 def : Pat<(v2i64 (AArch64saddv V128:$Rn)),
           (INSERT_SUBREG (v2i64 (IMPLICIT_DEF)), (ADDPv2i64p V128:$Rn), dsub)>;
 def : Pat<(v2i64 (AArch64uaddv V128:$Rn)),
           (INSERT_SUBREG (v2i64 (IMPLICIT_DEF)), (ADDPv2i64p V128:$Rn), dsub)>;
 def : Pat<(f32 (int_aarch64_neon_faddv (v2f32 V64:$Rn))),
           (FADDPv2i32p V64:$Rn)>;
 def : Pat<(f32 (int_aarch64_neon_faddv (v4f32 V128:$Rn))),
           (FADDPv2i32p (EXTRACT_SUBREG (FADDPv4f32 V128:$Rn, V128:$Rn), dsub))>;
 def : Pat<(f64 (int_aarch64_neon_faddv (v2f64 V128:$Rn))),
           (FADDPv2i64p V128:$Rn)>;
+def : Pat<(f32 (int_aarch64_neon_faddv (v2f32 V64:$Rn))),
+          (FADDP_FPEv2i32p V64:$Rn)>;
+def : Pat<(f32 (int_aarch64_neon_faddv (v4f32 V128:$Rn))),
+          (FADDP_FPEv2i32p (EXTRACT_SUBREG
+                            (FADDP_FPEv4f32 V128:$Rn, V128:$Rn), dsub))>;
+def : Pat<(f64 (int_aarch64_neon_faddv (v2f64 V128:$Rn))),
+          (FADDP_FPEv2i64p V128:$Rn)>;
 def : Pat<(f32 (int_aarch64_neon_fmaxnmv (v2f32 V64:$Rn))),
           (FMAXNMPv2i32p V64:$Rn)>;
 def : Pat<(f64 (int_aarch64_neon_fmaxnmv (v2f64 V128:$Rn))),
           (FMAXNMPv2i64p V128:$Rn)>;
 def : Pat<(f32 (int_aarch64_neon_fmaxv (v2f32 V64:$Rn))),
           (FMAXPv2i32p V64:$Rn)>;
 def : Pat<(f64 (int_aarch64_neon_fmaxv (v2f64 V128:$Rn))),
           (FMAXPv2i64p V128:$Rn)>;

 // Use pair-wise add instructions when summing up the lanes for v2f64, v2i64
 // or v2f32.
 def : Pat<(i64 (add (vector_extract (v2i64 FPR128:$Rn), (i64 0)),
                     (vector_extract (v2i64 FPR128:$Rn), (i64 1)))),
            (i64 (ADDPv2i64p (v2i64 FPR128:$Rn)))>;
 def : Pat<(f64 (fadd (vector_extract (v2f64 FPR128:$Rn), (i64 0)),
                      (vector_extract (v2f64 FPR128:$Rn), (i64 1)))),
            (f64 (FADDPv2i64p (v2f64 FPR128:$Rn)))>;
+def : Pat<(f64 (faddwchain (vector_extract (v2f64 FPR128:$Rn), (i64 0)),
+                     (vector_extract (v2f64 FPR128:$Rn), (i64 1)))),
+           (f64 (FADDP_FPEv2i64p (v2f64 FPR128:$Rn)))>;
     // vector_extract on 64-bit vectors gets promoted to a 128 bit vector,
     // so we match on v4f32 here, not v2f32. This will also catch adding
     // the low two lanes of a true v4f32 vector.
 def : Pat<(fadd (vector_extract (v4f32 FPR128:$Rn), (i64 0)),
                 (vector_extract (v4f32 FPR128:$Rn), (i64 1))),
           (f32 (FADDPv2i32p (EXTRACT_SUBREG FPR128:$Rn, dsub)))>;
+def : Pat<(faddwchain (vector_extract (v4f32 FPR128:$Rn), (i64 0)),
+                (vector_extract (v4f32 FPR128:$Rn), (i64 1))),
+          (f32 (FADDP_FPEv2i32p (EXTRACT_SUBREG FPR128:$Rn, dsub)))>;
 
 // Scalar 64-bit shifts in FPR64 registers.
 def : Pat<(i64 (int_aarch64_neon_sshl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (SSHLv1i64 FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(i64 (int_aarch64_neon_ushl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (USHLv1i64 FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(i64 (int_aarch64_neon_srshl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (SRSHLv1i64 FPR64:$Rn, FPR64:$Rm)>;

lib/Target/ARM/ARMISelLowering.cpp

if (VT.isInteger() && VT != PromotedBitwiseVT) {
     AddPromotedToType (ISD::AND, VT, PromotedBitwiseVT);
     setOperationAction(ISD::OR,  VT, Promote);
     AddPromotedToType (ISD::OR,  VT, PromotedBitwiseVT);
     setOperationAction(ISD::XOR, VT, Promote);
     AddPromotedToType (ISD::XOR, VT, PromotedBitwiseVT);
   }
 
   // Neon does not support vector divide/remainder operations.
-  setOperationAction(ISD::SDIV, VT, Expand);
-  setOperationAction(ISD::UDIV, VT, Expand);
-  setOperationAction(ISD::FDIV, VT, Expand);
-  setOperationAction(ISD::SREM, VT, Expand);
-  setOperationAction(ISD::UREM, VT, Expand);
-  setOperationAction(ISD::FREM, VT, Expand);
+  setOperationAction(ISD::SDIV,         VT, Expand);
+  setOperationAction(ISD::UDIV,         VT, Expand);
+  setOperationAction(ISD::FDIV,         VT, Expand);
+  setOperationAction(ISD::FDIV_W_CHAIN, VT, Expand);
+  setOperationAction(ISD::SREM,         VT, Expand);
+  setOperationAction(ISD::UREM,         VT, Expand);
+  setOperationAction(ISD::FREM,         VT, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, VT, Expand);
 
   if (VT.isInteger()) {
     setOperationAction(ISD::SABSDIFF, VT, Legal);
     setOperationAction(ISD::UABSDIFF, VT, Legal);
   }
   if (!VT.isFloatingPoint() &&
       VT != MVT::v2i64 && VT != MVT::v1i64)
     for (unsigned Opcode : {ISD::SMIN, ISD::SMAX, ISD::UMIN, ISD::UMAX})
if (Subtarget->hasNEON()) {
     addQRTypeForNEON(MVT::v4i32);
     addQRTypeForNEON(MVT::v2i64);
 
     // v2f64 is legal so that QR subregs can be extracted as f64 elements, but
     // neither Neon nor VFP support any arithmetic operations on it.
     // The same with v4f32. But keep in mind that vadd, vsub, vmul are natively
     // supported for v4f32.
     setOperationAction(ISD::FADD, MVT::v2f64, Expand);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::v2f64, Expand);
     setOperationAction(ISD::FSUB, MVT::v2f64, Expand);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::v2f64, Expand);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::v2f64, Expand);
     setOperationAction(ISD::FMUL, MVT::v2f64, Expand);
     // FIXME: Code duplication: FDIV and FREM are expanded always, see
     // ARMTargetLowering::addTypeForNEON method for details.
     setOperationAction(ISD::FDIV, MVT::v2f64, Expand);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::v2f64, Expand);
     setOperationAction(ISD::FREM, MVT::v2f64, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, MVT::v2f64, Expand);
     // FIXME: Create unittest.
     // In another words, find a way when "copysign" appears in DAG with vector
     // operands.
     setOperationAction(ISD::FCOPYSIGN, MVT::v2f64, Expand);
     // FIXME: Code duplication: SETCC has custom operation action, see
     // ARMTargetLowering::addTypeForNEON method for details.
     setOperationAction(ISD::SETCC, MVT::v2f64, Expand);
     // FIXME: Create unittest for FNEG and for FABS.
ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
   if (!Subtarget->isThumb1Only())
     setTargetDAGCombine(ISD::ADDC);
 
   if (Subtarget->isFPOnlySP()) {
     // When targeting a floating-point unit with only single-precision
     // operations, f64 is legal for the few double-precision instructions which
     // are present However, no double-precision operations other than moves,
     // loads and stores are provided by the hardware.
-    setOperationAction(ISD::FADD,       MVT::f64, Expand);
-    setOperationAction(ISD::FSUB,       MVT::f64, Expand);
-    setOperationAction(ISD::FMUL,       MVT::f64, Expand);
-    setOperationAction(ISD::FMA,        MVT::f64, Expand);
-    setOperationAction(ISD::FDIV,       MVT::f64, Expand);
-    setOperationAction(ISD::FREM,       MVT::f64, Expand);
-    setOperationAction(ISD::FCOPYSIGN,  MVT::f64, Expand);
-    setOperationAction(ISD::FGETSIGN,   MVT::f64, Expand);
-    setOperationAction(ISD::FNEG,       MVT::f64, Expand);
-    setOperationAction(ISD::FABS,       MVT::f64, Expand);
-    setOperationAction(ISD::FSQRT,      MVT::f64, Expand);
-    setOperationAction(ISD::FSIN,       MVT::f64, Expand);
-    setOperationAction(ISD::FCOS,       MVT::f64, Expand);
-    setOperationAction(ISD::FPOWI,      MVT::f64, Expand);
-    setOperationAction(ISD::FPOW,       MVT::f64, Expand);
-    setOperationAction(ISD::FLOG,       MVT::f64, Expand);
-    setOperationAction(ISD::FLOG2,      MVT::f64, Expand);
-    setOperationAction(ISD::FLOG10,     MVT::f64, Expand);
-    setOperationAction(ISD::FEXP,       MVT::f64, Expand);
-    setOperationAction(ISD::FEXP2,      MVT::f64, Expand);
-    setOperationAction(ISD::FCEIL,      MVT::f64, Expand);
-    setOperationAction(ISD::FTRUNC,     MVT::f64, Expand);
-    setOperationAction(ISD::FRINT,      MVT::f64, Expand);
-    setOperationAction(ISD::FNEARBYINT, MVT::f64, Expand);
-    setOperationAction(ISD::FFLOOR,     MVT::f64, Expand);
-    setOperationAction(ISD::SINT_TO_FP, MVT::i32, Custom);
-    setOperationAction(ISD::UINT_TO_FP, MVT::i32, Custom);
-    setOperationAction(ISD::FP_TO_SINT, MVT::i32, Custom);
-    setOperationAction(ISD::FP_TO_UINT, MVT::i32, Custom);
-    setOperationAction(ISD::FP_TO_SINT, MVT::f64, Custom);
-    setOperationAction(ISD::FP_TO_UINT, MVT::f64, Custom);
-    setOperationAction(ISD::FP_ROUND,   MVT::f32, Custom);
-    setOperationAction(ISD::FP_EXTEND,  MVT::f64, Custom);
+    setOperationAction(ISD::FADD,         MVT::f64, Expand);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FSUB,         MVT::f64, Expand);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FMUL,         MVT::f64, Expand);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FMA,          MVT::f64, Expand);
+    setOperationAction(ISD::FDIV,         MVT::f64, Expand);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FREM,         MVT::f64, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FCOPYSIGN,    MVT::f64, Expand);
+    setOperationAction(ISD::FGETSIGN,     MVT::f64, Expand);
+    setOperationAction(ISD::FNEG,         MVT::f64, Expand);
+    setOperationAction(ISD::FABS,         MVT::f64, Expand);
+    setOperationAction(ISD::FSQRT,        MVT::f64, Expand);
+    setOperationAction(ISD::FSIN,         MVT::f64, Expand);
+    setOperationAction(ISD::FCOS,         MVT::f64, Expand);
+    setOperationAction(ISD::FPOWI,        MVT::f64, Expand);
+    setOperationAction(ISD::FPOW,         MVT::f64, Expand);
+    setOperationAction(ISD::FLOG,         MVT::f64, Expand);
+    setOperationAction(ISD::FLOG2,        MVT::f64, Expand);
+    setOperationAction(ISD::FLOG10,       MVT::f64, Expand);
+    setOperationAction(ISD::FEXP,         MVT::f64, Expand);
+    setOperationAction(ISD::FEXP2,        MVT::f64, Expand);
+    setOperationAction(ISD::FCEIL,        MVT::f64, Expand);
+    setOperationAction(ISD::FTRUNC,       MVT::f64, Expand);
+    setOperationAction(ISD::FRINT,        MVT::f64, Expand);
+    setOperationAction(ISD::FNEARBYINT,   MVT::f64, Expand);
+    setOperationAction(ISD::FFLOOR,       MVT::f64, Expand);
+    setOperationAction(ISD::SINT_TO_FP,   MVT::i32, Custom);
+    setOperationAction(ISD::UINT_TO_FP,   MVT::i32, Custom);
+    setOperationAction(ISD::FP_TO_SINT,   MVT::i32, Custom);
+    setOperationAction(ISD::FP_TO_UINT,   MVT::i32, Custom);
+    setOperationAction(ISD::FP_TO_SINT,   MVT::f64, Custom);
+    setOperationAction(ISD::FP_TO_UINT,   MVT::f64, Custom);
+    setOperationAction(ISD::FP_ROUND,     MVT::f32, Custom);
+    setOperationAction(ISD::FP_EXTEND,    MVT::f64, Custom);
   }
 
   computeRegisterProperties(Subtarget->getRegisterInfo());
 
   // ARM does not have floating-point extending loads.
   for (MVT VT : MVT::fp_valuetypes()) {
     setLoadExtAction(ISD::EXTLOAD, VT, MVT::f32, Expand);
     setLoadExtAction(ISD::EXTLOAD, VT, MVT::f16, Expand);
ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
 
   setOperationAction(ISD::BRCOND,    MVT::Other, Expand);
   setOperationAction(ISD::BR_CC,     MVT::i32,   Custom);
   setOperationAction(ISD::BR_CC,     MVT::f32,   Custom);
   setOperationAction(ISD::BR_CC,     MVT::f64,   Custom);
   setOperationAction(ISD::BR_JT,     MVT::Other, Custom);
 
   // We don't support sin/cos/fmod/copysign/pow
-  setOperationAction(ISD::FSIN,      MVT::f64, Expand);
-  setOperationAction(ISD::FSIN,      MVT::f32, Expand);
-  setOperationAction(ISD::FCOS,      MVT::f32, Expand);
-  setOperationAction(ISD::FCOS,      MVT::f64, Expand);
-  setOperationAction(ISD::FSINCOS,   MVT::f64, Expand);
-  setOperationAction(ISD::FSINCOS,   MVT::f32, Expand);
-  setOperationAction(ISD::FREM,      MVT::f64, Expand);
-  setOperationAction(ISD::FREM,      MVT::f32, Expand);
+  setOperationAction(ISD::FSIN,         MVT::f64, Expand);
+  setOperationAction(ISD::FSIN,         MVT::f32, Expand);
+  setOperationAction(ISD::FCOS,         MVT::f32, Expand);
+  setOperationAction(ISD::FCOS,         MVT::f64, Expand);
+  setOperationAction(ISD::FSINCOS,      MVT::f64, Expand);
+  setOperationAction(ISD::FSINCOS,      MVT::f32, Expand);
+  setOperationAction(ISD::FREM,         MVT::f64, Expand);
+  setOperationAction(ISD::FREM,         MVT::f32, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f64, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f32, Expand);
   if (!Subtarget->useSoftFloat() && Subtarget->hasVFP2() &&
       !Subtarget->isThumb1Only()) {
     setOperationAction(ISD::FCOPYSIGN, MVT::f64, Custom);
     setOperationAction(ISD::FCOPYSIGN, MVT::f32, Custom);
   }
   setOperationAction(ISD::FPOW,      MVT::f64, Expand);
   setOperationAction(ISD::FPOW,      MVT::f32, Expand);
 

lib/Target/ARM/ARMInstrNEON.td

 
 //   VADD     : Vector Add (integer and floating-point)
 defm VADD     : N3V_QHSD<0, 0, 0b1000, 0, IIC_VBINiD, IIC_VBINiQ, "vadd", "i",
                          add, 1>;
 def  VADDfd   : N3VD<0, 0, 0b00, 0b1101, 0, IIC_VBIND, "vadd", "f32",
                      v2f32, v2f32, fadd, 1>;
 def  VADDfq   : N3VQ<0, 0, 0b00, 0b1101, 0, IIC_VBINQ, "vadd", "f32",
                      v4f32, v4f32, fadd, 1>;
+let isCodeGenOnly = 1 in {
+def  VADDfd_FPE : N3VD<0, 0, 0b00, 0b1101, 0, IIC_VBIND, "vadd", "f32",
+                       v2f32, v2f32, faddwchain, 1>;
+def  VADDfq_FPE : N3VQ<0, 0, 0b00, 0b1101, 0, IIC_VBINQ, "vadd", "f32",
+                       v4f32, v4f32, faddwchain, 1>;
+}
 //   VADDL    : Vector Add Long (Q = D + D)
 defm VADDLs   : N3VLExt_QHS<0,1,0b0000,0, IIC_VSHLiD, IIC_VSHLiD,
                             "vaddl", "s", add, sext, 1>;
 defm VADDLu   : N3VLExt_QHS<1,1,0b0000,0, IIC_VSHLiD, IIC_VSHLiD,
                             "vaddl", "u", add, zext, 1>;
 //   VADDW    : Vector Add Wide (Q = Q + D)
 defm VADDWs   : N3VW_QHS<0,1,0b0001,0, "vaddw", "s", add, sext, 0>;
 defm VADDWu   : N3VW_QHS<1,1,0b0001,0, "vaddw", "u", add, zext, 0>;

 def  VMULpd   : N3VDInt<1, 0, 0b00, 0b1001, 1, N3RegFrm, IIC_VMULi16D, "vmul",
                         "p8", v8i8, v8i8, int_arm_neon_vmulp, 1>;
 def  VMULpq   : N3VQInt<1, 0, 0b00, 0b1001, 1, N3RegFrm, IIC_VMULi16Q, "vmul",
                         "p8", v16i8, v16i8, int_arm_neon_vmulp, 1>;
 def  VMULfd   : N3VD<1, 0, 0b00, 0b1101, 1, IIC_VFMULD, "vmul", "f32",
                      v2f32, v2f32, fmul, 1>;
 def  VMULfq   : N3VQ<1, 0, 0b00, 0b1101, 1, IIC_VFMULQ, "vmul", "f32",
                      v4f32, v4f32, fmul, 1>;
+let isCodeGenOnly = 1 in {
+def  VMULfd_FPE : N3VD<1, 0, 0b00, 0b1101, 1, IIC_VFMULD, "vmul",
+                       "f32", v2f32, v2f32, fmulwchain, 1>;
+def  VMULfq_FPE : N3VQ<1, 0, 0b00, 0b1101, 1, IIC_VFMULQ, "vmul",
+                       "f32", v4f32, v4f32, fmulwchain, 1>;
+}
 defm VMULsl   : N3VSL_HS<0b1000, "vmul", mul>;
 def  VMULslfd : N3VDSL<0b10, 0b1001, IIC_VBIND, "vmul", "f32", v2f32, fmul>;
 def  VMULslfq : N3VQSL<0b10, 0b1001, IIC_VBINQ, "vmul", "f32", v4f32,
                        v2f32, fmul>;
 
 def : Pat<(v8i16 (mul (v8i16 QPR:$src1),
                       (v8i16 (NEONvduplane (v8i16 QPR:$src2), imm:$lane)))),
           (v8i16 (VMULslv8i16 (v8i16 QPR:$src1),

 
 //   VSUB     : Vector Subtract (integer and floating-point)
 defm VSUB     : N3V_QHSD<1, 0, 0b1000, 0, IIC_VSUBiD, IIC_VSUBiQ,
                          "vsub", "i", sub, 0>;
 def  VSUBfd   : N3VD<0, 0, 0b10, 0b1101, 0, IIC_VBIND, "vsub", "f32",
                      v2f32, v2f32, fsub, 0>;
 def  VSUBfq   : N3VQ<0, 0, 0b10, 0b1101, 0, IIC_VBINQ, "vsub", "f32",
                      v4f32, v4f32, fsub, 0>;
+let isCodeGenOnly = 1 in {
+def  VSUBfd_FPE : N3VD<0, 0, 0b10, 0b1101, 0, IIC_VBIND, "vsub", "f32",
+                       v2f32, v2f32, fsubwchain, 0>;
+def  VSUBfq_FPE : N3VQ<0, 0, 0b10, 0b1101, 0, IIC_VBINQ, "vsub", "f32",
+                       v4f32, v4f32, fsubwchain, 0>;
+}
 //   VSUBL    : Vector Subtract Long (Q = D - D)
 defm VSUBLs   : N3VLExt_QHS<0,1,0b0010,0, IIC_VSHLiD, IIC_VSHLiD,
                             "vsubl", "s", sub, sext, 0>;
 defm VSUBLu   : N3VLExt_QHS<1,1,0b0010,0, IIC_VSHLiD, IIC_VSHLiD,
                             "vsubl", "u", sub, zext, 0>;
 //   VSUBW    : Vector Subtract Wide (Q = Q - D)
 defm VSUBWs   : N3VW_QHS<0,1,0b0011,0, "vsubw", "s", sub, sext, 0>;
 defm VSUBWu   : N3VW_QHS<1,1,0b0011,0, "vsubw", "u", sub, zext, 0>;

 class NVCVTFIPat<SDNode OpNode, NeonI Inst>
   : NEONFPPat<(i32 (OpNode SPR:$a)),
               (i32 (EXTRACT_SUBREG
                      (v2f32 (Inst (INSERT_SUBREG (v2f32 (IMPLICIT_DEF)),
                                                  SPR:$a, ssub_0))),
                      ssub_0))>;
 
 def : N3VSPat<fadd, VADDfd>;
+def : N3VSPat<faddwchain, VADDfd_FPE>;
 def : N3VSPat<fsub, VSUBfd>;
+def : N3VSPat<fsubwchain, VSUBfd_FPE>;
 def : N3VSPat<fmul, VMULfd>;
+def : N3VSPat<fmulwchain, VMULfd_FPE>;
 def : N3VSMulOpPat<fmul, fadd, VMLAfd>,
       Requires<[HasNEON, UseNEONForFP, UseFPVMLx, DontUseFusedMAC]>;
 def : N3VSMulOpPat<fmul, fsub, VMLSfd>,
       Requires<[HasNEON, UseNEONForFP, UseFPVMLx, DontUseFusedMAC]>;
 def : N3VSMulOpPat<fmul, fadd, VFMAfd>,
       Requires<[HasVFP4, UseNEONForFP, UseFusedMAC]>;
 def : N3VSMulOpPat<fmul, fsub, VFMSfd>,
       Requires<[HasVFP4, UseNEONForFP, UseFusedMAC]>;

lib/Target/ARM/ARMInstrVFP.td

 def : VFP2MnemonicAlias<"fstmeax", "fstmiax">;
 def : VFP2MnemonicAlias<"fstmfdx", "fstmdbx">;
 
 //===----------------------------------------------------------------------===//
 // FP Binary Operations.
 //
 
 let TwoOperandAliasConstraint = "$Dn = $Dd" in
-def VADDD  : ADbI<0b11100, 0b11, 0, 0,
+class VADDD <SDNode OpCode> : ADbI<0b11100, 0b11, 0, 0,
                                    (outs DPR:$Dd), (ins DPR:$Dn, DPR:$Dm),
                                    IIC_fpALU64, "vadd", ".f64\t$Dd, $Dn, $Dm",
-                  [(set DPR:$Dd, (fadd DPR:$Dn, (f64 DPR:$Dm)))]>;
+                                   [(set DPR:$Dd,
+                                         (OpCode DPR:$Dn, (f64 DPR:$Dm)))]>;
 
 let TwoOperandAliasConstraint = "$Sn = $Sd" in
-def VADDS  : ASbIn<0b11100, 0b11, 0, 0,
+class VADDS <SDNode OpCode> : ASbIn<0b11100, 0b11, 0, 0,
                                     (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                                     IIC_fpALU32, "vadd", ".f32\t$Sd, $Sn, $Sm",
-                   [(set SPR:$Sd, (fadd SPR:$Sn, SPR:$Sm))]> {
+                                    [(set SPR:$Sd,
+                                          (OpCode SPR:$Sn, SPR:$Sm))]> {
   // Some single precision VFP instructions may be executed on both NEON and
   // VFP pipelines on A8.
   let D = VFPNeonA8Domain;
 }
 
 let TwoOperandAliasConstraint = "$Dn = $Dd" in
-def VSUBD  : ADbI<0b11100, 0b11, 1, 0,
+class VSUBD <SDNode OpCode> : ADbI<0b11100, 0b11, 1, 0,
                                    (outs DPR:$Dd), (ins DPR:$Dn, DPR:$Dm),
                                    IIC_fpALU64, "vsub", ".f64\t$Dd, $Dn, $Dm",
-                  [(set DPR:$Dd, (fsub DPR:$Dn, (f64 DPR:$Dm)))]>;
+                                   [(set DPR:$Dd,
+                                         (OpCode DPR:$Dn, (f64 DPR:$Dm)))]>;
 
 let TwoOperandAliasConstraint = "$Sn = $Sd" in
-def VSUBS  : ASbIn<0b11100, 0b11, 1, 0,
+class VSUBS <SDNode OpCode> : ASbIn<0b11100, 0b11, 1, 0,
                                     (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                                     IIC_fpALU32, "vsub", ".f32\t$Sd, $Sn, $Sm",
-                   [(set SPR:$Sd, (fsub SPR:$Sn, SPR:$Sm))]> {
+                                    [(set SPR:$Sd,
+                                          (OpCode SPR:$Sn, SPR:$Sm))]> {
   // Some single precision VFP instructions may be executed on both NEON and
   // VFP pipelines on A8.
   let D = VFPNeonA8Domain;
 }
 
 let TwoOperandAliasConstraint = "$Dn = $Dd" in
-def VDIVD  : ADbI<0b11101, 0b00, 0, 0,
+class VDIVD <SDNode OpCode> : ADbI<0b11101, 0b00, 0, 0,
                                    (outs DPR:$Dd), (ins DPR:$Dn, DPR:$Dm),
                                    IIC_fpDIV64, "vdiv", ".f64\t$Dd, $Dn, $Dm",
-                  [(set DPR:$Dd, (fdiv DPR:$Dn, (f64 DPR:$Dm)))]>;
+                                   [(set DPR:$Dd,
+                                         (OpCode DPR:$Dn, (f64 DPR:$Dm)))]>;
 
 let TwoOperandAliasConstraint = "$Sn = $Sd" in
-def VDIVS  : ASbI<0b11101, 0b00, 0, 0,
+class VDIVS <SDNode OpCode> : ASbI<0b11101, 0b00, 0, 0,
                                    (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                                    IIC_fpDIV32, "vdiv", ".f32\t$Sd, $Sn, $Sm",
-                  [(set SPR:$Sd, (fdiv SPR:$Sn, SPR:$Sm))]>;
+                                   [(set SPR:$Sd, (OpCode SPR:$Sn, SPR:$Sm))]>;
 
 let TwoOperandAliasConstraint = "$Dn = $Dd" in
-def VMULD  : ADbI<0b11100, 0b10, 0, 0,
+class VMULD <SDNode OpCode> : ADbI<0b11100, 0b10, 0, 0,
                                    (outs DPR:$Dd), (ins DPR:$Dn, DPR:$Dm),
                                    IIC_fpMUL64, "vmul", ".f64\t$Dd, $Dn, $Dm",
-                  [(set DPR:$Dd, (fmul DPR:$Dn, (f64 DPR:$Dm)))]>;
+                                   [(set DPR:$Dd,
+                                         (OpCode DPR:$Dn, (f64 DPR:$Dm)))]>;
 
 let TwoOperandAliasConstraint = "$Sn = $Sd" in
-def VMULS  : ASbIn<0b11100, 0b10, 0, 0,
+class VMULS <SDNode OpCode> : ASbIn<0b11100, 0b10, 0, 0,
                                     (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                                     IIC_fpMUL32, "vmul", ".f32\t$Sd, $Sn, $Sm",
-                   [(set SPR:$Sd, (fmul SPR:$Sn, SPR:$Sm))]> {
+                                    [(set SPR:$Sd,
+                                          (OpCode SPR:$Sn, SPR:$Sm))]> {
   // Some single precision VFP instructions may be executed on both NEON and
   // VFP pipelines on A8.
   let D = VFPNeonA8Domain;
 }
 
-def VNMULD : ADbI<0b11100, 0b10, 1, 0,
+class VNMULD <SDNode OpCode> : ADbI<0b11100, 0b10, 1, 0,
                                     (outs DPR:$Dd), (ins DPR:$Dn, DPR:$Dm),
                                     IIC_fpMUL64, "vnmul", ".f64\t$Dd, $Dn, $Dm",
-                  [(set DPR:$Dd, (fneg (fmul DPR:$Dn, (f64 DPR:$Dm))))]>;
+                                    [(set DPR:$Dd,
+                                          (fneg (OpCode DPR:$Dn,
+                                                (f64 DPR:$Dm))))]>;
 
-def VNMULS : ASbI<0b11100, 0b10, 1, 0,
+class VNMULS <SDNode OpCode> : ASbI<0b11100, 0b10, 1, 0,
                                     (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                                     IIC_fpMUL32, "vnmul", ".f32\t$Sd, $Sn, $Sm",
-                  [(set SPR:$Sd, (fneg (fmul SPR:$Sn, SPR:$Sm)))]> {
+                                    [(set SPR:$Sd,
+                                          (fneg (OpCode SPR:$Sn, SPR:$Sm)))]> {
   // Some single precision VFP instructions may be executed on both NEON and
   // VFP pipelines on A8.
   let D = VFPNeonA8Domain;
 }
 
+def VADDD : VADDD<fadd>;
+def VADDS : VADDS<fadd>;
+def VSUBD : VSUBD<fsub>;
+def VSUBS : VSUBS<fsub>;
+def VDIVD : VDIVD<fdiv>;
+def VDIVS : VDIVS<fdiv>;
+def VMULD : VMULD<fmul>;
+def VMULS : VMULS<fmul>;
+def VNMULD : VNMULD<fmul>;
+def VNMULS : VNMULS<fmul>;
+
+let isCodeGenOnly = 1 in {
+def VADDD_FPE : VADDD<faddwchain>;
+def VADDS_FPE : VADDS<faddwchain>;
+def VSUBD_FPE : VSUBD<fsubwchain>;
+def VSUBS_FPE : VSUBS<fsubwchain>;
+def VDIVD_FPE : VDIVD<fdivwchain>;
+def VDIVS_FPE : VDIVS<fdivwchain>;
+def VMULD_FPE : VMULD<fmulwchain>;
+def VMULS_FPE : VMULS<fmulwchain>;
+def VNMULD_FPE : VNMULD<fmulwchain>;
+def VNMULS_FPE : VNMULS<fmulwchain>;
+}
+
 multiclass vsel_inst<string op, bits<2> opc, int CC> {
   let DecoderNamespace = "VFPV8", PostEncoderMethod = "",
       Uses = [CPSR], AddedComplexity = 4 in {
     def S : ASbInp<0b11100, opc, 0,
                    (outs SPR:$Sd), (ins SPR:$Sn, SPR:$Sm),
                    NoItinerary, !strconcat("vsel", op, ".f32\t$Sd, $Sn, $Sm"),
                    [(set SPR:$Sd, (ARMcmov SPR:$Sm, SPR:$Sn, CC))]>,
                    Requires<[HasFPARMv8]>;

 // Be friendly and accept the old form of zero-compare
 def : VFP2DPInstAlias<"fcmpzd${p} $val", (VCMPZD DPR:$val, pred:$p)>;
 def : VFP2InstAlias<"fcmpzs${p} $val", (VCMPZS SPR:$val, pred:$p)>;
 
 
 def : VFP2InstAlias<"fmstat${p}", (FMSTAT pred:$p)>;
 def : VFP2InstAlias<"fadds${p} $Sd, $Sn, $Sm",
                     (VADDS SPR:$Sd, SPR:$Sn, SPR:$Sm, pred:$p)>;
+def : VFP2InstAlias<"fadds${p} $Sd, $Sn, $Sm",
+                    (VADDS_FPE SPR:$Sd, SPR:$Sn, SPR:$Sm, pred:$p)>;
 def : VFP2DPInstAlias<"faddd${p} $Dd, $Dn, $Dm",
                       (VADDD DPR:$Dd, DPR:$Dn, DPR:$Dm, pred:$p)>;
+def : VFP2DPInstAlias<"faddd${p} $Dd, $Dn, $Dm",
+                      (VADDD_FPE DPR:$Dd, DPR:$Dn, DPR:$Dm, pred:$p)>;
 def : VFP2InstAlias<"fsubs${p} $Sd, $Sn, $Sm",
                     (VSUBS SPR:$Sd, SPR:$Sn, SPR:$Sm, pred:$p)>;
+def : VFP2InstAlias<"fsubs${p} $Sd, $Sn, $Sm",
+                    (VSUBS_FPE SPR:$Sd, SPR:$Sn, SPR:$Sm, pred:$p)>;
 def : VFP2DPInstAlias<"fsubd${p} $Dd, $Dn, $Dm",
                       (VSUBD DPR:$Dd, DPR:$Dn, DPR:$Dm, pred:$p)>;
+def : VFP2DPInstAlias<"fsubd${p} $Dd, $Dn, $Dm",
+                      (VSUBD_FPE DPR:$Dd, DPR:$Dn, DPR:$Dm, pred:$p)>;
 
 // No need for the size suffix on VSQRT. It's implied by the register classes.
 def : VFP2InstAlias<"vsqrt${p} $Sd, $Sm", (VSQRTS SPR:$Sd, SPR:$Sm, pred:$p)>;
 def : VFP2DPInstAlias<"vsqrt${p} $Dd, $Dm", (VSQRTD DPR:$Dd, DPR:$Dm, pred:$p)>;
 
 // VLDR/VSTR accept an optional type suffix.
 def : VFP2InstAlias<"vldr${p}.32 $Sd, $addr",
                     (VLDRS SPR:$Sd, addrmode5:$addr, pred:$p)>;

lib/Target/Hexagon/HexagonISelLowering.cpp

for (unsigned IntExpOp :
          ISD::BSWAP,     ISD::SHL_PARTS, ISD::SRA_PARTS, ISD::SRL_PARTS,
          ISD::SMUL_LOHI, ISD::UMUL_LOHI }) {
     setOperationAction(IntExpOp, MVT::i32, Expand);
     setOperationAction(IntExpOp, MVT::i64, Expand);
   }
 
   for (unsigned FPExpOp :
        {ISD::FDIV, ISD::FREM, ISD::FSQRT, ISD::FSIN, ISD::FCOS, ISD::FSINCOS,
-        ISD::FPOW, ISD::FCOPYSIGN}) {
+        ISD::FPOW, ISD::FCOPYSIGN, ISD::FDIV_W_CHAIN, ISD::FREM_W_CHAIN}) {
     setOperationAction(FPExpOp, MVT::f32, Expand);
     setOperationAction(FPExpOp, MVT::f64, Expand);
   }
 
   // No extending loads from i32.
   for (MVT VT : MVT::integer_valuetypes()) {
     setLoadExtAction(ISD::ZEXTLOAD, VT, MVT::i32, Expand);
     setLoadExtAction(ISD::SEXTLOAD, VT, MVT::i32, Expand);
static const unsigned VectExpOps[] = {
     ISD::CTTZ_ZERO_UNDEF,
     // Floating point arithmetic/math functions:
     ISD::FADD,    ISD::FSUB,    ISD::FMUL,    ISD::FMA,     ISD::FDIV,
     ISD::FREM,    ISD::FNEG,    ISD::FABS,    ISD::FSQRT,   ISD::FSIN,
     ISD::FCOS,    ISD::FPOWI,   ISD::FPOW,    ISD::FLOG,    ISD::FLOG2,
     ISD::FLOG10,  ISD::FEXP,    ISD::FEXP2,   ISD::FCEIL,   ISD::FTRUNC,
     ISD::FRINT,   ISD::FNEARBYINT,            ISD::FROUND,  ISD::FFLOOR,
     ISD::FMINNUM, ISD::FMAXNUM, ISD::FSINCOS,
+
+    ISD::FADD_W_CHAIN,          ISD::FSUB_W_CHAIN,          ISD::FMUL_W_CHAIN,
+    ISD::FDIV_W_CHAIN,          ISD::FREM_W_CHAIN,
     // Misc:
     ISD::SELECT,  ISD::ConstantPool,
     // Vector:
     ISD::BUILD_VECTOR,          ISD::SCALAR_TO_VECTOR,
     ISD::EXTRACT_VECTOR_ELT,    ISD::INSERT_VECTOR_ELT,
     ISD::EXTRACT_SUBVECTOR,     ISD::INSERT_SUBVECTOR,
     ISD::CONCAT_VECTORS,        ISD::VECTOR_SHUFFLE
   };
if (UseHVXSgl) {
       setOperationAction(ISD::CONCAT_VECTORS, MVT::v32i64,  Custom);
     } else {
       llvm_unreachable("Unrecognized HVX mode");
     }
   }
   // Subtarget-specific operation actions.
   //
   if (Subtarget.hasV5TOps()) {
-    setOperationAction(ISD::FMA,  MVT::f64, Expand);
-    setOperationAction(ISD::FADD, MVT::f64, Expand);
-    setOperationAction(ISD::FSUB, MVT::f64, Expand);
-    setOperationAction(ISD::FMUL, MVT::f64, Expand);
+    setOperationAction(ISD::FMA,          MVT::f64, Expand);
+    setOperationAction(ISD::FADD,         MVT::f64, Expand);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FSUB,         MVT::f64, Expand);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::f64, Expand);
+    setOperationAction(ISD::FMUL,         MVT::f64, Expand);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::f64, Expand);
 
     setOperationAction(ISD::FP_TO_UINT, MVT::i1,  Promote);
     setOperationAction(ISD::FP_TO_UINT, MVT::i8,  Promote);
     setOperationAction(ISD::FP_TO_UINT, MVT::i16, Promote);
     setOperationAction(ISD::FP_TO_SINT, MVT::i1,  Promote);
     setOperationAction(ISD::FP_TO_SINT, MVT::i8,  Promote);
     setOperationAction(ISD::FP_TO_SINT, MVT::i16, Promote);
     setOperationAction(ISD::UINT_TO_FP, MVT::i1,  Promote);
if (Subtarget.hasV5TOps()) {
 
     setOperationAction(ISD::CTPOP, MVT::i8,  Expand);
     setOperationAction(ISD::CTPOP, MVT::i16, Expand);
     setOperationAction(ISD::CTPOP, MVT::i32, Expand);
     setOperationAction(ISD::CTPOP, MVT::i64, Expand);
 
     // Expand these operations for both f32 and f64:
     for (unsigned FPExpOpV4 :
-         {ISD::FADD, ISD::FSUB, ISD::FMUL, ISD::FABS, ISD::FNEG, ISD::FMA}) {
+         {ISD::FADD, ISD::FSUB, ISD::FMUL, ISD::FABS, ISD::FNEG, ISD::FMA,
+          ISD::FADD_W_CHAIN, ISD::FSUB_W_CHAIN, ISD::FMUL_W_CHAIN}) {
       setOperationAction(FPExpOpV4, MVT::f32, Expand);
       setOperationAction(FPExpOpV4, MVT::f64, Expand);
     }
 
     for (ISD::CondCode FPExpCCV4 :
          {ISD::SETOEQ, ISD::SETOGT, ISD::SETOLT, ISD::SETOGE, ISD::SETOLE,
           ISD::SETUO,  ISD::SETO}) {
       setCondCodeAction(FPExpCCV4, MVT::f32, Expand);

lib/Target/Hexagon/HexagonInstrInfoV5.td

Requires<[HasV5T]> {
     let Inst{12-8} = Rt;
     let Inst{7-5} = MinOp;
     let Inst{4-0} = Rd;
   }
 
 let isCommutable = 1 in {
   def F2_sfadd : T_MInstFloat < "sfadd", 0b000, 0b000>;
   def F2_sfmpy : T_MInstFloat < "sfmpy", 0b010, 0b000>;
+
+  let hasSideEffects = 1, isCodeGenOnly = 1 in {
+    def F2_sfaddwchain : T_MInstFloat < "sfadd", 0b000, 0b000>;
+    def F2_sfmpywchain : T_MInstFloat < "sfmpy", 0b010, 0b000>;
+  }
 }
 
 def F2_sfsub : T_MInstFloat < "sfsub", 0b000, 0b001>;
+let hasSideEffects = 1, isCodeGenOnly = 1 in
+  def F2_sfsubwchain : T_MInstFloat < "sfsub", 0b000, 0b001>;
 
 def: Pat<(f32 (fadd F32:$src1, F32:$src2)),
          (F2_sfadd F32:$src1, F32:$src2)>;
 
 def: Pat<(f32 (fsub F32:$src1, F32:$src2)),
          (F2_sfsub F32:$src1, F32:$src2)>;
 
 def: Pat<(f32 (fmul F32:$src1, F32:$src2)),
          (F2_sfmpy F32:$src1, F32:$src2)>;
 
+def: Pat<(f32 (faddwchain F32:$src1, F32:$src2)),
+         (F2_sfaddwchain F32:$src1, F32:$src2)>;
+
+def: Pat<(f32 (fsubwchain F32:$src1, F32:$src2)),
+         (F2_sfsubwchain F32:$src1, F32:$src2)>;
+
+def: Pat<(f32 (fmulwchain F32:$src1, F32:$src2)),
+         (F2_sfmpywchain F32:$src1, F32:$src2)>;
+
 let Itinerary = M_tc_3x_SLOT23 in {
   def F2_sfmax : T_MInstFloat < "sfmax", 0b100, 0b000>;
   def F2_sfmin : T_MInstFloat < "sfmin", 0b100, 0b001>;
 }
 
 let AddedComplexity = 100, Predicates = [HasV5T] in {
   def: Pat<(f32 (select (i1 (setolt F32:$src1, F32:$src2)),
                         F32:$src1, F32:$src2)),

lib/Target/Mips/MipsISelLowering.cpp

MipsTargetLowering::MipsTargetLowering(const MipsTargetMachine &TM,
   setOperationAction(ISD::FLOG,              MVT::f32,   Expand);
   setOperationAction(ISD::FLOG2,             MVT::f32,   Expand);
   setOperationAction(ISD::FLOG10,            MVT::f32,   Expand);
   setOperationAction(ISD::FEXP,              MVT::f32,   Expand);
   setOperationAction(ISD::FMA,               MVT::f32,   Expand);
   setOperationAction(ISD::FMA,               MVT::f64,   Expand);
   setOperationAction(ISD::FREM,              MVT::f32,   Expand);
   setOperationAction(ISD::FREM,              MVT::f64,   Expand);
+  setOperationAction(ISD::FREM_W_CHAIN,      MVT::f32,   Expand);
+  setOperationAction(ISD::FREM_W_CHAIN,      MVT::f64,   Expand);
 
   // Lower f16 conversion operations into library calls
   setOperationAction(ISD::FP16_TO_FP,        MVT::f32,   Expand);
   setOperationAction(ISD::FP_TO_FP16,        MVT::f32,   Expand);
   setOperationAction(ISD::FP16_TO_FP,        MVT::f64,   Expand);
   setOperationAction(ISD::FP_TO_FP16,        MVT::f64,   Expand);
 
   setOperationAction(ISD::EH_RETURN, MVT::Other, Custom);

lib/Target/Mips/MipsInstrFPU.td

 defm FDIV :  ADDS_M<"div.d", II_DIV_D, 0, fdiv>, ADDS_FM<0x03, 17>;
 def FMUL_S : MMRel, ADDS_FT<"mul.s", FGR32Opnd, II_MUL_S, 1, fmul>,
              ADDS_FM<0x02, 16>;
 defm FMUL :  ADDS_M<"mul.d", II_MUL_D, 1, fmul>, ADDS_FM<0x02, 17>;
 def FSUB_S : MMRel, ADDS_FT<"sub.s", FGR32Opnd, II_SUB_S, 0, fsub>,
              ADDS_FM<0x01, 16>;
 defm FSUB :  ADDS_M<"sub.d", II_SUB_D, 0, fsub>, ADDS_FM<0x01, 17>;
 
+let isCodeGenOnly = 1 in {
+  def FADD_S_FPE : MMRel, ADDS_FT<"add.s", FGR32Opnd, II_ADD_S, 1,
+                                     faddwchain>;
+  defm FADD_FPE :  ADDS_M<"add.d", II_ADD_D, 1, faddwchain>;
+
+  def FDIV_S_FPE : MMRel, ADDS_FT<"div.s", FGR32Opnd, II_DIV_S, 0,
+                                     fdivwchain>;
+  defm FDIV_FPE :  ADDS_M<"div.d", II_DIV_D, 0, fdivwchain>;
+
+  def FMUL_S_FPE : MMRel, ADDS_FT<"mul.s", FGR32Opnd, II_MUL_S, 1,
+                                     fmulwchain>;
+  defm FMUL_FPE :  ADDS_M<"mul.d", II_MUL_D, 1, fmulwchain>;
+
+  def FSUB_S_FPE : MMRel, ADDS_FT<"sub.s", FGR32Opnd, II_SUB_S, 0,
+                                     fsubwchain>;
+  defm FSUB_FPE :  ADDS_M<"sub.d", II_SUB_D, 0, fsubwchain>;
+}
+
 def MADD_S : MMRel, MADDS_FT<"madd.s", FGR32Opnd, II_MADD_S, fadd>,
              MADDS_FM<4, 0>, INSN_MIPS4_32R2_NOT_32R6_64R6;
 def MSUB_S : MMRel, MADDS_FT<"msub.s", FGR32Opnd, II_MSUB_S, fsub>,
              MADDS_FM<5, 0>, INSN_MIPS4_32R2_NOT_32R6_64R6;
 
+let isCodeGenOnly = 1 in {
+  def MADD_S_FPE : MADDS_FT<"madd.s", FGR32Opnd, II_MADD_S, faddwchain>;
+  def MSUB_S_FPE : MMRel, MADDS_FT<"msub.s", FGR32Opnd, II_MSUB_S, fsubwchain>;
+}
+
 let AdditionalPredicates = [NoNaNsFPMath] in {
   def NMADD_S : MMRel, NMADDS_FT<"nmadd.s", FGR32Opnd, II_NMADD_S, fadd>,
                 MADDS_FM<6, 0>, INSN_MIPS4_32R2_NOT_32R6_64R6;
   def NMSUB_S : MMRel, NMADDS_FT<"nmsub.s", FGR32Opnd, II_NMSUB_S, fsub>,
                 MADDS_FM<7, 0>, INSN_MIPS4_32R2_NOT_32R6_64R6;
 }
 
 def MADD_D32 : MMRel, MADDS_FT<"madd.d", AFGR64Opnd, II_MADD_D, fadd>,

lib/Target/NVPTX/NVPTXInstrInfo.td

if (&(N->getValueAPF().getSemantics()) != &llvm::APFloat::IEEEdouble)
       return false;
     double d = (double)N->getValueAPF().convertToDouble();
     return (d==1.0);
 }]>;
 
 defm FADD : F3<"add", fadd>;
 defm FSUB : F3<"sub", fsub>;
 defm FMUL : F3<"mul", fmul>;
+defm FADD_FPE : F3<"add", faddwchain>;
+defm FSUB_FPE : F3<"sub", fsubwchain>;
+defm FMUL_FPE : F3<"mul", fmulwchain>;
 
 defm FADD_rn : F3_rn<"add", fadd>;
 defm FSUB_rn : F3_rn<"sub", fsub>;
 defm FMUL_rn : F3_rn<"mul", fmul>;
+defm FADD_rn_FPE : F3_rn<"add", faddwchain>;
+defm FSUB_rn_FPE : F3_rn<"sub", fsubwchain>;
+defm FMUL_rn_FPE : F3_rn<"mul", fmulwchain>;
 
 defm FABS : F2<"abs", fabs>;
 defm FNEG : F2<"neg", fneg>;
 defm FSQRT : F2<"sqrt.rn", fsqrt>;
 
 //
 // F64 division
 //
-def FDIV641r : NVPTXInst<(outs Float64Regs:$dst),
-                      (ins f64imm:$a, Float64Regs:$b),
-                      "rcp.rn.f64 \t$dst, $b;",
-                      [(set Float64Regs:$dst,
-                        (fdiv DoubleConst1:$a, Float64Regs:$b))]>;
-def FDIV64rr : NVPTXInst<(outs Float64Regs:$dst),
-                      (ins Float64Regs:$a, Float64Regs:$b),
-                      "div.rn.f64 \t$dst, $a, $b;",
-                      [(set Float64Regs:$dst,
-                        (fdiv Float64Regs:$a, Float64Regs:$b))]>;
-def FDIV64ri : NVPTXInst<(outs Float64Regs:$dst),
-                      (ins Float64Regs:$a, f64imm:$b),
-                      "div.rn.f64 \t$dst, $a, $b;",
-                      [(set Float64Regs:$dst,
-                        (fdiv Float64Regs:$a, fpimm:$b))]>;
+multiclass FDIV64<SDNode OpNode> {
+  def 1r : NVPTXInst<(outs Float64Regs:$dst),
+                       (OpNode DoubleConst1:$a, Float64Regs:$b))]>;
+  def rr : NVPTXInst<(outs Float64Regs:$dst),
+                       (OpNode Float64Regs:$a, Float64Regs:$b))]>;
+  def ri : NVPTXInst<(outs Float64Regs:$dst),
+                       (OpNode Float64Regs:$a, fpimm:$b))]>;
+}
+defm FDIV64 : FDIV64<fdiv>;
+defm : FDIV64<fdivwchain>;
 
 //
 // F32 Approximate reciprocal
 //
 def FDIV321r_ftz : NVPTXInst<(outs Float32Regs:$dst),
                       (ins f32imm:$a, Float32Regs:$b),
                       "rcp.approx.ftz.f32 \t$dst, $b;",
                       [(set Float32Regs:$dst,
def FDIV321r_prec : NVPTXInst<(outs Float32Regs:$dst),
                       (ins f32imm:$a, Float32Regs:$b),
                        "rcp.rn.f32 \t$dst, $b;",
                       [(set Float32Regs:$dst,
                         (fdiv FloatConst1:$a, Float32Regs:$b))]>,
                       Requires<[reqPTX20]>;
 //
 // F32 Accurate division
 //
-def FDIV32rr_prec_ftz : NVPTXInst<(outs Float32Regs:$dst),
+multiclass FDIV32<SDNode OpNode> {
+  def rr_prec_ftz : NVPTXInst<(outs Float32Regs:$dst),
                         (ins Float32Regs:$a, Float32Regs:$b),
                         "div.rn.ftz.f32 \t$dst, $a, $b;",
                         [(set Float32Regs:$dst,
-                        (fdiv Float32Regs:$a, Float32Regs:$b))]>,
+                          (OpNode Float32Regs:$a, Float32Regs:$b))]>,
                         Requires<[doF32FTZ, reqPTX20]>;
-def FDIV32ri_prec_ftz : NVPTXInst<(outs Float32Regs:$dst),
+  def ri_prec_ftz : NVPTXInst<(outs Float32Regs:$dst),
                         (ins Float32Regs:$a, f32imm:$b),
                         "div.rn.ftz.f32 \t$dst, $a, $b;",
                         [(set Float32Regs:$dst,
-                        (fdiv Float32Regs:$a, fpimm:$b))]>,
+                          (OpNode Float32Regs:$a, fpimm:$b))]>,
                         Requires<[doF32FTZ, reqPTX20]>;
-def FDIV32rr_prec : NVPTXInst<(outs Float32Regs:$dst),
+  def rr_prec : NVPTXInst<(outs Float32Regs:$dst),
                         (ins Float32Regs:$a, Float32Regs:$b),
                         "div.rn.f32 \t$dst, $a, $b;",
                         [(set Float32Regs:$dst,
-                        (fdiv Float32Regs:$a, Float32Regs:$b))]>,
+                          (OpNode Float32Regs:$a, Float32Regs:$b))]>,
                         Requires<[reqPTX20]>;
-def FDIV32ri_prec : NVPTXInst<(outs Float32Regs:$dst),
+  def ri_prec : NVPTXInst<(outs Float32Regs:$dst),
                         (ins Float32Regs:$a, f32imm:$b),
                         "div.rn.f32 \t$dst, $a, $b;",
                         [(set Float32Regs:$dst,
-                        (fdiv Float32Regs:$a, fpimm:$b))]>,
+                          (OpNode Float32Regs:$a, fpimm:$b))]>,
                         Requires<[reqPTX20]>;
+}
+defm FDIV32 : FDIV32<fdiv>;
+defm : FDIV32<fdivwchain>;
 
 //
 // F32 rsqrt
 //
 
 def RSQRTF32approx1r : NVPTXInst<(outs Float32Regs:$dst), (ins Float32Regs:$b),
                        "rsqrt.approx.f32 \t$dst, $b;", []>;
 

lib/Target/PowerPC/PPCISelLowering.cpp

PPCTargetLowering::PPCTargetLowering(const PPCTargetMachine &TM,
 
   // We do not currently implement these libm ops for PowerPC.
   setOperationAction(ISD::FFLOOR, MVT::ppcf128, Expand);
   setOperationAction(ISD::FCEIL,  MVT::ppcf128, Expand);
   setOperationAction(ISD::FTRUNC, MVT::ppcf128, Expand);
   setOperationAction(ISD::FRINT,  MVT::ppcf128, Expand);
   setOperationAction(ISD::FNEARBYINT, MVT::ppcf128, Expand);
   setOperationAction(ISD::FREM, MVT::ppcf128, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::ppcf128, Expand);
 
   // PowerPC has no SREM/UREM instructions
   setOperationAction(ISD::SREM, MVT::i32, Expand);
   setOperationAction(ISD::UREM, MVT::i32, Expand);
   setOperationAction(ISD::SREM, MVT::i64, Expand);
   setOperationAction(ISD::UREM, MVT::i64, Expand);
 
   // Don't use SMUL_LOHI/UMUL_LOHI or SDIVREM/UDIVREM to lower SREM/UREM.
   setOperationAction(ISD::UMUL_LOHI, MVT::i32, Expand);
   setOperationAction(ISD::SMUL_LOHI, MVT::i32, Expand);
   setOperationAction(ISD::UMUL_LOHI, MVT::i64, Expand);
   setOperationAction(ISD::SMUL_LOHI, MVT::i64, Expand);
   setOperationAction(ISD::UDIVREM, MVT::i32, Expand);
   setOperationAction(ISD::SDIVREM, MVT::i32, Expand);
   setOperationAction(ISD::UDIVREM, MVT::i64, Expand);
   setOperationAction(ISD::SDIVREM, MVT::i64, Expand);
 
   // We don't support sin/cos/sqrt/fmod/pow
   setOperationAction(ISD::FSIN , MVT::f64, Expand);
   setOperationAction(ISD::FCOS , MVT::f64, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f64, Expand);
   setOperationAction(ISD::FREM , MVT::f64, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f64, Expand);
   setOperationAction(ISD::FPOW , MVT::f64, Expand);
   setOperationAction(ISD::FMA  , MVT::f64, Legal);
   setOperationAction(ISD::FSIN , MVT::f32, Expand);
   setOperationAction(ISD::FCOS , MVT::f32, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f32, Expand);
   setOperationAction(ISD::FREM , MVT::f32, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f32, Expand);
   setOperationAction(ISD::FPOW , MVT::f32, Expand);
   setOperationAction(ISD::FMA  , MVT::f32, Legal);
 
   setOperationAction(ISD::FLT_ROUNDS_, MVT::i32, Custom);
 
   // If we're enabling GP optimizations, use hardware square root
   if (!Subtarget.hasFSQRT() &&
       !(TM.Options.UnsafeFPMath && Subtarget.hasFRSQRTE() &&
for (MVT VT : MVT::vector_valuetypes()) {
 
       // No other operations are legal.
       setOperationAction(ISD::MUL , VT, Expand);
       setOperationAction(ISD::SDIV, VT, Expand);
       setOperationAction(ISD::SREM, VT, Expand);
       setOperationAction(ISD::UDIV, VT, Expand);
       setOperationAction(ISD::UREM, VT, Expand);
       setOperationAction(ISD::FDIV, VT, Expand);
+      setOperationAction(ISD::FDIV_W_CHAIN, VT, Expand);
       setOperationAction(ISD::FREM, VT, Expand);
+      setOperationAction(ISD::FREM_W_CHAIN, VT, Expand);
       setOperationAction(ISD::FNEG, VT, Expand);
       setOperationAction(ISD::FSQRT, VT, Expand);
       setOperationAction(ISD::FLOG, VT, Expand);
       setOperationAction(ISD::FLOG10, VT, Expand);
       setOperationAction(ISD::FLOG2, VT, Expand);
       setOperationAction(ISD::FEXP, VT, Expand);
       setOperationAction(ISD::FEXP2, VT, Expand);
       setOperationAction(ISD::FSIN, VT, Expand);
if (Subtarget.hasAltivec()) {
     addRegisterClass(MVT::v8i16, &PPC::VRRCRegClass);
     addRegisterClass(MVT::v16i8, &PPC::VRRCRegClass);
 
     setOperationAction(ISD::MUL, MVT::v4f32, Legal);
     setOperationAction(ISD::FMA, MVT::v4f32, Legal);
 
     if (TM.Options.UnsafeFPMath || Subtarget.hasVSX()) {
       setOperationAction(ISD::FDIV, MVT::v4f32, Legal);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f32, Legal);
       setOperationAction(ISD::FSQRT, MVT::v4f32, Legal);
     }
 
     if (Subtarget.hasP8Altivec())
       setOperationAction(ISD::MUL, MVT::v4i32, Legal);
     else
       setOperationAction(ISD::MUL, MVT::v4i32, Custom);
 
if (Subtarget.hasVSX()) {
       setOperationAction(ISD::FROUND, MVT::v2f64, Legal);
 
       setOperationAction(ISD::FROUND, MVT::v4f32, Legal);
 
       setOperationAction(ISD::MUL, MVT::v2f64, Legal);
       setOperationAction(ISD::FMA, MVT::v2f64, Legal);
 
       setOperationAction(ISD::FDIV, MVT::v2f64, Legal);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v2f64, Legal);
       setOperationAction(ISD::FSQRT, MVT::v2f64, Legal);
 
       setOperationAction(ISD::VSELECT, MVT::v16i8, Legal);
       setOperationAction(ISD::VSELECT, MVT::v8i16, Legal);
       setOperationAction(ISD::VSELECT, MVT::v4i32, Legal);
       setOperationAction(ISD::VSELECT, MVT::v4f32, Legal);
       setOperationAction(ISD::VSELECT, MVT::v2f64, Legal);
 
if (Subtarget.hasAltivec()) {
 
     if (Subtarget.hasP8Altivec()) {
       addRegisterClass(MVT::v2i64, &PPC::VRRCRegClass);
       addRegisterClass(MVT::v1i128, &PPC::VRRCRegClass);
     }
   }
 
   if (Subtarget.hasQPX()) {
-    setOperationAction(ISD::FADD, MVT::v4f64, Legal);
-    setOperationAction(ISD::FSUB, MVT::v4f64, Legal);
-    setOperationAction(ISD::FMUL, MVT::v4f64, Legal);
-    setOperationAction(ISD::FREM, MVT::v4f64, Expand);
+    setOperationAction(ISD::FADD,         MVT::v4f64, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::v4f64, Legal);
+    setOperationAction(ISD::FSUB,         MVT::v4f64, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::v4f64, Legal);
+    setOperationAction(ISD::FMUL,         MVT::v4f64, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::v4f64, Legal);
+    setOperationAction(ISD::FREM,         MVT::v4f64, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, MVT::v4f64, Expand);
 
     setOperationAction(ISD::FCOPYSIGN, MVT::v4f64, Legal);
     setOperationAction(ISD::FGETSIGN, MVT::v4f64, Expand);
 
     setOperationAction(ISD::LOAD  , MVT::v4f64, Custom);
     setOperationAction(ISD::STORE , MVT::v4f64, Custom);
 
     setTruncStoreAction(MVT::v4f64, MVT::v4f32, Custom);
if (Subtarget.hasQPX()) {
     setOperationAction(ISD::FMINNUM, MVT::v4f64, Legal);
     setOperationAction(ISD::FMAXNUM, MVT::v4f64, Legal);
 
     setIndexedLoadAction(ISD::PRE_INC, MVT::v4f64, Legal);
     setIndexedStoreAction(ISD::PRE_INC, MVT::v4f64, Legal);
 
     addRegisterClass(MVT::v4f64, &PPC::QFRCRegClass);
 
-    setOperationAction(ISD::FADD, MVT::v4f32, Legal);
-    setOperationAction(ISD::FSUB, MVT::v4f32, Legal);
-    setOperationAction(ISD::FMUL, MVT::v4f32, Legal);
-    setOperationAction(ISD::FREM, MVT::v4f32, Expand);
+    setOperationAction(ISD::FADD,         MVT::v4f32, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::v4f32, Legal);
+    setOperationAction(ISD::FSUB,         MVT::v4f32, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::v4f32, Legal);
+    setOperationAction(ISD::FMUL,         MVT::v4f32, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::v4f32, Legal);
+    setOperationAction(ISD::FREM,         MVT::v4f32, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, MVT::v4f32, Expand);
 
     setOperationAction(ISD::FCOPYSIGN, MVT::v4f32, Legal);
     setOperationAction(ISD::FGETSIGN, MVT::v4f32, Expand);
 
     setOperationAction(ISD::LOAD  , MVT::v4f32, Custom);
     setOperationAction(ISD::STORE , MVT::v4f32, Custom);
 
     if (!Subtarget.useCRBits())
if (Subtarget.hasQPX()) {
     setOperationAction(ISD::FNEARBYINT, MVT::v4f32, Expand);
 
     // These need to set FE_INEXACT, and so cannot be vectorized here.
     setOperationAction(ISD::FRINT, MVT::v4f64, Expand);
     setOperationAction(ISD::FRINT, MVT::v4f32, Expand);
 
     if (TM.Options.UnsafeFPMath) {
       setOperationAction(ISD::FDIV, MVT::v4f64, Legal);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f64, Legal);
       setOperationAction(ISD::FSQRT, MVT::v4f64, Legal);
 
       setOperationAction(ISD::FDIV, MVT::v4f32, Legal);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f32, Legal);
       setOperationAction(ISD::FSQRT, MVT::v4f32, Legal);
     } else {
       setOperationAction(ISD::FDIV, MVT::v4f64, Expand);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f64, Expand);
       setOperationAction(ISD::FSQRT, MVT::v4f64, Expand);
 
       setOperationAction(ISD::FDIV, MVT::v4f32, Expand);
+      setOperationAction(ISD::FDIV_W_CHAIN, MVT::v4f32, Expand);
       setOperationAction(ISD::FSQRT, MVT::v4f32, Expand);
     }
   }
 
   if (Subtarget.has64BitSupport())
     setOperationAction(ISD::PREFETCH, MVT::Other, Legal);
 
   setOperationAction(ISD::READCYCLECOUNTER, MVT::i64, isPPC64 ? Legal : Custom);
if (Subtarget.useCRBits()) {
     setTargetDAGCombine(ISD::TRUNCATE);
     setTargetDAGCombine(ISD::SETCC);
     setTargetDAGCombine(ISD::SELECT_CC);
   }
 
   // Use reciprocal estimates.
   if (TM.Options.UnsafeFPMath) {
     setTargetDAGCombine(ISD::FDIV);
+    setTargetDAGCombine(ISD::FDIV_W_CHAIN);
     setTargetDAGCombine(ISD::FSQRT);
   }
 
   // Darwin long double math library functions have $LDBL128 appended.
   if (Subtarget.isDarwin()) {
     setLibcallName(RTLIB::COS_PPCF128, "cosl$LDBL128");
     setLibcallName(RTLIB::POW_PPCF128, "powl$LDBL128");
     setLibcallName(RTLIB::REM_PPCF128, "fmodl$LDBL128");

lib/Target/PowerPC/PPCInstrAltivec.td

 def STVXL : XForm_8<31, 487, (outs), (ins vrrc:$rS, memrr:$dst),
                    "stvxl $rS, $dst", IIC_LdStStore,
                    [(int_ppc_altivec_stvxl v4i32:$rS, xoaddr:$dst)]>;
 }
 
 let PPC970_Unit = 5 in {  // VALU Operations.
 // VA-Form instructions.  3-input AltiVec ops.
 let isCommutable = 1 in {
-def VMADDFP : VAForm_1<46, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vC, vrrc:$vB),
+class VMADDFP : VAForm_1<46, (outs vrrc:$vD),
+                         (ins vrrc:$vA, vrrc:$vC, vrrc:$vB),
                          "vmaddfp $vD, $vA, $vC, $vB", IIC_VecFP,
                          [(set v4f32:$vD,
                                (fma v4f32:$vA, v4f32:$vC, v4f32:$vB))]>;
+def VMADDFP : VMADDFP;
+let hasSideEffects = 1, isCodeGenOnly = 1 in
+def VMADDFP_FPE : VMADDFP;
 
 // FIXME: The fma+fneg pattern won't match because fneg is not legal.
 def VNMSUBFP: VAForm_1<47, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vC, vrrc:$vB),
                        "vnmsubfp $vD, $vA, $vC, $vB", IIC_VecFP,
                        [(set v4f32:$vD, (fneg (fma v4f32:$vA, v4f32:$vC,
                                                   (fneg v4f32:$vB))))]>;
 
 def VMHADDSHS  : VA1a_Int_Ty<32, "vmhaddshs", int_ppc_altivec_vmhaddshs, v8i16>;

 // Shuffles.
 def VSLDOI  : VAForm_2<44, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB, u5imm:$SH),
                        "vsldoi $vD, $vA, $vB, $SH", IIC_VecFP,
                        [(set v16i8:$vD, 
                          (vsldoi_shuffle:$SH v16i8:$vA, v16i8:$vB))]>;
 
 // VX-Form instructions.  AltiVec arithmetic ops.
 let isCommutable = 1 in {
-def VADDFP : VXForm_1<10, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
+class VADDFP<SDNode OpNode> : VXForm_1<10, (outs vrrc:$vD),
+                                       (ins vrrc:$vA, vrrc:$vB),
                                        "vaddfp $vD, $vA, $vB", IIC_VecFP,
-                      [(set v4f32:$vD, (fadd v4f32:$vA, v4f32:$vB))]>;
-                      
+                                       [(set v4f32:$vD,
+                                             (OpNode v4f32:$vA, v4f32:$vB))]>;
+def VADDFP : VADDFP<fadd>;
+let isCodeGenOnly = 1 in
+def VADDFP_FPE : VADDFP<faddwchain>;
 def VADDUBM : VXForm_1<0, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vaddubm $vD, $vA, $vB", IIC_VecGeneral,
                       [(set v16i8:$vD, (add v16i8:$vA, v16i8:$vB))]>;
 def VADDUHM : VXForm_1<64, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vadduhm $vD, $vA, $vB", IIC_VecGeneral,
                       [(set v8i16:$vD, (add v8i16:$vA, v8i16:$vB))]>;
 def VADDUWM : VXForm_1<128, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vadduwm $vD, $vA, $vB", IIC_VecGeneral,

 def VRFIM     : VX2_Int_SP<714, "vrfim",     int_ppc_altivec_vrfim>;
 def VRFIN     : VX2_Int_SP<522, "vrfin",     int_ppc_altivec_vrfin>;
 def VRFIP     : VX2_Int_SP<650, "vrfip",     int_ppc_altivec_vrfip>;
 def VRFIZ     : VX2_Int_SP<586, "vrfiz",     int_ppc_altivec_vrfiz>;
 def VRSQRTEFP : VX2_Int_SP<330, "vrsqrtefp", int_ppc_altivec_vrsqrtefp>;
 
 def VSUBCUW : VX1_Int_Ty<1408, "vsubcuw", int_ppc_altivec_vsubcuw, v4i32>;
 
-def VSUBFP  : VXForm_1<74, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
+class VSUBFP<SDNode OpNode> : VXForm_1<74, (outs vrrc:$vD),
+                                       (ins vrrc:$vA, vrrc:$vB),
                                        "vsubfp $vD, $vA, $vB", IIC_VecGeneral,
-                      [(set v4f32:$vD, (fsub v4f32:$vA, v4f32:$vB))]>;
+                                       [(set v4f32:$vD,
+                                             (OpNode v4f32:$vA, v4f32:$vB))]>;
+def VSUBFP : VSUBFP<fsub>;
+let isCodeGenOnly = 1 in
+def : VSUBFP<fsubwchain>;
 def VSUBUBM : VXForm_1<1024, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vsububm $vD, $vA, $vB", IIC_VecGeneral,
                       [(set v16i8:$vD, (sub v16i8:$vA, v16i8:$vB))]>;
 def VSUBUHM : VXForm_1<1088, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vsubuhm $vD, $vA, $vB", IIC_VecGeneral,
                       [(set v8i16:$vD, (sub v8i16:$vA, v8i16:$vB))]>;
 def VSUBUWM : VXForm_1<1152, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                       "vsubuwm $vD, $vA, $vB", IIC_VecGeneral,

 
 def : Pat<(vnot_ppc (or v4i32:$A, v4i32:$B)),
           (VNOR $A, $B)>;
 def : Pat<(and v4i32:$A, (vnot_ppc v4i32:$B)),
           (VANDC $A, $B)>;
 
 def : Pat<(fmul v4f32:$vA, v4f32:$vB),
           (VMADDFP $vA, $vB,
-             (v4i32 (VSLW (V_SETALLONES), (V_SETALLONES))))>; 
+             (v4i32 (VSLW (V_SETALLONES), (V_SETALLONES))))>;
+def : Pat<(fmulwchain v4f32:$vA, v4f32:$vB),
+          (VMADDFP_FPE $vA, $vB,
+             (v4i32 (VSLW (V_SETALLONES), (V_SETALLONES))))>;
 
 // Fused multiply add and multiply sub for packed float.  These are represented
 // separately from the real instructions above, for operations that must have
 // the additional precision, such as Newton-Rhapson (used by divide, sqrt)
 def : Pat<(PPCvmaddfp v4f32:$A, v4f32:$B, v4f32:$C),
           (VMADDFP $A, $B, $C)>;
 def : Pat<(PPCvnmsubfp v4f32:$A, v4f32:$B, v4f32:$C),
           (VNMSUBFP $A, $B, $C)>;

lib/Target/PowerPC/PPCInstrInfo.td

defm FSELD : AForm_1r<63, 23,
                       "fsel", "$FRT, $FRA, $FRC, $FRB", IIC_FPGeneral,
                       [(set f64:$FRT, (PPCfsel f64:$FRA, f64:$FRC, f64:$FRB))]>;
 defm FSELS : AForm_1r<63, 23,
                       (outs f4rc:$FRT), (ins f8rc:$FRA, f4rc:$FRC, f4rc:$FRB),
                       "fsel", "$FRT, $FRA, $FRC, $FRB", IIC_FPGeneral,
                       [(set f32:$FRT, (PPCfsel f64:$FRA, f32:$FRC, f32:$FRB))]>;
 let Uses = [RM] in {
   let isCommutable = 1 in {
-  defm FADD  : AForm_2r<63, 21,
+  multiclass FADD <SDNode OpCode, string asmbase> :
+               AForm_2r<63, 21,
                         (outs f8rc:$FRT), (ins f8rc:$FRA, f8rc:$FRB),
-                        "fadd", "$FRT, $FRA, $FRB", IIC_FPAddSub,
-                        [(set f64:$FRT, (fadd f64:$FRA, f64:$FRB))]>;
-  defm FADDS : AForm_2r<59, 21,
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPAddSub,
+                        [(set f64:$FRT, (OpCode f64:$FRA, f64:$FRB))]>;
+  multiclass FADDS <SDNode OpCode, string asmbase> :
+               AForm_2r<59, 21,
                         (outs f4rc:$FRT), (ins f4rc:$FRA, f4rc:$FRB),
-                        "fadds", "$FRT, $FRA, $FRB", IIC_FPGeneral,
-                        [(set f32:$FRT, (fadd f32:$FRA, f32:$FRB))]>;
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPGeneral,
+                        [(set f32:$FRT, (OpCode f32:$FRA, f32:$FRB))]>;
+
+  defm FADD  : FADD<fadd, "fadd">;
+  defm FADDS : FADDS<fadd, "fadds">;
+
+  let isCodeGenOnly = 1, hasSideEffects = 1 in {
+  defm FADD_FPE  : FADD<faddwchain, "fadd_FPE">;
+  defm FADDS_FPE : FADDS<faddwchain, "fadds_FPE">;
+  }
+
   } // isCommutable
-  defm FDIV  : AForm_2r<63, 18,
+  multiclass FDIV <SDNode OpCode, string asmbase> :
+               AForm_2r<63, 18,
                         (outs f8rc:$FRT), (ins f8rc:$FRA, f8rc:$FRB),
-                        "fdiv", "$FRT, $FRA, $FRB", IIC_FPDivD,
-                        [(set f64:$FRT, (fdiv f64:$FRA, f64:$FRB))]>;
-  defm FDIVS : AForm_2r<59, 18,
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPDivD,
+                        [(set f64:$FRT, (OpCode f64:$FRA, f64:$FRB))]>;
+  multiclass FDIVS <SDNode OpCode, string asmbase> :
+               AForm_2r<59, 18,
                         (outs f4rc:$FRT), (ins f4rc:$FRA, f4rc:$FRB),
-                        "fdivs", "$FRT, $FRA, $FRB", IIC_FPDivS,
-                        [(set f32:$FRT, (fdiv f32:$FRA, f32:$FRB))]>;
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPDivS,
+                        [(set f32:$FRT, (OpCode f32:$FRA, f32:$FRB))]>;
+
+  defm FDIV  : FDIV<fdiv, "fdiv">;
+  defm FDIVS : FDIVS<fdiv, "fdivs">;
+
+  let isCodeGenOnly = 1, hasSideEffects = 1 in {
+  defm FDIV_FPE  : FDIV<fdivwchain, "fdiv_FPE">;
+  defm FDIVS_FPE : FDIVS<fdivwchain, "fdivs_FPE">;
+  }
+
   let isCommutable = 1 in {
-  defm FMUL  : AForm_3r<63, 25,
+  multiclass FMUL <SDNode OpCode, string asmbase> :
+               AForm_3r<63, 25,
                         (outs f8rc:$FRT), (ins f8rc:$FRA, f8rc:$FRC),
-                        "fmul", "$FRT, $FRA, $FRC", IIC_FPFused,
-                        [(set f64:$FRT, (fmul f64:$FRA, f64:$FRC))]>;
-  defm FMULS : AForm_3r<59, 25,
+                        asmbase, "$FRT, $FRA, $FRC", IIC_FPFused,
+                        [(set f64:$FRT, (OpCode f64:$FRA, f64:$FRC))]>;
+  multiclass FMULS <SDNode OpCode, string asmbase> :
+               AForm_3r<59, 25,
                         (outs f4rc:$FRT), (ins f4rc:$FRA, f4rc:$FRC),
-                        "fmuls", "$FRT, $FRA, $FRC", IIC_FPGeneral,
-                        [(set f32:$FRT, (fmul f32:$FRA, f32:$FRC))]>;
+                        asmbase, "$FRT, $FRA, $FRC", IIC_FPGeneral,
+                        [(set f32:$FRT, (OpCode f32:$FRA, f32:$FRC))]>;
+
+  defm FMUL  : FMUL<fmul, "fmul">;
+  defm FMULS : FMULS<fmul, "fmuls">;
+
+  let isCodeGenOnly = 1, hasSideEffects = 1 in {
+  defm FMUL_FPE  : FMUL<fmulwchain, "fmul_FPE">;
+  defm FMULS_FPE : FMULS<fmulwchain, "fmuls_FPE">;
+  }
+
   } // isCommutable
-  defm FSUB  : AForm_2r<63, 20,
+  multiclass FSUB <SDNode OpCode, string asmbase> :
+               AForm_2r<63, 20,
                         (outs f8rc:$FRT), (ins f8rc:$FRA, f8rc:$FRB),
-                        "fsub", "$FRT, $FRA, $FRB", IIC_FPAddSub,
-                        [(set f64:$FRT, (fsub f64:$FRA, f64:$FRB))]>;
-  defm FSUBS : AForm_2r<59, 20,
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPAddSub,
+                        [(set f64:$FRT, (OpCode f64:$FRA, f64:$FRB))]>;
+  multiclass FSUBS <SDNode OpCode, string asmbase> :
+               AForm_2r<59, 20,
                         (outs f4rc:$FRT), (ins f4rc:$FRA, f4rc:$FRB),
-                        "fsubs", "$FRT, $FRA, $FRB", IIC_FPGeneral,
-                        [(set f32:$FRT, (fsub f32:$FRA, f32:$FRB))]>;
+                        asmbase, "$FRT, $FRA, $FRB", IIC_FPGeneral,
+                        [(set f32:$FRT, (OpCode f32:$FRA, f32:$FRB))]>;
+
+  defm FSUB  : FSUB<fsub, "fsub">;
+  defm FSUBS : FSUBS<fsub, "fsubs">;
+
+  let isCodeGenOnly = 1, hasSideEffects = 1 in {
+  defm FSUB_FPE  : FSUB<fsubwchain, "fsub_FPE">;
+  defm FSUBS_FPE : FSUBS<fsubwchain, "fsubs_FPE">;
+  }
+
   }
 }
 
 let hasSideEffects = 0 in {
 let PPC970_Unit = 1 in {  // FXU Operations.
   let isSelect = 1 in
   def ISEL  : AForm_4<31, 15,
                      (outs gprc:$rT), (ins gprc_nor0:$rA, gprc:$rB, crbitrc:$cond),

lib/Target/PowerPC/PPCInstrQPX.td

let mayStore = 1 in {
     let RC = 1 in
     def QVSTFIWXA : XForm_8<31, 967,
                         (outs), (ins qfrc:$FRT, memrr:$dst),
                         "qvstfiwxa $FRT, $dst", IIC_LdStSTFD, []>;
   }
 }
 
 } // neverHasSideEffects
+
+let isCodeGenOnly = 1 in {
+  // Add Instructions
+  let isCommutable = 1 in {
+    def QVFADD_FPE : AForm_2<4, 21,
+                             (outs qfrc:$FRT), (ins qfrc:$FRA, qfrc:$FRB),
+                             "qvfadd $FRT, $FRA, $FRB", IIC_FPGeneral,
+                             [(set v4f64:$FRT,
+                                   (faddwchain v4f64:$FRA, v4f64:$FRB))]>;
+    def QVFADDSs_FPE : AForm_2<0, 21,
+                               (outs qsrc:$FRT), (ins qsrc:$FRA, qsrc:$FRB),
+                               "qvfadds $FRT, $FRA, $FRB", IIC_FPGeneral,
+                               [(set v4f32:$FRT,
+                                     (faddwchain v4f32:$FRA, v4f32:$FRB))]>;
+  }
+  def QVFSUB_FPE : AForm_2<4, 20,
+                           (outs qfrc:$FRT), (ins qfrc:$FRA, qfrc:$FRB),
+                           "qvfsub $FRT, $FRA, $FRB", IIC_FPGeneral,
+                           [(set v4f64:$FRT,
+                                 (fsubwchain v4f64:$FRA, v4f64:$FRB))]>;
+  def QVFSUBSs_FPE : AForm_2<0, 20,
+                             (outs qsrc:$FRT), (ins qsrc:$FRA, qsrc:$FRB),
+                             "qvfsubs $FRT, $FRA, $FRB", IIC_FPGeneral,
+                             [(set v4f32:$FRT,
+                                   (fsubwchain v4f32:$FRA, v4f32:$FRB))]>;
+  // Multiply Instructions
+  let isCommutable = 1 in {
+    def QVFMUL_FPE : AForm_3<4, 25,
+                                (outs qfrc:$FRT), (ins qfrc:$FRA, qfrc:$FRC),
+                                "qvfmul $FRT, $FRA, $FRC", IIC_FPGeneral,
+                                [(set v4f64:$FRT,
+                                      (fmulwchain v4f64:$FRA, v4f64:$FRC))]>;
+    def QVFMULSs_FPE : AForm_3<0, 25,
+                               (outs qsrc:$FRT), (ins qsrc:$FRA, qsrc:$FRC),
+                               "qvfmuls $FRT, $FRA, $FRC", IIC_FPGeneral,
+                               [(set v4f32:$FRT,
+                                     (fmulwchain v4f32:$FRA, v4f32:$FRC))]>;
+  }
+}
+
 }
 
 def : InstAlias<"qvfclr $FRT",
                 (QVFLOGICALb qbrc:$FRT, qbrc:$FRT, qbrc:$FRT, 0)>;
 def : InstAlias<"qvfand $FRT, $FRA, $FRB",
                 (QVFLOGICALb qbrc:$FRT, qbrc:$FRA, qbrc:$FRB, 1)>;
 def : InstAlias<"qvfandc $FRT, $FRA, $FRB",
                 (QVFLOGICALb qbrc:$FRT, qbrc:$FRA, qbrc:$FRB, 4)>;

lib/Target/PowerPC/PPCInstrVSX.td

let Uses = [RM] in {
   def XXSLDWI : XX3Form_2<60, 2,
                        (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB, u2imm:$SHW),
                        "xxsldwi $XT, $XA, $XB, $SHW", IIC_VecPerm, []>;
   def XXSPLTW : XX2Form_2<60, 164,
                        (outs vsrc:$XT), (ins vsrc:$XB, u2imm:$UIM),
                        "xxspltw $XT, $XB, $UIM", IIC_VecPerm, []>;
 } // hasSideEffects
 
+// Add/Mul Instructions
+let isCommutable = 1 in {
+  let isCodeGenOnly = 1 in {
+    def XSADDDP_FPE : XX3Form<60, 32,
+                              (outs vsfrc:$XT), (ins vsfrc:$XA, vsfrc:$XB),
+                              "xsadddp $XT, $XA, $XB", IIC_VecFP,
+                              [(set f64:$XT, (faddwchain f64:$XA, f64:$XB))]>;
+    def XSMULDP_FPE : XX3Form<60, 48,
+                              (outs vsfrc:$XT), (ins vsfrc:$XA, vsfrc:$XB),
+                              "xsmuldp $XT, $XA, $XB", IIC_VecFP,
+                              [(set f64:$XT, (fmulwchain f64:$XA, f64:$XB))]>;
+
+    def XVADDDP_FPE : XX3Form<60, 96,
+                              (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                              "xvadddp $XT, $XA, $XB", IIC_VecFP,
+                              [(set v2f64:$XT,
+                                    (faddwchain v2f64:$XA, v2f64:$XB))]>;
+
+    def XVADDSP_FPE : XX3Form<60, 64,
+                              (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                              "xvaddsp $XT, $XA, $XB", IIC_VecFP,
+                              [(set v4f32:$XT,
+                                    (faddwchain v4f32:$XA, v4f32:$XB))]>;
+
+    def XVMULDP_FPE : XX3Form<60, 112,
+                              (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                              "xvmuldp $XT, $XA, $XB", IIC_VecFP,
+                              [(set v2f64:$XT,
+                                    (fmulwchain v2f64:$XA, v2f64:$XB))]>;
+
+    def XVMULSP_FPE : XX3Form<60, 80,
+                              (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                              "xvmulsp $XT, $XA, $XB", IIC_VecFP,
+                              [(set v4f32:$XT,
+                                    (fmulwchain v4f32:$XA, v4f32:$XB))]>;
+  }
+}
+
+// Subtract Instructions
+let isCodeGenOnly = 1 in {
+  def XSSUBDP_FPE : XX3Form<60, 40,
+                            (outs vsfrc:$XT), (ins vsfrc:$XA, vsfrc:$XB),
+                            "xssubdp $XT, $XA, $XB", IIC_VecFP,
+                            [(set f64:$XT, (fsubwchain f64:$XA, f64:$XB))]>;
+
+  def XVSUBDP_FPE : XX3Form<60, 104,
+                            (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                            "xvsubdp $XT, $XA, $XB", IIC_VecFP,
+                            [(set v2f64:$XT,
+                                  (fsubwchain v2f64:$XA, v2f64:$XB))]>;
+  def XVSUBSP_FPE : XX3Form<60, 72,
+                            (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                            "xvsubsp $XT, $XA, $XB", IIC_VecFP,
+                            [(set v4f32:$XT,
+                                  (fsubwchain v4f32:$XA, v4f32:$XB))]>;
+}
+
+// Division Instructions
+let isCodeGenOnly = 1 in {
+  def XSDIVDP_FPE : XX3Form<60, 56,
+                            (outs vsfrc:$XT), (ins vsfrc:$XA, vsfrc:$XB),
+                            "xsdivdp $XT, $XA, $XB", IIC_FPDivD,
+                            [(set f64:$XT, (fdivwchain f64:$XA, f64:$XB))]>;
+
+  def XVDIVDP_FPE : XX3Form<60, 120,
+                            (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                            "xvdivdp $XT, $XA, $XB", IIC_FPDivD,
+                            [(set v2f64:$XT,
+                                  (fdivwchain v2f64:$XA, v2f64:$XB))]>;
+  def XVDIVSP_FPE : XX3Form<60, 88,
+                            (outs vsrc:$XT), (ins vsrc:$XA, vsrc:$XB),
+                            "xvdivsp $XT, $XA, $XB", IIC_FPDivS,
+                            [(set v4f32:$XT,
+                                  (fdivwchain v4f32:$XA, v4f32:$XB))]>;
+}
+
 // SELECT_CC_* - Used to implement the SELECT_CC DAG operation.  Expanded after
 // instruction selection into a branch sequence.
 let usesCustomInserter = 1,    // Expanded after instruction selection.
     PPC970_Single = 1 in {
 
   def SELECT_CC_VSRC: Pseudo<(outs vsrc:$dst),
                              (ins crrc:$cond, vsrc:$T, vsrc:$F, i32imm:$BROPC),
                              "#SELECT_CC_VSRC",
let isCommutable = 1 in {
     def XSADDSP : XX3Form<60, 0,
                           (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
                           "xsaddsp $XT, $XA, $XB", IIC_VecFP,
                           [(set f32:$XT, (fadd f32:$XA, f32:$XB))]>;
     def XSMULSP : XX3Form<60, 16,
                           (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
                           "xsmulsp $XT, $XA, $XB", IIC_VecFP,
                           [(set f32:$XT, (fmul f32:$XA, f32:$XB))]>;
+
+    let isCodeGenOnly = 1 in {
+      def XSADDSP_FPE : XX3Form<60, 0,
+                                (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
+                                "xsaddsp $XT, $XA, $XB", IIC_VecFP,
+                                [(set f32:$XT,
+                                      (faddwchain f32:$XA, f32:$XB))]>;
+      def XSMULSP_FPE : XX3Form<60, 16,
+                                (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
+                                "xsmulsp $XT, $XA, $XB", IIC_VecFP,
+                                [(set f32:$XT,
+                                      (fmulwchain f32:$XA, f32:$XB))]>;
+    }
   } // isCommutable
 
   def XSDIVSP : XX3Form<60, 24,
                         (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
                         "xsdivsp $XT, $XA, $XB", IIC_FPDivS,
                         [(set f32:$XT, (fdiv f32:$XA, f32:$XB))]>;
   def XSRESP : XX2Form<60, 26,
                         (outs vssrc:$XT), (ins vssrc:$XB),
                         "xsresp $XT, $XB", IIC_VecFP,
                         [(set f32:$XT, (PPCfre f32:$XB))]>;
   def XSSQRTSP : XX2Form<60, 11,
                         (outs vssrc:$XT), (ins vssrc:$XB),
                         "xssqrtsp $XT, $XB", IIC_FPSqrtS,
                         [(set f32:$XT, (fsqrt f32:$XB))]>;
   def XSRSQRTESP : XX2Form<60, 10,
                            (outs vssrc:$XT), (ins vssrc:$XB),
                            "xsrsqrtesp $XT, $XB", IIC_VecFP,
                            [(set f32:$XT, (PPCfrsqrte f32:$XB))]>;
   def XSSUBSP : XX3Form<60, 8,
                         (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
                         "xssubsp $XT, $XA, $XB", IIC_VecFP,
                         [(set f32:$XT, (fsub f32:$XA, f32:$XB))]>;
 
+  let isCodeGenOnly = 1 in {
+    def XSDIVSP_FPE : XX3Form<60, 24,
+                              (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
+                              "xsdivsp $XT, $XA, $XB", IIC_FPDivS,
+                              [(set f32:$XT, (fdivwchain f32:$XA, f32:$XB))]>;
+    def XSSUBSP_FPE : XX3Form<60, 8,
+                              (outs vssrc:$XT), (ins vssrc:$XA, vssrc:$XB),
+                              "xssubsp $XT, $XA, $XB", IIC_VecFP,
+                              [(set f32:$XT, (fsubwchain f32:$XA, f32:$XB))]>;
+  }
+
   // FMA Instructions
   let BaseName = "XSMADDASP" in {
   let isCommutable = 1 in
   def XSMADDASP : XX3Form<60, 1,
                           (outs vssrc:$XT),
                           (ins vssrc:$XTi, vssrc:$XA, vssrc:$XB),
                           "xsmaddasp $XT, $XA, $XB", IIC_VecFP,
                           [(set f32:$XT, (fma f32:$XA, f32:$XB, f32:$XTi))]>,

lib/Target/Sparc/SparcISelLowering.cpp

if (!Subtarget->isV9()) {
     setOperationAction(ISD::FNEG, MVT::f64, Custom);
     setOperationAction(ISD::FABS, MVT::f64, Custom);
   }
 
   setOperationAction(ISD::FSIN , MVT::f128, Expand);
   setOperationAction(ISD::FCOS , MVT::f128, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f128, Expand);
   setOperationAction(ISD::FREM , MVT::f128, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f128, Expand);
   setOperationAction(ISD::FMA  , MVT::f128, Expand);
   setOperationAction(ISD::FSIN , MVT::f64, Expand);
   setOperationAction(ISD::FCOS , MVT::f64, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f64, Expand);
   setOperationAction(ISD::FREM , MVT::f64, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f64, Expand);
   setOperationAction(ISD::FMA  , MVT::f64, Expand);
   setOperationAction(ISD::FSIN , MVT::f32, Expand);
   setOperationAction(ISD::FCOS , MVT::f32, Expand);
   setOperationAction(ISD::FSINCOS, MVT::f32, Expand);
   setOperationAction(ISD::FREM , MVT::f32, Expand);
+  setOperationAction(ISD::FREM_W_CHAIN, MVT::f32, Expand);
   setOperationAction(ISD::FMA  , MVT::f32, Expand);
   setOperationAction(ISD::CTTZ , MVT::i32, Expand);
   setOperationAction(ISD::CTTZ_ZERO_UNDEF, MVT::i32, Expand);
   setOperationAction(ISD::CTLZ , MVT::i32, Expand);
   setOperationAction(ISD::CTLZ_ZERO_UNDEF, MVT::i32, Expand);
   setOperationAction(ISD::ROTL , MVT::i32, Expand);
   setOperationAction(ISD::ROTR , MVT::i32, Expand);
   setOperationAction(ISD::BSWAP, MVT::i32, Expand);
if (Subtarget->isV9() && Subtarget->hasHardQuad()) {
     setOperationAction(ISD::LOAD, MVT::f128, Legal);
     setOperationAction(ISD::STORE, MVT::f128, Legal);
   } else {
     setOperationAction(ISD::LOAD, MVT::f128, Custom);
     setOperationAction(ISD::STORE, MVT::f128, Custom);
   }
 
   if (Subtarget->hasHardQuad()) {
-    setOperationAction(ISD::FADD,  MVT::f128, Legal);
-    setOperationAction(ISD::FSUB,  MVT::f128, Legal);
-    setOperationAction(ISD::FMUL,  MVT::f128, Legal);
-    setOperationAction(ISD::FDIV,  MVT::f128, Legal);
-    setOperationAction(ISD::FSQRT, MVT::f128, Legal);
-    setOperationAction(ISD::FP_EXTEND, MVT::f128, Legal);
-    setOperationAction(ISD::FP_ROUND,  MVT::f64, Legal);
+    setOperationAction(ISD::FADD,         MVT::f128, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::f128, Legal);
+    setOperationAction(ISD::FSUB,         MVT::f128, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::f128, Legal);
+    setOperationAction(ISD::FMUL,         MVT::f128, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::f128, Legal);
+    setOperationAction(ISD::FDIV,         MVT::f128, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::f128, Legal);
+    setOperationAction(ISD::FSQRT,        MVT::f128, Legal);
+    setOperationAction(ISD::FP_EXTEND,    MVT::f128, Legal);
+    setOperationAction(ISD::FP_ROUND,     MVT::f64,  Legal);
     if (Subtarget->isV9()) {
       setOperationAction(ISD::FNEG, MVT::f128, Legal);
       setOperationAction(ISD::FABS, MVT::f128, Legal);
     } else {
       setOperationAction(ISD::FNEG, MVT::f128, Custom);
       setOperationAction(ISD::FABS, MVT::f128, Custom);
     }
 
     if (!Subtarget->is64Bit()) {
       setLibcallName(RTLIB::FPTOSINT_F128_I64, "_Q_qtoll");
       setLibcallName(RTLIB::FPTOUINT_F128_I64, "_Q_qtoull");
       setLibcallName(RTLIB::SINTTOFP_I64_F128, "_Q_lltoq");
       setLibcallName(RTLIB::UINTTOFP_I64_F128, "_Q_ulltoq");
     }
 
   } else {
     // Custom legalize f128 operations.
 
-    setOperationAction(ISD::FADD,  MVT::f128, Custom);
-    setOperationAction(ISD::FSUB,  MVT::f128, Custom);
-    setOperationAction(ISD::FMUL,  MVT::f128, Custom);
-    setOperationAction(ISD::FDIV,  MVT::f128, Custom);
-    setOperationAction(ISD::FSQRT, MVT::f128, Custom);
-    setOperationAction(ISD::FNEG,  MVT::f128, Custom);
-    setOperationAction(ISD::FABS,  MVT::f128, Custom);
+    setOperationAction(ISD::FADD,         MVT::f128, Custom);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::f128, Custom);
+    setOperationAction(ISD::FSUB,         MVT::f128, Custom);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::f128, Custom);
+    setOperationAction(ISD::FMUL,         MVT::f128, Custom);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::f128, Custom);
+    setOperationAction(ISD::FDIV,         MVT::f128, Custom);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::f128, Custom);
+    setOperationAction(ISD::FSQRT,        MVT::f128, Custom);
+    setOperationAction(ISD::FNEG,         MVT::f128, Custom);
+    setOperationAction(ISD::FABS,         MVT::f128, Custom);
 
     setOperationAction(ISD::FP_EXTEND, MVT::f128, Custom);
     setOperationAction(ISD::FP_ROUND,  MVT::f64, Custom);
     setOperationAction(ISD::FP_ROUND,  MVT::f32, Custom);
 
     // Setup Runtime library names.
     if (Subtarget->is64Bit()) {
       setLibcallName(RTLIB::ADD_F128,  "_Qp_add");
SparcTargetLowering::LowerF128Op(SDValue Op, SelectionDAG &DAG,
   ArgListTy Args;
 
   MachineFrameInfo *MFI = DAG.getMachineFunction().getFrameInfo();
   auto PtrVT = getPointerTy(DAG.getDataLayout());
 
   SDValue Callee = DAG.getExternalSymbol(LibFuncName, PtrVT);
   Type *RetTy = Op.getValueType().getTypeForEVT(*DAG.getContext());
   Type *RetTyABI = RetTy;
-  SDValue Chain = DAG.getEntryNode();
+  bool hasChain = isFPOpWithChain(Op->getOpcode());
+  SDValue Chain = (hasChain ? Op.getOperand(0) : DAG.getEntryNode());
   SDValue RetPtr;
 
   if (RetTy->isFP128Ty()) {
     // Create a Stack Object to receive the return value of type f128.
     ArgListEntry Entry;
     int RetFI = MFI->CreateStackObject(16, 8, false);
     RetPtr = DAG.getFrameIndex(RetFI, PtrVT);
     Entry.Node = RetPtr;
     Entry.Ty   = PointerType::getUnqual(RetTy);
     if (!Subtarget->is64Bit())
       Entry.isSRet = true;
     Entry.isReturned = false;
     Args.push_back(Entry);
     RetTyABI = Type::getVoidTy(*DAG.getContext());
   }
 
   assert(Op->getNumOperands() >= numArgs && "Not enough operands!");
-  for (unsigned i = 0, e = numArgs; i != e; ++i) {
+  for (unsigned i = hasChain ? 1 : 0, e = numArgs; i != e; ++i) {
     Chain = LowerF128_LibCallArg(Chain, Args, Op.getOperand(i), SDLoc(Op), DAG);
   }
   TargetLowering::CallLoweringInfo CLI(DAG);
   CLI.setDebugLoc(SDLoc(Op)).setChain(Chain)
     .setCallee(CallingConv::C, RetTyABI, Callee, std::move(Args), 0);
 
   std::pair<SDValue, SDValue> CallInfo = LowerCallTo(CLI);
 
LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::VAARG:              return LowerVAARG(Op, DAG);
   case ISD::DYNAMIC_STACKALLOC: return LowerDYNAMIC_STACKALLOC(Op, DAG,
                                                                Subtarget);
 
   case ISD::LOAD:               return LowerLOAD(Op, DAG);
   case ISD::STORE:              return LowerSTORE(Op, DAG);
   case ISD::FADD:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::ADD_F128), 2);
+  case ISD::FADD_W_CHAIN:       return LowerF128Op(Op, DAG,
+                                       getLibcallName(RTLIB::ADD_F128), 3);
   case ISD::FSUB:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::SUB_F128), 2);
+  case ISD::FSUB_W_CHAIN:       return LowerF128Op(Op, DAG,
+                                       getLibcallName(RTLIB::SUB_F128), 3);
   case ISD::FMUL:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::MUL_F128), 2);
+  case ISD::FMUL_W_CHAIN:       return LowerF128Op(Op, DAG,
+                                       getLibcallName(RTLIB::MUL_F128), 3);
   case ISD::FDIV:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::DIV_F128), 2);
+  case ISD::FDIV_W_CHAIN:       return LowerF128Op(Op, DAG,
+                                       getLibcallName(RTLIB::DIV_F128), 3);
   case ISD::FSQRT:              return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::SQRT_F128),1);
   case ISD::FABS:
   case ISD::FNEG:               return LowerFNEGorFABS(Op, DAG, isV9);
   case ISD::FP_EXTEND:          return LowerF128_FPEXTEND(Op, DAG, *this);
   case ISD::FP_ROUND:           return LowerF128_FPROUND(Op, DAG, *this);
   case ISD::ADDC:
   case ISD::ADDE:

lib/Target/Sparc/SparcInstrInfo.td

def FSQRTQ : F3_3u<2, 0b110100, 0b000101011,
                   (outs QFPRegs:$rd), (ins QFPRegs:$rs2),
                   "fsqrtq $rs2, $rd",
                   [(set f128:$rd, (fsqrt f128:$rs2))]>,
                   Requires<[HasHardQuad]>;
 
 
 
 // Floating-point Add and Subtract Instructions, p. 146
-def FADDS  : F3_3<2, 0b110100, 0b001000001,
-                  (outs FPRegs:$rd), (ins FPRegs:$rs1, FPRegs:$rs2),
-                  "fadds $rs1, $rs2, $rd",
-                  [(set f32:$rd, (fadd f32:$rs1, f32:$rs2))]>;
-def FADDD  : F3_3<2, 0b110100, 0b001000010,
-                  (outs DFPRegs:$rd), (ins DFPRegs:$rs1, DFPRegs:$rs2),
-                  "faddd $rs1, $rs2, $rd",
-                  [(set f64:$rd, (fadd f64:$rs1, f64:$rs2))]>;
-def FADDQ  : F3_3<2, 0b110100, 0b001000011,
-                  (outs QFPRegs:$rd), (ins QFPRegs:$rs1, QFPRegs:$rs2),
-                  "faddq $rs1, $rs2, $rd",
-                  [(set f128:$rd, (fadd f128:$rs1, f128:$rs2))]>,
-                  Requires<[HasHardQuad]>;
+multiclass FADD<SDNode OpNode> {
+  def S : F3_3<2, 0b110100, 0b001000001,
+               [(set f32:$rd, (OpNode f32:$rs1, f32:$rs2))]>;
+  def D : F3_3<2, 0b110100, 0b001000010,
+               [(set f64:$rd, (OpNode f64:$rs1, f64:$rs2))]>;
+  def Q : F3_3<2, 0b110100, 0b001000011,
+               [(set f128:$rd, (OpNode f128:$rs1, f128:$rs2))]>,
+}
+defm FADD : FADD<fadd>;
+let isCodeGenOnly = 1 in
+  defm : FADD<faddwchain>;
 
-def FSUBS  : F3_3<2, 0b110100, 0b001000101,
-                  (outs FPRegs:$rd), (ins FPRegs:$rs1, FPRegs:$rs2),
-                  "fsubs $rs1, $rs2, $rd",
-                  [(set f32:$rd, (fsub f32:$rs1, f32:$rs2))]>;
-def FSUBD  : F3_3<2, 0b110100, 0b001000110,
-                  (outs DFPRegs:$rd), (ins DFPRegs:$rs1, DFPRegs:$rs2),
-                  "fsubd $rs1, $rs2, $rd",
-                  [(set f64:$rd, (fsub f64:$rs1, f64:$rs2))]>;
-def FSUBQ  : F3_3<2, 0b110100, 0b001000111,
-                  (outs QFPRegs:$rd), (ins QFPRegs:$rs1, QFPRegs:$rs2),
-                  "fsubq $rs1, $rs2, $rd",
-                  [(set f128:$rd, (fsub f128:$rs1, f128:$rs2))]>,
-                  Requires<[HasHardQuad]>;
+multiclass FSUB<SDNode OpNode> {
+  def S : F3_3<2, 0b110100, 0b001000101,
+               [(set f32:$rd, (OpNode f32:$rs1, f32:$rs2))]>;
+  def D : F3_3<2, 0b110100, 0b001000110,
+               [(set f64:$rd, (OpNode f64:$rs1, f64:$rs2))]>;
+  def Q : F3_3<2, 0b110100, 0b001000111,
+               [(set f128:$rd, (OpNode f128:$rs1, f128:$rs2))]>,
+}
+defm FSUB : FSUB<fsub>;
+let isCodeGenOnly = 1 in
+  defm : FSUB<fsubwchain>;
 
 
 // Floating-point Multiply and Divide Instructions, p. 147
-def FMULS  : F3_3<2, 0b110100, 0b001001001,
-                  (outs FPRegs:$rd), (ins FPRegs:$rs1, FPRegs:$rs2),
-                  "fmuls $rs1, $rs2, $rd",
-                  [(set f32:$rd, (fmul f32:$rs1, f32:$rs2))]>;
-def FMULD  : F3_3<2, 0b110100, 0b001001010,
-                  (outs DFPRegs:$rd), (ins DFPRegs:$rs1, DFPRegs:$rs2),
-                  "fmuld $rs1, $rs2, $rd",
-                  [(set f64:$rd, (fmul f64:$rs1, f64:$rs2))]>;
-def FMULQ  : F3_3<2, 0b110100, 0b001001011,
-                  (outs QFPRegs:$rd), (ins QFPRegs:$rs1, QFPRegs:$rs2),
-                  "fmulq $rs1, $rs2, $rd",
-                  [(set f128:$rd, (fmul f128:$rs1, f128:$rs2))]>,
-                  Requires<[HasHardQuad]>;
+multiclass FMUL<SDNode OpNode> {
+  def S : F3_3<2, 0b110100, 0b001001001,
+               [(set f32:$rd, (OpNode f32:$rs1, f32:$rs2))]>;
+  def D : F3_3<2, 0b110100, 0b001001010,
+               [(set f64:$rd, (OpNode f64:$rs1, f64:$rs2))]>;
+  def Q : F3_3<2, 0b110100, 0b001001011,
+               [(set f128:$rd, (OpNode f128:$rs1, f128:$rs2))]>,
+}
+defm FMUL : FMUL<fmul>;
+let isCodeGenOnly = 1 in
+  defm : FMUL<fmulwchain>;
 
-def FSMULD : F3_3<2, 0b110100, 0b001101001,
-                  (outs DFPRegs:$rd), (ins FPRegs:$rs1, FPRegs:$rs2),
+class FSMULD<SDNode OpNode> : F3_3<2, 0b110100, 0b001101001,
+                              (outs DFPRegs:$rd),
+                              (ins FPRegs:$rs1, FPRegs:$rs2),
                               "fsmuld $rs1, $rs2, $rd",
-                  [(set f64:$rd, (fmul (fextend f32:$rs1),
+                              [(set f64:$rd, (OpNode (fextend f32:$rs1),
                                                      (fextend f32:$rs2)))]>;
-def FDMULQ : F3_3<2, 0b110100, 0b001101110,
-                  (outs QFPRegs:$rd), (ins DFPRegs:$rs1, DFPRegs:$rs2),
+class FDMULQ<SDNode OpNode> : F3_3<2, 0b110100, 0b001101110,
+                              (outs QFPRegs:$rd),
+                              (ins DFPRegs:$rs1, DFPRegs:$rs2),
                               "fdmulq $rs1, $rs2, $rd",
-                  [(set f128:$rd, (fmul (fextend f64:$rs1),
+                              [(set f128:$rd, (OpNode (fextend f64:$rs1),
                                                       (fextend f64:$rs2)))]>,
                               Requires<[HasHardQuad]>;
+def FSMULD : FSMULD<fmul>;
+def FDMULQ : FDMULQ<fmul>;
+let isCodeGenOnly = 1 in {
+  def : FSMULD<fmulwchain>;
+  def : FDMULQ<fmulwchain>;
+}
 
-def FDIVS  : F3_3<2, 0b110100, 0b001001101,
-                 (outs FPRegs:$rd), (ins FPRegs:$rs1, FPRegs:$rs2),
-                 "fdivs $rs1, $rs2, $rd",
-                 [(set f32:$rd, (fdiv f32:$rs1, f32:$rs2))]>;
-def FDIVD  : F3_3<2, 0b110100, 0b001001110,
-                 (outs DFPRegs:$rd), (ins DFPRegs:$rs1, DFPRegs:$rs2),
-                 "fdivd $rs1, $rs2, $rd",
-                 [(set f64:$rd, (fdiv f64:$rs1, f64:$rs2))]>;
-def FDIVQ  : F3_3<2, 0b110100, 0b001001111,
-                 (outs QFPRegs:$rd), (ins QFPRegs:$rs1, QFPRegs:$rs2),
-                 "fdivq $rs1, $rs2, $rd",
-                 [(set f128:$rd, (fdiv f128:$rs1, f128:$rs2))]>,
-                 Requires<[HasHardQuad]>;
+multiclass FDIV<SDNode OpNode> {
+def S : F3_3<2, 0b110100, 0b001001101,
+             [(set f32:$rd, (OpNode f32:$rs1, f32:$rs2))]>;
+def D : F3_3<2, 0b110100, 0b001001110,
+             [(set f64:$rd, (OpNode f64:$rs1, f64:$rs2))]>;
+def Q : F3_3<2, 0b110100, 0b001001111,
+             [(set f128:$rd, (OpNode f128:$rs1, f128:$rs2))]>,
+}
+defm FDIV : FDIV<fdiv>;
+let isCodeGenOnly = 1 in
+  defm : FDIV<fdivwchain>;
 
 // Floating-point Compare Instructions, p. 148
 // Note: the 2nd template arg is different for these guys.
 // Note 2: the result of a FCMP is not available until the 2nd cycle
 // after the instr is retired, but there is no interlock in Sparc V8.
 // This behavior is modeled with a forced noop after the instruction in
 // DelaySlotFiller.
 

lib/Target/SystemZ/SystemZISelLowering.cpp

if (isTypeLegal(VT)) {
         setOperationAction(ISD::FROUND, VT, Legal);
       }
 
       // No special instructions for these.
       setOperationAction(ISD::FSIN, VT, Expand);
       setOperationAction(ISD::FCOS, VT, Expand);
       setOperationAction(ISD::FSINCOS, VT, Expand);
       setOperationAction(ISD::FREM, VT, Expand);
+      setOperationAction(ISD::FREM_W_CHAIN, VT, Expand);
       setOperationAction(ISD::FPOW, VT, Expand);
     }
   }
 
   // Handle floating-point vector types.
   if (Subtarget.hasVector()) {
     // Scalar-to-vector conversion is just a subreg.
     setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v4f32, Legal);
     setOperationAction(ISD::SCALAR_TO_VECTOR, MVT::v2f64, Legal);
 
     // Some insertions and extractions can be done directly but others
     // need to go via integers.
     setOperationAction(ISD::INSERT_VECTOR_ELT, MVT::v4f32, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT, MVT::v2f64, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v4f32, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v2f64, Custom);
 
     // These operations have direct equivalents.
     setOperationAction(ISD::FADD, MVT::v2f64, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN, MVT::v2f64, Legal);
     setOperationAction(ISD::FNEG, MVT::v2f64, Legal);
     setOperationAction(ISD::FSUB, MVT::v2f64, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN, MVT::v2f64, Legal);
     setOperationAction(ISD::FMUL, MVT::v2f64, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN, MVT::v2f64, Legal);
     setOperationAction(ISD::FMA, MVT::v2f64, Legal);
     setOperationAction(ISD::FDIV, MVT::v2f64, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN, MVT::v2f64, Legal);
     setOperationAction(ISD::FABS, MVT::v2f64, Legal);
     setOperationAction(ISD::FSQRT, MVT::v2f64, Legal);
     setOperationAction(ISD::FRINT, MVT::v2f64, Legal);
     setOperationAction(ISD::FNEARBYINT, MVT::v2f64, Legal);
     setOperationAction(ISD::FFLOOR, MVT::v2f64, Legal);
     setOperationAction(ISD::FCEIL, MVT::v2f64, Legal);
     setOperationAction(ISD::FTRUNC, MVT::v2f64, Legal);
     setOperationAction(ISD::FROUND, MVT::v2f64, Legal);

lib/Target/SystemZ/SystemZInstrFP.td

 //===----------------------------------------------------------------------===//
 
 // Addition.
 let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0xF in {
   let isCommutable = 1 in {
     def AEBR : BinaryRRE<"aeb", 0xB30A, fadd, FP32,  FP32>;
     def ADBR : BinaryRRE<"adb", 0xB31A, fadd, FP64,  FP64>;
     def AXBR : BinaryRRE<"axb", 0xB34A, fadd, FP128, FP128>;
+
+    let isCodeGenOnly = 1 in {
+      def AEBR_FPE : BinaryRRE<"aeb", 0xB30A, faddwchain, FP32, FP32>;
+      def ADBR_FPE : BinaryRRE<"adb", 0xB31A, faddwchain, FP64, FP64>;
+      def AXBR_FPE : BinaryRRE<"axb", 0xB34A, faddwchain, FP128, FP128>;
+    }
   }
   def AEB : BinaryRXE<"aeb", 0xED0A, fadd, FP32, load, 4>;
   def ADB : BinaryRXE<"adb", 0xED1A, fadd, FP64, load, 8>;
 }
 
 // Subtraction.
 let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0xF in {
   def SEBR : BinaryRRE<"seb", 0xB30B, fsub, FP32,  FP32>;
   def SDBR : BinaryRRE<"sdb", 0xB31B, fsub, FP64,  FP64>;
   def SXBR : BinaryRRE<"sxb", 0xB34B, fsub, FP128, FP128>;
 
+  let isCodeGenOnly = 1 in {
+    def SEBR_FPE : BinaryRRE<"seb", 0xB30B, fsubwchain, FP32,  FP32>;
+    def SDBR_FPE : BinaryRRE<"sdb", 0xB31B, fsubwchain, FP64,  FP64>;
+    def SXBR_FPE : BinaryRRE<"sxb", 0xB34B, fsubwchain, FP128, FP128>;
+  }
+
   def SEB : BinaryRXE<"seb",  0xED0B, fsub, FP32, load, 4>;
   def SDB : BinaryRXE<"sdb",  0xED1B, fsub, FP64, load, 8>;
 }
 
 // Multiplication.
 let isCommutable = 1 in {
   def MEEBR : BinaryRRE<"meeb", 0xB317, fmul, FP32,  FP32>;
   def MDBR  : BinaryRRE<"mdb",  0xB31C, fmul, FP64,  FP64>;
   def MXBR  : BinaryRRE<"mxb",  0xB34C, fmul, FP128, FP128>;
+
+  let isCodeGenOnly = 1 in {
+    def MEEBR_FPE : BinaryRRE<"meeb", 0xB317, fmulwchain, FP32, FP32>;
+    def MDBR_FPE  : BinaryRRE<"mdb",  0xB31C, fmulwchain, FP64, FP64>;
+    def MXBR_FPE  : BinaryRRE<"mxb",  0xB34C, fmulwchain, FP128, FP128>;
+  }
 }
 def MEEB : BinaryRXE<"meeb", 0xED17, fmul, FP32, load, 4>;
 def MDB  : BinaryRXE<"mdb",  0xED1C, fmul, FP64, load, 8>;
 
 // f64 multiplication of two FP32 registers.
 def MDEBR : BinaryRRE<"mdeb", 0xB30C, null_frag, FP64, FP32>;
 def : Pat<(fmul (f64 (fextend FP32:$src1)), (f64 (fextend FP32:$src2))),
           (MDEBR (INSERT_SUBREG (f64 (IMPLICIT_DEF)),

 // Division.
 def DEBR : BinaryRRE<"deb", 0xB30D, fdiv, FP32,  FP32>;
 def DDBR : BinaryRRE<"ddb", 0xB31D, fdiv, FP64,  FP64>;
 def DXBR : BinaryRRE<"dxb", 0xB34D, fdiv, FP128, FP128>;
 
 def DEB : BinaryRXE<"deb", 0xED0D, fdiv, FP32, load, 4>;
 def DDB : BinaryRXE<"ddb", 0xED1D, fdiv, FP64, load, 8>;
 
+let isCodeGenOnly = 1 in {
+  def DEBR_FPE : BinaryRRE<"deb", 0xB30D, fdivwchain, FP32,  FP32>;
+  def DDBR_FPE : BinaryRRE<"ddb", 0xB31D, fdivwchain, FP64,  FP64>;
+  def DXBR_FPE : BinaryRRE<"dxb", 0xB34D, fdivwchain, FP128, FP128>;
+}
+
 //===----------------------------------------------------------------------===//
 // Comparisons
 //===----------------------------------------------------------------------===//
 
 let Defs = [CC], CCValues = 0xF in {
   def CEBR : CompareRRE<"ceb", 0xB309, z_fcmp, FP32,  FP32>;
   def CDBR : CompareRRE<"cdb", 0xB319, z_fcmp, FP64,  FP64>;
   def CXBR : CompareRRE<"cxb", 0xB349, z_fcmp, FP128, FP128>;

lib/Target/SystemZ/SystemZInstrVector.td

multiclass VectorRounding<Instruction insn, TypedReg tr> {
   def : FPConversion<insn, ftrunc,     tr, tr, 4, 5>;
   def : FPConversion<insn, frnd,       tr, tr, 4, 1>;
 }
 
 let Predicates = [FeatureVector] in {
   // Add.
   def VFADB : BinaryVRRc<"vfadb", 0xE7E3, fadd, v128db, v128db, 3, 0>;
   def WFADB : BinaryVRRc<"wfadb", 0xE7E3, fadd, v64db, v64db, 3, 8>;
+  let isCodeGenOnly = 1 in {
+    def VFADB_FPE :
+      BinaryVRRc<"vfadb", 0xE7E3, faddwchain, v128db, v128db, 3, 0>;
+    def WFADB_FPE :
+      BinaryVRRc<"wfadb", 0xE7E3, faddwchain, v64db, v64db, 3, 8>;
+  }
 
   // Convert from fixed 64-bit.
   def VCDGB : TernaryVRRa<"vcdgb", 0xE7C3, null_frag, v128db, v128g, 3, 0>;
   def WCDGB : TernaryVRRa<"wcdgb", 0xE7C3, null_frag, v64db, v64g, 3, 8>;
   def : FPConversion<VCDGB, sint_to_fp, v128db, v128g, 0, 0>;
 
   // Convert from logical 64-bit.
   def VCDLGB : TernaryVRRa<"vcdlgb", 0xE7C1, null_frag, v128db, v128g, 3, 0>;
let Predicates = [FeatureVector] in {
   def VCLGDB : TernaryVRRa<"vclgdb", 0xE7C0, null_frag, v128g, v128db, 3, 0>;
   def WCLGDB : TernaryVRRa<"wclgdb", 0xE7C0, null_frag, v64g, v64db, 3, 8>;
   // Rounding mode should agree with SystemZInstrFP.td.
   def : FPConversion<VCLGDB, fp_to_uint, v128g, v128db, 0, 5>;
 
   // Divide.
   def VFDDB : BinaryVRRc<"vfddb", 0xE7E5, fdiv, v128db, v128db, 3, 0>;
   def WFDDB : BinaryVRRc<"wfddb", 0xE7E5, fdiv, v64db, v64db, 3, 8>;
+  let isCodeGenOnly = 1 in {
+    def VFDDB_FPE :
+      BinaryVRRc<"vfddb", 0xE7E5, fdivwchain, v128db, v128db, 3, 0>;
+    def WFDDB_FPE :
+      BinaryVRRc<"wfddb", 0xE7E5, fdivwchain, v64db, v64db, 3, 8>;
+  }
 
   // Load FP integer.
   def VFIDB : TernaryVRRa<"vfidb", 0xE7C7, int_s390_vfidb, v128db, v128db, 3, 0>;
   def WFIDB : TernaryVRRa<"wfidb", 0xE7C7, null_frag, v64db, v64db, 3, 8>;
   defm : VectorRounding<VFIDB, v128db>;
   defm : VectorRounding<WFIDB, v64db>;
 
   // Load lengthened.
   def VLDEB : UnaryVRRa<"vldeb", 0xE7C4, z_vextend, v128db, v128eb, 2, 0>;
   def WLDEB : UnaryVRRa<"wldeb", 0xE7C4, fextend, v64db, v32eb, 2, 8>;
 
   // Load rounded,
   def VLEDB : TernaryVRRa<"vledb", 0xE7C5, null_frag, v128eb, v128db, 3, 0>;
   def WLEDB : TernaryVRRa<"wledb", 0xE7C5, null_frag, v32eb, v64db, 3, 8>;
   def : Pat<(v4f32 (z_vround (v2f64 VR128:$src))), (VLEDB VR128:$src, 0, 0)>;
   def : FPConversion<WLEDB, fround, v32eb, v64db, 0, 0>;
 
   // Multiply.
   def VFMDB : BinaryVRRc<"vfmdb", 0xE7E7, fmul, v128db, v128db, 3, 0>;
   def WFMDB : BinaryVRRc<"wfmdb", 0xE7E7, fmul, v64db, v64db, 3, 8>;
+  let isCodeGenOnly = 1 in {
+    def VFMDB_FPE :
+      BinaryVRRc<"vfmdb", 0xE7E7, fmulwchain, v128db, v128db, 3, 0>;
+    def WFMDB_FPE :
+      BinaryVRRc<"wfmdb", 0xE7E7, fmulwchain, v64db, v64db, 3, 8>;
+  }
 
   // Multiply and add.
   def VFMADB : TernaryVRRe<"vfmadb", 0xE78F, fma, v128db, v128db, 0, 3>;
   def WFMADB : TernaryVRRe<"wfmadb", 0xE78F, fma, v64db, v64db, 8, 3>;
 
   // Multiply and subtract.
   def VFMSDB : TernaryVRRe<"vfmsdb", 0xE78E, fms, v128db, v128db, 0, 3>;
   def WFMSDB : TernaryVRRe<"wfmsdb", 0xE78E, fms, v64db, v64db, 8, 3>;
let Predicates = [FeatureVector] in {
 
   // Square root.
   def VFSQDB : UnaryVRRa<"vfsqdb", 0xE7CE, fsqrt, v128db, v128db, 3, 0>;
   def WFSQDB : UnaryVRRa<"wfsqdb", 0xE7CE, fsqrt, v64db, v64db, 3, 8>;
 
   // Subtract.
   def VFSDB : BinaryVRRc<"vfsdb", 0xE7E2, fsub, v128db, v128db, 3, 0>;
   def WFSDB : BinaryVRRc<"wfsdb", 0xE7E2, fsub, v64db, v64db, 3, 8>;
+  let isCodeGenOnly = 1 in {
+    def VFSDB_FPE :
+      BinaryVRRc<"vfsdb", 0xE7E2, fsubwchain, v128db, v128db, 3, 0>;
+    def WFSDB_FPE :
+      BinaryVRRc<"wfsdb", 0xE7E2, fsubwchain, v64db, v64db, 3, 8>;
+  }
 
   // Test data class immediate.
   let Defs = [CC] in {
     def VFTCIDB : BinaryVRIe<"vftcidb", 0xE74A, z_vftci, v128g, v128db, 3, 0>;
     def WFTCIDB : BinaryVRIe<"wftcidb", 0xE74A, null_frag, v64g, v64db, 3, 8>;
   }
 }
 

lib/Target/X86/X86FloatingPoint.cpp

 // Register File -> Register Stack Mapping Methods
 //===----------------------------------------------------------------------===//
 
 // OpcodeTable - Sorted map of register instructions to their stack version.
 // The first element is an register file pseudo instruction, the second is the
 // concrete X86 instruction which uses the register stack.
 //
 static const TableEntry OpcodeTable[] = {
-  { X86::ABS_Fp32     , X86::ABS_F     },
-  { X86::ABS_Fp64     , X86::ABS_F     },
-  { X86::ABS_Fp80     , X86::ABS_F     },
-  { X86::ADD_Fp32m    , X86::ADD_F32m  },
-  { X86::ADD_Fp64m    , X86::ADD_F64m  },
-  { X86::ADD_Fp64m32  , X86::ADD_F32m  },
-  { X86::ADD_Fp80m32  , X86::ADD_F32m  },
-  { X86::ADD_Fp80m64  , X86::ADD_F64m  },
-  { X86::ADD_FpI16m32 , X86::ADD_FI16m },
-  { X86::ADD_FpI16m64 , X86::ADD_FI16m },
-  { X86::ADD_FpI16m80 , X86::ADD_FI16m },
-  { X86::ADD_FpI32m32 , X86::ADD_FI32m },
-  { X86::ADD_FpI32m64 , X86::ADD_FI32m },
-  { X86::ADD_FpI32m80 , X86::ADD_FI32m },
-  { X86::CHS_Fp32     , X86::CHS_F     },
-  { X86::CHS_Fp64     , X86::CHS_F     },
-  { X86::CHS_Fp80     , X86::CHS_F     },
-  { X86::CMOVBE_Fp32  , X86::CMOVBE_F  },
-  { X86::CMOVBE_Fp64  , X86::CMOVBE_F  },
-  { X86::CMOVBE_Fp80  , X86::CMOVBE_F  },
-  { X86::CMOVB_Fp32   , X86::CMOVB_F   },
-  { X86::CMOVB_Fp64   , X86::CMOVB_F  },
-  { X86::CMOVB_Fp80   , X86::CMOVB_F  },
-  { X86::CMOVE_Fp32   , X86::CMOVE_F  },
-  { X86::CMOVE_Fp64   , X86::CMOVE_F   },
-  { X86::CMOVE_Fp80   , X86::CMOVE_F   },
-  { X86::CMOVNBE_Fp32 , X86::CMOVNBE_F },
-  { X86::CMOVNBE_Fp64 , X86::CMOVNBE_F },
-  { X86::CMOVNBE_Fp80 , X86::CMOVNBE_F },
-  { X86::CMOVNB_Fp32  , X86::CMOVNB_F  },
-  { X86::CMOVNB_Fp64  , X86::CMOVNB_F  },
-  { X86::CMOVNB_Fp80  , X86::CMOVNB_F  },
-  { X86::CMOVNE_Fp32  , X86::CMOVNE_F  },
-  { X86::CMOVNE_Fp64  , X86::CMOVNE_F  },
-  { X86::CMOVNE_Fp80  , X86::CMOVNE_F  },
-  { X86::CMOVNP_Fp32  , X86::CMOVNP_F  },
-  { X86::CMOVNP_Fp64  , X86::CMOVNP_F  },
-  { X86::CMOVNP_Fp80  , X86::CMOVNP_F  },
-  { X86::CMOVP_Fp32   , X86::CMOVP_F   },
-  { X86::CMOVP_Fp64   , X86::CMOVP_F   },
-  { X86::CMOVP_Fp80   , X86::CMOVP_F   },
-  { X86::COS_Fp32     , X86::COS_F     },
-  { X86::COS_Fp64     , X86::COS_F     },
-  { X86::COS_Fp80     , X86::COS_F     },
-  { X86::DIVR_Fp32m   , X86::DIVR_F32m },
-  { X86::DIVR_Fp64m   , X86::DIVR_F64m },
-  { X86::DIVR_Fp64m32 , X86::DIVR_F32m },
-  { X86::DIVR_Fp80m32 , X86::DIVR_F32m },
-  { X86::DIVR_Fp80m64 , X86::DIVR_F64m },
-  { X86::DIVR_FpI16m32, X86::DIVR_FI16m},
-  { X86::DIVR_FpI16m64, X86::DIVR_FI16m},
-  { X86::DIVR_FpI16m80, X86::DIVR_FI16m},
-  { X86::DIVR_FpI32m32, X86::DIVR_FI32m},
-  { X86::DIVR_FpI32m64, X86::DIVR_FI32m},
-  { X86::DIVR_FpI32m80, X86::DIVR_FI32m},
-  { X86::DIV_Fp32m    , X86::DIV_F32m  },
-  { X86::DIV_Fp64m    , X86::DIV_F64m  },
-  { X86::DIV_Fp64m32  , X86::DIV_F32m  },
-  { X86::DIV_Fp80m32  , X86::DIV_F32m  },
-  { X86::DIV_Fp80m64  , X86::DIV_F64m  },
-  { X86::DIV_FpI16m32 , X86::DIV_FI16m },
-  { X86::DIV_FpI16m64 , X86::DIV_FI16m },
-  { X86::DIV_FpI16m80 , X86::DIV_FI16m },
-  { X86::DIV_FpI32m32 , X86::DIV_FI32m },
-  { X86::DIV_FpI32m64 , X86::DIV_FI32m },
-  { X86::DIV_FpI32m80 , X86::DIV_FI32m },
-  { X86::ILD_Fp16m32  , X86::ILD_F16m  },
-  { X86::ILD_Fp16m64  , X86::ILD_F16m  },
-  { X86::ILD_Fp16m80  , X86::ILD_F16m  },
-  { X86::ILD_Fp32m32  , X86::ILD_F32m  },
-  { X86::ILD_Fp32m64  , X86::ILD_F32m  },
-  { X86::ILD_Fp32m80  , X86::ILD_F32m  },
-  { X86::ILD_Fp64m32  , X86::ILD_F64m  },
-  { X86::ILD_Fp64m64  , X86::ILD_F64m  },
-  { X86::ILD_Fp64m80  , X86::ILD_F64m  },
-  { X86::ISTT_Fp16m32 , X86::ISTT_FP16m},
-  { X86::ISTT_Fp16m64 , X86::ISTT_FP16m},
-  { X86::ISTT_Fp16m80 , X86::ISTT_FP16m},
-  { X86::ISTT_Fp32m32 , X86::ISTT_FP32m},
-  { X86::ISTT_Fp32m64 , X86::ISTT_FP32m},
-  { X86::ISTT_Fp32m80 , X86::ISTT_FP32m},
-  { X86::ISTT_Fp64m32 , X86::ISTT_FP64m},
-  { X86::ISTT_Fp64m64 , X86::ISTT_FP64m},
-  { X86::ISTT_Fp64m80 , X86::ISTT_FP64m},
-  { X86::IST_Fp16m32  , X86::IST_F16m  },
-  { X86::IST_Fp16m64  , X86::IST_F16m  },
-  { X86::IST_Fp16m80  , X86::IST_F16m  },
-  { X86::IST_Fp32m32  , X86::IST_F32m  },
-  { X86::IST_Fp32m64  , X86::IST_F32m  },
-  { X86::IST_Fp32m80  , X86::IST_F32m  },
-  { X86::IST_Fp64m32  , X86::IST_FP64m },
-  { X86::IST_Fp64m64  , X86::IST_FP64m },
-  { X86::IST_Fp64m80  , X86::IST_FP64m },
-  { X86::LD_Fp032     , X86::LD_F0     },
-  { X86::LD_Fp064     , X86::LD_F0     },
-  { X86::LD_Fp080     , X86::LD_F0     },
-  { X86::LD_Fp132     , X86::LD_F1     },
-  { X86::LD_Fp164     , X86::LD_F1     },
-  { X86::LD_Fp180     , X86::LD_F1     },
-  { X86::LD_Fp32m     , X86::LD_F32m   },
-  { X86::LD_Fp32m64   , X86::LD_F32m   },
-  { X86::LD_Fp32m80   , X86::LD_F32m   },
-  { X86::LD_Fp64m     , X86::LD_F64m   },
-  { X86::LD_Fp64m80   , X86::LD_F64m   },
-  { X86::LD_Fp80m     , X86::LD_F80m   },
-  { X86::MUL_Fp32m    , X86::MUL_F32m  },
-  { X86::MUL_Fp64m    , X86::MUL_F64m  },
-  { X86::MUL_Fp64m32  , X86::MUL_F32m  },
-  { X86::MUL_Fp80m32  , X86::MUL_F32m  },
-  { X86::MUL_Fp80m64  , X86::MUL_F64m  },
-  { X86::MUL_FpI16m32 , X86::MUL_FI16m },
-  { X86::MUL_FpI16m64 , X86::MUL_FI16m },
-  { X86::MUL_FpI16m80 , X86::MUL_FI16m },
-  { X86::MUL_FpI32m32 , X86::MUL_FI32m },
-  { X86::MUL_FpI32m64 , X86::MUL_FI32m },
-  { X86::MUL_FpI32m80 , X86::MUL_FI32m },
-  { X86::SIN_Fp32     , X86::SIN_F     },
-  { X86::SIN_Fp64     , X86::SIN_F     },
-  { X86::SIN_Fp80     , X86::SIN_F     },
-  { X86::SQRT_Fp32    , X86::SQRT_F    },
-  { X86::SQRT_Fp64    , X86::SQRT_F    },
-  { X86::SQRT_Fp80    , X86::SQRT_F    },
-  { X86::ST_Fp32m     , X86::ST_F32m   },
-  { X86::ST_Fp64m     , X86::ST_F64m   },
-  { X86::ST_Fp64m32   , X86::ST_F32m   },
-  { X86::ST_Fp80m32   , X86::ST_F32m   },
-  { X86::ST_Fp80m64   , X86::ST_F64m   },
-  { X86::ST_FpP80m    , X86::ST_FP80m  },
-  { X86::SUBR_Fp32m   , X86::SUBR_F32m },
-  { X86::SUBR_Fp64m   , X86::SUBR_F64m },
-  { X86::SUBR_Fp64m32 , X86::SUBR_F32m },
-  { X86::SUBR_Fp80m32 , X86::SUBR_F32m },
-  { X86::SUBR_Fp80m64 , X86::SUBR_F64m },
-  { X86::SUBR_FpI16m32, X86::SUBR_FI16m},
-  { X86::SUBR_FpI16m64, X86::SUBR_FI16m},
-  { X86::SUBR_FpI16m80, X86::SUBR_FI16m},
-  { X86::SUBR_FpI32m32, X86::SUBR_FI32m},
-  { X86::SUBR_FpI32m64, X86::SUBR_FI32m},
-  { X86::SUBR_FpI32m80, X86::SUBR_FI32m},
-  { X86::SUB_Fp32m    , X86::SUB_F32m  },
-  { X86::SUB_Fp64m    , X86::SUB_F64m  },
-  { X86::SUB_Fp64m32  , X86::SUB_F32m  },
-  { X86::SUB_Fp80m32  , X86::SUB_F32m  },
-  { X86::SUB_Fp80m64  , X86::SUB_F64m  },
-  { X86::SUB_FpI16m32 , X86::SUB_FI16m },
-  { X86::SUB_FpI16m64 , X86::SUB_FI16m },
-  { X86::SUB_FpI16m80 , X86::SUB_FI16m },
-  { X86::SUB_FpI32m32 , X86::SUB_FI32m },
-  { X86::SUB_FpI32m64 , X86::SUB_FI32m },
-  { X86::SUB_FpI32m80 , X86::SUB_FI32m },
-  { X86::TST_Fp32     , X86::TST_F     },
-  { X86::TST_Fp64     , X86::TST_F     },
-  { X86::TST_Fp80     , X86::TST_F     },
-  { X86::UCOM_FpIr32  , X86::UCOM_FIr  },
-  { X86::UCOM_FpIr64  , X86::UCOM_FIr  },
-  { X86::UCOM_FpIr80  , X86::UCOM_FIr  },
-  { X86::UCOM_Fpr32   , X86::UCOM_Fr   },
-  { X86::UCOM_Fpr64   , X86::UCOM_Fr   },
-  { X86::UCOM_Fpr80   , X86::UCOM_Fr   },
+  { X86::ABS_Fp32         , X86::ABS_F         },
+  { X86::ABS_Fp64         , X86::ABS_F         },
+  { X86::ABS_Fp80         , X86::ABS_F         },
+  { X86::ADD_FPE_Fp32m    , X86::ADD_FPE_F32m  },
+  { X86::ADD_FPE_Fp64m    , X86::ADD_FPE_F64m  },
+  { X86::ADD_FPE_Fp64m32  , X86::ADD_FPE_F32m  },
+  { X86::ADD_FPE_Fp80m32  , X86::ADD_FPE_F32m  },
+  { X86::ADD_FPE_Fp80m64  , X86::ADD_FPE_F64m  },
+  { X86::ADD_FPE_FpI16m32 , X86::ADD_FPE_FI16m },
+  { X86::ADD_FPE_FpI16m64 , X86::ADD_FPE_FI16m },
+  { X86::ADD_FPE_FpI16m80 , X86::ADD_FPE_FI16m },
+  { X86::ADD_FPE_FpI32m32 , X86::ADD_FPE_FI32m },
+  { X86::ADD_FPE_FpI32m64 , X86::ADD_FPE_FI32m },
+  { X86::ADD_FPE_FpI32m80 , X86::ADD_FPE_FI32m },
+  { X86::ADD_Fp32m        , X86::ADD_F32m      },
+  { X86::ADD_Fp64m        , X86::ADD_F64m      },
+  { X86::ADD_Fp64m32      , X86::ADD_F32m      },
+  { X86::ADD_Fp80m32      , X86::ADD_F32m      },
+  { X86::ADD_Fp80m64      , X86::ADD_F64m      },
+  { X86::ADD_FpI16m32     , X86::ADD_FI16m     },
+  { X86::ADD_FpI16m64     , X86::ADD_FI16m     },
+  { X86::ADD_FpI16m80     , X86::ADD_FI16m     },
+  { X86::ADD_FpI32m32     , X86::ADD_FI32m     },
+  { X86::ADD_FpI32m64     , X86::ADD_FI32m     },
+  { X86::ADD_FpI32m80     , X86::ADD_FI32m     },
+  { X86::CHS_Fp32         , X86::CHS_F         },
+  { X86::CHS_Fp64         , X86::CHS_F         },
+  { X86::CHS_Fp80         , X86::CHS_F         },
+  { X86::CMOVBE_Fp32      , X86::CMOVBE_F      },
+  { X86::CMOVBE_Fp64      , X86::CMOVBE_F      },
+  { X86::CMOVBE_Fp80      , X86::CMOVBE_F      },
+  { X86::CMOVB_Fp32       , X86::CMOVB_F       },
+  { X86::CMOVB_Fp64       , X86::CMOVB_F       },
+  { X86::CMOVB_Fp80       , X86::CMOVB_F       },
+  { X86::CMOVE_Fp32       , X86::CMOVE_F       },
+  { X86::CMOVE_Fp64       , X86::CMOVE_F       },
+  { X86::CMOVE_Fp80       , X86::CMOVE_F       },
+  { X86::CMOVNBE_Fp32     , X86::CMOVNBE_F     },
+  { X86::CMOVNBE_Fp64     , X86::CMOVNBE_F     },
+  { X86::CMOVNBE_Fp80     , X86::CMOVNBE_F     },
+  { X86::CMOVNB_Fp32      , X86::CMOVNB_F      },
+  { X86::CMOVNB_Fp64      , X86::CMOVNB_F      },
+  { X86::CMOVNB_Fp80      , X86::CMOVNB_F      },
+  { X86::CMOVNE_Fp32      , X86::CMOVNE_F      },
+  { X86::CMOVNE_Fp64      , X86::CMOVNE_F      },
+  { X86::CMOVNE_Fp80      , X86::CMOVNE_F      },
+  { X86::CMOVNP_Fp32      , X86::CMOVNP_F      },
+  { X86::CMOVNP_Fp64      , X86::CMOVNP_F      },
+  { X86::CMOVNP_Fp80      , X86::CMOVNP_F      },
+  { X86::CMOVP_Fp32       , X86::CMOVP_F       },
+  { X86::CMOVP_Fp64       , X86::CMOVP_F       },
+  { X86::CMOVP_Fp80       , X86::CMOVP_F       },
+  { X86::COS_Fp32         , X86::COS_F         },
+  { X86::COS_Fp64         , X86::COS_F         },
+  { X86::COS_Fp80         , X86::COS_F         },
+  { X86::DIVR_Fp32m       , X86::DIVR_F32m     },
+  { X86::DIVR_Fp64m       , X86::DIVR_F64m     },
+  { X86::DIVR_Fp64m32     , X86::DIVR_F32m     },
+  { X86::DIVR_Fp80m32     , X86::DIVR_F32m     },
+  { X86::DIVR_Fp80m64     , X86::DIVR_F64m     },
+  { X86::DIVR_FpI16m32    , X86::DIVR_FI16m    },
+  { X86::DIVR_FpI16m64    , X86::DIVR_FI16m    },
+  { X86::DIVR_FpI16m80    , X86::DIVR_FI16m    },
+  { X86::DIVR_FpI32m32    , X86::DIVR_FI32m    },
+  { X86::DIVR_FpI32m64    , X86::DIVR_FI32m    },
+  { X86::DIVR_FpI32m80    , X86::DIVR_FI32m    },
+  { X86::DIV_FPE_Fp32m    , X86::DIV_FPE_F32m  },
+  { X86::DIV_FPE_Fp64m    , X86::DIV_FPE_F64m  },
+  { X86::DIV_FPE_Fp64m32  , X86::DIV_FPE_F32m  },
+  { X86::DIV_FPE_Fp80m32  , X86::DIV_FPE_F32m  },
+  { X86::DIV_FPE_Fp80m64  , X86::DIV_FPE_F64m  },
+  { X86::DIV_FPE_FpI16m32 , X86::DIV_FPE_FI16m },
+  { X86::DIV_FPE_FpI16m64 , X86::DIV_FPE_FI16m },
+  { X86::DIV_FPE_FpI16m80 , X86::DIV_FPE_FI16m },
+  { X86::DIV_FPE_FpI32m32 , X86::DIV_FPE_FI32m },
+  { X86::DIV_FPE_FpI32m64 , X86::DIV_FPE_FI32m },
+  { X86::DIV_FPE_FpI32m80 , X86::DIV_FPE_FI32m },
+  { X86::DIV_Fp32m        , X86::DIV_F32m      },
+  { X86::DIV_Fp64m        , X86::DIV_F64m      },
+  { X86::DIV_Fp64m32      , X86::DIV_F32m      },
+  { X86::DIV_Fp80m32      , X86::DIV_F32m      },
+  { X86::DIV_Fp80m64      , X86::DIV_F64m      },
+  { X86::DIV_FpI16m32     , X86::DIV_FI16m     },
+  { X86::DIV_FpI16m64     , X86::DIV_FI16m     },
+  { X86::DIV_FpI16m80     , X86::DIV_FI16m     },
+  { X86::DIV_FpI32m32     , X86::DIV_FI32m     },
+  { X86::DIV_FpI32m64     , X86::DIV_FI32m     },
+  { X86::DIV_FpI32m80     , X86::DIV_FI32m     },
+  { X86::ILD_Fp16m32      , X86::ILD_F16m      },
+  { X86::ILD_Fp16m64      , X86::ILD_F16m      },
+  { X86::ILD_Fp16m80      , X86::ILD_F16m      },
+  { X86::ILD_Fp32m32      , X86::ILD_F32m      },
+  { X86::ILD_Fp32m64      , X86::ILD_F32m      },
+  { X86::ILD_Fp32m80      , X86::ILD_F32m      },
+  { X86::ILD_Fp64m32      , X86::ILD_F64m      },
+  { X86::ILD_Fp64m64      , X86::ILD_F64m      },
+  { X86::ILD_Fp64m80      , X86::ILD_F64m      },
+  { X86::ISTT_Fp16m32     , X86::ISTT_FP16m    },
+  { X86::ISTT_Fp16m64     , X86::ISTT_FP16m    },
+  { X86::ISTT_Fp16m80     , X86::ISTT_FP16m    },
+  { X86::ISTT_Fp32m32     , X86::ISTT_FP32m    },
+  { X86::ISTT_Fp32m64     , X86::ISTT_FP32m    },
+  { X86::ISTT_Fp32m80     , X86::ISTT_FP32m    },
+  { X86::ISTT_Fp64m32     , X86::ISTT_FP64m    },
+  { X86::ISTT_Fp64m64     , X86::ISTT_FP64m    },
+  { X86::ISTT_Fp64m80     , X86::ISTT_FP64m    },
+  { X86::IST_Fp16m32      , X86::IST_F16m      },
+  { X86::IST_Fp16m64      , X86::IST_F16m      },
+  { X86::IST_Fp16m80      , X86::IST_F16m      },
+  { X86::IST_Fp32m32      , X86::IST_F32m      },
+  { X86::IST_Fp32m64      , X86::IST_F32m      },
+  { X86::IST_Fp32m80      , X86::IST_F32m      },
+  { X86::IST_Fp64m32      , X86::IST_FP64m     },
+  { X86::IST_Fp64m64      , X86::IST_FP64m     },
+  { X86::IST_Fp64m80      , X86::IST_FP64m     },
+  { X86::LD_Fp032         , X86::LD_F0         },
+  { X86::LD_Fp064         , X86::LD_F0         },
+  { X86::LD_Fp080         , X86::LD_F0         },
+  { X86::LD_Fp132         , X86::LD_F1         },
+  { X86::LD_Fp164         , X86::LD_F1         },
+  { X86::LD_Fp180         , X86::LD_F1         },
+  { X86::LD_Fp32m         , X86::LD_F32m       },
+  { X86::LD_Fp32m64       , X86::LD_F32m       },
+  { X86::LD_Fp32m80       , X86::LD_F32m       },
+  { X86::LD_Fp64m         , X86::LD_F64m       },
+  { X86::LD_Fp64m80       , X86::LD_F64m       },
+  { X86::LD_Fp80m         , X86::LD_F80m       },
+  { X86::MUL_FPE_Fp32m    , X86::MUL_FPE_F32m  },
+  { X86::MUL_FPE_Fp64m    , X86::MUL_FPE_F64m  },
+  { X86::MUL_FPE_Fp64m32  , X86::MUL_FPE_F32m  },
+  { X86::MUL_FPE_Fp80m32  , X86::MUL_FPE_F32m  },
+  { X86::MUL_FPE_Fp80m64  , X86::MUL_FPE_F64m  },
+  { X86::MUL_FPE_FpI16m32 , X86::MUL_FPE_FI16m },
+  { X86::MUL_FPE_FpI16m64 , X86::MUL_FPE_FI16m },
+  { X86::MUL_FPE_FpI16m80 , X86::MUL_FPE_FI16m },
+  { X86::MUL_FPE_FpI32m32 , X86::MUL_FPE_FI32m },
+  { X86::MUL_FPE_FpI32m64 , X86::MUL_FPE_FI32m },
+  { X86::MUL_FPE_FpI32m80 , X86::MUL_FPE_FI32m },
+  { X86::MUL_Fp32m        , X86::MUL_F32m      },
+  { X86::MUL_Fp64m        , X86::MUL_F64m      },
+  { X86::MUL_Fp64m32      , X86::MUL_F32m      },
+  { X86::MUL_Fp80m32      , X86::MUL_F32m      },
+  { X86::MUL_Fp80m64      , X86::MUL_F64m      },
+  { X86::MUL_FpI16m32     , X86::MUL_FI16m     },
+  { X86::MUL_FpI16m64     , X86::MUL_FI16m     },
+  { X86::MUL_FpI16m80     , X86::MUL_FI16m     },
+  { X86::MUL_FpI32m32     , X86::MUL_FI32m     },
+  { X86::MUL_FpI32m64     , X86::MUL_FI32m     },
+  { X86::MUL_FpI32m80     , X86::MUL_FI32m     },
+  { X86::SIN_Fp32         , X86::SIN_F         },
+  { X86::SIN_Fp64         , X86::SIN_F         },
+  { X86::SIN_Fp80         , X86::SIN_F         },
+  { X86::SQRT_Fp32        , X86::SQRT_F        },
+  { X86::SQRT_Fp64        , X86::SQRT_F        },
+  { X86::SQRT_Fp80        , X86::SQRT_F        },
+  { X86::ST_Fp32m         , X86::ST_F32m       },
+  { X86::ST_Fp64m         , X86::ST_F64m       },
+  { X86::ST_Fp64m32       , X86::ST_F32m       },
+  { X86::ST_Fp80m32       , X86::ST_F32m       },
+  { X86::ST_Fp80m64       , X86::ST_F64m       },
+  { X86::ST_FpP80m        , X86::ST_FP80m      },
+  { X86::SUBR_Fp32m       , X86::SUBR_F32m     },
+  { X86::SUBR_Fp64m       , X86::SUBR_F64m     },
+  { X86::SUBR_Fp64m32     , X86::SUBR_F32m     },
+  { X86::SUBR_Fp80m32     , X86::SUBR_F32m     },
+  { X86::SUBR_Fp80m64     , X86::SUBR_F64m     },
+  { X86::SUBR_FpI16m32    , X86::SUBR_FI16m    },
+  { X86::SUBR_FpI16m64    , X86::SUBR_FI16m    },
+  { X86::SUBR_FpI16m80    , X86::SUBR_FI16m    },
+  { X86::SUBR_FpI32m32    , X86::SUBR_FI32m    },
+  { X86::SUBR_FpI32m64    , X86::SUBR_FI32m    },
+  { X86::SUBR_FpI32m80    , X86::SUBR_FI32m    },
+  { X86::SUB_FPE_Fp32m    , X86::SUB_FPE_F32m  },
+  { X86::SUB_FPE_Fp64m    , X86::SUB_FPE_F64m  },
+  { X86::SUB_FPE_Fp64m32  , X86::SUB_FPE_F32m  },
+  { X86::SUB_FPE_Fp80m32  , X86::SUB_FPE_F32m  },
+  { X86::SUB_FPE_Fp80m64  , X86::SUB_FPE_F64m  },
+  { X86::SUB_FPE_FpI16m32 , X86::SUB_FPE_FI16m },
+  { X86::SUB_FPE_FpI16m64 , X86::SUB_FPE_FI16m },
+  { X86::SUB_FPE_FpI16m80 , X86::SUB_FPE_FI16m },
+  { X86::SUB_FPE_FpI32m32 , X86::SUB_FPE_FI32m },
+  { X86::SUB_FPE_FpI32m64 , X86::SUB_FPE_FI32m },
+  { X86::SUB_FPE_FpI32m80 , X86::SUB_FPE_FI32m },
+  { X86::SUB_Fp32m        , X86::SUB_F32m      },
+  { X86::SUB_Fp64m        , X86::SUB_F64m      },
+  { X86::SUB_Fp64m32      , X86::SUB_F32m      },
+  { X86::SUB_Fp80m32      , X86::SUB_F32m      },
+  { X86::SUB_Fp80m64      , X86::SUB_F64m      },
+  { X86::SUB_FpI16m32     , X86::SUB_FI16m     },
+  { X86::SUB_FpI16m64     , X86::SUB_FI16m     },
+  { X86::SUB_FpI16m80     , X86::SUB_FI16m     },
+  { X86::SUB_FpI32m32     , X86::SUB_FI32m     },
+  { X86::SUB_FpI32m64     , X86::SUB_FI32m     },
+  { X86::SUB_FpI32m80     , X86::SUB_FI32m     },
+  { X86::TST_Fp32         , X86::TST_F         },
+  { X86::TST_Fp64         , X86::TST_F         },
+  { X86::TST_Fp80         , X86::TST_F         },
+  { X86::UCOM_FpIr32      , X86::UCOM_FIr      },
+  { X86::UCOM_FpIr64      , X86::UCOM_FIr      },
+  { X86::UCOM_FpIr80      , X86::UCOM_FIr      },
+  { X86::UCOM_Fpr32       , X86::UCOM_Fr       },
+  { X86::UCOM_Fpr64       , X86::UCOM_Fr       },
+  { X86::UCOM_Fpr80       , X86::UCOM_Fr       },
 };
 
 static unsigned getConcreteOpcode(unsigned Opcode) {
   ASSERT_SORTED(OpcodeTable);
   int Opc = Lookup(OpcodeTable, Opcode);
   assert(Opc != -1 && "FP Stack instruction not in OpcodeTable!");
   return Opc;
 }

 
 
 //===----------------------------------------------------------------------===//
 // Define tables of various ways to map pseudo instructions
 //
 
 // ForwardST0Table - Map: A = B op C  into: ST(0) = ST(0) op ST(i)
 static const TableEntry ForwardST0Table[] = {
-  { X86::ADD_Fp32  , X86::ADD_FST0r },
-  { X86::ADD_Fp64  , X86::ADD_FST0r },
-  { X86::ADD_Fp80  , X86::ADD_FST0r },
-  { X86::DIV_Fp32  , X86::DIV_FST0r },
-  { X86::DIV_Fp64  , X86::DIV_FST0r },
-  { X86::DIV_Fp80  , X86::DIV_FST0r },
-  { X86::MUL_Fp32  , X86::MUL_FST0r },
-  { X86::MUL_Fp64  , X86::MUL_FST0r },
-  { X86::MUL_Fp80  , X86::MUL_FST0r },
-  { X86::SUB_Fp32  , X86::SUB_FST0r },
-  { X86::SUB_Fp64  , X86::SUB_FST0r },
-  { X86::SUB_Fp80  , X86::SUB_FST0r },
+  { X86::ADD_FPE_Fp32 , X86::ADD_FST0r },
+  { X86::ADD_FPE_Fp64 , X86::ADD_FST0r },
+  { X86::ADD_FPE_Fp80 , X86::ADD_FST0r },
+  { X86::ADD_Fp32     , X86::ADD_FST0r },
+  { X86::ADD_Fp64     , X86::ADD_FST0r },
+  { X86::ADD_Fp80     , X86::ADD_FST0r },
+  { X86::DIV_FPE_Fp32 , X86::DIV_FST0r },
+  { X86::DIV_FPE_Fp64 , X86::DIV_FST0r },
+  { X86::DIV_FPE_Fp80 , X86::DIV_FST0r },
+  { X86::DIV_Fp32     , X86::DIV_FST0r },
+  { X86::DIV_Fp64     , X86::DIV_FST0r },
+  { X86::DIV_Fp80     , X86::DIV_FST0r },
+  { X86::MUL_FPE_Fp32 , X86::MUL_FST0r },
+  { X86::MUL_FPE_Fp64 , X86::MUL_FST0r },
+  { X86::MUL_FPE_Fp80 , X86::MUL_FST0r },
+  { X86::MUL_Fp32     , X86::MUL_FST0r },
+  { X86::MUL_Fp64     , X86::MUL_FST0r },
+  { X86::MUL_Fp80     , X86::MUL_FST0r },
+  { X86::SUB_FPE_Fp32 , X86::SUB_FST0r },
+  { X86::SUB_FPE_Fp64 , X86::SUB_FST0r },
+  { X86::SUB_FPE_Fp80 , X86::SUB_FST0r },
+  { X86::SUB_Fp32     , X86::SUB_FST0r },
+  { X86::SUB_Fp64     , X86::SUB_FST0r },
+  { X86::SUB_Fp80     , X86::SUB_FST0r },
 };
 
 // ReverseST0Table - Map: A = B op C  into: ST(0) = ST(i) op ST(0)
 static const TableEntry ReverseST0Table[] = {
-  { X86::ADD_Fp32  , X86::ADD_FST0r  },   // commutative
-  { X86::ADD_Fp64  , X86::ADD_FST0r  },   // commutative
-  { X86::ADD_Fp80  , X86::ADD_FST0r  },   // commutative
-  { X86::DIV_Fp32  , X86::DIVR_FST0r },
-  { X86::DIV_Fp64  , X86::DIVR_FST0r },
-  { X86::DIV_Fp80  , X86::DIVR_FST0r },
-  { X86::MUL_Fp32  , X86::MUL_FST0r  },   // commutative
-  { X86::MUL_Fp64  , X86::MUL_FST0r  },   // commutative
-  { X86::MUL_Fp80  , X86::MUL_FST0r  },   // commutative
-  { X86::SUB_Fp32  , X86::SUBR_FST0r },
-  { X86::SUB_Fp64  , X86::SUBR_FST0r },
-  { X86::SUB_Fp80  , X86::SUBR_FST0r },
+  { X86::ADD_FPE_Fp32 , X86::ADD_FST0r  }, // commutative
+  { X86::ADD_FPE_Fp64 , X86::ADD_FST0r  }, // commutative
+  { X86::ADD_FPE_Fp80 , X86::ADD_FST0r  }, // commutative
+  { X86::ADD_Fp32     , X86::ADD_FST0r  }, // commutative
+  { X86::ADD_Fp64     , X86::ADD_FST0r  }, // commutative
+  { X86::ADD_Fp80     , X86::ADD_FST0r  }, // commutative
+  { X86::DIV_FPE_Fp32 , X86::DIVR_FST0r },
+  { X86::DIV_FPE_Fp64 , X86::DIVR_FST0r },
+  { X86::DIV_FPE_Fp80 , X86::DIVR_FST0r },
+  { X86::DIV_Fp32     , X86::DIVR_FST0r },
+  { X86::DIV_Fp64     , X86::DIVR_FST0r },
+  { X86::DIV_Fp80     , X86::DIVR_FST0r },
+  { X86::MUL_FPE_Fp32 , X86::MUL_FST0r  }, // commutative
+  { X86::MUL_FPE_Fp64 , X86::MUL_FST0r  }, // commutative
+  { X86::MUL_FPE_Fp80 , X86::MUL_FST0r  }, // commutative
+  { X86::MUL_Fp32     , X86::MUL_FST0r  }, // commutative
+  { X86::MUL_Fp64     , X86::MUL_FST0r  }, // commutative
+  { X86::MUL_Fp80     , X86::MUL_FST0r  }, // commutative
+  { X86::SUB_FPE_Fp32 , X86::SUBR_FST0r },
+  { X86::SUB_FPE_Fp64 , X86::SUBR_FST0r },
+  { X86::SUB_FPE_Fp80 , X86::SUBR_FST0r },
+  { X86::SUB_Fp32     , X86::SUBR_FST0r },
+  { X86::SUB_Fp64     , X86::SUBR_FST0r },
+  { X86::SUB_Fp80     , X86::SUBR_FST0r },
 };
 
 // ForwardSTiTable - Map: A = B op C  into: ST(i) = ST(0) op ST(i)
 static const TableEntry ForwardSTiTable[] = {
-  { X86::ADD_Fp32  , X86::ADD_FrST0  },   // commutative
-  { X86::ADD_Fp64  , X86::ADD_FrST0  },   // commutative
-  { X86::ADD_Fp80  , X86::ADD_FrST0  },   // commutative
-  { X86::DIV_Fp32  , X86::DIVR_FrST0 },
-  { X86::DIV_Fp64  , X86::DIVR_FrST0 },
-  { X86::DIV_Fp80  , X86::DIVR_FrST0 },
-  { X86::MUL_Fp32  , X86::MUL_FrST0  },   // commutative
-  { X86::MUL_Fp64  , X86::MUL_FrST0  },   // commutative
-  { X86::MUL_Fp80  , X86::MUL_FrST0  },   // commutative
-  { X86::SUB_Fp32  , X86::SUBR_FrST0 },
-  { X86::SUB_Fp64  , X86::SUBR_FrST0 },
-  { X86::SUB_Fp80  , X86::SUBR_FrST0 },
+  { X86::ADD_FPE_Fp32 , X86::ADD_FrST0  }, // commutative
+  { X86::ADD_FPE_Fp64 , X86::ADD_FrST0  }, // commutative
+  { X86::ADD_FPE_Fp80 , X86::ADD_FrST0  }, // commutative
+  { X86::ADD_Fp32     , X86::ADD_FrST0  }, // commutative
+  { X86::ADD_Fp64     , X86::ADD_FrST0  }, // commutative
+  { X86::ADD_Fp80     , X86::ADD_FrST0  }, // commutative
+  { X86::DIV_FPE_Fp32 , X86::DIVR_FrST0 },
+  { X86::DIV_FPE_Fp64 , X86::DIVR_FrST0 },
+  { X86::DIV_FPE_Fp80 , X86::DIVR_FrST0 },
+  { X86::DIV_Fp32     , X86::DIVR_FrST0 },
+  { X86::DIV_Fp64     , X86::DIVR_FrST0 },
+  { X86::DIV_Fp80     , X86::DIVR_FrST0 },
+  { X86::MUL_FPE_Fp32 , X86::MUL_FrST0  }, // commutative
+  { X86::MUL_FPE_Fp64 , X86::MUL_FrST0  }, // commutative
+  { X86::MUL_FPE_Fp80 , X86::MUL_FrST0  }, // commutative
+  { X86::MUL_Fp32     , X86::MUL_FrST0  }, // commutative
+  { X86::MUL_Fp64     , X86::MUL_FrST0  }, // commutative
+  { X86::MUL_Fp80     , X86::MUL_FrST0  }, // commutative
+  { X86::SUB_FPE_Fp32 , X86::SUBR_FrST0 },
+  { X86::SUB_FPE_Fp64 , X86::SUBR_FrST0 },
+  { X86::SUB_FPE_Fp80 , X86::SUBR_FrST0 },
+  { X86::SUB_Fp32     , X86::SUBR_FrST0 },
+  { X86::SUB_Fp64     , X86::SUBR_FrST0 },
+  { X86::SUB_Fp80     , X86::SUBR_FrST0 },
 };
 
 // ReverseSTiTable - Map: A = B op C  into: ST(i) = ST(i) op ST(0)
 static const TableEntry ReverseSTiTable[] = {
-  { X86::ADD_Fp32  , X86::ADD_FrST0 },
-  { X86::ADD_Fp64  , X86::ADD_FrST0 },
-  { X86::ADD_Fp80  , X86::ADD_FrST0 },
-  { X86::DIV_Fp32  , X86::DIV_FrST0 },
-  { X86::DIV_Fp64  , X86::DIV_FrST0 },
-  { X86::DIV_Fp80  , X86::DIV_FrST0 },
-  { X86::MUL_Fp32  , X86::MUL_FrST0 },
-  { X86::MUL_Fp64  , X86::MUL_FrST0 },
-  { X86::MUL_Fp80  , X86::MUL_FrST0 },
-  { X86::SUB_Fp32  , X86::SUB_FrST0 },
-  { X86::SUB_Fp64  , X86::SUB_FrST0 },
-  { X86::SUB_Fp80  , X86::SUB_FrST0 },
+  { X86::ADD_FPE_Fp32 , X86::ADD_FrST0 },
+  { X86::ADD_FPE_Fp64 , X86::ADD_FrST0 },
+  { X86::ADD_FPE_Fp80 , X86::ADD_FrST0 },
+  { X86::ADD_Fp32     , X86::ADD_FrST0 },
+  { X86::ADD_Fp64     , X86::ADD_FrST0 },
+  { X86::ADD_Fp80     , X86::ADD_FrST0 },
+  { X86::DIV_FPE_Fp32 , X86::DIV_FrST0 },
+  { X86::DIV_FPE_Fp64 , X86::DIV_FrST0 },
+  { X86::DIV_FPE_Fp80 , X86::DIV_FrST0 },
+  { X86::DIV_Fp32     , X86::DIV_FrST0 },
+  { X86::DIV_Fp64     , X86::DIV_FrST0 },
+  { X86::DIV_Fp80     , X86::DIV_FrST0 },
+  { X86::MUL_FPE_Fp32 , X86::MUL_FrST0 },
+  { X86::MUL_FPE_Fp64 , X86::MUL_FrST0 },
+  { X86::MUL_FPE_Fp80 , X86::MUL_FrST0 },
+  { X86::MUL_Fp32     , X86::MUL_FrST0 },
+  { X86::MUL_Fp64     , X86::MUL_FrST0 },
+  { X86::MUL_Fp80     , X86::MUL_FrST0 },
+  { X86::SUB_FPE_Fp32 , X86::SUB_FrST0 },
+  { X86::SUB_FPE_Fp64 , X86::SUB_FrST0 },
+  { X86::SUB_FPE_Fp80 , X86::SUB_FrST0 },
+  { X86::SUB_Fp32     , X86::SUB_FrST0 },
+  { X86::SUB_Fp64     , X86::SUB_FrST0 },
+  { X86::SUB_Fp80     , X86::SUB_FrST0 },
 };
 
 
 /// handleTwoArgFP - Handle instructions like FADD and friends which are virtual
 /// instructions which need to be simplified and possibly transformed.
 ///
 /// Result: ST(0) = fsub  ST(0), ST(i)
 ///         ST(i) = fsub  ST(0), ST(i)

lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
       /// Combined add and sub on an FP vector.
       ADDSUB,
 
       //  FP vector ops with rounding mode.
       FADD_RND,
       FSUB_RND,
       FMUL_RND,
       FDIV_RND,
+      FADDWCHAIN_RND,
+      FSUBWCHAIN_RND,
+      FMULWCHAIN_RND,
+      FDIVWCHAIN_RND,
       FMAX_RND,
       FMIN_RND,
       FSQRT_RND,
 
       // FP vector get exponent 
       FGETEXP_RND,
       // Extract Normalized Mantissas
       VGETMANT,

lib/Target/X86/X86ISelLowering.cpp

   setOperationAction(ISD::FP_ROUND_INREG   , MVT::f32  , Expand);
 
   if (Subtarget->is32Bit() && Subtarget->isTargetKnownWindowsMSVC()) {
     // On 32 bit MSVC, `fmodf(f32)` is not defined - only `fmod(f64)`
     // is. We should promote the value to 64-bits to solve this.
     // This is what the CRT headers do - `fmodf` is an inline header
     // function casting to f64 and calling `fmod`.
     setOperationAction(ISD::FREM           , MVT::f32  , Promote);
+    setOperationAction(ISD::FREM_W_CHAIN   , MVT::f32  , Promote);
   } else {
     setOperationAction(ISD::FREM           , MVT::f32  , Expand);
+    setOperationAction(ISD::FREM_W_CHAIN   , MVT::f32  , Expand);
   }
 
   setOperationAction(ISD::FREM             , MVT::f64  , Expand);
   setOperationAction(ISD::FREM             , MVT::f80  , Expand);
+  setOperationAction(ISD::FREM_W_CHAIN     , MVT::f64  , Expand);
+  setOperationAction(ISD::FREM_W_CHAIN     , MVT::f80  , Expand);
   setOperationAction(ISD::FLT_ROUNDS_      , MVT::i32  , Custom);
 
   // Promote the i8 variants and force them on up to i32 which has a shorter
   // encoding.
   setOperationAction(ISD::CTTZ             , MVT::i8   , Promote);
   AddPromotedToType (ISD::CTTZ             , MVT::i8   , MVT::i32);
   setOperationAction(ISD::CTTZ_ZERO_UNDEF  , MVT::i8   , Promote);
   AddPromotedToType (ISD::CTTZ_ZERO_UNDEF  , MVT::i8   , MVT::i32);

 
   // First set operation action for all vector types to either promote
   // (for widening) or expand (for scalarization). Then we will selectively
   // turn on ones that can be effectively codegen'd.
   for (MVT VT : MVT::vector_valuetypes()) {
     setOperationAction(ISD::ADD , VT, Expand);
     setOperationAction(ISD::SUB , VT, Expand);
     setOperationAction(ISD::FADD, VT, Expand);
+    setOperationAction(ISD::FADD_W_CHAIN, VT, Expand);
     setOperationAction(ISD::FNEG, VT, Expand);
     setOperationAction(ISD::FSUB, VT, Expand);
+    setOperationAction(ISD::FSUB_W_CHAIN, VT, Expand);
     setOperationAction(ISD::MUL , VT, Expand);
     setOperationAction(ISD::FMUL, VT, Expand);
+    setOperationAction(ISD::FMUL_W_CHAIN, VT, Expand);
     setOperationAction(ISD::SDIV, VT, Expand);
     setOperationAction(ISD::UDIV, VT, Expand);
     setOperationAction(ISD::FDIV, VT, Expand);
+    setOperationAction(ISD::FDIV_W_CHAIN, VT, Expand);
     setOperationAction(ISD::SREM, VT, Expand);
     setOperationAction(ISD::UREM, VT, Expand);
     setOperationAction(ISD::LOAD, VT, Expand);
     setOperationAction(ISD::VECTOR_SHUFFLE, VT, Expand);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT,Expand);
     setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Expand);
     setOperationAction(ISD::EXTRACT_SUBVECTOR, VT,Expand);
     setOperationAction(ISD::INSERT_SUBVECTOR, VT,Expand);
     setOperationAction(ISD::FABS, VT, Expand);
     setOperationAction(ISD::FSIN, VT, Expand);
     setOperationAction(ISD::FSINCOS, VT, Expand);
     setOperationAction(ISD::FCOS, VT, Expand);
     setOperationAction(ISD::FSINCOS, VT, Expand);
     setOperationAction(ISD::FREM, VT, Expand);
+    setOperationAction(ISD::FREM_W_CHAIN, VT, Expand);
     setOperationAction(ISD::FMA,  VT, Expand);
     setOperationAction(ISD::FPOWI, VT, Expand);
     setOperationAction(ISD::FSQRT, VT, Expand);
     setOperationAction(ISD::FCOPYSIGN, VT, Expand);
     setOperationAction(ISD::FFLOOR, VT, Expand);
     setOperationAction(ISD::FCEIL, VT, Expand);
     setOperationAction(ISD::FTRUNC, VT, Expand);
     setOperationAction(ISD::FRINT, VT, Expand);

     setOperationAction(ISD::BITCAST,            MMXTy,      Expand);
   }
   setOperationAction(ISD::INSERT_VECTOR_ELT,  MVT::v1i64, Expand);
 
   if (!Subtarget->useSoftFloat() && Subtarget->hasSSE1()) {
     addRegisterClass(MVT::v4f32, &X86::VR128RegClass);
 
     setOperationAction(ISD::FADD,               MVT::v4f32, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v4f32, Legal);
     setOperationAction(ISD::FSUB,               MVT::v4f32, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v4f32, Legal);
     setOperationAction(ISD::FMUL,               MVT::v4f32, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v4f32, Legal);
     setOperationAction(ISD::FDIV,               MVT::v4f32, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v4f32, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v4f32, Legal);
     setOperationAction(ISD::FNEG,               MVT::v4f32, Custom);
     setOperationAction(ISD::FABS,               MVT::v4f32, Custom);
     setOperationAction(ISD::LOAD,               MVT::v4f32, Legal);
     setOperationAction(ISD::BUILD_VECTOR,       MVT::v4f32, Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE,     MVT::v4f32, Custom);
     setOperationAction(ISD::VSELECT,            MVT::v4f32, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v4f32, Custom);

     setOperationAction(ISD::MULHU,              MVT::v8i16, Legal);
     setOperationAction(ISD::MULHS,              MVT::v8i16, Legal);
     setOperationAction(ISD::SUB,                MVT::v16i8, Legal);
     setOperationAction(ISD::SUB,                MVT::v8i16, Legal);
     setOperationAction(ISD::SUB,                MVT::v4i32, Legal);
     setOperationAction(ISD::SUB,                MVT::v2i64, Legal);
     setOperationAction(ISD::MUL,                MVT::v8i16, Legal);
     setOperationAction(ISD::FADD,               MVT::v2f64, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v2f64, Legal);
     setOperationAction(ISD::FSUB,               MVT::v2f64, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v2f64, Legal);
     setOperationAction(ISD::FMUL,               MVT::v2f64, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v2f64, Legal);
     setOperationAction(ISD::FDIV,               MVT::v2f64, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v2f64, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v2f64, Legal);
     setOperationAction(ISD::FNEG,               MVT::v2f64, Custom);
     setOperationAction(ISD::FABS,               MVT::v2f64, Custom);
 
     setOperationAction(ISD::SMAX,               MVT::v8i16, Legal);
     setOperationAction(ISD::UMAX,               MVT::v16i8, Legal);
     setOperationAction(ISD::SMIN,               MVT::v8i16, Legal);
     setOperationAction(ISD::UMIN,               MVT::v16i8, Legal);

     addRegisterClass(MVT::v4i64,  &X86::VR256RegClass);
     addRegisterClass(MVT::v4f64,  &X86::VR256RegClass);
 
     setOperationAction(ISD::LOAD,               MVT::v8f32, Legal);
     setOperationAction(ISD::LOAD,               MVT::v4f64, Legal);
     setOperationAction(ISD::LOAD,               MVT::v4i64, Legal);
 
     setOperationAction(ISD::FADD,               MVT::v8f32, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v8f32, Legal);
     setOperationAction(ISD::FSUB,               MVT::v8f32, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v8f32, Legal);
     setOperationAction(ISD::FMUL,               MVT::v8f32, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v8f32, Legal);
     setOperationAction(ISD::FDIV,               MVT::v8f32, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v8f32, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v8f32, Legal);
     setOperationAction(ISD::FFLOOR,             MVT::v8f32, Legal);
     setOperationAction(ISD::FCEIL,              MVT::v8f32, Legal);
     setOperationAction(ISD::FTRUNC,             MVT::v8f32, Legal);
     setOperationAction(ISD::FRINT,              MVT::v8f32, Legal);
     setOperationAction(ISD::FNEARBYINT,         MVT::v8f32, Legal);
     setOperationAction(ISD::FNEG,               MVT::v8f32, Custom);
     setOperationAction(ISD::FABS,               MVT::v8f32, Custom);
 
     setOperationAction(ISD::FADD,               MVT::v4f64, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v4f64, Legal);
     setOperationAction(ISD::FSUB,               MVT::v4f64, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v4f64, Legal);
     setOperationAction(ISD::FMUL,               MVT::v4f64, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v4f64, Legal);
     setOperationAction(ISD::FDIV,               MVT::v4f64, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v4f64, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v4f64, Legal);
     setOperationAction(ISD::FFLOOR,             MVT::v4f64, Legal);
     setOperationAction(ISD::FCEIL,              MVT::v4f64, Legal);
     setOperationAction(ISD::FTRUNC,             MVT::v4f64, Legal);
     setOperationAction(ISD::FRINT,              MVT::v4f64, Legal);
     setOperationAction(ISD::FNEARBYINT,         MVT::v4f64, Legal);
     setOperationAction(ISD::FNEG,               MVT::v4f64, Custom);
     setOperationAction(ISD::FABS,               MVT::v4f64, Custom);

     setOperationAction(ISD::MUL,                MVT::i1,    Custom);
     setOperationAction(ISD::LOAD,               MVT::v16f32, Legal);
     setOperationAction(ISD::LOAD,               MVT::v8f64, Legal);
     setOperationAction(ISD::LOAD,               MVT::v8i64, Legal);
     setOperationAction(ISD::LOAD,               MVT::v16i32, Legal);
     setOperationAction(ISD::LOAD,               MVT::v16i1, Legal);
 
     setOperationAction(ISD::FADD,               MVT::v16f32, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v16f32, Legal);
     setOperationAction(ISD::FSUB,               MVT::v16f32, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v16f32, Legal);
     setOperationAction(ISD::FMUL,               MVT::v16f32, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v16f32, Legal);
     setOperationAction(ISD::FDIV,               MVT::v16f32, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v16f32, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v16f32, Legal);
     setOperationAction(ISD::FNEG,               MVT::v16f32, Custom);
 
     setOperationAction(ISD::FADD,               MVT::v8f64, Legal);
+    setOperationAction(ISD::FADD_W_CHAIN,       MVT::v8f64, Legal);
     setOperationAction(ISD::FSUB,               MVT::v8f64, Legal);
+    setOperationAction(ISD::FSUB_W_CHAIN,       MVT::v8f64, Legal);
     setOperationAction(ISD::FMUL,               MVT::v8f64, Legal);
+    setOperationAction(ISD::FMUL_W_CHAIN,       MVT::v8f64, Legal);
     setOperationAction(ISD::FDIV,               MVT::v8f64, Legal);
+    setOperationAction(ISD::FDIV_W_CHAIN,       MVT::v8f64, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v8f64, Legal);
     setOperationAction(ISD::FNEG,               MVT::v8f64, Custom);
     setOperationAction(ISD::FMA,                MVT::v8f64, Legal);
     setOperationAction(ISD::FMA,                MVT::v16f32, Legal);
 
     setOperationAction(ISD::FP_TO_SINT,         MVT::v16i32, Legal);
     setOperationAction(ISD::FP_TO_UINT,         MVT::v16i32, Legal);
     setOperationAction(ISD::FP_TO_UINT,         MVT::v8i32, Legal);

   case X86ISD::COMPRESS:           return "X86ISD::COMPRESS";
   case X86ISD::EXPAND:             return "X86ISD::EXPAND";
   case X86ISD::SELECT:             return "X86ISD::SELECT";
   case X86ISD::ADDSUB:             return "X86ISD::ADDSUB";
   case X86ISD::RCP28:              return "X86ISD::RCP28";
   case X86ISD::EXP2:               return "X86ISD::EXP2";
   case X86ISD::RSQRT28:            return "X86ISD::RSQRT28";
   case X86ISD::FADD_RND:           return "X86ISD::FADD_RND";
+  case X86ISD::FADDWCHAIN_RND:     return "X86ISD::FADDWCHAIN_RND";
   case X86ISD::FSUB_RND:           return "X86ISD::FSUB_RND";
+  case X86ISD::FSUBWCHAIN_RND:     return "X86ISD::FSUBWCHAIN_RND";
   case X86ISD::FMUL_RND:           return "X86ISD::FMUL_RND";
+  case X86ISD::FMULWCHAIN_RND:     return "X86ISD::FMULWCHAIN_RND";
   case X86ISD::FDIV_RND:           return "X86ISD::FDIV_RND";
+  case X86ISD::FDIVWCHAIN_RND:     return "X86ISD::FDIVWCHAIN_RND";
   case X86ISD::FSQRT_RND:          return "X86ISD::FSQRT_RND";
   case X86ISD::FGETEXP_RND:        return "X86ISD::FGETEXP_RND";
   case X86ISD::SCALEF:             return "X86ISD::SCALEF";
   case X86ISD::ADDS:               return "X86ISD::ADDS";
   case X86ISD::SUBS:               return "X86ISD::SUBS";
   case X86ISD::AVG:                return "X86ISD::AVG";
   case X86ISD::MULHRS:             return "X86ISD::MULHRS";
   case X86ISD::SINT_TO_FP_RND:     return "X86ISD::SINT_TO_FP_RND";

lib/Target/X86/X86InstrAVX512.td

defm SSZ : avx512_fp_scalar<opc, OpcodeStr#"ss", f32x_info, OpNode, VecNode,
                               XS, EVEX_4V, VEX_LIG,  EVEX_CD8<32, CD8VT1>;
   defm SDZ : avx512_fp_scalar<opc, OpcodeStr#"sd", f64x_info, OpNode, VecNode,
                               itins.d,                  IsCommutable>,
              avx512_fp_scalar_sae<opc, OpcodeStr#"sd", f64x_info, VecNode,
                               itins.d, IsCommutable>,
                               XD, VEX_W, EVEX_4V, VEX_LIG, EVEX_CD8<64, CD8VT1>;
 }
 defm VADD : avx512_binop_s_round<0x58, "vadd", fadd, X86faddRnd, SSE_ALU_ITINS_S, 1>;
+let isCodeGenOnly = 1 in {
+  defm VADD_FPE : avx512_binop_s_round<0x58, "vadd", faddwchain,
+                                       X86faddRndWChain, SSE_ALU_ITINS_S, 1>;
+  defm VSUB_FPE : avx512_binop_s_round<0x5C, "vsub", fsubwchain,
+                                       X86fsubRndWChain, SSE_ALU_ITINS_S, 0>;
+  defm VMUL_FPE : avx512_binop_s_round<0x59, "vmul", fmulwchain,
+                                       X86fmulRndWChain, SSE_ALU_ITINS_S, 1>;
+  defm VDIV_FPE : avx512_binop_s_round<0x5E, "vdiv", fdivwchain,
+                                       X86fdivRndWChain, SSE_ALU_ITINS_S, 0>;
+}
 defm VMUL : avx512_binop_s_round<0x59, "vmul", fmul, X86fmulRnd, SSE_ALU_ITINS_S, 1>;
 defm VSUB : avx512_binop_s_round<0x5C, "vsub", fsub, X86fsubRnd, SSE_ALU_ITINS_S, 0>;
 defm VDIV : avx512_binop_s_round<0x5E, "vdiv", fdiv, X86fdivRnd, SSE_ALU_ITINS_S, 0>;
 defm VMIN : avx512_binop_s_sae  <0x5D, "vmin", X86fmin, X86fminRnd, SSE_ALU_ITINS_S, 1>;
 defm VMAX : avx512_binop_s_sae  <0x5F, "vmax", X86fmax, X86fmaxRnd, SSE_ALU_ITINS_S, 1>;
 
 multiclass avx512_fp_packed<bits<8> opc, string OpcodeStr, SDNode OpNode,
                             X86VectorVTInfo _, bit IsCommutable> {
multiclass avx512_fp_binop_p_sae<bits<8> opc, string OpcodeStr, SDNode OpNodeRnd> {
   defm PSZ : avx512_fp_sae_packed<opc, OpcodeStr, OpNodeRnd, v16f32_info>,
                               EVEX_V512, PS, EVEX_CD8<32, CD8VF>;
   defm PDZ : avx512_fp_sae_packed<opc, OpcodeStr, OpNodeRnd, v8f64_info>,
                               EVEX_V512, PD, VEX_W,EVEX_CD8<64, CD8VF>;
 }
 
 defm VADD : avx512_fp_binop_p<0x58, "vadd", fadd, 1>,
             avx512_fp_binop_p_round<0x58, "vadd", X86faddRnd>;
+let isCodeGenOnly = 1 in {
+  defm VADD_FPE : avx512_fp_binop_p<0x58, "vadd", faddwchain, 1>;
+  defm VSUB_FPE : avx512_fp_binop_p<0x5C, "vsub", fsubwchain, 1>;
+  defm VMUL_FPE : avx512_fp_binop_p<0x59, "vmul", fmulwchain, 1>;
+  defm VDIV_FPE : avx512_fp_binop_p<0x5E, "vdiv", fdivwchain, 1>;
+}
 defm VMUL : avx512_fp_binop_p<0x59, "vmul", fmul, 1>,
             avx512_fp_binop_p_round<0x59, "vmul", X86fmulRnd>;
 defm VSUB : avx512_fp_binop_p<0x5C, "vsub", fsub>,
             avx512_fp_binop_p_round<0x5C, "vsub", X86fsubRnd>;
 defm VDIV : avx512_fp_binop_p<0x5E, "vdiv", fdiv>,
             avx512_fp_binop_p_round<0x5E, "vdiv", X86fdivRnd>;
 defm VMIN : avx512_fp_binop_p<0x5D, "vmin", X86fmin, 1>,
             avx512_fp_binop_p_sae<0x5D, "vmin", X86fminRnd>;

lib/Target/X86/X86InstrFPStack.td

 
 let Defs = [FPSW] in {
 // FPBinary_rr just defines pseudo-instructions, no need to set a scheduling
 // resources.
 defm ADD : FPBinary_rr<fadd>;
 defm SUB : FPBinary_rr<fsub>;
 defm MUL : FPBinary_rr<fmul>;
 defm DIV : FPBinary_rr<fdiv>;
+let isCodeGenOnly = 1 in {
+  defm ADD_FPE : FPBinary_rr<faddwchain>;
+  defm SUB_FPE : FPBinary_rr<fsubwchain>;
+  defm MUL_FPE : FPBinary_rr<fmulwchain>;
+  defm DIV_FPE : FPBinary_rr<fdivwchain>;
+}
 // Sets the scheduling resources for the actual NAME#_F<size>m defintions.
 let SchedRW = [WriteFAddLd] in {
 defm ADD : FPBinary<fadd, MRM0m, "add">;
+let isCodeGenOnly = 1 in
+  defm ADD_FPE : FPBinary<faddwchain, MRM0m, "add">;
 defm SUB : FPBinary<fsub, MRM4m, "sub">;
+let isCodeGenOnly = 1 in
+  defm SUB_FPE : FPBinary<fsubwchain, MRM4m, "sub">;
 defm SUBR: FPBinary<fsub ,MRM5m, "subr", 0>;
 }
 let SchedRW = [WriteFMulLd] in {
 defm MUL : FPBinary<fmul, MRM1m, "mul">;
+let isCodeGenOnly = 1 in
+  defm MUL_FPE : FPBinary<fmulwchain, MRM1m, "mul">;
 }
 let SchedRW = [WriteFDivLd] in {
 defm DIV : FPBinary<fdiv, MRM6m, "div">;
 defm DIVR: FPBinary<fdiv, MRM7m, "divr", 0>;
+let isCodeGenOnly = 1 in {
+  defm DIV_FPE  : FPBinary<fdivwchain, MRM6m, "div">;
+  defm DIVR_FPE : FPBinary<fdivwchain, MRM7m, "divr", 0>;
+}
 }
 }
 
 class FPST0rInst<Format fp, string asm>
   : FPI<0xD8, fp, (outs), (ins RST:$op), asm>;
 class FPrST0Inst<Format fp, string asm>
   : FPI<0xDC, fp, (outs), (ins RST:$op), asm>;
 class FPrST0PInst<Format fp, string asm>

lib/Target/X86/X86InstrFragmentsSIMD.td

 def X86Blendi    : SDNode<"X86ISD::BLENDI",   SDTBlend>;
 
 def X86Addsub    : SDNode<"X86ISD::ADDSUB", SDTFPBinOp>;
 
 def X86faddRnd   : SDNode<"X86ISD::FADD_RND",  SDTFPBinOpRound>;
 def X86fsubRnd   : SDNode<"X86ISD::FSUB_RND",  SDTFPBinOpRound>;
 def X86fmulRnd   : SDNode<"X86ISD::FMUL_RND",  SDTFPBinOpRound>;
 def X86fdivRnd   : SDNode<"X86ISD::FDIV_RND",  SDTFPBinOpRound>;
+def X86faddRndWChain : SDNode<"X86ISD::FADDWCHAIN_RND",  SDTFPBinOpRound>;
+def X86fsubRndWChain : SDNode<"X86ISD::FSUBWCHAIN_RND",  SDTFPBinOpRound>;
+def X86fmulRndWChain : SDNode<"X86ISD::FMULWCHAIN_RND",  SDTFPBinOpRound>;
+def X86fdivRndWChain : SDNode<"X86ISD::FDIVWCHAIN_RND",  SDTFPBinOpRound>;
 def X86fmaxRnd   : SDNode<"X86ISD::FMAX_RND",       SDTFPBinOpRound>;
 def X86scalef    : SDNode<"X86ISD::SCALEF",         SDTFPBinOpRound>;
 def X86fminRnd   : SDNode<"X86ISD::FMIN_RND",       SDTFPBinOpRound>;
 def X86fsqrtRnd     : SDNode<"X86ISD::FSQRT_RND",   SDTFPUnaryOpRound>;
 def X86fsqrtRnds    : SDNode<"X86ISD::FSQRT_RND",   STDFp2SrcRm>;
 def X86fgetexpRnd   : SDNode<"X86ISD::FGETEXP_RND", SDTFPUnaryOpRound>;
 def X86fgetexpRnds  : SDNode<"X86ISD::FGETEXP_RND", STDFp2SrcRm>;
 

lib/Target/X86/X86InstrSSE.td

def rm : SI<opc, MRMSrcMem, (outs RC:$dst), (ins RC:$src1, x86memop:$src2),
        [(set RC:$dst, (OpNode RC:$src1, (load addr:$src2)))], itins.rm, d>,
        Sched<[itins.Sched.Folded, ReadAfterLd]>;
 }
 
 /// sse12_fp_scalar_int - SSE 1 & 2 scalar instructions intrinsics class
 multiclass sse12_fp_scalar_int<bits<8> opc, string OpcodeStr, RegisterClass RC,
                              string asm, string SSEVer, string FPSizeStr,
                              Operand memopr, ComplexPattern mem_cpat,
-                             Domain d, OpndItins itins, bit Is2Addr = 1> {
-let isCodeGenOnly = 1 in {
+                             Domain d, OpndItins itins, bit Is2Addr, bit hse> {
+let isCodeGenOnly = 1, hasSideEffects = hse in {
   def rr_Int : SI_Int<opc, MRMSrcReg, (outs RC:$dst), (ins RC:$src1, RC:$src2),
        !if(Is2Addr,
            !strconcat(asm, "\t{$src2, $dst|$dst, $src2}"),
            !strconcat(asm, "\t{$src2, $src1, $dst|$dst, $src1, $src2}")),
        [(set RC:$dst, (!cast<Intrinsic>(
                  !strconcat("int_x86_sse", SSEVer, "_", OpcodeStr, FPSizeStr))
              RC:$src1, RC:$src2))], itins.rr, d>,
        Sched<[itins.Sched]>;
defm SS : sse12_fp_scalar<opc, !strconcat(OpcodeStr, "ss"),
                               itins.s>, XS;
     defm SD : sse12_fp_scalar<opc, !strconcat(OpcodeStr, "sd"),
                               OpNode, FR64, f64mem, SSEPackedDouble,
                               itins.d>, XD;
   }
 }
 
 multiclass basic_sse12_fp_binop_s_int<bits<8> opc, string OpcodeStr,
-                                      SizeItins itins> {
+                                      SizeItins itins, bit hse = 0> {
   defm V#NAME#SS : sse12_fp_scalar_int<opc, OpcodeStr, VR128,
                    !strconcat(OpcodeStr, "ss"), "", "_ss", ssmem, sse_load_f32,
-                   SSEPackedSingle, itins.s, 0>, XS, VEX_4V, VEX_LIG;
+                   SSEPackedSingle, itins.s, 0, hse>, XS, VEX_4V, VEX_LIG;
   defm V#NAME#SD : sse12_fp_scalar_int<opc, OpcodeStr, VR128,
                    !strconcat(OpcodeStr, "sd"), "2", "_sd", sdmem, sse_load_f64,
-                   SSEPackedDouble, itins.d, 0>, XD, VEX_4V, VEX_LIG;
+                   SSEPackedDouble, itins.d, 0, hse>, XD, VEX_4V, VEX_LIG;
 
   let Constraints = "$src1 = $dst" in {
     defm SS : sse12_fp_scalar_int<opc, OpcodeStr, VR128,
                    !strconcat(OpcodeStr, "ss"), "", "_ss", ssmem, sse_load_f32,
-                   SSEPackedSingle, itins.s>, XS;
+                   SSEPackedSingle, itins.s, 1, hse>, XS;
     defm SD : sse12_fp_scalar_int<opc, OpcodeStr, VR128,
                    !strconcat(OpcodeStr, "sd"), "2", "_sd", sdmem, sse_load_f64,
-                   SSEPackedDouble, itins.d>, XD;
+                   SSEPackedDouble, itins.d, 1, hse>, XD;
   }
 }
 
 // Binary Arithmetic instructions
 defm ADD : basic_sse12_fp_binop_p<0x58, "add", fadd, SSE_ALU_ITINS_P>,
            basic_sse12_fp_binop_s<0x58, "add", fadd, SSE_ALU_ITINS_S>,
            basic_sse12_fp_binop_s_int<0x58, "add", SSE_ALU_ITINS_S>;
+let isCodeGenOnly = 1 in
+defm ADD_FPE :
+           basic_sse12_fp_binop_p<0x58, "add", faddwchain, SSE_ALU_ITINS_P>,
+           basic_sse12_fp_binop_s<0x58, "add", faddwchain, SSE_ALU_ITINS_S>,
+           basic_sse12_fp_binop_s_int<0x58, "add", SSE_ALU_ITINS_S, 1>;
 defm MUL : basic_sse12_fp_binop_p<0x59, "mul", fmul, SSE_MUL_ITINS_P>,
            basic_sse12_fp_binop_s<0x59, "mul", fmul, SSE_MUL_ITINS_S>,
            basic_sse12_fp_binop_s_int<0x59, "mul", SSE_MUL_ITINS_S>;
+let isCodeGenOnly = 1 in
+defm MUL_FPE :
+           basic_sse12_fp_binop_p<0x59, "mul", fmulwchain, SSE_MUL_ITINS_P>,
+           basic_sse12_fp_binop_s<0x59, "mul", fmulwchain, SSE_MUL_ITINS_S>,
+           basic_sse12_fp_binop_s_int<0x59, "mul", SSE_MUL_ITINS_S, 1>;
 let isCommutable = 0 in {
   defm SUB : basic_sse12_fp_binop_p<0x5C, "sub", fsub, SSE_ALU_ITINS_P>,
              basic_sse12_fp_binop_s<0x5C, "sub", fsub, SSE_ALU_ITINS_S>,
              basic_sse12_fp_binop_s_int<0x5C, "sub", SSE_ALU_ITINS_S>;
+  let isCodeGenOnly = 1 in
+  defm SUB_FPE :
+             basic_sse12_fp_binop_p<0x5C, "sub", fsubwchain, SSE_ALU_ITINS_P>,
+             basic_sse12_fp_binop_s<0x5C, "sub", fsubwchain, SSE_ALU_ITINS_S>,
+             basic_sse12_fp_binop_s_int<0x5C, "sub", SSE_ALU_ITINS_S, 1>;
   defm DIV : basic_sse12_fp_binop_p<0x5E, "div", fdiv, SSE_DIV_ITINS_P>,
              basic_sse12_fp_binop_s<0x5E, "div", fdiv, SSE_DIV_ITINS_S>,
              basic_sse12_fp_binop_s_int<0x5E, "div", SSE_DIV_ITINS_S>;
+  let isCodeGenOnly = 1 in
+  defm DIV_FPE :
+             basic_sse12_fp_binop_p<0x5E, "div", fdivwchain, SSE_DIV_ITINS_P>,
+             basic_sse12_fp_binop_s<0x5E, "div", fdivwchain, SSE_DIV_ITINS_S>,
+             basic_sse12_fp_binop_s_int<0x5E, "div", SSE_DIV_ITINS_S, 1>;
   defm MAX : basic_sse12_fp_binop_p<0x5F, "max", X86fmax, SSE_ALU_ITINS_P>,
              basic_sse12_fp_binop_s<0x5F, "max", X86fmax, SSE_ALU_ITINS_S>,
              basic_sse12_fp_binop_s_int<0x5F, "max", SSE_ALU_ITINS_S>;
   defm MIN : basic_sse12_fp_binop_p<0x5D, "min", X86fmin, SSE_ALU_ITINS_P>,
              basic_sse12_fp_binop_s<0x5D, "min", X86fmin, SSE_ALU_ITINS_S>,
              basic_sse12_fp_binop_s_int<0x5D, "min", SSE_ALU_ITINS_S>;
 }
 
let Predicates = [HasAVX] in {
     // vector math op with insert via blend
     def : Pat<(v4f32 (X86Blendi (v4f32 VR128:$dst),
           (Op (v4f32 VR128:$dst), (v4f32 VR128:$src)), (i8 1))),
       (!cast<I>("V"#OpcPrefix#SSrr_Int) v4f32:$dst, v4f32:$src)>;
   }
 }
 
 defm : scalar_math_f32_patterns<fadd, "ADD">;
+defm : scalar_math_f32_patterns<faddwchain, "ADD_FPE">;
 defm : scalar_math_f32_patterns<fsub, "SUB">;
+defm : scalar_math_f32_patterns<fsubwchain, "SUB_FPE">;
 defm : scalar_math_f32_patterns<fmul, "MUL">;
+defm : scalar_math_f32_patterns<fmulwchain, "MUL_FPE">;
 defm : scalar_math_f32_patterns<fdiv, "DIV">;
+defm : scalar_math_f32_patterns<fdivwchain, "DIV_FPE">;
 
 multiclass scalar_math_f64_patterns<SDNode Op, string OpcPrefix> {
   let Predicates = [UseSSE2] in {
     // extracted scalar math op with insert via movsd
     def : Pat<(v2f64 (X86Movsd (v2f64 VR128:$dst), (v2f64 (scalar_to_vector
           (Op (f64 (vector_extract (v2f64 VR128:$dst), (iPTR 0))),
           FR64:$src))))),
       (!cast<I>(OpcPrefix#SDrr_Int) v2f64:$dst,
let Predicates = [HasAVX] in {
     // vector math op with insert via blend
     def : Pat<(v2f64 (X86Blendi (v2f64 VR128:$dst),
           (Op (v2f64 VR128:$dst), (v2f64 VR128:$src)), (i8 1))),
       (!cast<I>("V"#OpcPrefix#SDrr_Int) v2f64:$dst, v2f64:$src)>;
   }
 }
 
 defm : scalar_math_f64_patterns<fadd, "ADD">;
+defm : scalar_math_f64_patterns<faddwchain, "ADD_FPE">;
 defm : scalar_math_f64_patterns<fsub, "SUB">;
+defm : scalar_math_f64_patterns<fsubwchain, "SUB_FPE">;
 defm : scalar_math_f64_patterns<fmul, "MUL">;
+defm : scalar_math_f64_patterns<fmulwchain, "MUL_FPE">;
 defm : scalar_math_f64_patterns<fdiv, "DIV">;
+defm : scalar_math_f64_patterns<fdivwchain, "DIV_FPE">;
 
 
 /// Unop Arithmetic
 /// In addition, we also have a special variant of the scalar form here to
 /// represent the associated intrinsic operation.  This form is unlike the
 /// plain scalar form, in that it takes an entire vector (instead of a
 /// scalar) and leaves the top elements undefined.
 ///

test/CodeGen/ARM/fpenv-call-order.ll

+; RUN: llc -enable-except-access-fp-math -enable-control-access-fp-math -O0 < %s | FileCheck %s
+; RUN: llc -enable-except-access-fp-math -enable-control-access-fp-math < %s | FileCheck %s
+
+target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
+target triple = "armv7--linux-gnueabihf"
+
+; When floating-point environment is not examined, reordering instructions
+; doesn't harm, otherwise library call that checks floating-point environment
+; state can be moved before the actual computation, producing wrong results (not
+; affected by side-effects of operations).
+
+; CHECK: vadd.f32
+; -CHECK: vsub.f32
+; CHECK: bl function
+
+; Function Attrs: nounwind
+define float @f(float %x, float %y) {
+entry:
+  %res1 = fadd float %x, %y
+  %res2 = fsub float %x, %res1
+  %a = call i32 @function0()
+  tail call void @function(i32 %a)
+  ret float %res2
+}
+
+declare i32 @function0()
+declare void @function(i32)
+
+!llvm.module.flags = !{!0, !1}
+!llvm.ident = !{!2}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{i32 1, !"min_enum_size", i32 4}
+!2 = !{!""}

test/CodeGen/X86/avx2-vbroadcast.ll

 ; CHECK-NEXT:    .align 4, 0x90
 ; CHECK-NEXT:  LBB31_1: ## %footer349VF
 ; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    jmp LBB31_1
 WGLoopsEntry:
   br i1 undef, label %ret, label %footer329VF
 
 footer329VF:
-  %A.0.inVF = fmul float undef, 6.553600e+04
-  %B.0.in407VF = fmul <8 x float> undef, <float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04>
+  %A.0.inVF = fmul nrnd nexc float undef, 6.553600e+04
+  %B.0.in407VF = fmul nrnd nexc <8 x float> undef, <float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04, float 6.553600e+04>
   %A.0VF = fptosi float %A.0.inVF to i32
   %B.0408VF = fptosi <8 x float> %B.0.in407VF to <8 x i32>
   %0 = and <8 x i32> %B.0408VF, <i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535, i32 65535>
   %1 = and i32 %A.0VF, 65535
   %temp1098VF = insertelement <8 x i32> undef, i32 %1, i32 0
   %vector1099VF = shufflevector <8 x i32> %temp1098VF, <8 x i32> undef, <8 x i32> zeroinitializer
   br i1 undef, label %preload1201VF, label %footer349VF
 

test/CodeGen/X86/fma_patterns.ll

 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_f64_fneg_fmul:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; AVX512-NEXT:    vfnmsub213sd %xmm2, %xmm0, %xmm1
 ; AVX512-NEXT:    vmovaps %zmm1, %zmm0
 ; AVX512-NEXT:    retq
-  %m = fmul nsz double %x, %y
-  %n = fsub double -0.0, %m
+  %m = fmul nrnd nexc nsz double %x, %y
+  %n = fsub nrnd nexc double -0.0, %m
   ret double %n
 }
 
 define <4 x float> @test_v4f32_fneg_fmul(<4 x float> %x, <4 x float> %y) #0 {
 ; FMA-LABEL: test_v4f32_fneg_fmul:
 ; FMA:       # BB#0:
 ; FMA-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; FMA-NEXT:    vfnmsub213ps %xmm2, %xmm1, %xmm0
 ; FMA-NEXT:    retq
 ;
 ; FMA4-LABEL: test_v4f32_fneg_fmul:
 ; FMA4:       # BB#0:
 ; FMA4-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; FMA4-NEXT:    vfnmsubps %xmm2, %xmm1, %xmm0, %xmm0
 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v4f32_fneg_fmul:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; AVX512-NEXT:    vfnmsub213ps %xmm2, %xmm1, %xmm0
 ; AVX512-NEXT:    retq
-  %m = fmul nsz <4 x float> %x, %y
-  %n = fsub <4 x float> <float -0.0, float -0.0, float -0.0, float -0.0>, %m
+  %m = fmul nrnd nexc nsz <4 x float> %x, %y
+  %n = fsub nrnd nexc <4 x float> <float -0.0, float -0.0, float -0.0, float -0.0>, %m
   ret <4 x float> %n
 }
 
 define <4 x double> @test_v4f64_fneg_fmul(<4 x double> %x, <4 x double> %y) #0 {
 ; FMA-LABEL: test_v4f64_fneg_fmul:
 ; FMA:       # BB#0:
 ; FMA-NEXT:    vxorpd %ymm2, %ymm2, %ymm2
 ; FMA-NEXT:    vfnmsub213pd %ymm2, %ymm1, %ymm0
 ; FMA-NEXT:    retq
 ;
 ; FMA4-LABEL: test_v4f64_fneg_fmul:
 ; FMA4:       # BB#0:
 ; FMA4-NEXT:    vxorpd %ymm2, %ymm2, %ymm2
 ; FMA4-NEXT:    vfnmsubpd %ymm2, %ymm1, %ymm0, %ymm0
 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v4f64_fneg_fmul:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vxorps %ymm2, %ymm2, %ymm2
 ; AVX512-NEXT:    vfnmsub213pd %ymm2, %ymm1, %ymm0
 ; AVX512-NEXT:    retq
-  %m = fmul nsz <4 x double> %x, %y
-  %n = fsub <4 x double> <double -0.0, double -0.0, double -0.0, double -0.0>, %m
+  %m = fmul nrnd nexc nsz <4 x double> %x, %y
+  %n = fsub nrnd nexc <4 x double> <double -0.0, double -0.0, double -0.0, double -0.0>, %m
   ret <4 x double> %n
 }
 
 define <4 x double> @test_v4f64_fneg_fmul_no_nsz(<4 x double> %x, <4 x double> %y) #0 {
 ; ALL-LABEL: test_v4f64_fneg_fmul_no_nsz:
 ; ALL:       # BB#0:
 ; ALL-NEXT:    vmulpd %ymm1, %ymm0, %ymm0
 ; ALL-NEXT:    vxorpd {{.*}}(%rip), %ymm0, %ymm0
 ; ALL-NEXT:    retq
-  %m = fmul <4 x double> %x, %y
-  %n = fsub <4 x double> <double -0.0, double -0.0, double -0.0, double -0.0>, %m
+  %m = fmul nrnd nexc <4 x double> %x, %y
+  %n = fsub nrnd nexc <4 x double> <double -0.0, double -0.0, double -0.0, double -0.0>, %m
   ret <4 x double> %n
 }
 
 attributes #0 = { "unsafe-fp-math"="true" }

test/CodeGen/X86/fma_patterns_wide.ll

 ; FMA4-NEXT:    vfnmsubps %ymm4, %ymm3, %ymm1, %ymm1
 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v16f32_fneg_fmul:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vpxord %zmm2, %zmm2, %zmm2
 ; AVX512-NEXT:    vfnmsub213ps %zmm2, %zmm1, %zmm0
 ; AVX512-NEXT:    retq
-  %m = fmul nsz <16 x float> %x, %y
-  %n = fsub <16 x float> <float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0>, %m
+  %m = fmul nrnd nexc nsz <16 x float> %x, %y
+  %n = fsub nrnd nexc <16 x float> <float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0, float -0.0>, %m
   ret <16 x float> %n
 }
 
 define <8 x double> @test_v8f64_fneg_fmul(<8 x double> %x, <8 x double> %y) #0 {
 ; FMA-LABEL: test_v8f64_fneg_fmul:
 ; FMA:       # BB#0:
 ; FMA-NEXT:    vxorpd %ymm4, %ymm4, %ymm4
 ; FMA-NEXT:    vfnmsub213pd %ymm4, %ymm2, %ymm0
 ; FMA-NEXT:    vfnmsub213pd %ymm4, %ymm3, %ymm1
 ; FMA-NEXT:    retq
 ;
 ; FMA4-LABEL: test_v8f64_fneg_fmul:
 ; FMA4:       # BB#0:
 ; FMA4-NEXT:    vxorpd %ymm4, %ymm4, %ymm4
 ; FMA4-NEXT:    vfnmsubpd %ymm4, %ymm2, %ymm0, %ymm0
 ; FMA4-NEXT:    vfnmsubpd %ymm4, %ymm3, %ymm1, %ymm1
 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8f64_fneg_fmul:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vpxord %zmm2, %zmm2, %zmm2
 ; AVX512-NEXT:    vfnmsub213pd %zmm2, %zmm1, %zmm0
 ; AVX512-NEXT:    retq
-  %m = fmul nsz <8 x double> %x, %y
-  %n = fsub <8 x double> <double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0>, %m
+  %m = fmul nrnd nexc nsz <8 x double> %x, %y
+  %n = fsub nrnd nexc <8 x double> <double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0>, %m
   ret <8 x double> %n
 }
 
 define <8 x double> @test_v8f64_fneg_fmul_no_nsz(<8 x double> %x, <8 x double> %y) #0 {
 ; FMA-LABEL: test_v8f64_fneg_fmul_no_nsz:
 ; FMA:       # BB#0:
 ; FMA-NEXT:    vmulpd %ymm3, %ymm1, %ymm1
 ; FMA-NEXT:    vmulpd %ymm2, %ymm0, %ymm0

 ; FMA4-NEXT:    vxorpd %ymm2, %ymm1, %ymm1
 ; FMA4-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v8f64_fneg_fmul_no_nsz:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vmulpd %zmm1, %zmm0, %zmm0
 ; AVX512-NEXT:    vxorpd {{.*}}(%rip), %zmm0, %zmm0
 ; AVX512-NEXT:    retq
-  %m = fmul <8 x double> %x, %y
-  %n = fsub <8 x double> <double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0>, %m
+  %m = fmul nrnd nexc <8 x double> %x, %y
+  %n = fsub nrnd nexc <8 x double> <double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0, double -0.0>, %m
   ret <8 x double> %n
 }
 
 attributes #0 = { "unsafe-fp-math"="true" }