[FPEnv][RFC] Invert sense of MIFlag::FPExcept flag

[uweigand]

Summary

In D71841 we inverted the sense of the SDNode-level flag to ensure all nodes default to potentially raising FP exceptions unless otherwise specified -- i.e. if we forget to propagate the flag somewhere, the effect is now only lost performance, not incorrect code.

However, the related flag at the MI level still defaults to nodes not raising FP exceptions unless otherwise specified. To be fully on the (conservatively) safe side, we should invert that flag as well.

This patch does so by replacing MIFlag::FPExcept with MIFlag::NoFPExcept. (Note that this does also introduce an incompatible change in the MIR format.)

This should be NFC for all MI instructions emitted by SelectionDAG. However, we could see some changes now in these cases:

• MI instructions emitted by target code (post-ISEL passes). I believe I fixed all those on SystemZ -- I'm not seeing any assembler changes at all with this patch when building a full benchmark suite. However, there may be issues on other targets.

• Code emitted by FastISel and GlobalISel. These will now default to raising exceptions for all FP instructions (on targets that already define mayRaiseFPExceptions). For FastISel, this probably doesn't matter much since this should mostly be used at -O0, where the effect of instructions raising (or not) FP exceptions should be minimal anyway. GlobalISel will definitely need to be fixed to set the NoFPExcept flag where appropriate.

The patch introduces a number of changes to X86 test cases. Some of them are expected (replace fpexcept output with nofpexcept), some are a bit surprising.

• In two places in fp-intrinsics-flags.ll, we see an LD_Fp64m80 marked nofpexcept. This is wrong (but it was already wrong before, where it should have been marked fpexcept but wasn't). This is probably due to the X86 target somewhere not copying flags correctly.

• In fast-isel-select-sse.ll, I was seeing different output for the -fast-isel vs. not case. This turned out to be yet another artifact of the differing behavior of MachineCSE. With -fast-isel, FP instructions are now marked as raising exceptions. Therefore MachineCSE no longer coalesces some COPYs into the FP instruction. Therefore, we get some differences in relative lenghts of use-def chains which lead the final heuristic in TwoAddressInstructionPass::isProfitableToCommute to come to different conclusions. In the "bad" case this means one extra move instruction in the output. It's not quite clear to me that this can be "fixed" as the "good" case seems to be just plain luck here -- the heuristic there doesn't actually address the underlying problem at all. I'm now simply using -O0 with both -fast-isel and without, which gets back to identical output (including an extra copy). Given that -fast-isel tends to be used primarily with -O0 this may be fine ...

[craig.topper]

LGTM

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

I don't know how to fix this. This is a pattern match from a any extending load instruction. There's no flag to copy from. Unfortunately, X87 generates an exception if you load a NAN from a float or double memory location. But I don't know how to represent that in SelectionDAG or IR.

[pengfei]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

SSE instructions don't generate exceptions when only loading from memory to register. Maybe other targets too. And current strict FP semantics don't define a strict load. Can we add masking the #I before FLD and FPCW recovery operations into the pattern?

[pengfei]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

Oh, we also add those instructions in non-strict scenarios. So it's not practicable.

[craig.topper]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

What happens to the SNAN or denormal value that was loaded if the exception is masked. Does the SNAN get quieted or does it stay an SNAN in 80-bit format? If masking quiets it then the SNAN would never signal. For SSE it would be signalled when the SNAN is operated on by an arithmetic instruction.

[pengfei]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

sNaN will be kept sNaN in 80-bit while qNaN to be qNaN, tested by below code:

void foo(unsigned a) {
  fedisableexcept(FE_INVALID);
  asm("fld1\nflds %0\nfwait" :: "m"(a));
  feenableexcept(FE_INVALID);
  asm("fucom\nfwait");
}

But it becomes more complicated for denormal. Because any denormal in float and double will become normal value in 80-bit format. So we should keep #D unchanged, which means we may still raise exception when loading memory in X87, and we can't keep the same behavior with SSE.

[craig.topper]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

For the SNAN test, you need to clear the exception status bits before unmasking the exception. Otherwise the exception is still pending and will be taken when its unmasked.

[pengfei]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

You are correct! They all turn to qNaN after fld. The former exception came from the pending exception.

[uweigand]

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

9

The test case starts out as just a plain @llvm.experimental.constrained.sitofp.f64.i8
This gets translated at the start of isel to

    t3: i8,ch = load<(load 1 from %fixed-stack.0)> t0, FrameIndex:i32<-1>, undef:i32
  t4: f64,ch = strict_sint_to_fp t0, t3
t6: f80 = fp_extend t4

The strict_sint_to_fp then becomes a CVTSI2SDrr (correctly marked as raising exceptions), while the fp_extend becomes a combination of MOVSDmr and LD_Fp64m80.

It seems to me the problem originates with the fp_extend -- which already should be a strict_fp_extend, really. Then the strict_fp_extend should be converted to a series of fpexcept MI instructions.

I'm not sure exactly where the fp_extend comes from, but I'd assume this is a conversion mandated by the ABI? In that case, it might make sense to check whether the function is marked with the strictfp attribute and generate strict conversions by the ABI interface code in that case.

llvm/include/llvm/CodeGen/MachineInstr.h

enum MIFlag {
     FmReassoc    = 1 << 10,             // Instruction supports Fast math
                                         // reassociation of operand order.
     NoUWrap      = 1 << 11,             // Instruction supports binary operator
                                         // no unsigned wrap.
     NoSWrap      = 1 << 12,             // Instruction supports binary operator
                                         // no signed wrap.
     IsExact      = 1 << 13,             // Instruction supports division is
                                         // known to be exact.
-    FPExcept     = 1 << 14,             // Instruction may raise floating-point
-                                        // exceptions.
+    NoFPExcept   = 1 << 14,             // Instruction does not raise
+                                        // floatint-point exceptions.
   };
 
 private:
   const MCInstrDesc *MCID;              // Instruction descriptor.
   MachineBasicBlock *Parent = nullptr;  // Pointer to the owning basic block.
 
   // Operands are allocated by an ArrayRecycler.
   MachineOperand *Operands = nullptr;   // Pointer to the first operand.
bool mayLoadOrStore(QueryType Type = AnyInBundle) const {
     return mayLoad(Type) || mayStore(Type);
   }
 
   /// Return true if this instruction could possibly raise a floating-point
   /// exception.  This is the case if the instruction is a floating-point
   /// instruction that can in principle raise an exception, as indicated
   /// by the MCID::MayRaiseFPException property, *and* at the same time,
   /// the instruction is used in a context where we expect floating-point
-  /// exceptions might be enabled, as indicated by the FPExcept MI flag.
+  /// exceptions are not disabled, as indicated by the NoFPExcept MI flag.
   bool mayRaiseFPException() const {
     return hasProperty(MCID::MayRaiseFPException) &&
-           getFlag(MachineInstr::MIFlag::FPExcept);
+           !getFlag(MachineInstr::MIFlag::NoFPExcept);
   }
 
   //===--------------------------------------------------------------------===//
   // Flags that indicate whether an instruction can be modified by a method.
   //===--------------------------------------------------------------------===//
 
   /// Return true if this may be a 2- or 3-address
   /// instruction (of the form "X = op Y, Z, ..."), which produces the same

llvm/lib/CodeGen/MIRParser/MILexer.h

enum TokenKind {
     kw_nsz,
     kw_arcp,
     kw_contract,
     kw_afn,
     kw_reassoc,
     kw_nuw,
     kw_nsw,
     kw_exact,
-    kw_fpexcept,
+    kw_nofpexcept,
     kw_debug_location,
     kw_cfi_same_value,
     kw_cfi_offset,
     kw_cfi_rel_offset,
     kw_cfi_def_cfa_register,
     kw_cfi_def_cfa_offset,
     kw_cfi_adjust_cfa_offset,
     kw_cfi_escape,

llvm/lib/CodeGen/MIRParser/MILexer.cpp

return StringSwitch<MIToken::TokenKind>(Identifier)
       .Case("nsz", MIToken::kw_nsz)
       .Case("arcp", MIToken::kw_arcp)
       .Case("contract", MIToken::kw_contract)
       .Case("afn", MIToken::kw_afn)
       .Case("reassoc", MIToken::kw_reassoc)
       .Case("nuw" , MIToken::kw_nuw)
       .Case("nsw" , MIToken::kw_nsw)
       .Case("exact" , MIToken::kw_exact)
-      .Case("fpexcept", MIToken::kw_fpexcept)
+      .Case("nofpexcept", MIToken::kw_nofpexcept)
       .Case("debug-location", MIToken::kw_debug_location)
       .Case("same_value", MIToken::kw_cfi_same_value)
       .Case("offset", MIToken::kw_cfi_offset)
       .Case("rel_offset", MIToken::kw_cfi_rel_offset)
       .Case("def_cfa_register", MIToken::kw_cfi_def_cfa_register)
       .Case("def_cfa_offset", MIToken::kw_cfi_def_cfa_offset)
       .Case("adjust_cfa_offset", MIToken::kw_cfi_adjust_cfa_offset)
       .Case("escape", MIToken::kw_cfi_escape)

llvm/lib/CodeGen/MIRParser/MIParser.cpp

while (Token.is(MIToken::kw_frame_setup) ||
          Token.is(MIToken::kw_nsz) ||
          Token.is(MIToken::kw_arcp) ||
          Token.is(MIToken::kw_contract) ||
          Token.is(MIToken::kw_afn) ||
          Token.is(MIToken::kw_reassoc) ||
          Token.is(MIToken::kw_nuw) ||
          Token.is(MIToken::kw_nsw) ||
          Token.is(MIToken::kw_exact) ||
-         Token.is(MIToken::kw_fpexcept)) {
+         Token.is(MIToken::kw_nofpexcept)) {
     // Mine frame and fast math flags
     if (Token.is(MIToken::kw_frame_setup))
       Flags |= MachineInstr::FrameSetup;
     if (Token.is(MIToken::kw_frame_destroy))
       Flags |= MachineInstr::FrameDestroy;
     if (Token.is(MIToken::kw_nnan))
       Flags |= MachineInstr::FmNoNans;
     if (Token.is(MIToken::kw_ninf))
while (Token.is(MIToken::kw_frame_setup) ||
     if (Token.is(MIToken::kw_reassoc))
       Flags |= MachineInstr::FmReassoc;
     if (Token.is(MIToken::kw_nuw))
       Flags |= MachineInstr::NoUWrap;
     if (Token.is(MIToken::kw_nsw))
       Flags |= MachineInstr::NoSWrap;
     if (Token.is(MIToken::kw_exact))
       Flags |= MachineInstr::IsExact;
-    if (Token.is(MIToken::kw_fpexcept))
-      Flags |= MachineInstr::FPExcept;
+    if (Token.is(MIToken::kw_nofpexcept))
+      Flags |= MachineInstr::NoFPExcept;
 
     lex();
   }
   if (Token.isNot(MIToken::Identifier))
     return error("expected a machine instruction");
   StringRef InstrName = Token.stringValue();
   if (PFS.Target.parseInstrName(InstrName, OpCode))
     return error(Twine("unknown machine instruction name '") + InstrName + "'");

llvm/lib/CodeGen/MIRPrinter.cpp

void MIPrinter::print(const MachineInstr &MI) {
   if (MI.getFlag(MachineInstr::FmReassoc))
     OS << "reassoc ";
   if (MI.getFlag(MachineInstr::NoUWrap))
     OS << "nuw ";
   if (MI.getFlag(MachineInstr::NoSWrap))
     OS << "nsw ";
   if (MI.getFlag(MachineInstr::IsExact))
     OS << "exact ";
-  if (MI.getFlag(MachineInstr::FPExcept))
-    OS << "fpexcept ";
+  if (MI.getFlag(MachineInstr::NoFPExcept))
+    OS << "nofpexcept ";
 
   OS << TII->getName(MI.getOpcode());
   if (I < E)
     OS << ' ';
 
   bool NeedComma = false;
   for (; I < E; ++I) {
     if (NeedComma)

llvm/lib/CodeGen/MachineInstr.cpp

void MachineInstr::print(raw_ostream &OS, ModuleSlotTracker &MST,
   if (getFlag(MachineInstr::FmReassoc))
     OS << "reassoc ";
   if (getFlag(MachineInstr::NoUWrap))
     OS << "nuw ";
   if (getFlag(MachineInstr::NoSWrap))
     OS << "nsw ";
   if (getFlag(MachineInstr::IsExact))
     OS << "exact ";
-  if (getFlag(MachineInstr::FPExcept))
-    OS << "fpexcept ";
+  if (getFlag(MachineInstr::NoFPExcept))
+    OS << "nofpexcept ";
 
   // Print the opcode name.
   if (TII)
     OS << TII->getName(getOpcode());
   else
     OS << "UNKNOWN";
 
   if (SkipOpers)

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

if (Flags.hasNoUnsignedWrap())
       MI->setFlag(MachineInstr::MIFlag::NoUWrap);
 
     if (Flags.hasNoSignedWrap())
       MI->setFlag(MachineInstr::MIFlag::NoSWrap);
 
     if (Flags.hasExact())
       MI->setFlag(MachineInstr::MIFlag::IsExact);
 
-    if (MI->getDesc().mayRaiseFPException() && !Flags.hasNoFPExcept())
-      MI->setFlag(MachineInstr::MIFlag::FPExcept);
+    if (Flags.hasNoFPExcept())
+      MI->setFlag(MachineInstr::MIFlag::NoFPExcept);
   }
 
   // Emit all of the actual operands of this instruction, adding them to the
   // instruction as appropriate.
   bool HasOptPRefs = NumDefs > NumResults;
   assert((!HasOptPRefs || !HasPhysRegOuts) &&
          "Unable to cope with optional defs and phys regs defs!");
   unsigned NumSkip = HasOptPRefs ? NumDefs - NumResults : 0;

llvm/lib/Target/SystemZ/SystemZElimCompare.cpp

bool SystemZElimCompare::convertToLoadAndTest(
 
   // Rebuild to get the CC operand in the right place.
   auto MIB = BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(Opcode));
   for (const auto &MO : MI.operands())
     MIB.add(MO);
   MIB.setMemRefs(MI.memoperands());
   MI.eraseFromParent();
 
-  // Mark instruction as raising an FP exception if applicable.  We already
+  // Mark instruction as not raising an FP exception if applicable.  We already
   // verified earlier that this move is valid.
-  if (Compare.mayRaiseFPException())
-    MIB.setMIFlag(MachineInstr::MIFlag::FPExcept);
+  if (!Compare.mayRaiseFPException())
+    MIB.setMIFlag(MachineInstr::MIFlag::NoFPExcept);
 
   return true;
 }
 
 // See if MI is an instruction with an equivalent "logical" opcode that can
 // be used and replace MI. This is useful for EQ/NE comparisons where the
 // "nsw" flag is missing since the "logical" opcode always sets CC to reflect
 // the result being zero or non-zero.

llvm/test/CodeGen/X86/fast-isel-select-sse.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs                                          | FileCheck %s --check-prefix=SSE
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs -fast-isel -fast-isel-abort=1            | FileCheck %s --check-prefix=SSE
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs -O0                                          | FileCheck %s --check-prefix=SSE
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs -O0 -fast-isel -fast-isel-abort=1            | FileCheck %s --check-prefix=SSE
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs                               -mattr=avx | FileCheck %s --check-prefix=AVX
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs -fast-isel -fast-isel-abort=1 -mattr=avx | FileCheck %s --check-prefix=AVX
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs                               -mattr=avx512f | FileCheck %s --check-prefix=AVX512
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -verify-machineinstrs -fast-isel -fast-isel-abort=1 -mattr=avx512f | FileCheck %s --check-prefix=AVX512
 
 ; Test all cmp predicates that can be used with SSE.
 
 define float @select_fcmp_oeq_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_oeq_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpeqss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_oeq_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpeqss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_oeq_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpeqss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp oeq float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_oeq_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_oeq_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpeqsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_oeq_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpeqsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_oeq_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpeqsd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp oeq double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ogt_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ogt_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpltss %xmm0, %xmm1
-; SSE-NEXT:    andps %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andps %xmm2, %xmm0
 ; SSE-NEXT:    andnps %xmm3, %xmm1
-; SSE-NEXT:    orps %xmm2, %xmm1
+; SSE-NEXT:    orps %xmm0, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ogt_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpltss %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ogt_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpltss %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ogt float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ogt_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ogt_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpltsd %xmm0, %xmm1
-; SSE-NEXT:    andpd %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andpd %xmm2, %xmm0
 ; SSE-NEXT:    andnpd %xmm3, %xmm1
-; SSE-NEXT:    orpd %xmm2, %xmm1
-; SSE-NEXT:    movapd %xmm1, %xmm0
+; SSE-NEXT:    orpd %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ogt_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpltsd %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ogt_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpltsd %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ogt double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_oge_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_oge_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpless %xmm0, %xmm1
-; SSE-NEXT:    andps %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andps %xmm2, %xmm0
 ; SSE-NEXT:    andnps %xmm3, %xmm1
-; SSE-NEXT:    orps %xmm2, %xmm1
+; SSE-NEXT:    orps %xmm0, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_oge_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpless %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_oge_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpless %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp oge float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_oge_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_oge_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmplesd %xmm0, %xmm1
-; SSE-NEXT:    andpd %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andpd %xmm2, %xmm0
 ; SSE-NEXT:    andnpd %xmm3, %xmm1
-; SSE-NEXT:    orpd %xmm2, %xmm1
-; SSE-NEXT:    movapd %xmm1, %xmm0
+; SSE-NEXT:    orpd %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_oge_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmplesd %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_oge_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmplesd %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp oge double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_olt_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_olt_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpltss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_olt_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpltss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_olt_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpltss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp olt float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_olt_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_olt_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpltsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_olt_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpltsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_olt_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpltsd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp olt double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ole_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ole_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpless %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ole_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpless %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ole_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpless %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ole float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ole_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ole_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmplesd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ole_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmplesd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ole_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmplesd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ole double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ord_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ord_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpordss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ord_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpordss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ord_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpordss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ord float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ord_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ord_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpordsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ord_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpordsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ord_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpordsd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ord double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_uno_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_uno_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpunordss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_uno_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpunordss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_uno_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpunordss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp uno float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_uno_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_uno_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpunordsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_uno_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpunordsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_uno_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpunordsd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp uno double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ugt_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ugt_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnless %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ugt_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnless %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ugt_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnless %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ugt float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ugt_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ugt_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnlesd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ugt_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnlesd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ugt_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnlesd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ugt double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_uge_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_uge_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnltss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_uge_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnltss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_uge_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnltss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp uge float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_uge_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_uge_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnltsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_uge_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnltsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_uge_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnltsd %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp uge double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ult_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ult_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnless %xmm0, %xmm1
-; SSE-NEXT:    andps %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andps %xmm2, %xmm0
 ; SSE-NEXT:    andnps %xmm3, %xmm1
-; SSE-NEXT:    orps %xmm2, %xmm1
+; SSE-NEXT:    orps %xmm0, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ult_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnless %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ult_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnless %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ult float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ult_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ult_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnlesd %xmm0, %xmm1
-; SSE-NEXT:    andpd %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andpd %xmm2, %xmm0
 ; SSE-NEXT:    andnpd %xmm3, %xmm1
-; SSE-NEXT:    orpd %xmm2, %xmm1
-; SSE-NEXT:    movapd %xmm1, %xmm0
+; SSE-NEXT:    orpd %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ult_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnlesd %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ult_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnlesd %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ult double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_ule_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_ule_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnltss %xmm0, %xmm1
-; SSE-NEXT:    andps %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andps %xmm2, %xmm0
 ; SSE-NEXT:    andnps %xmm3, %xmm1
-; SSE-NEXT:    orps %xmm2, %xmm1
+; SSE-NEXT:    orps %xmm0, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ule_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnltss %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ule_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnltss %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ule float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_ule_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_ule_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpnltsd %xmm0, %xmm1
-; SSE-NEXT:    andpd %xmm1, %xmm2
+; SSE-NEXT:    movaps %xmm1, %xmm0
+; SSE-NEXT:    andpd %xmm2, %xmm0
 ; SSE-NEXT:    andnpd %xmm3, %xmm1
-; SSE-NEXT:    orpd %xmm2, %xmm1
-; SSE-NEXT:    movapd %xmm1, %xmm0
+; SSE-NEXT:    orpd %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_ule_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpnltsd %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_ule_f64:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpnltsd %xmm0, %xmm1, %k1
 ; AVX512-NEXT:    vmovsd %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovapd %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp ule double %a, %b
   %2 = select i1 %1, double %c, double %d
   ret double %2
 }
 
 define float @select_fcmp_une_f32(float %a, float %b, float %c, float %d) {
 ; SSE-LABEL: select_fcmp_une_f32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpneqss %xmm1, %xmm0
-; SSE-NEXT:    andps %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andps %xmm2, %xmm1
 ; SSE-NEXT:    andnps %xmm3, %xmm0
-; SSE-NEXT:    orps %xmm2, %xmm0
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_une_f32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpneqss %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvps %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: select_fcmp_une_f32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vcmpneqss %xmm1, %xmm0, %k1
 ; AVX512-NEXT:    vmovss %xmm2, %xmm3, %xmm3 {%k1}
 ; AVX512-NEXT:    vmovaps %xmm3, %xmm0
 ; AVX512-NEXT:    retq
   %1 = fcmp une float %a, %b
   %2 = select i1 %1, float %c, float %d
   ret float %2
 }
 
 define double @select_fcmp_une_f64(double %a, double %b, double %c, double %d) {
 ; SSE-LABEL: select_fcmp_une_f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpneqsd %xmm1, %xmm0
-; SSE-NEXT:    andpd %xmm0, %xmm2
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    andpd %xmm2, %xmm1
 ; SSE-NEXT:    andnpd %xmm3, %xmm0
-; SSE-NEXT:    orpd %xmm2, %xmm0
+; SSE-NEXT:    orpd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: select_fcmp_une_f64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vcmpneqsd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vblendvpd %xmm0, %xmm2, %xmm3, %xmm0
 ; AVX-NEXT:    retq
 ;

llvm/test/CodeGen/X86/fp-intrinsics-flags-x86_64.ll

 ; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s -stop-after=finalize-isel | FileCheck %s
 
 define i32 @f20u(double %x) #0 {
 ; CHECK-LABEL: name: f20u
 ; CHECK: liveins: $xmm0
 ; CHECK: [[COPY:%[0-9]+]]:fr64 = COPY $xmm0
-; CHECK: [[CVTTSD2SI64rr:%[0-9]+]]:gr64 = fpexcept CVTTSD2SI64rr [[COPY]], implicit $mxcsr
+; CHECK: [[CVTTSD2SI64rr:%[0-9]+]]:gr64 = CVTTSD2SI64rr [[COPY]], implicit $mxcsr
 ; CHECK: [[COPY1:%[0-9]+]]:gr32 = COPY [[CVTTSD2SI64rr]].sub_32bit
 ; CHECK: $eax = COPY [[COPY1]]
 ; CHECK: RET 0, $eax
 entry:
   %result = call i32 @llvm.experimental.constrained.fptoui.i32.f64(double %x, metadata !"fpexcept.strict") #0
   ret i32 %result
 }
 
 attributes #0 = { strictfp }
 
 declare i32 @llvm.experimental.constrained.fptoui.i32.f64(double, metadata)

llvm/test/CodeGen/X86/fp-intrinsics-flags.ll

 ; RUN: llc -O3 -mtriple=i686-pc-linux -mattr=sse2 -stop-after=finalize-isel < %s | FileCheck %s
 
 define double @sifdb(i8 %x) #0 {
 entry:
 ; CHECK-LABEL: name: sifdb
 ; CHECK: [[MOVSX32rm8_:%[0-9]+]]:gr32 = MOVSX32rm8 %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 1 from %fixed-stack.0, align 16)
 ; CHECK: [[CVTSI2SDrr:%[0-9]+]]:fr64 = CVTSI2SDrr killed [[MOVSX32rm8_]]
 ; CHECK: MOVSDmr %stack.0, 1, $noreg, 0, $noreg, killed [[CVTSI2SDrr]] :: (store 8 into %stack.0, align 4)
-; CHECK: [[LD_Fp64m80_:%[0-9]+]]:rfp80 = LD_Fp64m80 %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0, align 4)
+; CHECK: [[LD_Fp64m80_:%[0-9]+]]:rfp80 = nofpexcept LD_Fp64m80 %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0, align 4)
 ; CHECK: RET 0, killed [[LD_Fp64m80_]]
   %result = call double @llvm.experimental.constrained.sitofp.f64.i8(i8 %x, metadata !"round.dynamic", metadata !"fpexcept.strict") #0
   ret double %result
 }
 
 define double @sifdw(i16 %x) #0 {
 entry:
 ; CHECK-LABEL: name: sifdw
 ; CHECK: [[MOVSX32rm16_:%[0-9]+]]:gr32 = MOVSX32rm16 %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 2 from %fixed-stack.0, align 16)
 ; CHECK: [[CVTSI2SDrr:%[0-9]+]]:fr64 = CVTSI2SDrr killed [[MOVSX32rm16_]]
 ; CHECK: MOVSDmr %stack.0, 1, $noreg, 0, $noreg, killed [[CVTSI2SDrr]] :: (store 8 into %stack.0, align 4)
-; CHECK: [[LD_Fp64m80_:%[0-9]+]]:rfp80 = LD_Fp64m80 %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0, align 4)
+; CHECK: [[LD_Fp64m80_:%[0-9]+]]:rfp80 = nofpexcept LD_Fp64m80 %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0, align 4)
 ; CHECK: RET 0, killed [[LD_Fp64m80_]]
   %result = call double @llvm.experimental.constrained.sitofp.f64.i16(i16 %x, metadata !"round.dynamic", metadata !"fpexcept.strict") #0
   ret double %result
 }
 
 define i64 @f20u64(double %x) #0 {
 entry:
 ; CHECK-LABEL: name: f20u64
 ; CHECK: [[MOVSDrm_alt:%[0-9]+]]:fr64 = MOVSDrm_alt %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.0, align 16)
 ; CHECK: [[MOVSDrm_alt1:%[0-9]+]]:fr64 = MOVSDrm_alt $noreg, 1, $noreg, %const.0, $noreg :: (load 8 from constant-pool)
-; CHECK: fpexcept COMISDrr [[MOVSDrm_alt1]], [[MOVSDrm_alt]], implicit-def $eflags, implicit $mxcsr
+; CHECK: COMISDrr [[MOVSDrm_alt1]], [[MOVSDrm_alt]], implicit-def $eflags, implicit $mxcsr
 ; CHECK: [[FsFLD0SD:%[0-9]+]]:fr64 = FsFLD0SD
 ; CHECK: JCC_1
 ; CHECK: [[PHI:%[0-9]+]]:fr64 = PHI [[MOVSDrm_alt1]], {{.*}}, [[FsFLD0SD]], {{.*}}
-; CHECK: [[SUBSDrr:%[0-9]+]]:fr64 = fpexcept SUBSDrr [[MOVSDrm_alt]], killed [[PHI]], implicit $mxcsr
+; CHECK: [[SUBSDrr:%[0-9]+]]:fr64 = SUBSDrr [[MOVSDrm_alt]], killed [[PHI]], implicit $mxcsr
 ; CHECK: MOVSDmr %stack.0, 1, $noreg, 0, $noreg, killed [[SUBSDrr]] :: (store 8 into %stack.0)
 ; CHECK: [[SETCCr:%[0-9]+]]:gr8 = SETCCr 6, implicit $eflags
-; CHECK: [[LD_Fp64m:%[0-9]+]]:rfp64 = fpexcept LD_Fp64m %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0)
+; CHECK: [[LD_Fp64m:%[0-9]+]]:rfp64 = LD_Fp64m %stack.0, 1, $noreg, 0, $noreg, implicit-def dead $fpsw, implicit $fpcw :: (load 8 from %stack.0)
 ; CHECK: FNSTCW16m %stack.1, 1, $noreg, 0, $noreg, implicit-def $fpsw, implicit $fpcw :: (store 2 into %stack.1)
 ; CHECK: [[MOVZX32rm16_:%[0-9]+]]:gr32 = MOVZX32rm16 %stack.1, 1, $noreg, 0, $noreg :: (load 2 from %stack.1)
 ; CHECK: [[OR32ri:%[0-9]+]]:gr32 = OR32ri killed [[MOVZX32rm16_]], 3072, implicit-def $eflags
 ; CHECK: [[COPY3:%[0-9]+]]:gr16 = COPY killed [[OR32ri]].sub_16bit
 ; CHECK: MOV16mr %stack.2, 1, $noreg, 0, $noreg, killed [[COPY3]] :: (store 2 into %stack.2)
 ; CHECK: FLDCW16m %stack.2, 1, $noreg, 0, $noreg, implicit-def $fpsw, implicit-def $fpcw :: (load 2 from %stack.2)
 ; CHECK: IST_Fp64m64 %stack.0, 1, $noreg, 0, $noreg, [[LD_Fp64m]], implicit-def $fpsw, implicit $fpcw
 ; CHECK: FLDCW16m %stack.1, 1, $noreg, 0, $noreg, implicit-def $fpsw, implicit-def $fpcw :: (load 2 from %stack.1)
 ; CHECK: [[MOVZX32rr8_:%[0-9]+]]:gr32 = MOVZX32rr8 killed [[SETCCr]]
 ; CHECK: [[SHL32ri:%[0-9]+]]:gr32 = SHL32ri [[MOVZX32rr8_]], 31, implicit-def dead $eflags
 ; CHECK: [[XOR32rm:%[0-9]+]]:gr32 = XOR32rm [[SHL32ri]], %stack.0, 1, $noreg, 4, $noreg, implicit-def dead $eflags :: (load 4 from %stack.0 + 4)
 ; CHECK: [[MOV32rm:%[0-9]+]]:gr32 = MOV32rm %stack.0, 1, $noreg, 0, $noreg :: (load 4 from %stack.0, align 8)
 ; CHECK: $eax = COPY [[MOV32rm]]
 ; CHECK: $edx = COPY [[XOR32rm]]
 ; CHECK: RET 0, $eax, $edx
   %result = call i64 @llvm.experimental.constrained.fptoui.i64.f64(double %x, metadata !"fpexcept.strict") #0
   ret i64 %result
 }
 
 define i8 @f20s8(double %x) #0 {
 entry:
 ; CHECK-LABEL: name: f20s8
-; CHECK: [[CVTTSD2SIrm:%[0-9]+]]:gr32 = fpexcept CVTTSD2SIrm %fixed-stack.0, 1, $noreg, 0, $noreg, implicit $mxcsr :: (load 8 from %fixed-stack.0, align 16)
+; CHECK: [[CVTTSD2SIrm:%[0-9]+]]:gr32 = CVTTSD2SIrm %fixed-stack.0, 1, $noreg, 0, $noreg, implicit $mxcsr :: (load 8 from %fixed-stack.0, align 16)
 ; CHECK: [[COPY:%[0-9]+]]:gr32_abcd = COPY [[CVTTSD2SIrm]]
 ; CHECK: [[COPY1:%[0-9]+]]:gr8 = COPY [[COPY]].sub_8bit
 ; CHECK: $al = COPY [[COPY1]]
 ; CHECK: RET 0, $al
   %result = call i8 @llvm.experimental.constrained.fptosi.i8.f64(double %x, metadata !"fpexcept.strict") #0
   ret i8 %result
 }
 
 define i16 @f20s16(double %x) #0 {
 entry:
 ; CHECK-LABEL: name: f20s16
-; CHECK: [[CVTTSD2SIrm:%[0-9]+]]:gr32 = fpexcept CVTTSD2SIrm %fixed-stack.0, 1, $noreg, 0, $noreg, implicit $mxcsr :: (load 8 from %fixed-stack.0, align 16)
+; CHECK: [[CVTTSD2SIrm:%[0-9]+]]:gr32 = CVTTSD2SIrm %fixed-stack.0, 1, $noreg, 0, $noreg, implicit $mxcsr :: (load 8 from %fixed-stack.0, align 16)
 ; CHECK: [[COPY:%[0-9]+]]:gr16 = COPY [[CVTTSD2SIrm]].sub_16bit
 ; CHECK: $ax = COPY [[COPY]]
 ; CHECK: RET 0, $ax
   %result = call i16 @llvm.experimental.constrained.fptosi.i16.f64(double %x, metadata !"fpexcept.strict") #0
   ret i16 %result
 }
 
 define i32 @f20u(double %x) #0 {
 entry:
 ; CHECK-LABEL: name: f20u
 ; CHECK: [[MOVSDrm_alt:%[0-9]+]]:fr64 = MOVSDrm_alt %fixed-stack.0, 1, $noreg, 0, $noreg :: (load 8 from %fixed-stack.0, align 16)
 ; CHECK: [[MOVSDrm_alt1:%[0-9]+]]:fr64 = MOVSDrm_alt $noreg, 1, $noreg, %const.0, $noreg :: (load 8 from constant-pool)
-; CHECK: fpexcept COMISDrr [[MOVSDrm_alt1]], [[MOVSDrm_alt]], implicit-def $eflags, implicit $mxcsr
+; CHECK: COMISDrr [[MOVSDrm_alt1]], [[MOVSDrm_alt]], implicit-def $eflags, implicit $mxcsr
 ; CHECK: [[FsFLD0SD:%[0-9]+]]:fr64 = FsFLD0SD
 ; CHECK: JCC_1
 ; CHECK: [[PHI:%[0-9]+]]:fr64 = PHI [[MOVSDrm_alt1]], {{.*}}, [[FsFLD0SD]], {{.*}}
 ; CHECK: [[SETCCr:%[0-9]+]]:gr8 = SETCCr 6, implicit $eflags
 ; CHECK: [[MOVZX32rr8_:%[0-9]+]]:gr32 = MOVZX32rr8 killed [[SETCCr]]
 ; CHECK: [[SHL32ri:%[0-9]+]]:gr32 = SHL32ri [[MOVZX32rr8_]], 31, implicit-def dead $eflags
-; CHECK: [[SUBSDrr:%[0-9]+]]:fr64 = fpexcept SUBSDrr [[MOVSDrm_alt]], killed [[PHI]], implicit $mxcsr
-; CHECK: [[CVTTSD2SIrr:%[0-9]+]]:gr32 = fpexcept CVTTSD2SIrr killed [[SUBSDrr]], implicit $mxcsr
+; CHECK: [[SUBSDrr:%[0-9]+]]:fr64 = SUBSDrr [[MOVSDrm_alt]], killed [[PHI]], implicit $mxcsr
+; CHECK: [[CVTTSD2SIrr:%[0-9]+]]:gr32 = CVTTSD2SIrr killed [[SUBSDrr]], implicit $mxcsr
 ; CHECK: [[XOR32rr:%[0-9]+]]:gr32 = XOR32rr [[CVTTSD2SIrr]], killed [[SHL32ri]], implicit-def dead $eflags
 ; CHECK: $eax = COPY [[XOR32rr]]
 ; CHECK: RET 0, $eax
   %result = call i32 @llvm.experimental.constrained.fptoui.i32.f64(double %x, metadata !"fpexcept.strict") #0
   ret i32 %result
 }
 
 attributes #0 = { strictfp }
 
 declare double @llvm.experimental.constrained.sitofp.f64.i8(i8, metadata, metadata)
 declare double @llvm.experimental.constrained.sitofp.f64.i16(i16, metadata, metadata)
 declare i32 @llvm.experimental.constrained.fptoui.i32.f64(double, metadata)
 declare i64 @llvm.experimental.constrained.fptoui.i64.f64(double, metadata)
 declare i8 @llvm.experimental.constrained.fptosi.i8.f64(double, metadata)
 declare i16 @llvm.experimental.constrained.fptosi.i16.f64(double, metadata)

llvm/test/CodeGen/X86/sqrt-fastmath-mir.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2,fma -stop-after=finalize-isel 2>&1 | FileCheck %s
 
 declare float @llvm.sqrt.f32(float) #0
 
 define float @foo(float %f) #0 {
 ; CHECK: {{name: *foo}}
 ; CHECK: body:
 ; CHECK:     %0:fr32 = COPY $xmm0
 ; CHECK:     %1:fr32 = VRSQRTSSr killed %2, %0
-; CHECK:     %3:fr32 = VMULSSrr %0, %1
+; CHECK:     %3:fr32 = nofpexcept VMULSSrr %0, %1
 ; CHECK:     %4:fr32 = VMOVSSrm
-; CHECK:     %5:fr32 = VFMADD213SSr %1, killed %3, %4
+; CHECK:     %5:fr32 = nofpexcept VFMADD213SSr %1, killed %3, %4
 ; CHECK:     %6:fr32 = VMOVSSrm
-; CHECK:     %7:fr32 = VMULSSrr %1, %6
-; CHECK:     %8:fr32 = VMULSSrr killed %7, killed %5
-; CHECK:     %9:fr32 = VMULSSrr %0, %8
-; CHECK:     %10:fr32 = VFMADD213SSr %8, %9, %4
-; CHECK:     %11:fr32 = VMULSSrr %9, %6
-; CHECK:     %12:fr32 = VMULSSrr killed %11, killed %10
+; CHECK:     %7:fr32 = nofpexcept VMULSSrr %1, %6
+; CHECK:     %8:fr32 = nofpexcept VMULSSrr killed %7, killed %5
+; CHECK:     %9:fr32 = nofpexcept VMULSSrr %0, %8
+; CHECK:     %10:fr32 = nofpexcept VFMADD213SSr %8, %9, %4
+; CHECK:     %11:fr32 = nofpexcept VMULSSrr %9, %6
+; CHECK:     %12:fr32 = nofpexcept VMULSSrr killed %11, killed %10
 ; CHECK:     %14:fr32 = FsFLD0SS
-; CHECK:     %15:fr32 = VCMPSSrr %0, killed %14, 0
+; CHECK:     %15:fr32 = nofpexcept VCMPSSrr %0, killed %14, 0
 ; CHECK:     %17:vr128 = VPANDNrr killed %16, killed %13
 ; CHECK:     $xmm0 = COPY %18
 ; CHECK:     RET 0, $xmm0
   %call = tail call float @llvm.sqrt.f32(float %f) #1
   ret float %call
 }
 
 define float @rfoo(float %f) #0 {
 ; CHECK: {{name: *rfoo}}
 ; CHECK: body:             |
 ; CHECK:     %0:fr32 = COPY $xmm0
 ; CHECK:     %1:fr32 = VRSQRTSSr killed %2, %0
-; CHECK:     %3:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr %0, %1
+; CHECK:     %3:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr %0, %1
 ; CHECK:     %4:fr32 = VMOVSSrm
-; CHECK:     %5:fr32 = nnan ninf nsz arcp contract afn reassoc VFMADD213SSr %1, killed %3, %4
+; CHECK:     %5:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VFMADD213SSr %1, killed %3, %4
 ; CHECK:     %6:fr32 = VMOVSSrm
-; CHECK:     %7:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr %1, %6
-; CHECK:     %8:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr killed %7, killed %5
-; CHECK:     %9:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr %0, %8
-; CHECK:     %10:fr32 = nnan ninf nsz arcp contract afn reassoc VFMADD213SSr %8, killed %9, %4
-; CHECK:     %11:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr %8, %6
-; CHECK:     %12:fr32 = nnan ninf nsz arcp contract afn reassoc VMULSSrr killed %11, killed %10
+; CHECK:     %7:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr %1, %6
+; CHECK:     %8:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr killed %7, killed %5
+; CHECK:     %9:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr %0, %8
+; CHECK:     %10:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VFMADD213SSr %8, killed %9, %4
+; CHECK:     %11:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr %8, %6
+; CHECK:     %12:fr32 = nnan ninf nsz arcp contract afn reassoc nofpexcept VMULSSrr killed %11, killed %10
 ; CHECK:     $xmm0 = COPY %12
 ; CHECK:     RET 0, $xmm0
   %sqrt = tail call float @llvm.sqrt.f32(float %f)
   %div = fdiv fast float 1.0, %sqrt
   ret float %div
 }
 
 attributes #0 = { "unsafe-fp-math"="true" "reciprocal-estimates"="sqrt:2" }
 attributes #1 = { nounwind readnone }

llvm/test/CodeGen/X86/vector-constrained-fp-intrinsics-flags.ll

 ; RUN: llc -O3 -mtriple=x86_64-pc-linux -stop-after=finalize-isel < %s | FileCheck %s
 
 define <1 x float> @constrained_vector_fadd_v1f32() #0 {
 ; CHECK-LABEL: name: constrained_vector_fadd_v1f32
 ; CHECK: [[MOVSSrm_alt:%[0-9]+]]:fr32 = MOVSSrm_alt $rip, 1, $noreg, %const.0, $noreg :: (load 4 from constant-pool)
-; CHECK: [[ADDSSrm:%[0-9]+]]:fr32 = fpexcept ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
+; CHECK: [[ADDSSrm:%[0-9]+]]:fr32 = ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
 ; CHECK: $xmm0 = COPY [[ADDSSrm]]
 ; CHECK: RET 0, $xmm0
 entry:
   %add = call <1 x float> @llvm.experimental.constrained.fadd.v1f32(<1 x float> <float 0x7FF0000000000000>, <1 x float> <float 1.0>, metadata !"round.dynamic", metadata !"fpexcept.strict") #0
   ret <1 x float> %add
 }
 
 define <3 x float> @constrained_vector_fadd_v3f32() #0 {
 ; CHECK-LABEL: name: constrained_vector_fadd_v3f32
 ; CHECK: [[FsFLD0SS:%[0-9]+]]:fr32 = FsFLD0SS
 ; CHECK: [[MOVSSrm_alt:%[0-9]+]]:fr32 = MOVSSrm_alt $rip, 1, $noreg, %const.0, $noreg :: (load 4 from constant-pool)
-; CHECK: [[ADDSSrr:%[0-9]+]]:fr32 = fpexcept ADDSSrr [[MOVSSrm_alt]], killed [[FsFLD0SS]], implicit $mxcsr
-; CHECK: [[ADDSSrm:%[0-9]+]]:fr32 = fpexcept ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
-; CHECK: [[ADDSSrm1:%[0-9]+]]:fr32 = fpexcept ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.2, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
+; CHECK: [[ADDSSrr:%[0-9]+]]:fr32 = ADDSSrr [[MOVSSrm_alt]], killed [[FsFLD0SS]], implicit $mxcsr
+; CHECK: [[ADDSSrm:%[0-9]+]]:fr32 = ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
+; CHECK: [[ADDSSrm1:%[0-9]+]]:fr32 = ADDSSrm [[MOVSSrm_alt]], $rip, 1, $noreg, %const.2, $noreg, implicit $mxcsr :: (load 4 from constant-pool)
 ; CHECK: [[COPY:%[0-9]+]]:vr128 = COPY [[ADDSSrm1]]
 ; CHECK: [[COPY1:%[0-9]+]]:vr128 = COPY [[ADDSSrm]]
 ; CHECK: [[UNPCKLPSrr:%[0-9]+]]:vr128 = UNPCKLPSrr [[COPY1]], killed [[COPY]]
 ; CHECK: [[COPY2:%[0-9]+]]:vr128 = COPY [[ADDSSrr]]
 ; CHECK: [[UNPCKLPDrr:%[0-9]+]]:vr128 = UNPCKLPDrr [[UNPCKLPSrr]], killed [[COPY2]]
 ; CHECK: $xmm0 = COPY [[UNPCKLPDrr]]
 ; CHECK: RET 0, $xmm0
 entry:
   %add = call <3 x float> @llvm.experimental.constrained.fadd.v3f32(
            <3 x float> <float 0xFFFFFFFFE0000000, float 0xFFFFFFFFE0000000,
                         float 0xFFFFFFFFE0000000>,
            <3 x float> <float 2.0, float 1.0, float 0.0>,
            metadata !"round.dynamic",
            metadata !"fpexcept.strict") #0
   ret <3 x float> %add
 }
 
 define <4 x double> @constrained_vector_fadd_v4f64() #0 {
 ; CHECK-LABEL: name: constrained_vector_fadd_v4f64
 ; CHECK: [[MOVAPDrm:%[0-9]+]]:vr128 = MOVAPDrm $rip, 1, $noreg, %const.0, $noreg :: (load 16 from constant-pool)
-; CHECK: [[ADDPDrm:%[0-9]+]]:vr128 = fpexcept ADDPDrm [[MOVAPDrm]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 16 from constant-pool)
-; CHECK: [[ADDPDrm1:%[0-9]+]]:vr128 = fpexcept ADDPDrm [[MOVAPDrm]], $rip, 1, $noreg, %const.2, $noreg, implicit $mxcsr :: (load 16 from constant-pool)
+; CHECK: [[ADDPDrm:%[0-9]+]]:vr128 = ADDPDrm [[MOVAPDrm]], $rip, 1, $noreg, %const.1, $noreg, implicit $mxcsr :: (load 16 from constant-pool)
+; CHECK: [[ADDPDrm1:%[0-9]+]]:vr128 = ADDPDrm [[MOVAPDrm]], $rip, 1, $noreg, %const.2, $noreg, implicit $mxcsr :: (load 16 from constant-pool)
 ; CHECK: $xmm0 = COPY [[ADDPDrm]]
 ; CHECK: $xmm1 = COPY [[ADDPDrm1]]
 ; CHECK: RET 0, $xmm0, $xmm1
 entry:
   %add = call <4 x double> @llvm.experimental.constrained.fadd.v4f64(
            <4 x double> <double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF,
                          double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF>,
            <4 x double> <double 1.000000e+00, double 1.000000e-01,