[X86] Combine (cmov (and/or (setcc) (setcc))) into (cmov (cmov))

[ab]

Summary

Fold and/or of setcc's to double CMOV:

(CMOV F, T, ((cc1 | cc2) != 0)) -> (CMOV (CMOV F, T, cc1), T, cc2)
(CMOV F, T, ((cc1 & cc2) != 0)) -> (CMOV (CMOV T, F, !cc1), F, !cc2)

In practice, when we need to lower to control flow using a custom inserter,
we represent the double CMOV using a special node, CMOV2:

(CMOV F, T, ((cc1 | cc2) != 0)) -> (CMOV2 F, T, cc1, cc2)
(CMOV F, T, ((cc1 & cc2) != 0)) -> (CMOV2 T, F, !cc1, !cc2)

This enables the custom inserter to only need to insert one PHI, instead
of two if it looked at two independent CMOVs.

This combine lets us generate:

cmovcc1 (jcc1 if we don't have CMOV)
cmovcc2 (same)

instead of:

setcc1
setcc2
and/or
cmovne (jne if we don't have CMOV)

When we can't use the CMOV instruction, it might increase branch mispredicts.
When we can, or when there is no mispredict, this improves throughput and reduces register pressure.

---

This is pretty much the same thing as D7622, except here the pattern is exposed during ops legalization. By the time we get to the combiner, we already have X86ISD nodes, so even an independent combine wouldn't have been useful.

[EDIT: dropped the CMOV2 for now, I'll put that in a separate patch]
The CMOV2 pseudo is yucky, but I can't see an alternative. If you do it the obvious way (just form two CMOVs all the time) the custom inserter (which definitely should not look at other instructions) adds a PHI between the two jumps, which ends up creating a few copies all around. For instance, for the (sitofp (zext (fcmp une))) testcase:

        ucomiss %xmm1, %xmm0
        movss  <1.0f>, %xmm0
        movaps  %xmm0, %xmm1
        jne     .LBB5_2
        xorps   %xmm1, %xmm1
.LBB5_2:
        jp      .LBB5_4
        movaps  %xmm1, %xmm0
.LBB5_4:
        retq

We could avoid the problem by optimizing something like this:

  A
  | \
  |  B
  | /
  C
  | \
  |  D
  | /
  E

A: X = ...; Y = ...
B: empty
C: Z = PHI [X, A], [Y, B]
D: empty
E: PHI [X, C], [Z, D]

To this:

  A
  | \
  |  C
  | /|
  |/ |
  |  |
  |  D
  | /
  E

A: X = ...; Y = ...
D: empty
E: PHI [X, A], [X, C], [Z, D]

That sounds awfully specific though, and, having looked into it, I'm not sure which more general pass should do it.

Of course, we could just ignore the problem, and have a few extra MOVs around.

[ab]

• Add missing CMOV2 documentation.
• Isolate CMOV pseudo multiclass refactoring

[ab]

Add back the test file..

[ab]

• Split out CMOV2 changes

[MatzeB]

I've got some small nitpicks, but LGTM.

lib/Target/X86/X86ISelLowering.cpp

24030

I would move this before the switch statement to be closer to the def.

24255

I would not initialize this var here and rather make sure that checkBoolTestAndOrSetCCCombine() always sets isAndSetCC if it returns true.

[ab]

r231045.

I guess phabricator isn't content with a link to the revision to auto-close.

-Ahmed

lib/Target/X86/X86ISelLowering.h

enum NodeType {
       FGETSIGNx86,
 
       /// X86 conditional moves. Operand 0 and operand 1 are the two values
       /// to select from. Operand 2 is the condition code, and operand 3 is the
       /// flag operand produced by a CMP or TEST instruction. It also writes a
       /// flag result.
       CMOV,
 
+      /// Same as CMOV, but represents two successive conditional moves.
+      /// That is:
+      ///   (CMOV F, T, cc1, cc2, FLAGS)
+      /// is equivalent to:
+      ///   (CMOV (CMOV F, T, cc1, FLAGS), T, cc2, FLAGS)
+      ///
+      /// There is an additional operand for the second condition.
+      /// The flag operand is still the last one.
+      CMOV2,
+
       /// X86 conditional branches. Operand 0 is the chain operand, operand 1
       /// is the block to branch if condition is true, operand 2 is the
       /// condition code, and operand 3 is the flag operand produced by a CMP
       /// or TEST instruction.
       BRCOND,
 
       /// Return with a flag operand. Operand 0 is the chain operand, operand
       /// 1 is the number of bytes of stack to pop.

lib/Target/X86/X86ISelLowering.cpp

const char *X86TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case X86ISD::COMI:               return "X86ISD::COMI";
   case X86ISD::UCOMI:              return "X86ISD::UCOMI";
   case X86ISD::CMPM:               return "X86ISD::CMPM";
   case X86ISD::CMPMU:              return "X86ISD::CMPMU";
   case X86ISD::SETCC:              return "X86ISD::SETCC";
   case X86ISD::SETCC_CARRY:        return "X86ISD::SETCC_CARRY";
   case X86ISD::FSETCC:             return "X86ISD::FSETCC";
   case X86ISD::CMOV:               return "X86ISD::CMOV";
+  case X86ISD::CMOV2:              return "X86ISD::CMOV2";
   case X86ISD::BRCOND:             return "X86ISD::BRCOND";
   case X86ISD::RET_FLAG:           return "X86ISD::RET_FLAG";
   case X86ISD::REP_STOS:           return "X86ISD::REP_STOS";
   case X86ISD::REP_MOVS:           return "X86ISD::REP_MOVS";
   case X86ISD::GlobalBaseReg:      return "X86ISD::GlobalBaseReg";
   case X86ISD::Wrapper:            return "X86ISD::Wrapper";
   case X86ISD::WrapperRIP:         return "X86ISD::WrapperRIP";
   case X86ISD::PEXTRB:             return "X86ISD::PEXTRB";
X86TargetLowering::EmitLoweredSelect(MachineInstr *MI,
   //  thisMBB:
   //  ...
   //   TrueVal = ...
   //   cmpTY ccX, r1, r2
   //   bCC copy1MBB
   //   fallthrough --> copy0MBB
   MachineBasicBlock *thisMBB = BB;
   MachineFunction *F = BB->getParent();
+
+  // We also lower CMOV2 pseudo-instructions, which are equivalent to:
+  //   (CMOV (CMOV F, T, cc1), T, cc2)
+  // to two successives branches instead of one.
+  bool isCMOV2 = MI->getNumOperands() == 6;
+  MachineBasicBlock *jcc1MBB = nullptr;
+
+  // If MI is a CMOV2 instruction, we lower it to two successive branches to
+  // the same block.  EFLAGS is used by both, so mark it as live in the second.
+  if (isCMOV2) {
+    jcc1MBB = F->CreateMachineBasicBlock(LLVM_BB);
+    F->insert(It, jcc1MBB);
+    jcc1MBB->addLiveIn(X86::EFLAGS);
+  }
+
   MachineBasicBlock *copy0MBB = F->CreateMachineBasicBlock(LLVM_BB);
   MachineBasicBlock *sinkMBB = F->CreateMachineBasicBlock(LLVM_BB);
   F->insert(It, copy0MBB);
   F->insert(It, sinkMBB);
 
   // If the EFLAGS register isn't dead in the terminator, then claim that it's
   // live into the sink and copy blocks.
   const TargetRegisterInfo *TRI = Subtarget->getRegisterInfo();
   if (!MI->killsRegister(X86::EFLAGS) &&
       !checkAndUpdateEFLAGSKill(MI, BB, TRI)) {
     copy0MBB->addLiveIn(X86::EFLAGS);
     sinkMBB->addLiveIn(X86::EFLAGS);
   }
 
   // Transfer the remainder of BB and its successor edges to sinkMBB.
   sinkMBB->splice(sinkMBB->begin(), BB,
                   std::next(MachineBasicBlock::iterator(MI)), BB->end());
   sinkMBB->transferSuccessorsAndUpdatePHIs(BB);
 
   // Add the true and fallthrough blocks as its successors.
+  if (isCMOV2) {
+    // The fallthrough block may be jcc1MBB, if MI is a CMOV2 instruction.
+    BB->addSuccessor(jcc1MBB);
+
+    // In that case, jcc1MBB will itself fallthrough the copy0MBB, and
+    // jump to the sinkMBB.
+    jcc1MBB->addSuccessor(copy0MBB);
+    jcc1MBB->addSuccessor(sinkMBB);
+  } else {
     BB->addSuccessor(copy0MBB);
+  }
+
+  // The true block target of the first (or only) branch is always sinkMBB.
   BB->addSuccessor(sinkMBB);
 
   // Create the conditional branch instruction.
   unsigned Opc =
     X86::GetCondBranchFromCond((X86::CondCode)MI->getOperand(3).getImm());
   BuildMI(BB, DL, TII->get(Opc)).addMBB(sinkMBB);
 
+  if (isCMOV2) {
+    unsigned Opc2 =
+      X86::GetCondBranchFromCond((X86::CondCode)MI->getOperand(4).getImm());
+    BuildMI(jcc1MBB, DL, TII->get(Opc2)).addMBB(sinkMBB);
+  }
+
   //  copy0MBB:
   //   %FalseValue = ...
   //   # fallthrough to sinkMBB
   copy0MBB->addSuccessor(sinkMBB);
 
   //  sinkMBB:
   //   %Result = phi [ %FalseValue, copy0MBB ], [ %TrueValue, thisMBB ]
   //  ...
-  BuildMI(*sinkMBB, sinkMBB->begin(), DL,
-          TII->get(X86::PHI), MI->getOperand(0).getReg())
+  MachineInstrBuilder MIB =
+      BuildMI(*sinkMBB, sinkMBB->begin(), DL, TII->get(X86::PHI),
+              MI->getOperand(0).getReg())
           .addReg(MI->getOperand(1).getReg()).addMBB(copy0MBB)
           .addReg(MI->getOperand(2).getReg()).addMBB(thisMBB);
 
+  // If this was a CMOV2 instruction, the second Jcc provides the same incoming
+  // value as the first Jcc (the True operand of the SELECT_CC/CMOV nodes).
+  if (isCMOV2)
+    MIB.addReg(MI->getOperand(2).getReg()).addMBB(jcc1MBB);
+
   MI->eraseFromParent();   // The pseudo instruction is gone now.
   return sinkMBB;
 }
 
 MachineBasicBlock *
 X86TargetLowering::EmitLoweredSegAlloca(MachineInstr *MI,
                                         MachineBasicBlock *BB) const {
   MachineFunction *MF = BB->getParent();
X86TargetLowering::EmitInstrWithCustomInserter(MachineInstr *MI,
   case X86::CMOV_V16F32:
   case X86::CMOV_V8F64:
   case X86::CMOV_V8I64:
   case X86::CMOV_GR16:
   case X86::CMOV_GR32:
   case X86::CMOV_RFP32:
   case X86::CMOV_RFP64:
   case X86::CMOV_RFP80:
+  case X86::CMOV2_GR8:
+  case X86::CMOV2_FR32:
+  case X86::CMOV2_FR64:
+  case X86::CMOV2_V4F32:
+  case X86::CMOV2_V2F64:
+  case X86::CMOV2_V2I64:
+  case X86::CMOV2_V8F32:
+  case X86::CMOV2_V4F64:
+  case X86::CMOV2_V4I64:
+  case X86::CMOV2_V16F32:
+  case X86::CMOV2_V8F64:
+  case X86::CMOV2_V8I64:
+  case X86::CMOV2_GR16:
+  case X86::CMOV2_GR32:
+  case X86::CMOV2_RFP32:
+  case X86::CMOV2_RFP64:
+  case X86::CMOV2_RFP80:
     return EmitLoweredSelect(MI, BB);
 
   case X86::FP32_TO_INT16_IN_MEM:
   case X86::FP32_TO_INT32_IN_MEM:
   case X86::FP32_TO_INT64_IN_MEM:
   case X86::FP64_TO_INT16_IN_MEM:
   case X86::FP64_TO_INT32_IN_MEM:
   case X86::FP64_TO_INT64_IN_MEM:
if (needOppositeCond)
       CC = X86::GetOppositeBranchCondition(CC);
     return SetCC.getOperand(3);
   }
   }
 
   return SDValue();
 }
 
+/// Check whether Cond is an AND/OR of SETCCs off of the same EFLAGS.
+/// Match:
+///   (X86or (X86setcc) (X86setcc))
+///   (X86cmp (and (X86setcc) (X86setcc)), 0)
+static bool checkBoolTestAndOrSetCCCombine(SDValue Cond, X86::CondCode &CC0,
+                                           X86::CondCode &CC1, SDValue &Flags,
+                                           bool &isAnd) {
+  SDValue SetCC0, SetCC1;
+
+  if (Cond->getOpcode() == X86ISD::CMP) {
+    ConstantSDNode *CondOp1C = dyn_cast<ConstantSDNode>(Cond->getOperand(1));
+    if (!CondOp1C || !CondOp1C->isNullValue())
+      return false;
+
+    Cond = Cond->getOperand(0);
+  }
+
+  switch (Cond->getOpcode()) {
+  default: return false;
+  case ISD::AND:
+  case X86ISD::AND:
+    isAnd = true;
+    // fallthru
+  case ISD::OR:
+  case X86ISD::OR:
+    SetCC0 = Cond->getOperand(0);
+    SetCC1 = Cond->getOperand(1);
+    break;
+  };
+
+  // Make sure we have SETCC nodes, using the same flags value.
+  if (SetCC0.getOpcode() != X86ISD::SETCC ||
+      SetCC1.getOpcode() != X86ISD::SETCC ||
+      SetCC0->getOperand(1) != SetCC1->getOperand(1))
+    return false;
+
+  CC0 = (X86::CondCode)SetCC0->getConstantOperandVal(0);
+  CC1 = (X86::CondCode)SetCC1->getConstantOperandVal(0);
+  Flags = SetCC0->getOperand(1);
+  return true;
+}
+
 /// Optimize X86ISD::CMOV [LHS, RHS, CONDCODE (e.g. X86::COND_NE), CONDVAL]
 static SDValue PerformCMOVCombine(SDNode *N, SelectionDAG &DAG,
                                   TargetLowering::DAGCombinerInfo &DCI,
                                   const X86Subtarget *Subtarget) {
   SDLoc DL(N);
 
   // If the flag operand isn't dead, don't touch this CMOV.
   if (N->getNumValues() == 2 && !SDValue(N, 1).use_empty())
     return SDValue();
 
   SDValue FalseOp = N->getOperand(0);
   SDValue TrueOp = N->getOperand(1);
+  EVT VT = TrueOp.getValueType();
   X86::CondCode CC = (X86::CondCode)N->getConstantOperandVal(2);
   SDValue Cond = N->getOperand(3);
 
   if (CC == X86::COND_E || CC == X86::COND_NE) {
     switch (Cond.getOpcode()) {
     default: break;
     case X86ISD::BSR:
     case X86ISD::BSF:
if ((Cond.getOpcode() == X86ISD::CMP || Cond.getOpcode() == X86ISD::SUB) &&
           CmpAgainst == dyn_cast<ConstantSDNode>(TrueOp)) {
         SDValue Ops[] = { FalseOp, Cond.getOperand(0),
                           DAG.getConstant(CC, MVT::i8), Cond };
         return DAG.getNode(X86ISD::CMOV, DL, N->getVTList (), Ops);
       }
     }
   }
 
+  // Fold and/or of setcc's to double CMOV:
+  //   (CMOV F, T, ((cc1 | cc2) != 0)) -> (CMOV (CMOV F, T, cc1), T, cc2)
+  //   (CMOV F, T, ((cc1 & cc2) != 0)) -> (CMOV (CMOV T, F, !cc1), F, !cc2)
+  //
+  // In practice, when we need to lower to control flow using a custom inserter,
+  // we represent the double CMOV using a special node, CMOV2:
+  //   (CMOV F, T, ((cc1 | cc2) != 0)) -> (CMOV2 F, T, cc1, cc2)
+  //   (CMOV F, T, ((cc1 & cc2) != 0)) -> (CMOV2 T, F, !cc1, !cc2)
+  // This enables the custom inserter to only need to insert one PHI, instead
+  // of two if it looked at two independent CMOVs.
+  //
+  // This combine lets us generate:
+  //   cmovcc1 (jcc1 if we don't have CMOV)
+  //   cmovcc2 (same)
+  // instead of:
+  //   setcc1
+  //   setcc2
+  //   and/or
+  //   cmovne (jne if we don't have CMOV)
+  // When we can't use the CMOV instruction, it might increase branch
+  // mispredicts.
+  // When we can use CMOV, or when there is no mispredict, this improves
+  // throughput and reduces register pressure.
+  //
+  if (CC == X86::COND_NE) {
+    SDValue Flags;
+    X86::CondCode CC0, CC1;
+    bool isAndSetCC = false;
+    if (checkBoolTestAndOrSetCCCombine(Cond, CC0, CC1, Flags, isAndSetCC)) {
+      if (isAndSetCC) {
+        std::swap(FalseOp, TrueOp);
+        CC0 = X86::GetOppositeBranchCondition(CC0);
+        CC1 = X86::GetOppositeBranchCondition(CC1);
+      }
+
+      SDValue CMOV;
+      if (Subtarget->hasCMov() && VT.isInteger() && !VT.isVector()) {
+        SDValue LOps[] = {FalseOp, TrueOp, DAG.getConstant(CC0, MVT::i8),
+                          Flags};
+        SDValue LCMOV = DAG.getNode(X86ISD::CMOV, DL, N->getVTList(), LOps);
+        SDValue Ops[] = {LCMOV, TrueOp, DAG.getConstant(CC1, MVT::i8), Flags};
+        CMOV = DAG.getNode(X86ISD::CMOV, DL, N->getVTList(), Ops);
+      } else {
+        SDValue Ops[] = {FalseOp, TrueOp, DAG.getConstant(CC0, MVT::i8),
+                         DAG.getConstant(CC1, MVT::i8), Flags};
+        CMOV = DAG.getNode(X86ISD::CMOV2, DL, N->getVTList(), Ops);
+      }
+      DAG.ReplaceAllUsesOfValueWith(SDValue(N, 1), SDValue(CMOV.getNode(), 1));
+      return CMOV;
+    }
+  }
+
   return SDValue();
 }
 
 static SDValue PerformINTRINSIC_WO_CHAINCombine(SDNode *N, SelectionDAG &DAG,
                                                 const X86Subtarget *Subtarget) {
   unsigned IntNo = cast<ConstantSDNode>(N->getOperand(0))->getZExtValue();
   switch (IntNo) {
   default: return SDValue();

lib/Target/X86/X86InstrCompiler.td

 // CMOV* - Used to implement the SELECT DAG operation.  Expanded after
 // instruction selection into a branch sequence.
 multiclass CMOVrr_PSEUDO<RegisterClass RC, ValueType VT> {
   def CMOV#NAME  : I<0, Pseudo,
                     (outs RC:$dst), (ins RC:$t, RC:$f, i8imm:$cond),
                     "#CMOV_"#NAME#" PSEUDO!",
                     [(set RC:$dst, (VT (X86cmov RC:$t, RC:$f, imm:$cond,
                                                 EFLAGS)))]>;
+  def CMOV2#NAME : I<0, Pseudo,
+                    (outs RC:$dst), (ins RC:$t, RC:$f, i8imm:$cond, i8imm:$cond2),
+                    "#CMOV2_"#NAME#" PSEUDO!",
+                    [(set RC:$dst, (VT (X86cmov2 RC:$t, RC:$f, imm:$cond,
+                                                 imm:$cond2, EFLAGS)))]>;
 }
 
 let usesCustomInserter = 1, Uses = [EFLAGS] in {
   // X86 doesn't have 8-bit conditional moves. Use a customInserter to
   // emit control flow. An alternative to this is to mark i8 SELECT as Promote,
   // however that requires promoting the operands, and can induce additional
   // i8 register pressure.
   defm _GR8 : CMOVrr_PSEUDO<GR8, i8>;

lib/Target/X86/X86InstrInfo.td

 
 def SDTX86Cmps : SDTypeProfile<1, 3, [SDTCisFP<0>, SDTCisSameAs<1, 2>, SDTCisVT<3, i8>]>;
 //def SDTX86Cmpss : SDTypeProfile<1, 3, [SDTCisVT<0, f32>, SDTCisSameAs<1, 2>, SDTCisVT<3, i8>]>;
 
 def SDTX86Cmov    : SDTypeProfile<1, 4,
                                   [SDTCisSameAs<0, 1>, SDTCisSameAs<1, 2>,
                                    SDTCisVT<3, i8>, SDTCisVT<4, i32>]>;
 
+def SDTX86Cmov2   : SDTypeProfile<1, 5,
+                                  [SDTCisSameAs<0, 1>, SDTCisSameAs<1, 2>,
+                                   SDTCisVT<3, i8>, SDTCisSameAs<3, 4>,
+                                   SDTCisVT<5, i32>]>;
+
 // Unary and binary operator instructions that set EFLAGS as a side-effect.
 def SDTUnaryArithWithFlags : SDTypeProfile<2, 1,
                                            [SDTCisSameAs<0, 2>,
                                             SDTCisInt<0>, SDTCisVT<1, i32>]>;
 
 def SDTBinaryArithWithFlags : SDTypeProfile<2, 2,
                                             [SDTCisSameAs<0, 2>,
                                              SDTCisSameAs<0, 3>,

 def X86bsr     : SDNode<"X86ISD::BSR",      SDTUnaryArithWithFlags>;
 def X86shld    : SDNode<"X86ISD::SHLD",     SDTIntShiftDOp>;
 def X86shrd    : SDNode<"X86ISD::SHRD",     SDTIntShiftDOp>;
 
 def X86cmp     : SDNode<"X86ISD::CMP" ,     SDTX86CmpTest>;
 def X86bt      : SDNode<"X86ISD::BT",       SDTX86CmpTest>;
 
 def X86cmov    : SDNode<"X86ISD::CMOV",     SDTX86Cmov>;
+def X86cmov2   : SDNode<"X86ISD::CMOV2",    SDTX86Cmov2>;
 def X86brcond  : SDNode<"X86ISD::BRCOND",   SDTX86BrCond,
                         [SDNPHasChain]>;
 def X86setcc   : SDNode<"X86ISD::SETCC",    SDTX86SetCC>;
 def X86setcc_c : SDNode<"X86ISD::SETCC_CARRY", SDTX86SetCC_C>;
 
 def X86sahf    : SDNode<"X86ISD::SAHF",     SDTX86sahf>;
 
 def X86rdrand  : SDNode<"X86ISD::RDRAND",   SDTX86rdrand,

test/CodeGen/X86/cmovcmov.ll

+; RUN: llc < %s -asm-verbose=false -mtriple=x86_64-unknown-linux | FileCheck %s --check-prefix=CHECK --check-prefix=CMOV
+; RUN: llc < %s -asm-verbose=false -mtriple=i686-unknown-linux | FileCheck %s --check-prefix=CHECK --check-prefix=NOCMOV
+
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
+; Test 2xCMOV patterns exposed after legalization.
+; One way to do that is with (select (fcmp une/oeq)), which gets
+; legalized to setp/setne.
+
+; CHECK-LABEL: test_select_fcmp_oeq_i32:
+
+; CMOV-NEXT: ucomiss  %xmm1, %xmm0
+; CMOV-NEXT: cmovnel  %esi, %edi
+; CMOV-NEXT: cmovpl  %esi, %edi
+; CMOV-NEXT: movl  %edi, %eax
+; CMOV-NEXT: retq
+
+; NOCMOV-NEXT:  flds  8(%esp)
+; NOCMOV-NEXT:  flds  4(%esp)
+; NOCMOV-NEXT:  fucompp
+; NOCMOV-NEXT:  fnstsw  %ax
+; NOCMOV-NEXT:  sahf
+; NOCMOV-NEXT:  leal  16(%esp), %eax
+; NOCMOV-NEXT:  jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:  jp  [[TBB]]
+; NOCMOV-NEXT:  leal  12(%esp), %eax
+; NOCMOV-NEXT:[[TBB]]:
+; NOCMOV-NEXT:  movl  (%eax), %eax
+; NOCMOV-NEXT:  retl
+define i32 @test_select_fcmp_oeq_i32(float %a, float %b, i32 %c, i32 %d) #0 {
+entry:
+  %cmp = fcmp oeq float %a, %b
+  %r = select i1 %cmp, i32 %c, i32 %d
+  ret i32 %r
+}
+
+; CHECK-LABEL: test_select_fcmp_oeq_i64:
+
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   cmovneq  %rsi, %rdi
+; CMOV-NEXT:   cmovpq  %rsi, %rdi
+; CMOV-NEXT:   movq  %rdi, %rax
+; CMOV-NEXT:   retq
+
+; NOCMOV-NEXT:   flds  8(%esp)
+; NOCMOV-NEXT:   flds  4(%esp)
+; NOCMOV-NEXT:   fucompp
+; NOCMOV-NEXT:   fnstsw  %ax
+; NOCMOV-NEXT:   sahf
+; NOCMOV-NEXT:   leal  20(%esp), %ecx
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  12(%esp), %ecx
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  (%ecx), %eax
+; NOCMOV-NEXT:   orl  $4, %ecx
+; NOCMOV-NEXT:   movl  (%ecx), %edx
+; NOCMOV-NEXT:   retl
+define i64 @test_select_fcmp_oeq_i64(float %a, float %b, i64 %c, i64 %d) #0 {
+entry:
+  %cmp = fcmp oeq float %a, %b
+  %r = select i1 %cmp, i64 %c, i64 %d
+  ret i64 %r
+}
+
+; CHECK-LABEL: test_select_fcmp_une_i64:
+
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   cmovneq  %rdi, %rsi
+; CMOV-NEXT:   cmovpq  %rdi, %rsi
+; CMOV-NEXT:   movq  %rsi, %rax
+; CMOV-NEXT:   retq
+
+; NOCMOV-NEXT:   flds  8(%esp)
+; NOCMOV-NEXT:   flds  4(%esp)
+; NOCMOV-NEXT:   fucompp
+; NOCMOV-NEXT:   fnstsw  %ax
+; NOCMOV-NEXT:   sahf
+; NOCMOV-NEXT:   leal  12(%esp), %ecx
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  20(%esp), %ecx
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  (%ecx), %eax
+; NOCMOV-NEXT:   orl  $4, %ecx
+; NOCMOV-NEXT:   movl  (%ecx), %edx
+; NOCMOV-NEXT:   retl
+define i64 @test_select_fcmp_une_i64(float %a, float %b, i64 %c, i64 %d) #0 {
+entry:
+  %cmp = fcmp une float %a, %b
+  %r = select i1 %cmp, i64 %c, i64 %d
+  ret i64 %r
+}
+
+; CHECK-LABEL: test_select_fcmp_oeq_f64:
+
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; CMOV-NEXT:   jp  [[TBB]]
+; CMOV-NEXT:   movaps  %xmm2, %xmm3
+; CMOV-NEXT: [[TBB]]:
+; CMOV-NEXT:   movaps  %xmm3, %xmm0
+; CMOV-NEXT:   retq
+
+; NOCMOV-NEXT:   flds  8(%esp)
+; NOCMOV-NEXT:   flds  4(%esp)
+; NOCMOV-NEXT:   fucompp
+; NOCMOV-NEXT:   fnstsw  %ax
+; NOCMOV-NEXT:   sahf
+; NOCMOV-NEXT:   leal  20(%esp), %eax
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  12(%esp), %eax
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   fldl  (%eax)
+; NOCMOV-NEXT:   retl
+define double @test_select_fcmp_oeq_f64(float %a, float %b, double %c, double %d) #0 {
+entry:
+  %cmp = fcmp oeq float %a, %b
+  %r = select i1 %cmp, double %c, double %d
+  ret double %r
+}
+
+; CHECK-LABEL: test_select_fcmp_oeq_v4i32:
+
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; CMOV-NEXT:   jp  [[TBB]]
+; CMOV-NEXT:   movaps  %xmm2, %xmm3
+; CMOV-NEXT: [[TBB]]:
+; CMOV-NEXT:   movaps  %xmm3, %xmm0
+; CMOV-NEXT:   retq
+
+; NOCMOV-NEXT:   pushl  %edi
+; NOCMOV-NEXT:   pushl  %esi
+; NOCMOV-NEXT:   flds  20(%esp)
+; NOCMOV-NEXT:   flds  16(%esp)
+; NOCMOV-NEXT:   fucompp
+; NOCMOV-NEXT:   fnstsw  %ax
+; NOCMOV-NEXT:   sahf
+; NOCMOV-NEXT:   leal  40(%esp), %eax
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  24(%esp), %eax
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  (%eax), %eax
+; NOCMOV-NEXT:   leal  44(%esp), %ecx
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  28(%esp), %ecx
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  (%ecx), %ecx
+; NOCMOV-NEXT:   leal  48(%esp), %esi
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  32(%esp), %esi
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  12(%esp), %edx
+; NOCMOV-NEXT:   movl  (%esi), %esi
+; NOCMOV-NEXT:   leal  52(%esp), %edi
+; NOCMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; NOCMOV-NEXT:   jp  [[TBB]]
+; NOCMOV-NEXT:   leal  36(%esp), %edi
+; NOCMOV-NEXT: [[TBB]]:
+; NOCMOV-NEXT:   movl  (%edi), %edi
+; NOCMOV-NEXT:   movl  %edi, 12(%edx)
+; NOCMOV-NEXT:   movl  %esi, 8(%edx)
+; NOCMOV-NEXT:   movl  %ecx, 4(%edx)
+; NOCMOV-NEXT:   movl  %eax, (%edx)
+; NOCMOV-NEXT:   popl  %esi
+; NOCMOV-NEXT:   popl  %edi
+; NOCMOV-NEXT:   retl  $4
+define <4 x i32> @test_select_fcmp_oeq_v4i32(float %a, float %b, <4 x i32> %c, <4 x i32> %d) #0 {
+entry:
+  %cmp = fcmp oeq float %a, %b
+  %r = select i1 %cmp, <4 x i32> %c, <4 x i32> %d
+  ret <4 x i32> %r
+}
+
+; Also make sure we catch the original code-sequence of interest:
+
+; CMOV: [[ONE_F32_LCPI:.LCPI.*]]:
+; CMOV-NEXT:   .long  1065353216
+
+; CHECK-LABEL: test_zext_fcmp_une:
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   movss  [[ONE_F32_LCPI]](%rip), %xmm0
+; CMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; CMOV-NEXT:   jp  [[TBB]]
+; CMOV-NEXT:   xorps  %xmm0, %xmm0
+; CMOV-NEXT: [[TBB]]:
+; CMOV-NEXT:   retq
+
+; NOCMOV:        jne
+; NOCMOV-NEXT:   jp
+define float @test_zext_fcmp_une(float %a, float %b) #0 {
+entry:
+  %cmp = fcmp une float %a, %b
+  %conv1 = zext i1 %cmp to i32
+  %conv2 = sitofp i32 %conv1 to float
+  ret float %conv2
+}
+
+; CMOV: [[ONE_F32_LCPI:.LCPI.*]]:
+; CMOV-NEXT:   .long  1065353216
+
+; CHECK-LABEL: test_zext_fcmp_oeq:
+; CMOV-NEXT:   ucomiss  %xmm1, %xmm0
+; CMOV-NEXT:   xorps  %xmm0, %xmm0
+; CMOV-NEXT:   jne  [[TBB:.LBB[0-9_]+]]
+; CMOV-NEXT:   jp  [[TBB]]
+; CMOV-NEXT:   movss  [[ONE_F32_LCPI]](%rip), %xmm0
+; CMOV-NEXT: [[TBB]]:
+; CMOV-NEXT:   retq
+
+; NOCMOV:        jne
+; NOCMOV-NEXT:   jp
+define float @test_zext_fcmp_oeq(float %a, float %b) #0 {
+entry:
+  %cmp = fcmp oeq float %a, %b
+  %conv1 = zext i1 %cmp to i32
+  %conv2 = sitofp i32 %conv1 to float
+  ret float %conv2
+}
+
+attributes #0 = { nounwind }

test/CodeGen/X86/fast-isel-select-cmov2.ll

; CHECK-NEXT:  retq
   %1 = fcmp false double %a, %b
   %2 = select i1 %1, i64 %c, i64 %d
   ret i64 %2
 }
 
 define i64 @select_fcmp_oeq_cmov(double %a, double %b, i64 %c, i64 %d) {
 ; CHECK-LABEL: select_fcmp_oeq_cmov
 ; CHECK:       ucomisd %xmm1, %xmm0
-; CHECK-NEXT:  setnp %al
-; CHECK-NEXT:  sete %cl
-; CHECK-NEXT:  testb %al, %cl
-; CHECK-NEXT:  cmoveq %rsi, %rdi
+; CHECK-NEXT:  cmovneq %rsi, %rdi
+; CHECK-NEXT:  cmovpq %rsi, %rdi
+; CHECK-NEXT:  movq %rdi, %rax
   %1 = fcmp oeq double %a, %b
   %2 = select i1 %1, i64 %c, i64 %d
   ret i64 %2
 }
 
 define i64 @select_fcmp_ogt_cmov(double %a, double %b, i64 %c, i64 %d) {
 ; CHECK-LABEL: select_fcmp_ogt_cmov
 ; CHECK:       ucomisd %xmm1, %xmm0
; CHECK-NEXT:  cmovaq %rsi, %rdi
   %1 = fcmp ule double %a, %b
   %2 = select i1 %1, i64 %c, i64 %d
   ret i64 %2
 }
 
 define i64 @select_fcmp_une_cmov(double %a, double %b, i64 %c, i64 %d) {
 ; CHECK-LABEL: select_fcmp_une_cmov
 ; CHECK:       ucomisd %xmm1, %xmm0
-; CHECK-NEXT:  setp %al
-; CHECK-NEXT:  setne %cl
-; CHECK-NEXT:  orb %al, %cl
-; CHECK-NEXT:  cmoveq %rsi, %rdi
+; CHECK-NEXT:  cmovneq %rdi, %rsi
+; CHECK-NEXT:  cmovpq %rdi, %rsi
+; CHECK-NEXT:  movq %rsi, %rax
   %1 = fcmp une double %a, %b
   %2 = select i1 %1, i64 %c, i64 %d
   ret i64 %2
 }
 
 define i64 @select_fcmp_true_cmov(double %a, double %b, i64 %c, i64 %d) {
 ; CHECK-LABEL: select_fcmp_true_cmov
 ; CHECK:       movq %rdi, %rax