[SDAG] When performing post-legalize DAG combining, run the legalizer over each node in the worklist prior to combining.

[chandlerc]

Summary

This allows the combiner to produce new nodes which need to go back
through legalization. This is particularly useful when generating
operands to target specific nodes in a post-legalize DAG combine where
the operands are significantly easier to express as pre-legalized
operations. My immediate use case will be PSHUFB formation where we need
to build a constant shuffle mask with a build_vector node.

This also refactors the relevant functionality in the legalizer to
support this, and updates relevant tests. I've spoken to the R600 folks
and these changes look like improvements. The avx512 change needs to be
investigated, I suspect there is a disagreement between the legalizer
and the DAG combiner there, but it seems a minor issue so leaving it to
be re-evaluated after this patch.

[filcab]

LGTM.
It's “nice” to see we weren't copying debug info (although many combines and lowerings try to keep the original info).

[chandlerc]

Reflect the changes to the combiner that allowed this to handle more edge
cases. Also simplify some of the now unnecessary book keeping that was papering
over the flaws in the combiner itself.

[chandlerc]

FYI, I think this is essentially ready-to-go. Any final comments?

[hfinkel]

LGTM.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

185

Is this a fly-by fix, or something that was not needed previously?

4313

I actually dislike this comment because it calls something hacky without explaining why or what should be done instead.

4320

If we could get a vector scalarization here, I'd make this 64 instead of 16.

[chandlerc]

Thanks Hal, submitting with the fixes described below.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

185

Fly-by fix, and sadly neither Dave nor I had any good ideas about how to test. =/

4313

Yea, sorry, this comment was supposed to get deleted. It isn't hacky at all at this point I think, it dates from when I didn't even think this approach would work.

There is a proper doxygen comment in the header already. However, that comment was pretty dated as well. I've beefed it up to clarify a lot of the subtle semantic contracts here.

4320

I don't disagree completely, but the one that legalizes the *entire DAG* has only 16, so it seems weird to make this one larger. I'm going to keep this as-is for now, and maybe one of us can get measurements to drive a change here and/or above...

I'm also really frustrated by having both UpdatedNodes and LegalizedNodes. I'd like to eventually remove one of them, but haven't yet found a way.

[chandlerc]

Closed by commit rL214020 (authored by @chandlerc).

include/llvm/CodeGen/SelectionDAG.h

 // SDNode class and subclasses.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_SELECTIONDAG_H
 #define LLVM_CODEGEN_SELECTIONDAG_H
 
 #include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/ilist.h"
 #include "llvm/CodeGen/DAGCombine.h"
 #include "llvm/CodeGen/SelectionDAGNodes.h"
 #include "llvm/Support/RecyclingAllocator.h"
 #include "llvm/Target/TargetMachine.h"
 #include <cassert>
 #include <map>
#endif
   /// Legalize - This transforms the SelectionDAG into a SelectionDAG that is
   /// compatible with the target instruction selector, as indicated by the
   /// TargetLowering object.
   ///
   /// Note that this is an involved process that may invalidate pointers into
   /// the graph.
   void Legalize();
 
+  /// \brief Transforms a SelectionDAG node and any operands to it into a node
+  /// that is compatible with the target instruction selector, as indicated by
+  /// the TargetLowering object.
+  ///
+  /// This essentially runs a single recursive walk of the \c Legalize process
+  /// over the given node (and its operands). This can be used to incrementally
+  /// legalize the DAG.
+  bool LegalizeOp(SDNode *N, SmallSetVector<SDNode *, 16> &UpdatedNodes);
+
   /// LegalizeVectors - This transforms the SelectionDAG into a SelectionDAG
   /// that only uses vector math operations supported by the target.  This is
   /// necessary as a separate step from Legalize because unrolling a vector
   /// operation can introduce illegal types, which requires running
   /// LegalizeTypes again.
   ///
   /// This returns true if it made any changes; in that case, LegalizeTypes
   /// is called again before Legalize.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(),
        E = DAG.allnodes_end(); I != E; ++I)
     AddToWorklist(I);
 
   // Create a dummy node (which is not added to allnodes), that adds a reference
   // to the root node, preventing it from being deleted, and tracking any
   // changes of the root.
   HandleSDNode Dummy(DAG.getRoot());
 
-  // The root of the dag may dangle to deleted nodes until the dag combiner is
-  // done.  Set it to null to avoid confusion.
-  DAG.setRoot(SDValue());
-
   // while the worklist isn't empty, find a node and
   // try and combine it.
   while (!WorklistMap.empty()) {
     SDNode *N;
     // The Worklist holds the SDNodes in order, but it may contain null entries.
     do {
       N = Worklist.pop_back_val();
     } while (!N);
while (!WorklistMap.empty()) {
     // won't repeatedly process the same operand.
     CombinedNodes.insert(N);
     for (unsigned i = 0, e = N->getNumOperands(); i != e; ++i)
       if (!CombinedNodes.count(N->getOperand(i).getNode()))
         AddToWorklist(N->getOperand(i).getNode());
 
     WorklistRemover DeadNodes(*this);
 
+    // If this combine is running after legalizing the DAG, re-legalize any
+    // nodes pulled off the worklist.
+    if (Level == AfterLegalizeDAG) {
+      SmallSetVector<SDNode *, 16> UpdatedNodes;
+      bool NIsValid = DAG.LegalizeOp(N, UpdatedNodes);
+
+      for (SDNode *LN : UpdatedNodes) {
+        AddToWorklist(LN);
+        AddUsersToWorklist(LN);
+      }
+      if (!NIsValid)
+        continue;
+    }
+
     SDValue RV = combine(N);
 
     if (!RV.getNode())
       continue;
 
     ++NodesCombined;
 
     // If we get back the same node we passed in, rather than a new node or

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

 //===-- LegalizeDAG.cpp - Implement SelectionDAG::Legalize ----------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the SelectionDAG::Legalize method.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/SelectionDAG.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/IR/CallingConv.h"

 /// will attempt merge setcc and brc instructions into brcc's.
 ///
 namespace {
 class SelectionDAGLegalize : public SelectionDAG::DAGUpdateListener {
   const TargetMachine &TM;
   const TargetLowering &TLI;
   SelectionDAG &DAG;
 
-  /// LegalizePosition - The iterator for walking through the node list.
-  SelectionDAG::allnodes_iterator LegalizePosition;
+  /// \brief The iterator being used to walk the DAG. We hold a reference to it
+  /// in order to update it as necessary on node deletion.
+  SelectionDAG::allnodes_iterator &LegalizePosition;
+
+  /// \brief The set of nodes which have already been legalized. We hold a
+  /// reference to it in order to update as necessary on node deletion.
+  SmallPtrSetImpl<SDNode *> &LegalizedNodes;
 
-  /// LegalizedNodes - The set of nodes which have already been legalized.
-  SmallPtrSet<SDNode *, 16> LegalizedNodes;
+  /// \brief A set of all the nodes updated during legalization.
+  SmallSetVector<SDNode *, 16> *UpdatedNodes;
 
   EVT getSetCCResultType(EVT VT) const {
     return TLI.getSetCCResultType(*DAG.getContext(), VT);
   }
 
   // Libcall insertion helpers.
 
 public:
-  explicit SelectionDAGLegalize(SelectionDAG &DAG);
-
-  void LegalizeDAG();
+  SelectionDAGLegalize(SelectionDAG &DAG,
+                       SelectionDAG::allnodes_iterator &LegalizePosition,
+                       SmallPtrSetImpl<SDNode *> &LegalizedNodes,
+                       SmallSetVector<SDNode *, 16> *UpdatedNodes = nullptr)
+      : SelectionDAG::DAGUpdateListener(DAG), TM(DAG.getTarget()),
+        TLI(DAG.getTargetLoweringInfo()), DAG(DAG),
+        LegalizePosition(LegalizePosition), LegalizedNodes(LegalizedNodes),
+        UpdatedNodes(UpdatedNodes) {}
 
-private:
-  /// LegalizeOp - Legalizes the given operation.
+  /// \brief Legalizes the given operation.
   void LegalizeOp(SDNode *Node);
 
+private:
   SDValue OptimizeFloatStore(StoreSDNode *ST);
 
   void LegalizeLoadOps(SDNode *Node);
   void LegalizeStoreOps(SDNode *Node);
 
   /// PerformInsertVectorEltInMemory - Some target cannot handle a variable
   /// insertion index for the INSERT_VECTOR_ELT instruction.  In this case, it
   /// is necessary to spill the vector being inserted into to memory, perform
private:
 
   void ExpandNode(SDNode *Node);
   void PromoteNode(SDNode *Node);
 
   void ForgetNode(SDNode *N) {
     LegalizedNodes.erase(N);
     if (LegalizePosition == SelectionDAG::allnodes_iterator(N))
       ++LegalizePosition;
+    if (UpdatedNodes)
+      UpdatedNodes->remove(N);
   }
 
 public:
   // DAGUpdateListener implementation.
   void NodeDeleted(SDNode *N, SDNode *E) override {
     ForgetNode(N);
   }
   void NodeUpdated(SDNode *N) override {}
 
   // Node replacement helpers
   void ReplacedNode(SDNode *N) {
     if (N->use_empty()) {
       DAG.RemoveDeadNode(N);
     } else {
       ForgetNode(N);
     }
   }
   void ReplaceNode(SDNode *Old, SDNode *New) {
     DAG.ReplaceAllUsesWith(Old, New);
+    for (unsigned i = 0, e = Old->getNumValues(); i != e; ++i)
+      DAG.TransferDbgValues(SDValue(Old, i), SDValue(New, i));
     ReplacedNode(Old);
+    if (UpdatedNodes)
+      UpdatedNodes->insert(New);
   }
   void ReplaceNode(SDValue Old, SDValue New) {
     DAG.ReplaceAllUsesWith(Old, New);
     ReplacedNode(Old.getNode());
+    if (UpdatedNodes)
+      UpdatedNodes->insert(New.getNode());
   }
   void ReplaceNode(SDNode *Old, const SDValue *New) {
     DAG.ReplaceAllUsesWith(Old, New);
+    for (unsigned i = 0, e = Old->getNumValues(); i != e; ++i)
+      DAG.TransferDbgValues(SDValue(Old, i), New[i]);
     ReplacedNode(Old);
+    if (UpdatedNodes)
+      UpdatedNodes->insert(New->getNode());
   }
 };
 }
 
 /// ShuffleWithNarrowerEltType - Return a vector shuffle operation which
 /// performs the same shuffe in terms of order or result bytes, but on a type
 /// whose vector element type is narrower than the original shuffle type.
 /// e.g. <v4i32> <0, 1, 0, 1> -> v8i16 <0, 1, 2, 3, 0, 1, 2, 3>
for (unsigned j = 0; j != NumEltsGrowth; ++j) {
         NewMask.push_back(Idx * NumEltsGrowth + j);
     }
   }
   assert(NewMask.size() == NumDestElts && "Non-integer NumEltsGrowth?");
   assert(TLI.isShuffleMaskLegal(NewMask, NVT) && "Shuffle not legal?");
   return DAG.getVectorShuffle(NVT, dl, N1, N2, &NewMask[0]);
 }
 
-SelectionDAGLegalize::SelectionDAGLegalize(SelectionDAG &dag)
-  : SelectionDAG::DAGUpdateListener(dag),
-    TM(dag.getTarget()), TLI(dag.getTargetLoweringInfo()),
-    DAG(dag) {
-}
-
-void SelectionDAGLegalize::LegalizeDAG() {
-  DAG.AssignTopologicalOrder();
-
-  // Visit all the nodes. We start in topological order, so that we see
-  // nodes with their original operands intact. Legalization can produce
-  // new nodes which may themselves need to be legalized. Iterate until all
-  // nodes have been legalized.
-  for (;;) {
-    bool AnyLegalized = false;
-    for (LegalizePosition = DAG.allnodes_end();
-         LegalizePosition != DAG.allnodes_begin(); ) {
-      --LegalizePosition;
-
-      SDNode *N = LegalizePosition;
-      if (LegalizedNodes.insert(N)) {
-        AnyLegalized = true;
-        LegalizeOp(N);
-      }
-    }
-    if (!AnyLegalized)
-      break;
-
-  }
-
-  // Remove dead nodes now.
-  DAG.RemoveDeadNodes();
-}
-
 /// ExpandConstantFP - Expands the ConstantFP node to an integer constant or
 /// a load from the constant pool.
 SDValue
 SelectionDAGLegalize::ExpandConstantFP(ConstantFPSDNode *CFP, bool UseCP) {
   bool Extend = false;
   SDLoc dl(CFP);
 
   // If a FP immediate is precise when represented as a float and if the
case TargetLowering::Promote: {
       RChain = Res.getValue(1);
       break;
     }
     }
     if (RChain.getNode() != Node) {
       assert(RVal.getNode() != Node && "Load must be completely replaced");
       DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 0), RVal);
       DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 1), RChain);
+      if (UpdatedNodes) {
+        UpdatedNodes->insert(RVal.getNode());
+        UpdatedNodes->insert(RChain.getNode());
+      }
       ReplacedNode(Node);
     }
     return;
   }
 
   EVT SrcVT = LD->getMemoryVT();
   unsigned SrcWidth = SrcVT.getSizeInBits();
   unsigned Alignment = LD->getAlignment();
void SelectionDAGLegalize::LegalizeLoadOps(SDNode *Node) {
   }
 
   // Since loads produce two values, make sure to remember that we legalized
   // both of them.
   if (Chain.getNode() != Node) {
     assert(Value.getNode() != Node && "Load must be completely replaced");
     DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 0), Value);
     DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 1), Chain);
+    if (UpdatedNodes) {
+      UpdatedNodes->insert(Value.getNode());
+      UpdatedNodes->insert(Chain.getNode());
+    }
     ReplacedNode(Node);
   }
 }
 
 /// LegalizeOp - Return a legal replacement for the given operation, with
 /// all legal operands.
 void SelectionDAGLegalize::LegalizeOp(SDNode *Node) {
   if (Node->getOpcode() == ISD::TargetConstant) // Allow illegal target nodes.
case ISD::SHL_PARTS:
         NewNode = DAG.UpdateNodeOperands(Node, Node->getOperand(0),
                                          Node->getOperand(1),
                                          Handle.getValue());
       }
       break;
     }
 
     if (NewNode != Node) {
-      DAG.ReplaceAllUsesWith(Node, NewNode);
-      for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
-        DAG.TransferDbgValues(SDValue(Node, i), SDValue(NewNode, i));
-      ReplacedNode(Node);
+      ReplaceNode(Node, NewNode);
       Node = NewNode;
     }
     switch (Action) {
     case TargetLowering::Legal:
       return;
     case TargetLowering::Custom: {
       // FIXME: The handling for custom lowering with multiple results is
       // a complete mess.
       SDValue Res = TLI.LowerOperation(SDValue(Node, 0), DAG);
       if (Res.getNode()) {
         SmallVector<SDValue, 8> ResultVals;
         for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i) {
           if (e == 1)
             ResultVals.push_back(Res);
           else
             ResultVals.push_back(Res.getValue(i));
         }
-        if (Res.getNode() != Node || Res.getResNo() != 0) {
-          DAG.ReplaceAllUsesWith(Node, ResultVals.data());
-          for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
-            DAG.TransferDbgValues(SDValue(Node, i), ResultVals[i]);
-          ReplacedNode(Node);
-        }
+        if (Res.getNode() != Node || Res.getResNo() != 0)
+          ReplaceNode(Node, ResultVals.data());
         return;
       }
     }
       // FALL THROUGH
     case TargetLowering::Expand:
       ExpandNode(Node);
       return;
     case TargetLowering::Promote:
case ISD::VAARG: {
     Chain = Tmp1.getValue(1);
 
     Tmp2 = DAG.getNode(TruncOp, dl, OVT, Tmp1);
 
     // Modified the chain result - switch anything that used the old chain to
     // use the new one.
     DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 0), Tmp2);
     DAG.ReplaceAllUsesOfValueWith(SDValue(Node, 1), Chain);
+    if (UpdatedNodes) {
+      UpdatedNodes->insert(Tmp2.getNode());
+      UpdatedNodes->insert(Chain.getNode());
+    }
     ReplacedNode(Node);
     break;
   }
   case ISD::AND:
   case ISD::OR:
   case ISD::XOR: {
     unsigned ExtOp, TruncOp;
     if (OVT.isVector()) {
void SelectionDAGLegalize::PromoteNode(SDNode *Node) {
   // Replace the original node with the legalized result.
   if (!Results.empty())
     ReplaceNode(Node, Results.data());
 }
 
 // SelectionDAG::Legalize - This is the entry point for the file.
 //
 void SelectionDAG::Legalize() {
-  /// run - This is the main entry point to this class.
-  ///
-  SelectionDAGLegalize(*this).LegalizeDAG();
+  AssignTopologicalOrder();
+
+  allnodes_iterator LegalizePosition;
+  SmallPtrSet<SDNode *, 16> LegalizedNodes;
+  SelectionDAGLegalize Legalizer(*this, LegalizePosition, LegalizedNodes);
+
+  // Visit all the nodes. We start in topological order, so that we see
+  // nodes with their original operands intact. Legalization can produce
+  // new nodes which may themselves need to be legalized. Iterate until all
+  // nodes have been legalized.
+  for (;;) {
+    bool AnyLegalized = false;
+    for (LegalizePosition = allnodes_end();
+         LegalizePosition != allnodes_begin(); ) {
+      --LegalizePosition;
+
+      SDNode *N = LegalizePosition;
+      if (LegalizedNodes.insert(N)) {
+        AnyLegalized = true;
+        Legalizer.LegalizeOp(N);
+      }
+    }
+    if (!AnyLegalized)
+      break;
+
+  }
+
+  // Remove dead nodes now.
+  RemoveDeadNodes();
+}
+
+// This is a somewhat hacky entry point to legalize a single sub-graph rooted
+// at a node. This is used by the post-legalize DAG combiner to allow its DAG
+// combines to produce illegal nodes and still have a chance for them to be
+// legalized.
+bool SelectionDAG::LegalizeOp(SDNode *N,
+                              SmallSetVector<SDNode *, 16> &UpdatedNodes) {
+  allnodes_iterator LegalizePosition(N);
+  SmallPtrSet<SDNode *, 16> LegalizedNodes;
+  SelectionDAGLegalize Legalizer(*this, LegalizePosition, LegalizedNodes,
+                                 &UpdatedNodes);
+
+  // Directly insert the node in question, and legalize it. This will recurse
+  // as needed through operands.
+  LegalizedNodes.insert(N);
+  Legalizer.LegalizeOp(N);
+
+  return LegalizedNodes.count(N);
 }

test/CodeGen/R600/fp_to_sint.ll

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; Check that the compiler doesn't crash with a "cannot select" error
 ; SI: S_ENDPGM
 define void @fp_to_sint_i64 (i64 addrspace(1)* %out, float %in) {
 entry:
   %0 = fptosi float %in to i64
   store i64 %0, i64 addrspace(1)* %out
   ret void

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; SI: S_ENDPGM
 define void @fp_to_sint_v2i64(<2 x i64> addrspace(1)* %out, <2 x float> %x) {
   %conv = fptosi <2 x float> %x to <2 x i64>
   store <2 x i64> %conv, <2 x i64> addrspace(1)* %out
   ret void
 }
 

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; SI: S_ENDPGM
 define void @fp_to_sint_v4i64(<4 x i64> addrspace(1)* %out, <4 x float> %x) {
   %conv = fptosi <4 x float> %x to <4 x i64>
   store <4 x i64> %conv, <4 x i64> addrspace(1)* %out
   ret void
 }

test/CodeGen/R600/fp_to_uint.ll

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; SI: S_ENDPGM
 define void @fp_to_uint_i64(i64 addrspace(1)* %out, float %x) {
   %conv = fptoui float %x to i64
   store i64 %conv, i64 addrspace(1)* %out
   ret void
 }
 

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; SI: S_ENDPGM
 define void @fp_to_uint_v2i64(<2 x i64> addrspace(1)* %out, <2 x float> %x) {
   %conv = fptoui <2 x float> %x to <2 x i64>
   store <2 x i64> %conv, <2 x i64> addrspace(1)* %out
   ret void
 }
 

 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: LSHR
 ; EG-DAG: SUB_INT
 ; EG-DAG: AND_INT
 ; EG-DAG: ASHR
 ; EG-DAG: AND_INT
 ; EG-DAG: OR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHL
 ; EG-DAG: LSHL
 ; EG-DAG: SUB_INT
 ; EG-DAG: LSHR
 ; EG-DAG: LSHR
 ; EG-DAG: SETGT_UINT
 ; EG-DAG: SETGT_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: XOR_INT
 ; EG-DAG: SUB_INT
 ; EG-DAG: SUB_INT
-; EG-DAG: CNDGE_INT
-; EG-DAG: CNDGE_INT
+; EG-DAG: CNDE_INT
+; EG-DAG: CNDE_INT
 
 ; SI: S_ENDPGM
 define void @fp_to_uint_v4i64(<4 x i64> addrspace(1)* %out, <4 x float> %x) {
   %conv = fptoui <4 x float> %x to <4 x i64>
   store <4 x i64> %conv, <4 x i64> addrspace(1)* %out
   ret void
 }

test/CodeGen/R600/setcc-equivalent.ll

 ; RUN: llc -march=r600 -mcpu=cypress < %s | FileCheck -check-prefix=EG %s
-; XFAIL: *
 
 ; EG-LABEL: @and_setcc_setcc_i32
 ; EG: AND_INT
 ; EG-NEXT: SETE_INT
 define void @and_setcc_setcc_i32(i32 addrspace(1)* %out, i32 %a, i32 %b) {
   %cmp1 = icmp eq i32 %a, -1
   %cmp2 = icmp eq i32 %b, -1
   %and = and i1 %cmp1, %cmp2

 ; EG: SETE_INT
 define void @and_setcc_setcc_v4i32(<4 x i32> addrspace(1)* %out, <4 x i32> %a, <4 x i32> %b) {
   %cmp1 = icmp eq <4 x i32> %a, <i32 -1, i32 -1, i32 -1, i32 -1>
   %cmp2 = icmp eq <4 x i32> %b, <i32 -1, i32 -1, i32 -1, i32 -1>
   %and = and <4 x i1> %cmp1, %cmp2
   %ext = sext <4 x i1> %and to <4 x i32>
   store <4 x i32> %ext, <4 x i32> addrspace(1)* %out, align 4
   ret void
-}
+}
 

test/CodeGen/X86/avx512-cmp.ll

 l1:
   %c = fsub float %a, %b
   ret float %c
 l2:
   %c1 = fadd float %a, %b
   ret float %c1
 }
 
+; FIXME: Can use vcmpeqss and extract from the mask here in AVX512.
 ; CHECK-LABEL: test3
-; CHECK: vcmpeqss
-; CHECK: kmov
-; CHECK: ret
+; CHECK: vucomiss {{.*}}encoding: [0x62
 define i32 @test3(float %a, float %b) {
 
   %cmp10.i = fcmp oeq float %a, %b
   %conv11.i = zext i1 %cmp10.i to i32
   ret i32 %conv11.i
 }
 
 ; CHECK-LABEL: test5