[LegalizeVectorOps] Improve handling of multi-result operations.

[craig.topper]

Summary

This system wasn't very well designed for multi-result nodes. As
a consequence they weren't consistently registered in the
LegalizedNodes map leading to nodes being revisited for different
results.

I've removed the "Result" variable from the main LegalizeOp method
and used a SDNode* instead. The result number from the incoming
Op SDValue is only used for deciding which result to return to the
caller. When LegalizeOp is called it should always register a
legalized result for all of its results. Future calls for any other
result should be pulled for the LegalizedNodes map.

Legal nodes will now register all of their results in the map
instead of just the one we were called for.

The Expand and Promote handling to use a vector of results similar
to LegalizeDAG. Each of the new results is then re-legalized and
logged in the LegalizedNodes map for all of the Results for the
node being legalized. None of the handles register their own
results now. And none call ReplaceAllUsesOfValueWith now.

Custom handling now always passes result number 0 to LowerOperation.
This matches what LegalizeDAG does. Since the introduction of
STRICT nodes, I've encountered several issues with X86's custom
handling being called with an SDValue pointing at the chain and
our custom handlers using that to get a VT instead of result 0.
This should prevent us from having any more of those issues. On
return we will update the LegalizedNodes map for all results so
we shouldn't call the custom handler again for each result number.

I want to push SDNode* further into the Expand and Promote
handlers, but I've left that for a follow to keep this patch size
down. I've created a dummy SDValue(Node, 0) to keep the handlers
working.

This replaces D71861.

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

64

?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

64

Must have accidentally changed it when I copied the name into the commit message. Thanks!

[craig.topper]

Fix typo

[craig.topper]

Register the results of Expand and Promote in the LegalizedNodes map. Somehow I missed doing that before.

[arsenm]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

145–148

Extra space here.

Is there a real point to returning SDValues here instead of just the SDNode*? I assume this always returns (N, 0), (N, 1)

[kpn]

I'm happy to see the uses of ReplaceAllUsesOfValueWith() gone. Having nodes be only halfway spliced into the tree for a while seemed like it could cause bugs later. I imagine the scalar legalizer might could benefit from similar changes at some point in the future.

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

495

Is this a new case? Or was this a situation that could occur before your changes but was too subtle to see?

502

It seems like a code smell to have the rules duplicated in the Promote, Expand, and Custom cases. Is there a way to pull most of this block out into a function shared by the three cases? And yet the Custom case looks a little different from the others.

680

Nice. It's much cleaner than MERGE_VALUES.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

145–148

The handlers are calling out to TargetLowering and that's giving back two separate SDValues. I think they are different nodes. For example for UADDO, I think one result would be from an ADD and one would be from a SETCC.

495

It causes the assert below to fail in some cases. We never had the assert before and we only ever put one of the results from the custom handling into the map. You'll see there's a similar 1 result check in LegalizeDAG as well.

502

Custom is a bit nasty due to only returning one SDValue. I had to rework it again for D72238. I should probably move that part of the change over here as I'm not entirely sure the changes I had to make there aren't needed for other cases.

I'll look into merging the handling.

[efriedma]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

502

We should probably fix the whole LowerOperation/LowerOperationWrapper mess at some point, so custom legalization always returns a list of values. But I guess we don't have to do that here.

[craig.topper]

-Address review comments.
-Move the Expand/Promote result registration in the cache up to the caller as they need to use "Op" rather than "Node" to update the translation cache. Node almost always points to the same thing as Op, but the UpdateNodeOperands call that assigned Node might trigger CSE. When that happens we still need to use the original Op as the translation cache key since other nodes are holding that reference. This was the bug D71861 was trying to fix.
-Add the test case from D71861 which tests for that bug in the Custom path.

[kpn]

Looks like the test case didn't make it over.

[craig.topper]

With test case for real this time

[uweigand]

In general this all makes sense to me. See a few minor inline comments.

I agree with @kpn's comment about code duplication. I was wondering whether it might make sense to abstract some of that, even while the TLI.LowerOperation interface remains unchanged. E.g. we could have a routine like this:

bool CustomLower(SDNode *Node, SmallVectorImpl<SDValue> &Results) {
  if (SDValue Tmp = TLI.LowerOperation(SDValue(Node, 0), DAG)) {
    LLVM_DEBUG(dbgs() << "Successfully custom legalized node\n");
    if (Tmp == SDValue(Node, 0))
      return true;
    
    // Tmp might point to a single result from a multi result node, in that
    // case we need to use it's result number.
    if (Node->getNumValues() == 1) {
      Results.push_back(Tmp);
      return true;
    }
    
    // Otherwise it should be a multi-result node with the same number of
    // results.
    assert(Tmp->getNumValues() == Node->getNumValues() &&
           "Unexpected number of results");
    
    for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
      Results.push_back(Tmp.getValue(i));
    return true;
  }
  
  LLVM_DEBUG(dbgs() << "Could not custom legalize node\n");
  return false;
}

which could be used e.g. like so:

SmallVector<SDValue, 8> ResultVals;
if (CustomLower(Node, ResultVals)) {
  if (ResultVals.empty())
    return TranslateLegalizeResults(Op, Node);

  Changed = true;
  return RecursivelyLegalizeResults(Op, ResultVals);
}
LLVM_FALLTHROUGH;  // to expand

which would make the interfaces of the CustomLower / Expand / Promote cases the same, probably enabling further code sharing at the call site(s).

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

266

Lowered.getNode() must always be Node here, right?

480

Could just return the result of TranslateLegalizeResults, right?

517

Likewise, just return the result.

856

It's a bit odd to hard-code that some expanders may return a null SDValue and others may not. It may be preferable to just pass the Results vector through.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

856

Agreed. I was trying to minimize the changes to the handlers in this patch to keep the size down. I'm going to do a follow up patch to pass SDNode *N instead of Op to all the handlers. I can push Results while I'm doing that.

[craig.topper]

Add a LowerOperationWrapper helper that updates a Result vector. It looks a lot like X86's override of TargetLowering::LowerOperationWrapper except the bool result. The TargetLowering version doesn't allow for the possibility of keeping the existing node since it only called by the type legalizer today. I think in the future we could try to merge this together, but it will require changes to other targets.

Pass Results vector into the overflow operation expanders instead of returning std::pair. I plan to pass Results to all expanders in a follow up. Or possibly inline some of the smaller functions into the switch.

[uweigand]

OK, this looks all good to me now. I agree that there can be future API cleanup, but that can be done in follow-on patches. I think we should go with this for now.

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

 class VectorLegalizer {
   SelectionDAG& DAG;
   const TargetLowering &TLI;
   bool Changed = false; // Keep track of whether anything changed
 
   /// For nodes that are of legal width, and that have more than one use, this
   /// map indicates what regularized operand to use.  This allows us to avoid
   /// legalizing the same thing more than once.
   SmallDenseMap<SDValue, SDValue, 64> LegalizedNodes;
 
   /// Adds a node to the translation cache.
   void AddLegalizedOperand(SDValue From, SDValue To) {
     LegalizedNodes.insert(std::make_pair(From, To));
     // If someone requests legalization of the new node, return itself.
     if (From != To)
       LegalizedNodes.insert(std::make_pair(To, To));
   }
 
   /// Legalizes the given node.
   SDValue LegalizeOp(SDValue Op);
 
   /// Assuming the node is legal, "legalize" the results.
-  SDValue TranslateLegalizeResults(SDValue Op, SDValue Result);
+  SDValue TranslateLegalizeResults(SDValue Op, SDNode *Result);
 
   /// Implements unrolling a VSETCC.
   SDValue UnrollVSETCC(SDValue Op);
 
   /// Implement expand-based legalization of vector operations.
   ///
   /// This is just a high-level routine to dispatch to specific code paths for
   /// operations to legalize them.
-  SDValue Expand(SDValue Op);
+  SDValue Expand(SDNode *Node, unsigned ResNo);
 
   /// Implements expansion for FP_TO_UINT; falls back to UnrollVectorOp if
   /// FP_TO_SINT isn't legal.
-  SDValue ExpandFP_TO_UINT(SDValue Op);
+  void ExpandFP_TO_UINT(SDValue Op, SmallVectorImpl<SDValue> &Results);
 
   /// Implements expansion for UINT_TO_FLOAT; falls back to UnrollVectorOp if
   /// SINT_TO_FLOAT and SHR on vectors isn't legal.
-  SDValue ExpandUINT_TO_FLOAT(SDValue Op);
+  void ExpandUINT_TO_FLOAT(SDValue Op,
+                           SmallVectorImpl<SDValue> &Results);
 
   /// Implement expansion for SIGN_EXTEND_INREG using SRL and SRA.
   SDValue ExpandSEXTINREG(SDValue Op);
 
   /// Implement expansion for ANY_EXTEND_VECTOR_INREG.
   ///
   /// Shuffles the low lanes of the operand into place and bitcasts to the proper
   /// type. The contents of the bits in the extended part of each element are
class VectorLegalizer {
 
   /// Expand bswap of vectors into a shuffle if legal.
   SDValue ExpandBSWAP(SDValue Op);
 
   /// Implement vselect in terms of XOR, AND, OR when blend is not
   /// supported by the target.
   SDValue ExpandVSELECT(SDValue Op);
   SDValue ExpandSELECT(SDValue Op);
-  std::pair<SDValue, SDValue> ExpandLoad(SDValue Op);
-  SDValue ExpandStore(SDValue Op);
+  std::pair<SDValue, SDValue> ExpandLoad(SDNode *N);
+  SDValue ExpandStore(SDNode *N);
   SDValue ExpandFNEG(SDValue Op);
   SDValue ExpandFSUB(SDValue Op);
   SDValue ExpandBITREVERSE(SDValue Op);
   SDValue ExpandCTPOP(SDValue Op);
   SDValue ExpandCTLZ(SDValue Op);
   SDValue ExpandCTTZ(SDValue Op);
   SDValue ExpandFunnelShift(SDValue Op);
   SDValue ExpandROT(SDValue Op);
   SDValue ExpandFMINNUM_FMAXNUM(SDValue Op);
-  SDValue ExpandUADDSUBO(SDValue Op);
-  SDValue ExpandSADDSUBO(SDValue Op);
-  SDValue ExpandMULO(SDValue Op);
+  std::pair<SDValue, SDValue>  ExpandUADDSUBO(SDValue Op);
+  std::pair<SDValue, SDValue>  ExpandSADDSUBO(SDValue Op);
+  std::pair<SDValue, SDValue>  ExpandMULO(SDValue Op);
   SDValue ExpandAddSubSat(SDValue Op);
   SDValue ExpandFixedPointMul(SDValue Op);
-  SDValue ExpandStrictFPOp(SDValue Op);
+  void ExpandStrictFPOp(SDValue Op, SmallVectorImpl<SDValue> &Results);
 
-  SDValue UnrollStrictFPOp(SDValue Op);
+  void UnrollStrictFPOp(SDValue Op, SmallVectorImpl<SDValue> &Results);
 
   /// Implements vector promotion.
   ///
   /// This is essentially just bitcasting the operands to a different type and
   /// bitcasting the result back to the original type.
-  SDValue Promote(SDValue Op);
+  SDValue Promote(SDNode *Node, unsigned ResNo);
 
   /// Implements [SU]INT_TO_FP vector promotion.
   ///
   /// This is a [zs]ext of the input operand to a larger integer type.
-  SDValue PromoteINT_TO_FP(SDValue Op);
+  void PromoteINT_TO_FP(SDValue Op, SmallVectorImpl<SDValue> &Results);
 
   /// Implements FP_TO_[SU]INT vector promotion of the result type.
   ///
   /// It is promoted to a larger integer type.  The result is then
   /// truncated back to the original type.
-  SDValue PromoteFP_TO_INT(SDValue Op);
+  void PromoteFP_TO_INT(SDValue Op, SmallVectorImpl<SDValue> &Results);
 
 public:
   VectorLegalizer(SelectionDAG& dag) :
       DAG(dag), TLI(dag.getTargetLoweringInfo()) {}
 
   /// Begin legalizer the vector operations in the DAG.
   bool Run();
 };
bool VectorLegalizer::Run() {
   LegalizedNodes.clear();
 
   // Remove dead nodes now.
   DAG.RemoveDeadNodes();
 
   return Changed;
 }
 
-SDValue VectorLegalizer::TranslateLegalizeResults(SDValue Op, SDValue Result) {
+SDValue VectorLegalizer::TranslateLegalizeResults(SDValue Op, SDNode *Result) {
   // Generic legalization: just pass the operand through.
   for (unsigned i = 0, e = Op.getNode()->getNumValues(); i != e; ++i)
-    AddLegalizedOperand(Op.getValue(i), Result.getValue(i));
-  return Result.getValue(Op.getResNo());
+    AddLegalizedOperand(Op.getValue(i), SDValue(Result, i));
+  return SDValue(Result, Op.getResNo());
 }
 
 SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   // Note that LegalizeOp may be reentered even from single-use nodes, which
   // means that we always must cache transformed nodes.
   DenseMap<SDValue, SDValue>::iterator I = LegalizedNodes.find(Op);
   if (I != LegalizedNodes.end()) return I->second;
 
-  SDNode* Node = Op.getNode();
-
   // Legalize the operands
   SmallVector<SDValue, 8> Ops;
-  for (const SDValue &Op : Node->op_values())
-    Ops.push_back(LegalizeOp(Op));
+  for (const SDValue &Oper : Op->op_values())
+    Ops.push_back(LegalizeOp(Oper));
 
-  SDValue Result = SDValue(DAG.UpdateNodeOperands(Op.getNode(), Ops),
-                           Op.getResNo());
+  SDNode *Node = DAG.UpdateNodeOperands(Op.getNode(), Ops);
 
   if (Op.getOpcode() == ISD::LOAD) {
-    LoadSDNode *LD = cast<LoadSDNode>(Op.getNode());
+    LoadSDNode *LD = cast<LoadSDNode>(Node);
     ISD::LoadExtType ExtType = LD->getExtensionType();
     if (LD->getMemoryVT().isVector() && ExtType != ISD::NON_EXTLOAD) {
       LLVM_DEBUG(dbgs() << "\nLegalizing extending vector load: ";
                  Node->dump(&DAG));
       switch (TLI.getLoadExtAction(LD->getExtensionType(), LD->getValueType(0),
                                    LD->getMemoryVT())) {
       default: llvm_unreachable("This action is not supported yet!");
       case TargetLowering::Legal:
-        return TranslateLegalizeResults(Op, Result);
+        return TranslateLegalizeResults(Op, Node);
       case TargetLowering::Custom:
-        if (SDValue Lowered = TLI.LowerOperation(Result, DAG)) {
+        if (SDValue Lowered = TLI.LowerOperation(SDValue(Node, 0), DAG)) {
           assert(Lowered->getNumValues() == Op->getNumValues() &&
                  "Unexpected number of results");
-          if (Lowered != Result) {
+          if (Lowered != SDValue(Node, 0)) {
             // Make sure the new code is also legal.
             Lowered = LegalizeOp(Lowered);
             Changed = true;
           }
-          return TranslateLegalizeResults(Op, Lowered);
+          return TranslateLegalizeResults(Op, Lowered.getNode());
         }
         LLVM_FALLTHROUGH;
       case TargetLowering::Expand: {
         Changed = true;
-        std::pair<SDValue, SDValue> Tmp = ExpandLoad(Result);
+        std::pair<SDValue, SDValue> Tmp = ExpandLoad(Node);
         AddLegalizedOperand(Op.getValue(0), Tmp.first);
         AddLegalizedOperand(Op.getValue(1), Tmp.second);
         return Op.getResNo() ? Tmp.first : Tmp.second;
       }
       }
     }
   } else if (Op.getOpcode() == ISD::STORE) {
-    StoreSDNode *ST = cast<StoreSDNode>(Op.getNode());
+    StoreSDNode *ST = cast<StoreSDNode>(Node);
     EVT StVT = ST->getMemoryVT();
     MVT ValVT = ST->getValue().getSimpleValueType();
     if (StVT.isVector() && ST->isTruncatingStore()) {
       LLVM_DEBUG(dbgs() << "\nLegalizing truncating vector store: ";
                  Node->dump(&DAG));
       switch (TLI.getTruncStoreAction(ValVT, StVT)) {
       default: llvm_unreachable("This action is not supported yet!");
       case TargetLowering::Legal:
-        return TranslateLegalizeResults(Op, Result);
+        return TranslateLegalizeResults(Op, Node);
       case TargetLowering::Custom: {
-        SDValue Lowered = TLI.LowerOperation(Result, DAG);
-        if (Lowered != Result) {
+        SDValue Lowered = TLI.LowerOperation(SDValue(Node, 0), DAG);
+        if (Lowered != SDValue(Node, 0)) {
           // Make sure the new code is also legal.
           Lowered = LegalizeOp(Lowered);
           Changed = true;
         }
-        return TranslateLegalizeResults(Op, Lowered);
+        return TranslateLegalizeResults(Op, Lowered.getNode());
       }
       case TargetLowering::Expand: {
         Changed = true;
-        SDValue Chain = ExpandStore(Result);
+        SDValue Chain = ExpandStore(Node);
         AddLegalizedOperand(Op, Chain);
         return Chain;
       }
       }
     }
   }
 
   bool HasVectorValueOrOp = false;
   for (auto J = Node->value_begin(), E = Node->value_end(); J != E; ++J)
     HasVectorValueOrOp |= J->isVector();
-  for (const SDValue &Op : Node->op_values())
-    HasVectorValueOrOp |= Op.getValueType().isVector();
+  for (const SDValue &Oper : Node->op_values())
+    HasVectorValueOrOp |= Oper.getValueType().isVector();
 
   if (!HasVectorValueOrOp)
-    return TranslateLegalizeResults(Op, Result);
+    return TranslateLegalizeResults(Op, Node);
 
   TargetLowering::LegalizeAction Action = TargetLowering::Legal;
   EVT ValVT;
   switch (Op.getOpcode()) {
   default:
-    return TranslateLegalizeResults(Op, Result);
+    return TranslateLegalizeResults(Op, Node);
 #define INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN)                   \
   case ISD::STRICT_##DAGN:
 #include "llvm/IR/ConstrainedOps.def"
     ValVT = Node->getValueType(0);
     if (Op.getOpcode() == ISD::STRICT_SINT_TO_FP ||
         Op.getOpcode() == ISD::STRICT_UINT_TO_FP)
       ValVT = Node->getOperand(1).getValueType();
     Action = TLI.getOperationAction(Node->getOpcode(), ValVT);
Action = TLI.getOperationAction(Node->getOpcode(),
                                     Node->getOperand(0).getValueType());
     break;
   }
 
   LLVM_DEBUG(dbgs() << "\nLegalizing vector op: "; Node->dump(&DAG));
 
   switch (Action) {
   default: llvm_unreachable("This action is not supported yet!");
-  case TargetLowering::Promote:
-    Result = Promote(Op);
-    Changed = true;
-    break;
+  case TargetLowering::Promote: {
+    LLVM_DEBUG(dbgs() << "Promoting\n");
+    return Promote(Node, Op.getResNo());
+  }
   case TargetLowering::Legal:
     LLVM_DEBUG(dbgs() << "Legal node: nothing to do\n");
-    break;
+    TranslateLegalizeResults(Op, Node);
+    return SDValue(Node, Op.getResNo());
   case TargetLowering::Custom: {
     LLVM_DEBUG(dbgs() << "Trying custom legalization\n");
-    if (SDValue Tmp1 = TLI.LowerOperation(Op, DAG)) {
+    if (SDValue Tmp = TLI.LowerOperation(SDValue(Node, 0), DAG)) {
       LLVM_DEBUG(dbgs() << "Successfully custom legalized node\n");
-      Result = Tmp1;
-      break;
+      if (Tmp != SDValue(Node, 0)) {
+        // Make sure that the generated code is itself legal.
+        Tmp = LegalizeOp(Tmp);
+        Changed = true;
+      }
+
+      // Tmp might point to a single result from a multi result node, in that
+      // case we need to use it's result number.
+      if (Node->getNumValues() == 1) {
+        AddLegalizedOperand(Op, Tmp);
+        return Tmp;
+      }
+
+      // Otherwise it should be a multi-result node with the same number of
+      // results.
+      assert(Tmp->getNumValues() == Node->getNumValues() &&
+             "Unexpected number of results");
+      return TranslateLegalizeResults(Op, Tmp.getNode());
     }
     LLVM_DEBUG(dbgs() << "Could not custom legalize node\n");
     LLVM_FALLTHROUGH;
   }
   case TargetLowering::Expand:
-    Result = Expand(Op);
+    LLVM_DEBUG(dbgs() << "Expanding\n");
+    return Expand(Node, Op.getResNo());
   }
 
-  // Make sure that the generated code is itself legal.
-  if (Result != Op) {
-    Result = LegalizeOp(Result);
-    Changed = true;
-  }
+  llvm_unreachable("Unexpected legalization behavior!");
 
-  // Note that LegalizeOp may be reentered even from single-use nodes, which
-  // means that we always must cache transformed nodes.
-  AddLegalizedOperand(Op, Result);
-  return Result;
-}
+SDValue VectorLegalizer::Promote(SDNode *Node, unsigned ResNo) {
+  SDValue Op(Node, 0); // FIXME: Use Node throughout.
+
+  SmallVector<SDValue, 8> Results;
 
-SDValue VectorLegalizer::Promote(SDValue Op) {
   // For a few operations there is a specific concept for promotion based on
   // the operand's type.
   switch (Op.getOpcode()) {
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
   case ISD::STRICT_SINT_TO_FP:
   case ISD::STRICT_UINT_TO_FP:
     // "Promote" the operation by extending the operand.
-    return PromoteINT_TO_FP(Op);
+    PromoteINT_TO_FP(Op, Results);
+    break;
   case ISD::FP_TO_UINT:
   case ISD::FP_TO_SINT:
   case ISD::STRICT_FP_TO_UINT:
   case ISD::STRICT_FP_TO_SINT:
     // Promote the operation by extending the operand.
-    return PromoteFP_TO_INT(Op);
+    PromoteFP_TO_INT(Op, Results);
+    break;
   case ISD::FP_ROUND:
   case ISD::FP_EXTEND:
     // These operations are used to do promotion so they can't be promoted
     // themselves.
     llvm_unreachable("Don't know how to promote this operation!");
-  }
-
+  default: {
     // There are currently two cases of vector promotion:
     // 1) Bitcasting a vector of integers to a different type to a vector of the
     //    same overall length. For example, x86 promotes ISD::AND v2i32 to v1i64.
     // 2) Extending a vector of floats to a vector of the same number of larger
     //    floats. For example, AArch64 promotes ISD::FADD on v4f16 to v4f32.
     MVT VT = Op.getSimpleValueType();
     assert(Op.getNode()->getNumValues() == 1 &&
            "Can't promote a vector with multiple results!");
     MVT NVT = TLI.getTypeToPromoteTo(Op.getOpcode(), VT);
     SDLoc dl(Op);
     SmallVector<SDValue, 4> Operands(Op.getNumOperands());
 
     for (unsigned j = 0; j != Op.getNumOperands(); ++j) {
       if (Op.getOperand(j).getValueType().isVector())
         if (Op.getOperand(j)
                 .getValueType()
                 .getVectorElementType()
                 .isFloatingPoint() &&
             NVT.isVector() && NVT.getVectorElementType().isFloatingPoint())
           Operands[j] = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Op.getOperand(j));
         else
           Operands[j] = DAG.getNode(ISD::BITCAST, dl, NVT, Op.getOperand(j));
       else
         Operands[j] = Op.getOperand(j);
     }
 
-  Op = DAG.getNode(Op.getOpcode(), dl, NVT, Operands, Op.getNode()->getFlags());
+    Op = DAG.getNode(Op.getOpcode(), dl, NVT, Operands,
+                     Op.getNode()->getFlags());
+
+    SDValue Res;
     if ((VT.isFloatingPoint() && NVT.isFloatingPoint()) ||
         (VT.isVector() && VT.getVectorElementType().isFloatingPoint() &&
          NVT.isVector() && NVT.getVectorElementType().isFloatingPoint()))
-    return DAG.getNode(ISD::FP_ROUND, dl, VT, Op, DAG.getIntPtrConstant(0, dl));
+      Res = DAG.getNode(ISD::FP_ROUND, dl, VT, Op, DAG.getIntPtrConstant(0, dl));
     else
-    return DAG.getNode(ISD::BITCAST, dl, VT, Op);
+      Res = DAG.getNode(ISD::BITCAST, dl, VT, Op);
+
+    Results.push_back(Res);
+    break;
   }
+  }
+
+  assert(Results.size() == Node->getNumValues() &&
+         "Unexpected number of results");
 
-SDValue VectorLegalizer::PromoteINT_TO_FP(SDValue Op) {
+  // Make sure that the generated code is itself legal.
+  for (unsigned i = 0, e = Results.size(); i != e; ++i) {
+    Results[i] = LegalizeOp(Results[i]);
+    AddLegalizedOperand(SDValue(Node, i), Results[i]);
+  }
+
+  Changed = true;
+  return Results[ResNo];
+}
+
+void VectorLegalizer::PromoteINT_TO_FP(SDValue Op,
+                                       SmallVectorImpl<SDValue> &Results) {
   // INT_TO_FP operations may require the input operand be promoted even
   // when the type is otherwise legal.
   bool IsStrict = Op->isStrictFPOpcode();
   MVT VT = Op.getOperand(IsStrict ? 1 : 0).getSimpleValueType();
   MVT NVT = TLI.getTypeToPromoteTo(Op.getOpcode(), VT);
   assert(NVT.getVectorNumElements() == VT.getVectorNumElements() &&
          "Vectors have different number of elements!");
 
   SDLoc dl(Op);
   SmallVector<SDValue, 4> Operands(Op.getNumOperands());
 
   unsigned Opc = (Op.getOpcode() == ISD::UINT_TO_FP ||
                   Op.getOpcode() == ISD::STRICT_UINT_TO_FP)
                      ? ISD::ZERO_EXTEND
                      : ISD::SIGN_EXTEND;
   for (unsigned j = 0; j != Op.getNumOperands(); ++j) {
     if (Op.getOperand(j).getValueType().isVector())
       Operands[j] = DAG.getNode(Opc, dl, NVT, Op.getOperand(j));
     else
       Operands[j] = Op.getOperand(j);
   }
 
-  if (IsStrict)
-    return DAG.getNode(Op.getOpcode(), dl, {Op.getValueType(), MVT::Other},
-                       Operands);
+  if (IsStrict) {
+    SDValue Res = DAG.getNode(Op.getOpcode(), dl,
+                              {Op.getValueType(), MVT::Other}, Operands);
+    Results.push_back(Res);
+    Results.push_back(Res.getValue(1));
+    return;
+  }
 
-  return DAG.getNode(Op.getOpcode(), dl, Op.getValueType(), Operands);
+  SDValue Res = DAG.getNode(Op.getOpcode(), dl, Op.getValueType(), Operands);
+  Results.push_back(Res);
 }
 
 // For FP_TO_INT we promote the result type to a vector type with wider
 // elements and then truncate the result.  This is different from the default
 // PromoteVector which uses bitcast to promote thus assumning that the
 // promoted vector type has the same overall size.
-SDValue VectorLegalizer::PromoteFP_TO_INT(SDValue Op) {
+void VectorLegalizer::PromoteFP_TO_INT(SDValue Op,
+                                       SmallVectorImpl<SDValue> &Results) {
   MVT VT = Op.getSimpleValueType();
   MVT NVT = TLI.getTypeToPromoteTo(Op.getOpcode(), VT);
   bool IsStrict = Op->isStrictFPOpcode();
   assert(NVT.getVectorNumElements() == VT.getVectorNumElements() &&
          "Vectors have different number of elements!");
 
   unsigned NewOpc = Op->getOpcode();
   // Change FP_TO_UINT to FP_TO_SINT if possible.
if (Op->getOpcode() == ISD::FP_TO_UINT ||
       Op->getOpcode() == ISD::STRICT_FP_TO_UINT)
     NewOpc = ISD::AssertZext;
   else
     NewOpc = ISD::AssertSext;
 
   Promoted = DAG.getNode(NewOpc, dl, NVT, Promoted,
                          DAG.getValueType(VT.getScalarType()));
   Promoted = DAG.getNode(ISD::TRUNCATE, dl, VT, Promoted);
+  Results.push_back(Promoted);
   if (IsStrict)
-    return DAG.getMergeValues({Promoted, Chain}, dl);
-
-  return Promoted;
+    Results.push_back(Chain);
 }
 
-std::pair<SDValue, SDValue> VectorLegalizer::ExpandLoad(SDValue Op) {
-  LoadSDNode *LD = cast<LoadSDNode>(Op.getNode());
+std::pair<SDValue, SDValue> VectorLegalizer::ExpandLoad(SDNode *N) {
+  LoadSDNode *LD = cast<LoadSDNode>(N);
 
   EVT SrcVT = LD->getMemoryVT();
   EVT SrcEltVT = SrcVT.getScalarType();
   unsigned NumElem = SrcVT.getVectorNumElements();
 
   SDValue NewChain;
   SDValue Value;
   if (SrcVT.getVectorNumElements() > 1 && !SrcEltVT.isByteSized()) {
-    SDLoc dl(Op);
+    SDLoc dl(N);
 
     SmallVector<SDValue, 8> Vals;
     SmallVector<SDValue, 8> LoadChains;
 
     EVT DstEltVT = LD->getValueType(0).getScalarType();
     SDValue Chain = LD->getChain();
     SDValue BasePTR = LD->getBasePtr();
     ISD::LoadExtType ExtType = LD->getExtensionType();
for (unsigned Idx = 0; Idx != NumElem; ++Idx) {
         Lo = DAG.getNode(ISD::SRA, dl, WideVT, Lo, ShAmt);
         Lo = DAG.getSExtOrTrunc(Lo, dl, DstEltVT);
         break;
       }
       Vals.push_back(Lo);
     }
 
     NewChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, LoadChains);
-    Value = DAG.getBuildVector(Op.getNode()->getValueType(0), dl, Vals);
+    Value = DAG.getBuildVector(N->getValueType(0), dl, Vals);
   } else {
     std::tie(Value, NewChain) = TLI.scalarizeVectorLoad(LD, DAG);
   }
 
   return std::make_pair(Value, NewChain);
 }
 
-SDValue VectorLegalizer::ExpandStore(SDValue Op) {
-  StoreSDNode *ST = cast<StoreSDNode>(Op.getNode());
+SDValue VectorLegalizer::ExpandStore(SDNode *N) {
+  StoreSDNode *ST = cast<StoreSDNode>(N);
   SDValue TF = TLI.scalarizeVectorStore(ST, DAG);
   return TF;
 }
 
-SDValue VectorLegalizer::Expand(SDValue Op) {
+SDValue VectorLegalizer::Expand(SDNode *Node, unsigned ResNo) {
+  SDValue Op(Node, 0); // FIXME: Just pass Node to all the expanders.
+
+  SmallVector<SDValue, 8> Results;
   switch (Op->getOpcode()) {
   case ISD::SIGN_EXTEND_INREG:
-    return ExpandSEXTINREG(Op);
+    Results.push_back(ExpandSEXTINREG(Op));
+    break;
   case ISD::ANY_EXTEND_VECTOR_INREG:
-    return ExpandANY_EXTEND_VECTOR_INREG(Op);
+    Results.push_back(ExpandANY_EXTEND_VECTOR_INREG(Op));
+    break;
   case ISD::SIGN_EXTEND_VECTOR_INREG:
-    return ExpandSIGN_EXTEND_VECTOR_INREG(Op);
+    Results.push_back(ExpandSIGN_EXTEND_VECTOR_INREG(Op));
+    break;
   case ISD::ZERO_EXTEND_VECTOR_INREG:
-    return ExpandZERO_EXTEND_VECTOR_INREG(Op);
+    Results.push_back(ExpandZERO_EXTEND_VECTOR_INREG(Op));
+    break;
   case ISD::BSWAP:
-    return ExpandBSWAP(Op);
+    Results.push_back(ExpandBSWAP(Op));
+    break;
   case ISD::VSELECT:
-    return ExpandVSELECT(Op);
+    Results.push_back(ExpandVSELECT(Op));
+    break;
   case ISD::SELECT:
-    return ExpandSELECT(Op);
+    Results.push_back(ExpandSELECT(Op));
+    break;
   case ISD::FP_TO_UINT:
-    return ExpandFP_TO_UINT(Op);
+    ExpandFP_TO_UINT(Op, Results);
+    break;
   case ISD::UINT_TO_FP:
-    return ExpandUINT_TO_FLOAT(Op);
+    ExpandUINT_TO_FLOAT(Op, Results);
+    break;
   case ISD::FNEG:
-    return ExpandFNEG(Op);
+    Results.push_back(ExpandFNEG(Op));
+    break;
   case ISD::FSUB:
-    return ExpandFSUB(Op);
+    if (SDValue Tmp = ExpandFSUB(Op))
+      Results.push_back(Tmp);
+    break;
   case ISD::SETCC:
-    return UnrollVSETCC(Op);
+    Results.push_back(UnrollVSETCC(Op));
+    break;
   case ISD::ABS:
-    return ExpandABS(Op);
+    Results.push_back(ExpandABS(Op));
+    break;
   case ISD::BITREVERSE:
-    return ExpandBITREVERSE(Op);
+    if (SDValue Tmp = ExpandBITREVERSE(Op))
+      Results.push_back(Tmp);
+    break;
   case ISD::CTPOP:
-    return ExpandCTPOP(Op);
+    Results.push_back(ExpandCTPOP(Op));
+    break;
   case ISD::CTLZ:
   case ISD::CTLZ_ZERO_UNDEF:
-    return ExpandCTLZ(Op);
+    Results.push_back(ExpandCTLZ(Op));
+    break;
   case ISD::CTTZ:
   case ISD::CTTZ_ZERO_UNDEF:
-    return ExpandCTTZ(Op);
+    Results.push_back(ExpandCTTZ(Op));
+    break;
   case ISD::FSHL:
   case ISD::FSHR:
-    return ExpandFunnelShift(Op);
+    Results.push_back(ExpandFunnelShift(Op));
+    break;
   case ISD::ROTL:
   case ISD::ROTR:
-    return ExpandROT(Op);
+    Results.push_back(ExpandROT(Op));
+    break;
   case ISD::FMINNUM:
   case ISD::FMAXNUM:
-    return ExpandFMINNUM_FMAXNUM(Op);
+    Results.push_back(ExpandFMINNUM_FMAXNUM(Op));
+    break;
   case ISD::UADDO:
-  case ISD::USUBO:
-    return ExpandUADDSUBO(Op);
+  case ISD::USUBO: {
+    SDValue Result, Overflow;
+    std::tie(Result, Overflow) = ExpandUADDSUBO(Op);
+    Results.push_back(Result);
+    Results.push_back(Overflow);
+    break;
+  }
   case ISD::SADDO:
-  case ISD::SSUBO:
-    return ExpandSADDSUBO(Op);
+  case ISD::SSUBO: {
+    SDValue Result, Overflow;
+    std::tie(Result, Overflow) = ExpandSADDSUBO(Op);
+    Results.push_back(Result);
+    Results.push_back(Overflow);
+    break;
+  }
   case ISD::UMULO:
-  case ISD::SMULO:
-    return ExpandMULO(Op);
+  case ISD::SMULO: {
+    SDValue Result, Overflow;
+    std::tie(Result, Overflow) = ExpandMULO(Op);
+    Results.push_back(Result);
+    Results.push_back(Overflow);
+    break;
+  }
   case ISD::USUBSAT:
   case ISD::SSUBSAT:
   case ISD::UADDSAT:
   case ISD::SADDSAT:
-    return ExpandAddSubSat(Op);
+    Results.push_back(ExpandAddSubSat(Op));
+    break;
   case ISD::SMULFIX:
   case ISD::UMULFIX:
-    return ExpandFixedPointMul(Op);
+    Results.push_back(ExpandFixedPointMul(Op));
+    break;
   case ISD::SMULFIXSAT:
   case ISD::UMULFIXSAT:
     // FIXME: We do not expand SMULFIXSAT/UMULFIXSAT here yet, not sure exactly
     // why. Maybe it results in worse codegen compared to the unroll for some
     // targets? This should probably be investigated. And if we still prefer to
     // unroll an explanation could be helpful.
-    return DAG.UnrollVectorOp(Op.getNode());
+    Results.push_back(DAG.UnrollVectorOp(Op.getNode()));
+    break;
 #define INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN)                   \
   case ISD::STRICT_##DAGN:
 #include "llvm/IR/ConstrainedOps.def"
-    return ExpandStrictFPOp(Op);
+    ExpandStrictFPOp(Op, Results);
+    break;
   case ISD::VECREDUCE_ADD:
   case ISD::VECREDUCE_MUL:
   case ISD::VECREDUCE_AND:
   case ISD::VECREDUCE_OR:
   case ISD::VECREDUCE_XOR:
   case ISD::VECREDUCE_SMAX:
   case ISD::VECREDUCE_SMIN:
   case ISD::VECREDUCE_UMAX:
   case ISD::VECREDUCE_UMIN:
   case ISD::VECREDUCE_FADD:
   case ISD::VECREDUCE_FMUL:
   case ISD::VECREDUCE_FMAX:
   case ISD::VECREDUCE_FMIN:
-    return TLI.expandVecReduce(Op.getNode(), DAG);
+    Results.push_back(TLI.expandVecReduce(Op.getNode(), DAG));
+    break;
   default:
-    return DAG.UnrollVectorOp(Op.getNode());
+    Results.push_back(DAG.UnrollVectorOp(Op.getNode()));
+    break;
   }
+
+  if (Results.empty()) {
+    TranslateLegalizeResults(Op, Node);
+    return SDValue(Node, ResNo);
+  }
+
+  assert(Results.size() == Node->getNumValues() &&
+         "Unexpected number of results");
+
+  // Make sure that the generated code is itself legal.
+  for (unsigned i = 0, e = Results.size(); i != e; ++i) {
+    Results[i] = LegalizeOp(Results[i]);
+    AddLegalizedOperand(SDValue(Node, i), Results[i]);
+  }
+
+  Changed = true;
+  return Results[ResNo];
 }
 
 SDValue VectorLegalizer::ExpandSELECT(SDValue Op) {
   // Lower a select instruction where the condition is a scalar and the
   // operands are vectors. Lower this select to VSELECT and implement it
   // using XOR AND OR. The selector bit is broadcasted.
   EVT VT = Op.getValueType();
   SDLoc DL(Op);
SDValue VectorLegalizer::ExpandBITREVERSE(SDValue Op) {
   // than unrolling and expanding each component.
   if (!TLI.isOperationLegalOrCustom(ISD::SHL, VT) ||
       !TLI.isOperationLegalOrCustom(ISD::SRL, VT) ||
       !TLI.isOperationLegalOrCustomOrPromote(ISD::AND, VT) ||
       !TLI.isOperationLegalOrCustomOrPromote(ISD::OR, VT))
     return DAG.UnrollVectorOp(Op.getNode());
 
   // Let LegalizeDAG handle this later.
-  return Op;
+  return SDValue();
 }
 
 SDValue VectorLegalizer::ExpandVSELECT(SDValue Op) {
   // Implement VSELECT in terms of XOR, AND, OR
   // on platforms which do not support blend natively.
   SDLoc DL(Op);
 
   SDValue Mask = Op.getOperand(0);
SDValue VectorLegalizer::ExpandABS(SDValue Op) {
   SDValue Result;
   if (TLI.expandABS(Op.getNode(), Result, DAG))
     return Result;
 
   // Otherwise go ahead and unroll.
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
-SDValue VectorLegalizer::ExpandFP_TO_UINT(SDValue Op) {
+void VectorLegalizer::ExpandFP_TO_UINT(SDValue Op,
+                                       SmallVectorImpl<SDValue> &Results) {
   // Attempt to expand using TargetLowering.
   SDValue Result, Chain;
   if (TLI.expandFP_TO_UINT(Op.getNode(), Result, Chain, DAG)) {
+    Results.push_back(Result);
     if (Op->isStrictFPOpcode())
-      // Relink the chain
-      DAG.ReplaceAllUsesOfValueWith(Op.getValue(1), Chain);
-    return Result;
+      Results.push_back(Chain);
+    return;
   }
 
   // Otherwise go ahead and unroll.
-  if (Op->isStrictFPOpcode())
-    return UnrollStrictFPOp(Op);
-  return DAG.UnrollVectorOp(Op.getNode());
+  if (Op->isStrictFPOpcode()) {
+    UnrollStrictFPOp(Op, Results);
+    return;
+  }
+
+  Results.push_back(DAG.UnrollVectorOp(Op.getNode()));
 }
 
-SDValue VectorLegalizer::ExpandUINT_TO_FLOAT(SDValue Op) {
+void VectorLegalizer::ExpandUINT_TO_FLOAT(SDValue Op,
+                                          SmallVectorImpl<SDValue> &Results) {
   bool IsStrict = Op.getNode()->isStrictFPOpcode();
   unsigned OpNo = IsStrict ? 1 : 0;
   SDValue Src = Op.getOperand(OpNo);
   EVT VT = Src.getValueType();
   SDLoc DL(Op);
 
   // Attempt to expand using TargetLowering.
   SDValue Result;
   SDValue Chain;
   if (TLI.expandUINT_TO_FP(Op.getNode(), Result, Chain, DAG)) {
+    Results.push_back(Result);
     if (IsStrict)
-      // Relink the chain
-      DAG.ReplaceAllUsesOfValueWith(Op.getValue(1), Chain);
-    return Result;
+      Results.push_back(Chain);
+    return;
   }
 
   // Make sure that the SINT_TO_FP and SRL instructions are available.
   if (((!IsStrict && TLI.getOperationAction(ISD::SINT_TO_FP, VT) ==
                          TargetLowering::Expand) ||
        (IsStrict && TLI.getOperationAction(ISD::STRICT_SINT_TO_FP, VT) ==
                         TargetLowering::Expand)) ||
       TLI.getOperationAction(ISD::SRL, VT) == TargetLowering::Expand) {
-    if (IsStrict)
-      return UnrollStrictFPOp(Op);
-    return DAG.UnrollVectorOp(Op.getNode());
+    if (IsStrict) {
+      UnrollStrictFPOp(Op, Results);
+      return;
+    }
+
+    Results.push_back(DAG.UnrollVectorOp(Op.getNode()));
+    return;
   }
 
   unsigned BW = VT.getScalarSizeInBits();
   assert((BW == 64 || BW == 32) &&
          "Elements in vector-UINT_TO_FP must be 32 or 64 bits wide");
 
   SDValue HalfWord = DAG.getConstant(BW / 2, DL, VT);
 
SDValue fLO =
         DAG.getNode(ISD::STRICT_SINT_TO_FP, DL, {Op.getValueType(), MVT::Other},
                     {SDValue(fHI.getNode(), 1), LO});
 
     // Add the two halves
     SDValue Result =
         DAG.getNode(ISD::STRICT_FADD, DL, {Op.getValueType(), MVT::Other},
                     {SDValue(fLO.getNode(), 1), fHI, fLO});
 
-    // Relink the chain
-    DAG.ReplaceAllUsesOfValueWith(Op.getValue(1), SDValue(Result.getNode(), 1));
-    return Result;
+    Results.push_back(Result);
+    Results.push_back(Result.getValue(1));
+    return;
   }
 
   // Convert hi and lo to floats
   // Convert the hi part back to the upper values
   // TODO: Can any fast-math-flags be set on these nodes?
   SDValue fHI = DAG.getNode(ISD::SINT_TO_FP, DL, Op.getValueType(), HI);
           fHI = DAG.getNode(ISD::FMUL, DL, Op.getValueType(), fHI, TWOHW);
   SDValue fLO = DAG.getNode(ISD::SINT_TO_FP, DL, Op.getValueType(), LO);
 
   // Add the two halves
-  return DAG.getNode(ISD::FADD, DL, Op.getValueType(), fHI, fLO);
+  Results.push_back(DAG.getNode(ISD::FADD, DL, Op.getValueType(), fHI, fLO));
 }
 
 SDValue VectorLegalizer::ExpandFNEG(SDValue Op) {
   if (TLI.isOperationLegalOrCustom(ISD::FSUB, Op.getValueType())) {
     SDLoc DL(Op);
     SDValue Zero = DAG.getConstantFP(-0.0, DL, Op.getValueType());
     // TODO: If FNEG had fast-math-flags, they'd get propagated to this FSUB.
     return DAG.getNode(ISD::FSUB, DL, Op.getValueType(),
                        Zero, Op.getOperand(0));
   }
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
 SDValue VectorLegalizer::ExpandFSUB(SDValue Op) {
   // For floating-point values, (a-b) is the same as a+(-b). If FNEG is legal,
   // we can defer this to operation legalization where it will be lowered as
   // a+(-b).
   EVT VT = Op.getValueType();
   if (TLI.isOperationLegalOrCustom(ISD::FNEG, VT) &&
       TLI.isOperationLegalOrCustom(ISD::FADD, VT))
-    return Op; // Defer to LegalizeDAG
+    return SDValue(); // Defer to LegalizeDAG
 
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
 SDValue VectorLegalizer::ExpandCTPOP(SDValue Op) {
   SDValue Result;
   if (TLI.expandCTPOP(Op.getNode(), Result, DAG))
     return Result;

 }
 
 SDValue VectorLegalizer::ExpandFMINNUM_FMAXNUM(SDValue Op) {
   if (SDValue Expanded = TLI.expandFMINNUM_FMAXNUM(Op.getNode(), DAG))
     return Expanded;
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
-SDValue VectorLegalizer::ExpandUADDSUBO(SDValue Op) {
+std::pair<SDValue, SDValue> VectorLegalizer::ExpandUADDSUBO(SDValue Op) {
   SDValue Result, Overflow;
   TLI.expandUADDSUBO(Op.getNode(), Result, Overflow, DAG);
-
-  if (Op.getResNo() == 0) {
-    AddLegalizedOperand(Op.getValue(1), LegalizeOp(Overflow));
-    return Result;
-  } else {
-    AddLegalizedOperand(Op.getValue(0), LegalizeOp(Result));
-    return Overflow;
-  }
+  return std::make_pair(Result, Overflow);
 }
 
-SDValue VectorLegalizer::ExpandSADDSUBO(SDValue Op) {
+std::pair<SDValue, SDValue> VectorLegalizer::ExpandSADDSUBO(SDValue Op) {
   SDValue Result, Overflow;
   TLI.expandSADDSUBO(Op.getNode(), Result, Overflow, DAG);
-
-  if (Op.getResNo() == 0) {
-    AddLegalizedOperand(Op.getValue(1), LegalizeOp(Overflow));
-    return Result;
-  } else {
-    AddLegalizedOperand(Op.getValue(0), LegalizeOp(Result));
-    return Overflow;
-  }
+  return std::make_pair(Result, Overflow);
 }
 
-SDValue VectorLegalizer::ExpandMULO(SDValue Op) {
+std::pair<SDValue, SDValue> VectorLegalizer::ExpandMULO(SDValue Op) {
   SDValue Result, Overflow;
   if (!TLI.expandMULO(Op.getNode(), Result, Overflow, DAG))
     std::tie(Result, Overflow) = DAG.UnrollVectorOverflowOp(Op.getNode());
-
-  if (Op.getResNo() == 0) {
-    AddLegalizedOperand(Op.getValue(1), LegalizeOp(Overflow));
-    return Result;
-  } else {
-    AddLegalizedOperand(Op.getValue(0), LegalizeOp(Result));
-    return Overflow;
-  }
+  return std::make_pair(Result, Overflow);
 }
 
 SDValue VectorLegalizer::ExpandAddSubSat(SDValue Op) {
   if (SDValue Expanded = TLI.expandAddSubSat(Op.getNode(), DAG))
     return Expanded;
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
 SDValue VectorLegalizer::ExpandFixedPointMul(SDValue Op) {
   if (SDValue Expanded = TLI.expandFixedPointMul(Op.getNode(), DAG))
     return Expanded;
   return DAG.UnrollVectorOp(Op.getNode());
 }
 
-SDValue VectorLegalizer::ExpandStrictFPOp(SDValue Op) {
-  if (Op.getOpcode() == ISD::STRICT_UINT_TO_FP)
-    return ExpandUINT_TO_FLOAT(Op);
-  if (Op.getOpcode() == ISD::STRICT_FP_TO_UINT)
-    return ExpandFP_TO_UINT(Op);
+void VectorLegalizer::ExpandStrictFPOp(SDValue Op,
+                                       SmallVectorImpl<SDValue> &Results) {
+  if (Op.getOpcode() == ISD::STRICT_UINT_TO_FP) {
+    ExpandUINT_TO_FLOAT(Op, Results);
+    return;
+  }
+  if (Op.getOpcode() == ISD::STRICT_FP_TO_UINT) {
+    ExpandFP_TO_UINT(Op, Results);
+    return;
+  }
 
-  return UnrollStrictFPOp(Op);
+  UnrollStrictFPOp(Op, Results);
 }
 
-SDValue VectorLegalizer::UnrollStrictFPOp(SDValue Op) {
+void VectorLegalizer::UnrollStrictFPOp(SDValue Op,
+                                       SmallVectorImpl<SDValue> &Results) {
   EVT VT = Op.getValue(0).getValueType();
   EVT EltVT = VT.getVectorElementType();
   unsigned NumElems = VT.getVectorNumElements();
   unsigned NumOpers = Op.getNumOperands();
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   EVT TmpEltVT = EltVT;
   if (Op->getOpcode() == ISD::STRICT_FSETCC ||
for (unsigned i = 0; i < NumElems; ++i) {
 
     OpValues.push_back(ScalarResult);
     OpChains.push_back(ScalarChain);
   }
 
   SDValue Result = DAG.getBuildVector(VT, dl, OpValues);
   SDValue NewChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OpChains);
 
-  AddLegalizedOperand(Op.getValue(0), Result);
-  AddLegalizedOperand(Op.getValue(1), NewChain);
-
-  return Op.getResNo() ? NewChain : Result;
+  Results.push_back(Result);
+  Results.push_back(NewChain);
 }
 
 SDValue VectorLegalizer::UnrollVSETCC(SDValue Op) {
   EVT VT = Op.getValueType();
   unsigned NumElems = VT.getVectorNumElements();
   EVT EltVT = VT.getVectorElementType();
   SDValue LHS = Op.getOperand(0), RHS = Op.getOperand(1), CC = Op.getOperand(2);
   EVT TmpEltVT = LHS.getValueType().getVectorElementType();