[SelectionDAG] Add expansion and promotion of [US]MUL_LOHI

[nhaehnle]

Summary

Most targets set the action for these nodes to Expand even though there
isn't actually any code for them in ExpandNode. Instead, targets simply
relied on the fact that no code generates these nodes as long as the
nodes aren't legal or custom.

However, generating these nodes can be useful e.g. for divide-by-constant
in wider integer types. Targets will want to deal with [US]MUL_LOHI
differently depending on the available instructions, e.g. targets with a
native 64-bit multiply will want to Promote the 32-bit [US]MUL_LOHI
instructions, and targets with native MULH[US] instructions will want to
Split.

This patch intends to not change the generated code, but indirect effects
are possible since expansions/promotions that were previously done in
DAGCombine may now be done in LegalizeDAG.

See D24822 for a change that actually uses the new expansion.

[nhaehnle]

Ping.

[efriedma]

include/llvm/Target/TargetLowering.h

147

It doesn't look like None is actually used anywhere... did you mean to use it somehow?

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3156

Why are you using ADDC+ADDE rather than just plain ADD? It seems strange to split the operation, given that it might be legal. (I might be missing something here, though.)

lib/Target/Sparc/SparcISelLowering.cpp

1696

This FIXME doesn't make sense here; Promote is just the right thing to do here.

lib/Target/X86/X86ISelLowering.cpp

32532

I'm confused; you're returning "Scalarize" for scalar types?

[nhaehnle]

Thank you for taking a look. This patch should address all your comments,
plus I'm adding a guard against incorrectly using the known bits
optimizations with vector types.

[nhaehnle]

include/llvm/Target/TargetLowering.h

147

I used it at some point to preserve the original behaviour. I'll remove it.

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3156

That's just how the code happened to evolve because AMDGPU (which is where this all started) doesn't have 64-bit additions, but it's a good point. I'm changing it to use additions in the larger type.

lib/Target/Sparc/SparcISelLowering.cpp

1696

FIXME removed.

lib/Target/X86/X86ISelLowering.cpp

32532

On X86 this only gets called for vector types. I'm adding an assertion to make that explicit.

[jyknight]

I don't really understand why this needs to be a new target hook. Can't it have similar code as the MUL expansion code at LegalizeDAG.cpp:3316 (in the base version): if MULH and MUL are LegalOrCustom, then you'd split the MUL_LOHI into those two operations. Otherwise, you'd do half-width multiplies?

Also, I've fixed the sparc backend to properly implement [US]MUL_LOHI in r283381, removing the FIXME here.

[nhaehnle]

In D24956#562725, @jyknight wrote:

I don't really understand why this needs to be a new target hook. Can't it have similar code as the MUL expansion code at LegalizeDAG.cpp:3316 (in the base version): if MULH and MUL are LegalOrCustom, then you'd split the MUL_LOHI into those two operations. Otherwise, you'd do half-width multiplies?

IMHO, explicit is better than implicit. Also, how would you choose between scalarizing and half-width multiplies for vector types? You might be able to come up with a magic heuristic that works now, only to be broken when a new backend comes along.

Also, I've fixed the sparc backend to properly implement [US]MUL_LOHI in r283381, removing the FIXME here.

Thanks!

[jyknight]

IMHO, explicit is better than implicit. Also, how would you choose between scalarizing and half-width multiplies for vector types? You might be able to come up with a magic heuristic that works now, only to be broken when a new backend comes along.

I don't think it makes much sense to require backends to decide between Split and Half; whenever you can split into MULH and MUL, you should, and do 4 half-width multiplies otherwise.

I don't know much about the vector question -- how is that done for other operations? Surely the question about which axis to "expand" when expanding a vector operation isn't only an question for multiplication, so a MUL-specific solution to this issue feels strange -- I'd expect that either there's already a solution in use elsewhere, or a wider problem.

[RKSimon]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3035

If this is going to start being used by vectors, shouldn't these getSizeInBits() calls be replaced with getScalarSizeInBits()?

[efriedma]

Legalization for certain vector operations uses "PROMOTE" to indicate that the operation should be performed in a different vector type. That really stretches the meaning if we're using it to mean "split into multiple vector multiplies", but I guess it could work. See VectorLegalizer::Promote in LegalizeVectorOps.cpp.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3331

Sorry, I really haven't worked with this code in a while; I should have spotted the issue here earlier. :(

We can't ever scalarize an operation in LegalizeDAG: scalarizing can produce illegal types, and LegalizeDAG can't handle them. We have to handle this a bit earlier, in LegalizeVectorOps.

[nhaehnle]

Thank you all for looking into this and sorry for the delay. I think I've
addressed all your comments:

• getMulExpansion is gone
• vector scalarization is done in LegalizeVectorOps
• use getScalarSizeInBits() throughout to prepare for vector use

There's also the question of "promoting" / using half-width multiplies for
vectors. That could become useful to have more aggressive replacement of
divide-by-constant with the equivalent multiply+shift sequence also for
vector types, i.e. pushing D24822 even further.

However, there are more problems there (X86 gets unnecessarily worse looking
instruction sequences in some tests), so that should be a separate change.

[nhaehnle]

Simplify the Expand part in LegalizeDAG, since vectors are now expanded
earlier.

[nhaehnle]

Ping.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3331

Okay, fixing this.

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3035

Fixed, thanks.

[nhaehnle]

Ping^2

[efriedma]

Sorry about the delay...

Have you seen https://reviews.llvm.org/D26628 ? Does that interact with the patch in any way?

TargetLowering::expandMUL_LOHI is kind of difficult to review with all the different unrelated changes involved; would it be possible to separate out the non-functional changes?

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3330

Maybe add an assertion here that HalfType is a legal type? Some architectures could trip over this...

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3203

Should there be an assertion here? If expandMUL_LOHI isn't returning exactly two results, something went wrong.

[nhaehnle]

Thank you for taking a look.

D26628 does not directly interact, since it affected a custom sequence used
by DAGTypeLegalizer if the call to TargetLowering has failed. I suppose now
that TargetLowering no longer insists on only using legal instructions one
could think about using that mode instead of the custom sequence in
DAGTypeLegalizer, but that sequence is also qualitatively different (using a
regular NxN-bit multiply on values where everything but the low N/2 bits
have been masked out).

I've addressed your minor comments and split this up into three patches,
the first two being D27063 and D27064.

I'm going to add them as dependencies as well. Unfortunately, handling
patch series with Phabricator seems really awkward :(

[nhaehnle]

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3330

I missed this one, sorry.

[nhaehnle]

Add assertion that HalfType is legal.

[nhaehnle]

Ping.

[efriedma]

include/llvm/Target/TargetLowering.h

3031

Please make OnlyLegalOrCustom an enum; otherwise it's completely impossible to tell what the parameter does without looking at the header.

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2197

Can you replace this code with a call to expandMUL?

[nhaehnle]

• Use an enum instead of OnlyLegalOrCustom
• Handle a corner case where shift amounts are too large for the native shift type

[nhaehnle]

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2197

I played around with that, but I'd rather not do it in this patch. The generated code sequences are different, and in particular we'd generate non-legal instructions that at least the X86 target doesn't expect and doesn't handle properly. This is visible in test/CodeGen/X86/imul-256/512/1024.ll.

[efriedma]

LGTM with minor comments addressed.

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2197

Okay, that's fine.

lib/CodeGen/SelectionDAG/TargetLowering.cpp

3079

MaskedValueIsZero should do the right thing for vectors, I think, with recent changes to computeKnownBits.

3113

Hmm...

I was going to say that ShiftAmount always fits into ShiftAmountTy because it's a defined shift (ShiftAmount < OuterBitSize < maximum unsigned value), but getShiftAmountTy doesn't always return a sensible result for illegal types. Please just note this problem with a FIXME.

[chapuni]

I pruned three of \param(s) in r289057. Could you recheck them?

llvm/trunk/include/llvm/Target/TargetLowering.h

3051 ↗

(On Diff #80754)

HalfVT and OnlyLegalOrCustom cannot be seen here.

3067 ↗

(On Diff #80754)

ditto.

include/llvm/Target/TargetLowering.h

enum class AtomicExpansionKind {
     None,    // Don't expand the instruction.
     LLSC,    // Expand the instruction into loadlinked/storeconditional; used
              // by ARM/AArch64.
     LLOnly,  // Expand the (load) instruction into just a load-linked, which has
              // greater atomic guarantees than a normal load.
     CmpXChg, // Expand the instruction into cmpxchg; used by at least X86.
   };
 
+  /// Enum that specifies what [US]MUL_LOHI is expanded to.
+  enum class MulExpansion {
+    None,      // Do not actually expand anything
+    Split,     // Split into MUL + MULH[US]
+    Scalarize, // Scalarize vector operations
+    HalfWidth, // Sequence of half-width multiplications and additions
+  };
+
   static ISD::NodeType getExtendForContent(BooleanContent Content) {
     switch (Content) {
     case UndefinedBooleanContent:
       // Extend by adding rubbish bits.
       return ISD::ANY_EXTEND;
     case ZeroOrOneBooleanContent:
       // Extend by adding zero bits.
       return ISD::ZERO_EXTEND;
public:
   bool isSlowDivBypassed() const { return !BypassSlowDivWidths.empty(); }
 
   /// Returns map of slow types for division or remainder with corresponding
   /// fast types
   const DenseMap<unsigned int, unsigned int> &getBypassSlowDivWidths() const {
     return BypassSlowDivWidths;
   }
 
+  /// Returns which of a number of possible expansions should be used for
+  /// [US]MUL_LOHI instructions. This is only used when the corresponding
+  /// operation actions is set to Expand.
+  virtual MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const {
+    return MulExpansion::Split;
+  }
+
   /// Return true if Flow Control is an expensive operation that should be
   /// avoided.
   bool isJumpExpensive() const { return JumpIsExpensive; }
 
   /// Return true if selects are only cheaper than branches if the branch is
   /// unlikely to be predicted right.
   bool isPredictableSelectExpensive() const {
     return PredictableSelectIsExpensive;
virtual SDValue getRecipEstimate(SDValue Operand, DAGCombinerInfo &DCI,
                                    unsigned &RefinementSteps) const {
     return SDValue();
   }
 
   //===--------------------------------------------------------------------===//
   // Legalization utility functions
   //
 
+  /// Expand a MUL or [US]MUL_LOHI of n-bit values into two or four nodes,
+  /// respectively, each computing an n/2-bit part of the result.
+  /// \param Result A vector that will be filled with the parts of the result
+  ///        in little-endian order.
+  /// \param HalfVT The value type to use for the result nodes.
+  /// \param OnlyLegalOrCustom Only legal or custom instructions are used.
+  /// \param LL Low bits of the LHS of the MUL.  You can use this parameter
+  ///        if you want to control how low bits are extracted from the LHS.
+  /// \param LH High bits of the LHS of the MUL.  See LL for meaning.
+  /// \param RL Low bits of the RHS of the MUL.  See LL for meaning
+  /// \param RH High bits of the RHS of the MUL.  See LL for meaning.
+  /// \returns true if the node has been expanded, false if it has not
+  bool expandMUL_LOHI(unsigned Opcode, EVT VT, SDLoc dl, SDValue LHS,
+                      SDValue RHS, SmallVectorImpl<SDValue> &Result, EVT HalfVT,
+                      SelectionDAG &DAG, bool OnlyLegalOrCustom,
+                      SDValue LL = SDValue(), SDValue LH = SDValue(),
+                      SDValue RL = SDValue(), SDValue RH = SDValue()) const;
+
   /// Expand a MUL into two nodes.  One that computes the high bits of
   /// the result and one that computes the low bits.
   /// \param HiLoVT The value type to use for the Lo and Hi nodes.
+  /// \param OnlyLegalOrCustom Only legal or custom instructions are used.
   /// \param LL Low bits of the LHS of the MUL.  You can use this parameter
   ///        if you want to control how low bits are extracted from the LHS.
   /// \param LH High bits of the LHS of the MUL.  See LL for meaning.
   /// \param RL Low bits of the RHS of the MUL.  See LL for meaning
   /// \param RH High bits of the RHS of the MUL.  See LL for meaning.
   /// \returns true if the node has been expanded. false if it has not
   bool expandMUL(SDNode *N, SDValue &Lo, SDValue &Hi, EVT HiLoVT,
-                 SelectionDAG &DAG, SDValue LL = SDValue(),
-                 SDValue LH = SDValue(), SDValue RL = SDValue(),
-                 SDValue RH = SDValue()) const;
+                 SelectionDAG &DAG, bool OnlyLegalOrCustom,
+                 SDValue LL = SDValue(), SDValue LH = SDValue(),
+                 SDValue RL = SDValue(), SDValue RH = SDValue()) const;
 
   /// Expand float(f32) to SINT(i64) conversion
   /// \param N Node to expand
   /// \param Result output after conversion
   /// \returns True, if the expansion was successful, false otherwise
   bool expandFP_TO_SINT(SDNode *N, SDValue &Result, SelectionDAG &DAG) const;
 
   /// Turn load of vector type into a load of the individual elements.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

if (TLI.isOperationLegalOrCustom(DivRemOpc, VT)) {
       Tmp1 = DAG.getNode(DivRemOpc, dl, VTs, Node->getOperand(0),
                          Node->getOperand(1));
       Results.push_back(Tmp1);
     }
     break;
   }
   case ISD::MULHU:
   case ISD::MULHS: {
-    unsigned ExpandOpcode = Node->getOpcode() == ISD::MULHU ? ISD::UMUL_LOHI :
-                                                              ISD::SMUL_LOHI;
+    unsigned ExpandOpcode =
+        Node->getOpcode() == ISD::MULHU ? ISD::UMUL_LOHI : ISD::SMUL_LOHI;
     EVT VT = Node->getValueType(0);
     SDVTList VTs = DAG.getVTList(VT, VT);
-    assert(TLI.isOperationLegalOrCustom(ExpandOpcode, VT) &&
-           "If this wasn't legal, it shouldn't have been created!");
+
     Tmp1 = DAG.getNode(ExpandOpcode, dl, VTs, Node->getOperand(0),
                        Node->getOperand(1));
     Results.push_back(Tmp1.getValue(1));
     break;
   }
+  case ISD::UMUL_LOHI:
+  case ISD::SMUL_LOHI: {
+    SDValue LHS = Node->getOperand(0);
+    SDValue RHS = Node->getOperand(1);
+    MVT VT = LHS.getSimpleValueType();
+    TargetLowering::MulExpansion Kind =
+        TLI.getMulExpansion(Node->getOpcode(), VT);
+
+    switch (Kind) {
+    case TargetLowering::MulExpansion::None:
+      break;
+
+    case TargetLowering::MulExpansion::Split:
+      Results.push_back(DAG.getNode(ISD::MUL, dl, VT, LHS, RHS));
+      Results.push_back(DAG.getNode(
+          Node->getOpcode() == ISD::UMUL_LOHI ? ISD::MULHU : ISD::MULHS, dl, VT,
+          LHS, RHS));
+      break;
+
+    case TargetLowering::MulExpansion::Scalarize: {
+      MVT ScalarVT = VT.getScalarType();
+      unsigned NumElements = VT.getVectorNumElements();
+      MVT IdxTy = TLI.getVectorIdxTy(DAG.getDataLayout());
+      SmallVector<SDValue, 16> Scalars;
+
+      for (unsigned i = 0; i < NumElements; ++i) {
+        SDValue Idx = DAG.getConstant(i, dl, IdxTy);
+        Scalars.push_back(DAG.getNode(
+            Node->getOpcode(), dl, DAG.getVTList(ScalarVT, ScalarVT),
+            DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, ScalarVT, LHS, Idx),
+            DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, ScalarVT, RHS, Idx)));
+      }
+
+      for (unsigned i = 0; i < 2; ++i) {
+        SmallVector<SDValue, 16> Elements;
+        for (unsigned j = 0; j < NumElements; ++j)
+          Elements.push_back(Scalars[j].getValue(i));
+        Results.push_back(DAG.getNode(ISD::BUILD_VECTOR, dl, VT, Elements));
+      }
+      break;
+    }
+
+    case TargetLowering::MulExpansion::HalfWidth: {
+      SmallVector<SDValue, 4> Halves;
+      EVT HalfType;
+      if (!VT.isVector()) {
+        HalfType = EVT(VT).getHalfSizedIntegerVT(*DAG.getContext());
+      } else {
+        HalfType = EVT::getVectorVT(
+            *DAG.getContext(),
+            EVT(VT.getScalarType()).getHalfSizedIntegerVT(*DAG.getContext()),
+            VT.getVectorNumElements());
+      }
+      if (TLI.expandMUL_LOHI(Node->getOpcode(), VT, Node, LHS, RHS, Halves,
+                             HalfType, DAG, false)) {
+        for (unsigned i = 0; i < 2; ++i) {
+          SDValue Lo = DAG.getNode(ISD::ZERO_EXTEND, dl, VT, Halves[2 * i]);
+          SDValue Hi = DAG.getNode(ISD::ANY_EXTEND, dl, VT, Halves[2 * i + 1]);
+          SDValue Shift = DAG.getConstant(
+              HalfType.getSizeInBits(), dl,
+              TLI.getShiftAmountTy(HalfType, DAG.getDataLayout()));
+          Hi = DAG.getNode(ISD::SHL, dl, VT, Hi, Shift);
+          Results.push_back(DAG.getNode(ISD::OR, dl, VT, Lo, Hi));
+        }
+        break;
+      }
+      break;
+    }
+    }
+    break;
+  }
   case ISD::MUL: {
     EVT VT = Node->getValueType(0);
     SDVTList VTs = DAG.getVTList(VT, VT);
     // See if multiply or divide can be lowered using two-result operations.
     // We just need the low half of the multiply; try both the signed
     // and unsigned forms. If the target supports both SMUL_LOHI and
     // UMUL_LOHI, form a preference by checking which forms of plain
     // MULH it supports.
case ISD::MUL: {
     }
 
     SDValue Lo, Hi;
     EVT HalfType = VT.getHalfSizedIntegerVT(*DAG.getContext());
     if (TLI.isOperationLegalOrCustom(ISD::ZERO_EXTEND, VT) &&
         TLI.isOperationLegalOrCustom(ISD::ANY_EXTEND, VT) &&
         TLI.isOperationLegalOrCustom(ISD::SHL, VT) &&
         TLI.isOperationLegalOrCustom(ISD::OR, VT) &&
-        TLI.expandMUL(Node, Lo, Hi, HalfType, DAG)) {
+        TLI.expandMUL(Node, Lo, Hi, HalfType, DAG, true)) {
       Lo = DAG.getNode(ISD::ZERO_EXTEND, dl, VT, Lo);
       Hi = DAG.getNode(ISD::ANY_EXTEND, dl, VT, Hi);
       SDValue Shift =
           DAG.getConstant(HalfType.getSizeInBits(), dl,
                           TLI.getShiftAmountTy(HalfType, DAG.getDataLayout()));
       Hi = DAG.getNode(ISD::SHL, dl, VT, Hi, Shift);
       Results.push_back(DAG.getNode(ISD::OR, dl, VT, Lo, Hi));
     }
case ISD::XOR: {
     // Promote each of the values to the new type.
     Tmp1 = DAG.getNode(ExtOp, dl, NVT, Node->getOperand(0));
     Tmp2 = DAG.getNode(ExtOp, dl, NVT, Node->getOperand(1));
     // Perform the larger operation, then convert back
     Tmp1 = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2);
     Results.push_back(DAG.getNode(TruncOp, dl, OVT, Tmp1));
     break;
   }
+  case ISD::UMUL_LOHI:
+  case ISD::SMUL_LOHI: {
+    // Promote to a multiply in a wider integer type.
+    unsigned ExtOp = Node->getOpcode() == ISD::UMUL_LOHI ? ISD::ZERO_EXTEND
+                                                         : ISD::SIGN_EXTEND;
+    Tmp1 = DAG.getNode(ExtOp, dl, NVT, Node->getOperand(0));
+    Tmp2 = DAG.getNode(ExtOp, dl, NVT, Node->getOperand(1));
+    Tmp1 = DAG.getNode(ISD::MUL, dl, NVT, Tmp1, Tmp2);
+
+    auto &DL = DAG.getDataLayout();
+    unsigned OriginalSize = OVT.getScalarSizeInBits();
+    Tmp2 = DAG.getNode(
+        ISD::SRL, dl, NVT, Tmp1,
+        DAG.getConstant(OriginalSize, dl, TLI.getScalarShiftAmountTy(DL, NVT)));
+    Results.push_back(DAG.getNode(ISD::TRUNCATE, dl, OVT, Tmp1));
+    Results.push_back(DAG.getNode(ISD::TRUNCATE, dl, OVT, Tmp2));
+    break;
+  }
   case ISD::SELECT: {
     unsigned ExtOp, TruncOp;
     if (Node->getValueType(0).isVector() ||
         Node->getValueType(0).getSizeInBits() == NVT.getSizeInBits()) {
       ExtOp   = ISD::BITCAST;
       TruncOp = ISD::BITCAST;
     } else if (Node->getValueType(0).isInteger()) {
       ExtOp   = ISD::ANY_EXTEND;

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

void DAGTypeLegalizer::ExpandIntRes_MUL(SDNode *N,
   EVT VT = N->getValueType(0);
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
   SDLoc dl(N);
 
   SDValue LL, LH, RL, RH;
   GetExpandedInteger(N->getOperand(0), LL, LH);
   GetExpandedInteger(N->getOperand(1), RL, RH);
 
-  if (TLI.expandMUL(N, Lo, Hi, NVT, DAG, LL, LH, RL, RH))
+  if (TLI.expandMUL(N, Lo, Hi, NVT, DAG, true, LL, LH, RL, RH))
     return;
 
   // If nothing else, we can make a libcall.
   RTLIB::Libcall LC = RTLIB::UNKNOWN_LIBCALL;
   if (VT == MVT::i16)
     LC = RTLIB::MUL_I16;
   else if (VT == MVT::i32)
     LC = RTLIB::MUL_I32;
   else if (VT == MVT::i64)
     LC = RTLIB::MUL_I64;
   else if (VT == MVT::i128)
     LC = RTLIB::MUL_I128;
 
   if (LC == RTLIB::UNKNOWN_LIBCALL) {
     // We'll expand the multiplication by brute force because we have no other
     // options. This is a trivially-generalized version of the code from
     // Hacker's Delight (itself derived from Knuth's Algorithm M from section
     // 4.3.1).
     unsigned Bits = NVT.getSizeInBits();
     unsigned HalfBits = Bits >> 1;
     SDValue Mask = DAG.getConstant(APInt::getLowBitsSet(Bits, HalfBits), dl,
                                    NVT);
     SDValue LLL = DAG.getNode(ISD::AND, dl, NVT, LL, Mask);
     SDValue RLL = DAG.getNode(ISD::AND, dl, NVT, RL, Mask);
 
     SDValue T = DAG.getNode(ISD::MUL, dl, NVT, LLL, RLL);

lib/CodeGen/SelectionDAG/TargetLowering.cpp

verifyReturnAddressArgumentIsConstant(SDValue Op, SelectionDAG &DAG) const {
 
   return false;
 }
 
 //===----------------------------------------------------------------------===//
 // Legalization Utilities
 //===----------------------------------------------------------------------===//
 
-bool TargetLowering::expandMUL(SDNode *N, SDValue &Lo, SDValue &Hi, EVT HiLoVT,
-                               SelectionDAG &DAG, SDValue LL, SDValue LH,
-                               SDValue RL, SDValue RH) const {
-  EVT VT = N->getValueType(0);
-  SDLoc dl(N);
-
-  bool HasMULHS = isOperationLegalOrCustom(ISD::MULHS, HiLoVT);
-  bool HasMULHU = isOperationLegalOrCustom(ISD::MULHU, HiLoVT);
-  bool HasSMUL_LOHI = isOperationLegalOrCustom(ISD::SMUL_LOHI, HiLoVT);
-  bool HasUMUL_LOHI = isOperationLegalOrCustom(ISD::UMUL_LOHI, HiLoVT);
-  if (HasMULHU || HasMULHS || HasUMUL_LOHI || HasSMUL_LOHI) {
-    unsigned OuterBitSize = VT.getSizeInBits();
-    unsigned InnerBitSize = HiLoVT.getSizeInBits();
-    unsigned LHSSB = DAG.ComputeNumSignBits(N->getOperand(0));
-    unsigned RHSSB = DAG.ComputeNumSignBits(N->getOperand(1));
+bool TargetLowering::expandMUL_LOHI(unsigned Opcode, EVT VT, SDLoc dl,
+                                    SDValue LHS, SDValue RHS,
+                                    SmallVectorImpl<SDValue> &Result,
+                                    EVT HalfVT, SelectionDAG &DAG,
+                                    bool OnlyLegalOrCustom, SDValue LL,
+                                    SDValue LH, SDValue RL, SDValue RH) const {
+  assert(Opcode == ISD::MUL || Opcode == ISD::UMUL_LOHI ||
+         Opcode == ISD::SMUL_LOHI);
+
+  bool HasMULHS =
+      !OnlyLegalOrCustom || isOperationLegalOrCustom(ISD::MULHS, HalfVT);
+  bool HasMULHU =
+      !OnlyLegalOrCustom || isOperationLegalOrCustom(ISD::MULHU, HalfVT);
+  bool HasSMUL_LOHI =
+      !OnlyLegalOrCustom || isOperationLegalOrCustom(ISD::SMUL_LOHI, HalfVT);
+  bool HasUMUL_LOHI =
+      !OnlyLegalOrCustom || isOperationLegalOrCustom(ISD::UMUL_LOHI, HalfVT);
+  unsigned InnerBitSize = HalfVT.getSizeInBits();
+  unsigned LHSSB = DAG.ComputeNumSignBits(LHS);
+  unsigned RHSSB = DAG.ComputeNumSignBits(RHS);
 
-    // LL, LH, RL, and RH must be either all NULL or all set to a value.
-    assert((LL.getNode() && LH.getNode() && RL.getNode() && RH.getNode()) ||
-           (!LL.getNode() && !LH.getNode() && !RL.getNode() && !RH.getNode()));
 
-    if (!LL.getNode() && !RL.getNode() &&
-        isOperationLegalOrCustom(ISD::TRUNCATE, HiLoVT)) {
-      LL = DAG.getNode(ISD::TRUNCATE, dl, HiLoVT, N->getOperand(0));
-      RL = DAG.getNode(ISD::TRUNCATE, dl, HiLoVT, N->getOperand(1));
-    }
-
-    if (!LL.getNode())
-      return false;
+  if (!HasMULHS && !HasMULHU && !HasSMUL_LOHI && !HasUMUL_LOHI)
 
-    APInt HighMask = APInt::getHighBitsSet(OuterBitSize, InnerBitSize);
-    if (DAG.MaskedValueIsZero(N->getOperand(0), HighMask) &&
-        DAG.MaskedValueIsZero(N->getOperand(1), HighMask)) {
-      // The inputs are both zero-extended.
-      if (HasUMUL_LOHI) {
-        // We can emit a umul_lohi.
-        Lo = DAG.getNode(ISD::UMUL_LOHI, dl, DAG.getVTList(HiLoVT, HiLoVT), LL,
-                         RL);
-        Hi = SDValue(Lo.getNode(), 1);
-        return true;
-      }
-      if (HasMULHU) {
-        // We can emit a mulhu+mul.
-        Lo = DAG.getNode(ISD::MUL, dl, HiLoVT, LL, RL);
-        Hi = DAG.getNode(ISD::MULHU, dl, HiLoVT, LL, RL);
-        return true;
-      }
-    }
-    if (LHSSB > InnerBitSize && RHSSB > InnerBitSize) {
-      // The input values are both sign-extended.
-      if (HasSMUL_LOHI) {
-        // We can emit a smul_lohi.
-        Lo = DAG.getNode(ISD::SMUL_LOHI, dl, DAG.getVTList(HiLoVT, HiLoVT), LL,
-                         RL);
-        Hi = SDValue(Lo.getNode(), 1);
-        return true;
-      }
-      if (HasMULHS) {
-        // We can emit a mulhs+mul.
-        Lo = DAG.getNode(ISD::MUL, dl, HiLoVT, LL, RL);
-        Hi = DAG.getNode(ISD::MULHS, dl, HiLoVT, LL, RL);
-        return true;
-      }
-    }
+  SDVTList VTs = DAG.getVTList(HalfVT, HalfVT);
+  auto MakeUMUL_LOHI = [&](SDValue L, SDValue R, SDValue &Lo,
+                           SDValue &Hi) -> bool {
+      Lo = DAG.getNode(ISD::UMUL_LOHI, dl, VTs, L, R);
+      Lo = DAG.getNode(ISD::MUL, dl, HalfVT, L, R);
+      Hi = DAG.getNode(ISD::MULHU, dl, HalfVT, L, R);
+    return false;
+  };
+  auto MakeSMUL_LOHI = [&](SDValue L, SDValue R, SDValue &Lo,
+                           SDValue &Hi) -> bool {
+      Lo = DAG.getNode(ISD::SMUL_LOHI, dl, VTs, L, R);
+      Lo = DAG.getNode(ISD::MUL, dl, HalfVT, L, R);
+      Hi = DAG.getNode(ISD::MULHS, dl, HalfVT, L, R);
+      return true;
+    }
+    return false;
+  };
+
+  SDValue Lo, Hi;
+
+  if (!LL.getNode() && !RL.getNode() &&
+      isOperationLegalOrCustom(ISD::TRUNCATE, HalfVT)) {
+    LL = DAG.getNode(ISD::TRUNCATE, dl, HalfVT, LHS);
+    RL = DAG.getNode(ISD::TRUNCATE, dl, HalfVT, RHS);
+  }
+
+  if (!LL.getNode())
+    return false;
+
+  APInt HighMask = APInt::getHighBitsSet(OuterBitSize, InnerBitSize);
+  if (DAG.MaskedValueIsZero(LHS, HighMask) &&
+      DAG.MaskedValueIsZero(RHS, HighMask)) {
+    // The inputs are both zero-extended.
+    if (MakeUMUL_LOHI(LL, RL, Lo, Hi)) {
+      Result.push_back(Lo);
+      Result.push_back(Hi);
+      if (Opcode != ISD::MUL) {
+        SDValue Zero = DAG.getConstant(0, dl, HalfVT);
+        Result.push_back(Zero);
+        Result.push_back(Zero);
+      }
+      return true;
+    }
+  }
+
+  if (LHSSB > InnerBitSize && RHSSB > InnerBitSize && Opcode == ISD::MUL) {
+    // The input values are both sign-extended.
+    // TODO non-MUL case?
+    if (MakeSMUL_LOHI(LL, RL, Lo, Hi)) {
+      Result.push_back(Lo);
+      Result.push_back(Hi);
 
-    if (!LH.getNode() && !RH.getNode() &&
-        isOperationLegalOrCustom(ISD::SRL, VT) &&
-        isOperationLegalOrCustom(ISD::TRUNCATE, HiLoVT)) {
-      auto &DL = DAG.getDataLayout();
-      unsigned ShiftAmt = VT.getSizeInBits() - HiLoVT.getSizeInBits();
-      SDValue Shift = DAG.getConstant(ShiftAmt, dl, getShiftAmountTy(VT, DL));
-      LH = DAG.getNode(ISD::SRL, dl, VT, N->getOperand(0), Shift);
-      LH = DAG.getNode(ISD::TRUNCATE, dl, HiLoVT, LH);
-      RH = DAG.getNode(ISD::SRL, dl, VT, N->getOperand(1), Shift);
-      RH = DAG.getNode(ISD::TRUNCATE, dl, HiLoVT, RH);
-    }
+      isOperationLegalOrCustom(ISD::TRUNCATE, HalfVT)) {
+    unsigned ShiftAmt = VT.getSizeInBits() - HalfVT.getSizeInBits();
+    LH = DAG.getNode(ISD::SRL, dl, VT, LHS, Shift);
+    LH = DAG.getNode(ISD::TRUNCATE, dl, HalfVT, LH);
+    RH = DAG.getNode(ISD::SRL, dl, VT, RHS, Shift);
+    RH = DAG.getNode(ISD::TRUNCATE, dl, HalfVT, RH);
 
-    if (!LH.getNode())
-      return false;
 
-    if (HasUMUL_LOHI) {
-      // Lo,Hi = umul LHS, RHS.
-      SDValue UMulLOHI = DAG.getNode(ISD::UMUL_LOHI, dl,
-                                     DAG.getVTList(HiLoVT, HiLoVT), LL, RL);
-      Lo = UMulLOHI;
-      Hi = UMulLOHI.getValue(1);
-      RH = DAG.getNode(ISD::MUL, dl, HiLoVT, LL, RH);
-      LH = DAG.getNode(ISD::MUL, dl, HiLoVT, LH, RL);
-      Hi = DAG.getNode(ISD::ADD, dl, HiLoVT, Hi, RH);
-      Hi = DAG.getNode(ISD::ADD, dl, HiLoVT, Hi, LH);
-      return true;
-    }
-    if (HasMULHU) {
-      Lo = DAG.getNode(ISD::MUL, dl, HiLoVT, LL, RL);
-      Hi = DAG.getNode(ISD::MULHU, dl, HiLoVT, LL, RL);
-      RH = DAG.getNode(ISD::MUL, dl, HiLoVT, LL, RH);
-      LH = DAG.getNode(ISD::MUL, dl, HiLoVT, LH, RL);
-      Hi = DAG.getNode(ISD::ADD, dl, HiLoVT, Hi, RH);
-      Hi = DAG.getNode(ISD::ADD, dl, HiLoVT, Hi, LH);
-      return true;
-    }
+  if (!MakeUMUL_LOHI(LL, RL, Lo, Hi))
+    return false;
+
+  Result.push_back(Lo);
+
+  if (Opcode == ISD::MUL) {
+    RH = DAG.getNode(ISD::MUL, dl, HalfVT, LL, RH);
+    LH = DAG.getNode(ISD::MUL, dl, HalfVT, LH, RL);
+    Hi = DAG.getNode(ISD::ADD, dl, HalfVT, Hi, RH);
+    Hi = DAG.getNode(ISD::ADD, dl, HalfVT, Hi, LH);
+    Result.push_back(Hi);
+
+  SDValue Next = Hi;
+  if (!MakeUMUL_LOHI(LL, RH, Lo, Hi))
+    return false;
+
+  SDVTList CarryVTs = DAG.getVTList(HalfVT, MVT::Glue);
+  SDValue SumLo, SumHi;
+  SumHi = Hi;
+  SumLo = DAG.getNode(ISD::ADDC, dl, CarryVTs, Next, Lo);
+
+  if (!MakeUMUL_LOHI(LH, RL, Lo, Hi))
+    return false;
+
+  SumHi = DAG.getNode(ISD::ADDE, dl, CarryVTs, SumHi, Hi, SumLo.getValue(1));
+  SumLo = DAG.getNode(ISD::ADDC, dl, CarryVTs, SumLo, Lo);
+  Result.push_back(SumLo);
+
+  if (!(Opcode == ISD::UMUL_LOHI ? MakeUMUL_LOHI(LH, RH, Lo, Hi)
+                                 : MakeSMUL_LOHI(LH, RH, Lo, Hi)))
+    return false;
+
+  SDValue Zero = DAG.getConstant(0, dl, HalfVT);
+  SumLo = DAG.getNode(ISD::ADDE, dl, CarryVTs, SumHi, Lo, SumLo.getValue(1));
+  SumHi = DAG.getNode(ISD::ADDE, dl, CarryVTs, Hi, Zero, SumHi.getValue(1));
+  SumHi = DAG.getNode(ISD::ADDE, dl, CarryVTs, SumHi, Zero, SumLo.getValue(1));
+
+  if (Opcode == ISD::SMUL_LOHI) {
+    SDValue LoSub = DAG.getNode(ISD::SUBC, dl, CarryVTs, SumLo, RL);
+    SDValue HiSub =
+        DAG.getNode(ISD::SUBE, dl, CarryVTs, SumHi, Zero, LoSub.getValue(1));
+
+    SumLo = DAG.getSelectCC(dl, LH, Zero, LoSub, SumLo, ISD::SETLT);
+    SumHi = DAG.getSelectCC(dl, LH, Zero, HiSub, SumHi, ISD::SETLT);
+
+    LoSub = DAG.getNode(ISD::SUBC, dl, CarryVTs, SumLo, LL);
+    HiSub =
+        DAG.getNode(ISD::SUBE, dl, CarryVTs, SumHi, Zero, LoSub.getValue(1));
+
+    SumLo = DAG.getSelectCC(dl, RH, Zero, LoSub, SumLo, ISD::SETLT);
+    SumHi = DAG.getSelectCC(dl, RH, Zero, HiSub, SumHi, ISD::SETLT);
+  }
+
+  Result.push_back(SumLo);
+  Result.push_back(SumHi);
+
+bool TargetLowering::expandMUL(SDNode *N, SDValue &Lo, SDValue &Hi, EVT HiLoVT,
+                               SelectionDAG &DAG, bool OnlyLegalOrCustom,
+                               SDValue LL, SDValue LH, SDValue RL,
+                               SDValue RH) const {
+  SmallVector<SDValue, 2> Result;
+  bool Ok = expandMUL_LOHI(N->getOpcode(), N->getValueType(0), N,
+                           N->getOperand(0), N->getOperand(1), Result, HiLoVT,
+                           DAG, OnlyLegalOrCustom, LL, LH, RL, RH);
+  if (Result.size() >= 2) {
+    Lo = Result[0];
+    Hi = Result[1];
   }
-  return false;
+  return Ok;
 }
 
 bool TargetLowering::expandFP_TO_SINT(SDNode *Node, SDValue &Result,
                                SelectionDAG &DAG) const {
   EVT VT = Node->getOperand(0).getValueType();
   EVT NVT = Node->getValueType(0);
   SDLoc dl(SDValue(Node, 0));
 
   // FIXME: Only f32 to i64 conversions are supported.
   if (VT != MVT::f32 || NVT != MVT::i64)

lib/Target/AMDGPU/AMDGPUISelLowering.h

public:
   bool isTruncateFree(Type *Src, Type *Dest) const override;
 
   bool isZExtFree(Type *Src, Type *Dest) const override;
   bool isZExtFree(EVT Src, EVT Dest) const override;
   bool isZExtFree(SDValue Val, EVT VT2) const override;
 
   bool isNarrowingProfitable(EVT VT1, EVT VT2) const override;
 
+  MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const override;
+
   MVT getVectorIdxTy(const DataLayout &) const override;
   bool isSelectSupported(SelectSupportKind) const override;
 
   bool isFPImmLegal(const APFloat &Imm, EVT VT) const override;
   bool ShouldShrinkFPConstant(EVT VT) const override;
   bool shouldReduceLoadWidth(SDNode *Load,
                              ISD::LoadExtType ExtType,
                              EVT ExtVT) const override;

lib/Target/AMDGPU/AMDGPUISelLowering.cpp

bool AMDGPUTargetLowering::isNarrowingProfitable(EVT SrcVT, EVT DestVT) const {
   // limited number of native 64-bit operations. Shrinking an operation to fit
   // in a single 32-bit register should always be helpful. As currently used,
   // this is much less general than the name suggests, and is only used in
   // places trying to reduce the sizes of loads. Shrinking loads to < 32-bits is
   // not profitable, and may actually be harmful.
   return SrcVT.getSizeInBits() > 32 && DestVT.getSizeInBits() == 32;
 }
 
+TargetLowering::MulExpansion
+AMDGPUTargetLowering::getMulExpansion(unsigned Opcode, MVT VT) const {
+  if (VT.getSizeInBits() > 32)
+    return MulExpansion::HalfWidth;
+  return MulExpansion::Split;
+}
+
 //===---------------------------------------------------------------------===//
 // TargetLowering Callbacks
 //===---------------------------------------------------------------------===//
 
 /// The SelectionDAGBuilder will automatically promote function arguments
 /// with illegal types.  However, this does not work for the AMDGPU targets
 /// since the function arguments are stored in memory as these illegal types.
 /// In order to handle this properly we need to get the original types sizes

lib/Target/BPF/BPFISelLowering.h

public:
 
   // This method returns the name of a target specific DAG node.
   const char *getTargetNodeName(unsigned Opcode) const override;
 
   MachineBasicBlock *
   EmitInstrWithCustomInserter(MachineInstr &MI,
                               MachineBasicBlock *BB) const override;
 
+  MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const override;
+
 private:
   SDValue LowerBR_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT_CC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerGlobalAddress(SDValue Op, SelectionDAG &DAG) const;
 
   // Lower the result values of a call, copying them out of physregs into vregs
   SDValue LowerCallResult(SDValue Chain, SDValue InFlag,
                           CallingConv::ID CallConv, bool IsVarArg,

lib/Target/BPF/BPFISelLowering.cpp

BPFTargetLowering::BPFTargetLowering(const TargetMachine &TM,
   setPrefFunctionAlignment(3);
 
   // inline memcpy() for kernel to see explicit copy
   MaxStoresPerMemset = MaxStoresPerMemsetOptSize = 128;
   MaxStoresPerMemcpy = MaxStoresPerMemcpyOptSize = 128;
   MaxStoresPerMemmove = MaxStoresPerMemmoveOptSize = 128;
 }
 
+TargetLowering::MulExpansion BPFTargetLowering::getMulExpansion(unsigned Opcode,
+                                                                MVT VT) const {
+  return MulExpansion::HalfWidth;
+}
+
 SDValue BPFTargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   switch (Op.getOpcode()) {
   case ISD::BR_CC:
     return LowerBR_CC(Op, DAG);
   case ISD::GlobalAddress:
     return LowerGlobalAddress(Op, DAG);
   case ISD::SELECT_CC:
     return LowerSELECT_CC(Op, DAG);

lib/Target/PowerPC/PPCISelLowering.h

public:
     SDValue expandVSXLoadForLE(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue expandVSXStoreForLE(SDNode *N, DAGCombinerInfo &DCI) const;
 
     SDValue PerformDAGCombine(SDNode *N, DAGCombinerInfo &DCI) const override;
 
     SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                           std::vector<SDNode *> *Created) const override;
 
+    MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const override;
+
     unsigned getRegisterByName(const char* RegName, EVT VT,
                                SelectionDAG &DAG) const override;
 
     void computeKnownBitsForTargetNode(const SDValue Op,
                                        APInt &KnownZero,
                                        APInt &KnownOne,
                                        const SelectionDAG &DAG,
                                        unsigned Depth = 0) const override;

lib/Target/PowerPC/PPCISelLowering.cpp

if (IsNegPow2) {
     Op = DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(0, DL, VT), Op);
     if (Created)
       Created->push_back(Op.getNode());
   }
 
   return Op;
 }
 
+TargetLowering::MulExpansion PPCTargetLowering::getMulExpansion(unsigned Opcode,
+                                                                MVT VT) const {
+  if (VT.isVector())
+    return MulExpansion::Scalarize;
+  return MulExpansion::Split;
+}
+
 //===----------------------------------------------------------------------===//
 // Inline Assembly Support
 //===----------------------------------------------------------------------===//
 
 void PPCTargetLowering::computeKnownBitsForTargetNode(const SDValue Op,
                                                       APInt &KnownZero,
                                                       APInt &KnownOne,
                                                       const SelectionDAG &DAG,

lib/Target/Sparc/SparcISelLowering.h

public:
 
     bool shouldInsertFencesForAtomic(const Instruction *I) const override {
       // FIXME: We insert fences for each atomics and generate
       // sub-optimal code for PSO/TSO. (Approximately nobody uses any
       // mode but TSO, which makes this even more silly)
       return true;
     }
 
+    MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const override;
+
     AtomicExpansionKind shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
 
     void ReplaceNodeResults(SDNode *N,
                             SmallVectorImpl<SDValue>& Results,
                             SelectionDAG &DAG) const override;
 
     MachineBasicBlock *expandSelectCC(MachineInstr &MI, MachineBasicBlock *BB,
                                       unsigned BROpcode) const;
     MachineBasicBlock *emitEHSjLjSetJmp(MachineInstr &MI,
                                         MachineBasicBlock *MBB) const;
     MachineBasicBlock *emitEHSjLjLongJmp(MachineInstr &MI,
                                          MachineBasicBlock *MBB) const;
   };
 } // end namespace llvm
 
 #endif    // SPARC_ISELLOWERING_H

lib/Target/Sparc/SparcISelLowering.cpp

SparcTargetLowering::LowerCall_64(TargetLowering::CallLoweringInfo &CLI,
 
   return Chain;
 }
 
 //===----------------------------------------------------------------------===//
 // TargetLowering Implementation
 //===----------------------------------------------------------------------===//
 
+TargetLowering::MulExpansion
+SparcTargetLowering::getMulExpansion(unsigned Opcode, MVT VT) const {
+  if (VT.getSizeInBits() > 32)
+    return MulExpansion::HalfWidth;
+  return MulExpansion::Split;
+}
+
 TargetLowering::AtomicExpansionKind SparcTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   if (AI->getOperation() == AtomicRMWInst::Xchg &&
       AI->getType()->getPrimitiveSizeInBits() == 32)
     return AtomicExpansionKind::None; // Uses xchg instruction
 
   return AtomicExpansionKind::CmpXChg;
 }
 
SparcTargetLowering::SparcTargetLowering(const TargetMachine &TM,
   setOperationAction(ISD::FPOW , MVT::f128, Expand);
   setOperationAction(ISD::FPOW , MVT::f64, Expand);
   setOperationAction(ISD::FPOW , MVT::f32, Expand);
 
   setOperationAction(ISD::SHL_PARTS, MVT::i32, Expand);
   setOperationAction(ISD::SRA_PARTS, MVT::i32, Expand);
   setOperationAction(ISD::SRL_PARTS, MVT::i32, Expand);
 
+  if (Subtarget->is64Bit()) {
     // FIXME: Sparc provides these multiplies, but we don't have them yet.
-  setOperationAction(ISD::UMUL_LOHI, MVT::i32, Expand);
-  setOperationAction(ISD::SMUL_LOHI, MVT::i32, Expand);
+    setOperationAction(ISD::UMUL_LOHI, MVT::i32, Promote);
+    setOperationAction(ISD::SMUL_LOHI, MVT::i32, Promote);
 
-  if (Subtarget->is64Bit()) {
     setOperationAction(ISD::UMUL_LOHI, MVT::i64, Expand);
     setOperationAction(ISD::SMUL_LOHI, MVT::i64, Expand);
     setOperationAction(ISD::MULHU,     MVT::i64, Expand);
     setOperationAction(ISD::MULHS,     MVT::i64, Expand);
 
     setOperationAction(ISD::UMULO,     MVT::i64, Custom);
     setOperationAction(ISD::SMULO,     MVT::i64, Custom);
 
     setOperationAction(ISD::SHL_PARTS, MVT::i64, Expand);
     setOperationAction(ISD::SRA_PARTS, MVT::i64, Expand);
     setOperationAction(ISD::SRL_PARTS, MVT::i64, Expand);
+  } else {
+    // FIXME: Sparc provides these multiplies, but we don't have them yet.
+    setOperationAction(ISD::UMUL_LOHI, MVT::i32, Expand);
+    setOperationAction(ISD::SMUL_LOHI, MVT::i32, Expand);
   }
 
   // VASTART needs to be custom lowered to use the VarArgsFrameIndex.
   setOperationAction(ISD::VASTART           , MVT::Other, Custom);
   // VAARG needs to be lowered to not do unaligned accesses for doubles.
   setOperationAction(ISD::VAARG             , MVT::Other, Custom);
 
   setOperationAction(ISD::TRAP              , MVT::Other, Legal);

lib/Target/X86/X86ISelLowering.h

public:
 
     bool isNoopAddrSpaceCast(unsigned SrcAS, unsigned DestAS) const override;
 
     /// \brief Customize the preferred legalization strategy for certain types.
     LegalizeTypeAction getPreferredVectorAction(EVT VT) const override;
 
     bool isIntDivCheap(EVT VT, AttributeSet Attr) const override;
 
+    MulExpansion getMulExpansion(unsigned Opcode, MVT VT) const override;
+
     bool supportSwiftError() const override;
 
   protected:
     std::pair<const TargetRegisterClass *, uint8_t>
     findRepresentativeClass(const TargetRegisterInfo *TRI,
                             MVT VT) const override;
 
   private:

lib/Target/X86/X86ISelLowering.cpp

bool X86TargetLowering::isIntDivCheap(EVT VT, AttributeSet Attr) const {
   // integer division, leaving the division as-is is a loss even in terms of
   // size, because it will have to be scalarized, while the alternative code
   // sequence can be performed in vector form.
   bool OptSize = Attr.hasAttribute(AttributeSet::FunctionIndex,
                                    Attribute::MinSize);
   return OptSize && !VT.isVector();
 }
 
+TargetLowering::MulExpansion X86TargetLowering::getMulExpansion(unsigned Opcode,
+                                                                MVT VT) const {
+  if (VT.getScalarSizeInBits() >= 64)
+    return MulExpansion::Scalarize;
+  return MulExpansion::Split;
+}
+
 void X86TargetLowering::initializeSplitCSR(MachineBasicBlock *Entry) const {
   if (!Subtarget.is64Bit())
     return;
 
   // Update IsSplitCSR in X86MachineFunctionInfo.
   X86MachineFunctionInfo *AFI =
     Entry->getParent()->getInfo<X86MachineFunctionInfo>();
   AFI->setIsSplitCSR(true);