[Intrinsic] Signed Fixed Point Multiplication Intrinsic

[leonardchan]

Summary

Add an intrinsic that takes 2 signed integers with the scale of them provided as the third argument and performs fixed point multiplication on them.

This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics.

[craig.topper]

Need to update LangRef.rst which I think we also missed for the saturating intrinsics.

llvm/include/llvm/CodeGen/ISDOpcodes.h

278

This is says it saturates but I didn't see that implemented in the expansion function.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

625

Get rid of Op1 and Op2 and just do GetPromotedInteger(N->getOperand(0))

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5093

Shift amount constants should get their type from getShiftAmountTy.

5096

This is really an OR isn't it? DAG combiner will turn it into that so might as well just use OR.

5097

No need for an else after a return.

llvm/lib/IR/Verifier.cpp

4531

argumenr->argument

4532

I think you need to check that Op3 is a ConstantInt as well. And that it fits in 32 bits.

[leonardchan]

In D54719#1303553, @craig.topper wrote:

Need to update LangRef.rst which I think we also missed for the saturating intrinsics.

Funny, I actually just uploaded a patch for the LangRef docs on the other intrinsics (https://reviews.llvm.org/D54729). Also added the docs for this intrinsic in this patch.

llvm/include/llvm/CodeGen/ISDOpcodes.h

278

My bad, this intrinsic doesn't perform saturation.

[bjope]

llvm/docs/LangRef.rst

12604 ↗

(On Diff #174703)

I'm not sure if/how the auto-vectorizers are aware that only the first two operands should be vectorized and that the third operand should stay scalar. So I guess it is unlikely that we ever get any vector operands at the moment (except for handwritten lit tests).

So I guess this should work for now (after all it is simpler than having a vector of scales). But perhaps we need to allow the scale to be given as a vector as well in the future to support vectorization (shl is for example similar, but afaik it will get the shift counts as a vector when being vectorized).

llvm/include/llvm/CodeGen/TargetLowering.h

801

Shouldn't this say "scale" instead of "saturation bit width"?

819

Same as above, isn't the check about "scales" and not "saturation widths"?

llvm/include/llvm/IR/Intrinsics.td

738

nit: In the langref you are using "Saturation Arithmetic Intrinsics" and "Fixed Point Arithemetic Instrinsics" as two separate chapters. And here we put all of them into one category "Fixed Point Intrinsics". I'm not sure if the saturating arithmetics should be in a fixed point category.

Anyway, when adding a int_smul_fix_sat later I guess it will be in the "Fixed Point Arithemetic Instrinsics" in the langref (even if it also is saturating). So maybe it is better to not having this split into two categories after all.

[RKSimon]

llvm/test/CodeGen/X86/smul_fix.ll

4

You should be able to drop -mcpu, also please can you use -check-prefix=X64 and -check-prefix=X86 ?

10

Add nounwind to all the tests to reduce stack codegen?

[ebevhan]

llvm/docs/LangRef.rst

12628 ↗

(On Diff #174703)

typo "up"

12629 ↗

(On Diff #174703)

This doc doesn't say what happens on overflow. Truncation/wraparound? Or should we consider it undefined?

If it says that the rounding is dependent on the target, will the rounding mode be target-configurable and exposed through TLI/TTI? We essentially can't touch these intrinsics (constant folding, optimization, even legalization) otherwise.

Looking at the legalization sequence, it will definitely lower these into a round-down form. If a target has some legal operations which round up and some non-legal operations, then the legal ones will round up and the non-legal ones will round down. That's sort of messy.

It might be safer to say that the rounding is indeterminate, but that's even worse for optimization.

llvm/include/llvm/CodeGen/ISDOpcodes.h

278

The intrinsic docs in the LangRef mentions that the last value must be constant, but this comment doesn't.

llvm/include/llvm/CodeGen/TargetLowering.h

3785

This comment clashes with the assertions in the function. It doesn't take vectors.

llvm/include/llvm/Target/TargetSelectionDAG.td

380

It's marked SDTIntBinOp, but is supposed to have three input operands. I think these nodes might need a new SDT.

Also, multiplication is obviously commutative, but I don't know if SDNPCommutative works on nodes that have anything except two operands. It might not have an effect at all. Someone who knows more about DAG might have more info on that.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2545

Maybe I'm missing something here, but isn't this just a normal, expanded multiplication?

2546

If we hit this return, doesn't this mean that legalization failed? Do we want to catch this here?

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5092

Could use a couple comments explaining what we're doing with the values/SRL/SHL.

Does this work if MULHS in VT is of dubious legality?

[craig.topper]

llvm/include/llvm/Target/TargetSelectionDAG.td

380

Tablegen should use the first two operands as the commutative ones. It used to be an error or an assertion. But I changed it sometime in the last year or so to make it work for FMA in X86.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5092

Ideally we'd use MUL_LOHI if the target supports it. That should allow X86 to use a single multiply instruction in the test cases.

[leonardchan]

llvm/docs/LangRef.rst

12629 ↗

(On Diff #174703)

Overflow I was going to leave as undefined behavior. Added this into the docs.

By target dependent, I meant that whatever legal operation a target could replace this intrinsic with has the choice of rounding up or down. I'm not sure how bad this would be for optimization, but I imagine rounding isn't something that needs to be touched immediately since with this intrinsic, I'm just attempting to match what the standard says. I think one of the conclusions we also came up to in the long thread on llvm-dev was that, for now, we don't need to do more than what's necessary to implement the spec.

I changed the docs though to say rounding is indeterminable, as the spec says.

llvm/include/llvm/IR/Intrinsics.td

738

Split into 2 categories. Was also planning on putting the saturating fixed point ones under "Fixed Point Intrinsics".

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2545

This should actually be an expansion of SMUL_LOHI since we need the high bits when scaling down the result after multiplication. Updated the code, but one thing I found was that expandMUL_LOHI depends on ADDC but does not have a check for it. When rerunning the tests, it seems that X86 does not support 32 bit ADDE (and ADDC).

LLVM ERROR: Cannot select: t81: i32,glue = adde t121, t44:1, t80:1

so I removed these from the lit tests, but if we want to support expansion of this, I'm not sure if the solution is to add a check into expandMUL_LOHI to check isOperationLegalOrCustom(ISD::ADDC/E, VT) and if it is not legal, write out the expanded version. I found a comment in ExpandIntRes_ADDSUB() saying that there currently does not appear to be a way of generating a value of MTV::Glue if we were to expand the result.

2546

Added a fatal error report here.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5092

Added checks to see if we can use MULHS or SMUL_LOHI.

[ebevhan]

llvm/docs/LangRef.rst

12629 ↗

(On Diff #174703)

Okay. If we claim that it's indeterminate I guess we can still fold however we like, but the results would seem a bit inconsistent.

12635 ↗

(On Diff #174858)

I think "operation" might be clearer than "conversion". Or don't mention it at all and just say "It is undefined behavior if the source value does not fit within the range of the fixed point type."

The rounding section is also a bit wordy. "If the result value cannot be precisely represented in the given scale, the value is rounded up or down to the closest representable value. The rounding direction is unspecified."

Also, my choice of the word "indeterminate" was a bit unfortunate, I think the rest of the LangRef uses "unspecified".

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2545

It's a bit odd that x86 doesn't support ADDC/ADDE. Those nodes are deprecated in favor of ADDCARRY, so expandMUL_LOHI should probably be making expansions with ADDCARRY instead.

@craig.topper probably knows more about this.

llvm/test/CodeGen/X86/smul_fix.ll

29

Interesting that the 32-bit target produces better code in most cases.

[craig.topper]

I think @eli.friedman might know more about the ADDE/ADDC vs ADDCARRY than I do.

[craig.topper]

Do we have examples of real hardware that implements this sort of instruction?

Right now this intrinsic looks like its just reimplementing what you would get if you just emitted this IR

%a = sext iX %x to i2X
%b = sext iX %y to i2X
%c = mul i2X %a, %b
%d = lshr i2X %c, PRECISION
%e = trunc i2X %d to iX

So I'm not sure if there's a need for an intrinsic unless that pattern is difficult to match to an instruction.

[leonardchan]

In D54719#1305542, @craig.topper wrote:

Do we have examples of real hardware that implements this sort of instruction?

Right now this intrinsic looks like its just reimplementing what you would get if you just emitted this IR

%a = sext iX %x to i2X
%b = sext iX %y to i2X
%c = mul i2X %a, %b
%d = lshr i2X %c, PRECISION
%e = trunc i2X %d to iX

So I'm not sure if there's a need for an intrinsic unless that pattern is difficult to match to an instruction.

I think @ebevhan and @bjope may have hardware with fixed point instructions

[bjope]

In D54719#1305550, @leonardchan wrote:

In D54719#1305542, @craig.topper wrote:

Do we have examples of real hardware that implements this sort of instruction?

Right now this intrinsic looks like its just reimplementing what you would get if you just emitted this IR

%a = sext iX %x to i2X
%b = sext iX %y to i2X
%c = mul i2X %a, %b
%d = lshr i2X %c, PRECISION
%e = trunc i2X %d to iX

So I'm not sure if there's a need for an intrinsic unless that pattern is difficult to match to an instruction.

I think @ebevhan and @bjope may have hardware with fixed point instructions

Yes, our HW support various kinds of fixed point multiplication (such as llvm.smul.fix.i32(i32 %x, i32 %y, i32 31) and llvm.smul.fix.i24(i24 %x, i24 %y, i32 15)). We already have an intrinsic like llvm.smul.fix downstream.

For our downstream target we keep the intrinsic all the way to ISel. When compiling for other targets we have so far used an IR pass that transform the intrinsic to regular IR early in opt. I'm afraid that if we do that kind of expansion also for our target, we would introduce mul/lshr using non-legal types like i64 or i48. If we can't successfully match the IR back to a native instruction we will get really bad performance by doing the multiplication using the non-legal types. I'm also afraid that the pattern matching easily can be broken, for example by an IR pass that sinks the trunc to a later BB.

[efriedma]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2545

Yes, ADDC/ADDE are deprecated, and only work on targets which specifically implement lowering support, so there are a lot of targets where they simply can't be used (either because the target uses ADDCARRY instead, or because the operation simply doesn't exist on the target). expandMUL_LOHI is only used in very few places, though, so I guess we haven't hit this issue before.

There are a few different ways to expand an ADD; see DAGTypeLegalizer::ExpandIntRes_ADDSUB. Ideally, we should just generate ADDCARRY and let legalization take care of the rest, but I don't think ADDCARRY legalization is currently implemented. Probably not hard to implement, though. (Maybe keep the existing ADDC/ADDE code for targets which use them.)

[leonardchan]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2545

I changed expandMUL_LOHI to use ADDCARRY if ADDC and ADDE are not available, and the corresponding expansion in the DAGLegalizer.

[leonardchan]

*ping* Does anyone have any more comments on this patch?

[ebevhan]

LGTM.

[leonardchan]

@craig.topper @RKSimon @efriedma Any more comments on this patch?

[efriedma]

You might need PromoteIntOp_SMULFIX for certain targets, like 64-bit RISCV.

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

"The rounding direction is unspecified" seems scary to me... is there really no standard for rounding these operations?

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

624

Do you need to sign-extend here?

3259

Maybe worth adding a comment that the operand that needs to be expanded must be the third operand, since the other two operands have the same type as the result.

And actually, I'm not sure how you would hit this in practice; it seems unusual to split a boolean value.

[ebevhan]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

The Embedded-C standard says that the rounding direction is implementation-defined, and is allowed to be indeterminable.

I also think it's a bit unfortunate that we can't express the rounding direction, since it inhibits optimization, but locking it down to either rounding up or down will make things annoying for any target with operations that round the other direction.

Saying that it's unspecified lets us properly legalize this to something sensible while allowing targets to implement it with legal operations if they have them.

For what it's worth, the legalization will make it round down (toward negative infinity).

[bjope]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

Maybe it should say "target specific" instead? And then the legalization (and any constant folding etc) could use some target transform hook to decide in which direction to do the rounding (I guess it could be set to up/down/any).

Does "unspecified" mean that ConstantFolding use one strategy for rounding and legalization another strategy? Or should we only allow constant folding when rounding isn't a problem (such as multiply by zero)?

Things could still be annoying for a target that does care about rounding. Or maybe it isn't allowed for a target to decide what the implementation-defined behavior should be? Or should it be the same for all targets?

A first implementation could of course go for only supporting "unspecified" (if it is clear what kind of constant folding and other optimizations that are allowed). If a target needs a "target specific" behavior, then I assume the code could be sprinkled with hooks at a later stage.

[leonardchan]

In D54719#1317563, @efriedma wrote:

You might need PromoteIntOp_SMULFIX for certain targets, like 64-bit RISCV.

Added, although RISCV doesn't seem to support [US]MUL_LOHI, so I couldn't add a test for that.

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

Maybe it should say "target specific" instead? And then the legalization (and any constant folding etc) could use some target transform hook to decide in which direction to do the rounding (I guess it could be set to up/down/any).

I would be ok with adding this later. For now I was focusing on laying down the basic code necessary for just using this intrinsic.

Does "unspecified" mean that ConstantFolding use one strategy for rounding and legalization another strategy? Or should we only allow constant folding when rounding isn't a problem (such as multiply by zero)?

I initially had it as "target specific" but changed it to "unspecified" since the rest of the LangRef doc seems to use this term for describing either implementation-defined, or undefined, behavior.

Things could still be annoying for a target that does care about rounding. Or maybe it isn't allowed for a target to decide what the implementation-defined behavior should be? Or should it be the same for all targets?
A first implementation could of course go for only supporting "unspecified" (if it is clear what kind of constant folding and other optimizations that are allowed). If a target needs a "target specific" behavior, then I assume the code could be sprinkled with hooks at a later stage.

The hooks are a good idea. I figure for optimizations, depending on what's supported (up/down/any), InstCombine and others can use the hook to determine rounding direction.

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

624

Yup

3259

Oh. You're right, this shouldn't have happened in the first place. When expanding to use ADDCARRY in expandMUL_LOHI, I accidentally created the constant with a VT that was the same as the operands instead of an MTV::i1.

Fixed this, and this method is no longer necessary.

[efriedma]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

Would it make sense to make the rounding mode a parameter to the intrinsic? That way, frontends can choose the semantics they want, and the semantics are clear throughout the optimization pipeline. clang could choose the rounding mode in a target-specific manner if necessary.

[leonardchan]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

The only thing that I think would be of concern is just having too many parameters, or having an extra intrinsic to dictate rounding direction. Other than this, I have no other strong opinions regarding this.

I imagine though that having an extra parameter/intrinsic to specify rounding would imply that a particular target has the option of specifying rounding in both directions, when really a target would likely just have native rounding in one direction (although this is just an assumption).

[leonardchan]

*ping* @craig.topper @RKSimon Any more comments on this patch?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2532

Just get N->getConstantOperandVal(2)?

2563

This result is only used by one of the 2 ifs below. Should it be pulled into that if.

Can you add a comment to describe what's happening here?

[leonardchan]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

@ebevhan @bjope Do you think an extra parameter/intrinsic is worth adding also in this patch?

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2556

if Scale is grater than NVTSize then this number is negative.

2571

Why is ResultLL being shifted right twice? I don't think I understand that.

2588

If Scale == NVTSize then the result is exactly in ResultHL and ResultLH, but nothing can prevent at least 1 bit of ResultHH from being used here. The shift amount for the ResultHH shift must be between 0 and NVTSize-1 to not be undefined.

[leonardchan]

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

2571

My bad. It shouldn't have shifted twice.

2588

Done. Added test case for this also.

[ebevhan]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

It's definitely handy to be able to express the rounding of the operations in the intrinsic, but I think some of the downsides are:

• The number of parameters grows, making the intrinsic more unwieldy.
• There are other roundings than just up and down; rounding towards or away from zero (the former of which I think the fixed-point division ought to try and always follow) are also possibilities.
• We need to support legalization of every rounding mode we add to the intrinsic. Then again, if we add a hook we need to do the same anyway.
• The legalization checks need to check both rounding and scale.

And as you say, it's more likely that the rounding of a specific target should be the same across the board rather than be configurable per operation, if it has support for fixed-point.

[ebevhan]

llvm/docs/LangRef.rst

12830 ↗

(On Diff #175795)

Ignore what I said about the division rounding; I was totally off mark there.

I think we can just keep it unspecified for now and if the need arises, add rounding mode with a hook.

[RKSimon]

Use getConstantOperandVal where possible

llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1132

unsigned Scale = Node->getConstantOperandVal(2);

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

407

unsigned Scale = Node->getConstantOperandVal(2);

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5085

unsigned Scale = Node->getConstantOperandVal(2);

[leonardchan]

*ping* Any more comments on this patch?

[craig.topper]

LGTM

[lebedev.ri]

llvm/trunk/docs/LangRef.rst

12820–12829 ↗

(On Diff #177822)

To someone unfamiliar with what fixed-point math is, this is somewhat vague.

Would it please be possible to reword this with some more details?
Overview section in https://llvm.org/docs/LangRef.html#llvm-fshl-intrinsic is a nice example of it would look best.

In particular, it isn't all that obvious how scale interacts with lhs/rhs.
My current guess:

%r = call i4 @llvm.smul.fix.i4(i4 %a, i4 %b, i32 %c)
  =>
%a2 = sext i4 %a to i8
%b2 = sext i4 %b to i8
%mul = mul nsw nuw i8 %a, %b
%c2 = trunc i32 %c to i8
%scale = ashr i8 %mul, i8 %c ; does not convey the randomness of rounding though
%r = trunc i8 %scale to i4

llvm/include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// RESULT = [US]SUBSAT(LHS, RHS) - Perform saturation subtraction on 2
     /// integers with the same bit width (W). If the true value of LHS - RHS
     /// exceeds the largest value that can be represented by W bits, the
     /// resulting value is this maximum value. Otherwise, if this value is less
     /// than the smallest value that can be represented by W bits, the
     /// resulting value is this minimum value.
     SSUBSAT, USUBSAT,
 
+    /// RESULT = SMULFIX(LHS, RHS, SCALE) - Perform fixed point multiplication on
+    /// 2 integers with the same width and scale. SCALE represents the scale of
+    /// both operands as fixed point numbers. A scale of zero is effectively
+    /// performing saturation multiplication on 2 integers.
+    SMULFIX,
+
     /// Simple binary floating point operators.
     FADD, FSUB, FMUL, FDIV, FREM,
 
     /// Constrained versions of the binary floating point operators.
     /// These will be lowered to the simple operators before final selection.
     /// They are used to limit optimizations while the DAG is being
     /// optimized.
     STRICT_FADD, STRICT_FSUB, STRICT_FMUL, STRICT_FDIV, STRICT_FREM,

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   LegalizeAction getOperationAction(unsigned Op, EVT VT) const {
     if (VT.isExtended()) return Expand;
     // If a target-specific SDNode requires legalization, require the target
     // to provide custom legalization for it.
     if (Op >= array_lengthof(OpActions[0])) return Custom;
     return OpActions[(unsigned)VT.getSimpleVT().SimpleTy][Op];
   }
 
+  /// Custom method defined by each target to indicate if an operation which
+  /// may require a saturation bit width is supported natively by the target.
+  /// If not, the operation is illegal.
+  virtual bool isSupportedFixedPointOperation(unsigned Op, EVT VT,
+                                              unsigned Scale) const {
+    return false;
+  }
+
+  /// Some fixed point operations may be natively supported by the target but
+  /// only for specific scales. This method allows for checking
+  /// if the width is supported by the target for a given operation that may
+  /// depend on scale.
+  LegalizeAction getFixedPointOperationAction(unsigned Op, EVT VT,
+                                              unsigned Scale) const {
+    auto Action = getOperationAction(Op, VT);
+    if (Action != Legal)
+      return Action;
+
+    // This operation is supported in this type but may only work on specific
+    // saturation widths.
+    bool Supported;
+    switch (Op) {
+    default:
+      llvm_unreachable("Unexpected fixed point operation.");
+    case ISD::SMULFIX:
+      Supported = isSupportedFixedPointOperation(Op, VT, Scale);
+      break;
+    }
+
+    return Supported ? Action : Expand;
+  }
+
   LegalizeAction getStrictFPOperationAction(unsigned Op, EVT VT) const {
     unsigned EqOpc;
     switch (Op) {
       default: llvm_unreachable("Unexpected FP pseudo-opcode");
       case ISD::STRICT_FADD: EqOpc = ISD::FADD; break;
       case ISD::STRICT_FSUB: EqOpc = ISD::FSUB; break;
       case ISD::STRICT_FMUL: EqOpc = ISD::FMUL; break;
       case ISD::STRICT_FDIV: EqOpc = ISD::FDIV; break;

   SDValue getVectorElementPointer(SelectionDAG &DAG, SDValue VecPtr, EVT VecVT,
                                   SDValue Index) const;
 
   /// Method for building the DAG expansion of ISD::[US][ADD|SUB]SAT. This
   /// method accepts integers or vectors of integers as its arguments.
   SDValue getExpandedSaturationAdditionSubtraction(SDNode *Node,
                                                    SelectionDAG &DAG) const;
 
+  /// Method for building the DAG expansion of ISD::SMULFIX. This method accepts
+  /// integers or vectors of integers as its arguments.
+  SDValue getExpandedFixedPointMultiplication(SDNode *Node,
+                                              SelectionDAG &DAG) const;
+
   //===--------------------------------------------------------------------===//
   // Instruction Emitting Hooks
   //
 
   /// This method should be implemented by targets that mark instructions with
   /// the 'usesCustomInserter' flag.  These instructions are special in various
   /// ways, which require special support to insert.  The specified MachineInstr
   /// is created but not inserted into any basic blocks, and this method is

llvm/include/llvm/IR/Intrinsics.td

 
 def int_smul_with_overflow : Intrinsic<[llvm_anyint_ty, llvm_i1_ty],
                                        [LLVMMatchType<0>, LLVMMatchType<0>],
                                        [IntrNoMem, IntrSpeculatable]>;
 def int_umul_with_overflow : Intrinsic<[llvm_anyint_ty, llvm_i1_ty],
                                        [LLVMMatchType<0>, LLVMMatchType<0>],
                                        [IntrNoMem, IntrSpeculatable]>;
 
 //===------------------------- Fixed Point Intrinsics ---------------------===//
 //
 def int_sadd_sat : Intrinsic<[llvm_anyint_ty],
                              [LLVMMatchType<0>, LLVMMatchType<0>],
                              [IntrNoMem, IntrSpeculatable, Commutative]>;
 def int_uadd_sat : Intrinsic<[llvm_anyint_ty],
                              [LLVMMatchType<0>, LLVMMatchType<0>],
                              [IntrNoMem, IntrSpeculatable, Commutative]>;
 def int_ssub_sat : Intrinsic<[llvm_anyint_ty],
                              [LLVMMatchType<0>, LLVMMatchType<0>],
                              [IntrNoMem, IntrSpeculatable]>;
 def int_usub_sat : Intrinsic<[llvm_anyint_ty],
                              [LLVMMatchType<0>, LLVMMatchType<0>],
                              [IntrNoMem, IntrSpeculatable]>;
+def int_smul_fix : Intrinsic<[llvm_anyint_ty],
+                             [LLVMMatchType<0>, LLVMMatchType<0>, llvm_i32_ty],
+                             [IntrNoMem, IntrSpeculatable, Commutative]>;
 
 //===------------------------- Memory Use Markers -------------------------===//
 //
 def int_lifetime_start  : Intrinsic<[],
                                     [llvm_i64_ty, llvm_anyptr_ty],
                                     [IntrArgMemOnly, NoCapture<1>]>;
 def int_lifetime_end    : Intrinsic<[],
                                     [llvm_i64_ty, llvm_anyptr_ty],

llvm/include/llvm/Target/TargetSelectionDAG.td

def umin       : SDNode<"ISD::UMIN"      , SDTIntBinOp,
                                   [SDNPCommutative, SDNPAssociative]>;
 def umax       : SDNode<"ISD::UMAX"      , SDTIntBinOp,
                                   [SDNPCommutative, SDNPAssociative]>;
 
 def saddsat    : SDNode<"ISD::SADDSAT"   , SDTIntBinOp, [SDNPCommutative]>;
 def uaddsat    : SDNode<"ISD::UADDSAT"   , SDTIntBinOp, [SDNPCommutative]>;
 def ssubsat    : SDNode<"ISD::SSUBSAT"   , SDTIntBinOp>;
 def usubsat    : SDNode<"ISD::USUBSAT"   , SDTIntBinOp>;
+def smulfix    : SDNode<"ISD::SMULFIX"   , SDTIntBinOp, [SDNPCommutative]>;
 
 def sext_inreg : SDNode<"ISD::SIGN_EXTEND_INREG", SDTExtInreg>;
 def sext_invec : SDNode<"ISD::SIGN_EXTEND_VECTOR_INREG", SDTExtInvec>;
 def zext_invec : SDNode<"ISD::ZERO_EXTEND_VECTOR_INREG", SDTExtInvec>;
 
 def abs        : SDNode<"ISD::ABS"        , SDTIntUnaryOp>;
 def bitreverse : SDNode<"ISD::BITREVERSE" , SDTIntUnaryOp>;
 def bswap      : SDNode<"ISD::BSWAP"      , SDTIntUnaryOp>;

llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

case ISD::STRICT_FTRUNC:
     break;
   case ISD::SADDSAT:
   case ISD::UADDSAT:
   case ISD::SSUBSAT:
   case ISD::USUBSAT: {
     Action = TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0));
     break;
   }
+  case ISD::SMULFIX: {
+    unsigned Scale = cast<ConstantSDNode>(Node->getOperand(2))->getZExtValue();
+    Action = TLI.getFixedPointOperationAction(Node->getOpcode(),
+                                              Node->getValueType(0), Scale);
+    break;
+  }
   case ISD::MSCATTER:
     Action = TLI.getOperationAction(Node->getOpcode(),
                     cast<MaskedScatterSDNode>(Node)->getValue().getValueType());
     break;
   case ISD::MSTORE:
     Action = TLI.getOperationAction(Node->getOpcode(),
                     cast<MaskedStoreSDNode>(Node)->getValue().getValueType());
     break;
bool SelectionDAGLegalize::ExpandNode(SDNode *Node) {
   }
   case ISD::SADDSAT:
   case ISD::UADDSAT:
   case ISD::SSUBSAT:
   case ISD::USUBSAT: {
     Results.push_back(TLI.getExpandedSaturationAdditionSubtraction(Node, DAG));
     break;
   }
+  case ISD::SMULFIX: {
+    Results.push_back(TLI.getExpandedFixedPointMultiplication(Node, DAG));
+    break;
+  }
   case ISD::SADDO:
   case ISD::SSUBO: {
     SDValue LHS = Node->getOperand(0);
     SDValue RHS = Node->getOperand(1);
     SDValue Sum = DAG.getNode(Node->getOpcode() == ISD::SADDO ?
                               ISD::ADD : ISD::SUB, dl, LHS.getValueType(),
                               LHS, RHS);
     Results.push_back(Sum);

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
 
   case ISD::ADDCARRY:
   case ISD::SUBCARRY:    Res = PromoteIntRes_ADDSUBCARRY(N, ResNo); break;
 
   case ISD::SADDSAT:
   case ISD::UADDSAT:
   case ISD::SSUBSAT:
   case ISD::USUBSAT:     Res = PromoteIntRes_ADDSUBSAT(N); break;
+  case ISD::SMULFIX:     Res = PromoteIntRes_SMULFIX(N); break;
 
   case ISD::ATOMIC_LOAD:
     Res = PromoteIntRes_Atomic0(cast<AtomicSDNode>(N)); break;
 
   case ISD::ATOMIC_LOAD_ADD:
   case ISD::ATOMIC_LOAD_SUB:
   case ISD::ATOMIC_LOAD_AND:
   case ISD::ATOMIC_LOAD_CLR:
SDValue DAGTypeLegalizer::PromoteIntRes_ADDSUBSAT(SDNode *N) {
   Op2Promoted =
       DAG.getNode(ISD::SHL, dl, PromotedType, Op2Promoted, ShiftAmount);
 
   SDValue Result =
       DAG.getNode(Opcode, dl, PromotedType, Op1Promoted, Op2Promoted);
   return DAG.getNode(ShiftOp, dl, PromotedType, Result, ShiftAmount);
 }
 
+SDValue DAGTypeLegalizer::PromoteIntRes_SMULFIX(SDNode *N) {
+  // Can just promote the operands then continue with operation.
+  SDLoc dl(N);
+  SDValue Op1 = N->getOperand(0);
+  SDValue Op2 = N->getOperand(1);
+  SDValue Op1Promoted = GetPromotedInteger(Op1);
+  SDValue Op2Promoted = GetPromotedInteger(Op2);
+  EVT PromotedType = Op1Promoted.getValueType();
+  return DAG.getNode(N->getOpcode(), dl, PromotedType, Op1Promoted, Op2Promoted,
+                     N->getOperand(2));
+}
+
 SDValue DAGTypeLegalizer::PromoteIntRes_SADDSUBO(SDNode *N, unsigned ResNo) {
   if (ResNo == 1)
     return PromoteIntRes_Overflow(N);
 
   // The operation overflowed iff the result in the larger type is not the
   // sign extension of its truncation to the original type.
   SDValue LHS = SExtPromotedInteger(N->getOperand(0));
   SDValue RHS = SExtPromotedInteger(N->getOperand(1));
#endif
   case ISD::USUBO: ExpandIntRes_UADDSUBO(N, Lo, Hi); break;
   case ISD::UMULO:
   case ISD::SMULO: ExpandIntRes_XMULO(N, Lo, Hi); break;
 
   case ISD::SADDSAT:
   case ISD::UADDSAT:
   case ISD::SSUBSAT:
   case ISD::USUBSAT: ExpandIntRes_ADDSUBSAT(N, Lo, Hi); break;
+  case ISD::SMULFIX: ExpandIntRes_SMULFIX(N, Lo, Hi); break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
   if (Lo.getNode())
     SetExpandedInteger(SDValue(N, ResNo), Lo, Hi);
 }
 
 /// Lower an atomic node to the appropriate builtin call.

 }
 
 void DAGTypeLegalizer::ExpandIntRes_ADDSUBSAT(SDNode *N, SDValue &Lo,
                                               SDValue &Hi) {
   SDValue Result = TLI.getExpandedSaturationAdditionSubtraction(N, DAG);
   SplitInteger(Result, Lo, Hi);
 }
 
+void DAGTypeLegalizer::ExpandIntRes_SMULFIX(SDNode *N, SDValue &Lo,
+                                            SDValue &Hi) {
+  SDLoc dl(N);
+  EVT VT = N->getValueType(0);
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  unsigned Scale = cast<ConstantSDNode>(N->getOperand(2))->getZExtValue();
+  if (!Scale) {
+    SDValue Result = DAG.getNode(ISD::MUL, dl, VT, LHS, RHS);
+    SplitInteger(Result, Lo, Hi);
+    return;
+  }
+
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
+  SDValue LL, LH, RL, RH;
+  GetExpandedInteger(LHS, LL, LH);
+  GetExpandedInteger(RHS, RL, RH);
+  if (!TLI.expandMUL(
+          DAG.getNode(ISD::MUL, dl, VT, LHS, RHS).getNode(), Lo, Hi, NVT, DAG,
+          TargetLowering::MulExpansionKind::OnlyLegalOrCustom, LL, LH, RL, RH))
+    return;
+
+  Lo = DAG.getNode(ISD::SRL, dl, NVT, Lo, DAG.getConstant(Scale, dl, NVT));
+  Hi = DAG.getNode(ISD::SHL, dl, NVT, Hi,
+                   DAG.getConstant(NVT.getScalarSizeInBits() - Scale, dl, NVT));
+}
+
 void DAGTypeLegalizer::ExpandIntRes_SADDSUBO(SDNode *Node,
                                              SDValue &Lo, SDValue &Hi) {
   SDValue LHS = Node->getOperand(0);
   SDValue RHS = Node->getOperand(1);
   SDLoc dl(Node);
 
   // Expand the result by simply replacing it with the equivalent
   // non-overflow-checking operation.
   SDValue Sum = DAG.getNode(Node->getOpcode() == ISD::SADDO ?
                             ISD::ADD : ISD::SUB, dl, LHS.getValueType(),
                             LHS, RHS);
   SplitInteger(Sum, Lo, Hi);
 
   // Compute the overflow.
   //
   //   LHSSign -> LHS >= 0
   //   RHSSign -> RHS >= 0
   //   SumSign -> Sum >= 0
   //
   //   Add:
   //   Overflow -> (LHSSign == RHSSign) && (LHSSign != SumSign)
   //   Sub:
   //   Overflow -> (LHSSign != RHSSign) && (LHSSign != SumSign)
   //
   EVT OType = Node->getValueType(1);
   SDValue Zero = DAG.getConstant(0, dl, LHS.getValueType());
 
   SDValue LHSSign = DAG.getSetCC(dl, OType, LHS, Zero, ISD::SETGE);
   SDValue RHSSign = DAG.getSetCC(dl, OType, RHS, Zero, ISD::SETGE);
   SDValue SignsMatch = DAG.getSetCC(dl, OType, LHSSign, RHSSign,
                                     Node->getOpcode() == ISD::SADDO ?
                                     ISD::SETEQ : ISD::SETNE);
 
   SDValue SumSign = DAG.getSetCC(dl, OType, Sum, Zero, ISD::SETGE);
   SDValue SumSignNE = DAG.getSetCC(dl, OType, LHSSign, SumSign, ISD::SETNE);
 
   SDValue Cmp = DAG.getNode(ISD::AND, dl, OType, SignsMatch, SumSignNE);
 
   // Use the calculated overflow everywhere.
   ReplaceValueWith(SDValue(Node, 1), Cmp);
 }
 
 void DAGTypeLegalizer::ExpandIntRes_SDIV(SDNode *N,
                                          SDValue &Lo, SDValue &Hi) {
SDValue DAGTypeLegalizer::ExpandIntOp_BR_CC(SDNode *N) {
 
   // Update N to have the operands specified.
   return SDValue(DAG.UpdateNodeOperands(N, N->getOperand(0),
                                 DAG.getCondCode(CCCode), NewLHS, NewRHS,
                                 N->getOperand(4)), 0);
 }
 
 SDValue DAGTypeLegalizer::ExpandIntOp_SELECT_CC(SDNode *N) {
   SDValue NewLHS = N->getOperand(0), NewRHS = N->getOperand(1);
   ISD::CondCode CCCode = cast<CondCodeSDNode>(N->getOperand(4))->get();
   IntegerExpandSetCCOperands(NewLHS, NewRHS, CCCode, SDLoc(N));
 
   // If ExpandSetCCOperands returned a scalar, we need to compare the result
   // against zero to select between true and false values.
   if (!NewRHS.getNode()) {
     NewRHS = DAG.getConstant(0, SDLoc(N), NewLHS.getValueType());
     CCCode = ISD::SETNE;

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue PromoteIntRes_SRL(SDNode *N);
   SDValue PromoteIntRes_TRUNCATE(SDNode *N);
   SDValue PromoteIntRes_UADDSUBO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_ADDSUBCARRY(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_UNDEF(SDNode *N);
   SDValue PromoteIntRes_VAARG(SDNode *N);
   SDValue PromoteIntRes_XMULO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_ADDSUBSAT(SDNode *N);
+  SDValue PromoteIntRes_SMULFIX(SDNode *N);
 
   // Integer Operand Promotion.
   bool PromoteIntegerOperand(SDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_ANY_EXTEND(SDNode *N);
   SDValue PromoteIntOp_ATOMIC_STORE(AtomicSDNode *N);
   SDValue PromoteIntOp_BITCAST(SDNode *N);
   SDValue PromoteIntOp_BUILD_PAIR(SDNode *N);
   SDValue PromoteIntOp_BR_CC(SDNode *N, unsigned OpNo);
private:
   void ExpandIntRes_Shift             (SDNode *N, SDValue &Lo, SDValue &Hi);
 
   void ExpandIntRes_MINMAX            (SDNode *N, SDValue &Lo, SDValue &Hi);
 
   void ExpandIntRes_SADDSUBO          (SDNode *N, SDValue &Lo, SDValue &Hi);
   void ExpandIntRes_UADDSUBO          (SDNode *N, SDValue &Lo, SDValue &Hi);
   void ExpandIntRes_XMULO             (SDNode *N, SDValue &Lo, SDValue &Hi);
   void ExpandIntRes_ADDSUBSAT         (SDNode *N, SDValue &Lo, SDValue &Hi);
+  void ExpandIntRes_SMULFIX           (SDNode *N, SDValue &Lo, SDValue &Hi);
 
   void ExpandIntRes_ATOMIC_LOAD       (SDNode *N, SDValue &Lo, SDValue &Hi);
 
   void ExpandShiftByConstant(SDNode *N, const APInt &Amt,
                              SDValue &Lo, SDValue &Hi);
   bool ExpandShiftWithKnownAmountBit(SDNode *N, SDValue &Lo, SDValue &Hi);
   bool ExpandShiftWithUnknownAmountBit(SDNode *N, SDValue &Lo, SDValue &Hi);
 
private:
   SDValue ScalarizeVecRes_SCALAR_TO_VECTOR(SDNode *N);
   SDValue ScalarizeVecRes_VSELECT(SDNode *N);
   SDValue ScalarizeVecRes_SELECT(SDNode *N);
   SDValue ScalarizeVecRes_SELECT_CC(SDNode *N);
   SDValue ScalarizeVecRes_SETCC(SDNode *N);
   SDValue ScalarizeVecRes_UNDEF(SDNode *N);
   SDValue ScalarizeVecRes_VECTOR_SHUFFLE(SDNode *N);
 
+  SDValue ScalarizeVecRes_SMULFIX(SDNode *N);
+
   // Vector Operand Scalarization: <1 x ty> -> ty.
   bool ScalarizeVectorOperand(SDNode *N, unsigned OpNo);
   SDValue ScalarizeVecOp_BITCAST(SDNode *N);
   SDValue ScalarizeVecOp_UnaryOp(SDNode *N);
   SDValue ScalarizeVecOp_CONCAT_VECTORS(SDNode *N);
   SDValue ScalarizeVecOp_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue ScalarizeVecOp_VSELECT(SDNode *N);
   SDValue ScalarizeVecOp_VSETCC(SDNode *N);
private:
   void SplitVecRes_BinOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_TernaryOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_UnaryOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_ExtendOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_InregOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_ExtVecInRegOp(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_StrictFPOp(SDNode *N, SDValue &Lo, SDValue &Hi);
 
+  void SplitVecRes_SMULFIX(SDNode *N, SDValue &Lo, SDValue &Hi);
+
   void SplitVecRes_BITCAST(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_BUILD_VECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_CONCAT_VECTORS(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_EXTRACT_SUBVECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_INSERT_SUBVECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_FPOWI(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_FCOPYSIGN(SDNode *N, SDValue &Lo, SDValue &Hi);
   void SplitVecRes_INSERT_VECTOR_ELT(SDNode *N, SDValue &Lo, SDValue &Hi);

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   case ISD::UMUL_LOHI:
   case ISD::FCANONICALIZE:
   case ISD::SADDSAT:
   case ISD::UADDSAT:
   case ISD::SSUBSAT:
   case ISD::USUBSAT:
     Action = TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0));
     break;
+  case ISD::SMULFIX: {
+    unsigned Scale = cast<ConstantSDNode>(Node->getOperand(2))->getZExtValue();
+    Action = TLI.getFixedPointOperationAction(Node->getOpcode(),
+                                              Node->getValueType(0), Scale);
+    break;
+  }
   case ISD::FP_ROUND_INREG:
     Action = TLI.getOperationAction(Node->getOpcode(),
                cast<VTSDNode>(Node->getOperand(1))->getVT());
     break;
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
     Action = TLI.getOperationAction(Node->getOpcode(),
                                     Node->getOperand(0).getValueType());

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
     R = ScalarizeVecRes_StrictFPOp(N);
     break;
+  case ISD::SMULFIX:
+    R = ScalarizeVecRes_SMULFIX(N);
+    break;
   }
 
   // If R is null, the sub-method took care of registering the result.
   if (R.getNode())
     SetScalarizedVector(SDValue(N, ResNo), R);
 }
 
 SDValue DAGTypeLegalizer::ScalarizeVecRes_BinOp(SDNode *N) {
   SDValue LHS = GetScalarizedVector(N->getOperand(0));
   SDValue RHS = GetScalarizedVector(N->getOperand(1));
   return DAG.getNode(N->getOpcode(), SDLoc(N),
                      LHS.getValueType(), LHS, RHS, N->getFlags());
 }
 
 SDValue DAGTypeLegalizer::ScalarizeVecRes_TernaryOp(SDNode *N) {
   SDValue Op0 = GetScalarizedVector(N->getOperand(0));
   SDValue Op1 = GetScalarizedVector(N->getOperand(1));
   SDValue Op2 = GetScalarizedVector(N->getOperand(2));
   return DAG.getNode(N->getOpcode(), SDLoc(N),
                      Op0.getValueType(), Op0, Op1, Op2);
 }
 
+SDValue DAGTypeLegalizer::ScalarizeVecRes_SMULFIX(SDNode *N) {
+  SDValue Op0 = GetScalarizedVector(N->getOperand(0));
+  SDValue Op1 = GetScalarizedVector(N->getOperand(1));
+  SDValue Op2 = N->getOperand(2);
+  return DAG.getNode(N->getOpcode(), SDLoc(N), Op0.getValueType(), Op0, Op1,
+                     Op2);
+}
+
 SDValue DAGTypeLegalizer::ScalarizeVecRes_StrictFPOp(SDNode *N) {
   EVT VT = N->getValueType(0).getVectorElementType();
   unsigned NumOpers = N->getNumOperands();
   SDValue Chain = N->getOperand(0);
   EVT ValueVTs[] = {VT, MVT::Other};
   SDLoc dl(N);
 
   SmallVector<SDValue, 4> Opers;
#endif
   case ISD::STRICT_FMAXNUM:
   case ISD::STRICT_FMINNUM:
   case ISD::STRICT_FCEIL:
   case ISD::STRICT_FFLOOR:
   case ISD::STRICT_FROUND:
   case ISD::STRICT_FTRUNC:
     SplitVecRes_StrictFPOp(N, Lo, Hi);
     break;
+  case ISD::SMULFIX:
+    SplitVecRes_SMULFIX(N, Lo, Hi);
+    break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
   if (Lo.getNode())
     SetSplitVector(SDValue(N, ResNo), Lo, Hi);
 }
 
 void DAGTypeLegalizer::SplitVecRes_BinOp(SDNode *N, SDValue &Lo,
void DAGTypeLegalizer::SplitVecRes_TernaryOp(SDNode *N, SDValue &Lo,
   SDLoc dl(N);
 
   Lo = DAG.getNode(N->getOpcode(), dl, Op0Lo.getValueType(),
                    Op0Lo, Op1Lo, Op2Lo);
   Hi = DAG.getNode(N->getOpcode(), dl, Op0Hi.getValueType(),
                    Op0Hi, Op1Hi, Op2Hi);
 }
 
+void DAGTypeLegalizer::SplitVecRes_SMULFIX(SDNode *N, SDValue &Lo,
+                                           SDValue &Hi) {
+  SDValue LHSLo, LHSHi;
+  GetSplitVector(N->getOperand(0), LHSLo, LHSHi);
+  SDValue RHSLo, RHSHi;
+  GetSplitVector(N->getOperand(1), RHSLo, RHSHi);
+  SDLoc dl(N);
+  SDValue Op2 = N->getOperand(2);
+
+  unsigned Opcode = N->getOpcode();
+  Lo = DAG.getNode(Opcode, dl, LHSLo.getValueType(), LHSLo, RHSLo, Op2);
+  Hi = DAG.getNode(Opcode, dl, LHSHi.getValueType(), LHSHi, RHSHi, Op2);
+}
+
 void DAGTypeLegalizer::SplitVecRes_BITCAST(SDNode *N, SDValue &Lo,
                                            SDValue &Hi) {
   // We know the result is a vector.  The input may be either a vector or a
   // scalar value.
   EVT LoVT, HiVT;
   std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(N->getValueType(0));
   SDLoc dl(N);
 

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::ssub_sat: {
     return nullptr;
   }
   case Intrinsic::usub_sat: {
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     setValue(&I, DAG.getNode(ISD::USUBSAT, sdl, Op1.getValueType(), Op1, Op2));
     return nullptr;
   }
+  case Intrinsic::smul_fix: {
+    SDValue Op1 = getValue(I.getArgOperand(0));
+    SDValue Op2 = getValue(I.getArgOperand(1));
+    SDValue Op3 = getValue(I.getArgOperand(2));
+    setValue(&I,
+             DAG.getNode(ISD::SMULFIX, sdl, Op1.getValueType(), Op1, Op2, Op3));
+    return nullptr;
+  }
   case Intrinsic::stacksave: {
     SDValue Op = getRoot();
     Res = DAG.getNode(
         ISD::STACKSAVE, sdl,
         DAG.getVTList(TLI.getPointerTy(DAG.getDataLayout()), MVT::Other), Op);
     setValue(&I, Res);
     DAG.setRoot(Res.getValue(1));
     return nullptr;

llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::SHL_PARTS:                  return "shl_parts";
   case ISD::SRA_PARTS:                  return "sra_parts";
   case ISD::SRL_PARTS:                  return "srl_parts";
 
   case ISD::SADDSAT:                    return "saddsat";
   case ISD::UADDSAT:                    return "uaddsat";
   case ISD::SSUBSAT:                    return "ssubsat";
   case ISD::USUBSAT:                    return "usubsat";
+  case ISD::SMULFIX:                    return "smulfix";
 
   // Conversion operators.
   case ISD::SIGN_EXTEND:                return "sign_extend";
   case ISD::ZERO_EXTEND:                return "zero_extend";
   case ISD::ANY_EXTEND:                 return "any_extend";
   case ISD::SIGN_EXTEND_INREG:          return "sign_extend_inreg";
   case ISD::ANY_EXTEND_VECTOR_INREG:    return "any_extend_vector_inreg";
   case ISD::SIGN_EXTEND_VECTOR_INREG:   return "sign_extend_vector_inreg";

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (Opcode == ISD::UADDSAT) {
     APInt MaxVal = APInt::getSignedMaxValue(BitWidth);
     SDValue SatMin = DAG.getConstant(MinVal, dl, ResultType);
     SDValue SatMax = DAG.getConstant(MaxVal, dl, ResultType);
     SDValue SumNeg = DAG.getSetCC(dl, BoolVT, SumDiff, Zero, ISD::SETLT);
     Result = DAG.getSelect(dl, ResultType, SumNeg, SatMax, SatMin);
     return DAG.getSelect(dl, ResultType, Overflow, Result, SumDiff);
   }
 }
+
+SDValue
+TargetLowering::getExpandedFixedPointMultiplication(SDNode *Node,
+                                                    SelectionDAG &DAG) const {
+  assert(Node->getOpcode() == ISD::SMULFIX && "Expected opcode to be SMULFIX.");
+  assert(Node->getNumOperands() == 3 &&
+         "Expected signed fixed point multiplication to have 3 operands.");
+
+  SDLoc dl(Node);
+  SDValue LHS = Node->getOperand(0);
+  SDValue RHS = Node->getOperand(1);
+  assert(LHS.getValueType().isScalarInteger() &&
+         "Expected operands to be integers. Vector of int arguments should "
+         "already be unrolled.");
+  assert(RHS.getValueType().isScalarInteger() &&
+         "Expected operands to be integers. Vector of int arguments should "
+         "already be unrolled.");
+  assert(LHS.getValueType() == RHS.getValueType() &&
+         "Expected both operands to be the same type");
+
+  unsigned Scale = cast<ConstantSDNode>(Node->getOperand(2))->getZExtValue();
+  EVT VT = LHS.getValueType();
+  assert(Scale < VT.getScalarSizeInBits() &&
+         "Expected scale to be less than the number of bits.");
+
+  SDValue Lo = DAG.getNode(ISD::MUL, dl, VT, LHS, RHS);
+  if (Scale) {
+    SDValue Hi = DAG.getNode(ISD::MULHS, dl, VT, LHS, RHS);
+    Lo = DAG.getNode(ISD::SRL, dl, VT, Lo, DAG.getConstant(Scale, dl, VT));
+    Hi = DAG.getNode(ISD::SHL, dl, VT, Hi,
+                     DAG.getConstant(VT.getScalarSizeInBits() - Scale, dl, VT));
+    return DAG.getNode(ISD::ADD, dl, VT, Lo, Hi);
+  } else {
+    return Lo;
+  }
+}

llvm/lib/CodeGen/TargetLoweringBase.cpp

for (MVT VT : MVT::all_valuetypes()) {
     setOperationAction(ISD::SMAX, VT, Expand);
     setOperationAction(ISD::UMIN, VT, Expand);
     setOperationAction(ISD::UMAX, VT, Expand);
     setOperationAction(ISD::ABS, VT, Expand);
     setOperationAction(ISD::SADDSAT, VT, Expand);
     setOperationAction(ISD::UADDSAT, VT, Expand);
     setOperationAction(ISD::SSUBSAT, VT, Expand);
     setOperationAction(ISD::USUBSAT, VT, Expand);
+    setOperationAction(ISD::SMULFIX, VT, Expand);
 
     // Overflow operations default to expand
     setOperationAction(ISD::SADDO, VT, Expand);
     setOperationAction(ISD::SSUBO, VT, Expand);
     setOperationAction(ISD::UADDO, VT, Expand);
     setOperationAction(ISD::USUBO, VT, Expand);
     setOperationAction(ISD::SMULO, VT, Expand);
     setOperationAction(ISD::UMULO, VT, Expand);

llvm/lib/IR/Verifier.cpp

case Intrinsic::usub_sat: {
     Assert(Op1->getType()->isIntOrIntVectorTy(),
            "first operand of [us][add|sub]_sat must be an int type or vector "
            "of ints");
     Assert(Op2->getType()->isIntOrIntVectorTy(),
            "second operand of [us][add|sub]_sat must be an int type or vector "
            "of ints");
     break;
   }
+  case Intrinsic::smul_fix: {
+    Value *Op1 = CS.getArgOperand(0);
+    Value *Op2 = CS.getArgOperand(1);
+    Value *Op3 = CS.getArgOperand(2);
+    Assert(Op1->getType()->isIntOrIntVectorTy(),
+           "first operand of smul_fix must be an int type or vector "
+           "of ints");
+    Assert(Op2->getType()->isIntOrIntVectorTy(),
+           "second operand of smul_fix must be an int type or vector "
+           "of ints");
+    Assert(Op3->getType()->isIntegerTy(),
+           "third argumenr of smul_fix must be an int");
+    break;
+  }
   };
 }
 
 /// Carefully grab the subprogram from a local scope.
 ///
 /// This carefully grabs the subprogram from a local scope, avoiding the
 /// built-in assertions that would typically fire.
 static DISubprogram *getSubprogram(Metadata *LocalScope) {

llvm/test/CodeGen/X86/smul_fix.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mcpu=generic -mtriple=x86_64-linux | FileCheck %s
+; RUN: llc < %s -mcpu=generic -mtriple=i686 -mattr=cmov | FileCheck %s --check-prefix=CHECK32
+
+declare  i4  @llvm.smul.fix.i4   (i4,  i4, i32)
+declare  i32 @llvm.smul.fix.i32  (i32, i32, i32)
+declare  i64 @llvm.smul.fix.i64  (i64, i64, i32)
+declare  <4 x i32> @llvm.smul.fix.v4i32(<4 x i32>, <4 x i32>, i32)
+
+define i32 @func(i32 %x, i32 %y) {
+; CHECK-LABEL: func:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movslq %edi, %rcx
+; CHECK-NEXT:    imull %esi, %edi
+; CHECK-NEXT:    movslq %esi, %rax
+; CHECK-NEXT:    imulq %rcx, %rax
+; CHECK-NEXT:    shrq $32, %rax
+; CHECK-NEXT:    shldl $30, %edi, %eax
+; CHECK-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movl %eax, %ecx
+; CHECK32-NEXT:    imull %edx, %ecx
+; CHECK32-NEXT:    imull %edx
+; CHECK32-NEXT:    shrdl $2, %edx, %ecx
+; CHECK32-NEXT:    movl %ecx, %eax
+; CHECK32-NEXT:    retl
+  %tmp = call i32 @llvm.smul.fix.i32(i32 %x, i32 %y, i32 2);
+  ret i32 %tmp;
+}
+
+define i64 @func2(i64 %x, i64 %y) {
+; CHECK-LABEL: func2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    movq %rdi, %rcx
+; CHECK-NEXT:    imulq %rsi, %rcx
+; CHECK-NEXT:    imulq %rsi
+; CHECK-NEXT:    movq %rdx, %rax
+; CHECK-NEXT:    shldq $62, %rcx, %rax
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func2:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    pushl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    .cfi_offset %esi, -8
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    movl %ecx, %eax
+; CHECK32-NEXT:    mull %esi
+; CHECK32-NEXT:    shrl $2, %eax
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    addl %ecx, %edx
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    addl %esi, %edx
+; CHECK32-NEXT:    shll $30, %edx
+; CHECK32-NEXT:    popl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK32-NEXT:    retl
+  %tmp = call i64 @llvm.smul.fix.i64(i64 %x, i64 %y, i32 2);
+  ret i64 %tmp;
+}
+
+define i4 @func3(i4 %x, i4 %y) {
+; CHECK-LABEL: func3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    mulb %sil
+; CHECK-NEXT:    shrb $2, %al
+; CHECK-NEXT:    movsbl %sil, %ecx
+; CHECK-NEXT:    movsbl %dil, %edx
+; CHECK-NEXT:    imull %ecx, %edx
+; CHECK-NEXT:    shrl $8, %edx
+; CHECK-NEXT:    shlb $6, %dl
+; CHECK-NEXT:    orb %dl, %al
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func3:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    movsbl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movsbl {{[0-9]+}}(%esp), %edx
+; CHECK32-NEXT:    movl %eax, %ecx
+; CHECK32-NEXT:    imull %edx, %ecx
+; CHECK32-NEXT:    shlb $6, %ch
+; CHECK32-NEXT:    # kill: def $al killed $al killed $eax
+; CHECK32-NEXT:    mulb %dl
+; CHECK32-NEXT:    shrb $2, %al
+; CHECK32-NEXT:    orb %ch, %al
+; CHECK32-NEXT:    retl
+  %tmp = call i4 @llvm.smul.fix.i4(i4 %x, i4 %y, i32 2);
+  ret i4 %tmp;
+}
+
+define <4 x i32> @vec(<4 x i32> %x, <4 x i32> %y) {
+; CHECK-LABEL: vec:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; CHECK-NEXT:    movd %xmm2, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; CHECK-NEXT:    movd %xmm2, %ecx
+; CHECK-NEXT:    movslq %ecx, %rdx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    imulq %rdx, %rax
+; CHECK-NEXT:    shrq $32, %rax
+; CHECK-NEXT:    shldl $30, %ecx, %eax
+; CHECK-NEXT:    movd %eax, %xmm2
+; CHECK-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; CHECK-NEXT:    movd %xmm3, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; CHECK-NEXT:    movd %xmm3, %ecx
+; CHECK-NEXT:    movslq %ecx, %rdx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    imulq %rdx, %rax
+; CHECK-NEXT:    shrq $32, %rax
+; CHECK-NEXT:    shldl $30, %ecx, %eax
+; CHECK-NEXT:    movd %eax, %xmm3
+; CHECK-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; CHECK-NEXT:    movd %xmm1, %eax
+; CHECK-NEXT:    movd %xmm0, %ecx
+; CHECK-NEXT:    movslq %ecx, %rdx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    imulq %rdx, %rax
+; CHECK-NEXT:    shrq $32, %rax
+; CHECK-NEXT:    shldl $30, %ecx, %eax
+; CHECK-NEXT:    movd %eax, %xmm2
+; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; CHECK-NEXT:    movd %xmm1, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; CHECK-NEXT:    movd %xmm0, %ecx
+; CHECK-NEXT:    movslq %ecx, %rdx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    imulq %rdx, %rax
+; CHECK-NEXT:    shrq $32, %rax
+; CHECK-NEXT:    shldl $30, %ecx, %eax
+; CHECK-NEXT:    movd %eax, %xmm0
+; CHECK-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; CHECK-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; CHECK-NEXT:    movdqa %xmm2, %xmm0
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: vec:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    pushl %ebp
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    pushl %ebx
+; CHECK32-NEXT:    .cfi_def_cfa_offset 12
+; CHECK32-NEXT:    pushl %edi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 16
+; CHECK32-NEXT:    pushl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 20
+; CHECK32-NEXT:    .cfi_offset %esi, -20
+; CHECK32-NEXT:    .cfi_offset %edi, -16
+; CHECK32-NEXT:    .cfi_offset %ebx, -12
+; CHECK32-NEXT:    .cfi_offset %ebp, -8
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movl %eax, %ebp
+; CHECK32-NEXT:    imull %ecx, %ebp
+; CHECK32-NEXT:    imull %ecx
+; CHECK32-NEXT:    movl %edx, %ecx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    shldl $30, %ebp, %ecx
+; CHECK32-NEXT:    movl %eax, %ebp
+; CHECK32-NEXT:    imull %esi, %ebp
+; CHECK32-NEXT:    imull %esi
+; CHECK32-NEXT:    movl %edx, %esi
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    shldl $30, %ebp, %esi
+; CHECK32-NEXT:    movl %eax, %ebp
+; CHECK32-NEXT:    imull %edi, %ebp
+; CHECK32-NEXT:    imull %edi
+; CHECK32-NEXT:    movl %edx, %edi
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    shldl $30, %ebp, %edi
+; CHECK32-NEXT:    movl %eax, %ebp
+; CHECK32-NEXT:    imull %ebx, %ebp
+; CHECK32-NEXT:    imull %ebx
+; CHECK32-NEXT:    shldl $30, %ebp, %edx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movl %edx, 12(%eax)
+; CHECK32-NEXT:    movl %edi, 8(%eax)
+; CHECK32-NEXT:    movl %esi, 4(%eax)
+; CHECK32-NEXT:    movl %ecx, (%eax)
+; CHECK32-NEXT:    popl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 16
+; CHECK32-NEXT:    popl %edi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 12
+; CHECK32-NEXT:    popl %ebx
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    popl %ebp
+; CHECK32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK32-NEXT:    retl $4
+  %tmp = call <4 x i32> @llvm.smul.fix.v4i32(<4 x i32> %x, <4 x i32> %y, i32 2);
+  ret <4 x i32> %tmp;
+}
+
+; These result in regular integer multiplication
+define i32 @func4(i32 %x, i32 %y) {
+; CHECK-LABEL: func4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    imull %esi, %eax
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func4:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    retl
+  %tmp = call i32 @llvm.smul.fix.i32(i32 %x, i32 %y, i32 0);
+  ret i32 %tmp;
+}
+
+define i64 @func5(i64 %x, i64 %y) {
+; CHECK-LABEL: func5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    imulq %rsi, %rax
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func5:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    pushl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    .cfi_offset %esi, -8
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    movl %ecx, %eax
+; CHECK32-NEXT:    mull %esi
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    addl %ecx, %edx
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    addl %esi, %edx
+; CHECK32-NEXT:    popl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK32-NEXT:    retl
+  %tmp = call i64 @llvm.smul.fix.i64(i64 %x, i64 %y, i32 0);
+  ret i64 %tmp;
+}
+
+define i4 @func6(i4 %x, i4 %y) {
+; CHECK-LABEL: func6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    # kill: def $al killed $al killed $eax
+; CHECK-NEXT:    mulb %sil
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: func6:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    movb {{[0-9]+}}(%esp), %al
+; CHECK32-NEXT:    mulb {{[0-9]+}}(%esp)
+; CHECK32-NEXT:    retl
+  %tmp = call i4 @llvm.smul.fix.i4(i4 %x, i4 %y, i32 0);
+  ret i4 %tmp;
+}
+
+define <4 x i32> @vec2(<4 x i32> %x, <4 x i32> %y) {
+; CHECK-LABEL: vec2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; CHECK-NEXT:    movd %xmm2, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; CHECK-NEXT:    movd %xmm2, %ecx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    movd %ecx, %xmm2
+; CHECK-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; CHECK-NEXT:    movd %xmm3, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; CHECK-NEXT:    movd %xmm3, %ecx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    movd %ecx, %xmm3
+; CHECK-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; CHECK-NEXT:    movd %xmm1, %eax
+; CHECK-NEXT:    movd %xmm0, %ecx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    movd %ecx, %xmm2
+; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; CHECK-NEXT:    movd %xmm1, %eax
+; CHECK-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; CHECK-NEXT:    movd %xmm0, %ecx
+; CHECK-NEXT:    imull %eax, %ecx
+; CHECK-NEXT:    movd %ecx, %xmm0
+; CHECK-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; CHECK-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; CHECK-NEXT:    movdqa %xmm2, %xmm0
+; CHECK-NEXT:    retq
+;
+; CHECK32-LABEL: vec2:
+; CHECK32:       # %bb.0:
+; CHECK32-NEXT:    pushl %edi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    pushl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 12
+; CHECK32-NEXT:    .cfi_offset %esi, -12
+; CHECK32-NEXT:    .cfi_offset %edi, -8
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %edi
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %edi
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %esi
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %edx
+; CHECK32-NEXT:    imull {{[0-9]+}}(%esp), %ecx
+; CHECK32-NEXT:    movl %ecx, 12(%eax)
+; CHECK32-NEXT:    movl %edx, 8(%eax)
+; CHECK32-NEXT:    movl %esi, 4(%eax)
+; CHECK32-NEXT:    movl %edi, (%eax)
+; CHECK32-NEXT:    popl %esi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-NEXT:    popl %edi
+; CHECK32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK32-NEXT:    retl $4
+  %tmp = call <4 x i32> @llvm.smul.fix.v4i32(<4 x i32> %x, <4 x i32> %y, i32 0);
+  ret <4 x i32> %tmp;
+}