[ARM] Enable SMLAL[B|T] instruction selection

[samparker]

Summary

Enable the selection of the 64-bit signed multiply accumulate instructions which operate on 16-bit operands. These are enabled for ARMv5TE onwards for ARM and for V6T2 and other DSP enabled Thumb architectures. Added a new file to help with pattern matching across ISelLowering and ISelDAG.

[t.p.northover]

lib/Target/ARM/ARMISelDAGToDAG.cpp

3132–3146 ↗

(On Diff #88727)

The only functional change here appears to be limiting the first block to Thumb2, but the second block is no more correct on Thumb1 machines so I don't think it actually improves anything.

3150 ↗

(On Diff #88727)

You shouldn't need C++ for this. Patterns embedded in Instructions can't produce more than one value, but by a quirk of notation instantiations of Pat can. So you should be able to write something like

def : Pat<(smlalbb GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
   (SMLALBB $Rn, $Rm, $RLo, $RHi)>;

(after mapping ARMISD::SMLALBB to smlalbb of course).

lib/Target/ARM/ARMISelLowering.cpp

9472 ↗

(On Diff #88727)

What if it's not a constant?

9476 ↗

(On Diff #88727)

What if it's not the same mul as before?

9481 ↗

(On Diff #88727)

Don't you also need to know that the base type was i16? I don't see any checks here.

lib/Target/ARM/ARMPatternHelpers.h

1 ↗

(On Diff #88727)

Names beginning with a underscore and an upper-case letter are reserved in all contexts so you should drop the leading underscore.

You'll also need to add a copyright header. Though this whole file becomes irrelevant if you implement the actual selection in .td files and these functions can move to ARMISelLowering.h.

[efriedma]

Please put an explanation of the .td file changes into the commit message. (I think I follow why it's related, but it wasn't immediately obvious.)

Would it make sense to use ComputeNumSignBits to figure out whether a value is sign-extended? That would handle, for example, cases where the 16-bit inputs are loads, or one of the 16-bit inputs is a constant.

lib/Target/ARM/ARMISelLowering.cpp

9448 ↗

(On Diff #88727)

The .td file says v5TE is required; this says DSP is required. Which is correct?

lib/Target/ARM/ARMPatternHelpers.cpp

7 ↗

(On Diff #88727)

I know you're just moving the code, but it doesn't make sense to check for SIGN_EXTEND here. ARM doesn't have 16-bit integer registers.

test/CodeGen/ARM/longMAC.ll

243 ↗

(On Diff #88727)

The "CHECK-V6-THUMB2-NOT" aren't that useful; can you instead use CHECK-NEXT to check that there aren't any extra instructions in the function?

You might as well just use explicit register names here; given the calling convention is known, there's only one possible register assignment.

[samparker]

In D30044#678987, @efriedma wrote:

Would it make sense to use ComputeNumSignBits to figure out whether a value is sign-extended? That would handle, for example, cases where the 16-bit inputs are loads, or one of the 16-bit inputs is a constant.

This operations only act upon registers, but I will definitely look into this and the load issue, thanks.

lib/Target/ARM/ARMISelDAGToDAG.cpp

3150 ↗

(On Diff #88727)

Ah excellent, I will clean all this up.

lib/Target/ARM/ARMISelLowering.cpp

9448 ↗

(On Diff #88727)

Yes good point, hasDSP is only the Thumb check.

9476 ↗

(On Diff #88727)

Sorry I don't get your point, what I am missing?

lib/Target/ARM/ARMPatternHelpers.h

1 ↗

(On Diff #88727)

Ok. I'd still want to keep the helpers because they're used for pattern matching of different instructions in DAGToDAG as well.

test/CodeGen/ARM/longMAC.ll

243 ↗

(On Diff #88727)

I will change to explicit registers names, but I would like to keep the extra NOT checks because it caught me when I had originally lowered the operands incorrectly, which left the sxth and shifts in.

[samparker]

test/CodeGen/ARM/longMAC.ll

243 ↗

(On Diff #88727)

Sorry, I now understand your comment! I will make the changes

[samparker]

• Enabled for V5TE and added it as a test target.
• Simplifed the tests, because there were three targets testing for the same output.
• Numerous fixes that Tim pointed out.
• Replaced the final isel with tablegen matching.
• Tried using ComputeNumSignBits for identifying sign extending, but this didn't work because for the top values we need to perform SRA, but this then appears like a sext and results in the wrong opcodes being selected.
• Added support for sign extending loads as well as a test for this

[efriedma]

Tried using ComputeNumSignBits for identifying sign extending, but this didn't work because for the top values we need to perform SRA, but this then appears like a sext and results in the wrong opcodes being selected.

That should just be a matter of performing the checks in the right order, I think; you want to generate some sort of SMLAL if the inputs are sign-extended, but you want to special-case operands which are exactly "sra x, 16" or "SIGN_EXTEND_INREG(x)".

[t.p.northover]

lib/Target/ARM/ARMISelLowering.cpp

9476 ↗

(On Diff #88727)

Just knowing that the input to the SRA is some multiply operation isn't sufficient. You need to make sure it's the same one that produced the low bits.

[samparker]

Now use ComputeNumSignBits for bottom 16-bit checking, after checking for the special case of sra, and renamed the function appropriately.

[samparker]

lib/Target/ARM/ARMISelLowering.cpp

9476 ↗

(On Diff #88727)

Is this not what I have done? I'm checking that the MUL operand of the ADDC is the same operand to the SRA.

[t.p.northover]

lib/Target/ARM/ARMISelLowering.cpp

9476 ↗

(On Diff #88727)

Bother, sorry I completely misread it.

lib/Target/ARM/ARMInstrInfo.td

4202 ↗

(On Diff #89096)

I think these should be ARMPat or they'll still be valid in Thumb mode. You'll also need Thumb patterns to select that variant.

lib/Target/ARM/ARMPatternHelpers.cpp

45–60 ↗

(On Diff #89096)

This combination looks extremely dodgy to me. ComputeNumSignBits makes no promises about how it's going to find those bits, and in fact traverses nodes down to a depth of 6.

You've already found two cases where the final SDNode's operand(0) isn't actually the unextended node, I see no reason to think that's all there is.

[samparker]

lib/Target/ARM/ARMInstrInfo.td

4202 ↗

(On Diff #89096)

cheers!

lib/Target/ARM/ARMPatternHelpers.cpp

45–60 ↗

(On Diff #89096)

Ah yes... Do you think it's worth adding a function to SelectionDAG that returns the SDValue that ComputeNumSignBits finds and uses for its calculation?

[efriedma]

lib/Target/ARM/ARMPatternHelpers.cpp

45–60 ↗

(On Diff #89096)

The function you want already exists; it's called SimplifyDemandedBits.

[samparker]

lib/Target/ARM/ARMPatternHelpers.cpp

45–60 ↗

(On Diff #89096)

Great, I'll look into it. Thanks

[samparker]

A large change because I wanted to remove the new pattern helper file, so I've moved the smulwb, smulwt isel code from iseldag to lowering. The accumulator versions of those instructions are now handled in tablegen.

SimplifyDemandedBits is now used to get a signed 16-bit for the various instructions. I manually check LOAD and AssertSext nodes as these weren't handled by SimplifyDemandedBits.

[efriedma]

(See also https://reviews.llvm.org/D30401 ; not sure if it conflicts.)

Could you split the change to match ARMISD::SMULWB earlier into a separate patch? (You can make this patch depend on that, if you want; it just makes it easier to review, or track down regressions, if changes are smaller.)

lib/Target/ARM/ARMISelLowering.cpp

1503 ↗

(On Diff #90180)

This is a weird way to use SimplifyDemandedBits; normally you would first generate the ARMISD::SMLALBB node, then separately run SimplifyDemandedBits on all ARMISD::SMLALBB nodes. It composes better with other optimizations (which might expose additional simplifications), and it's more obviously correct (it isn't clear whether N has other users).

[samparker]

Yeah sure, i'll split it up and I'll have a look at more examples of SimplifyDemandedBits. cheers

[samparker]

Split out the SMULWTB stuff and committed it. This has now been rebased and we also now use the helper functions around SimplifyDemandeBits.

[efriedma]

LGTM.

[efriedma]

Just committed https://reviews.llvm.org/rL298752 . Please test more carefully in the future; reviewers don't always spot every issue.

llvm/trunk/lib/Target/ARM/ARMISelLowering.h

enum NodeType : unsigned {
       VMULLs,       // ...signed
       VMULLu,       // ...unsigned
 
       SMULWB,       // Signed multiply word by half word, bottom
       SMULWT,       // Signed multiply word by half word, top
       UMLAL,        // 64bit Unsigned Accumulate Multiply
       SMLAL,        // 64bit Signed Accumulate Multiply
       UMAAL,        // 64-bit Unsigned Accumulate Accumulate Multiply
+      SMLALBB,      // 64-bit signed accumulate multiply bottom, bottom 16
+      SMLALBT,      // 64-bit signed accumulate multiply bottom, top 16
+      SMLALTB,      // 64-bit signed accumulate multiply top, bottom 16
+      SMLALTT,      // 64-bit signed accumulate multiply top, top 16
 
       // Operands of the standard BUILD_VECTOR node are not legalized, which
       // is fine if BUILD_VECTORs are always lowered to shuffles or other
       // operations, but for ARM some BUILD_VECTORs are legal as-is and their
       // operands need to be legalized.  Define an ARM-specific version of
       // BUILD_VECTOR for this purpose.
       BUILD_VECTOR,
 

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

const char *ARMTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case ARMISD::VTRN:          return "ARMISD::VTRN";
   case ARMISD::VTBL1:         return "ARMISD::VTBL1";
   case ARMISD::VTBL2:         return "ARMISD::VTBL2";
   case ARMISD::VMULLs:        return "ARMISD::VMULLs";
   case ARMISD::VMULLu:        return "ARMISD::VMULLu";
   case ARMISD::UMAAL:         return "ARMISD::UMAAL";
   case ARMISD::UMLAL:         return "ARMISD::UMLAL";
   case ARMISD::SMLAL:         return "ARMISD::SMLAL";
+  case ARMISD::SMLALBB:       return "ARMISD::SMLALBB";
+  case ARMISD::SMLALBT:       return "ARMISD::SMLALBT";
+  case ARMISD::SMLALTB:       return "ARMISD::SMLALTB";
+  case ARMISD::SMLALTT:       return "ARMISD::SMLALTT";
   case ARMISD::SMULWB:        return "ARMISD::SMULWB";
   case ARMISD::SMULWT:        return "ARMISD::SMULWT";
   case ARMISD::BUILD_VECTOR:  return "ARMISD::BUILD_VECTOR";
   case ARMISD::BFI:           return "ARMISD::BFI";
   case ARMISD::VORRIMM:       return "ARMISD::VORRIMM";
   case ARMISD::VBICIMM:       return "ARMISD::VBICIMM";
   case ARMISD::VBSL:          return "ARMISD::VBSL";
   case ARMISD::MEMCPY:        return "ARMISD::MEMCPY";

 
 static SDValue findMUL_LOHI(SDValue V) {
   if (V->getOpcode() == ISD::UMUL_LOHI ||
       V->getOpcode() == ISD::SMUL_LOHI)
     return V;
   return SDValue();
 }
 
+static SDValue AddCombineTo64BitSMLAL16(SDNode *AddcNode, SDNode *AddeNode,
+                                        TargetLowering::DAGCombinerInfo &DCI,
+                                        const ARMSubtarget *Subtarget) {
+
+  if (Subtarget->isThumb()) {
+    if (!Subtarget->hasDSP())
+      return SDValue();
+  } else if (!Subtarget->hasV5TEOps())
+    return SDValue();
+
+  // SMLALBB, SMLALBT, SMLALTB, SMLALTT multiply two 16-bit values and
+  // accumulates the product into a 64-bit value. The 16-bit values will
+  // be sign extended somehow or SRA'd into 32-bit values
+  // (addc (adde (mul 16bit, 16bit), lo), hi)
+  SDValue Mul = AddcNode->getOperand(0);
+  SDValue Hi = AddcNode->getOperand(1);
+  if (Mul.getOpcode() != ISD::MUL) {
+    Hi = AddcNode->getOperand(0);
+    Mul = AddcNode->getOperand(1);
+    if (Mul.getOpcode() != ISD::MUL)
+      return SDValue();
+  }
+
+  SDValue SRA = AddeNode->getOperand(0);
+  SDValue Lo = AddeNode->getOperand(1);
+  if (SRA.getOpcode() != ISD::SRA) {
+    SRA = AddeNode->getOperand(1);
+    Lo = AddeNode->getOperand(0);
+    if (SRA.getOpcode() != ISD::SRA)
+      return SDValue();
+  }
+  if (auto Const = dyn_cast<ConstantSDNode>(SRA.getOperand(1))) {
+    if (Const->getZExtValue() != 31)
+      return SDValue();
+  } else
+    return SDValue();
+
+  if (SRA.getOperand(0) != Mul)
+    return SDValue();
+
+  SelectionDAG &DAG = DCI.DAG;
+  SDLoc dl(AddcNode);
+  unsigned Opcode = 0;
+  SDValue Op0;
+  SDValue Op1;
+
+  if (isS16(Mul.getOperand(0), DAG) && isS16(Mul.getOperand(1), DAG)) {
+    Opcode = ARMISD::SMLALBB;
+    Op0 = Mul.getOperand(0);
+    Op1 = Mul.getOperand(1);
+  } else if (isS16(Mul.getOperand(0), DAG) && isSRA16(Mul.getOperand(1))) {
+    Opcode = ARMISD::SMLALBT;
+    Op0 = Mul.getOperand(0);
+    Op1 = Mul.getOperand(1).getOperand(0);
+  } else if (isSRA16(Mul.getOperand(0)) && isS16(Mul.getOperand(1), DAG)) {
+    Opcode = ARMISD::SMLALTB;
+    Op0 = Mul.getOperand(0).getOperand(0);
+    Op1 = Mul.getOperand(1);
+  } else if (isSRA16(Mul.getOperand(0)) && isSRA16(Mul.getOperand(1))) {
+    Opcode = ARMISD::SMLALTT;
+    Op0 = Mul->getOperand(0).getOperand(0);
+    Op1 = Mul->getOperand(1).getOperand(0);
+  }
+
+  if (!Op0 || !Op1)
+    return SDValue();
+
+  SDValue SMLAL = DAG.getNode(Opcode, dl, DAG.getVTList(MVT::i32, MVT::i32),
+                              Op0, Op1, Lo, Hi);
+  // Replace the ADDs' nodes uses by the MLA node's values.
+  SDValue HiMLALResult(SMLAL.getNode(), 1);
+  SDValue LoMLALResult(SMLAL.getNode(), 0);
+
+  DAG.ReplaceAllUsesOfValueWith(SDValue(AddcNode, 0), LoMLALResult);
+  DAG.ReplaceAllUsesOfValueWith(SDValue(AddeNode, 0), HiMLALResult);
+
+  // Return original node to notify the driver to stop replacing.
+  SDValue resNode(AddcNode, 0);
+  return resNode;
+}
+
 static SDValue AddCombineTo64bitMLAL(SDNode *AddeNode,
-                                     TargetLowering::DAGCombinerInfo &DCI) {
+                                     TargetLowering::DAGCombinerInfo &DCI,
+                                     const ARMSubtarget *Subtarget) {
   // Look for multiply add opportunities.
   // The pattern is a ISD::UMUL_LOHI followed by two add nodes, where
   // each add nodes consumes a value from ISD::UMUL_LOHI and there is
   // a glue link from the first add to the second add.
   // If we find this pattern, we can replace the U/SMUL_LOHI, ADDC, and ADDE by
   // a S/UMLAL instruction.
   //                  UMUL_LOHI
   //                 / :lo    \ :hi
static SDValue AddCombineTo64bitMLAL(SDNode *AddeNode,
   // Check if the two operands are from the same mul_lohi node.
   if (AddcOp0.getNode() == AddcOp1.getNode())
     return SDValue();
 
   assert(AddcNode->getNumValues() == 2 &&
          AddcNode->getValueType(0) == MVT::i32 &&
          "Expect ADDC with two result values. First: i32");
 
-  // Check that the ADDC adds the low result of the S/UMUL_LOHI.
+  // Check that the ADDC adds the low result of the S/UMUL_LOHI. If not, it
+  // maybe a SMLAL which multiplies two 16-bit values.
   if (AddcOp0->getOpcode() != ISD::UMUL_LOHI &&
       AddcOp0->getOpcode() != ISD::SMUL_LOHI &&
       AddcOp1->getOpcode() != ISD::UMUL_LOHI &&
       AddcOp1->getOpcode() != ISD::SMUL_LOHI)
-    return SDValue();
+    return AddCombineTo64BitSMLAL16(AddcNode, AddeNode, DCI, Subtarget);
 
   // Check for the triangle shape.
   SDValue AddeOp0 = AddeNode->getOperand(0);
   SDValue AddeOp1 = AddeNode->getOperand(1);
 
   // Make sure that the ADDE operands are not coming from the same node.
   if (AddeOp0.getNode() == AddeOp1.getNode())
     return SDValue();
static SDValue AddCombineTo64bitUMAAL(SDNode *AddeNode,
                                       const ARMSubtarget *Subtarget) {
   // UMAAL is similar to UMLAL except that it adds two unsigned values.
   // While trying to combine for the other MLAL nodes, first search for the
   // chance to use UMAAL. Check if Addc uses a node which has already
   // been combined into a UMLAL. The other pattern is UMLAL using Addc/Adde
   // as the addend, and it's handled in PerformUMLALCombine.
 
   if (!Subtarget->hasV6Ops() || !Subtarget->hasDSP())
-    return AddCombineTo64bitMLAL(AddeNode, DCI);
+    return AddCombineTo64bitMLAL(AddeNode, DCI, Subtarget);
 
   // Check that we have a glued ADDC node.
   SDNode* AddcNode = AddeNode->getOperand(2).getNode();
   if (AddcNode->getOpcode() != ARMISD::ADDC)
     return SDValue();
 
   // Find the converted UMAAL or quit if it doesn't exist.
   SDNode *UmlalNode = nullptr;
   SDValue AddHi;
   if (AddcNode->getOperand(0).getOpcode() == ARMISD::UMLAL) {
     UmlalNode = AddcNode->getOperand(0).getNode();
     AddHi = AddcNode->getOperand(1);
   } else if (AddcNode->getOperand(1).getOpcode() == ARMISD::UMLAL) {
     UmlalNode = AddcNode->getOperand(1).getNode();
     AddHi = AddcNode->getOperand(0);
   } else {
-    return AddCombineTo64bitMLAL(AddeNode, DCI);
+    return AddCombineTo64bitMLAL(AddeNode, DCI, Subtarget);
   }
 
   // The ADDC should be glued to an ADDE node, which uses the same UMLAL as
   // the ADDC as well as Zero.
   if (!isNullConstant(UmlalNode->getOperand(3)))
     return SDValue();
 
   if ((isNullConstant(AddeNode->getOperand(0)) &&
SDValue ARMTargetLowering::PerformDAGCombine(SDNode *N,
   }
   case ARMISD::SMULWT: {
     unsigned BitWidth = N->getValueType(0).getSizeInBits();
     APInt DemandedMask = APInt::getHighBitsSet(BitWidth, 16);
     if (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI))
       return SDValue();
     break;
   }
+  case ARMISD::SMLALBB: {
+    unsigned BitWidth = N->getValueType(0).getSizeInBits();
+    APInt DemandedMask = APInt::getLowBitsSet(BitWidth, 16);
+    if ((SimplifyDemandedBits(N->getOperand(0), DemandedMask, DCI)) ||
+        (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI)))
+      return SDValue();
+    break;
+  }
+  case ARMISD::SMLALBT: {
+    unsigned LowWidth = N->getOperand(0).getValueType().getSizeInBits();
+    APInt LowMask = APInt::getLowBitsSet(LowWidth, 16);
+    unsigned HighWidth = N->getOperand(1).getValueType().getSizeInBits();
+    APInt HighMask = APInt::getHighBitsSet(HighWidth, 16);
+    if ((SimplifyDemandedBits(N->getOperand(0), LowMask, DCI)) ||
+        (SimplifyDemandedBits(N->getOperand(1), HighMask, DCI)))
+      return SDValue();
+    break;
+  }
+  case ARMISD::SMLALTB: {
+    unsigned HighWidth = N->getOperand(0).getValueType().getSizeInBits();
+    APInt HighMask = APInt::getHighBitsSet(HighWidth, 16);
+    unsigned LowWidth = N->getOperand(1).getValueType().getSizeInBits();
+    APInt LowMask = APInt::getLowBitsSet(LowWidth, 16);
+    if ((SimplifyDemandedBits(N->getOperand(0), HighMask, DCI)) ||
+        (SimplifyDemandedBits(N->getOperand(1), LowMask, DCI)))
+      return SDValue();
+    break;
+  }
+  case ARMISD::SMLALTT: {
+    unsigned BitWidth = N->getValueType(0).getSizeInBits();
+    APInt DemandedMask = APInt::getHighBitsSet(BitWidth, 16);
+    if ((SimplifyDemandedBits(N->getOperand(0), DemandedMask, DCI)) ||
+        (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI)))
+      return SDValue();
+    break;
+  }
   case ISD::INTRINSIC_VOID:
   case ISD::INTRINSIC_W_CHAIN:
     switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
     case Intrinsic::arm_neon_vld1:
     case Intrinsic::arm_neon_vld2:
     case Intrinsic::arm_neon_vld3:
     case Intrinsic::arm_neon_vld4:
     case Intrinsic::arm_neon_vld2lane:

llvm/trunk/lib/Target/ARM/ARMInstrInfo.td

 // SDTBinaryArithWithFlagsInOut - RES1, CPSR = op LHS, RHS, CPSR
 def SDTBinaryArithWithFlagsInOut : SDTypeProfile<2, 3,
                                             [SDTCisSameAs<0, 2>,
                                              SDTCisSameAs<0, 3>,
                                              SDTCisInt<0>,
                                              SDTCisVT<1, i32>,
                                              SDTCisVT<4, i32>]>;
 
+def SDT_LongMac  : SDTypeProfile<2, 4, [SDTCisVT<0, i32>,
+                                        SDTCisSameAs<0, 1>,
+                                        SDTCisSameAs<0, 2>,
+                                        SDTCisSameAs<0, 3>,
+                                        SDTCisSameAs<0, 4>,
+                                        SDTCisSameAs<0, 5>]>;
+
 // Node definitions.
 def ARMWrapper       : SDNode<"ARMISD::Wrapper",     SDTIntUnaryOp>;
 def ARMWrapperPIC    : SDNode<"ARMISD::WrapperPIC",  SDTIntUnaryOp>;
 def ARMWrapperJT     : SDNode<"ARMISD::WrapperJT",   SDTIntUnaryOp>;
 
 def ARMcallseq_start : SDNode<"ISD::CALLSEQ_START", SDT_ARMCallSeqStart,
                               [SDNPHasChain, SDNPSideEffect, SDNPOutGlue]>;
 def ARMcallseq_end   : SDNode<"ISD::CALLSEQ_END",   SDT_ARMCallSeqEnd,

 def ARMbfi           : SDNode<"ARMISD::BFI", SDT_ARMBFI>;
 
 def ARMmemcopy : SDNode<"ARMISD::MEMCPY", SDT_ARMMEMCPY,
                         [SDNPHasChain, SDNPInGlue, SDNPOutGlue,
                          SDNPMayStore, SDNPMayLoad]>;
 
 def ARMsmulwb       : SDNode<"ARMISD::SMULWB", SDTIntBinOp, []>;
 def ARMsmulwt       : SDNode<"ARMISD::SMULWT", SDTIntBinOp, []>;
+def ARMsmlalbb      : SDNode<"ARMISD::SMLALBB", SDT_LongMac, []>;
+def ARMsmlalbt      : SDNode<"ARMISD::SMLALBT", SDT_LongMac, []>;
+def ARMsmlaltb      : SDNode<"ARMISD::SMLALTB", SDT_LongMac, []>;
+def ARMsmlaltt      : SDNode<"ARMISD::SMLALTT", SDT_LongMac, []>;
 
 //===----------------------------------------------------------------------===//
 // ARM Instruction Predicate Definitions.
 //
 def HasV4T           : Predicate<"Subtarget->hasV4TOps()">,
                                  AssemblerPredicate<"HasV4TOps", "armv4t">;
 def NoV4T            : Predicate<"!Subtarget->hasV4TOps()">;
 def HasV5T           : Predicate<"Subtarget->hasV5TOps()">,
def WT : AMulxyIa<0b0001001, 0b10, (outs GPRnopc:$Rd),
            Sched<[WriteMAC16, ReadMUL, ReadMUL, ReadMAC]>;
   }
 }
 
 defm SMUL : AI_smul<"smul">;
 defm SMLA : AI_smla<"smla">;
 
 // Halfword multiply accumulate long: SMLAL<x><y>.
-def SMLALBB : AMulxyI64<0b0001010, 0b00, (outs GPRnopc:$RdLo, GPRnopc:$RdHi),
-                      (ins GPRnopc:$Rn, GPRnopc:$Rm),
-                      IIC_iMAC64, "smlalbb", "\t$RdLo, $RdHi, $Rn, $Rm", []>,
-              Requires<[IsARM, HasV5TE]>,
-              Sched<[WriteMAC64Lo, WriteMAC64Hi, ReadMUL, ReadMUL, ReadMAC, ReadMAC]>;
-
-def SMLALBT : AMulxyI64<0b0001010, 0b10, (outs GPRnopc:$RdLo, GPRnopc:$RdHi),
-                      (ins GPRnopc:$Rn, GPRnopc:$Rm),
-                      IIC_iMAC64, "smlalbt", "\t$RdLo, $RdHi, $Rn, $Rm", []>,
-              Requires<[IsARM, HasV5TE]>,
-              Sched<[WriteMAC64Lo, WriteMAC64Hi, ReadMUL, ReadMUL, ReadMAC, ReadMAC]>;
+class SMLAL<bits<2> opc1, string asm>
+ : AMulxyI64<0b0001010, opc1,
+        (outs GPRnopc:$RdLo, GPRnopc:$RdHi),
+        (ins GPRnopc:$Rn, GPRnopc:$Rm, GPRnopc:$RLo, GPRnopc:$RHi),
+        IIC_iMAC64, asm, "\t$RdLo, $RdHi, $Rn, $Rm", []>,
+        RegConstraint<"$RLo = $RdLo, $RHi = $RdHi">,
 
-def SMLALTB : AMulxyI64<0b0001010, 0b01, (outs GPRnopc:$RdLo, GPRnopc:$RdHi),
-                      (ins GPRnopc:$Rn, GPRnopc:$Rm),
-                      IIC_iMAC64, "smlaltb", "\t$RdLo, $RdHi, $Rn, $Rm", []>,
-              Requires<[IsARM, HasV5TE]>,
-              Sched<[WriteMAC64Lo, WriteMAC64Hi, ReadMUL, ReadMUL, ReadMAC, ReadMAC]>;
-
-def SMLALTT : AMulxyI64<0b0001010, 0b11, (outs GPRnopc:$RdLo, GPRnopc:$RdHi),
-                      (ins GPRnopc:$Rn, GPRnopc:$Rm),
-                      IIC_iMAC64, "smlaltt", "\t$RdLo, $RdHi, $Rn, $Rm", []>,
-              Requires<[IsARM, HasV5TE]>,
-              Sched<[WriteMAC64Lo, WriteMAC64Hi, ReadMUL, ReadMUL, ReadMAC, ReadMAC]>;
+def SMLALBB : SMLAL<0b00, "smlalbb">;
+def SMLALBT : SMLAL<0b10, "smlalbt">;
+def SMLALTB : SMLAL<0b01, "smlaltb">;
+def SMLALTT : SMLAL<0b11, "smlaltt">;
+
+def : ARMV5TEPat<(ARMsmlalbb GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                 (SMLALBB $Rn, $Rm, $RLo, $RHi)>;
+def : ARMV5TEPat<(ARMsmlalbt GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                 (SMLALBT $Rn, $Rm, $RLo, $RHi)>;
+def : ARMV5TEPat<(ARMsmlaltb GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                 (SMLALTB $Rn, $Rm, $RLo, $RHi)>;
+def : ARMV5TEPat<(ARMsmlaltt GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                 (SMLALTT $Rn, $Rm, $RLo, $RHi)>;
 
 // Helper class for AI_smld.
 class AMulDualIbase<bit long, bit sub, bit swap, dag oops, dag iops,
                     InstrItinClass itin, string opc, string asm>
   : AI<oops, iops, MulFrm, itin, opc, asm, []>, Requires<[IsARM, HasV6]> {
   bits<4> Rn;
   bits<4> Rm;
   let Inst{27-23} = 0b01110;

llvm/trunk/lib/Target/ARM/ARMInstrThumb2.td

def : Thumb2DSPMulPat<(add rGPR:$Ra, (mul sext_16_node:$Rn, sext_16_node:$Rm)),
                       (t2SMLABB rGPR:$Rn, rGPR:$Rm, rGPR:$Ra)>;
 def : Thumb2DSPMulPat<(add rGPR:$Ra,
                         (mul sext_16_node:$Rn, (sra rGPR:$Rm, (i32 16)))),
                       (t2SMLABT rGPR:$Rn, rGPR:$Rm, rGPR:$Ra)>;
 def : Thumb2DSPMulPat<(add rGPR:$Ra,
                         (mul (sra rGPR:$Rn, (i32 16)), sext_16_node:$Rm)),
                       (t2SMLATB rGPR:$Rn, rGPR:$Rm, rGPR:$Ra)>;
 
-class T2SMLAL<bits<3> op22_20, bits<4> op7_4, string opc, list<dag> pattern>
-  : T2FourReg_mac<1, op22_20, op7_4,
-                  (outs rGPR:$Ra, rGPR:$Rd),
-                  (ins rGPR:$Rn, rGPR:$Rm),
-                  IIC_iMAC64, opc, "\t$Ra, $Rd, $Rn, $Rm", []>,
-                  Requires<[IsThumb2, HasDSP]>,
-    Sched<[WriteMAC64Lo, WriteMAC64Hi, ReadMUL, ReadMUL, ReadMAC, ReadMAC]>;
-
 // Halfword multiple accumulate long: SMLAL<x><y>
-def t2SMLALBB : T2SMLAL<0b100, 0b1000, "smlalbb", []>;
-def t2SMLALBT : T2SMLAL<0b100, 0b1001, "smlalbt", []>;
-def t2SMLALTB : T2SMLAL<0b100, 0b1010, "smlaltb", []>;
-def t2SMLALTT : T2SMLAL<0b100, 0b1011, "smlaltt", []>;
+def t2SMLALBB : T2MlaLong<0b100, 0b1000, "smlalbb">,
+                          Requires<[IsThumb2, HasDSP]>;
+def t2SMLALBT : T2MlaLong<0b100, 0b1001, "smlalbt">,
+                          Requires<[IsThumb2, HasDSP]>;
+def t2SMLALTB : T2MlaLong<0b100, 0b1010, "smlaltb">,
+                          Requires<[IsThumb2, HasDSP]>;
+def t2SMLALTT : T2MlaLong<0b100, 0b1011, "smlaltt">,
+                          Requires<[IsThumb2, HasDSP]>;
+
+def : Thumb2DSPPat<(ARMsmlalbb GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                   (t2SMLALBB $Rn, $Rm, $RLo, $RHi)>;
+def : Thumb2DSPPat<(ARMsmlalbt GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                   (t2SMLALBT $Rn, $Rm, $RLo, $RHi)>;
+def : Thumb2DSPPat<(ARMsmlaltb GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                   (t2SMLALTB $Rn, $Rm, $RLo, $RHi)>;
+def : Thumb2DSPPat<(ARMsmlaltt GPR:$Rn, GPR:$Rm, GPR:$RLo, GPR:$RHi),
+                   (t2SMLALTT $Rn, $Rm, $RLo, $RHi)>;
 
 class T2DualHalfMul<bits<3> op22_20, bits<4> op7_4, string opc>
   : T2ThreeReg_mac<0, op22_20, op7_4,
                    (outs rGPR:$Rd),
                    (ins rGPR:$Rn, rGPR:$Rm),
                    IIC_iMAC32, opc, "\t$Rd, $Rn, $Rm", []>,
                    Requires<[IsThumb2, HasDSP]>,
    Sched<[WriteMAC32, ReadMUL, ReadMUL, ReadMAC]> {

llvm/trunk/test/CodeGen/ARM/longMAC.ll

 ; RUN: llc -mtriple=arm-eabi %s -o - | FileCheck %s -check-prefix=CHECK --check-prefix=CHECK-LE
 ; RUN: llc -mtriple=armv7-eabi %s -o - | FileCheck %s --check-prefix=CHECK-V7-LE
 ; RUN: llc -mtriple=armeb-eabi %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-BE
 ; RUN: llc -mtriple=armebv7-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V7-BE
 ; RUN: llc -mtriple=thumbv6-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V6-THUMB
-; RUN: llc -mtriple=thumbv6t2-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V6-THUMB2
-; RUN: llc -mtriple=thumbv7-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V7-THUMB
+; RUN: llc -mtriple=thumbv6t2-eabi %s -o - | FileCheck %s -check-prefix=CHECK-T2-DSP
+; RUN: llc -mtriple=thumbv7-eabi %s -o - | FileCheck %s -check-prefix=CHECK-T2-DSP
 ; RUN: llc -mtriple=thumbebv7-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V7-THUMB-BE
 ; RUN: llc -mtriple=thumbv6m-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V6M-THUMB
 ; RUN: llc -mtriple=thumbv7m-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V7M-THUMB
-; RUN: llc -mtriple=thumbv7em-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V7EM-THUMB
+; RUN: llc -mtriple=thumbv7em-eabi %s -o - | FileCheck %s -check-prefix=CHECK-T2-DSP
+; RUN: llc -mtriple=armv5te-eabi %s -o - | FileCheck %s -check-prefix=CHECK-V5TE
 ; Check generated signed and unsigned multiply accumulate long.
 
 define i64 @MACLongTest1(i32 %a, i32 %b, i64 %c) {
 ;CHECK-LABEL: MACLongTest1:
 ;CHECK-V6-THUMB-NOT: umlal
 ;CHECK-LE: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-LE: mov r0, [[RDLO]]
 ;CHECK-LE: mov r1, [[RDHI]]
 ;CHECK-BE: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-BE: mov r0, [[RDHI]]
 ;CHECK-BE: mov r1, [[RDLO]]
-;CHECK-V6-THUMB2: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V6-THUMB2: mov r0, [[RDLO]]
-;CHECK-V6-THUMB2: mov r1, [[RDHI]]
-;CHECK-V7-THUMB: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7-THUMB: mov r0, [[RDLO]]
-;CHECK-V7-THUMB: mov r1, [[RDHI]]
+;CHECK-T2-DSP: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
+;CHECK-T2-DSP-NEXT: mov r0, [[RDLO]]
+;CHECK-T2-DSP-NEXT: mov r1, [[RDHI]]
 ;CHECK-V7-THUMB-BE: umlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-THUMB-BE: mov r0, [[RDHI]]
 ;CHECK-V7-THUMB-BE: mov r1, [[RDLO]]
   %conv = zext i32 %a to i64
   %conv1 = zext i32 %b to i64
   %mul = mul i64 %conv1, %conv
   %add = add i64 %mul, %c
   ret i64 %add
 }
 
 define i64 @MACLongTest2(i32 %a, i32 %b, i64 %c)  {
 ;CHECK-LABEL: MACLongTest2:
 ;CHECK-LE: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-LE: mov r0, [[RDLO]]
 ;CHECK-LE: mov r1, [[RDHI]]
 ;CHECK-BE: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-BE: mov r0, [[RDHI]]
 ;CHECK-BE: mov r1, [[RDLO]]
-;CHECK-V6-THUMB2: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V6-THUMB2: mov r0, [[RDLO]]
-;CHECK-V6-THUMB2: mov r1, [[RDHI]]
-;CHECK-V7-THUMB: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7-THUMB: mov r0, [[RDLO]]
-;CHECK-V7-THUMB: mov r1, [[RDHI]]
+;CHECK-T2-DSP: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
+;CHECK-T2-DSP-NEXT: mov r0, [[RDLO]]
+;CHECK-T2-DSP-NEXT: mov r1, [[RDHI]]
 ;CHECK-V7-THUMB-BE: smlal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-THUMB-BE: mov r0, [[RDHI]]
 ;CHECK-V7-THUMB-BE: mov r1, [[RDLO]]
   %conv = sext i32 %a to i64
   %conv1 = sext i32 %b to i64
   %mul = mul nsw i64 %conv1, %conv
   %add = add nsw i64 %mul, %c
   ret i64 %add

 ;CHECK-LE: mov [[RDHI:r[0-9]+]], #0
 ;CHECK-LE: umlal [[RDLO:r[0-9]+]], [[RDHI]], r1, r0
 ;CHECK-LE: mov r0, [[RDLO]]
 ;CHECK-LE: mov r1, [[RDHI]]
 ;CHECK-BE: mov [[RDHI:r[0-9]+]], #0
 ;CHECK-BE: umlal [[RDLO:r[0-9]+]], [[RDHI]], r1, r0
 ;CHECK-BE: mov r0, [[RDHI]]
 ;CHECK-BE: mov r1, [[RDLO]]
-;CHECK-V6-THUMB2: umlal
-;CHECK-V7-THUMB: umlal
+;CHECK-T2-DSP: umlal
 ;CHECK-V6-THUMB-NOT: umlal
   %conv = zext i32 %b to i64
   %conv1 = zext i32 %a to i64
   %mul = mul i64 %conv, %conv1
   %conv2 = zext i32 %c to i64
   %add = add i64 %mul, %conv2
   ret i64 %add
 }
 
 define i64 @MACLongTest4(i32 %a, i32 %b, i32 %c) {
 ;CHECK-LABEL: MACLongTest4:
 ;CHECK-V6-THUMB-NOT: smlal
-;CHECK-V6-THUMB2: smlal
-;CHECK-V7-THUMB: smlal
+;CHECK-T2-DSP: smlal
 ;CHECK-LE: asr [[RDHI:r[0-9]+]], [[RDLO:r[0-9]+]], #31
 ;CHECK-LE: smlal [[RDLO]], [[RDHI]], r1, r0
 ;CHECK-LE: mov r0, [[RDLO]]
 ;CHECK-LE: mov r1, [[RDHI]]
 ;CHECK-BE: asr [[RDHI:r[0-9]+]], [[RDLO:r[0-9]+]], #31
 ;CHECK-BE: smlal [[RDLO]], [[RDHI]], r1, r0
 ;CHECK-BE: mov r0, [[RDHI]]
 ;CHECK-BE: mov r1, [[RDLO]]
   %conv = sext i32 %b to i64
   %conv1 = sext i32 %a to i64
   %mul = mul nsw i64 %conv, %conv1
   %conv2 = sext i32 %c to i64
   %add = add nsw i64 %mul, %conv2
   ret i64 %add
 }
 
 define i64 @MACLongTest6(i32 %a, i32 %b, i32 %c, i32 %d) {
 ;CHECK-LABEL: MACLongTest6:
 ;CHECK-V6-THUMB-NOT: smull
 ;CHECK-V6-THUMB-NOT: smlal
 ;CHECK: smull   r12, lr, r1, r0
 ;CHECK: smlal   r12, lr, r3, r2
 ;CHECK-V7: smull   [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], r1, r0
 ;CHECK-V7: smlal   [[RDLO]], [[RDHI]], [[Rn:r[0-9]+]], [[Rm:r[0-9]+]]
-;CHECK-V7-THUMB: smull   [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], r1, r0
-;CHECK-V7-THUMB: smlal   [[RDLO]], [[RDHI]], [[Rn:r[0-9]+]], [[Rm:r[0-9]+]]
-;CHECK-V6-THUMB2: smull   [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], r1, r0
-;CHECK-V6-THUMB2: smlal   [[RDLO]], [[RDHI]], [[Rn:r[0-9]+]], [[Rm:r[0-9]+]]
+;CHECK-T2-DSP: smull   [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], r1, r0
+;CHECK-T2-DSP: smlal   [[RDLO]], [[RDHI]], [[Rn:r[0-9]+]], [[Rm:r[0-9]+]]
   %conv = sext i32 %a to i64
   %conv1 = sext i32 %b to i64
   %mul = mul nsw i64 %conv1, %conv
   %conv2 = sext i32 %c to i64
   %conv3 = sext i32 %d to i64
   %mul4 = mul nsw i64 %conv3, %conv2
   %add = add nsw i64 %mul4, %mul
   ret i64 %add

 define i64 @MACLongTest9(i32 %lhs, i32 %rhs, i32 %lo, i32 %hi) {
 ;CHECK-LABEL: MACLongTest9:
 ;CHECK-V7-LE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-LE: mov r0, [[RDLO]]
 ;CHECK-V7-LE: mov r1, [[RDHI]]
 ;CHECK-V7-BE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-BE: mov r0, [[RDHI]]
 ;CHECK-V7-BE: mov r1, [[RDLO]]
-;CHECK-V6-THUMB2: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V6-THUMB2: mov r0, [[RDLO]]
-;CHECK-V6-THUMB2: mov r1, [[RDHI]]
-;CHECK-V7-THUMB: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7-THUMB: mov r0, [[RDLO]]
-;CHECK-V7-THUMB: mov r1, [[RDHI]]
+;CHECK-T2-DSP: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
+;CHECK-T2-DSP-NEXT: mov r0, [[RDLO]]
+;CHECK-T2-DSP-NEXT: mov r1, [[RDHI]]
 ;CHECK-V7-THUMB-BE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-THUMB-BE: mov r0, [[RDHI]]
 ;CHECK-V7-THUMB-BE: mov r1, [[RDLO]]
-;CHECK-V7EM-THUMB: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7EM-THUMB: mov r0, [[RDLO]]
-;CHECK-V7EM-THUMB: mov r1, [[RDHI]]
 ;CHECK-NOT:umaal
 ;CHECK-V6-THUMB-NOT: umaal
 ;CHECK-V6M-THUMB-NOT: umaal
 ;CHECK-V7M-THUMB-NOT: umaal
   %conv = zext i32 %lhs to i64
   %conv1 = zext i32 %rhs to i64
   %mul = mul nuw i64 %conv1, %conv
   %conv2 = zext i32 %lo to i64
   %add = add i64 %mul, %conv2
   %conv3 = zext i32 %hi to i64
   %add2 = add i64 %add, %conv3
   ret i64 %add2
 }
 
 define i64 @MACLongTest10(i32 %lhs, i32 %rhs, i32 %lo, i32 %hi) {
 ;CHECK-LABEL: MACLongTest10:
 ;CHECK-V7-LE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-LE: mov r0, [[RDLO]]
 ;CHECK-V7-LE: mov r1, [[RDHI]]
 ;CHECK-V7-BE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-BE: mov r0, [[RDHI]]
 ;CHECK-V7-BE: mov r1, [[RDLO]]
-;CHECK-V6-THUMB2: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V6-THUMB2: mov r0, [[RDLO]]
-;CHECK-V6-THUMB2: mov r1, [[RDHI]]
-;CHECK-V7-THUMB: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7-THUMB: mov r0, [[RDLO]]
-;CHECK-V7-THUMB: mov r1, [[RDHI]]
+;CHECK-T2-DSP: umaal r2, r3, r1, r0
+;CHECK-T2-DSP-NEXT: mov r0, r2
+;CHECK-T2-DSP-NEXT: mov r1, r3
 ;CHECK-V7-THUMB-BE: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
 ;CHECK-V7-THUMB-BE: mov r0, [[RDHI]]
 ;CHECK-V7-THUMB-BE: mov r1, [[RDLO]]
-;CHECK-V7EM-THUMB: umaal [[RDLO:r[0-9]+]], [[RDHI:r[0-9]+]], [[LHS:r[0-9]+]], [[RHS:r[0-9]+]]
-;CHECK-V7EM-THUMB: mov r0, [[RDLO]]
-;CHECK-V7EM-THUMB: mov r1, [[RDHI]]
 ;CHECK-NOT:umaal
 ;CHECK-V6-THUMB-NOT:umaal
 ;CHECK-V6M-THUMB-NOT: umaal
 ;CHECK-V7M-THUMB-NOT: umaal
   %conv = zext i32 %lhs to i64
   %conv1 = zext i32 %rhs to i64
   %mul = mul nuw i64 %conv1, %conv
   %conv2 = zext i32 %lo to i64
   %conv3 = zext i32 %hi to i64
   %add = add i64 %conv2, %conv3
   %add2 = add i64 %add, %mul
   ret i64 %add2
 }
+
+define i64 @MACLongTest11(i16 %a, i16 %b, i64 %c)  {
+;CHECK-LABEL: MACLongTest11:
+;CHECK-T2-DSP-NOT: sxth
+;CHECK-T2-DSP: smlalbb r3, r2
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-V5TE-NOT: sxth
+;CHECK-V5TE: smlalbb r3, r2
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE-NOT: sxth
+;CHECK-V7-LE: smlalbb r3, r2
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlalbb r2, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlalbb
+;CHECK-BE-NOT: smlalbb
+;CHECK-V6M-THUMB-NOT: smlalbb
+;CHECK-V7M-THUMB-NOT: smlalbb
+  %conv = sext i16 %a to i32
+  %conv1 = sext i16 %b to i32
+  %mul = mul nsw i32 %conv1, %conv
+  %conv2 = sext i32 %mul to i64
+  %add = add nsw i64 %conv2, %c
+  ret i64 %add
+}
+
+define i64 @MACLongTest12(i16 %b, i32 %t, i64 %c)  {
+;CHECK-LABEL: MACLongTest12:
+;CHECK-T2-DSP-NOT: sxth
+;CHECK-T2-DSP-NOT: {{asr|lsr}}
+;CHECK-T2-DSP: smlalbt r3, r2, r0, r1
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-T2-DSP-NOT: sxth
+;CHECK-V5TE-NOT: sxth
+;CHECK-V5TE-NOT: {{asr|lsr}}
+;CHECK-V5TE: smlalbt r3, r2, r0, r1
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE-NOT: sxth
+;CHECK-V7-LE-NOT: {{asr|lsr}}
+;CHECK-V7-LE: smlalbt r3, r2, r0, r1
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlalbt r2, r3,
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlalbt
+;CHECK-BE-NOT: smlalbt
+;CHECK-V6M-THUMB-NOT: smlalbt
+;CHECK-V7M-THUMB-NOT: smlalbt
+  %conv0 = sext i16 %b to i32
+  %conv1 = ashr i32 %t, 16
+  %mul = mul nsw i32 %conv0, %conv1
+  %conv2 = sext i32 %mul to i64
+  %add = add nsw i64 %conv2, %c
+  ret i64 %add
+}
+
+define i64 @MACLongTest13(i32 %t, i16 %b, i64 %c)  {
+;CHECK-LABEL: MACLongTest13:
+;CHECK-T2-DSP-NOT: sxth
+;CHECK-T2-DSP-NOT: {{asr|lsr}}
+;CHECK-T2-DSP: smlaltb r3, r2, r0, r1
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-V5TE-NOT: sxth
+;CHECK-V5TE-NOT: {{asr|lsr}}
+;CHECK-V5TE: smlaltb r3, r2, r0, r1
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE-NOT: sxth
+;CHECK-V7-LE-NOT: {{asr|lsr}}
+;CHECK-V7-LE: smlaltb r3, r2, r0, r1
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlaltb r2, r3, r0, r1
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlaltb
+;CHECK-BE-NOT: smlaltb
+;CHECK-V6M-THUMB-NOT: smlaltb
+;CHECK-V7M-THUMB-NOT: smlaltb
+  %conv0 = ashr i32 %t, 16
+  %conv1= sext i16 %b to i32
+  %mul = mul nsw i32 %conv0, %conv1
+  %conv2 = sext i32 %mul to i64
+  %add = add nsw i64 %conv2, %c
+  ret i64 %add
+}
+
+define i64 @MACLongTest14(i32 %a, i32 %b, i64 %c)  {
+;CHECK-LABEL: MACLongTest14:
+;CHECK-T2-DSP-NOT: {{asr|lsr}}
+;CHECK-T2-DSP: smlaltt r3, r2,
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-V5TE-NOT: {{asr|lsr}}
+;CHECK-V5TE: smlaltt r3, r2,
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE-NOT: {{asr|lsr}}
+;CHECK-V7-LE: smlaltt r3, r2,
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlaltt r2, r3,
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlaltt
+;CHECK-BE-NOT: smlaltt
+;CHECK-V6M-THUMB-NOT: smlaltt
+;CHECK-V7M-THUMB-NOT: smlaltt
+  %conv0 = ashr i32 %a, 16
+  %conv1 = ashr i32 %b, 16
+  %mul = mul nsw i32 %conv1, %conv0
+  %conv2 = sext i32 %mul to i64
+  %add = add nsw i64 %conv2, %c
+  ret i64 %add
+}
+
+@global_b = external global i16, align 2
+;CHECK-LABEL: MACLongTest15
+;CHECK-T2-DSP-NOT: {{asr|lsr}}
+;CHECK-T2-DSP: smlaltb r3, r2, r0, r1
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-V5TE-NOT: {{asr|lsr}}
+;CHECK-V5TE: smlaltb r3, r2, r0, r1
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE-NOT: {{asr|lsr}}
+;CHECK-V7-LE: smlaltb r3, r2, r0, r1
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlaltb r2, r3, r0, r1
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlaltb
+;CHECK-BE-NOT: smlaltb
+;CHECK-V6M-THUMB-NOT: smlaltb
+;CHECK-V7M-THUMB-NOT: smlaltb
+define i64 @MACLongTest15(i32 %t, i64 %acc) {
+entry:
+  %0 = load i16, i16* @global_b, align 2
+  %conv = sext i16 %0 to i32
+  %shr = ashr i32 %t, 16
+  %mul = mul nsw i32 %shr, %conv
+  %conv1 = sext i32 %mul to i64
+  %add = add nsw i64 %conv1, %acc
+  ret i64 %add
+}
+
+;CHECK-LABEL: MACLongTest16
+;CHECK-T2-DSP-NOT: {{asr|lsr}}
+;CHECK-T2-DSP: smlalbt r3, r2, r1, r0
+;CHECK-T2-DSP-NEXT: mov r0, r3
+;CHECK-T2-DSP-NEXT: mov r1, r2
+;CHECK-V5TE-NOT: {{asr|lsr}}
+;CHECK-V5TE: smlalbt r3, r2, r1, r0
+;CHECK-V5TE-NEXT: mov r0, r3
+;CHECK-V5TE-NEXT: mov r1, r2
+;CHECK-V7-LE: smlalbt r3, r2, r1, r0
+;CHECK-V7-LE-NEXT: mov r0, r3
+;CHECK-V7-LE-NEXT: mov r1, r2
+;CHECK-V7-THUMB-BE: smlalbt r2, r3, r1, r0
+;CHECK-V7-THUMB-BE-NEXT: mov r0, r3
+;CHECK-V7-THUMB-BE-NEXT: mov r1, r2
+;CHECK-LE-NOT: smlalbt
+;CHECK-BE-NOT: smlalbt
+;CHECK-V6M-THUMB-NOT: smlalbt
+;CHECK-V7M-THUMB-NOT: smlalbt
+define i64 @MACLongTest16(i32 %t, i64 %acc) {
+entry:
+  %0 = load i16, i16* @global_b, align 2
+  %conv = sext i16 %0 to i32
+  %shr = ashr i32 %t, 16
+  %mul = mul nsw i32 %conv, %shr
+  %conv1 = sext i32 %mul to i64
+  %add = add nsw i64 %conv1, %acc
+  ret i64 %add
+}