[PowerPC] Add combine logic to use MADDLD/MADDHD/MADDHDU in multiply-add patterns

[tingwang]

Summary

Some i64 multiply-add patterns can use MADDLD/MADDHD/MADDHDU to simplify as shown in this patch.

[shchenz]

llvm/test/CodeGen/PowerPC/adde-sube-int128-madd.ll

4 ↗

(On Diff #453857)

commit the new tests as a NFC patch first please.

[tingwang]

Update test case to show the change.

[tingwang]

llvm/test/CodeGen/PowerPC/adde-sube-int128-madd.ll

4 ↗

(On Diff #453857)

Sure, will do. Updated test case to highlight the delta.

[tingwang]

Move the combine logic in front of type legalization to simplify the pattern.

[lkail]

This comment was removed by lkail.

[tingwang]

The third version has issue that it missed some opportunities that should have been picked up by the second version. For example:

%conv = sext i64 %a to i128
%conv1 = sext i64 %b to i128
%or = or i128 %conv, %conv1
%mul = mul nsw i128 %conv1, %or
%conv2 = sext i64 %c to i128
%add = add nsw i128 %mul, %conv2

It is not obvious to me that %or = or i128 %conv, %conv1 in the context can be represented by sext i64 (or i64 %a, %b) to i128. I'm not aware of any API that can help deduce how many bits are the same as signed bit, so that we can apply truncate here. KnownBits calculates ones and zeros, can work for unsigned case, however cannot help signed case.

After reading some code, realized maybe I can use ComputeNumSignBits to determine signed case.

[tingwang]

(1) Validate pattern by using NumSignBits and SignBitIsZero
(2) Add a test case to show the pattern

[shchenz]

IMO, this version is better than the first one, after the type legalization recognizing the legalized i128 types seems more complicated than recognizing the MADD pattern for type i128 before the type legalization.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

17262

Can we do a std::swap first to put the MUL operand to a settled index? So that we don't need to check MUL operand many times in below code

17281

We may need some false check test cases here, like extending i65 to i128?

17346

Maybe we need to explicitly check this function is called before type legalization(isBeforeLegalize)? BUILD_PAIR is a node which should only be generated before type legalization.

llvm/test/CodeGen/PowerPC/add-sub-int128-madd.ll

4 ↗

(On Diff #457142)

Can we add some cases for other types extension like i32/i16 to i128? I think for these cases, we may don't need maddhd/maddhdu to generate the higher 64 bits of the i128.

[tingwang]

Address comments:
(1) Add isBeforeLegalize() check (updated function scope to allow access to DAGCombinerInfo).
(2) Add test case to show invalid case is not transformed.
(3) Update code structure, reduce duplicated logic.
(4) Add check to identify cases which lead to zero higher half result.

[shchenz]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

17264

I am worried about the swapping for SUB and then adding a SUB(0, swapped_result) to get it back. They are not equal I think.

For example for your below case:

define i128 @sub_int128_CmAxB(i64 noundef %a, i64 noundef %b, i64 noundef %c) local_unnamed_addr #0 {
; CHECK-P9-NEXT:    neg 4, 4
; CHECK-P9-NEXT:    maddld 6, 4, 3, 5
; CHECK-P9-NEXT:    maddhd 4, 4, 3, 5
; CHECK-P9-NEXT:    mr 3, 6
; CHECK-P9-NEXT:    blr
entry:
  %conv = sext i64 %c to i128
  %conv1 = sext i64 %a to i128
  %conv2 = sext i64 %b to i128
  %mul = mul nsw i128 %conv2, %conv1
  %sub = sub nsw i128 %conv, %mul
  ret i128 %sub
}

Suppose %b is the most negative value.

17276

maddld and maddhd are valid for 64 bit integers which include sign bit at the bit-0. So I think here we should expect there are at least 64 + 1= 65 sign bits?
For the 64 sign bits and zero sign bit case, I guess the case is like: we have 0 in all the high 64 bits, and we have 1 in the first bit of the low 64 bits?(Otherwise, the sign bits number must be at least 65?) If so, this is not a case can be handled correctly either. We are expecting zero extension, but the low 64 bits value which can be accessed by maddld/maddhd is a signed value.

17314

This seems overkill. For signed values, cases that some operands are negative while others are positive are still valid.

I think for add(mul(A, B), C),

• if A, B C all have zero sign bit, it is equal to maddhdu + maddld.
• if any of A, B, C can not be proven have zero sign bit, it is equal to maddhd + maddld

[tingwang]

Update according to comments:
(1) Drop sub patterns.
(2) Update check criteria to allow combined signed/unsigned operands as long as NumSignBits >= 65, and is served by madd[l/h]d.
(3) NumSignBits == 64 and SignBitIsZero is the only case served by maddhdu.

All IR test pattern results compared between P8 and P9 by attached script.

madd_gen_test.pl8 KBDownload

[tingwang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

17276

Thank you for pointing out. Updated accordingly.

llvm/lib/Target/PowerPC/PPCISelLowering.h

private:
     SDValue DAGCombineTruncBoolExt(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineStoreFPToInt(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineFPToIntToFP(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSHL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSRA(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSRL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineMUL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineADD(SDNode *N, DAGCombinerInfo &DCI) const;
+    SDValue combineADDToMADD(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineFMALike(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineTRUNCATE(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSetCC(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineABS(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVSelect(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVectorShuffle(ShuffleVectorSDNode *SVN,
                                  SelectionDAG &DAG) const;
     SDValue combineVReverseMemOP(ShuffleVectorSDNode *SVN, LSBaseSDNode *LSBase,

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

static SDValue combineADDToMAT_PCREL_ADDR(SDNode *N, SelectionDAG &DAG,
   SDValue GA =
       DAG.getTargetGlobalAddress(GSDN->getGlobal(), DL, GSDN->getValueType(0),
                                  NewOffset, GSDN->getTargetFlags());
   SDValue MatPCRel =
       DAG.getNode(PPCISD::MAT_PCREL_ADDR, DL, GSDN->getValueType(0), GA);
   return MatPCRel;
 }
 
+// Look for i128 multiply-add opportunities:
+// A, B, C are i128 operands
+// NumSignBits >= 65:
+// (add (mul A B) C) => (build_pair (maddld (trunc A), (trunc B), (trunc C)),
+//                                  (maddhd (trunc A), (trunc B), (trunc C)))
+// NumSignBits == 64 && SignBitIsZero:
+// (add (mul A B) C) => (build_pair (maddld (trunc A), (trunc B), (trunc C)),
+//                                  (maddhdu (trunc A), (trunc B), (trunc C)))
+SDValue PPCTargetLowering::combineADDToMADD(SDNode *N,
+                                            DAGCombinerInfo &DCI) const {
+  SelectionDAG &DAG = DCI.DAG;
+  if (!DCI.isBeforeLegalize())
+    return SDValue();
+
+  if (!Subtarget.isPPC64() || !Subtarget.isISA3_0())
+    return SDValue();
+
+  if (N->getOpcode() != ISD::ADD)
+    return SDValue();
+
+  if (N->getValueType(0) != MVT::i128)
+    return SDValue();
+
+  SDValue MulOp = N->getOperand(0);
+  SDValue AddOp = N->getOperand(1);
+
+  if (MulOp.getOpcode() != ISD::MUL)
+    std::swap(MulOp, AddOp);
+
+  if (MulOp.getOpcode() != ISD::MUL)
+    return SDValue();
+
+  // Do not transform if there are other users of the mul.
+  if (!MulOp.hasOneUse())
+    return SDValue();
+
+  // Operands have at least 65 NumSignBits can be handled by madd signed.
+  // Operands have 64 NumSignBits and proved SignBitIsZero should be handled
+  // by madd unsigned.
+  auto IsLegalOperand = [&DAG](SDValue Op, unsigned &NumSignBits,
+                               bool &IsUnsigned) -> bool {
+    IsUnsigned = false;
+    if (DAG.SignBitIsZero(Op))
+      IsUnsigned = true;
+
+    NumSignBits = DAG.ComputeNumSignBits(Op);
+    // Madd signed case.
+    if (NumSignBits >= 65)
+      return true;
+
+    // Madd unsigned case.
+    if (NumSignBits == 64 && IsUnsigned)
+      return true;
+
+    return false;
+  };
+
+  SmallVector<SDValue, 3> MADDOps;
+  MADDOps.push_back(MulOp->getOperand(0));
+  MADDOps.push_back(MulOp->getOperand(1));
+  MADDOps.push_back(AddOp);
+
+  SmallVector<unsigned, 3> MADDOpsNumSignBits;
+  bool AllUnsigned = true;
+  bool ForceUnsigned = false;
+
+  for (unsigned i = 0; i < 3; ++i) {
+    unsigned OpNumSignBits = 0;
+    bool OpIsUnsigned = false;
+    if (!IsLegalOperand(MADDOps[i], OpNumSignBits, OpIsUnsigned)) {
+      return SDValue();
+    }
+    MADDOpsNumSignBits.push_back(OpNumSignBits);
+    ForceUnsigned |= (OpNumSignBits == 64 && OpIsUnsigned);
+    AllUnsigned &= OpIsUnsigned;
+  }
+
+  // Check for Madd unsigned/signed conflict.
+  if (ForceUnsigned && !AllUnsigned)
+    return SDValue();
+
+  SDLoc dl(N);
+  SDValue Op0 = DAG.getNode(ISD::TRUNCATE, dl, MVT::i64, MADDOps[0]);
+  SDValue Op1 = DAG.getNode(ISD::TRUNCATE, dl, MVT::i64, MADDOps[1]);
+  SDValue Op2 = DAG.getNode(ISD::TRUNCATE, dl, MVT::i64, MADDOps[2]);
+
+  // For unsigned case, in case multiply operands have total less than 64 bits,
+  // and the other add operand has less than 64 bits, then the higher half
+  // result will be zero.
+  bool MAddHIsZero = false;
+  if (AllUnsigned &&
+      (((128 - MADDOpsNumSignBits[0]) + (128 - MADDOpsNumSignBits[1])) < 64) &&
+      ((128 - MADDOpsNumSignBits[2]) < 64))
+    MAddHIsZero = true;
+
+  SDValue MAddL =
+      BuildIntrinsicOp(Intrinsic::ppc_maddld, Op0, Op1, Op2, DAG, dl);
+  SDValue MAddH = MAddHIsZero
+                      ? DAG.getConstant(0, dl, MVT::i64)
+                      : BuildIntrinsicOp(ForceUnsigned ? Intrinsic::ppc_maddhdu
+                                                       : Intrinsic::ppc_maddhd,
+                                         Op0, Op1, Op2, DAG, dl);
+  SDValue Combined = DAG.getNode(ISD::BUILD_PAIR, dl, MVT::i128, MAddL, MAddH);
+  return Combined;
+}
+
 SDValue PPCTargetLowering::combineADD(SDNode *N, DAGCombinerInfo &DCI) const {
   if (auto Value = combineADDToADDZE(N, DCI.DAG, Subtarget))
     return Value;
 
   if (auto Value = combineADDToMAT_PCREL_ADDR(N, DCI.DAG, Subtarget))
     return Value;
 
+  if (auto Value = combineADDToMADD(N, DCI))
+    return Value;
+
   return SDValue();
 }
 
 // Detect TRUNCATE operations on bitcasts of float128 values.
 // What we are looking for here is the situtation where we extract a subset
 // of bits from a 128 bit float.
 // This can be of two forms:
 // 1) BITCAST of f128 feeding TRUNCATE
 // 2) BITCAST of f128 feeding SRL (a shift) feeding TRUNCATE

llvm/test/CodeGen/PowerPC/add-int128-madd.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s --check-prefix=CHECK-P9
 ; RUN: llc -verify-machineinstrs -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s --check-prefix=CHECK-P8
 
 define i128 @add_int64_sext(i64 noundef %a, i64 noundef %b, i64 noundef %c) {
 ; CHECK-P9-LABEL: add_int64_sext:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhd 4, 4, 3
-; CHECK-P9-NEXT:    sradi 7, 5, 63
-; CHECK-P9-NEXT:    addc 3, 6, 5
-; CHECK-P9-NEXT:    adde 4, 4, 7
+; CHECK-P9-NEXT:    maddld 6, 4, 3, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 3, 5
+; CHECK-P9-NEXT:    mr 3, 6
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int64_sext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhd 4, 4, 3
 ; CHECK-P8-NEXT:    sradi 7, 5, 63
 ; CHECK-P8-NEXT:    addc 3, 6, 5
 ; CHECK-P8-NEXT:    adde 4, 4, 7
 ; CHECK-P8-NEXT:    blr
 entry:
   %conv = sext i64 %a to i128
   %conv1 = sext i64 %b to i128
   %mul = mul nsw i128 %conv1, %conv
   %conv2 = sext i64 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_int64_zext(i64 noundef %a, i64 noundef %b, i64 noundef %c) {
 ; CHECK-P9-LABEL: add_int64_zext:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhdu 4, 4, 3
-; CHECK-P9-NEXT:    addc 3, 6, 5
-; CHECK-P9-NEXT:    addze 4, 4
+; CHECK-P9-NEXT:    maddld 6, 4, 3, 5
+; CHECK-P9-NEXT:    maddhdu 4, 4, 3, 5
+; CHECK-P9-NEXT:    mr 3, 6
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int64_zext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhdu 4, 4, 3
 ; CHECK-P8-NEXT:    addc 3, 6, 5
 ; CHECK-P8-NEXT:    addze 4, 4
entry:
   %conv2 = sext i64 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_int64_swap(i64 noundef %a, i64 noundef %b, i64 noundef %c) {
 ; CHECK-P9-LABEL: add_int64_swap:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhd 4, 4, 3
-; CHECK-P9-NEXT:    sradi 7, 5, 63
-; CHECK-P9-NEXT:    addc 3, 5, 6
-; CHECK-P9-NEXT:    adde 4, 7, 4
+; CHECK-P9-NEXT:    maddld 6, 4, 3, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 3, 5
+; CHECK-P9-NEXT:    mr 3, 6
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int64_swap:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhd 4, 4, 3
 ; CHECK-P8-NEXT:    sradi 7, 5, 63
 ; CHECK-P8-NEXT:    addc 3, 5, 6
 ; CHECK-P8-NEXT:    adde 4, 7, 4
 ; CHECK-P8-NEXT:    blr
 entry:
   %conv = sext i64 %a to i128
   %conv1 = sext i64 %b to i128
   %mul = mul nsw i128 %conv1, %conv
   %conv2 = sext i64 %c to i128
   %add = add nsw i128 %conv2, %mul
   ret i128 %add
 }
 
 define i128 @add_mix_zext(i63 noundef %a, i63 noundef %b, i64 noundef %c) {
 ; CHECK-P9-LABEL: add_mix_zext:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    clrldi 3, 3, 1
+; CHECK-P9-NEXT:    clrldi 6, 3, 1
 ; CHECK-P9-NEXT:    clrldi 4, 4, 1
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhdu 4, 4, 3
-; CHECK-P9-NEXT:    addc 3, 6, 5
-; CHECK-P9-NEXT:    addze 4, 4
+; CHECK-P9-NEXT:    maddld 3, 4, 6, 5
+; CHECK-P9-NEXT:    maddhdu 4, 4, 6, 5
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_mix_zext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    clrldi 3, 3, 1
 ; CHECK-P8-NEXT:    clrldi 4, 4, 1
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhdu 4, 4, 3
 ; CHECK-P8-NEXT:    addc 3, 6, 5
 ; CHECK-P8-NEXT:    addze 4, 4
 ; CHECK-P8-NEXT:    blr
 entry:
   %conv = zext i63 %a to i128
   %conv1 = zext i63 %b to i128
   %mul = mul nsw i128 %conv1, %conv
   %conv2 = zext i64 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_int63_sext(i63 noundef %a, i63 noundef %b, i63 noundef %c) {
 ; CHECK-P9-LABEL: add_int63_sext:
 ; CHECK-P9:       # %bb.0: # %entry
 ; CHECK-P9-NEXT:    sldi 3, 3, 1
-; CHECK-P9-NEXT:    sldi 4, 4, 1
-; CHECK-P9-NEXT:    sradi 3, 3, 1
-; CHECK-P9-NEXT:    sradi 4, 4, 1
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhd 4, 4, 3
-; CHECK-P9-NEXT:    sldi 3, 5, 1
-; CHECK-P9-NEXT:    sradi 5, 3, 1
-; CHECK-P9-NEXT:    sradi 7, 3, 63
-; CHECK-P9-NEXT:    addc 3, 6, 5
-; CHECK-P9-NEXT:    adde 4, 4, 7
+; CHECK-P9-NEXT:    sldi 5, 5, 1
+; CHECK-P9-NEXT:    sradi 6, 3, 1
+; CHECK-P9-NEXT:    sldi 3, 4, 1
+; CHECK-P9-NEXT:    sradi 5, 5, 1
+; CHECK-P9-NEXT:    sradi 4, 3, 1
+; CHECK-P9-NEXT:    maddld 3, 4, 6, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 6, 5
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int63_sext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    sldi 3, 3, 1
 ; CHECK-P8-NEXT:    sldi 4, 4, 1
 ; CHECK-P8-NEXT:    sradi 3, 3, 1
 ; CHECK-P8-NEXT:    sradi 4, 4, 1
entry:
   %conv2 = sext i63 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_int63_zext(i63 noundef %a, i63 noundef %b, i63 noundef %c) {
 ; CHECK-P9-LABEL: add_int63_zext:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    clrldi 3, 3, 1
+; CHECK-P9-NEXT:    clrldi 5, 5, 1
+; CHECK-P9-NEXT:    clrldi 6, 3, 1
 ; CHECK-P9-NEXT:    clrldi 4, 4, 1
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhdu 4, 4, 3
-; CHECK-P9-NEXT:    clrldi 3, 5, 1
-; CHECK-P9-NEXT:    addc 3, 6, 3
-; CHECK-P9-NEXT:    addze 4, 4
+; CHECK-P9-NEXT:    maddld 3, 4, 6, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 6, 5
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int63_zext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    clrldi 3, 3, 1
 ; CHECK-P8-NEXT:    clrldi 4, 4, 1
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhdu 4, 4, 3
 ; CHECK-P8-NEXT:    clrldi 3, 5, 1
 ; CHECK-P8-NEXT:    addc 3, 6, 3
 ; CHECK-P8-NEXT:    addze 4, 4
 ; CHECK-P8-NEXT:    blr
 entry:
   %conv = zext i63 %a to i128
   %conv1 = zext i63 %b to i128
   %mul = mul nsw i128 %conv1, %conv
   %conv2 = zext i63 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_int63_hybrid_ext(i63 noundef %a, i63 noundef %b, i63 noundef %c) {
 ; CHECK-P9-LABEL: add_int63_hybrid_ext:
 ; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    sldi 5, 5, 1
 ; CHECK-P9-NEXT:    sldi 3, 3, 1
+; CHECK-P9-NEXT:    sradi 5, 5, 1
 ; CHECK-P9-NEXT:    clrldi 4, 4, 1
-; CHECK-P9-NEXT:    sradi 3, 3, 1
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhd 4, 4, 3
-; CHECK-P9-NEXT:    sldi 3, 5, 1
-; CHECK-P9-NEXT:    sradi 5, 3, 1
-; CHECK-P9-NEXT:    sradi 7, 3, 63
-; CHECK-P9-NEXT:    addc 3, 6, 5
-; CHECK-P9-NEXT:    adde 4, 4, 7
+; CHECK-P9-NEXT:    sradi 6, 3, 1
+; CHECK-P9-NEXT:    maddld 3, 4, 6, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 6, 5
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_int63_hybrid_ext:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    sldi 3, 3, 1
 ; CHECK-P8-NEXT:    clrldi 4, 4, 1
 ; CHECK-P8-NEXT:    sradi 3, 3, 1
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
entry:
   %conv2 = sext i64 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_highDWZero(i32 noundef %a, i31 noundef %b, i63 noundef %c) {
 ; CHECK-P9-LABEL: add_highDWZero:
 ; CHECK-P9:       # %bb.0: # %entry
+; CHECK-P9-NEXT:    clrldi 5, 5, 1
 ; CHECK-P9-NEXT:    clrldi 3, 3, 32
 ; CHECK-P9-NEXT:    clrldi 4, 4, 33
-; CHECK-P9-NEXT:    clrldi 5, 5, 1
 ; CHECK-P9-NEXT:    maddld 3, 4, 3, 5
 ; CHECK-P9-NEXT:    li 4, 0
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_highDWZero:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    clrldi 3, 3, 32
 ; CHECK-P8-NEXT:    clrldi 4, 4, 33
entry:
   %conv2 = zext i63 %c to i128
   %add = add nsw i128 %mul, %conv2
   ret i128 %add
 }
 
 define i128 @add_sext_highDWNonZero(i16 noundef %a, i16 noundef %b, i16 noundef %c) {
 ; CHECK-P9-LABEL: add_sext_highDWNonZero:
 ; CHECK-P9:       # %bb.0: # %entry
-; CHECK-P9-NEXT:    extsh 3, 3
+; CHECK-P9-NEXT:    extsh 5, 5
+; CHECK-P9-NEXT:    extsh 6, 3
 ; CHECK-P9-NEXT:    extsh 4, 4
-; CHECK-P9-NEXT:    mulld 6, 4, 3
-; CHECK-P9-NEXT:    mulhd 4, 4, 3
-; CHECK-P9-NEXT:    extsh 3, 5
-; CHECK-P9-NEXT:    sradi 5, 3, 63
-; CHECK-P9-NEXT:    addc 3, 6, 3
-; CHECK-P9-NEXT:    adde 4, 4, 5
+; CHECK-P9-NEXT:    maddld 3, 4, 6, 5
+; CHECK-P9-NEXT:    maddhd 4, 4, 6, 5
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: add_sext_highDWNonZero:
 ; CHECK-P8:       # %bb.0: # %entry
 ; CHECK-P8-NEXT:    extsh 3, 3
 ; CHECK-P8-NEXT:    extsh 4, 4
 ; CHECK-P8-NEXT:    mulld 6, 4, 3
 ; CHECK-P8-NEXT:    mulhd 4, 4, 3

llvm/test/CodeGen/PowerPC/mulld.ll

 ; RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple=powerpc64-unknown-linux-gnu < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s
 ; RUN: llc -verify-machineinstrs -mcpu=pwr8 -mtriple=powerpc64-unknown-linux-gnu < %s | FileCheck %s \
 ; RUN: --check-prefix=CHECK-ITIN
 ; RUN: llc -verify-machineinstrs -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s \
 ; RUN: --check-prefix=CHECK-ITIN
 ; RUN: llc -verify-machineinstrs -mcpu=pwr7 -mtriple=powerpc64-unknown-linux-gnu < %s | FileCheck %s \
 ; RUN: --check-prefix=CHECK-ITIN
 
 define void @bn_mul_comba8(ptr nocapture %r, ptr nocapture readonly %a, ptr nocapture readonly %b) {
 ; CHECK-LABEL: bn_mul_comba8:
 ; CHECK:    mulhdu
-; CHECK-NEXT:    mulld
-; CHECK:         mulhdu
-; CHECK:         mulld
-; CHECK-NEXT:    mulhdu
+; CHECK:    maddhdu
+; CHECK-NEXT:    maddld
+; CHECK:    maddhdu
 
 
 ; CHECK-ITIN-LABEL: bn_mul_comba8:
 ; CHECK-ITIN:    mulhdu
 ; CHECK-ITIN-NEXT:    mulld
 ; CHECK-ITIN-NEXT:    mulhdu
 ; CHECK-ITIN-NEXT:    mulld
 ; CHECK-ITIN-NEXT:    mulhdu