[X86] Machine combine vnni instruction.

[LuoYuanke]

Summary

"vpmaddwd + vpaddd" can be combined to vpdpwssd and the latency is
reduced after combination. However when vpdpwssd is in a critical path
the combination get less ILP. It happens when vpdpwssd is in a loop, the
vpmaddwd can be executed in parallel in multi-iterations while vpdpwssd
has data dependency for each iterations. If vpaddd is in a critical path
while vpmaddwd is not, it is profitable to split vpdpwssd into "vpmaddwd
+ vpaddd ".
This patch is based on the machine combiner framework to acheive decision
on "vpmaddwd + vpaddd" combination. The typical example code is as
below.

__m256i foo(int cnt, __m256i c, __m256i b, __m256i *p) {

    for (int i = 0; i < cnt; ++i) {
        __m256i a = p[i];
        __m256i m = _mm256_madd_epi16 (b, a);
        c = _mm256_add_epi32(m, c);
    }

    return c;
}

[LuoYuanke]

I'll add more patterns if it is the right way to solve https://github.com/llvm/llvm-project/issues/62026.

[goldstein.w.n]

llvm/lib/CodeGen/MachineCombiner.cpp

173

What is this change for?

427

This seems like an unrelated change. Can you split it?

llvm/lib/Target/X86/X86InstrInfo.cpp

9823

Maybe this belongs in X86FixupInstTuning.cpp?

llvm/test/CodeGen/X86/avxvnni-combine.ll

4

Can you precommit the test so we can see the diff?

[LuoYuanke]

Fix the AArch64 test case failure.

[LuoYuanke]

llvm/lib/CodeGen/MachineCombiner.cpp

173

For some reason the new created register for new created instructions is not managed by MRI, so it would miss adding the DefInstr's latency. I guess may be due to the new instructions are not inserted in the machine function. I'll debug more on it.

llvm/lib/Target/X86/X86InstrInfo.cpp

9823

The method is to override the virtual function of TargetInstrInfo.
X86FixupInstTuning.cpp may be integrated to MachineCombine.

llvm/test/CodeGen/X86/avxvnni-combine.ll

4

Sure, I'll precommit the test case first.

[LuoYuanke]

Rebase

[RKSimon]

llvm/test/CodeGen/X86/avxvnni-combine.ll

2–291

Add AVX512-VNNI support as well?

300

Please can you add test coverage for a case where there isn't the cross-loop dependency?

auto bar(int cnt, __m256i *c, __m256i b, __m256i *p) {
    for (int i = 0; i < cnt; ++i) {
        __m256i a = p[i];
        __m256i m = _mm256_madd_epi16 (b, a);
        c[i] = _mm256_add_epi32(m, c[i]);
    }
}

[LuoYuanke]

Add test case that there isn't the cross-loop dependency.

[LuoYuanke]

llvm/test/CodeGen/X86/avxvnni-combine.ll

429

vpmaddwd and vmovdqa (line 159) can be issued in parallel.

[goldstein.w.n]

llvm/lib/Target/X86/X86InstrInfo.cpp

9823

It seems.you are just querying for an opcode and replacing 1-1? Not that it can only be done in X86Fixup... But seems to fit there with less code.

[craig.topper]

llvm/lib/Target/X86/X86InstrInfo.cpp

9823

It’s replacing 1 instruction with 2 after doing analysis that consults the scheduling model.

[LuoYuanke]

Add (avx512, 128, 256, 512) support.

[LuoYuanke]

Update test case.

[LuoYuanke]

Fix an opcode bug.

[craig.topper]

llvm/lib/CodeGen/MachineCombiner.cpp

426–432

Put curly braces around the if body for consistency with the else body.

llvm/lib/Target/X86/X86InstrInfo.cpp

9791

The Vp prefix on these variable names isn't providing much value. You can probably drop it.

[LuoYuanke]

Address Craig's comments.

[LuoYuanke]

Update description

[LuoYuanke]

Rebase

[LuoYuanke]

ping

[pengfei]

LGTM with some nits.

llvm/lib/Target/X86/X86InstrInfo.cpp

9797–9800

indent.

I think we don't need to duplicate for ymm and zmm. And maybe better to choose rr version as example.

9851

Avoid auto

llvm/lib/Target/X86/X86InstrInfo.h

605

Nit: Should copy+paste the comment here like others.

610

ditto.

[LuoYuanke]

Address Phoebe's comments.

[RKSimon]

@LuoYuanke I'm seeing tests failures on my local machine with expensive checks enabled - one buildbot is reporting something similar: https://lab.llvm.org/buildbot/#/builders/16/builds/47297

[LuoYuanke]

In D148980#4305430, @RKSimon wrote:

@LuoYuanke I'm seeing tests failures on my local machine with expensive checks enabled - one buildbot is reporting something similar: https://lab.llvm.org/buildbot/#/builders/16/builds/47297

Thanks, @RKSimon for notifying. I'll revert the patch first.

[LuoYuanke]

There is conflict when I try to revert the patch. I fixed the issue with rG40222ddcf8f5.

llvm/include/llvm/CodeGen/MachineCombinerPattern.h

enum class MachineCombinerPattern {
   FMULv8i16_indexed_OP1,
   FMULv8i16_indexed_OP2,
 
   // RISCV FMADD, FMSUB, FNMSUB patterns
   FMADD_AX,
   FMADD_XA,
   FMSUB,
   FNMSUB,
+
+  // X86 VNNI
+  DPWSSD,
 };
 
 } // end namespace llvm
 
 #endif

llvm/include/llvm/CodeGen/TargetInstrInfo.h

public:
   /// \param InstIdxForVirtReg - map of virtual register to instruction in
   /// InsInstr that defines it
   virtual void genAlternativeCodeSequence(
       MachineInstr &Root, MachineCombinerPattern Pattern,
       SmallVectorImpl<MachineInstr *> &InsInstrs,
       SmallVectorImpl<MachineInstr *> &DelInstrs,
       DenseMap<unsigned, unsigned> &InstIdxForVirtReg) const;
 
+  /// When calculate the latency of the root instruction, accumulate the
+  /// latency of the sequence to the root latency.
+  /// \param Root - Instruction that could be combined with one of its operands
+  virtual bool accumulateInstrSeqToRootLatency(MachineInstr &Root) const {
+    return true;
+  }
+
   /// Attempt to reassociate \P Root and \P Prev according to \P Pattern to
   /// reduce critical path length.
   void reassociateOps(MachineInstr &Root, MachineInstr &Prev,
                       MachineCombinerPattern Pattern,
                       SmallVectorImpl<MachineInstr *> &InsInstrs,
                       SmallVectorImpl<MachineInstr *> &DelInstrs,
                       DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const;
 

llvm/lib/CodeGen/MachineCombiner.cpp

MachineCombiner() : MachineFunctionPass(ID) {
     initializeMachineCombinerPass(*PassRegistry::getPassRegistry());
   }
   void getAnalysisUsage(AnalysisUsage &AU) const override;
   bool runOnMachineFunction(MachineFunction &MF) override;
   StringRef getPassName() const override { return "Machine InstCombiner"; }
 
 private:
   bool combineInstructions(MachineBasicBlock *);
-  MachineInstr *getOperandDef(const MachineOperand &MO);
+  MachineInstr *getOperandDef(const MachineOperand &MO,
+                              SmallVectorImpl<MachineInstr *> &InsInstrs);
   bool isTransientMI(const MachineInstr *MI);
   unsigned getDepth(SmallVectorImpl<MachineInstr *> &InsInstrs,
                     DenseMap<unsigned, unsigned> &InstrIdxForVirtReg,
                     MachineTraceMetrics::Trace BlockTrace,
                     const MachineBasicBlock &MBB);
   unsigned getLatency(MachineInstr *Root, MachineInstr *NewRoot,
                       MachineTraceMetrics::Trace BlockTrace);
   bool
void MachineCombiner::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addPreserved<MachineLoopInfo>();
   AU.addRequired<MachineTraceMetrics>();
   AU.addPreserved<MachineTraceMetrics>();
   AU.addRequired<LazyMachineBlockFrequencyInfoPass>();
   AU.addRequired<ProfileSummaryInfoWrapperPass>();
   MachineFunctionPass::getAnalysisUsage(AU);
 }
 
-MachineInstr *MachineCombiner::getOperandDef(const MachineOperand &MO) {
+MachineInstr *
+MachineCombiner::getOperandDef(const MachineOperand &MO,
+                               SmallVectorImpl<MachineInstr *> &InsInstrs) {
   MachineInstr *DefInstr = nullptr;
   // We need a virtual register definition.
   if (MO.isReg() && MO.getReg().isVirtual())
     DefInstr = MRI->getUniqueVRegDef(MO.getReg());
+  // Since the new instructions are not inserted into the machine function,
+  // the def-use information is not added in MRI. So it is possible that
+  // the register is defined in new instructions.
+  if (!DefInstr) {
+    for (auto *MI : InsInstrs) {
+      for (const MachineOperand &DefMO : MI->operands()) {
+        if (!(DefMO.isReg() && DefMO.getReg().isVirtual()))
+          continue;
+        if (!DefMO.isDef())
+          continue;
+        if (DefMO.getReg() != MO.getReg())
+          continue;
+        DefInstr = MI;
+      }
+    }
+  }
   // PHI's have no depth etc.
   if (DefInstr && DefInstr->isPHI())
     DefInstr = nullptr;
   return DefInstr;
 }
 
 /// Return true if MI is unlikely to generate an actual target instruction.
 bool MachineCombiner::isTransientMI(const MachineInstr *MI) {
for (const MachineOperand &MO : InstrPtr->operands()) {
         assert(DefInstr &&
                "There must be a definition for a new virtual register");
         DepthOp = InstrDepth[II->second];
         int DefIdx = DefInstr->findRegisterDefOperandIdx(MO.getReg());
         int UseIdx = InstrPtr->findRegisterUseOperandIdx(MO.getReg());
         LatencyOp = TSchedModel.computeOperandLatency(DefInstr, DefIdx,
                                                       InstrPtr, UseIdx);
       } else {
-        MachineInstr *DefInstr = getOperandDef(MO);
+        MachineInstr *DefInstr = getOperandDef(MO, InsInstrs);
         if (DefInstr && (TII->getMachineCombinerTraceStrategy() !=
                              MachineTraceStrategy::TS_Local ||
                          DefInstr->getParent() == &MBB)) {
           DepthOp = BlockTrace.getInstrCycles(*DefInstr).Depth;
           if (!isTransientMI(DefInstr))
             LatencyOp = TSchedModel.computeOperandLatency(
                 DefInstr, DefInstr->findRegisterDefOperandIdx(MO.getReg()),
                 InstrPtr, InstrPtr->findRegisterUseOperandIdx(MO.getReg()));
bool MachineCombiner::improvesCriticalPathLen(
   }
 
   // A more flexible cost calculation for the critical path includes the slack
   // of the original code sequence. This may allow the transform to proceed
   // even if the instruction depths (data dependency cycles) become worse.
 
   // Account for the latency of the inserted and deleted instructions by
   unsigned NewRootLatency, RootLatency;
+  if (TII->accumulateInstrSeqToRootLatency(*Root))
     std::tie(NewRootLatency, RootLatency) =
         getLatenciesForInstrSequences(*Root, InsInstrs, DelInstrs, BlockTrace);
+  else {
+    NewRootLatency = TSchedModel.computeInstrLatency(InsInstrs.back());
+    RootLatency = TSchedModel.computeInstrLatency(Root);
+  }
 
   unsigned RootSlack = BlockTrace.getInstrSlack(*Root);
   unsigned NewCycleCount = NewRootDepth + NewRootLatency;
   unsigned OldCycleCount =
       RootDepth + RootLatency + (SlackIsAccurate ? RootSlack : 0);
   LLVM_DEBUG(dbgs() << "\n\tNewRootLatency: " << NewRootLatency
                     << "\tRootLatency: " << RootLatency << "\n\tRootSlack: "
                     << RootSlack << " SlackIsAccurate=" << SlackIsAccurate

llvm/lib/Target/X86/X86InstrInfo.h

MachineInstr *commuteInstructionImpl(MachineInstr &MI, bool NewMI,
                                        unsigned CommuteOpIdx2) const override;
 
   /// If the specific machine instruction is a instruction that moves/copies
   /// value from one register to another register return destination and source
   /// registers as machine operands.
   std::optional<DestSourcePair>
   isCopyInstrImpl(const MachineInstr &MI) const override;
 
+  bool
+  getMachineCombinerPatterns(MachineInstr &Root,
+                             SmallVectorImpl<MachineCombinerPattern> &Patterns,
+                             bool DoRegPressureReduce) const override;
+
+  void genAlternativeCodeSequence(
+      MachineInstr &Root, MachineCombinerPattern Pattern,
+      SmallVectorImpl<MachineInstr *> &InsInstrs,
+      SmallVectorImpl<MachineInstr *> &DelInstrs,
+      DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const override;
+
+  /// When calculate the latency of the root instruction, accumulate the
+  /// latency of the sequence to the root latency.
+  /// \param Root - Instruction that could be combined with one of its operands
+  /// For X86 instruction (vpmaddwd + vpmaddwd) -> vpdpwssd, the vpmaddwd
+  /// is not in the critical path, so the root latency only include vpmaddwd.
+  bool accumulateInstrSeqToRootLatency(MachineInstr &Root) const override {
+    return false;
+  }
+
 private:
   /// This is a helper for convertToThreeAddress for 8 and 16-bit instructions.
   /// We use 32-bit LEA to form 3-address code by promoting to a 32-bit
   /// super-register and then truncating back down to a 8/16-bit sub-register.
   MachineInstr *convertToThreeAddressWithLEA(unsigned MIOpc, MachineInstr &MI,
                                              LiveVariables *LV,
                                              LiveIntervals *LIS,
                                              bool Is8BitOp) const;

llvm/lib/Target/X86/X86InstrInfo.cpp

 #include "X86MachineFunctionInfo.h"
 #include "X86Subtarget.h"
 #include "X86TargetMachine.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Sequence.h"
 #include "llvm/CodeGen/LiveIntervals.h"
 #include "llvm/CodeGen/LivePhysRegs.h"
 #include "llvm/CodeGen/LiveVariables.h"
+#include "llvm/CodeGen/MachineCombinerPattern.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
if (C.CallConstructionID == MachineOutlinerTailCall) {
     It = MBB.insert(It,
                   BuildMI(MF, DebugLoc(), get(X86::CALL64pcrel32))
                       .addGlobalAddress(M.getNamedValue(MF.getName())));
   }
 
   return It;
 }
 
+bool X86InstrInfo::getMachineCombinerPatterns(
+    MachineInstr &Root, SmallVectorImpl<MachineCombinerPattern> &Patterns,
+    bool DoRegPressureReduce) const {
+  unsigned Opc = Root.getOpcode();
+  switch (Opc) {
+  default:
+    return TargetInstrInfo::getMachineCombinerPatterns(Root, Patterns,
+                                                       DoRegPressureReduce);
+  case X86::VPDPWSSDrr:
+  case X86::VPDPWSSDrm:
+  case X86::VPDPWSSDYrr:
+  case X86::VPDPWSSDYrm: {
+    Patterns.push_back(MachineCombinerPattern::DPWSSD);
+    return true;
+  }
+  case X86::VPDPWSSDZ128r:
+  case X86::VPDPWSSDZ128m:
+  case X86::VPDPWSSDZ256r:
+  case X86::VPDPWSSDZ256m:
+  case X86::VPDPWSSDZr:
+  case X86::VPDPWSSDZm: {
+    if (Subtarget.hasBWI())
+      Patterns.push_back(MachineCombinerPattern::DPWSSD);
+    return true;
+  }
+  }
+}
+
+static void
+genAlternativeDpCodeSequence(MachineInstr &Root, const TargetInstrInfo &TII,
+                             SmallVectorImpl<MachineInstr *> &InsInstrs,
+                             SmallVectorImpl<MachineInstr *> &DelInstrs,
+                             DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) {
+  MachineFunction *MF = Root.getMF();
+  MachineRegisterInfo &RegInfo = MF->getRegInfo();
+
+  unsigned Opc = Root.getOpcode();
+  unsigned VpAddOpc;
+  unsigned VpMaddOpc;
+  switch (Opc) {
+  default:
+    assert("It should not reach here");
+    break;
+    // vpdpwssd xmm2,xmm3,XMMWORD PTR [r8+0x20]
+    // -->
+    // vpmaddwd xmm3,xmm3,XMMWORD PTR [r8+0x20]
+    // vpaddd xmm2,xmm2,xmm3
+  case X86::VPDPWSSDrr:
+    VpMaddOpc = X86::VPMADDWDrr;
+    VpAddOpc = X86::VPADDDrr;
+    break;
+  case X86::VPDPWSSDrm:
+    VpMaddOpc = X86::VPMADDWDrm;
+    VpAddOpc = X86::VPADDDrr;
+    break;
+  case X86::VPDPWSSDZ128r:
+    VpMaddOpc = X86::VPMADDWDZ128rr;
+    VpAddOpc = X86::VPADDDZ128rr;
+    break;
+  case X86::VPDPWSSDZ128m:
+    VpMaddOpc = X86::VPMADDWDYrm;
+    VpAddOpc = X86::VPADDDZ128rr;
+    break;
+    // vpdpwssd ymm2,ymm3,YMMWORD PTR [r8+0x20]
+    // -->
+    // vpmaddwd ymm3,ymm3,YMMWORD PTR [r8+0x20]
+    // vpaddd ymm2,ymm2,ymm3
+  case X86::VPDPWSSDYrr:
+    VpMaddOpc = X86::VPMADDWDYrr;
+    VpAddOpc = X86::VPADDDYrr;
+    break;
+  case X86::VPDPWSSDYrm:
+    VpMaddOpc = X86::VPMADDWDYrm;
+    VpAddOpc = X86::VPADDDYrr;
+    break;
+  case X86::VPDPWSSDZ256r:
+    VpMaddOpc = X86::VPMADDWDZ256rr;
+    VpAddOpc = X86::VPADDDZ256rr;
+    break;
+  case X86::VPDPWSSDZ256m:
+    VpMaddOpc = X86::VPMADDWDZ256rm;
+    VpAddOpc = X86::VPADDDZ256rr;
+    break;
+    // vpdpwssd zmm2,zmm3,ZMMWORD PTR [r8+0x20]
+    // -->
+    // vpmaddwd zmm3,zmm3,ZMMWORD PTR [r8+0x20]
+    // vpaddd zmm2,zmm2,zmm3
+  case X86::VPDPWSSDZr:
+    VpMaddOpc = X86::VPMADDWDZrr;
+    VpAddOpc = X86::VPADDDZrr;
+    break;
+  case X86::VPDPWSSDZm:
+    VpMaddOpc = X86::VPMADDWDZrm;
+    VpAddOpc = X86::VPADDDZrr;
+    break;
+  }
+  // Create vpmaddwd.
+  auto *RC = RegInfo.getRegClass(Root.getOperand(0).getReg());
+  Register NewReg = RegInfo.createVirtualRegister(RC);
+  MachineInstr *VpMadd = Root.getMF()->CloneMachineInstr(&Root);
+  VpMadd->setDesc(TII.get(VpMaddOpc));
+  VpMadd->untieRegOperand(1);
+  VpMadd->removeOperand(1);
+  VpMadd->getOperand(0).setReg(NewReg);
+  // Create vpaddd.
+  Register DstReg = Root.getOperand(0).getReg();
+  bool IsKill = Root.getOperand(1).isKill();
+  MachineInstr *VpAdd =
+      BuildMI(*MF, MIMetadata(Root), TII.get(VpAddOpc), DstReg)
+          .addReg(Root.getOperand(1).getReg(), getKillRegState(IsKill))
+          .addReg(VpMadd->getOperand(0).getReg(), getKillRegState(true));
+  InstrIdxForVirtReg.insert(std::make_pair(DstReg, 0));
+  InsInstrs.push_back(VpMadd);
+  InsInstrs.push_back(VpAdd);
+  DelInstrs.push_back(&Root);
+}
+
+void X86InstrInfo::genAlternativeCodeSequence(
+    MachineInstr &Root, MachineCombinerPattern Pattern,
+    SmallVectorImpl<MachineInstr *> &InsInstrs,
+    SmallVectorImpl<MachineInstr *> &DelInstrs,
+    DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const {
+  switch (Pattern) {
+  default:
+    // Reassociate instructions.
+    TargetInstrInfo::genAlternativeCodeSequence(Root, Pattern, InsInstrs,
+                                                DelInstrs, InstrIdxForVirtReg);
+    return;
+  case MachineCombinerPattern::DPWSSD:
+    genAlternativeDpCodeSequence(Root, *this, InsInstrs, DelInstrs,
+                                 InstrIdxForVirtReg);
+    return;
+  }
+}
+
 #define GET_INSTRINFO_HELPERS
 #include "X86GenInstrInfo.inc"

llvm/test/CodeGen/X86/avx512vnni-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=sapphirerapids | FileCheck %s
 
 define <8 x i64> @foo_reg_512(<8 x i64> %0, <8 x i64> %1, <8 x i64> %2, <8 x i64> %3, <8 x i64> %4, <8 x i64> %5) {
 ; CHECK-LABEL: foo_reg_512:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vpdpwssd %zmm2, %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd %zmm3, %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd %zmm4, %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd %zmm5, %zmm1, %zmm0
+; CHECK-NEXT:    vpmaddwd %zmm3, %zmm1, %zmm2
+; CHECK-NEXT:    vpaddd %zmm2, %zmm0, %zmm0
+; CHECK-NEXT:    vpmaddwd %zmm4, %zmm1, %zmm2
+; CHECK-NEXT:    vpaddd %zmm2, %zmm0, %zmm0
+; CHECK-NEXT:    vpmaddwd %zmm5, %zmm1, %zmm1
+; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
 ; CHECK-NEXT:    retq
   %7 = bitcast <8 x i64> %0 to <16 x i32>
   %8 = bitcast <8 x i64> %1 to <16 x i32>
   %9 = bitcast <8 x i64> %2 to <16 x i32>
   %10 = tail call <16 x i32> @llvm.x86.avx512.vpdpwssd.512(<16 x i32> %7, <16 x i32> %8, <16 x i32> %9)
   %11 = bitcast <8 x i64> %3 to <16 x i32>
   %12 = tail call <16 x i32> @llvm.x86.avx512.vpdpwssd.512(<16 x i32> %10, <16 x i32> %8, <16 x i32> %11)
   %13 = bitcast <8 x i64> %4 to <16 x i32>

 ; CHECK-NEXT:    jmp .LBB1_3
 ; CHECK-NEXT:  .LBB1_7:
 ; CHECK-NEXT:    andl $-4, %edx
 ; CHECK-NEXT:    leaq 192(%rsi), %rdi
 ; CHECK-NEXT:    xorl %ecx, %ecx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB1_8: # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vpdpwssd -192(%rdi), %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd -128(%rdi), %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd -64(%rdi), %zmm1, %zmm0
-; CHECK-NEXT:    vpdpwssd (%rdi), %zmm1, %zmm0
+; CHECK-NEXT:    vpmaddwd -128(%rdi), %zmm1, %zmm2
+; CHECK-NEXT:    vpaddd %zmm2, %zmm0, %zmm0
+; CHECK-NEXT:    vpmaddwd -64(%rdi), %zmm1, %zmm2
+; CHECK-NEXT:    vpaddd %zmm2, %zmm0, %zmm0
+; CHECK-NEXT:    vpmaddwd (%rdi), %zmm1, %zmm2
+; CHECK-NEXT:    vpaddd %zmm2, %zmm0, %zmm0
 ; CHECK-NEXT:    addq $4, %rcx
 ; CHECK-NEXT:    addq $256, %rdi # imm = 0x100
 ; CHECK-NEXT:    cmpq %rcx, %rdx
 ; CHECK-NEXT:    jne .LBB1_8
 ; CHECK-NEXT:  .LBB1_3:
 ; CHECK-NEXT:    testq %rax, %rax
 ; CHECK-NEXT:    je .LBB1_6
 ; CHECK-NEXT:  # %bb.4: # %.preheader

 ; CHECK-NEXT:    movl $64, %r8d
 ; CHECK-NEXT:    xorl %ecx, %ecx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB2_7: # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vmovdqa64 -64(%rsi,%r8), %zmm1
 ; CHECK-NEXT:    vmovdqa64 (%rsi,%r8), %zmm2
 ; CHECK-NEXT:    vpdpwssd -64(%rdx,%r8), %zmm0, %zmm1
 ; CHECK-NEXT:    vmovdqa64 %zmm1, -64(%rsi,%r8)
-; CHECK-NEXT:    vpdpwssd (%rdx,%r8), %zmm0, %zmm2
-; CHECK-NEXT:    vmovdqa64 %zmm2, (%rsi,%r8)
+; CHECK-NEXT:    vpmaddwd (%rdx,%r8), %zmm0, %zmm1
+; CHECK-NEXT:    vpaddd %zmm1, %zmm2, %zmm1
+; CHECK-NEXT:    vmovdqa64 %zmm1, (%rsi,%r8)
 ; CHECK-NEXT:    addq $2, %rcx
 ; CHECK-NEXT:    subq $-128, %r8
 ; CHECK-NEXT:    cmpq %rcx, %rdi
 ; CHECK-NEXT:    jne .LBB2_7
 ; CHECK-NEXT:  .LBB2_3:
 ; CHECK-NEXT:    testb $1, %al
 ; CHECK-NEXT:    je .LBB2_5
 ; CHECK-NEXT:  # %bb.4:
 ; CHECK-NEXT:    shlq $6, %rcx
-; CHECK-NEXT:    vmovdqa64 (%rsi,%rcx), %zmm1
-; CHECK-NEXT:    vpdpwssd (%rdx,%rcx), %zmm0, %zmm1
-; CHECK-NEXT:    vmovdqa64 %zmm1, (%rsi,%rcx)
+; CHECK-NEXT:    vpmaddwd (%rdx,%rcx), %zmm0, %zmm0
+; CHECK-NEXT:    vpaddd (%rsi,%rcx), %zmm0, %zmm0
+; CHECK-NEXT:    vmovdqa64 %zmm0, (%rsi,%rcx)
 ; CHECK-NEXT:  .LBB2_5:
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
   %5 = icmp sgt i32 %0, 0
   br i1 %5, label %6, label %22
 
 6:                                                ; preds = %4
   %7 = bitcast <8 x i64> %2 to <16 x i32>

llvm/test/CodeGen/X86/avxvnni-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=alderlake | FileCheck %s --check-prefixes=AVX
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=sapphirerapids | FileCheck %s --check-prefixes=AVX
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=icelake-server | FileCheck %s --check-prefixes=AVX512
 
 define <2 x i64> @foo_reg_128(<2 x i64> %0, <2 x i64> %1, <2 x i64> %2, <2 x i64> %3, <2 x i64> %4, <2 x i64> %5) {
 ; AVX-LABEL: foo_reg_128:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    {vex} vpdpwssd %xmm2, %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd %xmm3, %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd %xmm4, %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd %xmm5, %xmm1, %xmm0
+; AVX-NEXT:    vpmaddwd %xmm3, %xmm1, %xmm2
+; AVX-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    vpmaddwd %xmm4, %xmm1, %xmm2
+; AVX-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    vpmaddwd %xmm5, %xmm1, %xmm1
+; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: foo_reg_128:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vpdpwssd %xmm2, %xmm1, %xmm0
-; AVX512-NEXT:    vpdpwssd %xmm3, %xmm1, %xmm0
-; AVX512-NEXT:    vpdpwssd %xmm4, %xmm1, %xmm0
-; AVX512-NEXT:    vpdpwssd %xmm5, %xmm1, %xmm0
+; AVX512-NEXT:    vpmaddwd %xmm3, %xmm1, %xmm2
+; AVX512-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
+; AVX512-NEXT:    vpmaddwd %xmm4, %xmm1, %xmm2
+; AVX512-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
+; AVX512-NEXT:    vpmaddwd %xmm5, %xmm1, %xmm1
+; AVX512-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX512-NEXT:    retq
   %7 = bitcast <2 x i64> %0 to <4 x i32>
   %8 = bitcast <2 x i64> %1 to <4 x i32>
   %9 = bitcast <2 x i64> %2 to <4 x i32>
   %10 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %7, <4 x i32> %8, <4 x i32> %9)
   %11 = bitcast <2 x i64> %3 to <4 x i32>
   %12 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %10, <4 x i32> %8, <4 x i32> %11)
   %13 = bitcast <2 x i64> %4 to <4 x i32>
   %14 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %12, <4 x i32> %8, <4 x i32> %13)
   %15 = bitcast <2 x i64> %5 to <4 x i32>
   %16 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %14, <4 x i32> %8, <4 x i32> %15)
   %17 = bitcast <4 x i32> %16 to <2 x i64>
   ret <2 x i64> %17
 }
 
 declare <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32>, <4 x i32>, <4 x i32>) #1
 
 define <2 x i64> @foo_128(i32 %0, <2 x i64> %1, <2 x i64> %2, ptr %3) {
-; AVX-LABEL: foo_128:
-; AVX:       # %bb.0:
-; AVX-NEXT:    testl %edi, %edi
-; AVX-NEXT:    jle .LBB1_6
-; AVX-NEXT:  # %bb.1:
-; AVX-NEXT:    movl %edi, %edx
-; AVX-NEXT:    movl %edx, %eax
-; AVX-NEXT:    andl $3, %eax
-; AVX-NEXT:    cmpl $4, %edi
-; AVX-NEXT:    jae .LBB1_7
-; AVX-NEXT:  # %bb.2:
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    jmp .LBB1_3
-; AVX-NEXT:  .LBB1_7:
-; AVX-NEXT:    andl $-4, %edx
-; AVX-NEXT:    leaq 48(%rsi), %rdi
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB1_8: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    {vex} vpdpwssd -48(%rdi), %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd -32(%rdi), %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd -16(%rdi), %xmm1, %xmm0
-; AVX-NEXT:    {vex} vpdpwssd (%rdi), %xmm1, %xmm0
-; AVX-NEXT:    addq $4, %rcx
-; AVX-NEXT:    addq $64, %rdi
-; AVX-NEXT:    cmpq %rcx, %rdx
-; AVX-NEXT:    jne .LBB1_8
-; AVX-NEXT:  .LBB1_3:
-; AVX-NEXT:    testq %rax, %rax
-; AVX-NEXT:    je .LBB1_6
-; AVX-NEXT:  # %bb.4: # %.preheader
-; AVX-NEXT:    shlq $4, %rcx
-; AVX-NEXT:    addq %rcx, %rsi
-; AVX-NEXT:    shlq $4, %rax
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB1_5: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    {vex} vpdpwssd (%rsi,%rcx), %xmm1, %xmm0
-; AVX-NEXT:    addq $16, %rcx
-; AVX-NEXT:    cmpq %rcx, %rax
-; AVX-NEXT:    jne .LBB1_5
-; AVX-NEXT:  .LBB1_6:
-; AVX-NEXT:    retq
-;
 ; AVX512-LABEL: foo_128:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    testl %edi, %edi
 ; AVX512-NEXT:    jle .LBB1_6
 ; AVX512-NEXT:  # %bb.1:
 ; AVX512-NEXT:    movl %edi, %edx
 ; AVX512-NEXT:    movl %edx, %eax
 ; AVX512-NEXT:    andl $3, %eax
 ; AVX512-NEXT:    cmpl $4, %edi
 ; AVX512-NEXT:    jae .LBB1_7
 ; AVX512-NEXT:  # %bb.2:
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    jmp .LBB1_3
 ; AVX512-NEXT:  .LBB1_7:
 ; AVX512-NEXT:    andl $-4, %edx
 ; AVX512-NEXT:    leaq 48(%rsi), %rdi
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    .p2align 4, 0x90
 ; AVX512-NEXT:  .LBB1_8: # =>This Inner Loop Header: Depth=1
 ; AVX512-NEXT:    vpdpwssd -48(%rdi), %xmm1, %xmm0
-; AVX512-NEXT:    vpdpwssd -32(%rdi), %xmm1, %xmm0
-; AVX512-NEXT:    vpdpwssd -16(%rdi), %xmm1, %xmm0
+; AVX512-NEXT:    vpmaddwd -32(%rdi), %xmm1, %xmm2
+; AVX512-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
+; AVX512-NEXT:    vpmaddwd -16(%rdi), %xmm1, %xmm2
+; AVX512-NEXT:    vpaddd %xmm2, %xmm0, %xmm0
 ; AVX512-NEXT:    vpdpwssd (%rdi), %xmm1, %xmm0
 ; AVX512-NEXT:    addq $4, %rcx
 ; AVX512-NEXT:    addq $64, %rdi
 ; AVX512-NEXT:    cmpq %rcx, %rdx
 ; AVX512-NEXT:    jne .LBB1_8
 ; AVX512-NEXT:  .LBB1_3:
 ; AVX512-NEXT:    testq %rax, %rax
 ; AVX512-NEXT:    je .LBB1_6
 ; AVX512-NEXT:  # %bb.4: # %.preheader
 ; AVX512-NEXT:    shlq $4, %rcx
 ; AVX512-NEXT:    addq %rcx, %rsi
 ; AVX512-NEXT:    shlq $4, %rax
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    .p2align 4, 0x90
 ; AVX512-NEXT:  .LBB1_5: # =>This Inner Loop Header: Depth=1
 ; AVX512-NEXT:    vpdpwssd (%rsi,%rcx), %xmm1, %xmm0
 ; AVX512-NEXT:    addq $16, %rcx
 ; AVX512-NEXT:    cmpq %rcx, %rax
 ; AVX512-NEXT:    jne .LBB1_5
 ; AVX512-NEXT:  .LBB1_6:
 ; AVX512-NEXT:    retq
   %5 = icmp sgt i32 %0, 0
   br i1 %5, label %6, label %33
 
 6:                                                ; preds = %4
   %7 = bitcast <2 x i64> %2 to <8 x i16>
   %8 = bitcast <2 x i64> %1 to <4 x i32>
   %9 = zext i32 %0 to i64
   %10 = and i64 %9, 3
   %11 = icmp ult i32 %0, 4
   br i1 %11, label %14, label %12
 
 12:                                               ; preds = %6
   %13 = and i64 %9, 4294967292
   br label %35
 
 14:                                               ; preds = %35, %6
   %15 = phi <4 x i32> [ undef, %6 ], [ %57, %35 ]
   %16 = phi i64 [ 0, %6 ], [ %58, %35 ]
   %17 = phi <4 x i32> [ %8, %6 ], [ %57, %35 ]
   %18 = icmp eq i64 %10, 0
   br i1 %18, label %30, label %19
 
 19:                                               ; preds = %14, %19
   %20 = phi i64 [ %27, %19 ], [ %16, %14 ]
   %21 = phi <4 x i32> [ %26, %19 ], [ %17, %14 ]
   %22 = phi i64 [ %28, %19 ], [ 0, %14 ]
   %23 = getelementptr inbounds <2 x i64>, ptr %3, i64 %20
   %24 = load <8 x i16>, ptr %23, align 16
   %25 = tail call <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16> %7, <8 x i16> %24)
   %26 = add <4 x i32> %25, %21
   %27 = add nuw nsw i64 %20, 1
   %28 = add i64 %22, 1
   %29 = icmp eq i64 %28, %10
   br i1 %29, label %30, label %19
 
 30:                                               ; preds = %19, %14
   %31 = phi <4 x i32> [ %15, %14 ], [ %26, %19 ]
   %32 = bitcast <4 x i32> %31 to <2 x i64>
   br label %33
 
 33:                                               ; preds = %30, %4
   %34 = phi <2 x i64> [ %32, %30 ], [ %1, %4 ]
   ret <2 x i64> %34
 
 35:                                               ; preds = %35, %12
   %36 = phi i64 [ 0, %12 ], [ %58, %35 ]
   %37 = phi <4 x i32> [ %8, %12 ], [ %57, %35 ]
   %38 = phi i64 [ 0, %12 ], [ %59, %35 ]
   %39 = getelementptr inbounds <2 x i64>, ptr %3, i64 %36
   %40 = load <8 x i16>, ptr %39, align 16
   %41 = tail call <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16> %7, <8 x i16> %40)
   %42 = add <4 x i32> %41, %37
   %43 = or i64 %36, 1
   %44 = getelementptr inbounds <2 x i64>, ptr %3, i64 %43
   %45 = load <8 x i16>, ptr %44, align 16
   %46 = tail call <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16> %7, <8 x i16> %45)
   %47 = add <4 x i32> %46, %42
   %48 = or i64 %36, 2
   %49 = getelementptr inbounds <2 x i64>, ptr %3, i64 %48
   %50 = load <8 x i16>, ptr %49, align 16
   %51 = tail call <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16> %7, <8 x i16> %50)
   %52 = add <4 x i32> %51, %47
   %53 = or i64 %36, 3
   %54 = getelementptr inbounds <2 x i64>, ptr %3, i64 %53
   %55 = load <8 x i16>, ptr %54, align 16
   %56 = tail call <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16> %7, <8 x i16> %55)
   %57 = add <4 x i32> %56, %52
   %58 = add nuw nsw i64 %36, 4
   %59 = add i64 %38, 4
   %60 = icmp eq i64 %59, %13
   br i1 %60, label %14, label %35
 }
 
 define void @bar_128(i32 %0, ptr %1, <2 x i64> %2, ptr %3) {
-; AVX-LABEL: bar_128:
-; AVX:       # %bb.0:
-; AVX-NEXT:    testl %edi, %edi
-; AVX-NEXT:    jle .LBB2_5
-; AVX-NEXT:  # %bb.1:
-; AVX-NEXT:    movl %edi, %eax
-; AVX-NEXT:    cmpl $1, %edi
-; AVX-NEXT:    jne .LBB2_6
-; AVX-NEXT:  # %bb.2:
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    jmp .LBB2_3
-; AVX-NEXT:  .LBB2_6:
-; AVX-NEXT:    movl %eax, %edi
-; AVX-NEXT:    andl $-2, %edi
-; AVX-NEXT:    movl $16, %r8d
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB2_7: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    vmovdqa -16(%rsi,%r8), %xmm1
-; AVX-NEXT:    vmovdqa (%rsi,%r8), %xmm2
-; AVX-NEXT:    {vex} vpdpwssd -16(%rdx,%r8), %xmm0, %xmm1
-; AVX-NEXT:    vmovdqa %xmm1, -16(%rsi,%r8)
-; AVX-NEXT:    {vex} vpdpwssd (%rdx,%r8), %xmm0, %xmm2
-; AVX-NEXT:    vmovdqa %xmm2, (%rsi,%r8)
-; AVX-NEXT:    addq $2, %rcx
-; AVX-NEXT:    addq $32, %r8
-; AVX-NEXT:    cmpq %rcx, %rdi
-; AVX-NEXT:    jne .LBB2_7
-; AVX-NEXT:  .LBB2_3:
-; AVX-NEXT:    testb $1, %al
-; AVX-NEXT:    je .LBB2_5
-; AVX-NEXT:  # %bb.4:
-; AVX-NEXT:    shlq $4, %rcx
-; AVX-NEXT:    vmovdqa (%rsi,%rcx), %xmm1
-; AVX-NEXT:    {vex} vpdpwssd (%rdx,%rcx), %xmm0, %xmm1
-; AVX-NEXT:    vmovdqa %xmm1, (%rsi,%rcx)
-; AVX-NEXT:  .LBB2_5:
-; AVX-NEXT:    retq
-;
 ; AVX512-LABEL: bar_128:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    testl %edi, %edi
 ; AVX512-NEXT:    jle .LBB2_5
 ; AVX512-NEXT:  # %bb.1:
 ; AVX512-NEXT:    movl %edi, %eax
 ; AVX512-NEXT:    cmpl $1, %edi
 ; AVX512-NEXT:    jne .LBB2_6
 ; AVX512-NEXT:  # %bb.2:
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    jmp .LBB2_3
 ; AVX512-NEXT:  .LBB2_6:
 ; AVX512-NEXT:    movl %eax, %edi
 ; AVX512-NEXT:    andl $-2, %edi
 ; AVX512-NEXT:    movl $16, %r8d
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    .p2align 4, 0x90
 ; AVX512-NEXT:  .LBB2_7: # =>This Inner Loop Header: Depth=1
 ; AVX512-NEXT:    vmovdqa -16(%rsi,%r8), %xmm1
 ; AVX512-NEXT:    vmovdqa (%rsi,%r8), %xmm2
 ; AVX512-NEXT:    vpdpwssd -16(%rdx,%r8), %xmm0, %xmm1
 ; AVX512-NEXT:    vmovdqa %xmm1, -16(%rsi,%r8)
 ; AVX512-NEXT:    vpdpwssd (%rdx,%r8), %xmm0, %xmm2
 ; AVX512-NEXT:    vmovdqa %xmm2, (%rsi,%r8)
 ; AVX512-NEXT:    addq $2, %rcx
 ; AVX512-NEXT:    addq $32, %r8
 ; AVX512-NEXT:    cmpq %rcx, %rdi
 ; AVX512-NEXT:    jne .LBB2_7
 ; AVX512-NEXT:  .LBB2_3:
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB2_5
 ; AVX512-NEXT:  # %bb.4:
 ; AVX512-NEXT:    shlq $4, %rcx
-; AVX512-NEXT:    vmovdqa (%rsi,%rcx), %xmm1
-; AVX512-NEXT:    vpdpwssd (%rdx,%rcx), %xmm0, %xmm1
-; AVX512-NEXT:    vmovdqa %xmm1, (%rsi,%rcx)
+; AVX512-NEXT:    vpmaddwd (%rdx,%rcx), %xmm0, %xmm0
+; AVX512-NEXT:    vpaddd (%rsi,%rcx), %xmm0, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rsi,%rcx)
 ; AVX512-NEXT:  .LBB2_5:
 ; AVX512-NEXT:    retq
   %5 = icmp sgt i32 %0, 0
   br i1 %5, label %6, label %22
 
 6:                                                ; preds = %4
   %7 = bitcast <2 x i64> %2 to <4 x i32>
   %8 = zext i32 %0 to i64
   %9 = and i64 %8, 1
   %10 = icmp eq i32 %0, 1
   br i1 %10, label %13, label %11
 
 11:                                               ; preds = %6
   %12 = and i64 %8, 4294967294
   br label %23
 
 13:                                               ; preds = %23, %6
   %14 = phi i64 [ 0, %6 ], [ %37, %23 ]
   %15 = icmp eq i64 %9, 0
   br i1 %15, label %22, label %16
 
 16:                                               ; preds = %13
   %17 = getelementptr inbounds <2 x i64>, ptr %3, i64 %14
   %18 = load <4 x i32>, ptr %17, align 16
   %19 = getelementptr inbounds <2 x i64>, ptr %1, i64 %14
   %20 = load <4 x i32>, ptr %19, align 16
   %21 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %20, <4 x i32> %7, <4 x i32> %18)
   store <4 x i32> %21, ptr %19, align 16
   br label %22
 
 22:                                               ; preds = %16, %13, %4
   ret void
 
 23:                                               ; preds = %23, %11
   %24 = phi i64 [ 0, %11 ], [ %37, %23 ]
   %25 = phi i64 [ 0, %11 ], [ %38, %23 ]
   %26 = getelementptr inbounds <2 x i64>, ptr %3, i64 %24
   %27 = load <4 x i32>, ptr %26, align 16
   %28 = getelementptr inbounds <2 x i64>, ptr %1, i64 %24
   %29 = load <4 x i32>, ptr %28, align 16
   %30 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %29, <4 x i32> %7, <4 x i32> %27)
   store <4 x i32> %30, ptr %28, align 16
   %31 = or i64 %24, 1
   %32 = getelementptr inbounds <2 x i64>, ptr %3, i64 %31
   %33 = load <4 x i32>, ptr %32, align 16
   %34 = getelementptr inbounds <2 x i64>, ptr %1, i64 %31
   %35 = load <4 x i32>, ptr %34, align 16
   %36 = tail call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %35, <4 x i32> %7, <4 x i32> %33)
   store <4 x i32> %36, ptr %34, align 16
   %37 = add nuw nsw i64 %24, 2
   %38 = add i64 %25, 2
   %39 = icmp eq i64 %38, %12
   br i1 %39, label %13, label %23
 }
 
 declare <4 x i32> @llvm.x86.sse2.pmadd.wd(<8 x i16>, <8 x i16>) #1
 
 define <4 x i64> @foo_reg_256(<4 x i64> %0, <4 x i64> %1, <4 x i64> %2, <4 x i64> %3, <4 x i64> %4, <4 x i64> %5) {
 ; AVX-LABEL: foo_reg_256:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    {vex} vpdpwssd %ymm2, %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd %ymm3, %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd %ymm4, %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd %ymm5, %ymm1, %ymm0
+; AVX-NEXT:    vpmaddwd %ymm3, %ymm1, %ymm2
+; AVX-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX-NEXT:    vpmaddwd %ymm4, %ymm1, %ymm2
+; AVX-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX-NEXT:    vpmaddwd %ymm5, %ymm1, %ymm1
+; AVX-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: foo_reg_256:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vpdpwssd %ymm2, %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd %ymm3, %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd %ymm4, %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd %ymm5, %ymm1, %ymm0
+; AVX512-NEXT:    vpmaddwd %ymm3, %ymm1, %ymm2
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX512-NEXT:    vpmaddwd %ymm4, %ymm1, %ymm2
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX512-NEXT:    vpmaddwd %ymm5, %ymm1, %ymm1
+; AVX512-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %7 = bitcast <4 x i64> %0 to <8 x i32>
   %8 = bitcast <4 x i64> %1 to <8 x i32>
   %9 = bitcast <4 x i64> %2 to <8 x i32>
   %10 = tail call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %7, <8 x i32> %8, <8 x i32> %9)
   %11 = bitcast <4 x i64> %3 to <8 x i32>
   %12 = tail call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %10, <8 x i32> %8, <8 x i32> %11)
   %13 = bitcast <4 x i64> %4 to <8 x i32>
   %14 = tail call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %12, <8 x i32> %8, <8 x i32> %13)
   %15 = bitcast <4 x i64> %5 to <8 x i32>
   %16 = tail call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %14, <8 x i32> %8, <8 x i32> %15)
   %17 = bitcast <8 x i32> %16 to <4 x i64>
   ret <4 x i64> %17
 }
 
 ; __m256i foo(int cnt, __m256i c, __m256i b, __m256i *p) {
 ;     for (int i = 0; i < cnt; ++i) {
 ;         __m256i a = p[i];
 ;         __m256i m = _mm256_madd_epi16 (b, a);
 ;         c = _mm256_add_epi32(m, c);
 ;     }
 ;     return c;
 ; }
 
 define <4 x i64> @foo_256(i32 %0, <4 x i64> %1, <4 x i64> %2, ptr %3) {
-; AVX-LABEL: foo_256:
-; AVX:       # %bb.0:
-; AVX-NEXT:    testl %edi, %edi
-; AVX-NEXT:    jle .LBB4_6
-; AVX-NEXT:  # %bb.1:
-; AVX-NEXT:    movl %edi, %edx
-; AVX-NEXT:    movl %edx, %eax
-; AVX-NEXT:    andl $3, %eax
-; AVX-NEXT:    cmpl $4, %edi
-; AVX-NEXT:    jae .LBB4_7
-; AVX-NEXT:  # %bb.2:
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    jmp .LBB4_3
-; AVX-NEXT:  .LBB4_7:
-; AVX-NEXT:    andl $-4, %edx
-; AVX-NEXT:    leaq 96(%rsi), %rdi
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB4_8: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    {vex} vpdpwssd -96(%rdi), %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd -64(%rdi), %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd -32(%rdi), %ymm1, %ymm0
-; AVX-NEXT:    {vex} vpdpwssd (%rdi), %ymm1, %ymm0
-; AVX-NEXT:    addq $4, %rcx
-; AVX-NEXT:    subq $-128, %rdi
-; AVX-NEXT:    cmpq %rcx, %rdx
-; AVX-NEXT:    jne .LBB4_8
-; AVX-NEXT:  .LBB4_3:
-; AVX-NEXT:    testq %rax, %rax
-; AVX-NEXT:    je .LBB4_6
-; AVX-NEXT:  # %bb.4: # %.preheader
-; AVX-NEXT:    shlq $5, %rcx
-; AVX-NEXT:    addq %rcx, %rsi
-; AVX-NEXT:    shlq $5, %rax
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB4_5: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    {vex} vpdpwssd (%rsi,%rcx), %ymm1, %ymm0
-; AVX-NEXT:    addq $32, %rcx
-; AVX-NEXT:    cmpq %rcx, %rax
-; AVX-NEXT:    jne .LBB4_5
-; AVX-NEXT:  .LBB4_6:
-; AVX-NEXT:    retq
-;
 ; AVX512-LABEL: foo_256:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    testl %edi, %edi
 ; AVX512-NEXT:    jle .LBB4_6
 ; AVX512-NEXT:  # %bb.1:
 ; AVX512-NEXT:    movl %edi, %edx
 ; AVX512-NEXT:    movl %edx, %eax
 ; AVX512-NEXT:    andl $3, %eax
 ; AVX512-NEXT:    cmpl $4, %edi
 ; AVX512-NEXT:    jae .LBB4_7
 ; AVX512-NEXT:  # %bb.2:
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    jmp .LBB4_3
 ; AVX512-NEXT:  .LBB4_7:
 ; AVX512-NEXT:    andl $-4, %edx
 ; AVX512-NEXT:    leaq 96(%rsi), %rdi
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    .p2align 4, 0x90
 ; AVX512-NEXT:  .LBB4_8: # =>This Inner Loop Header: Depth=1
 ; AVX512-NEXT:    vpdpwssd -96(%rdi), %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd -64(%rdi), %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd -32(%rdi), %ymm1, %ymm0
-; AVX512-NEXT:    vpdpwssd (%rdi), %ymm1, %ymm0
+; AVX512-NEXT:    vpmaddwd -64(%rdi), %ymm1, %ymm2
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX512-NEXT:    vpmaddwd -32(%rdi), %ymm1, %ymm2
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; AVX512-NEXT:    vpmaddwd (%rdi), %ymm1, %ymm2
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    addq $4, %rcx
 ; AVX512-NEXT:    subq $-128, %rdi
 ; AVX512-NEXT:    cmpq %rcx, %rdx
 ; AVX512-NEXT:    jne .LBB4_8
 ; AVX512-NEXT:  .LBB4_3:
 ; AVX512-NEXT:    testq %rax, %rax
 ; AVX512-NEXT:    je .LBB4_6
 ; AVX512-NEXT:  # %bb.4: # %.preheader

 declare <8 x i32> @llvm.x86.avx2.pmadd.wd(<16 x i16>, <16 x i16>)
 
 ; void bar(int cnt, __m256i *c, __m256i b, __m256i *p) {
 ;     for (int i = 0; i < cnt; ++i) {
 ;         __m256i a = p[i];
 ;         c[i] = _mm256_dpwssd_epi32(c[i], b, a);
 ;     }
 ; }
 define void @bar_256(i32 %0, ptr %1, <4 x i64> %2, ptr %3) {
-; AVX-LABEL: bar_256:
-; AVX:       # %bb.0:
-; AVX-NEXT:    testl %edi, %edi
-; AVX-NEXT:    jle .LBB5_5
-; AVX-NEXT:  # %bb.1:
-; AVX-NEXT:    movl %edi, %eax
-; AVX-NEXT:    cmpl $1, %edi
-; AVX-NEXT:    jne .LBB5_6
-; AVX-NEXT:  # %bb.2:
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    jmp .LBB5_3
-; AVX-NEXT:  .LBB5_6:
-; AVX-NEXT:    movl %eax, %edi
-; AVX-NEXT:    andl $-2, %edi
-; AVX-NEXT:    movl $32, %r8d
-; AVX-NEXT:    xorl %ecx, %ecx
-; AVX-NEXT:    .p2align 4, 0x90
-; AVX-NEXT:  .LBB5_7: # =>This Inner Loop Header: Depth=1
-; AVX-NEXT:    vmovdqa -32(%rsi,%r8), %ymm1
-; AVX-NEXT:    vmovdqa (%rsi,%r8), %ymm2
-; AVX-NEXT:    {vex} vpdpwssd -32(%rdx,%r8), %ymm0, %ymm1
-; AVX-NEXT:    vmovdqa %ymm1, -32(%rsi,%r8)
-; AVX-NEXT:    {vex} vpdpwssd (%rdx,%r8), %ymm0, %ymm2
-; AVX-NEXT:    vmovdqa %ymm2, (%rsi,%r8)
-; AVX-NEXT:    addq $2, %rcx
-; AVX-NEXT:    addq $64, %r8
-; AVX-NEXT:    cmpq %rcx, %rdi
-; AVX-NEXT:    jne .LBB5_7
-; AVX-NEXT:  .LBB5_3:
-; AVX-NEXT:    testb $1, %al
-; AVX-NEXT:    je .LBB5_5
-; AVX-NEXT:  # %bb.4:
-; AVX-NEXT:    shlq $5, %rcx
-; AVX-NEXT:    vmovdqa (%rsi,%rcx), %ymm1
-; AVX-NEXT:    {vex} vpdpwssd (%rdx,%rcx), %ymm0, %ymm1
-; AVX-NEXT:    vmovdqa %ymm1, (%rsi,%rcx)
-; AVX-NEXT:  .LBB5_5:
-; AVX-NEXT:    vzeroupper
-; AVX-NEXT:    retq
-;
 ; AVX512-LABEL: bar_256:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    testl %edi, %edi
 ; AVX512-NEXT:    jle .LBB5_5
 ; AVX512-NEXT:  # %bb.1:
 ; AVX512-NEXT:    movl %edi, %eax
 ; AVX512-NEXT:    cmpl $1, %edi
 ; AVX512-NEXT:    jne .LBB5_6
 ; AVX512-NEXT:  # %bb.2:
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    jmp .LBB5_3
 ; AVX512-NEXT:  .LBB5_6:
 ; AVX512-NEXT:    movl %eax, %edi
 ; AVX512-NEXT:    andl $-2, %edi
 ; AVX512-NEXT:    movl $32, %r8d
 ; AVX512-NEXT:    xorl %ecx, %ecx
 ; AVX512-NEXT:    .p2align 4, 0x90
 ; AVX512-NEXT:  .LBB5_7: # =>This Inner Loop Header: Depth=1
 ; AVX512-NEXT:    vmovdqa -32(%rsi,%r8), %ymm1
 ; AVX512-NEXT:    vmovdqa (%rsi,%r8), %ymm2
 ; AVX512-NEXT:    vpdpwssd -32(%rdx,%r8), %ymm0, %ymm1
 ; AVX512-NEXT:    vmovdqa %ymm1, -32(%rsi,%r8)
-; AVX512-NEXT:    vpdpwssd (%rdx,%r8), %ymm0, %ymm2
-; AVX512-NEXT:    vmovdqa %ymm2, (%rsi,%r8)
+; AVX512-NEXT:    vpmaddwd (%rdx,%r8), %ymm0, %ymm1
+; AVX512-NEXT:    vpaddd %ymm1, %ymm2, %ymm1
+; AVX512-NEXT:    vmovdqa %ymm1, (%rsi,%r8)
 ; AVX512-NEXT:    addq $2, %rcx
 ; AVX512-NEXT:    addq $64, %r8
 ; AVX512-NEXT:    cmpq %rcx, %rdi
 ; AVX512-NEXT:    jne .LBB5_7
 ; AVX512-NEXT:  .LBB5_3:
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB5_5
 ; AVX512-NEXT:  # %bb.4:
 ; AVX512-NEXT:    shlq $5, %rcx
-; AVX512-NEXT:    vmovdqa (%rsi,%rcx), %ymm1
-; AVX512-NEXT:    vpdpwssd (%rdx,%rcx), %ymm0, %ymm1
-; AVX512-NEXT:    vmovdqa %ymm1, (%rsi,%rcx)
+; AVX512-NEXT:    vpmaddwd (%rdx,%rcx), %ymm0, %ymm0
+; AVX512-NEXT:    vpaddd (%rsi,%rcx), %ymm0, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rsi,%rcx)
 ; AVX512-NEXT:  .LBB5_5:
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
   %5 = icmp sgt i32 %0, 0
   br i1 %5, label %6, label %22
 
 6:                                                ; preds = %4
   %7 = bitcast <4 x i64> %2 to <8 x i32>