[SystemZ] Improve decoding in case of instructions with four register operands.

[jonpa]

Summary

Since z13, the max group size will be 2 if any μop has more than 3 register sources.

This has been ignored sofar in the SystemZHazardRecognizer, but is now handled by recognizing those instructions and adjusting the tracking of decoding and the cost heuristic for grouping.

• If the current decoder group has 2 uops, and the candidate has 4RegOps, then a cost of 1 is returned from groupingCost().

• Some of the relevant instructions (non-vector) group alone. The handling of these should be as before.

• The absolute grouping cost is not adjusted in groupingCost() between 2 or 3 depending on CurrGroupHas4RegOps, since it should not matter.

• If the current group has an instruction with 4 reg operands, the max group size becomes 2, except in the case of a single group-alone instruction which per default has 3 uops to represent a full decoder group. This also means that a cracked instruction that has 4 reg ops (if there is such) would group alone.

• One very minor SystemZ test update.

[uweigand]

lib/Target/SystemZ/SystemZHazardRecognizer.cpp

123

Have you verified that this doesn't count things like CC?

[jonpa]

These are the opcodes that would return true from has4RegOps:

z196, zEC12:
Group alone:
LMD, MAD, MADB, MAE, MAEB, MAY, MAYH, MAYL, MSD, MSDB, MSE, MSEB, MVCK, MVCP, MVCS, MY, MYH, MYL, PLO, SLDT, SLXT, SRDT, SRXT

Z13, in addition:
Normal grouping:
VAC, VACCC, VACCCQ, VACQ, VFMA, VFMADB, VFMS, VFMSDB, VGFMA, VGFMAB, VGFMAF, VGFMAG, VGFMAH, VMAE, VMAEB, VMAEF, VMAEH, VMAH, VMAHB, VMAHF, VMAHH, VMAL, VMALB, VMALE, VMALEB, VMALEF, VMALEH, VMALF, VMALH, VMALHB, VMALHF, VMALHH, VMALHW, VMALO, VMALOB, VMALOF, VMALOH, VMAO, VMAOB, VMAOF, VMAOH, VPERM, VSBCBI, VSBCBIQ, VSBI, VSBIQ, VSEL, VSTRC, VSTRCB, VSTRCBS, VSTRCF, VSTRCFS, VSTRCH, VSTRCHS, VSTRCZB, VSTRCZBS, VSTRCZF, VSTRCZFS, VSTRCZH, VSTRCZHS, WFMADB, WFMSDB

Z14, in addition:
Normal grouping:
VFMASB, VFMSSB, VFNMA, VFNMADB, VFNMASB, VFNMS, VFNMSDB, VFNMSSB, VMSL, VMSLG, WFMASB, WFMAXB, WFMSSB, WFMSXB, WFNMADB, WFNMASB, WFNMAXB, WFNMSDB, WFNMSSB, WFNMSXB

I have built SPEC on all subtargets from z196 and on, and found:

• z196 and zEC12 both NFC on SPEC, as expected since all affected instructions group alone.

• Instructions actually exposed to scheduler (emitted) of the above:

z196: none
zEC12: none
z13:

 10 VMALB
616 VMALF
 10 VMALHW
797 VPERM
424 VSEL

z14:

  10 VMALB
 616 VMALF
  10 VMALHW
1978 VPERM
 432 VSEL

lib/Target/SystemZ/SystemZHazardRecognizer.cpp

123

Yes, the NumOperands are the explicit defs and uses. Then there is in addition the lists of implicit defs and uses.

So, AG has
getNumOperands = 5
MID.hasImplicitDefOfPhysReg(SystemZ::CC) = true
The operands get printed like
DEF[GR64Bit] TIED-USE[GR64Bit] USE[ADDR64Bit] [IMM] USE[ADDR64Bit]
The count of regops = 3 (excluding any tied use).

[uweigand]

Ah, OK. Thanks for checking!

Patch LGTM.

[jonpa]

TII made available also with NDEBUG.

[jonpa]

Committed as r338368.

[bjope]

lib/Target/SystemZ/SystemZHazardRecognizer.cpp

104

You probably want to do assert((something) && "") instead of assert((something) || "") here.

[jonpa]

lib/Target/SystemZ/SystemZHazardRecognizer.cpp

104

oops... fixed with r338429. thanks.

lib/Target/SystemZ/SystemZHazardRecognizer.h

 #include "llvm/Support/raw_ostream.h"
 #include <string>
 
 namespace llvm {
 
 /// SystemZHazardRecognizer maintains the state for one MBB during scheduling.
 class SystemZHazardRecognizer : public ScheduleHazardRecognizer {
 
-#ifndef NDEBUG
   const SystemZInstrInfo *TII;
-#endif
   const TargetSchedModel *SchedModel;
 
   /// Keep track of the number of decoder slots used in the current
   /// decoder group.
   unsigned CurrGroupSize;
 
+  /// True if an instruction with four reg operands have been scheduled into
+  /// the current decoder group.
+  bool CurrGroupHas4RegOps;
+
   /// The tracking of resources here are quite similar to the common
   /// code use of a critical resource. However, z13 differs in the way
   /// that it has two processor sides which may be interesting to
   /// model in the future (a work in progress).
 
   /// Counters for the number of uops scheduled per processor
   /// resource.
   SmallVector<int, 0> ProcResourceCounters;
 
   /// This is the resource with the greatest queue, which the
   /// scheduler tries to avoid.
   unsigned CriticalResourceIdx;
 
   /// Return the number of decoder slots MI requires.
   inline unsigned getNumDecoderSlots(SUnit *SU) const;
 
   /// Return true if MI fits into current decoder group.
   bool fitsIntoCurrentGroup(SUnit *SU) const;
 
+  /// Return true if this instruction has four register operands.
+  bool has4RegOps(const MachineInstr *MI) const;
+
   /// Two decoder groups per cycle are formed (for z13), meaning 2x3
   /// instructions. This function returns a number between 0 and 5,
   /// representing the current decoder slot of the current cycle.  If an SU
   /// is passed which will begin a new decoder group, the returned value is
   /// the cycle index of the next group.
   unsigned getCurrCycleIdx(SUnit *SU = nullptr) const;
 
   /// LastFPdOpCycleIdx stores the numbeer returned by getCurrCycleIdx()
class SystemZHazardRecognizer : public ScheduleHazardRecognizer {
   bool isFPdOpPreferred_distance(SUnit *SU) const;
 
   /// Last emitted instruction or nullptr.
   MachineInstr *LastEmittedMI;
 
 public:
   SystemZHazardRecognizer(const SystemZInstrInfo *tii,
                           const TargetSchedModel *SM)
-      :
-#ifndef NDEBUG
-        TII(tii),
-#endif
-        SchedModel(SM) {
+      : TII(tii), SchedModel(SM) {
     Reset();
   }
 
   HazardType getHazardType(SUnit *m, int Stalls = 0) override;
   void Reset() override;
   void EmitInstruction(SUnit *SU) override;
 
   /// Resolves and cache a resolved scheduling class for an SUnit.

lib/Target/SystemZ/SystemZHazardRecognizer.cpp

 
 ScheduleHazardRecognizer::HazardType SystemZHazardRecognizer::
 getHazardType(SUnit *m, int Stalls) {
   return (fitsIntoCurrentGroup(m) ? NoHazard : Hazard);
 }
 
 void SystemZHazardRecognizer::Reset() {
   CurrGroupSize = 0;
+  CurrGroupHas4RegOps = false;
   clearProcResCounters();
   GrpCount = 0;
   LastFPdOpCycleIdx = UINT_MAX;
   LastEmittedMI = nullptr;
   LLVM_DEBUG(CurGroupDbg = "";);
 }
 
 bool
 SystemZHazardRecognizer::fitsIntoCurrentGroup(SUnit *SU) const {
   const MCSchedClassDesc *SC = getSchedClass(SU);
   if (!SC->isValid())
     return true;
 
   // A cracked instruction only fits into schedule if the current
   // group is empty.
   if (SC->BeginGroup)
     return (CurrGroupSize == 0);
 
+  // An instruction with 4 register operands will not fit in last slot.
+  assert ((CurrGroupSize < 2 || !CurrGroupHas4RegOps) ||
+          "Current decoder group is already full!");
+  if (CurrGroupSize == 2 && has4RegOps(SU->getInstr()))
+    return false;
+
   // Since a full group is handled immediately in EmitInstruction(),
   // SU should fit into current group. NumSlots should be 1 or 0,
   // since it is not a cracked or expanded instruction.
   assert ((getNumDecoderSlots(SU) <= 1) && (CurrGroupSize < 3) &&
           "Expected normal instruction to fit in non-full group!");
 
   return true;
 }
 
+bool SystemZHazardRecognizer::has4RegOps(const MachineInstr *MI) const {
+  const MachineFunction &MF = *MI->getParent()->getParent();
+  const TargetRegisterInfo *TRI = &TII->getRegisterInfo();
+  const MCInstrDesc &MID = MI->getDesc();
+  unsigned Count = 0;
+  for (unsigned OpIdx = 0; OpIdx < MID.getNumOperands(); OpIdx++) {
+    const TargetRegisterClass *RC = TII->getRegClass(MID, OpIdx, TRI, MF);
+    if (RC == nullptr)
+      continue;
+    if (OpIdx >= MID.getNumDefs() &&
+        MID.getOperandConstraint(OpIdx, MCOI::TIED_TO) != -1)
+      continue;
+    Count++;
+  }
+  return Count >= 4;
+}
+
 void SystemZHazardRecognizer::nextGroup() {
   if (CurrGroupSize == 0)
     return;
 
   LLVM_DEBUG(dumpCurrGroup("Completed decode group"));
   LLVM_DEBUG(CurGroupDbg = "";);
 
   GrpCount++;
 
   // Reset counter for next group.
   CurrGroupSize = 0;
+  CurrGroupHas4RegOps = false;
 
   // Decrease counters for execution units by one.
   for (unsigned i = 0; i < SchedModel->getNumProcResourceKinds(); ++i)
     if (ProcResourceCounters[i] > 0)
       ProcResourceCounters[i]--;
 
   // Clear CriticalResourceIdx if it is now below the threshold.
   if (CriticalResourceIdx != UINT_MAX &&
void SystemZHazardRecognizer::dumpSU(SUnit *SU, raw_ostream &OS) const {
   if (SC->BeginGroup && SC->EndGroup)
     OS << "/GroupsAlone";
   else if (SC->BeginGroup)
     OS << "/BeginsGroup";
   else if (SC->EndGroup)
     OS << "/EndsGroup";
   if (SU->isUnbuffered)
     OS << "/Unbuffered";
+  if (has4RegOps(SU->getInstr()))
+    OS << "/4RegOps";
 }
 
 void SystemZHazardRecognizer::dumpCurrGroup(std::string Msg) const {
   dbgs() << "++ " << Msg;
   dbgs() << ": ";
 
   if (CurGroupDbg.empty())
     dbgs() << " <empty>\n";
   else {
     dbgs() << "{ " << CurGroupDbg << " }";
     dbgs() << " (" << CurrGroupSize << " decoder slot"
            << (CurrGroupSize > 1 ? "s":"")
+           << (CurrGroupHas4RegOps ? ", 4RegOps" : "")
            << ")\n";
   }
 }
 
 void SystemZHazardRecognizer::dumpProcResourceCounters() const {
   bool any = false;
 
   for (unsigned i = 0; i < SchedModel->getNumProcResourceKinds(); ++i)
if (SU->isUnbuffered) {
     LastFPdOpCycleIdx = getCurrCycleIdx(SU);
     LLVM_DEBUG(dbgs() << "++ Last FPd cycle index: " << LastFPdOpCycleIdx
                       << "\n";);
   }
 
   // Insert SU into current group by increasing number of slots used
   // in current group.
   CurrGroupSize += getNumDecoderSlots(SU);
-  assert (CurrGroupSize <= 3);
+  CurrGroupHas4RegOps |= has4RegOps(SU->getInstr());
+  unsigned GroupLim =
+    ((CurrGroupHas4RegOps && getNumDecoderSlots(SU) < 3) ? 2 : 3);
+  assert (CurrGroupSize <= GroupLim && "SU does not fit into decoder group!");
 
   // Check if current group is now full/ended. If so, move on to next
   // group to be ready to evaluate more candidates.
-  if (CurrGroupSize == 3 || SC->EndGroup)
+  if (CurrGroupSize == GroupLim || SC->EndGroup)
     nextGroup();
 }
 
 int SystemZHazardRecognizer::groupingCost(SUnit *SU) const {
   const MCSchedClassDesc *SC = getSchedClass(SU);
   if (!SC->isValid())
     return 0;
   
int SystemZHazardRecognizer::groupingCost(SUnit *SU) const {
   if (SC->EndGroup) {
     unsigned resultingGroupSize =
       (CurrGroupSize + getNumDecoderSlots(SU));
     if (resultingGroupSize < 3)
       return (3 - resultingGroupSize);
     return -1;
   }
 
+  // An instruction with 4 register operands will not fit in last slot.
+  if (CurrGroupSize == 2 && has4RegOps(SU->getInstr()))
+    return 1;
+
   // Most instructions can be placed in any decoder slot.
   return 0;
 }
 
 bool SystemZHazardRecognizer::isFPdOpPreferred_distance(SUnit *SU) const {
   assert (SU->isUnbuffered);
   // If this is the first FPd op, it should be scheduled high.
   if (LastFPdOpCycleIdx == UINT_MAX)

lib/Target/SystemZ/SystemZMachineScheduler.cpp

SUnit *SystemZPostRASchedStrategy::pickNode(bool &IsTopNode) {
 
   // If only one choice, return it.
   if (Available.size() == 1) {
     LLVM_DEBUG(dbgs() << "** Only one: ";
                HazardRec->dumpSU(*Available.begin(), dbgs()); dbgs() << "\n";);
     return *Available.begin();
   }
 
-  // All nodes that are possible to schedule are stored by in the
-  // Available set.
+  // All nodes that are possible to schedule are stored in the Available set.
   LLVM_DEBUG(dbgs() << "** Available: "; Available.dump(*HazardRec););
 
   Candidate Best;
   for (auto *SU : Available) {
 
     // SU is the next candidate to be compared against current Best.
     Candidate c(SU, *HazardRec);
 

test/CodeGen/SystemZ/vec-cmp-cmp-logic-select.ll

 ; CHECK-NEXT:    vmrhf %v19, %v26, %v26
 ; CHECK-NEXT:    vfchdb %v7, %v27, %v7
 ; CHECK-NEXT:    vfchdb %v6, %v25, %v6
 ; CHECK-NEXT:    vfchdb %v5, %v31, %v5
 ; CHECK-NEXT:    vfchdb %v4, %v29, %v4
 ; CHECK-NEXT:    vpkg %v6, %v6, %v7
 ; CHECK-NEXT:    vpkg %v4, %v4, %v5
 ; CHECK-NEXT:    vn %v5, %v16, %v6
-; CHECK-NEXT:    vsel %v24, %v3, %v2, %v5
-; CHECK-NEXT:    vldeb %v17, %v17
+; CHECK-DAG:    vsel %v24, %v3, %v2, %v5
+; CHECK-DAG:    vldeb %v17, %v17
 ; CHECK-NEXT:    vldeb %v18, %v18
 ; CHECK-NEXT:    vfchdb %v17, %v18, %v17
 ; CHECK-NEXT:    vmrhf %v18, %v30, %v30
 ; CHECK-NEXT:    vldeb %v18, %v18
 ; CHECK-NEXT:    vldeb %v19, %v19
 ; CHECK-NEXT:    vfchdb %v18, %v19, %v18
 ; CHECK-NEXT:    vpkg %v17, %v18, %v17
 ; CHECK-NEXT:    vn %v4, %v17, %v4