[NFC] Add SchedState to allow forwarding the Scheduling state between MBB

[steven.zhang]

Summary

This is a NFC patch to improve the Generic scheduler to allow forwarding the scheduling state between MBB. A new class named SchedState is added to keep all the scheduling state that could be forwarded to next MBB.
Later patch would be committed to leverage this to improve the scheduler for some target.

If MBB2 has single pred MBB1, the MBB1's scheduler state could be forwarded to MBB2 to model the scheduler more precise.

[fhahn]

On a first look, the patch looks useful! Any chance to provide the follow-up patch that uses it as well? that would be helpful to see the bigger picture. Also, are you seeing any benefits by preserving the state?

llvm/include/llvm/CodeGen/MachineScheduler.h

604–605

Doc comment (///)?

609

Why do we need an explicit assignment operator? Does the default one not work?

716

Why not have getState() which returns const SchedState& and setState? This way, if we need a copy in the caller, it needs to be explicit.

I think it would also be better to do the restore at the point where we initialize the ScheduleBoundary, rather than providing a restore helper. Maybe have SchedBoundary take a SchedState object, which defaults to SchedState()?

llvm/lib/CodeGen/MachineScheduler.cpp

1834

Personally, I think it would be simpler to have the constructor create a 'resetted' object. In SchedBoundary::reset(), we then can either assign a 'resetted' object or a user provided one.

[jonpa]

I did something similar for the SystemZ PostRA scheduler. Is this patch taking that into consideration? I have not looked at this in detail, but it would be nice if the SystemZ implementation could use this or at least parts of it...

[steven.zhang]

In D59480#1433808, @jonpa wrote:

I did something similar for the SystemZ PostRA scheduler. Is this patch taking that into consideration? I have not looked at this in detail, but it would be nice if the SystemZ implementation could use this or at least parts of it...

Yes, I will take this into consideration. Thank you for the information.

[steven.zhang]

In D59480#1432943, @fhahn wrote:

On a first look, the patch looks useful! Any chance to provide the follow-up patch that uses it as well? that would be helpful to see the bigger picture. Also, are you seeing any benefits by preserving the state?

I believe SystemZ has already seen some benefit basing on this, as it seems that, they have already committed some patches in SystemZ target to preserve the state. There will be 4 patches, including this one. 1 NFC patch to extend the MI scheduler to support the state forward to next MBB. 1 NFC patch for SystemZ to adopt the new infrastructure. And 1 new patch to enable this feature for PowerPC. I haven't seen the benefit from the spec2017 yet on PowerPC , and I am still investigating, In theory, we should have some benefit as the resource is modeled more precise.

[steven.zhang]

Address comments.

llvm/include/llvm/CodeGen/MachineScheduler.h

604–605

typo.

716

Sounds good, and that is exactly what I want to do.

llvm/lib/CodeGen/MachineScheduler.cpp

1834

Sounds good. I will leave the way how to reset/save/store the state into next NFC infrastructure patch if this one is accepted.

[steven.zhang]

gentle ping ...

[fhahn]

llvm/lib/CodeGen/MachineScheduler.cpp

1834

I think it would be helpful if you could post your follow-up patches on Phabricator as well, and link them. That would make it easier to see the full picture.

[steven.zhang]

ok. I will prepare another two patches first, so that, we could continue the review.

[steven.zhang]

@fhahn I have completed all the needed patches, and it would be great if you have the time to continue the review. This is the whole picture.
https://reviews.llvm.org/D61248 is the 1st patch to allow the schedule strategy to forward the schedule state between MBB.
https://reviews.llvm.org/D61249 is the 2nd patch to update the schedule strategy for SystemZ target to adapt with 1st patch.
https://reviews.llvm.org/D59480 (this patch) is the 3rd patch to add the scheduled state data structure, so that, it could be kept somewhere.
https://reviews.llvm.org/D61250 is the last patch to forward the scheduled state 3rd patch added for PostGenericScheduler and enable it for PowerPC target.

[steven.zhang]

Gentle ping ...

[steven.zhang]

Gentle ping ... Thank you!

[fhahn]

Thanks for the update. I think it's better structured now for preserving the relevant state. It would be good to see if @atrick and @MatzeB are happy with that direction as well.

llvm/lib/CodeGen/MachineScheduler.cpp

1834

I guess this could go into the header.

1919–1920

I guess with this change, it would also make sense to use the getXXX functions here and elsewhere, instead of manually getting them from State.

[steven.zhang]

Address reviewer's comments.

• Move the ctor to the header.
• Declare the SchedState as "class" instead of "struct" (Is it ok?).
• Rebase the patch to master and resolve the conflict.

[steven.zhang]

llvm/lib/CodeGen/MachineScheduler.cpp

1834

ok

1919–1920

The State will be read and updated by SchedBoundary. i.e.

RetiredMOps += IncMOps

If we want to have getXXX, we have to provide another setXXX, and my understanding is that, it is an internal data structure for SchedBoundary. Maybe, we could declare it as class, and it is a friend class of SchedBoundary ?

[jonpa]

please rebase

llvm/include/llvm/CodeGen/MachineScheduler.h

void reset() {
     RemIssueCount = 0;
     IsAcyclicLatencyLimited = false;
     RemainingCounts.clear();
   }
 
   void init(ScheduleDAGMI *DAG, const TargetSchedModel *SchedModel);
 };
 
-/// Each Scheduling boundary is associated with ready queues. It tracks the
-/// current cycle in the direction of movement, and maintains the state
-/// of "hazards" and other interlocks at the current cycle.
-class SchedBoundary {
-public:
-  /// SUnit::NodeQueueId: 0 (none), 1 (top), 2 (bot), 3 (both)
-  enum {
-    TopQID = 1,
-    BotQID = 2,
-    LogMaxQID = 2
-  };
-
-  ScheduleDAGMI *DAG = nullptr;
-  const TargetSchedModel *SchedModel = nullptr;
-  SchedRemainder *Rem = nullptr;
-
-  ReadyQueue Available;
-  ReadyQueue Pending;
-
-  ScheduleHazardRecognizer *HazardRec = nullptr;
-
+/// Each SchedState is associated with SchedBoundary. It tracks the resource
+/// state in the direction of movement.
+class SchedState {
+  friend class SchedBoundary;
 private:
-  /// True if the pending Q should be checked/updated before scheduling another
-  /// instruction.
-  bool CheckPending;
-
   /// Number of cycles it takes to issue the instructions scheduled in this
   /// zone. It is defined as: scheduled-micro-ops / issue-width + stalls.
   /// See getStalls().
   unsigned CurrCycle;
 
   /// Micro-ops issued in the current cycle
   unsigned CurrMOps;
 
   /// MinReadyCycle - Cycle of the soonest available instruction.
   unsigned MinReadyCycle;
 
   // The expected latency of the critical path in this scheduled zone.
   unsigned ExpectedLatency;
private:
   // it.
   SmallVector<unsigned, 16> ReservedCyclesIndex;
 
 #ifndef NDEBUG
   // Remember the greatest possible stall as an upper bound on the number of
   // times we should retry the pending queue because of a hazard.
   unsigned MaxObservedStall;
 #endif
+public:
+  SchedState() :  CurrCycle(0), CurrMOps(0),
+    MinReadyCycle(std::numeric_limits<unsigned>::max()), ExpectedLatency(0),
+    DependentLatency(0), RetiredMOps(0), MaxExecutedResCount(0), 
+    ZoneCritResIdx(0), IsResourceLimited(false)
+  #ifndef NDEBUG
+    // Track the maximum number of stall cycles that could arise either from the
+    // latency of a DAG edge or the number of cycles that a processor resource
+    // is reserved (SchedState::ReservedCycles).
+    , MaxObservedStall(0)
+  #endif
+  {
+  // Reserve a zero-count for invalid CritResIdx.
+  ExecutedResCounts.resize(1);
+  assert(!ExecutedResCounts[0] && "nonzero count for bad resource");
+  }
+};
 
+/// Each Scheduling boundary is associated with ready queues. It tracks the
+/// current cycle in the direction of movement, and maintains the state
+/// of "hazards" and other interlocks at the current cycle.
+class SchedBoundary {
+public:
+  /// SUnit::NodeQueueId: 0 (none), 1 (top), 2 (bot), 3 (both)
+  enum {
+    TopQID = 1,
+    BotQID = 2,
+    LogMaxQID = 2
+  };
+
+  ScheduleDAGMI *DAG = nullptr;
+  const TargetSchedModel *SchedModel = nullptr;
+  SchedRemainder *Rem = nullptr;
+
+  ReadyQueue Available;
+  ReadyQueue Pending;
+
+  ScheduleHazardRecognizer *HazardRec = nullptr;
+
+private:
+  /// True if the pending Q should be checked/updated before scheduling another
+  /// instruction.
+  bool CheckPending;
+
+  /// Scheduling resource state.
+  SchedState State;
 public:
   /// Pending queues extend the ready queues with the same ID and the
   /// PendingFlag set.
   SchedBoundary(unsigned ID, const Twine &Name):
     Available(ID, Name+".A"), Pending(ID << LogMaxQID, Name+".P") {
     reset();
   }
 
   ~SchedBoundary();
 
   void reset();
 
   void init(ScheduleDAGMI *dag, const TargetSchedModel *smodel,
             SchedRemainder *rem);
 
   bool isTop() const {
     return Available.getID() == TopQID;
   }
 
   /// Number of cycles to issue the instructions scheduled in this zone.
-  unsigned getCurrCycle() const { return CurrCycle; }
+  unsigned getCurrCycle() const { return State.CurrCycle; }
 
   /// Micro-ops issued in the current cycle
-  unsigned getCurrMOps() const { return CurrMOps; }
+  unsigned getCurrMOps() const { return State.CurrMOps; }
 
   // The latency of dependence chains leading into this zone.
-  unsigned getDependentLatency() const { return DependentLatency; }
+  unsigned getDependentLatency() const { return State.DependentLatency; }
 
   /// Get the number of latency cycles "covered" by the scheduled
   /// instructions. This is the larger of the critical path within the zone
   /// and the number of cycles required to issue the instructions.
   unsigned getScheduledLatency() const {
-    return std::max(ExpectedLatency, CurrCycle);
+    return std::max(State.ExpectedLatency, State.CurrCycle);
   }
 
   unsigned getUnscheduledLatency(SUnit *SU) const {
     return isTop() ? SU->getHeight() : SU->getDepth();
   }
 
   unsigned getResourceCount(unsigned ResIdx) const {
-    return ExecutedResCounts[ResIdx];
+    return State.ExecutedResCounts[ResIdx];
   }
 
   /// Get the scaled count of scheduled micro-ops and resources, including
   /// executed resources.
   unsigned getCriticalCount() const {
-    if (!ZoneCritResIdx)
-      return RetiredMOps * SchedModel->getMicroOpFactor();
-    return getResourceCount(ZoneCritResIdx);
+    if (!State.ZoneCritResIdx)
+      return State.RetiredMOps * SchedModel->getMicroOpFactor();
+    return getResourceCount(State.ZoneCritResIdx);
   }
 
   /// Get a scaled count for the minimum execution time of the scheduled
   /// micro-ops that are ready to execute by getExecutedCount. Notice the
   /// feedback loop.
   unsigned getExecutedCount() const {
-    return std::max(CurrCycle * SchedModel->getLatencyFactor(),
-                    MaxExecutedResCount);
+    return std::max(State.CurrCycle * SchedModel->getLatencyFactor(),
+                    State.MaxExecutedResCount);
   }
 
-  unsigned getZoneCritResIdx() const { return ZoneCritResIdx; }
+  unsigned getZoneCritResIdx() const { return State.ZoneCritResIdx; }
 
   // Is the scheduled region resource limited vs. latency limited.
-  bool isResourceLimited() const { return IsResourceLimited; }
+  bool isResourceLimited() const { return State.IsResourceLimited; }
 
   /// Get the difference between the given SUnit's ready time and the current
   /// cycle.
   unsigned getLatencyStallCycles(SUnit *SU);
 
   unsigned getNextResourceCycleByInstance(unsigned InstanceIndex,
                                           unsigned Cycles);
 

 /// PostGenericScheduler - Interface to the scheduling algorithm used by
 /// ScheduleDAGMI.
 ///
 /// Callbacks from ScheduleDAGMI:
 ///   initPolicy -> initialize(DAG) -> registerRoots -> pickNode ...
 class PostGenericScheduler : public GenericSchedulerBase {
 protected:
   ScheduleDAGMI *DAG;
-  SchedBoundary Top;
   SmallVector<SUnit*, 8> BotRoots;
 
+protected:
+  SchedBoundary Top;
+
 public:
   PostGenericScheduler(const MachineSchedContext *C):
     GenericSchedulerBase(C), Top(SchedBoundary::TopQID, "TopQ") {}
 
   ~PostGenericScheduler() override = default;
 
   void initPolicy(MachineBasicBlock::iterator Begin,
                   MachineBasicBlock::iterator End,

llvm/lib/CodeGen/MachineScheduler.cpp

void CopyConstrain::apply(ScheduleDAGInstrs *DAGInstrs) {
   }
 }
 
 //===----------------------------------------------------------------------===//
 // MachineSchedStrategy helpers used by GenericScheduler, GenericPostScheduler
 // and possibly other custom schedulers.
 //===----------------------------------------------------------------------===//
 
 static const unsigned InvalidCycle = ~0U;
 
 SchedBoundary::~SchedBoundary() { delete HazardRec; }
 
 /// Given a Count of resource usage and a Latency value, return true if a
 /// SchedBoundary becomes resource limited.
 /// If we are checking after scheduling a node, we should return true when
 /// we just reach the resource limit.
 static bool checkResourceLimit(unsigned LFactor, unsigned Count,
void SchedBoundary::reset() {
   // invalid, placeholder HazardRecs.
   if (HazardRec && HazardRec->isEnabled()) {
     delete HazardRec;
     HazardRec = nullptr;
   }
   Available.clear();
   Pending.clear();
   CheckPending = false;
-  CurrCycle = 0;
-  CurrMOps = 0;
-  MinReadyCycle = std::numeric_limits<unsigned>::max();
-  ExpectedLatency = 0;
-  DependentLatency = 0;
-  RetiredMOps = 0;
-  MaxExecutedResCount = 0;
-  ZoneCritResIdx = 0;
-  IsResourceLimited = false;
-  ReservedCycles.clear();
-  ReservedCyclesIndex.clear();
-#ifndef NDEBUG
-  // Track the maximum number of stall cycles that could arise either from the
-  // latency of a DAG edge or the number of cycles that a processor resource is
-  // reserved (SchedBoundary::ReservedCycles).
-  MaxObservedStall = 0;
-#endif
-  // Reserve a zero-count for invalid CritResIdx.
-  ExecutedResCounts.resize(1);
-  assert(!ExecutedResCounts[0] && "nonzero count for bad resource");
+
+  State = SchedState();
 }
 
 void SchedRemainder::
 init(ScheduleDAGMI *DAG, const TargetSchedModel *SchedModel) {
   reset();
   if (!SchedModel->hasInstrSchedModel())
     return;
   RemainingCounts.resize(SchedModel->getNumProcResourceKinds());

 void SchedBoundary::
 init(ScheduleDAGMI *dag, const TargetSchedModel *smodel, SchedRemainder *rem) {
   reset();
   DAG = dag;
   SchedModel = smodel;
   Rem = rem;
   if (SchedModel->hasInstrSchedModel()) {
     unsigned ResourceCount = SchedModel->getNumProcResourceKinds();
-    ReservedCyclesIndex.resize(ResourceCount);
-    ExecutedResCounts.resize(ResourceCount);
+    State.ReservedCyclesIndex.resize(ResourceCount);
+    State.ExecutedResCounts.resize(ResourceCount);
     unsigned NumUnits = 0;
 
     for (unsigned i = 0; i < ResourceCount; ++i) {
-      ReservedCyclesIndex[i] = NumUnits;
+      State.ReservedCyclesIndex[i] = NumUnits;
       NumUnits += SchedModel->getProcResource(i)->NumUnits;
     }
 
-    ReservedCycles.resize(NumUnits, InvalidCycle);
+    State.ReservedCycles.resize(NumUnits, InvalidCycle);
   }
 }
 
 /// Compute the stall cycles based on this SUnit's ready time. Heuristics treat
 /// these "soft stalls" differently than the hard stall cycles based on CPU
 /// resources and computed by checkHazard(). A fully in-order model
 /// (MicroOpBufferSize==0) will not make use of this since instructions are not
 /// available for scheduling until they are ready. However, a weaker in-order
 /// model may use this for heuristics. For example, if a processor has in-order
 /// behavior when reading certain resources, this may come into play.
 unsigned SchedBoundary::getLatencyStallCycles(SUnit *SU) {
   if (!SU->isUnbuffered)
     return 0;
 
   unsigned ReadyCycle = (isTop() ? SU->TopReadyCycle : SU->BotReadyCycle);
-  if (ReadyCycle > CurrCycle)
-    return ReadyCycle - CurrCycle;
+  if (ReadyCycle > State.CurrCycle)
+    return ReadyCycle - State.CurrCycle;
   return 0;
 }
 
 /// Compute the next cycle at which the given processor resource unit
 /// can be scheduled.
 unsigned SchedBoundary::getNextResourceCycleByInstance(unsigned InstanceIdx,
                                                        unsigned Cycles) {
-  unsigned NextUnreserved = ReservedCycles[InstanceIdx];
+  unsigned NextUnreserved = State.ReservedCycles[InstanceIdx];
   // If this resource has never been used, always return cycle zero.
   if (NextUnreserved == InvalidCycle)
     return 0;
   // For bottom-up scheduling add the cycles needed for the current operation.
   if (!isTop())
     NextUnreserved += Cycles;
   return NextUnreserved;
 }
 
 /// Compute the next cycle at which the given processor resource can be
 /// scheduled.  Returns the next cycle and the index of the processor resource
 /// instance in the reserved cycles vector.
 std::pair<unsigned, unsigned>
 SchedBoundary::getNextResourceCycle(unsigned PIdx, unsigned Cycles) {
   unsigned MinNextUnreserved = InvalidCycle;
   unsigned InstanceIdx = 0;
-  unsigned StartIndex = ReservedCyclesIndex[PIdx];
+  unsigned StartIndex = State.ReservedCyclesIndex[PIdx];
   unsigned NumberOfInstances = SchedModel->getProcResource(PIdx)->NumUnits;
   assert(NumberOfInstances > 0 &&
          "Cannot have zero instances of a ProcResource");
 
   for (unsigned I = StartIndex, End = StartIndex + NumberOfInstances; I < End;
        ++I) {
     unsigned NextUnreserved = getNextResourceCycleByInstance(I, Cycles);
     if (MinNextUnreserved > NextUnreserved) {

 /// TODO: Also check whether the SU must start a new group.
 bool SchedBoundary::checkHazard(SUnit *SU) {
   if (HazardRec->isEnabled()
       && HazardRec->getHazardType(SU) != ScheduleHazardRecognizer::NoHazard) {
     return true;
   }
 
   unsigned uops = SchedModel->getNumMicroOps(SU->getInstr());
-  if ((CurrMOps > 0) && (CurrMOps + uops > SchedModel->getIssueWidth())) {
+  if ((State.CurrMOps > 0) && 
+      (State.CurrMOps + uops > SchedModel->getIssueWidth())) {
     LLVM_DEBUG(dbgs() << "  SU(" << SU->NodeNum << ") uops="
                       << SchedModel->getNumMicroOps(SU->getInstr()) << '\n');
     return true;
   }
 
-  if (CurrMOps > 0 &&
+  if (State.CurrMOps > 0 &&
       ((isTop() && SchedModel->mustBeginGroup(SU->getInstr())) ||
        (!isTop() && SchedModel->mustEndGroup(SU->getInstr())))) {
     LLVM_DEBUG(dbgs() << "  hazard: SU(" << SU->NodeNum << ") must "
                       << (isTop() ? "begin" : "end") << " group\n");
     return true;
   }
 
   if (SchedModel->hasInstrSchedModel() && SU->hasReservedResource) {
     const MCSchedClassDesc *SC = DAG->getSchedClass(SU);
     for (const MCWriteProcResEntry &PE :
           make_range(SchedModel->getWriteProcResBegin(SC),
                      SchedModel->getWriteProcResEnd(SC))) {
       unsigned ResIdx = PE.ProcResourceIdx;
       unsigned Cycles = PE.Cycles;
       unsigned NRCycle, InstanceIdx;
       std::tie(NRCycle, InstanceIdx) = getNextResourceCycle(ResIdx, Cycles);
-      if (NRCycle > CurrCycle) {
+      if (NRCycle > State.CurrCycle) {
 #ifndef NDEBUG
-        MaxObservedStall = std::max(Cycles, MaxObservedStall);
+        State.MaxObservedStall = std::max(Cycles, State.MaxObservedStall);
 #endif
         LLVM_DEBUG(dbgs() << "  SU(" << SU->NodeNum << ") "
-                          << SchedModel->getResourceName(ResIdx)
-                          << '[' << InstanceIdx - ReservedCyclesIndex[ResIdx]  << ']'
-                          << "=" << NRCycle << "c\n");
+                          << SchedModel->getResourceName(ResIdx) << '[' 
+                          << InstanceIdx - State.ReservedCyclesIndex[ResIdx]
+                          << ']' << "=" << NRCycle << "c\n");
         return true;
       }
     }
   }
   return false;
 }
 
 // Find the unscheduled node in ReadySUs with the highest latency.

 // resource index, or zero if the zone is issue limited.
 unsigned SchedBoundary::
 getOtherResourceCount(unsigned &OtherCritIdx) {
   OtherCritIdx = 0;
   if (!SchedModel->hasInstrSchedModel())
     return 0;
 
   unsigned OtherCritCount = Rem->RemIssueCount
-    + (RetiredMOps * SchedModel->getMicroOpFactor());
+    + (State.RetiredMOps * SchedModel->getMicroOpFactor());
   LLVM_DEBUG(dbgs() << "  " << Available.getName() << " + Remain MOps: "
                     << OtherCritCount / SchedModel->getMicroOpFactor() << '\n');
   for (unsigned PIdx = 1, PEnd = SchedModel->getNumProcResourceKinds();
        PIdx != PEnd; ++PIdx) {
     unsigned OtherCount = getResourceCount(PIdx) + Rem->RemainingCounts[PIdx];
     if (OtherCount > OtherCritCount) {
       OtherCritCount = OtherCount;
       OtherCritIdx = PIdx;

 
 void SchedBoundary::releaseNode(SUnit *SU, unsigned ReadyCycle) {
   assert(SU->getInstr() && "Scheduled SUnit must have instr");
 
 #ifndef NDEBUG
   // ReadyCycle was been bumped up to the CurrCycle when this node was
   // scheduled, but CurrCycle may have been eagerly advanced immediately after
   // scheduling, so may now be greater than ReadyCycle.
-  if (ReadyCycle > CurrCycle)
-    MaxObservedStall = std::max(ReadyCycle - CurrCycle, MaxObservedStall);
+  if (ReadyCycle > State.CurrCycle)
+    State.MaxObservedStall =
+    std::max(ReadyCycle - State.CurrCycle, State.MaxObservedStall);
 #endif
 
-  if (ReadyCycle < MinReadyCycle)
-    MinReadyCycle = ReadyCycle;
+  if (ReadyCycle < State.MinReadyCycle)
+    State.MinReadyCycle = ReadyCycle;
 
   // Check for interlocks first. For the purpose of other heuristics, an
   // instruction that cannot issue appears as if it's not in the ReadyQueue.
   bool IsBuffered = SchedModel->getMicroOpBufferSize() != 0;
-  if ((!IsBuffered && ReadyCycle > CurrCycle) || checkHazard(SU) ||
+  if ((!IsBuffered && ReadyCycle > State.CurrCycle) || checkHazard(SU) ||
       Available.size() >= ReadyListLimit)
     Pending.push(SU);
   else
     Available.push(SU);
 }
 
 /// Move the boundary of scheduled code by one cycle.
 void SchedBoundary::bumpCycle(unsigned NextCycle) {
   if (SchedModel->getMicroOpBufferSize() == 0) {
-    assert(MinReadyCycle < std::numeric_limits<unsigned>::max() &&
+    assert(State.MinReadyCycle < std::numeric_limits<unsigned>::max() &&
            "MinReadyCycle uninitialized");
-    if (MinReadyCycle > NextCycle)
-      NextCycle = MinReadyCycle;
+    if (State.MinReadyCycle > NextCycle)
+      NextCycle = State.MinReadyCycle;
   }
   // Update the current micro-ops, which will issue in the next cycle.
-  unsigned DecMOps = SchedModel->getIssueWidth() * (NextCycle - CurrCycle);
-  CurrMOps = (CurrMOps <= DecMOps) ? 0 : CurrMOps - DecMOps;
+  unsigned DecMOps = SchedModel->getIssueWidth() *
+    (NextCycle - State.CurrCycle);
+  State.CurrMOps = (State.CurrMOps <= DecMOps) ? 0 : State.CurrMOps - DecMOps;
 
   // Decrement DependentLatency based on the next cycle.
-  if ((NextCycle - CurrCycle) > DependentLatency)
-    DependentLatency = 0;
+  if ((NextCycle - State.CurrCycle) > State.DependentLatency)
+    State.DependentLatency = 0;
   else
-    DependentLatency -= (NextCycle - CurrCycle);
+    State.DependentLatency -= (NextCycle - State.CurrCycle);
 
   if (!HazardRec->isEnabled()) {
     // Bypass HazardRec virtual calls.
-    CurrCycle = NextCycle;
+    State.CurrCycle = NextCycle;
   } else {
     // Bypass getHazardType calls in case of long latency.
-    for (; CurrCycle != NextCycle; ++CurrCycle) {
+    for (; State.CurrCycle != NextCycle; ++State.CurrCycle) {
       if (isTop())
         HazardRec->AdvanceCycle();
       else
         HazardRec->RecedeCycle();
     }
   }
   CheckPending = true;
-  IsResourceLimited =
+  State.IsResourceLimited =
       checkResourceLimit(SchedModel->getLatencyFactor(), getCriticalCount(),
                          getScheduledLatency(), true);
 
-  LLVM_DEBUG(dbgs() << "Cycle: " << CurrCycle << ' ' << Available.getName()
-                    << '\n');
+  LLVM_DEBUG(dbgs() << "Cycle: " << State.CurrCycle << ' '
+                    << Available.getName() << '\n');
 }
 
 void SchedBoundary::incExecutedResources(unsigned PIdx, unsigned Count) {
-  ExecutedResCounts[PIdx] += Count;
-  if (ExecutedResCounts[PIdx] > MaxExecutedResCount)
-    MaxExecutedResCount = ExecutedResCounts[PIdx];
+  State.ExecutedResCounts[PIdx] += Count;
+  if (State.ExecutedResCounts[PIdx] > State.MaxExecutedResCount)
+    State.MaxExecutedResCount = State.ExecutedResCounts[PIdx];
 }
 
 /// Add the given processor resource to this scheduled zone.
 ///
 /// \param Cycles indicates the number of consecutive (non-pipelined) cycles
 /// during which this resource is consumed.
 ///
 /// \return the next cycle at which the instruction may execute without
 /// oversubscribing resources.
 unsigned SchedBoundary::
 countResource(unsigned PIdx, unsigned Cycles, unsigned NextCycle) {
   unsigned Factor = SchedModel->getResourceFactor(PIdx);
   unsigned Count = Factor * Cycles;
   LLVM_DEBUG(dbgs() << "  " << SchedModel->getResourceName(PIdx) << " +"
                     << Cycles << "x" << Factor << "u\n");
 
   // Update Executed resources counts.
   incExecutedResources(PIdx, Count);
   assert(Rem->RemainingCounts[PIdx] >= Count && "resource double counted");
   Rem->RemainingCounts[PIdx] -= Count;
 
   // Check if this resource exceeds the current critical resource. If so, it
   // becomes the critical resource.
-  if (ZoneCritResIdx != PIdx && (getResourceCount(PIdx) > getCriticalCount())) {
-    ZoneCritResIdx = PIdx;
+  if (State.ZoneCritResIdx != PIdx &&
+      (getResourceCount(PIdx) > getCriticalCount())) {
+    State.ZoneCritResIdx = PIdx;
     LLVM_DEBUG(dbgs() << "  *** Critical resource "
                       << SchedModel->getResourceName(PIdx) << ": "
                       << getResourceCount(PIdx) / SchedModel->getLatencyFactor()
                       << "c\n");
   }
   // For reserved resources, record the highest cycle using the resource.
   unsigned NextAvailable, InstanceIdx;
   std::tie(NextAvailable, InstanceIdx) = getNextResourceCycle(PIdx, Cycles);
-  if (NextAvailable > CurrCycle) {
+  if (NextAvailable > State.CurrCycle) {
     LLVM_DEBUG(dbgs() << "  Resource conflict: "
-                      << SchedModel->getResourceName(PIdx)
-                      << '[' << InstanceIdx - ReservedCyclesIndex[PIdx]  << ']'
+                      << SchedModel->getResourceName(PIdx) << '[' 
+                      << InstanceIdx - State.ReservedCyclesIndex[PIdx]  << ']'
                       << " reserved until @" << NextAvailable << "\n");
   }
   return NextAvailable;
 }
 
 /// Move the boundary of scheduled code by one SUnit.
 void SchedBoundary::bumpNode(SUnit *SU) {
   // Update the reservation table.
   if (HazardRec->isEnabled()) {
     if (!isTop() && SU->isCall) {
       // Calls are scheduled with their preceding instructions. For bottom-up
       // scheduling, clear the pipeline state before emitting.
       HazardRec->Reset();
     }
     HazardRec->EmitInstruction(SU);
     // Scheduling an instruction may have made pending instructions available.
     CheckPending = true;
   }
   // checkHazard should prevent scheduling multiple instructions per cycle that
   // exceed the issue width.
   const MCSchedClassDesc *SC = DAG->getSchedClass(SU);
   unsigned IncMOps = SchedModel->getNumMicroOps(SU->getInstr());
-  assert(
-      (CurrMOps == 0 || (CurrMOps + IncMOps) <= SchedModel->getIssueWidth()) &&
+  assert((State.CurrMOps == 0 ||
+          (State.CurrMOps + IncMOps) <= SchedModel->getIssueWidth()) &&
          "Cannot schedule this instruction's MicroOps in the current cycle.");
 
   unsigned ReadyCycle = (isTop() ? SU->TopReadyCycle : SU->BotReadyCycle);
   LLVM_DEBUG(dbgs() << "  Ready @" << ReadyCycle << "c\n");
 
-  unsigned NextCycle = CurrCycle;
+  unsigned NextCycle = State.CurrCycle;
   switch (SchedModel->getMicroOpBufferSize()) {
   case 0:
-    assert(ReadyCycle <= CurrCycle && "Broken PendingQueue");
+    assert(ReadyCycle <= State.CurrCycle && "Broken PendingQueue");
     break;
   case 1:
     if (ReadyCycle > NextCycle) {
       NextCycle = ReadyCycle;
       LLVM_DEBUG(dbgs() << "  *** Stall until: " << ReadyCycle << "\n");
     }
     break;
   default:
     // We don't currently model the OOO reorder buffer, so consider all
     // scheduled MOps to be "retired". We do loosely model in-order resource
     // latency. If this instruction uses an in-order resource, account for any
     // likely stall cycles.
     if (SU->isUnbuffered && ReadyCycle > NextCycle)
       NextCycle = ReadyCycle;
     break;
   }
-  RetiredMOps += IncMOps;
+  State.RetiredMOps += IncMOps;
 
   // Update resource counts and critical resource.
   if (SchedModel->hasInstrSchedModel()) {
     unsigned DecRemIssue = IncMOps * SchedModel->getMicroOpFactor();
     assert(Rem->RemIssueCount >= DecRemIssue && "MOps double counted");
     Rem->RemIssueCount -= DecRemIssue;
-    if (ZoneCritResIdx) {
+    if (State.ZoneCritResIdx) {
       // Scale scheduled micro-ops for comparing with the critical resource.
       unsigned ScaledMOps =
-        RetiredMOps * SchedModel->getMicroOpFactor();
+        State.RetiredMOps * SchedModel->getMicroOpFactor();
 
       // If scaled micro-ops are now more than the previous critical resource by
       // a full cycle, then micro-ops issue becomes critical.
-      if ((int)(ScaledMOps - getResourceCount(ZoneCritResIdx))
+      if ((int)(ScaledMOps - getResourceCount(State.ZoneCritResIdx))
           >= (int)SchedModel->getLatencyFactor()) {
-        ZoneCritResIdx = 0;
+        State.ZoneCritResIdx = 0;
         LLVM_DEBUG(dbgs() << "  *** Critical resource NumMicroOps: "
                           << ScaledMOps / SchedModel->getLatencyFactor()
                           << "c\n");
       }
     }
     for (TargetSchedModel::ProcResIter
            PI = SchedModel->getWriteProcResBegin(SC),
            PE = SchedModel->getWriteProcResEnd(SC); PI != PE; ++PI) {
if (SU->hasReservedResource) {
       for (TargetSchedModel::ProcResIter
              PI = SchedModel->getWriteProcResBegin(SC),
              PE = SchedModel->getWriteProcResEnd(SC); PI != PE; ++PI) {
         unsigned PIdx = PI->ProcResourceIdx;
         if (SchedModel->getProcResource(PIdx)->BufferSize == 0) {
           unsigned ReservedUntil, InstanceIdx;
           std::tie(ReservedUntil, InstanceIdx) = getNextResourceCycle(PIdx, 0);
           if (isTop()) {
-            ReservedCycles[InstanceIdx] =
+            State.ReservedCycles[InstanceIdx] =
                 std::max(ReservedUntil, NextCycle + PI->Cycles);
           } else
-            ReservedCycles[InstanceIdx] = NextCycle;
+            State.ReservedCycles[InstanceIdx] = NextCycle;
         }
       }
     }
   }
   // Update ExpectedLatency and DependentLatency.
-  unsigned &TopLatency = isTop() ? ExpectedLatency : DependentLatency;
-  unsigned &BotLatency = isTop() ? DependentLatency : ExpectedLatency;
+  unsigned &TopLatency = isTop() ?
+    State.ExpectedLatency : State.DependentLatency;
+  unsigned &BotLatency = isTop() ?
+    State.DependentLatency : State.ExpectedLatency;
   if (SU->getDepth() > TopLatency) {
     TopLatency = SU->getDepth();
     LLVM_DEBUG(dbgs() << "  " << Available.getName() << " TopLatency SU("
                       << SU->NodeNum << ") " << TopLatency << "c\n");
   }
   if (SU->getHeight() > BotLatency) {
     BotLatency = SU->getHeight();
     LLVM_DEBUG(dbgs() << "  " << Available.getName() << " BotLatency SU("
                       << SU->NodeNum << ") " << BotLatency << "c\n");
   }
   // If we stall for any reason, bump the cycle.
-  if (NextCycle > CurrCycle)
+  if (NextCycle > State.CurrCycle)
     bumpCycle(NextCycle);
   else
     // After updating ZoneCritResIdx and ExpectedLatency, check if we're
     // resource limited. If a stall occurred, bumpCycle does this.
-    IsResourceLimited =
+    State.IsResourceLimited =
         checkResourceLimit(SchedModel->getLatencyFactor(), getCriticalCount(),
                            getScheduledLatency(), true);
 
   // Update CurrMOps after calling bumpCycle to handle stalls, since bumpCycle
   // resets CurrMOps. Loop to handle instructions with more MOps than issue in
   // one cycle.  Since we commonly reach the max MOps here, opportunistically
   // bump the cycle to avoid uselessly checking everything in the readyQ.
-  CurrMOps += IncMOps;
+  State.CurrMOps += IncMOps;
 
   // Bump the cycle count for issue group constraints.
   // This must be done after NextCycle has been adjust for all other stalls.
   // Calling bumpCycle(X) will reduce CurrMOps by one issue group and set
   // currCycle to X.
   if ((isTop() &&  SchedModel->mustEndGroup(SU->getInstr())) ||
       (!isTop() && SchedModel->mustBeginGroup(SU->getInstr()))) {
     LLVM_DEBUG(dbgs() << "  Bump cycle to " << (isTop() ? "end" : "begin")
                       << " group\n");
     bumpCycle(++NextCycle);
   }
 
-  while (CurrMOps >= SchedModel->getIssueWidth()) {
-    LLVM_DEBUG(dbgs() << "  *** Max MOps " << CurrMOps << " at cycle "
-                      << CurrCycle << '\n');
+  while (State.CurrMOps >= SchedModel->getIssueWidth()) {
+    LLVM_DEBUG(dbgs() << "  *** Max MOps " << State.CurrMOps << " at cycle "
+                      << State.CurrCycle << '\n');
     bumpCycle(++NextCycle);
   }
   LLVM_DEBUG(dumpScheduledState());
 }
 
 /// Release pending ready nodes in to the available queue. This makes them
 /// visible to heuristics.
 void SchedBoundary::releasePending() {
   // If the available queue is empty, it is safe to reset MinReadyCycle.
   if (Available.empty())
-    MinReadyCycle = std::numeric_limits<unsigned>::max();
+    State.MinReadyCycle = std::numeric_limits<unsigned>::max();
 
   // Check to see if any of the pending instructions are ready to issue.  If
   // so, add them to the available queue.
   bool IsBuffered = SchedModel->getMicroOpBufferSize() != 0;
   for (unsigned i = 0, e = Pending.size(); i != e; ++i) {
     SUnit *SU = *(Pending.begin()+i);
     unsigned ReadyCycle = isTop() ? SU->TopReadyCycle : SU->BotReadyCycle;
 
-    if (ReadyCycle < MinReadyCycle)
-      MinReadyCycle = ReadyCycle;
+    if (ReadyCycle < State.MinReadyCycle)
+      State.MinReadyCycle = ReadyCycle;
 
-    if (!IsBuffered && ReadyCycle > CurrCycle)
+    if (!IsBuffered && ReadyCycle > State.CurrCycle)
       continue;
 
     if (checkHazard(SU))
       continue;
 
     if (Available.size() >= ReadyListLimit)
       break;
 

 
 /// If this queue only has one ready candidate, return it. As a side effect,
 /// defer any nodes that now hit a hazard, and advance the cycle until at least
 /// one node is ready. If multiple instructions are ready, return NULL.
 SUnit *SchedBoundary::pickOnlyChoice() {
   if (CheckPending)
     releasePending();
 
-  if (CurrMOps > 0) {
+  if (State.CurrMOps > 0) {
     // Defer any ready instrs that now have a hazard.
     for (ReadyQueue::iterator I = Available.begin(); I != Available.end();) {
       if (checkHazard(*I)) {
         Pending.push(*I);
         I = Available.remove(I);
         continue;
       }
       ++I;
     }
   }
   for (unsigned i = 0; Available.empty(); ++i) {
 //  FIXME: Re-enable assert once PR20057 is resolved.
 //    assert(i <= (HazardRec->getMaxLookAhead() + MaxObservedStall) &&
 //           "permanent hazard");
     (void)i;
-    bumpCycle(CurrCycle + 1);
+    bumpCycle(State.CurrCycle + 1);
     releasePending();
   }
 
   LLVM_DEBUG(Pending.dump());
   LLVM_DEBUG(Available.dump());
 
   if (Available.size() == 1)
     return *Available.begin();
   return nullptr;
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 // This is useful information to dump after bumpNode.
 // Note that the Queue contents are more useful before pickNodeFromQueue.
 LLVM_DUMP_METHOD void SchedBoundary::dumpScheduledState() const {
   unsigned ResFactor;
   unsigned ResCount;
-  if (ZoneCritResIdx) {
-    ResFactor = SchedModel->getResourceFactor(ZoneCritResIdx);
-    ResCount = getResourceCount(ZoneCritResIdx);
+  if (State.ZoneCritResIdx) {
+    ResFactor = SchedModel->getResourceFactor(State.ZoneCritResIdx);
+    ResCount = getResourceCount(State.ZoneCritResIdx);
   } else {
     ResFactor = SchedModel->getMicroOpFactor();
-    ResCount = RetiredMOps * ResFactor;
+    ResCount = State.RetiredMOps * ResFactor;
   }
   unsigned LFactor = SchedModel->getLatencyFactor();
-  dbgs() << Available.getName() << " @" << CurrCycle << "c\n"
-         << "  Retired: " << RetiredMOps;
+  dbgs() << Available.getName() << " @" << State.CurrCycle << "c\n"
+         << "  Retired: " << State.RetiredMOps;
   dbgs() << "\n  Executed: " << getExecutedCount() / LFactor << "c";
   dbgs() << "\n  Critical: " << ResCount / LFactor << "c, "
          << ResCount / ResFactor << " "
-         << SchedModel->getResourceName(ZoneCritResIdx)
-         << "\n  ExpectedLatency: " << ExpectedLatency << "c\n"
-         << (IsResourceLimited ? "  - Resource" : "  - Latency")
+         << SchedModel->getResourceName(State.ZoneCritResIdx)
+         << "\n  ExpectedLatency: " << State.ExpectedLatency << "c\n"
+         << (State.IsResourceLimited ? "  - Resource" : "  - Latency")
          << " limited.\n";
 }
 #endif
 
 //===----------------------------------------------------------------------===//
 // GenericScheduler - Generic implementation of MachineSchedStrategy.
 //===----------------------------------------------------------------------===//