[llvm-mca] Refactor event listeners to make the backend agnostic to event types.

[courbet]

Summary

This is a first step towards making the pipeline configurable.

[andreadb]

Thanks Clement.

Please run clang-format on the patch. I found that the code was not correctly formatted in several places.
See my comments below.

tools/llvm-mca/Backend.h

85

80-col.

tools/llvm-mca/BackendStatistics.cpp

24–36

clang-format.

tools/llvm-mca/BackendStatistics.h

122–123

Remove this change. Alternatively, run clang-format on this file.

123–124

clang-format.

tools/llvm-mca/Dispatch.cpp

154–167

80-col.

tools/llvm-mca/HWEventListener.h

24–27

We need a better (doxygen) comment here.

29–49

Could you please explain what is the long term goal? Do you plan to remove this EventType?

44

Period at end of sentence.

tools/llvm-mca/ResourcePressureView.cpp

48–50

You don't need braces here.

tools/llvm-mca/ResourcePressureView.h

63

Why this change?

tools/llvm-mca/Scheduler.h

457–458

Is the plan to remove this dependency in the long term?

[RKSimon]

Some minor style comments

tools/llvm-mca/Backend.h

85

Add an assertion before dereferencing pointer?

tools/llvm-mca/HWEventListener.h

64

UsedResources(UsedResources) ? scary - don't name arguments the same as class members

tools/llvm-mca/Scheduler.h

489

Avoid the same name for a argument as a member.

[courbet]

• clang-format
• more doc

[courbet]

Thanks, PTAL.

tools/llvm-mca/HWEventListener.h

24–27

oops.

29–49

I've given it a shot, tell me what you think.

tools/llvm-mca/Scheduler.h

457–458

Not really. If we want the Backend class to be agnostic to the pipeline structure, we need inter-component dependencies to be carried by the component themselves. The pipeline creation function (in the subtarget, see https://reviews.llvm.org/D43951?id=136547#inline-384507) should be the one that does that the plugging.

However it would be better if the DispatchUnit was given in the ctor, maybe through some wrapper object (but let's do that if the need arises).

[andreadb]

tools/llvm-mca/HWEventListener.h

30–35

Thanks for the description.

In my view, all the enum values in EventType should be known by all targets. Those should be all known concepts. This would minimize the impact of a redesign on the current views. Targets could then extend this with custom event types.
We should probably find a better name for this enum. Something like 'BaseEventType' or 'GenericEventType'. Also, please add a 'LastGenericEventType' enum value at the end.

73

s/simulator/backend pipeline

tools/llvm-mca/Scheduler.h

457–458

Essentially - correct me if I am wrong - your plan is to make dependencies between components (i.e. pipeline stages) explicit. Each component knows who to notify/interact with (like in this case).
The advantage is that the Backend is now no longer responsible for orchestrating the execution. This allows the definition of a simpler Backend class which acts as a container of stages, and starts the execution. Components will know how to coordinate with each other (since they _explicitly_ know their position in the pipeline).

If my understanding is correct, then this design is still quite rigid, since components now explicitly know other componets of the pipeline. That being said, we can still achieve modularity by letting users define their custom 'DispatchUnit'/'Scheduler'(etc.) classes with (potentially) different dependencies.

Please could you share (i.e. add as a description to this review) your design and give us more details on how you plan to restructure this code? Modularity can be achieved in many different ways. I just want to make sure that we all agree with the final design. At the moment, it is still a bit unclear what you have in mind.

[courbet]

tools/llvm-mca/HWEventListener.h

30–35

In my view, all the enum values in EventType should be known by all targets. Those should be all known concepts. This would minimize the impact of a redesign on the current views. Targets could then extend this with custom event types.

I fully agree. That's what I meant by "Of course some event types (e.g. Retired) can be shared among subtargets". Right now all the values in the enum apply to all subtargets, but as soon as the fonrtend is modelled, this won't be the case anymore (there are weird subtarget-specific things such as LCP stalls for Intel). I have made that clearer.

The point of the fixme was more about where this enum should be defined: I think eventually it should be in a base TD class that subtargets can extend. But that's not important right now.

We should probably find a better name for this enum.

Went for GenericEventType.

tools/llvm-mca/Scheduler.h

457–458

Essentially - correct me if I am wrong - your plan is to make dependencies between components (i.e. pipeline stages) explicit. Each component knows who to notify/interact with (like in this case).

The advantage is that the Backend is now no longer responsible for orchestrating the execution. This allows the definition of a simpler Backend class which acts as a container of stages, and starts the execution. Components will know how to coordinate with each other (since they _explicitly_ know their position in the pipeline).

Exactly.

If my understanding is correct, then this design is still quite rigid, since components now explicitly know other components of the pipeline.

In the current design, there are two things that are actually hardcoded:
(1) - The pipeline structure itself. This is because the Backend class has a list of exact components that it manipulates through their exact types (not through a Component of PipelineStage interface). If a subtarget wants to add a new component with type MyComponent, we need to add a std::unique_ptr<MyComponent> to class Backend, and add plumbing to let MyComponent communicate its state changes with listeners.
(2) - Interaction between components. This is because the interaction between HWS and DU` happens in Backend. If a subtarget wants to add a component in between HWS and DU, it has to modify classes HWS, DU and Backend.

This patch:

• Addresses the second part of (1): Components can emit new events, without changes to Backend.
• Improves (2) a bit: you now only have to change HWS` and DU, and not Backend.

I was actually considering this patch more as a refactoring (I'm not really changing anything ) than a change of design, to allow me to present a suggested design more easily in a next patch.

That being said, we can still achieve modularity by letting users define their custom 'DispatchUnit'/'Scheduler'(etc.) classes with (potentially) different dependencies.

Exactly. And we should also reduce how much Scheduler and DispatchUnit have to know about each other.
In our approach, all a component has to know about the previous (resp. next) element in the pipeline is what type of elements it creates (resp. accepts): Instruction/Uop/InsertYourType. Nothing has to be known about what the component is or does. That means you can substitute any component for another in the piplene as long as it has the same input and output type, and you can add a component in the middle of the pipeline as long as it manipulates elements without changing their type.
Without going as far as this, the only way we can ensure that subtargets can reuse "generic" components is to abstract a bit the type of the components.

[RKSimon]

tools/llvm-mca/TimelineView.cpp

44

Isn't this repeated at the end of the function?

[courbet]

Remove redundant instruction.

[andreadb]

Thanks Clement!

I only have one minor nit. Otherwise, the patch looks good to me.

-Andrea

tools/llvm-mca/HWEventListener.h

64

UR should be a const ArrayRef.

[courbet]

rebase

[courbet]

Thanks !

tools/llvm-mca/TimelineView.cpp

44

Indeed, thanks for the catch.

tools/llvm-mca/Backend.h

 #include "Dispatch.h"
 #include "InstrBuilder.h"
 #include "Scheduler.h"
 #include "SourceMgr.h"
 
 namespace mca {
 
 class HWEventListener;
+class HWInstructionEvent;
 
 /// \brief An out of order backend for a specific subtarget.
 ///
 /// It emulates an out-of-order execution of instructions. Instructions are
 /// fetched from a MCInst sequence managed by an object of class SourceMgr.
 /// Instructions are firstly dispatched to the schedulers and then executed.
 /// This class tracks the lifetime of an instruction from the moment where
 /// it gets dispatched to the schedulers, to the moment where it finishes
Backend(const llvm::MCSubtargetInfo &Subtarget, const llvm::MCInstrInfo &MCII,
         HWS(llvm::make_unique<Scheduler>(this, Subtarget.getSchedModel(),
                                          LoadQueueSize, StoreQueueSize,
                                          AssumeNoAlias)),
         DU(llvm::make_unique<DispatchUnit>(
             this, MRI, Subtarget.getSchedModel().MicroOpBufferSize,
             RegisterFileSize, MaxRetirePerCycle, DispatchWidth, HWS.get())),
         SM(Source), Cycles(0) {
     IB = llvm::make_unique<InstrBuilder>(MCII, getProcResourceMasks());
+    HWS->setDispatchUnit(DU.get());
   }
 
   void run() {
     while (SM.hasNext() || !DU->isRCUEmpty())
       runCycle(Cycles++);
   }
 
   unsigned getNumIterations() const { return SM.getNumIterations(); }
   unsigned getNumInstructions() const { return SM.size(); }
+  const Instruction &getInstruction(unsigned Index) const {
+    const auto It = Instructions.find(Index);
+    assert(It != Instructions.end() && "no running instructions with index");
+    assert(It->second);
+    return *Instructions.find(Index)->second;
+  }
+  void eraseInstruction(unsigned Index) { Instructions.erase(Index); }
   unsigned getNumCycles() const { return Cycles; }
   unsigned getTotalRegisterMappingsCreated() const {
     return DU->getTotalRegisterMappingsCreated();
   }
   unsigned getMaxUsedRegisterMappings() const {
     return DU->getMaxUsedRegisterMappings();
   }
   unsigned getDispatchWidth() const { return DU->getDispatchWidth(); }
public:
   const InstrDesc &getInstrDesc(const llvm::MCInst &Inst) const {
     return IB->getOrCreateInstrDesc(STI, Inst);
   }
 
   const SourceMgr &getSourceMgr() const { return SM; }
 
   void addEventListener(HWEventListener *Listener);
   void notifyCycleBegin(unsigned Cycle);
-  void notifyInstructionDispatched(unsigned Index);
-  void notifyInstructionReady(unsigned Index);
-  void notifyInstructionIssued(
-      unsigned Index,
-      const llvm::ArrayRef<std::pair<ResourceRef, unsigned>> &Used);
-  void notifyInstructionExecuted(unsigned Index);
+  void notifyInstructionEvent(const HWInstructionEvent &Event);
   void notifyResourceAvailable(const ResourceRef &RR);
-  void notifyInstructionRetired(unsigned Index);
   void notifyCycleEnd(unsigned Cycle);
 };
 
 } // namespace mca
 
 #endif

tools/llvm-mca/Backend.cpp

void Backend::notifyCycleBegin(unsigned Cycle) {
   DEBUG(dbgs() << "[E] Cycle begin: " << Cycle << '\n');
   for (HWEventListener *Listener : Listeners)
     Listener->onCycleBegin(Cycle);
 
   DU->cycleEvent(Cycle);
   HWS->cycleEvent(Cycle);
 }
 
-void Backend::notifyInstructionDispatched(unsigned Index) {
-  DEBUG(dbgs() << "[E] Instruction Dispatched: " << Index << '\n');
+void Backend::notifyInstructionEvent(const HWInstructionEvent &Event) {
   for (HWEventListener *Listener : Listeners)
-    Listener->onInstructionDispatched(Index);
-}
-
-void Backend::notifyInstructionReady(unsigned Index) {
-  DEBUG(dbgs() << "[E] Instruction Ready: " << Index << '\n');
-  for (HWEventListener *Listener : Listeners)
-    Listener->onInstructionReady(Index);
-}
-
-void Backend::notifyInstructionIssued(
-    unsigned Index, const ArrayRef<std::pair<ResourceRef, unsigned>> &Used) {
-  DEBUG(
-    dbgs() << "[E] Instruction Issued: " << Index << '\n';
-    for (const std::pair<ResourceRef, unsigned> &Resource : Used) {
-      dbgs() << "[E] Resource Used: [" << Resource.first.first << '.'
-             << Resource.first.second << "]\n";
-      dbgs() << "           cycles: " << Resource.second << '\n';
-    }
-  );
-
-  for (HWEventListener *Listener : Listeners)
-    Listener->onInstructionIssued(Index, Used);
-}
-
-void Backend::notifyInstructionExecuted(unsigned Index) {
-  DEBUG(dbgs() << "[E] Instruction Executed: " << Index << '\n');
-  for (HWEventListener *Listener : Listeners)
-    Listener->onInstructionExecuted(Index);
-
-  const Instruction &IS = *Instructions[Index];
-  DU->onInstructionExecuted(IS.getRCUTokenID());
-}
-
-void Backend::notifyInstructionRetired(unsigned Index) {
-  DEBUG(dbgs() << "[E] Instruction Retired: " << Index << '\n');
-  for (HWEventListener *Listener : Listeners)
-    Listener->onInstructionRetired(Index);
-
-  const Instruction &IS = *Instructions[Index];
-  DU->invalidateRegisterMappings(IS);
-  Instructions.erase(Index);
+    Listener->onInstructionEvent(Event);
 }
 
 void Backend::notifyResourceAvailable(const ResourceRef &RR) {
   DEBUG(dbgs() << "[E] Resource Available: [" << RR.first << '.' << RR.second
                << "]\n");
   for (HWEventListener *Listener : Listeners)
     Listener->onResourceAvailable(RR);
 }
 
 void Backend::notifyCycleEnd(unsigned Cycle) {
   DEBUG(dbgs() << "[E] Cycle end: " << Cycle << "\n\n");
   for (HWEventListener *Listener : Listeners)
     Listener->onCycleEnd(Cycle);
 }
 
 } // namespace mca.

tools/llvm-mca/BackendStatistics.h

void printIssuePerCycle(const Histogram &IssuePerCycle,
                           unsigned TotalCycles) const;
   void printSchedulerUsage(llvm::raw_ostream &OS, const llvm::MCSchedModel &SM,
                            const llvm::ArrayRef<BufferUsageEntry> &Usage) const;
 
 public:
   BackendStatistics(const Backend &backend)
       : B(backend), NumDispatched(0), NumIssued(0), NumRetired(0) {}
 
-  void onInstructionDispatched(unsigned Index) override { NumDispatched++; }
-  void
-  onInstructionIssued(unsigned Index,
-                      const llvm::ArrayRef<std::pair<ResourceRef, unsigned>>
-                          & /* unused */) override {
-    NumIssued++;
-  }
-  void onInstructionRetired(unsigned Index) override { NumRetired++; }
+  void onInstructionEvent(const HWInstructionEvent &Event) override;
 
   void onCycleBegin(unsigned Cycle) override { NumCycles++; }
 
   void onCycleEnd(unsigned Cycle) override { updateHistograms(); }
 
   void printView(llvm::raw_ostream &OS) const override {
     printDispatchStalls(OS, B.getNumRATStalls(), B.getNumRCUStalls(),
                         B.getNumSQStalls(), B.getNumLDQStalls(),
                         B.getNumSTQStalls(), B.getNumDispatchGroupStalls());
     printRATStatistics(OS, B.getTotalRegisterMappingsCreated(),
                        B.getMaxUsedRegisterMappings());
     printDispatchUnitStatistics(OS);
     printSchedulerStatistics(OS);
     printRetireUnitStatistics(OS);
 
     std::vector<BufferUsageEntry> Usage;
     B.getBuffersUsage(Usage);
     printSchedulerUsage(OS, B.getSchedModel(), Usage);
   }
 };
 
 } // namespace mca
 
 #endif

tools/llvm-mca/BackendStatistics.cpp

 
 #include "BackendStatistics.h"
 #include "llvm/Support/Format.h"
 
 using namespace llvm;
 
 namespace mca {
 
+void BackendStatistics::onInstructionEvent(const HWInstructionEvent &Event) {
+  switch (Event.Type) {
+  case HWInstructionEvent::Retired:
+    ++NumRetired;
+    break;
+  case HWInstructionEvent::Issued:
+    ++NumIssued;
+    break;
+  case HWInstructionEvent::Dispatched:
+    ++NumDispatched;
+    break;
+  default:
+    break;
+  }
+}
+
 void BackendStatistics::printRetireUnitStatistics(llvm::raw_ostream &OS) const {
   std::string Buffer;
   raw_string_ostream TempStream(Buffer);
   TempStream << "\n\nRetire Control Unit - "
              << "number of cycles where we saw N instructions retired:\n";
   TempStream << "[# retired], [# cycles]\n";
 
   for (const std::pair<unsigned, unsigned> &Entry : RetiredPerCycle) {

tools/llvm-mca/Dispatch.cpp

 ///
 /// This file implements methods declared by class RegisterFile, DispatchUnit
 /// and RetireControlUnit.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "Dispatch.h"
 #include "Backend.h"
+#include "HWEventListener.h"
 #include "Scheduler.h"
 #include "llvm/Support/Debug.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "llvm-mca"
 
 namespace mca {
unsigned RetireControlUnit::reserveSlot(unsigned Index, unsigned NumMicroOps) {
   Queue[NextAvailableSlotIdx] = {Index, NormalizedQuantity, false};
   NextAvailableSlotIdx += NormalizedQuantity;
   NextAvailableSlotIdx %= Queue.size();
   AvailableSlots -= NormalizedQuantity;
   return TokenID;
 }
 
 void DispatchUnit::notifyInstructionDispatched(unsigned Index) {
-  Owner->notifyInstructionDispatched(Index);
+  DEBUG(dbgs() << "[E] Instruction Dispatched: " << Index << '\n');
+  Owner->notifyInstructionEvent(
+      HWInstructionEvent(HWInstructionEvent::Dispatched, Index));
 }
 
 void DispatchUnit::notifyInstructionRetired(unsigned Index) {
-  Owner->notifyInstructionRetired(Index);
+  DEBUG(dbgs() << "[E] Instruction Retired: " << Index << '\n');
+  Owner->notifyInstructionEvent(
+      HWInstructionEvent(HWInstructionEvent::Retired, Index));
+
+  const Instruction &IS = Owner->getInstruction(Index);
+  invalidateRegisterMappings(IS);
+  Owner->eraseInstruction(Index);
 }
 
 void RetireControlUnit::cycleEvent() {
   if (isEmpty())
     return;
 
   unsigned NumRetired = 0;
   while (!isEmpty()) {
     if (MaxRetirePerCycle != 0 && NumRetired == MaxRetirePerCycle)
unsigned DispatchUnit::dispatch(unsigned IID, Instruction *NewInst) {
   } else {
     assert(AvailableEntries >= NumMicroOps);
     AvailableEntries -= NumMicroOps;
   }
 
   // Reserve slots in the RCU.
   unsigned RCUTokenID = RCU->reserveSlot(IID, NumMicroOps);
   NewInst->setRCUTokenID(RCUTokenID);
-  Owner->notifyInstructionDispatched(IID);
+  notifyInstructionDispatched(IID);
 
   SC->scheduleInstruction(IID, NewInst);
   return RCUTokenID;
 }
 
 #ifndef NDEBUG
 void DispatchUnit::dump() const {
   RAT->dump();

tools/llvm-mca/HWEventListener.h

 #ifndef LLVM_TOOLS_LLVM_MCA_HWEVENTLISTENER_H
 #define LLVM_TOOLS_LLVM_MCA_HWEVENTLISTENER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include <utility>
 
 namespace mca {
 
-class HWEventListener {
+// An HWInstructionEvent represents state changes of instructions that
+// listeners might be interested in. Listeners can choose to ignore any event
+// they are not interested in.
+class HWInstructionEvent {
 public:
+  // This is the list of event types that are shared by all targets, that
+  // generic subtarget-agnostic classes (e.g. Backend, HWInstructionEvent, ...)
+  // and generic Views can manipulate.
+  // Subtargets are free to define additional event types, that are goin to be
+  // handled by generic components as opaque values, but can still be
+  // emitted by subtarget-specific pipeline components (e.g. Scheduler,
+  // DispatchUnit, ...) and interpreted by subtarget-specific EventListener
+  // implementations.
+  enum GenericEventType {
+    Invalid = 0,
     // Events generated by the Retire Control Unit.
-  virtual void onInstructionRetired(unsigned Index) {};
-
+    Retired,
     // Events generated by the Scheduler.
+    Ready,
+    Issued,
+    Executed,
+    // Events generated by the Dispatch logic.
+    Dispatched,
+
+    LastGenericEventType,
+  };
+
+  HWInstructionEvent(unsigned Type, unsigned Index)
+      : Type(Type), Index(Index) {}
+
+  // The event type. The exact meaning depends on the subtarget.
+  const unsigned Type;
+  // The index of the instruction in the source manager.
+  const unsigned Index;
+};
+
+class HWInstructionIssuedEvent : public HWInstructionEvent {
+public:
   using ResourceRef = std::pair<uint64_t, uint64_t>;
-  virtual void
-  onInstructionIssued(unsigned Index,
-                      const llvm::ArrayRef<std::pair<ResourceRef, unsigned>> &Used) {}
-  virtual void onInstructionExecuted(unsigned Index) {}
-  virtual void onInstructionReady(unsigned Index) {}
-  virtual void onResourceAvailable(const ResourceRef &RRef) {};
+  HWInstructionIssuedEvent(unsigned Index,
+                           llvm::ArrayRef<std::pair<ResourceRef, unsigned>> UR)
+      : HWInstructionEvent(HWInstructionEvent::Issued, Index),
+        UsedResources(UR) {}
 
-  // Events generated by the Dispatch logic.
-  virtual void onInstructionDispatched(unsigned Index) {}
+  const llvm::ArrayRef<std::pair<ResourceRef, unsigned>> UsedResources;
+};
 
-  // Generic events generated by the Backend.
+class HWEventListener {
+public:
+  // Generic events generated by the backend pipeline.
   virtual void onCycleBegin(unsigned Cycle) {}
   virtual void onCycleEnd(unsigned Cycle) {}
 
-  virtual ~HWEventListener() = default;
+  virtual void onInstructionEvent(const HWInstructionEvent &Event) {}
+
+  using ResourceRef = std::pair<uint64_t, uint64_t>;
+  virtual void onResourceAvailable(const ResourceRef &RRef) {}
+
+  virtual ~HWEventListener() {}
+
+private:
   virtual void anchor();
 };
 
 } // namespace mca
 
 #endif

tools/llvm-mca/ResourcePressureView.h

 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_MCA_RESOURCEPRESSUREVIEW_H
 #define LLVM_TOOLS_LLVM_MCA_RESOURCEPRESSUREVIEW_H
 
 #include "SourceMgr.h"
 #include "View.h"
 #include "llvm/MC/MCInstPrinter.h"
 #include "llvm/MC/MCSubtargetInfo.h"
 #include <map>
 
 namespace mca {
 
 class Backend;
 
 /// This class collects resource pressure statistics and it is able to print

 public:
   ResourcePressureView(const llvm::MCSubtargetInfo &ST,
                        llvm::MCInstPrinter &Printer, const SourceMgr &SM,
                        const llvm::ArrayRef<uint64_t> ProcResourceMasks)
       : STI(ST), MCIP(Printer), Source(SM) {
     initialize(ProcResourceMasks);
   }
 
-  void onInstructionIssued(
-      unsigned Index,
-      const llvm::ArrayRef<std::pair<ResourceRef, unsigned>> &Used) override;
+  void onInstructionEvent(const HWInstructionEvent &Event) override;
 
   void printView(llvm::raw_ostream &OS) const override {
     unsigned Executions = Source.getNumIterations();
     printResourcePressurePerIteration(OS, Executions);
     printResourcePressurePerInstruction(OS, Executions);
   }
 };
 
 } // namespace mca
 
 #endif

tools/llvm-mca/ResourcePressureView.cpp

for (unsigned I = 0, E = SM.getNumProcResourceKinds(); I < E; ++I) {
     R2VIndex += ProcResource.NumUnits;
   }
 
   NumResourceUnits = R2VIndex;
   ResourceUsage.resize(NumResourceUnits * (Source.size() + 1));
   std::fill(ResourceUsage.begin(), ResourceUsage.end(), 0);
 }
 
-void ResourcePressureView::onInstructionIssued(
-    unsigned Index, const ArrayRef<std::pair<ResourceRef, unsigned>> &Used) {
-  unsigned SourceIdx = Index % Source.size();
-  for (const std::pair<ResourceRef, unsigned> &Use : Used) {
+void ResourcePressureView::onInstructionEvent(const HWInstructionEvent &Event) {
+  // We're only interested in Issue events.
+  if (Event.Type != HWInstructionEvent::Issued)
+    return;
+  const auto &IssueEvent = static_cast<const HWInstructionIssuedEvent &>(Event);
+  unsigned SourceIdx = Event.Index % Source.size();
+  for (const std::pair<ResourceRef, unsigned> &Use : IssueEvent.UsedResources) {
     const ResourceRef &RR = Use.first;
     assert(Resource2VecIndex.find(RR.first) != Resource2VecIndex.end());
     unsigned R2VIndex = Resource2VecIndex[RR.first];
     R2VIndex += countTrailingZeros(RR.second);
     ResourceUsage[R2VIndex + NumResourceUnits * SourceIdx] += Use.second;
     ResourceUsage[R2VIndex + NumResourceUnits * Source.size()] += Use.second;
   }
 }

tools/llvm-mca/Scheduler.h

 #include "LSUnit.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/MC/MCSubtargetInfo.h"
 #include <map>
 
 namespace mca {
 
 class Backend;
+class DispatchUnit;
+
 /// Used to notify the internal state of a processor resource.
 ///
 /// A processor resource is available if it is not reserved, and there are
 /// available slots in the buffer.  A processor resource is unavailable if it
 /// is either reserved, or the associated buffer is full. A processor resource
 /// with a buffer size of -1 is always available if it is not reserved.
 ///
 /// Values of type ResourceStateEvent are returned by method

 class Scheduler {
   const llvm::MCSchedModel &SM;
 
   // Hardware resources that are managed by this scheduler.
   std::unique_ptr<ResourceManager> Resources;
   std::unique_ptr<LSUnit> LSU;
 
   // The Backend gets notified when instructions are ready/issued/executed.
-  Backend *Owner;
+  Backend *const Owner;
+
+  // The dispatch unit gets notified when instructions are executed.
+  DispatchUnit *DU;
 
   using QueueEntryTy = std::pair<unsigned, Instruction *>;
   std::map<unsigned, Instruction *> WaitQueue;
   std::map<unsigned, Instruction *> ReadyQueue;
   std::map<unsigned, Instruction *> IssuedQueue;
 
   void notifyInstructionIssued(
       unsigned Index,

 public:
   Scheduler(Backend *B, const llvm::MCSchedModel &Model, unsigned LoadQueueSize,
             unsigned StoreQueueSize, bool AssumeNoAlias)
       : SM(Model), Resources(llvm::make_unique<ResourceManager>(SM)),
         LSU(llvm::make_unique<LSUnit>(LoadQueueSize, StoreQueueSize,
                                       AssumeNoAlias)),
         Owner(B) {}
 
+  void setDispatchUnit(DispatchUnit *DispUnit) { DU = DispUnit; }
+
   /// Scheduling events.
   ///
   /// The DispatchUnit is responsible for querying the Scheduler before
   /// dispatching new instructions. Queries are performed through method
   /// `Scheduler::CanBeDispatched`, which returns an instance of this enum to
   /// tell if the dispatch would fail or not.  If scheduling resources are
   /// available, and the instruction can be dispatched, then the query returns
   /// HWS_AVAILABLE.  A values different than HWS_AVAILABLE means that the

tools/llvm-mca/Scheduler.cpp

 //===--------------------- Scheduler.cpp ------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // A scheduler for processor resource units and processor resource groups.
 //
 //===----------------------------------------------------------------------===//
 
 #include "Scheduler.h"
 #include "Backend.h"
+#include "HWEventListener.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 
 #define DEBUG_TYPE "llvm-mca"
 
 namespace mca {
 
 using namespace llvm;
if (Entry.second->isExecuted()) {
                    << " is still executing.\n");
       ++I;
     }
   }
 }
 
 void Scheduler::notifyInstructionIssued(
     unsigned Index, const ArrayRef<std::pair<ResourceRef, unsigned>> &Used) {
-  Owner->notifyInstructionIssued(Index, Used);
+  DEBUG(dbgs() << "[E] Instruction Issued: " << Index << '\n';
+        for (const std::pair<ResourceRef, unsigned> &Resource
+             : Used) {
+          dbgs() << "[E] Resource Used: [" << Resource.first.first << '.'
+                 << Resource.first.second << "]\n";
+          dbgs() << "           cycles: " << Resource.second << '\n';
+        });
+  Owner->notifyInstructionEvent(HWInstructionIssuedEvent(Index, Used));
 }
 
 void Scheduler::notifyInstructionExecuted(unsigned Index) {
   LSU->onInstructionExecuted(Index);
-  Owner->notifyInstructionExecuted(Index);
+  DEBUG(dbgs() << "[E] Instruction Executed: " << Index << '\n');
+  Owner->notifyInstructionEvent(
+      HWInstructionEvent(HWInstructionEvent::Executed, Index));
+
+  const Instruction &IS = Owner->getInstruction(Index);
+  DU->onInstructionExecuted(IS.getRCUTokenID());
 }
 
 void Scheduler::notifyInstructionReady(unsigned Index) {
-  Owner->notifyInstructionReady(Index);
+  DEBUG(dbgs() << "[E] Instruction Ready: " << Index << '\n');
+  Owner->notifyInstructionEvent(
+      HWInstructionEvent(HWInstructionEvent::Ready, Index));
 }
 
 void Scheduler::notifyResourceAvailable(const ResourceRef &RR) {
   Owner->notifyResourceAvailable(RR);
 }
 } // namespace mca

tools/llvm-mca/TimelineView.h

TimelineView(const llvm::MCSubtargetInfo &sti, llvm::MCInstPrinter &Printer,
       : STI(sti), MCIP(Printer), AsmSequence(Sequence), CurrentCycle(0),
         MaxCycle(Cycles == 0 ? 80 : Cycles), LastCycle(0) {
     initialize(MaxIterations);
   }
 
   void initialize(unsigned MaxIterations);
 
   // Event handlers.
-  void onInstructionDispatched(unsigned Index) override;
-  void onInstructionReady(unsigned Index) override;
-  void onInstructionIssued(
-      unsigned Index,
-      const llvm::ArrayRef<std::pair<ResourceRef, unsigned>> &Used) override;
-  void onInstructionExecuted(unsigned Index) override;
-  void onInstructionRetired(unsigned Index) override;
   void onCycleBegin(unsigned Cycle) override { CurrentCycle = Cycle; }
+  void onInstructionEvent(const HWInstructionEvent &Event) override;
 
   // print functionalities.
   void printTimeline(llvm::raw_ostream &OS) const;
   void printAverageWaitTimes(llvm::raw_ostream &OS) const;
   void printView(llvm::raw_ostream &OS) const override {
     printTimeline(OS);
     printAverageWaitTimes(OS);
   }
 };
 
 } // namespace mca
 
 #endif

tools/llvm-mca/TimelineView.cpp

void TimelineView::initialize(unsigned MaxIterations) {
   TimelineViewEntry NullTVEntry = {0, 0, 0, 0, 0};
   std::fill(Timeline.begin(), Timeline.end(), NullTVEntry);
 
   WaitTime.resize(AsmSequence.size());
   WaitTimeEntry NullWTEntry = {0, 0, 0, 0};
   std::fill(WaitTime.begin(), WaitTime.end(), NullWTEntry);
 }
 
-void TimelineView::onInstructionDispatched(unsigned Index) {
-  if (CurrentCycle >= MaxCycle || Index >= Timeline.size())
+void TimelineView::onInstructionEvent(const HWInstructionEvent &Event) {
+  if (CurrentCycle >= MaxCycle || Event.Index >= Timeline.size())
     return;
-  Timeline[Index].CycleDispatched = CurrentCycle;
-  LastCycle = std::max(LastCycle, CurrentCycle);
-}
-
-void TimelineView::onInstructionReady(unsigned Index) {
-  if (CurrentCycle >= MaxCycle || Index >= Timeline.size())
-    return;
-  Timeline[Index].CycleReady = CurrentCycle;
-  LastCycle = std::max(LastCycle, CurrentCycle);
-}
-
-void TimelineView::onInstructionIssued(
-    unsigned Index,
-    const ArrayRef<std::pair<ResourceRef, unsigned>> & /* Unused */) {
-  if (CurrentCycle >= MaxCycle || Index >= Timeline.size())
-    return;
-  Timeline[Index].CycleIssued = CurrentCycle;
-  LastCycle = std::max(LastCycle, CurrentCycle);
-}
-
-void TimelineView::onInstructionExecuted(unsigned Index) {
-  if (CurrentCycle >= MaxCycle || Index >= Timeline.size())
-    return;
-  Timeline[Index].CycleExecuted = CurrentCycle;
-  LastCycle = std::max(LastCycle, CurrentCycle);
-}
-
-void TimelineView::onInstructionRetired(unsigned Index) {
-  if (CurrentCycle >= MaxCycle || Index >= Timeline.size())
-    return;
-  TimelineViewEntry &TVEntry = Timeline[Index];
+  switch (Event.Type) {
+  case HWInstructionEvent::Retired: {
+    TimelineViewEntry &TVEntry = Timeline[Event.Index];
     TVEntry.CycleRetired = CurrentCycle;
-  LastCycle = std::max(LastCycle, CurrentCycle);
 
     // Update the WaitTime entry which corresponds to this Index.
-
-  WaitTimeEntry &WTEntry = WaitTime[Index % AsmSequence.size()];
+    WaitTimeEntry &WTEntry = WaitTime[Event.Index % AsmSequence.size()];
     WTEntry.Executions++;
     WTEntry.CyclesSpentInSchedulerQueue +=
         TVEntry.CycleIssued - TVEntry.CycleDispatched;
     assert(TVEntry.CycleDispatched <= TVEntry.CycleReady);
-  WTEntry.CyclesSpentInSQWhileReady += TVEntry.CycleIssued - TVEntry.CycleReady;
+    WTEntry.CyclesSpentInSQWhileReady +=
+        TVEntry.CycleIssued - TVEntry.CycleReady;
     WTEntry.CyclesSpentAfterWBAndBeforeRetire +=
         (TVEntry.CycleRetired - 1) - TVEntry.CycleExecuted;
+    break;
+  }
+  case HWInstructionEvent::Ready:
+    Timeline[Event.Index].CycleReady = CurrentCycle;
+    break;
+  case HWInstructionEvent::Issued:
+    Timeline[Event.Index].CycleIssued = CurrentCycle;
+    break;
+  case HWInstructionEvent::Executed:
+    Timeline[Event.Index].CycleExecuted = CurrentCycle;
+    break;
+  case HWInstructionEvent::Dispatched:
+    Timeline[Event.Index].CycleDispatched = CurrentCycle;
+    break;
+  default:
+    return;
+  }
+  LastCycle = std::max(LastCycle, CurrentCycle);
 }
 
 void TimelineView::printWaitTimeEntry(raw_string_ostream &OS,
                                       const WaitTimeEntry &Entry,
                                       unsigned SourceIndex) const {
   OS << SourceIndex << '.';
   if (SourceIndex < 10)
     OS << "    ";