[llvm-mca] Add support for in-order CPUs

[asavonic]

Summary

This patch adds a simplified pipeline to support in-order CPUs such as
ARM Cortex-A55.

In-order pipeline implements a simplified version of Dispatch,
Scheduler and Execute stages as a single stage. Entry and Retire
stages are shared for both in-order and out-of-order pipelines.

[RKSimon]

llvm/lib/MCA/CMakeLists.txt

21

sorting

[andreadb]

Thanks for working on this feature!

I went through the patch and overall the approach seems sound.

There are however a few questions and concerns about the overall design.
That being said, overall it shouldn't be difficult to address most of my comments.

Speaking about the design:

Your model assumes an unbounded queue of instructions (something like rudimental reservation station) where to store dispatched instructions.

Correct me if I am wrong, but in-order processor don't use a reservation station.
In the absence of structural hazards, if data dependencies are met, then uOPs are directly issued to the underlying execution units.
So the dispatch event is not decoupled from the issue event.

The fact that your patch adds an unbounded queue sounds a bit strange to me. Not sure what @dmgreen
thinks about it. But this basically means that dispatch and issue are different events.

I also noticed how there are no checks on NumMicroOps. Is there a reason why you don't check for it?
In one of the tests, the target is dual-issue. However, there are cycles where three opcodes are dispatched.
See for example the test where two loads are dispatched in a same cycle (with the first load decoded into two uOPs).
Note also that the presence of '=' chars in the timeline represents cycles spent by an instruction while waiting in a scheduler to be issued.
To be honest, I was not expecting to see those characters at all.

Btw. There is also one particular case where two instructions seems to execute out of order.

You can find other comments below.

Thanks,
-Andrea

llvm/include/llvm/MCA/Stages/InOrderIssueStage.h

42–52

Please use unsigned quantities instead of signed integers.

llvm/lib/MCA/Context.cpp

31–33

I suggest to move the support for in-order pipeline composition into a separate function. I think it would help in terms of readability.

31–40

For readability reasons, I suggest to split this method into two methods.

For example, something like if (!SM.isOutOfOrder()) createInOrderPipeline(opts, srcMgr);.

You then need to move all the pipeline composition logic for in-order processors inside createInOrderPipeline().

I think it would be much more readable (just my opinion though).

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

35–38

We also need to check if IR "begins a group".
Instructions that begin a group, must always be the first instructions to be dispatched in a cycle. See how the isAvailable() check is implemented by the DispatchStage.

36

Shouldn't we also add checks on NumMicroOps somewhere?
One of the tests reports two loads dispatched within a same cycle. However one of the loads is 2 uOps, so - correct me if I am wrong - it shouldn't be possible to dispatch two of those in a same cycle.

Out of curiosity:

ldr	w4, [x2], #4

is the resource consumption info correct for that instruction?

50

As mentioned in another comment we should use unsigned quanitites whenever possible. So, int quantities that are only used to describe zero or more cycles should really be converted into unsigned.

108–115

I expect register renaming to only occur for out-of-order processors. So move elimination shouldn't ever happen in this context.

That being said, instructions that are fully eliminated at register renaming stage still have their writes propagated to the register file. So, if you want to correctly handle that case, then you cannot early exit; you still need to add writes.

Note that the only instructions that can be eliminated at register renaming stage are register moves. See the definition and description of tablegen class RegisterFile in
https://github.com/llvm-mirror/llvm/blob/master/include/llvm/Target/TargetSchedule.td

I don't expect any of this to really affect in-order targets. So, personally I suggest to replace the initial if-stmt with an assert (i.e. check tha IS is NOT eliminated).

143

Maybe add a comment here explaining why we are passing 0 as token-id.
Since this is an in-order processor, the retire stage (and therefore the token id) are not really required in practice. We don't expect the retire stage to do anything in particular.

149–151

Bandwidth should be set to zero if IR ends a group.

You probaby need something semantically equivalent to what method DispatchStage::dispatch(InstRef IR) does:

// Check if this instructions ends the dispatch group.                                                                                                                      if (Desc.EndGroup)
  AvailableEntries = 0;

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

115–127

Interesting.
According to this design, the "dispatch" event is still decoupled from the "issue" event. Is that expected?
I am asking because in my mind, at least for in-order processors, the two events should coincide. Basically there is no reservation station, and uOPs are directly issued to the underlying execution unit at a IssueWidth maximum rate.

117–118

Why are these two executing out of order?

[asavonic]

Thanks for the review Andrea!

In D94928#2506972, @andreadb wrote:

Your model assumes an unbounded queue of instructions (something like rudimental reservation station) where to store dispatched instructions.

If you mean InstQueue, then it is bounded by Bandwidth variable - the
maximum number of instructions that can be issued in the next cycle.

Correct me if I am wrong, but in-order processor don't use a reservation station.
In the absence of structural hazards, if data dependencies are met, then uOPs are directly issued to the underlying execution units.
So the dispatch event is not decoupled from the issue event.

The fact that your patch adds an unbounded queue sounds a bit strange to me. Not sure what @dmgreen
thinks about it. But this basically means that dispatch and issue are different events.

That is true. However, the problem here is that MCA timeline view counts stalls
as a number of cycles between dispatch and issue events. If dispatch and issue
always happen in the same cycle, stalls are not displayed:

[0,3]     .  DeeER  .    .    add	w13, w30, #1
[0,4]     .  DeeeER .    .    smulh	x30, x29, x28
[0,5]     .     DeeeER   .    smulh	x27, x30, x28
[0,6]     .        DeeeER.    smulh	xzr, x27, x26
[0,7]     .    .    DeeeER    umulh	x30, x29, x28

To avoid this, the implementation emits a dispatch event for instructions that
should be executed in the next cycle. If an instruction is unable to execute due
to a hazard, it is delayed and a stall is displayed starting from the dispatch
event:

[0,3]     . DeeeER  .    .    add	w13, w30, #4095, lsl #12
[0,4]     . DeeeeER .    .    smulh	x30, x29, x28
[0,5]     .  D==eeeeER   .    smulh	x27, x30, x28
[0,6]     .  D=====eeeeER.    smulh	xzr, x27, x26
[0,7]     .    .  D=eeeeER    umulh	x30, x29, x28

I remember that I did this intentionally, but now I'm not really convinced that
this difference is worth extra complexity. Let me know what you think about
this.

I also noticed how there are no checks on NumMicroOps. Is there a reason why you don't check for it?

Good point, I will fix that.

In one of the tests, the target is dual-issue. However, there are cycles where three opcodes are dispatched.
See for example the test where two loads are dispatched in a same cycle (with the first load decoded into two uOPs).

I think this should not happen. I will add a check for NumMicroOps.

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

117–118

Madd and add are issued in the same cycle, subs is issued next.
However, they should not retire out-of-order. Some instructions can
retire out-of-order, but not these.

I have to look into this. Probably an RCU is actually needed for the
in-order pipeline.

[andreadb]

In D94928#2510059, @asavonic wrote:

Thanks for the review Andrea!

In D94928#2506972, @andreadb wrote:

Your model assumes an unbounded queue of instructions (something like rudimental reservation station) where to store dispatched instructions.

If you mean InstQueue, then it is bounded by Bandwidth variable - the
maximum number of instructions that can be issued in the next cycle.

Correct me if I am wrong, but in-order processor don't use a reservation station.
In the absence of structural hazards, if data dependencies are met, then uOPs are directly issued to the underlying execution units.
So the dispatch event is not decoupled from the issue event.

The fact that your patch adds an unbounded queue sounds a bit strange to me. Not sure what @dmgreen
thinks about it. But this basically means that dispatch and issue are different events.

That is true. However, the problem here is that MCA timeline view counts stalls
as a number of cycles between dispatch and issue events. If dispatch and issue
always happen in the same cycle, stalls are not displayed:

[0,3]     .  DeeER  .    .    add	w13, w30, #1
[0,4]     .  DeeeER .    .    smulh	x30, x29, x28
[0,5]     .     DeeeER   .    smulh	x27, x30, x28
[0,6]     .        DeeeER.    smulh	xzr, x27, x26
[0,7]     .    .    DeeeER    umulh	x30, x29, x28

To avoid this, the implementation emits a dispatch event for instructions that
should be executed in the next cycle. If an instruction is unable to execute due
to a hazard, it is delayed and a stall is displayed starting from the dispatch
event:

[0,3]     . DeeeER  .    .    add	w13, w30, #4095, lsl #12
[0,4]     . DeeeeER .    .    smulh	x30, x29, x28
[0,5]     .  D==eeeeER   .    smulh	x27, x30, x28
[0,6]     .  D=====eeeeER.    smulh	xzr, x27, x26
[0,7]     .    .  D=eeeeER    umulh	x30, x29, x28

I remember that I did this intentionally, but now I'm not really convinced that
this difference is worth extra complexity. Let me know what you think about
this.

Ok, I understand now.
It was done intentionally to emphasize issue stalls.

Personally I prefer if the 'dispatch' event coincides with the 'issue' event.
From a theoretical point of view, those events should really be the same.

Conceptually, "dispatch" is only really useful when describing the lifetime of an instruction in an out-of-order processor.
For those processors it makes sense to distinguish between cycles where uOPs are picked from the decoder's queue, from cycles where uOPs are actually sent by the scheduler(s) to the underlying pipeline(s).

In future, we may introduce a new option for the timeline to toggle the generation of extra chars for dispatch stalls (possibly using a different character from = to avoid confusions).
That might be contributed by a separate patch in future. For now, if you don't mind I'd rather keep your patch simple.

I also noticed how there are no checks on NumMicroOps. Is there a reason why you don't check for it?

Good point, I will fix that.

In one of the tests, the target is dual-issue. However, there are cycles where three opcodes are dispatched.
See for example the test where two loads are dispatched in a same cycle (with the first load decoded into two uOPs).

I think this should not happen. I will add a check for NumMicroOps.

Thanks!

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

117–118

In theory, younger instructions should not be allowed to reach the write-back stage before older instructions because that would lead to out-of-order execution.
In this case I was expecting a compulsory stall to artificially delay the issue of the add so that it can write-back in-order w.r.t. the madd.
What are those cases where it is allowed to write-back instructions out of order? Shouldn't architectural commits always happen in-order?

[asavonic]

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

117–118

In theory, younger instructions should not be allowed to reach the write-back stage before older instructions because that would lead to out-of-order execution.
In this case I was expecting a compulsory stall to artificially delay the issue of the add so that it can write-back in-order w.r.t. the madd.

I wonder how this works for instructions with early termination (sdiv, udiv).
@dmgreen, can you please comment on this?

What are those cases where it is allowed to write-back instructions out of order? Shouldn't architectural commits always happen in-order?

From Cortex-A55 optimization manual, s3.5.1 "Instructions with out-of-order completion":

While the Cortex-A55 core only issues instructions in-order, due to the number of cycles required to complete more complex floating-point and NEON instructions, out-of-order retire is allowed on the instructions described in this section. The nature of the Cortex-A55 microarchitecture is such that NEON and floating-point instructions of the same type have the same timing characteristics.

[dmgreen]

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

117–118

Yeah, I was about to quote the same thing!

The Cortex-R52, another in-order AArch32 cpu similar to the A53 mentions:
https://developer.arm.com/documentation/100026/0103/Cycle-Timings-and-Interlock-Behavior/Pipeline-behavior/Dual-issuing

If the Cortex-R52 processor determines that the pair must be dual-issued, it remains a pair until both instructions are retired.

So I believe it can depend on the CPU and possibly the instructions issued.

[andreadb]

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

117–118

I see.
That unfortunately complicates the whole implementation.
We do need a Retire Stage after all...

Is out-of-order execution limited in some ways in these processors?
For example: do these processors perform register renaming to break false dependencies?
If we don't need to worry about register renaming, then the implementation is simpler. I really hope that out-of-order execution is only really allowed in case of completely independent instructions , and just to avoid any bottlenecks caused by long latency instructions...

By the way, do we have a mechanism in place to identify instructions that can be executed out-of-order? I don't remember to have ever seen anything like that in the scheduling model.
If not, then we need a way to mark those instructions somehow. In case, we might be and be able to reuse the MCSchedPredicate framework for that (fingers crossed).

[asavonic]

• Added RCU support for the in-order pipeline. Some instructions should retire out-of-order, so a new flag was added to SchedClass.

• Dispatch event is now emitted in the same cycle as an issue event.

• Fixed code style issues.

[asavonic]

llvm/include/llvm/MCA/Stages/InOrderIssueStage.h

42–52

Done.

llvm/lib/MCA/CMakeLists.txt

21

Thanks, fixed.

llvm/lib/MCA/Context.cpp

31–33

Done.

31–40

Done.

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

35–38

Thank you. I added a check for both BeginGroup and EndGroup.

36

Added a check for NumMicroOps.

Regarding the ldr: the optimization manual for Cortex-A55 states that throughput for most load instructions is 1, so I guess it is correct.

50

Done.

108–115

Thank you for the explanation! Replaced the if statement with an assert.

143

The code is changed a bit to accommodate an RCU. Let me know if it still needs a comment.

149–151

Done.

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

115–127

Fixed. Now a dispatch event and an issue event occur in the same cycle.

117–118

I changed the RetireStage, RCU and the scheduler model to support this.
RetireControlUnit is now used if it is defined in the scheduler model. I ran a couple of tests on hardware, and MCA results seem to match it pretty well.

Some instructions require out-of-order retire. I added RetireOOO flag to MCSchedClass to handle this feature. Instructions with this flag do not block other instructions and may retire as soon as they complete execution.

[asavonic]

Fixed LIT InvalidMCSchedClassDesc.td.

[andreadb]

Thanks for the updated patch.

I left a couple of minor comments below. Otherwise the design looks good to me.

@dmgreen @evgeny777, what is your opinion on this simulation pipeline? Do you think that this design might work well for other ARM in-order processors or there are other things that should be improved? Also, if you could try a few experiments with this patch, then that would be great.

Thanks,
-Andrea

llvm/include/llvm/Target/TargetSchedule.td

265–266

Note that instructions can have multiple writes (typically one per each register definition). So the comment should say that a 'write' is allowed to retire out-of-order.

Also, this field is only used by llvm-mca. The value of that field is ignored if the model doesn't describe an in-order processors.

llvm/utils/TableGen/SubtargetEmitter.cpp

1099

Basically, an instruction is allowed to retire out-of-order if RetireOOO is true for at least one of its writes.
This field is only meaningful for in-order subtargets, and is ignored for other targets.

I think that this should be better described in a comment (in TargetPredicates.td).

[RKSimon]

llvm/include/llvm/MC/MCSchedule.h

119

This is a separate problem, but windows builds currently takes 4 bytes for this (before and after this patch) as bool is treated as an int. Would it be possible to change this to:

uint16_t NumMicroOps : 13;
uint16_t BeginGroup : 1;
uint16_t EndGroup : 1;
uint16_t RetireOOO : 1;

Ideally replace the bools in an initial patch before this one?

[asavonic]

Updated the comment for RetireOOO.

[asavonic]

llvm/include/llvm/MC/MCSchedule.h

119

Thanks. Uploaded a fix to https://reviews.llvm.org/D95954.

[andreadb]

Now that https://reviews.llvm.org/D95954 has been fixed, there are only a few things left to do:

1. Disable the bottleneck analysis for in-order processors.
2. Add a couple of lines to the release notes describing this new feature.
3. Update the llvm-mca docs.

About the bottleneck analysis view:
Current implementation works under the assumption that the processor is out-of-order. The analysis internally observes events generated by the scheduler, as well as instruction state transitions. It correlates those events with scheduler queue availablity to track pressure increases due to data dependencies etc. For in-order processors, we need a completely different logic; the existing logic would not work (as you can tell from one of the tests that you have added, for which no bottlenecks were reported for data dependencies!).

We should raise a warning from the llvm-mca driver if the user explicitly requested the bottleneck analysis, and the processor is in-order. The warning should explain how the bottleneck analysis is currently unsupported for in-order processors. Note also that flag -all-views should not enable the bottleneck analysis if the simulated processor is in-order.

About the documentation: in a few paragraphs we mention "out-of-order" backends. In most cases, it is just a matter of removing the word "out-of-order", and the rest of the paragraph is still valid.
However, section "Instruction Flow" requires an extra paragraph for in-order processors. The instruction flow is much simpler for those processors, and the dispatch event is not decoupled from the issue event (it is all handled by your new "InOrderIssue" stage).
Please add a new section for your new "InOrderIssue" stage (ideally, it should go after the "Write-Back and Retire Stage" section). There you should briefly mention what happens to instructions during that stage. Please also add a small paragraph explaining how the current model allows for out-of-order execution of writes that are specially marked in the scheduling model.

I think this is all.
I plan to accept the patch once those three points are addressed.

Thanks,
-Andrea

[dmgreen]

Thanks for working on this, it looks like it will be very useful.

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

82

How come this asserts on ReadAdvance < 0? I though it was relatively common to have certain instructions requiring operands before the main stage the pipeline is based on.

[andreadb]

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

82

I agree with Dave here. This logic must also work for the case where ReadAdvance is a negative values. There is no reason why we should restrict this logic to cases where ReadAdvance is >=0.

You also don't need the assert for UNKNOWN_CYCLES at line 82. If the number of "write cycles left" for a register is unknown, then you can safely ignore it, and leave StallCycles as is (i.e. simply continue to the next iteration of the loop).

If however the number of write cycles left is different than UNKNOWN_CYCLES, then you should always update StallCycles with the std::max between the actual StallCycles and CyclesLeft - ReadAdvance. A negative ReadAdvance would effectively "increase" the CyclesLeft (which is what we want to model here).

For this to work, CyclesLeft must be manipulated as a signed quantity (note that UNKNOWN_CYCLES is also a signed quantity; its value is -1). std::max requires for operands to be of the same type, so this may require an extra cast for StallCycles.
You should then get rid of the check at line 84, and always compute the std::max, to update StallCycles.

[asavonic]

Supported UNKNOWN_CYCLES left and negative ReadAdvance.
Disabled bottleneck analysis for in-order CPUs.
Updated documentation.

[asavonic]

In D94928#2568386, @andreadb wrote:

Now that https://reviews.llvm.org/D95954 has been fixed, there are only a few things left to do:

1. Disable the bottleneck analysis for in-order processors.
2. Add a couple of lines to the release notes describing this new feature.
3. Update the llvm-mca docs.

Thank you! Can you please check if wording in the docs is ok?

Regarding the bottleneck analysis: it seems to work for some cases
(A55-all-views.s), but not for others (A55-out-of-order-retire.s).
I will check what is wrong here. The feature is disabled for now as you
suggested.

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

82

How come this asserts on ReadAdvance < 0? I though it was relatively common to have certain instructions requiring operands before the main stage the pipeline is based on.

I wasn't sure that it is a valid case. Thank you for the explanation. I've
adjusted the code to handle negative ReadAdvance. There seems to be no such cases for A55, so I added a test for M7.

82

You also don't need the assert for UNKNOWN_CYCLES at line 82. If the number of "write cycles left" for a register is unknown, then you can safely ignore it, and leave StallCycles as is (i.e. simply continue to the next iteration of the loop).

I'm not sure that we can just ignore it. If it is unknown, then it is not zero,
so a register hazard should be detected. I've modified the code to return 1 cycle
stall, so we can check that the value is "known" in the next cycle.

In any case, this should never happen, because Instruction::execute is called
before a subsequent instruction issue, and it always sets CyclesLeft to a
known value.

If however the number of write cycles left is different than UNKNOWN_CYCLES, then you should always update StallCycles with the std::max between the actual StallCycles and CyclesLeft - ReadAdvance. A negative ReadAdvance would effectively "increase" the CyclesLeft (which is what we want to model here).

For this to work, CyclesLeft must be manipulated as a signed quantity (note that UNKNOWN_CYCLES is also a signed quantity; its value is -1). std::max requires for operands to be of the same type, so this may require an extra cast for StallCycles.

Right, this is fixed now.

You should then get rid of the check at line 84, and always compute the std::max, to update StallCycles.

The check is still needed to ensure that we don't cast a negative value to
unsigned.

[andreadb]

In D94928#2596996, @asavonic wrote:

In D94928#2568386, @andreadb wrote:

Now that https://reviews.llvm.org/D95954 has been fixed, there are only a few things left to do:

1. Disable the bottleneck analysis for in-order processors.
2. Add a couple of lines to the release notes describing this new feature.
3. Update the llvm-mca docs.

Thank you! Can you please check if wording in the docs is ok?

Regarding the bottleneck analysis: it seems to work for some cases
(A55-all-views.s), but not for others (A55-out-of-order-retire.s).
I will check what is wrong here. The feature is disabled for now as you
suggested.

Thanks Andrew for the updated patch!

The doc changes look good to me.

The code changes also look good, except for one thing (I left a comment below).

About the bottleneck analysis:
The current analysis assumes an out-of-order pipeline. So I wouldn't be surprised if there are cases where it doesn't work well.
As you might already know, the bottleneck analysis effectively observes pressure events which are dynamically generated by a Scheduler component. Based on the observed trace, and it the predicts if throughput was somehow limited by the lack of processor resources, and/or data dependencies.
The good thing about your patch is that your code already generates pressure events (from your new canExecute() method). A bottleneck analysis for in-order processors would be much simpler, since we don't need any complex analysis based on the knowledge of which instructions are in a PENDING/READY state in a scheduler buffer.

The best thing to do for now is to raise an llvm-mca bug about implementing a bottleneck analysis for in-order processors. You can raise it now, or wait until this patch is finally committed (either way is fine).

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

84

As I wrote before, you should simply ignore the case where the number of cycles left is equal to UNKNOWN_CYCLES.
You shouldn't early exit with an arbitrary number of cycles.

Basically what I am saying is that you should replace this return statement with a continue;

[andreadb]

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

82

Sorry, I have only read your comment now.

It is true that, strictly speaking, UNKNOWN_CYCLES isn't equivalent to zero cycles.
However, in practice it shouldn't happen to have to deal with writes of UNKNOWN_CYCLES.
If I remember correctly, for the out-of-order simulator, we optimistically ignore it.

If you really want to enforce at least a 1-cycle delay for it, then fair enough. However you shouldn't return immediately. Instead, you should simply update the stall cycles quantity and then continue to the next iteration of the loop.

[asavonic]

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

84

As I wrote before, you should simply ignore the case where the number of cycles left is equal to UNKNOWN_CYCLES.
You shouldn't early exit with an arbitrary number of cycles.

Basically what I am saying is that you should replace this return statement with a continue;

I mentioned this above, but I might be missing something.

What exactly UNKNOWN_CYCLES means in this case? I assume that for in-order
pipeline it means that an instruction is issued (since we issue in-order), but
we don't know when its write is going to be completed. If this is correct, then
we should not issue any instructions that depend on this write until we know that the write
is completed (and has CyclesLeft == 0).

return 1 here stalls the instruction for 1 cycle, and it will be checked again
in the next cycle. If we continue here instead, then the write is ignored.

[asavonic]

Update StallCycles and continue instead of return,

[andreadb]

LGTM.

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

84

OK, I did some archaeology (it has been a while since when I wrote that code.)

Long story short:
When simulating in-order processors, you should never end up in a situation where the number of cycles left is UNKNOWN_CYCLES. Your code was not wrong. Adding a continue is also OK (but not necessarily better). Ideally, you could just assert that the number of cycles is different than UNKNOWN_CYCLES. Up to you.

Long story:
Each register write is associated with a CyclesLeft quantity. The number of cycles left is unknown until the instruction is issued. Only at that point, we know exactly how many cycles are left before writes are committed to the register file.

On instruction issue, dependent instructions get notified, so that the CyclesLeft quantities associated with their register reads are also updated.

This is important when simulating an out-of-order backend, because the scheduler may contain a chain of dependent instructions, and not all the instructions of that chain are in a "ready" state.

The relevant code is in Instruction.cpp (see for example method WriteState::onInstructionIssued).

Back to our case:

For in-order processors, instructions are always issued in-order, and instructions are not buffered internally by a scheduler. So the "dispatch" and "issue" are basically the same event.

In the presence of data dependencies, instructions are artificially delayed until all inputs become available (i.e. inputs have a known number of cycles left). Only at that point, instructions are issued to the underlying execution units.

By construction, you cannot end up in a situation where the "current instruction" depends on an older instruction which hasn't started execution yet.

To put it in another way: By construction, it is always guaranteed that "older instruction" have already started execution. Therefore, the CyclesLeft value from each write is always known.

[asavonic]

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

84

Thanks for the confirmation. This matches my analysis as well.
Instruction::execute always updates CyclesLeft, so we never have
UNKNOWN_CYCLES here. Let's keep the handling in case this changes at some point.

[andreadb]

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

84

Sounds good.

Thanks for working on this feature!

-Andrea

llvm/docs/CommandGuide/llvm-mca.rst

 DESCRIPTION
 -----------
 
 :program:`llvm-mca` is a performance analysis tool that uses information
 available in LLVM (e.g. scheduling models) to statically measure the performance
 of machine code in a specific CPU.
 
 Performance is measured in terms of throughput as well as processor resource
-consumption. The tool currently works for processors with an out-of-order
-backend, for which there is a scheduling model available in LLVM.
+consumption. The tool currently works for processors with a backend for which
+there is a scheduling model available in LLVM.
 
 The main goal of this tool is not just to predict the performance of the code
 when run on the target, but also help with diagnosing potential performance
 issues.
 
 Given an assembly code sequence, :program:`llvm-mca` estimates the Instructions
 Per Cycle (IPC), as well as hardware resource pressure. The analysis and
 reporting style were inspired by the IACA tool from Intel.
.. option:: -instruction-tables
   view because it doesn't require that the code is simulated. It instead prints
   the theoretical uniform distribution of resource pressure for every
   instruction in sequence.
 
 .. option:: -bottleneck-analysis
 
   Print information about bottlenecks that affect the throughput. This analysis
   can be expensive, and it is disabled by default.  Bottlenecks are highlighted
-  in the summary view.
+  in the summary view. Bottleneck analysis is currently not supported for
+  processors with an in-order backend.
 
 .. option:: -json
 
   Print the requested views in JSON format. The instructions and the processor
   resources are printed as members of special top level JSON objects.  The
   individual views refer to them by index.
 
 

 
 The report is structured in three main sections.  The first section collects a
 few performance numbers; the goal of this section is to give a very quick
 overview of the performance throughput. Important performance indicators are
 **IPC**, **uOps Per Cycle**, and  **Block RThroughput** (Block Reciprocal
 Throughput).
 
 Field *DispatchWidth* is the maximum number of micro opcodes that are dispatched
-to the out-of-order backend every simulated cycle.
+to the out-of-order backend every simulated cycle. For processors with an
+in-order backend, *DispatchWidth* is the maximum number of micro opcodes issued
+to the backend every simulated cycle.
 
 IPC is computed dividing the total number of simulated instructions by the total
 number of cycles.
 
 Field *Block RThroughput* is the reciprocal of the block throughput. Block
 throughput is a theoretical quantity computed as the maximum number of blocks
 (i.e. iterations) that can be executed per simulated clock cycle in the absence
 of loop carried dependencies. Block throughput is superiorly limited by the

 to the simulation. It is annotated to provide extra information about critical
 register dependencies and resource interferences between instructions.
 
 Instructions from the critical sequence are expected to significantly impact
 performance. By construction, the accuracy of this analysis is strongly
 dependent on the simulation and (as always) by the quality of the processor
 model in llvm.
 
+Bottleneck analysis is currently not supported for processors with an in-order
+backend.
 
 Extra Statistics to Further Diagnose Performance Issues
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 The ``-all-stats`` command line option enables extra statistics and performance
 counters for the dispatch logic, the reorder buffer, the retire control unit,
 and the register file.
 
 Below is an example of ``-all-stats`` output generated by  :program:`llvm-mca`

 The default pipeline implements the following sequence of stages used to
 process instructions.
 
 * Dispatch (Instruction is dispatched to the schedulers).
 * Issue (Instruction is issued to the processor pipelines).
 * Write Back (Instruction is executed, and results are written back).
 * Retire (Instruction is retired; writes are architecturally committed).
 
-The default pipeline only models the out-of-order portion of a processor.
-Therefore, the instruction fetch and decode stages are not modeled. Performance
-bottlenecks in the frontend are not diagnosed. :program:`llvm-mca` assumes that
-instructions have all been decoded and placed into a queue before the simulation
-start.  Also, :program:`llvm-mca` does not model branch prediction.
+The in-order pipeline implements the following sequence of stages:
+* InOrderIssue (Instruction is issued to the processor pipelines).
+* Retire (Instruction is retired; writes are architecturally committed).
+
+:program:`llvm-mca` assumes that instructions have all been decoded and placed
+into a queue before the simulation start. Therefore, the instruction fetch and
+decode stages are not modeled. Performance bottlenecks in the frontend are not
+diagnosed. Also, :program:`llvm-mca` does not model branch prediction.
 
 Instruction Dispatch
 """"""""""""""""""""
 During the dispatch stage, instructions are picked in program order from a
 queue of already decoded instructions, and dispatched in groups to the
 simulated hardware schedulers.
 
 The size of a dispatch group depends on the availability of the simulated

 In conclusion, the full set of load/store consistency rules are:
 
 #. A store may not pass a previous store.
 #. A store may not pass a previous load (regardless of ``-noalias``).
 #. A store has to wait until an older store barrier is fully executed.
 #. A load may pass a previous load.
 #. A load may not pass a previous store unless ``-noalias`` is set.
 #. A load has to wait until an older load barrier is fully executed.
+
+In-order Issue and Execute
+""""""""""""""""""""""""""""""""""""
+In-order processors are modelled as a single ``InOrderIssueStage`` stage. It
+bypasses Dispatch, Scheduler and Load/Store unit. Instructions are issued as
+soon as their operand registers are available and resource requirements are
+met. Multiple instructions can be issued in one cycle according to the value of
+the ``IssueWidth`` parameter in LLVM's scheduling model.
+
+Once issued, an instruction is moved to ``IssuedInst`` set until it is ready to
+retire. If ``RetireControlUnit`` is defined in the LLVM's scheduling model,
+:program:`llvm-mca` ensures that instructions are retired in-order. However, an
+instruction is allowed to retire out-of-order if ``RetireOOO`` property is true
+for at least one of its writes.

llvm/docs/ReleaseNotes.rst

 During this release ...
 
 Changes to the LLVM tools
 ---------------------------------
 
 * The options ``--build-id-link-{dir,input,output}`` have been deleted.
   (`D96310 <https://reviews.llvm.org/D96310>`_)
 
+* Support for in-order processors has been added to ``llvm-mca``.
+  (`D94928 <https://reviews.llvm.org/D94928>`_)
+
 Changes to LLDB
 ---------------------------------
 
 Changes to Sanitizers
 ---------------------
 
 External Open Source Projects Using LLVM 13
 ===========================================

llvm/include/llvm/MC/MCSchedule.h

struct MCReadAdvanceEntry {
   }
 };
 
 /// Summarize the scheduling resources required for an instruction of a
 /// particular scheduling class.
 ///
 /// Defined as an aggregate struct for creating tables with initializer lists.
 struct MCSchedClassDesc {
-  static const unsigned short InvalidNumMicroOps = (1U << 14) - 1;
+  static const unsigned short InvalidNumMicroOps = (1U << 13) - 1;
   static const unsigned short VariantNumMicroOps = InvalidNumMicroOps - 1;
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   const char* Name;
 #endif
-  uint16_t NumMicroOps : 14;
+  uint16_t NumMicroOps : 13;
   uint16_t BeginGroup : 1;
   uint16_t EndGroup : 1;
+  uint16_t RetireOOO : 1;
   uint16_t WriteProcResIdx; // First index into WriteProcResTable.
   uint16_t NumWriteProcResEntries;
   uint16_t WriteLatencyIdx; // First index into WriteLatencyTable.
   uint16_t NumWriteLatencyEntries;
   uint16_t ReadAdvanceIdx; // First index into ReadAdvanceTable.
   uint16_t NumReadAdvanceEntries;
 
   bool isValid() const {

llvm/include/llvm/MCA/Context.h

public:
   void addHardwareUnit(std::unique_ptr<HardwareUnit> H) {
     Hardware.push_back(std::move(H));
   }
 
   /// Construct a basic pipeline for simulating an out-of-order pipeline.
   /// This pipeline consists of Fetch, Dispatch, Execute, and Retire stages.
   std::unique_ptr<Pipeline> createDefaultPipeline(const PipelineOptions &Opts,
                                                   SourceMgr &SrcMgr);
+
+  /// Construct a basic pipeline for simulating an in-order pipeline.
+  /// This pipeline consists of Fetch, InOrderIssue, and Retire stages.
+  std::unique_ptr<Pipeline> createInOrderPipeline(const PipelineOptions &Opts,
+                                                  SourceMgr &SrcMgr);
 };
 
 } // namespace mca
 } // namespace llvm
 #endif // LLVM_MCA_CONTEXT_H

llvm/include/llvm/MCA/HardwareUnits/RegisterFile.h

class RegisterFile : public HardwareUnit {
   void allocatePhysRegs(const RegisterRenamingInfo &Entry,
                         MutableArrayRef<unsigned> UsedPhysRegs);
 
   // Releases previously allocated physical registers from the register file(s).
   // This method is called from `invalidateRegisterMapping()`.
   void freePhysRegs(const RegisterRenamingInfo &Entry,
                     MutableArrayRef<unsigned> FreedPhysRegs);
 
-  // Collects writes that are in a RAW dependency with RS.
-  // This method is called from `addRegisterRead()`.
-  void collectWrites(const ReadState &RS,
-                     SmallVectorImpl<WriteRef> &Writes) const;
-
   // Create an instance of RegisterMappingTracker for every register file
   // specified by the processor model.
   // If no register file is specified, then this method creates a default
   // register file with an unbounded number of physical registers.
   void initialize(const MCSchedModel &SM, unsigned NumRegs);
 
 public:
   RegisterFile(const MCSchedModel &SM, const MCRegisterInfo &mri,
                unsigned NumRegs = 0);
 
+  // Collects writes that are in a RAW dependency with RS.
+  void collectWrites(const ReadState &RS,
+                     SmallVectorImpl<WriteRef> &Writes) const;
+
   // This method updates the register mappings inserting a new register
   // definition. This method is also responsible for updating the number of
   // allocated physical registers in each register file modified by the write.
   // No physical regiser is allocated if this write is from a zero-idiom.
   void addRegisterWrite(WriteRef Write, MutableArrayRef<unsigned> UsedPhysRegs);
 
   // Collect writes that are in a data dependency with RS, and update RS
   // internal state.

llvm/include/llvm/MCA/HardwareUnits/RetireControlUnit.h

public:
   void consumeCurrentToken();
 
   // Update the RCU token to represent the executed state.
   void onInstructionExecuted(unsigned TokenID);
 
 #ifndef NDEBUG
   void dump() const;
 #endif
+
+  // Assigned to instructions that are not handled by the RCU.
+  static const unsigned UnhandledTokenID = ~0U;
 };
 
 } // namespace mca
 } // namespace llvm
 
 #endif // LLVM_MCA_HARDWAREUNITS_RETIRECONTROLUNIT_H

llvm/include/llvm/MCA/Instruction.h

struct InstrDesc {
   // subtarget when computing the reciprocal throughput.
   unsigned SchedClassID;
 
   bool MayLoad;
   bool MayStore;
   bool HasSideEffects;
   bool BeginGroup;
   bool EndGroup;
+  bool RetireOOO;
 
   // True if all buffered resources are in-order, and there is at least one
   // buffer which is a dispatch hazard (BufferSize = 0).
   bool MustIssueImmediately;
 
   // A zero latency instruction doesn't consume any scheduler resources.
   bool isZeroLatency() const { return !MaxLatency && Resources.empty(); }
 

llvm/include/llvm/MCA/Stages/InOrderIssueStage.h

+//===---------------------- InOrderIssueStage.h -----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// InOrderIssueStage implements an in-order execution pipeline.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_MCA_IN_ORDER_ISSUE_STAGE_H
+#define LLVM_MCA_IN_ORDER_ISSUE_STAGE_H
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/MCA/SourceMgr.h"
+#include "llvm/MCA/Stages/Stage.h"
+
+#include <queue>
+
+namespace llvm {
+struct MCSchedModel;
+class MCSubtargetInfo;
+
+namespace mca {
+class RegisterFile;
+class ResourceManager;
+struct RetireControlUnit;
+
+class InOrderIssueStage final : public Stage {
+  const MCSchedModel &SM;
+  const MCSubtargetInfo &STI;
+  RetireControlUnit &RCU;
+  RegisterFile &PRF;
+  std::unique_ptr<ResourceManager> RM;
+
+  /// Instructions that were issued, but not executed yet.
+  SmallVector<InstRef, 4> IssuedInst;
+
+  /// Number of instructions issued in the current cycle.
+  unsigned NumIssued;
+
+  /// If an instruction cannot execute due to an unmet register or resource
+  /// dependency, the it is stalled for StallCyclesLeft.
+  InstRef StalledInst;
+  unsigned StallCyclesLeft;
+
+  /// Number of instructions that can be issued in the current cycle.
+  unsigned Bandwidth;
+
+  InOrderIssueStage(const InOrderIssueStage &Other) = delete;
+  InOrderIssueStage &operator=(const InOrderIssueStage &Other) = delete;
+
+  /// If IR has an unmet register or resource dependency, canExecute returns
+  /// false. StallCycles is set to the number of cycles left before the
+  /// instruction can be issued.
+  bool canExecute(const InstRef &IR, unsigned *StallCycles) const;
+
+  /// Issue the instruction, or update StallCycles if IR is stalled.
+  Error tryIssue(InstRef &IR, unsigned *StallCycles);
+
+  /// Update status of instructions from IssuedInst.
+  Error updateIssuedInst();
+
+public:
+  InOrderIssueStage(RetireControlUnit &RCU, RegisterFile &PRF,
+                    const MCSchedModel &SM, const MCSubtargetInfo &STI)
+      : SM(SM), STI(STI), RCU(RCU), PRF(PRF),
+        RM(std::make_unique<ResourceManager>(SM)), StallCyclesLeft(0),
+        Bandwidth(0) {}
+
+  bool isAvailable(const InstRef &) const override;
+  bool hasWorkToComplete() const override;
+  Error execute(InstRef &IR) override;
+  Error cycleStart() override;
+  Error cycleEnd() override;
+};
+
+} // namespace mca
+} // namespace llvm
+
+#endif // LLVM_MCA_IN_ORDER_ISSUE_STAGE_H

llvm/include/llvm/MCA/Stages/RetireStage.h

 /// The RetireStage represents the process logic that interacts with the
 /// simulated RetireControlUnit hardware.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_MCA_STAGES_RETIRESTAGE_H
 #define LLVM_MCA_STAGES_RETIRESTAGE_H
 
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/MCA/HardwareUnits/LSUnit.h"
 #include "llvm/MCA/HardwareUnits/RegisterFile.h"
 #include "llvm/MCA/HardwareUnits/RetireControlUnit.h"
 #include "llvm/MCA/Stages/Stage.h"
 
 namespace llvm {
 namespace mca {
 
 class RetireStage final : public Stage {
   // Owner will go away when we move listeners/eventing to the stages.
   RetireControlUnit &RCU;
   RegisterFile &PRF;
   LSUnitBase &LSU;
+  SmallVector<InstRef, 4> RetireInst;
 
   RetireStage(const RetireStage &Other) = delete;
   RetireStage &operator=(const RetireStage &Other) = delete;
 
 public:
   RetireStage(RetireControlUnit &R, RegisterFile &F, LSUnitBase &LS)
       : Stage(), RCU(R), PRF(F), LSU(LS) {}
 
-  bool hasWorkToComplete() const override { return !RCU.isEmpty(); }
+  bool hasWorkToComplete() const override {
+    return !RCU.isEmpty() || !RetireInst.empty();
+  }
   Error cycleStart() override;
   Error execute(InstRef &IR) override;
   void notifyInstructionRetired(const InstRef &IR) const;
 };
 
 } // namespace mca
 } // namespace llvm
 
 #endif // LLVM_MCA_STAGES_RETIRESTAGE_H

llvm/include/llvm/Target/TargetSchedule.td

class ProcWriteResources<list<ProcResourceKind> resources> {
   bit BeginGroup = false;
   bit EndGroup = false;
   // Allow a processor to mark some scheduling classes as unsupported
   // for stronger verification.
   bit Unsupported = false;
   // Allow a processor to mark some scheduling classes as single-issue.
   // SingleIssue is an alias for Begin/End Group.
   bit SingleIssue = false;
+  // An instruction is allowed to retire out-of-order if RetireOOO is
+  // true for at least one of its writes. This field is only used by
+  // MCA for in-order subtargets, and is ignored for other targets.
+  bit RetireOOO = false;
   SchedMachineModel SchedModel = ?;
 }
 
 // Define the resources and latency of a SchedWrite. This will be used
 // directly by targets that have no itinerary classes. In this case,
 // SchedWrite is defined by the target, while WriteResources is
 // defined by the subtarget, and maps the SchedWrite to processor
 // resources.

llvm/lib/MCA/CMakeLists.txt

 add_llvm_component_library(LLVMMCA
   CodeEmitter.cpp
   Context.cpp
   HWEventListener.cpp
   HardwareUnits/HardwareUnit.cpp
   HardwareUnits/LSUnit.cpp
   HardwareUnits/RegisterFile.cpp
   HardwareUnits/ResourceManager.cpp
   HardwareUnits/RetireControlUnit.cpp
   HardwareUnits/Scheduler.cpp
   InstrBuilder.cpp
   Instruction.cpp
   Pipeline.cpp
   Stages/DispatchStage.cpp
   Stages/EntryStage.cpp
   Stages/ExecuteStage.cpp
+  Stages/InOrderIssueStage.cpp
   Stages/InstructionTables.cpp
   Stages/MicroOpQueueStage.cpp
   Stages/RetireStage.cpp
   Stages/Stage.cpp
   Support.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/MCA
 
   LINK_COMPONENTS
   MC
   Support
   )

llvm/lib/MCA/Context.cpp

 
 #include "llvm/MCA/Context.h"
 #include "llvm/MCA/HardwareUnits/RegisterFile.h"
 #include "llvm/MCA/HardwareUnits/RetireControlUnit.h"
 #include "llvm/MCA/HardwareUnits/Scheduler.h"
 #include "llvm/MCA/Stages/DispatchStage.h"
 #include "llvm/MCA/Stages/EntryStage.h"
 #include "llvm/MCA/Stages/ExecuteStage.h"
+#include "llvm/MCA/Stages/InOrderIssueStage.h"
 #include "llvm/MCA/Stages/MicroOpQueueStage.h"
 #include "llvm/MCA/Stages/RetireStage.h"
 
 namespace llvm {
 namespace mca {
 
 std::unique_ptr<Pipeline>
 Context::createDefaultPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr) {
   const MCSchedModel &SM = STI.getSchedModel();
 
+  if (!SM.isOutOfOrder())
+    return createInOrderPipeline(Opts, SrcMgr);
+
   // Create the hardware units defining the backend.
   auto RCU = std::make_unique<RetireControlUnit>(SM);
   auto PRF = std::make_unique<RegisterFile>(SM, MRI, Opts.RegisterFileSize);
   auto LSU = std::make_unique<LSUnit>(SM, Opts.LoadQueueSize,
                                        Opts.StoreQueueSize, Opts.AssumeNoAlias);
   auto HWS = std::make_unique<Scheduler>(SM, *LSU);
 
   // Create the pipeline stages.
   auto Fetch = std::make_unique<EntryStage>(SrcMgr);
   auto Dispatch = std::make_unique<DispatchStage>(STI, MRI, Opts.DispatchWidth,
                                                    *RCU, *PRF);
if (Opts.MicroOpQueueSize)
     StagePipeline->appendStage(std::make_unique<MicroOpQueueStage>(
         Opts.MicroOpQueueSize, Opts.DecodersThroughput));
   StagePipeline->appendStage(std::move(Dispatch));
   StagePipeline->appendStage(std::move(Execute));
   StagePipeline->appendStage(std::move(Retire));
   return StagePipeline;
 }
 
+std::unique_ptr<Pipeline>
+Context::createInOrderPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr) {
+  const MCSchedModel &SM = STI.getSchedModel();
+  auto RCU = std::make_unique<RetireControlUnit>(SM);
+  auto PRF = std::make_unique<RegisterFile>(SM, MRI, Opts.RegisterFileSize);
+  auto LSU = std::make_unique<LSUnit>(SM, Opts.LoadQueueSize,
+                                      Opts.StoreQueueSize, Opts.AssumeNoAlias);
+
+  auto Entry = std::make_unique<EntryStage>(SrcMgr);
+  auto InOrderIssue = std::make_unique<InOrderIssueStage>(*RCU, *PRF, SM, STI);
+  auto Retire = std::make_unique<RetireStage>(*RCU, *PRF, *LSU);
+
+  auto StagePipeline = std::make_unique<Pipeline>();
+  StagePipeline->appendStage(std::move(Entry));
+  StagePipeline->appendStage(std::move(InOrderIssue));
+  StagePipeline->appendStage(std::move(Retire));
+
+  addHardwareUnit(std::move(RCU));
+  addHardwareUnit(std::move(PRF));
+  addHardwareUnit(std::move(LSU));
+
+  return StagePipeline;
+}
+
 } // namespace mca
 } // namespace llvm

llvm/lib/MCA/HardwareUnits/RetireControlUnit.cpp

RetireControlUnit::RetireControlUnit(const MCSchedModel &SM)
   // and the maximum number of instructions retired per cycle.
   if (SM.hasExtraProcessorInfo()) {
     const MCExtraProcessorInfo &EPI = SM.getExtraProcessorInfo();
     if (EPI.ReorderBufferSize)
       AvailableEntries = EPI.ReorderBufferSize;
     MaxRetirePerCycle = EPI.MaxRetirePerCycle;
   }
   NumROBEntries = AvailableEntries;
+  bool IsOutOfOrder = SM.MicroOpBufferSize;
+  if (!IsOutOfOrder && !NumROBEntries)
+    return;
   assert(NumROBEntries && "Invalid reorder buffer size!");
   Queue.resize(2 * NumROBEntries);
 }
 
 // Reserves a number of slots, and returns a new token.
 unsigned RetireControlUnit::dispatch(const InstRef &IR) {
+  if (!NumROBEntries)
+    return UnhandledTokenID;
+
   const Instruction &Inst = *IR.getInstruction();
   unsigned Entries = normalizeQuantity(Inst.getNumMicroOps());
   assert((AvailableEntries >= Entries) && "Reorder Buffer unavailable!");
 
   unsigned TokenID = NextAvailableSlotIdx;
   Queue[NextAvailableSlotIdx] = {IR, Entries, false};
   NextAvailableSlotIdx += std::max(1U, Entries);
   NextAvailableSlotIdx %= Queue.size();
+  assert(TokenID < UnhandledTokenID && "Invalid token ID");
 
   AvailableEntries -= Entries;
   return TokenID;
 }
 
 const RetireControlUnit::RUToken &RetireControlUnit::getCurrentToken() const {
   const RetireControlUnit::RUToken &Current = Queue[CurrentInstructionSlotIdx];
 #ifndef NDEBUG

llvm/lib/MCA/InstrBuilder.cpp

if (MCDesc.isReturn() && FirstReturnInst) {
     FirstReturnInst = false;
   }
 
   ID->MayLoad = MCDesc.mayLoad();
   ID->MayStore = MCDesc.mayStore();
   ID->HasSideEffects = MCDesc.hasUnmodeledSideEffects();
   ID->BeginGroup = SCDesc.BeginGroup;
   ID->EndGroup = SCDesc.EndGroup;
+  ID->RetireOOO = SCDesc.RetireOOO;
 
   initializeUsedResources(*ID, SCDesc, STI, ProcResourceMasks);
   computeMaxLatency(*ID, MCDesc, SCDesc, STI);
 
   if (Error Err = verifyOperands(MCDesc, MCI))
     return std::move(Err);
 
   populateWrites(*ID, MCI, SchedClassID);

llvm/lib/MCA/Stages/InOrderIssueStage.cpp

+//===---------------------- InOrderIssueStage.cpp ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// InOrderIssueStage implements an in-order execution pipeline.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/MCA/Stages/InOrderIssueStage.h"
+
+#include "llvm/MC/MCSchedule.h"
+#include "llvm/MCA/HWEventListener.h"
+#include "llvm/MCA/HardwareUnits/RegisterFile.h"
+#include "llvm/MCA/HardwareUnits/ResourceManager.h"
+#include "llvm/MCA/HardwareUnits/RetireControlUnit.h"
+#include "llvm/MCA/Instruction.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Error.h"
+
+#include <algorithm>
+
+#define DEBUG_TYPE "llvm-mca"
+namespace llvm {
+namespace mca {
+
+bool InOrderIssueStage::hasWorkToComplete() const {
+  return !IssuedInst.empty() || StalledInst;
+}
+
+bool InOrderIssueStage::isAvailable(const InstRef &IR) const {
+  const Instruction &Inst = *IR.getInstruction();
+  unsigned NumMicroOps = Inst.getNumMicroOps();
+  const InstrDesc &Desc = Inst.getDesc();
+
+  if (Bandwidth < NumMicroOps)
+    return false;
+
+  // Instruction with BeginGroup must be the first instruction to be issued in a
+  // cycle.
+  if (Desc.BeginGroup && NumIssued != 0)
+    return false;
+
+  return true;
+}
+
+static bool hasResourceHazard(const ResourceManager &RM, const InstRef &IR) {
+  if (RM.checkAvailability(IR.getInstruction()->getDesc())) {
+    LLVM_DEBUG(dbgs() << "[E] Stall #" << IR << '\n');
+    return true;
+  }
+
+  return false;
+}
+
+/// Return a number of cycles left until register requirements of the
+/// instructions are met.
+static unsigned checkRegisterHazard(const RegisterFile &PRF,
+                                    const MCSchedModel &SM,
+                                    const MCSubtargetInfo &STI,
+                                    const InstRef &IR) {
+  unsigned StallCycles = 0;
+  SmallVector<WriteRef, 4> Writes;
+
+  for (const ReadState &RS : IR.getInstruction()->getUses()) {
+    const ReadDescriptor &RD = RS.getDescriptor();
+    const MCSchedClassDesc *SC = SM.getSchedClassDesc(RD.SchedClassID);
+
+    PRF.collectWrites(RS, Writes);
+    for (const WriteRef &WR : Writes) {
+      const WriteState *WS = WR.getWriteState();
+      unsigned WriteResID = WS->getWriteResourceID();
+      int ReadAdvance = STI.getReadAdvanceCycles(SC, RD.UseIndex, WriteResID);
+      LLVM_DEBUG(dbgs() << "[E] ReadAdvance for #" << IR << ": " << ReadAdvance
+                        << '\n');
+
+      if (WS->getCyclesLeft() == UNKNOWN_CYCLES) {
+        // Try again in the next cycle until the value is known
+        StallCycles = std::max(StallCycles, 1U);
+        continue;
+      }
+
+      int CyclesLeft = WS->getCyclesLeft() - ReadAdvance;
+      if (CyclesLeft > 0) {
+        LLVM_DEBUG(dbgs() << "[E] Register hazard: " << WS->getRegisterID()
+                          << '\n');
+        StallCycles = std::max(StallCycles, (unsigned)CyclesLeft);
+      }
+    }
+    Writes.clear();
+  }
+
+  return StallCycles;
+}
+
+bool InOrderIssueStage::canExecute(const InstRef &IR,
+                                   unsigned *StallCycles) const {
+  *StallCycles = 0;
+
+  if (unsigned RegStall = checkRegisterHazard(PRF, SM, STI, IR)) {
+    *StallCycles = RegStall;
+    // FIXME: add a parameter to HWStallEvent to indicate a number of cycles.
+    for (unsigned I = 0; I < RegStall; ++I) {
+      notifyEvent<HWStallEvent>(
+          HWStallEvent(HWStallEvent::RegisterFileStall, IR));
+      notifyEvent<HWPressureEvent>(
+          HWPressureEvent(HWPressureEvent::REGISTER_DEPS, IR));
+    }
+  } else if (hasResourceHazard(*RM, IR)) {
+    *StallCycles = 1;
+    notifyEvent<HWStallEvent>(
+        HWStallEvent(HWStallEvent::DispatchGroupStall, IR));
+    notifyEvent<HWPressureEvent>(
+        HWPressureEvent(HWPressureEvent::RESOURCES, IR));
+  }
+
+  return *StallCycles == 0;
+}
+
+static void addRegisterReadWrite(RegisterFile &PRF, Instruction &IS,
+                                 unsigned SourceIndex,
+                                 const MCSubtargetInfo &STI,
+                                 SmallVectorImpl<unsigned> &UsedRegs) {
+  assert(!IS.isEliminated());
+
+  for (ReadState &RS : IS.getUses())
+    PRF.addRegisterRead(RS, STI);
+
+  for (WriteState &WS : IS.getDefs())
+    PRF.addRegisterWrite(WriteRef(SourceIndex, &WS), UsedRegs);
+}
+
+static void notifyInstructionExecute(
+    const InstRef &IR,
+    const SmallVectorImpl<std::pair<ResourceRef, ResourceCycles>> &UsedRes,
+    const Stage &S) {
+
+  S.notifyEvent<HWInstructionEvent>(
+      HWInstructionEvent(HWInstructionEvent::Ready, IR));
+  S.notifyEvent<HWInstructionEvent>(HWInstructionIssuedEvent(IR, UsedRes));
+
+  LLVM_DEBUG(dbgs() << "[E] Issued #" << IR << "\n");
+}
+
+static void notifyInstructionDispatch(const InstRef &IR, unsigned Ops,
+                                      const SmallVectorImpl<unsigned> &UsedRegs,
+                                      const Stage &S) {
+
+  S.notifyEvent<HWInstructionEvent>(
+      HWInstructionDispatchedEvent(IR, UsedRegs, Ops));
+
+  LLVM_DEBUG(dbgs() << "[E] Dispatched #" << IR << "\n");
+}
+
+llvm::Error InOrderIssueStage::execute(InstRef &IR) {
+  Instruction &IS = *IR.getInstruction();
+  const InstrDesc &Desc = IS.getDesc();
+
+  unsigned RCUTokenID = RetireControlUnit::UnhandledTokenID;
+  if (!Desc.RetireOOO)
+    RCUTokenID = RCU.dispatch(IR);
+  IS.dispatch(RCUTokenID);
+
+  if (Desc.EndGroup) {
+    Bandwidth = 0;
+  } else {
+    unsigned NumMicroOps = IR.getInstruction()->getNumMicroOps();
+    assert(Bandwidth >= NumMicroOps);
+    Bandwidth -= NumMicroOps;
+  }
+
+  if (llvm::Error E = tryIssue(IR, &StallCyclesLeft))
+    return E;
+
+  if (StallCyclesLeft) {
+    StalledInst = IR;
+    Bandwidth = 0;
+  }
+
+  return llvm::ErrorSuccess();
+}
+
+llvm::Error InOrderIssueStage::tryIssue(InstRef &IR, unsigned *StallCycles) {
+  Instruction &IS = *IR.getInstruction();
+  unsigned SourceIndex = IR.getSourceIndex();
+
+  if (!canExecute(IR, StallCycles)) {
+    LLVM_DEBUG(dbgs() << "[E] Stalled #" << IR << " for " << *StallCycles
+                      << " cycles\n");
+    return llvm::ErrorSuccess();
+  }
+
+  SmallVector<unsigned, 4> UsedRegs(PRF.getNumRegisterFiles());
+  addRegisterReadWrite(PRF, IS, SourceIndex, STI, UsedRegs);
+
+  notifyInstructionDispatch(IR, IS.getDesc().NumMicroOps, UsedRegs, *this);
+
+  SmallVector<std::pair<ResourceRef, ResourceCycles>, 4> UsedResources;
+  RM->issueInstruction(IS.getDesc(), UsedResources);
+  IS.execute(SourceIndex);
+
+  // Replace resource masks with valid resource processor IDs.
+  for (std::pair<ResourceRef, ResourceCycles> &Use : UsedResources) {
+    uint64_t Mask = Use.first.first;
+    Use.first.first = RM->resolveResourceMask(Mask);
+  }
+  notifyInstructionExecute(IR, UsedResources, *this);
+
+  IssuedInst.push_back(IR);
+  ++NumIssued;
+
+  return llvm::ErrorSuccess();
+}
+
+llvm::Error InOrderIssueStage::updateIssuedInst() {
+  // Update other instructions. Executed instructions will be retired during the
+  // next cycle.
+  unsigned NumExecuted = 0;
+  for (auto I = IssuedInst.begin(), E = IssuedInst.end();
+       I != (E - NumExecuted);) {
+    InstRef &IR = *I;
+    Instruction &IS = *IR.getInstruction();
+
+    IS.cycleEvent();
+    if (!IS.isExecuted()) {
+      LLVM_DEBUG(dbgs() << "[E] Instruction #" << IR
+                        << " is still executing\n");
+      ++I;
+      continue;
+    }
+    notifyEvent<HWInstructionEvent>(
+        HWInstructionEvent(HWInstructionEvent::Executed, IR));
+
+    LLVM_DEBUG(dbgs() << "[E] Instruction #" << IR << " is executed\n");
+    ++NumExecuted;
+    std::iter_swap(I, E - NumExecuted);
+  }
+
+  // Retire instructions in the next cycle
+  if (NumExecuted) {
+    for (auto I = IssuedInst.end() - NumExecuted, E = IssuedInst.end(); I != E;
+         ++I) {
+      if (llvm::Error E = moveToTheNextStage(*I))
+        return E;
+    }
+    IssuedInst.resize(IssuedInst.size() - NumExecuted);
+  }
+
+  return llvm::ErrorSuccess();
+}
+
+llvm::Error InOrderIssueStage::cycleStart() {
+  NumIssued = 0;
+
+  // Release consumed resources.
+  SmallVector<ResourceRef, 4> Freed;
+  RM->cycleEvent(Freed);
+
+  if (llvm::Error E = updateIssuedInst())
+    return E;
+
+  // Issue instructions scheduled for this cycle
+  if (!StallCyclesLeft && StalledInst) {
+    if (llvm::Error E = tryIssue(StalledInst, &StallCyclesLeft))
+      return E;
+  }
+
+  if (!StallCyclesLeft) {
+    StalledInst.invalidate();
+    assert(NumIssued <= SM.IssueWidth && "Overflow.");
+    Bandwidth = SM.IssueWidth - NumIssued;
+  } else {
+    // The instruction is still stalled, cannot issue any new instructions in
+    // this cycle.
+    Bandwidth = 0;
+  }
+
+  return llvm::ErrorSuccess();
+}
+
+llvm::Error InOrderIssueStage::cycleEnd() {
+  if (StallCyclesLeft > 0)
+    --StallCyclesLeft;
+  return llvm::ErrorSuccess();
+}
+
+} // namespace mca
+} // namespace llvm

llvm/lib/MCA/Stages/RetireStage.cpp

 #include "llvm/Support/Debug.h"
 
 #define DEBUG_TYPE "llvm-mca"
 
 namespace llvm {
 namespace mca {
 
 llvm::Error RetireStage::cycleStart() {
-  if (RCU.isEmpty())
-    return llvm::ErrorSuccess();
-
   const unsigned MaxRetirePerCycle = RCU.getMaxRetirePerCycle();
   unsigned NumRetired = 0;
   while (!RCU.isEmpty()) {
     if (MaxRetirePerCycle != 0 && NumRetired == MaxRetirePerCycle)
       break;
     const RetireControlUnit::RUToken &Current = RCU.getCurrentToken();
     if (!Current.Executed)
       break;
     notifyInstructionRetired(Current.IR);
     RCU.consumeCurrentToken();
     NumRetired++;
   }
 
+  // Retire instructions that are not controlled by the RCU
+  for (InstRef &IR : RetireInst) {
+    IR.getInstruction()->retire();
+    notifyInstructionRetired(IR);
+  }
+  RetireInst.resize(0);
+
   return llvm::ErrorSuccess();
 }
 
 llvm::Error RetireStage::execute(InstRef &IR) {
-  RCU.onInstructionExecuted(IR.getInstruction()->getRCUTokenID());
+  Instruction &IS = *IR.getInstruction();
+
+  unsigned TokenID = IS.getRCUTokenID();
+  if (TokenID != RetireControlUnit::UnhandledTokenID) {
+    RCU.onInstructionExecuted(TokenID);
+    return llvm::ErrorSuccess();
+  }
+
+  RetireInst.push_back(IR);
   return llvm::ErrorSuccess();
 }
 
 void RetireStage::notifyInstructionRetired(const InstRef &IR) const {
   LLVM_DEBUG(llvm::dbgs() << "[E] Instruction Retired: #" << IR << '\n');
   llvm::SmallVector<unsigned, 4> FreedRegs(PRF.getNumRegisterFiles());
   const Instruction &Inst = *IR.getInstruction();
 

llvm/lib/Target/AArch64/AArch64SchedA55.td

 
 // FP ALU specific new schedwrite definitions
 def CortexA55WriteFPALU_F3 : SchedWriteRes<[CortexA55UnitFPALU]> { let Latency = 3;}
 def CortexA55WriteFPALU_F4 : SchedWriteRes<[CortexA55UnitFPALU]> { let Latency = 4;}
 def CortexA55WriteFPALU_F5 : SchedWriteRes<[CortexA55UnitFPALU]> { let Latency = 5;}
 
 // FP Mul, Div, Sqrt. Div/Sqrt are not pipelined
 def : WriteRes<WriteFMul, [CortexA55UnitFPMAC]> { let Latency = 4; }
+
+let RetireOOO = 1 in {
 def : WriteRes<WriteFDiv, [CortexA55UnitFPDIV]> { let Latency = 22;
                                             let ResourceCycles = [29]; }
 def CortexA55WriteFMAC : SchedWriteRes<[CortexA55UnitFPMAC]> { let Latency = 4; }
 def CortexA55WriteFDivHP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 8;
                                                      let ResourceCycles = [5]; }
 def CortexA55WriteFDivSP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 13;
                                                      let ResourceCycles = [10]; }
 def CortexA55WriteFDivDP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 22;
                                                      let ResourceCycles = [19]; }
 def CortexA55WriteFSqrtHP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 8;
                                                       let ResourceCycles = [5]; }
 def CortexA55WriteFSqrtSP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 12;
                                                       let ResourceCycles = [9]; }
 def CortexA55WriteFSqrtDP : SchedWriteRes<[CortexA55UnitFPDIV]> { let Latency = 22;
                                                       let ResourceCycles = [19]; }
-
+}
 //===----------------------------------------------------------------------===//
 // Subtarget-specific SchedRead types.
 
 def : ReadAdvance<ReadVLD, 0>;
 def : ReadAdvance<ReadExtrHi, 1>;
 def : ReadAdvance<ReadAdrBase, 1>;
 
 // ALU - ALU input operands are generally needed in EX1. An operand produced in

 def : InstRW<[CortexA55WriteFDivSP], (instrs FDIVSrr)>;
 def : InstRW<[CortexA55WriteFDivDP], (instrs FDIVDrr)>;
 def : InstRW<[CortexA55WriteFDivHP], (instregex "^FDIVv.*16$")>;
 def : InstRW<[CortexA55WriteFDivSP], (instregex "^FDIVv.*32$")>;
 def : InstRW<[CortexA55WriteFDivDP], (instregex "^FDIVv.*64$")>;
 def : InstRW<[CortexA55WriteFSqrtHP], (instregex "^.*SQRT.*16$")>;
 def : InstRW<[CortexA55WriteFSqrtSP], (instregex "^.*SQRT.*32$")>;
 def : InstRW<[CortexA55WriteFSqrtDP], (instregex "^.*SQRT.*64$")>;
+
+def A55RCU : RetireControlUnit<64, 0>;
 }

llvm/test/TableGen/InvalidMCSchedClassDesc.td

let CompleteModel = 0 in {
   def SchedModel_A: SchedMachineModel;
   def SchedModel_B: SchedMachineModel;
   def SchedModel_C: SchedMachineModel;
 }
 
 // Inst_B didn't have the resoures, and it is invalid.
 // CHECK: SchedModel_ASchedClasses[] = {
 // CHECK: {DBGFIELD("Inst_A")             1
-// CHECK-NEXT: {DBGFIELD("Inst_B")             16383 
+// CHECK-NEXT: {DBGFIELD("Inst_B")             8191
 let SchedModel = SchedModel_A in {
   def Write_A : SchedWriteRes<[]>;
   def : InstRW<[Write_A], (instrs Inst_A)>;
 }
 
 // Inst_A didn't have the resoures, and it is invalid.
 // CHECK: SchedModel_BSchedClasses[] = {
-// CHECK: {DBGFIELD("Inst_A")             16383 
+// CHECK: {DBGFIELD("Inst_A")             8191
 // CHECK-NEXT: {DBGFIELD("Inst_B")             1 
 let SchedModel = SchedModel_B in {
   def Write_B: SchedWriteRes<[]>; 
   def : InstRW<[Write_B], (instrs Inst_B)>;
 }
 
 // CHECK: SchedModel_CSchedClasses[] = {
 // CHECK: {DBGFIELD("Inst_A")             1

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-add-sequence.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --timeline --iterations=2 < %s | FileCheck %s
+
+add      w2, w3, #1
+add      w4, w3, #2, lsl #12
+add      w0, w4, #3
+add      w1, w0, #4
+
+# CHECK:      Iterations:        2
+# CHECK-NEXT: Instructions:      8
+# CHECK-NEXT: Total Cycles:      10
+# CHECK-NEXT: Total uOps:        8
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.80
+# CHECK-NEXT: IPC:               0.80
+# CHECK-NEXT: Block RThroughput: 2.0
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      3     0.50                        add w2, w3, #1
+# CHECK-NEXT:  1      3     0.50                        add w4, w3, #2, lsl #12
+# CHECK-NEXT:  1      3     0.50                        add w0, w4, #3
+# CHECK-NEXT:  1      3     0.50                        add w1, w0, #4
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - CortexA55UnitALU
+# CHECK-NEXT: [0.1] - CortexA55UnitALU
+# CHECK-NEXT: [1]   - CortexA55UnitB
+# CHECK-NEXT: [2]   - CortexA55UnitDiv
+# CHECK-NEXT: [3.0] - CortexA55UnitFPALU
+# CHECK-NEXT: [3.1] - CortexA55UnitFPALU
+# CHECK-NEXT: [4]   - CortexA55UnitFPDIV
+# CHECK-NEXT: [5.0] - CortexA55UnitFPMAC
+# CHECK-NEXT: [5.1] - CortexA55UnitFPMAC
+# CHECK-NEXT: [6]   - CortexA55UnitLd
+# CHECK-NEXT: [7]   - CortexA55UnitMAC
+# CHECK-NEXT: [8]   - CortexA55UnitSt
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]
+# CHECK-NEXT: 2.00   2.00    -      -      -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]    Instructions:
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -     add w2, w3, #1
+# CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -     add w4, w3, #2, lsl #12
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -     add w0, w4, #3
+# CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -     add w1, w0, #4
+
+# CHECK:      Timeline view:
+# CHECK-NEXT: Index     0123456789
+
+# CHECK:      [0,0]     DeeER.   .   add  w2, w3, #1
+# CHECK-NEXT: [0,1]     DeeER.   .   add  w4, w3, #2, lsl #12
+# CHECK-NEXT: [0,2]     .DeeER   .   add  w0, w4, #3
+# CHECK-NEXT: [0,3]     . DeeER  .   add  w1, w0, #4
+# CHECK-NEXT: [1,0]     . DeeER  .   add  w2, w3, #1
+# CHECK-NEXT: [1,1]     .  DeeER .   add  w4, w3, #2, lsl #12
+# CHECK-NEXT: [1,2]     .   DeeER.   add  w0, w4, #3
+# CHECK-NEXT: [1,3]     .    DeeER   add  w1, w0, #4
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     2     0.0    0.0    0.0       add  w2, w3, #1
+# CHECK-NEXT: 1.     2     0.0    0.0    0.0       add  w4, w3, #2, lsl #12
+# CHECK-NEXT: 2.     2     0.0    0.0    0.0       add  w0, w4, #3
+# CHECK-NEXT: 3.     2     0.0    0.0    0.0       add  w1, w0, #4
+# CHECK-NEXT:        2     0.0    0.0    0.0       <total>

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-stats.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --all-stats --iterations=2 < %s | FileCheck %s
+
+ldr w4, [x2], #4
+ldr w5, [x3]
+madd  w0, w5, w4, w0
+add x3, x3, x13
+subs  x1, x1, #1
+str  w0, [x21, x18, lsl #2]
+
+# CHECK:      Iterations:        2
+# CHECK-NEXT: Instructions:      12
+# CHECK-NEXT: Total Cycles:      21
+# CHECK-NEXT: Total uOps:        14
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.67
+# CHECK-NEXT: IPC:               0.57
+# CHECK-NEXT: Block RThroughput: 3.5
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  2      3     1.00    *                   ldr w4, [x2], #4
+# CHECK-NEXT:  1      3     1.00    *                   ldr w5, [x3]
+# CHECK-NEXT:  1      4     1.00                        madd  w0, w5, w4, w0
+# CHECK-NEXT:  1      3     0.50                        add x3, x3, x13
+# CHECK-NEXT:  1      3     0.50                        subs  x1, x1, #1
+# CHECK-NEXT:  1      4     1.00           *            str w0, [x21, x18, lsl #2]
+
+# CHECK:      Dynamic Dispatch Stall Cycles:
+# CHECK-NEXT: RAT     - Register unavailable:                      10  (47.6%)
+# CHECK-NEXT: RCU     - Retire tokens unavailable:                 0
+# CHECK-NEXT: SCHEDQ  - Scheduler full:                            0
+# CHECK-NEXT: LQ      - Load queue full:                           0
+# CHECK-NEXT: SQ      - Store queue full:                          0
+# CHECK-NEXT: GROUP   - Static restrictions on the dispatch group: 0
+
+# CHECK:      Dispatch Logic - number of cycles where we saw N micro opcodes dispatched:
+# CHECK-NEXT: [# dispatched], [# cycles]
+# CHECK-NEXT:  0,              11  (52.4%)
+# CHECK-NEXT:  1,              6  (28.6%)
+# CHECK-NEXT:  2,              4  (19.0%)
+
+# CHECK:      Schedulers - number of cycles where we saw N micro opcodes issued:
+# CHECK-NEXT: [# issued], [# cycles]
+# CHECK-NEXT:  0,          11  (52.4%)
+# CHECK-NEXT:  1,          6  (28.6%)
+# CHECK-NEXT:  2,          4  (19.0%)
+
+# CHECK:      Scheduler's queue usage:
+# CHECK-NEXT: No scheduler resources used.
+
+# CHECK:      Retire Control Unit - number of cycles where we saw N instructions retired:
+# CHECK-NEXT: [# retired], [# cycles]
+# CHECK-NEXT:  0,           14  (66.7%)
+# CHECK-NEXT:  1,           4  (19.0%)
+# CHECK-NEXT:  2,           1  (4.8%)
+# CHECK-NEXT:  3,           2  (9.5%)
+
+# CHECK:      Total ROB Entries:                64
+# CHECK-NEXT: Max Used ROB Entries:             6  ( 9.4% )
+# CHECK-NEXT: Average Used ROB Entries per cy:  2  ( 3.1% )
+
+# CHECK:      Register File statistics:
+# CHECK-NEXT: Total number of mappings created:    14
+# CHECK-NEXT: Max number of mappings used:         6
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - CortexA55UnitALU
+# CHECK-NEXT: [0.1] - CortexA55UnitALU
+# CHECK-NEXT: [1]   - CortexA55UnitB
+# CHECK-NEXT: [2]   - CortexA55UnitDiv
+# CHECK-NEXT: [3.0] - CortexA55UnitFPALU
+# CHECK-NEXT: [3.1] - CortexA55UnitFPALU
+# CHECK-NEXT: [4]   - CortexA55UnitFPDIV
+# CHECK-NEXT: [5.0] - CortexA55UnitFPMAC
+# CHECK-NEXT: [5.1] - CortexA55UnitFPMAC
+# CHECK-NEXT: [6]   - CortexA55UnitLd
+# CHECK-NEXT: [7]   - CortexA55UnitMAC
+# CHECK-NEXT: [8]   - CortexA55UnitSt
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]
+# CHECK-NEXT: 1.00   1.00    -      -      -      -      -      -      -     2.00   1.00   1.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]    Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     ldr w4, [x2], #4
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     ldr w5, [x3]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -     madd  w0, w5, w4, w0
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -     add x3, x3, x13
+# CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -     subs  x1, x1, #1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00   str w0, [x21, x18, lsl #2]

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-all-views.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --all-views --iterations=2 < %s | FileCheck %s
+
+ldr w4, [x2], #4
+ldr w5, [x3]
+madd  w0, w5, w4, w0
+add x3, x3, x13
+subs  x1, x1, #1
+str  w0, [x21, x18, lsl #2]
+
+# CHECK:      Iterations:        2
+# CHECK-NEXT: Instructions:      12
+# CHECK-NEXT: Total Cycles:      21
+# CHECK-NEXT: Total uOps:        14
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.67
+# CHECK-NEXT: IPC:               0.57
+# CHECK-NEXT: Block RThroughput: 3.5
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  2      3     1.00    *                   ldr w4, [x2], #4
+# CHECK-NEXT:  1      3     1.00    *                   ldr w5, [x3]
+# CHECK-NEXT:  1      4     1.00                        madd  w0, w5, w4, w0
+# CHECK-NEXT:  1      3     0.50                        add x3, x3, x13
+# CHECK-NEXT:  1      3     0.50                        subs  x1, x1, #1
+# CHECK-NEXT:  1      4     1.00           *            str w0, [x21, x18, lsl #2]
+
+# CHECK:      Dynamic Dispatch Stall Cycles:
+# CHECK-NEXT: RAT     - Register unavailable:                      10  (47.6%)
+# CHECK-NEXT: RCU     - Retire tokens unavailable:                 0
+# CHECK-NEXT: SCHEDQ  - Scheduler full:                            0
+# CHECK-NEXT: LQ      - Load queue full:                           0
+# CHECK-NEXT: SQ      - Store queue full:                          0
+# CHECK-NEXT: GROUP   - Static restrictions on the dispatch group: 0
+
+# CHECK:      Dispatch Logic - number of cycles where we saw N micro opcodes dispatched:
+# CHECK-NEXT: [# dispatched], [# cycles]
+# CHECK-NEXT:  0,              11  (52.4%)
+# CHECK-NEXT:  1,              6  (28.6%)
+# CHECK-NEXT:  2,              4  (19.0%)
+
+# CHECK:      Schedulers - number of cycles where we saw N micro opcodes issued:
+# CHECK-NEXT: [# issued], [# cycles]
+# CHECK-NEXT:  0,          11  (52.4%)
+# CHECK-NEXT:  1,          6  (28.6%)
+# CHECK-NEXT:  2,          4  (19.0%)
+
+# CHECK:      Scheduler's queue usage:
+# CHECK-NEXT: No scheduler resources used.
+
+# CHECK:      Retire Control Unit - number of cycles where we saw N instructions retired:
+# CHECK-NEXT: [# retired], [# cycles]
+# CHECK-NEXT:  0,           14  (66.7%)
+# CHECK-NEXT:  1,           4  (19.0%)
+# CHECK-NEXT:  2,           1  (4.8%)
+# CHECK-NEXT:  3,           2  (9.5%)
+
+# CHECK:      Total ROB Entries:                64
+# CHECK-NEXT: Max Used ROB Entries:             6  ( 9.4% )
+# CHECK-NEXT: Average Used ROB Entries per cy:  2  ( 3.1% )
+
+# CHECK:      Register File statistics:
+# CHECK-NEXT: Total number of mappings created:    14
+# CHECK-NEXT: Max number of mappings used:         6
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - CortexA55UnitALU
+# CHECK-NEXT: [0.1] - CortexA55UnitALU
+# CHECK-NEXT: [1]   - CortexA55UnitB
+# CHECK-NEXT: [2]   - CortexA55UnitDiv
+# CHECK-NEXT: [3.0] - CortexA55UnitFPALU
+# CHECK-NEXT: [3.1] - CortexA55UnitFPALU
+# CHECK-NEXT: [4]   - CortexA55UnitFPDIV
+# CHECK-NEXT: [5.0] - CortexA55UnitFPMAC
+# CHECK-NEXT: [5.1] - CortexA55UnitFPMAC
+# CHECK-NEXT: [6]   - CortexA55UnitLd
+# CHECK-NEXT: [7]   - CortexA55UnitMAC
+# CHECK-NEXT: [8]   - CortexA55UnitSt
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]
+# CHECK-NEXT: 1.00   1.00    -      -      -      -      -      -      -     2.00   1.00   1.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]    Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     ldr w4, [x2], #4
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -     ldr w5, [x3]
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -     1.00    -     madd  w0, w5, w4, w0
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -     add x3, x3, x13
+# CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -     subs  x1, x1, #1
+# CHECK-NEXT:  -      -      -      -      -      -      -      -      -      -      -     1.00   str w0, [x21, x18, lsl #2]
+
+# CHECK:      Timeline view:
+# CHECK-NEXT:                     0123456789
+# CHECK-NEXT: Index     0123456789          0
+
+# CHECK:      [0,0]     DeeER.    .    .    .   ldr w4, [x2], #4
+# CHECK-NEXT: [0,1]     .DeeER    .    .    .   ldr w5, [x3]
+# CHECK-NEXT: [0,2]     .   DeeeER.    .    .   madd  w0, w5, w4, w0
+# CHECK-NEXT: [0,3]     .   DeeE-R.    .    .   add x3, x3, x13
+# CHECK-NEXT: [0,4]     .    DeeER.    .    .   subs  x1, x1, #1
+# CHECK-NEXT: [0,5]     .    . DeeeER  .    .   str w0, [x21, x18, lsl #2]
+# CHECK-NEXT: [1,0]     .    .  DeeER  .    .   ldr w4, [x2], #4
+# CHECK-NEXT: [1,1]     .    .   DeeER .    .   ldr w5, [x3]
+# CHECK-NEXT: [1,2]     .    .    . DeeeER  .   madd  w0, w5, w4, w0
+# CHECK-NEXT: [1,3]     .    .    . DeeE-R  .   add x3, x3, x13
+# CHECK-NEXT: [1,4]     .    .    .  DeeER  .   subs  x1, x1, #1
+# CHECK-NEXT: [1,5]     .    .    .    DeeeER   str w0, [x21, x18, lsl #2]
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     2     0.0    0.0    0.0       ldr  w4, [x2], #4
+# CHECK-NEXT: 1.     2     0.0    0.0    0.0       ldr  w5, [x3]
+# CHECK-NEXT: 2.     2     0.0    0.0    0.0       madd w0, w5, w4, w0
+# CHECK-NEXT: 3.     2     0.0    0.0    1.0       add  x3, x3, x13
+# CHECK-NEXT: 4.     2     0.0    0.0    0.0       subs x1, x1, #1
+# CHECK-NEXT: 5.     2     0.0    0.0    0.0       str  w0, [x21, x18, lsl #2]
+# CHECK-NEXT:        2     0.0    0.0    0.2       <total>

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-in-order-retire.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --all-stats --all-views --iterations=2 < %s | FileCheck %s
+
+sdiv  w12, w21, w0
+add w8, w8, #1
+add w1, w2, w0
+add w3, w4, #1
+add w5, w6, w0
+add w7, w9, w0
+
+# CHECK:      Iterations:        2
+# CHECK-NEXT: Instructions:      12
+# CHECK-NEXT: Total Cycles:      18
+# CHECK-NEXT: Total uOps:        12
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.67
+# CHECK-NEXT: IPC:               0.67
+# CHECK-NEXT: Block RThroughput: 8.0
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      8     8.00                        sdiv  w12, w21, w0
+# CHECK-NEXT:  1      3     0.50                        add w8, w8, #1
+# CHECK-NEXT:  1      3     0.50                        add w1, w2, w0
+# CHECK-NEXT:  1      3     0.50                        add w3, w4, #1
+# CHECK-NEXT:  1      3     0.50                        add w5, w6, w0
+# CHECK-NEXT:  1      3     0.50                        add w7, w9, w0
+
+# CHECK:      Dynamic Dispatch Stall Cycles:
+# CHECK-NEXT: RAT     - Register unavailable:                      0
+# CHECK-NEXT: RCU     - Retire tokens unavailable:                 0
+# CHECK-NEXT: SCHEDQ  - Scheduler full:                            0
+# CHECK-NEXT: LQ      - Load queue full:                           0
+# CHECK-NEXT: SQ      - Store queue full:                          0
+# CHECK-NEXT: GROUP   - Static restrictions on the dispatch group: 5  (27.8%)
+
+# CHECK:      Dispatch Logic - number of cycles where we saw N micro opcodes dispatched:
+# CHECK-NEXT: [# dispatched], [# cycles]
+# CHECK-NEXT:  0,              12  (66.7%)
+# CHECK-NEXT:  2,              6  (33.3%)
+
+# CHECK:      Schedulers - number of cycles where we saw N micro opcodes issued:
+# CHECK-NEXT: [# issued], [# cycles]
+# CHECK-NEXT:  0,          12  (66.7%)
+# CHECK-NEXT:  2,          6  (33.3%)
+
+# CHECK:      Scheduler's queue usage:
+# CHECK-NEXT: No scheduler resources used.
+
+# CHECK:      Retire Control Unit - number of cycles where we saw N instructions retired:
+# CHECK-NEXT: [# retired], [# cycles]
+# CHECK-NEXT:  0,           16  (88.9%)
+# CHECK-NEXT:  6,           2  (11.1%)
+
+# CHECK:      Total ROB Entries:                64
+# CHECK-NEXT: Max Used ROB Entries:             8  ( 12.5% )
+# CHECK-NEXT: Average Used ROB Entries per cy:  5  ( 7.8% )
+
+# CHECK:      Register File statistics:
+# CHECK-NEXT: Total number of mappings created:    12
+# CHECK-NEXT: Max number of mappings used:         8
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - CortexA55UnitALU
+# CHECK-NEXT: [0.1] - CortexA55UnitALU
+# CHECK-NEXT: [1]   - CortexA55UnitB
+# CHECK-NEXT: [2]   - CortexA55UnitDiv
+# CHECK-NEXT: [3.0] - CortexA55UnitFPALU
+# CHECK-NEXT: [3.1] - CortexA55UnitFPALU
+# CHECK-NEXT: [4]   - CortexA55UnitFPDIV
+# CHECK-NEXT: [5.0] - CortexA55UnitFPMAC
+# CHECK-NEXT: [5.1] - CortexA55UnitFPMAC
+# CHECK-NEXT: [6]   - CortexA55UnitLd
+# CHECK-NEXT: [7]   - CortexA55UnitMAC
+# CHECK-NEXT: [8]   - CortexA55UnitSt
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]
+# CHECK-NEXT: 2.50   2.50    -     8.00    -      -      -      -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]    Instructions:
+# CHECK-NEXT:  -      -      -     8.00    -      -      -      -      -      -      -      -     sdiv  w12, w21, w0
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w8, w8, #1
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w1, w2, w0
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w3, w4, #1
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w5, w6, w0
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w7, w9, w0
+
+# CHECK:      Timeline view:
+# CHECK-NEXT:                     01234567
+# CHECK-NEXT: Index     0123456789
+
+# CHECK:      [0,0]     DeeeeeeeER.    . .   sdiv w12, w21, w0
+# CHECK-NEXT: [0,1]     DeeE-----R.    . .   add  w8, w8, #1
+# CHECK-NEXT: [0,2]     .DeeE----R.    . .   add  w1, w2, w0
+# CHECK-NEXT: [0,3]     .DeeE----R.    . .   add  w3, w4, #1
+# CHECK-NEXT: [0,4]     . DeeE---R.    . .   add  w5, w6, w0
+# CHECK-NEXT: [0,5]     . DeeE---R.    . .   add  w7, w9, w0
+# CHECK-NEXT: [1,0]     .    .  DeeeeeeeER   sdiv w12, w21, w0
+# CHECK-NEXT: [1,1]     .    .  DeeE-----R   add  w8, w8, #1
+# CHECK-NEXT: [1,2]     .    .   DeeE----R   add  w1, w2, w0
+# CHECK-NEXT: [1,3]     .    .   DeeE----R   add  w3, w4, #1
+# CHECK-NEXT: [1,4]     .    .    DeeE---R   add  w5, w6, w0
+# CHECK-NEXT: [1,5]     .    .    DeeE---R   add  w7, w9, w0
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     2     0.0    0.0    0.0       sdiv w12, w21, w0
+# CHECK-NEXT: 1.     2     0.0    0.0    5.0       add  w8, w8, #1
+# CHECK-NEXT: 2.     2     0.0    0.0    4.0       add  w1, w2, w0
+# CHECK-NEXT: 3.     2     0.0    0.0    4.0       add  w3, w4, #1
+# CHECK-NEXT: 4.     2     0.0    0.0    3.0       add  w5, w6, w0
+# CHECK-NEXT: 5.     2     0.0    0.0    3.0       add  w7, w9, w0
+# CHECK-NEXT:        2     0.0    0.0    3.2       <total>

llvm/test/tools/llvm-mca/AArch64/Cortex/A55-out-of-order-retire.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --all-stats --all-views --iterations=2 < %s | FileCheck %s
+
+fdiv  s1, s2, s3
+add w8, w8, #1
+add w1, w2, w0
+add w3, w4, #1
+add w5, w6, w0
+add w7, w9, w0
+
+# CHECK:      Iterations:        2
+# CHECK-NEXT: Instructions:      12
+# CHECK-NEXT: Total Cycles:      25
+# CHECK-NEXT: Total uOps:        12
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.48
+# CHECK-NEXT: IPC:               0.48
+# CHECK-NEXT: Block RThroughput: 10.0
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      13    10.00                       fdiv  s1, s2, s3
+# CHECK-NEXT:  1      3     0.50                        add w8, w8, #1
+# CHECK-NEXT:  1      3     0.50                        add w1, w2, w0
+# CHECK-NEXT:  1      3     0.50                        add w3, w4, #1
+# CHECK-NEXT:  1      3     0.50                        add w5, w6, w0
+# CHECK-NEXT:  1      3     0.50                        add w7, w9, w0
+
+# CHECK:      Dynamic Dispatch Stall Cycles:
+# CHECK-NEXT: RAT     - Register unavailable:                      0
+# CHECK-NEXT: RCU     - Retire tokens unavailable:                 0
+# CHECK-NEXT: SCHEDQ  - Scheduler full:                            0
+# CHECK-NEXT: LQ      - Load queue full:                           0
+# CHECK-NEXT: SQ      - Store queue full:                          0
+# CHECK-NEXT: GROUP   - Static restrictions on the dispatch group: 7  (28.0%)
+
+# CHECK:      Dispatch Logic - number of cycles where we saw N micro opcodes dispatched:
+# CHECK-NEXT: [# dispatched], [# cycles]
+# CHECK-NEXT:  0,              19  (76.0%)
+# CHECK-NEXT:  2,              6  (24.0%)
+
+# CHECK:      Schedulers - number of cycles where we saw N micro opcodes issued:
+# CHECK-NEXT: [# issued], [# cycles]
+# CHECK-NEXT:  0,          19  (76.0%)
+# CHECK-NEXT:  2,          6  (24.0%)
+
+# CHECK:      Scheduler's queue usage:
+# CHECK-NEXT: No scheduler resources used.
+
+# CHECK:      Retire Control Unit - number of cycles where we saw N instructions retired:
+# CHECK-NEXT: [# retired], [# cycles]
+# CHECK-NEXT:  0,           18  (72.0%)
+# CHECK-NEXT:  1,           2  (8.0%)
+# CHECK-NEXT:  2,           5  (20.0%)
+
+# CHECK:      Total ROB Entries:                64
+# CHECK-NEXT: Max Used ROB Entries:             7  ( 10.9% )
+# CHECK-NEXT: Average Used ROB Entries per cy:  2  ( 3.1% )
+
+# CHECK:      Register File statistics:
+# CHECK-NEXT: Total number of mappings created:    12
+# CHECK-NEXT: Max number of mappings used:         7
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - CortexA55UnitALU
+# CHECK-NEXT: [0.1] - CortexA55UnitALU
+# CHECK-NEXT: [1]   - CortexA55UnitB
+# CHECK-NEXT: [2]   - CortexA55UnitDiv
+# CHECK-NEXT: [3.0] - CortexA55UnitFPALU
+# CHECK-NEXT: [3.1] - CortexA55UnitFPALU
+# CHECK-NEXT: [4]   - CortexA55UnitFPDIV
+# CHECK-NEXT: [5.0] - CortexA55UnitFPMAC
+# CHECK-NEXT: [5.1] - CortexA55UnitFPMAC
+# CHECK-NEXT: [6]   - CortexA55UnitLd
+# CHECK-NEXT: [7]   - CortexA55UnitMAC
+# CHECK-NEXT: [8]   - CortexA55UnitSt
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]
+# CHECK-NEXT: 2.50   2.50    -      -      -      -     10.00   -      -      -      -      -
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2]    [3.0]  [3.1]  [4]    [5.0]  [5.1]  [6]    [7]    [8]    Instructions:
+# CHECK-NEXT:  -      -      -      -      -      -     10.00   -      -      -      -      -     fdiv  s1, s2, s3
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w8, w8, #1
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w1, w2, w0
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w3, w4, #1
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w5, w6, w0
+# CHECK-NEXT: 0.50   0.50    -      -      -      -      -      -      -      -      -      -     add w7, w9, w0
+
+# CHECK:      Timeline view:
+# CHECK-NEXT:                     0123456789
+# CHECK-NEXT: Index     0123456789          01234
+
+# CHECK:      [0,0]     DeeeeeeeeeeeeER.    .   .   fdiv  s1, s2, s3
+# CHECK-NEXT: [0,1]     DeeER.    .    .    .   .   add w8, w8, #1
+# CHECK-NEXT: [0,2]     .DeeER    .    .    .   .   add w1, w2, w0
+# CHECK-NEXT: [0,3]     .DeeER    .    .    .   .   add w3, w4, #1
+# CHECK-NEXT: [0,4]     . DeeER   .    .    .   .   add w5, w6, w0
+# CHECK-NEXT: [0,5]     . DeeER   .    .    .   .   add w7, w9, w0
+# CHECK-NEXT: [1,0]     .    .    DeeeeeeeeeeeeER   fdiv  s1, s2, s3
+# CHECK-NEXT: [1,1]     .    .    DeeER.    .   .   add w8, w8, #1
+# CHECK-NEXT: [1,2]     .    .    .DeeER    .   .   add w1, w2, w0
+# CHECK-NEXT: [1,3]     .    .    .DeeER    .   .   add w3, w4, #1
+# CHECK-NEXT: [1,4]     .    .    . DeeER   .   .   add w5, w6, w0
+# CHECK-NEXT: [1,5]     .    .    . DeeER   .   .   add w7, w9, w0
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     2     0.0    0.0    0.0       fdiv s1, s2, s3
+# CHECK-NEXT: 1.     2     0.0    0.0    0.0       add  w8, w8, #1
+# CHECK-NEXT: 2.     2     0.0    0.0    0.0       add  w1, w2, w0
+# CHECK-NEXT: 3.     2     0.0    0.0    0.0       add  w3, w4, #1
+# CHECK-NEXT: 4.     2     0.0    0.0    0.0       add  w5, w6, w0
+# CHECK-NEXT: 5.     2     0.0    0.0    0.0       add  w7, w9, w0
+# CHECK-NEXT:        2     0.0    0.0    0.0       <total>

llvm/test/tools/llvm-mca/AArch64/Cortex/in-order-bottleneck-analysis.s

+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --all-views < %s | FileCheck %s
+# CHECK-NOT: Throughput Bottlenecks
+
+# RUN: llvm-mca -mtriple=aarch64 -mcpu=cortex-a55 --bottleneck-analysis < %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=CHECK-WARN
+# CHECK-WARN: warning: bottleneck analysis is not supported for in-order CPU 'cortex-a55'
+
+add      w2, w3, #1
+

llvm/test/tools/llvm-mca/ARM/m7-negative-readadvance.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=arm -mcpu=cortex-m7 --timeline --iterations=1 < %s | FileCheck %s
+
+add r1, r1, #1
+# ReadAdvance: 0
+add r1, r1, #2
+# ReadAdvance: -1
+vldr d0, [r1]
+
+# CHECK:      Iterations:        1
+# CHECK-NEXT: Instructions:      3
+# CHECK-NEXT: Total Cycles:      7
+# CHECK-NEXT: Total uOps:        3
+
+# CHECK:      Dispatch Width:    2
+# CHECK-NEXT: uOps Per Cycle:    0.43
+# CHECK-NEXT: IPC:               0.43
+# CHECK-NEXT: Block RThroughput: 1.5
+
+# CHECK:      Instruction Info:
+# CHECK-NEXT: [1]: #uOps
+# CHECK-NEXT: [2]: Latency
+# CHECK-NEXT: [3]: RThroughput
+# CHECK-NEXT: [4]: MayLoad
+# CHECK-NEXT: [5]: MayStore
+# CHECK-NEXT: [6]: HasSideEffects (U)
+
+# CHECK:      [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
+# CHECK-NEXT:  1      1     0.50                        add.w r1, r1, #1
+# CHECK-NEXT:  1      1     0.50                        add.w r1, r1, #2
+# CHECK-NEXT:  1      3     1.00    *                   vldr  d0, [r1]
+
+# CHECK:      Resources:
+# CHECK-NEXT: [0.0] - M7UnitALU
+# CHECK-NEXT: [0.1] - M7UnitALU
+# CHECK-NEXT: [1]   - M7UnitBranch
+# CHECK-NEXT: [2.0] - M7UnitLoad
+# CHECK-NEXT: [2.1] - M7UnitLoad
+# CHECK-NEXT: [3]   - M7UnitMAC
+# CHECK-NEXT: [4]   - M7UnitSIMD
+# CHECK-NEXT: [5]   - M7UnitShift1
+# CHECK-NEXT: [6]   - M7UnitShift2
+# CHECK-NEXT: [7]   - M7UnitStore
+# CHECK-NEXT: [8]   - M7UnitVFP
+# CHECK-NEXT: [9.0] - M7UnitVPort
+# CHECK-NEXT: [9.1] - M7UnitVPort
+
+# CHECK:      Resource pressure per iteration:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2.0]  [2.1]  [3]    [4]    [5]    [6]    [7]    [8]    [9.0]  [9.1]
+# CHECK-NEXT: 1.00   1.00    -      -     1.00    -      -      -      -      -      -      -     2.00
+
+# CHECK:      Resource pressure by instruction:
+# CHECK-NEXT: [0.0]  [0.1]  [1]    [2.0]  [2.1]  [3]    [4]    [5]    [6]    [7]    [8]    [9.0]  [9.1]  Instructions:
+# CHECK-NEXT:  -     1.00    -      -      -      -      -      -      -      -      -      -      -     add.w  r1, r1, #1
+# CHECK-NEXT: 1.00    -      -      -      -      -      -      -      -      -      -      -      -     add.w  r1, r1, #2
+# CHECK-NEXT:  -      -      -      -     1.00    -      -      -      -      -      -      -     2.00   vldr d0, [r1]
+
+# CHECK:      Timeline view:
+# CHECK-NEXT: Index     0123456
+
+# CHECK:      [0,0]     DER  ..   add.w r1, r1, #1
+# CHECK-NEXT: [0,1]     .DER ..   add.w r1, r1, #2
+# CHECK-NEXT: [0,2]     .  DeER   vldr  d0, [r1]
+
+# CHECK:      Average Wait times (based on the timeline view):
+# CHECK-NEXT: [0]: Executions
+# CHECK-NEXT: [1]: Average time spent waiting in a scheduler's queue
+# CHECK-NEXT: [2]: Average time spent waiting in a scheduler's queue while ready
+# CHECK-NEXT: [3]: Average time elapsed from WB until retire stage
+
+# CHECK:            [0]    [1]    [2]    [3]
+# CHECK-NEXT: 0.     1     0.0    0.0    0.0       add.w  r1, r1, #1
+# CHECK-NEXT: 1.     1     0.0    0.0    0.0       add.w  r1, r1, #2
+# CHECK-NEXT: 2.     1     0.0    0.0    0.0       vldr d0, [r1]
+# CHECK-NEXT:        1     0.0    0.0    0.0       <total>

llvm/test/tools/llvm-mca/X86/in-order-cpu.s

-# RUN: not llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=atom -o /dev/null 2>&1 | FileCheck %s
-
-# CHECK: error: please specify an out-of-order cpu. 'atom' is an in-order cpu.
+# RUN: llvm-mca %s -mtriple=x86_64-unknown-unknown -mcpu=atom -o /dev/null 2>&1 | FileCheck %s
+# CHECK: warning: support for in-order CPU 'atom' is experimental.
+movsbw  %al, %di

llvm/tools/llvm-mca/llvm-mca.cpp

 }
 } // end of anonymous namespace
 
 static void processOptionImpl(cl::opt<bool> &O, const cl::opt<bool> &Default) {
   if (!O.getNumOccurrences() || O.getPosition() < Default.getPosition())
     O = Default.getValue();
 }
 
-static void processViewOptions() {
+static void processViewOptions(bool IsOutOfOrder) {
   if (!EnableAllViews.getNumOccurrences() &&
       !EnableAllStats.getNumOccurrences())
     return;
 
   if (EnableAllViews.getNumOccurrences()) {
     processOptionImpl(PrintSummaryView, EnableAllViews);
+    if (IsOutOfOrder)
       processOptionImpl(EnableBottleneckAnalysis, EnableAllViews);
     processOptionImpl(PrintResourcePressureView, EnableAllViews);
     processOptionImpl(PrintTimelineView, EnableAllViews);
     processOptionImpl(PrintInstructionInfoView, EnableAllViews);
   }
 
   const cl::opt<bool> &Default =
       EnableAllViews.getPosition() < EnableAllStats.getPosition()
           ? EnableAllStats
int main(int argc, char **argv) {
 
   ErrorOr<std::unique_ptr<MemoryBuffer>> BufferPtr =
       MemoryBuffer::getFileOrSTDIN(InputFilename);
   if (std::error_code EC = BufferPtr.getError()) {
     WithColor::error() << InputFilename << ": " << EC.message() << '\n';
     return 1;
   }
 
-  // Apply overrides to llvm-mca specific options.
-  processViewOptions();
-
   if (MCPU == "native")
     MCPU = std::string(llvm::sys::getHostCPUName());
 
   std::unique_ptr<MCSubtargetInfo> STI(
       TheTarget->createMCSubtargetInfo(TripleName, MCPU, MATTR));
   assert(STI && "Unable to create subtarget info!");
   if (!STI->isCPUStringValid(MCPU))
     return 1;
 
-  if (!PrintInstructionTables && !STI->getSchedModel().isOutOfOrder()) {
-    WithColor::error() << "please specify an out-of-order cpu. '" << MCPU
-                       << "' is an in-order cpu.\n";
-    return 1;
+  bool IsOutOfOrder = STI->getSchedModel().isOutOfOrder();
+  if (!PrintInstructionTables && !IsOutOfOrder) {
+    WithColor::warning() << "support for in-order CPU '" << MCPU
+                         << "' is experimental.\n";
   }
 
   if (!STI->getSchedModel().hasInstrSchedModel()) {
     WithColor::error()
         << "unable to find instruction-level scheduling information for"
         << " target triple '" << TheTriple.normalize() << "' and cpu '" << MCPU
         << "'.\n";
 
     if (STI->getSchedModel().InstrItineraries)
       WithColor::note()
           << "cpu '" << MCPU << "' provides itineraries. However, "
           << "instruction itineraries are currently unsupported.\n";
     return 1;
   }
 
+  // Apply overrides to llvm-mca specific options.
+  processViewOptions(IsOutOfOrder);
+
   std::unique_ptr<MCRegisterInfo> MRI(TheTarget->createMCRegInfo(TripleName));
   assert(MRI && "Unable to create target register info!");
 
   MCTargetOptions MCOptions = mc::InitMCTargetOptionsFromFlags();
   std::unique_ptr<MCAsmInfo> MAI(
       TheTarget->createMCAsmInfo(*MRI, TripleName, MCOptions));
   assert(MAI && "Unable to create target asm info!");
 
if (PrintJson)
       Printer.addView(
           std::make_unique<mca::InstructionView>(*STI, *IP, Insts, MCPU));
 
     if (PrintSummaryView)
       Printer.addView(
           std::make_unique<mca::SummaryView>(SM, Insts, DispatchWidth));
 
     if (EnableBottleneckAnalysis) {
+      if (!IsOutOfOrder) {
+        WithColor::warning()
+            << "bottleneck analysis is not supported for in-order CPU '" << MCPU
+            << "'.\n";
+      }
       Printer.addView(std::make_unique<mca::BottleneckAnalysis>(
           *STI, *IP, Insts, S.getNumIterations()));
     }
 
     if (PrintInstructionInfoView)
       Printer.addView(std::make_unique<mca::InstructionInfoView>(
           *STI, *MCII, CE, ShowEncoding, Insts, *IP));
 

llvm/utils/TableGen/SubtargetEmitter.cpp

for (const CodeGenSchedClass &SC : SchedModels.schedClasses()) {
     LLVM_DEBUG(SC.dump(&SchedModels));
 
     SCTab.resize(SCTab.size() + 1);
     MCSchedClassDesc &SCDesc = SCTab.back();
     // SCDesc.Name is guarded by NDEBUG
     SCDesc.NumMicroOps = 0;
     SCDesc.BeginGroup = false;
     SCDesc.EndGroup = false;
+    SCDesc.RetireOOO = false;
     SCDesc.WriteProcResIdx = 0;
     SCDesc.WriteLatencyIdx = 0;
     SCDesc.ReadAdvanceIdx = 0;
 
     // A Variant SchedClass has no resources of its own.
     bool HasVariants = false;
     for (const CodeGenSchedTransition &CGT :
            make_range(SC.Transitions.begin(), SC.Transitions.end())) {
for (unsigned W : Writes) {
           break;
         }
         WLEntry.Cycles += WriteRes->getValueAsInt("Latency");
         SCDesc.NumMicroOps += WriteRes->getValueAsInt("NumMicroOps");
         SCDesc.BeginGroup |= WriteRes->getValueAsBit("BeginGroup");
         SCDesc.EndGroup |= WriteRes->getValueAsBit("EndGroup");
         SCDesc.BeginGroup |= WriteRes->getValueAsBit("SingleIssue");
         SCDesc.EndGroup |= WriteRes->getValueAsBit("SingleIssue");
+        SCDesc.RetireOOO |= WriteRes->getValueAsBit("RetireOOO");
 
         // Create an entry for each ProcResource listed in WriteRes.
         RecVec PRVec = WriteRes->getValueAsListOfDefs("ProcResources");
         std::vector<int64_t> Cycles =
           WriteRes->getValueAsListOfInts("ResourceCycles");
 
         if (Cycles.empty()) {
           // If ResourceCycles is not provided, default to one cycle per
void SubtargetEmitter::EmitSchedClassTables(SchedClassTables &SchedTables,
   for (CodeGenSchedModels::ProcIter PI = SchedModels.procModelBegin(),
          PE = SchedModels.procModelEnd(); PI != PE; ++PI) {
     if (!PI->hasInstrSchedModel())
       continue;
 
     std::vector<MCSchedClassDesc> &SCTab =
       SchedTables.ProcSchedClasses[1 + (PI - SchedModels.procModelBegin())];
 
-    OS << "\n// {Name, NumMicroOps, BeginGroup, EndGroup,"
+    OS << "\n// {Name, NumMicroOps, BeginGroup, EndGroup, RetireOOO,"
        << " WriteProcResIdx,#, WriteLatencyIdx,#, ReadAdvanceIdx,#}\n";
     OS << "static const llvm::MCSchedClassDesc "
        << PI->ModelName << "SchedClasses[] = {\n";
 
     // The first class is always invalid. We no way to distinguish it except by
     // name and position.
     assert(SchedModels.getSchedClass(0).Name == "NoInstrModel"
            && "invalid class not first");
     OS << "  {DBGFIELD(\"InvalidSchedClass\")  "
        << MCSchedClassDesc::InvalidNumMicroOps
-       << ", false, false,  0, 0,  0, 0,  0, 0},\n";
+       << ", false, false, false, 0, 0,  0, 0,  0, 0},\n";
 
     for (unsigned SCIdx = 1, SCEnd = SCTab.size(); SCIdx != SCEnd; ++SCIdx) {
       MCSchedClassDesc &MCDesc = SCTab[SCIdx];
       const CodeGenSchedClass &SchedClass = SchedModels.getSchedClass(SCIdx);
       OS << "  {DBGFIELD(\"" << SchedClass.Name << "\") ";
       if (SchedClass.Name.size() < 18)
         OS.indent(18 - SchedClass.Name.size());
       OS << MCDesc.NumMicroOps
          << ", " << ( MCDesc.BeginGroup ? "true" : "false" )
          << ", " << ( MCDesc.EndGroup ? "true" : "false" )
+         << ", " << ( MCDesc.RetireOOO ? "true" : "false" )
          << ", " << format("%2d", MCDesc.WriteProcResIdx)
          << ", " << MCDesc.NumWriteProcResEntries
          << ", " << format("%2d", MCDesc.WriteLatencyIdx)
          << ", " << MCDesc.NumWriteLatencyEntries
          << ", " << format("%2d", MCDesc.ReadAdvanceIdx)
          << ", " << MCDesc.NumReadAdvanceEntries
          << "}, // #" << SCIdx << '\n';
     }