[X86][Znver1] Specify Register Files, RCU; FP scheduler capacity.

[lebedev.ri]

Summary

First off: i do not have any access to that processor,
so this is purely theoretical, no benchmarks.

I have been looking into bdver2 scheduling profile, and while cross-referencing
the existing btver2, znver1 profiles, and the reference docs
(Software Optimization Guide for AMD Family {15,16,17}h Processors),
i have noticed that only btver2 scheduling profile specifies these.

Also, there is no mca test coverage.

So i'm wondering if this info should be specified?

[GGanesh]

lib/Target/X86/X86ScheduleZnver1.td

96

I am not sure why we include CCR register.

100

This number is fine.

111

The retire unit is shared between integer and FP ops. In SMT mode it is 96 entry per thread. So, I think we shall consider only 96 entry as a conservative value.

113

I assume these are the load/store queue entries. The FPU has

1. 44 entry Load Queue
2. 72 Out of Order Loads
3. 44 entry Store Queue

So, we are concerned only about the queues, we can have only 44 marked for LD and ST.

[lebedev.ri]

Addressed @GGanesh's review notes.

[lebedev.ri]

lib/Target/X86/X86ScheduleZnver1.td

111

Aha, i was wondering how SMT was considered here.
But then what about MicroOpBufferSize in SchedMachineModel?
Is that supposed to be keep at 192?

113

I think i would prefer to keep this just as a fixme note for now, since
while llvm-mca does have an options to specify the sizes of these queues,
it does not seem to read those from the sched model otherwise,
i guess because it can't currently be expressed here.

[lebedev.ri]

Any further comments?

[lebedev.ri]

Split off the baseline into separate differential, rebased.

[andreadb]

lib/Target/X86/X86ScheduleZnver1.td

96

As a rule of thumb:
a register class should only be added if you want to allow those registers to be renamed. In this case, CCR should be added if you want to count EFLAGS renames agains the number of available physical registers in the PRF.
If you don't think EFLAGS renames should be counted against the total, then it is correct to remove it. Otherwise, it is incorrect.

111

llvm-mca doesn't make assumptions on whether the CPU is in SMT mode or not. Same for the scheduling model, which assumes an optimistic micro-op buffer. For now, it is better to specify the resources seen available by a single thread running on the CPU (i.e. with no other concurrent threads).
Basically, I think you should not go for a conservative value here.

113

In theory, the processor model could specify load/store queues as a buffered processor resources that are consumed by instructions.
In practice, llvm-mca has the concept of LSUnit. There is an open bugzilla about how to better describe load/store queues in llvm-mca and the scheduling model.

[lebedev.ri]

Rebased, added CCR back.

lib/Target/X86/X86ScheduleZnver1.td

96

I'm not seeing anything in docs, so i'm going to be conservative, and add it back.

111

Hmm. Is that documented somewhere?
I would have naively expected that what @GGanesh wrote is the approach..

113

(That matches what i said)

[andreadb]

lib/Target/X86/X86ScheduleZnver1.td

111

This limitation was mentioned in the RFC.

You cannot possibly make any reasonable assumptions with SMT. The problem is not just the reorder buffer but any other resources which may or may not be competitively shared (or statically/dynamically partitioned).
Even if we know that the cpu is multi threaded, there is no way to predict how the other thread will use/consume hardware resources.

[lebedev.ri]

lib/Target/X86/X86ScheduleZnver1.td

111

Hmm.
https://support.amd.com/TechDocs/55723_SOG_Fam_17h_Processors_3.00.pdf
Page 35:
The retire queue can hold up to 192 micro ops or 96 per thread in SMT mode.

[lebedev.ri]

@GGanesh any further thoughts?

[andreadb]

lib/Target/X86/X86ScheduleZnver1.td

111

You shouldn't be making assumptions on SMT and use a conservative value for the retire queue. See my previous comment.
Without a proper framework to emulate SMT in llvm-mca, It is better to keep this value to the theoretical maximum. At least, you will get more accurate numbers when there is only a single thread active on the cpu.

[lebedev.ri]

lib/Target/X86/X86ScheduleZnver1.td

111

You shouldn't be making assumptions on SMT

Ah, on the SMT itself too, not only on what happens in SMT mode. I missed that remark initially.
Does the scheduling use RetireControlUnit<> values, or is it only used in mca right now?

[andreadb]

lib/Target/X86/X86ScheduleZnver1.td

111

My point is that in an SMT processor, resources (not just the ROB) are shared by multiple hardware threads.
Some resources may be competitively shared among the threads. Not knowing what the other thread is doing, makes it impossible to correctly predict how buffers are used.

By setting the RCU size to half of its capacity, you are assuming that the processor is conservatively running in SMT mode. However, you are mixing apples with oranges here. You are not saying how other resources should behave in SMT mode. What about the schedulers, register file, dispatch logic in SMT mode?

My point is: there is no framework at the moment to deal with SMT in general. So, there is no advantage in using a conservative value for the RCU here (unless you think it better matches your perf results).

To answer to your last question:
for now, the RetireControlUnit information is only used by mca.

[lebedev.ri]

Don't be conservative re RetireControlUnit size.

lib/Target/X86/X86ScheduleZnver1.td

111

Thank you for this detailed explanation!

To answer to your last question:

for now, the RetireControlUnit information is only used by mca.

Ok then!

[GGanesh]

LGTM!

[lebedev.ri]

In D47676#1137291, @GGanesh wrote:

LGTM!

Thank you for the review!
Could you please also click the accept button? :)

lib/Target/X86/X86ScheduleZnver1.td

 
 // Four FPU units are defined below
 def ZnFPU0 : ProcResource<1>;
 def ZnFPU1 : ProcResource<1>;
 def ZnFPU2 : ProcResource<1>;
 def ZnFPU3 : ProcResource<1>;
 
 // FPU grouping
-def ZnFPU     : ProcResGroup<[ZnFPU0, ZnFPU1, ZnFPU2, ZnFPU3]>;
 def ZnFPU013  : ProcResGroup<[ZnFPU0, ZnFPU1, ZnFPU3]>;
 def ZnFPU01   : ProcResGroup<[ZnFPU0, ZnFPU1]>;
 def ZnFPU12   : ProcResGroup<[ZnFPU1, ZnFPU2]>;
 def ZnFPU13   : ProcResGroup<[ZnFPU1, ZnFPU3]>;
 def ZnFPU23   : ProcResGroup<[ZnFPU2, ZnFPU3]>;
 def ZnFPU02   : ProcResGroup<[ZnFPU0, ZnFPU2]>;
 def ZnFPU03   : ProcResGroup<[ZnFPU0, ZnFPU3]>;
 

 def ZnMultiplier : ProcResource<1>;
 
 // Integer division issued on ALU2.
 def ZnDivider : ProcResource<1>;
 
 // 4 Cycles load-to use Latency is captured
 def : ReadAdvance<ReadAfterLd, 4>;
 
+// The Integer PRF for Zen is 168 entries, and it holds the architectural and
+// speculative version of the 64-bit integer registers.
+// Reference: "Software Optimization Guide for AMD Family 17h Processors"
+def ZnIntegerPRF : RegisterFile<168, [GR8, GR16, GR32, GR64]>;
+
+// 36 Entry (9x4 entries) floating-point Scheduler
+def ZnFPU     : ProcResGroup<[ZnFPU0, ZnFPU1, ZnFPU2, ZnFPU3]> {
+let BufferSize=36;
+}
+
+// The Zen FP Retire Queue renames SIMD and FP uOps onto a pool of 160 128-bit
+// registers. Operations on 256-bit data types are cracked into two COPs.
+// Reference: "Software Optimization Guide for AMD Family 17h Processors"
+def ZnFpuPRF: RegisterFile<160, [VR64, VR128, VR256], [1, 1, 2]>;
+
+// The unit can track up to 192 macro ops in-flight.
+// The retire unit handles in-order commit of up to 8 macro ops per cycle.
+// Reference: "Software Optimization Guide for AMD Family 17h Processors"
+// However, the retire unit is shared between integer and FP ops. In SMT mode
+// it is 96 entry per thread. So we shall use that as conservative value.
+def ZnRCU : RetireControlUnit<96, 8>;
+
+// FIXME: there are 72 read buffers and 44 write buffers.
+
 // (a folded load is an instruction that loads and does some operation)
 // Ex: ADDPD xmm,[mem]-> This instruction has two micro-ops
 // Instructions with folded loads are usually micro-fused, so they only appear
 // as two micro-ops.
 //      a. load and
 //      b. addpd
 // This multiclass is for folded loads for integer units.
 multiclass ZnWriteResPair<X86FoldableSchedWrite SchedRW,

test/tools/llvm-mca/X86/register-file-statistics.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=broadwell -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=BDWELL
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=btver2 -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=BTVER2
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=haswell -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=HASWELL
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=sandybridge -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SANDYBRID
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=skylake -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SKLCLI
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=skylake-avx512 -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SKLSER
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver1 -iterations=1 -register-file-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=ZNVER1
+
+xor %eax, %eax
+
+# ALL:            Iterations:        1
+# ALL-NEXT:       Instructions:      1
+# ALL-NEXT:       Total Cycles:      4
+
+# BDWELL-NEXT:    Dispatch Width:    4
+# BTVER2-NEXT:    Dispatch Width:    2
+# HASWELL-NEXT:   Dispatch Width:    4
+# SANDYBRID-NEXT: Dispatch Width:    4
+# SKLCLI-NEXT:    Dispatch Width:    6
+# SKLSER-NEXT:    Dispatch Width:    6
+# ZNVER1-NEXT:    Dispatch Width:    4
+
+# ALL-NEXT:       IPC:               0.25
+
+# BDWELL-NEXT:    Block RThroughput: 0.3
+# BTVER2-NEXT:    Block RThroughput: 0.5
+# HASWELL-NEXT:   Block RThroughput: 0.3
+# SANDYBRID-NEXT: Block RThroughput: 0.3
+# SKLCLI-NEXT:    Block RThroughput: 0.3
+# SKLSER-NEXT:    Block RThroughput: 0.3
+# ZNVER1-NEXT:    Block RThroughput: 0.3
+
+# ALL:            Register File statistics:
+# ALL-NEXT:       Total number of mappings created:    2
+# ALL-NEXT:       Max number of mappings used:         2
+
+# BTVER2:         *  Register File #1 -- JFpuPRF:
+# BTVER2-NEXT:       Number of physical registers:     72
+# BTVER2-NEXT:       Total number of mappings created: 0
+# BTVER2-NEXT:       Max number of mappings used:      0
+
+# ZNVER1:         *  Register File #1 -- ZnFpuPRF:
+# ZNVER1-NEXT:       Number of physical registers:     160
+# ZNVER1-NEXT:       Total number of mappings created: 0
+# ZNVER1-NEXT:       Max number of mappings used:      0
+
+# BTVER2:         *  Register File #2 -- JIntegerPRF:
+# BTVER2-NEXT:       Number of physical registers:     64
+# BTVER2-NEXT:       Total number of mappings created: 2
+# BTVER2-NEXT:       Max number of mappings used:      2
+
+# ZNVER1:         *  Register File #2 -- ZnIntegerPRF:
+# ZNVER1-NEXT:       Number of physical registers:     168
+# ZNVER1-NEXT:       Total number of mappings created: 1
+# ZNVER1-NEXT:       Max number of mappings used:      1

test/tools/llvm-mca/X86/scheduler-queue-usage.s

+# NOTE: Assertions have been autogenerated by utils/update_mca_test_checks.py
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=broadwell -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=BDWELL
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=btver2 -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=BTVER2
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=haswell -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=HASWELL
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=sandybridge -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SANDYBRID
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=skylake -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SKLCLI
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=skylake-avx512 -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=SKLSER
+# RUN: llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=znver1 -iterations=1 -scheduler-stats -instruction-info=false -resource-pressure=false < %s | FileCheck %s -check-prefix=ALL -check-prefix=ZNVER1
+
+xor %eax, %eax
+
+# ALL:            Iterations:        1
+# ALL-NEXT:       Instructions:      1
+# ALL-NEXT:       Total Cycles:      4
+
+# BDWELL-NEXT:    Dispatch Width:    4
+# BTVER2-NEXT:    Dispatch Width:    2
+# HASWELL-NEXT:   Dispatch Width:    4
+# SANDYBRID-NEXT: Dispatch Width:    4
+# SKLCLI-NEXT:    Dispatch Width:    6
+# SKLSER-NEXT:    Dispatch Width:    6
+# ZNVER1-NEXT:    Dispatch Width:    4
+
+# ALL-NEXT:       IPC:               0.25
+
+# BDWELL-NEXT:    Block RThroughput: 0.3
+# BTVER2-NEXT:    Block RThroughput: 0.5
+# HASWELL-NEXT:   Block RThroughput: 0.3
+# SANDYBRID-NEXT: Block RThroughput: 0.3
+# SKLCLI-NEXT:    Block RThroughput: 0.3
+# SKLSER-NEXT:    Block RThroughput: 0.3
+# ZNVER1-NEXT:    Block RThroughput: 0.3
+
+# ALL:            Schedulers - number of cycles where we saw N instructions issued:
+# ALL-NEXT:       [# issued], [# cycles]
+# ALL-NEXT:        0,          3  (75.0%)
+# ALL-NEXT:        1,          1  (25.0%)
+
+# BDWELL:         Scheduler's queue usage:
+# BDWELL-NEXT:    BWPortAny,  1/60
+
+# HASWELL:        Scheduler's queue usage:
+# HASWELL-NEXT:   HWPortAny,  1/60
+
+# BTVER2:         Scheduler's queue usage:
+# BTVER2-NEXT:    JALU01,  1/20
+# BTVER2-NEXT:    JFPU01,  0/18
+# BTVER2-NEXT:    JLSAGU,  0/12
+
+# SANDYBRID:      Scheduler's queue usage:
+# SANDYBRID-NEXT: SBPortAny,  1/54
+
+# SKLCLI:         Scheduler's queue usage:
+# SKLCLI-NEXT:    SKLPortAny,  1/60
+
+# SKLSER:         Scheduler's queue usage:
+# SKLSER-NEXT:    SKXPortAny,  1/60
+
+# ZNVER1:         Scheduler's queue usage:
+# ZNVER1-NEXT:    ZnAGU,  0/28
+# ZNVER1-NEXT:    ZnALU,  1/56
+# ZNVER1-NEXT:    ZnFPU,  0/36