[llvm-exegesis] Add benchmark mode that uses LBR for more precise measurements.

[oontvoo]

Summary

Add benchmark mode that uses LBR for more precise measurements.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

86

Let's move this to the X86 target, and rename it to x86-lbr-sample-period, with a default of 0. If non-zero, we're going to use LBR. Else, we'll use the non-LBR runner.

[courbet]

And thanks for the contribution.

[oontvoo]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

115

Yes, it would've ... Changed to Counter*

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

87

Yes - it should be macro guarded. (I mindlessly deleted the guards while moving things around)
Added it back.

[oontvoo]

Addressed review comments:

• remove the new mode and merge it into latency
• various refactorings and comments-cleanup per courbet@'s requests

[oontvoo]

Fix typos and rebase

[oontvoo]

Fix clang-tidy warning

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

Can you expand on what "hiding blocks" means ? e.g. an external reference ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

96

Apart from the PerfEvent, this does not really have anything to do with LBR.
On the one hand, this is not really a full-fledged FunctionExecutor, as it just throws away the CounterName argument and always measures the latency. This would be an argument for moving this to the X86 backend.
On the other hand, this has a lot of target-independent code, so it's a good thing that other targets can use it. The main issue, and the only thing that's really specific to LBR here is the perf event. Letting the target create the perf event as well as the counter does fix this.

So what we can do:

• change Target::createCounter to take a counter name rather than a PerfEvent. The default implementation creates the PerfEvent, validates it and wraps it in a Counter. The X86 implementation tests if we're asking for the cycles counter in LBR mode, and creates the relevant PerfEvent and LBRCounter. It falls back to the default implementation else.
• We stop ignoring the counter name argument in LbrFunctionExecutorImpl::runAndMeasure, and just propagate it to the ExegesisTarget.
• We rename LbrFunctionExecutorImpl into WorkerThreadFunctionExecutorImpl (and FunctionExecutorImpl could be renamed to SameThreadFunctionExecutor, and everything becomes target agnostic.

204

Again, this is too specific to X86. I think it's one of these cases where is actually makes more sense for the generic code to know nothing about LBR. What about:

• Moving LbrSamplePeriod from llvm-exegesis to X86/Target.cpp
• Making createFunctionExecutor() a virtual member of the target. The default implementation returns a FunctionExecutorImpl.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

50 ↗

(On Diff #255837)

The LLVM style is FIXME

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

32

do we really need all these ?

600

[nit] s/PFM package available../HAVE_LIBPFM/ the package might be available but specifically disabled

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

17

Does this work under windows ? If not, please use one of the LLVM wrappers.

30

internal functions use a static keyword instead of anonymous namespace in the LLVM style guide.

35

s/time out/timeout

39

static

53

This would benefit from a pointer to documentation (or argued speculation about) whether rewriting code inside a loop for another thread is valid and how it behaves w.r.t. caches and DSB.

57

static

84

static

126

you can remove this now

198

FIXME. Also, please give more explanations so that somebody else can look at that too ;)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

2

This file, and the cpp one, are missing the LLVM headers.

[oontvoo]

• Make internal helpers static functions (instead of anonymous ns members)
• Reword confusing comments
• Add LLVM headers

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

typo: ..

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

113

std::unique_ptr + no dtor ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

50 ↗

(On Diff #255837)

FIXME and no name :)

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

199

The LBR on Haswell has only the addresses of the branches. To get the timing, Skylake is the minimal requirement.

207

I'd add a mention that this applies to the LBR ratency mode.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

192

As noted above - Haswell and Broadwell have the LBR, but it does not report the number of cycles between the branches.

[oontvoo]

Addressed review comments.

[oontvoo]

Add more comment

[oontvoo]

Thanks for the thorough reviews! :-)

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

96

Done - instead of moving the executor out, I've just let the Target decides whether to use WorkerThread executor or otherwise.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

17

Per offline discussion, we'll ifdef this out for non-linux and will add support for windows in follow-up CLs

53

I meant to rewrite this hack to use a condition var for stopping the loop from another thread, per Ondrej's suggestion .
I just haven't got to it yet :-) ... (would need more changes in the assembler.cpp )

So I'll add a FIXME for next CLs

198

Per offline discussions, this requires re-working the read() interface because we need to return more than one value.
(Specifically, the lbr-entries offer a list of cycles-latencies for a bunch of consec branches ... we don't want to just ignore the older branches and only look at the most recent)

But because this patch is already quite big , we'll keep this as TODO and follow up.

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

96

That forces passing the RunArg to the target, which has no reason to know about it.

What I'm suggesting is the following:

• move LbrSamplePeriod to X86/Target.cpp. This whole SamplePeriod thing, and the existence of the option is really just a X86 detail, the target-independent code has no need to know about any of this.
• Get rid of SamplePeriod, in the target-independant code. RunArg can go too.
• createCounter does not need Description, or SamplePeriod. The X86 target handles it internally.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

621

This should really onely be:

Expected<std::unique_ptr<pfm::Counter>>
createCounter(const char *CounterName) {
  if (CounterName == kCyclesCounter && LbrSamplingPeriod > 0) {
    return std::make_unique<X86LbrCounter>(PerfEvent("LBR", LbrSamplingPeriod));
  }
  return ExegesisTarget::createCounter(CounterName);
}

623

This will fail to compile under windows because X86LbrCounter does not exist there.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

348

This compile-time guarded runtime check can now go, because the x86 specific concepts are handled in the x86 target.

[oontvoo]

Addressed review comments:

• remove no-longer needed RunArg
• move target-specific to X86/Target

[oontvoo]

Add check for the LBR format to ensure it contains cycles.

[oontvoo]

Update perf_event_attr to use BR_INST_RETIRED.NEAR_TAKEN per eranian@'s recommendation.

[oontvoo]

ran clang-format

[oontvoo]

revert unrelated change

[oontvoo]

Fix dup options

[oontvoo]

Add new cpp file to makefile

[oontvoo]

Update perf-even-attr for raw-encoding

[oontvoo]

Updated patch to cover what we discussed in the sync last week and this week, specifically:

• Remove the worker-thread executor (not needed. It's fine to use the same thread)
• Make PerfEvent virtual so the X86Counter can create its own PerfEvent with LBR-specific attributes.

Remaining questions:

• Counter should be able to return more than one readings.
• We should validate that repetion-mode=loop for LBR

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

On X86,

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

12

you're no longer using <thread> and <mutex>.

14

is this needed ?

63

s/auto/pfm::Counter/

llvm/tools/llvm-exegesis/lib/PerfHelper.h

45

why is the dtor virtual ? I don't see any virtual methods ?

llvm/tools/llvm-exegesis/lib/Target.h

69 ↗

(On Diff #265315)

Can you submit this change independently and rebase ? Thanks !

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

595

&& LbrSamplingPeriod > 0 ?

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

65

fills

184

[style]

if (PollResult == -1)
  return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.", llvm::errc::io_error);
if (PollResult != 0) 
  continue;
...

185

how often does that happen ? Do we want that much logging ?

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

60

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/Target.cpp

36 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

[style] no braces for 1-line conditional bodies

131

[style] no braces for 1-line conditional bodies

144

[style] no braces for 1-line conditional bodies

206

[style] no braces for 1-line conditional bodies

216

ditto

223

ditto

[oontvoo]

Addressed review comments

[oontvoo]

llvm/tools/llvm-exegesis/lib/Target.h

69 ↗

(On Diff #265315)

Done https://reviews.llvm.org/D80610

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

185

It could happen 100% of the time if the code happens to have no branches.
With "proper" input, it happens less frequently. I don't really have a sense of how often ... it's probably be fine to change this to llvm::debug() or something?

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

184

you don't need the else after the return.

185

I'm not worried about the no-branch case as we can avoid that. We might as well remove it, because we don't really care if it fanally succeeded, and if it did not we can print an error there.

190

you don't need the else here after the continue.

[oontvoo]

Rebase to reduce diff

[oontvoo]

Simplified re-try

[courbet]

OK, I think this is in good shape now, but we still need a test for it. The only issue is that not all buildbots support LBR, so we need a mechanism for skipping the test when not available. You can create a subdir llvm-project/llvm/test/tools/llvm-exegesis/X86/lbr with a custom lit similar to llvm-project/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg to check lbr support before running the test.

[oontvoo]

Add test and option to check for expected CPU.

I can't seem to find a way to get the host CPU version (much less name or pmu capabilities) from the lit config, so I had to add the check to llvm-exegesis.

[oontvoo]

Rebase

[oontvoo]

This comment was removed by oontvoo.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291

This is a bit brittle, because we could imagine the name of some unrelated CPUs being substrings of others. What about having a repeated option --allowed-host-cpu=skylake --allowed-host-cpu=skylake-avx512 --allowed-host-cpu=whateverlake, and check that the exact value is one of these ?

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

This is more of a philosophical comment: IIRC, the LBR records the times when the branch instructions are retired. With correct branch prediction, the instructions in the measured block would start executing speculatively before the branch is retired, and so the number we get is probably closer to the throughput.

211

I'm wondering how this applies to this use case. The issue with basic blocks skipped might be when sampling all basic blocks in a larger application over a longer period of time, but this is not our case.
In our case, we're interested in a single basic block, and run it in a tight loop for relatively long, so that the LBR sampling catches this exact basic block.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

I think we also need to filter by the addresses of the branches (or store them) to make sure that we take the cycles only for the measured basic block, not for the other basic blocks surrounding it.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291

Ideally, this should be defined in terms of CPU features (e.g. CPUID bits). Even better - each target should know which counters it supports, based on its platform-specific feature discovery mechanism. I understand that this would be a huge change for this CL, but we should at least have a FIXME here.

[oontvoo]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291

Actually, I had a question here that I was gonna ask in an email But here goes.

We know that the LBR formats could be queried from the perf-cap MSR
Specifically, we want: MSR IA32_PERF_CAPABILITIES[5:0]" == 000110B (bit 59...63 is not relevant)

If I'm not mistaken, perf_event_mmap_page::capabilities should(?) give us that.
Except, when I run this on both Broadwell and Skylake, the capabilities field has value of 30 for both platforms. (It shouldn't be). Of course, the difference here is that the cycle entries are all zeroes on Broadwell .
I haven't looked in details, so I don't know if this is some implementation detail of the libpfm or if we're just mis-interpreting the SDM here.

Thoughts?

[ondrasej]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291

This is strange.

I understand the SDM section about IA32_PERF_CAPABILITIES the same way, i.e. we should see 000110B if there are timings, and this should happen only on Skylake or onwards.

That said, I don't think that perf_event_mmap_page::capabilities contains the IA32_PERF_CAPABILITIES MSR. Instead, it looks like a struct that is defined and filled by the kernel. Check out the definition/docs of the struct (and the code that fills it). They set individual bit fields (and even the bit field defined in the union does not match the other bits defined in the MSR).

After some digging through kernel sources, I see where the kernel keeps the data internally, but I didn't see a way to get it out of the kernel. We might need to do more research on that.

[oontvoo]

Filter and keep only records for the benchmarked codeblock

[oontvoo]

Add option to not filter the records too

[oontvoo]

Updated diff

[oontvoo]

clang-tidy

[courbet]

Only style comments left

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

78

let's use a c++ cast here.

80

Actually I'm thinking you can just pass a StringRef to readOrError

llvm/tools/llvm-exegesis/lib/PerfHelper.h

88

Please explain what the arguments are (and switch to StringRef as mentioned above ?)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

150

braces

205

style: no braces

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

9

This could use a file comment with a link to LBR documentation.

[oontvoo]

updated diff

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

179

Let's return an error when FunctionBytes.empty(). Right now this will assert later.

[oontvoo]

Return error for empty function

[ondrasej]

Looks good overall.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

48–49

[style] Remove the empty line.

[oontvoo]

Removed empty line

[oontvoo]

Fixed warning [-Wmissing-field-initializers] in code

[erichkeane]

This patch doesn't seem to build for me:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp: In function ‘llvm::Error llvm::exegesis::parseDataBuffer(const char*, size_t, const void*, const void*, llvm::SmallVector<long int, 4>*)’:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp:99:37: error: ‘struct perf_branch_entry’ has no member named ‘cycles’

CycleArray->push_back(Entry.cycles);

I'm on RHEL7, so I have kernel 3.10, so it doesn't have 'cycles'.

According ot this: https://elixir.bootlin.com/linux/v4.3/source/include/uapi/linux/perf_event.h#L963 kernel 4.3 is the first time that 'cycles' appeared in this structure.

[courbet]

If you're not using llvm-exegesis you can disable this with the LLVM_ENABLE_LIBPFM cmake option.
But I'll revert this until @oontvoo can fix llvm/cmake/modules/FindLibpfm.cmake

[oontvoo]

Check for existence of cycles before enabling LBR mode.

[gchatelet]

LGTM

[oontvoo]

Relanded.
https://reviews.llvm.org/rGee7caa75939afb75547c00744c5df4d04d45e517

llvm/docs/CommandGuide/llvm-exegesis.rst

 
 .. option:: -snippets-file=<filename>
 
  Specify the custom code snippet to measure. See example 2 for details.
  Either `opcode-index`, `opcode-name` or `snippets-file` must be set.
 
 .. option:: -mode=[latency|uops|inverse_throughput|analysis]
 
- Specify the run mode. Note that if you pick `analysis` mode, you also need
- to specify at least one of the `-analysis-clusters-output-file=` and
- `-analysis-inconsistencies-output-file=`.
+ Specify the run mode. Note that some modes have additional requirements and options.
+
+ `latency` mode can be  make use of either RDTSC or LBR.
+ `latency[LBR]` is only available on X86 (at least `Skylake`).
+  To run in this mode, a positive value  must be specified for `x86-lbr-sample-period` and `--repetition-mode=loop`
+
+ In `analysis` mode, you also need to specify at least one of the
+ `-analysis-clusters-output-file=` and `-analysis-inconsistencies-output-file=`.
+
+.. option:: -x86-lbr-sample-period=<nBranches/sample>
+
+  Specify the LBR sampling period - how many branches before we take a sample.
+  When a positive value is specified for this option and when the mode is `latency`,
+  we will use LBRs for measuring.
+  On choosing the "right" sampling period, a small value is preferred, but throttling
+  could occur if the sampling is too frequent. A prime number should be used to
+  avoid consistently skipping certain blocks.
   
 .. option:: -repetition-mode=[duplicate|loop|min]
 
  Specify the repetition mode. `duplicate` will create a large, straight line
  basic block with `num-repetitions` copies of the snippet. `loop` will wrap
  the snippet in a loop which will be run `num-repetitions` times. The `loop`
  mode tends to better hide the effects of the CPU frontend on architectures
  that cache decoded instructions, but consumes a register for counting
  iterations. If performing an analysis over many opcodes, it may be best

llvm/test/tools/llvm-exegesis/X86/lbr/Inputs/mov_add.att

+# LLVM-EXEGESIS-LIVEIN RDI
+# LLVM-EXEGESIS-DEFREG XMM1 42
+movq $2, %rdi
+addq $0x10, %rdi
 No newline at end of file

llvm/test/tools/llvm-exegesis/X86/lbr/lit.local.cfg

+import subprocess
+import lit.util
+
+if not ('X86' in config.root.targets):
+    # We need support for X86.
+    config.unsupported = True
+
+elif not ('x86_64' in config.root.host_triple):
+    # We need to be running on an X86 host.
+    config.unsupported = True
+    
+else:    
+    # We need libpfm to be installed and the host to be at least skylake.
+    llvm_exegesis_exe = lit.util.which('llvm-exegesis', config.llvm_tools_dir)
+    if not llvm_exegesis_exe:
+        print('llvm-exegesis not found')
+        config.unsupported = True
+    else:
+      try:
+          with open(os.devnull, 'w') as quiet:
+              check_llvm_exegesis_uops_result = subprocess.call(
+                [llvm_exegesis_exe, '-expected-host-cpu', 'lake', '-mode', 'uops', '-snippets-file', '/dev/null'], stdout=quiet, stderr=quiet)
+              check_llvm_exegesis_latency_result = subprocess.call(
+                [llvm_exegesis_exe, '-expected-host-cpu', 'lake', '-mode', 'latency', '-snippets-file', '/dev/null'], stdout=quiet, stderr=quiet)
+      except OSError:
+          print('could not exec llvm-exegesis')
+          config.unsupported = True
+      if not check_llvm_exegesis_uops_result == 0:
+        config.unsupported = True
+      if not check_llvm_exegesis_latency_result == 0:
+        config.unsupported = True

llvm/test/tools/llvm-exegesis/X86/lbr/mov-add.s

+# RUN: llvm-exegesis -mode=latency --repetition-mode=loop --x86-lbr-sample-period=521 --snippets-file=%p/Inputs/mov_add.att
+
+
+CHECK:      ---
+CHECK-NEXT: mode: latency
+CHECK-NEXT: key:
+CHECK-NEXT:   instructions:
+CHECK-NEXT:     'MOV64ri32 RDI i_0x2'
+CHECK-NEXT:     'ADD64ri8 RDI RDI i_0x10'
+CHECK-NEXT: config: ''
+CHECK-NEXT: {{.*}}
+CHECK-NEXT: {{.*}}
+CHECK-NEXT: {{.*}}
+CHECK-NEXT: {{.*}}
+CHECK-NEXT: num_repetitions: 10000
+CHECK-NEXT: measurements:
+CHECK-NEXT: {{.*}} value: 0.0001, per_snippet_value: 0.0002 {{.*}}
+CHECK-LAST: ...

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

 //===-- BenchmarkRunner.cpp -------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <array>
 #include <memory>
 #include <string>
 
 #include "Assembler.h"
 #include "BenchmarkRunner.h"
 #include "Error.h"
 #include "MCInstrDescView.h"
 #include "PerfHelper.h"
 #include "Target.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"

 public:
   FunctionExecutorImpl(const LLVMState &State,
                        object::OwningBinary<object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : State(State), Function(State.createTargetMachine(), std::move(Obj)),
         Scratch(Scratch) {}
 
 private:
   Expected<int64_t> runAndMeasure(const char *Counters) const override {
     // We sum counts when there are several counters for a single ProcRes
     // (e.g. P23 on SandyBridge).
     int64_t CounterValue = 0;
     SmallVector<StringRef, 2> CounterNames;
     StringRef(Counters).split(CounterNames, '+');
     char *const ScratchPtr = Scratch->ptr();
     for (auto &CounterName : CounterNames) {
       CounterName = CounterName.trim();
       auto CounterOrError =
           State.getExegesisTarget().createCounter(CounterName, State);
 
       if (!CounterOrError)
         return CounterOrError.takeError();
 
       pfm::Counter *Counter = CounterOrError.get().get();
       Scratch->clear();
       {
         CrashRecoveryContext CRC;
         CrashRecoveryContext::Enable();
         const bool Crashed = !CRC.RunSafely([this, Counter, ScratchPtr]() {
           Counter->start();
           this->Function(ScratchPtr);
           Counter->stop();
         });
         CrashRecoveryContext::Disable();
         // FIXME: Better diagnosis.
         if (Crashed)
           return make_error<SnippetCrash>("snippet crashed while running");
       }
-      CounterValue += Counter->read();
+
+      auto From = (void *)Function.getFunctionBytes().data();
+      auto To = (void *)(((char *)From) + Function.getFunctionBytes().size());
+      auto ValueOrError = Counter->readOrError(From, To);
+      if (!ValueOrError)
+        return ValueOrError.takeError();
+
+      CounterValue += ValueOrError.get();
+     );
     }
     return CounterValue;
   }
 
   const LLVMState &State;
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
 } // namespace
 
 Expected<InstructionBenchmark> BenchmarkRunner::runConfiguration(
     const BenchmarkCode &BC, unsigned NumRepetitions,
     ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
     bool DumpObjectToDisk) const {
   InstructionBenchmark InstrBenchmark;
   InstrBenchmark.Mode = Mode;
   InstrBenchmark.CpuName = std::string(State.getTargetMachine().getTargetCPU());
   InstrBenchmark.LLVMTriple =
       State.getTargetMachine().getTargetTriple().normalize();
   InstrBenchmark.NumRepetitions = NumRepetitions;
   InstrBenchmark.Info = BC.Info;
 
   const std::vector<MCInst> &Instructions = BC.Key.Instructions;
 
   InstrBenchmark.Key = BC.Key;
 
   // If we end up having an error, and we've previously succeeded with
   // some other Repetitor, we want to discard the previous measurements.
   struct ClearBenchmarkOnReturn {
     ClearBenchmarkOnReturn(InstructionBenchmark *IB) : IB(IB) {}
     ~ClearBenchmarkOnReturn() {
       if (Clear)
         IB->Measurements.clear();
     }
     void disarm() { Clear = false; }
 
   private:
if (Error E = NewMeasurements.takeError()) {
       InstrBenchmark.Error = toString(std::move(E));
       return InstrBenchmark;
     }
     assert(InstrBenchmark.NumRepetitions > 0 && "invalid NumRepetitions");
     for (BenchmarkMeasure &BM : *NewMeasurements) {
       // Scale the measurements by instruction.
       BM.PerInstructionValue /= InstrBenchmark.NumRepetitions;
       // Scale the measurements by snippet.
       BM.PerSnippetValue *= static_cast<double>(Instructions.size()) /
                             InstrBenchmark.NumRepetitions;
     }
     if (InstrBenchmark.Measurements.empty()) {
       InstrBenchmark.Measurements = std::move(*NewMeasurements);
       continue;
     }
 
     assert(Repetitors.size() > 1 && !InstrBenchmark.Measurements.empty() &&
            "We're in an 'min' repetition mode, and need to aggregate new "
            "result to the existing result.");
     assert(InstrBenchmark.Measurements.size() == NewMeasurements->size() &&
            "Expected to have identical number of measurements.");
     for (auto I : zip(InstrBenchmark.Measurements, *NewMeasurements)) {
       BenchmarkMeasure &Measurement = std::get<0>(I);
       BenchmarkMeasure &NewMeasurement = std::get<1>(I);
       assert(Measurement.Key == NewMeasurement.Key &&
              "Expected measurements to be symmetric");
 
       Measurement.PerInstructionValue = std::min(
           Measurement.PerInstructionValue, NewMeasurement.PerInstructionValue);

llvm/tools/llvm-exegesis/lib/PerfHelper.h

 class PerfEvent {
 public:
   // http://perfmon2.sourceforge.net/manv4/libpfm.html
   // Events are expressed as strings. e.g. "INSTRUCTION_RETIRED"
   explicit PerfEvent(StringRef PfmEventString);
 
   PerfEvent(const PerfEvent &) = delete;
   PerfEvent(PerfEvent &&other);
   ~PerfEvent();
 
   // The pfm_event_string passed at construction time.
   StringRef name() const;
 
   // Whether the event was successfully created.
   bool valid() const;
 
   // The encoded event to be passed to the Kernel.
   const perf_event_attr *attribute() const;
 
   // The fully qualified name for the event.
   // e.g. "snb_ep::INSTRUCTION_RETIRED:e=0:i=0:c=0:t=0:u=1:k=0:mg=0:mh=1"
   StringRef getPfmEventString() const;
 
-private:
-  const std::string EventString;
+protected:
+  PerfEvent() = default;
+  std::string EventString;
   std::string FullQualifiedEventString;
   perf_event_attr *Attr;
 };
 
 // Uses a valid PerfEvent to configure the Kernel so we can measure the
 // underlying event.
 class Counter {
 public:
public:
   virtual void start();
 
   /// Stops the measurement of the event.
   void stop();
 
   /// Returns the current value of the counter or -1 if it cannot be read.
   int64_t read() const;
 
   /// Returns the current value of the counter or error if it cannot be read.
-  virtual llvm::Expected<int64_t> readOrError() const;
+  /// If both non-null, the parameters specify the address of the benchmark
+  /// code.
+  virtual llvm::Expected<int64_t> readOrError(void *From = nullptr,
+                                              void *To = nullptr) const;
 
-private:
+protected:
   PerfEvent Event;
 #ifdef HAVE_LIBPFM
   int FileDescriptor = -1;
 #endif
 };
 
 } // namespace pfm
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

#ifdef HAVE_LIBPFM
   if (Fstr) {
     FullQualifiedEventString = Fstr;
     free(Fstr);
   }
 #endif
 }
 
 StringRef PerfEvent::name() const { return EventString; }
 
 bool PerfEvent::valid() const { return !FullQualifiedEventString.empty(); }
 
 const perf_event_attr *PerfEvent::attribute() const { return Attr; }
 
 StringRef PerfEvent::getPfmEventString() const {
   return FullQualifiedEventString;
 }
 
int64_t Counter::read() const {
   auto ValueOrError = readOrError();
   if (ValueOrError)
     return ValueOrError.get();
 
   errs() << ValueOrError.takeError() << "\n";
   return -1;
 }
 
-llvm::Expected<int64_t> Counter::readOrError() const {
+llvm::Expected<int64_t> Counter::readOrError(void *From, void *To) const {
   int64_t Count = 0;
   ssize_t ReadSize = ::read(FileDescriptor, &Count, sizeof(Count));
   if (ReadSize != sizeof(Count))
     return llvm::make_error<llvm::StringError>("Failed to read event counter",
                                                llvm::errc::io_error);
 
   return Count;
 }
 
 #else
 
 Counter::Counter(PerfEvent &&Event) : Event(std::move(Event)) {}
 
 Counter::~Counter() = default;
 
 void Counter::start() {}
 
 void Counter::stop() {}
 
 int64_t Counter::read() const { return 42; }
 
-llvm::Expected<int64_t> Counter::readOrError() const {
+llvm::Expected<int64_t> Counter::readOrError(void *From, void *To) const {
   return llvm::make_error<llvm::StringError>("Not implemented",
                                              llvm::errc::io_error);
 }
 
 #endif
 
 } // namespace pfm
 } // namespace exegesis
 } // namespace llvm

llvm/tools/llvm-exegesis/lib/X86/CMakeLists.txt

 include_directories(
   ${LLVM_MAIN_SRC_DIR}/lib/Target/X86
   ${LLVM_BINARY_DIR}/lib/Target/X86
   )
 
 add_library(LLVMExegesisX86
   STATIC
   Target.cpp
+  X86Counter.cpp
   )
 
 llvm_update_compile_flags(LLVMExegesisX86)
 llvm_map_components_to_libnames(libs
   X86
   Exegesis
   )
 
 target_link_libraries(LLVMExegesisX86 ${libs})
 set_target_properties(LLVMExegesisX86 PROPERTIES FOLDER "Libraries")

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

 //===-- Target.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "../Target.h"
 
 #include "../Error.h"
 #include "../ParallelSnippetGenerator.h"
 #include "../SerialSnippetGenerator.h"
 #include "../SnippetGenerator.h"
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "X86.h"
+#include "X86Counter.h"
 #include "X86RegisterInfo.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/Sequence.h"
 #include "llvm/MC/MCInstBuilder.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FormatVariadic.h"
 
+#include <memory>
+#include <string>
+#include <vector>
+
 namespace llvm {
 namespace exegesis {
 
+static cl::OptionCategory
+    BenchmarkOptions("llvm-exegesis benchmark x86-options");
+
+// If a positive value is specified, we are going to use the LBR in
+// latency-mode.
+//
+// Note:
+//  -  A small value is preferred, but too low a value could result in
+//     throttling.
+//  -  A prime number is preferred to avoid always skipping certain blocks.
+//
+static cl::opt<unsigned> LbrSamplingPeriod(
+    "x86-lbr-sample-period",
+    cl::desc("The sample period (nbranches/sample), used for LBR sampling"),
+    cl::cat(BenchmarkOptions), cl::init(0));
+
+// FIXME: Validates that repetition-mode is loop if LBR is requested.
+
 // Returns a non-null reason if we cannot handle the memory references in this
 // instruction.
 static const char *isInvalidMemoryInstr(const Instruction &Instr) {
   switch (Instr.Description.TSFlags & X86II::FormMask) {
   default:
     llvm_unreachable("Unknown FormMask value");
   // These have no memory access.
   case X86II::Pseudo:
if (Bytes - ByteOffset >= 1)
     add(fillStackSpace(
         X86::MOV8mi, ByteOffset,
         WideConstant.extractBits(8, ByteOffset * 8).getZExtValue()));
 }
 
 #include "X86GenExegesis.inc"
 
 namespace {
+
 class ExegesisX86Target : public ExegesisTarget {
 public:
   ExegesisX86Target() : ExegesisTarget(X86CpuPfmCounters) {}
 
+  Expected<std::unique_ptr<pfm::Counter>>
+  createCounter(StringRef CounterName, const LLVMState &State) const override {
+    // If LbrSamplingPeriod was provided, then ignore the
+    // CounterName because we only have one for LBR.
+    if (LbrSamplingPeriod > 0) {
+      // Can't use LBR without HAVE_LIBPFM, or __linux__ (for now)
+#if defined(HAVE_LIBPFM) && defined(__linux__)
+      return std::make_unique<X86LbrCounter>(
+          X86LbrPerfEvent(LbrSamplingPeriod));
+#else
+      return llvm::make_error<llvm::StringError>(
+          "LBR counter requested without HAVE_LIBPFM or running on Linux.",
+          llvm::errc::invalid_argument);
+#endif
+    }
+    return ExegesisTarget::createCounter(CounterName, State);
+  }
+
 private:
   void addTargetSpecificPasses(PassManagerBase &PM) const override;
 
   unsigned getScratchMemoryRegister(const Triple &TT) const override;
 
   unsigned getLoopCounterRegister(const Triple &) const override;
 
   unsigned getMaxMemoryAccessSize() const override { return 64; }
 
   Error randomizeTargetMCOperand(const Instruction &Instr, const Variable &Var,
                                  MCOperand &AssignedValue,
                                  const BitVector &ForbiddenRegs) const override;
 
   void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                           unsigned Offset) const override;
 
   void decrementLoopCounterAndJump(MachineBasicBlock &MBB,
                                    MachineBasicBlock &TargetMBB,
                                    const MCInstrInfo &MII) const override;
 
   std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                const APInt &Value) const override;

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

+//===-- X86Counter.h --------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H
+#define LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H
+
+#include "../PerfHelper.h"
+#include "llvm/Support/Error.h"
+
+// FIXME: Use appropriate wrappers for poll.h and mman.h
+// to support Windows and remove this linux-only guard.
+#if defined(__linux__) && defined(HAVE_LIBPFM)
+
+namespace llvm {
+namespace exegesis {
+
+class X86LbrPerfEvent : public pfm::PerfEvent {
+public:
+  X86LbrPerfEvent(unsigned SamplingPeriod);
+};
+
+class X86LbrCounter : public pfm::Counter {
+public:
+  explicit X86LbrCounter(pfm::PerfEvent &&Event);
+
+  virtual ~X86LbrCounter();
+
+  void start() override;
+
+  llvm::Expected<int64_t> readOrError(void *From = nullptr,
+                                      void *To = nullptr) const override;
+
+private:
+  void *MMappedBuffer = nullptr;
+};
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif // defined(__linux__) && defined(HAVE_LIBPFM)
+
+#endif // LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

+//===-- X86Counter.cpp ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86Counter.h"
+
+// FIXME: Use appropriate wrappers for poll.h and mman.h
+// to support Windows and remove this linux-only guard.
+#ifdef __linux__
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Errc.h"
+
+#ifdef HAVE_LIBPFM
+#include "perfmon/perf_event.h"
+#include "perfmon/pfmlib.h"
+#include "perfmon/pfmlib_perf_event.h"
+#endif // HAVE_LIBPFM
+
+#include <atomic>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <vector>
+
+#ifdef HAVE_LIBPFM
+namespace llvm {
+namespace exegesis {
+
+static constexpr size_t kBufferPages = 8;
+static const size_t kDataBufferSize = kBufferPages * getpagesize();
+
+// Waits for the LBR perf events.
+static int pollLbrPerfEvent(const int FileDescriptor) {
+  struct pollfd PollFd;
+  PollFd.fd = FileDescriptor;
+  PollFd.events = POLLIN;
+  PollFd.revents = 0;
+  return poll(&PollFd, 1 /* num of fds */, 10000 /* timeout in ms */);
+}
+
+// Copies the data-buffer into Buf, given the pointer to MMapped.
+static void copyDataBuffer(void *MMappedBuffer, char *Buf, uint64_t Tail,
+                           size_t DataSize) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page.
+  char *Start = reinterpret_cast<char *>(MMappedBuffer) + getpagesize();
+  // The LBR buffer is a cyclic buffer, we copy data to another buffer.
+  uint64_t Offset = Tail % kDataBufferSize;
+  size_t CopySize = kDataBufferSize - Offset;
+  memcpy(Buf, Start + Offset, CopySize);
+  if (CopySize >= DataSize)
+    return;
+
+  memcpy(Buf + CopySize, Start, Offset);
+  return;
+}
+
+// Parses the given data-buffer for stats and fill the CycleArray.
+// If data has been extracted successfully, also modifies the code to jump
+// out the benchmark loop.
+static llvm::Error parseDataBuffer(const char *DataBuf, size_t DataSize,
+                                   void *From, void *To,
+                                   std::vector<int64_t> *CycleArray) {
+  assert(From != nullptr && To != nullptr);
+  const char *DataPtr = DataBuf;
+  while (DataPtr < DataBuf + DataSize) {
+    struct perf_event_header Header;
+    memcpy(&Header, DataPtr, sizeof(struct perf_event_header));
+    if (Header.type != PERF_RECORD_SAMPLE) {
+      // Ignores non-sample records.
+      DataPtr += Header.size;
+      continue;
+    }
+    DataPtr += sizeof(Header);
+    uint64_t Count = llvm::support::endian::read64(DataPtr, support::native);
+    DataPtr += sizeof(Count);
+
+    struct perf_branch_entry Entry;
+    memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    // Read the perf_branch_entry array.
+    for (uint64_t i = 0; i < Count; ++i) {
+      if (reinterpret_cast<uint64_t>(From) >= Entry.from &&
+          reinterpret_cast<uint64_t>(To) <= Entry.to)
+        CycleArray->push_back(Entry.cycles);
+      if (i == Count - 1)
+        // We've reached the last entry.
+        return llvm::Error::success();
+
+      // Advance to next entry
+      DataPtr += sizeof(Entry);
+      memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    }
+  }
+  return llvm::make_error<llvm::StringError>("Unable to parse databuffer.",
+                                             llvm::errc::io_error);
+}
+
+#ifdef HAVE_LIBPFM
+X86LbrPerfEvent::X86LbrPerfEvent(unsigned SamplingPeriod) {
+  assert(SamplingPeriod > 0 && "SamplingPeriod must be positive");
+  EventString = "BR_INST_RETIRED.NEAR_TAKEN";
+  Attr = new perf_event_attr();
+  *Attr = {0};
+  Attr->size = sizeof(*Attr);
+  Attr->type = PERF_TYPE_RAW;
+  // FIXME This is SKL's encoding. Not sure if it'll change.
+  Attr->config = 0x20c4; // BR_INST_RETIRED.NEAR_TAKEN
+  Attr->sample_type = PERF_SAMPLE_BRANCH_STACK;
+  // Don't need to specify "USER" because we've already excluded HV and Kernel.
+  Attr->branch_sample_type = PERF_SAMPLE_BRANCH_ANY;
+  Attr->sample_period = SamplingPeriod;
+  Attr->wakeup_events = 1; // We need this even when using ioctl REFRESH.
+  Attr->disabled = 1;
+  Attr->exclude_kernel = 1;
+  Attr->exclude_hv = 1;
+  Attr->read_format = PERF_FORMAT_GROUP;
+
+  FullQualifiedEventString = EventString;
+}
+#else
+X86LbrPerfEvent::X86LbrPerfEvent(unsigned SamplingPeriod) {
+  EventString = "";
+  Attr = nullptr;
+}
+#endif
+
+X86LbrCounter::X86LbrCounter(pfm::PerfEvent &&NewEvent)
+    : Counter(std::move(NewEvent)) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page, so we allocate one more page.
+  MMappedBuffer = mmap(nullptr, (kBufferPages + 1) * getpagesize(),
+                       PROT_READ | PROT_WRITE, MAP_SHARED, FileDescriptor, 0);
+  if (MMappedBuffer == MAP_FAILED) {
+    llvm::errs() << "Failed to mmap buffer.";
+  }
+}
+
+X86LbrCounter::~X86LbrCounter() { close(FileDescriptor); }
+
+void X86LbrCounter::start() {
+  ioctl(FileDescriptor, PERF_EVENT_IOC_REFRESH, 1024 /* kMaxPollsPerFd */);
+}
+
+llvm::Expected<int64_t> X86LbrCounter::readOrError(void *From, void *To) const {
+  // The max number of time-outs/retries before we give up.
+  static constexpr int kMaxTimeouts = 160;
+
+  // Disable the event before reading
+  ioctl(FileDescriptor, PERF_EVENT_IOC_DISABLE, 0);
+
+  // Parses the LBR buffer and fills CycleArray with the sequence of cycle
+  // counts from the buffer.
+  std::vector<int64_t> CycleArray;
+  std::unique_ptr<char[]> DataBuf(new char[kDataBufferSize]);
+  int NumTimeouts = 0;
+  int PollResult = 0;
+  while (PollResult <= 0) {
+    PollResult = pollLbrPerfEvent(FileDescriptor);
+    if (PollResult > 0)
+      break;
+    if (PollResult == -1)
+      return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.",
+                                                 llvm::errc::io_error);
+    if (NumTimeouts++ >= kMaxTimeouts)
+      return llvm::make_error<llvm::StringError>(
+          "LBR polling still timed out after max number of attempts.",
+          llvm::errc::device_or_resource_busy);
+  }
+
+  struct perf_event_mmap_page Page;
+  memcpy(&Page, MMappedBuffer, sizeof(struct perf_event_mmap_page));
+
+  const uint64_t DataTail = Page.data_tail;
+  const uint64_t DataHead = Page.data_head;
+  // We're supposed to use a barrier after reading data_head.
+  std::atomic_thread_fence(std::memory_order_acq_rel);
+  const size_t DataSize = DataHead - DataTail;
+  if (DataSize > kDataBufferSize)
+    return llvm::make_error<llvm::StringError>(
+        "DataSize larger than buffer size.", llvm::errc::invalid_argument);
+
+  copyDataBuffer(MMappedBuffer, DataBuf.get(), DataTail, DataSize);
+
+  llvm::Error error =
+      parseDataBuffer(DataBuf.get(), DataSize, From, To, &CycleArray);
+  if (!error)
+    // FIXME Rework the Counter interface to allow returning more than one reads
+    // because We should report the cycles count for all jumps, not just the
+    // most recent.
+    return CycleArray[0];
+  return error;
+}
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif // HAVE_LIBPFM
+#endif // __linux__

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

cl::values(clEnumValN(exegesis::InstructionBenchmark::Latency, "latency",
                           "Instruction Inverse Throughput"),
                clEnumValN(exegesis::InstructionBenchmark::Uops, "uops",
                           "Uop Decomposition"),
                // When not asking for a specific benchmark mode,
                // we'll analyse the results.
                clEnumValN(exegesis::InstructionBenchmark::Unknown, "analysis",
                           "Analysis")));
 
 static cl::opt<exegesis::InstructionBenchmark::RepetitionModeE> RepetitionMode(
     "repetition-mode", cl::desc("how to repeat the instruction snippet"),
     cl::cat(BenchmarkOptions),
     cl::values(
         clEnumValN(exegesis::InstructionBenchmark::Duplicate, "duplicate",
                    "Duplicate the snippet"),
         clEnumValN(exegesis::InstructionBenchmark::Loop, "loop",
                    "Loop over the snippet"),
         clEnumValN(exegesis::InstructionBenchmark::AggregateMin, "min",
                    "All of the above and take the minimum of measurements")),
     cl::init(exegesis::InstructionBenchmark::Duplicate));
 

 static cl::opt<std::string>
     AnalysisClustersOutputFile("analysis-clusters-output-file", cl::desc(""),
                                cl::cat(AnalysisOptions), cl::init(""));
 static cl::opt<std::string>
     AnalysisInconsistenciesOutputFile("analysis-inconsistencies-output-file",
                                       cl::desc(""), cl::cat(AnalysisOptions),
                                       cl::init(""));
 
+static cl::opt<std::string>
+    ExpectedHostCpu("expected-host-cpu",
+                    cl::desc("if non-empty, only run the benchmark if the host "
+                             "CPU matches the given name."),
+                    cl::cat(Options), cl::init(""));
+
 static cl::opt<bool> AnalysisDisplayUnstableOpcodes(
     "analysis-display-unstable-clusters",
     cl::desc("if there is more than one benchmark for an opcode, said "
              "benchmarks may end up not being clustered into the same cluster "
              "if the measured performance characteristics are different. by "
              "default all such opcodes are filtered out. this flag will "
              "instead show only such unstable opcodes"),
     cl::cat(AnalysisOptions), cl::init(false));
#endif
 
   InitializeNativeTarget();
   InitializeNativeTargetAsmPrinter();
   InitializeNativeTargetAsmParser();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
 
+  if (!ExpectedHostCpu.empty()) {
+    // The actual name could include variations, such as "skylake" vs
+    // "skylake-avx512" so we don't look for exact match.
+    std::string actual(State.getTargetMachine().getTargetCPU().data());
+    if (actual.find(ExpectedHostCpu) == std::string::npos)
+      ExitWithError("Unexpected host CPU " + actual);
+  }
   const std::unique_ptr<BenchmarkRunner> Runner = ExitOnErr(
       State.getExegesisTarget().createBenchmarkRunner(BenchmarkMode, State));
   if (!Runner) {
     ExitWithError("cannot create benchmark runner");
   }
 
   const auto Opcodes = getOpcodesOrDie(State.getInstrInfo());
 
for (const unsigned Opcode : Opcodes) {
       }
       std::move(ConfigsForInstr->begin(), ConfigsForInstr->end(),
                 std::back_inserter(Configurations));
     }
   } else {
     Configurations = ExitOnErr(readSnippets(State, SnippetsFile));
   }
 
   if (NumRepetitions == 0) {
     ExitOnErr.setBanner("llvm-exegesis: ");
     ExitWithError("--num-repetitions must be greater than zero");
   }
 
   // Write to standard output if file is not set.
   if (BenchmarkFile.empty())
     BenchmarkFile = "-";