[llvm-exegesis] Add benchmark mode that uses LBR for more precise measurements.

[oontvoo]

Summary

Add benchmark mode that uses LBR for more precise measurements.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

88

Let's move this to the X86 target, and rename it to x86-lbr-sample-period, with a default of 0. If non-zero, we're going to use LBR. Else, we'll use the non-LBR runner.

[courbet]

And thanks for the contribution.

[oontvoo]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

106

Yes, it would've ... Changed to Counter*

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

91

Yes - it should be macro guarded. (I mindlessly deleted the guards while moving things around)
Added it back.

[oontvoo]

Addressed review comments:

• remove the new mode and merge it into latency
• various refactorings and comments-cleanup per courbet@'s requests

[oontvoo]

Fix typos and rebase

[oontvoo]

Fix clang-tidy warning

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

220

Can you expand on what "hiding blocks" means ? e.g. an external reference ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

87

Apart from the PerfEvent, this does not really have anything to do with LBR.
On the one hand, this is not really a full-fledged FunctionExecutor, as it just throws away the CounterName argument and always measures the latency. This would be an argument for moving this to the X86 backend.
On the other hand, this has a lot of target-independent code, so it's a good thing that other targets can use it. The main issue, and the only thing that's really specific to LBR here is the perf event. Letting the target create the perf event as well as the counter does fix this.

So what we can do:

• change Target::createCounter to take a counter name rather than a PerfEvent. The default implementation creates the PerfEvent, validates it and wraps it in a Counter. The X86 implementation tests if we're asking for the cycles counter in LBR mode, and creates the relevant PerfEvent and LBRCounter. It falls back to the default implementation else.
• We stop ignoring the counter name argument in LbrFunctionExecutorImpl::runAndMeasure, and just propagate it to the ExegesisTarget.
• We rename LbrFunctionExecutorImpl into WorkerThreadFunctionExecutorImpl (and FunctionExecutorImpl could be renamed to SameThreadFunctionExecutor, and everything becomes target agnostic.

195

Again, this is too specific to X86. I think it's one of these cases where is actually makes more sense for the generic code to know nothing about LBR. What about:

• Moving LbrSamplePeriod from llvm-exegesis to X86/Target.cpp
• Making createFunctionExecutor() a virtual member of the target. The default implementation returns a FunctionExecutorImpl.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

49

The LLVM style is FIXME

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

45

do we really need all these ?

760

[nit] s/PFM package available../HAVE_LIBPFM/ the package might be available but specifically disabled

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

16 ↗

(On Diff #255837)

Does this work under windows ? If not, please use one of the LLVM wrappers.

29 ↗

(On Diff #255837)

internal functions use a static keyword instead of anonymous namespace in the LLVM style guide.

34 ↗

(On Diff #255837)

s/time out/timeout

38 ↗

(On Diff #255837)

static

52 ↗

(On Diff #255837)

This would benefit from a pointer to documentation (or argued speculation about) whether rewriting code inside a loop for another thread is valid and how it behaves w.r.t. caches and DSB.

56 ↗

(On Diff #255837)

static

83 ↗

(On Diff #255837)

static

125 ↗

(On Diff #255837)

you can remove this now

197 ↗

(On Diff #255837)

FIXME. Also, please give more explanations so that somebody else can look at that too ;)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

1 ↗

(On Diff #255837)

This file, and the cpp one, are missing the LLVM headers.

[oontvoo]

• Make internal helpers static functions (instead of anonymous ns members)
• Reword confusing comments
• Add LLVM headers

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

220

typo: ..

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

104

std::unique_ptr + no dtor ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

49

FIXME and no name :)

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

209

The LBR on Haswell has only the addresses of the branches. To get the timing, Skylake is the minimal requirement.

217

I'd add a mention that this applies to the LBR ratency mode.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

183

As noted above - Haswell and Broadwell have the LBR, but it does not report the number of cycles between the branches.

[oontvoo]

Addressed review comments.

[oontvoo]

Add more comment

[oontvoo]

Thanks for the thorough reviews! :-)

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

87

Done - instead of moving the executor out, I've just let the Target decides whether to use WorkerThread executor or otherwise.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

16 ↗

(On Diff #255837)

Per offline discussion, we'll ifdef this out for non-linux and will add support for windows in follow-up CLs

52 ↗

(On Diff #255837)

I meant to rewrite this hack to use a condition var for stopping the loop from another thread, per Ondrej's suggestion .
I just haven't got to it yet :-) ... (would need more changes in the assembler.cpp )

So I'll add a FIXME for next CLs

197 ↗

(On Diff #255837)

Per offline discussions, this requires re-working the read() interface because we need to return more than one value.
(Specifically, the lbr-entries offer a list of cycles-latencies for a bunch of consec branches ... we don't want to just ignore the older branches and only look at the most recent)

But because this patch is already quite big , we'll keep this as TODO and follow up.

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

87

That forces passing the RunArg to the target, which has no reason to know about it.

What I'm suggesting is the following:

• move LbrSamplePeriod to X86/Target.cpp. This whole SamplePeriod thing, and the existence of the option is really just a X86 detail, the target-independent code has no need to know about any of this.
• Get rid of SamplePeriod, in the target-independant code. RunArg can go too.
• createCounter does not need Description, or SamplePeriod. The X86 target handles it internally.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

781

This should really onely be:

Expected<std::unique_ptr<pfm::Counter>>
createCounter(const char *CounterName) {
  if (CounterName == kCyclesCounter && LbrSamplingPeriod > 0) {
    return std::make_unique<X86LbrCounter>(PerfEvent("LBR", LbrSamplingPeriod));
  }
  return ExegesisTarget::createCounter(CounterName);
}

783

This will fail to compile under windows because X86LbrCounter does not exist there.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

342

This compile-time guarded runtime check can now go, because the x86 specific concepts are handled in the x86 target.

[oontvoo]

Addressed review comments:

• remove no-longer needed RunArg
• move target-specific to X86/Target

[oontvoo]

Add check for the LBR format to ensure it contains cycles.

[oontvoo]

Update perf_event_attr to use BR_INST_RETIRED.NEAR_TAKEN per eranian@'s recommendation.

[oontvoo]

ran clang-format

[oontvoo]

revert unrelated change

[oontvoo]

Fix dup options

[oontvoo]

Add new cpp file to makefile

[oontvoo]

Update perf-even-attr for raw-encoding

[oontvoo]

Updated patch to cover what we discussed in the sync last week and this week, specifically:

• Remove the worker-thread executor (not needed. It's fine to use the same thread)
• Make PerfEvent virtual so the X86Counter can create its own PerfEvent with LBR-specific attributes.

Remaining questions:

• Counter should be able to return more than one readings.
• We should validate that repetion-mode=loop for LBR

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

207

On X86,

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

13

you're no longer using <thread> and <mutex>.

16

is this needed ?

69

s/auto/pfm::Counter/

llvm/tools/llvm-exegesis/lib/PerfHelper.h

45

why is the dtor virtual ? I don't see any virtual methods ?

llvm/tools/llvm-exegesis/lib/Target.h

69

Can you submit this change independently and rebase ? Thanks !

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

755

&& LbrSamplingPeriod > 0 ?

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

64 ↗

(On Diff #265315)

fills

183 ↗

(On Diff #265315)

[style]

if (PollResult == -1)
  return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.", llvm::errc::io_error);
if (PollResult != 0) 
  continue;
...

184 ↗

(On Diff #265315)

how often does that happen ? Do we want that much logging ?

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

66

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/Target.cpp

34

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

87 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

130 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

143 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

205 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

215 ↗

(On Diff #265315)

ditto

222 ↗

(On Diff #265315)

ditto

[oontvoo]

Addressed review comments

[oontvoo]

llvm/tools/llvm-exegesis/lib/Target.h

69

Done https://reviews.llvm.org/D80610

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

184 ↗

(On Diff #265315)

It could happen 100% of the time if the code happens to have no branches.
With "proper" input, it happens less frequently. I don't really have a sense of how often ... it's probably be fine to change this to llvm::debug() or something?

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

184 ↗

(On Diff #265315)

I'm not worried about the no-branch case as we can avoid that. We might as well remove it, because we don't really care if it fanally succeeded, and if it did not we can print an error there.

183 ↗

(On Diff #266422)

you don't need the else after the return.

189 ↗

(On Diff #266422)

you don't need the else here after the continue.

[oontvoo]

Rebase to reduce diff

[oontvoo]

Simplified re-try

[courbet]

OK, I think this is in good shape now, but we still need a test for it. The only issue is that not all buildbots support LBR, so we need a mechanism for skipping the test when not available. You can create a subdir llvm-project/llvm/test/tools/llvm-exegesis/X86/lbr with a custom lit similar to llvm-project/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg to check lbr support before running the test.

[oontvoo]

Add test and option to check for expected CPU.

I can't seem to find a way to get the host CPU version (much less name or pmu capabilities) from the lit config, so I had to add the check to llvm-exegesis.

[oontvoo]

Rebase

[oontvoo]

This comment was removed by oontvoo.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

292

This is a bit brittle, because we could imagine the name of some unrelated CPUs being substrings of others. What about having a repeated option --allowed-host-cpu=skylake --allowed-host-cpu=skylake-avx512 --allowed-host-cpu=whateverlake, and check that the exact value is one of these ?

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

207

This is more of a philosophical comment: IIRC, the LBR records the times when the branch instructions are retired. With correct branch prediction, the instructions in the measured block would start executing speculatively before the branch is retired, and so the number we get is probably closer to the throughput.

221

I'm wondering how this applies to this use case. The issue with basic blocks skipped might be when sampling all basic blocks in a larger application over a longer period of time, but this is not our case.
In our case, we're interested in a single basic block, and run it in a tight loop for relatively long, so that the LBR sampling catches this exact basic block.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

87 ↗

(On Diff #268037)

I think we also need to filter by the addresses of the branches (or store them) to make sure that we take the cycles only for the measured basic block, not for the other basic blocks surrounding it.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

292

Ideally, this should be defined in terms of CPU features (e.g. CPUID bits). Even better - each target should know which counters it supports, based on its platform-specific feature discovery mechanism. I understand that this would be a huge change for this CL, but we should at least have a FIXME here.

[oontvoo]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

292

Actually, I had a question here that I was gonna ask in an email But here goes.

We know that the LBR formats could be queried from the perf-cap MSR
Specifically, we want: MSR IA32_PERF_CAPABILITIES[5:0]" == 000110B (bit 59...63 is not relevant)

If I'm not mistaken, perf_event_mmap_page::capabilities should(?) give us that.
Except, when I run this on both Broadwell and Skylake, the capabilities field has value of 30 for both platforms. (It shouldn't be). Of course, the difference here is that the cycle entries are all zeroes on Broadwell .
I haven't looked in details, so I don't know if this is some implementation detail of the libpfm or if we're just mis-interpreting the SDM here.

Thoughts?

[ondrasej]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

292

This is strange.

I understand the SDM section about IA32_PERF_CAPABILITIES the same way, i.e. we should see 000110B if there are timings, and this should happen only on Skylake or onwards.

That said, I don't think that perf_event_mmap_page::capabilities contains the IA32_PERF_CAPABILITIES MSR. Instead, it looks like a struct that is defined and filled by the kernel. Check out the definition/docs of the struct (and the code that fills it). They set individual bit fields (and even the bit field defined in the union does not match the other bits defined in the MSR).

After some digging through kernel sources, I see where the kernel keeps the data internally, but I didn't see a way to get it out of the kernel. We might need to do more research on that.

[oontvoo]

Filter and keep only records for the benchmarked codeblock

[oontvoo]

Add option to not filter the records too

[oontvoo]

Updated diff

[oontvoo]

clang-tidy

[courbet]

Only style comments left

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

78

let's use a c++ cast here.

80

Actually I'm thinking you can just pass a StringRef to readOrError

llvm/tools/llvm-exegesis/lib/PerfHelper.h

84–85

Please explain what the arguments are (and switch to StringRef as mentioned above ?)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

149 ↗

(On Diff #273878)

braces

204 ↗

(On Diff #273878)

style: no braces

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

8 ↗

(On Diff #273878)

This could use a file comment with a link to LBR documentation.

[oontvoo]

updated diff

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

178 ↗

(On Diff #276122)

Let's return an error when FunctionBytes.empty(). Right now this will assert later.

[oontvoo]

Return error for empty function

[ondrasej]

Looks good overall.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

50–51

[style] Remove the empty line.

[oontvoo]

Removed empty line

[oontvoo]

Fixed warning [-Wmissing-field-initializers] in code

[erichkeane]

This patch doesn't seem to build for me:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp: In function ‘llvm::Error llvm::exegesis::parseDataBuffer(const char*, size_t, const void*, const void*, llvm::SmallVector<long int, 4>*)’:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp:99:37: error: ‘struct perf_branch_entry’ has no member named ‘cycles’

CycleArray->push_back(Entry.cycles);

I'm on RHEL7, so I have kernel 3.10, so it doesn't have 'cycles'.

According ot this: https://elixir.bootlin.com/linux/v4.3/source/include/uapi/linux/perf_event.h#L963 kernel 4.3 is the first time that 'cycles' appeared in this structure.

[courbet]

If you're not using llvm-exegesis you can disable this with the LLVM_ENABLE_LIBPFM cmake option.
But I'll revert this until @oontvoo can fix llvm/cmake/modules/FindLibpfm.cmake

[oontvoo]

Check for existence of cycles before enabling LBR mode.

[gchatelet]

LGTM

[oontvoo]

Relanded.
https://reviews.llvm.org/rGee7caa75939afb75547c00744c5df4d04d45e517

llvm/docs/CommandGuide/llvm-exegesis.rst

  a comma-separated list. See example 1 for details.
  Either `opcode-index`, `opcode-name` or `snippets-file` must be set.
 
 .. option:: -snippets-file=<filename>
 
  Specify the custom code snippet to measure. See example 2 for details.
  Either `opcode-index`, `opcode-name` or `snippets-file` must be set.
 
-.. option:: -mode=[latency|uops|inverse_throughput|analysis]
+.. option:: -mode=[latency|lbr_latency|uops|inverse_throughput|analysis]
+
+ Specify the run mode. Note that some modes have additional requirements.
+
+ `lbr_latency` mode makes use of LBR, which, starting with Skylake, contains the
+ precise number of cycles between the two consecutive branches. This will be
+ significantly more precise than the method using RDTSC. This mode should be run
+ with at least `Haswell`, but preferably `Skylake` for more precise measurements.
+ Using `lrb_latency` requires setting `lbr-sample-period`.
+
+ In `analysis` mode, you also need to specify at least one of the
+ `-analysis-clusters-output-file=` and `-analysis-inconsistencies-output-file=`.
 
- Specify the run mode. Note that if you pick `analysis` mode, you also need
- to specify at least one of the `-analysis-clusters-output-file=` and
- `-analysis-inconsistencies-output-file=`.
 
 .. option:: -repetition-mode=[duplicate|loop|min]
 
  Specify the repetition mode. `duplicate` will create a large, straight line
  basic block with `num-repetitions` copies of the snippet. `loop` will wrap
  the snippet in a loop which will be run `num-repetitions` times. The `loop`
  mode tends to better hide the effects of the CPU frontend on architectures
  that cache decoded instructions, but consumes a register for counting
  iterations. If performing an analysis over many opcodes, it may be best
  to instead use the `min` mode, which will run each other mode, and produce
  the minimal measured result.
 
 .. option:: -num-repetitions=<Number of repetitions>
 
  Specify the number of repetitions of the asm snippet.
  Higher values lead to more accurate measurements but lengthen the benchmark.
 
 .. option:: -max-configs-per-opcode=<value>
 
  Specify the maximum configurations that can be generated for each opcode.
  By default this is `1`, meaning that we assume that a single measurement is
  enough to characterize an opcode. This might not be true of all instructions:
  for example, the performance characteristics of the LEA instruction on X86
  depends on the value of assigned registers and immediates. Setting a value of

llvm/tools/llvm-exegesis/lib/BenchmarkResult.h

 //===-- BenchmarkResult.h ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
struct BenchmarkMeasure {
   // This is the per-snippet value, i.e. measured quantity for one repetition of
   // the whole snippet.
   double PerSnippetValue;
 };
 
 // The result of an instruction benchmark.
 struct InstructionBenchmark {
   InstructionBenchmarkKey Key;
-  enum ModeE { Unknown, Latency, Uops, InverseThroughput };
+  enum ModeE {
+    Unknown,
+    Latency,
+    Uops,
+    InverseThroughput,
+    // LbrLatency mode is only available on x86.
+    // Must be used with at least, Haswell.
+    LbrLatency,
+  };
   ModeE Mode;
   std::string CpuName;
   std::string LLVMTriple;
   // Which instruction is being benchmarked here?
   const MCInst &keyInstruction() const { return Key.Instructions[0]; }
   // The number of instructions inside the repeated snippet. For example, if a
   // snippet of 3 instructions is repeated 4 times, this is 12.
   int NumRepetitions = 0;

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

 //===-- BenchmarkRunner.h ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// Defines the abstract BenchmarkRunner class for measuring a certain execution
 /// property of instructions (e.g. latency).
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 
 #include "Assembler.h"
 #include "BenchmarkCode.h"
 #include "BenchmarkResult.h"
 #include "LlvmState.h"
 #include "MCInstrDescView.h"
+#include "PerfHelper.h"
 #include "SnippetRepetitor.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/Error.h"
 #include <cstdlib>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace exegesis {
 
 // Common code for all benchmark modes.
 class BenchmarkRunner {
 public:
+  struct RunArg {
+    const BenchmarkCode &CodeSnippet;
+    unsigned NumRepetitions;
+    ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors;
+    bool DumpObjectToDisk;
+    unsigned LbrSamplePeriod;
+  };
   explicit BenchmarkRunner(const LLVMState &State,
                            InstructionBenchmark::ModeE Mode);
 
   virtual ~BenchmarkRunner();
 
+  // TODO(vyng) Make this "Deprecated" and switch caller to using RunArg
   Expected<InstructionBenchmark>
   runConfiguration(const BenchmarkCode &Configuration, unsigned NumRepetitions,
                    ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
                    bool DumpObjectToDisk) const;
 
+  Expected<InstructionBenchmark> runConfiguration(const RunArg &Arg) const;
+
   // Scratch space to run instructions that touch memory.
   struct ScratchSpace {
     static constexpr const size_t kAlignment = 1024;
     static constexpr const size_t kSize = 1 << 20; // 1MB.
     ScratchSpace()
         : UnalignedPtr(std::make_unique<char[]>(kSize + kAlignment)),
           AlignedPtr(
               UnalignedPtr.get() + kAlignment -

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

 //===-- BenchmarkRunner.cpp -------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <array>
+#include <memory>
+#include <mutex>
 #include <string>
+#include <thread>
 
 #include "Assembler.h"
+#include "BenchmarkResult.h"
 #include "BenchmarkRunner.h"
 #include "Error.h"
 #include "MCInstrDescView.h"
 #include "PerfHelper.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 
 namespace llvm {
 namespace exegesis {
 
 BenchmarkRunner::BenchmarkRunner(const LLVMState &State,
                                  InstructionBenchmark::ModeE Mode)
     : State(State), Mode(Mode), Scratch(std::make_unique<ScratchSpace>()) {}
 
 BenchmarkRunner::~BenchmarkRunner() = default;
 
 namespace {
 class FunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
 public:
   FunctionExecutorImpl(const LLVMState &State,
                        object::OwningBinary<object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : Function(State.createTargetMachine(), std::move(Obj)),
         Scratch(Scratch) {}
 
 private:
   Expected<int64_t> runAndMeasure(const char *Counters) const override {
     // We sum counts when there are several counters for a single ProcRes
     // (e.g. P23 on SandyBridge).
     int64_t CounterValue = 0;
     SmallVector<StringRef, 2> CounterNames;
     StringRef(Counters).split(CounterNames, '+');
     char *const ScratchPtr = Scratch->ptr();
     for (auto &CounterName : CounterNames) {
       CounterName = CounterName.trim();
       pfm::PerfEvent PerfEvent(CounterName);
       if (!PerfEvent.valid())
         return make_error<Failure>(
             Twine("invalid perf event '").concat(CounterName).concat("'"));
       pfm::Counter Counter(PerfEvent);
       Scratch->clear();
       {
         CrashRecoveryContext CRC;
         CrashRecoveryContext::Enable();
         const bool Crashed = !CRC.RunSafely([this, &Counter, ScratchPtr]() {
           Counter.start();
           this->Function(ScratchPtr);
           Counter.stop();
         });
         CrashRecoveryContext::Disable();
         // FIXME: Better diagnosis.
         if (Crashed)
           return make_error<SnippetCrash>("snippet crashed while running");
       }
       CounterValue += Counter.read();
     }
     return CounterValue;
   }
 
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
+
+class LbrFunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
+public:
+  LbrFunctionExecutorImpl(
+      const LLVMState &State,
+      llvm::object::OwningBinary<llvm::object::ObjectFile> Obj,
+      BenchmarkRunner::ScratchSpace *Scratch, unsigned LbrSamplePeriod)
+      : Function(State.createTargetMachine(), std::move(Obj)), Scratch(Scratch),
+        LbrSamplePeriod(LbrSamplePeriod) {}
+
+private:
+  struct ThreadArg {
+    std::mutex Mutex;
+    pfm::Counter *CounterPtr;
+    std::condition_variable ConditionVariable;
+    bool Crashed;
+  };
+
+  void workerThread(ThreadArg *Arg) const {
+    pfm::PerfEvent PerfEvent("LBR", LbrSamplePeriod);
+    pfm::Counter Counter(PerfEvent);
+    {
+      std::lock_guard<std::mutex> Lock(Arg->Mutex);
+      Arg->CounterPtr = &Counter;
+    }
+    Arg->ConditionVariable.notify_one();
+    {
+      llvm::CrashRecoveryContext CRC;
+      llvm::CrashRecoveryContext::Enable();
+      const bool Crashed = !CRC.RunSafely([&]() {
+        Counter.start();
+        Function(Scratch->ptr());
+        Counter.stop();
+      });
+      llvm::CrashRecoveryContext::Disable();
+      {
+        std::lock_guard<std::mutex> Lock(Arg->Mutex);
+        Arg->CounterPtr = nullptr;
+      }
+      // FIXME: Better diagnosis.
+      if (Crashed) {
+        Arg->Crashed = true;
+        return;
+      }
+    }
+    Arg->Crashed = false;
+  }
+
+  llvm::Expected<int64_t> runAndMeasure(const char *) const override {
+    Scratch->clear();
+    struct ThreadArg Arg;
+    Arg.CounterPtr = nullptr;
+    std::thread Worker(&LbrFunctionExecutorImpl::workerThread, this, &Arg);
+    {
+      // Waits until the counter is ready.
+      std::unique_lock<std::mutex> Lock(Arg.Mutex);
+      Arg.ConditionVariable.wait(Lock,
+                                 [&Arg] { return Arg.CounterPtr != nullptr; });
+    }
+    if (Arg.Crashed) {
+      return make_error<SnippetCrash>(
+          "Snippet crashed while running in thread.");
+    }
+
+    auto valueOrError = Arg.CounterPtr->readOrError();
+    Worker.join();
+    if (valueOrError) {
+      return valueOrError.takeError();
+    }
+    return valueOrError.get();
+  }
+
+  const ExecutableFunction Function;
+  BenchmarkRunner::ScratchSpace *const Scratch;
+  unsigned LbrSamplePeriod;
+};
+
+std::unique_ptr<BenchmarkRunner::FunctionExecutor>
+createFunctionExecutor(InstructionBenchmark::ModeE Mode, const LLVMState &State,
+                       object::OwningBinary<object::ObjectFile> ObjectFile,
+                       BenchmarkRunner::ScratchSpace *Scratch,
+                       const BenchmarkRunner::RunArg &Arg) {
+  if (Mode == InstructionBenchmark::ModeE::LbrLatency) {
+    return std::make_unique<LbrFunctionExecutorImpl>(
+        State, std::move(ObjectFile), Scratch, Arg.LbrSamplePeriod);
+  }
+  return std::make_unique<FunctionExecutorImpl>(State, std::move(ObjectFile),
+                                                Scratch);
+}
+
 } // namespace
 
 Expected<InstructionBenchmark> BenchmarkRunner::runConfiguration(
     const BenchmarkCode &BC, unsigned NumRepetitions,
     ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
     bool DumpObjectToDisk) const {
+  RunArg arg{BC, NumPrepetitions, Repetitors, DumObjectToDisk};
+  return runConfiguration(arg);
+}
+
+Expected<InstructionBenchmark>
+BenchmarkRunner::runConfiguration(const RunArg &Arg) const {
   InstructionBenchmark InstrBenchmark;
   InstrBenchmark.Mode = Mode;
   InstrBenchmark.CpuName = std::string(State.getTargetMachine().getTargetCPU());
   InstrBenchmark.LLVMTriple =
       State.getTargetMachine().getTargetTriple().normalize();
-  InstrBenchmark.NumRepetitions = NumRepetitions;
-  InstrBenchmark.Info = BC.Info;
+  InstrBenchmark.NumRepetitions = Arg.NumRepetitions;
+  InstrBenchmark.Info = Arg.CodeSnippet.Info;
 
-  const std::vector<MCInst> &Instructions = BC.Key.Instructions;
+  const std::vector<MCInst> &Instructions = Arg.CodeSnippet.Key.Instructions;
 
-  InstrBenchmark.Key = BC.Key;
+  InstrBenchmark.Key = Arg.CodeSnippet.Key;
 
   // If we end up having an error, and we've previously succeeded with
   // some other Repetitor, we want to discard the previous measurements.
   struct ClearBenchmarkOnReturn {
     ClearBenchmarkOnReturn(InstructionBenchmark *IB) : IB(IB) {}
     ~ClearBenchmarkOnReturn() {
       if (Clear)
         IB->Measurements.clear();
     }
     void disarm() { Clear = false; }
 
   private:
     InstructionBenchmark *const IB;
     bool Clear = true;
   };
   ClearBenchmarkOnReturn CBOR(&InstrBenchmark);
 
-  for (const std::unique_ptr<const SnippetRepetitor> &Repetitor : Repetitors) {
+  for (const std::unique_ptr<const SnippetRepetitor> &Repetitor :
+       Arg.Repetitors) {
     // Assemble at least kMinInstructionsForSnippet instructions by repeating
     // the snippet for debug/analysis. This is so that the user clearly
     // understands that the inside instructions are repeated.
     constexpr const int kMinInstructionsForSnippet = 16;
     {
       SmallString<0> Buffer;
       raw_svector_ostream OS(Buffer);
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
-              BC.LiveIns, BC.Key.RegisterInitialValues,
+              Arg.CodeSnippet.LiveIns,
+              Arg.CodeSnippet.Key.RegisterInitialValues,
               Repetitor->Repeat(Instructions, kMinInstructionsForSnippet),
               OS)) {
         return std::move(E);
       }
+
       const ExecutableFunction EF(State.createTargetMachine(),
                                   getObjectFromBuffer(OS.str()));
       const auto FnBytes = EF.getFunctionBytes();
       InstrBenchmark.AssembledSnippet.insert(
           InstrBenchmark.AssembledSnippet.end(), FnBytes.begin(),
           FnBytes.end());
     }
 
if (DumpObjectToDisk) {
       outs() << "Check generated assembly with: /usr/bin/objdump -d "
              << *ObjectFilePath << "\n";
       ObjectFile = getObjectFromFile(*ObjectFilePath);
     } else {
       SmallString<0> Buffer;
       raw_svector_ostream OS(Buffer);
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
-              BC.LiveIns, BC.Key.RegisterInitialValues, Filler, OS)) {
+              Arg.CodeSnippet.LiveIns,
+              Arg.CodeSnippet.Key.RegisterInitialValues, Filler, OS)) {
         return std::move(E);
       }
       ObjectFile = getObjectFromBuffer(OS.str());
     }
 
-    const FunctionExecutorImpl Executor(State, std::move(ObjectFile),
-                                        Scratch.get());
-    auto NewMeasurements = runMeasurements(Executor);
+    std::unique_ptr<BenchmarkRunner::FunctionExecutor> Executor =
+        createFunctionExecutor(Mode, State, std::move(ObjectFile),
+                               Scratch.get(), Arg);
+
+    auto NewMeasurements = runMeasurements(*Executor);
     if (Error E = NewMeasurements.takeError()) {
       if (!E.isA<SnippetCrash>())
         return std::move(E);
       InstrBenchmark.Error = toString(std::move(E));
       return InstrBenchmark;
     }
     assert(InstrBenchmark.NumRepetitions > 0 && "invalid NumRepetitions");
     for (BenchmarkMeasure &BM : *NewMeasurements) {

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.h

 //===-- LatencyBenchmarkRunner.h --------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
public:
   LatencyBenchmarkRunner(const LLVMState &State,
                          InstructionBenchmark::ModeE Mode);
   ~LatencyBenchmarkRunner() override;
 
 private:
   Expected<std::vector<BenchmarkMeasure>>
   runMeasurements(const FunctionExecutor &Executor) const override;
 };
+
+class LbrLatencyBenchmarkRunner : public BenchmarkRunner {
+public:
+  LbrLatencyBenchmarkRunner(const LLVMState &State);
+  ~LbrLatencyBenchmarkRunner() = default;
+
+private:
+  Expected<std::vector<BenchmarkMeasure>>
+  runMeasurements(const FunctionExecutor &Executor) const override;
+};
+
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_LATENCY_H

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

 //===-- LatencyBenchmarkRunner.cpp ------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
case InstructionBenchmark::InverseThroughput:
     Result = {BenchmarkMeasure::Create("inverse_throughput", MinValue)};
     break;
   default:
     break;
   }
   return std::move(Result);
 }
 
+LbrLatencyBenchmarkRunner::LbrLatencyBenchmarkRunner(const LLVMState &State)
+    : BenchmarkRunner(State, InstructionBenchmark::LbrLatency) {}
+
+llvm::Expected<std::vector<BenchmarkMeasure>>
+LbrLatencyBenchmarkRunner::runMeasurements(
+    const FunctionExecutor &Executor) const {
+  // TODO(vyng) Maybe verify that we're running with the right CPU/config
+  auto ExpectedCounterValue = Executor.runAndMeasure("LBR");
+  if (!ExpectedCounterValue)
+    return ExpectedCounterValue.takeError();
+  std::vector<BenchmarkMeasure> Result = {
+      BenchmarkMeasure::Create("lbr-latency", *ExpectedCounterValue)};
+  return std::move(Result);
+}
+
 } // namespace exegesis
 } // namespace llvm

llvm/tools/llvm-exegesis/lib/PerfHelper.h

 //===-- PerfHelper.h ------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// Helpers for measuring perf events.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Error.h"
+#include <cstdint>
 #include <functional>
 #include <memory>
 
 struct perf_event_attr;
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 // Returns true on error.
 bool pfmInitialize();
 void pfmTerminate();
 
 // Retrieves the encoding for the event described by pfm_event_string.
 // NOTE: pfm_initialize() must be called before creating PerfEvent objects.
 class PerfEvent {
 public:
   // http://perfmon2.sourceforge.net/manv4/libpfm.html
   // Events are expressed as strings. e.g. "INSTRUCTION_RETIRED"
-  explicit PerfEvent(StringRef pfm_event_string);
+  explicit PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod = 0);
 
   PerfEvent(const PerfEvent &) = delete;
   PerfEvent(PerfEvent &&other);
   ~PerfEvent();
 
   // The pfm_event_string passed at construction time.
   StringRef name() const;
 
   // Whether the event was successfully created.
   bool valid() const;
 
   // The encoded event to be passed to the Kernel.
   const perf_event_attr *attribute() const;
 
   // The fully qualified name for the event.
   // e.g. "snb_ep::INSTRUCTION_RETIRED:e=0:i=0:c=0:t=0:u=1:k=0:mg=0:mh=1"
   StringRef getPfmEventString() const;
 
+  // Returns true if it should use LBR.
+  bool useLbr() const;
+
 private:
+  void initPerfEvent();
+  void initPerfEventForLbr();
+  unsigned LbrSamplePeriod;
+
   const std::string EventString;
   std::string FullQualifiedEventString;
   perf_event_attr *Attr;
 };
 
 // Uses a valid PerfEvent to configure the Kernel so we can measure the
 // underlying event.
-struct Counter {
+class Counter {
+public:
   // event: the PerfEvent to measure.
   explicit Counter(const PerfEvent &event);
 
   Counter(const Counter &) = delete;
   Counter(Counter &&other) = default;
 
-  ~Counter();
+  virtual ~Counter();
 
-  void start();         // Starts the measurement of the event.
-  void stop();          // Stops the measurement of the event.
-  int64_t read() const; // Return the current value of the counter.
+  /// Starts the measurement of the event.
+  virtual void start();
 
-private:
+  /// Stops the measurement of the event.
+  void stop();
+
+  /// Returns the current value of the counter.
+  virtual int64_t read() const;
+
+  /// Returns the current value of the counter or error if it cannot be read.
+  virtual llvm::Expected<int64_t> readOrError() const { return read(); }
+
+protected:
 #ifdef HAVE_LIBPFM
   int FileDescriptor = -1;
-#endif
+#endif // HAVE_LIBPFM
 };
 
 // Helper to measure a list of PerfEvent for a particular function.
 // callback is called for each successful measure (PerfEvent needs to be valid).
 template <typename Function>
 void Measure(
     ArrayRef<PerfEvent> Events,
     const std::function<void(const PerfEvent &Event, int64_t Value)> &Callback,

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

 //===-- PerfHelper.cpp ------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "PerfHelper.h"
+#include "third_party/llvm/llvm-project/llvm/include/llvm/Support/Error.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Errc.h"
 #include "llvm/Support/raw_ostream.h"
+#include <memory>
 #ifdef HAVE_LIBPFM
 #include "perfmon/perf_event.h"
 #include "perfmon/pfmlib.h"
 #include "perfmon/pfmlib_perf_event.h"
 #endif
+#include <atomic>
 #include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 #ifdef HAVE_LIBPFM
 static bool isPfmError(int Code) { return Code != PFM_SUCCESS; }
 #endif

 PerfEvent::~PerfEvent() {
 #ifdef HAVE_LIBPFM
   delete Attr;
   ;
 #endif
 }
 
 PerfEvent::PerfEvent(PerfEvent &&Other)
-    : EventString(std::move(Other.EventString)),
+    : LbrSamplePeriod(Other.LbrSamplePeriod),
+      EventString(std::move(Other.EventString)),
       FullQualifiedEventString(std::move(Other.FullQualifiedEventString)),
       Attr(Other.Attr) {
   Other.Attr = nullptr;
 }
 
-PerfEvent::PerfEvent(StringRef PfmEventString)
-    : EventString(PfmEventString.str()), Attr(nullptr) {
+void PerfEvent::initPerfEvent() {
 #ifdef HAVE_LIBPFM
   char *Fstr = nullptr;
   pfm_perf_encode_arg_t Arg = {};
   Attr = new perf_event_attr();
   Arg.attr = Attr;
   Arg.fstr = &Fstr;
   Arg.size = sizeof(pfm_perf_encode_arg_t);
   const int Result = pfm_get_os_event_encoding(EventString.c_str(), PFM_PLM3,
                                                PFM_OS_PERF_EVENT, &Arg);
   if (isPfmError(Result)) {
     // We don't know beforehand which counters are available (e.g. 6 uops ports
     // on Sandybridge but 8 on Haswell) so we report the missing counter without
     // crashing.
     errs() << pfm_strerror(Result) << " - cannot create event " << EventString
            << "\n";
   }
   if (Fstr) {
     FullQualifiedEventString = Fstr;
     free(Fstr);
   }
 #endif
 }
 
+void PerfEvent::initPerfEventForLbr() {
+  Attr = new perf_event_attr();
+  *Attr = {0};
+  Attr->size = sizeof(*Attr);
+  Attr->type = PERF_TYPE_HARDWARE;
+  Attr->config = PERF_COUNT_HW_BRANCH_INSTRUCTIONS;
+  Attr->sample_type = PERF_SAMPLE_BRANCH_STACK;
+  Attr->branch_sample_type = PERF_SAMPLE_BRANCH_USER | PERF_SAMPLE_BRANCH_ANY;
+  Attr->sample_period = LbrSamplePeriod;
+  Attr->wakeup_events = 1; // We need this even when using ioctl REFRESH.
+  Attr->disabled = 1;
+  Attr->exclude_kernel = 1;
+  Attr->exclude_hv = 1;
+  Attr->read_format = PERF_FORMAT_GROUP;
+
+  FullQualifiedEventString = "LBR";
+}
+
+PerfEvent::PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod)
+    : LbrSamplePeriod(SamplingPeriod), EventString(PfmEventString.str()),
+      Attr(nullptr) {
+  if (EventString == "LBR") {
+    initPerfEventForLbr();
+  } else {
+    initPerfEvent();
+  }
+}
+
 StringRef PerfEvent::name() const { return EventString; }
 
 bool PerfEvent::valid() const { return !FullQualifiedEventString.empty(); }
 
+bool PerfEvent::useLbr() const { return EventString == "LBR"; }
+
 const perf_event_attr *PerfEvent::attribute() const { return Attr; }
 
 StringRef PerfEvent::getPfmEventString() const {
   return FullQualifiedEventString;
 }
 
 #ifdef HAVE_LIBPFM
 Counter::Counter(const PerfEvent &Event) {

llvm/tools/llvm-exegesis/lib/Target.h

 //===-- Target.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 ///
 /// Classes that handle the creation of target-specific objects. This is
 /// similar to Target/TargetRegistry.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H
 
 #include "BenchmarkResult.h"
 #include "BenchmarkRunner.h"
 #include "Error.h"
 #include "LlvmState.h"
 #include "SnippetGenerator.h"
+#include "third_party/llvm/llvm-project/llvm/include/llvm/Support/Error.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/MC/MCRegisterInfo.h"
 
 namespace llvm {
struct CpuAndPfmCounters {
   bool operator<(StringRef S) const { return StringRef(CpuName) < S; }
 };
 
 class ExegesisTarget {
 public:
   explicit ExegesisTarget(ArrayRef<CpuAndPfmCounters> CpuPfmCounters)
       : CpuPfmCounters(CpuPfmCounters) {}
 
+  virtual Expected<pfm::Counter>
+  CreateCounter(const pfm::PerfEvent &Event) const {
+    return pfm::Counter(Event);
+  }
+
   // Targets can use this to add target-specific passes in assembleToStream();
   virtual void addTargetSpecificPasses(PassManagerBase &PM) const {}
 
   // Generates code to move a constant into a the given register.
   // Precondition: Value must fit into Reg.
   virtual std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                        const APInt &Value) const = 0;
 
private:
   std::unique_ptr<SnippetGenerator> virtual createSerialSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<SnippetGenerator> virtual createParallelSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<BenchmarkRunner> virtual createLatencyBenchmarkRunner(
       const LLVMState &State, InstructionBenchmark::ModeE Mode) const;
   std::unique_ptr<BenchmarkRunner> virtual createUopsBenchmarkRunner(
       const LLVMState &State) const;
+  std::unique_ptr<BenchmarkRunner> virtual createLbrLatencyBenchmarkRunner(
+      const LLVMState &State) const;
 
   const ExegesisTarget *Next = nullptr;
   const ArrayRef<CpuAndPfmCounters> CpuPfmCounters;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H

llvm/tools/llvm-exegesis/lib/Target.cpp

 //===-- Target.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "Target.h"
const ExegesisTarget *ExegesisTarget::lookup(Triple TT) {
   }
   return nullptr;
 }
 
 void ExegesisTarget::registerTarget(ExegesisTarget *Target) {
   if (FirstTarget == nullptr) {
     FirstTarget = Target;
     return;
   }
   if (Target->Next != nullptr)
     return; // Already registered.
   Target->Next = FirstTarget;
   FirstTarget = Target;
 }
 
 std::unique_ptr<SnippetGenerator> ExegesisTarget::createSnippetGenerator(
     InstructionBenchmark::ModeE Mode, const LLVMState &State,
     const SnippetGenerator::Options &Opts) const {
   switch (Mode) {
   case InstructionBenchmark::Unknown:
     return nullptr;
   case InstructionBenchmark::Latency:
     return createSerialSnippetGenerator(State, Opts);
   case InstructionBenchmark::Uops:
   case InstructionBenchmark::InverseThroughput:
     return createParallelSnippetGenerator(State, Opts);
+  case InstructionBenchmark::LbrLatency:
+    return createSerialSnippetGenerator(State, Opts);
   }
   return nullptr;
 }
 
 Expected<std::unique_ptr<BenchmarkRunner>>
 ExegesisTarget::createBenchmarkRunner(InstructionBenchmark::ModeE Mode,
                                       const LLVMState &State) const {
   PfmCountersInfo PfmCounters = State.getPfmCounters();
if (!PfmCounters.CycleCounter) {
               .concat("' mode, sched model does not define a cycle counter."));
     }
     return createLatencyBenchmarkRunner(State, Mode);
   case InstructionBenchmark::Uops:
     if (!PfmCounters.UopsCounter && !PfmCounters.IssueCounters)
       return make_error<Failure>("can't run 'uops' mode, sched model does not "
                                  "define uops or issue counters.");
     return createUopsBenchmarkRunner(State);
+  case InstructionBenchmark::LbrLatency:
+    return createLbrLatencyBenchmarkRunner(State);
   }
   return nullptr;
 }
 
 std::unique_ptr<SnippetGenerator> ExegesisTarget::createSerialSnippetGenerator(
     const LLVMState &State, const SnippetGenerator::Options &Opts) const {
   return std::make_unique<SerialSnippetGenerator>(State, Opts);
 }
 
 std::unique_ptr<SnippetGenerator> ExegesisTarget::createParallelSnippetGenerator(
     const LLVMState &State, const SnippetGenerator::Options &Opts) const {
   return std::make_unique<ParallelSnippetGenerator>(State, Opts);
 }
 
 std::unique_ptr<BenchmarkRunner> ExegesisTarget::createLatencyBenchmarkRunner(
     const LLVMState &State, InstructionBenchmark::ModeE Mode) const {
   return std::make_unique<LatencyBenchmarkRunner>(State, Mode);
 }
 
 std::unique_ptr<BenchmarkRunner>
 ExegesisTarget::createUopsBenchmarkRunner(const LLVMState &State) const {
   return std::make_unique<UopsBenchmarkRunner>(State);
 }
 
+std::unique_ptr<BenchmarkRunner>
+ExegesisTarget::createLbrLatencyBenchmarkRunner(const LLVMState &State) const {
+#if defined(__x86_64__) || defined(__x86__) || defined(__i386__)
+  return std::make_unique<LbrLatencyBenchmarkRunner>(State);
+#else
+#warning "LBR benchmark runner being created for non-X86 arch."
+#endif
+  return nullptr;
+}
+
 static_assert(std::is_pod<PfmCountersInfo>::value,
               "We shouldn't have dynamic initialization here");
 const PfmCountersInfo PfmCountersInfo::Default = {nullptr, nullptr, nullptr,
                                                   0u};
 
 const PfmCountersInfo &ExegesisTarget::getPfmCounters(StringRef CpuName) const {
   assert(std::is_sorted(
              CpuPfmCounters.begin(), CpuPfmCounters.end(),

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

 //===-- Target.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "../Target.h"
 
 #include "../Error.h"
 #include "../ParallelSnippetGenerator.h"
 #include "../SerialSnippetGenerator.h"
 #include "../SnippetGenerator.h"
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "X86.h"
 #include "X86RegisterInfo.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/Sequence.h"
 #include "llvm/MC/MCInstBuilder.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FormatVariadic.h"
+#include <memory>
+#include <vector>
+#ifdef HAVE_LIBPFM
+#include "perfmon/perf_event.h"
+#include "perfmon/pfmlib.h"
+#include "perfmon/pfmlib_perf_event.h"
+#endif
+#include <atomic>
+#include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
 
 namespace llvm {
 namespace exegesis {
 
 // Returns a non-null reason if we cannot handle the memory references in this
 // instruction.
 static const char *isInvalidMemoryInstr(const Instruction &Instr) {
   switch (Instr.Description.TSFlags & X86II::FormMask) {
   default:
     llvm_unreachable("Unknown FormMask value");
   // These have no memory access.
   case X86II::Pseudo:
   case X86II::RawFrm:
   case X86II::AddCCFrm:
   case X86II::PrefixByte:
   case X86II::MRMDestReg:
if (Bytes - ByteOffset >= 1)
     add(fillStackSpace(
         X86::MOV8mi, ByteOffset,
         WideConstant.extractBits(8, ByteOffset * 8).getZExtValue()));
 }
 
 #include "X86GenExegesis.inc"
 
 namespace {
+#ifdef HAVE_LIBPFM
+class X86Counter : public pfm::Counter {
+public:
+  explicit X86Counter(const pfm::PerfEvent &Event) : Counter(Event) {
+    if (Event.useLbr()) {
+      // First page is reserved for perf_event_mmap_page. Data buffer starts on
+      // the next page, so we allocate one more page.
+      MMappedBuffer =
+          mmap(nullptr, (kBufferPages + 1) * getpagesize(),
+               PROT_READ | PROT_WRITE, MAP_SHARED, FileDescriptor, 0);
+      if (MMappedBuffer == MAP_FAILED) {
+        llvm::errs() << "Failed to mmap buffer.";
+      }
+      LbrMode = true;
+    }
+  }
+  virtual ~X86Counter() {}
+
+  void start() override {
+    if (LbrMode) {
+      ioctl(FileDescriptor, PERF_EVENT_IOC_REFRESH, 1024 /* kMaxPollsPerFd */);
+    } else {
+      ioctl(FileDescriptor, PERF_EVENT_IOC_RESET, 0);
+    }
+  }
+
+  int64_t read() const override {
+    auto error = readOrError();
+    if (!error) {
+      return error.get();
+    }
+    llvm::errs() << "Error reading counter: " << error.takeError() << "\n";
+    return 0;
+  }
+
+  llvm::Expected<int64_t> readOrError() const override {
+    // Parses the LBR buffer and fills CycleArray with the sequence of cycle
+    // counts from the buffer.
+    std::vector<int64_t> CycleArray;
+    std::unique_ptr<char[]> DataBuf(new char[DataBufferSize]);
+    size_t Pos = 0;
+    for (;;) {
+      if (pollLbrPerfEvent() == -1) {
+        return llvm::make_error<llvm::StringError>(
+            "Cannot poll LBR perf event.", llvm::errc::io_error);
+      }
+      // First page is reserved for perf_event_mmap_page. Data buffer starts on
+      // the next page.
+      struct perf_event_mmap_page Page;
+      memcpy(&Page, MMappedBuffer, sizeof(struct perf_event_mmap_page));
+      const uint64_t DataTail = Page.data_tail;
+      const uint64_t DataHead = Page.data_head;
+      // We're supposed to use a barrier after reading data_head.
+      std::atomic_thread_fence(std::memory_order_acq_rel);
+      const size_t DataSize = DataHead - DataTail;
+      if (DataSize > DataBufferSize) {
+        return llvm::make_error<llvm::StringError>(
+            "DataSize larger than buffer size.", llvm::errc::invalid_argument);
+      }
+      copyDataBuffer(DataBuf.get(), DataTail, DataSize);
+      llvm::Error error =
+          readDataBuffer(DataBuf.get(), DataSize, &CycleArray, Pos);
+      if (!error) {
+        // TODO(vyng) Analyse the array and get proper value.
+        return CycleArray[0];
+      }
+    }
+    return llvm::make_error<llvm::StringError>("Unknown error.",
+                                               llvm::errc::io_error);
+  }
+
+private:
+  // Waits for the LBR perf events.
+  int pollLbrPerfEvent() const {
+    struct pollfd PollFd;
+    PollFd.fd = FileDescriptor;
+    PollFd.events = POLLIN;
+    PollFd.revents = 0;
+    return poll(&PollFd, 1 /* num of fds */, 10000 /* time out */);
+  }
+
+  void copyDataBuffer(char *Buf, uint64_t Tail, size_t DataSize) const {
+    // First page is reserved for perf_event_mmap_page. Data buffer starts on
+    // the next page.
+    char *Start = reinterpret_cast<char *>(MMappedBuffer) + getpagesize();
+    // The LBR buffer is a cyclic buffer, we copy data to another buffer.
+    uint64_t Offset = Tail % DataBufferSize;
+    size_t CopySize = DataBufferSize - Offset;
+    memcpy(Buf, Start + Offset, CopySize);
+    if (CopySize >= DataSize)
+      return;
+    memcpy(Buf + CopySize, Start, Offset);
+  }
+
+  llvm::Error readDataBuffer(const char *DataBuf, size_t DataSize,
+                             std::vector<int64_t> *CycleArray,
+                             size_t Pos) const {
+    const char *DataPtr = DataBuf;
+    while (DataPtr < DataBuf + DataSize) {
+      struct perf_event_header Header;
+      memcpy(&Header, DataPtr, sizeof(struct perf_event_header));
+      if (Header.type != PERF_RECORD_SAMPLE) {
+        // Ignores non-sample records.
+        DataPtr += Header.size;
+        continue;
+      }
+      DataPtr += sizeof(Header);
+      uint64_t Count = llvm::support::endian::read64(DataPtr, support::native);
+      DataPtr += sizeof(Count);
+
+      struct perf_branch_entry Entry;
+      memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+      // Read the perf_branch_entry array.
+      char *JumpPC = reinterpret_cast<char *>(Entry.from);
+      int64_t MinCycle = Entry.cycles;
+      for (int i = 0; i < Count; ++i) {
+        // We use the JumpPC from the entry with min cycle to avoid the entry
+        // that is returning from kernel.
+        if (MinCycle > Entry.cycles) {
+          MinCycle = Entry.cycles;
+          JumpPC = reinterpret_cast<char *>(Entry.from);
+        }
+        (*CycleArray)[Pos++] = Entry.cycles;
+        if (Pos == CycleArray->size()) {
+          patchBasicBlockToEndBenchmarkedLoop(JumpPC);
+          return llvm::Error::success();
+        }
+        // Advance to next entry
+        DataPtr += sizeof(Entry);
+        memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+      }
+    }
+    return llvm::make_error<llvm::StringError>("Unable to read databuffer.",
+                                               llvm::errc::io_error);
+  }
+
+  void patchBasicBlockToEndBenchmarkedLoop(char *pc) const {
+    auto prot = PROT_READ | PROT_WRITE | PROT_EXEC;
+    auto page = reinterpret_cast<void *>(reinterpret_cast<uintptr_t>(pc) &
+                                         ~(getpagesize() - 1));
+    mprotect(page, getpagesize(), prot);
+    // Update the last jump back instruction, pc is the start of jump
+    // instruction.
+    if ((reinterpret_cast<uint64_t>(pc) & 0xf) != 0xf) {
+      // We have at least two bytes in the same cacheline, so we can use an
+      // atomic write to replace the jmp with pop, ret.
+      *reinterpret_cast<uint16_t *>(pc) = 0xc35b; // pop %rbx; ret;
+    } else {
+      // pc and pc+1 might be at different cacheline, so we first add the pop,
+      // ret instructions, and then update jump offset to jump to pop
+      // instruction.
+      *reinterpret_cast<uint16_t *>(pc + 5) = 0xc35b; // pop %rbx; ret;
+      // An automic update on the displacement.
+      if (pc[0] == 0xeb /* x86 short jump */) {
+        // offset is 1 byte
+        pc[1] = 0x03;
+      } else { /* x86 near jump */
+        // offset is 4 bytes
+        *reinterpret_cast<uint32_t *>(pc + 1) = 0;
+      }
+    }
+  }
+
+  bool LbrMode = false;
+  void *MMappedBuffer = nullptr;
+  size_t DataBufferSize;
+  static constexpr int kBufferPages = 8;
+};
+#endif
+
 class ExegesisX86Target : public ExegesisTarget {
 public:
   ExegesisX86Target() : ExegesisTarget(X86CpuPfmCounters) {}
 
+  Expected<pfm::Counter>
+  CreateCounter(const pfm::PerfEvent &Event) const override {
+#ifdef HAVE_LIBPFM
+    return X86Counter(Event);
+#else
+    return pfm::Counter(Event);
+#endif
+  }
+
 private:
   void addTargetSpecificPasses(PassManagerBase &PM) const override;
 
   unsigned getScratchMemoryRegister(const Triple &TT) const override;
 
   unsigned getLoopCounterRegister(const Triple &) const override;
 
   unsigned getMaxMemoryAccessSize() const override { return 64; }
 
   Error randomizeTargetMCOperand(const Instruction &Instr, const Variable &Var,
                                  MCOperand &AssignedValue,
                                  const BitVector &ForbiddenRegs) const override;
 
   void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                           unsigned Offset) const override;
 
   void decrementLoopCounterAndJump(MachineBasicBlock &MBB,
                                    MachineBasicBlock &TargetMBB,
                                    const MCInstrInfo &MII) const override;
 
   std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                const APInt &Value) const override;
 
   ArrayRef<unsigned> getUnavailableRegisters() const override {
     return makeArrayRef(kUnavailableRegisters,
                         sizeof(kUnavailableRegisters) /
                             sizeof(kUnavailableRegisters[0]));
   }
 
   bool allowAsBackToBack(const Instruction &Instr) const override {
     const unsigned Opcode = Instr.Description.Opcode;

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

 //===-- llvm-exegesis.cpp ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
BenchmarkFile("benchmarks-file",
                            "(latency/uops/inverse_throughput modes) benchmark "
                            "results. “-” uses stdin/stdout."),
                   cl::cat(Options), cl::init(""));
 
 static cl::opt<exegesis::InstructionBenchmark::ModeE> BenchmarkMode(
     "mode", cl::desc("the mode to run"), cl::cat(Options),
     cl::values(clEnumValN(exegesis::InstructionBenchmark::Latency, "latency",
                           "Instruction Latency"),
+               clEnumValN(exegesis::InstructionBenchmark::LbrLatency,
+                          "lbr_latency", "Instruction Latency using LBR"),
                clEnumValN(exegesis::InstructionBenchmark::InverseThroughput,
                           "inverse_throughput",
                           "Instruction Inverse Throughput"),
                clEnumValN(exegesis::InstructionBenchmark::Uops, "uops",
                           "Uop Decomposition"),
                // When not asking for a specific benchmark mode,
                // we'll analyse the results.
                clEnumValN(exegesis::InstructionBenchmark::Unknown, "analysis",
                           "Analysis")));
 
+static cl::opt<unsigned>
+    LbrSamplePeriod("lbr-sample-period",
+                    cl::desc("The sample period in msec used for LBR sampling"),
+                    cl::cat(BenchmarkOptions), cl::init(10000));
+
 static cl::opt<exegesis::InstructionBenchmark::RepetitionModeE> RepetitionMode(
     "repetition-mode", cl::desc("how to repeat the instruction snippet"),
     cl::cat(BenchmarkOptions),
     cl::values(
         clEnumValN(exegesis::InstructionBenchmark::Duplicate, "duplicate",
                    "Duplicate the snippet"),
         clEnumValN(exegesis::InstructionBenchmark::Loop, "loop",
                    "Loop over the snippet"),
#endif
   InitializeNativeTarget();
   InitializeNativeTargetAsmPrinter();
   InitializeNativeTargetAsmParser();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
 
   const std::unique_ptr<BenchmarkRunner> Runner = ExitOnErr(
       State.getExegesisTarget().createBenchmarkRunner(BenchmarkMode, State));
   if (!Runner) {
     ExitWithError("cannot create benchmark runner");
   }
 
   const auto Opcodes = getOpcodesOrDie(State.getInstrInfo());
 
   SmallVector<std::unique_ptr<const SnippetRepetitor>, 2> Repetitors;
   if (RepetitionMode != InstructionBenchmark::RepetitionModeE::AggregateMin)
for (const unsigned Opcode : Opcodes) {
       }
       std::move(ConfigsForInstr->begin(), ConfigsForInstr->end(),
                 std::back_inserter(Configurations));
     }
   } else {
     Configurations = ExitOnErr(readSnippets(State, SnippetsFile));
   }
 
+#if !defined(__x86_64__) && !defined(__x86__) && !defined(__i386__)
+  if (exegesis::BenchmarkMode == exegesis::InstructionBenchmark::LbrLatency) {
+    ExitOnErr.setBanner("llvm-exegesis: ");
+    ExitWithError("LBR mode must be run on x86 arch.");
+  }
+#endif
+
   if (NumRepetitions == 0) {
     ExitOnErr.setBanner("llvm-exegesis: ");
     ExitWithError("--num-repetitions must be greater than zero");
   }
 
   // Write to standard output if file is not set.
   if (BenchmarkFile.empty())
     BenchmarkFile = "-";
 
   for (const BenchmarkCode &Conf : Configurations) {
-    InstructionBenchmark Result = ExitOnErr(Runner->runConfiguration(
-        Conf, NumRepetitions, Repetitors, DumpObjectToDisk));
+    BenchmarkRunner::RunArg RunArg{Conf, NumRepetitions, Repetitors,
+                                   DumpObjectToDisk, LbrSamplePeriod};
+
+    InstructionBenchmark Result = ExitOnErr(Runner->runConfiguration(RunArg));
     ExitOnFileError(BenchmarkFile, Result.writeYaml(State, BenchmarkFile));
   }
   exegesis::pfm::pfmTerminate();
 }
 
 // Prints the results of running analysis pass `Pass` to file `OutputFilename`
 // if OutputFilename is non-empty.
 template <typename Pass>