[llvm-exegesis] Add benchmark mode that uses LBR for more precise measurements.

[oontvoo]

Summary

Add benchmark mode that uses LBR for more precise measurements.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

88 ↗

(On Diff #254993)

Let's move this to the X86 target, and rename it to x86-lbr-sample-period, with a default of 0. If non-zero, we're going to use LBR. Else, we'll use the non-LBR runner.

[courbet]

And thanks for the contribution.

[oontvoo]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

107

Yes, it would've ... Changed to Counter*

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

90

Yes - it should be macro guarded. (I mindlessly deleted the guards while moving things around)
Added it back.

[oontvoo]

Addressed review comments:

• remove the new mode and merge it into latency
• various refactorings and comments-cleanup per courbet@'s requests

[oontvoo]

Fix typos and rebase

[oontvoo]

Fix clang-tidy warning

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

Can you expand on what "hiding blocks" means ? e.g. an external reference ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

88

Apart from the PerfEvent, this does not really have anything to do with LBR.
On the one hand, this is not really a full-fledged FunctionExecutor, as it just throws away the CounterName argument and always measures the latency. This would be an argument for moving this to the X86 backend.
On the other hand, this has a lot of target-independent code, so it's a good thing that other targets can use it. The main issue, and the only thing that's really specific to LBR here is the perf event. Letting the target create the perf event as well as the counter does fix this.

So what we can do:

• change Target::createCounter to take a counter name rather than a PerfEvent. The default implementation creates the PerfEvent, validates it and wraps it in a Counter. The X86 implementation tests if we're asking for the cycles counter in LBR mode, and creates the relevant PerfEvent and LBRCounter. It falls back to the default implementation else.
• We stop ignoring the counter name argument in LbrFunctionExecutorImpl::runAndMeasure, and just propagate it to the ExegesisTarget.
• We rename LbrFunctionExecutorImpl into WorkerThreadFunctionExecutorImpl (and FunctionExecutorImpl could be renamed to SameThreadFunctionExecutor, and everything becomes target agnostic.

196

Again, this is too specific to X86. I think it's one of these cases where is actually makes more sense for the generic code to know nothing about LBR. What about:

• Moving LbrSamplePeriod from llvm-exegesis to X86/Target.cpp
• Making createFunctionExecutor() a virtual member of the target. The default implementation returns a FunctionExecutorImpl.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

42

The LLVM style is FIXME

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

32

do we really need all these ?

597

[nit] s/PFM package available../HAVE_LIBPFM/ the package might be available but specifically disabled

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

17

Does this work under windows ? If not, please use one of the LLVM wrappers.

30

internal functions use a static keyword instead of anonymous namespace in the LLVM style guide.

35

s/time out/timeout

39

static

53

This would benefit from a pointer to documentation (or argued speculation about) whether rewriting code inside a loop for another thread is valid and how it behaves w.r.t. caches and DSB.

57

static

84

static

126

you can remove this now

198

FIXME. Also, please give more explanations so that somebody else can look at that too ;)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

2

This file, and the cpp one, are missing the LLVM headers.

[oontvoo]

• Make internal helpers static functions (instead of anonymous ns members)
• Reword confusing comments
• Add LLVM headers

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

typo: ..

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

105

std::unique_ptr + no dtor ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

42

FIXME and no name :)

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

199

The LBR on Haswell has only the addresses of the branches. To get the timing, Skylake is the minimal requirement.

207

I'd add a mention that this applies to the LBR ratency mode.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

184

As noted above - Haswell and Broadwell have the LBR, but it does not report the number of cycles between the branches.

[oontvoo]

Addressed review comments.

[oontvoo]

Add more comment

[oontvoo]

Thanks for the thorough reviews! :-)

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

88

Done - instead of moving the executor out, I've just let the Target decides whether to use WorkerThread executor or otherwise.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

17

Per offline discussion, we'll ifdef this out for non-linux and will add support for windows in follow-up CLs

53

I meant to rewrite this hack to use a condition var for stopping the loop from another thread, per Ondrej's suggestion .
I just haven't got to it yet :-) ... (would need more changes in the assembler.cpp )

So I'll add a FIXME for next CLs

198

Per offline discussions, this requires re-working the read() interface because we need to return more than one value.
(Specifically, the lbr-entries offer a list of cycles-latencies for a bunch of consec branches ... we don't want to just ignore the older branches and only look at the most recent)

But because this patch is already quite big , we'll keep this as TODO and follow up.

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

88

That forces passing the RunArg to the target, which has no reason to know about it.

What I'm suggesting is the following:

• move LbrSamplePeriod to X86/Target.cpp. This whole SamplePeriod thing, and the existence of the option is really just a X86 detail, the target-independent code has no need to know about any of this.
• Get rid of SamplePeriod, in the target-independant code. RunArg can go too.
• createCounter does not need Description, or SamplePeriod. The X86 target handles it internally.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

618

This should really onely be:

Expected<std::unique_ptr<pfm::Counter>>
createCounter(const char *CounterName) {
  if (CounterName == kCyclesCounter && LbrSamplingPeriod > 0) {
    return std::make_unique<X86LbrCounter>(PerfEvent("LBR", LbrSamplingPeriod));
  }
  return ExegesisTarget::createCounter(CounterName);
}

620

This will fail to compile under windows because X86LbrCounter does not exist there.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

348 ↗

(On Diff #256477)

This compile-time guarded runtime check can now go, because the x86 specific concepts are handled in the x86 target.

[oontvoo]

Addressed review comments:

• remove no-longer needed RunArg
• move target-specific to X86/Target

[oontvoo]

Add check for the LBR format to ensure it contains cycles.

[oontvoo]

Update perf_event_attr to use BR_INST_RETIRED.NEAR_TAKEN per eranian@'s recommendation.

[oontvoo]

ran clang-format

[oontvoo]

revert unrelated change

[oontvoo]

Fix dup options

[oontvoo]

Add new cpp file to makefile

[oontvoo]

Update perf-even-attr for raw-encoding

[oontvoo]

Updated patch to cover what we discussed in the sync last week and this week, specifically:

• Remove the worker-thread executor (not needed. It's fine to use the same thread)
• Make PerfEvent virtual so the X86Counter can create its own PerfEvent with LBR-specific attributes.

Remaining questions:

• Counter should be able to return more than one readings.
• We should validate that repetion-mode=loop for LBR

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

On X86,

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

13

you're no longer using <thread> and <mutex>.

16

is this needed ?

70

s/auto/pfm::Counter/

llvm/tools/llvm-exegesis/lib/PerfHelper.h

45

why is the dtor virtual ? I don't see any virtual methods ?

llvm/tools/llvm-exegesis/lib/Target.h

69

Can you submit this change independently and rebase ? Thanks !

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

592

&& LbrSamplingPeriod > 0 ?

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

65

fills

184

[style]

if (PollResult == -1)
  return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.", llvm::errc::io_error);
if (PollResult != 0) 
  continue;
...

185

how often does that happen ? Do we want that much logging ?

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

67

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/Target.cpp

34

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

[style] no braces for 1-line conditional bodies

131

[style] no braces for 1-line conditional bodies

144

[style] no braces for 1-line conditional bodies

206

[style] no braces for 1-line conditional bodies

216

ditto

223

ditto

[oontvoo]

Addressed review comments

[oontvoo]

llvm/tools/llvm-exegesis/lib/Target.h

69

Done https://reviews.llvm.org/D80610

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

185

It could happen 100% of the time if the code happens to have no branches.
With "proper" input, it happens less frequently. I don't really have a sense of how often ... it's probably be fine to change this to llvm::debug() or something?

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

184

you don't need the else after the return.

185

I'm not worried about the no-branch case as we can avoid that. We might as well remove it, because we don't really care if it fanally succeeded, and if it did not we can print an error there.

190

you don't need the else here after the continue.

[oontvoo]

Rebase to reduce diff

[oontvoo]

Simplified re-try

[courbet]

OK, I think this is in good shape now, but we still need a test for it. The only issue is that not all buildbots support LBR, so we need a mechanism for skipping the test when not available. You can create a subdir llvm-project/llvm/test/tools/llvm-exegesis/X86/lbr with a custom lit similar to llvm-project/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg to check lbr support before running the test.

[oontvoo]

Add test and option to check for expected CPU.

I can't seem to find a way to get the host CPU version (much less name or pmu capabilities) from the lit config, so I had to add the check to llvm-exegesis.

[oontvoo]

Rebase

[oontvoo]

This comment was removed by oontvoo.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291 ↗

(On Diff #268037)

This is a bit brittle, because we could imagine the name of some unrelated CPUs being substrings of others. What about having a repeated option --allowed-host-cpu=skylake --allowed-host-cpu=skylake-avx512 --allowed-host-cpu=whateverlake, and check that the exact value is one of these ?

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

This is more of a philosophical comment: IIRC, the LBR records the times when the branch instructions are retired. With correct branch prediction, the instructions in the measured block would start executing speculatively before the branch is retired, and so the number we get is probably closer to the throughput.

211

I'm wondering how this applies to this use case. The issue with basic blocks skipped might be when sampling all basic blocks in a larger application over a longer period of time, but this is not our case.
In our case, we're interested in a single basic block, and run it in a tight loop for relatively long, so that the LBR sampling catches this exact basic block.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

I think we also need to filter by the addresses of the branches (or store them) to make sure that we take the cycles only for the measured basic block, not for the other basic blocks surrounding it.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291 ↗

(On Diff #268037)

Ideally, this should be defined in terms of CPU features (e.g. CPUID bits). Even better - each target should know which counters it supports, based on its platform-specific feature discovery mechanism. I understand that this would be a huge change for this CL, but we should at least have a FIXME here.

[oontvoo]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291 ↗

(On Diff #268037)

Actually, I had a question here that I was gonna ask in an email But here goes.

We know that the LBR formats could be queried from the perf-cap MSR
Specifically, we want: MSR IA32_PERF_CAPABILITIES[5:0]" == 000110B (bit 59...63 is not relevant)

If I'm not mistaken, perf_event_mmap_page::capabilities should(?) give us that.
Except, when I run this on both Broadwell and Skylake, the capabilities field has value of 30 for both platforms. (It shouldn't be). Of course, the difference here is that the cycle entries are all zeroes on Broadwell .
I haven't looked in details, so I don't know if this is some implementation detail of the libpfm or if we're just mis-interpreting the SDM here.

Thoughts?

[ondrasej]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

291 ↗

(On Diff #268037)

This is strange.

I understand the SDM section about IA32_PERF_CAPABILITIES the same way, i.e. we should see 000110B if there are timings, and this should happen only on Skylake or onwards.

That said, I don't think that perf_event_mmap_page::capabilities contains the IA32_PERF_CAPABILITIES MSR. Instead, it looks like a struct that is defined and filled by the kernel. Check out the definition/docs of the struct (and the code that fills it). They set individual bit fields (and even the bit field defined in the union does not match the other bits defined in the MSR).

After some digging through kernel sources, I see where the kernel keeps the data internally, but I didn't see a way to get it out of the kernel. We might need to do more research on that.

[oontvoo]

Filter and keep only records for the benchmarked codeblock

[oontvoo]

Add option to not filter the records too

[oontvoo]

Updated diff

[oontvoo]

clang-tidy

[courbet]

Only style comments left

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

79

let's use a c++ cast here.

81

Actually I'm thinking you can just pass a StringRef to readOrError

llvm/tools/llvm-exegesis/lib/PerfHelper.h

78–79

Please explain what the arguments are (and switch to StringRef as mentioned above ?)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

150

braces

205

style: no braces

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

9

This could use a file comment with a link to LBR documentation.

[oontvoo]

updated diff

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

179

Let's return an error when FunctionBytes.empty(). Right now this will assert later.

[oontvoo]

Return error for empty function

[ondrasej]

Looks good overall.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

51–52

[style] Remove the empty line.

[oontvoo]

Removed empty line

[oontvoo]

Fixed warning [-Wmissing-field-initializers] in code

[erichkeane]

This patch doesn't seem to build for me:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp: In function ‘llvm::Error llvm::exegesis::parseDataBuffer(const char*, size_t, const void*, const void*, llvm::SmallVector<long int, 4>*)’:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp:99:37: error: ‘struct perf_branch_entry’ has no member named ‘cycles’

CycleArray->push_back(Entry.cycles);

I'm on RHEL7, so I have kernel 3.10, so it doesn't have 'cycles'.

According ot this: https://elixir.bootlin.com/linux/v4.3/source/include/uapi/linux/perf_event.h#L963 kernel 4.3 is the first time that 'cycles' appeared in this structure.

[courbet]

If you're not using llvm-exegesis you can disable this with the LLVM_ENABLE_LIBPFM cmake option.
But I'll revert this until @oontvoo can fix llvm/cmake/modules/FindLibpfm.cmake

[oontvoo]

Check for existence of cycles before enabling LBR mode.

[gchatelet]

LGTM

[oontvoo]

Relanded.
https://reviews.llvm.org/rGee7caa75939afb75547c00744c5df4d04d45e517

llvm/docs/CommandGuide/llvm-exegesis.rst

 
 .. option:: -snippets-file=<filename>
 
  Specify the custom code snippet to measure. See example 2 for details.
  Either `opcode-index`, `opcode-name` or `snippets-file` must be set.
 
 .. option:: -mode=[latency|uops|inverse_throughput|analysis]
 
- Specify the run mode. Note that if you pick `analysis` mode, you also need
- to specify at least one of the `-analysis-clusters-output-file=` and
- `-analysis-inconsistencies-output-file=`.
+ Specify the run mode. Note that some modes have additional requirements and options.
+
+ `latency` mode can be  make use of either RDTSC or LBR.
+ `latency[LBR]` is only available on X86 (at least `Skylake`).
+  To run in this mode, a positive value  must be specified for `x86-lbr-sample-period`
+
+ In `analysis` mode, you also need to specify at least one of the
+ `-analysis-clusters-output-file=` and `-analysis-inconsistencies-output-file=`.
+
+.. option:: -x86-lbr-sample-period=<nBranches/sample>
+
+  Specify the LBR sampling period - how many branches before we take a sample.
+  When a positive value is specified for this option and when the mode is `latency`,
+  we will use LBRs for measuring.
+  On choosing the "right" sampling period, a small value is preferred, but throttling
+  could occur if the sampling is too frequent. A prime number should be used to
+  avoid consistently skipping certain blocks.
   
 .. option:: -repetition-mode=[duplicate|loop|min]
 
  Specify the repetition mode. `duplicate` will create a large, straight line
  basic block with `num-repetitions` copies of the snippet. `loop` will wrap
  the snippet in a loop which will be run `num-repetitions` times. The `loop`
  mode tends to better hide the effects of the CPU frontend on architectures
  that cache decoded instructions, but consumes a register for counting
  iterations. If performing an analysis over many opcodes, it may be best

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 
 #include "Assembler.h"
 #include "BenchmarkCode.h"
 #include "BenchmarkResult.h"
 #include "LlvmState.h"
 #include "MCInstrDescView.h"
+#include "PerfHelper.h"
 #include "SnippetRepetitor.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/Error.h"
 #include <cstdlib>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace exegesis {
 
 // Common code for all benchmark modes.
 class BenchmarkRunner {
 public:
   explicit BenchmarkRunner(const LLVMState &State,
                            InstructionBenchmark::ModeE Mode);
 
   virtual ~BenchmarkRunner();
 
   Expected<InstructionBenchmark>
   runConfiguration(const BenchmarkCode &Configuration, unsigned NumRepetitions,
                    ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
                    bool DumpObjectToDisk) const;
 
   // Scratch space to run instructions that touch memory.
   struct ScratchSpace {
     static constexpr const size_t kAlignment = 1024;
     static constexpr const size_t kSize = 1 << 20; // 1MB.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

 //===-- BenchmarkRunner.cpp -------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <array>
+#include <memory>
+#include <mutex>
 #include <string>
+#include <thread>
 
 #include "Assembler.h"
+#include "BenchmarkResult.h"
 #include "BenchmarkRunner.h"
 #include "Error.h"
 #include "MCInstrDescView.h"
 #include "PerfHelper.h"
+#include "Target.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 
 namespace llvm {
 namespace exegesis {
 
 BenchmarkRunner::BenchmarkRunner(const LLVMState &State,
                                  InstructionBenchmark::ModeE Mode)
     : State(State), Mode(Mode), Scratch(std::make_unique<ScratchSpace>()) {}
 
 BenchmarkRunner::~BenchmarkRunner() = default;
 
 namespace {
 class FunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
 public:
   FunctionExecutorImpl(const LLVMState &State,
                        object::OwningBinary<object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : Function(State.createTargetMachine(), std::move(Obj)),
         Scratch(Scratch) {}
 
 private:
   Expected<int64_t> runAndMeasure(const char *Counters) const override {
     // We sum counts when there are several counters for a single ProcRes
     // (e.g. P23 on SandyBridge).
     int64_t CounterValue = 0;
     SmallVector<StringRef, 2> CounterNames;
     StringRef(Counters).split(CounterNames, '+');
     char *const ScratchPtr = Scratch->ptr();
     for (auto &CounterName : CounterNames) {
       CounterName = CounterName.trim();
       pfm::PerfEvent PerfEvent(CounterName);
       if (!PerfEvent.valid())
         return make_error<Failure>(
             Twine("invalid perf event '").concat(CounterName).concat("'"));
       pfm::Counter Counter(std::move(PerfEvent));
       Scratch->clear();
       {
         CrashRecoveryContext CRC;
         CrashRecoveryContext::Enable();
         const bool Crashed = !CRC.RunSafely([this, &Counter, ScratchPtr]() {
           Counter.start();
           this->Function(ScratchPtr);
           Counter.stop();
         });
         CrashRecoveryContext::Disable();
         // FIXME: Better diagnosis.
         if (Crashed)
           return make_error<SnippetCrash>("snippet crashed while running");
       }
       CounterValue += Counter.read();
     }
     return CounterValue;
   }
 
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
+
+class WorkerThreadFunctionExecutorImpl
+    : public BenchmarkRunner::FunctionExecutor {
+public:
+  WorkerThreadFunctionExecutorImpl(
+      const LLVMState &State,
+      llvm::object::OwningBinary<llvm::object::ObjectFile> Obj,
+      BenchmarkRunner::ScratchSpace *Scratch)
+      : State(State), Function(State.createTargetMachine(), std::move(Obj)),
+        Scratch(Scratch) {}
+
+  struct ThreadArg {
+    std::mutex Mutex;
+    std::unique_ptr<pfm::Counter> CounterPtr;
+    std::condition_variable ConditionVariable;
+    bool Crashed;
+    std::string ErrMsg;
+    const char *CounterName;
+  };
+
+  void workerThread(struct ThreadArg *Arg) const {
+    auto CounterOrError =
+        State.getExegesisTarget().createCounter(Arg->CounterName, State);
+    if (!CounterOrError) {
+      std::lock_guard<std::mutex> Lock(Arg->Mutex);
+      Arg->Crashed = true;
+      llvm::errs() << "Error creating counter: " << CounterOrError.takeError();
+      Arg->ErrMsg = "Cannot create counter.";
+      Arg->ConditionVariable.notify_one();
+      return;
+    }
+    {
+      std::lock_guard<std::mutex> Lock(Arg->Mutex);
+      Arg->CounterPtr.reset(CounterOrError.get().get());
+    }
+    Arg->ConditionVariable.notify_one();
+    {
+      llvm::CrashRecoveryContext CRC;
+      llvm::CrashRecoveryContext::Enable();
+      const bool Crashed = !CRC.RunSafely([&]() {
+        Arg->CounterPtr->start();
+        Function(Scratch->ptr());
+        Arg->CounterPtr->stop();
+      });
+      llvm::CrashRecoveryContext::Disable();
+      {
+        std::lock_guard<std::mutex> Lock(Arg->Mutex);
+        Arg->CounterPtr.reset(nullptr);
+      }
+      // FIXME: Better diagnosis.
+      if (Crashed) {
+        Arg->Crashed = true;
+        return;
+      }
+    }
+    Arg->Crashed = false;
+  }
+
+  llvm::Expected<int64_t>
+  runAndMeasure(const char *CounterName) const override {
+    Scratch->clear();
+    struct ThreadArg Arg;
+    Arg.CounterPtr = nullptr;
+    Arg.CounterName = CounterName;
+    std::thread Worker(&WorkerThreadFunctionExecutorImpl::workerThread, this,
+                       &Arg);
+    {
+      // Waits until the counter is ready or a "Crashed" signal if
+      // it could not be created.
+      std::unique_lock<std::mutex> Lock(Arg.Mutex);
+      Arg.ConditionVariable.wait(
+          Lock, [&Arg] { return Arg.Crashed || Arg.CounterPtr != nullptr; });
+    }
+    if (Arg.Crashed) {
+      return make_error<SnippetCrash>(
+          "Snippet crashed while running in thread. Reason: [" + Arg.ErrMsg +
+          "]");
+    }
+
+    auto ValueOrError = Arg.CounterPtr->readOrError();
+    Worker.join();
+    if (!ValueOrError) {
+      return ValueOrError.takeError();
+    }
+    return ValueOrError.get();
+  }
+
+  const LLVMState &State;
+  const ExecutableFunction Function;
+  BenchmarkRunner::ScratchSpace *const Scratch;
+};
+
+std::unique_ptr<BenchmarkRunner::FunctionExecutor>
+createFunctionExecutor(InstructionBenchmark::ModeE Mode, const LLVMState &State,
+                       object::OwningBinary<object::ObjectFile> ObjectFile,
+                       BenchmarkRunner::ScratchSpace *Scratch) {
+  const ExegesisTarget &Target = State.getExegesisTarget();
+  if (Target.useWorkerThreadForBenchmark(State, Mode)) {
+    return std::make_unique<WorkerThreadFunctionExecutorImpl>(
+        State, std::move(ObjectFile), Scratch);
+  }
+
+  return std::make_unique<FunctionExecutorImpl>(State, std::move(ObjectFile),
+                                                Scratch);
+}
+
 } // namespace
 
 Expected<InstructionBenchmark> BenchmarkRunner::runConfiguration(
     const BenchmarkCode &BC, unsigned NumRepetitions,
     ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
     bool DumpObjectToDisk) const {
   InstructionBenchmark InstrBenchmark;
   InstrBenchmark.Mode = Mode;
   InstrBenchmark.CpuName = std::string(State.getTargetMachine().getTargetCPU());
   InstrBenchmark.LLVMTriple =
       State.getTargetMachine().getTargetTriple().normalize();
   InstrBenchmark.NumRepetitions = NumRepetitions;
constexpr const int kMinInstructionsForSnippet = 16;
       raw_svector_ostream OS(Buffer);
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
               BC.LiveIns, BC.Key.RegisterInitialValues,
               Repetitor->Repeat(Instructions, kMinInstructionsForSnippet),
               OS)) {
         return std::move(E);
       }
+
       const ExecutableFunction EF(State.createTargetMachine(),
                                   getObjectFromBuffer(OS.str()));
       const auto FnBytes = EF.getFunctionBytes();
       InstrBenchmark.AssembledSnippet.insert(
           InstrBenchmark.AssembledSnippet.end(), FnBytes.begin(),
           FnBytes.end());
     }
 
if (DumpObjectToDisk) {
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
               BC.LiveIns, BC.Key.RegisterInitialValues, Filler, OS)) {
         return std::move(E);
       }
       ObjectFile = getObjectFromBuffer(OS.str());
     }
 
-    const FunctionExecutorImpl Executor(State, std::move(ObjectFile),
-                                        Scratch.get());
-    auto NewMeasurements = runMeasurements(Executor);
+    std::unique_ptr<BenchmarkRunner::FunctionExecutor> Executor =
+        createFunctionExecutor(Mode, State, std::move(ObjectFile),
+
+    auto NewMeasurements = runMeasurements(*Executor);
     if (Error E = NewMeasurements.takeError()) {
       if (!E.isA<SnippetCrash>())
         return std::move(E);
       InstrBenchmark.Error = toString(std::move(E));
       return InstrBenchmark;
     }
     assert(InstrBenchmark.NumRepetitions > 0 && "invalid NumRepetitions");
     for (BenchmarkMeasure &BM : *NewMeasurements) {

llvm/tools/llvm-exegesis/lib/PerfHelper.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Error.h"
+#include <cstdint>
 #include <functional>
 #include <memory>
 
 struct perf_event_attr;
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 // Returns true on error.
 bool pfmInitialize();
 void pfmTerminate();
 
 // Retrieves the encoding for the event described by pfm_event_string.
 // NOTE: pfm_initialize() must be called before creating PerfEvent objects.
 class PerfEvent {
 public:
   // http://perfmon2.sourceforge.net/manv4/libpfm.html
   // Events are expressed as strings. e.g. "INSTRUCTION_RETIRED"
-  explicit PerfEvent(StringRef pfm_event_string);
+  explicit PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod = 0);
 
   PerfEvent(const PerfEvent &) = delete;
   PerfEvent(PerfEvent &&other);
   ~PerfEvent();
 
   // The pfm_event_string passed at construction time.
   StringRef name() const;
 
   // Whether the event was successfully created.
   bool valid() const;
 
   // The encoded event to be passed to the Kernel.
   const perf_event_attr *attribute() const;
 
   // The fully qualified name for the event.
   // e.g. "snb_ep::INSTRUCTION_RETIRED:e=0:i=0:c=0:t=0:u=1:k=0:mg=0:mh=1"
   StringRef getPfmEventString() const;
 
 private:
+  void initPerfEvent();
+  void initPerfEventForLbr(unsigned SamplingPeriod);
+
   const std::string EventString;
   std::string FullQualifiedEventString;
   perf_event_attr *Attr;
 };
 
 // Uses a valid PerfEvent to configure the Kernel so we can measure the
 // underlying event.
-struct Counter {
+class Counter {
+public:
   // event: the PerfEvent to measure.
   explicit Counter(PerfEvent &&event);
 
   Counter(const Counter &) = delete;
   Counter(Counter &&other) = default;
 
-  ~Counter();
+  virtual ~Counter();
 
-  void start();         // Starts the measurement of the event.
-  void stop();          // Stops the measurement of the event.
-  int64_t read() const; // Return the current value of the counter.
+  /// Starts the measurement of the event.
+  virtual void start();
 
-private:
-  PerfEvent Event;
+  /// Stops the measurement of the event.
+  void stop();
+
+  /// Returns the current value of the counter.
+  virtual int64_t read() const;
+
+  /// Returns the current value of the counter or error if it cannot be read.
+  virtual llvm::Expected<int64_t> readOrError() const { return read(); }
+
+protected:
 #ifdef HAVE_LIBPFM
   int FileDescriptor = -1;
-#endif
+#endif // HAVE_LIBPFM
+
+private:
+  PerfEvent Event;
 };
 
 } // namespace pfm
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

 //===-- PerfHelper.cpp ------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "PerfHelper.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/raw_ostream.h"
+#include <memory>
 #ifdef HAVE_LIBPFM
 #include "perfmon/perf_event.h"
 #include "perfmon/pfmlib.h"
 #include "perfmon/pfmlib_perf_event.h"
 #endif
+#include <atomic>
 #include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 #ifdef HAVE_LIBPFM
 static bool isPfmError(int Code) { return Code != PFM_SUCCESS; }
 #endif

 
 PerfEvent::PerfEvent(PerfEvent &&Other)
     : EventString(std::move(Other.EventString)),
       FullQualifiedEventString(std::move(Other.FullQualifiedEventString)),
       Attr(Other.Attr) {
   Other.Attr = nullptr;
 }
 
-PerfEvent::PerfEvent(StringRef PfmEventString)
-    : EventString(PfmEventString.str()), Attr(nullptr) {
+void PerfEvent::initPerfEvent() {
 #ifdef HAVE_LIBPFM
   char *Fstr = nullptr;
   pfm_perf_encode_arg_t Arg = {};
   Attr = new perf_event_attr();
   Arg.attr = Attr;
   Arg.fstr = &Fstr;
   Arg.size = sizeof(pfm_perf_encode_arg_t);
   const int Result = pfm_get_os_event_encoding(EventString.c_str(), PFM_PLM3,
                                                PFM_OS_PERF_EVENT, &Arg);
   if (isPfmError(Result)) {
     // We don't know beforehand which counters are available (e.g. 6 uops ports
     // on Sandybridge but 8 on Haswell) so we report the missing counter without
     // crashing.
     errs() << pfm_strerror(Result) << " - cannot create event " << EventString
            << "\n";
   }
   if (Fstr) {
     FullQualifiedEventString = Fstr;
     free(Fstr);
   }
 #endif
 }
 
+void PerfEvent::initPerfEventForLbr(unsigned SamplingPeriod) {
+#ifdef HAVE_LIBPFM
+  assert(SamplingPeriod > 0 && "SamplingPeriod must be positive");
+  Attr = new perf_event_attr();
+  *Attr = {0};
+  Attr->size = sizeof(*Attr);
+  Attr->type = PERF_TYPE_HARDWARE;
+  // Look up encodings at
+  // https://download.01.org/perfmon/SKL/skylake_core_v48.json
+  Attr->config = 0x20c4; // BR_INST_RETIRED.NEAR_TAKEN
+  Attr->sample_type = PERF_SAMPLE_BRANCH_STACK;
+  // Don't need to specify "USER" because we've already excluded HV and Kernel.
+  Attr->branch_sample_type = PERF_SAMPLE_BRANCH_ANY;
+  Attr->sample_period = SamplingPeriod;
+  Attr->wakeup_events = 1; // We need this even when using ioctl REFRESH.
+  Attr->disabled = 1;
+  Attr->exclude_kernel = 1;
+  Attr->exclude_hv = 1;
+  Attr->read_format = PERF_FORMAT_GROUP;
+
+  FullQualifiedEventString = "LBR";
+#endif
+}
+
+PerfEvent::PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod)
+    : EventString(PfmEventString.str()), Attr(nullptr) {
+  if (EventString == "LBR") {
+    initPerfEventForLbr(SamplingPeriod);
+  } else {
+    initPerfEvent();
+  }
+}
+
 StringRef PerfEvent::name() const { return EventString; }
 
 bool PerfEvent::valid() const { return !FullQualifiedEventString.empty(); }
 
 const perf_event_attr *PerfEvent::attribute() const { return Attr; }
 
 StringRef PerfEvent::getPfmEventString() const {
   return FullQualifiedEventString;

llvm/tools/llvm-exegesis/lib/Target.h

 #include "LlvmState.h"
 #include "SnippetGenerator.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/MC/MCRegisterInfo.h"
+#include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace exegesis {
 
 struct PfmCountersInfo {
   // An optional name of a performance counter that can be used to measure
   // cycles.
   const char *CycleCounter;
struct CpuAndPfmCounters {
   bool operator<(StringRef S) const { return StringRef(CpuName) < S; }
 };
 
 class ExegesisTarget {
 public:
   explicit ExegesisTarget(ArrayRef<CpuAndPfmCounters> CpuPfmCounters)
       : CpuPfmCounters(CpuPfmCounters) {}
 
+  /// Returns true if the benchmark should use a worker-thread.
+  /// For most target/benchmark-mode, it is not necessary.
+  virtual bool useWorkerThreadForBenchmark(const LLVMState &,
+                                           InstructionBenchmark::ModeE) const {
+    return false;
+  }
+
+  virtual Expected<std::unique_ptr<pfm::Counter>>
+  createCounter(const char *CounterName, const LLVMState &State) const;
+
   // Targets can use this to add target-specific passes in assembleToStream();
   virtual void addTargetSpecificPasses(PassManagerBase &PM) const {}
 
   // Generates code to move a constant into a the given register.
   // Precondition: Value must fit into Reg.
   virtual std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                        const APInt &Value) const = 0;
 
private:
   std::unique_ptr<SnippetGenerator> virtual createSerialSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<SnippetGenerator> virtual createParallelSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<BenchmarkRunner> virtual createLatencyBenchmarkRunner(
       const LLVMState &State, InstructionBenchmark::ModeE Mode) const;
   std::unique_ptr<BenchmarkRunner> virtual createUopsBenchmarkRunner(
       const LLVMState &State) const;
 
   const ExegesisTarget *Next = nullptr;
   const ArrayRef<CpuAndPfmCounters> CpuPfmCounters;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H

llvm/tools/llvm-exegesis/lib/Target.cpp

 const ExegesisTarget *ExegesisTarget::lookup(Triple TT) {
   for (const ExegesisTarget *T = FirstTarget; T != nullptr; T = T->Next) {
     if (T->matchesArch(TT.getArch()))
       return T;
   }
   return nullptr;
 }
 
+Expected<std::unique_ptr<pfm::Counter>>
+ExegesisTarget::createCounter(const char *CounterName,
+                              const LLVMState &) const {
+  return std::make_unique<pfm::Counter>(pfm::PerfEvent(CounterName));
+}
+
 void ExegesisTarget::registerTarget(ExegesisTarget *Target) {
   if (FirstTarget == nullptr) {
     FirstTarget = Target;
     return;
   }
   if (Target->Next != nullptr)
     return; // Already registered.
   Target->Next = FirstTarget;

 }
 
 std::unique_ptr<BenchmarkRunner>
 ExegesisTarget::createUopsBenchmarkRunner(const LLVMState &State) const {
   return std::make_unique<UopsBenchmarkRunner>(State);
 }
 
 static_assert(std::is_pod<PfmCountersInfo>::value,
               "We shouldn't have dynamic initialization here");
 const PfmCountersInfo PfmCountersInfo::Default = {nullptr, nullptr, nullptr,
                                                   0u};
 
 const PfmCountersInfo &ExegesisTarget::getPfmCounters(StringRef CpuName) const {
   assert(llvm::is_sorted(
              CpuPfmCounters,
              [](const CpuAndPfmCounters &LHS, const CpuAndPfmCounters &RHS) {
                return strcmp(LHS.CpuName, RHS.CpuName) < 0;

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

 //===-- Target.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "../Target.h"
 
 #include "../Error.h"
 #include "../ParallelSnippetGenerator.h"
 #include "../SerialSnippetGenerator.h"
 #include "../SnippetGenerator.h"
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "X86.h"
+#include "X86Counter.h"
 #include "X86RegisterInfo.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/Sequence.h"
 #include "llvm/MC/MCInstBuilder.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FormatVariadic.h"
 
+#include <memory>
+#include <string>
+#include <vector>
+
 namespace llvm {
 namespace exegesis {
 
+static cl::OptionCategory BenchmarkOptions("llvm-exegesis benchmark x86-options");
+
+// If a positive value is specified, we are going to use the LBR in
+// latency-mode.
+//
+// Note:
+//  -  A small value is preferred, but too low a value could result in
+//     throttling.
+//  -  A prime number is preferred to avoid always skipping certain blocks.
+//
+static cl::opt<unsigned> LbrSamplingPeriod(
+    "x86-lbr-sample-period",
+    cl::desc("The sample period (nbranches/sample), used for LBR sampling"),
+    cl::cat(BenchmarkOptions), cl::init(0));
+
 // Returns a non-null reason if we cannot handle the memory references in this
 // instruction.
 static const char *isInvalidMemoryInstr(const Instruction &Instr) {
   switch (Instr.Description.TSFlags & X86II::FormMask) {
   default:
     llvm_unreachable("Unknown FormMask value");
   // These have no memory access.
   case X86II::Pseudo:
if (Bytes - ByteOffset >= 1)
     add(fillStackSpace(
         X86::MOV8mi, ByteOffset,
         WideConstant.extractBits(8, ByteOffset * 8).getZExtValue()));
 }
 
 #include "X86GenExegesis.inc"
 
 namespace {
+
 class ExegesisX86Target : public ExegesisTarget {
 public:
   ExegesisX86Target() : ExegesisTarget(X86CpuPfmCounters) {}
 
+  bool
+  useWorkerThreadForBenchmark(const LLVMState &State,
+                              InstructionBenchmark::ModeE Mode) const override {
+    // If it is Latency with LBR, should use worker-thread
+    return Mode == InstructionBenchmark::ModeE::Latency &&
+           LbrSamplingPeriod > 0;
+  }
+
+  Expected<std::unique_ptr<pfm::Counter>>
+  createCounter(const char *CounterName,
+                const LLVMState &State) const override {
+    if (CounterName == State.getPfmCounters().CycleCounter) {
+      // Can't use LBR without HAVE_LIBPFM, or __linux__ (for now)
+#if defined(HAVE_LIBPFM) && defined(__linux__)
+      return std::make_unique<X86LbrCounter>(
+          PerfEvent("LBR", LbrSamplingPeriod));
+#else
+      return llvm::make_error<llvm::StringError>(
+          "LBR counter requested without HAVE_LIBPFM or running on Linux.",
+          llvm::errc::invalid_argument);
+#endif
+    }
+    return ExegesisTarget::createCounter(CounterName, State);
+  }
+
 private:
   void addTargetSpecificPasses(PassManagerBase &PM) const override;
 
   unsigned getScratchMemoryRegister(const Triple &TT) const override;
 
   unsigned getLoopCounterRegister(const Triple &) const override;
 
   unsigned getMaxMemoryAccessSize() const override { return 64; }
 
   Error randomizeTargetMCOperand(const Instruction &Instr, const Variable &Var,
                                  MCOperand &AssignedValue,
                                  const BitVector &ForbiddenRegs) const override;
 
   void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                           unsigned Offset) const override;

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

+//===-- X86Counter.h --------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H
+#define LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H
+
+#include "../PerfHelper.h"
+#include "llvm/Support/Error.h"
+
+// FIXME: Use appropriate wrappers for poll.h and mman.h
+// to support Windows and remove this linux-only guard.
+#if defined(__linux__) && defined(HAVE_LIBPFM)
+
+namespace llvm {
+namespace exegesis {
+
+class X86LbrCounter : public pfm::Counter {
+public:
+  explicit X86LbrCounter(const pfm::PerfEvent &Event);
+
+  virtual ~X86LbrCounter();
+
+  void start() override;
+  int64_t read() const override;
+  llvm::Expected<int64_t> readOrError() const override;
+
+private:
+  void *MMappedBuffer = nullptr;
+};
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif // defined(__linux__) && defined(HAVE_LIBPFM)
+
+#endif // LLVM_TOOLS_LLVM_EXEGESIS_X86COUNTER_H

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

+//===-- X86Counter.cpp ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86Counter.h"
+
+// FIXME: Use appropriate wrappers for poll.h and mman.h
+// to support Windows and remove this linux-only guard.
+#ifdef __linux__
+#include "llvm/Support/Errc.h"
+
+#ifdef HAVE_LIBPFM
+#include "perfmon/perf_event.h"
+#include "perfmon/pfmlib.h"
+#include "perfmon/pfmlib_perf_event.h"
+#endif // HAVE_LIBPFM
+
+#include <atomic>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <vector>
+
+#ifdef HAVE_LIBPFM
+namespace llvm {
+namespace exegesis {
+
+static constexpr int KBufferPages = 8;
+static const int kDataBufferSize = kBufferPages * getpagesize();
+
+// Waits for the LBR perf events.
+static int pollLbrPerfEvent(const int FileDescriptor) {
+  struct pollfd PollFd;
+  PollFd.fd = FileDescriptor;
+  PollFd.events = POLLIN;
+  PollFd.revents = 0;
+  return poll(&PollFd, 1 /* num of fds */, 10000 /* timeout in ms */);
+}
+
+// Copies the data-buffer into Buf, given the pointer to MMapped.
+static void copyDataBuffer(void *MMappedBuffer, char *Buf, uint64_t Tail,
+                           size_t DataSize) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page.
+  char *Start = reinterpret_cast<char *>(MMappedBuffer) + getpagesize();
+  // The LBR buffer is a cyclic buffer, we copy data to another buffer.
+  uint64_t Offset = Tail % kDataBufferSize;
+  size_t CopySize = kDataBufferSize - Offset;
+  memcpy(Buf, Start + Offset, CopySize);
+  if (CopySize >= DataSize)
+    return;
+  memcpy(Buf + CopySize, Start, Offset);
+}
+
+// FIXME: Remove this hack.
+// Option #1: Make the assembler produce code with a condition var for stopping
+// the loop, which can be found statically (at fixed addr) and remove this
+// hack of modifying the code while running.
+// Option #2: Run benchmarked code in the same thread, and make use of
+// Freeze_LBRs_On_PMI
+//
+// Rewrites the jmp (from the BM loop) with `pop $rbx; ret` to get
+// out of the benchmarked code.
+//
+// pc: Must point at the start of the jmp instruction.
+static void patchBasicBlockToEndBenchmarkedLoop(char *pc) const {
+  void *page = reinterpret_cast<void *>(reinterpret_cast<uintptr_t>(pc) &
+                                        ~(getpagesize() - 1));
+  mprotect(page, getpagesize(), PROT_READ | PROT_WRITE | PROT_EXEC);
+  if ((reinterpret_cast<uint64_t>(pc) & 0xf) != 0xf) {
+    // We have at least two bytes in the same cacheline, so we can use an
+    // atomic write to replace the jmp with pop, ret.
+    *reinterpret_cast<uint16_t *>(pc) = 0xc35b; // pop %rbx; ret;
+  } else {
+    // pc and pc+1 might be at different cacheline, so we first add the pop,
+    // ret instructions, and then update jump offset to jump to pop
+    // instruction.
+    *reinterpret_cast<uint16_t *>(pc + 5) = 0xc35b; // pop %rbx; ret;
+    // An automic update on the displacement.
+    if (pc[0] == 0xeb /* x86 short jump */) {
+      // offset is 1 byte
+      pc[1] = 0x03;
+    } else { /* x86 near jump */
+      // offset is 4 bytes
+      *reinterpret_cast<uint32_t *>(pc + 1) = 0;
+    }
+  }
+}
+
+// Parses the given data-buffer for stats and fill the CycleArray.
+// If data has been extracted successfully, also modifies the code to jump
+// out the benchmark loop.
+static llvm::Error parseDataBuffer(const char const *DataBuf, size_t DataSize,
+                                   std::vector<int64_t> *CycleArray) const {
+  const char *DataPtr = DataBuf;
+  while (DataPtr < DataBuf + DataSize) {
+    struct perf_event_header Header;
+    memcpy(&Header, DataPtr, sizeof(struct perf_event_header));
+    if (Header.type != PERF_RECORD_SAMPLE) {
+      // Ignores non-sample records.
+      DataPtr += Header.size;
+      continue;
+    }
+    DataPtr += sizeof(Header);
+    uint64_t Count = llvm::support::endian::read64(DataPtr, support::native);
+    DataPtr += sizeof(Count);
+
+    struct perf_branch_entry Entry;
+    memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    // Read the perf_branch_entry array.
+    char *JumpPC = reinterpret_cast<char *>(Entry.from);
+    int64_t MinCycle = Entry.cycles;
+    for (int i = 0; i < Count; ++i) {
+      // We use the JumpPC from the entry with min cycle to avoid the entry
+      // that is returning from kernel.
+      if (MinCycle > Entry.cycles) {
+        MinCycle = Entry.cycles;
+        JumpPC = reinterpret_cast<char *>(Entry.from);
+      }
+      CycleArray->push_back(Entry.cycles);
+      if (i == Count - 1) {
+        // If we're at the last entry, then terminate the loop.
+        patchBasicBlockToEndBenchmarkedLoop(JumpPC);
+        return llvm::Error::success();
+      }
+      // Advance to next entry
+      DataPtr += sizeof(Entry);
+      memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    }
+  }
+  return llvm::make_error<llvm::StringError>("Unable to read databuffer.",
+                                             llvm::errc::io_error);
+}
+
+X86LbrCounter::X86LbrCounter(const pfm::PerfEvent &Event) : Counter(Event) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page, so we allocate one more page.
+  MMappedBuffer = mmap(nullptr, (kBufferPages + 1) * getpagesize(),
+                       PROT_READ | PROT_WRITE, MAP_SHARED, FileDescriptor, 0);
+  if (MMappedBuffer == MAP_FAILED) {
+    llvm::errs() << "Failed to mmap buffer.";
+  }
+}
+
+X86LbrCounter::~X86LbrCounter() { close(FileDescriptor); }
+
+void X86LbrCounter::start() override {
+  ioctl(FileDescriptor, PERF_EVENT_IOC_REFRESH, 1024 /* kMaxPollsPerFd */);
+}
+
+int64_t X86LbrCounter::read() const override {
+  auto error = readOrError();
+  if (!error) {
+    return error.get();
+  }
+  llvm::errs() << "Error reading counter: " << error.takeError() << "\n";
+  return 0;
+}
+
+static bool CheckLbrFormats(const struct perf_event_mmap_page &Page) {
+  static const uint64_t mask = 0x0000003C;
+  // We are looking for the format that contains "cycles".
+  // (Intel SDM, vol 3B 17.4.8.1)
+  // "MSR IA32_PERF_CAPABILITIES[5:0]" == 000110B
+
+  // perf_event_mmap_page::capabilities is a union of {unit64_t,
+  // or a struct with 64 bits}. But you're allowed to read common prefix
+  // regardless of which union member is active.
+  // So it should be safe to just access the `unit64_t capabilitites`
+  // member.
+  return (Page.capabilities & mask) == 0x00000006;
+}
+
+llvm::Expected<int64_t> X86LbrCounter::readOrError() const override {
+  // The max number of time-outs/retries before we give up.
+  static constexpr int kMaxTimeouts = 160;
+
+  // Parses the LBR buffer and fills CycleArray with the sequence of cycle
+  // counts from the buffer.
+  std::vector<int64_t> CycleArray;
+  std::unique_ptr<char[]> DataBuf(new char[kDataBufferSize]);
+  int NumTimeouts = 0;
+  int PollResult = 0;
+  while (PollResult <= 0) {
+    PollResult = pollLbrPerfEvent(FileDescriptor);
+    if (PollResult == -1) {
+      return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.",
+                                                 llvm::errc::io_error);
+    } else if (PollResult == 0) {
+      llvm::errs() << "LBR polling timed out without result, NumTimeouts ="
+                   << NumTimeouts << ". ";
+      if (NumTimeouts < kMaxTimeouts) {
+        llvm::errs() << "Retrying ...\n";
+        ++NumTimeouts;
+        continue;
+      } else {
+        llvm::errs() << "At max-timeouts. Giving up.\n";
+        return llvm::make_error<llvm::StringError>(
+            "LBR polling still timed out after max number of attempts.",
+            llvm::errc::device_or_resource_busy);
+      }
+    }
+  }
+
+  struct perf_event_mmap_page Page;
+  memcpy(&Page, MMappedBuffer, sizeof(struct perf_event_mmap_page));
+  // Check to see if the LBR format is expected.
+  // FIXME: It would be nicer if we could detect the format way earlier.
+  if (!checkLbrFormat(Page)) {
+    return llvm::make_error<llvm::StringError>("Unexpected LBR format",
+                                               llvm::errc::not_supported);
+  }
+
+  const uint64_t DataTail = Page.data_tail;
+  const uint64_t DataHead = Page.data_head;
+  // We're supposed to use a barrier after reading data_head.
+  std::atomic_thread_fence(std::memory_order_acq_rel);
+  const size_t DataSize = DataHead - DataTail;
+  if (DataSize > kDataBufferSize) {
+    return llvm::make_error<llvm::StringError>(
+        "DataSize larger than buffer size.", llvm::errc::invalid_argument);
+  }
+  copyDataBuffer(MMappedBuffer, DataBuf.get(), DataTail, DataSize);
+  llvm::Error error = parseDataBuffer(DataBuf.get(), DataSize, &CycleArray);
+  if (!error) {
+    // FIXME We should report the cycles count for all jumps,
+    // not just the most recent.
+    return CycleArray[0];
+  }
+
+  return error;
+}
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif // HAVE_LIBPFM
+#endif // __linux__