[llvm-exegesis] Add benchmark mode that uses LBR for more precise measurements.

[oontvoo]

Summary

Add benchmark mode that uses LBR for more precise measurements.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

86

Let's move this to the X86 target, and rename it to x86-lbr-sample-period, with a default of 0. If non-zero, we're going to use LBR. Else, we'll use the non-LBR runner.

[courbet]

And thanks for the contribution.

[oontvoo]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

109

Yes, it would've ... Changed to Counter*

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

90

Yes - it should be macro guarded. (I mindlessly deleted the guards while moving things around)
Added it back.

[oontvoo]

Addressed review comments:

• remove the new mode and merge it into latency
• various refactorings and comments-cleanup per courbet@'s requests

[oontvoo]

Fix typos and rebase

[oontvoo]

Fix clang-tidy warning

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

Can you expand on what "hiding blocks" means ? e.g. an external reference ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

90

Apart from the PerfEvent, this does not really have anything to do with LBR.
On the one hand, this is not really a full-fledged FunctionExecutor, as it just throws away the CounterName argument and always measures the latency. This would be an argument for moving this to the X86 backend.
On the other hand, this has a lot of target-independent code, so it's a good thing that other targets can use it. The main issue, and the only thing that's really specific to LBR here is the perf event. Letting the target create the perf event as well as the counter does fix this.

So what we can do:

• change Target::createCounter to take a counter name rather than a PerfEvent. The default implementation creates the PerfEvent, validates it and wraps it in a Counter. The X86 implementation tests if we're asking for the cycles counter in LBR mode, and creates the relevant PerfEvent and LBRCounter. It falls back to the default implementation else.
• We stop ignoring the counter name argument in LbrFunctionExecutorImpl::runAndMeasure, and just propagate it to the ExegesisTarget.
• We rename LbrFunctionExecutorImpl into WorkerThreadFunctionExecutorImpl (and FunctionExecutorImpl could be renamed to SameThreadFunctionExecutor, and everything becomes target agnostic.

198

Again, this is too specific to X86. I think it's one of these cases where is actually makes more sense for the generic code to know nothing about LBR. What about:

• Moving LbrSamplePeriod from llvm-exegesis to X86/Target.cpp
• Making createFunctionExecutor() a virtual member of the target. The default implementation returns a FunctionExecutorImpl.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

50

The LLVM style is FIXME

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

32

do we really need all these ?

594

[nit] s/PFM package available../HAVE_LIBPFM/ the package might be available but specifically disabled

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

16

Does this work under windows ? If not, please use one of the LLVM wrappers.

29

internal functions use a static keyword instead of anonymous namespace in the LLVM style guide.

34

s/time out/timeout

38

static

52

This would benefit from a pointer to documentation (or argued speculation about) whether rewriting code inside a loop for another thread is valid and how it behaves w.r.t. caches and DSB.

56

static

83

static

125

you can remove this now

197

FIXME. Also, please give more explanations so that somebody else can look at that too ;)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

1

This file, and the cpp one, are missing the LLVM headers.

[oontvoo]

• Make internal helpers static functions (instead of anonymous ns members)
• Reword confusing comments
• Add LLVM headers

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

210

typo: ..

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

107

std::unique_ptr + no dtor ?

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

50

FIXME and no name :)

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

199

The LBR on Haswell has only the addresses of the branches. To get the timing, Skylake is the minimal requirement.

207

I'd add a mention that this applies to the LBR ratency mode.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

186

As noted above - Haswell and Broadwell have the LBR, but it does not report the number of cycles between the branches.

[oontvoo]

Addressed review comments.

[oontvoo]

Add more comment

[oontvoo]

Thanks for the thorough reviews! :-)

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

90

Done - instead of moving the executor out, I've just let the Target decides whether to use WorkerThread executor or otherwise.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

16

Per offline discussion, we'll ifdef this out for non-linux and will add support for windows in follow-up CLs

52

I meant to rewrite this hack to use a condition var for stopping the loop from another thread, per Ondrej's suggestion .
I just haven't got to it yet :-) ... (would need more changes in the assembler.cpp )

So I'll add a FIXME for next CLs

197

Per offline discussions, this requires re-working the read() interface because we need to return more than one value.
(Specifically, the lbr-entries offer a list of cycles-latencies for a bunch of consec branches ... we don't want to just ignore the older branches and only look at the most recent)

But because this patch is already quite big , we'll keep this as TODO and follow up.

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

90

That forces passing the RunArg to the target, which has no reason to know about it.

What I'm suggesting is the following:

• move LbrSamplePeriod to X86/Target.cpp. This whole SamplePeriod thing, and the existence of the option is really just a X86 detail, the target-independent code has no need to know about any of this.
• Get rid of SamplePeriod, in the target-independant code. RunArg can go too.
• createCounter does not need Description, or SamplePeriod. The X86 target handles it internally.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

615

This should really onely be:

Expected<std::unique_ptr<pfm::Counter>>
createCounter(const char *CounterName) {
  if (CounterName == kCyclesCounter && LbrSamplingPeriod > 0) {
    return std::make_unique<X86LbrCounter>(PerfEvent("LBR", LbrSamplingPeriod));
  }
  return ExegesisTarget::createCounter(CounterName);
}

617

This will fail to compile under windows because X86LbrCounter does not exist there.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

349

This compile-time guarded runtime check can now go, because the x86 specific concepts are handled in the x86 target.

[oontvoo]

Addressed review comments:

• remove no-longer needed RunArg
• move target-specific to X86/Target

[oontvoo]

Add check for the LBR format to ensure it contains cycles.

[oontvoo]

Update perf_event_attr to use BR_INST_RETIRED.NEAR_TAKEN per eranian@'s recommendation.

[oontvoo]

ran clang-format

[oontvoo]

revert unrelated change

[oontvoo]

Fix dup options

[oontvoo]

Add new cpp file to makefile

[oontvoo]

Update perf-even-attr for raw-encoding

[oontvoo]

Updated patch to cover what we discussed in the sync last week and this week, specifically:

• Remove the worker-thread executor (not needed. It's fine to use the same thread)
• Make PerfEvent virtual so the X86Counter can create its own PerfEvent with LBR-specific attributes.

Remaining questions:

• Counter should be able to return more than one readings.
• We should validate that repetion-mode=loop for LBR

[courbet]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

On X86,

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

13

you're no longer using <thread> and <mutex>.

16

is this needed ?

72

s/auto/pfm::Counter/

llvm/tools/llvm-exegesis/lib/PerfHelper.h

45

why is the dtor virtual ? I don't see any virtual methods ?

llvm/tools/llvm-exegesis/lib/Target.h

69

Can you submit this change independently and rebase ? Thanks !

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

589

&& LbrSamplingPeriod > 0 ?

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

65

fills

184

[style]

if (PollResult == -1)
  return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.", llvm::errc::io_error);
if (PollResult != 0) 
  continue;
...

185

how often does that happen ? Do we want that much logging ?

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

69

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/Target.cpp

36 ↗

(On Diff #265315)

[style] no braces for 1-line conditional bodies

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

[style] no braces for 1-line conditional bodies

131

[style] no braces for 1-line conditional bodies

144

[style] no braces for 1-line conditional bodies

206

[style] no braces for 1-line conditional bodies

216

ditto

223

ditto

[oontvoo]

Addressed review comments

[oontvoo]

llvm/tools/llvm-exegesis/lib/Target.h

69

Done https://reviews.llvm.org/D80610

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

185

It could happen 100% of the time if the code happens to have no branches.
With "proper" input, it happens less frequently. I don't really have a sense of how often ... it's probably be fine to change this to llvm::debug() or something?

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

184

you don't need the else after the return.

185

I'm not worried about the no-branch case as we can avoid that. We might as well remove it, because we don't really care if it fanally succeeded, and if it did not we can print an error there.

190

you don't need the else here after the continue.

[oontvoo]

Rebase to reduce diff

[oontvoo]

Simplified re-try

[courbet]

OK, I think this is in good shape now, but we still need a test for it. The only issue is that not all buildbots support LBR, so we need a mechanism for skipping the test when not available. You can create a subdir llvm-project/llvm/test/tools/llvm-exegesis/X86/lbr with a custom lit similar to llvm-project/llvm/test/tools/llvm-exegesis/X86/lit.local.cfg to check lbr support before running the test.

[oontvoo]

Add test and option to check for expected CPU.

I can't seem to find a way to get the host CPU version (much less name or pmu capabilities) from the lit config, so I had to add the check to llvm-exegesis.

[oontvoo]

Rebase

[oontvoo]

This comment was removed by oontvoo.

[courbet]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

299

This is a bit brittle, because we could imagine the name of some unrelated CPUs being substrings of others. What about having a repeated option --allowed-host-cpu=skylake --allowed-host-cpu=skylake-avx512 --allowed-host-cpu=whateverlake, and check that the exact value is one of these ?

[ondrasej]

llvm/docs/CommandGuide/llvm-exegesis.rst

197

This is more of a philosophical comment: IIRC, the LBR records the times when the branch instructions are retired. With correct branch prediction, the instructions in the measured block would start executing speculatively before the branch is retired, and so the number we get is probably closer to the throughput.

211

I'm wondering how this applies to this use case. The issue with basic blocks skipped might be when sampling all basic blocks in a larger application over a longer period of time, but this is not our case.
In our case, we're interested in a single basic block, and run it in a tight loop for relatively long, so that the LBR sampling catches this exact basic block.

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

88

I think we also need to filter by the addresses of the branches (or store them) to make sure that we take the cycles only for the measured basic block, not for the other basic blocks surrounding it.

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

299

Ideally, this should be defined in terms of CPU features (e.g. CPUID bits). Even better - each target should know which counters it supports, based on its platform-specific feature discovery mechanism. I understand that this would be a huge change for this CL, but we should at least have a FIXME here.

[oontvoo]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

299

Actually, I had a question here that I was gonna ask in an email But here goes.

We know that the LBR formats could be queried from the perf-cap MSR
Specifically, we want: MSR IA32_PERF_CAPABILITIES[5:0]" == 000110B (bit 59...63 is not relevant)

If I'm not mistaken, perf_event_mmap_page::capabilities should(?) give us that.
Except, when I run this on both Broadwell and Skylake, the capabilities field has value of 30 for both platforms. (It shouldn't be). Of course, the difference here is that the cycle entries are all zeroes on Broadwell .
I haven't looked in details, so I don't know if this is some implementation detail of the libpfm or if we're just mis-interpreting the SDM here.

Thoughts?

[ondrasej]

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

299

This is strange.

I understand the SDM section about IA32_PERF_CAPABILITIES the same way, i.e. we should see 000110B if there are timings, and this should happen only on Skylake or onwards.

That said, I don't think that perf_event_mmap_page::capabilities contains the IA32_PERF_CAPABILITIES MSR. Instead, it looks like a struct that is defined and filled by the kernel. Check out the definition/docs of the struct (and the code that fills it). They set individual bit fields (and even the bit field defined in the union does not match the other bits defined in the MSR).

After some digging through kernel sources, I see where the kernel keeps the data internally, but I didn't see a way to get it out of the kernel. We might need to do more research on that.

[oontvoo]

Filter and keep only records for the benchmarked codeblock

[oontvoo]

Add option to not filter the records too

[oontvoo]

Updated diff

[oontvoo]

clang-tidy

[courbet]

Only style comments left

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

81

let's use a c++ cast here.

83

Actually I'm thinking you can just pass a StringRef to readOrError

llvm/tools/llvm-exegesis/lib/PerfHelper.h

83–84

Please explain what the arguments are (and switch to StringRef as mentioned above ?)

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

150

braces

205

style: no braces

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

9

This could use a file comment with a link to LBR documentation.

[oontvoo]

updated diff

[courbet]

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

179

Let's return an error when FunctionBytes.empty(). Right now this will assert later.

[oontvoo]

Return error for empty function

[ondrasej]

Looks good overall.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

53–54

[style] Remove the empty line.

[oontvoo]

Removed empty line

[oontvoo]

Fixed warning [-Wmissing-field-initializers] in code

[erichkeane]

This patch doesn't seem to build for me:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp: In function ‘llvm::Error llvm::exegesis::parseDataBuffer(const char*, size_t, const void*, const void*, llvm::SmallVector<long int, 4>*)’:
/iusers/ekeane1/workspaces/llvm-project/llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp:99:37: error: ‘struct perf_branch_entry’ has no member named ‘cycles’

CycleArray->push_back(Entry.cycles);

I'm on RHEL7, so I have kernel 3.10, so it doesn't have 'cycles'.

According ot this: https://elixir.bootlin.com/linux/v4.3/source/include/uapi/linux/perf_event.h#L963 kernel 4.3 is the first time that 'cycles' appeared in this structure.

[courbet]

If you're not using llvm-exegesis you can disable this with the LLVM_ENABLE_LIBPFM cmake option.
But I'll revert this until @oontvoo can fix llvm/cmake/modules/FindLibpfm.cmake

[oontvoo]

Check for existence of cycles before enabling LBR mode.

[gchatelet]

LGTM

[oontvoo]

Relanded.
https://reviews.llvm.org/rGee7caa75939afb75547c00744c5df4d04d45e517

llvm/docs/CommandGuide/llvm-exegesis.rst

 
 .. option:: -snippets-file=<filename>
 
  Specify the custom code snippet to measure. See example 2 for details.
  Either `opcode-index`, `opcode-name` or `snippets-file` must be set.
 
 .. option:: -mode=[latency|uops|inverse_throughput|analysis]
 
- Specify the run mode. Note that if you pick `analysis` mode, you also need
- to specify at least one of the `-analysis-clusters-output-file=` and
- `-analysis-inconsistencies-output-file=`.
+ Specify the run mode. Note that some modes have additional requirements and options.
+
+ `latency` mode can be  make use of either RDTSC or LBR.
+   `latency[LBR]` is only available on X86 (at least `Haswell`, but preferably
+   `Skylake` for more precise measurements). To run in this mode, a positive value
+   must be specified for `x86-lbr-sample-period`
+
+ In `analysis` mode, you also need to specify at least one of the
+ `-analysis-clusters-output-file=` and `-analysis-inconsistencies-output-file=`.
+
+.. option:: -x86-lbr-sample-period=<nBranches/sample>
+
+  Specify the sampling period - how many branches before we take a sample.
+  On choosing the "right" sampling period, a small value is preferred, but throttling
+  could occur if the sampling is too frequent. A prime number should be used to
+  avoid hiding blocks.
+  With various testing/experiments, `521` seems like the best value so far.
   
 .. option:: -repetition-mode=[duplicate|loop|min]
 
  Specify the repetition mode. `duplicate` will create a large, straight line
  basic block with `num-repetitions` copies of the snippet. `loop` will wrap
  the snippet in a loop which will be run `num-repetitions` times. The `loop`
  mode tends to better hide the effects of the CPU frontend on architectures
  that cache decoded instructions, but consumes a register for counting
  iterations. If performing an analysis over many opcodes, it may be best

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 
 #include "Assembler.h"
 #include "BenchmarkCode.h"
 #include "BenchmarkResult.h"
 #include "LlvmState.h"
 #include "MCInstrDescView.h"
+#include "PerfHelper.h"
 #include "SnippetRepetitor.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/Error.h"
 #include <cstdlib>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace exegesis {
 
 // Common code for all benchmark modes.
 class BenchmarkRunner {
 public:
+  // The argument to be passed to runConfiguration().
+  struct RunArg {
+    const BenchmarkCode &CodeSnippet;
+    unsigned NumRepetitions;
+    ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors;
+    bool DumpObjectToDisk;
+    unsigned LbrSamplePeriod;
+  };
   explicit BenchmarkRunner(const LLVMState &State,
                            InstructionBenchmark::ModeE Mode);
 
   virtual ~BenchmarkRunner();
 
+  // TODO(vyng) Make this "Deprecated" and switch caller to using RunArg
   Expected<InstructionBenchmark>
   runConfiguration(const BenchmarkCode &Configuration, unsigned NumRepetitions,
                    ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
                    bool DumpObjectToDisk) const;
 
+  Expected<InstructionBenchmark> runConfiguration(const RunArg &Arg) const;
+
   // Scratch space to run instructions that touch memory.
   struct ScratchSpace {
     static constexpr const size_t kAlignment = 1024;
     static constexpr const size_t kSize = 1 << 20; // 1MB.
     ScratchSpace()
         : UnalignedPtr(std::make_unique<char[]>(kSize + kAlignment)),
           AlignedPtr(
               UnalignedPtr.get() + kAlignment -
               (reinterpret_cast<intptr_t>(UnalignedPtr.get()) % kAlignment)) {}
     char *ptr() const { return AlignedPtr; }
     void clear() { std::memset(ptr(), 0, kSize); }
 
   private:
     const std::unique_ptr<char[]> UnalignedPtr;
     char *const AlignedPtr;
   };
 
   // A helper to measure counters while executing a function in a sandboxed
   // context.
   class FunctionExecutor {
   public:
     virtual ~FunctionExecutor();
     virtual Expected<int64_t> runAndMeasure(const char *Counters) const = 0;
+    // Optional description, if available.
+    virtual std::string description() const = 0;
   };
 
 protected:
   const LLVMState &State;
   const InstructionBenchmark::ModeE Mode;
 
 private:
   virtual Expected<std::vector<BenchmarkMeasure>>

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

 //===-- BenchmarkRunner.cpp -------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <array>
+#include <memory>
+#include <mutex>
 #include <string>
+#include <thread>
 
 #include "Assembler.h"
+#include "BenchmarkResult.h"
 #include "BenchmarkRunner.h"
 #include "Error.h"
 #include "MCInstrDescView.h"
 #include "PerfHelper.h"
+#include "Target.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 
 namespace llvm {
 namespace exegesis {
 
 BenchmarkRunner::BenchmarkRunner(const LLVMState &State,
                                  InstructionBenchmark::ModeE Mode)
     : State(State), Mode(Mode), Scratch(std::make_unique<ScratchSpace>()) {}
 
 BenchmarkRunner::~BenchmarkRunner() = default;
 
 namespace {
 class FunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
 public:
   FunctionExecutorImpl(const LLVMState &State,
                        object::OwningBinary<object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : Function(State.createTargetMachine(), std::move(Obj)),
         Scratch(Scratch) {}
 
+  std::string description() const override { return ""; }
+
 private:
   Expected<int64_t> runAndMeasure(const char *Counters) const override {
     // We sum counts when there are several counters for a single ProcRes
     // (e.g. P23 on SandyBridge).
     int64_t CounterValue = 0;
     SmallVector<StringRef, 2> CounterNames;
     StringRef(Counters).split(CounterNames, '+');
     char *const ScratchPtr = Scratch->ptr();
     for (auto &CounterName : CounterNames) {
       CounterName = CounterName.trim();
       pfm::PerfEvent PerfEvent(CounterName);
       if (!PerfEvent.valid())
         return make_error<Failure>(
             Twine("invalid perf event '").concat(CounterName).concat("'"));
       pfm::Counter Counter(PerfEvent);
       Scratch->clear();
       {
         CrashRecoveryContext CRC;
         CrashRecoveryContext::Enable();
         const bool Crashed = !CRC.RunSafely([this, &Counter, ScratchPtr]() {
           Counter.start();
           this->Function(ScratchPtr);
           Counter.stop();
         });
         CrashRecoveryContext::Disable();
         // FIXME: Better diagnosis.
         if (Crashed)
           return make_error<SnippetCrash>("snippet crashed while running");
       }
       CounterValue += Counter.read();
     }
     return CounterValue;
   }
 
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
+
+class LbrFunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
+public:
+  LbrFunctionExecutorImpl(
+      const LLVMState &State,
+      llvm::object::OwningBinary<llvm::object::ObjectFile> Obj,
+      BenchmarkRunner::ScratchSpace *Scratch, unsigned LbrSamplePeriod)
+      : Target(State.getExegesisTarget()),
+        Function(State.createTargetMachine(), std::move(Obj)), Scratch(Scratch),
+        LbrSamplePeriod(LbrSamplePeriod) {}
+
+  std::string description() const override { return "LBR"; }
+  struct ThreadArg {
+    std::mutex Mutex;
+    pfm::Counter *CounterPtr;
+    std::condition_variable ConditionVariable;
+    bool Crashed;
+    std::string ErrMsg;
+    ~ThreadArg() {
+      if (CounterPtr != nullptr) {
+        delete CounterPtr;
+      }
+    }
+  };
+
+  void workerThread(struct ThreadArg *Arg) const {
+    pfm::PerfEvent PerfEvent("LBR", LbrSamplePeriod);
+    auto CounterOrError = Target.createCounter(PerfEvent);
+    if (!CounterOrError) {
+      std::lock_guard<std::mutex> Lock(Arg->Mutex);
+      Arg->Crashed = true;
+      llvm::errs() << "Error creating counter: " << CounterOrError.takeError();
+      Arg->ErrMsg = "Cannot create counter.";
+      Arg->ConditionVariable.notify_one();
+      return;
+    }
+    pfm::Counter *Counter = CounterOrError.get();
+    {
+      std::lock_guard<std::mutex> Lock(Arg->Mutex);
+      Arg->CounterPtr = Counter;
+    }
+    Arg->ConditionVariable.notify_one();
+    {
+      llvm::CrashRecoveryContext CRC;
+      llvm::CrashRecoveryContext::Enable();
+      const bool Crashed = !CRC.RunSafely([&]() {
+        Counter->start();
+        Function(Scratch->ptr());
+        Counter->stop();
+      });
+      llvm::CrashRecoveryContext::Disable();
+      {
+        std::lock_guard<std::mutex> Lock(Arg->Mutex);
+        Arg->CounterPtr = nullptr;
+      }
+      // FIXME: Better diagnosis.
+      if (Crashed) {
+        Arg->Crashed = true;
+        return;
+      }
+    }
+    Arg->Crashed = false;
+  }
+
+  llvm::Expected<int64_t> runAndMeasure(const char *) const override {
+    Scratch->clear();
+    struct ThreadArg Arg;
+    Arg.CounterPtr = nullptr;
+    std::thread Worker(&LbrFunctionExecutorImpl::workerThread, this, &Arg);
+    {
+      // Waits until the counter is ready or a "Crashed" signal if
+      // it could not be created.
+      std::unique_lock<std::mutex> Lock(Arg.Mutex);
+      Arg.ConditionVariable.wait(
+          Lock, [&Arg] { return Arg.Crashed || Arg.CounterPtr != nullptr; });
+    }
+    if (Arg.Crashed) {
+      return make_error<SnippetCrash>(
+          "Snippet crashed while running in thread. Reason: [" + Arg.ErrMsg +
+          "]");
+    }
+
+    auto ValueOrError = Arg.CounterPtr->readOrError();
+    Worker.join();
+    if (!ValueOrError) {
+      return ValueOrError.takeError();
+    }
+    return ValueOrError.get();
+  }
+
+  const ExegesisTarget &Target;
+  const ExecutableFunction Function;
+  BenchmarkRunner::ScratchSpace *const Scratch;
+  unsigned LbrSamplePeriod;
+};
+
+bool supportsLbr(const LLVMState &State) {
+  static const std::set<std::string> SupportedCpus{"haswell", "broadwell",
+                                                   "skylake", "kabylake"};
+  return SupportedCpus.find(State.getTargetMachine().getTargetCPU().str()) !=
+         SupportedCpus.end();
+}
+
+std::unique_ptr<BenchmarkRunner::FunctionExecutor>
+createFunctionExecutor(InstructionBenchmark::ModeE Mode, const LLVMState &State,
+                       object::OwningBinary<object::ObjectFile> ObjectFile,
+                       BenchmarkRunner::ScratchSpace *Scratch,
+                       const BenchmarkRunner::RunArg &Arg) {
+  if (Mode == InstructionBenchmark::ModeE::Latency) {
+    if (Arg.LbrSamplePeriod > 0) {
+      if (supportsLbr(State)) {
+        return std::make_unique<LbrFunctionExecutorImpl>(
+            State, std::move(ObjectFile), Scratch, Arg.LbrSamplePeriod);
+      } else {
+        llvm::errs() << "LBR not supported on given target["
+                     << State.getTargetMachine().getTargetCPU().str() << "].\n";
+        // Falls back to the default executor.
+      }
+    }
+  }
+  return std::make_unique<FunctionExecutorImpl>(State, std::move(ObjectFile),
+                                                Scratch);
+}
+
 } // namespace
 
 Expected<InstructionBenchmark> BenchmarkRunner::runConfiguration(
     const BenchmarkCode &BC, unsigned NumRepetitions,
     ArrayRef<std::unique_ptr<const SnippetRepetitor>> Repetitors,
     bool DumpObjectToDisk) const {
+  RunArg Arg{BC, NumRepetitions, Repetitors, DumpObjectToDisk,
+             0 /*LbrSamplePeriod*/};
+  return runConfiguration(Arg);
+}
+
+Expected<InstructionBenchmark>
+BenchmarkRunner::runConfiguration(const RunArg &Arg) const {
   InstructionBenchmark InstrBenchmark;
   InstrBenchmark.Mode = Mode;
   InstrBenchmark.CpuName = std::string(State.getTargetMachine().getTargetCPU());
   InstrBenchmark.LLVMTriple =
       State.getTargetMachine().getTargetTriple().normalize();
-  InstrBenchmark.NumRepetitions = NumRepetitions;
-  InstrBenchmark.Info = BC.Info;
+  InstrBenchmark.NumRepetitions = Arg.NumRepetitions;
+  InstrBenchmark.Info = Arg.CodeSnippet.Info;
 
-  const std::vector<MCInst> &Instructions = BC.Key.Instructions;
+  const std::vector<MCInst> &Instructions = Arg.CodeSnippet.Key.Instructions;
 
-  InstrBenchmark.Key = BC.Key;
+  InstrBenchmark.Key = Arg.CodeSnippet.Key;
 
   // If we end up having an error, and we've previously succeeded with
   // some other Repetitor, we want to discard the previous measurements.
   struct ClearBenchmarkOnReturn {
     ClearBenchmarkOnReturn(InstructionBenchmark *IB) : IB(IB) {}
     ~ClearBenchmarkOnReturn() {
       if (Clear)
         IB->Measurements.clear();
     }
     void disarm() { Clear = false; }
 
   private:
     InstructionBenchmark *const IB;
     bool Clear = true;
   };
   ClearBenchmarkOnReturn CBOR(&InstrBenchmark);
 
-  for (const std::unique_ptr<const SnippetRepetitor> &Repetitor : Repetitors) {
+  for (const std::unique_ptr<const SnippetRepetitor> &Repetitor :
+       Arg.Repetitors) {
     // Assemble at least kMinInstructionsForSnippet instructions by repeating
     // the snippet for debug/analysis. This is so that the user clearly
     // understands that the inside instructions are repeated.
     constexpr const int kMinInstructionsForSnippet = 16;
     {
       SmallString<0> Buffer;
       raw_svector_ostream OS(Buffer);
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
-              BC.LiveIns, BC.Key.RegisterInitialValues,
+              Arg.CodeSnippet.LiveIns,
+              Arg.CodeSnippet.Key.RegisterInitialValues,
               Repetitor->Repeat(Instructions, kMinInstructionsForSnippet),
               OS)) {
         return std::move(E);
       }
+
       const ExecutableFunction EF(State.createTargetMachine(),
                                   getObjectFromBuffer(OS.str()));
       const auto FnBytes = EF.getFunctionBytes();
       InstrBenchmark.AssembledSnippet.insert(
           InstrBenchmark.AssembledSnippet.end(), FnBytes.begin(),
           FnBytes.end());
     }
 
     // Assemble NumRepetitions instructions repetitions of the snippet for
     // measurements.
     const auto Filler =
         Repetitor->Repeat(Instructions, InstrBenchmark.NumRepetitions);
 
     object::OwningBinary<object::ObjectFile> ObjectFile;
-    if (DumpObjectToDisk) {
-      auto ObjectFilePath = writeObjectFile(BC, Filler);
+    if (Arg.DumpObjectToDisk) {
+      auto ObjectFilePath = writeObjectFile(Arg.CodeSnippet, Filler);
       if (Error E = ObjectFilePath.takeError()) {
         InstrBenchmark.Error = toString(std::move(E));
         return InstrBenchmark;
       }
       outs() << "Check generated assembly with: /usr/bin/objdump -d "
              << *ObjectFilePath << "\n";
       ObjectFile = getObjectFromFile(*ObjectFilePath);
     } else {
       SmallString<0> Buffer;
       raw_svector_ostream OS(Buffer);
       if (Error E = assembleToStream(
               State.getExegesisTarget(), State.createTargetMachine(),
-              BC.LiveIns, BC.Key.RegisterInitialValues, Filler, OS)) {
+              Arg.CodeSnippet.LiveIns,
+              Arg.CodeSnippet.Key.RegisterInitialValues, Filler, OS)) {
         return std::move(E);
       }
       ObjectFile = getObjectFromBuffer(OS.str());
     }
 
-    const FunctionExecutorImpl Executor(State, std::move(ObjectFile),
-                                        Scratch.get());
-    auto NewMeasurements = runMeasurements(Executor);
+    std::unique_ptr<BenchmarkRunner::FunctionExecutor> Executor =
+        createFunctionExecutor(Mode, State, std::move(ObjectFile),
+                               Scratch.get(), Arg);
+
+    auto NewMeasurements = runMeasurements(*Executor);
     if (Error E = NewMeasurements.takeError()) {
       if (!E.isA<SnippetCrash>())
         return std::move(E);
       InstrBenchmark.Error = toString(std::move(E));
       return InstrBenchmark;
     }
     assert(InstrBenchmark.NumRepetitions > 0 && "invalid NumRepetitions");
     for (BenchmarkMeasure &BM : *NewMeasurements) {
       // Scale the measurements by instruction.
       BM.PerInstructionValue /= InstrBenchmark.NumRepetitions;
       // Scale the measurements by snippet.
       BM.PerSnippetValue *= static_cast<double>(Instructions.size()) /
                             InstrBenchmark.NumRepetitions;
     }
     if (InstrBenchmark.Measurements.empty()) {
       InstrBenchmark.Measurements = std::move(*NewMeasurements);
       continue;
     }
 
-    assert(Repetitors.size() > 1 && !InstrBenchmark.Measurements.empty() &&
+    assert(Arg.Repetitors.size() > 1 && !InstrBenchmark.Measurements.empty() &&
            "We're in an 'min' repetition mode, and need to aggregate new "
            "result to the existing result.");
     assert(InstrBenchmark.Measurements.size() == NewMeasurements->size() &&
            "Expected to have identical number of measurements.");
     for (auto I : zip(InstrBenchmark.Measurements, *NewMeasurements)) {
       BenchmarkMeasure &Measurement = std::get<0>(I);
       BenchmarkMeasure &NewMeasurement = std::get<1>(I);
       assert(Measurement.Key == NewMeasurement.Key &&

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.h

 class LatencyBenchmarkRunner : public BenchmarkRunner {
 public:
   LatencyBenchmarkRunner(const LLVMState &State,
                          InstructionBenchmark::ModeE Mode);
   ~LatencyBenchmarkRunner() override;
 
 private:
   Expected<std::vector<BenchmarkMeasure>>
+  runMeasurementsWithLbr(const FunctionExecutor &Executor) const;
+
+  Expected<std::vector<BenchmarkMeasure>>
+  runMeasurements(const FunctionExecutor &Executor) const override;
+};
+
+class LbrLatencyBenchmarkRunner : public BenchmarkRunner {
+public:
+  LbrLatencyBenchmarkRunner(const LLVMState &State);
+  ~LbrLatencyBenchmarkRunner() = default;
+
+private:
+  Expected<std::vector<BenchmarkMeasure>>
   runMeasurements(const FunctionExecutor &Executor) const override;
 };
+
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_LATENCY_H

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

for (size_t I = 0; I < NumMeasurements; ++I) {
     if (!ExpectedCounterValue)
       return ExpectedCounterValue.takeError();
     if (*ExpectedCounterValue < MinValue)
       MinValue = *ExpectedCounterValue;
   }
   std::vector<BenchmarkMeasure> Result;
   switch (Mode) {
   case InstructionBenchmark::Latency:
-    Result = {BenchmarkMeasure::Create("latency", MinValue)};
+    Result = {BenchmarkMeasure::Create(
+        "latency[" + Executor.description() + "]", MinValue)};
     break;
   case InstructionBenchmark::InverseThroughput:
-    Result = {BenchmarkMeasure::Create("inverse_throughput", MinValue)};
+    Result = {BenchmarkMeasure::Create(
+        "inverse_throughput[" + Executor.description() + "]", MinValue)};
     break;
   default:
     break;
   }
   return std::move(Result);
 }
 
 } // namespace exegesis
 } // namespace llvm

llvm/tools/llvm-exegesis/lib/PerfHelper.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Error.h"
+#include <cstdint>
 #include <functional>
 #include <memory>
 
 struct perf_event_attr;
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 // Returns true on error.
 bool pfmInitialize();
 void pfmTerminate();
 
 // Retrieves the encoding for the event described by pfm_event_string.
 // NOTE: pfm_initialize() must be called before creating PerfEvent objects.
 class PerfEvent {
 public:
   // http://perfmon2.sourceforge.net/manv4/libpfm.html
   // Events are expressed as strings. e.g. "INSTRUCTION_RETIRED"
-  explicit PerfEvent(StringRef pfm_event_string);
+  explicit PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod = 0);
 
   PerfEvent(const PerfEvent &) = delete;
   PerfEvent(PerfEvent &&other);
   ~PerfEvent();
 
   // The pfm_event_string passed at construction time.
   StringRef name() const;
 
   // Whether the event was successfully created.
   bool valid() const;
 
   // The encoded event to be passed to the Kernel.
   const perf_event_attr *attribute() const;
 
   // The fully qualified name for the event.
   // e.g. "snb_ep::INSTRUCTION_RETIRED:e=0:i=0:c=0:t=0:u=1:k=0:mg=0:mh=1"
   StringRef getPfmEventString() const;
 
+  // Returns true if it should use LBR.
+  bool useLbr() const;
+
 private:
+  void initPerfEvent();
+  void initPerfEventForLbr(unsigned SamplingPeriod);
+
   const std::string EventString;
   std::string FullQualifiedEventString;
   perf_event_attr *Attr;
 };
 
 // Uses a valid PerfEvent to configure the Kernel so we can measure the
 // underlying event.
-struct Counter {
+class Counter {
+public:
   // event: the PerfEvent to measure.
   explicit Counter(const PerfEvent &event);
 
   Counter(const Counter &) = delete;
   Counter(Counter &&other) = default;
 
-  ~Counter();
+  virtual ~Counter();
 
-  void start();         // Starts the measurement of the event.
-  void stop();          // Stops the measurement of the event.
-  int64_t read() const; // Return the current value of the counter.
+  /// Starts the measurement of the event.
+  virtual void start();
 
-private:
+  /// Stops the measurement of the event.
+  void stop();
+
+  /// Returns the current value of the counter.
+  virtual int64_t read() const;
+
+  /// Returns the current value of the counter or error if it cannot be read.
+  virtual llvm::Expected<int64_t> readOrError() const { return read(); }
+
+protected:
 #ifdef HAVE_LIBPFM
   int FileDescriptor = -1;
-#endif
+#endif // HAVE_LIBPFM
 };
 
 // Helper to measure a list of PerfEvent for a particular function.
 // callback is called for each successful measure (PerfEvent needs to be valid).
 template <typename Function>
 void Measure(
     ArrayRef<PerfEvent> Events,
     const std::function<void(const PerfEvent &Event, int64_t Value)> &Callback,

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

 //===-- PerfHelper.cpp ------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "PerfHelper.h"
 #include "llvm/Config/config.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/raw_ostream.h"
+#include <memory>
 #ifdef HAVE_LIBPFM
 #include "perfmon/perf_event.h"
 #include "perfmon/pfmlib.h"
 #include "perfmon/pfmlib_perf_event.h"
 #endif
+#include <atomic>
 #include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
 
 namespace llvm {
 namespace exegesis {
 namespace pfm {
 
 #ifdef HAVE_LIBPFM
 static bool isPfmError(int Code) { return Code != PFM_SUCCESS; }
 #endif

 
 PerfEvent::PerfEvent(PerfEvent &&Other)
     : EventString(std::move(Other.EventString)),
       FullQualifiedEventString(std::move(Other.FullQualifiedEventString)),
       Attr(Other.Attr) {
   Other.Attr = nullptr;
 }
 
-PerfEvent::PerfEvent(StringRef PfmEventString)
-    : EventString(PfmEventString.str()), Attr(nullptr) {
+void PerfEvent::initPerfEvent() {
 #ifdef HAVE_LIBPFM
   char *Fstr = nullptr;
   pfm_perf_encode_arg_t Arg = {};
   Attr = new perf_event_attr();
   Arg.attr = Attr;
   Arg.fstr = &Fstr;
   Arg.size = sizeof(pfm_perf_encode_arg_t);
   const int Result = pfm_get_os_event_encoding(EventString.c_str(), PFM_PLM3,
                                                PFM_OS_PERF_EVENT, &Arg);
   if (isPfmError(Result)) {
     // We don't know beforehand which counters are available (e.g. 6 uops ports
     // on Sandybridge but 8 on Haswell) so we report the missing counter without
     // crashing.
     errs() << pfm_strerror(Result) << " - cannot create event " << EventString
            << "\n";
   }
   if (Fstr) {
     FullQualifiedEventString = Fstr;
     free(Fstr);
   }
 #endif
 }
 
+void PerfEvent::initPerfEventForLbr(unsigned SamplingPeriod) {
+#ifdef HAVE_LIBPFM
+  assert(SamplingPeriod > 0 && "SamplingPeriod must be positive");
+  Attr = new perf_event_attr();
+  *Attr = {0};
+  Attr->size = sizeof(*Attr);
+  Attr->type = PERF_TYPE_HARDWARE;
+  Attr->config = PERF_COUNT_HW_BRANCH_INSTRUCTIONS;
+  Attr->sample_type = PERF_SAMPLE_BRANCH_STACK;
+  Attr->branch_sample_type = PERF_SAMPLE_BRANCH_USER | PERF_SAMPLE_BRANCH_ANY;
+  Attr->sample_period = SamplingPeriod;
+  Attr->wakeup_events = 1; // We need this even when using ioctl REFRESH.
+  Attr->disabled = 1;
+  Attr->exclude_kernel = 1;
+  Attr->exclude_hv = 1;
+  Attr->read_format = PERF_FORMAT_GROUP;
+
+  FullQualifiedEventString = "LBR";
+#endif
+}
+
+PerfEvent::PerfEvent(StringRef PfmEventString, unsigned SamplingPeriod)
+    : EventString(PfmEventString.str()), Attr(nullptr) {
+  if (EventString == "LBR") {
+    initPerfEventForLbr(SamplingPeriod);
+  } else {
+    initPerfEvent();
+  }
+}
+
 StringRef PerfEvent::name() const { return EventString; }
 
 bool PerfEvent::valid() const { return !FullQualifiedEventString.empty(); }
 
+bool PerfEvent::useLbr() const { return EventString == "LBR"; }
+
 const perf_event_attr *PerfEvent::attribute() const { return Attr; }
 
 StringRef PerfEvent::getPfmEventString() const {
   return FullQualifiedEventString;
 }
 
 #ifdef HAVE_LIBPFM
 Counter::Counter(const PerfEvent &Event) {

llvm/tools/llvm-exegesis/lib/Target.h

 #include "LlvmState.h"
 #include "SnippetGenerator.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/MC/MCRegisterInfo.h"
+#include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace exegesis {
 
 struct PfmCountersInfo {
   // An optional name of a performance counter that can be used to measure
   // cycles.
   const char *CycleCounter;
struct CpuAndPfmCounters {
   bool operator<(StringRef S) const { return StringRef(CpuName) < S; }
 };
 
 class ExegesisTarget {
 public:
   explicit ExegesisTarget(ArrayRef<CpuAndPfmCounters> CpuPfmCounters)
       : CpuPfmCounters(CpuPfmCounters) {}
 
+  virtual Expected<pfm::Counter *>
+  createCounter(const pfm::PerfEvent &Event) const {
+    return new pfm::Counter(Event);
+  }
+
   // Targets can use this to add target-specific passes in assembleToStream();
   virtual void addTargetSpecificPasses(PassManagerBase &PM) const {}
 
   // Generates code to move a constant into a the given register.
   // Precondition: Value must fit into Reg.
   virtual std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                        const APInt &Value) const = 0;
 
private:
   std::unique_ptr<SnippetGenerator> virtual createSerialSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<SnippetGenerator> virtual createParallelSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<BenchmarkRunner> virtual createLatencyBenchmarkRunner(
       const LLVMState &State, InstructionBenchmark::ModeE Mode) const;
   std::unique_ptr<BenchmarkRunner> virtual createUopsBenchmarkRunner(
       const LLVMState &State) const;
 
   const ExegesisTarget *Next = nullptr;
   const ArrayRef<CpuAndPfmCounters> CpuPfmCounters;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

 //===-- Target.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "../Target.h"
 
 #include "../Error.h"
 #include "../ParallelSnippetGenerator.h"
 #include "../SerialSnippetGenerator.h"
 #include "../SnippetGenerator.h"
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "X86.h"
+#include "X86Counter.h"
 #include "X86RegisterInfo.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/Sequence.h"
 #include "llvm/MC/MCInstBuilder.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/FormatVariadic.h"
 
+#ifdef HAVE_LIBPFM
+#include "perfmon/perf_event.h"
+#include "perfmon/pfmlib.h"
+#include "perfmon/pfmlib_perf_event.h"
+#endif // HAVE_LIBPFM
+
+#include <atomic>
+#include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <vector>
+
 namespace llvm {
 namespace exegesis {
 
 // Returns a non-null reason if we cannot handle the memory references in this
 // instruction.
 static const char *isInvalidMemoryInstr(const Instruction &Instr) {
   switch (Instr.Description.TSFlags & X86II::FormMask) {
   default:
if (Bytes - ByteOffset >= 1)
     add(fillStackSpace(
         X86::MOV8mi, ByteOffset,
         WideConstant.extractBits(8, ByteOffset * 8).getZExtValue()));
 }
 
 #include "X86GenExegesis.inc"
 
 namespace {
+
 class ExegesisX86Target : public ExegesisTarget {
 public:
   ExegesisX86Target() : ExegesisTarget(X86CpuPfmCounters) {}
 
+  Expected<pfm::Counter *>
+  createCounter(const pfm::PerfEvent &Event) const override {
+    // Can't use LBR without LIB PFM.
+    if (Event.useLbr()) {
+#ifdef HAVE_LIBPFM
+      return new X86LbrCounter(Event);
+#else
+      return llvm::make_error<llvm::StringError>(
+          "LBR counter requested without PFM package available..",
+          llvm::errc::invalid_argument);
+#endif
+    }
+    return new pfm::Counter(Event);
+  }
+
 private:
   void addTargetSpecificPasses(PassManagerBase &PM) const override;
 
   unsigned getScratchMemoryRegister(const Triple &TT) const override;
 
   unsigned getLoopCounterRegister(const Triple &) const override;
 
   unsigned getMaxMemoryAccessSize() const override { return 64; }
 
   Error randomizeTargetMCOperand(const Instruction &Instr, const Variable &Var,
                                  MCOperand &AssignedValue,
                                  const BitVector &ForbiddenRegs) const override;
 
   void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                           unsigned Offset) const override;
 
   void decrementLoopCounterAndJump(MachineBasicBlock &MBB,
                                    MachineBasicBlock &TargetMBB,
                                    const MCInstrInfo &MII) const override;
 
   std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                const APInt &Value) const override;
 
   ArrayRef<unsigned> getUnavailableRegisters() const override {
     return makeArrayRef(kUnavailableRegisters,

llvm/tools/llvm-exegesis/lib/X86/X86Counter.h

+#include "../PerfHelper.h"
+#include "llvm/Support/Error.h"
+
+namespace llvm {
+namespace exegesis {
+
+#ifdef HAVE_LIBPFM
+
+class X86LbrCounter : public pfm::Counter {
+public:
+  explicit X86LbrCounter(const pfm::PerfEvent &Event);
+
+  virtual ~X86LbrCounter();
+
+  void start() override;
+  int64_t read() const override;
+  llvm::Expected<int64_t> readOrError() const override;
+
+private:
+  void *MMappedBuffer = nullptr;
+};
+#endif // HAVE_LIBPFM
+
+} // namespace exegesis
+} // namespace llvm

llvm/tools/llvm-exegesis/lib/X86/X86Counter.cpp

+#include "X86Counter.h"
+
+#include "llvm/Support/Errc.h"
+
+#ifdef HAVE_LIBPFM
+#include "perfmon/perf_event.h"
+#include "perfmon/pfmlib.h"
+#include "perfmon/pfmlib_perf_event.h"
+#endif // HAVE_LIBPFM
+
+#include <atomic>
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <poll.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <vector>
+
+#ifdef HAVE_LIBPFM
+namespace llvm {
+namespace exegesis {
+namespace {
+
+static constexpr int KBufferPages = 8;
+static const int kDataBufferSize = kBufferPages * getpagesize();
+
+// Waits for the LBR perf events.
+int pollLbrPerfEvent(const int FileDescriptor) {
+  struct pollfd PollFd;
+  PollFd.fd = FileDescriptor;
+  PollFd.events = POLLIN;
+  PollFd.revents = 0;
+  return poll(&PollFd, 1 /* num of fds */, 10000 /* time out */);
+}
+
+// Copies the data-buffer into Buf, given the pointer to MMapped.
+void copyDataBuffer(void *MMappedBuffer, char *Buf, uint64_t Tail,
+                    size_t DataSize) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page.
+  char *Start = reinterpret_cast<char *>(MMappedBuffer) + getpagesize();
+  // The LBR buffer is a cyclic buffer, we copy data to another buffer.
+  uint64_t Offset = Tail % kDataBufferSize;
+  size_t CopySize = kDataBufferSize - Offset;
+  memcpy(Buf, Start + Offset, CopySize);
+  if (CopySize >= DataSize)
+    return;
+  memcpy(Buf + CopySize, Start, Offset);
+}
+
+// Rewrites the jmp (from the BM loop) with `pop $rbx; ret` to get
+// out of the benchmarked code.
+//
+// pc: Must point at the start of the jmp instruction.
+void patchBasicBlockToEndBenchmarkedLoop(char *pc) const {
+  void *page = reinterpret_cast<void *>(reinterpret_cast<uintptr_t>(pc) &
+                                        ~(getpagesize() - 1));
+  mprotect(page, getpagesize(), PROT_READ | PROT_WRITE | PROT_EXEC);
+  if ((reinterpret_cast<uint64_t>(pc) & 0xf) != 0xf) {
+    // We have at least two bytes in the same cacheline, so we can use an
+    // atomic write to replace the jmp with pop, ret.
+    *reinterpret_cast<uint16_t *>(pc) = 0xc35b; // pop %rbx; ret;
+  } else {
+    // pc and pc+1 might be at different cacheline, so we first add the pop,
+    // ret instructions, and then update jump offset to jump to pop
+    // instruction.
+    *reinterpret_cast<uint16_t *>(pc + 5) = 0xc35b; // pop %rbx; ret;
+    // An automic update on the displacement.
+    if (pc[0] == 0xeb /* x86 short jump */) {
+      // offset is 1 byte
+      pc[1] = 0x03;
+    } else { /* x86 near jump */
+      // offset is 4 bytes
+      *reinterpret_cast<uint32_t *>(pc + 1) = 0;
+    }
+  }
+}
+
+// Parses the given data-buffer for stats and fill the CycleArray.
+// If data has been extracted successfully, also modifies the code to jump
+// out the benchmark loop.
+llvm::Error parseDataBuffer(const char const *DataBuf, size_t DataSize,
+                            std::vector<int64_t> *CycleArray,
+                            size_t Pos) const {
+  const char *DataPtr = DataBuf;
+  while (DataPtr < DataBuf + DataSize) {
+    struct perf_event_header Header;
+    memcpy(&Header, DataPtr, sizeof(struct perf_event_header));
+    if (Header.type != PERF_RECORD_SAMPLE) {
+      // Ignores non-sample records.
+      DataPtr += Header.size;
+      continue;
+    }
+    DataPtr += sizeof(Header);
+    uint64_t Count = llvm::support::endian::read64(DataPtr, support::native);
+    DataPtr += sizeof(Count);
+
+    struct perf_branch_entry Entry;
+    memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    // Read the perf_branch_entry array.
+    char *JumpPC = reinterpret_cast<char *>(Entry.from);
+    int64_t MinCycle = Entry.cycles;
+    for (int i = 0; i < Count; ++i) {
+      // We use the JumpPC from the entry with min cycle to avoid the entry
+      // that is returning from kernel.
+      if (MinCycle > Entry.cycles) {
+        MinCycle = Entry.cycles;
+        JumpPC = reinterpret_cast<char *>(Entry.from);
+      }
+      (*CycleArray)[Pos++] = Entry.cycles;
+      if (Pos == CycleArray->size()) {
+        patchBasicBlockToEndBenchmarkedLoop(JumpPC);
+        return llvm::Error::success();
+      }
+      // Advance to next entry
+      DataPtr += sizeof(Entry);
+      memcpy(&Entry, DataPtr, sizeof(struct perf_branch_entry));
+    }
+  }
+  return llvm::make_error<llvm::StringError>("Unable to read databuffer.",
+                                             llvm::errc::io_error);
+}
+
+} // namespace
+
+X86LbrCounter::X86LbrCounter(const pfm::PerfEvent &Event) : Counter(Event) {
+  // First page is reserved for perf_event_mmap_page. Data buffer starts on
+  // the next page, so we allocate one more page.
+  MMappedBuffer = mmap(nullptr, (kBufferPages + 1) * getpagesize(),
+                       PROT_READ | PROT_WRITE, MAP_SHARED, FileDescriptor, 0);
+  if (MMappedBuffer == MAP_FAILED) {
+    llvm::errs() << "Failed to mmap buffer.";
+  }
+}
+
+X86LbrCounter::~X86LbrCounter() { close(FileDescriptor); }
+
+void X86LbrCounter::start() override {
+  ioctl(FileDescriptor, PERF_EVENT_IOC_REFRESH, 1024 /* kMaxPollsPerFd */);
+}
+
+int64_t X86LbrCounter::read() const override {
+  auto error = readOrError();
+  if (!error) {
+    return error.get();
+  }
+  llvm::errs() << "Error reading counter: " << error.takeError() << "\n";
+  return 0;
+}
+
+llvm::Expected<int64_t> X86LbrCounter::readOrError() const override {
+  // The max number of time-outs/retries before we give up.
+  static constexpr int kMaxTimeouts = 160;
+
+  // Parses the LBR buffer and fills CycleArray with the sequence of cycle
+  // counts from the buffer.
+  std::vector<int64_t> CycleArray;
+  std::unique_ptr<char[]> DataBuf(new char[kDataBufferSize]);
+  size_t Pos = 0;
+  int NumTimeouts = 0;
+  while (true) {
+    const int PollResult = pollLbrPerfEvent(FileDescriptor);
+    if (PollResult == -1) {
+      return llvm::make_error<llvm::StringError>("Cannot poll LBR perf event.",
+                                                 llvm::errc::io_error);
+    } else if (PollResult == 0) {
+      llvm::errs() << "LBR polling timed out without result, NumTimeouts ="
+                   << NumTimeouts << ". ";
+      if (NumTimeouts < kMaxTimeouts) {
+        llvm::errs() << "Retrying ...\n";
+        ++NumTimeouts;
+        continue;
+      } else {
+        llvm::errs() << "At max-timeouts. Giving up.\n";
+        return llvm::make_error<llvm::StringError>(
+            "LBR polling still timed out after max number of attempts.",
+            llvm::errc::device_or_resource_busy);
+      }
+    }
+
+    struct perf_event_mmap_page Page;
+    memcpy(&Page, MMappedBuffer, sizeof(struct perf_event_mmap_page));
+    const uint64_t DataTail = Page.data_tail;
+    const uint64_t DataHead = Page.data_head;
+    // We're supposed to use a barrier after reading data_head.
+    std::atomic_thread_fence(std::memory_order_acq_rel);
+    const size_t DataSize = DataHead - DataTail;
+    if (DataSize > kDataBufferSize) {
+      return llvm::make_error<llvm::StringError>(
+          "DataSize larger than buffer size.", llvm::errc::invalid_argument);
+    }
+    copyDataBuffer(MMappedBuffer, DataBuf.get(), DataTail, DataSize);
+    llvm::Error error =
+        parseDataBuffer(DataBuf.get(), DataSize, &CycleArray, Pos);
+    if (!error) {
+      // TODO(vyng) Analyse the array and get proper value.
+      return CycleArray[0];
+    }
+  }
+  return llvm::make_error<llvm::StringError>("Unknown error.",
+                                             llvm::errc::io_error);
+}
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif // HAVE_LIBPFM

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

cl::values(clEnumValN(exegesis::InstructionBenchmark::Latency, "latency",
                           "Instruction Inverse Throughput"),
                clEnumValN(exegesis::InstructionBenchmark::Uops, "uops",
                           "Uop Decomposition"),
                // When not asking for a specific benchmark mode,
                // we'll analyse the results.
                clEnumValN(exegesis::InstructionBenchmark::Unknown, "analysis",
                           "Analysis")));
 
+// If a positive value is specified, we are going to use the LBR in
+// latency-mode.
+//
+// Note:
+//  -  A small value is preferred, but too low a value could result in
+//     throttling.
+//  -  A prime number is preferred to avoid hiding blocks.
+//
+// Based on various testings, `521` seems like the recommended value so far.
+static cl::opt<unsigned> LbrSamplePeriod(
+    "x86-lbr-sample-period",
+    cl::desc("The sample period (nbranches/sample), used for LBR sampling"),
+    cl::cat(BenchmarkOptions), cl::init(0));
+
 static cl::opt<exegesis::InstructionBenchmark::RepetitionModeE> RepetitionMode(
     "repetition-mode", cl::desc("how to repeat the instruction snippet"),
     cl::cat(BenchmarkOptions),
     cl::values(
         clEnumValN(exegesis::InstructionBenchmark::Duplicate, "duplicate",
                    "Duplicate the snippet"),
         clEnumValN(exegesis::InstructionBenchmark::Loop, "loop",
                    "Loop over the snippet"),
#endif
   InitializeNativeTarget();
   InitializeNativeTargetAsmPrinter();
   InitializeNativeTargetAsmParser();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
 
   const std::unique_ptr<BenchmarkRunner> Runner = ExitOnErr(
       State.getExegesisTarget().createBenchmarkRunner(BenchmarkMode, State));
   if (!Runner) {
     ExitWithError("cannot create benchmark runner");
   }
 
   const auto Opcodes = getOpcodesOrDie(State.getInstrInfo());
 
   SmallVector<std::unique_ptr<const SnippetRepetitor>, 2> Repetitors;
   if (RepetitionMode != InstructionBenchmark::RepetitionModeE::AggregateMin)
for (const unsigned Opcode : Opcodes) {
       }
       std::move(ConfigsForInstr->begin(), ConfigsForInstr->end(),
                 std::back_inserter(Configurations));
     }
   } else {
     Configurations = ExitOnErr(readSnippets(State, SnippetsFile));
   }
 
+#if !defined(__x86_64__) && !defined(__x86__) && !defined(__i386__)
+  if (exegesis::BenchmarkMode == exegesis::InstructionBenchmark::Latency &&
+      LbrSamplePeriod > 0) {
+    ExitOnErr.setBanner("llvm-exegesis: ");
+    ExitWithError("LBR mode must be run on x86 arch.");
+  }
+#endif
+
   if (NumRepetitions == 0) {
     ExitOnErr.setBanner("llvm-exegesis: ");
     ExitWithError("--num-repetitions must be greater than zero");
   }
 
   // Write to standard output if file is not set.
   if (BenchmarkFile.empty())
     BenchmarkFile = "-";
 
   for (const BenchmarkCode &Conf : Configurations) {
-    InstructionBenchmark Result = ExitOnErr(Runner->runConfiguration(
-        Conf, NumRepetitions, Repetitors, DumpObjectToDisk));
+    BenchmarkRunner::RunArg RunArg{Conf, NumRepetitions, Repetitors,
+                                   DumpObjectToDisk, LbrSamplePeriod};
+
+    InstructionBenchmark Result = ExitOnErr(Runner->runConfiguration(RunArg));
     ExitOnFileError(BenchmarkFile, Result.writeYaml(State, BenchmarkFile));
   }
   exegesis::pfm::pfmTerminate();
 }
 
 // Prints the results of running analysis pass `Pass` to file `OutputFilename`
 // if OutputFilename is non-empty.
 template <typename Pass>