[llvm-exegesis] Let Counter returns up to 16 entries.

[oontvoo]

Summary

LBR contains (up to) 16 entries for last x branches and the X86LBRCounter (from D77422) should be able to return all those.
Currently, it just returns the latest entry, which could lead to mis-leading measurements.
This patch aslo changes the LatencyBenchmarkRunner to accommodate multi-value readings.

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

73

Please get rif of the magic number. Why 16 ? what about Counter->numValues() ?

110–114

Please split this off to void accumulateCounterValues(ValueOrError.get(), CounterValues)

111

[style]

if (!ValueOrError)
  return ValueOrError.takeError();
...

117

Why not += ?

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

33

ComputeVariance ?

43

This is missing a square.

66

So computing the min and stddev across values in runAndMeasureMulti() requires them to be measuring the same thing. From the documentation it's not clear to me what the values represent. Are they always homogeneous ? Can you give an example of what they are in the LBR case.

72

why not std::numeric_limits<double> ?

77

Technically you're computing a variance.

78

Because of short-circuiting, if WithMinStdev.empty(), then CurStdev will not be evaluated, and CurStdev will be zero. Then Stdev will be set to 0, preventing any further updates.

79

Was this supposed to be the other way around ? Here we are selecting the largest stddev.

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

137–139

This returns a vector with Count elements set to zero. If this passes all tests then we are clearly missing tests :(

[ondrasej]

This looks good in general, but we should be careful about aggregating the values from the measurements (and aggregation when the counter returns multiple values). In particular for the LBR, we'd be losing interesting and potentially useful information by aggregating all the values into a single number.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

72

This should be llvm::SmallVector<int64_t, N> with some small N (e.g. 4) to avoid memory allocations when there is just one return value. This is the case for virtually all counters except the LBR.

73

Consider calling this runAndSample().

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

38

You could do
const double Sum = std::accumulate(Values.begin(), Values.end(), 0.0);
instead.

90

I'd prefer to have more flexibility about the numbers that are returned and reported by the tool. The original code collapsed the measurements to a single value (the minimum). This is useful when looking for lower bounds/optimistic numbers, but other processing would also make sense:

• the mean of the measurements - this might give a better idea of the actual performance when running in the loop.
• the list of all measured values - so that the user can analyze the distribution of the measurements by themselves (and go beyond the mean/variance).

In particular for the LBR measurements: being able to see the raw timings of individual loop iterations is what makes that measurement method so appealing, and we'd lose all that information if we just took an aggregate value, be it a min or a mean. The min and mean from the LBR do have their value and I'd expect them to be more precise than the measurements over an unrolled loop, so ideally we'd have a way to see both.

I think a good solution would be to add an argument to this method that determines how the values are aggregated over the measurements:

• min,
• mean,
• min variance,
• keep all.
  
  and over the contents of the returned vector:
• min,
• mean,
• keep all.

This will mean more changes up the call stack, but it would give us a lot more flexibility in using the tool. What do you think?

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

121–122

You might want to check that there is at least one element.

131

This should be llvm::SmallVector<int64_t, some small N> - most counters return just a single value.

[oontvoo]

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

90

Yes, that's a good idea. I've thought about this a bit more and realised even keeping min-variance (of each read) could still be completely wrong.

Imagine your benchmarked code has a number of distinct branches; then there is no reason to expect the cycles from these branches to correlate. The min-variance approach is only meaningful if it's the same code-path.

[oontvoo]

Updated diff

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

66

Yes, they're supposed to be homogenous, representing the measurements of the same block sampled at fixed rate.

For the LBR specifically:

• Each value in the vector is the number of elapsed cycles since last branch-retire
• When we take a sample, we have 16 of such values, representing the last 16 branches.
• If we have a loop whose body is a basic-block, then effectively these are the measurements for the last 16 iterations[0]. if the loop body has some branches then we'd need a different aggregation strategy.

[0] I think this is where it's a bit "wrong" right now. We always only read the last 16 branches. What we probably want is to be able to have the BenchmarkRunner pause at fixed period and take a sample.

• (Just made a note so we could talk about this tomorrow @courbet @ondrasej )

77

¯\_(ツ)_/¯ indeed!

[oontvoo]

Fix clang-tidy issues

[ondrasej]

I've added a couple of style comment + one bigger comment on the aggregation of results from multiple runs/counter buffer.

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

56

NewValues should be a const reference, no?

65

This is probably not a big deal, but consider:

const size_t NumValues = std::max(NewValues.size(), Result->size());
if (NumValues > Result->size()) Result->resize(NumValues, 0);
for (size_t i = 0, End = NewValues.size(); i < End; ++i) {
  (*Result)[i] += NewValues[i];
}

for unrolled + vectorized code.

73

s/reserved/Reserved/

87

I'd prefer not using assignment in function arguments - this might be easily overlooked or mistaken for a bug (assignment in place of equality comparison).

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

41

std::pow is somewhat heavy-weight as it is not optimized for integer exponents,
const double Delta = V - Mean;
Ret += Delta * Delta;
will likely be significantly faster.

53

A shorter way to write this:

return std::accumulate(
  Values.begin(), Values.end(), std::numeric_limits<int64_t>::max(),
  [](int64_t A, int64_t B) { return A < B ? A : B; });

And similar for FindMax below.

81

Nit: Since we have WithMinVariance, consider renaming Variance to MinVariance, and CurVariance to just Variance.

82

Ideally, this should use move semantics (if they are supported by SmallVector).

83

This would update Variance (MinVariance with the rename proposed above) even if the variance increased. What you probably want is

if (Variance < MinVariance) {
  WithMinVariance = *ExpectedCounterValues;
  MinVariancec = Variance;
}

98

I'd still split this into two arguments:
One that decides what happens with the return values of runAndSample() with {Concatenate, MinVariance}.

And another one that decides what to do with the results of the previous step (Min, Max, Mean, return as is}.

With the current MinVariance filtering in all cases, we're changing the behavior for scalar counters, where we might drop some values (and we might drop some values also in the LBR case).

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

155

This should also be llvm::SmallVector (please test with HAVE_LIBPFM undefined/false).

[oontvoo]

Fixed another clang-tidy issue

[oontvoo]

Updated diff

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

98

Ah, I think we can infer how to accumulate the values returnt by runAndSample().

• If the return vector has more than 1 element, then keep the set with min variance (because it wouldn't make sense to concat all the values from different runs together)
• If the vector only has 1 element, then concat them together to find Min/Max/Mean

[ondrasej]

One last comment, but otherwise this looks good. I'll leave the approval to Clement.

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

72

Very nit: if we're working with doubles, std::numeric_limits<double>::infinity() might be even better.

[oontvoo]

std::numeric_limits<double>::infinity()

[courbet]

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

62

[style] no braces

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

53

Why not std::min_element ?

llvm/tools/llvm-exegesis/lib/Target.cpp

74

Let's do this now to avoid being in an inconsistent state.

[oontvoo]

update diff

[courbet]

llvm/tools/llvm-exegesis/lib/Target.cpp

121

I the default Target implementation does not use it, let's just do:

std::unique_ptr<BenchmarkRunner> ExegesisTarget::createUopsBenchmarkRunner(
    const LLVMState &State,
    InstructionBenchmark::ResultAggregationModeE /*unused*/) const {
  return std::make_unique<UopsBenchmarkRunner>(State);
}

[oontvoo]

llvm/tools/llvm-exegesis/lib/Target.cpp

121

Should we change the impl to use it?

[oontvoo]

Removed unused ResultAggMode

[oontvoo]

update diff

llvm/tools/llvm-exegesis/lib/BenchmarkResult.h

struct InstructionBenchmark {
   // snippet of 3 instructions is repeated 4 times, this is 12.
   int NumRepetitions = 0;
   enum RepetitionModeE { Duplicate, Loop, AggregateMin };
   // Note that measurements are per instruction.
   std::vector<BenchmarkMeasure> Measurements;
   std::string Error;
   std::string Info;
   std::vector<uint8_t> AssembledSnippet;
-
+  // How to aggregate measurements.
+  enum ResultAggregationModeE { Min, Max, MinVariance };
   // Read functions.
   static Expected<InstructionBenchmark> readYaml(const LLVMState &State,
                                                  StringRef Filename);
 
   static Expected<std::vector<InstructionBenchmark>>
   readYamls(const LLVMState &State, StringRef Filename);
 
   class Error readYamlFrom(const LLVMState &State, StringRef InputContent);

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.h

 #define LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 
 #include "Assembler.h"
 #include "BenchmarkCode.h"
 #include "BenchmarkResult.h"
 #include "LlvmState.h"
 #include "MCInstrDescView.h"
 #include "SnippetRepetitor.h"
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/Error.h"
 #include <cstdlib>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace exegesis {
private:
     char *const AlignedPtr;
   };
 
   // A helper to measure counters while executing a function in a sandboxed
   // context.
   class FunctionExecutor {
   public:
     virtual ~FunctionExecutor();
+    // FIXME deprecate this.
     virtual Expected<int64_t> runAndMeasure(const char *Counters) const = 0;
+
+    virtual Expected<llvm::SmallVector<int64_t, 4>>
+    runAndSample(const char *Counters) const = 0;
   };
 
 protected:
   const LLVMState &State;
   const InstructionBenchmark::ModeE Mode;
 
 private:
   virtual Expected<std::vector<BenchmarkMeasure>>

llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

public:
   FunctionExecutorImpl(const LLVMState &State,
                        object::OwningBinary<object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : State(State), Function(State.createTargetMachine(), std::move(Obj)),
         Scratch(Scratch) {}
 
 private:
   Expected<int64_t> runAndMeasure(const char *Counters) const override {
+    auto ResultOrError = runAndSample(Counters);
+    if (ResultOrError)
+      return ResultOrError.get()[0];
+    return ResultOrError.takeError();
+  }
+
+  static void
+  accumulateCounterValues(const llvm::SmallVector<int64_t, 4> NewValues,
+                          llvm::SmallVector<int64_t, 4> *Result) {
+    size_t I = 0;
+    for (const int64_t &Value : NewValues) {
+      if (I >= Result->size())
+        Result->push_back(Value);
+      else
+        (*Result)[I] += Value;
+      ++I;
+    }
+  }
+
+  Expected<llvm::SmallVector<int64_t, 4>>
+  runAndSample(const char *Counters) const override {
     // We sum counts when there are several counters for a single ProcRes
     // (e.g. P23 on SandyBridge).
-    int64_t CounterValue = 0;
+    llvm::SmallVector<int64_t, 4> CounterValues;
+    int Reserved = 0;
     SmallVector<StringRef, 2> CounterNames;
     StringRef(Counters).split(CounterNames, '+');
     char *const ScratchPtr = Scratch->ptr();
     for (auto &CounterName : CounterNames) {
       CounterName = CounterName.trim();
       auto CounterOrError =
           State.getExegesisTarget().createCounter(CounterName, State);
 
       if (!CounterOrError)
         return CounterOrError.takeError();
 
       pfm::Counter *Counter = CounterOrError.get().get();
+      if (Reserved == 0)
+        CounterValues.reserve(Reserved = Counter->numValues());
+      else if (Reserved != Counter->numValues())
+        // It'd be wrong to accumulate vectors of different sizes.
+        return make_error<Failure>(
+            llvm::Twine("Inconsistent number of values for counter ")
+                .concat(CounterName)
+                .concat(std::to_string(Counter->numValues()))
+                .concat(" vs expected of ")
+                .concat(std::to_string(Reserved)));
       Scratch->clear();
       {
         CrashRecoveryContext CRC;
         CrashRecoveryContext::Enable();
         const bool Crashed = !CRC.RunSafely([this, Counter, ScratchPtr]() {
           Counter->start();
           this->Function(ScratchPtr);
           Counter->stop();
         });
         CrashRecoveryContext::Disable();
         // FIXME: Better diagnosis.
         if (Crashed)
           return make_error<SnippetCrash>("snippet crashed while running");
       }
-      CounterValue += Counter->read();
+      auto ValueOrError = Counter->readOrError();
+      if (!ValueOrError)
+        return ValueOrError.takeError();
+
+      accumulateCounterValues(ValueOrError.get(), &CounterValues);
     }
-    return CounterValue;
+    return CounterValues;
   }
 
   const LLVMState &State;
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
 } // namespace
 
 Expected<InstructionBenchmark> BenchmarkRunner::runConfiguration(

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.h

 
 #include "BenchmarkRunner.h"
 
 namespace llvm {
 namespace exegesis {
 
 class LatencyBenchmarkRunner : public BenchmarkRunner {
 public:
-  LatencyBenchmarkRunner(const LLVMState &State,
-                         InstructionBenchmark::ModeE Mode);
+  LatencyBenchmarkRunner(
+      const LLVMState &State, InstructionBenchmark::ModeE Mode,
+      InstructionBenchmark::ResultAggregationModeE ResultAggMode);
   ~LatencyBenchmarkRunner() override;
 
 private:
   Expected<std::vector<BenchmarkMeasure>>
   runMeasurements(const FunctionExecutor &Executor) const override;
+
+  InstructionBenchmark::ResultAggregationModeE ResultAggMode;
 };
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_LATENCY_H

llvm/tools/llvm-exegesis/lib/LatencyBenchmarkRunner.cpp

 //===-- LatencyBenchmarkRunner.cpp ------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "LatencyBenchmarkRunner.h"
 
-#include "Target.h"
 #include "BenchmarkRunner.h"
+#include "Target.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Error.h"
+#include <algorithm>
+#include <cmath>
 
 namespace llvm {
 namespace exegesis {
 
-LatencyBenchmarkRunner::LatencyBenchmarkRunner(const LLVMState &State,
-                                               InstructionBenchmark::ModeE Mode)
+LatencyBenchmarkRunner::LatencyBenchmarkRunner(
+    const LLVMState &State, InstructionBenchmark::ModeE Mode,
+    InstructionBenchmark::ResultAggregationModeE ResultAgg)
     : BenchmarkRunner(State, Mode) {
   assert((Mode == InstructionBenchmark::Latency ||
           Mode == InstructionBenchmark::InverseThroughput) &&
          "invalid mode");
+  ResultAggMode = ResultAgg;
 }
 
 LatencyBenchmarkRunner::~LatencyBenchmarkRunner() = default;
 
+static double computeVariance(const llvm::SmallVector<int64_t, 4> &Values) {
+  if (Values.empty())
+    return 0.0;
+  double Sum = std::accumulate(Values.begin(), Values.end(), 0.0);
+
+  const double Mean = Sum / Values.size();
+  double Ret = 0;
+  for (const auto &V : Values)
+    Ret += std::pow(V - Mean, 2);
+
+  return Ret / Values.size();
+}
+
+static int64_t findMin(const llvm::SmallVector<int64_t, 4> &Values) {
+  if (Values.empty())
+    return 0;
+  int64_t Min = std::numeric_limits<int64_t>::max();
+  for (const int64_t &V : Values)
+    if (V < Min)
+      Min = V;
+  return Min;
+}
+
+static int64_t findMax(const llvm::SmallVector<int64_t, 4> &Values) {
+  if (Values.empty())
+    return 0;
+  int64_t Max = std::numeric_limits<int64_t>::min();
+  for (const int64_t &V : Values)
+    if (V > Max)
+      Max = V;
+  return Max;
+}
+
 Expected<std::vector<BenchmarkMeasure>> LatencyBenchmarkRunner::runMeasurements(
     const FunctionExecutor &Executor) const {
   // Cycle measurements include some overhead from the kernel. Repeat the
   // measure several times and take the minimum value.
   constexpr const int NumMeasurements = 30;
-  int64_t MinValue = std::numeric_limits<int64_t>::max();
+  llvm::SmallVector<int64_t, 4> WithMinVariance;
+  double Variance = std::numeric_limits<double>::max();
   const char *CounterName = State.getPfmCounters().CycleCounter;
   for (size_t I = 0; I < NumMeasurements; ++I) {
-    auto ExpectedCounterValue = Executor.runAndMeasure(CounterName);
-    if (!ExpectedCounterValue)
-      return ExpectedCounterValue.takeError();
-    if (*ExpectedCounterValue < MinValue)
-      MinValue = *ExpectedCounterValue;
+    auto ExpectedCounterValues = Executor.runAndSample(CounterName);
+    if (!ExpectedCounterValues)
+      return ExpectedCounterValues.takeError();
+
+    // We'll keep the reading with lowest variance (ie., most stable)
+    double CurVariance = computeVariance(*ExpectedCounterValues);
+    if (Variance > CurVariance)
+      WithMinVariance = *ExpectedCounterValues;
+    Variance = CurVariance;
   }
-  std::vector<BenchmarkMeasure> Result;
+
+  std::string ModeName;
   switch (Mode) {
   case InstructionBenchmark::Latency:
-    Result = {BenchmarkMeasure::Create("latency", MinValue)};
+    ModeName = "latency";
     break;
   case InstructionBenchmark::InverseThroughput:
-    Result = {BenchmarkMeasure::Create("inverse_throughput", MinValue)};
+    ModeName = "inverse_throughput";
     break;
   default:
     break;
   }
+
+  switch (ResultAggMode) {
+  case InstructionBenchmark::MinVariance: {
+    std::vector<BenchmarkMeasure> Result;
+    Result.reserve(WithMinVariance.size());
+    for (const int64_t Value : WithMinVariance)
+      Result.push_back(BenchmarkMeasure::Create(ModeName, Value));
+    return std::move(Result);
+  }
+  case InstructionBenchmark::Min: {
+    std::vector<BenchmarkMeasure> Result;
+    Result.push_back(
+        BenchmarkMeasure::Create(ModeName, findMin(WithMinVariance)));
     return std::move(Result);
   }
+  case InstructionBenchmark::Max: {
+    std::vector<BenchmarkMeasure> Result;
+    Result.push_back(
+        BenchmarkMeasure::Create(ModeName, findMax(WithMinVariance)));
+    return std::move(Result);
+  }
+  }
+  return llvm::make_error<Failure>(llvm::Twine("Unexpected benchmark mode(")
+                                       .concat(std::to_string(Mode))
+                                       .concat(" and unexpected ResultAggMode ")
+                                       .concat(std::to_string(ResultAggMode)));
+}
 
 } // namespace exegesis
 } // namespace llvm

llvm/tools/llvm-exegesis/lib/PerfHelper.h

 /// Helpers for measuring perf events.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Config/config.h"
 #include "llvm/Support/Error.h"
+
 #include <cstdint>
 #include <functional>
 #include <memory>
 
 struct perf_event_attr;
 
 namespace llvm {
 namespace exegesis {
public:
 
   /// Stops the measurement of the event.
   void stop();
 
   /// Returns the current value of the counter or -1 if it cannot be read.
   int64_t read() const;
 
   /// Returns the current value of the counter or error if it cannot be read.
-  virtual llvm::Expected<int64_t> readOrError() const;
+  virtual llvm::Expected<llvm::SmallVector<int64_t, 4>> readOrError() const;
+
+  virtual int numValues() const;
 
 private:
   PerfEvent Event;
 #ifdef HAVE_LIBPFM
   int FileDescriptor = -1;
 #endif
 };
 
 } // namespace pfm
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_PERFHELPER_H

llvm/tools/llvm-exegesis/lib/PerfHelper.cpp

 
 Counter::~Counter() { close(FileDescriptor); }
 
 void Counter::start() { ioctl(FileDescriptor, PERF_EVENT_IOC_RESET, 0); }
 
 void Counter::stop() { ioctl(FileDescriptor, PERF_EVENT_IOC_DISABLE, 0); }
 
 int64_t Counter::read() const {
   auto ValueOrError = readOrError();
-  if (ValueOrError)
-    return ValueOrError.get();
-
+  if (ValueOrError) {
+    if (!ValueOrError.get().empty())
+      return ValueOrError.get()[0];
+    errs() << "Counter has no reading\n";
+  } else
     errs() << ValueOrError.takeError() << "\n";
   return -1;
 }
 
-llvm::Expected<int64_t> Counter::readOrError() const {
+llvm::Expected<llvm::SmallVector<int64_t, 4>> Counter::readOrError() const {
   int64_t Count = 0;
   ssize_t ReadSize = ::read(FileDescriptor, &Count, sizeof(Count));
   if (ReadSize != sizeof(Count))
     return llvm::make_error<llvm::StringError>("Failed to read event counter",
                                                llvm::errc::io_error);
-
-  return Count;
+  llvm::SmallVector<int64_t, 4> Result;
+  Result.push_back(Count);
+  return Result;
 }
 
+int Counter::numValues() const { return 1; }
 #else
 
 Counter::Counter(PerfEvent &&Event) : Event(std::move(Event)) {}
 
 Counter::~Counter() = default;
 
 void Counter::start() {}
 
 void Counter::stop() {}
 
 int64_t Counter::read() const { return 42; }
 
-llvm::Expected<int64_t> Counter::readOrError() const {
+llvm::Expected<llvm : SmallVector<int64_t, 4>> Counter::readOrError() const {
   return llvm::make_error<llvm::StringError>("Not implemented",
                                              llvm::errc::io_error);
 }
 
+int Counter::numValues() const { return 1; }
+
 #endif
 
 } // namespace pfm
 } // namespace exegesis
 } // namespace llvm

llvm/tools/llvm-exegesis/lib/Target.h

public:
   }
 
   // Creates a snippet generator for the given mode.
   std::unique_ptr<SnippetGenerator>
   createSnippetGenerator(InstructionBenchmark::ModeE Mode,
                          const LLVMState &State,
                          const SnippetGenerator::Options &Opts) const;
   // Creates a benchmark runner for the given mode.
-  Expected<std::unique_ptr<BenchmarkRunner>>
-  createBenchmarkRunner(InstructionBenchmark::ModeE Mode,
-                        const LLVMState &State) const;
+  Expected<std::unique_ptr<BenchmarkRunner>> createBenchmarkRunner(
+      InstructionBenchmark::ModeE Mode, const LLVMState &State,
+      InstructionBenchmark::ResultAggregationModeE ResultAggMode =
+          InstructionBenchmark::Min) const;
 
   // Returns the ExegesisTarget for the given triple or nullptr if the target
   // does not exist.
   static const ExegesisTarget *lookup(Triple TT);
   // Returns the default (unspecialized) ExegesisTarget.
   static const ExegesisTarget &getDefault();
   // Registers a target. Not thread safe.
   static void registerTarget(ExegesisTarget *T);
private:
 
   // Targets can implement their own snippet generators/benchmarks runners by
   // implementing these.
   std::unique_ptr<SnippetGenerator> virtual createSerialSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<SnippetGenerator> virtual createParallelSnippetGenerator(
       const LLVMState &State, const SnippetGenerator::Options &Opts) const;
   std::unique_ptr<BenchmarkRunner> virtual createLatencyBenchmarkRunner(
-      const LLVMState &State, InstructionBenchmark::ModeE Mode) const;
+      const LLVMState &State, InstructionBenchmark::ModeE Mode,
+      InstructionBenchmark::ResultAggregationModeE ResultAggMode) const;
   std::unique_ptr<BenchmarkRunner> virtual createUopsBenchmarkRunner(
       const LLVMState &State) const;
 
   const ExegesisTarget *Next = nullptr;
   const ArrayRef<CpuAndPfmCounters> CpuPfmCounters;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_TARGET_H

llvm/tools/llvm-exegesis/lib/Target.cpp

std::unique_ptr<SnippetGenerator> ExegesisTarget::createSnippetGenerator(
   case InstructionBenchmark::Uops:
   case InstructionBenchmark::InverseThroughput:
     return createParallelSnippetGenerator(State, Opts);
   }
   return nullptr;
 }
 
 Expected<std::unique_ptr<BenchmarkRunner>>
-ExegesisTarget::createBenchmarkRunner(InstructionBenchmark::ModeE Mode,
-                                      const LLVMState &State) const {
+ExegesisTarget::createBenchmarkRunner(
+    InstructionBenchmark::ModeE Mode, const LLVMState &State,
+    InstructionBenchmark::ResultAggregationModeE ResultAggMode) const {
+  // FIXME propagate ResultAggMode to other BenchmarkRunner.
   PfmCountersInfo PfmCounters = State.getPfmCounters();
   switch (Mode) {
   case InstructionBenchmark::Unknown:
     return nullptr;
   case InstructionBenchmark::Latency:
   case InstructionBenchmark::InverseThroughput:
     if (!PfmCounters.CycleCounter) {
       const char *ModeName = Mode == InstructionBenchmark::Latency
                                  ? "latency"
                                  : "inverse_throughput";
       return make_error<Failure>(
           Twine("can't run '")
               .concat(ModeName)
               .concat("' mode, sched model does not define a cycle counter."));
     }
-    return createLatencyBenchmarkRunner(State, Mode);
+    return createLatencyBenchmarkRunner(State, Mode, ResultAggMode);
   case InstructionBenchmark::Uops:
     if (!PfmCounters.UopsCounter && !PfmCounters.IssueCounters)
       return make_error<Failure>("can't run 'uops' mode, sched model does not "
                                  "define uops or issue counters.");
     return createUopsBenchmarkRunner(State);
   }
   return nullptr;
 }
 
 std::unique_ptr<SnippetGenerator> ExegesisTarget::createSerialSnippetGenerator(
     const LLVMState &State, const SnippetGenerator::Options &Opts) const {
   return std::make_unique<SerialSnippetGenerator>(State, Opts);
 }
 
 std::unique_ptr<SnippetGenerator> ExegesisTarget::createParallelSnippetGenerator(
     const LLVMState &State, const SnippetGenerator::Options &Opts) const {
   return std::make_unique<ParallelSnippetGenerator>(State, Opts);
 }
 
 std::unique_ptr<BenchmarkRunner> ExegesisTarget::createLatencyBenchmarkRunner(
-    const LLVMState &State, InstructionBenchmark::ModeE Mode) const {
-  return std::make_unique<LatencyBenchmarkRunner>(State, Mode);
+    const LLVMState &State, InstructionBenchmark::ModeE Mode,
+    InstructionBenchmark::ResultAggregationModeE ResultAggMode) const {
+  return std::make_unique<LatencyBenchmarkRunner>(State, Mode, ResultAggMode);
 }
 
 std::unique_ptr<BenchmarkRunner>
 ExegesisTarget::createUopsBenchmarkRunner(const LLVMState &State) const {
   return std::make_unique<UopsBenchmarkRunner>(State);
 }
 
 static_assert(std::is_pod<PfmCountersInfo>::value,
               "We shouldn't have dynamic initialization here");
 const PfmCountersInfo PfmCountersInfo::Default = {nullptr, nullptr, nullptr,
                                                   0u};
 
 const PfmCountersInfo &ExegesisTarget::getPfmCounters(StringRef CpuName) const {
   assert(llvm::is_sorted(
              CpuPfmCounters,
              [](const CpuAndPfmCounters &LHS, const CpuAndPfmCounters &RHS) {

llvm/tools/llvm-exegesis/llvm-exegesis.cpp

cl::values(clEnumValN(exegesis::InstructionBenchmark::Latency, "latency",
                           "Instruction Inverse Throughput"),
                clEnumValN(exegesis::InstructionBenchmark::Uops, "uops",
                           "Uop Decomposition"),
                // When not asking for a specific benchmark mode,
                // we'll analyse the results.
                clEnumValN(exegesis::InstructionBenchmark::Unknown, "analysis",
                           "Analysis")));
 
+static cl::opt<exegesis::InstructionBenchmark::ResultAggregationModeE>
+    ResultAggMode(
+        "result-aggregation-mode",
+        cl::desc("How to aggregate multi-values result"), cl::cat(Options),
+        cl::values(clEnumValN(exegesis::InstructionBenchmark::Min, "min",
+                              "Keep min reading"),
+                   clEnumValN(exegesis::InstructionBenchmark::Max, "max",
+                              "Keep max reading"),
+                   clEnumValN(exegesis::InstructionBenchmark::MinVariance,
+                              "min-variance",
+                              "Keep readings set with min-variance")),
+        cl::init(exegesis::InstructionBenchmark::Min));
+
 static cl::opt<exegesis::InstructionBenchmark::RepetitionModeE> RepetitionMode(
     "repetition-mode", cl::desc("how to repeat the instruction snippet"),
     cl::cat(BenchmarkOptions),
     cl::values(
         clEnumValN(exegesis::InstructionBenchmark::Duplicate, "duplicate",
                    "Duplicate the snippet"),
         clEnumValN(exegesis::InstructionBenchmark::Loop, "loop",
                    "Loop over the snippet"),
#endif
 
   InitializeNativeTarget();
   InitializeNativeTargetAsmPrinter();
   InitializeNativeTargetAsmParser();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
 
-  const std::unique_ptr<BenchmarkRunner> Runner = ExitOnErr(
-      State.getExegesisTarget().createBenchmarkRunner(BenchmarkMode, State));
+  const std::unique_ptr<BenchmarkRunner> Runner =
+      ExitOnErr(State.getExegesisTarget().createBenchmarkRunner(
+          BenchmarkMode, State, ResultAggMode));
   if (!Runner) {
     ExitWithError("cannot create benchmark runner");
   }
 
   const auto Opcodes = getOpcodesOrDie(State.getInstrInfo());
 
   SmallVector<std::unique_ptr<const SnippetRepetitor>, 2> Repetitors;
   if (RepetitionMode != InstructionBenchmark::RepetitionModeE::AggregateMin)