[llvm-exegesis] Add loop mode for repeating the snippet.

[courbet]

Summary

Before this change the Executable function was made by duplicating the
snippet. This change adds a --repetion-mode={loop|duplicate} flag that
allows choosing between this behaviour and wrapping the snippet instructions
in a loop.

The new mode can help measurements when the snippet fits in the DSB by
short-cirtcuiting decoding. The loop adds a dec + jmp to the measurements, but
since these are not part of the critical path, they execute in parallel
with the measured code and do not impact measurements in practice.

Overview of the change:

• New SnippetRepetitor abstraction that handles repeating the snippet. The assembler delegates repeating the instructions to this class.
• ExegesisTarget learns how to decrement loop counter and jump.
• Some refactoring of the assembler into FunctionFiller/BasicBlockFiller.

[lebedev.ri]

Can you point me to the test that actually checks the new assembly?

[courbet]

In D68125#1685579, @lebedev.ri wrote:

Can you point me to the test that actually checks the new assembly?

I was hesitant here: I only added integration tests there instead of unit tests because it felt like unit tests would really only be a repetition of the class internals. Given your comment, I'm going to add actual unit tests too.

[gchatelet]

llvm/tools/llvm-exegesis/lib/Assembler.cpp

122 ↗

(On Diff #222107)

remove braces

219 ↗

(On Diff #222107)

How about FillWith?

llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp

67 ↗

(On Diff #222107)

Can you extract the constant to a varaible?

78 ↗

(On Diff #222107)

Remove braces

81 ↗

(On Diff #222107)

ditto

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

507 ↗

(On Diff #222107)

Add a comment on why R8

557 ↗

(On Diff #222107)

Maybe use a constant instead of repeating R8 everywhere.

[courbet]

Add unit tests for SnippetRepetitor.

[courbet]

More tests: check live ins.

[courbet]

Address comments.

[courbet]

Thanks.

[thakis]

This upsets gcc: http://lab.llvm.org:8011/builders/lld-x86_64-ubuntu-fast/builds/6437/steps/build-unified-tree/logs/stdio

/home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/unittests/tools/llvm-exegesis/X86/SnippetRepetitorTest.cpp:66:27: error: declaration of ‘std::unique_ptr<llvm::Module> llvm::exegesis::{anonymous}::X86SnippetRepetitorTest::Module’ [-fpermissive]
   std::unique_ptr<Module> Module;
                           ^~~~~~
In file included from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/Analysis/TargetLibraryInfo.h:17:0,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/Analysis/AliasAnalysis.h:45,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/CodeGen/MachineInstr.h:23,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/CodeGen/MachineBasicBlock.h:21,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/CodeGen/MachineFunction.h:30,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/tools/llvm-exegesis/lib/Assembler.h:23,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/unittests/tools/llvm-exegesis/X86/../Common/AssemblerUtils.h:12,
                 from /home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/unittests/tools/llvm-exegesis/X86/SnippetRepetitorTest.cpp:9:
/home/buildbot/as-builder-4/lld-x86_64-ubuntu-fast/llvm/include/llvm/IR/Module.h:65:7: error: changes meaning of ‘Module’ from ‘class llvm::Module’ [-fpermissive]
 class Module {
       ^~~~~~

llvm/trunk/test/tools/llvm-exegesis/X86/inverse_throughput-by-opcode-name.s

-# RUN: llvm-exegesis -mode=inverse_throughput -opcode-name=ADD32rr | FileCheck %s
+# RUN: llvm-exegesis -mode=inverse_throughput -opcode-name=ADD32rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=inverse_throughput -opcode-name=ADD32rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      ---
 CHECK-NEXT: mode: inverse_throughput
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32rr
 CHECK: key: inverse_throughput

llvm/trunk/test/tools/llvm-exegesis/X86/latency-CMOV32rr.s

-# RUN: llvm-exegesis -mode=latency -opcode-name=CMOV32rr | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=CMOV32rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=CMOV32rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      ---
 CHECK-NEXT: mode: latency
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     'CMOV32rr {{.*}} i_0x{{[0-9a-f]}}'
 CHECK-NEXT: config: ''
 CHECK-LAST: ...

llvm/trunk/test/tools/llvm-exegesis/X86/latency-SBB8rr.s

-# RUN: llvm-exegesis -mode=latency -opcode-name=SBB8rr | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=SBB8rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=SBB8rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      ---
 CHECK-NEXT: mode: latency
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     SBB8rr
 CHECK-NEXT: config: ''
 CHECK-NEXT: register_initial_values:
 CHECK-DAG: - '[[REG1:[A-Z0-9]+]]=0x0'
 CHECK-LAST: ...

llvm/trunk/test/tools/llvm-exegesis/X86/latency-by-opcode-name.s

-# RUN: llvm-exegesis -mode=latency -opcode-name=ADD32rr | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=ADD32rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=latency -opcode-name=ADD32rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      ---
 CHECK-NEXT: mode: latency
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32rr
 CHECK-NEXT: config: ''
 CHECK-NEXT: register_initial_values:
 CHECK-DAG: - '[[REG1:[A-Z0-9]+]]=0x0'
 CHECK-LAST: ...

llvm/trunk/test/tools/llvm-exegesis/X86/uops-ADD32mi8.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mi8 | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mi8 -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mi8 -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32mi8

llvm/trunk/test/tools/llvm-exegesis/X86/uops-ADD32mr.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mr | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32mr -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32mr

llvm/trunk/test/tools/llvm-exegesis/X86/uops-ADD32rm.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rm | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rm -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rm -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32rm

llvm/trunk/test/tools/llvm-exegesis/X86/uops-BEXTR32rm.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=BEXTR32rm | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BEXTR32rm -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BEXTR32rm -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     BEXTR32rm

llvm/trunk/test/tools/llvm-exegesis/X86/uops-BSF16rm.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=BSF16rm | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BSF16rm -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BSF16rm -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     BSF16rm

llvm/trunk/test/tools/llvm-exegesis/X86/uops-BTR64mr.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=BTR64mr | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BTR64mr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=BTR64mr -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     BTR64mr

llvm/trunk/test/tools/llvm-exegesis/X86/uops-VFMADDSS4rm.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=VFMADDSS4rm | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=VFMADDSS4rm -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=VFMADDSS4rm -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     VFMADDSS4rm

llvm/trunk/test/tools/llvm-exegesis/X86/uops-XCHG64rr.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=XCHG64rr | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=XCHG64rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=XCHG64rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     XCHG64rr

llvm/trunk/test/tools/llvm-exegesis/X86/uops-by-opcode-name.s

-# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rr | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rr -repetition-mode=duplicate | FileCheck %s
+# RUN: llvm-exegesis -mode=uops -opcode-name=ADD32rr -repetition-mode=loop | FileCheck %s
 
 CHECK:      mode:            uops
 CHECK-NEXT: key:
 CHECK-NEXT:   instructions:
 CHECK-NEXT:     ADD32rr

llvm/trunk/tools/llvm-exegesis/lib/Assembler.h

 namespace exegesis {
 
 class ExegesisTarget;
 
 // Gather the set of reserved registers (depends on function's calling
 // convention and target machine).
 llvm::BitVector getFunctionReservedRegs(const llvm::TargetMachine &TM);
 
+// Helper to fill in a basic block.
+class BasicBlockFiller {
+public:
+  BasicBlockFiller(MachineFunction &MF, MachineBasicBlock *MBB,
+                   const MCInstrInfo *MCII);
+
+  void addInstruction(const MCInst &Inst, const DebugLoc &DL = DebugLoc());
+  void addInstructions(ArrayRef<llvm::MCInst> Insts,
+                       const DebugLoc &DL = DebugLoc());
+
+  void addReturn(const DebugLoc &DL = DebugLoc());
+
+  MachineFunction &MF;
+  MachineBasicBlock *const MBB;
+  const MCInstrInfo *const MCII;
+};
+
+// Helper to fill in a function.
+class FunctionFiller {
+public:
+  FunctionFiller(MachineFunction &MF, std::vector<unsigned> RegistersSetUp);
+
+  // Adds a basic block to the function.
+  BasicBlockFiller addBasicBlock();
+
+  // Returns the function entry point.
+  BasicBlockFiller getEntry() { return Entry; }
+
+  MachineFunction &MF;
+  const MCInstrInfo *const MCII;
+
+  // Returns the set of registers in the snippet setup code.
+  ArrayRef<unsigned> getRegistersSetUp() const;
+
+private:
+  BasicBlockFiller Entry;
+  // The set of registers that are set up in the basic block.
+  std::vector<unsigned> RegistersSetUp;
+};
+
+// A callback that fills a function.
+using FillFunction = std::function<void(FunctionFiller &)>;
+
 // Creates a temporary `void foo(char*)` function containing the provided
 // Instructions. Runs a set of llvm Passes to provide correct prologue and
 // epilogue. Once the MachineFunction is ready, it is assembled for TM to
 // AsmStream, the temporary function is eventually discarded.
 void assembleToStream(const ExegesisTarget &ET,
                       std::unique_ptr<llvm::LLVMTargetMachine> TM,
                       llvm::ArrayRef<unsigned> LiveIns,
                       llvm::ArrayRef<RegisterValue> RegisterInitialValues,
-                      llvm::ArrayRef<llvm::MCInst> Instructions,
+                      const FillFunction &Fill,
                       llvm::raw_pwrite_stream &AsmStream);
 
 // Creates an ObjectFile in the format understood by the host.
 // Note: the resulting object keeps a copy of Buffer so it can be discarded once
 // this function returns.
 llvm::object::OwningBinary<llvm::object::ObjectFile>
 getObjectFromBuffer(llvm::StringRef Buffer);
 
void operator()(char *Memory) const {
     ((void (*)(char *))(intptr_t)FunctionBytes.data())(Memory);
   }
 
   std::unique_ptr<llvm::LLVMContext> Context;
   std::unique_ptr<llvm::ExecutionEngine> ExecEngine;
   llvm::StringRef FunctionBytes;
 };
 
+// Creates a void(int8*) MachineFunction.
+llvm::MachineFunction &
+createVoidVoidPtrMachineFunction(llvm::StringRef FunctionID,
+                                 llvm::Module *Module,
+                                 llvm::MachineModuleInfo *MMI);
+
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_ASSEMBLER_H

llvm/trunk/tools/llvm-exegesis/lib/Assembler.cpp

 //===-- Assembler.cpp -------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Assembler.h"
 
+#include "SnippetRepetitor.h"
 #include "Target.h"
 #include "llvm/CodeGen/GlobalISel/CallLowering.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/ExecutionEngine/SectionMemoryManager.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/MC/MCInstrInfo.h"
 #include "llvm/Support/MemoryBuffer.h"
 
 namespace llvm {
 namespace exegesis {
 
 static constexpr const char ModuleID[] = "ExegesisInfoTest";
 static constexpr const char FunctionID[] = "foo";
 
-static std::vector<llvm::MCInst>
+// Fills the given basic block with register setup code, and returns true if
+// all registers could be setup correctly.
+static bool
 generateSnippetSetupCode(const ExegesisTarget &ET,
                          const llvm::MCSubtargetInfo *const MSI,
                          llvm::ArrayRef<RegisterValue> RegisterInitialValues,
-                         bool &IsSnippetSetupComplete) {
-  IsSnippetSetupComplete = true;
-  std::vector<llvm::MCInst> Result;
+                         BasicBlockFiller &BBF) {
+  bool IsSnippetSetupComplete = true;
   for (const RegisterValue &RV : RegisterInitialValues) {
     // Load a constant in the register.
     const auto SetRegisterCode = ET.setRegTo(*MSI, RV.Register, RV.Value);
     if (SetRegisterCode.empty())
       IsSnippetSetupComplete = false;
-    Result.insert(Result.end(), SetRegisterCode.begin(), SetRegisterCode.end());
+    BBF.addInstructions(SetRegisterCode);
   }
-  return Result;
+  return IsSnippetSetupComplete;
 }
 
 // Small utility function to add named passes.
 static bool addPass(llvm::PassManagerBase &PM, llvm::StringRef PassName,
                     llvm::TargetPassConfig &TPC) {
   const llvm::PassRegistry *PR = llvm::PassRegistry::getPassRegistry();
   const llvm::PassInfo *PI = PR->getPassInfo(PassName);
   if (!PI) {
     llvm::errs() << " run-pass " << PassName << " is not registered.\n";
     return true;
   }
 
   if (!PI->getNormalCtor()) {
     llvm::errs() << " cannot create pass: " << PI->getPassName() << "\n";
     return true;
   }
   llvm::Pass *P = PI->getNormalCtor()();
   std::string Banner = std::string("After ") + std::string(P->getPassName());
   PM.add(P);
   TPC.printAndVerify(Banner);
 
   return false;
 }
 
-// Creates a void(int8*) MachineFunction.
-static llvm::MachineFunction &
+llvm::MachineFunction &
 createVoidVoidPtrMachineFunction(llvm::StringRef FunctionID,
                                  llvm::Module *Module,
                                  llvm::MachineModuleInfo *MMI) {
   llvm::Type *const ReturnType = llvm::Type::getInt32Ty(Module->getContext());
   llvm::Type *const MemParamType = llvm::PointerType::get(
       llvm::Type::getInt8Ty(Module->getContext()), 0 /*default address space*/);
   llvm::FunctionType *FunctionType =
       llvm::FunctionType::get(ReturnType, {MemParamType}, false);
   llvm::Function *const F = llvm::Function::Create(
       FunctionType, llvm::GlobalValue::InternalLinkage, FunctionID, Module);
   // Making sure we can create a MachineFunction out of this Function even if it
   // contains no IR.
   F->setIsMaterializable(true);
   return MMI->getOrCreateMachineFunction(*F);
 }
 
-static void fillMachineFunction(llvm::MachineFunction &MF,
-                                llvm::ArrayRef<unsigned> LiveIns,
-                                llvm::ArrayRef<llvm::MCInst> Instructions) {
-  llvm::MachineBasicBlock *MBB = MF.CreateMachineBasicBlock();
-  MF.push_back(MBB);
-  for (const unsigned Reg : LiveIns)
-    MBB->addLiveIn(Reg);
-  const llvm::MCInstrInfo *MCII = MF.getTarget().getMCInstrInfo();
-  llvm::DebugLoc DL;
-  for (const llvm::MCInst &Inst : Instructions) {
-    const unsigned Opcode = Inst.getOpcode();
-    const llvm::MCInstrDesc &MCID = MCII->get(Opcode);
-    llvm::MachineInstrBuilder Builder = llvm::BuildMI(MBB, DL, MCID);
-    for (unsigned OpIndex = 0, E = Inst.getNumOperands(); OpIndex < E;
-         ++OpIndex) {
-      const llvm::MCOperand &Op = Inst.getOperand(OpIndex);
-      if (Op.isReg()) {
-        const bool IsDef = OpIndex < MCID.getNumDefs();
-        unsigned Flags = 0;
-        const llvm::MCOperandInfo &OpInfo = MCID.operands().begin()[OpIndex];
-        if (IsDef && !OpInfo.isOptionalDef())
-          Flags |= llvm::RegState::Define;
-        Builder.addReg(Op.getReg(), Flags);
-      } else if (Op.isImm()) {
-        Builder.addImm(Op.getImm());
-      } else if (!Op.isValid()) {
-        llvm_unreachable("Operand is not set");
-      } else {
-        llvm_unreachable("Not yet implemented");
-      }
-    }
-  }
+BasicBlockFiller::BasicBlockFiller(llvm::MachineFunction &MF,
+                                   llvm::MachineBasicBlock *MBB,
+                                   const llvm::MCInstrInfo *MCII)
+    : MF(MF), MBB(MBB), MCII(MCII) {}
+
+void BasicBlockFiller::addInstruction(const llvm::MCInst &Inst,
+                                      const llvm::DebugLoc &DL) {
+
+void BasicBlockFiller::addInstructions(ArrayRef<llvm::MCInst> Insts,
+                                       const llvm::DebugLoc &DL) {
+  for (const MCInst &Inst : Insts)
+    addInstruction(Inst, DL);
+}
+
+void BasicBlockFiller::addReturn(const llvm::DebugLoc &DL) {
   // Insert the return code.
   const llvm::TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
   if (TII->getReturnOpcode() < TII->getNumOpcodes()) {
     llvm::BuildMI(MBB, DL, TII->get(TII->getReturnOpcode()));
   } else {
     llvm::MachineIRBuilder MIB(MF);
     MIB.setMBB(*MBB);
     MF.getSubtarget().getCallLowering()->lowerReturn(MIB, nullptr, {});
   }
 }
 
+FunctionFiller::FunctionFiller(llvm::MachineFunction &MF,
+                               std::vector<unsigned> RegistersSetUp)
+    : MF(MF), MCII(MF.getTarget().getMCInstrInfo()), Entry(addBasicBlock()),
+      RegistersSetUp(std::move(RegistersSetUp)) {}
+
+BasicBlockFiller FunctionFiller::addBasicBlock() {
+  llvm::MachineBasicBlock *MBB = MF.CreateMachineBasicBlock();
+  MF.push_back(MBB);
+  return BasicBlockFiller(MF, MBB, MCII);
+}
+
+ArrayRef<unsigned> FunctionFiller::getRegistersSetUp() const {
+  return RegistersSetUp;
+}
+
 static std::unique_ptr<llvm::Module>
 createModule(const std::unique_ptr<llvm::LLVMContext> &Context,
              const llvm::DataLayout DL) {
   auto Module = std::make_unique<llvm::Module>(ModuleID, *Context);
   Module->setDataLayout(DL);
   return Module;
 }
 
llvm::BitVector getFunctionReservedRegs(const llvm::TargetMachine &TM) {
   // Saving reserved registers for client.
   return MF.getSubtarget().getRegisterInfo()->getReservedRegs(MF);
 }
 
 void assembleToStream(const ExegesisTarget &ET,
                       std::unique_ptr<llvm::LLVMTargetMachine> TM,
                       llvm::ArrayRef<unsigned> LiveIns,
                       llvm::ArrayRef<RegisterValue> RegisterInitialValues,
-                      llvm::ArrayRef<llvm::MCInst> Instructions,
+                      const FillFunction &Fill,
                       llvm::raw_pwrite_stream &AsmStream) {
   std::unique_ptr<llvm::LLVMContext> Context =
       std::make_unique<llvm::LLVMContext>();
   std::unique_ptr<llvm::Module> Module =
       createModule(Context, TM->createDataLayout());
   std::unique_ptr<llvm::MachineModuleInfo> MMI =
       std::make_unique<llvm::MachineModuleInfo>(TM.get());
   llvm::MachineFunction &MF =
       createVoidVoidPtrMachineFunction(FunctionID, Module.get(), MMI.get());
 
   // We need to instruct the passes that we're done with SSA and virtual
   // registers.
   auto &Properties = MF.getProperties();
   Properties.set(llvm::MachineFunctionProperties::Property::NoVRegs);
   Properties.reset(llvm::MachineFunctionProperties::Property::IsSSA);
+  Properties.set(llvm::MachineFunctionProperties::Property::NoPHIs);
 
   for (const unsigned Reg : LiveIns)
     MF.getRegInfo().addLiveIn(Reg);
 
-  bool IsSnippetSetupComplete;
-  std::vector<llvm::MCInst> Code =
-      generateSnippetSetupCode(ET, TM->getMCSubtargetInfo(),
-                               RegisterInitialValues, IsSnippetSetupComplete);
+  std::vector<unsigned> RegistersSetUp;
+  for (const auto &InitValue : RegisterInitialValues) {
+    RegistersSetUp.push_back(InitValue.Register);
+  }
+  FunctionFiller Sink(MF, std::move(RegistersSetUp));
+  auto Entry = Sink.getEntry();
+  for (const unsigned Reg : LiveIns)
+    Entry.MBB->addLiveIn(Reg);
 
-  Code.insert(Code.end(), Instructions.begin(), Instructions.end());
+  const bool IsSnippetSetupComplete = generateSnippetSetupCode(
+      ET, TM->getMCSubtargetInfo(), RegisterInitialValues, Entry);
 
   // If the snippet setup is not complete, we disable liveliness tracking. This
   // means that we won't know what values are in the registers.
   if (!IsSnippetSetupComplete)
     Properties.reset(llvm::MachineFunctionProperties::Property::TracksLiveness);
 
+  Fill(Sink);
+
   // prologue/epilogue pass needs the reserved registers to be frozen, this
   // is usually done by the SelectionDAGISel pass.
   MF.getRegInfo().freezeReservedRegs(MF);
 
-  // Fill the MachineFunction from the instructions.
-  fillMachineFunction(MF, LiveIns, Code);
-
   // We create the pass manager, run the passes to populate AsmBuffer.
   llvm::MCContext &MCContext = MMI->getContext();
   llvm::legacy::PassManager PM;
 
   llvm::TargetLibraryInfoImpl TLII(llvm::Triple(Module->getTargetTriple()));
   PM.add(new llvm::TargetLibraryInfoWrapperPass(TLII));
 
   llvm::TargetPassConfig *TPC = TM->createPassConfig(PM);

llvm/trunk/tools/llvm-exegesis/lib/BenchmarkResult.h

struct InstructionBenchmark {
   ModeE Mode;
   std::string CpuName;
   std::string LLVMTriple;
   // Which instruction is being benchmarked here?
   const llvm::MCInst &keyInstruction() const { return Key.Instructions[0]; }
   // The number of instructions inside the repeated snippet. For example, if a
   // snippet of 3 instructions is repeated 4 times, this is 12.
   int NumRepetitions = 0;
+  enum RepetitionModeE { Duplicate, Loop };
+  RepetitionModeE RepetitionMode;
   // Note that measurements are per instruction.
   std::vector<BenchmarkMeasure> Measurements;
   std::string Error;
   std::string Info;
   std::vector<uint8_t> AssembledSnippet;
 
   // Read functions.
   static llvm::Expected<InstructionBenchmark>

llvm/trunk/tools/llvm-exegesis/lib/BenchmarkRunner.h

 #ifndef LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 #define LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H
 
 #include "Assembler.h"
 #include "BenchmarkCode.h"
 #include "BenchmarkResult.h"
 #include "LlvmState.h"
 #include "MCInstrDescView.h"
+#include "SnippetRepetitor.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/Error.h"
 #include <cstdlib>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace exegesis {

 public:
   explicit BenchmarkRunner(const LLVMState &State,
                            InstructionBenchmark::ModeE Mode);
 
   virtual ~BenchmarkRunner();
 
   InstructionBenchmark runConfiguration(const BenchmarkCode &Configuration,
                                         unsigned NumRepetitions,
+                                        const SnippetRepetitor &Repetitor,
                                         bool DumpObjectToDisk) const;
 
   // Scratch space to run instructions that touch memory.
   struct ScratchSpace {
     static constexpr const size_t kAlignment = 1024;
     static constexpr const size_t kSize = 1 << 20; // 1MB.
     ScratchSpace()
         : UnalignedPtr(std::make_unique<char[]>(kSize + kAlignment)),
protected:
   const InstructionBenchmark::ModeE Mode;
 
 private:
   virtual llvm::Expected<std::vector<BenchmarkMeasure>>
   runMeasurements(const FunctionExecutor &Executor) const = 0;
 
   llvm::Expected<std::string>
   writeObjectFile(const BenchmarkCode &Configuration,
-                  llvm::ArrayRef<llvm::MCInst> Code) const;
+                  const FillFunction &Fill) const;
 
   const std::unique_ptr<ScratchSpace> Scratch;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif // LLVM_TOOLS_LLVM_EXEGESIS_BENCHMARKRUNNER_H

llvm/trunk/tools/llvm-exegesis/lib/BenchmarkRunner.cpp

BenchmarkFailure::BenchmarkFailure(const llvm::Twine &S)
     : llvm::StringError(S, llvm::inconvertibleErrorCode()) {}
 
 BenchmarkRunner::BenchmarkRunner(const LLVMState &State,
                                  InstructionBenchmark::ModeE Mode)
     : State(State), Mode(Mode), Scratch(std::make_unique<ScratchSpace>()) {}
 
 BenchmarkRunner::~BenchmarkRunner() = default;
 
-// Repeat the snippet until there are at least MinInstructions in the resulting
-// code.
-static std::vector<llvm::MCInst>
-GenerateInstructions(const BenchmarkCode &BC, const size_t MinInstructions) {
-  if (BC.Instructions.empty())
-    return {};
-  std::vector<llvm::MCInst> Code = BC.Instructions;
-  for (int I = 0; Code.size() < MinInstructions; ++I)
-    Code.push_back(BC.Instructions[I % BC.Instructions.size()]);
-  return Code;
-}
 
 namespace {
 class FunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
 public:
   FunctionExecutorImpl(const LLVMState &State,
                        llvm::object::OwningBinary<llvm::object::ObjectFile> Obj,
                        BenchmarkRunner::ScratchSpace *Scratch)
       : Function(State.createTargetMachine(), std::move(Obj)),
llvm::Expected<int64_t> runAndMeasure(const char *Counters) const override {
     return CounterValue;
   }
 
   const ExecutableFunction Function;
   BenchmarkRunner::ScratchSpace *const Scratch;
 };
 } // namespace
 
-InstructionBenchmark
-BenchmarkRunner::runConfiguration(const BenchmarkCode &BC,
-                                  unsigned NumRepetitions,
-                                  bool DumpObjectToDisk) const {
+InstructionBenchmark BenchmarkRunner::runConfiguration(
+    const BenchmarkCode &BC, unsigned NumRepetitions,
+    const SnippetRepetitor &Repetitor, bool DumpObjectToDisk) const {
   InstructionBenchmark InstrBenchmark;
   InstrBenchmark.Mode = Mode;
   InstrBenchmark.CpuName = State.getTargetMachine().getTargetCPU();
   InstrBenchmark.LLVMTriple =
       State.getTargetMachine().getTargetTriple().normalize();
   InstrBenchmark.NumRepetitions = NumRepetitions;
   InstrBenchmark.Info = BC.Info;
 
   const std::vector<llvm::MCInst> &Instructions = BC.Instructions;
 
   InstrBenchmark.Key.Instructions = Instructions;
   InstrBenchmark.Key.RegisterInitialValues = BC.RegisterInitialValues;
 
   // Assemble at least kMinInstructionsForSnippet instructions by repeating the
   // snippet for debug/analysis. This is so that the user clearly understands
   // that the inside instructions are repeated.
   constexpr const int kMinInstructionsForSnippet = 16;
   {
     llvm::SmallString<0> Buffer;
     llvm::raw_svector_ostream OS(Buffer);
-    assembleToStream(State.getExegesisTarget(), State.createTargetMachine(),
-                     BC.LiveIns, BC.RegisterInitialValues,
-                     GenerateInstructions(BC, kMinInstructionsForSnippet), OS);
+    assembleToStream(
+        State.getExegesisTarget(), State.createTargetMachine(), BC.LiveIns,
+        BC.RegisterInitialValues,
+        Repetitor.Repeat(BC.Instructions, kMinInstructionsForSnippet), OS);
     const ExecutableFunction EF(State.createTargetMachine(),
                                 getObjectFromBuffer(OS.str()));
     const auto FnBytes = EF.getFunctionBytes();
     InstrBenchmark.AssembledSnippet.assign(FnBytes.begin(), FnBytes.end());
   }
 
   // Assemble NumRepetitions instructions repetitions of the snippet for
   // measurements.
-  const auto Code = GenerateInstructions(BC, InstrBenchmark.NumRepetitions);
+  const auto Filler =
+      Repetitor.Repeat(BC.Instructions, InstrBenchmark.NumRepetitions);
 
   llvm::object::OwningBinary<llvm::object::ObjectFile> ObjectFile;
   if (DumpObjectToDisk) {
-    auto ObjectFilePath = writeObjectFile(BC, Code);
+    auto ObjectFilePath = writeObjectFile(BC, Filler);
     if (llvm::Error E = ObjectFilePath.takeError()) {
       InstrBenchmark.Error = llvm::toString(std::move(E));
       return InstrBenchmark;
     }
     llvm::outs() << "Check generated assembly with: /usr/bin/objdump -d "
                  << *ObjectFilePath << "\n";
     ObjectFile = getObjectFromFile(*ObjectFilePath);
   } else {
     llvm::SmallString<0> Buffer;
     llvm::raw_svector_ostream OS(Buffer);
     assembleToStream(State.getExegesisTarget(), State.createTargetMachine(),
-                     BC.LiveIns, BC.RegisterInitialValues, Code, OS);
+                     BC.LiveIns, BC.RegisterInitialValues, Filler, OS);
     ObjectFile = getObjectFromBuffer(OS.str());
   }
 
   const FunctionExecutorImpl Executor(State, std::move(ObjectFile),
                                       Scratch.get());
   auto Measurements = runMeasurements(Executor);
   if (llvm::Error E = Measurements.takeError()) {
     InstrBenchmark.Error = llvm::toString(std::move(E));
BM.PerSnippetValue *= static_cast<double>(BC.Instructions.size()) /
                           InstrBenchmark.NumRepetitions;
   }
 
   return InstrBenchmark;
 }
 
 llvm::Expected<std::string>
 BenchmarkRunner::writeObjectFile(const BenchmarkCode &BC,
-                                 llvm::ArrayRef<llvm::MCInst> Code) const {
+                                 const FillFunction &FillFunction) const {
   int ResultFD = 0;
   llvm::SmallString<256> ResultPath;
   if (llvm::Error E = llvm::errorCodeToError(llvm::sys::fs::createTemporaryFile(
           "snippet", "o", ResultFD, ResultPath)))
     return std::move(E);
   llvm::raw_fd_ostream OFS(ResultFD, true /*ShouldClose*/);
   assembleToStream(State.getExegesisTarget(), State.createTargetMachine(),
-                   BC.LiveIns, BC.RegisterInitialValues, Code, OFS);
+                   BC.LiveIns, BC.RegisterInitialValues, FillFunction, OFS);
   return ResultPath.str();
 }
 
 BenchmarkRunner::FunctionExecutor::~FunctionExecutor() {}
 
 } // namespace exegesis
 } // namespace llvm

llvm/trunk/tools/llvm-exegesis/lib/CMakeLists.txt

add_library(LLVMExegesis
   Latency.cpp
   LlvmState.cpp
   MCInstrDescView.cpp
   PerfHelper.cpp
   RegisterAliasing.cpp
   RegisterValue.cpp
   SchedClassResolution.cpp
   SnippetGenerator.cpp
+  SnippetRepetitor.cpp
   Target.cpp
   Uops.cpp
   )
 
 llvm_update_compile_flags(LLVMExegesis)
 llvm_map_components_to_libnames(libs
   Analysis
   CodeGen

llvm/trunk/tools/llvm-exegesis/lib/SnippetGenerator.h

 class SnippetGenerator {
 public:
   explicit SnippetGenerator(const LLVMState &State);
 
   virtual ~SnippetGenerator();
 
   // Calls generateCodeTemplate and expands it into one or more BenchmarkCode.
   llvm::Expected<std::vector<BenchmarkCode>>
-  generateConfigurations(const Instruction &Instr) const;
+  generateConfigurations(const Instruction &Instr,
+                         const llvm::BitVector &ExtraForbiddenRegs) const;
 
   // Given a snippet, computes which registers the setup code needs to define.
   std::vector<RegisterValue> computeRegisterInitialValues(
       const std::vector<InstructionTemplate> &Snippet) const;
 
 protected:
   const LLVMState &State;
 

llvm/trunk/tools/llvm-exegesis/lib/SnippetGenerator.cpp

 SnippetGeneratorFailure::SnippetGeneratorFailure(const llvm::Twine &S)
     : llvm::StringError(S, llvm::inconvertibleErrorCode()) {}
 
 SnippetGenerator::SnippetGenerator(const LLVMState &State) : State(State) {}
 
 SnippetGenerator::~SnippetGenerator() = default;
 
 llvm::Expected<std::vector<BenchmarkCode>>
-SnippetGenerator::generateConfigurations(const Instruction &Instr) const {
+SnippetGenerator::generateConfigurations(
+    const Instruction &Instr, const llvm::BitVector &ExtraForbiddenRegs) const {
   llvm::BitVector ForbiddenRegs = State.getRATC().reservedRegisters();
-
+  ForbiddenRegs |= ExtraForbiddenRegs;
   // If the instruction has memory registers, prevent the generator from
   // using the scratch register and its aliasing registers.
   if (Instr.hasMemoryOperands()) {
     const auto &ET = State.getExegesisTarget();
     unsigned ScratchSpacePointerInReg =
         ET.getScratchMemoryRegister(State.getTargetMachine().getTargetTriple());
     if (ScratchSpacePointerInReg == 0)
       return llvm::make_error<BenchmarkFailure>(

llvm/trunk/tools/llvm-exegesis/lib/SnippetRepetitor.h

+//===-- SnippetRepetitor.h --------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Defines helpers to fill functions with repetitions of a snippet.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_EXEGESIS_FUNCTIONFILLER_H
+#define LLVM_TOOLS_LLVM_EXEGESIS_FUNCTIONFILLER_H
+
+#include "Assembler.h"
+#include "BenchmarkResult.h"
+#include "LlvmState.h"
+#include "llvm/ADT/BitVector.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/MC/MCInst.h"
+#include "llvm/MC/MCInstrInfo.h"
+#include "llvm/Object/Binary.h"
+
+namespace llvm {
+namespace exegesis {
+
+class SnippetRepetitor {
+public:
+  static std::unique_ptr<const SnippetRepetitor>
+  Create(InstructionBenchmark::RepetitionModeE Mode, const LLVMState &State);
+
+  virtual ~SnippetRepetitor();
+
+  // Returns the set of registers that are reserved by the repetitor.
+  virtual BitVector getReservedRegs() const = 0;
+
+  // Returns a functor that repeats `Instructions` so that the function executes
+  // at least `MinInstructions` instructions.
+  virtual FillFunction Repeat(ArrayRef<MCInst> Instructions,
+                              unsigned MinInstructions) const = 0;
+
+  explicit SnippetRepetitor(const LLVMState &State) : State(State) {}
+
+protected:
+  const LLVMState &State;
+};
+
+} // namespace exegesis
+} // namespace llvm
+
+#endif

llvm/trunk/tools/llvm-exegesis/lib/SnippetRepetitor.cpp

+//===-- SnippetRepetitor.cpp ------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <array>
+#include <string>
+
+#include "SnippetRepetitor.h"
+#include "Target.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+
+namespace llvm {
+namespace exegesis {
+namespace {
+
+class DuplicateSnippetRepetitor : public SnippetRepetitor {
+public:
+  using SnippetRepetitor::SnippetRepetitor;
+
+  // Repeats the snippet until there are at least MinInstructions in the
+  // resulting code.
+  FillFunction Repeat(ArrayRef<MCInst> Instructions,
+                      unsigned MinInstructions) const override {
+    return [Instructions, MinInstructions](FunctionFiller &Filler) {
+      auto Entry = Filler.getEntry();
+      if (!Instructions.empty()) {
+        // Add the whole snippet at least once.
+        Entry.addInstructions(Instructions);
+        for (unsigned I = Instructions.size(); I < MinInstructions; ++I) {
+          Entry.addInstruction(Instructions[I % Instructions.size()]);
+        }
+      }
+      Entry.addReturn();
+    };
+  }
+
+  BitVector getReservedRegs() const override {
+    // We're using no additional registers.
+    return State.getRATC().emptyRegisters();
+  }
+};
+
+class LoopSnippetRepetitor : public SnippetRepetitor {
+public:
+  explicit LoopSnippetRepetitor(const LLVMState &State)
+      : SnippetRepetitor(State),
+        LoopCounter(State.getExegesisTarget().getLoopCounterRegister(
+            State.getTargetMachine().getTargetTriple())) {}
+
+  // Loop over the snippet ceil(MinInstructions / Instructions.Size()) times.
+  FillFunction Repeat(ArrayRef<MCInst> Instructions,
+                      unsigned MinInstructions) const override {
+    return [this, Instructions, MinInstructions](FunctionFiller &Filler) {
+      const auto &ET = State.getExegesisTarget();
+      auto Entry = Filler.getEntry();
+      auto Loop = Filler.addBasicBlock();
+      auto Exit = Filler.addBasicBlock();
+
+      // Set loop counter to the right value:
+      const APInt LoopCount(32, (MinInstructions + Instructions.size() - 1) /
+                                    Instructions.size());
+      for (const MCInst &Inst :
+           ET.setRegTo(State.getSubtargetInfo(), LoopCounter, LoopCount))
+        Entry.addInstruction(Inst);
+
+      // Set up the loop basic block.
+      Entry.MBB->addSuccessor(Loop.MBB, llvm::BranchProbability::getOne());
+      Loop.MBB->addSuccessor(Loop.MBB, llvm::BranchProbability::getOne());
+      // The live ins are: the loop counter, the registers that were setup by
+      // the entry block, and entry block live ins.
+      Loop.MBB->addLiveIn(LoopCounter);
+      for (unsigned Reg : Filler.getRegistersSetUp())
+        Loop.MBB->addLiveIn(Reg);
+      for (const auto &LiveIn : Entry.MBB->liveins())
+        Loop.MBB->addLiveIn(LiveIn);
+      Loop.addInstructions(Instructions);
+      ET.decrementLoopCounterAndLoop(*Loop.MBB, State.getInstrInfo());
+
+      // Set up the exit basic block.
+      Loop.MBB->addSuccessor(Exit.MBB, llvm::BranchProbability::getZero());
+      Exit.addReturn();
+    };
+  }
+
+  BitVector getReservedRegs() const override {
+    // We're using a single loop counter, but we have to reserve all aliasing
+    // registers.
+    return State.getRATC().getRegister(LoopCounter).aliasedBits();
+  }
+
+private:
+  const unsigned LoopCounter;
+};
+
+} // namespace
+
+SnippetRepetitor::~SnippetRepetitor() {}
+
+std::unique_ptr<const SnippetRepetitor>
+SnippetRepetitor::Create(InstructionBenchmark::RepetitionModeE Mode,
+                         const LLVMState &State) {
+  switch (Mode) {
+  case InstructionBenchmark::Duplicate:
+    return std::make_unique<DuplicateSnippetRepetitor>(State);
+  case InstructionBenchmark::Loop:
+    return std::make_unique<LoopSnippetRepetitor>(State);
+  }
+}
+
+} // namespace exegesis
+} // namespace llvm

llvm/trunk/tools/llvm-exegesis/lib/Target.h

public:
   // default calling convention.
   virtual unsigned getScratchMemoryRegister(const llvm::Triple &) const {
     return 0;
   }
 
   // Fills memory operands with references to the address at [Reg] + Offset.
   virtual void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                                   unsigned Offset) const {
-
     llvm_unreachable(
         "fillMemoryOperands() requires getScratchMemoryRegister() > 0");
   }
 
+  // Returns a counter usable as a loop counter.
+  virtual unsigned getLoopCounterRegister(const llvm::Triple &) const {
+    return 0;
+  }
+
+  // Adds the code to decrement the loop counter and
+  virtual void decrementLoopCounterAndLoop(MachineBasicBlock &MBB,
+                                           const llvm::MCInstrInfo &MII) const {
+    llvm_unreachable("decrementLoopCounterAndBranch() requires "
+                     "getLoopCounterRegister() > 0");
+  }
+
   // Returns a list of unavailable registers.
   // Targets can use this to prevent some registers to be automatically selected
   // for use in snippets.
   virtual ArrayRef<unsigned> getUnavailableRegisters() const { return {}; }
 
   // Returns the maximum number of bytes a load/store instruction can access at
   // once. This is typically the size of the largest register available on the
   // processor. Note that this only used as a hint to generate independant

llvm/trunk/tools/llvm-exegesis/lib/X86/Target.cpp

 public:
   ExegesisX86Target() : ExegesisTarget(X86CpuPfmCounters) {}
 
 private:
   void addTargetSpecificPasses(llvm::PassManagerBase &PM) const override;
 
   unsigned getScratchMemoryRegister(const llvm::Triple &TT) const override;
 
+  unsigned getLoopCounterRegister(const llvm::Triple &) const override;
+
   unsigned getMaxMemoryAccessSize() const override { return 64; }
 
   void randomizeMCOperand(const Instruction &Instr, const Variable &Var,
                           llvm::MCOperand &AssignedValue,
                           const llvm::BitVector &ForbiddenRegs) const override;
 
   void fillMemoryOperands(InstructionTemplate &IT, unsigned Reg,
                           unsigned Offset) const override;
 
+  void decrementLoopCounterAndLoop(MachineBasicBlock &MBB,
+                                   const llvm::MCInstrInfo &MII) const override;
+
   std::vector<llvm::MCInst> setRegTo(const llvm::MCSubtargetInfo &STI,
                                      unsigned Reg,
                                      const llvm::APInt &Value) const override;
 
   ArrayRef<unsigned> getUnavailableRegisters() const override {
     return makeArrayRef(kUnavailableRegisters,
                         sizeof(kUnavailableRegisters) /
                             sizeof(kUnavailableRegisters[0]));
private:
 
   static const unsigned kUnavailableRegisters[4];
 };
 
 // We disable a few registers that cannot be encoded on instructions with a REX
 // prefix.
 const unsigned ExegesisX86Target::kUnavailableRegisters[4] = {X86::AH, X86::BH,
                                                               X86::CH, X86::DH};
+
+// We're using one of R8-R15 because these registers are never hardcoded in
+// instructions (e.g. MOVS writes to EDI, ESI, EDX), so they have less
+// conflicts.
+constexpr const unsigned kLoopCounterReg = X86::R8;
+
 } // namespace
 
 void ExegesisX86Target::addTargetSpecificPasses(
     llvm::PassManagerBase &PM) const {
   // Lowers FP pseudo-instructions, e.g. ABS_Fp32 -> ABS_F.
   PM.add(llvm::createX86FloatingPointStackifierPass());
 }
 
 unsigned
 ExegesisX86Target::getScratchMemoryRegister(const llvm::Triple &TT) const {
   if (!TT.isArch64Bit()) {
     // FIXME: This would require popping from the stack, so we would have to
     // add some additional setup code.
     return 0;
   }
   return TT.isOSWindows() ? llvm::X86::RCX : llvm::X86::RDI;
 }
 
+unsigned
+ExegesisX86Target::getLoopCounterRegister(const llvm::Triple &TT) const {
+  if (!TT.isArch64Bit()) {
+    return 0;
+  }
+  return kLoopCounterReg;
+}
+
 void ExegesisX86Target::randomizeMCOperand(
     const Instruction &Instr, const Variable &Var,
     llvm::MCOperand &AssignedValue,
     const llvm::BitVector &ForbiddenRegs) const {
   ExegesisTarget::randomizeMCOperand(Instr, Var, AssignedValue, ForbiddenRegs);
 
   const Operand &Op = Instr.getPrimaryOperand(Var);
   switch (Op.getExplicitOperandInfo().OperandType) {
void ExegesisX86Target::fillMemoryOperands(InstructionTemplate &IT,
   };
   SetOp(MemOpIdx + 0, MCOperand::createReg(Reg));    // BaseReg
   SetOp(MemOpIdx + 1, MCOperand::createImm(1));      // ScaleAmt
   SetOp(MemOpIdx + 2, MCOperand::createReg(0));      // IndexReg
   SetOp(MemOpIdx + 3, MCOperand::createImm(Offset)); // Disp
   SetOp(MemOpIdx + 4, MCOperand::createReg(0));      // Segment
 }
 
+void ExegesisX86Target::decrementLoopCounterAndLoop(
+    MachineBasicBlock &MBB, const llvm::MCInstrInfo &MII) const {
+  BuildMI(&MBB, DebugLoc(), MII.get(X86::ADD64ri8))
+      .addDef(kLoopCounterReg)
+      .addUse(kLoopCounterReg)
+      .addImm(-1);
+  BuildMI(&MBB, DebugLoc(), MII.get(X86::JCC_1))
+      .addMBB(&MBB)
+      .addImm(X86::COND_NE);
+}
+
 std::vector<llvm::MCInst>
 ExegesisX86Target::setRegTo(const llvm::MCSubtargetInfo &STI, unsigned Reg,
                             const llvm::APInt &Value) const {
   if (llvm::X86::GR8RegClass.contains(Reg))
     return {loadImmediate(Reg, 8, Value)};
   if (llvm::X86::GR16RegClass.contains(Reg))
     return {loadImmediate(Reg, 16, Value)};
   if (llvm::X86::GR32RegClass.contains(Reg))

llvm/trunk/tools/llvm-exegesis/llvm-exegesis.cpp

 //===----------------------------------------------------------------------===//
 
 #include "lib/Analysis.h"
 #include "lib/BenchmarkResult.h"
 #include "lib/BenchmarkRunner.h"
 #include "lib/Clustering.h"
 #include "lib/LlvmState.h"
 #include "lib/PerfHelper.h"
+#include "lib/SnippetRepetitor.h"
 #include "lib/Target.h"
 #include "lib/TargetSelect.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/MC/MCInstBuilder.h"
 #include "llvm/MC/MCObjectFileInfo.h"
 #include "llvm/MC/MCParser/MCAsmParser.h"
 #include "llvm/MC/MCParser/MCTargetAsmParser.h"
cl::values(clEnumValN(exegesis::InstructionBenchmark::Latency, "latency",
                           "Instruction Inverse Throughput"),
                clEnumValN(exegesis::InstructionBenchmark::Uops, "uops",
                           "Uop Decomposition"),
                // When not asking for a specific benchmark mode,
                // we'll analyse the results.
                clEnumValN(exegesis::InstructionBenchmark::Unknown, "analysis",
                           "Analysis")));
 
+static cl::opt<exegesis::InstructionBenchmark::RepetitionModeE> RepetitionMode(
+    "repetition-mode", cl::desc("how to repeat the instruction snippet"),
+    cl::cat(BenchmarkOptions),
+    cl::values(clEnumValN(exegesis::InstructionBenchmark::Duplicate,
+                          "duplicate", "Duplicate the snippet"),
+               clEnumValN(exegesis::InstructionBenchmark::Loop, "loop",
+                          "Loop over the snippet")));
+
 static cl::opt<unsigned>
     NumRepetitions("num-repetitions",
                    cl::desc("number of time to repeat the asm snippet"),
                    cl::cat(BenchmarkOptions), cl::init(10000));
 
 static cl::opt<bool> IgnoreInvalidSchedClass(
     "ignore-invalid-sched-class",
     cl::desc("ignore instructions that do not define a sched class"),
else
       llvm::report_fatal_error(
           llvm::Twine("unknown opcode ").concat(OpcodeName));
   }
   return Result;
 }
 
 // Generates code snippets for opcode `Opcode`.
 static llvm::Expected<std::vector<BenchmarkCode>>
-generateSnippets(const LLVMState &State, unsigned Opcode) {
+generateSnippets(const LLVMState &State, unsigned Opcode,
+                 const llvm::BitVector &ForbiddenRegs) {
   const Instruction &Instr = State.getIC().getInstr(Opcode);
   const llvm::MCInstrDesc &InstrDesc = *Instr.Description;
   // Ignore instructions that we cannot run.
   if (InstrDesc.isPseudo())
     return llvm::make_error<BenchmarkFailure>("Unsupported opcode: isPseudo");
   if (InstrDesc.isBranch() || InstrDesc.isIndirectBranch())
     return llvm::make_error<BenchmarkFailure>(
         "Unsupported opcode: isBranch/isIndirectBranch");
   if (InstrDesc.isCall() || InstrDesc.isReturn())
     return llvm::make_error<BenchmarkFailure>(
         "Unsupported opcode: isCall/isReturn");
 
   const std::unique_ptr<SnippetGenerator> Generator =
       State.getExegesisTarget().createSnippetGenerator(BenchmarkMode, State);
   if (!Generator)
     llvm::report_fatal_error("cannot create snippet generator");
-  return Generator->generateConfigurations(Instr);
+  return Generator->generateConfigurations(Instr, ForbiddenRegs);
 }
 
 namespace {
 
 // An MCStreamer that reads a BenchmarkCode definition from a file.
 // The BenchmarkCode definition is just an asm file, with additional comments to
 // specify which registers should be defined or are live on entry.
 class BenchmarkCodeStreamer : public llvm::MCStreamer,
#endif
   llvm::InitializeNativeTarget();
   llvm::InitializeNativeTargetAsmPrinter();
   llvm::InitializeNativeTargetAsmParser();
   InitializeNativeExegesisTarget();
 
   const LLVMState State(CpuName);
   const auto Opcodes = getOpcodesOrDie(State.getInstrInfo());
 
+  const auto Repetitor = SnippetRepetitor::Create(RepetitionMode, State);
+
   std::vector<BenchmarkCode> Configurations;
   if (!Opcodes.empty()) {
     for (const unsigned Opcode : Opcodes) {
       // Ignore instructions without a sched class if
       // -ignore-invalid-sched-class is passed.
       if (IgnoreInvalidSchedClass &&
           State.getInstrInfo().get(Opcode).getSchedClass() == 0) {
         llvm::errs() << State.getInstrInfo().getName(Opcode)
                      << ": ignoring instruction without sched class\n";
         continue;
       }
-      auto ConfigsForInstr = generateSnippets(State, Opcode);
+      auto ConfigsForInstr =
+          generateSnippets(State, Opcode, Repetitor->getReservedRegs());
       if (!ConfigsForInstr) {
         llvm::logAllUnhandledErrors(
             ConfigsForInstr.takeError(), llvm::errs(),
             llvm::Twine(State.getInstrInfo().getName(Opcode)).concat(": "));
         continue;
       }
       std::move(ConfigsForInstr->begin(), ConfigsForInstr->end(),
                 std::back_inserter(Configurations));
#endif
   if (NumRepetitions == 0)
     llvm::report_fatal_error("--num-repetitions must be greater than zero");
 
   // Write to standard output if file is not set.
   if (BenchmarkFile.empty())
     BenchmarkFile = "-";
 
   for (const BenchmarkCode &Conf : Configurations) {
-    InstructionBenchmark Result =
-        Runner->runConfiguration(Conf, NumRepetitions, DumpObjectToDisk);
+    InstructionBenchmark Result = Runner->runConfiguration(
+        Conf, NumRepetitions, *Repetitor, DumpObjectToDisk);
     ExitOnErr(Result.writeYaml(State, BenchmarkFile));
   }
   exegesis::pfm::pfmTerminate();
 }
 
 // Prints the results of running analysis pass `Pass` to file `OutputFilename`
 // if OutputFilename is non-empty.
 template <typename Pass>

llvm/trunk/unittests/tools/llvm-exegesis/Common/AssemblerUtils.h

MachineFunctionGeneratorBaseTest(const std::string &TT,
     if (!CanExecute) {
       llvm::outs() << "Skipping execution, host:"
                    << llvm::sys::getProcessTriple() << ", target:" << TT
                    << "\n";
     }
   }
 
   template <class... Bs>
-  inline void Check(llvm::ArrayRef<RegisterValue> RegisterInitialValues,
-                    llvm::MCInst MCInst, Bs... Bytes) {
+  inline void Check(ArrayRef<RegisterValue> RegisterInitialValues, MCInst Inst,
+                    Bs... Bytes) {
     ExecutableFunction Function =
-        (MCInst.getOpcode() == 0)
-            ? assembleToFunction(RegisterInitialValues, {})
-            : assembleToFunction(RegisterInitialValues, {MCInst});
+        (Inst.getOpcode() == 0)
+            ? assembleToFunction(RegisterInitialValues, [](FunctionFiller &) {})
+            : assembleToFunction(RegisterInitialValues,
+                                 [Inst](FunctionFiller &Filler) {
+                                   Filler.getEntry().addInstruction(Inst);
+                                 });
     ASSERT_THAT(Function.getFunctionBytes().str(),
                 testing::ElementsAre(Bytes...));
     if (CanExecute) {
       BenchmarkRunner::ScratchSpace Scratch;
       Function(Scratch.ptr());
     }
   }
 
 private:
   std::unique_ptr<llvm::LLVMTargetMachine> createTargetMachine() {
     std::string Error;
     const llvm::Target *TheTarget =
         llvm::TargetRegistry::lookupTarget(TT, Error);
     EXPECT_TRUE(TheTarget) << Error << " " << TT;
     const llvm::TargetOptions Options;
     llvm::TargetMachine *TM = TheTarget->createTargetMachine(
         TT, CpuName, "", Options, llvm::Reloc::Model::Static);
     EXPECT_TRUE(TM) << TT << " " << CpuName;
     return std::unique_ptr<llvm::LLVMTargetMachine>(
         static_cast<llvm::LLVMTargetMachine *>(TM));
   }
 
   ExecutableFunction
   assembleToFunction(llvm::ArrayRef<RegisterValue> RegisterInitialValues,
-                     llvm::ArrayRef<llvm::MCInst> Instructions) {
+                     FillFunction Fill) {
     llvm::SmallString<256> Buffer;
     llvm::raw_svector_ostream AsmStream(Buffer);
     assembleToStream(*ET, createTargetMachine(), /*LiveIns=*/{},
-                     RegisterInitialValues, Instructions, AsmStream);
+                     RegisterInitialValues, Fill, AsmStream);
     return ExecutableFunction(createTargetMachine(),
                               getObjectFromBuffer(AsmStream.str()));
   }
 
   const std::string TT;
   const std::string CpuName;
   const bool CanExecute;
   const ExegesisTarget *const ET;
 };
 
 } // namespace exegesis
 } // namespace llvm
 
 #endif

llvm/trunk/unittests/tools/llvm-exegesis/X86/CMakeLists.txt

set(LLVM_LINK_COMPONENTS
   )
 
 add_llvm_unittest(LLVMExegesisX86Tests
   AssemblerTest.cpp
   BenchmarkResultTest.cpp
   RegisterAliasingTest.cpp
   SchedClassResolutionTest.cpp
   SnippetGeneratorTest.cpp
+  SnippetRepetitorTest.cpp
   TargetTest.cpp
   )
 target_link_libraries(LLVMExegesisX86Tests PRIVATE
   LLVMExegesis
   LLVMExegesisX86)

llvm/trunk/unittests/tools/llvm-exegesis/X86/SnippetGeneratorTest.cpp

TEST_F(FakeSnippetGeneratorTest, MemoryUse_Movsb) {
   // - Var0 [Op0]
   // - Var1 [Op1]
   // - Var2 [Op2]
   // - hasMemoryOperands
   // - hasAliasingImplicitRegisters (execution is always serial)
   // - hasAliasingRegisters
   const unsigned Opcode = llvm::X86::MOVSB;
   const Instruction &Instr = State.getIC().getInstr(Opcode);
-  auto Error = Generator.generateConfigurations(Instr).takeError();
+  auto Error =
+      Generator.generateConfigurations(Instr, State.getRATC().emptyRegisters())
+          .takeError();
   EXPECT_TRUE((bool)Error);
   llvm::consumeError(std::move(Error));
 }
 
 TEST_F(FakeSnippetGeneratorTest, ComputeRegisterInitialValuesAdd16ri) {
   // ADD16ri:
   // explicit def 0       : reg RegClass=GR16
   // explicit use 1       : reg RegClass=GR16 | TIED_TO:0

llvm/trunk/unittests/tools/llvm-exegesis/X86/SnippetRepetitorTest.cpp

+//===-- SnippetRepetitorTest.cpp --------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "../Common/AssemblerUtils.h"
+#include "Latency.h"
+#include "LlvmState.h"
+#include "MCInstrDescView.h"
+#include "RegisterAliasing.h"
+#include "Uops.h"
+#include "X86InstrInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+
+namespace llvm {
+namespace exegesis {
+
+void InitializeX86ExegesisTarget();
+
+namespace {
+
+using testing::ElementsAre;
+using testing::Eq;
+using testing::Field;
+using testing::Property;
+using testing::UnorderedElementsAre;
+
+class X86SnippetRepetitorTest : public ::testing::Test {
+protected:
+  X86SnippetRepetitorTest() : State("x86_64-unknown-linux", "haswell") {}
+
+  static void SetUpTestCase() {
+    LLVMInitializeX86TargetInfo();
+    LLVMInitializeX86TargetMC();
+    LLVMInitializeX86Target();
+    LLVMInitializeX86AsmPrinter();
+    InitializeX86ExegesisTarget();
+  }
+
+  void SetUp() {
+    TM = State.createTargetMachine();
+    Context = std::make_unique<LLVMContext>();
+    Module =
+        std::make_unique<llvm::Module>("X86SnippetRepetitorTest", *Context);
+    Module->setDataLayout(TM->createDataLayout());
+    MMI = std::make_unique<MachineModuleInfo>(TM.get());
+    MF = &createVoidVoidPtrMachineFunction("TestFn", Module.get(), MMI.get());
+  }
+
+  void TestCommon(InstructionBenchmark::RepetitionModeE RepetitionMode) {
+    const auto Repetitor = SnippetRepetitor::Create(RepetitionMode, State);
+    const std::vector<MCInst> Instructions = {MCInstBuilder(X86::NOOP)};
+    FunctionFiller Sink(*MF, {X86::EAX});
+    const auto Fill = Repetitor->Repeat(Instructions, kMinInstructions);
+    Fill(Sink);
+  }
+
+  static constexpr const unsigned kMinInstructions = 3;
+
+  const LLVMState State;
+  std::unique_ptr<llvm::LLVMTargetMachine> TM;
+  std::unique_ptr<LLVMContext> Context;
+  std::unique_ptr<Module> Module;
+  std::unique_ptr<MachineModuleInfo> MMI;
+  MachineFunction *MF = nullptr;
+};
+
+static auto HasOpcode = [](unsigned Opcode) {
+  return Property(&MachineInstr::getOpcode, Eq(Opcode));
+};
+
+static auto LiveReg = [](unsigned Reg) {
+  return Field(&MachineBasicBlock::RegisterMaskPair::PhysReg, Eq(Reg));
+};
+
+TEST_F(X86SnippetRepetitorTest, Duplicate) {
+  TestCommon(InstructionBenchmark::Duplicate);
+  // Duplicating creates a single basic block that repeats the instructions.
+  ASSERT_EQ(MF->getNumBlockIDs(), 1u);
+  EXPECT_THAT(MF->getBlockNumbered(0)->instrs(),
+              ElementsAre(HasOpcode(X86::NOOP), HasOpcode(X86::NOOP),
+                          HasOpcode(X86::NOOP), HasOpcode(X86::RETQ)));
+}
+
+TEST_F(X86SnippetRepetitorTest, Loop) {
+  TestCommon(InstructionBenchmark::Loop);
+  // Duplicating creates an entry block, a loop body and a ret block.
+  ASSERT_EQ(MF->getNumBlockIDs(), 3u);
+  const auto &LoopBlock = *MF->getBlockNumbered(1);
+  EXPECT_THAT(LoopBlock.instrs(),
+              ElementsAre(HasOpcode(X86::NOOP), HasOpcode(X86::ADD64ri8),
+                          HasOpcode(X86::JCC_1)));
+  EXPECT_THAT(LoopBlock.liveins(),
+              UnorderedElementsAre(
+                  LiveReg(X86::EAX),
+                  LiveReg(State.getExegesisTarget().getLoopCounterRegister(
+                      State.getTargetMachine().getTargetTriple()))));
+  EXPECT_THAT(MF->getBlockNumbered(2)->instrs(),
+              ElementsAre(HasOpcode(X86::RETQ)));
+}
+
+} // namespace
+} // namespace exegesis
+} // namespace llvm