[llvm-exegesis] Add Target Memory Utility Functions

[aidengrossman]

Summary

This patch adds in several functions to ExegesisTarget that will assist
in setting up memory for the planned memory annotations.

[aidengrossman]

Rebase

[aidengrossman]

Format + rebase

[RKSimon]

Please remember to add context to the diffs with -UU999999 - https://www.llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

[aidengrossman]

Rebase

[courbet]

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

978

All these are extremely complex. Can't we insert a call to a separate module instead ?

Given that we don't actually care about measuring the setup code, we don't need to inline each function here.

[aidengrossman]

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

978

I agree that they're complex and the readability is not great.

Theoretically, we could do it in a separate module and call into that, but the requirements for setting up a deterministic execution environment with regard to memory require that all the code has to be in blocks of fixed size at known addresses. I don't see any feasible way to do that by just implementing these functions in C/C++ in the llvm-exegesis code base. It's theoretically possible to have an implementation that lives outside of the main llvm-exegesis code base and gets built/linked in to the JITed snippet and emitted to the same block of memory, but decided against that approach since there was still a lot of complexity with that method and it seemed to be a less prominent path.

Very open to suggestions to improve the readability/complexity though since I'm sure I haven't explored the solution space that thoroughly.

[aidengrossman]

Address reviewer feedback (from dependent review), rebase.

[aidengrossman]

Rebase

[aidengrossman]

Rebase

[courbet]

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

978

We discussed a couple solutions or this with Ondrej and Guillaume, for the record:

• Writing the code in syscalls in asm or c++ in a separate module and call to that: We agree that this would be very hard to maintain two separate modules.
• Writing this as textual asm here and using the snippet parsing facilities to transform the result to MCInsts: Still painful because there a a bunch of values that are not constant.

In the end we don;t really have a much better apporach than this oone, but we think that it could be made much more readable by refactoring, e.g. generateSyscall(SYS_munmap, ...) method, roundTo(4096, X86::RSI);

992

should this be using getpagesize() ?

[aidengrossman]

Address reviewer feedback.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

978

That would definitely improve readability! Thanks for the suggestion. I've done some refactoring to try and make things more readable based on your suggestions in addition to adding in comments where it seems like things aren't clear. Let me know what you think and if you have any other suggestions.

[aidengrossman]

Rebase

[aidengrossman]

@courbet This one should be ready for another round of review too when you get a chance. Thanks for all your time and patience so far with reviews!

[courbet]

General comment: all these blindly clobber registers (which is an issue, see my comment in D151025). Should we always save and restore registers ?

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

990

PageSizeShift ?

994–1001

Another, maybe simpler option is: AND64ri register, -page_size

[courbet]

(BTW thanks for the refactoring, it makes everything much simpler to understand)

[courbet]

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

1036

[nit] Use RDI for consistency with the rest of the file

1055

Please create a constant for this.

[aidengrossman]

Right, all these functions do clobber registers (at least currently). However, the only one that actually matters is configurePerfCounter as it is the only one that runs after the registers get initialized to their specified values. I have a TODO to fix this.

[courbet]

In D151023#4427407, @aidengrossman wrote:

Right, all these functions do clobber registers (at least currently).

Let's document this then :)

However, the only one that actually matters is configurePerfCounter as it is the only one that runs after the registers get initialized to their specified values. I have a TODO to fix this.

SG, let's do it now given that as such the code won't work as is.

[aidengrossman]

Address reviewer feedback

[aidengrossman]

Just added in register saving to configurePerfCounter() and noted that the functions may clobber registers in Target.h. Probably should've just done it to begin with. New changes should be ready for review.

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

994–1001

I find the shift left/right pattern to be a little bit more intuitive since using an AND instruction involves understanding a bitwise "trick" when reading, but it definitely is shorter. I don't have any particularly strong feelings on this though, so if you want me to change it, just let me know.

[courbet]

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

994–1001

I find the shift left/right pattern to be a little bit more intuitive since using an AND instruction involves understanding a bitwise "trick" when reading, but it definitely is shorter. I don't have any particularly strong feelings on this though, so if you want me to change it, just let me know.

No strong opinion, you can keep it as it if you want.

llvm/tools/llvm-exegesis/lib/Target.h

   // Precondition: Value must fit into Reg.
   virtual std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                        const APInt &Value) const = 0;
+  
+  // Generates the code for the lower munmap call
+  virtual std::vector<MCInst> generateLowerMunmap() const {
+    report_fatal_error("generateLowerMunmap is not implemented on the current architecture");
+  }
+
+  // Generates the upper munmap call
+  virtual std::vector<MCInst> generateUpperMunmap() const {
+    report_fatal_error("generateUpperMunmap is not implemented on the current architecture");
+  }
+
+  // Generates the code for an exit syscall
+  virtual std::vector<MCInst> generateExitSyscall(unsigned ExitCode) const {
+    report_fatal_error("generateExitSyscall is not implemented on the current architecture");
+  }
+
+  // Generates the code to mmap a region of code
+  virtual std::vector<MCInst> generateMmap(intptr_t Address, size_t Length,
+                                           intptr_t FileDescriptorAddress) const {
+    report_fatal_error(
+        "generateMmap is not implemented on the current architecture");
+  }
+
+  // Generates the mmap code for the aux memory
+  virtual std::vector<MCInst> generateMmapAuxMem() const {
+    report_fatal_error(
+        "generateMmapAuxMem is not implemented on the current architecture\n");
+  }
+
+  // Moves argument registers into other registers that won't get clobbered
+  // while making syscalls
+  virtual std::vector<MCInst> moveArgumentRegisters() const {
+    report_fatal_error("moveArgumentRegisters is not implemented on the "
+                       "current architecture\n");
+  }
+
+  // Sets the stack register to the auxiliary memory so that operations
+  // requiring the stack can be formed (e.g., setting large registers).
+  virtual std::vector<MCInst> setStackRegisterToAuxMem() const {
+    report_fatal_error("setStackRegisterToAuxMem is not implemented on the "
+                       "current architectures");
+  }
+
+  virtual intptr_t getAuxiliaryMemoryStartAddress() const {
+    report_fatal_error("getAuxiliaryMemoryStartAddress is not implemented on "
+                       "the current architecture");
+  }
+
+  // Generates the necessary ioctl system calls to configure the perf counters
+  virtual std::vector<MCInst> configurePerfCounter(long Request) const {
+    report_fatal_error(
+        "configurePerfCounter is not implemented on the current architecture");
+  }
+
+  // Gets the ABI dependent registers that are used to pass arguments in a
+  // function call
+  virtual std::vector<unsigned> getArgumentRegisters() const {
+    report_fatal_error(
+        "getArgumentRegisters is not implemented on the current architecture");
+  };
 
   // Returns the register pointing to scratch memory, or 0 if this target
   // does not support memory operands. The benchmark function uses the

llvm/tools/llvm-exegesis/lib/X86/Target.cpp

 #include "../ParallelSnippetGenerator.h"
 #include "../SerialSnippetGenerator.h"
 #include "../SnippetGenerator.h"
+#include "../SubprocessMemory.h"
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "X86.h"

 #include <float.h> // For _clearfp in ~X86SavedState().
 #endif
 
+#ifdef __linux__
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+#endif
+
 namespace llvm {
 namespace exegesis {
 

     return ExegesisTarget::createCounter(CounterName, State, ProcessPID);
   }
 
+  enum ArgumentRegisters { CodeSize = X86::R12, AuxiliaryMemoryFD = X86::R13 };
+
 private:
   void addTargetSpecificPasses(PassManagerBase &PM) const override;
 

   std::vector<MCInst> setRegTo(const MCSubtargetInfo &STI, unsigned Reg,
                                const APInt &Value) const override;
 
+#ifdef __linux__
+  std::vector<MCInst> generateLowerMunmap() const override;
+
+  std::vector<MCInst> generateUpperMunmap() const override;
+
+  std::vector<MCInst> generateExitSyscall(unsigned ExitCode) const override;
+
+  std::vector<MCInst>
+  generateMmap(intptr_t Address, size_t Length,
+               intptr_t FileDescriptorAddress) const override;
+
+  std::vector<MCInst> generateMmapAuxMem() const override;
+
+  std::vector<MCInst> moveArgumentRegisters() const override;
+
+  std::vector<MCInst> setStackRegisterToAuxMem() const override;
+
+  intptr_t getAuxiliaryMemoryStartAddress() const override;
+
+  std::vector<MCInst> configurePerfCounter(long Request) const override;
+
+  std::vector<unsigned> getArgumentRegisters() const override;
+#endif // __linux__
+
   ArrayRef<unsigned> getUnavailableRegisters() const override {
     if (DisableUpperSSERegisters)
       return ArrayRef(kUnavailableRegistersSSE,

   return {}; // Not yet implemented.
 }
 
+#ifdef __linux__
+std::vector<MCInst> ExegesisX86Target::generateLowerMunmap() const {
+  std::vector<MCInst> LowerMunmapCode;
+  LowerMunmapCode.push_back(loadImmediate(X86::RDI, 64, APInt(64, 0)));
+  // Get the current function pointer so we know where to unmap up to.
+  LowerMunmapCode.push_back(MCInstBuilder(X86::LEA64r)
+                                .addReg(X86::RSI)
+                                .addReg(X86::RIP)
+                                .addImm(1)
+                                .addReg(0)
+                                .addImm(0)
+                                .addReg(0));
+  // The below two instructions round to the nearest 4096 byte page.
+  LowerMunmapCode.push_back(
+      MCInstBuilder(X86::SHR64ri).addReg(X86::RSI).addReg(X86::RSI).addImm(12));
+  LowerMunmapCode.push_back(
+      MCInstBuilder(X86::SHL64ri).addReg(X86::RSI).addReg(X86::RSI).addImm(12));
+  // Subtract a page from the end of the unmap so we don't unmap the currently
+  // executing section.
+  LowerMunmapCode.push_back(MCInstBuilder(X86::SUB64ri32)
+                                .addReg(X86::RSI)
+                                .addReg(X86::RSI)
+                                .addImm(getpagesize()));
+  LowerMunmapCode.push_back(loadImmediate(X86::RAX, 64, APInt(64, SYS_munmap)));
+  LowerMunmapCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return LowerMunmapCode;
+}
+
+std::vector<MCInst> ExegesisX86Target::generateUpperMunmap() const {
+  std::vector<MCInst> UpperMunmapCode;
+  UpperMunmapCode.push_back(MCInstBuilder(X86::LEA64r)
+                                .addReg(X86::R8)
+                                .addReg(X86::RIP)
+                                .addImm(1)
+                                .addReg(0)
+                                .addImm(0)
+                                .addReg(0));
+  // Load in RDI from from the argument registers.
+  UpperMunmapCode.push_back(MCInstBuilder(X86::MOV64rr)
+                                .addReg(X86::RDI)
+                                .addReg(ArgumentRegisters::AuxiliaryMemoryFD));
+  UpperMunmapCode.push_back(MCInstBuilder(X86::ADD64rr)
+                                .addReg(X86::RDI)
+                                .addReg(X86::RDI)
+                                .addReg(X86::R8));
+  UpperMunmapCode.push_back(
+      MCInstBuilder(X86::SHR64ri).addReg(X86::RDI).addReg(X86::RDI).addImm(12));
+  UpperMunmapCode.push_back(
+      MCInstBuilder(X86::SHL64ri).addReg(X86::RDI).addReg(X86::RDI).addImm(12));
+  UpperMunmapCode.push_back(MCInstBuilder(X86::ADD64ri32)
+                                .addReg(X86::RDI)
+                                .addReg(X86::RDI)
+                                .addImm(getpagesize()));
+  // Unmap to just one page under the ceiling of the address space.
+  UpperMunmapCode.push_back(loadImmediate(
+      X86::RSI, 64, APInt(64, 0x0000800000000000 - getpagesize())));
+  UpperMunmapCode.push_back(MCInstBuilder(X86::SUB64rr)
+                                .addReg(X86::RSI)
+                                .addReg(X86::RSI)
+                                .addReg(X86::RDI));
+  UpperMunmapCode.push_back(loadImmediate(X86::RAX, 64, APInt(64, SYS_munmap)));
+  UpperMunmapCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return UpperMunmapCode;
+}
+
+std::vector<MCInst>
+ExegesisX86Target::generateExitSyscall(unsigned ExitCode) const {
+  std::vector<MCInst> ExitCallCode;
+  ExitCallCode.push_back(loadImmediate(X86::RDI, 64, APInt(64, ExitCode)));
+  ExitCallCode.push_back(loadImmediate(X86::RAX, 64, APInt(64, SYS_exit)));
+  ExitCallCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return ExitCallCode;
+}
+
+std::vector<MCInst>
+ExegesisX86Target::generateMmap(intptr_t Address, size_t Length,
+                                intptr_t FileDescriptorAddress) const {
+  std::vector<MCInst> MmapCode;
+  MmapCode.push_back(loadImmediate(X86::RAX, 64, APInt(64, SYS_mmap)));
+  MmapCode.push_back(loadImmediate(X86::RDI, 64, APInt(64, Address)));
+  MmapCode.push_back(loadImmediate(X86::RSI, 64, APInt(64, Length)));
+  MmapCode.push_back(
+      loadImmediate(X86::RDX, 64, APInt(64, PROT_READ | PROT_WRITE)));
+  MmapCode.push_back(
+      loadImmediate(X86::R10, 64, APInt(64, MAP_SHARED | MAP_FIXED_NOREPLACE)));
+  // Copy file descriptor location from aux memory into R8
+  MmapCode.push_back(
+      loadImmediate(X86::R8, 64, APInt(64, FileDescriptorAddress)));
+  // Dereference file descriptor into FD argument register
+  MmapCode.push_back(MCInstBuilder(X86::MOV32rm)
+                         .addReg(X86::R8D)
+                         .addReg(X86::R8)
+                         .addImm(1)
+                         .addReg(0)
+                         .addImm(0)
+                         .addReg(0));
+  MmapCode.push_back(loadImmediate(X86::R9, 64, APInt(64, 0)));
+  MmapCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return MmapCode;
+}
+
+std::vector<MCInst> ExegesisX86Target::generateMmapAuxMem() const {
+  std::vector<MCInst> MmapAuxMemCode;
+  MmapAuxMemCode.push_back(loadImmediate(X86::RAX, 64, APInt(64, SYS_mmap)));
+  MmapAuxMemCode.push_back(
+      loadImmediate(X86::RDI, 64, APInt(64, getAuxiliaryMemoryStartAddress())));
+  MmapAuxMemCode.push_back(
+      loadImmediate(X86::RSI, 64, APInt(64, AuxiliaryMemorySize)));
+  MmapAuxMemCode.push_back(
+      loadImmediate(X86::RDX, 64, APInt(64, PROT_READ | PROT_WRITE)));
+  MmapAuxMemCode.push_back(
+      loadImmediate(X86::R10, 64, APInt(64, MAP_SHARED | MAP_FIXED_NOREPLACE)));
+  MmapAuxMemCode.push_back(MCInstBuilder(X86::MOV64rr)
+                               .addReg(X86::R8)
+                               .addReg(ArgumentRegisters::CodeSize));
+  MmapAuxMemCode.push_back(loadImmediate(X86::R9, 64, APInt(64, 0)));
+  MmapAuxMemCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return MmapAuxMemCode;
+}
+
+std::vector<MCInst> ExegesisX86Target::moveArgumentRegisters() const {
+  std::vector<MCInst> MoveArgumentRegistersCode;
+  MoveArgumentRegistersCode.push_back(MCInstBuilder(X86::MOV64rr)
+                                          .addReg(ArgumentRegisters::CodeSize)
+                                          .addReg(X86::RSI));
+  MoveArgumentRegistersCode.push_back(
+      MCInstBuilder(X86::MOV64rr)
+          .addReg(ArgumentRegisters::AuxiliaryMemoryFD)
+          .addReg(X86::RDI));
+  return MoveArgumentRegistersCode;
+}
+
+std::vector<MCInst> ExegesisX86Target::setStackRegisterToAuxMem() const {
+  // Moves %rsp to the end of the auxiliary memory
+  return {MCInstBuilder(X86::MOV64ri)
+              .addReg(X86::RSP)
+              .addImm(getAuxiliaryMemoryStartAddress() + AuxiliaryMemorySize)};
+}
+
+intptr_t ExegesisX86Target::getAuxiliaryMemoryStartAddress() const {
+  // Return the second to last page in the virtual address space to try and
+  // prevent interference with memory annotations in the snippet
+  return 0x00007fffffffe000;
+}
+
+std::vector<MCInst>
+ExegesisX86Target::configurePerfCounter(long Request) const {
+  // TOOD(boomanaiden154): This currently will clobber RAX, RDI, and RSI.
+  // Fix this, probably by pushing data to the stack
+  std::vector<MCInst> ConfigurePerfCounterCode;
+  ConfigurePerfCounterCode.push_back(
+      loadImmediate(X86::RAX, 64, APInt(64, SYS_ioctl)));
+  ConfigurePerfCounterCode.push_back(
+      loadImmediate(X86::RDI, 64, APInt(64, getAuxiliaryMemoryStartAddress())));
+  ConfigurePerfCounterCode.push_back(MCInstBuilder(X86::MOV32rm)
+                                         .addReg(X86::EDI)
+                                         .addReg(X86::RDI)
+                                         .addImm(1)
+                                         .addReg(0)
+                                         .addImm(0)
+                                         .addReg(0));
+  ConfigurePerfCounterCode.push_back(
+      loadImmediate(X86::RSI, 64, APInt(64, Request)));
+  ConfigurePerfCounterCode.push_back(MCInstBuilder(X86::SYSCALL));
+  return ConfigurePerfCounterCode;
+}
+
+std::vector<unsigned> ExegesisX86Target::getArgumentRegisters() const {
+  return {X86::RDI, X86::RSI};
+}
+#endif // __linux__
+
 // Instruction can have some variable operands, and we may want to see how
 // different operands affect performance. So for each operand position,
 // precompute all the possible choices we might care about,

llvm/unittests/tools/llvm-exegesis/X86/TargetTest.cpp

 #include <memory>
 
 #include "MCTargetDesc/X86MCTargetDesc.h"
+#include "SubprocessMemory.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/TargetSelect.h"
 #include "gmock/gmock.h"

 
 #include "llvm/MC/MCInstPrinter.h"
 
+#ifdef __linux__
+#include <sys/syscall.h>
+#include <sys/mman.h>
+#endif // __linux__
+
 namespace llvm {
 
 bool operator==(const MCOperand &a, const MCOperand &b) {

   return AllOf(OpcodeIs(Opcode), ElementsAre(IsReg(Reg), IsImm(Value)));
 }
 
+Matcher<MCInst> IsMovRegToReg(unsigned Opcode, int64_t Reg1, int64_t Reg2) {
+  return AllOf(OpcodeIs(Opcode), ElementsAre(IsReg(Reg1), IsReg(Reg2)));
+}
+
 Matcher<MCInst> IsMovValueToStack(unsigned Opcode, int64_t Value,
                                   size_t Offset) {
   return AllOf(OpcodeIs(Opcode),

       State.getExegesisTarget().allowAsBackToBack(getInstr(X86::LEA64r)));
 }
 
+#ifdef __linux__
+TEST_F(X86Core2TargetTest, GenerateLowerMunmapTest) {
+  EXPECT_THAT(State.getExegesisTarget().generateLowerMunmap(),
+              ElementsAre(IsMovImmediate(X86::MOV64ri, X86::RDI, 0),
+                          OpcodeIs(X86::LEA64r), OpcodeIs(X86::SHR64ri),
+                          OpcodeIs(X86::SHL64ri), OpcodeIs(X86::SUB64ri32),
+                          IsMovImmediate(X86::MOV64ri, X86::RAX, SYS_munmap),
+                          OpcodeIs(X86::SYSCALL)));
+}
+
+TEST_F(X86Core2TargetTest, GenerateUpperMunmapTest) {
+  EXPECT_THAT(
+      State.getExegesisTarget().generateUpperMunmap(),
+      ElementsAreArray({OpcodeIs(X86::LEA64r), OpcodeIs(X86::MOV64rr),
+                             OpcodeIs(X86::ADD64rr), OpcodeIs(X86::SHR64ri),
+                             OpcodeIs(X86::SHL64ri), OpcodeIs(X86::ADD64ri32),
+                             IsMovImmediate(X86::MOV64ri, X86::RSI,
+                                            0x0000800000000000 - getpagesize()),
+                             OpcodeIs(X86::SUB64rr),
+                             IsMovImmediate(X86::MOV64ri, X86::RAX, SYS_munmap),
+                             OpcodeIs(X86::SYSCALL)}));
+}
+
+TEST_F(X86Core2TargetTest, GenerateExitSyscallTest) {
+  EXPECT_THAT(State.getExegesisTarget().generateExitSyscall(127),
+              ElementsAre(IsMovImmediate(X86::MOV64ri, X86::RDI, 127),
+                          IsMovImmediate(X86::MOV64ri, X86::RAX, SYS_exit),
+                          OpcodeIs(X86::SYSCALL)));
+}
+
+TEST_F(X86Core2TargetTest, GenerateMmapTest) {
+  EXPECT_THAT(
+      State.getExegesisTarget().generateMmap(0x1000, 4096, 0x2000),
+      ElementsAre(
+          IsMovImmediate(X86::MOV64ri, X86::RAX, SYS_mmap),
+          IsMovImmediate(X86::MOV64ri, X86::RDI, 0x1000),
+          IsMovImmediate(X86::MOV64ri, X86::RSI, 4096),
+          IsMovImmediate(X86::MOV64ri, X86::RDX, PROT_READ | PROT_WRITE),
+          IsMovImmediate(X86::MOV64ri, X86::R10,
+                         MAP_SHARED | MAP_FIXED_NOREPLACE),
+          IsMovImmediate(X86::MOV64ri, X86::R8, 0x2000), OpcodeIs(X86::MOV32rm),
+          IsMovImmediate(X86::MOV64ri, X86::R9, 0), OpcodeIs(X86::SYSCALL)));
+}
+
+TEST_F(X86Core2TargetTest, GenerateMmapAuxMemTest) {
+  EXPECT_THAT(
+      State.getExegesisTarget().generateMmapAuxMem(),
+      ElementsAre(
+          IsMovImmediate(X86::MOV64ri, X86::RAX, SYS_mmap),
+          IsMovImmediate(
+              X86::MOV64ri, X86::RDI,
+              State.getExegesisTarget().getAuxiliaryMemoryStartAddress()),
+          IsMovImmediate(X86::MOV64ri, X86::RSI, AuxiliaryMemorySize),
+          IsMovImmediate(X86::MOV64ri, X86::RDX, PROT_READ | PROT_WRITE),
+          IsMovImmediate(X86::MOV64ri, X86::R10,
+                         MAP_SHARED | MAP_FIXED_NOREPLACE),
+          OpcodeIs(X86::MOV64rr), IsMovImmediate(X86::MOV64ri, X86::R9, 0),
+          OpcodeIs(X86::SYSCALL)));
+}
+
+TEST_F(X86Core2TargetTest, MoveArgumentRegistersTest) {
+  EXPECT_THAT(State.getExegesisTarget().moveArgumentRegisters(),
+              ElementsAre(IsMovRegToReg(X86::MOV64rr, X86::R12, X86::RSI),
+                          IsMovRegToReg(X86::MOV64rr, X86::R13, X86::RDI)));
+}
+#endif // __linux__
+
 } // namespace
 } // namespace exegesis
 } // namespace llvm