[SectionMemoryManager] Abstract out mmap, munmap, mprotect even more ; NFC

[sanjoy]

Summary

This will let ORC JIT clients plug in custom logic for the mmap, munmap and
mprotect paths.

[lhames]

Out of interest, what’s the use case for this? I’d have expected sys::Memory to be good enough for block-based allocation (though there’s the known issue with PaX on BSD that I need to fix), and using a non block-based underlying allocator seems like it’d risk unnecessary fragmentation.

[sanjoy]

In D39300#910042, @lhames wrote:

Out of interest, what’s the use case for this? I’d have expected sys::Memory to be good enough for block-based allocation (though there’s the known issue with PaX on BSD that I need to fix), and using a non block-based underlying allocator seems like it’d risk unnecessary fragmentation.

This is just so that the mmap can come from a file or a memfd, which will make the mapping show up with a name and not as "anon" under perf etc.

[sanjoy]

ping!

[asbirlea]

LGTM.

include/llvm/ExecutionEngine/SectionMemoryManager.h

73

Remove "is".

[jlebar]

include/llvm/ExecutionEngine/SectionMemoryManager.h

43

reasons

56

, in which case

[lhames]

In D39300#910081, @sanjoy wrote:

In D39300#910042, @lhames wrote:

Out of interest, what’s the use case for this? I’d have expected sys::Memory to be good enough for block-based allocation (though there’s the known issue with PaX on BSD that I need to fix), and using a non block-based underlying allocator seems like it’d risk unnecessary fragmentation.

This is just so that the mmap can come from a file or a memfd, which will make the mapping show up with a name and not as "anon" under perf etc.

Ahh. Makes sense.

Apologies for the delayed reply: I've been on the road. Thanks asbirlea and jlebar for stepping in.

include/llvm/ExecutionEngine/SectionMemoryManager.h

 ///
 /// Any client using this memory manager MUST ensure that section-specific
 /// page permissions have been applied before attempting to execute functions
 /// in the JITed object.  Permissions can be applied either by calling
 /// MCJIT::finalizeObject or by calling SectionMemoryManager::finalizeMemory
 /// directly.  Clients of MCJIT should call MCJIT::finalizeObject.
 class SectionMemoryManager : public RTDyldMemoryManager {
 public:
-  SectionMemoryManager() = default;
-  SectionMemoryManager(const SectionMemoryManager&) = delete;
-  void operator=(const SectionMemoryManager&) = delete;
+  /// This enum describes the various reason to allocate pages from
+  /// allocateMappedMemory.
+  enum class AllocationPurpose {
+    Code,
+    ROData,
+    RWData,
+  };
+
+  /// Implementations of this interface are used by SectionMemoryManager to
+  /// request pages from the operating system.
+  class MemoryMapper {
+  public:
+    /// This method attempts to allocate \p NumBytes bytes of virtual memory for
+    /// \p Purpose.  \p NearBlock may point to an existing allocation in which
+    /// case an attempt is made to allocate more memory near the existing block.
+    /// The actual allocated address is not guaranteed to be near the requested
+    /// address.  \p Flags is used to set the initial protection flags for the
+    /// block of the memory.  \p EC [out] returns an object describing any error
+    /// that occurs.
+    ///
+    /// This method may allocate more than the number of bytes requested.  The
+    /// actual number of bytes allocated is indicated in the returned
+    /// MemoryBlock.
+    ///
+    /// The start of the allocated block must be aligned with the
+    /// system allocation granularity (64K on Windows, page size on Linux).
+    /// If the address following \p NearBlock is not so aligned, it will be
+    /// rounded up to the next allocation granularity boundary.
+    ///
+    /// \r a non-null MemoryBlock if the function was successful,
+    /// otherwise a null MemoryBlock is with \p EC describing the error.
+    virtual sys::MemoryBlock
+    allocateMappedMemory(AllocationPurpose Purpose, size_t NumBytes,
+                         const sys::MemoryBlock *const NearBlock,
+                         unsigned Flags, std::error_code &EC) = 0;
+
+    /// This method sets the protection flags for a block of memory to the
+    /// state specified by \p Flags.  The behavior is not specified if the
+    /// memory was not allocated using the allocateMappedMemory method.
+    /// \p Block describes the memory block to be protected.
+    /// \p Flags specifies the new protection state to be assigned to the block.
+    ///
+    /// If \p Flags is MF_WRITE, the actual behavior varies
+    /// with the operating system (i.e. MF_READ | MF_WRITE on Windows) and the
+    /// target architecture (i.e. MF_WRITE -> MF_READ | MF_WRITE on i386).
+    ///
+    /// \r error_success if the function was successful, or an error_code
+    /// describing the failure if an error occurred.
+    virtual std::error_code protectMappedMemory(const sys::MemoryBlock &Block,
+                                                unsigned Flags) = 0;
+
+    /// This method releases a block of memory that was allocated with the
+    /// allocateMappedMemory method. It should not be used to release any
+    /// memory block allocated any other way.
+    /// \p Block describes the memory to be released.
+    ///
+    /// \r error_success if the function was successful, or an error_code
+    /// describing the failure if an error occurred.
+    virtual std::error_code releaseMappedMemory(sys::MemoryBlock &M) = 0;
+
+    virtual ~MemoryMapper();
+  };
+
+  /// Creates a SectionMemoryManager instance with \p MM as the associated
+  /// memory mapper.  If \p MM is nullptr then a default memory mapper is used
+  /// that directly calls into the operating system.
+  SectionMemoryManager(MemoryMapper *MM = nullptr);
+  SectionMemoryManager(const SectionMemoryManager &) = delete;
+  void operator=(const SectionMemoryManager &) = delete;
   ~SectionMemoryManager() override;
 
   /// \brief Allocates a memory block of (at least) the given size suitable for
   /// executable code.
   ///
   /// The value of \p Alignment must be a power of two.  If \p Alignment is zero
   /// a default alignment of 16 will be used.
   uint8_t *allocateCodeSection(uintptr_t Size, unsigned Alignment,
struct MemoryGroup {
     SmallVector<FreeMemBlock, 16> FreeMem;
 
     // All memory blocks that have been requested from the system
     SmallVector<sys::MemoryBlock, 16> AllocatedMem;
 
     sys::MemoryBlock Near;
   };
 
-  uint8_t *allocateSection(MemoryGroup &MemGroup, uintptr_t Size,
+  uint8_t *allocateSection(AllocationPurpose Purpose, uintptr_t Size,
                            unsigned Alignment);
 
   std::error_code applyMemoryGroupPermissions(MemoryGroup &MemGroup,
                                               unsigned Permissions);
 
   MemoryGroup CodeMem;
   MemoryGroup RWDataMem;
   MemoryGroup RODataMem;
+  MemoryMapper &MMapper;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_EXECUTION_ENGINE_SECTION_MEMORY_MANAGER_H

lib/ExecutionEngine/SectionMemoryManager.cpp

 namespace llvm {
 
 uint8_t *SectionMemoryManager::allocateDataSection(uintptr_t Size,
                                                    unsigned Alignment,
                                                    unsigned SectionID,
                                                    StringRef SectionName,
                                                    bool IsReadOnly) {
   if (IsReadOnly)
-    return allocateSection(RODataMem, Size, Alignment);
-  return allocateSection(RWDataMem, Size, Alignment);
+    return allocateSection(SectionMemoryManager::AllocationPurpose::ROData,
+                           Size, Alignment);
+  return allocateSection(SectionMemoryManager::AllocationPurpose::RWData, Size,
+                         Alignment);
 }
 
 uint8_t *SectionMemoryManager::allocateCodeSection(uintptr_t Size,
                                                    unsigned Alignment,
                                                    unsigned SectionID,
                                                    StringRef SectionName) {
-  return allocateSection(CodeMem, Size, Alignment);
+  return allocateSection(SectionMemoryManager::AllocationPurpose::Code, Size,
+                         Alignment);
 }
 
-uint8_t *SectionMemoryManager::allocateSection(MemoryGroup &MemGroup,
-                                               uintptr_t Size,
-                                               unsigned Alignment) {
+uint8_t *SectionMemoryManager::allocateSection(
+    SectionMemoryManager::AllocationPurpose Purpose, uintptr_t Size,
   if (!Alignment)
     Alignment = 16;
 
   assert(!(Alignment & (Alignment - 1)) && "Alignment must be a power of two.");
 
-  uintptr_t RequiredSize = Alignment * ((Size + Alignment - 1)/Alignment + 1);
+  uintptr_t RequiredSize = Alignment * ((Size + Alignment - 1) / Alignment + 1);
   uintptr_t Addr = 0;
 
+  MemoryGroup &MemGroup = [&]() -> MemoryGroup & {
+    switch (Purpose) {
+    case AllocationPurpose::Code:
+      return CodeMem;
+    case AllocationPurpose::ROData:
+      return RODataMem;
+    case AllocationPurpose::RWData:
+      return RWDataMem;
+    }
+  }();
+
   // Look in the list of free memory regions and use a block there if one
   // is available.
   for (FreeMemBlock &FreeMB : MemGroup.FreeMem) {
     if (FreeMB.Free.size() >= RequiredSize) {
       Addr = (uintptr_t)FreeMB.Free.base();
       uintptr_t EndOfBlock = Addr + FreeMB.Free.size();
       // Align the address.
       Addr = (Addr + Alignment - 1) & ~(uintptr_t)(Alignment - 1);
 
       if (FreeMB.PendingPrefixIndex == (unsigned)-1) {
         // The part of the block we're giving out to the user is now pending
         MemGroup.PendingMem.push_back(sys::MemoryBlock((void *)Addr, Size));
 
         // Remember this pending block, such that future allocations can just
         // modify it rather than creating a new one
         FreeMB.PendingPrefixIndex = MemGroup.PendingMem.size() - 1;
       } else {
-        sys::MemoryBlock &PendingMB = MemGroup.PendingMem[FreeMB.PendingPrefixIndex];
-        PendingMB = sys::MemoryBlock(PendingMB.base(), Addr + Size - (uintptr_t)PendingMB.base());
+        sys::MemoryBlock &PendingMB =
+            MemGroup.PendingMem[FreeMB.PendingPrefixIndex];
+        PendingMB = sys::MemoryBlock(PendingMB.base(),
+                                     Addr + Size - (uintptr_t)PendingMB.base());
       }
 
       // Remember how much free space is now left in this block
-      FreeMB.Free = sys::MemoryBlock((void *)(Addr + Size), EndOfBlock - Addr - Size);
-      return (uint8_t*)Addr;
+      FreeMB.Free =
+          sys::MemoryBlock((void *)(Addr + Size), EndOfBlock - Addr - Size);
+      return (uint8_t *)Addr;
     }
   }
 
   // No pre-allocated free block was large enough. Allocate a new memory region.
   // Note that all sections get allocated as read-write.  The permissions will
   // be updated later based on memory group.
   //
   // FIXME: It would be useful to define a default allocation size (or add
   // it as a constructor parameter) to minimize the number of allocations.
   //
   // FIXME: Initialize the Near member for each memory group to avoid
   // interleaving.
   std::error_code ec;
-  sys::MemoryBlock MB = sys::Memory::allocateMappedMemory(RequiredSize,
-                                                          &MemGroup.Near,
-                                                          sys::Memory::MF_READ |
-                                                            sys::Memory::MF_WRITE,
-                                                          ec);
+  sys::MemoryBlock MB = MMapper.allocateMappedMemory(
+      Purpose, RequiredSize, &MemGroup.Near,
+      sys::Memory::MF_READ | sys::Memory::MF_WRITE, ec);
   if (ec) {
     // FIXME: Add error propagation to the interface.
     return nullptr;
   }
 
   // Save this address as the basis for our next request
   MemGroup.Near = MB;
 
   // Remember that we allocated this memory
   MemGroup.AllocatedMem.push_back(MB);
   Addr = (uintptr_t)MB.base();
   uintptr_t EndOfBlock = Addr + MB.size();
 
   // Align the address.
   Addr = (Addr + Alignment - 1) & ~(uintptr_t)(Alignment - 1);
 
   // The part of the block we're giving out to the user is now pending
   MemGroup.PendingMem.push_back(sys::MemoryBlock((void *)Addr, Size));
 
   // The allocateMappedMemory may allocate much more memory than we need. In
   // this case, we store the unused memory as a free memory block.
-  unsigned FreeSize = EndOfBlock-Addr-Size;
+  unsigned FreeSize = EndOfBlock - Addr - Size;
   if (FreeSize > 16) {
     FreeMemBlock FreeMB;
-    FreeMB.Free = sys::MemoryBlock((void*)(Addr + Size), FreeSize);
+    FreeMB.Free = sys::MemoryBlock((void *)(Addr + Size), FreeSize);
     FreeMB.PendingPrefixIndex = (unsigned)-1;
     MemGroup.FreeMem.push_back(FreeMB);
   }
 
   // Return aligned address
-  return (uint8_t*)Addr;
+  return (uint8_t *)Addr;
 }
 
-bool SectionMemoryManager::finalizeMemory(std::string *ErrMsg)
-{
+bool SectionMemoryManager::finalizeMemory(std::string *ErrMsg) {
   // FIXME: Should in-progress permissions be reverted if an error occurs?
   std::error_code ec;
 
   // Make code memory executable.
   ec = applyMemoryGroupPermissions(CodeMem,
                                    sys::Memory::MF_READ | sys::Memory::MF_EXEC);
   if (ec) {
     if (ErrMsg) {
static sys::MemoryBlock trimBlockToPageSize(sys::MemoryBlock M) {
 
   size_t StartOverlap =
       (PageSize - ((uintptr_t)M.base() % PageSize)) % PageSize;
 
   size_t TrimmedSize = M.size();
   TrimmedSize -= StartOverlap;
   TrimmedSize -= TrimmedSize % PageSize;
 
-  sys::MemoryBlock Trimmed((void *)((uintptr_t)M.base() + StartOverlap), TrimmedSize);
+  sys::MemoryBlock Trimmed((void *)((uintptr_t)M.base() + StartOverlap),
+                           TrimmedSize);
 
   assert(((uintptr_t)Trimmed.base() % PageSize) == 0);
   assert((Trimmed.size() % PageSize) == 0);
   assert(M.base() <= Trimmed.base() && Trimmed.size() <= M.size());
 
   return Trimmed;
 }
 
-
 std::error_code
 SectionMemoryManager::applyMemoryGroupPermissions(MemoryGroup &MemGroup,
                                                   unsigned Permissions) {
   for (sys::MemoryBlock &MB : MemGroup.PendingMem)
-    if (std::error_code EC = sys::Memory::protectMappedMemory(MB, Permissions))
+    if (std::error_code EC = MMapper.protectMappedMemory(MB, Permissions))
       return EC;
 
   MemGroup.PendingMem.clear();
 
   // Now go through free blocks and trim any of them that don't span the entire
   // page because one of the pending blocks may have overlapped it.
   for (FreeMemBlock &FreeMB : MemGroup.FreeMem) {
     FreeMB.Free = trimBlockToPageSize(FreeMB.Free);

 void SectionMemoryManager::invalidateInstructionCache() {
   for (sys::MemoryBlock &Block : CodeMem.PendingMem)
     sys::Memory::InvalidateInstructionCache(Block.base(), Block.size());
 }
 
 SectionMemoryManager::~SectionMemoryManager() {
   for (MemoryGroup *Group : {&CodeMem, &RWDataMem, &RODataMem}) {
     for (sys::MemoryBlock &Block : Group->AllocatedMem)
-      sys::Memory::releaseMappedMemory(Block);
+      MMapper.releaseMappedMemory(Block);
+  }
+}
+
+SectionMemoryManager::MemoryMapper::~MemoryMapper() {}
+
+namespace {
+// Trivial implementation of SectionMemoryManager::MemoryMapper that just calls
+// into sys::Memory.
+class DefaultMMapper final : public SectionMemoryManager::MemoryMapper {
+public:
+  sys::MemoryBlock
+  allocateMappedMemory(SectionMemoryManager::AllocationPurpose Purpose,
+                       size_t NumBytes, const sys::MemoryBlock *const NearBlock,
+                       unsigned Flags, std::error_code &EC) override {
+    return sys::Memory::allocateMappedMemory(NumBytes, NearBlock, Flags, EC);
+  }
+
+  std::error_code protectMappedMemory(const sys::MemoryBlock &Block,
+                                      unsigned Flags) override {
+    return sys::Memory::protectMappedMemory(Block, Flags);
   }
+
+  std::error_code releaseMappedMemory(sys::MemoryBlock &M) override {
+    return sys::Memory::releaseMappedMemory(M);
   }
+};
+
+DefaultMMapper DefaultMMapperInstance;
+} // namespace
+
+SectionMemoryManager::SectionMemoryManager(MemoryMapper *MM)
+    : MMapper(MM ? *MM : DefaultMMapperInstance) {}
 
 } // namespace llvm