[Memory] Add basic support for large/huge memory pages

[aganea]

Summary

This adds Memory::MF_HUGE_HINT which indicates that allocateMappedMemory() shall return a pointer to a large memory page. However the flag is merly a hint because we're not guaranteed in any way that we will get back a large memory page. There are several restrictions:

• Large/huge memory pages aren't enabled by default on modern OSes (Windows 10 and Linux at least), and should be manually enabled/reserved.
• Once enabled, it should be kept in mind that large pages are physical only, they can't be swapped.
• Memory fragmentation can affect the availability of large pages, especially after running the OS for a long time and/or running along many other applications.

allocateMappedMemory() will fallback to 4KB pages if it can't allocate 2MB large pages (when Memory::MF_HUGE_HINT is provided)

Currently, this patch implements Memory::MF_HUGE_HINT only on Windows. The hint will be ignored on Linux, and 4KB pages will always be returned. Unfortunately I don't have a Linux box at hand, and WSL does not seem to support huge pages at the moment.

Also, testing on the build system is a bit tricky. I've added a unit test to exercise the codepath, although I can't ensure the OS will return a large/huge memory page. The test would be too fragile otherwise and could fail occasionally.

[aganea]

Fix tests on WSL (Ubuntu).

[rnk]

lgtm

lib/Support/Unix/Memory.inc

48 ↗

(On Diff #188547)

At the call site, we ought to add // FIXME: Handle huge page requests..

[zturner]

lib/Support/Windows/Memory.inc

59 ↗

(On Diff #188547)

Can we mark this function static?

65 ↗

(On Diff #188547)

Can you do an early out here if it failed?

72 ↗

(On Diff #188547)

Why do we actually even need to set this privilege on the process token? Will huge pages not work without it?

Also, it's strange to be doing this from a function called getLargePageSize which sounds like it doesn't modify anything.

75 ↗

(On Diff #188547)

There's a handle leak here.

[zturner]

lib/Support/Windows/Memory.inc

72 ↗

(On Diff #188547)

Actually I found this that explains it:

https://blogs.msdn.microsoft.com/oldnewthing/20110128-00/?p=11643

Is looking up the privilege value and opening the token expensive? If so, maybe you want to put this behind an llvm::call_once()

[aganea]

lib/Support/Windows/Memory.inc

72 ↗

(On Diff #188547)

getLargePageSize() returns into a static variable (see L114) which shall be thread-safe as per C++11 specification section 6.7.4. I know there were issues previously with VS2012, but any modern C++11/14 compiler should support that (I know for sure it works since VS2015).

Is there any reason for calling llvm::call_once() over static initialization in this case?

[aganea]

Adressed all comments, except for llvm::call_once.

[zturner]

lib/Support/Windows/Memory.inc

105–115 ↗

(On Diff #188602)

FWIW, this comment about static locals is no longer accurate. I wonder if we could write this as:

static size_t DefaultGranularity = getAllocationGranularity();
static Optional<size_t> LargePageGranularity = getLargePargeGranularity();

DWORD Flags = MEM_RESERVE | MEM_COMMIT;

size_t Granularity = DefaultGranularity;
if ((Flags & MF_HUGE_HINT) && LargePageGranularity.hasValue()) {
  Flags |= MEM_LARGE_PAGES;
  Granularity = *LargePageGranularity;
}

[aganea]

lib/Support/Windows/Memory.inc

105–115 ↗

(On Diff #188602)

I was tempted to change that code at first, but I said maybe in a later commit. Looks much better now - thank you!

[aganea]

Added comments.

llvm/trunk/include/llvm/Support/Memory.h

public:
     MemoryBlock() : Address(nullptr), Size(0) { }
     MemoryBlock(void *addr, size_t size) : Address(addr), Size(size) { }
     void *base() const { return Address; }
     size_t size() const { return Size; }
 
   private:
     void *Address;    ///< Address of first byte of memory area
     size_t Size;      ///< Size, in bytes of the memory area
+    unsigned Flags = 0;
     friend class Memory;
   };
 
   /// This class provides various memory handling functions that manipulate
   /// MemoryBlock instances.
   /// @since 1.4
   /// An abstraction for memory operations.
   class Memory {
   public:
     enum ProtectionFlags {
-      MF_READ  = 0x1000000,
+      MF_READ = 0x1000000,
       MF_WRITE = 0x2000000,
-      MF_EXEC  = 0x4000000
+      MF_EXEC = 0x4000000,
+      MF_RWE_MASK = 0x7000000,
+      MF_HUGE_HINT = 0x0000001
     };
 
     /// This method allocates a block of memory that is suitable for loading
     /// dynamically generated code (e.g. JIT). An attempt to allocate
     /// \p NumBytes bytes of virtual memory is made.
     /// \p NearBlock may point to an existing allocation in which case
     /// an attempt is made to allocate more memory near the existing block.
     /// The actual allocated address is not guaranteed to be near the requested

llvm/trunk/lib/Support/Unix/Memory.inc

 extern "C" void sys_icache_invalidate(const void *Addr, size_t len);
 #else
 extern "C" void __clear_cache(void *, void*);
 #endif
 
 namespace {
 
 int getPosixProtectionFlags(unsigned Flags) {
-  switch (Flags) {
+  switch (Flags & llvm::sys::Memory::MF_RWE_MASK) {
   case llvm::sys::Memory::MF_READ:
     return PROT_READ;
   case llvm::sys::Memory::MF_WRITE:
     return PROT_WRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_WRITE:
     return PROT_READ | PROT_WRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_EXEC:
     return PROT_READ | PROT_EXEC;
#endif
 
   // Use any near hint and the page size to set a page-aligned starting address
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                       NearBlock->size() : 0;
   static const size_t PageSize = Process::getPageSize();
   if (Start && Start % PageSize)
     Start += PageSize - Start % PageSize;
 
+  // FIXME: Handle huge page requests (MF_HUGE_HINT).
   void *Addr = ::mmap(reinterpret_cast<void *>(Start), NumBytes, Protect,
                       MMFlags, fd, 0);
   if (Addr == MAP_FAILED) {
     if (NearBlock) //Try again without a near hint
       return allocateMappedMemory(NumBytes, nullptr, PFlags, EC);
 
     EC = std::error_code(errno, std::generic_category());
     return MemoryBlock();

llvm/trunk/lib/Support/Windows/Memory.inc

 #include "llvm/Support/WindowsError.h"
 
 // The Windows.h header must be the last one included.
 #include "WindowsSupport.h"
 
 namespace {
 
 DWORD getWindowsProtectionFlags(unsigned Flags) {
-  switch (Flags) {
+  switch (Flags & llvm::sys::Memory::MF_RWE_MASK) {
   // Contrary to what you might expect, the Windows page protection flags
   // are not a bitwise combination of RWX values
   case llvm::sys::Memory::MF_READ:
     return PAGE_READONLY;
   case llvm::sys::Memory::MF_WRITE:
     // Note: PAGE_WRITE is not supported by VirtualProtect
     return PAGE_READWRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_WRITE:
     return PAGE_READWRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE_READ;
   case llvm::sys::Memory::MF_READ |
          llvm::sys::Memory::MF_WRITE |
          llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE_READWRITE;
   case llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE;
   default:
     llvm_unreachable("Illegal memory protection flag specified!");
   }
   // Provide a default return value as required by some compilers.
   return PAGE_NOACCESS;
 }
 
+// While we'd be happy to allocate single pages, the Windows allocation
+// granularity may be larger than a single page (in practice, it is 64K)
+// so mapping less than that will create an unreachable fragment of memory.
 size_t getAllocationGranularity() {
   SYSTEM_INFO  Info;
   ::GetSystemInfo(&Info);
   if (Info.dwPageSize > Info.dwAllocationGranularity)
     return Info.dwPageSize;
   else
     return Info.dwAllocationGranularity;
 }
 
+// Large/huge memory pages need explicit process permissions in order to be
+// used. See https://blogs.msdn.microsoft.com/oldnewthing/20110128-00/?p=11643
+// Also large pages need to be manually enabled on your OS. If all this is
+// sucessfull, we return the minimal large memory page size.
+static size_t enableProcessLargePages() {
+  HANDLE Token = 0;
+  size_t LargePageMin = GetLargePageMinimum();
+  if (LargePageMin)
+    OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY,
+                     &Token);
+  if (!Token)
+    return 0;
+  LUID Luid;
+  if (!LookupPrivilegeValue(0, SE_LOCK_MEMORY_NAME, &Luid)) {
+    CloseHandle(Token);
+    return 0;
+  }
+  TOKEN_PRIVILEGES TP{};
+  TP.PrivilegeCount = 1;
+  TP.Privileges[0].Luid = Luid;
+  TP.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+  if (!AdjustTokenPrivileges(Token, FALSE, &TP, 0, 0, 0)) {
+    CloseHandle(Token);
+    return 0;
+  }
+  DWORD E = GetLastError();
+  CloseHandle(Token);
+  if (E == ERROR_SUCCESS)
+    return LargePageMin;
+  return 0;
+}
+
 } // namespace
 
 namespace llvm {
 namespace sys {
 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only Win32 specific code
 //===          and must not be UNIX code
 //===----------------------------------------------------------------------===//
 
 MemoryBlock Memory::allocateMappedMemory(size_t NumBytes,
                                          const MemoryBlock *const NearBlock,
                                          unsigned Flags,
                                          std::error_code &EC) {
   EC = std::error_code();
   if (NumBytes == 0)
     return MemoryBlock();
 
-  // While we'd be happy to allocate single pages, the Windows allocation
-  // granularity may be larger than a single page (in practice, it is 64K)
-  // so mapping less than that will create an unreachable fragment of memory.
-  // Avoid using one-time initialization of static locals here, since they
-  // aren't thread safe with MSVC.
-  static volatile size_t GranularityCached;
-  size_t Granularity = GranularityCached;
-  if (Granularity == 0) {
-    Granularity = getAllocationGranularity();
-    GranularityCached = Granularity;
+  static size_t DefaultGranularity = getAllocationGranularity();
+  static Optional<size_t> LargePageGranularity = enableProcessLargePages();
+
+  DWORD AllocType = MEM_RESERVE | MEM_COMMIT;
+  bool HugePages = false;
+  size_t Granularity = DefaultGranularity;
+
+  if ((Flags & MF_HUGE_HINT) && LargePageGranularity.hasValue()) {
+    AllocType |= MEM_LARGE_PAGES;
+    HugePages = true;
+    Granularity = *LargePageGranularity;
   }
 
-  const size_t NumBlocks = (NumBytes+Granularity-1)/Granularity;
+  size_t NumBlocks = (NumBytes + Granularity - 1) / Granularity;
 
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                 NearBlock->size()
                            : 0;
 
   // If the requested address is not aligned to the allocation granularity,
   // round up to get beyond NearBlock. VirtualAlloc would have rounded down.
   if (Start && Start % Granularity != 0)
     Start += Granularity - Start % Granularity;
 
   DWORD Protect = getWindowsProtectionFlags(Flags);
 
-  void *PA = ::VirtualAlloc(reinterpret_cast<void*>(Start),
-                            NumBlocks*Granularity,
-                            MEM_RESERVE | MEM_COMMIT, Protect);
+  void *PA = ::VirtualAlloc(reinterpret_cast<void *>(Start),
+                            NumBlocks * Granularity, AllocType, Protect);
   if (PA == NULL) {
-    if (NearBlock) {
-      // Try again without the NearBlock hint
-      return allocateMappedMemory(NumBytes, NULL, Flags, EC);
+    if (NearBlock || HugePages) {
+      // Try again without the NearBlock hint and without large memory pages
+      return allocateMappedMemory(NumBytes, NULL, Flags & ~MF_HUGE_HINT, EC);
     }
     EC = mapWindowsError(::GetLastError());
     return MemoryBlock();
   }
 
   MemoryBlock Result;
   Result.Address = PA;
   Result.Size = NumBlocks*Granularity;
+  Result.Flags = (Flags & ~MF_HUGE_HINT) | (HugePages ? MF_HUGE_HINT : 0);
 
   if (Flags & MF_EXEC)
     Memory::InvalidateInstructionCache(Result.Address, Result.Size);
 
   return Result;
 }
 
   std::error_code Memory::releaseMappedMemory(MemoryBlock &M) {

llvm/trunk/unittests/Support/MemoryTest.cpp

TEST_P(MappedMemoryTest, AllocAndRelease) {
   EXPECT_EQ(std::error_code(), EC);
 
   EXPECT_NE((void*)nullptr, M1.base());
   EXPECT_LE(sizeof(int), M1.size());
 
   EXPECT_FALSE(Memory::releaseMappedMemory(M1));
 }
 
+TEST_P(MappedMemoryTest, AllocAndReleaseHuge) {
+  CHECK_UNSUPPORTED();
+  std::error_code EC;
+  MemoryBlock M1 = Memory::allocateMappedMemory(
+      sizeof(int), nullptr, Flags | Memory::MF_HUGE_HINT, EC);
+  EXPECT_EQ(std::error_code(), EC);
+
+  // Test large/huge memory pages. In the worst case, 4kb pages should be
+  // returned, if large pages aren't available.
+
+  EXPECT_NE((void *)nullptr, M1.base());
+  EXPECT_LE(sizeof(int), M1.size());
+
+  EXPECT_FALSE(Memory::releaseMappedMemory(M1));
+}
+
 TEST_P(MappedMemoryTest, MultipleAllocAndRelease) {
   CHECK_UNSUPPORTED();
   std::error_code EC;
   MemoryBlock M1 = Memory::allocateMappedMemory(16, nullptr, Flags, EC);
   EXPECT_EQ(std::error_code(), EC);
   MemoryBlock M2 = Memory::allocateMappedMemory(64, nullptr, Flags, EC);
   EXPECT_EQ(std::error_code(), EC);
   MemoryBlock M3 = Memory::allocateMappedMemory(32, nullptr, Flags, EC);