[Memory] Add basic support for large/huge memory pages

[aganea]

Summary

This adds Memory::MF_HUGE_HINT which indicates that allocateMappedMemory() shall return a pointer to a large memory page. However the flag is merly a hint because we're not guaranteed in any way that we will get back a large memory page. There are several restrictions:

• Large/huge memory pages aren't enabled by default on modern OSes (Windows 10 and Linux at least), and should be manually enabled/reserved.
• Once enabled, it should be kept in mind that large pages are physical only, they can't be swapped.
• Memory fragmentation can affect the availability of large pages, especially after running the OS for a long time and/or running along many other applications.

allocateMappedMemory() will fallback to 4KB pages if it can't allocate 2MB large pages (when Memory::MF_HUGE_HINT is provided)

Currently, this patch implements Memory::MF_HUGE_HINT only on Windows. The hint will be ignored on Linux, and 4KB pages will always be returned. Unfortunately I don't have a Linux box at hand, and WSL does not seem to support huge pages at the moment.

Also, testing on the build system is a bit tricky. I've added a unit test to exercise the codepath, although I can't ensure the OS will return a large/huge memory page. The test would be too fragile otherwise and could fail occasionally.

[aganea]

Fix tests on WSL (Ubuntu).

[rnk]

lgtm

lib/Support/Unix/Memory.inc

48

At the call site, we ought to add // FIXME: Handle huge page requests..

[zturner]

lib/Support/Windows/Memory.inc

62

Can we mark this function static?

68

Can you do an early out here if it failed?

75

Why do we actually even need to set this privilege on the process token? Will huge pages not work without it?

Also, it's strange to be doing this from a function called getLargePageSize which sounds like it doesn't modify anything.

78

There's a handle leak here.

[zturner]

lib/Support/Windows/Memory.inc

75

Actually I found this that explains it:

https://blogs.msdn.microsoft.com/oldnewthing/20110128-00/?p=11643

Is looking up the privilege value and opening the token expensive? If so, maybe you want to put this behind an llvm::call_once()

[aganea]

lib/Support/Windows/Memory.inc

75

getLargePageSize() returns into a static variable (see L114) which shall be thread-safe as per C++11 specification section 6.7.4. I know there were issues previously with VS2012, but any modern C++11/14 compiler should support that (I know for sure it works since VS2015).

Is there any reason for calling llvm::call_once() over static initialization in this case?

[aganea]

Adressed all comments, except for llvm::call_once.

[zturner]

lib/Support/Windows/Memory.inc

111–123

FWIW, this comment about static locals is no longer accurate. I wonder if we could write this as:

static size_t DefaultGranularity = getAllocationGranularity();
static Optional<size_t> LargePageGranularity = getLargePargeGranularity();

DWORD Flags = MEM_RESERVE | MEM_COMMIT;

size_t Granularity = DefaultGranularity;
if ((Flags & MF_HUGE_HINT) && LargePageGranularity.hasValue()) {
  Flags |= MEM_LARGE_PAGES;
  Granularity = *LargePageGranularity;
}

[aganea]

lib/Support/Windows/Memory.inc

111–123

I was tempted to change that code at first, but I said maybe in a later commit. Looks much better now - thank you!

[aganea]

Added comments.

include/llvm/Support/Memory.h

public:
     MemoryBlock() : Address(nullptr), Size(0) { }
     MemoryBlock(void *addr, size_t size) : Address(addr), Size(size) { }
     void *base() const { return Address; }
     size_t size() const { return Size; }
 
   private:
     void *Address;    ///< Address of first byte of memory area
     size_t Size;      ///< Size, in bytes of the memory area
+    unsigned Flags = 0;
     friend class Memory;
   };
 
   /// This class provides various memory handling functions that manipulate
   /// MemoryBlock instances.
   /// @since 1.4
   /// An abstraction for memory operations.
   class Memory {
   public:
     enum ProtectionFlags {
-      MF_READ  = 0x1000000,
+      MF_READ = 0x1000000,
       MF_WRITE = 0x2000000,
-      MF_EXEC  = 0x4000000
+      MF_EXEC = 0x4000000,
+      MF_RWE_MASK = 0x7000000,
+      MF_HUGE_HINT = 0x0000001
     };
 
     /// This method allocates a block of memory that is suitable for loading
     /// dynamically generated code (e.g. JIT). An attempt to allocate
     /// \p NumBytes bytes of virtual memory is made.
     /// \p NearBlock may point to an existing allocation in which case
     /// an attempt is made to allocate more memory near the existing block.
     /// The actual allocated address is not guaranteed to be near the requested

lib/Support/Unix/Memory.inc

 extern "C" void sys_icache_invalidate(const void *Addr, size_t len);
 #else
 extern "C" void __clear_cache(void *, void*);
 #endif
 
 namespace {
 
 int getPosixProtectionFlags(unsigned Flags) {
-  switch (Flags) {
+  switch (Flags & llvm::sys::Memory::MF_RWE_MASK) {
   case llvm::sys::Memory::MF_READ:
     return PROT_READ;
   case llvm::sys::Memory::MF_WRITE:
     return PROT_WRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_WRITE:
     return PROT_READ | PROT_WRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_EXEC:
     return PROT_READ | PROT_EXEC;
#endif
 
   // Use any near hint and the page size to set a page-aligned starting address
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                       NearBlock->size() : 0;
   static const size_t PageSize = Process::getPageSize();
   if (Start && Start % PageSize)
     Start += PageSize - Start % PageSize;
 
+  // FIXME: Handle huge page requests (MF_HUGE_HINT).
   void *Addr = ::mmap(reinterpret_cast<void *>(Start), NumBytes, Protect,
                       MMFlags, fd, 0);
   if (Addr == MAP_FAILED) {
     if (NearBlock) //Try again without a near hint
       return allocateMappedMemory(NumBytes, nullptr, PFlags, EC);
 
     EC = std::error_code(errno, std::generic_category());
     return MemoryBlock();

lib/Support/Windows/Memory.inc

 #include "llvm/Support/WindowsError.h"
 
 // The Windows.h header must be the last one included.
 #include "WindowsSupport.h"
 
 namespace {
 
 DWORD getWindowsProtectionFlags(unsigned Flags) {
-  switch (Flags) {
+  switch (Flags & llvm::sys::Memory::MF_RWE_MASK) {
   // Contrary to what you might expect, the Windows page protection flags
   // are not a bitwise combination of RWX values
   case llvm::sys::Memory::MF_READ:
     return PAGE_READONLY;
   case llvm::sys::Memory::MF_WRITE:
     // Note: PAGE_WRITE is not supported by VirtualProtect
     return PAGE_READWRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_WRITE:
     return PAGE_READWRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE_READ;
   case llvm::sys::Memory::MF_READ |
          llvm::sys::Memory::MF_WRITE |
          llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE_READWRITE;
   case llvm::sys::Memory::MF_EXEC:
     return PAGE_EXECUTE;
   default:
     llvm_unreachable("Illegal memory protection flag specified!");
   }
   // Provide a default return value as required by some compilers.
   return PAGE_NOACCESS;
 }
 
+// While we'd be happy to allocate single pages, the Windows allocation
+// granularity may be larger than a single page (in practice, it is 64K)
+// so mapping less than that will create an unreachable fragment of memory.
 size_t getAllocationGranularity() {
   SYSTEM_INFO  Info;
   ::GetSystemInfo(&Info);
   if (Info.dwPageSize > Info.dwAllocationGranularity)
     return Info.dwPageSize;
   else
     return Info.dwAllocationGranularity;
 }
 
+// Large/huge memory pages need explicit process permissions in order to be
+// used. See https://blogs.msdn.microsoft.com/oldnewthing/20110128-00/?p=11643
+// Also large pages need to be manually enabled on your OS. If all this is
+// sucessfull, we return the minimal large memory page size.
+static size_t enableProcessLargePages() {
+  HANDLE Token = 0;
+  size_t LargePageMin = GetLargePageMinimum();
+  if (LargePageMin)
+    OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY,
+                     &Token);
+  if (!Token)
+    return 0;
+  LUID Luid;
+  if (!LookupPrivilegeValue(0, SE_LOCK_MEMORY_NAME, &Luid)) {
+    CloseHandle(Token);
+    return 0;
+  }
+  TOKEN_PRIVILEGES TP{};
+  TP.PrivilegeCount = 1;
+  TP.Privileges[0].Luid = Luid;
+  TP.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+  if (!AdjustTokenPrivileges(Token, FALSE, &TP, 0, 0, 0)) {
+    CloseHandle(Token);
+    return 0;
+  }
+  DWORD E = GetLastError();
+  CloseHandle(Token);
+  if (E == ERROR_SUCCESS)
+    return LargePageMin;
+  return 0;
+}
+
 } // namespace
 
 namespace llvm {
 namespace sys {
 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only Win32 specific code
 //===          and must not be UNIX code
 //===----------------------------------------------------------------------===//
 
 MemoryBlock Memory::allocateMappedMemory(size_t NumBytes,
                                          const MemoryBlock *const NearBlock,
                                          unsigned Flags,
                                          std::error_code &EC) {
   EC = std::error_code();
   if (NumBytes == 0)
     return MemoryBlock();
 
-  // While we'd be happy to allocate single pages, the Windows allocation
-  // granularity may be larger than a single page (in practice, it is 64K)
-  // so mapping less than that will create an unreachable fragment of memory.
-  // Avoid using one-time initialization of static locals here, since they
-  // aren't thread safe with MSVC.
-  static volatile size_t GranularityCached;
-  size_t Granularity = GranularityCached;
-  if (Granularity == 0) {
-    Granularity = getAllocationGranularity();
-    GranularityCached = Granularity;
+  static size_t DefaultGranularity = getAllocationGranularity();
+  static Optional<size_t> LargePageGranularity = enableProcessLargePages();
+
+  DWORD AllocType = MEM_RESERVE | MEM_COMMIT;
+  bool HugePages = false;
+  size_t Granularity = DefaultGranularity;
+
+  if ((Flags & MF_HUGE_HINT) && LargePageGranularity.hasValue()) {
+    AllocType |= MEM_LARGE_PAGES;
+    HugePages = true;
+    Granularity = *LargePageGranularity;
   }
 
-  const size_t NumBlocks = (NumBytes+Granularity-1)/Granularity;
+  size_t NumBlocks = (NumBytes + Granularity - 1) / Granularity;
 
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                 NearBlock->size()
                            : 0;
 
   // If the requested address is not aligned to the allocation granularity,
   // round up to get beyond NearBlock. VirtualAlloc would have rounded down.
   if (Start && Start % Granularity != 0)
     Start += Granularity - Start % Granularity;
 
   DWORD Protect = getWindowsProtectionFlags(Flags);
 
-  void *PA = ::VirtualAlloc(reinterpret_cast<void*>(Start),
-                            NumBlocks*Granularity,
-                            MEM_RESERVE | MEM_COMMIT, Protect);
+  void *PA = ::VirtualAlloc(reinterpret_cast<void *>(Start),
+                            NumBlocks * Granularity, AllocType, Protect);
   if (PA == NULL) {
-    if (NearBlock) {
-      // Try again without the NearBlock hint
-      return allocateMappedMemory(NumBytes, NULL, Flags, EC);
+    if (NearBlock || HugePages) {
+      // Try again without the NearBlock hint and without large memory pages
+      return allocateMappedMemory(NumBytes, NULL, Flags & ~MF_HUGE_HINT, EC);
     }
     EC = mapWindowsError(::GetLastError());
     return MemoryBlock();
   }
 
   MemoryBlock Result;
   Result.Address = PA;
   Result.Size = NumBlocks*Granularity;
+  Result.Flags = (Flags & ~MF_HUGE_HINT) | (HugePages ? MF_HUGE_HINT : 0);
 
   if (Flags & MF_EXEC)
     Memory::InvalidateInstructionCache(Result.Address, Result.Size);
 
   return Result;
 }
 
   std::error_code Memory::releaseMappedMemory(MemoryBlock &M) {

unittests/Support/MemoryTest.cpp

TEST_P(MappedMemoryTest, AllocAndRelease) {
   EXPECT_EQ(std::error_code(), EC);
 
   EXPECT_NE((void*)nullptr, M1.base());
   EXPECT_LE(sizeof(int), M1.size());
 
   EXPECT_FALSE(Memory::releaseMappedMemory(M1));
 }
 
+TEST_P(MappedMemoryTest, AllocAndReleaseHuge) {
+  CHECK_UNSUPPORTED();
+  std::error_code EC;
+  MemoryBlock M1 = Memory::allocateMappedMemory(
+      sizeof(int), nullptr, Flags | Memory::MF_HUGE_HINT, EC);
+  EXPECT_EQ(std::error_code(), EC);
+
+  // Test large/huge memory pages. In the worst case, 4kb pages should be
+  // returned, if large pages aren't available.
+
+  EXPECT_NE((void *)nullptr, M1.base());
+  EXPECT_LE(sizeof(int), M1.size());
+
+  EXPECT_FALSE(Memory::releaseMappedMemory(M1));
+}
+
 TEST_P(MappedMemoryTest, MultipleAllocAndRelease) {
   CHECK_UNSUPPORTED();
   std::error_code EC;
   MemoryBlock M1 = Memory::allocateMappedMemory(16, nullptr, Flags, EC);
   EXPECT_EQ(std::error_code(), EC);
   MemoryBlock M2 = Memory::allocateMappedMemory(64, nullptr, Flags, EC);
   EXPECT_EQ(std::error_code(), EC);
   MemoryBlock M3 = Memory::allocateMappedMemory(32, nullptr, Flags, EC);