[Support] Support MF_HUGE_HINT on Linux and FreeBSD

[riccibruno]

Summary

Implement support for large page requests with the allocateMappedMemory interface on Linux and FreeBSD. If the MF_HUGE_HINT flag is passed we first try an mmap with the relevant flag. If that fails we fall back to a normal mmap. On some systems (such as Linux with transparent huge pages) it is possible to indicate with madvise that a certain range of memory should use large pages. We do this if supported and the first mmap with the large page flag failed.

Note that only a little test is added since the allocateMappedMemory interface is already well-tested. It seems difficult to reliably test that we got large pages. I am open to suggestions here (maybe with userfaultfd ?).

[riccibruno]

llvm/lib/Support/Unix/Memory.inc

163

Note that the choice to first try without the large page hint is arbitrary. If anyone has an argument for one way or another please go ahead.

[rnk]

+@aganea, who added this flag.

Note that only a little test is added since the allocateMappedMemory interface is already well-tested. It seems difficult to reliably test that we got large pages. I am open to suggestions here (maybe with userfaultfd ?).

At a certain point, this would really be a test for the OS. That's pretty hard. I think this is fine without additional testing.

llvm/lib/Support/Unix/Memory.inc

79

Prefer static over anon ns:
https://llvm.org/docs/CodingStandards.html#anonymous-namespaces

169

This code pattern of a recursive tail call shows up in a few places in old LLVM code like this. Personally, I don't like it. It seems like a do / while loop around the mmap call itself would be clearer, since we don't need to re-run most of the flag calculations above. This would also avoid the Impl rename.

[riccibruno]

In D71975#1799355, @rnk wrote:

+@aganea, who added this flag.

Note that only a little test is added since the allocateMappedMemory interface is already well-tested. It seems difficult to reliably test that we got large pages. I am open to suggestions here (maybe with userfaultfd ?).

At a certain point, this would really be a test for the OS. That's pretty hard. I think this is fine without additional testing.

So I did rework the patch to address your comments, but I found out an "interesting" design decision with how mmap/munmap work on linux:

• mmap with the MAP_HUGETLB flag automatically adjusts the length parameter to be a multiple of the size of a huge page. But there is no easy way to find out to what value the length was adjusted to. There is also no easy way to find what the size of a huge page is. The recommended method (and what libhugetlbfs does) is to parse /proc/meminfo.
• munmap requires the adjusted length.

So I need to:

• Add a getHugePageSize similar to sys::Process::getPageSize which will do its best to find out about the huge page size(s) on the platform. This needs to be done for Windows too.
• Use that in allocateMappedMemory.

llvm/lib/Support/Unix/Memory.inc

169

I agree that your suggestion is nicer.

[aganea]

In D71975#1803542, @riccibruno wrote:

So I need to:

• Add a getHugePageSize similar to sys::Process::getPageSize which will do its best to find out about the huge page size(s) on the platform. This needs to be done for Windows too.

Do you need to know the large page size outside of Memory.inc? If no, you could mark it as a static function, in the same way as the Windows enableProcessLargePages() (which detects the minimum large page size). Additionally, you can also take a look at how rpmalloc does it: rpmalloc.c, L1802-L1871 in D71786, as it already handles the same platforms as LLVM.

[bsdjhb]

For FreeBSD, MAP_ALIGNED_SUPER probably isn't what you want. The default mmap() for MAP_ANON on FreeBSD tries to do what MAP_ALIGNED_SUPER does if the request is at least one large page in size first, and if that fails to find available address space, falls back to finding any possible virtual addresses (it does the two steps you are doing inside of mmap() itself). Specifying MAP_ALIGNED_SUPER tells mmap() to only try the first pass and fail without trying the second pass, but then the second call to mmap() that you would make as a result would just do both passes anyway.

FreeBSD doesn't currently have a "must-only-use-large-pages-and-never-demote-to-small-pages" flag which is probably what you would want to use here instead. (We've talked about adding one, but no one has done it to date.)

[riccibruno]

In D71975#1809205, @bsdjhb wrote:

For FreeBSD, MAP_ALIGNED_SUPER probably isn't what you want. The default mmap() for MAP_ANON on FreeBSD tries to do what MAP_ALIGNED_SUPER does if the request is at least one large page in size first, and if that fails to find available address space, falls back to finding any possible virtual addresses (it does the two steps you are doing inside of mmap() itself). Specifying MAP_ALIGNED_SUPER tells mmap() to only try the first pass and fail without trying the second pass, but then the second call to mmap() that you would make as a result would just do both passes anyway.

Ah, I did not know that. Thanks for your comment. The reason I used MAP_ALIGNED_SUPER is the note in the man page for mmap(2):

The MAP_ALIGNED_SUPER flag is an optimization that will align the mapping request to the size of a large page similar to MAP_ALIGNED [...]

but what you are saying is that for large enough requests with MAP_ANON the default is already MAP_ALIGNED_SUPER with a fallback. The only difference with using MAP_ALIGNED_SUPER is that we can detect if the first mmap fails and clear the MF_HUGE_HINT flag. However this is not actually useful since the MF_HUGE_HINT flag in the returned block of memory has no reliable meaning (on some systems the block of memory may use large pages even if MF_HUGE_HINT is cleared, and on some other systems the block of memory may use small pages even if MF_HUGE_HINT is set).

FreeBSD doesn't currently have a "must-only-use-large-pages-and-never-demote-to-small-pages" flag which is probably what you would want to use here instead. (We've talked about adding one, but no one has done it to date.)

Not really, MF_HUGE_HINT is only a hint and a fallback to small pages is entirely acceptable.

llvm/include/llvm/Support/Memory.h

namespace sys {
   /// This class encapsulates the notion of a memory block which has an address
   /// and a size. It is used by the Memory class (a friend) as the result of
   /// various memory allocation operations.
   /// @see Memory
   /// Memory block abstraction.
   class MemoryBlock {
   public:
     MemoryBlock() : Address(nullptr), AllocatedSize(0) {}
-    MemoryBlock(void *addr, size_t allocatedSize)
-        : Address(addr), AllocatedSize(allocatedSize) {}
+    MemoryBlock(void *addr, size_t allocatedSize, unsigned Flags = 0)
+        : Address(addr), AllocatedSize(allocatedSize), Flags(Flags) {}
     void *base() const { return Address; }
     /// The size as it was allocated. This is always greater or equal to the
     /// size that was originally requested.
     size_t allocatedSize() const { return AllocatedSize; }
-  
+    unsigned flags() const { return Flags; }
   private:
-    void *Address;    ///< Address of first byte of memory area
+    void *Address;        ///< Address of first byte of memory area
     size_t AllocatedSize; ///< Size, in bytes of the memory area
     unsigned Flags = 0;
     friend class Memory;
   };
 
   /// This class provides various memory handling functions that manipulate
   /// MemoryBlock instances.
   /// @since 1.4

llvm/lib/Support/Unix/Memory.inc

 #endif
   default:
     llvm_unreachable("Illegal memory protection flag specified!");
   }
   // Provide a default return value as required by some compilers.
   return PROT_NONE;
 }
 
+int getHugeMmapFlags(unsigned Flags) {
+  if (Flags & llvm::sys::Memory::MF_HUGE_HINT) {
+#if defined(MAP_HUGETLB)
+    return MAP_HUGETLB; // Linux >=2.6.32
+#elif defined(MAP_ALIGNED_SUPER)
+    return MAP_ALIGNED_SUPER; // FreeBSD >=9
+#endif
+    // fallthrough
+  }
+  return 0;
+}
+
+int getHugeMadviseFlags(unsigned Flags) {
+  if (Flags & llvm::sys::Memory::MF_HUGE_HINT) {
+#if defined(MADV_HUGEPAGE) // Linux >= 2.6.38
+    return MADV_HUGEPAGE;
+#endif
+    // fallthrough
+  }
+  return 0;
+}
+
 } // anonymous namespace
 
 namespace llvm {
 namespace sys {
 
-MemoryBlock
-Memory::allocateMappedMemory(size_t NumBytes,
+static MemoryBlock allocateMappedMemoryImpl(size_t NumBytes,
                                             const MemoryBlock *const NearBlock,
                                             unsigned PFlags,
-                             std::error_code &EC) {
+                                            std::error_code &EC,
+                                            bool MmapHugePageFailed) {
   EC = std::error_code();
   if (NumBytes == 0)
     return MemoryBlock();
 
   // On platforms that have it, we can use MAP_ANON to get a memory-mapped
   // page without file backing, but we need a fallback of opening /dev/zero
   // for strictly POSIX platforms instead.
   int fd;
#endif
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                       NearBlock->allocatedSize() : 0;
   static const size_t PageSize = Process::getPageSizeEstimate();
   const size_t NumPages = (NumBytes+PageSize-1)/PageSize;
 
   if (Start && Start % PageSize)
     Start += PageSize - Start % PageSize;
 
-  // FIXME: Handle huge page requests (MF_HUGE_HINT).
-  void *Addr = ::mmap(reinterpret_cast<void *>(Start), PageSize*NumPages, Protect,
-                      MMFlags, fd, 0);
-  if (Addr == MAP_FAILED) {
-    if (NearBlock) { //Try again without a near hint
+  // Handle huge page requests (MF_HUGE_HINT) as follow:
+  // 1. Try to directly request huge pages by passing the appropriate flag
+  //    to mmap.
+  // 2. If the above fails, try again without the huge page request. In this
+  //    case if supported, call madvise with a huge page hint.
+  //
+  // If we have both an address hint an a huge page hint, on failure first try
+  // without the huge page hint and then try without both.
+  int HugeMmapFlags = getHugeMmapFlags(PFlags);
+
+  void *Addr = ::mmap(reinterpret_cast<void *>(Start), PageSize * NumPages,
+                      Protect, MMFlags | HugeMmapFlags, fd, 0);
+
 #if !defined(MAP_ANON)
-      close(fd);
+  close(fd); // No longer needed.
 #endif
-      return allocateMappedMemory(NumBytes, nullptr, PFlags, EC);
-    }
 
+  if (Addr == MAP_FAILED) {
+    // The failure could be due to either a huge page request, or
+    // an address hint. Try first without the huge page request.
+    if (PFlags & llvm::sys::Memory::MF_HUGE_HINT)
+      return allocateMappedMemoryImpl(NumBytes, NearBlock,
+                                      PFlags & ~llvm::sys::Memory::MF_HUGE_HINT,
+                                      EC, /*MmapHugePageFailed=*/true);
+
+    // Then try again without a near hint.
+    if (NearBlock)
+      return allocateMappedMemoryImpl(NumBytes, /*NearBlock=*/nullptr, PFlags,
+                                      EC, MmapHugePageFailed);
+
+    // Ok, we really failed. Give up.
     EC = std::error_code(errno, std::generic_category());
-#if !defined(MAP_ANON)
-    close(fd);
-#endif
     return MemoryBlock();
   }
 
-#if !defined(MAP_ANON)
-  close(fd);
-#endif
+  // Hint that we want huge pages if supported and we failed to directly ask
+  // for it above. If the madvise call succeeds we will set the MF_HUGE_HINT
+  // flag in the returned memory block.
+  int HugeMadviseFlags = getHugeMadviseFlags(llvm::sys::Memory::MF_HUGE_HINT);
+  bool HugeMadviseSuccess = false;
+  if (MmapHugePageFailed && HugeMadviseFlags) {
+    HugeMadviseSuccess =
+        ::madvise(Addr, PageSize * NumPages, HugeMadviseFlags) == 0;
+  }
 
-  MemoryBlock Result;
-  Result.Address = Addr;
-  Result.AllocatedSize = PageSize*NumPages;
-  Result.Flags = PFlags;
+  MemoryBlock Result(
+      Addr, PageSize * NumPages,
+      PFlags | (HugeMadviseSuccess ? llvm::sys::Memory::MF_HUGE_HINT : 0));
 
   // Rely on protectMappedMemory to invalidate instruction cache.
-  if (PFlags & MF_EXEC) {
-    EC = Memory::protectMappedMemory (Result, PFlags);
+  if (PFlags & llvm::sys::Memory::MF_EXEC) {
+    EC = Memory::protectMappedMemory(Result, PFlags);
     if (EC != std::error_code())
       return MemoryBlock();
   }
 
   return Result;
 }
 
-std::error_code
-Memory::releaseMappedMemory(MemoryBlock &M) {
+MemoryBlock Memory::allocateMappedMemory(size_t NumBytes,
+                                         const MemoryBlock *const NearBlock,
+                                         unsigned PFlags, std::error_code &EC) {
+  return allocateMappedMemoryImpl(NumBytes, NearBlock, PFlags, EC,
+                                  /*MmapHugePageFailed=*/false);
+}
+
+std::error_code Memory::releaseMappedMemory(MemoryBlock &M) {
   if (M.Address == nullptr || M.AllocatedSize == 0)
     return std::error_code();
 
   if (0 != ::munmap(M.Address, M.AllocatedSize))
     return std::error_code(errno, std::generic_category());
 
   M.Address = nullptr;
   M.AllocatedSize = 0;

llvm/unittests/Support/MemoryTest.cpp

TEST_P(MappedMemoryTest, AllocAndRelease) {
   CHECK_UNSUPPORTED();
   std::error_code EC;
   MemoryBlock M1 = Memory::allocateMappedMemory(sizeof(int), nullptr, Flags,EC);
   EXPECT_EQ(std::error_code(), EC);
 
   EXPECT_NE((void*)nullptr, M1.base());
   EXPECT_LE(sizeof(int), M1.allocatedSize());
 
+  // The returned block of memory should not magically
+  // get the MF_HUGE_HINT flag if we did not ask for it.
+  EXPECT_FALSE(M1.flags() & Memory::MF_HUGE_HINT);
+
   EXPECT_FALSE(Memory::releaseMappedMemory(M1));
 }
 
 TEST_P(MappedMemoryTest, AllocAndReleaseHuge) {
   CHECK_UNSUPPORTED();
   std::error_code EC;
   MemoryBlock M1 = Memory::allocateMappedMemory(
       sizeof(int), nullptr, Flags | Memory::MF_HUGE_HINT, EC);