Enable pdbpagesize to allow support for PDB file sizes > 4GB

[chrdavis]

Summary

Enable the pdbpagesize flag to allow linking of PDB files > 4GB. Also includes a couple small fixes to change to uint64_t to support the larger file sizes. I updated the max file size check in MSFBuilder.cpp to take into account the page size.

[thakis]

Nice!

Before we turn this on, we should probably have some tests for /pdbpagesize: -- at least checking that llvm-pdbutil can read pdbs with different page size. If you have a pdb > 4gb sitting around locally, it'd also be good to verify that llvm-pdbutil dump or llvm-pdbutil pretty --native can open and read those large pdbs.

[chrdavis]

I'll add some unit tests over the weekend.

Here are some examples of a > 4GB PDB with 16k page size

D:\llvm-project\build\Debug\bin\llvm-pdbutil.exe pretty --native d:\scratch\4GBPDBWork\unit_tests\16384\unit_tests.exe.pdb

Summary for d:\scratch\4GBPDBWork\unit_tests\16384\unit_tests.exe.pdb
  Size: 4531617792 bytes
  Guid: {257AC4E5-274E-9019-4C4C-44205044422E}
  Age: 1
  Attributes: HasPrivateSymbols

D:\llvm-project\build\Debug\bin\llvm-pdbutil.exe dump -summary d:\scratch\4GBPDBWork\unit_tests\16384\unit_tests.exe.pdb

                          Summary
============================================================
  Block Size: 16384
  Number of blocks: 276588
  Number of streams: 18124
  Signature: 628802789
  Age: 1
  GUID: {257AC4E5-274E-9019-4C4C-44205044422E}
  Features: 0x1
  Has Debug Info: true
  Has Types: true
  Has IDs: true
  Has Globals: true
  Has Publics: true
  Is incrementally linked: false
  Has conflicting types: false
  Is stripped: false

[thakis]

(from the presubmit bots, looks like this needs an update to an existing test too: https://buildkite.com/llvm-project/premerge-checks/builds/68334#25aa8614-2e89-4dd7-a7ae-229dd5a587d3 => Failed Tests (1): lld :: COFF/pdbpagesize.test)

[chrdavis]

Updated pdbpagesize test

[chrdavis]

Fix for pdbpagesize unit tests

[thakis]

This basically lgtm. One comment to fix a diagnostic below.

I patched this in and checked that lld now indeed produces PDBs > 4GiB when needed (with /pdbpagesize:8192, with the repro in crbug.com/1179085) and that llvm-pdbutil can dump the summary of such large PDBs.

Do you need me to commit this for you?

llvm/lib/DebugInfo/MSF/MSFBuilder.cpp

360–361

Take a look at llvm/lib/DebugInfo/MSF/MSFError.cpp:

case msf_error_code::size_overflow:
  return "Output data is larger than 4 GiB.";

The lld linker for > 4GB files is: lld-link: error: Output data is larger than 4 GiB. File size would have been 4,593,000,448

But we'll use the "larger than 4 GiB" text even with page size 8192.

The easiest fix is probably to rename the current size_overflow to size_overflow_4096 and add size_overflow_8192, size_overflow_16384, size_overflow_32768. (Maybe there's a nicer fix too, but that'd work.)

Maybe the string returned by MSFError.cpp could mention ", the maximum for a PDB with page size XXX" so that users get the idea that the page size is now configurable.

[chrdavis]

Update size overflow error text

[chrdavis]

@thakis - Yes if you could commit on my behalf I would appreciate it.

[thakis]

Is --author='Chris Davis <[e-mail]>' good? Or should I use something else?

[chrdavis]

That works thanks!

lld/COFF/DriverUtils.cpp

if (s.getAsInteger(0, v)) {
     error("/pdbpagesize: invalid argument: " + s);
     return;
   }
   if (v != 4096 && v != 8192 && v != 16384 && v != 32768) {
     error("/pdbpagesize: invalid argument: " + s);
     return;
   }
 
-  // FIXME: Remove this once other page sizes work.
-  if (v != 4096) {
-    warn("/pdbpagesize: page sizes != 4096 not yet implemented, ignoring flag");
-    v = 4096;
-  }
-
   config->pdbPageSize = v;
 }
 
 static uint32_t parseSectionAttributes(StringRef s) {
   uint32_t ret = 0;
   for (char c : s.lower()) {
     switch (c) {
     case 'd':

lld/test/COFF/pdbpagesize.test

 # RUN: yaml2obj %p/Inputs/empty.yaml -o %t.obj
 
 # RUN: not lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:hi 2>&1 \
 # RUN:     | FileCheck --check-prefix=INVALID %s
 # RUN: not lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:15 2>&1 \
 # RUN:     | FileCheck --check-prefix=INVALID %s
 # INVALID: error: /pdbpagesize: invalid argument:
 
 # RUN: lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:4096
 # RUN: llvm-pdbutil pdb2yaml %t.pdb | FileCheck --check-prefix=PAGE4096 %s
 # PAGE4096: BlockSize: 4096
 
-# RUN: lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:8192 2>&1 \
-# RUN:     | FileCheck --check-prefix=TODO %s
-# TODO: warning: /pdbpagesize: page sizes != 4096 not yet implemented, ignoring flag
+# RUN: lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:8192
+# RUN: llvm-pdbutil pdb2yaml %t.pdb | FileCheck --check-prefix=PAGE8192 %s
+# PAGE8192: BlockSize: 8192
+
+# RUN: lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:16384
+# RUN: llvm-pdbutil pdb2yaml %t.pdb | FileCheck --check-prefix=PAGE16384 %s
+# PAGE16384: BlockSize: 16384
+
+# RUN: lld-link /entry:main %t.obj /out:%t.exe /debug /pdbpagesize:32768
+# RUN: llvm-pdbutil pdb2yaml %t.pdb | FileCheck --check-prefix=PAGE32768 %s
+# PAGE32768: BlockSize: 32768

llvm/include/llvm/DebugInfo/MSF/MSFCommon.h

inline bool isValidBlockSize(uint32_t Size) {
   case 8192:
   case 16384:
   case 32768:
     return true;
   }
   return false;
 }
 
+/// Given the specified block size, returns the maximum possible file size.
+/// Block Size  |  Max File Size
+/// <= 4096     |      4GB
+///    8192     |      8GB
+///   16384     |      16GB
+///   32768     |      32GB
+/// \p Size - the block size of the MSF
+inline uint64_t getMaxFileSizeFromBlockSize(uint32_t Size) {
+  switch (Size) {
+  case 8192:
+    return (uint64_t)UINT32_MAX * 2ULL;
+  case 16384:
+    return (uint64_t)UINT32_MAX * 3ULL;
+  case 32768:
+    return (uint64_t)UINT32_MAX * 4ULL;
+  default:
+    return (uint64_t)UINT32_MAX;
+  }
+}
+
 // Super Block, Fpm0, Fpm1, and Block Map
 inline uint32_t getMinimumBlockCount() { return 4; }
 
 // Super Block, Fpm0, and Fpm1 are reserved.  The Block Map, although required
 // need not be at block 3.
 inline uint32_t getFirstUnreservedBlock() { return 3; }
 
 inline uint64_t bytesToBlocks(uint64_t NumBytes, uint64_t BlockSize) {

llvm/include/llvm/DebugInfo/MSF/MSFError.h

 
 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace msf {
 enum class msf_error_code {
   unspecified = 1,
   insufficient_buffer,
-  size_overflow,
+  size_overflow_4096,
+  size_overflow_8192,
+  size_overflow_16384,
+  size_overflow_32768,
   not_writable,
   no_stream,
   invalid_format,
   block_in_use
 };
 } // namespace msf
 } // namespace llvm
 

llvm/include/llvm/DebugInfo/PDB/Native/PDBFile.h

public:
   uint32_t getNumDirectoryBlocks() const;
   uint64_t getBlockMapOffset() const;
 
   uint32_t getNumStreams() const override;
   uint32_t getMaxStreamSize() const;
   uint32_t getStreamByteSize(uint32_t StreamIndex) const override;
   ArrayRef<support::ulittle32_t>
   getStreamBlockList(uint32_t StreamIndex) const override;
-  uint32_t getFileSize() const;
+  uint64_t getFileSize() const;
 
   Expected<ArrayRef<uint8_t>> getBlockData(uint32_t BlockIndex,
                                            uint32_t NumBytes) const override;
   Error setBlockData(uint32_t BlockIndex, uint32_t Offset,
                      ArrayRef<uint8_t> Data) const override;
 
   ArrayRef<support::ulittle32_t> getStreamSizes() const {
     return ContainerLayout.StreamSizes;

llvm/lib/DebugInfo/MSF/MSFBuilder.cpp

Expected<FileBufferByteStream> MSFBuilder::commit(StringRef Path,
                                                   MSFLayout &Layout) {
   Expected<MSFLayout> L = generateLayout();
   if (!L)
     return L.takeError();
 
   Layout = std::move(*L);
 
   uint64_t FileSize = uint64_t(Layout.SB->BlockSize) * Layout.SB->NumBlocks;
-  if (FileSize > UINT32_MAX) {
-    // FIXME: Changing the BinaryStream classes to use 64-bit numbers lets
-    // us create PDBs larger than 4 GiB successfully. The file format is
-    // block-based and as long as each stream is small enough, PDBs larger than
-    // 4 GiB might work. Check if tools can handle these large PDBs, and if so
-    // add support for writing them.
+  // Ensure that the file size is under the limit for the specified block size.
+  if (FileSize > getMaxFileSizeFromBlockSize(Layout.SB->BlockSize)) {
+    msf_error_code error_code;
+    switch (Layout.SB->BlockSize) {
+    case 8192:
+      error_code = msf_error_code::size_overflow_8192;
+    case 16384:
+      error_code = msf_error_code::size_overflow_16384;
+    case 32768:
+      error_code = msf_error_code::size_overflow_32768;
+    default:
+      error_code = msf_error_code::size_overflow_4096;
+    }
+
     return make_error<MSFError>(
-        msf_error_code::size_overflow,
-        formatv("File size would have been {0,1:N}", FileSize));
+        error_code,
+        formatv("File size would have been {0,1:N} which exceeds the maximum \
+                 file size for the current page size {1}",
+                FileSize, Layout.SB->BlockSize));
   }
 
   auto OutFileOrError = FileOutputBuffer::create(Path, FileSize);
   if (auto EC = OutFileOrError.takeError())
     return std::move(EC);
 
   FileBufferByteStream Buffer(std::move(*OutFileOrError),
                               llvm::support::little);

llvm/lib/DebugInfo/MSF/MSFError.cpp

public:
   const char *name() const noexcept override { return "llvm.msf"; }
   std::string message(int Condition) const override {
     switch (static_cast<msf_error_code>(Condition)) {
     case msf_error_code::unspecified:
       return "An unknown error has occurred.";
     case msf_error_code::insufficient_buffer:
       return "The buffer is not large enough to read the requested number of "
              "bytes.";
-    case msf_error_code::size_overflow:
+    case msf_error_code::size_overflow_4096:
       return "Output data is larger than 4 GiB.";
+    case msf_error_code::size_overflow_8192:
+      return "Output data is larger than 8 GiB.";
+    case msf_error_code::size_overflow_16384:
+      return "Output data is larger than 16 GiB.";
+    case msf_error_code::size_overflow_32768:
+      return "Output data is larger than 32 GiB.";
     case msf_error_code::not_writable:
       return "The specified stream is not writable.";
     case msf_error_code::no_stream:
       return "The specified stream does not exist.";
     case msf_error_code::invalid_format:
       return "The data is in an unexpected format.";
     case msf_error_code::block_in_use:
       return "The block is already in use.";

llvm/lib/DebugInfo/PDB/Native/PDBFile.cpp

uint32_t PDBFile::getStreamByteSize(uint32_t StreamIndex) const {
   return ContainerLayout.StreamSizes[StreamIndex];
 }
 
 ArrayRef<support::ulittle32_t>
 PDBFile::getStreamBlockList(uint32_t StreamIndex) const {
   return ContainerLayout.StreamMap[StreamIndex];
 }
 
-uint32_t PDBFile::getFileSize() const { return Buffer->getLength(); }
+uint64_t PDBFile::getFileSize() const { return Buffer->getLength(); }
 
 Expected<ArrayRef<uint8_t>> PDBFile::getBlockData(uint32_t BlockIndex,
                                                   uint32_t NumBytes) const {
   uint64_t StreamBlockOffset = msf::blockToOffset(BlockIndex, getBlockSize());
 
   ArrayRef<uint8_t> Result;
   if (auto EC = Buffer->readBytes(StreamBlockOffset, NumBytes, Result))
     return std::move(EC);

llvm/tools/llvm-pdbutil/PdbYaml.h

 namespace yaml {
 struct SerializationContext;
 
 struct MSFHeaders {
   msf::SuperBlock SuperBlock;
   uint32_t NumDirectoryBlocks = 0;
   std::vector<uint32_t> DirectoryBlocks;
   uint32_t NumStreams = 0;
-  uint32_t FileSize = 0;
+  uint64_t FileSize = 0;
 };
 
 struct StreamBlockList {
   std::vector<uint32_t> Blocks;
 };
 
 struct NamedStreamMapping {
   StringRef StreamName;