[Object] Support large archive members

[jhenderson]

Summary

The archive library truncated the size of archive members whose size was greater than max uint32_t, leading to errors in tools that use the library. This patch fixes the issue and adds some unit tests to verify.

[MaskRay]

Command Output (stderr):
--

bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Success
	Input file = /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/lld/test/ELF/Inputs/large-files1.o.bz2, output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

As you see, many systems may not be able to run such tests. Deleting the test may be fine.

llvm/lib/Object/Archive.cpp

396

Does return Header.getSize() work?

[grimar]

I am also not fond of having tests that might consume gigabytes to run.
Perhaps you could craft an object that has a broken size written to a header, e.g. (MAX_UINT32 + 1) ant test
that LLD does not crash and report an error with that would contain the size value found?

[jhenderson]

In D75742#1911048, @grimar wrote:

Perhaps you could craft an object that has a broken size written to a header, e.g. (MAX_UINT32 + 1) ant test
that LLD does not crash and report an error with that would contain the size value found?

I'm not sure I understand what you mean here. The bug fixed by this patch manifests as attempting to read a section header table past the end of the file, because the result of the truncated size is failing to create a buffer that is large enough. This fix can therefore only be verified by attempting to use data in the latter part of a buffer that is > 4GB in size. The only alternative I can think of to having large files as inputs, is to create sparse files, but I'm not sure if the test would be portable consequently.

[jhenderson]

Simplify size handling slightly.

I've not deleted the test yet, but I will do so in due course before this gets committed. I'd like to see if we can come up with an alternative test option first, if we can. For the record, the test passes locally with my patch, but didn't without the fix (it was how I found the bug in the first place).

[jhenderson]

I realised that I could at least test Archive::Child::getSize() function using gtest unittests. I'll take a look at that, and see if getBuffer is possible too.

[jhenderson]

Test with unit tests.

[ruiu]

LGTM

[grimar]

LGTM. Using unit test is good I think.

In D75742#1912125, @jhenderson wrote:

In D75742#1911048, @grimar wrote:

Perhaps you could craft an object that has a broken size written to a header, e.g. (MAX_UINT32 + 1) ant test
that LLD does not crash and report an error with that would contain the size value found?

I'm not sure I understand what you mean here. The bug fixed by this patch manifests as attempting to read a section header table past the end of the file, because the result of the truncated size is failing to create a buffer that is large enough. This fix can therefore only be verified by attempting to use data in the latter part of a buffer that is > 4GB in size. The only alternative I can think of to having large files as inputs, is to create sparse files, but I'm not sure if the test would be portable consequently.

Idea was based on a testing of a values reported in a error message.
See the code this patch changes:

Expected<StringRef> Archive::Child::getBuffer() const {
  Expected<bool> isThinOrErr = isThinMember();
  if (!isThinOrErr)
    return isThinOrErr.takeError();
  bool isThin = isThinOrErr.get();
  if (!isThin) {
    Expected<uint64_t> Size = getSize();
    if (!Size)
      return Size.takeError();
    return StringRef(Data.data() + StartOfFile, Size.get());
  }

The problem is with the original logic after Expected<uint64_t> Size = getSize();.
Imagine we have a broken object and getSize() returns (uint64)-1.
We do not report it and return a StringRef that has overruns the file size.
But we could report an error (i.e. add a new one) instead.

This error could contain an information about a size of a broken child.
It would be printed differently in Expected<uint64_t> Size and Expected<uint32_t> Size
cases I think. That is what something I was thinking about.

llvm/unittests/Object/ArchiveTest.cpp

25 ↗

(On Diff #249089)

static char const -> static const char?

66 ↗

(On Diff #249089)

Not strong suggestion:

MemoryBufferRef Source(ArchiveWithMember, "regular");
Expected<std::unique_ptr<Archive>> A = Archive::create(Source);
ASSERT_THAT_EXPECTED(A, Succeeded());
Error ChildErr = Error::success();
auto Child = (*A)->child_begin(ChildErr);
ASSERT_THAT_ERROR(std::move(ChildErr), Succeeded());

This part is almost similar to one in ArchiveChildGetSizeThinArchive and ArchiveChildGetSizeRegularArchive.
Should this code be moved to a helper that returns a Child?

[jhenderson]

llvm/unittests/Object/ArchiveTest.cpp

66 ↗

(On Diff #249089)

I considered that. Unfortunately, ASSERT_* macros return void, so can't be inside other functions that return something else. Consequently, we have to put them in yet another function, then check to see if the thing is valid anyway (rendering the ASSERT pointless), and the logic becomes somewhat complex and hard to follow. I don't think there's much benefit as a result.

[rupprecht]

In D75742#1912140, @jhenderson wrote:

Simplify size handling slightly.

I've not deleted the test yet, but I will do so in due course before this gets committed. I'd like to see if we can come up with an alternative test option first, if we can. For the record, the test passes locally with my patch, but didn't without the fix (it was how I found the bug in the first place).

I don't know if it's portable (you'd have to add requires for Linux/shell, or maybe do some lit configuration to detect if it's available), but you could try using fallocate:

$ time fallocate -l 3G test.large
fallocate -l 3G test.large  0.00s user 0.01s system 39% cpu 0.017 total
$ time cat test.large > /dev/null
cat test.large > /dev/null  0.01s user 1.18s system 99% cpu 1.193 total

If you want to try that, I would still recommend keeping the unit test that runs on all platforms, and just having this lit one be an extra test.

llvm/unittests/Object/ArchiveTest.cpp

66 ↗

(On Diff #249089)

One way you might be able to refactor this is to not have the assert in the helper method:

Expected<Child> createChild(StringRef src, StringRef name) {
  MemoryBufferRef Source(src, name);
  Expected<std::unique_ptr<Archive>> A = Archive::create(Source);
  if (!A) return A.takeError();

  Error ChildErr = Error::success();
  auto Child = (*A)->child_begin(ChildErr);
  if (ChildErr) return std::move(ChildErr);
  return std::move(Child);
}

TEST(ArchiveTests, ArchiveChildGetBuffer) {
  auto Child = createChild(ArchiveWithMember, "regular");
  ASSERT_THAT_EXPECTED(Child, Succeeded());
...

(heavy pseudocode, I almost definitely got a lot of types wrong... and I'm also ignoring the lifetime issues)

71 ↗

(On Diff #249089)

ASSERT_TRUE((bool)Buffer);

[jhenderson]

In D75742#1914799, @rupprecht wrote:

In D75742#1912140, @jhenderson wrote:

Simplify size handling slightly.

I've not deleted the test yet, but I will do so in due course before this gets committed. I'd like to see if we can come up with an alternative test option first, if we can. For the record, the test passes locally with my patch, but didn't without the fix (it was how I found the bug in the first place).

I don't know if it's portable (you'd have to add requires for Linux/shell, or maybe do some lit configuration to detect if it's available), but you could try using fallocate:

$ time fallocate -l 3G test.large
fallocate -l 3G test.large  0.00s user 0.01s system 39% cpu 0.017 total
$ time cat test.large > /dev/null
cat test.large > /dev/null  0.01s user 1.18s system 99% cpu 1.193 total

If you want to try that, I would still recommend keeping the unit test that runs on all platforms, and just having this lit one be an extra test.

I'm not sure that's going to work. We'll still need an object of the right size in the first place, to punch holes in, as otherwise it won't be a valid linker input, so it doesn't avoid the size issue. Furthermore, I doubt that llvm-ar or LLD will preserve the sparseness, so it won't be useful for testing this or LLD's behaviour.

[jhenderson]

Address review comments (char const -> const char, pull code out into helper).

[MaskRay]

In D75742#1914799, @rupprecht wrote:

In D75742#1912140, @jhenderson wrote:

Simplify size handling slightly.

I've not deleted the test yet, but I will do so in due course before this gets committed. I'd like to see if we can come up with an alternative test option first, if we can. For the record, the test passes locally with my patch, but didn't without the fix (it was how I found the bug in the first place).

I don't know if it's portable (you'd have to add requires for Linux/shell, or maybe do some lit configuration to detect if it's available), but you could try using fallocate:

$ time fallocate -l 3G test.large
fallocate -l 3G test.large  0.00s user 0.01s system 39% cpu 0.017 total
$ time cat test.large > /dev/null
cat test.large > /dev/null  0.01s user 1.18s system 99% cpu 1.193 total

If you want to try that, I would still recommend keeping the unit test that runs on all platforms, and just having this lit one be an extra test.

Nice tip! Sparse files is even supported on ext2. So running it on Linux will be safe. However, if the disk cannot allocate 3G space (if the space is less 3G, then it surely cannot), fallocate(2) can fail with ENOSPC 28 No space left on device.

Additionally, if a test can take more than one second, I will still be hesitant..

ld.lld -m elf_x86_64 -b binary test.large -o out can take quite a few seconds. 2.5s in tmpfs with warn cache on my pc. It could be much slower with a different configuration.

[grimar]

FTR.

The current size of LLVM build folder for me is 26.1 Gb.
I think it is reasonable to avoid letting it go to 30.1 Gb (e.g. +4 Gb, +15%).

We have huge-temporarily-file.s test in LLD with the following comment:

1. This inputs previously created a 4gb temporarily file under 32 bit
2. configuration. Issue was fixed. There is no clean way to check that from here.
3. This testcase added for documentation purposes.

I believe that +15% of peak free space consumption is probably too much. It might be better to
have no test at all sometimes.

[jhenderson]

In D75742#1916287, @grimar wrote:

FTR.

The current size of LLVM build folder for me is 26.1 Gb.
I think it is reasonable to avoid letting it go to 30.1 Gb (e.g. +4 Gb, +15%).

We have huge-temporarily-file.s test in LLD with the following comment:

1. This inputs previously created a 4gb temporarily file under 32 bit
2. configuration. Issue was fixed. There is no clean way to check that from here.
3. This testcase added for documentation purposes.

I believe that +15% of peak free space consumption is probably too much. It might be better to
have no test at all sometimes.

We have or had that file? I couldn't find it in git as things stand.

[grimar]

In D75742#1916519, @jhenderson wrote:

In D75742#1916287, @grimar wrote:

FTR.

The current size of LLVM build folder for me is 26.1 Gb.
I think it is reasonable to avoid letting it go to 30.1 Gb (e.g. +4 Gb, +15%).

We have huge-temporarily-file.s test in LLD with the following comment:

1. This inputs previously created a 4gb temporarily file under 32 bit
2. configuration. Issue was fixed. There is no clean way to check that from here.
3. This testcase added for documentation purposes.

I believe that +15% of peak free space consumption is probably too much. It might be better to
have no test at all sometimes.

We have or had that file? I couldn't find it in git as things stand.

I believe we have it:
https://github.com/llvm/llvm-project/blob/master/lld/test/ELF/linkerscript/huge-temporary-file.s

[jhenderson]

In D75742#1916549, @grimar wrote:

In D75742#1916519, @jhenderson wrote:

In D75742#1916287, @grimar wrote:

FTR.

The current size of LLVM build folder for me is 26.1 Gb.
I think it is reasonable to avoid letting it go to 30.1 Gb (e.g. +4 Gb, +15%).

We have huge-temporarily-file.s test in LLD with the following comment:

1. This inputs previously created a 4gb temporarily file under 32 bit
2. configuration. Issue was fixed. There is no clean way to check that from here.
3. This testcase added for documentation purposes.

I believe that +15% of peak free space consumption is probably too much. It might be better to
have no test at all sometimes.

We have or had that file? I couldn't find it in git as things stand.

I believe we have it:
https://github.com/llvm/llvm-project/blob/master/lld/test/ELF/linkerscript/huge-temporary-file.s

Thanks. I didn't look in the linkerscript directory.

lld/test/ELF/large-files.s

+# REQUIRES: x86
+
+## Show that LLD can create large output files, generated from multiple smaller
+## input files.
+
+# RUN: rm -rf %t.dir
+# RUN: mkdir -p %t.dir
+# RUN: cd %t.dir
+
+## These two inputs were created from the following assembly:
+##
+## .align 0x1000
+## .global SYMBOL1
+## SYMBOL1:
+## .space 0x80000000
+## SYMBOL2:
+## 
+## .data
+## .align 0x4000
+## SYMBOL3:
+## .byte 1
+## SYMBOL4:
+##
+## with the following instructions run on it:
+##
+## sed -e s/SYMBOL1/_start/g %s > t
+## sed -e s/SYMBOL2/end1/g t > t2
+## sed -e s/SYMBOL3/data1/g t2 > t3
+## sed -e s/SYMBOL4/enddata1/g t3 > t1.s
+## sed -e s/SYMBOL1/sym2/g %s > t
+## sed -e s/SYMBOL2/end2/g t > t2
+## sed -e s/SYMBOL3/data2/g t2 > t3
+## sed -e s/SYMBOL4/enddata2/g t3 > t2.s
+## llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu t1.s -o t1.o
+## llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu t2.s -o t2.o
+## bzip2 -c t1.o > %p/Inputs/large-files1.o.bz2
+## bzip2 -c t2.o > %p/Inputs/large-files2.o.bz2
+
+# RUN: bzip2 -kdc %p/Inputs/large-files1.o.bz2 > t1.o
+# RUN: bzip2 -kdc %p/Inputs/large-files2.o.bz2 > t2.o
+
+## Test a large output (>4GB) file can be created.
+# RUN: ld.lld t1.o t2.o -o t.elf
+# RUN: llvm-nm t.elf | FileCheck %s --check-prefixes=CHECK,DATA1
+
+## Clean up immediately so that we are not using more disk space than we need at
+## any one time.
+# RUN: rm t.elf
+
+## Test a large input can be used with a section size >4GB and a section at
+## offset >4GB.
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o t3.o
+# RUN: ld.lld t1.o t2.o -o t.ro -r
+# RUN: ld.lld t.ro t3.o -o t.elf
+# RUN: llvm-nm t.elf | FileCheck %s --check-prefixes=CHECK,MORE
+# RUN: rm t.elf
+# RUN: rm t1.o
+# RUN: rm t2.o
+
+## Show that the link works when the large files are in an archive, for both
+## thin and regular archives.
+# RUN: llvm-ar rcsT t.a t.ro t3.o
+# RUN: ld.lld t.a --undefined _start --undefined sym2 --undefined sym3 -m elf_x86_64 -o t.elf
+# RUN: llvm-nm t.elf | FileCheck %s --check-prefixes=CHECK,MORE
+
+# RUN: rm t.elf
+# RUN: rm t.a
+
+# RUN: llvm-ar rcs t.a t.ro t3.o
+# RUN: rm t.ro
+# RUN: ld.lld t.a --undefined _start --undefined sym2 --undefined sym3 -m elf_x86_64 -o t.elf
+# RUN: llvm-nm t.elf | FileCheck %s --check-prefixes=CHECK,MORE
+
+## Clean up so that we do not have several GB of data leftover after this test
+## finishes.
+# RUN: cd ..
+# RUN: rm -rf %t.dir
+
+# CHECK:      0000000000202000 {{.*}} _start
+# CHECK-NEXT: 0000000100204000 {{.*}} data1
+# CHECK-NEXT: 0000000100208000 {{.*}} data2
+# MORE-NEXT:  000000010020c000 {{.*}} data3
+# CHECK-NEXT: 0000000080202000 {{.*}} end1
+# CHECK-NEXT: 0000000100202000 {{.*}} end2
+# MORE-NEXT:  0000000100202001 {{.*}} end3
+# CHECK-NEXT: 0000000100204001 {{.*}} enddata1
+# CHECK-NEXT: 0000000100208001 {{.*}} enddata2
+# MORE-NEXT:  000000010020c001 {{.*}} enddata3
+# CHECK-NEXT: 0000000080202000 {{.*}} sym2
+# MORE-NEXT:  0000000100202000 {{.*}} sym3
+
+.align 0x1000
+.global sym3
+sym3:
+    nop
+end3:
+
+.data
+.align 0x4000
+data3:
+    .byte 0
+enddata3:

llvm/lib/Object/Archive.cpp

if (Name.substr(3).rtrim(' ').getAsInteger(10, NameSize)) {
       return;
     }
     StartOfFile += NameSize;
   }
 }
 
 Expected<uint64_t> Archive::Child::getSize() const {
   if (Parent->IsThin) {
-    Expected<uint32_t> Size = Header.getSize();
+    Expected<uint64_t> Size = Header.getSize();
     if (!Size)
       return Size.takeError();
     return Size.get();
   }
   return Data.size() - StartOfFile;
 }
 
 Expected<uint64_t> Archive::Child::getRawSize() const {

 }
 
 Expected<StringRef> Archive::Child::getBuffer() const {
   Expected<bool> isThinOrErr = isThinMember();
   if (!isThinOrErr)
     return isThinOrErr.takeError();
   bool isThin = isThinOrErr.get();
   if (!isThin) {
-    Expected<uint32_t> Size = getSize();
+    Expected<uint64_t> Size = getSize();
     if (!Size)
       return Size.takeError();
     return StringRef(Data.data() + StartOfFile, Size.get());
   }
   Expected<std::string> FullNameOrErr = getFullName();
   if (!FullNameOrErr)
     return FullNameOrErr.takeError();
   const std::string &FullName = *FullNameOrErr;