Add support for writing 64-bit symbol tables for archives when offsets become too large for 32-bit

[jakehehrlich]

Summary

This should fix https://bugs.llvm.org//show_bug.cgi?id=34189

This change makes it so that if writing a K_GNU style archive, you need to output a > 32-bit offset it should output in K_GNU64 style instead.

[mcgrathr]

You have a bunch of changes here to support individual members >=4G, but I'm not sure there is any need for that.
The bug is just about the whole archive being >=4G so that the member offsets need to be 64 bits.
If you want to support members >=4G I think that can be a separate change since it's not related to the bug.

[emaste]

include/llvm/Object/Archive.h

51–52 ↗

(On Diff #111438)

The comment still references a 32-bit limit.

[phosek]

lib/Object/ArchiveWriter.cpp

502 ↗

(On Diff #111438)

You should use static_cast here.

[phosek]

lib/Object/ArchiveWriter.cpp

152 ↗

(On Diff #111438)

This function is exactly the same as print32 except for the Val type, couldn't you use a single templated function and avoid the printNBits altogether?

[jakehehrlich]

1. Removed changes to allow for > 4GB members as Roland recommended
2. This had the consequence of nullifying the need for the static cast and the comment change
3. I merged print32 and print64 into a single file but I did not get rid of printNBits. The decision to switch to outputting a 64-bit symbol table is made dynamically so there will need to be an if statement that calls one template instance or the other of any template that might merge the behaviors. I see 3 possible options

1. use printNBits as I am
2. tempalte writeSymbolTable and replace the one call to printNBits in writeArchive with an if statement that decides which print to call. This would also need an if statement to decide which writeSymbolTable to call.
3. template writeSymbolTable and make a template that does what writeArchive does, call it writeArchiveT. writeArchive will the decide which type to instantiate writeArchiveT with while writeArchiveT will do the work that writeArchive does now.

option 1) seems the least invasive to me personally.

[jakehehrlich]

ping

[jakehehrlich]

I agree with Rafael's comment. K_MIPS64 is now K_GNU64 and printNBits now uses Kind to determine the size.

[jakehehrlich]

1. I committed the rename and rebased so that this diff now assumes that K_MIPS64 is now K_GNU64
2. I made all the helpers static

[• rafael]

With r314844 in place it should now be much easier to check if a 64 bit symbol table is needed or not.

[jakehehrlich]

I updated this to use the new changes that rafael made. Much clearner. Thanks Rafael!

[jakehehrlich]

This change adds --print-armap to the test. After https://reviews.llvm.org/D39379 lands this test will work. If the wrong index is used for "main" then the correct file is unlikely to show up making so this test should check that "main" is being looked up in the correct place.

llvm/trunk/lib/Object/ArchiveWriter.cpp

static void printWithSpacePadding(raw_ostream &OS, T Data, unsigned Size) {
   unsigned SizeSoFar = OS.tell() - OldPos;
   assert(SizeSoFar <= Size && "Data doesn't fit in Size");
   OS.indent(Size - SizeSoFar);
 }
 
 static bool isBSDLike(object::Archive::Kind Kind) {
   switch (Kind) {
   case object::Archive::K_GNU:
+  case object::Archive::K_GNU64:
     return false;
   case object::Archive::K_BSD:
   case object::Archive::K_DARWIN:
     return true;
-  case object::Archive::K_GNU64:
   case object::Archive::K_DARWIN64:
   case object::Archive::K_COFF:
     break;
   }
   llvm_unreachable("not supported for writting");
 }
 
-static void print32(raw_ostream &Out, object::Archive::Kind Kind,
-                    uint32_t Val) {
+template <class T>
+static void print(raw_ostream &Out, object::Archive::Kind Kind, T Val) {
   if (isBSDLike(Kind))
     support::endian::Writer<support::little>(Out).write(Val);
   else
     support::endian::Writer<support::big>(Out).write(Val);
 }
 
 static void printRestOfMemberHeader(
     raw_ostream &Out, const sys::TimePoint<std::chrono::seconds> &ModTime,
#ifdef LLVM_ON_WIN32
   // Replace backslashes with slashes so that the path is portable between *nix
   // and Windows.
   std::replace(Relative.begin(), Relative.end(), '\\', '/');
 #endif
 
   return Relative.str();
 }
 
+static bool is64BitKind(object::Archive::Kind Kind) {
+  switch (Kind) {
+  case object::Archive::K_GNU:
+  case object::Archive::K_BSD:
+  case object::Archive::K_DARWIN:
+  case object::Archive::K_COFF:
+    return false;
+  case object::Archive::K_DARWIN64:
+  case object::Archive::K_GNU64:
+    return true;
+  }
+  llvm_unreachable("not supported for writting");
+}
+
 static void addToStringTable(raw_ostream &Out, StringRef ArcName,
                              const NewArchiveMember &M, bool Thin) {
   StringRef ID = M.Buf->getBufferIdentifier();
   if (Thin) {
     if (M.IsNew)
       Out << computeRelativePath(ArcName, ID);
     else
       Out << ID;
static bool isArchiveSymbol(const object::BasicSymbolRef &S) {
   if (!(Symflags & object::SymbolRef::SF_Global))
     return false;
   if (Symflags & object::SymbolRef::SF_Undefined &&
       !(Symflags & object::SymbolRef::SF_Indirect))
     return false;
   return true;
 }
 
+static void printNBits(raw_ostream &Out, object::Archive::Kind Kind,
+                       uint64_t Val) {
+  if (is64BitKind(Kind))
+    print<uint64_t>(Out, Kind, Val);
+  else
+    print<uint32_t>(Out, Kind, Val);
+}
+
 static void writeSymbolTable(raw_ostream &Out, object::Archive::Kind Kind,
                              bool Deterministic, ArrayRef<MemberData> Members,
                              StringRef StringTable) {
   if (StringTable.empty())
     return;
 
   unsigned NumSyms = 0;
   for (const MemberData &M : Members)
     NumSyms += M.Symbols.size();
 
   unsigned Size = 0;
-  Size += 4; // Number of entries
+  Size += is64BitKind(Kind) ? 8 : 4; // Number of entries
   if (isBSDLike(Kind))
     Size += NumSyms * 8; // Table
+  else if (is64BitKind(Kind))
+    Size += NumSyms * 8; // Table
   else
     Size += NumSyms * 4; // Table
   if (isBSDLike(Kind))
     Size += 4; // byte count
   Size += StringTable.size();
   // ld64 expects the members to be 8-byte aligned for 64-bit content and at
   // least 4-byte aligned for 32-bit content.  Opt for the larger encoding
   // uniformly.
   // We do this for all bsd formats because it simplifies aligning members.
   unsigned Alignment = isBSDLike(Kind) ? 8 : 2;
   unsigned Pad = OffsetToAlignment(Size, Alignment);
   Size += Pad;
 
   if (isBSDLike(Kind))
     printBSDMemberHeader(Out, Out.tell(), "__.SYMDEF", now(Deterministic), 0, 0,
                          0, Size);
+  else if (is64BitKind(Kind))
+    printGNUSmallMemberHeader(Out, "/SYM64", now(Deterministic), 0, 0, 0, Size);
   else
     printGNUSmallMemberHeader(Out, "", now(Deterministic), 0, 0, 0, Size);
 
   uint64_t Pos = Out.tell() + Size;
 
   if (isBSDLike(Kind))
-    print32(Out, Kind, NumSyms * 8);
+    print<uint32_t>(Out, Kind, NumSyms * 8);
   else
-    print32(Out, Kind, NumSyms);
+    printNBits(Out, Kind, NumSyms);
 
   for (const MemberData &M : Members) {
     for (unsigned StringOffset : M.Symbols) {
       if (isBSDLike(Kind))
-        print32(Out, Kind, StringOffset);
-      print32(Out, Kind, Pos); // member offset
+        print<uint32_t>(Out, Kind, StringOffset);
+      printNBits(Out, Kind, Pos); // member offset
     }
     Pos += M.Header.size() + M.Data.size() + M.Padding.size();
   }
 
   if (isBSDLike(Kind))
-    print32(Out, Kind, StringTable.size()); // byte count of the string table
+    // byte count of the string table
+    print<uint32_t>(Out, Kind, StringTable.size());
   Out << StringTable;
 
   while (Pad--)
     Out.write(uint8_t(0));
 }
 
 static Expected<std::vector<unsigned>>
 getSymbols(MemoryBufferRef Buf, raw_ostream &SymNames, bool &HasObject) {
Expected<std::vector<MemberData>> DataOrErr =
       computeMemberData(StringTable, SymNames, Kind, Thin, ArcName, NewMembers);
   if (Error E = DataOrErr.takeError())
     return E;
   std::vector<MemberData> &Data = *DataOrErr;
 
   if (!StringTableBuf.empty())
     Data.insert(Data.begin(), computeStringTable(StringTableBuf));
 
+  // We would like to detect if we need to switch to a 64-bit symbol table.
+  if (WriteSymtab) {
+    uint64_t MaxOffset = 0;
+    uint64_t LastOffset = MaxOffset;
+    for (const auto& M : Data) {
+      // Record the start of the member's offset
+      LastOffset = MaxOffset;
+      // Account for the size of each part associated with the member.
+      MaxOffset += M.Header.size() + M.Data.size() + M.Padding.size();
+      // We assume 32-bit symbols to see if 32-bit symbols are possible or not.
+      MaxOffset += M.Symbols.size() * 4;
+    }
+    // If LastOffset isn't going to fit in a 32-bit varible we need to switch
+    // to 64-bit. Note that the file can be larger than 4GB as long as the last
+    // member starts before the 4GB offset.
+    if (LastOffset >> 32 != 0)
+      Kind = object::Archive::K_GNU64;
+  }
+
   SmallString<128> TmpArchive;
   int TmpArchiveFD;
   if (auto EC = sys::fs::createUniqueFile(ArcName + ".temp-archive-%%%%%%%.a",
                                           TmpArchiveFD, TmpArchive))
     return errorCodeToError(EC);
 
   ToolOutputFile Output(TmpArchive, TmpArchiveFD);
   raw_fd_ostream &Out = Output.os();

llvm/trunk/test/Object/archive-SYM64-write.test

+# RUN: yaml2obj %s > %t
+# RUN: rm -f %t.lib
+# RUN: cp %t %t2
+# RUN: llvm-ar cr %t.lib %t %t2 %p/Inputs/trivial-object-test.elf-x86-64
+# RUN: llvm-nm --print-armap %t.lib | FileCheck %s
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+  Machine:         EM_X86_64
+Sections:
+  - Name:            .data
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC ]
+    AddressAlign:    0x0000000000000001
+    Content:         "00"
+    Size:            2147483648
+
+# CHECK:      Archive map
+# CHECK-NEXT: main in trivial-object-test.elf-x86-64
+
+# CHECK:    archive-SYM64-write.test.tmp:
+
+# CHECK:    archive-SYM64-write.test.tmp2:
+
+# CHECK:    trivial-object-test.elf-x86-64:
+# CHECK-NEXT:                     U SomeOtherFunction
+# CHECK-NEXT:    0000000000000000 T main
+# CHECK-NEXT:                     U puts