[AIX][XCOFF] Add support for 64-bit file header and section header writing

[MaryamBen]

Summary

Support 64-bit file header and section header writing.
This patch is the first partition of D103696

[jhenderson]

As a non-XCOFF developer, this mostly looks fine to me, but needs a test case.

[Helflym]

Ok for the XCOFF part

[sfertile]

In D104639#2832518, @jhenderson wrote:

As a non-XCOFF developer, this mostly looks fine to me, but needs a test case.

This is important, we can't approve and land a patch without testing, and this patch is currently in an untestable state as we will hit a fatal error before hitting the code changes in writeFileHeader/writeSectionHeaderTable. On the converse side, there is nothing to stop object file writing after emitting the file header and section header tables.

I've left some inline comments on this patch showing where we will have to make adjustments for your changes to be testable. There is one other modification we need to make in PPCXCOFFObjectWriter::getRelocTypeAndSignSize which gets called as we finalize the layout. We need to add support for mapping the FK_Data_8 fixup type with a VK_None modifier (all the other modifiers should be left supported for now). Note that since the last field is the signdness and (1 - the length) and we are relocating 8 byte data now it will differ slightly from the existing relocations. Something like

case FK_Data_8:
  switch (Modifier) {
  default:
    report_fatal_error("Unsupported modifier");
  case MCSymbolRefExpr::VK_None:
    return {XCOFF::RelocationType::R_POS, EncodedSignednessIndicator | 63};
  }

should be fine for this patch, and we can handle the other modifiers in the follow up patches.

llvm/include/llvm/BinaryFormat/XCOFF.h

30 ↗

(On Diff #353352)

I'm not sure why these changes are showing up in this patch as they are all landed upstream already (they landed in the patch adding support for yaml2obj support for xcoff)

llvm/lib/MC/XCOFFObjectWriter.cpp

368

This fatal error needs to be removed to be able to reach the point where we emit the file-header and section header table.

380

The implementation of this function will need to be modified to be correct for 64-bit object generation since the name is always stored in the string table (or in a debug section) and not in the symbol table entry itself.

586

This fatal error will also need to be removed to be able to observe your changes to writeFileHeader and writeSectionheaderTable

593

We should insert an early return after this line, with a comment saying this is the temporary end of the 64-bit implementation.

[MaryamBen]

Thanks you for your reviews,

I tried to make changes you told me to see if I could make it in time. Because tomorow is my last day at work, I don't have enough time to redo all the patchs especially the tests.
When I applied your recommendations, I got the error "The end of the file was unexpectedly encountered". I think it would require more time to do things properly.

Otherwise, there are tests for the file header and the section header in further patches.

[sfertile]

In D104639#2954342, @MaryamBen wrote:

I tried to make changes you told me to see if I could make it in time. Because tomorow is my last day at work, I don't have enough time to redo all the patchs especially the tests.

Sorry I couldn't get these reviewed and landed in time before your internship finished. Thank you for all the work you did, It is a pretty impressive the amount you got accomplished.

In D104639#2954342, @MaryamBen wrote:

When I applied your recommendations, I got the error "The end of the file was unexpectedly encountered". I think it would require more time to do things properly.

Your right, its because we are writing values into the file-header or section header table that implies there is following info (like a symbol table offset and non-zero count). I've tested setting those fields to 0 (as well as offsets to relocations and section data) and then we can dump the file headers and section header table. For existing lit tests that dump symbol tables or section contents I would expect llvm-readobj to give an error as those parts are missing from the object file.

In D104639#2954342, @MaryamBen wrote:

Otherwise, there are tests for the file header and the section header in further patches.

Each patch that adds/alters functionality has to be accompanied by testing of said functionality. We can add further testing in subsequent patches but only patches that contain no function changes are allowed to be committed without new testing (or updated testing).

llvm/lib/MC/XCOFFObjectWriter.cpp

58

I don't see any need to update this structure until we start emitting relocations.

80–81

Isn't the 64-bit object format also limited to a 4 byte symbol table index?

[MaryamBen]

When I did this work, I had no idea that I could remove "64-bit XCOFF object files are not supported yet." in order to create tests.
Unfortunately, I learned this late,
Thank you so much for your clarifications.

llvm/include/llvm/BinaryFormat/XCOFF.h

30 ↗

(On Diff #353352)

I think the patch you are talking about was recently merged.
When I created the pacth (2 months ago) , these changes weren't applied.

llvm/lib/MC/XCOFFObjectWriter.cpp

80–81

Yes I made a mistake, I will change it right now.

llvm/lib/MC/XCOFFObjectWriter.cpp

 constexpr unsigned DefaultSectionAlign = 4;
 constexpr int16_t MaxSectionIndex = INT16_MAX;
 
 // Packs the csect's alignment and type into a byte.
 uint8_t getEncodedType(const MCSectionXCOFF *);
 
 struct XCOFFRelocation {
   uint32_t SymbolTableIndex;
-  uint32_t FixupOffsetInCsect;
+  uint64_t FixupOffsetInCsect;
   uint8_t SignAndSize;
   uint8_t Type;
 };
 
 // Wrapper around an MCSymbolXCOFF.
 struct Symbol {
   const MCSymbolXCOFF *const MCSym;
   uint32_t SymbolTableIndex;
 
   XCOFF::StorageClass getStorageClass() const {
     return MCSym->getStorageClass();
   }
   StringRef getSymbolTableName() const { return MCSym->getSymbolTableName(); }
   Symbol(const MCSymbolXCOFF *MCSym) : MCSym(MCSym), SymbolTableIndex(-1) {}
 };
 
 // Wrapper for an MCSectionXCOFF.
 // It can be a Csect or debug section or DWARF section and so on.
 struct XCOFFSection {
   const MCSectionXCOFF *const MCSec;
   uint32_t SymbolTableIndex;
-  uint32_t Address;
-  uint32_t Size;
+  uint64_t Address;
+  uint64_t Size;
 
   SmallVector<Symbol, 1> Syms;
   SmallVector<XCOFFRelocation, 1> Relocations;
   StringRef getSymbolTableName() const { return MCSec->getSymbolTableName(); }
   XCOFFSection(const MCSectionXCOFF *MCSec)
       : MCSec(MCSec), SymbolTableIndex(-1), Address(-1), Size(0) {}
 };
 
 // Type to be used for a container representing a set of csects with
 // (approximately) the same storage mapping class. For example all the csects
 // with a storage mapping class of `xmc_pr` will get placed into the same
 // container.
 using CsectGroup = std::deque<XCOFFSection>;
 using CsectGroups = std::deque<CsectGroup *>;
 
 // The basic section entry defination. This Section represents a section entry
 // in XCOFF section header table.
 struct SectionEntry {
   char Name[XCOFF::NameSize];
   // The physical/virtual address of the section. For an object file
   // these values are equivalent.
-  uint32_t Address;
-  uint32_t Size;
-  uint32_t FileOffsetToData;
-  uint32_t FileOffsetToRelocations;
+  uint64_t Address;
+  uint64_t Size;
+  uint64_t FileOffsetToData;
+  uint64_t FileOffsetToRelocations;
   uint32_t RelocationCount;
   int32_t Flags;
 
   int16_t Index;
 
   // XCOFF has special section numbers for symbols:
   // -2 Specifies N_DEBUG, a special symbolic debugging symbol.
   // -1 Specifies N_ABS, an absolute symbol. The symbol has a value but is not
struct CsectSectionEntry : public SectionEntry {
   }
 
   virtual ~CsectSectionEntry() {}
 };
 
 class XCOFFObjectWriter : public MCObjectWriter {
 
   uint32_t SymbolTableEntryCount = 0;
-  uint32_t SymbolTableOffset = 0;
+  uint64_t SymbolTableOffset = 0;
   uint16_t SectionCount = 0;
   uint32_t RelocationEntryOffset = 0;
 
   support::endian::Writer W;
   std::unique_ptr<MCXCOFFObjectTargetWriter> TargetObjectWriter;
   StringTableBuilder Strings;
 
   // Maps the MCSection representation to its corresponding XCOFFSection

 static MCSectionXCOFF *getContainingCsect(const MCSymbolXCOFF *XSym) {
   if (XSym->isDefined())
     return cast<MCSectionXCOFF>(XSym->getFragment()->getParent());
   return XSym->getRepresentedCsect();
 }
 
 void XCOFFObjectWriter::executePostLayoutBinding(MCAssembler &Asm,
                                                  const MCAsmLayout &Layout) {
   if (TargetObjectWriter->is64Bit())
     report_fatal_error("64-bit XCOFF object files are not supported yet.");
 
   for (const auto &S : Asm) {
     const auto *MCSec = cast<const MCSectionXCOFF>(&S);
     assert(SectionMap.find(MCSec) == SectionMap.end() &&
            "Cannot add a section twice.");
     assert(XCOFF::XTY_ER != MCSec->getCSectType() &&
            "An undefined csect should not get registered.");
 
     // If the name does not fit in the storage provided in the symbol table
     // entry, add it to the string table.
     if (nameShouldBeInStringTable(MCSec->getSymbolTableName()))
       Strings.add(MCSec->getSymbolTableName());
 
     CsectGroup &Group = getCsectGroup(MCSec);
     Group.emplace_back(MCSec);
     SectionMap[MCSec] = &Group.back();
   }
 
   for (const MCSymbol &S : Asm.symbols()) {
else if (Type == XCOFF::RelocationType::R_TOC ||
 
     FixedValue = TOCEntryOffset;
   }
 
   assert(
       (TargetObjectWriter->is64Bit() ||
        Fixup.getOffset() <= UINT32_MAX - Layout.getFragmentOffset(Fragment)) &&
       "Fragment offset + fixup offset is overflowed in 32-bit mode.");
-  uint32_t FixupOffsetInCsect =
+  uint64_t FixupOffsetInCsect =
       Layout.getFragmentOffset(Fragment) + Fixup.getOffset();
 
   XCOFFRelocation Reloc = {Index, FixupOffsetInCsect, SignAndSize, Type};
   MCSectionXCOFF *RelocationSec = cast<MCSectionXCOFF>(Fragment->getParent());
   assert(SectionMap.find(RelocationSec) != SectionMap.end() &&
          "Expected containing csect to exist in map.");
   SectionMap[RelocationSec]->Relocations.push_back(Reloc);
 
void XCOFFObjectWriter::recordRelocation(MCAssembler &Asm,
   SectionMap[RelocationSec]->Relocations.push_back(RelocB);
   // We already folded "SymbolA + imm64" above when Type is R_POS for SymbolA,
   // now we just need to fold "- SymbolB" here.
   FixedValue -= getVirtualAddress(SymB, SymBSec);
 }
 
 void XCOFFObjectWriter::writeSections(const MCAssembler &Asm,
                                       const MCAsmLayout &Layout) {
-  uint32_t CurrentAddressLocation = 0;
+  uint64_t CurrentAddressLocation = 0;
   for (const auto *Section : Sections) {
     // Nothing to write for this Section.
     if (Section->Index == SectionEntry::UninitializedIndex ||
         Section->IsVirtual)
       continue;
 
     // There could be a gap (without corresponding zero padding) between
     // sections.

 uint64_t XCOFFObjectWriter::writeObject(MCAssembler &Asm,
                                         const MCAsmLayout &Layout) {
   // We always emit a timestamp of 0 for reproducibility, so ensure incremental
   // linking is not enabled, in case, like with Windows COFF, such a timestamp
   // is incompatible with incremental linking of XCOFF.
   if (Asm.isIncrementalLinkerCompatible())
     report_fatal_error("Incremental linking not supported for XCOFF.");
 
   if (TargetObjectWriter->is64Bit())
     report_fatal_error("64-bit XCOFF object files are not supported yet.");
 
   finalizeSectionInfo();
   uint64_t StartOffset = W.OS.tell();
 
   writeFileHeader();
   writeSectionHeaderTable();
   writeSections(Asm, Layout);
   writeRelocations();
 
   writeSymbolTable(Layout);
   // Write the string table.
   Strings.write(W.OS);
 
   return W.OS.tell() - StartOffset;
void XCOFFObjectWriter::writeSymbolTableEntryForControlSection(
   // Reserved (x_stab).
   W.write<uint32_t>(0);
   // Reserved (x_snstab).
   W.write<uint16_t>(0);
 }
 
 void XCOFFObjectWriter::writeFileHeader() {
   // Magic.
+  if (TargetObjectWriter->is64Bit())
+    W.write<uint16_t>(0x01f7);
+  else
     W.write<uint16_t>(0x01df);
   // Number of sections.
   W.write<uint16_t>(SectionCount);
   // Timestamp field. For reproducible output we write a 0, which represents no
   // timestamp.
   W.write<int32_t>(0);
   // Byte Offset to the start of the symbol table.
+  if (TargetObjectWriter->is64Bit())
+    W.write<uint64_t>(SymbolTableOffset);
+  else {
     W.write<uint32_t>(SymbolTableOffset);
     // Number of entries in the symbol table.
     W.write<int32_t>(SymbolTableEntryCount);
+  }
   // Size of the optional header.
   W.write<uint16_t>(0);
   // Flags.
   W.write<uint16_t>(0);
+  if (TargetObjectWriter->is64Bit())
+    // Number of entries in the symbol table.
+    W.write<int32_t>(SymbolTableEntryCount);
 }
 
 void XCOFFObjectWriter::writeSectionHeaderTable() {
   for (const auto *Sec : Sections) {
     // Nothing to write for this Section.
     if (Sec->Index == SectionEntry::UninitializedIndex)
       continue;
 
     // Write Name.
     ArrayRef<char> NameRef(Sec->Name, XCOFF::NameSize);
     W.write(NameRef);
 
+    if (TargetObjectWriter->is64Bit()) {
+      // Write the Physical Address and Virtual Address. In an object file these
+      // are the same.
+      W.write<uint64_t>(Sec->Address);
+      W.write<uint64_t>(Sec->Address);
+
+      W.write<uint64_t>(Sec->Size);
+      W.write<uint64_t>(Sec->FileOffsetToData);
+      W.write<uint64_t>(Sec->FileOffsetToRelocations);
+
+      // Line number pointer. Not supported yet.
+      W.write<uint64_t>(0);
+
+      W.write<uint32_t>(Sec->RelocationCount);
+
+      // Line number counts. Not supported yet.
+      W.write<uint32_t>(0);
+
+      W.write<int32_t>(Sec->Flags);
+      W.write<int32_t>(0);
+    } else {
       // Write the Physical Address and Virtual Address. In an object file these
       // are the same.
       W.write<uint32_t>(Sec->Address);
       W.write<uint32_t>(Sec->Address);
 
       W.write<uint32_t>(Sec->Size);
       W.write<uint32_t>(Sec->FileOffsetToData);
       W.write<uint32_t>(Sec->FileOffsetToRelocations);
 
       // Line number pointer. Not supported yet.
       W.write<uint32_t>(0);
 
       W.write<uint16_t>(Sec->RelocationCount);
 
       // Line number counts. Not supported yet.
       W.write<uint16_t>(0);
 
       W.write<int32_t>(Sec->Flags);
     }
   }
+}
 
 void XCOFFObjectWriter::writeRelocation(XCOFFRelocation Reloc,
                                         const XCOFFSection &CSection) {
   W.write<uint32_t>(CSection.Address + Reloc.FixupOffsetInCsect);
   W.write<uint32_t>(Reloc.SymbolTableIndex);
   W.write<uint8_t>(Reloc.SignAndSize);
   W.write<uint8_t>(Reloc.Type);
 }
void XCOFFObjectWriter::finalizeSectionInfo() {
 
   // Calculate the file offset to the relocation entries.
   uint64_t RawPointer = RelocationEntryOffset;
   for (auto Sec : Sections) {
     if (Sec->Index == SectionEntry::UninitializedIndex || !Sec->RelocationCount)
       continue;
 
     Sec->FileOffsetToRelocations = RawPointer;
-    const uint32_t RelocationSizeInSec =
+    uint64_t RelocationSizeInSec;
+    if (TargetObjectWriter->is64Bit())
+      RelocationSizeInSec =
+          Sec->RelocationCount * XCOFF::RelocationSerializationSize64;
+    else
+      RelocationSizeInSec =
           Sec->RelocationCount * XCOFF::RelocationSerializationSize32;
+
     RawPointer += RelocationSizeInSec;
-    if (RawPointer > UINT32_MAX)
+    if (RawPointer > UINT32_MAX && !TargetObjectWriter->is64Bit())
       report_fatal_error("Relocation data overflowed this object file.");
   }
 
   // TODO Error check that the number of symbol table entries fits in 32-bits
   // signed ...
   if (SymbolTableEntryCount)
     SymbolTableOffset = RawPointer;
 }
for (auto &Csect : UndefinedCsects) {
     SymbolIndexMap[Csect.MCSec->getQualNameSymbol()] = Csect.SymbolTableIndex;
     // 1 main and 1 auxiliary symbol table entry for each contained symbol.
     SymbolTableIndex += 2;
   }
 
   // The address corrresponds to the address of sections and symbols in the
   // object file. We place the shared address 0 immediately after the
   // section header table.
-  uint32_t Address = 0;
+  uint64_t Address = 0;
   // Section indices are 1-based in XCOFF.
   int32_t SectionIndex = 1;
   bool HasTDataSection = false;
 
   for (auto *Section : Sections) {
     const bool IsEmpty =
         llvm::all_of(Section->Groups,
                      [](const CsectGroup *Group) { return Group->empty(); });
for (auto *Section : Sections) {
     // DefaultSectionAlign.
     Address = alignTo(Address, DefaultSectionAlign);
     Section->Size = Address - Section->Address;
   }
 
   SymbolTableEntryCount = SymbolTableIndex;
 
   // Calculate the RawPointer value for each section.
-  uint64_t RawPointer = XCOFF::FileHeaderSize32 + auxiliaryHeaderSize() +
-                        SectionCount * XCOFF::SectionHeaderSize32;
+  uint64_t RawPointer;
+  if (TargetObjectWriter->is64Bit())
+    RawPointer = XCOFF::FileHeaderSize64 + auxiliaryHeaderSize() +
+                 SectionCount * XCOFF::SectionHeaderSize64;
+  else
+    RawPointer = XCOFF::FileHeaderSize32 + auxiliaryHeaderSize() +
   for (auto *Sec : Sections) {
     if (Sec->Index == SectionEntry::UninitializedIndex || Sec->IsVirtual)
       continue;
 
     Sec->FileOffsetToData = RawPointer;
     RawPointer += Sec->Size;
-    if (RawPointer > UINT32_MAX)
+    if (RawPointer > UINT32_MAX && !TargetObjectWriter->is64Bit())
       report_fatal_error("Section raw data overflowed this object file.");
   }
 
   RelocationEntryOffset = RawPointer;
 }
 
 // Takes the log base 2 of the alignment and shifts the result into the 5 most
 // significant bits of a byte, then or's in the csect type into the least