[yaml2obj]Allow explicit setting of p_filesz, p_memsz, and p_offset

[jhenderson]

Summary

yaml2obj currently derives the p_filesz, p_memsz, and p_offset values from sections. This makes writing tests for certain formats more complex, and sometimes impossible. This patch allows setting these fields explicitly, overriding the default value, when relevant.

[jakehehrlich]

This is much needed. Thanks!

[jakehehrlich]

I think there are some really funky tests in llvm-objcopy that we can fix now but it's going to take a careful eye to see which ones.

[jakehehrlich]

Oh also we can add PHDR tests to llvm-objcopy with this!

[Higuoxing]

LGTM

[jhenderson]

Thanks! Just to note that this does NOT force the sections that are in the segments to appear at different offsets than they otherwise would. That will require some different changes at some point, which I would certainly appreciate, but don't need at the moment, because you can use the section's address align value to achieve the same effect usually.

[grimar]

I have a minor question. I do not think it is really important so should not be stoppable for landing this.

test/tools/yaml2obj/program-header-size-offset.yaml

53 ↗

(On Diff #190663)

So out of curiosity: was it important for some reason to have an arbitrary numeric type here?
(and not a some kind of regular header)

[jhenderson]

test/tools/yaml2obj/program-header-size-offset.yaml

53 ↗

(On Diff #190663)

This was requested in an internal review of this before I put up a similar patch up for review previously. The idea is that the standard PT_* types have special semantics, and therefore by using a non-standard type, it bypasses any issues to do with it (and hopefully doesn't suggest that the type is important, together with the comment).

[grimar]

test/tools/yaml2obj/program-header-size-offset.yaml

53 ↗

(On Diff #190663)

I see, thanks.

(to clarify: the first thing I tried when saw this is to lookup in LLVM for 6abcdef0 and 0xabcdef0,
I did not instantly realize that it is just a set of alphabetical characters in order, but now after your reply
I re-read this place and that's looking fine for me now).

llvm/trunk/include/llvm/ObjectYAML/ELFYAML.h

 };
 
 struct ProgramHeader {
   ELF_PT Type;
   ELF_PF Flags;
   llvm::yaml::Hex64 VAddr;
   llvm::yaml::Hex64 PAddr;
   Optional<llvm::yaml::Hex64> Align;
+  Optional<llvm::yaml::Hex64> FileSize;
+  Optional<llvm::yaml::Hex64> MemSize;
+  Optional<llvm::yaml::Hex64> Offset;
   std::vector<SectionName> Sections;
 };
 
 struct Symbol {
   StringRef Name;
   ELF_STT Type;
   StringRef Section;
   Optional<ELF_SHN> Index;

llvm/trunk/lib/ObjectYAML/ELFYAML.cpp

 void MappingTraits<ELFYAML::ProgramHeader>::mapping(
     IO &IO, ELFYAML::ProgramHeader &Phdr) {
   IO.mapRequired("Type", Phdr.Type);
   IO.mapOptional("Flags", Phdr.Flags, ELFYAML::ELF_PF(0));
   IO.mapOptional("Sections", Phdr.Sections);
   IO.mapOptional("VAddr", Phdr.VAddr, Hex64(0));
   IO.mapOptional("PAddr", Phdr.PAddr, Hex64(0));
   IO.mapOptional("Align", Phdr.Align);
+  IO.mapOptional("FileSize", Phdr.FileSize);
+  IO.mapOptional("MemSize", Phdr.MemSize);
+  IO.mapOptional("Offset", Phdr.Offset);
 }
 
 namespace {
 
 struct NormalizedOther {
   NormalizedOther(IO &)
       : Visibility(ELFYAML::ELF_STV(0)), Other(ELFYAML::ELF_STO(0)) {}
   NormalizedOther(IO &, uint8_t Original)

llvm/trunk/test/tools/yaml2obj/program-header-size-offset.yaml

+# Show that yaml2obj properly emits program headers with explicit file size,
+# memory size and offset parameters.
+
+# RUN: yaml2obj %s -o %t
+# RUN: llvm-readobj %t --program-headers | FileCheck %s
+
+# CHECK: ProgramHeaders [
+# CHECK:    Offset: 0x1234
+# CHECK:    FileSize: 1111
+# CHECK:    MemSize: 9999
+
+# CHECK:    Offset: 0x2000
+# CHECK:    FileSize: 6
+# CHECK:    MemSize: 6
+
+# CHECK:    Offset: 0x2000
+# CHECK:    FileSize: 4
+# CHECK:    MemSize: 6
+
+# CHECK:    Offset: 0x1FFF
+# CHECK:    FileSize: 5
+# CHECK:    MemSize: 5
+
+# CHECK:    Offset: 0xFFE
+# CHECK:    FileSize: 7
+# CHECK:    MemSize: 9
+
+# CHECK:    Offset: 0x3000
+# CHECK:    FileSize: 3
+# CHECK:    MemSize: 2
+# CHECK: ]
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+  Machine:         EM_X86_64
+Sections:
+  - Name: .text
+    Type: SHT_PROGBITS
+    Size: 4
+    AddressAlign: 0x1000
+  - Name: .rodata
+    Type: SHT_PROGBITS
+    Size: 4
+    AddressAlign: 0x1000
+  - Name: .data
+    Type: SHT_PROGBITS
+    Size: 4
+ProgramHeaders:
+  # Program header with no sections.
+  - Type:     0x6abcdef0 # arbitrary type
+    Offset:   0x1234
+    FileSize: 1111
+    MemSize:  9999
+  # Program header with only file size set.
+  - Type:     0x6abcdef0
+    FileSize: 6
+    Sections:
+      - Section: .rodata
+  # Program header with only mem size set.
+  - Type:     0x6abcdef0
+    MemSize: 6
+    Sections:
+      - Section: .rodata
+  # Program header with only offset set.
+  - Type:     0x6abcdef0
+    Offset:   0x1fff
+    Sections:
+      - Section: .rodata
+  # Program header with sections, valid properties.
+  - Type:     0x6abcdef0
+    Offset:   0xffe
+    FileSize: 7
+    MemSize:  9
+    Sections:
+      - Section: .text
+  # Program header with sections, invalid properties.
+  - Type:     0x6abcdef0
+    Offset:   0x3000
+    FileSize: 3
+    MemSize:  2
+    Sections:
+      - Section: .data

llvm/trunk/tools/yaml2obj/yaml2elf.cpp

 
 template <class ELFT>
 void ELFState<ELFT>::setProgramHeaderLayout(std::vector<Elf_Phdr> &PHeaders,
                                             std::vector<Elf_Shdr> &SHeaders) {
   uint32_t PhdrIdx = 0;
   for (auto &YamlPhdr : Doc.ProgramHeaders) {
     auto &PHeader = PHeaders[PhdrIdx++];
 
+    if (YamlPhdr.Offset) {
+      PHeader.p_offset = *YamlPhdr.Offset;
+    } else {
       if (YamlPhdr.Sections.size())
         PHeader.p_offset = UINT32_MAX;
       else
         PHeader.p_offset = 0;
 
       // Find the minimum offset for the program header.
       for (auto SecName : YamlPhdr.Sections) {
         uint32_t Index = 0;
         SN2I.lookup(SecName.Section, Index);
         const auto &SHeader = SHeaders[Index];
         PHeader.p_offset = std::min(PHeader.p_offset, SHeader.sh_offset);
       }
+    }
 
-    // Find the maximum offset of the end of a section in order to set p_filesz.
+    // Find the maximum offset of the end of a section in order to set p_filesz,
+    // if not set explicitly.
+    if (YamlPhdr.FileSize) {
+      PHeader.p_filesz = *YamlPhdr.FileSize;
+    } else {
       PHeader.p_filesz = 0;
       for (auto SecName : YamlPhdr.Sections) {
         uint32_t Index = 0;
         SN2I.lookup(SecName.Section, Index);
         const auto &SHeader = SHeaders[Index];
         uint64_t EndOfSection;
         if (SHeader.sh_type == llvm::ELF::SHT_NOBITS)
           EndOfSection = SHeader.sh_offset;
         else
           EndOfSection = SHeader.sh_offset + SHeader.sh_size;
         uint64_t EndOfSegment = PHeader.p_offset + PHeader.p_filesz;
         EndOfSegment = std::max(EndOfSegment, EndOfSection);
         PHeader.p_filesz = EndOfSegment - PHeader.p_offset;
       }
+    }
 
-    // Find the memory size by adding the size of sections at the end of the
-    // segment. These should be empty (size of zero) and NOBITS sections.
+    // If not set explicitly, find the memory size by adding the size of
+    // sections at the end of the segment. These should be empty (size of zero)
+    // and NOBITS sections.
+    if (YamlPhdr.MemSize) {
+      PHeader.p_memsz = *YamlPhdr.MemSize;
+    } else {
       PHeader.p_memsz = PHeader.p_filesz;
       for (auto SecName : YamlPhdr.Sections) {
         uint32_t Index = 0;
         SN2I.lookup(SecName.Section, Index);
         const auto &SHeader = SHeaders[Index];
         if (SHeader.sh_offset == PHeader.p_offset + PHeader.p_filesz)
           PHeader.p_memsz += SHeader.sh_size;
       }
+    }
 
     // Set the alignment of the segment to be the same as the maximum alignment
     // of the sections with the same offset so that by default the segment
     // has a valid and sensible alignment.
     if (YamlPhdr.Align) {
       PHeader.p_align = *YamlPhdr.Align;
     } else {
       PHeader.p_align = 1;