[yaml2obj] - Add a support for defining null sections in YAMLs.

[grimar]

Summary

ELF spec shows (Figure 4-10: Section Header Table Entry:Index 0,
http://www.sco.com/developers/gabi/latest/ch4.sheader.html)
that section header at index 0 (null section) can have sh_size and
sh_link fields set to non-zero values.

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

If the number of sections is greater than or equal to SHN_LORESERVE (0xff00),
this member has the value zero and the actual number of section header table
entries is contained in the sh_size field of the section header at index 0.
Otherwise, the sh_size member of the initial entry contains 0.

and:

If the section name string table section index is greater than or equal to SHN_LORESERVE
(0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section
name string table section is contained in the sh_link field of the section header at index 0.
Otherwise, the sh_link member of the initial entry contains 0.

At this moment it is not possible to create custom section headers at index 0 using yaml2obj.
This patch implements this.

[MaskRay]

In the description:

name string table section is contained in thesh_link field of the section header at index 0.

Typo: thesh_link -> the sh_link

tools/yaml2obj/yaml2elf.cpp

135

Period at the end.

[grimar]

• Addressed review comment.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

6

Calling this prefix CASE1 when it is used by two test cases is a bit confusion. I recommend "DEFAULT" or something like that.

22

which fields are all zeroed -> with fields all zeroed

23

to the section created

(don't need both the "normally" and "by default").

41

Specify Link explicitly with an integer here (i.e. Link: 0).

What about Info, and Address and perhaps sh_offset?

76

Similar to above, this is called "CASE3" but is used in two different test cases. Perhaps REDEFINE?

95

Nit: double blank line

96

"The same as above"

114

a error -> an error
to unknown -> to an unknown

130

Perhaps worth a slight rewording here:

"Check that null section fields are set to zero, if they are unspecified."

142

How about a test case for a SHT_NULL section that is not the first listed section? I don't think there's anything in the ELF spec saying that you can't have one elsewhere in the section header table.

tools/yaml2obj/yaml2elf.cpp

301

I wonder if it would make sense to treat the null section the same as any other implicit section? What do you think?

[MaskRay]

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

The Solaris Linker and Libraries Guide can serve as a reference if the official ELF spec is missing something that we agree upon, but for the SHN_LORESERVE thing you can just quote http://www.sco.com/developers/gabi/latest/ch4.eheader.html :

e_shnum
...
If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), this member has the value zero and the actual number of section header table entries is contained in the sh_size field of the section header at index 0. (Otherwise, the sh_size member of the initial entry contains 0.)

e_shstrndx
...
If the section name string table section index is greater than or equal to SHN_LORESERVE (0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section name string table section is contained in the sh_link field of the section header at index 0. (Otherwise, the sh_link member of the initial entry contains 0.)

[MaskRay]

test/tools/yaml2obj/elf-custom-null-section.yaml

46

CASE2 -> a descriptive name. (I'd call it EXPLICIT-ZERO but there may be a better name).

76

Or
CASE3 -> SEC-REF-LINK
CASE4 -> NUMERAL-LINK

tools/yaml2obj/yaml2elf.cpp

301

How to do that? SHN_UNDEF (of SHT_NULL) is the first section, different from other implicit sections...

[grimar]

test/tools/yaml2obj/elf-custom-null-section.yaml

41

What about Info, and Address and perhaps sh_offset?

I focused on a use cases which are correct from ELF spec view in this patch.

Adding support for another fields can be a part of follow-up refactoring.
(I think we might be able to handle SHT_NULL in the same loop as other sections in initSectionHeaders,
that would allow to support setting all other fields at once).

tools/yaml2obj/yaml2elf.cpp

301

Yes, it is very different. I also do not see a clean way to handle it as a regular implicit section.
It would require lot of changes everywhere, at least because we place implicit sections to the end,
what doesn't work for SHT_NULL.

But seems the solution is to land D64999 first. It should allow to significantly simplify this patch.

[grimar]

• Reimplemented.

Now rebased on top of D65087.

test/tools/yaml2obj/elf-custom-null-section.yaml

41

What about Info, and Address and perhaps sh_offset?

Oops, I misread slightly. I added Info and Address zeroed fields here.

This patch still doesn't support setting them to something else though.
I didn't add ShOffset or ShSize, this is out of the area of this patch and
anyways not supported either.

46

I used ZERO-VS-NONZERO because the test case above already explicitly defines the zero section.

142

I think that wasn't never intentionally supported, I added a test to check we do not crash.
Not sure if we want to check something else here.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

49

I don't like "ZERO-VS-NONZERO" here, because it's not really anything to do with that. This test case is for testing that other sections can be added after the NULL section. "ZERO-VS-NONZERO" doesn't clarify that. How about "OTHER-SECTION"?

72

Nit: Link: 0 (be consistent with above change).

146

if we have

163–164

Perhaps worth setting some fields to different explicit values, to show that it's possible.

[grimar]

• Addressed review comments.

[jhenderson]

LGTM. Thanks!

test/tools/yaml2obj/elf-custom-null-section.yaml

+## In this test we check that can redefine the null section in the YAML.
+
+## Test the default output first.
+
+# RUN: yaml2obj --docnum=1 %s -o %t1
+# RUN: llvm-readelf --sections %t1 | FileCheck %s --check-prefix=CASE1
+
+# CASE1:      Section Headers:
+# CASE1-NEXT:  [Nr] Name      Type   Address          Off    Size   ES Flg Lk Inf Al
+# CASE1-NEXT:  [ 0]           NULL   0000000000000000 000000 000000 00 0   0  0
+# CASE1-NEXT:  [ 1] .symtab   SYMTAB 0000000000000000 000140 000018 18 2   1  8
+# CASE1-NEXT:  [ 2] .strtab   STRTAB 0000000000000000 000158 000001 00 0   0  1
+# CASE1-NEXT:  [ 3] .shstrtab STRTAB 0000000000000000 000159 00001b 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+
+## Now define a SHT_NULL section which fields are all zeroed.
+## In this case it is equal to section normally created by default.
+
+# RUN: yaml2obj --docnum=2 %s -o %t2
+# RUN: llvm-readelf --sections %t2 | FileCheck %s --check-prefix=CASE1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         ""
+
+## Check we are still able to describe other sections too.
+
+# RUN: yaml2obj --docnum=3 %s -o %t3
+# RUN: llvm-readelf --sections %t3 | FileCheck %s --check-prefix=CASE2
+
+# CASE2:      Section Headers:
+# CASE2-NEXT:   [Nr] Name      Type     Address          Off    Size   ES Flg Lk Inf Al
+# CASE2-NEXT:   [ 0]           NULL     0000000000000000 000000 000000 00 0   0  0
+# CASE2-NEXT:   [ 1] foo       PROGBITS 0000000000000000 000180 000000 00 0   0  0
+# CASE2-NEXT:   [ 2] .symtab   SYMTAB   0000000000000000 000180 000018 18 3   1  8
+# CASE2-NEXT:   [ 3] .strtab   STRTAB   0000000000000000 000198 000001 00 0   0  1
+# CASE2-NEXT:   [ 4] .shstrtab STRTAB   0000000000000000 000199 00001f 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         ""
+  - Type: SHT_PROGBITS
+    Name: 'foo'
+
+## Check we can redefine sh_size and sh_link fields of the SHT_NULL section.
+
+# RUN: yaml2obj --docnum=4 %s -o %t4
+# RUN: llvm-readelf --sections %t4 | FileCheck %s --check-prefix=CASE3
+
+# CASE3:      Section Headers:
+# CASE3-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# CASE3-NEXT:  [ 0]      NULL 0000000000000000 000000 000123 00     1   0  0
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+
+## The same, but using a number as a Link value.
+
+# RUN: yaml2obj --docnum=5 %s -o %t5
+# RUN: llvm-readelf --sections %t5 | FileCheck %s --check-prefix=CASE3
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: 1
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+## Check we report a error if null section sh_link field refers to unknown section.
+
+# RUN: not yaml2obj --docnum=6 %s -o %t6 2>&1 | FileCheck %s --check-prefix=CASE4
+
+# CASE4: error: Unknown section referenced: '.foo' at YAML section ''.
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+
+## Check that null section fields are zeroed by default.
+
+# RUN: yaml2obj --docnum=7 %s -o %t7
+# RUN: llvm-readelf --sections %t7 | FileCheck %s --check-prefix=CASE1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL

tools/yaml2obj/yaml2elf.cpp

class ELFState {
 
   /// The future ".dynstr" section.
   StringTableBuilder DotDynstr{StringTableBuilder::ELF};
 
   NameToIdxMap SN2I;
   NameToIdxMap SymN2I;
   const ELFYAML::Object &Doc;
 
+  // Non-zero when document explicitly describes SHT_NULL section.
+  ELFYAML::Section *DocNullSec = nullptr;
+
   bool buildSectionIndex();
   bool buildSymbolIndex(ArrayRef<ELFYAML::Symbol> Symbols);
   void initELFHeader(Elf_Ehdr &Header);
   void initProgramHeaders(std::vector<Elf_Phdr> &PHeaders);
   bool initImplicitHeader(ELFState<ELFT> &State, ContiguousBlobAccumulator &CBA,
                           Elf_Shdr &Header, StringRef SecName,
                           ELFYAML::Section *YAMLSec);
   bool initSectionHeaders(ELFState<ELFT> &State,

 
 template <class ELFT>
 bool ELFState<ELFT>::initSectionHeaders(ELFState<ELFT> &State,
                                         std::vector<Elf_Shdr> &SHeaders,
                                         ContiguousBlobAccumulator &CBA) {
   // Build a list of sections we are going to add implicitly.
   std::vector<StringRef> ImplicitSections;
   for (StringRef Name : State.implicitSectionNames())
-    if (State.SN2I.get(Name) > Doc.Sections.size())
+    if (State.SN2I.get(Name) > (Doc.Sections.size() - (DocNullSec ? 1 : 0)))
       ImplicitSections.push_back(Name);
 
   // Ensure SHN_UNDEF entry is present. An all-zero section header is a
   // valid SHN_UNDEF entry since SHT_NULL == 0.
-  SHeaders.resize(Doc.Sections.size() + ImplicitSections.size() + 1);
+  SHeaders.resize(Doc.Sections.size() + ImplicitSections.size() +
+                  (DocNullSec ? 0 : 1));
   zero(SHeaders[0]);
 
-  for (size_t I = 1; I < Doc.Sections.size() + ImplicitSections.size() + 1; ++I) {
+  // If the null section header is explicitly defined in YAML, handle it.
+  if (DocNullSec) {
+    Elf_Shdr &Hdr = SHeaders[0];
+    if (auto S = dyn_cast<ELFYAML::RawContentSection>(DocNullSec))
+      if (S->Size)
+        Hdr.sh_size = *S->Size;
+
+    if (!DocNullSec->Link.empty()) {
+      unsigned Index;
+      if (!convertSectionIndex(SN2I, DocNullSec->Name, DocNullSec->Link, Index))
+        return false;
+      Hdr.sh_link = Index;
+    }
+  }
+
+  for (size_t I = 1; I < SHeaders.size(); ++I) {
     Elf_Shdr &SHeader = SHeaders[I];
     zero(SHeader);
-    ELFYAML::Section *Sec =
-        I > Doc.Sections.size() ? nullptr : Doc.Sections[I - 1].get();
+
+    ELFYAML::Section *Sec = nullptr;
+    if (I < (Doc.Sections.size() + (DocNullSec ? 0 : 1)))
+      Sec = Doc.Sections[I - (DocNullSec ? 0 : 1)].get();
 
     // We have a few sections like string or symbol tables that are usually
     // added implicitly to the end. However, if they are explicitly specified
     // in the YAML, we need to write them here. This ensures the file offset
     // remains correct.
     StringRef SecName =
-        Sec ? Sec->Name : ImplicitSections[I - Doc.Sections.size() - 1];
+        Sec ? Sec->Name
+            : ImplicitSections[I - Doc.Sections.size() - (DocNullSec ? 0 : 1)];
+
     if (initImplicitHeader(State, CBA, SHeader, SecName, Sec))
       continue;
 
     assert(Sec && "It can't be null unless it is an implicit section. But all "
                   "implicit sections should already have been handled above.");
 
     SHeader.sh_name = DotShStrtab.getOffset(dropUniqueSuffix(SecName));
     SHeader.sh_type = Sec->Type;
bool ELFState<ELFT>::writeSectionContent(Elf_Shdr &SHeader,
   }
   if (Section.Content)
     Section.Content->writeAsBinary(OS);
 
   return true;
 }
 
 template <class ELFT> bool ELFState<ELFT>::buildSectionIndex() {
-  for (unsigned i = 0, e = Doc.Sections.size(); i != e; ++i) {
-    StringRef Name = Doc.Sections[i]->Name;
+  // We might have an explicitly defined zero section header.
+  if (!Doc.Sections.empty() && Doc.Sections[0]->Type == ELF::SHT_NULL)
+    DocNullSec = Doc.Sections[0].get();
+
+  for (unsigned I = 0, E = Doc.Sections.size(); I != E; ++I) {
+    if (DocNullSec && I == 0)
+      continue;
+
+    StringRef Name = Doc.Sections[I]->Name;
     DotShStrtab.add(dropUniqueSuffix(Name));
-    // "+ 1" to take into account the SHT_NULL entry.
-    if (!SN2I.addName(Name, i + 1)) {
+    // "+ 1" to take into account the SHT_NULL entry if not explicitly defined.
+    if (!SN2I.addName(Name, I + (DocNullSec ? 0 : 1))) {
       WithColor::error() << "Repeated section name: '" << Name
-                         << "' at YAML section number " << i << ".\n";
+                         << "' at YAML section number " << I << ".\n";
       return false;
     }
   }
 
-  auto SecNo = 1 + Doc.Sections.size();
+  unsigned SecNo = Doc.Sections.size() + (DocNullSec ? 0 : 1);
   // Add special sections after input sections, if necessary.
   for (StringRef Name : implicitSectionNames())
     if (SN2I.addName(Name, SecNo)) {
       // Account for this section, since it wasn't in the Doc
       ++SecNo;
       DotShStrtab.add(Name);
     }