[yaml2obj] - Add a support for defining null sections in YAMLs.

[grimar]

Summary

ELF spec shows (Figure 4-10: Section Header Table Entry:Index 0,
http://www.sco.com/developers/gabi/latest/ch4.sheader.html)
that section header at index 0 (null section) can have sh_size and
sh_link fields set to non-zero values.

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

If the number of sections is greater than or equal to SHN_LORESERVE (0xff00),
this member has the value zero and the actual number of section header table
entries is contained in the sh_size field of the section header at index 0.
Otherwise, the sh_size member of the initial entry contains 0.

and:

If the section name string table section index is greater than or equal to SHN_LORESERVE
(0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section
name string table section is contained in the sh_link field of the section header at index 0.
Otherwise, the sh_link member of the initial entry contains 0.

At this moment it is not possible to create custom section headers at index 0 using yaml2obj.
This patch implements this.

[MaskRay]

In the description:

name string table section is contained in thesh_link field of the section header at index 0.

Typo: thesh_link -> the sh_link

tools/yaml2obj/yaml2elf.cpp

136

Period at the end.

[grimar]

• Addressed review comment.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

7

Calling this prefix CASE1 when it is used by two test cases is a bit confusion. I recommend "DEFAULT" or something like that.

23

which fields are all zeroed -> with fields all zeroed

24

to the section created

(don't need both the "normally" and "by default").

42

Specify Link explicitly with an integer here (i.e. Link: 0).

What about Info, and Address and perhaps sh_offset?

77

Similar to above, this is called "CASE3" but is used in two different test cases. Perhaps REDEFINE?

96

Nit: double blank line

97

"The same as above"

115

a error -> an error
to unknown -> to an unknown

131

Perhaps worth a slight rewording here:

"Check that null section fields are set to zero, if they are unspecified."

143

How about a test case for a SHT_NULL section that is not the first listed section? I don't think there's anything in the ELF spec saying that you can't have one elsewhere in the section header table.

tools/yaml2obj/yaml2elf.cpp

324–325

I wonder if it would make sense to treat the null section the same as any other implicit section? What do you think?

[MaskRay]

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

The Solaris Linker and Libraries Guide can serve as a reference if the official ELF spec is missing something that we agree upon, but for the SHN_LORESERVE thing you can just quote http://www.sco.com/developers/gabi/latest/ch4.eheader.html :

e_shnum
...
If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), this member has the value zero and the actual number of section header table entries is contained in the sh_size field of the section header at index 0. (Otherwise, the sh_size member of the initial entry contains 0.)

e_shstrndx
...
If the section name string table section index is greater than or equal to SHN_LORESERVE (0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section name string table section is contained in the sh_link field of the section header at index 0. (Otherwise, the sh_link member of the initial entry contains 0.)

[MaskRay]

test/tools/yaml2obj/elf-custom-null-section.yaml

47

CASE2 -> a descriptive name. (I'd call it EXPLICIT-ZERO but there may be a better name).

77

Or
CASE3 -> SEC-REF-LINK
CASE4 -> NUMERAL-LINK

tools/yaml2obj/yaml2elf.cpp

324–325

How to do that? SHN_UNDEF (of SHT_NULL) is the first section, different from other implicit sections...

[grimar]

test/tools/yaml2obj/elf-custom-null-section.yaml

42

What about Info, and Address and perhaps sh_offset?

I focused on a use cases which are correct from ELF spec view in this patch.

Adding support for another fields can be a part of follow-up refactoring.
(I think we might be able to handle SHT_NULL in the same loop as other sections in initSectionHeaders,
that would allow to support setting all other fields at once).

tools/yaml2obj/yaml2elf.cpp

324–325

Yes, it is very different. I also do not see a clean way to handle it as a regular implicit section.
It would require lot of changes everywhere, at least because we place implicit sections to the end,
what doesn't work for SHT_NULL.

But seems the solution is to land D64999 first. It should allow to significantly simplify this patch.

[grimar]

• Reimplemented.

Now rebased on top of D65087.

test/tools/yaml2obj/elf-custom-null-section.yaml

42

What about Info, and Address and perhaps sh_offset?

Oops, I misread slightly. I added Info and Address zeroed fields here.

This patch still doesn't support setting them to something else though.
I didn't add ShOffset or ShSize, this is out of the area of this patch and
anyways not supported either.

47

I used ZERO-VS-NONZERO because the test case above already explicitly defines the zero section.

143

I think that wasn't never intentionally supported, I added a test to check we do not crash.
Not sure if we want to check something else here.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

49

I don't like "ZERO-VS-NONZERO" here, because it's not really anything to do with that. This test case is for testing that other sections can be added after the NULL section. "ZERO-VS-NONZERO" doesn't clarify that. How about "OTHER-SECTION"?

72

Nit: Link: 0 (be consistent with above change).

146

if we have

163–164

Perhaps worth setting some fields to different explicit values, to show that it's possible.

[grimar]

• Addressed review comments.

[jhenderson]

LGTM. Thanks!

test/tools/yaml2obj/duplicate-section-names.test

- Name: '.foo [1]'
     Type: SHT_PROGBITS
   - Name: .foo2
     Type: SHT_PROGBITS
 
 ## Check that yaml2obj reports an error in case we have
 ## sections with equal names and suffixes.
 
 # RUN: not yaml2obj --docnum=2 %s 2>&1 | FileCheck %s --check-prefix=CASE2
-# CASE2: error: Repeated section name: '.foo [1]' at YAML section number 1.
+# CASE2: error: Repeated section name: '.foo [1]' at YAML section number 2.
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_X86_64
 Sections:
   - Name: '.foo [1]'
     Type: SHT_PROGBITS
   - Name: '.foo [1]'
     Type: SHT_PROGBITS
 
 ## Check that yaml2obj reports an error in case we have
 ## symbols without suffixes in the names and their
 ## names are equal.
 
 # RUN: not yaml2obj --docnum=3 %s 2>&1 | FileCheck %s --check-prefix=CASE3
-# CASE3: error: Repeated section name: '.foo' at YAML section number 1.
+# CASE3: error: Repeated section name: '.foo' at YAML section number 2.
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_X86_64
 Sections:

test/tools/yaml2obj/elf-custom-null-section.yaml

+## In this test we check that can redefine the null section in the YAML.
+
+## Test the default output first.
+
+# RUN: yaml2obj --docnum=1 %s -o %t1
+# RUN: llvm-readelf --sections %t1 | FileCheck %s --check-prefix=DEFAULT
+
+# DEFAULT:      Section Headers:
+# DEFAULT-NEXT:  [Nr] Name      Type   Address          Off    Size   ES Flg Lk Inf Al
+# DEFAULT-NEXT:  [ 0]           NULL   0000000000000000 000000 000000 00 0   0  0
+# DEFAULT-NEXT:  [ 1] .symtab   SYMTAB 0000000000000000 000140 000018 18 2   1  8
+# DEFAULT-NEXT:  [ 2] .strtab   STRTAB 0000000000000000 000158 000001 00 0   0  1
+# DEFAULT-NEXT:  [ 3] .shstrtab STRTAB 0000000000000000 000159 00001b 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+
+## Now define a SHT_NULL section with fields all zeroed.
+## In this case it is equal to the section created by default.
+
+# RUN: yaml2obj --docnum=2 %s -o %t2
+# RUN: llvm-readelf --sections %t2 | FileCheck %s --check-prefix=DEFAULT
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         0
+    Info:         0
+    Address:      0x0
+
+## Check we are still able to describe other sections too.
+
+# RUN: yaml2obj --docnum=3 %s -o %t3
+# RUN: llvm-readelf --sections %t3 | FileCheck %s --check-prefix=OTHER-SECTION
+
+# OTHER-SECTION:      Section Headers:
+# OTHER-SECTION-NEXT:   [Nr] Name      Type     Address          Off    Size   ES Flg Lk Inf Al
+# OTHER-SECTION-NEXT:   [ 0]           NULL     0000000000000000 000000 000000 00 0   0  0
+# OTHER-SECTION-NEXT:   [ 1] foo       PROGBITS 0000000000000000 000180 000000 00 0   0  0
+# OTHER-SECTION-NEXT:   [ 2] .symtab   SYMTAB   0000000000000000 000180 000018 18 3   1  8
+# OTHER-SECTION-NEXT:   [ 3] .strtab   STRTAB   0000000000000000 000198 000001 00 0   0  1
+# OTHER-SECTION-NEXT:   [ 4] .shstrtab STRTAB   0000000000000000 000199 00001f 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         0
+  - Type: SHT_PROGBITS
+    Name: 'foo'
+
+## Check we can redefine sh_size and sh_link fields of the SHT_NULL section.
+
+# RUN: yaml2obj --docnum=4 %s -o %t4
+# RUN: llvm-readelf --sections %t4 | FileCheck %s --check-prefix=REDEFINE
+
+# REDEFINE:      Section Headers:
+# REDEFINE-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# REDEFINE-NEXT:  [ 0]      NULL 0000000000000000 000000 000123 00     1   0  0
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+## The same as above, but using a number as a Link value.
+
+# RUN: yaml2obj --docnum=5 %s -o %t5
+# RUN: llvm-readelf --sections %t5 | FileCheck %s --check-prefix=REDEFINE
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: 1
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+## Check we report an error if null section sh_link field refers to an unknown section.
+
+# RUN: not yaml2obj --docnum=6 %s -o %t6 2>&1 | FileCheck %s --check-prefix=CASE4
+
+# CASE4: error: Unknown section referenced: '.foo' at YAML section ''.
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+
+## Check that null section fields are set to zero, if they are unspecified.
+
+# RUN: yaml2obj --docnum=7 %s -o %t7
+# RUN: llvm-readelf --sections %t7 | FileCheck %s --check-prefix=DEFAULT
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+
+## Check we do not crash if we have more than one SHT_NULL section.
+
+# RUN: yaml2obj --docnum=8 %s -o %t8
+# RUN: llvm-readelf --sections %t8 | FileCheck %s --check-prefix=MULTIPLE
+
+# MULTIPLE:      Section Headers:
+# MULTIPLE-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# MULTIPLE-NEXT:  [ 0]      NULL 0000000000000000 000000 000000 00 0   0  0
+# MULTIPLE-NEXT:  [ 1]      NULL 0000000000000123 000180 000020 10 A   1  2   0
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+  - Type:    SHT_NULL
+    Flags:   [ SHF_ALLOC ]
+    Size:    0x20
+    EntSize: 0x10
+    Link:    1
+    Info:    2
+    Address: 0x123

tools/yaml2obj/yaml2elf.cpp

class ELFState {
 
   /// The future ".dynstr" section.
   StringTableBuilder DotDynstr{StringTableBuilder::ELF};
 
   NameToIdxMap SN2I;
   NameToIdxMap SymN2I;
   ELFYAML::Object &Doc;
 
   bool buildSectionIndex();
   bool buildSymbolIndex(ArrayRef<ELFYAML::Symbol> Symbols);
   void initELFHeader(Elf_Ehdr &Header);
   void initProgramHeaders(std::vector<Elf_Phdr> &PHeaders);
   bool initImplicitHeader(ELFState<ELFT> &State, ContiguousBlobAccumulator &CBA,
                           Elf_Shdr &Header, StringRef SecName,
                           ELFYAML::Section *YAMLSec);
   bool initSectionHeaders(ELFState<ELFT> &State,
                           std::vector<Elf_Shdr> &SHeaders,

 
 template <class ELFT>
 ELFState<ELFT>::ELFState(ELFYAML::Object &D) : Doc(D) {
   StringSet<> DocSections;
   for (std::unique_ptr<ELFYAML::Section> &D : Doc.Sections)
     if (!D->Name.empty())
       DocSections.insert(D->Name);
 
+  // Insert SHT_NULL section implicitly when it is not defined in YAML.
+  if (Doc.Sections.empty() || Doc.Sections.front()->Type != ELF::SHT_NULL)
+    Doc.Sections.insert(
+        Doc.Sections.begin(),
+        llvm::make_unique<ELFYAML::Section>(
+            ELFYAML::Section::SectionKind::RawContent, true /*IsImplicit*/));
+
   std::vector<StringRef> ImplicitSections = {".symtab", ".strtab", ".shstrtab"};
   if (!Doc.DynamicSymbols.empty())
     ImplicitSections.insert(ImplicitSections.end(), {".dynsym", ".dynstr"});
 
   // Insert placeholders for implicit sections that are not
   // defined explicitly in YAML.
   for (StringRef SecName : ImplicitSections) {
     if (DocSections.count(SecName))
static StringRef dropUniqueSuffix(StringRef S) {
   if (SuffixPos == StringRef::npos)
     return S;
   return S.substr(0, SuffixPos);
 }
 
 template <class ELFT>
 bool ELFState<ELFT>::initSectionHeaders(ELFState<ELFT> &State,
                                         std::vector<Elf_Shdr> &SHeaders,
                                         ContiguousBlobAccumulator &CBA) {
   // Ensure SHN_UNDEF entry is present. An all-zero section header is a
   // valid SHN_UNDEF entry since SHT_NULL == 0.
-  SHeaders.resize(Doc.Sections.size() + 1);
-  zero(SHeaders[0]);
+  SHeaders.resize(Doc.Sections.size());
 
-  for (size_t I = 1; I < Doc.Sections.size() + 1; ++I) {
+  for (size_t I = 0; I < Doc.Sections.size(); ++I) {
     Elf_Shdr &SHeader = SHeaders[I];
-    zero(SHeader);
-    ELFYAML::Section *Sec = Doc.Sections[I - 1].get();
+    ELFYAML::Section *Sec = Doc.Sections[I].get();
+
+    if (I == 0) {
+      if (Sec->IsImplicit)
+        continue;
+
+      if (auto S = dyn_cast<ELFYAML::RawContentSection>(Sec))
+        if (S->Size)
+          SHeader.sh_size = *S->Size;
+
+      if (!Sec->Link.empty()) {
+        unsigned Index;
+        if (!convertSectionIndex(SN2I, Sec->Name, Sec->Link, Index))
+          return false;
+        SHeader.sh_link = Index;
+      }
+      continue;
+    }
 
     // We have a few sections like string or symbol tables that are usually
     // added implicitly to the end. However, if they are explicitly specified
     // in the YAML, we need to write them here. This ensures the file offset
     // remains correct.
     if (initImplicitHeader(State, CBA, SHeader, Sec->Name,
                            Sec->IsImplicit ? nullptr : Sec))
       continue;
bool ELFState<ELFT>::writeSectionContent(Elf_Shdr &SHeader,
   }
   if (Section.Content)
     Section.Content->writeAsBinary(OS);
 
   return true;
 }
 
 template <class ELFT> bool ELFState<ELFT>::buildSectionIndex() {
-  for (unsigned i = 0, e = Doc.Sections.size(); i != e; ++i) {
-    StringRef Name = Doc.Sections[i]->Name;
+  for (unsigned I = 1, E = Doc.Sections.size(); I != E; ++I) {
+    StringRef Name = Doc.Sections[I]->Name;
     DotShStrtab.add(dropUniqueSuffix(Name));
-    // "+ 1" to take into account the SHT_NULL entry.
-    if (!SN2I.addName(Name, i + 1)) {
+    if (!SN2I.addName(Name, I)) {
       WithColor::error() << "Repeated section name: '" << Name
-                         << "' at YAML section number " << i << ".\n";
+                         << "' at YAML section number " << I << ".\n";
       return false;
     }
   }
 
   DotShStrtab.finalize();
   return true;
 }