[yaml2obj] - Add a support for defining null sections in YAMLs.

[grimar]

Summary

ELF spec shows (Figure 4-10: Section Header Table Entry:Index 0,
http://www.sco.com/developers/gabi/latest/ch4.sheader.html)
that section header at index 0 (null section) can have sh_size and
sh_link fields set to non-zero values.

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

If the number of sections is greater than or equal to SHN_LORESERVE (0xff00),
this member has the value zero and the actual number of section header table
entries is contained in the sh_size field of the section header at index 0.
Otherwise, the sh_size member of the initial entry contains 0.

and:

If the section name string table section index is greater than or equal to SHN_LORESERVE
(0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section
name string table section is contained in the sh_link field of the section header at index 0.
Otherwise, the sh_link member of the initial entry contains 0.

At this moment it is not possible to create custom section headers at index 0 using yaml2obj.
This patch implements this.

[MaskRay]

In the description:

name string table section is contained in thesh_link field of the section header at index 0.

Typo: thesh_link -> the sh_link

tools/yaml2obj/yaml2elf.cpp

135 ↗

(On Diff #210529)

Period at the end.

[grimar]

• Addressed review comment.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

6 ↗

(On Diff #210792)

Calling this prefix CASE1 when it is used by two test cases is a bit confusion. I recommend "DEFAULT" or something like that.

22 ↗

(On Diff #210792)

which fields are all zeroed -> with fields all zeroed

23 ↗

(On Diff #210792)

to the section created

(don't need both the "normally" and "by default").

41 ↗

(On Diff #210792)

Specify Link explicitly with an integer here (i.e. Link: 0).

What about Info, and Address and perhaps sh_offset?

76 ↗

(On Diff #210792)

Similar to above, this is called "CASE3" but is used in two different test cases. Perhaps REDEFINE?

95 ↗

(On Diff #210792)

Nit: double blank line

96 ↗

(On Diff #210792)

"The same as above"

114 ↗

(On Diff #210792)

a error -> an error
to unknown -> to an unknown

130 ↗

(On Diff #210792)

Perhaps worth a slight rewording here:

"Check that null section fields are set to zero, if they are unspecified."

142 ↗

(On Diff #210792)

How about a test case for a SHT_NULL section that is not the first listed section? I don't think there's anything in the ELF spec saying that you can't have one elsewhere in the section header table.

tools/yaml2obj/yaml2elf.cpp

301 ↗

(On Diff #210792)

I wonder if it would make sense to treat the null section the same as any other implicit section? What do you think?

[MaskRay]

It says (https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtc9/index.html):

The Solaris Linker and Libraries Guide can serve as a reference if the official ELF spec is missing something that we agree upon, but for the SHN_LORESERVE thing you can just quote http://www.sco.com/developers/gabi/latest/ch4.eheader.html :

e_shnum
...
If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), this member has the value zero and the actual number of section header table entries is contained in the sh_size field of the section header at index 0. (Otherwise, the sh_size member of the initial entry contains 0.)

e_shstrndx
...
If the section name string table section index is greater than or equal to SHN_LORESERVE (0xff00), this member has the value SHN_XINDEX (0xffff) and the actual index of the section name string table section is contained in the sh_link field of the section header at index 0. (Otherwise, the sh_link member of the initial entry contains 0.)

[MaskRay]

test/tools/yaml2obj/elf-custom-null-section.yaml

46 ↗

(On Diff #210792)

CASE2 -> a descriptive name. (I'd call it EXPLICIT-ZERO but there may be a better name).

76 ↗

(On Diff #210792)

Or
CASE3 -> SEC-REF-LINK
CASE4 -> NUMERAL-LINK

tools/yaml2obj/yaml2elf.cpp

301 ↗

(On Diff #210792)

How to do that? SHN_UNDEF (of SHT_NULL) is the first section, different from other implicit sections...

[grimar]

test/tools/yaml2obj/elf-custom-null-section.yaml

41 ↗

(On Diff #210792)

What about Info, and Address and perhaps sh_offset?

I focused on a use cases which are correct from ELF spec view in this patch.

Adding support for another fields can be a part of follow-up refactoring.
(I think we might be able to handle SHT_NULL in the same loop as other sections in initSectionHeaders,
that would allow to support setting all other fields at once).

tools/yaml2obj/yaml2elf.cpp

301 ↗

(On Diff #210792)

Yes, it is very different. I also do not see a clean way to handle it as a regular implicit section.
It would require lot of changes everywhere, at least because we place implicit sections to the end,
what doesn't work for SHT_NULL.

But seems the solution is to land D64999 first. It should allow to significantly simplify this patch.

[grimar]

• Reimplemented.

Now rebased on top of D65087.

test/tools/yaml2obj/elf-custom-null-section.yaml

41 ↗

(On Diff #210792)

What about Info, and Address and perhaps sh_offset?

Oops, I misread slightly. I added Info and Address zeroed fields here.

This patch still doesn't support setting them to something else though.
I didn't add ShOffset or ShSize, this is out of the area of this patch and
anyways not supported either.

46 ↗

(On Diff #210792)

I used ZERO-VS-NONZERO because the test case above already explicitly defines the zero section.

142 ↗

(On Diff #210792)

I think that wasn't never intentionally supported, I added a test to check we do not crash.
Not sure if we want to check something else here.

[jhenderson]

test/tools/yaml2obj/elf-custom-null-section.yaml

48 ↗

(On Diff #211098)

I don't like "ZERO-VS-NONZERO" here, because it's not really anything to do with that. This test case is for testing that other sections can be added after the NULL section. "ZERO-VS-NONZERO" doesn't clarify that. How about "OTHER-SECTION"?

71 ↗

(On Diff #211098)

Nit: Link: 0 (be consistent with above change).

145 ↗

(On Diff #211098)

if we have

162–163 ↗

(On Diff #211098)

Perhaps worth setting some fields to different explicit values, to show that it's possible.

[grimar]

• Addressed review comments.

[jhenderson]

LGTM. Thanks!

llvm/trunk/test/tools/yaml2obj/elf-custom-null-section.yaml

+## In this test we check that can redefine the null section in the YAML.
+
+## Test the default output first.
+
+# RUN: yaml2obj --docnum=1 %s -o %t1
+# RUN: llvm-readelf --sections %t1 | FileCheck %s --check-prefix=DEFAULT
+
+# DEFAULT:      Section Headers:
+# DEFAULT-NEXT:  [Nr] Name      Type   Address          Off    Size   ES Flg Lk Inf Al
+# DEFAULT-NEXT:  [ 0]           NULL   0000000000000000 000000 000000 00 0   0  0
+# DEFAULT-NEXT:  [ 1] .symtab   SYMTAB 0000000000000000 000140 000018 18 2   1  8
+# DEFAULT-NEXT:  [ 2] .strtab   STRTAB 0000000000000000 000158 000001 00 0   0  1
+# DEFAULT-NEXT:  [ 3] .shstrtab STRTAB 0000000000000000 000159 00001b 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+
+## Now define a SHT_NULL section with fields all zeroed.
+## In this case it is equal to the section created by default.
+
+# RUN: yaml2obj --docnum=2 %s -o %t2
+# RUN: llvm-readelf --sections %t2 | FileCheck %s --check-prefix=DEFAULT
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         0
+    Info:         0
+    Address:      0x0
+
+## Check we are still able to describe other sections too.
+
+# RUN: yaml2obj --docnum=3 %s -o %t3
+# RUN: llvm-readelf --sections %t3 | FileCheck %s --check-prefix=OTHER-SECTION
+
+# OTHER-SECTION:      Section Headers:
+# OTHER-SECTION-NEXT:   [Nr] Name      Type     Address          Off    Size   ES Flg Lk Inf Al
+# OTHER-SECTION-NEXT:   [ 0]           NULL     0000000000000000 000000 000000 00 0   0  0
+# OTHER-SECTION-NEXT:   [ 1] foo       PROGBITS 0000000000000000 000180 000000 00 0   0  0
+# OTHER-SECTION-NEXT:   [ 2] .symtab   SYMTAB   0000000000000000 000180 000018 18 3   1  8
+# OTHER-SECTION-NEXT:   [ 3] .strtab   STRTAB   0000000000000000 000198 000001 00 0   0  1
+# OTHER-SECTION-NEXT:   [ 4] .shstrtab STRTAB   0000000000000000 000199 00001f 00 0   0  1
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type:         SHT_NULL
+    Name:         ''
+    Flags:        [ ]
+    AddressAlign: 0x0
+    Size:         0x0
+    EntSize:      0x0
+    Link:         0
+  - Type: SHT_PROGBITS
+    Name: 'foo'
+
+## Check we can redefine sh_size and sh_link fields of the SHT_NULL section.
+
+# RUN: yaml2obj --docnum=4 %s -o %t4
+# RUN: llvm-readelf --sections %t4 | FileCheck %s --check-prefix=REDEFINE
+
+# REDEFINE:      Section Headers:
+# REDEFINE-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# REDEFINE-NEXT:  [ 0]      NULL 0000000000000000 000000 000123 00     1   0  0
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+## The same as above, but using a number as a Link value.
+
+# RUN: yaml2obj --docnum=5 %s -o %t5
+# RUN: llvm-readelf --sections %t5 | FileCheck %s --check-prefix=REDEFINE
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: 1
+    Size: 0x123
+  - Type: SHT_PROGBITS
+    Name: .foo
+
+## Check we report an error if null section sh_link field refers to an unknown section.
+
+# RUN: not yaml2obj --docnum=6 %s -o %t6 2>&1 | FileCheck %s --check-prefix=CASE4
+
+# CASE4: error: Unknown section referenced: '.foo' at YAML section ''.
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+    Link: .foo
+
+## Check that null section fields are set to zero, if they are unspecified.
+
+# RUN: yaml2obj --docnum=7 %s -o %t7
+# RUN: llvm-readelf --sections %t7 | FileCheck %s --check-prefix=DEFAULT
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+
+## Check we do not crash if we have more than one SHT_NULL section.
+
+# RUN: yaml2obj --docnum=8 %s -o %t8
+# RUN: llvm-readelf --sections %t8 | FileCheck %s --check-prefix=MULTIPLE
+
+# MULTIPLE:      Section Headers:
+# MULTIPLE-NEXT:  [Nr] Name Type Address          Off    Size   ES Flg Lk Inf Al
+# MULTIPLE-NEXT:  [ 0]      NULL 0000000000000000 000000 000000 00 0   0  0
+# MULTIPLE-NEXT:  [ 1]      NULL 0000000000000123 000180 000020 10 A   1  2   0
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_X86_64
+Sections:
+  - Type: SHT_NULL
+  - Type:    SHT_NULL
+    Flags:   [ SHF_ALLOC ]
+    Size:    0x20
+    EntSize: 0x10
+    Link:    1
+    Info:    2
+    Address: 0x123

llvm/trunk/tools/yaml2obj/yaml2elf.cpp

 
 template <class ELFT>
 ELFState<ELFT>::ELFState(ELFYAML::Object &D) : Doc(D) {
   StringSet<> DocSections;
   for (std::unique_ptr<ELFYAML::Section> &D : Doc.Sections)
     if (!D->Name.empty())
       DocSections.insert(D->Name);
 
-  // Insert SHT_NULL section implicitly.
+  // Insert SHT_NULL section implicitly when it is not defined in YAML.
+  if (Doc.Sections.empty() || Doc.Sections.front()->Type != ELF::SHT_NULL)
     Doc.Sections.insert(
         Doc.Sections.begin(),
         llvm::make_unique<ELFYAML::Section>(
           ELFYAML::Section::SectionKind::RawContent, /*IsImplicit=*/true));
 
   std::vector<StringRef> ImplicitSections = {".symtab", ".strtab", ".shstrtab"};
   if (!Doc.DynamicSymbols.empty())
     ImplicitSections.insert(ImplicitSections.end(), {".dynsym", ".dynstr"});
 
   // Insert placeholders for implicit sections that are not
   // defined explicitly in YAML.

 template <class ELFT>
 bool ELFState<ELFT>::initSectionHeaders(ELFState<ELFT> &State,
                                         std::vector<Elf_Shdr> &SHeaders,
                                         ContiguousBlobAccumulator &CBA) {
   // Ensure SHN_UNDEF entry is present. An all-zero section header is a
   // valid SHN_UNDEF entry since SHT_NULL == 0.
   SHeaders.resize(Doc.Sections.size());
 
-  for (size_t I = 1; I < Doc.Sections.size(); ++I) {
+  for (size_t I = 0; I < Doc.Sections.size(); ++I) {
     Elf_Shdr &SHeader = SHeaders[I];
     ELFYAML::Section *Sec = Doc.Sections[I].get();
 
+    if (I == 0) {
+      if (Sec->IsImplicit)
+        continue;
+
+      if (auto S = dyn_cast<ELFYAML::RawContentSection>(Sec))
+        if (S->Size)
+          SHeader.sh_size = *S->Size;
+
+      if (!Sec->Link.empty()) {
+        unsigned Index;
+        if (!convertSectionIndex(SN2I, Sec->Name, Sec->Link, Index))
+          return false;
+        SHeader.sh_link = Index;
+      }
+      continue;
+    }
+
     // We have a few sections like string or symbol tables that are usually
     // added implicitly to the end. However, if they are explicitly specified
     // in the YAML, we need to write them here. This ensures the file offset
     // remains correct.
     if (initImplicitHeader(State, CBA, SHeader, Sec->Name,
                            Sec->IsImplicit ? nullptr : Sec))
       continue;