This is an archive of the discontinued LLVM Phabricator instance.

[yaml2obj] - Automatically assign sh_addr for allocatable sections.
ClosedPublic

Authored by grimar on Feb 18 2020, 6:09 AM.

Details

Summary

I've noticed that it is not convenient to create YAMLs from
binaries (using obj2yaml) that have to be test cases for obj2yaml
later (after applying yaml2obj).

The problem, for example is that obj2yaml emits "DynamicSymbols:"
key instead of .dynsym. It also does not create .dynstr.
And when a YAML document without explicitly defined .dynsym/.dynstr
is given to yaml2obj, we have issues:

  1. These sections are placed after non-allocatable sections (I've fixed it in D74756).
  2. They have VA == 0. User needs create descriptions for such sections explicitly manually to set a VA.

This patch addresses (2). I suggest to let yaml2obj assign virtual addresses by itself.
It makes an output binary to be much closer to "normal" ELF.
(It is still possible to use "Address: 0x0" for a section to get the original behavior
if it is needed)

Diff Detail

Event Timeline

grimar created this revision.Feb 18 2020, 6:09 AM
Herald added a project: Restricted Project. · View Herald Transcript

I think this is the correct approach. It will definitely simplify tests.

Nit: in the description, use sh_addr instead of VA because sh_addr may be more familiar to readers.

llvm/lib/ObjectYAML/ELFEmitter.cpp
222

Section *

396

Add a test that ShSize (overriding sh_size, usually used for abnormal values) does not affect LocationCounter.

526

Delete. I think non-SHF_ALLOC sections follow the same rule.

530

getValueOr

llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
32

0x1000 is fine but probably not the best.

If .text.any.addr is included by a PT_LOAD, one will expect sh_offset%p_align = sh_addr%p_align (a PT_LOAD's p_align is usually 0x1000 (maxpagesize) on x86).

One approach is to specify sh_addralign and set sh_addr to sh_addralign plus a multiple of p_align.

Address: 0x1200
AddressAlign: 0x200
MaskRay added inline comments.Feb 18 2020, 5:26 PM
llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
55

Add a ShSize test somewhere.

65

Add a non-SHF_ALLOC section (e.g. .rela.debug_info) to test this rule applies on non-SHF_ALLOC sections.

grimar updated this revision to Diff 245384.Feb 19 2020, 5:22 AM
grimar marked 11 inline comments as done.
grimar retitled this revision from [yaml2obj][WIP] - Automatically assign VA for allocatable sections. to [yaml2obj][WIP] - Automatically assign sh_addr for allocatable sections..
  • Addressed review comments.
llvm/lib/ObjectYAML/ELFEmitter.cpp
526

Not sure I follow. We should not assign sh_addr for non-allocatable sections. Isn't?
In LLD we just drop counter to 0:

void LinkerScript::assignOffsets(OutputSection *sec) {
  if (!(sec->flags & SHF_ALLOC))
     dot = 0;

GNU linkers do not set addresses to sh_addr for non-allocatable sections too.
Why should yaml2obj do different thing?

530

getValueOr, which is a method of Optional<>? There is no optional here. I am testing sh_addralign for 0 because
alignTo expects it to be !=0:

inline uint64_t alignTo(uint64_t Value, uint64_t Align, uint64_t Skew = 0) {
  assert(Align != 0u && "Align can't be 0.");
...
llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
32

I do not think it is a problem here. yaml2obj do just what it do: produce an output from a given description.
This test just documents the behavior of assigning addresses. It does not have intention to create an
ELF that has no issues.

And so it probably does not make much sence to work hard and put 0x1200 and AddressAlign here.
At least because it works only for .text*. See, the offset of .data* is broken:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00     0   0  0
  [ 1] .text.any.addr    PROGBITS        0000000000001200 000200 000003 00   A 0   0 512
  [ 2] .text.shsize      PROGBITS        0000000000001203 000203 001234 00   A 0   0  0
  [ 3] .text.align       PROGBITS        0000000000001300 000300 000004 00   A 0   0 256
  [ 4] .data.any.addr    PROGBITS        0000000000002000 000304 000001 00   A 0   0  0
  [ 5] .data.after.fill  PROGBITS        0000000000002101 000405 000001 00   A 0   0  0
  [ 6] .data.return.back PROGBITS        0000000000001500 000406 000001 00   A 0   0  0
  [ 7] .data.return.back.foo PROGBITS    0000000000001501 000407 000000 00   A 0   0  0
  [ 8] .dynsym           DYNSYM          0000000000001508 000408 000018 18   A 9   1  8
  [ 9] .dynstr           STRTAB          0000000000001520 000420 000001 00   A 0   0  1
  [10] .strtab           STRTAB          0000000000000000 000421 000001 00 0   0  1
  [11] .shstrtab         STRTAB          0000000000000000 000422 000093 00 0   0  1

Yes, it is possible to adjust other sections too, add a few properties to make them follow the rule, but that requires adding
additional fields, and I am not sure I see how it is useful in the context of this test case.
I see why it can be bad though :) It would make the test case larger than it can be.

65

We have .strtab and .shstrtab added implicitly. They are non-SHF_ALLOC.
I think I just do not understand what exactly you want me to test for them?

Now I am going to fix all test cases that are failing and remove the [WIP] status after that update.

The code change looks good to me now.

llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
65

.shstr and .shstrtab are non-SHF_ALLOC. I think it is fine not adding another non-SHF_ALLOC section.

grimar updated this revision to Diff 245605.Feb 20 2020, 2:01 AM
  • Fixed test cases.
  • Disabled auto VA assign for sections in relocatable objects.
grimar retitled this revision from [yaml2obj][WIP] - Automatically assign sh_addr for allocatable sections. to [yaml2obj] - Automatically assign sh_addr for allocatable sections..Feb 20 2020, 2:02 AM
grimar edited the summary of this revision. (Show Details)
MaskRay added inline comments.Feb 20 2020, 2:08 PM
llvm/lib/ObjectYAML/ELFEmitter.cpp
528

The two conditions can be combined.

sh_addr represents the address in the memory image of a process. Sections in a relocatable object file or non-allocatable sections do not need sh_addr assignment.

llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
1

Test that yaml2obj automatically assigns sh_addr to allocatable sections for ET_EXEC/ET_DYN files.

(ET_CORE is possible but they don't usually have the section header table.)

grimar updated this revision to Diff 245799.Feb 21 2020, 12:38 AM
grimar marked 2 inline comments as done.
  • Addressed review comments.
MaskRay accepted this revision.Feb 21 2020, 9:35 AM
This revision is now accepted and ready to land.Feb 21 2020, 9:35 AM
This revision was automatically updated to reflect the committed changes.

Sorry, I didn't get a chance to look at this earlier. It's been a pretty hectic few days. Two comment-related nits from me.

Also, not worth fixing them now, but in some of those tests, we probably didn't need to check the actual address value at all, so they could have been replaced with {{.*}} or similar.

llvm/lib/ObjectYAML/ELFEmitter.cpp
712–714

Nit: this comment looks stale now.

llvm/test/tools/yaml2obj/ELF/section-address-assign.yaml
87

I think you mean "go back" not "return back".

grimar marked 2 inline comments as done.Feb 25 2020, 2:30 AM

Two comment-related nits from me.

Thanks, I've addressed them in https://reviews.llvm.org/rG157b3d505f6

Also, not worth fixing them now, but in some of those tests, we probably didn't need to check the actual address value at all, so they could have been replaced with {{.*}} or similar.

In some of them the check for address seems to be useful thing. Like in llvm-readobj tests where we now checking that we dump addresses properly.