This is an archive of the discontinued LLVM Phabricator instance.

[ELF] - Redesign of program headers creation code.
AbandonedPublic

Authored by grimar on Jan 26 2016, 3:42 AM.

Details

Reviewers
ruiu
rafael
Summary

During review of "[ELF] Support PHDRS command" (http://reviews.llvm.org/D15191)
it was mentioned that current code to create a program header it's not easy to read.
And solution could be to have a class or a function that create a list of program header fields,
and associate OutputSections to zero or more program header field. I am agree with that.

So this patch do this. It introduces new method scanHeaders() which builds the program
headers map. This map contains the list of all headers, its flags and output sections lists
associated to each. assignAddresses() was changed to use that.

This prepares code for linker script implementation.

Diff Detail

Event Timeline

grimar updated this revision to Diff 45967.Jan 26 2016, 3:42 AM
grimar retitled this revision from to [ELF] - redesign of program headers creation code..
grimar updated this object.
grimar added reviewers: ruiu, rafael.
grimar added subscribers: llvm-commits, grimar.
grimar updated this revision to Diff 45973.Jan 26 2016, 4:28 AM
  • Last minute fixes
rafael edited edge metadata.Jan 26 2016, 2:57 PM

The idea is that with linker script scanHeaders is not used and instead we use the data from the script to construct a mapping and the rest of the code is not changed? If so, I like the general direction.

ELF/Writer.cpp
103

Why std::list and not std::vector?

106

Thin can be a std::list<Phdr> or a std::vector<std::unique_ptr<Phdr>>, no?

1052

Add a small comment.

1056

Not sure if having this if in here is actually more readable.

1143

Do you really need this flag? Looks like the only special case is that the first load starts at 0, not at the start of its first section.

Is that the same behaviour one gets with linker scripts?

ruiu added inline comments.Jan 26 2016, 5:23 PM
ELF/Writer.cpp
1122–1130

Do you think you can reduce number of variables and boolean flags used in this function by separating into small functions?

The fundamental reason why the previous code was not easy to read is because too many flags are entangled. If the code makes a decision based on n boolean flags, the possible combination is 2^n, which is exponential. Moreover, some variables are updated in the middle of a process, that made it hard to follow the logic. (If a variable gets a new value in one path but don't in the other, you'll get 2^n combinations, again.) We'd like to use less information. I think it is acceptable to iterate over the same data structure more than once because I don't think this is a performance-critical path.

grimar updated this revision to Diff 46124.Jan 27 2016, 5:44 AM
grimar retitled this revision from [ELF] - redesign of program headers creation code. to [ELF] - Redesign of program headers creation code..
grimar edited edge metadata.
grimar marked 5 inline comments as done.
  • Review comments addressed.

The idea is that with linker script scanHeaders is not used and instead we use the data from the script to construct a mapping and the rest of the code is not changed? If so, I like the general direction.

Yes, script could use some special version of scanHeaders(), like scanHeadersLinkerScript() which will generate some structure (header map) with all required parameters based on options parsed by LS. And then assignAddresses should be able to handle it without changes in other code.
This is of course initial implemention and things are subject to change during work on LS (assignAddresses will need tweaking I think), but I think that whole approach should work.

ELF/Writer.cpp
103

Because I dont know the amount of sections to be assigned to header. So I cant reserve() and dont want multiple reallocates that can happen when using the vector. Also vector can bring overhead here because of reserving memory upfront.
And I also dont need any indexed access so just see no reasons to use vector here.

106

Changed to std::list<Phdr>.

1056

Changed AddIf -> AddHdr, moved condition check outside.

1122–1130

Was able to get rid of some variables. Also separated TLS adresses asigning logic into another method. Now there are 3 passes logically separated.

1143

assignAddresses() definetely will requre some tweaking for linker script logic I am sure, currently it is the base concept that assigns addresses in according to some pre-generated map.
It seems that first load will be special anyways, just the parameters will be different. That way probably parameters from linkersript will be used for first load. During implementation of linker script it will be more clear what exactly changes required.

I placed that flag to header description and also moved the code from inside the loop. That added few lines but probaby looks more readable now.

emaste added a subscriber: emaste.Jan 27 2016, 5:57 AM

I can add that I am working on "[ELF] Support PHDRS command" (http://reviews.llvm.org/D15191) reimplementation now. And it seems that with this patch it is going OK now.

grimar updated this revision to Diff 46249.Jan 28 2016, 1:52 AM
  • Introduced PhdrsContext which contains helpfull info found during scanHeaders() call like index of TLS, Relro phdrs.
  • Get rid of excess assignAddresses() pass. Now there are two left, which seems reasonable amount.
grimar updated this revision to Diff 46268.Jan 28 2016, 7:00 AM
  • bool FirstLoad -> size_t FirstLoadId
  • Assign phdr's flags and types only at the very end.
  • Cleanup + added comments.
ruiu added inline comments.Jan 28 2016, 2:58 PM
ELF/Writer.cpp
46

This needs comments.

50

Please do not use std::list unless you have a reason to do so because it is generally slower than std::vector.

52

This needs comments, too. If a purpose of a struct or class is not obvious, please describe.

58

What is this for?

1061

You are not using a return value.

1130

Please do not use auto unless type is obvious by RHS.

1150

Hmm, I'm honestly not sure if this is net positive... This code hasn't changed that much, and this is what I wanted to simplify.

grimar updated this revision to Diff 46371.Jan 29 2016, 3:30 AM
grimar marked 7 inline comments as done.
  • Addressed Rui's comments.
ELF/Writer.cpp
50

Output section amount that should be attached to each phdr is unknown is the main reason for this.
If there are many possible output sections then multiple reallocations can happen if using vector here. I have no argument for calling reserve().
If in opposite we assume that output sections count is little then there is no any perfomance hit during iterations in this case either, but we win a bit of memory probably that is not forwardly allocated as vector do.

58

Removed using, it was used before but no more needed.

1061

Actually I am. In one place, but it is very useful there:

uint32_t LoadType = (Config->EMachine == EM_AMDGPU) ? getAmdgpuPhdr(Sec)
                                                    : (uint32_t)PT_LOAD;
Load = AddHdr(LoadType, NewFlags);
Flags = NewFlags;
1130

Done.

1150

Generally this patch is not about simplification, main aim to have it is to split address assigning into scanHeaders() and assignAddressed() methods, what is needed to implement initial variant of linker script PHDRS I believe. After some futher tweaks for assignAddressed() it should be possible to use some scanHeadersLinkerScript() to affect output. I thinks it is good that code looks close to initial but at fact have significantly more possible functionality now.

But I cant agree that it was not simplified.
For example initial code has few local variables/flags in one place that affected execution flow in one huge method:

int PhdrIdx = 0;
Elf_Phdr *Interp = nullptr;
Elf_Phdr GnuRelroPhdr = {};
Elf_Phdr TlsPhdr{};
bool RelroAligned = false;
uintX_t ThreadBssOffset = 0;

This code has only one of them left:

bool RelroAligned = false;

And TLS address assigning was moved to separate method, what simplified logic a bit.

grimar updated this revision to Diff 46517.Feb 1 2016, 2:09 AM
grimar updated this object.

Simplified:

  • All specific headers (PT_DyNAMIC, PT_INTERP etc) are processed at once in universal way.
  • First pass in assignAddressed now handles all PT_LOAD segments and assigns addresses to all sections including TLS ones.
ruiu edited edge metadata.Feb 1 2016, 6:21 PM

This does not compile, too.

grimar updated this revision to Diff 46627.Feb 2 2016, 1:20 AM
grimar edited edge metadata.
  • Fixed compilation error and warning under linux.
  • Rebased.
grimar updated this revision to Diff 46646.Feb 2 2016, 6:30 AM
  • Minor refactor: placed some logic from assignAddresses() to newly created setupSecVA() and setupSecFileOff() methods.
grimar updated this revision to Diff 46773.Feb 3 2016, 5:02 AM
  • Moved out PT_LOAD header creation from assignAddress() method.
  • Resynced with head.

I have applied this locally and I have some ideas on how to simplify
it. I will upload a proposed change after lunch.

Cheers,
Rafael

grimar added a comment.Feb 5 2016, 3:47 PM

I have applied this locally and I have some ideas on how to simplify
it. I will upload a proposed change after lunch.

Cheers,
Rafael

Interesting to see. Thanks !

grimar added a comment.Feb 6 2016, 2:38 PM

This is what I have so far. There is more work to be done, but the idea is that

  • scanHeaders does nothing more than map section to program headers.
  • We then find out which sections need extra alignment because of page

boundaries.

  • We assign address and offset to each *section*, which is now a simple loop
  • We use the above info to assign sizes and address to each program header.

Things I want to try before considering this a real patch

  • Create dummy sections for the file and program headers. This should

avoid the hacks regarding the first PT_LOAD.

  • Store just first and last section. A program header is a contiguous range.
  • Have the Phdr structure be just a Elf_Phdr and the first and last section.

Cheers,
Rafael

Ok.
If you don't mind I will try your suggestions and update the patch then ?

George.

grimar abandoned this revision.Feb 11 2016, 3:25 AM

Another version of that by Rafael Ávila de Espíndola was commited as r260453 (http://reviews.llvm.org/D16991).