This is an archive of the discontinued LLVM Phabricator instance.

[llvm-objcopy] Add support for nested and overlapping segments
ClosedPublic

Authored by jakehehrlich on Aug 9 2017, 4:24 PM.

Diff Detail

Repository
rL LLVM

Event Timeline

jakehehrlich created this revision.Aug 9 2017, 4:24 PM
jhenderson edited edge metadata.Aug 30 2017, 2:54 AM

Lots of small nits, and one or two slightly larger issues to address here. I like the overall approach though.

I'm not all that happy with the level of test coverage available, but at the same time, I'm not sure that there's much that can be done without unit-testing a lot of this, or creating large numbers of virtually identical Lit tests, which seems a little excessive.

tools/llvm-objcopy/Object.cpp
170

Nit: full stop.

171

segmentOverlappsSegment -> segmentOverlapsSegment

174–175

I don't think this is quite right - in the case of two adjacent segments (i.e. where the end of one and the start of the next are the same), the second ends up being treated as a child of the first. I think the second clause should be strictly greater than.

206–207

Couple of typos in this comment: 1) though -> through; 2) match up nested segments up. -> match up segments.

210

"will be a child of itself" should probably be "will overlap with itself", to match the function name.

213

cononical -> canonical

214

Nit: full stop.

222

I think there's one case missing here - we don't want the case where two segments with identical offsets and file sizes to be parents of each other. In this situation, we also can't simply say something like "if (Parent->ParentSegment == Child) do nothing", you could end up with segment 1 being parent of segment 2 which is parent of segment 3 which is parent of segment 1. I think in that situation, the segment that is first in the table should be treated as the parent of the other two.

232

Nit: unnecessary blank line.

401

properlly -> properly

433

parrent -> parent

435

Full stop.

437

Could you rename PSeg to Parent, please?

438

Should this be:
Segment->Offset = PSeg->Offset + Segment->OriginalOffset - PSeg->OriginalOffset;
?

Lots of small nits, and one or two slightly larger issues to address here. I like the overall approach though.

I'm not all that happy with the level of test coverage available, but at the same time, I'm not sure that there's much that can be done without unit-testing a lot of this, or creating large numbers of virtually identical Lit tests, which seems a little excessive.

I'm not that happy with test coverage either. You caught a mistake that wasn't just a little mess sup on my part (Segment->OriginalOffset - PSeg->OriginalOffset vs Segment->Offset - PSeg->Offset). I think it's pretty clear that the tests are not enough here. There are lots of branches in this code that likely aren't being checked as well. I don't know of a way to add unit tests in LLVM.

Are you aware of some way of adding unit tests in llvm?

Are you aware of some way of adding unit tests in llvm?

We have unittests in llvm/unittests. We generally prefer regression tests for various reasons, but the option exists if you need it.

  1. Switched to using Index instead of FileSize for parent determination. FileSize was only being used to disambiguate the case where two segments had the same offset. Index does that better for reasons James pointed out.
  2. Fixed bug where Offset was being used instead of OriginalOffset
  3. Fixed lots of typos

Are you aware of some way of adding unit tests in llvm?

As @efriedma mentioned, there are plenty of unit tests in llvm/unittests (I don't know if you're developing on Windows or Linux, but at least in the Visual Studio solution, most of the projects under the "Test" group are unit test projects). I have a little experience with adding unit tests to existing tests, but no experience with setting up a whole new set of unit tests for a new program, such as llvm-objcopy. I do believe that it would require moving all the code that we want to test this way into a separate support library. In general, my experience has been that writing unit tests has been easier to generate sufficient coverage than regression tests, but that was with an internal non-LLVM product, so the principle may not so readily apply here.

I think if the alternative is large numbers of pre-built binaries, I think we have to unit test this, if we want sensible coverage. However, I also think this a broader problem with llvm-objcopy, so I think it's not part of this change. However, I do think it's something that should be looked at sooner rather than later (from experience, the bigger a project grows, the harder it is to unit test, because certain design decisions that make unit testing hard get harder to undo). This all might be something to ask about on the mailing list (i.e. how to start unit testing the program), as I haven't discovered any documentation on it anywhere.

I'm happy with the code as is now, but I'm not sure how we can exercise the parts of the code where the bugs were from a Lit perspective until we add the ability to actually do things to the ELF to llvm-objcopy (e.g. strip sections). I'd therefore like the unit testing discussion/investigation to happen before I accept this review (although if you don't think it's worth it, I could probably be persuaded otherwise).

  1. Added condition to check for PT_GNU_STACK in layout because the new layout algorithm threw and error in the case
  2. Added 3 test to try and get as much coverage of the O(n^2) loop over program headers. I tried to make sure every branch was covered.

Turning this into a library isn't really an option right now. Petr Hosek talked with Rui about the possibility of that in future. Namely LLD's synthetic sections and my sections look a lot alike so I think deduplicating those and merging them into a library makes a lot of sense at some point in the future but not right now. If this stuff gets turned into a library I think that's the form it will take.

I think we should probably settle for adding a few tests (possibly more, please recommend more if you can think of any) here and then wait to add some stripping capability next. After stripping is implemented we can write better tests. I plan on adding some stripping capabilities right after dynamic stuff has been submitted. I'll start writing code for it soon and put it up for review as soon as possible.

What kind of stripping would be most useful for testing at first? I think removing a section by name and then removing a symbol by name would be best. That way we can tailor tests to see what happens when specific sections/symbols are removed. After that we can implement more useful types of stripping that remove multiple sections and symbols at once.

So my proposal is the following:

  1. Compromise between adding a bunch of very similar .test files and unit testing by adding a few more .test files (please recommend more) and waiting for better tests to come up after we have section and symbol removal
  2. Write section removal by name next.
  3. Write a collection of tests using section removal works well and to get more test coverage
  4. Write symbol removal by name after that.
  5. Write a collection of tests using symbol removal to get better coverage.
  6. Add more advanced kinds of stripping like --strip-all and --strip-debug
jhenderson added a comment.EditedSep 7 2017, 2:08 AM

I think we should probably settle for adding a few tests (possibly more, please recommend more if you can think of any) here and then wait to add some stripping capability next. After stripping is implemented we can write better tests. I plan on adding some stripping capabilities right after dynamic stuff has been submitted. I'll start writing code for it soon and put it up for review as soon as possible.

What kind of stripping would be most useful for testing at first? I think removing a section by name and then removing a symbol by name would be best. That way we can tailor tests to see what happens when specific sections/symbols are removed. After that we can implement more useful types of stripping that remove multiple sections and symbols at once.

So my proposal is the following:

  1. Compromise between adding a bunch of very similar .test files and unit testing by adding a few more .test files (please recommend more) and waiting for better tests to come up after we have section and symbol removal
  2. Write section removal by name next.
  3. Write a collection of tests using section removal works well and to get more test coverage
  4. Write symbol removal by name after that.
  5. Write a collection of tests using symbol removal to get better coverage.
  6. Add more advanced kinds of stripping like --strip-all and --strip-debug

I think what you've proposed here sounds reasonable. Section stripping is something I've made use of in the past, but not symbol stripping, so I think section stripping should be first, definitely. It would also allow us to test this area (nested segments) more directly as well.

As for suggested tests, I think you've done a pretty good job of covering the cases I can think of. One I didn't see, so might be missing, was two adjacent segments (i.e. the end of one is the same value as the start of the next), perhaps with different alignments, so that the first can move but not the second (or they can both move, but are not tied together, so one moves further than the other). That would test the second half of "segmentOverlapsSegment" I think. It might just be part of one of the other tests. As there are several tests now that test similar, but slightly different cases, I think you should add a comment in each test to describe what exactly is being tested (e.g. adjacent segments, segments that are identical, chains of segments that partially overlap), essentially describing why that particular case is interesting. For example, in the test I've suggested, the comment would read something like "Check the case where two non-empty segments are adjacent in the file, i.e. the end of one is the start of the next. In this case, the two should move independently of each other."

test/tools/llvm-objcopy/same-segment.test
1

I'd call this test identical-segment.test (same-segment implies that two different things refer to one segment).

test/tools/llvm-objcopy/tripple-overlap.test
1

tripple-overlap.test -> triple-overlap.test

tools/llvm-objcopy/Object.cpp
432

Why is PT_GNU_STACK special?

Added test and fixed test names

tools/llvm-objcopy/Object.cpp
432

For the purposes of skipping it here it is special because it has no alignment and it's offset must be zero. In general segments should have non-zero alignment.. It's more generally special however. All of it's fields are zero (including address, offset, and size) except for the flags. It isn't nested in another loadable segment but it also doesn't cover any section.

I skip it here because it needs to maintain a zero offset and it causes align to fail (when I originally wrote this diff, the new layout algorithm wasn't a part of it so I didn't have this issue)

One of the renamed tests needs another tweak to the name, and another of the tests still has a typo in its name.

test/tools/llvm-objcopy/identical-segment.test
1 ↗(On Diff #114430)

Sorry, should be identical-segments.test (plural).

2 ↗(On Diff #114430)

based in -> based on.

test/tools/llvm-objcopy/tripple-overlap.test
4

the case which -> the case where

tools/llvm-objcopy/Object.cpp
432

LLD can emit other segments with zero in every field. I have a linker script, for example, that requests a PT_INTERP segment, but LLD does not assign anything to it, so the segment is empty. Every field apart from the type is zero. Empty segments don't need an alignment or address, so I think the check should be for any empty segments at offset zero.

  1. Fixed typos and test names
  2. Made recommended change to skipping certain segments. I used MemSize to check that the size is zero because FileSize could be zero while MemSize would not be. If a segment covers only SHT_NOBITS read only sections then it's offset is technically free to be whatever I believe and it would have to have FileSize be zero. In practice it will be something more sensible but it seemed right t cover this case.

tripple-overlap.test still needs renaming!

I think it may be possible to have segments with zero MemSize but non-zero FileSize. I think it's unlikely to happen that they appear at offset zero, but I wouldn't want to guarantee it. As such, these should probably be considered in the algorithm as well, rather than being skipped. If they overlap another later segment, then the latter must stay relative to the former. We should have tests for these two cases as well, possibly independent of the completely empty segment.

Sorry about not getting that changed faster. That's just not ok. I apologize.

As for the FileSize < MemSize issue. The standard states "The file size may not be larger than the memory size" here: http://www.sco.com/developers/gabi/2012-12-31/ch5.pheader.html

I haven't seen anything producing any such binaries. Also LLD considers it a bug when it happens so that's at least one linker that promises to never do it. I think I'd rather throw an error on read in if this happens. Would that be ok?

In case you're cool with just throwing an error in the case that p_filesz > p_memsz I've gone ahead an added that as well. Adding a test is tricky for two reasons 1) It's tricky to make a binary to trigger this case and 2) I would have to upload a binary to do so.

Sorry about not getting that changed faster. That's just not ok. I apologize.

As for the FileSize < MemSize issue. The standard states "The file size may not be larger than the memory size" here: http://www.sco.com/developers/gabi/2012-12-31/ch5.pheader.html

I haven't seen anything producing any such binaries. Also LLD considers it a bug when it happens so that's at least one linker that promises to never do it. I think I'd rather throw an error on read in if this happens. Would that be ok?

No problem, it happens. Sorry for getting back to you slowly - I've been on annual leave for a few days.

That standard quote only refers to PT_LOAD segments. Other segment types are not constrained by this, so the new error will spuriously catch such cases. We cannot rely on the linker preventing this case for other segment types, because it is entirely reasonable for specific targets to have segments that are not loaded on the target, so don't need address or memory size allocated - see the "NOLOAD" linker script directive, for example. I could also imagine a target which does not assign addresses for the ELF header/program header table, but theoretically they could be assigned to a (non-loaded) segment.

Ultimately though, I think this is all irrelevant - the actual problem here is the use of alignTo with zero alignment, but according to the ELF spec, a value of 0 or 1 means no alignment, so if we see an align of zero, we should align as if aligning to 1, I think.

Right, the alignment trick makes perfect sense. Also I removed the error because, you're right, it only makes sense for PT_LOAD. I still think there might be an issue with some special nearly "all zero" sections but it's kind of hard to figure out how exactly they should be handled. Anything that has an offset of zero and a file size of zero will currently be handled correctly so I can't think of such a case that isn't handled correctly.

This revision is now accepted and ready to land.Sep 19 2017, 1:25 AM
This revision was automatically updated to reflect the committed changes.