This is an archive of the discontinued LLVM Phabricator instance.

[llvm-objcopy] Add support for removing sections
ClosedPublic

Authored by jakehehrlich on Sep 25 2017, 2:14 PM.

Details

Summary

This change adds support for removing sections using the -R field (as GNU objcopy does as well). This change should let us add many helpful tests and is a proper stepping stone for adding more general kinds of stripping.

Diff Detail

Repository
rL LLVM

Event Timeline

jakehehrlich created this revision.Sep 25 2017, 2:14 PM
jhenderson edited edge metadata.Sep 26 2017, 6:35 AM

There are no tests here for what happens to segments. Two cases I can think of: sections between segments, sections in segments.

test/tools/llvm-objcopy/reloc-error-remove-symtab.test
8 ↗(On Diff #116611)

Minor quibble, but this should probably be ET_REL, since it's a relocatable object file.

test/tools/llvm-objcopy/remove-section.test
3 ↗(On Diff #116611)

Could you please check the ELF header here, as well, and show that e_shnum is correct.

tools/llvm-objcopy/Object.cpp
157 ↗(On Diff #116611)

"referenced by the symbol table" -> "referenced by a symbol table", or "referenced by the symbol table" + Name

299 ↗(On Diff #116611)

Why has this changed?

635 ↗(On Diff #116611)

This function strikes me as overly complex for what is doing. I think the SecsToRemove set and the second stable_partition can be removed by changing the first one's predicate to test both the section itself for removal, and if it is a relocation section, the section it refers to:

if (ToRemove(*Sec))
  return false;
if (auto RelSec = dyn_cast<DefinedOnRelocSection>(Sec.get()))
  return !ToRemove(*RelSec->SecToApplyRel);
return true;
652 ↗(On Diff #116611)

"... throw errors here." -> "... emit an error here instead."

tools/llvm-objcopy/Object.h
199 ↗(On Diff #116611)

What is the purpose of this new class? Can't we just use RelocationSectionBase?

tools/llvm-objcopy/llvm-objcopy.cpp
59 ↗(On Diff #116611)

objcopy also allows --remove-section as an alias. It also allows specifying the option multiple times to strip multiple sections.

72 ↗(On Diff #116611)

Why a lambda here? Also, the explicit return type is unnecessary.

jakehehrlich added inline comments.Sep 26 2017, 6:37 PM
tools/llvm-objcopy/Object.cpp
299 ↗(On Diff #116611)

Oh whoops. I forgot to mention this. It was wrong before. I was using StringTableSection for what under the hood was actually a Section. This was fine because I wasn't using anything that wasn't in SectionBase here. Additionally dynamic string tables have SHT_STRTAB so the cast went though just fine. I think to avoid this mistake in the future I should probably add a check that StringTableSection is not allocated.

tools/llvm-objcopy/llvm-objcopy.cpp
59 ↗(On Diff #116611)

I'll add --remove-section as an alias. Would you take making ToRemove a cl::list as a todo?

72 ↗(On Diff #116611)

What alternative are you proposing? Or are you just asking why this method takes a lambda? That's because many other conditions for removal will be added. This interface should cover all of them however.

  1. Added --remove-section alias (actually, I made --remove-section the main parameter and -R the alias)
  2. Simplified removal algorithm as recommended by James
  3. Added check for e_shnum to a test.
jakehehrlich marked 3 inline comments as done.Sep 26 2017, 7:12 PM
jakehehrlich added inline comments.
tools/llvm-objcopy/Object.h
199 ↗(On Diff #116611)

It lets me cast both to DefinedOnRelocSection so that I can handle both cases uniformly.

I uploaded the wrong diff. This is the right one.

jhenderson added inline comments.Sep 27 2017, 1:27 AM
test/tools/llvm-objcopy/remove-section.test
22 ↗(On Diff #116750)

As things stand, this test only shows that .test2 is not between .test1 and .test3. I think there are two ways to change this: 1) CHECK-NOT: .test2 between every block of CHECKs or 2) CHECK for each section (not just .test1 and .test3) by name specifically.

tools/llvm-objcopy/Object.cpp
299 ↗(On Diff #116611)

Ok, that sort of makes sense. In that case, however, could you make this change in a separate review/commit, please, as it's not really related to removing sections.

tools/llvm-objcopy/Object.h
216 ↗(On Diff #116750)

I think this member should now be deleted?

199 ↗(On Diff #116611)

Ah, okay, I forgot that we had templated RelocationSectionBase. In that case, could we rename them, because I do not understand what "DefinedOnRelocSection" is supposed to represent - what is defined here?

I would seriously consider renaming DefinedOnRelocSection "RelocationSectionBase", since that is the base class for relocation sections, and then maybe RelocationSectionBase to "RelocSectionWithSymtabBase" or similar.

tools/llvm-objcopy/llvm-objcopy.cpp
59 ↗(On Diff #116611)

Yes, that's fine with me.

72 ↗(On Diff #116611)

The alternative would be to pass in the section name(s) to the function and test them there, but if the plan is to add other ways to remove sections, then this is fine with me. The explicit return type is still unnecessary, I believe.

jakehehrlich updated this revision to Diff 116859.EditedSep 27 2017, 12:16 PM
  1. Removed "-> bool"
  2. Renamed relocation class hierarchy as recommended
  3. Reverted fixing the StringTableSection vs SectionBase issue. I will upload another patch to fix that
  4. Removed duplicate member

Not sure if this got missed from my earlier comments, but I think you need tests for removing sections when there are segments present, to show that the addresses are unchanged, even if the offset changes, and to show the restriction on moving sections within segments etc. I've suggested changing the remaining yaml2obj-based tests to use ET_REL, to crudely indicate these are missing segments (feel free to ignore those comments), and then any new tests involving segments would use ET_EXEC.

test/tools/llvm-objcopy/remove-section-with-symbol.test
9 ↗(On Diff #116859)

ET_EXEC -> ET_REL

test/tools/llvm-objcopy/remove-section.test
9 ↗(On Diff #116859)

ET_EXEC -> ET_REL

tools/llvm-objcopy/Object.h
213 ↗(On Diff #116859)

Could we add a comment to explain why we need separate RelocationSectionBase and RelocationSectionWtihSymtabBase classes, please.

Also, I think we need to prevent users directly instantiating RelocationSectionBase somehow, maybe by giving it a protected constructor?

jhenderson added inline comments.Sep 28 2017, 1:46 AM
tools/llvm-objcopy/llvm-objcopy.cpp
62 ↗(On Diff #116859)

Alias for -quiet -> copy and paste error?

(Spotted this when looking at D38335!)

  1. Changed test ELF types to ET_REL were appropriate
  2. Added segment test that I forgot to add
  3. Added a comment describing the need for the strange hierarchy around relocations.

I think we could use one more test, if it is possible to construct: a section between two segments that is then removed, allowing the second segment to move in the file (but not in memory):

| Section 1 | Section 2 | Section 3 |
| Segment 1 |-----------| Segment 2 |

Don't worry about it, if it's not possible to sensibly, as I think this an edge case that is unlikely to be seen in most situations.

test/tools/llvm-objcopy/segment-test-remove-section.test
4 ↗(On Diff #117027)

it's -> its

test/tools/llvm-objcopy/symtab-error-on-remove-strtab.test
11 ↗(On Diff #117027)

by the symbol table .symtab

tools/llvm-objcopy/Object.h
200 ↗(On Diff #117027)

However some... -> However, some

206 ↗(On Diff #117027)

the other handles... -> another which handles

221 ↗(On Diff #117027)

Too many full stops this time!

222 ↗(On Diff #117027)

two relocation types -> two symbol table types

Two relocation types implies to me "Rel" and "Rela" relocations, which isn't what is relevant here. What's relevant is the symbol table being referred to.

I think we could use one more test, if it is possible to construct: a section between two segments that is then removed, allowing the second segment to move in the file (but not in memory):

| Section 1 | Section 2 | Section 3 |
| Segment 1 |-----------| Segment 2 |

Don't worry about it, if it's not possible to sensibly, as I think this an edge case that is unlikely to be seen in most situations.

A test like this is possible but I'm not sure it will test the desired behavior.

If an input file starts out like this

| Section 1 | Section 2 | Section 3 |
| Segment 1 |-----------| Segment 2 |

Then it will end up like this after copy (note that Section 2 is not removed) because of the new layout algortihm

| Section 1 | Section 3 | Section 2 | 
| Segment 1 | Segment 2 | ---------- |

I can add the test but it's worth pointing out that the reason the segments get placed together has nothing to do with Section 2 being removed or not removed.

  1. I was playing with the recommended test that I said would not prove anything and it actually exposed a bunch of issues. I submitted https://reviews.llvm.org/D38436 for review and that should be landed before this is landed.
  2. I made the recommended grammatical changes
  3. The new test in https://reviews.llvm.org/D38436 named "segment-shift.test" is exactly the test case James recommended minus the section removal.

I can add the test but it's worth pointing out that the reason the segments get placed together has nothing to do with Section 2 being removed or not removed.

I see your point. However, I think it is still worth adding a remove-section version of the test you added in D38436, as it will demonstrate that the section is removed properly, and that no space is allocated for it at the end. It means that even if the layout algorithm changes, the correct behaviour is still tested for this case. Ideally, I'd then like to see that the trailing sections (.symtab, .strtab etc) appear directly after the final segment, without any gap caused by the removed section.

test/tools/llvm-objcopy/segment-test-remove-section.test
2 ↗(On Diff #117249)

There's another it's -> its here.

  1. Added test
  2. Fixed "it's vs its" typo

Don't LGTM this just yet. I want to add things from D38335 beforehand. I need to add a bunch of tests still but I'll upload the code for review tonight.

tests I want to add

  1. Trigger the error for .shstrtab being removed. Right now this should always be an error
  2. Make sure segment data is kept regardless of section removal.
  1. PT_LOAD Segment data is now copied though regardless of section removal
  2. Added checks for .shstrtab and .symtab to handle removal of those sections

I also think an error should be added if an allocated section is removed. It isn't clear to me what should happen in that case and the best thing I can come up with in that case is *very* different from what I think GNU objcopy would do. So until we know what should happen when an allocated section is removed I think would should throw an error saying that removing an allocated section isn't supported.

  1. Added test to make sure that segment data is kept regardless of section removal
  2. Added test to make sure that .symtab can be removed without causing error
  3. Added test to make sure that .shstrtab can't be removed. Once we have an option to not emit the section header table we can also add an option to allow .shstrtab to be removed in that case.

I did an experiment using objcopy, to see what happened if it removes a SHF_ALLOC section in a linked ELF with segments. As it turns out, objcopy removes the section without warning or error, and leaves 0s in its place. This could be dangerous on x86 at least, I think. If you do this to an executable section, and there's still the ability to jump there, then this could lead to execution of invalid instructions. At the very least, we should write trap instructions in place of the removed section, if it is in the executable segment. Alternatively, I'm quite happy you emit an error in this case.

A possible suggestion might be to have both behaviours, but have the "blank section" behaviour under another switch (e.g. --blank-section). This could be useful if the user wants to send somebody an ELF to inspect some specific characteristics, but without allowing them to actually run the program or see certain bits of data for whatever reason. I'd still replace the section contents with trap instructions in this case.

test/tools/llvm-objcopy/segment-data-kept.test
3 ↗(On Diff #117708)

Why not fold this and segment-test-remove-section.test together?

By the way, not sure if you noticed my comment in D38335:

I'm not sure it [stripping the section header string table] should be an error anyway - it looks like GNU binutils at the very least can handle an ELF with no section header string table, but with section headers, without too much problem, although I'd expect the e_shstrndx field to be set to 0 in this case.

Can I just confirm here that you are deliberately not allowing this behaviour? FWIW, objcopy refuses to strip the section, although it is silent about doing so.

By the way, not sure if you noticed my comment in D38335:

I'm not sure it [stripping the section header string table] should be an error anyway - it looks like GNU binutils at the very least can handle an ELF with no section header string table, but with section headers, without too much problem, although I'd expect the e_shstrndx field to be set to 0 in this case.

Can I just confirm here that you are deliberately not allowing this behaviour? FWIW, objcopy refuses to strip the section, although it is silent about doing so.

Sorry, yes. I heard this and want to add that ability in another change. Specifically the --strip-sections change.

jakehehrlich added a comment.EditedOct 5 2017, 6:00 PM

I did an experiment using objcopy, to see what happened if it removes a SHF_ALLOC section in a linked ELF with segments. As it turns out, objcopy removes the section without warning or error, and leaves 0s in its place. This could be dangerous on x86 at least, I think. If you do this to an executable section, and there's still the ability to jump there, then this could lead to execution of invalid instructions. At the very least, we should write trap instructions in place of the removed section, if it is in the executable segment. Alternatively, I'm quite happy you emit an error in this case.

A possible suggestion might be to have both behaviours, but have the "blank section" behaviour under another switch (e.g. --blank-section). This could be useful if the user wants to send somebody an ELF to inspect some specific characteristics, but without allowing them to actually run the program or see certain bits of data for whatever reason. I'd still replace the section contents with trap instructions in this case.

So someone on IRC requested that llvm-objcopy support the -j option. The particular way this is used is in conjunction with -O binary. This winds up showing a difference between what I did and what GNU objcopy actually does under the hood. GNU objcopy is section oriented not segment oriented. So if you use -O binary and -j in conjunction it will output just the data from that specific section. In the process of thinking about how to support this it became clear to me that removing allocated sections is something that's important to support, we can't just throw an error. It also became clear that I need to make -O binary work differently but more on that later. I talked with Roland McGrath about what seemed like expected behavior since I was quite ambivalent about what should happen when you remove a section that's in the middle of a segment. I pitched writing with gap fill (zero by default but using <gap fill> from --gap-fill=<gap fill> otherwise) vs just keeping the segment data. Roland seemed to think that keeping the segment data made no sense at all. This being the case it makes most sense to copy what GNU objcopy does. This also gives you both behaviors by default. Namely if you don't want a section overwritten with zeros just don't remove it. So to support the only change we would make is to just leave Object::writeSectionData alone rather than having it write segment data out first. In the --strip-sections diff we then remove only the non-allocated sections and don't emit the section header table. Everything should be consistent with what GNU objcopy does in the relevant cases.

In summary I propose the following:

  1. Leave writeSectionData alone so that it writes zeros. When we implement --gap-fill that can change slightly
  2. In --strip-sections we should only remove non-allocated sections
  1. Removed data-kept test because a) it was a bit redundant as James pointed out and b) I no longer think that is correct functionality as I explained. Basically I wasn't sure what correct functionality was and after talking with Roland McGrath and reading what James wrote I think I'm convinced that overwriting with zeros by default is better than keeping segment data.
  2. I made Object::writeSegmentData just behave like it used to which will mimic what GNU objcopy does when an allocated section is removed.

By the way, not sure if you noticed my comment in D38335:

I'm not sure it [stripping the section header string table] should be an error anyway - it looks like GNU binutils at the very least can handle an ELF with no section header string table, but with section headers, without too much problem, although I'd expect the e_shstrndx field to be set to 0 in this case.

Can I just confirm here that you are deliberately not allowing this behaviour? FWIW, objcopy refuses to strip the section, although it is silent about doing so.

Sorry, yes. I heard this and want to add that ability in another change. Specifically the --strip-sections change.

Okay, that's fine with me.

In summary I propose the following:

  1. Leave writeSectionData alone so that it writes zeros. When we implement --gap-fill that can change slightly
  2. In --strip-sections we should only remove non-allocated sections

I agree with this as an approach.

I think we need a test to demonstrate that we have zeros instead though, to demonstrate that this is the desired behaviour. I guess it would look very similar to the kept data test, but with zeroes instead of the old data.

Added test to make sure that zeros are being output

Sorry, this should have occurred to me earlier, but the tests that actually remove sections need to have checks for the e_shnum field, to show that it has been updated.

Once that's done, I'm happy with this change.

Added checks for section header table size to each test.

jhenderson accepted this revision.Oct 10 2017, 2:45 AM

LGTM, aside from one comment addition.

test/tools/llvm-objcopy/remove-section-with-symbol.test
31 ↗(On Diff #118284)

Could you add a comment saying what the other 4 section headers should be, please, to avoid confusion (I assume it is null, .symtab, .strtab, and .shstrtab).

This revision is now accepted and ready to land.Oct 10 2017, 2:45 AM
This revision was automatically updated to reflect the committed changes.