This is an archive of the discontinued LLVM Phabricator instance.

[llvm-objcopy] Refactor llvm-objcopy to use reader and writer objects
ClosedPublic

Authored by jakehehrlich on Jan 17 2018, 5:54 PM.

Details

Summary

While writing code for input and output formats in llvm-objcopy it became apparent that there was a code health problem. This change attempts to solve that problem by refactoring the code to use Reader and Writer objects that can read in different objects in different formats, convert them to a single shared internal representation, and then write them to any other representation.

New classes:
Reader: the base class used to construct instances of the internal representation
Writer: the base class used to write out instances of the internal representation
ELFBuilder: a helper class for ELFWriter that takes an ELFFile and converts it to a Object
SectionVisitor: it became necessary to remove writeSection from SectionBase because, under the new Reader/Writer scheme, it's possible to convert between ELF Types such as ELF32LE and ELF32BE. This isn't possible with writeSection because it (dynamically) depends on the underlying section type *and* (statically) depends on the ELF type. Bad things would happen if the underlying sections for ELF32LE were used for writing to ELF64BE. To avoid this code smell (which would have compiled, run, and output some nonsesnse) I decoupled writing of sections from a class.
SectionWriter: This is just the ELFT templated implementation of SectionVisitor. Many classes now have this class as a friend so that the writing methods in this class can write out private data.
ELFWriter: This is the Writer that outputs to ELF
BinaryWriter: This is the Writer that outputs to Binary
ElfType: Because the ELF Type is not a part of the Object anymore we need a way to construct the correct default Writer based on properties of the Reader. This enum just keeps track of the ELF type of the input so it can be used as the default output type as well.

Object has correspondingly undergone some serious changes as well. It now has more generic methods for building and manipulating ELF binaries. This interface makes ELFBuilder easy enough to use and will make the BinaryReader/Builder easy to create as well. Most changes in this diff are cosmetic and deal with the fact that a method has been moved from one class to another or a change from a pointer to a reference. Almost no changes should result in a functional difference (this is after all a refactor). One minor functional change was made and the result can be seen in remove-shstrtab-error.test. The fact that it fails hasn't changed but the error message has changed because that failure is detected at a later point in the code now (because WriteSectionHeaders is a property of the ElfWriter *not* a property of the Object). I'd say roughly 80-90% of this code is cosmetically different, 10-19% is different but functionally the same, and 1-5% is functionally different despite not causing a change in tests.

Known Code Smells:

I'm actully aware of a number of issues I'd personally like to fix before this is submitted. This code is still rough and in need of some polish. For instance there's a bit of code duplication that I haven't removed. There are several "if(auto *o = dyn_cast<ELFObjectFile<...>(...))" blocks around that I'd like to remove. Also the handling of BufPtr and SectionWriter in BianryWriter and ElfWriter is duplicated. I'd like to remove that (I think that one can be accomplished by just folding it into the Writer base class). Also it might be possible to split this up into separate diffs. Any recommendations on how to split it up would be welcome. I figured now that it a) compiles and b) appears to pass all tests that I could start getting some feedback.

Diff Detail

Repository
rL LLVM

Event Timeline

jakehehrlich created this revision.Jan 17 2018, 5:54 PM

Overall, the approach seems sound. I particularly like how it removes the templates from many of our classes, and removes the unfortunate SymbolTableSectionImpl class. Have you tried extending this with new input types, to make sure this design works?

I agree with your points about the code smells, and think they should be fixed too. As to how to split it up, could the SectionWriter/SectionVisitor related changes be done first? After that a logical split would be to do the Reader-related classes, then the Writer classes, if it's possible without having to make too many intermediate changes that are just going to be thrown away.

tools/llvm-objcopy/Object.cpp
77 ↗(On Diff #130334)

This should go above the previous function.

667 ↗(On Diff #130334)

I'm not sure you want this to take an r-value reference? Shouldn't it be a const reference, since you don't need to change the range?

966 ↗(On Diff #130334)

Braces here aren't needed.

1060 ↗(On Diff #130334)

This line looks a little dodgy. Doesn't that mean we'll write ELF64LE sections to binary output, no matter what the input was?

tools/llvm-objcopy/Object.h
76–82 ↗(On Diff #130334)

It would probably make more sense for this to come after SectionVisitor.

499 ↗(On Diff #130334)

What items are in this namespace that means we need this using in a header?

503–505 ↗(On Diff #130334)

This will require us to always have the ELFReader allocated until we write, which seems a little less than ideal?

518 ↗(On Diff #130334)

Do you plan on having any child classes? Otherwise, why protected?

tools/llvm-objcopy/llvm-objcopy.cpp
175–182 ↗(On Diff #130334)

Any particular reason this is no longer in its own function?

317 ↗(On Diff #130334)

Unused return value and unnecessary braces.

329 ↗(On Diff #130334)

What's complicated? :)

Have you tried extending this with new input types, to make sure this design works?

No but I was thinking about the case while designing this approach. I should probably actually do that just to make sure. In principal it should just be as simple as making a new Reader class and constructs all the needed parts.

tools/llvm-objcopy/Object.cpp
667 ↗(On Diff #130334)

It's not an rvalue reference, it's a "universal reference". It's whatever the user passes it.

1060 ↗(On Diff #130334)

Ah yes, I remember having this dilemma before pushing it to the side in favor of progress. I was thinking about splitting up SectionWriter into a non-ELFT-templated base class and a separate ELFT-template base class and then *also* a binary section writer (that is non-ELFT-templated) that throws errors for the methods that need ELFT to output correctly. How does that sound?

tools/llvm-objcopy/Object.h
499 ↗(On Diff #130334)

ELFFile, ELFObjectFile, and maybe something else I'm not quite remembering. I should probably just explicitly use "using" on those things I need rather than massively pollute the whole namespace, huh?

503–505 ↗(On Diff #130334)

I want to keep it so that the data between the Object and the Binary is not duplicated. I do agree that it doesn't seem like the Reader shouldn't *have* to stay open the whole time but I'd like the Binary to stay open the whole time (until all output files have been written at least). I think this can be remedied in two opposite ways. One way is for the Reader to maintain ownership of the Objects so that the Binary and the Objects that reference it's data have a shared lifetime. The other idea is that this could perhaps be decoupled and the Reader could store a reference to the Binary. It then becomes the user's job to make sure that the Bianry stays open as long as any object created by the Reader. Right now it's (oddly) the user's job to make sure that the Reader stays open as like as any object created by it. I agree that's wrong.

What are your thoughts?

518 ↗(On Diff #130334)

Good, point. I don't think I'll ever have any child classes. It was mostly holdover from the previous Object type.

tools/llvm-objcopy/llvm-objcopy.cpp
175–182 ↗(On Diff #130334)

Not that I remember, I'll put it back.

329 ↗(On Diff #130334)

Haha, that was a comment to my self that I left in, whoops. I belive that was one of the "update to use reader and writer" todos that I wasn't 100% clear on how to acomplish.

As to how to split it up, could the SectionWriter/SectionVisitor related changes be done first? After that a logical split would be to do the Reader-related classes, then the Writer classes, if it's possible without having to make too many intermediate changes that are just going to be thrown away.

I'll look into that. I think the SectionWriter/SectionVisitor split is very doable and should be done without question. The Reader and Writer split was the split I was thinking of initially (I was grouping the SectionVisitor split as being a part of the Writer split). I think I'm just going to have to try that one and see what happens as I don't have a good intuition for how hard that split would be.

  1. Fixed BufPtr duplication in Writers
  2. Moved Writer class to below SectionWriter
  3. Removed using namespace object in favor of using specific types
  4. Removed protected members (made them private)
  5. Move SectionVisitor destructor to above the first SectionWriter call
  6. Removed curly braces on if-stmt for WhiteSectionHeaders
  7. Put code for SplitDWOToFile back in a function
  8. Removed unused variable for section return
  9. Removed TODO, whoops
jhenderson added inline comments.Jan 19 2018, 2:12 AM
tools/llvm-objcopy/Object.cpp
667 ↗(On Diff #130334)

Huh, I can't say I was aware of the distinction - I had to go and look it up!

1060 ↗(On Diff #130334)

I think that would make things a little clearer, yes. I'd like to see which methods you're referring to that should throw errors, and why we don't need them for a binary section writer though.

tools/llvm-objcopy/Object.h
499 ↗(On Diff #130334)

Yes, I think that's probably right in a header file (no problem in cpp files, mind you).

503–505 ↗(On Diff #130334)

I'm actually wondering whether we could transfer ownership of either the Reader, or just the Binary associated with it to the Object. That way the original Reader can be disposed of at any point after creating the Object. Unfortunately, this doesn't play particularly well with multiple input types. Of the two ideas you suggested I'd prefer that Readers have ownership of their Objects.

jakehehrlich added inline comments.Jan 22 2018, 11:55 AM
tools/llvm-objcopy/Object.cpp
1060 ↗(On Diff #130334)

Non-allocated relocations, SHT_SYMTAB, and GnuDebugLinkSections (which are non-allocated as well) are the only ones that need to be written as such. There are many section types that should never make into the binary output that these methods would not throw errors on (string tables for instance) but there are no cases in which the above 3 sections should be output for -O binary. That way if this ever comes up and we get an error it either means a) the code is messed up or b) someone marked .symtab or .gnu-debuglink as allocated.

tools/llvm-objcopy/Object.h
503–505 ↗(On Diff #130334)

I wonder if there's a way to use reference counting here? I can't find a way to make a binary shared_ptr but that would be the time to use it if there ever was one. I played around with Readers owning Objects though and it can be enforced really well using references. Basically objects are stored in a vector and references to them are returned on creation (I haven't considered how collection of these objects should occur). If the ELFBuilder (which is owned by the ELFReader) owns the binary then we don't even need to use unique pointers to allocate objects (they can just be allocated in the vector).

Added BinarySectionWriter to avoid always using ELFWriter<ELF64LE> for writing binaries. Errors are now thrown for certain input we don't know how to handle (like allocated SHT_SYMTAB, allocated relcoations, and allocated .gnudebuglink)

Updated code to handle data ownership in a better way. Objects now own all their data using a shared_ptr. This eliminates the potential for dangling references if the Reader is not left open.

Okay, I like the changes now. Perhaps worth adding a test with a crazy input being converted to Binary, resulting in errors on those section types? Alternatively, instead of emitting an error, maybe it makes sense to simply write the table as some arbitrary binary blob and warn? I doubt there's a use case for this though, so I think emitting an error for now is fine, and change if the need arises.

tools/llvm-objcopy/Object.cpp
73 ↗(On Diff #130955)

bianry -> binary (also other two cases below)

I think that it would be better if these error messages had the section name included.

tools/llvm-objcopy/Object.h
94–102 ↗(On Diff #130955)

To keep a consistent order of classes, this should go below ELFSectionWriter, or ELFWriter should go below BinaryWriter.

  1. Moved BinarySectionWriter to after ELFWriter
  2. Fixed bianry/binary typos
  3. Added section names to error messages
jakehehrlich updated this revision to Diff 131149.EditedJan 23 2018, 3:00 PM

Added test to check one of the error messages that BianrySectionWriter can generate. I only test the symtab case. The other two cases are hard to test without a way to edit the sections after they've been read in (with like --set-section-flags or something). Relocations are read in as DynamicRelocationSections when allocated so this whole issue is bypassed. Since GnuDebuglinkSection is never read in, only added, there isn't a way to make a binary that will trigger that code path. Once --set-section-flags exists it will be able to dynamically change these values and trigger these cases.

Oh and alloc-symtab.o is just a small relocatable ELF file where I manually flipped the flag bit to mark it allocated. I literally had to use xxd to binary patch the file. GNU objcopy won't respect --set-section-flags on symbol tables and no tool I'm aware of will produce an allocated SHT_SYMTAB section.

One small inline comment. Otherwise, I'm happy with this as-is. Are you still planning on splitting it up? It'll make it somewhat easier, I hope to spot any problems, but if it's a lot of extra work, may no longer be worth it.

test/tools/llvm-objcopy/binary-out-error.test
1 ↗(On Diff #131149)

What's going on with the "/dev/null" here?

One small inline comment. Otherwise, I'm happy with this as-is. Are you still planning on splitting it up? It'll make it somewhat easier, I hope to spot any problems, but if it's a lot of extra work, may no longer be worth it.

It's a lot of work to split it up. I'd rather not.

test/tools/llvm-objcopy/binary-out-error.test
1 ↗(On Diff #131149)

We need to throw an error but that error is written to stderr not stdout but the pipe only pipes stdout to FileCheck. So we redirect stderr (2) to where stdout is going (&1) and then we redirect stdout to /dev/null so nothing comes from stdout (because as is both stderr and stdout would be pipped to FileCheck).

In this case it doesn't matter but I copy-pasted this from other code that checks errors and that come along with it. It's part of the standard way of checking error messages in lit tests. Run "git grep /dev/null | wc -l" in the test directory. I get 1113 uses but not all of those are to suppress stdout, some suppress stderr.

jhenderson accepted this revision.Jan 25 2018, 1:41 AM

Okay, LGTM.

This revision is now accepted and ready to land.Jan 25 2018, 1:41 AM
This revision was automatically updated to reflect the committed changes.