Page MenuHomePhabricator

Enhancing MCStreamer to support referential (rather than copy) emission of bytes and use that to reduce the memory footprint of llvm-dwp
Needs ReviewPublic

Authored by dblaikie on Feb 28 2016, 8:45 AM.



MCStreamer currently always copies the contents into a temporary buffer before emitting the contents of that buffer to the file. This is problematic when the input is large and already accessible elsewhere.

This change (see the separate sub-diffs/commits within it to follow along more easily) adds support for referential emission of bytes into the object file and uses that to reduce the high water mark in a large dwp run from 9.6 GB to 2.3 GB.

The implementation is totally ad-hoc/prototype code, I don't know MC terribly well enough to know much better and /totally/ looking for input on what this should look like. Should I support referential and non-referential output to the same section? Should it collapse the contents as I've implemented it, or interleave (using the old Contents buffer as a fragment buffer and represent the non-referential chunks?)? Just not support that at all and assert fail if you mix referential and non-referential output in the same section?

Any/all ideas welcome, totally happy to rewrite this in any direction now that I've demonstrated the value in the outcome.

Diff Detail

Event Timeline

dblaikie updated this revision to Diff 49318.Feb 28 2016, 8:45 AM
dblaikie retitled this revision from to Enhancing MCStreamer to support referential (rather than copy) emission of bytes and use that to reduce the memory footprint of llvm-dwp.
dblaikie updated this object.
dblaikie added reviewers: chandlerc, echristo, grosbach.
dblaikie added a subscriber: llvm-commits.

Oh, bother. I was hoping it would show my local commits as separately accessible/diffable changes, but it seems not.

Well, if the patch is in any way hard to follow, or you'd otherwise like to seep the smaller steps, do let me know - happy to send them as attachments, etc.

chandlerc resigned from this revision.Apr 6 2016, 11:30 PM
chandlerc removed a reviewer: chandlerc.

I don't really feel like I'm familiar enough with the MC layer to give a useful review, but I can try if no one else comments... But I wanted to voice some strong support of getting this kind of functionality into the tree, as it seems really awesome.

David and I discussed a bit on IRC, but looks like it never got logged. Sorry.

So, from the discussion:

  • dwp is not really a linker (no relocations if I remember correctly)
  • dwp is not really an assembler (no relaxations)

So it seems that what needs to happen is that instead of dwp using MC,
lib/MC/ELFObjectWriter.cpp should be refactored into a small library
that bot MC and dwp can use.