MCStreamer currently always copies the contents into a temporary buffer before emitting the contents of that buffer to the file. This is problematic when the input is large and already accessible elsewhere.
This change (see the separate sub-diffs/commits within it to follow along more easily) adds support for referential emission of bytes into the object file and uses that to reduce the high water mark in a large dwp run from 9.6 GB to 2.3 GB.
The implementation is totally ad-hoc/prototype code, I don't know MC terribly well enough to know much better and /totally/ looking for input on what this should look like. Should I support referential and non-referential output to the same section? Should it collapse the contents as I've implemented it, or interleave (using the old Contents buffer as a fragment buffer and represent the non-referential chunks?)? Just not support that at all and assert fail if you mix referential and non-referential output in the same section?
Any/all ideas welcome, totally happy to rewrite this in any direction now that I've demonstrated the value in the outcome.