This is an archive of the discontinued LLVM Phabricator instance.

Make sure DataBufferLLVM contents are writable
ClosedPublic

Authored by labath on Nov 15 2017, 8:00 AM.

Details

Summary

We sometimes need to write to the object file we've mapped into memory,
generally to apply relocations to debug info sections. We've had that
ability before, but with the introduction of DataBufferLLVM, we have
lost it, as the underlying llvm class (MemoryBuffer) only supports
read-only mappings. The way DataBufferLLVM implemented writable buffers
was through the Private=true flag, which causes the file contents to be
read into a malloc()ed buffer.

This wasn't a problem then, since we weren't aware of any cases where we
need to relocate the debug info, and we just always set Private=false
for the cases where we mmapped the full object file. This has changed in
D38142, where needed to relocate the FreeBSD kernel, and we've flipped
the private flag to true, without even realising the full impact of
that. (The impact is significant slowdown in debug info loading and a
huge number of dirty pages).

This (not particularly elegant) commit tries to restore the performance
and bring down memory usage by bringing back the private copy-on-write
file mappings. It does this by falling back to the lower-level llvm
primitive (mapped_file_region), which supports this type of mappings.
Unfortunately, this primitive does not support non-mmapped regions, so
for volatile regions I have to still use the MemoryBuffer class.

I have also removed the "Private" argument to DataBufferLLVM factory
functions -- what it really meant was "read-only", but we generally
assume all our databuffers are writable. If we ever want to have
read-only buffers, we can bring it back with an appropriate name.

Diff Detail

Repository
rL LLVM

Event Timeline

labath created this revision.Nov 15 2017, 8:00 AM
clayborg edited edge metadata.Nov 15 2017, 10:05 AM

I had suggested in D38142 that we have ObjectFileELF check the file type of the file and only map with write abilities if the ELF file is an object file since that is the only time we need relocations. If we can pass a flag through to indicate how to map, would that provide any additional performance improvements?

I had suggested in D38142 that we have ObjectFileELF check the file type of the file and only map with write abilities if the ELF file is an object file since that is the only time we need relocations. If we can pass a flag through to indicate how to map, would that provide any additional performance improvements?

Sorry, I hadn't seen that comment.

But to answer to question, it would not provide *additional* performance improvements -- the memory will still be shared until you actually try to write to it (with the read-only shared mapping you'll get a SEGV, with a private read-write mapping, the page will be COWed).

However, I suppose something like that could be done *instead* of this patch. That way we would be using mmap for most object files, and only resorting to malloc+read for those that need relocating. It's not an ideal solution, but then again, neither is this.

zturner edited edge metadata.Nov 15 2017, 4:35 PM

On second thought I actually think the strategy used here is fine. But we have llvm::WritableBinaryStream which could server as a single base interface for a mapped_file_region based implementation as well as a malloc-based implementation.

For the malloc based implementation, we already have WritableBinaryByteStream, so you don't even need to implement that.

The nice thing about this is that it could later be improved to use the stream reader / stream writer idiom, which gets rid of a ton of manual pointer arithmetic and management.

labath updated this revision to Diff 127500.Dec 19 2017, 4:59 AM

It took a while, but we've finally landed an llvm::WritableMemoryBuffer class.
This patch now becomes simple, as all I need to do is use that instead of the
MemoryBuffer class.

Well.. almost. The new class does not have the null-termination capability, so I
also update our buffer uses to not require null termination.

clayborg requested changes to this revision.Dec 19 2017, 9:04 AM

See inlined comment. Quick fix and this will be good to go.

source/Plugins/ObjectFile/ELF/ObjectFileELF.cpp
409–410 ↗(On Diff #127500)

We should put the DataBufferLLVM::CreateSliceFromPath() call into a new static function in ObjectFile:

static DataBufferSP ObjectFile::MapFileData(FileSpec file, uint64_t Size, uint64_t Offset);

Then when we need to make fixes, we change just one place instead of all of them. We should add this new function and switch all ObjectFile subclasses over to using it.

This revision now requires changes to proceed.Dec 19 2017, 9:04 AM
labath updated this revision to Diff 127687.Dec 20 2017, 5:07 AM

Add the suggested MapFileData function

clayborg accepted this revision.Dec 20 2017, 10:45 AM

Looks good, thanks for the making the changes. This will make future tweaks to reading of file data in object files just a single change in ObjectFile.cpp.

This revision is now accepted and ready to land.Dec 20 2017, 10:45 AM
This revision was automatically updated to reflect the committed changes.