This is an archive of the discontinued LLVM Phabricator instance.

[ELF] Set maximum page size to 65536 on AArch64
ClosedPublic

Authored by evgeny777 on Sep 30 2016, 1:38 AM.

Diff Detail

Event Timeline

evgeny777 updated this revision to Diff 73005.Sep 30 2016, 1:38 AM
evgeny777 retitled this revision from to [ELF] Set maximum page size to 65536 on AArch64.
evgeny777 updated this object.
evgeny777 added reviewers: ruiu, rafael.
evgeny777 set the repository for this revision to rL LLVM.
evgeny777 added a project: lld.
evgeny777 added subscribers: grimar, ikudrin, llvm-commits.
ruiu edited edge metadata.Sep 30 2016, 8:13 AM

LGTM as long as Peter is okay with this.

peter.smith edited edge metadata.Sep 30 2016, 9:47 AM

I think I'm going to need a bit of time to do some research. Both gold and ld.bfd have the concept of a maximum page size and a common page size. The maximum page size is 64k but the common page size is 4k. The maximum (abi page size in gold) is used in some places .rel.ro but not in others. I'm a bit concerned that this may be setting both maximum and common page size to 64k which may not be the best default, for example it certainly used to be true that a 64k page size prevented AArch32 (ARM) user space applications from running as these needed a 4kb page size.

If anyone has more information and can add some references I'd be most grateful.

I'll have a look as soon as I can, I'm at a conference this week and am flying back to UK this evening so I may not be able to finish today. Will pick it back up on Monday if I don't.

I think AArch32 is handled by ARMTargetInfo in lld, which has PageSize == MaxPageSize == 4096.

To my understanding gold uses common_pagesize (analog of PageSize in lld) to align sections to machine page boundary within segment
in order to save one memory page, in case OS kernel uses 4KB pages instead of 64KB. IMHO, we are quite far from this.

zatrazz edited edge metadata.Sep 30 2016, 11:56 AM

I do not seem any impending issue with this patch, however we must be aware that since max alignment is used as the default one for page alignment for some sections it might interfere with both memory utilization and relro setup. There is some information on a recent powerpc64le issue that occurred from a similar change in binutils [1].

Since lld currently does not define a 'common' page size, running lld with this modification on 4K kernels might result on both issues described above.

[1] https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1412553

I've had a chance to talk with a few colleagues:

  • There is no consensus amongst linux distributions about what the page size should be for AARCH64, Ubuntu have chosen 4k, fedora 64k
  • A 64k page size kernel can't run ARM binaries that assume a 4k page size
  • There are some applications that don't work on a 64k page size kernel if program segments are only aligned at 4k
  • LLD doesn't have a concept of common-page-size and max-page-size so if 64k page size is chosen all output sections are aligned to 64k boundary as well as program segments.
  • Aligning everything to a 64k page size should work on a 4k page size kernel but depending on how many output sections a program has this could be quite wasteful for a 4k page size kernel.

To summarise this looks like:

  • keeping the default at 4k will mean that people with 64k page size kernels will need to set the max page size to 64k
  • changing the default to 64k will mean that people with 4k page size kernels may see non trivial increases in image size

It seems like whatever default is chosen the other side won't be happy. I don't think that we can look to gold/ld.bfd here as they won't see the same size increases by setting the abi page size.

My instinct is to go for correctness over performance, but this does mean changing long established behaviour. Do you have a specific reason to want to change the default beyond matching some parts of what gold/bfd implement?

  • LLD doesn't have a concept of common-page-size and max-page-size so if 64k page size is chosen all output sections are aligned to 64k boundary as well as program segments.

AFAIK LLD does have a 'max-page-size' option and it exactly overrides the target's default value (check ELF/Driver.cpp:665). That's why I think this change in general is not wrong, but as you said it will add some possible issues to system with uses 4K as default.

Do you have a specific reason to want to change the default beyond matching some parts of what gold/bfd implement?

I'm using lld to link kernel for AArch64, so I decided to to make this patch in order to have similar behavior with ld/gold. Also I think switching to default page size of 65536 will make room for further optimizations for a case when one has 4K kernel / 64K alignment. You're right saying that using 64K alignment increases image size, but I suppose 4K systems are using '-z max-page-size=4096' (just because ld or gold is used) to overcome this problem, aren't they?

Given that on ld/gold max-page-size=64k common-page-size=4k, which doesn't increase the image size much I doubt many people with 4k page size systems will explicitly chang it. Furthermore I expect the binutils for each distro has a specs file with the appropriate defaults.

I've not got any strong personal opinions on page size, but I do want to make sure we don't needless upset someone by changing the default. I'll check with the BSD team as they are probably the largest stakeholder for AARCH64 support in lld right now.

Hello Ed,

Would changing the maximum page size to 64k in lld for AArch64 give you any cause for concern? My understanding is that it would increase ELF file size as each OutputSection would be aligned on a 64k boundary, but this would mean that both systems with a 4k and 64k page size could run with the default configuration.

Peter

each OutputSection would be aligned on a 64k boundary

I suppose you mean segment not section, don't you?

I did, although I may be mistaken. Looking at the tests the values for some of the section addresses gave me the impression that each output section was being aligned on a page boundary, although looking a bit deeper it looks to be only at the RO/RW boundar. If it is just on segment boundary I'm much less concerned.

I'll check this afternoon properly, my apologies if I was wrong, I had limited opportunity to download and rebuild last Friday.

No problem. FYI only sections which start PT_LOAD segment are page aligned. See Writer<ELFT>::fixSectionAlignments()

peter.smith accepted this revision.Oct 3 2016, 4:11 AM
peter.smith edited edge metadata.

Ok, in that case I'm happy. I suggest waiting a little before committing to see if BSD have any objections.

This revision is now accepted and ready to land.Oct 3 2016, 4:11 AM
This revision was automatically updated to reflect the committed changes.
emaste edited edge metadata.Oct 4 2016, 1:43 PM
emaste added a subscriber: andrew.

Hello Ed,

Would changing the maximum page size to 64k in lld for AArch64 give you any cause for concern?

Sorry I didn't get a chance to look prior to commit. I don't think it's a problem but will ping @andrew for confirmation.

Hello Ed,

Would changing the maximum page size to 64k in lld for AArch64 give you any cause for concern?

Sorry I didn't get a chance to look prior to commit. I don't think it's a problem but will ping @andrew for confirmation.

I can't think of any problems for FreeBSD. There might be advantages to making sure we can align things in memory on 2MB boundaries so we can use larger page table entries & reduce tlb usage.