[BOLT] Move from RuntimeDyld to JITLink

[jobnoorman]

Summary

RuntimeDyld has been deprecated in favor of JITLink. [1] This patch
replaces all uses of RuntimeDyld in BOLT with JITLink.

Care has been taken to minimize the impact on the code structure in
order to ease the inspection of this (rather large) changeset. Since
BOLT relied on the RuntimeDyld API in multiple places, this wasn't
always possible though and I'll explain the changes in code structure
first.

Design note: BOLT uses a JIT linker to perform what essentially is
static linking. No linked code is ever executed; the result of linking
is simply written back to an executable file. For this reason, I
restricted myself to the use of the core JITLink library and avoided ORC
as much as possible.

RuntimeDyld contains methods for loading objects (loadObject) and symbol
lookup (getSymbol). Since JITLink doesn't provide a class with a similar
interface, the BOLTLinker abstract class was added to implement it. It
was added to Core since both the Rewrite and RuntimeLibs libraries make
use of it. Wherever a RuntimeDyld object was used before, it was
replaced with a BOLTLinker object.

There is one major difference between the RuntimeDyld and BOLTLinker
interfaces: in JITLink, section allocation and the application of fixups
(relocation) happens in a single call (jitlink::link). That is, there is
no separate method like finalizeWithMemoryManagerLocking in RuntimeDyld.
BOLT used to remap sections between allocating (loadObject) and linking
them (finalizeWithMemoryManagerLocking). This doesn't work anymore with
JITLink. Instead, BOLTLinker::loadObject accepts a callback that is
called before fixups are applied which is used to remap sections.

The actual implementation of the BOLTLinker interface lives in the
JITLinkLinker class in the Rewrite library. It's the only part of the
BOLT code that should directly interact with the JITLink API.

For loading object, JITLinkLinker first creates a LinkGraph
(jitlink::createLinkGraphFromObject) and then links it (jitlink::link).
For the latter, it uses a custom JITLinkContext with the following
properties:

• Use BOLT's ExecutableFileMemoryManager. This one was updated to implement the JITLinkMemoryManager interface. Since BOLT never executes code, its finalization step is a no-op.
• Pass config: don't use the default target passes since they modify DWARF sections in a way that seems incompatible with BOLT. Also run a custom pre-prune pass that makes sure sections without symbols are not pruned by JITLink.
• Implement symbol lookup. This used to be implemented by BOLTSymbolResolver.
• Call the section mapper callback before the final linking step.
• Copy symbol values when the LinkGraph is resolved. Symbols are stored inside JITLinkLinker to ensure that later objects (i.e., instrumentation libraries) can find them. This functionality used to be provided by RuntimeDyld but I did not find a way to use JITLink directly for this.

Some more minor points of interest:

• BinarySection::SectionID: JITLink doesn't have something equivalent to RuntimeDyld's Section IDs. Instead, sections can only be referred to by name. Hence, SectionID was updated to a string.
• There seem to be no tests for Mach-O. I've tested a small hello-world style binary but not more than that.
• On Mach-O, JITLink "normalizes" section names to include the segment name. I had to parse the section name back from this manually which feels slightly hacky.

@lhames: I've added you as an (optional) reviewer. If you have time and
interest to look at this patch, feedback on the use of JITLink would be
greatly appreciated.

There's one thing in particular I'm curious about: is there a better way
to load multiple objects? I currently have to manually keep track of the
symbols of loaded objects so that when other objects are loaded later,
they can refer to them. It feels like this is something JITLink should
be able to do? Also, this strategy cannot support earlier objects
referring to symbols of later ones (not an issue for BOLT currently but
still feels like a downside). I was expecting there to be a way to merge
multiple LinkGraphs but couldn't find one.

[1] https://reviews.llvm.org/D145686#4222642

[lhames]

Design note: BOLT uses a JIT linker to perform what essentially is
static linking. No linked code is ever executed; the result of linking
is simply written back to an executable file. For this reason, I
restricted myself to the use of the core JITLink library and avoided ORC
as much as possible.

I'm very surprised to hear this. I guess at this point I should confess that I haven't looked closely at how Bolt uses the JIT APIs.

If you only need static linking, wouldn't lld be a better fit than the JIT APIs?

[lhames]

If you only need static linking, wouldn't lld be a better fit than the JIT APIs?

Expanding on this thought a little:

If you're doing dynamic instrumentation (or dynamically changing layout) in a JIT process then I think ORC/JITLink would be a good fit. If you're doing static instrumentation for a static executable, I'd have thought lld would be a better fit. I could see a system using both: The JIT for initial investigation using dynamic instrumentation and dynamic re-layout (to find an efficient layout), then lld to produce a final executable using the chosen layout.

ORC/JITLink can probably be substituted for RuntimeDyld, but I'd need to understand how BOLT is using RuntimeDyld a little better before I could say for sure.

@jobnoorman @rafaelauler If a real-time chat would help I'm on the LLVM discord.

[jobnoorman]

Thanks for your comments, @lhames!

If you only need static linking, wouldn't lld be a better fit than the JIT APIs?

I think you're right, at least in theory. In fact, when looking into this JITLink port, I briefly considered using lld. I decided against it for the following reasons:

• lld's API is rather primitive: it's basically just a wrapper around calling ld.lld's main function with command-line arguments. This means that, for example, inputs always need to be written out to disk and cannot be processed completely in memory (as BOLT now does). We would also need to write linker scripts on-the-fly to support custom section remapping. This is probably not a huge problem but feels a bit inelegant.
• I felt this would need a huge rewrite of BOLT's logic. BOLT doesn't relink all code. Some parts are simply copied over from the input file to the output. References from the relinked code to other parts of the binary are resolved using a custom symbol lookup. There's quite some custom logic for reassembling the final ELF file (handling GOTs, PLTs, eh_frame,...). etc. I wasn't confident this could easily be adapted to the use of lld.

Of course, these are not good reasons for why BOLT has to use a JIT linker. However, since my main goal for this patch is to move BOLT away from the deprecated RuntimeDyld library, I felt like switching to JITLink was the easier and more pragmatic choice.

My assessment might be a bit off, though, and maybe there are good reasons for BOLT to use a JIT linker. I'd be very interested in hearing the opinions of the BOLT devs on this.

[rafauler]

My assessment might be a bit off, though, and maybe there are good reasons for BOLT to use a JIT linker. I'd be very interested in hearing the opinions of the BOLT devs on this.

I agree, I'll speak for myself, maybe later Maksim can give his view, since he originally wrote the code that uses RuntimeDyld (he's on vacation at the moment). I don't think lld was written to be (easily) used as library. The way to communicate with lld would be writing stuff to disk / abusing linker scripts/ trying to do all sorts of hacks to make the linker stop doing linker stuff. I'll try to make myself more clear below.

The reason BOLT used RuntimeDyld is because final linking already took place in the binary, and BOLT just needed to patch a few references without doing any other classic linker jobs, and RuntimeDyld was a simple component that provided exactly that with few changes (such as mapping sections to the addresses seen in the input binary, instead of mapping them to a random page of JIT memory). BOLT would never require generating PLT tables, for example, because this is already done in the input. The only feature that BOLT requires from the linker is actually patching the relocations to addresses from the BOLT symbol table. It does NOT need to: combine multiple sections into one, generate extra code, generate extra data structures, understand ELF symbol versioning, consider weak symbols, generate TLS/GOT/etc.

Given this unique use case to just run a tiny bit of the functionality of a linker, which is precisely a for loop going through all relocations and patching them, it was way easier to just use RuntimeDyld as directly as possible. We actually considered just writing this code to patch relocations inside BOLT, but it seemed silly to duplicate code already written in RuntimeDyld.

[lhames]

@jobnoorman I've had a chance to read over this a bit more carefully. I still don't grok how BOLT works, but this is a pretty amazing application of JITLink to an area that it was never designed for. ;)

There's one thing in particular I'm curious about: is there a better way
to load multiple objects?

Maybe. ORC certainly supports linking multiple objects together into a single JIT process, but whether it's approach is applicable to BOLT I'm less sure.

I currently have to manually keep track of the symbols of loaded objects
so that when other objects are loaded later, they can refer to them. It feels
like this is something JITLink should be able to do? Also, this strategy
cannot support earlier objects referring to symbols of later ones (not an
issue for BOLT currently but still feels like a downside). I was expecting there
to be a way to merge multiple LinkGraphs but couldn't find one.

JITLink usually publishes and looks up symbols in the ORC symbol tables via notifyResolved and lookup respectively. ORC's symbol tables support async lookups (and you can see this reflected in the signature of JITLinkContext::lookup, which is designed to dovetail with ORC): when a link for an earlier object looks up a symbol in a defined in a later object the JITLinkAsyncLookupContinuation for the earlier link will just sit in the ORC symbol table until the symbol addresses from the later object become available, then ORC runs the continuation and the earlier link proceeds. Supporting async lookup in a custom JITLinkContext is possible, but there may be a way to just use ORC for this too, if you ever find that you need to do it.

In D147544#4246122, @rafauler wrote:

The reason BOLT used RuntimeDyld is because final linking already took place in the binary...

Is BOLT taking a relocatable file as input, or an executable / dylib?

And is BOLT using RuntimeDyld to apply relocations so that the final executable is bound to a specific address? Or is it creating new relocations in the output based on the ones RuntimeDyld produces, and skipping fixups?

Depending on what it's doing the best option might be to implement a merge operation on LinkGraphs, and then process the merged graph in BOLT rather than using JITLink. The applyFixup functions are already public for each architecture, so I think you might be able to do this:
(1) Parse and merge graphs for all inputs.
(2) Assign virtual addresses to all blocks in the merged graph.
(3) Walk the graph and apply fixups to internal edges (edges between blocks in the merged graph), using the architecture-specific applyFixups function.
(4) Write out the merged graph, with including fixup info for external edges.

It might also be worth involving some lld folks in this discussion -- maybe their APIs are more amenable to this these days, or maybe this is something they could support with minimal effort. It definitely feels like a close fit to what LLD would be doing day-to-day.

[lhames]

^^^ None one the above should be read as a reason not to accept this patch, just thinking about future directions.

[rafauler]

Is BOLT taking a relocatable file as input, or an executable / dylib?

BOLT's input is a final executable / .so file.

Here's how BOLT operates at a high level.

1. Run a disassembler to create MCInst objects for each instruction in a function it wants to patch. (it does more than that, such as creating cfgs, but I'll skip for the purposes of this discussion)
2. After MCInsts get shuffled around to take advantage of a better code layout, we wend it to the assembler (via the MCStreamer interface).
3. Assembler resolves internal fixups, generates regular ELF relocations for symbols that are not local (for example, references to other functions or global data in that binary).
4. We feed that to RuntimeDyld to patch these relocations for us. RuntimeDyld actually has no idea where are these global symbols, so it asks us "where is foobar", we know where it is because we read the full symbol table of that binary (and using other techniques as well), so we answer RuntimeDyld back with all of the addresses it needs to resolve.
5. Relocations are processed and the code is available in a blob of memory.
6. We copy that blob of memory back to the final binary, at the location we want to patch.

BOLT has different modes of operation too, but at a high level, from a linker perspective, that's what it is doing.

[lhames]

In D147544#4247007, @rafauler wrote:

Is BOLT taking a relocatable file as input, or an executable / dylib?

BOLT's input is a final executable / .so file.

Here's how BOLT operates at a high level.

1. Run a disassembler to create MCInst objects for each instruction in a function it wants to patch. (it does more than that, such as creating cfgs, but I'll skip for the purposes of this discussion)
2. After MCInsts get shuffled around to take advantage of a better code layout, we wend it to the assembler (via the MCStreamer interface).
3. Assembler resolves internal fixups, generates regular ELF relocations for symbols that are not local (for example, references to other functions or global data in that binary).
4. We feed that to RuntimeDyld to patch these relocations for us. RuntimeDyld actually has no idea where are these global symbols, so it asks us "where is foobar", we know where it is because we read the full symbol table of that binary (and using other techniques as well), so we answer RuntimeDyld back with all of the addresses it needs to resolve.
5. Relocations are processed and the code is available in a blob of memory.
6. We copy that blob of memory back to the final binary, at the location we want to patch.

Is this operating per-function and patching the relocated code for that function back into the original binary then? I had assumed that you were creating a whole new ELF container.

[Amir]

In D147544#4254306, @lhames wrote:

In D147544#4247007, @rafauler wrote:

Is BOLT taking a relocatable file as input, or an executable / dylib?

BOLT's input is a final executable / .so file.

Here's how BOLT operates at a high level.

1. Run a disassembler to create MCInst objects for each instruction in a function it wants to patch. (it does more than that, such as creating cfgs, but I'll skip for the purposes of this discussion)
2. After MCInsts get shuffled around to take advantage of a better code layout, we wend it to the assembler (via the MCStreamer interface).
3. Assembler resolves internal fixups, generates regular ELF relocations for symbols that are not local (for example, references to other functions or global data in that binary).
4. We feed that to RuntimeDyld to patch these relocations for us. RuntimeDyld actually has no idea where are these global symbols, so it asks us "where is foobar", we know where it is because we read the full symbol table of that binary (and using other techniques as well), so we answer RuntimeDyld back with all of the addresses it needs to resolve.
5. Relocations are processed and the code is available in a blob of memory.
6. We copy that blob of memory back to the final binary, at the location we want to patch.

Is this operating per-function and patching the relocated code for that function back into the original binary then? I had assumed that you were creating a whole new ELF container.

BOLT operates per-function, the optimized function is placed in a newly-created .text section (or is split between .text and .text.cold), or, if the function is not updated, it's copied to .bolt.org.text and only the external references are patched.

Creating a new ELF container is something that BOLT wants to avoid – because in that case BOLT would need to understand and update all relocations, including data-to-data relocations, which is outside of our scope. There's an ongoing effort to support this mode to reduce the final binary size but it's still a WIP (D144560).

[rafauler]

I've been testing these changes and I think you need to run this with an address sanitizer. My address sanitizer builds are complaining about memory problems on at least these tests:

BOLT :: AArch64/asm-func-debug.test
BOLT :: AArch64/update-debug-reloc.test
BOLT :: X86/asm-func-debug.test
BOLT :: X86/dwarf5-call-pc-function-null-check.test
BOLT :: X86/dwarf5-call-pc.test
BOLT :: X86/dwarf5-debug-line.s
BOLT :: X86/dwarf5-debug-loclists.s
BOLT :: X86/dwarf5-df-dualcu-loclist.test
BOLT :: X86/dwarf5-df-dualcu.test
BOLT :: X86/dwarf5-df-mono-dualcu.test
BOLT :: X86/dwarf5-dwarf4-monolithic.test
BOLT :: X86/dwarf5-ftypes-dwp-input-dwo-output.test
BOLT :: X86/dwarf5-label-low-pc.s
BOLT :: X86/dwarf5-loclist-offset-form.test
BOLT :: X86/dwarf5-lowpc-highpc-convert.s
BOLT :: X86/dwarf5-rangeoffset-to-rangeindex.s
BOLT :: X86/dwarf5-return-pc.test
BOLT :: X86/dwarf5-shared-str-offset-base.s
BOLT :: X86/dwarf5-split-dwarf4-monolithic.test
BOLT :: X86/dwarf5-two-loclists.test
BOLT :: X86/dwarf5-two-rnglists.test
BOLT :: X86/inline-debug-info.test
BOLT :: X86/inlined-function-mixed.test
BOLT :: X86/line-number.test
BOLT :: keep-aranges.test
BOLT :: non-empty-debug-line.test

[rafauler]

In one of our large tests on big x86 binaries, it's also firing this assertion:

JITLinkLinker.cpp:52: void llvm::bolt::(anonymous namespace)::reassignSectionAddress(jitlink::LinkGraph &, const llvm::bolt::BinarySection &, uint64_t): Assertion `JLSection && "cannot find section in LinkGraph"' failed

I took a look and it looks like the sectionID of the second argument (the binarysection object) is somehow storing garbage data. That's why I took a look at our asan builds. I got this when printing the section name it was trying to fetch from the link graph:

(...)
(earlier correct call) reassignsection address for id: .local.text.funcname/1
(buggy one) reassignsection address for id: ),:$R↓_ZZN5°⎺┌┌≤6␍␊├▒␋┌4C

[jobnoorman]

Fix memory leak reported by ASan.

The .debug_line_str section wasn't processed by RuntimeDyld before. Since
JITLink would err without it, this patch modified BOLT to allocate it using the
JIT memory manager. However, it was also still manually allocated by
DwarfLineTable, ultimately resulting in a memory leak. This updates makes sure
.debug_line_str is *only* allocated by the memory manager.

[jobnoorman]

In D147544#4263504, @rafauler wrote:

In one of our large tests on big x86 binaries, it's also firing this assertion:

JITLinkLinker.cpp:52: void llvm::bolt::(anonymous namespace)::reassignSectionAddress(jitlink::LinkGraph &, const llvm::bolt::BinarySection &, uint64_t): Assertion `JLSection && "cannot find section in LinkGraph"' failed

I took a look and it looks like the sectionID of the second argument (the binarysection object) is somehow storing garbage data. That's why I took a look at our asan builds. I got this when printing the section name it was trying to fetch from the link graph:

(...)
(earlier correct call) reassignsection address for id: .local.text.funcname/1
(buggy one) reassignsection address for id: ),:$R↓_ZZN5°⎺┌┌≤6␍␊├▒␋┌4C

Thanks for testing this so thoroughly, @rafauler!

Even though I fixed the ASan issues, I'm not convinced this assertion is caused by that. If it's still triggered, would it be possible to share a binary that allows me to reproduce this locally?

[rafauler]

In D147544#4264602, @jobnoorman wrote:

In D147544#4263504, @rafauler wrote:

In one of our large tests on big x86 binaries, it's also firing this assertion:

JITLinkLinker.cpp:52: void llvm::bolt::(anonymous namespace)::reassignSectionAddress(jitlink::LinkGraph &, const llvm::bolt::BinarySection &, uint64_t): Assertion `JLSection && "cannot find section in LinkGraph"' failed

I took a look and it looks like the sectionID of the second argument (the binarysection object) is somehow storing garbage data. That's why I took a look at our asan builds. I got this when printing the section name it was trying to fetch from the link graph:

(...)
(earlier correct call) reassignsection address for id: .local.text.funcname/1
(buggy one) reassignsection address for id: ),:$R↓_ZZN5°⎺┌┌≤6␍␊├▒␋┌4C

Thanks for testing this so thoroughly, @rafauler!

Even though I fixed the ASan issues, I'm not convinced this assertion is caused by that. If it's still triggered, would it be possible to share a binary that allows me to reproduce this locally?

I can't share this binary. Luckily ASan was able to pinpoint the problem as a use-after-free. I don't know if it helps, but here is the stack trace: https://pastebin.com/MWTFvYPn

If that stack trace doesn't help, when I have more time, I'll take a closer look at it.

[lhames]

In D147544#4255091, @Amir wrote:

...
BOLT operates per-function, the optimized function is placed in a newly-created .text section (or is split between .text and .text.cold), or, if the function is not updated, it's copied to .bolt.org.text and only the external references are patched.

Thanks @Amir!

This patch is impressive, and I think I see how bolt is (ab)using the JIT linker to do its job. I'll just mention two final things from my perspective: (1) you should expect more API churn in JITLink compared to RuntimeDyld, given that JITLink is being more actively developed (though I don't expect it to be prohibitive given the JITLink API surface you're using), (2) JITLink is missing some relocations, e.g. for the static relocation model. They're easy to add, we just haven't had anyone ask for them yet. If you hit missing relocations -- that's what's going on.

• Lang.

[rafauler]

In D147544#4270631, @lhames wrote:

In D147544#4255091, @Amir wrote:

...
BOLT operates per-function, the optimized function is placed in a newly-created .text section (or is split between .text and .text.cold), or, if the function is not updated, it's copied to .bolt.org.text and only the external references are patched.

Thanks @Amir!

This patch is impressive, and I think I see how bolt is (ab)using the JIT linker to do its job. I'll just mention two final things from my perspective: (1) you should expect more API churn in JITLink compared to RuntimeDyld, given that JITLink is being more actively developed (though I don't expect it to be prohibitive given the JITLink API surface you're using), (2) JITLink is missing some relocations, e.g. for the static relocation model. They're easy to add, we just haven't had anyone ask for them yet. If you hit missing relocations -- that's what's going on.

• Lang.

If it is not the intent of JITLink to support such use cases, and given that BOLT has a unique use case on these components, one thing we can consider doing is to just copy RuntimeDyld to bolt's folder as a specific bolt-only private linker. In that way, BOLT doesn't get in the way of deprecating RuntimeDyld and doesn't put unwanted requirements on the JITLink API.

[jobnoorman]

Fix use-after-free in RewriteInstance::mapCodeSections (same fix as in D148427).

[jobnoorman]

In D147544#4270547, @rafauler wrote:

I can't share this binary. Luckily ASan was able to pinpoint the problem as a use-after-free. I don't know if it helps, but here is the stack trace: https://pastebin.com/MWTFvYPn

Thanks for the stack trace! It helped me pinpoint the issue. Interestingly, the issue also exists on main but probably only causes problems now since SectionID is a std::string instead of an integer.

[jobnoorman]

In D147544#4270637, @rafauler wrote:

If it is not the intent of JITLink to support such use cases, and given that BOLT has a unique use case on these components, one thing we can consider doing is to just copy RuntimeDyld to bolt's folder as a specific bolt-only private linker. In that way, BOLT doesn't get in the way of deprecating RuntimeDyld and doesn't put unwanted requirements on the JITLink API.

While this would solve the immediate RuntimeDyld issue, I feel this approach would be suboptimal since it puts the burden of maintaining the linker entirely on the shoulders of the BOLT devs. For example, RISC-V support in RuntimeDyld is incomplete and would need to be implemented in isolation as part of BOLT. Besides BOLT not benefiting from an externally maintained project, I believe the reverse is also true: using an external linker like JITLink will also benefit that project (e.g., a few minor fixes have been upstreamed while developing this patch).

Another example of this is support for linker relaxation; something that would probably be desirable to have on RISC-V in BOLT. Since this is rather subtle to implement correctly, it makes much more sense to me to do this as part of something like JITLink than privately in BOLT.

As for putting unwanted requirements on the JITLink API, I don't think this patch actually does that (please correct me if I'm wrong @lhames). I believe the way LinkGraphs are created and linked is entirely within the intended use of the API. The only thing that is "weird" is the way the result of linking is used (i.e., read back from memory instead of being executed). But I don't think this should put any unwanted requirements on JITLink.

[alexander-shaposhnikov]

bolt/lib/Rewrite/JITLinkLinker.cpp

52

(just in case) is it true that (at the moment) we have only one block within JLSection ?

[alexander-shaposhnikov]

bolt/lib/Rewrite/JITLinkLinker.cpp

52

(in non-relocation mode)

[rafauler]

I'm currently investigating why this diff causes BOLT to output different eh_frame sections. It's very important for BOLT to correctly support these. I don't have all the answers yet, but I did notice a few weird things I'll comment in the code, so you can help me understand what's happening.

bolt/lib/Rewrite/JITLinkLinker.cpp

91

First thing I noticed is that the size of our eh_frame is missing a few bytes. I thought that maybe we're missing the NULL terminator.

Shouldn't we add something like:

Config.PrePrunePasses.push_back(jitlink::EHFrameNullTerminator(".eh_frame"));

here? But then I did that and noticed that the terminators we originally had were.. ugh... maybe illegal (in the middle of the eh_frame). So we can probably live without this, but bear in mind that if there is no terminator here, eh_frame will miscompare (versus the previous version).

This version currently has a final terminator in the end of the eh_frame so that should be fine.

bolt/lib/Rewrite/RewriteInstance.cpp

1664

Second thing I noticed was this. This just looks wrong because here we're adding a reloc against a new symbol Sym, but then we're adding its own Value as an addend too. That Value probably should be zero, instead:

RelocatedEHFrameSection->addRelocation(Offset, Sym, RelType, 0);

But then I did that and .eh_frame still looks completely different (the addresses). I'm not sure what's happening.

To see that, you need to build a test case with exceptions with two versions of BOLT, with and without your change, and then compare the eh_frame sections in them. To dump the eh_frame contents you can use a utility such as "eu-readelf -e binary".

Finally, to check that your JITLink enabled BOLT is generating the same output of the old BOLT and thus, proving that this diff is safe, I suggest using this utility: https://github.com/llvm/llvm-project/blob/main/bolt/utils/llvm-bolt-wrapper.py

This utility basically replaces llvm-bolt in your binary folder, and when you run "check-bolt", it will trigger two BOLT runs, with and without your cahnge, and then compare the binaries produced by both BOLTs. That's essentially part of the testing that we do when we have to approve a diff on BOLT, we take a look if the diff is not unintentionally changing some parts of the binary that it shouldn't, otherwise it's not safe to go to production.

Essentially, your diff is blocked atm because it is generating different eh_frame contents in x86 in a way that doesn't look right.

[jobnoorman]

Fix .eh_frame addresses.

[jobnoorman]

bolt/lib/Rewrite/RewriteInstance.cpp

1664

Second thing I noticed was this. This just looks wrong because here we're adding a reloc against a new symbol Sym, but then we're adding its own Value as an addend too. That Value probably should be zero, instead:

RelocatedEHFrameSection->addRelocation(Offset, Sym, RelType, 0);

But then I did that and .eh_frame still looks completely different (the addresses). I'm not sure what's happening.

This was indeed the problem. However, when setting the addend to 0, the symbol will get patched with the address of the relocated function and not with the original value (which is what BOLT wants iiuc). Hence the different addresses. The "trick" I used to fix this is to create a symbol at address 0x0 and then set the addend to the value. As my comments explain, this is is necessary as JITLink crashes when adding an ABS symbol as was done before.

To see that, you need to build a test case with exceptions with two versions of BOLT, with and without your change, and then compare the eh_frame sections in them. To dump the eh_frame contents you can use a utility such as "eu-readelf -e binary".

TIL about eu-readelf, thanks! :)

Finally, to check that your JITLink enabled BOLT is generating the same output of the old BOLT and thus, proving that this diff is safe, I suggest using this utility: https://github.com/llvm/llvm-project/blob/main/bolt/utils/llvm-bolt-wrapper.py

I've only manually tested using a simple binary for now and will try this asap.

Essentially, your diff is blocked atm because it is generating different eh_frame contents in x86 in a way that doesn't look right.

The null-terminator is now the only difference I see on a test binary.

[jobnoorman]

I've been investigating the remaining differences using llvm-bolt-wrapper.py and they seem to fall in three categories.

main:
  [ 8] .text.cold.2      PROGBITS        0000000000600080 400080 000014 00  AX  0   0 64
cmp:
  [ 8] .text.cold.3      PROGBITS        0000000000600080 400080 00000a 00  AX  0   0 64

headers mismatch

This seems serious at a first glance but turned out to be caused by the order in which JITLink processes sections. Sections are stored in a DenseMap so iteration doesn't happen in insertion order (which seems to happen in RuntimeDyld). I've attached a patch to use a VectorMap instead which causes these errors to go away (the patch is just for testing of course :))

0001-Deterministic-JITLink-section-order.patch1 KBDownload

main:
  [ 3] .eh_frame         PROGBITS        00000000004001c0 2001c0 00007c 00   A  0   0  8
cmp:
  [ 3] .eh_frame         PROGBITS        00000000004001c0 2001c0 000074 00   A  0   0  8

headers mismatch

The different size of .eh_frame is caused by the difference in the way terminators are added. iiuc, this is benign.

main:
  Start of section headers:          4198288 (bytes into file)
cmp:
  Start of section headers:          4198272 (bytes into file)

headers mismatch

I believe this is simply a consequence of the previous difference. The section headers are stored after the sections. I checked a few files manually and the difference in the section header offset equals the size difference of .eh_frame aligned to 4 bytes.

[jobnoorman]

bolt/lib/Rewrite/JITLinkLinker.cpp

52

Yes, that seems to be the case (assert(JLSection->blocks_size() == 1) doesn't break anything).

IIRC, I added this when still using JITLink's built-in .eh_frame and PLT/GOT support which generates sections with multiple blocks. I'm not sure if this could ever happen when creating a LinkGraph from an ELF file so we could add the above assert and simplify this function a bit.

[rafauler]

We can't have non-determinism in BOLT

BOLT, and compilers in general, need to be deterministic. I understand how that might be overlooked in a JIT, but for a static tool, we can't generate a different binary given the input hasn't changed. There are build tools that rely on this. For example, you might want to compute the hash on the output binary, but if your build system is non-deterministic and generate a different binary in a new run, then there's no point in hashing the binaries. I would argue that even for a JIT, having the layout being non-deterministic is undesirable. Suppose you're debugging something and want to restart the run, it is bad if the layout changes and your bug goes away or manifests differently, in case of a memory bug.

I saw that you provided a patch to make the section order deterministic. I used that, and also used a change to insert the null terminator. And then I got the binaries to match perfectly for some tests. But what's weird is that that happened only for some runs. Other runs would generate un-matching binaries, which is something that we can't have in BOLT. I didn't investigate why is that happening, but it is because of my change to add the prepass to add the null terminator. But assuming we don't have this change, if the section orders aren't deterministic, we also can't have that, even if the code is correct.

[jobnoorman]

In D147544#4285166, @rafauler wrote:

We can't have non-determinism in BOLT

Agreed.

I dug a bit deeper into this and fortunately, the order of output sections is already controlled by BOLT here. Since compareSections does not implement a strict total order, the result of stable_sort depends on the initial order of sections which is ultimately determined here. The source of non-determinism comes from the last comparison between the SectionNumber of sections which is based on the creation order of sections (e.g., here).

When using RuntimeDyld, the creation order of sections (that are created during emitAndLink) is out of BOLT's hands since RuntimeDyld calls ExecutableFileMemoryManager::allocateSection in some order. The way this order is determined is quite complex (e.g., it depends on symbol order) and I doubt any guarantees are made about it but at least it seems deterministic.

When using JITLink, whole LinkGraphs are allocated at once so BOLT has full control over the order in which sections are allocated. The current version of the patch simply uses the order of LinkGraph::sections() which is non-deterministic since a DenseMap is used to store sections. However, we can make it deterministic by first sorting the sections using any deterministic property (e.g., their names). Note that this doesn't guarantee the same order as BOLT currently produces but it will be deterministic.

The other way to make it deterministic would of course be to apply the patch I posted above. Since this also happens to produce the same section order as current BOLT, and hence simplifies verification of this patch, I would like to ask your opinion about this, @lhames. Would it make sense for JITLink to make the section order deterministic? The patch above replaces the DenseMap used to store sections with a MapVector preserving insertion order (which, when creating a link graph from an object, makes sure the sections are stored in the same order as in the object file).

I saw that you provided a patch to make the section order deterministic. I used that, and also used a change to insert the null terminator. And then I got the binaries to match perfectly for some tests. But what's weird is that that happened only for some runs. Other runs would generate un-matching binaries, which is something that we can't have in BOLT. I didn't investigate why is that happening, but it is because of my change to add the prepass to add the null terminator. But assuming we don't have this change, if the section orders aren't deterministic, we also can't have that, even if the code is correct.

About the null terminator: when looking at .eh_frame in some test binaries, the terminator is already inserted by the current version of the patch. The difference with current BOLT is that some spurious null terminators in the middle of the section are not there anymore. So why are you trying to add the prepass to insert the terminator?

When you refer to "other runs", do you mean subsequent runs on the same binary? That would obviously be a serious bug that needs to be fixed. Or are you referring to runs on other binaries? The only binary differences I still see in the test cases are caused by the .eh_frame null terminator difference which, iiuc, is benign.

[rafauler]

So why are you trying to add the prepass to insert the terminator?

Just as a practical way of comparing this patch with production BOLT. If we mismatch, then I have 80 different binaries mismatching that I need to look at. Even for a single binary, for some of our internal large production binaries, it's not possible to analyze all differences manually because once you produce a different eh_frame size, you might have code references to sections past eh_frame that are now offset by a few bytes, for example, and then you have code mismatches all over the place. There are different ways to validate your diff, but I just thought that the fastest way would be to produce equal size, equal contents eh_frame, and I thought it would be simple enough to just add a new prepass, and was surprised with the nondeterminism.

We've dealt with large changes to BOLT a few times before, and the easiest way to guarantee that we're not missing anything was always to change the code until it matches the binaries, so we're fully aware of all of the changes that are going in production.

This diff is a rather large change to how things are done and I basically need to guarantee this is not going to surprise us when this is productionized.

[rafauler]

I think I forgot to clarify this: when I mentioned nondeterminism when running the terminator insertion pass, I meant same binary, same inputs, same everything, but getting different outputs from run to run. I meant:

$ ninja check-bolt
87 tests failed

Then again
$ ninja check-bolt
84 tests failed

without changing anything.

Also, to be clear, I never intend to commit this (to use the null insertion pass), this is just for validation purposes. I can try to pad eh_frame to have the same size as before in another way. I was just curious why is that happening, but didn't have time to look into it.

But another source of non-determinism that we can't have is iteration over DenseMaps, because those change if we build a new version of BOLT and cause a bunch of headaches when we are qualifying a new release.

[jobnoorman]

Make use of JITLink fully deterministic:

• Iterate sections in order of creation (by ordinal).
• Iterate blocks by address

[jobnoorman]

In D147544#4288096, @rafauler wrote:

We've dealt with large changes to BOLT a few times before, and the easiest way to guarantee that we're not missing anything was always to change the code until it matches the binaries, so we're fully aware of all of the changes that are going in production.

Makes sense. I got the patch in a state where it now produces 100% matching binaries on all test cases (with a few tweaks, see below).

In D147544#4288256, @rafauler wrote:

I think I forgot to clarify this: when I mentioned nondeterminism when running the terminator insertion pass, I meant same binary, same inputs, same everything, but getting different outputs from run to run.

You were right that using the terminator insertion pass should make this patch match the current behavior of BOLT. The non-determinism you observed was caused by the blocks of sections being stored in a DenseSet in JITLink. I fixed this by ordering blocks by address when iterating.

But another source of non-determinism that we can't have is iteration over DenseMaps, because those change if we build a new version of BOLT and cause a bunch of headaches when we are qualifying a new release.

I discovered that JITLink keeps track of the creation order of sections so this source of non-determinism has also been fixed now.

To make binaries produced by this patch match the ones BOLT currently produces, a few extra tweaks are necessary. I've attached them in a separate patch because I believe they should only be necessary for validation purposes. This patch does the following:

• Switch the order of .bss.bolt.extra and .data.bolt.extra sections. For some reason, RuntimeDyLd did not allocate those sections in the same order as they are in the input file.
• Use 1-byte allocations for empty sections. RuntimeDyld forces the minimum allocation size to 1 byte. I believe it should be fine to skip this and simply allocate an empty section. (This happens with empty .text or .text.cold sections in X86/bb-with-two-tail-calls.s and X86/bug-reorder-bb-jrcxz.s.)
• Explicitly ignore .note.gnu.property sections. This section comes from instr.cpp.o and is marked as allocatable in the ELF file but was still skipped by RuntimeDyld.
• Add .eh_frame terminator passes.

0001-RuntimeDyld-simulation.patch4 KBDownload

To make all binaries match completely deterministically, the reference version of BOLT also needs to be tweaked, unfortunately: when RuntimeDyld forces the minimum allocation to 1 byte (as explained above), it doesn't actually initialize that byte. This causes the section contents of empty sections to sometimes mismatch. The following patch fixes this.

0001-Make-RuntimeDyld-deterministic.patch979 BDownload

[rafauler]

bolt/lib/Rewrite/ExecutableFileMemoryManager.cpp

34

Some warnings from clang here:

ExecutableFileMemoryManager.cpp:34:10: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]

return std::move(Sections);
       ^

ExecutableFileMemoryManager.cpp:34:10: note: remove std::move call here

return std::move(Sections);

bolt/lib/Rewrite/JITLinkLinker.cpp

192

some warning from clang here:

JITLinkLinker.cpp:191:10: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]

return std::move(Blocks);
       ^

note: remove std::move call here

return std::move(Blocks);

[jobnoorman]

Fix Clang warnings.

[jobnoorman]

bolt/lib/Rewrite/ExecutableFileMemoryManager.cpp

34

Woops, darn you GCC :)

[rafauler]

Noticed another difference here between RuntimeDyld vs. JITLink. Writing here for documentation only, I think it is harmless.

My hugify lib's file are built with one additional .rodata.extra section (.rodata.str1.16.bolt.extra.1) that is entirely useless. The error strings originally stored in this section are materialized in code with movabs to stack space (I think this is an unintended side effect of the way the code is written, see https://github.com/llvm/llvm-project/blob/main/bolt/runtime/hugify.cpp#L119, which forces the compiler to allocate the error message in the stack), rendering this rodata section unnecessary/unreferenced, but still present in the object file. RuntimeDyld doesn't allocate this unnecessary section, but JITLink does allocate.

I'm not sure if RuntimeDyld is being smart about it, or if its a bug. But the binary is not going to crash because we have no relocs against this unnecessary rodata section.

[maksfb]

In D147544#4293847, @rafauler wrote:

Noticed another difference here between RuntimeDyld vs. JITLink. Writing here for documentation only, I think it is harmless.

My hugify lib's file are built with one additional .rodata.extra section (.rodata.str1.16.bolt.extra.1) that is entirely useless. The error strings originally stored in this section are materialized in code with movabs to stack space (I think this is an unintended side effect of the way the code is written, see https://github.com/llvm/llvm-project/blob/main/bolt/runtime/hugify.cpp#L119, which forces the compiler to allocate the error message in the stack), rendering this rodata section unnecessary/unreferenced, but still present in the object file. RuntimeDyld doesn't allocate this unnecessary section, but JITLink does allocate.

I'm not sure if RuntimeDyld is being smart about it, or if its a bug. But the binary is not going to crash because we have no relocs against this unnecessary rodata section.

Right. RuntimeDyld will not emit a section without relocations. E.g., for .debug_line, we have to make the section allocatable (https://github.com/llvm/llvm-project/blob/99cfaf0d5ed68d5d4e292fc87a10b1bb26201787/bolt/lib/Core/BinaryEmitter.cpp#L190-L199) and then force a dummy relocation against it (https://github.com/llvm/llvm-project/blob/5f2b0892d5b732a822728f96a57144f6542c155e/bolt/lib/Core/DebugData.cpp#L1631-L1637).

Sounds like with JITLink at least the hack for the relocation is not needed. @lhames, could you please confirm that it's by design?

Overall, the switch to JITLink is going full-swing and it might be too late to stop it :) Besides BOLT, what other major projects are going to be (or perhaps already are) using JITLink/ORC?

@yota9, @yavtuk, @treapster, did you test this diff on aarch64?

We also need to compare BOLT's CPU time and memory for this change before it can be accepted.

[rafauler]

bolt/lib/Rewrite/RewriteInstance.cpp

4017–4018

Delete duplicate code that was already committed.

[rafauler]

Getting

BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:280: R_AARCH64_CONDBR19

on large AArch64 binaries (SPEC CPU2017 binaries)

[rafauler]

We have one MachO test failing. this is currently not available on llvm repo, I posted a diff with them:

D149113

Not sure if we want to commit these because they are in pure binary form, albeit very small. The largest one is 80kb.

[jobnoorman]

In D147544#4294147, @rafauler wrote:

Getting

BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:280: R_AARCH64_CONDBR19

on large AArch64 binaries (SPEC CPU2017 binaries)

Posted D149138 for this.

[jobnoorman]

In D147544#4294205, @rafauler wrote:

We have one MachO test failing. this is currently not available on llvm repo, I posted a diff with them:

I think this is a bug in BOLT on main. See my comments in D149113. I'll try to figure out what's going on there.

[jobnoorman]

In D147544#4293906, @maksfb wrote:

Right. RuntimeDyld will not emit a section without relocations. E.g., for .debug_line, we have to make the section allocatable (https://github.com/llvm/llvm-project/blob/99cfaf0d5ed68d5d4e292fc87a10b1bb26201787/bolt/lib/Core/BinaryEmitter.cpp#L190-L199) and then force a dummy relocation against it (https://github.com/llvm/llvm-project/blob/5f2b0892d5b732a822728f96a57144f6542c155e/bolt/lib/Core/DebugData.cpp#L1631-L1637).

Sounds like with JITLink at least the hack for the relocation is not needed. @lhames, could you please confirm that it's by design?

In JITLink, the pruning of sections is based on symbol liveness. Only sections (blocks actually) that have live symbols pointing to them will be kept for linking. Which symbols are live, is controlled by BOLT. See markSectionsLive in JITLinkLinker.cpp in this patch.

The hack you are talking about has indeed been removed by this patch.

[jobnoorman]

Remove duplicate code.

[jobnoorman]

In D147544#4295243, @jobnoorman wrote:

In D147544#4294205, @rafauler wrote:

We have one MachO test failing. this is currently not available on llvm repo, I posted a diff with them:

I think this is a bug in BOLT on main. See my comments in D149113. I'll try to figure out what's going on there.

The problem is that on main, MachORewriteInstance::emitAndLink never calls RuntimeDyld::finalizeWithMemoryManagerLocking so relocations are never processed. The following patch fixes this and with it, BOLT on main produces the same (correct) output as with the JITLink patch. I'm not sure if it's worth it to post this patch for review.

macho.patch564 BDownload

[Amir]

In D147544#4291902, @jobnoorman wrote:

In D147544#4288096, @rafauler wrote:

We've dealt with large changes to BOLT a few times before, and the easiest way to guarantee that we're not missing anything was always to change the code until it matches the binaries, so we're fully aware of all of the changes that are going in production.

Makes sense. I got the patch in a state where it now produces 100% matching binaries on all test cases (with a few tweaks, see below).

In D147544#4288256, @rafauler wrote:

I think I forgot to clarify this: when I mentioned nondeterminism when running the terminator insertion pass, I meant same binary, same inputs, same everything, but getting different outputs from run to run.

You were right that using the terminator insertion pass should make this patch match the current behavior of BOLT. The non-determinism you observed was caused by the blocks of sections being stored in a DenseSet in JITLink. I fixed this by ordering blocks by address when iterating.

But another source of non-determinism that we can't have is iteration over DenseMaps, because those change if we build a new version of BOLT and cause a bunch of headaches when we are qualifying a new release.

I discovered that JITLink keeps track of the creation order of sections so this source of non-determinism has also been fixed now.

To make binaries produced by this patch match the ones BOLT currently produces, a few extra tweaks are necessary. I've attached them in a separate patch because I believe they should only be necessary for validation purposes. This patch does the following:

• Switch the order of .bss.bolt.extra and .data.bolt.extra sections. For some reason, RuntimeDyLd did not allocate those sections in the same order as they are in the input file.
• Use 1-byte allocations for empty sections. RuntimeDyld forces the minimum allocation size to 1 byte. I believe it should be fine to skip this and simply allocate an empty section. (This happens with empty .text or .text.cold sections in X86/bb-with-two-tail-calls.s and X86/bug-reorder-bb-jrcxz.s.)
• Explicitly ignore .note.gnu.property sections. This section comes from instr.cpp.o and is marked as allocatable in the ELF file but was still skipped by RuntimeDyld.
• Add .eh_frame terminator passes.

0001-RuntimeDyld-simulation.patch4 KBDownload

To make all binaries match completely deterministically, the reference version of BOLT also needs to be tweaked, unfortunately: when RuntimeDyld forces the minimum allocation to 1 byte (as explained above), it doesn't actually initialize that byte. This causes the section contents of empty sections to sometimes mismatch. The following patch fixes this.

0001-Make-RuntimeDyld-deterministic.patch979 BDownload

Thanks for the second patch, we can incorporate it since it addresses an existing non-determinism in BOLT (see https://github.com/llvm/llvm-project/issues/59008)

[Amir]

In D147544#4295564, @jobnoorman wrote:

In D147544#4295243, @jobnoorman wrote:

In D147544#4294205, @rafauler wrote:

We have one MachO test failing. this is currently not available on llvm repo, I posted a diff with them:

I think this is a bug in BOLT on main. See my comments in D149113. I'll try to figure out what's going on there.

The problem is that on main, MachORewriteInstance::emitAndLink never calls RuntimeDyld::finalizeWithMemoryManagerLocking so relocations are never processed. The following patch fixes this and with it, BOLT on main produces the same (correct) output as with the JITLink patch. I'm not sure if it's worth it to post this patch for review.

macho.patch564 BDownload

This fix should also be incorporated, please submit it as a separate diff.

[lhames]

Hi All,

Looks like there has been a lot of activity on this -- apologies, but I'm on vacation at the moment and don't have time to catch up on everything. I'll be back in the office May 8th.

I did catch:

We can't have non-determinism in BOLT...

Non-determinism hasn't caused any problems for the JIT so far, but determinism would be nice to have and I don't think it'd add too much in terms of cost. I'd be happy to discuss this.

[jobnoorman]

In D147544#4296001, @Amir wrote:

Thanks for the second patch, we can incorporate it since it addresses an existing non-determinism in BOLT (see https://github.com/llvm/llvm-project/issues/59008)

D149243

In D147544#4296002, @Amir wrote:

macho.patch564 BDownload

This fix should also be incorporated, please submit it as a separate diff.

D149244

[jobnoorman]

Rebase

[rafauler]

Hi all,

It would be nice if somebody with better access to AArch64 binaries test this diff. I took another look at this diff and most of my small AArch64 tests look fine, except for one that crashes with:

"BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:279: R_AARCH64_TSTBR14"

But I only have trivial SPEC programs built for AArch64, probably @yota9 is in a better position to evaluate this on more complex AArch64 binaries.

I have two issues on the X86 side that needs investigating: first, in most tests we look fine, but there is one binary that is mismatching eh_frame contents. Upon further inspection, I noticed that addresses of some functions in FDEs look off:

To the left, we have reference BOLT, to the right, BOLT with this patch. This address 0x1c0c322d doesn't exist in the binary. This is HHVM binary (c++ with exceptions), built with GCC, being processed with "-lite=1 -reorder-functions=hfsort", but the issue might reproduce in any binary with lite=1 and eh_frame. It looks like addresses of functions that are not hot are getting corrupted, but it's just a guess.

Second issue is a large regression in wall time when running with assertions enabled. Running BOLT with -time-rewrite option and optimizing clang itself, we have:

Original BOLT:

 ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
33.3125 ( 32.0%)   0.2820 (  0.4%)  33.5946 ( 18.9%)  33.5972 ( 51.4%)  emit and link

BOLT with JITLink:

 ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
134.1916 ( 64.9%)   0.1047 (  0.1%)  134.2963 ( 47.2%)  134.3034 ( 79.6%)  emit and link

We run BOLT with assertions and they're usually not _that_ expensive.

[jobnoorman]

In D147544#4348350, @rafauler wrote:

Hi all,

It would be nice if somebody with better access to AArch64 binaries test this diff. I took another look at this diff and most of my small AArch64 tests look fine, except for one that crashes with:

"BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:279: R_AARCH64_TSTBR14"

Opened D150778 for this.

[jobnoorman]

In D147544#4348350, @rafauler wrote:

Second issue is a large regression in wall time when running with assertions enabled. Running BOLT with -time-rewrite option and optimizing clang itself, we have:

Original BOLT:

 ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
33.3125 ( 32.0%)   0.2820 (  0.4%)  33.5946 ( 18.9%)  33.5972 ( 51.4%)  emit and link

BOLT with JITLink:

 ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
134.1916 ( 64.9%)   0.1047 (  0.1%)  134.2963 ( 47.2%)  134.3034 ( 79.6%)  emit and link

We run BOLT with assertions and they're usually not _that_ expensive.

I traced to performance issue to one particularly expensive assert in JITLink.

I posted D150874 to propose a solution but since it adds complexity just to improve the performance of an assert, I'm not quite sure if that will get accepted.

[jobnoorman]

In D147544#4348350, @rafauler wrote:

I have two issues on the X86 side that needs investigating: first, in most tests we look fine, but there is one binary that is mismatching eh_frame contents. Upon further inspection, I noticed that addresses of some functions in FDEs look off:

To the left, we have reference BOLT, to the right, BOLT with this patch. This address 0x1c0c322d doesn't exist in the binary. This is HHVM binary (c++ with exceptions), built with GCC, being processed with "-lite=1 -reorder-functions=hfsort", but the issue might reproduce in any binary with lite=1 and eh_frame. It looks like addresses of functions that are not hot are getting corrupted, but it's just a guess.

Since HHVM is an open source project, I was wondering if it's possible to share a binary? I'm having trouble getting HHVM compiled myself and the binaries I find online don't have the necessary symbols/relocations. I'm also not able to reproduce this with simple binaries containing an .eh_frame.

[rafauler]

bolt/lib/Rewrite/RewriteInstance.cpp

1665

I was digging a bit into the problem of bad eh_frames for HHVM and discovered this is the problem. HHVM has some symbols that are initially set to zero but later BOLT will relocate them. These are special hot_start/hot_end markers that delimit the location of hot code in case your application really wants to know the location of hot functions (older versions of HHVM were doing self huge page allocation based on that) . So these symbols are registered at address zero in BinaryContext, but we will move them later. When you create a reloc here that points to a MCSymbol returned by BinaryContext::getOrCreateGlobalSymbol(), BinaryContext checks its internal maps, realizes it already has a symbol registered at address zero, and then returns to you "hot_end", which is unfortunately a special symbol that will not evaluate to zero after binary rewriting is complete.

If you really want a symbol that always evaluates to zero and will never be relocated, I guess you will need to create a special MCSymbol whenever we initialize BinaryContext and return that in a special getter, and then adapt our symbol resolver to, whenever the linker wants the address of the special zero marker, set it to zero. Maybe there are better solutions, but that's all I can think of at the moment without digging deeper, unless you can make the absolute relocs work in JITLink.

[jobnoorman]

Remove JITLink ABS symbol workaround.

[jobnoorman]

bolt/lib/Rewrite/RewriteInstance.cpp

1665

I'm not entirely sure what changed in JITLink but this workaround doesn't seem necessary anymore so I removed it. Could you check if this solves your HHVM issue?

Note that for some reason, some sections changed order when using RuntimeDyld so I had to update the simulation patch to make sure llvm-bolt-wrapper.py passes on all tests:

0001-RuntimeDyld-simulation.patch4 KBDownload

[rafauler]

LGTM

[jobnoorman]

Thanks a lot for reviewing this @rafauler!

I guess we're still waiting for confirmation that everything is fine on AArch64 before landing this? @yota9, @yavtuk, @treapster: did you have time to test this patch?

[rafauler]

Thank you for your patience and excellent work. You're free to commit, just give me a heads up because I'll have to land another commit removing that slow assertion internally. If AArch64 folks want to test the diff, we can wait a bit longer.

[jobnoorman]

Alright, then I'll commit in 24h (around 10:00 UTC on May 24) if we don't hear any complaints from the AArch64 folks.

[yota9]

Hello! Sorry, for long reply, I'm on vacation. @Elvina would test this diff :)

[Elvina]

Hello! I tested the patch on our binaries, and everything looks fine. But on the small tests, as Rafael mentioned, I also ran into "BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:274: R_AARCH64_ADR_PREL_L021 " crash. It seems that the problem appears only when using lld linker (small tests use it).

[jobnoorman]

In D147544#4366578, @Elvina wrote:

Hello! I tested the patch on our binaries, and everything looks fine.

Thanks for testing this!

But on the small tests, as Rafael mentioned, I also ran into "BOLT-ERROR: JITLink failed: Unsupported aarch64 relocation:274: R_AARCH64_ADR_PREL_L021 " crash. It seems that the problem appears only when using lld linker (small tests use it).

Posted D151305 for this. I'll wait until that one lands before committing this patch.

[jobnoorman]

Since D151305 landed, I think this can finally be committed now.

@rafauler: since you wanted a heads-up: I will probably commit when I get back in the office tomorrow, 15/06 around 11am CEST.

[ayermolo]

bolt/lib/Core/BinaryEmitter.cpp

199

Why was this added?

[jobnoorman]

bolt/lib/Core/BinaryEmitter.cpp

199

JITLink only accepts relocations that reference symbols in allocatable sections. More specifically, this assert is triggered when not marking .debug_line_str allocatable.

An example of a failing test without this is X86/dwarf5-debug-line.s. Here, the sections .local.text.main and .debug_line have relocations pointing to .debug_line_str.

[treapster]

Hello, @jobnoorman, out team is currently evaluating possible improvements to LongJmp pass, and we're wondering what are the opportunities for doing relaxation/stub insertion on JITLink side? I see currently JITLink can build PLT with TableManager and relax some calls to it and GOT accesses, do you think it's possible/worth it to to extend this functionality further to support a more general replacement of instructions as we do in LongJMP? It seems it already works like that for RISCV because of D149526, does that mean we can implement similar patches for AArch64/x86 and throw LongJmp away? I guess it would be tricky for X86 because of all the encoding mess, but i'm wondering what your thoughts are.

[rafauler]

In D147544#4473532, @treapster wrote:

Hello, @jobnoorman, out team is currently evaluating possible improvements to LongJmp pass, and we're wondering what are the opportunities for doing relaxation/stub insertion on JITLink side? I see currently JITLink can build PLT with TableManager and relax some calls to it and GOT accesses, do you think it's possible/worth it to to extend this functionality further to support a more general replacement of instructions as we do in LongJMP? It seems it already works like that for RISCV because of D149526, does that mean we can implement similar patches for AArch64/x86 and throw LongJmp away? I guess it would be tricky for X86 because of all the encoding mess, but i'm wondering what your thoughts are.

I think I already asked this but remind me again, why is LongJmp being used for X86? Is this golang support? We shouldn't use longjmp for X86. This pass was written for AArch64 and RISC's short ranged branches.

[treapster]

In D147544#4474522, @rafauler wrote:

I think I already asked this but remind me again, why is LongJmp being used for X86? Is this golang support?

Yes, it's golang support. @yota9, can you please remind us why we need LongJmp in golang?

[yota9]

Currently it is golang specific thing since we need to know exact layout to re-create golang-specific data. As we've discussed previously @rafaelauler it won't be submitted in open source and as a first step we would just relax all x86 instructions for golang.

[jobnoorman]

In D147544#4473532, @treapster wrote:

Hello, @jobnoorman, out team is currently evaluating possible improvements to LongJmp pass, and we're wondering what are the opportunities for doing relaxation/stub insertion on JITLink side? I see currently JITLink can build PLT with TableManager and relax some calls to it and GOT accesses, do you think it's possible/worth it to to extend this functionality further to support a more general replacement of instructions as we do in LongJMP? It seems it already works like that for RISCV because of D149526, does that mean we can implement similar patches for AArch64/x86 and throw LongJmp away? I guess it would be tricky for X86 because of all the encoding mess, but i'm wondering what your thoughts are.

I think it should be possible to do this in JITLink, at least for the normal range-extension thunks that a static AArch64 linker would insert. One difficulty I see is that unlike PLTTableManager, which adds all stubs to a new section, these thunks would need to be added in the middle of existing sections. I think this should be possible with LinkGraph::splitBlock but I don't know JITLink well enough to be sure. @lhames: do you have any thoughts on this?

Note that currently, doing this (as well as RISC-V linker relaxation) causes problems in BOLT because it doesn't handle the linker changing symbols well. See D154604 for context.

I'm not familiar with LongJmpPass but I did notice the following comment:

We pull this pass inside BOLT because here we can do a better job at stub inserting by manipulating the CFG, something linkers can't do.

So wouldn't you lose something if this was moved to JITLink?

[treapster]

In D147544#4477299, @jobnoorman wrote:

I'm not familiar with LongJmpPass but I did notice the following comment:

We pull this pass inside BOLT because here we can do a better job at stub inserting by manipulating the CFG, something linkers can't do.

So wouldn't you lose something if this was moved to JITLink?

You're right, losing CFG will make branches impossible to relax. We'll need MC layer to emit relocs for every branch which is apparently default behavior for RISC-V but not AArch64.
AFAIK currently JITLink doesn't relax branches for RISC-V, but if you intend to implement it, we can see how it works out and consider doing it for AArch64(starting with some MC option to emit R_AARCH64_JUMP26 and R_AARCH64_CONDBR19). As for updating the output values, it seems you mostly solved it in D154604.
@rafauler, do you think it's feasible to move LongJMP to JITLink if we force MC to generate branch relocs, and are there some other reasons to manipulate the actual CFG?

[rafauler]

In D147544#4497505, @treapster wrote:

In D147544#4477299, @jobnoorman wrote:

I'm not familiar with LongJmpPass but I did notice the following comment:

We pull this pass inside BOLT because here we can do a better job at stub inserting by manipulating the CFG, something linkers can't do.

So wouldn't you lose something if this was moved to JITLink?

You're right, losing CFG will make branches impossible to relax. We'll need MC layer to emit relocs for every branch which is apparently default behavior for RISC-V but not AArch64.
AFAIK currently JITLink doesn't relax branches for RISC-V, but if you intend to implement it, we can see how it works out and consider doing it for AArch64(starting with some MC option to emit R_AARCH64_JUMP26 and R_AARCH64_CONDBR19). As for updating the output values, it seems you mostly solved it in D154604.
@rafauler, do you think it's feasible to move LongJMP to JITLink if we force MC to generate branch relocs, and are there some other reasons to manipulate the actual CFG?

In theory, if we feed the linker with as many relocations as it needs, it should be able to do any job. There is no specific reason to do that with a CFG other than "it's nicer to write a pass that goes over a real IR instead of just going over a stream of bytes + relocations", IMO. However, maintaining LongJmp pass in BOLT has its own complications too, such as the need to manually reproduce how the binary is going to be written before it is actually written. One could change something in BOLT and make LongJmp out of sync. In this case, LongJmp could not longer be accurately predicting how far apart instructions are from each other and which ones need trampoline stubs to extend their range.

bolt/include/bolt/Core/BinaryContext.h

 class MCDisassembler;
 class MCInstPrinter;
 
 using namespace object;
 
 namespace bolt {
 
 class BinaryFunction;
-class ExecutableFileMemoryManager;
 
 /// Information on loadable part of the file.
 struct SegmentInfo {
   uint64_t Address;           /// Address of the segment in memory.
   uint64_t Size;              /// Size of the segment in memory.
   uint64_t FileOffset;        /// Offset in the file.
   uint64_t FileSize;          /// Size in file.
   uint64_t Alignment;         /// Alignment of the segment.
public:
   using binary_data_iterator = BinaryDataMapType::iterator;
   using binary_data_const_iterator = BinaryDataMapType::const_iterator;
   BinaryDataMapType BinaryDataMap;
 
   using FilteredBinaryDataConstIterator =
       FilterIterator<binary_data_const_iterator>;
   using FilteredBinaryDataIterator = FilterIterator<binary_data_iterator>;
 
-  /// Memory manager for sections and segments. Used to communicate with ORC
-  /// among other things.
-  std::shared_ptr<ExecutableFileMemoryManager> EFMM;
-
   StringRef getFilename() const { return Filename; }
   void setFilename(StringRef Name) { Filename = std::string(Name); }
 
   std::optional<StringRef> getFileBuildID() const {
     if (FileBuildID)
       return StringRef(*FileBuildID);
 
     return std::nullopt;

bolt/include/bolt/Core/BinarySection.h

bool IsFinalized{false};         // Has this section had output information
                                    // finalized?
   std::string OutputName;          // Output section name (if the section has
                                    // been renamed)
   uint64_t OutputAddress{0};       // Section address for the rewritten binary.
   uint64_t OutputSize{0};          // Section size in the rewritten binary.
   uint64_t OutputFileOffset{0};    // File offset in the rewritten binary file.
   StringRef OutputContents;        // Rewritten section contents.
   const uint64_t SectionNumber;    // Order in which the section was created.
-  unsigned SectionID{-1u};         // Unique ID used for address mapping.
+  std::string SectionID;           // Unique ID used for address mapping.
                                    // Set by ExecutableFileMemoryManager.
   uint32_t Index{0};               // Section index in the output file.
   mutable bool IsReordered{false}; // Have the contents been reordered?
   bool IsAnonymous{false};         // True if the name should not be included
                                    // in the output file.
 
   uint64_t hash(const BinaryData &BD,
                 std::map<const BinaryData *, uint64_t> &Cache) const;
const uint8_t *getOutputData() const {
     return reinterpret_cast<const uint8_t *>(getOutputContents().data());
   }
   StringRef getOutputContents() const { return OutputContents; }
   uint64_t getAllocAddress() const {
     return reinterpret_cast<uint64_t>(getOutputData());
   }
   uint64_t getOutputAddress() const { return OutputAddress; }
   uint64_t getOutputFileOffset() const { return OutputFileOffset; }
-  unsigned getSectionID() const {
+  StringRef getSectionID() const {
     assert(hasValidSectionID() && "trying to use uninitialized section id");
     return SectionID;
   }
-  bool hasValidSectionID() const { return SectionID != -1u; }
+  bool hasValidSectionID() const { return !SectionID.empty(); }
   bool hasValidIndex() { return Index != 0; }
   uint32_t getIndex() const { return Index; }
 
   // mutation
   void setOutputAddress(uint64_t Address) { OutputAddress = Address; }
   void setOutputFileOffset(uint64_t Offset) { OutputFileOffset = Offset; }
-  void setSectionID(unsigned ID) {
+  void setSectionID(StringRef ID) {
     assert(!hasValidSectionID() && "trying to set section id twice");
     SectionID = ID;
   }
   void setIndex(uint32_t I) { Index = I; }
   void setOutputName(const Twine &Name) { OutputName = Name.str(); }
   void setAnonymous(bool Flag) { IsAnonymous = Flag; }
 
   /// Emit the section as data, possibly with relocations.

bolt/include/bolt/Core/Linker.h

+//===- bolt/Core/Linker.h - BOLTLinker interface ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains the interface BOLT uses for linking.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef BOLT_CORE_LINKER_H
+#define BOLT_CORE_LINKER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/MemoryBufferRef.h"
+
+#include <cstdint>
+#include <functional>
+#include <optional>
+
+namespace llvm {
+namespace bolt {
+
+class BinarySection;
+
+class BOLTLinker {
+public:
+  using SectionMapper =
+      std::function<void(const BinarySection &Section, uint64_t Address)>;
+  using SectionsMapper = std::function<void(SectionMapper)>;
+
+  virtual ~BOLTLinker() = default;
+
+  /// Load and link \p Obj. \p MapSections will be called before the object is
+  /// linked to allow section addresses to be remapped. When called, the address
+  /// of a section can be changed by calling the passed SectionMapper.
+  virtual void loadObject(MemoryBufferRef Obj, SectionsMapper MapSections) = 0;
+
+  /// Return the address of a symbol or std::nullopt if it cannot be found.
+  virtual std::optional<uint64_t> lookupSymbol(StringRef Name) const = 0;
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif // BOLT_CORE_LINKER_H

bolt/include/bolt/Rewrite/ExecutableFileMemoryManager.h

 //===- bolt/Rewrite/ExecutableFileMemoryManager.h ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef BOLT_REWRITE_EXECUTABLE_FILE_MEMORY_MANAGER_H
 #define BOLT_REWRITE_EXECUTABLE_FILE_MEMORY_MANAGER_H
 
 #include "llvm/ADT/StringRef.h"
-#include "llvm/ExecutionEngine/RuntimeDyld.h"
+#include "llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h"
 #include <cstdint>
 #include <string>
 
 namespace llvm {
 
 namespace bolt {
 class BinaryContext;
 
 /// Class responsible for allocating and managing code and data sections.
-class ExecutableFileMemoryManager : public RuntimeDyld::MemoryManager {
+class ExecutableFileMemoryManager : public jitlink::JITLinkMemoryManager {
 private:
-  uint8_t *allocateSection(uintptr_t Size, unsigned Alignment,
-                           unsigned SectionID, StringRef SectionName,
-                           bool IsCode, bool IsReadOnly);
-  BinaryContext &BC;
-  bool AllowStubs;
+  void updateSection(const jitlink::Section &Section, uint8_t *Contents,
+                     size_t Size, size_t Alignment);
 
-  struct AllocInfo {
-    uint8_t *Address;
-    size_t Size;
-    size_t Alignment;
-  };
-  SmallVector<AllocInfo, 8> AllocatedSections;
+  BinaryContext &BC;
 
   // All new sections will be identified by the following prefix.
   std::string NewSecPrefix;
 
   // Name prefix used for sections from the input.
   std::string OrgSecPrefix;
 
 public:
   // Our linker's main purpose is to handle a single object file, created
   // by RewriteInstance after reading the input binary and reordering it.
   // After objects finish loading, we increment this. Therefore, whenever
   // this is greater than zero, we are dealing with additional objects that
   // will not be managed by BinaryContext but only exist to support linking
   // user-supplied objects into the main input executable.
   uint32_t ObjectsLoaded{0};
 
-  ExecutableFileMemoryManager(BinaryContext &BC, bool AllowStubs)
-      : BC(BC), AllowStubs(AllowStubs) {}
+  ExecutableFileMemoryManager(BinaryContext &BC) : BC(BC) {}
 
-  ~ExecutableFileMemoryManager();
+  void allocate(const jitlink::JITLinkDylib *JD, jitlink::LinkGraph &G,
+                OnAllocatedFunction OnAllocated) override;
 
-  uint8_t *allocateCodeSection(uintptr_t Size, unsigned Alignment,
-                               unsigned SectionID,
-                               StringRef SectionName) override {
-    return allocateSection(Size, Alignment, SectionID, SectionName,
-                           /*IsCode=*/true, true);
-  }
-
-  uint8_t *allocateDataSection(uintptr_t Size, unsigned Alignment,
-                               unsigned SectionID, StringRef SectionName,
-                               bool IsReadOnly) override {
-    return allocateSection(Size, Alignment, SectionID, SectionName,
-                           /*IsCode=*/false, IsReadOnly);
-  }
-
-  // Ignore TLS sections by treating them as a regular data section
-  TLSSection allocateTLSSection(uintptr_t Size, unsigned Alignment,
-                                unsigned SectionID,
-                                StringRef SectionName) override {
-    TLSSection Res;
-    Res.Offset = 0;
-    Res.InitializationImage = allocateDataSection(
-        Size, Alignment, SectionID, SectionName, /*IsReadOnly=*/false);
-    return Res;
-  }
-
-  bool allowStubAllocation() const override { return AllowStubs; }
-
-  /// Count processed objects and skip memory finalization.
-  bool finalizeMemory(std::string *ErrMsg) override {
-    ++ObjectsLoaded;
-    return false;
-  }
-
-  /// Ignore EH frames.
-  void registerEHFrames(uint8_t *Addr, uint64_t LoadAddr,
-                        size_t Size) override {}
-  void deregisterEHFrames() override {}
+  void deallocate(std::vector<FinalizedAlloc> Allocs,
+                  OnDeallocatedFunction OnDeallocated) override;
+  using JITLinkMemoryManager::deallocate;
 
   /// Section name management.
   void setNewSecPrefix(StringRef Prefix) { NewSecPrefix = Prefix; }
   void setOrgSecPrefix(StringRef Prefix) { OrgSecPrefix = Prefix; }
 };
 
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/Rewrite/JITLinkLinker.h

+//===- bolt/Rewrite/JITLinkLinker.h - Linker using JITLink ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// BOLTLinker using JITLink.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef BOLT_REWRITE_JITLINK_LINKER_H
+#define BOLT_REWRITE_JITLINK_LINKER_H
+
+#include "bolt/Core/Linker.h"
+#include "bolt/Rewrite/ExecutableFileMemoryManager.h"
+#include "llvm/ExecutionEngine/JITLink/JITLinkDylib.h"
+
+#include <map>
+#include <memory>
+#include <vector>
+
+namespace llvm {
+namespace bolt {
+
+class BinaryContext;
+
+class JITLinkLinker : public BOLTLinker {
+private:
+  struct Context;
+  friend struct Context;
+
+  BinaryContext &BC;
+  std::unique_ptr<ExecutableFileMemoryManager> MM;
+  jitlink::JITLinkDylib Dylib{"main"};
+  std::vector<ExecutableFileMemoryManager::FinalizedAlloc> Allocs;
+  std::map<std::string, uint64_t> Symtab;
+
+public:
+  JITLinkLinker(BinaryContext &BC,
+                std::unique_ptr<ExecutableFileMemoryManager> MM);
+  ~JITLinkLinker();
+
+  void loadObject(MemoryBufferRef Obj, SectionsMapper MapSections) override;
+  std::optional<uint64_t> lookupSymbol(StringRef Name) const override;
+
+  static SmallVector<jitlink::Block *, 2>
+  orderedBlocks(const jitlink::Section &Section);
+  static size_t sectionSize(const jitlink::Section &Section);
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif // BOLT_REWRITE_JITLINK_LINKER_H

bolt/include/bolt/Rewrite/MachORewriteInstance.h

 //===- bolt/Rewrite/MachORewriteInstance.h - MachO rewriter -----*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Interface to control an instance of a macho binary rewriting process.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef BOLT_REWRITE_MACHO_REWRITE_INSTANCE_H
 #define BOLT_REWRITE_MACHO_REWRITE_INSTANCE_H
 
+#include "bolt/Core/Linker.h"
 #include "bolt/Utils/NameResolver.h"
 #include "llvm/Support/Error.h"
 #include <memory>
 
 namespace llvm {
 class ToolOutputFile;
-class RuntimeDyld;
 class raw_pwrite_stream;
 namespace object {
 class MachOObjectFile;
 } // namespace object
 
 namespace bolt {
 
 class BinaryContext;
 class ProfileReaderBase;
 
 class MachORewriteInstance {
   object::MachOObjectFile *InputFile;
   StringRef ToolPath;
   std::unique_ptr<BinaryContext> BC;
 
   NameResolver NR;
 
-  std::unique_ptr<RuntimeDyld> RTDyld;
+  std::unique_ptr<BOLTLinker> Linker;
 
   std::unique_ptr<ToolOutputFile> Out;
 
   std::unique_ptr<ProfileReaderBase> ProfileReader;
   void preprocessProfileData();
   void processProfileDataPreCFG();
   void processProfileData();
 
   static StringRef getNewSecPrefix() { return ".bolt.new"; }
   static StringRef getOrgSecPrefix() { return ".bolt.org"; }
 
-  void mapInstrumentationSection(StringRef SectionName);
-  void mapCodeSections();
+  void mapInstrumentationSection(StringRef SectionName,
+                                 BOLTLinker::SectionMapper MapSection);
+  void mapCodeSections(BOLTLinker::SectionMapper MapSection);
 
   void adjustCommandLineOptions();
   void readSpecialSections();
   void discoverFileObjects();
   void disassembleFunctions();
   void buildFunctionsCFG();
   void postProcessFunctions();
   void runOptimizationPasses();

bolt/include/bolt/Rewrite/RewriteInstance.h

 //===- bolt/Rewrite/RewriteInstance.h - ELF rewriter ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Interface to control an instance of a binary rewriting process.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef BOLT_REWRITE_REWRITE_INSTANCE_H
 #define BOLT_REWRITE_REWRITE_INSTANCE_H
 
 #include "bolt/Core/BinaryContext.h"
+#include "bolt/Core/Linker.h"
 #include "bolt/Utils/NameResolver.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/MC/StringTableBuilder.h"
 #include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/Error.h"
 #include <map>
 #include <set>
private:
 
   /// Encode MCDecodedPseudoProbe.
   void encodePseudoProbes();
 
   /// Return the list of code sections in the output order.
   std::vector<BinarySection *> getCodeSections();
 
   /// Map all sections to their final addresses.
-  void mapFileSections(RuntimeDyld &RTDyld);
+  void mapFileSections(BOLTLinker::SectionMapper MapSection);
 
   /// Map code sections generated by BOLT.
-  void mapCodeSections(RuntimeDyld &RTDyld);
+  void mapCodeSections(BOLTLinker::SectionMapper MapSection);
 
   /// Map the rest of allocatable sections.
-  void mapAllocatableSections(RuntimeDyld &RTDyld);
+  void mapAllocatableSections(BOLTLinker::SectionMapper MapSection);
 
   /// Update output object's values based on the final \p Layout.
   void updateOutputValues(const MCAsmLayout &Layout);
 
   /// Rewrite back all functions (hopefully optimized) that fit in the original
   /// memory footprint for that function. If the function is now larger and does
   /// not fit in the binary, reject it and preserve the original version of the
   /// function. If we couldn't understand the function for some reason in
private:
   StringRef ToolPath;
 
   std::unique_ptr<ProfileReaderBase> ProfileReader;
 
   std::unique_ptr<BinaryContext> BC;
   std::unique_ptr<CFIReaderWriter> CFIRdWrt;
 
   // Run ExecutionEngine linker with custom memory manager and symbol resolver.
-  std::unique_ptr<RuntimeDyld> RTDyld;
+  std::unique_ptr<BOLTLinker> Linker;
 
   /// Output file where we mix original code from the input binary and
   /// optimized code for selected functions.
   std::unique_ptr<ToolOutputFile> Out;
 
   /// Offset in the input file where non-allocatable sections start.
   uint64_t FirstNonAllocatableOffset{0};
 

bolt/include/bolt/RuntimeLibs/HugifyRuntimeLibrary.h

public:
   /// Add custom section names generated by the runtime libraries to \p
   /// SecNames.
   void addRuntimeLibSections(std::vector<std::string> &SecNames) const final {}
 
   void adjustCommandLineOptions(const BinaryContext &BC) const final;
 
   void emitBinary(BinaryContext &BC, MCStreamer &Streamer) final {}
 
-  void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
-            std::function<void(RuntimeDyld &)> OnLoad) final;
+  void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
+            BOLTLinker::SectionsMapper MapSections) override;
 };
 
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h

public:
   void addRuntimeLibSections(std::vector<std::string> &SecNames) const final {
     SecNames.push_back(".bolt.instr.counters");
   }
 
   void adjustCommandLineOptions(const BinaryContext &BC) const final;
 
   void emitBinary(BinaryContext &BC, MCStreamer &Streamer) final;
 
-  void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
-            std::function<void(RuntimeDyld &)> OnLoad) final;
+  void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
+            BOLTLinker::SectionsMapper MapSections) override;
 
 private:
   std::string buildTables(BinaryContext &BC);
 
   /// Create a non-allocatable ELF section with read-only tables necessary for
   /// writing the instrumented data profile during program finish. The runtime
   /// library needs to open the program executable file and read this data from
   /// disk, this is not loaded by the system.

bolt/include/bolt/RuntimeLibs/RuntimeLibrary.h

 // provides all the necessary utilities to link runtime libraries during binary
 // rewriting, such as the instrumentation runtime library.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef BOLT_RUNTIMELIBS_RUNTIME_LIBRARY_H
 #define BOLT_RUNTIMELIBS_RUNTIME_LIBRARY_H
 
+#include "bolt/Core/Linker.h"
 #include "llvm/ADT/StringRef.h"
 #include <functional>
 #include <vector>
 
 namespace llvm {
 
 class MCStreamer;
-class RuntimeDyld;
 
 namespace bolt {
 
 class BinaryContext;
 
 class RuntimeLibrary {
   // vtable anchor.
   virtual void anchor();
public:
   /// Validity check and modify if necessary all the command line options
   /// for this runtime library.
   virtual void adjustCommandLineOptions(const BinaryContext &BC) const = 0;
 
   /// Emit data structures that will be necessary during runtime.
   virtual void emitBinary(BinaryContext &BC, MCStreamer &Streamer) = 0;
 
   /// Link with the library code.
-  virtual void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
-                    std::function<void(RuntimeDyld &)> OnLoad) = 0;
+  virtual void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
+                    BOLTLinker::SectionsMapper MapSections) = 0;
 
 protected:
   /// The fini and init address set by the runtime library.
   uint64_t RuntimeFiniAddress{0};
   uint64_t RuntimeStartAddress{0};
 
   /// Get the full path to a runtime library specified by \p LibFileName.
   static std::string getLibPath(StringRef ToolPath, StringRef LibFileName);
 
   /// Load a static runtime library specified by \p LibPath.
-  static void loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld);
+  static void loadLibrary(StringRef LibPath, BOLTLinker &Linker,
+                          BOLTLinker::SectionsMapper MapSections);
 };
 
 } // namespace bolt
 } // namespace llvm
 
 #endif // BOLT_RUNTIMELIBS_RUNTIME_LIBRARY_H

bolt/lib/Core/BinaryContext.cpp

while (NameRange.first != NameRange.second) {
     ++NameRange.first;
   }
 }
 
 void BinaryContext::deregisterUnusedSections() {
   ErrorOr<BinarySection &> AbsSection = getUniqueSectionByName("<absolute>");
   for (auto SI = Sections.begin(); SI != Sections.end();) {
     BinarySection *Section = *SI;
-    if (Section->hasSectionRef() || Section->getOutputSize() ||
+    // We check getOutputData() instead of getOutputSize() because sometimes
+    // zero-sized .text.cold sections are allocated.
+    if (Section->hasSectionRef() || Section->getOutputData() ||
         (AbsSection && Section == &AbsSection.get())) {
       ++SI;
       continue;
     }
 
     LLVM_DEBUG(dbgs() << "LLVM-DEBUG: deregistering " << Section->getName()
                       << '\n';);
     deregisterSectionName(*Section);

bolt/lib/Core/BinaryEmitter.cpp

 
 } // anonymous namespace
 
 void BinaryEmitter::emitAll(StringRef OrgSecPrefix) {
   Streamer.initSections(false, *BC.STI);
 
   if (opts::UpdateDebugSections && BC.isELF()) {
     // Force the emission of debug line info into allocatable section to ensure
-    // RuntimeDyld will process it without ProcessAllSections flag.
+    // JITLink will process it.
     //
     // NB: on MachO all sections are required for execution, hence no need
     //     to change flags/attributes.
     MCSectionELF *ELFDwarfLineSection =
         static_cast<MCSectionELF *>(BC.MOFI->getDwarfLineSection());
     ELFDwarfLineSection->setFlags(ELF::SHF_ALLOC);
+    MCSectionELF *ELFDwarfLineStrSection =
+        static_cast<MCSectionELF *>(BC.MOFI->getDwarfLineStrSection());
+    ELFDwarfLineStrSection->setFlags(ELF::SHF_ALLOC);
   }
 
   if (RuntimeLibrary *RtLibrary = BC.getRuntimeLibrary())
     RtLibrary->emitBinary(BC, Streamer);
 
   BC.getTextSection()->setAlignment(Align(opts::AlignText));
 
   emitFunctions();

bolt/lib/Core/DebugData.cpp

void DwarfLineTable::emitCU(MCStreamer *MCOS, MCDwarfLineTableParams Params,
                             std::optional<MCDwarfLineStr> &LineStr,
                             BinaryContext &BC) const {
   if (!RawData.empty()) {
     assert(MCLineSections.getMCLineEntries().empty() &&
            InputSequences.empty() &&
            "cannot combine raw data with new line entries");
     MCOS->emitLabel(getLabel());
     MCOS->emitBytes(RawData);
-
-    // Emit fake relocation for RuntimeDyld to always allocate the section.
-    //
-    // FIXME: remove this once RuntimeDyld stops skipping allocatable sections
-    //        without relocations.
-    MCOS->emitRelocDirective(
-        *MCConstantExpr::create(0, *BC.Ctx), "BFD_RELOC_NONE",
-        MCSymbolRefExpr::create(getLabel(), *BC.Ctx), SMLoc(), *BC.STI);
-
     return;
   }
 
   MCSymbol *LineEndSym = Header.Emit(MCOS, Params, LineStr).second;
 
   // Put out the line tables.
   for (const auto &LineSec : MCLineSections.getMCLineEntries())
     emitDwarfLineTable(MCOS, LineSec.first, LineSec.second);
for (auto &CUIDTablePair : LineTables) {
     CUIDTablePair.second.emitCU(&Streamer, Params, LineStr, BC);
   }
 
   // Resetting DWARF version for rest of the flow.
   BC.Ctx->setDwarfVersion(DwarfVersion);
 
   // Still need to write the section out for the ExecutionEngine, and temp in
   // memory object we are constructing.
-  if (LineStr) {
+  if (LineStr)
     LineStr->emitSection(&Streamer);
-    SmallString<0> Data = LineStr->getFinalizedData();
-    BC.registerOrUpdateNoteSection(".debug_line_str", copyByteArray(Data.str()),
-                                   Data.size());
-  }
 }
 
 } // namespace bolt
 } // namespace llvm

bolt/lib/Rewrite/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   DebugInfoDWARF
   DWP
-  ExecutionEngine
+  JITLink
   MC
   Object
   Support
   TargetParser
   )
 
 add_llvm_library(LLVMBOLTRewrite
   BinaryPassManager.cpp
   BoltDiff.cpp
   DWARFRewriter.cpp
   ExecutableFileMemoryManager.cpp
+  JITLinkLinker.cpp
   MachORewriteInstance.cpp
   RewriteInstance.cpp
 
   DISABLE_LLVM_LINK_LLVM_DYLIB
 
   LINK_LIBS
   ${LLVM_PTHREAD_LIB}
   )

bolt/lib/Rewrite/ExecutableFileMemoryManager.cpp

 //===- bolt/Rewrite/ExecutableFileMemoryManager.cpp -----------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Rewrite/ExecutableFileMemoryManager.h"
+#include "bolt/Rewrite/JITLinkLinker.h"
 #include "bolt/Rewrite/RewriteInstance.h"
+#include "llvm/ExecutionEngine/JITLink/JITLink.h"
 #include "llvm/Support/MemAlloc.h"
 
 #undef  DEBUG_TYPE
 #define DEBUG_TYPE "efmm"
 
 using namespace llvm;
 using namespace object;
 using namespace bolt;
 
 namespace llvm {
 
 namespace bolt {
 
-uint8_t *ExecutableFileMemoryManager::allocateSection(
-    uintptr_t Size, unsigned Alignment, unsigned SectionID,
-    StringRef SectionName, bool IsCode, bool IsReadOnly) {
-  uint8_t *Ret = static_cast<uint8_t *>(llvm::allocate_buffer(Size, Alignment));
-  AllocatedSections.push_back(AllocInfo{Ret, Size, Alignment});
-
-  // A Size of 1 might mean an empty section for which RuntimeDyld decided to
-  // allocate 1 byte. In this case, the allocation will never be initialized
-  // causing non-deterministic output section contents.
-  if (Size == 1)
-    *Ret = 0;
+namespace {
+
+SmallVector<jitlink::Section *> orderedSections(jitlink::LinkGraph &G) {
+  SmallVector<jitlink::Section *> Sections(
+      llvm::map_range(G.sections(), [](auto &S) { return &S; }));
+  llvm::sort(Sections, [](const auto *LHS, const auto *RHS) {
+    return LHS->getOrdinal() < RHS->getOrdinal();
+  });
+  return Sections;
+}
+
+size_t sectionAlignment(const jitlink::Section &Section) {
+  assert(!Section.empty() && "Cannot get alignment for empty section");
+  return JITLinkLinker::orderedBlocks(Section).front()->getAlignment();
+}
+
+StringRef sectionName(const jitlink::Section &Section,
+                      const BinaryContext &BC) {
+  auto Name = Section.getName();
+
+  if (BC.isMachO()) {
+    // JITLink "normalizes" section names as "SegmentName,SectionName" on
+    // Mach-O. BOLT internally refers to sections just by the section name so
+    // strip-off the segment name.
+    auto SegmentEnd = Name.find(',');
+    assert(SegmentEnd != StringRef::npos && "Mach-O segment not found");
+    Name = Name.substr(SegmentEnd + 1);
+  }
+
+  return Name;
+}
+
+struct SectionAllocInfo {
+  void *Address;
+  size_t Size;
+  size_t Alignment;
+};
+
+struct AllocInfo {
+  SmallVector<SectionAllocInfo, 8> AllocatedSections;
+
+  ~AllocInfo() {
+    for (auto &Section : AllocatedSections)
+      deallocate_buffer(Section.Address, Section.Size, Section.Alignment);
+  }
+
+  SectionAllocInfo allocateSection(const jitlink::Section &Section) {
+    auto Size = JITLinkLinker::sectionSize(Section);
+    auto Alignment = sectionAlignment(Section);
+    auto *Buf = allocate_buffer(Size, Alignment);
+    SectionAllocInfo Alloc{Buf, Size, Alignment};
+    AllocatedSections.push_back(Alloc);
+    return Alloc;
+  }
+};
+
+struct BOLTInFlightAlloc : ExecutableFileMemoryManager::InFlightAlloc {
+  // Even though this is passed using a raw pointer in FinalizedAlloc, we keep
+  // it in a unique_ptr as long as possible to enjoy automatic cleanup when
+  // something goes wrong.
+  std::unique_ptr<AllocInfo> Alloc;
+
+public:
+  BOLTInFlightAlloc(std::unique_ptr<AllocInfo> Alloc)
+      : Alloc(std::move(Alloc)) {}
+
+  virtual void abandon(OnAbandonedFunction OnAbandoned) override {
+    llvm_unreachable("unexpected abandoned allocation");
+  }
+
+  virtual void finalize(OnFinalizedFunction OnFinalized) override {
+    OnFinalized(ExecutableFileMemoryManager::FinalizedAlloc(
+        orc::ExecutorAddr::fromPtr(Alloc.release())));
+  }
+};
+
+} // anonymous namespace
+
+void ExecutableFileMemoryManager::updateSection(
+    const jitlink::Section &JLSection, uint8_t *Contents, size_t Size,
+    size_t Alignment) {
+  auto SectionID = JLSection.getName();
+  auto SectionName = sectionName(JLSection, BC);
+  auto Prot = JLSection.getMemProt();
+  auto IsCode = (Prot & orc::MemProt::Exec) != orc::MemProt::None;
+  auto IsReadOnly = (Prot & orc::MemProt::Write) == orc::MemProt::None;
 
   // Register a debug section as a note section.
   if (!ObjectsLoaded && RewriteInstance::isDebugSection(SectionName)) {
     BinarySection &Section =
-        BC.registerOrUpdateNoteSection(SectionName, Ret, Size, Alignment);
+        BC.registerOrUpdateNoteSection(SectionName, Contents, Size, Alignment);
     Section.setSectionID(SectionID);
     assert(!Section.isAllocatable() && "note sections cannot be allocatable");
-    return Ret;
+    return;
   }
 
   if (!IsCode && (SectionName == ".strtab" || SectionName == ".symtab" ||
                   SectionName == "" || SectionName.startswith(".rela.")))
-    return Ret;
+    return;
 
   SmallVector<char, 256> Buf;
   if (ObjectsLoaded > 0) {
     if (BC.isELF()) {
       SectionName = (Twine(SectionName) + ".bolt.extra." + Twine(ObjectsLoaded))
                         .toStringRef(Buf);
     } else if (BC.isMachO()) {
       assert((SectionName == "__text" || SectionName == "__data" ||
void ExecutableFileMemoryManager::updateSection(
   if (!OrgSecPrefix.empty() && SectionName.startswith(OrgSecPrefix)) {
     // Update the original section contents.
     ErrorOr<BinarySection &> OrgSection =
         BC.getUniqueSectionByName(SectionName.substr(OrgSecPrefix.length()));
     assert(OrgSection && OrgSection->isAllocatable() &&
            "Original section must exist and be allocatable.");
 
     Section = &OrgSection.get();
-    Section->updateContents(Ret, Size);
+    Section->updateContents(Contents, Size);
   } else {
     // If the input contains a section with the section name, rename it in the
     // output file to avoid the section name conflict and emit the new section
     // under a unique internal name.
     ErrorOr<BinarySection &> OrgSection =
         BC.getUniqueSectionByName(SectionName);
     bool UsePrefix = false;
     if (OrgSection && OrgSection->hasSectionRef()) {
       OrgSection->setOutputName(OrgSecPrefix + SectionName);
       UsePrefix = true;
     }
 
     // Register the new section under a unique name to avoid name collision with
     // sections in the input file.
     BinarySection &NewSection = BC.registerOrUpdateSection(
         UsePrefix ? NewSecPrefix + SectionName : SectionName, ELF::SHT_PROGBITS,
-        BinarySection::getFlags(IsReadOnly, IsCode, true), Ret, Size,
+        BinarySection::getFlags(IsReadOnly, IsCode, true), Contents, Size,
         Alignment);
     if (UsePrefix)
       NewSection.setOutputName(SectionName);
     Section = &NewSection;
   }
 
   LLVM_DEBUG({
     dbgs() << "BOLT: allocating "
            << (IsCode ? "code" : (IsReadOnly ? "read-only data" : "data"))
            << " section : " << Section->getOutputName() << " ("
            << Section->getName() << ")"
            << " with size " << Size << ", alignment " << Alignment << " at "
-           << Ret << ", ID = " << SectionID << "\n";
+           << Contents << ", ID = " << SectionID << "\n";
   });
 
   Section->setSectionID(SectionID);
+}
+
+void ExecutableFileMemoryManager::allocate(const jitlink::JITLinkDylib *JD,
+                                           jitlink::LinkGraph &G,
+                                           OnAllocatedFunction OnAllocated) {
+  auto Alloc = std::make_unique<AllocInfo>();
 
-  return Ret;
+  for (auto *Section : orderedSections(G)) {
+    if (Section->empty())
+      continue;
+
+    auto SectionAlloc = Alloc->allocateSection(*Section);
+    updateSection(*Section, static_cast<uint8_t *>(SectionAlloc.Address),
+                  SectionAlloc.Size, SectionAlloc.Alignment);
+
+    size_t CurrentOffset = 0;
+    auto *Buf = static_cast<char *>(SectionAlloc.Address);
+    for (auto *Block : JITLinkLinker::orderedBlocks(*Section)) {
+      CurrentOffset = jitlink::alignToBlock(CurrentOffset, *Block);
+      auto BlockSize = Block->getSize();
+      auto *BlockBuf = Buf + CurrentOffset;
+
+      if (Block->isZeroFill())
+        std::memset(BlockBuf, 0, BlockSize);
+      else
+        std::memcpy(BlockBuf, Block->getContent().data(), BlockSize);
+
+      Block->setMutableContent({BlockBuf, Block->getSize()});
+      CurrentOffset += BlockSize;
     }
+  }
+
+  OnAllocated(std::make_unique<BOLTInFlightAlloc>(std::move(Alloc)));
+}
+
+void ExecutableFileMemoryManager::deallocate(
+    std::vector<FinalizedAlloc> Allocs, OnDeallocatedFunction OnDeallocated) {
+  for (auto &Alloc : Allocs)
+    delete Alloc.release().toPtr<AllocInfo *>();
 
-ExecutableFileMemoryManager::~ExecutableFileMemoryManager() {
-  for (const AllocInfo &AI : AllocatedSections)
-    llvm::deallocate_buffer(AI.Address, AI.Size, AI.Alignment);
+  OnDeallocated(Error::success());
 }
 
 } // namespace bolt
 
 } // namespace llvm

bolt/lib/Rewrite/JITLinkLinker.cpp

+//===- bolt/Rewrite/JITLinkLinker.cpp - BOLTLinker using JITLink ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "bolt/Rewrite/JITLinkLinker.h"
+#include "bolt/Core/BinaryData.h"
+#include "bolt/Rewrite/RewriteInstance.h"
+#include "llvm/ExecutionEngine/JITLink/JITLink.h"
+#include "llvm/ExecutionEngine/Orc/Shared/ExecutorAddress.h"
+#include "llvm/ExecutionEngine/Orc/Shared/ExecutorSymbolDef.h"
+#include "llvm/Support/Debug.h"
+
+#define DEBUG_TYPE "bolt"
+
+namespace llvm {
+namespace bolt {
+
+namespace {
+
+bool hasSymbols(const jitlink::Block &B) {
+  return llvm::any_of(B.getSection().symbols(),
+                      [&B](const auto &S) { return &S->getBlock() == &B; });
+}
+
+/// Liveness in JITLink is based on symbols so sections that do not contain
+/// any symbols will always be pruned. This pass adds anonymous symbols to
+/// needed sections to prevent pruning.
+Error markSectionsLive(jitlink::LinkGraph &G) {
+  for (auto &Section : G.sections()) {
+    // We only need allocatable sections.
+    if (Section.getMemLifetimePolicy() == orc::MemLifetimePolicy::NoAlloc)
+      continue;
+
+    // Skip empty sections.
+    if (JITLinkLinker::sectionSize(Section) == 0)
+      continue;
+
+    for (auto *Block : Section.blocks()) {
+      // No need to add symbols if it already has some.
+      if (hasSymbols(*Block))
+        continue;
+
+      G.addAnonymousSymbol(*Block, /*Offset=*/0, /*Size=*/0,
+                           /*IsCallable=*/false, /*IsLive=*/true);
+    }
+  }
+
+  return jitlink::markAllSymbolsLive(G);
+}
+
+void reassignSectionAddress(jitlink::LinkGraph &LG,
+                            const BinarySection &BinSection, uint64_t Address) {
+  auto *JLSection = LG.findSectionByName(BinSection.getSectionID());
+  assert(JLSection && "cannot find section in LinkGraph");
+
+  auto BlockAddress = Address;
+  for (auto *Block : JITLinkLinker::orderedBlocks(*JLSection)) {
+    // FIXME it would seem to make sense to align here. However, in
+    // non-relocation mode, we simply use the original address of functions
+    // which might not be aligned with the minimum alignment used by
+    // BinaryFunction (2). Example failing test when aligning:
+    // bolt/test/X86/addr32.s
+    Block->setAddress(orc::ExecutorAddr(BlockAddress));
+    BlockAddress += Block->getSize();
+  }
+}
+
+} // anonymous namespace
+
+struct JITLinkLinker::Context : jitlink::JITLinkContext {
+  JITLinkLinker &Linker;
+  JITLinkLinker::SectionsMapper MapSections;
+
+  Context(JITLinkLinker &Linker, JITLinkLinker::SectionsMapper MapSections)
+      : JITLinkContext(&Linker.Dylib), Linker(Linker),
+        MapSections(MapSections) {}
+
+  jitlink::JITLinkMemoryManager &getMemoryManager() override {
+    return *Linker.MM;
+  }
+
+  bool shouldAddDefaultTargetPasses(const Triple &TT) const override {
+    // The default passes manipulate DWARF sections in a way incompatible with
+    // BOLT.
+    // TODO check if we can actually use these passes to remove some of the
+    // DWARF manipulation done in BOLT.
+    return false;
+  }
+
+  Error modifyPassConfig(jitlink::LinkGraph &G,
+                         jitlink::PassConfiguration &Config) override {
+    Config.PrePrunePasses.push_back(markSectionsLive);
+    return Error::success();
+  }
+
+  void notifyFailed(Error Err) override {
+    errs() << "BOLT-ERROR: JITLink failed: " << Err << '\n';
+    exit(1);
+  }
+
+  void
+  lookup(const LookupMap &Symbols,
+         std::unique_ptr<jitlink::JITLinkAsyncLookupContinuation> LC) override {
+    jitlink::AsyncLookupResult AllResults;
+
+    for (const auto &Symbol : Symbols) {
+      std::string SymName = Symbol.first.str();
+      LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
+
+      if (auto Address = Linker.lookupSymbol(SymName)) {
+        LLVM_DEBUG(dbgs() << "Resolved to address 0x"
+                          << Twine::utohexstr(*Address) << "\n");
+        AllResults[Symbol.first] = orc::ExecutorSymbolDef(
+            orc::ExecutorAddr(*Address), JITSymbolFlags());
+        continue;
+      }
+
+      if (const BinaryData *I = Linker.BC.getBinaryDataByName(SymName)) {
+        uint64_t Address = I->isMoved() && !I->isJumpTable()
+                               ? I->getOutputAddress()
+                               : I->getAddress();
+        LLVM_DEBUG(dbgs() << "Resolved to address 0x"
+                          << Twine::utohexstr(Address) << "\n");
+        AllResults[Symbol.first] = orc::ExecutorSymbolDef(
+            orc::ExecutorAddr(Address), JITSymbolFlags());
+        continue;
+      }
+      LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
+      AllResults[Symbol.first] =
+          orc::ExecutorSymbolDef(orc::ExecutorAddr(0), JITSymbolFlags());
+    }
+
+    LC->run(std::move(AllResults));
+  }
+
+  Error notifyResolved(jitlink::LinkGraph &G) override {
+    MapSections([&G](const BinarySection &Section, uint64_t Address) {
+      reassignSectionAddress(G, Section, Address);
+    });
+
+    for (auto *Symbol : G.defined_symbols()) {
+      Linker.Symtab.insert(
+          {Symbol->getName().str(), Symbol->getAddress().getValue()});
+    }
+
+    return Error::success();
+  }
+
+  void notifyFinalized(
+      jitlink::JITLinkMemoryManager::FinalizedAlloc Alloc) override {
+    Linker.Allocs.push_back(std::move(Alloc));
+    ++Linker.MM->ObjectsLoaded;
+  }
+};
+
+JITLinkLinker::JITLinkLinker(BinaryContext &BC,
+                             std::unique_ptr<ExecutableFileMemoryManager> MM)
+    : BC(BC), MM(std::move(MM)) {}
+
+JITLinkLinker::~JITLinkLinker() { cantFail(MM->deallocate(std::move(Allocs))); }
+
+void JITLinkLinker::loadObject(MemoryBufferRef Obj,
+                               SectionsMapper MapSections) {
+  auto LG = jitlink::createLinkGraphFromObject(Obj);
+  if (auto E = LG.takeError()) {
+    errs() << "BOLT-ERROR: JITLink failed: " << E << '\n';
+    exit(1);
+  }
+
+  auto Ctx = std::make_unique<Context>(*this, MapSections);
+  jitlink::link(std::move(*LG), std::move(Ctx));
+}
+
+std::optional<uint64_t> JITLinkLinker::lookupSymbol(StringRef Name) const {
+  auto It = Symtab.find(Name.data());
+  if (It == Symtab.end())
+    return std::nullopt;
+
+  return It->second;
+}
+
+SmallVector<jitlink::Block *, 2>
+JITLinkLinker::orderedBlocks(const jitlink::Section &Section) {
+  SmallVector<jitlink::Block *, 2> Blocks(Section.blocks());
+  llvm::sort(Blocks, [](const auto *LHS, const auto *RHS) {
+    return LHS->getAddress() < RHS->getAddress();
+  });
+  return Blocks;
+}
+
+size_t JITLinkLinker::sectionSize(const jitlink::Section &Section) {
+  size_t Size = 0;
+
+  for (const auto *Block : orderedBlocks(Section)) {
+    Size = jitlink::alignToBlock(Size, *Block);
+    Size += Block->getSize();
+  }
+
+  return Size;
+}
+
+} // namespace bolt
+} // namespace llvm

bolt/lib/Rewrite/MachORewriteInstance.cpp

 #include "bolt/Core/BinaryFunction.h"
 #include "bolt/Core/JumpTable.h"
 #include "bolt/Core/MCPlusBuilder.h"
 #include "bolt/Passes/Instrumentation.h"
 #include "bolt/Passes/PatchEntries.h"
 #include "bolt/Profile/DataReader.h"
 #include "bolt/Rewrite/BinaryPassManager.h"
 #include "bolt/Rewrite/ExecutableFileMemoryManager.h"
+#include "bolt/Rewrite/JITLinkLinker.h"
 #include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
 #include "bolt/Utils/Utils.h"
 #include "llvm/MC/MCAsmLayout.h"
 #include "llvm/MC/MCObjectStreamer.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include <memory>
Manager.registerPass(
       std::make_unique<FixupBranches>(opts::PrintAfterBranchFixup));
   // This pass should always run last.*
   Manager.registerPass(
       std::make_unique<FinalizeFunctions>(opts::PrintFinalized));
 
   Manager.runPasses();
 }
 
-void MachORewriteInstance::mapInstrumentationSection(StringRef SectionName) {
+void MachORewriteInstance::mapInstrumentationSection(
+    StringRef SectionName, BOLTLinker::SectionMapper MapSection) {
   if (!opts::Instrument)
     return;
   ErrorOr<BinarySection &> Section = BC->getUniqueSectionByName(SectionName);
   if (!Section) {
     llvm::errs() << "Cannot find " + SectionName + " section\n";
     exit(1);
   }
   if (!Section->hasValidSectionID())
     return;
-  RTDyld->reassignSectionAddress(Section->getSectionID(),
-                                 Section->getAddress());
+  MapSection(*Section, Section->getAddress());
 }
 
-void MachORewriteInstance::mapCodeSections() {
+void MachORewriteInstance::mapCodeSections(
+    BOLTLinker::SectionMapper MapSection) {
   for (BinaryFunction *Function : BC->getAllBinaryFunctions()) {
     if (!Function->isEmitted())
       continue;
     if (Function->getOutputAddress() == 0)
       continue;
     ErrorOr<BinarySection &> FuncSection = Function->getCodeSection();
     if (!FuncSection)
       report_error(
           (Twine("Cannot find section for function ") + Function->getOneName())
               .str(),
           FuncSection.getError());
 
     FuncSection->setOutputAddress(Function->getOutputAddress());
     LLVM_DEBUG(dbgs() << "BOLT: mapping 0x"
                  << Twine::utohexstr(FuncSection->getAllocAddress()) << " to 0x"
                  << Twine::utohexstr(Function->getOutputAddress()) << '\n');
-    RTDyld->reassignSectionAddress(FuncSection->getSectionID(),
-                                   Function->getOutputAddress());
+    MapSection(*FuncSection, Function->getOutputAddress());
     Function->setImageAddress(FuncSection->getAllocAddress());
     Function->setImageSize(FuncSection->getOutputSize());
   }
 
   if (opts::Instrument) {
     ErrorOr<BinarySection &> BOLT = BC->getUniqueSectionByName("__bolt");
     if (!BOLT) {
       llvm::errs() << "Cannot find __bolt section\n";
       exit(1);
     }
     uint64_t Addr = BOLT->getAddress();
     for (BinaryFunction *Function : BC->getAllBinaryFunctions()) {
       if (!Function->isEmitted())
         continue;
       if (Function->getOutputAddress() != 0)
         continue;
       ErrorOr<BinarySection &> FuncSection = Function->getCodeSection();
       assert(FuncSection && "cannot find section for function");
       Addr = llvm::alignTo(Addr, 4);
       FuncSection->setOutputAddress(Addr);
-      RTDyld->reassignSectionAddress(FuncSection->getSectionID(), Addr);
+      MapSection(*FuncSection, Addr);
       Function->setFileOffset(Addr - BOLT->getAddress() +
                               BOLT->getInputFileOffset());
       Function->setImageAddress(FuncSection->getAllocAddress());
       Function->setImageSize(FuncSection->getOutputSize());
       BC->registerNameAtAddress(Function->getOneName(), Addr, 0, 0);
       Addr += FuncSection->getOutputSize();
     }
   }
 }
 
-namespace {
-
-class BOLTSymbolResolver : public LegacyJITSymbolResolver {
-  BinaryContext &BC;
-public:
-  BOLTSymbolResolver(BinaryContext &BC) : BC(BC) {}
-
-  JITSymbol findSymbolInLogicalDylib(const std::string &Name) override {
-    return JITSymbol(nullptr);
-  }
-
-  JITSymbol findSymbol(const std::string &Name) override {
-    LLVM_DEBUG(dbgs() << "BOLT: looking for " << Name << "\n");
-    if (BinaryData *I = BC.getBinaryDataByName(Name)) {
-      const uint64_t Address = I->isMoved() && !I->isJumpTable()
-                                   ? I->getOutputAddress()
-                                   : I->getAddress();
-      LLVM_DEBUG(dbgs() << "Resolved to address 0x" << Twine::utohexstr(Address)
-                        << "\n");
-      return JITSymbol(Address, JITSymbolFlags());
-    }
-    LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
-    return JITSymbol(nullptr);
-  }
-};
-
-} // end anonymous namespace
-
 void MachORewriteInstance::emitAndLink() {
   std::error_code EC;
   std::unique_ptr<::llvm::ToolOutputFile> TempOut =
       std::make_unique<::llvm::ToolOutputFile>(
           opts::OutputFilename + ".bolt.o", EC, sys::fs::OF_None);
   check_error(EC, "cannot create output object file");
 
   if (opts::KeepTmp)
void MachORewriteInstance::emitAndLink() {
 
   std::unique_ptr<MemoryBuffer> ObjectMemBuffer =
       MemoryBuffer::getMemBuffer(BOS->str(), "in-memory object file", false);
   std::unique_ptr<object::ObjectFile> Obj = cantFail(
       object::ObjectFile::createObjectFile(ObjectMemBuffer->getMemBufferRef()),
       "error creating in-memory object");
   assert(Obj && "createObjectFile cannot return nullptr");
 
-  BOLTSymbolResolver Resolver = BOLTSymbolResolver(*BC);
-
   MCAsmLayout FinalLayout(
       static_cast<MCObjectStreamer *>(Streamer.get())->getAssembler());
 
-  BC->EFMM.reset(new ExecutableFileMemoryManager(*BC, /*AllowStubs*/ false));
-  BC->EFMM->setOrgSecPrefix(getOrgSecPrefix());
-  BC->EFMM->setNewSecPrefix(getNewSecPrefix());
-
-  RTDyld.reset(new decltype(RTDyld)::element_type(*BC->EFMM, Resolver));
-  RTDyld->setProcessAllSections(true);
-  RTDyld->loadObject(*Obj);
-  if (RTDyld->hasError()) {
-    outs() << "BOLT-ERROR: RTDyld failed.\n";
-    exit(1);
-  }
-
-  // Assign addresses to all sections. If key corresponds to the object
-  // created by ourselves, call our regular mapping function. If we are
-  // loading additional objects as part of runtime libraries for
-  // instrumentation, treat them as extra sections.
-  mapCodeSections();
-  mapInstrumentationSection("__counters");
-  mapInstrumentationSection("__tables");
-  RTDyld->finalizeWithMemoryManagerLocking();
+  auto EFMM = std::make_unique<ExecutableFileMemoryManager>(*BC);
+  EFMM->setNewSecPrefix(getNewSecPrefix());
+  EFMM->setOrgSecPrefix(getOrgSecPrefix());
+
+  Linker = std::make_unique<JITLinkLinker>(*BC, std::move(EFMM));
+  Linker->loadObject(ObjectMemBuffer->getMemBufferRef(),
+                     [this](auto MapSection) {
+                       // Assign addresses to all sections. If key corresponds
+                       // to the object created by ourselves, call our regular
+                       // mapping function. If we are loading additional objects
+                       // as part of runtime libraries for instrumentation,
+                       // treat them as extra sections.
+                       mapCodeSections(MapSection);
+                       mapInstrumentationSection("__counters", MapSection);
+                       mapInstrumentationSection("__tables", MapSection);
+                     });
 
-          // TODO: Refactor addRuntimeLibSections to work properly on Mach-O
-          // and use it here.
-  //FIXME! Put this in RtLibrary->link
-//          mapInstrumentationSection("I__setup");
-//          mapInstrumentationSection("I__fini");
-//          mapInstrumentationSection("I__data");
-//          mapInstrumentationSection("I__text");
-//          mapInstrumentationSection("I__cstring");
-//          mapInstrumentationSection("I__literal16");
-
-//  if (auto *RtLibrary = BC->getRuntimeLibrary()) {
-//    RtLibrary->link(*BC, ToolPath, *ES, *OLT);
-//  }
+  // if (auto *RtLibrary = BC->getRuntimeLibrary()) {
+  //   RtLibrary->link(*BC, ToolPath, *Linker, [this](auto MapSection) {
+  //     mapInstrumentationSection("I__setup", MapSection);
+  //     mapInstrumentationSection("I__fini", MapSection);
+  //     mapInstrumentationSection("I__data", MapSection);
+  //     mapInstrumentationSection("I__text", MapSection);
+  //     mapInstrumentationSection("I__cstring", MapSection);
+  //     mapInstrumentationSection("I__literal16", MapSection);
+  //   });
+  // }
 }
 
 void MachORewriteInstance::writeInstrumentationSection(StringRef SectionName,
                                                        raw_pwrite_stream &OS) {
   if (!opts::Instrument)
     return;
   ErrorOr<BinarySection &> Section = BC->getUniqueSectionByName(SectionName);
   if (!Section) {

bolt/lib/Rewrite/RewriteInstance.cpp

 #include "bolt/Profile/BoltAddressTranslation.h"
 #include "bolt/Profile/DataAggregator.h"
 #include "bolt/Profile/DataReader.h"
 #include "bolt/Profile/YAMLProfileReader.h"
 #include "bolt/Profile/YAMLProfileWriter.h"
 #include "bolt/Rewrite/BinaryPassManager.h"
 #include "bolt/Rewrite/DWARFRewriter.h"
 #include "bolt/Rewrite/ExecutableFileMemoryManager.h"
+#include "bolt/Rewrite/JITLinkLinker.h"
 #include "bolt/RuntimeLibs/HugifyRuntimeLibrary.h"
 #include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
 #include "bolt/Utils/CommandLineOpts.h"
 #include "bolt/Utils/Utils.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugFrame.h"
-#include "llvm/ExecutionEngine/RuntimeDyld.h"
 #include "llvm/IR/Function.h"
 #include "llvm/MC/MCAsmBackend.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCAsmLayout.h"
 #include "llvm/MC/MCDisassembler/MCDisassembler.h"
 #include "llvm/MC/MCObjectStreamer.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSymbol.h"
case dwarf::DW_EH_PE_udata8:
       Offset -= 8;
       break;
     }
 
     // Create a relocation against an absolute value since the goal is to
     // preserve the contents of the section independent of the new values
     // of referenced symbols.
     RelocatedEHFrameSection->addRelocation(Offset, nullptr, RelType, Value);
   };
 
   Error E = EHFrameParser::parse(DE, EHFrameSection->getAddress(), createReloc);
   check_error(std::move(E), "failed to patch EH frame");
 }
 
 ArrayRef<uint8_t> RewriteInstance::getLSDAData() {
   return ArrayRef<uint8_t>(LSDASection->getData(),
                            LSDASection->getContents().size());
 }

 }
 
 void RewriteInstance::runOptimizationPasses() {
   NamedRegionTimer T("runOptimizationPasses", "run optimization passes",
                      TimerGroupName, TimerGroupDesc, opts::TimeRewrite);
   BinaryFunctionPassManager::runAllPasses(*BC);
 }
 
-namespace {
-
-class BOLTSymbolResolver : public JITSymbolResolver {
-  BinaryContext &BC;
-
-public:
-  BOLTSymbolResolver(BinaryContext &BC) : BC(BC) {}
-
-  // We are responsible for all symbols
-  Expected<LookupSet> getResponsibilitySet(const LookupSet &Symbols) override {
-    return Symbols;
-  }
-
-  // Some of our symbols may resolve to zero and this should not be an error
-  bool allowsZeroSymbols() override { return true; }
-
-  /// Resolves the address of each symbol requested
-  void lookup(const LookupSet &Symbols,
-              OnResolvedFunction OnResolved) override {
-    JITSymbolResolver::LookupResult AllResults;
-
-    if (BC.EFMM->ObjectsLoaded) {
-      for (const StringRef &Symbol : Symbols) {
-        std::string SymName = Symbol.str();
-        LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
-        // Resolve to a PLT entry if possible
-        if (const BinaryData *I = BC.getPLTBinaryDataByName(SymName)) {
-          AllResults[Symbol] =
-              JITEvaluatedSymbol(I->getAddress(), JITSymbolFlags());
-          continue;
-        }
-        OnResolved(make_error<StringError>(
-            "Symbol not found required by runtime: " + Symbol,
-            inconvertibleErrorCode()));
-        return;
-      }
-      OnResolved(std::move(AllResults));
-      return;
-    }
-
-    for (const StringRef &Symbol : Symbols) {
-      std::string SymName = Symbol.str();
-      LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
-
-      if (BinaryData *I = BC.getBinaryDataByName(SymName)) {
-        uint64_t Address = I->isMoved() && !I->isJumpTable()
-                               ? I->getOutputAddress()
-                               : I->getAddress();
-        LLVM_DEBUG(dbgs() << "Resolved to address 0x"
-                          << Twine::utohexstr(Address) << "\n");
-        AllResults[Symbol] = JITEvaluatedSymbol(Address, JITSymbolFlags());
-        continue;
-      }
-      LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
-      AllResults[Symbol] = JITEvaluatedSymbol(0, JITSymbolFlags());
-    }
-
-    OnResolved(std::move(AllResults));
-  }
-};
-
-} // anonymous namespace
-
 void RewriteInstance::preregisterSections() {
   // Preregister sections before emission to set their order in the output.
   const unsigned ROFlags = BinarySection::getFlags(/*IsReadOnly*/ true,
                                                    /*IsText*/ false,
                                                    /*IsAllocatable*/ true);
   if (BinarySection *EHFrameSection = getSection(getEHFrameSectionName())) {
     // New .eh_frame.
     BC->registerOrUpdateSection(getNewSecPrefix() + getEHFrameSectionName(),
void RewriteInstance::emitAndLink() {
 
   //////////////////////////////////////////////////////////////////////////////
   // Assign addresses to new sections.
   //////////////////////////////////////////////////////////////////////////////
 
   // Get output object as ObjectFile.
   std::unique_ptr<MemoryBuffer> ObjectMemBuffer =
       MemoryBuffer::getMemBuffer(BOS->str(), "in-memory object file", false);
-  std::unique_ptr<object::ObjectFile> Obj = cantFail(
-      object::ObjectFile::createObjectFile(ObjectMemBuffer->getMemBufferRef()),
-      "error creating in-memory object");
 
-  BOLTSymbolResolver Resolver = BOLTSymbolResolver(*BC);
+  auto EFMM = std::make_unique<ExecutableFileMemoryManager>(*BC);
+  EFMM->setNewSecPrefix(getNewSecPrefix());
+  EFMM->setOrgSecPrefix(getOrgSecPrefix());
+
+  Linker = std::make_unique<JITLinkLinker>(*BC, std::move(EFMM));
+  Linker->loadObject(ObjectMemBuffer->getMemBufferRef(),
+                     [this](auto MapSection) { mapFileSections(MapSection); });
 
   MCAsmLayout FinalLayout(
       static_cast<MCObjectStreamer *>(Streamer.get())->getAssembler());
 
-  // Disable stubs because RuntimeDyld may try to increase the size of
-  // sections accounting for stubs. We need those sections to match the
-  // same size seen in the input binary, in case this section is a copy
-  // of the original one seen in the binary.
-  BC->EFMM.reset(new ExecutableFileMemoryManager(*BC, /*AllowStubs=*/false));
-  BC->EFMM->setNewSecPrefix(getNewSecPrefix());
-  BC->EFMM->setOrgSecPrefix(getOrgSecPrefix());
-
-  RTDyld.reset(new decltype(RTDyld)::element_type(*BC->EFMM, Resolver));
-  RTDyld->setProcessAllSections(false);
-  RTDyld->loadObject(*Obj);
-
-  // Assign addresses to all sections. If key corresponds to the object
-  // created by ourselves, call our regular mapping function. If we are
-  // loading additional objects as part of runtime libraries for
-  // instrumentation, treat them as extra sections.
-  mapFileSections(*RTDyld);
-
-  RTDyld->finalizeWithMemoryManagerLocking();
-  if (RTDyld->hasError()) {
-    errs() << "BOLT-ERROR: RTDyld failed: " << RTDyld->getErrorString() << "\n";
-    exit(1);
-  }
-
   // Update output addresses based on the new section map and
   // layout. Only do this for the object created by ourselves.
   updateOutputValues(FinalLayout);
 
   if (opts::UpdateDebugSections)
     DebugInfoRewriter->updateLineTableOffsets(FinalLayout);
 
   if (RuntimeLibrary *RtLibrary = BC->getRuntimeLibrary())
-    RtLibrary->link(*BC, ToolPath, *RTDyld, [this](RuntimeDyld &R) {
+    RtLibrary->link(*BC, ToolPath, *Linker, [this](auto MapSection) {
       // Map newly registered sections.
-      this->mapAllocatableSections(*RTDyld);
+      this->mapAllocatableSections(MapSection);
     });
 
   // Once the code is emitted, we can rename function sections to actual
   // output sections and de-register sections used for emission.
   for (BinaryFunction *Function : BC->getAllBinaryFunctions()) {
     ErrorOr<BinarySection &> Section = Function->getCodeSection();
     if (Section &&
         (Function->getImageAddress() == 0 || Function->getImageSize() == 0))
void RewriteInstance::updateLKMarkers() {
   }
   outs() << "BOLT-INFO: patching linux kernel sections. Total patches per "
             "section are as follows:\n";
   for (const std::pair<const std::string, uint64_t> &KV : PatchCounts)
     outs() << "  Section: " << KV.first << ", patch-counts: " << KV.second
            << '\n';
 }
 
-void RewriteInstance::mapFileSections(RuntimeDyld &RTDyld) {
+void RewriteInstance::mapFileSections(BOLTLinker::SectionMapper MapSection) {
   BC->deregisterUnusedSections();
 
   // If no new .eh_frame was written, remove relocated original .eh_frame.
   BinarySection *RelocatedEHFrameSection =
       getSection(".relocated" + getEHFrameSectionName());
   if (RelocatedEHFrameSection && RelocatedEHFrameSection->hasValidSectionID()) {
     BinarySection *NewEHFrameSection =
         getSection(getNewSecPrefix() + getEHFrameSectionName());
     if (!NewEHFrameSection || !NewEHFrameSection->isFinalized()) {
-      // RTDyld will still have to process relocations for the section, hence
+      // JITLink will still have to process relocations for the section, hence
       // we need to assign it the address that wouldn't result in relocation
       // processing failure.
-      RTDyld.reassignSectionAddress(RelocatedEHFrameSection->getSectionID(),
-                                    NextAvailableAddress);
+      MapSection(*RelocatedEHFrameSection, NextAvailableAddress);
       BC->deregisterSection(*RelocatedEHFrameSection);
     }
   }
 
-  mapCodeSections(RTDyld);
+  mapCodeSections(MapSection);
 
   // Map the rest of the sections.
-  mapAllocatableSections(RTDyld);
+  mapAllocatableSections(MapSection);
 }
 
 std::vector<BinarySection *> RewriteInstance::getCodeSections() {
   std::vector<BinarySection *> CodeSections;
   for (BinarySection &Section : BC->textSections())
     if (Section.hasValidSectionID())
       CodeSections.emplace_back(&Section);
 
std::vector<BinarySection *> RewriteInstance::getCodeSections() {
   };
 
   // Determine the order of sections.
   llvm::stable_sort(CodeSections, compareSections);
 
   return CodeSections;
 }
 
-void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
+void RewriteInstance::mapCodeSections(BOLTLinker::SectionMapper MapSection) {
   if (BC->HasRelocations) {
     // Map sections for functions with pre-assigned addresses.
     for (BinaryFunction *InjectedFunction : BC->getInjectedBinaryFunctions()) {
       const uint64_t OutputAddress = InjectedFunction->getOutputAddress();
       if (!OutputAddress)
         continue;
 
       ErrorOr<BinarySection &> FunctionSection =
           InjectedFunction->getCodeSection();
       assert(FunctionSection && "function should have section");
       FunctionSection->setOutputAddress(OutputAddress);
-      RTDyld.reassignSectionAddress(FunctionSection->getSectionID(),
-                                    OutputAddress);
+      MapSection(*FunctionSection, OutputAddress);
       InjectedFunction->setImageAddress(FunctionSection->getAllocAddress());
       InjectedFunction->setImageSize(FunctionSection->getOutputSize());
     }
 
     // Populate the list of sections to be allocated.
     std::vector<BinarySection *> CodeSections = getCodeSections();
 
     // Remove sections that were pre-allocated (patch sections).
if (!AllocationDone)
       NextAvailableAddress = allocateAt(NextAvailableAddress);
 
     // Do the mapping for ORC layer based on the allocation.
     for (BinarySection *Section : CodeSections) {
       LLVM_DEBUG(
           dbgs() << "BOLT: mapping " << Section->getName() << " at 0x"
                  << Twine::utohexstr(Section->getAllocAddress()) << " to 0x"
                  << Twine::utohexstr(Section->getOutputAddress()) << '\n');
-      RTDyld.reassignSectionAddress(Section->getSectionID(),
-                                    Section->getOutputAddress());
+      MapSection(*Section, Section->getOutputAddress());
       Section->setOutputFileOffset(
           getFileOffsetForAddress(Section->getOutputAddress()));
     }
 
     // Check if we need to insert a padding section for hot text.
     if (PaddingSize && !opts::UseOldText)
       outs() << "BOLT-INFO: padding code to 0x"
              << Twine::utohexstr(NextAvailableAddress)
for (auto &BFI : BC->getBinaryFunctions()) {
     bool TooLarge = false;
     ErrorOr<BinarySection &> FuncSection = Function.getCodeSection();
     assert(FuncSection && "cannot find section for function");
     FuncSection->setOutputAddress(Function.getAddress());
     LLVM_DEBUG(dbgs() << "BOLT: mapping 0x"
                       << Twine::utohexstr(FuncSection->getAllocAddress())
                       << " to 0x" << Twine::utohexstr(Function.getAddress())
                       << '\n');
-    RTDyld.reassignSectionAddress(FuncSection->getSectionID(),
-                                  Function.getAddress());
+    MapSection(*FuncSection, Function.getAddress());
     Function.setImageAddress(FuncSection->getAllocAddress());
     Function.setImageSize(FuncSection->getOutputSize());
     if (Function.getImageSize() > Function.getMaxSize()) {
       TooLarge = true;
       FailedAddresses.emplace_back(Function.getAddress());
     }
 
     // Map jump tables if updating in-place.
     if (opts::JumpTables == JTS_BASIC) {
       for (auto &JTI : Function.JumpTables) {
         JumpTable *JT = JTI.second;
         BinarySection &Section = JT->getOutputSection();
         Section.setOutputAddress(JT->getAddress());
         Section.setOutputFileOffset(getFileOffsetForAddress(JT->getAddress()));
         LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mapping JT " << Section.getName()
                           << " to 0x" << Twine::utohexstr(JT->getAddress())
                           << '\n');
-        RTDyld.reassignSectionAddress(Section.getSectionID(), JT->getAddress());
+        MapSection(Section, JT->getAddress());
       }
     }
 
     if (!Function.isSplit())
       continue;
 
     assert(Function.getLayout().isHotColdSplit() &&
            "Cannot allocate more than two fragments per function in "
if (TooLarge) {
       FF.setFileOffset(getFileOffsetForAddress(NextAvailableAddress));
       ColdSection->setOutputAddress(FF.getAddress());
     }
 
     LLVM_DEBUG(
         dbgs() << formatv(
             "BOLT: mapping cold fragment {0:x+} to {1:x+} with size {2:x+}\n",
             FF.getImageAddress(), FF.getAddress(), FF.getImageSize()));
-    RTDyld.reassignSectionAddress(ColdSection->getSectionID(), FF.getAddress());
+    MapSection(*ColdSection, FF.getAddress());
 
     if (TooLarge)
       BC->deregisterSection(*ColdSection);
 
     NextAvailableAddress += FF.getImageSize();
   }
 
   // Add the new text section aggregating all existing code sections.
   // This is pseudo-section that serves a purpose of creating a corresponding
   // entry in section header table.
   int64_t NewTextSectionSize =
BinarySection &Section =
                                   NewTextSectionSize,
                                   16);
     Section.setOutputAddress(NewTextSectionStartAddress);
     Section.setOutputFileOffset(
         getFileOffsetForAddress(NewTextSectionStartAddress));
   }
 }
 
-void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
+void RewriteInstance::mapAllocatableSections(
+    BOLTLinker::SectionMapper MapSection) {
   // Allocate read-only sections first, then writable sections.
   enum : uint8_t { ST_READONLY, ST_READWRITE };
   for (uint8_t SType = ST_READONLY; SType <= ST_READWRITE; ++SType) {
     const uint64_t LastNextAvailableAddress = NextAvailableAddress;
     if (SType == ST_READWRITE) {
       // Align R+W segment to regular page size
       NextAvailableAddress = alignTo(NextAvailableAddress, BC->RegularPageSize);
       NewWritableSegmentAddress = NextAvailableAddress;
for (BinarySection &Section : BC->allocatableSections()) {
 
       if (Section.hasSectionRef()) {
         LLVM_DEBUG({
           dbgs() << "BOLT-DEBUG: mapping original section " << Section.getName()
                  << " to 0x" << Twine::utohexstr(Section.getAddress()) << '\n';
         });
         Section.setOutputAddress(Section.getAddress());
         Section.setOutputFileOffset(Section.getInputFileOffset());
-        RTDyld.reassignSectionAddress(Section.getSectionID(),
-                                      Section.getAddress());
+        MapSection(Section, Section.getAddress());
       } else {
         NextAvailableAddress =
             alignTo(NextAvailableAddress, Section.getAlignment());
         LLVM_DEBUG({
           dbgs() << "BOLT: mapping section " << Section.getName() << " (0x"
                  << Twine::utohexstr(Section.getAllocAddress()) << ") to 0x"
                  << Twine::utohexstr(NextAvailableAddress) << ":0x"
                  << Twine::utohexstr(NextAvailableAddress +
                                      Section.getOutputSize())
                  << '\n';
         });
 
-        RTDyld.reassignSectionAddress(Section.getSectionID(),
-                                      NextAvailableAddress);
+        MapSection(Section, NextAvailableAddress);
         Section.setOutputAddress(NextAvailableAddress);
         Section.setOutputFileOffset(
             getFileOffsetForAddress(NextAvailableAddress));
 
         NextAvailableAddress += Section.getOutputSize();
       }
     }
 
if (RelocatedEHFrameSection) {
     BC->deregisterSection(*RelocatedEHFrameSection);
   }
 
   LLVM_DEBUG(dbgs() << "BOLT-DEBUG: size of .eh_frame after merge is "
                     << NewEHFrameSection->getOutputSize() << '\n');
 }
 
 uint64_t RewriteInstance::getNewValueForSymbol(const StringRef Name) {
-  uint64_t Value = RTDyld->getSymbol(Name).getAddress();
-  if (Value != 0)
-    return Value;
+  auto Value = Linker->lookupSymbol(Name);
+  if (Value)
+    return *Value;
 
   // Return the original value if we haven't emitted the symbol.
   BinaryData *BD = BC->getBinaryDataByName(Name);
   if (!BD)
     return 0;
 
   return BD->getAddress();
 }

bolt/lib/RuntimeLibs/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   BinaryFormat
+  JITLink
   MC
   Object
-  RuntimeDyld
   Support
   )
 
 add_llvm_library(LLVMBOLTRuntimeLibs
   RuntimeLibrary.cpp
   HugifyRuntimeLibrary.cpp
   InstrumentationRuntimeLibrary.cpp
 

bolt/lib/RuntimeLibs/HugifyRuntimeLibrary.cpp

 //===- bolt/RuntimeLibs/HugifyRuntimeLibrary.cpp - Hugify RT Library ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the HugifyRuntimeLibrary class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/RuntimeLibs/HugifyRuntimeLibrary.h"
 #include "bolt/Core/BinaryFunction.h"
-#include "llvm/ExecutionEngine/RuntimeDyld.h"
+#include "bolt/Core/Linker.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/CommandLine.h"
 
 using namespace llvm;
 using namespace bolt;
 
 namespace opts {
if (!BC.StartFunctionAddress) {
     errs() << "BOLT-ERROR: hugify runtime libraries require a known entry "
               "point of "
               "the input binary\n";
     exit(1);
   }
 }
 
 void HugifyRuntimeLibrary::link(BinaryContext &BC, StringRef ToolPath,
-                                RuntimeDyld &RTDyld,
-                                std::function<void(RuntimeDyld &)> OnLoad) {
+                                BOLTLinker &Linker,
+                                BOLTLinker::SectionsMapper MapSections) {
+
   std::string LibPath = getLibPath(ToolPath, opts::RuntimeHugifyLib);
-  loadLibrary(LibPath, RTDyld);
-  OnLoad(RTDyld);
-  RTDyld.finalizeWithMemoryManagerLocking();
-  if (RTDyld.hasError()) {
-    outs() << "BOLT-ERROR: RTDyld failed: " << RTDyld.getErrorString() << "\n";
-    exit(1);
-  }
+  loadLibrary(LibPath, Linker, MapSections);
 
   assert(!RuntimeStartAddress &&
          "We don't currently support linking multiple runtime libraries");
-  RuntimeStartAddress = RTDyld.getSymbol("__bolt_hugify_self").getAddress();
+  RuntimeStartAddress = Linker.lookupSymbol("__bolt_hugify_self").value_or(0);
   if (!RuntimeStartAddress) {
     errs() << "BOLT-ERROR: instrumentation library does not define "
               "__bolt_hugify_self: "
            << LibPath << "\n";
     exit(1);
   }
 }

bolt/lib/RuntimeLibs/InstrumentationRuntimeLibrary.cpp

 //===- bolt/RuntimeLibs/InstrumentationRuntimeLibrary.cpp -----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the InstrumentationRuntimeLibrary class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
 #include "bolt/Core/BinaryFunction.h"
 #include "bolt/Core/JumpTable.h"
+#include "bolt/Core/Linker.h"
 #include "bolt/Utils/CommandLineOpts.h"
-#include "llvm/ExecutionEngine/RuntimeDyld.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/CommandLine.h"
 
 using namespace llvm;
 using namespace bolt;
 
 namespace opts {
MCSection *TablesSection = BC.Ctx->getMachOSection(
         "__BOLT", "__tables", MachO::S_REGULAR, SectionKind::getData());
     TablesSection->setAlignment(llvm::Align(BC.RegularPageSize));
     Streamer.switchSection(TablesSection);
     emitString("__bolt_instr_tables", buildTables(BC));
   }
 }
 
 void InstrumentationRuntimeLibrary::link(
-    BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
-    std::function<void(RuntimeDyld &)> OnLoad) {
+    BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
+    BOLTLinker::SectionsMapper MapSections) {
   std::string LibPath = getLibPath(ToolPath, opts::RuntimeInstrumentationLib);
-  loadLibrary(LibPath, RTDyld);
-  OnLoad(RTDyld);
-  RTDyld.finalizeWithMemoryManagerLocking();
-  if (RTDyld.hasError()) {
-    outs() << "BOLT-ERROR: RTDyld failed: " << RTDyld.getErrorString() << "\n";
-    exit(1);
-  }
+  loadLibrary(LibPath, Linker, MapSections);
 
   if (BC.isMachO())
     return;
 
-  RuntimeFiniAddress = RTDyld.getSymbol("__bolt_instr_fini").getAddress();
+  RuntimeFiniAddress = Linker.lookupSymbol("__bolt_instr_fini").value_or(0);
   if (!RuntimeFiniAddress) {
     errs() << "BOLT-ERROR: instrumentation library does not define "
               "__bolt_instr_fini: "
            << LibPath << "\n";
     exit(1);
   }
-  RuntimeStartAddress = RTDyld.getSymbol("__bolt_instr_start").getAddress();
+  RuntimeStartAddress = Linker.lookupSymbol("__bolt_instr_start").value_or(0);
   if (!RuntimeStartAddress) {
     errs() << "BOLT-ERROR: instrumentation library does not define "
               "__bolt_instr_start: "
            << LibPath << "\n";
     exit(1);
   }
   outs() << "BOLT-INFO: output linked against instrumentation runtime "
             "library, lib entry point is 0x"
          << Twine::utohexstr(RuntimeFiniAddress) << "\n";
   outs() << "BOLT-INFO: clear procedure is 0x"
          << Twine::utohexstr(
-                RTDyld.getSymbol("__bolt_instr_clear_counters").getAddress())
+                Linker.lookupSymbol("__bolt_instr_clear_counters").value_or(0))
          << "\n";
 
   emitTablesAsELFNote(BC);
 }
 
 std::string InstrumentationRuntimeLibrary::buildTables(BinaryContext &BC) {
   std::string TablesStr;
   raw_string_ostream OS(TablesStr);

bolt/lib/RuntimeLibs/RuntimeLibrary.cpp

 //===- bolt/RuntimeLibs/RuntimeLibrary.cpp - Runtime Library --------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the RuntimeLibrary class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/RuntimeLibs/RuntimeLibrary.h"
+#include "bolt/Core/Linker.h"
 #include "bolt/RuntimeLibs/RuntimeLibraryVariables.inc"
 #include "bolt/Utils/Utils.h"
 #include "llvm/BinaryFormat/Magic.h"
-#include "llvm/ExecutionEngine/RuntimeDyld.h"
 #include "llvm/Object/Archive.h"
+#include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/Path.h"
 
 #define DEBUG_TYPE "bolt-rtlib"
 
 using namespace llvm;
 using namespace bolt;
 
 void RuntimeLibrary::anchor() {}
std::string RuntimeLibrary::getLibPath(StringRef ToolPath,
   llvm::sys::path::append(LibPath, LibFileName);
   if (!llvm::sys::fs::exists(LibPath)) {
     errs() << "BOLT-ERROR: library not found: " << LibPath << "\n";
     exit(1);
   }
   return std::string(LibPath.str());
 }
 
-void RuntimeLibrary::loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld) {
+void RuntimeLibrary::loadLibrary(StringRef LibPath, BOLTLinker &Linker,
+                                 BOLTLinker::SectionsMapper MapSections) {
   ErrorOr<std::unique_ptr<MemoryBuffer>> MaybeBuf =
       MemoryBuffer::getFile(LibPath, false, false);
   check_error(MaybeBuf.getError(), LibPath);
   std::unique_ptr<MemoryBuffer> B = std::move(MaybeBuf.get());
   file_magic Magic = identify_magic(B->getBuffer());
 
   if (Magic == file_magic::archive) {
     Error Err = Error::success();
     object::Archive Archive(B.get()->getMemBufferRef(), Err);
     for (const object::Archive::Child &C : Archive.children(Err)) {
       std::unique_ptr<object::Binary> Bin = cantFail(C.getAsBinary());
       if (object::ObjectFile *Obj = dyn_cast<object::ObjectFile>(&*Bin))
-        RTDyld.loadObject(*Obj);
+        Linker.loadObject(Obj->getMemoryBufferRef(), MapSections);
     }
     check_error(std::move(Err), B->getBufferIdentifier());
   } else if (Magic == file_magic::elf_relocatable ||
              Magic == file_magic::elf_shared_object) {
     std::unique_ptr<object::ObjectFile> Obj = cantFail(
         object::ObjectFile::createObjectFile(B.get()->getMemBufferRef()),
         "error creating in-memory object");
-    RTDyld.loadObject(*Obj);
+    Linker.loadObject(Obj->getMemoryBufferRef(), MapSections);
   } else {
     errs() << "BOLT-ERROR: unrecognized library format: " << LibPath << "\n";
     exit(1);
   }
 }