This is an archive of the discontinued LLVM Phabricator instance.

[Orc] Add JITLink debug support plugin for ELF x86-64
ClosedPublic

Authored by sgraenitz on Feb 23 2021, 2:03 PM.

Details

Summary

This patch adds a new ObjectLinkingLayer plugin DebugObjectManagerPlugin and infrastructure to handle creation of DebugObjects and their registration in OrcTargetProcess. The current implementation only covers ELF on x86-64, but the infrastructure is not limited to that.

The journey starts with a new LinkGraph / JITLinkContext pair being created for a MaterializationResponsibility in ORC's ObjectLinkingLayer. It sends a notifyMaterializing() notification, which is forwarded to all registered plugins. The DebugObjectManagerPlugin aims to create a DebugObject form the provided target triple and object buffer. (Future implementations might create DebugObjects from a LinkGraph in other ways.) On success it will track it as the pending DebugObject for the MaterializationResponsibility.

This patch only implements the ELFDebugObject for x86-64 targets. It follows the RuntimeDyld approach for debug object setup: it captures a copy of the input object, parses all section headers and prepares to patch their load-address fields with their final addresses in target memory. It instructs the plugin to report the section load-addresses once they are available. The plugin overrides modifyPassConfig() and installs a JITLink post-allocation pass to capture them.

Once JITLink emitted the finalized executable, the plugin emits and registers the DebugObject. For emission it requests a new JITLinkMemoryManager::Allocation with a single read-only segment, copies the object with patched section load-addresses over to working memory and triggers finalization to target memory. For registration, it notifies the DebugObjectRegistrar provided in the constructor and stores the previously pending`DebugObject` as registered for the corresponding MaterializationResponsibility.

The DebugObjectRegistrar registers the DebugObject with the target process. llvm-jitlink uses the TPCDebugObjectRegistrar, which calls llvm_orc_registerJITLoaderGDBWrapper() in the target process via TargetProcessControl to emit a jit_code_entry compatible with the GDB JIT interface [1]. So far the implementation only supports registration and no removal. It appears to me that it wouldn't raise any new design questions, so I left this as an addition for the near future.

[1] https://sourceware.org/gdb/current/onlinedocs/gdb/JIT-Interface.html

Diff Detail

Event Timeline

sgraenitz created this revision.Feb 23 2021, 2:03 PM
sgraenitz requested review of this revision.Feb 23 2021, 2:03 PM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 23 2021, 2:03 PM
sgraenitz added inline comments.Feb 23 2021, 3:28 PM
llvm/lib/ExecutionEngine/JITLink/DebugSupport.cpp
60 ↗(On Diff #325896)

Is it possible to set an allocation's working memory to an existing buffer? Is it worth adding?

llvm/lib/ExecutionEngine/Orc/DebugObjectManagerPlugin.cpp
35

This still holds. Can we have more than one resource per MR?

llvm/lib/ExecutionEngine/Orc/TargetProcess/JITLoaderGDB.cpp
65

This replicates the code for the GDB JIT interface definition fom ExecutionEngine and causes the duplication in D97339 eventually. Is there a way to shared it without creating a dedicated library? If there's a better way, I am eager to learn about it.

Otherwise, maybe one more reason to look at this https://reviews.llvm.org/D95747#2532432 ?

This looks excellent.

I still think that we want to approach this using debug object construction in the long run, but this is too useful to hold up on that point. One thing that I would like to see (and that I think is doable with this scheme as-is) is to move the changes out of JITLink and entirely into Orc / ObjectLinkingLayer. Keeping them out of the deeper levels of the stack (i.e. JITLink) should make them easier to re-do later when we switch to the "right" long term solution.

So, I would be inclined to move DebugObject into ORC, and void notifyMaterializing(jitlink::LinkGraph &G) and Expected<std::unique_ptr<DebugObject>> createDebugObject(LinkGraph &G) from JITLinkContext to ObjectLinkingLayerJITLinkContext (the decl for which may need to be hoisted into the ObjectLinkingLayer.h header for now). The new methods should carry comments saying that they're deprecated -- we can leave them in for the purposes of DebugObjectManagerPlugin for now, but we don't want other people to start using them.

llvm/lib/ExecutionEngine/JITLink/DebugSupport.cpp
60 ↗(On Diff #325896)

Good question. I ran across another limitation of the existing memory management API today: If you want to mutate a block in a way that changes its size (e.g. prepending or appending) then you need to allocate memory for that in the graph and copy the content into that, only to have to copy all of that back out into working memory later. What you really wanted was a way to say
"reserve me X bytes of additional memory for this block and *I'll* copy the content".

Let's stick to a copy for now, but I think we want to revisit the memory manager API in the not-too-distant-future.

llvm/lib/ExecutionEngine/Orc/DebugObjectManagerPlugin.cpp
35

In this context a "resource" is some abstract thing that you manage. You can certainly manage more than one kind of resource on behalf of an MR (e.g. for an ObjC++ object you have allocated memory tracked by ObjectLinkingLayer, eh-frames tracked by the EHFrameRegistrationPlugin, and runtime data tracked by the platform plugin).

The usual idiom is to hold a DenseMap<ResourceKey, std::vector<ResourceT>>, where ResourceT is whatever resource type(s) the plugin is managing, which I think is what you're doing.

// We should never have more than one pending debug object.

Is this true? JITLink supports running multiple concurrent instances -- If the plugin is to support concurrency it should be possible to have more than one debug object in-flight at a time.

sgraenitz updated this revision to Diff 326822.Feb 26 2021, 3:23 PM

Make DebugObject an implementation detail of the plugin

Thanks for the feedback.

So, I would be inclined to move DebugObject into ORC, and void notifyMaterializing(jitlink::LinkGraph &G) and Expected<std::unique_ptr<DebugObject>> createDebugObject(LinkGraph &G) from JITLinkContext to ObjectLinkingLayerJITLinkContext

At first I followed the approach from EH frame registration and thought I could reuse some JITLink code for handling ELF headers, but yes there is no real need to have the debug object classes there. And yes moving them over to ORC also avoids extra API. It was a bit more reorganization than I expected initially, but I think the result is better than the first version.

The new methods should carry comments saying that they're deprecated -- we can leave them in for the purposes of DebugObjectManagerPlugin for now, but we don't want other people to start using them.

I need to double-check my previous comments and will add the note then.

(the decl for which may need to be hoisted into the ObjectLinkingLayer.h header for now).

Instead of creating a new interface, I just added a MemoryBufferRef parameter to Plugin::notifyMaterializing() for now. Is that acceptable? Otherwise, can you give a rough description of the API you had in mind?

What I realized during my ExtensibleRTTI hack is, that I could also capture the incoming object file with an extra ObjectLayer above the linking layer. At first sight it doesn't look impossible. I'd also have to add some support code first and I don't know if plugin-layer-coop is a design question you already considered or rejected. What do you think?

sgraenitz updated this revision to Diff 326927.Feb 27 2021, 2:05 PM

Allow more than one pending debug object per MaterializationUnit

sgraenitz added inline comments.Feb 27 2021, 2:36 PM
llvm/lib/ExecutionEngine/Orc/DebugObjectManagerPlugin.cpp
35

If the plugin is to support concurrency it should be possible to have more than one debug object in-flight at a time.

Yes, but wouldn't they all have their own MaterializationResponsibilities?

My first thought was: yes, I can let modifyPassConfig() and notifyLoaded() simply act on all pending debug objects registered for the MR. I submitted the change and just realized now, that it is incorrect. Please see my other inline comment.

395

This obviously won't work because it adds a pass for each pending object. But what would be the right way? Sending each section range to all the pending objects? That can't seem to be correct. What am I missing?

sgraenitz updated this revision to Diff 326932.Feb 27 2021, 2:50 PM

Notify the debugger on notifyEmitted(), not notifyLoded(). This caused a funny behavior for pending breakpoints in LLDB: they were resolved and implemented correctly but never hit, because the JIT had overwritten the instruction in the meantime!

lhames accepted this revision.Mar 1 2021, 5:21 PM

Instead of creating a new interface, I just added a MemoryBufferRef parameter to Plugin::notifyMaterializing() for now. Is that acceptable? Otherwise, can you give a rough description of the API you had in mind?

I think that sounds good for now. Could you just add a note saying that it's deprecated? As soon as JITLink reaches the point where we can represent debug info in the graph we'll switch to using that to construct a new object,

llvm/lib/ExecutionEngine/Orc/DebugObjectManagerPlugin.cpp
35

Oh -- You're right. Each in-flight object will have a distinct MaterializationResponsibility, so you can assume one debug object per MR.

Sorry for the confusion!

This revision is now accepted and ready to land.Mar 1 2021, 5:21 PM
sgraenitz edited the summary of this revision. (Show Details)Mar 2 2021, 3:09 AM
sgraenitz updated this revision to Diff 327420.Mar 2 2021, 5:40 AM

Revert support for multiple pending debug objects per MR. Update documentation in comments and polish.

This revision was landed with ongoing or failed builds.Mar 2 2021, 6:08 AM
This revision was automatically updated to reflect the committed changes.
thakis added a subscriber: thakis.Mar 2 2021, 12:58 PM

I see you've now disabled this on Windows. I think it works on Windows with

diff --git a/llvm/tools/llvm-jitlink/CMakeLists.txt b/llvm/tools/llvm-jitlink/CMakeLists.txt
index 8d511b17fca2..34e892f30ed6 100644
--- a/llvm/tools/llvm-jitlink/CMakeLists.txt
+++ b/llvm/tools/llvm-jitlink/CMakeLists.txt
@@ -24,4 +24,16 @@ add_llvm_tool(llvm-jitlink
   llvm-jitlink-macho.cpp
   )
 
+# export_executable_symbols() is a no-op on Windows if neither
+# LLVM_EXPORTED_SYMBOL_FILE nor LLVM_EXPORT_SYMBOLS_FOR_PLUGINS are set, but
+# the jitlink tests need llvm_orc_registerJITLoaderGDBWrapper to be exported
+# from the executable to work.
+# FIXME: Find a better workaround. Maybe this should use LLVM_EXPORTED_SYMBOL_FILE
+# and an .exports file now that the binary has a required export.
+if (WIN32)
+  target_link_options(llvm-jitlink PRIVATE
+    "/export:llvm_orc_registerJITLoaderGDBWrapper"
+    )
+endif()
+
 export_executable_symbols(llvm-jitlink)

but as the comment says, setting LLVM_EXPORTED_SYMBOL_FILE seems like it'd probably be a nicer fix.

(But I don't have a problem with this being disabled on Windows.)

thakis added a comment.Mar 2 2021, 3:12 PM

Since your win

thakis added a comment.Mar 2 2021, 3:13 PM

fix attempt in http://reviews.llvm.org/rGbbdb4c8c9bcef0e8db751630accc04ad874f54e7 didn't help, i landed that in 900f07611330. please see if you want to follow up on the fixme therein.

ASDenysPetrov reopened this revision.EditedMar 4 2021, 7:27 AM
ASDenysPetrov added a subscriber: ASDenysPetrov.

My build fails after hack to this patch https://reviews.llvm.org/rG900f076113302e26e1939541b546b0075e3e9721. Could you revise your solution?
I'm using Win10, GCC+ninja
Here is my build output:

[32/92] Linking CXX executable bin\llvm-jitlink.exe
FAILED: bin/llvm-jitlink.exe
cmd.exe /C "cd . && D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2 -Wl,--stack,16777216    /export:llvm_orc_registerJITLoaderGDBWrapper tools/llvm-jitlink/CMakeFiles/llvm-jitlink.dir/llvm-jitlink.cpp.obj tools/llvm-jitlink/CMakeFiles/llvm-jitlink.dir/llvm-jitlink-elf.cpp.obj tools/llvm-jitlink/CMakeFiles/llvm-jitlink.dir/llvm-jitlink-macho.cpp.obj -o bin\llvm-jitlink.exe -Wl,--out-implib,lib\libllvm-jitlink.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/libLLVMX86Desc.a  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a  lib/libLLVMBinaryFormat.a  lib/libLLVMExecutionEngine.a  lib/libLLVMJITLink.a  lib/libLLVMMC.a  lib/libLLVMObject.a  lib/libLLVMOrcJIT.a  lib/libLLVMOrcShared.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMSupport.a  lib/libLLVMMCDisassembler.a  lib/libLLVMExecutionEngine.a  lib/libLLVMJITLink.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMOrcShared.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMPasses.a  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMLinker.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMVectorize.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMObject.a  lib/libLLVMBitReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMProfileData.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lpsapi  -lshell32  -lole32  -luuid  -ladvapi32  D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a  lib/libLLVMDemangle.a  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
c++.exe: error: /export:llvm_orc_registerJITLoaderGDBWrapper: No such file or directory
[47/92] Linking CXX executable bin\clang-check.exe
ninja: build stopped: subcommand failed.

Here is my setup output if you're wondering cmake -GNinja ../llvm -DLLVM_LIT_ARGS=-sv -DLLVM_ENABLE_PROJECTS=clang -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../../install -DLLVM_ENABLE_ASSERTIONS=ON:

This revision is now accepted and ready to land.Mar 4 2021, 7:27 AM

My build fails after hack to this patch https://reviews.llvm.org/rG900f076113302e26e1939541b546b0075e3e9721. Could you revise your solution?

Hello Denys, I reverted the temporary hack and fixed the issue that Nico described with e984c2b06f0c59265ba2172bbdb2811c72123701
Does it fix the issue for you too?

ASDenysPetrov requested changes to this revision.Mar 9 2021, 3:01 AM

Does it fix the issue for you too?

Hi, Stefan. Thanks. I've checked. Yes, this fix (rGe984c2b06f0c59265ba2172bbdb2811c72123701) works for me.

This revision now requires changes to proceed.Mar 9 2021, 3:01 AM
sgraenitz accepted this revision.Mar 9 2021, 10:34 AM

Attempting to close the review after it was re-opened.

lhames accepted this revision.Mar 9 2021, 10:41 AM

@ASDenysPetrov I guess you set the Needs Revision status accidentally?

ASDenysPetrov added a comment.EditedMar 9 2021, 11:59 AM

@ASDenysPetrov I guess you set the Needs Revision status accidentally?

Right. Probably the action was choosed previously in the drop-down list and I missed it. Sorry.

Right. Probably the action was choosed previously in the drop-down list and I missed it. Sorry.

Can you withdraw that, i.e. accept the patch? Otherwise, I cannot close the review.

ASDenysPetrov accepted this revision.Mar 9 2021, 2:25 PM
This revision is now accepted and ready to land.Mar 9 2021, 2:25 PM
mehdi_amini closed this revision.Mar 9 2021, 3:15 PM