Page MenuHomePhabricator

[ORC] Add LookupKind::TransitiveStatic for linker-resolved transitive dependencies
AbandonedPublic

Authored by sgraenitz on Feb 24 2020, 3:07 PM.

Details

Reviewers
lhames
Summary

This lookup kind tells symbol generators that the lookup is being performed for transitive module dependencies. ThinLtoJIT wants to avoid loading and emitting modules for transitive dependencies where possible (if all requested symbols for one module are callable) and instead emit synthetic call-through stubs, which will load and emit the module when reached.

Event Timeline

sgraenitz created this revision.Feb 24 2020, 3:07 PM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 24 2020, 3:07 PM
sgraenitz added inline comments.Feb 24 2020, 3:33 PM
llvm/include/llvm/ExecutionEngine/Orc/Core.h
89

This would need a description in the comment. Not sure though, if it's the best way to achieve my goal in the first place.

llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp
54

Alternatively, the generator could make the decision dependent on the required symbol state. However, currently this information isn't passed to the generators.

lhames added a comment.Mar 3 2020, 5:11 PM

So the aim here is to enable generation of lazy stubs without having to materialize any IR, right?

I think the right way to do this: They’re supposed to have just enough information to decide whether the materialization unit being added should be part of the JIT’d program or not. They’re not intended for making smart decisions about how to build the materialization units.

I think a better approach would be to define a ModuleSummaryMaterializationUnit (or something similar) and have that be the thing that you add to your JIT layer. The layer would build stubs based on the interface defined in the module summary and defer compilation of the IR.

Let me know if I’m missing problems with that approach though: I’m afraid I haven’t had time to think about this deeply.

btw — I’m on #llvm on discord a lot these days. This might be a handy one to chat about in real time.

sgraenitz marked an inline comment as done.Mar 4 2020, 2:36 PM

So the aim here is to enable generation of lazy stubs without having to materialize any IR, right?

Right, AND without having to parse the respective bitcode files. In many cases the main module has transitive dependencies to a huge number of modules and materializing it causes parsing and emitting reexports for all of them. I do see that the case, which I refer to as TransitiveStatic here, could be the default for my generator. Entry points would then get a synthetic call-through too and load&emit the module on call (that would be acceptable). The problem, however, would then be the lookups issued from discovery. These should cause materialization of the requested modules. Unfortunately, there isn't a good way for the generator to distinguish them either (see inline comment).

I think the right way to do this: They’re supposed to have just enough information to decide whether the materialization unit being added should be part of the JIT’d program or not. They’re not intended for making smart decisions about how to build the materialization units.

I think a better approach would be to define a ModuleSummaryMaterializationUnit (or something similar) and have that be the thing that you add to your JIT layer. The layer would build stubs based on the interface defined in the module summary and defer compilation of the IR.

Not sure if that goes too much into the details, but one issue here is that module summaries only provide GUIDs. I don't have the full interface before parsing bitcode. The only symbol names I know are the requested ones.

The underlying question for me was, whether -- in general -- it could make sense for generators to distinguish between lookups coming from the linker and lookups from the "user".

llvm/examples/ThinLtoJIT/ThinLtoJIT.cpp
263

This is the lookup issued from discovery. It's intended to force materialization of the modules that define LookaheadSymbols. As described above, I could assume TransitiveStatic the default. Then I'd probably pass LookupKind::DLSym here for the generator to indicate forced materialization, but that would be an abuse too.

sgraenitz abandoned this revision.Aug 14 2020, 7:59 AM