Page MenuHomePhabricator

[opt] Ensure the IR-Linker is available to plugins.
Needs RevisionPublic

Authored by Meinersbur on Jul 25 2019, 11:25 AM.

Details

Summary

When using a statically linked opt executable (no BUILD_SHARED_LIBS or LLVM_LINK_LLVM_DYLIB), the linker only includes symbols used (directly or transitively) by the opt program itself. However, when loading another module using the "-load" mechanism from Passes/PassPlugin.cpp or Support/PluginLoader.cpp, it may require symbols that have not been included into the executable.

In this specific case, Polly uses the LLVM-IR linker, which is not included in the statically linked opt executable. As a result, some Polly regressions tests may fail with the error message

opt: symbol lookup error: LLVMPolly.so: undefined symbol: _ZN4llvm6LinkerC1ERNS_6ModuleE

(It does work using clang which uses the LLVM linker).

This patch ensures that the LLVMLinker is available in the opt executable by referencing it in one of opt's main object files. The exported symbol ensurePluginSymbols creates a linker object without being called anywhere.

This is a pragmatic fix for Polly and unfortunately does not solve the more general problem that not all symbols are available to plugins in statically linked tools. I am open for better solutions.

Diff Detail

Event Timeline

Meinersbur created this revision.Jul 25 2019, 11:25 AM
mehdi_amini requested changes to this revision.Jul 25 2019, 5:58 PM

unfortunately does not solve the more general problem that not all symbols are available to plugins in statically linked tools. I am open for better solutions.

What about "force-load" the archive libraries on link? See what we do in MLIR here: https://github.com/tensorflow/mlir/blob/master/CMakeLists.txt#L22

This revision now requires changes to proceed.Jul 25 2019, 5:58 PM

@Meinersbur what about linking with LLVM-IR linker in your shared module?

What about "force-load" the archive libraries on link? See what we do in MLIR here: https://github.com/tensorflow/mlir/blob/master/CMakeLists.txt#L22

Thank you for the suggestion. During a first test with --whole-archive, I got multiple definitions errors. I will try a bit more.

As another possibility in cmake we could link the object libraries instead of the archive,

@Meinersbur what about linking with LLVM-IR linker in your shared module?

This would cause global variables to be present multiple times within the same address space. If it is an cl::opt, it will immediately error-out.

beanz requested changes to this revision.Jul 26 2019, 9:45 AM

The CMake build system option PLUGIN_TOOL on add_llvm_loadable_module is for this specific purpose, it instructs the link step for the plugin dylib to resolve symbols against the tool.

Polly likely needs to generate different libraries for being a clang plugin vs being an opt plugin, but that is expected and the correct solution to this problem.

The CMake build system option PLUGIN_TOOL on add_llvm_loadable_module is for this specific purpose, it instructs the link step for the plugin dylib to resolve symbols against the tool.

Polly likely needs to generate different libraries for being a clang plugin vs being an opt plugin, but that is expected and the correct solution to this problem.

Thank you for the hint. I was under the impression that this option was added to allow plugins under Windows. I will try it out.

If this is the proper way, why do we even support add_llvm_library(${NAME} MODULE without PLUGIN_TOOL?

beanz added a comment.Jul 26 2019, 2:13 PM

If this is the proper way, why do we even support add_llvm_library(${NAME} MODULE without PLUGIN_TOOL?

The LLVMGold plugin's loader tool isn't part of the LLVM build tree.

The CMake build system option PLUGIN_TOOL on add_llvm_loadable_module is for this specific purpose, it instructs the link step for the plugin dylib to resolve symbols against the tool.

Polly likely needs to generate different libraries for being a clang plugin vs being an opt plugin, but that is expected and the correct solution to this problem.

I'm not sure I understood: does it mean that the plugin dylib would embed the symbols it needs unless it finds them inside the tool (opt)?