[flang] Enabling flang to build with BUILD_SHARED_LIBS=On

[Renaud-K]

Summary

When building with static libs, the linker only needs to find the minimal dependencies required to arrive to an executable.
When building with shared libs, the linker requires that all the symbols in the shared libs be resolved which exposes more dependencies.

Making the dependency with FortranEvaluate explicit so we can build with BUILD_SHARED_LIBS=On

[awarzynski]

Hi @Renaud-K , thank you for submitting this!

Interestingly, flang-aarch64-ubuntu-sharedlibs is fine. This Buildbot worker was set-up specifically to test BUILD_SHARED_LIBS. Could you double-check whether this change is really needed? I will also test tomorrow.

[Renaud-K]

The change does NOT appear to be necessary for main of llvm-project, however I could reproduce it with fir-dev branch of f18-llvm-project. I was told that changes to this file should go through phabricator and percolate to fir-dev. I am not too sure what to do now.

[richard.barton.arm]

The change does NOT appear to be necessary for main of llvm-project, however I could reproduce it with fir-dev branch of f18-llvm-project. I was told that changes to this file should go through phabricator and percolate to fir-dev.

I don't understand that. If there is a build issue on fir-dev but not on main then the bugfix only needs to go on fir-dev and not main. When the extra code on fir-dev that causes the bug comes to main then the bugfix can come with it at that point. main has buildbots for shared libs that ensure we notice it if that happens.

[Renaud-K]

I have been looking some more into how the flang-new driver is being built.
I see that flangFrontend and flangFrontendTool libraries have been introduced and include dependencies with the Fortran libraries. It makes things look elegant. When we link, we just have to say flangFrontend and we get all the Fortran libraries. However when building with shared libs , you can see that the flangFrontEnd libraries are very small, they no longer include any of the code they depend on. So the attempt to have an umbrella library does not work because the linker does not link with the dependent libraries. In a shared lib flow, you have to make them explicit.

If we had some very tight "C like" interfaces, you could guarantee that the driver only makes calls into the objects of the flangFrontend libraries and this could work. But with all the rich header files included which have inlined templated code you have no control over what gets called and it is only a matter of time until this breaks. Here is the change, precisely single header file change, from the fir-dev branch that is causing the link to fail. It is a very small, very reasonable change. https://github.com/flang-compiler/f18-llvm-project/blame/fir-dev/flang/include/flang/Evaluate/expression.h#L97

I guess we could make it so the call is not inlined. But my feeling is that the umbrella library approach is brittle and that this issue will come up again.

[Renaud-K]

Fixed issue this way: https://github.com/flang-compiler/f18-llvm-project/pull/822

[awarzynski]

In D103540#2797278, @Renaud-K wrote:

I see that flangFrontend and flangFrontendTool libraries have been introduced and include dependencies with the Fortran libraries.

No. These libraries implement the frontend driver. They depend on the frontend libraries, because they "drive" it.

In D103540#2797278, @Renaud-K wrote:

However when building with shared libs , you can see that the flangFrontEnd libraries are very small, they no longer include any of the code they depend on.

This is simply the difference between static and shared libraries, right? There is nothing special about flangFrontend here.

So the attempt to have an umbrella library does not work because the linker does not link with the dependent libraries. In a shared lib flow, you have to make them explicit.

In both cases (static and shared) all dependencies should be expressed explicitly. It just so happens that in the case of static libraries, one can miss a dependency and things will build just fine anyway. That's because linking is delayed until the final binary is build (e.g. flang-new): as long as the missing dependency is also present in the final binary, everything is fine. So the missing dependency could be coming from yet another library.

I guess we could make it so the call is not inlined. But my feeling is that the umbrella library approach is brittle and that this issue will come up again.

This did indeed come up in the past:

• https://github.com/llvm/llvm-project/commit/d007bbd986d9ab004c34efaca27aff4a7633b9e9
• https://reviews.llvm.org/D89111
• https://reviews.llvm.org/D88484

In order to defend Flang from such failures in the future, we've set-up a Buildbot worker that builds with BUILD_SHARED_LIBS set to On.

As you admitted, there is no issue with llvm-project/flang and this change is not required.

In D103540#2799260, @Renaud-K wrote:

Fixed issue this way: https://github.com/flang-compiler/f18-llvm-project/pull/822

Great, it sounds like you've resolved your original issue!

flang/tools/flang-driver/CMakeLists.txt

target_link_libraries(flang-new
   flangFrontend
   flangFrontendTool
 )
 
 clang_target_link_libraries(flang-new
   PRIVATE
   clangDriver
   clangBasic
+  FortranEvaluate
 )
 
 install(TARGETS flang-new DESTINATION bin)