Page MenuHomePhabricator

LLVM overhaul to avoid linking LLVM component libraries with libLLVM

Authored by jwhowarth on Feb 6 2016, 8:44 AM.



Current trunk suffers several thousand llvm test suite failures for the LLVM_LINK_LLVM_DYLIB build due to incorrect linkages of both libLLVM and the individual LLVM component static libraries. The proposed changes to the llvm_config() macro in cmake/modules/LLVM-Config.cmake and the llvm_add_library() function in cmake/modules/AddLLVM.cmake solves this issue by stripping the individual LLVM component libraries from LLVM_LINK_COMPONENTS and LINK_COMPONENTS for the LLVM_LINK_LLVM_DYLIB build. To address the remaining instances where the unittests executables are incorrectly linked against both libLLVM and libLLVMSupport in the LLVM_LINK_LLVM_DYLIB build, the explicit linkage of LLVMSupport is removed from utils/unittest/CMakeLists.txt. However, the explicit linkage on LLVMSupport in the llvm_add_library name function of cmake/modules/AddLLVM.cmake and utils/unittest/UnitTestMain/CMakeLists.txt are retained and conditionalized on LLVM_LINK_LLVM_DYLIB. These are required to prevent the BUILD_SHARED_LIBS build from missing required the LLVMSupport linkages on unittests executables. Finally a new --enable-llvm-link-llvm-dylib is introduced in utils/llvm-build/llvmbuild/ and utilized in the top-level CMakeLists.txt file for the LLVM_LINK_LLVM_DYLIB build. In utils/llvm-build/llvmbuild/, this new option sets a use_llvm_link_llvm_dylib global which is then tested in the write_library_table routine to control pruning 'Support' from the required dependencies of the gtest library. This produces the desired result for the DLLVM_LINK_LLVM_DYLIB build of the producing a tools/llvm-config/ file containing...

{ "gtest", "libgtest.a", false, { } },

instead of

{ "gtest", "libgtest.a", false, { "support" } },

The command...

find . -name link.txt -print0 | xargs -0 cat | grep libgtest | grep -v -c libLLVMSupport

was used to verify that none of the linkages on libgtest were missing the required linkages of libLLVMSupport for the stock and BUILD_SHARED_LIBS builds.

find . -name link.txt -print0 | xargs -0 cat | grep libgtest | grep libLLVM | grep -c libLLVMSupport

was used to verify that none of the unitests linkages on the LLVM_LINK_LLVM_DYLIB build had both libLLVM.dylib and libLLVMSupport.

Confirmed bootstraps on x86_64 darwin for the stock (static), BUILD_SHARED_LIBS and LLVM_LINK_LLVM_DYLIB builds of current trunk with this patch.

Diff Detail


Event Timeline

jwhowarth updated this revision to Diff 47085.Feb 6 2016, 8:44 AM
jwhowarth retitled this revision from to LLVM overhaul to avoid linking LLVM component libraries with libLLVM.
jwhowarth updated this object.
jwhowarth added reviewers: axw, beanz.
jwhowarth set the repository for this revision to rL LLVM.

Confirmed for 3-stage bootstraps (with stage-2/stage-3 file comparison) on x86-64 darwin of llvm/clang/clang-extras/compiler-rt/polly/openmp/libc++ svn trunk with diff 47085 applied that no regressions on the stock (static), BUILD_SHARED_LIBS and LLVM_LINK_LLVM_DYLIB builds are produced.

The original bug report this bug address is [[ URL | ]].

Unfortunately, I haven't caught every single hard-coding of LLVMSupport linkages. We still have a libLLVMSupport.a linkage with libLLVM and libgtest for FileCheck which seems to be hardcoded in cmake/modules/TableGen.cmake. So we will have to conditionalize that on. We are also getting those for the linkage of not, llvm-tblgen and yams-bench as well.

jwhowarth updated this revision to Diff 47114.Feb 6 2016, 7:40 PM

Eliminated linkages against both libLLVM and libLLVMSupport for FileCheck, KillTheDoctor, not and yams-bench with addition of a conditional on LLVM_LINK_LLVM_DYLIB in their CMakeLists.txt. Also, restored LIBS assignment using a conditional in unittest/CMakeLists.txt. I suspect the cmake files in the utils subdirectories are configured differently because TableGen build needs to link against selective static LLVM components as it is used in the later LLVM component builds.

Tested Diff 47114 with 3-stage bootstraps and stage2/stage3 file comparison for llvm/clang/clang-tools-extra/polly/openmp/libc++ on x86_64 darwin. The stock (static), BUILD_SHARED_LIBS and LLVM_LINK_LLVM_DYLIB builds all show no regressions in the test suite. Also confirmed on the LLVM_LINK_LLVM_DYLIB build using...

find . -name link.txt -print0 | xargs -0 cat | grep libLLVM | grep libLLVMSupport | grep -v "/usr/bin/ar"

that the only executable files linked against libLLVMSupport.a are llvm-tblgen and clang-tblgen and neither (correctly) are linked to libLLVM.dylib. Also repeated the same testing regime with the patch applied to current 3.8 branch without issues. So the patch now looks complete to me.

Confirmed that the instance of 'LLVM' appended on target_link_libraries for LLVM_LINK_LLVM_DYLIB is essential in utils/FileCheck/CMakeLists.txt, utils/KillTheDoctor/CMakeLists.txt, utils/not/CMakeLists.txt and utils/yaml-bench/CMakeLists.txt. Removal of 'LLVM" from those instances of target_link_libraries() for LLVM_LINK_LLVM_DYLIB in Diff ID 47114 prunes it from the generated link.txt.

Likewise, also confirmed that the instance of the appended 'LLVM' on add_llvm_library() in utils/unittest/UnitTestMain/CMakeLists.txt is essential. Removal of 'LLVM" from those instances of target_link_libraries() for LLVM_LINK_LLVM_DYLIB in Diff ID 47114 prunes it from the generated link.txt.

With regard to the changes in utils/unittest/CMakeLists.txt, these were introduced with the commit of and then refactored in Since it is unclear exactly where utils/unittest/CMakeLists.txt is used, I think we should treat it very conservatively as I do in Diff ID 47114.

While a change to utils/unittest/CMakeLists.txt is required to avoid these failures for the LLVM_LINK_LLVM_DYLIB build...

Failing Tests (4):

Clang-Unit :: Tooling/ToolingTests/addTargetAndModeForProgramName.AddsTargetAndMode
Clang-Unit :: Tooling/ToolingTests/addTargetAndModeForProgramName.IgnoresExistingMode
Clang-Unit :: Tooling/ToolingTests/addTargetAndModeForProgramName.IgnoresExistingTarget
Clang-Unit :: Tooling/ToolingTests/addTargetAndModeForProgramName.PathIgnored

the fix can be reduced to just placing a 'if (NOT LLVM_LINK_LLVM_DYLIB)' wrapper around the setting of LIBS to LLVMSupport. The linkage for libLLVM is already being added elsewhere for these test cases.

jwhowarth updated this revision to Diff 47155.Feb 7 2016, 1:11 PM

Simplify fix for utils/unittest/CMakeLists.txt to just adding 'if (NOT LLVM_LINK_LLVM_DYLIB)' wrapper.

Confirmed for Diff ID 47155 that a 3-stage bootstrap with stage2/stage3 file comparison for llvm/clang/clang-tools-extra/polly/openmp/libc++ on x86_64 darwin built with LLVM_LINK_LLVM_DYLIB produces no test suite regressions on both trunk and 3.8 branch. All other sections of the proposed patch have now been verified as essential.

axw requested changes to this revision.Feb 8 2016, 2:59 AM
axw edited edge metadata.

I hope what I've said is clear - please ask if not. I can see if I can find some time to propose an alternative, but also very busy with day job.

890–901 ↗(On Diff #47155)

Just remove the LLVMSupport. CMake will link the program with LLVM or LLVMSupport because gtest depends on it.

5–9 ↗(On Diff #47155)

A couple of issues:

  • this is subverting the explicit "DISABLE_LLVM_LINK_LLVM_DYLIB" in add_llvm_utility
  • if that weren't there, you could/should just do:
add_llvm_utility(FileCheck FileCheck.cpp)

(llvm_config will decide which libraries to link with depending on LLVM_LINK_LLVM_DYLIB)

Is there any harm in the utilities not using libLLVM? When I made the changes, only the packaged/distributed tools were made to link to libLLVM. I don't think the internal utilities really need to.

Ditto for the other utilities.

5 ↗(On Diff #47155)

as above

353 ↗(On Diff #47155)

This is a bit of a dirty hack (hard-coding component names and such), and I'm not convinced modifying LLVMBuild is the right layer to be changing anyway. Doing this means that "llvm-config --libs --link-static gtest" won't do the right thing anymore.

I really think you should be doing this at the CMake layer. I think what you want to do, instead, is to have LLVMBuild output CMake fragments with the component names rather than library names, and then include them in the LLVM_LINK_COMPONENTS handling.

That is:

  • don't remove the dependency from gtest to the Support component
  • in CMake, interpret the "required_libraries: Support" as "add Support to LLVM_LINK_COMPONENTS"; we'll then either link with libLLVM or libLLVMSupport as we do already
35–39 ↗(On Diff #47155)

AFAICT, the explicit dependency is not even needed, as long as you don't remove the dependency of gtest->{LLVM,LLVMSupport}.

1–17 ↗(On Diff #47155)

You should just be able to remove the LLVMSupport line, and otherwise stick with the original.
gtest_main depends on gtest, and gtest depends on LLVM or LLVMSupport. CMake will link the transitive closure.

This revision now requires changes to proceed.Feb 8 2016, 2:59 AM

Any suggestions on how to adjust cmake to interpret the "required_libraries: Support" as "add Support to LLVM_LINK_COMPONENTS"? I am lost on that one.

890–901 ↗(On Diff #47155)

This works as long as the associated unittests explicitly add Support to LLVM_LINK_COMPONENTS, Currently I have found five CMakeLists.txt across the llvm, cfe and clang-tools-extra sources that need this fix.

5–9 ↗(On Diff #47155)

Without a linkage on libLLVM, the LLVM_LINK_LLVM_DYLIB build of FileCheck fails on numerous unresolved symbols like llvm::cl::Option::anchor().

353 ↗(On Diff #47155)

Any suggestions in how to adjust cmake to interpret the "required_libraries: Support" as "add Support to LLVM_LINK_COMPONENTS"?

1–17 ↗(On Diff #47155)

Removal of the LLVMSupport from the add_llvm_library arguments here will require all unittests to explicitly define...


in their local CMakeLists.txt file. If that is acceptable to force on the maintainers for code with unittests then we can go that route. Currently llvm/unittests/DebugInfo/DWARF/CMakeLists.txt, llvm/unittests/DebugInfo/PDB/CMakeLists.txt, llvm/unittests/Linker/CMakeLists.txt, cfe/unittests/libclang/CMakeLists.txt and clang-tools-extra/unittests/clang-apply-replacements/CMakeLists.txt need those fixes.

jwhowarth added a comment.EditedFeb 8 2016, 10:35 AM

Doesn't the correct handling of the 'support' library dependency of 'gtest' have to be special cased for LLVM_LINK_LLVM_DYLIB in expand_topologically() of cmake/modules/LLVM-Config.cmake? It preceded by the comment...

# Perform a post-order traversal of the dependency graph.
# This duplicates the algorithm used by llvm-config, originally
# in tools/llvm-config/llvm-config.cpp, function ComputeLibsForComponents.
jwhowarth updated this revision to Diff 47227.Feb 8 2016, 11:18 AM
jwhowarth edited edge metadata.

This version still requires the pruning of 'support' from the required_libraries of the 'gtest' library at the cmake level.

$ find . -name link.txt -print0 | xargs -0 cat | grep libgtest | grep libLLVM.dylib | grep -c libLLVMSupport

Using message statements to print out link_components in explicit_llvm_config() of cmake/modules/LLVM-Config.cmake shows that the 'gtest' linkage never passes through there.

I suspect the problem *might* be at...

# Add the explicit dependency information for this library.
# It would be nice to verify that we have the dependencies for this library
# name, but using get_property(... SET) doesn't suffice to determine if a
# property has been set to an empty value.
get_property(lib_deps GLOBAL PROPERTY LLVMBUILD_LIB_DEPS_${name})

in the llvm_add_library() function of cmake/modules/AddLLVM.cmake. I see it is passed 'gtest' during the cmake run on lib_deps, however I don't understand how to deconstruction the LLVMBUILD_LIB_DEPS_gtest that is being invoked.

It does look like the 'get_property(lib_deps GLOBAL PROPERTY LLVMBUILD_LIB_DEPS_${name})' call for name as gtest is the problem. The build directory has...


in the generated LLVMBuild.cmake.

jwhowarth updated this revision to Diff 47276.Feb 8 2016, 7:42 PM
jwhowarth edited edge metadata.

For LLVM_LINK_LLVM_DYLIB builds, set the library dependencies of the gtest library to LLVM within llvm_add_library(), using a set_property() on LLVMBUILD_LIB_DEPS_gtest, rather than reading them from tools/llvm-config/ using get_property(). The patch avoids any attempt to refactor the unitests build machinery to keep these changes suitable for back porting to 3.8 branch. Confirmed that bootstraps of llvm/clang/clang-tools-extra/compiler-rt/openmp/polly/libc++ on x86_64 darwin for the stock (static), BUILD_SHARED_LIBS and LLVM_LINK_LLVM_DYLIB builds of current trunk produce no test suite regressions. Refactoring the unitests build machinery should be a separate follow-on patch to trunk.

jwhowarth added inline comments.Feb 8 2016, 8:16 PM
5–9 ↗(On Diff #47276)

Reconfirmed that changing...

Index: utils/FileCheck/CMakeLists.txt
--- utils/FileCheck/CMakeLists.txt      (revision 260055)
+++ utils/FileCheck/CMakeLists.txt      (working copy)
@@ -1,5 +1,5 @@
-target_link_libraries(FileCheck LLVMSupport)

for the LLVM_LINK_LLVM_DYLIB build with Diff ID 47276 ends up linking against libLLVMSupport.a rather than libLLVM.dylib. So the full conditional with explicit passing of LLVM on target_link_libraries() for LLVM_LINK_LLVM_DYLIB in Diff ID 47276 is required.

jwhowarth updated this revision to Diff 47297.Feb 8 2016, 11:13 PM

Drop changes to the linkage of the utils programs from patch. Successsfully regression tested on x86_64 darwin for a llvm/clang/clang-tools-extra/compiler-rt/polly/openmp/libc++ build using LLVM_LINK_LLVM_DYLIB.

Diff 6 ID 47297 also tested for the same llvm/clang/clang-tools-extra/compiler-rt/openmp/polly/libc++ build on x86_64 darwin for the stock (static) and BUILD_SHARED_LIBS builds without regression. Likewise tested the patch back ported to 3.8 branch for the LLVM_LINK_LLVM_DYLIB build without regressions.

chapuni added a subscriber: chapuni.Feb 9 2016, 3:35 PM
beanz edited edge metadata.Feb 9 2016, 6:43 PM

Sorry I haven't been engaged on this thread. I'll try to do better.

Comments inline.

477 ↗(On Diff #47297)

I'm really not sure I like this approach, but if you're going to do it, rather than doing this here, why not do it in the root CMakeLists.txt?

I think after line 491 you could add:


If you do it there, then I don't think any of the property reading code needs to change here.

The reason I don't like this approach is that the reason llvm-build exists is to supply dependency information the same way to projects using multiple build systems. That said, in this case it is probably fine because the gtest stuff should only be used by LLVM projects. So I think this is safe.

484 ↗(On Diff #47297)

I don't believe this is correct. What happens if LLVM_LINK_COMPONENTS contains things not in LLVM_DYLIB_COMPNENTS?

This is why I had suggested on an earlier thread that we are doing the component filtering at the wrong layer. We probably should do it inside llvm_map_components_to_libnames.

894 ↗(On Diff #47297)

Playing devil's advocate, what if LLVM_DYLIB_COMPONENTS didn't include LLVMSupport?

axw added inline comments.Feb 9 2016, 7:11 PM
484 ↗(On Diff #47297)

How about just using llvm_config here instead, passing USE_SHARED as appropriate? That macro has already been fixed so that it links with libLLVM if USE_SHARED, but also with the component libraries not included in libLLVM.

jwhowarth updated this revision to Diff 47421.Feb 9 2016, 10:10 PM
jwhowarth edited edge metadata.

Use refactored patch from Andrew Wilkins instead. Tested stock (static), BUILD_SHARED_LIBS and LLVM_LINK_LLVM_DYLIB builds of current trunk with this patch on x86_64 darwin for llvm/clang/clang-tools-extra/compiler-rt/polly/openmp/libc++ with no regressions introduced and proper linkages verified. Okay for trunk and back port to 3.8 branch?

axw resigned from this revision.Feb 10 2016, 5:28 PM
axw removed a reviewer: axw.

IMO this is good to go, but I'm biased as I was involved in writing it. I'll resign as reviewer for that reason. @beanz, PTAL.

You're quite right that there's an issue if LLVM_DYLIB_COMPONENTS does not include all components, but I'm not sure that changing llvm_map_components_to_libnames is the right thing to do. For one thing, that's what libLLVM uses to determine the library names; that could be changed, of course. For another, there's llvm_config which essentially does what we want: conditionally link with libLLVM, and with any libraries that are not included in that.

The immediate issue is resolved, and LLVM_LINK_LLVM_DYLIB is opt-in and still fairly new. I'd like to defer the missing-component fix to a follow-up if you don't mind.

beanz accepted this revision.Feb 10 2016, 10:39 PM
beanz edited edge metadata.

LGTM. Sorry this took so much back and forth.

Thank you both of you for all your work on this patch.


This revision is now accepted and ready to land.Feb 10 2016, 10:39 PM
This revision was automatically updated to reflect the committed changes.