[runtimes] Enable LTO when supported

[ldionne]

Summary

We were not using LTO to build the various runtime libraries. Enabling
this is a no-brainer.

I did measure the code size changes in the hopes that it would provide
an improvement in the size of the dylib. However, after stripping the
dylibs, there is no improvement:

File Without LTO (-O2 -g) With Thin LTO (-flto=thin -O2 -g)

libc++.1.0.dylib 1.0M 1.0M
libc++.a 9.2M 12M
libc++abi.1.0.dylib 252K 252K
libc++abi.a 1.5M 2.0M

The static archives become larger, however this is expected since we
are now including more information that would allow LTO to happen when
users produce their final linked image.

Note that this patch does not enable LTO for libunwind, because this
currently crashes when trying to link libunwind.dylib. I followed up
with linker folks to investigate the issue.

[mstorsjo]

While enabling LTO, if possible probably is fine for the shared library, it's a much less clear cut deal for the static library.

Regarding "if possible", the CI run seems to show that it was enabled in cases where it probably didn't work in the end - e.g. the CI linux configurations, compiling with clang but linking with the system GNU linkers, probably without a suitable LLVM LTO plugin, or perhaps it's a case where it would need to use llvm-ar instead of GNU ar or something like that.

With the static library, if built with LTO, it pushes the compilation of the library onto every single user who links an executable. If linking a single executable that's probably fine, but for a generic toolchain that can be used in a multitude of ways, it's probably a different tradeoff. If linking e.g. hundreds of small executables against a static libc++ (for users who don't enable LTO otherwise), you'd now end up compiling the libc++ code hundreds of times, with no actual benefit if the rest of the executables linked isn't built with LTO.

I believe this issue should show up in CI too, for the static-only configurations, where the runtime of the testsuite that builds thousands of test executables probably would skyrocket. (The current CI run seems to have failed earlier than that, so I couldn't check how it actually went this time.)

So IMO, whether to enable LTO for the static library build or not should probably be a build time option, for toolchain vendors to decide upon.

[phosek]

In D152967#4423591, @mstorsjo wrote:

While enabling LTO, if possible probably is fine for the shared library, it's a much less clear cut deal for the static library.

Regarding "if possible", the CI run seems to show that it was enabled in cases where it probably didn't work in the end - e.g. the CI linux configurations, compiling with clang but linking with the system GNU linkers, probably without a suitable LLVM LTO plugin, or perhaps it's a case where it would need to use llvm-ar instead of GNU ar or something like that.

With the static library, if built with LTO, it pushes the compilation of the library onto every single user who links an executable. If linking a single executable that's probably fine, but for a generic toolchain that can be used in a multitude of ways, it's probably a different tradeoff. If linking e.g. hundreds of small executables against a static libc++ (for users who don't enable LTO otherwise), you'd now end up compiling the libc++ code hundreds of times, with no actual benefit if the rest of the executables linked isn't built with LTO.

I believe this issue should show up in CI too, for the static-only configurations, where the runtime of the testsuite that builds thousands of test executables probably would skyrocket. (The current CI run seems to have failed earlier than that, so I couldn't check how it actually went this time.)

So IMO, whether to enable LTO for the static library build or not should probably be a build time option, for toolchain vendors to decide upon.

I agree completely, for static libraries LTO is undesirable since it affects usage of those libraries and isn't transparent to users. This is one of the motivations for FatLTO support @paulkirth is working on (see the stack of changes starting at D146776) which allows building a static library that can be used for regular builds as well as LTO. Those changes should be landing soon. Even though only ELF is supported at the moment, there's no technical reason why we couldn't support FatLTO for COFF or Mach-O, we just need to implement the necessary LLD support.

I think we should have two flags to separately control the use of LTO for shared and static libraries. For shared libraries, we could default to ON (we should also have a CI configuration to test the OFF configuration). For static libraries the default should be OFF for now. Once FatLTO support lands, we can revisit this and consider changing the default for static libraries to ON. Alternatively, we could omit this change altogether and let the user control the use of LTO via CMAKE_INTERPROCEDURAL_OPTIMIZATION.

Regarding size reduction, in my experience ThinLTO doesn't typically help with size, in fact it often leads to size increase due to a more aggressive inlining and devirtualization. We had a much better success with regular LTO which I think is a better fit for runtime libraries due to their small size.

[ldionne]

Thanks for chiming in, folks. I think I am coming to agree with you and in particular I believe @phosek 's suggestion to let this be controlled by CMAKE_INTERPROCEDURAL_OPTIMIZATION makes a lot of sense -- so there'd be nothing for us to do and this patch can be abandoned.

I'll abandon, if someone disagrees and thinks we should move forward with some version of this patch, please comment.

libcxx/CMakeLists.txt

 
 set(CMAKE_FOLDER "libc++")
 
 set(LIBCXX_SOURCE_DIR  ${CMAKE_CURRENT_SOURCE_DIR})
 set(LIBCXX_BINARY_DIR  ${CMAKE_CURRENT_BINARY_DIR})
 set(LIBCXX_BINARY_INCLUDE_DIR "${LIBCXX_BINARY_DIR}/include/c++build")
 
 include(GNUInstallDirs)
-include(WarningFlags)
 
 # Require out of source build.
 include(MacroEnsureOutOfSourceBuild)
 MACRO_ENSURE_OUT_OF_SOURCE_BUILD(
  "${PROJECT_NAME} requires an out of source build. Please create a separate
  build directory and run 'cmake /path/to/${PROJECT_NAME} [options]' there."
  )
 if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" AND "${CMAKE_CXX_SIMULATE_ID}" STREQUAL "MSVC")

 
 if (WIN32 AND LIBCXX_ENABLE_STATIC_ABI_LIBRARY)
   # If linking libcxxabi statically into libcxx, skip the dllimport attributes
   # on symbols we refer to from libcxxabi.
   add_definitions(-D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS)
 endif()
 
 # Setup all common build flags =================================================
+include(LTO)
+include(WarningFlags)
 function(cxx_add_common_build_flags target)
   cxx_add_basic_build_flags(${target})
   cxx_add_warning_flags(${target} ${LIBCXX_ENABLE_WERROR} ${LIBCXX_ENABLE_PEDANTIC})
   cxx_add_windows_flags(${target})
   cxx_add_exception_flags(${target})
   cxx_add_rtti_flags(${target})
   cxx_add_module_flags(${target})
   cxx_link_system_libraries(${target})
+  cxx_enable_lto(${target})
 endfunction()
 
 #===============================================================================
 # Setup Source Code And Tests
 #===============================================================================
 add_subdirectory(include)
 add_subdirectory(src)
 add_subdirectory(utils)

libcxxabi/src/CMakeLists.txt

 split_list(LIBCXXABI_LINK_FLAGS)
 
 # FIXME: libc++abi.so will not link when modules are enabled because it depends
 # on symbols defined in libc++.so which has not yet been built.
 if (LLVM_ENABLE_MODULES)
   string(REPLACE "-Wl,-z,defs" "" CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS}")
 endif()
 
+include(LTO)
 include(WarningFlags)
 
 # Build the shared library.
 add_library(cxxabi_shared_objects OBJECT EXCLUDE_FROM_ALL ${LIBCXXABI_SOURCES} ${LIBCXXABI_HEADERS})
 cxx_add_warning_flags(cxxabi_shared_objects ${LIBCXXABI_ENABLE_WERROR} ${LIBCXXABI_ENABLE_PEDANTIC})
+cxx_enable_lto(cxxabi_shared_objects)
 if (LIBCXXABI_USE_LLVM_UNWINDER)
   if (LIBCXXABI_STATICALLY_LINK_UNWINDER_IN_SHARED_LIBRARY)
     target_link_libraries(cxxabi_shared_objects PUBLIC unwind_shared_objects) # propagate usage requirements
     target_sources(cxxabi_shared_objects PUBLIC $<TARGET_OBJECTS:unwind_shared_objects>)
   else()
     target_link_libraries(cxxabi_shared_objects PUBLIC unwind_shared)
   endif()
 endif()
if (LIBCXXABI_ENABLE_EXCEPTIONS)
       endif()
     endif()
   endif()
 endif()
 
 # Build the static library.
 add_library(cxxabi_static_objects OBJECT EXCLUDE_FROM_ALL ${LIBCXXABI_SOURCES} ${LIBCXXABI_HEADERS})
 cxx_add_warning_flags(cxxabi_static_objects ${LIBCXXABI_ENABLE_WERROR} ${LIBCXXABI_ENABLE_PEDANTIC})
+cxx_enable_lto(cxxabi_static_objects)
 if (LIBCXXABI_USE_LLVM_UNWINDER AND LIBCXXABI_STATICALLY_LINK_UNWINDER_IN_STATIC_LIBRARY)
   target_link_libraries(cxxabi_static_objects PUBLIC unwind_static_objects) # propagate usage requirements
   target_sources(cxxabi_static_objects PUBLIC $<TARGET_OBJECTS:unwind_static_objects>)
 endif()
 target_link_libraries(cxxabi_static_objects PRIVATE cxx-headers ${LIBCXXABI_STATIC_LIBRARIES} ${LIBCXXABI_LIBRARIES})
 target_link_libraries(cxxabi_static_objects PUBLIC cxxabi-headers)
 set_target_properties(cxxabi_static_objects
   PROPERTIES

runtimes/cmake/Modules/LTO.cmake

+include(CheckIPOSupported)
+check_ipo_supported(RESULT _ipo_supported LANGUAGES C CXX)
+
+function(cxx_enable_lto target)
+  if (_ipo_supported)
+    set_property(TARGET ${target} PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)
+  endif()
+endfunction()