[CMake] Don't LTO optimize targets that aren't part of any distribution

[phosek]

Summary

When using distributions, targets that aren't included in any
distribution don't need to be as optimized as targets that are
included since those targets are typically only used for tests.

We might consider avoiding LTO for these targets altogether, see
https://lists.llvm.org/pipermail/llvm-dev/2021-April/149843.html

[phosek]

llvm/cmake/modules/AddLLVM.cmake

240

I'm wondering if we should avoid this branch for non-distribution targets as well to further reduce cost, what do you think?

[smeenai]

This is a neat idea.

llvm/CMakeLists.txt

934

Is this moved up just to get the distribution target map built in time for the new use?

llvm/cmake/modules/AddLLVM.cmake

221

In https://reviews.llvm.org/D89177?vs=on&id=343946#change-XYKH4tDojBz0, I had a get_llvm_distribution target in LLVMDistributionSupport that would tell you whether a target was in a distribution or not. I removed it from the final version of that diff because I added a more specific function instead that handled all the use cases I had, but for this use case, I think adding that function back would be nicer, so that e.g. the LLVM_DISTRIBUTION_FOR_* property naming scheme remains centralized inside LLVMDistributionSupport.

The other thing that function handled was umbrella targets (e.g. making sure llvm-libraries pulled in all LLVM libraries). Do you care about that support here? I haven't actually thought about how LTO would work if you're distributing (static) libraries as part of your distribution, as in would you wanna build them as bitcode or regular archives.

240

I think it'd make sense. Not related to your diff though, but depending on where the linker handles dead-stripping, I imagine it might actually make links *faster*, since it's reducing the amount of data the linker has to process. I think @thakis might have observed that when implementing dead-stripping in LLD for Mach-O.

[phosek]

llvm/CMakeLists.txt

934

Yes, I need LLVM_DISTRIBUTION_FOR_* to be populated before creating any targets.

llvm/cmake/modules/AddLLVM.cmake

221

I like that idea so I'll go ahead do that. I haven't thought of including static libraries when using LTO and I'm not sure how useful that's going to be since those libraries would contain just bitcode. This could be another potential use case for -fembed-bitcode where the static library would contain both machine code and bitcode and consumers can decide which of the two to use (at the expense of increased binary size).

240

Makes sense, I can keep dead-stripping and just drop the -Wl,-O3.

[smeenai]

llvm/cmake/modules/LLVMDistributionSupport.cmake

59–62

The second sentence of this comment "By convention, ..." should be removed ... that's handled by get_target_export_arg now.

131

I think we need an else here to preserve the previous behavior of this function?

[smeenai]

LGTM; thanks for cleaning up get_target_export_arg as well :)

[bogner]

llvm/CMakeLists.txt

934

I'm seeing test failures if I do ninja distribution test-depends and then ninja check and I suspect moving this is related. It's an llvm-config test that fails because lib/libLLVMObjCMetadata.a wasn't built though, so it's entirely possible that moving this just exposed a missing dependency somewhere else.

Do you happen to have any insight here?

[phosek]

llvm/CMakeLists.txt

934

Is it possible to reproduce that failure? We're using ninja distribution with https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/Fuchsia-stage2.cmake and haven't seen that issue but our distribution may contain a different set of targets.

[bogner]

llvm/CMakeLists.txt

934

This might be a problem on my side. Sorry for the noise.

llvm/CMakeLists.txt

 
 # Make sure we don't get -rdynamic in every binary. For those that need it,
 # use export_executable_symbols(target).
 set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "")
 
 include(AddLLVM)
 include(TableGen)
 
+include(LLVMDistributionSupport)
+
 if( MINGW AND NOT "${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" )
   # People report that -O3 is unreliable on MinGW. The traditional
   # build also uses -O2 for that reason:
   llvm_replace_compiler_option(CMAKE_CXX_FLAGS_RELEASE "-O3" "-O2")
 endif()
 
 # Put this before tblgen. Else we have a circular dependence.
 add_subdirectory(lib/Demangle)
foreach(lib ${LLVM_LIBS})
         add_dependencies(install-llvm-libraries-stripped install-${lib}-stripped)
       endif()
     endforeach()
   endif()
 endif()
 
 # This must be at the end of the LLVM root CMakeLists file because it must run
 # after all targets are created.
-include(LLVMDistributionSupport)
 llvm_distribution_add_targets()
 process_llvm_pass_plugins(GEN_CONFIG)
 
 # This allows us to deploy the Universal CRT DLLs by passing -DCMAKE_INSTALL_UCRT_LIBRARIES=ON to CMake
 if (MSVC AND CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows" AND CMAKE_INSTALL_UCRT_LIBRARIES)
   include(InstallRequiredSystemLibraries)
 endif()
 

llvm/cmake/modules/AddLLVM.cmake

elseif(NOT WIN32)
     else()
       set(LLVM_LINKER_DETECTED NO CACHE INTERNAL "")
       message(STATUS "Linker detection: unknown")
     endif()
   endif()
 endif()
 
 function(add_link_opts target_name)
+  get_llvm_distribution(${target_name} in_distribution in_distribution_var)
+  if(NOT in_distribution)
+    # Don't LTO optimize targets that aren't part of any distribution.
+    if (LLVM_ENABLE_LTO)
+      # We may consider avoiding LTO altogether by using -fembed-bitcode
+      # and teaching the linker to select machine code from .o files, see
+      # https://lists.llvm.org/pipermail/llvm-dev/2021-April/149843.html
+      if((UNIX OR MINGW) AND LLVM_USE_LINKER STREQUAL "lld")
+        set_property(TARGET ${target_name} APPEND_STRING PROPERTY
+                      LINK_FLAGS " -Wl,--lto-O0")
+      elseif(LINKER_IS_LLD_LINK)
+        set_property(TARGET ${target_name} APPEND_STRING PROPERTY
+                      LINK_FLAGS " /opt:lldlto=0")
+      endif()
+    endif()
+  endif()
+
   # Don't use linker optimizations in debug builds since it slows down the
   # linker in a context where the optimizations are not important.
   if (NOT uppercase_CMAKE_BUILD_TYPE STREQUAL "DEBUG")
 
     # Pass -O3 to the linker. This enabled different optimizations on different
     # linkers.
-    if(NOT (CMAKE_SYSTEM_NAME MATCHES "Darwin|SunOS|AIX|OS390" OR WIN32))
+    if(NOT (CMAKE_SYSTEM_NAME MATCHES "Darwin|SunOS|AIX|OS390" OR WIN32) AND in_distribution)
       set_property(TARGET ${target_name} APPEND_STRING PROPERTY
                    LINK_FLAGS " -Wl,-O3")
     endif()
 
     if(NOT LLVM_NO_DEAD_STRIP)
       if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
         # ld64's implementation of -dead_strip breaks tools that use plugins.
         set_property(TARGET ${target_name} APPEND_STRING PROPERTY

llvm/cmake/modules/LLVMDistributionSupport.cmake

foreach(distribution ${LLVM_DISTRIBUTIONS})
     endforeach()
   endforeach()
 endfunction()
 
 # The include guard ensures this will only be called once. The rest of this file
 # only defines other functions (i.e. it doesn't execute any more code directly).
 llvm_distribution_build_target_map()
 
+# Look up the distribution a particular target belongs to.
+# - target: The target to look up.
+# - in_distribution_var: The variable with this name is set in the caller's
+#   scope to indicate if the target is in any distribution. If no distributions
+#   have been configured, this will always be set to true.
+# - distribution_var: The variable with this name is set in the caller's scope
+#   to indicate the distribution name for the target. If the target belongs to
+#   the default (unnamed) distribution, or if no distributions have been
+#   configured, it's set to the empty string.
+# - UMBRELLA: The (optional) umbrella target that the target is a part of. For
+#   example, all LLVM libraries have the umbrella target llvm-libraries.
+function(get_llvm_distribution target in_distribution_var distribution_var)
+  if(NOT LLVM_DISTRIBUTION_COMPONENTS AND NOT LLVM_DISTRIBUTIONS)
+    set(${in_distribution_var} YES PARENT_SCOPE)
+    set(${distribution_var} "" PARENT_SCOPE)
+    return()
+  endif()
+
+  cmake_parse_arguments(ARG "" UMBRELLA "" ${ARGN})
+  get_property(distribution GLOBAL PROPERTY LLVM_DISTRIBUTION_FOR_${target})
+  if(ARG_UMBRELLA)
+    get_property(umbrella_distribution GLOBAL PROPERTY LLVM_DISTRIBUTION_FOR_${ARG_UMBRELLA})
+    if(distribution AND umbrella_distribution AND NOT distribution STREQUAL umbrella_distribution)
+      message(SEND_ERROR "Target ${target} has different distribution ${distribution} from its"
+                         " umbrella target ${ARG_UMBRELLA} distribution ${umbrella_distribution}")
+    elseif(NOT distribution)
+      set(distribution ${umbrella_distribution})
+    endif()
+  endif()
+  if(distribution)
+    set(${in_distribution_var} YES PARENT_SCOPE)
+    if(distribution STREQUAL "<DEFAULT>")
+      set(distribution "")
+    endif()
+    set(${distribution_var} "${distribution}" PARENT_SCOPE)
+  else()
+    set(${in_distribution_var} NO PARENT_SCOPE)
+  endif()
+endfunction()
+
 # Get the EXPORT argument to use for an install command for a target in a
 # project. As explained at the top of the file, the project export set for a
 # distribution is named ${project}{distribution}Targets (except for LLVM where
 # it's named ${project}{distribution}Exports for legacy reasons). Also set the
 # ${PROJECT}_${DISTRIBUTION}_HAS_EXPORTS global property to mark the project as
 # having exports for the distribution.
 # - target: The target to get the EXPORT argument for.
 # - project: The project to produce the argument for. IMPORTANT: The casing of
 #   this argument should match the casing used by the project's Config.cmake
 #   file. The correct casing for the LLVM projects is Clang, Flang, LLD, LLVM,
 #   and MLIR.
 # - export_arg_var The variable with this name is set in the caller's scope to
 #   the EXPORT argument for the target for the project.
 # - UMBRELLA: The (optional) umbrella target that the target is a part of. For
 #   example, all LLVM libraries have the umbrella target llvm-libraries.
 function(get_target_export_arg target project export_arg_var)
   string(TOUPPER "${project}" project_upper)
   if(project STREQUAL "LLVM")
     set(suffix "Exports") # legacy
   else()
     set(suffix "Targets")
   endif()
 
-  if(NOT LLVM_DISTRIBUTION_COMPONENTS AND NOT LLVM_DISTRIBUTIONS)
-    set(${export_arg_var} EXPORT ${project}${suffix} PARENT_SCOPE)
-    set_property(GLOBAL PROPERTY ${project_upper}_HAS_EXPORTS True)
-    return()
-  endif()
+  get_llvm_distribution(${target} in_distribution distribution ${ARGN})
 
-  cmake_parse_arguments(ARG "" UMBRELLA "" ${ARGN})
-  get_property(distribution GLOBAL PROPERTY LLVM_DISTRIBUTION_FOR_${target})
-  if(ARG_UMBRELLA)
-    get_property(umbrella_distribution GLOBAL PROPERTY LLVM_DISTRIBUTION_FOR_${ARG_UMBRELLA})
-    if(distribution AND umbrella_distribution AND NOT distribution STREQUAL umbrella_distribution)
-      message(SEND_ERROR "Target ${target} has different distribution ${distribution} from its"
-                         " umbrella target ${ARG_UMBRELLA} distribution ${umbrella_distribution}")
-    elseif(NOT distribution)
-      set(distribution ${umbrella_distribution})
-    endif()
-  endif()
-  if(distribution)
-    if(distribution STREQUAL "<DEFAULT>")
-      set(${export_arg_var} EXPORT ${project}${suffix} PARENT_SCOPE)
-      set_property(GLOBAL PROPERTY ${project_upper}_HAS_EXPORTS True)
-    else()
-      set(${export_arg_var} EXPORT ${project}${distribution}${suffix} PARENT_SCOPE)
+  if(in_distribution)
+    if(distribution)
       string(TOUPPER "${distribution}" distribution_upper)
       set_property(GLOBAL PROPERTY ${project_upper}_${distribution_upper}_HAS_EXPORTS True)
+    else()
+      set_property(GLOBAL PROPERTY ${project_upper}_HAS_EXPORTS True)
     endif()
   else()
     set(${export_arg_var} "" PARENT_SCOPE)
   endif()
 endfunction()
 
 # Produce a string of CMake include() commands to include the exported targets
 # files for all distributions. See the comment at the top of this file for
 # various assumptions made.