[build][NFC] limit parallel link to reduce memory usage

[Kokan]

Summary

Measurerament I took with the following configuration:

• 32 GB memory
• 16 thread
• compiler clang/clang++ (version: 9.0.1)
  
  With ld as a linker (version: 2.33.1) ====

-DLLVM_PARALLEL_LINK_JOBS=any => out of memory
-DLLVM_PARALLEL_LINK_JOBS=5 => no more than 30 GB memory
-DLLVM_PARALLEL_LINK_JOBS=2 => no more than 14 GB memory
-DLLVM_PARALLEL_LINK_JOBS=1 => no more than 10 GB memory

With lld as a linker (version: 2.33.1) => -DLLVM_USE_LINKER=lld
====

-DLLVM_PARALLEL_LINK_JOBS=any => no more then 9 GB memory
-DLLVM_PARALLEL_LINK_JOBS=2 => no more than 6 GB memory

The current defaults for LLVM_PARALLEL_LINK_JOBS is empty, meaning any number
(only limited by ninja/make parallel option).

The LLVM_PARALLEL_LINK_JOBS=2 is a better default option,
if the linker is not lld.

Signed-off-by: Kokai Peter <[e-mail]>

[dblaikie]

Could we do this the other way around - and "AND LINKER_IS_GNULD"? Because so far as I know gold has reasonable memory usage too (& LD64), etc. I /think/ it's just GNULD that's the outlier here. (I can't speak for SOLARISLD, admittedly)

[Kokan]

In D72402#1810269, @dblaikie wrote:

Could we do this the other way around - and "AND LINKER_IS_GNULD"? Because so far as I know gold has reasonable memory usage too (& LD64), etc. I /think/ it's just GNULD that's the outlier here. (I can't speak for SOLARISLD, admittedly)

This looks reasonable for me. The part that creates LINKER_IS_GNULD is implemented in include(AddLLVM), that is included much later then this HandleLLVMOptions. I will look into if that can be changed.

llvm/cmake/modules/HandleLLVMOptions.cmake

 
 set(LLVM_PARALLEL_LINK_JOBS "" CACHE STRING
   "Define the maximum number of concurrent link jobs (Ninja only).")
 if(CMAKE_MAKE_PROGRAM MATCHES "ninja")
   if(NOT LLVM_PARALLEL_LINK_JOBS AND uppercase_LLVM_ENABLE_LTO STREQUAL "THIN")
     message(STATUS "ThinLTO provides its own parallel linking - limiting parallel link jobs to 2.")
     set(LLVM_PARALLEL_LINK_JOBS "2")
   endif()
+  if (NOT LLVM_PARALLEL_LINK_JOBS AND NOT LINKER_IS_LLD_LINK)
+    set(LLVM_PARALLEL_LINK_JOBS "2")
+  endif()
   if(LLVM_PARALLEL_LINK_JOBS)
     set_property(GLOBAL APPEND PROPERTY JOB_POOLS link_job_pool=${LLVM_PARALLEL_LINK_JOBS})
     set(CMAKE_JOB_POOL_LINK link_job_pool)
   endif()
 elseif(LLVM_PARALLEL_LINK_JOBS)
   message(WARNING "Job pooling is only available with Ninja generators.")
 endif()