This is an archive of the discontinued LLVM Phabricator instance.

[build][NFC] limit parallel link to reduce memory usage
Needs ReviewPublic

Authored by Kokan on Jan 8 2020, 8:56 AM.

Details

Reviewers
rnk
Group Reviewers
Restricted Project
Summary

Measurerament I took with the following configuration:

  • 32 GB memory
  • 16 thread
  • compiler clang/clang++ (version: 9.0.1)

    With ld as a linker (version: 2.33.1) ====

-DLLVM_PARALLEL_LINK_JOBS=any => out of memory
-DLLVM_PARALLEL_LINK_JOBS=5 => no more than 30 GB memory
-DLLVM_PARALLEL_LINK_JOBS=2 => no more than 14 GB memory
-DLLVM_PARALLEL_LINK_JOBS=1 => no more than 10 GB memory

With lld as a linker (version: 2.33.1) => -DLLVM_USE_LINKER=lld
====

-DLLVM_PARALLEL_LINK_JOBS=any => no more then 9 GB memory
-DLLVM_PARALLEL_LINK_JOBS=2 => no more than 6 GB memory

The current defaults for LLVM_PARALLEL_LINK_JOBS is empty, meaning any number
(only limited by ninja/make parallel option).

The LLVM_PARALLEL_LINK_JOBS=2 is a better default option,
if the linker is not lld.

Signed-off-by: Kokai Peter <kokaipeter@gmail.com>

Diff Detail

Event Timeline

Kokan created this revision.Jan 8 2020, 8:56 AM
Herald added a project: Restricted Project. · View Herald TranscriptJan 8 2020, 8:56 AM
dblaikie edited reviewers, added: rnk; removed: chandlerc.Jan 8 2020, 9:00 AM
dblaikie added a subscriber: dblaikie.

Could we do this the other way around - and "AND LINKER_IS_GNULD"? Because so far as I know gold has reasonable memory usage too (& LD64), etc. I /think/ it's just GNULD that's the outlier here. (I can't speak for SOLARISLD, admittedly)

SouraVX added a subscriber: SouraVX.Jan 9 2020, 3:18 AM
Kokan added a comment.Jan 9 2020, 10:03 AM

Could we do this the other way around - and "AND LINKER_IS_GNULD"? Because so far as I know gold has reasonable memory usage too (& LD64), etc. I /think/ it's just GNULD that's the outlier here. (I can't speak for SOLARISLD, admittedly)

This looks reasonable for me. The part that creates LINKER_IS_GNULD is implemented in include(AddLLVM), that is included much later then this HandleLLVMOptions. I will look into if that can be changed.