Page MenuHomePhabricator

[ThinLTO] During module importing, close one source module before open another one for distributed mode.
ClosedPublic

Authored by wmi on Mar 29 2021, 5:13 PM.

Details

Summary

Currently during module importing, ThinLTO opens all the source modules, collect functions to be imported and append them to the destination module, then leave all the modules open through out the lto backend pipeline. This patch refactors it in the way that one source module will be closed before another source module is opened. All the source modules will be closed after importing phase is done. It will save some amount of memory when there are many source modules to be imported.

Note that this patch only changes the distributed thinlto mode. For in process thinlto mode, one source module is shared acorss different thinlto backend threads so it is not changed in this patch.

Diff Detail

Event Timeline

wmi created this revision.Mar 29 2021, 5:13 PM
wmi requested review of this revision.Mar 29 2021, 5:13 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptMar 29 2021, 5:13 PM
Herald added a subscriber: cfe-commits. · View Herald Transcript
tejohnson accepted this revision.Mar 30 2021, 10:28 AM

LGTM with a couple of comment requests noted below.

clang/lib/CodeGen/BackendUtil.cpp
1583

Document constant parameter

llvm/include/llvm/LTO/LTOBackend.h
48–53

Can you add a note here about the behavior with regards to ModuleMap depending on whether it is null or not?

This revision is now accepted and ready to land.Mar 30 2021, 10:28 AM
wmi updated this revision to Diff 334240.Mar 30 2021, 11:37 AM

Address Teresa's comment.

clang/lib/CodeGen/BackendUtil.cpp
1583

Done.

llvm/include/llvm/LTO/LTOBackend.h
48–53

Add the comment.

Does this help non-distributed ThinLTO as well? cc: @weiwang

Does this help non-distributed ThinLTO as well? cc: @weiwang

No. See the note in Wei's description about this:

Note that this patch only changes the distributed thinlto mode. For in process thinlto mode, one source module is shared across different thinlto backend threads so it is not changed in this patch.

Specifically, for in-process ThinLTO all modules are kept open so that we don't need to reload a module for each thread that wants to import from it.