This is an archive of the discontinued LLVM Phabricator instance.

CodeGen: Introduce LinkedCodeGen and teach LTOCodeGenerator to use it.
ClosedPublic

Authored by pcc on Aug 21 2015, 7:31 PM.

Details

Summary

llvm::LinkedCodeGen is a function that implements the core of parallel LTO
code generation. It uses llvm::SplitModule to split the module into linkable
partitions and spawning one code generation thread per partition. The function
produces multiple object files which can be linked in the usual way.

This has been threaded through to LTOCodeGenerator (and llvm-lto for testing
purposes). Separate patches will add parallel LTO support to the gold plugin
and lld.

Diff Detail

Event Timeline

pcc updated this revision to Diff 32894.Aug 21 2015, 7:31 PM
pcc retitled this revision from to CodeGen: Introduce LinkedCodeGen and teach LTOCodeGenerator to use it..
pcc updated this object.
pcc added reviewers: mehdi_amini, dexonsmith, chandlerc.
pcc added a subscriber: llvm-commits.
pcc updated this revision to Diff 33017.Aug 24 2015, 3:59 PM

Rebase

pcc added inline comments.Aug 24 2015, 11:08 PM
lib/CodeGen/Parallel.cpp
43 ↗(On Diff #33017)

Although this API takes an array of stream pointers, all current clients (llvm-lto, gold-plugin D12308, lld D12309) own their streams, making this API a little awkward to use. Probably a better choice of argument type here would be ArrayRef<std::unique_ptr<raw_pwrite_stream>>. This would mean a few extra allocations, but the cost would be dwarfed by that of LTO.

pcc updated this revision to Diff 33150.Aug 25 2015, 4:14 PM
  • Fix Windows build
mehdi_amini edited edge metadata.Aug 26 2015, 8:17 AM

A few comments.

lib/CodeGen/Parallel.cpp
70 ↗(On Diff #33150)

Add a comment? Something like: we need to do this because we want to clone the module in a new context to multi-thread the codegen. The "dumping()" part has to be performed in the caller thread and not the one we create just after because we are still in the old LLVMContext

It seems LLVM is missing the ability to clone a module in a new context directly...

72 ↗(On Diff #33150)

What is copied? Might replace the = with an explicit list if possible. I'm not sure why you want to copy "Options" for instance? Also, some values are passed as argument to the function, the cut is not clear to me.

lib/LTO/LTOCodeGenerator.cpp
504

rename, not longer involved in CodeGen

tools/llvm-lto/llvm-lto.cpp
272

Tools are usually using tool_output_file, is it appropriate here?

279

Why? I mean any particular reason or just for convenience of the implementation?

pcc updated this revision to Diff 33359.Aug 27 2015, 2:57 PM
pcc edited edge metadata.
  • Address review comments
lib/CodeGen/Parallel.cpp
44 ↗(On Diff #33150)

This doesn't seem like it will work, as we do have a client that does not own the stream (LTOCodeGenerator::compileOptimizedToFile, and now llvm-lto). Let's keep the existing API for now.

70 ↗(On Diff #33150)

Done

72 ↗(On Diff #33150)

This is now a list of copied things.

It seems that we don't need to copy Options, but this is a micro-optimization not worth thinking about, so I'd rather just copy it.

lib/LTO/LTOCodeGenerator.cpp
504

Done

tools/llvm-lto/llvm-lto.cpp
272

Yes, I suppose so -- we might want to clean up if one of the files cannot be opened. Done.

279

The latter. Since we wouldn't be able to gain any extra test coverage by supporting this case, we don't need to.

pcc updated this revision to Diff 33360.Aug 27 2015, 2:59 PM
  • Update file headers
pcc updated this revision to Diff 33361.Aug 27 2015, 3:01 PM
  • Update guard
mehdi_amini accepted this revision.Aug 27 2015, 3:50 PM
mehdi_amini edited edge metadata.

LGTM. One inline comment.

lib/CodeGen/ParallelCG.cpp
87

No move needed here since it is a const ref right?

This revision is now accepted and ready to land.Aug 27 2015, 3:50 PM
This revision was automatically updated to reflect the committed changes.
pcc added inline comments.Aug 27 2015, 4:38 PM
lib/CodeGen/ParallelCG.cpp
87

The std::move here is needed for the std::thread constructor to move (rather than copy) BC into thread-accessible storage (the const ref is taken from that storage). Added a comment clarifying.