This is an archive of the discontinued LLVM Phabricator instance.

[ThinLTO] Option to invoke ThinLTO backend passes and importing
ClosedPublic

Authored by tejohnson on Nov 26 2015, 7:16 AM.

Details

Summary

Adds new option -fthinlto-backend=<file> to invoke the LTO pipeline
along with function importing via clang using the supplied function
summary index file. This supports invoking the parallel ThinLTO
backend processes in a distributed build environment via clang.

Additionally, this causes the module linker to be invoked on the bitcode
file being compiled to perform any necessary promotion and renaming of
locals that are exported via the function summary index file.

Add a couple tests that confirm we get expected errors when we try to
use the new option on a file that isn't bitcode, or specify an invalid
index file. The tests also confirm that we trigger the expected function
import pass.

Depends on D15024

Diff Detail

Event Timeline

tejohnson updated this revision to Diff 41251.Nov 26 2015, 7:16 AM
tejohnson retitled this revision from to [ThinLTO] Option to invoke ThinLTO backend passes and importing.
tejohnson updated this object.
tejohnson added reviewers: mehdi_amini, dexonsmith.
tejohnson added subscribers: cfe-commits, davidxl.
mehdi_amini added inline comments.Nov 26 2015, 10:14 AM
lib/CodeGen/BackendUtil.cpp
309

It does not seem to be nicely integrated within the context of this function. What about all the options set just a few line below? It is not clear if CodeGenOpts.DisableLLVMOpts is well honored either.

lib/CodeGen/CodeGenAction.cpp
31

Not well sorted :)

190

Is this an unrelated change to the -fthinlto-backend one that could be committed separately?

821

This lambda is the same as the one just above for getFunctionIndexForFile, name it and define it once?

826

So for the renaming we need to duplicate completely the module? It cannot be done by morphing the existing module in place? That seems quite inefficient :(

Thanks for the review, comments below.
Teresa

lib/CodeGen/CodeGenAction.cpp
190

It is related in that I am now calling linkerDiagnosticHandler from outside BackendConsumer, which required moving it out of that class, and therefore we need to pass in the DiagnosticsEngine.

821

Ok

826

For now that is how the ModuleLinker works unfortunately. Rafael is working on a change to split this into two parts, and the symbol resolution would be done in place. So this should be resolved eventually.

tejohnson marked an inline comment as done.Dec 1 2015, 11:12 AM

Updated patch coming shortly.

lib/CodeGen/BackendUtil.cpp
309

This is basically the same LTO pipeline we setup in both gold-plugin.cpp and in LTOCodeGenerator::optimize (used by the llvm-lto tool and presumably ld64). I added that info to the comments. It is a special pipeline for this mode. That being said, I have made a couple of changes in my latest version to be uploaded shortly: Skip this special pipeline if DisableLLVMOpts is true; set LoopVectorize to CodeGenOpts.VectorizeLoop as in the below code; set SLPVectorize to CodeGenOpts.VectorizeSLP as in the below code.

One thing I may do first actually is to refactor this code out of gold-plugin and LTOCodeGenerator into a PassManagerBuilder helper function.

tejohnson updated this revision to Diff 41542.Dec 1 2015, 11:25 AM
  • Address Mehdi's comments.
mehdi_amini added inline comments.Dec 1 2015, 5:51 PM
lib/CodeGen/BackendUtil.cpp
291

The !CodeGenOpts.DisableLLVMOpts part is unclear to me. This may lead to unexpected user experience?

309

This duplication of PMBuilder / CodeGenOpts setup is what triggered my initial comment.

323–334

Could you move your code here and reuse the same PBBuilder initialized here?
I expect that you don't need to do (almost) anything else than:

if (!CodeGenOpts.ThinLTOIndexFile.empty()) {
  legacy::PassManager *MPM = getPerModulePasses();
  PMB.populateLTOPassManager(MPM);
  return;
}
398–399

Could we move the inlining setup before your code so that you don't need to it?

398–399

Could this be moved earlier so that it is before your code and you don't need to duplicate it?

lib/CodeGen/CodeGenAction.cpp
791

I think I didn't express what I meant by "name it", it can still be a lambda:

auto DiagHandler = [&] (const DiagnosticInfo &DI) {
   linkerDiagnosticHandler(DI, TheModule.get(),
                        getCompilerInstance().getDiagnostics());
};

(there not real reason to use std::bind() for anything now that we have lambda I think)

819

Side note: I found terrible that we can't do the renaming/linkage upgrade in-place :(

tejohnson marked 4 inline comments as done.Dec 3 2015, 12:11 PM

New patch coming I think addresses all your comments.
Thanks, Teresa

lib/CodeGen/BackendUtil.cpp
291

I'm not sure what the effect of DisableLLVMOpts should be on importing. With the changes to the latest patch I will upload shortly that moves this down and uses the same PM setup as the below code, we will still perform importing, but the optimization level is turned down to 0. That is probably reasonable, I think.

lib/CodeGen/CodeGenAction.cpp
819

This is an unfortunate effect of the currently module linker code, assuming that I want to use the same renaming/promotion support. I suppose I can add some support for doing this in place on the module, it was convenient to just reuse the support I put in the module linker initially.

tejohnson updated this revision to Diff 41787.Dec 3 2015, 12:12 PM
  • Address more comments.
mehdi_amini edited edge metadata.Dec 3 2015, 12:18 PM

Looks good to me but I'd like another opinion on the name of "-fthinlto-backend", Duncan?

include/clang/Driver/Types.h
68

I guess you could already commit this separately

lib/CodeGen/CodeGenAction.cpp
819

I know, we can improve it later, I just wanted to share this so we're on the same page on the ultimate goal :)

Talked with Duncan, we're not convinced about -fthinlto-backend=... for the option name: the word backend does not seems right here. What about -fthinlto-index=...?

lib/CodeGen/CodeGenAction.cpp
822

Could we refactor this in a helper in llvm?

test/Driver/thinlto_backend.c
6

Why not pipe to FileCheck here?

Talked with Duncan, we're not convinced about -fthinlto-backend=... for the option name: the word backend does not seems right here. What about -fthinlto-index=...?

I wanted to express that this was invoking a different pipeline, and distinguish from a normal compilation that takes a source file. But I'm ok with your suggestion if you prefer that.

lib/CodeGen/CodeGenAction.cpp
822

I can create a new Linker interface that takes the index path, builds the index, does the link and returns the ErrorOr index.

test/Driver/thinlto_backend.c
6

Will do. I think I cloned this from another test.

mehdi_amini added inline comments.Dec 4 2015, 8:15 AM
lib/CodeGen/CodeGenAction.cpp
822

I was thinking about a free function that would be something like: std::unique_ptr<Module> llvm::renameModuleForThinLTO(std::unique_ptr<Module> &M, const FunctionInfoIndex &Index);

tejohnson added inline comments.Dec 4 2015, 8:59 AM
lib/CodeGen/CodeGenAction.cpp
822

We also need to get the Index back so that it can be passed down to EmitBackendOutput below (to get it on the pass manager builder). So what I was thinking was create the Combined Module unique_ptr here and pass it in (as we do currently for LinkModules), and return the std::unique_ptr<FunctionInfoIndex> .

mehdi_amini added inline comments.Dec 4 2015, 9:03 AM
lib/CodeGen/CodeGenAction.cpp
822

You pass a reference to the index, you already have it.

tejohnson added inline comments.Dec 4 2015, 9:14 AM
lib/CodeGen/CodeGenAction.cpp
822

Oh I see, you are just talking about the Combined module creation and LinkModules call.

I was thinking you meant refactor out the getFunctionIndexForFile + LinkModules. Just doing the module creation and LinkModules is easier, will do that.

tejohnson updated this revision to Diff 41926.Dec 4 2015, 12:29 PM
tejohnson edited edge metadata.
  • Use new renameModuleForThinLTO helper function from llvm.
  • Fix tests

    I have not change the option name yet. Wanted to be sure we have a name that reflects the fact that this is a special compile that changes the compilation pipeline and is only used for the last part of a ThinLTO build in a distributed build situation. Let me know if you feel -fthinlto-index is sufficient for this, or if there is a better alternative to my current -fthinlto-backend.
tejohnson updated this revision to Diff 41935.Dec 4 2015, 1:41 PM
  • Move ownership of FunctionInfoIndex onto EmitAssemblyHelper
tejohnson updated this revision to Diff 42069.Dec 7 2015, 7:41 AM
  • Rename -fthinlto-backend to -fthinlto-index as per review and IRC.
mehdi_amini accepted this revision.Dec 7 2015, 8:25 AM
mehdi_amini edited edge metadata.

LGTM, thanks!

This revision is now accepted and ready to land.Dec 7 2015, 8:25 AM
This revision was automatically updated to reflect the committed changes.