This is an archive of the discontinued LLVM Phabricator instance.

[ThinLTO] Add option to emit imports files for distributed backends
ClosedPublic

Authored by tejohnson on Apr 27 2016, 3:42 PM.

Details

Summary

Add a new thinlto-emit-imports-files plugin option to enable emission of
plaintext lists of the imported files for each distributed backend compilation.
Used for distributed build file staging. This option is only valid with
thinlto-index-only (i.e. for distributed builds).

Depends on D19556.

Diff Detail

Event Timeline

tejohnson updated this revision to Diff 55342.Apr 27 2016, 3:42 PM
tejohnson retitled this revision from to [ThinLTO] Add option to emit imports files for distributed backends.
tejohnson updated this object.
tejohnson added a reviewer: mehdi_amini.
tejohnson added a subscriber: llvm-commits.
mehdi_amini edited edge metadata.Apr 27 2016, 11:03 PM

It'd be nice to have this mirrored in llvm-lto.

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

In D19636#415046, @joker.eph wrote:

It'd be nice to have this mirrored in llvm-lto.

I will do that and add a test.

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

I could move this particular code into a helper in FunctionImport.cpp that takes in the ImportsLists and a file name as input, and do the file I/O there.
(Unrelated to this patch, but to the above comment: as noted in my response to D19556 this morning, I will pull the preceding code to compute the ModuleToSummariesForIndexMap into a helper in FunctionImport.cpp.)

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

I could move this particular code into a helper in FunctionImport.cpp that takes in the ImportsLists and a file name as input, and do the file I/O there.
(Unrelated to this patch, but to the above comment: as noted in my response to D19556 this morning, I will pull the preceding code to compute the ModuleToSummariesForIndexMap into a helper in FunctionImport.cpp.)

Yes this is not about this patch in particular, this is about this plugin file that drifted over time. Its scope should be very narrowed to the interaction with gold IMO. All other logic should be refactored in some LLVM libraries.

tejohnson updated this revision to Diff 55528.Apr 28 2016, 9:57 PM
tejohnson edited edge metadata.
  • Address comments: refactor into a helper and add libLTO and llvm-lto support and tests

Oh sorry I missed the update to the patch, will look this morning (hopefully)

In D19636#421031, @joker.eph wrote:

Oh sorry I missed the update to the patch, will look this morning (hopefully)

Thanks, actually I meant to ping the one this is dependent on first (D19556 for emitting individual backend index files), but they were both updated last week (along with D19644) and are ready for review.

mehdi_amini accepted this revision.May 4 2016, 2:04 PM
mehdi_amini edited edge metadata.

LGTM, with two remarks.

include/llvm/LTO/ThinLTOCodeGenerator.h
198 ↗(On Diff #55528)

for *each* module? I'd say for \p ModulePath instead.

201 ↗(On Diff #55528)

Same remark as the previous path: this does not really use anything specific to the ThinLTOCodeGenerator.

This revision is now accepted and ready to land.May 4 2016, 2:04 PM

thanks

include/llvm/LTO/ThinLTOCodeGenerator.h
198 ↗(On Diff #55528)

Yes the old comment wasn't really correct, will fix

201 ↗(On Diff #55528)

Will change this to a static method too.

This patch is not yet committed. However, another patch I committed this morning (D19556) is the cause of this. I'm working on tracking it down (unfortunately, it isn't obvious since we shouldn't even execute the routine in question and don't with a non-msan compiler. Trying to build an msan compiler right now to track down further). Should I revert in the meantime?

My mistake, it is D19556. I searched by sub-string in the title.
It's better to revert if a fix is not obvious.

This revision was automatically updated to reflect the committed changes.