This is an archive of the discontinued LLVM Phabricator instance.

[ThinLTO] Option to control path of distributed backend files
ClosedPublic

Authored by tejohnson on Apr 27 2016, 9:16 PM.

Details

Summary

Add new thinlto-prefix-replace plugin option to control where files
for a distributed backend (the individual index files and optional
imports files) are created.

If specified, expects a string of the form "oldprefix:newprefix", and
instead of generating these files in the same directory path as the
corresponding bitcode file, will use a path formed by replacing the
bitcode file's path prefix matching oldprefix with newprefix.

Depends on D19636.

Diff Detail

Repository
rL LLVM

Event Timeline

tejohnson updated this revision to Diff 55374.Apr 27 2016, 9:16 PM
tejohnson retitled this revision from to [ThinLTO] Option to control path of distributed backend files.
tejohnson updated this object.
tejohnson added a reviewer: mehdi_amini.
tejohnson added a subscriber: llvm-commits.
mehdi_amini added inline comments.Apr 27 2016, 11:06 PM
test/tools/gold/X86/thinlto_prefix_replace.ll
4 ↗(On Diff #55374)

Deserve a comment

tools/gold/gold-plugin.cpp
198 ↗(On Diff #55374)

What is the motivation for that?

tejohnson added inline comments.Apr 28 2016, 9:46 AM
test/tools/gold/X86/thinlto_prefix_replace.ll
4 ↗(On Diff #55374)

Will add.

tools/gold/gold-plugin.cpp
198 ↗(On Diff #55374)

In our particular instance it helps enable sharing of bitcode files across builds of different libraries or binaries. I.e. the bitcode files, the result of the first compile step, are output into one tree. When the ThinLTO link and backend steps execute, we want the resulting index/imports files to go into a different tree specific to the target binary/library. That way the same bitcode tree can be used as input across builds for multiple libraries/binaries. But this generally seems like useful functionality to allow flexibility in the build system.

tejohnson updated this revision to Diff 55592.Apr 29 2016, 6:25 AM

Add comment to test. Rebase.

mehdi_amini edited edge metadata.May 4 2016, 2:08 PM

Can this be exposed to llvm-lto as well? It'll help testing when refactoring everything behind a new API.

test/tools/gold/X86/thinlto_prefix_replace.ll
13 ↗(On Diff #55592)

I think you can just write ls %T/newpath/thinlto_prefix_replace.o.thinlto.bc without any FileCheck, it should be enough to test the existence of the file.

In D19644#421629, @joker.eph wrote:

Can this be exposed to llvm-lto as well? It'll help testing when refactoring everything behind a new API.

I wasn't sure the best way to do that initially, but thinking some more I think this is what I can do: In D19556 (individual index files) I will sink the lifetime of the raw_fd_ostream into WriteIndexToFile (similar to how it is done within EmitImportsFiles in D19636). Then I can move getOutputModulePath into FunctionImport.cpp as well, and just pass the new and old prefix into those routines and do the replacement there before opening the files.

test/tools/gold/X86/thinlto_prefix_replace.ll
13 ↗(On Diff #55592)

Ok, will give that a try.

In D19644#421629, @joker.eph wrote:

Can this be exposed to llvm-lto as well? It'll help testing when refactoring everything behind a new API.

I wasn't sure the best way to do that initially, but thinking some more I think this is what I can do: In D19556 (individual index files) I will sink the lifetime of the raw_fd_ostream into WriteIndexToFile (similar to how it is done within EmitImportsFiles in D19636). Then I can move getOutputModulePath into FunctionImport.cpp as well, and just pass the new and old prefix into those routines and do the replacement there before opening the files.

I remembered why I didn't do this - what I describe above doesn't work since WriteIndexToFile is in the bitcode writer, and I don't want to add a dependence from it to the function importer or vice versa. It's difficult to sink the code here, which is just doing path and file operations, into the function importer since the result is used both by EmitImports (in the function importer) and for WriteIndexToFile (in the bitcode writer).

What I did instead was to extract the path manipulation done here to replace the matching path prefix with a new prefix into a new path method in libSupport (and added unit tests). I then invoke that both from gold-plugin as well as from llvm-lto. There is some code duplicated between gold-plugin and llvm-lto but it is now pretty minimal (just invoking some basic option parsing and the path support library routines). I added a companion llvm-lto test as well.

tejohnson updated this revision to Diff 56908.May 11 2016, 7:09 AM
tejohnson edited edge metadata.

Extract path manipulation into libSupport path helper, add unit tests. Add
similar support to llvm-lto along with test.

tejohnson updated this revision to Diff 56910.May 11 2016, 7:21 AM

Forgot to add new llvm-lto based test in previous update, add it here.

Ping.
Thanks,
Teresa

mehdi_amini accepted this revision.May 16 2016, 4:20 PM
mehdi_amini edited edge metadata.

LGTM, with some suggestions inline (no need to review again)

lib/Support/Path.cpp
533 ↗(On Diff #56910)

You mentioned as precondition in the doc:

 @param Path If \a Path starts with \a OldPrefix modify to instead
///        start with \a NewPrefix.

Here you're still doing the check. I'd either assert or update the comment.

539 ↗(On Diff #56910)

If OldPrefix.size() == NewPrefix.size() we have a possible fast-path in-place.
(the unittest should also be updated to test this case)

tools/gold/gold-plugin.cpp
1217 ↗(On Diff #56910)

This is a user-supplied command line flag, if you assert he I'd expect it to be appropriately validated with a report_fatal_error (or equivalent) at parse time (line 226 above)

This revision is now accepted and ready to land.May 16 2016, 4:20 PM
mehdi_amini added inline comments.May 16 2016, 4:21 PM
tools/llvm-lto/llvm-lto.cpp
331 ↗(On Diff #56910)

The duplication with the gold plugin isn't great for these two functions.

tejohnson added inline comments.May 16 2016, 9:47 PM
lib/Support/Path.cpp
533 ↗(On Diff #56910)

I think the code matches the description, which indicates that we only do the replacement if Path starts with OldPrefix. Here we are skipping the replacement if Path does not start with OldPrefix.

539 ↗(On Diff #56910)

I can do that.

tools/gold/gold-plugin.cpp
1217 ↗(On Diff #56910)

Ok

tools/llvm-lto/llvm-lto.cpp
331 ↗(On Diff #56910)

Agree, but I couldn't figure out a good way to avoid that. They didn't seem general enough to move into lib/Support either.

mehdi_amini added inline comments.May 16 2016, 9:54 PM
lib/Support/Path.cpp
533 ↗(On Diff #56910)

Somehow I missed the "If" in the comment when I copy/pasted... Nevermind!

tools/llvm-lto/llvm-lto.cpp
331 ↗(On Diff #56910)

It is not critical if there was no convenient way.

This revision was automatically updated to reflect the committed changes.