This is an archive of the discontinued LLVM Phabricator instance.

Driver: generate file lists to pass through to linkers
AbandonedPublic

Authored by abdulras on Jun 11 2014, 2:40 PM.

Details

Summary

Add support for linker file lists. A file list is similar to linker response
files, but slightly more restricted. A file list may only contain a list of
input files. Flags must remain on the actual command invocation. Runs of files
are collapsed into a single input which is marked as a file list by prepending
the filename with an '@' symbol. The linker will read this file and expand the
contents as the input items.

This is supported by at least the GNU linker as well as Microsoft's link.exe
linker. The use of the option is controlled through a driver flag, defaulting
to disabled.

This functionality aids in reducing the length of the subcommand invoked to
perform the final link. Some systems have a limit on the command line, and when
linking a large number of object files, it is possible to overrun that limit.

Diff Detail

Event Timeline

abdulras updated this revision to Diff 10334.Jun 11 2014, 2:40 PM
abdulras retitled this revision from to Driver: add support for linker file lists.
abdulras updated this object.
abdulras edited the test plan for this revision. (Show Details)
abdulras added a reviewer: rui314.
abdulras set the repository for this revision to rL LLVM.
abdulras added subscribers: Unknown Object (MLST), ruiu, compnerd.
ruiu edited reviewers, added: ruiu; removed: rui314.Jun 11 2014, 2:54 PM
ruiu removed a subscriber: ruiu.
ruiu resigned from this revision.Jun 11 2014, 3:32 PM
ruiu edited reviewers, added: hans; removed: ruiu.

I first thought that GNU and Microsoft linker support a feature called "linker file lists", but they are actually not -- this patch is just to pass input file lists to subcommand using a response file. Is this understanding correct?

I think I don't have an ownership in this area. Hans was workikng on Clang driver, so reassigning to him.

rnk added a subscriber: rnk.Jun 11 2014, 4:01 PM

This is a general problem: any subcommand we execute may overflow the system command line length limit. I'd rather solve it transparently for -cc1 as well as linker invocations by exposing a command line length estimate from Support and then estimating our command line length in the driver before invoking the subcommand, optionally with a response file. I know that Sony has an internal patch to do this for -cc1 invocations, so it's not just linker stuff.

abdulras updated this revision to Diff 10366.Jun 12 2014, 9:40 AM
abdulras retitled this revision from Driver: add support for linker file lists to Driver: generate file lists to pass through to linkers.

Make getLinkerFileListName more similar to getDependencyFileName. Add an assertion. Tweak some variable names as per review comments.

rnk added a comment.Jun 18 2014, 1:39 PM

Broadly, I'd like to rework this to be more generic. We should have some mechanism in Support to get the max command line length. The driver should have some mechanism for estimating if the command line will overflow that. The Tool should query that, and if it is likely to overflow, go back and put parts of the command into filelists or generic response files. If you really need to support a linker that only supports file lists, then we'll have to change the representation of the command line to be a list of strings that know if they are inputs or flags. That seems OK.

lib/Driver/Tools.cpp
180

Use llvm::raw_fd_ostream. It has less moving parts.

213

This won't be correct for many linkers if the filename contains whitespace. I think we need to know what kind of linker we're calling out to in order to get this right. It's easy for link.exe and gnu ld, since we know exactly how they quote and unquote arguments. See the response file parsing code in llvm/lib/Support/CommandLine.cpp.

abdulras abandoned this revision.Oct 21 2014, 9:26 AM

Seems that an alternative implementation has been committed.