Page MenuHomePhabricator

[clang][driver] Set LTO mode based on input files
Needs RevisionPublic

Authored by tbaeder on Fri, Oct 30, 5:48 AM.

Details

Summary

After setting the LTO mode from the -flto option, look at the input files as well. If any of them is an object file containing LLVM bitcode,
set the LTO mode to either thin or full, depending on the input file.

This makes the following sample work:

clang test.c -c -flto
clang test.o

Which also works when using GCC. Currently this makes non-lld linkers print an error, for example when using ld.bfd:

$ clang test.o
test.o: file not recognized: file format not recognized
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

Diff Detail

Unit TestsFailed

TimeTest
360 mslinux > HWAddressSanitizer-x86_64.TestCases::sizes.cpp
Script: -- : 'RUN: at line 3'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -m64 -gline-tables-only -fsanitize=hwaddress -fuse-ld=lld -mcmodel=large -mllvm -hwasan-globals -mllvm -hwasan-use-short-granules -mllvm -hwasan-instrument-landing-pads=0 -mllvm -hwasan-instrument-personality-functions /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp -nostdlib++ -lstdc++ -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp
480 mswindows > Clang.Driver::thinlto_backend.c
Script: -- : 'RUN: at line 1'; c:\ws\w1\llvm-project\premerge-checks\build\bin\clang.exe -O2 C:\ws\w1\llvm-project\premerge-checks\clang\test\Driver\thinlto_backend.c -flto=thin -c -o C:\ws\w1\llvm-project\premerge-checks\build\tools\clang\test\Driver\Output\thinlto_backend.c.tmp.o
290 mswindows > lld.ELF/invalid::symtab-sh-info.s
Script: -- : 'RUN: at line 4'; c:\ws\w1\llvm-project\premerge-checks\build\bin\yaml2obj.exe --docnum=1 C:\ws\w1\llvm-project\premerge-checks\lld\test\ELF\invalid\symtab-sh-info.s -o C:\ws\w1\llvm-project\premerge-checks\build\tools\lld\test\ELF\invalid\Output\symtab-sh-info.s.tmp.o

Event Timeline

tbaeder created this revision.Fri, Oct 30, 5:48 AM
tbaeder requested review of this revision.Fri, Oct 30, 5:48 AM
MaskRay added a comment.EditedWed, Nov 11, 11:56 PM

Is the motivation just to avoid -flto or -flto=lto at link time? I am afraid that the advantage probably is not large enough to justify the potentially costly object file parsing in the driver. Before this, as I understand it, object files are opaque to the driver. The driver just passes all file names to the linker. With this change, every object file will be opened and llvm::getBitcodeFileContents will be called on every object file.

Other thoughts include:

  • An LTO link can mix regular and thin LTO bitcode files. I am not very familiar with the thinlto internals so I don't know whether it is appropriate to stop after we immediately see a full LTO bitcode file.
  • This provides a convenience method in the most simplest link (no other LTO related option). If you specify any other LTO related option (e.g --thinlto-index-only, --thinlto-cache-dir), the save of a -flto option appears to give too small of a value with an extra parsing.
  • clangDriver's dependency on LLVMBitReader is a bit odd.

In any case, it is just my opinion that this may not fit well as a driver feature. Hope Teresa can share more authoritative thoughts on this.

clang/lib/Driver/Driver.cpp
1205

On line 1133, there is a similar setLTOMode. If we are going to add the logic, probably consider unifying the logic. However, I am concerned with the cost, see my main comment

MaskRay requested changes to this revision.Wed, Nov 18, 5:52 PM
This revision now requires changes to proceed.Wed, Nov 18, 5:52 PM

Is the motivation just to avoid -flto or -flto=lto at link time?

Essentially, yes. Lots of projects seem to depend on this behavior of GCC. I.e. the pass -flto when compiling but not when linking.

I am afraid that the advantage probably is not large enough to justify the potentially costly object file parsing in the driver. Before this, as I understand it, object files are opaque to the driver. The driver just passes all file names to the linker. With this change, every object file will be opened and llvm::getBitcodeFileContents will be called on every object file.

I thought about the potential performance penalties as well. All of this only matters if -flto is really missing of course, otherwise the driver is not going to do any more work than it did before.

Other thoughts include:

  • An LTO link can mix regular and thin LTO bitcode files. I am not very familiar with the thinlto internals so I don't know whether it is appropriate to stop after we immediately see a full LTO bitcode file.
  • This provides a convenience method in the most simplest link (no other LTO related option). If you specify any other LTO related option (e.g --thinlto-index-only, --thinlto-cache-dir), the save of a -flto option appears to give too small of a value with an extra parsing.

Right, the purpose of this is not to save typing a "-flto", but to make linking in the simplest cast just work.

clang/lib/Driver/Driver.cpp
1205

Setting the LTO mode from the input files requires the Inputs list, but calling the existing setLTOMode() this late causes problems.