This is an archive of the discontinued LLVM Phabricator instance.

[Driver][OpenMP][CUDA] Add capability to bundle object files in sections of the host binary format.
ClosedPublic

Authored by sfantao on Jun 29 2016, 11:28 AM.

Details

Summary

This patch adds the capability to bundle object files in sections of the host binary using a designated naming convention for these sections. This patch uses the functionality of the object reader already in the LLVM library to read bundled files, and invokes clang with the incremental linking options to create bundle files.

Bundling files involves creating an IR file with the contents of the bundle assigned as initializers of globals binded to the designated sections. This way the bundling implementation is agnostic of the host object format.

The features added by this patch were requested in the RFC discussion in http://lists.llvm.org/pipermail/cfe-dev/2016-February/047547.html.

Diff Detail

Event Timeline

sfantao updated this revision to Diff 62255.Jun 29 2016, 11:28 AM
sfantao retitled this revision from to [Driver][OpenMP][CUDA] Add capability to bundle object files in sections of the host binary format..
sfantao updated this object.
sfantao added reviewers: echristo, tra, jlebar, hfinkel, ABataev.
ABataev added inline comments.Jun 29 2016, 8:48 PM
tools/clang-offload-bundler/ClangOffloadBundler.cpp
337–347

3 slashes, please

348

'final'?

350

No '\brief's

358

'const'? or 'static'?

463

Not sure that this a good solution to crash the compiler in this case. I think it must exit gracefully.

481

Again, maybe just emit an error an exit?

sfantao updated this revision to Diff 62550.Jul 1 2016, 3:20 PM
sfantao marked 6 inline comments as done.
  • Remove \brief and return errors instead of crashing when temporary tiles can't be created.

Hi Alexey,

Thanks for the review!

tools/clang-offload-bundler/ClangOffloadBundler.cpp
358

It can be static.

463

Ok, replaced the unreachable code with logic to emit an error and exist the tool.

481

Emitting an error now.

sfantao updated this revision to Diff 62584.Jul 1 2016, 5:20 PM
  • Start staic function name with caps.
Hahnfeld added inline comments.Jul 7 2016, 2:37 AM
tools/clang-offload-bundler/ClangOffloadBundler.cpp
724–737

test/Driver/clang-offload-bundler.c gives me

/..//bin/ld: unrecognised emulation mode: elf64lppc
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 i386linux elf_l1om elf_k1om

and therefore fails.

I'm on an x86_64 Linux and obviously my GNU ld version 2.23.52.0.1-55.el7 20130226 doesn't support Power :-(

Hi Jonas,

Thanks for trying that out!

tools/clang-offload-bundler/ClangOffloadBundler.cpp
724–737

Oh, right... I cannot run the bundler in the regression tests to tests the bundler. I guess I need some sort of dry run option to check the commands are correct without actually running them.

I'll fix that.

Another overall question: Back in February you said that it would be possible to have a "default" object that can be taken without knowledge of the bundler (see http://lists.llvm.org/pipermail/cfe-dev/2016-February/047555.html).

In my tests with the patch, this has not worked yet - am I missing something or is this not yet implemented?

tools/clang-offload-bundler/ClangOffloadBundler.cpp
724–737

Yes, comparable to -### in clang?

Another option (that I don't really prefer, just for completeness) would be to have separate tests that have appropriate REQUIRES...

Another overall question: Back in February you said that it would be possible to have a "default" object that can be taken without knowledge of the bundler (see http://lists.llvm.org/pipermail/cfe-dev/2016-February/047555.html).

In my tests with the patch, this has not worked yet - am I missing something or is this not yet implemented?

What I have proposed in the patch should be able to produce a bundled object file that can be read by other tools (the device image is embedded in designated sections, so these other tools may just ignore these sections). Also, if you feed the bundled an object file that does not have device sections, it should produce empty device files instead of just failing. That's what I was proposing there. Did you have something different in mind?

That's actually tested in the regression tests that goes with the patch. It won't work for you as is because of the ppc64le specific problem that I have to fix, but if you change that to x86_64 you may be able to test it.

tools/clang-offload-bundler/ClangOffloadBundler.cpp
724–737

Yes, -### is exactly what I was aiming at.

sfantao marked 4 inline comments as done.Jul 8 2016, 5:21 PM

That was probably not the reason why you were getting the failure, but the bundler was not prepared to bundle files if the host was not the first input. I fixed that.

I also replaced the test that was running the linker by something that checks the command and temporary files using -#. However, I'm still producing temporary files with -#. This problem is similar to the linker script test in http://reviews.llvm.org/D21847. Once we figure out how to tests the linker script I'll replicate the solution here too.

sfantao updated this revision to Diff 63361.Jul 8 2016, 5:23 PM
  • Allow host object to come in any order when generating the binary. Add testing for that.
  • Add save temps and command print option.
  • Add bundled object for testing.

That was probably not the reason why you were getting the failure, but the bundler was not prepared to bundle files if the host was not the first input. I fixed that.

This was indeed the problem, this now looks better and has the host object as default (though I don't yet understand why the order in -targets= is different for me...).

Thanks,
Jonas

mkuron added a subscriber: mkuron.Jul 26 2016, 5:27 AM
sfantao updated this revision to Diff 66025.Jul 28 2016, 2:51 PM
  • Refactor code to dump contents of temporary file instead of creating the actual file in dry-run mode.
  • Rebase.
Hahnfeld accepted this revision.Aug 12 2016, 2:14 AM
Hahnfeld added a reviewer: Hahnfeld.

LGTM

tools/clang-offload-bundler/ClangOffloadBundler.cpp
432

Unnecessary return, please remove as Alexey requested on the other patch

This revision is now accepted and ready to land.Aug 12 2016, 2:14 AM
sfantao updated this revision to Diff 68089.Aug 15 2016, 3:11 PM
sfantao marked an inline comment as done.
sfantao edited edge metadata.
  • Remove redundant return statement.
sfantao closed this revision.Aug 24 2016, 8:47 AM