[OffloadPackager] Add option to extract files from images

[jhuber6]

Summary

We use the clang-offload-packager too bundle many files into a single
binary format containing metadata. This is used for offloading
compilation which may contain multiple device binaries of different
types and architectures in a single file. We use this special binary
format to store these files along with some necessary metadata around
them. We use this format because of the difficulty of determining the
filesize of the various binary inputs that will be passed to the
offloading toolchain rather than engineering a solution for each input.

Previously we only support packaing many files into a single binary.
This patch adds support for doing the reverse by using the same
--image= syntax. To unpackage a binary we now present an input file
instead of an output.

[jhuber6]

llvm/tools/llvm-objcopy/ObjcopyOpts.td

200

I think the syntax for this flag could be improved, users may want to dump all files matching a triple, architecture, etc. Let me know if you have any suggestions.

[jhuber6]

Leftover objdump in test

[tra]

It would be great to allow dumping all offloading binaries at once. E.g. something like --dump-offloading all=file-name or just --dump-offloading file-name which would save all binaries into file-name.0 .. file-name.N.

llvm/tools/llvm-objcopy/ObjcopyOptions.cpp

865

--dump-section -> --dump-offloading ?

[MaskRay]

The feature doesn't fit into llvm-objcopy's purpose. llvm-objcopy is a binary manipulation tool which only understands ELF. --dump-offloading seems to parse the content using knowledge of the underlying format.

[jhuber6]

In D129507#3643246, @MaskRay wrote:

The feature doesn't fit into llvm-objcopy's purpose. llvm-objcopy is a binary manipulation tool which only understands ELF. --dump-offloading seems to parse the content using knowledge of the underlying format.

I see, I simply chose it because it had a similar feature for --dump-sections. Should this be placed in llvm-objdump as well then?

In D129507#3643241, @tra wrote:

It would be great to allow dumping all offloading binaries at once. E.g. something like --dump-offloading all=file-name or just --dump-offloading file-name which would save all binaries into file-name.0 .. file-name.N.

That's a good idea, I'll add it.

[tra]

In D129507#3643246, @MaskRay wrote:

The feature doesn't fit into llvm-objcopy's purpose. llvm-objcopy is a binary manipulation tool which only understands ELF. --dump-offloading seems to parse the content using knowledge of the underlying format.

Which tool would be appropriate for something like this? Do you suggest that we need a new tool?

The "only understands ELF" part may be open to interpretation. If one were to implement an extension to ELF format, would objcopy be the right place to support it? While objdump does have knowledge of the underlying formats, it does not deal with extraction of such data, but rather interprets it for human consumption.
This use case is somewhere in between -- it involves binary data manipulation, which is what objcopy does, but it does require some knowledge of the underlying format.

IMO, the offloading packing format is closer to being an extension of ELF, than being yet another ISA to disassemble (i.e. it's simple, generic and unlikely to change much) and objcopy seems like a decent fit. Offloading data manipulation may eventually be extended to insert new offload binaries, too, which would also fit objcopy functionality nicely.

[jhuber6]

In D129507#3643246, @MaskRay wrote:

The feature doesn't fit into llvm-objcopy's purpose. llvm-objcopy is a binary manipulation tool which only understands ELF. --dump-offloading seems to parse the content using knowledge of the underlying format.

Will this feature be acceptable llvm-objcopy considering we already have code there? I'd like to know where you think this code should live before I put in the effort to make any changes.

[MaskRay]

In D129507#3643307, @tra wrote:

In D129507#3643246, @MaskRay wrote:

The feature doesn't fit into llvm-objcopy's purpose. llvm-objcopy is a binary manipulation tool which only understands ELF. --dump-offloading seems to parse the content using knowledge of the underlying format.

Which tool would be appropriate for something like this? Do you suggest that we need a new tool?

The "only understands ELF" part may be open to interpretation. If one were to implement an extension to ELF format, would objcopy be the right place to support it? While objdump does have knowledge of the underlying formats, it does not deal with extraction of such data, but rather interprets it for human consumption.
This use case is somewhere in between -- it involves binary data manipulation, which is what objcopy does, but it does require some knowledge of the underlying format.

For example, @avl wanted to create a DWARF utility and I vaguely recall there was a discussion whether it applied to llvm-objcopy.
The answer is no as llvm-objcopy shouldn't know how to parse llvm-objcopy.

IMO, the offloading packing format is closer to being an extension of ELF, than being yet another ISA to disassemble (i.e. it's simple, generic and unlikely to change much) and objcopy seems like a decent fit. Offloading data manipulation may eventually be extended to insert new offload binaries, too, which would also fit objcopy functionality nicely.

It's not an extension of ELF. It is a new format with content embedded in an ELF section (according to while (Offset < Buffer->getBufferSize()) { and OffloadBinary::create). I don't really keep up-to-date with the already-plethora of GPU/offloading tools so I do not have an idea out of my head. If adding a new tool seems to be overkill, GPU/offloading folks may need to re-think how to reorganize the tools.
The question isn't raised for the first time why we have so many llvm-*/clang-* tools doing GPU/offloading stuff now.

[MaskRay]

Note: llvm-objcopy --dump-section .text=text a.out rewrites a.out (the output is nearly indistinguishable, but the byte stream may be different). You need llvm-objcopy --dump-section .text=text a.out /dev/null to suppress the rewrite.
This means --dump-offloading likely doesn't have good ergonomics in llvm-objcopy.

[tra]

The question isn't raised for the first time why we have so many llvm-*/clang-* tools doing GPU/offloading stuff now.

Well, it's a bit more complicated, but I do agree that offloading machinery we have is a bit of a mess and could use some streamlining.

That said, the issue at hand here is orthogonal to that. I'll take your word that llvm-objcopy is not the right place for offloading image manipulation.
That leaves some other tool.
The options I see are:

• llvm-objdump -- may not be the best fit as we want to save offload binaries to files, not just print them out. Whether we may ever want to embed offloading binaries is an open question. I think it could be useful, but it's probably a very niche use case, even for those few of us who care what's under the hood in the offloading binaries.
• incorporate functionality into clang-offload-packager.
• write a new tool?

I personally would prefer to avoid creating yet another tool. clang-offload-packager may be a reasonable place for this, considering that it already deals with the new offloading binary packaging style.

[jhuber6]

In D129507#3646755, @MaskRay wrote:

It's not an extension of ELF. It is a new format with content embedded in an ELF section (according to while (Offset < Buffer->getBufferSize()) { and OffloadBinary::create). I don't really keep up-to-date with the already-plethora of GPU/offloading tools so I do not have an idea out of my head. If adding a new tool seems to be overkill, GPU/offloading folks may need to re-think how to reorganize the tools.
The question isn't raised for the first time why we have so many llvm-*/clang-* tools doing GPU/offloading stuff now.

FWIW I've pretty much deprecated three of those GPU/offloading tools and introduced two. We have a lot of these tools mostly because of legacy which I'm hoping will be removed in time. Generally the work I'm doing is noisy, but a net reduction in complexity. The features I'm adding here are not strictly necessary for functionality, but are meant to be convenient tools for users familiar with stuff like cuobjdump from Nvidia. I wouldn't think a new tool is necessary as the actual code required to support these features is relatively small and in-line with many other data formats embedded in an ELF.

In D129507#3646757, @MaskRay wrote:

Note: llvm-objcopy --dump-section .text=text a.out rewrites a.out (the output is nearly indistinguishable, but the byte stream may be different). You need llvm-objcopy --dump-section .text=text a.out /dev/null to suppress the rewrite.
This means --dump-offloading likely doesn't have good ergonomics in llvm-objcopy.

Noted, thanks.

[jhuber6]

In D129507#3646824, @tra wrote:

I personally would prefer to avoid creating yet another tool. clang-offload-packager may be a reasonable place for this, considering that it already deals with the new offloading binary packaging style.

That tool is more or less a straight clone of Nvidia's fatbinary. We could potentially put it there, as it already creates the binaries that we embed inside the host, but it's a little out of scope. I can definitely put it there if people have major objections to putting stuff like this in llvm-objdump I could've made everything work without it, but it made the job of passing this information from the compiler driver to the back-end much more straightforward. Theoretically we could get rid of that and merge the clang-linker-wrapper into lld and bring the number of tools to zero.

[jhenderson]

I'm not familiar with some of the mentioned tools, but I tend to agree with @MaskRay's conclusions.

1. The functionality doesn't really fit in llvm-objcopy because the tool is intended for modifying a file in-place or making a (modified) copy to another place, neither of which is what you'll actually want it for. I think it would be appropriate for extracting the offloading section from an ELF binary as a whole, since that would basically just be --dump-section. Ideally, we'd have the ability to extract by section type, not just section name, but that's getting out-of-scope for this discussion.

2. llvm-objdump shouldn't be writing things to a file (or producing "binary" output that is intended for piping directly to a file). It's goal is printing information about object files. It is appropriate for it to be able to dump the offloading sections in an interpreted format, but not to extract them.

That means the functionality needs adding to a different tool, I feel (whether that tool is a new one or an existing one, I don't know).

[jhuber6]

In D129507#3647420, @jhenderson wrote:

I'm not familiar with some of the mentioned tools, but I tend to agree with @MaskRay's conclusions.

1. The functionality doesn't really fit in llvm-objcopy because the tool is intended for modifying a file in-place or making a (modified) copy to another place, neither of which is what you'll actually want it for. I think it would be appropriate for extracting the offloading section from an ELF binary as a whole, since that would basically just be --dump-section. Ideally, we'd have the ability to extract by section type, not just section name, but that's getting out-of-scope for this discussion.

2. llvm-objdump shouldn't be writing things to a file (or producing "binary" output that is intended for piping directly to a file). It's goal is printing information about object files. It is appropriate for it to be able to dump the offloading sections in an interpreted format, but not to extract them.

That means the functionality needs adding to a different tool, I feel (whether that tool is a new one or an existing one, I don't know).

llvm-objdump has the -s -j <secname> functionality, but that just dumps straight to the screen in hex format. I could put this in the clang-offload-packger, the tools that makes these blobs in the first place, but since it's a clang tool and doesn't care about object files I figured it wouldn't be a good fit. But if people are opposed to this kind of functionality in llvm-objdump, llvm-readobj, or llvm-objcopy I can just put it in there so it's at least available.

[MaskRay]

In D129507#3648113, @jhuber6 wrote:

In D129507#3647420, @jhenderson wrote:

I'm not familiar with some of the mentioned tools, but I tend to agree with @MaskRay's conclusions.

1. The functionality doesn't really fit in llvm-objcopy because the tool is intended for modifying a file in-place or making a (modified) copy to another place, neither of which is what you'll actually want it for. I think it would be appropriate for extracting the offloading section from an ELF binary as a whole, since that would basically just be --dump-section. Ideally, we'd have the ability to extract by section type, not just section name, but that's getting out-of-scope for this discussion.

2. llvm-objdump shouldn't be writing things to a file (or producing "binary" output that is intended for piping directly to a file). It's goal is printing information about object files. It is appropriate for it to be able to dump the offloading sections in an interpreted format, but not to extract them.

That means the functionality needs adding to a different tool, I feel (whether that tool is a new one or an existing one, I don't know).

llvm-objdump has the -s -j <secname> functionality, but that just dumps straight to the screen in hex format. I could put this in the clang-offload-packger, the tools that makes these blobs in the first place, but since it's a clang tool and doesn't care about object files I figured it wouldn't be a good fit. But if people are opposed to this kind of functionality in llvm-objdump, llvm-readobj, or llvm-objcopy I can just put it in there so it's at least available.

I think none of llvm-objdump, llvm-readobj, or llvm-objcopy is applicable. I don't know the offloading ecosystem well, but it seems that if a tool already does something like extracting an offloading section, it seems natural for it to support --dump-offloading if such a debugging functionality is needed.
If clang-offload-packager seems misnomer, since the utility is still new, a rename can be considered. If its main code is in clang/lib, you can add the dump functionality in llvm/lib/Object (to sit beside code which parses/processes some binary formats) and let the tool call that code.

[jhuber6]

In D129507#3652600, @MaskRay wrote:

I think none of llvm-objdump, llvm-readobj, or llvm-objcopy is applicable. I don't know the offloading ecosystem well, but it seems that if a tool already does something like extracting an offloading section, it seems natural for it to support --dump-offloading if such a debugging functionality is needed.
If clang-offload-packager seems misnomer, since the utility is still new, a rename can be considered. If its main code is in clang/lib, you can add the dump functionality in llvm/lib/Object (to sit beside code which parses/processes some binary formats) and let the tool call that code.

Fair enough, I'll just stash this with the clang-offload-packager.

[jhuber6]

Moving to clang packager.

[tra]

clang/test/Driver/offload-packager.c

12–14 ↗

(On Diff #453354)

So, with no -o option the tool treats --image to both choose the embedded image to dump and the file name to dump it into.
The patch appears to have more dumping modes. E.g. file name appears to be deriveable from the embedded image info. That should be tested, too.

Does user need to specify complete kind/triple/arch? Can I dump a subset of images -- e.g. all images with the same triple, or all with arch starting with sm_7?

It would also be very convenient if we could dump an image by its index in the executable and, also, all images. I suspect these two modes would be the ones used most often interactively.

BTW, we do not seem to have a way to list those embedded images. It would be great to add it.

15 ↗

(On Diff #453354)

I'd use diff -q or maybe cmp, though I don't know if it's available on Windows. We only want to know if the files differ and do not want it to accidentally dump binary on the screen.

clang/tools/clang-offload-packager/ClangOffloadPackager.cpp

70 ↗

(On Diff #453354)

Hooray for c++17!

168 ↗

(On Diff #453354)

A comment would be useful here. Or conversion of --image=k1=v1,k2=v2,k3=v3... into a kN->vN map could be extracted into a helper function with a meaningful name.

193 ↗

(On Diff #453354)

Nit: Args.count("file") ? Args["file"] : Saver.save(...) has one less !.

194–196 ↗

(On Diff #453354)

Is this guaranteed to be unique? I'd add an image index to the name.

204 ↗

(On Diff #453354)

You could use llvm::copy which works with ranges.

[jhuber6]

Thanks for the comments.

clang/test/Driver/offload-packager.c

12–14 ↗

(On Diff #453354)

So, with no -o option the tool treats --image to both choose the embedded image to dump and the file name to dump it into.
The patch appears to have more dumping modes. E.g. file name appears to be derivable from the embedded image info. That should be tested, too.

I tried to write a test for this. Given no explicit filename and multiple matches it should dump it as <input>-<triple>-<arch>.o or similar. It works when I test it locally but when I wrote a test for this I don't think it was writing to the correct directory because the lit test couldn't find them.

Does user need to specify complete kind/triple/arch? Can I dump a subset of images -- e.g. all images with the same triple, or all with arch starting with sm_7?

As written now, it should just write every image it finds that matches the image values. So --image=kind=openmp would dump everything whose producer was OpenMP. If it was --image=kind=openmp,file=out.o it would then write to out.o multiple times so you'd just end up with the last one. If you didn't provide a filename it would derive one as above. Not sure if it's worth handling this more intelligently since I think we should keep the user's requested name even if it's ambiguous.

It would also be very convenient if we could dump an image by its index in the executable and, also, all images. I suspect these two modes would be the ones used most often interactively.

BTW, we do not seem to have a way to list those embedded images. It would be great to add it.

That should be was llvm-objdump --offloading does, right? Or do you mean something different from just printing what's embedded.

clang/tools/clang-offload-packager/ClangOffloadPackager.cpp

70 ↗

(On Diff #453354)

Indeed.

168 ↗

(On Diff #453354)

I was thinking about that as it's somewhat obtuse, I'll try to make it clearer.

193 ↗

(On Diff #453354)

The formatting looked a little weird when I did it that way. Not a huge deal however.

194–196 ↗

(On Diff #453354)

Not necessarily, you're right that it might be good to use the image's index to provide a unique name in the case of conflicts. As mentioned in another comment I'm not sure if we should modify the output file if the user specified it however.

204 ↗

(On Diff #453354)

Good point.

[tra]

clang/test/Driver/offload-packager.c

12–14 ↗

(On Diff #453354)

lit test couldn't find them.

I see. The directory the test is run from is not necessarily the temporary directory. For that matter, such a test will fail in some setups where the source tree is read-only.
I think we need something like ar --output=dir which is used for very similar purposes.

Can I dump a subset of images ?

Given that we can narrow the subset to particular kind/triple/arch, it should do for now.

That should be was llvm-objdump --offloading does, right?

Now that lang-offload-packager got more functionality, it would make sense to make it the tool for listing embedded images, too, IMO.

clang/tools/clang-offload-packager/ClangOffloadPackager.cpp

194–196 ↗

(On Diff #453354)

Agreed, user input takes precedence.

My concern was for the clashes caused by the tool itself. There may conceivably be embedded images with identical properties and we need a way to distinguish between them.

[jhuber6]

clang/test/Driver/offload-packager.c

12–14 ↗

(On Diff #453354)

I think we need something like ar --output=dir which is used for very similar purposes.

That will work for now, I'll add that and update this patch with better testing.

Now that clang-offload-packager got more functionality, it would make sense to make it the tool for listing embedded images, too, IMO.

It shouldn't be too difficult to add since we'll be iterating through all of these already. I could do that in another patch.

clang/tools/clang-offload-packager/ClangOffloadPackager.cpp

194–196 ↗

(On Diff #453354)

Yeah, I'll add .x to the filename for that reason.

[jhuber6]

Update, still having problems making the test but I figured I'd just update now.

[tra]

In D129507#3730464, @jhuber6 wrote:

Update, still having problems making the test but I figured I'd just update now.

You could try something like this:
https://github.com/llvm/llvm-project/blob/d20e632853ad978e8008747d838d51b51d823a53/lld/test/MachO/stabs.s#L48

# RUN: cd %t && %lld -lSystem test.o foo.o no-debug.o -o test

[jhuber6]

Using @tra's suggestion to use cd. I had to make the test not apply to Windows
however, since I had to use realpath. But we don't support Windows anyway.

[tra]

clang/test/Driver/offload-packager.c

26 ↗

(On Diff #454106)

I'd also check that only openmp files got extracted. You may also want to delete all files in the temp directory -- I'm not sure if lit guarantees that it will be clean. I've seen other tests manually cleaning it in some cases.

[jhuber6]

Updating documentation, cleaning up, and adjusting tests.

[bader]

clang/test/Driver/offload-packager.c

2–3 ↗

(On Diff #454808)

Are nvptx and amdgpu target required for this test?
Latest version of the test invokes clang only for x86 target and clang-offload-packager just adds triple as metadata string without using llvm target. Right?

[jhuber6]

clang/test/Driver/offload-packager.c

2–3 ↗

(On Diff #454808)

You're right, we could get rid of those. Also this made me realize that the test for the bitcode file isn't actually using bitcode, I should probably fix that.

llvm/include/llvm/ObjCopy/CommonConfig.h

struct CommonConfig {
   StringRef SplitDWO;
   StringRef SymbolsPrefix;
   StringRef AllocSectionsPrefix;
   DiscardType DiscardMode = DiscardType::None;
 
   // Repeated options
   std::vector<NewSectionInfo> AddSection;
   std::vector<StringRef> DumpSection;
+  std::vector<StringRef> DumpOffloading;
   std::vector<NewSectionInfo> UpdateSection;
 
   // Section matchers
   NameMatcher KeepSection;
   NameMatcher OnlySection;
   NameMatcher ToRemove;
 
   // Symbol matchers

llvm/lib/ObjCopy/ConfigManager.cpp

 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace objcopy {
 
 Expected<const COFFConfig &> ConfigManager::getCOFFConfig() const {
   if (!Common.SplitDWO.empty() || !Common.SymbolsPrefix.empty() ||
       !Common.AllocSectionsPrefix.empty() || !Common.DumpSection.empty() ||
-      !Common.KeepSection.empty() || !Common.SymbolsToGlobalize.empty() ||
-      !Common.SymbolsToKeep.empty() || !Common.SymbolsToLocalize.empty() ||
-      !Common.SymbolsToWeaken.empty() || !Common.SymbolsToKeepGlobal.empty() ||
-      !Common.SectionsToRename.empty() || !Common.SetSectionAlignment.empty() ||
-      Common.ExtractDWO || Common.PreserveDates || Common.StripDWO ||
-      Common.StripNonAlloc || Common.StripSections || Common.Weaken ||
-      Common.DecompressDebugSections ||
+      !Common.DumpOffloading.empty() || !Common.KeepSection.empty() ||
+      !Common.SymbolsToGlobalize.empty() || !Common.SymbolsToKeep.empty() ||
+      !Common.SymbolsToLocalize.empty() || !Common.SymbolsToWeaken.empty() ||
+      !Common.SymbolsToKeepGlobal.empty() || !Common.SectionsToRename.empty() ||
+      !Common.SetSectionAlignment.empty() || Common.ExtractDWO ||
+      Common.PreserveDates || Common.StripDWO || Common.StripNonAlloc ||
+      Common.StripSections || Common.Weaken || Common.DecompressDebugSections ||
       Common.DiscardMode == DiscardType::Locals || !Common.SymbolsToAdd.empty())
     return createStringError(llvm::errc::invalid_argument,
                              "option is not supported for COFF");
 
   return COFF;
 }
 
 Expected<const MachOConfig &> ConfigManager::getMachOConfig() const {
   if (!Common.SplitDWO.empty() || !Common.SymbolsPrefix.empty() ||
       !Common.AllocSectionsPrefix.empty() || !Common.KeepSection.empty() ||
-      !Common.SymbolsToGlobalize.empty() || !Common.SymbolsToKeep.empty() ||
-      !Common.SymbolsToLocalize.empty() || !Common.SymbolsToWeaken.empty() ||
-      !Common.SymbolsToKeepGlobal.empty() || !Common.SectionsToRename.empty() ||
+      !Common.DumpOffloading.empty() || !Common.SymbolsToGlobalize.empty() ||
+      !Common.SymbolsToKeep.empty() || !Common.SymbolsToLocalize.empty() ||
+      !Common.SymbolsToWeaken.empty() || !Common.SymbolsToKeepGlobal.empty() ||
+      !Common.SectionsToRename.empty() ||
       !Common.UnneededSymbolsToRemove.empty() ||
       !Common.SetSectionAlignment.empty() || !Common.SetSectionFlags.empty() ||
       Common.ExtractDWO || Common.PreserveDates || Common.StripAllGNU ||
       Common.StripDWO || Common.StripNonAlloc || Common.StripSections ||
       Common.Weaken || Common.DecompressDebugSections || Common.StripUnneeded ||
       Common.DiscardMode == DiscardType::Locals || !Common.SymbolsToAdd.empty())
     return createStringError(llvm::errc::invalid_argument,
                              "option is not supported for MachO");
 
   return MachO;
 }
 
 Expected<const WasmConfig &> ConfigManager::getWasmConfig() const {
   if (!Common.AddGnuDebugLink.empty() || Common.ExtractPartition ||
       !Common.SplitDWO.empty() || !Common.SymbolsPrefix.empty() ||
-      !Common.AllocSectionsPrefix.empty() ||
+      !Common.DumpOffloading.empty() || !Common.AllocSectionsPrefix.empty() ||
       Common.DiscardMode != DiscardType::None || !Common.SymbolsToAdd.empty() ||
       !Common.SymbolsToGlobalize.empty() || !Common.SymbolsToLocalize.empty() ||
       !Common.SymbolsToKeep.empty() || !Common.SymbolsToRemove.empty() ||
       !Common.UnneededSymbolsToRemove.empty() ||
       !Common.SymbolsToWeaken.empty() || !Common.SymbolsToKeepGlobal.empty() ||
       !Common.SectionsToRename.empty() || !Common.SetSectionAlignment.empty() ||
       !Common.SetSectionFlags.empty() || !Common.SymbolsToRename.empty())
     return createStringError(llvm::errc::invalid_argument,
                              "only flags for section dumping, removal, and "
                              "addition are supported");
 
   return Wasm;
 }
 
 Expected<const XCOFFConfig &> ConfigManager::getXCOFFConfig() const {
   if (!Common.AddGnuDebugLink.empty() || Common.ExtractPartition ||
       !Common.SplitDWO.empty() || !Common.SymbolsPrefix.empty() ||
-      !Common.AllocSectionsPrefix.empty() ||
+      !Common.DumpOffloading.empty() || !Common.AllocSectionsPrefix.empty() ||
       Common.DiscardMode != DiscardType::None || !Common.AddSection.empty() ||
       !Common.DumpSection.empty() || !Common.SymbolsToAdd.empty() ||
       !Common.KeepSection.empty() || !Common.OnlySection.empty() ||
       !Common.ToRemove.empty() || !Common.SymbolsToGlobalize.empty() ||
       !Common.SymbolsToKeep.empty() || !Common.SymbolsToLocalize.empty() ||
       !Common.SymbolsToRemove.empty() ||
       !Common.UnneededSymbolsToRemove.empty() ||
       !Common.SymbolsToWeaken.empty() || !Common.SymbolsToKeepGlobal.empty() ||

llvm/lib/ObjCopy/ELF/ELFObjcopy.cpp

 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/ObjCopy/CommonConfig.h"
 #include "llvm/ObjCopy/ELF/ELFConfig.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Object/Error.h"
+#include "llvm/Object/OffloadBinary.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/FileSystem.h"
if (Sec.Name == SecName) {
         return E;
       return Error::success();
     }
   }
   return createStringError(object_error::parse_failed, "section '%s' not found",
                            SecName.str().c_str());
 }
 
+static Error dumpOffloadToFile(StringRef Identifier, StringRef Filename,
+                               Object &Obj) {
+  for (auto &Sec : Obj.sections()) {
+    if (Sec.Type != ELF::SHT_LLVM_OFFLOADING)
+      continue;
+
+    // This data can be misaligned if extracted from an archive.
+    std::unique_ptr<MemoryBuffer> Buffer =
+        MemoryBuffer::getMemBuffer(toStringRef(Sec.OriginalData), "", false);
+    if (!isAddrAligned(Align(OffloadBinary::getAlignment()),
+                       Buffer->getBufferStart()))
+      Buffer = MemoryBuffer::getMemBufferCopy(Buffer->getBuffer(),
+                                              Buffer->getBufferIdentifier());
+    uint64_t Offset = 0;
+    uint64_t Index = 0;
+    while (Offset < Buffer->getBufferSize()) {
+      MemoryBufferRef BufferRef =
+          MemoryBufferRef(Buffer->getBuffer().drop_front(Offset), "");
+      auto BinaryOrErr = OffloadBinary::create(BufferRef);
+      if (!BinaryOrErr)
+        return createStringError(object_error::parse_failed,
+                                 "while extracting offloading files: " +
+                                     toString(BinaryOrErr.takeError()));
+      const OffloadBinary &Binary = **BinaryOrErr;
+
+      if (Identifier == std::to_string(Index)) {
+        Expected<std::unique_ptr<FileOutputBuffer>> OutputOrErr =
+            FileOutputBuffer::create(Filename, Binary.getImage().size());
+        if (!OutputOrErr)
+          return OutputOrErr.takeError();
+        std::unique_ptr<FileOutputBuffer> Output = std::move(*OutputOrErr);
+        std::copy(Binary.getImage().bytes_begin(),
+                  Binary.getImage().bytes_end(), Output->getBufferStart());
+        if (Error E = Output->commit())
+          return std::move(E);
+        return Error::success();
+      }
+
+      Offset += Binary.getSize();
+      Index++;
+    }
+  }
+  return createStringError(object_error::parse_failed,
+                           "Offloading image with index '%s' not found",
+                           Identifier.str().c_str());
+}
+
 static bool isCompressable(const SectionBase &Sec) {
   return !(Sec.Flags & ELF::SHF_COMPRESSED) &&
          StringRef(Sec.Name).startswith(".debug");
 }
 
 static Error replaceDebugSections(
     Object &Obj, function_ref<bool(const SectionBase &)> ShouldReplace,
     function_ref<Expected<SectionBase *>(const SectionBase *)> AddSection) {
static Error handleArgs(const CommonConfig &Config, const ELFConfig &ELFConfig,
   for (StringRef Flag : Config.DumpSection) {
     StringRef SectionName;
     StringRef FileName;
     std::tie(SectionName, FileName) = Flag.split('=');
     if (Error E = dumpSectionToFile(SectionName, FileName, Obj))
       return E;
   }
 
+  // Dump embedded offloading images matching the input identifier.
+  for (StringRef Flag : Config.DumpOffloading) {
+    StringRef Identifer;
+    StringRef FileName;
+    std::tie(Identifer, FileName) = Flag.split('=');
+    if (Error E = dumpOffloadToFile(Identifer, FileName, Obj))
+      return E;
+  }
+
   // It is important to remove the sections first. For example, we want to
   // remove the relocation sections before removing the symbols. That allows
   // us to avoid reporting the inappropriate errors about removing symbols
   // named in relocations.
   if (Error E = replaceAndRemoveSections(Config, ELFConfig, Obj))
     return E;
 
   if (Error E = updateAndRemoveSymbols(Config, ELFConfig, Obj))

llvm/test/tools/llvm-objcopy/Offloading/Inputs/binary.yaml

+# Input image with a single character to dump.
+!Offload
+Members:
+  - ImageKind:      IMG_Bitcode
+    OffloadKind:    OFK_OpenMP
+    String:
+    - Key:            "triple"
+      Value:          "amdgcn-amd-amdhsa"
+    - Key:            "arch"
+      Value:          "gfx908"
+    Content:        "41" # A
+  - ImageKind:      IMG_Bitcode
+    OffloadKind:    OFK_OpenMP
+    String:
+    - Key:            "triple"
+      Value:          "amdgcn-amd-amdhsa"
+    - Key:            "arch"
+      Value:          "gfx90a"
+    Content:        "42" # B
+  - ImageKind:      IMG_Cubin
+    OffloadKind:    OFK_OpenMP
+    String:
+    - Key:            "triple"
+      Value:          "nvptx64-nvidia-cuda"
+    - Key:            "arch"
+      Value:          "sm_52"
+    Content:        "43" # C
+  - ImageKind:      IMG_None
+    OffloadKind:    OFK_OpenMP
+    String:
+    - Key:            "triple"
+      Value:          "nvptx64-nvidia-cuda"
+    - Key:            "arch"
+      Value:          "sm_70"
+    Content:        "44" # D

llvm/test/tools/llvm-objcopy/Offloading/alignment.test

+## Ensure we can dump the contents even if the alignment is bad.
+# RUN: yaml2obj %s -o %t.elf
+# RUN: yaml2obj %S/Inputs/binary.yaml -o %t.bin
+# RUN: llvm-objcopy --update-section .llvm.offloading=%t.bin %t.elf
+# RUN: llvm-objcopy --dump-offloading 0=- %t.elf | FileCheck %s -DFILENAME=%t
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+Sections:
+  - Name:            .misaligned
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         "41424300"
+  - Name:            .llvm.offloading
+    Type:            SHT_LLVM_OFFLOADING
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+
+# CHECK: A

llvm/test/tools/llvm-objcopy/Offloading/binary.test

+## Check dumping the offloading image contents by index.
+# RUN: yaml2obj %S/Inputs/binary.yaml -o %t.bin
+# RUN: yaml2obj %s -o %t.elf
+# RUN: llvm-objcopy --update-section .llvm.offloading=%t.bin %t.elf
+# RUN: llvm-objcopy --dump-offloading 0=- --dump-offloading 1=- --dump-offloading 2=- --dump-offloading 3=- %t.elf \
+# RUN:   | FileCheck %s --match-full-lines --strict-whitespace --implicit-check-not={{.}}
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+Sections:
+  - Name:            .llvm.offloading
+    Type:            SHT_LLVM_OFFLOADING
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000008
+
+# CHECK:ABCD

llvm/test/tools/llvm-objcopy/Offloading/failure.test

+# RUN: yaml2obj %s -o %t
+# RUN: not llvm-objcopy --dump-offloading 0=b %t 2>&1 | FileCheck -DFILENAME=%t %s
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+Sections:
+  - Name:            .llvm.offloading
+    Type:            SHT_LLVM_OFFLOADING
+    Flags:           [ SHF_EXCLUDE ]
+    Address:         0x0
+    AddressAlign:    0x0000000000000008
+    Content:         "10ffb0ad"
+
+# CHECK: error: '[[FILENAME]]': while extracting offloading files: Invalid data was encountered while parsing the file

llvm/test/tools/llvm-objcopy/Offloading/non-elf.test

+# RUN: yaml2obj %s -o %t
+# RUN: not llvm-objcopy --dump-offloading 0=%t.out %t 2>&1 | FileCheck -DFILENAME=%t %s
+
+--- !COFF
+header:
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: []
+sections:
+  - Name:            .rdata
+    Characteristics: []
+    SectionData:     00
+symbols:
+
+# CHECK: error: option is not supported for COFF

llvm/test/tools/llvm-objcopy/Offloading/not-found.test

+## Ensure we can dump the contents even if the alignment is bad.
+# RUN: yaml2obj %s -o %t.elf
+# RUN: yaml2obj %S/Inputs/binary.yaml -o %t.bin
+# RUN: llvm-objcopy --update-section .llvm.offloading=%t.bin %t.elf
+# RUN: not llvm-objcopy --dump-offloading 9999999=- %t.elf 2>&1 | FileCheck %s -DFILENAME=%t.elf
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+Sections:
+  - Name:            .misaligned
+    Type:            SHT_PROGBITS
+    AddressAlign:    0x0000000000000001
+    Content:         "41424300"
+  - Name:            .llvm.offloading
+    Type:            SHT_LLVM_OFFLOADING
+    Flags:           [ SHF_EXCLUDE ]
+    AddressAlign:    0x0000000000000001
+
+# CHECK: error: '[[FILENAME]]': Offloading image with index '9999999' not found

llvm/tools/llvm-objcopy/ObjcopyOptions.cpp

objcopy::parseObjcopyOptions(ArrayRef<const char *> RawArgsArr,
   for (auto *Arg : InputArgs.filtered(OBJCOPY_dump_section)) {
     StringRef Value(Arg->getValue());
     if (Value.split('=').second.empty())
       return createStringError(
           errc::invalid_argument,
           "bad format for --dump-section, expected section=file");
     Config.DumpSection.push_back(Value);
   }
+  for (auto *Arg : InputArgs.filtered(OBJCOPY_dump_offloading)) {
+    StringRef Value(Arg->getValue());
+    if (Value.split('=').second.empty())
+      return createStringError(
+          errc::invalid_argument,
+          "bad format for --dump-section, expected identifier=file");
+    Config.DumpOffloading.push_back(Value);
+  }
   Config.StripAll = InputArgs.hasArg(OBJCOPY_strip_all);
   Config.StripAllGNU = InputArgs.hasArg(OBJCOPY_strip_all_gnu);
   Config.StripDebug = InputArgs.hasArg(OBJCOPY_strip_debug);
   Config.StripDWO = InputArgs.hasArg(OBJCOPY_strip_dwo);
   Config.StripSections = InputArgs.hasArg(OBJCOPY_strip_sections);
   Config.StripNonAlloc = InputArgs.hasArg(OBJCOPY_strip_non_alloc);
   Config.StripUnneeded = InputArgs.hasArg(OBJCOPY_strip_unneeded);
   Config.ExtractDWO = InputArgs.hasArg(OBJCOPY_extract_dwo);

llvm/tools/llvm-objcopy/ObjcopyOpts.td

: Eq<"keep-symbols",
          "'#'. Leading and trailing whitespace is stripped from each line. May "
          "be repeated to read symbols from many files.">,
       MetaVarName<"filename">;
 
 defm dump_section
     : Eq<"dump-section",
          "Dump contents of section named <section> into file <file>">,
       MetaVarName<"section=file">;
+defm dump_offloading
+    : Eq<"dump-offloading",
+         "Dump the <index> embedded offloading buffer into file <file>">,
+      MetaVarName<"index=file">;
 defm prefix_symbols
     : Eq<"prefix-symbols", "Add <prefix> to the start of every symbol name">,
       MetaVarName<"prefix">;
 
 defm prefix_alloc_sections
     : Eq<"prefix-alloc-sections", "Add <prefix> to the start of every allocated section name">,
       MetaVarName<"prefix">;