Add Statically Linked Libraries

[ashi1]

Summary

Add GNU Static Lib Tool, which supports the --emit-static-lib
flag. For HIP, a static library archive will be created and
consist of HIP Fat Binary host object with the device images embedded.
Using MC Directives, embed the device images and define symbols.
Using llvm-ar to create the static archive. Also, delete existing
output file to ensure a new archive is created each time.

[ashi1]

Adding a new diff with full context.

[JonChesterfield]

clang/lib/Driver/ToolChain.cpp

582

What does ar do with the symbol table for this? I'd expect it to include host symbols but none of the device ones

[tra]

clang/lib/Driver/ToolChain.cpp

582

Also, can we use llvm-ar? It should reduce the amount of surprises we may run into with whatever ar we may find in the PATH.

[ashi1]

clang/lib/Driver/ToolChain.cpp

582

Hi Jon, the ar tool will embed the host objects, .o, and the device fat binary object, .hipfbo, into a global symbol __hip_fatbin. HIP runtime has the capability to extract this symbol, and digest the HIP fat binary, and extract the device symbols to run.

Hi tra, I will update this to use llvm-ar, thanks.

[ashi1]

Changed to use llvm-ar tool.

[yaxunl]

LGTM. Thanks.

clang/lib/Driver/ToolChains/CommonArgs.cpp

1421 ↗

(On Diff #259969)

should be HIPToolChain

1469 ↗

(On Diff #259969)

extra blank line

[tra]

Added GNU Static Lib Tool, which supports the --hip-emit-static-lib flag with -fgpu-rdc. A static library archive will be created and include both the host object and device object embedded.

What's going to be in that static library in the end? The description above is somewhat confusing. On one hand it says include both the host object and device object embedded.. Does it mean host object and device object to be embeddeed? I.e. two separate object files? Or does it mean that the device object is embedded in the host object and the host object is included in the library?

Using llvm-mc to embed the HIP fat binary into __hip_fatbin symbol, and system ar to archive. Also, delete existing output file to ensure a new archive with index is created each time.

__hip_fatbin, presumably in the host object. If the device object is embedded in the host object, why do we need to put it into the library? In the end theuser would only see the host object and we may just link it directly without making it a library. What do I miss?

It would be great to add some details to the test, because right now it's really hard to figure out what's being done and almost impossible to understand why it is done that way.

The test example is also odd in a sense that it compiles two source files. Is the purpose of this patch to allow creation of a self-contained archive with GPU-side objects that still need final linking step further down the build pipeline?

If that's indeed the driving motivation behind the patch, it may need some improvement, as it's unlikely to work well in practice, where pretty much every build system compiles one file at a time. Giving single compilation large number of inputs is simply not going to scale in practice. We need to figure out a way to implement the final link for object files compiled with -fgpu-rdc in separate compilations.

[yaxunl]

The purpose of this patch is to generate a static library that can be used as an "ordinary" static library, i.e., can be linked by a system linker, given a bunch of HIP programs that need -fgpu-rdc to compile.

This is a common use case for a library written with HIP. That is, the library provides a bunch of host APIs so that other C++ programs can use it. The library does not provide device code that need to be linked with the user's device code. The users want to use it as an ordinary host library.

If the users of the library just need dynamic library, then they do not need this feature, since clang is already able to do that.

However, clang is not able to generate a static library for that purpose. This is what this feature is trying to achieve.

So basically this patch will link the device code in these HIP programs, generate device ISA, and create a host object containing the ISA, and add the host object to the generated static library. This is exactly what clang does for -fgpu-rdc when generating a shared library. The only difference is that, instead of generating a shared library, now clang generates a static library.

[tra]

In D78759#2014039, @yaxunl wrote:

The purpose of this patch is to generate a static library that can be used as an "ordinary" static library, i.e., can be linked by a system linker, given a bunch of HIP programs that need -fgpu-rdc to compile.

This is a common use case for a library written with HIP. That is, the library provides a bunch of host APIs so that other C++ programs can use it. The library does not provide device code that need to be linked with the user's device code. The users want to use it as an ordinary host library.

Is the intention that all GPU-side code inside this library is linked together, even though individual sources are compiled to object files?

Is there a way to compile .hip ->.o with GPU-side object and then
link together GPU objects from multiple such .o files into a single .o with partially linked host object and fully linked GPU code?

If the users of the library just need dynamic library, then they do not need this feature, since clang is already able to do that.

Can you give me an example how this shared library is produced?

However, clang is not able to generate a static library for that purpose. This is what this feature is trying to achieve.

So basically this patch will link the device code in these HIP programs, generate device ISA, and create a host object containing the ISA, and add the host object to the generated static library. This is exactly what clang does for -fgpu-rdc when generating a shared library. The only difference is that, instead of generating a shared library, now clang generates a static library.

Can you give me a small example of using clang to generate such .so ? If this approach relies on compiling multiple *source* files at once, that will have practicall use issues. If it can be done on .o as inputs, then it may be OK.

[ashi1]

In D78759#2014185, @tra wrote:

In D78759#2014039, @yaxunl wrote:

The purpose of this patch is to generate a static library that can be used as an "ordinary" static library, i.e., can be linked by a system linker, given a bunch of HIP programs that need -fgpu-rdc to compile.

This is a common use case for a library written with HIP. That is, the library provides a bunch of host APIs so that other C++ programs can use it. The library does not provide device code that need to be linked with the user's device code. The users want to use it as an ordinary host library.

Is the intention that all GPU-side code inside this library is linked together, even though individual sources are compiled to object files?

Is there a way to compile .hip ->.o with GPU-side object and then
link together GPU objects from multiple such .o files into a single .o with partially linked host object and fully linked GPU code?

When clang detects the .o files are clang-offload-bundle or host object, it will extract the bitcode and link them and generate ISA, then create a host object containing it, then link them together. We can link multiple objects together to create either a shared library or a static library (with this patch).

If the users of the library just need dynamic library, then they do not need this feature, since clang is already able to do that.

Can you give me an example how this shared library is produced?

In a ROCm project, RCCL is building the shared library using -fgpu-rdc. That project compiles sources using -c and -fgpu-rdc, and then links them together into a shared library. We require this patch to allow them to produce a static library too.

However, clang is not able to generate a static library for that purpose. This is what this feature is trying to achieve.

So basically this patch will link the device code in these HIP programs, generate device ISA, and create a host object containing the ISA, and add the host object to the generated static library. This is exactly what clang does for -fgpu-rdc when generating a shared library. The only difference is that, instead of generating a shared library, now clang generates a static library.

Can you give me a small example of using clang to generate such .so ? If this approach relies on compiling multiple *source* files at once, that will have practicall use issues. If it can be done on .o as inputs, then it may be OK.

Yes, this can be done on .o files. If you have multiple hip objects, foo.o and bar.o with their own device code, we can pass them into clang to generate a shared library file. clang currently extracts their bitcode, and links them together, and creates host objects containing the device code, and will link them together. It can accept multiple .o as inputs. With this patch, using the flag hip-emit-static-lib, we can generate a .a static library file.

[JonChesterfield]

In D78759#2031560, @ashi1 wrote:

When clang detects the .o files are clang-offload-bundle or host object, it will extract the bitcode and link them and generate ISA, then create a host object containing it, then link them together. We can link multiple objects together to create either a shared library or a static library (with this patch).

The semantics of this patch are a bit unclear to me.

When linking a static archive, the linker searches for each currently unresolved symbol in the archive and links in only the object that exposes that symbol. Unused objects are left in the archive. That's useful behaviour in various settings, and overridden with --whole-archive when all the objects are wanted.

The behaviour I would expect, from a user perspective, is to link in the host code which satisfies unresolved symbols for the host, and the device code that satisfies unresolved symbols for the device, and no extra files, regardless of whether the files are machine code for either arch, bitcode for either arch, or an offload-bundle potentially containing some of each. That's what we have today for mixtures of bitcode and machine code for a single architecture. And extensible in the obvious fashion for > 2 architectures.

That seems implementable - skip over objects for architectures other than the one currently of interest, and reach into clang-offload-bundle if present to do the same thing. Optionally provide a symbol table per arch to make it faster.

I think you're suggesting that the bitcode from every object is extracted and linked together, not just the ones that are actually used. If so, do you also change the semantics for the host code part of the object? What about for host objects that don't contain any bitcode? What about bitcode files for the device that don't have a corresponding host part?

[JonChesterfield]

clang/lib/Driver/ToolChain.cpp

582

I'm not really following you here. Longer comment at the top level.

It seems a shame to spawn llvm-ar to make the archive. It's not a very complicated file format and we already have the implementation of llvm-ar in tree. Can we use it as a library instead of forking?

[yaxunl]

In D78759#2031649, @JonChesterfield wrote:

In D78759#2031560, @ashi1 wrote:

When clang detects the .o files are clang-offload-bundle or host object, it will extract the bitcode and link them and generate ISA, then create a host object containing it, then link them together. We can link multiple objects together to create either a shared library or a static library (with this patch).

The semantics of this patch are a bit unclear to me.

When linking a static archive, the linker searches for each currently unresolved symbol in the archive and links in only the object that exposes that symbol. Unused objects are left in the archive. That's useful behaviour in various settings, and overridden with --whole-archive when all the objects are wanted.

The behaviour I would expect, from a user perspective, is to link in the host code which satisfies unresolved symbols for the host, and the device code that satisfies unresolved symbols for the device, and no extra files, regardless of whether the files are machine code for either arch, bitcode for either arch, or an offload-bundle potentially containing some of each. That's what we have today for mixtures of bitcode and machine code for a single architecture. And extensible in the obvious fashion for > 2 architectures.

That seems implementable - skip over objects for architectures other than the one currently of interest, and reach into clang-offload-bundle if present to do the same thing. Optionally provide a symbol table per arch to make it faster.

I think you're suggesting that the bitcode from every object is extracted and linked together, not just the ones that are actually used. If so, do you also change the semantics for the host code part of the object? What about for host objects that don't contain any bitcode? What about bitcode files for the device that don't have a corresponding host part?

The use case of this patch is that the library only needs to provide host APIs which can be called by C++ programs but do not need to expose device code to the users. Therefore the device code in the library are linked, finalized, and embedded as ISA in host objects and not exposed to library users. We have user requests to support such use case.

There may be use cases that users want to store both host codes and device codes in a static library and expose both to users, however that is out of scope of this patch.

clang/lib/Driver/ToolChain.cpp

582

Even if we can do that in-process, we still need the action and tool to represent that compilation step. Besides, llvm-ar does not support static library used by MSVC. If we want to extend this to MSVC toolchain, we need these API's.

[JonChesterfield]

In D78759#2031741, @yaxunl wrote:

The use case of this patch is that the library only needs to provide host APIs which can be called by C++ programs but do not need to expose device code to the users. Therefore the device code in the library are linked, finalized, and embedded as ISA in host objects and not exposed to library users. We have user requests to support such use case.

There may be use cases that users want to store both host codes and device codes in a static library and expose both to users, however that is out of scope of this patch.

Ah, OK. So it's HIP specific, and not a general purpose static archive. In that case, the many functions with names based on StaticLib should probably all be renamed HIPStaticLib or similar to indicate that it doesn't work for other toolchains. Can any of this code be moved into the HIP.cpp toolchain file?

[yaxunl]

In D78759#2031780, @JonChesterfield wrote:

In D78759#2031741, @yaxunl wrote:

The use case of this patch is that the library only needs to provide host APIs which can be called by C++ programs but do not need to expose device code to the users. Therefore the device code in the library are linked, finalized, and embedded as ISA in host objects and not exposed to library users. We have user requests to support such use case.

There may be use cases that users want to store both host codes and device codes in a static library and expose both to users, however that is out of scope of this patch.

Ah, OK. So it's HIP specific, and not a general purpose static archive. In that case, the many functions with names based on StaticLib should probably all be renamed HIPStaticLib or similar to indicate that it doesn't work for other toolchains. Can any of this code be moved into the HIP.cpp toolchain file?

It is not HIP specific. The use case is generic and common. Many cases users do not care about device code internals and only interested in using the functionality of a library through usual host APIs. If other toolchains would like to support the same use case they can easily extend it to their own use.

Also the implementation requires introduction of static lib tool in general since to get a tool it has to go through the generic interface of toolchain. There have already been all kinds of Tools and JobActions in the generic interface that are for specific toolchains, e.g. clang-offload-bundler, so this is not something new.

[JonChesterfield]

In D78759#2031921, @yaxunl wrote:

It is not HIP specific. The use case is generic and common. Many cases users do not care about device code internals and only interested in using the functionality of a library through usual host APIs. If other toolchains would like to support the same use case they can easily extend it to their own use.

Regardless of how popular the use case might be, it's not building a static library. It uses the same file format, which seems adequate for the purpose, but with different semantics.

Can you split this patch into a function for building static libraries, which other toolchains could use unmodified, and some HIP specific code which makes use of that function?

In D78759#2031921, @yaxunl wrote:

Also the implementation requires introduction of static lib tool in general since to get a tool it has to go through the generic interface of toolchain. There have already been all kinds of Tools and JobActions in the generic interface that are for specific toolchains, e.g. clang-offload-bundler, so this is not something new.

No objection to another tool. The objection is to calling it 'static lib', on the basis that it isn't one, and other toolchains may want to build a conventional static library from the clang driver.

[yaxunl]

In D78759#2031963, @JonChesterfield wrote:

Can you split this patch into a function for building static libraries, which other toolchains could use unmodified, and some HIP specific code which makes use of that function?

The current patch already does that. For other toolchains, this option will generate a normal static library, like ar does.

For HIP, it does something extra (link, finalize and embeds device code), then generate a static library for host object like ar does.

Maybe Aaron can add a lit test for using this option for C language.

[yaxunl]

This patch follows a generic design. It introduces generic option, JobAction, and Tool about creating static library which can be customized for each toolchain.

For linux toolchain, it is just the conventional ar, which is implemented in this patch.

For HIP toolchain, currently it implements a specific behavior to finalize device code and generate a common host static library. It is a static library from the host point of view.

This does not preventing it to be extended to generating other format of static libraries. We can easily introduce some options to generate static library in a different way, e.g. archive of clang-offload-bundler bundles, or clang-offload-bundler bundles of archives.

For other toolchains, they just need to override virtual functions getStaticLibTool and implement their own static lib tool to implement their own command to create static libs.

[JonChesterfield]

In D78759#2032019, @yaxunl wrote:

In D78759#2031963, @JonChesterfield wrote:

Can you split this patch into a function for building static libraries, which other toolchains could use unmodified, and some HIP specific code which makes use of that function?

The current patch already does that. For other toolchains, this option will generate a normal static library, like ar does.

Some comments inline in StaticLibTool::ConstructJob

Unrelated to the above, but a consequence of reading the code a few times, there's a lot of duplication with tools::AddHIPLinkerScript that doesn't seem necessary.

clang/lib/Driver/ToolChains/CommonArgs.cpp

1427 ↗

(On Diff #259969)

This code looked familiar - it seems to have a lot in common with AddHIPLinkerScript just above. Can the common parts be factored out?

1486 ↗

(On Diff #259969)

There isn't a linker script involved here. Copied & pasted from above?

clang/lib/Driver/ToolChains/Gnu.cpp

350

This function is named (and starts off) generic

373

Then does some things which aren't obviously generic. What's special about -stdlib here?

392

Finally does something that is definitely hip specific

[ashi1]

Maybe Aaron can add a lit test for using this option for C language.

I can add a lit test for C language.

clang/lib/Driver/ToolChains/CommonArgs.cpp

1427 ↗

(On Diff #259969)

Could re-factoring be in a separate commit? It may be significant and risky. I think there can be other optimizations made here.

1486 ↗

(On Diff #259969)

Okay I will fix these comments, the comment can be file generator script.

clang/lib/Driver/ToolChains/Gnu.cpp

350

These two lines were added for the D.Diag error checking below.

373

This is a generic line from the Linker::ConstructJob function

392

There is a check in this function which will skip this function if not HIP host toolchain above.

// If this is not a HIP host toolchain, we don't need to do anything.
if (!JA.isHostOffloading(Action::OFK_HIP))
  return;

[ashi1]

ping

[tra]

Few cosmetic nits. LGTM in general. I'll leave the approval to @JonChesterfield

clang/lib/Driver/ToolChains/CommonArgs.cpp

1417–1420 ↗

(On Diff #263769)

You don't need to static_cast in order to do getTriple(), regular ToolChain should do the job.
With the cast removed, folding C.getSingleOffloadToolChain<Action::OFK_HIP>()->getTriple().getArch() == llvm::Triple::amdgcn into assert(). would also let you remove the (void)HIPTC below.

1466 ↗

(On Diff #263769)

Nit: Would it make sense to this if with the one above, so the file name construction is in one place for both names?

1473 ↗

(On Diff #263769)

Nit: It's only used once. Fold it into MakeArgString()?

clang/lib/Driver/ToolChains/Gnu.cpp

348–349

Nit: the cast may be unnecessary. Both getDriver() and AddLinkerInput() operate on base ToolChain.

379

This is odd.
We check is the file is writable or not a regular file. So far so good.
But then we issue a message that we can't *remove* the file, even though we didn't try to remove anything yet.
The checks we've done do not necessarily mean that the file is not removable. E.g. a read-only file in a writable directory is removable. On UNIX file unlinking is an operation on a directory, not on the file. There are also files that are not regular files that are also possible to delete. E.g. there are symlinks, device nodes, sockets.

IMO, if we report an error it should be due to an error. In this case we produce the diagnostic for something we didn't even try.

Perhaps just remove these predictive checks altogether and rely on the error returned by fs::remove() below?

[JonChesterfield]

I don't like the copy and paste but at least it's contained within HIP specific functions.

The control flow in tools::gnutools::StaticLibTool::ConstructJob doesn't seem good though. It's a generic function that unconditionally calls a hip specific function which happens to return immediately in non-hip cases. That really should be a hip specific function calling the generic one, then doing more hip specific things afterwards.

However, @tra knows the clang code base and conventions rather better than I do, so I'm going to defer back.

[tra]

In D78759#2049750, @JonChesterfield wrote:

The control flow in tools::gnutools::StaticLibTool::ConstructJob doesn't seem good though. It's a generic function that unconditionally calls a hip specific function which happens to return immediately in non-hip cases. That really should be a hip specific function calling the generic one, then doing more hip specific things afterwards.

+1. Indeed it would be great to reuse existing tool either by making it more flexible, or via extracting common things into a helper.

However, @tra knows the clang code base and conventions rather better than I do, so I'm going to defer back.

I may be too used to the odd things we have to do to deal with GPUs. I've pinged @echristo for the second opinion.

[ashi1]

Thank you for your reviews, I've cleaned up the patch, and reduced the size of the StaticLib::ConstructJob.

[ashi1]

In D78759#2049973, @tra wrote:

In D78759#2049750, @JonChesterfield wrote:

The control flow in tools::gnutools::StaticLibTool::ConstructJob doesn't seem good though. It's a generic function that unconditionally calls a hip specific function which happens to return immediately in non-hip cases. That really should be a hip specific function calling the generic one, then doing more hip specific things afterwards.

+1. Indeed it would be great to reuse existing tool either by making it more flexible, or via extracting common things into a helper.

There might require a lot of changes to make this more flexible, and making a hip specific function call generic functions. I think it may be out of the scope for this patch.

[echristo]

I think I've got a lot of the same concerns that Jon had. In addition I'd like to see some reasoning and abstraction if we're going to use this for creating archives in general. No need to make windows work, but instead could just return a "we can't do that yet" error instead?

clang/lib/Driver/Driver.cpp

3546–3550

I'm really not a fan of this here. I think this needs to be abstracted a bit.

clang/lib/Driver/ToolChains/CommonArgs.cpp

1398 ↗

(On Diff #266302)

This needs comments.

1422 ↗

(On Diff #266302)

I'd really like to avoid a target specific assert here.

1427 ↗

(On Diff #259969)

You can refactor ahead of time to make this a bit more clear?

[ashi1]

In D78759#2055746, @echristo wrote:

I think I've got a lot of the same concerns that Jon had. In addition I'd like to see some reasoning and abstraction if we're going to use this for creating archives in general. No need to make windows work, but instead could just return a "we can't do that yet" error instead?

Hi Eric, this patch is to support users who want to create a static library on the HIP language and ROCm architecture. I will add some comments to the code to make it more clear.

clang/lib/Driver/Driver.cpp

3546–3550

Could you please provide some advice on how to make this more abstract? I am not familiar with this area of the code.

[ashi1]

I've refactored the code, and removed the AddHIPLinkerScript function, which can be replaced by these MC directives approach.

[ashi1]

ping - is this refactored version ready for submission? Thanks.

[yaxunl]

clang/lib/Driver/Driver.cpp

3546–3550

You may introduce a member function Driver::shouldEmitStaticLibrary() and probably change -hip-emit-static-lib to -emit-static-lib

[ashi1]

Thank you for the tips, I've updated the code with Driver::ShouldEmitStaticLibrary function, and renamed to flag to a more general name, --emit-static-lib.

[ashi1]

ping @echristo

[yaxunl]

@echristo Aaron has addressed most of the comments. The major remaining issue is that the general linker
calls AddGenerateObjFileFromHIPFatBinary, which is supposed to be part of HIP
toolchain. This issue is not new, since the existing gnutools::Linker::ConstructJob
already did the same thing. Is it OK to land this patch and leave the refactoring of
AddGenerateObjFileFromHIPFatBinary to a separate patch? Thanks.

[ashi1]

In D78759#2031780, @JonChesterfield wrote:

In D78759#2031741, @yaxunl wrote:

The use case of this patch is that the library only needs to provide host APIs which can be called by C++ programs but do not need to expose device code to the users. Therefore the device code in the library are linked, finalized, and embedded as ISA in host objects and not exposed to library users. We have user requests to support such use case.

There may be use cases that users want to store both host codes and device codes in a static library and expose both to users, however that is out of scope of this patch.

Ah, OK. So it's HIP specific, and not a general purpose static archive. In that case, the many functions with names based on StaticLib should probably all be renamed HIPStaticLib or similar to indicate that it doesn't work for other toolchains. Can any of this code be moved into the HIP.cpp toolchain file?

Hi @JonChesterfield and @echristo , I am opening a follow up commit that will move the linking logic into the HIP Toolchain rather than being in GNU.cpp. I will rebase this on top of master as well.

[ashi1]

Rebased on master, and added --emit-static-lib to hip-link-save-temps.hip testcase.

[ashi1]

Removed the HIP Linker changes, that will come in a different patch. Keeping this change more general.

[JonChesterfield]

This appears to have been committed without addressing all the comments or waiting for an acceptance from someone outside of our organisation. That doesn't seem right - am I missing part of the thread here?

[ashi1]

Hi Jon, I may have committed this in combination with D81963 as the parent commit. It should have waited for a LGTM from someone outside our org.

Please let me know unaddressed comments on this change, and I will fix them asap. Sorry about the confusion.

clang/include/clang/Driver/Action.h

enum ActionClass {
     IfsMergeJobClass,
     LipoJobClass,
     DsymutilJobClass,
     VerifyDebugInfoJobClass,
     VerifyPCHJobClass,
     OffloadBundlingJobClass,
     OffloadUnbundlingJobClass,
     OffloadWrapperJobClass,
+    StaticLibJobClass,
 
     JobClassFirst = PreprocessJobClass,
-    JobClassLast = OffloadWrapperJobClass
+    JobClassLast = StaticLibJobClass
   };
 
   // The offloading kind determines if this action is binded to a particular
   // programming model. Each entry reserves one bit. We also have a special kind
   // to designate the host offloading tool chain.
   enum OffloadKind {
     OFK_None = 0x00,
 

 public:
   OffloadWrapperJobAction(ActionList &Inputs, types::ID Type);
 
   static bool classof(const Action *A) {
     return A->getKind() == OffloadWrapperJobClass;
   }
 };
 
+class StaticLibJobAction : public JobAction {
+  void anchor() override;
+
+public:
+  StaticLibJobAction(ActionList &Inputs, types::ID Type);
+
+  static bool classof(const Action *A) {
+    return A->getKind() == StaticLibJobClass;
+  }
+};
+
 } // namespace driver
 } // namespace clang
 
 #endif // LLVM_CLANG_DRIVER_ACTION_H

clang/include/clang/Driver/Driver.h

public:
   /// ShouldUseClangCompiler - Should the clang compiler be used to
   /// handle this action.
   bool ShouldUseClangCompiler(const JobAction &JA) const;
 
   /// ShouldUseFlangCompiler - Should the flang compiler be used to
   /// handle this action.
   bool ShouldUseFlangCompiler(const JobAction &JA) const;
 
+  /// ShouldEmitStaticLibrary - Should the linker emit a static library.
+  bool ShouldEmitStaticLibrary(const llvm::opt::ArgList &Args) const;
+
   /// Returns true if we are performing any kind of LTO.
   bool isUsingLTO() const { return LTOMode != LTOK_None; }
 
   /// Get the specific kind of LTO being performed.
   LTOKind getLTOMode() const { return LTOMode; }
 
 private:
 

clang/include/clang/Driver/Options.td

def offload_arch_EQ : Joined<["--"], "offload-arch=">, Flags<[DriverOption]>,
   HelpText<"CUDA/HIP offloading device architecture (e.g. sm_35, gfx906).  May be specified more than once.">;
 def cuda_gpu_arch_EQ : Joined<["--"], "cuda-gpu-arch=">, Flags<[DriverOption]>,
   Alias<offload_arch_EQ>;
 def hip_link : Flag<["--"], "hip-link">,
   HelpText<"Link clang-offload-bundler bundles for HIP">;
 def no_offload_arch_EQ : Joined<["--"], "no-offload-arch=">, Flags<[DriverOption]>,
   HelpText<"Remove CUDA/HIP offloading device architecture (e.g. sm_35, gfx906) from the list of devices to compile for. "
            "'all' resets the list to its default value.">;
+def emit_static_lib : Flag<["--"], "emit-static-lib">,
+  HelpText<"Enable linker job to emit a static library.">;
 def no_cuda_gpu_arch_EQ : Joined<["--"], "no-cuda-gpu-arch=">, Flags<[DriverOption]>,
   Alias<no_offload_arch_EQ>;
 def cuda_noopt_device_debug : Flag<["--"], "cuda-noopt-device-debug">,
   HelpText<"Enable device-side debug info generation. Disables ptxas optimizations.">;
 def no_cuda_version_check : Flag<["--"], "no-cuda-version-check">,
   HelpText<"Don't error out if the detected version of the CUDA install is "
            "too low for the requested CUDA gpu architecture.">;
 def no_cuda_noopt_device_debug : Flag<["--"], "no-cuda-noopt-device-debug">;

clang/include/clang/Driver/ToolChain.h

private:
 
   /// The list of toolchain specific path prefixes to search for programs.
   path_list ProgramPaths;
 
   mutable std::unique_ptr<Tool> Clang;
   mutable std::unique_ptr<Tool> Flang;
   mutable std::unique_ptr<Tool> Assemble;
   mutable std::unique_ptr<Tool> Link;
+  mutable std::unique_ptr<Tool> StaticLibTool;
   mutable std::unique_ptr<Tool> IfsMerge;
   mutable std::unique_ptr<Tool> OffloadBundler;
   mutable std::unique_ptr<Tool> OffloadWrapper;
 
   Tool *getClang() const;
   Tool *getFlang() const;
   Tool *getAssemble() const;
   Tool *getLink() const;
+  Tool *getStaticLibTool() const;
   Tool *getIfsMerge() const;
   Tool *getClangAs() const;
   Tool *getOffloadBundler() const;
   Tool *getOffloadWrapper() const;
 
   mutable std::unique_ptr<SanitizerArgs> SanitizerArguments;
   mutable std::unique_ptr<XRayArgs> XRayArguments;
 
protected:
 
   ToolChain(const Driver &D, const llvm::Triple &T,
             const llvm::opt::ArgList &Args);
 
   void setTripleEnvironment(llvm::Triple::EnvironmentType Env);
 
   virtual Tool *buildAssembler() const;
   virtual Tool *buildLinker() const;
+  virtual Tool *buildStaticLibTool() const;
   virtual Tool *getTool(Action::ActionClass AC) const;
 
   /// \name Utilities for implementing subclasses.
   ///@{
   static void addSystemInclude(const llvm::opt::ArgList &DriverArgs,
                                llvm::opt::ArgStringList &CC1Args,
                                const Twine &Path);
   static void addExternCSystemInclude(const llvm::opt::ArgList &DriverArgs,
public:
 
   std::string GetFilePath(const char *Name) const;
   std::string GetProgramPath(const char *Name) const;
 
   /// Returns the linker path, respecting the -fuse-ld= argument to determine
   /// the linker suffix or name.
   std::string GetLinkerPath() const;
 
+  /// Returns the linker path for emitting a static library.
+  std::string GetStaticLibToolPath() const;
+
   /// Dispatch to the specific toolchain for verbose printing.
   ///
   /// This is used when handling the verbose option to print detailed,
   /// toolchain-specific information useful for understanding the behavior of
   /// the driver on a specific platform.
   virtual void printVerboseInfo(raw_ostream &OS) const {}
 
   // Platform defaults information

clang/lib/Driver/Action.cpp

const char *Action::getClassName(ActionClass AC) {
   case VerifyDebugInfoJobClass: return "verify-debug-info";
   case VerifyPCHJobClass: return "verify-pch";
   case OffloadBundlingJobClass:
     return "clang-offload-bundler";
   case OffloadUnbundlingJobClass:
     return "clang-offload-unbundler";
   case OffloadWrapperJobClass:
     return "clang-offload-wrapper";
+  case StaticLibJobClass:
+    return "static-lib-linker";
   }
 
   llvm_unreachable("invalid class");
 }
 
 void Action::propagateDeviceOffloadInfo(OffloadKind OKind, const char *OArch) {
   // Offload action set its own kinds on their dependences.
   if (Kind == OffloadClass)

 OffloadUnbundlingJobAction::OffloadUnbundlingJobAction(Action *Input)
     : JobAction(OffloadUnbundlingJobClass, Input, Input->getType()) {}
 
 void OffloadWrapperJobAction::anchor() {}
 
 OffloadWrapperJobAction::OffloadWrapperJobAction(ActionList &Inputs,
                                                  types::ID Type)
   : JobAction(OffloadWrapperJobClass, Inputs, Type) {}
+
+void StaticLibJobAction::anchor() {}
+
+StaticLibJobAction::StaticLibJobAction(ActionList &Inputs, types::ID Type)
+    : JobAction(StaticLibJobClass, Inputs, Type) {}

clang/lib/Driver/Driver.cpp

for (auto &I : Inputs) {
     // Add any top level actions generated for offloading.
     OffloadBuilder.appendTopLevelActions(Actions, Current, InputArg);
   }
 
   // Add a link action if necessary.
   if (!LinkerInputs.empty()) {
     if (Action *Wrapper = OffloadBuilder.makeHostLinkAction())
       LinkerInputs.push_back(Wrapper);
-    Action *LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
+    Action *LA;
+    // Check if this Linker Job should emit a static library.
+    if (ShouldEmitStaticLibrary(Args)) {
+      LA = C.MakeAction<StaticLibJobAction>(LinkerInputs, types::TY_Image);
+    } else {
+      LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
+    }
     LA = OffloadBuilder.processHostLinkAction(LA);
     Actions.push_back(LA);
   }
 
   // Add an interface stubs merge action if necessary.
   if (!MergerInputs.empty())
     Actions.push_back(
         C.MakeAction<IfsMergeJobAction>(MergerInputs, types::TY_Image));
bool Driver::ShouldUseFlangCompiler(const JobAction &JA) const {
 
   // And say "no" if this is not a kind of action flang understands.
   if (!isa<PreprocessJobAction>(JA) && !isa<CompileJobAction>(JA) && !isa<BackendJobAction>(JA))
     return false;
 
   return true;
 }
 
+bool Driver::ShouldEmitStaticLibrary(const ArgList &Args) const {
+  // Only emit static library if the flag is set explicitly.
+  if (Args.hasArg(options::OPT_emit_static_lib))
+    return true;
+  return false;
+}
+
 /// GetReleaseVersion - Parse (([0-9]+)(.([0-9]+)(.([0-9]+)?))?)? and return the
 /// grouped values as integers. Numbers which are not provided are set to 0.
 ///
 /// \return True if the entire string was parsed (9.2), or all groups were
 /// parsed (10.3.5extrastuff).
 bool Driver::GetReleaseVersion(StringRef Str, unsigned &Major, unsigned &Minor,
                                unsigned &Micro, bool &HadExtra) {
   HadExtra = false;

clang/lib/Driver/ToolChain.cpp

 Tool *ToolChain::buildAssembler() const {
   return new tools::ClangAs(*this);
 }
 
 Tool *ToolChain::buildLinker() const {
   llvm_unreachable("Linking is not supported by this toolchain");
 }
 
+Tool *ToolChain::buildStaticLibTool() const {
+  llvm_unreachable("Creating static lib is not supported by this toolchain");
+}
+
 Tool *ToolChain::getAssemble() const {
   if (!Assemble)
     Assemble.reset(buildAssembler());
   return Assemble.get();
 }
 
 Tool *ToolChain::getClangAs() const {
   if (!Assemble)
     Assemble.reset(new tools::ClangAs(*this));
   return Assemble.get();
 }
 
 Tool *ToolChain::getLink() const {
   if (!Link)
     Link.reset(buildLinker());
   return Link.get();
 }
 
+Tool *ToolChain::getStaticLibTool() const {
+  if (!StaticLibTool)
+    StaticLibTool.reset(buildStaticLibTool());
+  return StaticLibTool.get();
+}
+
 Tool *ToolChain::getIfsMerge() const {
   if (!IfsMerge)
     IfsMerge.reset(new tools::ifstool::Merger(*this));
   return IfsMerge.get();
 }
 
 Tool *ToolChain::getOffloadBundler() const {
   if (!OffloadBundler)
case Action::AssembleJobClass:
     return getAssemble();
 
   case Action::IfsMergeJobClass:
     return getIfsMerge();
 
   case Action::LinkJobClass:
     return getLink();
 
+  case Action::StaticLibJobClass:
+    return getStaticLibTool();
+
   case Action::InputClass:
   case Action::BindArchClass:
   case Action::OffloadClass:
   case Action::LipoJobClass:
   case Action::DsymutilJobClass:
   case Action::VerifyDebugInfoJobClass:
     llvm_unreachable("Invalid tool kind.");
 
std::string ToolChain::GetLinkerPath() const {
   }
 
   if (A)
     getDriver().Diag(diag::err_drv_invalid_linker_name) << A->getAsString(Args);
 
   return GetProgramPath(getDefaultLinker());
 }
 
+std::string ToolChain::GetStaticLibToolPath() const {
+  // TODO: Add support for static lib archiving on Windows
+  return GetProgramPath("llvm-ar");
+}
+
 types::ID ToolChain::LookupTypeForExtension(StringRef Ext) const {
   types::ID id = types::lookupTypeForExtension(Ext);
 
   // Flang always runs the preprocessor and has no notion of "preprocessed
   // fortran". Here, TY_PP_Fortran is coerced to TY_Fortran to avoid treating
   // them differently.
   if (D.IsFlangMode() && id == types::TY_PP_Fortran)
     id = types::TY_Fortran;

clang/lib/Driver/ToolChains/Gnu.h

public:
   bool hasIntegratedCPP() const override { return false; }
   bool isLinkJob() const override { return true; }
 
   void ConstructJob(Compilation &C, const JobAction &JA,
                     const InputInfo &Output, const InputInfoList &Inputs,
                     const llvm::opt::ArgList &TCArgs,
                     const char *LinkingOutput) const override;
 };
+
+class LLVM_LIBRARY_VISIBILITY StaticLibTool : public GnuTool {
+public:
+  StaticLibTool(const ToolChain &TC)
+      : GnuTool("GNU::StaticLibTool", "static-lib-linker", TC) {}
+
+  bool hasIntegratedCPP() const override { return false; }
+  bool isLinkJob() const override { return true; }
+
+  void ConstructJob(Compilation &C, const JobAction &JA,
+                    const InputInfo &Output, const InputInfoList &Inputs,
+                    const llvm::opt::ArgList &TCArgs,
+                    const char *LinkingOutput) const override;
+};
 } // end namespace gnutools
 
 /// gcc - Generic GCC tool implementations.
 namespace gcc {
 class LLVM_LIBRARY_VISIBILITY Common : public GnuTool {
 public:
   Common(const char *Name, const char *ShortName, const ToolChain &TC)
       : GnuTool(Name, ShortName, TC) {}

clang/lib/Driver/ToolChains/Gnu.cpp

static bool getStaticPIE(const ArgList &Args, const ToolChain &TC) {
   return HasStaticPIE;
 }
 
 static bool getStatic(const ArgList &Args) {
   return Args.hasArg(options::OPT_static) &&
       !Args.hasArg(options::OPT_static_pie);
 }
 
+void tools::gnutools::StaticLibTool::ConstructJob(
+    Compilation &C, const JobAction &JA, const InputInfo &Output,
+    const InputInfoList &Inputs, const ArgList &Args,
+    const char *LinkingOutput) const {
+  const Driver &D = getToolChain().getDriver();
+
+  // Silence warning for "clang -g foo.o -o foo"
+  Args.ClaimAllArgs(options::OPT_g_Group);
+  // and "clang -emit-llvm foo.o -o foo"
+  Args.ClaimAllArgs(options::OPT_emit_llvm);
+  // and for "clang -w foo.o -o foo". Other warning options are already
+  // handled somewhere else.
+  Args.ClaimAllArgs(options::OPT_w);
+  // Silence warnings when linking C code with a C++ '-stdlib' argument.
+  Args.ClaimAllArgs(options::OPT_stdlib_EQ);
+
+  // GNU ar tool command "ar <options> <output_file> <input_files>".
+  ArgStringList CmdArgs;
+  // Create and insert file members with a deterministic index.
+  CmdArgs.push_back("rcsD");
+  CmdArgs.push_back(Output.getFilename());
+  AddLinkerInputs(getToolChain(), Inputs, Args, CmdArgs, JA);
+
+  // Delete old output archive file if it already exists before generating a new
+  // archive file.
+  auto OutputFileName = Output.getFilename();
+  if (Output.isFilename() && llvm::sys::fs::exists(OutputFileName)) {
+    if (std::error_code EC = llvm::sys::fs::remove(OutputFileName)) {
+      D.Diag(diag::err_drv_unable_to_remove_file) << EC.message();
+      return;
+    }
+  }
+
+  const char *Exec = Args.MakeArgString(getToolChain().GetStaticLibToolPath());
+  C.addCommand(std::make_unique<Command>(JA, *this, Exec, CmdArgs, Inputs));
+}
+
 void tools::gnutools::Linker::ConstructJob(Compilation &C, const JobAction &JA,
                                            const InputInfo &Output,
                                            const InputInfoList &Inputs,
                                            const ArgList &Args,
                                            const char *LinkingOutput) const {
   // FIXME: The Linker class constructor takes a ToolChain and not a
   // Generic_ELF, so the static_cast might return a reference to a invalid
   // instance (see PR45061). Ideally, the Linker constructor needs to take a
   // Generic_ELF instead.
   const toolchains::Generic_ELF &ToolChain =
       static_cast<const toolchains::Generic_ELF &>(getToolChain());
   const Driver &D = ToolChain.getDriver();
 
   const llvm::Triple &Triple = getToolChain().getEffectiveTriple();
 
   const llvm::Triple::ArchType Arch = ToolChain.getArch();
   const bool isAndroid = ToolChain.getTriple().isAndroid();
   const bool IsIAMCU = ToolChain.getTriple().isOSIAMCU();
   const bool IsPIE = getPIE(Args, ToolChain);
   const bool IsStaticPIE = getStaticPIE(Args, ToolChain);

clang/lib/Driver/ToolChains/Linux.h

public:
 
   llvm::DenormalMode getDefaultDenormalModeForType(
       const llvm::opt::ArgList &DriverArgs, const JobAction &JA,
       const llvm::fltSemantics *FPType = nullptr) const override;
 
 protected:
   Tool *buildAssembler() const override;
   Tool *buildLinker() const override;
+  Tool *buildStaticLibTool() const override;
 
   std::string getMultiarchTriple(const Driver &D,
                                  const llvm::Triple &TargetTriple,
                                  StringRef SysRoot) const override;
 };
 
 } // end namespace toolchains
 } // end namespace driver
 } // end namespace clang
 
 #endif // LLVM_CLANG_LIB_DRIVER_TOOLCHAINS_LINUX_H

clang/lib/Driver/ToolChains/Linux.cpp

if (getTriple().isAndroid())
     return ToolChain::CST_Libcxx;
   return ToolChain::CST_Libstdcxx;
 }
 
 bool Linux::HasNativeLLVMSupport() const { return true; }
 
 Tool *Linux::buildLinker() const { return new tools::gnutools::Linker(*this); }
 
+Tool *Linux::buildStaticLibTool() const {
+  return new tools::gnutools::StaticLibTool(*this);
+}
+
 Tool *Linux::buildAssembler() const {
   return new tools::gnutools::Assembler(*this);
 }
 
 std::string Linux::computeSysRoot() const {
   if (!getDriver().SysRoot.empty())
     return getDriver().SysRoot;
 

clang/test/Driver/bindings.c

 // RUN: %clang -target powerpc-unknown-unknown -ccc-print-bindings %s -S 2>&1 | FileCheck %s --check-prefix=CHECK12
 // CHECK12: "clang", inputs: ["{{.*}}bindings.c"], output: "bindings.s"
 
 // Darwin bindings
 // RUN: %clang -target i386-apple-darwin9 -no-integrated-as -ccc-print-bindings %s 2>&1 | FileCheck %s --check-prefix=CHECK14
 // CHECK14: "clang", inputs: ["{{.*}}bindings.c"], output: "{{.*}}.s"
 // CHECK14: "darwin::Assembler", inputs: ["{{.*}}.s"], output: "{{.*}}.o"
 // CHECK14: "darwin::Linker", inputs: ["{{.*}}.o"], output: "a.out"
+
+// GNU StaticLibTool binding
+// RUN: %clang -target x86_64-linux-gnu -ccc-print-bindings --emit-static-lib %s 2>&1 | FileCheck %s --check-prefix=CHECK15
+// CHECK15: "x86_64-unknown-linux-gnu" - "GNU::StaticLibTool", inputs: ["{{.*}}.o"], output: "a.out"

clang/test/Driver/hip-link-save-temps.hip

 // -fgpu-rdc link without output
 // RUN: touch %T/obj1.o
 // RUN: touch %T/obj2.o
 // RUN: %clang -### -target x86_64-linux-gnu -nogpulib -save-temps \
 // RUN:   --hip-link -fgpu-rdc --cuda-gpu-arch=gfx900 \
 // RUN:   --offload-arch=gfx906  %T/obj1.o %T/obj2.o 2>&1 | \
 // RUN:   FileCheck -check-prefixes=CHECK,NOUT %s
 
+// -fgpu-rdc link with output and --emit-static-lib
+// RUN: touch %T/obj1.o
+// RUN: touch %T/obj2.o
+// RUN: %clang -### -target x86_64-linux-gnu -nogpulib -save-temps \
+// RUN:   --hip-link -o libTest.a -fgpu-rdc --cuda-gpu-arch=gfx900 \
+// RUN:   --emit-static-lib \
+// RUN:   --offload-arch=gfx906  %T/obj1.o %T/obj2.o 2>&1 | \
+// RUN:   FileCheck -check-prefixes=CHECK,SLO %s
+
+// -fgpu-rdc link without output and --emit-static-lib
+// RUN: touch %T/obj1.o
+// RUN: touch %T/obj2.o
+// RUN: %clang -### -target x86_64-linux-gnu -nogpulib -save-temps \
+// RUN:   --hip-link -fgpu-rdc --cuda-gpu-arch=gfx900 \
+// RUN:   --emit-static-lib \
+// RUN:   --offload-arch=gfx906  %T/obj1.o %T/obj2.o 2>&1 | \
+// RUN:   FileCheck -check-prefixes=CHECK,SLNO %s
+
 // CHECK: "{{.*clang-offload-bundler.*}}" {{.*}} "-outputs=obj1-host-x86_64-unknown-linux-gnu.o,obj1-hip-amdgcn-amd-amdhsa-gfx900.o,obj1-hip-amdgcn-amd-amdhsa-gfx906.o" "-unbundle"
 // CHECK: "{{.*clang-offload-bundler.*}}" {{.*}} "-outputs=obj2-host-x86_64-unknown-linux-gnu.o,obj2-hip-amdgcn-amd-amdhsa-gfx900.o,obj2-hip-amdgcn-amd-amdhsa-gfx906.o" "-unbundle"
 // CHECK-NOT: llvm-link
 // CHECK-NOT: opt
 // CHECK-NOT: llc
 // CHECK: "{{.*lld.*}}" {{.*}} "-mllvm" "-amdgpu-internalize-symbols"
 // CHECK-SAME: "-o" "a.out-hip-amdgcn-amd-amdhsa-gfx900" "obj1-hip-amdgcn-amd-amdhsa-gfx900.o" "obj2-hip-amdgcn-amd-amdhsa-gfx900.o"
 // CHECK: "{{.*lld.*}}" {{.*}} "-mllvm" "-amdgpu-internalize-symbols"
 // CHECK-SAME: "-o" "a.out-hip-amdgcn-amd-amdhsa-gfx906" "obj1-hip-amdgcn-amd-amdhsa-gfx906.o" "obj2-hip-amdgcn-amd-amdhsa-gfx906.o"
 // CHECK: {{".*llvm-mc.*"}} "-triple" "amdgcn-amd-amdhsa" "-o"
 // CHECK-SAME: "[[OBJBUNDLE:.*.o]]" "{{.*}}.mcin" "--filetype=obj"
 // OUT: "{{.*ld.*}}" {{.*}} "-o" "executable" {{.*}} "[[OBJBUNDLE]]"
 // NOUT: "{{.*ld.*}}" {{.*}} "-o" "a.out" {{.*}} "[[OBJBUNDLE]]"
+// SLO: "{{.*llvm-ar.*}}" "rcsD" "libTest.a" {{.*}} "[[OBJBUNDLE]]"
+// SLNO: "{{.*llvm-ar.*}}" "rcsD" "a.out" {{.*}} "[[OBJBUNDLE]]"

clang/test/Driver/hip-link-static-library.hip

+// REQUIRES: clang-driver
+// REQUIRES: x86-registered-target
+// REQUIRES: amdgpu-registered-target
+
+// RUN: touch %t.o
+// RUN: %clang --hip-link -ccc-print-bindings -target x86_64-linux-gnu \
+// RUN:   --emit-static-lib \
+// RUN:   --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 -fgpu-rdc %t.o\
+// RUN: 2>&1 | FileCheck %s
+
+// CHECK: # "x86_64-unknown-linux-gnu" - "offload bundler", inputs: ["[[IN:.*o]]"], outputs: ["[[HOSTOBJ:.*o]]", "{{.*o}}", "{{.*o}}"]
+// CHECK: # "amdgcn-amd-amdhsa" - "offload bundler", inputs: ["[[IN]]"], outputs: ["{{.*o}}", "[[DOBJ1:.*o]]", "[[DOBJ2:.*o]]"]
+// CHECK: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[DOBJ1]]"], output: "[[IMG1:.*out]]"
+// CHECK-NOT: offload bundler
+// CHECK: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[DOBJ2]]"], output: "[[IMG2:.*out]]"
+// CHECK-NOT: offload bundler
+// CHECK: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[IMG1]]", "[[IMG2]]"], output: "[[FATBINOBJ:.*o]]"
+// CHECK-NOT: offload bundler
+// CHECK: # "x86_64-unknown-linux-gnu" - "GNU::StaticLibTool", inputs: ["[[HOSTOBJ]]", "[[FATBINOBJ]]"], output: "a.out"
+
+// RUN: %clang --hip-link -ccc-print-bindings -target x86_64-linux-gnu \
+// RUN:   --emit-static-lib \
+// RUN:   --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 %t.o\
+// RUN: 2>&1 | FileCheck -check-prefix=NORDC %s
+
+// NORDC-NOT: offload bundler
+// NORDC: # "x86_64-unknown-linux-gnu" - "GNU::StaticLibTool", inputs: ["{{.*o}}"], output: "a.out"

clang/test/Driver/hip-toolchain-rdc-static-lib.hip

+// REQUIRES: clang-driver
+// REQUIRES: x86-registered-target
+// REQUIRES: amdgpu-registered-target
+
+// RUN: %clang -### -target x86_64-linux-gnu \
+// RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
+// RUN:   --emit-static-lib -nogpulib \
+// RUN:   -fuse-ld=lld -fgpu-rdc -nogpuinc \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
+// RUN: 2>&1 | FileCheck %s
+
+// emit objects for host side path
+// CHECK: [[CLANG:".*clang.*"]] "-cc1" "-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-aux-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-emit-obj"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu"
+// CHECK-SAME: {{.*}} "-o" [[A_OBJ_HOST:".*o"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC:".*a.cu"]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-aux-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-emit-obj"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip"
+// CHECK-SAME: {{.*}} "-o" [[B_OBJ_HOST:".*o"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC:".*b.hip"]]
+
+// generate image for device side path on gfx803
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu"
+// CHECK-SAME: "-fcuda-is-device" "-fgpu-rdc"
+// CHECK-SAME: "-target-cpu" "gfx803"
+// CHECK-SAME: {{.*}} "-o" [[A_BC1:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip"
+// CHECK-SAME: "-fcuda-is-device" "-fgpu-rdc"
+// CHECK-SAME: "-target-cpu" "gfx803"
+// CHECK-SAME: {{.*}} "-o" [[B_BC1:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC]]
+
+// CHECK-NOT: "*.llvm-link"
+// CHECK-NOT: ".*opt"
+// CHECK-NOT: ".*llc"
+// CHECK: [[LLD: ".*lld"]] {{.*}} "-o" "[[IMG_DEV1:.*out]]" [[A_BC1]] [[B_BC1]]
+
+// generate image for device side path on gfx900
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu"
+// CHECK-SAME: "-fcuda-is-device" "-fgpu-rdc"
+// CHECK-SAME: "-target-cpu" "gfx900"
+// CHECK-SAME: {{.*}} "-o" [[A_BC2:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64-unknown-linux-gnu"
+// CHECK-SAME: "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip"
+// CHECK-SAME: "-fcuda-is-device" "-fgpu-rdc"
+// CHECK-SAME: "-target-cpu" "gfx900"
+// CHECK-SAME: {{.*}} "-o" [[B_BC2:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC]]
+
+// CHECK-NOT: "*.llvm-link"
+// CHECK-NOT: ".*opt"
+// CHECK-NOT: ".*llc"
+// CHECK: [[LLD]] {{.*}} "-o" "[[IMG_DEV2:.*out]]" [[A_BC2]] [[B_BC2]]
+
+// combine images generated into hip fat binary object
+// CHECK: [[BUNDLER:".*clang-offload-bundler"]] "-type=o"
+// CHECK-SAME: "-targets={{.*}},hip-amdgcn-amd-amdhsa-gfx803,hip-amdgcn-amd-amdhsa-gfx900"
+// CHECK-SAME: "-inputs={{.*}},[[IMG_DEV1]],[[IMG_DEV2]]" "-outputs=[[BUNDLE:.*hipfb]]"
+
+// CHECK: [[MC:".*llvm-mc"]] "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-o" [[OBJBUNDLE:".*o"]] "{{.*}}.mcin" "--filetype=obj"
+
+// CHECK: [[AR:".*llvm-ar.*"]] "rcsD" "{{.*}}.out" [[A_OBJ_HOST]] [[B_OBJ_HOST]] [[OBJBUNDLE]]