Add HIP toolchain

[yaxunl]

Summary

This patch adds HIP toolchain to support HIP language mode. It includes:

Create specific compiler jobs for HIP.

Choose specific libraries for HIP.

With contribution from Greg Rodgers.

[yaxunl]

Fixed typo which causes lit test failures. Now all lit tests pass.

[rjmccall]

The other changes I see here seem reasonable, but please do split the patch.

include/clang/Basic/Cuda.h

61 ↗

(On Diff #140827)

Does this actually have anything to do with HIP? You have a lot of changes in this patch which seem to just be about supporting more GPU revisions.

include/clang/Driver/Options.td

564

Do we absolutely need the non-CUDA-related aliases here? We generally try to be good about namespacing extension-specific language options.

I understand that you're probably trying to maintain command-line compatibility with some existing toolchain, but if it's possible to avoid this, I would be much happier.

include/clang/Driver/ToolChain.h

124 ↗

(On Diff #140827)

"Backend" is a really generic name for this thing that is probably hyper-specific to the CUDA translation model.

[yaxunl]

include/clang/Basic/Cuda.h

61 ↗

(On Diff #140827)

This patch not only adds support of HIP language mode but also adds support of amdgpu to CUDA toolchain.

Currently HIP extension is only supported by amdgpu although in the future it may be supported by other targets.

include/clang/Driver/Options.td

564

There were discussions about a uniform clang option for offloading sub-arcs

http://lists.llvm.org/pipermail/llvm-dev/2017-February/109896.html

the consensus seem to be -offload-arch or -offload-archs.

This patch attempts to make that transition to use this new option.

We can separate this change to a different patch though.

include/clang/Driver/ToolChain.h

124 ↗

(On Diff #140827)

Agree. This tool actually links bunch of bitcode libraries (so called device libraries). For non-GPU targets, this is usually unnecessary since they support ISA-level linking. However most GPU targets do not support that, therefore they need this stage.

How about renaming it as BitcodeLink?

[rjmccall]

include/clang/Basic/Cuda.h

61 ↗

(On Diff #140827)

I understand that, but I think you can separate those two patches without too much difficulty.

include/clang/Driver/Options.td

564

I don't mind the -offload-* options; I'm more concerned about --host-only and so on.

include/clang/Driver/ToolChain.h

124 ↗

(On Diff #140827)

DeviceLibraryLink, maybe? I wouldn't want someone to think this was related to ordinary LTO.

[Hahnfeld]

include/clang/Basic/Cuda.h

61 ↗

(On Diff #140827)

There already is D42800 with lots of unanswered comments, so those need to be addressed first.

[yaxunl]

include/clang/Basic/Cuda.h

61 ↗

(On Diff #140827)

Will split the changes for supporting amdgcn to another review.

61 ↗

(On Diff #140827)

Will address those comments in the split review.

include/clang/Driver/Options.td

564

Will drop those options for now. Since HIP is just a language mode of CUDA, it is OK to use CUDA options e.g. 'cuda-device-only' for HIP.

include/clang/Driver/ToolChain.h

124 ↗

(On Diff #140827)

That sounds good. Will change to that.

[yaxunl]

Revised by reviewers' comments, including comments from previous review.

[rjmccall]

This looks okay to me, but I think it would better if someone with more expertise in the design of the driver and frontend code could review this.

lib/Driver/Driver.cpp

2204

The extra parens are weird here.

[Hahnfeld]

Can this revision be split further? The summary mentions many things that might make up multiple independent changes...

lib/Driver/ToolChains/Cuda.cpp

263 ↗

(On Diff #141221)

Will this override the user's value, e.g. -std=c++14?

[yaxunl]

lib/Driver/Driver.cpp

2204

will remove it.

lib/Driver/ToolChains/Cuda.cpp

263 ↗

(On Diff #141221)

No. We will add diagnotics in a separate patch.

[yaxunl]

Separate file type changes to another patch.

[yaxunl]

In D45212#1060628, @Hahnfeld wrote:

Can this revision be split further? The summary mentions many things that might make up multiple independent changes...

I separated file type changes to https://reviews.llvm.org/D45489 and deferred some other changes for future.

It is kind of difficult to split this patch further since they depend on each other.

[Hahnfeld]

In D45212#1063186, @yaxunl wrote:

In D45212#1060628, @Hahnfeld wrote:

Can this revision be split further? The summary mentions many things that might make up multiple independent changes...

I separated file type changes to https://reviews.llvm.org/D45489 and deferred some other changes for future.

It is kind of difficult to split this patch further since they depend on each other.

You can add Related Revisions to make this easy to see.

[rjmccall]

I'd still prefer if someone with more driver-design expertise weighed in, but we might not have any specialists there.

LGTM, although you might consider changing your tests a bit: FileCheck recently added support for a -D argument where you can predefine variables in the command line. But that's just a suggestion, not something I'm asking you to do as part of review.

[Hahnfeld]

In D45212#1066842, @rjmccall wrote:

I'd still prefer if someone with more driver-design expertise weighed in, but we might not have any specialists there.

I think you should at least give @tra the possibility to take a look. Last time we essentially ended up reverting a patch.

[tra]

Sorry about the delay. I've been out for few days.

include/clang/Driver/Options.td

563–564

The discussion you've mentioned appears to be unfinished. @jlebar has raised a valid point regarding -no-offload-arch[s] and his email went unanswered AFAICT.

If you propose to generalize a way to specify offload targets, it should probably be done in a separate patch.

I'd just stick with --cuda-gpu-arch for the time being until we figure out how we're going to handle target specification in general. It works well enough for the moment and the new options are not required for the functionality implemented by this patch.

lib/Driver/Driver.cpp

2289–2297

You really want to either have your own error message or change existing one to something more generic. unsupported use of NVPTX for host compilation. will sound very confusing tof someone who tries to compile for AMD GPU.

3462

Would it make sense for HIP to have its own offloading kind? You seem to be adding similar checks in few other places.

3607–3609

llvm::sys::path::extension ?

4035–4042

I'm not sure why we do this here. Nor does it seem relevant to HIP.

4107–4108

Same here -- I'm not sure why we need this nor how it's related to HIP.

lib/Driver/ToolChains/Cuda.cpp

352 ↗

(On Diff #141863)

L*L*C_OUTPUT ?

356 ↗

(On Diff #141863)

Please use routines in`llvm::sys::path` here and in other places where you manipulate paths.

360–363 ↗

(On Diff #141863)

You could use .append({...})

CmdArgs2.append({"foo","bar","buz"});

436–439 ↗

(On Diff #141863)

This is rather awkward -- you're looking for /libdevice under all paths specified by -L, but there's no way to explicitly point to the directory with the bitcode library. If device library may be in a non-canonical location, I'd rather there was an explicit option to specify it. Furthermore, my understanding is that you will need to find these bitcode libraries during -c compilation. Using -L to derive bitcode search path during compilation looks wrong to me.

445 ↗

(On Diff #141863)

This (and other places where you're calculating libdevice path relative to driver dir) should probably be derived from the resource dir. Driver's path may not be the 'root' the compiler has been told to use.

447 ↗

(On Diff #141863)

LIBRARY_PATH sounds rather generic. Perhaps it should have HIP somewhere in its name.

462 ↗

(On Diff #141863)

Why do we need to silence the warnings?

498–501 ↗

(On Diff #141863)

This does not look like the right place to disable particular passes. Shouldn't it be done somewhere in LLVM?

[yaxunl]

include/clang/Driver/Options.td

563–564

Will use --cuda-gpu-arch for this patch.

lib/Driver/Driver.cpp

2289–2297

Will generalise this error message.

3462

Yes it makes sense to let HIP have its own offloading kind. Will do.

3607–3609

will do

4035–4042

will remove it

4107–4108

will remove

lib/Driver/ToolChains/Cuda.cpp

352 ↗

(On Diff #141863)

will change to LLC_OUTPUT

356 ↗

(On Diff #141863)

will do

360–363 ↗

(On Diff #141863)

will do

436–439 ↗

(On Diff #141863)

Will use --hip-device-lib-path= and drop /libdevice.

445 ↗

(On Diff #141863)

will remove this

447 ↗

(On Diff #141863)

will use HIP_DEVICE_LIB_PATH

462 ↗

(On Diff #141863)

will remove this

[yaxunl]

lib/Driver/ToolChains/Cuda.cpp

498–501 ↗

(On Diff #141863)

These are not disabling the passes but adding these passes. They are some optimizations which are usually improving performance for amdgcn.

[yaxunl]

Revised by Artem's comments.

[yaxunl]

sync to ToT.

[yaxunl]

minor bug fix: do not add CUDA specific link options for HIP.

[yaxunl]

lib/Driver/ToolChains/Cuda.cpp

498–501 ↗

(On Diff #141863)

Since opt is now able to adjust passes based on -mtriple and -mcpu, will remove these manually added passes and add -mtriple and -mcpu instead.

[yaxunl]

Remove manually added passes from opt and add -mtriple.

[yaxunl]

ping

[yaxunl]

clean up code and separate action builder to another review.

[yaxunl]

ping

[yaxunl]

Hi Artem,

I've addressed your comments. Any further changes are needed? Thanks.

[tra]

Hi,

Sorry about the long silence. I'm back to continue the reviews. I'll handle what I can today and will continue with the rest on Tuesday.

It looks like patch description needs to be updated:

Use clang-offload-bindler to create binary for device ISA.

I don't see anything related to offload-bundler in this patch any more.

include/clang/Driver/Options.td

588

I'm not sure about i_Group? This will cause this option to be passed to all preprocessor jobs. It will also be passed to host and device side compilations, while you probably only want/need it on device side only.

lib/Driver/ToolChains/Cuda.cpp

323 ↗

(On Diff #145297)

FullName is already result of Args.MakeArgString. You only need to do it once.

329 ↗

(On Diff #145297)

This function is too large to easily see that we're actually constructing sequence of commands.
I'd probably split construction of individual tool's command line into its own function.

336 ↗

(On Diff #145297)

No need for the leading space in the message.

344–345 ↗

(On Diff #145297)

for (const InputInfo &it : Inputs) ?

350 ↗

(On Diff #145297)

All-caps name looks like a macro. Rename to GfxName ?

354–359 ↗

(On Diff #145297)

for (path : Args.getAllArgValues(...)) {
   LibraryPaths.push_back(Args.MakeArgString(path));
}

375–378 ↗

(On Diff #145297)

This is somewhat unreadable. Perhaps you could construct the name in a temp variable.

384 ↗

(On Diff #145297)

You don't need to use c_str() for MakeArgString. It will happily accept std::string.

394 ↗

(On Diff #145297)

BitcodeOutputFile?

417 ↗

(On Diff #145297)

I think you can get rid of the temp var here without hurting readability.

420 ↗

(On Diff #145297)

I wonder if we could derive temp file name from the input's name. This may make it easier to find relevant temp files if/when we need to debug the compilation process.

422 ↗

(On Diff #145297)

No need for c_str() here.

439 ↗

(On Diff #145297)

c_str(), again.

764 ↗

(On Diff #145297)

I'd just add something like this and leave existing if unchanged:

// There's no need for CUDA-specific bitcode linking with HIP.
if( DeviceOffloadingKind == Action::OFK_HIP)
  return;

926–938 ↗

(On Diff #145297)

All these should be in the derived toolchain.

lib/Driver/ToolChains/Cuda.h

137–138 ↗

(On Diff #145297)

Where does amdgcn-link come from? Does it accept --options-file ?

153–154 ↗

(On Diff #145297)

You may want to derive your own HIPToolChain as you're unlikely to reuse NVPTX-specific tools.

172 ↗

(On Diff #145297)

In HIPToolchain this one would become an inline function returning true.

[tra]

One more thing -- it would be really good to add some tests to make sure your commands are constructed the way you want.

[yaxunl]

In D45212#1105177, @tra wrote:

Hi,

Sorry about the long silence. I'm back to continue the reviews. I'll handle what I can today and will continue with the rest on Tuesday.

It looks like patch description needs to be updated:

Use clang-offload-bindler to create binary for device ISA.

I don't see anything related to offload-bundler in this patch any more.

You are right. Using clang-offload-bundler to create binary for device ISA has been moved to another patch. Will update the description of this patch.

In D45212#1105282, @tra wrote:

One more thing -- it would be really good to add some tests to make sure your commands are constructed the way you want.

will do

include/clang/Driver/Options.td

588

will change to Link_Group

lib/Driver/ToolChains/Cuda.cpp

323 ↗

(On Diff #145297)

will fix

329 ↗

(On Diff #145297)

will do

336 ↗

(On Diff #145297)

will fix.

344–345 ↗

(On Diff #145297)

will fix

350 ↗

(On Diff #145297)

will fix

354–359 ↗

(On Diff #145297)

will fix

375–378 ↗

(On Diff #145297)

will do

384 ↗

(On Diff #145297)

will fix

394 ↗

(On Diff #145297)

will change

417 ↗

(On Diff #145297)

will do

420 ↗

(On Diff #145297)

will do

422 ↗

(On Diff #145297)

will do

439 ↗

(On Diff #145297)

will fix

764 ↗

(On Diff #145297)

will change

926–938 ↗

(On Diff #145297)

will do

lib/Driver/ToolChains/Cuda.h

137–138 ↗

(On Diff #145297)

amdgcn-link is the short name of the amdgcn bitcode linker. It is not a real program and does not support response file. Will remove the arguments about response file.

153–154 ↗

(On Diff #145297)

Will do

172 ↗

(On Diff #145297)

will do

[yaxunl]

Revised by Artem's comments.

[tra]

lib/Driver/ToolChains/HIP.cpp

30–48

FullName may remain uninitialized if LibraryPaths are empty which will probably crash compiler when you attempt to pass it to MakeArgString.
If empty LibraryPaths is not expected there should be an assert.

If the library is not found, we issue an error, but we still proceed to append the FullName to the CmdArgs. I don't think we should do that. FullName will be either NULL or pointing to the last directory in the LibraryPaths.

You seem to be relying on diagnostics to deal with errors and are not using return value of the function. You may as well make it void.

I'd move CmdArgs.push_back(...) under if(::exists(FullName)) and change break to return;
Then you can get rid of FoundLibDevice and just issue the error if we ever reach the end of the function.

80–82

No need for intermediate values here -- just '+' all parts together.

134

Nit: I think explicit llvm::Twine is unnecessary here.

156–161

Nit: THis could be collapsed into ArgStringList LlcArgs({...});

180–182

Same here: ArgStringList LldArgs({"-flavor", "gnu", "--no-undefined", "-shared", "-o"});

213–216

Right now the code is structured as if you're appending to the same TempFile string which is not the case here. I'd give intermediate variables their own names -- OptCommand,LlcCommand.
This would make it easier to see that you are chaining separate commands, each producing its own temp output file.

[yaxunl]

lib/Driver/ToolChains/HIP.cpp

30–48

Will CmdArgs.push_back(...) under if(::exists(FullName)) and change break to return; and change return type to void.

80–82

will do

134

will remove

156–161

will do

180–182

will do

213–216

will do

[yaxunl]

Revised by Artem's comments.

[tra]

One small nit. LGTM otherwise.

lib/Driver/ToolChains/HIP.cpp

45

You don't need FoundLibDevice any more as you will always return from inside the loop if it is ever true.

[yaxunl]

lib/Driver/ToolChains/HIP.cpp

45

will remove when committing. Thanks!

include/clang/Driver/Options.td

HelpText<"Compile CUDA code for both host and device (default).  Has no "
            "effect on non-CUDA compilations.">;
 def cuda_include_ptx_EQ : Joined<["--"], "cuda-include-ptx=">, Flags<[DriverOption]>,
   HelpText<"Include PTX for the follwing GPU architecture (e.g. sm_35) or 'all'. May be specified more than once.">;
 def no_cuda_include_ptx_EQ : Joined<["--"], "no-cuda-include-ptx=">, Flags<[DriverOption]>,
   HelpText<"Do not include PTX for the follwing GPU architecture (e.g. sm_35) or 'all'. May be specified more than once.">;
 def cuda_gpu_arch_EQ : Joined<["--"], "cuda-gpu-arch=">, Flags<[DriverOption]>,
   HelpText<"CUDA GPU architecture (e.g. sm_35).  May be specified more than once.">;
 def hip_link : Flag<["--"], "hip-link">,
   HelpText<"Link clang-offload-bundler bundles for HIP">;
 def no_cuda_gpu_arch_EQ : Joined<["--"], "no-cuda-gpu-arch=">, Flags<[DriverOption]>,
   HelpText<"Remove GPU architecture (e.g. sm_35) from the list of GPUs to compile for. "
            "'all' resets the list to its default value.">;
 def cuda_noopt_device_debug : Flag<["--"], "cuda-noopt-device-debug">,
   HelpText<"Enable device-side debug info generation. Disables ptxas optimizations.">;
 def no_cuda_version_check : Flag<["--"], "no-cuda-version-check">,
   HelpText<"Don't error out if the detected version of the CUDA install is "
            "too low for the requested CUDA gpu architecture.">;
 def no_cuda_noopt_device_debug : Flag<["--"], "no-cuda-noopt-device-debug">;
 def cuda_path_EQ : Joined<["--"], "cuda-path=">, Group<i_Group>,
   HelpText<"CUDA installation path">;
 def cuda_path_ignore_env : Flag<["--"], "cuda-path-ignore-env">, Group<i_Group>,
   HelpText<"Ignore environment variables to detect CUDA installation">;
 def ptxas_path_EQ : Joined<["--"], "ptxas-path=">, Group<i_Group>,
   HelpText<"Path to ptxas (used for compiling CUDA code)">;
 def fcuda_flush_denormals_to_zero : Flag<["-"], "fcuda-flush-denormals-to-zero">,
   Flags<[CC1Option]>, HelpText<"Flush denormal floating point values to zero in CUDA device mode.">;
 def fno_cuda_flush_denormals_to_zero : Flag<["-"], "fno-cuda-flush-denormals-to-zero">;
 def fcuda_approx_transcendentals : Flag<["-"], "fcuda-approx-transcendentals">,
   Flags<[CC1Option]>, HelpText<"Use approximate transcendental functions">;
 def fno_cuda_approx_transcendentals : Flag<["-"], "fno-cuda-approx-transcendentals">;
 def fcuda_rdc : Flag<["-"], "fcuda-rdc">, Flags<[CC1Option]>,
   HelpText<"Generate relocatable device code, also known as separate compilation mode.">;
 def fno_cuda_rdc : Flag<["-"], "fno-cuda-rdc">;
 def fcuda_short_ptr : Flag<["-"], "fcuda-short-ptr">, Flags<[CC1Option]>,
   HelpText<"Use 32-bit pointers for accessing const/local/shared address spaces.">;
 def fno_cuda_short_ptr : Flag<["-"], "fno-cuda-short-ptr">;
+def hip_device_lib_path_EQ : Joined<["--"], "hip-device-lib-path=">, Group<Link_Group>,
+  HelpText<"HIP device library path">;
+def hip_device_lib_EQ : Joined<["--"], "hip-device-lib=">, Group<Link_Group>,
+  HelpText<"HIP device library">;
 def fhip_dump_offload_linker_script : Flag<["-"], "fhip-dump-offload-linker-script">,
   Group<f_Group>, Flags<[NoArgumentUnused, HelpHidden]>;
 def dA : Flag<["-"], "dA">, Group<d_Group>;
 def dD : Flag<["-"], "dD">, Group<d_Group>, Flags<[CC1Option]>,
   HelpText<"Print macro definitions in -E mode in addition to normal output">;
 def dI : Flag<["-"], "dI">, Group<d_Group>, Flags<[CC1Option]>,
   HelpText<"Print include directives in -E mode in addition to normal output">;
 def dM : Flag<["-"], "dM">, Group<d_Group>, Flags<[CC1Option]>,

lib/Driver/CMakeLists.txt

add_clang_library(clangDriver
   ToolChains/CrossWindows.cpp
   ToolChains/Cuda.cpp
   ToolChains/Darwin.cpp
   ToolChains/DragonFly.cpp
   ToolChains/FreeBSD.cpp
   ToolChains/Fuchsia.cpp
   ToolChains/Gnu.cpp
   ToolChains/Haiku.cpp
+  ToolChains/HIP.cpp
   ToolChains/Hexagon.cpp
   ToolChains/Linux.cpp
   ToolChains/MipsLinux.cpp
   ToolChains/MinGW.cpp
   ToolChains/Minix.cpp
   ToolChains/MSVC.cpp
   ToolChains/Myriad.cpp
   ToolChains/NaCl.cpp

lib/Driver/Driver.cpp

 //===--- Driver.cpp - Clang GCC Compatible Driver -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Driver/Driver.h"
 #include "InputInfo.h"
 #include "ToolChains/AMDGPU.h"
 #include "ToolChains/AVR.h"
 #include "ToolChains/Ananas.h"
+#include "ToolChains/BareMetal.h"
 #include "ToolChains/Clang.h"
 #include "ToolChains/CloudABI.h"
 #include "ToolChains/Contiki.h"
 #include "ToolChains/CrossWindows.h"
 #include "ToolChains/Cuda.h"
 #include "ToolChains/Darwin.h"
 #include "ToolChains/DragonFly.h"
 #include "ToolChains/FreeBSD.h"
 #include "ToolChains/Fuchsia.h"
 #include "ToolChains/Gnu.h"
-#include "ToolChains/BareMetal.h"
+#include "ToolChains/HIP.h"
 #include "ToolChains/Haiku.h"
 #include "ToolChains/Hexagon.h"
 #include "ToolChains/Lanai.h"
 #include "ToolChains/Linux.h"
+#include "ToolChains/MSVC.h"
 #include "ToolChains/MinGW.h"
 #include "ToolChains/Minix.h"
 #include "ToolChains/MipsLinux.h"
-#include "ToolChains/MSVC.h"
 #include "ToolChains/Myriad.h"
 #include "ToolChains/NaCl.h"
 #include "ToolChains/NetBSD.h"
 #include "ToolChains/OpenBSD.h"
 #include "ToolChains/PS4CPU.h"
 #include "ToolChains/Solaris.h"
 #include "ToolChains/TCE.h"
 #include "ToolChains/WebAssembly.h"

 #include "llvm/Option/Option.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
+#include "llvm/Support/StringSaver.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/raw_ostream.h"
-#include "llvm/Support/StringSaver.h"
 #include <map>
 #include <memory>
 #include <utility>
 #if LLVM_ON_UNIX
 #include <unistd.h> // getpid
 #endif
 
 using namespace clang::driver;

 }
 
 void Driver::CreateOffloadingDeviceToolChains(Compilation &C,
                                               InputList &Inputs) {
 
   //
   // CUDA/HIP
   //
-  // We need to generate a CUDA toolchain if any of the inputs has a CUDA
+  // We need to generate a CUDA/HIP toolchain if any of the inputs has a CUDA
   // or HIP type. However, mixed CUDA/HIP compilation is not supported.
   bool IsCuda =
       llvm::any_of(Inputs, [](std::pair<types::ID, const llvm::opt::Arg *> &I) {
         return types::isCuda(I.first);
       });
   bool IsHIP =
       llvm::any_of(Inputs,
                    [](std::pair<types::ID, const llvm::opt::Arg *> &I) {
                      return types::isHIP(I.first);
                    }) ||
       C.getInputArgs().hasArg(options::OPT_hip_link);
   if (IsCuda && IsHIP) {
     Diag(clang::diag::err_drv_mix_cuda_hip);
     return;
   }
-  if (IsCuda || IsHIP) {
+  if (IsCuda) {
     const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
     const llvm::Triple &HostTriple = HostTC->getTriple();
     StringRef DeviceTripleStr;
-    auto OFK = IsHIP ? Action::OFK_HIP : Action::OFK_Cuda;
-    if (IsHIP) {
-      // HIP is only supported on amdgcn.
-      DeviceTripleStr = "amdgcn-amd-amdhsa";
-    } else {
-      // CUDA is only supported on nvptx.
-      DeviceTripleStr = HostTriple.isArch64Bit() ? "nvptx64-nvidia-cuda"
-                                                 : "nvptx-nvidia-cuda";
-    }
+    auto OFK = Action::OFK_Cuda;
+    DeviceTripleStr =
+        HostTriple.isArch64Bit() ? "nvptx64-nvidia-cuda" : "nvptx-nvidia-cuda";
     llvm::Triple CudaTriple(DeviceTripleStr);
-    // Use the CUDA/HIP and host triples as the key into the ToolChains map,
+    // Use the CUDA and host triples as the key into the ToolChains map,
     // because the device toolchain we create depends on both.
     auto &CudaTC = ToolChains[CudaTriple.str() + "/" + HostTriple.str()];
     if (!CudaTC) {
       CudaTC = llvm::make_unique<toolchains::CudaToolChain>(
           *this, CudaTriple, *HostTC, C.getInputArgs(), OFK);
     }
     C.addOffloadDeviceToolChain(CudaTC.get(), OFK);
+  } else if (IsHIP) {
+    const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
+    const llvm::Triple &HostTriple = HostTC->getTriple();
+    StringRef DeviceTripleStr;
+    auto OFK = Action::OFK_HIP;
+    DeviceTripleStr = "amdgcn-amd-amdhsa";
+    llvm::Triple HIPTriple(DeviceTripleStr);
+    // Use the HIP and host triples as the key into the ToolChains map,
+    // because the device toolchain we create depends on both.
+    auto &HIPTC = ToolChains[HIPTriple.str() + "/" + HostTriple.str()];
+    if (!HIPTC) {
+      HIPTC = llvm::make_unique<toolchains::HIPToolChain>(
+          *this, HIPTriple, *HostTC, C.getInputArgs());
+    }
+    C.addOffloadDeviceToolChain(HIPTC.get(), OFK);
   }
 
   //
   // OpenMP
   //
   // We need to generate an OpenMP toolchain if the user specified targets with
   // the -fopenmp-targets option.
   if (Arg *OpenMPTargets =
ActionBuilderReturnCode addDeviceDepences(Action *HostAction) override {
       // the host uses the CUDA offload kind.
       if (auto *IA = dyn_cast<InputAction>(HostAction)) {
         assert(!GpuArchList.empty() &&
                "We should have at least one GPU architecture.");
 
         // If the host input is not CUDA or HIP, we don't need to bother about
         // this input.
         if (IA->getType() != types::TY_CUDA &&
             IA->getType() != types::TY_HIP) {
           // The builder will ignore this input.
           IsActive = false;
           return ABRT_Inactive;
         }
 
         // Set the flag to true, so that the builder acts on the current input.
         IsActive = true;
 
bool initialize() override {
         return false;
 
       // We don't need to support HIP.
       if (AssociatedOffloadKind == Action::OFK_HIP &&
           !C.hasOffloadToolChain<Action::OFK_HIP>())
         return false;
 
       const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
       assert(HostTC && "No toolchain for host compilation.");
       if (HostTC->getTriple().isNVPTX() ||
           HostTC->getTriple().getArch() == llvm::Triple::amdgcn) {
         // We do not support targeting NVPTX/AMDGCN for host compilation. Throw
         // an error and abort pipeline construction early so we don't trip
         // asserts that assume device-side compilation.
         C.getDriver().Diag(diag::err_drv_cuda_host_arch)
             << HostTC->getTriple().getArchName();
         return true;
       }
 
       ToolChains.push_back(
           AssociatedOffloadKind == Action::OFK_Cuda
               ? C.getSingleOffloadToolChain<Action::OFK_Cuda>()
               : C.getSingleOffloadToolChain<Action::OFK_HIP>());
 
       Arg *PartialCompilationArg = Args.getLastArg(
static void AppendCollapsedOffloadAction(ActionList &CollapsedOffloadAction,
                                            unsigned ElementNum) {
     assert(ElementNum <= ActionInfo.size() && "Invalid number of elements.");
     for (unsigned I = 0; I < ElementNum; ++I)
       CollapsedOffloadAction.append(ActionInfo[I].SavedOffloadAction.begin(),
                                     ActionInfo[I].SavedOffloadAction.end());
   }
 
   /// Functions that attempt to perform the combining. They detect if that is
   /// legal, and if so they update the inputs \a Inputs and the offload action
   /// that were collapsed in \a CollapsedOffloadAction. A tool that deals with
   /// the combined action is returned. If the combining is not legal or if the
   /// tool does not exist, null is returned.
   /// Currently three kinds of collapsing are supported:
   ///  - Assemble + Backend + Compile;
   ///  - Assemble + Backend ;
   ///  - Backend + Compile.
   const Tool *
ToolSelector(const JobAction *BaseAction, const ToolChain &TC,
                const Compilation &C, bool SaveTemps, bool EmbedBitcode)
       : TC(TC), C(C), BaseAction(BaseAction), SaveTemps(SaveTemps),
         EmbedBitcode(EmbedBitcode) {
     assert(BaseAction && "Invalid base action.");
     IsHostSelector = BaseAction->getOffloadingDeviceKind() == Action::OFK_None;
   }
 
   /// Check if a chain of actions can be combined and return the tool that can
   /// handle the combination of actions. The pointer to the current inputs \a
   /// Inputs and the list of offload actions \a CollapsedOffloadActions
   /// connected to collapsed actions are updated accordingly. The latter enables
   /// the caller of the selector to process them afterwards instead of just
   /// dropping them. If no suitable tool is found, null will be returned.
   const Tool *getTool(const ActionList *&Inputs,
                       ActionList &CollapsedOffloadAction) {
     //
     // Get the largest chain of actions that we could combine.
     //
 
const char *Driver::GetNamedOutputPath(Compilation &C, const JobAction &JA,
   }
 
   // Output to a temporary file?
   if ((!AtTopLevel && !isSaveTempsEnabled() &&
        !C.getArgs().hasArg(options::OPT__SLASH_Fo)) ||
       CCGenDiagnostics) {
     StringRef Name = llvm::sys::path::filename(BaseInput);
     std::pair<StringRef, StringRef> Split = Name.split('.');
     std::string TmpName = GetTemporaryPath(
         Split.first, types::getTypeTempSuffix(JA.getType(), IsCLMode()));
     return C.addTempFile(C.getArgs().MakeArgString(TmpName));
   }
 
   SmallString<128> BasePath(BaseInput);
   StringRef BaseName;
 
   // Dsymutil actions should use the full path.
   if (isa<DsymutilJobAction>(JA) || isa<VerifyJobAction>(JA))
     BaseName = BasePath;
   else
     BaseName = llvm::sys::path::filename(BasePath);
 
   // Determine what the derived output name should be.
   const char *NamedOutput;
if ((JA.getType() == types::TY_Object || JA.getType() == types::TY_LTO_BC) &&
     }
     // When using both -save-temps and -emit-llvm, use a ".tmp.bc" suffix for
     // the unoptimized bitcode so that it does not get overwritten by the ".bc"
     // optimized bitcode output.
     if (!AtTopLevel && C.getArgs().hasArg(options::OPT_emit_llvm) &&
         JA.getType() == types::TY_LLVM_BC)
       Suffixed += ".tmp";
     Suffixed += '.';
     Suffixed += Suffix;
     NamedOutput = C.getArgs().MakeArgString(Suffixed.c_str());
   }
 
   // Prepend object file path if -save-temps=obj
   if (!AtTopLevel && isSaveTempsObj() && C.getArgs().hasArg(options::OPT_o) &&
       JA.getType() != types::TY_PCH) {
     Arg *FinalOutput = C.getArgs().getLastArg(options::OPT_o);
     SmallString<128> TempPath(FinalOutput->getValue());
     llvm::sys::path::remove_filename(TempPath);

lib/Driver/ToolChains/HIP.h

+//===--- HIP.h - HIP ToolChain Implementations ------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_DRIVER_TOOLCHAINS_HIP_H
+#define LLVM_CLANG_LIB_DRIVER_TOOLCHAINS_HIP_H
+
+#include "clang/Driver/ToolChain.h"
+#include "clang/Driver/Tool.h"
+
+namespace clang {
+namespace driver {
+
+namespace tools {
+
+namespace AMDGCN {
+// Runs llvm-link/opt/llc/lld, which links multiple LLVM bitcode, together with
+// device library, then compiles it to ISA in a shared object.
+class LLVM_LIBRARY_VISIBILITY Linker : public Tool {
+public:
+  Linker(const ToolChain &TC) : Tool("AMDGCN::Linker", "amdgcn-link", TC) {}
+
+  bool hasIntegratedCPP() const override { return false; }
+
+  void ConstructJob(Compilation &C, const JobAction &JA,
+                    const InputInfo &Output, const InputInfoList &Inputs,
+                    const llvm::opt::ArgList &TCArgs,
+                    const char *LinkingOutput) const override;
+
+private:
+  /// \return llvm-link output file name.
+  const char *constructLLVMLinkCommand(Compilation &C, const JobAction &JA,
+                                       const InputInfoList &Inputs,
+                                       const llvm::opt::ArgList &Args,
+                                       llvm::StringRef SubArchName,
+                                       llvm::StringRef OutputFilePrefix) const;
+
+  /// \return opt output file name.
+  const char *constructOptCommand(Compilation &C, const JobAction &JA,
+                                  const InputInfoList &Inputs,
+                                  const llvm::opt::ArgList &Args,
+                                  llvm::StringRef SubArchName,
+                                  llvm::StringRef OutputFilePrefix,
+                                  const char *InputFileName) const;
+
+  /// \return llc output file name.
+  const char *constructLlcCommand(Compilation &C, const JobAction &JA,
+                                  const InputInfoList &Inputs,
+                                  const llvm::opt::ArgList &Args,
+                                  llvm::StringRef SubArchName,
+                                  llvm::StringRef OutputFilePrefix,
+                                  const char *InputFileName) const;
+
+  void constructLldCommand(Compilation &C, const JobAction &JA,
+                           const InputInfoList &Inputs, const InputInfo &Output,
+                           const llvm::opt::ArgList &Args,
+                           const char *InputFileName) const;
+};
+
+} // end namespace AMDGCN
+} // end namespace tools
+
+namespace toolchains {
+
+class LLVM_LIBRARY_VISIBILITY HIPToolChain : public ToolChain {
+public:
+  HIPToolChain(const Driver &D, const llvm::Triple &Triple,
+                const ToolChain &HostTC, const llvm::opt::ArgList &Args);
+
+  const llvm::Triple *getAuxTriple() const override {
+    return &HostTC.getTriple();
+  }
+
+  llvm::opt::DerivedArgList *
+  TranslateArgs(const llvm::opt::DerivedArgList &Args, StringRef BoundArch,
+                Action::OffloadKind DeviceOffloadKind) const override;
+  void addClangTargetOptions(const llvm::opt::ArgList &DriverArgs,
+                             llvm::opt::ArgStringList &CC1Args,
+                             Action::OffloadKind DeviceOffloadKind) const override;
+
+  bool useIntegratedAs() const override { return true; }
+  bool isCrossCompiling() const override { return true; }
+  bool isPICDefault() const override { return false; }
+  bool isPIEDefault() const override { return false; }
+  bool isPICDefaultForced() const override { return false; }
+  bool SupportsProfiling() const override { return false; }
+  bool IsMathErrnoDefault() const override { return false; }
+
+  void addClangWarningOptions(llvm::opt::ArgStringList &CC1Args) const override;
+  CXXStdlibType GetCXXStdlibType(const llvm::opt::ArgList &Args) const override;
+  void
+  AddClangSystemIncludeArgs(const llvm::opt::ArgList &DriverArgs,
+                            llvm::opt::ArgStringList &CC1Args) const override;
+  void AddClangCXXStdlibIncludeArgs(
+      const llvm::opt::ArgList &Args,
+      llvm::opt::ArgStringList &CC1Args) const override;
+  void AddIAMCUIncludeArgs(const llvm::opt::ArgList &DriverArgs,
+                           llvm::opt::ArgStringList &CC1Args) const override;
+
+  SanitizerMask getSupportedSanitizers() const override;
+
+  VersionTuple
+  computeMSVCVersion(const Driver *D,
+                     const llvm::opt::ArgList &Args) const override;
+
+  unsigned GetDefaultDwarfVersion() const override { return 2; }
+
+  const ToolChain &HostTC;
+
+protected:
+  Tool *buildLinker() const override;
+};
+
+} // end namespace toolchains
+} // end namespace driver
+} // end namespace clang
+
+#endif // LLVM_CLANG_LIB_DRIVER_TOOLCHAINS_HIP_H

lib/Driver/ToolChains/HIP.cpp

+//===--- HIP.cpp - HIP Tool and ToolChain Implementations -------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "HIP.h"
+#include "CommonArgs.h"
+#include "InputInfo.h"
+#include "clang/Basic/Cuda.h"
+#include "clang/Driver/Compilation.h"
+#include "clang/Driver/Driver.h"
+#include "clang/Driver/DriverDiagnostic.h"
+#include "clang/Driver/Options.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+
+using namespace clang::driver;
+using namespace clang::driver::toolchains;
+using namespace clang::driver::tools;
+using namespace clang;
+using namespace llvm::opt;
+
+namespace {
+
+static void addBCLib(Compilation &C, const ArgList &Args,
+                     ArgStringList &CmdArgs, ArgStringList LibraryPaths,
+                     StringRef BCName) {
+  StringRef FullName;
+  for (std::string LibraryPath : LibraryPaths) {
+    SmallString<128> Path(LibraryPath);
+    llvm::sys::path::append(Path, BCName);
+    FullName = Path;
+    if (llvm::sys::fs::exists(FullName)) {
+      CmdArgs.push_back(Args.MakeArgString(FullName));
+      return;
+    }
+  }
+  C.getDriver().Diag(diag::err_drv_no_such_file) << BCName;
+}
+
+} // namespace
+
+const char *AMDGCN::Linker::constructLLVMLinkCommand(
+    Compilation &C, const JobAction &JA, const InputInfoList &Inputs,
+    const ArgList &Args, StringRef SubArchName,
+    StringRef OutputFilePrefix) const {
+  ArgStringList CmdArgs;
+  // Add the input bc's created by compile step.
+  for (const auto &II : Inputs)
+    CmdArgs.push_back(II.getFilename());
+
+  ArgStringList LibraryPaths;
+
+  // Find in --hip-device-lib-path and HIP_LIBRARY_PATH.
+  for (auto Path : Args.getAllArgValues(options::OPT_hip_device_lib_path_EQ))
+    LibraryPaths.push_back(Args.MakeArgString(Path));
+
+  addDirectoryList(Args, LibraryPaths, "-L", "HIP_DEVICE_LIB_PATH");
+
+  llvm::SmallVector<std::string, 10> BCLibs;
+
+  // Add bitcode library in --hip-device-lib.
+  for (auto Lib : Args.getAllArgValues(options::OPT_hip_device_lib_EQ)) {
+    BCLibs.push_back(Args.MakeArgString(Lib));
+  }
+
+  // If --hip-device-lib is not set, add the default bitcode libraries.
+  if (BCLibs.empty()) {
+    // Get the bc lib file name for ISA version. For example,
+    // gfx803 => oclc_isa_version_803.amdgcn.bc.
+    std::string ISAVerBC =
+        "oclc_isa_version_" + SubArchName.drop_front(3).str() + ".amdgcn.bc";
+
+    BCLibs.append({"hip.amdgcn.bc", "hc.amdgcn.bc", "opencl.amdgcn.bc",
+                   "ockl.amdgcn.bc", "irif.amdgcn.bc", "ocml.amdgcn.bc",
+                   "oclc_finite_only_off.amdgcn.bc",
+                   "oclc_daz_opt_off.amdgcn.bc",
+                   "oclc_correctly_rounded_sqrt_on.amdgcn.bc",
+                   "oclc_unsafe_math_off.amdgcn.bc", "hc.amdgcn.bc", ISAVerBC});
+  }
+  for (auto Lib : BCLibs)
+    addBCLib(C, Args, CmdArgs, LibraryPaths, Lib);
+
+  // Add an intermediate output file.
+  CmdArgs.push_back("-o");
+  std::string TmpName =
+      C.getDriver().GetTemporaryPath(OutputFilePrefix.str() + "-linked", "bc");
+  const char *OutputFileName =
+      C.addTempFile(C.getArgs().MakeArgString(TmpName));
+  CmdArgs.push_back(OutputFileName);
+  SmallString<128> ExecPath(C.getDriver().Dir);
+  llvm::sys::path::append(ExecPath, "llvm-link");
+  const char *Exec = Args.MakeArgString(ExecPath);
+  C.addCommand(llvm::make_unique<Command>(JA, *this, Exec, CmdArgs, Inputs));
+  return OutputFileName;
+}
+
+const char *AMDGCN::Linker::constructOptCommand(
+    Compilation &C, const JobAction &JA, const InputInfoList &Inputs,
+    const llvm::opt::ArgList &Args, llvm::StringRef SubArchName,
+    llvm::StringRef OutputFilePrefix, const char *InputFileName) const {
+  // Construct opt command.
+  ArgStringList OptArgs;
+  // The input to opt is the output from llvm-link.
+  OptArgs.push_back(InputFileName);
+  // Pass optimization arg to opt.
+  if (Arg *A = Args.getLastArg(options::OPT_O_Group)) {
+    StringRef OOpt = "3";
+    if (A->getOption().matches(options::OPT_O4) ||
+        A->getOption().matches(options::OPT_Ofast))
+      OOpt = "3";
+    else if (A->getOption().matches(options::OPT_O0))
+      OOpt = "0";
+    else if (A->getOption().matches(options::OPT_O)) {
+      // -Os, -Oz, and -O(anything else) map to -O2
+      OOpt = llvm::StringSwitch<const char *>(A->getValue())
+                 .Case("1", "1")
+                 .Case("2", "2")
+                 .Case("3", "3")
+                 .Case("s", "2")
+                 .Case("z", "2")
+                 .Default("2");
+    }
+    OptArgs.push_back(Args.MakeArgString("-O" + OOpt));
+  }
+  OptArgs.push_back("-mtriple=amdgcn-amd-amdhsa");
+  OptArgs.push_back(Args.MakeArgString("-mcpu=" + SubArchName));
+  OptArgs.push_back("-o");
+  std::string TmpFileName = C.getDriver().GetTemporaryPath(
+      OutputFilePrefix.str() + "-optimized", "bc");
+  const char *OutputFileName =
+      C.addTempFile(C.getArgs().MakeArgString(TmpFileName));
+  OptArgs.push_back(OutputFileName);
+  SmallString<128> OptPath(C.getDriver().Dir);
+  llvm::sys::path::append(OptPath, "opt");
+  const char *OptExec = Args.MakeArgString(OptPath);
+  C.addCommand(llvm::make_unique<Command>(JA, *this, OptExec, OptArgs, Inputs));
+  return OutputFileName;
+}
+
+const char *AMDGCN::Linker::constructLlcCommand(
+    Compilation &C, const JobAction &JA, const InputInfoList &Inputs,
+    const llvm::opt::ArgList &Args, llvm::StringRef SubArchName,
+    llvm::StringRef OutputFilePrefix, const char *InputFileName) const {
+  // Construct llc command.
+  ArgStringList LlcArgs{InputFileName, "-mtriple=amdgcn-amd-amdhsa",
+                        "-filetype=obj",
+                        Args.MakeArgString("-mcpu=" + SubArchName), "-o"};
+  std::string LlcOutputFileName =
+      C.getDriver().GetTemporaryPath(OutputFilePrefix, "o");
+  const char *LlcOutputFile =
+      C.addTempFile(C.getArgs().MakeArgString(LlcOutputFileName));
+  LlcArgs.push_back(LlcOutputFile);
+  SmallString<128> LlcPath(C.getDriver().Dir);
+  llvm::sys::path::append(LlcPath, "llc");
+  const char *Llc = Args.MakeArgString(LlcPath);
+  C.addCommand(llvm::make_unique<Command>(JA, *this, Llc, LlcArgs, Inputs));
+  return LlcOutputFile;
+}
+
+void AMDGCN::Linker::constructLldCommand(Compilation &C, const JobAction &JA,
+                                          const InputInfoList &Inputs,
+                                          const InputInfo &Output,
+                                          const llvm::opt::ArgList &Args,
+                                          const char *InputFileName) const {
+  // Construct lld command.
+  // The output from ld.lld is an HSA code object file.
+  ArgStringList LldArgs{"-flavor",    "gnu", "--no-undefined",
+                        "-shared",    "-o",  Output.getFilename(),
+                        InputFileName};
+  SmallString<128> LldPath(C.getDriver().Dir);
+  llvm::sys::path::append(LldPath, "lld");
+  const char *Lld = Args.MakeArgString(LldPath);
+  C.addCommand(llvm::make_unique<Command>(JA, *this, Lld, LldArgs, Inputs));
+}
+
+// For amdgcn the inputs of the linker job are device bitcode and output is
+// object file. It calls llvm-link, opt, llc, then lld steps.
+void AMDGCN::Linker::ConstructJob(Compilation &C, const JobAction &JA,
+                                   const InputInfo &Output,
+                                   const InputInfoList &Inputs,
+                                   const ArgList &Args,
+                                   const char *LinkingOutput) const {
+
+  const auto &TC =
+      static_cast<const toolchains::HIPToolChain &>(getToolChain());
+  assert(TC.getTriple().getArch() == llvm::Triple::amdgcn &&
+         "Unsupported target");
+
+  std::string SubArchName = JA.getOffloadingArch();
+  assert(StringRef(SubArchName).startswith("gfx") && "Unsupported sub arch");
+
+  // Prefix for temporary file name.
+  std::string Prefix =
+      llvm::sys::path::stem(Inputs[0].getFilename()).str() + "-" + SubArchName;
+
+  // Each command outputs different files.
+  const char *LLVMLinkCommand =
+      constructLLVMLinkCommand(C, JA, Inputs, Args, SubArchName, Prefix);
+  const char *OptCommand = constructOptCommand(C, JA, Inputs, Args, SubArchName,
+                                               Prefix, LLVMLinkCommand);
+  const char *LlcCommand =
+      constructLlcCommand(C, JA, Inputs, Args, SubArchName, Prefix, OptCommand);
+  constructLldCommand(C, JA, Inputs, Output, Args, LlcCommand);
+}
+
+HIPToolChain::HIPToolChain(const Driver &D, const llvm::Triple &Triple,
+                             const ToolChain &HostTC, const ArgList &Args)
+    : ToolChain(D, Triple, Args), HostTC(HostTC) {
+  // Lookup binaries into the driver directory, this is used to
+  // discover the clang-offload-bundler executable.
+  getProgramPaths().push_back(getDriver().Dir);
+}
+
+void HIPToolChain::addClangTargetOptions(
+    const llvm::opt::ArgList &DriverArgs,
+    llvm::opt::ArgStringList &CC1Args,
+    Action::OffloadKind DeviceOffloadingKind) const {
+  HostTC.addClangTargetOptions(DriverArgs, CC1Args, DeviceOffloadingKind);
+
+  StringRef GpuArch = DriverArgs.getLastArgValue(options::OPT_march_EQ);
+  assert(!GpuArch.empty() && "Must have an explicit GPU arch.");
+  assert(DeviceOffloadingKind == Action::OFK_HIP &&
+         "Only HIP offloading kinds are supported for GPUs.");
+
+  CC1Args.push_back("-fcuda-is-device");
+
+  if (DriverArgs.hasFlag(options::OPT_fcuda_flush_denormals_to_zero,
+                         options::OPT_fno_cuda_flush_denormals_to_zero, false))
+    CC1Args.push_back("-fcuda-flush-denormals-to-zero");
+
+  if (DriverArgs.hasFlag(options::OPT_fcuda_approx_transcendentals,
+                         options::OPT_fno_cuda_approx_transcendentals, false))
+    CC1Args.push_back("-fcuda-approx-transcendentals");
+
+  if (DriverArgs.hasFlag(options::OPT_fcuda_rdc, options::OPT_fno_cuda_rdc,
+                         false))
+    CC1Args.push_back("-fcuda-rdc");
+}
+
+llvm::opt::DerivedArgList *
+HIPToolChain::TranslateArgs(const llvm::opt::DerivedArgList &Args,
+                             StringRef BoundArch,
+                             Action::OffloadKind DeviceOffloadKind) const {
+  DerivedArgList *DAL =
+      HostTC.TranslateArgs(Args, BoundArch, DeviceOffloadKind);
+  if (!DAL)
+    DAL = new DerivedArgList(Args.getBaseArgs());
+
+  const OptTable &Opts = getDriver().getOpts();
+
+  for (Arg *A : Args) {
+    if (A->getOption().matches(options::OPT_Xarch__)) {
+      // Skip this argument unless the architecture matches BoundArch
+      if (BoundArch.empty() || A->getValue(0) != BoundArch)
+        continue;
+
+      unsigned Index = Args.getBaseArgs().MakeIndex(A->getValue(1));
+      unsigned Prev = Index;
+      std::unique_ptr<Arg> XarchArg(Opts.ParseOneArg(Args, Index));
+
+      // If the argument parsing failed or more than one argument was
+      // consumed, the -Xarch_ argument's parameter tried to consume
+      // extra arguments. Emit an error and ignore.
+      //
+      // We also want to disallow any options which would alter the
+      // driver behavior; that isn't going to work in our model. We
+      // use isDriverOption() as an approximation, although things
+      // like -O4 are going to slip through.
+      if (!XarchArg || Index > Prev + 1) {
+        getDriver().Diag(diag::err_drv_invalid_Xarch_argument_with_args)
+            << A->getAsString(Args);
+        continue;
+      } else if (XarchArg->getOption().hasFlag(options::DriverOption)) {
+        getDriver().Diag(diag::err_drv_invalid_Xarch_argument_isdriver)
+            << A->getAsString(Args);
+        continue;
+      }
+      XarchArg->setBaseArg(A);
+      A = XarchArg.release();
+      DAL->AddSynthesizedArg(A);
+    }
+    DAL->append(A);
+  }
+
+  if (!BoundArch.empty()) {
+    DAL->eraseArg(options::OPT_march_EQ);
+    DAL->AddJoinedArg(nullptr, Opts.getOption(options::OPT_march_EQ), BoundArch);
+  }
+
+  return DAL;
+}
+
+Tool *HIPToolChain::buildLinker() const {
+  assert(getTriple().getArch() == llvm::Triple::amdgcn);
+  return new tools::AMDGCN::Linker(*this);
+}
+
+void HIPToolChain::addClangWarningOptions(ArgStringList &CC1Args) const {
+  HostTC.addClangWarningOptions(CC1Args);
+}
+
+ToolChain::CXXStdlibType
+HIPToolChain::GetCXXStdlibType(const ArgList &Args) const {
+  return HostTC.GetCXXStdlibType(Args);
+}
+
+void HIPToolChain::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
+                                              ArgStringList &CC1Args) const {
+  HostTC.AddClangSystemIncludeArgs(DriverArgs, CC1Args);
+}
+
+void HIPToolChain::AddClangCXXStdlibIncludeArgs(const ArgList &Args,
+                                                 ArgStringList &CC1Args) const {
+  HostTC.AddClangCXXStdlibIncludeArgs(Args, CC1Args);
+}
+
+void HIPToolChain::AddIAMCUIncludeArgs(const ArgList &Args,
+                                        ArgStringList &CC1Args) const {
+  HostTC.AddIAMCUIncludeArgs(Args, CC1Args);
+}
+
+SanitizerMask HIPToolChain::getSupportedSanitizers() const {
+  // The HIPToolChain only supports sanitizers in the sense that it allows
+  // sanitizer arguments on the command line if they are supported by the host
+  // toolchain. The HIPToolChain will actually ignore any command line
+  // arguments for any of these "supported" sanitizers. That means that no
+  // sanitization of device code is actually supported at this time.
+  //
+  // This behavior is necessary because the host and device toolchains
+  // invocations often share the command line, so the device toolchain must
+  // tolerate flags meant only for the host toolchain.
+  return HostTC.getSupportedSanitizers();
+}
+
+VersionTuple HIPToolChain::computeMSVCVersion(const Driver *D,
+                                               const ArgList &Args) const {
+  return HostTC.computeMSVCVersion(D, Args);
+}

test/Driver/hip-toolchain.hip

+// REQUIRES: clang-driver
+// REQUIRES: x86-registered-target
+// REQUIRES: amdgpu-registered-target
+
+// RUN: %clang -### -target x86_64-linux-gnu \
+// RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
+// RUN:   --hip-device-lib=lib1.bc --hip-device-lib=lib2.bc \
+// RUN:   --hip-device-lib-path=%S/Inputs/hip_multiple_inputs/lib1 \
+// RUN:   --hip-device-lib-path=%S/Inputs/hip_multiple_inputs/lib2 \
+// RUN:   -fuse-ld=lld \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
+// RUN: 2>&1 | FileCheck %s
+
+// CHECK: [[CLANG:".*clang.*"]] "-cc1" "-triple" "amdgcn-amd-amdhsa" 
+// CHECK-SAME: "-aux-triple" "x86_64--linux-gnu" "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu" {{.*}} "-fcuda-is-device"
+// CHECK-SAME: {{.*}} "-o" [[A_BC:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC:".*a.cu"]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64--linux-gnu" "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip" {{.*}} "-fcuda-is-device"
+// CHECK-SAME: {{.*}} "-o" [[B_BC:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC:".*b.hip"]]
+
+// CHECK: [[LLVM_LINK:"*.llvm-link"]] [[A_BC]] [[B_BC]]
+// CHECK-SAME: "{{.*}}lib1.bc" "{{.*}}lib2.bc"
+// CHECK-SAME: "-o" [[LINKED_BC_DEV1:".*-gfx803-linked-.*bc"]]
+
+// CHECK: [[OPT:".*opt"]] [[LINKED_BC_DEV1]] "-mtriple=amdgcn-amd-amdhsa"
+// CHECK-SAME: "-mcpu=gfx803"
+// CHECK-SAME: "-o" [[OPT_BC_DEV1:".*-gfx803-optimized.*bc"]]
+
+// CHECK: [[LLC: ".*llc"]] [[OPT_BC_DEV1]] "-mtriple=amdgcn-amd-amdhsa"
+// CHECK-SAME: "-filetype=obj" "-mcpu=gfx803" "-o" [[OBJ_DEV1:".*-gfx803-.*o"]]
+
+// CHECK: [[LLD: ".*lld"]] "-flavor" "gnu" "--no-undefined" "-shared"
+// CHECK-SAME: "-o" "[[IMG_DEV1:.*out]]" [[OBJ_DEV1]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa" 
+// CHECK-SAME: "-aux-triple" "x86_64--linux-gnu" "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu" {{.*}} "-fcuda-is-device"
+// CHECK-SAME: {{.*}} "-o" [[A_BC:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "amdgcn-amd-amdhsa"
+// CHECK-SAME: "-aux-triple" "x86_64--linux-gnu" "-emit-llvm-bc"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip" {{.*}} "-fcuda-is-device"
+// CHECK-SAME: {{.*}} "-o" [[B_BC:".*bc"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC]]
+
+// CHECK: [[LLVM_LINK]] [[A_BC]] [[B_BC]]
+// CHECK-SAME: "{{.*}}lib1.bc" "{{.*}}lib2.bc"
+// CHECK-SAME: "-o" [[LINKED_BC_DEV2:".*-gfx900-linked-.*bc"]]
+
+// CHECK: [[OPT]] [[LINKED_BC_DEV2]] "-mtriple=amdgcn-amd-amdhsa"
+// CHECK-SAME: "-mcpu=gfx900"
+// CHECK-SAME: "-o" [[OPT_BC_DEV2:".*-gfx900-optimized.*bc"]]
+
+// CHECK: [[LLC]] [[OPT_BC_DEV2]] "-mtriple=amdgcn-amd-amdhsa"
+// CHECK-SAME: "-filetype=obj" "-mcpu=gfx900" "-o" [[OBJ_DEV2:".*-gfx900-.*o"]]
+
+// CHECK: [[LLD]] "-flavor" "gnu" "--no-undefined" "-shared"
+// CHECK-SAME: "-o" "[[IMG_DEV2:.*out]]" [[OBJ_DEV2]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "x86_64--linux-gnu" 
+// CHECK-SAME: "-aux-triple" "amdgcn-amd-amdhsa" "-emit-obj"
+// CHECK-SAME: {{.*}} "-main-file-name" "a.cu"
+// CHECK-SAME: {{.*}} "-o" [[A_OBJ_HOST:".*o"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[A_SRC]]
+
+// CHECK: [[CLANG]] "-cc1" "-triple" "x86_64--linux-gnu"
+// CHECK-SAME: "-aux-triple" "amdgcn-amd-amdhsa" "-emit-obj"
+// CHECK-SAME: {{.*}} "-main-file-name" "b.hip"
+// CHECK-SAME: {{.*}} "-o" [[B_OBJ_HOST:".*o"]] "-x" "hip"
+// CHECK-SAME: {{.*}} [[B_SRC]]
+
+// CHECK: [[BUNDLER:".*clang-offload-bundler"]] "-type=o"
+// CHECK-SAME: "-targets={{.*}},hip-amdgcn-amd-amdhsa-gfx803,hip-amdgcn-amd-amdhsa-gfx900"
+// CHECK-SAME: "-inputs={{.*}},[[IMG_DEV1]],[[IMG_DEV2]]" "-outputs=[[BUNDLE:.*o]]"
+
+// CHECK: [[LD:".*ld.lld"]] {{.*}} [[A_OBJ_HOST]] [[B_OBJ_HOST]]
+// CHECK-SAME: {{.*}} "-T" "{{.*}}.lk"