[HIP]: Add -fhip-emit-relocatable to override link job creation for -fno-gpu-rdc

[jrbyrnes]

Summary

Provide control over clang job / action creation. This feature provides the phase pipeline for an upcoming COMGR action : AMD_COMGR_ACTION_COMPILE_SOURCE_WITH_DEVICE_LIBS_TO_RELOCATABLE

[jrbyrnes]

Formatting

[yaxunl]

clang/test/Driver/hip-phases.hip

259

probably just use

// RELOC-NOT: linker

same for below

also, we need a test for -fgpu-rdc case

[jrbyrnes]

Fix tests + add tests. Add phase test for -fgpu-rdc --no-gpu-link-output (these are not intended to be used together)

[jhuber6]

What's the difference here between this and the existing --hip-link?

[jrbyrnes]

In D153667#4450517, @jhuber6 wrote:

What's the difference here between this and the existing --hip-link?

Hi @jhuber6

The commit is poorly named, the main purpose is to introduce -no-gpu-link-output.

We want a way to produce relocatable from source. In terms of the Driver, this means building actions and jobs for phases up to phases::Assemble. -no-gpu-link-output does this by overriding BuildActions to stop after phases::Assemble (similar to -no-gpu-bundle-output). -gpu-link-output is NFCI. COMGR would be the client of this, and it would be up to COMGR to handle linking of the relocatable.

AFAICT, -hip-link allows for linking of offload-bundles / linking through HIPAMD toolchain, so it is conceptually different. We can get (somewhat) close to what we with -emit-llvm -hip-link, but that is probably more due to -emit-llvm. -hip-link by itself produces linker actions / jobs which what we are trying to avoid here.

[jhuber6]

In D153667#4450705, @jrbyrnes wrote:

In D153667#4450517, @jhuber6 wrote:

What's the difference here between this and the existing --hip-link?

Hi @jhuber6

The commit is poorly named, the main purpose is to introduce -no-gpu-link-output.

We want a way to produce relocatable from source. In terms of the Driver, this means building actions and jobs for phases up to phases::Assemble. -no- gpu-link-output does this by overriding BuildActions to stop after phases::Assemble (similar to -no-gpu-bundle-output). -gpu-link-output is NFCI. COMGR would be the client of this, and it would be up to COMGR to handle linking of the relocatable.

AFAICT, -hip-link allows for linking of offload-bundles, so it is conceptually different. We can get (somewhat) close to what we with -emit-llvm -hip-link, but that is probably more due to -emit-llvm. -hip-link by itself produces linker actions / jobs which what we are trying to avoid here.

So, you run the backend and obtain a relocatable ELF, but do not link it via lld? If I'm understanding this correctly, that would be the difference between -flto and -fno-lto, or -foffload-lto and -fno-offload-lto, AMDGPU always having -flto on currently. Also I recall AMDGPU / HIP completely disabling the backend step at some point, so it only emits LLVM-IR.

[yaxunl]

In D153667#4450724, @jhuber6 wrote:

In D153667#4450705, @jrbyrnes wrote:

In D153667#4450517, @jhuber6 wrote:

What's the difference here between this and the existing --hip-link?

Hi @jhuber6

The commit is poorly named, the main purpose is to introduce -no-gpu-link-output.

We want a way to produce relocatable from source. In terms of the Driver, this means building actions and jobs for phases up to phases::Assemble. -no- gpu-link-output does this by overriding BuildActions to stop after phases::Assemble (similar to -no-gpu-bundle-output). -gpu-link-output is NFCI. COMGR would be the client of this, and it would be up to COMGR to handle linking of the relocatable.

AFAICT, -hip-link allows for linking of offload-bundles, so it is conceptually different. We can get (somewhat) close to what we with -emit-llvm -hip-link, but that is probably more due to -emit-llvm. -hip-link by itself produces linker actions / jobs which what we are trying to avoid here.

So, you run the backend and obtain a relocatable ELF, but do not link it via lld? If I'm understanding this correctly, that would be the difference between -flto and -fno-lto, or -foffload-lto and -fno-offload-lto, AMDGPU always having -flto on currently. Also I recall AMDGPU / HIP completely disabling the backend step at some point, so it only emits LLVM-IR.

For -fno-gpu-rdc case we do not use lto. Since -fno-gpu-rdc has one TU only, we use the non-lto backend to get relocatable object, and use lld for relocatable to shared object. This patch allows us to stop at the relocatable object since comgr needs that.

[jhuber6]

In D153667#4455943, @yaxunl wrote:

For -fno-gpu-rdc case we do not use lto. Since -fno-gpu-rdc has one TU only, we use the non-lto backend to get relocatable object, and use lld for relocatable to shared object. This patch allows us to stop at the relocatable object since comgr needs that.

I see, so conceptually this is like -Xarch-device -c (if such a thing worked)?

[yaxunl]

I am thinking we probably want to rename the option as -fhip-emit-relocatable and limit it to be used with -cuda-device-only and -fno-gpu-rdc only.

[yaxunl]

In D153667#4455998, @jhuber6 wrote:

In D153667#4455943, @yaxunl wrote:

For -fno-gpu-rdc case we do not use lto. Since -fno-gpu-rdc has one TU only, we use the non-lto backend to get relocatable object, and use lld for relocatable to shared object. This patch allows us to stop at the relocatable object since comgr needs that.

I see, so conceptually this is like -Xarch-device -c (if such a thing worked)?

It is like that

[jrbyrnes]

In D153667#4450724, @jhuber6 wrote:

In D153667#4450705, @jrbyrnes wrote:

In D153667#4450517, @jhuber6 wrote:

What's the difference here between this and the existing --hip-link?

Hi @jhuber6

The commit is poorly named, the main purpose is to introduce -no-gpu-link-output.

We want a way to produce relocatable from source. In terms of the Driver, this means building actions and jobs for phases up to phases::Assemble. -no- gpu-link-output does this by overriding BuildActions to stop after phases::Assemble (similar to -no-gpu-bundle-output). -gpu-link-output is NFCI. COMGR would be the client of this, and it would be up to COMGR to handle linking of the relocatable.

AFAICT, -hip-link allows for linking of offload-bundles, so it is conceptually different. We can get (somewhat) close to what we with -emit-llvm -hip-link, but that is probably more due to -emit-llvm. -hip-link by itself produces linker actions / jobs which what we are trying to avoid here.

So, you run the backend and obtain a relocatable ELF, but do not link it via lld? If I'm understanding this correctly, that would be the difference between -flto and -fno-lto, or -foffload-lto and -fno-offload-lto, AMDGPU always having -flto on currently. Also I recall AMDGPU / HIP completely disabling the backend step at some point, so it only emits LLVM-IR.

The whole point of this work is to give hiprtc a way to compile-to-bitcode and optimize sources in a single step, to make (user-passed) flag handling less weird. Since the intent of LTO is to defer this optimization step, I would assume any way we try to use it here would not be correct.

[jrbyrnes]

Naming + -cuda-device-only and -fno-gpu-rdc only

[yaxunl]

clang/lib/Driver/Driver.cpp

3367–3369

There are data members Relocatable and CompileDeviceOnly in the base class. You can use them instead of using local variables.

3374

need to emit a diag diag::err_opt_not_valid_with_opt for fgpu-rdc
and diag::err_opt_not_valid_without_opt if it is not device only.

[jrbyrnes]

Use member variabls + add diagnostic + tests

[yaxunl]

clang/lib/Driver/Driver.cpp

3364–3366

probably needs to be moved to ctor of CudaActionBuilderBase since they are needed by both Cuda and HIP action builders.

[jrbyrnes]

Address Comment

clang/lib/Driver/Driver.cpp

3364–3366

Thanks

[yaxunl]

LGTM. Thanks

clang/include/clang/Driver/Options.td

def gpu_instrument_lib_EQ : Joined<["--"], "gpu-instrument-lib=">,
   "__cyg_profile_func_enter and __cyg_profile_func_exit">;
 def fgpu_sanitize : Flag<["-"], "fgpu-sanitize">, Group<f_Group>,
   HelpText<"Enable sanitizer for AMDGPU target">;
 def fno_gpu_sanitize : Flag<["-"], "fno-gpu-sanitize">, Group<f_Group>;
 def gpu_bundle_output : Flag<["--"], "gpu-bundle-output">,
   Group<f_Group>, HelpText<"Bundle output files of HIP device compilation">;
 def no_gpu_bundle_output : Flag<["--"], "no-gpu-bundle-output">,
   Group<f_Group>, HelpText<"Do not bundle output files of HIP device compilation">;
+def fhip_emit_relocatable : Flag<["-"], "fhip-emit-relocatable">, Group<f_Group>,
+  HelpText<"Compile HIP source to relocatable">;
+def fno_hip_emit_relocatable : Flag<["-"], "fno-hip-emit-relocatable">, Group<f_Group>,
+  HelpText<"Do not override toolchain to compile HIP source to relocatable">;
 def cuid_EQ : Joined<["-"], "cuid=">, Flags<[CC1Option]>,
   HelpText<"An ID for compilation unit, which should be the same for the same "
            "compilation unit but different for different compilation units. "
            "It is used to externalize device-side static variables for single "
            "source offloading languages CUDA and HIP so that they can be "
            "accessed by the host code of the same compilation unit.">,
   MarshallingInfoString<LangOpts<"CUID">>;
 def fuse_cuid_EQ : Joined<["-"], "fuse-cuid=">,

clang/lib/Driver/Driver.cpp

protected:
 
     /// Compilation unit ID specified by option '-cuid='.
     StringRef FixedCUID;
 
   public:
     CudaActionBuilderBase(Compilation &C, DerivedArgList &Args,
                           const Driver::InputList &Inputs,
                           Action::OffloadKind OFKind)
-        : DeviceActionBuilder(C, Args, Inputs, OFKind) {}
+        : DeviceActionBuilder(C, Args, Inputs, OFKind) {
+
+      CompileDeviceOnly = C.getDriver().offloadDeviceOnly();
+      Relocatable = Args.hasFlag(options::OPT_fgpu_rdc,
+                                 options::OPT_fno_gpu_rdc, /*Default=*/false);
+    }
 
     ActionBuilderReturnCode addDeviceDependences(Action *HostAction) override {
       // While generating code for CUDA, we only depend on the host input action
       // to trigger the creation of all the CUDA device actions.
 
       // If we are dealing with an input action, replicate it for each GPU
       // architecture. If we are in host-only mode we return 'success' so that
       // the host uses the CUDA offload kind.
bool initialize() override {
           !C.hasOffloadToolChain<Action::OFK_Cuda>())
         return false;
 
       // We don't need to support HIP.
       if (AssociatedOffloadKind == Action::OFK_HIP &&
           !C.hasOffloadToolChain<Action::OFK_HIP>())
         return false;
 
-      Relocatable = Args.hasFlag(options::OPT_fgpu_rdc,
-                                 options::OPT_fno_gpu_rdc, /*Default=*/false);
-
       const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
       assert(HostTC && "No toolchain for host compilation.");
       if (HostTC->getTriple().isNVPTX() ||
           HostTC->getTriple().getArch() == llvm::Triple::amdgcn) {
         // We do not support targeting NVPTX/AMDGCN for host compilation. Throw
         // an error and abort pipeline construction early so we don't trip
         // asserts that assume device-side compilation.
         C.getDriver().Diag(diag::err_drv_cuda_host_arch)
             << HostTC->getTriple().getArchName();
         return true;
       }
 
       ToolChains.push_back(
           AssociatedOffloadKind == Action::OFK_Cuda
               ? C.getSingleOffloadToolChain<Action::OFK_Cuda>()
               : C.getSingleOffloadToolChain<Action::OFK_HIP>());
 
       CompileHostOnly = C.getDriver().offloadHostOnly();
-      CompileDeviceOnly = C.getDriver().offloadDeviceOnly();
       EmitLLVM = Args.getLastArg(options::OPT_emit_llvm);
       EmitAsm = Args.getLastArg(options::OPT_S);
       FixedCUID = Args.getLastArgValue(options::OPT_cuid_EQ);
       if (Arg *A = Args.getLastArg(options::OPT_fuse_cuid_EQ)) {
         StringRef UseCUIDStr = A->getValue();
         UseCUID = llvm::StringSwitch<UseCUIDKind>(UseCUIDStr)
                       .Case("hash", CUID_Hash)
                       .Case("random", CUID_Random)
class HIPActionBuilder final : public CudaActionBuilderBase {
     /// The linker inputs obtained for each device arch.
     SmallVector<ActionList, 8> DeviceLinkerInputs;
     // The default bundling behavior depends on the type of output, therefore
     // BundleOutput needs to be tri-value: None, true, or false.
     // Bundle code objects except --no-gpu-output is specified for device
     // only compilation. Bundle other type of output files only if
     // --gpu-bundle-output is specified for device only compilation.
     std::optional<bool> BundleOutput;
+    std::optional<bool> EmitReloc;
 
   public:
     HIPActionBuilder(Compilation &C, DerivedArgList &Args,
                      const Driver::InputList &Inputs)
         : CudaActionBuilderBase(C, Args, Inputs, Action::OFK_HIP) {
+
       DefaultCudaArch = CudaArch::GFX906;
+
+      if (Args.hasArg(options::OPT_fhip_emit_relocatable,
+                      options::OPT_fno_hip_emit_relocatable)) {
+        EmitReloc = Args.hasFlag(options::OPT_fhip_emit_relocatable,
+                                 options::OPT_fno_hip_emit_relocatable, false);
+
+        if (*EmitReloc) {
+          if (Relocatable) {
+            C.getDriver().Diag(diag::err_opt_not_valid_with_opt)
+                << "-fhip-emit-relocatable"
+                << "-fgpu-rdc";
+          }
+
+          if (!CompileDeviceOnly) {
+            C.getDriver().Diag(diag::err_opt_not_valid_without_opt)
+                << "-fhip-emit-relocatable"
+                << "--cuda-device-only";
+          }
+        }
+      }
+
       if (Args.hasArg(options::OPT_gpu_bundle_output,
                       options::OPT_no_gpu_bundle_output))
         BundleOutput = Args.hasFlag(options::OPT_gpu_bundle_output,
-                                    options::OPT_no_gpu_bundle_output, true);
+                                    options::OPT_no_gpu_bundle_output, true) &&
+                       (!EmitReloc || !*EmitReloc);
     }
 
     bool canUseBundlerUnbundler() const override { return true; }
 
     StringRef getCanonicalOffloadArch(StringRef IdStr) override {
       llvm::StringMap<bool> Features;
       // getHIPOffloadTargetTriple() is known to return valid value as it has
       // been called successfully in the CreateOffloadingDeviceToolChains().
getDeviceDependences(OffloadAction::DeviceDependences &DA,
         return ABRT_Success;
 
       assert(((CurPhase == phases::Link && Relocatable) ||
               CudaDeviceActions.size() == GpuArchList.size()) &&
              "Expecting one action per GPU architecture.");
       assert(!CompileHostOnly &&
              "Not expecting HIP actions in host-only compilation.");
 
+      bool ShouldLink = !EmitReloc || !*EmitReloc;
+
       if (!Relocatable && CurPhase == phases::Backend && !EmitLLVM &&
-          !EmitAsm) {
+          !EmitAsm && ShouldLink) {
         // If we are in backend phase, we attempt to generate the fat binary.
         // We compile each arch to IR and use a link action to generate code
         // object containing ISA. Then we use a special "link" action to create
         // a fat binary containing all the code objects for different GPU's.
         // The fat binary is then an input to the host action.
         for (unsigned I = 0, E = GpuArchList.size(); I != E; ++I) {
           if (C.getDriver().isUsingLTO(/*IsOffload=*/true)) {
             // When LTO is enabled, skip the backend and assemble phases and
getDeviceDependences(OffloadAction::DeviceDependences &DA,
 
           // Remove the CUDA actions as they are already connected to an host
           // action or fat binary.
           CudaDeviceActions.clear();
         }
 
         return CompileDeviceOnly ? ABRT_Ignore_Host : ABRT_Success;
       } else if (CurPhase == phases::Link) {
+        if (!ShouldLink)
+          return ABRT_Success;
         // Save CudaDeviceActions to DeviceLinkerInputs for each GPU subarch.
         // This happens to each device action originated from each input file.
         // Later on, device actions in DeviceLinkerInputs are used to create
         // device link actions in appendLinkDependences and the created device
         // link actions are passed to the offload action as device dependence.
         DeviceLinkerInputs.resize(CudaDeviceActions.size());
         auto LI = DeviceLinkerInputs.begin();
         for (auto *A : CudaDeviceActions) {
getDeviceDependences(OffloadAction::DeviceDependences &DA,
           CudaDeviceActions[I] = C.MakeAction<OffloadAction>(
               DDep, CudaDeviceActions[I]->getType());
         }
         CudaFatBinary =
             C.MakeAction<OffloadBundlingJobAction>(CudaDeviceActions);
         CudaDeviceActions.clear();
       }
 
-      return (CompileDeviceOnly && CurPhase == FinalPhase) ? ABRT_Ignore_Host
-                                                           : ABRT_Success;
+      return (CompileDeviceOnly &&
+              (CurPhase == FinalPhase ||
+               (!ShouldLink && CurPhase == phases::Assemble)))
+                 ? ABRT_Ignore_Host
     }
 
     void appendLinkDeviceActions(ActionList &AL) override {
       if (DeviceLinkerInputs.size() == 0)
         return;
 
       assert(DeviceLinkerInputs.size() == GpuArchList.size() &&
              "Linker inputs and GPU arch list sizes do not match.");
addDeviceDependencesToHostAction(Action *HostAction, const Arg *InputArg,
     auto &OffloadKind = InputArgToOffloadKindMap[InputArg];
     unsigned InactiveBuilders = 0u;
     unsigned IgnoringBuilders = 0u;
     for (auto *SB : SpecializedBuilders) {
       if (!SB->isValid()) {
         ++InactiveBuilders;
         continue;
       }
-
       auto RetCode =
           SB->getDeviceDependences(DDeps, CurPhase, FinalPhase, Phases);
 
       // If the builder explicitly says the host action should be ignored,
       // we need to increment the variable that tracks the builders that request
       // the host object to be ignored.
       if (RetCode == DeviceActionBuilder::ABRT_Ignore_Host)
         ++IgnoringBuilders;

clang/test/Driver/hip-dependent-options.hip

+// RUN: %clang -### --target=x86_64-linux-gnu \
+// RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
+// RUN:   -c -fhip-emit-relocatable -nogpuinc -nogpulib --cuda-device-only -fgpu-rdc \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip --gpu-bundle-output \
+// RUN: 2>&1 | FileCheck -check-prefixes=RELOCRDC %s
+
+// RELOCRDC: error: option '-fhip-emit-relocatable' cannot be specified with '-fgpu-rdc'
+
+// RUN: %clang -### --target=x86_64-linux-gnu \
+// RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
+// RUN:   -c -fhip-emit-relocatable -nogpuinc -nogpulib \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip --gpu-bundle-output \
+// RUN: 2>&1 | FileCheck -check-prefixes=RELOCHOST %s
+
+// RELOCHOST: error: option '-fhip-emit-relocatable' cannot be specified without '--cuda-device-only'

clang/test/Driver/hip-device-compile.hip

 // Output unbundled assembly.
 // RUN: %clang -c -S --cuda-device-only -### --target=x86_64-linux-gnu \
 // RUN:   -o a.s -x hip --cuda-gpu-arch=gfx900 --no-gpu-bundle-output \
 // RUN:   --hip-device-lib=lib1.bc \
 // RUN:   --hip-device-lib-path=%S/Inputs/hip_multiple_inputs/lib1 \
 // RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
 // RUN: 2>&1 | FileCheck -check-prefixes=CHECK,ASM,NBUN %s
 
+// Output relocatable.
+// RUN: %clang -c --cuda-device-only -### --target=x86_64-linux-gnu \
+// RUN:   -o a.o -x hip --cuda-gpu-arch=gfx900 -fhip-emit-relocatable \
+// RUN:   --hip-device-lib=lib1.bc \
+// RUN:   --hip-device-lib-path=%S/Inputs/hip_multiple_inputs/lib1 \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN: 2>&1 | FileCheck -check-prefixes=CHECK,NBUN,RELOC %s
+
 // Output bundled assembly.
 // RUN: %clang -c -S --cuda-device-only -### --target=x86_64-linux-gnu \
 // RUN:   -o a.s -x hip --cuda-gpu-arch=gfx900 --no-gpu-bundle-output \
 // RUN:   --hip-device-lib=lib1.bc \
 // RUN:   --hip-device-lib-path=%S/Inputs/hip_multiple_inputs/lib1 \
 // RUN:   %S/Inputs/hip_multiple_inputs/a.cu --gpu-bundle-output \
 // RUN: 2>&1 | FileCheck -check-prefixes=CHECK,ASMBUN %s
 
 // CHECK: {{".*clang.*"}} "-cc1" "-triple" "amdgcn-amd-amdhsa"
 // CHECK-SAME: "-aux-triple" "x86_64-unknown-linux-gnu"
 // BC-SAME: "-emit-llvm-bc"
 // LL-SAME: "-emit-llvm"
 // ASM-NOT: "-emit-llvm"
 // CHECK-SAME: "-main-file-name" "a.cu"
 // CHECK-SAME: "-fcuda-is-device"
 // CHECK-SAME: {{".*lib1.bc"}}
 // CHECK-SAME: "-target-cpu" "gfx900"
 // BC-SAME: "-o" "a.bc"
 // BCBUN-SAME: "-o" "{{.*}}.bc"
 // LL-SAME: "-o" "a.ll"
 // LLBUN-SAME: "-o" "{{.*}}.ll"
 // ASM-SAME: "-o" "a.s"
 // ASMBUN-SAME: "-o" "{{.*}}.s"
+// RELOC-SAME: "-o" "a.o"
 // CHECK-SAME: {{".*a.cu"}}
 
 // CHECK-NOT: {{"*.llvm-link"}}
 // CHECK-NOT: {{".*opt"}}
 // CHECK-NOT: {{".*llc"}}
 // CHECK-NOT: {{".*lld.*"}}
 // NBUN-NOT: {{".*clang-offload-bundler"}}
 // BCBUN: {{".*clang-offload-bundler"}}{{.*}}"-output=a.bc"

clang/test/Driver/hip-phases.hip

 // DASM-DAG: [[P1:[0-9]+]]: preprocessor, {[[P0]]}, [[T]]-cpp-output, (device-[[T]], [[ARCH]])
 // DASM-DAG: [[P2:[0-9]+]]: compiler, {[[P1]]}, ir, (device-[[T]], [[ARCH]])
 // DASM-DAG: [[P3:[0-9]+]]: backend, {[[P2]]}, assembler, (device-[[T]], [[ARCH]])
 // DASM-DAG: [[P4:[0-9]+]]: offload, "device-[[T]] (amdgcn-amd-amdhsa:[[ARCH]])" {[[P3]]}, assembler
 // DASM-NOT: clang-offload-bundler
 // DASM-NOT: host
 
 //
+// Test single gpu architecture with compile to relocatable in device-only
+// compilation mode.
+//
+// RUN: %clang -x hip --target=x86_64-unknown-linux-gnu -ccc-print-phases \
+// RUN: --cuda-gpu-arch=gfx803 %s --cuda-device-only -fhip-emit-relocatable 2>&1 \
+// RUN: | FileCheck -check-prefixes=RELOC %s
+// RELOC-DAG: [[P0:[0-9]+]]: input, "{{.*}}hip-phases.hip", [[T:hip]], (device-[[T]], [[ARCH:gfx803]])
+// RELOC-DAG: [[P1:[0-9]+]]: preprocessor, {[[P0]]}, [[T]]-cpp-output, (device-[[T]], [[ARCH]])
+// RELOC-DAG: [[P2:[0-9]+]]: compiler, {[[P1]]}, ir, (device-[[T]], [[ARCH]])
+// RELOC-DAG: [[P3:[0-9]+]]: backend, {[[P2]]}, assembler, (device-[[T]], [[ARCH]])
+// RELOC-DAG: [[P4:[0-9]+]]: assembler, {[[P3]]}, object, (device-[[T]], [[ARCH]])
+// RELOC-NOT: linker
+// RELOC-DAG: [[P5:[0-9]+]]: offload, "device-[[T]] (amdgcn-amd-amdhsa:[[ARCH]])" {[[P4]]}, object
+
+//
+// Test two gpu architectures with compile to relocatable in device-only
+// compilation mode.
+//
+// RUN: %clang -x hip --target=x86_64-unknown-linux-gnu -ccc-print-phases \
+// RUN: --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 %s --cuda-device-only -fhip-emit-relocatable 2>&1 \
+// RUN: | FileCheck -check-prefixes=RELOC2 %s
+// RELOC2-DAG: [[P0:[0-9]+]]: input, "{{.*}}hip-phases.hip", [[T:hip]], (device-[[T]], [[ARCH:gfx803]])
+// RELOC2-DAG: [[P1:[0-9]+]]: preprocessor, {[[P0]]}, [[T]]-cpp-output, (device-[[T]], [[ARCH]])
+// RELOC2-DAG: [[P2:[0-9]+]]: compiler, {[[P1]]}, ir, (device-[[T]], [[ARCH]])
+// RELOC2-DAG: [[P3:[0-9]+]]: backend, {[[P2]]}, assembler, (device-[[T]], [[ARCH]])
+// RELOC2-DAG: [[P4:[0-9]+]]: assembler, {[[P3]]}, object, (device-[[T]], [[ARCH]])
+// RELOC2-NOT: [[P5:[0-9]+]]: linker, {[[P4]]}, image, (device-[[T]], [[ARCH]])
+// RELOC2-DAG: [[P5:[0-9]+]]: offload, "device-[[T]] (amdgcn-amd-amdhsa:[[ARCH]])" {[[P4]]}, object
+// RELOC2-DAG: [[P6:[0-9]+]]: input, "{{.*}}hip-phases.hip", [[T:hip]], (device-[[T]], [[ARCH2:gfx900]])
+// RELOC2-DAG: [[P7:[0-9]+]]: preprocessor, {[[P6]]}, [[T]]-cpp-output, (device-[[T]], [[ARCH2]])
+// RELOC2-DAG: [[P8:[0-9]+]]: compiler, {[[P7]]}, ir, (device-[[T]], [[ARCH2]])
+// RELOC2-DAG: [[P9:[0-9]+]]: backend, {[[P8]]}, assembler, (device-[[T]], [[ARCH2]])
+// RELOC2-DAG: [[P10:[0-9]+]]: assembler, {[[P9]]}, object, (device-[[T]], [[ARCH2]])
+// RELOC2-NOT: linker
+// RELOC2-DAG: [[P11:[0-9]+]]: offload, "device-[[T]] (amdgcn-amd-amdhsa:[[ARCH2]])" {[[P10]]}, object
+
+//
 // Test two gpu architectures with complete compilation in device-only
 // compilation mode.
 //
 // RUN: %clang -x hip --target=x86_64-unknown-linux-gnu -ccc-print-phases \
 // RUN: --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 %s --cuda-device-only \
 // RUN: 2>&1 | FileCheck -check-prefixes=DBIN2 %s
 // DBIN2-DAG: [[P0:[0-9]+]]: input, "{{.*}}hip-phases.hip", [[T:hip]], (device-[[T]], [[ARCH:gfx803]])
 // DBIN2-DAG: [[P1:[0-9]+]]: preprocessor, {[[P0]]}, [[T]]-cpp-output, (device-[[T]], [[ARCH]])

clang/test/Driver/hip-rdc-device-only.hip

 
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
 // RUN:   -c -emit-llvm -nogpuinc -nogpulib --cuda-device-only -fgpu-rdc \
 // RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip --gpu-bundle-output \
 // RUN: 2>&1 | FileCheck -check-prefixes=COMMON,EMITBC %s
 
+// With `-fno-hip-emit-relocatable`, the output should be the same as the aforementioned line
+// as `-fgpu-rdc` in HIP implies `-fno-hip-emit-relocatable`.
+
+// RUN: %clang -### --target=x86_64-linux-gnu \
+// RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
+// RUN:   -c -fno-hip-emit-relocatable -nogpuinc -nogpulib --cuda-device-only -fgpu-rdc \
+// RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip --gpu-bundle-output \
+// RUN: 2>&1 | FileCheck -check-prefixes=COMMON,EMITBC %s
+
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   -x hip --cuda-gpu-arch=gfx803 --cuda-gpu-arch=gfx900 \
 // RUN:   -S -nogpuinc -nogpulib --cuda-device-only -fgpu-rdc \
 // RUN:   %S/Inputs/hip_multiple_inputs/a.cu \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip --gpu-bundle-output \
 // RUN: 2>&1 | FileCheck -check-prefixes=COMMON,EMITLL %s
 
 // With `-emit-llvm`, the output should be the same as the aforementioned line