[CUDA] Allow the new driver to compile CUDA in non-RDC mode

[jhuber6]

Summary

The new driver primarily allows us to support RDC-mode compilations with
proper linking. This is not needed for non-RDC mode compilation, but we
still would like the new driver to be able to handle this mode so we can
transition away from the old driver in the future. This patch adds the
necessary code to support creating a fatbinary for CUDA code generation
as well as removing old assumptions and errors about RDC-mode with the
new driver.

[jhuber6]

clang/lib/Driver/ToolChains/Clang.cpp

6993–6994

Why do we add -fgpu-rdc here when it's already added on line 6322 above?

[tra]

Nice.

clang/lib/Driver/ToolChains/Clang.cpp

6994

This should not be necessary -- we've already added it for all CUDA/HIP compilations above.

6998

I'm not sure I understand when we'd end up in this "else" branch.
Perhaps the whole sequence could be restructured this way:

if ((IsCuda || IsHIP) && CudaDeviceInput) { // old-style embedding
    CmdArgs.push_back("-fcuda-include-gpubinary");
    CmdArgs.push_back(CudaDeviceInput->getFilename());
} else if (!HostOffloadingInputs.empty()) { // new-driver offloading mode
  if (Cuda && !IsRDCMode) {
    assert(HostOffloadingInputs.size() == 1); // We should have only one GPU fatbinary to embed. Uses old-style embedding.
    CmdArgs.push_back("-fcuda-include-gpubinary");
    CmdArgs.push_back(HostOffloadingInputs.front().getFilename());
  } else { // new driver embedding of GPU object files.
     for (const InputInfo Input : HostOffloadingInputs)
        CmdArgs.push_back(Args.MakeArgString("-fembed-offload-object=" +
                                             TC.getInputFilename(Input)));
  }
}

[jhuber6]

It's also worth noting that this doesn't include the PTX output for JIT in the fatbinary, it would be relatively easy to include that but I wanted to ask how we should handle that.

clang/lib/Driver/ToolChains/Clang.cpp

6994

What I figured, I think I'm just going to prune it in a separate commit to clean this up.

6998

This is clearer, I'll change it.

[jhuber6]

Updating and making suggested changes. I removed the old fgpu-rdc in rG6abaa8e2103760025cee76528f555de7cf6698e6 since I considered it to be reviewed already from the discussion here.

[tra]

clang/lib/Driver/ToolChains/Clang.cpp

6982

I think more details/history would be helpful here. Unlike you and me, the future readers will have little to no context to understand why we do things three different ways here.

[jhuber6]

clang/lib/Driver/ToolChains/Clang.cpp

6982

I'd like a future with only one way of doing it, but I'll add a comment mentioning the new and old drivers to make it clearer.

clang/include/clang/Basic/DiagnosticDriverKinds.td

 def err_drv_offload_bad_gpu_arch : Error<"unsupported %0 gpu architecture: %1">;
 def err_drv_no_cuda_installation : Error<
   "cannot find CUDA installation; provide its path via '--cuda-path', or pass "
   "'-nocudainc' to build without CUDA includes">;
 def err_drv_no_cuda_libdevice : Error<
   "cannot find libdevice for %0; provide path to different CUDA installation "
   "via '--cuda-path', or pass '-nocudalib' to build without linking with "
   "libdevice">;
-def err_drv_no_rdc_new_driver : Error<
-  "Using '--offload-new-driver' requires '-fgpu-rdc'">;
 
 def err_drv_no_rocm_device_lib : Error<
   "cannot find ROCm device library%select{| for %1|for ABI version %1}0; provide its path via "
   "'--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build "
   "without ROCm device library">;
 def err_drv_no_hip_runtime : Error<
   "cannot find HIP runtime; provide its path via '--rocm-path', or pass "
   "'-nogpuinc' to build without HIP runtime">;

clang/lib/CodeGen/CGCUDANV.cpp

static std::unique_ptr<MangleContext> InitDeviceMC(CodeGenModule &CGM) {
 
   return std::unique_ptr<MangleContext>(CGM.getContext().createMangleContext(
       CGM.getContext().getAuxTargetInfo()));
 }
 
 CGNVCUDARuntime::CGNVCUDARuntime(CodeGenModule &CGM)
     : CGCUDARuntime(CGM), Context(CGM.getLLVMContext()),
       TheModule(CGM.getModule()),
-      RelocatableDeviceCode(CGM.getLangOpts().GPURelocatableDeviceCode ||
-                            CGM.getLangOpts().OffloadingNewDriver),
+      RelocatableDeviceCode(CGM.getLangOpts().GPURelocatableDeviceCode),
       DeviceMC(InitDeviceMC(CGM)) {
   CodeGen::CodeGenTypes &Types = CGM.getTypes();
   ASTContext &Ctx = CGM.getContext();
 
   IntTy = CGM.IntTy;
   SizeTy = CGM.SizeTy;
   VoidTy = CGM.VoidTy;
 
for (auto &&Info : DeviceVars) {
            Kind == DeviceVarFlags::Surface ||
            Kind == DeviceVarFlags::Texture) &&
           Info.D->isUsed() && !Info.D->hasAttr<UsedAttr>()) {
         CGM.addCompilerUsedGlobal(Info.Var);
       }
     }
     return nullptr;
   }
-  if (!(CGM.getLangOpts().OffloadingNewDriver && RelocatableDeviceCode))
+  if (CGM.getLangOpts().OffloadingNewDriver && RelocatableDeviceCode)
+    createOffloadingEntries();
+  else
     return makeModuleCtorFunction();
 
-  createOffloadingEntries();
   return nullptr;
 }
 
 llvm::GlobalValue *CGNVCUDARuntime::getKernelHandle(llvm::Function *F,
                                                     GlobalDecl GD) {
   auto Loc = KernelHandles.find(F);
   if (Loc != KernelHandles.end())
     return Loc->second;

clang/lib/Driver/Driver.cpp

bool initialize() override {
         return false;
 
       // We don't need to support HIP.
       if (AssociatedOffloadKind == Action::OFK_HIP &&
           !C.hasOffloadToolChain<Action::OFK_HIP>())
         return false;
 
       Relocatable = Args.hasFlag(options::OPT_fgpu_rdc,
                                  options::OPT_fno_gpu_rdc, /*Default=*/false);
 
       const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
       assert(HostTC && "No toolchain for host compilation.");
       if (HostTC->getTriple().isNVPTX() ||
           HostTC->getTriple().getArch() == llvm::Triple::amdgcn) {
         // We do not support targeting NVPTX/AMDGCN for host compilation. Throw
         // an error and abort pipeline construction early so we don't trip
         // asserts that assume device-side compilation.
for (Action *A : DeviceActions) {
       OffloadActions.push_back(C.MakeAction<OffloadAction>(DDep, A->getType()));
       ++TCAndArch;
     }
   }
 
   if (offloadDeviceOnly())
     return C.MakeAction<OffloadAction>(DDeps, types::TY_Nothing);
 
-  Action *OffloadPackager =
-      C.MakeAction<OffloadPackagerJobAction>(OffloadActions, types::TY_Image);
   OffloadAction::DeviceDependences DDep;
-  DDep.add(*OffloadPackager, *C.getSingleOffloadToolChain<Action::OFK_Host>(),
+  if (C.isOffloadingHostKind(Action::OFK_Cuda) &&
+      !Args.hasFlag(options::OPT_fgpu_rdc, options::OPT_fno_gpu_rdc, false)) {
+    // If we are not in RDC-mode we just emit the final CUDA fatbinary for each
+    // translation unit without requiring any linking.
+    Action *FatbinAction =
+        C.MakeAction<LinkJobAction>(OffloadActions, types::TY_CUDA_FATBIN);
+    DDep.add(*FatbinAction, *C.getSingleOffloadToolChain<Action::OFK_Cuda>(),
+             nullptr, Action::OFK_Cuda);
+  } else {
+    // Package all the offloading actions into a single output that can be
+    // embedded in the host and linked.
+    Action *PackagerAction =
+        C.MakeAction<OffloadPackagerJobAction>(OffloadActions, types::TY_Image);
+    DDep.add(*PackagerAction, *C.getSingleOffloadToolChain<Action::OFK_Host>(),
              nullptr, Action::OFK_None);
+  }
   OffloadAction::HostDependence HDep(
       *HostAction, *C.getSingleOffloadToolChain<Action::OFK_Host>(),
       /*BoundArch=*/nullptr, isa<CompileJobAction>(HostAction) ? DDep : DDeps);
   return C.MakeAction<OffloadAction>(
       HDep, isa<CompileJobAction>(HostAction) ? DDep : DDeps);
 }
 
 Action *Driver::ConstructPhaseAction(

clang/lib/Driver/ToolChains/Clang.cpp

void Clang::ConstructJob(Compilation &C, const JobAction &JA,
   bool IsHostOffloadingAction =
       (JA.isHostOffloading(Action::OFK_OpenMP) &&
        Args.hasFlag(options::OPT_fopenmp_new_driver,
                     options::OPT_no_offload_new_driver, true)) ||
       (JA.isHostOffloading(C.getActiveOffloadKinds()) &&
        Args.hasFlag(options::OPT_offload_new_driver,
                     options::OPT_no_offload_new_driver, false));
 
+  bool IsRDCMode =
+      Args.hasFlag(options::OPT_fgpu_rdc, options::OPT_fno_gpu_rdc, false);
   bool IsUsingLTO = D.isUsingLTO(IsDeviceOffloadAction);
   auto LTOMode = D.getLTOMode(IsDeviceOffloadAction);
 
   // A header module compilation doesn't have a main input file, so invent a
   // fake one as a placeholder.
-  const char *ModuleName = [&]{
+  const char *ModuleName = [&] {
     auto *ModuleNameArg = Args.getLastArg(options::OPT_fmodule_name_EQ);
     return ModuleNameArg ? ModuleNameArg->getValue() : "";
   }();
   InputInfo HeaderModuleInput(Inputs[0].getType(), ModuleName, ModuleName);
 
   // Extract API doesn't have a main input file, so invent a fake one as a
   // placeholder.
   InputInfo ExtractAPIPlaceholderInput(Inputs[0].getType(), "extract-api",
if (IsHIP) {
     if (Args.hasFlag(options::OPT_fgpu_allow_device_init,
                      options::OPT_fno_gpu_allow_device_init, false))
       CmdArgs.push_back("-fgpu-allow-device-init");
     Args.addOptInFlag(CmdArgs, options::OPT_fhip_kernel_arg_name,
                       options::OPT_fno_hip_kernel_arg_name);
   }
 
   if (IsCuda || IsHIP) {
-    if (!Args.hasFlag(options::OPT_fgpu_rdc, options::OPT_fno_gpu_rdc, false) &&
-        Args.hasArg(options::OPT_offload_new_driver))
-      D.Diag(diag::err_drv_no_rdc_new_driver);
-    if (Args.hasFlag(options::OPT_fgpu_rdc, options::OPT_fno_gpu_rdc, false))
+    if (IsRDCMode)
       CmdArgs.push_back("-fgpu-rdc");
     if (Args.hasFlag(options::OPT_fgpu_defer_diag,
                      options::OPT_fno_gpu_defer_diag, false))
       CmdArgs.push_back("-fgpu-defer-diag");
     if (Args.hasFlag(options::OPT_fgpu_exclude_wrong_side_overloads,
                      options::OPT_fno_gpu_exclude_wrong_side_overloads,
                      false)) {
       CmdArgs.push_back("-fgpu-exclude-wrong-side-overloads");
if (TC.UseDwarfDebugFlags() || GRecordSwitches) {
       CmdArgs.push_back(FlagsArgString);
     }
     if (FRecordSwitches) {
       CmdArgs.push_back("-record-command-line");
       CmdArgs.push_back(FlagsArgString);
     }
   }
 
-  // Host-side cuda compilation receives all device-side outputs in a single
-  // fatbin as Inputs[1]. Include the binary with -fcuda-include-gpubinary.
+  // Host-side offloading compilation receives all device-side outputs. Include
+  // them in the host compilation depending on the target. If the host inputs
+  // are not empty we use the new-driver scheme, otherwise use the old scheme.
   if ((IsCuda || IsHIP) && CudaDeviceInput) {
-      CmdArgs.push_back("-fcuda-include-gpubinary");
-      CmdArgs.push_back(CudaDeviceInput->getFilename());
+  } else if (!HostOffloadingInputs.empty()) {
+    if (IsCuda && !IsRDCMode) {
+      assert(HostOffloadingInputs.size() == 1 && "Only one input expected");
+      CmdArgs.push_back("-fcuda-include-gpubinary");
+      CmdArgs.push_back(HostOffloadingInputs.front().getFilename());
+    } else {
+      for (const InputInfo Input : HostOffloadingInputs)
+        CmdArgs.push_back(Args.MakeArgString("-fembed-offload-object=" +
+                                             TC.getInputFilename(Input)));
+    }
   }
 
   if (IsCuda) {
     if (Args.hasFlag(options::OPT_fcuda_short_ptr,
                      options::OPT_fno_cuda_short_ptr, false))
       CmdArgs.push_back("-fcuda-short-ptr");
   }
 
   if (IsCuda || IsHIP) {
     // Determine the original source input.

   if (IsOpenMPDevice) {
     CmdArgs.push_back("-fopenmp-is-device");
     if (OpenMPDeviceInput) {
       CmdArgs.push_back("-fopenmp-host-ir-file-path");
       CmdArgs.push_back(Args.MakeArgString(OpenMPDeviceInput->getFilename()));
     }
   }
 
-  // Host-side offloading recieves the device object files and embeds it in a
-  // named section including the associated target triple and architecture.
-  for (const InputInfo Input : HostOffloadingInputs)
-    CmdArgs.push_back(Args.MakeArgString("-fembed-offload-object=" +
-                                         TC.getInputFilename(Input)));
-
   if (Triple.isAMDGPU()) {
     handleAMDGPUCodeObjectVersionOptions(D, Args, CmdArgs);
 
     Args.addOptInFlag(CmdArgs, options::OPT_munsafe_fp_atomics,
                       options::OPT_mno_unsafe_fp_atomics);
   }
 
   // For all the host OpenMP offloading compile jobs we need to pass the targets

clang/test/CodeGenCUDA/offloading-entries.cu

 // NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --check-globals --global-value-regex ".omp_offloading.entry.*"
-// RUN: %clang_cc1 -std=c++11 -triple x86_64-unknown-linux-gnu \
+// RUN: %clang_cc1 -std=c++11 -triple x86_64-unknown-linux-gnu -fgpu-rdc \
 // RUN:   --offload-new-driver -emit-llvm -o - -x cuda  %s | FileCheck \
 // RUN:   --check-prefix=CUDA %s
-// RUN: %clang_cc1 -std=c++11 -triple x86_64-unknown-linux-gnu \
+// RUN: %clang_cc1 -std=c++11 -triple x86_64-unknown-linux-gnu -fgpu-rdc \
 // RUN:   --offload-new-driver -emit-llvm -o - -x hip  %s | FileCheck \
 // RUN:   --check-prefix=HIP %s
 
 #include "Inputs/cuda.h"
 
 //.
 // CUDA: @.omp_offloading.entry_name = internal unnamed_addr constant [8 x i8] c"_Z3foov\00"
 // CUDA: @.omp_offloading.entry._Z3foov = weak constant %struct.__tgt_offload_entry { ptr @_Z18__device_stub__foov, ptr @.omp_offloading.entry_name, i64 0, i32 0, i32 0 }, section "cuda_offloading_entries", align 1

clang/test/Driver/cuda-omp-unsupported-debug-options.cu

 // RUN: | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -c %s -ggdb3 -gdwarf-aranges 2>&1 \
 // RUN: | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -c %s -g -gcolumn-info -fdebug-types-section 2>&1 \
 // RUN: | FileCheck %s --check-prefixes WARN,COMMON
 
 // Same tests for OpenMP
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -g -gz 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -g -gz 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf -fdebug-info-for-profiling 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -gdwarf -fdebug-info-for-profiling 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf-2 -gsplit-dwarf 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -gdwarf-2 -gsplit-dwarf 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf-3 -glldb 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -gdwarf-3 -glldb 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf-4 -gcodeview 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -gdwarf-4 -gcodeview 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf-5 -gmodules 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -gdwarf-5 -gmodules 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -ggdb1 -fdebug-macro 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -ggdb1 -fdebug-macro 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -ggdb2 -ggnu-pubnames 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -ggdb2 -ggnu-pubnames 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -ggdb3 -gdwarf-aranges 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -ggdb3 -gdwarf-aranges 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -g -gcolumn-info -fdebug-types-section 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
+// RUN:   -fgpu-rdc -g -gcolumn-info -fdebug-types-section 2>&1 | FileCheck %s --check-prefixes WARN,COMMON
 
 // RUN: %clang -### -target x86_64-linux-gnu -c %s -gdwarf-5 -gembed-source 2>&1 \
 // RUN: | FileCheck %s --check-prefixes WARN-GES,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -c %s -ggdb -gembed-source -gdwarf-5 2>&1 \
 // RUN: | FileCheck %s --check-prefixes WARN-GES,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -gdwarf-5 -gembed-source 2>&1 | FileCheck %s --check-prefixes WARN-GES,COMMON
+// RUN:   -fgpu-rdc -gdwarf-5 -gembed-source 2>&1 | FileCheck %s --check-prefixes WARN-GES,COMMON
 // RUN: %clang -### -target x86_64-linux-gnu -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -c %s \
-// RUN:   -ggdb -gembed-source -gdwarf-5 2>&1 | FileCheck %s --check-prefixes WARN-GES,COMMON
+// RUN:   -fgpu-rdc -ggdb -gembed-source -gdwarf-5 2>&1 | FileCheck %s --check-prefixes WARN-GES,COMMON
 
 // COMMON: warning: debug information option '{{-gz|-fdebug-info-for-profiling|-gsplit-dwarf|-glldb|-gcodeview|-gmodules|-gembed-source|-fdebug-macro|-ggnu-pubnames|-gdwarf-aranges|-fdebug-types-section}}' is not supported
 // WARN-SAME: for target 'nvptx64-nvidia-cuda' [-Wunsupported-target-opt]
 // WARN-GES-SAME: requires DWARF-5 but target 'nvptx64-nvidia-cuda' only provides DWARF-2 [-Wunsupported-target-opt]
 // COMMON-NOT: debug information option '{{.*}}' is not supported for target 'x86
 // COMMON: "-triple" "nvptx64-nvidia-cuda"
 // COMMON-NOT: {{-compress-debug|-fdebug-info-for-profiling|lldb|codeview|module-format|embed-source|debug-info-macro|gnu-pubnames|generate-arange-section|generate-type-units}}
 // COMMON: "-triple" "x86_64
 // COMMON-SAME: {{-compress-debug|-fdebug-info-for-profiling|split-dwarf|lldb|codeview|module-format|embed-source|debug-info-macro|gnu-pubnames|generate-arange-section|generate-type-units}}

clang/test/Driver/cuda-openmp-driver.cu

 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib -ccc-print-bindings -fgpu-rdc \
 // RUN:        --offload-new-driver --offload-arch=sm_35 --offload-arch=sm_70 %s 2>&1 \
 // RUN: | FileCheck -check-prefix BINDINGS %s
 
 //      BINDINGS: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[PTX_SM_35:.+]]"
 // BINDINGS-NEXT: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[PTX_SM_35]]"], output: "[[CUBIN_SM_35:.+]]"
 // BINDINGS-NEXT: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT]]"], output: "[[PTX_SM_70:.+]]"
 // BINDINGS-NEXT: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[PTX_SM_70:.+]]"], output: "[[CUBIN_SM_70:.+]]"
 // BINDINGS-NEXT: "x86_64-unknown-linux-gnu" - "Offload::Packager", inputs: ["[[CUBIN_SM_35]]", "[[CUBIN_SM_70]]"], output: "[[BINARY:.+]]"
 // BINDINGS-NEXT: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[INPUT]]", "[[BINARY]]"], output: "[[HOST_OBJ:.+]]"
 // BINDINGS-NEXT: "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[HOST_OBJ]]"], output: "a.out"
 
-// RUN: %clang -### -nocudalib --offload-new-driver %s 2>&1 | FileCheck -check-prefix RDC %s
-// RDC: error: Using '--offload-new-driver' requires '-fgpu-rdc'
-
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib -ccc-print-bindings -fgpu-rdc \
 // RUN:        --offload-new-driver --offload-arch=sm_35 --offload-arch=sm_70 %s 2>&1 \
 // RUN: | FileCheck -check-prefix BINDINGS-HOST %s
 
 // BINDINGS-HOST: # "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[OUTPUT:.+]]"
 // BINDINGS-HOST: # "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[OUTPUT]]"], output: "a.out"
 
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib -ccc-print-bindings -fgpu-rdc \
 // RUN:        --offload-new-driver --offload-arch=sm_35 --offload-arch=sm_70 %s 2>&1 \
 // RUN: | FileCheck -check-prefix BINDINGS-DEVICE %s
 
 // BINDINGS-DEVICE: # "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[PTX:.+]]"
 // BINDINGS-DEVICE: # "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[PTX]]"], output: "[[CUBIN:.+]]"
 
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib --cuda-feature=+ptx61 --offload-arch=sm_70 %s 2>&1 | FileCheck -check-prefix MANUAL-FEATURE %s
 // MANUAL-FEATURE: -cc1{{.*}}-target-feature{{.*}}+ptx61
 
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib -ccc-print-bindings --offload-link %s 2>&1 \
 // RUN: | FileCheck -check-prefix DEVICE-LINK %s
 
 // DEVICE-LINK: "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[INPUT:.+]]"], output: "a.out"
+
+// RUN: %clang -### -target x86_64-linux-gnu -nocudalib --offload-new-driver \
+// RUN:   --offload-arch=sm_35 --offload-arch=sm_70 %s 2>&1 \
+// RUN: | FileCheck -check-prefix GPU-BINARY %s
+
+// GPU-BINARY: fatbinary{{.*}}"--create" "{{.*}}.fatbin"
+// GPU-BINARY: -cc1{{.*}}-fcuda-include-gpubinary" "{{.*}}.fatbin"

clang/test/Driver/cuda-phases.cu

 // DASM2-DAG: [[P7:[0-9]+]]: compiler, {[[P6]]}, ir, (device-[[T]], [[ARCH2]])
 // DASM2-DAG: [[P8:[0-9]+]]: backend, {[[P7]]}, assembler, (device-[[T]], [[ARCH2]])
 // DASM2-DAG: [[P9:[0-9]+]]: offload, "device-[[T]] ([[TRIPLE]]:[[ARCH2]])" {[[P8]]}, assembler
 // DASM2-NOT: host
 
 //
 // Test the phases generated when using the new offloading driver.
 //
-// RUN: %clang -### -target powerpc64le-ibm-linux-gnu -ccc-print-phases --offload-new-driver \
-// RUN: --offload-arch=sm_52 --offload-arch=sm_70 %s 2>&1 | FileCheck --check-prefix=NEW_DRIVER %s
-// NEW_DRIVER: 0: input, "[[INPUT:.+]]", cuda
-// NEW_DRIVER: 1: preprocessor, {0}, cuda-cpp-output
-// NEW_DRIVER: 2: compiler, {1}, ir
-// NEW_DRIVER: 3: input, "[[INPUT]]", cuda, (device-cuda, sm_52)
-// NEW_DRIVER: 4: preprocessor, {3}, cuda-cpp-output, (device-cuda, sm_52)
-// NEW_DRIVER: 5: compiler, {4}, ir, (device-cuda, sm_52)
-// NEW_DRIVER: 6: backend, {5}, assembler, (device-cuda, sm_52)
-// NEW_DRIVER: 7: assembler, {6}, object, (device-cuda, sm_52)
-// NEW_DRIVER: 8: offload, "device-cuda (nvptx64-nvidia-cuda:sm_52)" {7}, object
-// NEW_DRIVER: 9: input, "[[INPUT]]", cuda, (device-cuda, sm_70)
-// NEW_DRIVER: 10: preprocessor, {9}, cuda-cpp-output, (device-cuda, sm_70)
-// NEW_DRIVER: 11: compiler, {10}, ir, (device-cuda, sm_70)
-// NEW_DRIVER: 12: backend, {11}, assembler, (device-cuda, sm_70)
-// NEW_DRIVER: 13: assembler, {12}, object, (device-cuda, sm_70)
-// NEW_DRIVER: 14: offload, "device-cuda (nvptx64-nvidia-cuda:sm_70)" {13}, object
-// NEW_DRIVER: 15: clang-offload-packager, {8, 14}, image
-// NEW_DRIVER: 16: offload, " (powerpc64le-ibm-linux-gnu)" {2}, " (powerpc64le-ibm-linux-gnu)" {15}, ir
-// NEW_DRIVER: 17: backend, {16}, assembler, (host-cuda)
-// NEW_DRIVER: 18: assembler, {17}, object, (host-cuda)
-// NEW_DRIVER: 19: clang-linker-wrapper, {18}, image, (host-cuda)
+// RUN: %clang -### -target powerpc64le-ibm-linux-gnu -ccc-print-phases --offload-new-driver -fgpu-rdc \
+// RUN: --offload-arch=sm_52 --offload-arch=sm_70 %s 2>&1 | FileCheck --check-prefix=NEW-DRIVER-RDC %s
+// NEW-DRIVER-RDC: 0: input, "[[INPUT:.+]]", cuda
+// NEW-DRIVER-RDC: 1: preprocessor, {0}, cuda-cpp-output
+// NEW-DRIVER-RDC: 2: compiler, {1}, ir
+// NEW-DRIVER-RDC: 3: input, "[[INPUT]]", cuda, (device-cuda, sm_52)
+// NEW-DRIVER-RDC: 4: preprocessor, {3}, cuda-cpp-output, (device-cuda, sm_52)
+// NEW-DRIVER-RDC: 5: compiler, {4}, ir, (device-cuda, sm_52)
+// NEW-DRIVER-RDC: 6: backend, {5}, assembler, (device-cuda, sm_52)
+// NEW-DRIVER-RDC: 7: assembler, {6}, object, (device-cuda, sm_52)
+// NEW-DRIVER-RDC: 8: offload, "device-cuda (nvptx64-nvidia-cuda:sm_52)" {7}, object
+// NEW-DRIVER-RDC: 9: input, "[[INPUT]]", cuda, (device-cuda, sm_70)
+// NEW-DRIVER-RDC: 10: preprocessor, {9}, cuda-cpp-output, (device-cuda, sm_70)
+// NEW-DRIVER-RDC: 11: compiler, {10}, ir, (device-cuda, sm_70)
+// NEW-DRIVER-RDC: 12: backend, {11}, assembler, (device-cuda, sm_70)
+// NEW-DRIVER-RDC: 13: assembler, {12}, object, (device-cuda, sm_70)
+// NEW-DRIVER-RDC: 14: offload, "device-cuda (nvptx64-nvidia-cuda:sm_70)" {13}, object
+// NEW-DRIVER-RDC: 15: clang-offload-packager, {8, 14}, image
+// NEW-DRIVER-RDC: 16: offload, " (powerpc64le-ibm-linux-gnu)" {2}, " (powerpc64le-ibm-linux-gnu)" {15}, ir
+// NEW-DRIVER-RDC: 17: backend, {16}, assembler, (host-cuda)
+// NEW-DRIVER-RDC: 18: assembler, {17}, object, (host-cuda)
+// NEW-DRIVER-RDC: 19: clang-linker-wrapper, {18}, image, (host-cuda)
+
+// RUN: %clang -### -target powerpc64le-ibm-linux-gnu -ccc-print-phases --offload-new-driver -fgpu-rdc \
+// RUN: --offload-arch=sm_52 --offload-arch=sm_70 %s 2>&1 | FileCheck --check-prefix=NEW-DRIVER %s
+// NEW-DRIVER: 0: input, "[[INPUT:.+]]", cuda
+// NEW-DRIVER: 1: preprocessor, {0}, cuda-cpp-output
+// NEW-DRIVER: 2: compiler, {1}, ir
+// NEW-DRIVER: 3: input, "[[INPUT]]", cuda, (device-cuda, sm_52)
+// NEW-DRIVER: 4: preprocessor, {3}, cuda-cpp-output, (device-cuda, sm_52)
+// NEW-DRIVER: 5: compiler, {4}, ir, (device-cuda, sm_52)
+// NEW-DRIVER: 6: backend, {5}, assembler, (device-cuda, sm_52)
+// NEW-DRIVER: 7: assembler, {6}, object, (device-cuda, sm_52)
+// NEW-DRIVER: 8: offload, "device-cuda (nvptx64-nvidia-cuda:sm_52)" {7}, object
+// NEW-DRIVER: 9: input, "[[INPUT]]", cuda, (device-cuda, sm_70)
+// NEW-DRIVER: 10: preprocessor, {9}, cuda-cpp-output, (device-cuda, sm_70)
+// NEW-DRIVER: 11: compiler, {10}, ir, (device-cuda, sm_70)
+// NEW-DRIVER: 12: backend, {11}, assembler, (device-cuda, sm_70)
+// NEW-DRIVER: 13: assembler, {12}, object, (device-cuda, sm_70)
+// NEW-DRIVER: 14: offload, "device-cuda (nvptx64-nvidia-cuda:sm_70)" {13}, object
+// NEW-DRIVER: 15: clang-offload-packager, {8, 14}, image
+// NEW-DRIVER: 16: offload, " (powerpc64le-ibm-linux-gnu)" {2}, " (powerpc64le-ibm-linux-gnu)" {15}, ir
+// NEW-DRIVER: 17: backend, {16}, assembler, (host-cuda)
+// NEW-DRIVER: 18: assembler, {17}, object, (host-cuda)
+// NEW-DRIVER: 19: clang-linker-wrapper, {18}, image, (host-cuda)