[Clang] Fix using LTO with the new driver in RDC-mode

[jhuber6]

Summary

The new driver supports LTO for RDC-mode compilations. However, this was
not correctly handled for non-LTO compilations. HIP can handle this as
it is fed to lld which will perform the LTO itself. CUDA however would
require every work which is wholly useless in non-RDC mode so it should
report an error.

[yaxunl]

should we test with -ccc-print-phases instead? It is not clear what actions are produced by driver.

[jhuber6]

In D135305#3838412, @yaxunl wrote:

should we test with -ccc-print-phases instead? It is not clear what actions are produced by driver.

Of the four run lines added, three use -ccc-print-phases. The last uses just -### because otherwise the error message would not be triggered as it would quit before trying to generate the Clang job.

[jhuber6]

In D135305#3838412, @yaxunl wrote:

should we test with -ccc-print-phases instead? It is not clear what actions are produced by driver.

Ah, realized you meant with phases. This is good enough for NVPTX as there is no explicit assembler phase. LTO makes no difference for AMDGPU as it already outputs bitcode.

[yaxunl]

In D135305#3838435, @jhuber6 wrote:

In D135305#3838412, @yaxunl wrote:

should we test with -ccc-print-phases instead? It is not clear what actions are produced by driver.

Ah, realized you meant with phases. This is good enough for NVPTX as there is no explicit assembler phase. LTO makes no difference for AMDGPU as it already outputs bitcode.

It is not clear whether the driver respects -foffload-lto when -fno-gpu-rdc is specified. -ccc-print-phases shows whether the driver generates bc or object. -ccc-print-bindings does not because it does not show the file types.

[jhuber6]

In D135305#3838490, @yaxunl wrote:

In D135305#3838435, @jhuber6 wrote:

In D135305#3838412, @yaxunl wrote:

should we test with -ccc-print-phases instead? It is not clear what actions are produced by driver.

Ah, realized you meant with phases. This is good enough for NVPTX as there is no explicit assembler phase. LTO makes no difference for AMDGPU as it already outputs bitcode.

It is not clear whether the driver respects -foffload-lto when -fno-gpu-rdc is specified. -ccc-print-phases shows whether the driver generates bc or object. -ccc-print-bindings does not because it does not show the file types.

There's an existing test in the OpenMP toolchain that uses phases I believe, using bindings was just easier for making the tests. If you think it's worth changing then I'll do it, but it was just easier this way.

[jhuber6]

Changing two tests to use phases instead to illustrate the lto-bc usage.

[yaxunl]

clang/test/Driver/hip-phases.hip

553

do we need a test for -foffload-lto -fno-gpu-rdc ?

[jhuber6]

clang/test/Driver/hip-phases.hip

553

The test above still uses the bindings, all the matters for that one is that the jobs generated are legal and still the same.

[yaxunl]

LGTM. Thanks.

clang/lib/Driver/Driver.cpp

for (phases::ID Phase : PL) {
           A = C.MakeAction<OffloadAction>(HDep, DDep);
         }
         ++TCAndArch;
       }
     }
 
     // Compiling HIP in non-RDC mode requires linking each action individually.
     for (Action *&A : DeviceActions) {
-      if (A->getType() != types::TY_Object || Kind != Action::OFK_HIP ||
+      if ((A->getType() != types::TY_Object &&
+           A->getType() != types::TY_LTO_BC) ||
+          Kind != Action::OFK_HIP ||
           Args.hasFlag(options::OPT_fgpu_rdc, options::OPT_fno_gpu_rdc, false))
         continue;
       ActionList LinkerInput = {A};
       A = C.MakeAction<LinkJobAction>(LinkerInput, types::TY_Image);
     }
 
     auto TCAndArch = TCAndArchs.begin();
     for (Action *A : DeviceActions) {

clang/lib/Driver/ToolChains/Clang.cpp

if (isa<AnalyzeJobAction>(JA)) {
     // Preserve use-list order by default when emitting bitcode, so that
     // loading the bitcode up in 'opt' or 'llc' and running passes gives the
     // same result as running passes here.  For LTO, we don't need to preserve
     // the use-list order, since serialization to bitcode is part of the flow.
     if (JA.getType() == types::TY_LLVM_BC)
       CmdArgs.push_back("-emit-llvm-uselists");
 
     if (IsUsingLTO) {
-      // Only AMDGPU supports device-side LTO.
       if (IsDeviceOffloadAction && !JA.isDeviceOffloading(Action::OFK_OpenMP) &&
           !Args.hasFlag(options::OPT_offload_new_driver,
                         options::OPT_no_offload_new_driver, false) &&
           !Triple.isAMDGPU()) {
         D.Diag(diag::err_drv_unsupported_opt_for_target)
             << Args.getLastArg(options::OPT_foffload_lto,
                                options::OPT_foffload_lto_EQ)
                    ->getAsString(Args)
             << Triple.getTriple();
+      } else if (Triple.isNVPTX() && !IsRDCMode) {
+        D.Diag(diag::err_drv_unsupported_opt_for_language_mode)
+            << Args.getLastArg(options::OPT_foffload_lto,
+                               options::OPT_foffload_lto_EQ)
+                   ->getAsString(Args)
+            << "-fno-gpu-rdc";
       } else {
         assert(LTOMode == LTOK_Full || LTOMode == LTOK_Thin);
         CmdArgs.push_back(Args.MakeArgString(
             Twine("-flto=") + (LTOMode == LTOK_Thin ? "thin" : "full")));
         CmdArgs.push_back("-flto-unit");
       }
     }
   }

clang/test/Driver/cuda-bindings.cu

 // Check to ensure that we can use '-save-temps' when operating in RDC-mode.
 //
 // RUN: %clang -### -target powerpc64le-ibm-linux-gnu -save-temps --offload-new-driver \
 // RUN:        -fgpu-rdc --offload-arch=sm_70 --offload-arch=sm_52 -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix=SAVE-TEMPS %s
 // SAVE-TEMPS: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_52"
 // SAVE-TEMPS: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_70"
 // SAVE-TEMPS: "-cc1" "-triple" "powerpc64le-ibm-linux-gnu"
+
+//
+// Check to ensure that we cannot use '-foffload' when not operating in RDC-mode.
+//
+// RUN: %clang -### -target powerpc64le-ibm-linux-gnu -fno-gpu-rdc --offload-new-driver \
+// RUN:        -foffload-lto --offload-arch=sm_70 --offload-arch=sm_52 -c %s 2>&1 \
+// RUN: | FileCheck -check-prefix=LTO-NO-RDC %s
+// LTO-NO-RDC: error: unsupported option '-foffload-lto' for language mode '-fno-gpu-rdc'

clang/test/Driver/cuda-phases.cu

 // NON-CUDA-INPUT-NEXT: 17: backend, {16}, assembler, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 18: assembler, {17}, object, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 19: input, "[[CPP:.+]]", c++, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 20: preprocessor, {19}, c++-cpp-output, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 21: compiler, {20}, ir, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 22: backend, {21}, assembler, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 23: assembler, {22}, object, (host-cuda)
 // NON-CUDA-INPUT-NEXT: 24: clang-linker-wrapper, {18, 23}, image, (host-cuda)
+
+//
+// Test the phases using the new driver in LTO-mode.
+//
+// RUN: %clang -### -target powerpc64le-ibm-linux-gnu --offload-new-driver -ccc-print-phases \
+// RUN:        --offload-arch=sm_70 --offload-arch=sm_52 -foffload-lto -fgpu-rdc -c %s 2>&1 \
+// RUN: | FileCheck -check-prefix=LTO %s
+//      LTO: 0: input, "[[INPUT:.+]]", cuda, (host-cuda)
+// LTO-NEXT: 1: preprocessor, {0}, cuda-cpp-output, (host-cuda)
+// LTO-NEXT: 2: compiler, {1}, ir, (host-cuda)
+// LTO-NEXT: 3: input, "[[INPUT]]", cuda, (device-cuda, sm_52)
+// LTO-NEXT: 4: preprocessor, {3}, cuda-cpp-output, (device-cuda, sm_52)
+// LTO-NEXT: 5: compiler, {4}, ir, (device-cuda, sm_52)
+// LTO-NEXT: 6: backend, {5}, lto-bc, (device-cuda, sm_52)
+// LTO-NEXT: 7: offload, "device-cuda (nvptx64-nvidia-cuda:sm_52)" {6}, lto-bc
+// LTO-NEXT: 8: input, "[[INPUT]]", cuda, (device-cuda, sm_70)
+// LTO-NEXT: 9: preprocessor, {8}, cuda-cpp-output, (device-cuda, sm_70)
+// LTO-NEXT: 10: compiler, {9}, ir, (device-cuda, sm_70)
+// LTO-NEXT: 11: backend, {10}, lto-bc, (device-cuda, sm_70)
+// LTO-NEXT: 12: offload, "device-cuda (nvptx64-nvidia-cuda:sm_70)" {11}, lto-bc
+// LTO-NEXT: 13: clang-offload-packager, {7, 12}, image, (device-cuda)
+// LTO-NEXT: 14: offload, "host-cuda (powerpc64le-ibm-linux-gnu)" {2}, "device-cuda (powerpc64le-ibm-linux-gnu)" {13}, ir
+// LTO-NEXT: 15: backend, {14}, assembler, (host-cuda)
+// LTO-NEXT: 16: assembler, {15}, object, (host-cuda)

clang/test/Driver/hip-binding.hip

 // driver.
 // 
 // RUN: %clang -### -target x86_64-unknown-linux-gnu --offload-new-driver \
 // RUN:        -fsyntax-only --offload-arch=gfx90a --offload-arch=gfx908 -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix=SYNTAX-ONLY %s
 // SYNTAX-ONLY: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-fsyntax-only"
 // SYNTAX-ONLY: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-fsyntax-only"
 // SYNTAX-ONLY: "-cc1" "-triple" "x86_64-unknown-linux-gnu"{{.*}}"-fsyntax-only"
+
+//
+// Check to ensure that we can use '-foffload' when not operating in RDC-mode.
+//
+// RUN: %clang -### --target=x86_64-linux-gnu -fno-gpu-rdc --offload-new-driver -ccc-print-bindings \
+// RUN:        -foffload-lto --offload-arch=gfx90a --offload-arch=gfx908 -c %s 2>&1 \
+// RUN: | FileCheck -check-prefix=LTO-NO-RDC %s
+//      LTO-NO-RDC: # "amdgcn-amd-amdhsa" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[LTO_908:.+]]"
+// LTO-NO-RDC-NEXT: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[LTO_908]]"], output: "[[OBJ_908:.+]]"
+// LTO-NO-RDC-NEXT: # "amdgcn-amd-amdhsa" - "clang", inputs: ["[[INPUT]]"], output: "[[LTO_90A:.+]]"
+// LTO-NO-RDC-NEXT: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[LTO_90A]]"], output: "[[OBJ_90A:.+]]"
+// LTO-NO-RDC-NEXT: # "amdgcn-amd-amdhsa" - "AMDGCN::Linker", inputs: ["[[OBJ_908]]", "[[OBJ_90A]]"], output: "[[HIPFB:.+]]"

clang/test/Driver/hip-phases.hip

 
 // CHECK: [[B0:[0-9]+]]: input, "{{.*}}bitcodeB.bc", ir
 // CHECK: [[B1:[0-9]+]]: clang-offload-unbundler, {[[B0]]}, ir
 // CHECK: [[B2:[0-9]+]]: compiler, {[[B1]]}, ir, (device-hip, [[ARCH]])
 // CHECK: [[B3:[0-9]+]]: backend, {[[B2]]}, ir, (device-hip, [[ARCH]])
 
 // CHECK: [[L0:[0-9]+]]: linker, {[[A3]], [[B3]]}, ir, (device-hip, [[ARCH]])
 // CHECK: offload, "device-hip (amdgcn-amd-amdhsa:[[ARCH]])" {[[L0]]}, ir
+
+//
+// Test the bindings using the new driver in LTO-mode.
+//
+// RUN: %clang -### --target=x86_64-linux-gnu --offload-new-driver -ccc-print-phases \
+// RUN:        --offload-arch=gfx90a --offload-arch=gfx908 -foffload-lto -fgpu-rdc -c %s 2>&1 \
+// RUN: | FileCheck -check-prefix=LTO %s
+//      LTO: 0: input, "[[INPUT:.+]]", hip, (host-hip)
+// LTO-NEXT: 1: preprocessor, {0}, hip-cpp-output, (host-hip)
+// LTO-NEXT: 2: compiler, {1}, ir, (host-hip)
+// LTO-NEXT: 3: input, "[[INPUT]]", hip, (device-hip, gfx908)
+// LTO-NEXT: 4: preprocessor, {3}, hip-cpp-output, (device-hip, gfx908)
+// LTO-NEXT: 5: compiler, {4}, ir, (device-hip, gfx908)
+// LTO-NEXT: 6: backend, {5}, lto-bc, (device-hip, gfx908)
+// LTO-NEXT: 7: offload, "device-hip (amdgcn-amd-amdhsa:gfx908)" {6}, lto-bc
+// LTO-NEXT: 8: input, "[[INPUT]]", hip, (device-hip, gfx90a)
+// LTO-NEXT: 9: preprocessor, {8}, hip-cpp-output, (device-hip, gfx90a)
+// LTO-NEXT: 10: compiler, {9}, ir, (device-hip, gfx90a)
+// LTO-NEXT: 11: backend, {10}, lto-bc, (device-hip, gfx90a)
+// LTO-NEXT: 12: offload, "device-hip (amdgcn-amd-amdhsa:gfx90a)" {11}, lto-bc
+// LTO-NEXT: 13: clang-offload-packager, {7, 12}, image, (device-hip)
+// LTO-NEXT: 14: offload, "host-hip (x86_64-unknown-linux-gnu)" {2}, "device-hip (x86_64-unknown-linux-gnu)" {13}, ir
+// LTO-NEXT: 15: backend, {14}, assembler, (host-hip)
+// LTO-NEXT: 16: assembler, {15}, object, (host-hip)