[Clang] Introduce `--offload-link` option to perform offload device linking

[jhuber6]

Summary

The new driver uses an augmented linker wrapper to perform the device
linking phase, but to the user looks like a regular linker invocation.
Contrary to the old driver, the new driver contains all the information
necessary to produce a linked device image in the host object itself.
Currently, we infer the usage of the device linker by the user
specifying an offloading toolchain, e.g. (--offload-arch=...) or
(-fopenmp-targets=...), but this shouldn't be strictly necessary.
This patch introduces a new option --offload-link to tell
the driver to use the offloading linker instead. So a compilation flow
can now look like this,

clang foo.cu --offload-new-driver -fgpu-rdc --offload-arch=sm_70 -c
clang foo.o --offload-link -lcudart

I was considering if this could be merged into the -fuse-ld option,
but because the device linker wraps over the users linker it would
conflict with that. In the future it's possible to merge this into lld
completely or gold via a plugin and we would use this option to
enable the device linking feature. Let me know what you think for this.

[tra]

Currently, we infer the usage of the device linker by the user
specifying an offloading toolchain, e.g. (--offload-arch=...) or
(-fopenmp-targets=...), but this shouldn't be strictly necessary.

Yup. Whether we want to perform device link or not is orthogonal to those options.

This patch introduces a new option -dl to tell the driver to use the
offloading linker instead. So a compilation flow can now look like this,

clang foo.cu --offload-new-driver -fgpu-rdc --offload-arch=sm_70 -c
clang foo.o -dl -lcudart

It's an essential feature, as we do want to be able to produce libraries with host object files, but with fully linked GPU executables.
Naming, as usual, is hard. I would prefer a more explicit --offload-link which would be in line with other --offload* options we have by now.
-dl is cryptic for uninitiated and is uncomfortably close to commonly used -ldl. If it gets mistyped as -ld, it would lead to a legitimate but unrelated error about missing libd. Or it might silently succeed linking with libd without actually doing any device linking.

I was considering if this could be merged into the -fuse-ld option,
but becuse the device linker wraps over the users linker it would
conflict with that. In the future it's possible to merge this into lld
completely or gold via a plugin. Let me know what you think for this.

[jhuber6]

In D126398#3537942, @tra wrote:

Naming, as usual, is hard. I would prefer a more explicit --offload-link which would be in line with other --offload* options we have by now.
-dl is cryptic for uninitiated and is uncomfortably close to commonly used -ldl. If it gets mistyped as -ld, it would lead to a legitimate but unrelated error about missing libd. Or it might silently succeed linking with libd without actually doing any device linking.

Yeah, I can see your point, --offload-link definitely works but it would be nice to have something less verbose. Maybe could just use -dlink or something.

I was considering if this could be merged into the -fuse-ld option,
but becuse the device linker wraps over the users linker it would
conflict with that. In the future it's possible to merge this into lld
completely or gold via a plugin. Let me know what you think for this.

I should also add, even if we built-in support for this into LLD or Gold, we'll probably still need a flag like this to tell Gold to use the plugin, or LLD to do the extra processing.

Unrelated, but in the future I'm also considering making the linker wrapper add the necessary libraries for whatever offloading kinds it found. E.g. if the link job finds embedded OpenMP code it will add -lomp -lomptarget if not already present.

[jhuber6]

Changing to use --offload-link and use -dlink as an alias.

[tra]

clang/include/clang/Driver/Options.td

825–826

We typically use option aliases to provide compatibility with the legacy options. AFAICT there are no current uses of Alias for the sake of saving a few characters in an option name.

Is -dlink really needed? It's not going to be typed manually all that often, and a slightly longer option does not make any difference for cmake or make files.

I assume that partial motivation for -dlink is that it's a shortened alias used by NVCC for its functionally similar --device-link option. I do not think it buys us anything. We never intended to be option-compatible with nvcc and clang's CUDA compilation and relevant options have only partial overlap with NVCC's functionality-wise and almost none syntax-wise. Adding one rarely used option for the same of matching NVCC's is not worth it, IMO.

[jhuber6]

clang/include/clang/Driver/Options.td

825–826

Yeah, it's somewhat mental compatibility with Nvidia parlance, see -fgpu-rdc and -rdc=true, if you think that's not necessary I'll just make it --offload-link.

[jhuber6]

Removing -dlink

clang/docs/ClangCommandLineReference.rst

 .. option:: -Ttext<addr>
 
 Set starting address of TEXT to <addr>
 
 .. option:: -Wl,<arg>,<arg2>...
 
 Pass the comma separated arguments in <arg> to the linker
 
+.. option:: -dlink, --offload-link
+
+Use the linker supporting offloading device linking.
+
 .. option:: -X
 
 .. option:: -Xlinker <arg>, --for-linker <arg>, --for-linker=<arg>
 
 Pass <arg> to the linker
 
 .. option:: -Xoffload-linker <arg>, -Xoffload-linker-<triple> <arg>
 

clang/include/clang/Driver/Options.td

 def Xopenmp_target : Separate<["-"], "Xopenmp-target">, Group<CompileOnly_Group>,
   HelpText<"Pass <arg> to the target offloading toolchain.">, MetaVarName<"<arg>">;
 def Xopenmp_target_EQ : JoinedAndSeparate<["-"], "Xopenmp-target=">, Group<CompileOnly_Group>,
   HelpText<"Pass <arg> to the target offloading toolchain identified by <triple>.">,
   MetaVarName<"<triple> <arg>">;
 def z : Separate<["-"], "z">, Flags<[LinkerInput, RenderAsInput]>,
   HelpText<"Pass -z <arg> to the linker">, MetaVarName<"<arg>">,
   Group<Link_Group>;
+def offload_link : Flag<["--"], "offload-link">, Group<Link_Group>,
+  HelpText<"Use the new offloading linker to perform the link job.">;
+def device_link : Flag<["-"], "dlink">, Group<Link_Group>,
+  Alias<offload_link>;
 def Xlinker : Separate<["-"], "Xlinker">, Flags<[LinkerInput, RenderAsInput]>,
   HelpText<"Pass <arg> to the linker">, MetaVarName<"<arg>">,
   Group<Link_Group>;
 def Xoffload_linker : JoinedAndSeparate<["-"], "Xoffload-linker">,
   HelpText<"Pass <arg> to the offload linkers or the ones idenfied by -<triple>">, 
   MetaVarName<"<triple> <arg>">, Group<Link_Group>;
 def Xpreprocessor : Separate<["-"], "Xpreprocessor">, Group<Preprocessor_Group>,
   HelpText<"Pass <arg> to the preprocessor">, MetaVarName<"<arg>">;

clang/lib/Driver/Driver.cpp

void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   if (!LinkerInputs.empty()) {
     if (!UseNewOffloadingDriver)
       if (Action *Wrapper = OffloadBuilder.makeHostLinkAction())
         LinkerInputs.push_back(Wrapper);
     Action *LA;
     // Check if this Linker Job should emit a static library.
     if (ShouldEmitStaticLibrary(Args)) {
       LA = C.MakeAction<StaticLibJobAction>(LinkerInputs, types::TY_Image);
-    } else if (UseNewOffloadingDriver) {
+    } else if (UseNewOffloadingDriver ||
+               Args.hasArg(options::OPT_offload_link)) {
       LA = C.MakeAction<LinkerWrapperJobAction>(LinkerInputs, types::TY_Image);
       LA->propagateHostOffloadInfo(C.getActiveOffloadKinds(),
                                    /*BoundArch=*/nullptr);
     } else {
       LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
     }
     if (!UseNewOffloadingDriver)
       LA = OffloadBuilder.processHostLinkAction(LA);

clang/test/Driver/cuda-openmp-driver.cu

 // RUN: | FileCheck -check-prefix BINDINGS-DEVICE %s
 
 // BINDINGS-DEVICE: # "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[PTX:.+]]"
 // BINDINGS-DEVICE: # "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[PTX]]"], output: "[[CUBIN:.+]]"
 // BINDINGS-DEVICE: # "nvptx64-nvidia-cuda" - "NVPTX::Linker", inputs: ["[[CUBIN]]", "[[PTX]]"], output: "{{.*}}.fatbin"
 
 // RUN: %clang -### -target x86_64-linux-gnu -nocudalib --cuda-feature=+ptx61 --offload-arch=sm_70 %s 2>&1 | FileCheck -check-prefix MANUAL-FEATURE %s
 // MANUAL-FEATURE: -cc1{{.*}}-target-feature{{.*}}+ptx61
+
+// RUN: %clang -### -target x86_64-linux-gnu -nocudalib -ccc-print-bindings -dlink %s 2>&1 \
+// RUN: | FileCheck -check-prefix DEVICE-LINK %s
+
+// DEVICE-LINK: "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[INPUT:.+]]"], output: "a.out"