[Clang] Introduce Clang Linker Wrapper Tool

[jhuber6]

Summary

This patch introduces a linker wrapper tool that allows us to preprocess
files before they are sent to the linker. This adds a dummy action and
job to the driver stage that builds the linker command as usual and then
replaces the command line with the wrapper tool.

Depends on D116543

[JonChesterfield]

I think this is reasonable. It's a lot of boilerplate to put the interception shim in the right place, but we do want a specific point to inject offloading functionality and this is guarded by OPT_fopenmp_new_driver so should be safe. It'll be easy to adjust later if the patches that build on it end up needing different behaviour.

[saiislam]

Can you please add this tool's doc in clang/docs?

[jhuber6]

Adding documentation for tool.

[jyknight]

"clang-linker-wrapper" seems like a very generic name for a command which is OpenMP offloading specific?

[jhuber6]

In D116544#3285369, @jyknight wrote:

"clang-linker-wrapper" seems like a very generic name for a command which is OpenMP offloading specific?

This could potentially be used to handle CUDA / HIP offloading as well, so I deliberately chose not bind it too tightly to OpenMP offloading. I can change the name if people think that it's too confusing.

clang/docs/ClangLinkerWrapper.rst

+====================
+Clang Linker Wrapper
+====================
+
+.. contents::
+   :local:
+
+.. _clang-linker-wrapper:
+
+Introduction
+============
+
+This tool works as a wrapper over a linking job. The tool is used to create
+linked device images for offloading. It scans the linker's input for embedded
+device offloading data stored in sections ``.llvm.offloading.<triple>.<arch>``
+and extracts it as a temporary file. The extracted device files will then be
+passed to a device linking job to create a final device image.
+
+Usage
+=====
+
+This tool can be used with the following options. Arguments to the host linker
+being wrapper around are passed as positional arguments using the ``--`` flag to
+override parsing.
+
+.. code-block:: console
+
+  USAGE: clang-linker-wrapper [options] <options to be passed to linker>...
+  
+  OPTIONS:
+  
+  Generic Options:
+  
+    --help                    - Display available options (--help-hidden for more)
+    --help-list               - Display list of available options (--help-list-hidden for more)
+    --version                 - Display the version of this program
+  
+  clang-linker-wrapper options:
+  
+    --host-triple=<string>    - Triple to use for the host compilation
+    --linker-path=<string>    - Path of linker binary
+    --opt-level=<string>      - Optimization level for LTO
+    --ptxas-option=<string>   - Argument to pass to the ptxas invocation
+    --save-temps              - Save intermediary results.
+    --strip-sections          - Strip offloading sections from the host object file.
+    --target-embed-bc         - Embed linked bitcode instead of an executable device image
+    --target-feature=<string> - Target features for triple
+    --target-library=<string> - Path for the target bitcode library
+    -v                        - Verbose output from tools
+
+Example
+=======
+
+This tool links object files with offloading images embedded within it using the
+``-fembed-offload-object`` flag in Clang. Given an input file containing the
+magic section we can pass it to this tool to extract the data contained at that
+section and run a device linking job on it.
+
+.. code-block:: console
+
+  clang-linker-wrapper -host-triple x86_64-unknown-linux-gnu -linker-path /usr/bin/ld -- <Args>

clang/docs/ReleaseNotes.rst

- GCC doesn't pack non-POD members in packed structs unless the packed
   such packing. Clang now matches the gcc behavior (except on Darwin and PS4).
   You can switch back to the old ABI behavior with the flag:
   ``-fclang-abi-compat=13.0``.
 
 OpenMP Support in Clang
 -----------------------
 
 - ``clang-nvlink-wrapper`` tool introduced to support linking of cubin files archived in an archive. See :doc:`ClangNvlinkWrapper`.
-
+- ``clang-linker-wrapper`` tool introduced to support linking using a new OpenMP target offloading method. See :doc:`ClangLinkerWrapper`.
 
 CUDA Support in Clang
 ---------------------
 
 - ...
 
 X86 Support in Clang
 --------------------

clang/include/clang/Driver/Action.h

enum ActionClass {
     IfsMergeJobClass,
     LipoJobClass,
     DsymutilJobClass,
     VerifyDebugInfoJobClass,
     VerifyPCHJobClass,
     OffloadBundlingJobClass,
     OffloadUnbundlingJobClass,
     OffloadWrapperJobClass,
+    LinkerWrapperJobClass,
     StaticLibJobClass,
 
     JobClassFirst = PreprocessJobClass,
     JobClassLast = StaticLibJobClass
   };
 
   // The offloading kind determines if this action is binded to a particular
   // programming model. Each entry reserves one bit. We also have a special kind

 public:
   OffloadWrapperJobAction(ActionList &Inputs, types::ID Type);
 
   static bool classof(const Action *A) {
     return A->getKind() == OffloadWrapperJobClass;
   }
 };
 
+class LinkerWrapperJobAction : public JobAction {
+  void anchor() override;
+
+public:
+  LinkerWrapperJobAction(ActionList &Inputs, types::ID Type);
+
+  static bool classof(const Action *A) {
+    return A->getKind() == LinkerWrapperJobClass;
+  }
+};
+
 class StaticLibJobAction : public JobAction {
   void anchor() override;
 
 public:
   StaticLibJobAction(ActionList &Inputs, types::ID Type);
 
   static bool classof(const Action *A) {
     return A->getKind() == StaticLibJobClass;
   }
 };
 
 } // namespace driver
 } // namespace clang
 
 #endif // LLVM_CLANG_DRIVER_ACTION_H

clang/include/clang/Driver/Job.h

public:
   /// \remark If the environment remains unset, then the environment
   ///         from the parent process will be used.
   virtual void setEnvironment(llvm::ArrayRef<const char *> NewEnvironment);
 
   void replaceArguments(llvm::opt::ArgStringList List) {
     Arguments = std::move(List);
   }
 
+  void replaceExecutable(const char *Exe) { Executable = Exe; }
+
   const char *getExecutable() const { return Executable; }
 
   const llvm::opt::ArgStringList &getArguments() const { return Arguments; }
 
   const std::vector<InputInfo> &getInputInfos() const { return InputInfoList; }
 
   const std::vector<std::string> &getOutputFilenames() const {
     return OutputFilenames;

clang/include/clang/Driver/ToolChain.h

private:
   mutable std::unique_ptr<Tool> Clang;
   mutable std::unique_ptr<Tool> Flang;
   mutable std::unique_ptr<Tool> Assemble;
   mutable std::unique_ptr<Tool> Link;
   mutable std::unique_ptr<Tool> StaticLibTool;
   mutable std::unique_ptr<Tool> IfsMerge;
   mutable std::unique_ptr<Tool> OffloadBundler;
   mutable std::unique_ptr<Tool> OffloadWrapper;
+  mutable std::unique_ptr<Tool> LinkerWrapper;
 
   Tool *getClang() const;
   Tool *getFlang() const;
   Tool *getAssemble() const;
   Tool *getLink() const;
   Tool *getStaticLibTool() const;
   Tool *getIfsMerge() const;
   Tool *getClangAs() const;
   Tool *getOffloadBundler() const;
   Tool *getOffloadWrapper() const;
+  Tool *getLinkerWrapper() const;
 
   mutable bool SanitizerArgsChecked = false;
   mutable std::unique_ptr<XRayArgs> XRayArguments;
 
   /// The effective clang triple for the current Job.
   mutable llvm::Triple EffectiveTriple;
 
   /// Set the toolchain's effective clang triple.

clang/lib/Driver/Action.cpp

const char *Action::getClassName(ActionClass AC) {
   case VerifyDebugInfoJobClass: return "verify-debug-info";
   case VerifyPCHJobClass: return "verify-pch";
   case OffloadBundlingJobClass:
     return "clang-offload-bundler";
   case OffloadUnbundlingJobClass:
     return "clang-offload-unbundler";
   case OffloadWrapperJobClass:
     return "clang-offload-wrapper";
+  case LinkerWrapperJobClass:
+    return "clang-linker-wrapper";
   case StaticLibJobClass:
     return "static-lib-linker";
   }
 
   llvm_unreachable("invalid class");
 }
 
 void Action::propagateDeviceOffloadInfo(OffloadKind OKind, const char *OArch) {
OffloadUnbundlingJobAction::OffloadUnbundlingJobAction(Action *Input)
     : JobAction(OffloadUnbundlingJobClass, Input, Input->getType()) {}
 
 void OffloadWrapperJobAction::anchor() {}
 
 OffloadWrapperJobAction::OffloadWrapperJobAction(ActionList &Inputs,
                                                  types::ID Type)
   : JobAction(OffloadWrapperJobClass, Inputs, Type) {}
 
+void LinkerWrapperJobAction::anchor() {}
+
+LinkerWrapperJobAction::LinkerWrapperJobAction(ActionList &Inputs,
+                                               types::ID Type)
+    : JobAction(LinkerWrapperJobClass, Inputs, Type) {}
+
 void StaticLibJobAction::anchor() {}
 
 StaticLibJobAction::StaticLibJobAction(ActionList &Inputs, types::ID Type)
     : JobAction(StaticLibJobClass, Inputs, Type) {}

clang/lib/Driver/Driver.cpp

void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   if (!LinkerInputs.empty()) {
     if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       if (Action *Wrapper = OffloadBuilder.makeHostLinkAction())
         LinkerInputs.push_back(Wrapper);
     Action *LA;
     // Check if this Linker Job should emit a static library.
     if (ShouldEmitStaticLibrary(Args)) {
       LA = C.MakeAction<StaticLibJobAction>(LinkerInputs, types::TY_Image);
+    } else if (Args.hasArg(options::OPT_fopenmp_new_driver) &&
+               OffloadKinds != Action::OFK_None) {
+      LA = C.MakeAction<LinkerWrapperJobAction>(LinkerInputs, types::TY_Image);
+      LA->propagateHostOffloadInfo(OffloadKinds,
+                                   /*BoundArch=*/nullptr);
     } else {
       LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
     }
     if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       LA = OffloadBuilder.processHostLinkAction(LA);
-    if (Args.hasArg(options::OPT_fopenmp_new_driver))
-      LA->propagateHostOffloadInfo(OffloadKinds,
-                                   /*BoundArch=*/nullptr);
     Actions.push_back(LA);
   }
 
   // Add an interface stubs merge action if necessary.
   if (!MergerInputs.empty())
     Actions.push_back(
         C.MakeAction<IfsMergeJobAction>(MergerInputs, types::TY_Image));
 

clang/lib/Driver/ToolChain.cpp

 }
 
 Tool *ToolChain::getOffloadWrapper() const {
   if (!OffloadWrapper)
     OffloadWrapper.reset(new tools::OffloadWrapper(*this));
   return OffloadWrapper.get();
 }
 
+Tool *ToolChain::getLinkerWrapper() const {
+  if (!LinkerWrapper)
+    LinkerWrapper.reset(new tools::LinkerWrapper(*this, getLink()));
+  return LinkerWrapper.get();
+}
+
 Tool *ToolChain::getTool(Action::ActionClass AC) const {
   switch (AC) {
   case Action::AssembleJobClass:
     return getAssemble();
 
   case Action::IfsMergeJobClass:
     return getIfsMerge();
 
case Action::BackendJobClass:
     return getClang();
 
   case Action::OffloadBundlingJobClass:
   case Action::OffloadUnbundlingJobClass:
     return getOffloadBundler();
 
   case Action::OffloadWrapperJobClass:
     return getOffloadWrapper();
+  case Action::LinkerWrapperJobClass:
+    return getLinkerWrapper();
   }
 
   llvm_unreachable("Invalid tool kind.");
 }
 
 static StringRef getArchNameForCompilerRTLib(const ToolChain &TC,
                                              const ArgList &Args) {
   const llvm::Triple &Triple = TC.getTriple();

clang/lib/Driver/ToolChains/Clang.h

public:
 
   bool hasIntegratedCPP() const override { return false; }
   void ConstructJob(Compilation &C, const JobAction &JA,
                     const InputInfo &Output, const InputInfoList &Inputs,
                     const llvm::opt::ArgList &TCArgs,
                     const char *LinkingOutput) const override;
 };
 
+/// Linker wrapper tool.
+class LLVM_LIBRARY_VISIBILITY LinkerWrapper final : public Tool {
+  const Tool *Linker;
+
+public:
+  LinkerWrapper(const ToolChain &TC, const Tool *Linker)
+      : Tool("Offload::Linker", "linker", TC), Linker(Linker) {}
+
+  bool hasIntegratedCPP() const override { return false; }
+  void ConstructJob(Compilation &C, const JobAction &JA,
+                    const InputInfo &Output, const InputInfoList &Inputs,
+                    const llvm::opt::ArgList &TCArgs,
+                    const char *LinkingOutput) const override;
+};
+
 } // end namespace tools
 
 } // end namespace driver
 } // end namespace clang
 
 #endif // LLVM_CLANG_LIB_DRIVER_TOOLCHAINS_CLANG_H

clang/lib/Driver/ToolChains/Clang.cpp

for (const InputInfo &I : Inputs) {
     CmdArgs.push_back(I.getFilename());
   }
 
   C.addCommand(std::make_unique<Command>(
       JA, *this, ResponseFileSupport::None(),
       Args.MakeArgString(getToolChain().GetProgramPath(getShortName())),
       CmdArgs, Inputs, Output));
 }
+
+void LinkerWrapper::ConstructJob(Compilation &C, const JobAction &JA,
+                                 const InputInfo &Output,
+                                 const InputInfoList &Inputs,
+                                 const ArgList &Args,
+                                 const char *LinkingOutput) const {
+  ArgStringList CmdArgs;
+
+  // Construct the link job so we can wrap around it.
+  Linker->ConstructJob(C, JA, Output, Inputs, Args, LinkingOutput);
+  const auto &LinkCommand = C.getJobs().getJobs().back();
+
+  CmdArgs.push_back("-linker-path");
+  CmdArgs.push_back(LinkCommand->getExecutable());
+  for (const char *LinkArg : LinkCommand->getArguments())
+    CmdArgs.push_back(LinkArg);
+
+  const char *Exec =
+      Args.MakeArgString(getToolChain().GetProgramPath("clang-linker-wrapper"));
+
+  // Replace the executable and arguments associated with the link job to the
+  // wrapper.
+  LinkCommand->replaceExecutable(Exec);
+  LinkCommand->replaceArguments(CmdArgs);
+}

clang/tools/CMakeLists.txt

 create_subdirectory_options(CLANG TOOL)
 
 add_clang_subdirectory(diagtool)
 add_clang_subdirectory(driver)
 add_clang_subdirectory(apinotes-test)
 add_clang_subdirectory(clang-diff)
 add_clang_subdirectory(clang-format)
 add_clang_subdirectory(clang-format-vs)
 add_clang_subdirectory(clang-fuzzer)
 add_clang_subdirectory(clang-import-test)
 add_clang_subdirectory(clang-nvlink-wrapper)
+add_clang_subdirectory(clang-linker-wrapper)
 add_clang_subdirectory(clang-offload-bundler)
 add_clang_subdirectory(clang-offload-wrapper)
 add_clang_subdirectory(clang-scan-deps)
 add_clang_subdirectory(clang-repl)
 
 add_clang_subdirectory(c-index-test)
 
 add_clang_subdirectory(clang-rename)

clang/tools/clang-linker-wrapper/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS BitWriter Core Object Support)
+
+if(NOT CLANG_BUILT_STANDALONE)
+  set(tablegen_deps intrinsics_gen)
+endif()
+
+add_clang_executable(clang-linker-wrapper
+  ClangLinkerWrapper.cpp
+
+  DEPENDS
+  ${tablegen_deps}
+  )
+
+set(CLANG_LINKER_WRAPPER_LIB_DEPS
+  clangBasic
+  )
+
+add_dependencies(clang clang-linker-wrapper)
+
+target_link_libraries(clang-linker-wrapper
+  PRIVATE
+  ${CLANG_LINKER_WRAPPER_LIB_DEPS}
+  )
+
+install(TARGETS clang-linker-wrapper RUNTIME DESTINATION bin)

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp

+//===-- clang-linker-wrapper/ClangLinkerWrapper.cpp - wrapper over linker-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===---------------------------------------------------------------------===//
+///
+//===---------------------------------------------------------------------===//
+
+#include "clang/Basic/Version.h"
+#include "llvm/Object/Archive.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/StringSaver.h"
+#include "llvm/Support/WithColor.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+static cl::opt<bool> Help("h", cl::desc("Alias for -help"), cl::Hidden);
+
+// Mark all our options with this category, everything else (except for -help)
+// will be hidden.
+static cl::OptionCategory
+    ClangLinkerWrapperCategory("clang-linker-wrapper options");
+
+static cl::opt<std::string> LinkerUserPath("linker-path",
+                                           cl::desc("Path of linker binary"),
+                                           cl::cat(ClangLinkerWrapperCategory));
+
+// Do not parse linker options
+static cl::list<std::string>
+    LinkerArgs(cl::Sink, cl::desc("<options to be passed to linker>..."));
+
+static Error runLinker(std::string LinkerPath,
+                       SmallVectorImpl<std::string> &Args) {
+  std::vector<StringRef> LinkerArgs;
+  LinkerArgs.push_back(LinkerPath);
+  for (auto &Arg : Args)
+    LinkerArgs.push_back(Arg);
+
+  if (sys::ExecuteAndWait(LinkerPath, LinkerArgs))
+    return createStringError(inconvertibleErrorCode(), "'linker' failed");
+  return Error::success();
+}
+
+static void PrintVersion(raw_ostream &OS) {
+  OS << clang::getClangToolFullVersion("clang-linker-wrapper") << '\n';
+}
+
+int main(int argc, const char **argv) {
+  sys::PrintStackTraceOnErrorSignal(argv[0]);
+  cl::SetVersionPrinter(PrintVersion);
+  cl::HideUnrelatedOptions(ClangLinkerWrapperCategory);
+  cl::ParseCommandLineOptions(
+      argc, argv,
+      "A wrapper utility over the host linker. It scans the input files for\n"
+      "sections that require additional processing prior to linking. The tool\n"
+      "will then transparently pass all arguments and input to the specified\n"
+      "host linker to create the final binary.\n");
+
+  if (Help) {
+    cl::PrintHelpMessage();
+    return EXIT_SUCCESS;
+  }
+
+  auto reportError = [argv](Error E) {
+    logAllUnhandledErrors(std::move(E), WithColor::error(errs(), argv[0]));
+    exit(EXIT_FAILURE);
+  };
+
+  // TODO: Scan input object files for offloading sections and extract them.
+  // TODO: Perform appropriate device linking action.
+  // TODO: Wrap device image in a host binary and pass it to the linker.
+  WithColor::warning(errs(), argv[0]) << "Offload linking not yet supported.\n";
+
+  SmallVector<std::string, 0> Argv;
+  for (const std::string &Arg : LinkerArgs)
+    Argv.push_back(Arg);
+
+  if (Error Err = runLinker(LinkerUserPath, Argv))
+    reportError(std::move(Err));
+
+  return EXIT_SUCCESS;
+}