[OpenMP] Introduce new flag to change offloading driver pipeline

[jhuber6]

Summary

This patch introduces the -fopenmp-new-driver option which instructs
the compiler to use a new driver scheme for producing offloading code.
In this scheme we create a complete offloading object file and then pass
it as input to the host compilation phase. This will allow us to embed
the object code in the backend phase.

This is the start of a series of commits to rework the OpenMP offloading driver
pipeline. The goal of this is to simplify the steps required for creating an
offloading program. This patch changes the driver's configuration to simply pass
the device file back to the host as an input so it can be embedded as an LLVM IR
global during the backend, then simply passes that object file to the linker.

This driver implementation will currently create the following phases,

$ clang input.c -fopenmp -fopenmp-targets=nvptx64 -fopenmp-new-driver -ccc-print-phases
               +- 0: input, "input.c", c, (host-openmp)
            +- 1: preprocessor, {0}, cpp-output, (host-openmp)
         +- 2: compiler, {1}, ir, (host-openmp)
         |        |     +- 3: input, "input.c", c, (device-openmp)
         |        |  +- 4: preprocessor, {3}, cpp-output, (device-openmp)
         |        |- 5: compiler, {4}, ir, (device-openmp)
         |     +- 6: offload, "host-openmp (x86_64-unknown-linux-gnu)" {2}, "device-openmp (nvptx64)" {5}, ir
         |  +- 7: backend, {6}, assembler, (device-openmp)
         |- 8: assembler, {7}, object, (device-openmp)
      +- 9: offload, "host-openmp (x86_64-unknown-linux-gnu)" {2}, "device-openmp (nvptx64)" {8}, ir
   +- 10: backend, {9}, assembler, (host-openmp)
+- 11: assembler, {10}, object, (host-openmp)
12: clang-linker-wrapper, {11}, image, (host-openmp)

Which will map to the following bindings

# "x86_64-unknown-linux-gnu" - "clang", inputs: ["input.c"], output: "/tmp/input-bae62e.bc"
# "nvptx64" - "clang", inputs: ["input.c", "/tmp/input-bae62e.bc"], output: "/tmp/input-76784e.s"
# "nvptx64" - "NVPTX::Assembler", inputs: ["/tmp/input-76784e.s"], output: "/tmp/input-8f29db.o"
# "x86_64-unknown-linux-gnu" - "clang", inputs: ["/tmp/input-bae62e.bc", "/tmp/input-8f29db.o"], output: "/tmp/input-545450.o"
# "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["/tmp/input-545450.o"], output: "a.out"

[jhuber6]

Updating to only contain this commit.

[JonChesterfield]

This looks reasonable to me, though I'd prefer we keep the forward declare. The scalar->vector transform is mechanical and the if (Args.hasArg(options::OPT_fopenmp_new_driver)) cruft will disappear when we return to a single driver implementation.

clang/include/clang/Driver/Driver.h

45

This looks like it should be a breaking change - InputInfo is no longer forward declared. Would it be reasonable to keep the forward declaration and put the typedef between class statements and the LTOKind enum?

[jhuber6]

clang/include/clang/Driver/Driver.h

45

We can't forward declare a struct and use it by-value in a container, I would need to change it to a pointer. It's doable, but I don't think it's ideal. I'm not sure why this would break anything, the forward declarations simply were there to avoid including more files here. I could be wrong however.

[thakis]

Looks like this breaks tests on macOS: http://45.33.8.238/macm1/26856/step_7.txt

Please take a look and revert for now if it takes a while to fix (maybe just needs an explicit triple?)

[jhuber6]

In D116541#3285455, @thakis wrote:

Looks like this breaks tests on macOS: http://45.33.8.238/macm1/26856/step_7.txt

Please take a look and revert for now if it takes a while to fix (maybe just needs an explicit triple?)

Not sure what I expected when I hard-coded the host-triple in the test. I pushed a change in rGb79e2a1ccd3b, can you check it again?

[thakis]

Still failing: http://45.33.8.238/macm1/26873/step_7.txt

[jhuber6]

In D116541#3285927, @thakis wrote:

Still failing: http://45.33.8.238/macm1/26873/step_7.txt

Weird, can you show me what -fopenmp -fopenmp-targets=nvptx64 -fopenmp-new-driver -ccc-print-bindings looks like there? I'm not sure why but it doesn't seem to be getting one of the expected arguments but I'm not sure how to reproduce it.

[jhuber6]

In D116541#3285927, @thakis wrote:

Still failing: http://45.33.8.238/macm1/26873/step_7.txt

It seems what's happening here is that we are building the host.bc twice, this will work fine but isn't ideal. I prevent this manually by checking the cache if one of the jobs was already created, but for some reason that doesn't seem to be happening here. I'll need to figure out how to reproduce this.

[thakis]

Tests have been failing on Mac for over 20 hours now. Time to revert and fix async?

 % bin/clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -fopenmp-new-driver -no-canonical-prefixes -ccc-print-bindings /Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c -o openmp-offload-gpu -fopenmp -fopenmp-targets=nvptx64 -fopenmp-new-driver -ccc-print-bindings
clang version 14.0.0
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: bin
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"
# "nvptx64-nvidia-cuda" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"
# "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"
# "x86_64-apple-darwin19.6.0" - "Offload::Linker", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"], output: "openmp-offload-gpu"

[jhuber6]

In D116541#3287330, @thakis wrote:

Tests have been failing on Mac for over 20 hours now. Time to revert and fix async?

 % bin/clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -fopenmp-new-driver -no-canonical-prefixes -ccc-print-bindings /Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c -o openmp-offload-gpu -fopenmp -fopenmp-targets=nvptx64 -fopenmp-new-driver -ccc-print-bindings
clang version 14.0.0
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: bin
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"
# "nvptx64-nvidia-cuda" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"
# "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"
# "x86_64-apple-darwin19.6.0" - "Offload::Linker", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"], output: "openmp-offload-gpu"

I'd rather just disable the test, this patch has like 20 others that depend on it so we'd need to revert all of those. It's definitely not grabbing the bitcode from the cache as expected. I can disable this part of the test for now, but do you know how I could figure out how to reproduce this? I've been tracking some other buildbots and they don't seem to have the same issue so I'm not sure what's special about this one.

[thakis]

In D116541#3287342, @jhuber6 wrote:

In D116541#3287330, @thakis wrote:

Tests have been failing on Mac for over 20 hours now. Time to revert and fix async?

 % bin/clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda -fopenmp-new-driver -no-canonical-prefixes -ccc-print-bindings /Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c -o openmp-offload-gpu -fopenmp -fopenmp-targets=nvptx64 -fopenmp-new-driver -ccc-print-bindings
clang version 14.0.0
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: bin
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"
# "nvptx64-nvidia-cuda" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-729b05.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"
# "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a35969.s"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/Users/thakis/src/llvm-project/clang/test/Driver/openmp-offload-gpu.c", "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-a3f7f0.o"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"
# "x86_64-apple-darwin19.6.0" - "clang", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-f53552.bc"], output: "/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"
# "x86_64-apple-darwin19.6.0" - "Offload::Linker", inputs: ["/var/folders/qt/hxckwtm545l643cnk200wzt00000gn/T/openmp-offload-gpu-86f846.o"], output: "openmp-offload-gpu"

I'd rather just disable the test, this patch has like 20 others that depend on it so we'd need to revert all of those. It's definitely not grabbing the bitcode from the cache as expected. I can disable this part of the test for now, but do you know how I could figure out how to reproduce this? I've been tracking some other buildbots and they don't seem to have the same issue so I'm not sure what's special about this one.

Just build and run tests on any mac. This fails on 3 different macs I tried (2x arm, 1x intel), in a bunch of different build configs.

For the particular build I sent the output from, the cmake invocation looked like /Applications/CMake.app/Contents/bin/cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS='compiler-rt;libcxx;clang' -DLLVM_APPEND_VC_REV=NO -DCMAKE_C_COMPILER=$HOME/src/chrome/src/third_party/llvm-build/Release+Asserts/bin/clang -DCMAKE_CXX_COMPILER=$HOME/src/chrome/src/third_party/llvm-build/Release+Asserts/bin/clang++ -DCMAKE_OSX_SYSROOT=/Users/thakis/src/llvm-project/sysroot/MacOSX.sdk -DDARWIN_macosx_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/MacOSX.sdk -DDARWIN_iphoneos_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/iPhoneOS.sdk -DDARWIN_iphonesimulator_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/iPhoneSimulator.sdk ../llvm

[thakis]

ps "This is inconvenient to revert since 20 patches landed" means you're landing too many patches too quickly. This landed less than 24h ago, so if it's difficult to revert, that's a bit of a problem for the project, right?

[jhuber6]

In D116541#3287379, @thakis wrote:

Just build and run tests on any mac. This fails on 3 different macs I tried (2x arm, 1x intel), in a bunch of different build configs.

For the particular build I sent the output from, the cmake invocation looked like /Applications/CMake.app/Contents/bin/cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS='compiler-rt;libcxx;clang' -DLLVM_APPEND_VC_REV=NO -DCMAKE_C_COMPILER=$HOME/src/chrome/src/third_party/llvm-build/Release+Asserts/bin/clang -DCMAKE_CXX_COMPILER=$HOME/src/chrome/src/third_party/llvm-build/Release+Asserts/bin/clang++ -DCMAKE_OSX_SYSROOT=/Users/thakis/src/llvm-project/sysroot/MacOSX.sdk -DDARWIN_macosx_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/MacOSX.sdk -DDARWIN_iphoneos_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/iPhoneOS.sdk -DDARWIN_iphonesimulator_CACHED_SYSROOT=/Users/thakis/src/llvm-project/sysroot/iPhoneSimulator.sdk ../llvm

I don't have access to a mac computer right now. I'm just going to remove the problematic check line so this passes and add it in later once I figure out what's going on here. The output you're getting should still work, it's just not ideal because we're regenerating the bitcode file so this isn't a breaking issue.

[jhuber6]

Removed the two lines in rG28c15341368b, let me know if this lets the tests pass. I'll look into getting an access somehow so I can reproduce this and figure it out.

[thakis]

Tests are passing again.

clang/include/clang/Driver/Driver.h

 //===--- Driver.h - Clang GCC Compatible Driver -----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_DRIVER_DRIVER_H
 #define LLVM_CLANG_DRIVER_DRIVER_H
 
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Driver/Action.h"
+#include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/Phases.h"
 #include "clang/Driver/ToolChain.h"
 #include "clang/Driver/Types.h"
 #include "clang/Driver/Util.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Option/Arg.h"

 class FileSystem;
 }
 } // namespace llvm
 
 namespace clang {
 
 namespace driver {
 
-  class Command;
-  class Compilation;
-  class InputInfo;
-  class JobList;
-  class JobAction;
-  class SanitizerArgs;
-  class ToolChain;
+typedef SmallVector<InputInfo, 4> InputInfoList;
+
 
 /// Describes the kind of LTO mode selected via -f(no-)?lto(=.*)? options.
 enum LTOKind {
   LTOK_None,
   LTOK_Full,
   LTOK_Thin,
   LTOK_Unknown
 };
public:
   std::string CCPrintOptionsFilename;
 
   /// The file to log CC_PRINT_HEADERS output to, if enabled.
   std::string CCPrintHeadersFilename;
 
   /// The file to log CC_LOG_DIAGNOSTICS output to, if enabled.
   std::string CCLogDiagnosticsFilename;
 
+  /// An input type and its arguments.
+  using InputTy = std::pair<types::ID, const llvm::opt::Arg *>;
+
   /// A list of inputs and their types for the given arguments.
-  typedef SmallVector<std::pair<types::ID, const llvm::opt::Arg *>, 16>
-      InputList;
+  using InputList = SmallVector<InputTy, 16>;
 
   /// Whether the driver should follow g++ like behavior.
   bool CCCIsCXX() const { return Mode == GXXMode; }
 
   /// Whether the driver is just the preprocessor.
   bool CCCIsCPP() const { return Mode == CPPMode; }
 
   /// Whether the driver should follow gcc like behavior.
public:
   /// BuildUniversalActions - Construct the list of actions to perform
   /// for the given arguments, which may require a universal build.
   ///
   /// \param C - The compilation that is being built.
   /// \param TC - The default host tool chain.
   void BuildUniversalActions(Compilation &C, const ToolChain &TC,
                              const InputList &BAInputs) const;
 
+  /// BuildOffloadingActions - Construct the list of actions to perform for the
+  /// offloading toolchain that will be embedded in the host.
+  ///
+  /// \param C - The compilation that is being built.
+  /// \param Args - The input arguments.
+  /// \param Input - The input type and arguments
+  /// \param HostAction - The host action used in the offloading toolchain.
+  Action *BuildOffloadingActions(Compilation &C,
+                                 llvm::opt::DerivedArgList &Args,
+                                 const InputTy &Input,
+                                 Action *HostAction) const;
+
   /// Check that the file referenced by Value exists. If it doesn't,
   /// issue a diagnostic and return false.
   /// If TypoCorrect is true and the file does not exist, see if it looks
   /// like a likely typo for a flag and if so print a "did you mean" blurb.
   bool DiagnoseInputExistence(const llvm::opt::DerivedArgList &Args,
                               StringRef Value, types::ID Ty,
                               bool TypoCorrect) const;
 
public:
   Action *ConstructPhaseAction(
       Compilation &C, const llvm::opt::ArgList &Args, phases::ID Phase,
       Action *Input,
       Action::OffloadKind TargetDeviceOffloadKind = Action::OFK_None) const;
 
   /// BuildJobsForAction - Construct the jobs to perform for the action \p A and
   /// return an InputInfo for the result of running \p A.  Will only construct
   /// jobs for a given (Action, ToolChain, BoundArch, DeviceKind) tuple once.
-  InputInfo
-  BuildJobsForAction(Compilation &C, const Action *A, const ToolChain *TC,
-                     StringRef BoundArch, bool AtTopLevel, bool MultipleArchs,
-                     const char *LinkingOutput,
-                     std::map<std::pair<const Action *, std::string>, InputInfo>
-                         &CachedResults,
-                     Action::OffloadKind TargetDeviceOffloadKind) const;
+  InputInfoList BuildJobsForAction(
+      Compilation &C, const Action *A, const ToolChain *TC, StringRef BoundArch,
+      bool AtTopLevel, bool MultipleArchs, const char *LinkingOutput,
+      std::map<std::pair<const Action *, std::string>, InputInfoList>
 
   /// Returns the default name for linked images (e.g., "a.out").
   const char *getDefaultImageName() const;
 
   /// GetNamedOutputPath - Return the name to use for the output of
   /// the action \p JA. The result is appended to the compilation's
   /// list of temporary or result files, as appropriate.
   ///
private:
 
   /// Get bitmasks for which option flags to include and exclude based on
   /// the driver mode.
   std::pair<unsigned, unsigned> getIncludeExcludeOptionFlagMasks(bool IsClCompatMode) const;
 
   /// Helper used in BuildJobsForAction.  Doesn't use the cache when building
   /// jobs specifically for the given action, but will use the cache when
   /// building jobs for the Action's inputs.
-  InputInfo BuildJobsForActionNoCache(
+  InputInfoList BuildJobsForActionNoCache(
       Compilation &C, const Action *A, const ToolChain *TC, StringRef BoundArch,
       bool AtTopLevel, bool MultipleArchs, const char *LinkingOutput,
-      std::map<std::pair<const Action *, std::string>, InputInfo>
+      std::map<std::pair<const Action *, std::string>, InputInfoList>
           &CachedResults,
       Action::OffloadKind TargetDeviceOffloadKind) const;
 
 public:
   /// GetReleaseVersion - Parse (([0-9]+)(.([0-9]+)(.([0-9]+)?))?)? and
   /// return the grouped values as integers. Numbers which are not
   /// provided are set to 0.
   ///

clang/include/clang/Driver/Options.td

defm openmp_target_new_runtime: BoolFOption<"openmp-target-new-runtime",
   LangOpts<"OpenMPTargetNewRuntime">, DefaultTrue,
   PosFlag<SetTrue, [CC1Option], "Use the new bitcode library for OpenMP offloading">,
   NegFlag<SetFalse>>;
 defm openmp_optimistic_collapse : BoolFOption<"openmp-optimistic-collapse",
   LangOpts<"OpenMPOptimisticCollapse">, DefaultFalse,
   PosFlag<SetTrue, [CC1Option]>, NegFlag<SetFalse>, BothFlags<[NoArgumentUnused, HelpHidden]>>;
 def static_openmp: Flag<["-"], "static-openmp">,
   HelpText<"Use the static host OpenMP runtime while linking.">;
+def fopenmp_new_driver : Flag<["-"], "fopenmp-new-driver">, Flags<[CC1Option]>, Group<Action_Group>,
+  HelpText<"Use the new driver for OpenMP offloading.">;
 def fno_optimize_sibling_calls : Flag<["-"], "fno-optimize-sibling-calls">, Group<f_Group>;
 def foptimize_sibling_calls : Flag<["-"], "foptimize-sibling-calls">, Group<f_Group>;
 defm escaping_block_tail_calls : BoolFOption<"escaping-block-tail-calls",
   CodeGenOpts<"NoEscapingBlockTailCalls">, DefaultFalse,
   NegFlag<SetTrue, [CC1Option]>, PosFlag<SetFalse>>;
 def force__cpusubtype__ALL : Flag<["-"], "force_cpusubtype_ALL">;
 def force__flat__namespace : Flag<["-"], "force_flat_namespace">;
 def force__load : Separate<["-"], "force_load">;

clang/lib/Driver/Driver.cpp

if (Arg *A = Args.getLastArg(options::OPT__SLASH_o)) {
     }
   }
 
   handleArguments(C, Args, Inputs, Actions);
 
   // Builder to be used to build offloading actions.
   OffloadingActionBuilder OffloadBuilder(C, Args, Inputs);
 
+  // Offload kinds active for this compilation.
+  unsigned OffloadKinds = Action::OFK_None;
+  if (C.hasOffloadToolChain<Action::OFK_OpenMP>())
+    OffloadKinds |= Action::OFK_OpenMP;
+
   // Construct the actions to perform.
   HeaderModulePrecompileJobAction *HeaderModuleAction = nullptr;
   ActionList LinkerInputs;
   ActionList MergerInputs;
 
   for (auto &I : Inputs) {
     types::ID InputType = I.first;
     const Arg *InputArg = I.second;
 
     auto PL = types::getCompilationPhases(*this, Args, InputType);
     if (PL.empty())
       continue;
 
     auto FullPL = types::getCompilationPhases(InputType);
 
     // Build the pipeline for this file.
     Action *Current = C.MakeAction<InputAction>(*InputArg, InputType);
 
     // Use the current host action in any of the offloading actions, if
     // required.
+    if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       if (OffloadBuilder.addHostDependenceToDeviceActions(Current, InputArg))
         break;
 
     for (phases::ID Phase : PL) {
 
       // Add any offload action the host action depends on.
+      if (!Args.hasArg(options::OPT_fopenmp_new_driver))
         Current = OffloadBuilder.addDeviceDependencesToHostAction(
             Current, InputArg, Phase, PL.back(), FullPL);
       if (!Current)
         break;
 
       // Queue linker inputs.
       if (Phase == phases::Link) {
         assert(Phase == PL.back() && "linking must be final compilation step.");
         LinkerInputs.push_back(Current);
         Current = nullptr;
for (phases::ID Phase : PL) {
       // separate PCH.
       if (Phase == phases::Precompile && HeaderModuleAction &&
           getPrecompiledType(InputType) == types::TY_PCH) {
         HeaderModuleAction->addModuleHeaderInput(Current);
         Current = nullptr;
         break;
       }
 
+      // Try to build the offloading actions and add the result as a dependency
+      // to the host.
+      if (Args.hasArg(options::OPT_fopenmp_new_driver))
+        Current = BuildOffloadingActions(C, Args, I, Current);
+
       // FIXME: Should we include any prior module file outputs as inputs of
       // later actions in the same command line?
 
       // Otherwise construct the appropriate action.
       Action *NewCurrent = ConstructPhaseAction(C, Args, Phase, Current);
 
       // We didn't create a new action, so we will just move to the next phase.
       if (NewCurrent == Current)
         continue;
 
       if (auto *HMA = dyn_cast<HeaderModulePrecompileJobAction>(NewCurrent))
         HeaderModuleAction = HMA;
 
       Current = NewCurrent;
 
       // Use the current host action in any of the offloading actions, if
       // required.
+      if (!Args.hasArg(options::OPT_fopenmp_new_driver))
         if (OffloadBuilder.addHostDependenceToDeviceActions(Current, InputArg))
           break;
 
       if (Current->getType() == types::TY_Nothing)
         break;
     }
 
     // If we ended with something, add to the output list.
     if (Current)
       Actions.push_back(Current);
 
     // Add any top level actions generated for offloading.
+    if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       OffloadBuilder.appendTopLevelActions(Actions, Current, InputArg);
+    else if (Current)
+      Current->propagateHostOffloadInfo(OffloadKinds,
+                                        /*BoundArch=*/nullptr);
   }
 
   // Add a link action if necessary.
 
   if (LinkerInputs.empty()) {
     Arg *FinalPhaseArg;
     if (getFinalPhase(Args, &FinalPhaseArg) == phases::Link)
       OffloadBuilder.appendDeviceLinkActions(Actions);
   }
 
   if (!LinkerInputs.empty()) {
+    if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       if (Action *Wrapper = OffloadBuilder.makeHostLinkAction())
         LinkerInputs.push_back(Wrapper);
     Action *LA;
     // Check if this Linker Job should emit a static library.
     if (ShouldEmitStaticLibrary(Args)) {
       LA = C.MakeAction<StaticLibJobAction>(LinkerInputs, types::TY_Image);
     } else {
       LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
     }
+    if (!Args.hasArg(options::OPT_fopenmp_new_driver))
       LA = OffloadBuilder.processHostLinkAction(LA);
+    if (Args.hasArg(options::OPT_fopenmp_new_driver))
+      LA->propagateHostOffloadInfo(OffloadKinds,
+                                   /*BoundArch=*/nullptr);
     Actions.push_back(LA);
   }
 
   // Add an interface stubs merge action if necessary.
   if (!MergerInputs.empty())
     Actions.push_back(
         C.MakeAction<IfsMergeJobAction>(MergerInputs, types::TY_Image));
 
void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   Args.ClaimAllArgs(options::OPT_cl_ignored_Group);
 
   // Claim --cuda-host-only and --cuda-compile-host-device, which may be passed
   // to non-CUDA compilations and should not trigger warnings there.
   Args.ClaimAllArgs(options::OPT_cuda_host_only);
   Args.ClaimAllArgs(options::OPT_cuda_compile_host_device);
 }
 
+Action *Driver::BuildOffloadingActions(Compilation &C,
+                                       llvm::opt::DerivedArgList &Args,
+                                       const InputTy &Input,
+                                       Action *HostAction) const {
+  if (!isa<CompileJobAction>(HostAction))
+    return HostAction;
+
+  SmallVector<const ToolChain *, 2> ToolChains;
+  ActionList DeviceActions;
+
+  types::ID InputType = Input.first;
+  const Arg *InputArg = Input.second;
+
+  auto OpenMPTCRange = C.getOffloadToolChains<Action::OFK_OpenMP>();
+  for (auto TI = OpenMPTCRange.first, TE = OpenMPTCRange.second; TI != TE; ++TI)
+    ToolChains.push_back(TI->second);
+
+  for (unsigned I = 0; I < ToolChains.size(); ++I)
+    DeviceActions.push_back(C.MakeAction<InputAction>(*InputArg, InputType));
+
+  if (DeviceActions.empty())
+    return HostAction;
+
+  auto PL = types::getCompilationPhases(*this, Args, InputType);
+
+  for (phases::ID Phase : PL) {
+    if (Phase == phases::Link) {
+      assert(Phase == PL.back() && "linking must be final compilation step.");
+      break;
+    }
+
+    auto TC = ToolChains.begin();
+    for (Action *&A : DeviceActions) {
+      A = ConstructPhaseAction(C, Args, Phase, A);
+
+      if (isa<CompileJobAction>(A)) {
+        HostAction->setCannotBeCollapsedWithNextDependentAction();
+        OffloadAction::HostDependence HDep(
+            *HostAction, *C.getSingleOffloadToolChain<Action::OFK_Host>(),
+            /*BourdArch=*/nullptr, Action::OFK_OpenMP);
+        OffloadAction::DeviceDependences DDep;
+        DDep.add(*A, **TC, /*BoundArch=*/nullptr, Action::OFK_OpenMP);
+        A = C.MakeAction<OffloadAction>(HDep, DDep);
+      }
+      ++TC;
+    }
+  }
+
+  OffloadAction::DeviceDependences DDeps;
+
+  auto TC = ToolChains.begin();
+  for (Action *A : DeviceActions) {
+    DDeps.add(*A, **TC, /*BoundArch=*/nullptr, Action::OFK_OpenMP);
+    TC++;
+  }
+
+  OffloadAction::HostDependence HDep(
+      *HostAction, *C.getSingleOffloadToolChain<Action::OFK_Host>(),
+      /*BoundArch=*/nullptr, DDeps);
+  return C.MakeAction<OffloadAction>(HDep, DDeps);
+}
+
 Action *Driver::ConstructPhaseAction(
     Compilation &C, const ArgList &Args, phases::ID Phase, Action *Input,
     Action::OffloadKind TargetDeviceOffloadKind) const {
   llvm::PrettyStackTraceString CrashInfo("Constructing phase actions");
 
   // Some types skip the assembler phase (e.g., llvm-bc), but we can't
   // encode this in the steps because the intermediate type depends on
   // arguments. Just special case here.
void Driver::BuildJobs(Compilation &C) const {
   // Collect the list of architectures.
   llvm::StringSet<> ArchNames;
   if (RawTriple.isOSBinFormatMachO())
     for (const Arg *A : C.getArgs())
       if (A->getOption().matches(options::OPT_arch))
         ArchNames.insert(A->getValue());
 
   // Set of (Action, canonical ToolChain triple) pairs we've built jobs for.
-  std::map<std::pair<const Action *, std::string>, InputInfo> CachedResults;
+  std::map<std::pair<const Action *, std::string>, InputInfoList> CachedResults;
   for (Action *A : C.getActions()) {
     // If we are linking an image for multiple archs then the linker wants
     // -arch_multiple and -final_output <final image name>. Unfortunately, this
     // doesn't fit in cleanly because we have to pass this information down.
     //
     // FIXME: This is a hack; find a cleaner way to integrate this into the
     // process.
     const char *LinkingOutput = nullptr;
if (!BoundArch.empty()) {
     TriplePlusArch += "-";
     TriplePlusArch += BoundArch;
   }
   TriplePlusArch += "-";
   TriplePlusArch += Action::GetOffloadKindName(OffloadKind);
   return TriplePlusArch;
 }
 
-InputInfo Driver::BuildJobsForAction(
+InputInfoList Driver::BuildJobsForAction(
     Compilation &C, const Action *A, const ToolChain *TC, StringRef BoundArch,
     bool AtTopLevel, bool MultipleArchs, const char *LinkingOutput,
-    std::map<std::pair<const Action *, std::string>, InputInfo> &CachedResults,
+    std::map<std::pair<const Action *, std::string>, InputInfoList>
+        &CachedResults,
     Action::OffloadKind TargetDeviceOffloadKind) const {
   std::pair<const Action *, std::string> ActionTC = {
       A, GetTriplePlusArchString(TC, BoundArch, TargetDeviceOffloadKind)};
   auto CachedResult = CachedResults.find(ActionTC);
   if (CachedResult != CachedResults.end()) {
     return CachedResult->second;
   }
-  InputInfo Result = BuildJobsForActionNoCache(
+  InputInfoList Result = BuildJobsForActionNoCache(
       C, A, TC, BoundArch, AtTopLevel, MultipleArchs, LinkingOutput,
       CachedResults, TargetDeviceOffloadKind);
   CachedResults[ActionTC] = Result;
   return Result;
 }
 
-InputInfo Driver::BuildJobsForActionNoCache(
+InputInfoList Driver::BuildJobsForActionNoCache(
     Compilation &C, const Action *A, const ToolChain *TC, StringRef BoundArch,
     bool AtTopLevel, bool MultipleArchs, const char *LinkingOutput,
-    std::map<std::pair<const Action *, std::string>, InputInfo> &CachedResults,
+    std::map<std::pair<const Action *, std::string>, InputInfoList>
+        &CachedResults,
     Action::OffloadKind TargetDeviceOffloadKind) const {
   llvm::PrettyStackTraceString CrashInfo("Building compilation jobs");
 
   InputInfoList OffloadDependencesInputInfo;
   bool BuildingForOffloadDevice = TargetDeviceOffloadKind != Action::OFK_None;
   if (const OffloadAction *OA = dyn_cast<OffloadAction>(A)) {
     // The 'Darwin' toolchain is initialized only when its arguments are
     // computed. Get the default arguments for OFK_None to ensure that
if (const OffloadAction *OA = dyn_cast<OffloadAction>(A)) {
     //
     // For a) and b), we just return the job generated for the dependence. For
     // c) and d) we override the current action with the host/device dependence
     // if the current toolchain is host/device and set the offload dependences
     // info with the jobs obtained from the device/host dependence(s).
 
     // If there is a single device option, just generate the job for it.
     if (OA->hasSingleDeviceDependence()) {
-      InputInfo DevA;
+      InputInfoList DevA;
       OA->doOnEachDeviceDependence([&](Action *DepA, const ToolChain *DepTC,
                                        const char *DepBoundArch) {
         DevA =
             BuildJobsForAction(C, DepA, DepTC, DepBoundArch, AtTopLevel,
                                /*MultipleArchs*/ !!DepBoundArch, LinkingOutput,
                                CachedResults, DepA->getOffloadingDeviceKind());
       });
       return DevA;
     }
 
     // If 'Action 2' is host, we generate jobs for the device dependences and
     // override the current action with the host dependence. Otherwise, we
     // generate the host dependences and override the action with the device
     // dependence. The dependences can't therefore be a top-level action.
     OA->doOnEachDependence(
         /*IsHostDependence=*/BuildingForOffloadDevice,
         [&](Action *DepA, const ToolChain *DepTC, const char *DepBoundArch) {
-          OffloadDependencesInputInfo.push_back(BuildJobsForAction(
+          OffloadDependencesInputInfo.append(BuildJobsForAction(
               C, DepA, DepTC, DepBoundArch, /*AtTopLevel=*/false,
               /*MultipleArchs*/ !!DepBoundArch, LinkingOutput, CachedResults,
               DepA->getOffloadingDeviceKind()));
         });
 
     A = BuildingForOffloadDevice
             ? OA->getSingleDeviceDependence(/*DoNotConsiderHostActions=*/true)
             : OA->getHostDependence();
+
+    // We may have already built this action as a part of the offloading
+    // toolchain, return the cached input if so.
+    std::pair<const Action *, std::string> ActionTC = {
+        OA->getHostDependence(),
+        GetTriplePlusArchString(TC, BoundArch, TargetDeviceOffloadKind)};
+    if (CachedResults.find(ActionTC) != CachedResults.end()) {
+      InputInfoList Inputs = CachedResults[ActionTC];
+      Inputs.append(OffloadDependencesInputInfo);
+      return Inputs;
+    }
   }
 
   if (const InputAction *IA = dyn_cast<InputAction>(A)) {
     // FIXME: It would be nice to not claim this here; maybe the old scheme of
     // just using Args was better?
     const Arg &Input = IA->getInputArg();
     Input.claim();
     if (Input.getOption().matches(options::OPT_INPUT)) {
       const char *Name = Input.getValue();
-      return InputInfo(A, Name, /* _BaseInput = */ Name);
+      return {InputInfo(A, Name, /* _BaseInput = */ Name)};
     }
-    return InputInfo(A, &Input, /* _BaseInput = */ "");
+    return {InputInfo(A, &Input, /* _BaseInput = */ "")};
   }
 
   if (const BindArchAction *BAA = dyn_cast<BindArchAction>(A)) {
     const ToolChain *TC;
     StringRef ArchName = BAA->getArchName();
 
     if (!ArchName.empty())
       TC = &getToolChain(C.getArgs(),
InputInfoList Driver::BuildJobsForActionNoCache(
   const JobAction *JA = cast<JobAction>(A);
   ActionList CollapsedOffloadActions;
 
   ToolSelector TS(JA, *TC, C, isSaveTempsEnabled(),
                   embedBitcodeInObject() && !isUsingLTO());
   const Tool *T = TS.getTool(Inputs, CollapsedOffloadActions);
 
   if (!T)
-    return InputInfo();
+    return {InputInfo()};
 
   if (BuildingForOffloadDevice &&
       A->getOffloadingDeviceKind() == Action::OFK_OpenMP) {
     if (TC->getTriple().isAMDGCN()) {
       // AMDGCN treats backend and assemble actions as no-op because
       // linker does not support object files.
       if (const BackendJobAction *BA = dyn_cast<BackendJobAction>(A)) {
         return BuildJobsForAction(C, *BA->input_begin(), TC, BoundArch,
InputInfoList Driver::BuildJobsForActionNoCache(
   }
 
   // If we've collapsed action list that contained OffloadAction we
   // need to build jobs for host/device-side inputs it may have held.
   for (const auto *OA : CollapsedOffloadActions)
     cast<OffloadAction>(OA)->doOnEachDependence(
         /*IsHostDependence=*/BuildingForOffloadDevice,
         [&](Action *DepA, const ToolChain *DepTC, const char *DepBoundArch) {
-          OffloadDependencesInputInfo.push_back(BuildJobsForAction(
+          OffloadDependencesInputInfo.append(BuildJobsForAction(
               C, DepA, DepTC, DepBoundArch, /* AtTopLevel */ false,
               /*MultipleArchs=*/!!DepBoundArch, LinkingOutput, CachedResults,
               DepA->getOffloadingDeviceKind()));
         });
 
   // Only use pipes when there is exactly one input.
   InputInfoList InputInfos;
   for (const Action *Input : Inputs) {
     // Treat dsymutil and verify sub-jobs as being at the top-level too, they
     // shouldn't get temporary output names.
     // FIXME: Clean this up.
     bool SubJobAtTopLevel =
         AtTopLevel && (isa<DsymutilJobAction>(A) || isa<VerifyJobAction>(A));
-    InputInfos.push_back(BuildJobsForAction(
+    InputInfos.append(BuildJobsForAction(
         C, Input, TC, BoundArch, SubJobAtTopLevel, MultipleArchs, LinkingOutput,
         CachedResults, A->getOffloadingDeviceKind()));
   }
 
   // Always use the first file input as the base input.
   const char *BaseInput = InputInfos[0].getBaseInput();
   for (auto &Info : InputInfos) {
     if (Info.isFilename()) {
for (auto &UI : UA->getDependentActionsInfo()) {
         if (UI.DependentOffloadKind == Action::OFK_Host)
           Arch = StringRef();
         else
           Arch = UI.DependentBoundArch;
       } else
         Arch = BoundArch;
 
       CachedResults[{A, GetTriplePlusArchString(UI.DependentToolChain, Arch,
-                                                UI.DependentOffloadKind)}] =
-          CurI;
+                                                UI.DependentOffloadKind)}] = {
+          CurI};
     }
 
     // Now that we have all the results generated, select the one that should be
     // returned for the current depending action.
     std::pair<const Action *, std::string> ActionTC = {
         A, GetTriplePlusArchString(TC, BoundArch, TargetDeviceOffloadKind)};
     assert(CachedResults.find(ActionTC) != CachedResults.end() &&
            "Result does not exist??");
-    Result = CachedResults[ActionTC];
+    Result = CachedResults[ActionTC].front();
   } else if (JA->getType() == types::TY_Nothing)
-    Result = InputInfo(A, BaseInput);
+    Result = {InputInfo(A, BaseInput)};
   else {
     // We only have to generate a prefix for the host if this is not a top-level
     // action.
     std::string OffloadingPrefix = Action::GetOffloadingFileNamePrefix(
         A->getOffloadingDeviceKind(), TC->getTriple().normalize(),
         /*CreatePrefixForHost=*/!!A->getOffloadingHostActiveKinds() &&
             !AtTopLevel);
     if (isa<OffloadWrapperJobAction>(JA)) {
if (UnbundlingResults.empty())
           C.getArgsForToolChain(TC, BoundArch, JA->getOffloadingDeviceKind()),
           LinkingOutput);
     else
       T->ConstructJobMultipleOutputs(
           C, *JA, UnbundlingResults, InputInfos,
           C.getArgsForToolChain(TC, BoundArch, JA->getOffloadingDeviceKind()),
           LinkingOutput);
   }
-  return Result;
+  return {Result};
 }
 
 const char *Driver::getDefaultImageName() const {
   llvm::Triple Target(llvm::Triple::normalize(TargetTriple));
   return Target.isOSWindows() ? "a.exe" : "a.out";
 }
 
 /// Create output filename based on ArgValue, which could either be a

clang/lib/Driver/ToolChains/Clang.cpp

void Clang::ConstructJob(Compilation &C, const JobAction &JA,
   // second input. Module precompilation accepts a list of header files to
   // include as part of the module. All other jobs are expected to have exactly
   // one input.
   bool IsCuda = JA.isOffloading(Action::OFK_Cuda);
   bool IsCudaDevice = JA.isDeviceOffloading(Action::OFK_Cuda);
   bool IsHIP = JA.isOffloading(Action::OFK_HIP);
   bool IsHIPDevice = JA.isDeviceOffloading(Action::OFK_HIP);
   bool IsOpenMPDevice = JA.isDeviceOffloading(Action::OFK_OpenMP);
+  bool IsOpenMPHost = JA.isHostOffloading(Action::OFK_OpenMP);
   bool IsHeaderModulePrecompile = isa<HeaderModulePrecompileJobAction>(JA);
   bool IsDeviceOffloadAction = !(JA.isDeviceOffloading(Action::OFK_None) ||
                                  JA.isDeviceOffloading(Action::OFK_Host));
   bool IsUsingLTO = D.isUsingLTO(IsDeviceOffloadAction);
   auto LTOMode = D.getLTOMode(IsDeviceOffloadAction);
 
   // A header module compilation doesn't have a main input file, so invent a
   // fake one as a placeholder.
   const char *ModuleName = [&]{
     auto *ModuleNameArg = Args.getLastArg(options::OPT_fmodule_name_EQ);
     return ModuleNameArg ? ModuleNameArg->getValue() : "";
   }();
   InputInfo HeaderModuleInput(Inputs[0].getType(), ModuleName, ModuleName);
 
   const InputInfo &Input =
       IsHeaderModulePrecompile ? HeaderModuleInput : Inputs[0];
 
   InputInfoList ModuleHeaderInputs;
   const InputInfo *CudaDeviceInput = nullptr;
   const InputInfo *OpenMPDeviceInput = nullptr;
+  const InputInfo *OpenMPHostInput = nullptr;
   for (const InputInfo &I : Inputs) {
     if (&I == &Input) {
       // This is the primary input.
     } else if (IsHeaderModulePrecompile &&
                types::getPrecompiledType(I.getType()) == types::TY_PCH) {
       types::ID Expected = HeaderModuleInput.getType();
       if (I.getType() != Expected) {
         D.Diag(diag::err_drv_module_header_wrong_kind)
             << I.getFilename() << types::getTypeName(I.getType())
             << types::getTypeName(Expected);
       }
       ModuleHeaderInputs.push_back(I);
     } else if ((IsCuda || IsHIP) && !CudaDeviceInput) {
       CudaDeviceInput = &I;
     } else if (IsOpenMPDevice && !OpenMPDeviceInput) {
       OpenMPDeviceInput = &I;
+    } else if (IsOpenMPHost && !OpenMPHostInput) {
+      OpenMPHostInput = &I;
     } else {
       llvm_unreachable("unexpectedly given multiple inputs");
     }
   }
 
   const llvm::Triple *AuxTriple =
       (IsCuda || IsHIP) ? TC.getAuxTriple() : nullptr;
   bool IsWindowsMSVC = RawTriple.isWindowsMSVCEnvironment();

clang/test/Driver/openmp-offload-gpu.c

 // SAVE_TEMPS_NAMES-NOT: "GNU::Linker"{{.*}}["[[SAVE_TEMPS_INPUT1:.*\.o]]", "[[SAVE_TEMPS_INPUT1]]"]
 
 // RUN:   %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64 -Xopenmp-target=nvptx64 -march=sm_35 \
 // RUN:          -save-temps -no-canonical-prefixes %s -o openmp-offload-gpu 2>&1 \
 // RUN:   | FileCheck -check-prefix=TRIPLE %s
 
 // TRIPLE: "-triple" "nvptx64-nvidia-cuda"
 // TRIPLE: "-target-cpu" "sm_35"
+
+// RUN:   %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda \
+// RUN:          -fopenmp-new-driver -no-canonical-prefixes -ccc-print-bindings %s -o openmp-offload-gpu 2>&1 \
+// RUN:   | FileCheck -check-prefix=NEW_DRIVER %s
+
+// NEW_DRIVER: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[HOST_INPUT:.+]]"], output: "[[HOST_BC:.+]]" 
+// NEW_DRIVER: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[DEVICE_INPUT:.+]]", "[[HOST_BC]]"], output: "[[DEVICE_ASM:.+]]"
+// NEW_DRIVER: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[DEVICE_ASM]]"], output: "[[DEVICE_OBJ:.+]]" 
+// NEW_DRIVER: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[HOST_BC]]", "[[DEVICE_OBJ]]"], output: "[[HOST_OBJ:.+]]" 
+// NEW_DRIVER: "x86_64-unknown-linux-gnu" - "[[LINKER:.+]]", inputs: ["[[HOST_OBJ]]"], output: "openmp-offload-gpu"