[OpenMP] Try to Infer target triples using the offloading architecture

[jhuber6]

Summary

Currently we require the -fopenmp-targets= option to specify the
triple to use for the offloading toolchains, and the -Xopenmp-target=
option to specify architectures to a specific toolchain. The changes
made in D124721 allowed us to use --offload-arch= to specify multiple
target architectures. However, this can become combersome with many
different architectures. This patch introduces functinality that
attempts to deduce the target triple and architectures from the
offloading action. Currently we will deduce known GPU architectures when
only -fopenmp is specified.

This required a bit of a hack to cache the deduced architectures,
without this we would've just thrown an error when we tried to look up
the architecture again when generating the job. Normally we require the
user to manually specify the toolchain arguments, but here they would
confict unless we overrode them.

Depends on: D124721

[tra]

Just a drive-by style review. I didn't look at the functionality of the changes much.

clang/lib/Driver/Driver.cpp

788–789

So, specifying just -fopenmp will result in IsOpenMP=false? This seems odd.
Perhaps the variable should be called IsOpenMPOffload ?

821–822

This seems a bit too restrictive as it would fail if we've failed to get either of those triples. We may be able to proceed with only one of them in many cases.
I'd remove this check and would make the loop below more forgiving.

845

if (NvidiaTriple && IsNVIDIAGpuArch(...))
...
else if (AMDTriple && IsAMDGpuArch(...))
...
else {
   Diag(); 
   return;
}

[jhuber6]

Address comments.

[jhuber6]

clang/lib/Driver/Driver.cpp

788–789

Will do.

[saiislam]

Looks good to me.

Will it work with -fno-openmp? Sometimes -fno-openmp is used by the end-user to override system provided -fopenmp flag for some translation units.

Please have a look at following examples:

// RUN: %clang -### -target x86_64-linux-gnu \
// RUN:   --offload-arch=gfx906 \
// RUN:   %s 2>&1 | FileCheck -check-prefix=OFFLOAD %s
// OFFLOAD: warning: argument unused during compilation: '--offload-arch=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   --offload-arch=gfx906 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=OFFLOAD1 %s
// OFFLOAD1: warning: argument unused during compilation: '--offload-arch=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=LEGACY %s
// LEGACY: warning: '-fopenmp-targets' must be used in conjunction with a '-fopenmp' option compatible with offloading; e.g., '-fopenmp=libomp' or '-fopenmp=libiomp5'
// LEGACY-NEXT: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   --offload-arch=gfx906 \
// RUN:   --offload-arch=gfx908 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=MOFFLOAD %s
// MOFFLOAD: warning: argument unused during compilation: '--offload-arch=gfx906'
// MOFFLOAD-NEXT: warning: argument unused during compilation: '--offload-arch=gfx908'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=MLEGACY %s
// MLEGACY: warning: '-fopenmp-targets' must be used in conjunction with a '-fopenmp' option compatible with offloading; e.g., '-fopenmp=libomp' or '-fopenmp=libiomp5'
// MLEGACY: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906'
// MLEGACY: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908'

[jhuber6]

In D125050#3496289, @saiislam wrote:

Looks good to me.

Will it work with -fno-openmp? Sometimes -fno-openmp is used by the end-user to override system provided -fopenmp flag for some translation units.

Please have a look at following examples:

// RUN: %clang -### -target x86_64-linux-gnu \
// RUN:   --offload-arch=gfx906 \
// RUN:   %s 2>&1 | FileCheck -check-prefix=OFFLOAD %s
// OFFLOAD: warning: argument unused during compilation: '--offload-arch=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   --offload-arch=gfx906 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=OFFLOAD1 %s
// OFFLOAD1: warning: argument unused during compilation: '--offload-arch=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=LEGACY %s
// LEGACY: warning: '-fopenmp-targets' must be used in conjunction with a '-fopenmp' option compatible with offloading; e.g., '-fopenmp=libomp' or '-fopenmp=libiomp5'
// LEGACY-NEXT: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   --offload-arch=gfx906 \
// RUN:   --offload-arch=gfx908 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=MOFFLOAD %s
// MOFFLOAD: warning: argument unused during compilation: '--offload-arch=gfx906'
// MOFFLOAD-NEXT: warning: argument unused during compilation: '--offload-arch=gfx908'

// RUN: %clang -### -target x86_64-linux-gnu -fopenmp\
// RUN:   -fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 \
// RUN:   -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908 \
// RUN:   -fno-openmp \
// RUN:   %s 2>&1 | FileCheck -check-prefix=MLEGACY %s
// MLEGACY: warning: '-fopenmp-targets' must be used in conjunction with a '-fopenmp' option compatible with offloading; e.g., '-fopenmp=libomp' or '-fopenmp=libiomp5'
// MLEGACY: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906'
// MLEGACY: warning: argument unused during compilation: '-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908'

It should, I could add a simple test for that.

[jhuber6]

Add test for disabling openmp with -fno-openmp

[saiislam]

Thanks!
LGTM.

[yaxunl]

clang/lib/Driver/Driver.cpp

789

probably should be

C.getInputArgs().hasArg(options::OPT_offload_arch_EQ) && !IsHIP && !IsCUDA

otherwise a HIP program may get both HIP offloading and OpenMP offloading at the same time. Currently it is not supported.

[yaxunl]

could you please add a test to https://github.com/llvm/llvm-project/blob/main/clang/test/Driver/hip-offload-arch.hip

to make sure when -fopenmp is specified, no openmp offloading is performed? Thanks.

[jhuber6]

clang/lib/Driver/Driver.cpp

789

I do that below, line 815. The logic states that we only take --offloading-arch to imply OpenMP offloading if it's not currently CUDA or HIP. I'm pretty sure there's an existing test that covers this as it used to fail one of those HIP-interop tests until I added !IsHIP && !IsCUDA below.

clang/include/clang/Basic/DiagnosticDriverKinds.td

def warn_drv_avr_linker_section_addresses_not_implemented: Warning<
   "support for passing the data section address to the linker for "
   "microcontroller '%0' is not implemented">,
   InGroup<AVRRtlibLinkingQuirks>;
 def warn_drv_avr_stdlib_not_linked: Warning<
   "standard library not linked and so no interrupt vector table or "
   "compiler runtime routines will be linked">,
   InGroup<AVRRtlibLinkingQuirks>;
 def err_drv_cuda_bad_gpu_arch : Error<"unsupported CUDA gpu architecture: %0">;
+def err_drv_offload_bad_gpu_arch : Error<"unsupported %0 gpu architecture: %1">;
 def err_drv_no_cuda_installation : Error<
   "cannot find CUDA installation; provide its path via '--cuda-path', or pass "
   "'-nocudainc' to build without CUDA includes">;
 def err_drv_no_cuda_libdevice : Error<
   "cannot find libdevice for %0; provide path to different CUDA installation "
   "via '--cuda-path', or pass '-nocudalib' to build without linking with "
   "libdevice">;
 def err_drv_no_rdc_new_driver : Error<

 def err_drv_omp_host_ir_file_not_found : Error<
   "provided host compiler IR file '%0' is required to generate code for OpenMP "
   "target regions but cannot be found">;
 def err_drv_omp_host_target_not_supported : Error<
   "target '%0' is not a supported OpenMP host target">;
 def err_drv_expecting_fopenmp_with_fopenmp_targets : Error<
   "'-fopenmp-targets' must be used in conjunction with a '-fopenmp' option "
   "compatible with offloading; e.g., '-fopenmp=libomp' or '-fopenmp=libiomp5'">;
+def err_drv_failed_to_deduce_target_from_arch : Error<
+  "failed to deduce triple for target architecture '%0'; specify the triple "
+  "using '-fopenmp-targets' and '-Xopenmp-target' instead.">;
 def err_drv_omp_offload_target_missingbcruntime : Error<
   "no library '%0' found in the default clang lib directory or in LIBRARY_PATH"
   "; use '--libomptarget-%1-bc-path' to specify %1 bitcode library">;
 def err_drv_omp_offload_target_bcruntime_not_found : Error<
   "bitcode library '%0' does not exist">;
 def err_drv_omp_offload_target_cuda_version_not_support : Error<
   "NVPTX target requires CUDA 9.2 or above; CUDA %0 detected">;
 def warn_drv_omp_offload_target_duplicate : Warning<

clang/include/clang/Driver/Driver.h

private:
 
   /// Cache of all the ToolChains in use by the driver.
   ///
   /// This maps from the string representation of a triple to a ToolChain
   /// created targeting that triple. The driver owns all the ToolChain objects
   /// stored in it, and will clean them up when torn down.
   mutable llvm::StringMap<std::unique_ptr<ToolChain>> ToolChains;
 
+  /// Cache of known offloading architectures for the ToolChain already derived.
+  /// This should only be modified when we first initialize the offloading
+  /// toolchains.
+  llvm::DenseMap<const ToolChain *, llvm::DenseSet<llvm::StringRef>> KnownArchs;
+
 private:
   /// TranslateInputArgs - Create a new derived argument list from the input
   /// arguments, after applying the standard argument translations.
   llvm::opt::DerivedArgList *
   TranslateInputArgs(const llvm::opt::InputArgList &Args) const;
 
   // handleArguments - All code related to claiming and printing diagnostics
   // related to arguments to the driver are done here.
public:
   /// \param Args - The input arguments.
   /// \param Input - The input type and arguments
   /// \param HostAction - The host action used in the offloading toolchain.
   Action *BuildOffloadingActions(Compilation &C,
                                  llvm::opt::DerivedArgList &Args,
                                  const InputTy &Input,
                                  Action *HostAction) const;
 
+  /// Returns the set of bound architectures active for this offload kind.
+  /// If there are no bound architctures we return a set containing only the
+  /// empty string.
+  llvm::DenseSet<StringRef>
+  getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
+                  Action::OffloadKind Kind, const ToolChain *TC) const;
+
   /// Check that the file referenced by Value exists. If it doesn't,
   /// issue a diagnostic and return false.
   /// If TypoCorrect is true and the file does not exist, see if it looks
   /// like a likely typo for a flag and if so print a "did you mean" blurb.
   bool DiagnoseInputExistence(const llvm::opt::DerivedArgList &Args,
                               StringRef Value, types::ID Ty,
                               bool TypoCorrect) const;
 

clang/lib/Driver/Driver.cpp

if (IsCuda) {
     assert(HIPTC && "Could not create offloading device tool chain.");
     C.addOffloadDeviceToolChain(HIPTC, OFK);
   }
 
   //
   // OpenMP
   //
   // We need to generate an OpenMP toolchain if the user specified targets with
-  // the -fopenmp-targets option.
+  // the -fopenmp-targets option or used --offload-arch with OpenMP enabled.
+  bool IsOpenMPOffloading =
+      C.getInputArgs().hasFlag(options::OPT_fopenmp, options::OPT_fopenmp_EQ,
+                               options::OPT_fno_openmp, false) &&
+      (C.getInputArgs().hasArg(options::OPT_fopenmp_targets_EQ) ||
+       C.getInputArgs().hasArg(options::OPT_offload_arch_EQ));
+  if (IsOpenMPOffloading) {
+    // We expect that -fopenmp-targets is always used in conjunction with the
+    // option -fopenmp specifying a valid runtime with offloading support, i.e.
+    // libomp or libiomp.
+    OpenMPRuntimeKind RuntimeKind = getOpenMPRuntime(C.getInputArgs());
+    if (RuntimeKind != OMPRT_OMP && RuntimeKind != OMPRT_IOMP5) {
+      Diag(clang::diag::err_drv_expecting_fopenmp_with_fopenmp_targets);
+      return;
+    }
+
+    llvm::StringMap<llvm::DenseSet<StringRef>> DerivedArchs;
+    llvm::StringMap<StringRef> FoundNormalizedTriples;
+    llvm::SmallVector<StringRef, 4> OpenMPTriples;
+
+    // If the user specified -fopenmp-targets= we create a toolchain for each
+    // valid triple. Otherwise, if only --offload-arch= was specified we instead
+    // attempt to derive the appropriate toolchains from the arguments.
     if (Arg *OpenMPTargets =
             C.getInputArgs().getLastArg(options::OPT_fopenmp_targets_EQ)) {
-    if (OpenMPTargets->getNumValues()) {
-      // We expect that -fopenmp-targets is always used in conjunction with the
-      // option -fopenmp specifying a valid runtime with offloading support,
-      // i.e. libomp or libiomp.
-      bool HasValidOpenMPRuntime = C.getInputArgs().hasFlag(
-          options::OPT_fopenmp, options::OPT_fopenmp_EQ,
-          options::OPT_fno_openmp, false);
-      if (HasValidOpenMPRuntime) {
-        OpenMPRuntimeKind OpenMPKind = getOpenMPRuntime(C.getInputArgs());
-        HasValidOpenMPRuntime =
-            OpenMPKind == OMPRT_OMP || OpenMPKind == OMPRT_IOMP5;
+      if (OpenMPTargets && !OpenMPTargets->getNumValues()) {
+        Diag(clang::diag::warn_drv_empty_joined_argument)
+            << OpenMPTargets->getAsString(C.getInputArgs());
+        return;
+      }
+      llvm::copy(OpenMPTargets->getValues(), std::back_inserter(OpenMPTriples));
+    } else if (C.getInputArgs().hasArg(options::OPT_offload_arch_EQ) &&
+               !IsHIP && !IsCuda) {
+      const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
+      auto AMDTriple = getHIPOffloadTargetTriple(*this, C.getInputArgs());
+      auto NVPTXTriple = getNVIDIAOffloadTargetTriple(*this, C.getInputArgs(),
+                                                      HostTC->getTriple());
+
+      // Attempt to deduce the offloading triple from the set of architectures.
+      // We can only correctly deduce NVPTX / AMDGPU triples currently.
+      llvm::DenseSet<StringRef> Archs =
+          getOffloadArchs(C, C.getArgs(), Action::OFK_OpenMP, nullptr);
+      for (StringRef Arch : Archs) {
+        if (NVPTXTriple && IsNVIDIAGpuArch(StringToCudaArch(
+                               getProcessorFromTargetID(*NVPTXTriple, Arch)))) {
+          DerivedArchs[NVPTXTriple->getTriple()].insert(Arch);
+        } else if (AMDTriple &&
+                   IsAMDGpuArch(StringToCudaArch(
+                       getProcessorFromTargetID(*AMDTriple, Arch)))) {
+          DerivedArchs[AMDTriple->getTriple()].insert(Arch);
+        } else {
+          Diag(clang::diag::err_drv_failed_to_deduce_target_from_arch) << Arch;
+          return;
+        }
       }
 
-      if (HasValidOpenMPRuntime) {
-        llvm::StringMap<const char *> FoundNormalizedTriples;
-        for (const char *Val : OpenMPTargets->getValues()) {
-          llvm::Triple TT(ToolChain::getOpenMPTriple(Val));
-          std::string NormalizedName = TT.normalize();
+      for (const auto &TripleAndArchs : DerivedArchs)
+        OpenMPTriples.push_back(TripleAndArchs.first());
+    }
+
+    for (StringRef Val : OpenMPTriples) {
 
-          // Make sure we don't have a duplicate triple.
-          auto Duplicate = FoundNormalizedTriples.find(NormalizedName);
-          if (Duplicate != FoundNormalizedTriples.end()) {
-            Diag(clang::diag::warn_drv_omp_offload_target_duplicate)
-                << Val << Duplicate->second;
-            continue;
-          }
 
-          // Store the current triple so that we can check for duplicates in the
-          // following iterations.
-          FoundNormalizedTriples[NormalizedName] = Val;
 
-          // If the specified target is invalid, emit a diagnostic.
-          if (TT.getArch() == llvm::Triple::UnknownArch)
-            Diag(clang::diag::err_drv_invalid_omp_target) << Val;
-          else {
-            const ToolChain *TC;
-            // Device toolchains have to be selected differently. They pair host
-            // and device in their implementation.
-            if (TT.isNVPTX() || TT.isAMDGCN()) {
-              const ToolChain *HostTC =
-                  C.getSingleOffloadToolChain<Action::OFK_Host>();
-              assert(HostTC && "Host toolchain should be always defined.");
-              auto &DeviceTC =
-                  ToolChains[TT.str() + "/" + HostTC->getTriple().normalize()];
-              if (!DeviceTC) {
-                if (TT.isNVPTX())
-                  DeviceTC = std::make_unique<toolchains::CudaToolChain>(
-                      *this, TT, *HostTC, C.getInputArgs(), Action::OFK_OpenMP);
-                else if (TT.isAMDGCN())
-                  DeviceTC =
-                      std::make_unique<toolchains::AMDGPUOpenMPToolChain>(
-                          *this, TT, *HostTC, C.getInputArgs());
-                else
-                  assert(DeviceTC && "Device toolchain not defined.");
-              }
+              DeviceTC = std::make_unique<toolchains::AMDGPUOpenMPToolChain>(
 
-              TC = DeviceTC.get();
-            } else
-              TC = &getToolChain(C.getInputArgs(), TT);
-            C.addOffloadDeviceToolChain(TC, Action::OFK_OpenMP);
-          }
-        }
-      } else
-        Diag(clang::diag::err_drv_expecting_fopenmp_with_fopenmp_targets);
-    } else
-      Diag(clang::diag::warn_drv_empty_joined_argument)
-          << OpenMPTargets->getAsString(C.getInputArgs());
+        if (DerivedArchs.find(TT.getTriple()) != DerivedArchs.end())
+          KnownArchs[TC] = DerivedArchs[TT.getTriple()];
+  } else if (C.getInputArgs().hasArg(options::OPT_fopenmp_targets_EQ)) {
+    return;
   }
 
   //
   // TODO: Add support for other offloading programming models here.
   //
 }
 
 /// Looks the given directories for the specified file.
void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   Args.ClaimAllArgs(options::OPT_cl_ignored_Group);
 
   // Claim --offload-host-only and --offload-compile-host-device, which may be
   // passed to non-CUDA compilations and should not trigger warnings there.
   Args.ClaimAllArgs(options::OPT_offload_host_only);
   Args.ClaimAllArgs(options::OPT_offload_host_device);
 }
 
-/// Returns the canonical name for the offloading architecture when using HIP or
-/// CUDA.
+/// Returns the canonical name for the offloading architecture when using a HIP
+/// or CUDA architecture.
 static StringRef getCanonicalArchString(Compilation &C,
                                         const llvm::opt::DerivedArgList &Args,
                                         StringRef ArchStr,
-                                        Action::OffloadKind Kind,
-                                        const ToolChain *TC) {
-  if (Kind == Action::OFK_Cuda ||
-      (Kind == Action::OFK_OpenMP && TC->getTriple().isNVPTX())) {
-    CudaArch Arch = StringToCudaArch(ArchStr);
-    if (Arch == CudaArch::UNKNOWN || !IsNVIDIAGpuArch(Arch)) {
-      C.getDriver().Diag(clang::diag::err_drv_cuda_bad_gpu_arch) << ArchStr;
-      return StringRef();
-    }
+                                        const llvm::Triple &Triple) {
+  // Lookup the CUDA / HIP architecture string. Only report an error if we were
+  // expecting the triple to be only NVPTX / AMDGPU.
+  CudaArch Arch = StringToCudaArch(getProcessorFromTargetID(Triple, ArchStr));
+  if (Triple.isNVPTX() &&
+      (Arch == CudaArch::UNKNOWN || !IsNVIDIAGpuArch(Arch))) {
+    C.getDriver().Diag(clang::diag::err_drv_offload_bad_gpu_arch)
+        << "CUDA" << ArchStr;
+    return StringRef();
+  } else if (Triple.isAMDGPU() &&
+             (Arch == CudaArch::UNKNOWN || !IsAMDGpuArch(Arch))) {
+    C.getDriver().Diag(clang::diag::err_drv_offload_bad_gpu_arch)
+        << "HIP" << ArchStr;
+
+  if (IsNVIDIAGpuArch(Arch))
     return Args.MakeArgStringRef(CudaArchToString(Arch));
-  } else if (Kind == Action::OFK_HIP ||
-             (Kind == Action::OFK_OpenMP && TC->getTriple().isAMDGPU())) {
+
+  if (IsAMDGpuArch(Arch)) {
     llvm::StringMap<bool> Features;
-    // getHIPOffloadTargetTriple() is known to return valid value as it has
-    // been called successfully in the CreateOffloadingDeviceToolChains().
-    auto Arch = parseTargetID(
-        *getHIPOffloadTargetTriple(C.getDriver(), C.getInputArgs()), ArchStr,
-        &Features);
+    auto HIPTriple = getHIPOffloadTargetTriple(C.getDriver(), C.getInputArgs());
+    if (!HIPTriple)
+      return StringRef();
+    auto Arch = parseTargetID(*HIPTriple, ArchStr, &Features);
     if (!Arch) {
       C.getDriver().Diag(clang::diag::err_drv_bad_target_id) << ArchStr;
       C.setContainsError();
       return StringRef();
     }
     return Args.MakeArgStringRef(
         getCanonicalTargetID(Arch.getValue(), Features));
   }
+
   // If the input isn't CUDA or HIP just return the architecture.
   return ArchStr;
 }
 
 /// Checks if the set offloading architectures does not conflict. Returns the
 /// incompatible pair if a conflict occurs.
 static llvm::Optional<std::pair<llvm::StringRef, llvm::StringRef>>
 getConflictOffloadArchCombination(const llvm::DenseSet<StringRef> &Archs,
                                   Action::OffloadKind Kind) {
   if (Kind != Action::OFK_HIP)
     return None;
 
   std::set<StringRef> ArchSet;
   llvm::copy(Archs, std::inserter(ArchSet, ArchSet.begin()));
   return getConflictTargetIDCombination(ArchSet);
 }
 
-/// Returns the set of bound architectures active for this compilation kind.
-/// This function returns a set of bound architectures, if there are no bound
-/// architctures we return a set containing only the empty string.
-static llvm::DenseSet<StringRef>
-getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
-                Action::OffloadKind Kind, const ToolChain *TC) {
+llvm::DenseSet<StringRef>
+Driver::getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
+                        Action::OffloadKind Kind, const ToolChain *TC) const {
+  if (!TC)
+    TC = &C.getDefaultToolChain();
 
   // --offload and --offload-arch options are mutually exclusive.
   if (Args.hasArgNoClaim(options::OPT_offload_EQ) &&
       Args.hasArgNoClaim(options::OPT_offload_arch_EQ,
                          options::OPT_no_offload_arch_EQ)) {
     C.getDriver().Diag(diag::err_opt_not_valid_with_opt)
         << "--offload"
         << (Args.hasArgNoClaim(options::OPT_offload_arch_EQ)
                 ? "--offload-arch"
                 : "--no-offload-arch");
   }
 
+  if (KnownArchs.find(TC) != KnownArchs.end())
+    return KnownArchs.lookup(TC);
+
   llvm::DenseSet<StringRef> Archs;
   for (auto &Arg : Args) {
     if (Arg->getOption().matches(options::OPT_offload_arch_EQ)) {
-      Archs.insert(getCanonicalArchString(C, Args, Arg->getValue(), Kind, TC));
+      Archs.insert(
+          getCanonicalArchString(C, Args, Arg->getValue(), TC->getTriple()));
     } else if (Arg->getOption().matches(options::OPT_no_offload_arch_EQ)) {
       if (Arg->getValue() == StringRef("all"))
         Archs.clear();
       else
-        Archs.erase(getCanonicalArchString(C, Args, Arg->getValue(), Kind, TC));
+        Archs.erase(
+            getCanonicalArchString(C, Args, Arg->getValue(), TC->getTriple()));
     }
   }
 
   if (auto ConflictingArchs = getConflictOffloadArchCombination(Archs, Kind)) {
     C.getDriver().Diag(clang::diag::err_drv_bad_offload_arch_combo)
         << ConflictingArchs.getValue().first
         << ConflictingArchs.getValue().second;
     C.setContainsError();

clang/lib/Driver/ToolChains/Clang.cpp

if (Triple.isAMDGPU()) {
 
     Args.addOptInFlag(CmdArgs, options::OPT_munsafe_fp_atomics,
                       options::OPT_mno_unsafe_fp_atomics);
   }
 
   // For all the host OpenMP offloading compile jobs we need to pass the targets
   // information using -fopenmp-targets= option.
   if (JA.isHostOffloading(Action::OFK_OpenMP)) {
-    SmallString<128> TargetInfo("-fopenmp-targets=");
+    SmallString<128> Targets("-fopenmp-targets=");
 
-    Arg *Tgts = Args.getLastArg(options::OPT_fopenmp_targets_EQ);
-    assert(Tgts && Tgts->getNumValues() &&
-           "OpenMP offloading has to have targets specified.");
-    for (unsigned i = 0; i < Tgts->getNumValues(); ++i) {
-      if (i)
-        TargetInfo += ',';
-      // We need to get the string from the triple because it may be not exactly
-      // the same as the one we get directly from the arguments.
-      llvm::Triple T(Tgts->getValue(i));
-      TargetInfo += T.getTriple();
-    }
-    CmdArgs.push_back(Args.MakeArgString(TargetInfo.str()));
+    SmallVector<std::string, 4> Triples;
+    auto TCRange = C.getOffloadToolChains<Action::OFK_OpenMP>();
+    std::transform(TCRange.first, TCRange.second, std::back_inserter(Triples),
+                   [](auto TC) { return TC.second->getTripleString(); });
+    CmdArgs.push_back(Args.MakeArgString(Targets + llvm::join(Triples, ",")));
   }
 
   bool VirtualFunctionElimination =
       Args.hasFlag(options::OPT_fvirtual_function_elimination,
                    options::OPT_fno_virtual_function_elimination, false);
   if (VirtualFunctionElimination) {
     // VFE requires full LTO (currently, this might be relaxed to allow ThinLTO
     // in the future).

clang/test/Driver/openmp-offload-infer.c

+// REQUIRES: x86-registered-target
+// REQUIRES: nvptx-registered-target
+// REQUIRES: amdgpu-registered-target
+
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -fopenmp \
+// RUN:          --offload-arch=sm_52 --offload-arch=gfx803 \
+// RUN:          --libomptarget-amdgpu-bc-path=%S/Inputs/hip_dev_lib/libomptarget-amdgpu-gfx803.bc \
+// RUN:          --libomptarget-nvptx-bc-path=%S/Inputs/libomptarget/libomptarget-nvptx-test.bc %s 2>&1 \
+// RUN:   | FileCheck %s
+
+// verify the tools invocations
+// CHECK: "-cc1" "-triple" "x86_64-unknown-linux-gnu"{{.*}}"-emit-llvm-bc"{{.*}}"-x" "c"
+// CHECK: "-cc1" "-triple" "amdgcn-amd-amdhsa" "-aux-triple" "x86_64-unknown-linux-gnu"{{.*}}"-target-cpu" "gfx803"
+// CHECK: "-cc1" "-triple" "nvptx64-nvidia-cuda" "-aux-triple" "x86_64-unknown-linux-gnu"{{.*}}"-target-cpu" "sm_52"
+// CHECK: "-cc1" "-triple" "x86_64-unknown-linux-gnu"{{.*}}"-emit-obj"
+// CHECK: clang-linker-wrapper{{.*}}"--"{{.*}} "-o" "a.out"
+
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -ccc-print-bindings -fopenmp \
+// RUN:     --offload-arch=sm_70 --offload-arch=gfx908:sramecc+:xnack- \
+// RUN:     -nogpulib %s 2>&1 | FileCheck %s --check-prefix=CHECK-NVIDIA-AMDGPU
+
+// CHECK-NVIDIA-AMDGPU: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[INPUT:.+]]"], output: "[[HOST_BC:.+]]"
+// CHECK-NVIDIA-AMDGPU: "amdgcn-amd-amdhsa" - "clang", inputs: ["[[INPUT]]", "[[HOST_BC]]"], output: "[[AMD_BC:.+]]"
+// CHECK-NVIDIA-AMDGPU: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT]]", "[[HOST_BC]]"], output: "[[NVIDIA_PTX:.+]]"
+// CHECK-NVIDIA-AMDGPU: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[NVIDIA_PTX]]"], output: "[[NVIDIA_CUBIN:.+]]"
+// CHECK-NVIDIA-AMDGPU: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[HOST_BC]]", "[[AMD_BC]]", "[[NVIDIA_CUBIN]]"], output: "[[HOST_OBJ:.+]]"
+// CHECK-NVIDIA-AMDGPU: "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[HOST_OBJ]]"], output: "a.out"
+
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -ccc-print-bindings -fopenmp \
+// RUN:     --offload-arch=sm_52 --offload-arch=sm_70 -nogpulib %s 2>&1 | FileCheck %s --check-prefix=CHECK-ARCH-BINDINGS
+
+// CHECK-ARCH-BINDINGS: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[INPUT:.*]]"], output: "[[HOST_BC:.*]]"
+// CHECK-ARCH-BINDINGS: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT]]", "[[HOST_BC]]"], output: "[[DEVICE_BC_SM_52:.*]]"
+// CHECK-ARCH-BINDINGS: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[DEVICE_BC_SM_52]]"], output: "[[DEVICE_OBJ_SM_52:.*]]"
+// CHECK-ARCH-BINDINGS: "nvptx64-nvidia-cuda" - "clang", inputs: ["[[INPUT]]", "[[HOST_BC]]"], output: "[[DEVICE_BC_SM_70:.*]]"
+// CHECK-ARCH-BINDINGS: "nvptx64-nvidia-cuda" - "NVPTX::Assembler", inputs: ["[[DEVICE_BC_SM_70]]"], output: "[[DEVICE_OBJ_SM_70:.*]]"
+// CHECK-ARCH-BINDINGS: "x86_64-unknown-linux-gnu" - "clang", inputs: ["[[HOST_BC]]", "[[DEVICE_OBJ_SM_52]]", "[[DEVICE_OBJ_SM_70]]"], output: "[[HOST_OBJ:.*]]"
+// CHECK-ARCH-BINDINGS: "x86_64-unknown-linux-gnu" - "Offload::Linker", inputs: ["[[HOST_OBJ]]"], output: "a.out"
+
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -ccc-print-bindings -fopenmp \
+// RUN:     --offload-arch=sm_70 --offload-arch=gfx908 --offload-arch=native \
+// RUN:     -nogpulib %s 2>&1 | FileCheck %s --check-prefix=CHECK-FAILED
+
+// CHECK-FAILED: error: failed to deduce triple for target architecture 'native'; specify the triple using '-fopenmp-targets' and '-Xopenmp-target' instead.
+
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -ccc-print-bindings -fopenmp \
+// RUN:     --offload-arch=sm_70 --offload-arch=gfx908 -fno-openmp \
+// RUN:     -nogpulib %s 2>&1 | FileCheck %s --check-prefix=CHECK-DISABLED
+
+// CHECK-DISABLED-NOT: "nvptx64-nvidia-cuda" - "clang",