[OpenMP] Add support for '--offload-arch=native' to OpenMP offloading

[jhuber6]

Summary

This patch adds support for '--offload-arch=native' to OpenMP
offloading. This will automatically generate the toolchains required to
fulfil whatever GPUs the user has installed. Getting this to work
requires a bit of a hack. The problem is that we need the ToolChain to
launch its searching program. But we do not yet have that ToolChain
built. I had to temporarily make the ToolChain and also add some logic
to ignore regular warnings & errors.

Depends on D141078

[JonChesterfield]

Possible naming hazard here. march=native means target the local processor architecture, zen2 or whatever, and we have the host CPU as an offloading target already. So what I'd expect this to do is host offloading with the openmp runtime compiled for the local variant of x86 or aarch64, not for it to have a guess at a GPU target.

What you think of offload-arch=GPU for pick a plausible GPU? That distinguishes it from other things we might want to offload to. Open question whether it should create a vgpu instance if it can't detect a physical card.

[jhuber6]

In D141105#4031103, @JonChesterfield wrote:

Possible naming hazard here. march=native means target the local processor architecture, zen2 or whatever, and we have the host CPU as an offloading target already. So what I'd expect this to do is host offloading with the openmp runtime compiled for the local variant of x86 or aarch64, not for it to have a guess at a GPU target.

What you think of offload-arch=GPU for pick a plausible GPU? That distinguishes it from other things we might want to offload to. Open question whether it should create a vgpu instance if it can't detect a physical card.

There is some prior art here, e.g. CUDA and CMake, but we don't necessarily need to follow them. I'm in favor of native because it tracks with what the user "expects" native to do. It may be somewhat ambiguous given that we can offload to the host, but I think native in the future should just go with whatever we can detect no the user's system. So if someone has some future FPGA it'll detect that. We can still use native in the host way with this syntax, we just can't infer the triple. e.g. clang foo.c -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu --offload-arch=native will do what you except.

[jhuber6]

Add test

[jdoerfert]

Looks reasonable to me. See comments though

clang/include/clang/Driver/Driver.h

488

I wouldn't call it query but it's not too bad either. I'd call it "SuppressErrors".

clang/lib/Driver/Driver.cpp

894

Does this necessarily mean we failed with =native, if so it's ok. Just didn't follow the logic all the way.

[jhuber6]

clang/lib/Driver/Driver.cpp

894

I think this might trigger if someone passed --offload-arch= maybe we should treat that the same as native?

[jhuber6]

Change to SuppressError and make`--offload-arch=` just default to native.

[cmtice]

Just a heads up: This commit is causing our bootstrap build to fail (your new openmp-system-arch.c test is failing in our stage1 compiler).

[gribozavr2]

FYI: I fixed the problem in https://github.com/llvm/llvm-project/commit/0a11a1b1868dd2ab183c4313ccbfbe126e91ca08.

[jhuber6]

In D141105#4046400, @gribozavr2 wrote:

FYI: I fixed the problem in https://github.com/llvm/llvm-project/commit/0a11a1b1868dd2ab183c4313ccbfbe126e91ca08.

Thanks, I forgot to update that test after fixing a similar problem before.

clang/include/clang/Driver/Driver.h

public:
   /// \param HostAction - The host action used in the offloading toolchain.
   Action *BuildOffloadingActions(Compilation &C,
                                  llvm::opt::DerivedArgList &Args,
                                  const InputTy &Input,
                                  Action *HostAction) const;
 
   /// Returns the set of bound architectures active for this offload kind.
   /// If there are no bound architctures we return a set containing only the
-  /// empty string.
+  /// empty string. The \p Query option is used to suppress errors on failure.
   llvm::DenseSet<StringRef>
   getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
-                  Action::OffloadKind Kind, const ToolChain *TC) const;
+                  Action::OffloadKind Kind, const ToolChain *TC,
+                  bool Query = false) const;
 
   /// Check that the file referenced by Value exists. If it doesn't,
   /// issue a diagnostic and return false.
   /// If TypoCorrect is true and the file does not exist, see if it looks
   /// like a likely typo for a flag and if so print a "did you mean" blurb.
   bool DiagnoseInputExistence(const llvm::opt::DerivedArgList &Args,
                               StringRef Value, types::ID Ty,
                               bool TypoCorrect) const;

clang/lib/Driver/Driver.cpp

if (IsOpenMPOffloading) {
     } else if (C.getInputArgs().hasArg(options::OPT_offload_arch_EQ) &&
                !IsHIP && !IsCuda) {
       const ToolChain *HostTC = C.getSingleOffloadToolChain<Action::OFK_Host>();
       auto AMDTriple = getHIPOffloadTargetTriple(*this, C.getInputArgs());
       auto NVPTXTriple = getNVIDIAOffloadTargetTriple(*this, C.getInputArgs(),
                                                       HostTC->getTriple());
 
       // Attempt to deduce the offloading triple from the set of architectures.
-      // We can only correctly deduce NVPTX / AMDGPU triples currently.
-      llvm::DenseSet<StringRef> Archs =
-          getOffloadArchs(C, C.getArgs(), Action::OFK_OpenMP, nullptr);
+      // We can only correctly deduce NVPTX / AMDGPU triples currently. We need
+      // to temporarily create these toolchains so that we can access tools for
+      // inferring architectures.
+      llvm::DenseSet<StringRef> Archs;
+      if (NVPTXTriple) {
+        auto TempTC = std::make_unique<toolchains::CudaToolChain>(
+            *this, *NVPTXTriple, *HostTC, C.getInputArgs());
+        for (StringRef Arch : getOffloadArchs(
+                 C, C.getArgs(), Action::OFK_OpenMP, &*TempTC, true))
+          Archs.insert(Arch);
+      }
+      if (AMDTriple) {
+        auto TempTC = std::make_unique<toolchains::AMDGPUOpenMPToolChain>(
+            *this, *AMDTriple, *HostTC, C.getInputArgs());
+        for (StringRef Arch : getOffloadArchs(
+                 C, C.getArgs(), Action::OFK_OpenMP, &*TempTC, true))
+          Archs.insert(Arch);
+      }
+      if (!AMDTriple && !NVPTXTriple) {
+        for (StringRef Arch :
+             getOffloadArchs(C, C.getArgs(), Action::OFK_OpenMP, nullptr, true))
+          Archs.insert(Arch);
+      }
+
       for (StringRef Arch : Archs) {
         if (NVPTXTriple && IsNVIDIAGpuArch(StringToCudaArch(
                                getProcessorFromTargetID(*NVPTXTriple, Arch)))) {
           DerivedArchs[NVPTXTriple->getTriple()].insert(Arch);
         } else if (AMDTriple &&
                    IsAMDGpuArch(StringToCudaArch(
                        getProcessorFromTargetID(*AMDTriple, Arch)))) {
           DerivedArchs[AMDTriple->getTriple()].insert(Arch);
         } else {
           Diag(clang::diag::err_drv_failed_to_deduce_target_from_arch) << Arch;
           return;
         }
       }
 
+      // If the set is empty then we failed to find a native architecture.
+      if (Archs.empty()) {
+        Diag(clang::diag::err_drv_failed_to_deduce_target_from_arch)
+            << "native";
+        return;
+      }
+
       for (const auto &TripleAndArchs : DerivedArchs)
         OpenMPTriples.push_back(TripleAndArchs.first());
     }
 
     for (StringRef Val : OpenMPTriples) {
       llvm::Triple TT(ToolChain::getOpenMPTriple(Val));
       std::string NormalizedName = TT.normalize();
 
void Driver::BuildActions(Compilation &C, DerivedArgList &Args,
   Args.ClaimAllArgs(options::OPT_cl_ignored_Group);
 }
 
 /// Returns the canonical name for the offloading architecture when using a HIP
 /// or CUDA architecture.
 static StringRef getCanonicalArchString(Compilation &C,
                                         const llvm::opt::DerivedArgList &Args,
                                         StringRef ArchStr,
-                                        const llvm::Triple &Triple) {
+                                        const llvm::Triple &Triple,
+                                        bool Query = false) {
   // Lookup the CUDA / HIP architecture string. Only report an error if we were
   // expecting the triple to be only NVPTX / AMDGPU.
   CudaArch Arch = StringToCudaArch(getProcessorFromTargetID(Triple, ArchStr));
-  if (Triple.isNVPTX() &&
+  if (!Query && Triple.isNVPTX() &&
       (Arch == CudaArch::UNKNOWN || !IsNVIDIAGpuArch(Arch))) {
     C.getDriver().Diag(clang::diag::err_drv_offload_bad_gpu_arch)
         << "CUDA" << ArchStr;
     return StringRef();
-  } else if (Triple.isAMDGPU() &&
+  } else if (!Query && Triple.isAMDGPU() &&
              (Arch == CudaArch::UNKNOWN || !IsAMDGpuArch(Arch))) {
     C.getDriver().Diag(clang::diag::err_drv_offload_bad_gpu_arch)
         << "HIP" << ArchStr;
     return StringRef();
   }
 
   if (IsNVIDIAGpuArch(Arch))
     return Args.MakeArgStringRef(CudaArchToString(Arch));
getConflictOffloadArchCombination(const llvm::DenseSet<StringRef> &Archs,
 
   std::set<StringRef> ArchSet;
   llvm::copy(Archs, std::inserter(ArchSet, ArchSet.begin()));
   return getConflictTargetIDCombination(ArchSet);
 }
 
 llvm::DenseSet<StringRef>
 Driver::getOffloadArchs(Compilation &C, const llvm::opt::DerivedArgList &Args,
-                        Action::OffloadKind Kind, const ToolChain *TC) const {
+                        Action::OffloadKind Kind, const ToolChain *TC,
+                        bool Query) const {
   if (!TC)
     TC = &C.getDefaultToolChain();
 
   // --offload and --offload-arch options are mutually exclusive.
   if (Args.hasArgNoClaim(options::OPT_offload_EQ) &&
       Args.hasArgNoClaim(options::OPT_offload_arch_EQ,
                          options::OPT_no_offload_arch_EQ)) {
     C.getDriver().Diag(diag::err_opt_not_valid_with_opt)
for (auto *Arg : Args) {
 
     // Add or remove the seen architectures in order of appearance. If an
     // invalid architecture is given we simply exit.
     if (Arg->getOption().matches(options::OPT_offload_arch_EQ)) {
       for (StringRef Arch : llvm::split(Arg->getValue(), ",")) {
         if (Arch == "native") {
           auto GPUsOrErr = TC->getSystemGPUArchs(Args);
           if (!GPUsOrErr) {
+            if (Query)
+              llvm::consumeError(GPUsOrErr.takeError());
+            else
               TC->getDriver().Diag(diag::err_drv_undetermined_gpu_arch)
                   << llvm::Triple::getArchTypeName(TC->getArch())
                   << llvm::toString(GPUsOrErr.takeError()) << "--offload-arch";
             continue;
           }
 
-          for (auto ArchStr : *GPUsOrErr)
-            Archs.insert(
-                getCanonicalArchString(C, Args, ArchStr, TC->getTriple()));
+          for (auto ArchStr : *GPUsOrErr) {
+            Archs.insert(getCanonicalArchString(
+                C, Args, Args.MakeArgString(ArchStr), TC->getTriple(), Query));
+          }
         } else {
           StringRef ArchStr =
-              getCanonicalArchString(C, Args, Arch, TC->getTriple());
+              getCanonicalArchString(C, Args, Arch, TC->getTriple(), Query);
           if (ArchStr.empty())
             return Archs;
           Archs.insert(ArchStr);
         }
       }
     } else if (Arg->getOption().matches(options::OPT_no_offload_arch_EQ)) {
       for (StringRef Arch : llvm::split(Arg->getValue(), ",")) {
         if (Arch == "all") {
           Archs.clear();
         } else {
           StringRef ArchStr =
-              getCanonicalArchString(C, Args, Arch, TC->getTriple());
+              getCanonicalArchString(C, Args, Arch, TC->getTriple(), Query);
           if (ArchStr.empty())
             return Archs;
           Archs.erase(ArchStr);
         }
       }
     }
   }
 
   if (auto ConflictingArchs = getConflictOffloadArchCombination(Archs, Kind)) {
     C.getDriver().Diag(clang::diag::err_drv_bad_offload_arch_combo)
         << ConflictingArchs->first << ConflictingArchs->second;
     C.setContainsError();
   }
 
+  // Skip filling defaults if we're just querying what is availible.
+  if (Query)
+    return Archs;
+
   if (Archs.empty()) {
     if (Kind == Action::OFK_Cuda)
       Archs.insert(CudaArchToString(CudaArch::CudaDefault));
     else if (Kind == Action::OFK_HIP)
       Archs.insert(CudaArchToString(CudaArch::HIPDefault));
     else if (Kind == Action::OFK_OpenMP)
       Archs.insert(StringRef());
   } else {

clang/test/Driver/openmp-system-arch.c

+// RUN: mkdir -p %t
+// RUN: cp %S/Inputs/amdgpu-arch/amdgpu_arch_fail %t/
+// RUN: cp %S/Inputs/amdgpu-arch/amdgpu_arch_gfx906 %t/
+// RUN: cp %S/Inputs/nvptx-arch/nvptx_arch_fail %t/
+// RUN: cp %S/Inputs/nvptx-arch/nvptx_arch_sm_70 %t/
+// RUN: echo '#!/bin/sh' > %t/amdgpu_arch_empty
+// RUN: chmod +x %t/amdgpu_arch_fail
+// RUN: chmod +x %t/amdgpu_arch_gfx906
+// RUN: chmod +x %t/amdgpu_arch_empty
+// RUN: echo '#!/bin/sh' > %t/nvptx_arch_empty
+// RUN: chmod +x %t/nvptx_arch_fail
+// RUN: chmod +x %t/nvptx_arch_sm_70
+// RUN: chmod +x %t/nvptx_arch_empty
+
+// case when nvptx-arch and amdgpu-arch return nothing or fails
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_fail --amdgpu-arch-tool=%t/amdgpu_arch_fail %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=NO-OUTPUT-ERROR
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_empty --amdgpu-arch-tool=%t/amdgpu_arch_empty %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=NO-OUTPUT-ERROR
+// NO-OUTPUT-ERROR: error: failed to deduce triple for target architecture 'native'; specify the triple using '-fopenmp-targets' and '-Xopenmp-target' instead.
+
+// case when amdgpu-arch succeeds.
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_fail --amdgpu-arch-tool=%t/amdgpu_arch_gfx906 %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=ARCH-GFX906
+// ARCH-GFX906: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-target-cpu" "gfx906"
+
+// case when nvptx-arch succeeds.
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_sm_70 --amdgpu-arch-tool=%t/amdgpu_arch_fail %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=ARCH-SM_70
+// ARCH-SM_70: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_70"
+
+// case when both nvptx-arch and amdgpu-arch succeed.
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_sm_70 --amdgpu-arch-tool=%t/amdgpu_arch_gfx906 %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=ARCH-SM_70-GFX906
+// ARCH-SM_70-GFX906: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-target-cpu" "gfx906"
+// ARCH-SM_70-GFX906: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_70"
+
+// case when both nvptx-arch and amdgpu-arch succeed with other archs.
+// RUN:   %clang -### --target=x86_64-unknown-linux-gnu -nogpulib -fopenmp --offload-arch=native,sm_75,gfx1030 \
+// RUN:     --nvptx-arch-tool=%t/nvptx_arch_sm_70 --amdgpu-arch-tool=%t/amdgpu_arch_gfx906 %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=ARCH-MULTIPLE
+// ARCH-MULTIPLE: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-target-cpu" "gfx1030"
+// ARCH-MULTIPLE: "-cc1" "-triple" "amdgcn-amd-amdhsa"{{.*}}"-target-cpu" "gfx906"
+// ARCH-MULTIPLE: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_70"
+// ARCH-MULTIPLE: "-cc1" "-triple" "nvptx64-nvidia-cuda"{{.*}}"-target-cpu" "sm_75"