[OpenMP] Add a driver flag to enable the new device runtime library

[jhuber6]

Summary

This patch adds a driver flag -fopenmp-target-new-runtime to optionally enable the new device runtime
bitcode library. This allows users to enable the new experimental runtime
before it becomes the default in the future.

[jdoerfert]

We need a driver test.

[jhuber6]

Adding test

[ye-luo]

Not clear from the summary what is the new driver option.

[JonChesterfield]

Can we call this something other than new? We don't tend to remove command line arguments and this won't make much sense once it's the only runtime.

I'd be inclined to add an argument called 'use_legacy_runtime' or similar, which defaults to true, instead of this one. I.e. invert the logic

[JonChesterfield]

clang/lib/Driver/ToolChains/AMDGPUOpenMP.cpp

229

Likewise here, how about amdgcn-legacy-. Taking advantage of the monorepo + no guarantees that mix&match clang and devicertl works.

Side note, someone should probably rename the amdgcn devicertl to amdgpu, since the gfx10 stuff is the 'rdna' arch instead of the 'gcn' one.

[jhuber6]

In D106793#2904661, @ye-luo wrote:

Not clear from the summary what is the new driver option.

It's a rewrite of the current device runtime. This option will enable it.

In D106793#2904771, @JonChesterfield wrote:

Can we call this something other than new? We don't tend to remove command line arguments and this won't make much sense once it's the only runtime.

I'd be inclined to add an argument called 'use_legacy_runtime' or similar, which defaults to true, instead of this one. I.e. invert the logic

I agree somewhat that it might be easier to just call this the legacy, I'm open to it. But we had other options like -enable-new-pm for awhile before it became the default so it's probably not a big deal.

clang/lib/Driver/ToolChains/AMDGPUOpenMP.cpp

229

I think changing the BC name is something we can do any time, so just calling it libomptarget-new is fine for now.

[ye-luo]

In D106793#2904943, @jhuber6 wrote:

In D106793#2904661, @ye-luo wrote:

Not clear from the summary what is the new driver option.

It's a rewrite of the current device runtime. This option will enable it.

What is the option name? I tried not to read the source code but only the summary. Considering one day, I just want to figure out the answer via the commit log.

[jdoerfert]

I think changing the BC name is something we can do any time, so just calling it libomptarget-new is fine for now.

Yes, let's not disrupt old workflows. The "default" runtime keeps it's name always and we swap it out at some point and remove the option (or make it a no-op for one release).

[jdoerfert]

In D106793#2904771, @JonChesterfield wrote:

Can we call this something other than new? We don't tend to remove command line arguments and this won't make much sense once it's the only runtime.

We removed multiple openmp ones between 12 and 13. We'll remove this one as well as soon as we can.
Commit message spells out the option. LG

clang/include/clang/Basic/LangOptions.def

 LANGOPT(OpenMPUseTLS      , 1, 0, "Use TLS for threadprivates or runtime calls")
 LANGOPT(OpenMPIsDevice    , 1, 0, "Generate code only for OpenMP target device")
 LANGOPT(OpenMPCUDAMode    , 1, 0, "Generate code for OpenMP pragmas in SIMT/SPMD mode")
 LANGOPT(OpenMPIRBuilder   , 1, 0, "Use the experimental OpenMP-IR-Builder codegen path.")
 LANGOPT(OpenMPCUDAForceFullRuntime , 1, 0, "Force to use full runtime in all constructs when offloading to CUDA devices")
 LANGOPT(OpenMPCUDANumSMs  , 32, 0, "Number of SMs for CUDA devices.")
 LANGOPT(OpenMPCUDABlocksPerSM  , 32, 0, "Number of blocks per SM for CUDA devices.")
 LANGOPT(OpenMPCUDAReductionBufNum , 32, 1024, "Number of the reduction records in the intermediate reduction buffer used for the teams reductions.")
+LANGOPT(OpenMPTargetNewRuntime , 1, 0, "Use the new bitcode library for OpenMP offloading")
 LANGOPT(OpenMPOptimisticCollapse  , 1, 0, "Use at most 32 bits to represent the collapsed loop nest counter.")
 LANGOPT(RenderScript      , 1, 0, "RenderScript")
 
 LANGOPT(CUDAIsDevice      , 1, 0, "compiling for CUDA device")
 LANGOPT(CUDAAllowVariadicFunctions, 1, 0, "allowing variadic functions in CUDA device code")
 LANGOPT(CUDAHostDeviceConstexpr, 1, 1, "treating unattributed constexpr functions as __host__ __device__")
 LANGOPT(CUDADeviceApproxTranscendentals, 1, 0, "using approximate transcendental functions")
 LANGOPT(GPURelocatableDeviceCode, 1, 0, "generate relocatable device code")

clang/include/clang/Driver/Options.td

 def fno_openmp_cuda_force_full_runtime : Flag<["-"], "fno-openmp-cuda-force-full-runtime">, Group<f_Group>,
   Flags<[NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_number_of_sm_EQ : Joined<["-"], "fopenmp-cuda-number-of-sm=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_blocks_per_sm_EQ : Joined<["-"], "fopenmp-cuda-blocks-per-sm=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_teams_reduction_recs_num_EQ : Joined<["-"], "fopenmp-cuda-teams-reduction-recs-num=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+defm openmp_target_new_runtime: BoolFOption<"openmp-target-new-runtime",
+  LangOpts<"OpenMPTargetNewRuntime">, DefaultFalse,
+  PosFlag<SetTrue, [CC1Option], "Use the new bitcode library for OpenMP offloading">,
+  NegFlag<SetFalse>>;
 defm openmp_optimistic_collapse : BoolFOption<"openmp-optimistic-collapse",
   LangOpts<"OpenMPOptimisticCollapse">, DefaultFalse,
   PosFlag<SetTrue, [CC1Option]>, NegFlag<SetFalse>, BothFlags<[NoArgumentUnused, HelpHidden]>>;
 def static_openmp: Flag<["-"], "static-openmp">,
   HelpText<"Use the static host OpenMP runtime while linking.">;
 def fno_optimize_sibling_calls : Flag<["-"], "fno-optimize-sibling-calls">, Group<f_Group>;
 def foptimize_sibling_calls : Flag<["-"], "foptimize-sibling-calls">, Group<f_Group>;
 defm escaping_block_tail_calls : BoolFOption<"escaping-block-tail-calls",

clang/lib/Driver/ToolChains/AMDGPUOpenMP.cpp

assert(DeviceOffloadingKind == Action::OFK_OpenMP &&
          "Only OpenMP offloading kinds are supported.");
 
   CC1Args.push_back("-target-cpu");
   CC1Args.push_back(DriverArgs.MakeArgStringRef(GPUArch));
   CC1Args.push_back("-fcuda-is-device");
 
   if (DriverArgs.hasArg(options::OPT_nogpulib))
     return;
-  std::string BitcodeSuffix = "amdgcn-" + GPUArch;
+
+  std::string BitcodeSuffix;
+  if (DriverArgs.hasFlag(options::OPT_fopenmp_target_new_runtime,
+                         options::OPT_fno_openmp_target_new_runtime, false))
+    BitcodeSuffix = "new-amdgcn-" + GPUArch;
+  else
+    BitcodeSuffix = "amdgcn-" + GPUArch;
+
   addOpenMPDeviceRTL(getDriver(), DriverArgs, CC1Args, BitcodeSuffix,
                      getTriple());
 }
 
 llvm::opt::DerivedArgList *AMDGPUOpenMPToolChain::TranslateArgs(
     const llvm::opt::DerivedArgList &Args, StringRef BoundArch,
     Action::OffloadKind DeviceOffloadKind) const {
   DerivedArgList *DAL =

clang/lib/Driver/ToolChains/Cuda.cpp

#undef CASE_CUDA_VERSION
   if (DeviceOffloadingKind == Action::OFK_OpenMP) {
     if (CudaInstallationVersion < CudaVersion::CUDA_92) {
       getDriver().Diag(
           diag::err_drv_omp_offload_target_cuda_version_not_support)
           << CudaVersionToString(CudaInstallationVersion);
       return;
     }
 
-    std::string BitcodeSuffix = "nvptx-" + GpuArch.str();
+    std::string BitcodeSuffix;
+    if (DriverArgs.hasFlag(options::OPT_fopenmp_target_new_runtime,
+                           options::OPT_fno_openmp_target_new_runtime, false))
+      BitcodeSuffix = "new-nvptx-" + GpuArch.str();
+    else
+      BitcodeSuffix = "nvptx-" + GpuArch.str();
+
     addOpenMPDeviceRTL(getDriver(), DriverArgs, CC1Args, BitcodeSuffix,
                        getTriple());
   }
 }
 
 llvm::DenormalMode CudaToolChain::getDefaultDenormalModeForType(
     const llvm::opt::ArgList &DriverArgs, const JobAction &JA,
     const llvm::fltSemantics *FPType) const {

clang/test/Driver/Inputs/libomptarget/libomptarget-new-nvptx-sm_35.bc

+

clang/test/Driver/openmp-offload-gpu.c

 /// ###########################################################################
 
 /// Check that the runtime bitcode library is part of the compile line. Create a bogus
 /// bitcode library and add it to the LIBRARY_PATH.
 // RUN:   env LIBRARY_PATH=%S/Inputs/libomptarget %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda \
 // RUN:   -Xopenmp-target -march=sm_35 --cuda-path=%S/Inputs/CUDA_102/usr/local/cuda \
 // RUN:   -fopenmp-relocatable-target -save-temps -no-canonical-prefixes %s 2>&1 \
 // RUN:   | FileCheck -check-prefix=CHK-BCLIB %s
+/// Check with the new runtime enabled
+// RUN:   env LIBRARY_PATH=%S/Inputs/libomptarget %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda \
+// RUN:   -Xopenmp-target -march=sm_35 --cuda-path=%S/Inputs/CUDA_102/usr/local/cuda \
+// RUN:   -fopenmp-relocatable-target -fopenmp-target-new-runtime -save-temps -no-canonical-prefixes %s 2>&1 \
+// RUN:   | FileCheck -check-prefix=CHK-BCLIB-NEW %s
 /// The user can override default detection using --libomptarget-nvptx-bc-path=.
 // RUN:   %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda \
 // RUN:   --libomptarget-nvptx-bc-path=%S/Inputs/libomptarget/libomptarget-nvptx-test.bc \
 // RUN:   -Xopenmp-target -march=sm_35 --cuda-path=%S/Inputs/CUDA_102/usr/local/cuda \
 // RUN:   -fopenmp-relocatable-target -save-temps -no-canonical-prefixes %s 2>&1 \
 // RUN:   | FileCheck -check-prefix=CHK-BCLIB-USER %s
 
 // CHK-BCLIB: clang{{.*}}-triple{{.*}}nvptx64-nvidia-cuda{{.*}}-mlink-builtin-bitcode{{.*}}libomptarget-nvptx-sm_35.bc
+// CHK-BCLIB-NEW: clang{{.*}}-triple{{.*}}nvptx64-nvidia-cuda{{.*}}-mlink-builtin-bitcode{{.*}}libomptarget-new-nvptx-sm_35.bc
 // CHK-BCLIB-USER: clang{{.*}}-triple{{.*}}nvptx64-nvidia-cuda{{.*}}-mlink-builtin-bitcode{{.*}}libomptarget-nvptx-test.bc
 // CHK-BCLIB-NOT: {{error:|warning:}}
 
 /// ###########################################################################
 
 /// Check that the warning is thrown when the libomptarget bitcode library is not found.
 /// Libomptarget requires sm_35 or newer so an sm_35 bitcode library should never exist.
 // RUN:   %clang -### -fopenmp=libomp -fopenmp-targets=nvptx64-nvidia-cuda \