[OpenMP][MLIR][Flang][Driver][bbc] Lower and apply Module FlagsAttr

[agozillon]

Summary

This patch adds the lowering for the Module FlagsAttr to FIR (the OpenMP
dialect component) for the bbc/flang-new tool and compiler.

In addition It ports additional flags from the shared Clang compiler options to
provide this lowering to the OpenMP Dialect.

This patch enables the flags below (and any equals or inverse variants) for
Flang that exist in Clang:

-fopenmp-target-debug
-fopenmp-assume-threads-oversubscription
-fopenmp-assume-teams-oversubscription
-fopenmp-assume-no-nested-parallelism
-fopenmp-assume-no-thread-state

For the bbc tool it only utilises the primary variants to minimize additional
complexity in the tool.

The patch also provides lowering of the FlagsAttr to LLVM-IR,
each individual frontend flag (an argument to the OpenMP MLIR attribute)
is lowered to a global.

In short the patch generates an OpenMP MLIR Dialect FlagsAttr to hold
provided flag information so that it can proliferate down to the translation
to LLVM-IR that occurs in OpenMPToLLVMIRTranslation, allowing use of the
OpenMPIRBuilder to create LLVM-IR globals for the OpenMP runtime to
eventually utilise.

[agozillon]

This builds on the following phabricator patches that are in review at the moment, so there are components that are used here that are added from these:

https://reviews.llvm.org/D144864 : add fopenmp-is-device and an mlir is-device attribute for the OpenMP dialect
https://reviews.llvm.org/D144896 : the RTLModuleFlagsAttr that is in use within this patch for lowering and application
https://reviews.llvm.org/D144883 : an infrastructure addition for the MLIR LLVM translation interface

For the moment, this patch is primarily to give context to the usage of the RTLModuleFlagsAttr mlir attribute and give some further understanding for a reviewer, however, please feel free to review it if you have time as I will eventually ask for some revision of it after all the components it depends on have been progressed.

[agozillon]

• [Flang][MLIR][OpenMP] Fix rebase inconsistencies
• [Flang][bbc][Tools] Add OpenMP RTL flags to bbc and adjust setOffloadModuleInterfaceAttributes to use LangOpts
• [Flang][bbc][Tools] Update rtl-flags.f90 test for bbc tests and generally fix due to rebase changes

[agozillon]

All the mechanisms for this patch to now function are upstream, so I believe it is ready for a review if at all possible @kiranchandramohan @awarzynski and other reviewers that this patch may be of interest to.

Although if we wish for it to be split into two separate Phabricator patches, one for the lowering, the other for the flag additions and attribute creation, I can do so. However, I think this patch gives the big picture which may be useful.

As a side-note there will be a follow up commit in the near future (tomorrow if nothing steals my attention) which will do the following:

• Add an additional mlir-translate test, as I believe I owe @kiranchandramohan a test I promised from another patch!
• The clang/test/Driver/flang/flang-omp.f90 will be converted into a flang specific test and likely moved into the existing omp-frontend-forwarding.f90 test

[agozillon]

• [Flang][Driver][Test] Move flang-omp.f90 tests into omp-frontend-forwarding.f90

[agozillon]

• [Flang][Driver][Test] Move flang-omp.f90 tests into omp-frontend-forwarding.f90

[agozillon]

Recent update(s) moved flag/driver testing of the flags from Clang to Flang's omp-frontend-forwarding.f90 aligning it with previous tests.

[agozillon]

Unfortunately I do not believe an mlir-translate test that tests if the OffloadModuleInterface is accessible when directly utilizing mlir-translate is possible for this patch... I forgot I removed the is device check as it is already done at the initial creation of the attribute. However, I do have a future patch that it will be utilised in and when @jsjodin's initial TargetOp work is in I can likely create a test around that functionality.

[awarzynski]

The driver plumbing looks good to me. I will defer reviewing the OpenMP changes to experts. Thanks for working on this!

clang/include/clang/Driver/Options.td

2688

Many of these options have identical flags. While not really needed for this change, it would still be nice to re-organise them a bit. This file could really benefit from some love :) Here's an example of what I have in mind: https://github.com/llvm/llvm-project/blob/cf60d3f1a688671c8eb7859bf0572c403c3c0cca/clang/include/clang/Driver/Options.td#L6575-L6600

flang/tools/bbc/CMakeLists.txt

29 ↗

(On Diff #509628)

This a frontend driver library and so far bbc and flang-new -fc1 have been entirely separate. Could this dependency be avoided?

[agozillon]

clang/include/clang/Driver/Options.td

2688

I can have a look into doing this in this patch :-)

flang/tools/bbc/CMakeLists.txt

29 ↗

(On Diff #509628)

I had hoped to share LangOpts so that the setOffloadModuleInterfaceAttributes function wouldn't turn into a monolithic set of arguments whenever it's invoked if more arguments are added, but the dependency isn't ideal I agree!

I could perhaps look into making some sort of shared data structure to be put inside of CrossToolHelpers that might remove the dependency and be similarly useable to how LangOpts works at the moment. If that doesn't work, I can revert the change to just be a regular argument list and we can revisit the topic if new options are ever added?

[agozillon]

flang/tools/bbc/CMakeLists.txt

29 ↗

(On Diff #509628)

Although looking at @domada's https://reviews.llvm.org/D146612 patch it reminds me that they could just be separate functions shared across tools (I perhaps got a little fixated on the idea of it being similar to a driver function handling all the options). Please do tell me whichever you'd prefer :-)

[agozillon]

Squash commits, format and apply requested changes

[agozillon]

clang/include/clang/Driver/Options.td

2688

Made an attempt at tidying these up with the recent commit, hopefully this is close to what you expected!

flang/tools/bbc/CMakeLists.txt

29 ↗

(On Diff #509628)

I removed the dependency via a helper class, that will keep flang-new a little cleaner hopefully and allow bbc to utilise the same logic provided it populates an intermediate structure!

But alternatively it's possible to split into separate functions for each attribute that depends on options if that's the desired direction!

[awarzynski]

@agozillon Thanks for the updates, LGTM! Please wait for somebody to review the OpenMP changes before merging.

[kiranchandramohan]

Please split this patch into three:

1. Code changes and testing for the driver and the FIR+OpenMP dialect generated.
2. Code changes and test for FIR+OpenMP to LLVM+OpenMP dialect.
3. Code changes and testing for the translation from LLVM + OpenMP dialect to LLVM IR. Code in OpenMPToLLVMIRTranslation.cpp should be tested with mlir-translate.

In D145264#4233358, @agozillon wrote:

Unfortunately I do not believe an mlir-translate test that tests if the OffloadModuleInterface is accessible when directly utilizing mlir-translate is possible for this patch... I forgot I removed the is device check as it is already done at the initial creation of the attribute. However, I do have a future patch that it will be utilised in and when @jsjodin's initial TargetOp work is in I can likely create a test around that functionality.

You may delay the translation code till then.

[agozillon]

In D145264#4236173, @kiranchandramohan wrote:

Please split this patch into three:

1. Code changes and testing for the driver and the FIR+OpenMP dialect generated.
2. Code changes and test for FIR+OpenMP to LLVM+OpenMP dialect.
3. Code changes and testing for the translation from LLVM + OpenMP dialect to LLVM IR. Code in OpenMPToLLVMIRTranslation.cpp should be tested with mlir-translate.

More than happy to do all of the above, however, there is no code for step 2 in this case as it undergoes no rewriting or transformation when going from FIR+OpenMP -> LLVM+OpenMP. However, I am happy to make a test to assure it is retained during the conversion, but I believe it would just be a test in the patch, would it be okay to merge patch 2 and 3 from your comment? Meaning there would be two patches, the driver additions and the attribute generation from the flags. And then the lowering of the attribute to LLVM-IR (with additional mlir-translate tests for LLVM+OpenMP -> LLVM-IR and FIR+OpenMP -> LLVM+OpenMP).

In D145264#4233358, @agozillon wrote:

Unfortunately I do not believe an mlir-translate test that tests if the OffloadModuleInterface is accessible when directly utilizing mlir-translate is possible for this patch... I forgot I removed the is device check as it is already done at the initial creation of the attribute. However, I do have a future patch that it will be utilised in and when @jsjodin's initial TargetOp work is in I can likely create a test around that functionality.

You may delay the translation code till then.

Thank you.

[agozillon]

Fragmented this patch into https://reviews.llvm.org/D147344 (lowering) and https://reviews.llvm.org/D147324 (driver/tool changes and application of attribute). I will keep this patch open until the others are closed to give a big picture for easier reference.

[agozillon]

Closing this patch now that each individual component is upstreamed via seperate phabricator patches! Thank you to all the reviewers for your time and help.

clang/include/clang/Driver/Options.td

 def fno_openmp_cuda_mode : Flag<["-"], "fno-openmp-cuda-mode">, Group<f_Group>,
   Flags<[NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_number_of_sm_EQ : Joined<["-"], "fopenmp-cuda-number-of-sm=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_blocks_per_sm_EQ : Joined<["-"], "fopenmp-cuda-blocks-per-sm=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_cuda_teams_reduction_recs_num_EQ : Joined<["-"], "fopenmp-cuda-teams-reduction-recs-num=">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
-def fopenmp_target_debug : Flag<["-"], "fopenmp-target-debug">, Group<f_Group>, Flags<[CC1Option, NoArgumentUnused]>,
+def fopenmp_target_debug : Flag<["-"], "fopenmp-target-debug">, Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused]>,
   HelpText<"Enable debugging in the OpenMP offloading device RTL">;
-def fno_openmp_target_debug : Flag<["-"], "fno-openmp-target-debug">, Group<f_Group>, Flags<[NoArgumentUnused]>;
-def fopenmp_target_debug_EQ : Joined<["-"], "fopenmp-target-debug=">, Group<f_Group>, Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+def fno_openmp_target_debug : Flag<["-"], "fno-openmp-target-debug">, Group<f_Group>, Flags<[NoArgumentUnused, FC1Option]>;
+def fopenmp_target_debug_EQ : Joined<["-"], "fopenmp-target-debug=">, Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_assume_teams_oversubscription : Flag<["-"], "fopenmp-assume-teams-oversubscription">,
-  Group<f_Group>, Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+  Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_assume_threads_oversubscription : Flag<["-"], "fopenmp-assume-threads-oversubscription">,
-  Group<f_Group>, Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+  Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>;
 def fno_openmp_assume_teams_oversubscription : Flag<["-"], "fno-openmp-assume-teams-oversubscription">,
-  Group<f_Group>, Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+  Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>;
 def fno_openmp_assume_threads_oversubscription : Flag<["-"], "fno-openmp-assume-threads-oversubscription">,
-  Group<f_Group>, Flags<[CC1Option, NoArgumentUnused, HelpHidden]>;
+  Group<f_Group>, Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>;
 def fopenmp_assume_no_thread_state : Flag<["-"], "fopenmp-assume-no-thread-state">, Group<f_Group>,
-  Flags<[CC1Option, NoArgumentUnused, HelpHidden]>,
+  Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>,
   HelpText<"Assert no thread in a parallel region modifies an ICV">,
   MarshallingInfoFlag<LangOpts<"OpenMPNoThreadState">>;
 def fopenmp_assume_no_nested_parallelism : Flag<["-"], "fopenmp-assume-no-nested-parallelism">, Group<f_Group>,
-  Flags<[CC1Option, NoArgumentUnused, HelpHidden]>,
+  Flags<[CC1Option, FC1Option, NoArgumentUnused, HelpHidden]>,
   HelpText<"Assert no nested parallel regions in the GPU">,
   MarshallingInfoFlag<LangOpts<"OpenMPNoNestedParallelism">>;
 def fopenmp_offload_mandatory : Flag<["-"], "fopenmp-offload-mandatory">, Group<f_Group>,
   Flags<[CC1Option, NoArgumentUnused]>,
   HelpText<"Do not create a host fallback if offloading to the device fails.">,
   MarshallingInfoFlag<LangOpts<"OpenMPOffloadMandatory">>;
 def fopenmp_target_jit : Flag<["-"], "fopenmp-target-jit">, Group<f_Group>,
   Flags<[CoreOption, NoArgumentUnused]>,

clang/lib/Driver/ToolChains/Flang.h

private:
   /// Extract target options from the driver arguments and add them to
   /// the command arguments.
   ///
   /// \param [in] Args The list of input driver arguments
   /// \param [out] CmdArgs The list of output command arguments
   void addTargetOptions(const llvm::opt::ArgList &Args,
                         llvm::opt::ArgStringList &CmdArgs) const;
 
+  /// Extract offload options from the driver arguments and add them to
+  /// the command arguments.
+  ///
+  /// \param [in] JA The job action
+  /// \param [out] CmdArgs The list of output command arguments
+  void addOffloadOptions(const JobAction &JA, const llvm::opt::ArgList &Args,
+                         llvm::opt::ArgStringList &CmdArgs) const;
+
   /// Extract other compilation options from the driver arguments and add them
   /// to the command arguments.
   ///
   /// \param [in] Args The list of input driver arguments
   /// \param [out] CmdArgs The list of output command arguments
   void addOtherOptions(const llvm::opt::ArgList &Args,
                        llvm::opt::ArgStringList &CmdArgs) const;
 

clang/lib/Driver/ToolChains/Flang.cpp

void Flang::addTargetOptions(const ArgList &Args,
   case llvm::Triple::x86_64:
     getTargetFeatures(D, Triple, Args, CmdArgs, /*ForAs*/ false);
     break;
   }
 
   // TODO: Add target specific flags, ABI, mtune option etc.
 }
 
+void Flang::addOffloadOptions(const JobAction &JA, const ArgList &Args,
+                              ArgStringList &CmdArgs) const {
+  bool IsOpenMPDevice = JA.isDeviceOffloading(Action::OFK_OpenMP);
+
+  if (IsOpenMPDevice) {
+    // When in OpenMP offloading mode, enable debugging on the device.
+    Args.AddAllArgs(CmdArgs, options::OPT_fopenmp_target_debug_EQ);
+    if (Args.hasFlag(options::OPT_fopenmp_target_debug,
+                     options::OPT_fno_openmp_target_debug, /*Default=*/false))
+      CmdArgs.push_back("-fopenmp-target-debug");
+
+    // When in OpenMP offloading mode, forward assumptions information about
+    // thread and team counts in the device.
+    if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
+                     options::OPT_fno_openmp_assume_teams_oversubscription,
+                     /*Default=*/false))
+      CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
+    if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
+                     options::OPT_fno_openmp_assume_threads_oversubscription,
+                     /*Default=*/false))
+      CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
+    if (Args.hasArg(options::OPT_fopenmp_assume_no_thread_state))
+      CmdArgs.push_back("-fopenmp-assume-no-thread-state");
+    if (Args.hasArg(options::OPT_fopenmp_assume_no_nested_parallelism))
+      CmdArgs.push_back("-fopenmp-assume-no-nested-parallelism");
+  }
+}
+
 static void addFloatingPointOptions(const Driver &D, const ArgList &Args,
                                     ArgStringList &CmdArgs) {
   StringRef FPContract;
   bool HonorINFs = true;
   bool HonorNaNs = true;
   bool ApproxFunc = false;
   bool SignedZeros = true;
   bool AssociativeMath = false;
void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   addPicOptions(Args, CmdArgs);
 
   // Floating point related options
   addFloatingPointOptions(D, Args, CmdArgs);
 
   // Add target args, features, etc.
   addTargetOptions(Args, CmdArgs);
 
+  // Offloading related options
+  addOffloadOptions(JA, Args, CmdArgs);
+
   // Add other compile options
   addOtherOptions(Args, CmdArgs);
 
   // Forward -Xflang arguments to -fc1
   Args.AddAllArgValues(CmdArgs, options::OPT_Xflang);
 
   // Forward -mllvm options to the LLVM option parser. In practice, this means
   // forwarding to `-fc1` as that's where the LLVM parser is run.

clang/test/Driver/flang/flang-omp.f90

+! Check that flang -fc1 is invoked when in --driver-mode=flang 
+! and the relevant openmp and openmp offload flags are utilised
+! and passed down correctly
+
+! Test -fopenmp with offload for RTL Flag Options
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-assume-threads-oversubscription %s 2>&1 | FileCheck --check-prefixes=CHECK-THREADS-OVS %s
+! CHECK-THREADS-OVS: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-assume-threads-oversubscription" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-assume-teams-oversubscription %s 2>&1 | FileCheck --check-prefixes=CHECK-TEAMS-OVS %s
+! CHECK-TEAMS-OVS: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-assume-teams-oversubscription" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-assume-no-nested-parallelism %s 2>&1 | FileCheck --check-prefixes=CHECK-NEST-PAR %s
+! CHECK-NEST-PAR: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-assume-no-nested-parallelism" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-assume-no-thread-state %s 2>&1 | FileCheck --check-prefixes=CHECK-THREAD-STATE %s
+! CHECK-THREAD-STATE: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-assume-no-thread-state" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-target-debug %s 2>&1 | FileCheck --check-prefixes=CHECK-TARGET-DEBUG %s
+! CHECK-TARGET-DEBUG: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-target-debug" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-target-debug=111 %s 2>&1 | FileCheck --check-prefixes=CHECK-TARGET-DEBUG-EQ %s
+! CHECK-TARGET-DEBUG-EQ: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-target-debug=111" {{.*}}.f90"
+
+! RUN: %clang --driver-mode=flang -### -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -fopenmp-target-debug -fopenmp-assume-threads-oversubscription \
+! RUN: -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism -fopenmp-assume-no-thread-state \
+! RUN: %s 2>&1 | FileCheck --check-prefixes=CHECK-RTL-ALL %s
+! CHECK-RTL-ALL: "{{[^"]*}}flang-new" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-device" "-fopenmp-target-debug" "-fopenmp-assume-teams-oversubscription" 
+! CHECK-RTL-ALL: "-fopenmp-assume-threads-oversubscription" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
+! CHECK-RTL-ALL: {{.*}}.f90"
 No newline at end of file

flang/include/flang/Frontend/LangOptions.def

 /// calculation
 LANGOPT(ApproxFunc, 1, false)
 /// Allow optimizations that ignore the sign of floating point zeros
 LANGOPT(NoSignedZeros, 1, false)
 /// Allow reassociation transformations for floating-point instructions
 LANGOPT(AssociativeMath, 1, false)
 /// Allow division operations to be reassociated
 LANGOPT(ReciprocalMath, 1, false)
+/// Enable debugging in the OpenMP offloading device RTL
+LANGOPT(OpenMPTargetDebug, 32, 0)
+/// Assume work-shared loops do not have more iterations than participating
+/// threads.
+LANGOPT(OpenMPThreadSubscription, 1, 0)
+/// Assume distributed loops do not have more iterations than participating
+/// teams.
+LANGOPT(OpenMPTeamSubscription, 1, 0)
+/// Assume that no thread in a parallel region will modify an ICV.
+LANGOPT(OpenMPNoThreadState, 1, 0)
+/// Assume that no thread in a parallel region will encounter a parallel region
+LANGOPT(OpenMPNoNestedParallelism, 1, 0)
 
 #undef LANGOPT
 #undef ENUM_LANGOPT

flang/lib/Frontend/CompilerInvocation.cpp

static bool parseDialectArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
   // -fopenmp and -fopenacc
   if (args.hasArg(clang::driver::options::OPT_fopenacc)) {
     res.getFrontendOpts().features.Enable(
         Fortran::common::LanguageFeature::OpenACC);
   }
   if (args.hasArg(clang::driver::options::OPT_fopenmp)) {
     res.getFrontendOpts().features.Enable(
         Fortran::common::LanguageFeature::OpenMP);
+
+    if (args.hasArg(clang::driver::options::OPT_fopenmp_is_device)) {
+      if (args.hasFlag(
+              clang::driver::options::OPT_fopenmp_assume_teams_oversubscription,
+              clang::driver::options::
+                  OPT_fno_openmp_assume_teams_oversubscription,
+              /*Default=*/false))
+        res.getLangOpts().OpenMPTeamSubscription = true;
+
+      if (args.hasArg(
+              clang::driver::options::OPT_fopenmp_assume_no_thread_state))
+        res.getLangOpts().OpenMPNoThreadState = 1;
+
+      if (args.hasArg(
+              clang::driver::options::OPT_fopenmp_assume_no_nested_parallelism))
+        res.getLangOpts().OpenMPNoNestedParallelism = 1;
+
+      if (args.hasFlag(clang::driver::options::
+                           OPT_fopenmp_assume_threads_oversubscription,
+                       clang::driver::options::
+                           OPT_fno_openmp_assume_threads_oversubscription,
+                       /*Default=*/false))
+        res.getLangOpts().OpenMPThreadSubscription = true;
+
+      if ((args.hasArg(clang::driver::options::OPT_fopenmp_target_debug) ||
+           args.hasArg(clang::driver::options::OPT_fopenmp_target_debug_EQ))) {
+        res.getLangOpts().OpenMPTargetDebug = getLastArgIntValue(
+            args, clang::driver::options::OPT_fopenmp_target_debug_EQ,
+            res.getLangOpts().OpenMPTargetDebug, diags);
+
+        if (!res.getLangOpts().OpenMPTargetDebug &&
+            args.hasArg(clang::driver::options::OPT_fopenmp_target_debug))
+          res.getLangOpts().OpenMPTargetDebug = 1;
+      }
+    }
   }
 
   // -pedantic
   if (args.hasArg(clang::driver::options::OPT_pedantic)) {
     res.setEnableConformanceChecks();
   }
   // -std=f2018 (currently this implies -pedantic)
   // TODO: Set proper options when more fortran standards

flang/lib/Frontend/FrontendActions.cpp

lower::LoweringBridge lb = Fortran::lower::LoweringBridge::create(
       ci.getInvocation().getSemanticsContext().intrinsics(),
       ci.getInvocation().getSemanticsContext().targetCharacteristics(),
       ci.getParsing().allCooked(), ci.getInvocation().getTargetOpts().triple,
       kindMap, ci.getInvocation().getLoweringOpts(),
       ci.getInvocation().getFrontendOpts().envDefaults);
 
   // Fetch module from lb, so we can set
   mlirModule = std::make_unique<mlir::ModuleOp>(lb.getModule());
+
+  if (ci.getInvocation().getFrontendOpts().features.IsEnabled(
+          Fortran::common::LanguageFeature::OpenMP)) {
+    if (ci.getInvocation().getLangOpts().OpenMPIsDevice)
+      mlir::omp::OpenMPDialect::setRTLFlags(
+          *mlirModule, ci.getInvocation().getLangOpts().OpenMPTargetDebug,
+          ci.getInvocation().getLangOpts().OpenMPTeamSubscription,
+          ci.getInvocation().getLangOpts().OpenMPThreadSubscription,
+          ci.getInvocation().getLangOpts().OpenMPNoThreadState,
+          ci.getInvocation().getLangOpts().OpenMPNoNestedParallelism);
+  }
+
   setUpTargetMachine();
   const llvm::DataLayout &dl = tm->createDataLayout();
   setMLIRDataLayout(*mlirModule, dl);
 
   // Create a parse tree and lower it to FIR
   Fortran::parser::Program &parseTree{*ci.getParsing().parseTree()};
   lb.lower(parseTree, ci.getInvocation().getSemanticsContext());
 

flang/test/Driver/driver-help.f90

 ! HELP-FC1-NEXT: -fno-analyzed-objects-for-unparse
 ! HELP-FC1-NEXT:                        Do not use the analyzed objects when unparsing
 ! HELP-FC1-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! HELP-FC1-NEXT: -fno-debug-pass-manager Disables debug printing for the new pass manager
 ! HELP-FC1-NEXT: -fno-reformat          Dump the cooked character stream in -E mode
 ! HELP-FC1-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros
 ! HELP-FC1-NEXT: -fno-stack-arrays      Allocate array temporaries on the heap (default)
 ! HELP-FC1-NEXT: -fopenacc              Enable OpenACC
+! HELP-FC1-NEXT: -fopenmp-target-debug  Enable debugging in the OpenMP offloading device RTL
 ! HELP-FC1-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
 ! HELP-FC1-NEXT: -fpass-plugin=<dsopath> Load pass plugin from a dynamic shared object file (only with new pass manager).
 ! HELP-FC1-NEXT: -freciprocal-math      Allow division operations to be reassociated
 ! HELP-FC1-NEXT: -fstack-arrays         Attempt to allocate array temporaries on the stack, no matter their size
 ! HELP-FC1-NEXT: -fsyntax-only          Run the preprocessor, parser and semantic analysis stages
 ! HELP-FC1-NEXT: -funderscoring         Appends one trailing underscore to external names
 ! HELP-FC1-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! HELP-FC1-NEXT: -help                  Display available options

flang/test/Lower/OpenMP/rtl-flags.f90

+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DEFAULT-DEVICE-FIR
+!RUN: %flang_fc1 -emit-fir -fopenmp %s -o - | FileCheck %s --check-prefix=DEFAULT-HOST-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DEFAULT-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-target-debug -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DBG-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-target-debug -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DBG-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-target-debug=111 -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DBG-EQ-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-target-debug=111 -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=DBG-EQ-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=TEAMS-OSUB-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=TEAMS-OSUB-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-assume-threads-oversubscription -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=THREAD-OSUB-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-assume-threads-oversubscription -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=THREAD-OSUB-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-assume-no-thread-state -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=THREAD-STATE-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-assume-no-thread-state -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=THREAD-STATE-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-assume-no-nested-parallelism -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=NEST-PAR-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-assume-no-nested-parallelism -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=NEST-PAR-DEVICE-LLVM
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-target-debug -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism -fopenmp-assume-threads-oversubscription -fopenmp-assume-no-thread-state -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=ALL-DEVICE-FIR
+!RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-target-debug -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism -fopenmp-assume-threads-oversubscription -fopenmp-assume-no-thread-state -fopenmp-is-device %s -o - | FileCheck %s --check-prefix=ALL-DEVICE-LLVM
+
+!DEFAULT-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<>}
+!DEFAULT-HOST-FIR: module attributes {{{.*}}, omp.is_device = false}
+!DEFAULT-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
+!DEFAULT-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!DEFAULT-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!DEFAULT-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!DEFAULT-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!DBG-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<debug_kind = 1>}
+!DBG-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 1
+!DBG-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!DBG-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!DBG-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!DBG-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!DBG-EQ-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<debug_kind = 111>}
+!DBG-EQ-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 111
+!DBG-EQ-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!DBG-EQ-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!DBG-EQ-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!DBG-EQ-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!TEAMS-OSUB-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<assume_teams_oversubscription = true>}
+!TEAMS-OSUB-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
+!TEAMS-OSUB-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 1
+!TEAMS-OSUB-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!TEAMS-OSUB-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!TEAMS-OSUB-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!THREAD-OSUB-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<assume_threads_oversubscription = true>}
+!THREAD-OSUB-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
+!THREAD-OSUB-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!THREAD-OSUB-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 1
+!THREAD-OSUB-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!THREAD-OSUB-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!THREAD-STATE-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<assume_no_thread_state = true>}
+!THREAD-STATE-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
+!THREAD-STATE-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!THREAD-STATE-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!THREAD-STATE-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 1
+!THREAD-STATE-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
+
+!NEST-PAR-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<assume_no_nested_parallelism = true>}
+!NEST-PAR-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
+!NEST-PAR-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 0
+!NEST-PAR-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
+!NEST-PAR-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
+!NEST-PAR-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 1
+
+!ALL-DEVICE-FIR: module attributes {{{.*}}, omp.is_device = true, omp.rtlmoduleflags = #omp.rtlmoduleflags<debug_kind = 1, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = true, assume_no_nested_parallelism = true>}
+!ALL-DEVICE-LLVM: @__omp_rtl_debug_kind = weak_odr hidden constant i32 1
+!ALL-DEVICE-LLVM: @__omp_rtl_assume_teams_oversubscription = weak_odr hidden constant i32 1
+!ALL-DEVICE-LLVM: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 1
+!ALL-DEVICE-LLVM: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 1
+!ALL-DEVICE-LLVM: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 1
+subroutine omp_subroutine()
+end subroutine omp_subroutine

mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp

convertOmpThreadprivate(Operation &opInst, llvm::IRBuilderBase &builder,
   llvm::Value *callInst =
       moduleTranslation.getOpenMPBuilder()->createCachedThreadPrivate(
           ompLoc, data, size, cacheName);
   llvm::Value *result = builder.CreateBitCast(callInst, globalValue->getType());
   moduleTranslation.mapValue(opInst.getResult(0), result);
   return success();
 }
 
+/// Lowers the RTLModuleFlagsAttr which is applied to the module on the device
+/// pass when offloading, this attribute contains OpenMP RTL globals that can
+/// be passed as flags to the frontend, otherwise they are set to default
+LogicalResult
+convertRTLModuleFlagsAttr(Operation *op,
+                          mlir::omp::RTLModuleFlagsAttr attribute,
+                          LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  ompBuilder->createGlobalFlag(
+      attribute.getDebugKind() /*LangOpts().OpenMPTargetDebug*/,
+      "__omp_rtl_debug_kind");
+  ompBuilder->createGlobalFlag(
+      attribute
+          .getAssumeTeamsOversubscription() /*LangOpts().OpenMPTeamSubscription*/
+      ,
+      "__omp_rtl_assume_teams_oversubscription");
+  ompBuilder->createGlobalFlag(
+      attribute
+          .getAssumeThreadsOversubscription() /*LangOpts().OpenMPThreadSubscription*/
+      ,
+      "__omp_rtl_assume_threads_oversubscription");
+  ompBuilder->createGlobalFlag(
+      attribute.getAssumeNoThreadState() /*LangOpts().OpenMPNoThreadState*/,
+      "__omp_rtl_assume_no_thread_state");
+  ompBuilder->createGlobalFlag(
+      attribute
+          .getAssumeNoNestedParallelism() /*LangOpts().OpenMPNoNestedParallelism*/
+      ,
+      "__omp_rtl_assume_no_nested_parallelism");
+
+  return success();
+}
+
 namespace {
 
 /// Implementation of the dialect interface that converts operations belonging
 /// to the OpenMP dialect to LLVM IR.
 class OpenMPDialectLLVMIRTranslationInterface
     : public LLVMTranslationDialectInterface {
 public:
   using LLVMTranslationDialectInterface::LLVMTranslationDialectInterface;
 
   /// Translates the given operation to LLVM IR using the provided IR builder
   /// and saving the state in `moduleTranslation`.
   LogicalResult
   convertOperation(Operation *op, llvm::IRBuilderBase &builder,
                    LLVM::ModuleTranslation &moduleTranslation) const final;
+
+  LogicalResult
+  amendOperation(Operation *op, NamedAttribute attribute,
+                 LLVM::ModuleTranslation &moduleTranslation) const final;
 };
 
 } // namespace
 
+/// Given an OpenMP MLIR attribute, create the corresponding LLVM-IR, runtime
+/// calls, or operation amendments
+LogicalResult OpenMPDialectLLVMIRTranslationInterface::amendOperation(
+    Operation *op, NamedAttribute attribute,
+    LLVM::ModuleTranslation &moduleTranslation) const {
+
+  return llvm::TypeSwitch<Attribute, LogicalResult>(attribute.getValue())
+      .Case([&](mlir::omp::RTLModuleFlagsAttr rtlAttr) {
+        return convertRTLModuleFlagsAttr(op, rtlAttr, moduleTranslation);
+      })
+      .Default([&](Attribute attr) {
+        // fall through for omp attributes that do not require lowering and/or
+        // have no concrete definition and thus no type to define a case on
+        // e.g. omp.is_device
+        return success();
+      });
+
+  return failure();
+}
+
 /// Given an OpenMP MLIR operation, create the corresponding LLVM IR
 /// (including OpenMP runtime calls).
 LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
     Operation *op, llvm::IRBuilderBase &builder,
     LLVM::ModuleTranslation &moduleTranslation) const {
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();