[OpenMP][MLIR] Add OpenMP version attribute to OMP dialect

[domada]

Summary

The intent of OpenMP version attribute is to be applied to a module and then hold information on OpenMP version flag.

Information about OpenMP version can be specified in Clang by flag -fopenmp-version and it is stored in LLVM-IR module metadata:

!llvm.module.flags = !{!0, !1}
!0 = !{i32 7, !"openmp", i32 51}
!1 = !{i32 7, !"openmp-device", i32 51}

OpenMP MLIR version attributes will allow to lower Flang frontend flag to OpenMP MLIR code and then to LLVM IR.

There are two OpenMP MLIR version attributes. The first one "omp.version" MLIR attribute corresponds to host OpenMP version. The second one corresponds to "openmp-device" LLVM-IR metadata. "openmp-device" LLVM-IR metadata is attached only for offloaded code.

[kiranchandramohan]

Will the translation to LLVM be through amendOperation?

mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td

75

There seem to be two different defaults, 4.5 and 5.0 (In the Interface).

What is the real default and is it different for openmp and openmp-device?

[domada]

Fixed default version of OpenMP

[domada]

mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td

75

I corrected it to 50. In clang we use one flag to specify the version of OpenMP for the device and host. By default this flag is set to 50.

[domada]

In D150351#4334598, @kiranchandramohan wrote:

Will the translation to LLVM be through amendOperation?

That's my plan. If you wish I can wait till the output of @skatrak proposal: https://reviews.llvm.org/D150275

[kiranchandramohan]

In D150351#4334608, @domada wrote:

In D150351#4334598, @kiranchandramohan wrote:

Will the translation to LLVM be through amendOperation?

That's my plan. If you wish I can wait till the output of @skatrak proposal: https://reviews.llvm.org/D150275

No need to wait. I believe that proposal was only to demonstrate that there is a way for out of tree users to use the Offload Attributes in the Dialect.
Also, if it uses amendOperation then you can consider removing the getVersion since we will not be using it.

Another question I have is whether the flags attribute will always be present with -fopenmp or only with some device/target specific options.

[skatrak]

In D150351#4334608, @domada wrote:

In D150351#4334598, @kiranchandramohan wrote:

Will the translation to LLVM be through amendOperation?

That's my plan. If you wish I can wait till the output of @skatrak proposal: https://reviews.llvm.org/D150275

My patch was just to show that there are alternatives, purely within MLIR, to the amendOperation flow for lowering discardable dialect attributes attached to operations from other dialects. AFAIK there's no intention to use that approach for the time being at least.

[agozillon]

In D150351#4334624, @kiranchandramohan wrote:

In D150351#4334608, @domada wrote:

In D150351#4334598, @kiranchandramohan wrote:

Will the translation to LLVM be through amendOperation?

That's my plan. If you wish I can wait till the output of @skatrak proposal: https://reviews.llvm.org/D150275

No need to wait. I believe that proposal was only to demonstrate that there is a way for out of tree users to use the Offload Attributes in the Dialect.
Also, if it uses amendOperation then you can consider removing the getVersion since we will not be using it.

Another question I have is whether the flags attribute will always be present with -fopenmp or only with some device/target specific options.

Currently the flags were only present on device (and the original intent was it only being these flags, but it makes sense for it to extend broader to cover these flags as well), but that was because the flags that were in the attribute originally are only ever present on the device/target. However, all you would need to do to change this is move the is_device check that applies them in the frontend (initial attribute creation) to the back end (MLIR attribute -> LLVM IR) and then use the is_device flag in amendOperation to filter and generate the flags you wish. This would mean the attribute is present for host and device whenever -fopenmp is specified however, although I don't necessarily see a problem with that!

[domada]

Resolve dependency between parent and child revision.

[domada]

In D150351#4334629, @agozillon wrote:

In D150351#4334624, @kiranchandramohan wrote:

In D150351#4334608, @domada wrote:

In D150351#4334598, @kiranchandramohan wrote:

Will the translation to LLVM be through amendOperation?

That's my plan. If you wish I can wait till the output of @skatrak proposal: https://reviews.llvm.org/D150275

No need to wait. I believe that proposal was only to demonstrate that there is a way for out of tree users to use the Offload Attributes in the Dialect.
Also, if it uses amendOperation then you can consider removing the getVersion since we will not be using it.

Another question I have is whether the flags attribute will always be present with -fopenmp or only with some device/target specific options.

Currently the flags were only present on device (and the original intent was it only being these flags, but it makes sense for it to extend broader to cover these flags as well), but that was because the flags that were in the attribute originally are only ever present on the device/target. However, all you would need to do to change this is move the is_device check that applies them in the frontend (initial attribute creation) to the back end (MLIR attribute -> LLVM IR) and then use the is_device flag in amendOperation to filter and generate the flags you wish. This would mean the attribute is present for host and device whenever -fopenmp is specified however, although I don't necessarily see a problem with that!

Clang specifies two metadata (for host and the device) for OpenMP version:

!llvm.module.flags = !{!0, !1}
!0 = !{i32 7, !"openmp", i32 51}
!1 = !{i32 7, !"openmp-device", i32 51}

That's why I decided to reflect it in MLIR and add two fields in MLIR code (one for host - and the second for the device).

[kiranchandramohan]

Could you clarify in the commit message that there are two attributes here: One representing the device version (in Flags Attr) and a standalone one called omp.version?

mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td

86–108

Since this is not related to Device or Offloading, I think the omp.version set or get should be removed from this interface.

mlir/test/Dialect/OpenMP/attr.mlir

32

Can we have a roundtrip test for omp.version?

[domada]

Applied remarks

[domada]

In D150351#4337358, @kiranchandramohan wrote:

Could you clarify in the commit message that there are two attributes here: One representing the device version (in Flags Attr) and a standalone one called omp.version?

Please look at the modified description of this patch. Is it ok for you?

[domada]

mlir/test/Dialect/OpenMP/attr.mlir

32

I added test case:
// CHECK: module attributes {omp.version = #omp.version<version = 51>} {
module attributes {omp.version = #omp.version<version = 51>} {}

[kiranchandramohan]

LG.

Where will the omp.version be set in Flang? Assuming it is in a separate patch.

flang/include/flang/Tools/CrossToolHelpers.h

64–65

[domada]

In D150351#4342442, @kiranchandramohan wrote:

LG.

Where will the omp.version be set in Flang? Assuming it is in a separate patch.

Please look at: https://reviews.llvm.org/D150354 . Andrzej made remark about default version of OpenMP. If you wish you can express your opinion about it.

flang/include/flang/Tools/CrossToolHelpers.h

void setOffloadModuleInterfaceAttributes(
     mlir::ModuleOp &module, OffloadModuleOpts Opts) {
   // Should be registered by the OpenMPDialect
   if (auto offloadMod = llvm::dyn_cast<mlir::omp::OffloadModuleInterface>(
           module.getOperation())) {
     offloadMod.setIsDevice(Opts.OpenMPIsDevice);
     if (Opts.OpenMPIsDevice) {
       offloadMod.setFlags(Opts.OpenMPTargetDebug, Opts.OpenMPTeamSubscription,
           Opts.OpenMPThreadSubscription, Opts.OpenMPNoThreadState,
-          Opts.OpenMPNoNestedParallelism);
+          Opts.OpenMPNoNestedParallelism,
+          50 /* Default OpenMP Device version */);
 
       if (!Opts.OMPHostIRFile.empty())
         offloadMod.setHostIRFilePath(Opts.OMPHostIRFile);
     }
   }
 }
 
 //  Shares assinging of the OpenMP OffloadModuleInterface and its TargetCPU

mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td

 def IsDeviceAttr : OpenMP_Attr<"IsDevice", "isdevice"> {
   let parameters = (ins
     "bool":$is_device
   );
 
   let assemblyFormat = "`<` struct(params) `>`";
 }
 
+def VersionAttr : OpenMP_Attr<"Version", "version"> {
+  let parameters = (ins
+    "uint32_t":$version
+  );
+
+  let assemblyFormat = "`<` struct(params) `>`";
+}
+
 //===----------------------------------------------------------------------===//
 // Runtime library flag's attribute that holds information for lowering to LLVM
 //===----------------------------------------------------------------------===//
 
 def FlagsAttr : OpenMP_Attr<"Flags", "flags"> {
   let parameters = (ins
     DefaultValuedParameter<"uint32_t", "0">:$debug_kind,
     DefaultValuedParameter<"bool", "false">:$assume_teams_oversubscription,
     DefaultValuedParameter<"bool", "false">:$assume_threads_oversubscription,
     DefaultValuedParameter<"bool", "false">:$assume_no_thread_state,
-    DefaultValuedParameter<"bool", "false">:$assume_no_nested_parallelism
+    DefaultValuedParameter<"bool", "false">:$assume_no_nested_parallelism,
+    DefaultValuedParameter<"uint32_t", "50">:$openmp_device_version
   );
 
   let assemblyFormat = "`<` struct(params) `>`";
 }
 
 def TargetAttr : OpenMP_Attr<"Target", "target"> {
   let parameters = (ins
     StringRefParameter<>:$target_cpu,

mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td

InterfaceMethod<
       (ins), [{}], [{
         if (Attribute isDevice = $_op->getAttr("omp.is_device"))
           if (::llvm::isa<mlir::omp::IsDeviceAttr>(isDevice))
             return ::llvm::dyn_cast<IsDeviceAttr>(isDevice).getIsDevice();
         return false;
       }]>,
       InterfaceMethod<
       /*description=*/[{
         Get the FlagsAttr attribute on the current module if it exists
         and return the attribute, if it doesn't exit it returns a nullptr
       }],
       /*retTy=*/"mlir::omp::FlagsAttr",
       /*methodName=*/"getFlags",
       (ins), [{}], [{
         if (Attribute flags = $_op->getAttr("omp.flags"))
           return flags.dyn_cast_or_null<mlir::omp::FlagsAttr>();
         return nullptr;
       }]>,
       InterfaceMethod<
       /*description=*/[{
         Apply an omp.FlagsAttr to a module with the specified values
         for the flags
       }],
       /*retTy=*/"void",
       /*methodName=*/"setFlags",
       (ins "uint32_t":$debugKind,
             "bool":$assumeTeamsOversubscription,
             "bool":$assumeThreadsOversubscription,
             "bool":$assumeNoThreadState,
-            "bool":$assumeNoNestedParallelism), [{}], [{
+            "bool":$assumeNoNestedParallelism,
+            "uint32_t":$openmpDeviceVersion), [{}], [{
         $_op->setAttr(("omp." + mlir::omp::FlagsAttr::getMnemonic()).str(),
                   mlir::omp::FlagsAttr::get($_op->getContext(), debugKind,
                       assumeTeamsOversubscription, assumeThreadsOversubscription,
-                      assumeNoThreadState, assumeNoNestedParallelism));
+                      assumeNoThreadState, assumeNoNestedParallelism, openmpDeviceVersion));
       }]>,
     InterfaceMethod<
       /*description=*/[{
         Get the Target attribute on the current module if it exists
         and return the attribute, if it doesn't exist it returns a nullptr.
       }],
       /*retTy=*/"mlir::omp::TargetAttr",
       /*methodName=*/"getTarget",

mlir/test/Dialect/OpenMP/attr.mlir

 // RUN: mlir-opt %s | mlir-opt | FileCheck %s
 
 // CHECK: module attributes {omp.flags = #omp.flags<>} {
-module attributes {omp.flags = #omp.flags<debug_kind = 0, assume_teams_oversubscription = false, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false>} {}
+module attributes {omp.flags = #omp.flags<debug_kind = 0, assume_teams_oversubscription = false, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false, openmp_device_version = 50>} {}
 
 // CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 20>} {
 module attributes {omp.flags = #omp.flags<debug_kind = 20, assume_teams_oversubscription = false, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false>} {}
 
 // CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 100, assume_teams_oversubscription = true>} {
 module attributes {omp.flags = #omp.flags<debug_kind = 100, assume_teams_oversubscription = true, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false>} {}
 
 // CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 200, assume_teams_oversubscription = true, assume_threads_oversubscription = true>} {

 // CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true>} {
 module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true>} {}
 
 // CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true>} {
 module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true>} {}
 
 // CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true>} {
 module attributes {omp.flags = #omp.flags<assume_no_thread_state = true, assume_teams_oversubscription = true>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 20, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<debug_kind = 20, assume_teams_oversubscription = false, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false, openmp_device_version = 51>} {}
+
+//: module attributes {omp.flags = #omp.flags<debug_kind = 100, assume_teams_oversubscription = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<debug_kind = 100, assume_teams_oversubscription = true, assume_threads_oversubscription = false, assume_no_thread_state = false, assume_no_nested_parallelism = false, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 200, assume_teams_oversubscription = true, assume_threads_oversubscription = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<debug_kind = 200, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = false, assume_no_nested_parallelism = false, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 300, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<debug_kind = 300, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = true, assume_no_nested_parallelism = false, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<debug_kind = 400, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = true, assume_no_nested_parallelism = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<debug_kind = 400, assume_teams_oversubscription = true, assume_threads_oversubscription = true, assume_no_thread_state = true, assume_no_nested_parallelism = true, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true, openmp_device_version = 51>} {
+module attributes {omp.flags = #omp.flags<assume_no_thread_state = true, assume_teams_oversubscription = true, openmp_device_version = 51>} {}
+
+// CHECK: module attributes {omp.version = #omp.version<version = 51>} {
+module attributes {omp.version = #omp.version<version = 51>} {}