[RFC] Flang - generation of LLVM IR function attributes

[domada]

Summary

This patch proposes the way how LLVM IR function attributes should be generated for Fortran code.

1. Expected output:

Flang should be able to attach LLVM IR function attributes as Clang does:

; Clang *.ll file:
define dso_local i32 @foo() #0 {
; some code
}
attributes #0 = { list of attributes for example: nounwind, target-cpu="cpu_name", etc}

2. Passing MLIR attributes to LLVM IR for Fortran functions:

Flang can attach LLVM IR function attributes to passthrough MLIR attribute. passthrough attribute is lowered to LLVM IR by MLIR (see patch: https://reviews.llvm.org/D77072 ). The attributes which are dependent on the frontend options (like target-cpu, target-features etc.) should be passed directly to lower functions as it is done in this patch. Attributes which are dependent on the content of Fortran function (for example: alwaysinline) should be added to MLIR code during lowering from PFT to MLIR phase.

3. Handling LLVM IR attributes for OpenMP related LLVM IR

OpenMP MLIR operations are lowered to LLVM IR by OpenMPIRBuilder. That's why OpenMPIRbuilder should be responsible for attaching LLVM IR attributes. The attributes which are dependent on frontend options (like target-cpu) should be passed to MLIR code as it is done here: https://reviews.llvm.org/D146612. Proposition of lowering these attributes is put here: https://reviews.llvm.org/D147313 and it is dependent on the result of discussion for patch: https://reviews.llvm.org/D147344 . OpenMP functions attributes which are not dependent on the frontend options should be added during LLVM IR generation like it is done for nounwind attribute: https://reviews.llvm.org/D147321

[schweitz]

flang/test/Lower/passthrough_attributes.f90

14

Is there a reason each distinct function in the compilation unit has its own target attribute? (The target attribute is attached to the module, no?)

[vzakhari]

+@ftynse and @mehdi_amini
I believe Alex or Mehdi said before that the passthrough support was just a temporary solution and we must not rely on it in more places.

[domada]

flang/test/Lower/passthrough_attributes.f90

14

Clang allows to generate different target for every function via __attribute macro:

__attribute__((target("arch=atom")))
void foo() {} // will be called on 'atom' processors.
__attribute__((target("default")))
void foo() {} // will b

That's why Clang generates different target info for every function. I think we should follow this way for Fortran.

[clementval]

flang/test/Lower/passthrough_attributes.f90

14

I might be wrong but the __attribute__ macro is not something we have support in the frontend/fortran. What's the plan here?

[domada]

flang/test/Lower/passthrough_attributes.f90

14

Currently I am not planning to add support for __attribute__ macro in frontend.

IMO we should enable support for __attribute___ in MLIR code even if we currently have no support for __attribute__ macro in the frontend.

[tschuett]

They are attributes. They are not macros. Clang Sema analyses them and reports misbehaviour. IDK, whether Fortran supports attributes, but you can mark functions as hot and cold or assign them different cpus, which can be useful.

GFortran seems to support attributes.

[ftynse]

+@ftynse and @mehdi_amini
I believe Alex or Mehdi said before that the passthrough support was just a temporary solution and we must not rely on it in more places.

Indeed. "Passthrough" must not be used unless for quick prototyping to check which attributes are needed before modeling them as properly defined first-class attributes in MLIR. There is no verification, it has strings everywhere. I'll update the documentation to better reflect this.

[domada]

In D149450#4326431, @ftynse wrote:

+@ftynse and @mehdi_amini
I believe Alex or Mehdi said before that the passthrough support was just a temporary solution and we must not rely on it in more places.

Indeed. "Passthrough" must not be used unless for quick prototyping to check which attributes are needed before modeling them as properly defined first-class attributes in MLIR. There is no verification, it has strings everywhere. I'll update the documentation to better reflect this.

Hi Alex,
Thank you for your feedback. I would like to ask if there are any plans to add LLVM-IR attributes (noinline, nounwind, etc.) to LLVM dialect?

[ftynse]

Thank you for your feedback. I would like to ask if there are any plans to add LLVM-IR attributes (noinline, nounwind, etc.) to LLVM dialect?

I don't have such plans. But I am happy to review design / patches if somebody has got time to implement these.

flang/include/flang/Lower/Bridge.h

 struct Program;
 } // namespace parser
 namespace semantics {
 class SemanticsContext;
 } // namespace semantics
 
 namespace lower {
 
+struct Attribute {
+  std::string attribute;
+  std::optional<std::string> attributeVal;
+  Attribute(llvm::StringRef attribute)
+      : attribute{attribute.str()}, attributeVal{} {};
+  Attribute(llvm::StringRef attribute, llvm::StringRef attributeVal)
+      : attribute{attribute.str()}, attributeVal{attributeVal.str()} {};
+};
+
 //===----------------------------------------------------------------------===//
 // Lowering bridge
 //===----------------------------------------------------------------------===//
 
 /// The lowering bridge converts the front-end parse trees and semantics
 /// checking residual to MLIR (FIR dialect) code.
 class LoweringBridge {
 public:
   /// Create a lowering bridge instance.
   static LoweringBridge
   create(mlir::MLIRContext &ctx,
          Fortran::semantics::SemanticsContext &semanticsContext,
          const Fortran::common::IntrinsicTypeDefaultKinds &defaultKinds,
          const Fortran::evaluate::IntrinsicProcTable &intrinsics,
          const Fortran::evaluate::TargetCharacteristics &targetCharacteristics,
          const Fortran::parser::AllCookedSources &allCooked,
          llvm::StringRef triple, fir::KindMapping &kindMap,
+         const llvm::SmallVector<Attribute> &llvmFuncAttributes,
          const Fortran::lower::LoweringOptions &loweringOptions,
          const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults) {
     return LoweringBridge(ctx, semanticsContext, defaultKinds, intrinsics,
                           targetCharacteristics, allCooked, triple, kindMap,
-                          loweringOptions, envDefaults);
+                          llvmFuncAttributes, loweringOptions, envDefaults);
   }
 
   //===--------------------------------------------------------------------===//
   // Getters
   //===--------------------------------------------------------------------===//
 
   mlir::MLIRContext &getMLIRContext() { return context; }
 
private:
   explicit LoweringBridge(
       mlir::MLIRContext &ctx,
       Fortran::semantics::SemanticsContext &semanticsContext,
       const Fortran::common::IntrinsicTypeDefaultKinds &defaultKinds,
       const Fortran::evaluate::IntrinsicProcTable &intrinsics,
       const Fortran::evaluate::TargetCharacteristics &targetCharacteristics,
       const Fortran::parser::AllCookedSources &cooked, llvm::StringRef triple,
       fir::KindMapping &kindMap,
+      const llvm::SmallVector<Attribute> &funcAttributes,
       const Fortran::lower::LoweringOptions &loweringOptions,
       const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults);
   LoweringBridge() = delete;
   LoweringBridge(const LoweringBridge &) = delete;
 
   Fortran::semantics::SemanticsContext &semanticsContext;
   Fortran::lower::StatementContext functionContext;
   const Fortran::common::IntrinsicTypeDefaultKinds &defaultKinds;
   const Fortran::evaluate::IntrinsicProcTable &intrinsics;
   const Fortran::evaluate::TargetCharacteristics &targetCharacteristics;
   const Fortran::parser::AllCookedSources *cooked;
   mlir::MLIRContext &context;
   std::unique_ptr<mlir::ModuleOp> module;
   fir::KindMapping &kindMap;
+  const llvm::SmallVector<Attribute> &funcAttributes;
   const Fortran::lower::LoweringOptions &loweringOptions;
   const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults;
 };
 
 } // namespace lower
 } // namespace Fortran
 
 #endif // FORTRAN_LOWER_BRIDGE_H

flang/lib/Frontend/FrontendActions.cpp

if (getCurrentInput().getKind().getLanguage() != Language::Fortran) {
     ci.getDiagnostics().Report(diagID);
     return false;
   }
   bool res = runPrescan() && runParse() && runSemanticChecks() &&
              generateRtTypeTables();
   if (!res)
     return res;
 
+  llvm::SmallVector<lower::Attribute> loweringAttributes;
+  if (!setUpTargetMachine())
+    return false;
+
+  if (!tm->getTargetCPU().empty()) {
+    lower::Attribute attr("target-cpu", tm->getTargetCPU());
+    loweringAttributes.push_back(attr);
+  }
+  if (!tm->getTargetFeatureString().empty()) {
+    lower::Attribute attr("target-features", tm->getTargetFeatureString());
+    loweringAttributes.push_back(attr);
+  }
   // Create a LoweringBridge
   const common::IntrinsicTypeDefaultKinds &defKinds =
       ci.getInvocation().getSemanticsContext().defaultKinds();
   fir::KindMapping kindMap(mlirCtx.get(), llvm::ArrayRef<fir::KindTy>{
                                               fir::fromDefaultKinds(defKinds)});
   lower::LoweringBridge lb = Fortran::lower::LoweringBridge::create(
       *mlirCtx, ci.getInvocation().getSemanticsContext(), defKinds,
       ci.getInvocation().getSemanticsContext().intrinsics(),
       ci.getInvocation().getSemanticsContext().targetCharacteristics(),
       ci.getParsing().allCooked(), ci.getInvocation().getTargetOpts().triple,
-      kindMap, ci.getInvocation().getLoweringOpts(),
+      kindMap, loweringAttributes, ci.getInvocation().getLoweringOpts(),
       ci.getInvocation().getFrontendOpts().envDefaults);
 
   // Fetch module from lb, so we can set
   mlirModule = std::make_unique<mlir::ModuleOp>(lb.getModule());
 
-  if (!setUpTargetMachine())
-    return false;
 
   if (ci.getInvocation().getFrontendOpts().features.IsEnabled(
           Fortran::common::LanguageFeature::OpenMP)) {
     setOffloadModuleInterfaceAttributes(*mlirModule,
                                         ci.getInvocation().getLangOpts());
     setOffloadModuleInterfaceTargetAttribute(*mlirModule, tm->getTargetCPU(),
                                              tm->getTargetFeatureString());
   }

flang/lib/Lower/Bridge.cpp

 class FirConverter : public Fortran::lower::AbstractConverter {
 public:
   explicit FirConverter(Fortran::lower::LoweringBridge &bridge)
       : Fortran::lower::AbstractConverter(bridge.getLoweringOptions()),
         bridge{bridge}, foldingContext{bridge.createFoldingContext()} {}
   virtual ~FirConverter() = default;
 
   /// Convert the PFT to FIR.
-  void run(Fortran::lower::pft::Program &pft) {
+  void run(Fortran::lower::pft::Program &pft,
+           const llvm::SmallVector<Fortran::lower::Attribute> &funcAttributes) {
     // Preliminary translation pass.
 
     // - Lower common blocks from the PFT common block list that contains a
     // consolidated list of the common blocks (with the initialization if any in
     // the Program, and with the common block biggest size in all its
     // appearance). This is done before lowering any scope declarations because
     // it is not know at the local scope level what MLIR type common blocks
     // should have to suit all its usage in the compilation unit.
for (Fortran::lower::pft::Program::Units &u : pft.getUnits()) {
                      [&](Fortran::lower::pft::BlockDataUnit &b) {},
                      [&](Fortran::lower::pft::CompilerDirectiveUnit &d) {},
                  },
                  u);
     }
 
     // Primary translation pass.
     for (Fortran::lower::pft::Program::Units &u : pft.getUnits()) {
-      std::visit(
-          Fortran::common::visitors{
-              [&](Fortran::lower::pft::FunctionLikeUnit &f) { lowerFunc(f); },
-              [&](Fortran::lower::pft::ModuleLikeUnit &m) { lowerMod(m); },
+      std::visit(Fortran::common::visitors{
+                     [&](Fortran::lower::pft::FunctionLikeUnit &f) {
+                       lowerFunc(f, funcAttributes);
+                     },
+                     [&](Fortran::lower::pft::ModuleLikeUnit &m) {
+                       lowerMod(m, funcAttributes);
+                     },
                      [&](Fortran::lower::pft::BlockDataUnit &b) {},
                      [&](Fortran::lower::pft::CompilerDirectiveUnit &d) {},
                  },
                  u);
     }
 
     /// Once all the code has been translated, create runtime type info
     /// global data structure for the derived types that have been
     /// processed.
     createGlobalOutsideOfFunctionLowering(
         [&]() { runtimeTypeInfoConverter.createTypeInfoGlobals(*this); });
 
void instantiateVar(const Fortran::lower::pft::Variable &var,
     Fortran::lower::instantiateVariable(*this, var, localSymbols, storeMap);
     if (var.hasSymbol() &&
         var.getSymbol().test(
             Fortran::semantics::Symbol::Flag::OmpThreadprivate))
       Fortran::lower::genThreadprivateOp(*this, var);
   }
 
   /// Start translation of a function.
-  void startNewFunction(Fortran::lower::pft::FunctionLikeUnit &funit) {
+  void startNewFunction(
+      Fortran::lower::pft::FunctionLikeUnit &funit,
+      const llvm::SmallVector<Fortran::lower::Attribute> &funcAttributes) {
     assert(!builder && "expected nullptr");
     bridge.fctCtx().pushScope();
     const Fortran::semantics::Scope &scope = funit.getScope();
     LLVM_DEBUG(llvm::dbgs() << "\n[bridge - startNewFunction]";
                if (auto *sym = scope.symbol()) llvm::dbgs() << " " << *sym;
                llvm::dbgs() << "\n");
     Fortran::lower::CalleeInterface callee(funit, *this);
     mlir::func::FuncOp func = callee.addEntryBlockAndMapArguments();
     builder = new fir::FirOpBuilder(func, bridge.getKindMap());
     assert(builder && "FirOpBuilder did not instantiate");
     builder->setFastMathFlags(bridge.getLoweringOptions().getMathOptions());
     builder->setInsertionPointToStart(&func.front());
     func.setVisibility(mlir::SymbolTable::Visibility::Public);
     assert(blockId == 0 && "invalid blockId");
     assert(activeConstructStack.empty() && "invalid construct stack state");
 
     mapDummiesAndResults(funit, callee);
+    mlir::Attribute passthroughAttr = func->getAttr("passthrough");
+    if (!passthroughAttr) {
+      llvm::SmallVector<mlir::Attribute> passthroughAttrsVec;
+      for (const auto &attr : funcAttributes) {
+        if (attr.attributeVal) {
+          passthroughAttrsVec.push_back(
+              builder->getStrArrayAttr({attr.attribute, *attr.attributeVal}));
+        } else {
+          passthroughAttrsVec.push_back(builder->getStringAttr(attr.attribute));
+        }
+      }
+      if (!passthroughAttrsVec.empty())
+        func->setAttr(builder->getStringAttr("passthrough"),
+                      builder->getArrayAttr(passthroughAttrsVec));
+    }
 
     // Map host associated symbols from parent procedure if any.
     if (funit.parentHasHostAssoc())
       funit.parentHostAssoc().internalProcedureBindings(*this, localSymbols);
 
     // Non-primary results of a function with multiple entry points.
     // These result values share storage with the primary result.
     llvm::SmallVector<Fortran::lower::pft::Variable> deferredFuncResultList;

   /// Create fir::Global for all the common blocks that appear in the program.
   void
   lowerCommonBlocks(const Fortran::semantics::CommonBlockList &commonBlocks) {
     createGlobalOutsideOfFunctionLowering(
         [&]() { Fortran::lower::defineCommonBlocks(*this, commonBlocks); });
   }
 
   /// Lower a procedure (nest).
-  void lowerFunc(Fortran::lower::pft::FunctionLikeUnit &funit) {
+  void lowerFunc(
+      Fortran::lower::pft::FunctionLikeUnit &funit,
+      const llvm::SmallVector<Fortran::lower::Attribute> &funcAttributes) {
     setCurrentPosition(funit.getStartingSourceLoc());
     for (int entryIndex = 0, last = funit.entryPointList.size();
          entryIndex < last; ++entryIndex) {
       funit.setActiveEntry(entryIndex);
-      startNewFunction(funit); // the entry point for lowering this procedure
+      startNewFunction(
+          funit, funcAttributes); // the entry point for lowering this procedure
       for (Fortran::lower::pft::Evaluation &eval : funit.evaluationList)
         genFIR(eval);
       endNewFunction(funit);
     }
     funit.setActiveEntry(0);
     for (Fortran::lower::pft::FunctionLikeUnit &f : funit.nestedFunctions)
-      lowerFunc(f); // internal procedure
+      lowerFunc(f, funcAttributes); // internal procedure
   }
 
   /// Lower module variable definitions to fir::globalOp and OpenMP/OpenACC
   /// declarative construct.
   void lowerModuleDeclScope(Fortran::lower::pft::ModuleLikeUnit &mod) {
     setCurrentPosition(mod.getStartingSourceLoc());
     createGlobalOutsideOfFunctionLowering([&]() {
       auto &scopeVariableListMap =
           Fortran::lower::pft::getScopeVariableListMap(mod);
       for (const auto &var : Fortran::lower::pft::getScopeVariableList(
                mod.getScope(), scopeVariableListMap)) {
         // Only define the variables owned by this module.
         const Fortran::semantics::Scope *owningScope = var.getOwningScope();
         if (!owningScope || mod.getScope() == *owningScope)
           Fortran::lower::defineModuleVariable(*this, var);
       }
       for (auto &eval : mod.evaluationList)
         genFIR(eval);
     });
   }
 
   /// Lower functions contained in a module.
-  void lowerMod(Fortran::lower::pft::ModuleLikeUnit &mod) {
+  void
+  lowerMod(Fortran::lower::pft::ModuleLikeUnit &mod,
+           const llvm::SmallVector<Fortran::lower::Attribute> &funcAttributes) {
     for (Fortran::lower::pft::FunctionLikeUnit &f : mod.nestedFunctions)
-      lowerFunc(f);
+      lowerFunc(f, funcAttributes);
   }
 
   void setCurrentPosition(const Fortran::parser::CharBlock &position) {
     if (position != Fortran::parser::CharBlock{})
       currentPosition = position;
   }
 
   /// Set current position at the location of \p parseTreeNode. Note that the

 void Fortran::lower::LoweringBridge::lower(
     const Fortran::parser::Program &prg,
     const Fortran::semantics::SemanticsContext &semanticsContext) {
   std::unique_ptr<Fortran::lower::pft::Program> pft =
       Fortran::lower::createPFT(prg, semanticsContext);
   if (dumpBeforeFir)
     Fortran::lower::dumpPFT(llvm::errs(), *pft);
   FirConverter converter{*this};
-  converter.run(*pft);
+  converter.run(*pft, funcAttributes);
 }
 
 void Fortran::lower::LoweringBridge::parseSourceFile(llvm::SourceMgr &srcMgr) {
   mlir::OwningOpRef<mlir::ModuleOp> owningRef =
       mlir::parseSourceFile<mlir::ModuleOp>(srcMgr, &context);
   module.reset(new mlir::ModuleOp(owningRef.get().getOperation()));
   owningRef.release();
 }
 
 Fortran::lower::LoweringBridge::LoweringBridge(
     mlir::MLIRContext &context,
     Fortran::semantics::SemanticsContext &semanticsContext,
     const Fortran::common::IntrinsicTypeDefaultKinds &defaultKinds,
     const Fortran::evaluate::IntrinsicProcTable &intrinsics,
     const Fortran::evaluate::TargetCharacteristics &targetCharacteristics,
     const Fortran::parser::AllCookedSources &cooked, llvm::StringRef triple,
     fir::KindMapping &kindMap,
+    const llvm::SmallVector<Attribute> &funcAttributes,
     const Fortran::lower::LoweringOptions &loweringOptions,
     const std::vector<Fortran::lower::EnvironmentDefault> &envDefaults)
     : semanticsContext{semanticsContext}, defaultKinds{defaultKinds},
       intrinsics{intrinsics}, targetCharacteristics{targetCharacteristics},
       cooked{&cooked}, context{context}, kindMap{kindMap},
-      loweringOptions{loweringOptions}, envDefaults{envDefaults} {
+      funcAttributes{funcAttributes}, loweringOptions{loweringOptions},
+      envDefaults{envDefaults} {
   // Register the diagnostic handler.
   context.getDiagEngine().registerHandler([](mlir::Diagnostic &diag) {
     llvm::raw_ostream &os = llvm::errs();
     switch (diag.getSeverity()) {
     case mlir::DiagnosticSeverity::Error:
       os << "error: ";
       break;
     case mlir::DiagnosticSeverity::Remark:

flang/test/Driver/save-mlir-temps.f90

 ! RUN: %flang -c -fno-integrated-as -save-temps=%t -o %t/out.o %s 2>&1
 ! RUN: FileCheck %s -input-file=%t/save-mlir-temps-fir.mlir -check-prefix=MLIR-FIR
 ! RUN: FileCheck %s -input-file=%t/save-mlir-temps-llvmir.mlir -check-prefix=MLIR-LLVMIR
 
 !--------------------------
 ! Content to check from the MLIR outputs
 !--------------------------
 ! MLIR-FIR-NOT: llvm.func
-! MLIR-FIR: func.func @{{.*}}main() {
+! MLIR-FIR: func.func @{{.*}}main()
 
 ! MLIR-FIR-NOT: func.func
-! MLIR-LLVMIR: llvm.func @{{.*}}main() {
+! MLIR-LLVMIR: llvm.func @{{.*}}main()
 
 end program

flang/test/Lower/OpenMP/target_cpu_features.f90

 !REQUIRES: amdgpu-registered-target
 !RUN: %flang_fc1 -emit-fir -triple amdgcn-amd-amdhsa -target-cpu gfx908 -fopenmp %s -o - | FileCheck %s
 
 !===============================================================================
 ! Target_Enter Simple
 !===============================================================================
 
 !CHECK: omp.target = #omp.target<target_cpu = "gfx908",
 !CHECK-SAME: target_features = "+dot3-insts,+dot4-insts,+s-memtime-inst,
 !CHECK-SAME: +16-bit-insts,+s-memrealtime,+dot6-insts,+dl-insts,+wavefrontsize64,
 !CHECK-SAME: +gfx9-insts,+gfx8-insts,+ci-insts,+dot10-insts,+dot7-insts,
 !CHECK-SAME: +dot1-insts,+dot5-insts,+mai-insts,+dpp,+dot2-insts">
-!CHECK-LABEL: func.func @_QPomp_target_simple() {
+!CHECK-LABEL: func.func @_QPomp_target_simple()
 subroutine omp_target_simple
 end subroutine omp_target_simple
 

flang/test/Lower/passthrough_attributes.f90

+! REQUIRES: x86-registered-target
+
+!RUN: %flang_fc1 -emit-fir -triple x86_64-unknown-linux-gnu  %s -o - | FileCheck %s -check-prefix=CHECK-X86
+
+!RUN: %flang_fc1 -emit-fir -triple x86_64-unknown-linux-gnu -target-cpu znver4  %s -o - | FileCheck %s -check-prefix=CHECK-ZEN4
+
+!RUN: %flang_fc1 -emit-fir -triple x86_64-unknown-linux-gnu -target-cpu znver4 -target-feature +mmx  %s -o - | FileCheck %s -check-prefix=CHECK-ZEN4-MMX
+
+subroutine test_passthrough
+end subroutine
+
+!CHECK-X86:  func.func @_QPtest_passthrough() {
+!CHECK-ZEN4:   func.func @_QPtest_passthrough() attributes {passthrough =
+!CHECK-ZEN4-SAME: target-cpu
+!CHECK-ZEN4-SAME: znver4
+!CHECK-ZEN4-NOT: mmx
+!CHECK-ZEN4-MMX:   func.func @_QPtest_passthrough() attributes {passthrough =
+!CHECK-ZEN4-MMX-SAME: target-cpu
+!CHECK-ZEN4-MMX-SAME: znver4
+!CHECK-ZEN4-MMX-SAME: target-features
+!CHECK-ZEN4-MMX-SAME: mmx
+

flang/tools/bbc/bbc.cpp

fir::KindMapping kindMap(
       &ctx, llvm::ArrayRef<fir::KindTy>{fir::fromDefaultKinds(defKinds)});
   // Use default lowering options for bbc.
   Fortran::lower::LoweringOptions loweringOptions{};
   loweringOptions.setPolymorphicTypeImpl(enablePolymorphic);
   loweringOptions.setLowerToHighLevelFIR(useHLFIR);
   auto burnside = Fortran::lower::LoweringBridge::create(
       ctx, semanticsContext, defKinds, semanticsContext.intrinsics(),
       semanticsContext.targetCharacteristics(), parsing.allCooked(), "",
-      kindMap, loweringOptions, {});
+      kindMap, {}, loweringOptions, {});
   burnside.lower(parseTree, semanticsContext);
   mlir::ModuleOp mlirModule = burnside.getModule();
   if (enableOpenMP) {
     auto offloadModuleOpts =
         OffloadModuleOpts(setOpenMPTargetDebug, setOpenMPTeamSubscription,
                           setOpenMPThreadSubscription, setOpenMPNoThreadState,
                           setOpenMPNoNestedParallelism, enableOpenMPDevice);
     setOffloadModuleInterfaceAttributes(mlirModule, offloadModuleOpts);