[mlir][openacc] Add conversion pass OpenACC to LLVM with initial acc.enter_data conversion

[clementval]

Summary

This patch begins to lower acc.enter_data operation to call to tgt runtime call.
It currently only lower create operands of memref type. This acts as a basis to add support
for FIR types in the Flang/OpenACC support. It follows more or less a similar path than clang
with omp target enter data map directives.

OpenACC support in Flang will rely on the current OpenMP runtime where 1:1 lowering can be
applied. Some extension will be added where features are not available yet.

Big part of this code will be shared for other standalone data operations in the OpenACC
dialect such as acc.exit_data and acc.update.

It is likely that parts of the lowering can also be shared later with the ops for
standalone data directives in the OpenMP dialect when they are introduced.

This is an initial lowering and it probably needs more work.

[clementval]

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

96

The global name is obviously just a temporary placeholder. Would it be a good idea to enable unnamed GlobalOp? @ftynse

109

Same here, the name doesn't really matter so unnamed GlobalOp would be nice.

[jdoerfert]

I left some comments to show how this duplicates/exposes things that it should not.

Basically, we fist need the OpenMPIRBuilder entry points for the __tgt_XXX functions and then we can call them during the conversion.

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

52

This should never be needed. ident_t is an implementation artifact of the runtime that should not leak into the user space.

77

Not necessary, see use below.

79–145

161

Again, implementation detail, subject to change.

241

This duplicates what
OpenMPIRBuilder::getOrCreateRuntimeFunctionPtr(OMPRTL___tgt_target_data_begin_mapper);
does but with drawbacks.

[clementval]

In D100678#2698875, @jdoerfert wrote:

I left some comments to show how this duplicates/exposes things that it should not.

Basically, we fist need the OpenMPIRBuilder entry points for the __tgt_XXX functions and then we can call them during the conversion.

Thanks for the pointer in the OpenMPIRBuilder. For sure it makes sense to reuse what is there. I guess it means that instead of a conversion this will has to be moved to a translation.

[clementval]

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

79–145

Fully agree with this if we are in the translation. Currently I don't think there is a way to have the OMPBuilder in the conversion step. I don't say that we should find a way for that but probably the global location creation should be at least deferred until the translation phase in order to make correct use of the OMPBuilder. Just thinking out loud :-)

@mehdi_amini or @ftynse do you have any insight how this could be handled properly?

161

Surely the creation of the global can be done in OMPBuilder but the information in the flags array needs to be collected from the MLIR information.

[kiranchandramohan]

Great to see that we are now making progress in the LLVM codegen for OpenACC. Clarity here will help the OpenMP device side work as well. I have a general question inline.

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

79–145

Yes, I think we cannot use the OpenMP IRBuilder while converting to LLVM Dialect. It has to be during the translation phase.

I believe you were investigating using the OpenMP IRBuilder and considering extending it. Did you face issues there or in interfacing at the translation level with the OpenMP IRBuilder. Or are you pointing out that the conversion to LLVM dialect requires special handling irrespective of whether OpenMP IRBuilder is used or not.

[clementval]

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

79–145

Yeah that's correct. The OpenMP IR Builder works directly on the translated module so there is no way to use it in a conversion pass.

There is no problem using/extending the OpenMP IRBuilder but as you mentioned it has to be used at the translation level. This patch was trying to achieve the lowering with a conversion pass to LLVM dialect. I'm planing to rewrite this lowering at the translation level and so directly targeting LLVM IR. That's the only way to reuse the OpenMP IRBuilder code unless we find another way around.

[clementval]

Patch D101504 is taking the new approach by doing a direct translation to LLVM IR with the use of the OpenMPIRBuilder.

mlir/include/mlir/Conversion/OpenACCToLLVM/ConvertOpenACCToLLVM.h

+//===- ConvertOpenACCToLLVM.h - OpenACC conversion pass entrypoint --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#ifndef MLIR_CONVERSION_OPENACCTOLLVM_CONVERTOPENACCTOLLVMPASS_H_
+#define MLIR_CONVERSION_OPENACCTOLLVM_CONVERTOPENACCTOLLVMPASS_H_
+
+#include <memory>
+
+namespace mlir {
+class LLVMTypeConverter;
+class ModuleOp;
+template <typename T>
+class OperationPass;
+class RewritePatternSet;
+
+/// Collect the patterns to convert from the OpenACC dialect LLVMIR dialect.
+void populateOpenACCToLLVMConversionPatterns(LLVMTypeConverter &converter,
+                                             RewritePatternSet &patterns);
+
+/// Create a pass to convert the OpenACC dialect into the LLVMIR dialect.
+std::unique_ptr<OperationPass<ModuleOp>> createConvertOpenACCToLLVMPass();
+
+} // namespace mlir
+
+#endif // MLIR_CONVERSION_OPENACCTOLLVM_CONVERTOPENACCTOLLVMPASS_H_

mlir/include/mlir/Conversion/Passes.h

 #include "mlir/Conversion/GPUCommon/GPUCommonPass.h"
 #include "mlir/Conversion/GPUToNVVM/GPUToNVVMPass.h"
 #include "mlir/Conversion/GPUToROCDL/GPUToROCDLPass.h"
 #include "mlir/Conversion/GPUToSPIRV/GPUToSPIRVPass.h"
 #include "mlir/Conversion/GPUToVulkan/ConvertGPUToVulkanPass.h"
 #include "mlir/Conversion/LinalgToLLVM/LinalgToLLVM.h"
 #include "mlir/Conversion/LinalgToSPIRV/LinalgToSPIRVPass.h"
 #include "mlir/Conversion/LinalgToStandard/LinalgToStandard.h"
+#include "mlir/Conversion/OpenACCToLLVM/ConvertOpenACCToLLVM.h"
 #include "mlir/Conversion/OpenMPToLLVM/ConvertOpenMPToLLVM.h"
 #include "mlir/Conversion/PDLToPDLInterp/PDLToPDLInterp.h"
 #include "mlir/Conversion/SCFToGPU/SCFToGPUPass.h"
 #include "mlir/Conversion/SCFToOpenMP/SCFToOpenMP.h"
 #include "mlir/Conversion/SCFToSPIRV/SCFToSPIRVPass.h"
 #include "mlir/Conversion/SCFToStandard/SCFToStandard.h"
 #include "mlir/Conversion/SPIRVToLLVM/SPIRVToLLVMPass.h"
 #include "mlir/Conversion/ShapeToStandard/ShapeToStandard.h"

mlir/include/mlir/Conversion/Passes.td

let description = [{
     This pass converts supported Linalg ops to SPIR-V ops. It's quite
     experimental and are expected to migrate to other proper conversions.
   }];
   let constructor = "mlir::createLinalgToSPIRVPass()";
   let dependentDialects = ["spirv::SPIRVDialect"];
 }
 
 //===----------------------------------------------------------------------===//
+// OpenACCToLLVM
+//===----------------------------------------------------------------------===//
+
+def ConvertOpenACCToLLVM : Pass<"convert-openacc-to-llvm", "ModuleOp"> {
+  let summary = "Convert the OpenACC ops to LLVM dialect";
+  let constructor = "mlir::createConvertOpenACCToLLVMPass()";
+  let dependentDialects = ["LLVM::LLVMDialect"];
+}
+
+//===----------------------------------------------------------------------===//
 // OpenMPToLLVM
 //===----------------------------------------------------------------------===//
 
 def ConvertOpenMPToLLVM : Pass<"convert-openmp-to-llvm", "ModuleOp"> {
   let summary = "Convert the OpenMP ops to OpenMP ops with LLVM dialect";
   let constructor = "mlir::createConvertOpenMPToLLVMPass()";
   let dependentDialects = ["LLVM::LLVMDialect"];
 }

mlir/lib/Conversion/CMakeLists.txt

 add_subdirectory(AffineToStandard)
 add_subdirectory(AsyncToLLVM)
 add_subdirectory(ComplexToLLVM)
 add_subdirectory(GPUCommon)
 add_subdirectory(GPUToNVVM)
 add_subdirectory(GPUToROCDL)
 add_subdirectory(GPUToSPIRV)
 add_subdirectory(GPUToVulkan)
 add_subdirectory(LinalgToLLVM)
 add_subdirectory(LinalgToSPIRV)
 add_subdirectory(LinalgToStandard)
+add_subdirectory(OpenACCToLLVM)
 add_subdirectory(OpenMPToLLVM)
 add_subdirectory(PDLToPDLInterp)
 add_subdirectory(SCFToGPU)
 add_subdirectory(SCFToOpenMP)
 add_subdirectory(SCFToSPIRV)
 add_subdirectory(SCFToStandard)
 add_subdirectory(ShapeToStandard)
 add_subdirectory(SPIRVToLLVM)

mlir/lib/Conversion/OpenACCToLLVM/CMakeLists.txt

+add_mlir_conversion_library(MLIROpenACCToLLVM
+  OpenACCToLLVM.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Conversion/OpenACCToLLVM
+
+  DEPENDS
+  MLIRConversionPassIncGen
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  Core
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  MLIRLLVMIR
+  MLIROpenACC
+  MLIRStandardToLLVM
+  MLIRTransforms
+  )

mlir/lib/Conversion/OpenACCToLLVM/OpenACCToLLVM.cpp

+//===- OpenACCToLLVM.cpp - conversion from OpenACC to LLVM dialect --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "../PassDetail.h"
+#include "mlir/Conversion/OpenACCToLLVM/ConvertOpenACCToLLVM.h"
+#include "mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+
+#include "llvm/IR/GlobalValue.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace mlir::acc;
+
+//===----------------------------------------------------------------------===//
+// Utility functions
+//===----------------------------------------------------------------------===//
+
+static constexpr StringRef kmpIdentTypeName = "struct.ident_t";
+static constexpr StringRef globalOffloadMaptypesName = ".offload_maptypes";
+static constexpr StringRef globalOffloadMapnamesName = ".offload_mapnames";
+static constexpr StringRef tgtTargetDataBeginMapperName =
+    "__tgt_target_data_begin_mapper";
+static constexpr int64_t defaultDeviceId = -1;
+
+/// 0 = alloc
+static constexpr int64_t createFlag = 0;
+
+static LLVM::LLVMStructType getKmpIdentType(MLIRContext *ctx) {
+  // Struct defined in openmp/libomptarget/include/SourceInfo.h
+  // Create a struct type corresponding to the C struct below.
+  // struct ident_t {
+  //   // Ident_t flags described in kmp.h.
+  //   int32_t reserved_1;
+  //   int32_t flags;
+  //   int32_t reserved_2;
+  //   int32_t reserved_3;
+  //   char const *psource;
+  // };
+  auto i32Ty = IntegerType::get(ctx, 32);
+  auto i8Ty = IntegerType::get(ctx, 8);
+  auto i8Ptr = LLVM::LLVMPointerType::get(i8Ty);
+  auto type = LLVM::LLVMStructType::getIdentified(ctx, kmpIdentTypeName);
+  type.setBody({i32Ty, i32Ty, i32Ty, i32Ty, i8Ptr}, /*isPacked=*/false);
+  return type;
+}
+
+/// Function defined in openmp/libomptarget/include/omptarget.h
+/// void __tgt_target_data_begin_mapper(ident_t *loc, int64_t device_id,
+///                                     int32_t arg_num, void **args_base,
+///                                     void **args, int64_t *arg_sizes,
+///                                     int64_t *arg_types,
+///                                     map_var_info_t *arg_names,
+///                                     void **arg_mappers);
+static LLVM::LLVMFunctionType
+getTgtTargetDataBeginMapperType(MLIRContext *ctx) {
+  auto llvmVoidType = LLVM::LLVMVoidType::get(ctx);
+  auto identType = getKmpIdentType(ctx);
+  auto identTypePtr = LLVM::LLVMPointerType::get(identType);
+
+  auto i32Ty = IntegerType::get(ctx, 32); // i32
+  auto i8Ty = IntegerType::get(ctx, 8);
+  auto i8PtrTy = LLVM::LLVMPointerType::get(i8Ty);
+  auto i8PtrPtrTy = LLVM::LLVMPointerType::get(i8PtrTy); // i8**
+  auto i64Ty = IntegerType::get(ctx, 64);
+  auto i64PtrTy = LLVM::LLVMPointerType::get(i64Ty); // i64*
+
+  return LLVM::LLVMFunctionType::get(
+      llvmVoidType, {identTypePtr, i64Ty, i32Ty, i8PtrPtrTy, i8PtrPtrTy,
+                     i64PtrTy, i64PtrTy, i8PtrPtrTy, i8PtrPtrTy});
+}
+
+static LLVM::GlobalOp createLocationInfoGlobalOp(Location loc, EnterDataOp &op,
+                                                 OpBuilder &builder) {
+  MLIRContext *ctx = builder.getContext();
+  std::string locStr;
+  llvm::raw_string_ostream locOS(locStr);
+  if (auto fileLoc = loc.dyn_cast<FileLineColLoc>()) {
+    auto funcOp = op.getOperation()->getParentOfType<FuncOp>();
+    StringRef funcName = funcOp ? funcOp.getName() : "unknown";
+    locOS << ";" << fileLoc.getFilename() << ";" << funcName << ";"
+          << fileLoc.getLine() << ";" << fileLoc.getColumn() << ";;";
+  } else {
+    locOS << ";unknown;unknown;0;0;;";
+  }
+  auto globalStringType =
+      LLVM::LLVMArrayType::get(IntegerType::get(ctx, 8), locOS.str().size());
+  auto globalStr = builder.create<LLVM::GlobalOp>(
+      loc, globalStringType,
+      /*isConstant=*/true, LLVM::Linkage::Private, "0",
+      builder.getStringAttr(locOS.str()));
+  globalStr.unnamed_addrAttr(
+      LLVM::UnnamedAddrAttr::get(ctx, LLVM::UnnamedAddr::Global));
+
+  auto structType = getKmpIdentType(ctx);
+
+  // Create a ident_t struct
+  auto llvmI32Type = IntegerType::get(ctx, 32);
+  auto i8Ty = IntegerType::get(ctx, 8);
+  auto i8PtrTy = LLVM::LLVMPointerType::get(i8Ty);
+  auto global = builder.create<LLVM::GlobalOp>(
+      loc, structType,
+      /*isConstant=*/true, LLVM::Linkage::Private, "1", Attribute());
+  global.unnamed_addrAttr(
+      LLVM::UnnamedAddrAttr::get(ctx, LLVM::UnnamedAddr::Global));
+
+  Location globalLoc = global.getLoc();
+  Region &region = global.getInitializerRegion();
+  Block *block = builder.createBlock(&region);
+  builder.setInsertionPoint(block, block->begin());
+
+  Value structValue = builder.create<LLVM::UndefOp>(globalLoc, structType);
+  Value zero = builder.create<LLVM::ConstantOp>(globalLoc, llvmI32Type,
+                                                builder.getI64IntegerAttr(0));
+  Value two = builder.create<LLVM::ConstantOp>(globalLoc, llvmI32Type,
+                                               builder.getI64IntegerAttr(2));
+  structValue = builder.create<LLVM::InsertValueOp>(
+      globalLoc, structType, structValue, zero,
+      ArrayAttr::get(ctx, {builder.getIntegerAttr(builder.getI32Type(), 0)}));
+  structValue = builder.create<LLVM::InsertValueOp>(
+      globalLoc, structType, structValue, two,
+      ArrayAttr::get(ctx, {builder.getIntegerAttr(builder.getI32Type(), 1)}));
+  structValue = builder.create<LLVM::InsertValueOp>(
+      globalLoc, structType, structValue, zero,
+      ArrayAttr::get(ctx, {builder.getIntegerAttr(builder.getI32Type(), 2)}));
+  structValue = builder.create<LLVM::InsertValueOp>(
+      globalLoc, structType, structValue, zero,
+      ArrayAttr::get(ctx, {builder.getIntegerAttr(builder.getI32Type(), 3)}));
+  Value addrOfGlobalStr =
+      builder.create<LLVM::AddressOfOp>(globalLoc, globalStr);
+  auto globalStrGep = builder.create<LLVM::GEPOp>(
+      globalLoc, i8PtrTy, addrOfGlobalStr, ArrayRef<Value>{zero, zero});
+  structValue = builder.create<LLVM::InsertValueOp>(
+      globalLoc, structType, structValue, globalStrGep,
+      ArrayAttr::get(ctx, {builder.getIntegerAttr(builder.getI32Type(), 4)}));
+  builder.create<LLVM::ReturnOp>(globalLoc, ArrayRef<Value>({structValue}));
+  builder.setInsertionPointAfter(global);
+  return global;
+}
+
+/// Create offload maptypes global constant
+static LLVM::GlobalOp createOffloadMaptypesGlobal(Location loc,
+                                                  OpBuilder &builder,
+                                                  unsigned nbArg,
+                                                  ArrayRef<int64_t> flags) {
+  MLIRContext *ctx = builder.getContext();
+  auto offloadMaptypesType =
+      LLVM::LLVMArrayType::get(IntegerType::get(ctx, 64), nbArg);
+  auto offloadMaptypes = builder.create<LLVM::GlobalOp>(
+      loc, offloadMaptypesType, /*isConstant=*/true, LLVM::Linkage::Private,
+      globalOffloadMaptypesName, builder.getI64VectorAttr(flags));
+  offloadMaptypes.unnamed_addrAttr(
+      LLVM::UnnamedAddrAttr::get(ctx, LLVM::UnnamedAddr::Global));
+  return offloadMaptypes;
+}
+
+static LLVM::GlobalOp
+createOffloadMapnamesGlobal(Location loc, OpBuilder &builder, unsigned nbArg,
+                            SmallVector<std::string> &names) {
+  MLIRContext *ctx = builder.getContext();
+  SmallVector<LLVM::GlobalOp> globals;
+  unsigned index = 0;
+  for (StringRef name : names) {
+    std::string locStr;
+    llvm::raw_string_ostream locOS(locStr);
+    if (auto fileLoc = loc.dyn_cast<FileLineColLoc>()) {
+      locOS << ";" << name << ";" << fileLoc.getFilename() << ";0;"
+            << name.size() << ";;";
+    } else {
+      locOS << ";" << name << ";unknown;0;0;;";
+    }
+    auto globalStringType =
+        LLVM::LLVMArrayType::get(IntegerType::get(ctx, 8), locOS.str().size());
+    std::string globalName = std::string(llvm::formatv("{0}_mapname", index));
+    auto globalStr = builder.create<LLVM::GlobalOp>(
+        loc, globalStringType,
+        /*isConstant=*/true, LLVM::Linkage::Private, globalName,
+        builder.getStringAttr(locOS.str()));
+    globalStr.unnamed_addrAttr(
+        LLVM::UnnamedAddrAttr::get(ctx, LLVM::UnnamedAddr::Global));
+    globals.push_back(globalStr);
+    ++index;
+  }
+
+  auto llvmI32Type = IntegerType::get(ctx, 32);
+  auto i8Ty = IntegerType::get(ctx, 8);
+  auto i8PtrTy = LLVM::LLVMPointerType::get(i8Ty);
+
+  auto offloadMapnamesType =
+      LLVM::LLVMArrayType::get(LLVM::LLVMPointerType::get(i8Ty), nbArg);
+
+  auto offloadMapnames = builder.create<LLVM::GlobalOp>(
+      loc, offloadMapnamesType, /*isConstant=*/true, LLVM::Linkage::Private,
+      globalOffloadMapnamesName, Attribute());
+
+  // Initialize the global with global string created above.
+  Location globalLoc = offloadMapnames.getLoc();
+  Region &region = offloadMapnames.getInitializerRegion();
+  Block *block = builder.createBlock(&region);
+  builder.setInsertionPoint(block, block->begin());
+
+  Value mapValue =
+      builder.create<LLVM::UndefOp>(globalLoc, offloadMapnamesType);
+  Value zero = builder.create<LLVM::ConstantOp>(globalLoc, llvmI32Type,
+                                                builder.getI64IntegerAttr(0));
+  index = 0;
+  for (LLVM::GlobalOp global : globals) {
+    Value addrOfGlobal = builder.create<LLVM::AddressOfOp>(globalLoc, global);
+    auto globalGep = builder.create<LLVM::GEPOp>(
+        globalLoc, i8PtrTy, addrOfGlobal, ArrayRef<Value>{zero, zero});
+    mapValue = builder.create<LLVM::InsertValueOp>(
+        globalLoc, offloadMapnamesType, mapValue, globalGep,
+        ArrayAttr::get(ctx,
+                       {builder.getIntegerAttr(builder.getI32Type(), index)}));
+    ++index;
+  }
+  builder.create<LLVM::ReturnOp>(globalLoc, ArrayRef<Value>({mapValue}));
+  builder.setInsertionPointAfter(offloadMapnames);
+  return offloadMapnames;
+}
+
+static LLVM::LLVMFuncOp getTgtTargetDataBeginMapperFuncOp(ModuleOp &module,
+                                                          OpBuilder &builder) {
+  MLIRContext *ctx = module.getContext();
+  auto mapperFunc =
+      module.lookupSymbol<LLVM::LLVMFuncOp>(tgtTargetDataBeginMapperName);
+  if (!mapperFunc) {
+    OpBuilder::InsertionGuard guard(builder);
+    builder.setInsertionPointToStart(module.getBody());
+    auto mapperFuncTy = getTgtTargetDataBeginMapperType(ctx);
+    mapperFunc = builder.create<LLVM::LLVMFuncOp>(
+        builder.getUnknownLoc(), tgtTargetDataBeginMapperName, mapperFuncTy);
+  }
+  return mapperFunc;
+}
+
+//===----------------------------------------------------------------------===//
+// Conversion patterns
+//===----------------------------------------------------------------------===//
+
+struct EnterDataOpConversion : public ConvertOpToLLVMPattern<EnterDataOp> {
+  using ConvertOpToLLVMPattern<EnterDataOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(EnterDataOp enterDataOp, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    // acc.enter_data operation is lowered to a call to
+    // __tgt_target_data_begin_mapper function from the libomptarget runtime.
+
+    // TODO
+    // - handle copyin operands
+    // - handle create_zero operands
+    // - handle attach operands
+    // - handle async
+    // - wait
+
+    auto loc = enterDataOp->getLoc();
+    auto module = enterDataOp->getParentOfType<ModuleOp>();
+    MLIRContext *ctx = module.getContext();
+
+    OpBuilder moduleBuilder(module.getBodyRegion(), rewriter.getListener());
+    auto globalLocationInfo =
+        createLocationInfoGlobalOp(loc, enterDataOp, moduleBuilder);
+
+    auto deviceId = rewriter.create<LLVM::ConstantOp>(
+        enterDataOp->getLoc(), rewriter.getI64Type(),
+        rewriter.getI64IntegerAttr(defaultDeviceId));
+
+    // Number of arguments in the enter_data operation.
+    // TODO include copyin, create_zero and attach operands.
+    unsigned nbArg = enterDataOp.createOperands().size();
+
+    auto argNumCstOp = rewriter.create<LLVM::ConstantOp>(
+        enterDataOp->getLoc(), rewriter.getI32Type(),
+        rewriter.getI32IntegerAttr(nbArg));
+
+    auto i8Ty = IntegerType::get(ctx, 8);                       // i8
+    auto i8PtrTy = LLVM::LLVMPointerType::get(i8Ty);            // i8*
+    auto i8PtrPtrTy = LLVM::LLVMPointerType::get(i8PtrTy);      // i8**
+    auto i8PtrArrTy = LLVM::LLVMArrayType::get(i8PtrTy, nbArg); // [nbArg x i8*]
+    auto i8PtrArrPtrTy =
+        LLVM::LLVMPointerType::get(i8PtrArrTy);              // [nbArg x i8*]*
+    auto i64Ty = IntegerType::get(ctx, 64);                  // i64
+    auto i64PtrTy = LLVM::LLVMPointerType::get(i64Ty);       // i64*
+    auto i64ArrTy = LLVM::LLVMArrayType::get(i64Ty, nbArg);  // [nbArg x i64*]
+    auto i64ArrTyPtr = LLVM::LLVMPointerType::get(i64ArrTy); // [nbArg x i64*]*
+
+    // %4 = alloca [nbArg x i8*], align 8
+    auto argsBasePtr = rewriter.create<LLVM::AllocaOp>(
+        loc, i8PtrArrPtrTy, argNumCstOp, /*alignment=*/0);
+    // %5 = alloca [nbArg x i8*], align 8
+    auto argsPtr = rewriter.create<LLVM::AllocaOp>(
+        loc, i8PtrArrPtrTy, argNumCstOp, /*alignment=*/0);
+    // %7 = alloca [nbArg x i64], align 8
+    auto argSizesPtr = rewriter.create<LLVM::AllocaOp>(
+        loc, i64ArrTyPtr, argNumCstOp, /*alignment=*/0);
+
+    auto zero = rewriter.create<LLVM::ConstantOp>(
+        loc, IntegerType::get(ctx, 32), rewriter.getI32IntegerAttr(0));
+
+    SmallVector<int64_t> flags;
+    SmallVector<std::string> names;
+    unsigned index = 0;
+    // create operands are handled as alloc call.
+    for (auto data : enterDataOp.createOperands()) {
+      Value crtIndex = rewriter.create<LLVM::ConstantOp>(
+          loc, IntegerType::get(ctx, 32), rewriter.getI32IntegerAttr(index));
+
+      // Handle operands that were converted to MemRefDescriptors.
+      if (auto structType = data.getType().dyn_cast<LLVM::LLVMStructType>()) {
+        // Calculate the size of the memref and store it for the fct call.
+        Value sizeBytes = getSizeInBytes(loc, structType, rewriter);
+        auto sizesPtr = rewriter.create<LLVM::GEPOp>(
+            loc, i64PtrTy, argSizesPtr, ArrayRef<Value>{zero, zero});
+        rewriter.create<LLVM::StoreOp>(loc, sizeBytes, sizesPtr);
+
+        MemRefDescriptor descriptor(data);
+        Value allocatedPtr = descriptor.allocatedPtr(rewriter, loc);
+
+        auto basePtr = rewriter.create<LLVM::GEPOp>(
+            loc, i8PtrTy, argsBasePtr, ArrayRef<Value>{zero, crtIndex});
+        Value basePtrCast = rewriter.create<LLVM::BitcastOp>(
+            loc, LLVM::LLVMPointerType::get(descriptor.getElementPtrType()),
+            basePtr);
+        rewriter.create<LLVM::StoreOp>(loc, allocatedPtr, basePtrCast);
+
+        auto ptr = rewriter.create<LLVM::GEPOp>(
+            loc, i8PtrTy, argsPtr, ArrayRef<Value>{zero, crtIndex});
+        Value ptrCast = rewriter.create<LLVM::BitcastOp>(
+            loc, LLVM::LLVMPointerType::get(descriptor.getElementPtrType()),
+            ptr);
+        rewriter.create<LLVM::StoreOp>(loc, allocatedPtr, ptrCast);
+
+        flags.push_back(createFlag);
+
+        // TODO: pass more information about line location and name of memref.
+        names.push_back(std::string(llvm::formatv("memref_{0}", index)));
+      } else {
+        // Not supported types for now.
+        enterDataOp.emitOpError("Unsupported type ") << data.getType();
+        return failure();
+      }
+      ++index;
+    }
+
+    auto offloadMaptypes =
+        createOffloadMaptypesGlobal(loc, moduleBuilder, nbArg, flags);
+
+    auto offloadMapnames =
+        createOffloadMapnamesGlobal(loc, moduleBuilder, nbArg, names);
+
+    Value addrLocInfo =
+        rewriter.create<LLVM::AddressOfOp>(loc, globalLocationInfo);
+    auto argsBasePtrArg = rewriter.create<LLVM::GEPOp>(
+        loc, i8PtrPtrTy, argsBasePtr, ArrayRef<Value>{zero, zero});
+    auto argsPtrArg = rewriter.create<LLVM::GEPOp>(loc, i8PtrPtrTy, argsPtr,
+                                                   ArrayRef<Value>{zero, zero});
+    auto argSizesPtrArg = rewriter.create<LLVM::GEPOp>(
+        loc, i64PtrTy, argSizesPtr, ArrayRef<Value>{zero, zero});
+    Value addrOffloadMaptypes =
+        rewriter.create<LLVM::AddressOfOp>(loc, offloadMaptypes);
+    auto offloadMaptypesArg = rewriter.create<LLVM::GEPOp>(
+        loc, i64PtrTy, addrOffloadMaptypes, ArrayRef<Value>{zero, zero});
+    Value addrOffloadMapnames =
+        rewriter.create<LLVM::AddressOfOp>(loc, offloadMapnames);
+    auto offloadMapnamesArg = rewriter.create<LLVM::GEPOp>(
+        loc, i8PtrPtrTy, addrOffloadMapnames, ArrayRef<Value>{zero, zero});
+    auto nullPtr = rewriter.create<LLVM::NullOp>(loc, i8PtrPtrTy);
+
+    auto mapperFuncOp = getTgtTargetDataBeginMapperFuncOp(module, rewriter);
+    rewriter.replaceOpWithNewOp<LLVM::CallOp>(
+        enterDataOp, mapperFuncOp,
+        ValueRange{addrLocInfo, deviceId, argNumCstOp, argsBasePtrArg,
+                   argsPtrArg, argSizesPtrArg, offloadMaptypesArg,
+                   offloadMapnamesArg, nullPtr});
+
+    return success();
+  }
+};
+
+void mlir::populateOpenACCToLLVMConversionPatterns(
+    LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
+  patterns.add<EnterDataOpConversion>(converter);
+}
+
+namespace {
+struct ConvertOpenACCToLLVMPass
+    : public ConvertOpenACCToLLVMBase<ConvertOpenACCToLLVMPass> {
+  void runOnOperation() override;
+};
+} // namespace
+
+void ConvertOpenACCToLLVMPass::runOnOperation() {
+  auto op = getOperation();
+  auto *context = op.getContext();
+
+  // Convert to OpenACC operations with LLVM IR dialect
+  RewritePatternSet patterns(context);
+  LLVMTypeConverter converter(context);
+  populateStdToLLVMConversionPatterns(converter, patterns);
+  populateOpenACCToLLVMConversionPatterns(converter, patterns);
+
+  ConversionTarget target(*context);
+  target.addLegalDialect<LLVM::LLVMDialect>();
+
+  if (failed(applyPartialConversion(op, target, std::move(patterns))))
+    signalPassFailure();
+}
+
+std::unique_ptr<OperationPass<ModuleOp>>
+mlir::createConvertOpenACCToLLVMPass() {
+  return std::make_unique<ConvertOpenACCToLLVMPass>();
+}

mlir/test/Conversion/OpenACCToLLVM/convert-standalone-data-to-llvmir.mlir

+// RUN: mlir-opt -convert-std-to-llvm -convert-openacc-to-llvm %s | FileCheck %s
+
+func @testenterdataop(%a: memref<10xf32>, %b: memref<10xf32>) -> () {
+  acc.enter_data create(%a, %b : memref<10xf32>, memref<10xf32>)
+  return
+}
+
+// CHECK: llvm.func @__tgt_target_data_begin_mapper(!llvm.ptr<struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>>, i64, i32, !llvm.ptr<ptr<i8>>, !llvm.ptr<ptr<i8>>, !llvm.ptr<i64>, !llvm.ptr<i64>, !llvm.ptr<ptr<i8>>, !llvm.ptr<ptr<i8>>)
+// CHECK:      llvm.mlir.global private constant [[LOCSTR:@.*]](";{{.*}};{{.*}};4;3;;") {unnamed_addr = 2 : i64}
+// CHECK:      llvm.mlir.global private constant [[IDENT_T:@.*]]() {unnamed_addr = 2 : i64} : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)> {
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.undef : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   [[ZERO:%.*]] = llvm.mlir.constant(0 : i64) : i32
+// CHECK-NEXT:   [[TWO:%.*]] = llvm.mlir.constant(2 : i64) : i32
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue [[ZERO]], %{{.*}}[0 : i32] : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue [[TWO]], %{{.*}}[1 : i32] : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue [[ZERO]], %{{.*}}[2 : i32] : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue [[ZERO]], %{{.*}}[3 : i32] : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.addressof [[LOCSTR]] : !llvm.ptr<array<{{.*}} x i8>>
+// CHECK-NEXT:   %{{.*}} = llvm.getelementptr %{{.*}}[%{{.*}}, %{{.*}}] : (!llvm.ptr<array<{{.*}} x i8>>, i32, i32) -> !llvm.ptr<i8>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue %{{.*}}, %{{.*}}[4 : i32] : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT:   llvm.return %{{.*}} : !llvm.struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>
+// CHECK-NEXT: }
+// CHECK:      llvm.mlir.global private constant @".offload_maptypes"(dense<0> : vector<{{.*}}xi64>) {unnamed_addr = 2 : i64} : !llvm.array<{{.*}} x i64>
+// CHECK:      llvm.mlir.global private constant [[MAPNAME0:@.*]](";memref_0;{{.*}};0;8;;") {unnamed_addr = 2 : i64}
+// CHECK:      llvm.mlir.global private constant [[MAPNAME1:@.*]](";memref_1;{{.*}};0;8;;") {unnamed_addr = 2 : i64}
+// CHECK:      llvm.mlir.global private constant @".offload_mapnames"() : !llvm.array<2 x ptr<i8>> {
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.undef : !llvm.array<2 x ptr<i8>>
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.constant(0 : i64) : i32
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.addressof [[MAPNAME0]] : !llvm.ptr<array<{{.*}} x i8>>
+// CHECK-NEXT:   %{{.*}} = llvm.getelementptr %{{.*}}[%{{.*}}, %{{.*}}] : (!llvm.ptr<array<{{.*}} x i8>>, i32, i32) -> !llvm.ptr<i8>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue %{{.*}}, %{{.*}}[0 : i32] : !llvm.array<2 x ptr<i8>>
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.addressof [[MAPNAME1]] : !llvm.ptr<array<{{.*}} x i8>>
+// CHECK-NEXT:   %{{.*}} = llvm.getelementptr %{{.*}}[%1, %1] : (!llvm.ptr<array<{{.*}} x i8>>, i32, i32) -> !llvm.ptr<i8>
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue %{{.*}}, %{{.*}}[1 : i32] : !llvm.array<2 x ptr<i8>>
+// CHECK-NEXT:   llvm.return %7 : !llvm.array<2 x ptr<i8>>
+// CHECK-NEXT: }
+
+// CHECK: llvm.func @testenterdataop
+
+// CHECK: [[DEVICEID:%.*]] = llvm.mlir.constant(-1 : i64) : i64
+// CHECK: [[ARGNUM:%.*]] = llvm.mlir.constant(2 : i32) : i32
+
+// CHECK: [[ARGS_BASE:%.*]] = llvm.alloca %{{.*}} x !llvm.array<{{.*}} x ptr<i8>> : (i32) -> !llvm.ptr<array<{{.*}} x ptr<i8>>>
+// CHECK: [[ARGS:%.*]] = llvm.alloca %{{.*}} x !llvm.array<{{.*}} x ptr<i8>> : (i32) -> !llvm.ptr<array<{{.*}} x ptr<i8>>>
+// CHECK: [[SIZES:%.*]] = llvm.alloca %{{.*}} x !llvm.array<{{.*}} x i64> : (i32) -> !llvm.ptr<array<{{.*}} x i64>>
+
+// CHECK: llvm.call @__tgt_target_data_begin_mapper(%{{.*}}, [[DEVICEID]], [[ARGNUM]], %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.ptr<struct<"struct.ident_t", (i32, i32, i32, i32, ptr<i8>)>>, i64, i32, !llvm.ptr<ptr<i8>>, !llvm.ptr<ptr<i8>>, !llvm.ptr<i64>, !llvm.ptr<i64>, !llvm.ptr<ptr<i8>>, !llvm.ptr<ptr<i8>>) -> ()
+// CHECK-NOT: acc.enter_data