[mlir][openacc] Introduce OpenACC dialect with parallel, data, loop operations

[clementval]

Summary

This patch introduces the OpenACC dialect with three operation defined
parallel, data and loop operations with custom parsing and printing.

OpenACC dialect RFC can be find here: https://llvm.discourse.group/t/rfc-openacc-dialect/546/2

[clementval]

Fix some formatting issues

[rriddle]

mlir/include/mlir/Dialect/OpenACC/OpenACC.h

14

nit: Please fix this lint error.

30

nit: Please add documentation here.

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

18

nit: Drop the extra newline.

35

Same here.

68

nit: You could invoke the ODS generated accessor here(region()), or just remove this method in favor of that.

150

nit: Other dialects generally put this in the base Op class:

let printer = [{ return ::print(p, *this); }];
let verifier = [{ return ::verify(*this); }];
let parser = [{ return ::parse$cppClass(parser, result); }];

The assemblyFormat overrides the parser/printer fields when present, and if an op doesn't need a verifier you can use let verifier = ?;

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

43

nit: Drop trivial braces, also unsigned int -> unsigned in several places.

47

nit: Can you spell out auto here.

59

nit: Replace this method with parser.parseOptionalAttrDictWithKeyword, it does this for you.

76

nit: Please use proper punctuation in comments.

94

nit: parser.resolveOperands?

132

nit: Please use /// for top-level comments.

255

nit: Spell out auto here.

258

nit: Please add /*paramName=*/ comments to these constants.

471

You can add ParentOneOf to the op definition to capture this constraint.

475

nit: Please avoid the use of e as an iterator variable that isn't end, it can be confusing.

[rriddle]

Thanks, added some initial comments.

[clementval]

Address comments

[clementval]

@rriddle Thanks for the feedback. Just pushed a new version of the patch.

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

150

This is now defined in the base class. I added let verifier = ?; on ops for now until they are implemented in a follow-up patch.

[kiranchandramohan]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

62

I am probably missing a point here. (For reference, I have posted below all the clauses for the parallel construct for Open ACC 3.0.)

1. Where are the allowed attributes specified?
2. Are only a subset of the clauses modelled here?

async [( int-expr )]
wait [( int-expr-list )]
num_gangs( int-expr )
num_workers( int-expr )
vector_length( int-expr )
device_type( device-type-list )
if( condition )
self [( condition )]
reduction( operator:var-list )
copy( var-list )
copyin( [readonly:]var-list )
copyout( [zero:]var-list )
create( [zero:]var-list )
no_create( var-list )
present( var-list )
deviceptr( var-list )
attach( var-list )
private( var-list )
firstprivate( var-list )
default( none | present )

[clementval]

Add some "clauses" to parallel op and loop op

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

62

The operation does not have currently all the clauses that are normally allowed. I have updated the patch to add most of them but some will come later when we will implement them such as device_type.
ParallelOp has just a single attribute now that is the reduction operation.

[clementval]

Fix header guard

[kiranchandramohan]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

106

Will this preclude code with control flow? Or is this because the standard does not allow such kind of code in OpenACC parallel regions?

[clementval]

Update region for parallel, loop and data op

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

106

Actually this is an old copy-paste from my old branch. I just updated the patch so the ops accept AnyRegion similar to GPU func op or omp.parallel op.

[kiranchandramohan]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

106

Thanks for making the change.

Actually this is a for my information question also. I am interested in knowing whether having anyregion causes any problems when lowering (particularly the loop) to GPU or linalg dialects?

[SouraVX]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

74

Could you please provide motivation/rationale for this Op having a single region with a single block ?

[clementval]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

74

This is not limited to a single region. But with the SingleBlockImplicitTerminator it limits to 0 or 1 block per region and this is not what is intended. I'll update the ops to accpet more than a single block in the region.

106

I'm not seeing any problem on having 1 or more region as I mentioned during our call.

[SouraVX]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

179

Probably I'm missing the bigger picture here, but can't we gave a generic terminator ? This by naming seems specific to DataEndOP ? I

[clementval]

Address review comments

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

179

Good point. Since there is only this op right now that need this default terminator I named it after the op but a generic one would be a better fit. Updated in the patch right now.

[SouraVX]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

179

Didn't noticed the test, for this op. In 'ops.mlir'. Is it too obvious ? Too ignore?

mlir/test/Dialect/OpenMP/ops.mlir

109 ↗

(On Diff #282480)

I believe, this change must be unintended.
?
Please take care of it whenever you revise.

[SouraVX]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

177

Is this "_" in naming of Op intentional ? I tried your patch locally, It's resulting in name as acc._terminator. That's a bit odd (leading underscore)considering naming of the rest of the ops in dialect.
This is also why I mentioned for the test case for this Op too. test case helps to capture these in very early stage.

[clementval]

Address review comments

[clementval]

Updated the patch

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

177

I don't remember which other dialect is using this naming style but I'm fine with removing the underscore from the name.

179

Added in the test,

mlir/test/Dialect/OpenMP/ops.mlir

109 ↗

(On Diff #282480)

Sure this is unintended.

[SouraVX]

LGTM(at least the part I reviewed). But it would be good to wait for @kiranchandramohan and IIRC @ftynse has contributed a lot in GPU dialect. So it would be nice to have there views/thoughts also.
Thanks for your patience :)

[clementval]

In D84268#2193486, @SouraVX wrote:

LGTM(at least the part I reviewed). But it would be good to wait for @kiranchandramohan and IIRC @ftynse has contributed a lot in GPU dialect. So it would be nice to have there views/thoughts also.
Thanks for your patience :)

Sure will wait on their review. It's the first bits of the dialect so there is no problem to wait some more.

[rriddle]

mlir/include/mlir/Dialect/OpenACC/OpenACC.h

30

Top Level comments should use ///

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

80

Please follow the same format as the other dialects for mlir snippets, i.e.

Example:

<triple-tick-here>mlir
<no_indent_snippet_here>
<triple-tick-here>

201

Missing a period here?

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

92

Use llvm::enumerate instead of a separate index.

95

nit: Spell out auto here.

95

nit: Use llvm::interleaveComma here instead of an indexed loop

142

nit: Just move this to a new line instead of trailing the previous.

537

I don't think this is used anywhere.

579

This isn't actually necessary.

[kiranchandramohan]

This looks ready. A few nit comments.

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

234

Nit: space after full stop.

238

Is a custom parser/printer not needed since there is only one variadic operand?
Noticed that there are no tests for yield with operand.

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

126

Nit: not in sync with contents.

478

Nit: Do you want to specify reduction here?

[clementval]

Address review comments

[clementval]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

238

Yeah since there is only one variadic operand the op is not flagged as AttrSizedOperandSegments. The assemblyFormat is fine to generate the parser/printer.

It is hard to put together a test using yield with operands without the full reduction being defined, So it will come together I guess in a follow-up patch.

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

478

Yes! good catch. Added woker well.

[rriddle]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

238

Yeah since there is only one variadic operand the op is not flagged as AttrSizedOperandSegments. The assemblyFormat is fine to generate the parser/printer.

The assemblyFormat can already handle AttrSizedOperandSegments BTW.

[clementval]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

238

Good to know. Do you have an example of op using this? Would be interested to check how it works.

[rriddle]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

238

It gets handled automatically, you don't have to do anything. E.g. CondBranchOp:
https://github.com/llvm/llvm-project/blob/823ffef009152ba1210740c44d472d1d6e56afa3/mlir/include/mlir/Dialect/StandardOps/IR/Ops.td#L1191

If you use the operands directive instead of listing each operand group individually, the segment size attribute just gets printed along with the attribute dictionary.

[clementval]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

238

Very nice! Thanks for the example.

[kiranchandramohan]

LGTM. Please wait for @rriddle to approve.

mlir/test/Dialect/OpenACC/ops.mlir

167

Nit:

[clementval]

New line at the end of file

[rriddle]

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

23

nit: This looks like what you would put in the summary field.

39

This looks weird, remove?

80

Can you add in the Example: part? It is consistent with the other dialects.

87

nit: Move the verifier to later in the op definitions.

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

18

I think some of these can be trimmed.

29

I think this needs to be updated after a recent refactoring.

35

What unknown operations are being allowed here? Why do we need to do this?

42

nit: Drop llvm:: here, this along with many others are re-exported in the MLIR namespace.

82

nit: Just return this.

103

nit: This type doesn't look like it's being mutated, drop the reference specifier?

142

nit: Spell out auto here.

578

nit: Drop the trivial braces here.

[clementval]

Address review comments

[clementval]

@rriddle I pushed a new version of the patch with changes to address your comments. Are we good to go like this?

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

18

Yeah I have removed a bunch.

35

I'm gonna remove this since it is not necessary at this stage. Will add it again if there is a need later.

mlir/include/mlir/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
+add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
 add_subdirectory(Vector)

mlir/include/mlir/Dialect/OpenACC/CMakeLists.txt

+set(LLVM_TARGET_DEFINITIONS OpenACCOps.td)
+mlir_tablegen(OpenACCOpsDialect.h.inc -gen-dialect-decls -dialect=acc)
+mlir_tablegen(OpenACCOps.h.inc -gen-op-decls)
+mlir_tablegen(OpenACCOps.cpp.inc -gen-op-defs)
+mlir_tablegen(OpenACCOpsEnums.h.inc -gen-enum-decls)
+mlir_tablegen(OpenACCOpsEnums.cpp.inc -gen-enum-defs)
+add_mlir_doc(OpenACCOps -gen-dialect-doc OpenACCDialect Dialects/)
+add_public_tablegen_target(MLIROpenACCOpsIncGen)
+

mlir/include/mlir/Dialect/OpenACC/OpenACC.h

+//===- OpenACC.h - MLIR OpenACC Dialect -------------------------*- C++ -*-===//
+//
+// Part of the MLIR Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+// ============================================================================
+//
+// This file declares the OpenACC dialect in MLIR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_OPENACC_H_
+#define MLIR_DIALECT_OPENACC_H_
+
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+
+#include "mlir/Dialect/OpenACC/OpenACCOpsEnums.h.inc"
+
+namespace mlir {
+namespace acc {
+
+#define GET_OP_CLASSES
+#include "mlir/Dialect/OpenACC/OpenACCOps.h.inc"
+
+#include "mlir/Dialect/OpenACC/OpenACCOpsDialect.h.inc"
+
+// Enumeration used to encode the execution mapping on a loop construct.
+// They refer directly to the OpenACC 3.0 standard:
+// 2.9.2. gang
+// 2.9.3. worker
+// 2.9.4. vector
+// 2.9.5. seq
+//
+// Value can be combined bitwise to reflect the mapping applied to the
+// construct. e.g. `acc.loop gang vector`, the `gang` and `vector` could be
+// combined and the final mapping value would be 5 (4 & 1).
+enum OpenACCExecMapping { NONE = 0, VECTOR = 1, WORKER = 2, GANG = 4, SEQ = 8 };
+
+} // end namespace acc
+} // end namespace mlir
+
+#endif // MLIR_DIALECT_OPENACC_H_

mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

+//===- OpenACC.td - OpenACC operation definitions ----------*- tablegen -*-===//
+//
+// Part of the MLIR Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+// =============================================================================
+//
+// Defines MLIR OpenACC operations.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef OPENACC_OPS
+#define OPENACC_OPS
+
+include "mlir/IR/OpBase.td"
+
+def OpenACC_Dialect : Dialect {
+  let name = "acc";
+
+  let description = [{
+    An OpenACC dialect for MLIR.
+
+    This dialect models the construct from the OpenACC 3.0 directive language.
+  }];
+
+  let cppNamespace = "acc";
+}
+
+// Base class for OpenACC dialect ops.
+class OpenACC_Op<string mnemonic, list<OpTrait> traits = []> :
+  Op<OpenACC_Dialect, mnemonic, traits> {
+  // For every OpenACC op, there needs to be a:
+  //   * void print(OpAsmPrinter &p, ${C++ class of Op} op)
+  //   * LogicalResult verify(${C++ class of Op} op)
+  //   * ParseResult parse${C++ class of Op}(OpAsmParser &parser,
+  //                                         OperationState &result)
+  // functions.
+  let printer = [{ return ::print(p, *this); }];
+  let verifier = [{ return ::verify(*this); }];
+  let parser = [{ return ::parse$cppClass(parser, result); }];
+}
+
+//===----------------------------------------------------------------------===//
+// 2.5.1 parallel Construct
+//===----------------------------------------------------------------------===//
+
+def OpenACC_ParallelOp : OpenACC_Op<"parallel",
+    [SingleBlockImplicitTerminator<"YieldOp">,
+     AttrSizedOperandSegments]> {
+  let summary = "parallel construct";
+  let description = [{
+    The "acc.parallel" operation represents a parallel construct block. It has
+    one region to be executued in parallel on the current device.
+
+    acc.parallel {
+      ... // body
+    }
+  }];
+
+  let verifier = ?;
+  let arguments = (ins Optional<Index>:$numGangs,
+                       Optional<Index>:$numWorkers,
+                       Optional<Index>:$vectorLength,
+                       Optional<I1>:$ifCond,
+                       Variadic<AnyType>:$gangPrivateOperands,
+                       Variadic<AnyType>:$gangFirstPrivateOperands,
+                       Variadic<AnyType>:$createOperands,
+                       Variadic<AnyType>:$copyinOperands,
+                       Variadic<AnyType>:$copyoutOperands);
+  let regions = (region SizedRegion<1>:$region);
+
+  let extraClassDeclaration = [{
+    static StringRef getNumGangsKeyword() { return "num_gangs"; }
+    static StringRef getNumWorkersKeyword() { return "num_workers"; }
+    static StringRef getVectorLengthKeyword() { return "vector_length"; }
+    static StringRef getPrivateKeyword() { return "private"; }
+    static StringRef getFirstPrivateKeyword() { return "firstprivate"; }
+    static StringRef getCreateKeyword() { return "create"; }
+    static StringRef getCopyinKeyword() { return "copyin"; }
+    static StringRef getCopyoutKeyword() { return "copyout"; }
+    static StringRef getIfKeyword() { return "if"; }
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// 2.6.5 data Construct
+//===----------------------------------------------------------------------===//
+
+def OpenACC_DataOp : OpenACC_Op<"data",
+    [SingleBlockImplicitTerminator<"DataEndOp">,
+     AttrSizedOperandSegments]> {
+  let summary = "data construct";
+  let description = [{
+    The "acc.data" operation represents a data construct. It defines vars to
+    be allocated in the current device memory for the duration of the region,
+    whether data should be copied from local memory to the current device
+    memory upon region entry , and copied from device memory to local memory
+    upon region exit.
+
+    acc.data {
+      ... // body
+    }
+  }];
+
+  let verifier = ?;
+  let arguments = (ins Variadic<AnyType>:$presentOperands,
+                       Variadic<AnyType>:$copyOperands,
+                       Variadic<AnyType>:$copyinOperands,
+                       Variadic<AnyType>:$copyoutOperands,
+                       Variadic<AnyType>:$createOperands,
+                       Variadic<AnyType>:$noCreateOperands,
+                       Variadic<AnyType>:$deleteOperands,
+                       Variadic<AnyType>:$attachOperands,
+                       Variadic<AnyType>:$detachOperands);
+  let regions = (region SizedRegion<1>:$region);
+  let extraClassDeclaration = [{
+    static StringRef getAttachKeyword() { return "attach"; }
+    static StringRef getDeleteKeyword() { return "delete"; }
+    static StringRef getDetachKeyword() { return "detach"; }
+    static StringRef getCopyinKeyword() { return "copyin"; }
+    static StringRef getCopyKeyword() { return "copy"; }
+    static StringRef getCopyoutKeyword() { return "copyout"; }
+    static StringRef getCreateKeyword() { return "create"; }
+    static StringRef getNoCreateKeyword() { return "no_create"; }
+    static StringRef getPresentKeyword() { return "present"; }
+  }];
+}
+
+def OpenACC_DataEndOp : OpenACC_Op<"_data_end", [Terminator]> {
+  let summary = "terminator for OpenACC data regions";
+  let description = [{
+    A terminator operation for regions that appear in the body of OpenACC data
+    operation.  Data construct regions are not expected to return any value so
+    the terminator takes no operands. The terminator op returns control to the
+    enclosing op.
+  }];
+
+  let verifier = ?;
+  let assemblyFormat = "attr-dict";
+}
+
+//===----------------------------------------------------------------------===//
+// 2.9 loop Construct
+//===----------------------------------------------------------------------===//
+
+def OpenACC_LoopOp : OpenACC_Op<"loop",
+                        [SingleBlockImplicitTerminator<"YieldOp">]> {
+  let summary = "loop construct";
+  let description = [{
+    The "acc.loop" operation represent the acc loop construct
+  }];
+
+  let verifier = ?;
+  let results = (outs Variadic<AnyType>:$results);
+  let regions = (region SizedRegion<1>:$region);
+
+  let extraClassDeclaration = [{
+    static StringRef getCollapseAttrName() { return "collapse"; }
+    static StringRef getExecutionMappingAttrName() { return "exec_mapping"; }
+    static StringRef getGangAttrName() { return "gang"; }
+    static StringRef getSeqAttrName() { return "seq"; }
+    static StringRef getVectorAttrName() { return "vector"; }
+    static StringRef getWorkerAttrName() { return "worker"; }
+  }];
+}
+
+// Yield operation for the acc.loop and acc.parallel operations.
+def OpenACC_YieldOp : OpenACC_Op<"yield", [Terminator,
+    ParentOneOf<["ParallelOp, LoopOp"]>]> {
+  let summary = "Acc yield and termination operation";
+  let description = [{
+
+  }];
+  let arguments = (ins Variadic<AnyType>:$operands);
+  let builders = [
+    OpBuilder<"OpBuilder &builder, OperationState &result",
+              [{ /* nothing to do */ }]>
+  ];
+
+  let verifier = ?;
+  let assemblyFormat = "attr-dict ($operands^ `:` type($operands))?";
+}
+
+#endif // OPENACC_OPS
+

mlir/include/mlir/IR/DialectSymbolRegistry.def

 DEFINE_SYM_KIND_RANGE(TENSORFLOW_CONTROL)
 DEFINE_SYM_KIND_RANGE(TENSORFLOW_EXECUTOR)
 DEFINE_SYM_KIND_RANGE(TENSORFLOW)
 DEFINE_SYM_KIND_RANGE(LLVM)
 DEFINE_SYM_KIND_RANGE(QUANTIZATION)
 DEFINE_SYM_KIND_RANGE(IREE) // IREE stands for IR Execution Engine
 DEFINE_SYM_KIND_RANGE(LINALG) // Linear Algebra Dialect
 DEFINE_SYM_KIND_RANGE(FIR) // Flang Fortran IR Dialect
+DEFINE_SYM_KIND_RANGE(OPENACC) // OpenACC IR Dialect
 DEFINE_SYM_KIND_RANGE(OPENMP) // OpenMP IR Dialect
 DEFINE_SYM_KIND_RANGE(TOY) // Toy language (tutorial) Dialect
 DEFINE_SYM_KIND_RANGE(SPIRV) // SPIR-V dialect
 DEFINE_SYM_KIND_RANGE(XLA_HLO) // XLA HLO dialect
 DEFINE_SYM_KIND_RANGE(SHAPE) // Shape dialect
 
 // The following ranges are reserved for experimenting with MLIR dialects in a
 // private context without having to register them here.

mlir/include/mlir/InitAllDialects.h

 #include "mlir/Dialect/AVX512/AVX512Dialect.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/GPU/GPUDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMAVX512Dialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
 #include "mlir/Dialect/Linalg/IR/LinalgOps.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/Quant/QuantOps.h"
 #include "mlir/Dialect/SCF/SCF.h"
 #include "mlir/Dialect/SDBM/SDBMDialect.h"
 #include "mlir/Dialect/SPIRV/SPIRVDialect.h"
 #include "mlir/Dialect/Shape/IR/Shape.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/IR/Dialect.h"
 
 namespace mlir {
 
 // This function should be called before creating any MLIRContext if one expect
 // all the possible dialects to be made available to the context automatically.
 inline void registerAllDialects() {
   static bool init_once = []() {
+    registerDialect<acc::OpenACCDialect>();
     registerDialect<AffineDialect>();
     registerDialect<avx512::AVX512Dialect>();
     registerDialect<gpu::GPUDialect>();
     registerDialect<LLVM::LLVMAVX512Dialect>();
     registerDialect<LLVM::LLVMDialect>();
     registerDialect<linalg::LinalgDialect>();
     registerDialect<scf::SCFDialect>();
     registerDialect<omp::OpenMPDialect>();

mlir/lib/Dialect/CMakeLists.txt

 add_subdirectory(Affine)
 add_subdirectory(AVX512)
 add_subdirectory(GPU)
 add_subdirectory(Linalg)
 add_subdirectory(LLVMIR)
+add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(Quant)
 add_subdirectory(SCF)
 add_subdirectory(SDBM)
 add_subdirectory(Shape)
 add_subdirectory(SPIRV)
 add_subdirectory(StandardOps)
 add_subdirectory(Vector)

mlir/lib/Dialect/OpenACC/CMakeLists.txt

+add_mlir_dialect_library(MLIROpenACC
+  IR/OpenACC.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/OpenACC
+
+  DEPENDS
+  MLIROpenACCOpsIncGen
+
+  LINK_LIBS PUBLIC
+  MLIRIR
+  )
+

mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp

+//===- OpenACC.cpp - OpenACC MLIR Operations ------------------------------===//
+//
+// Part of the MLIR Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+// =============================================================================
+
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/Function.h"
+#include "mlir/IR/Matchers.h"
+#include "mlir/IR/Module.h"
+#include "mlir/IR/OpImplementation.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/StandardTypes.h"
+#include "mlir/IR/Value.h"
+
+#include "mlir/Dialect/OpenACC/OpenACCOpsEnums.cpp.inc"
+
+using namespace mlir;
+using namespace acc;
+
+//===----------------------------------------------------------------------===//
+// OpenACC operations
+//===----------------------------------------------------------------------===//
+
+OpenACCDialect::OpenACCDialect(MLIRContext *context)
+    : Dialect(getDialectNamespace(), context) {
+  addOperations<
+#define GET_OP_LIST
+#include "mlir/Dialect/OpenACC/OpenACCOps.cpp.inc"
+      >();
+  allowsUnknownOperations();
+}
+
+template <typename StructureOp>
+static ParseResult parseRegions(OpAsmParser &parser, OperationState &state,
+                                unsigned nRegions = 1) {
+
+  llvm::SmallVector<Region *, 2> regions;
+  for (unsigned i = 0; i < nRegions; ++i)
+    regions.push_back(state.addRegion());
+
+  for (Region *region : regions) {
+    if (parser.parseRegion(*region, /*arguments=*/{}, /*argTypes=*/{}))
+      return failure();
+    StructureOp::ensureTerminator(*region, parser.getBuilder(), state.location);
+  }
+
+  return success();
+}
+
+static ParseResult
+parseOperandList(OpAsmParser &parser, StringRef keyword,
+                 SmallVectorImpl<OpAsmParser::OperandType> &args,
+                 SmallVectorImpl<Type> &argTypes, OperationState &result) {
+  if (failed(parser.parseOptionalKeyword(keyword)))
+    return success();
+
+  if (failed(parser.parseLParen()))
+    return failure();
+
+  // Exit early if the list is empty.
+  if (succeeded(parser.parseOptionalRParen()))
+    return success();
+
+  do {
+    OpAsmParser::OperandType arg;
+    Type type;
+
+    if (parser.parseRegionArgument(arg) || parser.parseColonType(type))
+      return failure();
+
+    args.push_back(arg);
+    argTypes.push_back(type);
+  } while (succeeded(parser.parseOptionalComma()));
+
+  if (failed(parser.parseRParen()))
+    return failure();
+
+  if (parser.resolveOperands(args, argTypes, parser.getCurrentLocation(),
+                             result.operands))
+    return failure();
+
+  return success();
+}
+
+static void printOperandList(Operation::operand_range operands,
+                             StringRef listName, OpAsmPrinter &printer) {
+  if (operands.size() > 0) {
+    printer << " " << listName << "(";
+    for (auto operand : operands)
+      printer << operand << ": " << operand.getType();
+    printer << ")";
+  }
+}
+
+static ParseResult parseOptionalOperand(OpAsmParser &parser, StringRef keyword,
+                                        OpAsmParser::OperandType &operand,
+                                        Type &type, bool &hasOptional,
+                                        OperationState &result) {
+  hasOptional = false;
+  if (succeeded(parser.parseOptionalKeyword(keyword))) {
+    hasOptional = true;
+    if (parser.parseLParen() || parser.parseOperand(operand) ||
+        parser.resolveOperand(operand, type, result.operands) ||
+        parser.parseRParen())
+      return failure();
+  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// ParallelOp
+//===----------------------------------------------------------------------===//
+
+/// Parse acc.parallel operation
+/// operation := `acc.parallel` `num_gangs(value)?` `num_workers(value)?`
+///                             `vector_length(value)?` `private(value)?`
+///                             `private(value-list)?`
+///                             `firstprivate(value-list)`?
+///                             region attr-dict?
+static ParseResult parseParallelOp(OpAsmParser &parser,
+                                   OperationState &result) {
+  auto &builder = parser.getBuilder();
+  SmallVector<OpAsmParser::OperandType, 8> privateOperands,
+      firstprivateOperands, createOperands, copyinOperands, copyoutOperands;
+  SmallVector<Type, 8> operandTypes;
+  OpAsmParser::OperandType numGangs, numWorkers, vectorLength, ifCond;
+  bool hasNumGangs = false, hasNumWorkers = false, hasVectorLength = false,
+       hasIfCond = false;
+
+  Type indexType = builder.getIndexType();
+  Type i1Type = builder.getI1Type();
+
+  // num_gangs(value)?
+  if (failed(parseOptionalOperand(parser, ParallelOp::getNumGangsKeyword(),
+                                  numGangs, indexType, hasNumGangs, result)))
+    return failure();
+
+  // num_workers(value)?
+  if (failed(parseOptionalOperand(parser, ParallelOp::getNumWorkersKeyword(),
+                                  numWorkers, indexType, hasNumWorkers,
+                                  result)))
+    return failure();
+
+  // vector_length(value)?
+  if (failed(parseOptionalOperand(parser, ParallelOp::getVectorLengthKeyword(),
+                                  vectorLength, indexType, hasVectorLength,
+                                  result)))
+    return failure();
+
+  // private()?
+  if (failed(parseOperandList(parser, ParallelOp::getPrivateKeyword(),
+                              privateOperands, operandTypes, result)))
+    return failure();
+
+  // firstprivate()?
+  if (failed(parseOperandList(parser, ParallelOp::getFirstPrivateKeyword(),
+                              firstprivateOperands, operandTypes, result)))
+    return failure();
+
+  // create()?
+  if (failed(parseOperandList(parser, ParallelOp::getCreateKeyword(),
+                              createOperands, operandTypes, result)))
+    return failure();
+
+  // copyin()?
+  if (failed(parseOperandList(parser, ParallelOp::getCopyinKeyword(),
+                              copyinOperands, operandTypes, result)))
+    return failure();
+
+  // copyout()?
+  if (failed(parseOperandList(parser, ParallelOp::getCopyoutKeyword(),
+                              copyoutOperands, operandTypes, result)))
+    return failure();
+
+  // if()?
+  if (failed(parseOptionalOperand(parser, ParallelOp::getIfKeyword(), ifCond,
+                                  i1Type, hasIfCond, result)))
+    return failure();
+
+  // Parallel op region
+  if (failed(parseRegions<ParallelOp>(parser, result)))
+    return failure();
+
+  result.addAttribute(ParallelOp::getOperandSegmentSizeAttr(),
+                      builder.getI32VectorAttr(
+                          {static_cast<int32_t>(hasNumGangs ? 1 : 0),
+                           static_cast<int32_t>(hasNumWorkers ? 1 : 0),
+                           static_cast<int32_t>(hasVectorLength ? 1 : 0),
+                           static_cast<int32_t>(hasIfCond ? 1 : 0),
+                           static_cast<int32_t>(privateOperands.size()),
+                           static_cast<int32_t>(firstprivateOperands.size()),
+                           static_cast<int32_t>(createOperands.size()),
+                           static_cast<int32_t>(copyinOperands.size()),
+                           static_cast<int32_t>(copyoutOperands.size())}));
+
+  // Additional attributes
+
+  if (failed(parser.parseOptionalAttrDictWithKeyword(result.attributes)))
+    return failure();
+
+  return success();
+}
+
+static void print(OpAsmPrinter &printer, ParallelOp &op) {
+  printer << ParallelOp::getOperationName();
+
+  // num_gangs()?
+  if (auto numGangs = op.numGangs())
+    printer << " " << ParallelOp::getNumGangsKeyword() << "(" << numGangs
+            << ")";
+
+  // num_workers()?
+  if (auto numWorkers = op.numWorkers())
+    printer << " " << ParallelOp::getNumWorkersKeyword() << "(" << numWorkers
+            << ")";
+
+  // private()?
+  printOperandList(op.gangPrivateOperands(), ParallelOp::getPrivateKeyword(),
+                   printer);
+
+  // firstprivate()?
+  printOperandList(op.gangFirstPrivateOperands(),
+                   ParallelOp::getFirstPrivateKeyword(), printer);
+
+  // create()?
+  printOperandList(op.createOperands(), ParallelOp::getCreateKeyword(),
+                   printer);
+
+  // copyin()?
+  printOperandList(op.copyinOperands(), ParallelOp::getCopyinKeyword(),
+                   printer);
+
+  // copyout()?
+  printOperandList(op.copyoutOperands(), ParallelOp::getCopyoutKeyword(),
+                   printer);
+
+  // if()?
+  if (Value ifCond = op.ifCond())
+    printer << " " << ParallelOp::getIfKeyword() << "(" << ifCond << ")";
+
+  printer.printRegion(op.region(),
+                      /*printEntryBlockArgs=*/false,
+                      /*printBlockTerminators=*/false);
+  printer.printOptionalAttrDictWithKeyword(
+      op.getAttrs(), ParallelOp::getOperandSegmentSizeAttr());
+}
+
+//===----------------------------------------------------------------------===//
+// DataOp
+//===----------------------------------------------------------------------===//
+
+/// Parse acc.data operation
+/// operation := `acc.parallel` `present(value-list)?` `copy(value-list)?`
+///                             `copyin(value-list)?` `copyout(value-list)?`
+///                             `create(value-list)?` `no_create(value-list)`?
+///                             `delete(value-list)?` `attach(value-list)`?
+///                             `detach(value-list)?`
+///                             region attr-dict?
+static ParseResult parseDataOp(OpAsmParser &parser, OperationState &result) {
+  auto &builder = parser.getBuilder();
+  SmallVector<OpAsmParser::OperandType, 8> presentOperands, copyOperands,
+      copyinOperands, copyoutOperands, createOperands, noCreateOperands,
+      deleteOperands, attachOperands, detachOperands;
+  SmallVector<Type, 8> operandsTypes;
+
+  // present(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getPresentKeyword(),
+                              presentOperands, operandsTypes, result)))
+    return failure();
+
+  // copy(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getCopyKeyword(), copyOperands,
+                              operandsTypes, result)))
+    return failure();
+
+  // copyin(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getCopyinKeyword(),
+                              copyinOperands, operandsTypes, result)))
+    return failure();
+
+  // copyout(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getCopyoutKeyword(),
+                              copyoutOperands, operandsTypes, result)))
+    return failure();
+
+  // create(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getCreateKeyword(),
+                              createOperands, operandsTypes, result)))
+    return failure();
+
+  // no_create(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getCreateKeyword(),
+                              noCreateOperands, operandsTypes, result)))
+    return failure();
+
+  // delete(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getDeleteKeyword(),
+                              deleteOperands, operandsTypes, result)))
+    return failure();
+
+  // attach(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getAttachKeyword(),
+                              attachOperands, operandsTypes, result)))
+    return failure();
+
+  // detach(value-list)?
+  if (failed(parseOperandList(parser, DataOp::getDetachKeyword(),
+                              detachOperands, operandsTypes, result)))
+    return failure();
+
+  // Data op region
+  if (failed(parseRegions<ParallelOp>(parser, result)))
+    return failure();
+
+  result.addAttribute(
+      ParallelOp::getOperandSegmentSizeAttr(),
+      builder.getI32VectorAttr({static_cast<int32_t>(presentOperands.size()),
+                                static_cast<int32_t>(copyOperands.size()),
+                                static_cast<int32_t>(copyinOperands.size()),
+                                static_cast<int32_t>(copyoutOperands.size()),
+                                static_cast<int32_t>(createOperands.size()),
+                                static_cast<int32_t>(noCreateOperands.size()),
+                                static_cast<int32_t>(deleteOperands.size()),
+                                static_cast<int32_t>(attachOperands.size()),
+                                static_cast<int32_t>(detachOperands.size())}));
+
+  // Additional attributes
+  if (failed(parser.parseOptionalAttrDictWithKeyword(result.attributes)))
+    return failure();
+
+  return success();
+}
+
+static void print(OpAsmPrinter &printer, DataOp &op) {
+  printer << DataOp::getOperationName();
+
+  // present(value-list)?
+  printOperandList(op.presentOperands(), DataOp::getPresentKeyword(), printer);
+
+  // copy(value-list)?
+  printOperandList(op.copyOperands(), DataOp::getCopyKeyword(), printer);
+
+  // copyin(value-list)?
+  printOperandList(op.copyinOperands(), DataOp::getCopyinKeyword(), printer);
+
+  // copyout(value-list)?
+  printOperandList(op.copyoutOperands(), DataOp::getCopyoutKeyword(), printer);
+
+  // create(value-list)?
+  printOperandList(op.createOperands(), DataOp::getCreateKeyword(), printer);
+
+  // no_create(value-list)?
+  printOperandList(op.noCreateOperands(), DataOp::getNoCreateKeyword(),
+                   printer);
+
+  // delete(value-list)?
+  printOperandList(op.deleteOperands(), DataOp::getDeleteKeyword(), printer);
+
+  // attach(value-list)?
+  printOperandList(op.attachOperands(), DataOp::getAttachKeyword(), printer);
+
+  // detach(value-list)?
+  printOperandList(op.detachOperands(), DataOp::getDetachKeyword(), printer);
+
+  printer.printRegion(op.region(),
+                      /*printEntryBlockArgs=*/false,
+                      /*printBlockTerminators=*/false);
+  printer.printOptionalAttrDictWithKeyword(
+      op.getAttrs(), ParallelOp::getOperandSegmentSizeAttr());
+}
+
+//===----------------------------------------------------------------------===//
+// LoopOp
+//===----------------------------------------------------------------------===//
+
+/// Parse acc.loop operation
+/// operation := `acc.loop` `gang?` `vector?` `seq?` `private(value-list)?`
+///                         region attr-dict?
+static ParseResult parseLoopOp(OpAsmParser &parser, OperationState &result) {
+
+  Builder &builder = parser.getBuilder();
+  unsigned executionMapping = 0;
+
+  // gang?
+  if (succeeded(parser.parseOptionalKeyword(LoopOp::getGangAttrName())))
+    executionMapping |= OpenACCExecMapping::GANG;
+
+  // vector?
+  if (succeeded(parser.parseOptionalKeyword(LoopOp::getVectorAttrName())))
+    executionMapping |= OpenACCExecMapping::VECTOR;
+
+  // worker?
+  if (succeeded(parser.parseOptionalKeyword(LoopOp::getWorkerAttrName())))
+    executionMapping |= OpenACCExecMapping::WORKER;
+
+  // seq?
+  if (succeeded(parser.parseOptionalKeyword(LoopOp::getSeqAttrName())))
+    executionMapping |= OpenACCExecMapping::SEQ;
+
+  if (executionMapping != 0)
+    result.addAttribute(LoopOp::getExecutionMappingAttrName(),
+                        builder.getI64IntegerAttr(executionMapping));
+
+  // Parse optional results in case there is a reduce.
+  if (parser.parseOptionalArrowTypeList(result.types))
+    return failure();
+
+  if (failed(parseRegions<LoopOp>(parser, result)))
+    return failure();
+
+  if (failed(parser.parseOptionalAttrDictWithKeyword(result.attributes)))
+    return failure();
+
+  return success();
+}
+
+static void print(OpAsmPrinter &printer, LoopOp &op) {
+  printer << LoopOp::getOperationName();
+  bool printBlockTerminators = false;
+
+  unsigned execMapping =
+      (op.getAttrOfType<IntegerAttr>(LoopOp::getExecutionMappingAttrName()) !=
+       nullptr)
+          ? op.getAttrOfType<IntegerAttr>(LoopOp::getExecutionMappingAttrName())
+                .getInt()
+          : 0;
+  if ((execMapping & OpenACCExecMapping::GANG) == OpenACCExecMapping::GANG)
+    printer << " " << LoopOp::getGangAttrName();
+
+  if ((execMapping & OpenACCExecMapping::WORKER) == OpenACCExecMapping::WORKER)
+    printer << " " << LoopOp::getWorkerAttrName();
+
+  if ((execMapping & OpenACCExecMapping::VECTOR) == OpenACCExecMapping::VECTOR)
+    printer << " " << LoopOp::getVectorAttrName();
+
+  if ((execMapping & OpenACCExecMapping::SEQ) == OpenACCExecMapping::SEQ)
+    printer << " " << LoopOp::getSeqAttrName();
+
+  if (op.getNumResults() > 0) {
+    printer << " -> (" << op.getResultTypes() << ")";
+    printBlockTerminators = true;
+  }
+
+  printer.printRegion(op.region(),
+                      /*printEntryBlockArgs=*/false,
+                      /*printBlockTerminators=*/printBlockTerminators);
+
+  printer.printOptionalAttrDictWithKeyword(
+      op.getAttrs(), {LoopOp::getExecutionMappingAttrName()});
+}
+
+namespace mlir {
+namespace acc {
+#define GET_OP_CLASSES
+#include "mlir/Dialect/OpenACC/OpenACCOps.cpp.inc"
+} // namespace acc
+} // namespace mlir

mlir/test/Dialect/OpenACC/ops.mlir

+// RUN: mlir-opt %s | FileCheck %s
+// Verify the printed output can be parsed.
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+// Verify the generic form can be parsed.
+// RUN: mlir-opt -mlir-print-op-generic %s | mlir-opt | FileCheck %s
+
+func @compute1(%A: memref<10x10xf32>, %B: memref<10x10xf32>, %C: memref<10x10xf32>) -> memref<10x10xf32> {
+  %c0 = constant 0 : index
+  %c10 = constant 10 : index
+  %c1 = constant 1 : index
+
+  acc.parallel {
+    acc.loop gang vector {
+      scf.for %arg3 = %c0 to %c10 step %c1 {
+        scf.for %arg4 = %c0 to %c10 step %c1 {
+          scf.for %arg5 = %c0 to %c10 step %c1 {
+            %a = load %A[%arg3, %arg5] : memref<10x10xf32>
+            %b = load %B[%arg5, %arg4] : memref<10x10xf32>
+            %cij = load %C[%arg3, %arg4] : memref<10x10xf32>
+            %p = mulf %a, %b : f32
+            %co = addf %cij, %p : f32
+            store %co, %C[%arg3, %arg4] : memref<10x10xf32>
+          }
+        }
+      }
+    } attributes { collapse = 3 }
+  } attributes { async = 1 }
+
+  return %C : memref<10x10xf32>
+}
+
+// CHECK-LABEL: func @compute1(
+//  CHECK-NEXT:   %{{.*}} = constant 0 : index
+//  CHECK-NEXT:   %{{.*}} = constant 10 : index
+//  CHECK-NEXT:   %{{.*}} = constant 1 : index
+//  CHECK-NEXT:   acc.parallel {
+//  CHECK-NEXT:     acc.loop gang vector {
+//  CHECK-NEXT:       scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:         scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:           scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = mulf %{{.*}}, %{{.*}} : f32
+//  CHECK-NEXT:             %{{.*}} = addf %{{.*}}, %{{.*}} : f32
+//  CHECK-NEXT:             store %{{.*}}, %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:           }
+//  CHECK-NEXT:         }
+//  CHECK-NEXT:       }
+//  CHECK-NEXT:     } attributes {collapse = 3 : i64}
+//  CHECK-NEXT:   } attributes {async = 1 : i64}
+//  CHECK-NEXT:   return %{{.*}} : memref<10x10xf32>
+//  CHECK-NEXT: }
+
+func @compute2(%A: memref<10x10xf32>, %B: memref<10x10xf32>, %C: memref<10x10xf32>) -> memref<10x10xf32> {
+  %c0 = constant 0 : index
+  %c10 = constant 10 : index
+  %c1 = constant 1 : index
+
+  acc.parallel {
+    acc.loop seq {
+      scf.for %arg3 = %c0 to %c10 step %c1 {
+        scf.for %arg4 = %c0 to %c10 step %c1 {
+          scf.for %arg5 = %c0 to %c10 step %c1 {
+            %a = load %A[%arg3, %arg5] : memref<10x10xf32>
+            %b = load %B[%arg5, %arg4] : memref<10x10xf32>
+            %cij = load %C[%arg3, %arg4] : memref<10x10xf32>
+            %p = mulf %a, %b : f32
+            %co = addf %cij, %p : f32
+            store %co, %C[%arg3, %arg4] : memref<10x10xf32>
+          }
+        }
+      }
+    }
+  }
+
+  return %C : memref<10x10xf32>
+}
+
+// CHECK-LABEL: func @compute2(
+//  CHECK-NEXT:   %{{.*}} = constant 0 : index
+//  CHECK-NEXT:   %{{.*}} = constant 10 : index
+//  CHECK-NEXT:   %{{.*}} = constant 1 : index
+//  CHECK-NEXT:   acc.parallel {
+//  CHECK-NEXT:     acc.loop seq {
+//  CHECK-NEXT:       scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:         scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:           scf.for %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} {
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:             %{{.*}} = mulf %{{.*}}, %{{.*}} : f32
+//  CHECK-NEXT:             %{{.*}} = addf %{{.*}}, %{{.*}} : f32
+//  CHECK-NEXT:             store %{{.*}}, %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+//  CHECK-NEXT:           }
+//  CHECK-NEXT:         }
+//  CHECK-NEXT:       }
+//  CHECK-NEXT:     }
+//  CHECK-NEXT:   }
+//  CHECK-NEXT:   return %{{.*}} : memref<10x10xf32>
+//  CHECK-NEXT: }
+
+
+func @compute3(%a: memref<10x10xf32>, %b: memref<10x10xf32>, %c: memref<10xf32>, %d: memref<10xf32>) -> memref<10xf32> {
+  %lb = constant 0 : index
+  %st = constant 1 : index
+  %c10 = constant 10 : index
+
+  acc.parallel num_gangs(%c10) num_workers(%c10) private(%c : memref<10xf32>) {
+    acc.loop gang {
+      scf.for %x = %lb to %c10 step %st {
+        acc.loop worker {
+          scf.for %y = %lb to %c10 step %st {
+            %axy = load %a[%x, %y] : memref<10x10xf32>
+            %bxy = load %b[%x, %y] : memref<10x10xf32>
+            %tmp = addf %axy, %bxy : f32
+            store %tmp, %c[%y] : memref<10xf32>
+          }
+        }
+
+        acc.loop seq {
+          // for i = 0 to 10 step 1
+          //   d[x] += c[i]
+          scf.for %i = %lb to %c10 step %st {
+            %ci = load %c[%i] : memref<10xf32>
+            %dx = load %d[%x] : memref<10xf32>
+            %z = addf %ci, %dx : f32
+            store %z, %d[%x] : memref<10xf32>
+          }
+        }
+      }
+    }
+  }
+
+  return %d : memref<10xf32>
+}
+
+// CHECK:      func @compute3({{.*}}: memref<10x10xf32>, {{.*}}: memref<10x10xf32>, [[ARG2:%.*]]: memref<10xf32>, {{.*}}: memref<10xf32>) -> memref<10xf32> {
+// CHECK-NEXT:   [[C0:%.*]] = constant 0 : index
+// CHECK-NEXT:   [[C1:%.*]] = constant 1 : index
+// CHECK-NEXT:   [[C10:%.*]] = constant 10 : index
+// CHECK-NEXT:   acc.parallel num_gangs([[C10]]) num_workers([[C10]]) private([[ARG2]]: memref<10xf32>) {
+// CHECK-NEXT:     acc.loop gang {
+// CHECK-NEXT:       scf.for %{{.*}} = [[C0]] to [[C10]] step [[C1]] {
+// CHECK-NEXT:         acc.loop worker {
+// CHECK-NEXT:           scf.for %{{.*}} = [[C0]] to [[C10]] step [[C1]] {
+// CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+// CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}, %{{.*}}] : memref<10x10xf32>
+// CHECK-NEXT:             %{{.*}} = addf %{{.*}}, %{{.*}} : f32
+// CHECK-NEXT:             store %{{.*}}, %{{.*}}[%{{.*}}] : memref<10xf32>
+// CHECK-NEXT:           }
+// CHECK-NEXT:         }
+// CHECK-NEXT:         acc.loop seq {
+// CHECK-NEXT:           scf.for %{{.*}} = [[C0]] to [[C10]] step [[C1]] {
+// CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}] : memref<10xf32>
+// CHECK-NEXT:             %{{.*}} = load %{{.*}}[%{{.*}}] : memref<10xf32>
+// CHECK-NEXT:             %{{.*}} = addf %{{.*}}, %{{.*}} : f32
+// CHECK-NEXT:             store %{{.*}}, %{{.*}}[%{{.*}}] : memref<10xf32>
+// CHECK-NEXT:           }
+// CHECK-NEXT:         }
+// CHECK-NEXT:       }
+// CHECK-NEXT:     }
+// CHECK-NEXT:   }
+// CHECK-NEXT:   return %{{.*}} : memref<10xf32>
+// CHECK-NEXT: }
+