[mlir] Add loop.parallel, loop.reduce and loop.reduce.return operations.

[akuegel]

Summary

These operations can be used to specify a loop nest with a body that can contain reductions.
The iteration space can be iterated in any order.

RFC: https://groups.google.com/a/tensorflow.org/d/topic/mlir/pwtSgiKFPis/discussion

[merge_guards_bot]

Unit tests: pass. 61132 tests passed, 0 failed and 728 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[akuegel]

Rebased to head.

[merge_guards_bot]

Unit tests: pass. 61306 tests passed, 0 failed and 736 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[akuegel]

Improved description of parallel loop operation.

[akuegel]

Also add description for loop.yield (thanks herhut@)

[merge_guards_bot]

Unit tests: pass. 61312 tests passed, 0 failed and 736 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[merge_guards_bot]

Unit tests: pass. 61312 tests passed, 0 failed and 736 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[rriddle]

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

168

These should all be wrapped at 80 characters.

173

Keep these operations sorted alphabetically.

211

Use:

mlir

mlir/lib/Dialect/LoopOps/LoopOps.cpp

231

Drop the const &

nit: auto -> Value

233

Don't use -> or * with values.

246

nit: auto -> Block

251

Drop the &.

252

Same here, drop all of the -> and *

332

Stream in the results directly:
<< op.getResultTypes()

362

Drop extra newline

416

Stream the result and the type.

[herhut]

We pair programmed some of this code, so someone else needs to review. Obviously, I am happy for this to land :)

[akuegel]

Addressed review comments.

[akuegel]

Fix copy/paste issue.

[merge_guards_bot]

Unit tests: pass. 61338 tests passed, 0 failed and 738 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[merge_guards_bot]

Unit tests: pass. 61673 tests passed, 0 failed and 779 were skipped.

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[rriddle]

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

171

Can you format this a bit better?

let arguments = (ins Variadic...,

Variadic...,
Variadic...);

290

This looks like it needs to be formatted.

297

nit: Can we name this something other than 'operation'? Seems like this is likely to lead to confusion in the future. e.g. computation or body would achieve the same effect.

mlir/lib/Dialect/LoopOps/LoopOps.cpp

240

I would imagine that we would check this before the above loop.

262

Use camelCase for variable names.

341

Drop the -> here.

342

I don't see a check for empty block, can you add one.

383

Drop -> here. Same in a few other places below.

[nicolasvasilache]

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

152

typo: individual
typo2: modeled

163

Please add a few more examples that also illustrate reduction and the yield ops.

261

Do we foresee another use case for this than reduction?
If not, does it make sense to call this a ReductionOp or a ReduceOp?

290

can we make this example a bit more realistic?
Unless I am misunderstanding, there should be a reduction operation in the region?

[nicolasvasilache]

mlir/test/Dialect/Loops/invalid.mlir

130

Here and below: we usually omit the checks that are automatically tablegen'd.

[akuegel]

Address review comments.

[akuegel]

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

163

I extended the example so that it also has a loop.reduce inside.

171

I hope I understood your suggestion correctly. I have now indented it such that the words Variadic align.

261

I renamed it to ReduceOp, and accordingly the other op to ReduceReturnOp instead of YieldReturnOp.
Also, the textual format is now loop.reduce and loop.reduce.return.

290

My editor decided it should add a tab when I hit carriage return. Visually it looked good in my editor, but of course I should use spaces here for indentation :)

297

What about reductionOperator?

mlir/lib/Dialect/LoopOps/LoopOps.cpp

342

I think this is automatically checked because it is defined as a SizedRegion. See the test yield_no_block in invalid.mlir
But as far as I understand Nicolas' comment, I actually should remove that test, too (leaving it here for now for verification if maybe I misunderstood something).

[merge_guards_bot]

Unit tests: pass. 61744 tests passed, 0 failed and 780 were skipped.

clang-tidy: pass.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[akuegel]

Ran git-clang-format.

[merge_guards_bot]

Unit tests: pass. 61744 tests passed, 0 failed and 780 were skipped.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[rriddle]

mlir/lib/Dialect/LoopOps/LoopOps.cpp

332

nit: Remove trivial braces

342

SizedRegion means the number of blocks, it doesn't constrain the number of operations within the blocks. e.g. an empty block would crash the isa<ReduceReturnOp> check below.

359

Still seems like there is a newline here.

[nicolasvasilache]

LGTM once River's comments are addressed, thanks a lot for pushing this!

[akuegel]

Addressed review comments.

[akuegel]

mlir/lib/Dialect/LoopOps/LoopOps.cpp

342

Ah, I mixed up those two. Thanks for explaining :)
I have now added a check for this as well, and modified the test to not test for no block, but instead for whether the block is empty.

[merge_guards_bot]

Unit tests: pass. 61776 tests passed, 0 failed and 780 were skipped.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[rriddle]

mlir/lib/Dialect/LoopOps/LoopOps.cpp

348

nit: You can use emitOpError to have the message prefixed with the operation name: 'foo.op' op <your-message>

353

nit: use lower case to start the message

[akuegel]

Addressed River's review comments.

[merge_guards_bot]

Unit tests: pass. 61801 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[akuegel]

I fixed the clang-format error in a followup commit.
Sorry for not noticing it sooner :-(

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

let description = [{
     The lower and upper bounds specify a half-open range: the range includes the
     lower bound but does not include the upper bound.
 
     The body region must contain exactly one block that terminates with
     "loop.terminator".  Calling ForOp::build will create such region and insert
     the terminator, so will the parsing even in cases when it is absent from the
     custom format. For example:
 
+    ```mlir
        loop.for %iv = %lb to %ub step %step {
          ... // body
        }
+    ```
   }];
   let arguments = (ins Index:$lowerBound, Index:$upperBound, Index:$step);
   let regions = (region SizedRegion<1>:$region);
 
   let skipDefaultBuilders = 1;
   let builders = [
     OpBuilder<"Builder *builder, OperationState &result, "
               "Value lowerBound, Value upperBound, Value step">

 def IfOp : Loop_Op<"if",
       [SingleBlockImplicitTerminator<"TerminatorOp">]> {
   let summary = "if-then-else operation";
   let description = [{
     The "loop.if" operation represents an if-then-else construct for
     conditionally executing two regions of code. The operand to an if operation
     is a boolean value. The operation produces no results. For example:
 
+    ```mlir
        loop.if %b  {
          ...
        } else {
          ...
        }
+    ```
 
     The 'else' block is optional, and may be omitted. For
     example:
 
+    ```mlir
        loop.if %b  {
          ...
        }
+    ```
   }];
   let arguments = (ins I1:$condition);
   let regions = (region SizedRegion<1>:$thenRegion, AnyRegion:$elseRegion);
 
   let skipDefaultBuilders = 1;
   let builders = [
     OpBuilder<"Builder *builder, OperationState &result, "
               "Value cond, bool withElseRegion">
   ];
 
   let extraClassDeclaration = [{
     OpBuilder getThenBodyBuilder() {
       assert(!thenRegion().empty() && "Unexpected empty 'then' region.");
       Block &body = thenRegion().front();
       return OpBuilder(&body, std::prev(body.end()));
     }
     OpBuilder getElseBodyBuilder() {
       assert(!elseRegion().empty() && "Unexpected empty 'else' region.");
       Block &body = elseRegion().front();
       return OpBuilder(&body, std::prev(body.end()));
     }
   }];
 }
 
-def TerminatorOp :
-    Loop_Op<"terminator", [NativeOpTrait<"IsTerminator">]> {
+def ParallelOp : Loop_Op<"parallel",
+    [SameVariadicOperandSize, SingleBlockImplicitTerminator<"TerminatorOp">]> {
+  let summary = "parallel for operation";
+  let description = [{
+    The "loop.parallel" operation represents a loop nest taking 3 groups of SSA
+    values as operands that represent the lower bounds, upper bounds and steps,
+    respectively. The operation defines a variadic number of SSA values for its
+    induction variables. It has one region capturing the loop body. The
+    induction variables are represented as an argument of this region. These SSA
+    values always have type index, which is the size of the machine word. The
+    steps are values of type index, required to be positive.
+    The lower and upper bounds specify a half-open range: the range includes the
+    lower bound but does not include the upper bound.
+
+    Semantically we require that the iteration space can be iterated in any
+    order, and the loop body can be executed in parallel. If there are data
+    races, the behavior is undefined.
+
+    The parallel loop operation supports reduction of values produced by
+    individual iterations into a single result. This is modeled using the
+    loop.reduce operation (see loop.reduce for details). Each result of a
+    loop.parallel operation is associated with a reduce operation that is an
+    immediate child. Reduces are matched to result values in order of their
+    appearance in the body. Consequently, we require that the body region has
+    the same number of results as it has reduce operations.
+
+    The body region must contain exactly one block that terminates with
+    "loop.terminator". Parsing ParallelOp will create such region and insert the
+    terminator when it is absent from the custom format. For example:
+
+    ```mlir
+       loop.parallel (%iv) = (%lb) to (%ub) step (%step) {
+         %zero = constant 0.0 : f32
+         loop.reduce(%zero) {
+           ^bb0(%lhs : f32, %rhs: f32):
+             %res = addf %lhs, %rhs : f32
+             loop.reduce.return %res : f32
+         } : f32
+       }
+    ```
+   }];
+
+  let arguments = (ins Variadic<Index>:$lowerBound,
+                       Variadic<Index>:$upperBound,
+                       Variadic<Index>:$step);
+  let results = (outs Variadic<AnyType>:$results);
+  let regions = (region SizedRegion<1>:$body);
+}
+
+def ReduceOp : Loop_Op<"reduce", [HasParent<"ParallelOp">]> {
+  let summary = "reduce operation for parallel for";
+  let description = [{
+    "loop.reduce" is an operation occuring inside "loop.parallel" operations. It
+    consists of one block with two arguments which have the same type as the
+    operand of "loop.reduce".
+
+    "loop.reduce" is used to model the value for reduction computations of a
+    "loop.parallel" operation. It has to appear as an immediate child of a
+    "loop.parallel" and is associated with a result value of its parent
+    operation.
+
+    Association is in the order of appearance in the body where the first result
+    of a parallel loop operation corresponds to the first "loop.reduce" in the
+    operation's body region. The reduce operation takes a single operand, which
+    is the value to be used in the reduction.
+
+    The reduce operation contains a region whose entry block expects two
+    arguments of the same type as the operand. As the iteration order of the
+    parallel loop and hence reduction order is unspecified, the result of
+    reduction may be non-deterministic unless the operation is associative and
+    commutative.
+
+    The result of the reduce operation's body must have the same type as the
+    operands and associated result value of the parallel loop operation.
+    Example:
+
+    ```mlir
+      %zero = constant 0.0 : f32
+      loop.reduce(%zero) {
+        ^bb0(%lhs : f32, %rhs: f32):
+          %res = addf %lhs, %rhs : f32
+          loop.reduce.return %res : f32
+      } : f32
+    ```
+
+  }];
+
+  let arguments = (ins AnyType:$operand);
+  let regions = (region SizedRegion<1>:$reductionOperator);
+}
+
+def ReduceReturnOp :
+    Loop_Op<"reduce.return", [HasParent<"ReduceOp">, Terminator]> {
+  let summary = "terminator for reduce operation";
+  let description = [{
+    "loop.reduce.return" is a special terminator operation for the block inside
+    "loop.reduce". It terminates the region. It should have the same type as the
+    operand of "loop.reduce". Example for the custom format:
+
+    ```mlir
+      loop.reduce.return %res : f32
+    ```
+  }];
+
+  let arguments = (ins AnyType:$result);
+}
+
+def TerminatorOp : Loop_Op<"terminator", [Terminator]> {
   let summary = "cf terminator operation";
   let description = [{
     "loop.terminator" is a special terminator operation for blocks inside
     loops. It terminates the region. This operation does _not_ have a custom
     syntax. However, `std` control operations omit the terminator in their
     custom syntax for brevity.
 
+    ```mlir
        loop.terminator
+    ```
   }];
 
   // No custom parsing/printing form.
   let parser = ?;
   let printer = ?;
 
   // Fully specified by traits.
   let verifier = ?;
 }
 
 #endif // LOOP_OPS

mlir/lib/Dialect/LoopOps/LoopOps.cpp

static ParseResult parseIfOp(OpAsmParser &parser, OperationState &result) {
   auto &builder = parser.getBuilder();
   OpAsmParser::OperandType cond;
   Type i1Type = builder.getIntegerType(1);
   if (parser.parseOperand(cond) ||
       parser.resolveOperand(cond, i1Type, result.operands))
     return failure();
 
   // Parse the 'then' region.
-  if (parser.parseRegion(*thenRegion, {}, {}))
+  if (parser.parseRegion(*thenRegion, /*arguments=*/{}, /*argTypes=*/{}))
     return failure();
   IfOp::ensureTerminator(*thenRegion, parser.getBuilder(), result.location);
 
   // If we find an 'else' keyword then parse the 'else' region.
   if (!parser.parseOptionalKeyword("else")) {
-    if (parser.parseRegion(*elseRegion, {}, {}))
+    if (parser.parseRegion(*elseRegion, /*arguments=*/{}, /*argTypes=*/{}))
       return failure();
     IfOp::ensureTerminator(*elseRegion, parser.getBuilder(), result.location);
   }
 
   // Parse the optional attribute list.
   if (parser.parseOptionalAttrDict(result.attributes))
     return failure();
 
p.printRegion(elseRegion,
                   /*printEntryBlockArgs=*/false,
                   /*printBlockTerminators=*/false);
   }
 
   p.printOptionalAttrDict(op.getAttrs());
 }
 
 //===----------------------------------------------------------------------===//
+// ParallelOp
+//===----------------------------------------------------------------------===//
+
+static LogicalResult verify(ParallelOp op) {
+  // Check that there is at least one value in lowerBound, upperBound and step.
+  // It is sufficient to test only step, because it is ensured already that the
+  // number of elements in lowerBound, upperBound and step are the same.
+  Operation::operand_range stepValues = op.step();
+  if (stepValues.empty())
+    return op.emitOpError(
+        "needs at least one tuple element for lowerBound, upperBound and step");
+
+  // Check whether all constant step values are positive.
+  for (Value stepValue : stepValues)
+    if (auto cst = dyn_cast_or_null<ConstantIndexOp>(stepValue.getDefiningOp()))
+      if (cst.getValue() <= 0)
+        return op.emitOpError("constant step operand must be positive");
+
+  // Check that the body defines the same number of block arguments as the
+  // number of tuple elements in step.
+  Block *body = &op.body().front();
+  if (body->getNumArguments() != stepValues.size())
+    return op.emitOpError(
+        "expects the same number of induction variables as bound and step "
+        "values");
+  for (auto arg : body->getArguments())
+    if (!arg.getType().isIndex())
+      return op.emitOpError(
+          "expects arguments for the induction variable to be of index type");
+
+  // Check that the number of results is the same as the number of ReduceOps.
+  SmallVector<ReduceOp, 4> reductions(body->getOps<ReduceOp>());
+  if (op.results().size() != reductions.size())
+    return op.emitOpError(
+        "expects number of results to be the same as number of reductions");
+
+  // Check that the types of the results and reductions are the same.
+  for (auto resultAndReduce : llvm::zip(op.results(), reductions)) {
+    auto resultType = std::get<0>(resultAndReduce).getType();
+    auto reduceOp = std::get<1>(resultAndReduce);
+    auto reduceType = reduceOp.operand().getType();
+    if (resultType != reduceType)
+      return reduceOp.emitOpError()
+             << "expects type of reduce to be the same as result type: "
+             << resultType;
+  }
+  return success();
+}
+
+static ParseResult parseParallelOp(OpAsmParser &parser,
+                                   OperationState &result) {
+  auto &builder = parser.getBuilder();
+  // Parse an opening `(` followed by induction variables followed by `)`
+  SmallVector<OpAsmParser::OperandType, 4> ivs;
+  if (parser.parseRegionArgumentList(ivs, /*requiredOperandCount=*/-1,
+                                     OpAsmParser::Delimiter::Paren))
+    return failure();
+
+  // Parse loop bounds.
+  SmallVector<OpAsmParser::OperandType, 4> lower;
+  if (parser.parseEqual() ||
+      parser.parseOperandList(lower, ivs.size(),
+                              OpAsmParser::Delimiter::Paren) ||
+      parser.resolveOperands(lower, builder.getIndexType(), result.operands))
+    return failure();
+
+  SmallVector<OpAsmParser::OperandType, 4> upper;
+  if (parser.parseKeyword("to") ||
+      parser.parseOperandList(upper, ivs.size(),
+                              OpAsmParser::Delimiter::Paren) ||
+      parser.resolveOperands(upper, builder.getIndexType(), result.operands))
+    return failure();
+
+  // Parse step value.
+  SmallVector<OpAsmParser::OperandType, 4> steps;
+  if (parser.parseKeyword("step") ||
+      parser.parseOperandList(steps, ivs.size(),
+                              OpAsmParser::Delimiter::Paren) ||
+      parser.resolveOperands(steps, builder.getIndexType(), result.operands))
+    return failure();
+
+  // Now parse the body.
+  Region *body = result.addRegion();
+  SmallVector<Type, 4> types(ivs.size(), builder.getIndexType());
+  if (parser.parseRegion(*body, ivs, types))
+    return failure();
+
+  // Parse attributes and optional results (in case there is a reduce).
+  if (parser.parseOptionalAttrDict(result.attributes) ||
+      parser.parseOptionalColonTypeList(result.types))
+    return failure();
+
+  // Add a terminator if none was parsed.
+  ForOp::ensureTerminator(*body, builder, result.location);
+
+  return success();
+}
+
+static void print(OpAsmPrinter &p, ParallelOp op) {
+  p << op.getOperationName() << " (";
+  p.printOperands(op.body().front().getArguments());
+  p << ") = (" << op.lowerBound() << ") to (" << op.upperBound() << ") step ("
+    << op.step() << ")";
+  p.printRegion(op.body(), /*printEntryBlockArgs=*/false);
+  p.printOptionalAttrDict(op.getAttrs());
+  if (!op.results().empty())
+    p << " : " << op.getResultTypes();
+}
+
+//===----------------------------------------------------------------------===//
+// ReduceOp
+//===----------------------------------------------------------------------===//
+
+static LogicalResult verify(ReduceOp op) {
+  // The region of a ReduceOp has two arguments of the same type as its operand.
+  auto type = op.operand().getType();
+  Block &block = op.reductionOperator().front();
+  if (block.empty())
+    return op.emitOpError("the block inside reduce should not be empty");
+  if (block.getNumArguments() != 2 ||
+      llvm::any_of(block.getArguments(), [&](const BlockArgument &arg) {
+        return arg.getType() != type;
+      }))
+    return op.emitOpError() << "expects two arguments to reduce block of type "
+                            << type;
+
+  // Check that the block is terminated by a ReduceReturnOp.
+  if (!isa<ReduceReturnOp>(block.getTerminator()))
+    return op.emitOpError("the block inside reduce should be terminated with a "
+                          "'loop.reduce.return' op");
+
+  return success();
+}
+
+static ParseResult parseReduceOp(OpAsmParser &parser, OperationState &result) {
+  // Parse an opening `(` followed by the reduced value followed by `)`
+  OpAsmParser::OperandType operand;
+  if (parser.parseLParen() || parser.parseOperand(operand) ||
+      parser.parseRParen())
+    return failure();
+
+  // Now parse the body.
+  Region *body = result.addRegion();
+  if (parser.parseRegion(*body, /*arguments=*/{}, /*argTypes=*/{}))
+    return failure();
+
+  // And the type of the operand (and also what reduce computes on).
+  Type resultType;
+  if (parser.parseColonType(resultType) ||
+      parser.resolveOperand(operand, resultType, result.operands))
+    return failure();
+
+  return success();
+}
+
+static void print(OpAsmPrinter &p, ReduceOp op) {
+  p << op.getOperationName() << "(" << op.operand() << ") ";
+  p.printRegion(op.reductionOperator());
+  p << " : " << op.operand().getType();
+}
+
+//===----------------------------------------------------------------------===//
+// ReduceReturnOp
+//===----------------------------------------------------------------------===//
+
+static LogicalResult verify(ReduceReturnOp op) {
+  // The type of the return value should be the same type as the type of the
+  // operand of the enclosing ReduceOp.
+  auto reduceOp = cast<ReduceOp>(op.getParentOp());
+  Type reduceType = reduceOp.operand().getType();
+  if (reduceType != op.result().getType())
+    return op.emitOpError() << "needs to have type " << reduceType
+                            << " (the type of the enclosing ReduceOp)";
+  return success();
+}
+
+static ParseResult parseReduceReturnOp(OpAsmParser &parser,
+                                       OperationState &result) {
+  OpAsmParser::OperandType operand;
+  Type resultType;
+  if (parser.parseOperand(operand) || parser.parseColonType(resultType) ||
+      parser.resolveOperand(operand, resultType, result.operands))
+    return failure();
+
+  return success();
+}
+
+static void print(OpAsmPrinter &p, ReduceReturnOp op) {
+  p << op.getOperationName() << " " << op.result() << " : "
+    << op.result().getType();
+}
+
+//===----------------------------------------------------------------------===//
 // TableGen'd op method definitions
 //===----------------------------------------------------------------------===//
 
 #define GET_OP_CLASSES
 #include "mlir/Dialect/LoopOps/LoopOps.cpp.inc"

mlir/test/Dialect/Loops/invalid.mlir

func @loop_if_illegal_block_argument(%arg0: i1) {
   // expected-error@+1 {{requires that child entry blocks have no arguments}}
   "loop.if"(%arg0) ({
     ^bb0(%0 : index):
       "loop.terminator"() : () -> ()
   }, {}): (i1) -> ()
   return
 }
 
+// -----
+
+func @parallel_arguments_different_tuple_size(
+    %arg0: index, %arg1: index, %arg2: index) {
+  // expected-error@+1 {{custom op 'loop.parallel' expected 1 operands}}
+  loop.parallel (%i0) = (%arg0) to (%arg1, %arg2) step () {
+  }
+  return
+}
+
+// -----
+
+func @parallel_body_arguments_wrong_type(
+    %arg0: index, %arg1: index, %arg2: index) {
+  // expected-error@+1 {{'loop.parallel' op expects arguments for the induction variable to be of index type}}
+  "loop.parallel"(%arg0, %arg1, %arg2) ({
+    ^bb0(%i0: f32):
+      "loop.terminator"() : () -> ()
+  }): (index, index, index) -> ()
+  return
+}
+
+// -----
+
+func @parallel_body_wrong_number_of_arguments(
+    %arg0: index, %arg1: index, %arg2: index) {
+  // expected-error@+1 {{'loop.parallel' op expects the same number of induction variables as bound and step values}}
+  "loop.parallel"(%arg0, %arg1, %arg2) ({
+    ^bb0(%i0: index, %i1: index):
+      "loop.terminator"() : () -> ()
+  }): (index, index, index) -> ()
+  return
+}
+
+// -----
+
+func @parallel_no_tuple_elements() {
+  // expected-error@+1 {{'loop.parallel' op needs at least one tuple element for lowerBound, upperBound and step}}
+  loop.parallel () = () to () step () {
+  }
+  return
+}
+
+// -----
+
+func @parallel_step_not_positive(
+    %arg0: index, %arg1: index, %arg2: index, %arg3: index) {
+  // expected-error@+3 {{constant step operand must be positive}}
+  %c0 = constant 1 : index
+  %c1 = constant 0 : index
+  loop.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3) step (%c0, %c1) {
+  }
+  return
+}
+
+// -----
+
+func @parallel_fewer_results_than_reduces(
+    %arg0 : index, %arg1: index, %arg2: index) {
+  // expected-error@+1 {{expects number of results to be the same as number of reductions}}
+  loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
+    %c0 = constant 1.0 : f32
+    loop.reduce(%c0) {
+      ^bb0(%lhs: f32, %rhs: f32):
+        loop.reduce.return %lhs : f32
+    } : f32
+  }
+  return
+}
+
+// -----
+
+func @parallel_more_results_than_reduces(
+    %arg0 : index, %arg1 : index, %arg2 : index) {
+  // expected-error@+1 {{expects number of results to be the same as number of reductions}}
+  %res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
+  } : f32
+
+  return
+}
+
+// -----
+
+func @parallel_different_types_of_results_and_reduces(
+    %arg0 : index, %arg1: index, %arg2: index) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg1) step (%arg2) {
+    // expected-error@+1 {{expects type of reduce to be the same as result type: 'f32'}}
+    loop.reduce(%arg0) {
+      ^bb0(%lhs: index, %rhs: index):
+        loop.reduce.return %lhs : index
+    } : index
+  } : f32
+  return
+}
+
+// -----
+
+func @top_level_reduce(%arg0 : f32) {
+  // expected-error@+1 {{expects parent op 'loop.parallel'}}
+  loop.reduce(%arg0) {
+    ^bb0(%lhs : f32, %rhs : f32):
+      loop.reduce.return %lhs : f32
+  } : f32
+  return
+}
+
+// -----
+
+func @reduce_empty_block(%arg0 : index, %arg1 : f32) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+    // expected-error@+1 {{the block inside reduce should not be empty}}
+    loop.reduce(%arg1) {
+      ^bb0(%lhs : f32, %rhs : f32):
+    } : f32
+  } : f32
+  return
+}
+
+// -----
+
+func @reduce_too_many_args(%arg0 : index, %arg1 : f32) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+    // expected-error@+1 {{expects two arguments to reduce block of type 'f32'}}
+    loop.reduce(%arg1) {
+      ^bb0(%lhs : f32, %rhs : f32, %other : f32):
+        loop.reduce.return %lhs : f32
+    } : f32
+  } : f32
+  return
+}
+
+// -----
+
+func @reduce_wrong_args(%arg0 : index, %arg1 : f32) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+    // expected-error@+1 {{expects two arguments to reduce block of type 'f32'}}
+    loop.reduce(%arg1) {
+      ^bb0(%lhs : f32, %rhs : i32):
+        loop.reduce.return %lhs : f32
+    } : f32
+  } : f32
+  return
+}
+
+
+// -----
+
+func @reduce_wrong_terminator(%arg0 : index, %arg1 : f32) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+    // expected-error@+1 {{the block inside reduce should be terminated with a 'loop.reduce.return' op}}
+    loop.reduce(%arg1) {
+      ^bb0(%lhs : f32, %rhs : f32):
+        "loop.terminator"(): () -> ()
+    } : f32
+  } : f32
+  return
+}
+
+// -----
+
+func @reduceReturn_wrong_type(%arg0 : index, %arg1: f32) {
+  %res = loop.parallel (%i0) = (%arg0) to (%arg0) step (%arg0) {
+    loop.reduce(%arg1) {
+      ^bb0(%lhs : f32, %rhs : f32):
+        %c0 = constant 1 : index
+        // expected-error@+1 {{needs to have type 'f32' (the type of the enclosing ReduceOp)}}
+        loop.reduce.return %c0 : index
+    } : f32
+  } : f32
+  return
+}
+
+// -----
+
+func @reduceReturn_not_inside_reduce(%arg0 : f32) {
+  "foo.region"() ({
+    // expected-error@+1 {{expects parent op 'loop.reduce'}}
+    loop.reduce.return %arg0 : f32
+  }): () -> ()
+  return
+}

mlir/test/Dialect/Loops/ops.mlir

func @std_if_else(%arg0: i1, %arg1: f32) {
   }
   return
 }
 // CHECK-LABEL: func @std_if_else(
 //  CHECK-NEXT:   loop.if %{{.*}} {
 //  CHECK-NEXT:     %{{.*}} = addf %{{.*}}, %{{.*}} : f32
 //  CHECK-NEXT:   } else {
 //  CHECK-NEXT:     %{{.*}} = addf %{{.*}}, %{{.*}} : f32
+
+func @std_parallel_loop(%arg0 : index, %arg1 : index, %arg2 : index,
+                        %arg3 : index, %arg4 : index) {
+  %step = constant 1 : index
+  loop.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
+                                          step (%arg4, %step) {
+    %min_cmp = cmpi "slt", %i0, %i1 : index
+    %min = select %min_cmp, %i0, %i1 : index
+    %max_cmp = cmpi "sge", %i0, %i1 : index
+    %max = select %max_cmp, %i0, %i1 : index
+    %red = loop.parallel (%i2) = (%min) to (%max) step (%i1) {
+      %zero = constant 0.0 : f32
+      loop.reduce(%zero) {
+        ^bb0(%lhs : f32, %rhs: f32):
+          %res = addf %lhs, %rhs : f32
+          loop.reduce.return %res : f32
+      } : f32
+    } : f32
+  }
+  return
+}
+// CHECK-LABEL: func @std_parallel_loop(
+//  CHECK-SAME: %[[ARG0:[A-Za-z0-9]+]]:
+//  CHECK-SAME: %[[ARG1:[A-Za-z0-9]+]]:
+//  CHECK-SAME: %[[ARG2:[A-Za-z0-9]+]]:
+//  CHECK-SAME: %[[ARG3:[A-Za-z0-9]+]]:
+//  CHECK-SAME: %[[ARG4:[A-Za-z0-9]+]]:
+//       CHECK:   %[[STEP:.*]] = constant 1 : index
+//  CHECK-NEXT:   loop.parallel (%[[I0:.*]], %[[I1:.*]]) = (%[[ARG0]], %[[ARG1]]) to
+//       CHECK:   (%[[ARG2]], %[[ARG3]]) step (%[[ARG4]], %[[STEP]]) {
+//  CHECK-NEXT:     %[[MIN_CMP:.*]] = cmpi "slt", %[[I0]], %[[I1]] : index
+//  CHECK-NEXT:     %[[MIN:.*]] = select %[[MIN_CMP]], %[[I0]], %[[I1]] : index
+//  CHECK-NEXT:     %[[MAX_CMP:.*]] = cmpi "sge", %[[I0]], %[[I1]] : index
+//  CHECK-NEXT:     %[[MAX:.*]] = select %[[MAX_CMP]], %[[I0]], %[[I1]] : index
+//  CHECK-NEXT:     loop.parallel (%{{.*}}) = (%[[MIN]]) to (%[[MAX]]) step (%[[I1]]) {
+//  CHECK-NEXT:       %[[ZERO:.*]] = constant 0.000000e+00 : f32
+//  CHECK-NEXT:       loop.reduce(%[[ZERO]]) {
+//  CHECK-NEXT:       ^bb0(%[[LHS:.*]]: f32, %[[RHS:.*]]: f32):
+//  CHECK-NEXT:         %[[RES:.*]] = addf %[[LHS]], %[[RHS]] : f32
+//  CHECK-NEXT:         loop.reduce.return %[[RES]] : f32
+//  CHECK-NEXT:       } : f32
+//  CHECK-NEXT:       "loop.terminator"() : () -> ()
+//  CHECK-NEXT:     } : f32
+//  CHECK-NEXT:     "loop.terminator"() : () -> ()