[MLIR][Linalg] Lower linalg.generic to ploops.

[pifon2a]

In the follow-up PR linalg->ploops will also add support for reduce loops.

[merge_guards_bot]

Unit tests: pass. 62304 tests passed, 0 failed and 838 were skipped.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[ftynse]

What happens when you have non-parallel loops in the generic op? Can we have a test for this case (even if there's no transform happening, just make sure we bail out instead of generating incorrectly parallel loop?)

mlir/include/mlir/Dialect/Linalg/Passes.h

38

Given that we have parallel loops to sequential loops, does it make sense to keep Linalg to sequential loops?

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

179–186

Any specific reason for this rename? I find "body" to be a better name than 'region".

mlir/lib/Dialect/LoopOps/LoopOps.cpp

235

Ultra-nit: drop trivial braces

[pifon2a]

Addressed comments.

[pifon2a]

mlir/include/mlir/Dialect/Linalg/Passes.h

38

Yes, it does. We want to go LHLO->Linalg->Ploops->GPU.

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

179–186

This is what we have in loop::ForOp. Just wanted to be consistent. I can change both though. I can also change getBody()->body() in ForOp and in ParallelOp.

[pifon2a]

In D73684#1849025, @ftynse wrote:

What happens when you have non-parallel loops in the generic op? Can we have a test for this case (even if there's no transform happening, just make sure we bail out instead of generating incorrectly parallel loop?)

Thank you for the comments. I added a test and a check, that should fail if we are lowering to ploops with any non-zero number of window/reduce loops.

[merge_guards_bot]

Unit tests: fail. 62336 tests passed, 1 failed and 838 were skipped.

failed: libc++.std/containers/sequences/array/array_creation/to_array.fail.cpp

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[ftynse]

mlir/include/mlir/Dialect/Linalg/Passes.h

38

Why? How having Linalg->Loops helps you? You are going to have LHLO->Linalg->Ploops->GPU, and you can also have LHLO->Linalg->Ploops->Loops->e.g.CPU because we have Ploops->Loops. I don't see the need for Linalg->Loops in this scheme, it will be mostly duplicate code.

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

179–186

Consistency is good. Change it in a follow-up if you want.

mlir/lib/Dialect/Linalg/Transforms/LinalgToLoops.cpp

416

Nit: MLIR uses lowerCamelCase for function names

mlir/test/Dialect/Linalg/parallel_loops.mlir

2

Do we need really need --dump-input=always ?

[rriddle]

mlir/include/mlir/Dialect/Linalg/EDSC/Builders.h

80

nit: conditional_t here and below

mlir/include/mlir/Dialect/LoopOps/LoopOps.h

47

///

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

184

Is this wrapped at 80?

mlir/include/mlir/EDSC/Builders.h

224

Can you add a comment to this?

mlir/lib/Dialect/Linalg/EDSC/Builders.cpp

119

nit: lbs, ubs, steps?

mlir/lib/Dialect/Linalg/Transforms/LinalgToLoops.cpp

606

///

mlir/lib/Dialect/LoopOps/LoopOps.cpp

351

dyn_cast?

mlir/lib/EDSC/Builders.cpp

196

llvm::to_vector<4>(lbHandles|ubHandles|steps)

202

*Drop trivial braces
*Cache the end iterator of for loops.

mlir/test/Dialect/Linalg/parallel_loops.mlir

2

dump-input-on-failure is a better choice here.

[pifon2a]

Addressed the comments.

[pifon2a]

@rriddle @ftynse Thank you for the comments!

mlir/include/mlir/Dialect/Linalg/Passes.h

38

We will have tiling and fusion on Ploops before going to GPU.

mlir/include/mlir/Dialect/LoopOps/LoopOps.h

47

There are also comments for these functions in Passes.h. I am removing the comments from .cpp.

mlir/lib/EDSC/Builders.cpp

196

this did not work

[merge_guards_bot]

Unit tests: fail. 62344 tests passed, 1 failed and 839 were skipped.

failed: libc++.std/containers/sequences/array/array_creation/to_array.fail.cpp

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[herhut]

Whoohoo. \0/ Thanks!

mlir/include/mlir/Dialect/Linalg/Passes.h

38

I think you are cross-talking here. What ftynse suggests is always going

LinAlg->Ploops->SeqLoops->...

instead of also having

LinAlg->SeqLoops.

That is a good point. I would suggest we build out LinAlg->Ploop first before retiring the other lowering, though.

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

179–186

Just adding my 2 <insert small change of currency of choice> here. I find the name body misleading, especially when there also is a getBody() method on the same op. Hence I prefer region for the region and body to return the first block (if there is a single block in the region).

mlir/lib/Dialect/LoopOps/LoopOps.cpp

234

ParallelOp::ensureTerminator

329

I just saw it is wrong here, too :(

[nicolasvasilache]

Coming a little late to the party but this looks great, thanks Alex-DE!
+1 to Alex-FR's suggestion of Linalg -> ploop -> loop when it makes sense, and similarly in the future Linalg -> affine.ploop -> affine.loop.

[vsk]

Hi @pifon2a, I noticed that your work appeared on GitHub at remotes/llvm/cl291881283. Please stage work on your fork in the future, I think those branches are just for releases. Thanks.

mlir/include/mlir/Dialect/Linalg/EDSC/Builders.h

 /// ranges.
 template <typename LoopTy> class GenericLoopNestRangeBuilder {
 public:
   GenericLoopNestRangeBuilder(ArrayRef<edsc::ValueHandle *> ivs,
                               ArrayRef<Value> ranges);
   void operator()(std::function<void(void)> fun = nullptr) { (*builder)(fun); }
 
 private:
-  typedef typename std::conditional<std::is_same<LoopTy, AffineForOp>::value,
-                                    AffineLoopNestBuilder,
-                                    LoopNestRangeBuilder>::type BuilderType;
+  using LoopOrAffineLoopBuilder =
+      typename std::conditional_t<std::is_same<LoopTy, AffineForOp>::value,
+                                  AffineLoopNestBuilder, LoopNestRangeBuilder>;
+  using BuilderType =
+      typename std::conditional_t<std::is_same<LoopTy, loop::ParallelOp>::value,
+                                  ParallelLoopNestBuilder,
+                                  LoopOrAffineLoopBuilder>;
+
   std::unique_ptr<BuilderType> builder;
 };
 
 enum class IterType { Parallel, Reduction };
 
 inline StringRef toString(IterType t) {
   switch (t) {
   case IterType::Parallel:

mlir/include/mlir/Dialect/Linalg/Passes.h

 
 std::unique_ptr<OpPassBase<FuncOp>>
 createLinalgPromotionPass(bool dynamicBuffers);
 
 /// Create a pass to convert Linalg operations to loop.for loops and
 /// std.load/std.store accesses.
 std::unique_ptr<OpPassBase<FuncOp>> createConvertLinalgToLoopsPass();
 
+/// Create a pass to convert Linalg operations to loop.parallel loops and
+/// std.load/std.store accesses.
+std::unique_ptr<OpPassBase<FuncOp>> createConvertLinalgToParallelLoopsPass();
+
 /// Create a pass to convert Linalg operations to affine.for loops and
 /// affine_load/affine_store accesses.
 /// Placeholder for now, this is NYI.
 std::unique_ptr<OpPassBase<FuncOp>> createConvertLinalgToAffineLoopsPass();
 
 } // namespace mlir
 
 #endif // MLIR_DIALECT_LINALG_PASSES_H_

mlir/include/mlir/Dialect/Linalg/Transforms/LinalgTransformPatterns.td

 
 //===----------------------------------------------------------------------===//
 // Linalg to loop patterns.
 //===----------------------------------------------------------------------===//
 class LinalgOpToLoops<string OpType> : NativeCodeCall<
   "if (failed(linalgOpToLoops<" # OpType # ">($_builder, op))) " #
   "  return matchFailure();">;
 
+class LinalgOpToParallelLoops<string OpType> : NativeCodeCall<
+  "if (failed(linalgOpToParallelLoops<" # OpType # ">($_builder, op))) " #
+  "  return matchFailure();">;
+
 class LinalgOpToAffineLoops<string OpType> : NativeCodeCall<
   "if (failed(linalgOpToAffineLoops<" # OpType # ">($_builder, op))) " #
   "  return matchFailure();">;
 
 //===----------------------------------------------------------------------===//
 // Linalg to vector patterns precondition and DRR.
 //===----------------------------------------------------------------------===//
 def PreconditionVectorizeLinalgOp : CPred<

mlir/include/mlir/Dialect/Linalg/Transforms/LinalgTransforms.h

 LogicalResult tileAndFuseLinalgOpAndSetMarker(
     PatternRewriter &rewriter, Operation *op, ArrayRef<int64_t> sizes,
     ArrayRef<int64_t> operandIndicesToFuse, StringRef linalgMarker);
 
 /// Emits a loop nest of `loop.for` with the proper body for `op`.
 template <typename ConcreteOp>
 LogicalResult linalgOpToLoops(PatternRewriter &rewriter, Operation *op);
 
+/// Emits a loop nest of `loop.parallel` with the proper body for `op`.
+template <typename ConcreteOp>
+LogicalResult linalgOpToParallelLoops(PatternRewriter &rewriter, Operation *op);
+
 /// Emits a loop nest of `affine.for` with the proper body for `op`.
 template <typename ConcreteOp>
 LogicalResult linalgOpToAffineLoops(PatternRewriter &rewriter, Operation *op);
 
 /// Rewrite a linalg.generic into a suitable vector.contraction op.
 LogicalResult vectorizeLinalgOpPrecondition(Operation *op);
 SmallVector<Value, 0> vectorizeLinalgOp(PatternRewriter &rewriter,
                                         Operation *op);

mlir/include/mlir/Dialect/LoopOps/LoopOps.h

 // the location is specified by `loc`. If the region is empty, insert a new
 // block first.
 void ensureLoopTerminator(Region &region, Builder &builder, Location loc);
 
 /// Returns the loop parent of an induction variable. If the provided value is
 /// not an induction variable, then return nullptr.
 ForOp getForInductionVarOwner(Value val);
 
+/// Returns the parallel loop parent of an induction variable. If the provided
+// value is not an induction variable, then return nullptr.
+ParallelOp getParallelForInductionVarOwner(Value val);
+
 } // end namespace loop
 } // end namespace mlir
 #endif // MLIR_LOOPOPS_OPS_H_

mlir/include/mlir/Dialect/LoopOps/LoopOps.td

```mlir
        }
     ```
    }];
 
   let arguments = (ins Variadic<Index>:$lowerBound,
                        Variadic<Index>:$upperBound,
                        Variadic<Index>:$step);
   let results = (outs Variadic<AnyType>:$results);
-  let regions = (region SizedRegion<1>:$body);
+  let regions = (region SizedRegion<1>:$region);
+
+  let skipDefaultBuilders = 1;
+  let builders = [
+    OpBuilder<"Builder *builder, OperationState &result, "
+              "ValueRange lowerBounds, ValueRange upperBounds, "
+              "ValueRange steps">
+  ];
 
   let extraClassDeclaration = [{
+    Block *getBody() { return &region().front(); }
     iterator_range<Block::args_iterator> getInductionVars() {
-      Block &block = body().front();
-      return {block.args_begin(), block.args_end()};
+      return {getBody()->args_begin(), getBody()->args_end()};
     }
   }];
 }
 
 def ReduceOp : Loop_Op<"reduce", [HasParent<"ParallelOp">]> {
   let summary = "reduce operation for parallel for";
   let description = [{
     "loop.reduce" is an operation occurring inside "loop.parallel" operations.

mlir/include/mlir/EDSC/Builders.h

 class LoopBuilder : public NestedBuilder {
 public:
   /// Constructs a new AffineForOp and captures the associated induction
   /// variable. A ValueHandle pointer is passed as the first argument and is the
   /// *only* way to capture the loop induction variable.
   static LoopBuilder makeAffine(ValueHandle *iv,
                                 ArrayRef<ValueHandle> lbHandles,
                                 ArrayRef<ValueHandle> ubHandles, int64_t step);
+  /// Constructs a new loop::ParallelOp and captures the associated induction
+  /// variables. An array of ValueHandle pointers is passed as the first
+  /// argument and is the *only* way to capture loop induction variables.
+  static LoopBuilder makeParallel(ArrayRef<ValueHandle *> ivs,
+                                  ArrayRef<ValueHandle> lbHandles,
+                                  ArrayRef<ValueHandle> ubHandles,
+                                  ArrayRef<ValueHandle> steps);
   /// Constructs a new loop::ForOp and captures the associated induction
   /// variable. A ValueHandle pointer is passed as the first argument and is the
   /// *only* way to capture the loop induction variable.
   static LoopBuilder makeLoop(ValueHandle *iv, ValueHandle lbHandle,
                               ValueHandle ubHandle, ValueHandle stepHandle);
   LoopBuilder(const LoopBuilder &) = delete;
   LoopBuilder(LoopBuilder &&) = default;
 
AffineLoopNestBuilder(ArrayRef<ValueHandle *> ivs, ArrayRef<ValueHandle> lbs,
                         ArrayRef<ValueHandle> ubs, ArrayRef<int64_t> steps);
 
   void operator()(function_ref<void(void)> fun = nullptr);
 
 private:
   SmallVector<LoopBuilder, 4> loops;
 };
 
+/// Helper class to sugar building loop.parallel loop nests from lower/upper
+/// bounds and step sizes.
+class ParallelLoopNestBuilder {
+public:
+  ParallelLoopNestBuilder(ArrayRef<ValueHandle *> ivs,
+                          ArrayRef<ValueHandle> lbs, ArrayRef<ValueHandle> ubs,
+                          ArrayRef<ValueHandle> steps);
+
+  void operator()(function_ref<void(void)> fun = nullptr);
+
+private:
+  SmallVector<LoopBuilder, 4> loops;
+};
+
 /// Helper class to sugar building loop.for loop nests from ranges.
 /// This is similar to edsc::AffineLoopNestBuilder except it operates on
 /// loop.for.
 class LoopNestBuilder {
 public:
   LoopNestBuilder(ArrayRef<edsc::ValueHandle *> ivs, ArrayRef<ValueHandle> lbs,
                   ArrayRef<ValueHandle> ubs, ArrayRef<ValueHandle> steps);
   void operator()(std::function<void(void)> fun = nullptr);

mlir/lib/Conversion/LoopToStandard/ConvertLoopToStandard.cpp

for (auto loop_operands :
     Value iv, lower, upper, step;
     std::tie(iv, lower, upper, step) = loop_operands;
     ForOp forOp = rewriter.create<ForOp>(loc, lower, upper, step);
     mapping.map(iv, forOp.getInductionVar());
     rewriter.setInsertionPointToStart(forOp.getBody());
   }
 
   // Now copy over the contents of the body.
-  for (auto &op : parallelOp.body().front().without_terminator())
+  for (auto &op : parallelOp.getBody()->without_terminator())
     rewriter.clone(op, mapping);
 
   rewriter.eraseOp(parallelOp);
 
   return matchSuccess();
 }
 
 void mlir::populateLoopToStdConversionPatterns(

mlir/lib/Dialect/Linalg/EDSC/Builders.cpp

ValueHandle LoopNestRangeBuilder::LoopNestRangeBuilder::operator()(
   for (auto &lit : reverse(loops)) {
     lit({});
   }
   return ValueHandle::null();
 }
 
 namespace mlir {
 namespace edsc {
+
 template <>
 GenericLoopNestRangeBuilder<loop::ForOp>::GenericLoopNestRangeBuilder(
     ArrayRef<edsc::ValueHandle *> ivs, ArrayRef<Value> ranges) {
   builder = std::make_unique<LoopNestRangeBuilder>(ivs, ranges);
 }
 
 template <>
 GenericLoopNestRangeBuilder<AffineForOp>::GenericLoopNestRangeBuilder(
     ArrayRef<ValueHandle *> ivs, ArrayRef<Value> ranges) {
   SmallVector<ValueHandle, 4> lbs;
   SmallVector<ValueHandle, 4> ubs;
   SmallVector<int64_t, 4> steps;
   for (Value range : ranges) {
     assert(range.getType() && "expected linalg.range type");
     assert(range.getDefiningOp() && "need operations to extract range parts");
     RangeOp rangeOp = cast<RangeOp>(range.getDefiningOp());
-    lbs.emplace_back(ValueHandle(rangeOp.min()));
-    ubs.emplace_back(ValueHandle(rangeOp.max()));
-    steps.emplace_back(ValueHandle(rangeOp.step()));
+    lbs.emplace_back(rangeOp.min());
+    ubs.emplace_back(rangeOp.max());
+    steps.emplace_back(rangeOp.step());
   }
   builder = std::make_unique<AffineLoopNestBuilder>(ivs, lbs, ubs, steps);
 }
+
+template <>
+GenericLoopNestRangeBuilder<loop::ParallelOp>::GenericLoopNestRangeBuilder(
+    ArrayRef<ValueHandle *> ivs, ArrayRef<Value> ranges) {
+  SmallVector<ValueHandle, 4> lbs, ubs, steps;
+  for (Value range : ranges) {
+    assert(range.getType() && "expected linalg.range type");
+    assert(range.getDefiningOp() && "need operations to extract range parts");
+    RangeOp rangeOp = cast<RangeOp>(range.getDefiningOp());
+    lbs.emplace_back(rangeOp.min());
+    ubs.emplace_back(rangeOp.max());
+    steps.emplace_back(rangeOp.step());
+  }
+  builder = std::make_unique<ParallelLoopNestBuilder>(ivs, lbs, ubs, steps);
+}
+
 } // namespace edsc
 } // namespace mlir
 
 static void getMaxDimIndex(ArrayRef<StructuredIndexed> structuredIndices,
                            unsigned &pos) {
   for (auto sidx : structuredIndices) {
     for (auto expr : sidx.getExprs()) {
       expr.walk([&pos](AffineExpr e) {

mlir/lib/Dialect/Linalg/Transforms/LinalgToLoops.cpp

 // guaranteed to be a success, (2) it is not allowed erase/replace, and (3) an
 // encompassing pattern must take care of the erasure logic.
 template <typename LoopTy, typename IndexedValueTy, typename ConcreteOpTy>
 class LinalgOpToLoopsImpl {
 public:
   static LogicalResult doit(Operation *op, PatternRewriter &rewriter);
 };
 
+template <typename LoopTy>
+bool loweringIsAllowed(int numParallelLoops, int numLoops) {
+  return true;
+}
+template <>
+bool loweringIsAllowed<loop::ParallelOp>(int numParallelLoops, int numLoops) {
+  return numParallelLoops == numLoops;
+}
+
 template <typename LoopTy, typename IndexedValueTy, typename ConcreteOpTy>
 LogicalResult LinalgOpToLoopsImpl<LoopTy, IndexedValueTy, ConcreteOpTy>::doit(
     Operation *op, PatternRewriter &rewriter) {
   OpBuilder b(op);
   ScopedContext scope(b, op->getLoc());
 
   // The flattened loopToOperandRangesMaps is expected to be an invertible
   // permutation map (which is asserted in the inverse calculation).
   auto linalgOp = cast<ConcreteOpTy>(op);
   assert(linalgOp.hasBufferSemantics() &&
          "expected linalg op with buffer semantics");
+  auto nPar = linalgOp.getNumParallelLoops();
+  auto nRed = linalgOp.getNumReductionLoops();
+  auto nWin = linalgOp.getNumWindowLoops();
+  auto nLoops = nPar + nRed + nWin;
+  if (!loweringIsAllowed<LoopTy>(nPar, nLoops))
+    return failure();
   auto invertedMap =
       inversePermutation(concatAffineMaps(loopToOperandRangesMaps(linalgOp)));
   if (!invertedMap) {
     LinalgScopedEmitter<IndexedValueTy, ConcreteOpTy>::emitScalarImplementation(
         {}, linalgOp);
     return success();
   }
 
-  auto nPar = linalgOp.getNumParallelLoops();
-  auto nRed = linalgOp.getNumReductionLoops();
-  auto nWin = linalgOp.getNumWindowLoops();
-  SmallVector<IndexHandle, 4> allIvs(nPar + nRed + nWin);
+  SmallVector<IndexHandle, 4> allIvs(nLoops);
   SmallVector<ValueHandle *, 4> allPIvs =
       makeHandlePointers(MutableArrayRef<IndexHandle>(allIvs));
   auto loopRanges = emitLoopRanges(scope.getBuilder(), scope.getLocation(),
                                    invertedMap, getViewSizes(b, linalgOp));
   assert(loopRanges.size() == allIvs.size());
 
   GenericLoopNestRangeBuilder<LoopTy>(allPIvs, loopRanges)([&] {
     auto allIvValues = extractValues(allIvs);
void LowerLinalgToLoopsPass<LoopType, IndexedValueType>::runOnFunction() {
   FillRewritePatterns<LoopType, IndexedValueType>(patterns, context);
   DimOp::getCanonicalizationPatterns(patterns, context);
   AffineApplyOp::getCanonicalizationPatterns(patterns, context);
   patterns.insert<FoldAffineOp>(context);
   // Just apply the patterns greedily.
   applyPatternsGreedily(this->getFunction(), patterns);
 }
 
-/// Create a pass to convert Linalg operations to loop.for loops and
-/// std.load/std.store accesses.
 std::unique_ptr<OpPassBase<FuncOp>> mlir::createConvertLinalgToLoopsPass() {
   return std::make_unique<
       LowerLinalgToLoopsPass<loop::ForOp, IndexedStdValue>>();
 }
 
-/// Create a pass to convert Linalg operations to affine.for loops and
-/// affine_load/affine_store accesses.
-/// Placeholder for now, this is NYI.
+std::unique_ptr<OpPassBase<FuncOp>>
+mlir::createConvertLinalgToParallelLoopsPass() {
+  return std::make_unique<
+      LowerLinalgToLoopsPass<loop::ParallelOp, IndexedStdValue>>();
+}
+
 std::unique_ptr<OpPassBase<FuncOp>>
 mlir::createConvertLinalgToAffineLoopsPass() {
   return std::make_unique<
       LowerLinalgToLoopsPass<AffineForOp, IndexedAffineValue>>();
 }
 
-// Emits a loop nest of `loop.for` with the proper body for `op`.
+/// Emits a loop nest of `loop.for` with the proper body for `op`.
 template <typename ConcreteOp>
 LogicalResult mlir::linalg::linalgOpToLoops(PatternRewriter &rewriter,
                                             Operation *op) {
   return LinalgOpToLoopsImpl<loop::ForOp, IndexedStdValue, ConcreteOp>::doit(
       op, rewriter);
 }
 
-// Emits a loop nest of `affine.for` with the proper body for `op`.
+/// Emits a loop nest of `affine.for` with the proper body for `op`.
 template <typename ConcreteOp>
 LogicalResult mlir::linalg::linalgOpToAffineLoops(PatternRewriter &rewriter,
                                                   Operation *op) {
   return LinalgOpToLoopsImpl<AffineForOp, IndexedAffineValue, ConcreteOp>::doit(
       op, rewriter);
 }
 
+/// Emits a loop nest of `loop.parallel` with the proper body for `op`.
+template <typename ConcreteOp>
+LogicalResult mlir::linalg::linalgOpToParallelLoops(PatternRewriter &rewriter,
+                                                    Operation *op) {
+  return LinalgOpToLoopsImpl<loop::ParallelOp, IndexedStdValue,
+                             ConcreteOp>::doit(op, rewriter);
+}
+
 // TODO(ntv) Need to make these instantiations more future-proof to avoid the
 // need to update as soon as we add new ops.
 #define INSTANTIATE_LINALG_OP_TO_LOOPS(OP_TYPE)                                \
   template LogicalResult mlir::linalg::linalgOpToLoops<OP_TYPE>(               \
       PatternRewriter & rewriter, Operation * op);                             \
   template LogicalResult mlir::linalg::linalgOpToAffineLoops<OP_TYPE>(         \
       PatternRewriter & rewriter, Operation * op);
 
 INSTANTIATE_LINALG_OP_TO_LOOPS(CopyOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(FillOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(DotOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(MatvecOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(MatmulOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(ConvOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(GenericOp)
 INSTANTIATE_LINALG_OP_TO_LOOPS(IndexedGenericOp)
 
+// TODO(pifon): Enable lowering to parallel loops for ops other than
+// linalg.generic for now to be on the safe side.
+template LogicalResult
+mlir::linalg::linalgOpToParallelLoops<GenericOp>(PatternRewriter &rewriter,
+                                                 Operation *op);
+
 static PassRegistration<LowerLinalgToLoopsPass<loop::ForOp, IndexedStdValue>>
     structuredLoopsPass(
         "convert-linalg-to-loops",
         "Lower the operations from the linalg dialect into loops");
 
+static PassRegistration<
+    LowerLinalgToLoopsPass<loop::ParallelOp, IndexedStdValue>>
+    parallelLoopsPass(
+        "convert-linalg-to-parallel-loops",
+        "Lower the operations from the linalg dialect into parallel loops");
+
 static PassRegistration<LowerLinalgToLoopsPass<AffineForOp, IndexedAffineValue>>
     affineLoopsPass(
         "convert-linalg-to-affine-loops",
         "Lower the operations from the linalg dialect into affine loops");

mlir/lib/Dialect/LoopOps/LoopOps.cpp

static void print(OpAsmPrinter &p, IfOp op) {
 
   p.printOptionalAttrDict(op.getAttrs());
 }
 
 //===----------------------------------------------------------------------===//
 // ParallelOp
 //===----------------------------------------------------------------------===//
 
+void ParallelOp::build(Builder *builder, OperationState &result, ValueRange lbs,
+                       ValueRange ubs, ValueRange steps) {
+  result.addOperands(lbs);
+  result.addOperands(ubs);
+  result.addOperands(steps);
+  Region *bodyRegion = result.addRegion();
+  ParallelOp::ensureTerminator(*bodyRegion, *builder, result.location);
+  for (size_t i = 0; i < steps.size(); ++i)
+    bodyRegion->front().addArgument(builder->getIndexType());
+}
+
 static LogicalResult verify(ParallelOp op) {
   // Check that there is at least one value in lowerBound, upperBound and step.
   // It is sufficient to test only step, because it is ensured already that the
   // number of elements in lowerBound, upperBound and step are the same.
   Operation::operand_range stepValues = op.step();
   if (stepValues.empty())
     return op.emitOpError(
         "needs at least one tuple element for lowerBound, upperBound and step");
 
   // Check whether all constant step values are positive.
   for (Value stepValue : stepValues)
     if (auto cst = dyn_cast_or_null<ConstantIndexOp>(stepValue.getDefiningOp()))
       if (cst.getValue() <= 0)
         return op.emitOpError("constant step operand must be positive");
 
   // Check that the body defines the same number of block arguments as the
   // number of tuple elements in step.
-  Block *body = &op.body().front();
+  Block *body = op.getBody();
   if (body->getNumArguments() != stepValues.size())
     return op.emitOpError(
         "expects the same number of induction variables as bound and step "
         "values");
   for (auto arg : body->getArguments())
     if (!arg.getType().isIndex())
       return op.emitOpError(
           "expects arguments for the induction variable to be of index type");
if (parser.parseRegion(*body, ivs, types))
     return failure();
 
   // Parse attributes and optional results (in case there is a reduce).
   if (parser.parseOptionalAttrDict(result.attributes) ||
       parser.parseOptionalColonTypeList(result.types))
     return failure();
 
   // Add a terminator if none was parsed.
   ForOp::ensureTerminator(*body, builder, result.location);
 
   return success();
 }
 
 static void print(OpAsmPrinter &p, ParallelOp op) {
   p << op.getOperationName() << " (";
-  p.printOperands(op.body().front().getArguments());
+  p.printOperands(op.getBody()->getArguments());
   p << ") = (" << op.lowerBound() << ") to (" << op.upperBound() << ") step ("
     << op.step() << ")";
-  p.printRegion(op.body(), /*printEntryBlockArgs=*/false);
+  p.printRegion(op.region(), /*printEntryBlockArgs=*/false);
   p.printOptionalAttrDict(op.getAttrs());
   if (!op.results().empty())
     p << " : " << op.getResultTypes();
 }
 
+ParallelOp mlir::loop::getParallelForInductionVarOwner(Value val) {
+  auto ivArg = val.dyn_cast<BlockArgument>();
+  if (!ivArg)
+    return ParallelOp();
+  assert(ivArg.getOwner() && "unlinked block argument");
+  auto *containingInst = ivArg.getOwner()->getParentOp();
+  return dyn_cast<ParallelOp>(containingInst);
+}
+
 //===----------------------------------------------------------------------===//
 // ReduceOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(ReduceOp op) {
   // The region of a ReduceOp has two arguments of the same type as its operand.
   auto type = op.operand().getType();
   Block &block = op.reductionOperator().front();

mlir/lib/EDSC/Builders.cpp

*iv = ValueHandle::create<AffineForOp>(
         ubs, ScopedContext::getBuilder().getMultiDimIdentityMap(ubs.size()),
         step);
   }
   auto *body = getForInductionVarOwner(iv->getValue()).getBody();
   result.enter(body, /*prev=*/1);
   return result;
 }
 
+mlir::edsc::LoopBuilder mlir::edsc::LoopBuilder::makeParallel(
+    ArrayRef<ValueHandle *> ivs, ArrayRef<ValueHandle> lbHandles,
+    ArrayRef<ValueHandle> ubHandles, ArrayRef<ValueHandle> steps) {
+  mlir::edsc::LoopBuilder result;
+  auto opHandle = OperationHandle::create<loop::ParallelOp>(
+      SmallVector<Value, 4>(lbHandles.begin(), lbHandles.end()),
+      SmallVector<Value, 4>(ubHandles.begin(), ubHandles.end()),
+      SmallVector<Value, 4>(steps.begin(), steps.end()));
+
+  loop::ParallelOp parallelOp =
+      cast<loop::ParallelOp>(*opHandle.getOperation());
+  for (size_t i = 0, e = ivs.size(); i < e; ++i)
+    *ivs[i] = ValueHandle(parallelOp.getBody()->getArgument(i));
+  result.enter(parallelOp.getBody(), /*prev=*/1);
+  return result;
+}
+
 mlir::edsc::LoopBuilder
 mlir::edsc::LoopBuilder::makeLoop(ValueHandle *iv, ValueHandle lbHandle,
                                   ValueHandle ubHandle,
                                   ValueHandle stepHandle) {
   mlir::edsc::LoopBuilder result;
   auto forOp =
       OperationHandle::createOp<loop::ForOp>(lbHandle, ubHandle, stepHandle);
   *iv = ValueHandle(forOp.getInductionVar());
void mlir::edsc::AffineLoopNestBuilder::operator()(
   // The iteration order is from innermost to outermost because enter/exit needs
   // to be asymmetric (i.e. enter() occurs on LoopBuilder construction, exit()
   // occurs on calling operator()). The asymmetry is required for properly
   // nesting imperfectly nested regions (see LoopBuilder::operator()).
   for (auto lit = loops.rbegin(), eit = loops.rend(); lit != eit; ++lit)
     (*lit)();
 }
 
+mlir::edsc::ParallelLoopNestBuilder::ParallelLoopNestBuilder(
+    ArrayRef<ValueHandle *> ivs, ArrayRef<ValueHandle> lbs,
+    ArrayRef<ValueHandle> ubs, ArrayRef<ValueHandle> steps) {
+  assert(ivs.size() == lbs.size() && "Mismatch in number of arguments");
+  assert(ivs.size() == ubs.size() && "Mismatch in number of arguments");
+  assert(ivs.size() == steps.size() && "Mismatch in number of arguments");
+
+  loops.emplace_back(LoopBuilder::makeParallel(ivs, lbs, ubs, steps));
+}
+
+void mlir::edsc::ParallelLoopNestBuilder::operator()(
+    function_ref<void(void)> fun) {
+  if (fun)
+    fun();
+  // Iterate on the calling operator() on all the loops in the nest.
+  // The iteration order is from innermost to outermost because enter/exit needs
+  // to be asymmetric (i.e. enter() occurs on LoopBuilder construction, exit()
+  // occurs on calling operator()). The asymmetry is required for properly
+  // nesting imperfectly nested regions (see LoopBuilder::operator()).
+  for (auto lit = loops.rbegin(), eit = loops.rend(); lit != eit; ++lit)
+    (*lit)();
+}
+
 mlir::edsc::LoopNestBuilder::LoopNestBuilder(ArrayRef<ValueHandle *> ivs,
                                              ArrayRef<ValueHandle> lbs,
                                              ArrayRef<ValueHandle> ubs,
                                              ArrayRef<ValueHandle> steps) {
   assert(ivs.size() == lbs.size() && "expected size of ivs and lbs to match");
   assert(ivs.size() == ubs.size() && "expected size of ivs and ubs to match");
   assert(ivs.size() == steps.size() &&
          "expected size of ivs and steps to match");

mlir/test/Dialect/Linalg/parallel_loops.mlir

+// RUN: mlir-opt %s -convert-linalg-to-parallel-loops -split-input-file | FileCheck %s --dump-input-on-failure
+
+#map0 = affine_map<(d0, d1) -> (d0, d1)>
+func @linalg_generic_sum(%lhs: memref<2x2xf32>,
+                         %rhs: memref<2x2xf32>,
+                         %sum: memref<2x2xf32>) {
+  linalg.generic {
+    args_in = 2 : i64,
+    args_out = 1 : i64,
+    indexing_maps = [#map0, #map0, #map0],
+    iterator_types = ["parallel", "parallel"]
+  } %lhs, %rhs, %sum {
+    ^bb0(%lhs_in: f32, %rhs_in: f32, %sum_out: f32):   // no predecessors
+      %0 = addf %lhs_in, %rhs_in : f32
+      linalg.yield %0 : f32
+  }: memref<2x2xf32>, memref<2x2xf32>, memref<2x2xf32>
+  return
+}
+// CHECK-LABEL: @linalg_generic_sum
+// CHECK:   (%[[LHS:.*]]:{{.*}}, %[[RHS:.*]]:{{.*}}, %[[SUM:.*]]:{{.*}})
+// CHECK-DAG: %[[C2:.*]] = constant 2
+// CHECK-DAG: %[[C0:.*]] = constant 0
+// CHECK-DAG: %[[C1:.*]] = constant 1
+// CHECK: loop.parallel (%[[I:.*]], %[[J:.*]]) = {{.*}}
+// CHECK:   %[[LHS_ELEM:.*]] = load %[[LHS]][%[[I]], %[[J]]]
+// CHECK:   %[[RHS_ELEM:.*]] = load %[[RHS]][%[[I]], %[[J]]]
+// CHECK:   %[[SUM_ELEM:.*]] = load %[[SUM]][%[[I]], %[[J]]]
+// CHECK:   %[[SUM:.*]] = addf %[[LHS_ELEM]], %[[RHS_ELEM]] : f32
+// CHECK:   store %[[SUM]], %{{.*}}[%[[I]], %[[J]]]
+// CHECK:   "loop.terminator"() : () -> ()
+
+// -----
+
+#accesses = [
+  affine_map<(m, n) -> (m, n)>,
+  affine_map<(m, n) -> (m)>
+]
+#trait = {
+  args_in = 1,
+  args_out = 1,
+  iterator_types = ["parallel", "reduction"],
+  indexing_maps = #accesses
+}
+
+func @do_not_lower_reduce(%A: memref<2x4xf32>, %B: memref<2xf32>) {
+  linalg.generic #trait %A, %B {
+    ^bb0(%a: f32, %b: f32):
+      linalg.yield %a: f32
+  } : memref<2x4xf32>, memref<2xf32>
+  return
+}
+// CHECK-LABEL: @do_not_lower_reduce
+// CHECK: linalg.generic