[mlir][Linalg] Fuse sequence of Linalg operation (on buffers)

[mravishankar]

Summary

Enhance the tile+fuse logic to allow fusing a sequence of operations.

Depends On D91101

[mravishankar]

Rebase

[mravishankar]

Rebase and fix test.

[mravishankar]

Split out parts not related to Fusion itself into separate changes.

[mravishankar]

Move missing files from parent change to here.

[bondhugula]

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

227

All methods including static/local ones are expected to have doc comments.

[bondhugula]

mlir/include/mlir/Dialect/Linalg/Passes.h

25 ↗

(On Diff #303977)

The td description doesn't say as to what happens when this parameter is missing - there is no doc comment here either.

mlir/include/mlir/Dialect/Linalg/Passes.td

40 ↗

(On Diff #303977)

Set the tile sizes to use -> Tiles sizes to use

Please don't use an imperative form here - instead just document what the parameter is.

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

194–198

Why has the doc comment been deleted here?

[mravishankar]

Move pass to test pass.

mlir/include/mlir/Dialect/Linalg/Passes.h

25 ↗

(On Diff #303977)

I moved the pass itself to test directory. This pass was added for testing.

mlir/include/mlir/Dialect/Linalg/Passes.td

40 ↗

(On Diff #303977)

Moved to test.

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

194–198

The original function was split to reuse functionality. The actual doc comment was moved down to the split function below. Adding comments here.

[mravishankar]

Rebase

[mravishankar]

Rebase

[hanchung]

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

204–206

nit drop braces for simple single-statement body of for

233–238

drop trivial braces

261

[optional]: I would name it pos because the body region is simple, and it's easy to know this is for producer because the right above line just states this.

567

nit: use auto with initializers like cast<Foo>(...)

See https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable

570

ditto

598–600

nit: I would just return it directly. (the function name already states that it's returning consumerLoopToProducerLoop, so no need to capture it with a name for readability?)

607

The logic here only checks for contiguous pair. Is it possible to happen on non-contiguous pair? For example,

AffineMap: (d0, d1, d2) -> (d2, d1, d0)

and the fusableLoops only contains d0 and d2.

If it's must be contiguous, I'm confused that why this is a set not an integer or a pair-like type like [start, end).

607–612

This is my personal preference (and it's clearer that we are comparing with neighbors). How about rewrite like this:

for (unsigned curr = 1, e = map.getNumResults(); curr < e; ++curr) {
  unsigned posCurr = map.getResult(curr).cast<AffineDimExpr>().getPosition();
  unsigned posPrev = map.getResult(curr - 1).cast<AffineDimExpr>().getPosition();
  if (fusableLoops.count(posPrev) && fusableLoops.count(posCurr) && posPrev >= posCurr)
    return true;

With this way, we don't need to consider the relationship between i and j. It's more straightforward to me.

654

The TODO statement should be an actionable item. If this is a note, we don't need todo. If this is TODO, then what to do?

830

add assert(tileSizes.size() >= op.getNumLoops()) ?

mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp

174–176

Remove trivial braces

187–188

I think we don't need to pass LinalgTilingOptions() because that is the default constructor?

[mravishankar]

Make the tileAndFuse only tile the fusable loops. Also address comments.

[mravishankar]

Make Tile+Fuse only tile the fusable loops and leave the unfused loops
as is. The TileAndFuse pattern does the tiling of the unfused
operations.
Address other comments.

[antiagainst]

mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h

138

tiledLastLinalgOp could be a more clear name with self-documentation.

142

These are the newly generated ops that inside the fused loops? Would be nice to be clear.

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

227

No need to have helper function to given it does not provide much additional value.

566

Is it necessary to copy the dependence struct here?

781

auto tileSizes = llvm::to_vecor<4>(tileSizeVector)?

879

I think renaming this function as collectTilableAndFusableLoops would be clearer and read better.

mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp

168

We don't care about the order here? (producers is only used at L182 for count). A normal SmallSet will do?

187

This sentence does not read properly for me.

202

This constant op can be created outside the for loop?

[hanchung]

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

611–616

lastFusableLoops should update for each iteration. Otherwise, it only checks if pos < the first fusable loop. E.g.,

(d1, d2, d3) -> (d1, d3, d2)

If all the loops are fusable, the lastFusableLoops is 0 forever, and we won't catch this case.

615

I would use < because they should be identical?

630–633

drop trivial braces

711–712

nit: I think we can use auto here because llvm::seq<xxx> states the type.

783–786

I think LLVM style doesn't want braces for this. See an example in https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements

if (isa<VarDecl>(D)) {
  for (auto *A : D.attrs())
    if (shouldProcessAttr(A))
      handleAttr(A);
}

The inner for-body is not wrapped with braces.

[mravishankar]

Address comments and rebase

[mravishankar]

mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h

138

This is consistent with the TiledLinalgOp described above. I am inclined to leave as is.

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

261

Ack. I was confused when I saw it second time (before I had that longer name).

566

Just copying the LinalgDependenceGraphElem and not the entire DependenceGraph. I think the convention is to pass by value.

607–612

Thanks for catching this. Indeed the logic was flawed. Updated with the fix.

611–616

Ah you are right. Thanks for catching that as well. Changed.

615

It could be identical (the transpose name might mislead here, but I dont know of a better name)

You could have an `affine_namp<(d0, d1, d2) -> (d1, d1, d1)>

Still might be fusable, but havent tested it.

630–633

Spanning a couple of lines. I think the convention is to still keep it.

654

Added some more comments on the todo.

781

I think its the same really.

830

This part is removed now.

879

Just made it collectFusableLoops.

mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp

187

Removed the sentence. It is not needed anymore

187–188

Not needed anymore

[mravishankar]

Rebase

[asaadaldien]

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

229

Note: This is an example of what I was talking about last week =). If we have the computation that is used to compute loop bounds as part of the IR of being computed as part of the Linalg-> Loops transformation things like that will be easier.

[hanchung]

LGTM

[mravishankar]

Reopening since the commit was reverted.

[mravishankar]

Updating the change with https://reviews.llvm.org/D91503 to address some failures.

mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h

struct TiledLinalgOp {
   LinalgOp op;
   SmallVector<Operation *, 8> loops;
   SmallVector<Value, 4> tensorResults;
 };
 
 struct TiledAndFusedLinalgOps {
   LinalgOp op;
   SmallVector<LinalgOp, 1> fusedProducers;
-  SmallVector<LinalgOp, 1> originalProducers;
   SmallVector<Operation *, 4> fusedLoops;
-  SmallVector<Operation *, 4> unfusedLoops;
+  SmallVector<SmallVector<Operation *, 4>, 1> unfusedLoops;
 };
 
 /// Populates patterns for vectorization of all ConvN-D ops.
 void populateConvVectorizationPatterns(
     MLIRContext *context, SmallVectorImpl<OwningRewritePatternList> &patterns,
     ArrayRef<int64_t> tileSizes);
 
 /// Populates the given list with patterns to bufferize linalg ops.

 ///
 /// E.g. the permutation `(i,j,k) -> (j,k,i)` is expressed by
 /// `interchangeVector = [1,2,0]`. All values in `interchangeVector` must be
 /// integers, in the range 0..`tileSizes.size()` without duplications
 /// (i.e. `[1,1,2]` is an invalid permutation).
 Optional<TiledLinalgOp> tileLinalgOp(OpBuilder &b, LinalgOp op,
                                      const LinalgTilingOptions &options);
 
-/// Tile and fuse the `op` with its producers. The tile and fuse proceeds in
-/// three steps
-/// - Find tile loops that are fusable with its producer tile loops (a.k.a. tile
-///   + fuse loops).
-/// - Tile just these loops of the consumer (root operation) and fuse with
-///   the producer.
-/// - Tile again the tiled consumer operation produced above to do rest of
-///   the tiling specified by the `tilingOptions`.
+/// Fuse a sequence of linalg operations (`ops`) using tile-and-fuse. This
+/// proceeds as follows:
+/// - Find outer parallel loops in these ops that can be fused.
+/// - Tile fusable outer parallel loops of the last operation in the sequence.
+/// - Fuse the remaining operations with the tiled operation
+/// - Tile the unfused loops in each of the fused operations id needed.
 ///
 /// For example, consider the sequence of matmul below
 ///
 ///   linalg.matmul ins(%arg0, %arg1 : memref<256x32xf32>, memref<32x32xf32>)
 ///                 outs(%arg2 : memref<256x32xf32>)
 ///   linalg.matmul ins(%arg2, %arg3 : memref<256x32xf32>, memref<32x32xf32>)
 ///                 outs(%arg4 : memref<256x32xf32>)
 ///

 ///   %2 = subview %arg0[%arg5, 0] [16, 32] [1, 1]
 ///     : memref<256x32xf32> to memref<16x32xf32, #map0>
 ///   %3 = subview %arg1[0, 0] [32, 32] [1, 1]
 ///     : memref<32x32xf32> to memref<32x32xf32, #map1>
 ///   linalg.matmul
 ///     ins(%2, %3 : memref<16x32xf32, #map0>, memref<32x32xf32, #map1>)
 ///     outs(%0 : memref<16x32xf32, #map0>)
 ///   scf.parallel (%arg6) = (%c0) to (%c32) step (%c8) {
-///   scf.for %arg7 = %c0 to %c32 step %c4 {
-///     %4 = subview %0[0, %arg7] [16, 4] [1, 1]
-///       : memref<16x32xf32, #map0> to memref<16x4xf32, #map0>
-///     %5 = subview %arg3[%arg7, %arg6] [4, 8] [1, 1]
-///       : memref<32x32xf32> to memref<4x8xf32, #map0>
-///     %6 = subview %1[0, %arg6] [16, 8] [1, 1]
-///       : memref<16x32xf32, #map0> to memref<16x8xf32, #map0>
-///     linalg.matmul
-///       ins(%4, %5 : memref<16x4xf32, #map0>, memref<4x8xf32, #map0>)
-///       outs(%6 : memref<16x8xf32, #map0>)
+///     scf.for %arg7 = %c0 to %c32 step %c4 {
+///       %4 = subview %0[0, %arg7] [16, 4] [1, 1]
+///         : memref<16x32xf32, #map0> to memref<16x4xf32, #map0>
+///       %5 = subview %arg3[%arg7, %arg6] [4, 8] [1, 1]
+///         : memref<32x32xf32> to memref<4x8xf32, #map0>
+///       %6 = subview %1[0, %arg6] [16, 8] [1, 1]
+///         : memref<16x32xf32, #map0> to memref<16x8xf32, #map0>
+///       linalg.matmul
+///         ins(%4, %5 : memref<16x4xf32, #map0>, memref<4x8xf32, #map0>)
+///         outs(%6 : memref<16x8xf32, #map0>)
 ///     }
 ///     scf.yield
 ///   }
 ///   scf.yield
 /// }
 ///
-/// The following tiling options are handled differently in tile+fuse (compared
-/// to tile only)
+/// `tilingOptions` are used to tile the corresponding operation in `ops` (the
+/// size of the former should be same as size of the latter. Based on how
+/// tile+fuse is implemented, the fused loops are generated based on the last
+/// operation in the sequence. For example, the tile sizes for the fused loops
+/// is obtained from `tilingOptions.back()`. The following tiling options are
+/// handled differently in tile+fuse (compared to tile only)
 /// - Interchange of the tiling loops is not supported right now.
-/// - Distribution is only done for the tile+fuse loops. The tiled loops
+/// - Distribution is only done for the fused loops. The tiled loops
 ///   generated by the second tiling is not distributed.
 Optional<TiledAndFusedLinalgOps>
-tileAndFuseLinalgOps(PatternRewriter &rewriter, LinalgOp op,
+tileAndFuseLinalgOps(OpBuilder &builder, ArrayRef<LinalgOp> ops,
                      const LinalgDependenceGraph &dependenceGraph,
-                     const LinalgTilingOptions &tilingOptions,
-                     const LinalgFusionOptions &fusionOptions);
+                     ArrayRef<LinalgTilingOptions> tilingOptions);
 
 /// Interchanges the `iterator_types` and `iterator_maps` dimensions of `op`.
 /// This is an in-place transformation controlled by `interchangeVector`.
 /// An empty vector is interpreted as the identity permutation and the
 /// transformation returns early.
 ///
 /// E.g. the permutation `(i,j,k) -> (j,k,i)` is expressed with
 /// `interchangeVector = [1,2,0]`. All values in `interchangeVector` must be
 /// integers, in the range 0..`op.rank` without duplications
 /// (i.e. `[1,1,2]` is an invalid permutation).
 LinalgOp interchange(LinalgOp op, ArrayRef<unsigned> interchangeVector);
 

mlir/lib/Dialect/Linalg/Transforms/Fusion.cpp

 #include "mlir/IR/AffineMap.h"
 #include "mlir/IR/Dominance.h"
 #include "mlir/Support/LLVM.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 
+#include <map>
 #include <set>
 
 #define DEBUG_TYPE "linalg-fusion"
 
 using namespace mlir;
 using namespace mlir::edsc;
 using namespace mlir::edsc::intrinsics;
 using namespace mlir::linalg;
for (auto en2 : llvm::enumerate(map.getResults())) {
                    << "getShapeDefiningLoopRange shape: " << shape << "\n");
         return ShapeDimension{shape, static_cast<unsigned>(en2.index())};
       }
     }
   }
   llvm_unreachable("Expect to be able to extract a shape defining loop range");
 }
 
-/// Fuses the producer of `producerIdx` into the loop immediately enclosing
-/// `consumer`. This is achieved by "recomputing" the `producer` at the time it
-/// is needed just before the `consumer.
-///
-/// Depending on the type of `consumer.getShapedOperand(consumerIdx)`, there are
-/// 2 cases:
-///   1. Buffer case: `producerIdx` is the index of the buffer in
-///      `producer.getOutputBuffers()`.
-///   2. Tensor case: `producerIdx` is the index of the tensor in
-///      `producer.getResults()`.
-static LinalgOp fuse(OpBuilder &b, LinalgOp producer, unsigned producerIdx,
-                     LinalgOp consumer, unsigned consumerIdx) {
-  Operation *shapeProducingOp =
-      consumer.getShapedOperand(consumerIdx).getDefiningOp();
-  assert((isa<SubViewOp>(shapeProducingOp) ||
-          isa<SubTensorOp>(shapeProducingOp)) &&
-         "SubviewOp or SubTensorOp expected");
-
-  // loopToOperandRangesMaps are permutations-only by construction:
-  //   we can always identify a data dimension with a (at least one) loop
-  //   dimension.
-  // TODO: extend this with range inference.
-  AffineMap producerMap = producer.getOutputIndexingMap(producerIdx);
-  LLVM_DEBUG(llvm::dbgs() << "Producer Idx: " << producerIdx
-                          << ", producer map: " << producerMap << "\n");
+/// Fuse the producer by cloning the `producer`. The `fusedLoopsAndRanges`
+/// provides the loop range information for the fused loops. The rest are
+/// obtained from the producer itself, since they are not tiled + fused.
+static LinalgOp fuse(OpBuilder &b, LinalgOp producer,
+                     const DenseMap<unsigned, Range> &fusedLoopsAndRanges) {
 
   unsigned nPar = producer.getNumParallelLoops();
   unsigned nRed = producer.getNumReductionLoops();
   unsigned nWin = producer.getNumWindowLoops();
   SmallVector<Range, 8> loopRanges(nPar + nRed + nWin);
-
-  // Iterate over dimensions identified by the producer map for `producerIdx`.
-  // This defines a subset of the loop ranges that we need to complete later.
-  auto loc = consumer.getLoc();
-  for (auto en : llvm::enumerate(producerMap.getResults())) {
-    unsigned posInProducerLoop = en.value().cast<AffineDimExpr>().getPosition();
-    loopRanges[posInProducerLoop] =
-        isa<SubViewOp>(shapeProducingOp)
-            ? cast<SubViewOp>(shapeProducingOp)
-                  .getOrCreateRanges(b, loc)[en.index()]
-            : cast<SubTensorOp>(shapeProducingOp)
-                  .getOrCreateRanges(b, loc)[en.index()];
+  for (auto fusedLoops : fusedLoopsAndRanges) {
+    loopRanges[fusedLoops.first] = fusedLoops.second;
   }
 
   // Iterate over all dimensions. For the dimensions not identified by the
   // producer map for `producerIdx`, we need to explicitly compute the shape
   // that defines the loop ranges using the `producer`.
   for (unsigned i = 0, nLoops = loopRanges.size(); i < nLoops; ++i) {
     if (loopRanges[i].offset)
       LLVM_DEBUG(llvm::dbgs()
                  << "existing LoopRange: " << loopRanges[i] << "\n");
     else {
       auto shapeDim = getShapeDefiningLoopRange(producer, i);
       loopRanges[i] = Range{std_constant_index(0),
                             std_dim(shapeDim.shape, shapeDim.dimension),
                             std_constant_index(1)};
       LLVM_DEBUG(llvm::dbgs() << "new LoopRange: " << loopRanges[i] << "\n");
     }
   }
 
-  return cloneWithLoopRanges(b, loc, producer, loopRanges);
+  return cloneWithLoopRanges(b, producer.getLoc(), producer, loopRanges);
+}
+
+/// Helper function to get the loop range for a dimension `dim` based on the
+/// `shapedOperand`. It is expected to be defined by a subview op or a subtensor
+/// op.
+static Range getRangeFromOperandShape(OpBuilder &b, Location loc,
+                                      Value shapedOperand, unsigned dim) {
+  Operation *shapeProducingOp = shapedOperand.getDefiningOp();
+  if (auto subViewOp = dyn_cast<SubViewOp>(shapeProducingOp)) {
+    return subViewOp.getOrCreateRanges(b, loc)[dim];
+  }
+  if (auto subTensorOp = dyn_cast<SubTensorOp>(shapeProducingOp)) {
+    return subTensorOp.getOrCreateRanges(b, loc)[dim];
+  }
+  llvm_unreachable("SubviewOp or SubTensorOp expected");
+}
+
+/// Fuses the producer of `producerIdx` into the loop immediately enclosing
+/// `consumer`. This is achieved by "recomputing" the `producer` at the time it
+/// is needed just before the `consumer.
+///
+/// Depending on the type of `consumer.getShapedOperand(consumerIdx)`, there are
+/// 2 cases:
+///   1. Buffer case: `producerIdx` is the index of the buffer in
+///      `producer.getOutputBuffers()`.
+///   2. Tensor case: `producerIdx` is the index of the tensor in
+///      `producer.getResults()`.
+static LinalgOp fuse(OpBuilder &b, LinalgOp producer, unsigned producerIdx,
+                     LinalgOp consumer, unsigned consumerIdx) {
+  AffineMap producerMap = producer.getOutputIndexingMap(producerIdx);
+  LLVM_DEBUG(llvm::dbgs() << "Producer Idx: " << producerIdx
+                          << ", producer map: " << producerMap << "\n");
+  DenseMap<unsigned, Range> fusedLoopsAndRanges;
+  Location loc = consumer.getLoc();
+  Value shapedOperand = consumer.getShapedOperand(consumerIdx);
+  for (auto en : llvm::enumerate(producerMap.getResults())) {
+    unsigned posInProducerLoop = en.value().cast<AffineDimExpr>().getPosition();
+    fusedLoopsAndRanges[posInProducerLoop] =
+        getRangeFromOperandShape(b, loc, shapedOperand, en.index());
+  }
+  return fuse(b, producer, fusedLoopsAndRanges);
 }
 
 // Encode structural fusion safety preconditions.
 // Some of these will be lifted in the future with better analysis.
 static bool isStructurallyFusableProducer(LinalgOp producer, Value consumedView,
                                           LinalgOp consumer) {
   assert(producer.hasBufferSemantics() &&
          "expected linalg op with buffer semantics");

 
 /// Returns the positions of the loop in `op` that can be tiled based on the
 /// operations that are to be fused with it. For example, in a
 ///
 ///   linalg.matmul ins(%a, %b : ...) outs(%c : ...)
 ///
 /// if the producer of %a needs to be fused with this op, only the `i` loop of
 /// the matmul can be tiled while fusing. If producer of %a, and %b are to be
 /// fused, then no loops can be tiled while fusing. The conditions used are:
 /// 1. Only parallel loops can be used for tile + fuse. Find the number of
 ///    common outer parallel loops between the op and its producers being fused.
 /// 2. Of the parallel loops only some can be fused. Only those loops can be
 ///    fused such where the fused loop iteration space only touches one tile of
 ///    the fused operation. This is because the producer (which is writing the
 ///    fused subview) has update semantics. To compute this,
 ///    a. Find the mapping from iterations in the consumer that write to the
 ///       same location as the iterations in the producer. To do so use
 ///       - indexing map of the fused view in the consumer : consumerIndexMap
 ///       - indexing map of the fused view in the producer : producerIndexMap
 ///       consumerLoopToProducerLoop =
 ///         inverse(producerIndexMap).compose(consumerIndexMap)

 ///     consumerIndexMap = affine_map<(i, j, k) -> (i, j)>
 ///     consumerLoopToProducerLoop = affine_map<(i, j, k) -> (i, j)>
 ///     Fused dimensions : i, j
 ///
 /// Example 2:
 ///   linalg.matmul ins(%a, %b) outs(%c)
 ///   linalg.generic {indexing_maps = [affine_map<(i, j) -> (j, i)>, ...
 ///                   iterator_types = ["parallel", "parallel"]}
 ///     ins(%c) ...
 ///
 ///     Number of parallel loops = 2:
 ///     producerIndexMap (projected to parallel loops) =
 ///       affine_map<(i, j) -> (i, j)>
 ///     consumerLoopToProducerLoop2 = affine_map<(i, j) -> (j, i)>
 ///     Fused dimensions : i, j
 ///
 /// Example 3:
 ///   linalg.copy(%s, %b)
 ///   linalg.matmul ins(%a, %b) outs(%c)
 ///
 ///   Number of parallel loops = 2
 ///   produceIndexMap : affine_map<(i, j) -> (i, j)>
 ///   consumerLoopToProduceLoops = affine_map<(i, j, k) -> (k, j)>
 ///     submap with only parallel loops = affine_map<(i, j) -> (j)>
 ///   Fused dimensions : j
 static std::set<unsigned>
-collectTileAndFuseLoops(LinalgOp op,
+collectTileAndFuseLoops(ArrayRef<LinalgOp> ops,
                         const FusableOpDependencesTy &fusableDependences) {
+  assert(!ops.empty());
   auto getNumOuterParallelLoops = [](LinalgOp linalgOp) {
     return linalgOp.iterator_types()
         .getValue()
         .take_while([](Attribute attr) -> bool {
           return attr.cast<StringAttr>().getValue() ==
                  getParallelIteratorTypeName();
         })
         .size();
   };
 
-  LLVM_DEBUG({
-    llvm::dbgs() << "Op : ";
-    op.getOperation()->print(llvm::dbgs(), OpPrintingFlags().useLocalScope());
-    llvm::dbgs() << "\n";
-  });
-
-  size_t numOuterParallelLoops = getNumOuterParallelLoops(op);
-  for (auto dependence : fusableDependences) {
-    linalg::LinalgOp producer = cast<linalg::LinalgOp>(dependence.first);
+  size_t numOuterParallelLoops = getNumOuterParallelLoops(ops.back());
+  for (auto op : ops.drop_back()) {
     numOuterParallelLoops =
-        std::min(numOuterParallelLoops, getNumOuterParallelLoops(producer));
+        std::min(numOuterParallelLoops, getNumOuterParallelLoops(op));
   }
 
   std::set<unsigned> fusableLoops;
   auto range = llvm::seq<unsigned>(0, numOuterParallelLoops);
   fusableLoops.insert(range.begin(), range.end());
-  for (auto dependence : fusableDependences) {
+
+  for (auto op : reverse(ops)) {
+    for (auto dependence : fusableDependences.lookup(op)) {
       LLVM_DEBUG({
         llvm::dbgs() << "\t fusable :";
         for (unsigned i : fusableLoops)
           llvm::dbgs() << " " << i;
         llvm::dbgs() << "\n";
       });
-    linalg::LinalgOp producer = cast<linalg::LinalgOp>(dependence.first);
 
-    assert(!dependence.second.empty() &&
-           "unexpected producer but not dependences");
-    AffineMap producerIndexingMap = producer.getIndexingMap(
-        dependence.second.front().dependentOpView.operandNum);
+      AffineMap producerIndexingMap =
+          op.getIndexingMap(dependence.dependentOpView.operandNum);
       AffineMap prunedProducerIndexingMap = pruneReductionDimsFromMap(
-        producer.iterator_types().getValue(), producerIndexingMap);
+          op.iterator_types().getValue(), producerIndexingMap);
       if (!prunedProducerIndexingMap.isPermutation())
         return {};
 
+      LinalgOp consumer = cast<LinalgOp>(dependence.indexingOpView.op);
       AffineMap consumerIndexingMap =
-        op.getIndexingMap(dependence.second.front().indexingOpView.operandNum);
+          consumer.getIndexingMap(dependence.indexingOpView.operandNum);
       if (consumerIndexingMap.getNumResults() !=
           prunedProducerIndexingMap.getNumResults())
         return {};
 
       LLVM_DEBUG({
         llvm::dbgs() << "\t producerMap : ";
         producerIndexingMap.print(llvm::dbgs());
         llvm::dbgs() << "  pruned : ";
         prunedProducerIndexingMap.print(llvm::dbgs());
         llvm::dbgs() << "\n";
         llvm::dbgs() << "\t consumerMap : ";
         consumerIndexingMap.print(llvm::dbgs());
         llvm::dbgs() << "\n";
       });
 
       AffineMap invProducerIndexMap =
           inversePermutation(prunedProducerIndexingMap);
       if (!invProducerIndexMap)
         return {};
 
       AffineMap consumerLoopToProducerLoop =
           invProducerIndexMap.compose(consumerIndexingMap);
       std::set<unsigned> candidates;
       for (AffineExpr expr : consumerLoopToProducerLoop.getResults()) {
         unsigned position = expr.cast<AffineDimExpr>().getPosition();
         if (fusableLoops.count(position))
           candidates.insert(position);
       }
       LLVM_DEBUG({
         llvm::dbgs() << "\t candidates :";
         for (unsigned i : candidates)
           llvm::dbgs() << " " << i;
         llvm::dbgs() << "\n";
       });
       if (candidates.empty())
         return {};
       std::swap(candidates, fusableLoops);
     }
+  }
 
   return fusableLoops;
 }
 
 /// Find all dependences that are to be fusable.
 static FusableOpDependencesTy
-findAllFusableDependences(LinalgOp op,
-                          const LinalgDependenceGraph &dependenceGraph,
-                          const LinalgFusionOptions &fusionOptions) {
+findAllFusableDependences(ArrayRef<LinalgOp> ops,
+                          const LinalgDependenceGraph &dependenceGraph) {
   FusableOpDependencesTy fusableDependences;
   // TODO: Currently fusion would not be legal if the fusable dependence is to
   // the same producer but different indexing map in the consumer. Fix this, but
   // in the meanwhile disallow such a fusion.
   DenseMap<Operation *, AffineMap> fusedProducerIndexingMap;
-  for (auto operandIndex : fusionOptions.indicesToFuse) {
-    auto fusableDependence =
-        findFusableProducer(op, operandIndex, dependenceGraph);
+  for (LinalgOp op : reverse(ops)) {
+    for (unsigned operandNum :
+         llvm::seq<unsigned>(0, op.getNumInputsAndOutputBuffers())) {
+      Optional<LinalgDependenceGraph::LinalgDependenceGraphElem>
+          fusableDependence =
+              findFusableProducer(op, operandNum, dependenceGraph);
       if (!fusableDependence)
-      return FusableOpDependencesTy{};
-    LinalgOp producerOp = cast<LinalgOp>(fusableDependence->dependentOpView.op);
+        continue;
+      LinalgOp producerOp =
+          cast<LinalgOp>(fusableDependence->dependentOpView.op);
       // Do not fuse dependences that are to operations not in the same basic
       // block. This avoid moving fused operations across loops that might
       // themselves carry dependency making the fusion illegal.
       if (producerOp.getOperation()->getBlock() !=
           op.getOperation()->getBlock()) {
         op.emitRemark("unhandled fusion of ops in different basic blocks");
         return FusableOpDependencesTy{};
       }
       // Make sure that the indexing map of the view used for fusion in the
       // producer is a projected permutation.
       unsigned producerIdx = fusableDependence->dependentOpView.operandNum;
       AffineMap producerMap = producerOp.getIndexingMap(producerIdx);
       if (!producerMap.isProjectedPermutation()) {
-      op.emitRemark("unhandled non permutation indexing map for fused view in "
-                    "producer for operand at index ")
-          << operandIndex;
+        op.emitRemark(
+            "unhandled non permutation indexing map for fused view in "
+            << operandNum;
         return FusableOpDependencesTy{};
       }
 
       unsigned consumerIdx = fusableDependence->indexingOpView.operandNum;
       AffineMap consumerMap = op.getIndexingMap(consumerIdx);
       if (!consumerMap.isProjectedPermutation()) {
         op.emitRemark(
-          "unhandled case where indexing map for fused view in the consumer is "
-          "not a projected permutation while fusing at index ")
-          << operandIndex;
+            "unhandled case where indexing map for fused view in the consumer "
+            "is "
+            "not a projected permuration while fusing at index ")
+            << operandNum;
         return FusableOpDependencesTy{};
       }
 
       // Check if the producer is already a fusion candidate. Cannot fuse this
-    // dependence if it has a different indexing map when used in the consumer.
+      // dependence if it has a different indexing map when used in the
+      // consumer.
       if (fusedProducerIndexingMap.count(producerOp.getOperation()) &&
           fusedProducerIndexingMap[producerOp.getOperation()] != consumerMap) {
-      op.emitRemark("unhandled fusion to the same producer but with different "
-                    "indexing maps");
+        op.emitRemark(
+            "unhandled fusion to the same producer but with different "
         return FusableOpDependencesTy{};
       }
       fusedProducerIndexingMap[producerOp.getOperation()] = consumerMap;
 
-    fusableDependences[producerOp.getOperation()].push_back(*fusableDependence);
+      fusableDependences[producerOp.getOperation()].push_back(
+          *fusableDependence);
+    }
   }
   return fusableDependences;
 }
 
 static bool isZero(Value v) {
   if (auto cst = v.getDefiningOp<ConstantIndexOp>())
     return cst.getValue() == 0;
   return false;
 }
 
+/// Tile the fused loops in the root operation, by setting the tile sizes for
+/// all other loops to zero (those will be tiled later).
+static Optional<TiledLinalgOp> tileRootOperation(
+    OpBuilder &builder, LinalgOp op, ArrayRef<Value> tileSizeVector,
+    const LinalgTilingOptions &options, const std::set<unsigned> &fusedLoops) {
+  SmallVector<Value, 4> tileSizes(tileSizeVector.begin(), tileSizeVector.end());
+  auto zero = std_constant_index(0);
+  for (unsigned i = 0, e = tileSizes.size(); i != e; ++i) {
+    if (!fusedLoops.count(i))
+      tileSizes[i] = zero;
+  }
+  LinalgTilingOptions tileFusedLoopsOptions = options;
+  tileFusedLoopsOptions.setTileSizes(tileSizes);
+  return tileLinalgOp(builder, op, tileFusedLoopsOptions);
+}
+
+/// Fuse the operations in `fusionCandidates` with `tiledOp`. Latter is expected
+/// to be a tiled operation such that it is valid to fuse all operations in
+/// `fusionCandidates`, i.e. move the operation within the inter-tile loops of
+/// `tiledOp`.
+static SmallVector<LinalgOp, 1>
+fuseOperations(OpBuilder &builder, LinalgOp tiledOp,
+               ArrayRef<LinalgOp> fusionCandidates,
+               const std::set<unsigned> &fusedLoops) {
+  OpBuilder::InsertionGuard guard(builder);
+  builder.setInsertionPoint(tiledOp);
+  DenseMap<unsigned, Range> fusedLoopsAndRanges;
+  for (unsigned loop : fusedLoops) {
+    ShapeDimension shapeDim = getShapeDefiningLoopRange(tiledOp, loop);
+    fusedLoopsAndRanges[loop] = getRangeFromOperandShape(
+        builder, tiledOp.getLoc(), shapeDim.shape, shapeDim.dimension);
+  }
+  SmallVector<LinalgOp, 1> fusedOps(fusionCandidates.size());
+  for (auto candidate : enumerate(llvm::reverse(fusionCandidates))) {
+    LinalgOp fusedOp = fuse(builder, candidate.value(), fusedLoopsAndRanges);
+    fusedOps[fusionCandidates.size() - candidate.index() - 1] = fusedOp;
+    builder.setInsertionPoint(fusedOp);
+  }
+  return fusedOps;
+}
+
+/// Post fusion, tile all the unfused loops of the fused operations.
+static Optional<TiledLinalgOp> tileUnfusedLoops(
+    OpBuilder &builder, LinalgOp op, ArrayRef<Value> tileSizeVector,
+    const LinalgTilingOptions &options, const std::set<unsigned> &fusedLoops) {
+  OpBuilder::InsertionGuard guard(builder);
+  builder.setInsertionPoint(op);
+  SmallVector<Value, 4> tileSizes(tileSizeVector.begin(), tileSizeVector.end());
+  auto zero = std_constant_index(0);
+  for (unsigned i = 0, e = tileSizes.size(); i != e; ++i) {
+    if (fusedLoops.count(i))
+      tileSizes[i] = zero;
+  }
+  tileSizes.resize(op.getNumLoops());
+  if (llvm::all_of(tileSizes, isZero))
+    return TiledLinalgOp{op, {}, {}};
+  LinalgTilingOptions tileUnfusedLoopsOptions = options;
+  tileUnfusedLoopsOptions.setTileSizes(tileSizes);
+  tileUnfusedLoopsOptions.distribution = llvm::None;
+  return tileLinalgOp(builder, op, tileUnfusedLoopsOptions);
+}
+
 template <typename LoopType>
 static Optional<TiledAndFusedLinalgOps>
-tileAndFuseLinalgOpsImpl(PatternRewriter &rewriter, LinalgOp op,
+tileAndFuseLinalgOpsImpl(OpBuilder &builder, ArrayRef<LinalgOp> ops,
                          const LinalgDependenceGraph &dependenceGraph,
-                         const LinalgTilingOptions &tilingOptions,
-                         const LinalgFusionOptions &fusionOptions) {
-  assert(op.hasBufferSemantics() && "expected linalg op with buffer semantics");
-  // Some of the tiling options might not be supportable with tile and fuse.
-  // TODO: Support interchange with tile + fuse.
-  if (!tilingOptions.interchangeVector.empty()) {
-    op.emitError("unable to handle tile and fuse with interchange");
+                         ArrayRef<LinalgTilingOptions> tilingOptions) {
+  if (ops.empty())
+    return llvm::None;
+  LinalgOp rootOp = ops.back();
+  if (ops.size() != tilingOptions.size()) {
+    rootOp.emitError(
+        "need tiling options for all the operations that are to be fused");
+    return llvm::None;
+  }
+  for (auto op : enumerate(ops)) {
+    // TODO: Nothing in the fusion of sequence of ops is specific to
+    // buffers. This check can be removed after it is tested on tensors.
+    LinalgOp linalgOp = op.value();
+    if (!linalgOp.hasBufferSemantics()) {
+      linalgOp.emitError("tile and fuse only tested for buffer operation");
       return llvm::None;
     }
+    // TODO: Support interchange with tile + fuse. This might actually help do
+    // better fusion.
+    if (!tilingOptions[op.index()].interchangeVector.empty()) {
+      linalgOp.emitError("unable to handle tile and fuse with interchange");
+      return llvm::None;
+    }
+  }
 
-  OpBuilder::InsertionGuard g(rewriter);
-  rewriter.setInsertionPoint(op);
-  ScopedContext scope(rewriter, op.getLoc());
+  OpBuilder::InsertionGuard guard(builder);
+  builder.setInsertionPoint(rootOp);
+  ScopedContext scope(builder, rootOp.getLoc());
 
   // Find all the producers.
   FusableOpDependencesTy fusableDependences =
-      findAllFusableDependences(op, dependenceGraph, fusionOptions);
+      findAllFusableDependences(ops, dependenceGraph);
   if (fusableDependences.empty())
     return llvm::None;
 
-  // Enforce the convention that "tiling by zero" skips tiling a particular
-  // dimension. This convention is significantly simpler to handle instead of
-  // adjusting affine maps to account for missing dimensions.
-  auto nLoops = op.getNumLoops();
-  SmallVector<Value, 4> tileSizeVector =
-      tilingOptions.tileSizeComputationFunction(rewriter, op);
-  if (tileSizeVector.size() < nLoops) {
-    auto zero = std_constant_index(0);
-    tileSizeVector.append(nLoops - tileSizeVector.size(), zero);
-  }
-
-  TiledAndFusedLinalgOps ret;
-
   // Find the loops that can be tiled and fused.
   std::set<unsigned> tileFuseLoops =
-      collectTileAndFuseLoops(op, fusableDependences);
+      collectTileAndFuseLoops(ops, fusableDependences);
 
   // If there are no fusable dependences or there are no tile+fusable loops,
   // just return.
   if (tileFuseLoops.empty()) {
     return llvm::None;
   }
 
-  // Get the tile sizes for the first and second tiling steps. For the first
-  // step the tile size are set to zero for the loops that arent
-  // fused. Similarly for the second step, the tile sizes are set to zero for
-  // the loops that are fused. For example, if for the following input
-  //
-  // ```
-  //   linalg.add ins(%a, %b) outs(%c)
-  //   linalg.matmul ins(%d, %c) outs(%e)
-  // ```
-  //
-  // if the tile sizes of the `{i, j, k}` loops where given as `{ti, tj, tk}`
-  // respectively, and since only `j` can be tiled and fused. The tile sizes
-  // would be `{0, t_j, 0}` for the first tiling that tiles just the fusable
-  // loops. The second tiling would be use tile sizes of `{t_i, 0, t_k}` to tile
-  // the tiled matmul generated by the first tiling step.
-  SmallVector<Value, 4> tileAndFuseSizes, tileSizes;
-  for (auto tileSize : enumerate(tileSizeVector)) {
-    auto zero = std_constant_index(0);
-    if (tileFuseLoops.count(tileSize.index())) {
-      tileAndFuseSizes.push_back(tileSize.value());
-      tileSizes.push_back(zero);
-    } else {
-      tileSizes.push_back(tileSize.value());
-      tileAndFuseSizes.push_back(zero);
-    }
+  // Tile the fused loops in the last operation in the list.
+  SmallVector<Value, 4> tileSizeVector =
+      tilingOptions.back().tileSizeComputationFunction(builder, rootOp);
+  TiledAndFusedLinalgOps ret;
+  Optional<TiledLinalgOp> tiledRootOp = tileRootOperation(
+      builder, rootOp, tileSizeVector, tilingOptions.back(), tileFuseLoops);
+  if (!tiledRootOp) {
+    rootOp.emitError("failed to tile the fused loops");
+    return llvm::None;
   }
+  ret.op = tiledRootOp->op;
+  ret.fusedLoops.assign(tiledRootOp->loops.begin(), tiledRootOp->loops.end());
 
-  // Tile for the loops that can be fused.
-  LinalgTilingOptions firstTilingOptions = tilingOptions;
-  firstTilingOptions.setTileSizes(tileAndFuseSizes);
-  Optional<TiledLinalgOp> firstTiledOp =
-      tileLinalgOp(rewriter, op, firstTilingOptions);
-  if (!firstTiledOp)
+  // Fuse the other operations into the fused inter-tile loops produced above.
+  ret.fusedProducers =
+      fuseOperations(builder, ret.op, ops.drop_back(), tileFuseLoops);
+
+  // Tile the unfused loops for the fused ops.
+  ret.unfusedLoops.resize(ops.size());
+  for (auto op : enumerate(ret.fusedProducers)) {
+    const LinalgTilingOptions &option = tilingOptions[op.index()];
+    if (!option.tileSizeComputationFunction)
+      continue;
+    SmallVector<Value, 4> producerTileSizeVector =
+        option.tileSizeComputationFunction(builder, op.value());
+    Optional<TiledLinalgOp> tiledOp = tileUnfusedLoops(
+        builder, op.value(), producerTileSizeVector, option, tileFuseLoops);
+    if (!tiledOp) {
+      op.value().emitError("unable to tile unfused loops");
       return llvm::None;
-  ret.op = firstTiledOp->op;
-  ret.fusedLoops.assign(firstTiledOp->loops.begin(), firstTiledOp->loops.end());
+    }
+    if (tiledOp->op != op.value()) {
+      builder.eraseOp(op.value());
+      ret.fusedProducers[op.index()] = tiledOp->op;
+    }
+    ret.unfusedLoops[op.index()].assign(tiledOp->loops.begin(),
+                                        tiledOp->loops.end());
+    builder.setInsertionPoint(tiledOp->op);
+  }
 
-  rewriter.setInsertionPoint(ret.op);
-  // Fuse the operands.
-  for (auto dependence : fusableDependences) {
-    LinalgOp producerOp = cast<LinalgOp>(dependence.first);
-    unsigned producerIdx = dependence.second.front().dependentOpView.operandNum;
-    unsigned consumerIdx = dependence.second.front().indexingOpView.operandNum;
-    LinalgOp fusedOp = fuse(rewriter, producerOp,
-                            producerOp.getOutputIndex(producerIdx).getValue(),
-                            ret.op, consumerIdx);
-    ret.fusedProducers.push_back(fusedOp);
-    ret.originalProducers.push_back(producerOp);
-  }
-
-  if (!llvm::all_of(tileSizes, isZero)) {
-    // Tile the remaining loops of the root operation.
-    LinalgTilingOptions secondTilingOptions = tilingOptions;
-    // The distribution is done only for the tile+fused loops.
-    secondTilingOptions.distribution = llvm::None;
-    secondTilingOptions.setTileSizes(tileSizes);
-    Optional<TiledLinalgOp> secondTiledOp =
-        tileLinalgOp(rewriter, ret.op, secondTilingOptions);
-    if (!secondTiledOp)
-      return llvm::None;
-    ret.unfusedLoops.assign(secondTiledOp->loops.begin(),
-                            secondTiledOp->loops.end());
-    rewriter.eraseOp(ret.op);
-    ret.op = secondTiledOp->op;
+  // Tile the unfused loops for the tiled root operation.
+  Optional<TiledLinalgOp> rootTileUnfusedLoopsOp = tileUnfusedLoops(
+      builder, ret.op, tileSizeVector, tilingOptions.back(), tileFuseLoops);
+  if (!rootTileUnfusedLoopsOp) {
+    ret.op.emitError("unable to tile unfused loops");
   }
-
+  if (rootTileUnfusedLoopsOp->op != ret.op) {
+    builder.eraseOp(ret.op);
+    ret.op = rootTileUnfusedLoopsOp->op;
+  }
+  ret.unfusedLoops.back().assign(rootTileUnfusedLoopsOp->loops.begin(),
+                                 rootTileUnfusedLoopsOp->loops.end());
   return ret;
 }
 
-Optional<TiledAndFusedLinalgOps>
-mlir::linalg::tileAndFuseLinalgOps(PatternRewriter &rewriter, LinalgOp op,
-                                   const LinalgDependenceGraph &dependenceGraph,
-                                   const LinalgTilingOptions &tilingOptions,
-                                   const LinalgFusionOptions &fusionOptions) {
-  switch (tilingOptions.loopType) {
+Optional<TiledAndFusedLinalgOps> mlir::linalg::tileAndFuseLinalgOps(
+    OpBuilder &builder, ArrayRef<LinalgOp> ops,
+    ArrayRef<LinalgTilingOptions> tilingOptions) {
+  switch (tilingOptions.back().loopType) {
   case LinalgTilingLoopType::Loops:
-    return tileAndFuseLinalgOpsImpl<scf::ForOp>(rewriter, op, dependenceGraph,
-                                                tilingOptions, fusionOptions);
+    return tileAndFuseLinalgOpsImpl<scf::ForOp>(builder, ops, dependenceGraph,
+                                                tilingOptions);
   case LinalgTilingLoopType::ParallelLoops:
     return tileAndFuseLinalgOpsImpl<scf::ParallelOp>(
-        rewriter, op, dependenceGraph, tilingOptions, fusionOptions);
+        builder, ops, dependenceGraph, tilingOptions);
   default:;
   }
   return llvm::None;
 }

mlir/lib/Dialect/Linalg/Transforms/Transforms.cpp

LogicalResult mlir::linalg::LinalgBaseTileAndFusePattern::matchAndRewrite(
   LinalgOp linalgOp = dyn_cast<LinalgOp>(op);
   if (!linalgOp)
     return failure();
   if (failed(marker.checkAndNotify(rewriter, linalgOp)))
     return failure();
   if (!linalgOp.hasBufferSemantics())
     return failure();
 
+  llvm::SetVector<Operation *> producers;
+  producers.insert(linalgOp);
+  for (auto dependence : dependenceGraph.getDependentOperations(linalgOp)) {
+    if (!fusionOptions.indicesToFuse.count(
+            dependence.indexingOpView.operandNum))
+      continue;
+    if (isa<LinalgOp>(dependence.dependentOpView.op)) {
+      producers.insert(dependence.dependentOpView.op);
+    }
+  }
+
+  SmallVector<LinalgOp, 1> fusionOps;
+  for (auto it = op->getBlock()->begin(), ie = Block::iterator(op); it != ie;
+       ++it) {
+    auto producerLinalgOp = dyn_cast<LinalgOp>(&(*it));
+    if (producerLinalgOp && producers.count(producerLinalgOp))
+      fusionOps.push_back(producerLinalgOp);
+  }
+
+  SmallVector<LinalgTilingOptions, 2> tilingOptionsVec(fusionOps.size(),
+                                                       LinalgTilingOptions());
+  fusionOps.push_back(linalgOp);
+  tilingOptionsVec.push_back(tilingOptions);
+
   Optional<TiledAndFusedLinalgOps> tiledAndFusedOps = tileAndFuseLinalgOps(
-      rewriter, op, dependenceGraph, tilingOptions, fusionOptions);
+      rewriter, fusionOps, dependenceGraph, tilingOptionsVec);
   if (!tiledAndFusedOps)
     return failure();
   marker.replaceLinalgMarker(rewriter, tiledAndFusedOps->op.getOperation());
   for (auto fusedOp : tiledAndFusedOps->fusedProducers) {
     fusedOpMarker.replaceLinalgMarker(rewriter, fusedOp.getOperation());
   }
-  for (auto origProducerOp : tiledAndFusedOps->originalProducers)
+  for (auto origProducerOp : ArrayRef<LinalgOp>(fusionOps).drop_back()) {
     originalOpMarker.replaceLinalgMarker(rewriter,
                                          origProducerOp.getOperation());
+  }
   rewriter.updateRootInPlace(
       op, [&]() { originalOpMarker.replaceLinalgMarker(rewriter, op); });
   return success();
 }
 
 /// Linalg base interchange pattern.
 mlir::linalg::LinalgBaseInterchangePattern::LinalgBaseInterchangePattern(
     StringRef opName, MLIRContext *context,

mlir/test/Dialect/Linalg/fusion-pattern.mlir

 //      CHECK:     %[[SV2:.+]] = subview %[[ARG1]][0, %[[IV1]]]
 // CHECK-SAME:       %[[K_2]], %[[TILE_N]]
 //      CHECK:     %[[M_2:.+]] = dim %[[ARG2]], %[[C0]]
 //      CHECK:     %[[TILE_M_2:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M_2]]]
 //      CHECK:     %[[N_2:.+]] = dim %[[ARG2]], %[[C1]]
 //      CHECK:     %[[TILE_N_2:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N_2]]]
 //      CHECK:     %[[SV3:.+]] = subview %[[ARG2]][%[[IV0]], %[[IV1]]]
 // CHECK-SAME:       [%[[TILE_M_2]], %[[TILE_N_2]]]
-//      CHECK:     linalg.fill(%[[SV3]], %[[CST]])
+//      CHECK:     %[[SV3_2:.+]] = subview %[[ARG2]][%[[IV0]], %[[IV1]]]
+// CHECK-SAME:       [%[[TILE_M]], %[[TILE_N]]]
+//      CHECK:     linalg.fill(%[[SV3_2]], %[[CST]])
 // CHECK-SAME:       __internal_linalg_transform__ = "after_basic_fusion_producer"
 //      CHECK:     scf.for %[[IV2:.+]] = %[[C0]] to %[[K]] step %[[C16]] {
 //      CHECK:       %[[TILE_K:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[K]]]
 //      CHECK:       %[[SV4:.+]] = subview %[[SV1]][0, %[[IV2]]]
 // CHECK-SAME:         [%[[TILE_M]], %[[TILE_K]]]
 //      CHECK:       %[[TILE_K_2:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[K_2]]]
 //      CHECK:       %[[SV5:.+]] = subview %[[SV2]][%[[IV2]], 0]
 // CHECK-SAME:         [%[[TILE_K_2]], %[[TILE_N]]]

 //      CHECK:     %[[TILE_N:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[N]]]
 //      CHECK:     %[[SV1:.+]] = subview %[[ARG2]][0, %[[IV0]]]
 // CHECK-SAME:       [%[[K]], %[[TILE_N]]]
 //      CHECK:     %[[M:.+]] = dim %[[ARG3]], %[[C0]]
 //      CHECK:     %[[N_2:.+]] = dim %[[ARG3]], %[[C1]]
 //      CHECK:     %[[TILE_N_2:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[N_2]]]
 //      CHECK:     %[[SV2:.+]] = subview %[[ARG3]][0, %[[IV0]]]
 // CHECK-SAME:       [%[[M]], %[[TILE_N_2]]]
+//      CHECK:     %[[K_2:.+]] = dim %[[ARG1]], %[[C0]]
 //      CHECK:     %[[SV3:.+]] = subview %[[ARG1]][0, %[[IV0]]]
-// CHECK-SAME:       [%[[K]], %[[TILE_N]]]
-//      CHECK:     linalg.copy(%[[SV3]], %[[SV1]])
+// CHECK-SAME:       [%[[K_2]], %[[TILE_N]]]
+//      CHECK:     %[[SV3_2:.+]] = subview %[[ARG2]][0, %[[IV0]]]
+// CHECK-SAME:       [%[[K_2]], %[[TILE_N]]]
+//      CHECK:     linalg.copy(%[[SV3]], %[[SV3_2]])
 // CHECK-SAME:       __internal_linalg_transform__ = "after_rhs_fusion_producer"
 //  CHECK-NOT:     linalg.fill
 //  CHECK-DAG:     %[[M_2:.+]] = dim %[[ARG0]], %[[C0]]
 //  CHECK-DAG:     %[[K_2:.+]] = dim %[[ARG0]], %[[C1]]
 //      CHECK:     scf.parallel (%[[IV1:.+]]) =
 // CHECK-SAME:       (%[[C0]]) to (%[[M_2]]) step (%[[C32]]) {
 // CHECK-NEXT:       scf.for %[[IV2:.+]] = %[[C0]] to %[[K_2]] step %[[C16]] {
 //      CHECK:         %[[TILE_M:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[M_2]]]

 //      CHECK:     %[[K:.+]] = dim %[[ARG1]], %[[C1]]
 //      CHECK:     %[[SV1:.+]] = subview %[[ARG1]][%[[IV0]], 0]
 // CHECK-SAME:       [%[[TILE_M]], %[[K]]]
 //      CHECK:     %[[M_2:.+]] = dim %[[ARG3]], %[[C0]]
 //      CHECK:     %[[TILE_M_2:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M_2]]]
 //      CHECK:     %[[N:.+]] = dim %[[ARG3]], %[[C1]]
 //      CHECK:     %[[SV2:.+]] = subview %[[ARG3]][%[[IV0]], 0]
 // CHECK-SAME:       [%[[TILE_M_2]], %[[N]]]
+//      CHECK:     %[[SV2_2:.+]] = subview %[[ARG3]][%[[IV0]], 0]
+// CHECK-SAME:       [%[[TILE_M]], %[[N]]]
+//      CHECK:     %[[K_2:.+]] = dim %[[ARG0]], %[[C1]]
 //      CHECK:     %[[SV3:.+]] = subview %[[ARG0]][%[[IV0]], 0]
-// CHECK-SAME:       [%[[TILE_M]], %[[K]]]
-//      CHECK:     linalg.copy(%[[SV3]], %[[SV1]])
+// CHECK-SAME:       [%[[TILE_M]], %[[K_2]]]
+//      CHECK:     %[[SV1_2:.+]] = subview %[[ARG1]][%[[IV0]], 0]
+// CHECK-SAME:       [%[[TILE_M]], %[[K_2]]]
+//      CHECK:     linalg.copy(%[[SV3]], %[[SV1_2]])
 // CHECK-SAME:       __internal_linalg_transform__ = "after_two_operand_fusion_producer"
-//      CHECK:     linalg.fill(%[[SV2]], %[[CST]])
+//      CHECK:     linalg.fill(%[[SV2_2]], %[[CST]])
 // CHECK-SAME:       __internal_linalg_transform__ = "after_two_operand_fusion_producer"
 //  CHECK-DAG:     %[[N_2:.+]] = dim %[[ARG2]], %[[C1]]
 //      CHECK:     scf.parallel (%[[IV1:.+]]) =
 // CHECK-SAME:       (%[[C0]]) to (%[[N_2]]) step (%[[C64]]) {
 // CHECK-NEXT:       scf.for %[[IV2:.+]] = %[[C0]] to %[[K]] step %[[C16]] {
 //      CHECK:         %[[TILE_K:.+]] = affine.min #[[MAP2]](%[[IV2]])[%[[K]]]
 //      CHECK:         %[[SV4:.+]] = subview %[[SV1]][0, %[[IV2]]]
 // CHECK-SAME:           [%[[TILE_M]], %[[TILE_K]]]

 //  CHECK-DAG:   %[[C64:.+]] = constant 64 : index
 //  CHECK-DAG:   %[[C16:.+]] = constant 16 : index
 //      CHECK:   linalg.matmul
 // CHECK-SAME:     __internal_linalg_transform__ = "after_lhs_fusion_original"
 //  CHECK-DAG:   %[[M:.+]] = dim %[[ARG2]], %[[C0]]
 //      CHECK:   scf.parallel (%[[IV0:.+]]) =
 // CHECK-SAME:     (%[[C0]]) to (%[[M]]) step (%[[C32]]) {
 //      CHECK:     %[[TILE_M:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M]]]
-//      CHECK:     %[[K2:.+]] = dim %[[ARG2]], %[[C1]]
+//      CHECK:     %[[K:.+]] = dim %[[ARG2]], %[[C1]]
 //      CHECK:     %[[SV1:.+]] = subview %[[ARG2]][%[[IV0]], 0]
-// CHECK-SAME:       [%[[TILE_M]], %[[K2]]]
+// CHECK-SAME:       [%[[TILE_M]], %[[K]]]
 //      CHECK:     %[[M_2:.+]] = dim %[[ARG4]], %[[C0]]
 //      CHECK:     %[[TILE_M_2:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M_2]]]
 //      CHECK:     %[[N:.+]] = dim %[[ARG4]], %[[C1]]
 //      CHECK:     %[[SV2:.+]] = subview %[[ARG4]][%[[IV0]], 0]
 // CHECK-SAME:       [%[[TILE_M_2]], %[[N]]]
-//      CHECK:     %[[K1:.+]] = dim %[[ARG0]], %[[C1]]
+//      CHECK:     %[[N_2:.+]] = dim %[[ARG1]], %[[C1]]
+//      CHECK:     %[[K_2:.+]] = dim %[[ARG0]], %[[C1]]
 //      CHECK:     %[[SV3:.+]] = subview %[[ARG0]][%[[IV0]], 0]
-// CHECK-SAME:       [%[[TILE_M]], %[[K1]]]
-//      CHECK:     %[[SV4:.+]] = subview %[[ARG1]][0, 0] [%[[K1]], %[[K2]]]
+// CHECK-SAME:       [%[[TILE_M]], %[[K_2]]]
+//      CHECK:     %[[SV4:.+]] = subview %[[ARG1]][0, 0] [%[[K_2]], %[[N_2]]]
+//      CHECK:     %[[SV5:.+]] = subview %[[ARG2]][%[[IV0]], 0]
+//      CHECK:       [%[[TILE_M]], %[[N_2]]]
 //      CHECK:     linalg.matmul
 // CHECK-SAME:         __internal_linalg_transform__ = "after_lhs_fusion_producer"
 // CHECK-SAME:         ins(%[[SV3]], %[[SV4]]
 // CHECK-SAME:           : memref<?x?xf32, #[[MAP1]]>, memref<?x?xf32, #[[MAP1]]>)
-// CHECK-SAME:         outs(%[[SV1]] : memref<?x?xf32, #[[MAP1]]>)
+// CHECK-SAME:         outs(%[[SV5]] : memref<?x?xf32, #[[MAP1]]>)
 //  CHECK-DAG:     %[[N_2:.+]] = dim %[[ARG3]], %[[C1]]
 //      CHECK:     scf.parallel (%[[IV1:.+]]) =
 // CHECK-SAME:       (%[[C0]]) to (%[[N_2]]) step (%[[C64]]) {
 // CHECK-NEXT:       scf.for %[[IV2:.+]] = %[[C0]] to %[[K]] step %[[C16]] {
 //      CHECK:         %[[TILE_K:.+]] = affine.min #[[MAP2]](%[[IV2]])[%[[K]]]
 //      CHECK:         %[[SV6:.+]] = subview %[[SV1]][0, %[[IV2]]]
 // CHECK-SAME:           [%[[TILE_M]], %[[TILE_K]]]
 //      CHECK:         %[[K_2:.+]] = dim %[[ARG3]], %[[C0]]

mlir/test/Dialect/Linalg/fusion-sequence.mlir

+// RUN: mlir-opt -pass-pipeline="test-linalg-tile-and-fuse{tile-sizes=16,32,64},canonicalize,cse" -split-input-file %s | FileCheck %s
+
+module {
+  func @three_op_fusion(%arg0: memref<?x?xf32>, %arg1: memref<?x?xf32>,
+                        %arg2: memref<?xf32>, %arg3 : memref<?x?xf32>) {
+    %cst = constant 0.000000e+00 : f32
+    %c0 = constant 0 : index
+    %c1 = constant 1 : index
+    %d0 = dim %arg0, %c0 : memref<?x?xf32>
+    %d1 = dim %arg1, %c1 : memref<?x?xf32>
+    %0 = alloc(%d0, %d1) : memref<?x?xf32>
+    linalg.fill(%0, %cst) : memref<?x?xf32>, f32
+    linalg.matmul ins(%arg0, %arg1 : memref<?x?xf32>, memref<?x?xf32>)
+      outs(%0 : memref<?x?xf32>)
+    linalg.generic
+      {indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>,
+                        affine_map<(d0, d1) -> (d1)>,
+                        affine_map<(d0, d1) -> (d0, d1)>],
+       iterator_types = ["parallel", "parallel"]}
+      ins(%0, %arg2 : memref<?x?xf32>, memref<?xf32>)
+      outs(%arg3 : memref<?x?xf32>) {
+      ^bb0(%arg4 : f32, %arg5 : f32, %arg6 : f32) :
+        %5 = addf %arg4, %arg5 : f32
+        linalg.yield %5 : f32
+      }
+    return
+  }
+}
+
+//   CHECK-DAG: #[[MAP2:.+]] = affine_map<(d0, d1)[s0, s1] -> (d0 * s1 + s0 + d1)>
+//   CHECK-DAG: #[[MAP3:.+]] = affine_map<(d0)[s0] -> (d0 + s0)>
+//       CHECK: func @three_op_fusion
+//  CHECK-SAME:   %[[ARG0:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG1:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG2:[a-zA-Z0-9_]+]]: memref<?xf32>
+//  CHECK-SAME:   %[[ARG3:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//       CHECK:   %[[TEMP:.+]] = alloc(%{{.*}}, %{{.*}}) : memref<?x?xf32>
+//       CHECK:   scf.parallel (%[[IV0:.+]], %[[IV1:.+]]) = {{.*}} {
+//   CHECK-DAG:     %[[SV_TEMP:.+]] = subview %[[TEMP]][%[[IV0]], %[[IV1]]]
+//   CHECK-DAG:     %[[SV_ARG2:.+]] = subview %[[ARG2]][%[[IV1]]]
+//   CHECK-DAG:     %[[SV_ARG3:.+]] = subview %[[ARG3]][%[[IV0]], %[[IV1]]]
+//   CHECK-DAG:     %[[SV_ARG0:.+]] = subview %[[ARG0]][%[[IV0]], 0]
+//   CHECK-DAG:     %[[SV_ARG1:.+]] = subview %[[ARG1]][0, %[[IV1]]]
+//       CHECK:     linalg.fill(%[[SV_TEMP]], %{{.+}})
+//       CHECK:     scf.for %[[IV2:.+]] = %{{.+}} to %{{.+}} step %{{.+}} {
+//       CHECK:       %[[SV_ARG0_K:.+]] = subview %[[SV_ARG0]][0, %[[IV2]]]
+//       CHECK:       %[[SV_ARG1_K:.+]] = subview %[[SV_ARG1]][%[[IV2]], 0]
+//       CHECK:       linalg.matmul
+//  CHECK-SAME:         ins(%[[SV_ARG0_K]], %[[SV_ARG1_K]]
+//  CHECK-SAME:           : memref<?x?xf32, #[[MAP2]]>, memref<?x?xf32, #[[MAP2]]>)
+//  CHECK-SAME:         outs(%[[SV_TEMP]] : memref<?x?xf32, #[[MAP2]]>)
+//       CHECK:     }
+//       CHECK:     linalg.generic
+//  CHECK-SAME:       ins(%[[SV_TEMP]], %[[SV_ARG2]]
+//  CHECK-SAME:         : memref<?x?xf32, #[[MAP2]]>, memref<?xf32, #[[MAP3]]>)
+//  CHECK-SAME:       outs(%[[SV_ARG3]] : memref<?x?xf32, #[[MAP2]]>)
+//       CHECK:     scf.yield
+//       CHECK:   }
+
+// -----
+
+module {
+  func @sequence_of_matmul(%arg0: memref<?x?xf32>, %arg1: memref<?x?xf32>,
+                           %arg2: memref<?x?xf32>, %arg3: memref<?x?xf32>,
+         %arg4: memref<?x?xf32>) {
+    %cst = constant 0.000000e+00 : f32
+    %c0 = constant 0 : index
+    %c1 = constant 1 : index
+    %m = dim %arg0, %c0 : memref<?x?xf32>
+    %n1 = dim %arg1, %c1 : memref<?x?xf32>
+    %n2 = dim %arg2, %c1 : memref<?x?xf32>
+    %n3 = dim %arg3, %c1 : memref<?x?xf32>
+    %0 = alloc(%m, %n1) : memref<?x?xf32>
+    %1 = alloc(%m, %n2) : memref<?x?xf32>
+    linalg.fill(%0, %cst) : memref<?x?xf32>, f32
+    linalg.matmul ins(%arg0, %arg1 : memref<?x?xf32>, memref<?x?xf32>)
+      outs(%0 : memref<?x?xf32>)
+    linalg.fill(%1, %cst) : memref<?x?xf32>, f32
+    linalg.matmul ins(%0, %arg2 : memref<?x?xf32>, memref<?x?xf32>)
+      outs(%1 : memref<?x?xf32>)
+    linalg.fill(%arg4, %cst) : memref<?x?xf32>, f32
+    linalg.matmul ins(%1, %arg3 : memref<?x?xf32>, memref<?x?xf32>)
+      outs(%arg4 : memref<?x?xf32>)
+    return
+  }
+}
+
+//   CHECK-DAG: #[[MAP0:.+]] = affine_map<(d0)[s0] -> (16, -d0 + s0)>
+//   CHECK-DAG: #[[MAP1:.+]] = affine_map<(d0, d1)[s0, s1] -> (d0 * s1 + s0 + d1)>
+//   CHECK-DAG: #[[MAP2:.+]] = affine_map<(d0)[s0] -> (32, -d0 + s0)>
+//   CHECK-DAG: #[[MAP3:.+]] = affine_map<(d0)[s0] -> (64, -d0 + s0)>
+//       CHECK: func @sequence_of_matmul
+//  CHECK-SAME:   %[[ARG0:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG1:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG2:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG3:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//  CHECK-SAME:   %[[ARG4:[a-zA-Z0-9_]+]]: memref<?x?xf32>
+//   CHECK-DAG:   %[[C0:.+]] = constant 0 : index
+//   CHECK-DAG:   %[[C1:.+]] = constant 1 : index
+//   CHECK-DAG:   %[[C16:.+]] = constant 16 : index
+//   CHECK-DAG:   %[[C32:.+]] = constant 32 : index
+//   CHECK-DAG:   %[[C64:.+]] = constant 64 : index
+//   CHECK-DAG:   %[[M:.+]] = dim %[[ARG0]], %[[C0]]
+//   CHECK-DAG:   %[[N1:.+]] = dim %[[ARG1]], %[[C1]]
+//   CHECK-DAG:   %[[N2:.+]] = dim %[[ARG2]], %[[C1]]
+//       CHECK:   %[[ALLOC1:.+]] = alloc(%[[M]], %[[N1]])
+//       CHECK:   %[[ALLOC2:.+]] = alloc(%[[M]], %[[N2]])
+//       CHECK:   scf.parallel (%[[IV0:.+]]) = (%[[C0]]) to (%[[M]])
+//  CHECK-SAME:     step (%[[C16]]) {
+//       CHECK:     %[[TILE_M:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M]]]
+//       CHECK:     %[[SV_ALLOC2:.+]] = subview %[[ALLOC2]][%[[IV0]], 0]
+//  CHECK-SAME:       [%[[TILE_M]], %[[N2]]]
+//       CHECK:     %[[M_2:.+]] = dim %[[ARG4]], %[[C0]]
+//       CHECK:     %[[TILE_M_2:.+]] = affine.min #[[MAP0]](%[[IV0]])[%[[M_2]]]
+//       CHECK:     %[[N3:.+]] = dim %[[ARG4]], %[[C1]]
+//       CHECK:     %[[SV_ARG4:.+]] = subview %[[ARG4]][%[[IV0]], 0]
+//  CHECK-SAME:       [%[[TILE_M_2]], %[[N3]]]
+//       CHECK:     %[[SV_ARG4_2:.+]] = subview %[[ARG4]][%[[IV0]], 0]
+//  CHECK-SAME:       [%[[TILE_M]], %[[N3]]]
+//       CHECK:     %[[SV_ALLOC1:.+]] = subview %[[ALLOC1]][%[[IV0]], 0]
+//  CHECK-SAME:       [%[[TILE_M]], %[[N1]]]
+//       CHECK:     %[[SV_ARG2:.+]] = subview %[[ARG2]][0, 0] [%[[N1]], %[[N2]]]
+//       CHECK:     %[[N0:.+]] = dim %[[ARG0]], %[[C1]]
+//       CHECK:     %[[SV_ARG0:.+]] = subview %[[ARG0]][%[[IV0]], 0]
+//  CHECK-SAME:       [%[[TILE_M:.+]], %[[N0]]]
+//       CHECK:     %[[SV_ARG1:.+]] = subview %[[ARG1]][0, 0] [%[[N0]], %[[N1]]]
+
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N1]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       %[[TILE_N1:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N1]]]
+//       CHECK:       %[[SV_SV_ALLOC1:.+]] = subview %[[SV_ALLOC1]][0, %[[IV1]]]
+//  CHECK-SAME:         [%[[TILE_M]], %[[TILE_N1]]]
+//       CHECK:       linalg.fill(%[[SV_SV_ALLOC1]], %{{.+}})
+//       CHECK:     }
+
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N1]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       scf.for %[[IV2:.+]] = %[[C0]] to %[[N0]] step %[[C64]] {
+//       CHECK:         %[[TILE_N0:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[N0]]]
+//       CHECK:         %[[SV_SV_ARG0:.+]] = subview %[[SV_ARG0]][0, %[[IV2]]]
+//  CHECK-SAME:           [%[[TILE_M]], %[[TILE_N0]]]
+//       CHECK:         %[[TILE_N1:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N1]]]
+//       CHECK:         %[[SV_SV_ARG1:.+]] = subview %[[SV_ARG1]][%[[IV2]], %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_N0]], %[[TILE_N1]]]
+//       CHECK:         %[[SV_SV_ALLOC1:.+]] = subview %[[SV_ALLOC1]][0, %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_M]], %[[TILE_N1]]]
+//       CHECK:         linalg.matmul ins(%[[SV_SV_ARG0]], %[[SV_SV_ARG1]]
+//  CHECK-SAME:            : memref<?x?xf32, #[[MAP1]]>, memref<?x?xf32, #[[MAP1]]>)
+//  CHECK-SAME:            outs(%[[SV_SV_ALLOC1]] : memref<?x?xf32, #[[MAP1]]>)
+//       CHECK:       }
+//       CHECK:       scf.yield
+//       CHECK:     }
+
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N2]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       %[[TILE_N2:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N2]]]
+//       CHECK:       %[[SV_SV_ALLOC2:.+]] = subview %[[SV_ALLOC2]][0, %[[IV1]]]
+//  CHECK-SAME:         [%[[TILE_M]], %[[TILE_N2]]]
+//       CHECK:       linalg.fill(%[[SV_SV_ALLOC2]], %{{.+}})
+//       CHECK:     }
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N2]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       scf.for %[[IV2:.+]] = %[[C0]] to %[[N1]] step %[[C64]] {
+//       CHECK:         %[[TILE_N1:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[N1]]]
+//       CHECK:         %[[SV_SV_ALLOC1:.+]] = subview %[[SV_ALLOC1]][0, %[[IV2]]]
+//  CHECK-SAME:           [%[[TILE_M]], %[[TILE_N1]]]
+//       CHECK:         %[[TILE_N2:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N2]]]
+//       CHECK:         %[[SV_SV_ARG2:.+]] = subview %[[SV_ARG2]][%[[IV2]], %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_N1]], %[[TILE_N2]]]
+//       CHECK:         %[[SV_SV_ALLOC2:.+]] = subview %[[SV_ALLOC2]][0, %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_M]], %[[TILE_N2]]]
+//       CHECK:         linalg.matmul ins(%[[SV_SV_ALLOC1]], %[[SV_SV_ARG2]]
+//  CHECK-SAME:            : memref<?x?xf32, #[[MAP1]]>, memref<?x?xf32, #[[MAP1]]>)
+//  CHECK-SAME:            outs(%[[SV_SV_ALLOC2]] : memref<?x?xf32, #[[MAP1]]>)
+//       CHECK:       }
+//       CHECK:       scf.yield
+//       CHECK:     }
+
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N3]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       %[[TILE_N3:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N3]]]
+//       CHECK:       %[[SV_SV_ARG4:.+]] = subview %[[SV_ARG4_2]][0, %[[IV1]]]
+//  CHECK-SAME:         [%[[TILE_M]], %[[TILE_N3]]]
+//       CHECK:       linalg.fill(%[[SV_SV_ARG4]], %{{.+}})
+//       CHECK:     }
+//       CHECK:     %[[N3_2:.+]] = dim %[[ARG3]], %[[C1]]
+//       CHECK:     scf.parallel (%[[IV1:.+]]) = (%[[C0]]) to (%[[N3_2]])
+//  CHECK-SAME:       step (%[[C32]]) {
+//       CHECK:       scf.for %[[IV2:.+]] = %[[C0]] to %[[N2]] step %[[C64]] {
+//       CHECK:         %[[TILE_N2:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[N2]]]
+//       CHECK:         %[[SV_SV_ALLOC2:.+]] = subview %[[SV_ALLOC2]][0, %[[IV2]]]
+//  CHECK-SAME:           [%[[TILE_M]], %[[TILE_N2]]]
+//       CHECK:         %[[N2_2:.+]] = dim %[[ARG3]], %[[C0]]
+//       CHECK:         %[[TILE_N2:.+]] = affine.min #[[MAP3]](%[[IV2]])[%[[N2_2]]]
+//       CHECK:         %[[TILE_N3:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N3_2]]]
+//       CHECK:         %[[SV_ARG3:.+]] = subview %[[ARG3]][%[[IV2]], %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_N2]], %[[TILE_N3]]]
+//       CHECK:         %[[TILE_N3_2:.+]] = affine.min #[[MAP2]](%[[IV1]])[%[[N3]]]
+//       CHECK:         %[[SV_SV_ARG4:.+]] = subview %[[SV_ARG4]][0, %[[IV1]]]
+//  CHECK-SAME:           [%[[TILE_M_2]], %[[TILE_N3_2]]]
+//       CHECK:         linalg.matmul ins(%[[SV_SV_ALLOC2]], %[[SV_ARG3]]
+//  CHECK-SAME:            : memref<?x?xf32, #[[MAP1]]>, memref<?x?xf32, #[[MAP1]]>)
+//  CHECK-SAME:            outs(%[[SV_SV_ARG4]] : memref<?x?xf32, #[[MAP1]]>)
+//       CHECK:       }
+//       CHECK:       scf.yield
+//       CHECK:     }
+

mlir/test/lib/Transforms/TestLinalgFusionTransforms.cpp

while (succeeded(fuseLinalgOpsGreedily(getFunction()))) {
       pm.addPass(createCanonicalizerPass());
       pm.addPass(createCSEPass());
       LogicalResult res = pm.run(getFunction().getParentOfType<ModuleOp>());
       if (failed(res))
         this->signalPassFailure();
     }
   }
 };
+
+/// Pass to test tile and fuse of sequence of operations. Intended only for
+/// testing.
+struct TestLinalgTileAndFuseSequencePass
+    : public PassWrapper<TestLinalgTileAndFuseSequencePass, FunctionPass> {
+  TestLinalgTileAndFuseSequencePass() = default;
+  TestLinalgTileAndFuseSequencePass(
+      const TestLinalgTileAndFuseSequencePass &pass){};
+
+  ListOption<int64_t> tileSizes{
+      *this, "tile-sizes", llvm::cl::desc("Tile sizes to use for ops"),
+      llvm::cl::ZeroOrMore, llvm::cl::MiscFlags::CommaSeparated};
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<AffineDialect, linalg::LinalgDialect, scf::SCFDialect>();
+  }
+
+  void runOnFunction() override {
+    FuncOp funcOp = getOperation();
+    auto &blocks = funcOp.getBody().getBlocks();
+    if (!llvm::hasSingleElement(blocks)) {
+      return;
+    }
+    SmallVector<LinalgOp, 2> linalgOps =
+        llvm::to_vector<2>(blocks.front().getOps<LinalgOp>());
+    Aliases aliases;
+    LinalgDependenceGraph dependenceGraph(aliases, linalgOps);
+    SmallVector<LinalgTilingOptions, 4> tilingOptions(
+        linalgOps.size(),
+        LinalgTilingOptions().setTileSizes(tileSizes).setLoopType(
+            LinalgTilingLoopType::ParallelLoops));
+    OpBuilder builder(funcOp.getContext());
+    Optional<TiledAndFusedLinalgOps> tileAndFuseOps = tileAndFuseLinalgOps(
+        builder, linalgOps, dependenceGraph, tilingOptions);
+    if (!tileAndFuseOps)
+      return signalPassFailure();
+    for (auto op : linalgOps)
+      builder.eraseOp(op);
+  }
+};
 } // namespace
 
 namespace mlir {
 namespace test {
 void registerTestLinalgFusionTransforms() {
   PassRegistration<TestLinalgFusionTransforms> testFusionTransformsPass(
       "test-linalg-fusion-transform-patterns",
       "Test Linalg fusion transformation patterns by applying them greedily.");
 }
 void registerTestLinalgGreedyFusion() {
   PassRegistration<TestLinalgGreedyFusion> testFusionTransformsPass(
       "test-linalg-greedy-fusion",
       "Test Linalg fusion by applying a greedy test transformation.");
 }
+void registerTestLinalgTileAndFuseSequencePass() {
+  PassRegistration<TestLinalgTileAndFuseSequencePass>
+      testTileAndFuseSequencePass(
+          "test-linalg-tile-and-fuse",
+          "Test Linalg tiling and fusion of a sequence of Linalg operations.");
+}
+
 } // namespace test
 } // namespace mlir

mlir/tools/mlir-opt/mlir-opt.cpp

 void registerTestExpandTanhPass();
 void registerTestFinalizingBufferizePass();
 void registerTestGpuParallelLoopMappingPass();
 void registerTestInterfaces();
 void registerTestLinalgCodegenStrategy();
 void registerTestLinalgFusionTransforms();
 void registerTestLinalgGreedyFusion();
 void registerTestLinalgHoisting();
+void registerTestLinalgTileAndFuseSequencePass();
 void registerTestLinalgTransforms();
 void registerTestLivenessPass();
 void registerTestLoopFusion();
 void registerTestLoopMappingPass();
 void registerTestLoopUnrollingPass();
 void registerTestMemRefDependenceCheck();
 void registerTestMemRefStrideCalculation();
 void registerTestOpaqueLoc();
#endif
   test::registerTestExpandTanhPass();
   test::registerTestFinalizingBufferizePass();
   test::registerTestGpuParallelLoopMappingPass();
   test::registerTestInterfaces();
   test::registerTestLinalgCodegenStrategy();
   test::registerTestLinalgFusionTransforms();
   test::registerTestLinalgGreedyFusion();
   test::registerTestLinalgHoisting();
+  test::registerTestLinalgTileAndFuseSequencePass();
   test::registerTestLinalgTransforms();
   test::registerTestLivenessPass();
   test::registerTestLoopFusion();
   test::registerTestLoopMappingPass();
   test::registerTestLoopUnrollingPass();
   test::registerTestMemRefDependenceCheck();
   test::registerTestMemRefStrideCalculation();
   test::registerTestOpaqueLoc();