[mlir][TilingInterface] Add mode to tile + fuse that allows yielding the replacements for fused producers.

[mravishankar]

Summary

Tile and fuse using TilingInterface allows fusing producers with
consumers at tile granularity, i.e. using a tiled implementation of
the producer to compute the tile needed for producer a tile of the
consumer. Current implementation of tile and fuse only yields the
value to replace the consumer. In some cases it is also useful to have
the loop nest yield a value to replace the tiled + fused producer.
This is only valid to do in cases where the producer is not computed
in a redundant fashion, i.e. after tiling the innermost loop body
produces a disjoint portion of the producer. This patch modifies the
implementation of tile and fuse to support this use case.

Currently there is no way to know automatically if there are redundant
computations introduced while fusing the producer. So a new option is
added which allows the caller to assert that this is indeed the case,
i.e. the tile sizes selected for the consumer are such that this
assumption can be made. Under this option, the generated tiled loop
nest also yeilds a value that can be used to replace the untiled
producer.

[nicolasvasilache]

Putting a blocker until I had time to review.
At first sight this looks even more complicated than https://reviews.llvm.org/D132085 for which the consensus was to break into small independent pieces, so without going into details yet this raises a few red flags for me.

[nicolasvasilache]

mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h

67

Quick fly-by comment.

Genereally we want to go towards SubsetInsert/Extract interfaces, committing more APIs to offsets + sizes is not where we want to go IMO.

[mravishankar]

In D133474#3776709, @nicolasvasilache wrote:

Putting a blocker until I had time to review.
At first sight this looks even more complicated than https://reviews.llvm.org/D132085 for which the consensus was to break into small independent pieces, so without going into details yet this raises a few red flags for me.

A lot of it is refactoring, and happy to iterate, but AFAICS it is definitely much less complicated than the earlier attempt. I'll leave some high level pointers here hoping it helps in reviewing

1. Most the logic is done by three methods
   • tileConsumer which generates the tile loop nest and tiles the consumer op. This is mostly refactoring.
   • fuseProducer which tiles and fuses a producer with all currently tiled and fused operations. While the implementation itself is done using swap of op -> tensor.extract_slice pattern, but refactoring the logic into using an "operation centric" view makes it much easier to handle the case where the producer values also needs to be produced by the tiled loop.
   • yieldTiledValues which modifies loop nest to yield values from within the tiled loop body. THe key here is that neither tileConsumer nor fuseProducer does this, i.e. after each of those calls the scf.for nest created does not yield anything yet. The reason being having/manipulating iter_args is extremely involved, and hard to reason about. Instead the core transformations just create the loop nest and body, and this method orchestrates the yield. This separation makes sense to me since the use of iter_args etc. is very tied to use of scf.for. So while the first two methods are not really tied to use of scf.for (any looping construct would work), this method manages the complications with iter_args.

The cadence of the transformation is fairly easy to follow (see the returningMatchAndRewrite methods for the tiling pattern and tile + fuse pattern). The rest is just details. I am still working out a few kinks in those details, but the generated code matches what I would expect. So in my mind this decomposition makes sense. I am happy to iterate further. Each of these methods have a lot of detail, but "what they do" is fairly well defined. Note that the part of yielding replacements for the fused producer is all gaurded by a flag. The rest of the changes are just NFC. I am happy to stress test it and see where it leads, drop it if there is a better way to do it. Personally I am confident this approach is more maintainable. This is my second attempt at this :) , and is a real blocker at this point (cant kick this can further).

[mravishankar]

Rebase.

[mravishankar]

Remove dropping of dead yields from tiled loop. That needs to be revisited later.

[mravishankar]

Abandoning this one to try a different approach.

mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h

SCFTilingOptions &setInterchange(ArrayRef<unsigned> interchange) {
     interchangeVector = llvm::to_vector(interchange);
     return *this;
   }
 };
 
 struct SCFTilingResult {
   Operation *tiledOp;
   SmallVector<scf::ForOp> loops;
+  llvm::SmallBitVector tiledLoops;
+  SmallVector<OpFoldResult> tileOffsets;
+  SmallVector<OpFoldResult> tileSizes;
 };
 
 /// Pattern to tile an op that implements the `TilingInterface` using
 /// `scf.for` for iterating over the tiles.
 struct TileUsingSCFForOp : public OpInterfaceRewritePattern<TilingInterface> {
   /// Construct a generic pattern applied to all TilingInterface ops.
   TileUsingSCFForOp(MLIRContext *context, SCFTilingOptions options,
                     PatternBenefit benefit = 1);

 /// ```mlir
 /// %0 = linalg.fill ...
 /// %1 = linalg.matmul ... outs(%0 : ...) ...
 /// ```
 ///
 /// it is legal to fuse the fill with the matmul only if the matmul is tiled
 /// along the parallel dimensions and not the reduction dimension, i.e. the tile
 /// size for the reduction dimension should be 0.
+
+struct SCFTileAndFuseOptions {
+  SCFTilingOptions tilingOptions;
+  SCFTileAndFuseOptions &setTilingOptions(SCFTilingOptions options) {
+    tilingOptions = options;
+    return *this;
+  }
+
+  /// When the access pattern between producer and consumer is such that
+  /// the producer is fused with the consumer without the producer being
+  /// recomputed, then it is possible to yield the value of the producer from
+  /// within the loop nest. For example, consider
+  //
+  /// ```mlir
+  /// %0 = linalg.matmul ins(%lhs0, %rhs0) outs(%init0)
+  /// %1 = linalg.matmul ins(%0, %rhs1) outs(%init1)
+  /// ```
+  ///
+  /// If the tile sizes chosen are such that the second `linalg.matmul`
+  /// is tiled along the outer two dimensions of the op, then fusing
+  /// the first `linalg.matmul` using tile and fuse results in
+  /// recomputation of parts of the fused producer during computation
+  /// of different tiles of the consumer. Instead if only the outer dimension
+  /// is chosen, then the producer is not recomputed.
+  ///
+  /// ```mlir
+  /// scf.for %iv0 =
+  ///    %lhs0_slice = tensor.extract_slice %lhs0[%iv0, 0]
+  ///    %rhs0_slice = tensor.extract_slice %rhs0[0, 0]
+  ///    %init0_slice = tensor.extract_slice %init0[%iv0, 0]
+  ///    %0 = linalg.matmul ins(%lhs0_slice, %rhs0_slice) outs(%init0_slice)
+  ///    %rhs1_slice = tensor.extract_slice %rhs1[0, 0]
+  ///    %init1_slice = tensor.extract_slice %init1[%iv0, 0]
+  ///    %1 = linalg.matmul ins(%0, %rhs1_slice) outs(%init1_slice)
+  /// ```
+  ///
+  /// If needed the value of the untiled first matmul can be reconstructed
+  /// using the tiled and fused operation (similar to how the replacement of the
+  /// consumer is done).
+  /// It is unclear how to automatically determine when the producer is
+  /// recomputed and when it is not (especially through the `TilingInterface`).
+  /// So for now this is left as a option for the caller which is expected to
+  /// set the tile sizes appropriately to ensure the producer is not recomputed.
+  /// With this based on the uses of the untiled producer, a replacement value
+  /// for this is yielded by the tiled loop nest.
+  ///
+  /// One way to ensure this in the producer is when the producer and consumer
+  /// are LinalgOps, is
+  /// ```cpp
+  /// bool canProducerBeFusedWithoutRedundantComputation(
+  ///     llvm::SmallBitVector tiledLoops, OpOperand *fusedOperand) {
+  ///   auto consumerOp = cast<LinalgOp>(fusedOperand->getOwner());
+  ///   AffineMap consumerIndexingMap =
+  ///     consumerOp.getTiedIndexingMap(fusedOperand);
+  ///   AffineMap projectedConsumerMap =
+  ///     getProjectedMap(consumerOp, tiledLoops);
+  ///   AffineMap projectedProducerMap =
+  ///     getProjectedMap(producerOp, tiledLoops);
+  ///   return projectedConsumerMap.isIdentity();
+  /// }
+  /// ```
+  bool producerCanBeFusedWithoutRedundantComputations = false;
+  SCFTileAndFuseOptions &
+  setProducerCanBeFusedWithoutRedundantComputations(bool val) {
+    producerCanBeFusedWithoutRedundantComputations = val;
+    return *this;
+  }
+};
 struct SCFTileAndFuseResult {
+  SmallVector<Operation *> fusableProducers;
   SmallVector<Operation *> tiledAndFusedOps;
   SmallVector<scf::ForOp> loops;
 };
-struct TileConsumerAndFuseProducersUsingSCFForOp
+
+struct TileConsumerAndFuseProducersGreedilyUsingSCFForOp
     : public OpInterfaceRewritePattern<TilingInterface> {
 
   /// Construct a generic pattern applied to all TilingInterface ops.
-  TileConsumerAndFuseProducersUsingSCFForOp(MLIRContext *context,
-                                            SCFTilingOptions options,
-                                            PatternBenefit benefit = 1);
+  TileConsumerAndFuseProducersGreedilyUsingSCFForOp(
+      MLIRContext *context, SCFTileAndFuseOptions options,
 
   /// Construct a generic pattern applied to `opName`.
-  TileConsumerAndFuseProducersUsingSCFForOp(StringRef opName,
-                                            MLIRContext *context,
-                                            SCFTilingOptions options,
-                                            PatternBenefit benefit = 1);
+  TileConsumerAndFuseProducersGreedilyUsingSCFForOp(
+      StringRef opName, MLIRContext *context, SCFTileAndFuseOptions options,
 
   /// `matchAndRewrite` implementation that returns the significant transformed
   /// pieces of IR.
   FailureOr<SCFTileAndFuseResult>
   returningMatchAndRewrite(TilingInterface op, PatternRewriter &rewriter) const;
 
   LogicalResult matchAndRewrite(TilingInterface op,
                                 PatternRewriter &rewriter) const override {
     return returningMatchAndRewrite(op, rewriter);
   }
 
 private:
-  /// This pattern uses the tiling pattern. Instead of using inheritance, use
-  /// the patterns as private object that is instantiated at the same time as
-  /// this pattern.
-  TileUsingSCFForOp tilingPattern;
+  SCFTileAndFuseOptions tileAndFuseOptions;
 };
 
 /// Pattern to lower operations that implement the `TilingInterface` to
 /// loops/scalar IR using `scf.for`.
 struct LowerToLoopsUsingSCFForOp
     : public OpInterfaceRewritePattern<TilingInterface> {
   using OpInterfaceRewritePattern<TilingInterface>::OpInterfaceRewritePattern;
 

mlir/include/mlir/Dialect/SCF/Utils/Utils.h

 /// values returned by the callback should match the number of new
 /// initialization values. This function
 /// - Moves (i.e. doesnt clone) operations from the `loop` to the newly created
 ///   loop
 /// - Replaces the uses of `loop` with the new loop.
 /// - `loop` isnt erased, but is left in a "no-op" state where the body of the
 ///   loop just yields the basic block arguments that correspond to the
 ///   initialization values of a loop. The loop is dead after this method.
-/// - All uses of the `newIterOperands` within the generated new loop
-///   are replaced with the corresponding `BlockArgument` in the loop body.
 using NewYieldValueFn = std::function<SmallVector<Value>(
     OpBuilder &b, Location loc, ArrayRef<BlockArgument> newBBArgs)>;
 scf::ForOp replaceLoopWithNewYields(OpBuilder &builder, scf::ForOp loop,
                                     ValueRange newIterOperands,
                                     const NewYieldValueFn &newYieldValuesFn);
 
 /// Update a perfectly nested loop nest to yield new values from the innermost
 /// loop and propagating it up through the loop nest. This function
 /// - Expects `loopNest` to be a perfectly nested loop with outer most loop
 ///   first and innermost loop last.
 /// - `newIterOperands` are the initialization values to be used for the
 ///    outermost loop
 /// - `newYielValueFn` is the callback that generates the new values to be
 ///   yielded from within the innermost loop.
 /// - The original loops are not erased,  but are left in a "no-op" state where
 ///   the body of the loop just yields the basic block arguments that correspond
 ///   to the initialization values of a loop. The original loops are dead after
 ///   this method.
-/// - All uses of the `newIterOperands` within the generated new loop
-///   are replaced with the corresponding `BlockArgument` in the loop body.
 SmallVector<scf::ForOp>
 replaceLoopNestWithNewYields(OpBuilder &builder, ArrayRef<scf::ForOp> loopNest,
                              ValueRange newIterOperands,
                              const NewYieldValueFn &newYieldValueFn);
 
 /// Outline a region with a single block into a new FuncOp.
 /// Assumes the FuncOp result types is the type of the yielded operands of the
 /// single block. This constraint makes it easy to determine the result.

mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp

 #include "mlir/Dialect/Arithmetic/IR/Arithmetic.h"
 #include "mlir/Dialect/Arithmetic/Utils/Utils.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/SCF/Utils/Utils.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Interfaces/TilingInterface.h"
+#include "llvm/ADT/TypeSwitch.h"
 #include "llvm/Support/Debug.h"
 
 #define DEBUG_TYPE "tile-using-interface"
 
 using namespace mlir;
 
 scf::SCFTilingOptions &
 scf::SCFTilingOptions::setTileSizes(ArrayRef<int64_t> ts) {
auto loop = builder.create<scf::ForOp>(
         });
     offsets[loopRange.index()] = loop.getInductionVar();
     loops.push_back(loop);
     builder.setInsertionPoint(loop.getBody()->getTerminator());
   }
   return loops;
 }
 
-scf::TileUsingSCFForOp::TileUsingSCFForOp(MLIRContext *context,
-                                          scf::SCFTilingOptions options,
-                                          PatternBenefit benefit)
-    : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
-      options(std::move(options)) {}
+/// If producers of an destination operand are fused with its consumer
+/// then while yielding the value of the consumer, the destination of
+/// the producer is preferable to use as the initial value for the
+/// iter_arg. This removes an unnecessary use of the producer as the
+/// init value of the yielded result. For example, for this sequence
+///
+/// ```mlir
+///   %0 = linalg.init_tensor ...
+///   %1 = linalg.fill ins(...) outs(%0 : ...)
+///   %2 = linalg.matmul ins(...) outs(%1 : ...)
+/// ```
+///
+/// when tiled and fused
+///
+/// ```mlir
+/// %0 = linalg.init_tensor
+/// %1 = scf.for ... iter_args(%arg0 = %0)
+///  %2 = scf.for ... iter_args(%arg1 = %arg0)
+///     %3 = tensor.extract_slice %arg1 ...
+///     %4 = linalg.fill ins(...) outs(%3 : ...)
+///     %5 = linalg.matmul ins(...) outs(%4 : ...)
+/// ```
+///
+/// is preferable to
+///
+/// ```mlir
+/// %0 = linalg.init_tensor
+/// %1 = linalg.fill ... outs(%0)
+/// %2 = scf.for ... iter_args(%arg0 = %1)
+///  %3 = scf.for ... iter_args(%arg1 = %arg0)
+///     %3 = tensor.extract_slice %arg1 ...
+///     %4 = linalg.fill ins(...) outs(%3 : ...)
+///     %5 = linalg.matmul ins(...) outs(%4 : ...)
+/// ```
+///
+/// This method tracks through the tiled and fused ops to get to the
+/// actual value to use for the destination.
+// TODO: This will be much easier to do when the DestinationPassingStyle
+// interface is moved to `mlir/Interfaces`.
+static Value getTileDestination(OpBuilder &builder, OpResult result) {
+  Operation *definingOp = result.getOwner();
+  while (definingOp) {
+    Value destinationTile =
+        TypeSwitch<Operation *, Value>(definingOp)
+            .Case<TilingInterface>([&](auto interfaceOp) {
+              return interfaceOp.getDestinationOperands(
+                  builder)[result.getResultNumber()];
+            })
+            .Default([&](Operation *) -> Value { return nullptr; });
+    definingOp = destinationTile ? destinationTile.getDefiningOp() : nullptr;
+    result = definingOp ? destinationTile.cast<OpResult>() : result;
+  }
+  return result;
+}
+
+/// Given a list of `untiledOps` that are the original untiled operations, and
+/// their tiled counterparts (`tiledOps`) yield the values of the results
+/// through the generated `tilingLoops`. The result of the outermost loop forms
+/// a replacement for the `untiledOps`. This method performs the replacement as
+/// well.
+static LogicalResult yieldTiledValues(RewriterBase &rewriter,
+                                      ArrayRef<TilingInterface> untiledOps,
+                                      ArrayRef<Operation *> tiledOps,
+                                      ArrayRef<OpFoldResult> tileOffsets,
+                                      ArrayRef<OpFoldResult> tileSizes,
+                                      MutableArrayRef<scf::ForOp> tilingLoops) {
+  SmallVector<tensor::ExtractSliceOp> destSliceOps;
+  SmallVector<Value> initValues;
+  for (auto tiledOp : llvm::enumerate(tiledOps)) {
+    for (auto result : llvm::enumerate(tiledOp.value()->getResults())) {
+      Value destinationTile = getTileDestination(rewriter, result.value());
+      tensor::ExtractSliceOp sliceOp =
+          destinationTile
+              ? destinationTile.getDefiningOp<tensor::ExtractSliceOp>()
+              : nullptr;
+      destSliceOps.push_back(sliceOp);
+      if (sliceOp) {
+        initValues.push_back(sliceOp.getSource());
+      } else {
+        TilingInterface untiledOp = untiledOps[tiledOp.index()];
+        initValues.push_back(
+            untiledOp.getDestinationOperands(rewriter)[result.index()]);
+      }
+    }
+  }
+  // 5b. `scf.for` with tensor semantics requires the loop nest to yield the
+  // replacement values using destructive updates. Use the `TilingInterface`
+  // to get the position of the result tiles and use that to generate the
+  // destructive update pattern, i.e.,
+  //
+  // ```mlir
+  // scf.for %iv0 = ... {
+  //   %0 = tiled_op
+  // }
+  // ```
+  //
+  // is transformed to
+  //
+  // ```mlir
+  // %result = scf.for %iv0 = ... iter_args(%arg = %init) -> .. {
+  //   %0 = tiled_op
+  //   %1 = tensor.insert_slice %0 into %arg[..] [..] [..]
+  //   scf.yield %1
+  // }
+  // ```
+  NewYieldValueFn yieldValueFn =
+      [&](OpBuilder &b, Location loc,
+          ArrayRef<BlockArgument> newBBArgs) -> SmallVector<Value> {
+    SmallVector<Value> yieldedValues;
+    Attribute one = b.getIndexAttr(1);
+    Attribute zero = b.getIndexAttr(0);
+    unsigned bbArgNum = 0;
+    for (auto it : llvm::enumerate(untiledOps)) {
+      TilingInterface untiledOp = it.value();
+      SmallVector<OpFoldResult> opTileOffsets(tileOffsets),
+          opTileSizes(tileSizes);
+      unsigned opRank = untiledOp.getLoopIteratorTypes().size();
+      opTileOffsets.resize(opRank, zero);
+      opTileSizes.resize(opRank, zero);
+      for (auto source : llvm::enumerate(tiledOps[it.index()]->getResults())) {
+        SmallVector<OpFoldResult> resultTileOffsets, resultTileSizes;
+        if (failed(untiledOp.getResultTilePosition(
+                b, source.index(), opTileOffsets, opTileSizes,
+                resultTileOffsets, resultTileSizes))) {
+          return {};
+        }
 
-scf::TileUsingSCFForOp::TileUsingSCFForOp(StringRef opName,
-                                          MLIRContext *context,
-                                          scf::SCFTilingOptions options,
-                                          PatternBenefit benefit)
-    : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
-      options(std::move(options)) {}
+        SmallVector<OpFoldResult> resultTileStrides(resultTileOffsets.size(),
+                                                    one);
+        tensor::ExtractSliceOp destSliceOp = destSliceOps[bbArgNum];
+        if (destSliceOp && destSliceOp->getBlock() == b.getInsertionBlock())
+          destSliceOp.setOperand(0, newBBArgs[bbArgNum]);
+        Value yieldedValue = b.create<tensor::InsertSliceOp>(
+            loc, source.value(), newBBArgs[bbArgNum++], resultTileOffsets,
+            resultTileSizes, resultTileStrides);
+        yieldedValues.push_back(yieldedValue);
+      }
+    }
+    return yieldedValues;
+  };
 
-FailureOr<scf::SCFTilingResult>
-scf::TileUsingSCFForOp::returningMatchAndRewrite(
-    TilingInterface op, PatternRewriter &rewriter) const {
+  // Modify the loop nest to yield the result values.
+  SmallVector<scf::ForOp> newLoops = replaceLoopNestWithNewYields(
+      rewriter, tilingLoops, initValues, yieldValueFn);
+  for (const auto &loop : llvm::enumerate(tilingLoops)) {
+    rewriter.eraseOp(loop.value());
+    tilingLoops[loop.index()] = newLoops[loop.index()];
+  }
+  scf::ForOp outerMost = tilingLoops.front();
+  llvm::SmallDenseSet<Operation *> untiledOpsSet;
+  untiledOpsSet.insert(untiledOps.begin(), untiledOps.end());
+  unsigned resultNum = 0;
+  for (auto untiledOp : untiledOps) {
+    if (untiledOp->getNumResults() + resultNum > outerMost.getNumResults()) {
+      return rewriter.notifyMatchFailure(
+          untiledOp, "failed to yield results of operation");
+    }
+    // Replace all uses of the untiled op with values returned from the
+    // loop except for in `tensor.dim` operations or other ops that are fused
+    // here. Those need to be resolved separately.
+    // TODO: Find a better way to handle replacements.
+    rewriter.replaceOpWithIf(
+        untiledOp,
+        outerMost.getResults().drop_front(resultNum).take_front(
+            untiledOp->getNumResults()),
+        [&](OpOperand &use) {
+          Operation *user = use.getOwner();
+          return !untiledOpsSet.count(user) &&
+                 !isa<tensor::DimOp>(use.getOwner());
+        });
+    resultNum += untiledOp->getNumResults();
+  }
+  return success();
+}
+
+/// Implementation of tiling transformation of `op` that implements the
+/// `TilingInterface` using `scf.for` to iterate over the tiles.
+static FailureOr<scf::SCFTilingResult>
+tileConsumer(RewriterBase &rewriter, TilingInterface op,
+             scf::SCFTilingOptions options) {
   OpBuilder::InsertionGuard guard(rewriter);
   rewriter.setInsertionPointAfter(op);
 
   if (!options.tileSizeComputationFunction) {
     return rewriter.notifyMatchFailure(
         op, "missing tile size computation function");
   }
 
tileConsumer(RewriterBase &rewriter, TilingInterface op,
   SmallVector<Value> tileSizeVector =
       options.tileSizeComputationFunction(rewriter, op);
   if (tileSizeVector.size() < iterationDomain.size()) {
     auto zero = rewriter.create<arith::ConstantIndexOp>(op.getLoc(), 0);
     tileSizeVector.append(numLoops - tileSizeVector.size(), zero);
   }
 
   scf::SCFTilingResult tilingResult;
+  tilingResult.tiledLoops.resize(numLoops);
+  for (auto tileSize : llvm::enumerate(tileSizeVector))
+    if (!isConstantIntValue(tileSize.value(), 0))
+      tilingResult.tiledLoops.set(tileSize.index());
+
   SmallVector<OpFoldResult> offsets, sizes;
   {
     // If there is an interchange specified, permute the iteration domain and
     // the tile sizes.
     SmallVector<unsigned> interchangeVector;
     if (!options.interchangeVector.empty()) {
       interchangeVector = fillInterchangeVector(options.interchangeVector,
                                                 iterationDomain.size());
if (!tilingResult.loops.empty())
           tilingResult.loops.back().getBody()->getTerminator());
     SmallVector<Operation *> tiledImplementation =
         op.getTiledImplementation(rewriter, offsets, sizes);
     if (tiledImplementation.size() != 1) {
       return rewriter.notifyMatchFailure(
           op, "expected tiled implementation to return a single op");
     }
     tilingResult.tiledOp = tiledImplementation[0];
+    std::swap(tilingResult.tileOffsets, offsets);
+    std::swap(tilingResult.tileSizes, sizes);
 
     LLVM_DEBUG({
       if (!tilingResult.loops.empty()) {
         llvm::errs() << "After tiled implementation :\n";
         tilingResult.loops.front().dump();
         llvm::errs() << "\n";
       }
     });
   }
 
+  return tilingResult;
+}
+
+scf::TileUsingSCFForOp::TileUsingSCFForOp(MLIRContext *context,
+                                          scf::SCFTilingOptions options,
+                                          PatternBenefit benefit)
+    : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
+      options(std::move(options)) {}
+
+scf::TileUsingSCFForOp::TileUsingSCFForOp(StringRef opName,
+                                          MLIRContext *context,
+                                          scf::SCFTilingOptions options,
+                                          PatternBenefit benefit)
+    : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
+      options(std::move(options)) {}
+
+FailureOr<scf::SCFTilingResult>
+scf::TileUsingSCFForOp::returningMatchAndRewrite(
+    TilingInterface op, PatternRewriter &rewriter) const {
+
+  FailureOr<scf::SCFTilingResult> tilingResult =
+      tileConsumer(rewriter, op, options);
+  if (failed(tilingResult))
+    return rewriter.notifyMatchFailure(op, "failed to tile operation");
+
+  // If there are no results (i.e. buffer semantics), there is nothing to do.
+  // Erase op and return.
   if (op->getNumResults() == 0) {
     rewriter.eraseOp(op);
     return tilingResult;
   }
 
-  // 5. If the original operations has results, modify the loop nest to yield
-  // the replacement values.
-  SmallVector<Value> replacements;
-  if (tilingResult.loops.empty()) {
-    // 5a. If there were no loops, the tiled implementation results are the
-    // replacements.
-    rewriter.replaceOp(op, tilingResult.tiledOp->getResults());
+  // If there are no loops, there is nothing more to do.
+  if (tilingResult->loops.empty()) {
+    // Replace the original op with the tiled op.
+    rewriter.replaceOp(op, tilingResult->tiledOp->getResults());
     return tilingResult;
   }
 
-  // 5b. `scf.for` with tensor semantics requires the loop nest to yield the
-  // replacement values using destructive updates. Use the `TilingInterface`
-  // to get the position of the result tiles and use that to generate the
-  // destructive update pattern, i.e.,
-  //
-  // ```mlir
-  // scf.for %iv0 = ... {
-  //   %0 = tiled_op
-  // }
-  // ```
-  //
-  // is transformed to
-  //
-  // ```mlir
-  // %result = scf.for %iv0 = ... iter_args(%arg = %init) -> .. {
-  //   %0 = tiled_op
-  //   %1 = tensor.insert_slice %0 into %arg[..] [..] [..]
-  //   scf.yield %1
-  // }
-  // ```
-  NewYieldValueFn yieldValueFn =
-      [&](OpBuilder &b, Location loc,
-          ArrayRef<BlockArgument> newBBArgs) -> SmallVector<Value> {
-    SmallVector<Value> yieldedValues;
-    Attribute one = b.getIndexAttr(1);
-    for (auto resultNum : llvm::seq<unsigned>(0, op->getNumResults())) {
-      SmallVector<OpFoldResult> resultTileOffsets, resultTileSizes;
-      if (failed(op.getResultTilePosition(b, resultNum, offsets, sizes,
-                                          resultTileOffsets,
-                                          resultTileSizes))) {
-        op.emitOpError("unable to get position of result ")
-            << resultNum << " of the tiled implementation";
-        return {};
-      }
-      SmallVector<OpFoldResult> resultTileStrides(resultTileOffsets.size(),
-                                                  one);
-      Value yieldedValue = b.create<tensor::InsertSliceOp>(
-          op->getLoc(), tilingResult.tiledOp->getResult(resultNum),
-          newBBArgs[resultNum], resultTileOffsets, resultTileSizes,
-          resultTileStrides);
-      yieldedValues.push_back(yieldedValue);
-    }
-    return yieldedValues;
-  };
-  SmallVector<scf::ForOp> newLoops = replaceLoopNestWithNewYields(
-      rewriter, tilingResult.loops, op.getDestinationOperands(rewriter),
-      yieldValueFn);
-  for (const auto &loop : llvm::enumerate(tilingResult.loops)) {
-    rewriter.eraseOp(loop.value());
-    tilingResult.loops[loop.index()] = newLoops[loop.index()];
-  }
-  rewriter.replaceOp(op, tilingResult.loops.front().getResults());
+  // From the tiled ops reconstruct the value to replace the result of the
+  // untiled op using destructive updates.
+  if (failed(yieldTiledValues(rewriter, op, tilingResult->tiledOp,
+                              tilingResult->tileOffsets,
+                              tilingResult->tileSizes, tilingResult->loops)))
+    return failure();
   return tilingResult;
 }
 
 //===----------------------------------------------------------------------===//
-// TileConsumerAndFuseProducersUsingSCFForOp pattern implementation.
+// TileConsumerAndFuseProducersGreedilyUsingSCFForOp pattern implementation.
 //===----------------------------------------------------------------------===//
 
-scf::TileConsumerAndFuseProducersUsingSCFForOp::
-    TileConsumerAndFuseProducersUsingSCFForOp(MLIRContext *context,
-                                              scf::SCFTilingOptions options,
-                                              PatternBenefit benefit)
+scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp::
+    TileConsumerAndFuseProducersGreedilyUsingSCFForOp(
+        MLIRContext *context, scf::SCFTileAndFuseOptions options,
     : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
-      tilingPattern(context, std::move(options)) {}
+      tileAndFuseOptions(std::move(options)) {}
 
-scf::TileConsumerAndFuseProducersUsingSCFForOp::
-    TileConsumerAndFuseProducersUsingSCFForOp(StringRef opName,
-                                              MLIRContext *context,
-                                              scf::SCFTilingOptions options,
-                                              PatternBenefit benefit)
+scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp::
+    TileConsumerAndFuseProducersGreedilyUsingSCFForOp(
+        StringRef opName, MLIRContext *context,
+        scf::SCFTileAndFuseOptions options, PatternBenefit benefit)
     : OpInterfaceRewritePattern<TilingInterface>(context, benefit),
-      tilingPattern(context, std::move(options)) {}
+      tileAndFuseOptions(std::move(options)) {}
+
+/// Collect all transitive producers of `op` and return then in sorted order
+/// (i.e def before use).
+static SmallVector<Operation *> collectTransitiveProducers(TilingInterface op) {
+  SetVector<Operation *> visited;
+  SmallVector<Operation *> worklist;
+  SmallVector<Operation *> sortedOps;
+  worklist.push_back(op);
+  while (!worklist.empty()) {
+    TilingInterface currOp = worklist.back();
+    bool addedProducer = false;
+    for (OpOperand &operand : currOp->getOpOperands()) {
+      auto producerOp = operand.get().getDefiningOp<TilingInterface>();
+      if (!producerOp || visited.count(producerOp))
+        continue;
+      addedProducer = true;
+      worklist.push_back(producerOp);
+      visited.insert(producerOp);
+    }
+    if (!addedProducer) {
+      if (op != currOp)
+        sortedOps.push_back(currOp);
+      worklist.pop_back();
+    }
+  }
+  return sortedOps;
+}
+
+/// For a given untiled op `producer` find all instances where
+/// slices of this operation are used in `tiledOps`.
+struct CandidateSliceOp {
+  tensor::ExtractSliceOp sliceOp;
+  OpResult producerResult;
+};
+static SmallVector<CandidateSliceOp>
+collectAllSlicesToProducer(ArrayRef<Operation *> tiledOps,
+                           TilingInterface producer) {
+  SmallVector<CandidateSliceOp> slicesOfProducer;
+  for (auto tiledOp : tiledOps) {
+    for (OpOperand &operand : tiledOp->getOpOperands()) {
+      auto sliceOp = operand.get().getDefiningOp<tensor::ExtractSliceOp>();
+      if (!sliceOp)
+        continue;
 
-/// Return the `Value` that is defined by an operation that implements
-/// the `TilingInterface`. Looks through `iter_args` of scf.for nest
-/// if required.
-static Optional<OpResult> getFusableProducer(Value v) {
-  while (auto blockArg = v.dyn_cast<BlockArgument>()) {
+      Value source = sliceOp.getSource();
+      while (auto blockArg = source.dyn_cast<BlockArgument>()) {
         auto loopOp = dyn_cast<scf::ForOp>(blockArg.getOwner()->getParentOp());
         if (!loopOp)
-      return llvm::None;
-    v = loopOp.getOpOperandForRegionIterArg(blockArg).get();
+          break;
+        source = loopOp.getOpOperandForRegionIterArg(blockArg).get();
       }
-  if (!isa_and_nonnull<TilingInterface>(v.getDefiningOp()))
-    return llvm::None;
-  return v.cast<OpResult>();
-}
-
-// Replace iter args of the outer most loop with region args of the inner most
-// one.
-static void replaceIterArgs(scf::ForOp outerFor, scf::ForOp innerFor,
-                            PatternRewriter &rewriter) {
-  assert(outerFor.getNumIterOperands() == innerFor.getNumIterOperands() &&
-         "expect same number of iter args");
-  Block *block = &(*innerFor.getRegion().begin());
-  for (auto it :
-       llvm::zip(outerFor.getIterOperands(), innerFor.getRegionIterArgs())) {
-    Value source = std::get<0>(it);
-    Value target = std::get<1>(it);
-    source.replaceUsesWithIf(target, [&](OpOperand &use) {
-      return use.getOwner()->getBlock() == block;
-    });
+
+      if (source.getDefiningOp() == producer.getOperation()) {
+        slicesOfProducer.emplace_back(
+            CandidateSliceOp{sliceOp, source.cast<OpResult>()});
+      }
+    }
+  }
+  return slicesOfProducer;
+}
+
+/// Implementation of the the steps to fuse an untiled `producer` with
+/// all uses of it in `tiledOps`. If `isFusableWithRedundantComputation` is
+/// - `false` : each slice is replaced with a tiled version of the producer
+///             that produces the
+/// - `true`  : it is assumed that a single instance of tiling the producer
+///             can be used to replace all (slice) uses of the untiled
+///             producer in `tiledOps`.
+static FailureOr<SmallVector<Operation *>>
+fuseProducer(RewriterBase &rewriter, ArrayRef<Operation *> tiledOps,
+             TilingInterface producer,
+             bool isFusableWithoutRedundantComputation = false) {
+  SmallVector<Operation *> fusedProducers;
+  SmallVector<CandidateSliceOp> slicesOfProducer =
+      collectAllSlicesToProducer(tiledOps, producer);
+  if (slicesOfProducer.empty())
+    return fusedProducers;
+
+  if (!isFusableWithoutRedundantComputation) {
+    // Simpler usage, with lesser constraints. Just replace each slice
+    // with tiled implementation of the producer.
+    for (auto candidateSliceOp : slicesOfProducer) {
+      tensor::ExtractSliceOp sliceOp = candidateSliceOp.sliceOp;
+      OpBuilder::InsertionGuard g(rewriter);
+      rewriter.setInsertionPoint(sliceOp);
+      FailureOr<Value> fusedProducerValue =
+          tensor::replaceExtractSliceWithTiledProducer(
+              rewriter, sliceOp, candidateSliceOp.producerResult);
+      if (failed(fusedProducerValue)) {
+        // The fusion failed. Continue fusing other slices.
+        continue;
+      }
+      rewriter.replaceOp(sliceOp, fusedProducerValue.value());
+      fusedProducers.push_back(fusedProducerValue->getDefiningOp());
     }
+    return fusedProducers;
+  }
+
+  // Assume one instance of the tiled producer can replace all uses in
+  // `tiledOps`. Take the first slice op and use that to produce the tiled
+  // implementation.
+  auto currSliceOp = slicesOfProducer.front().sliceOp;
+  OpBuilder::InsertionGuard g(rewriter);
+  rewriter.setInsertionPoint(currSliceOp);
+  FailureOr<Value> fusedProducerVal =
+      tensor::replaceExtractSliceWithTiledProducer(
+          rewriter, currSliceOp, slicesOfProducer.front().producerResult);
+  if (failed(fusedProducerVal)) {
+    // The fusion failed. Continue fusing other ops.
+    return fusedProducers;
+  }
+  TilingInterface tiledProducer =
+      fusedProducerVal->getDefiningOp<TilingInterface>();
+  if (!tiledProducer ||
+      tiledProducer->getNumResults() != producer->getNumResults()) {
+    return rewriter.notifyMatchFailure(
+        producer,
+        "unhandled case where tiled implementation does not return a single "
+        "operation with as many results as the untiled operation");
+  }
+  fusedProducers.push_back(tiledProducer);
+
+  // 3c. Replace the slice uses with the corresponding producer use.
+  for (auto candidateSliceOp : slicesOfProducer) {
+    unsigned resultNumber = candidateSliceOp.producerResult.getResultNumber();
+    rewriter.replaceOp(candidateSliceOp.sliceOp,
+                       tiledProducer->getResult(resultNumber));
+  }
+  return fusedProducers;
 }
 
 FailureOr<scf::SCFTileAndFuseResult>
-scf::TileConsumerAndFuseProducersUsingSCFForOp::returningMatchAndRewrite(
-    TilingInterface op, PatternRewriter &rewriter) const {
+scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp::
+    returningMatchAndRewrite(TilingInterface op,
+                             PatternRewriter &rewriter) const {
   // This transformation is only valid for ops that return values (i.e. not
   // valid to use with operations that have memref operands).
   if (!op->getNumResults()) {
     return rewriter.notifyMatchFailure(
         op, "invalid pattern for op with no results");
   }
 
   // 1. First tile the consumer.
   SCFTileAndFuseResult tileAndFuseResult;
+  SmallVector<OpFoldResult> consumerTileOffsets, consumerTileSizes;
   {
     FailureOr<SCFTilingResult> tilingResult =
-        tilingPattern.returningMatchAndRewrite(op, rewriter);
-    if (failed(tilingResult)) {
-      return failure();
-    }
+        tileConsumer(rewriter, op, tileAndFuseOptions.tilingOptions);
+    if (failed(tilingResult))
+      return rewriter.notifyMatchFailure(op, "failed to tile consumer");
     tileAndFuseResult.tiledAndFusedOps.push_back(tilingResult->tiledOp);
     tileAndFuseResult.loops = std::move(tilingResult->loops);
+    std::swap(tilingResult->tileOffsets, consumerTileOffsets);
+    std::swap(tilingResult->tileSizes, consumerTileSizes);
   }
 
-  // 2. Typically, the operands of the tiled operation are slices of the
-  //    operands of the untiled operation. These are expressed in IR using
-  //    `tensor.extract_slice` operations with source being the operands of the
-  //    untiled operation. Create a worklist of these `tensor.extract_slice`
-  //    operations. If the producers of the source of the `tensor.extract_slice`
-  //    can be tiled such that the tiled value is generated in-place, that
-  //    effectively tiles + fuses the operations.
-  auto addCandidateSlices = [](Operation *fusedOp,
-                               std::deque<tensor::ExtractSliceOp> &candidates) {
-    for (Value operand : fusedOp->getOperands())
-      if (auto sliceOp = operand.getDefiningOp<tensor::ExtractSliceOp>())
-        candidates.push_back(sliceOp);
-  };
+  // If there are no loops generated, fusion is immaterial.
+  if (tileAndFuseResult.loops.empty())
+    return tileAndFuseResult;
 
-  std::deque<tensor::ExtractSliceOp> candidates;
-  addCandidateSlices(tileAndFuseResult.tiledAndFusedOps.back(), candidates);
-  OpBuilder::InsertionGuard g(rewriter);
-  while (!candidates.empty()) {
-    // 2a. Traverse the slices in BFS fashion.
-    tensor::ExtractSliceOp candidateSliceOp = candidates.front();
-    candidates.pop_front();
-
-    // 2b. Get the producer of the source (potentially walking through
-    // `iter_args` of nested `scf.for`)
-    Optional<OpResult> fusableProducer =
-        getFusableProducer(candidateSliceOp.getSource());
-    if (!fusableProducer)
+  // 2. Collect a list of producers of the original operation that are to be
+  // tiled and fused.
+  tileAndFuseResult.fusableProducers = collectTransitiveProducers(op);
+
+  // 3. Iterate through the producers in reverse and tile and fuse them.
+  for (Operation *producerOp :
+       llvm::reverse(tileAndFuseResult.fusableProducers)) {
+    auto producer = cast<TilingInterface>(producerOp);
+
+    FailureOr<SmallVector<Operation *>> fusedProducers = fuseProducer(
+        rewriter, tileAndFuseResult.tiledAndFusedOps, producer,
+        tileAndFuseOptions.producerCanBeFusedWithoutRedundantComputations);
+    if (failed(fusedProducers)) {
+      // Fusion failed. Continue with other producers.
       continue;
+    }
 
-    // 2c. Generate the tiled implementation of the producer of the source
-    rewriter.setInsertionPoint(candidateSliceOp);
-    FailureOr<Value> fusedProducerValue =
-        tensor::replaceExtractSliceWithTiledProducer(rewriter, candidateSliceOp,
-                                                     fusableProducer.value());
-    if (failed(fusedProducerValue))
-      continue;
-    rewriter.replaceOp(candidateSliceOp, fusedProducerValue.value());
+    if (tileAndFuseOptions.producerCanBeFusedWithoutRedundantComputations &&
+        fusedProducers->size() != 1) {
+      return rewriter.notifyMatchFailure(
+          producer, "expected single operation for the fused producer");
+    }
 
-    // 2d. The operands of the fused producer might themselved be slices of
-    //     values produced by operations that implement the `TilingInterface`.
-    //     Add these operations to the worklist.
-    Operation *fusedProducer = fusedProducerValue->getDefiningOp();
-    tileAndFuseResult.tiledAndFusedOps.push_back(fusedProducer);
-    addCandidateSlices(fusedProducer, candidates);
-
-    // 2e. If the operation being fused creates a value that is used as `outs`
-    //     in the tiled operation, the result of the unfused operation will be
-    //     used in the `iter_args` of the tiled loop generated. When the
-    //     operation is fused, this use in `iter_args` needs to be modified to
-    //     use the destination of the fused operation. For example, starting
-    //     with
-    //
-    //     ```mlir
-    //     %0 = linalg.init_tensor ...
-    //     %1 = linalg.fill ... outs(%0:...)...
-    //     %2 = linalg.matmul ... outs(%1:...)....
-    //     ```
-    //
-    //     First the `linalg.matmul` gets tiled
-    //
-    //     ```mlir
-    //     %0 = linalg.init_tensor
-    //     %1 = linalg.fill
-    //     %2 = scf.for .... iter_args(%arg0 = %1)...
-    //        ...
-    //        ... = linalg.matmul ...
-    //
-    //     ```
-    //
-    //     When the `linalg.fill` gets fused, the `iter_args` needs to be
-    //     modified
-    //
-    //     ```mlir
-    //     %0 = linalg.init_tensor
-    //     %1 = scf.for ... iter_args(%arg0 = %0)...
-    //        ...
-    //        %2 = linalg.fill ...
-    //        %3 = linalg.matmul ... outs(%2: ...)...
-    //     ```
-    TilingInterface unfusedProducerOp =
-        cast<TilingInterface>(fusableProducer->getOwner());
-    scf::ForOp outerMostTiledLoop = tileAndFuseResult.loops.front();
-    SmallVector<Value> unfusedProducerOpDestValues =
-        unfusedProducerOp.getDestinationOperands(rewriter);
-    for (OpOperand &uses : unfusedProducerOp->getUses()) {
-      if (uses.getOwner() == outerMostTiledLoop.getOperation()) {
-        unsigned resultNumber = uses.get().cast<OpResult>().getResultNumber();
-        unsigned operandNumber = uses.getOperandNumber();
-        outerMostTiledLoop->setOperand(
-            operandNumber, unfusedProducerOpDestValues[resultNumber]);
-      }
+    tileAndFuseResult.tiledAndFusedOps.append(fusedProducers.value());
+
+  // 4. Finally reconstruct the replacements for the untiled operations
+  // using destructive updates. If
+  // `producerCanBeFusedWithoutRedundantComputation` is
+  // - `true` : Yield the results of all the producers. It is assumed
+  //            to be valid.
+  // - `false` : Yield the results of just the tiled consumer.
+  SmallVector<TilingInterface> untiledOps;
+  ArrayRef<Operation *> tiledOps = {tileAndFuseResult.tiledAndFusedOps.front()};
+  untiledOps.push_back(op);
+  if (tileAndFuseOptions.producerCanBeFusedWithoutRedundantComputations) {
+    if (tileAndFuseResult.tiledAndFusedOps.size() !=
+        tileAndFuseResult.fusableProducers.size() + 1) {
+      return rewriter.notifyMatchFailure(
+          op, "expected as many tiled and fused ops as producer");
     }
+    untiledOps.append(tileAndFuseResult.fusableProducers.rbegin(),
+                      tileAndFuseResult.fusableProducers.rend());
+    tiledOps = llvm::makeArrayRef(tileAndFuseResult.tiledAndFusedOps);
+  }
+  if (failed(yieldTiledValues(rewriter, untiledOps, tiledOps,
+                              consumerTileOffsets, consumerTileSizes,
+                              tileAndFuseResult.loops))) {
+    return rewriter.notifyMatchFailure(op, "failed to yield values");
   }
-  replaceIterArgs(tileAndFuseResult.loops.front(),
-                  tileAndFuseResult.loops.back(), rewriter);
+
   return tileAndFuseResult;
 }
 
 //===----------------------------------------------------------------------===//
 // LowerToLoopsUsingSCFForOp
 //===----------------------------------------------------------------------===//
 
 FailureOr<SmallVector<scf::ForOp>>

mlir/lib/Dialect/SCF/Utils/Utils.cpp

mlir::replaceLoopWithNewYields(OpBuilder &builder, scf::ForOp loop,
 
   // Remap the BlockArguments from the original loop to the new loop
   // BlockArguments.
   ArrayRef<BlockArgument> bbArgs = loopBody->getArguments();
   for (auto it :
        llvm::zip(bbArgs, newLoopBody->getArguments().take_front(bbArgs.size())))
     std::get<0>(it).replaceAllUsesWith(std::get<1>(it));
 
-  // Replace all uses of `newIterOperands` with the corresponding basic block
-  // arguments.
-  for (auto it : llvm::zip(newIterOperands, newBBArgs)) {
-    std::get<0>(it).replaceUsesWithIf(std::get<1>(it), [&](OpOperand &use) {
-      Operation *user = use.getOwner();
-      return newLoop->isProperAncestor(user);
-    });
-  }
-
   // Replace all uses of the original loop with corresponding values from the
   // new loop.
   loop.replaceAllUsesWith(
       newLoop.getResults().take_front(loop.getNumResults()));
 
   // Add a fake yield to the original loop body that just returns the
   // BlockArguments corresponding to the iter_args. This makes it a no-op loop.
   // The loop is dead. The caller is expected to erase it.
NewYieldValueFn fn = [&](OpBuilder &innerBuilder, Location loc,
                              ArrayRef<BlockArgument> innerNewBBArgs) {
       SmallVector<Value> newYields(
           newLoopNest[loopDepth + 1]->getResults().take_back(
               newIterOperands.size()));
       return newYields;
     };
     newLoopNest[loopDepth] = replaceLoopWithNewYields(
         builder, loopNest[loopDepth], newIterOperands, fn);
+    newLoopNest[loopDepth + 1].getInitArgsMutable().assign(
+        newLoopNest[loopDepth].getRegionIterArgs());
   }
   return newLoopNest;
 }
 
 /// Outline a region with a single block into a new FuncOp.
 /// Assumes the FuncOp result types is the type of the yielded operands of the
 /// single block. This constraint makes it easy to determine the result.
 /// This method also clones the `arith::ConstantIndexOp` at the start of

mlir/test/Interfaces/TilingInterface/tile-and-fuse-using-interface.mlir

 //  CHECK-DAG:       %[[RHS_TILE:.+]] = tensor.extract_slice %[[ARG1]][0, %[[IV1]]]
 //  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV0]], %[[IV1]]]
 //      CHECK:       %[[FILL_TILE:.+]] = linalg.fill
 // CHECK-SAME:           outs(%[[INIT_TILE]] :
 //      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
 // CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
 // CHECK-SAME:           outs(%[[FILL_TILE]] :
 //      CHECK:       %[[INSERT:.+]] = tensor.insert_slice %[[GEMM_TILE]] into %[[ITERARG1]][%[[IV0]], %[[IV1]]]
-//      CHECK        scf.yield %[[INSERT]]
+//      CHECK:       scf.yield %[[INSERT]]
 
 // -----
 
 func.func @gemm_generic_fusion(%arg0 : tensor<?x?xf32>, %arg1 : tensor<?x?xf32>,
     %arg2 : tensor<?xf32>) -> tensor<?x?xf32> {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %cst = arith.constant 0.0 : f32

 // CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: tensor<?xf32>)
 //      CHECK:   %[[INIT:.+]] = linalg.init_tensor
 //      CHECK:   scf.for %[[IV0:[a-zA-Z0-9]+]] =
 // CHECK-SAME:       iter_args(%[[ITERARG0:.+]] = %[[INIT]])
 //      CHECK:     scf.for %[[IV1:[a-zA-Z0-9]+]] =
 // CHECK-SAME:         iter_args(%[[ITERARG1:.+]] = %[[ITERARG0]])
 //  CHECK-DAG:       %[[LHS_TILE:.+]] = tensor.extract_slice %[[ARG0]][%[[IV0]], 0]
 //  CHECK-DAG:       %[[RHS_TILE:.+]] = tensor.extract_slice %[[ARG1]][0, %[[IV1]]]
-//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV0]], %[[IV1]]]
+//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[INIT]][%[[IV0]], %[[IV1]]]
 //      CHECK:       %[[FILL_TILE:.+]] = linalg.fill
 // CHECK-SAME:           outs(%[[INIT_TILE]] :
 //      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
 // CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
 // CHECK-SAME:           outs(%[[FILL_TILE]] :
 //  CHECK-DAG:       %[[BIAS_TILE:.+]] = tensor.extract_slice %[[ARG2]][%[[IV1]]]
 //  CHECK-DAG:       %[[OUTS_TILE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV0]], %[[IV1]]]
 //      CHECK:       %[[GENERIC_TILE:.+]] = linalg.generic
 // CHECK-SAME:           ins(%[[GEMM_TILE]], %[[BIAS_TILE]] :
 // CHECK-SAME:           outs(%[[OUTS_TILE]] :
 //      CHECK:       %[[INSERT:.+]] = tensor.insert_slice %[[GENERIC_TILE]] into %[[ITERARG1]][%[[IV0]], %[[IV1]]]
-//      CHECK        scf.yield %[[INSERT]]
+//      CHECK:       scf.yield %[[INSERT]]
 
 // -----
 
 func.func @gemm_gemm_fusion(%lhs0 : tensor<?x?xf32>, %rhs0 : tensor<?x?xf32>, %rhs1 : tensor<?x?xf32>) -> tensor<?x?xf32> {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %cst = arith.constant 0.0 : f32
   %d0 = tensor.dim %lhs0, %c0 : tensor<?x?xf32>

 //  CHECK-DAG:     %[[RHS1_TILE:.+]] = tensor.extract_slice %[[RHS1]][0, 0]
 //  CHECK-DAG:     %[[INIT1_TILE:.+]] = tensor.extract_slice %[[ITERARG]][%[[IV]], 0]
 //      CHECK:     %[[FILL1_TILE:.+]] = linalg.fill
 // CHECK-SAME:         outs(%[[INIT1_TILE]] :
 //      CHECK:     %[[GEMM1_TILE:.+]] = linalg.matmul
 // CHECK-SAME:         ins(%[[GEMM0_TILE]], %[[RHS1_TILE]] :
 // CHECK-SAME:         outs(%[[FILL1_TILE]] :
 //      CHECK:     %[[INSERT:.+]] = tensor.insert_slice %[[GEMM1_TILE]] into %[[ITERARG]][%[[IV]], 0]
-//      CHECK      scf.yield %[[INSERT]]
+//      CHECK:     scf.yield %[[INSERT]]
 
 // -----
 
 func.func @gemm_transpose_fusion(%arg0 : tensor<?x?xf32>, %arg1 : tensor<?x?xf32>) -> tensor<?x?xf32> {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %cst = arith.constant 0.0 : f32
   %d0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>

 //      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
 // CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
 // CHECK-SAME:           outs(%[[FILL_TILE]] :
 //  CHECK-DAG:       %[[OUTS_TILE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV1]], %[[IV0]]]
 //      CHECK:       %[[GENERIC_TILE:.+]] = linalg.generic
 // CHECK-SAME:           ins(%[[GEMM_TILE]] :
 // CHECK-SAME:           outs(%[[OUTS_TILE]] :
 //      CHECK:       %[[INSERT:.+]] = tensor.insert_slice %[[GENERIC_TILE]] into %[[ITERARG1]][%[[IV1]], %[[IV0]]]
-//      CHECK        scf.yield %[[INSERT]]
+//      CHECK:       scf.yield %[[INSERT]]
 
 // -----
 
 func.func @interchange_matmul_fusion(%arg0 : tensor<?x?xf32>, %arg1 : tensor<?x?xf32>) -> tensor<?x?xf32> {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %d0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
   %d1 = tensor.dim %arg1, %c1 : tensor<?x?xf32>

 // CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]: tensor<?x?xf32>)
 //      CHECK:   %[[INIT:.+]] = linalg.init_tensor
 //      CHECK:   scf.for %[[IV0:[a-zA-Z0-9]+]] =
 // CHECK-SAME:       iter_args(%[[ITERARG0:.+]] = %[[INIT]])
 //      CHECK:     scf.for %[[IV1:[a-zA-Z0-9]+]] =
 // CHECK-SAME:         iter_args(%[[ITERARG1:.+]] = %[[ITERARG0]])
 //  CHECK-DAG:       %[[LHS_TILE:.+]] = tensor.extract_slice %[[ARG0]][%[[IV1]], 0]
 //  CHECK-DAG:       %[[RHS_TILE:.+]] = tensor.extract_slice %[[ARG1]][0, %[[IV0]]]
-//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV1]], %[[IV0]]]
+//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[INIT]][%[[IV1]], %[[IV0]]]
 //      CHECK:       %[[FILL_TILE:.+]] = linalg.fill
 // CHECK-SAME:           outs(%[[INIT_TILE]] :
 //      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
 // CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
 // CHECK-SAME:           outs(%[[FILL_TILE]] :
 //      CHECK:       %[[INIT_TILE_2:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV1]], %[[IV0]]]
 //      CHECK:       %[[GENERIC_TILE:.+]] = linalg.generic
 // CHECK-SAME:           ins(%[[GEMM_TILE]] :
 // CHECK-SAME:           outs(%[[INIT_TILE_2]] :
 //      CHECK:       %[[INSERT:.+]] = tensor.insert_slice %[[GENERIC_TILE]] into %[[ITERARG1]][%[[IV1]], %[[IV0]]]
-//      CHECK        scf.yield %[[INSERT]]
+//      CHECK:       scf.yield %[[INSERT]]
 
 // -----
 
 func.func @matmul_plus_matmul(%arg0: tensor<?x?xf32>, %arg1: tensor<?x?xf32>,
                          %arg2: tensor<?x?xf32>) -> tensor<?x?xf32>{
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %0 = tensor.dim %arg2, %c0 : tensor<?x?xf32>

mlir/test/Interfaces/TilingInterface/tile-and-fuse-yield-producers.mlir

+// RUN: mlir-opt -test-tiling-interface=tile-consumer-and-fuse-yield-producer-using-scf-for -cse -split-input-file %s | FileCheck %s
+
+func.func @gemm_generic_fusion(%arg0 : tensor<?x?xf32>, %arg1 : tensor<?x?xf32>,
+    %arg2 : tensor<?xf32>) -> (tensor<?x?xf32>, tensor<?x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %cst = arith.constant 0.0 : f32
+  %d0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %d1 = tensor.dim %arg1, %c1 : tensor<?x?xf32>
+  %init0 = linalg.init_tensor [%d0, %d1] : tensor<?x?xf32>
+  %fill = linalg.fill ins(%cst : f32) outs(%init0 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %gemm = linalg.matmul
+      ins(%arg0, %arg1 : tensor<?x?xf32>, tensor<?x?xf32>)
+      outs(%fill : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %init1 = linalg.init_tensor [%d0, %d1] : tensor<?x?xf32>
+  %generic = linalg.generic {
+      __internal_linalg_transform__ = "fusion",
+      indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d1)>, affine_map<(d0, d1) -> (d0, d1)>],
+      iterator_types = ["parallel", "parallel"]}
+      ins(%gemm, %arg2 : tensor<?x?xf32>, tensor<?xf32>) outs(%init1 : tensor<?x?xf32>) {
+    ^bb0(%b0 : f32, %b1 : f32, %b2 : f32):
+      %add = arith.addf %b0, %b1 : f32
+      linalg.yield %add : f32 
+  } -> tensor<?x?xf32>
+  return %gemm, %generic : tensor<?x?xf32>, tensor<?x?xf32> 
+}
+//      CHECK: func.func @gemm_generic_fusion(
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: tensor<?x?xf32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]: tensor<?x?xf32>,
+// CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: tensor<?xf32>)
+//      CHECK:   %[[INIT:.+]] = linalg.init_tensor
+//      CHECK:   %[[RESULT:[a-zA-Z0-9]+]]:3 = scf.for %[[IV0:[a-zA-Z0-9]+]] =
+// CHECK-SAME:       iter_args(%[[ITERARG0:.+]] = %[[INIT]], %[[ITERARG1:.+]] = %[[INIT]], %[[ITERARG2:.+]] = %[[INIT]])
+//      CHECK:     %[[INNER_RESULT:[a-zA-Z0-9]+]]:3 = scf.for %[[IV1:[a-zA-Z0-9]+]] =
+// CHECK-SAME:         iter_args(%[[INNER_ITERARG0:[a-zA-Z0-9]+]] = %[[ITERARG0]], %[[INNER_ITERARG1:[a-zA-Z0-9]+]] = %[[ITERARG1]], %[[INNER_ITERARG2:[a-zA-Z0-9]+]] = %[[ITERARG2]])
+//  CHECK-DAG:       %[[LHS_TILE:.+]] = tensor.extract_slice %[[ARG0]][%[[IV0]], 0]
+//  CHECK-DAG:       %[[RHS_TILE:.+]] = tensor.extract_slice %[[ARG1]][0, %[[IV1]]]
+//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[INNER_ITERARG2]][%[[IV0]], %[[IV1]]]
+//      CHECK:       %[[FILL_TILE:.+]] = linalg.fill
+// CHECK-SAME:           outs(%[[INIT_TILE]] :
+//      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
+// CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
+// CHECK-SAME:           outs(%[[FILL_TILE]] :
+//  CHECK-DAG:       %[[BIAS_TILE:.+]] = tensor.extract_slice %[[ARG2]][%[[IV1]]]
+//  CHECK-DAG:       %[[OUTS_TILE:.+]] = tensor.extract_slice %[[INNER_ITERARG0]][%[[IV0]], %[[IV1]]]
+//      CHECK:       %[[GENERIC_TILE:.+]] = linalg.generic
+// CHECK-SAME:           ins(%[[GEMM_TILE]], %[[BIAS_TILE]] :
+// CHECK-SAME:           outs(%[[OUTS_TILE]] :
+//  CHECK-DAG:       %[[INSERT1:.+]] = tensor.insert_slice %[[GENERIC_TILE]] into %[[INNER_ITERARG0]][%[[IV0]], %[[IV1]]]
+//  CHECK-DAG:       %[[INSERT2:.+]] = tensor.insert_slice %[[GEMM_TILE]] into %[[INNER_ITERARG1]][%[[IV0]], %[[IV1]]]
+//  CHECK-DAG:       %[[INSERT3:.+]] = tensor.insert_slice %[[FILL_TILE]] into %[[INNER_ITERARG2]][%[[IV0]], %[[IV1]]]
+//      CHECK:       scf.yield %[[INSERT1]], %[[INSERT2]], %[[INSERT3]]
+//      CHECK:     scf.yield %[[INNER_RESULT]]#0, %[[INNER_RESULT]]#1, %[[INNER_RESULT]]#2
+//      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#0
+
+// -----
+
+func.func @gemm_gemm_fusion(%lhs0 : tensor<?x?xf32>, %rhs0 : tensor<?x?xf32>, %rhs1 : tensor<?x?xf32>)
+    -> (tensor<?x?xf32>, tensor<?x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %cst = arith.constant 0.0 : f32
+  %d0 = tensor.dim %lhs0, %c0 : tensor<?x?xf32>
+  %d1 = tensor.dim %rhs0, %c1 : tensor<?x?xf32>
+  %init0 = linalg.init_tensor [%d0, %d1] : tensor<?x?xf32>
+  %fill0 = linalg.fill ins(%cst : f32) outs(%init0 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %gemm0 = linalg.matmul
+      ins(%lhs0, %rhs0 : tensor<?x?xf32>, tensor<?x?xf32>) outs(%fill0 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %d2 = tensor.dim %rhs1, %c1 : tensor<?x?xf32>
+  %init1 = linalg.init_tensor [%d0, %d2] : tensor<?x?xf32>
+  %fill1 = linalg.fill ins(%cst : f32) outs(%init1 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %gemm1 = linalg.matmul  {__internal_linalg_transform__ = "gemm_fusion"}
+      ins(%gemm0, %rhs1 : tensor<?x?xf32>, tensor<?x?xf32>) outs(%fill1 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %gemm0, %gemm1 : tensor<?x?xf32>, tensor<?x?xf32>
+}
+//      CHECK: func.func @gemm_gemm_fusion(
+// CHECK-SAME:     %[[LHS0:[a-zA-Z0-9]+]]: tensor<?x?xf32>
+// CHECK-SAME:     %[[RHS0:[a-zA-Z0-9]+]]: tensor<?x?xf32>,
+// CHECK-SAME:     %[[RHS1:[a-zA-Z0-9]+]]: tensor<?x?xf32>)
+//  CHECK-DAG:   %[[C0:.+]] = arith.constant 0 : index
+//  CHECK-DAG:   %[[C1:.+]] = arith.constant 1 : index
+//  CHECK-DAG:   %[[D0:.+]] = tensor.dim %[[LHS0]], %[[C0]]
+//  CHECK-DAG:   %[[D1:.+]] = tensor.dim %[[RHS0]], %[[C1]]
+//  CHECK-DAG:   %[[INIT0:.+]] = linalg.init_tensor [%[[D0]], %[[D1]]]
+//  CHECK-DAG:   %[[D2:.+]] = tensor.dim %[[RHS1]], %[[C1]]
+//      CHECK:   %[[INIT1:.+]] = linalg.init_tensor [%[[D0]], %[[D2]]]
+//      CHECK:   %[[RESULT:[a-zA-Z0-9]+]]:4 = scf.for %[[IV:[a-zA-Z0-9]+]] =
+// CHECK-SAME:       iter_args(%[[ITERARG0:[a-zA-Z0-9]+]] = %[[INIT1]], %[[ITERARG1:[a-zA-Z0-9]+]] = %[[INIT0]], %[[ITERARG2:[a-zA-Z0-9]+]] = %[[INIT0]], %[[ITERARG3:[a-zA-Z0-9]+]] = %[[INIT1]])
+//  CHECK-DAG:     %[[LHS0_TILE:.+]] = tensor.extract_slice %[[LHS0]][%[[IV]], 0]
+//  CHECK-DAG:     %[[RHS0_TILE:.+]] = tensor.extract_slice %[[RHS0]][0, 0]
+//  CHECK-DAG:     %[[INIT0_TILE:.+]] = tensor.extract_slice %[[ITERARG2]][%[[IV]], 0]
+//      CHECK:     %[[FILL0_TILE:.+]] = linalg.fill
+// CHECK-SAME:         outs(%[[INIT0_TILE]] :
+//      CHECK:     %[[GEMM0_TILE:.+]] = linalg.matmul
+// CHECK-SAME:         ins(%[[LHS0_TILE]], %[[RHS0_TILE]] :
+// CHECK-SAME:         outs(%[[FILL0_TILE]] :
+//  CHECK-DAG:     %[[RHS1_TILE:.+]] = tensor.extract_slice %[[RHS1]][0, 0]
+//  CHECK-DAG:     %[[INIT1_TILE:.+]] = tensor.extract_slice %[[ITERARG3]][%[[IV]], 0]
+//      CHECK:     %[[FILL1_TILE:.+]] = linalg.fill
+// CHECK-SAME:         outs(%[[INIT1_TILE]] :
+//      CHECK:     %[[GEMM1_TILE:.+]] = linalg.matmul
+// CHECK-SAME:         ins(%[[GEMM0_TILE]], %[[RHS1_TILE]] :
+// CHECK-SAME:         outs(%[[FILL1_TILE]] :
+//  CHECK-DAG:     %[[INSERT0:.+]] = tensor.insert_slice %[[GEMM1_TILE]] into %[[ITERARG0]][%[[IV]], 0]
+//  CHECK-DAG:     %[[INSERT1:.+]] = tensor.insert_slice %[[GEMM0_TILE]] into %[[ITERARG1]][%[[IV]], 0]
+//  CHECK-DAG:     %[[INSERT2:.+]] = tensor.insert_slice %[[FILL0_TILE]] into %[[ITERARG2]][%[[IV]], 0]
+//  CHECK-DAG:     %[[INSERT3:.+]] = tensor.insert_slice %[[FILL1_TILE]] into %[[ITERARG3]][%[[IV]], 0]
+//      CHECK:     scf.yield %[[INSERT0]], %[[INSERT1]], %[[INSERT2]], %[[INSERT3]]
+//      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#0
+
+// -----
+
+func.func @interchange_matmul_fusion(%arg0 : tensor<?x?xf32>, %arg1 : tensor<?x?xf32>)
+    -> (tensor<?x?xf32>, tensor<?x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %d0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %d1 = tensor.dim %arg1, %c1 : tensor<?x?xf32>
+  %cst = arith.constant 0.0 : f32
+  %0 = linalg.init_tensor [%d0, %d1] : tensor<?x?xf32>
+  %1 = linalg.fill ins(%cst : f32) outs(%0 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %2 = linalg.matmul
+      ins(%arg0, %arg1 : tensor<?x?xf32>, tensor<?x?xf32>)
+      outs(%1 : tensor<?x?xf32>) -> tensor<?x?xf32>
+  %3 = linalg.init_tensor [%d0, %d1] : tensor<?x?xf32>
+  %4 = linalg.generic {
+      __internal_linalg_transform__ = "gemm_interchange_fusion",
+      indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0, d1)>],
+      iterator_types = ["parallel", "parallel"]}
+      ins(%2 : tensor<?x?xf32>) outs(%3 : tensor<?x?xf32>) {
+      ^bb0(%b0 : f32, %b1 : f32):
+        %4 = arith.addf %b0, %b0 : f32
+        linalg.yield %4 : f32
+      } -> tensor<?x?xf32>
+  return %2, %4 : tensor<?x?xf32>, tensor<?x?xf32>
+}
+//      CHECK: func.func @interchange_matmul_fusion(
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: tensor<?x?xf32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]: tensor<?x?xf32>)
+//      CHECK:   %[[INIT0:.+]] = linalg.init_tensor
+//      CHECK:   %[[RESULT:[a-zA-Z0-9]+]]:3 = scf.for %[[IV0:[a-zA-Z0-9]+]] =
+// CHECK-SAME:       iter_args(%[[ITERARG0:[a-zA-Z0-9]+]] = %[[INIT]], %[[ITERARG1:[a-zA-Z0-9]+]] = %[[INIT]], %[[ITERARG2:[a-zA-Z0-9]+]] = %[[INIT]])
+//      CHECK:     %[[INNER_RESULT:[a-zA-Z0-9]+]]:3 = scf.for %[[IV1:[a-zA-Z0-9]+]] =
+// CHECK-SAME:         iter_args(%[[INNER_ITERARG0:[a-zA-Z0-9]+]] = %[[ITERARG0]], %[[INNER_ITERARG1:[a-zA-Z0-9]+]] = %[[ITERARG1]], %[[INNER_ITERARG2:[a-zA-Z0-9]+]] = %[[ITERARG2]])
+//  CHECK-DAG:       %[[LHS_TILE:.+]] = tensor.extract_slice %[[ARG0]][%[[IV1]], 0]
+//  CHECK-DAG:       %[[RHS_TILE:.+]] = tensor.extract_slice %[[ARG1]][0, %[[IV0]]]
+//  CHECK-DAG:       %[[INIT_TILE:.+]] = tensor.extract_slice %[[INNER_ITERARG2]][%[[IV1]], %[[IV0]]]
+//      CHECK:       %[[FILL_TILE:.+]] = linalg.fill
+// CHECK-SAME:           outs(%[[INIT_TILE]] :
+//      CHECK:       %[[GEMM_TILE:.+]] = linalg.matmul
+// CHECK-SAME:           ins(%[[LHS_TILE]], %[[RHS_TILE]] :
+// CHECK-SAME:           outs(%[[FILL_TILE]] :
+//      CHECK:       %[[INIT_TILE_2:.+]] = tensor.extract_slice %[[INNER_ITERARG0]][%[[IV1]], %[[IV0]]]
+//      CHECK:       %[[GENERIC_TILE:.+]] = linalg.generic
+// CHECK-SAME:           ins(%[[GEMM_TILE]] :
+// CHECK-SAME:           outs(%[[INIT_TILE_2]] :
+//  CHECK-DAG:       %[[INSERT0:.+]] = tensor.insert_slice %[[GENERIC_TILE]] into %[[INNER_ITERARG0]][%[[IV1]], %[[IV0]]]
+//  CHECK-DAG:       %[[INSERT1:.+]] = tensor.insert_slice %[[GEMM_TILE]] into %[[INNER_ITERARG1]][%[[IV1]], %[[IV0]]]
+//  CHECK-DAG:       %[[INSERT2:.+]] = tensor.insert_slice %[[FILL_TILE]] into %[[INNER_ITERARG2]][%[[IV1]], %[[IV0]]]
+//      CHECK:       scf.yield %[[INSERT0]], %[[INSERT1]], %[[INSERT2]]
+//      CHECK:     scf.yield %[[INNER_RESULT]]#0, %[[INNER_RESULT]]#1, %[[INNER_RESULT]]#2
+//      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#0
+
+// -----
+
+func.func @matmul_sequence_fusion(%arg0: tensor<?x?xf32>, %arg1: tensor<?x?xf32>,
+    %arg2: tensor<?x?xf32>, %arg3: tensor<?x?xf32>, %arg4: tensor<?x?xf32>,
+    %arg5: tensor<?x?xf32>, %arg6: tensor<?x?xf32>)
+    -> (tensor<?x?xf32>, tensor<?x?xf32>, tensor<?x?xf32>) {
+  %0 = linalg.matmul ins(%arg0, %arg1 : tensor<?x?xf32>, tensor<?x?xf32>)
+    outs(%arg2 : tensor<?x?xf32>) -> tensor<?x?xf32> // [M, N0] * [N0, N1]
+  %1 = linalg.matmul ins(%0, %arg3 : tensor<?x?xf32>, tensor<?x?xf32>)
+    outs(%arg4 : tensor<?x?xf32>) -> tensor<?x?xf32> // [M, N1] * [N1, N2]
+  %2 = linalg.matmul
+    {__internal_linalg_transform__ = "gemm_sequence_fusion"}
+    ins(%1, %arg5 : tensor<?x?xf32>, tensor<?x?xf32>)
+    outs(%arg6 : tensor<?x?xf32>) -> tensor<?x?xf32> // [M, N2] * [N2, N3]
+  return %0, %1, %2 : tensor<?x?xf32>, tensor<?x?xf32>, tensor<?x?xf32>
+}
+
+//       CHECK: #[[MAP:.+]] = affine_map<(d0)[s0, s1] -> (10, -d0 + s1)>
+//       CHECK: func @matmul_sequence_fusion(
+//  CHECK-SAME:   %[[ARG0:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG2:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG3:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG4:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG5:[a-zA-Z0-9_]+]]: tensor<?x?xf32>
+//  CHECK-SAME:   %[[ARG6:[a-zA-Z0-9_]+]]: tensor<?x?xf32>)
+//   CHECK-DAG:   %[[C0:.+]] = arith.constant 0 : index
+//   CHECK-DAG:   %[[C1:.+]] = arith.constant 1 : index
+//   CHECK-DAG:   %[[N0:.+]] = tensor.dim %[[ARG0]], %[[C1]]
+//   CHECK-DAG:   %[[ORIG_GEMM1:.+]] = linalg.matmul ins(%[[ARG0]], %[[ARG1]] :
+//   CHECK-DAG:   %[[N1:.+]] = tensor.dim %[[ORIG_GEMM1]], %[[C1]]
+//   CHECK-DAG:   %[[ORIG_GEMM2:.+]] = linalg.matmul ins(%[[ORIG_GEMM1]], %[[ARG3]] :
+//   CHECK-DAG:   %[[M:.+]] = tensor.dim %[[ORIG_GEMM2]], %[[C0]]
+//   CHECK-DAG:   %[[N2:.+]] = tensor.dim %[[ORIG_GEMM2]], %[[C1]]
+//   CHECK-DAG:   %[[N3:.+]] = tensor.dim %[[ARG5]], %[[C1]]
+//       CHECK:   %[[R0:[a-zA-Z0-9]+]]:3 = scf.for %[[IV:[a-zA-Z0-9_]+]] =
+//  CHECK-SAME:       iter_args(%[[ARG8:[a-zA-Z0-9]+]] = %[[ARG6]], %[[ARG9:[a-zA-Z0-9]+]] = %[[ARG4]], %[[ARG10:[a-zA-Z0-9]+]] = %[[ARG2]])
+//   CHECK-DAG:     %[[TILE_M:.+]] = affine.min #[[MAP]](%[[IV]])[%{{.+}}, %[[M]]]
+//   CHECK-DAG:     %[[SLICE_ARG0:.+]] = tensor.extract_slice %[[ARG0]][%[[IV]], 0] [%[[TILE_M]], %[[N0]]]
+//   CHECK-DAG:     %[[SLICE_ARG1:.+]] = tensor.extract_slice %[[ARG1]][0, 0] [%[[N0]], %[[N1]]]
+//   CHECK-DAG:     %[[SLICE_ARG2:.+]] = tensor.extract_slice %[[ARG10]][%[[IV]], 0] [%[[TILE_M]], %[[N1]]]
+//   CHECK-DAG:     %[[TILE_GEMM1:.+]] = linalg.matmul ins(%[[SLICE_ARG0]], %[[SLICE_ARG1]] :
+//  CHECK-SAME:         outs(%[[SLICE_ARG2]] :
+//   CHECK-DAG:     %[[SLICE_ARG3:.+]] = tensor.extract_slice %[[ARG3]][0, 0] [%[[N1]], %[[N2]]]
+//   CHECK-DAG:     %[[SLICE_ARG4:.+]] = tensor.extract_slice %[[ARG9]][%[[IV]], 0] [%[[TILE_M]], %[[N2]]]
+//   CHECK-DAG:     %[[TILE_GEMM2:.+]] = linalg.matmul ins(%[[TILE_GEMM1]], %[[SLICE_ARG3]] :
+//  CHECK-SAME:         outs(%[[SLICE_ARG4]] :
+//   CHECK-DAG:     %[[SLICE_ARG5:.+]] = tensor.extract_slice %[[ARG5]][0, 0] [%[[N2]], %[[N3]]]
+//   CHECK-DAG:     %[[SLICE_ARG6:.+]] = tensor.extract_slice %[[ARG8]][%[[IV]], 0] [%[[TILE_M]], %[[N3]]]
+//   CHECK-DAG:     %[[TILE_GEMM3:.+]] = linalg.matmul
+//  CHECK-SAME:         ins(%[[TILE_GEMM2]], %[[SLICE_ARG5]] :
+//  CHECK-SAME:         outs(%[[SLICE_ARG6]] :
+//   CHECK-DAG:     %[[UPDATE0:.+]] = tensor.insert_slice %[[TILE_GEMM3]] into %[[ARG8]][%[[IV]], 0] [%[[TILE_M]], %[[N3]]]
+//   CHECK-DAG:     %[[UPDATE1:.+]] = tensor.insert_slice %[[TILE_GEMM2]] into %[[ARG9]][%[[IV]], 0] [%[[TILE_M]], %[[N3]]]
+//   CHECK-DAG:     %[[UPDATE2:.+]] = tensor.insert_slice %[[TILE_GEMM1]] into %[[ARG10]][%[[IV]], 0] [%[[TILE_M]], %[[N3]]]
+//       CHECK:     scf.yield %[[UPDATE0]], %[[UPDATE1]], %[[UPDATE2]]
+//       CHECK:   return %[[R0]]#2, %[[R0]]#1, %[[R0]]#0
+
+// -----
+
+func.func @reduction_sequence(%arg0: tensor<30x3xf32>) -> tensor<30x3xf32> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %cst_0 = arith.constant 0xFF800000 : f32
+  %0 = linalg.init_tensor [30] : tensor<30xf32>
+  %1 = linalg.fill ins(%cst_0 : f32) outs(%0 : tensor<30xf32>) -> tensor<30xf32>
+  %2 = linalg.generic {
+      indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0)>],
+      iterator_types = ["parallel", "reduction"]}
+      ins(%arg0 : tensor<30x3xf32>) outs(%1 : tensor<30xf32>) {
+    ^bb0(%arg1: f32, %arg2: f32):
+      %8 = arith.maxf %arg2, %arg1 : f32
+      linalg.yield %8 : f32
+    } -> tensor<30xf32>
+  %3 = linalg.init_tensor [30, 3] : tensor<30x3xf32>
+  %4 = linalg.fill ins(%cst : f32) outs(%0 : tensor<30xf32>) -> tensor<30xf32>
+  %5:2 = linalg.generic {
+      indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0)>,
+                       affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0)>],
+      iterator_types = ["parallel", "reduction"]}
+      ins(%arg0, %2 : tensor<30x3xf32>, tensor<30xf32>) outs(%3, %4 : tensor<30x3xf32>, tensor<30xf32>) {
+    ^bb0(%arg1: f32, %arg2: f32, %arg3: f32, %arg4: f32):
+      %8 = arith.subf %arg1, %arg2 : f32
+      %9 = math.exp %8 : f32
+      %10 = arith.addf %arg4, %9 : f32
+      linalg.yield %9, %10 : f32, f32
+    } -> (tensor<30x3xf32>, tensor<30xf32>)
+  %6 = linalg.generic {
+      __internal_linalg_transform__ = "reduction_sequence_fusion",
+      indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0)>,
+                       affine_map<(d0, d1) -> (d0, d1)>],
+      iterator_types = ["parallel", "parallel"]}
+      ins(%5#0, %5#1 : tensor<30x3xf32>, tensor<30xf32>) outs(%3 : tensor<30x3xf32>) {
+    ^bb0(%arg1: f32, %arg2: f32, %arg3: f32):
+      %8 = arith.divf %arg1, %arg2 : f32
+      linalg.yield %8 : f32
+    } -> tensor<30x3xf32>
+  return %6 : tensor<30x3xf32>
+}
+//       CHECK: func @reduction_sequence(%[[ARG0:.+]]: tensor<30x3xf32>)
+//   CHECK-DAG:   %[[INIT0:.+]] = linalg.init_tensor [30]
+//   CHECK-DAG:   %[[INIT1:.+]] = linalg.init_tensor [30, 3]
+//       CHECK:   %[[RESULT:[a-zA-Z0-9]+]]:6 = scf.for %[[IV:[a-zA-Z0-9]+]]
+//  CHECK-SAME:       iter_args(%[[ITERARG0:[a-zA-Z0-9]+]] = %[[INIT1]], %[[ITERARG1:[a-zA-Z0-9]+]] = %[[INIT1]], %[[ITERARG2:[a-zA-Z0-9]+]] = %[[INIT0]], %[[ITERARG3:[a-zA-Z0-9]+]] = %[[INIT0]], %[[ITERARG4:[a-zA-Z0-9]+]] = %[[INIT0]], %[[ITERARG5:[a-zA-Z0-9]+]] = %[[INIT0]]
+//   CHECK-DAG:     %[[ARG0_SLICE:.+]] = tensor.extract_slice %[[ARG0]][%[[IV]], 0]
+//   CHECK-DAG:     %[[ITERARG4_SLICE:.+]] = tensor.extract_slice %[[ITERARG4]][%[[IV]]]
+//       CHECK:     %[[FILL0:.+]] = linalg.fill
+//  CHECK-SAME:         outs(%[[ITERARG4_SLICE]] :
+//       CHECK:     %[[GENERIC0:.+]] = linalg.generic
+//  CHECK-SAME:         ins(%[[ARG0_SLICE]] :
+//  CHECK-SAME:         outs(%[[FILL0]] :
+//   CHECK-DAG:     %[[ITERARG1_SLICE:.+]] = tensor.extract_slice %[[ITERARG1]][%[[IV]], 0]
+//   CHECK-DAG:     %[[ITERARG5_SLICE:.+]] = tensor.extract_slice %[[ITERARG5]][%[[IV]]]
+//       CHECK:     %[[FILL1:.+]] = linalg.fill
+//  CHECK-SAME:         outs(%[[ITERARG5_SLICE]] :
+//       CHECK:     %[[GENERIC1:.+]]:2 = linalg.generic
+//  CHECK-SAME:         ins(%[[ARG0_SLICE]], %[[GENERIC0]] :
+//  CHECK-SAME:         outs(%[[ITERARG1_SLICE]], %[[FILL1]] :
+//       CHECK:     %[[ITERARG0_SLICE:.+]] = tensor.extract_slice %[[ITERARG0]][%[[IV]], 0]
+//       CHECK:     %[[GENERIC2:.+]] = linalg.generic
+//  CHECK-SAME:         ins(%[[GENERIC1]]#0, %[[GENERIC1]]#1 :
+//  CHECK-SAME:         outs(%[[ITERARG0_SLICE]] :
+//   CHECK-DAG:     %[[INSERTSLICE0:.+]] = tensor.insert_slice %[[GENERIC2]] into %[[ITERARG0]][%[[IV]], 0]
+//   CHECK-DAG:     %[[INSERTSLICE1:.+]] = tensor.insert_slice %[[GENERIC1]]#0 into %[[ITERARG1]][%[[IV]], 0]
+//   CHECK-DAG:     %[[INSERTSLICE2:.+]] = tensor.insert_slice %[[GENERIC1]]#1 into %[[ITERARG2]][%[[IV]]]
+//   CHECK-DAG:     %[[INSERTSLICE3:.+]] = tensor.insert_slice %[[GENERIC0]] into %[[ITERARG3]][%[[IV]]]
+//   CHECK-DAG:     %[[INSERTSLICE4:.+]] = tensor.insert_slice %[[FILL0]] into %[[ITERARG4]][%[[IV]]]
+//   CHECK-DAG:     %[[INSERTSLICE5:.+]] = tensor.insert_slice %[[FILL1]] into %[[ITERARG5]][%[[IV]]]
+//       CHECK:     scf.yield %[[INSERTSLICE0]], %[[INSERTSLICE1]], %[[INSERTSLICE2]], %[[INSERTSLICE3]], %[[INSERTSLICE4]], %[[INSERTSLICE5]]
+//       CHECK:   return %[[RESULT]]#0

mlir/test/lib/Interfaces/TilingInterface/TestTilingInterface.cpp

 private:
   linalg::LinalgTransformationFilter filter;
 };
 
 /// Pattern for testing `TileConsumerAndFuseProducersUsingSCFForOp` pattern
 /// (that tiles and fuses operations using the `TilingInterface` with `scf.for`
 /// ops for iterating over the tiles) while using a `filter` to avoid recursive
 /// application.
-struct TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter
-    : public scf::TileConsumerAndFuseProducersUsingSCFForOp {
-  TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter(
-      MLIRContext *context, scf::SCFTilingOptions options,
+struct TestTileConsumerAndFuseProducersGreedilyUsingSCFForOpWithFilter
+    : public scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp {
+  TestTileConsumerAndFuseProducersGreedilyUsingSCFForOpWithFilter(
+      MLIRContext *context, scf::SCFTileAndFuseOptions options,
       linalg::LinalgTransformationFilter filter =
           linalg::LinalgTransformationFilter(),
       PatternBenefit benefit = 1)
-      : scf::TileConsumerAndFuseProducersUsingSCFForOp(context, options,
+      : scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp(context, options,
                                                                benefit),
         filter(filter) {}
 
   /// Construct a generic pattern applied to `opName`.
-  TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter(
-      StringRef opName, MLIRContext *context, scf::SCFTilingOptions options,
+  TestTileConsumerAndFuseProducersGreedilyUsingSCFForOpWithFilter(
+      StringRef opName, MLIRContext *context,
+      scf::SCFTileAndFuseOptions options,
       linalg::LinalgTransformationFilter filter =
           linalg::LinalgTransformationFilter(),
       PatternBenefit benefit = 1)
-      : scf::TileConsumerAndFuseProducersUsingSCFForOp(context, options,
+      : scf::TileConsumerAndFuseProducersGreedilyUsingSCFForOp(context, options,
                                                                benefit),
         filter(filter) {}
 
   LogicalResult matchAndRewrite(TilingInterface op,
                                 PatternRewriter &rewriter) const override {
     if (failed(filter.checkAndNotify(rewriter, op)))
       return failure();
 
     auto tileAndFuseResult = returningMatchAndRewrite(op, rewriter);
Option<bool> testTiling{
       llvm::cl::init(false)};
 
   Option<bool> testTileConsumerAndFuseProducer{
       *this, "tile-consumer-and-fuse-producer-using-scf-for",
       llvm::cl::desc("Test tile and fuse transformation using TilingInterface "
                      "with scf.for operations"),
       llvm::cl::init(false)};
 
+  Option<bool> testTileConsumerAndFuseProducerWithoutRedundancy{
+      *this, "tile-consumer-and-fuse-yield-producer-using-scf-for",
+      llvm::cl::desc(
+          "Test tile and fuse transformation using TilingInterface with "
+          "scf.for operations, when producers can be fused without redundancy"),
+      llvm::cl::init(false)};
+
   Option<bool> testLoweringToScalar{
       *this, "lower-to-scalar-using-scf-for",
       llvm::cl::desc("Test lowering to scalar implementation using "
                      "TilingInterface with scf.for operations"),
       llvm::cl::init(false)};
 
   void runOnOperation() override;
 
 private:
   void addTestPatterns(MLIRContext *context, RewritePatternSet &patterns);
 };
 } // namespace
 
-template <class Pattern>
-static void
-addPatternForTiling(MLIRContext *context, RewritePatternSet &patterns,
-                    StringRef filterName, ArrayRef<int64_t> tileSizes,
+static void addPatternForTiling(MLIRContext *context,
+                                RewritePatternSet &patterns,
+                                StringRef filterName,
+                                ArrayRef<int64_t> tileSizes,
                                 ArrayRef<unsigned> interchange = {}) {
   scf::SCFTilingOptions tilingOptions;
   tilingOptions.setTileSizes(tileSizes).setInterchange(interchange);
   linalg::LinalgTransformationFilter filter(
       StringAttr::get(context, filterName), StringAttr::get(context, "tiled"));
-  patterns.add<Pattern>(context, tilingOptions, filter);
+  patterns.add<TestTileUsingSCFForOpWithFilter>(context, tilingOptions, filter);
+}
+
+static void
+addPatternForTileAndFuse(MLIRContext *context, RewritePatternSet &patterns,
+                         StringRef filterName, ArrayRef<int64_t> tileSizes,
+                         bool producerCanBeFusedWithoutRedundancy = false,
+                         ArrayRef<unsigned> interchange = {}) {
+  scf::SCFTileAndFuseOptions tileAndFuseOptions;
+  tileAndFuseOptions.tilingOptions.setTileSizes(tileSizes).setInterchange(
+      interchange);
+  tileAndFuseOptions.setProducerCanBeFusedWithoutRedundantComputations(
+      producerCanBeFusedWithoutRedundancy);
+  linalg::LinalgTransformationFilter filter(
+      StringAttr::get(context, filterName), StringAttr::get(context, "tiled"));
+  patterns.add<TestTileConsumerAndFuseProducersGreedilyUsingSCFForOpWithFilter>(
+      context, tileAndFuseOptions, filter);
 }
 
 void TestTilingInterfacePass::addTestPatterns(MLIRContext *context,
                                               RewritePatternSet &patterns) {
   if (testTiling) {
     // 1. Tiling M and N dims of `linalg.matmul` on tensors.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "simple_gemm", {10, 20});
+    addPatternForTiling(context, patterns, "simple_gemm", {10, 20});
     // 2. Tiling M, N and K of `linalg.matmul` on buffers.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "simple_gemm_memref", {10, 20, 30});
+    addPatternForTiling(context, patterns, "simple_gemm_memref", {10, 20, 30});
     // 3. Tiling 3D parallel generic op which implements a transpose
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "parallel_generic_transpose", {10, 0, 20});
+    addPatternForTiling(context, patterns, "parallel_generic_transpose",
+                        {10, 0, 20});
     // 4. Tiling 2D conv op.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "simple_conv", {0, 0, 0, 0, 10, 20, 30});
+    addPatternForTiling(context, patterns, "simple_conv",
+                        {0, 0, 0, 0, 10, 20, 30});
     // 5. Tiling a simple op with `linalg.index` inside.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "indexed_semantics", {10, 20});
+    addPatternForTiling(context, patterns, "indexed_semantics", {10, 20});
     // 6. Tiling + interchange of an operation
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "gemm_interchange", {10, 20, 30}, {1, 2, 0});
+    addPatternForTiling(context, patterns, "gemm_interchange", {10, 20, 30},
+                        {1, 2, 0});
     // 7. Tiling for 2D pad tensor operations.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "pad_2dtiling", {2, 3});
+    addPatternForTiling(context, patterns, "pad_2dtiling", {2, 3});
     // 8. Tiling inner dimension of 2d pad tensor operations.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "pad_inner_tiling", {0, 3});
+    addPatternForTiling(context, patterns, "pad_inner_tiling", {0, 3});
     // 9. Tiling inner dimension of 2d pad tensor operations.
-    addPatternForTiling<TestTileUsingSCFForOpWithFilter>(
-        context, patterns, "pad_outer_tiling", {2, 3});
+    addPatternForTiling(context, patterns, "pad_outer_tiling", {2, 3});
 
     return;
   }
   if (testTileConsumerAndFuseProducer) {
-    // 1. Tile and fuse of gemm with bias-add operation.
-    addPatternForTiling<
-        TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter>(
-        context, patterns, "fusion", {10, 20});
-    addPatternForTiling<
-        TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter>(
-        context, patterns, "gemm_fusion", {10});
-    addPatternForTiling<
-        TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter>(
-        context, patterns, "gemm_interchange_fusion", {10, 20}, {1, 0});
-    addPatternForTiling<
-        TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter>(
-        context, patterns, "gemm_plus_gemm_fusion", {10, 20});
-    addPatternForTiling<
-        TestTileConsumerAndFuseProducersUsingSCFForOpWithFilter>(
-        context, patterns, "gemm_sequence_fusion", {10});
+    // 1. Tile and fuse of gemm with fill producer and bias-add consumer.
+    addPatternForTileAndFuse(context, patterns, "fusion", {10, 20});
+    // 2. Tile and fuse sequence of GEMMs, by fusing only along M.
+    addPatternForTileAndFuse(context, patterns, "gemm_fusion", {10});
+    // 3. Tile and fuse gemm with consumer + interchange of tiled loops.
+    addPatternForTileAndFuse(context, patterns, "gemm_interchange_fusion",
+                             {10, 20}, false, {1, 0});
+    // 4. Tile and fuse matmul + transpose(matmul). Will introduce redundant
+    // computations.
+    addPatternForTileAndFuse(context, patterns, "gemm_plus_gemm_fusion",
+                             {10, 20});
+    // 5. Tile and fuse a sequence of GEMMs by tiling and fusing only along M
+    // dimension.
+    addPatternForTileAndFuse(context, patterns, "gemm_sequence_fusion", {10});
     return;
   }
+  if (testTileConsumerAndFuseProducerWithoutRedundancy) {
+    // 1. Tile and fuse of gemm with fill producer and bias-add consumer,
+    // returning both the gemm value and the bias-add value.
+    addPatternForTileAndFuse(context, patterns, "fusion", {10, 20}, true);
+    // 2. Tile and fuse sequence of GEMMs, by fusing only along M, return all
+    // the individual GEMM values.
+    addPatternForTileAndFuse(context, patterns, "gemm_fusion", {10}, true);
+    // 3. Tile and fuse gemm with consumer + interchange of tiled loops.
+    addPatternForTileAndFuse(context, patterns, "gemm_interchange_fusion",
+                             {10, 20}, true, {1, 0});
+    // 4.  Tile and fuse a sequence of GEMMs by tiling and fusing only along M
+    // dimension.
+    addPatternForTileAndFuse(context, patterns, "gemm_sequence_fusion", {10},
+                             true);
+    // 5. Fusion of back-to-back-reduction ops
+    addPatternForTileAndFuse(context, patterns, "reduction_sequence_fusion",
+                             {10}, true);
+  }
   if (testLoweringToScalar) {
     patterns.add<scf::LowerToLoopsUsingSCFForOp>(context);
   }
 }
 
 void TestTilingInterfacePass::runOnOperation() {
   MLIRContext *context = &getContext();