[mlir][Vector] Add option to fully unroll for VectorTransfer to SCF lowering

[nicolasvasilache]

Summary

Previously, the only support partial lowering from vector transfers to SCF was
going through loops. This requires a dedicated allocation and extra memory
roundtrips because LLVM aggregates cannot be indexed dynamically (for more
details see the deep-dive).

This revision allows specifying full unrolling which removes this additional roundtrip.
This should be used carefully though because full unrolling will spill, negating the
benefits of removing the interim alloc in the first place.

Proper heuristics are left for a later time.

[nicolasvasilache]

Update

[nicolasvasilache]

Rebase.

[ftynse]

mlir/include/mlir/Conversion/VectorToSCF/VectorToSCF.h

16

struct or you'll break msvc

57

Something's missing here

59

Copy-pasta?

[ftynse]

mlir/include/mlir/Dialect/Vector/VectorOps.td

377

I was confused by this comment wrt the builder above. Consider "which assumes the values in position are defined by ConstantIndexOp`".

544

The position of this comment is just weird

545

Nit: no need for #, string literals are auto-concatenated like in C

mlir/lib/Conversion/VectorToSCF/VectorToSCF.cpp

149

Nit: we mildly prefer using auto when the type is either very long or obvious from the context. In this function, types are almost never obvious and one has to look up the declarations of other functions.

226

Typo: inbBounds

243

Plz remind me if Append() appends to the end of the block or before the terminator

261

https://llvm.org/docs/CodingStandards.html#do-not-use-braced-initializer-lists-to-call-a-constructor

Maybe just do vector = vector_insert(vector, ...); loop_yield(vector);, reads better IMO

329

inbBounds again

mlir/lib/Dialect/Vector/VectorUtils.cpp

36

std::accumulate(basis.begin(), basis.end(), 1, std::multiplies<int64_t>()); ?

[aartbik]

mlir/include/mlir/Conversion/VectorToSCF/VectorToSCF.h

78

nit: broadcasts (same verb form)

mlir/test/Conversion/VectorToLoops/vector-to-loops.mlir

225 ↗

(On Diff #264668)

empty line?

[nicolasvasilache]

mlir/include/mlir/Conversion/VectorToSCF/VectorToSCF.h

16

dropped it from here

mlir/lib/Conversion/VectorToSCF/VectorToSCF.cpp

243

before the terminator if it is there otherwise end of the block.

[nicolasvasilache]

Address review.

[nicolasvasilache]

Rebase.

[rriddle]

mlir/lib/Dialect/Vector/VectorOps.cpp

478

getDefiningOp can return null, which would make this crash instead of assert.

757

Same here.

mlir/include/mlir/Conversion/VectorToSCF/VectorToSCF.h

 //===- VectorToSCF.h - Utils to convert from the vector dialect -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_CONVERSION_VECTORTOSCF_VECTORTOSCF_H_
 #define MLIR_CONVERSION_VECTORTOSCF_VECTORTOSCF_H_
 
+#include "mlir/IR/PatternMatch.h"
+
 namespace mlir {
 class MLIRContext;
 class OwningRewritePatternList;
 
-/// Collect a set of patterns to convert from the Vector dialect to loops + std.
-void populateVectorToSCFConversionPatterns(OwningRewritePatternList &patterns,
-                                           MLIRContext *context);
+/// Control whether unrolling is used when lowering vector transfer ops to SCF.
+///
+/// Case 1:
+/// =======
+/// When `unroll` is false, a temporary buffer is created through which
+/// individual 1-D vector are staged. this is consistent with the lack of an
+/// LLVM instruction to dynamically index into an aggregate (see the Vector
+/// dialect lowering to LLVM deep dive).
+/// An instruction such as:
+/// ```
+///    vector.transfer_write %vec, %A[%base, %base] :
+///      vector<17x15xf32>, memref<?x?xf32>
+/// ```
+/// Lowers to pseudo-IR resembling:
+/// ```
+///    %0 = alloc() : memref<17xvector<15xf32>>
+///    %1 = vector.type_cast %0 :
+///      memref<17xvector<15xf32>> to memref<vector<17x15xf32>>
+///    store %vec, %1[] : memref<vector<17x15xf32>>
+///    %dim = dim %A, 0 : memref<?x?xf32>
+///    affine.for %I = 0 to 17 {
+///      %add = affine.apply %I + %base
+///      %cmp = cmpi "slt", %add, %dim : index
+///      scf.if %cmp {
+///        %vec_1d = load %0[%I] : memref<17xvector<15xf32>>
+///        vector.transfer_write %vec_1d, %A[%add, %base] :
+///          vector<15xf32>, memref<?x?xf32>
+/// ```
+///
+/// Case 2:
+/// =======
+/// When `unroll` is true, the temporary buffer is skipped and static indices
+/// into aggregates can be used (see the Vector dialect lowering to LLVM deep
+/// dive).
+/// An instruction such as:
+/// ```
+///    vector.transfer_write %vec, %A[%base, %base] :
+///      vector<3x15xf32>, memref<?x?xf32>
+/// ```
+/// Lowers to pseudo-IR resembling:
+/// ```
+///    %0 = vector.extract %arg2[0] : vector<3x15xf32>
+///    vector.transfer_write %0, %arg0[%arg1, %arg1] : vector<15xf32>,
+///    memref<?x?xf32> %1 = affine.apply #map1()[%arg1] %2 = vector.extract
+///    %arg2[1] : vector<3x15xf32> vector.transfer_write %2, %arg0[%1, %arg1] :
+///    vector<15xf32>, memref<?x?xf32> %3 = affine.apply #map2()[%arg1] %4 =
+///    vector.extract %arg2[2] : vector<3x15xf32> vector.transfer_write %4,
+///    %arg0[%3, %arg1] : vector<15xf32>, memref<?x?xf32>
+/// ```
+struct VectorTransferToSCFOptions {
+  bool unroll = false;
+  VectorTransferToSCFOptions &setUnroll(bool u) {
+    unroll = u;
+    return *this;
+  }
+};
+
+/// Implements lowering of TransferReadOp and TransferWriteOp to a
+/// proper abstraction for the hardware.
+///
+/// There are multiple cases.
+///
+/// Case A: Permutation Map does not permute or broadcast.
+/// ======================================================
+///
+/// Progressive lowering occurs to 1-D vector transfer ops according to the
+/// description in `VectorTransferToSCFOptions`.
+///
+/// Case B: Permutation Map permutes and/or broadcasts.
+/// ======================================================
+///
+/// This path will be progressively deprecated and folded into the case above by
+/// using vector broadcast and transpose operations.
+///
+/// This path only emits a simple loop nest that performs clipped pointwise
+/// copies from a remote to a locally allocated memory.
+///
+/// Consider the case:
+///
+/// ```mlir
+///    // Read the slice `%A[%i0, %i1:%i1+256, %i2:%i2+32]` into
+///    // vector<32x256xf32> and pad with %f0 to handle the boundary case:
+///    %f0 = constant 0.0f : f32
+///    scf.for %i0 = 0 to %0 {
+///      scf.for %i1 = 0 to %1 step %c256 {
+///        scf.for %i2 = 0 to %2 step %c32 {
+///          %v = vector.transfer_read %A[%i0, %i1, %i2], %f0
+///               {permutation_map: (d0, d1, d2) -> (d2, d1)} :
+///               memref<?x?x?xf32>, vector<32x256xf32>
+///    }}}
+/// ```
+///
+/// The rewriters construct loop and indices that access MemRef A in a pattern
+/// resembling the following (while guaranteeing an always full-tile
+/// abstraction):
+///
+/// ```mlir
+///    scf.for %d2 = 0 to %c256 {
+///      scf.for %d1 = 0 to %c32 {
+///        %s = %A[%i0, %i1 + %d1, %i2 + %d2] : f32
+///        %tmp[%d2, %d1] = %s
+///      }
+///    }
+/// ```
+///
+/// In the current state, only a clipping transfer is implemented by `clip`,
+/// which creates individual indexing expressions of the form:
+///
+/// ```mlir-dsc
+///    auto condMax = i + ii < N;
+///    auto max = std_select(condMax, i + ii, N - one)
+///    auto cond = i + ii < zero;
+///    std_select(cond, zero, max);
+/// ```
+///
+/// In the future, clipping should not be the only way and instead we should
+/// load vectors + mask them. Similarly on the write side, load/mask/store for
+/// implementing RMW behavior.
+///
+/// Lowers TransferOp into a combination of:
+///   1. local memory allocation;
+///   2. perfect loop nest over:
+///      a. scalar load/stores from local buffers (viewed as a scalar memref);
+///      a. scalar store/load to original memref (with clipping).
+///   3. vector_load/store
+///   4. local memory deallocation.
+/// Minor variations occur depending on whether a TransferReadOp or
+/// a TransferWriteOp is rewritten.
+template <typename TransferOpTy>
+struct VectorTransferRewriter : public RewritePattern {
+  explicit VectorTransferRewriter(VectorTransferToSCFOptions options,
 
+  /// Used for staging the transfer in a local buffer.
+  MemRefType tmpMemRefType(TransferOpTy transfer) const;
+
+  /// Performs the rewrite.
+  LogicalResult matchAndRewrite(Operation *op,
+                                PatternRewriter &rewriter) const override;
+
+  /// See description of `VectorTransferToSCFOptions`.
+  VectorTransferToSCFOptions options;
+};
+
+/// Collect a set of patterns to convert from the Vector dialect to SCF + std.
+void populateVectorToSCFConversionPatterns(
+    OwningRewritePatternList &patterns, MLIRContext *context,
+    const VectorTransferToSCFOptions &options = VectorTransferToSCFOptions());
+
 } // namespace mlir
 
 #endif // MLIR_CONVERSION_VECTORTOSCF_VECTORTOSCF_H_

mlir/include/mlir/Dialect/Vector/VectorOps.td

let description = [{
 
     ```mlir
     %1 = vector.extract %0[3]: vector<4x8x16xf32>
     %2 = vector.extract %0[3, 3, 3]: vector<4x8x16xf32>
     ```
   }];
   let builders = [OpBuilder<
     "OpBuilder &builder, OperationState &result, Value source,"
-    "ArrayRef<int64_t>">];
+    "ArrayRef<int64_t> position">,
+    // Convenience builder which assumes the values in `position` are defined by
+    // ConstantIndexOp.
+    OpBuilder<"OpBuilder &builder, OperationState &result, Value source,"
+    "ValueRange position">];
   let extraClassDeclaration = [{
     static StringRef getPositionAttrName() { return "position"; }
     VectorType getVectorType() {
       return vector().getType().cast<VectorType>();
     }
   }];
 }
 
let description = [{
     %5 = vector.insert %3, %4[3, 3, 3] : f32 into vector<4x8x16xf32>
     ```
   }];
   let assemblyFormat = [{
     $source `,` $dest $position attr-dict `:` type($source) `into` type($dest)
   }];
 
   let builders = [OpBuilder<
-    "OpBuilder &builder, OperationState &result, Value source, " #
-    "Value dest, ArrayRef<int64_t>">];
+    "OpBuilder &builder, OperationState &result, Value source, "
+    "Value dest, ArrayRef<int64_t> position">,
+    // Convenience builder which assumes all values are constant indices.
+    OpBuilder<
+    "OpBuilder &builder, OperationState &result, Value source, "
+    "Value dest, ValueRange position">];
   let extraClassDeclaration = [{
     static StringRef getPositionAttrName() { return "position"; }
     Type getSourceType() { return source().getType(); }
     VectorType getDestVectorType() {
       return dest().getType().cast<VectorType>();
     }
   }];
 }

mlir/include/mlir/Dialect/Vector/VectorUtils.h

-//===- VectorUtils.h - VectorOps Utilities ------------------*- C++ -*-=======//
+//===- VectorUtils.h - Vector Utilities -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_DIALECT_VECTOR_VECTORUTILS_H_

 class AffineMap;
 class Location;
 class MemRefType;
 class OpBuilder;
 class Operation;
 class Value;
 class VectorType;
 
+/// Return the number of elements of basis, `0` if empty.
+int64_t computeMaxLinearIndex(ArrayRef<int64_t> basis);
+
+/// Given a shape with sizes greater than 0 along all dimensions,
+/// return the distance, in number of elements, between a slice in a dimension
+/// and the next slice in the same dimension.
+///   e.g. shape[3, 4, 5] -> linearization_basis[20, 5, 1]
+SmallVector<int64_t, 8> computeStrides(ArrayRef<int64_t> shape);
+
 /// Given the shape and sizes of a vector, returns the corresponding
 /// strides for each dimension.
+/// TODO: needs better doc of how it is used.
 SmallVector<int64_t, 4> computeStrides(ArrayRef<int64_t> shape,
                                        ArrayRef<int64_t> sizes);
 
 /// Computes and returns the linearized index of 'offsets' w.r.t. 'basis'.
 int64_t linearize(ArrayRef<int64_t> offsets, ArrayRef<int64_t> basis);
 
-/// Given the slice strides together with a linear index in the dimension
+/// Given the strides together with a linear index in the dimension
 /// space, returns the vector-space offsets in each dimension for a
 /// de-linearized index.
-SmallVector<int64_t, 4> delinearize(ArrayRef<int64_t> sliceStrides,
+SmallVector<int64_t, 4> delinearize(ArrayRef<int64_t> strides,
                                     int64_t linearIndex);
 
 /// Given the target sizes of a vector, together with vector-space offsets,
 /// returns the element-space offsets for each dimension.
 SmallVector<int64_t, 4>
 computeElementOffsetsFromVectorSliceOffsets(ArrayRef<int64_t> sizes,
                                             ArrayRef<int64_t> vectorOffsets);
 

mlir/lib/Conversion/VectorToSCF/VectorToSCF.cpp

 
 #include "mlir/Conversion/VectorToSCF/VectorToSCF.h"
 #include "mlir/Dialect/Affine/EDSC/Intrinsics.h"
 #include "mlir/Dialect/SCF/EDSC/Builders.h"
 #include "mlir/Dialect/SCF/EDSC/Intrinsics.h"
 #include "mlir/Dialect/StandardOps/EDSC/Intrinsics.h"
 #include "mlir/Dialect/Vector/EDSC/Intrinsics.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
+#include "mlir/Dialect/Vector/VectorUtils.h"
 #include "mlir/IR/AffineExpr.h"
 #include "mlir/IR/AffineMap.h"
 #include "mlir/IR/Attributes.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/Location.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/PatternMatch.h"

 ///   }
 ///   %res = load(%tmp, %0): memref<(major_dims) x vector<minor_dim x type>>):
 //      vector<(major_dims) x (minor_dims) x type>
 /// ```
 ///
 template <typename ConcreteOp>
 class NDTransferOpHelper {
 public:
-  NDTransferOpHelper(PatternRewriter &rewriter, ConcreteOp xferOp)
-      : rewriter(rewriter), loc(xferOp.getLoc()),
+  NDTransferOpHelper(PatternRewriter &rewriter, ConcreteOp xferOp,
+                     const VectorTransferToSCFOptions &options)
+      : rewriter(rewriter), options(options), loc(xferOp.getLoc()),
         scope(std::make_unique<ScopedContext>(rewriter, loc)), xferOp(xferOp),
         op(xferOp.getOperation()) {
     vectorType = xferOp.getVectorType();
     // TODO(ntv, ajcbik): when we go to k > 1-D vectors adapt minorRank.
     minorRank = 1;
     majorRank = vectorType.getRank() - minorRank;
     leadingRank = xferOp.getMemRefType().getRank() - (majorRank + minorRank);
     majorVectorType =

 
 private:
   /// Creates the loop nest on the "major" dimensions and calls the
   /// `loopBodyBuilder` lambda in the context of the loop nest.
   template <typename Lambda>
   void emitLoops(Lambda loopBodyBuilder);
 
   /// Operate within the body of `emitLoops` to:
-  ///   1. Compute the indexings `majorIvs + majorOffsets`.
-  ///   2. Compute a boolean that determines whether the first `majorIvs.rank()`
+  ///   1. Compute the indexings `majorIvs + majorOffsets` and save them in
+  ///      `majorIvsPlusOffsets`.
+  ///   2. Return a boolean that determines whether the first `majorIvs.rank()`
   ///      dimensions `majorIvs + majorOffsets` are all within `memrefBounds`.
-  ///   3. Create an IfOp conditioned on the boolean in step 2.
-  ///   4. Call a `thenBlockBuilder` and an `elseBlockBuilder` to append
-  ///      operations to the IfOp blocks as appropriate.
-  template <typename LambdaThen, typename LambdaElse>
-  void emitInBounds(ValueRange majorIvs, ValueRange majorOffsets,
+  Value emitInBoundsCondition(ValueRange majorIvs, ValueRange majorOffsets,
                               MemRefBoundsCapture &memrefBounds,
-                    LambdaThen thenBlockBuilder, LambdaElse elseBlockBuilder);
+                              SmallVectorImpl<Value> &majorIvsPlusOffsets);
 
   /// Common state to lower vector transfer ops.
   PatternRewriter &rewriter;
+  const VectorTransferToSCFOptions &options;
   Location loc;
   std::unique_ptr<ScopedContext> scope;
   ConcreteOp xferOp;
   Operation *op;
   // A vector transfer copies data between:
   //   - memref<(leading_dims) x (major_dims) x (minor_dims) x type>
   //   - vector<(major_dims) x (minor_dims) x type>
   unsigned minorRank;         // for now always 1
   unsigned majorRank;         // vector rank - minorRank
   unsigned leadingRank;       // memref rank - vector rank
   VectorType vectorType;      // vector<(major_dims) x (minor_dims) x type>
   VectorType majorVectorType; // vector<(major_dims) x type>
   VectorType minorVectorType; // vector<(minor_dims) x type>
   MemRefType memRefMinorVectorType; // memref<vector<(minor_dims) x type>>
 };
 
 template <typename ConcreteOp>
 template <typename Lambda>
 void NDTransferOpHelper<ConcreteOp>::emitLoops(Lambda loopBodyBuilder) {
   /// Loop nest operates on the major dimensions
   MemRefBoundsCapture memrefBoundsCapture(xferOp.memref());
+
+  if (options.unroll) {
+    auto shape = majorVectorType.getShape();
+    auto strides = computeStrides(shape);
+    unsigned numUnrolledInstances = computeMaxLinearIndex(shape);
+    ValueRange indices(xferOp.indices());
+    for (unsigned idx = 0; idx < numUnrolledInstances; ++idx) {
+      SmallVector<int64_t, 4> offsets = delinearize(strides, idx);
+      SmallVector<Value, 4> offsetValues =
+          llvm::to_vector<4>(llvm::map_range(offsets, [](int64_t off) -> Value {
+            return std_constant_index(off);
+          }));
+      loopBodyBuilder(offsetValues, indices.take_front(leadingRank),
+                      indices.drop_front(leadingRank).take_front(majorRank),
+                      indices.take_back(minorRank), memrefBoundsCapture);
+    }
+  } else {
     VectorBoundsCapture vectorBoundsCapture(majorVectorType);
     auto majorLbs = vectorBoundsCapture.getLbs();
     auto majorUbs = vectorBoundsCapture.getUbs();
     auto majorSteps = vectorBoundsCapture.getSteps();
     SmallVector<Value, 8> majorIvs(vectorBoundsCapture.rank());
     AffineLoopNestBuilder(majorIvs, majorLbs, majorUbs, majorSteps)([&] {
       ValueRange indices(xferOp.indices());
       loopBodyBuilder(majorIvs, indices.take_front(leadingRank),
                       indices.drop_front(leadingRank).take_front(majorRank),
                       indices.take_back(minorRank), memrefBoundsCapture);
     });
   }
+}
 
 template <typename ConcreteOp>
-template <typename LambdaThen, typename LambdaElse>
-void NDTransferOpHelper<ConcreteOp>::emitInBounds(
+Value NDTransferOpHelper<ConcreteOp>::emitInBoundsCondition(
     ValueRange majorIvs, ValueRange majorOffsets,
-    MemRefBoundsCapture &memrefBounds, LambdaThen thenBlockBuilder,
-    LambdaElse elseBlockBuilder) {
-  Value inBounds;
-  SmallVector<Value, 4> majorIvsPlusOffsets;
+    MemRefBoundsCapture &memrefBounds,
+    SmallVectorImpl<Value> &majorIvsPlusOffsets) {
+  Value inBoundsCondition;
   majorIvsPlusOffsets.reserve(majorIvs.size());
   unsigned idx = 0;
   for (auto it : llvm::zip(majorIvs, majorOffsets, memrefBounds.getUbs())) {
     Value iv = std::get<0>(it), off = std::get<1>(it), ub = std::get<2>(it);
     using namespace mlir::edsc::op;
     majorIvsPlusOffsets.push_back(iv + off);
     if (xferOp.isMaskedDim(leadingRank + idx)) {
-      Value inBounds2 = majorIvsPlusOffsets.back() < ub;
-      inBounds = (inBounds) ? (inBounds && inBounds2) : inBounds2;
+      Value inBounds = majorIvsPlusOffsets.back() < ub;
+      inBoundsCondition =
+          (inBoundsCondition) ? (inBoundsCondition && inBounds) : inBounds;
     }
     ++idx;
   }
-
-  if (inBounds) {
-    auto ifOp = ScopedContext::getBuilderRef().create<scf::IfOp>(
-        ScopedContext::getLocation(), TypeRange{}, inBounds,
-        /*withElseRegion=*/std::is_same<ConcreteOp, TransferReadOp>());
-    BlockBuilder(&ifOp.thenRegion().front(),
-                 Append())([&] { thenBlockBuilder(majorIvsPlusOffsets); });
-    if (std::is_same<ConcreteOp, TransferReadOp>())
-      BlockBuilder(&ifOp.elseRegion().front(),
-                   Append())([&] { elseBlockBuilder(majorIvsPlusOffsets); });
-  } else {
-    // Just build the body of the then block right here.
-    thenBlockBuilder(majorIvsPlusOffsets);
-  }
+  return inBoundsCondition;
 }
 
 template <>
 LogicalResult NDTransferOpHelper<TransferReadOp>::doReplace() {
-  Value alloc = std_alloc(memRefMinorVectorType);
+  Value alloc, result;
+  if (options.unroll)
+    result = std_splat(vectorType, xferOp.padding());
+  else
+    alloc = std_alloc(memRefMinorVectorType);
 
   emitLoops([&](ValueRange majorIvs, ValueRange leadingOffsets,
                 ValueRange majorOffsets, ValueRange minorOffsets,
                 MemRefBoundsCapture &memrefBounds) {
-    // If in-bounds, index into memref and lower to 1-D transfer read.
-    auto thenBlockBuilder = [&](ValueRange majorIvsPlusOffsets) {
+    /// Lambda to load 1-D vector in the current loop ivs + offset context.
+    auto load1DVector = [&](ValueRange majorIvsPlusOffsets) -> Value {
       SmallVector<Value, 8> indexing;
       indexing.reserve(leadingRank + majorRank + minorRank);
       indexing.append(leadingOffsets.begin(), leadingOffsets.end());
       indexing.append(majorIvsPlusOffsets.begin(), majorIvsPlusOffsets.end());
       indexing.append(minorOffsets.begin(), minorOffsets.end());
-
       Value memref = xferOp.memref();
       auto map = TransferReadOp::getTransferMinorIdentityMap(
           xferOp.getMemRefType(), minorVectorType);
       ArrayAttr masked;
       if (xferOp.isMaskedDim(xferOp.getVectorType().getRank() - 1)) {
         OpBuilder &b = ScopedContext::getBuilderRef();
         masked = b.getBoolArrayAttr({true});
       }
-      auto loaded1D = vector_transfer_read(minorVectorType, memref, indexing,
-                                           AffineMapAttr::get(map),
-                                           xferOp.padding(), masked);
-      // Store the 1-D vector.
-      std_store(loaded1D, alloc, majorIvs);
+      return vector_transfer_read(minorVectorType, memref, indexing,
+                                  AffineMapAttr::get(map), xferOp.padding(),
+                                  masked);
     };
-    // If out-of-bounds, just store a splatted vector.
-    auto elseBlockBuilder = [&](ValueRange majorIvsPlusOffsets) {
-      auto vector = std_splat(minorVectorType, xferOp.padding());
+
+    // 1. Compute the inBoundsCondition in the current loops ivs + offset
+    // context.
+    SmallVector<Value, 4> majorIvsPlusOffsets;
+    Value inBoundsCondition = emitInBoundsCondition(
+        majorIvs, majorOffsets, memrefBounds, majorIvsPlusOffsets);
+
+    if (inBoundsCondition) {
+      // 2. If the condition is not null, we need an IfOp, which may yield
+      // if `options.unroll` is true.
+      SmallVector<Type, 1> resultType;
+      if (options.unroll)
+        resultType.push_back(vectorType);
+      auto ifOp = ScopedContext::getBuilderRef().create<scf::IfOp>(
+          ScopedContext::getLocation(), resultType, inBoundsCondition,
+          /*withElseRegion=*/true);
+
+      // 3.a. If in-bounds, progressively lower to a 1-D transfer read.
+      BlockBuilder(&ifOp.thenRegion().front(), Append())([&] {
+        Value vector = load1DVector(majorIvsPlusOffsets);
+        // 3.a.i. If `options.unroll` is true, insert the 1-D vector in the
+        // aggregate. We must yield and merge with the `else` branch.
+        if (options.unroll) {
+          vector = vector_insert(vector, result, majorIvs);
+          (loop_yield(vector));
+          return;
+        }
+        // 3.a.ii. Otherwise, just go through the temporary `alloc`.
         std_store(vector, alloc, majorIvs);
-    };
-    emitInBounds(majorIvs, majorOffsets, memrefBounds, thenBlockBuilder,
-                 elseBlockBuilder);
       });
 
-  Value loaded =
-      std_load(vector_type_cast(MemRefType::get({}, vectorType), alloc));
-  rewriter.replaceOp(op, loaded);
+      // 3.b. If not in-bounds, splat a 1-D vector.
+      BlockBuilder(&ifOp.elseRegion().front(), Append())([&] {
+        Value vector = std_splat(minorVectorType, xferOp.padding());
+        // 3.a.i. If `options.unroll` is true, insert the 1-D vector in the
+        // aggregate. We must yield and merge with the `then` branch.
+        if (options.unroll) {
+          vector = vector_insert(vector, result, majorIvs);
+          (loop_yield(vector));
+          return;
+        }
+        // 3.b.ii. Otherwise, just go through the temporary `alloc`.
+        std_store(vector, alloc, majorIvs);
+      });
+      if (!resultType.empty())
+        result = *ifOp.results().begin();
+    } else {
+      // 4. Guaranteed in-bounds, progressively lower to a 1-D transfer read.
+      Value loaded1D = load1DVector(majorIvsPlusOffsets);
+      // 5.a. If `options.unroll` is true, insert the 1-D vector in the
+      // aggregate.
+      if (options.unroll)
+        result = vector_insert(loaded1D, result, majorIvs);
+      // 5.b. Otherwise, just go through the temporary `alloc`.
+      else
+        std_store(loaded1D, alloc, majorIvs);
+    }
+  });
+
+  assert((!options.unroll ^ result) && "Expected resulting Value iff unroll");
+  if (!result)
+    result = std_load(vector_type_cast(MemRefType::get({}, vectorType), alloc));
+  rewriter.replaceOp(op, result);
 
   return success();
 }
 
 template <>
 LogicalResult NDTransferOpHelper<TransferWriteOp>::doReplace() {
-  Value alloc = std_alloc(memRefMinorVectorType);
-
+  Value alloc;
+  if (!options.unroll) {
+    alloc = std_alloc(memRefMinorVectorType);
     std_store(xferOp.vector(),
               vector_type_cast(MemRefType::get({}, vectorType), alloc));
+  }
 
   emitLoops([&](ValueRange majorIvs, ValueRange leadingOffsets,
                 ValueRange majorOffsets, ValueRange minorOffsets,
                 MemRefBoundsCapture &memrefBounds) {
-    auto thenBlockBuilder = [&](ValueRange majorIvsPlusOffsets) {
+    // Lower to 1-D vector_transfer_write and let recursion handle it.
+    auto emitTransferWrite = [&](ValueRange majorIvsPlusOffsets) {
       SmallVector<Value, 8> indexing;
       indexing.reserve(leadingRank + majorRank + minorRank);
       indexing.append(leadingOffsets.begin(), leadingOffsets.end());
       indexing.append(majorIvsPlusOffsets.begin(), majorIvsPlusOffsets.end());
       indexing.append(minorOffsets.begin(), minorOffsets.end());
-      // Lower to 1-D vector_transfer_write and let recursion handle it.
-      Value loaded1D = std_load(alloc, majorIvs);
+      Value result;
+      // If `options.unroll` is true, extract the 1-D vector from the
+      // aggregate.
+      if (options.unroll)
+        result = vector_extract(xferOp.vector(), majorIvs);
+      else
+        result = std_load(alloc, majorIvs);
       auto map = TransferWriteOp::getTransferMinorIdentityMap(
           xferOp.getMemRefType(), minorVectorType);
       ArrayAttr masked;
       if (xferOp.isMaskedDim(xferOp.getVectorType().getRank() - 1)) {
         OpBuilder &b = ScopedContext::getBuilderRef();
         masked = b.getBoolArrayAttr({true});
       }
-      vector_transfer_write(loaded1D, xferOp.memref(), indexing,
+      vector_transfer_write(result, xferOp.memref(), indexing,
                             AffineMapAttr::get(map), masked);
     };
-    // Don't write anything when out of bounds.
-    auto elseBlockBuilder = [&](ValueRange majorIvsPlusOffsets) {};
-    emitInBounds(majorIvs, majorOffsets, memrefBounds, thenBlockBuilder,
-                 elseBlockBuilder);
+
+    // 1. Compute the inBoundsCondition in the current loops ivs + offset
+    // context.
+    SmallVector<Value, 4> majorIvsPlusOffsets;
+    Value inBoundsCondition = emitInBoundsCondition(
+        majorIvs, majorOffsets, memrefBounds, majorIvsPlusOffsets);
+
+    if (inBoundsCondition) {
+      // 2.a. If the condition is not null, we need an IfOp, to write
+      // conditionally. Progressively lower to a 1-D transfer write.
+      auto ifOp = ScopedContext::getBuilderRef().create<scf::IfOp>(
+          ScopedContext::getLocation(), TypeRange{}, inBoundsCondition,
+          /*withElseRegion=*/false);
+      BlockBuilder(&ifOp.thenRegion().front(),
+                   Append())([&] { emitTransferWrite(majorIvsPlusOffsets); });
+    } else {
+      // 2.b. Guaranteed in-bounds. Progressively lower to a 1-D transfer write.
+      emitTransferWrite(majorIvsPlusOffsets);
+    }
   });
 
   rewriter.eraseOp(op);
 
   return success();
 }
 
 } // namespace
if (loopIndex < 0) {
       clippedScalarAccessExprs[memRefDim] =
           std_select(i_plus_ii < zero, zero, select_1);
     }
   }
 
   return clippedScalarAccessExprs;
 }
 
-namespace {
-
-/// Implements lowering of TransferReadOp and TransferWriteOp to a
-/// proper abstraction for the hardware.
-///
-/// For now, we only emit a simple loop nest that performs clipped pointwise
-/// copies from a remote to a locally allocated memory.
-///
-/// Consider the case:
-///
-/// ```mlir
-///    // Read the slice `%A[%i0, %i1:%i1+256, %i2:%i2+32]` into
-///    // vector<32x256xf32> and pad with %f0 to handle the boundary case:
-///    %f0 = constant 0.0f : f32
-///    scf.for %i0 = 0 to %0 {
-///      scf.for %i1 = 0 to %1 step %c256 {
-///        scf.for %i2 = 0 to %2 step %c32 {
-///          %v = vector.transfer_read %A[%i0, %i1, %i2], %f0
-///               {permutation_map: (d0, d1, d2) -> (d2, d1)} :
-///               memref<?x?x?xf32>, vector<32x256xf32>
-///    }}}
-/// ```
-///
-/// The rewriters construct loop and indices that access MemRef A in a pattern
-/// resembling the following (while guaranteeing an always full-tile
-/// abstraction):
-///
-/// ```mlir
-///    scf.for %d2 = 0 to %c256 {
-///      scf.for %d1 = 0 to %c32 {
-///        %s = %A[%i0, %i1 + %d1, %i2 + %d2] : f32
-///        %tmp[%d2, %d1] = %s
-///      }
-///    }
-/// ```
-///
-/// In the current state, only a clipping transfer is implemented by `clip`,
-/// which creates individual indexing expressions of the form:
-///
-/// ```mlir-dsc
-///    auto condMax = i + ii < N;
-///    auto max = std_select(condMax, i + ii, N - one)
-///    auto cond = i + ii < zero;
-///    std_select(cond, zero, max);
-/// ```
-///
-/// In the future, clipping should not be the only way and instead we should
-/// load vectors + mask them. Similarly on the write side, load/mask/store for
-/// implementing RMW behavior.
-///
-/// Lowers TransferOp into a combination of:
-///   1. local memory allocation;
-///   2. perfect loop nest over:
-///      a. scalar load/stores from local buffers (viewed as a scalar memref);
-///      a. scalar store/load to original memref (with clipping).
-///   3. vector_load/store
-///   4. local memory deallocation.
-/// Minor variations occur depending on whether a TransferReadOp or
-/// a TransferWriteOp is rewritten.
 template <typename TransferOpTy>
-struct VectorTransferRewriter : public RewritePattern {
-  explicit VectorTransferRewriter(MLIRContext *context)
-      : RewritePattern(TransferOpTy::getOperationName(), 1, context) {}
+VectorTransferRewriter<TransferOpTy>::VectorTransferRewriter(
+    VectorTransferToSCFOptions options, MLIRContext *context)
+    : RewritePattern(TransferOpTy::getOperationName(), 1, context),
+      options(options) {}
 
-  /// Used for staging the transfer in a local scalar buffer.
-  MemRefType tmpMemRefType(TransferOpTy transfer) const {
-    auto vectorType = transfer.getVectorType();
-    return MemRefType::get(vectorType.getShape(), vectorType.getElementType(),
-                           {}, 0);
-  }
+/// Used for staging the transfer in a local buffer.
+template <typename TransferOpTy>
+MemRefType VectorTransferRewriter<TransferOpTy>::tmpMemRefType(
+    TransferOpTy transfer) const {
+  return MemRefType::get(vectorType.getShape(), vectorType.getElementType(), {},
+                         0);
 
-  /// Performs the rewrite.
-  LogicalResult matchAndRewrite(Operation *op,
-                                PatternRewriter &rewriter) const override;
-};
-
 /// Lowers TransferReadOp into a combination of:
 ///   1. local memory allocation;
 ///   2. perfect loop nest over:
 ///      a. scalar load from local buffers (viewed as a scalar memref);
 ///      a. scalar store to original memref (with clipping).
 ///   3. vector_load from local buffer (viewed as a memref<1 x vector>);
 ///   4. local memory deallocation.
 ///

 LogicalResult VectorTransferRewriter<TransferReadOp>::matchAndRewrite(
     Operation *op, PatternRewriter &rewriter) const {
   using namespace mlir::edsc::op;
 
   TransferReadOp transfer = cast<TransferReadOp>(op);
   if (AffineMap::isMinorIdentity(transfer.permutation_map())) {
     // If > 1D, emit a bunch of loops around 1-D vector transfers.
     if (transfer.getVectorType().getRank() > 1)
-      return NDTransferOpHelper<TransferReadOp>(rewriter, transfer).doReplace();
+      return NDTransferOpHelper<TransferReadOp>(rewriter, transfer, options)
+          .doReplace();
     // If 1-D this is now handled by the target-specific lowering.
     if (transfer.getVectorType().getRank() == 1)
       return failure();
   }
 
   // Conservative lowering to scalar load / stores.
   // 1. Setup all the captures.
   ScopedContext scope(rewriter, transfer.getLoc());

 LogicalResult VectorTransferRewriter<TransferWriteOp>::matchAndRewrite(
     Operation *op, PatternRewriter &rewriter) const {
   using namespace edsc::op;
 
   TransferWriteOp transfer = cast<TransferWriteOp>(op);
   if (AffineMap::isMinorIdentity(transfer.permutation_map())) {
     // If > 1D, emit a bunch of loops around 1-D vector transfers.
     if (transfer.getVectorType().getRank() > 1)
-      return NDTransferOpHelper<TransferWriteOp>(rewriter, transfer)
+      return NDTransferOpHelper<TransferWriteOp>(rewriter, transfer, options)
           .doReplace();
     // If 1-D this is now handled by the target-specific lowering.
     if (transfer.getVectorType().getRank() == 1)
       return failure();
   }
 
   // 1. Setup all the captures.
   ScopedContext scope(rewriter, transfer.getLoc());
loopNestBuilder(lbs, ubs, steps, [&](ValueRange loopIvs) {
     remote(clip(transfer, memRefBoundsCapture, ivs)) = local(ivs);
   });
   (std_dealloc(tmp)); // vexing parse...
 
   rewriter.eraseOp(op);
   return success();
 }
 
-} // namespace
-
 void mlir::populateVectorToSCFConversionPatterns(
-    OwningRewritePatternList &patterns, MLIRContext *context) {
+    OwningRewritePatternList &patterns, MLIRContext *context,
+    const VectorTransferToSCFOptions &options) {
   patterns.insert<VectorTransferRewriter<vector::TransferReadOp>,
-                  VectorTransferRewriter<vector::TransferWriteOp>>(context);
+                  VectorTransferRewriter<vector::TransferWriteOp>>(options,
+                                                                   context);
 }

mlir/lib/Dialect/Vector/VectorOps.cpp

void vector::ExtractOp::build(OpBuilder &builder, OperationState &result,
                               Value source, ArrayRef<int64_t> position) {
   result.addOperands(source);
   auto positionAttr = getVectorSubscriptAttr(builder, position);
   result.addTypes(inferExtractOpResultType(source.getType().cast<VectorType>(),
                                            positionAttr));
   result.addAttribute(getPositionAttrName(), positionAttr);
 }
 
+// Convenience builder which assumes the values are constant indices.
+void vector::ExtractOp::build(OpBuilder &builder, OperationState &result,
+                              Value source, ValueRange position) {
+  SmallVector<int64_t, 4> positionConstants =
+      llvm::to_vector<4>(llvm::map_range(position, [](Value pos) {
+        return pos.getDefiningOp<ConstantIndexOp>().getValue();
+      }));
+  build(builder, result, source, positionConstants);
+}
+
 static void print(OpAsmPrinter &p, vector::ExtractOp op) {
   p << op.getOperationName() << " " << op.vector() << op.position();
   p.printOptionalAttrDict(op.getAttrs(), {"position"});
   p << " : " << op.vector().getType();
 }
 
 static ParseResult parseExtractOp(OpAsmParser &parser, OperationState &result) {
   llvm::SMLoc attributeLoc, typeLoc;

 void InsertOp::build(OpBuilder &builder, OperationState &result, Value source,
                      Value dest, ArrayRef<int64_t> position) {
   result.addOperands({source, dest});
   auto positionAttr = getVectorSubscriptAttr(builder, position);
   result.addTypes(dest.getType());
   result.addAttribute(getPositionAttrName(), positionAttr);
 }
 
+// Convenience builder which assumes the values are constant indices.
+void InsertOp::build(OpBuilder &builder, OperationState &result, Value source,
+                     Value dest, ValueRange position) {
+  SmallVector<int64_t, 4> positionConstants =
+      llvm::to_vector<4>(llvm::map_range(position, [](Value pos) {
+        return pos.getDefiningOp<ConstantIndexOp>().getValue();
+      }));
+  build(builder, result, source, dest, positionConstants);
+}
+
 static LogicalResult verify(InsertOp op) {
   auto positionAttr = op.position().getValue();
   if (positionAttr.empty())
     return op.emitOpError("expected non-empty position attribute");
   auto destVectorType = op.getDestVectorType();
   if (positionAttr.size() > static_cast<unsigned>(destVectorType.getRank()))
     return op.emitOpError(
         "expected position attribute of rank smaller than dest vector rank");

mlir/lib/Dialect/Vector/VectorTransforms.cpp

 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 
 #define DEBUG_TYPE "vector-to-vector"
 
 using namespace mlir;
 using llvm::dbgs;
 
-/// Given a shape with sizes greater than 0 along all dimensions,
-/// returns the distance, in number of elements, between a slice in a dimension
-/// and the next slice in the same dimension.
-///   e.g. shape[3, 4, 5] -> linearization_basis[20, 5, 1]
-static SmallVector<int64_t, 8> computeStrides(ArrayRef<int64_t> shape) {
-  if (shape.empty())
-    return {};
-  SmallVector<int64_t, 8> tmp;
-  tmp.reserve(shape.size());
-  int64_t running = 1;
-  for (auto size : llvm::reverse(shape)) {
-    assert(size > 0 && "size must be nonnegative");
-    tmp.push_back(running);
-    running *= size;
-  }
-  return SmallVector<int64_t, 8>(tmp.rbegin(), tmp.rend());
-}
-
-static int64_t computeMaxLinearIndex(ArrayRef<int64_t> basis) {
-  if (basis.empty())
-    return 0;
-  int64_t res = 1;
-  for (auto b : basis)
-    res *= b;
-  return res;
-}
-
 // Clones `op` into a new operations that takes `operands` and returns
 // `resultTypes`.
 static Operation *cloneOpWithOperandsAndTypes(OpBuilder &builder, Location loc,
                                               Operation *op,
                                               ArrayRef<Value> operands,
                                               ArrayRef<Type> resultTypes) {
   OperationState res(loc, op->getName().getStringRef(), operands, resultTypes,
                      op->getAttrs());

mlir/lib/Dialect/Vector/VectorUtils.cpp

 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"
 #include "mlir/Dialect/Vector/VectorOps.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/IntegerSet.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/Support/LLVM.h"
 #include "mlir/Support/MathExtras.h"
+#include <numeric>
 
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SetVector.h"
 
 using llvm::SetVector;
 
 using namespace mlir;
 
+/// Return the number of elements of basis, `0` if empty.
+int64_t mlir::computeMaxLinearIndex(ArrayRef<int64_t> basis) {
+  if (basis.empty())
+    return 0;
+  return std::accumulate(basis.begin(), basis.end(), 1,
+                         std::multiplies<int64_t>());
+}
+
+/// Given a shape with sizes greater than 0 along all dimensions,
+/// return the distance, in number of elements, between a slice in a dimension
+/// and the next slice in the same dimension.
+///   e.g. shape[3, 4, 5] -> linearization_basis[20, 5, 1]
+SmallVector<int64_t, 8> mlir::computeStrides(ArrayRef<int64_t> shape) {
+  if (shape.empty())
+    return {};
+  SmallVector<int64_t, 8> tmp;
+  tmp.reserve(shape.size());
+  int64_t running = 1;
+  for (auto size : llvm::reverse(shape)) {
+    assert(size > 0 && "size must be nonnegative");
+    tmp.push_back(running);
+    running *= size;
+  }
+  return SmallVector<int64_t, 8>(tmp.rbegin(), tmp.rend());
+}
+
 SmallVector<int64_t, 4> mlir::computeStrides(ArrayRef<int64_t> shape,
                                              ArrayRef<int64_t> sizes) {
   int64_t rank = shape.size();
   // Compute the count for each dimension.
   SmallVector<int64_t, 4> sliceDimCounts(rank);
   for (int64_t r = 0; r < rank; ++r)
     sliceDimCounts[r] = ceilDiv(shape[r], sizes[r]);
   // Use that to compute the slice stride for each dimension.

mlir/test/Conversion/VectorToSCF/vector-to-loops.mlir

 // RUN: mlir-opt %s -test-convert-vector-to-scf -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -test-convert-vector-to-scf=full-unroll=true -split-input-file | FileCheck %s --check-prefix=FULL-UNROLL
 
 // CHECK-LABEL: func @materialize_read_1d() {
 func @materialize_read_1d() {
   %f0 = constant 0.0: f32
   %A = alloc () : memref<7x42xf32>
   affine.for %i0 = 0 to 7 step 4 {
     affine.for %i1 = 0 to 42 step 4 {
       %f1 = vector.transfer_read %A[%i0, %i1], %f0 {permutation_map = affine_map<(d0, d1) -> (d0)>} : memref<7x42xf32>, vector<4xf32>
func @materialize_write(%M: index, %N: index, %O: index, %P: index) {
   return
 }
 
 // -----
 
 // CHECK-DAG: #[[MAP0:.*]] = affine_map<(d0)[s0] -> (d0 + s0)>
 // CHECK-DAG: #[[MAP1:.*]] = affine_map<(d0, d1) -> (d1)>
 
+// FULL-UNROLL-DAG: #[[MAP0:.*]] = affine_map<(d0, d1) -> (d1)>
+// FULL-UNROLL-DAG: #[[MAP1:.*]] = affine_map<()[s0] -> (s0 + 1)>
+// FULL-UNROLL-DAG: #[[MAP2:.*]] = affine_map<()[s0] -> (s0 + 2)>
+
+
 // CHECK-LABEL: transfer_read_progressive(
 //  CHECK-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
 //  CHECK-SAME:   %[[base:[a-zA-Z0-9]+]]: index
-func @transfer_read_progressive(%A : memref<?x?xf32>, %base: index) -> vector<17x15xf32> {
+
+// FULL-UNROLL-LABEL: transfer_read_progressive(
+//  FULL-UNROLL-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
+//  FULL-UNROLL-SAME:   %[[base:[a-zA-Z0-9]+]]: index
+
+func @transfer_read_progressive(%A : memref<?x?xf32>, %base: index) -> vector<3x15xf32> {
   // CHECK: %[[cst:.*]] = constant 7.000000e+00 : f32
   %f7 = constant 7.0: f32
 
   // CHECK-DAG: %[[splat:.*]] = constant dense<7.000000e+00> : vector<15xf32>
-  // CHECK-DAG: %[[alloc:.*]] = alloc() : memref<17xvector<15xf32>>
+  // CHECK-DAG: %[[alloc:.*]] = alloc() : memref<3xvector<15xf32>>
   // CHECK-DAG: %[[dim:.*]] = dim %[[A]], 0 : memref<?x?xf32>
-  // CHECK: affine.for %[[I:.*]] = 0 to 17 {
+  // CHECK: affine.for %[[I:.*]] = 0 to 3 {
   // CHECK:   %[[add:.*]] = affine.apply #[[MAP0]](%[[I]])[%[[base]]]
   // CHECK:   %[[cond1:.*]] = cmpi "slt", %[[add]], %[[dim]] : index
   // CHECK:   scf.if %[[cond1]] {
   // CHECK:     %[[vec_1d:.*]] = vector.transfer_read %[[A]][%[[add]], %[[base]]], %[[cst]] : memref<?x?xf32>, vector<15xf32>
-  // CHECK:     store %[[vec_1d]], %[[alloc]][%[[I]]] : memref<17xvector<15xf32>>
+  // CHECK:     store %[[vec_1d]], %[[alloc]][%[[I]]] : memref<3xvector<15xf32>>
   // CHECK:   } else {
-  // CHECK:     store %[[splat]], %[[alloc]][%[[I]]] : memref<17xvector<15xf32>>
+  // CHECK:     store %[[splat]], %[[alloc]][%[[I]]] : memref<3xvector<15xf32>>
   // CHECK:   }
-  // CHECK: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<17xvector<15xf32>> to memref<vector<17x15xf32>>
-  // CHECK: %[[cst:.*]] = load %[[vmemref]][] : memref<vector<17x15xf32>>
-  %f = vector.transfer_read %A[%base, %base], %f7
-      {permutation_map = affine_map<(d0, d1) -> (d0, d1)>} :
-    memref<?x?xf32>, vector<17x15xf32>
+  // CHECK: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<3xvector<15xf32>> to memref<vector<3x15xf32>>
+  // CHECK: %[[cst:.*]] = load %[[vmemref]][] : memref<vector<3x15xf32>>
+
+  // FULL-UNROLL: %[[pad:.*]] = constant 7.000000e+00 : f32
+  // FULL-UNROLL: %[[VEC0:.*]] = constant dense<7.000000e+00> : vector<3x15xf32>
+  // FULL-UNROLL: %[[SPLAT:.*]] = constant dense<7.000000e+00> : vector<15xf32>
+  // FULL-UNROLL: %[[DIM:.*]] = dim %[[A]], 0 : memref<?x?xf32>
+  // FULL-UNROLL: cmpi "slt", %[[base]], %[[DIM]] : index
+  // FULL-UNROLL: %[[VEC1:.*]] = scf.if %{{.*}} -> (vector<3x15xf32>) {
+  // FULL-UNROLL:   vector.transfer_read %[[A]][%[[base]], %[[base]]], %[[pad]] : memref<?x?xf32>, vector<15xf32>
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC0]] [0] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: } else {
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC0]] [0] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: }
+  // FULL-UNROLL: affine.apply #[[MAP1]]()[%[[base]]]
+  // FULL-UNROLL: cmpi "slt", %{{.*}}, %[[DIM]] : index
+  // FULL-UNROLL: %[[VEC2:.*]] = scf.if %{{.*}} -> (vector<3x15xf32>) {
+  // FULL-UNROLL:   vector.transfer_read %[[A]][%{{.*}}, %[[base]]], %[[pad]] : memref<?x?xf32>, vector<15xf32>
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC1]] [1] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: } else {
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC1]] [1] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: }
+  // FULL-UNROLL: affine.apply #[[MAP2]]()[%[[base]]]
+  // FULL-UNROLL: cmpi "slt", %{{.*}}, %[[DIM]] : index
+  // FULL-UNROLL: %[[VEC3:.*]] = scf.if %{{.*}} -> (vector<3x15xf32>) {
+  // FULL-UNROLL:   vector.transfer_read %[[A]][%{{.*}}, %[[base]]], %[[pad]] : memref<?x?xf32>, vector<15xf32>
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC2]] [2] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: } else {
+  // FULL-UNROLL:   vector.insert %{{.*}}, %[[VEC2]] [2] : vector<15xf32> into vector<3x15xf32>
+  // FULL-UNROLL:   scf.yield %{{.*}} : vector<3x15xf32>
+  // FULL-UNROLL: }
+
+  %f = vector.transfer_read %A[%base, %base], %f7 :
+    memref<?x?xf32>, vector<3x15xf32>
 
-  return %f: vector<17x15xf32>
+  return %f: vector<3x15xf32>
 }
 
 // -----
 
 // CHECK-DAG: #[[MAP0:.*]] = affine_map<(d0)[s0] -> (d0 + s0)>
 // CHECK-DAG: #[[MAP1:.*]] = affine_map<(d0, d1) -> (d1)>
 
+// FULL-UNROLL-DAG: #[[MAP0:.*]] = affine_map<(d0, d1) -> (d1)>
+// FULL-UNROLL-DAG: #[[MAP1:.*]] = affine_map<()[s0] -> (s0 + 1)>
+// FULL-UNROLL-DAG: #[[MAP2:.*]] = affine_map<()[s0] -> (s0 + 2)>
+
 // CHECK-LABEL: transfer_write_progressive(
 //  CHECK-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
 //  CHECK-SAME:   %[[base:[a-zA-Z0-9]+]]: index,
-//  CHECK-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<17x15xf32>
-func @transfer_write_progressive(%A : memref<?x?xf32>, %base: index, %vec: vector<17x15xf32>) {
-  // CHECK: %[[alloc:.*]] = alloc() : memref<17xvector<15xf32>>
-  // CHECK: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<17xvector<15xf32>> to memref<vector<17x15xf32>>
-  // CHECK: store %[[vec]], %[[vmemref]][] : memref<vector<17x15xf32>>
+//  CHECK-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<3x15xf32>
+// FULL-UNROLL-LABEL: transfer_write_progressive(
+//  FULL-UNROLL-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
+//  FULL-UNROLL-SAME:   %[[base:[a-zA-Z0-9]+]]: index,
+//  FULL-UNROLL-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<3x15xf32>
+func @transfer_write_progressive(%A : memref<?x?xf32>, %base: index, %vec: vector<3x15xf32>) {
+  // CHECK: %[[alloc:.*]] = alloc() : memref<3xvector<15xf32>>
+  // CHECK: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<3xvector<15xf32>> to memref<vector<3x15xf32>>
+  // CHECK: store %[[vec]], %[[vmemref]][] : memref<vector<3x15xf32>>
   // CHECK: %[[dim:.*]] = dim %[[A]], 0 : memref<?x?xf32>
-  // CHECK: affine.for %[[I:.*]] = 0 to 17 {
+  // CHECK: affine.for %[[I:.*]] = 0 to 3 {
   // CHECK:   %[[add:.*]] = affine.apply #[[MAP0]](%[[I]])[%[[base]]]
   // CHECK:   %[[cmp:.*]] = cmpi "slt", %[[add]], %[[dim]] : index
   // CHECK:   scf.if %[[cmp]] {
-  // CHECK:     %[[vec_1d:.*]] = load %0[%[[I]]] : memref<17xvector<15xf32>>
+  // CHECK:     %[[vec_1d:.*]] = load %0[%[[I]]] : memref<3xvector<15xf32>>
   // CHECK:     vector.transfer_write %[[vec_1d]], %[[A]][%[[add]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
   // CHECK:   }
-  vector.transfer_write %vec, %A[%base, %base]
-      {permutation_map = affine_map<(d0, d1) -> (d0, d1)>} :
-    vector<17x15xf32>, memref<?x?xf32>
+
+  // FULL-UNROLL: %[[DIM:.*]] = dim %[[A]], 0 : memref<?x?xf32>
+  // FULL-UNROLL: %[[CMP0:.*]] = cmpi "slt", %[[base]], %[[DIM]] : index
+  // FULL-UNROLL: scf.if %[[CMP0]] {
+  // FULL-UNROLL:   %[[V0:.*]] = vector.extract %[[vec]][0] : vector<3x15xf32>
+  // FULL-UNROLL:   vector.transfer_write %[[V0]], %[[A]][%[[base]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+  // FULL-UNROLL: }
+  // FULL-UNROLL: %[[I1:.*]] = affine.apply #[[MAP1]]()[%[[base]]]
+  // FULL-UNROLL: %[[CMP1:.*]] = cmpi "slt", %[[I1]], %[[DIM]] : index
+  // FULL-UNROLL: scf.if %[[CMP1]] {
+  // FULL-UNROLL:   %[[V1:.*]] = vector.extract %[[vec]][1] : vector<3x15xf32>
+  // FULL-UNROLL:   vector.transfer_write %[[V1]], %[[A]][%[[I1]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+  // FULL-UNROLL: }
+  // FULL-UNROLL: %[[I2:.*]] = affine.apply #[[MAP2]]()[%[[base]]]
+  // FULL-UNROLL: %[[CMP2:.*]] = cmpi "slt", %[[I2]], %[[DIM]] : index
+  // FULL-UNROLL: scf.if %[[CMP2]] {
+  // FULL-UNROLL:   %[[V2:.*]] = vector.extract %[[vec]][2] : vector<3x15xf32>
+  // FULL-UNROLL:   vector.transfer_write %[[V2]], %[[A]][%[[I2]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+  // FULL-UNROLL: }
+
+  vector.transfer_write %vec, %A[%base, %base] :
+    vector<3x15xf32>, memref<?x?xf32>
   return
 }
 
 // -----
 
 // CHECK-DAG: #[[MAP0:.*]] = affine_map<(d0)[s0] -> (d0 + s0)>
 // CHECK-DAG: #[[MAP1:.*]] = affine_map<(d0, d1) -> (d1)>
 
+// FULL-UNROLL-DAG: #[[MAP0:.*]] = affine_map<(d0, d1) -> (d1)>
+// FULL-UNROLL-DAG: #[[MAP1:.*]] = affine_map<()[s0] -> (s0 + 1)>
+// FULL-UNROLL-DAG: #[[MAP2:.*]] = affine_map<()[s0] -> (s0 + 2)>
+
 // CHECK-LABEL: transfer_write_progressive_not_masked(
 //  CHECK-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
 //  CHECK-SAME:   %[[base:[a-zA-Z0-9]+]]: index,
-//  CHECK-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<17x15xf32>
-func @transfer_write_progressive_not_masked(%A : memref<?x?xf32>, %base: index, %vec: vector<17x15xf32>) {
+//  CHECK-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<3x15xf32>
+// FULL-UNROLL-LABEL: transfer_write_progressive_not_masked(
+//  FULL-UNROLL-SAME:   %[[A:[a-zA-Z0-9]+]]: memref<?x?xf32>,
+//  FULL-UNROLL-SAME:   %[[base:[a-zA-Z0-9]+]]: index,
+//  FULL-UNROLL-SAME:   %[[vec:[a-zA-Z0-9]+]]: vector<3x15xf32>
+func @transfer_write_progressive_not_masked(%A : memref<?x?xf32>, %base: index, %vec: vector<3x15xf32>) {
   // CHECK-NOT:    scf.if
-  // CHECK-NEXT: %[[alloc:.*]] = alloc() : memref<17xvector<15xf32>>
-  // CHECK-NEXT: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<17xvector<15xf32>> to memref<vector<17x15xf32>>
-  // CHECK-NEXT: store %[[vec]], %[[vmemref]][] : memref<vector<17x15xf32>>
-  // CHECK-NEXT: affine.for %[[I:.*]] = 0 to 17 {
+  // CHECK-NEXT: %[[alloc:.*]] = alloc() : memref<3xvector<15xf32>>
+  // CHECK-NEXT: %[[vmemref:.*]] = vector.type_cast %[[alloc]] : memref<3xvector<15xf32>> to memref<vector<3x15xf32>>
+  // CHECK-NEXT: store %[[vec]], %[[vmemref]][] : memref<vector<3x15xf32>>
+  // CHECK-NEXT: affine.for %[[I:.*]] = 0 to 3 {
   // CHECK-NEXT:   %[[add:.*]] = affine.apply #[[MAP0]](%[[I]])[%[[base]]]
-  // CHECK-NEXT:   %[[vec_1d:.*]] = load %0[%[[I]]] : memref<17xvector<15xf32>>
+  // CHECK-NEXT:   %[[vec_1d:.*]] = load %0[%[[I]]] : memref<3xvector<15xf32>>
   // CHECK-NEXT:   vector.transfer_write %[[vec_1d]], %[[A]][%[[add]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+
+  // FULL-UNROLL: %[[VEC0:.*]] = vector.extract %[[vec]][0] : vector<3x15xf32>
+  // FULL-UNROLL: vector.transfer_write %[[VEC0]], %[[A]][%[[base]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+  // FULL-UNROLL: %[[I1:.*]] = affine.apply #[[MAP1]]()[%[[base]]]
+  // FULL-UNROLL: %[[VEC1:.*]] = vector.extract %[[vec]][1] : vector<3x15xf32>
+  // FULL-UNROLL: vector.transfer_write %2, %[[A]][%[[I1]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
+  // FULL-UNROLL: %[[I2:.*]] = affine.apply #[[MAP2]]()[%[[base]]]
+  // FULL-UNROLL: %[[VEC2:.*]] = vector.extract %[[vec]][2] : vector<3x15xf32>
+  // FULL-UNROLL: vector.transfer_write %[[VEC2:.*]], %[[A]][%[[I2]], %[[base]]] : vector<15xf32>, memref<?x?xf32>
   vector.transfer_write %vec, %A[%base, %base] {masked = [false, false]} :
-    vector<17x15xf32>, memref<?x?xf32>
+    vector<3x15xf32>, memref<?x?xf32>
   return
 }

mlir/test/lib/Transforms/TestVectorToSCFConversion.cpp

 #include "mlir/Transforms/Passes.h"
 
 using namespace mlir;
 
 namespace {
 
 struct TestVectorToSCFPass
     : public PassWrapper<TestVectorToSCFPass, FunctionPass> {
+  TestVectorToSCFPass() = default;
+  TestVectorToSCFPass(const TestVectorToSCFPass &pass) {}
+
+  Option<bool> fullUnroll{
+      *this, "full-unroll",
+      llvm::cl::desc(
+          "Perform full unrolling when converting vector transfers to SCF"),
+      llvm::cl::init(false)};
+
   void runOnFunction() override {
     OwningRewritePatternList patterns;
     auto *context = &getContext();
-    populateVectorToSCFConversionPatterns(patterns, context);
+    populateVectorToSCFConversionPatterns(
+        patterns, context, VectorTransferToSCFOptions().setUnroll(fullUnroll));
     applyPatternsAndFoldGreedily(getFunction(), patterns);
   }
 };
 
 } // end anonymous namespace
 
 namespace mlir {
 void registerTestVectorToSCFPass() {
   PassRegistration<TestVectorToSCFPass> pass(
       "test-convert-vector-to-scf",
       "Converts vector transfer ops to loops over scalars and vector casts");
 }
 } // namespace mlir