[MLIR] Introduce memref vector cast op

[bondhugula]

Summary

Introduce memref_vector_cast op, which casts a memref of an elemental type
to a memref of a vector of that elemental type. This only supports
1-d vectorization of the shape and is meant to be used for vectorization
in the common case. Lowering to LLVM included.

[bondhugula]

Combine match and rewrite; update LLVM lowering style.

[bondhugula]

Fix inadvertent change.

[bondhugula]

Fix yet another inadvertent change.

[ftynse]

Was there an RFC for this op? How this connects to https://llvm.discourse.group/t/memref-cast/1514?

Can this be used as a replacement/generalization of https://mlir.llvm.org/docs/Dialects/Vector/#vectortype_cast-vectortypecastop ?

[ftynse]

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

2354

Type conversion may fail, please check the result for null and bail out on errors.

2356

MemRefDescriptor::undef(targetStructType)

2365

Please use MemRefDescriptor API to work with descriptors instead of spreading around the assumptions about the descriptor structure.

2381

I can't seem to find a verifier check that prevents this, so the assert may trigger on valid IR.

2384

This would have been one line using MemRefDescriptor API

[bondhugula]

In D85885#2215499, @ftynse wrote:

Was there an RFC for this op? How this connects to https://llvm.discourse.group/t/memref-cast/1514?

See later in the discussion you are referring to:
https://llvm.discourse.group/t/memref-cast/1514/10

There were multiple requests for having this op contributed - this can be in the future integrated into one of the memref cast ops.
https://llvm.discourse.group/t/understanding-the-vector-abstraction-in-mlir/989/16
https://llvm.discourse.group/t/memref-cast/1514/10

Can this be used as a replacement/generalization of https://mlir.llvm.org/docs/Dialects/Vector/#vectortype_cast-vectortypecastop ?

Yes, this op can subsume such a cast. To start with, I don't think that vector type cast should have ideally been introduced in the vector dialect; my guess is that it was temporary and for local experimentation before it could be subsumed or generalized by something in the standard dialect. Because that's really a cast on memref's - not vectors.

In any case, I expect this memref_vector_cast to potentially be merged into another cast (memref_reinterpret_cast for example) depending on how loaded we want each of the cast ops to be.

[bondhugula]

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

2381

Changed to return failure - thanks!

2384

Thanks - switched to the memref descriptor API.

[bondhugula]

Address review comments.

[bondhugula]

Address remaining review comments.

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

2356

Thanks.

[aartbik]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1894

or rank (also not changed)

1895

unless that last dimension is a ?

I would add some more explanation on what is allowed in those cases

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

2342

this is a pretty big block of memref related manipulations without any comments
please add some for readability of what is being done where

mlir/lib/Dialect/StandardOps/IR/Ops.cpp

1980

don't you want to test that the division is exact, i.e. no remainder?

[bondhugula]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1894

Thanks, updated. It's actually more than that: 'rank' as well as the extents along all but the last dimension remain unchanged the way it is.

1895

Even if the last dimension is dynamic, it's still supported. We divide the dynamic size by the vector width. So it works for both static and dynamic and the lowering already supports it. The second example below has it.

[bondhugula]

Update doc comment

[nicolasvasilache]

If it wants to subsume vector.type_cast, how does this interact with n-D vectors for n>1?

There are also implications related to data layout that I don't see how to handle in the general case.
This is the main reason vector.type_cast exists: it caters to a very specific use case.

This revision conveniently omits any discussion about layout which is critical.
How does this all work with non-"powers of 2" vectors?

[AlexEichenberger]

This functionality makes it very straightforward to develop SIMD code and replicate at a high level the ability to view memory as either scalar or SIMD code with very few restrictions.

[ftynse]

My implementation-level concerns have been addressed. Leaving the floor to others for conceptual discussions.

[bondhugula]

In D85885#2225686, @nicolasvasilache wrote:

If it wants to subsume vector.type_cast, how does this interact with n-D vectors for n>1?

More than 1-d has not really been thought out. I guess I just meant it could subsume what vector type_cast would do 1-d vectors.

There are also implications related to data layout that I don't see how to handle in the general case.

Yes, the implications with layout come in for the case of n > 1. This isn't really meant / designed for that.

This is the main reason vector.type_cast exists: it caters to a very specific use case.

This revision conveniently omits any discussion about layout which is critical.

Are you referring to non-identity layouts for memrefs? This isn't really meant to handle non-identity layouts. I'm not sure what you mean by "discussion" - this isn't a paper or a document to omit or include discussion. This is just implementing the functionality it describes that's really missing and that multiple folks requested for and need. I also see the adjective "conveniently omits" as making absolutely no sense in this context.

How does this all work with non-"powers of 2" vectors?

There is no restriction on the vector width as far as I can see. It's just a floordiv by vector width. Separately and unrelatedly, I'm yet to see hardware with non power of two vectors though. Why should something deal with non power of two vectors to be useful?

[aartbik]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1895

Yes, I realized that. My comment was indented to ask for proper documentation of that case, since the text and example did not fully agree (it is unclear if ? is at least vector sized for example while reading the text). Sorry if that was not clear.

[bondhugula]

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

1895

That's right, whether ? or constant, the last dimension is expected to be vector sized. Let me know if you want me to augment this doc in some other way.

mlir/include/mlir/Dialect/StandardOps/IR/Ops.td

let extraClassDeclaration = [{
     static bool areCastCompatible(Type a, Type b);
 
     /// The result of a memref_cast is always a memref.
     Type getType() { return getResult().getType(); }
   }];
 }
 
 //===----------------------------------------------------------------------===//
+// MemRefVectorCastOp
+//===----------------------------------------------------------------------===//
+
+def MemRefVectorCastOp : Std_Op<"memref_vector_cast", [NoSideEffect]> {
+  let summary = "memref elemental type vectorizing cast operation";
+  let description = [{
+    The `memref_vector_cast` operation converts a memref with a non-vector
+    element type to a memref of a vector element type while not changing
+    the former memref's base element type, rank, and its dimension sizes along
+    all but the last dimension. The last dimension size of the source dimension
+    is divided (floor division) by the vector size to obtain the corresponding
+    dimension size for the target memref type. The source memref type's last
+    dimension is expected to be at least vector sized. Examples:
+
+    ```
+    %MV = memref_vector_cast %M : memref<8x16xf32> to memref<8x2xvector<8xf32>>
+    %AV = memref_vector_cast %A : memref<?x?xf32> to memref<?x?xvector<8xf32>>
+    ```
+  }];
+
+  let arguments = (ins AnyMemRef:$source);
+  let results = (outs AnyMemRef);
+
+  let parser = [{
+    return impl::parseCastOp(parser, result);
+  }];
+  let printer = [{
+    return printStandardCastOp(this->getOperation(), p);
+  }];
+
+  let verifier = [{ return ::verifyCastOp(*this); }];
+
+  let extraClassDeclaration = [{
+    /// Returns true if `a` and `b` are valid operand and result pairs for
+    /// the operation.
+    static bool areCastCompatible(Type a, Type b);
+
+    /// The result of a memref_vector_cast is always a memref.
+    MemRefType getType() { return getResult().getType().cast<MemRefType>(); }
+  }];
+
+  let hasFolder = 1;
+}
+
+//===----------------------------------------------------------------------===//
 // MulFOp
 //===----------------------------------------------------------------------===//
 
 def MulFOp : FloatArithmeticOp<"mulf"> {
   let summary = "floating point multiplication operation";
   let description = [{
     Syntax:
 
def TensorLoadOp : Std_Op<"tensor_load",
   let builders = [OpBuilder<
     "OpBuilder &builder, OperationState &result, Value memref", [{
       result.addOperands(memref);
       result.addTypes(getTensorTypeFromMemRefType(memref.getType()));
   }]>];
 
   let extraClassDeclaration = [{
     /// The result of a tensor_load is always a tensor.
-    TensorType getType() { 
+    TensorType getType() {
       Type resultType = getResult().getType();
       if (resultType.isa<TensorType>())
         return resultType.cast<TensorType>();
       return {};
     }
   }];
 
   let assemblyFormat = "$memref attr-dict `:` type($memref)";

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

if (srcType.isa<MemRefType>() && dstType.isa<UnrankedMemRefType>()) {
       auto loadOp = rewriter.create<LLVM::LoadOp>(loc, castPtr);
       rewriter.replaceOp(op, loadOp.getResult());
     } else {
       llvm_unreachable("Unsupported unranked memref to unranked memref cast");
     }
   }
 };
 
+struct MemRefVectorCastOpLowering
+    : public ConvertOpToLLVMPattern<MemRefVectorCastOp> {
+  using ConvertOpToLLVMPattern<MemRefVectorCastOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto memRefShapeCastOp = cast<MemRefVectorCastOp>(op);
+    MemRefType sourceType =
+        memRefShapeCastOp.getOperand().getType().cast<MemRefType>();
+    MemRefType targetType = memRefShapeCastOp.getType();
+    if (!isSupportedMemRefType(targetType) ||
+        !isSupportedMemRefType(sourceType))
+      return failure();
+
+    MemRefVectorCastOp::Adaptor transformed(operands);
+    MemRefDescriptor srcMemRefDesc(transformed.source());
+
+    Type targetStructType =
+        typeConverter.convertType(memRefShapeCastOp.getType());
+    if (!targetStructType)
+      return failure();
+    Location loc = op->getLoc();
+    MemRefDescriptor memRefDescriptor =
+        MemRefDescriptor::undef(rewriter, loc, targetStructType);
+    LLVM::LLVMType targetElementPtrType = memRefDescriptor.getElementType();
+
+    Value srcBuffer = srcMemRefDesc.allocatedPtr(rewriter, loc);
+    Value targetBuffer = rewriter.create<LLVM::BitcastOp>(
+        loc, targetElementPtrType, ArrayRef<Value>(srcBuffer));
+    memRefDescriptor.setAllocatedPtr(rewriter, loc, targetBuffer);
+
+    Value srcBufferAligned = srcMemRefDesc.alignedPtr(rewriter, loc);
+    Value targetBufAligned = rewriter.create<LLVM::BitcastOp>(
+        loc, targetElementPtrType, ArrayRef<Value>(srcBufferAligned));
+    memRefDescriptor.setAlignedPtr(rewriter, loc, targetBufAligned);
+
+    int64_t offset;
+    SmallVector<int64_t, 4> strides;
+    if (failed(getStridesAndOffset(targetType, strides, offset)))
+      return failure();
+
+    // Unhandled dynamic offset.
+    if (offset == MemRefType::getDynamicStrideOrOffset())
+      return failure();
+
+    memRefDescriptor.setOffset(rewriter, loc,
+                               createIndexConstant(rewriter, loc, offset));
+
+    // Get the sizes of the memref: all but the last one are copied from the
+    // source memref. If the dimension size was static, the target memref would
+    // have the same size.
+    SmallVector<Value, 4> sizes;
+    sizes.reserve(targetType.getRank());
+    for (unsigned pos = 0, e = targetType.getRank() - 1; pos < e; ++pos) {
+      int64_t dimSize = targetType.getDimSize(pos);
+      if (dimSize == MemRefType::kDynamicSize)
+        sizes.push_back(srcMemRefDesc.size(rewriter, loc, pos));
+      else
+        sizes.push_back(createIndexConstant(rewriter, loc, dimSize));
+    }
+
+    if (targetType.getShape().back() != MemRefType::kDynamicSize) {
+      // The op is already verified to have the right size for the last
+      // dimension.
+      sizes.push_back(
+          createIndexConstant(rewriter, loc, targetType.getShape().back()));
+    } else {
+      // We need to divide the dynamic size on the source by the vector width.
+      Value vecWidth = createIndexConstant(
+          rewriter, loc,
+          targetType.getElementType().cast<ShapedType>().getNumElements());
+      sizes.push_back(rewriter.create<LLVM::UDivOp>(
+          loc, srcMemRefDesc.size(rewriter, loc, sourceType.getRank() - 1),
+          vecWidth));
+    }
+
+    assert(!sizes.empty() && "target memref rank can't be zero");
+
+    // Compute the total number of memref elements.
+    Value cumulativeSize = sizes.front();
+    for (unsigned i = 1, e = sizes.size(); i < e; ++i)
+      cumulativeSize = rewriter.create<LLVM::MulOp>(
+          loc, getIndexType(), ArrayRef<Value>{cumulativeSize, sizes[i]});
+
+    // Calculate the strides.
+    Value runningStride = nullptr;
+    // Iterate strides in reverse order, compute runningStride and strideValues.
+    unsigned nStrides = strides.size();
+    SmallVector<Value, 4> strideValues(nStrides, nullptr);
+    for (auto indexedStride : llvm::enumerate(llvm::reverse(strides))) {
+      int64_t index = nStrides - 1 - indexedStride.index();
+      if (strides[index] == MemRefType::getDynamicStrideOrOffset())
+        // Identity layout map is enforced in the match function, so we compute:
+        //   `runningStride *= sizes[index + 1]`.
+        runningStride = runningStride
+                            ? rewriter.create<LLVM::MulOp>(loc, runningStride,
+                                                           sizes[index + 1])
+                            : createIndexConstant(rewriter, loc, 1);
+      else
+        runningStride = createIndexConstant(rewriter, loc, strides[index]);
+      strideValues[index] = runningStride;
+    }
+
+    // Fill size and stride descriptors in memref.
+    for (auto indexedSize : llvm::enumerate(sizes)) {
+      int64_t index = indexedSize.index();
+      memRefDescriptor.setSize(rewriter, loc, index, indexedSize.value());
+      memRefDescriptor.setStride(rewriter, loc, index, strideValues[index]);
+    }
+
+    rewriter.replaceOp(op, {memRefDescriptor});
+    return success();
+  }
+};
+
 struct DialectCastOpLowering
     : public ConvertOpToLLVMPattern<LLVM::DialectCastOp> {
   using ConvertOpToLLVMPattern<LLVM::DialectCastOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
   matchAndRewrite(Operation *op, ArrayRef<Value> operands,
                   ConversionPatternRewriter &rewriter) const override {
     auto castOp = cast<LLVM::DialectCastOp>(op);
void mlir::populateStdToLLVMMemoryConversionPatterns(
     LLVMTypeConverter &converter, OwningRewritePatternList &patterns) {
   // clang-format off
   patterns.insert<
       AssumeAlignmentOpLowering,
       DeallocOpLowering,
       DimOpLowering,
       LoadOpLowering,
       MemRefCastOpLowering,
+      MemRefVectorCastOpLowering,
       RankOpLowering,
       StoreOpLowering,
       SubViewOpLowering,
       ViewOpLowering,
       AllocOpLowering>(converter);
   // clang-format on
 }
 

mlir/lib/Dialect/StandardOps/IR/Ops.cpp

bool MemRefCastOp::areCastCompatible(Type a, Type b) {
   return false;
 }
 
 OpFoldResult MemRefCastOp::fold(ArrayRef<Attribute> operands) {
   return impl::foldCastOp(*this);
 }
 
 //===----------------------------------------------------------------------===//
+// MemRefVectorCastOp
+//===----------------------------------------------------------------------===//
+
+bool MemRefVectorCastOp::areCastCompatible(Type a, Type b) {
+  auto aT = a.dyn_cast<MemRefType>();
+  auto bT = b.dyn_cast<MemRefType>();
+
+  if (!aT || !bT)
+    return false;
+
+  if (aT.getAffineMaps() != bT.getAffineMaps())
+    return false;
+
+  if (aT.getMemorySpace() != bT.getMemorySpace())
+    return false;
+
+  // With rank 0, there is no vec cast.
+  if (aT.getRank() == 0)
+    return false;
+
+  if (aT.getRank() != bT.getRank())
+    return false;
+
+  // Should have the same shape up until the last n-1 dimensions.
+  if (!std::equal(aT.getShape().begin(), std::prev(aT.getShape().end()),
+                  bT.getShape().begin()))
+    return false;
+
+  // The source memref can't have a vector elemental type.
+  if (auto shapedEltType = aT.getElementType().dyn_cast<ShapedType>())
+    return false;
+
+  // The destination memref elt type has be a vector type.
+  auto vectorEltTypeB = bT.getElementType().dyn_cast<VectorType>();
+  if (!vectorEltTypeB)
+    return false;
+
+  auto eltA = aT.getElementType();
+  auto eltB = vectorEltTypeB.getElementType();
+  if (eltA != eltB)
+    return false;
+
+  int64_t lastDimA = aT.getShape().back();
+  int64_t lastDimB = bT.getShape().back();
+
+  // If one of them is dynamic but not the other, they are incompatible.
+  if (lastDimA * lastDimB < 0)
+    return false;
+
+  // The last dim of the target should be of the right size.
+  if (lastDimB != MemRefType::kDynamicSize &&
+      lastDimA / vectorEltTypeB.getNumElements() != lastDimB)
+    return false;
+
+  return true;
+}
+
+OpFoldResult MemRefVectorCastOp::fold(ArrayRef<Attribute> operands) {
+  return impl::foldCastOp(*this);
+}
+
+//===----------------------------------------------------------------------===//
 // MulFOp
 //===----------------------------------------------------------------------===//
 
 OpFoldResult MulFOp::fold(ArrayRef<Attribute> operands) {
   return constFoldBinaryOp<FloatAttr>(
       operands, [](APFloat a, APFloat b) { return a * b; });
 }
 

mlir/test/Conversion/StandardToLLVM/convert-dynamic-memref-ops.mlir

 // CHECK-LABEL: func @memref_cast_unranked_to_ranked
 func @memref_cast_unranked_to_ranked(%arg : memref<*xf32>) {
 //      CHECK: %[[p:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(i64, ptr<i8>)>
 // CHECK-NEXT: llvm.bitcast %[[p]] : !llvm.ptr<i8> to !llvm.ptr<struct<(ptr<float>, ptr<float>, i64, array<4 x i64>, array<4 x i64>)>>
   %0 = memref_cast %arg : memref<*xf32> to memref<?x?x10x2xf32>
   return
 }
 
+// CHECK-LABEL: func @memref_vector_cast_dynamic
+func @memref_vector_cast_dynamic(%M : memref<?x?xf32>) -> memref<?x?xvector<16xf32>> {
+// CHECK:        %[[SRC:.*]] = llvm.insertvalue %{{.*}}, %{{.*}}[4, 1] : !llvm.struct<(ptr<float>, ptr<float>,
+// CHECK-NEXT:   %[[U:.*]] = llvm.mlir.undef : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.extractvalue %{{.*}}[0] : !llvm.struct<(ptr<float>, ptr<float>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.bitcast %{{.*}} : !llvm.ptr<float> to !llvm.ptr<vec<16 x float>>
+// CHECK-NEXT:   llvm.insertvalue %{{.*}}, %{{.*}}[0] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.extractvalue %{{.*}}[1] : !llvm.struct<(ptr<float>, ptr<float>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.bitcast %{{.*}} : !llvm.ptr<float> to !llvm.ptr<vec<16 x float>>
+// CHECK-NEXT:   llvm.insertvalue %{{.*}}, %{{.*}}[1] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   %[[OFFSET:.*]] = llvm.mlir.constant(0 : index) : !llvm.i64
+// CHECK-NEXT:   llvm.insertvalue %[[OFFSET]], %{{.*}}[2] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   %[[size1:.*]] = llvm.extractvalue %[[SRC]][3, 0]
+// CHECK-NEXT:   %[[VEC_WIDTH:.*]] = llvm.mlir.constant(16 : index)
+// CHECK-NEXT:   %[[src_size2:.*]] = llvm.extractvalue %[[SRC]][3, 1]
+// CHECK-NEXT:   %[[size2:.*]] = llvm.udiv %[[src_size2]], %[[VEC_WIDTH]]
+// CHECK-NEXT:   %[[cum_size:.*]] = llvm.mul %[[size1]], %[[size2]]
+// CHECK-NEXT:   %[[st2:.*]] = llvm.mlir.constant(1 : index)
+// CHECK-NEXT:   %[[st1:.*]] = llvm.mul %[[st2]], %[[size2]]
+// CHECK-NEXT:   llvm.insertvalue %[[size1]], %{{.*}}[3, 0] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[st1]], %{{.*}}[4, 0] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[size2]], %{{.*}}[3, 1] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[st2]], %{{.*}}[4, 1] : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.return %{{.*}} : !llvm.struct<(ptr<vec<16 x float>>, ptr<vec<16 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+  %MV = memref_vector_cast %M : memref<?x?xf32> to memref<?x?xvector<16xf32>>
+  return %MV : memref<?x?xvector<16xf32>>
+}
+
 // CHECK-LABEL: func @mixed_memref_dim
 func @mixed_memref_dim(%mixed : memref<42x?x?x13x?xf32>) {
 // CHECK: llvm.mlir.constant(42 : index) : !llvm.i64
   %c0 = constant 0 : index
   %0 = dim %mixed, %c0 : memref<42x?x?x13x?xf32>
 // CHECK: llvm.extractvalue %[[ld:.*]][3, 1] : !llvm.struct<(ptr<float>, ptr<float>, i64, array<5 x i64>, array<5 x i64>)>
   %c1 = constant 1 : index
   %1 = dim %mixed, %c1 : memref<42x?x?x13x?xf32>

mlir/test/Conversion/StandardToLLVM/convert-static-memref-ops.mlir

 // BAREPTR-NEXT: %[[bc:.*]] = llvm.bitcast %[[ptr]] : !llvm.ptr<float> to !llvm.ptr<i8>
 // BAREPTR-NEXT: llvm.call @free(%[[bc]]) : (!llvm.ptr<i8>) -> ()
   dealloc %static : memref<10x8xf32>
   return
 }
 
 // -----
 
+// CHECK-LABEL: func @memref_vector_cast
+func @memref_vector_cast(%M : memref<42x16xf32>) -> memref<42x4xvector<4xf32>> {
+// CHECK:        %[[SRC:.*]] = llvm.insertvalue %{{.*}}, %{{.*}}[4, 1]
+// CHECK-NEXT:   llvm.mlir.undef : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.extractvalue %{{.*}}[0] : !llvm.struct<(ptr<float>, ptr<float>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.bitcast %{{.*}} : !llvm.ptr<float> to !llvm.ptr<vec<4 x float>>
+// CHECK-NEXT:   llvm.insertvalue %{{.*}}, %{{.*}}[0] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.extractvalue %{{.*}}[1] : !llvm.struct<(ptr<float>, ptr<float>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.bitcast %{{.*}} : !llvm.ptr<float> to !llvm.ptr<vec<4 x float>>
+// CHECK-NEXT:   llvm.insertvalue %{{.*}}, %{{.*}}[1] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   %{{.*}} = llvm.mlir.constant(0 : index) : !llvm.i64
+// CHECK-NEXT:   %{{.*}} = llvm.insertvalue %{{.*}}, %{{.*}}[2] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   %[[size1:.*]] = llvm.mlir.constant(42 : index) : !llvm.i64
+// CHECK-NEXT:   %[[size2:.*]] = llvm.mlir.constant(4 : index) : !llvm.i64
+// CHECK-NEXT:   %[[cumul_size:.*]] = llvm.mul %[[size1]], %[[size2]]
+// CHECK-NEXT:   %[[st2:.*]] = llvm.mlir.constant(1 : index)
+// CHECK-NEXT:   %[[st1:.*]] = llvm.mlir.constant(4 : index)
+// CHECK-NEXT:   llvm.insertvalue %[[size1]], %{{.*}}[3, 0] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[st1]], %{{.*}}[4, 0] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[size2]], %{{.*}}[3, 1] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.insertvalue %[[st2]], %{{.*}}[4, 1] : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+// CHECK-NEXT:   llvm.return %{{.*}} : !llvm.struct<(ptr<vec<4 x float>>, ptr<vec<4 x float>>, i64, array<2 x i64>, array<2 x i64>)>
+  %MV = memref_vector_cast %M : memref<42x16xf32> to memref<42x4xvector<4xf32>>
+  return %MV : memref<42x4xvector<4xf32>>
+}
+
+// -----
+
 // CHECK-LABEL: func @zero_d_load
 // BAREPTR-LABEL: func @zero_d_load(%{{.*}}: !llvm.ptr<float>) -> !llvm.float
 func @zero_d_load(%arg0: memref<f32>) -> f32 {
 //      CHECK:  %[[ptr:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(ptr<float>, ptr<float>, i64)>
 // CHECK-NEXT:  %[[c0:.*]] = llvm.mlir.constant(0 : index) : !llvm.i64
 // CHECK-NEXT:  %[[addr:.*]] = llvm.getelementptr %[[ptr]][%[[c0]]] : (!llvm.ptr<float>, !llvm.i64) -> !llvm.ptr<float>
 // CHECK-NEXT:  %{{.*}} = llvm.load %[[addr]] : !llvm.ptr<float>
 

mlir/test/IR/core-ops.mlir

func @memref_cast(%arg0: memref<4xf32>, %arg1 : memref<?xf32>, %arg2 : memref<64x16x4xf32, offset: 0, strides: [64, 4, 1]>) {
   // CHECK: memref_cast %{{.*}} : memref<4xf32> to memref<*xf32>
   %4 = memref_cast %1 : memref<4xf32> to memref<*xf32>
 
   // CHECK: memref_cast %{{.*}} : memref<*xf32> to memref<4xf32>
   %5 = memref_cast %4 : memref<*xf32> to memref<4xf32>
   return
 }
 
+// CHECK-LABEL: func @memref_vector_cast
+func @memref_vector_cast(%arg0: memref<60x8xf32>, %arg1: memref<?x?xf32>) -> memref<60x2xvector<4xf32>> {
+  // CHECK: {{.*}} = memref_vector_cast %arg0 : memref<60x8xf32> to memref<60x2xvector<4xf32>>
+  %0 = memref_vector_cast %arg0 : memref<60x8xf32> to memref<60x2xvector<4xf32>>
+  // CHECK: memref_vector_cast %arg1 : memref<?x?xf32> to memref<?x?xvector<4xf32>>
+  %1 = memref_vector_cast %arg1 : memref<?x?xf32> to memref<?x?xvector<4xf32>>
+  return %0 : memref<60x2xvector<4xf32>>
+}
+
 // Check that unranked memrefs with non-default memory space roundtrip
 // properly.
 // CHECK-LABEL: @unranked_memref_roundtrip(memref<*xf32, 4>)
 func @unranked_memref_roundtrip(memref<*xf32, 4>)
 
 // CHECK-LABEL: func @memref_view(%arg0
 func @memref_view(%arg0 : index, %arg1 : index, %arg2 : index) {
   %0 = alloc() : memref<2048xi8>

mlir/test/IR/invalid-ops.mlir

func @invalid_memref_cast() {
   %1 = memref_cast %0 : memref<2x5xf32, 0> to memref<*xf32, 0>
   // expected-error@+1 {{operand type 'memref<*xf32>' and result type 'memref<*xf32>' are cast incompatible}}
   %2 = memref_cast %1 : memref<*xf32, 0> to memref<*xf32, 0>
   return
 }
 
 // -----
 
+func @invalid_memref_vector_cast_1() {
+  %0 = alloc() : memref<2x4xf32>
+  %1 = memref_vector_cast %0 : memref<2x4xf32> to memref<2x2xf64>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<2x2xf64>' are cast incompatible}}
+}
+
+// -----
+
+func @invalid_memref_vector_cast_2() {
+  %0 = alloc() : memref<2x4xf32>
+  memref_vector_cast %0 : memref<2x4xf32> to memref<2xvector<4xf32>>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<2xvector<4xf32>>' are cast incompatible}}
+  return
+}
+
+// -----
+
+func @invalid_memref_vector_cast_3() {
+  %0 = alloc() : memref<2x4xf32>
+  memref_vector_cast %0 : memref<2x4xf32> to memref<2x?xvector<4xf32>>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<2x?xvector<4xf32>>' are cast incompatible}}
+  return
+}
+
+// -----
+
+func @invalid_memref_vector_cast_4() {
+  %0 = alloc() : memref<2x4xf32>
+  memref_vector_cast %0 : memref<2x4xf32> to memref<?x1xvector<4xf32>>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<?x1xvector<4xf32>>' are cast incompatible}}
+  return
+}
+
+// -----
+
+func @invalid_memref_vector_cast_5() {
+  %0 = alloc() : memref<2x4xf32>
+  %1 = memref_vector_cast %0 : memref<2x4xf32> to memref<4x2xf32>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<4x2xf32>' are cast incompatible}}
+}
+
+// -----
+
+func @invalid_memref_vector_cast_6() {
+  %0 = alloc() : memref<2x4xf32>
+  %1 = memref_vector_cast %0 : memref<2x4xf32> to memref<2x4xf32>
+  // expected-error@-1 {{operand type 'memref<2x4xf32>' and result type 'memref<2x4xf32>' are cast incompatible}}
+}
+
+// -----
+
 func @atomic_rmw_idxs_rank_mismatch(%I: memref<16x10xf32>, %i : index, %val : f32) {
   // expected-error@+1 {{expects the number of subscripts to be equal to memref rank}}
   %x = atomic_rmw "addf" %val, %I[%i] : (f32, memref<16x10xf32>) -> f32
   return
 }
 
 // -----