[mlir] [VectorOps] Canonicalization of 1-D memory operations

[aartbik]

Summary

Masked loading/storing in various forms can be optimized
into simpler memory operations when the mask is all true
or all false. Note that the backend does similar optimizations
but doing this early may expose more opportunities for further
optimizations. This further prepares progressively lowering
transfer read and write into 1-D memory operations.

[bondhugula]

Documentation and comments are missing throughout.

mlir/lib/Dialect/Vector/VectorOps.cpp

41

Missing doc comment.

46

Avoid auto here.

60

Avoid auto.

63

Comment for this block.

72

Doc comment doesn't describe the method. Output ref parameters not documented. Triple doc comment.

[aartbik]

mlir/lib/Dialect/Vector/VectorOps.cpp

46

What is your reasoning on when auto is okay and when not?

[bondhugula]

mlir/lib/Dialect/Vector/VectorOps.cpp

46

The reasoning is not mine, but what's being followed in other revisions I've seen and repeated by @mehdi_amini and @rriddle as well multiple times is to avoid using auto where it does NOT improve readability. The usage here is hurting readability however obvious it might appear to you now.

@rriddle @mehdi_amini - shouldn't we be documenting this guideline in the style guidelines on mlir.llvm.org?

[aartbik]

addressed comments

[aartbik]

mlir/lib/Dialect/Vector/VectorOps.cpp

46

Just to make sure, I was not disagreeing with your feedback (I usually follow the advice of my Google mentor from long time back that the reviewer is almost always right :-), I was genuinely interested in some form of guideline on this.

[aartbik]

Oh, and I would be very interested in seeing your work on memref casting being upstreamed, just to unify some of the vector specific ops we have been using so far!

[bondhugula]

In D85769#2211420, @aartbik wrote:

Oh, and I would be very interested in seeing your work on memref casting being upstreamed, just to unify some of the vector specific ops we have been using so far!

Sure - I'm submitting that revision for review today.

[bondhugula]

mlir/lib/Dialect/Vector/VectorOps.cpp

46

No worries. :) We should definitely have a guideline on the "coding style" page since this has come up often. I'll bring this up on discord.

[bondhugula]

I'll defer to the original reviewers on the list for a real review!

[aartbik]

Nicolas is not going to look at this soon, so can I ask any of the other reviewers to please have a look?

[rriddle]

mlir/lib/Dialect/Vector/VectorOps.cpp

46

Please put functions at the top level, marked with static, and not within a namespace.

re: auto (and the comment above), these are in the LLVM coding standards:
https://llvm.org/docs/CodingStandards.html
https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable

53

Are these all i1? If so, use denseElts.getValues<bool>().

[ThomasRaoux]

Looks good to me

mlir/lib/Dialect/Vector/VectorOps.cpp

84

nit: the function name suggests that it doesn't have side-effects even though it creates the cast if possible. Maybe there is a better name?

[aartbik]

addressed comments

[aartbik]

Thanks River and Thomas. I removed the namespace, used static, and renamed to "castedToMemRef" to make the side effect a bit more clear.....

mlir/lib/Dialect/Vector/VectorOps.cpp

46

Marked with static. Removed namespace.

84

ok, rephrased slightly to make that more clear.

mlir/include/mlir/Dialect/Vector/VectorOps.td

VectorType getPassThruVectorType() {
       return pass_thru().getType().cast<VectorType>();
     }
     VectorType getResultVectorType() {
       return result().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "$base `,` $mask `,` $pass_thru attr-dict `:` "
     "type($base) `,` type($mask) `,` type($pass_thru) `into` type($result)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_MaskedStoreOp :
   Vector_Op<"maskedstore">,
     Arguments<(ins AnyMemRef:$base,
                VectorOfRankAndType<[1], [I1]>:$mask,
                VectorOfRank<[1]>:$value)> {
 
VectorType getMaskVectorType() {
       return mask().getType().cast<VectorType>();
     }
     VectorType getValueVectorType() {
       return value().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "$base `,` $mask `,` $value attr-dict `:` "
     "type($mask) `,` type($value) `into` type($base)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_GatherOp :
   Vector_Op<"gather">,
     Arguments<(ins AnyMemRef:$base,
                VectorOfRankAndType<[1], [AnyInteger]>:$indices,
                VectorOfRankAndType<[1], [I1]>:$mask,
                Variadic<VectorOfRank<[1]>>:$pass_thru)>,
VectorType getPassThruVectorType() {
         ? VectorType()
         : (*pass_thru().begin()).getType().cast<VectorType>();
     }
     VectorType getResultVectorType() {
       return result().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_ScatterOp :
   Vector_Op<"scatter">,
     Arguments<(ins AnyMemRef:$base,
                VectorOfRankAndType<[1], [AnyInteger]>:$indices,
                VectorOfRankAndType<[1], [I1]>:$mask,
                VectorOfRank<[1]>:$value)> {
VectorType getMaskVectorType() {
       return mask().getType().cast<VectorType>();
     }
     VectorType getValueVectorType() {
       return value().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "$base `,` $indices `,` $mask `,` $value attr-dict `:` "
     "type($indices) `,` type($mask) `,` type($value) `into` type($base)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_ExpandLoadOp :
   Vector_Op<"expandload">,
     Arguments<(ins AnyMemRef:$base,
                VectorOfRankAndType<[1], [I1]>:$mask,
                VectorOfRank<[1]>:$pass_thru)>,
     Results<(outs VectorOfRank<[1]>:$result)> {
VectorType getPassThruVectorType() {
       return pass_thru().getType().cast<VectorType>();
     }
     VectorType getResultVectorType() {
       return result().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "$base `,` $mask `,` $pass_thru attr-dict `:` "
     "type($base) `,` type($mask) `,` type($pass_thru) `into` type($result)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_CompressStoreOp :
   Vector_Op<"compressstore">,
     Arguments<(ins AnyMemRef:$base,
                VectorOfRankAndType<[1], [I1]>:$mask,
                VectorOfRank<[1]>:$value)> {
 
VectorType getMaskVectorType() {
       return mask().getType().cast<VectorType>();
     }
     VectorType getValueVectorType() {
       return value().getType().cast<VectorType>();
     }
   }];
   let assemblyFormat = "$base `,` $mask `,` $value attr-dict `:` "
     "type($base) `,` type($mask) `,` type($value)";
+  let hasCanonicalizer = 1;
 }
 
 def Vector_ShapeCastOp :
   Vector_Op<"shape_cast", [NoSideEffect]>,
     Arguments<(ins AnyTypeOf<[AnyVector, TupleOf<[AnyVector]>]>:$source)>,
     Results<(outs AnyTypeOf<[AnyVector, TupleOf<[AnyVector]>]>:$result)> {
   let summary = "shape_cast casts between vector shapes";
   let description = [{

mlir/lib/Dialect/Vector/VectorOps.cpp

 #include "mlir/Support/LLVM.h"
 #include "mlir/Support/MathExtras.h"
 #include "llvm/ADT/StringSet.h"
 #include <numeric>
 
 using namespace mlir;
 using namespace mlir::vector;
 
+/// Helper enum to classify mask value.
+enum class MaskFormat {
+  AllTrue = 0,
+  AllFalse = 1,
+  Unknown = 2,
+};
+
+/// Helper method to classify a 1-D mask value. Currently, the method
+/// looks "under the hood" of a constant value with dense attributes
+/// and a constant mask operation (since the client may be called at
+/// various stages during progressive lowering).
+static MaskFormat get1DMaskFormat(Value mask) {
+  if (auto c = mask.getDefiningOp<ConstantOp>()) {
+    // Inspect constant dense values. We count up for bits that
+    // are set, count down for bits that are cleared, and bail
+    // when a mix is detected.
+    if (auto denseElts = c.value().dyn_cast<DenseIntElementsAttr>()) {
+      int64_t val = 0;
+      for (llvm::APInt b : denseElts)
+        if (b.getBoolValue() && val >= 0)
+          val++;
+        else if (!b.getBoolValue() && val <= 0)
+          val--;
+        else
+          return MaskFormat::Unknown;
+      if (val > 0)
+        return MaskFormat::AllTrue;
+      if (val < 0)
+        return MaskFormat::AllFalse;
+    }
+  } else if (auto m = mask.getDefiningOp<ConstantMaskOp>()) {
+    // Inspect constant mask index. If the index exceeds the
+    // dimension size, all bits are set. If the index is zero
+    // or less, no bits are set.
+    ArrayAttr masks = m.mask_dim_sizes();
+    assert(masks.size() == 1);
+    int64_t i = masks[0].cast<IntegerAttr>().getInt();
+    int64_t u = m.getType().cast<VectorType>().getDimSize(0);
+    if (i >= u)
+      return MaskFormat::AllTrue;
+    if (i <= 0)
+      return MaskFormat::AllFalse;
+  }
+  return MaskFormat::Unknown;
+}
+
+/// Helper method to cast a 1-D memref<10xf32> "base" into a
+/// memref<vector<10xf32>> in the output parameter "newBase",
+/// using the 'element' vector type "vt". Returns true on success.
+static bool castedToMemRef(Location loc, Value base, MemRefType mt,
+                           VectorType vt, PatternRewriter &rewriter,
+                           Value &newBase) {
+  // The vector.type_cast operation does not accept unknown memref<?xf32>.
+  // TODO: generalize the cast and accept this case too
+  if (!mt.hasStaticShape())
+    return false;
+  newBase = rewriter.create<TypeCastOp>(loc, MemRefType::get({}, vt), base);
+  return true;
+}
+
 //===----------------------------------------------------------------------===//
 // VectorDialect
 //===----------------------------------------------------------------------===//
 
 void VectorDialect::initialize() {
   addOperations<
 #define GET_OP_LIST
 #include "mlir/Dialect/Vector/VectorOps.cpp.inc"
static LogicalResult verify(MaskedLoadOp op) {
 
   if (resVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected result dim to match mask dim");
   if (resVType != passVType)
     return op.emitOpError("expected pass_thru of same type as result type");
   return success();
 }
 
+namespace {
+class MaskedLoadFolder final : public OpRewritePattern<MaskedLoadOp> {
+public:
+  using OpRewritePattern<MaskedLoadOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(MaskedLoadOp load,
+                                PatternRewriter &rewriter) const override {
+    Value newBase;
+    switch (get1DMaskFormat(load.mask())) {
+    case MaskFormat::AllTrue:
+      if (!castedToMemRef(load.getLoc(), load.base(), load.getMemRefType(),
+                          load.getResultVectorType(), rewriter, newBase))
+        return failure();
+      rewriter.replaceOpWithNewOp<LoadOp>(load, newBase);
+      return success();
+    case MaskFormat::AllFalse:
+      rewriter.replaceOp(load, load.pass_thru());
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void MaskedLoadOp::getCanonicalizationPatterns(
+    OwningRewritePatternList &results, MLIRContext *context) {
+  results.insert<MaskedLoadFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // MaskedStoreOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(MaskedStoreOp op) {
   VectorType maskVType = op.getMaskVectorType();
   VectorType valueVType = op.getValueVectorType();
 
   if (valueVType.getElementType() != op.getMemRefType().getElementType())
     return op.emitOpError("base and value element type should match");
 
   if (valueVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected value dim to match mask dim");
   return success();
 }
 
+namespace {
+class MaskedStoreFolder final : public OpRewritePattern<MaskedStoreOp> {
+public:
+  using OpRewritePattern<MaskedStoreOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(MaskedStoreOp store,
+                                PatternRewriter &rewriter) const override {
+    Value newBase;
+    switch (get1DMaskFormat(store.mask())) {
+    case MaskFormat::AllTrue:
+      if (!castedToMemRef(store.getLoc(), store.base(), store.getMemRefType(),
+                          store.getValueVectorType(), rewriter, newBase))
+        return failure();
+      rewriter.replaceOpWithNewOp<StoreOp>(store, store.value(), newBase);
+      return success();
+    case MaskFormat::AllFalse:
+      rewriter.eraseOp(store);
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void MaskedStoreOp::getCanonicalizationPatterns(
+    OwningRewritePatternList &results, MLIRContext *context) {
+  results.insert<MaskedStoreFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // GatherOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(GatherOp op) {
   VectorType indicesVType = op.getIndicesVectorType();
   VectorType maskVType = op.getMaskVectorType();
   VectorType resVType = op.getResultVectorType();
 
   if (resVType.getElementType() != op.getMemRefType().getElementType())
     return op.emitOpError("base and result element type should match");
 
   if (resVType.getDimSize(0) != indicesVType.getDimSize(0))
     return op.emitOpError("expected result dim to match indices dim");
   if (resVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected result dim to match mask dim");
   if (llvm::size(op.pass_thru()) != 0) {
     VectorType passVType = op.getPassThruVectorType();
     if (resVType != passVType)
       return op.emitOpError("expected pass_thru of same type as result type");
   }
   return success();
 }
 
+namespace {
+class GatherFolder final : public OpRewritePattern<GatherOp> {
+public:
+  using OpRewritePattern<GatherOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(GatherOp gather,
+                                PatternRewriter &rewriter) const override {
+    switch (get1DMaskFormat(gather.mask())) {
+    case MaskFormat::AllTrue:
+      return failure(); // no unmasked equivalent
+    case MaskFormat::AllFalse:
+      rewriter.replaceOp(gather, gather.pass_thru());
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void GatherOp::getCanonicalizationPatterns(OwningRewritePatternList &results,
+                                           MLIRContext *context) {
+  results.insert<GatherFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // ScatterOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(ScatterOp op) {
   VectorType indicesVType = op.getIndicesVectorType();
   VectorType maskVType = op.getMaskVectorType();
   VectorType valueVType = op.getValueVectorType();
 
   if (valueVType.getElementType() != op.getMemRefType().getElementType())
     return op.emitOpError("base and value element type should match");
 
   if (valueVType.getDimSize(0) != indicesVType.getDimSize(0))
     return op.emitOpError("expected value dim to match indices dim");
   if (valueVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected value dim to match mask dim");
   return success();
 }
 
+namespace {
+class ScatterFolder final : public OpRewritePattern<ScatterOp> {
+public:
+  using OpRewritePattern<ScatterOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(ScatterOp scatter,
+                                PatternRewriter &rewriter) const override {
+    switch (get1DMaskFormat(scatter.mask())) {
+    case MaskFormat::AllTrue:
+      return failure(); // no unmasked equivalent
+    case MaskFormat::AllFalse:
+      rewriter.eraseOp(scatter);
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void ScatterOp::getCanonicalizationPatterns(OwningRewritePatternList &results,
+                                            MLIRContext *context) {
+  results.insert<ScatterFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // ExpandLoadOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(ExpandLoadOp op) {
   VectorType maskVType = op.getMaskVectorType();
   VectorType passVType = op.getPassThruVectorType();
   VectorType resVType = op.getResultVectorType();
 
   if (resVType.getElementType() != op.getMemRefType().getElementType())
     return op.emitOpError("base and result element type should match");
 
   if (resVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected result dim to match mask dim");
   if (resVType != passVType)
     return op.emitOpError("expected pass_thru of same type as result type");
   return success();
 }
 
+namespace {
+class ExpandLoadFolder final : public OpRewritePattern<ExpandLoadOp> {
+public:
+  using OpRewritePattern<ExpandLoadOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(ExpandLoadOp expand,
+                                PatternRewriter &rewriter) const override {
+    Value newBase;
+    switch (get1DMaskFormat(expand.mask())) {
+    case MaskFormat::AllTrue:
+      if (!castedToMemRef(expand.getLoc(), expand.base(),
+                          expand.getMemRefType(), expand.getResultVectorType(),
+                          rewriter, newBase))
+        return failure();
+      rewriter.replaceOpWithNewOp<LoadOp>(expand, newBase);
+      return success();
+    case MaskFormat::AllFalse:
+      rewriter.replaceOp(expand, expand.pass_thru());
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void ExpandLoadOp::getCanonicalizationPatterns(
+    OwningRewritePatternList &results, MLIRContext *context) {
+  results.insert<ExpandLoadFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // CompressStoreOp
 //===----------------------------------------------------------------------===//
 
 static LogicalResult verify(CompressStoreOp op) {
   VectorType maskVType = op.getMaskVectorType();
   VectorType valueVType = op.getValueVectorType();
 
   if (valueVType.getElementType() != op.getMemRefType().getElementType())
     return op.emitOpError("base and value element type should match");
 
   if (valueVType.getDimSize(0) != maskVType.getDimSize(0))
     return op.emitOpError("expected value dim to match mask dim");
   return success();
 }
 
+namespace {
+class CompressStoreFolder final : public OpRewritePattern<CompressStoreOp> {
+public:
+  using OpRewritePattern<CompressStoreOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(CompressStoreOp compress,
+                                PatternRewriter &rewriter) const override {
+    Value newBase;
+    switch (get1DMaskFormat(compress.mask())) {
+    case MaskFormat::AllTrue:
+      if (!castedToMemRef(compress.getLoc(), compress.base(),
+                          compress.getMemRefType(),
+                          compress.getValueVectorType(), rewriter, newBase))
+        return failure();
+      rewriter.replaceOpWithNewOp<StoreOp>(compress, compress.value(), newBase);
+      return success();
+    case MaskFormat::AllFalse:
+      rewriter.eraseOp(compress);
+      return success();
+    case MaskFormat::Unknown:
+      return failure();
+    }
+  }
+};
+} // namespace
+
+void CompressStoreOp::getCanonicalizationPatterns(
+    OwningRewritePatternList &results, MLIRContext *context) {
+  results.insert<CompressStoreFolder>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // ShapeCastOp
 //===----------------------------------------------------------------------===//
 
 /// Returns true if each element of 'a' is equal to the product of a contiguous
 /// sequence of the elements of 'b'. Returns false otherwise.
 static bool isValidShapeCast(ArrayRef<int64_t> a, ArrayRef<int64_t> b) {
   unsigned rankA = a.size();

 
 void CreateMaskOp::getCanonicalizationPatterns(
     OwningRewritePatternList &results, MLIRContext *context) {
   results.insert<CreateMaskFolder>(context);
 }
 
 void mlir::vector::populateVectorToVectorCanonicalizationPatterns(
     OwningRewritePatternList &patterns, MLIRContext *context) {
-  patterns.insert<CreateMaskFolder, StridedSliceConstantMaskFolder,
+  patterns.insert<CreateMaskFolder, MaskedLoadFolder, MaskedStoreFolder,
+                  GatherFolder, ScatterFolder, ExpandLoadFolder,
+                  CompressStoreFolder, StridedSliceConstantMaskFolder,
                   TransposeFolder>(context);
 }
 
 namespace mlir {
 namespace vector {
 
 #define GET_OP_CLASSES
 #include "mlir/Dialect/Vector/VectorOps.cpp.inc"
 
 } // namespace vector
 } // namespace mlir

mlir/test/Dialect/Vector/vector-mem-transforms.mlir

+// RUN: mlir-opt %s -test-vector-to-vector-conversion | FileCheck %s
+
+//
+// TODO: optimize this one too!
+//
+// CHECK-LABEL: func @maskedload0(
+// CHECK-SAME: %[[A0:.*]]: memref<?xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[M:.*]] = vector.constant_mask
+// CHECK-NEXT: %[[T:.*]] = vector.maskedload %[[A0]], %[[M]], %[[A1]] : memref<?xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+// CHECK-NEXT: return %[[T]] : vector<16xf32>
+
+func @maskedload0(%base: memref<?xf32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  %ld = vector.maskedload %base, %mask, %pass_thru
+    : memref<?xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @maskedload1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.type_cast %[[A0]] : memref<16xf32> to memref<vector<16xf32>>
+// CHECK-NEXT: %[[T1:.*]] = load %[[T0]][] : memref<vector<16xf32>>
+// CHECK-NEXT: return %[[T1]] : vector<16xf32>
+
+func @maskedload1(%base: memref<16xf32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  %ld = vector.maskedload %base, %mask, %pass_thru
+    : memref<16xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @maskedload2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: return %[[A1]] : vector<16xf32>
+
+func @maskedload2(%base: memref<16xf32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  %ld = vector.maskedload %base, %mask, %pass_thru
+    : memref<16xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @maskedstore1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.type_cast %[[A0]] : memref<16xf32> to memref<vector<16xf32>>
+// CHECK-NEXT: store %[[A1]], %[[T0]][] : memref<vector<16xf32>>
+// CHECK-NEXT: return
+
+func @maskedstore1(%base: memref<16xf32>, %value: vector<16xf32>) {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  vector.maskedstore %base, %mask, %value
+    : vector<16xi1>, vector<16xf32> into memref<16xf32>
+  return
+}
+
+// CHECK-LABEL: func @maskedstore2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: return
+
+func @maskedstore2(%base: memref<16xf32>, %value: vector<16xf32>)  {
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  vector.maskedstore %base, %mask, %value
+    : vector<16xi1>, vector<16xf32> into memref<16xf32>
+  return
+}
+
+// CHECK-LABEL: func @gather1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xi32>,
+// CHECK-SAME: %[[A2:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.constant_mask [16] : vector<16xi1>
+// CHECK-NEXT: %[[T1:.*]] = vector.gather %[[A0]], %[[A1]], %[[T0]], %[[A2]] : (memref<16xf32>, vector<16xi32>, vector<16xi1>, vector<16xf32>) -> vector<16xf32>
+// CHECK-NEXT: return %1 : vector<16xf32>
+
+func @gather1(%base: memref<16xf32>, %indices: vector<16xi32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  %ld = vector.gather %base, %indices, %mask, %pass_thru
+    : (memref<16xf32>, vector<16xi32>, vector<16xi1>, vector<16xf32>) -> vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @gather2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xi32>,
+// CHECK-SAME: %[[A2:.*]]: vector<16xf32>)
+// CHECK-NEXT: return %[[A2]] : vector<16xf32>
+
+func @gather2(%base: memref<16xf32>, %indices: vector<16xi32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  %ld = vector.gather %base, %indices, %mask, %pass_thru
+    : (memref<16xf32>, vector<16xi32>, vector<16xi1>, vector<16xf32>) -> vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @scatter1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xi32>,
+// CHECK-SAME: %[[A2:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.constant_mask [16] : vector<16xi1>
+// CHECK-NEXT: vector.scatter %[[A0]], %[[A1]], %[[T0]], %[[A2]] : vector<16xi32>, vector<16xi1>, vector<16xf32> into memref<16xf32>
+// CHECK-NEXT: return
+
+func @scatter1(%base: memref<16xf32>, %indices: vector<16xi32>, %value: vector<16xf32>) {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  vector.scatter %base, %indices, %mask, %value
+    : vector<16xi32>, vector<16xi1>, vector<16xf32> into memref<16xf32>
+  return
+}
+
+// CHECK-LABEL: func @scatter2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xi32>,
+// CHECK-SAME: %[[A2:.*]]: vector<16xf32>)
+// CHECK-NEXT: return
+
+func @scatter2(%base: memref<16xf32>, %indices: vector<16xi32>, %value: vector<16xf32>) {
+  %0 = vector.type_cast %base : memref<16xf32> to memref<vector<16xf32>>
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  vector.scatter %base, %indices, %mask, %value
+    : vector<16xi32>, vector<16xi1>, vector<16xf32> into memref<16xf32>
+  return
+}
+
+// CHECK-LABEL: func @expand1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.type_cast %[[A0]] : memref<16xf32> to memref<vector<16xf32>>
+// CHECK-NEXT: %[[T1:.*]] = load %[[T0]][] : memref<vector<16xf32>>
+// CHECK-NEXT: return %[[T1]] : vector<16xf32>
+
+func @expand1(%base: memref<16xf32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  %ld = vector.expandload %base, %mask, %pass_thru
+    : memref<16xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @expand2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: return %[[A1]] : vector<16xf32>
+
+func @expand2(%base: memref<16xf32>, %pass_thru: vector<16xf32>) -> vector<16xf32> {
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  %ld = vector.expandload %base, %mask, %pass_thru
+    : memref<16xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %ld : vector<16xf32>
+}
+
+// CHECK-LABEL: func @compress1(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: %[[T0:.*]] = vector.type_cast %[[A0]] : memref<16xf32> to memref<vector<16xf32>>
+// CHECK-NEXT: store %[[A1]], %[[T0]][] : memref<vector<16xf32>>
+// CHECK-NEXT: return
+
+func @compress1(%base: memref<16xf32>, %value: vector<16xf32>) {
+  %mask = vector.constant_mask [16] : vector<16xi1>
+  vector.compressstore %base, %mask, %value  : memref<16xf32>, vector<16xi1>, vector<16xf32>
+  return
+}
+
+// CHECK-LABEL: func @compress2(
+// CHECK-SAME: %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME: %[[A1:.*]]: vector<16xf32>)
+// CHECK-NEXT: return
+
+func @compress2(%base: memref<16xf32>, %value: vector<16xf32>) {
+  %mask = vector.constant_mask [0] : vector<16xi1>
+  vector.compressstore %base, %mask, %value : memref<16xf32>, vector<16xi1>, vector<16xf32>
+  return
+}