[mlir][spirv] First step to support spirv cooperative matrix extension.

[ThomasRaoux]

Summary

Add a new type to SPIRV dialect for cooperative matrix and add new op for cooperative matrix load. This is missing most instructions to support cooperative matrix extension but this is a stop-gap patch to avoid creating big review.

[mravishankar]

Could you also add some roundtrip tests to : https://github.com/llvm/llvm-project/blob/master/mlir/test/Dialect/SPIRV/ops.mlir (Actually a separate test file for these ops might also be fine)

mlir/include/mlir/Dialect/SPIRV/SPIRVBase.td

3228

The opcodes in this file are auto-generated using this script : https://github.com/llvm/llvm-project/blob/master/mlir/utils/spirv/define_opcodes.sh . Please use that instead of defining them here. (Sorry if the documentation doesnt point you in the right direction)

mlir/lib/Dialect/SPIRV/SPIRVDialect.cpp

282

Nit: Instead of using parseKeyword here, we could make this method : https://github.com/llvm/llvm-project/blob/7ee479a760e0a4402b4eb7fb6168768a44f66945/mlir/lib/Dialect/SPIRV/SPIRVOps.cpp#L147 a utility function and use that here?

mlir/lib/Dialect/SPIRV/Serialization/Deserializer.cpp

1237

I wonder if the automatic deserialization generation code would work here? (AFAIR the same code must already be generated by tblgen, but is being intercepted early here)

[ThomasRaoux]

Thanks Mahesh, I added roundtrip tests. Please take another look.

mlir/include/mlir/Dialect/SPIRV/SPIRVBase.td

3228

My bad, it was in the documentation, I had missed this part. Updated using the script.

mlir/lib/Dialect/SPIRV/SPIRVDialect.cpp

282

Done.

mlir/lib/Dialect/SPIRV/Serialization/Deserializer.cpp

1237

My understanding is that we get auto-generated code for the ops but here this if for deserialization of the new type. The type is not defined tblgen at all unless I'm missing something.

[mravishankar]

Sorry, I should have been more clear. These go through the serialization and deserialization of the ops. So while they are testing the parser and printer as well, they are primarily for testing that the serialization and deserialization works. Typically ops are tested for both serialization/deserialization through the SPIR-V binary, as well as just IR roundtrip. The IR round-trip tests live here : https://github.com/llvm/llvm-project/tree/master/mlir/test/Dialect/SPIRV . Regressions here indicate that the IR parsing/printing failed.

The serialization tests are within the Serialization sub-directory here : https://github.com/llvm/llvm-project/tree/master/mlir/test/Dialect/SPIRV/Serialization. These catch regressions in serialization/deserialization.

So I was asking for a roundtrip IR test.

Rest of the changes look fine to me.

mlir/lib/Dialect/SPIRV/Serialization/Deserializer.cpp

1237

Ah, your right. My bad.

[antiagainst]

Fantastic!! This is really great, Thomas! Thanks for taking on this! Overall looks good to me; I just have a few nits. Besides, can we add tests for

1. The coop mat type itself: https://github.com/llvm/llvm-project/blob/master/mlir/test/Dialect/SPIRV/types.mlir
2. (De)serialization for coop mat load op in the directory as pointed out by Mahesh: https://github.com/llvm/llvm-project/tree/master/mlir/test/Dialect/SPIRV/Serialization

mlir/include/mlir/Dialect/SPIRV/ParserUtils.h

2

The header explanation needs to be updated.

41

Nit: let's have an empty line before #endif to improve readability.

mlir/include/mlir/Dialect/SPIRV/SPIRVBase.td

2993–2994

Nit: can we keep this subsection alphabetically sorted?

mlir/include/mlir/Dialect/SPIRV/SPIRVCooperativeMatrixOps.td

23

What about copying the original doc for coopmat load from https://github.com/KhronosGroup/SPIRV-Registry/blob/master/extensions/NV/SPV_NV_cooperative_matrix.asciidoc#3328-memory-instructions here? We have been mostly following that convention for SPIR-V ops defined.

mlir/include/mlir/Dialect/SPIRV/SPIRVTypes.h

355

Nit: move this empy line after the function decl?

mlir/lib/Dialect/SPIRV/SPIRVDialect.cpp

269

spv.coopmatrix?

281

Can we put the scope ahead of the element type? I think it's more natural that way: element types typically appear together with row/col sizes. (The spec put it in an order like this but we don't need to follow that order.)

mlir/lib/Dialect/SPIRV/SPIRVOps.cpp

2623

spv.CooperativeMatrixLoadNV

mlir/lib/Dialect/SPIRV/SPIRVTypes.cpp

286

Nit: leave an empty line to separate the methods with fields.

315

We need to push SPV_NV_cooperative_matrix to extensions here.

320

We need to push CooperativeMatrixNV capability to capabilities here.

mlir/test/Dialect/SPIRV/Serialization/cooperative-matrix.mlir

4

We can drop the requires ... part. They are incorrect here given we actually need more capabilities and extensions for the following to be used.

[ThomasRaoux]

Thanks Lei. added some tests for the type in types.mir. About 2. I already had serialization/deserialization tests, am I missing something there?

mlir/test/Dialect/SPIRV/Serialization/cooperative-matrix.mlir

4

I get the following error if I remove the requires part. Maybe I missed something?

error: module must have 'vce_triple' attribute to be serializeable

[antiagainst]

Thanks Thomas! Just two final comments about the type assembly and tests. Feel free to land after addressing it. :)

mlir/test/Dialect/SPIRV/Serialization/cooperative-matrix.mlir

4

Ah, sorry. You are right. I mistakenly treated this test as one for op parsing/printing round-trip. (Hence also my comment to request tests for serialization.)

Then can we fix the required extension and capability? At the moment it's wrong. (We don't have validation for this yet but should have.) We need to have spv.vce<v1.0, [CooperativeMatrixNV], [SPV_NV_cooperative_matrix]> here.

mlir/test/Dialect/SPIRV/types.mlir

338

Hmm, sorry didn't point this out previously; I feel it's actually better to have the format of spv.coopmatrix<8x8xf32, Workgroup>. This is more consistent with vectors/tensors (vector<3x4xi32>) and pointers (spv.ptr<f32, PushConstant>) that way. How do you think?

[ThomasRaoux]

mlir/test/Dialect/SPIRV/Serialization/cooperative-matrix.mlir

4

Got it. Fixed.

mlir/test/Dialect/SPIRV/types.mlir

338

Looks much better indeed.

mlir/include/mlir/Dialect/SPIRV/ParserUtils.h

+//===------------ ParserUtils.h - Parse text to SPIR-V ops ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines utilities used for parsing types and ops for SPIR-V
+// dialect.
+//
+//===----------------------------------------------------------------------===//
+#ifndef MLIR_DIALECT_SPIRV_PARSERUTILS_H_
+#define MLIR_DIALECT_SPIRV_PARSERUTILS_H_
+
+#include "mlir/Dialect/SPIRV/SPIRVOps.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/OpImplementation.h"
+
+namespace mlir {
+/// Parses the next keyword in `parser` as an enumerant of the given
+/// `EnumClass`.
+template <typename EnumClass, typename ParserType>
+static ParseResult
+parseEnumKeywordAttr(EnumClass &value, ParserType &parser,
+                     StringRef attrName = spirv::attributeName<EnumClass>()) {
+  StringRef keyword;
+  SmallVector<NamedAttribute, 1> attr;
+  auto loc = parser.getCurrentLocation();
+  if (parser.parseKeyword(&keyword))
+    return failure();
+  if (Optional<EnumClass> attr = spirv::symbolizeEnum<EnumClass>(keyword)) {
+    value = attr.getValue();
+    return success();
+  }
+  return parser.emitError(loc, "invalid ")
+         << attrName << " attribute specification: " << keyword;
+}
+} // namespace mlir
+
+#endif // MLIR_DIALECT_SPIRV_PARSERUTILS_H_

mlir/include/mlir/Dialect/SPIRV/SPIRVBase.td

: Type<Or<[CPred<"$_self.isSignlessInteger(" # width # ")">,
                CPred<"$_self.isUnsignedInteger(" # width # ")">]>,
            width # "-bit signless/unsigned integer"> {
   int bitwidth = width;
 }
 
 class SignlessOrUnsignedIntOfWidths<list<int> widths> :
     AnyTypeOf<!foreach(w, widths, IOrUI<w>),
               StrJoinInt<widths, "/">.result # "-bit signless/unsigned integer">;
 
-def SPV_IsPtrType : CPred<"$_self.isa<::mlir::spirv::PointerType>()">;
 def SPV_IsArrayType : CPred<"$_self.isa<::mlir::spirv::ArrayType>()">;
+def SPV_IsCooperativeMatrixType :
+  CPred<"$_self.isa<::mlir::spirv::CooperativeMatrixNVType>()">;
+def SPV_IsPtrType : CPred<"$_self.isa<::mlir::spirv::PointerType>()">;
 def SPV_IsRTArrayType : CPred<"$_self.isa<::mlir::spirv::RuntimeArrayType>()">;
 def SPV_IsStructType : CPred<"$_self.isa<::mlir::spirv::StructType>()">;
 
 // See https://www.khronos.org/registry/spir-v/specs/unified1/SPIRV.html#_types
 // for the definition of the following types and type categories.
 
 def SPV_Void : TypeAlias<NoneType, "void">;
 def SPV_Bool : TypeAlias<I1, "bool">;
 def SPV_Integer : AnyIntOfWidths<[8, 16, 32, 64]>;
 def SPV_Float : FloatOfWidths<[16, 32, 64]>;
 def SPV_Float16or32 : FloatOfWidths<[16, 32]>;
 def SPV_Vector : VectorOfLengthAndType<[2, 3, 4],
                                        [SPV_Bool, SPV_Integer, SPV_Float]>;
 // Component type check is done in the type parser for the following SPIR-V
 // dialect-specific types so we use "Any" here.
 def SPV_AnyPtr : DialectType<SPIRV_Dialect, SPV_IsPtrType,
                              "any SPIR-V pointer type">;
 def SPV_AnyArray : DialectType<SPIRV_Dialect, SPV_IsArrayType,
                                "any SPIR-V array type">;
+def SPV_AnyCooperativeMatrix : DialectType<SPIRV_Dialect,
+                               SPV_IsCooperativeMatrixType,
+                               "any SPIR-V cooperative matrix type">;
 def SPV_AnyRTArray : DialectType<SPIRV_Dialect, SPV_IsRTArrayType,
                                  "any SPIR-V runtime array type">;
 def SPV_AnyStruct : DialectType<SPIRV_Dialect, SPV_IsStructType,
                                 "any SPIR-V struct type">;
 
 def SPV_Numerical : AnyTypeOf<[SPV_Integer, SPV_Float]>;
 def SPV_Scalar : AnyTypeOf<[SPV_Numerical, SPV_Bool]>;
 def SPV_Aggregate : AnyTypeOf<[SPV_AnyArray, SPV_AnyRTArray, SPV_AnyStruct]>;

 def SPV_OC_OpGroupNonUniformFMul       : I32EnumAttrCase<"OpGroupNonUniformFMul", 352>;
 def SPV_OC_OpGroupNonUniformSMin       : I32EnumAttrCase<"OpGroupNonUniformSMin", 353>;
 def SPV_OC_OpGroupNonUniformUMin       : I32EnumAttrCase<"OpGroupNonUniformUMin", 354>;
 def SPV_OC_OpGroupNonUniformFMin       : I32EnumAttrCase<"OpGroupNonUniformFMin", 355>;
 def SPV_OC_OpGroupNonUniformSMax       : I32EnumAttrCase<"OpGroupNonUniformSMax", 356>;
 def SPV_OC_OpGroupNonUniformUMax       : I32EnumAttrCase<"OpGroupNonUniformUMax", 357>;
 def SPV_OC_OpGroupNonUniformFMax       : I32EnumAttrCase<"OpGroupNonUniformFMax", 358>;
 def SPV_OC_OpSubgroupBallotKHR         : I32EnumAttrCase<"OpSubgroupBallotKHR", 4421>;
+def SPV_OC_OpTypeCooperativeMatrixNV   : I32EnumAttrCase<"OpTypeCooperativeMatrixNV", 5358>;
+def SPV_OC_OpCooperativeMatrixLoadNV   : I32EnumAttrCase<"OpCooperativeMatrixLoadNV", 5359>;
 
 def SPV_OpcodeAttr :
     SPV_I32EnumAttr<"Opcode", "valid SPIR-V instructions", [
       SPV_OC_OpNop, SPV_OC_OpUndef, SPV_OC_OpSourceContinued, SPV_OC_OpSource,
       SPV_OC_OpSourceExtension, SPV_OC_OpName, SPV_OC_OpMemberName, SPV_OC_OpString,
       SPV_OC_OpLine, SPV_OC_OpExtension, SPV_OC_OpExtInstImport, SPV_OC_OpExtInst,
       SPV_OC_OpMemoryModel, SPV_OC_OpEntryPoint, SPV_OC_OpExecutionMode,
       SPV_OC_OpCapability, SPV_OC_OpTypeVoid, SPV_OC_OpTypeBool, SPV_OC_OpTypeInt,
SPV_I32EnumAttr<"Opcode", "valid SPIR-V instructions", [
       SPV_OC_OpBranch, SPV_OC_OpBranchConditional, SPV_OC_OpReturn,
       SPV_OC_OpReturnValue, SPV_OC_OpUnreachable, SPV_OC_OpNoLine,
       SPV_OC_OpModuleProcessed, SPV_OC_OpGroupNonUniformElect,
       SPV_OC_OpGroupNonUniformBallot, SPV_OC_OpGroupNonUniformIAdd,
       SPV_OC_OpGroupNonUniformFAdd, SPV_OC_OpGroupNonUniformIMul,
       SPV_OC_OpGroupNonUniformFMul, SPV_OC_OpGroupNonUniformSMin,
       SPV_OC_OpGroupNonUniformUMin, SPV_OC_OpGroupNonUniformFMin,
       SPV_OC_OpGroupNonUniformSMax, SPV_OC_OpGroupNonUniformUMax,
-      SPV_OC_OpGroupNonUniformFMax, SPV_OC_OpSubgroupBallotKHR
+      SPV_OC_OpGroupNonUniformFMax, SPV_OC_OpSubgroupBallotKHR,
+      SPV_OC_OpTypeCooperativeMatrixNV, SPV_OC_OpCooperativeMatrixLoadNV
     ]>;
 
 // End opcode section. Generated from SPIR-V spec; DO NOT MODIFY!
 
 //===----------------------------------------------------------------------===//
 // SPIR-V op definitions
 //===----------------------------------------------------------------------===//
 

mlir/include/mlir/Dialect/SPIRV/SPIRVCooperativeMatrixOps.td

+//===- SPIRVCooperativeMatrixOps.td - cooperative matmul ---*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This is the op definition spec of cooperative matrix multiply extension ops.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SPIRV_COOPERATIVE_MATRIX_OPS
+#define SPIRV_COOPERATIVE_MATRIX_OPS
+
+// -----
+
+def SPV_CooperativeMatrixLoadNVOp : SPV_Op<"CooperativeMatrixLoadNV", []> {
+  let summary = "See extension SPV_NV_cooperative_matrix";
+
+  let description = [{
+    Load a cooperative matrix through a pointer.
+
+    Result Type is the type of the loaded object. It must be a cooperative
+    matrix type.
+
+    Pointer is a pointer into an array. Its type must be an OpTypePointer whose
+    Type operand is a scalar or vector type. The storage class of Pointer must
+    be Workgroup, StorageBuffer, or (if SPV_EXT_physical_storage_buffer is
+    supported) PhysicalStorageBufferEXT.
+
+    Stride is the number of elements in the array in memory between the first
+    component of consecutive rows (or columns) in the result. It must be a
+    scalar integer type.
+
+    ColumnMajor indicates whether the values loaded from memory are arranged in
+    column-major or row-major order. It must be a boolean constant instruction,
+    with false indicating row major and true indicating column major.
+
+    Memory Access must be a Memory Access literal. If not present, it is the
+    same as specifying None.
+
+    If ColumnMajor is false, then elements (row,*) of the result are taken in
+    order from contiguous locations starting at Pointer[row*Stride]. If
+    ColumnMajor is true, then elements (*,col) of the result are taken in order
+    from contiguous locations starting from Pointer[col*Stride]. Any ArrayStride
+    decoration on Pointer is ignored.
+
+    For a given dynamic instance of this instruction, all operands of this
+    instruction must be the same for all invocations in a given scope instance
+    (where the scope is the scope the cooperative matrix type was created with).
+    All invocations in a given scope instance must be active or all must be
+    inactive.
+
+    ### Custom assembly form
+
+    ``` {.ebnf}
+    cooperative-matrix-op ::= ssa-id `=` `spv.CooperativeMatrixLoadNV`
+                              storage-class ssa-use (`[` memory-access `]`)? `
+                              : ` cooperative-matrix-type
+    ```
+
+    For example:
+
+    ```
+    %0 = spv.CooperativeMatrixLoadNV "StorageBuffer" %ptr, %stride, %colMajor
+         : !spv.coopmatrix<i32, Workgroup, 16, 8>
+    ```
+  }];
+
+  let availability = [
+    MinVersion<SPV_V_1_0>,
+    MaxVersion<SPV_V_1_5>,
+    Extension<[SPV_NV_cooperative_matrix]>,
+    Capability<[SPV_C_CooperativeMatrixNV]>
+  ];
+
+  let arguments = (ins
+    SPV_AnyPtr:$pointer,
+    SPV_Integer:$stride,
+    SPV_Bool:$columnmajor,
+    OptionalAttr<SPV_MemoryAccessAttr>:$memory_access
+  );
+
+  let results = (outs
+    SPV_AnyCooperativeMatrix:$result
+  );
+
+  let verifier = [{ return success(); }];
+}
+
+// -----
+
+#endif // SPIRV_COOPERATIVE_MATRIX_OPS

mlir/include/mlir/Dialect/SPIRV/SPIRVOps.td

 
 include "mlir/Dialect/SPIRV/SPIRVBase.td"
 include "mlir/Dialect/SPIRV/SPIRVArithmeticOps.td"
 include "mlir/Dialect/SPIRV/SPIRVAtomicOps.td"
 include "mlir/Dialect/SPIRV/SPIRVBitOps.td"
 include "mlir/Dialect/SPIRV/SPIRVCastOps.td"
 include "mlir/Dialect/SPIRV/SPIRVCompositeOps.td"
 include "mlir/Dialect/SPIRV/SPIRVControlFlowOps.td"
+include "mlir/Dialect/SPIRV/SPIRVCooperativeMatrixOps.td"
 include "mlir/Dialect/SPIRV/SPIRVGLSLOps.td"
 include "mlir/Dialect/SPIRV/SPIRVGroupOps.td"
 include "mlir/Dialect/SPIRV/SPIRVLogicalOps.td"
 include "mlir/Dialect/SPIRV/SPIRVNonUniformOps.td"
 include "mlir/Dialect/SPIRV/SPIRVStructureOps.td"
 include "mlir/Interfaces/SideEffectInterfaces.td"
 
 // -----

mlir/include/mlir/Dialect/SPIRV/SPIRVTypes.h

 /// capabilities are implicitly declared by the given capability. Compared to
 /// the above function, this function collects implied capabilities recursively:
 /// if an implicitly declared capability implicitly declares a third one, the
 /// third one will also be returned.
 SmallVector<Capability, 0> getRecursiveImpliedCapabilities(Capability cap);
 
 namespace detail {
 struct ArrayTypeStorage;
+struct CooperativeMatrixTypeStorage;
 struct ImageTypeStorage;
 struct PointerTypeStorage;
 struct RuntimeArrayTypeStorage;
 struct StructTypeStorage;
 } // namespace detail
 
 namespace TypeKind {
 enum Kind {
   Array = Type::FIRST_SPIRV_TYPE,
+  CooperativeMatrix,
   Image,
   Pointer,
   RuntimeArray,
   Struct,
   LAST_SPIRV_TYPE = Struct,
 };
 }
 
void getMemberDecorations(
       unsigned i, SmallVectorImpl<spirv::Decoration> &memberDecorations) const;
 
   void getExtensions(SPIRVType::ExtensionArrayRefVector &extensions,
                      Optional<spirv::StorageClass> storage = llvm::None);
   void getCapabilities(SPIRVType::CapabilityArrayRefVector &capabilities,
                        Optional<spirv::StorageClass> storage = llvm::None);
 };
 
+// SPIR-V cooperative matrix type
+class CooperativeMatrixNVType
+    : public Type::TypeBase<CooperativeMatrixNVType, SPIRVType,
+                            detail::CooperativeMatrixTypeStorage> {
+public:
+  using Base::Base;
+
+  static bool kindof(unsigned kind) {
+    return kind == TypeKind::CooperativeMatrix;
+  }
+
+  static CooperativeMatrixNVType get(Type elementType, spirv::Scope scope,
+                                     unsigned rows, unsigned columns);
+  Type getElementType() const;
+
+  /// Return the scope of the cooperative matrix.
+  spirv::Scope getScope() const;
+  /// return the number of rows of the matrix.
+  unsigned getRows() const;
+  /// return the number of columns of the matrix.
+  unsigned getColumns() const;
+
+  void getExtensions(SPIRVType::ExtensionArrayRefVector &extensions,
+                     Optional<spirv::StorageClass> storage = llvm::None);
+  void getCapabilities(SPIRVType::CapabilityArrayRefVector &capabilities,
+                       Optional<spirv::StorageClass> storage = llvm::None);
+};
+
 } // end namespace spirv
 } // end namespace mlir
 
 #endif // MLIR_DIALECT_SPIRV_SPIRVTYPES_H_

mlir/lib/Dialect/SPIRV/SPIRVDialect.cpp

 //===- LLVMDialect.cpp - MLIR SPIR-V dialect ------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the SPIR-V dialect in MLIR.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/SPIRV/SPIRVDialect.h"
+#include "mlir/Dialect/SPIRV/ParserUtils.h"
 #include "mlir/Dialect/SPIRV/SPIRVOps.h"
 #include "mlir/Dialect/SPIRV/SPIRVTypes.h"
 #include "mlir/Dialect/SPIRV/TargetAndABI.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/MLIRContext.h"
 #include "mlir/IR/StandardTypes.h"
 #include "mlir/Parser.h"

 } // namespace
 
 //===----------------------------------------------------------------------===//
 // SPIR-V Dialect
 //===----------------------------------------------------------------------===//
 
 SPIRVDialect::SPIRVDialect(MLIRContext *context)
     : Dialect(getDialectNamespace(), context) {
-  addTypes<ArrayType, ImageType, PointerType, RuntimeArrayType, StructType>();
+  addTypes<ArrayType, CooperativeMatrixNVType, ImageType, PointerType,
+           RuntimeArrayType, StructType>();
 
   addAttributes<InterfaceVarABIAttr, TargetEnvAttr, VerCapExtAttr>();
 
   // Add SPIR-V ops.
   addOperations<
 #define GET_OP_LIST
 #include "mlir/Dialect/SPIRV/SPIRVOps.cpp.inc"
       >();
static Type parseArrayType(SPIRVDialect const &dialect,
   if (failed(parseOptionalArrayStride(dialect, parser, stride)))
     return Type();
 
   if (parser.parseGreater())
     return Type();
   return ArrayType::get(elementType, count, stride);
 }
 
+// cooperative-matrix-type ::= `!spv.coopmatrix` `<` element-type ',' scope ','
+//                                                   rows ',' coloumns>`
+static Type parseCooperativeMatrixType(SPIRVDialect const &dialect,
+                                       DialectAsmParser &parser) {
+  if (parser.parseLess())
+    return Type();
+
+  SmallVector<int64_t, 2> dims;
+  llvm::SMLoc countLoc = parser.getCurrentLocation();
+  if (parser.parseDimensionList(dims, /*allowDynamic=*/false))
+    return Type();
+
+  if (dims.size() != 2) {
+    parser.emitError(countLoc, "expected rows and columns size.");
+    return Type();
+  }
+
+  auto elementTy = parseAndVerifyType(dialect, parser);
+  if (!elementTy)
+    return Type();
+
+  Scope scope;
+  if (parser.parseComma() || parseEnumKeywordAttr(scope, parser, "scope <id>"))
+    return Type();
+
+  if (parser.parseGreater())
+    return Type();
+  return CooperativeMatrixNVType::get(elementTy, scope, dims[0], dims[1]);
+}
+
 // TODO(ravishankarm) : Reorder methods to be utilities first and parse*Type
 // methods in alphabetical order
 //
 // storage-class ::= `UniformConstant`
 //                 | `Uniform`
 //                 | `Workgroup`
 //                 | <and other storage classes...>
 //

 //              | struct-type
 Type SPIRVDialect::parseType(DialectAsmParser &parser) const {
   StringRef keyword;
   if (parser.parseKeyword(&keyword))
     return Type();
 
   if (keyword == "array")
     return parseArrayType(*this, parser);
+  if (keyword == "coopmatrix")
+    return parseCooperativeMatrixType(*this, parser);
   if (keyword == "image")
     return parseImageType(*this, parser);
   if (keyword == "ptr")
     return parsePointerType(*this, parser);
   if (keyword == "rtarray")
     return parseRuntimeArrayType(*this, parser);
   if (keyword == "struct")
     return parseStructType(*this, parser);
if (type.hasLayout() || !decorations.empty()) {
       os << "]";
     }
   };
   llvm::interleaveComma(llvm::seq<unsigned>(0, type.getNumElements()), os,
                         printMember);
   os << ">";
 }
 
+static void print(CooperativeMatrixNVType type, DialectAsmPrinter &os) {
+  os << "coopmatrix<" << type.getRows() << "x" << type.getColumns() << "x";
+  os << type.getElementType() << ", " << stringifyScope(type.getScope());
+  os << ">";
+}
+
 void SPIRVDialect::printType(Type type, DialectAsmPrinter &os) const {
   switch (type.getKind()) {
   case TypeKind::Array:
     print(type.cast<ArrayType>(), os);
     return;
+  case TypeKind::CooperativeMatrix:
+    print(type.cast<CooperativeMatrixNVType>(), os);
+    return;
   case TypeKind::Pointer:
     print(type.cast<PointerType>(), os);
     return;
   case TypeKind::RuntimeArray:
     print(type.cast<RuntimeArrayType>(), os);
     return;
   case TypeKind::Image:
     print(type.cast<ImageType>(), os);

mlir/lib/Dialect/SPIRV/SPIRVOps.cpp

 //===- SPIRVOps.cpp - MLIR SPIR-V operations ------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the operations in the SPIR-V dialect.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/SPIRV/SPIRVOps.h"
 
+#include "mlir/Dialect/SPIRV/ParserUtils.h"
 #include "mlir/Dialect/SPIRV/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/SPIRVDialect.h"
 #include "mlir/Dialect/SPIRV/SPIRVTypes.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/Function.h"
 #include "mlir/IR/FunctionImplementation.h"
 #include "mlir/IR/OpImplementation.h"
 #include "mlir/IR/StandardTypes.h"
parseEnumStrAttr(EnumClass &value, OpAsmParser &parser, OperationState &state,
   if (parseEnumStrAttr(value, parser)) {
     return failure();
   }
   state.addAttribute(attrName, parser.getBuilder().getI32IntegerAttr(
                                    llvm::bit_cast<int32_t>(value)));
   return success();
 }
 
-/// Parses the next keyword in `parser` as an enumerant of the given
-/// `EnumClass`.
-template <typename EnumClass>
-static ParseResult
-parseEnumKeywordAttr(EnumClass &value, OpAsmParser &parser,
-                     StringRef attrName = spirv::attributeName<EnumClass>()) {
-  StringRef keyword;
-  SmallVector<NamedAttribute, 1> attr;
-  auto loc = parser.getCurrentLocation();
-  if (parser.parseKeyword(&keyword))
-    return failure();
-  if (Optional<EnumClass> attr = spirv::symbolizeEnum<EnumClass>(keyword)) {
-    value = attr.getValue();
-    return success();
-  }
-  return parser.emitError(loc, "invalid ")
-         << attrName << " attribute specification: " << keyword;
-}
-
 /// Parses the next keyword in `parser` as an enumerant of the given `EnumClass`
 /// and inserts the enumerant into `state` as an 32-bit integer attribute with
 /// the enum class's name as attribute name.
 template <typename EnumClass>
 static ParseResult
 parseEnumKeywordAttr(EnumClass &value, OpAsmParser &parser,
                      OperationState &state,
                      StringRef attrName = spirv::attributeName<EnumClass>()) {
for (const auto &attr : {descriptorSetName, bindingName, builtInName}) {
     if (op->getAttr(attr))
       return varOp.emitOpError("cannot have '")
              << attr << "' attribute (only allowed in spv.globalVariable)";
   }
 
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// spv.CooperativeMatrixLoadNV
+//===----------------------------------------------------------------------===//
+
+static ParseResult parseCooperativeMatrixLoadNVOp(OpAsmParser &parser,
+                                                  OperationState &state) {
+  spirv::StorageClass storageClass;
+  SmallVector<OpAsmParser::OperandType, 3> operandInfo;
+  Type strideType = parser.getBuilder().getIntegerType(32);
+  Type columnMajorType = parser.getBuilder().getIntegerType(1);
+  Type elementType;
+  if (parseEnumStrAttr(storageClass, parser) ||
+      parser.parseOperandList(operandInfo, 3) ||
+      parseMemoryAccessAttributes(parser, state) || parser.parseColon() ||
+      parser.parseType(elementType)) {
+    return failure();
+  }
+
+  auto ptrType = spirv::PointerType::get(
+      elementType.cast<spirv::CooperativeMatrixNVType>().getElementType(),
+      storageClass);
+  SmallVector<Type, 3> OperandType = {ptrType, strideType, columnMajorType};
+  if (parser.resolveOperands(operandInfo, OperandType, parser.getNameLoc(),
+                             state.operands)) {
+    return failure();
+  }
+
+  state.addTypes(elementType);
+  return success();
+}
+
+static void print(spirv::CooperativeMatrixLoadNVOp M, OpAsmPrinter &printer) {
+  StringRef sc = stringifyStorageClass(
+      M.pointer().getType().cast<spirv::PointerType>().getStorageClass());
+  printer << spirv::CooperativeMatrixLoadNVOp::getOperationName() << " \"" << sc
+          << "\" " << M.pointer() << ", " << M.stride() << ", "
+          << M.columnmajor();
+  // Print optional memory access attribute.
+  if (auto memAccess = M.memory_access())
+    printer << " [\"" << stringifyMemoryAccess(*memAccess) << "\"]";
+  printer << " : " << M.getType();
+}
+
 namespace mlir {
 namespace spirv {
 
 // TableGen'erated operation interfaces for querying versions, extensions, and
 // capabilities.
 #include "mlir/Dialect/SPIRV/SPIRVAvailability.cpp.inc"
 
 // TablenGen'erated operation definitions.
 #define GET_OP_CLASSES
 #include "mlir/Dialect/SPIRV/SPIRVOps.cpp.inc"
 
 // TableGen'erated operation availability interface implementations.
 #include "mlir/Dialect/SPIRV/SPIRVOpAvailabilityImpl.inc"
 
 } // namespace spirv
 } // namespace mlir

mlir/lib/Dialect/SPIRV/SPIRVTypes.cpp

 
 //===----------------------------------------------------------------------===//
 // CompositeType
 //===----------------------------------------------------------------------===//
 
 bool CompositeType::classof(Type type) {
   switch (type.getKind()) {
   case TypeKind::Array:
+  case TypeKind::CooperativeMatrix:
   case TypeKind::RuntimeArray:
   case TypeKind::Struct:
     return true;
   case StandardTypes::Vector:
     return isValid(type.cast<VectorType>());
   default:
     return false;
   }
 }
 
 bool CompositeType::isValid(VectorType type) {
   return type.getRank() == 1 && type.getElementType().isa<ScalarType>() &&
          type.getNumElements() >= 2 && type.getNumElements() <= 4;
 }
 
 Type CompositeType::getElementType(unsigned index) const {
   switch (getKind()) {
   case spirv::TypeKind::Array:
     return cast<ArrayType>().getElementType();
+  case spirv::TypeKind::CooperativeMatrix:
+    return cast<CooperativeMatrixNVType>().getElementType();
   case spirv::TypeKind::RuntimeArray:
     return cast<RuntimeArrayType>().getElementType();
   case spirv::TypeKind::Struct:
     return cast<StructType>().getElementType(index);
   case StandardTypes::Vector:
     return cast<VectorType>().getElementType();
   default:
     llvm_unreachable("invalid composite type");
   }
 }
 
 unsigned CompositeType::getNumElements() const {
   switch (getKind()) {
   case spirv::TypeKind::Array:
     return cast<ArrayType>().getNumElements();
+  case spirv::TypeKind::CooperativeMatrix:
+    return cast<CooperativeMatrixNVType>().getRows() *
+           cast<CooperativeMatrixNVType>().getColumns();
   case spirv::TypeKind::RuntimeArray:
     llvm_unreachable(
         "invalid to query number of elements of spirv::RuntimeArray type");
   case spirv::TypeKind::Struct:
     return cast<StructType>().getNumElements();
   case StandardTypes::Vector:
     return cast<VectorType>().getNumElements();
   default:
     llvm_unreachable("invalid composite type");
   }
 }
 
 void CompositeType::getExtensions(
     SPIRVType::ExtensionArrayRefVector &extensions,
     Optional<StorageClass> storage) {
   switch (getKind()) {
   case spirv::TypeKind::Array:
     cast<ArrayType>().getExtensions(extensions, storage);
     break;
+  case spirv::TypeKind::CooperativeMatrix:
+    cast<CooperativeMatrixNVType>().getExtensions(extensions, storage);
+    break;
   case spirv::TypeKind::RuntimeArray:
     cast<RuntimeArrayType>().getExtensions(extensions, storage);
     break;
   case spirv::TypeKind::Struct:
     cast<StructType>().getExtensions(extensions, storage);
     break;
   case StandardTypes::Vector:
     cast<VectorType>().getElementType().cast<ScalarType>().getExtensions(
         extensions, storage);
     break;
   default:
     llvm_unreachable("invalid composite type");
   }
 }
 
 void CompositeType::getCapabilities(
     SPIRVType::CapabilityArrayRefVector &capabilities,
     Optional<StorageClass> storage) {
   switch (getKind()) {
   case spirv::TypeKind::Array:
     cast<ArrayType>().getCapabilities(capabilities, storage);
     break;
+  case spirv::TypeKind::CooperativeMatrix:
+    cast<CooperativeMatrixNVType>().getCapabilities(capabilities, storage);
+    break;
   case spirv::TypeKind::RuntimeArray:
     cast<RuntimeArrayType>().getCapabilities(capabilities, storage);
     break;
   case spirv::TypeKind::Struct:
     cast<StructType>().getCapabilities(capabilities, storage);
     break;
   case StandardTypes::Vector:
     cast<VectorType>().getElementType().cast<ScalarType>().getCapabilities(
         capabilities, storage);
     break;
   default:
     llvm_unreachable("invalid composite type");
   }
 }
 
 //===----------------------------------------------------------------------===//
+// CooperativeMatrixType
+//===----------------------------------------------------------------------===//
+
+struct spirv::detail::CooperativeMatrixTypeStorage : public TypeStorage {
+  using KeyTy = std::tuple<Type, Scope, unsigned, unsigned>;
+
+  static CooperativeMatrixTypeStorage *
+  construct(TypeStorageAllocator &allocator, const KeyTy &key) {
+    return new (allocator.allocate<CooperativeMatrixTypeStorage>())
+        CooperativeMatrixTypeStorage(key);
+  }
+
+  bool operator==(const KeyTy &key) const {
+    return key == KeyTy(elementType, getScope(), rows, columns);
+  }
+
+  CooperativeMatrixTypeStorage(const KeyTy &key)
+      : TypeStorage(static_cast<unsigned>(std::get<1>(key))),
+        elementType(std::get<0>(key)), rows(std::get<2>(key)),
+        columns(std::get<3>(key)) {}
+
+  Scope getScope() const { return static_cast<Scope>(getSubclassData()); }
+
+  Type elementType;
+  unsigned rows;
+  unsigned columns;
+};
+
+CooperativeMatrixNVType CooperativeMatrixNVType::get(Type elementType,
+                                                     Scope scope, unsigned rows,
+                                                     unsigned columns) {
+  return Base::get(elementType.getContext(), TypeKind::CooperativeMatrix,
+                   elementType, scope, rows, columns);
+}
+
+Type CooperativeMatrixNVType::getElementType() const {
+  return getImpl()->elementType;
+}
+
+Scope CooperativeMatrixNVType::getScope() const {
+  return getImpl()->getScope();
+}
+
+unsigned CooperativeMatrixNVType::getRows() const { return getImpl()->rows; }
+
+unsigned CooperativeMatrixNVType::getColumns() const {
+  return getImpl()->columns;
+}
+
+void CooperativeMatrixNVType::getExtensions(
+    SPIRVType::ExtensionArrayRefVector &extensions,
+    Optional<StorageClass> storage) {
+  getElementType().cast<SPIRVType>().getExtensions(extensions, storage);
+  extensions.push_back(Extension::SPV_NV_cooperative_matrix);
+}
+
+void CooperativeMatrixNVType::getCapabilities(
+    SPIRVType::CapabilityArrayRefVector &capabilities,
+    Optional<StorageClass> storage) {
+  getElementType().cast<SPIRVType>().getCapabilities(capabilities, storage);
+  capabilities.push_back(Capability::CooperativeMatrixNV);
+}
+
+//===----------------------------------------------------------------------===//
 // ImageType
 //===----------------------------------------------------------------------===//
 
 template <typename T> static constexpr unsigned getNumBits() { return 0; }
 template <> constexpr unsigned getNumBits<Dim>() {
   static_assert((1 << 3) > getMaxEnumValForDim(),
                 "Not enough bits to encode Dim value");
   return 3;

mlir/lib/Dialect/SPIRV/Serialization/Deserializer.cpp

private:
   bool isVoidType(Type type) const { return type.isa<NoneType>(); }
 
   /// Processes a SPIR-V type instruction with given `opcode` and `operands` and
   /// registers the type into `module`.
   LogicalResult processType(spirv::Opcode opcode, ArrayRef<uint32_t> operands);
 
   LogicalResult processArrayType(ArrayRef<uint32_t> operands);
 
+  LogicalResult processCooperativeMatrixType(ArrayRef<uint32_t> operands);
+
   LogicalResult processFunctionType(ArrayRef<uint32_t> operands);
 
   LogicalResult processRuntimeArrayType(ArrayRef<uint32_t> operands);
 
   LogicalResult processStructType(ArrayRef<uint32_t> operands);
 
   //===--------------------------------------------------------------------===//
   // Constant
if (!pointeeType) {
       return emitError(unknownLoc, "unknown OpTypePointer pointee type <id> ")
              << operands[2];
     }
     auto storageClass = static_cast<spirv::StorageClass>(operands[1]);
     typeMap[operands[0]] = spirv::PointerType::get(pointeeType, storageClass);
   } break;
   case spirv::Opcode::OpTypeArray:
     return processArrayType(operands);
+  case spirv::Opcode::OpTypeCooperativeMatrixNV:
+    return processCooperativeMatrixType(operands);
   case spirv::Opcode::OpTypeFunction:
     return processFunctionType(operands);
   case spirv::Opcode::OpTypeRuntimeArray:
     return processRuntimeArrayType(operands);
   case spirv::Opcode::OpTypeStruct:
     return processStructType(operands);
   default:
     return emitError(unknownLoc, "unhandled type instruction");
LogicalResult Deserializer::processFunctionType(ArrayRef<uint32_t> operands) {
   if (!isVoidType(returnType)) {
     returnTypes = llvm::makeArrayRef(returnType);
   }
   typeMap[operands[0]] = FunctionType::get(argTypes, returnTypes, context);
   return success();
 }
 
 LogicalResult
+Deserializer::processCooperativeMatrixType(ArrayRef<uint32_t> operands) {
+  if (operands.size() != 5) {
+    return emitError(unknownLoc, "OpTypeCooperativeMatrix must have element "
+                                 "type and row x column parameters");
+  }
+
+  Type elementTy = getType(operands[1]);
+  if (!elementTy) {
+    return emitError(unknownLoc,
+                     "OpTypeCooperativeMatrix references undefined <id> ")
+           << operands[1];
+  }
+
+  auto scope = spirv::symbolizeScope(operands[2]);
+  if (!scope) {
+    return emitError(unknownLoc,
+                     "OpTypeCooperativeMatrix references undefined scope <id> ")
+           << operands[2];
+  }
+
+  unsigned rows = operands[3];
+  unsigned columns = operands[4];
+
+  typeMap[operands[0]] = spirv::CooperativeMatrixNVType::get(
+      elementTy, scope.getValue(), rows, columns);
+  return success();
+}
+
+LogicalResult
 Deserializer::processRuntimeArrayType(ArrayRef<uint32_t> operands) {
   if (operands.size() != 2) {
     return emitError(unknownLoc, "OpTypeRuntimeArray must have two operands");
   }
   Type memberType = getType(operands[1]);
   if (!memberType) {
     return emitError(unknownLoc,
                      "OpTypeRuntimeArray references undefined <id> ")
LogicalResult Deserializer::processInstruction(spirv::Opcode opcode,
   case spirv::Opcode::OpTypeInt:
   case spirv::Opcode::OpTypeFloat:
   case spirv::Opcode::OpTypeVector:
   case spirv::Opcode::OpTypeArray:
   case spirv::Opcode::OpTypeFunction:
   case spirv::Opcode::OpTypeRuntimeArray:
   case spirv::Opcode::OpTypeStruct:
   case spirv::Opcode::OpTypePointer:
+  case spirv::Opcode::OpTypeCooperativeMatrixNV:
     return processType(opcode, operands);
   case spirv::Opcode::OpConstant:
     return processConstant(operands, /*isSpec=*/false);
   case spirv::Opcode::OpSpecConstant:
     return processConstant(operands, /*isSpec=*/true);
   case spirv::Opcode::OpConstantComposite:
     return processConstantComposite(operands);
   case spirv::Opcode::OpConstantTrue:

mlir/lib/Dialect/SPIRV/Serialization/Serializer.cpp

for (auto &memberDecoration : memberDecorations) {
                << memberDecoration.first << "-th member of " << structType
                << " with " << stringifyDecoration(memberDecoration.second);
       }
     }
     typeEnum = spirv::Opcode::OpTypeStruct;
     return success();
   }
 
+  if (auto cooperativeMatrixType =
+          type.dyn_cast<spirv::CooperativeMatrixNVType>()) {
+    uint32_t elementTypeID = 0;
+    if (failed(processType(loc, cooperativeMatrixType.getElementType(),
+                           elementTypeID))) {
+      return failure();
+    }
+    typeEnum = spirv::Opcode::OpTypeCooperativeMatrixNV;
+    operands.push_back(elementTypeID);
+    operands.push_back(static_cast<uint32_t>(cooperativeMatrixType.getScope()));
+    operands.push_back(cooperativeMatrixType.getRows());
+    operands.push_back(cooperativeMatrixType.getColumns());
+    return success();
+  }
+
   // TODO(ravishankarm) : Handle other types.
   return emitError(loc, "unhandled type in serialization: ") << type;
 }
 
 LogicalResult
 Serializer::prepareFunctionType(Location loc, FunctionType type,
                                 spirv::Opcode &typeEnum,
                                 SmallVectorImpl<uint32_t> &operands) {

mlir/test/Dialect/SPIRV/Serialization/cooperative-matrix.mlir

+// RUN: mlir-translate -test-spirv-roundtrip -split-input-file %s | FileCheck %s
+
+spv.module Logical GLSL450 requires #spv.vce<v1.0, [CooperativeMatrixNV], [SPV_NV_cooperative_matrix]> {
+  // CHECK-LABEL: @cooperative_matrix_load
+  spv.func @cooperative_matrix_load(%ptr : !spv.ptr<i32, StorageBuffer>, %stride : i32, %b : i1) "None" {
+    // CHECK: {{%.*}} = spv.CooperativeMatrixLoadNV "StorageBuffer" {{%.*}}, {{%.*}}, {{%.*}} : !spv.coopmatrix<16x8xi32, Workgroup>
+    %0 = spv.CooperativeMatrixLoadNV "StorageBuffer" %ptr, %stride, %b : !spv.coopmatrix<16x8xi32, Workgroup>
+    spv.Return
+  }
+
+  // CHECK-LABEL: @cooperative_matrix_load_memaccess
+  spv.func @cooperative_matrix_load_memaccess(%ptr : !spv.ptr<i32, StorageBuffer>, %stride : i32, %b : i1) "None" {
+    // CHECK: {{%.*}} = spv.CooperativeMatrixLoadNV "StorageBuffer" {{%.*}}, {{%.*}}, {{%.*}} ["Volatile"] : !spv.coopmatrix<8x16xi32, Subgroup>
+    %0 = spv.CooperativeMatrixLoadNV "StorageBuffer" %ptr, %stride, %b ["Volatile"] : !spv.coopmatrix<8x16xi32, Subgroup>
+    spv.Return
+  }
+}

mlir/test/Dialect/SPIRV/cooperative-matrix.mlir

+// RUN: mlir-opt -allow-unregistered-dialect -split-input-file -verify-diagnostics %s | FileCheck %s
+
+// CHECK-LABEL: @cooperative_matrix_load
+spv.func @cooperative_matrix_load(%ptr : !spv.ptr<i32, StorageBuffer>, %stride : i32, %b : i1) "None" {
+  // CHECK: {{%.*}} = spv.CooperativeMatrixLoadNV "StorageBuffer" {{%.*}}, {{%.*}}, {{%.*}} : !spv.coopmatrix<16x8xi32, Workgroup>
+  %0 = spv.CooperativeMatrixLoadNV "StorageBuffer" %ptr, %stride, %b : !spv.coopmatrix<16x8xi32, Workgroup>
+  spv.Return
+}
+
+// -----
+// CHECK-LABEL: @cooperative_matrix_load_memaccess
+spv.func @cooperative_matrix_load_memaccess(%ptr : !spv.ptr<i32, StorageBuffer>, %stride : i32, %b : i1) "None" {
+  // CHECK: {{%.*}} = spv.CooperativeMatrixLoadNV "StorageBuffer" {{%.*}}, {{%.*}}, {{%.*}} ["Volatile"] : !spv.coopmatrix<8x16xi32, Subgroup>
+  %0 = spv.CooperativeMatrixLoadNV "StorageBuffer" %ptr, %stride, %b ["Volatile"] : !spv.coopmatrix<8x16xi32, Subgroup>
+  spv.Return
+}

mlir/test/Dialect/SPIRV/types.mlir

 
 // expected-error @+1 {{expected ','}}
 func @struct_type_missing_comma(!spv.struct<f32 [0 NonWritable], i32 [4]>)
 
 // -----
 
 // expected-error @+1 {{expected ']'}}
 func @struct_type_missing_comma(!spv.struct<f32 [0, NonWritable NonReadable], i32 [4]>)
+
+// -----
+
+//===----------------------------------------------------------------------===//
+// CooperativeMatrix
+//===----------------------------------------------------------------------===//
+
+// CHECK: func @coop_matrix_type(!spv.coopmatrix<8x16xi32, Subgroup>, !spv.coopmatrix<8x8xf32, Workgroup>)
+func @coop_matrix_type(!spv.coopmatrix<8x16xi32, Subgroup>, !spv.coopmatrix<8x8xf32, Workgroup>) -> ()
+
+// -----
+
+// expected-error @+1 {{expected ','}}
+func @missing_scope(!spv.coopmatrix<8x16xi32>) -> ()
+
+// -----
+
+// expected-error @+1 {{expected rows and columns size}}
+func @missing_count(!spv.coopmatrix<8xi32, Subgroup>) -> ()
+