[Matrix] Tighten LangRef definitions and Verifier checks.

[SjoerdMeijer]

Summary

This tightens the matrix intrinsic definitions in LLVM LangRef and adds corresponding checks to the IR Verifier.

[fhahn]

Thanks for improving the LangRef and verifier. I am not entirely sure about referring to the matrixes as linearized (see inline comment), otherwise looks great.

llvm/docs/LangRef.rst

15498–15499

When I read linearized here, I thing about https://en.wikipedia.org/wiki/Linearization , so there might be potential for confusion.

It might be worth defining exactly what we mean be embedding here, then further uses should be un-ambigous: the columns of a matrix R x C are embedded into a vector such that the elements of subsequent columns are adjacent in the vector. Or more formally element I of column J is at index J * R + I in the vector (with indices starting at 0)

[SjoerdMeijer]

llvm/docs/LangRef.rst

15498–15499

Yep, thanks. I was looking how to rephrase "embedded", but agree that "linearization" is perhaps equally vague, so yes this is the best we can do:

Or more formally element I of column J is at index J * R + I in the vector (with indices starting at 0)

Will go for that one.

[fhahn]

llvm/docs/LangRef.rst

15498–15499

It would also be good to say that layout defaults to column major currently. It can be changed globally during the lowering to row-major as well, but we probably do not want to mention actual pass specifics here.

[SjoerdMeijer]

llvm/docs/LangRef.rst

15578

I am actually now also interested in defining %Stride better. Using our new definition:

For a R x C matrix, element i of column j is at index j * R + i in its vector, with indices starting at 0.

From the description of %Stride it follows that:

%Stride = ( (j+1) * R + 0) - (j * R + 0)
=>
%Stride = R

So double checking: we can simply the description of %Stride just by saying it is equal to the number of rows, is that correct?

[fhahn]

llvm/docs/LangRef.rst

15578

Stride can be > the number of rows.

For example, if you want to load a 2x2 sub-matrix from a 4x4 matrix, you would use `llvm.matrix.column.major.load(%start, 4, false, 2, 2), where %start points to the first element of the sub-matrix.

The function to compute column addresses has an extensive comment about how things work: https://github.com/llvm/llvm-project/blob/master/llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp#L92

It boils down to something like: the start address of column I in memory is computed as getelementptr %Start, I * Stride.

[SjoerdMeijer]

llvm/docs/LangRef.rst

15578

Ah yes, thanks, I see now. I will add this, and we have at least one more condition, Stride >= Rows, to add to the verifier.

[SjoerdMeijer]

llvm/docs/LangRef.rst

15578

ignore:

and we have at least one more condition, Stride >= Rows, to add to the verifier.

%Stride is not an immediate.

[fhahn]

llvm/docs/LangRef.rst

15578

yes, the stride can be an arbitrary value. In some (probably most) it will be a ConstantInt, so it might be worth just checking for ConstantInt.

[SjoerdMeijer]

As discussed:

• removed "linearization" and replaced it with the explanation how matrices are laid out in vectors.
• Similarly, spent some words how Stride is used/calculated
• added a check for Stride >= Rows.

[SjoerdMeijer]

Removed unnecessary extra newline.

[fhahn]

LGTM, thanks! Some optional nits related to wording inline (I think it would be good to start the sentences for the arguments with a The).

llvm/docs/LangRef.rst

15502

maybe something like in the corresponding vector instead of in its vector, where it might be a little unclear what its refers to.

15527–15531

The first ..?

15554–15559

The first... and the second ...?

15558

must all have ?

15588–15596

The first...?

15589

The second?

15592

The third

15628–15636

The first argument %In is a vector?

15629

The second argument %Ptr is a pointer to the?

15630

The third?

15633

The fourth?

15634

The arguments?

llvm/lib/IR/Verifier.cpp

5069

It would be good to be consistent with the capitalization/puncation with the existing message at 5073 or update the message there.

Also, it might be good to include vector element type in the message, as in the message for Op0.

[SjoerdMeijer]

Thanks for reviewing, and I will make those changes before committing.

[SjoerdMeijer]

Had to revert this because somehow I missed a few failing regression test. Regarding this, wanted to check one thing @fhahn.

In test/Transforms/LowerMatrixIntrinsics/strided-store-i32.ll, we have for example:

call void @llvm.matrix.column.major.store(<6 x i32> %in, i32* %out, ..

And I am thinking that this should be:

call void @llvm.matrix.column.major.store(<6 x i32> %in, <6 x i32>* %out, ..

This would match the intrinsic description which says that the second argument should be a pointer to the first matched type:

[llvm_anyvector_ty, LLVMAnyPointerType<LLVMMatchType<0>>,

Agree, or am I perhaps missing something? If not and you agree, I will modify the loads/store tests in test/Transforms/LowerMatrixIntrinsics/ before recommitting this.

[fhahn]

In D83477#2146778, @SjoerdMeijer wrote:

Had to revert this because somehow I missed a few failing regression test. Regarding this, wanted to check one thing @fhahn.

In test/Transforms/LowerMatrixIntrinsics/strided-store-i32.ll, we have for example:

call void @llvm.matrix.column.major.store(<6 x i32> %in, i32* %out, ..

And I am thinking that this should be:

call void @llvm.matrix.column.major.store(<6 x i32> %in, <6 x i32>* %out, ..

This would match the intrinsic description which says that the second argument should be a pointer to the first matched type:

[llvm_anyvector_ty, LLVMAnyPointerType<LLVMMatchType<0>>,

Agree, or am I perhaps missing something? If not and you agree, I will modify the loads/store tests in test/Transforms/LowerMatrixIntrinsics/ before recommitting this.

I think the intrinsic definition is wrong here (and it also seems like the LLVMAnyPointerType does not actually result in the expected check). I think should pass a pointer to the element type directly (rather than a pointer to a vector), because if stride > R we would access elements outside of the vector. Granted, nothing should really rely on the pointer type for aliasing purposes and so on, but it seems misleading to pass in e.g. <6 x i32>* and then access elements other than the first 6 i32, e.g. due to the stride being 10.

I missed that in the adjustments of the langref, I think we specify that %Ptr needs to be a pointer to the element type of the vector.

[SjoerdMeijer]

I think the intrinsic definition is wrong here (and it also seems like the LLVMAnyPointerType does not actually result in the expected check). I think should pass a pointer to the element type directly (rather than a pointer to a vector), because if stride > R we would access elements outside of the vector. Granted, nothing should really rely on the pointer type for aliasing purposes and so on, but it seems misleading to pass in e.g. <6 x i32>* and then access elements other than the first 6 i32, e.g. due to the stride being 10.

I missed that in the adjustments of the langref, I think we specify that %Ptr needs to be a pointer to the element type of the vector.

Okay, cool, that's actually what I was expecting.
Just checking that I don't get into your way, shall I prepare a patch for that?

[fhahn]

In D83477#2146889, @SjoerdMeijer wrote:

I think the intrinsic definition is wrong here (and it also seems like the LLVMAnyPointerType does not actually result in the expected check). I think should pass a pointer to the element type directly (rather than a pointer to a vector), because if stride > R we would access elements outside of the vector. Granted, nothing should really rely on the pointer type for aliasing purposes and so on, but it seems misleading to pass in e.g. <6 x i32>* and then access elements other than the first 6 i32, e.g. due to the stride being 10.

I missed that in the adjustments of the langref, I think we specify that %Ptr needs to be a pointer to the element type of the vector.

Okay, cool, that's actually what I was expecting.
Just checking that I don't get into your way, shall I prepare a patch for that?

I don't mind. The current patch is reverted at the moment, right? So it might be easiest to just fold those small changes directly into it? Otherwise I can do it as follow-up once the current patch lands.

[SjoerdMeijer]

I don't mind. The current patch is reverted at the moment, right? So it might be easiest to just fold those small changes directly into it? Otherwise I can do it as follow-up once the current patch lands.

Yep, it's reverted because we have different ptr types (ptr to vector vs. ptr to scalar) in different tests now, and these new Verifier checks don't like that.
As I need to make changes to this patch anyway in order to recommit it, you're right that I can just fold it into this, that's the easiest I guess so will do that then (and will put that up for review).

[fhahn]

In D83477#2146912, @SjoerdMeijer wrote:

I don't mind. The current patch is reverted at the moment, right? So it might be easiest to just fold those small changes directly into it? Otherwise I can do it as follow-up once the current patch lands.

Yep, it's reverted because we have different ptr types (ptr to vector vs. ptr to scalar) in different tests now, and these new Verifier checks don't like that.
As I need to make changes to this patch anyway in order to recommit it, you're right that I can just fold it into this, that's the easiest I guess so will do that then (and will put that up for review).

Great, thanks!

llvm/docs/LangRef.rst

 
 Arguments:
 """"""""""
 The argument to this intrinsic must be a vector of floating-point values.
 
 Matrix Intrinsics
 -----------------
 
 Operations on matrixes requiring shape information (like number of rows/columns
-or the memory layout) can be expressed using the matrix intrinsics. Matrixes are
-embedded in a flat vector and the intrinsics take the dimensions as arguments.
-Currently column-major layout is assumed. The intrinsics support both integer
-and floating point matrixes.
+or the memory layout) can be expressed using the matrix intrinsics. These
+intrinsics require matrix dimensions to be passed as immediate arguments, and
+matrixes are passed and returned as vectors. This means that for a ``R`` x
+``C`` matrix, element ``i`` of column ``j`` is at index ``j * R + i`` in its
+vector, with indices starting at 0. Currently column-major layout is assumed.
+The intrinsics support both integer and floating point matrixes.
 
 
 '``llvm.matrix.transpose.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
+This is an overloaded intrinsic.
 
 ::
 
       declare vectorty @llvm.matrix.transpose.*(vectorty %In, i32 <Rows>, i32 <Cols>)
 
 Overview:
 """""""""
 
-The '``llvm.matrix.transpose.*``' intrinsic treats %In as containing a matrix
-with <Rows> rows and <Cols> columns and returns the transposed matrix embedded in
-the result vector.
+The '``llvm.matrix.transpose.*``' intrinsics treat %In as a <Rows> x <Cols> matrix
+and return the transposed matrix in the result vector.
 
 Arguments:
 """"""""""
 
-The <Rows> and <Cols> arguments must be constant integers. The vector argument
-%In and the returned vector must have <Rows> * <Cols> elements.
+First argument %In is vector that corresponds to a <Rows> x <Cols> matrix.
+Thus, arguments <Rows> and <Cols> correspond to the number of rows and columns,
+respectively, and must be positive, constant integers. The returned vector must
+have <Rows> * <Cols> elements, and have the same float or integer element type
+as %In.
 
 '``llvm.matrix.multiply.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
+This is an overloaded intrinsic.
 
 ::
 
       declare vectorty @llvm.matrix.multiply.*(vectorty %A, vectorty %B, i32 <OuterRows>, i32 <Inner>, i32 <OuterColumns>)
 
 Overview:
 """""""""
 
-The '``llvm.matrix.multiply.*``' intrinsic treats %A as a matrix with <OuterRows>
-rows and <Inner> columns, %B as a matrix with <Inner> rows and <OuterColumns>
-columns and multiplies them. The result matrix is returned embedded in the
-result vector.
+The '``llvm.matrix.multiply.*``' intrinsics treat %A as a <OuterRows> x <Inner>
+matrix, %B as a <Inner> x <OuterColumns> matrix, and multiplies them. The result
+matrix is returned in the result vector.
 
 Arguments:
 """"""""""
 
-The <OuterRows>, <Inner> and <OuterColumns> arguments must be constant
-integers. The vector argument %A must have <OuterRows> * <Inner> elements, %B
-must have <Inner> * <OuterColumns> elements and the returned vector must have
-<OuterRows> * <OuterColumns> elements.
+First vector argument %A corresponds to a matrix with <OuterRows> * <Inner>
+elements, and second argument %B to a matrix with <Inner> * <OuterColumns>
+elements. Arguments <OuterRows>, <Inner> and <OuterColumns> must be positive,
+constant integers. The returned vector must have <OuterRows> * <OuterColumns>
+elements. Vectors %A, %B, and the returned vector all have the same float or
+integer element type.
 
 
 '``llvm.matrix.column.major.load.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
+This is an overloaded intrinsic.
 
 ::
 
       declare vectorty @llvm.matrix.column.major.load.*(
           ptrty %Ptr, i64 %Stride, i1 <IsVolatile>, i32 <Rows>, i32 <Cols>)
 
 Overview:
 """""""""
 
-The '``llvm.matrix.column.major.load.*``' intrinsic loads a matrix with <Rows>
-rows and <Cols> columns, using a stride of %Stride between columns. For two
-consecutive columns A and B, %Stride refers to the distance (the number of
-elements) between the start of column A and the start of column B. The result
-matrix is returned embedded in the result vector. This allows for convenient
-loading of sub matrixes.  If <IsVolatile> is true, the intrinsic is considered
-a :ref:`volatile memory access <volatile>`.
-
-If the %Ptr argument is known to be aligned to some boundary, this can be
-specified as an attribute on the argument.
+The '``llvm.matrix.column.major.load.*``' intrinsics load a <Rows> x <Cols>
+matrix using a stride of %Stride to compute the start address of the different
+columns.  This allows for convenient loading of sub matrixes. If <IsVolatile>
+is true, the intrinsic is considered a :ref:`volatile memory access
+<volatile>`. The result matrix is returned in the result vector. If the %Ptr
+argument is known to be aligned to some boundary, this can be specified as an
+attribute on the argument.
 
 Arguments:
 """"""""""
 
-The <IsVolatile>, <Rows> and <Cols> arguments must be constant integers. The
-returned vector must have <Rows> * <Cols> elements. %Stride must be >= <Rows>.
+First argument %Ptr is a pointer type to the returned vector type, and
+correponds to the start address to load from. Second argument %Stride is a
+postive, constant integer with %Stride ``>=`` <Rows>. %Stride is used to compute
+the column memory addresses. I.e., for a column ``C``, its start memory
+addresses is calculated with %Ptr + ``C`` * %Stride. Third Argument
+<IsVolatile> is a boolean value.  The fourth and fifth arguments, <Rows> and
+<Cols>, correspond to the number of rows and columns, respectively, and must be
+positive, constant integers. The returned vector must have <Rows> * <Cols>
+elements.
 
 The :ref:`align <attr_align>` parameter attribute can be provided
 for the %Ptr arguments.
 
 
 '``llvm.matrix.column.major.store.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
+This is an overloaded intrinsic.
 
 ::
 
       declare void @llvm.matrix.column.major.store.*(
           vectorty %In, ptrty %Ptr, i64 %Stride, i1 <IsVolatile>, i32 <Rows>, i32 <Cols>)
 
 Overview:
 """""""""
 
-The '``llvm.matrix.column.major.store.*``' intrinsic stores the matrix with
-<Rows> rows and <Cols> columns embedded in %In, using a stride of %Stride
-between columns. For two consecutive columns A and B, %Stride refers to the
-distance (the number of elements) between the start of column A and the start
-of column B. If <IsVolatile> is true, the intrinsic is considered a
-:ref:`volatile memory access <volatile>`.
+The '``llvm.matrix.column.major.store.*``' intrinsics store the <Rows> x <Cols>
+matrix in %In to memory using a stride of %Stride between columns. If
+<IsVolatile> is true, the intrinsic is considered a :ref:`volatile memory
+access <volatile>`.
 
 If the %Ptr argument is known to be aligned to some boundary, this can be
 specified as an attribute on the argument.
 
 Arguments:
 """"""""""
 
-The <IsVolatile>, <Rows>, <Cols> arguments must be constant integers. The
-vector argument %In must have <Rows> * <Cols> elements. %Stride must be >= <Rows>.
+First argument %In is vector that corresponds to a <Rows> x <Cols> matrix to be
+stored to memory. Second argument %Ptr is a pointer type to the vector type of
+%In, and is the start address of the matrix in memory.  Third argument %Stride
+is a positive, constant integer with %Stride ``>=`` <Rows>. %Stride is used to
+compute the column memory addresses. I.e., for a column ``C``, its start memory
+addresses is calculated with %Ptr + ``C`` * %Stride. Fourth argument
+<IsVolatile> is a boolean value. Arguments <Rows> and <Cols> correspond to the
+number of rows and columns, respectively, and must be positive, constant
+integers.
 
 The :ref:`align <attr_align>` parameter attribute can be provided
 for the %Ptr arguments.
 
 
 Half Precision Floating-Point Intrinsics
 ----------------------------------------
 

llvm/lib/IR/Verifier.cpp

case Intrinsic::bswap: {
     unsigned Size = Ty->getScalarSizeInBits();
     Assert(Size % 16 == 0, "bswap must be an even number of bytes", &Call);
     break;
   }
   case Intrinsic::matrix_multiply:
   case Intrinsic::matrix_transpose:
   case Intrinsic::matrix_column_major_load:
   case Intrinsic::matrix_column_major_store: {
+    Function *IF = Call.getCalledFunction();
+    ConstantInt *Stride = nullptr;
     ConstantInt *NumRows;
     ConstantInt *NumColumns;
-    VectorType *TypeToCheck;
+    VectorType *ResultTy;
+    Type *Op0ElemTy = nullptr;
+    Type *Op1ElemTy = nullptr;
     switch (ID) {
     case Intrinsic::matrix_multiply:
       NumRows = cast<ConstantInt>(Call.getArgOperand(2));
       NumColumns = cast<ConstantInt>(Call.getArgOperand(4));
-      TypeToCheck = cast<VectorType>(Call.getType());
+      ResultTy = cast<VectorType>(Call.getType());
+      Op0ElemTy =
+          cast<VectorType>(Call.getArgOperand(0)->getType())->getElementType();
+      Op1ElemTy =
+          cast<VectorType>(Call.getArgOperand(1)->getType())->getElementType();
       break;
     case Intrinsic::matrix_transpose:
       NumRows = cast<ConstantInt>(Call.getArgOperand(1));
       NumColumns = cast<ConstantInt>(Call.getArgOperand(2));
-      TypeToCheck = cast<VectorType>(Call.getType());
+      ResultTy = cast<VectorType>(Call.getType());
+      Op0ElemTy =
+          cast<VectorType>(Call.getArgOperand(0)->getType())->getElementType();
       break;
-    case Intrinsic::matrix_column_major_load:
+    case Intrinsic::matrix_column_major_load: {
+      Stride = dyn_cast<ConstantInt>(Call.getArgOperand(1));
       NumRows = cast<ConstantInt>(Call.getArgOperand(3));
       NumColumns = cast<ConstantInt>(Call.getArgOperand(4));
-      TypeToCheck = cast<VectorType>(Call.getType());
+      ResultTy = cast<VectorType>(Call.getType());
+      auto *VecTy = cast<VectorType>(
+          cast<PointerType>(Call.getArgOperand(0)->getType())->getElementType());
+      Op0ElemTy = VecTy->getElementType();
+      }
       break;
-    case Intrinsic::matrix_column_major_store:
+    case Intrinsic::matrix_column_major_store: {
+      Stride = dyn_cast<ConstantInt>(Call.getArgOperand(2));
       NumRows = cast<ConstantInt>(Call.getArgOperand(4));
       NumColumns = cast<ConstantInt>(Call.getArgOperand(5));
-      TypeToCheck = cast<VectorType>(Call.getArgOperand(0)->getType());
+      ResultTy = cast<VectorType>(Call.getArgOperand(0)->getType());
+      Op0ElemTy =
+          cast<VectorType>(Call.getArgOperand(0)->getType())->getElementType();
+      auto *VecTy = cast<VectorType>(
+          cast<PointerType>(Call.getArgOperand(1)->getType())->getElementType());
+      Op1ElemTy = VecTy->getElementType();
+      }
       break;
     default:
       llvm_unreachable("unexpected intrinsic");
     }
-    Assert(TypeToCheck->getNumElements() ==
+
+    Assert(ResultTy->getElementType()->isIntegerTy() ||
+           ResultTy->getElementType()->isFloatingPointTy(),
+           "Result type must be an integer or floating-point type!", IF);
+
+
+    Assert(ResultTy->getElementType() == Op0ElemTy,
+           "Vector element type mismatch of the result and first operand "
+           "vector!", IF);
+
+    if (Op1ElemTy)
+      Assert(ResultTy->getElementType() == Op1ElemTy,
+             "Type mismatch of the result and second operand vector!", IF);
+
+    Assert(ResultTy->getNumElements() ==
                NumRows->getZExtValue() * NumColumns->getZExtValue(),
            "result of a matrix operation does not fit in the returned vector");
+
+    if (Stride)
+      Assert(Stride->getZExtValue() >= NumRows->getZExtValue(),
+             "Stride must be greater or equal than the number of rows!", IF);
+
     break;
   }
   };
 }
 
 /// Carefully grab the subprogram from a local scope.
 ///
 /// This carefully grabs the subprogram from a local scope, avoiding the

llvm/test/Verifier/matrix-intrinsics.ll

 ; CHECK-NEXT: result of a matrix operation does not fit in the returned vector
 ; CHECK-NEXT: result of a matrix operation does not fit in the returned vector
   call void @llvm.matrix.column.major.store.v4f32.p0v4f32(<4 x float> zeroinitializer, <4 x float>* %m, i64 0, i1 false, i32 0, i32 0)
   call void @llvm.matrix.column.major.store.v4f32.p0v4f32(<4 x float> zeroinitializer, <4 x float>* %m, i64 2, i1 false, i32 1, i32 2)
   call void @llvm.matrix.column.major.store.v6f32.p0v6f32(<6 x float> zeroinitializer, <6 x float>* %n, i64 2, i1 false, i32 3, i32 3)
   call void @llvm.matrix.column.major.store.v6f32.p0v6f32(<6 x float> zeroinitializer, <6 x float>* %n, i64 %arg, i1 false, i32 3, i32 3)
   ret void
 }
+
+declare <4 x float> @llvm.matrix.transpose.v4f32.v4i32(<4 x i32>, i32, i32)
+declare <4 x i32> @llvm.matrix.transpose.v4i32.v4f32(<4 x float>, i32, i32)
+
+define <4 x float> @transpose_mixed_types(<4 x float> %fvec, <4 x i32> %ivec, i32 %arg) {
+;
+; CHECK-NEXT: Intrinsic has incorrect argument type!
+; CHECK-NEXT: <4 x float> (<4 x i32>, i32, i32)* @llvm.matrix.transpose.v4f32.v4i32
+; CHECK-NEXT: Intrinsic has incorrect argument type!
+; CHECK-NEXT: <4 x i32> (<4 x float>, i32, i32)* @llvm.matrix.transpose.v4i32.v4f32
+;
+  %result.0 = call <4 x float> @llvm.matrix.transpose.v4f32.v4i32(<4 x i32> %ivec, i32 0, i32 0)
+  %result.1 = call <4 x i32> @llvm.matrix.transpose.v4i32.v4f32(<4 x float> %result.0, i32 3, i32 2)
+  ret <4 x float> %result.0
+}
+
+declare <4 x i32>   @llvm.matrix.multiply.v4i32.v4f32.v4f32(<4 x float>, <4 x float>, i32, i32, i32)
+declare <4 x float> @llvm.matrix.multiply.v4f32.v4i32.v4f32(<4 x i32>, <4 x float>, i32, i32, i32)
+declare <4 x float> @llvm.matrix.multiply.v4f32.v4f32.v4i32(<4 x float>, <4 x i32>, i32, i32, i32)
+declare <4 x float> @llvm.matrix.multiply.v4f32.v4i32.v4i32(<4 x i32>, <4 x i32>, i32, i32, i32)
+
+define <4 x float> @multiply_mixed_types(<4 x i32> %ivec, <4 x float> %fvec, i32 %arg) {
+;
+; CHECK-NEXT: Vector element type mismatch of the result and first operand vector!
+; CHECK-NEXT: <4 x i32> (<4 x float>, <4 x float>, i32, i32, i32)* @llvm.matrix.multiply.v4i32.v4f32.v4f32
+; CHECK-NEXT: Vector element type mismatch of the result and first operand vector!
+; CHECK-NEXT: <4 x float> (<4 x i32>, <4 x float>, i32, i32, i32)* @llvm.matrix.multiply.v4f32.v4i32.v4f32
+; CHECK-NEXT: Type mismatch of the result and second operand vector!
+; CHECK-NEXT: <4 x float> (<4 x float>, <4 x i32>, i32, i32, i32)* @llvm.matrix.multiply.v4f32.v4f32.v4i32
+; CHECK-NEXT: Vector element type mismatch of the result and first operand vector!
+; CHECK-NEXT: <4 x float> (<4 x i32>, <4 x i32>, i32, i32, i32)* @llvm.matrix.multiply.v4f32.v4i32.v4i32
+;
+  %result.0 = call <4 x i32> @llvm.matrix.multiply.v4i32.v4f32.v4f32(<4 x float> %fvec, <4 x float> %fvec, i32 2, i32 2, i32 2)
+  %result.1 = call <4 x float> @llvm.matrix.multiply.v4f32.v4i32.v4f32(<4 x i32> %result.0, <4 x float> %fvec, i32 2, i32 2, i32 2)
+  %result.2 = call <4 x float> @llvm.matrix.multiply.v4f32.v4f32.v4i32(<4 x float> %fvec, <4 x i32> %ivec, i32 2, i32 2, i32 2)
+  %result.3 = call <4 x float> @llvm.matrix.multiply.v4f32.v4i32.v4i32(<4 x i32> %ivec, <4 x i32> %ivec, i32 2, i32 2, i32 2)
+  ret <4 x float> %result.3
+}
+
+declare <4 x float> @llvm.matrix.column.major.load.v4f32.p0v4i32(<4 x i32>*, i64, i1, i32, i32)
+declare <4 x i32> @llvm.matrix.column.major.load.v4i32.p0v4f32(<4 x float>*, i64, i1, i32, i32)
+
+define <4 x float> @column.major_load_mixed_types(<4 x i32>* %m, <4 x float>* %n, i32 %arg) {
+;
+; CHECK-NEXT: Vector element type mismatch of the result and first operand vector!
+; CHECK-NEXT: <4 x float> (<4 x i32>*, i64, i1, i32, i32)* @llvm.matrix.column.major.load.v4f32.p0v4i32
+; CHECK-NEXT: Vector element type mismatch of the result and first operand vector!
+; CHECK-NEXT: <4 x i32> (<4 x float>*, i64, i1, i32, i32)* @llvm.matrix.column.major.load.v4i32.p0v4f32
+;
+  %result.0 = call <4 x float> @llvm.matrix.column.major.load.v4f32.p0v4i32(<4 x i32>* %m, i64 2, i1 false, i32 2, i32 2)
+  %result.1 = call <4 x i32> @llvm.matrix.column.major.load.v4i32.p0v4f32(<4 x float>* %n, i64 2, i1 false, i32 2, i32 2)
+  ret <4 x float> %result.0
+}
+
+declare void @llvm.matrix.column.major.store.v4i32.p0v4f32(<4 x i32>, <4 x float>*, i64, i1, i32, i32)
+declare void @llvm.matrix.column.major.store.v4f32.p0v4i32(<4 x float>, <4 x i32>*, i64, i1, i32, i32)
+
+define void @column.major_store_mixed_types(<4 x float>* %m, <4 x i32>* %n, i64 %arg) {
+;
+; CHECK-NEXT: Type mismatch of the result and second operand vector!
+; CHECK-NEXT: void (<4 x i32>, <4 x float>*, i64, i1, i32, i32)* @llvm.matrix.column.major.store.v4i32.p0v4f32
+; CHECK-NEXT: Type mismatch of the result and second operand vector!
+; CHECK-NEXT: void (<4 x float>, <4 x i32>*, i64, i1, i32, i32)* @llvm.matrix.column.major.store.v4f32.p0v4i32
+;
+  call void @llvm.matrix.column.major.store.v4i32.p0v4f32(<4 x i32> zeroinitializer, <4 x float>* %m, i64 2, i1 false, i32 2, i32 2)
+  call void @llvm.matrix.column.major.store.v4f32.p0v4i32(<4 x float> zeroinitializer, <4 x i32>* %n, i64 2, i1 false, i32 2, i32 2)
+  ret void
+}
+
+declare void @llvm.matrix.column.major.store.v4f32p0.p0v4f32(<4 x float*>, <4 x float>*, i64, i1, i32, i32)
+
+define void @column.major_store_non_int_float_type(<4 x float>* %m, <4 x float>* %n, i64 %arg) {
+;
+; CHECK-NEXT: Result type must be an integer or floating-point type!
+; CHECK-NEXT: void (<4 x float*>, <4 x float>*, i64, i1, i32, i32)* @llvm.matrix.column.major.store.v4p0f32.p0v4f32
+;
+  call void @llvm.matrix.column.major.store.v4f32p0.p0v4f32(<4 x float*> zeroinitializer, <4 x float>* %n, i64 2, i1 false, i32 2, i32 2)
+  ret void
+}
+
+define <4 x float> @column.major_load_stride_too_small(<4 x float>* %m, i32 %arg) {
+;
+; CHECK-NEXT: Stride must be greater or equal than the number of rows!
+; CHECK-NEXT: <4 x float> (<4 x float>*, i64, i1, i32, i32)* @llvm.matrix.column.major.load.v4f32.p0v4f32
+;
+  %result.1 = call <4 x float> @llvm.matrix.column.major.load.v4f32.p0v4f32(<4 x float>* %m, i64 1, i1 false, i32 2, i32 2)
+  ret <4 x float> %result.1
+}
+
+define void @column.major_store_stride_too_small(<4 x float>* %m, i64 %arg) {
+;
+; CHECK-NEXT: Stride must be greater or equal than the number of rows!
+; CHECK-NEXT: void (<4 x float>, <4 x float>*, i64, i1, i32, i32)* @llvm.matrix.column.major.store.v4f32.p0v4f32
+;
+  call void @llvm.matrix.column.major.store.v4f32.p0v4f32(<4 x float> zeroinitializer, <4 x float>* %m, i64 1, i1 false, i32 2, i32 2)
+  ret void
+}
+
+