[Matrix] Add special case dot product lowering

[virnarula]

Summary

Add special case to matrix lowering for dot products. Normal matrix lowering if optimized for either row-major or column-major, which results in many shufflevector instructions being generated for one vector. We work around this in our special case. We can also use vector-reduce adds instead of sequential adds to sum the result of the element-wise multiplication, which takes advantage of SIMD instructions.

[virnarula]

Fix formatting with clang-format

[fhahn]

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1000

I don't think this is necessary and it looks a bit odd to overwrite the earlier Changed status. It should be sufficient to remove setting Changed on line 877.

1321

Please add a comment describing what this function does.

1324

I think we should be able to just look up the shape of operands 0 and 1 in ShapeMap instead?

1327

nit: coding style recommends omitting {} for single-line blocks.

1336

nit: Coding style uses UpperCase for variables.

1356

might be good to explain what costs we are comparing.

1359

nit: the style guide recommends avoiding auto in most cases where the type is not entirely obvious from the context (e.g. auto for dyn_cast is fine).

1369

nit: use UpperCase for variables.

1371

To check for a specific intrinsic, you could either use something like match(&I, m_Intrinsic<Intrinsic::matrix_transpose>(m_Value(TA))) or cast to IntrinsicInst and checking getIntrinsicID().

1399

I am not sure if that's correct. Here we need to load a vector with 1 row and N columns. llvm.matrix.column.load may have a stride > 1 between columns, which is ignored. We should probably limit this to .llvm.matrix.column.load with a stride of 1 here for now. Needs a test case.

1424

nit: according to the style guide, comments are meant to be full sentences, with the first letter capitalized and a period at the end. Applies to most comments added in the patch.

1430

nit: no need for {}

1433

nit: no need for {}

1437

nit: no need to return at the end of a void function.

[thegameg]

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1347

You can probably remove this and use IRBuilder::Create(F)AddReduce directly below?

1351

Maybe we can do all the FMF and cost checks before everything?

1379

You might want to specify what you mean by "differently" (not using LowerLoad because we would end up with scalar loads for the row vector).

1381

To add on top of what @fhahn said above, you can combine matchers like this:

match(&I, m_CombineOr(m_Load(), m_Intrinsic<Intrinsic::matrix_column_major_load>())

llvm/test/Transforms/LowerMatrixIntrinsics/dot-product.ll

165 ↗

(On Diff #449816)

Very nice!

[virnarula]

Address first round of comments.

[virnarula]

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1324

Using ShapeMap works when we load matrix inside the function. However, it doesn't if the matrix is passed in as an argument. As such, I have not implemented this.

1399

Most of our test cases didn't account for this (they used strides greater than 1). I am submitting an additional review that will update the test cases to use strides of 1 but keep some that don't as test cases.

1399

https://reviews.llvm.org/D131444

[thegameg]

LGTM, thank you!

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1344

Not sure what's going on here. Intrinsic with Instruction::Add? It also doesn't seem to be really used.

1360

You cold match the stride directly here:

ConstantInt *LoadStride = nullptr;
if (match(Val, m_Intrinsic<Intrinsic::matrix_column_major_load>(
                       m_Value(), m_ConstantInt(LoadStride), m_Value(), m_Value()) &&
    LoadStride->getZExtValue() == Stride) {

and pass Stride as an argument of GetBuiltinLoad.

[virnarula]

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1344

I use it to get the function type when estimating sequential add cost. line 1243. Do you suggest doing it a different way?

1360

I tried this but I realized that I would have to have a control flow to check if the load has size of n anyways. This is because if we return null, it could mean there's a LoadInst (instead of a builtin load) or the stride in incorrect, which require different behaviors.

[thegameg]

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

1344

It looks like you only use AddOpCode and Add is only used for the type, right?

[fhahn]

Rebased. I also extended int coverage, so we have enough coverage to land this and build in it.

[fhahn]

LGTM, thanks!

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

for (auto *BB : RPOT)
           MaybeFusableInsts.push_back(cast<CallInst>(&I));
         MatrixInsts.push_back(&I);
       }
 
     // Second, try to fuse candidates.
     SmallPtrSet<Instruction *, 16> FusedInsts;
     for (CallInst *CI : MaybeFusableInsts)
       LowerMatrixMultiplyFused(CI, FusedInsts);
+
+    // Third, try to lower any dot products
+    for (CallInst *CI : MaybeFusableInsts) {
+      if (FusedInsts.find(CI) != FusedInsts.end()) // skip if already fused
+        continue;
+      lowerDotProduct(CI, FusedInsts, getFastMathFlags(CI));
+    }
     Changed = !FusedInsts.empty();
 
-    // Third, lower remaining instructions with shape information.
+    // Fourth, lower remaining instructions with shape information.
     for (Instruction *Inst : MatrixInsts) {
       if (FusedInsts.count(Inst))
         continue;
 
       IRBuilder<> Builder(Inst);
 
       if (CallInst *CInst = dyn_cast<CallInst>(Inst))
         Changed |= VisitCallInst(CInst);
 
       Value *Op1;
       Value *Op2;
       if (auto *BinOp = dyn_cast<BinaryOperator>(Inst))
         Changed |= VisitBinaryOperator(BinOp);
       if (auto *UnOp = dyn_cast<UnaryOperator>(Inst))
for (Use &U : llvm::make_early_inc_range(Inst->uses())) {
       if (ShapeMap.find(U.getUser()) == ShapeMap.end()) {
         if (!Flattened)
           Flattened = Matrix.embedInVector(Builder);
         U.set(Flattened);
       }
     }
   }
 
+  /// Special case for MatMul lowering. Prevents scalar loads of row-major
+  /// vectors Lowers to vector reduction add instead of sequential add if
+  /// reassocation is enabled.
+  void lowerDotProduct(CallInst *MatMul,
+                       SmallPtrSet<Instruction *, 16> &FusedInsts,
+                       FastMathFlags FMF) {
+    ShapeInfo LShape(MatMul->getArgOperand(2), MatMul->getArgOperand(3));
+    ShapeInfo RShape(MatMul->getArgOperand(3), MatMul->getArgOperand(4));
+
+    if (LShape.NumRows != 1 || RShape.NumColumns != 1) // not a dot product
+      return;
+
+    Value *LHS = MatMul->getArgOperand(0);
+    Value *RHS = MatMul->getArgOperand(1);
+
+    Type *ElementType = cast<VectorType>(LHS->getType())->getElementType();
+    bool IsIntVec = ElementType->isIntegerTy();
+
+    // Floating point reductions require reassocation.
+    if (!IsIntVec && !FMF.allowReassoc())
+      return;
+
+    auto IsSupportedArg = [](Value *Op, unsigned N) {
+      if (!isa<Instruction>(Op))
+        return true;
+      return match(Op, m_OneUse(m_CombineOr(
+                           m_Load(m_Value()),
+                           m_Intrinsic<Intrinsic::matrix_column_major_load>(
+                               m_Value(), m_SpecificInt(N)))));
+    };
+    if (!IsSupportedArg(RHS, RShape.NumColumns) ||
+        !IsSupportedArg(LHS, LShape.NumColumns))
+      return;
+
+    // We compare the costs of a vector.reduce.add to sequential add.
+    int AddOpCode = IsIntVec ? Instruction::Add : Instruction::FAdd;
+    FastMathFlags FMFReassoc;
+    FMFReassoc.setAllowReassoc();
+    InstructionCost ReductionCost = TTI.getArithmeticReductionCost(
+        AddOpCode, cast<VectorType>(LHS->getType()), FMFReassoc);
+    InstructionCost SequentialAddCost =
+        TTI.getArithmeticInstrCost(AddOpCode, ElementType) *
+        (LShape.NumColumns - 1);
+    if (ReductionCost >= SequentialAddCost)
+      return;
+
+    FusedInsts.insert(MatMul);
+    IRBuilder<> Builder(MatMul);
+    auto FlattenArg = [&Builder, &FusedInsts](Value *Op) -> Value * {
+      // Matmul must be the only user of loads because we don't use LowerLoad
+      // for row vectors (LowerLoad results in scalar loads and shufflevectors
+      // instead of single vector load).
+      if (!match(Op, m_CombineOr(
+                         m_Load(m_Value()),
+                         m_Intrinsic<Intrinsic::matrix_column_major_load>()))) {
+        return Op;
+      }
+      FusedInsts.insert(cast<Instruction>(Op));
+
+      // If vector uses the builtin load, lower to a LoadInst
+      Value *Ptr;
+      if (match(Op, m_Intrinsic<Intrinsic::matrix_column_major_load>(
+                        m_Value(Ptr)))) {
+        auto *NewLoad = Builder.CreateLoad(Op->getType(), Ptr);
+        Op->replaceAllUsesWith(NewLoad);
+        cast<Instruction>(Op)->eraseFromParent();
+        return NewLoad;
+      }
+      return Op;
+    };
+    LHS = FlattenArg(LHS);
+    RHS = FlattenArg(RHS);
+
+    // Insert mul/fmul and llvm.vector.reduce.fadd
+    Value *Mul =
+        IsIntVec ? Builder.CreateMul(LHS, RHS) : Builder.CreateFMul(LHS, RHS);
+
+    Value *Result;
+    if (IsIntVec)
+      Result = Builder.CreateAddReduce(Mul);
+    else {
+      Result = Builder.CreateFAddReduce(
+          ConstantFP::get(cast<VectorType>(LHS->getType())->getElementType(),
+                          0.0),
+          Mul);
+      cast<Instruction>(Result)->setFastMathFlags(FMF);
+    }
+
+    // pack scalar back into a matrix and then replace matmul inst
+    Result = Builder.CreateInsertElement(PoisonValue::get(MatMul->getType()),
+                                         Result, uint64_t(0));
+    MatMul->replaceAllUsesWith(Result);
+    MatMul->eraseFromParent();
+  }
+
   /// Compute \p Result += \p A * \p B for input matrices with left-associating
   /// addition.
   ///
   /// We can fold a transpose into the operand that is used to extract scalars.
   /// This is the first operands with row-major and the second with
   /// column-major.  If \p IsScalarMatrixTransposed we assume the appropriate
   /// operand is transposed.
   void emitMatrixMultiply(MatrixTy &Result, const MatrixTy &A,
                           const MatrixTy &B, IRBuilder<> &Builder, bool IsTiled,
                           bool IsScalarMatrixTransposed, FastMathFlags FMF) {
     const unsigned VF = std::max<unsigned>(
         TTI.getRegisterBitWidth(TargetTransformInfo::RGK_FixedWidthVector)
                 .getFixedValue() /
             Result.getElementType()->getPrimitiveSizeInBits().getFixedValue(),
         1U);
     unsigned R = Result.getNumRows();
     unsigned C = Result.getNumColumns();
     unsigned M = A.getNumColumns();
 
     bool IsFP = Result.getElementType()->isFloatingPointTy();
     assert(A.isColumnMajor() == B.isColumnMajor() &&
            Result.isColumnMajor() == A.isColumnMajor() &&
            "operands must agree on matrix layout");
     unsigned NumComputeOps = 0;
 
     Builder.setFastMathFlags(FMF);
 
     if (A.isColumnMajor()) {
       // Multiply columns from the first operand with scalars from the second
       // operand. Then move along the K axes and accumulate the columns.  With

llvm/test/Transforms/LowerMatrixIntrinsics/dot-product-int.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; REQUIRES: aarch64-registered-target
 ; RUN: opt -passes='lower-matrix-intrinsics' -mtriple=arm64-apple-iphoneos -S < %s | FileCheck %s
 
 define <1 x i32> @dotproduct_i32_v8(<8 x i32> %a, <8 x i32> %b) {
 ; CHECK-LABEL: @dotproduct_i32_v8(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPLIT:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[SPLIT1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 1>
-; CHECK-NEXT:    [[SPLIT2:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 2>
-; CHECK-NEXT:    [[SPLIT3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 3>
-; CHECK-NEXT:    [[SPLIT4:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 4>
-; CHECK-NEXT:    [[SPLIT5:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 5>
-; CHECK-NEXT:    [[SPLIT6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 6>
-; CHECK-NEXT:    [[SPLIT7:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 7>
-; CHECK-NEXT:    [[SPLIT8:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i32> [[SPLIT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT:%.*]] = insertelement <1 x i32> poison, i32 [[TMP0]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = mul <1 x i32> [[BLOCK]], [[SPLAT_SPLAT]]
-; CHECK-NEXT:    [[BLOCK9:%.*]] = shufflevector <1 x i32> [[SPLIT1]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 1
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT10:%.*]] = insertelement <1 x i32> poison, i32 [[TMP2]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT11:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT10]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <1 x i32> [[BLOCK9]], [[SPLAT_SPLAT11]]
-; CHECK-NEXT:    [[TMP4:%.*]] = add <1 x i32> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[BLOCK12:%.*]] = shufflevector <1 x i32> [[SPLIT2]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 2
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT13:%.*]] = insertelement <1 x i32> poison, i32 [[TMP5]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT14:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT13]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <1 x i32> [[BLOCK12]], [[SPLAT_SPLAT14]]
-; CHECK-NEXT:    [[TMP7:%.*]] = add <1 x i32> [[TMP4]], [[TMP6]]
-; CHECK-NEXT:    [[BLOCK15:%.*]] = shufflevector <1 x i32> [[SPLIT3]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 3
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT16:%.*]] = insertelement <1 x i32> poison, i32 [[TMP8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT17:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT16]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = mul <1 x i32> [[BLOCK15]], [[SPLAT_SPLAT17]]
-; CHECK-NEXT:    [[TMP10:%.*]] = add <1 x i32> [[TMP7]], [[TMP9]]
-; CHECK-NEXT:    [[BLOCK18:%.*]] = shufflevector <1 x i32> [[SPLIT4]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 4
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT19:%.*]] = insertelement <1 x i32> poison, i32 [[TMP11]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT20:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT19]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = mul <1 x i32> [[BLOCK18]], [[SPLAT_SPLAT20]]
-; CHECK-NEXT:    [[TMP13:%.*]] = add <1 x i32> [[TMP10]], [[TMP12]]
-; CHECK-NEXT:    [[BLOCK21:%.*]] = shufflevector <1 x i32> [[SPLIT5]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 5
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT22:%.*]] = insertelement <1 x i32> poison, i32 [[TMP14]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT23:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT22]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP15:%.*]] = mul <1 x i32> [[BLOCK21]], [[SPLAT_SPLAT23]]
-; CHECK-NEXT:    [[TMP16:%.*]] = add <1 x i32> [[TMP13]], [[TMP15]]
-; CHECK-NEXT:    [[BLOCK24:%.*]] = shufflevector <1 x i32> [[SPLIT6]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 6
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT25:%.*]] = insertelement <1 x i32> poison, i32 [[TMP17]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT26:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT25]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP18:%.*]] = mul <1 x i32> [[BLOCK24]], [[SPLAT_SPLAT26]]
-; CHECK-NEXT:    [[TMP19:%.*]] = add <1 x i32> [[TMP16]], [[TMP18]]
-; CHECK-NEXT:    [[BLOCK27:%.*]] = shufflevector <1 x i32> [[SPLIT7]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP20:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 7
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT28:%.*]] = insertelement <1 x i32> poison, i32 [[TMP20]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT29:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT28]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP21:%.*]] = mul <1 x i32> [[BLOCK27]], [[SPLAT_SPLAT29]]
-; CHECK-NEXT:    [[TMP22:%.*]] = add <1 x i32> [[TMP19]], [[TMP21]]
-; CHECK-NEXT:    [[TMP23:%.*]] = shufflevector <1 x i32> [[TMP22]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP24:%.*]] = shufflevector <1 x i32> undef, <1 x i32> [[TMP23]], <1 x i32> <i32 1>
-; CHECK-NEXT:    ret <1 x i32> [[TMP24]]
+; CHECK-NEXT:    [[TMP0:%.*]] = mul <8 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> [[TMP0]])
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <1 x i32> poison, i32 [[TMP1]], i64 0
+; CHECK-NEXT:    ret <1 x i32> [[TMP2]]
 ;
 entry:
   %c = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32> %a, <8 x i32> %b, i32 1, i32 8, i32 1)
   ret <1 x i32> %c
 }
 
 declare <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32>, <8 x i32>, i32, i32, i32)
 
 declare void @use(<1 x i32>)
 
 define <1 x i32> @dotproduct_i32_v8_result_used_by_inst(<8 x i32> %a, <8 x i32> %b) {
 ; CHECK-LABEL: @dotproduct_i32_v8_result_used_by_inst(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPLIT:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[SPLIT1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 1>
-; CHECK-NEXT:    [[SPLIT2:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 2>
-; CHECK-NEXT:    [[SPLIT3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 3>
-; CHECK-NEXT:    [[SPLIT4:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 4>
-; CHECK-NEXT:    [[SPLIT5:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 5>
-; CHECK-NEXT:    [[SPLIT6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 6>
-; CHECK-NEXT:    [[SPLIT7:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 7>
-; CHECK-NEXT:    [[SPLIT8:%.*]] = shufflevector <8 x i32> [[B:%.*]], <8 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i32> [[SPLIT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT:%.*]] = insertelement <1 x i32> poison, i32 [[TMP0]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = mul <1 x i32> [[BLOCK]], [[SPLAT_SPLAT]]
-; CHECK-NEXT:    [[BLOCK9:%.*]] = shufflevector <1 x i32> [[SPLIT1]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 1
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT10:%.*]] = insertelement <1 x i32> poison, i32 [[TMP2]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT11:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT10]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <1 x i32> [[BLOCK9]], [[SPLAT_SPLAT11]]
-; CHECK-NEXT:    [[TMP4:%.*]] = add <1 x i32> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[BLOCK12:%.*]] = shufflevector <1 x i32> [[SPLIT2]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 2
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT13:%.*]] = insertelement <1 x i32> poison, i32 [[TMP5]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT14:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT13]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <1 x i32> [[BLOCK12]], [[SPLAT_SPLAT14]]
-; CHECK-NEXT:    [[TMP7:%.*]] = add <1 x i32> [[TMP4]], [[TMP6]]
-; CHECK-NEXT:    [[BLOCK15:%.*]] = shufflevector <1 x i32> [[SPLIT3]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 3
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT16:%.*]] = insertelement <1 x i32> poison, i32 [[TMP8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT17:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT16]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = mul <1 x i32> [[BLOCK15]], [[SPLAT_SPLAT17]]
-; CHECK-NEXT:    [[TMP10:%.*]] = add <1 x i32> [[TMP7]], [[TMP9]]
-; CHECK-NEXT:    [[BLOCK18:%.*]] = shufflevector <1 x i32> [[SPLIT4]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 4
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT19:%.*]] = insertelement <1 x i32> poison, i32 [[TMP11]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT20:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT19]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = mul <1 x i32> [[BLOCK18]], [[SPLAT_SPLAT20]]
-; CHECK-NEXT:    [[TMP13:%.*]] = add <1 x i32> [[TMP10]], [[TMP12]]
-; CHECK-NEXT:    [[BLOCK21:%.*]] = shufflevector <1 x i32> [[SPLIT5]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 5
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT22:%.*]] = insertelement <1 x i32> poison, i32 [[TMP14]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT23:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT22]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP15:%.*]] = mul <1 x i32> [[BLOCK21]], [[SPLAT_SPLAT23]]
-; CHECK-NEXT:    [[TMP16:%.*]] = add <1 x i32> [[TMP13]], [[TMP15]]
-; CHECK-NEXT:    [[BLOCK24:%.*]] = shufflevector <1 x i32> [[SPLIT6]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 6
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT25:%.*]] = insertelement <1 x i32> poison, i32 [[TMP17]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT26:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT25]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP18:%.*]] = mul <1 x i32> [[BLOCK24]], [[SPLAT_SPLAT26]]
-; CHECK-NEXT:    [[TMP19:%.*]] = add <1 x i32> [[TMP16]], [[TMP18]]
-; CHECK-NEXT:    [[BLOCK27:%.*]] = shufflevector <1 x i32> [[SPLIT7]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP20:%.*]] = extractelement <8 x i32> [[SPLIT8]], i64 7
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT28:%.*]] = insertelement <1 x i32> poison, i32 [[TMP20]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT29:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT28]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP21:%.*]] = mul <1 x i32> [[BLOCK27]], [[SPLAT_SPLAT29]]
-; CHECK-NEXT:    [[TMP22:%.*]] = add <1 x i32> [[TMP19]], [[TMP21]]
-; CHECK-NEXT:    [[TMP23:%.*]] = shufflevector <1 x i32> [[TMP22]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP24:%.*]] = shufflevector <1 x i32> undef, <1 x i32> [[TMP23]], <1 x i32> <i32 1>
-; CHECK-NEXT:    call void @use(<1 x i32> [[TMP24]])
-; CHECK-NEXT:    ret <1 x i32> [[TMP24]]
+; CHECK-NEXT:    [[TMP0:%.*]] = mul <8 x i32> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> [[TMP0]])
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <1 x i32> poison, i32 [[TMP1]], i64 0
+; CHECK-NEXT:    call void @use(<1 x i32> [[TMP2]])
+; CHECK-NEXT:    ret <1 x i32> [[TMP2]]
 ;
 entry:
   %c = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32> %a, <8 x i32> %b, i32 1, i32 8, i32 1)
   call void @use(<1 x i32> %c)
   ret <1 x i32> %c
 }
 
 
 define <1 x i32> @dotproduct_i32_v8_constvector(<8 x i32> %a) {
 ; CHECK-LABEL: @dotproduct_i32_v8_constvector(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPLIT:%.*]] = shufflevector <8 x i32> [[A:%.*]], <8 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[SPLIT1:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 1>
-; CHECK-NEXT:    [[SPLIT2:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 2>
-; CHECK-NEXT:    [[SPLIT3:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 3>
-; CHECK-NEXT:    [[SPLIT4:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 4>
-; CHECK-NEXT:    [[SPLIT5:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 5>
-; CHECK-NEXT:    [[SPLIT6:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 6>
-; CHECK-NEXT:    [[SPLIT7:%.*]] = shufflevector <8 x i32> [[A]], <8 x i32> poison, <1 x i32> <i32 7>
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i32> [[SPLIT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = mul <1 x i32> [[BLOCK]], <i32 1>
-; CHECK-NEXT:    [[BLOCK8:%.*]] = shufflevector <1 x i32> [[SPLIT1]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = mul <1 x i32> [[BLOCK8]], <i32 2>
-; CHECK-NEXT:    [[TMP2:%.*]] = add <1 x i32> [[TMP0]], [[TMP1]]
-; CHECK-NEXT:    [[BLOCK9:%.*]] = shufflevector <1 x i32> [[SPLIT2]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <1 x i32> [[BLOCK9]], <i32 3>
-; CHECK-NEXT:    [[TMP4:%.*]] = add <1 x i32> [[TMP2]], [[TMP3]]
-; CHECK-NEXT:    [[BLOCK10:%.*]] = shufflevector <1 x i32> [[SPLIT3]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = mul <1 x i32> [[BLOCK10]], <i32 4>
-; CHECK-NEXT:    [[TMP6:%.*]] = add <1 x i32> [[TMP4]], [[TMP5]]
-; CHECK-NEXT:    [[BLOCK11:%.*]] = shufflevector <1 x i32> [[SPLIT4]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP7:%.*]] = mul <1 x i32> [[BLOCK11]], <i32 5>
-; CHECK-NEXT:    [[TMP8:%.*]] = add <1 x i32> [[TMP6]], [[TMP7]]
-; CHECK-NEXT:    [[BLOCK12:%.*]] = shufflevector <1 x i32> [[SPLIT5]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = mul <1 x i32> [[BLOCK12]], <i32 6>
-; CHECK-NEXT:    [[TMP10:%.*]] = add <1 x i32> [[TMP8]], [[TMP9]]
-; CHECK-NEXT:    [[BLOCK13:%.*]] = shufflevector <1 x i32> [[SPLIT6]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP11:%.*]] = mul <1 x i32> [[BLOCK13]], <i32 7>
-; CHECK-NEXT:    [[TMP12:%.*]] = add <1 x i32> [[TMP10]], [[TMP11]]
-; CHECK-NEXT:    [[BLOCK14:%.*]] = shufflevector <1 x i32> [[SPLIT7]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP13:%.*]] = mul <1 x i32> [[BLOCK14]], <i32 8>
-; CHECK-NEXT:    [[TMP14:%.*]] = add <1 x i32> [[TMP12]], [[TMP13]]
-; CHECK-NEXT:    [[TMP15:%.*]] = shufflevector <1 x i32> [[TMP14]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP16:%.*]] = shufflevector <1 x i32> undef, <1 x i32> [[TMP15]], <1 x i32> <i32 1>
-; CHECK-NEXT:    ret <1 x i32> [[TMP16]]
+; CHECK-NEXT:    [[TMP0:%.*]] = mul <8 x i32> [[A:%.*]], <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8>
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> [[TMP0]])
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <1 x i32> poison, i32 [[TMP1]], i64 0
+; CHECK-NEXT:    ret <1 x i32> [[TMP2]]
 ;
 entry:
   %c = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32> %a, <8 x i32> <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8>, i32 1, i32 8, i32 1)
   ret <1 x i32> %c
 }
 
 define <1 x i32> @add_feeding_dotproduct_i32_v8_(<8 x i32> %a, <8 x i32> %b, <8 x i32> %c) {
 ; CHECK-LABEL: @add_feeding_dotproduct_i32_v8_(
entry:
   %add.2 = add <8 x i32> %add.1, %c
   %res = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32> %add.2, <8 x i32> %d, i32 1, i32 8, i32 1)
   ret <1 x i32> %res
 }
 
 define <1 x i64> @dotproduct_i64_v8(<8 x i64> %a, <8 x i64> %b) {
 ; CHECK-LABEL: @dotproduct_i64_v8(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[SPLIT:%.*]] = shufflevector <8 x i64> [[A:%.*]], <8 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[SPLIT1:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 1>
-; CHECK-NEXT:    [[SPLIT2:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 2>
-; CHECK-NEXT:    [[SPLIT3:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 3>
-; CHECK-NEXT:    [[SPLIT4:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 4>
-; CHECK-NEXT:    [[SPLIT5:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 5>
-; CHECK-NEXT:    [[SPLIT6:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 6>
-; CHECK-NEXT:    [[SPLIT7:%.*]] = shufflevector <8 x i64> [[A]], <8 x i64> poison, <1 x i32> <i32 7>
-; CHECK-NEXT:    [[SPLIT8:%.*]] = shufflevector <8 x i64> [[B:%.*]], <8 x i64> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i64> [[SPLIT]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT:%.*]] = insertelement <1 x i64> poison, i64 [[TMP0]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = mul <1 x i64> [[BLOCK]], [[SPLAT_SPLAT]]
-; CHECK-NEXT:    [[BLOCK9:%.*]] = shufflevector <1 x i64> [[SPLIT1]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 1
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT10:%.*]] = insertelement <1 x i64> poison, i64 [[TMP2]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT11:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT10]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <1 x i64> [[BLOCK9]], [[SPLAT_SPLAT11]]
-; CHECK-NEXT:    [[TMP4:%.*]] = add <1 x i64> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[BLOCK12:%.*]] = shufflevector <1 x i64> [[SPLIT2]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 2
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT13:%.*]] = insertelement <1 x i64> poison, i64 [[TMP5]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT14:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT13]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <1 x i64> [[BLOCK12]], [[SPLAT_SPLAT14]]
-; CHECK-NEXT:    [[TMP7:%.*]] = add <1 x i64> [[TMP4]], [[TMP6]]
-; CHECK-NEXT:    [[BLOCK15:%.*]] = shufflevector <1 x i64> [[SPLIT3]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 3
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT16:%.*]] = insertelement <1 x i64> poison, i64 [[TMP8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT17:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT16]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = mul <1 x i64> [[BLOCK15]], [[SPLAT_SPLAT17]]
-; CHECK-NEXT:    [[TMP10:%.*]] = add <1 x i64> [[TMP7]], [[TMP9]]
-; CHECK-NEXT:    [[BLOCK18:%.*]] = shufflevector <1 x i64> [[SPLIT4]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 4
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT19:%.*]] = insertelement <1 x i64> poison, i64 [[TMP11]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT20:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT19]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = mul <1 x i64> [[BLOCK18]], [[SPLAT_SPLAT20]]
-; CHECK-NEXT:    [[TMP13:%.*]] = add <1 x i64> [[TMP10]], [[TMP12]]
-; CHECK-NEXT:    [[BLOCK21:%.*]] = shufflevector <1 x i64> [[SPLIT5]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 5
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT22:%.*]] = insertelement <1 x i64> poison, i64 [[TMP14]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT23:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT22]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP15:%.*]] = mul <1 x i64> [[BLOCK21]], [[SPLAT_SPLAT23]]
-; CHECK-NEXT:    [[TMP16:%.*]] = add <1 x i64> [[TMP13]], [[TMP15]]
-; CHECK-NEXT:    [[BLOCK24:%.*]] = shufflevector <1 x i64> [[SPLIT6]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 6
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT25:%.*]] = insertelement <1 x i64> poison, i64 [[TMP17]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT26:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT25]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP18:%.*]] = mul <1 x i64> [[BLOCK24]], [[SPLAT_SPLAT26]]
-; CHECK-NEXT:    [[TMP19:%.*]] = add <1 x i64> [[TMP16]], [[TMP18]]
-; CHECK-NEXT:    [[BLOCK27:%.*]] = shufflevector <1 x i64> [[SPLIT7]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP20:%.*]] = extractelement <8 x i64> [[SPLIT8]], i64 7
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT28:%.*]] = insertelement <1 x i64> poison, i64 [[TMP20]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT29:%.*]] = shufflevector <1 x i64> [[SPLAT_SPLATINSERT28]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP21:%.*]] = mul <1 x i64> [[BLOCK27]], [[SPLAT_SPLAT29]]
-; CHECK-NEXT:    [[TMP22:%.*]] = add <1 x i64> [[TMP19]], [[TMP21]]
-; CHECK-NEXT:    [[TMP23:%.*]] = shufflevector <1 x i64> [[TMP22]], <1 x i64> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP24:%.*]] = shufflevector <1 x i64> undef, <1 x i64> [[TMP23]], <1 x i32> <i32 1>
-; CHECK-NEXT:    ret <1 x i64> [[TMP24]]
+; CHECK-NEXT:    [[TMP0:%.*]] = mul <8 x i64> [[A:%.*]], [[B:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call i64 @llvm.vector.reduce.add.v8i64(<8 x i64> [[TMP0]])
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <1 x i64> poison, i64 [[TMP1]], i64 0
+; CHECK-NEXT:    ret <1 x i64> [[TMP2]]
 ;
 entry:
   %c = tail call <1 x i64> @llvm.matrix.multiply.v1i64.v8i64.v8i64(<8 x i64> %a, <8 x i64> %b, i32 1, i32 8, i32 1)
   ret <1 x i64> %c
 }
 
 declare <1 x i64> @llvm.matrix.multiply.v1i64.v8i64.v8i64(<8 x i64>, <8 x i64>, i32, i32, i32)
 
 define <1 x i32> @intrinsic_column_major_load_dot_product_i32_v8(ptr %lhs_address, ptr %rhs_address) {
 ; CHECK-LABEL: @intrinsic_column_major_load_dot_product_i32_v8(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[COL_LOAD:%.*]] = load <8 x i32>, ptr [[LHS_ADDRESS:%.*]], align 4
-; CHECK-NEXT:    [[COL_LOAD1:%.*]] = load <1 x i32>, ptr [[RHS_ADDRESS:%.*]], align 4
-; CHECK-NEXT:    [[VEC_GEP:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 1
-; CHECK-NEXT:    [[COL_LOAD2:%.*]] = load <1 x i32>, ptr [[VEC_GEP]], align 4
-; CHECK-NEXT:    [[VEC_GEP3:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 2
-; CHECK-NEXT:    [[COL_LOAD4:%.*]] = load <1 x i32>, ptr [[VEC_GEP3]], align 4
-; CHECK-NEXT:    [[VEC_GEP5:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 3
-; CHECK-NEXT:    [[COL_LOAD6:%.*]] = load <1 x i32>, ptr [[VEC_GEP5]], align 4
-; CHECK-NEXT:    [[VEC_GEP7:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 4
-; CHECK-NEXT:    [[COL_LOAD8:%.*]] = load <1 x i32>, ptr [[VEC_GEP7]], align 4
-; CHECK-NEXT:    [[VEC_GEP9:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 5
-; CHECK-NEXT:    [[COL_LOAD10:%.*]] = load <1 x i32>, ptr [[VEC_GEP9]], align 4
-; CHECK-NEXT:    [[VEC_GEP11:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 6
-; CHECK-NEXT:    [[COL_LOAD12:%.*]] = load <1 x i32>, ptr [[VEC_GEP11]], align 4
-; CHECK-NEXT:    [[VEC_GEP13:%.*]] = getelementptr i32, ptr [[RHS_ADDRESS]], i64 7
-; CHECK-NEXT:    [[COL_LOAD14:%.*]] = load <1 x i32>, ptr [[VEC_GEP13]], align 4
-; CHECK-NEXT:    [[SPLIT:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[SPLIT15:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 1>
-; CHECK-NEXT:    [[SPLIT16:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 2>
-; CHECK-NEXT:    [[SPLIT17:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 3>
-; CHECK-NEXT:    [[SPLIT18:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 4>
-; CHECK-NEXT:    [[SPLIT19:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 5>
-; CHECK-NEXT:    [[SPLIT20:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 6>
-; CHECK-NEXT:    [[SPLIT21:%.*]] = shufflevector <8 x i32> [[COL_LOAD]], <8 x i32> poison, <1 x i32> <i32 7>
-; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <1 x i32> [[COL_LOAD1]], <1 x i32> [[COL_LOAD2]], <2 x i32> <i32 0, i32 1>
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <1 x i32> [[COL_LOAD4]], <1 x i32> [[COL_LOAD6]], <2 x i32> <i32 0, i32 1>
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <1 x i32> [[COL_LOAD8]], <1 x i32> [[COL_LOAD10]], <2 x i32> <i32 0, i32 1>
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <1 x i32> [[COL_LOAD12]], <1 x i32> [[COL_LOAD14]], <2 x i32> <i32 0, i32 1>
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> [[TMP1]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP2]], <2 x i32> [[TMP3]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
-; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i32> [[TMP4]], <4 x i32> [[TMP5]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[SPLIT22:%.*]] = shufflevector <8 x i32> [[TMP6]], <8 x i32> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i32> [[SPLIT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT:%.*]] = insertelement <1 x i32> poison, i32 [[TMP7]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP8:%.*]] = mul <1 x i32> [[BLOCK]], [[SPLAT_SPLAT]]
-; CHECK-NEXT:    [[BLOCK23:%.*]] = shufflevector <1 x i32> [[SPLIT15]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 1
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT24:%.*]] = insertelement <1 x i32> poison, i32 [[TMP9]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT25:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT24]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP10:%.*]] = mul <1 x i32> [[BLOCK23]], [[SPLAT_SPLAT25]]
-; CHECK-NEXT:    [[TMP11:%.*]] = add <1 x i32> [[TMP8]], [[TMP10]]
-; CHECK-NEXT:    [[BLOCK26:%.*]] = shufflevector <1 x i32> [[SPLIT16]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 2
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT27:%.*]] = insertelement <1 x i32> poison, i32 [[TMP12]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT28:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT27]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP13:%.*]] = mul <1 x i32> [[BLOCK26]], [[SPLAT_SPLAT28]]
-; CHECK-NEXT:    [[TMP14:%.*]] = add <1 x i32> [[TMP11]], [[TMP13]]
-; CHECK-NEXT:    [[BLOCK29:%.*]] = shufflevector <1 x i32> [[SPLIT17]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 3
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT30:%.*]] = insertelement <1 x i32> poison, i32 [[TMP15]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT31:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT30]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP16:%.*]] = mul <1 x i32> [[BLOCK29]], [[SPLAT_SPLAT31]]
-; CHECK-NEXT:    [[TMP17:%.*]] = add <1 x i32> [[TMP14]], [[TMP16]]
-; CHECK-NEXT:    [[BLOCK32:%.*]] = shufflevector <1 x i32> [[SPLIT18]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP18:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 4
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT33:%.*]] = insertelement <1 x i32> poison, i32 [[TMP18]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT34:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT33]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP19:%.*]] = mul <1 x i32> [[BLOCK32]], [[SPLAT_SPLAT34]]
-; CHECK-NEXT:    [[TMP20:%.*]] = add <1 x i32> [[TMP17]], [[TMP19]]
-; CHECK-NEXT:    [[BLOCK35:%.*]] = shufflevector <1 x i32> [[SPLIT19]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP21:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 5
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT36:%.*]] = insertelement <1 x i32> poison, i32 [[TMP21]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT37:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT36]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP22:%.*]] = mul <1 x i32> [[BLOCK35]], [[SPLAT_SPLAT37]]
-; CHECK-NEXT:    [[TMP23:%.*]] = add <1 x i32> [[TMP20]], [[TMP22]]
-; CHECK-NEXT:    [[BLOCK38:%.*]] = shufflevector <1 x i32> [[SPLIT20]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP24:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 6
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT39:%.*]] = insertelement <1 x i32> poison, i32 [[TMP24]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT40:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT39]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP25:%.*]] = mul <1 x i32> [[BLOCK38]], [[SPLAT_SPLAT40]]
-; CHECK-NEXT:    [[TMP26:%.*]] = add <1 x i32> [[TMP23]], [[TMP25]]
-; CHECK-NEXT:    [[BLOCK41:%.*]] = shufflevector <1 x i32> [[SPLIT21]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP27:%.*]] = extractelement <8 x i32> [[SPLIT22]], i64 7
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT42:%.*]] = insertelement <1 x i32> poison, i32 [[TMP27]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT43:%.*]] = shufflevector <1 x i32> [[SPLAT_SPLATINSERT42]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP28:%.*]] = mul <1 x i32> [[BLOCK41]], [[SPLAT_SPLAT43]]
-; CHECK-NEXT:    [[TMP29:%.*]] = add <1 x i32> [[TMP26]], [[TMP28]]
-; CHECK-NEXT:    [[TMP30:%.*]] = shufflevector <1 x i32> [[TMP29]], <1 x i32> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP31:%.*]] = shufflevector <1 x i32> undef, <1 x i32> [[TMP30]], <1 x i32> <i32 1>
-; CHECK-NEXT:    ret <1 x i32> [[TMP31]]
+; CHECK-NEXT:    [[TMP0:%.*]] = load <8 x i32>, ptr [[LHS_ADDRESS:%.*]], align 32
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i32>, ptr [[RHS_ADDRESS:%.*]], align 32
+; CHECK-NEXT:    [[TMP2:%.*]] = mul <8 x i32> [[TMP0]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> [[TMP2]])
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <1 x i32> poison, i32 [[TMP3]], i64 0
+; CHECK-NEXT:    ret <1 x i32> [[TMP4]]
 ;
 entry:
   %lhs = tail call <8 x i32> @llvm.matrix.column.major.load.v8i32.i64(ptr nonnull align 4 %lhs_address, i64 8, i1 false, i32 8, i32 1)
   %rhs = tail call <8 x i32> @llvm.matrix.column.major.load.v8i32.i64(ptr nonnull align 4 %rhs_address, i64 1, i1 false, i32 1, i32 8)
   %result = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v8i32.v8i32(<8 x i32> %lhs, <8 x i32> %rhs, i32 1, i32 8, i32 1)
   ret <1 x i32> %result
 }
 
entry:
   %result = tail call <1 x i32> @llvm.matrix.multiply.v1i32.v4i32.v4i32(<4 x i32> %lhs, <4 x i32> %rhs, i32 1, i32 4, i32 1)
   ret <1 x i32> %result
 }
 
 
 define <1 x i16> @LoadInst_dot_product_i16_v6(ptr %lhs_address, ptr %rhs_address) {
 ; CHECK-LABEL: @LoadInst_dot_product_i16_v6(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[COL_LOAD:%.*]] = load <1 x i16>, ptr [[LHS_ADDRESS:%.*]], align 16
-; CHECK-NEXT:    [[VEC_GEP:%.*]] = getelementptr i16, ptr [[LHS_ADDRESS]], i64 1
-; CHECK-NEXT:    [[COL_LOAD1:%.*]] = load <1 x i16>, ptr [[VEC_GEP]], align 2
-; CHECK-NEXT:    [[VEC_GEP2:%.*]] = getelementptr i16, ptr [[LHS_ADDRESS]], i64 2
-; CHECK-NEXT:    [[COL_LOAD3:%.*]] = load <1 x i16>, ptr [[VEC_GEP2]], align 4
-; CHECK-NEXT:    [[VEC_GEP4:%.*]] = getelementptr i16, ptr [[LHS_ADDRESS]], i64 3
-; CHECK-NEXT:    [[COL_LOAD5:%.*]] = load <1 x i16>, ptr [[VEC_GEP4]], align 2
-; CHECK-NEXT:    [[VEC_GEP6:%.*]] = getelementptr i16, ptr [[LHS_ADDRESS]], i64 4
-; CHECK-NEXT:    [[COL_LOAD7:%.*]] = load <1 x i16>, ptr [[VEC_GEP6]], align 8
-; CHECK-NEXT:    [[VEC_GEP8:%.*]] = getelementptr i16, ptr [[LHS_ADDRESS]], i64 5
-; CHECK-NEXT:    [[COL_LOAD9:%.*]] = load <1 x i16>, ptr [[VEC_GEP8]], align 2
-; CHECK-NEXT:    [[COL_LOAD10:%.*]] = load <6 x i16>, ptr [[RHS_ADDRESS:%.*]], align 16
-; CHECK-NEXT:    [[BLOCK:%.*]] = shufflevector <1 x i16> [[COL_LOAD]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT:%.*]] = insertelement <1 x i16> poison, i16 [[TMP0]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP1:%.*]] = mul <1 x i16> [[BLOCK]], [[SPLAT_SPLAT]]
-; CHECK-NEXT:    [[BLOCK11:%.*]] = shufflevector <1 x i16> [[COL_LOAD1]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 1
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT12:%.*]] = insertelement <1 x i16> poison, i16 [[TMP2]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT13:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT12]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <1 x i16> [[BLOCK11]], [[SPLAT_SPLAT13]]
-; CHECK-NEXT:    [[TMP4:%.*]] = add <1 x i16> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[BLOCK14:%.*]] = shufflevector <1 x i16> [[COL_LOAD3]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 2
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT15:%.*]] = insertelement <1 x i16> poison, i16 [[TMP5]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT16:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT15]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP6:%.*]] = mul <1 x i16> [[BLOCK14]], [[SPLAT_SPLAT16]]
-; CHECK-NEXT:    [[TMP7:%.*]] = add <1 x i16> [[TMP4]], [[TMP6]]
-; CHECK-NEXT:    [[BLOCK17:%.*]] = shufflevector <1 x i16> [[COL_LOAD5]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 3
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT18:%.*]] = insertelement <1 x i16> poison, i16 [[TMP8]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT19:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT18]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP9:%.*]] = mul <1 x i16> [[BLOCK17]], [[SPLAT_SPLAT19]]
-; CHECK-NEXT:    [[TMP10:%.*]] = add <1 x i16> [[TMP7]], [[TMP9]]
-; CHECK-NEXT:    [[BLOCK20:%.*]] = shufflevector <1 x i16> [[COL_LOAD7]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 4
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT21:%.*]] = insertelement <1 x i16> poison, i16 [[TMP11]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT22:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT21]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP12:%.*]] = mul <1 x i16> [[BLOCK20]], [[SPLAT_SPLAT22]]
-; CHECK-NEXT:    [[TMP13:%.*]] = add <1 x i16> [[TMP10]], [[TMP12]]
-; CHECK-NEXT:    [[BLOCK23:%.*]] = shufflevector <1 x i16> [[COL_LOAD9]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <6 x i16> [[COL_LOAD10]], i64 5
-; CHECK-NEXT:    [[SPLAT_SPLATINSERT24:%.*]] = insertelement <1 x i16> poison, i16 [[TMP14]], i64 0
-; CHECK-NEXT:    [[SPLAT_SPLAT25:%.*]] = shufflevector <1 x i16> [[SPLAT_SPLATINSERT24]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP15:%.*]] = mul <1 x i16> [[BLOCK23]], [[SPLAT_SPLAT25]]
-; CHECK-NEXT:    [[TMP16:%.*]] = add <1 x i16> [[TMP13]], [[TMP15]]
-; CHECK-NEXT:    [[TMP17:%.*]] = shufflevector <1 x i16> [[TMP16]], <1 x i16> poison, <1 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP18:%.*]] = shufflevector <1 x i16> undef, <1 x i16> [[TMP17]], <1 x i32> <i32 1>
-; CHECK-NEXT:    ret <1 x i16> [[TMP18]]
+; CHECK-NEXT:    [[LHS:%.*]] = load <6 x i16>, ptr [[LHS_ADDRESS:%.*]], align 16
+; CHECK-NEXT:    [[RHS:%.*]] = load <6 x i16>, ptr [[RHS_ADDRESS:%.*]], align 16
+; CHECK-NEXT:    [[TMP0:%.*]] = mul <6 x i16> [[LHS]], [[RHS]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call i16 @llvm.vector.reduce.add.v6i16(<6 x i16> [[TMP0]])
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <1 x i16> poison, i16 [[TMP1]], i64 0
+; CHECK-NEXT:    ret <1 x i16> [[TMP2]]
 ;
 entry:
   %lhs = load <6 x i16>, ptr %lhs_address
   %rhs = load <6 x i16>, ptr %rhs_address
   %result = tail call <1 x i16> @llvm.matrix.multiply.v1i16.v6i16.v6i16(<6 x i16> %lhs, <6 x i16> %rhs, i32 1, i32 6, i32 1)
   ret <1 x i16> %result
 }
 
 declare <1 x i16> @llvm.matrix.multiply.v1i16.v6i16.v6i16(<6 x i16>, <6 x i16>, i32, i32, i32)