[InstCombine] Matrix multiplication negation optimisation

[zjaffal]

Summary

If one of the operands in a matrix multiplication is negated we can optimise the equation by moving the negation to the smallest element of the operands or the result.

[fhahn]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

73

Why is this needed?

3289

here it should be sufficient to use a more compact mathematical notation: (-A) * B = -(A * B)

3320

can use cast here if you are not checking if FNegType is null. Same for similar uses uses of dyn_cast here

[fhahn]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

3331

I think this is not correct, you are updating all uses of FNegOp, but we only have to update the use in the matrix multiply. Can you add a test case where the FNeg also has other users in some different instructions? They should remain unchanged.

Also, it would probably make sense to limit this to fneg instructions with a single use. If there are other uses outside the multiply, we still need to negate the input and we only add an extra fneg.

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1889

Add an FP transform near these other FP intrinsics.

3331

Also, we don't typically replace operands - just create a new call. That's what instcombine's worklist iteration is expecting.
Look at existing FP transforms above for examples (and where I think this transform should be placed too).

[thegameg]

Are we looking to also support -(A * B) -> -A * B with the negation on the cheapest operand? (might need to check what the required fast math flags are)

[zjaffal]

Move code next to FP Intrinsics

[radford]

In D133300#3771911, @thegameg wrote:

Are we looking to also support -(A * B) -> -A * B with the negation on the cheapest operand? (might need to check what the required fast math flags are)

We should. In particular, there three places for the negation to go: -A * B = A * -B = -(A * B) and any one of the three might be smaller.

[spatel]

What happens if both args are negated? We need a test for that. Maybe that pattern should be handled before this, so we don't have to deal with the complication in this patch?

Tests should also include fast-math-flags in at least some cases, so we can see if those are propagated as expected.

[zjaffal]

In D133300#3774699, @radford wrote:

In D133300#3771911, @thegameg wrote:

Are we looking to also support -(A * B) -> -A * B with the negation on the cheapest operand? (might need to check what the required fast math flags are)

We should. In particular, there three places for the negation to go: -A * B = A * -B = -(A * B) and any one of the three might be smaller.

Yes I am not covering this case. I can add a test case for it and then introduce another patch for it

[zjaffal]

In D133300#3776842, @spatel wrote:

What happens if both args are negated? We need a test for that. Maybe that pattern should be handled before this, so we don't have to deal with the complication in this patch?

Tests should also include fast-math-flags in at least some cases, so we can see if those are propagated as expected.

I will add a test case for that now

[zjaffal]

Support where two operands are negated

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1850–1858

I don't think this is correct if an fneg has multiple uses (similar to the bug noted earlier, and I repeat my suggestion to create new instructions rather than modifying existing ones).

Please split this change and tests to its own review ahead of the original transforms in this patch.

[zjaffal]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1850–1858

We don't need this check anyways.
if both operands are negative then there is two cases:

1. Both of the negations will be moved to the result and then another pass will remove the negations
2. negation gets moved to an operand and then we have two negations on one operand which will be optimised as well.

[zjaffal]

remove unecessary check for two negations

[fhahn]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1833

optimise -> optimize (US spelling is commonly used in LLVM)

1835

nit: We -> we, optimise -> optimize (US spelling is commonly used in LLVM)

1837

smalest->smallest

1846

no need to dyn_cast here, the result is guaranteed to be a vector type, cast can be used.

1850

no need to capture a value you are not using , m_Value() should just work

Also, you only use the variable MatchOp0 in a single place, so it would be easier to read if you just use if (match(..))

1850–1858

@zjaffal do we have a test case where both operands are negated and at least one of the fnegs has multiple uses?

1884

nit: Period at end of sentence.

1886

There should be no need to cast to Instruction here?

1912

please remove the stray line change.

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1850–1858

Correct me if I'm wrong:

1. When both ops are negated, we don't care what the relative sizes of the values are, we always want to use the non-negated source ops.
2. When both ops are negated, we don't care if those ops have other uses, we always want to use the non-negated source ops.

Either way, we need tests to exercise those patterns.

define <4 x double> @matrix_multiply_v2f64_v2f64(<2 x double> %a, <2 x double> %b) {
  %a.neg = fneg <2 x double> %a
  %b.neg = fneg <2 x double> %b
  %res = call <4 x double> @llvm.matrix.multiply.v4f64.v2f64.v2f64(<2 x double> %a.neg, <2 x double> %b.neg, i32 2, i32 1, i32 2)
  ret <4 x double> %res
}

[zjaffal]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1850–1858

Correct me if I'm wrong:

1. When both ops are negated, we don't care what the relative sizes of the values are, we always want to use the non-negated source ops.

I am not sure I understand the question fully but in the case of the two operands the first operand gets handled first and we don't check if it is the larger of both

2. When both ops are negated, we don't care if those ops have other uses, we always want to use the non-negated source ops.

Currently if fmul has other uses we will still use the negated ops in the multiplication

Either way, we need tests to exercise those patterns.

define <4 x double> @matrix_multiply_v2f64_v2f64(<2 x double> %a, <2 x double> %b) {
  %a.neg = fneg <2 x double> %a
  %b.neg = fneg <2 x double> %b
  %res = call <4 x double> @llvm.matrix.multiply.v4f64.v2f64.v2f64(<2 x double> %a.neg, <2 x double> %b.neg, i32 2, i32 1, i32 2)
  ret <4 x double> %res
}

Yes I think I need to expand on the test cases where we have two operands.

[zjaffal]

• Move the test to a seperate patch.
• In the cases where both operands are negated we may need to introduce a seperate patch to handle that case

[spatel]

In D133300#3781177, @zjaffal wrote:

• In the cases where both operands are negated we may need to introduce a seperate patch to handle that case

Yes, and it should be written first. Without it, we have inconsistent behavior. This patch will reduce the 2nd example below, but it misses the 1st even though they are very similar patterns:

define <9 x double> @test_with_two_operands_negated2_commute(<3 x double> %a, <27 x double> %b){
  %a.neg = fneg <3 x double> %a
  %b.neg = fneg <27 x double> %b
  %res = call <9 x double> @llvm.matrix.multiply.v9f64.v3f64.v27f64(<3 x double> %a.neg, <27 x double> %b.neg, i32 1, i32 3, i32 9)
  ret <9 x double> %res
}

define <9 x double> @test_with_two_operands_negated2(<27 x double> %a, <3 x double> %b){
  %a.neg = fneg <27 x double> %a
  %b.neg = fneg <3 x double> %b
  %res = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> %a.neg, <3 x double> %b.neg, i32 9, i32 3, i32 1)
  ret <9 x double> %res
}

[spatel]

Currently if fmul has other uses we will still use the negated ops in the multiplication

I don't think that's correct - there are no use restrictions on this transform:
https://github.com/llvm/llvm-project/blob/24e1736d84fd0fb45097245706a523c3398beb69/llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp#L453

[zjaffal]

1. Add the optimization for two negations into a separate patch
2. Remove the check for single use for negation operation
3. Address style comments and spelling mistakes

[zjaffal]

replace dyn_cast with cast

[zjaffal]

update transforms

[zjaffal]

use patterns

[zjaffal]

Update variable names to patch comments

[fhahn]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1833

nit: We -> we, it's -> its.

On second thought, I think it would be better to just move the Case 1,2,3 comments to the code that actually deals with those cases and remove the sentence here.

1834

We -> we.

1862

IIUC this is the case where we move the negation from one to the other operand. Could you move the comment for Case 2 above here?

1867

If I read the code correctly, this may not be the second operand but could also the first one if the second one is negated?

1868

I thought an earlier version created a new call here, rather than updating the exist one. Did we agree that replaceOperand here is the right thing to do?

1872

IIUC this is the case where we move the negation from an argument to the result of the multiply. Could you move the comment from Case 3 here?

1877

there should be no need to cast to Instruction here, you should be able to just use Value.

[zjaffal]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1867

I think it might be better to name them

1. NegatedOperand
2. OtherOperand or NonNegatedOperand

1868

We agreed in using replaceOperand since it is used in other areas of the same file.

[zjaffal]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1877

You need it when we call FNegInst->setOperand()

[zjaffal]

Fix comments and variable names

[fhahn]

LGTM, thanks!

[spatel]

llvm/test/Transforms/InstCombine/matrix-multiplication-negation.ll

259

We added an fneg to this sequence without removing the existing one. How is this better?

[zjaffal]

llvm/test/Transforms/InstCombine/matrix-multiplication-negation.ll

259

This is a result of removing the check if the negation has a single use.

[spatel]

llvm/test/Transforms/InstCombine/matrix-multiplication-negation.ll

259

Right - it made sense when both operands are negated in the other patch, but not when only 1 is negated. The one-use check should be present for this optimization.

[zjaffal]

llvm/test/Transforms/InstCombine/matrix-multiplication-negation.ll

259

Perfect, I will add it now

[zjaffal]

Add condition to only optimize if the negated operand has one use

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

1887

This sequence of create/replace/reset operand seems shaky. This transform is different than the earlier ones because we are creating a new instruction after the existing matmul call.

It would be better to use the standard practice: Builder.CreateIntrinsic() followed by UnaryOperator::CreateFNeg().

This is also dropping all FMF on the fneg. Usually, we'd propagate the FMF from the matmul to the new fneg. We should adjust the tests to show this behavior more explicitly (ie, put some FMF on the fnegs in the tests).

[zjaffal]

Refactor the code for moving negation to the result. Now we preserve the fastmath flags on the created multiplication and negation instructions by passing them from the original multiplication instruction. FNeg is created using Builder instead of UnitaryOperand because
the latter caused build failures.
The changes in the fastflag behaviour can be seen on test_negation_move_to_result_with_fastflags test.

I added another 3 test cases using a subset of the fast flags and the flags on the negation instruction instead of the multiplication.

1. test_negation_move_to_result_with_nnan_flag
2. test_negation_move_to_result_with_nsz_flag
3. test_negation_move_to_result_with_fastflag_on_negation

[spatel]

LGTM

[fhahn]

Thanks for the latest update & @spatel for all your comments!

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/InstCombine/InstCombiner.h"
 #include "llvm/Transforms/Utils/AssumeBundleBuilder.h"
 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Utils/SimplifyLibCalls.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <utility>
 #include <vector>
 
 #define DEBUG_TYPE "instcombine"
 #include "llvm/Transforms/Utils/InstructionWorklist.h"
 
 using namespace llvm;
 using namespace PatternMatch;
if ((match(Arg0, m_OneUse(m_FNeg(m_Value(X)))) && Arg1 == X) ||
       if (IID == Intrinsic::minimum || IID == Intrinsic::minnum)
         R = Builder.CreateFNegFMF(R, II);
       return replaceInstUsesWith(*II, R);
     }
 
     break;
   }
   case Intrinsic::matrix_multiply: {
+    // Optimize negation in matrix multiplication.
+
     // -A * -B -> A * B
     Value *A, *B;
     if (match(II->getArgOperand(0), m_FNeg(m_Value(A))) &&
         match(II->getArgOperand(1), m_FNeg(m_Value(B)))) {
       replaceOperand(*II, 0, A);
       replaceOperand(*II, 1, B);
       return II;
     }
+
+    Value *Op0 = II->getOperand(0);
+    Value *Op1 = II->getOperand(1);
+    Value *OpNotNeg, *NegatedOp;
+    unsigned NegatedOpArg, OtherOpArg;
+    if (match(Op0, m_FNeg(m_Value(OpNotNeg)))) {
+      NegatedOp = Op0;
+      NegatedOpArg = 0;
+      OtherOpArg = 1;
+    } else if (match(Op1, m_FNeg(m_Value(OpNotNeg)))) {
+      NegatedOp = Op1;
+      NegatedOpArg = 1;
+      OtherOpArg = 0;
+    } else
+      // Multiplication doesn't have a negated operand.
+      break;
+
+    // Only optimize if the negated operand has only one use.
+    if (!NegatedOp->hasOneUse())
+      break;
+
+    Value *OtherOp = II->getOperand(OtherOpArg);
+    VectorType *RetTy = cast<VectorType>(II->getType());
+    VectorType *NegatedOpTy = cast<VectorType>(NegatedOp->getType());
+    VectorType *OtherOpTy = cast<VectorType>(OtherOp->getType());
+    ElementCount NegatedCount = NegatedOpTy->getElementCount();
+    ElementCount OtherCount = OtherOpTy->getElementCount();
+    ElementCount RetCount = RetTy->getElementCount();
+    // (-A) * B -> A * (-B), if it is cheaper to negate B and vice versa.
+    if (ElementCount::isKnownGT(NegatedCount, OtherCount) &&
+        ElementCount::isKnownLT(OtherCount, RetCount)) {
+      Value *InverseOtherOp = Builder.CreateFNeg(OtherOp);
+      replaceOperand(*II, NegatedOpArg, OpNotNeg);
+      replaceOperand(*II, OtherOpArg, InverseOtherOp);
+      return II;
+    }
+    // (-A) * B -> -(A * B), if it is cheaper to negate the result
+    if (ElementCount::isKnownGT(NegatedCount, RetCount)) {
+      SmallVector<Value *, 5> NewArgs(II->args());
+      NewArgs[NegatedOpArg] = OpNotNeg;
+      Instruction *NewMul =
+          Builder.CreateIntrinsic(II->getType(), IID, NewArgs, II);
+      return replaceInstUsesWith(*II, Builder.CreateFNegFMF(NewMul, II));
+    }
     break;
   }
   case Intrinsic::fmuladd: {
     // Canonicalize fast fmuladd to the separate fmul + fadd.
     if (II->isFast()) {
       BuilderTy::FastMathFlagGuard Guard(Builder);
       Builder.setFastMathFlags(II->getFastMathFlags());
       Value *Mul = Builder.CreateFMul(II->getArgOperand(0),
                                       II->getArgOperand(1));
       Value *Add = Builder.CreateFAdd(Mul, II->getArgOperand(2));
       Add->takeName(II);
       return replaceInstUsesWith(*II, Add);
     }
 
     // Try to simplify the underlying FMul.
     if (Value *V = simplifyFMulInst(II->getArgOperand(0), II->getArgOperand(1),
                                     II->getFastMathFlags(),
                                     SQ.getWithInstruction(II))) {
       auto *FAdd = BinaryOperator::CreateFAdd(V, II->getArgOperand(2));
       FAdd->copyFastMathFlags(II);
       return FAdd;
     }
 
     [[fallthrough]];
   }
   case Intrinsic::fma: {
     // fma fneg(x), fneg(y), z -> fma x, y, z
     Value *Src0 = II->getArgOperand(0);
     Value *Src1 = II->getArgOperand(1);
     Value *X, *Y;
     if (match(Src0, m_FNeg(m_Value(X))) && match(Src1, m_FNeg(m_Value(Y)))) {
       replaceOperand(*II, 0, X);
       replaceOperand(*II, 1, Y);
       return II;
case Intrinsic::experimental_gc_statepoint: {
     return CallBase::Create(&Call, NewBundle);
   }
   default: { break; }
   }
 
   return Changed ? &Call : nullptr;
 }
 
 /// If the callee is a constexpr cast of a function, attempt to move the cast to
 /// the arguments of the call/callbr/invoke.
 bool InstCombinerImpl::transformConstExprCastCall(CallBase &Call) {
   auto *Callee =
       dyn_cast<Function>(Call.getCalledOperand()->stripPointerCasts());
   if (!Callee)
     return false;
 
   // If this is a call to a thunk function, don't remove the cast. Thunks are
bool InstCombinerImpl::transformConstExprCastCall(CallBase &Call) {
 
   // Okay, this is a cast from a function to a different type.  Unless doing so
   // would cause a type conversion of one of our arguments, change this call to
   // be a direct call with arguments casted to the appropriate types.
   FunctionType *FT = Callee->getFunctionType();
   Type *OldRetTy = Caller->getType();
   Type *NewRetTy = FT->getReturnType();
 
   // Check to see if we are changing the return type...
   if (OldRetTy != NewRetTy) {
 
     if (NewRetTy->isStructTy())
       return false; // TODO: Handle multiple return values.
 
     if (!CastInst::isBitOrNoopPointerCastable(NewRetTy, OldRetTy, DL)) {
       if (Callee->isDeclaration())
         return false;   // Cannot transform this return value.
 
       if (!Caller->use_empty() &&
           // void -> non-void is handled specially
           !NewRetTy->isVoidTy())
         return false;   // Cannot transform this return value.
     }
 
     if (!CallerPAL.isEmpty() && !Caller->use_empty()) {
       AttrBuilder RAttrs(FT->getContext(), CallerPAL.getRetAttrs());
       if (RAttrs.overlaps(AttributeFuncs::typeIncompatible(NewRetTy)))
         return false;   // Attribute not compatible with transformed value.

llvm/test/Transforms/InstCombine/matrix-multiplication-negation.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=instcombine -S | FileCheck %s
 
 ; The result has the fewest vector elements between the result and the two operands so the negation can be moved there
 define <2 x double> @test_negation_move_to_result(<6 x double> %a, <3 x double> %b) {
 ; CHECK-LABEL: @test_negation_move_to_result(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <6 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A_NEG]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
-; CHECK-NEXT:    ret <2 x double> [[RES]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A:%.*]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
 ;
   %a.neg = fneg <6 x double> %a
   %res = tail call <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> %a.neg, <3 x double> %b, i32 2, i32 3, i32 1)
   ret <2 x double> %res
 }
 
 ; The result has the fewest vector elements between the result and the two operands so the negation can be moved there
 ; Fast flag should be preserved
 define <2 x double> @test_negation_move_to_result_with_fastflags(<6 x double> %a, <3 x double> %b) {
 ; CHECK-LABEL: @test_negation_move_to_result_with_fastflags(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <6 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call fast <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A_NEG]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
-; CHECK-NEXT:    ret <2 x double> [[RES]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call fast <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A:%.*]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg fast <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
 ;
   %a.neg = fneg <6 x double> %a
   %res = tail call fast <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> %a.neg, <3 x double> %b, i32 2, i32 3, i32 1)
   ret <2 x double> %res
 }
 
+define <2 x double> @test_negation_move_to_result_with_nnan_flag(<6 x double> %a, <3 x double> %b) {
+; CHECK-LABEL: @test_negation_move_to_result_with_nnan_flag(
+; CHECK-NEXT:    [[TMP1:%.*]] = call nnan <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A:%.*]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg nnan <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
+;
+  %a.neg = fneg <6 x double> %a
+  %res = tail call nnan <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> %a.neg, <3 x double> %b, i32 2, i32 3, i32 1)
+  ret <2 x double> %res
+}
+
+define <2 x double> @test_negation_move_to_result_with_nsz_flag(<6 x double> %a, <3 x double> %b) {
+; CHECK-LABEL: @test_negation_move_to_result_with_nsz_flag(
+; CHECK-NEXT:    [[TMP1:%.*]] = call nsz <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A:%.*]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg nsz <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
+;
+  %a.neg = fneg <6 x double> %a
+  %res = tail call nsz <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> %a.neg, <3 x double> %b, i32 2, i32 3, i32 1)
+  ret <2 x double> %res
+}
+
+define <2 x double> @test_negation_move_to_result_with_fastflag_on_negation(<6 x double> %a, <3 x double> %b) {
+; CHECK-LABEL: @test_negation_move_to_result_with_fastflag_on_negation(
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> [[A:%.*]], <3 x double> [[B:%.*]], i32 2, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
+;
+  %a.neg = fneg fast<6 x double> %a
+  %res = tail call <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double> %a.neg, <3 x double> %b, i32 2, i32 3, i32 1)
+  ret <2 x double> %res
+}
+
 ; %b has the fewest vector elements between the result and the two operands so the negation can be moved there
 define <9 x double> @test_move_negation_to_second_operand(<27 x double> %a, <3 x double> %b) {
 ; CHECK-LABEL: @test_move_negation_to_second_operand(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <27 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A_NEG]], <3 x double> [[B:%.*]], i32 9, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP1:%.*]] = fneg <3 x double> [[B:%.*]]
+; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A:%.*]], <3 x double> [[TMP1]], i32 9, i32 3, i32 1)
 ; CHECK-NEXT:    ret <9 x double> [[RES]]
 ;
   %a.neg = fneg <27 x double> %a
   %res = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> %a.neg, <3 x double> %b, i32 9, i32 3, i32 1)
   ret <9 x double> %res
 }
 
 ; %b has the fewest vector elements between the result and the two operands so the negation can be moved there
 ; Fast flag should be preserved
 define <9 x double> @test_move_negation_to_second_operand_with_fast_flags(<27 x double> %a, <3 x double> %b) {
 ; CHECK-LABEL: @test_move_negation_to_second_operand_with_fast_flags(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <27 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call fast <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A_NEG]], <3 x double> [[B:%.*]], i32 9, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP1:%.*]] = fneg <3 x double> [[B:%.*]]
+; CHECK-NEXT:    [[RES:%.*]] = tail call fast <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A:%.*]], <3 x double> [[TMP1]], i32 9, i32 3, i32 1)
 ; CHECK-NEXT:    ret <9 x double> [[RES]]
 ;
   %a.neg = fneg <27 x double> %a
   %res = tail call fast <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> %a.neg, <3 x double> %b, i32 9, i32 3, i32 1)
   ret <9 x double> %res
 }
 
 ; The result has the fewest vector elements between the result and the two operands so the negation can be moved there
 define <2 x double> @test_negation_move_to_result_from_second_operand(<3 x double> %a, <6 x double> %b){
 ; CHECK-LABEL: @test_negation_move_to_result_from_second_operand(
-; CHECK-NEXT:    [[B_NEG:%.*]] = fneg <6 x double> [[B:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <2 x double> @llvm.matrix.multiply.v2f64.v3f64.v6f64(<3 x double> [[A:%.*]], <6 x double> [[B_NEG]], i32 1, i32 3, i32 2)
-; CHECK-NEXT:    ret <2 x double> [[RES]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call <2 x double> @llvm.matrix.multiply.v2f64.v3f64.v6f64(<3 x double> [[A:%.*]], <6 x double> [[B:%.*]], i32 1, i32 3, i32 2)
+; CHECK-NEXT:    [[TMP2:%.*]] = fneg <2 x double> [[TMP1]]
+; CHECK-NEXT:    ret <2 x double> [[TMP2]]
 ;
   %b.neg = fneg <6 x double> %b
   %res = tail call <2 x double> @llvm.matrix.multiply.v2f64.v3f64.v6f64(<3 x double> %a, <6 x double> %b.neg, i32 1, i32 3, i32 2)
   ret <2 x double> %res
 }
 
 ; %a has the fewest vector elements between the result and the two operands so the negation can be moved there
 define <9 x double> @test_move_negation_to_first_operand(<3 x double> %a, <27 x double> %b) {
 ; CHECK-LABEL: @test_move_negation_to_first_operand(
-; CHECK-NEXT:    [[B_NEG:%.*]] = fneg <27 x double> [[B:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v3f64.v27f64(<3 x double> [[A:%.*]], <27 x double> [[B_NEG]], i32 1, i32 3, i32 9)
+; CHECK-NEXT:    [[TMP1:%.*]] = fneg <3 x double> [[A:%.*]]
+; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v3f64.v27f64(<3 x double> [[TMP1]], <27 x double> [[B:%.*]], i32 1, i32 3, i32 9)
 ; CHECK-NEXT:    ret <9 x double> [[RES]]
 ;
   %b.neg = fneg <27 x double> %b
   %res = tail call <9 x double> @llvm.matrix.multiply.v9f64.v3f64.v27f64(<3 x double> %a, <27 x double> %b.neg, i32 1, i32 3, i32 9)
   ret <9 x double> %res
 }
 
 ; %a has the fewest vector elements between the result and the two operands so the negation is not moved
;
   %res.3 = fadd <12 x double> %res.2, %res
   ret <12 x double> %res.3
 }
 
 define <12 x double> @fneg_with_multiple_uses_2(<15 x double> %a, <20 x double> %b, ptr %a_loc){
 ; CHECK-LABEL: @fneg_with_multiple_uses_2(
 ; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <15 x double> [[A:%.*]]
 ; CHECK-NEXT:    [[RES:%.*]] = tail call <12 x double> @llvm.matrix.multiply.v12f64.v15f64.v20f64(<15 x double> [[A_NEG]], <20 x double> [[B:%.*]], i32 3, i32 5, i32 4)
 ; CHECK-NEXT:    store <15 x double> [[A_NEG]], ptr [[A_LOC:%.*]], align 128
 ; CHECK-NEXT:    ret <12 x double> [[RES]]
 ;
   %a.neg = fneg <15 x double> %a
   %res = tail call <12 x double> @llvm.matrix.multiply.v12f64.v15f64.v20f64(<15 x double> %a.neg, <20 x double> %b, i32 3, i32 5, i32 4)
   store <15 x double> %a.neg, ptr %a_loc
   ret <12 x double> %res
 }
 ; negation should be moved to the second operand given it has the smallest operand count
 define <72 x double> @chain_of_matrix_mutliplies(<27 x double> %a, <3 x double> %b, <8 x double> %c) {
 ; CHECK-LABEL: @chain_of_matrix_mutliplies(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <27 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A_NEG]], <3 x double> [[B:%.*]], i32 9, i32 3, i32 1)
+; CHECK-NEXT:    [[TMP1:%.*]] = fneg <3 x double> [[B:%.*]]
+; CHECK-NEXT:    [[RES:%.*]] = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> [[A:%.*]], <3 x double> [[TMP1]], i32 9, i32 3, i32 1)
 ; CHECK-NEXT:    [[RES_2:%.*]] = tail call <72 x double> @llvm.matrix.multiply.v72f64.v9f64.v8f64(<9 x double> [[RES]], <8 x double> [[C:%.*]], i32 9, i32 1, i32 8)
 ; CHECK-NEXT:    ret <72 x double> [[RES_2]]
 ;
   %a.neg = fneg <27 x double> %a
   %res = tail call <9 x double> @llvm.matrix.multiply.v9f64.v27f64.v3f64(<27 x double> %a.neg, <3 x double> %b, i32 9, i32 3, i32 1)
   %res.2 = tail call <72 x double> @llvm.matrix.multiply.v72f64.v9f64.v8f64(<9 x double> %res, <8 x double> %c, i32 9, i32 1, i32 8)
   ret <72 x double> %res.2
 }
 
 ; first negation should be moved to %a
 ; second negation should be moved to the result of the second multipication
 define <6 x double> @chain_of_matrix_mutliplies_with_two_negations(<3 x double> %a, <5 x double> %b, <10 x double> %c) {
 ; CHECK-LABEL: @chain_of_matrix_mutliplies_with_two_negations(
-; CHECK-NEXT:    [[B_NEG:%.*]] = fneg <5 x double> [[B:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <15 x double> @llvm.matrix.multiply.v15f64.v3f64.v5f64(<3 x double> [[A:%.*]], <5 x double> [[B_NEG]], i32 3, i32 1, i32 5)
-; CHECK-NEXT:    [[RES_NEG:%.*]] = fneg <15 x double> [[RES]]
-; CHECK-NEXT:    [[RES_2:%.*]] = tail call <6 x double> @llvm.matrix.multiply.v6f64.v15f64.v10f64(<15 x double> [[RES_NEG]], <10 x double> [[C:%.*]], i32 3, i32 5, i32 2)
-; CHECK-NEXT:    ret <6 x double> [[RES_2]]
+; CHECK-NEXT:    [[TMP1:%.*]] = fneg <3 x double> [[A:%.*]]
+; CHECK-NEXT:    [[RES:%.*]] = tail call <15 x double> @llvm.matrix.multiply.v15f64.v3f64.v5f64(<3 x double> [[TMP1]], <5 x double> [[B:%.*]], i32 3, i32 1, i32 5)
+; CHECK-NEXT:    [[TMP2:%.*]] = call <6 x double> @llvm.matrix.multiply.v6f64.v15f64.v10f64(<15 x double> [[RES]], <10 x double> [[C:%.*]], i32 3, i32 5, i32 2)
+; CHECK-NEXT:    [[TMP3:%.*]] = fneg <6 x double> [[TMP2]]
+; CHECK-NEXT:    ret <6 x double> [[TMP3]]
 ;
   %b.neg = fneg <5 x double> %b
   %res = tail call <15 x double> @llvm.matrix.multiply.v15f64.v3f64.v5f64(<3 x double> %a, <5 x double> %b.neg, i32 3, i32 1, i32 5)
   %res.neg = fneg <15 x double> %res
   %res.2 = tail call <6 x double> @llvm.matrix.multiply.v6f64.v15f64.v10f64(<15 x double> %res.neg, <10 x double> %c, i32 3, i32 5, i32 2)
   ret <6 x double> %res.2
 }
 
 ; negation should be propagated to the result of the second matrix multiplication
 define <6 x double> @chain_of_matrix_mutliplies_propagation(<15 x double> %a, <20 x double> %b, <8 x double> %c){
 ; CHECK-LABEL: @chain_of_matrix_mutliplies_propagation(
-; CHECK-NEXT:    [[A_NEG:%.*]] = fneg <15 x double> [[A:%.*]]
-; CHECK-NEXT:    [[RES:%.*]] = tail call <12 x double> @llvm.matrix.multiply.v12f64.v15f64.v20f64(<15 x double> [[A_NEG]], <20 x double> [[B:%.*]], i32 3, i32 5, i32 4)
-; CHECK-NEXT:    [[RES_2:%.*]] = tail call <6 x double> @llvm.matrix.multiply.v6f64.v12f64.v8f64(<12 x double> [[RES]], <8 x double> [[C:%.*]], i32 3, i32 4, i32 2)
-; CHECK-NEXT:    ret <6 x double> [[RES_2]]
+; CHECK-NEXT:    [[TMP1:%.*]] = call <12 x double> @llvm.matrix.multiply.v12f64.v15f64.v20f64(<15 x double> [[A:%.*]], <20 x double> [[B:%.*]], i32 3, i32 5, i32 4)
+; CHECK-NEXT:    [[TMP2:%.*]] = call <6 x double> @llvm.matrix.multiply.v6f64.v12f64.v8f64(<12 x double> [[TMP1]], <8 x double> [[C:%.*]], i32 3, i32 4, i32 2)
+; CHECK-NEXT:    [[TMP3:%.*]] = fneg <6 x double> [[TMP2]]
+; CHECK-NEXT:    ret <6 x double> [[TMP3]]
 ;
   %a.neg = fneg <15 x double> %a
   %res = tail call <12 x double> @llvm.matrix.multiply.v12f64.v15f64.v20f64(<15 x double> %a.neg, <20 x double> %b, i32 3, i32 5, i32 4)
   %res.2 = tail call <6 x double> @llvm.matrix.multiply.v6f64.v12f64.v8f64(<12 x double> %res, <8 x double> %c, i32 3, i32 4, i32 2)
   ret <6 x double> %res.2
 }
 
 declare <2 x double> @llvm.matrix.multiply.v2f64.v6f64.v3f64(<6 x double>, <3 x double>, i32 immarg, i32 immarg, i32 immarg) #1