[Codegen] Support symmetric operations on complex numbers

[NickGuy]

Summary

This provides support for symmetric operations being performed on complex numbers, allowing for such patterns as o[i] = (x[i] * y[i]) + z[i]; and z[i] += x[i] * y[i] (where the expression isn't rearranged beyond the recognised patterns)

[dmgreen]

Sounds like a nice idea. It might be better not to represent it as "complex rotate with zero rotation", but as a binary operation that is applied equally to both real and imaginary values. Possibly a new ComplexDeinterleavingOperation type? It could then apply to any binary operator, so if you have R=fsub(a,b) and I=fsub(c,d), and it is valid to transform with identifyNode(a,c) and identifyNode(b,d), then it is valid to use a fsub on the interleaved version. It could run at the end of ComplexDeinterleavingGraph::identifyNode - if we haven't found anything else so far and the operations are equal binary operators then try and recurse into the operands. There are some unary operations too that might apply, like fneg and llvm.fabs intrinsics. I'm not sure how many of them commonly come up with complex numbers, but hopefully once some are added the others can be fairly simple additions.

[NickGuy]

I've changed the approach to not be so focused on "complex addition with zero rotation", and instead common operations applied to both sides (dubbed "symmetric operations"). The current implementation doesn't allow for targets to define their own supported operations, but also I'm not sure if there are many (if any at all) operations that wouldn't be common.

[pthariensflame]

llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll

355–507

Would it not be better to recognize the addend as part of the mla, thus saving an fadd? Or is that out of scope for this patch, or wrong for some subtle reason?

[dmgreen]

Sounds good. A few high level points:

• Some of the changes here look unrelated to these new operations. Can they be split out into a separate change.
• Add more tests for all the various instructions added so far.
• The creation of the new nodes doesn't need to happen in Arm/AArch64. It can be shared between the two in the pass.
• I'm not sure about the name Passthrough (a passthrough of what?), but I'm not sure I have a better suggestion. Maybe CommonOperation or just BinOp (even if they can also be UnaryOps)? It's not a very important point, but it is worth adding comments to explain the node type.

llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp

707–717

This checks various conditions about the operation, then checks for exact opcodes below, none of which will throw of have side effects. The switch table should be enough.

739

I may be missing it, but what checks that the two operations are the same?

llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll

355–507

That sounds like a good change, but I think it could be handled generically by the backend.

[NickGuy]

Split out unrelated changes, added more tests, unified the common logic, and renamed Passthrough to Symmetric.

In D142482#4098968, @dmgreen wrote:

This checks various conditions about the operation, then checks for exact opcodes below, none of which will throw of have side effects. The switch table should be enough.

It was an initial attempt to try and be clever, removed.

I may be missing it, but what checks that the two operations are the same?

You're not missing anything, it existed at one point, but it got removed in one of the iterations. I've readded it.

[dmgreen]

llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp

643

This needn't be here.

720

Formatting

852

This could be in the other cleanup patch.

907

Formatting.

[dmgreen]

If you can move the call to identifySymmetricOperation into this patch, then it LGTM. Cheers

llvm/include/llvm/CodeGen/ComplexDeinterleavingPass.h

public:
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 };
 
 enum class ComplexDeinterleavingOperation {
   CAdd,
   CMulPartial,
   // The following 'operations' are used to represent internal states. Backends
   // are not expected to try and support these in any capacity.
-  Shuffle
+  Shuffle,
+  Symmetric
 };
 
 enum class ComplexDeinterleavingRotation {
   Rotation_0 = 0,
   Rotation_90 = 1,
   Rotation_180 = 2,
   Rotation_270 = 3,
 };
 
 } // namespace llvm
 
 #endif // LLVM_CODEGEN_COMPLEXDEINTERLEAVING_H

llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp

private:
   /// Identifies a complex add pattern and its rotation, based on the following
   /// patterns.
   ///
   /// 90:  r: ar - bi
   ///      i: ai + br
   /// 270: r: ar + bi
   ///      i: ai - br
   NodePtr identifyAdd(Instruction *Real, Instruction *Imag);
+  NodePtr identifySymmetricOperation(Instruction *Real, Instruction *Imag);
 
   NodePtr identifyNode(Instruction *I, Instruction *J);
 
   Value *replaceNode(RawNodePtr Node);
 
 public:
   void dump() { dump(dbgs()); }
   void dump(raw_ostream &OS) {
ComplexDeinterleavingGraph::identifyPartialMul(Instruction *Real,
   Node->addOperand(CNode);
   return submitCompositeNode(Node);
 }
 
 ComplexDeinterleavingGraph::NodePtr
 ComplexDeinterleavingGraph::identifyAdd(Instruction *Real, Instruction *Imag) {
   LLVM_DEBUG(dbgs() << "identifyAdd " << *Real << " / " << *Imag << "\n");
 
   // Determine rotation
   ComplexDeinterleavingRotation Rotation;
   if ((Real->getOpcode() == Instruction::FSub &&
        Imag->getOpcode() == Instruction::FAdd) ||
       (Real->getOpcode() == Instruction::Sub &&
        Imag->getOpcode() == Instruction::Add))
     Rotation = ComplexDeinterleavingRotation::Rotation_90;
   else if ((Real->getOpcode() == Instruction::FAdd &&
             Imag->getOpcode() == Instruction::FSub) ||

 
 static bool isInstructionPairMul(Instruction *A, Instruction *B) {
   auto Pattern =
       m_BinOp(m_FMul(m_Value(), m_Value()), m_FMul(m_Value(), m_Value()));
 
   return match(A, Pattern) && match(B, Pattern);
 }
 
+static bool isInstructionPotentiallySymmetric(Instruction *I) {
+  switch (I->getOpcode()) {
+  case Instruction::FAdd:
+  case Instruction::FSub:
+  case Instruction::FMul:
+  case Instruction::FNeg:
+    return true;
+  default:
+    return false;
+  }
+}
+
+ComplexDeinterleavingGraph::NodePtr
+ComplexDeinterleavingGraph::identifySymmetricOperation(Instruction *Real,
+                                                       Instruction *Imag) {
+  if (Real->getOpcode() != Imag->getOpcode())
+    return nullptr;
+
+  if (!isInstructionPotentiallySymmetric(Real) ||
+      !isInstructionPotentiallySymmetric(Imag))
+    return nullptr;
+
+  auto *R0 = dyn_cast<Instruction>(Real->getOperand(0));
+  auto *I0 = dyn_cast<Instruction>(Imag->getOperand(0));
+
+  if (!R0 || !I0)
+    return nullptr;
+
+  NodePtr Op0 = identifyNode(R0, I0);
+  NodePtr Op1 = nullptr;
+  if (Op0 == nullptr)
+    return nullptr;
+
+  if (Real->isBinaryOp()) {
+    auto *R1 = dyn_cast<Instruction>(Real->getOperand(1));
+    auto *I1 = dyn_cast<Instruction>(Imag->getOperand(1));
+    if (!R1 || !I1)
+      return nullptr;
+
+    Op1 = identifyNode(R1, I1);
+    if (Op1 == nullptr)
+      return nullptr;
+  }
+
+  auto Node = prepareCompositeNode(ComplexDeinterleavingOperation::Symmetric,
+                                   Real, Imag);
+  Node->addOperand(Op0);
+  if (Real->isBinaryOp())
+    Node->addOperand(Op1);
+
+  return submitCompositeNode(Node);
+}
+
 ComplexDeinterleavingGraph::NodePtr
 ComplexDeinterleavingGraph::identifyNode(Instruction *Real, Instruction *Imag) {
   LLVM_DEBUG(dbgs() << "identifyNode on " << *Real << " / " << *Imag << "\n");
   if (NodePtr CN = getContainingComposite(Real, Imag)) {
     LLVM_DEBUG(dbgs() << " - Folding to existing node\n");
     return CN;
   }
 
if (RealShuffle && ImagShuffle) {
 
     NodePtr PlaceholderNode =
         prepareCompositeNode(llvm::ComplexDeinterleavingOperation::Shuffle,
                              RealShuffle, ImagShuffle);
     PlaceholderNode->ReplacementNode = RealShuffle->getOperand(0);
     return submitCompositeNode(PlaceholderNode);
   }
   if (RealShuffle || ImagShuffle) {
     LLVM_DEBUG(dbgs() << " - There's a shuffle where there shouldn't be.\n");
     return nullptr;
   }
 
   auto *VTy = cast<FixedVectorType>(Real->getType());
   auto *NewVTy =
       FixedVectorType::get(VTy->getScalarType(), VTy->getNumElements() * 2);
 
   if (TL->isComplexDeinterleavingOperationSupported(
           ComplexDeinterleavingOperation::CMulPartial, NewVTy) &&
       isInstructionPairMul(Real, Imag)) {
     return identifyPartialMul(Real, Imag);
   }
 
   if (TL->isComplexDeinterleavingOperationSupported(
           ComplexDeinterleavingOperation::CAdd, NewVTy) &&
       isInstructionPairAdd(Real, Imag)) {
     return identifyAdd(Real, Imag);
   }
 
-  return nullptr;
+  auto Symmetric = identifySymmetricOperation(Real, Imag);
+  LLVM_DEBUG(if (Symmetric == nullptr) dbgs()
+             << "  - Not recognised as a valid pattern.\n");
+  return Symmetric;
 }
 
 bool ComplexDeinterleavingGraph::identifyNodes(Instruction *RootI) {
   Instruction *Real;
   Instruction *Imag;
   if (!match(RootI, m_Shuffle(m_Instruction(Real), m_Instruction(Imag))))
     return false;
 
for (const auto &Node : CompositeNodes) {
     if (!Node->hasAllInternalUses(AllInstructions)) {
       LLVM_DEBUG(dbgs() << "  - Invalid internal uses\n");
       return false;
     }
   }
   return RootNode != nullptr;
 }
 
+static Value *replaceSymmetricNode(ComplexDeinterleavingGraph::RawNodePtr Node,
+                                   Value *InputA, Value *InputB) {
+  Instruction *I = Node->Real;
+  if (I->isUnaryOp())
+    assert(!InputB &&
+           "Unary symmetric operations need one input, but two were provided.");
+  else if (I->isBinaryOp())
+    assert(InputB && "Binary symmetric operations need two inputs, only one "
+                     "was provided.");
+
+  IRBuilder<> B(I);
+
+  switch (I->getOpcode()) {
+  case Instruction::FNeg:
+    return B.CreateFNeg(InputA);
+  case Instruction::FAdd:
+    return B.CreateFAdd(InputA, InputB);
+  case Instruction::FSub:
+    return B.CreateFSub(InputA, InputB);
+  case Instruction::FMul:
+    return B.CreateFMul(InputA, InputB);
+  }
+
+  return nullptr;
+}
+
 Value *ComplexDeinterleavingGraph::replaceNode(
     ComplexDeinterleavingGraph::RawNodePtr Node) {
   if (Node->ReplacementNode)
     return Node->ReplacementNode;
 
   Value *Input0 = replaceNode(Node->Operands[0]);
-  Value *Input1 = replaceNode(Node->Operands[1]);
+  Value *Input1 =
+      Node->Operands.size() > 1 ? replaceNode(Node->Operands[1]) : nullptr;
   Value *Accumulator =
       Node->Operands.size() > 2 ? replaceNode(Node->Operands[2]) : nullptr;
 
+  if (Input1)
     assert(Input0->getType() == Input1->getType() &&
            "Node inputs need to be of the same type");
 
+  if (Node->Operation == ComplexDeinterleavingOperation::Symmetric)
+    Node->ReplacementNode = replaceSymmetricNode(Node, Input0, Input1);
+  else
     Node->ReplacementNode = TL->createComplexDeinterleavingIR(
-      Node->Real, Node->Operation, Node->Rotation, Input0, Input1, Accumulator);
+        Node->Real, Node->Operation, Node->Rotation, Input0, Input1,
+        Accumulator);
 
   assert(Node->ReplacementNode && "Target failed to create Intrinsic call.");
   NumComplexTransformations += 1;
   return Node->ReplacementNode;
 }
 
 void ComplexDeinterleavingGraph::replaceNodes() {
   Value *R = replaceNode(RootNode.get());

llvm/test/CodeGen/AArch64/complex-deinterleaving-mixed-cases.ll

entry:
   store <4 x float> %otheruse, ptr %p
   %7 = fmul fast <2 x float> %6, %strided.vec
   %8 = fadd fast <2 x float> %3, %7
   %9 = fmul fast <2 x float> %2, %strided.vec
   %10 = fmul fast <2 x float> %6, %strided.vec35
   %11 = fsub fast <2 x float> %9, %10
   %interleaved.vec = shufflevector <2 x float> %11, <2 x float> %8, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
   ret <4 x float> %interleaved.vec
 }
+
+; Expected to transform
+define <4 x float> @mul_addequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_addequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fadd v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fadd fast <2 x float> %5, %c.real
+  %7 = fadd fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_subequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_subequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fsub v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fsub fast <2 x float> %5, %c.real
+  %7 = fsub fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+
+; Expected to transform
+define <4 x float> @mul_mulequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_mulequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v3.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fmul v0.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fmul fast <2 x float> %5, %c.real
+  %7 = fmul fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to not transform
+define <4 x float> @mul_divequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_divequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    ext v3.16b, v1.16b, v1.16b, #8
+; CHECK-NEXT:    ext v4.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    ext v16.16b, v2.16b, v2.16b, #8
+; CHECK-NEXT:    zip2 v5.2s, v1.2s, v3.2s
+; CHECK-NEXT:    zip1 v1.2s, v1.2s, v3.2s
+; CHECK-NEXT:    zip2 v6.2s, v0.2s, v4.2s
+; CHECK-NEXT:    zip1 v0.2s, v0.2s, v4.2s
+; CHECK-NEXT:    zip1 v4.2s, v2.2s, v16.2s
+; CHECK-NEXT:    zip2 v2.2s, v2.2s, v16.2s
+; CHECK-NEXT:    fmul v7.2s, v6.2s, v5.2s
+; CHECK-NEXT:    fneg v3.2s, v7.2s
+; CHECK-NEXT:    fmla v3.2s, v0.2s, v1.2s
+; CHECK-NEXT:    fmul v0.2s, v5.2s, v0.2s
+; CHECK-NEXT:    fmla v0.2s, v6.2s, v1.2s
+; CHECK-NEXT:    fdiv v3.2s, v3.2s, v4.2s
+; CHECK-NEXT:    fdiv v0.2s, v0.2s, v2.2s
+; CHECK-NEXT:    zip1 v0.4s, v3.4s, v0.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fdiv fast <2 x float> %5, %c.real
+  %7 = fdiv fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_negequal(<4 x float> %a, <4 x float> %b) {
+; CHECK-LABEL: mul_negequal:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    movi v2.2d, #0000000000000000
+; CHECK-NEXT:    fcmla v2.4s, v0.4s, v1.4s, #0
+; CHECK-NEXT:    fcmla v2.4s, v0.4s, v1.4s, #90
+; CHECK-NEXT:    fneg v0.4s, v2.4s
+; CHECK-NEXT:    ret
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %6 = fneg fast <2 x float> %5
+  %7 = fneg fast <2 x float> %2
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}

llvm/test/CodeGen/Thumb2/mve-complex-deinterleaving-mixed-cases.ll

entry:
   %7 = fmul fast <2 x float> %6, %strided.vec
   %8 = fadd fast <2 x float> %3, %7
   %9 = fmul fast <2 x float> %2, %strided.vec
   %10 = fmul fast <2 x float> %6, %strided.vec35
   %11 = fsub fast <2 x float> %9, %10
   %interleaved.vec = shufflevector <2 x float> %11, <2 x float> %8, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
   ret <4 x float> %interleaved.vec
 }
+
+; Expected to transform
+define <4 x float> @mul_addequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_addequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vadd.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fadd fast <2 x float> %5, %c.real
+  %7 = fadd fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_subequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_subequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vsub.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fsub fast <2 x float> %5, %c.real
+  %7 = fsub fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+
+; Expected to transform
+define <4 x float> @mul_mulequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_mulequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r1, sp
+; CHECK-NEXT:    vldrw.u32 q2, [r1]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    add r0, sp, #16
+; CHECK-NEXT:    vcmul.f32 q3, q0, q2, #0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vcmla.f32 q3, q0, q2, #90
+; CHECK-NEXT:    vmul.f32 q0, q3, q1
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fmul fast <2 x float> %5, %c.real
+  %7 = fmul fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to not transform
+define <4 x float> @mul_divequal(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: mul_divequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    .vsave {d10, d11}
+; CHECK-NEXT:    vpush {d10, d11}
+; CHECK-NEXT:    .vsave {d8}
+; CHECK-NEXT:    vpush {d8}
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    add r0, sp, #24
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    vmov.f32 s16, s1
+; CHECK-NEXT:    add.w r12, sp, #40
+; CHECK-NEXT:    vmov.f32 s12, s5
+; CHECK-NEXT:    vmov.f32 s13, s7
+; CHECK-NEXT:    vmov.f32 s1, s2
+; CHECK-NEXT:    vmov.f32 s8, s4
+; CHECK-NEXT:    vmul.f32 q5, q3, q0
+; CHECK-NEXT:    vmov.f32 s9, s6
+; CHECK-NEXT:    vldrw.u32 q1, [r12]
+; CHECK-NEXT:    vmov.f32 s17, s3
+; CHECK-NEXT:    vfma.f32 q5, q2, q4
+; CHECK-NEXT:    vmul.f32 q3, q4, q3
+; CHECK-NEXT:    vdiv.f32 s3, s21, s7
+; CHECK-NEXT:    vneg.f32 q3, q3
+; CHECK-NEXT:    vfma.f32 q3, q2, q0
+; CHECK-NEXT:    vdiv.f32 s1, s20, s5
+; CHECK-NEXT:    vdiv.f32 s2, s13, s6
+; CHECK-NEXT:    vdiv.f32 s0, s12, s4
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    vpop {d8}
+; CHECK-NEXT:    vpop {d10, d11}
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %c.real = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %c.imag = shufflevector <4 x float> %c, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %6 = fdiv fast <2 x float> %5, %c.real
+  %7 = fdiv fast <2 x float> %2, %c.imag
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}
+
+; Expected to transform
+define <4 x float> @mul_negequal(<4 x float> %a, <4 x float> %b) {
+; CHECK-LABEL: mul_negequal:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vmov d0, r0, r1
+; CHECK-NEXT:    mov r0, sp
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vmov d1, r2, r3
+; CHECK-NEXT:    vcmul.f32 q2, q0, q1, #0
+; CHECK-NEXT:    vcmla.f32 q2, q0, q1, #90
+; CHECK-NEXT:    vneg.f32 q0, q2
+; CHECK-NEXT:    vmov r0, r1, d0
+; CHECK-NEXT:    vmov r2, r3, d1
+; CHECK-NEXT:    bx lr
+entry:
+  %strided.vec = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %a.imag = shufflevector <4 x float> %a, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %b.real = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 0, i32 2>
+  %b.imag = shufflevector <4 x float> %b, <4 x float> poison, <2 x i32> <i32 1, i32 3>
+  %0 = fmul fast <2 x float> %b.imag, %strided.vec
+  %1 = fmul fast <2 x float> %b.real, %a.imag
+  %2 = fadd fast <2 x float> %1, %0
+  %3 = fmul fast <2 x float> %b.real, %strided.vec
+  %4 = fmul fast <2 x float> %a.imag, %b.imag
+  %5 = fsub fast <2 x float> %3, %4
+  %6 = fneg fast <2 x float> %5
+  %7 = fneg fast <2 x float> %2
+  %interleaved.vec = shufflevector <2 x float> %6, <2 x float> %7, <4 x i32> <i32 0, i32 2, i32 1, i32 3>
+  ret <4 x float> %interleaved.vec
+}