[AArch64] Add patterns for FMADD, FMSUB

[overmighty]

Summary

FMADD, FMSUB instructions perform better or the same compared to indexed
FMLA, FMLS.

For example, the Arm Cortex-A55 Software Optimization Guide lists "FP
multiply accumulate" FMADD, FMSUB instructions with a throughput of 2
IPC, whereas it lists "ASIMD FP multiply accumulate, by element" FMLA,
FMLS with a throughput of 1 IPC.

The Arm Cortex-A77 Software Optimization Guide, however, does not
separately list "by element" variants of the "ASIMD FP multiply
accumulate" instructions, which are listed with the same throughput of 2
IPC as "FP multiply accumulate" instructions.

[dmgreen]

It looks like there are patterns for A[0]*B+C and B*A[0]*C to handle commutivity. Are there tests for both forms too?

[samtebbs]

Thanks, this looks good to me with any extra tests required by David's comment.

[overmighty]

Add tests for A[0]*B+C patterns.

[overmighty]

If this still looks good to you, please commit it as "OverMighty <[e-mail]>". Thank you.

[overmighty]

The crashes in complex-deinterleaving-f16-mul.ll and fp16_intrinsic_lane.ll seem to be caused by D156831.

[dzhidzhoev]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5381–5393

The result type of (EXTRACT_SUBREG ...) here seems to be deduced to i16, which causes an assertion after applying this rule.
Explicit specification of the result type fixes crashes of tests:

FPR16:$Rn, (f16 (EXTRACT_SUBREG V128:$Rm, hsub)), FPR16:$Ra)>;

(f16 (EXTRACT_SUBREG V128:$Rn, hsub)), FPR16:$Rm, FPR16:$Ra)>

[dzhidzhoev]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5393

BTW, these lines add some patterns for fnmadd. Could you add some tests for them? Or are they already covered by existing ones?

[dzhidzhoev]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5418

Is it possible to use extractelt here? Since vector_extract is marked as deprecated in TargetSelectionDAG.td

[overmighty]

• Rebase new upstream commits.
• Replace usage of deprecated vector_extract with extractelt in new patterns.
• Add f16 operators before EXTRACT_SUBREGs in new *Hrrr instruction pattern results to fix crashing tests.

[overmighty]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5393

The patterns for FNMADD and FNMSUB are useless as there are no Neon vector equivalents of these instructions. Should I add a template argument to ThreeOperandFPData to prevent the patterns from being generated for them?

I previously had patterns in AArch64InstrInfo.td for FMADD and FMSUB only and it was suggested to me to move them to ThreeOperandFPData: https://reviews.llvm.org/D153207?id=532433#inline-1481961. I was asked to split that part of my previous patch into a second (this) one.

5418

I saw the comment marking it as deprecated but I also saw new commits using vector_extract. I should have asked for clarification earlier. Is vector_extract truly deprecated? Should I change patterns I added in previous patches to use extractelt too?

[dzhidzhoev]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

5393

The patterns for FNMADD and FNMSUB are useless as there are no Neon vector equivalents of these instructions. Should I add a template argument to ThreeOperandFPData to prevent the patterns from being generated for them?

Thank you for the explanation! There's no need for it since I can barely find a way it can interfere with other patterns.

5418

Thank you! Now I've noticed that the comment about the deprecation was made in 2015, and vector_extract is still in use. @arsenm, could you suggest whether it is relevant?

[overmighty]

Ping.

If you would like the current patch with extractelt to be committed, please commit it as "OverMighty <[e-mail]>". Thank you.

clang/test/CodeGen/aarch64-neon-scalar-x-indexed-elem-constrained.c

 // COMMONIR:        [[TMP1:%.*]] = bitcast <1 x double> %b to <8 x i8>
 // COMMONIR:        [[TMP2:%.*]] = bitcast <2 x double> %v to <16 x i8>
 // COMMONIR:        [[TMP3:%.*]] = bitcast <8 x i8> [[TMP0]] to double
 // COMMONIR:        [[TMP4:%.*]] = bitcast <8 x i8> [[TMP1]] to double
 // COMMONIR:        [[TMP5:%.*]] = bitcast <16 x i8> [[TMP2]] to <2 x double>
 // COMMONIR:        [[EXTRACT:%.*]] = extractelement <2 x double> [[TMP5]], i32 0
 // UNCONSTRAINED:   [[TMP6:%.*]] = call double @llvm.fma.f64(double [[TMP4]], double [[EXTRACT]], double [[TMP3]])
 // CONSTRAINED:     [[TMP6:%.*]] = call double @llvm.experimental.constrained.fma.f64(double [[TMP4]], double [[EXTRACT]], double [[TMP3]], metadata !"round.tonearest", metadata !"fpexcept.strict")
-// CHECK-ASM:       fmla d{{[0-9]+}}, d{{[0-9]+}}, v{{[0-9]+}}.d[{{[0-9]+}}]
+// CHECK-ASM:       fmadd d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
 // COMMONIR:        [[TMP7:%.*]] = bitcast double [[TMP6]] to <1 x double>
 // COMMONIR:        ret <1 x double> [[TMP7]]
 float64x1_t test_vfma_laneq_f64(float64x1_t a, float64x1_t b, float64x2_t v) {
   return vfma_laneq_f64(a, b, v, 0);
 }
 
 // COMMON-LABEL: test_vfms_laneq_f64
 // COMMONIR:        [[SUB:%.*]] = fneg <1 x double> %b
 // CHECK-ASM:       fneg d{{[0-9]+}}, d{{[0-9]+}}
 // COMMONIR:        [[TMP0:%.*]] = bitcast <1 x double> %a to <8 x i8>
 // COMMONIR:        [[TMP1:%.*]] = bitcast <1 x double> [[SUB]] to <8 x i8>
 // COMMONIR:        [[TMP2:%.*]] = bitcast <2 x double> %v to <16 x i8>
 // COMMONIR:        [[TMP3:%.*]] = bitcast <8 x i8> [[TMP0]] to double
 // COMMONIR:        [[TMP4:%.*]] = bitcast <8 x i8> [[TMP1]] to double
 // COMMONIR:        [[TMP5:%.*]] = bitcast <16 x i8> [[TMP2]] to <2 x double>
 // COMMONIR:        [[EXTRACT:%.*]] = extractelement <2 x double> [[TMP5]], i32 0
 // UNCONSTRAINED:   [[TMP6:%.*]] = call double @llvm.fma.f64(double [[TMP4]], double [[EXTRACT]], double [[TMP3]])
 // CONSTRAINED:     [[TMP6:%.*]] = call double @llvm.experimental.constrained.fma.f64(double [[TMP4]], double [[EXTRACT]], double [[TMP3]], metadata !"round.tonearest", metadata !"fpexcept.strict")
-// CHECK-ASM:       fmla d{{[0-9]+}}, d{{[0-9]+}}, v{{[0-9]+}}.d[{{[0-9]+}}]
+// CHECK-ASM:       fmadd d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
 // COMMONIR:        [[TMP7:%.*]] = bitcast double [[TMP6]] to <1 x double>
 // COMMONIR:        ret <1 x double> [[TMP7]]
 float64x1_t test_vfms_laneq_f64(float64x1_t a, float64x1_t b, float64x2_t v) {
   return vfms_laneq_f64(a, b, v, 0);
 }
 

llvm/lib/Target/AArch64/AArch64InstrFormats.td

def Srrr : BaseThreeOperandFPData<isNegated, isSub, FPR32, asm,
     let Inst{23-22} = 0b00; // 32-bit size flag
   }
 
   def Drrr : BaseThreeOperandFPData<isNegated, isSub, FPR64, asm,
             [(set FPR64:$Rd,
                   (node (f64 FPR64:$Rn), (f64 FPR64:$Rm), (f64 FPR64:$Ra)))]> {
     let Inst{23-22} = 0b01; // 64-bit size flag
   }
+
+  let Predicates = [HasFullFP16] in {
+  def : Pat<(f16 (node (f16 FPR16:$Rn),
+                       (f16 (vector_extract (v8f16 V128:$Rm), (i64 0))),
+                       (f16 FPR16:$Ra))),
+            (!cast<Instruction>(NAME # Hrrr)
+              FPR16:$Rn, (EXTRACT_SUBREG V128:$Rm, hsub), FPR16:$Ra)>;
+
+  def : Pat<(f16 (node (f16 (vector_extract (v8f16 V128:$Rn), (i64 0))),
+                       (f16 FPR16:$Rm),
+                       (f16 FPR16:$Ra))),
+            (!cast<Instruction>(NAME # Hrrr)
+              (EXTRACT_SUBREG V128:$Rn, hsub), FPR16:$Rm, FPR16:$Ra)>;
+  }
+
+  def : Pat<(f32 (node (f32 FPR32:$Rn),
+                       (f32 (vector_extract (v4f32 V128:$Rm), (i64 0))),
+                       (f32 FPR32:$Ra))),
+            (!cast<Instruction>(NAME # Srrr)
+              FPR32:$Rn, (EXTRACT_SUBREG V128:$Rm, ssub), FPR32:$Ra)>;
+
+  def : Pat<(f32 (node (f32 (vector_extract (v4f32 V128:$Rn), (i64 0))),
+                       (f32 FPR32:$Rm),
+                       (f32 FPR32:$Ra))),
+            (!cast<Instruction>(NAME # Srrr)
+              (EXTRACT_SUBREG V128:$Rn, ssub), FPR32:$Rm, FPR32:$Ra)>;
+
+  def : Pat<(f64 (node (f64 FPR64:$Rn),
+                       (f64 (vector_extract (v2f64 V128:$Rm), (i64 0))),
+                       (f64 FPR64:$Ra))),
+            (!cast<Instruction>(NAME # Drrr)
+              FPR64:$Rn, (EXTRACT_SUBREG V128:$Rm, dsub), FPR64:$Ra)>;
+
+  def : Pat<(f64 (node (f64 (vector_extract (v2f64 V128:$Rn), (i64 0))),
+                       (f64 FPR64:$Rm),
+                       (f64 FPR64:$Ra))),
+            (!cast<Instruction>(NAME # Drrr)
+              (EXTRACT_SUBREG V128:$Rn, dsub), FPR64:$Rm, FPR64:$Ra)>;
 }
 
 //---
 // Floating point data comparisons
 //---
 
 let mayLoad = 0, mayStore = 0, hasSideEffects = 0, mayRaiseFPException = 1, Uses = [FPCR] in
 class BaseOneOperandFPComparison<bit signalAllNans,
                                  RegisterClass regtype, string asm,

llvm/test/CodeGen/AArch64/complex-deinterleaving-f16-mul.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s --mattr=+complxnum,+neon,+fullfp16 -o - | FileCheck %s
 
 target triple = "aarch64"
 
 ; Expected to not transform
 define <2 x half> @complex_mul_v2f16(<2 x half> %a, <2 x half> %b) {
 ; CHECK-LABEL: complex_mul_v2f16:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    // kill: def $d1 killed $d1 def $q1
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    mov h3, v0.h[1]
-; CHECK-NEXT:    mov h2, v1.h[1]
-; CHECK-NEXT:    fmul h4, h0, v1.h[1]
-; CHECK-NEXT:    fnmul h2, h3, h2
-; CHECK-NEXT:    fmla h4, h3, v1.h[0]
-; CHECK-NEXT:    fmla h2, h0, v1.h[0]
-; CHECK-NEXT:    mov v2.h[1], v4.h[0]
-; CHECK-NEXT:    fmov d0, d2
+; CHECK-NEXT:    mov h2, v0.h[1]
+; CHECK-NEXT:    // kill: def $d1 killed $d1 def $q1
+; CHECK-NEXT:    fmul h3, h0, v1.h[1]
+; CHECK-NEXT:    fmul h4, h2, v1.h[1]
+; CHECK-NEXT:    fmadd h2, h1, h2, h3
+; CHECK-NEXT:    fnmsub h0, h1, h0, h4
+; CHECK-NEXT:    mov v0.h[1], v2.h[0]
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
 entry:
   %a.real   = shufflevector <2 x half> %a, <2 x half> poison, <1 x i32> <i32 0>
   %a.imag = shufflevector <2 x half> %a, <2 x half> poison, <1 x i32> <i32 1>
   %b.real = shufflevector <2 x half> %b, <2 x half> poison, <1 x i32> <i32 0>
   %b.imag = shufflevector <2 x half> %b, <2 x half> poison, <1 x i32> <i32 1>
   %0 = fmul fast <1 x half> %b.imag, %a.real
   %1 = fmul fast <1 x half> %b.real, %a.imag

llvm/test/CodeGen/AArch64/fp16_intrinsic_lane.ll

 ; CHECK-NEXT:    ret
 entry:
   %vecinit = insertelement <8 x half> undef, half %c, i32 0
   %vecinit7 = shufflevector <8 x half> %vecinit, <8 x half> undef, <8 x i32> zeroinitializer
   %0 = tail call <8 x half> @llvm.fma.v8f16(<8 x half> %b, <8 x half> %vecinit7, <8 x half> %a) #4
   ret <8 x half> %0
 }
 
-define dso_local half @t_vfmah_lane_f16(half %a, half %b, <4 x half> %c, i32 %lane) {
-; CHECK-LABEL: t_vfmah_lane_f16:
+define dso_local half @t_vfmah_lane_f16_0(half %a, half %b, <4 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmah_lane_f16_0:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d2 killed $d2 def $q2
-; CHECK-NEXT:    fmla h0, h1, v2.h[0]
+; CHECK-NEXT:    fmadd h0, h1, h2, h0
 ; CHECK-NEXT:    ret
 entry:
   %extract = extractelement <4 x half> %c, i32 0
   %0 = tail call half @llvm.fma.f16(half %b, half %extract, half %a)
   ret half %0
 }
 
-define dso_local half @t_vfmah_laneq_f16(half %a, half %b, <8 x half> %c, i32 %lane) {
-; CHECK-LABEL: t_vfmah_laneq_f16:
+define dso_local half @t_vfmah_lane_f16_3(half %a, half %b, <4 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmah_lane_f16_3:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    fmla h0, h1, v2.h[0]
+; CHECK-NEXT:    // kill: def $d2 killed $d2 def $q2
+; CHECK-NEXT:    fmla h0, h1, v2.h[3]
+; CHECK-NEXT:    ret
+entry:
+  %extract = extractelement <4 x half> %c, i32 3
+  %0 = tail call half @llvm.fma.f16(half %b, half %extract, half %a)
+  ret half %0
+}
+
+define dso_local half @t_vfmah_laneq_f16_0(half %a, half %b, <8 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmah_laneq_f16_0:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    fmadd h0, h1, h2, h0
 ; CHECK-NEXT:    ret
 entry:
   %extract = extractelement <8 x half> %c, i32 0
   %0 = tail call half @llvm.fma.f16(half %b, half %extract, half %a)
   ret half %0
 }
 
+define dso_local half @t_vfmah_laneq_f16_7(half %a, half %b, <8 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmah_laneq_f16_7:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    fmla h0, h1, v2.h[7]
+; CHECK-NEXT:    ret
+entry:
+  %extract = extractelement <8 x half> %c, i32 7
+  %0 = tail call half @llvm.fma.f16(half %b, half %extract, half %a)
+  ret half %0
+}
+
 define dso_local <4 x half> @t_vfms_lane_f16(<4 x half> %a, <4 x half> %b, <4 x half> %c, i32 %lane) {
 ; CHECK-LABEL: t_vfms_lane_f16:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d2 killed $d2 def $q2
 ; CHECK-NEXT:    fmls v0.4h, v1.4h, v2.h[0]
 ; CHECK-NEXT:    ret
 entry:
   %sub = fsub <4 x half> <half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000>, %b

 entry:
   %sub = fsub <8 x half> <half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000, half 0xH8000>, %b
   %vecinit = insertelement <8 x half> undef, half %c, i32 0
   %vecinit7 = shufflevector <8 x half> %vecinit, <8 x half> undef, <8 x i32> zeroinitializer
   %0 = tail call <8 x half> @llvm.fma.v8f16(<8 x half> %sub, <8 x half> %vecinit7, <8 x half> %a) #4
   ret <8 x half> %0
 }
 
-define dso_local half @t_vfmsh_lane_f16(half %a, half %b, <4 x half> %c, i32 %lane) {
-; CHECK-LABEL: t_vfmsh_lane_f16:
+define dso_local half @t_vfmsh_lane_f16_0(half %a, half %b, <4 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmsh_lane_f16_0:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d2 killed $d2 def $q2
-; CHECK-NEXT:    fmls h0, h1, v2.h[0]
+; CHECK-NEXT:    fmsub h0, h2, h1, h0
 ; CHECK-NEXT:    ret
 entry:
   %0 = fsub half 0xH8000, %b
   %extract = extractelement <4 x half> %c, i32 0
   %1 = tail call half @llvm.fma.f16(half %0, half %extract, half %a)
   ret half %1
 }
 
-define dso_local half @t_vfmsh_laneq_f16(half %a, half %b, <8 x half> %c, i32 %lane) {
-; CHECK-LABEL: t_vfmsh_laneq_f16:
+define dso_local half @t_vfmsh_lane_f16_3(half %a, half %b, <4 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmsh_lane_f16_3:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    // kill: def $d2 killed $d2 def $q2
+; CHECK-NEXT:    fmls h0, h1, v2.h[3]
+; CHECK-NEXT:    ret
+entry:
+  %0 = fsub half 0xH8000, %b
+  %extract = extractelement <4 x half> %c, i32 3
+  %1 = tail call half @llvm.fma.f16(half %0, half %extract, half %a)
+  ret half %1
+}
+
+define dso_local half @t_vfmsh_laneq_f16_0(half %a, half %b, <8 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmsh_laneq_f16_0:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    fmls h0, h1, v2.h[0]
+; CHECK-NEXT:    fmsub h0, h2, h1, h0
 ; CHECK-NEXT:    ret
 entry:
   %0 = fsub half 0xH8000, %b
   %extract = extractelement <8 x half> %c, i32 0
   %1 = tail call half @llvm.fma.f16(half %0, half %extract, half %a)
   ret half %1
 }
 
+define dso_local half @t_vfmsh_laneq_f16_7(half %a, half %b, <8 x half> %c, i32 %lane) {
+; CHECK-LABEL: t_vfmsh_laneq_f16_7:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    fmls h0, h1, v2.h[7]
+; CHECK-NEXT:    ret
+entry:
+  %0 = fsub half 0xH8000, %b
+  %extract = extractelement <8 x half> %c, i32 7
+  %1 = tail call half @llvm.fma.f16(half %0, half %extract, half %a)
+  ret half %1
+}
+
 define dso_local <4 x half> @t_vmul_laneq_f16(<4 x half> %a, <8 x half> %b, i32 %lane) {
 ; CHECK-LABEL: t_vmul_laneq_f16:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fmul v0.4h, v0.4h, v1.h[0]
 ; CHECK-NEXT:    ret
 entry:
   %shuffle = shufflevector <8 x half> %b, <8 x half> undef, <4 x i32> zeroinitializer
   %mul = fmul <4 x half> %shuffle, %a

llvm/test/CodeGen/AArch64/neon-scalar-by-elem-fma.ll

 ; RUN: llc < %s -verify-machineinstrs -mtriple=aarch64-none-linux-gnu -mattr=+neon -fp-contract=fast | FileCheck %s
 
 attributes #0 = { strictfp }
 
 declare float @llvm.fma.f32(float, float, float)
 declare double @llvm.fma.f64(double, double, double)
 declare float @llvm.experimental.constrained.fma.f32(float, float, float, metadata, metadata)
 declare double @llvm.experimental.constrained.fma.f64(double, double, double, metadata, metadata)
 
-define float @test_fmla_ss4S(float %a, float %b, <4 x float> %v) {
-  ; CHECK-LABEL: test_fmla_ss4S
+define float @test_fmla_ss4S_0(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmla_ss4S_0
+  ; CHECK: fmadd s0, s1, s2, s0
+  %tmp1 = extractelement <4 x float> %v, i32 0
+  %tmp2 = call float @llvm.fma.f32(float %b, float %tmp1, float %a)
+  ret float %tmp2
+}
+
+define float @test_fmla_ss4S_3(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmla_ss4S_3
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = call float @llvm.fma.f32(float %b, float %tmp1, float %a)
   ret float %tmp2
 }
 
-define float @test_fmla_ss4S_swap(float %a, float %b, <4 x float> %v) {
-  ; CHECK-LABEL: test_fmla_ss4S_swap
+define float @test_fmla_ss4S_3_swap(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmla_ss4S_3_swap
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = call float @llvm.fma.f32(float %tmp1, float %a, float %a)
   ret float %tmp2
 }
 
-define float @test_fmla_ss2S(float %a, float %b, <2 x float> %v) {
-  ; CHECK-LABEL: test_fmla_ss2S
+define float @test_fmla_ss2S_0(float %a, float %b, <2 x float> %v) {
+  ; CHECK-LABEL: test_fmla_ss2S_0
+  ; CHECK: fmadd s0, s1, s2, s0
+  %tmp1 = extractelement <2 x float> %v, i32 0
+  %tmp2 = call float @llvm.fma.f32(float %b, float %tmp1, float %a)
+  ret float %tmp2
+}
+
+define float @test_fmla_ss2S_1(float %a, float %b, <2 x float> %v) {
+  ; CHECK-LABEL: test_fmla_ss2S_1
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
   %tmp1 = extractelement <2 x float> %v, i32 1
   %tmp2 = call float @llvm.fma.f32(float %b, float %tmp1, float %a)
   ret float %tmp2
 }
 
-define double @test_fmla_ddD(double %a, double %b, <1 x double> %v) {
-  ; CHECK-LABEL: test_fmla_ddD
-  ; CHECK: {{fmla d[0-9]+, d[0-9]+, v[0-9]+.d\[0]|fmadd d[0-9]+, d[0-9]+, d[0-9]+, d[0-9]+}}
+define double @test_fmla_ddD_0(double %a, double %b, <1 x double> %v) {
+  ; CHECK-LABEL: test_fmla_ddD_0
+  ; CHECK: fmadd d0, d1, d2, d0
   %tmp1 = extractelement <1 x double> %v, i32 0
   %tmp2 = call double @llvm.fma.f64(double %b, double %tmp1, double %a)
   ret double %tmp2
 }
 
-define double @test_fmla_dd2D(double %a, double %b, <2 x double> %v) {
-  ; CHECK-LABEL: test_fmla_dd2D
+define double @test_fmla_dd2D_0(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmla_dd2D_0
+  ; CHECK: fmadd d0, d1, d2, d0
+  %tmp1 = extractelement <2 x double> %v, i32 0
+  %tmp2 = call double @llvm.fma.f64(double %b, double %tmp1, double %a)
+  ret double %tmp2
+}
+
+define double @test_fmla_dd2D_1(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmla_dd2D_1
   ; CHECK: fmla {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = call double @llvm.fma.f64(double %b, double %tmp1, double %a)
   ret double %tmp2
 }
 
-define double @test_fmla_dd2D_swap(double %a, double %b, <2 x double> %v) {
-  ; CHECK-LABEL: test_fmla_dd2D_swap
+define double @test_fmla_dd2D_1_swap(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmla_dd2D_1_swap
   ; CHECK: fmla {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = call double @llvm.fma.f64(double %tmp1, double %b, double %a)
   ret double %tmp2
 }
 
-define float @test_fmls_ss4S(float %a, float %b, <4 x float> %v) {
-  ; CHECK-LABEL: test_fmls_ss4S
+define float @test_fmls_ss4S_0(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmls_ss4S_0
+  ; CHECK: fmsub s0, s2, s1, s0
+entry:
+  %fneg = fneg float %b
+  %extract = extractelement <4 x float> %v, i64 0
+  %0 = tail call float @llvm.fma.f32(float %fneg, float %extract, float %a)
+  ret float %0
+}
+
+define float @test_fmls_ss4S_3(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmls_ss4S_3
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = fsub float -0.0, %tmp1
   %tmp3 = call float @llvm.fma.f32(float %tmp2, float %tmp1, float %a)
   ret float %tmp3
 }
 
-define float @test_fmls_ss4S_swap(float %a, float %b, <4 x float> %v) {
-  ; CHECK-LABEL: test_fmls_ss4S_swap
+define float @test_fmls_ss4S_3_swap(float %a, float %b, <4 x float> %v) {
+  ; CHECK-LABEL: test_fmls_ss4S_3_swap
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = fsub float -0.0, %tmp1
   %tmp3 = call float @llvm.fma.f32(float %tmp1, float %tmp2, float %a)
   ret float %tmp3
 }
 
 
-define float @test_fmls_ss2S(float %a, float %b, <2 x float> %v) {
-  ; CHECK-LABEL: test_fmls_ss2S
+define float @test_fmls_ss2S_0(float %a, float %b, <2 x float> %v) {
+  ; CHECK-LABEL: test_fmls_ss2S_0
+  ; CHECK: fmsub s0, s2, s1, s0
+entry:
+  %fneg = fneg float %b
+  %extract = extractelement <2 x float> %v, i64 0
+  %0 = tail call float @llvm.fma.f32(float %fneg, float %extract, float %a)
+  ret float %0
+}
+
+define float @test_fmls_ss2S_1(float %a, float %b, <2 x float> %v) {
+  ; CHECK-LABEL: test_fmls_ss2S_1
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
   %tmp1 = extractelement <2 x float> %v, i32 1
   %tmp2 = fsub float -0.0, %tmp1
   %tmp3 = call float @llvm.fma.f32(float %tmp2, float %tmp1, float %a)
   ret float %tmp3
 }
 
-define double @test_fmls_ddD(double %a, double %b, <1 x double> %v) {
-  ; CHECK-LABEL: test_fmls_ddD
-  ; CHECK: {{fmls d[0-9]+, d[0-9]+, v[0-9]+.d\[0]|fmsub d[0-9]+, d[0-9]+, d[0-9]+, d[0-9]+}}
-  %tmp1 = extractelement <1 x double> %v, i32 0
-  %tmp2 = fsub double -0.0, %tmp1
-  %tmp3 = call double @llvm.fma.f64(double %tmp2, double %tmp1, double %a)
-  ret double %tmp3
+define double @test_fmls_ddD_0(double %a, double %b, <1 x double> %v) {
+  ; CHECK-LABEL: test_fmls_ddD_0
+  ; CHECK: fmsub d0, d1, d2, d0
+entry:
+  %fneg = fneg double %b
+  %extract = extractelement <1 x double> %v, i64 0
+  %0 = tail call double @llvm.fma.f64(double %fneg, double %extract, double %a)
+  ret double %0
+}
+
+define double @test_fmls_dd2D_0(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmls_dd2D_0
+  ; CHECK: fmsub d0, d2, d1, d0
+entry:
+  %fneg = fneg double %b
+  %extract = extractelement <2 x double> %v, i64 0
+  %0 = tail call double @llvm.fma.f64(double %fneg, double %extract, double %a)
+  ret double %0
 }
 
-define double @test_fmls_dd2D(double %a, double %b, <2 x double> %v) {
-  ; CHECK-LABEL: test_fmls_dd2D
+define double @test_fmls_dd2D_1(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmls_dd2D_1
   ; CHECK: fmls {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = fsub double -0.0, %tmp1
   %tmp3 = call double @llvm.fma.f64(double %tmp2, double %tmp1, double %a)
   ret double %tmp3
 }
 
-define double @test_fmls_dd2D_swap(double %a, double %b, <2 x double> %v) {
-  ; CHECK-LABEL: test_fmls_dd2D_swap
+define double @test_fmls_dd2D_1_swap(double %a, double %b, <2 x double> %v) {
+  ; CHECK-LABEL: test_fmls_dd2D_1_swap
   ; CHECK: fmls {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = fsub double -0.0, %tmp1
   %tmp3 = call double @llvm.fma.f64(double %tmp1, double %tmp2, double %a)
   ret double %tmp3
 }
 
-define float @test_fmla_ss4S_strict(float %a, float %b, <4 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmla_ss4S_strict
+define float @test_fmla_ss4S_0_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ss4S_0_strict
+  ; CHECK: fmadd s0, s1, s2, s0
+  %tmp1 = extractelement <4 x float> %v, i32 0
+  %tmp2 = call float @llvm.experimental.constrained.fma.f32(float %b, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret float %tmp2
+}
+
+define float @test_fmla_ss4S_3_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ss4S_3_strict
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = call float @llvm.experimental.constrained.fma.f32(float %b, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp2
 }
 
-define float @test_fmla_ss4S_swap_strict(float %a, float %b, <4 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmla_ss4S_swap_strict
+define float @test_fmla_ss4S_3_swap_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ss4S_3_swap_strict
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = call float @llvm.experimental.constrained.fma.f32(float %tmp1, float %a, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp2
 }
 
-define float @test_fmla_ss2S_strict(float %a, float %b, <2 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmla_ss2S_strict
+define float @test_fmla_ss2S_0_strict(float %a, float %b, <2 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ss2S_0_strict
+  ; CHECK: fmadd s0, s1, s2, s0
+  %tmp1 = extractelement <2 x float> %v, i32 0
+  %tmp2 = call float @llvm.experimental.constrained.fma.f32(float %b, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret float %tmp2
+}
+
+define float @test_fmla_ss2S_1_strict(float %a, float %b, <2 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ss2S_1_strict
   ; CHECK: fmla {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
   %tmp1 = extractelement <2 x float> %v, i32 1
   %tmp2 = call float @llvm.experimental.constrained.fma.f32(float %b, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp2
 }
 
-define double @test_fmla_ddD_strict(double %a, double %b, <1 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmla_ddD_strict
-  ; CHECK: {{fmla d[0-9]+, d[0-9]+, v[0-9]+.d\[0]|fmadd d[0-9]+, d[0-9]+, d[0-9]+, d[0-9]+}}
+define double @test_fmla_ddD_0_strict(double %a, double %b, <1 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmla_ddD_0_strict
+  ; CHECK: fmadd d0, d1, d2, d0
   %tmp1 = extractelement <1 x double> %v, i32 0
   %tmp2 = call double @llvm.experimental.constrained.fma.f64(double %b, double %tmp1, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret double %tmp2
 }
 
-define double @test_fmla_dd2D_strict(double %a, double %b, <2 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmla_dd2D_strict
+define double @test_fmla_dd2D_0_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmla_dd2D_0_strict
+  ; CHECK: fmadd d0, d1, d2, d0
+  %tmp1 = extractelement <2 x double> %v, i32 0
+  %tmp2 = call double @llvm.experimental.constrained.fma.f64(double %b, double %tmp1, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret double %tmp2
+}
+
+define double @test_fmla_dd2D_1_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmla_dd2D_1_strict
   ; CHECK: fmla {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = call double @llvm.experimental.constrained.fma.f64(double %b, double %tmp1, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret double %tmp2
 }
 
-define double @test_fmla_dd2D_swap_strict(double %a, double %b, <2 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmla_dd2D_swap_strict
+define double @test_fmla_dd2D_1_swap_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmla_dd2D_1_swap_strict
   ; CHECK: fmla {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = call double @llvm.experimental.constrained.fma.f64(double %tmp1, double %b, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret double %tmp2
 }
 
-define float @test_fmls_ss4S_strict(float %a, float %b, <4 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmls_ss4S_strict
+define float @test_fmls_ss4S_0_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ss4S_0_strict
+  ; CHECK: fmsub s0, s2, s1, s0
+entry:
+  %fneg = fneg float %b
+  %extract = extractelement <4 x float> %v, i64 0
+  %0 = tail call float @llvm.experimental.constrained.fma.f32(float %fneg, float %extract, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret float %0
+}
+
+define float @test_fmls_ss4S_3_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ss4S_3_strict
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = fneg float %tmp1
   %tmp3 = call float @llvm.experimental.constrained.fma.f32(float %tmp2, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp3
 }
 
-define float @test_fmls_ss4S_swap_strict(float %a, float %b, <4 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmls_ss4S_swap_strict
+define float @test_fmls_ss4S_3_swap_strict(float %a, float %b, <4 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ss4S_3_swap_strict
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
   %tmp1 = extractelement <4 x float> %v, i32 3
   %tmp2 = fneg float %tmp1
   %tmp3 = call float @llvm.experimental.constrained.fma.f32(float %tmp1, float %tmp2, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp3
 }
 
-define float @test_fmls_ss2S_strict(float %a, float %b, <2 x float> %v) #0 {
-  ; CHECK-LABEL: test_fmls_ss2S_strict
+define float @test_fmls_ss2S_0_strict(float %a, float %b, <2 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ss2S_0_strict
+  ; CHECK: fmsub s0, s2, s1, s0
+entry:
+  %fneg = fneg float %b
+  %extract = extractelement <2 x float> %v, i64 0
+  %0 = tail call float @llvm.experimental.constrained.fma.f32(float %fneg, float %extract, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret float %0
+}
+
+define float @test_fmls_ss2S_1_strict(float %a, float %b, <2 x float> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ss2S_1_strict
   ; CHECK: fmls {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
   %tmp1 = extractelement <2 x float> %v, i32 1
   %tmp2 = fneg float %tmp1
   %tmp3 = call float @llvm.experimental.constrained.fma.f32(float %tmp2, float %tmp1, float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %tmp3
 }
 
-define double @test_fmls_ddD_strict(double %a, double %b, <1 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmls_ddD_strict
-  ; CHECK: {{fmls d[0-9]+, d[0-9]+, v[0-9]+.d\[0]|fmsub d[0-9]+, d[0-9]+, d[0-9]+, d[0-9]+}}
-  %tmp1 = extractelement <1 x double> %v, i32 0
-  %tmp2 = fneg double %tmp1
-  %tmp3 = call double @llvm.experimental.constrained.fma.f64(double %tmp2, double %tmp1, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
-  ret double %tmp3
+define double @test_fmls_ddD_0_strict(double %a, double %b, <1 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmls_ddD_0_strict
+  ; CHECK: fmsub d0, d2, d1, d0
+entry:
+  %fneg = fneg double %b
+  %extract = extractelement <1 x double> %v, i64 0
+  %0 = tail call double @llvm.experimental.constrained.fma.f64(double %fneg, double %extract, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret double %0
+}
+
+define double @test_fmls_dd2D_0_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmls_dd2D_0_strict
+  ; CHECK: fmsub d0, d2, d1, d0
+entry:
+  %fneg = fneg double %b
+  %extract = extractelement <2 x double> %v, i64 0
+  %0 = tail call double @llvm.experimental.constrained.fma.f64(double %fneg, double %extract, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ret double %0
 }
 
-define double @test_fmls_dd2D_strict(double %a, double %b, <2 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmls_dd2D_strict
+define double @test_fmls_dd2D_1_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmls_dd2D_1_strict
   ; CHECK: fmls {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = fneg double %tmp1
   %tmp3 = call double @llvm.experimental.constrained.fma.f64(double %tmp2, double %tmp1, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret double %tmp3
 }
 
-define double @test_fmls_dd2D_swap_strict(double %a, double %b, <2 x double> %v) #0 {
-  ; CHECK-LABEL: test_fmls_dd2D_swap_strict
+define double @test_fmls_dd2D_1_swap_strict(double %a, double %b, <2 x double> %v) #0 {
+  ; CHECK-LABEL: test_fmls_dd2D_1_swap_strict
   ; CHECK: fmls {{d[0-9]+}}, {{d[0-9]+}}, {{v[0-9]+}}.d[1]
   %tmp1 = extractelement <2 x double> %v, i32 1
   %tmp2 = fneg double %tmp1
   %tmp3 = call double @llvm.experimental.constrained.fma.f64(double %tmp1, double %tmp2, double %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret double %tmp3
 }