[X86][TwoAddressInstructionPass] Teach tryInstructionCommute to continue checking for commutable FMA operands in more cases.

[craig.topper]

Summary

Previously we would only check for another commutable operand if the first commute was an aggressive commute.

But if we have two kill operands and neither is tied to the def at the start, we should consider both operands as the one to use as the new def.

This improves the loop in the fma-commute-loop.ll test. This test is derived from a post from discourse here https://llvm.discourse.group/t/unnecessary-vmovapd-instructions-generated-can-you-hint-in-favor-of-vfmadd231pd/582

It does degrade some of the fastmath tests, but that's probably just due to the known problems with our decision making with physical register constraints from above and below on small code with multiple two address instructions.

[bondhugula]

It's really nice to see this patch. I think the instruction sequence pattern that this has an impact on (the one in fma-commute-loop.ll and on the discourse post) is arguably the most important one for HPC! I tested this patch on the code generated from MLIR in the context of this experimentation: https://github.com/bondhugula/llvm-project/blob/hop/mlir/docs/HighPerfCodeGen.md#tweaking-m_c-k_c-m_r-n_r-to-maximize-reuse
and all the fma 213pd's shown in the llc assembly output snippet therein are replaced by 231pd's. It didn't have any impact on the performance though since the 213pds in that sequence didn't lead to any additional inter register shuffling, but FWIW, it's much more intuitive to see the same consistent set of registers due to usage of 231s there.

[RKSimon]

There seems to be a few extra register moves with this change

llvm/test/CodeGen/X86/recip-fastmath.ll

819

Any chance that we can avoid this regression (and the other ones above)?

[craig.topper]

llvm/test/CodeGen/X86/recip-fastmath.ll

819

Not without a new heuristic. We would need to see that one of the kills we're choosing for vfmsub is tied to xmm0 and that the user of the vfmsub def, the vfnmadd132ps, has its def tied to xmm0 and that it's commutable into a form that would allow the output from the vfmsub to be tied to that def.

[RKSimon]

OK, please can you add the additional tests now?

[craig.topper]

Rebase after uploading test. Hopefully with context this time

[RKSimon]

LGTM, please can you raise a bug about fixing the extra move cases?

[chriselrod]

I'm still seeing this for a complex dot product on LLVM 11.0.1. Godbolt example

LLVM IR:

L61:                                              ; preds = %L61, %L30.preheader
  %value_phi20178 = phi <8 x double> [ %res.i118, %L61 ], [ zeroinitializer, %L30.preheader ]
  %value_phi19177 = phi <8 x double> [ %res.i119, %L61 ], [ zeroinitializer, %L30.preheader ]
  %value_phi16176 = phi <8 x double> [ %res.i124, %L61 ], [ zeroinitializer, %L30.preheader ]
  %value_phi15175 = phi <8 x double> [ %res.i125, %L61 ], [ zeroinitializer, %L30.preheader ]
  %value_phi9174 = phi i64 [ %ptr.2.i114, %L61 ], [ %11, %L30.preheader ]
  %value_phi173 = phi i64 [ %ptr.2.i117, %L61 ], [ %10, %L30.preheader ]
  %ptr.1.i148 = inttoptr i64 %value_phi173 to <16 x double>*
  %res.i149 = load <16 x double>, <16 x double>* %ptr.1.i148, align 8
  %res.i147 = shufflevector <16 x double> %res.i149, <16 x double> undef, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
  %res.i146 = shufflevector <16 x double> %res.i149, <16 x double> undef, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
  %ptr.0.i143 = inttoptr i64 %value_phi173 to double*
  %ptr.1.i144 = getelementptr inbounds double, double* %ptr.0.i143, i64 16
  %ptr.1.i141 = bitcast double* %ptr.1.i144 to <16 x double>*
  %res.i142 = load <16 x double>, <16 x double>* %ptr.1.i141, align 8
  %res.i140 = shufflevector <16 x double> %res.i142, <16 x double> undef, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
  %res.i139 = shufflevector <16 x double> %res.i142, <16 x double> undef, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
  %ptr.1.i137 = inttoptr i64 %value_phi9174 to <16 x double>*
  %res.i138 = load <16 x double>, <16 x double>* %ptr.1.i137, align 8
  %res.i136 = shufflevector <16 x double> %res.i138, <16 x double> undef, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
  %res.i135 = shufflevector <16 x double> %res.i138, <16 x double> undef, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
  %ptr.0.i132 = inttoptr i64 %value_phi9174 to double*
  %ptr.1.i133 = getelementptr inbounds double, double* %ptr.0.i132, i64 16
  %ptr.1.i130 = bitcast double* %ptr.1.i133 to <16 x double>*
  %res.i131 = load <16 x double>, <16 x double>* %ptr.1.i130, align 8
  %res.i129 = shufflevector <16 x double> %res.i131, <16 x double> undef, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
  %res.i128 = shufflevector <16 x double> %res.i131, <16 x double> undef, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
  %res.i127 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i146, <8 x double> %res.i135, <8 x double> %value_phi15175)
  %res.i126 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i139, <8 x double> %res.i128, <8 x double> %value_phi16176)
  %res.i125 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i147, <8 x double> %res.i136, <8 x double> %res.i127)
  %res.i124 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i140, <8 x double> %res.i129, <8 x double> %res.i126)
  %res.i123 = fneg nsz contract <8 x double> %res.i146
  %res.i122 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i123, <8 x double> %res.i136, <8 x double> %value_phi19177)
  %res.i121 = fneg nsz contract <8 x double> %res.i139
  %res.i120 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i121, <8 x double> %res.i129, <8 x double> %value_phi20178)
  %res.i119 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i147, <8 x double> %res.i135, <8 x double> %res.i122)
  %res.i118 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i140, <8 x double> %res.i128, <8 x double> %res.i120)
  %ptr.1.i116 = getelementptr inbounds double, double* %ptr.0.i143, i64 32
  %ptr.2.i117 = ptrtoint double* %ptr.1.i116 to i64
  %ptr.1.i113 = getelementptr inbounds double, double* %ptr.0.i132, i64 32
  %ptr.2.i114 = ptrtoint double* %ptr.1.i113 to i64
  %.not = icmp ugt double* %ptr.1.i116, %ptr.1.i157
  br i1 %.not, label %L88, label %L61

ASM:

.LBB0_5:                                # %L61
        vmovupd zmm8, zmmword ptr [rsi]
        vmovupd zmm6, zmmword ptr [rsi + 64]
        vmovupd zmm9, zmmword ptr [rsi + 128]
        vmovupd zmm7, zmmword ptr [rsi + 192]
        vmovapd zmm10, zmm8
        vpermt2pd       zmm10, zmm2, zmm6
        vpermt2pd       zmm8, zmm3, zmm6
        vmovapd zmm11, zmm9
        vpermt2pd       zmm11, zmm2, zmm7
        vmovupd zmm12, zmmword ptr [rdx]
        vmovupd zmm13, zmmword ptr [rdx + 64]
        vmovupd zmm14, zmmword ptr [rdx + 128]
        vpermt2pd       zmm9, zmm3, zmm7
        vmovupd zmm15, zmmword ptr [rdx + 192]
        vmovapd zmm6, zmm12
        vpermt2pd       zmm6, zmm2, zmm13
        vpermt2pd       zmm12, zmm3, zmm13
        vmovapd zmm7, zmm14
        vpermt2pd       zmm7, zmm2, zmm15
        vpermt2pd       zmm14, zmm3, zmm15
        vfmadd231pd     zmm0, zmm8, zmm12       # zmm0 = (zmm8 * zmm12) + zmm0
        vfmadd231pd     zmm1, zmm9, zmm14       # zmm1 = (zmm9 * zmm14) + zmm1
        vfmadd231pd     zmm0, zmm10, zmm6       # zmm0 = (zmm10 * zmm6) + zmm0
        vfmadd231pd     zmm1, zmm11, zmm7       # zmm1 = (zmm11 * zmm7) + zmm1
        vfmsub213pd     zmm6, zmm8, zmm5        # zmm6 = (zmm8 * zmm6) - zmm5
        vfmsub213pd     zmm7, zmm9, zmm4        # zmm7 = (zmm9 * zmm7) - zmm4
        vfmsub231pd     zmm6, zmm10, zmm12      # zmm6 = (zmm10 * zmm12) - zmm6
        vfmsub231pd     zmm7, zmm11, zmm14      # zmm7 = (zmm11 * zmm14) - zmm7
        add     rsi, 256
        add     rdx, 256
        vmovapd zmm4, zmm7
        vmovapd zmm5, zmm6
        cmp     rsi, r11
        jbe     .LBB0_5

Note the

        vfmsub213pd     zmm6, zmm8, zmm5        # zmm6 = (zmm8 * zmm6) - zmm5
        vfmsub213pd     zmm7, zmm9, zmm4        # zmm7 = (zmm9 * zmm7) - zmm4
        vfmsub231pd     zmm6, zmm10, zmm12      # zmm6 = (zmm10 * zmm12) - zmm6
        vfmsub231pd     zmm7, zmm11, zmm14      # zmm7 = (zmm11 * zmm14) - zmm7
# ...
        vmovapd zmm4, zmm7
        vmovapd zmm5, zmm6

EDIT:
This seems to be a separate (unrelated?) problem.

First, an aside pointing to what looks like a missed optimization, where there's an extra separaet loop preheader:

  %3 = load i64, i64* %2, align 8
  %res.i159 = shl nuw i64 %3, 1
  %indname.i = add i64 %res.i159, -32
  %9 = icmp ult i64 %3, 16
  br i1 %9, label %L93, label %L30.preheader

L30.preheader:                                    ; preds = %top
  %.not172 = icmp slt i64 %indname.i, 0
  br i1 %.not172, label %L88, label %L61

I'm not sure if the optimization of eliminating it/the extra branch is valid or not.
It'd be invalid if %3 were less than 0. but there is an assume on sgt %3, 0, but there could be other corner cases I'm missing. Running through the code witha few possible example valeues of %3 (and a few assumed impossible ones):

  %3 = [ -1, 0, 15, 16, 17 ]
  %4 = icmp sgt i64 %3, 0
  call void @llvm.assume(i1 %4) # the `-1` and `0` values aren't possible
  %res.i159 = shl nuw i64 %3, 1 = [ -2, 0, 30, 32, 34 ]
  %indname.i = add i64 %res.i159, -32 = [ -34, -32, -2, 0, 2 ]
  %9 = icmp ult i64 %3, 16 [ 0xffff...<16=false, 0<16=true, 15<16=true, 16<16=false, 17<16=false > 
  br i1 %9, label %L93, label %L30.preheader 

L30.preheader:                                    ; preds = %top
  %.not172 = icmp slt i64 %indname.i, 0 # <-34<0=true,couldn't reach, couldn't reach, 0<0=false, 2<0 false >
  br i1 %.not172, label %L88, label %L61

Anyway, the point of this long aside on the maybe missed optimization is that when I did ellimate the redundancy, I got t[[ https://godbolt.org/z/448vco | his Godbolt ]].

The two vmovapds associated with the vfmsubs are now gone (the remaining ones are all necessary because of the vperms).

So my theory (disclaimer: I am totally ignorant of LLVM's internals) is that the cause of those vmovapds is difficulty allocating registers consistently across all those blocks/in the presence of phi nodes.
Code with the extra vmovapds:

        vfmsub213pd     zmm6, zmm8, zmm5        # zmm6 = (zmm8 * zmm6) - zmm5
        vfmsub213pd     zmm7, zmm9, zmm4        # zmm7 = (zmm9 * zmm7) - zmm4
        vfmsub231pd     zmm6, zmm10, zmm12      # zmm6 = (zmm10 * zmm12) - zmm6
        vfmsub231pd     zmm7, zmm11, zmm14      # zmm7 = (zmm11 * zmm14) - zmm7
        add     rsi, 256
        add     rdx, 256
        vmovapd zmm4, zmm7
        vmovapd zmm5, zmm6
        cmp     rsi, r11
        jbe     .LBB0_5
        jmp     .LBB0_6
.LBB0_3:
        vxorpd  xmm0, xmm0, xmm0
        mov     rsi, r8
        vxorpd  xmm1, xmm1, xmm1
        vxorpd  xmm6, xmm6, xmm6
        vxorpd  xmm7, xmm7, xmm7
.LBB0_6:                                # %L88
        vaddpd  zmm2, zmm6, zmm7
        vaddpd  zmm0, zmm0, zmm1 # calculated earlier in the loop

corresponding IR:

  %res.i123 = fneg nsz contract <8 x double> %res.i146
  %res.i122 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i123, <8 x double> %res.i136, <8 x double> %value_phi19177)
  %res.i121 = fneg nsz contract <8 x double> %res.i139
  %res.i120 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i121, <8 x double> %res.i129, <8 x double> %value_phi20178)
  %res.i119 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i147, <8 x double> %res.i135, <8 x double> %res.i122)
  %res.i118 = call nsz contract <8 x double> @llvm.fma.v8f64(<8 x double> %res.i140, <8 x double> %res.i128, <8 x double> %res.i120)
  %ptr.1.i116 = getelementptr inbounds double, double* %ptr.0.i143, i64 32
  %ptr.2.i117 = ptrtoint double* %ptr.1.i116 to i64
  %ptr.1.i113 = getelementptr inbounds double, double* %ptr.0.i132, i64 32
  %ptr.2.i114 = ptrtoint double* %ptr.1.i113 to i64
  %.not = icmp ugt double* %ptr.1.i116, %ptr.1.i157
  br i1 %.not, label %L88, label %L61

L88:                                              ; preds = %L61, %L30.preheader
  %value_phi.lcssa = phi i64 [ %10, %L30.preheader ], [ %ptr.2.i117, %L61 ]
  %value_phi9.lcssa = phi i64 [ %11, %L30.preheader ], [ %ptr.2.i114, %L61 ]
  %value_phi15.lcssa = phi <8 x double> [ zeroinitializer, %L30.preheader ], [ %res.i125, %L61 ]
  %value_phi16.lcssa = phi <8 x double> [ zeroinitializer, %L30.preheader ], [ %res.i124, %L61 ]
  %value_phi19.lcssa = phi <8 x double> [ zeroinitializer, %L30.preheader ], [ %res.i119, %L61 ]
  %value_phi20.lcssa = phi <8 x double> [ zeroinitializer, %L30.preheader ], [ %res.i118, %L61 ]
  %res.i111 = fadd nsz contract <8 x double> %value_phi15.lcssa, %value_phi16.lcssa
  %res.i110 = fadd nsz contract <8 x double> %value_phi19.lcssa, %value_phi20.lcssa
  br label %L93

The start of the loop iteration takes zmm4 and zmm5 as inputs, and then updates them into zmm7 and zmm6.
Thus, the next loop iteration requires zmm4 and zmm5 to be updated -- hence, either vmovapds are necessary, or changing which register gets assigned in the second set of vmovapds. However, .LBB0_6: expects registers 6 and 7 as inputs. This would have to be changed as well.

Obviously it's possible, but it's bigger/more involved and thus understandably harder to do than the case when the extra blocks get elliminated, and the ASM instead looks like:

vfmsub231pd     zmm2, zmm6, zmm14       # zmm2 = (zmm6 * zmm14) - zmm2
vfmsub231pd     zmm4, zmm8, zmm12       # zmm4 = (zmm8 * zmm12) - zmm4
vfmsub231pd     zmm2, zmm10, zmm11      # zmm2 = (zmm10 * zmm11) - zmm2
vfmsub231pd     zmm4, zmm7, zmm13       # zmm4 = (zmm7 * zmm13) - zmm4
add     rsi, 256
add     rdx, 256
cmp     rsi, r11
jbe     .LBB0_3
vaddpd  zmm1, zmm0, zmm1  # calculated earlier in the loop
vaddpd  zmm0, zmm2, zmm4

corresponding IR:

  %res.i138 = fneg reassoc nsz arcp contract afn <8 x double> %res.i161
  %res.i137 = call reassoc nsz arcp contract afn <8 x double> @llvm.fma.v8f64(<8 x double> %res.i138, <8 x double> %res.i151, <8 x double> %value_phi18)
  %res.i136 = fneg reassoc nsz arcp contract afn <8 x double> %res.i154
  %res.i135 = call reassoc nsz arcp contract afn <8 x double> @llvm.fma.v8f64(<8 x double> %res.i136, <8 x double> %res.i144, <8 x double> %value_phi19)
  %res.i134 = call reassoc nsz arcp contract afn <8 x double> @llvm.fma.v8f64(<8 x double> %res.i162, <8 x double> %res.i150, <8 x double> %res.i137)
  %res.i133 = call reassoc nsz arcp contract afn <8 x double> @llvm.fma.v8f64(<8 x double> %res.i155, <8 x double> %res.i143, <8 x double> %res.i135)
  %ptr.1.i131 = getelementptr inbounds double, double* %ptr.0.i158, i64 32
  %ptr.2.i132 = ptrtoint double* %ptr.1.i131 to i64
  %ptr.1.i128 = getelementptr inbounds double, double* %ptr.0.i147, i64 32
  %ptr.2.i129 = ptrtoint double* %ptr.1.i128 to i64
  %.not = icmp ugt double* %ptr.1.i131, %ptr.1.i172
  br i1 %.not, label %L110, label %L51

L110:                                             ; preds = %L51
  %res.i126 = fadd nsz contract <8 x double> %res.i140, %res.i139
  %res.i125 = fadd nsz contract <8 x double> %res.i134, %res.i133
  br label %L129

No phi nodes here in the second block, and no unnecessary vmovapds in the ASM.

llvm/lib/CodeGen/TwoAddressInstructionPass.cpp

if (!DoCommute &&
       AggressiveCommute = true;
     }
 
     // If it's profitable to commute, try to do so.
     if (DoCommute && commuteInstruction(MI, DstOpIdx, BaseOpIdx, OtherOpIdx,
                                         Dist)) {
       MadeChange = true;
       ++NumCommuted;
-      if (AggressiveCommute) {
+      if (AggressiveCommute)
         ++NumAggrCommuted;
-        // There might be more than two commutable operands, update BaseOp and
-        // continue scanning.
-        // FIXME: This assumes that the new instruction's operands are in the
-        // same positions and were simply swapped.
-        BaseOpReg = OtherOpReg;
-        BaseOpKilled = OtherOpKilled;
-        // Resamples OpsNum in case the number of operands was reduced. This
-        // happens with X86.
-        OpsNum = MI->getDesc().getNumOperands();
-        continue;
-      }
-      // If this was a commute based on kill, we won't do better continuing.
-      return MadeChange;
+
     }
   }
   return MadeChange;
 }
 
 /// For the case where an instruction has a single pair of tied register
 /// operands, attempt some transformations that may either eliminate the tied
 /// operands or improve the opportunities for coalescing away the register copy.

llvm/test/CodeGen/X86/fma-commute-loop.ll

 ; CHECK-NEXT:    leaq (%rbx,%r15,8), %r15
 ; CHECK-NEXT:    vxorpd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    xorl %ebx, %ebx
 ; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %r13
 ; CHECK-NEXT:    addq %r12, %r13
 ; CHECK-NEXT:    addq {{[0-9]+}}(%rsp), %r12
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    vxorpd %xmm2, %xmm2, %xmm2
+; CHECK-NEXT:    vxorpd %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    vxorpd %xmm4, %xmm4, %xmm4
 ; CHECK-NEXT:    vxorpd %xmm5, %xmm5, %xmm5
-; CHECK-NEXT:    vxorpd %xmm3, %xmm3, %xmm3
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  LBB0_1: ## %bb15
 ; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmovapd %zmm5, %zmm6
-; CHECK-NEXT:    vmovapd %zmm4, %zmm7
-; CHECK-NEXT:    vmovupd (%rax,%r11,8), %zmm4
-; CHECK-NEXT:    vmovupd (%rax,%r13,8), %zmm5
+; CHECK-NEXT:    vmovupd (%rax,%r11,8), %zmm6
+; CHECK-NEXT:    vmovupd (%rax,%r13,8), %zmm7
 ; CHECK-NEXT:    vmovupd (%rax,%r12,8), %zmm8
 ; CHECK-NEXT:    vbroadcastsd (%r15,%rbx,8), %zmm9
-; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm0 = (zmm4 * zmm9) + zmm0
-; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm1 = (zmm5 * zmm9) + zmm1
+; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm0 = (zmm6 * zmm9) + zmm0
+; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm1 = (zmm7 * zmm9) + zmm1
 ; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm2 = (zmm8 * zmm9) + zmm2
 ; CHECK-NEXT:    vbroadcastsd (%r14,%rbx,8), %zmm9
-; CHECK-NEXT:    vfmadd213pd {{.*#+}} zmm4 = (zmm9 * zmm4) + zmm7
-; CHECK-NEXT:    vfmadd213pd {{.*#+}} zmm5 = (zmm9 * zmm5) + zmm6
-; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm3 = (zmm8 * zmm9) + zmm3
+; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm3 = (zmm9 * zmm6) + zmm3
+; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm4 = (zmm9 * zmm7) + zmm4
+; CHECK-NEXT:    vfmadd231pd {{.*#+}} zmm5 = (zmm8 * zmm9) + zmm5
 ; CHECK-NEXT:    incq %rbx
 ; CHECK-NEXT:    cmpq %rbx, %r10
 ; CHECK-NEXT:    jne LBB0_1
 ; CHECK-NEXT:  ## %bb.2: ## %bb51
 ; CHECK-NEXT:    vmovapd %zmm0, (%rdi)
 ; CHECK-NEXT:    vmovapd %zmm1, (%rsi)
 ; CHECK-NEXT:    vmovapd %zmm2, (%rdx)
-; CHECK-NEXT:    vmovapd %zmm4, (%rcx)
-; CHECK-NEXT:    vmovapd %zmm5, (%r8)
-; CHECK-NEXT:    vmovapd %zmm3, (%r9)
+; CHECK-NEXT:    vmovapd %zmm3, (%rcx)
+; CHECK-NEXT:    vmovapd %zmm4, (%r8)
+; CHECK-NEXT:    vmovapd %zmm5, (%r9)
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 bb:

llvm/test/CodeGen/X86/recip-fastmath.ll

 ; AVX-RECIP-NEXT:    vmulss %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: f32_one_step_variables:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
 ; FMA-RECIP-NEXT:    vmulss %xmm2, %xmm0, %xmm3
-; FMA-RECIP-NEXT:    vfmsub213ss {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; FMA-RECIP-NEXT:    vmovaps %xmm2, %xmm0
+; FMA-RECIP-NEXT:    vfmsub231ss {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: f32_one_step_variables:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
 ; BDVER2-NEXT:    vmulss %xmm2, %xmm0, %xmm3
 ; BDVER2-NEXT:    vfmsubss {{.*#+}} xmm0 = (xmm1 * xmm3) - xmm0
 ; BDVER2-NEXT:    vfnmaddss {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3

 ; SANDY-NEXT:    vmulss %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: f32_one_step_variables:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
 ; HASWELL-NEXT:    vmulss %xmm2, %xmm0, %xmm3
-; HASWELL-NEXT:    vfmsub213ss {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; HASWELL-NEXT:    vmovaps %xmm2, %xmm0
+; HASWELL-NEXT:    vfmsub231ss {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: f32_one_step_variables:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm0, %xmm3
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm3, %xmm1, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vsubss %xmm1, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: f32_one_step_variables:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
 ; AVX512-NEXT:    vmulss %xmm2, %xmm0, %xmm3
-; AVX512-NEXT:    vfmsub213ss {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; AVX512-NEXT:    vmovaps %xmm2, %xmm0
+; AVX512-NEXT:    vfmsub231ss {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast float %x, %y
   ret float %div
 }
 
 define float @f32_two_step(float %x) #2 {
 ; SSE-LABEL: f32_two_step:
 ; SSE:       # %bb.0:

 ; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; SANDY-NEXT:    vaddps %xmm0, %xmm1, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_one_step:
 ; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %xmm0, %xmm1
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - xmm2
-; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
+; HASWELL-NEXT:    vrcpps %xmm0, %xmm2
+; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; HASWELL-NEXT:    vfmsub231ps {{.*#+}} xmm1 = (xmm2 * xmm0) - xmm1
+; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} xmm1 = -(xmm1 * xmm2) + xmm2
+; HASWELL-NEXT:    vmovaps %xmm1, %xmm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_one_step:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm0, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; KNL-LABEL: v4f32_one_step:
 ; KNL:       # %bb.0:
-; KNL-NEXT:    vrcpps %xmm0, %xmm1
-; KNL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - xmm2
-; KNL-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
+; KNL-NEXT:    vrcpps %xmm0, %xmm2
+; KNL-NEXT:    vbroadcastss {{.*#+}} xmm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; KNL-NEXT:    vfmsub231ps {{.*#+}} xmm1 = (xmm2 * xmm0) - xmm1
+; KNL-NEXT:    vfnmadd132ps {{.*#+}} xmm1 = -(xmm1 * xmm2) + xmm2
+; KNL-NEXT:    vmovaps %xmm1, %xmm0
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: v4f32_one_step:
 ; SKX:       # %bb.0:
 ; SKX-NEXT:    vrcpps %xmm0, %xmm1
 ; SKX-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - mem
 ; SKX-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
 ; SKX-NEXT:    retq

 ; AVX-RECIP-NEXT:    vmulps %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v4f32_one_step_variables:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %xmm1, %xmm2
 ; FMA-RECIP-NEXT:    vmulps %xmm2, %xmm0, %xmm3
-; FMA-RECIP-NEXT:    vfmsub213ps {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; FMA-RECIP-NEXT:    vmovaps %xmm2, %xmm0
+; FMA-RECIP-NEXT:    vfmsub231ps {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v4f32_one_step_variables:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %xmm1, %xmm2
 ; BDVER2-NEXT:    vmulps %xmm2, %xmm0, %xmm3
 ; BDVER2-NEXT:    vfmsubps {{.*#+}} xmm0 = (xmm1 * xmm3) - xmm0
 ; BDVER2-NEXT:    vfnmaddps {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3

 ; SANDY-NEXT:    vmulps %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_one_step_variables:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %xmm1, %xmm2
 ; HASWELL-NEXT:    vmulps %xmm2, %xmm0, %xmm3
-; HASWELL-NEXT:    vfmsub213ps {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; HASWELL-NEXT:    vmovaps %xmm2, %xmm0
+; HASWELL-NEXT:    vfmsub231ps {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_one_step_variables:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm1, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm0, %xmm3
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm3, %xmm1, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vsubps %xmm1, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v4f32_one_step_variables:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpps %xmm1, %xmm2
 ; AVX512-NEXT:    vmulps %xmm2, %xmm0, %xmm3
-; AVX512-NEXT:    vfmsub213ps {{.*#+}} xmm1 = (xmm3 * xmm1) - xmm0
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm2 = -(xmm1 * xmm2) + xmm3
-; AVX512-NEXT:    vmovaps %xmm2, %xmm0
+; AVX512-NEXT:    vfmsub231ps {{.*#+}} xmm0 = (xmm3 * xmm1) - xmm0
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm2 * xmm0) + xmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast <4 x float> %x, %y
   ret <4 x float> %div
 }
 
 define <4 x float> @v4f32_two_step(<4 x float> %x) #2 {
 ; SSE-LABEL: v4f32_two_step:
 ; SSE:       # %bb.0:

 ; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step:
 ; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %ymm0, %ymm1
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm2
-; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
+; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
+; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; HASWELL-NEXT:    vfmsub231ps {{.*#+}} ymm1 = (ymm2 * ymm0) - ymm1
+; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} ymm1 = -(ymm1 * ymm2) + ymm2
+; HASWELL-NEXT:    vmovaps %ymm1, %ymm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v8f32_one_step:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm0, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; KNL-LABEL: v8f32_one_step:
 ; KNL:       # %bb.0:
-; KNL-NEXT:    vrcpps %ymm0, %ymm1
-; KNL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm2
-; KNL-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
+; KNL-NEXT:    vrcpps %ymm0, %ymm2
+; KNL-NEXT:    vbroadcastss {{.*#+}} ymm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; KNL-NEXT:    vfmsub231ps {{.*#+}} ymm1 = (ymm2 * ymm0) - ymm1
+; KNL-NEXT:    vfnmadd132ps {{.*#+}} ymm1 = -(ymm1 * ymm2) + ymm2
+; KNL-NEXT:    vmovaps %ymm1, %ymm0
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: v8f32_one_step:
 ; SKX:       # %bb.0:
 ; SKX-NEXT:    vrcpps %ymm0, %ymm1
 ; SKX-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - mem
 ; SKX-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
 ; SKX-NEXT:    retq

llvm/test/CodeGen/X86/recip-fastmath2.ll

 ; SANDY-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm1
 ; SANDY-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_one_step_2_divs:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %xmm0, %xmm1
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - xmm2
-; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm1
-; HASWELL-NEXT:    vmulps %xmm0, %xmm1, %xmm0
+; HASWELL-NEXT:    vfmsub231ps {{.*#+}} xmm2 = (xmm1 * xmm0) - xmm2
+; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} xmm2 = -(xmm2 * xmm1) + xmm1
+; HASWELL-NEXT:    vmulps {{.*}}(%rip), %xmm2, %xmm0
+; HASWELL-NEXT:    vmulps %xmm2, %xmm0, %xmm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_one_step_2_divs:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm0, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; KNL-LABEL: v4f32_one_step_2_divs:
 ; KNL:       # %bb.0:
 ; KNL-NEXT:    vrcpps %xmm0, %xmm1
 ; KNL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - xmm2
-; KNL-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
-; KNL-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm1
-; KNL-NEXT:    vmulps %xmm0, %xmm1, %xmm0
+; KNL-NEXT:    vfmsub231ps {{.*#+}} xmm2 = (xmm1 * xmm0) - xmm2
+; KNL-NEXT:    vfnmadd132ps {{.*#+}} xmm2 = -(xmm2 * xmm1) + xmm1
+; KNL-NEXT:    vmulps {{.*}}(%rip), %xmm2, %xmm0
+; KNL-NEXT:    vmulps %xmm2, %xmm0, %xmm0
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: v4f32_one_step_2_divs:
 ; SKX:       # %bb.0:
 ; SKX-NEXT:    vrcpps %xmm0, %xmm1
 ; SKX-NEXT:    vfmsub213ps {{.*#+}} xmm0 = (xmm1 * xmm0) - mem
 ; SKX-NEXT:    vfnmadd132ps {{.*#+}} xmm0 = -(xmm0 * xmm1) + xmm1
 ; SKX-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm1

 ; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step_2_divs:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm2
-; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm1
-; HASWELL-NEXT:    vmulps %ymm0, %ymm1, %ymm0
+; HASWELL-NEXT:    vfmsub231ps {{.*#+}} ymm2 = (ymm1 * ymm0) - ymm2
+; HASWELL-NEXT:    vfnmadd132ps {{.*#+}} ymm2 = -(ymm2 * ymm1) + ymm1
+; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm2, %ymm0
+; HASWELL-NEXT:    vmulps %ymm2, %ymm0, %ymm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v8f32_one_step_2_divs:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm0, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; KNL-LABEL: v8f32_one_step_2_divs:
 ; KNL:       # %bb.0:
 ; KNL-NEXT:    vrcpps %ymm0, %ymm1
 ; KNL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm2
-; KNL-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
-; KNL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm1
-; KNL-NEXT:    vmulps %ymm0, %ymm1, %ymm0
+; KNL-NEXT:    vfmsub231ps {{.*#+}} ymm2 = (ymm1 * ymm0) - ymm2
+; KNL-NEXT:    vfnmadd132ps {{.*#+}} ymm2 = -(ymm2 * ymm1) + ymm1
+; KNL-NEXT:    vmulps {{.*}}(%rip), %ymm2, %ymm0
+; KNL-NEXT:    vmulps %ymm2, %ymm0, %ymm0
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: v8f32_one_step_2_divs:
 ; SKX:       # %bb.0:
 ; SKX-NEXT:    vrcpps %ymm0, %ymm1
 ; SKX-NEXT:    vfmsub213ps {{.*#+}} ymm0 = (ymm1 * ymm0) - mem
 ; SKX-NEXT:    vfnmadd132ps {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
 ; SKX-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm1

llvm/test/CodeGen/X86/sqrt-fastmath.ll

 ; AVX1-NEXT:    vmulps %xmm0, %xmm1, %xmm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX512-LABEL: v4f32_estimate:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrsqrtps %xmm0, %xmm1
 ; AVX512-NEXT:    vmulps %xmm1, %xmm0, %xmm0
 ; AVX512-NEXT:    vbroadcastss {{.*#+}} xmm2 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]
-; AVX512-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm2
-; AVX512-NEXT:    vbroadcastss {{.*#+}} xmm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; AVX512-NEXT:    vmulps %xmm2, %xmm1, %xmm1
+; AVX512-NEXT:    vfmadd231ps {{.*#+}} xmm2 = (xmm1 * xmm0) + xmm2
+; AVX512-NEXT:    vbroadcastss {{.*#+}} xmm0 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; AVX512-NEXT:    vmulps %xmm0, %xmm1, %xmm0
+; AVX512-NEXT:    vmulps %xmm2, %xmm0, %xmm0
 ; AVX512-NEXT:    retq
   %sqrt = tail call <4 x float> @llvm.sqrt.v4f32(<4 x float> %x)
   %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %sqrt
   ret <4 x float> %div
 }
 
 define <8 x float> @v8f32_no_estimate(<8 x float> %x) #0 {
 ; SSE-LABEL: v8f32_no_estimate:

 ; AVX1-NEXT:    vmulps %ymm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX512-LABEL: v8f32_estimate:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrsqrtps %ymm0, %ymm1
 ; AVX512-NEXT:    vmulps %ymm1, %ymm0, %ymm0
 ; AVX512-NEXT:    vbroadcastss {{.*#+}} ymm2 = [-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0,-3.0E+0]
-; AVX512-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm1 * ymm0) + ymm2
-; AVX512-NEXT:    vbroadcastss {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; AVX512-NEXT:    vmulps %ymm2, %ymm1, %ymm1
+; AVX512-NEXT:    vfmadd231ps {{.*#+}} ymm2 = (ymm1 * ymm0) + ymm2
+; AVX512-NEXT:    vbroadcastss {{.*#+}} ymm0 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; AVX512-NEXT:    vmulps %ymm0, %ymm1, %ymm0
+; AVX512-NEXT:    vmulps %ymm2, %ymm0, %ymm0
 ; AVX512-NEXT:    retq
   %sqrt = tail call <8 x float> @llvm.sqrt.v8f32(<8 x float> %x)
   %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %sqrt
   ret <8 x float> %div
 }
 
 define <16 x float> @v16f32_no_estimate(<16 x float> %x) #0 {
 ; SSE-LABEL: v16f32_no_estimate: