[X86] Add HasAVX512 patterns for SS SSE intrinsics.

[ab]

Summary

These would currently fail to select.

[ab]

• Add S[SD]Zrr->rm folding table entries
• Cleanup patterns as rm variants aren't needed

[delena]

lib/Target/X86/X86InstrAVX512.td

3365

All AVX-512 intrinsics, even FP scalar, come with masks and with rounding mode.
I don't understand where these intrinsics _sse_avx512_ come from.
Due to very complex form of these inrinsics, we add them to X86IntrinsicsInfo.h

[ab]

lib/Target/X86/X86InstrAVX512.td

3365

These patterns are just for the SSE1/SSE2 intrinsics (_mm_add_ss, int_x86_sse_add_ss, ...; see tests), which will fail to select when used on AVX512, because the X86InstrSSE.td SI defs are all predicated on "UseAVX" rather than "HaveAVX".

I went with patterns because for _ss, IntrinsicsInfo would turn something like _mm_add_ss to a v4f32 fadd and later addps. We could add an IntrinsicType for _ss _2OP, or maybe even, an additional field in the intrinsic data tables saying that even though an intrinsic has vector type, it's just a scalar op?

[ab]

Ping!

[delena]

All problems of sse intrinsics should be resolved in X86InstrSSE.td. Please change UseAVX to HasAVX for these intrinsics. I don't think that we should add all SSE intrinsics to AVX512 now.

[qcolombet]

I don't think that we should add all SSE intrinsics to AVX512 now.

I kind of disagree.
When AVX512 is available we have additional registers we can use for those. I believe we should do the same thing as we do for thumb, i.e., choose the most permissive instruction encoding then shorten the encoding if possible (like the Thumb2SizeReduction pass).

For the long term we may want to find a better solution for such case (same instruction available with different encoding based on extension) to avoid the explosion in size of the description.

What do you think?

Cheers,
-Quentin

[ab]

In D9362#168427, @qcolombet wrote:

I don't think that we should add all SSE intrinsics to AVX512 now.

I kind of disagree.
When AVX512 is available we have additional registers we can use for those. I believe we should do the same thing as we do for thumb, i.e., choose the most permissive instruction encoding then shorten the encoding if possible (like the Thumb2SizeReduction pass).

I agree, this seems like the way to go; see http://llvm.org/bugs/show_bug.cgi?id=23376.

For the long term we may want to find a better solution for such case (same instruction available with different encoding based on extension) to avoid the explosion in size of the description.

I also agree. In fact I think there are two orthogonal issues here:

• like I proposed earlier, we should use IntrinsicsInfo for scalar intrinsics. I filed https://llvm.org/bugs/show_bug.cgi?id=23449
• in lots of cases, the main difference between SSE/AVX/AVX512 instructions is the encoding. Due to the way the backend evolved, they're mostly apart, with lots of duplications (now it's even worse because AVX512 is in a completely different file). There's room for refactoring, I think, but that's a big change.

I can repurpose this review to implement the first one, if people agree? Also, if the second makes sense, I'll file a PR as well.

-Ahmed

What do you think?

Cheers,
-Quentin

[delena]

Hi,

We encountered with another problem - some sse intrinsics don't work on AVX-512 mode at all, due to UseAVX prefix.
I'm going to fix this, just to let these intrinsics be mapped to AVX instructions.

But I agree, as a long term solution, all these intrinsics should be mapped to AVX-512 instructions. And it is not only scalar FP. SKX covers all AVX and AVX2 instructions with EVEX version.

I'm against writing all this manually, it will be the last option. I'll think how all this mapping may be auto-generated.

• Elena

[ab]

Alright then, for now let's say this is superseded by r237903. Closing.

lib/Target/X86/X86InstrAVX512.td

 }
 defm VADD : avx512_binop_s_round<0x58, "vadd", fadd, X86faddRnd, SSE_ALU_ITINS_S, 1>;
 defm VMUL : avx512_binop_s_round<0x59, "vmul", fmul, X86fmulRnd, SSE_ALU_ITINS_S, 1>;
 defm VSUB : avx512_binop_s_round<0x5C, "vsub", fsub, X86fsubRnd, SSE_ALU_ITINS_S, 0>;
 defm VDIV : avx512_binop_s_round<0x5E, "vdiv", fdiv, X86fdivRnd, SSE_ALU_ITINS_S, 0>;
 defm VMIN : avx512_binop_s_sae  <0x5D, "vmin", X86fmin, X86fminRnd, SSE_ALU_ITINS_S, 1>;
 defm VMAX : avx512_binop_s_sae  <0x5F, "vmax", X86fmax, X86fmaxRnd, SSE_ALU_ITINS_S, 1>;
 
+// Also match the pre-AVX512 intrinsics.
+multiclass avx512_sse_scalar_intrin_patterns<string OpcodeStr, string IntStr> {
+  let Predicates = [HasAVX512] in {
+    def : Pat<(!cast<Intrinsic>("int_x86_sse_"#IntStr#"_ss") VR128X:$src1,
+                                                             VR128X:$src2),
+              (COPY_TO_REGCLASS
+                (!cast<Instruction>("V"#OpcodeStr#"SSZrr")
+                  (COPY_TO_REGCLASS VR128X:$src1, FR32),
+                  (COPY_TO_REGCLASS VR128X:$src2, FR32)),
+               VR128X)>;
+    def : Pat<(!cast<Intrinsic>("int_x86_sse2_"#IntStr#"_sd") VR128X:$src1,
+                                                              VR128X:$src2),
+              (COPY_TO_REGCLASS
+                (!cast<Instruction>("V"#OpcodeStr#"SDZrr")
+                  (COPY_TO_REGCLASS VR128X:$src1, FR64),
+                  (COPY_TO_REGCLASS VR128X:$src2, FR64)),
+               VR128X)>;
+  }
+}
+
+defm : avx512_sse_scalar_intrin_patterns<"ADD", "add">;
+defm : avx512_sse_scalar_intrin_patterns<"MUL", "mul">;
+defm : avx512_sse_scalar_intrin_patterns<"SUB", "sub">;
+defm : avx512_sse_scalar_intrin_patterns<"DIV", "div">;
+defm : avx512_sse_scalar_intrin_patterns<"MIN", "min">;
+defm : avx512_sse_scalar_intrin_patterns<"MAX", "max">;
+
 multiclass avx512_fp_packed<bits<8> opc, string OpcodeStr, SDNode OpNode,
                             X86VectorVTInfo _, bit IsCommutable> {
   defm rr: AVX512_maskable<opc, MRMSrcReg, _, (outs _.RC:$dst),
                   (ins _.RC:$src1, _.RC:$src2), OpcodeStr##_.Suffix,
                   "$src2, $src1", "$src1, $src2",
                   (_.VT (OpNode _.RC:$src1, _.RC:$src2))>, EVEX_4V;
   let mayLoad = 1 in {
     defm rm: AVX512_maskable<opc, MRMSrcMem, _, (outs _.RC:$dst),

lib/Target/X86/X86InstrInfo.cpp

static const X86MemoryFoldTableEntry MemoryFoldTable2[] = {
     { X86::MULX32rr,          X86::MULX32rm,            0 },
     { X86::MULX64rr,          X86::MULX64rm,            0 },
     { X86::PDEP32rr,          X86::PDEP32rm,            0 },
     { X86::PDEP64rr,          X86::PDEP64rm,            0 },
     { X86::PEXT32rr,          X86::PEXT32rm,            0 },
     { X86::PEXT64rr,          X86::PEXT64rm,            0 },
 
     // AVX-512 foldable instructions
-    { X86::VADDPSZrr,         X86::VADDPSZrm,           0 },
     { X86::VADDPDZrr,         X86::VADDPDZrm,           0 },
-    { X86::VSUBPSZrr,         X86::VSUBPSZrm,           0 },
-    { X86::VSUBPDZrr,         X86::VSUBPDZrm,           0 },
-    { X86::VMULPSZrr,         X86::VMULPSZrm,           0 },
-    { X86::VMULPDZrr,         X86::VMULPDZrm,           0 },
-    { X86::VDIVPSZrr,         X86::VDIVPSZrm,           0 },
+    { X86::VADDPSZrr,         X86::VADDPSZrm,           0 },
+    { X86::VADDSDZrr,         X86::VADDSDZrm,           0 },
+    { X86::VADDSSZrr,         X86::VADDSSZrm,           0 },
     { X86::VDIVPDZrr,         X86::VDIVPDZrm,           0 },
-    { X86::VMINPSZrr,         X86::VMINPSZrm,           0 },
-    { X86::VMINPDZrr,         X86::VMINPDZrm,           0 },
-    { X86::VMAXPSZrr,         X86::VMAXPSZrm,           0 },
+    { X86::VDIVPSZrr,         X86::VDIVPSZrm,           0 },
+    { X86::VDIVSDZrr,         X86::VDIVSDZrm,           0 },
+    { X86::VDIVSSZrr,         X86::VDIVSSZrm,           0 },
     { X86::VMAXPDZrr,         X86::VMAXPDZrm,           0 },
+    { X86::VMAXPSZrr,         X86::VMAXPSZrm,           0 },
+    { X86::VMAXSDZrr,         X86::VMAXSDZrm,           0 },
+    { X86::VMAXSSZrr,         X86::VMAXSSZrm,           0 },
+    { X86::VMINPDZrr,         X86::VMINPDZrm,           0 },
+    { X86::VMINPSZrr,         X86::VMINPSZrm,           0 },
+    { X86::VMINSDZrr,         X86::VMINSDZrm,           0 },
+    { X86::VMINSSZrr,         X86::VMINSSZrm,           0 },
+    { X86::VMULPDZrr,         X86::VMULPDZrm,           0 },
+    { X86::VMULPSZrr,         X86::VMULPSZrm,           0 },
+    { X86::VMULSDZrr,         X86::VMULSDZrm,           0 },
+    { X86::VMULSSZrr,         X86::VMULSSZrm,           0 },
     { X86::VPADDDZrr,         X86::VPADDDZrm,           0 },
     { X86::VPADDQZrr,         X86::VPADDQZrm,           0 },
     { X86::VPERMPDZri,        X86::VPERMPDZmi,          0 },
     { X86::VPERMPSZrr,        X86::VPERMPSZrm,          0 },
     { X86::VPMAXSDZrr,        X86::VPMAXSDZrm,          0 },
     { X86::VPMAXSQZrr,        X86::VPMAXSQZrm,          0 },
     { X86::VPMAXUDZrr,        X86::VPMAXUDZrm,          0 },
     { X86::VPMAXUQZrr,        X86::VPMAXUQZrm,          0 },
     { X86::VPMINSDZrr,        X86::VPMINSDZrm,          0 },
     { X86::VPMINSQZrr,        X86::VPMINSQZrm,          0 },
     { X86::VPMINUDZrr,        X86::VPMINUDZrm,          0 },
     { X86::VPMINUQZrr,        X86::VPMINUQZrm,          0 },
     { X86::VPMULDQZrr,        X86::VPMULDQZrm,          0 },
     { X86::VPSLLVDZrr,        X86::VPSLLVDZrm,          0 },
     { X86::VPSLLVQZrr,        X86::VPSLLVQZrm,          0 },
     { X86::VPSRAVDZrr,        X86::VPSRAVDZrm,          0 },
     { X86::VPSRLVDZrr,        X86::VPSRLVDZrm,          0 },
     { X86::VPSRLVQZrr,        X86::VPSRLVQZrm,          0 },
     { X86::VPSUBDZrr,         X86::VPSUBDZrm,           0 },
     { X86::VPSUBQZrr,         X86::VPSUBQZrm,           0 },
+    { X86::VSUBPDZrr,         X86::VSUBPDZrm,           0 },
+    { X86::VSUBPSZrr,         X86::VSUBPSZrm,           0 },
+    { X86::VSUBSDZrr,         X86::VSUBSDZrm,           0 },
+    { X86::VSUBSSZrr,         X86::VSUBSSZrm,           0 },
     { X86::VSHUFPDZrri,       X86::VSHUFPDZrmi,         0 },
     { X86::VSHUFPSZrri,       X86::VSHUFPSZrmi,         0 },
     { X86::VALIGNQrri,        X86::VALIGNQrmi,          0 },
     { X86::VALIGNDrri,        X86::VALIGNDrmi,          0 },
     { X86::VPMULUDQZrr,       X86::VPMULUDQZrm,         0 },
     { X86::VBROADCASTSSZrkz,  X86::VBROADCASTSSZmkz,    TB_NO_REVERSE },
     { X86::VBROADCASTSDZrkz,  X86::VBROADCASTSDZmkz,    TB_NO_REVERSE },
 

test/CodeGen/X86/avx512-sse-intrinsics.ll

+; RUN: llc < %s -asm-verbose=true -mtriple=x86_64-unknown-unknown -mattr=+avx512f --show-mc-encoding | FileCheck %s
+
+;====== VMAX ==================================================================;
+
+; CHECK-LABEL: test_maxss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5f,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_maxss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.max.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_maxss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5f,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_maxss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.max.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_maxsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5f,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_maxsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.max.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_maxsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5f,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_maxsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.max.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_maxps:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxps %xmm1, %xmm0, %xmm0 # encoding: [0xc5,0xf8,0x5f,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_maxps(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.max.ps(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_maxps_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxps (%rdi), %xmm0, %xmm0 # encoding: [0xc5,0xf8,0x5f,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_maxps_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.max.ps(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_maxpd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxpd %xmm1, %xmm0, %xmm0 # encoding: [0xc5,0xf9,0x5f,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_maxpd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.max.pd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_maxpd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmaxpd (%rdi), %xmm0, %xmm0 # encoding: [0xc5,0xf9,0x5f,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_maxpd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.max.pd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.max.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.max.sd(<2 x double>, <2 x double>)
+declare <4 x float>  @llvm.x86.sse.max.ps(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.max.pd(<2 x double>, <2 x double>)
+
+;====== VMIN ==================================================================;
+
+; CHECK-LABEL: test_minss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5d,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_minss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.min.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_minss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5d,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_minss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.min.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_minsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5d,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_minsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.min.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_minsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5d,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_minsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.min.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_minps:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminps %xmm1, %xmm0, %xmm0 # encoding: [0xc5,0xf8,0x5d,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_minps(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.min.ps(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_minps_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminps (%rdi), %xmm0, %xmm0 # encoding: [0xc5,0xf8,0x5d,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_minps_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.min.ps(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_minpd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminpd %xmm1, %xmm0, %xmm0 # encoding: [0xc5,0xf9,0x5d,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_minpd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.min.pd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_minpd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vminpd (%rdi), %xmm0, %xmm0 # encoding: [0xc5,0xf9,0x5d,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_minpd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.min.pd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.min.ps(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.min.pd(<2 x double>, <2 x double>)
+declare <4 x float>  @llvm.x86.sse.min.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.min.sd(<2 x double>, <2 x double>)
+
+;====== VADD ==================================================================;
+
+; CHECK-LABEL: test_addss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vaddss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x58,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_addss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.add.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_addss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vaddss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x58,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_addss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.add.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_addsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vaddsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x58,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_addsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.add.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_addsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vaddsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x58,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_addsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.add.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.add.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.add.sd(<2 x double>, <2 x double>)
+
+;====== VSUB ==================================================================;
+
+; CHECK-LABEL: test_subss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vsubss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5c,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_subss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.sub.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_subss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vsubss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5c,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_subss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.sub.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_subsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vsubsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5c,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_subsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.sub.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_subsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vsubsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5c,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_subsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.sub.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.sub.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.sub.sd(<2 x double>, <2 x double>)
+
+;====== VMUL ==================================================================;
+
+; CHECK-LABEL: test_mulss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmulss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x59,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_mulss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.mul.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_mulss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmulss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x59,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_mulss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.mul.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_mulsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmulsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x59,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_mulsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.mul.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_mulsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vmulsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x59,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_mulsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.mul.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.mul.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.mul.sd(<2 x double>, <2 x double>)
+
+;====== VDIV ==================================================================;
+
+; CHECK-LABEL: test_divss:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vdivss %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5e,0xc1]
+; CHECK-NEXT: retq
+define <4 x float> @test_divss(<4 x float> %a0, <4 x float> %a1) #0 {
+  %res = call <4 x float> @llvm.x86.sse.div.ss(<4 x float> %a0, <4 x float> %a1)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_divss_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vdivss (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0x7e,0x08,0x5e,0x07]
+; CHECK-NEXT: retq
+define <4 x float> @test_divss_rm(<4 x float> %a0, <4 x float>* %a1) #0 {
+  %a1l = load <4 x float>, <4 x float>* %a1
+  %res = call <4 x float> @llvm.x86.sse.div.ss(<4 x float> %a0, <4 x float> %a1l)
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_divsd:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vdivsd %xmm1, %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5e,0xc1]
+; CHECK-NEXT: retq
+define <2 x double> @test_divsd(<2 x double> %a0, <2 x double> %a1) #0 {
+  %res = call <2 x double> @llvm.x86.sse2.div.sd(<2 x double> %a0, <2 x double> %a1)
+  ret <2 x double> %res
+}
+
+; CHECK-LABEL: test_divsd_rm:
+; CHECK-NEXT: BB#0:
+; CHECK-NEXT: vdivsd (%rdi), %xmm0, %xmm0 # encoding: [0x62,0xf1,0xff,0x08,0x5e,0x07]
+; CHECK-NEXT: retq
+define <2 x double> @test_divsd_rm(<2 x double> %a0, <2 x double>* %a1) #0 {
+  %a1l = load <2 x double>, <2 x double>* %a1
+  %res = call <2 x double> @llvm.x86.sse2.div.sd(<2 x double> %a0, <2 x double> %a1l)
+  ret <2 x double> %res
+}
+
+declare <4 x float>  @llvm.x86.sse.div.ss(<4 x float>, <4 x float>)
+declare <2 x double> @llvm.x86.sse2.div.sd(<2 x double>, <2 x double>)
+
+attributes #0 = { nounwind }