[X86][BtVer2] Fix latency and throughput of conditional SIMD store instructions.

[andreadb]

Summary

On BtVer2 conditional SIMD stores are heavily microcoded.
The latency is directly proportional to the number of packed elements extracted from the input vector. Also, according to micro-benchmarks, most of the computation seems to be done in the integer unit.

Only a minority of the uOPs is executed by the FPU. The observed behaviour on the FPU looks similar to this:

• The input MASK value is moved to the Integer Unit -- [ a VMOVMSK-like uOP - executed on JFPU0].
• In parallel, each element of the input XMM/YMM is extracted and then sent to the IntegerUnit through JFPU1.

As expected, a (conditional) store is executed for every extracted element. Interestingly, a (speculative) load is executed for every extracted element too. It is as-if a "LOAD - BIT_EXTRACT- CMOV" sequence of uOPs is repeated by the integer unit for every contionally stored element.
VMASKMOVDQU is a special case: the number of speculative loads is always 2 (presumably, one load per quadword). That means, extra shifts and masking is performed on (one of) the loaded quadwords before each conditional store (that also explains the big number of non-FP uOPs retired).

[RKSimon]

lib/Target/X86/X86ScheduleBtVer2.td

826

Store

863

Would we be better off just splitting WriteFMaskedStore into WriteFMaskedStore32 + WriteFMaskedStore64?

[andreadb]

lib/Target/X86/X86ScheduleBtVer2.td

826

Thanks. I will fix it.

863

I have been thinking about it before sending this patch. The possibility of adding new classes for conditional writes was not so bad to start.
However, btver2 is currently the only model that requires to special case the PS/PD variants. So, eventually I opted for this solution because it seemed like a good compromise. Maybe we could revisit this decision later on if we see that other models also require to special case these writes. What do you think?

[andreadb]

Patch updated.

Address review comments.

This patch replaces the existing writes for conditional SIMD stores (i.e. WriteFMaskedStore, and WriteFMaskedStoreY) with the following new writes:

• WriteFMaskedStore32 [ XMM Packed Single ]
• WriteFMaskedStore32Y [ YMM Packed Single ]
• WriteFMaskedStore64 [ XMM Packed Double ]
• WriteFMaskedStore64T [ YMM Packed Double ]

Added a wrapper class named X86SchedWriteMaskMove in X86Schedule.td to describe both RM and MR variants for conditional SIMD moves in a single tablegen definition.
Instances of that class are then passed in input to multiclass avx_movmask_rm when constructing MASKMOVPS/PD definitions.

Since this patch introduces new writes, I had to update all the X86 scheduling models.

This patch is a NFC for all x86 models except BtVer2.

[lebedev.ri]

FWIW this does not appear to be the case on BdVer2:

$ ./bin/llvm-exegesis --mode=uops --opcode-name=VMASKMOVPSmr
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-072799.o
---
mode:            uops
key:
  instructions:
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0x0 %noreg XMM6 XMM11'
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0x40 %noreg XMM4 XMM9'
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0x80 %noreg XMM12 XMM12'
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0xc0 %noreg XMM6 XMM2'
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0x100 %noreg XMM1 XMM7'
    - 'VMASKMOVPSmr RDI i_0x1 %noreg i_0x140 %noreg XMM10 XMM15'
  config:          ''
  register_initial_values:
    - 'XMM6=0x0'
    - 'XMM11=0x0'
    - 'XMM4=0x0'
    - 'XMM9=0x0'
    - 'XMM12=0x0'
    - 'XMM2=0x0'
    - 'XMM1=0x0'
    - 'XMM7=0x0'
    - 'XMM10=0x0'
    - 'XMM15=0x0'
cpu_name:        bdver2
llvm_triple:     x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
  - { key: PdFPU0, value: 8.0055, per_snippet_value: 48.033 }
  - { key: PdFPU1, value: 4.0124, per_snippet_value: 24.0744 }
  - { key: PdFPU2, value: 2.0042, per_snippet_value: 12.0252 }
  - { key: PdFPU3, value: 4.0078, per_snippet_value: 24.0468 }
  - { key: NumMicroOps, value: 18.0142, per_snippet_value: 108.085 }
error:           ''
info:            instruction is parallel, repeating a random one.
assembled_snippet: 4883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F34244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F1C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F24244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F0C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F24244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F14244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F0C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F3C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F14244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F3C244883C410C462492E1FC462592E4F40C462192EA780000000C4E2492E97C0000000C4E2712EBF00010000C462292EBF40010000C462492E1FC462592E4F40C462192EA780000000C4E2492E97C0000000C4E2712EBF00010000C462292EBF40010000C462492E1FC462592E4F40C462192EA780000000C4E2492E97C0000000C3
...
$ ./bin/llvm-exegesis --mode=uops --opcode-name=VMASKMOVPDmr
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-28613e.o
---
mode:            uops
key:
  instructions:
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0x0 %noreg XMM8 XMM7'
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0x40 %noreg XMM14 XMM0'
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0x80 %noreg XMM11 XMM5'
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0xc0 %noreg XMM4 XMM11'
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0x100 %noreg XMM12 XMM11'
    - 'VMASKMOVPDmr RDI i_0x1 %noreg i_0x140 %noreg XMM4 XMM0'
  config:          ''
  register_initial_values:
    - 'XMM8=0x0'
    - 'XMM7=0x0'
    - 'XMM14=0x0'
    - 'XMM0=0x0'
    - 'XMM11=0x0'
    - 'XMM5=0x0'
    - 'XMM4=0x0'
    - 'XMM12=0x0'
cpu_name:        bdver2
llvm_triple:     x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
  - { key: PdFPU0, value: 7.9896, per_snippet_value: 47.9376 }
  - { key: PdFPU1, value: 4.0235, per_snippet_value: 24.141 }
  - { key: PdFPU2, value: 2.0042, per_snippet_value: 12.0252 }
  - { key: PdFPU3, value: 4.0077, per_snippet_value: 24.0462 }
  - { key: NumMicroOps, value: 18.0128, per_snippet_value: 108.077 }
error:           ''
info:            instruction is parallel, repeating a random one.
assembled_snippet: 4883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F04244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F3C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F34244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F04244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F1C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F2C244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C5FA6F24244883C4104883EC10C7042400000000C744240400000000C744240800000000C744240C00000000C57A6F24244883C410C4E2392F3FC4E2092F4740C4E2212FAF80000000C462592F9FC0000000C462192F9F00010000C4E2592F8740010000C4E2392F3FC4E2092F4740C4E2212FAF80000000C462592F9FC0000000C462192F9F00010000C4E2592F8740010000C4E2392F3FC4E2092F4740C4E2212FAF80000000C462592F9FC0000000C3
...
$ ./bin/llvm-exegesis --mode=uops --opcode-name=VMASKMOVPSYmr
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-f26721.o
---
mode:            uops
key:
  instructions:
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0x0 %noreg YMM5 YMM4'
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0x40 %noreg YMM2 YMM0'
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0x80 %noreg YMM15 YMM14'
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0xc0 %noreg YMM10 YMM13'
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0x100 %noreg YMM7 YMM15'
    - 'VMASKMOVPSYmr RDI i_0x1 %noreg i_0x140 %noreg YMM15 YMM5'
  config:          ''
  register_initial_values:
    - 'YMM5=0x0'
    - 'YMM4=0x0'
    - 'YMM2=0x0'
    - 'YMM0=0x0'
    - 'YMM15=0x0'
    - 'YMM14=0x0'
    - 'YMM10=0x0'
    - 'YMM13=0x0'
    - 'YMM7=0x0'
cpu_name:        bdver2
llvm_triple:     x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
  - { key: PdFPU0, value: 15.9929, per_snippet_value: 95.9574 }
  - { key: PdFPU1, value: 8.089, per_snippet_value: 48.534 }
  - { key: PdFPU2, value: 2.0012, per_snippet_value: 12.0072 }
  - { key: PdFPU3, value: 8.0068, per_snippet_value: 48.0408 }
  - { key: NumMicroOps, value: 34.018, per_snippet_value: 204.108 }
error:           ''
info:            instruction is parallel, repeating a random one.
assembled_snippet: 4883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F2C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F24244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F14244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F04244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F3C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F34244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F14244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F2C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F3C244883C420C4E2552E27C4E26D2E4740C462052EB780000000C4622D2EAFC0000000C462452EBF00010000C4E2052EAF40010000C4E2552E27C4E26D2E4740C462052EB780000000C4622D2EAFC0000000C462452EBF00010000C4E2052EAF40010000C4E2552E27C4E26D2E4740C462052EB780000000C4622D2EAFC0000000C3
...
$ ./bin/llvm-exegesis --mode=uops --opcode-name=VMASKMOVPDYmr
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-e45324.o
---
mode:            uops
key:
  instructions:
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0x0 %noreg YMM15 YMM5'
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0x40 %noreg YMM9 YMM10'
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0x80 %noreg YMM10 YMM7'
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0xc0 %noreg YMM1 YMM8'
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0x100 %noreg YMM10 YMM10'
    - 'VMASKMOVPDYmr RDI i_0x1 %noreg i_0x140 %noreg YMM13 YMM9'
  config:          ''
  register_initial_values:
    - 'YMM15=0x0'
    - 'YMM5=0x0'
    - 'YMM9=0x0'
    - 'YMM10=0x0'
    - 'YMM7=0x0'
    - 'YMM1=0x0'
    - 'YMM8=0x0'
    - 'YMM13=0x0'
cpu_name:        bdver2
llvm_triple:     x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
  - { key: PdFPU0, value: 16.0013, per_snippet_value: 96.0078 }
  - { key: PdFPU1, value: 8.0093, per_snippet_value: 48.0558 }
  - { key: PdFPU2, value: 2.0018, per_snippet_value: 12.0108 }
  - { key: PdFPU3, value: 8.0068, per_snippet_value: 48.0408 }
  - { key: NumMicroOps, value: 34.0168, per_snippet_value: 204.101 }
error:           ''
info:            instruction is parallel, repeating a random one.
assembled_snippet: 4883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F3C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F2C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F0C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F14244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F3C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C5FE6F0C244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F04244883C4204883EC20C7042400000000C744240400000000C744240800000000C744240C00000000C744241000000000C744241400000000C744241800000000C744241C00000000C57E6F2C244883C420C4E2052F2FC462352F5740C4E22D2FBF80000000C462752F87C0000000C4622D2F9700010000C462152F8F40010000C4E2052F2FC462352F5740C4E22D2FBF80000000C462752F87C0000000C4622D2F9700010000C462152F8F40010000C4E2052F2FC462352F5740C4E22D2FBF80000000C462752F87C0000000C3
...

[RKSimon]

LGTM - thanks @andreadb I think this is the way to go. As ever its up to the people responsible for the other models to tweak as necessary, as you said this is NFC for everything but btver2.

I don't see accurate numbers for these ops on Agner/instlatx64 for any target, I'm curious how they've checked the perf range for different mask register values (although Agner does mention that btver2 is often bad with VMASKMOVPS loads when mask == 0).

@lebedev.ri By the looks of it llvm-exegesis always uses zero registers for those tests - does it alter if you hack in other values?

[andreadb]

In D66801#1653861, @RKSimon wrote:

LGTM - thanks @andreadb I think this is the way to go. As ever its up to the people responsible for the other models to tweak as necessary, as you said this is NFC for everything but btver2.

I don't see accurate numbers for these ops on Agner/instlatx64 for any target, I'm curious how they've checked the perf range for different mask register values (although Agner does mention that btver2 is often bad with VMASKMOVPS loads when mask == 0).

@lebedev.ri By the looks of it llvm-exegesis always uses zero registers for those tests - does it alter if you hack in other values?

I was about to post a similar comment.

It may be worthy to rerun those experiments by forcing a different mask value. Otherwise, we don't know for sure if the zero-mask is treated specially on bdver2.

More in general: it would be better if exegesis uses a all-ones default for initial register values. That is what I tend to do when doing throughput analysis (actually, I tend to test both cases, i.e. the all-zero case and the all-ones case). On Jaguar, I know that no optimization is performed if registers are not set via a zero idiom. On entry to the benchmark loop, I set those registers to all-ones. For XMM/YMM registers, as you know, it is really straightforward (just use an all-ones (v)pcmpeq instead of a zero-idiom (v)xorps).
When reading counters, make sure that the initialization code is not counted too (to minimize the noise - all-ones idioms are executed, while all-zeroes are eliminated). I don't know exegesis enough, but I would advice for that change in default values if possible. At least, give an option for testing all-ones...

lib/Target/X86/X86InstrSSE.td

let Predicates = [HasAVX1Only] in {
   defm : vextract_lowering<"VEXTRACTF128", v32i8,  v16i8>;
 }
 
 //===----------------------------------------------------------------------===//
 // VMASKMOV - Conditional SIMD Packed Loads and Stores
 //
 multiclass avx_movmask_rm<bits<8> opc_rm, bits<8> opc_mr, string OpcodeStr,
                           Intrinsic IntLd, Intrinsic IntLd256,
-                          Intrinsic IntSt, Intrinsic IntSt256> {
+                          Intrinsic IntSt, Intrinsic IntSt256,
+                          X86SchedWriteMaskMove schedX,
+                          X86SchedWriteMaskMove schedY> {
   def rm  : AVX8I<opc_rm, MRMSrcMem, (outs VR128:$dst),
              (ins VR128:$src1, f128mem:$src2),
              !strconcat(OpcodeStr, "\t{$src2, $src1, $dst|$dst, $src1, $src2}"),
              [(set VR128:$dst, (IntLd addr:$src2, VR128:$src1))]>,
-             VEX_4V, Sched<[WriteFMaskedLoad]>;
+             VEX_4V, Sched<[schedX.RM]>;
   def Yrm : AVX8I<opc_rm, MRMSrcMem, (outs VR256:$dst),
              (ins VR256:$src1, f256mem:$src2),
              !strconcat(OpcodeStr, "\t{$src2, $src1, $dst|$dst, $src1, $src2}"),
              [(set VR256:$dst, (IntLd256 addr:$src2, VR256:$src1))]>,
-             VEX_4V, VEX_L, Sched<[WriteFMaskedLoadY]>;
+             VEX_4V, VEX_L, Sched<[schedY.RM]>;
   def mr  : AVX8I<opc_mr, MRMDestMem, (outs),
              (ins f128mem:$dst, VR128:$src1, VR128:$src2),
              !strconcat(OpcodeStr, "\t{$src2, $src1, $dst|$dst, $src1, $src2}"),
              [(IntSt addr:$dst, VR128:$src1, VR128:$src2)]>,
-             VEX_4V, Sched<[WriteFMaskedStore]>;
+             VEX_4V, Sched<[schedX.MR]>;
   def Ymr : AVX8I<opc_mr, MRMDestMem, (outs),
              (ins f256mem:$dst, VR256:$src1, VR256:$src2),
              !strconcat(OpcodeStr, "\t{$src2, $src1, $dst|$dst, $src1, $src2}"),
              [(IntSt256 addr:$dst, VR256:$src1, VR256:$src2)]>,
-             VEX_4V, VEX_L, Sched<[WriteFMaskedStoreY]>;
+             VEX_4V, VEX_L, Sched<[schedY.MR]>;
 }
 
 let ExeDomain = SSEPackedSingle in
 defm VMASKMOVPS : avx_movmask_rm<0x2C, 0x2E, "vmaskmovps",
                                  int_x86_avx_maskload_ps,
                                  int_x86_avx_maskload_ps_256,
                                  int_x86_avx_maskstore_ps,
-                                 int_x86_avx_maskstore_ps_256>;
+                                 int_x86_avx_maskstore_ps_256,
+                                 WriteFMaskMove32, WriteFMaskMove32Y>;
 let ExeDomain = SSEPackedDouble in
 defm VMASKMOVPD : avx_movmask_rm<0x2D, 0x2F, "vmaskmovpd",
                                  int_x86_avx_maskload_pd,
                                  int_x86_avx_maskload_pd_256,
                                  int_x86_avx_maskstore_pd,
-                                 int_x86_avx_maskstore_pd_256>;
+                                 int_x86_avx_maskstore_pd_256,
+                                 WriteFMaskMove64, WriteFMaskMove64Y>;
 
 //===----------------------------------------------------------------------===//
 // VPERMIL - Permute Single and Double Floating-Point Values
 //
 
 multiclass avx_permil<bits<8> opc_rm, bits<8> opc_rmi, string OpcodeStr,
                       RegisterClass RC, X86MemOperand x86memop_f,
                       X86MemOperand x86memop_i,

lib/Target/X86/X86SchedBroadwell.td

 defm : X86WriteRes<WriteFMaskedLoad,   [BWPort23,BWPort5], 7, [1,2], 3>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [BWPort23,BWPort5], 8, [1,2], 3>;
 defm : X86WriteRes<WriteFStore,        [BWPort237,BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreX,       [BWPort237,BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreY,       [BWPort237,BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNT,      [BWPort237,BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTX,     [BWPort237,BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTY,     [BWPort237,BWPort4], 1, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStore,  [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
-defm : X86WriteRes<WriteFMaskedStoreY, [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore64,  [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [BWPort0,BWPort4,BWPort237,BWPort15], 5, [1,1,1,1], 4>;
+
 defm : X86WriteRes<WriteFMove,         [BWPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [BWPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [BWPort5], 1, [1], 1>;
 
 defm : BWWriteResPair<WriteFAdd,    [BWPort1],  3, [1], 1, 5>; // Floating point add/sub.
 defm : BWWriteResPair<WriteFAddX,   [BWPort1],  3, [1], 1, 5>; // Floating point add/sub (XMM).
 defm : BWWriteResPair<WriteFAddY,   [BWPort1],  3, [1], 1, 6>; // Floating point add/sub (YMM/ZMM).
 defm : X86WriteResPairUnsupported<WriteFAddZ>;

lib/Target/X86/X86SchedHaswell.td

 defm : X86WriteRes<WriteFMaskedLoad,   [HWPort23,HWPort5], 8, [1,2], 3>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [HWPort23,HWPort5], 9, [1,2], 3>;
 defm : X86WriteRes<WriteFStore,        [HWPort237,HWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreX,       [HWPort237,HWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreY,       [HWPort237,HWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNT,      [HWPort237,HWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTX,     [HWPort237,HWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTY,     [HWPort237,HWPort4], 1, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStore,  [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
-defm : X86WriteRes<WriteFMaskedStoreY, [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore64,  [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [HWPort0,HWPort4,HWPort237,HWPort15], 5, [1,1,1,1], 4>;
+
 defm : X86WriteRes<WriteFMove,         [HWPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [HWPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [HWPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteEMMS,          [HWPort01,HWPort15,HWPort015,HWPort0156], 31, [8,1,21,1], 31>;
 
 defm : HWWriteResPair<WriteFAdd,    [HWPort1],  3, [1], 1, 5>;
 defm : HWWriteResPair<WriteFAddX,   [HWPort1],  3, [1], 1, 6>;
 defm : HWWriteResPair<WriteFAddY,   [HWPort1],  3, [1], 1, 7>;

lib/Target/X86/X86SchedSandyBridge.td

 defm : X86WriteRes<WriteFMaskedLoad,   [SBPort23,SBPort05], 8, [1,2], 3>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [SBPort23,SBPort05], 9, [1,2], 3>;
 defm : X86WriteRes<WriteFStore,        [SBPort23,SBPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreX,       [SBPort23,SBPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreY,       [SBPort23,SBPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreNT,      [SBPort23,SBPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreNTX,     [SBPort23,SBPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreNTY,     [SBPort23,SBPort4], 1, [1,1], 1>;
-defm : X86WriteRes<WriteFMaskedStore,  [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
-defm : X86WriteRes<WriteFMaskedStoreY, [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
+defm : X86WriteRes<WriteFMaskedStore64,  [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [SBPort4,SBPort01,SBPort23], 5, [1,1,1], 3>;
+
 defm : X86WriteRes<WriteFMove,         [SBPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [SBPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [SBPort5], 1, [1], 1>;
 defm : X86WriteRes<WriteEMMS,          [SBPort015], 31, [31], 31>;
 
 defm : SBWriteResPair<WriteFAdd,    [SBPort1],  3, [1], 1, 6>;
 defm : SBWriteResPair<WriteFAddX,   [SBPort1],  3, [1], 1, 6>;
 defm : SBWriteResPair<WriteFAddY,   [SBPort1],  3, [1], 1, 7>;

lib/Target/X86/X86SchedSkylakeClient.td

 defm : X86WriteRes<WriteFMaskedLoad,   [SKLPort23,SKLPort015], 7, [1,1], 2>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [SKLPort23,SKLPort015], 8, [1,1], 2>;
 defm : X86WriteRes<WriteFStore,        [SKLPort237,SKLPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreX,       [SKLPort237,SKLPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreY,       [SKLPort237,SKLPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNT,      [SKLPort237,SKLPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTX,     [SKLPort237,SKLPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTY,     [SKLPort237,SKLPort4], 1, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStore,  [SKLPort237,SKLPort0], 2, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStoreY, [SKLPort237,SKLPort0], 2, [1,1], 2>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [SKLPort237,SKLPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [SKLPort237,SKLPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore64,  [SKLPort237,SKLPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [SKLPort237,SKLPort0], 2, [1,1], 2>;
+
 defm : X86WriteRes<WriteFMove,         [SKLPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [SKLPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [SKLPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteEMMS,          [SKLPort05,SKLPort0156], 10, [9,1], 10>;
 
 defm : SKLWriteResPair<WriteFAdd,     [SKLPort01],  4, [1], 1, 5>; // Floating point add/sub.
 defm : SKLWriteResPair<WriteFAddX,    [SKLPort01],  4, [1], 1, 6>;
 defm : SKLWriteResPair<WriteFAddY,    [SKLPort01],  4, [1], 1, 7>;

lib/Target/X86/X86SchedSkylakeServer.td

 defm : X86WriteRes<WriteFMaskedLoad,   [SKXPort23,SKXPort015], 7, [1,1], 2>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [SKXPort23,SKXPort015], 8, [1,1], 2>;
 defm : X86WriteRes<WriteFStore,        [SKXPort237,SKXPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreX,       [SKXPort237,SKXPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreY,       [SKXPort237,SKXPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNT,      [SKXPort237,SKXPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTX,     [SKXPort237,SKXPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteFStoreNTY,     [SKXPort237,SKXPort4], 1, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStore,  [SKXPort237,SKXPort0], 2, [1,1], 2>;
-defm : X86WriteRes<WriteFMaskedStoreY, [SKXPort237,SKXPort0], 2, [1,1], 2>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [SKXPort237,SKXPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [SKXPort237,SKXPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore64,  [SKXPort237,SKXPort0], 2, [1,1], 2>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [SKXPort237,SKXPort0], 2, [1,1], 2>;
+
 defm : X86WriteRes<WriteFMove,         [SKXPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [SKXPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [SKXPort015], 1, [1], 1>;
 defm : X86WriteRes<WriteEMMS,          [SKXPort05,SKXPort0156], 10, [9,1], 10>;
 
 defm : SKXWriteResPair<WriteFAdd,      [SKXPort01],  4, [1], 1, 5>; // Floating point add/sub.
 defm : SKXWriteResPair<WriteFAddX,     [SKXPort01],  4, [1], 1, 6>;
 defm : SKXWriteResPair<WriteFAddY,     [SKXPort01],  4, [1], 1, 7>;

lib/Target/X86/X86Schedule.td

 class X86SchedWriteMoveLS<SchedWrite MoveRR,
                           SchedWrite LoadRM,
                           SchedWrite StoreMR> {
   SchedWrite RR = MoveRR;
   SchedWrite RM = LoadRM;
   SchedWrite MR = StoreMR;
 }
 
+// Multiclass that wraps masked load/store writes for a vector width.
+class X86SchedWriteMaskMove<SchedWrite LoadRM, SchedWrite StoreMR> {
+  SchedWrite RM = LoadRM;
+  SchedWrite MR = StoreMR;
+}
+
 // Multiclass that wraps X86SchedWriteMoveLS for each vector width.
 class X86SchedWriteMoveLSWidths<X86SchedWriteMoveLS sScl,
                                 X86SchedWriteMoveLS s128,
                                 X86SchedWriteMoveLS s256,
                                 X86SchedWriteMoveLS s512> {
   X86SchedWriteMoveLS Scl = sScl; // Scalar float/double operations.
   X86SchedWriteMoveLS MMX = sScl; // MMX operations.
   X86SchedWriteMoveLS XMM = s128; // XMM operations.

 def  WriteFMaskedLoad   : SchedWrite;
 def  WriteFMaskedLoadY  : SchedWrite;
 def  WriteFStore        : SchedWrite;
 def  WriteFStoreX       : SchedWrite;
 def  WriteFStoreY       : SchedWrite;
 def  WriteFStoreNT      : SchedWrite;
 def  WriteFStoreNTX     : SchedWrite;
 def  WriteFStoreNTY     : SchedWrite;
-def  WriteFMaskedStore  : SchedWrite;
-def  WriteFMaskedStoreY : SchedWrite;
+
+def  WriteFMaskedStore32  : SchedWrite;
+def  WriteFMaskedStore64  : SchedWrite;
+def  WriteFMaskedStore32Y : SchedWrite;
+def  WriteFMaskedStore64Y : SchedWrite;
+
 def  WriteFMove         : SchedWrite;
 def  WriteFMoveX        : SchedWrite;
 def  WriteFMoveY        : SchedWrite;
 
 defm WriteFAdd    : X86SchedWritePair<ReadAfterVecLd>;  // Floating point add/sub.
 defm WriteFAddX   : X86SchedWritePair<ReadAfterVecXLd>; // Floating point add/sub (XMM).
 defm WriteFAddY   : X86SchedWritePair<ReadAfterVecYLd>; // Floating point add/sub (YMM).
 defm WriteFAddZ   : X86SchedWritePair<ReadAfterVecYLd>; // Floating point add/sub (ZMM).

 def WriteVecMoveLSNTX
  : X86SchedWriteMoveLS<WriteVecMoveX, WriteVecLoadNT, WriteVecStoreNT>;
 def WriteVecMoveLSNTY
  : X86SchedWriteMoveLS<WriteVecMoveY, WriteVecLoadNTY, WriteVecStoreNTY>;
 def SchedWriteVecMoveLSNT
   : X86SchedWriteMoveLSWidths<WriteVecMoveLSNT, WriteVecMoveLSNTX,
                               WriteVecMoveLSNTY, WriteVecMoveLSNTY>;
 
+// Conditional SIMD Packed Loads and Stores wrappers.
+def WriteFMaskMove32
+  : X86SchedWriteMaskMove<WriteFMaskedLoad, WriteFMaskedStore32>;
+def WriteFMaskMove64
+  : X86SchedWriteMaskMove<WriteFMaskedLoad, WriteFMaskedStore64>;
+def WriteFMaskMove32Y
+  : X86SchedWriteMaskMove<WriteFMaskedLoadY, WriteFMaskedStore32Y>;
+def WriteFMaskMove64Y
+  : X86SchedWriteMaskMove<WriteFMaskedLoadY, WriteFMaskedStore64Y>;
+
 // Vector width wrappers.
 def SchedWriteFAdd
  : X86SchedWriteWidths<WriteFAdd, WriteFAddX, WriteFAddY, WriteFAddZ>;
 def SchedWriteFAdd64
  : X86SchedWriteWidths<WriteFAdd64, WriteFAdd64X, WriteFAdd64Y, WriteFAdd64Z>;
 def SchedWriteFHAdd
  : X86SchedWriteWidths<WriteFHAdd, WriteFHAdd, WriteFHAddY, WriteFHAddZ>;
 def SchedWriteFCmp

lib/Target/X86/X86ScheduleAtom.td

 defm : X86WriteResUnsupported<WriteFMaskedLoadY>;
 
 def  : WriteRes<WriteFStore,        [AtomPort0]>;
 def  : WriteRes<WriteFStoreX,       [AtomPort0]>;
 defm : X86WriteResUnsupported<WriteFStoreY>;
 def  : WriteRes<WriteFStoreNT,      [AtomPort0]>;
 def  : WriteRes<WriteFStoreNTX,     [AtomPort0]>;
 defm : X86WriteResUnsupported<WriteFStoreNTY>;
-defm : X86WriteResUnsupported<WriteFMaskedStore>;
-defm : X86WriteResUnsupported<WriteFMaskedStoreY>;
+defm : X86WriteResUnsupported<WriteFMaskedStore32>;
+defm : X86WriteResUnsupported<WriteFMaskedStore32Y>;
+defm : X86WriteResUnsupported<WriteFMaskedStore64>;
+defm : X86WriteResUnsupported<WriteFMaskedStore64Y>;
 
 def  : WriteRes<WriteFMove,         [AtomPort01]>;
 def  : WriteRes<WriteFMoveX,        [AtomPort01]>;
 defm : X86WriteResUnsupported<WriteFMoveY>;
 
 defm : X86WriteRes<WriteEMMS,       [AtomPort01], 5, [5], 1>;
 
 defm : AtomWriteResPair<WriteFAdd,           [AtomPort0],  [AtomPort0],  5,  5,  [5],  [5]>;

lib/Target/X86/X86ScheduleBdVer2.td

def PdWriteVMOVUPDYmrVMOVUPSYmr : SchedWriteRes<[PdStore, PdFPU1,  PdFPSTO]> {
   let NumMicroOps = 8;
 }
 def : InstRW<[PdWriteVMOVUPDYmrVMOVUPSYmr], (instrs VMOVUPDYmr, VMOVUPSYmr)>;
 
 defm : PdWriteRes<WriteFStoreNT,           [PdStore, PdFPU1,  PdFPSTO], 3>;
 defm : PdWriteRes<WriteFStoreNTX,          [PdStore, PdFPU1,  PdFPSTO], 3>;
 defm : PdWriteRes<WriteFStoreNTY,          [PdStore, PdFPU1,  PdFPSTO], 3, [2, 2, 2], 4>;
 
-defm : PdWriteRes<WriteFMaskedStore,       [PdStore, PdFPU01, PdFPFMA], 6, [1, 1, 188], 18>;
-defm : PdWriteRes<WriteFMaskedStoreY,      [PdStore, PdFPU01, PdFPFMA], 6, [2, 2, 376], 34>;
+defm : PdWriteRes<WriteFMaskedStore32,     [PdStore, PdFPU01, PdFPFMA], 6, [1, 1, 188], 18>;
+defm : PdWriteRes<WriteFMaskedStore64,     [PdStore, PdFPU01, PdFPFMA], 6, [1, 1, 188], 18>;
+defm : PdWriteRes<WriteFMaskedStore32Y,    [PdStore, PdFPU01, PdFPFMA], 6, [2, 2, 376], 34>;
+defm : PdWriteRes<WriteFMaskedStore64Y,    [PdStore, PdFPU01, PdFPFMA], 6, [2, 2, 376], 34>;
 
 defm : PdWriteRes<WriteFMove,              [PdFPU01, PdFPFMA]>;
 defm : PdWriteRes<WriteFMoveX,             [PdFPU01, PdFPFMA], 1, [1, 2]>;
 defm : PdWriteRes<WriteFMoveY,             [PdFPU01, PdFPFMA], 2, [2, 2], 2>;
 
 defm : PdWriteRes<WriteEMMS,               [PdFPU01, PdFPFMA], 2>;
 
 defm : PdWriteResXMMPair<WriteFAdd,         [PdFPU0, PdFPFMA],  5>;

lib/Target/X86/X86ScheduleBtVer2.td

 defm : X86WriteRes<WriteFMaskedLoadY,  [JLAGU, JFPU01, JFPX], 6, [2, 4, 4], 2>;
 
 defm : X86WriteRes<WriteFStore,        [JSAGU, JFPU1,  JSTC], 2, [1, 1, 1], 1>;
 defm : X86WriteRes<WriteFStoreX,       [JSAGU, JFPU1,  JSTC], 1, [1, 1, 1], 1>;
 defm : X86WriteRes<WriteFStoreY,       [JSAGU, JFPU1,  JSTC], 1, [1, 1, 1], 1>;
 defm : X86WriteRes<WriteFStoreNT,      [JSAGU, JFPU1,  JSTC], 3, [1, 1, 1], 1>;
 defm : X86WriteRes<WriteFStoreNTX,     [JSAGU, JFPU1,  JSTC], 3, [1, 1, 1], 1>;
 defm : X86WriteRes<WriteFStoreNTY,     [JSAGU, JFPU1,  JSTC], 3, [2, 2, 2], 1>;
-defm : X86WriteRes<WriteFMaskedStore,  [JSAGU, JFPU01, JFPX], 6, [1, 1, 4], 1>;
-defm : X86WriteRes<WriteFMaskedStoreY, [JSAGU, JFPU01, JFPX], 6, [2, 2, 4], 2>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [JFPU0, JFPA, JFPU1, JSTC, JLAGU, JSAGU, JALU01], 16, [1,1, 5, 5,4,4,4], 19>;
+defm : X86WriteRes<WriteFMaskedStore64,  [JFPU0, JFPA, JFPU1, JSTC, JLAGU, JSAGU, JALU01], 13, [1,1, 2, 2,2,2,2], 10>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [JFPU0, JFPA, JFPU1, JSTC, JLAGU, JSAGU, JALU01], 22, [1,1,10,10,8,8,8], 36>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [JFPU0, JFPA, JFPU1, JSTC, JLAGU, JSAGU, JALU01], 16, [1,1, 4, 4,4,4,4], 18>;
 
 defm : X86WriteRes<WriteFMove,         [JFPU01, JFPX], 1, [1, 1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [JFPU01, JFPX], 1, [1, 1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [JFPU01, JFPX], 1, [2, 2], 2>;
 
 defm : X86WriteRes<WriteEMMS,          [JFPU01, JFPX], 2, [1, 1], 1>;
 
 defm : JWriteResFpuPair<WriteFAdd,         [JFPU0, JFPA],  3>;

 
 def JWriteJVZEROUPPER: SchedWriteRes<[]> {
   let Latency = 46;
   let NumMicroOps = 37;
 }
 def : InstRW<[JWriteJVZEROUPPER], (instrs VZEROUPPER)>;
 
 ///////////////////////////////////////////////////////////////////////////////
+//  SSE2/AVX Store Selected Bytes of Double Quadword - (V)MASKMOVDQ
+///////////////////////////////////////////////////////////////////////////////
+
+def JWriteMASKMOVDQU: SchedWriteRes<[JFPU0, JFPA, JFPU1, JSTC, JLAGU, JSAGU, JALU01]> {
+  let Latency = 34;
+  let ResourceCycles = [1, 1, 2, 2, 2, 16, 42];
+  let NumMicroOps = 63;
+}
+def : InstRW<[JWriteMASKMOVDQU], (instrs MASKMOVDQU, MASKMOVDQU64,
+                                         VMASKMOVDQU, VMASKMOVDQU64)>;
+
+///////////////////////////////////////////////////////////////////////////////
 //  SchedWriteVariant definitions.
 ///////////////////////////////////////////////////////////////////////////////
 
 def JWriteZeroLatency : SchedWriteRes<[]> {
   let Latency = 0;
 }
 
 def JWriteZeroIdiomYmm : SchedWriteRes<[JFPU01, JFPX]> {

 // Section 21.8 [Dependency-breaking instructions].
 
 def JWriteZeroIdiom : SchedWriteVariant<[
     SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [JWriteZeroLatency]>,
     SchedVar<NoSchedPred,                          [WriteALU]>
 ]>;
 def : InstRW<[JWriteZeroIdiom], (instrs SUB32rr, SUB64rr,
                                         XOR32rr, XOR64rr)>;
 
 def JWriteFZeroIdiom : SchedWriteVariant<[
     SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [JWriteZeroLatency]>,
     SchedVar<NoSchedPred,                          [WriteFLogic]>
 ]>;
 def : InstRW<[JWriteFZeroIdiom], (instrs XORPSrr, VXORPSrr, XORPDrr, VXORPDrr,
                                          ANDNPSrr, VANDNPSrr,
                                          ANDNPDrr, VANDNPDrr)>;
 

lib/Target/X86/X86ScheduleSLM.td

 def  : WriteRes<WriteFMaskedLoad,   [SLM_MEC_RSV]> { let Latency = 3; }
 def  : WriteRes<WriteFMaskedLoadY,  [SLM_MEC_RSV]> { let Latency = 3; }
 def  : WriteRes<WriteFStore,        [SLM_MEC_RSV]>;
 def  : WriteRes<WriteFStoreX,       [SLM_MEC_RSV]>;
 def  : WriteRes<WriteFStoreY,       [SLM_MEC_RSV]>;
 def  : WriteRes<WriteFStoreNT,      [SLM_MEC_RSV]>;
 def  : WriteRes<WriteFStoreNTX,     [SLM_MEC_RSV]>;
 def  : WriteRes<WriteFStoreNTY,     [SLM_MEC_RSV]>;
-def  : WriteRes<WriteFMaskedStore,  [SLM_MEC_RSV]>;
-def  : WriteRes<WriteFMaskedStoreY, [SLM_MEC_RSV]>;
+
+def  : WriteRes<WriteFMaskedStore32,    [SLM_MEC_RSV]>;
+def  : WriteRes<WriteFMaskedStore32Y,   [SLM_MEC_RSV]>;
+def  : WriteRes<WriteFMaskedStore64,    [SLM_MEC_RSV]>;
+def  : WriteRes<WriteFMaskedStore64Y,   [SLM_MEC_RSV]>;
+
 def  : WriteRes<WriteFMove,         [SLM_FPC_RSV01]>;
 def  : WriteRes<WriteFMoveX,        [SLM_FPC_RSV01]>;
 def  : WriteRes<WriteFMoveY,        [SLM_FPC_RSV01]>;
 defm : X86WriteRes<WriteEMMS,       [SLM_FPC_RSV01], 10, [10], 9>;
 
 defm : SLMWriteResPair<WriteFAdd,     [SLM_FPC_RSV1], 3>;
 defm : SLMWriteResPair<WriteFAddX,    [SLM_FPC_RSV1], 3>;
 defm : SLMWriteResPair<WriteFAddY,    [SLM_FPC_RSV1], 3>;

lib/Target/X86/X86ScheduleZnver1.td

 defm : X86WriteRes<WriteFMaskedLoad,   [ZnAGU,ZnFPU01], 8, [1,1], 1>;
 defm : X86WriteRes<WriteFMaskedLoadY,  [ZnAGU,ZnFPU01], 8, [1,2], 2>;
 defm : X86WriteRes<WriteFStore,        [ZnAGU], 1, [1], 1>;
 defm : X86WriteRes<WriteFStoreX,       [ZnAGU], 1, [1], 1>;
 defm : X86WriteRes<WriteFStoreY,       [ZnAGU], 1, [1], 1>;
 defm : X86WriteRes<WriteFStoreNT,      [ZnAGU,ZnFPU2], 8, [1,1], 1>;
 defm : X86WriteRes<WriteFStoreNTX,     [ZnAGU], 1, [1], 1>;
 defm : X86WriteRes<WriteFStoreNTY,     [ZnAGU], 1, [1], 1>;
-defm : X86WriteRes<WriteFMaskedStore,  [ZnAGU,ZnFPU01], 4, [1,1], 1>;
-defm : X86WriteRes<WriteFMaskedStoreY, [ZnAGU,ZnFPU01], 5, [1,2], 2>;
+
+defm : X86WriteRes<WriteFMaskedStore32,  [ZnAGU,ZnFPU01], 4, [1,1], 1>;
+defm : X86WriteRes<WriteFMaskedStore32Y, [ZnAGU,ZnFPU01], 5, [1,2], 2>;
+defm : X86WriteRes<WriteFMaskedStore64,  [ZnAGU,ZnFPU01], 4, [1,1], 1>;
+defm : X86WriteRes<WriteFMaskedStore64Y, [ZnAGU,ZnFPU01], 5, [1,2], 2>;
+
 defm : X86WriteRes<WriteFMove,         [ZnFPU], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveX,        [ZnFPU], 1, [1], 1>;
 defm : X86WriteRes<WriteFMoveY,        [ZnFPU], 1, [1], 1>;
 
 defm : ZnWriteResFpuPair<WriteFAdd,      [ZnFPU0],  3>;
 defm : ZnWriteResFpuPair<WriteFAddX,     [ZnFPU0],  3>;
 defm : ZnWriteResFpuPair<WriteFAddY,     [ZnFPU0],  3>;
 defm : X86WriteResPairUnsupported<WriteFAddZ>;

test/tools/llvm-mca/X86/BtVer2/resources-avx1.s

 # CHECK-NEXT:  2      9     2.00    *                   vhsubps (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  2      1     1.00                        vinsertf128 $1, %xmm0, %ymm1, %ymm2
 # CHECK-NEXT:  2      6     1.00    *                   vinsertf128 $1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      1     0.50                        vinsertps $1, %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  1      6     1.00    *                   vinsertps $1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  1      5     1.00    *                   vlddqu  (%rax), %xmm2
 # CHECK-NEXT:  1      5     1.00    *                   vlddqu  (%rax), %ymm2
 # CHECK-NEXT:  1      3     1.00    *             U     vldmxcsr  (%rax)
-# CHECK-NEXT:  1      1     1.00    *      *      U     vmaskmovdqu %xmm0, %xmm1
+# CHECK-NEXT:  63     34    21.00   *      *      U     vmaskmovdqu %xmm0, %xmm1
 # CHECK-NEXT:  1      6     1.00    *                   vmaskmovpd  (%rax), %xmm0, %xmm2
 # CHECK-NEXT:  2      6     2.00    *                   vmaskmovpd  (%rax), %ymm0, %ymm2
-# CHECK-NEXT:  1      6     2.00    *      *            vmaskmovpd  %xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  2      6     2.00    *      *            vmaskmovpd  %ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  10     13    2.00    *      *            vmaskmovpd  %xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  18     16    4.00    *      *            vmaskmovpd  %ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  1      6     1.00    *                   vmaskmovps  (%rax), %xmm0, %xmm2
 # CHECK-NEXT:  2      6     2.00    *                   vmaskmovps  (%rax), %ymm0, %ymm2
-# CHECK-NEXT:  1      6     2.00    *      *            vmaskmovps  %xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  2      6     2.00    *      *            vmaskmovps  %ymm0, %ymm1, (%rax)
+# CHECK-NEXT:  19     16    5.00    *      *            vmaskmovps  %xmm0, %xmm1, (%rax)
+# CHECK-NEXT:  36     22    10.00   *      *            vmaskmovps  %ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  1      2     1.00                        vmaxpd  %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  1      7     1.00    *                   vmaxpd  (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      2     2.00                        vmaxpd  %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  2      7     2.00    *                   vmaxpd  (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  1      2     1.00                        vmaxps  %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  1      7     1.00    *                   vmaxps  (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  2      2     2.00                        vmaxps  %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  2      7     2.00    *                   vmaxps  (%rax), %ymm1, %ymm2

 # CHECK-NEXT: [9]   - JSAGU
 # CHECK-NEXT: [10]  - JSTC
 # CHECK-NEXT: [11]  - JVALU0
 # CHECK-NEXT: [12]  - JVALU1
 # CHECK-NEXT: [13]  - JVIMUL
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]
-# CHECK-NEXT: 56.00   -      -     365.00 915.00 447.50 461.50 394.00  -     51.00  132.00 135.50 159.50 38.00
+# CHECK-NEXT: 86.00  30.00   -     362.00 907.00 449.50 480.50 414.00  -     78.00  154.00 135.50 159.50 38.00
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]   Instructions:
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vaddpd  %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     vaddpd  (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -      -      -      -      -      -      -      -     vaddpd  %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -     2.00    -      -      -      -      -      -     vaddpd  (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vaddps  %xmm0, %xmm1, %xmm2

 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -     2.00    -      -      -      -      -      -     vhsubps (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00    -      -      -      -      -      -      -     vinsertf128 $1, %xmm0, %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00   1.00    -      -      -      -      -      -     vinsertf128 $1, (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -     vinsertps $1, %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50   1.00    -      -      -      -      -      -     vinsertps $1, (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -      -      -     0.50   0.50   1.00    -      -      -     0.50   0.50    -     vlddqu  (%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -     0.50   0.50   1.00    -      -      -     0.50   0.50    -     vlddqu  (%rax), %ymm2
 # CHECK-NEXT:  -      -      -      -      -      -      -     1.00    -      -      -      -      -      -     vldmxcsr  (%rax)
-# CHECK-NEXT:  -      -      -      -      -      -     1.00    -      -     1.00   1.00    -      -      -     vmaskmovdqu  %xmm0, %xmm1
+# CHECK-NEXT: 21.00  21.00   -     1.00    -     1.00   2.00   2.00    -     16.00  2.00    -      -      -     vmaskmovdqu  %xmm0, %xmm1
 # CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00   1.00    -      -      -      -      -      -     vmaskmovpd  (%rax), %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00   2.00    -      -      -      -      -      -     vmaskmovpd  (%rax), %ymm0, %ymm2
-# CHECK-NEXT:  -      -      -     2.00   2.00   0.50   0.50    -      -     1.00    -      -      -      -     vmaskmovpd  %xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  -      -      -     2.00   2.00   1.00   1.00    -      -     2.00    -      -      -      -     vmaskmovpd  %ymm0, %ymm1, (%rax)
+# CHECK-NEXT: 1.00   1.00    -     1.00    -     1.00   2.00   2.00    -     2.00   2.00    -      -      -     vmaskmovpd  %xmm0, %xmm1, (%rax)
+# CHECK-NEXT: 2.00   2.00    -     1.00    -     1.00   4.00   4.00    -     4.00   4.00    -      -      -     vmaskmovpd  %ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  -      -      -     1.00   1.00   1.00   1.00   1.00    -      -      -      -      -      -     vmaskmovps  (%rax), %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     2.00   2.00   2.00   2.00   2.00    -      -      -      -      -      -     vmaskmovps  (%rax), %ymm0, %ymm2
-# CHECK-NEXT:  -      -      -     2.00   2.00   0.50   0.50    -      -     1.00    -      -      -      -     vmaskmovps  %xmm0, %xmm1, (%rax)
-# CHECK-NEXT:  -      -      -     2.00   2.00   1.00   1.00    -      -     2.00    -      -      -      -     vmaskmovps  %ymm0, %ymm1, (%rax)
+# CHECK-NEXT: 2.00   2.00    -     1.00    -     1.00   5.00   4.00    -     4.00   5.00    -      -      -     vmaskmovps  %xmm0, %xmm1, (%rax)
+# CHECK-NEXT: 4.00   4.00    -     1.00    -     1.00   10.00  8.00    -     8.00   10.00   -      -      -     vmaskmovps  %ymm0, %ymm1, (%rax)
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vmaxpd  %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     vmaxpd  (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -      -      -      -      -      -      -      -     vmaxpd  %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -     2.00    -      -      -      -      -      -     vmaxpd  (%rax), %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vmaxps  %xmm0, %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     vmaxps  (%rax), %xmm1, %xmm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -      -      -      -      -      -      -      -     vmaxps  %ymm0, %ymm1, %ymm2
 # CHECK-NEXT:  -      -      -     2.00    -     2.00    -     2.00    -      -      -      -      -      -     vmaxps  (%rax), %ymm1, %ymm2

test/tools/llvm-mca/X86/BtVer2/resources-sse2.s

 # CHECK-NEXT:  2      7     1.00                        cvttsd2si %xmm0, %rcx
 # CHECK-NEXT:  2      12    1.00    *                   cvttsd2si (%rax), %ecx
 # CHECK-NEXT:  2      12    1.00    *                   cvttsd2si (%rax), %rcx
 # CHECK-NEXT:  1      19    19.00                       divpd %xmm0, %xmm2
 # CHECK-NEXT:  1      24    19.00   *                   divpd (%rax), %xmm2
 # CHECK-NEXT:  1      19    19.00                       divsd %xmm0, %xmm2
 # CHECK-NEXT:  1      24    19.00   *                   divsd (%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00    *      *      U     lfence
-# CHECK-NEXT:  1      1     1.00    *      *      U     maskmovdqu  %xmm0, %xmm1
+# CHECK-NEXT:  63     34    21.00   *      *      U     maskmovdqu  %xmm0, %xmm1
 # CHECK-NEXT:  1      2     1.00                        maxpd %xmm0, %xmm2
 # CHECK-NEXT:  1      7     1.00    *                   maxpd (%rax), %xmm2
 # CHECK-NEXT:  1      2     1.00                        maxsd %xmm0, %xmm2
 # CHECK-NEXT:  1      7     1.00    *                   maxsd (%rax), %xmm2
 # CHECK-NEXT:  1      1     1.00    *      *      U     mfence
 # CHECK-NEXT:  1      2     1.00                        minpd %xmm0, %xmm2
 # CHECK-NEXT:  1      7     1.00    *                   minpd (%rax), %xmm2
 # CHECK-NEXT:  1      2     1.00                        minsd %xmm0, %xmm2

 # CHECK-NEXT: [9]   - JSAGU
 # CHECK-NEXT: [10]  - JSTC
 # CHECK-NEXT: [11]  - JVALU0
 # CHECK-NEXT: [12]  - JVALU1
 # CHECK-NEXT: [13]  - JVIMUL
 
 # CHECK:      Resource pressure per iteration:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]
-# CHECK-NEXT: 17.00   -      -     49.00  204.00 128.50 141.50 118.00  -     16.00  54.00  67.50  67.50  12.00
+# CHECK-NEXT: 38.00  21.00   -     50.00  204.00 129.50 142.50 120.00  -     31.00  55.00  67.50  67.50  12.00
 
 # CHECK:      Resource pressure by instruction:
 # CHECK-NEXT: [0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]   Instructions:
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     addpd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     addpd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     addsd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     addsd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -     0.50   0.50   0.50   0.50    -      -      -      -      -      -      -     andnpd  %xmm0, %xmm2

 # CHECK-NEXT: 1.00    -      -     1.00    -     1.00   1.00    -      -      -     1.00    -      -      -     cvttsd2si %xmm0, %rcx
 # CHECK-NEXT: 1.00    -      -     1.00    -     1.00   1.00   1.00    -      -     1.00    -      -      -     cvttsd2si (%rax), %ecx
 # CHECK-NEXT: 1.00    -      -     1.00    -     1.00   1.00   1.00    -      -     1.00    -      -      -     cvttsd2si (%rax), %rcx
 # CHECK-NEXT:  -      -      -      -     19.00   -     1.00    -      -      -      -      -      -      -     divpd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     19.00   -     1.00   1.00    -      -      -      -      -      -     divpd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -     19.00   -     1.00    -      -      -      -      -      -      -     divsd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -      -     19.00   -     1.00   1.00    -      -      -      -      -      -     divsd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -     lfence
-# CHECK-NEXT:  -      -      -      -      -      -     1.00    -      -     1.00   1.00    -      -      -     maskmovdqu  %xmm0, %xmm1
+# CHECK-NEXT: 21.00  21.00   -     1.00    -     1.00   2.00   2.00    -     16.00  2.00    -      -      -     maskmovdqu  %xmm0, %xmm1
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     maxpd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     maxpd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     maxsd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     maxsd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -      -      -      -      -      -      -     1.00    -      -      -      -     mfence
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     minpd %xmm0, %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -     1.00    -      -      -      -      -      -     minpd (%rax), %xmm2
 # CHECK-NEXT:  -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     minsd %xmm0, %xmm2