[X86] Improved sched models for X86 XCHG*rr and XADD*rr instructions

[avt77]

Summary

This patch improves sched models for XCHG*rr X86 instrs removing unnecessary redefinitions of instr infos. This patch is based on results of D48222.

[lebedev.ri]

I think this needs mca test coverage first.

llvm/test/tools/llvm-mca/X86$ grep -ri XCHG | wc -l
0

[avt77]

But we have a lot of XCHG tests in other places. For example:

.../llvm/test/CodeGen/X86$ grep -ri XCHG | wc -l
730

Or:

schedule-x86_32.ll:; ATOM-NEXT: xchgl %eax, %eax # sched: [2:1.00]
schedule-x86_32.ll:; ATOM-NEXT: xchgl %ecx, %eax # sched: [2:1.00]
schedule-x86_32.ll:; ATOM-NEXT: xchgl %eax, (%edx) # sched: [3:1.50]
schedule-x86_32.ll:; SLM-LABEL: test_xchg_32:
schedule-x86_32.ll:; SLM-NEXT: xchgl %eax, %eax # sched: [1:0.50]
schedule-x86_32.ll:; SLM-NEXT: xchgl %ecx, %eax # sched: [1:0.50]
schedule-x86_32.ll:; SLM-NEXT: xchgl %eax, (%edx) # sched: [4:2.00]
schedule-x86_32.ll:; SANDY-LABEL: test_xchg_32:
schedule-x86_32.ll:; SANDY-NEXT: xchgl %eax, %eax # sched: [2:1.00]
schedule-x86_32.ll:; SANDY-NEXT: xchgl %ecx, %eax # sched: [2:1.00]
schedule-x86_32.ll:; SANDY-NEXT: xchgl %eax, (%edx) # sched: [6:1.00]
schedule-x86_32.ll:; HASWELL-LABEL: test_xchg_32:
schedule-x86_32.ll:; HASWELL-NEXT: xchgl %eax, %eax # sched: [2:0.75]
schedule-x86_32.ll:; HASWELL-NEXT: xchgl %ecx, %eax # sched: [2:0.75]
schedule-x86_32.ll:; HASWELL-NEXT: xchgl %eax, (%edx) # sched: [9:1.00]
schedule-x86_32.ll:; BROADWELL-LABEL: test_xchg_32:
schedule-x86_32.ll:; BROADWELL-NEXT: xchgl %eax, %eax # sched: [2:0.75]
schedule-x86_32.ll:; BROADWELL-NEXT: xchgl %ecx, %eax # sched: [2:0.75]
schedule-x86_32.ll:; BROADWELL-NEXT: xchgl %eax, (%edx) # sched: [8:1.00]
schedule-x86_32.ll:; SKYLAKE-LABEL: test_xchg_32:
schedule-x86_32.ll:; SKYLAKE-NEXT: xchgl %eax, %eax # sched: [2:0.75]
schedule-x86_32.ll:; SKYLAKE-NEXT: xchgl %ecx, %eax # sched: [2:0.75]
schedule-x86_32.ll:; SKYLAKE-NEXT: xchgl %eax, (%edx) # sched: [10:1.25]

And this patch does not change any of those tests - it's very importent.

[avt77]

But of course you're right: we need such mca tests. I'll try to prepare such tests for XCHG* and CMPXCHG* instrs (I've just started to work with CMPXCHG*).

[lebedev.ri]

In general - looks good, assuming that after rebasing ontop of D49912 there are no regressions.
A few nits.
The SLM & btver2 - are those placeholders, or those are real values from agner?

lib/Target/X86/X86SchedHaswell.td

128 ↗

(On Diff #157497)

Nit: align?

lib/Target/X86/X86SchedSkylakeClient.td

115 ↗

(On Diff #157497)

Nit: align?

lib/Target/X86/X86Schedule.td

123 ↗

(On Diff #157497)

// Exchange the contents of two operands
?

lib/Target/X86/X86ScheduleBtVer2.td

173 ↗

(On Diff #157497)

These are from agner?

[avt77]

In D49861#1178401, @lebedev.ri wrote:

The SLM & btver2 - are those placeholders, or those are real values from agner?

They did not have any notes about XCHG*rr instrs and that means that default values were OK for them. I kept the default values and tests were OK but it's better to check with agner of course. I'll do it.

[avt77]

lib/Target/X86/X86ScheduleBtVer2.td

173 ↗

(On Diff #157497)

Agner shows different values for r8,r8 and r,r while AMD64_16h_InstrLatency_1.1 shows 'ucode' for all variants. I used the default values because it keeps the current tests untouched. That's our old problem: what should be selected as "right" value?

[lebedev.ri]

lib/Target/X86/X86ScheduleBtVer2.td

173 ↗

(On Diff #157497)

Ideally - what llvm-exegesis says for these.
Realistically, i'd guess the current change - just cleanup, don't change test output - is ok.

[RKSimon]

Tag XADD instructions with WriteXCHG as well?

[avt77]

In D49861#1184028, @RKSimon wrote:

Tag XADD instructions with WriteXCHG as well?

Should we use WriteXCHGADD in this case?

[RKSimon]

In D49861#1184168, @avt77 wrote:

In D49861#1184028, @RKSimon wrote:

Tag XADD instructions with WriteXCHG as well?

Should we use WriteXCHGADD in this case?

If you want to rename them that's OK - but XADD is just a form of XCHG (it just happens to perform the sum as well).

[avt77]

In D49861#1184028, @RKSimon wrote:

Tag XADD instructions with WriteXCHG as well?

Here we have the same situation like in CMPXCHG. We can implement it in 2 ways:

• like it's done in D50070
• or with help of WriteSequence like craig.topper suggests here.

I think the both implementations are identical but let's select the one.

[avt77]

XADD*rr instrs were added: I did not add rm version because we should decide how to do it beter:

like in D50070
or with help of WriteSequence suggested by craig.topper

[RKSimon]

lib/Target/X86/X86Schedule.td

123 ↗

(On Diff #157497)

Add

// Compare+Exchange - TODO RMW support.

[avt77]

The comment was added:

def WriteXCHG : SchedWrite; // Compare+Exchange - TODO RMW support.

[RKSimon]

I think you're almost there but D49912 must be completed and committed first

[avt77]

In D49861#1187225, @RKSimon wrote:

I think you're almost there but D49912 must be completed and committed first

I'm working on it but every change requires additional testing and time.

[avt77]

The patch was rebased after D49912.

[RKSimon]

LGTM

llvm/trunk/lib/Target/X86/X86InstrInfo.td

def NAME#64rm : RI<opc, MRMSrcMem, (outs GR64:$dst),
                          GR64:$dst,
                          (!cast<PatFrag>(frag # "_64") addr:$ptr, GR64:$val))]>;
   }
 }
 
 defm XCHG    : ATOMIC_SWAP<0x86, 0x87, "xchg", "atomic_swap">, NotMemoryFoldable;
 
 // Swap between registers.
-let SchedRW = [WriteALU] in {
+let SchedRW = [WriteXCHG] in {
 let Constraints = "$src1 = $dst1, $src2 = $dst2", hasSideEffects = 0 in {
 def XCHG8rr : I<0x86, MRMSrcReg, (outs GR8:$dst1, GR8:$dst2),
                 (ins GR8:$src1, GR8:$src2),
                 "xchg{b}\t{$src2, $src1|$src1, $src2}", []>, NotMemoryFoldable;
 def XCHG16rr : I<0x87, MRMSrcReg, (outs GR16:$dst1, GR16:$dst2),
                  (ins GR16:$src1, GR16:$src2),
                  "xchg{w}\t{$src2, $src1|$src1, $src2}", []>,
                  OpSize16, NotMemoryFoldable;
def XCHG32ar : I<0x90, AddRegFrm, (outs GR32:$dst), (ins GR32:$src),
                   "xchg{l}\t{$src, %eax|eax, $src}", []>, OpSize32;
 let Uses = [RAX], Defs = [RAX] in
 def XCHG64ar : RI<0x90, AddRegFrm, (outs GR64:$dst), (ins GR64:$src),
                   "xchg{q}\t{$src, %rax|rax, $src}", []>;
 }
 } // SchedRW
 
 let hasSideEffects = 0, Constraints = "$src1 = $dst1, $src2 = $dst2",
-    Defs = [EFLAGS], SchedRW = [WriteALU] in {
+    Defs = [EFLAGS], SchedRW = [WriteXCHG] in {
 def XADD8rr : I<0xC0, MRMDestReg, (outs GR8:$dst1, GR8:$dst2),
                 (ins GR8:$src1, GR8:$src2),
                 "xadd{b}\t{$src2, $src1|$src1, $src2}", []>, TB;
 def XADD16rr : I<0xC1, MRMDestReg, (outs GR16:$dst1, GR16:$dst2),
                  (ins GR16:$src1, GR16:$src2),
                  "xadd{w}\t{$src2, $src1|$src1, $src2}", []>, TB, OpSize16;
 def XADD32rr : I<0xC1, MRMDestReg, (outs GR32:$dst1, GR32:$dst2),
                   (ins GR32:$src1, GR32:$src2),

llvm/trunk/lib/Target/X86/X86SchedBroadwell.td

 defm : BWWriteResPair<WriteDiv64,  [BWPort0, BWDivider], 25, [1, 10]>;
 defm : BWWriteResPair<WriteIDiv8,  [BWPort0, BWDivider], 25, [1, 10]>;
 defm : BWWriteResPair<WriteIDiv16, [BWPort0, BWDivider], 25, [1, 10]>;
 defm : BWWriteResPair<WriteIDiv32, [BWPort0, BWDivider], 25, [1, 10]>;
 defm : BWWriteResPair<WriteIDiv64, [BWPort0, BWDivider], 25, [1, 10]>;
 
 defm : X86WriteRes<WriteBSWAP32,   [BWPort15], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64,   [BWPort06, BWPort15], 2, [1, 1], 2>;
+defm : X86WriteRes<WriteXCHG,      [BWPort0156], 2, [3], 3>;
 
 defm : BWWriteResPair<WriteCRC32, [BWPort1],   3>;
 def : WriteRes<WriteIMulH, []> { let Latency = 3; } // Integer multiplication, high part.
 
 def : WriteRes<WriteLEA, [BWPort15]>; // LEA instructions can't fold loads.
 
 defm : BWWriteResPair<WriteCMOV,  [BWPort06], 1>; // Conditional move.
 defm : BWWriteResPair<WriteCMOV2, [BWPort06,BWPort0156], 2, [1,1], 2>; // // Conditional (CF + ZF flag) move.

 def BWWriteResGroup28 : SchedWriteRes<[BWPort5]> {
   let Latency = 3;
   let NumMicroOps = 1;
   let ResourceCycles = [1];
 }
 def: InstRW<[BWWriteResGroup28], (instregex "VPBROADCASTBrr",
                                             "VPBROADCASTWrr")>;
 
-def BWWriteResGroup30 : SchedWriteRes<[BWPort0156]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-  let ResourceCycles = [3];
-}
-def: InstRW<[BWWriteResGroup30], (instrs XADD8rr, XADD16rr, XADD32rr, XADD64rr,
-                                         XCHG8rr, XCHG16rr, XCHG32rr, XCHG64rr,
-                                         XCHG16ar, XCHG32ar, XCHG64ar)>;
-
 def BWWriteResGroup33 : SchedWriteRes<[BWPort5,BWPort0156]> {
   let Latency = 3;
   let NumMicroOps = 3;
   let ResourceCycles = [2,1];
 }
 def: InstRW<[BWWriteResGroup33], (instregex "MMX_PACKSSDWirr",
                                             "MMX_PACKSSWBirr",
                                             "MMX_PACKUSWBirr")>;

llvm/trunk/lib/Target/X86/X86SchedHaswell.td

 // Arithmetic.
 defm : HWWriteResPair<WriteALU,    [HWPort0156], 1>;
 defm : HWWriteResPair<WriteADC,    [HWPort06, HWPort0156], 2, [1,1], 2>;
 defm : HWWriteResPair<WriteIMul,   [HWPort1],   3>;
 defm : HWWriteResPair<WriteIMul64, [HWPort1],   3>;
 
 defm : X86WriteRes<WriteBSWAP32,   [HWPort15], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64,   [HWPort06, HWPort15], 2, [1,1], 2>;
+defm : X86WriteRes<WriteXCHG, [HWPort0156], 2, [3], 3>;
 
 def  : WriteRes<WriteIMulH, []> { let Latency = 3; }
 
 // Integer shifts and rotates.
 defm : HWWriteResPair<WriteShift,  [HWPort06],  1>;
 
 // SHLD/SHRD.
 defm : X86WriteRes<WriteSHDrri, [HWPort1], 3, [1], 1>;
def HWWriteResGroup53_1 : SchedWriteRes<[HWPort5,HWPort23]> {
   let NumMicroOps = 2;
   let ResourceCycles = [1,1];
 }
 def: InstRW<[HWWriteResGroup53_1], (instregex "VPMOVSXBWYrm",
                                               "VPMOVSXDQYrm",
                                               "VPMOVSXWDYrm",
                                               "VPMOVZXWDYrm")>;
 
-def HWWriteResGroup54 : SchedWriteRes<[HWPort0156]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-  let ResourceCycles = [3];
-}
-def: InstRW<[HWWriteResGroup54], (instrs XADD8rr, XADD16rr, XADD32rr, XADD64rr,
-                                         XCHG8rr, XCHG16rr, XCHG32rr, XCHG64rr,
-                                         XCHG16ar, XCHG32ar, XCHG64ar)>;
-
 def HWWriteResGroup57 : SchedWriteRes<[HWPort5,HWPort0156]> {
   let Latency = 3;
   let NumMicroOps = 3;
   let ResourceCycles = [2,1];
 }
 def: InstRW<[HWWriteResGroup57], (instregex "MMX_PACKSSDWirr",
                                             "MMX_PACKSSWBirr",
                                             "MMX_PACKUSWBirr")>;

llvm/trunk/lib/Target/X86/X86SchedSandyBridge.td

 def : WriteRes<WriteMove,    [SBPort015]>;
 def : WriteRes<WriteZero,    []>;
 
 // Arithmetic.
 defm : SBWriteResPair<WriteALU,    [SBPort015], 1>;
 defm : SBWriteResPair<WriteADC,    [SBPort05,SBPort015], 2, [1,1], 2>;
 defm : SBWriteResPair<WriteIMul,   [SBPort1],   3>;
 defm : SBWriteResPair<WriteIMul64, [SBPort1],   3>;
+defm : X86WriteRes<WriteXCHG,      [SBPort015], 2, [3], 3>;
 
 defm : X86WriteRes<WriteBSWAP32,   [SBPort1], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64,   [SBPort1,SBPort05], 2, [1,1], 2>;
 
 defm : SBWriteResPair<WriteDiv8,   [SBPort0, SBDivider], 25, [1, 10]>;
 defm : SBWriteResPair<WriteDiv16,  [SBPort0, SBDivider], 25, [1, 10]>;
 defm : SBWriteResPair<WriteDiv32,  [SBPort0, SBDivider], 25, [1, 10]>;
 defm : SBWriteResPair<WriteDiv64,  [SBPort0, SBDivider], 25, [1, 10]>;
def SBWriteResGroup23_2 : SchedWriteRes<[SBPort05]> {
   let ResourceCycles = [3];
 }
 def: InstRW<[SBWriteResGroup23_2], (instregex "ROL(8|16|32|64)rCL",
                                               "ROR(8|16|32|64)rCL",
                                               "SAR(8|16|32|64)rCL",
                                               "SHL(8|16|32|64)rCL",
                                               "SHR(8|16|32|64)rCL")>;
 
-def SBWriteResGroup25 : SchedWriteRes<[SBPort015]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-  let ResourceCycles = [3];
-}
-def: InstRW<[SBWriteResGroup25], (instrs XADD8rr, XADD16rr, XADD32rr, XADD64rr,
-                                         XCHG8rr, XCHG16rr, XCHG32rr, XCHG64rr,
-                                         XCHG16ar, XCHG32ar, XCHG64ar)>;
-
 def SBWriteResGroup25_1 : SchedWriteRes<[SBPort23,SBPort015]> {
   let Latency = 7;
   let NumMicroOps = 3;
   let ResourceCycles = [1,2];
 }
 def: InstRW<[SBWriteResGroup25_1], (instrs LEAVE, LEAVE64)>;
 
 def SBWriteResGroup26_2 : SchedWriteRes<[SBPort0,SBPort1,SBPort5]> {

llvm/trunk/lib/Target/X86/X86SchedSkylakeClient.td

 // Arithmetic.
 defm : SKLWriteResPair<WriteALU,    [SKLPort0156], 1>; // Simple integer ALU op.
 defm : SKLWriteResPair<WriteADC,    [SKLPort06],   1>; // Integer ALU + flags op.
 defm : SKLWriteResPair<WriteIMul,   [SKLPort1],    3>; // Integer multiplication.
 defm : SKLWriteResPair<WriteIMul64, [SKLPort1],    3>; // Integer 64-bit multiplication.
 
 defm : X86WriteRes<WriteBSWAP32,    [SKLPort15], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64,    [SKLPort06, SKLPort15], 2, [1,1], 2>;
+defm : X86WriteRes<WriteXCHG, [SKLPort0156], 2, [3], 3>;
 
 defm : SKLWriteResPair<WriteDiv8,   [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteDiv16,  [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteDiv32,  [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteDiv64,  [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteIDiv8,  [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteIDiv16, [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
 defm : SKLWriteResPair<WriteIDiv32, [SKLPort0, SKLDivider], 25, [1,10], 1, 4>;
def SKLWriteResGroup33 : SchedWriteRes<[SKLPort06]> {
   let ResourceCycles = [3];
 }
 def: InstRW<[SKLWriteResGroup33], (instregex "ROL(8|16|32|64)rCL",
                                              "ROR(8|16|32|64)rCL",
                                              "SAR(8|16|32|64)rCL",
                                              "SHL(8|16|32|64)rCL",
                                              "SHR(8|16|32|64)rCL")>;
 
-def SKLWriteResGroup34 : SchedWriteRes<[SKLPort0156]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-  let ResourceCycles = [3];
-}
-def: InstRW<[SKLWriteResGroup34], (instrs XADD8rr, XADD16rr, XADD32rr, XADD64rr,
-                                          XCHG8rr, XCHG16rr, XCHG32rr, XCHG64rr,
-                                          XCHG16ar, XCHG32ar, XCHG64ar)>;
-
 def SKLWriteResGroup35 : SchedWriteRes<[SKLPort0,SKLPort5]> {
   let Latency = 3;
   let NumMicroOps = 3;
   let ResourceCycles = [1,2];
 }
 def: InstRW<[SKLWriteResGroup35], (instregex "MMX_PH(ADD|SUB)SWrr")>;
 
 def SKLWriteResGroup36 : SchedWriteRes<[SKLPort5,SKLPort01]> {

llvm/trunk/lib/Target/X86/X86SchedSkylakeServer.td

 // Arithmetic.
 defm : SKXWriteResPair<WriteALU,    [SKXPort0156], 1>; // Simple integer ALU op.
 defm : SKXWriteResPair<WriteADC,    [SKXPort06],   1>; // Integer ALU + flags op.
 defm : SKXWriteResPair<WriteIMul,   [SKXPort1],    3>; // Integer multiplication.
 defm : SKXWriteResPair<WriteIMul64, [SKXPort1],    3>; // Integer 64-bit multiplication.
 
 defm : X86WriteRes<WriteBSWAP32, [SKXPort15], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64, [SKXPort06, SKXPort15], 2, [1,1], 2>;
+defm : X86WriteRes<WriteXCHG,       [SKXPort0156], 2, [3], 3>;
 
 defm : SKXWriteResPair<WriteDiv8,   [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteDiv16,  [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteDiv32,  [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteDiv64,  [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteIDiv8,  [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteIDiv16, [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
 defm : SKXWriteResPair<WriteIDiv32, [SKXPort0, SKXDivider], 25, [1,10], 1, 4>;
def SKXWriteResGroup35 : SchedWriteRes<[SKXPort06]> {
   let ResourceCycles = [3];
 }
 def: InstRW<[SKXWriteResGroup35], (instregex "ROL(8|16|32|64)rCL",
                                              "ROR(8|16|32|64)rCL",
                                              "SAR(8|16|32|64)rCL",
                                              "SHL(8|16|32|64)rCL",
                                              "SHR(8|16|32|64)rCL")>;
 
-def SKXWriteResGroup36 : SchedWriteRes<[SKXPort0156]> {
-  let Latency = 2;
-  let NumMicroOps = 3;
-  let ResourceCycles = [3];
-}
-def: InstRW<[SKXWriteResGroup36], (instrs XADD8rr, XADD16rr, XADD32rr, XADD64rr,
-                                          XCHG8rr, XCHG16rr, XCHG32rr, XCHG64rr,
-                                          XCHG16ar, XCHG32ar, XCHG64ar)>;
-
 def SKXWriteResGroup37 : SchedWriteRes<[SKXPort0,SKXPort5]> {
   let Latency = 3;
   let NumMicroOps = 3;
   let ResourceCycles = [1,2];
 }
 def: InstRW<[SKXWriteResGroup37], (instregex "MMX_PH(ADD|SUB)SWrr")>;
 
 def SKXWriteResGroup38 : SchedWriteRes<[SKXPort5,SKXPort01]> {

llvm/trunk/lib/Target/X86/X86Schedule.td

 def  WriteADCRMW : WriteSequence<[WriteADCLd, WriteStore]>;
 defm WriteIMul   : X86SchedWritePair; // Integer multiplication.
 defm WriteIMul64 : X86SchedWritePair; // Integer 64-bit multiplication.
 def  WriteIMulH  : SchedWrite;        // Integer multiplication, high part.
 def  WriteLEA    : SchedWrite;        // LEA instructions can't fold loads.
 
 def  WriteBSWAP32 : SchedWrite; // Byte Order (Endianness) 32-bit Swap.
 def  WriteBSWAP64 : SchedWrite; // Byte Order (Endianness) 64-bit Swap.
+def  WriteXCHG   : SchedWrite;        // Compare+Exchange - TODO RMW support.
 
 // Integer division.
 defm WriteDiv8   : X86SchedWritePair;
 defm WriteDiv16  : X86SchedWritePair;
 defm WriteDiv32  : X86SchedWritePair;
 defm WriteDiv64  : X86SchedWritePair;
 defm WriteIDiv8  : X86SchedWritePair;
 defm WriteIDiv16 : X86SchedWritePair;

llvm/trunk/lib/Target/X86/X86ScheduleAtom.td

 // Arithmetic.
 ////////////////////////////////////////////////////////////////////////////////
 
 defm : AtomWriteResPair<WriteALU,    [AtomPort01], [AtomPort0]>;
 defm : AtomWriteResPair<WriteADC,    [AtomPort01], [AtomPort0]>;
 defm : AtomWriteResPair<WriteIMul,   [AtomPort01], [AtomPort01],  7,  7,  [7],  [7]>;
 defm : AtomWriteResPair<WriteIMul64, [AtomPort01], [AtomPort01], 12, 12, [12], [12]>;
 
+defm : X86WriteRes<WriteXCHG,        [AtomPort01], 2, [2], 1>;
 defm : X86WriteRes<WriteBSWAP32,     [AtomPort0], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64,     [AtomPort0], 1, [1], 1>;
 
 defm : AtomWriteResPair<WriteDiv8,   [AtomPort01], [AtomPort01], 50, 68, [50], [68]>;
 defm : AtomWriteResPair<WriteDiv16,  [AtomPort01], [AtomPort01], 50, 50, [50], [50]>;
 defm : AtomWriteResPair<WriteDiv32,  [AtomPort01], [AtomPort01], 50, 50, [50], [50]>;
 defm : AtomWriteResPair<WriteDiv64,  [AtomPort01], [AtomPort01],130,130,[130],[130]>;
 defm : AtomWriteResPair<WriteIDiv8,  [AtomPort01], [AtomPort01], 62, 62, [62], [62]>;
def AtomWrite01_2 : SchedWriteRes<[AtomPort01]> {
   let ResourceCycles = [2];
 }
 def : InstRW<[AtomWrite01_2], (instrs LEAVE, LEAVE64, POP16r,
                                       PUSH16rmm, PUSH32rmm, PUSH64rmm,
                                       LODSB, LODSL, LODSQ, LODSW,
                                       SCASB, SCASL, SCASQ, SCASW)>;
 def : InstRW<[AtomWrite01_2], (instregex "BT(C|R|S)(16|32|64)mi8",
                                          "PUSH(CS|DS|ES|FS|GS|SS)(16|32|64)",
-                                         "XADD(8|16|32|64)rr",
-                                         "XCHG(8|16|32|64)(ar|rr)",
                                          "(ST|ISTT)_F(P)?(16|32|64)?(m|rr)",
                                          "MMX_P(ADD|SUB)Qirr",
                                          "MOV(S|Z)X16rr8",
                                          "MOV(UPS|UPD|DQU)mr",
                                          "MASKMOVDQU(64)?",
                                          "P(ADD|SUB)Qrr")>;
 
 def AtomWrite01_3 : SchedWriteRes<[AtomPort01]> {

llvm/trunk/lib/Target/X86/X86ScheduleBtVer2.td

 defm : JWriteResIntPair<WriteALU,    [JALU01], 1>;
 defm : JWriteResIntPair<WriteADC,    [JALU01], 1, [2]>;
 defm : JWriteResIntPair<WriteIMul,   [JALU1, JMul], 3, [1, 1], 2>; // i8/i16/i32 multiplication
 defm : JWriteResIntPair<WriteIMul64, [JALU1, JMul], 6, [1, 4], 2>; // i64 multiplication
 defm : X86WriteRes<WriteIMulH,       [JALU1], 6, [4], 1>;
 
 defm : X86WriteRes<WriteBSWAP32, [JALU01], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64, [JALU01], 1, [1], 1>;
+defm : X86WriteRes<WriteXCHG,        [JALU01], 1, [1], 1>;
 
 defm : JWriteResIntPair<WriteDiv8,   [JALU1, JDiv], 12, [1, 12], 1>;
 defm : JWriteResIntPair<WriteDiv16,  [JALU1, JDiv], 17, [1, 17], 2>;
 defm : JWriteResIntPair<WriteDiv32,  [JALU1, JDiv], 25, [1, 25], 2>;
 defm : JWriteResIntPair<WriteDiv64,  [JALU1, JDiv], 41, [1, 41], 2>;
 defm : JWriteResIntPair<WriteIDiv8,  [JALU1, JDiv], 12, [1, 12], 1>;
 defm : JWriteResIntPair<WriteIDiv16, [JALU1, JDiv], 17, [1, 17], 2>;
 defm : JWriteResIntPair<WriteIDiv32, [JALU1, JDiv], 25, [1, 25], 2>;

llvm/trunk/lib/Target/X86/X86ScheduleSLM.td

 
 defm : SLMWriteResPair<WriteALU,    [SLM_IEC_RSV01], 1>;
 defm : SLMWriteResPair<WriteADC,    [SLM_IEC_RSV01], 1>;
 defm : SLMWriteResPair<WriteIMul,   [SLM_IEC_RSV1],  3>;
 defm : SLMWriteResPair<WriteIMul64, [SLM_IEC_RSV1],  3>;
 
 defm : X86WriteRes<WriteBSWAP32, [SLM_IEC_RSV01], 1, [1], 1>;
 defm : X86WriteRes<WriteBSWAP64, [SLM_IEC_RSV01], 1, [1], 1>;
+defm : X86WriteRes<WriteXCHG,       [SLM_IEC_RSV01], 1, [1], 1>;
 
 defm : SLMWriteResPair<WriteShift,  [SLM_IEC_RSV0],  1>;
 
 defm : X86WriteRes<WriteSHDrri, [SLM_IEC_RSV0],  1, [1], 1>;
 defm : X86WriteRes<WriteSHDrrcl,[SLM_IEC_RSV0],  1, [1], 1>;
 defm : X86WriteRes<WriteSHDmri, [SLM_MEC_RSV, SLM_IEC_RSV0], 4, [2, 1], 2>;
 defm : X86WriteRes<WriteSHDmrcl,[SLM_MEC_RSV, SLM_IEC_RSV0], 4, [2, 1], 2>;
 

llvm/trunk/lib/Target/X86/X86ScheduleZnver1.td

 def : WriteRes<WriteLEA, [ZnALU]>;
 defm : ZnWriteResPair<WriteALU,   [ZnALU], 1>;
 defm : ZnWriteResPair<WriteADC,   [ZnALU], 1>;
 defm : ZnWriteResPair<WriteIMul,   [ZnALU1, ZnMultiplier], 4>;
 defm : ZnWriteResPair<WriteIMul64, [ZnALU1, ZnMultiplier], 4, [1,1], 2>;
 
 defm : X86WriteRes<WriteBSWAP32, [ZnALU], 1, [4], 1>;
 defm : X86WriteRes<WriteBSWAP64, [ZnALU], 1, [4], 1>;
+defm : X86WriteRes<WriteXCHG, [ZnALU], 1, [2], 2>;
 
 defm : ZnWriteResPair<WriteShift, [ZnALU], 1>;
 
 defm : X86WriteRes<WriteSHDrri, [ZnALU], 1, [1], 1>;
 defm : X86WriteResUnsupported<WriteSHDrrcl>;
 defm : X86WriteResUnsupported<WriteSHDmri>;
 defm : X86WriteResUnsupported<WriteSHDmrcl>;
 

 // r16,m.
 def : InstRW<[WriteALULd, ReadAfterLd], (instregex "MOV16rm")>;
 
 // MOVSX, MOVZX.
 // r,m.
 def : InstRW<[WriteLoad], (instregex "MOV(S|Z)X32rm(8|16)")>;
 
 // XCHG.
-// r,r.
-def ZnWriteXCHG : SchedWriteRes<[ZnALU]> {
-  let NumMicroOps = 2;
-  let ResourceCycles = [2];
-}
-
-def : InstRW<[ZnWriteXCHG], (instregex "XCHG(8|16|32|64)rr", "XCHG(16|32|64)ar")>;
-
 // r,m.
 def ZnWriteXCHGrm : SchedWriteRes<[ZnAGU, ZnALU]> {
   let Latency = 5;
   let NumMicroOps = 2;
 }
 def : InstRW<[ZnWriteXCHGrm, ReadAfterLd], (instregex "XCHG(8|16|32|64)rm")>;
 
 def : InstRW<[WriteMicrocoded], (instrs XLAT)>;

 
 // SCAS.
 def : InstRW<[WriteMicrocoded], (instregex "SCAS(B|W|L|Q)")>;
 
 // STOS
 def : InstRW<[WriteMicrocoded], (instregex "STOS(B|L|Q|W)")>;
 
 // XADD.
+def ZnXADD : SchedWriteRes<[ZnALU]>;
+def : InstRW<[ZnXADD], (instregex "XADD(8|16|32|64)rr")>;
 def : InstRW<[WriteMicrocoded], (instregex "XADD(8|16|32|64)rm")>;
 
 //=== Floating Point x87 Instructions ===//
 //-- Move instructions --//
 
 def ZnWriteFLDr : SchedWriteRes<[ZnFPU13]> ;
 
 def ZnWriteSTr: SchedWriteRes<[ZnFPU23]> {