[x86] avoid false dependency stall on 'sbb' with same source reg

[spatel]

Summary

This is effectively inverting the transform added with D116804 because the downside of the false dependency of something like "sbb %eax, %eax" is much greater than the upside of eliminating a zeroing instruction on (all?) Intel CPUs.

[spatel]

llvm/test/CodeGen/X86/sbb-false-dep.ll

30

I'm not sure why this is a mov $0, but it is the same as with D116804 reverted.

[craig.topper]

Is the getSBBZero function you're modifying not in tree yet?

[spatel]

In D118843#3291793, @craig.topper wrote:

Is the getSBBZero function you're modifying not in tree yet?

Sorry - yes, I made that a local preliminary/NFC patch. I just pushed it up to main:
f523e83b204ea3e4ab80df6b

[craig.topper]

I think I prefer the opposite tuning flag polarity. It's not truely a false dep. It's more of a missed special case. The semantics of SBB imply the sources are read.

[RKSimon]

In D118843#3292007, @craig.topper wrote:

I think I prefer the opposite tuning flag polarity. It's not truely a false dep. It's more of a missed special case. The semantics of SBB imply the sources are read.

+1 TuningSBBDepBreaking flag makes more sense as an opt-in optimization

[spatel]

Patch updated:

1. Inverted polarity of the CPU attribute - default assumes we do not have the sbb idiom (we should zero the register).
2. Added a test file to check that the CPU attribute is on the right set of CPUs (Agner's guide says it is effectively any 64-bit AMD CPU).

We get more diffs than were created by D116804 because this is adding a zero op to every SETCC_CARRY node rather than just the ones that were altered by the earlier patch.

There might be a way to reduce the diffs, but I haven't found the predicate yet. Safer to err on the side of executing an unnecessary zero op rather than missing a false dependency stall?

[craig.topper]

llvm/lib/Target/X86/X86.td

448

I'm not sure about the use of the word "zero" in this. I liked TuningSBBDepBreaking

[spatel]

Patch updated:
Changed feature name from "HasSBBZeroIdiom" to "HasSBBDepBreaking".

[RKSimon]

LGTM - cheers

[craig.topper]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

468

This has a -Wparentheses warning with gcc.

[spatel]

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

468

Thanks!
be059a1263c6

llvm/lib/Target/X86/X86.td

 def TuningPOPCNTFalseDeps : SubtargetFeature<"false-deps-popcnt",
                                      "HasPOPCNTFalseDeps", "true",
                                      "POPCNT has a false dependency on dest register">;
 
 def TuningLZCNTFalseDeps : SubtargetFeature<"false-deps-lzcnt-tzcnt",
                                      "HasLZCNTFalseDeps", "true",
                                      "LZCNT/TZCNT have a false dependency on dest register">;
 
+def TuningSBBFalseDeps : SubtargetFeature<"false-deps-sbb",
+                                     "HasSBBFalseDeps", "true",
+                                     "SBB has a false dependency on same src register">;
+
 // On recent X86 (port bound) processors, its preferable to combine to a single shuffle
 // using a variable mask over multiple fixed shuffles.
 def TuningFastVariableCrossLaneShuffle
     : SubtargetFeature<"fast-variable-crosslane-shuffle",
                        "HasFastVariableCrossLaneShuffle",
                        "true", "Cross-lane shuffles with variable masks are fast">;
 def TuningFastVariablePerLaneShuffle
     : SubtargetFeature<"fast-variable-perlane-shuffle",
list<SubtargetFeature> X86_64V4Features = !listconcat(X86_64V3Features, [
     FeatureCDI,
     FeatureDQI,
     FeatureVLX,
   ]);
 
   // Nehalem
   list<SubtargetFeature> NHMFeatures = X86_64V2Features;
   list<SubtargetFeature> NHMTuning = [TuningMacroFusion,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
 
   // Westmere
   list<SubtargetFeature> WSMAdditionalFeatures = [FeaturePCLMUL];
   list<SubtargetFeature> WSMTuning = NHMTuning;
   list<SubtargetFeature> WSMFeatures =
     !listconcat(NHMFeatures, WSMAdditionalFeatures);
 
   // Sandybridge
   list<SubtargetFeature> SNBAdditionalFeatures = [FeatureAVX,
                                                   FeatureXSAVE,
                                                   FeatureXSAVEOPT];
   list<SubtargetFeature> SNBTuning = [TuningMacroFusion,
                                       TuningSlow3OpsLEA,
                                       TuningSlowDivide64,
                                       TuningSlowUAMem32,
                                       TuningFastScalarFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningPOPCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> SNBFeatures =
     !listconcat(WSMFeatures, SNBAdditionalFeatures);
 
   // Ivybridge
   list<SubtargetFeature> IVBAdditionalFeatures = [FeatureRDRAND,
                                                   FeatureF16C,
                                                   FeatureFSGSBase];
list<SubtargetFeature> HSWTuning = [TuningMacroFusion,
                                       TuningSlowDivide64,
                                       TuningFastScalarFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningFastVariableCrossLaneShuffle,
                                       TuningFastVariablePerLaneShuffle,
                                       TuningPOPCNTFalseDeps,
                                       TuningLZCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> HSWFeatures =
     !listconcat(IVBFeatures, HSWAdditionalFeatures);
 
   // Broadwell
   list<SubtargetFeature> BDWAdditionalFeatures = [FeatureADX,
                                                   FeatureRDSEED,
                                                   FeaturePRFCHW];
list<SubtargetFeature> SKLTuning = [TuningFastGather,
                                       TuningSlowDivide64,
                                       TuningFastScalarFSQRT,
                                       TuningFastVectorFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningFastVariableCrossLaneShuffle,
                                       TuningFastVariablePerLaneShuffle,
                                       TuningPOPCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> SKLFeatures =
     !listconcat(BDWFeatures, SKLAdditionalFeatures);
 
   // Skylake-AVX512
   list<SubtargetFeature> SKXAdditionalFeatures = [FeatureAES,
                                                   FeatureXSAVEC,
                                                   FeatureXSAVES,
list<SubtargetFeature> SKXTuning = [TuningFastGather,
                                       TuningFastScalarFSQRT,
                                       TuningFastVectorFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningFastVariableCrossLaneShuffle,
                                       TuningFastVariablePerLaneShuffle,
                                       TuningPrefer256Bit,
                                       TuningPOPCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> SKXFeatures =
     !listconcat(BDWFeatures, SKXAdditionalFeatures);
 
   // Cascadelake
   list<SubtargetFeature> CLXAdditionalFeatures = [FeatureVNNI];
   list<SubtargetFeature> CLXTuning = SKXTuning;
   list<SubtargetFeature> CLXFeatures =
list<SubtargetFeature> CNLTuning = [TuningFastGather,
                                       TuningSlowDivide64,
                                       TuningFastScalarFSQRT,
                                       TuningFastVectorFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningFastVariableCrossLaneShuffle,
                                       TuningFastVariablePerLaneShuffle,
                                       TuningPrefer256Bit,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> CNLFeatures =
     !listconcat(SKLFeatures, CNLAdditionalFeatures);
 
   // Icelake
   list<SubtargetFeature> ICLAdditionalFeatures = [FeatureBITALG,
                                                   FeatureVAES,
                                                   FeatureVBMI2,
list<SubtargetFeature> ICLTuning = [TuningFastGather,
                                       TuningSlowDivide64,
                                       TuningFastScalarFSQRT,
                                       TuningFastVectorFSQRT,
                                       TuningFastSHLDRotate,
                                       TuningFast15ByteNOP,
                                       TuningFastVariableCrossLaneShuffle,
                                       TuningFastVariablePerLaneShuffle,
                                       TuningPrefer256Bit,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> ICLFeatures =
     !listconcat(CNLFeatures, ICLAdditionalFeatures);
 
   // Icelake Server
   list<SubtargetFeature> ICXAdditionalFeatures = [FeaturePCONFIG,
                                                   FeatureCLWB,
                                                   FeatureWBNOINVD];
def ProcessorFeatures {
   list<SubtargetFeature> AtomTuning = [ProcIntelAtom,
                                        TuningSlowUAMem16,
                                        TuningLEAForSP,
                                        TuningSlowDivide32,
                                        TuningSlowDivide64,
                                        TuningSlowTwoMemOps,
                                        TuningLEAUsesAG,
                                        TuningPadShortFunctions,
+                                       TuningSBBFalseDeps,
                                        TuningInsertVZEROUPPER];
 
   // Silvermont
   list<SubtargetFeature> SLMAdditionalFeatures = [FeatureSSE42,
                                                   FeatureCRC32,
                                                   FeaturePOPCNT,
                                                   FeaturePCLMUL,
                                                   FeaturePRFCHW,
                                                   FeatureRDRAND];
   list<SubtargetFeature> SLMTuning = [TuningUseSLMArithCosts,
                                       TuningSlowTwoMemOps,
                                       TuningSlowLEA,
                                       TuningSlowIncDec,
                                       TuningSlowDivide64,
                                       TuningSlowPMULLD,
                                       TuningFast7ByteNOP,
                                       TuningFastMOVBE,
                                       TuningPOPCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> SLMFeatures =
     !listconcat(AtomFeatures, SLMAdditionalFeatures);
 
   // Goldmont
   list<SubtargetFeature> GLMAdditionalFeatures = [FeatureAES,
                                                   FeatureSHA,
                                                   FeatureRDSEED,
                                                   FeatureXSAVE,
                                                   FeatureXSAVEOPT,
                                                   FeatureXSAVEC,
                                                   FeatureXSAVES,
                                                   FeatureCLFLUSHOPT,
                                                   FeatureFSGSBase];
   list<SubtargetFeature> GLMTuning = [TuningUseGLMDivSqrtCosts,
                                       TuningSlowTwoMemOps,
                                       TuningSlowLEA,
                                       TuningSlowIncDec,
                                       TuningFastMOVBE,
                                       TuningPOPCNTFalseDeps,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> GLMFeatures =
     !listconcat(SLMFeatures, GLMAdditionalFeatures);
 
   // Goldmont Plus
   list<SubtargetFeature> GLPAdditionalFeatures = [FeaturePTWRITE,
                                                   FeatureRDPID];
   list<SubtargetFeature> GLPTuning = [TuningUseGLMDivSqrtCosts,
                                       TuningSlowTwoMemOps,
                                       TuningSlowLEA,
                                       TuningSlowIncDec,
                                       TuningFastMOVBE,
+                                      TuningSBBFalseDeps,
                                       TuningInsertVZEROUPPER];
   list<SubtargetFeature> GLPFeatures =
     !listconcat(GLMFeatures, GLPAdditionalFeatures);
 
   // Tremont
   list<SubtargetFeature> TRMAdditionalFeatures = [FeatureCLWB,
                                                   FeatureGFNI];
   list<SubtargetFeature> TRMTuning = GLPTuning;
list<SubtargetFeature> KNLFeatures = [FeatureX87,
                                         FeaturePRFCHW];
   list<SubtargetFeature> KNLTuning = [TuningSlowDivide64,
                                       TuningSlow3OpsLEA,
                                       TuningSlowIncDec,
                                       TuningSlowTwoMemOps,
                                       TuningPreferMaskRegisters,
                                       TuningFastGather,
                                       TuningFastMOVBE,
+                                      TuningSBBFalseDeps,
                                       TuningSlowPMADDWD];
   // TODO Add AVX5124FMAPS/AVX5124VNNIW features
   list<SubtargetFeature> KNMFeatures =
     !listconcat(KNLFeatures, [FeatureVPOPCNTDQ]);
 
   // Barcelona
   list<SubtargetFeature> BarcelonaFeatures = [FeatureX87,
                                               FeatureCMPXCHG8B,

 // constructor checks that any CPU used in 64-bit mode has Feature64Bit enabled.
 // It has no effect on code generation.
 def : ProcModel<"generic", SandyBridgeModel,
                 [FeatureX87, FeatureCMPXCHG8B, Feature64Bit],
                 [TuningSlow3OpsLEA,
                  TuningSlowDivide64,
                  TuningSlowIncDec,
                  TuningMacroFusion,
+                 TuningSBBFalseDeps,
                  TuningInsertVZEROUPPER]>;
 
 def : Proc<"i386",            [FeatureX87],
                               [TuningSlowUAMem16, TuningInsertVZEROUPPER]>;
 def : Proc<"i486",            [FeatureX87],
                               [TuningSlowUAMem16, TuningInsertVZEROUPPER]>;
 def : Proc<"i586",            [FeatureX87, FeatureCMPXCHG8B],
                               [TuningSlowUAMem16, TuningInsertVZEROUPPER]>;

 // knobs which need to be tuned differently for AMD chips, we might consider
 // forming a common base for them.
 def : ProcModel<"x86-64", SandyBridgeModel, ProcessorFeatures.X86_64V1Features,
 [
   TuningSlow3OpsLEA,
   TuningSlowDivide64,
   TuningSlowIncDec,
   TuningMacroFusion,
+  TuningSBBFalseDeps,
   TuningInsertVZEROUPPER
 ]>;
 
 // x86-64 micro-architecture levels.
 def : ProcModel<"x86-64-v2", SandyBridgeModel, ProcessorFeatures.X86_64V2Features,
                 ProcessorFeatures.SNBTuning>;
 // Close to Haswell.
 def : ProcModel<"x86-64-v3", HaswellModel, ProcessorFeatures.X86_64V3Features,

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

SDValue getSBBZero(SDNode *N) {
             CurDAG->getMachineNode(
                 TargetOpcode::SUBREG_TO_REG, dl, MVT::i64,
                 CurDAG->getTargetConstant(0, dl, MVT::i64), Zero,
                 CurDAG->getTargetConstant(X86::sub_32bit, dl, MVT::i32)),
             0);
       }
 
       // Copy flags to the EFLAGS register and glue it to next node.
-      SDValue EFLAGS = CurDAG->getCopyToReg(
-          CurDAG->getEntryNode(), dl, X86::EFLAGS, N->getOperand(2), SDValue());
+      unsigned Opcode = N->getOpcode();
+      assert(Opcode == X86ISD::SBB || Opcode == X86ISD::SETCC_CARRY &&
+             "Unexpected opcode for SBB materialization");
+      unsigned FlagOpIndex = Opcode == X86ISD::SBB ? 2 : 1;
+      SDValue EFLAGS =
+          CurDAG->getCopyToReg(CurDAG->getEntryNode(), dl, X86::EFLAGS,
+                               N->getOperand(FlagOpIndex), SDValue());
 
       // Create a 64-bit instruction if the result is 64-bits otherwise use the
       // 32-bit version.
       unsigned Opc = VT == MVT::i64 ? X86::SBB64rr : X86::SBB32rr;
       MVT SBBVT = VT == MVT::i64 ? MVT::i64 : MVT::i32;
       VTs = CurDAG->getVTList(SBBVT, MVT::i32);
       return SDValue(
           CurDAG->getMachineNode(Opc, dl, VTs,

   }
 
   case ISD::STORE:
     if (foldLoadStoreIntoMemOperand(Node))
       return;
     break;
 
   case X86ISD::SETCC_CARRY: {
+    MVT VT = Node->getSimpleValueType(0);
+    SDValue Result;
+    if (Subtarget->hasSBBFalseDeps()) {
+      Result = getSBBZero(Node);
+    } else {
       // We have to do this manually because tblgen will put the eflags copy in
       // the wrong place if we use an extract_subreg in the pattern.
-    MVT VT = Node->getSimpleValueType(0);
-
       // Copy flags to the EFLAGS register and glue it to next node.
       SDValue EFLAGS =
           CurDAG->getCopyToReg(CurDAG->getEntryNode(), dl, X86::EFLAGS,
                                Node->getOperand(1), SDValue());
 
       // Create a 64-bit instruction if the result is 64-bits otherwise use the
       // 32-bit version.
       unsigned Opc = VT == MVT::i64 ? X86::SETB_C64r : X86::SETB_C32r;
       MVT SetVT = VT == MVT::i64 ? MVT::i64 : MVT::i32;
-    SDValue Result = SDValue(
-        CurDAG->getMachineNode(Opc, dl, SetVT, EFLAGS, EFLAGS.getValue(1)), 0);
+      Result = SDValue(
+          CurDAG->getMachineNode(Opc, dl, SetVT, EFLAGS, EFLAGS.getValue(1)),
+          0);
+    }
 
     // For less than 32-bits we need to extract from the 32-bit node.
     if (VT == MVT::i8 || VT == MVT::i16) {
       int SubIndex = VT == MVT::i16 ? X86::sub_16bit : X86::sub_8bit;
       Result = CurDAG->getTargetExtractSubreg(SubIndex, dl, VT, Result);
     }
 
     ReplaceUses(SDValue(Node, 0), Result);

llvm/lib/Target/X86/X86Subtarget.h

class X86Subtarget final : public X86GenSubtargetInfo {
   bool UseLeaForSP = false;
 
   /// True if POPCNT instruction has a false dependency on the destination register.
   bool HasPOPCNTFalseDeps = false;
 
   /// True if LZCNT/TZCNT instructions have a false dependency on the destination register.
   bool HasLZCNTFalseDeps = false;
 
+  /// True if SBB instruction has a false dependency with same source register.
+  bool HasSBBFalseDeps = false;
+
   /// True if its preferable to combine to a single cross-lane shuffle
   /// using a variable mask over multiple fixed shuffles.
   bool HasFastVariableCrossLaneShuffle = false;
 
   /// True if its preferable to combine to a single per-lane shuffle
   /// using a variable mask over multiple fixed shuffles.
   bool HasFastVariablePerLaneShuffle = false;
 
public:
   bool isPMADDWDSlow() const { return IsPMADDWDSlow; }
   bool isUnalignedMem16Slow() const { return IsUAMem16Slow; }
   bool isUnalignedMem32Slow() const { return IsUAMem32Slow; }
   bool hasSSEUnalignedMem() const { return HasSSEUnalignedMem; }
   bool hasCmpxchg16b() const { return HasCmpxchg16b && is64Bit(); }
   bool useLeaForSP() const { return UseLeaForSP; }
   bool hasPOPCNTFalseDeps() const { return HasPOPCNTFalseDeps; }
   bool hasLZCNTFalseDeps() const { return HasLZCNTFalseDeps; }
+  bool hasSBBFalseDeps() const { return HasSBBFalseDeps; }
   bool hasFastVariableCrossLaneShuffle() const {
     return HasFastVariableCrossLaneShuffle;
   }
   bool hasFastVariablePerLaneShuffle() const {
     return HasFastVariablePerLaneShuffle;
   }
   bool insertVZEROUPPER() const { return InsertVZEROUPPER; }
   bool hasFastGather() const { return HasFastGather; }

llvm/test/CodeGen/X86/pr32588.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mcpu=generic -mtriple=x86_64-linux | FileCheck %s
 
 @c = external dso_local local_unnamed_addr global i32, align 4
 @b = external dso_local local_unnamed_addr global i32, align 4
 @d = external dso_local local_unnamed_addr global i32, align 4
 
 define void @fn1() {
 ; CHECK-LABEL: fn1:
 ; CHECK:       # %bb.0:
+; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    cmpl $1, c(%rip)
 ; CHECK-NEXT:    sbbl %eax, %eax
 ; CHECK-NEXT:    andl $1, %eax
 ; CHECK-NEXT:    movl %eax, d(%rip)
 ; CHECK-NEXT:    retq
   %t0 = load i32, i32* @c, align 4
   %tobool1 = icmp eq i32 %t0, 0
   %xor = zext i1 %tobool1 to i32

llvm/test/CodeGen/X86/sbb-false-dep.ll

 ; CHECK-NEXT:    movq %rsi, %rbx
 ; CHECK-NEXT:    movq %rdi, %r12
 ; CHECK-NEXT:    movq (%rsi), %rdi
 ; CHECK-NEXT:    movq 8(%rsi), %rsi
 ; CHECK-NEXT:    movq %rbx, %rdx
 ; CHECK-NEXT:    callq foo1@PLT
 ; CHECK-NEXT:    movq 8(%rbx), %rax
 ; CHECK-NEXT:    movq (%rax), %rdx
+; CHECK-NEXT:    xorl %ebp, %ebp
 ; CHECK-NEXT:    movl %r13d, %ecx
 ; CHECK-NEXT:    negl %ecx
-; CHECK-NEXT:    sbbq %rbp, %rbp
-; CHECK-NEXT:    orq %rdx, %rbp
-; CHECK-NEXT:    cmpl $1, %r13d
+; CHECK-NEXT:    movl $0, %eax
 ; CHECK-NEXT:    sbbq %rax, %rax
 ; CHECK-NEXT:    orq %rdx, %rax
+; CHECK-NEXT:    cmpl $1, %r13d
+; CHECK-NEXT:    sbbq %rbp, %rbp
+; CHECK-NEXT:    orq %rdx, %rbp
 ; CHECK-NEXT:    subq $8, %rsp
 ; CHECK-NEXT:    movq %r12, %rdi
 ; CHECK-NEXT:    movl %r15d, %esi
 ; CHECK-NEXT:    movl %r14d, %edx
 ; CHECK-NEXT:    xorl %ecx, %ecx
 ; CHECK-NEXT:    xorl %r8d, %r8d
 ; CHECK-NEXT:    xorl %r9d, %r9d
-; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    callq foo2@PLT
 ; CHECK-NEXT:    addq $40, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15

llvm/test/CodeGen/X86/select.ll

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpq $1, %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq %rsi, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test9:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpq $1, %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq %rsi, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test9:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %eax
 ; ATHLON-NEXT:    movl $-1, %edx

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpq $1, %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq %rsi, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test9a:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpq $1, %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq %rsi, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test9a:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %eax
 ; ATHLON-NEXT:    movl $-1, %edx

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpq $1, %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq %rsi, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test9b:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpq $1, %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq %rsi, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test9b:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    xorl %edx, %edx
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    sete %dl

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpq $1, %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq $1, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test10:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpq $1, %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq $1, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test10:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    xorl %edx, %edx
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %ecx

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    negq %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq %rsi, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test11:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negq %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq %rsi, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test11:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %eax
 ; ATHLON-NEXT:    movl $-1, %edx

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    negq %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq %rsi, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test11a:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negq %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq %rsi, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test11a:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %eax
 ; ATHLON-NEXT:    movl $-1, %edx

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    negl %edi
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    orl $42, %eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: eqzero_const_or_all_ones:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negl %edi
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    orl $42, %eax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: eqzero_const_or_all_ones:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    orl $42, %eax

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpl $1, %edi
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    orl $42, %eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: nezero_const_or_all_ones:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpl $1, %edi
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    orl $42, %eax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: nezero_const_or_all_ones:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    cmpl $1, {{[0-9]+}}(%esp)
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    orl $42, %eax
 ; ATHLON-NEXT:    retl

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpq $1, %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    orq $42, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: eqzero_all_ones_or_const:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpq $1, %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    orq $42, %rax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: eqzero_all_ones_or_const:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    xorl %edx, %edx
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    movl $-1, %ecx

 ; GENERIC-NEXT:    negb %dil
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    orb $42, %al
 ; GENERIC-NEXT:    ## kill: def $al killed $al killed $eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: nezero_all_ones_or_const:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negb %dil
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    orb $42, %al
 ; ATOM-NEXT:    ## kill: def $al killed $al killed $eax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: nezero_all_ones_or_const:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    cmpb {{[0-9]+}}(%esp), %al
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    orb $42, %al

 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    negl %edi
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    orl $1, %eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: PR53006:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negl %edi
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    orl $1, %eax
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: PR53006:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    orl $1, %eax

 ; GENERIC-LABEL: test13:
 ; GENERIC:       ## %bb.0:
 ; GENERIC-NEXT:    cmpl %esi, %edi
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test13:
 ; ATOM:       ## %bb.0:
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    cmpl %esi, %edi
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    nop
 ; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test13:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    retl

 ; GENERIC-LABEL: test15:
 ; GENERIC:       ## %bb.0: ## %entry
 ; GENERIC-NEXT:    negl %edi
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test15:
 ; ATOM:       ## %bb.0: ## %entry
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negl %edi
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    nop
 ; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test15:
 ; ATHLON:       ## %bb.0: ## %entry
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    retl

 ; GENERIC-LABEL: test16:
 ; GENERIC:       ## %bb.0: ## %entry
 ; GENERIC-NEXT:    negq %rdi
 ; GENERIC-NEXT:    sbbq %rax, %rax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test16:
 ; ATOM:       ## %bb.0: ## %entry
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negq %rdi
 ; ATOM-NEXT:    sbbq %rax, %rax
 ; ATOM-NEXT:    nop
 ; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test16:
 ; ATHLON:       ## %bb.0: ## %entry
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %ecx
 ; ATHLON-NEXT:    setne %al

 ; GENERIC:       ## %bb.0: ## %entry
 ; GENERIC-NEXT:    negw %di
 ; GENERIC-NEXT:    sbbl %eax, %eax
 ; GENERIC-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; GENERIC-NEXT:    retq
 ;
 ; ATOM-LABEL: test17:
 ; ATOM:       ## %bb.0: ## %entry
+; ATOM-NEXT:    xorl %eax, %eax
 ; ATOM-NEXT:    negw %di
 ; ATOM-NEXT:    sbbl %eax, %eax
 ; ATOM-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; ATOM-NEXT:    nop
 ; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
-; ATOM-NEXT:    nop
 ; ATOM-NEXT:    retq
 ;
 ; ATHLON-LABEL: test17:
 ; ATHLON:       ## %bb.0: ## %entry
 ; ATHLON-NEXT:    xorl %eax, %eax
 ; ATHLON-NEXT:    cmpw {{[0-9]+}}(%esp), %ax
 ; ATHLON-NEXT:    sbbl %eax, %eax
 ; ATHLON-NEXT:    ## kill: def $ax killed $ax killed $eax