X86: Don't emit SAHF/LAHF for 64-bit targets unless explicitly supported

[hans]

Summary

These instructions are not supported by all CPUs in 64-bit mode. Emitting them causes Chromium to crash on start-up for users with those chips.

(GCC puts this behind -msahf on 64-bit for the same reason.)

This patch adds FeatureSAHF, enables it by default for 32-bit targets and modern CPUs, and changes X86InstrInfo::copyPhysReg back to the lowering from before r244503 when it's not available.

I'm not familiar with all the x86 CPU models, but I believe the ones I updated are all modern enough.

Please take a look.

[jfb]

Looks like we raced: D15239!

[majnemer]

lib/Target/X86/X86Subtarget.cpp

192–202 ↗

(On Diff #41901)

Looks like some debugging code was left around?

[jfb]

I haven't looked through the list of CPU support, or tests yet.

lib/Target/X86/X86.td

186 ↗

(On Diff #41901)

I like my naming of HasLAHFSAHF since it's more comprehensive.

lib/Target/X86/X86ISelLowering.cpp

13935 ↗

(On Diff #41901)

This should be fixed before committing?

lib/Target/X86/X86InstrInfo.cpp

4468 ↗

(On Diff #41901)

I like my format for lib/Target/X86/X86InstrInfo.cpp in D15239 better ☺

lib/Target/X86/X86Subtarget.cpp

200 ↗

(On Diff #41901)

What name does GCC use?

[majnemer]

lib/Target/X86/X86Subtarget.cpp

200 ↗

(On Diff #41901)

https://gcc.gnu.org/onlinedocs/gcc-4.5.3/gcc/i386-and-x86_002d64-Options.html says -msahf

[hans]

Thanks for the quick review!

lib/Target/X86/X86.td

186 ↗

(On Diff #41901)

Renamed.

I would like to keep the "sahf" name though, to match gcc's -msahf flag.

lib/Target/X86/X86ISelLowering.cpp

13935 ↗

(On Diff #41901)

No, there is nothing to be fixed. 64-bit targets all have FCOMI, so we shouldn't be emitting SAHF for 64-bit here, and the assert shouldn't fire.

lib/Target/X86/X86InstrInfo.cpp

4468 ↗

(On Diff #41901)

It does look nicer :-)

lib/Target/X86/X86Subtarget.cpp

192–202 ↗

(On Diff #41901)

Oops. That should teach me to rush things out before lunch :-)

[hans]

Addressing comments.

[jfb]

Looks good overall, though I haven't validated the list of supported CPUs. How did you create it?

[hans]

In D15240#302911, @jfb wrote:

Looks good overall, though I haven't validated the list of supported CPUs. How did you create it?

I added the feature to all the architectures which seemed (by looking at Wikipedia) significantly newer than 2005. It's probably not an exhaustive list, but I believe it's conservatively correct.

[jfb]

Could you rebase the change? You're missing my recent patch which slightly conflicts.

[hans]

Rebase.

[jfb]

Sorry, a few more comments. lgtm otherwise.

lib/Target/X86/X86.td

564 ↗

(On Diff #41939)

Looks like the above two should also have lahf/sahf.

lib/Target/X86/X86InstrInfo.cpp

4398 ↗

(On Diff #41939)

IIUC you can assert(is64); here?

[hans]

[jfb]

llvm/trunk/lib/Target/X86/X86.td

561

I think this one could have it as well?

[hans]

llvm/trunk/lib/Target/X86/X86.td

561

Sorry, I missed the "above *two*" part of your previous comment. Added in r254801.

[hoffbrinkle]

@jfb, this brings back the issue we solved with http://reviews.llvm.org/D6629. We can't issue PUSHFQ/POPFQ as the POPFQ can corrupt EFLAGS by loading a stale value.

[jfb]

In D15240#331998, @hoffbrinkle wrote:

@jfb, this brings back the issue we solved with http://reviews.llvm.org/D6629. We can't issue PUSHFQ/POPFQ as the POPFQ can corrupt EFLAGS by loading a stale value.

Agreed, you'll need to compile with -mattr=+sahf to avoid that issue.

We could also teach LLVM to be smarter about flags, but that's quite a bit more work that we've already poured into this!

[hoffbrinkle]

In D15240#332004, @jfb wrote:

Agreed, you'll need to compile with -mattr=+sahf to avoid that issue.

Well, right, but when not using that attribute, we shouldn't generate incorrect code. llvm shouldn't issue a POPFQ unless it knows that it's not blowing away an updated EFLAGS.

[jfb]

In D15240#332012, @hoffbrinkle wrote:

In D15240#332004, @jfb wrote:

Agreed, you'll need to compile with -mattr=+sahf to avoid that issue.

Well, right, but when not using that attribute, we shouldn't generate incorrect code. llvm shouldn't issue a POPFQ unless it knows that it's not blowing away an updated EFLAGS.

Agreed. We're playing a game of whack-a-bug and a significant number of programs expect to run on machines with lahf/sahf so those uses won out on programs which get sad when EFLAGS are clobbered. That was the pre-D6629 status-quo, but the world is slightly better overall. It'll take another whack (and a big one at that) to fix the EFLAGS issue.

llvm/trunk/lib/Target/X86/X86.td

def FeatureADX     : SubtargetFeature<"adx", "HasADX", "true",
                                       "Support ADX instructions">;
 def FeatureSHA     : SubtargetFeature<"sha", "HasSHA", "true",
                                       "Enable SHA instructions",
                                       [FeatureSSE2]>;
 def FeaturePRFCHW  : SubtargetFeature<"prfchw", "HasPRFCHW", "true",
                                       "Support PRFCHW instructions">;
 def FeatureRDSEED  : SubtargetFeature<"rdseed", "HasRDSEED", "true",
                                       "Support RDSEED instruction">;
+def FeatureLAHFSAHF : SubtargetFeature<"sahf", "HasLAHFSAHF", "true",
+                                       "Support LAHF and SAHF instructions">;
 def FeatureMPX     : SubtargetFeature<"mpx", "HasMPX", "true",
                                       "Support MPX instructions">;
 def FeatureLEAForSP : SubtargetFeature<"lea-sp", "UseLeaForSP", "true",
                                      "Use LEA for adjusting the stack pointer">;
 def FeatureSlowDivide32 : SubtargetFeature<"idivl-to-divb",
                                      "HasSlowDivide32", "true",
                                      "Use 8-bit divide for positive values less than 256">;
 def FeatureSlowDivide64 : SubtargetFeature<"idivq-to-divw",

 
 // Intel Core 2 Solo/Duo.
 def : ProcessorModel<"core2", SandyBridgeModel, [
   FeatureSlowUAMem16,
   FeatureMMX,
   FeatureSSSE3,
   FeatureFXSR,
   FeatureCMPXCHG16B,
-  FeatureSlowBTMem
+  FeatureSlowBTMem,
+  FeatureLAHFSAHF
 ]>;
 def : ProcessorModel<"penryn", SandyBridgeModel, [
   FeatureSlowUAMem16,
   FeatureMMX,
   FeatureSSE41,
   FeatureFXSR,
   FeatureCMPXCHG16B,
-  FeatureSlowBTMem
+  FeatureSlowBTMem,
+  FeatureLAHFSAHF
 ]>;
 
 // Atom CPUs.
 class BonnellProc<string Name> : ProcessorModel<Name, AtomModel, [
   ProcIntelAtom,
   FeatureSlowUAMem16,
   FeatureMMX,
   FeatureSSSE3,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureMOVBE,
   FeatureSlowBTMem,
   FeatureLEAForSP,
   FeatureSlowDivide32,
   FeatureSlowDivide64,
   FeatureCallRegIndirect,
   FeatureLEAUsesAG,
-  FeaturePadShortFunctions
+  FeaturePadShortFunctions,
+  FeatureLAHFSAHF
 ]>;
 def : BonnellProc<"bonnell">;
 def : BonnellProc<"atom">; // Pin the generic name to the baseline.
 
 class SilvermontProc<string Name> : ProcessorModel<Name, SLMModel, [
   ProcIntelSLM,
   FeatureMMX,
   FeatureSSE42,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureMOVBE,
   FeaturePOPCNT,
   FeaturePCLMUL,
   FeatureAES,
   FeatureSlowDivide64,
   FeatureCallRegIndirect,
   FeaturePRFCHW,
   FeatureSlowLEA,
   FeatureSlowIncDec,
-  FeatureSlowBTMem
+  FeatureSlowBTMem,
+  FeatureLAHFSAHF
 ]>;
 def : SilvermontProc<"silvermont">;
 def : SilvermontProc<"slm">; // Legacy alias.
 
 // "Arrandale" along with corei3 and corei5
 class NehalemProc<string Name> : ProcessorModel<Name, SandyBridgeModel, [
   FeatureMMX,
   FeatureSSE42,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureSlowBTMem,
-  FeaturePOPCNT
+  FeaturePOPCNT,
+  FeatureLAHFSAHF
 ]>;
 def : NehalemProc<"nehalem">;
 def : NehalemProc<"corei7">;
 
 // Westmere is a similar machine to nehalem with some additional features.
 // Westmere is the corei3/i5/i7 path from nehalem to sandybridge
 class WestmereProc<string Name> : ProcessorModel<Name, SandyBridgeModel, [
   FeatureMMX,
   FeatureSSE42,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureSlowBTMem,
   FeaturePOPCNT,
   FeatureAES,
-  FeaturePCLMUL
+  FeaturePCLMUL,
+  FeatureLAHFSAHF
 ]>;
 def : WestmereProc<"westmere">;
 
 // SSE is not listed here since llvm treats AVX as a reimplementation of SSE,
 // rather than a superset.
 class SandyBridgeProc<string Name> : ProcessorModel<Name, SandyBridgeModel, [
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureSlowBTMem,
   FeatureSlowUAMem32,
   FeaturePOPCNT,
   FeatureAES,
   FeaturePCLMUL,
   FeatureXSAVE,
-  FeatureXSAVEOPT
+  FeatureXSAVEOPT,
+  FeatureLAHFSAHF
 ]>;
 def : SandyBridgeProc<"sandybridge">;
 def : SandyBridgeProc<"corei7-avx">; // Legacy alias.
 
 class IvyBridgeProc<string Name> : ProcessorModel<Name, SandyBridgeModel, [
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeatureSlowBTMem,
   FeatureSlowUAMem32,
   FeaturePOPCNT,
   FeatureAES,
   FeaturePCLMUL,
   FeatureXSAVE,
   FeatureXSAVEOPT,
   FeatureRDRAND,
   FeatureF16C,
-  FeatureFSGSBase
+  FeatureFSGSBase,
+  FeatureLAHFSAHF
 ]>;
 def : IvyBridgeProc<"ivybridge">;
 def : IvyBridgeProc<"core-avx-i">; // Legacy alias.
 
 class HaswellProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureMMX,
   FeatureAVX2,
   FeatureFXSR,
class HaswellProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureFSGSBase,
   FeatureMOVBE,
   FeatureLZCNT,
   FeatureBMI,
   FeatureBMI2,
   FeatureFMA,
   FeatureRTM,
   FeatureHLE,
-  FeatureSlowIncDec
+  FeatureSlowIncDec,
+  FeatureLAHFSAHF
 ]>;
 def : HaswellProc<"haswell">;
 def : HaswellProc<"core-avx2">; // Legacy alias.
 
 class BroadwellProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureMMX,
   FeatureAVX2,
   FeatureFXSR,
class BroadwellProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureLZCNT,
   FeatureBMI,
   FeatureBMI2,
   FeatureFMA,
   FeatureRTM,
   FeatureHLE,
   FeatureADX,
   FeatureRDSEED,
-  FeatureSlowIncDec
+  FeatureSlowIncDec,
+  FeatureLAHFSAHF
 ]>;
 def : BroadwellProc<"broadwell">;
 
 // FIXME: define KNL model
 class KnightsLandingProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureMMX,
   FeatureAVX512,
   FeatureFXSR,
class KnightsLandingProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureMOVBE,
   FeatureLZCNT,
   FeatureBMI,
   FeatureBMI2,
   FeatureFMA,
   FeatureRTM,
   FeatureHLE,
   FeatureSlowIncDec,
-  FeatureMPX
+  FeatureMPX,
+  FeatureLAHFSAHF
 ]>;
 def : KnightsLandingProc<"knl">;
 
 // FIXME: define SKX model
 class SkylakeProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureMMX,
   FeatureAVX512,
   FeatureFXSR,
class SkylakeProc<string Name> : ProcessorModel<Name, HaswellModel, [
   FeatureFMA,
   FeatureRTM,
   FeatureHLE,
   FeatureADX,
   FeatureRDSEED,
   FeatureSlowIncDec,
   FeatureMPX,
   FeatureXSAVEC,
-  FeatureXSAVES
+  FeatureXSAVES,
+  FeatureLAHFSAHF
 ]>;
 def : SkylakeProc<"skylake">;
 def : SkylakeProc<"skx">; // Legacy alias.
 
 
 // AMD CPUs.
 
 def : Proc<"k6",              [FeatureSlowUAMem16, FeatureMMX]>;

 def : Proc<"opteron-sse3",    [FeatureSlowUAMem16, FeatureSSE3,   Feature3DNowA,
                                FeatureFXSR, FeatureCMPXCHG16B, FeatureSlowBTMem,
                                FeatureSlowSHLD]>;
 def : Proc<"athlon64-sse3",   [FeatureSlowUAMem16, FeatureSSE3,   Feature3DNowA,
                                FeatureFXSR, FeatureCMPXCHG16B, FeatureSlowBTMem,
                                FeatureSlowSHLD]>;
 def : Proc<"amdfam10",        [FeatureSSE4A, Feature3DNowA, FeatureFXSR,
                                FeatureCMPXCHG16B, FeatureLZCNT, FeaturePOPCNT,
                                FeatureSlowBTMem, FeatureSlowSHLD]>;
 def : Proc<"barcelona",       [FeatureSSE4A, Feature3DNowA, FeatureFXSR,
                                FeatureCMPXCHG16B, FeatureLZCNT, FeaturePOPCNT,
-                               FeatureSlowBTMem, FeatureSlowSHLD]>;
+                               FeatureSlowBTMem, FeatureSlowSHLD, FeatureLAHFSAHF]>;
 
 // Bobcat
 def : Proc<"btver1", [
   FeatureMMX,
   FeatureSSSE3,
   FeatureSSE4A,
   FeatureFXSR,
   FeatureCMPXCHG16B,
   FeaturePRFCHW,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
-  FeatureSlowSHLD
+  FeatureSlowSHLD,
+  FeatureLAHFSAHF
 ]>;
 
 // Jaguar
 def : ProcessorModel<"btver2", BtVer2Model, [
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureSSE4A,
   FeatureCMPXCHG16B,
   FeaturePRFCHW,
   FeatureAES,
   FeaturePCLMUL,
   FeatureBMI,
   FeatureF16C,
   FeatureMOVBE,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureXSAVEOPT,
-  FeatureSlowSHLD
+  FeatureSlowSHLD,
+  FeatureLAHFSAHF
 ]>;
 
 // Bulldozer
 def : Proc<"bdver1", [
   FeatureXOP,
   FeatureFMA4,
   FeatureCMPXCHG16B,
   FeatureAES,
   FeaturePRFCHW,
   FeaturePCLMUL,
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureSSE4A,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
-  FeatureSlowSHLD
+  FeatureSlowSHLD,
+  FeatureLAHFSAHF
 ]>;
 // Piledriver
 def : Proc<"bdver2", [
   FeatureXOP,
   FeatureFMA4,
   FeatureCMPXCHG16B,
   FeatureAES,
   FeaturePRFCHW,
   FeaturePCLMUL,
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureSSE4A,
   FeatureF16C,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureBMI,
   FeatureTBM,
   FeatureFMA,
-  FeatureSlowSHLD
+  FeatureSlowSHLD,
+  FeatureLAHFSAHF
 ]>;
 
 // Steamroller
 def : Proc<"bdver3", [
   FeatureXOP,
   FeatureFMA4,
   FeatureCMPXCHG16B,
   FeatureAES,
   FeaturePRFCHW,
   FeaturePCLMUL,
   FeatureMMX,
   FeatureAVX,
   FeatureFXSR,
   FeatureSSE4A,
   FeatureF16C,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureBMI,
   FeatureTBM,
   FeatureFMA,
   FeatureXSAVEOPT,
   FeatureSlowSHLD,
-  FeatureFSGSBase
+  FeatureFSGSBase,
+  FeatureLAHFSAHF
 ]>;
 
 // Excavator
 def : Proc<"bdver4", [
   FeatureMMX,
   FeatureAVX2,
   FeatureFXSR,
   FeatureXOP,
   FeatureFMA4,
   FeatureCMPXCHG16B,
   FeatureAES,
   FeaturePRFCHW,
   FeaturePCLMUL,
   FeatureF16C,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureBMI,
   FeatureBMI2,
   FeatureTBM,
   FeatureFMA,
   FeatureXSAVEOPT,
-  FeatureFSGSBase
+  FeatureFSGSBase,
+  FeatureLAHFSAHF
 ]>;
 
 def : Proc<"geode",           [FeatureSlowUAMem16, Feature3DNowA]>;
 
 def : Proc<"winchip-c6",      [FeatureSlowUAMem16, FeatureMMX]>;
 def : Proc<"winchip2",        [FeatureSlowUAMem16, Feature3DNow]>;
 def : Proc<"c3",              [FeatureSlowUAMem16, Feature3DNow]>;
 def : Proc<"c3-2", [FeatureSlowUAMem16, FeatureMMX, FeatureSSE1, FeatureFXSR]>;

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

SDValue X86TargetLowering::ConvertCmpIfNecessary(SDValue Cmp,
   // build an SDNode sequence that transfers the result from FPSW into EFLAGS:
   // (X86sahf (trunc (srl (X86fp_stsw (trunc (X86cmp ...)), 8))))
   SDLoc dl(Cmp);
   SDValue TruncFPSW = DAG.getNode(ISD::TRUNCATE, dl, MVT::i16, Cmp);
   SDValue FNStSW = DAG.getNode(X86ISD::FNSTSW16r, dl, MVT::i16, TruncFPSW);
   SDValue Srl = DAG.getNode(ISD::SRL, dl, MVT::i16, FNStSW,
                             DAG.getConstant(8, dl, MVT::i8));
   SDValue TruncSrl = DAG.getNode(ISD::TRUNCATE, dl, MVT::i8, Srl);
+
+  // Some 64-bit targets lack SAHF support, but they do support FCOMI.
+  assert(Subtarget->hasLAHFSAHF() && "Target doesn't support SAHF or FCOMI?");
   return DAG.getNode(X86ISD::SAHF, dl, MVT::i32, TruncSrl);
 }
 
 /// The minimum architected relative accuracy is 2^-12. We need one
 /// Newton-Raphson step to have a good float result (24 bits of precision).
 SDValue X86TargetLowering::getRsqrtEstimate(SDValue Op,
                                             DAGCombinerInfo &DCI,
                                             unsigned &RefinementSteps,

llvm/trunk/lib/Target/X86/X86InstrInfo.cpp

if (Opc) {
     return;
   }
 
   bool FromEFLAGS = SrcReg == X86::EFLAGS;
   bool ToEFLAGS = DestReg == X86::EFLAGS;
   int Reg = FromEFLAGS ? DestReg : SrcReg;
   bool is32 = X86::GR32RegClass.contains(Reg);
   bool is64 = X86::GR64RegClass.contains(Reg);
+
   if ((FromEFLAGS || ToEFLAGS) && (is32 || is64)) {
+    int Mov = is64 ? X86::MOV64rr : X86::MOV32rr;
+    int Push = is64 ? X86::PUSH64r : X86::PUSH32r;
+    int PushF = is64 ? X86::PUSHF64 : X86::PUSHF32;
+    int Pop = is64 ? X86::POP64r : X86::POP32r;
+    int PopF = is64 ? X86::POPF64 : X86::POPF32;
+    int AX = is64 ? X86::RAX : X86::EAX;
+
+    if (!Subtarget.hasLAHFSAHF()) {
+      assert(is64 && "Not having LAHF/SAHF only happens on 64-bit.");
+      // Moving EFLAGS to / from another register requires a push and a pop.
+      // Notice that we have to adjust the stack if we don't want to clobber the
+      // first frame index. See X86FrameLowering.cpp - clobbersTheStack.
+      if (FromEFLAGS) {
+        BuildMI(MBB, MI, DL, get(PushF));
+        BuildMI(MBB, MI, DL, get(Pop), DestReg);
+      }
+      if (ToEFLAGS) {
+        BuildMI(MBB, MI, DL, get(Push))
+            .addReg(SrcReg, getKillRegState(KillSrc));
+        BuildMI(MBB, MI, DL, get(PopF));
+      }
+      return;
+    }
+
     // The flags need to be saved, but saving EFLAGS with PUSHF/POPF is
     // inefficient. Instead:
     //   - Save the overflow flag OF into AL using SETO, and restore it using a
     //     signed 8-bit addition of AL and INT8_MAX.
     //   - Save/restore the bottom 8 EFLAGS bits (CF, PF, AF, ZF, SF) to/from AH
     //     using LAHF/SAHF.
     //   - When RAX/EAX is live and isn't the destination register, make sure it
     //     isn't clobbered by PUSH/POP'ing it before and after saving/restoring
     //     the flags.
     // This approach is ~2.25x faster than using PUSHF/POPF.
     //
     // This is still somewhat inefficient because we don't know which flags are
     // actually live inside EFLAGS. Were we able to do a single SETcc instead of
     // SETO+LAHF / ADDB+SAHF the code could be 1.02x faster.
     //
     // PUSHF/POPF is also potentially incorrect because it affects other flags
     // such as TF/IF/DF, which LLVM doesn't model.
     //
     // Notice that we have to adjust the stack if we don't want to clobber the
     // first frame index. See X86FrameLowering.cpp - clobbersTheStack.
 
-    int Mov = is64 ? X86::MOV64rr : X86::MOV32rr;
-    int Push = is64 ? X86::PUSH64r : X86::PUSH32r;
-    int Pop = is64 ? X86::POP64r : X86::POP32r;
-    int AX = is64 ? X86::RAX : X86::EAX;
 
     bool AXDead = (Reg == AX);
     // FIXME: The above could figure out that AX is dead in more cases with:
     //          || (MachineBasicBlock::LQR_Dead ==
     //            MBB.computeRegisterLiveness(&getRegisterInfo(), AX, MI));
     //
     //        Unfortunately this is slightly broken, see PR24535 and the likely
     //        related PR25033 PR24991 PR24992 PR25201. These issues seem to

llvm/trunk/lib/Target/X86/X86InstrInfo.td

 def HasRTM       : Predicate<"Subtarget->hasRTM()">;
 def HasHLE       : Predicate<"Subtarget->hasHLE()">;
 def HasTSX       : Predicate<"Subtarget->hasRTM() || Subtarget->hasHLE()">;
 def HasADX       : Predicate<"Subtarget->hasADX()">;
 def HasSHA       : Predicate<"Subtarget->hasSHA()">;
 def HasPRFCHW    : Predicate<"Subtarget->hasPRFCHW()">;
 def HasRDSEED    : Predicate<"Subtarget->hasRDSEED()">;
 def HasPrefetchW : Predicate<"Subtarget->hasPRFCHW()">;
+def HasLAHFSAHF  : Predicate<"Subtarget->hasLAHFSAHF()">;
 def FPStackf32   : Predicate<"!Subtarget->hasSSE1()">;
 def FPStackf64   : Predicate<"!Subtarget->hasSSE2()">;
 def HasMPX       : Predicate<"Subtarget->hasMPX()">;
 def HasCmpxchg16b: Predicate<"Subtarget->hasCmpxchg16b()">;
 def Not64BitMode : Predicate<"!Subtarget->is64Bit()">,
                              AssemblerPredicate<"!Mode64Bit", "Not 64-bit mode">;
 def In64BitMode  : Predicate<"Subtarget->is64Bit()">,
                              AssemblerPredicate<"Mode64Bit", "64-bit mode">;
def MOV8rm_NOREX : I<0x8A, MRMSrcMem,
                      IIC_MOV_MEM>, Sched<[WriteLoad]>;
 }
 
 
 // Condition code ops, incl. set if equal/not equal/...
 let SchedRW = [WriteALU] in {
 let Defs = [EFLAGS], Uses = [AH] in
 def SAHF     : I<0x9E, RawFrm, (outs),  (ins), "sahf",
-                 [(set EFLAGS, (X86sahf AH))], IIC_AHF>;
+                 [(set EFLAGS, (X86sahf AH))], IIC_AHF>,
+               Requires<[HasLAHFSAHF]>;
 let Defs = [AH], Uses = [EFLAGS], hasSideEffects = 0 in
 def LAHF     : I<0x9F, RawFrm, (outs),  (ins), "lahf", [],
-                IIC_AHF>;  // AH = flags
+                IIC_AHF>,  // AH = flags
+               Requires<[HasLAHFSAHF]>;
 } // SchedRW
 
 //===----------------------------------------------------------------------===//
 // Bit tests instructions: BT, BTS, BTR, BTC.
 
 let Defs = [EFLAGS] in {
 let SchedRW = [WriteALU] in {
 def BT16rr : I<0xA3, MRMDestReg, (outs), (ins GR16:$src1, GR16:$src2),

llvm/trunk/lib/Target/X86/X86Subtarget.h

protected:
   bool HasSHA;
 
   /// Processor has PRFCHW instructions.
   bool HasPRFCHW;
 
   /// Processor has RDSEED instructions.
   bool HasRDSEED;
 
+  /// Processor has LAHF/SAHF instructions.
+  bool HasLAHFSAHF;
+
   /// True if BT (bit test) of memory instructions are slow.
   bool IsBTMemSlow;
 
   /// True if SHLD instructions are slow.
   bool IsSHLDSlow;
 
   /// True if unaligned memory accesses of 16-bytes are slow.
   bool IsUAMem16Slow;
public:
   bool hasBMI() const { return HasBMI; }
   bool hasBMI2() const { return HasBMI2; }
   bool hasRTM() const { return HasRTM; }
   bool hasHLE() const { return HasHLE; }
   bool hasADX() const { return HasADX; }
   bool hasSHA() const { return HasSHA; }
   bool hasPRFCHW() const { return HasPRFCHW; }
   bool hasRDSEED() const { return HasRDSEED; }
+  bool hasLAHFSAHF() const { return HasLAHFSAHF; }
   bool isBTMemSlow() const { return IsBTMemSlow; }
   bool isSHLDSlow() const { return IsSHLDSlow; }
   bool isUnalignedMem16Slow() const { return IsUAMem16Slow; }
   bool isUnalignedMem32Slow() const { return IsUAMem32Slow; }
   bool hasSSEUnalignedMem() const { return HasSSEUnalignedMem; }
   bool hasCmpxchg16b() const { return HasCmpxchg16b; }
   bool useLeaForSP() const { return UseLeaForSP; }
   bool hasSlowDivide32() const { return HasSlowDivide32; }

llvm/trunk/lib/Target/X86/X86Subtarget.cpp

void X86Subtarget::initSubtargetFeatures(StringRef CPU, StringRef FS) {
   std::string FullFS = FS;
   if (In64BitMode) {
     if (!FullFS.empty())
       FullFS = "+64bit,+sse2," + FullFS;
     else
       FullFS = "+64bit,+sse2";
   }
 
+  // LAHF/SAHF are always supported in non-64-bit mode.
+  if (!In64BitMode) {
+    if (!FullFS.empty())
+      FullFS = "+sahf," + FullFS;
+    else
+      FullFS = "+sahf";
+  }
+
+
   // Parse features string and set the CPU.
   ParseSubtargetFeatures(CPUName, FullFS);
 
   // All CPUs that implement SSE4.2 or SSE4A support unaligned accesses of
   // 16-bytes and under that are reasonably fast. These features were
   // introduced with Intel's Nehalem/Silvermont and AMD's Family10h
   // micro-architectures respectively.
   if (hasSSE42() || hasSSE4A())
void X86Subtarget::initializeEnvironment() {
   HasPFI = false;
   HasDQI = false;
   HasBWI = false;
   HasVLX = false;
   HasADX = false;
   HasSHA = false;
   HasPRFCHW = false;
   HasRDSEED = false;
+  HasLAHFSAHF = false;
   HasMPX = false;
   IsBTMemSlow = false;
   IsSHLDSlow = false;
   IsUAMem16Slow = false;
   IsUAMem32Slow = false;
   HasSSEUnalignedMem = false;
   HasCmpxchg16b = false;
   UseLeaForSP = false;

llvm/trunk/test/CodeGen/X86/cmpxchg-clobber-flags.ll

 ; RUN: llc -mtriple=i386-linux-gnu %s -o - | FileCheck %s -check-prefix=i386
 ; RUN: llc -mtriple=i386-linux-gnu -pre-RA-sched=fast %s -o - | FileCheck %s -check-prefix=i386f
+
 ; RUN: llc -mtriple=x86_64-linux-gnu %s -o - | FileCheck %s -check-prefix=x8664
 ; RUN: llc -mtriple=x86_64-linux-gnu -pre-RA-sched=fast %s -o - | FileCheck %s -check-prefix=x8664
+; RUN: llc -mtriple=x86_64-linux-gnu -mattr=+sahf %s -o - | FileCheck %s -check-prefix=x8664-sahf
+; RUN: llc -mtriple=x86_64-linux-gnu -mattr=+sahf -pre-RA-sched=fast %s -o - | FileCheck %s -check-prefix=x8664-sahf
+; RUN: llc -mtriple=x86_64-linux-gnu -mcpu=corei7 %s -o - | FileCheck %s -check-prefix=x8664-sahf
 
 ; FIXME: X86InstrInfo::copyPhysReg had code which figured out whether AX was
 ;        live or not to avoid save / restore when it's not needed. See FIXME in
 ;        that function for more details on which the code is currently
 ;        disabled. The extra push/pop are marked below and can be removed once
 ;        the issue is fixed.
 ;        -verify-machineinstrs should also be added back in the RUN lines above.
 

 ; i386f-NEXT: addb $127, %al
 ; i386f-NEXT: sahf
 ; ** FIXME Next line isn't actually necessary. **
 ; i386f-NEXT: popl %eax
 ; i386f-NEXT: jne
 
 ; x8664-LABEL: test_intervening_call:
 ; x8664: cmpxchgq
-; x8664: pushq %rax
-; x8664-NEXT: seto %al
-; x8664-NEXT: lahf
-; x8664-NEXT: movq %rax, [[FLAGS:%.*]]
-; x8664-NEXT: popq %rax
+; x8664: pushfq
+; x8664-NEXT: popq [[FLAGS:%.*]]
 ; x8664-NEXT: movq %rax, %rdi
 ; x8664-NEXT: callq bar
-; ** FIXME Next line isn't actually necessary. **
-; x8664-NEXT: pushq %rax
-; x8664-NEXT: movq [[FLAGS]], %rax
-; x8664-NEXT: addb $127, %al
-; x8664-NEXT: sahf
-; ** FIXME Next line isn't actually necessary. **
-; x8664-NEXT: popq %rax
+; x8664-NEXT: pushq [[FLAGS]]
+; x8664-NEXT: popfq
 ; x8664-NEXT: jne
 
+; x8664-sahf-LABEL: test_intervening_call:
+; x8664-sahf: cmpxchgq
+; x8664-sahf: pushq %rax
+; x8664-sahf-NEXT: seto %al
+; x8664-sahf-NEXT: lahf
+; x8664-sahf-NEXT: movq %rax, [[FLAGS:%.*]]
+; x8664-sahf-NEXT: popq %rax
+; x8664-sahf-NEXT: movq %rax, %rdi
+; x8664-sahf-NEXT: callq bar
+; ** FIXME Next line isn't actually necessary. **
+; x8664-sahf-NEXT: pushq %rax
+; x8664-sahf-NEXT: movq [[FLAGS]], %rax
+; x8664-sahf-NEXT: addb $127, %al
+; x8664-sahf-NEXT: sahf
+; ** FIXME Next line isn't actually necessary. **
+; x8664-sahf-NEXT: popq %rax
+; x8664-sahf-NEXT: jne
+
   %cx = cmpxchg i64* %foo, i64 %bar, i64 %baz seq_cst seq_cst
   %v = extractvalue { i64, i1 } %cx, 0
   %p = extractvalue { i64, i1 } %cx, 1
   call i32 @bar(i64 %v)
   br i1 %p, label %t, label %f
 
 t:
   ret i64 42

 ; i386f-LABEL: test_control_flow:
 ; i386f: cmpxchg
 ; i386f-NEXT: jne
 
 ; x8664-LABEL: test_control_flow:
 ; x8664: cmpxchg
 ; x8664-NEXT: jne
 
+; x8664-sahf-LABEL: test_control_flow:
+; x8664-sahf: cmpxchg
+; x8664-sahf-NEXT: jne
+
 entry:
   %cmp = icmp sgt i32 %i, %j
   br i1 %cmp, label %loop_start, label %cond.end
 
 loop_start:
   br label %while.condthread-pre-split.i
 
 while.condthread-pre-split.i:

 ; i386f-NEXT: calll foo
 ; i386f-NEXT: pushl %eax
 ; i386f-NEXT: movl [[FLAGS]], %eax
 ; i386f-NEXT: addb $127, %al
 ; i386f-NEXT: sahf
 ; i386f-NEXT: popl %eax
 
 ; x8664-LABEL: test_feed_cmov:
-; x8664: cmpxchgl
-; ** FIXME Next line isn't actually necessary. **
-; x8664: pushq %rax
-; x8664: seto %al
-; x8664-NEXT: lahf
-; x8664-NEXT: movq %rax, [[FLAGS:%.*]]
-; ** FIXME Next line isn't actually necessary. **
-; x8664-NEXT: popq %rax
+; x8664: cmpxchg
+; x8664: pushfq
+; x8664-NEXT: popq [[FLAGS:%.*]]
 ; x8664-NEXT: callq foo
-; x8664-NEXT: pushq %rax
-; x8664-NEXT: movq [[FLAGS]], %rax
-; x8664-NEXT: addb $127, %al
-; x8664-NEXT: sahf
-; x8664-NEXT: popq %rax
+; x8664-NEXT: pushq [[FLAGS]]
+; x8664-NEXT: popfq
+
+; x8664-sahf-LABEL: test_feed_cmov:
+; x8664-sahf: cmpxchgl
+; ** FIXME Next line isn't actually necessary. **
+; x8664-sahf: pushq %rax
+; x8664-sahf: seto %al
+; x8664-sahf-NEXT: lahf
+; x8664-sahf-NEXT: movq %rax, [[FLAGS:%.*]]
+; ** FIXME Next line isn't actually necessary. **
+; x8664-sahf-NEXT: popq %rax
+; x8664-sahf-NEXT: callq foo
+; x8664-sahf-NEXT: pushq %rax
+; x8664-sahf-NEXT: movq [[FLAGS]], %rax
+; x8664-sahf-NEXT: addb $127, %al
+; x8664-sahf-NEXT: sahf
+; x8664-sahf-NEXT: popq %rax
 
   %res = cmpxchg i32* %addr, i32 %desired, i32 %new seq_cst seq_cst
   %success = extractvalue { i32, i1 } %res, 1
 
   %rhs = call i32 @foo()
 
   %ret = select i1 %success, i32 %new, i32 %rhs
   ret i32 %ret
 }

llvm/trunk/test/CodeGen/X86/peephole-na-phys-copy-folding.ll

 ; RUN: llc -mtriple=i386-linux-gnu %s -o - | FileCheck %s
-; RUN: llc -mtriple=x86_64-linux-gnu %s -o - | FileCheck %s
+; RUN: llc -mtriple=x86_64-linux-gnu -mattr=+sahf %s -o - | FileCheck %s
 
 ; FIXME Add -verify-machineinstrs back when PR24535 is fixed.
 
 ; The peephole optimizer can elide some physical register copies such as
 ; EFLAGS. Make sure the flags are used directly, instead of needlessly using
 ; lahf, when possible.
 
 @L = external global i32