Disable the vzeroupper insertion pass on PS4

[ygao]

Summary

Hi,
This patch re-implements the work to disable the vzeroupper insertion pass
on PS4 based on review feedback from Hal and Sean.

I am not sure whether there are other processors that behave like Jaguar
when it comes to writing YMM registers.

[hfinkel]

LGTM.

[silvas]

LGTM.

[probinson]

As long as the consequence of running such code on a non-btver2 CPU is merely performance, not correctness.
I seem to remember that being a concern in the first attempt at turning off vzeroupper, years ago. Something about the consistency of behavior of code in a library, IIRC, when caller and callee were compiled for different CPUs and did not have the same concept of whether the upper parts had been zeroed. Sorry I don't remember the specifics better than that, and I certainly don't know enough about the microarchitectural details to say one way or the other.

[andreadb]

In D16837#343006, @probinson wrote:

As long as the consequence of running such code on a non-btver2 CPU is merely performance, not correctness.
I seem to remember that being a concern in the first attempt at turning off vzeroupper, years ago. Something about the consistency of behavior of code in a library, IIRC, when caller and callee were compiled for different CPUs and did not have the same concept of whether the upper parts had been zeroed. Sorry I don't remember the specifics better than that, and I certainly don't know enough about the microarchitectural details to say one way or the other.

My understanding is that this should only affect performance.

The problem is when you mix legacy SSE instructions with AVX instructions. Legacy SSE instructions do not affect the upper 128-bits of the YMM registers. This may cause false dependencies due to partial register writes.

So, if a library is built for a non AVX CPU (or if the library cannot avoid using legacy SSE code), the absence of vzeroupper in the code has the potential of causing stalls due to false dependencies (when there is a AVX-SSE transition).

On AMD Fam 15h processors (and Btver2) there is no penalty for AVX-SSE transitions. This is an important difference with respect to Intel processors where, for each SSE-AVX transition, the hardware saves and restores the upper 128 bits of the YMM registers. I think that is the reason why on Intel, vzeroupper is very fast, while on btver2 vzeroupper is microcoded (and extremely slow!).
Also, (since Fam 15) AMD processors implement an XMM register merge optimization; the hardware keeps track of XMM registers whose upper portions have been cleared to zeros.

[ygao]

I definitely remember there was some concern (or incident?) over correctness,
and it involves some library. Unfortunately I cannot recall the details.

In this patch I was setting the feature bit on btver2, but it probably also
applies to bdver[2..4].

lib/Target/X86/X86.td

def FeatureLEAUsesAG : SubtargetFeature<"lea-uses-ag", "LEAUsesAG", "true",
                                    "LEA instruction needs inputs at AG stage">;
 def FeatureSlowLEA : SubtargetFeature<"slow-lea", "SlowLEA", "true",
                                    "LEA instruction with certain arguments is slow">;
 def FeatureSlowIncDec : SubtargetFeature<"slow-incdec", "SlowIncDec", "true",
                                    "INC and DEC instructions are slower than ADD and SUB">;
 def FeatureSoftFloat
     : SubtargetFeature<"soft-float", "UseSoftFloat", "true",
                        "Use software floating point features.">;
+// On at least some AMD processors, there is no performance hazard to writing
+// only the lower parts of a YMM register without clearing the upper part.
+def FeatureFastPartialYMMWrite
+    : SubtargetFeature<"fast-partial-ymm-write", "HasFastPartialYMMWrite",
+                       "true", "Partial writes to YMM registers are fast">;
 
 //===----------------------------------------------------------------------===//
 // X86 processors supported.
 //===----------------------------------------------------------------------===//
 
 include "X86Schedule.td"
 
 def ProcIntelAtom : SubtargetFeature<"atom", "X86ProcFamily", "IntelAtom",
def : ProcessorModel<"btver2", BtVer2Model, [
   FeatureBMI,
   FeatureF16C,
   FeatureMOVBE,
   FeatureLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureXSAVEOPT,
   FeatureSlowSHLD,
-  FeatureLAHFSAHF
+  FeatureLAHFSAHF,
+  FeatureFastPartialYMMWrite
 ]>;
 
 // Bulldozer
 def : Proc<"bdver1", [
   FeatureXOP,
   FeatureFMA4,
   FeatureCMPXCHG16B,
   FeatureAES,

lib/Target/X86/X86Subtarget.h

protected:
   /// True if this processor has the CMPXCHG16B instruction;
   /// this is true for most x86-64 chips, but not the first AMD chips.
   bool HasCmpxchg16b;
 
   /// True if the LEA instruction should be used for adjusting
   /// the stack pointer. This is an optimization for Intel Atom processors.
   bool UseLeaForSP;
 
+  /// True if there is no performance penalty to writing only the lower parts
+  /// of a YMM register without clearing the upper part.
+  bool HasFastPartialYMMWrite;
+
   /// True if 8-bit divisions are significantly faster than
   /// 32-bit divisions and should be used when possible.
   bool HasSlowDivide32;
 
   /// True if 16-bit divides are significantly faster than
   /// 64-bit divisions and should be used when possible.
   bool HasSlowDivide64;
 
public:
   bool hasLAHFSAHF() const { return HasLAHFSAHF; }
   bool isBTMemSlow() const { return IsBTMemSlow; }
   bool isSHLDSlow() const { return IsSHLDSlow; }
   bool isUnalignedMem16Slow() const { return IsUAMem16Slow; }
   bool isUnalignedMem32Slow() const { return IsUAMem32Slow; }
   bool hasSSEUnalignedMem() const { return HasSSEUnalignedMem; }
   bool hasCmpxchg16b() const { return HasCmpxchg16b; }
   bool useLeaForSP() const { return UseLeaForSP; }
+  bool hasFastPartialYMMWrite() const { return HasFastPartialYMMWrite; }
   bool hasSlowDivide32() const { return HasSlowDivide32; }
   bool hasSlowDivide64() const { return HasSlowDivide64; }
   bool padShortFunctions() const { return PadShortFunctions; }
   bool callRegIndirect() const { return CallRegIndirect; }
   bool LEAusesAG() const { return LEAUsesAG; }
   bool slowLEA() const { return SlowLEA; }
   bool slowIncDec() const { return SlowIncDec; }
   bool hasCDI() const { return HasCDI; }

lib/Target/X86/X86Subtarget.cpp

void X86Subtarget::initializeEnvironment() {
   HasMPX = false;
   IsBTMemSlow = false;
   IsSHLDSlow = false;
   IsUAMem16Slow = false;
   IsUAMem32Slow = false;
   HasSSEUnalignedMem = false;
   HasCmpxchg16b = false;
   UseLeaForSP = false;
+  HasFastPartialYMMWrite = false;
   HasSlowDivide32 = false;
   HasSlowDivide64 = false;
   PadShortFunctions = false;
   CallRegIndirect = false;
   LEAUsesAG = false;
   SlowLEA = false;
   SlowIncDec = false;
   stackAlignment = 4;

lib/Target/X86/X86VZeroUpper.cpp

void VZeroUpperInserter::processBasicBlock(MachineBasicBlock &MBB) {
 
   BlockStates[MBB.getNumber()].ExitState = CurState;
 }
 
 /// runOnMachineFunction - Loop over all of the basic blocks, inserting
 /// vzeroupper instructions before function calls.
 bool VZeroUpperInserter::runOnMachineFunction(MachineFunction &MF) {
   const X86Subtarget &ST = MF.getSubtarget<X86Subtarget>();
-  if (!ST.hasAVX() || ST.hasAVX512())
+  if (!ST.hasAVX() || ST.hasAVX512() || ST.hasFastPartialYMMWrite())
     return false;
   TII = ST.getInstrInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   EverMadeChange = false;
 
   bool FnHasLiveInYmm = checkFnHasLiveInYmm(MRI);
 
   // Fast check: if the function doesn't use any ymm registers, we don't need

test/CodeGen/X86/avx-vzeroupper.ll

 ; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-apple-darwin -mattr=+avx | FileCheck %s
+; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-apple-darwin -mattr=+avx,+fast-partial-ymm-write | FileCheck --check-prefix=FASTYMM %s
+; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-apple-darwin -mcpu=btver2 | FileCheck --check-prefix=BTVER2 %s
+
+; FASTYMM-NOT: vzeroupper
+; BTVER2-NOT: vzeroupper
 
 declare i32 @foo()
 declare <4 x float> @do_sse(<4 x float>)
 declare <8 x float> @do_avx(<8 x float>)
 declare <4 x float> @llvm.x86.avx.vextractf128.ps.256(<8 x float>, i8) nounwind readnone
 @x = common global <4 x float> zeroinitializer, align 16
 @g = common global <8 x float> zeroinitializer, align 32