[X86] NFC: Convert LLVM command-line flag to target attribute

[davezarzycki]

Summary

Swift and perhaps other users of LLVM do not leave cl::opt style command-line option parsing enabled due to C++ global constructor overhead, therefore let's convert the VZEROUPPER option to a target attribute.

Furthermore, this change stops the conflation of VZEROUPPER emission with enabling/disabling fast partial YMM/ZMM updates. The compiler doesn't intentionally emit partial updates at the moment, but it could.

This patch also verifies that clang can pass arbitrary target features.

[RKSimon]

Add a comment to the ReleaseNotes X86 section?

lib/Target/X86/X86.td

316

Just say "some X86 processors" like above.

[davezarzycki]

Addressed recent feedback.

[davezarzycki]

Ping. If it helps, I can move the remaining few uses of cl::opt by the x86 backend to code gen attributes for consistency.

[craig.topper]

Doesn’t this leave the FastPartialWriteFlag unused?

With this change there is no way to disable the pass from the clang command line.

[craig.topper]

A brief google search suggests Swift has a -Xllvm option to reach the llvm command line parser or did at one time.

[davezarzycki]

In D69500#1728752, @craig.topper wrote:

A brief google search suggests Swift has a -Xllvm option to reach the llvm command line parser or did at one time.

Here you go: https://forums.swift.org/t/micro-architectural-tuning/28269/3

Quote: "-Xllvm flags are compiled out of Release builds, since they impact startup time."

[davezarzycki]

In D69500#1728720, @craig.topper wrote:

Doesn’t this leave the FastPartialWriteFlag unused?

With this change there is no way to disable the pass from the clang command line.

Yes, this leaves FeatureFastPartialYMMorZMMWrite unused. I looked into wiring up some code gen to use the flag, but I got distracted.

Your right about the clang behavior. I'll need to update the patch.

[davezarzycki]

This update also verifies that clang can pass arbitrary target features.

[craig.topper]

I still don’t understand why we’re not just renaming the existing flag? What are you planning to connect the old flag to?

[craig.topper]

I think I’d like to see the existing feature renamed and leave the command line alone. I’m not sure -target-feature can be reached from the clang driver rather than cc1. I’ll work on adding -mno-vzeroupper to clang to match gcc and then we can remove the command line.

[davezarzycki]

In D69500#1730264, @craig.topper wrote:

I still don’t understand why we’re not just renaming the existing flag? What are you planning to connect the old flag to?

I didn't remove it because it seemed like information would be "lost". Said differently, I didn't know that some AMD CPUs have fast partial updates too.

I made a quick attempt at using the "fast partial" flag so that it wouldn't be left unused, but I got distracted. (For example using SSE move instructions instead of VINSERT*)

And yes, the -target-feature flag can be reached from the driver via -Xclang.

[arsenm]

Terminology needs fixing. This is adding a subtarget feature, not a target attribute which would be a different thing

[davezarzycki]

Closed in favor of D69786

docs/ReleaseNotes.rst

 * v32i8 and v64i8 vectors with AVX512F enabled, but AVX512BW disabled will now
   be passed in ZMM registers for calls and returns. Previously they were passed
   in two YMM registers. Old behavior can be enabled by passing
   -x86-enable-old-knl-abi
 * -mprefer-vector-width=256 is now the default behavior skylake-avx512 and later
   Intel CPUs. This tries to limit the use of 512-bit registers which can cause a
   decrease in CPU frequency on these CPUs. This can be re-enabled by passing
   -mprefer-vector-width=512 to clang or passing -mattr=-prefer-256-bit to llc.
+* The hidden command line flag -x86-use-vzeroupper is now a target attribute:
+  omit-vzeroupper
 
 Changes to the AMDGPU Target
 -----------------------------
 
 Changes to the AVR Target
 -----------------------------
 
  During this release ...

lib/Target/X86/X86.td

: SubtargetFeature<"fast-variable-shuffle",
                        "HasFastVariableShuffle",
                        "true", "Shuffles with variable masks are fast">;
 // On some X86 processors, there is no performance hazard to writing only the
 // lower parts of a YMM or ZMM register without clearing the upper part.
 def FeatureFastPartialYMMorZMMWrite
     : SubtargetFeature<"fast-partial-ymm-or-zmm-write",
                        "HasFastPartialYMMorZMMWrite",
                        "true", "Partial writes to YMM/ZMM registers are fast">;
+// Disable automatic VZEROUPPER emission.
+def FeatureOmitVZEROUPPER
+    : SubtargetFeature<"omit-vzeroupper", "OmitVZEROUPPER", "true",
+                       "Omit VZEROUPPER instruction emission">;
+
 // FeatureFastScalarFSQRT should be enabled if scalar FSQRT has shorter latency
 // than the corresponding NR code. FeatureFastVectorFSQRT should be enabled if
 // vector FSQRT has higher throughput than the corresponding NR code.
 // The idea is that throughput bound code is likely to be vectorized, so for
 // vectorized code we should care about the throughput of SQRT operations.
 // But if the code is scalar that probably means that the code has some kind of
 // dependency and we should care more about reducing the latency.
 def FeatureFastScalarFSQRT
list<SubtargetFeature> KNLFeatures = [FeatureX87,
                                         FeatureLZCNT,
                                         FeatureBMI,
                                         FeatureBMI2,
                                         FeatureFMA,
                                         FeaturePRFCHW,
                                         FeaturePreferMaskRegisters,
                                         FeatureSlowTwoMemOps,
                                         FeatureFastPartialYMMorZMMWrite,
+                                        FeatureOmitVZEROUPPER,
                                         FeatureHasFastGather,
                                         FeatureSlowPMADDWD];
   // TODO Add AVX5124FMAPS/AVX5124VNNIW features
   list<SubtargetFeature> KNMFeatures =
     !listconcat(KNLFeatures, [FeatureVPOPCNTDQ]);
 
   // Barcelona
   list<SubtargetFeature> BarcelonaInheritableFeatures = [FeatureX87,
list<SubtargetFeature> BtVer2AdditionalFeatures = [FeatureAVX,
                                                      FeatureBMI,
                                                      FeatureF16C,
                                                      FeatureMOVBE,
                                                      FeatureXSAVE,
                                                      FeatureXSAVEOPT];
   list<SubtargetFeature> BtVer2SpecificFeatures = [FeatureFastLZCNT,
                                                    FeatureFastBEXTR,
                                                    FeatureFastPartialYMMorZMMWrite,
+                                                   FeatureOmitVZEROUPPER,
                                                    FeatureFastHorizontalOps];
   list<SubtargetFeature> BtVer2InheritableFeatures =
     !listconcat(BtVer1InheritableFeatures, BtVer2AdditionalFeatures);
   list<SubtargetFeature> BtVer2Features =
     !listconcat(BtVer2InheritableFeatures, BtVer2SpecificFeatures);
 
   // Bulldozer
   list<SubtargetFeature> BdVer1InheritableFeatures = [FeatureX87,

lib/Target/X86/X86Subtarget.h

protected:
   /// True if its preferable to combine to a single shuffle using a variable
   /// mask over multiple fixed shuffles.
   bool HasFastVariableShuffle = false;
 
   /// True if there is no performance penalty to writing only the lower parts
   /// of a YMM or ZMM register without clearing the upper part.
   bool HasFastPartialYMMorZMMWrite = false;
 
+  /// True if the VZEROUPPER instruction should be omitted.
+  bool OmitVZEROUPPER = false;
+
   /// True if there is no performance penalty for writing NOPs with up to
   /// 11 bytes.
   bool HasFast11ByteNOP = false;
 
   /// True if there is no performance penalty for writing NOPs with up to
   /// 15 bytes.
   bool HasFast15ByteNOP = false;
 
public:
   bool hasPOPCNTFalseDeps() const { return HasPOPCNTFalseDeps; }
   bool hasLZCNTFalseDeps() const { return HasLZCNTFalseDeps; }
   bool hasFastVariableShuffle() const {
     return HasFastVariableShuffle;
   }
   bool hasFastPartialYMMorZMMWrite() const {
     return HasFastPartialYMMorZMMWrite;
   }
+  bool omitVZEROUPPER() const { return OmitVZEROUPPER; }
   bool hasFastGather() const { return HasFastGather; }
   bool hasFastScalarFSQRT() const { return HasFastScalarFSQRT; }
   bool hasFastVectorFSQRT() const { return HasFastVectorFSQRT; }
   bool hasFastLZCNT() const { return HasFastLZCNT; }
   bool hasFastSHLDRotate() const { return HasFastSHLDRotate; }
   bool hasFastBEXTR() const { return HasFastBEXTR; }
   bool hasFastHorizontalOps() const { return HasFastHorizontalOps; }
   bool hasFastScalarShiftMasks() const { return HasFastScalarShiftMasks; }

lib/Target/X86/X86TargetMachine.cpp

I = std::make_unique<X86Subtarget>(
         TargetTriple, CPU, FS, *this,
         MaybeAlign(Options.StackAlignmentOverride), PreferVectorWidthOverride,
         RequiredVectorWidth);
   }
   return I.get();
 }
 
 //===----------------------------------------------------------------------===//
-// Command line options for x86
-//===----------------------------------------------------------------------===//
-static cl::opt<bool>
-UseVZeroUpper("x86-use-vzeroupper", cl::Hidden,
-  cl::desc("Minimize AVX to SSE transition penalty"),
-  cl::init(true));
-
-//===----------------------------------------------------------------------===//
 // X86 TTI query.
 //===----------------------------------------------------------------------===//
 
 TargetTransformInfo
 X86TargetMachine::getTargetTransformInfo(const Function &F) {
   return TargetTransformInfo(X86TTIImpl(this, F));
 }
 

 void X86PassConfig::addPreEmitPass() {
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(new X86ExecutionDomainFix());
     addPass(createBreakFalseDeps());
   }
 
   addPass(createX86IndirectBranchTrackingPass());
 
-  if (UseVZeroUpper)
-    addPass(createX86IssueVZeroUpperPass());
 
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(createX86FixupBWInsts());
     addPass(createX86PadShortFunctions());
     addPass(createX86FixupLEAs());
     addPass(createX86EvexToVexInsts());
   }
   addPass(createX86DiscriminateMemOpsPass());

lib/Target/X86/X86TargetTransformInfo.h

const FeatureBitset InlineFeatureIgnoreList = {
 
       // Codegen control options.
       X86::FeatureFast11ByteNOP,
       X86::FeatureFast15ByteNOP,
       X86::FeatureFastBEXTR,
       X86::FeatureFastHorizontalOps,
       X86::FeatureFastLZCNT,
       X86::FeatureFastPartialYMMorZMMWrite,
+      X86::FeatureOmitVZEROUPPER,
       X86::FeatureFastScalarFSQRT,
       X86::FeatureFastSHLDRotate,
       X86::FeatureFastScalarShiftMasks,
       X86::FeatureFastVectorShiftMasks,
       X86::FeatureFastVariableShuffle,
       X86::FeatureFastVectorFSQRT,
       X86::FeatureLEAForSP,
       X86::FeatureLEAUsesAG,

lib/Target/X86/X86VZeroUpper.cpp

void VZeroUpperInserter::processBasicBlock(MachineBasicBlock &MBB) {
 
   BlockStates[MBB.getNumber()].ExitState = CurState;
 }
 
 /// Loop over all of the basic blocks, inserting vzeroupper instructions before
 /// function calls.
 bool VZeroUpperInserter::runOnMachineFunction(MachineFunction &MF) {
   const X86Subtarget &ST = MF.getSubtarget<X86Subtarget>();
-  if (!ST.hasAVX() || ST.hasFastPartialYMMorZMMWrite())
+  if (!ST.hasAVX() || ST.omitVZEROUPPER())
     return false;
   TII = ST.getInstrInfo();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   EverMadeChange = false;
   IsX86INTR = MF.getFunction().getCallingConv() == CallingConv::X86_INTR;
 
   bool FnHasLiveInYmmOrZmm = checkFnHasLiveInYmmOrZmm(MRI);
 

test/CodeGen/X86/avx-vzeroupper.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefix=ALL --check-prefix=VZ --check-prefix=AVX
-; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefix=ALL --check-prefix=VZ --check-prefix=AVX512
-; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-unknown-unknown -mattr=+avx,+fast-partial-ymm-or-zmm-write | FileCheck %s --check-prefix=ALL --check-prefix=NO-VZ --check-prefix=FAST-ymm-zmm
-; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-unknown-unknown -mcpu=bdver2 | FileCheck %s --check-prefix=ALL --check-prefix=NO-VZ --check-prefix=BDVER2
-; RUN: llc < %s -x86-use-vzeroupper -mtriple=x86_64-unknown-unknown -mcpu=btver2 | FileCheck %s --check-prefix=ALL --check-prefix=NO-VZ --check-prefix=BTVER2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx     | FileCheck %s --check-prefixes=ALL,VZ,AVX
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefixes=ALL,VZ,AVX512
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=bdver2    | FileCheck %s --check-prefixes=ALL,NO-VZ,BDVER2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=btver2    | FileCheck %s --check-prefixes=ALL,NO-VZ,BTVER2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx,+fast-partial-ymm-or-zmm-write | FileCheck %s --check-prefixes=ALL,VZ,FAST-ymm-zmm
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx,+omit-vzeroupper | FileCheck %s --check-prefixes=ALL,NO-VZ,AVX-NO-VZ
 
 declare i32 @foo()
 declare <4 x float> @do_sse(<4 x float>)
 declare <8 x float> @do_avx(<8 x float>)
 declare <4 x float> @llvm.x86.avx.vextractf128.ps.256(<8 x float>, i8) nounwind readnone
 @x = common global <4 x float> zeroinitializer, align 16
 @g = common global <8 x float> zeroinitializer, align 32
 

 ; VZ-NEXT:    callq do_sse
 ; VZ-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
 ; VZ-NEXT:    callq do_sse
 ; VZ-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
 ; VZ-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
 ; VZ-NEXT:    addq $56, %rsp
 ; VZ-NEXT:    retq
 ;
-; FAST-ymm-zmm-LABEL: test01:
-; FAST-ymm-zmm:       # %bb.0:
-; FAST-ymm-zmm-NEXT:    subq $56, %rsp
-; FAST-ymm-zmm-NEXT:    vmovups %ymm2, (%rsp) # 32-byte Spill
-; FAST-ymm-zmm-NEXT:    vmovaps {{.*}}(%rip), %xmm0
-; FAST-ymm-zmm-NEXT:    callq do_sse
-; FAST-ymm-zmm-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
-; FAST-ymm-zmm-NEXT:    callq do_sse
-; FAST-ymm-zmm-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
-; FAST-ymm-zmm-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
-; FAST-ymm-zmm-NEXT:    addq $56, %rsp
-; FAST-ymm-zmm-NEXT:    retq
-;
 ; BDVER2-LABEL: test01:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    subq $56, %rsp
 ; BDVER2-NEXT:    vmovaps {{.*}}(%rip), %xmm0
 ; BDVER2-NEXT:    vmovups %ymm2, (%rsp) # 32-byte Spill
 ; BDVER2-NEXT:    vzeroupper
 ; BDVER2-NEXT:    callq do_sse
 ; BDVER2-NEXT:    vmovaps %xmm0, {{.*}}(%rip)

 ; BTVER2-NEXT:    vmovups %ymm2, (%rsp) # 32-byte Spill
 ; BTVER2-NEXT:    callq do_sse
 ; BTVER2-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
 ; BTVER2-NEXT:    callq do_sse
 ; BTVER2-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
 ; BTVER2-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
 ; BTVER2-NEXT:    addq $56, %rsp
 ; BTVER2-NEXT:    retq
+;
+; AVX-NO-VZ-LABEL: test01:
+; AVX-NO-VZ:       # %bb.0:
+; AVX-NO-VZ-NEXT:    subq $56, %rsp
+; AVX-NO-VZ-NEXT:    vmovups %ymm2, (%rsp) # 32-byte Spill
+; AVX-NO-VZ-NEXT:    vmovaps {{.*}}(%rip), %xmm0
+; AVX-NO-VZ-NEXT:    callq do_sse
+; AVX-NO-VZ-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
+; AVX-NO-VZ-NEXT:    callq do_sse
+; AVX-NO-VZ-NEXT:    vmovaps %xmm0, {{.*}}(%rip)
+; AVX-NO-VZ-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
+; AVX-NO-VZ-NEXT:    addq $56, %rsp
+; AVX-NO-VZ-NEXT:    retq
   %tmp = load <4 x float>, <4 x float>* @x, align 16
   %call = tail call <4 x float> @do_sse(<4 x float> %tmp) nounwind
   store <4 x float> %call, <4 x float>* @x, align 16
   %call2 = tail call <4 x float> @do_sse(<4 x float> %call) nounwind
   store <4 x float> %call2, <4 x float>* @x, align 16
   ret <8 x float> %c
 }
 
 ;; Check that vzeroupper is emitted for tail calls.
 
 define <4 x float> @test02(<8 x float> %a, <8 x float> %b) nounwind {
 ; VZ-LABEL: test02:
 ; VZ:       # %bb.0:
 ; VZ-NEXT:    vaddps %xmm1, %xmm0, %xmm0
 ; VZ-NEXT:    vzeroupper
 ; VZ-NEXT:    jmp do_sse # TAILCALL
 ;
-; FAST-ymm-zmm-LABEL: test02:
-; FAST-ymm-zmm:       # %bb.0:
-; FAST-ymm-zmm-NEXT:    vaddps %xmm1, %xmm0, %xmm0
-; FAST-ymm-zmm-NEXT:    jmp do_sse # TAILCALL
-;
 ; BDVER2-LABEL: test02:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vaddps %xmm1, %xmm0, %xmm0
 ; BDVER2-NEXT:    vzeroupper
 ; BDVER2-NEXT:    jmp do_sse # TAILCALL
 ;
 ; BTVER2-LABEL: test02:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vaddps %xmm1, %xmm0, %xmm0
 ; BTVER2-NEXT:    jmp do_sse # TAILCALL
+;
+; AVX-NO-VZ-LABEL: test02:
+; AVX-NO-VZ:       # %bb.0:
+; AVX-NO-VZ-NEXT:    vaddps %xmm1, %xmm0, %xmm0
+; AVX-NO-VZ-NEXT:    jmp do_sse # TAILCALL
   %add.i = fadd <8 x float> %a, %b
   %add.low = call <4 x float> @llvm.x86.avx.vextractf128.ps.256(<8 x float> %add.i, i8 0)
   %call3 = tail call <4 x float> @do_sse(<4 x float> %add.low) nounwind
   ret <4 x float> %call3
 }
 
 ;; Test the pass convergence and also that vzeroupper is only issued when necessary,
 ;; for this function it should be only once

 ; VZ-NEXT:    callq do_sse
 ; VZ-NEXT:    decl %ebx
 ; VZ-NEXT:    jne .LBB3_3
 ; VZ-NEXT:  # %bb.4: # %for.end
 ; VZ-NEXT:    addq $16, %rsp
 ; VZ-NEXT:    popq %rbx
 ; VZ-NEXT:    retq
 ;
-; FAST-ymm-zmm-LABEL: test03:
-; FAST-ymm-zmm:       # %bb.0: # %entry
-; FAST-ymm-zmm-NEXT:    pushq %rbx
-; FAST-ymm-zmm-NEXT:    subq $16, %rsp
-; FAST-ymm-zmm-NEXT:    vaddps %xmm1, %xmm0, %xmm0
-; FAST-ymm-zmm-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
-; FAST-ymm-zmm-NEXT:    .p2align 4, 0x90
-; FAST-ymm-zmm-NEXT:  .LBB3_1: # %while.cond
-; FAST-ymm-zmm-NEXT:    # =>This Inner Loop Header: Depth=1
-; FAST-ymm-zmm-NEXT:    callq foo
-; FAST-ymm-zmm-NEXT:    testl %eax, %eax
-; FAST-ymm-zmm-NEXT:    jne .LBB3_1
-; FAST-ymm-zmm-NEXT:  # %bb.2: # %for.body.preheader
-; FAST-ymm-zmm-NEXT:    movl $4, %ebx
-; FAST-ymm-zmm-NEXT:    vmovaps (%rsp), %xmm0 # 16-byte Reload
-; FAST-ymm-zmm-NEXT:    .p2align 4, 0x90
-; FAST-ymm-zmm-NEXT:  .LBB3_3: # %for.body
-; FAST-ymm-zmm-NEXT:    # =>This Inner Loop Header: Depth=1
-; FAST-ymm-zmm-NEXT:    callq do_sse
-; FAST-ymm-zmm-NEXT:    callq do_sse
-; FAST-ymm-zmm-NEXT:    vmovaps g+{{.*}}(%rip), %xmm0
-; FAST-ymm-zmm-NEXT:    callq do_sse
-; FAST-ymm-zmm-NEXT:    decl %ebx
-; FAST-ymm-zmm-NEXT:    jne .LBB3_3
-; FAST-ymm-zmm-NEXT:  # %bb.4: # %for.end
-; FAST-ymm-zmm-NEXT:    addq $16, %rsp
-; FAST-ymm-zmm-NEXT:    popq %rbx
-; FAST-ymm-zmm-NEXT:    retq
-;
 ; BDVER2-LABEL: test03:
 ; BDVER2:       # %bb.0: # %entry
 ; BDVER2-NEXT:    pushq %rbx
 ; BDVER2-NEXT:    subq $16, %rsp
 ; BDVER2-NEXT:    vaddps %xmm1, %xmm0, %xmm0
 ; BDVER2-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
 ; BDVER2-NEXT:    .p2align 4, 0x90
 ; BDVER2-NEXT:  .LBB3_1: # %while.cond

 ; BTVER2-NEXT:    vmovaps g+{{.*}}(%rip), %xmm0
 ; BTVER2-NEXT:    callq do_sse
 ; BTVER2-NEXT:    decl %ebx
 ; BTVER2-NEXT:    jne .LBB3_3
 ; BTVER2-NEXT:  # %bb.4: # %for.end
 ; BTVER2-NEXT:    addq $16, %rsp
 ; BTVER2-NEXT:    popq %rbx
 ; BTVER2-NEXT:    retq
+;
+; AVX-NO-VZ-LABEL: test03:
+; AVX-NO-VZ:       # %bb.0: # %entry
+; AVX-NO-VZ-NEXT:    pushq %rbx
+; AVX-NO-VZ-NEXT:    subq $16, %rsp
+; AVX-NO-VZ-NEXT:    vaddps %xmm1, %xmm0, %xmm0
+; AVX-NO-VZ-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
+; AVX-NO-VZ-NEXT:    .p2align 4, 0x90
+; AVX-NO-VZ-NEXT:  .LBB3_1: # %while.cond
+; AVX-NO-VZ-NEXT:    # =>This Inner Loop Header: Depth=1
+; AVX-NO-VZ-NEXT:    callq foo
+; AVX-NO-VZ-NEXT:    testl %eax, %eax
+; AVX-NO-VZ-NEXT:    jne .LBB3_1
+; AVX-NO-VZ-NEXT:  # %bb.2: # %for.body.preheader
+; AVX-NO-VZ-NEXT:    movl $4, %ebx
+; AVX-NO-VZ-NEXT:    vmovaps (%rsp), %xmm0 # 16-byte Reload
+; AVX-NO-VZ-NEXT:    .p2align 4, 0x90
+; AVX-NO-VZ-NEXT:  .LBB3_3: # %for.body
+; AVX-NO-VZ-NEXT:    # =>This Inner Loop Header: Depth=1
+; AVX-NO-VZ-NEXT:    callq do_sse
+; AVX-NO-VZ-NEXT:    callq do_sse
+; AVX-NO-VZ-NEXT:    vmovaps g+{{.*}}(%rip), %xmm0
+; AVX-NO-VZ-NEXT:    callq do_sse
+; AVX-NO-VZ-NEXT:    decl %ebx
+; AVX-NO-VZ-NEXT:    jne .LBB3_3
+; AVX-NO-VZ-NEXT:  # %bb.4: # %for.end
+; AVX-NO-VZ-NEXT:    addq $16, %rsp
+; AVX-NO-VZ-NEXT:    popq %rbx
+; AVX-NO-VZ-NEXT:    retq
 entry:
   %add.i = fadd <4 x float> %a, %b
   br label %while.cond
 
 while.cond:
   %call = tail call i32 @foo()
   %tobool = icmp eq i32 %call, 0
   br i1 %tobool, label %for.body, label %while.cond

 ; VZ-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
 ; VZ-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; VZ-NEXT:    callq do_avx
 ; VZ-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
 ; VZ-NEXT:    popq %rax
 ; VZ-NEXT:    vzeroupper
 ; VZ-NEXT:    retq
 ;
-; FAST-ymm-zmm-LABEL: test04:
-; FAST-ymm-zmm:       # %bb.0:
-; FAST-ymm-zmm-NEXT:    pushq %rax
-; FAST-ymm-zmm-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
-; FAST-ymm-zmm-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
-; FAST-ymm-zmm-NEXT:    callq do_avx
-; FAST-ymm-zmm-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
-; FAST-ymm-zmm-NEXT:    popq %rax
-; FAST-ymm-zmm-NEXT:    retq
-;
 ; BDVER2-LABEL: test04:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    pushq %rax
 ; BDVER2-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
 ; BDVER2-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; BDVER2-NEXT:    callq do_avx
 ; BDVER2-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
 ; BDVER2-NEXT:    popq %rax
 ; BDVER2-NEXT:    vzeroupper
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: test04:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    pushq %rax
 ; BTVER2-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
 ; BTVER2-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; BTVER2-NEXT:    callq do_avx
 ; BTVER2-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
 ; BTVER2-NEXT:    popq %rax
 ; BTVER2-NEXT:    retq
+;
+; AVX-NO-VZ-LABEL: test04:
+; AVX-NO-VZ:       # %bb.0:
+; AVX-NO-VZ-NEXT:    pushq %rax
+; AVX-NO-VZ-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; AVX-NO-VZ-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX-NO-VZ-NEXT:    callq do_avx
+; AVX-NO-VZ-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
+; AVX-NO-VZ-NEXT:    popq %rax
+; AVX-NO-VZ-NEXT:    retq
   %shuf = shufflevector <4 x float> %a, <4 x float> %b, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %call = call <8 x float> @do_avx(<8 x float> %shuf) nounwind
   %shuf2 = shufflevector <8 x float> %call, <8 x float> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   ret <4 x float> %shuf2
 }