[X86] Allow folding of reloads from stack slots when loading a subreg of the spilled reg

[mkuper]

Summary

Currently we don't support subregs in InlineSpiller::foldMemoryOperand() because targets may not deal with them correctly.

It seems that X86, at least, already does the right thing when folding a load from a stack slot when it only needs the low (e.g. not "ah") subreg.

This fixes PR30832.

[mkuper]

Add accidentally missing negative test.

[MatzeB]

include/llvm/Target/TargetInstrInfo.h

820–828

Shouldn't we better fix the existing foldMemoryOperand() implementations in the targets instead of adding more API? A "fix" in the sense to actually check the SubReg and return nullptr if it is set would be enough.

[mkuper]

include/llvm/Target/TargetInstrInfo.h

820–828

I'm somewhat wary of changing the default behavior to be more permissive.

It would be very easy to "fix" all the in-tree targets by sinking the SubReg check into their foldMemoryOperandImpl(), but it has the potential of introducing really nasty bugs for downstream out-of-tree targets.

[craig.topper]

lib/Target/X86/X86InstrInfo.h

387

Fix the formatting here to line up the arguments.

[zvi]

lib/Target/X86/X86InstrInfo.h

383

Can you please clarify the last sentence in the comment?

[mkuper]

lib/Target/X86/X86InstrInfo.h

383

Basically, this is the same as the LoadMI parameter to InlineSpiller::foldMemoryOperand(). There it's described as "Load instruction to use instead of stack slot when non-null." but I'm not sure that's any clearer.

There are 3 cases InlineSpiller::foldMemoryOperand() supports:

1. Fold a store to a stack slot into a def.
2. Fold a load from a stack slot into a use.
3. Fold an existing load instruction into a use.

Cases (2) and (3) are distinguished by actually passing the load instruction we want to fold through the API.
I can try to write that down more concisely (but hopefully in a clearer way than it is right now).

Note that this doesn't actually use LoadMI, just verifies it's non-null, but, conceivably, it could be useful. Do you think it would be better if I just pass a bool here for now, and change the API if we eventually need the instruction?

387

Whoops, will do, thanks.

[RKSimon]

Nice! I have an outstanding poor codegen issue with the inability to split+fold scalarization cases, do you think we would be able to expand this patch in the future to handle those cases?

__m128i popcnt1(__m128i *in) {
  return (__m128i) { __builtin_popcountll(in[0][0]), __builtin_popcountll(in[0][1]) };
}

popcnt1(long long __vector(2)*):
        vmovdqu (%rdi), %xmm0
        vmovq   %xmm0, %rax
        vpextrq $1, %xmm0, %rcx
        popcntq %rax, %rax
        popcntq %rcx, %rcx
        vmovq   %rcx, %xmm0
        vmovq   %rax, %xmm1
        vpunpcklqdq     %xmm0, %xmm1, %xmm0 # xmm0 = xmm1[0],xmm0[0]
        retq

Which would be better as:

popcnt1(long long __vector(2)*):
        popcntq (%rdi), %rax
        popcntq 8(%rdi), %rcx
        vmovq   %rcx, %xmm0
        vmovq   %rax, %xmm1
        vpunpcklqdq     %xmm0, %xmm1, %xmm0 # xmm0 = xmm1[0],xmm0[0]
        retq

test/CodeGen/X86/partial-fold.ll

1 ↗

(On Diff #77545)

Add i686 tests as well and regenerate with utils\update_llc_test_checks.py

[mkuper]

In D26521#592717, @RKSimon wrote:

Nice! I have an outstanding poor codegen issue with the inability to split+fold scalarization cases, do you think we would be able to expand this patch in the future to handle those cases?

I don't know the spill/reload code well enough to even tell if it's happening for similar reasons or not.
But CC me on the PR, I'll take a look.

test/CodeGen/X86/partial-fold.ll

1 ↗

(On Diff #77545)

Sure, I'll add i686.
I'd prefer not to use update_llc_test_checks here - there's a *lot* of push/pop noise because of forcing all the regs to be used.

[RKSimon]

In D26521#592842, @mkuper wrote:

In D26521#592717, @RKSimon wrote:

Nice! I have an outstanding poor codegen issue with the inability to split+fold scalarization cases, do you think we would be able to expand this patch in the future to handle those cases?

I don't know the spill/reload code well enough to even tell if it's happening for similar reasons or not.
But CC me on the PR, I'll take a look.

PR30986

test/CodeGen/X86/partial-fold.ll

1 ↗

(On Diff #77545)

That's fine - it might mean that you have to split this into 32/64 versions of the file as the existing inline asm will fail on i686

[wmi]

lib/CodeGen/TargetInstrInfo.cpp

572

In which case we will have (SubRegSize % 8) != 0?

lib/Target/X86/X86InstrInfo.cpp

6281

Why only folding from stack-slot is supported but folding from load is not?

[mkuper]

lib/CodeGen/TargetInstrInfo.cpp

572

It will never happen for x86, but it's possible in theory, and this is target-independent code.

lib/Target/X86/X86InstrInfo.cpp

6281

Because it's the simple case. :-)
I need to make sure the other case makes sense (and find a test-case), and assuming it does, I'll send a follow-up patch.

I'll add a TODO.

[mkuper]

(Hopefully) addressed comments.

[zvi]

Thanks for improving the documentation, Michael. LGTM.

[RKSimon]

Is it worth adding the partial-fold test files to trunk with current codegen?

lib/CodeGen/TargetInstrInfo.cpp

578

Do we need an assert(MemSize && "Empty stack slot") here?

[mkuper]

In D26521#594426, @RKSimon wrote:

Is it worth adding the partial-fold test files to trunk with current codegen?

I think that's rather less useful when not using update_llc_test_checks, since you don't see the full diff anyway. And, as I wrote above, I think update_llc_test_checks really does more harm than good in this case.
But I can add it anyway with just the relevant checks if you want the diff to be more explicit. What do you think?

lib/CodeGen/TargetInstrInfo.cpp

578

Sure, I'll add one.

[qcolombet]

include/llvm/Target/TargetInstrInfo.h

821

I am not sure I understand the intent. I believe this is just a problem of wording.
Basically, are you referring to this case:
v1 = ld

op v1.sub

or this case:
v1.sub = ld

op v1.sub

I believe this is the former and adding the example would help :).

[mkuper]

include/llvm/Target/TargetInstrInfo.h

821

Yes, it's the former, I'll add an example, thanks!

[mkuper]

Updated per comments, and rebased to show test diffs.

Also moved the code that computes MemSize to be before calling into the target's foldMemoryOperandImpl() because, apparently, the X86 implementation feels free to change MI, so accessing MI.getOperand(Idx) after the Impl call may produce unexpected results.

[mkuper]

ping

[MatzeB]

What about changing isSubregFoldable() to not take any parameters and just return true/false, so all the decision logic can stay in foldMemoryOperand()? At least that keeps the newly added API as simple as possible.
It also seems to match the usage pattern in https://reviews.llvm.org/D27002.

lib/CodeGen/TargetInstrInfo.cpp

576

(can't remove this comment because of phabricator bug http://llvm.org/PR30572)

[mkuper]

In D26521#603360, @MatzeB wrote:

What about changing isSubregFoldable() to not take any parameters and just return true/false, so all the decision logic can stay in foldMemoryOperand()?

Do you mean in the target's foldMemoryOperandImpl()? (The decision needs to be target-dependent)
If you do, I think that can work. Not sure it's going to be a net improvement, but I'll post a patch.

[MatzeB]

In D26521#603450, @mkuper wrote:

In D26521#603360, @MatzeB wrote:

What about changing isSubregFoldable() to not take any parameters and just return true/false, so all the decision logic can stay in foldMemoryOperand()?

Do you mean in the target's foldMemoryOperandImpl()? (The decision needs to be target-dependent)
If you do, I think that can work. Not sure it's going to be a net improvement, but I'll post a patch.

Yep, I would expect that to make the code (slightly) easier to follow as the logic is not split accross two callbacks anymore.

[mkuper]

Updated with suggested API.

[MatzeB]

Thanks.

I wanted to propose to only perform the check in the `foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI, unsigned OpNum,

ArrayRef<MachineOperand> MOs, MachineBasicBlock::iterator InsertPt, unsigned Size, unsigned Align, bool AllowCommute)`

variant, as the other two finally call into that anyway. However I see a MI.setDesc(get(NewOpc)); and I currently fail to reason about whether that common function may actually still abort after we changed the instruction type (soo much code I'm not that familiar with...).

So this LGTM as is.

include/llvm/Target/TargetInstrInfo.h

public:
   /// This function is called for all pseudo instructions
   /// that remain after register allocation. Many pseudo instructions are
   /// created to help register allocation. This is the place to convert them
   /// into real instructions. The target can edit MI in place, or it can insert
   /// new instructions and erase MI. The function should return true if
   /// anything was changed.
   virtual bool expandPostRAPseudo(MachineInstr &MI) const { return false; }
 
+  /// Check whether the target can fold a load that feeds a subreg operand MO
+  /// (or a subreg operand MO that feeds a store).
+  /// This can be used to check the foldability of both loads and stores.
+  /// When checking whether a store is foldable, LoadMI is always null.
+  /// When checking whether a load is foldable, LoadMI can either be:
+  /// 1) null when we are checking whether we can fold a stack slot reload
+  /// 2) non-null. In this case, we are checking whether we can fold LoadMI
+  /// into its use.
+  /// For example, X86 may want to return true if it can fold
+  /// movl (%esp), %eax
+  /// subb, %al, ...
+  /// Into:
+  /// subb (%esp), ...
+  ///
+  /// Ideally, we'd like the target implementation of foldMemoryOperand() to
+  /// reject subregs - but since this behavior used to be enforced in the
+  /// target-independent code, moving this responsibility to the targets
+  /// has the potential of causing nasty silent breakage in out-of-tree targets.
+  virtual bool isSubregFoldable(MachineOperand &MO,
+                                MachineInstr *LoadMI) const {
+    return false;
+  }
+
   /// Attempt to fold a load or store of the specified stack
   /// slot into the specified machine instruction for the specified operand(s).
   /// If this is possible, a new instruction is returned with the specified
   /// operand folded, otherwise NULL is returned.
   /// The new instruction is inserted before MI, and the client is responsible
   /// for removing the old instruction.
   MachineInstr *foldMemoryOperand(MachineInstr &MI, ArrayRef<unsigned> Ops,
                                   int FrameIndex,

lib/CodeGen/InlineSpiller.cpp

foldMemoryOperand(ArrayRef<std::pair<MachineInstr*, unsigned> > Ops,
   // Don't attempt folding in bundles.
   MachineInstr *MI = Ops.front().first;
   if (Ops.back().first != MI || MI->isBundled())
     return false;
 
   bool WasCopy = MI->isCopy();
   unsigned ImpReg = 0;
 
+  // We always want to spill subregs for stackmap/patchpoint pseudos.
   bool SpillSubRegs = (MI->getOpcode() == TargetOpcode::STATEPOINT ||
                        MI->getOpcode() == TargetOpcode::PATCHPOINT ||
                        MI->getOpcode() == TargetOpcode::STACKMAP);
 
   // TargetInstrInfo::foldMemoryOperand only expects explicit, non-tied
   // operands.
   SmallVector<unsigned, 8> FoldOps;
   for (const auto &OpPair : Ops) {
     unsigned Idx = OpPair.second;
     assert(MI == OpPair.first && "Instruction conflict during operand folding");
     MachineOperand &MO = MI->getOperand(Idx);
     if (MO.isImplicit()) {
       ImpReg = MO.getReg();
       continue;
     }
-    // FIXME: Teach targets to deal with subregs.
-    if (!SpillSubRegs && MO.getSubReg())
+
+    // Spill subregs if the target allows it.
+    if (!SpillSubRegs && MO.getSubReg() && !TII.isSubregFoldable(MO, LoadMI))
       return false;
     // We cannot fold a load instruction into a def.
     if (LoadMI && MO.isDef())
       return false;
     // Tied use operands should not be passed to foldMemoryOperand.
     if (!MI->isRegTiedToDefOperand(Idx))
       FoldOps.push_back(Idx);
   }

lib/CodeGen/TargetInstrInfo.cpp

if (MI.getOperand(Ops[i]).isDef())
       Flags |= MachineMemOperand::MOStore;
     else
       Flags |= MachineMemOperand::MOLoad;
 
   MachineBasicBlock *MBB = MI.getParent();
   assert(MBB && "foldMemoryOperand needs an inserted instruction");
   MachineFunction &MF = *MBB->getParent();
 
+  // If we're not folding a load into a subreg, the size of the load is the
+  // size of the spill slot. But if we are, we need to figure out what the
+  // actual load size is.
+  int64_t MemSize = 0;
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+
+  if (Flags & MachineMemOperand::MOStore) {
+    MemSize = MFI.getObjectSize(FI);
+  } else {
+    for (unsigned Idx : Ops) {
+      int64_t OpSize = MFI.getObjectSize(FI);
+
+      if (auto SubReg = MI.getOperand(Idx).getSubReg()) {
+        unsigned SubRegSize = TRI->getSubRegIdxSize(SubReg);
+        if (SubRegSize > 0 && !(SubRegSize % 8))
+          OpSize = SubRegSize / 8;
+      }
+
+      MemSize = std::max(MemSize, OpSize);
+    }
+  }
+
+  assert(MemSize && "Did not expect a zero-sized stack slot");
+
   MachineInstr *NewMI = nullptr;
 
   if (MI.getOpcode() == TargetOpcode::STACKMAP ||
       MI.getOpcode() == TargetOpcode::PATCHPOINT ||
       MI.getOpcode() == TargetOpcode::STATEPOINT) {
     // Fold stackmap/patchpoint.
     NewMI = foldPatchpoint(MF, MI, Ops, FI, *this);
     if (NewMI)
       MBB->insert(MI, NewMI);
   } else {
     // Ask the target to do the actual folding.
     NewMI = foldMemoryOperandImpl(MF, MI, Ops, MI, FI, LIS);
   }
 
   if (NewMI) {
     NewMI->setMemRefs(MI.memoperands_begin(), MI.memoperands_end());
     // Add a memory operand, foldMemoryOperandImpl doesn't do that.
     assert((!(Flags & MachineMemOperand::MOStore) ||
             NewMI->mayStore()) &&
            "Folded a def to a non-store!");
     assert((!(Flags & MachineMemOperand::MOLoad) ||
             NewMI->mayLoad()) &&
            "Folded a use to a non-load!");
-    const MachineFrameInfo &MFI = MF.getFrameInfo();
     assert(MFI.getObjectOffset(FI) != -1);
     MachineMemOperand *MMO = MF.getMachineMemOperand(
-        MachinePointerInfo::getFixedStack(MF, FI), Flags, MFI.getObjectSize(FI),
+        MachinePointerInfo::getFixedStack(MF, FI), Flags, MemSize,
         MFI.getObjectAlignment(FI));
     NewMI->addMemOperand(MF, MMO);
 
     return NewMI;
   }
 
   // Straight COPY may fold as load/store.
   if (!MI.isCopy() || Ops.size() != 1)
     return nullptr;
 
   const TargetRegisterClass *RC = canFoldCopy(MI, Ops[0]);
   if (!RC)
     return nullptr;
 
   const MachineOperand &MO = MI.getOperand(1 - Ops[0]);
   MachineBasicBlock::iterator Pos = MI;
-  const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
 
   if (Flags == MachineMemOperand::MOStore)
     storeRegToStackSlot(*MBB, Pos, MO.getReg(), MO.isKill(), FI, RC, TRI);
   else
     loadRegFromStackSlot(*MBB, Pos, MO.getReg(), FI, RC, TRI);
   return &*--Pos;
 }
 

lib/Target/X86/X86InstrInfo.h

void loadRegFromAddr(MachineFunction &MF, unsigned DestReg,
                        SmallVectorImpl<MachineOperand> &Addr,
                        const TargetRegisterClass *RC,
                        MachineInstr::mmo_iterator MMOBegin,
                        MachineInstr::mmo_iterator MMOEnd,
                        SmallVectorImpl<MachineInstr*> &NewMIs) const;
 
   bool expandPostRAPseudo(MachineInstr &MI) const override;
 
+  /// Check whether the target can fold a load that feeds a subreg operand MO
+  /// (or a subreg operand MO that feeds a store).
+  /// This can be used to check the foldability of both loads and stores.
+  /// When checking whether a store is foldable, LoadMI is always null.
+  /// When checking whether a load is foldable, LoadMI can either be:
+  /// 1) null when we are checking whether we can fold a stack slot reload
+  /// 2) non-null. In this case, we are checking whether we can fold LoadMI
+  /// into its use.
+  bool isSubregFoldable(MachineOperand &MO,
+                        MachineInstr *LoadMI) const override;
+
   /// foldMemoryOperand - If this target supports it, fold a load or store of
   /// the specified stack slot into the specified machine instruction for the
   /// specified operand(s).  If this is possible, the target should perform the
   /// folding and return true, otherwise it should return false.  If it folds
   /// the instruction, it is likely that the MachineInstruction the iterator
   /// references has been changed.
   MachineInstr *
   foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,

lib/Target/X86/X86InstrInfo.cpp

if (OpNum == 2) {
       }
     }
     break;
   };
 
   return nullptr;
 }
 
+bool X86InstrInfo::isSubregFoldable(MachineOperand &MO,
+                                    MachineInstr *LoadMI) const {
+  // We only support folding into loads from stack-slots.
+  // TODO: Check whether we can support the LoadMI case as well.
+  if (LoadMI || MO.isDef())
+    return false;
+
+  // We don't want to try to fold into subregs that have a non-zero offset
+  // from the register start. Generally, this should be a TRI query, but
+  // we know there's only one such case on x86.
+  return MO.getSubReg() != X86::sub_8bit_hi;
+}
+
 MachineInstr *X86InstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, unsigned OpNum,
     ArrayRef<MachineOperand> MOs, MachineBasicBlock::iterator InsertPt,
     unsigned Size, unsigned Align, bool AllowCommute) const {
   const DenseMap<unsigned,
                  std::pair<uint16_t, uint16_t> > *OpcodeTablePtr = nullptr;
   bool isCallRegIndirect = Subtarget.callRegIndirect();
   bool isTwoAddrFold = false;

test/CodeGen/X86/partial-fold32.ll

 ; RUN: llc -mtriple=i686-unknown-linux-gnu -enable-misched=false < %s | FileCheck %s
 
 define fastcc i8 @fold32to8(i32 %add, i8 %spill) {
 ; CHECK-LABEL: fold32to8:
 ; CHECK:    movl %ecx, (%esp) # 4-byte Spill
-; CHECK:    movl (%esp), %eax # 4-byte Reload
-; CHECK:    subb %al, %dl
+; CHECK:    subb (%esp), %dl  # 1-byte Folded Reload
 entry:
   tail call void asm sideeffect "", "~{eax},~{ebx},~{ecx},~{edi},~{esi},~{ebp},~{dirflag},~{fpsr},~{flags}"()
   %trunc = trunc i32 %add to i8
   %sub = sub i8 %spill, %trunc
   ret i8 %sub
 }
 
 ; Do not fold a 1-byte store into a 4-byte spill slot

test/CodeGen/X86/partial-fold64.ll

 ; RUN: llc -mtriple=x86_64-unknown-linux-gnu -enable-misched=false < %s | FileCheck %s
 
 define i32 @fold64to32(i64 %add, i32 %spill) {
 ; CHECK-LABEL: fold64to32:
 ; CHECK:    movq %rdi, -{{[0-9]+}}(%rsp) # 8-byte Spill
-; CHECK:    movq -{{[0-9]+}}(%rsp), %rax # 8-byte Reload
-; CHECK:    subl %eax, %esi
+; CHECK:    subl -{{[0-9]+}}(%rsp), %esi # 4-byte Folded Reload
 entry:
   tail call void asm sideeffect "", "~{rax},~{rbx},~{rcx},~{rdx},~{rdi},~{rbp},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15},~{dirflag},~{fpsr},~{flags}"()
   %trunc = trunc i64 %add to i32
   %sub = sub i32 %spill, %trunc
   ret i32 %sub
 }
 
 define i8 @fold64to8(i64 %add, i8 %spill) {
 ; CHECK-LABEL: fold64to8:
 ; CHECK:    movq %rdi, -{{[0-9]+}}(%rsp) # 8-byte Spill
-; CHECK:    movq -{{[0-9]+}}(%rsp), %rax # 8-byte Reload
-; CHECK:    subb %al, %sil
+; CHECK:    subb -{{[0-9]+}}(%rsp), %sil # 1-byte Folded Reload
 entry:
   tail call void asm sideeffect "", "~{rax},~{rbx},~{rcx},~{rdx},~{rdi},~{rbp},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15},~{dirflag},~{fpsr},~{flags}"()
   %trunc = trunc i64 %add to i8
   %sub = sub i8 %spill, %trunc
   ret i8 %sub
 }
 
 ; Do not fold a 4-byte store into a 8-byte spill slot

test/CodeGen/X86/vector-half-conversions.ll

 ; AVX1-NEXT:    movw %ax, %bx
 ; AVX1-NEXT:    shll $16, %ebx
 ; AVX1-NEXT:    vmovaps {{[0-9]+}}(%rsp), %xmm0 # 16-byte Reload
 ; AVX1-NEXT:    callq __truncdfhf2
 ; AVX1-NEXT:    movzwl %ax, %r14d
 ; AVX1-NEXT:    orl %ebx, %r14d
 ; AVX1-NEXT:    shlq $32, %r14
 ; AVX1-NEXT:    orq %r15, %r14
-; AVX1-NEXT:    vmovupd (%rsp), %ymm0 # 32-byte Reload
-; AVX1-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
-; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    vpermilpd $1, (%rsp), %xmm0 # 16-byte Folded Reload
+; AVX1-NEXT:    # xmm0 = mem[1,0]
 ; AVX1-NEXT:    callq __truncdfhf2
 ; AVX1-NEXT:    movw %ax, %bx
 ; AVX1-NEXT:    shll $16, %ebx
 ; AVX1-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
 ; AVX1-NEXT:    # kill: %XMM0<def> %XMM0<kill> %YMM0<kill>
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    callq __truncdfhf2
 ; AVX1-NEXT:    movzwl %ax, %r15d

 ; AVX2-NEXT:    movw %ax, %bx
 ; AVX2-NEXT:    shll $16, %ebx
 ; AVX2-NEXT:    vmovaps {{[0-9]+}}(%rsp), %xmm0 # 16-byte Reload
 ; AVX2-NEXT:    callq __truncdfhf2
 ; AVX2-NEXT:    movzwl %ax, %r14d
 ; AVX2-NEXT:    orl %ebx, %r14d
 ; AVX2-NEXT:    shlq $32, %r14
 ; AVX2-NEXT:    orq %r15, %r14
-; AVX2-NEXT:    vmovupd (%rsp), %ymm0 # 32-byte Reload
-; AVX2-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
-; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    vpermilpd $1, (%rsp), %xmm0 # 16-byte Folded Reload
+; AVX2-NEXT:    # xmm0 = mem[1,0]
 ; AVX2-NEXT:    callq __truncdfhf2
 ; AVX2-NEXT:    movw %ax, %bx
 ; AVX2-NEXT:    shll $16, %ebx
 ; AVX2-NEXT:    vmovups (%rsp), %ymm0 # 32-byte Reload
 ; AVX2-NEXT:    # kill: %XMM0<def> %XMM0<kill> %YMM0<kill>
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    callq __truncdfhf2
 ; AVX2-NEXT:    movzwl %ax, %r15d

 ; AVX1-NEXT:    movw %ax, {{[0-9]+}}(%rsp) # 2-byte Spill
 ; AVX1-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vmovapd %xmm0, {{[0-9]+}}(%rsp) # 16-byte Spill
 ; AVX1-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    callq __truncdfhf2
 ; AVX1-NEXT:    movw %ax, {{[0-9]+}}(%rsp) # 2-byte Spill
-; AVX1-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
-; AVX1-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
-; AVX1-NEXT:    vzeroupper
+; AVX1-NEXT:    vpermilpd $1, {{[0-9]+}}(%rsp), %xmm0 # 16-byte Folded Reload
+; AVX1-NEXT:    # xmm0 = mem[1,0]
 ; AVX1-NEXT:    callq __truncdfhf2
 ; AVX1-NEXT:    movl %eax, %r12d
 ; AVX1-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vmovapd %xmm0, {{[0-9]+}}(%rsp) # 16-byte Spill
 ; AVX1-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    callq __truncdfhf2

 ; AVX2-NEXT:    movw %ax, {{[0-9]+}}(%rsp) # 2-byte Spill
 ; AVX2-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vmovapd %xmm0, {{[0-9]+}}(%rsp) # 16-byte Spill
 ; AVX2-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    callq __truncdfhf2
 ; AVX2-NEXT:    movw %ax, {{[0-9]+}}(%rsp) # 2-byte Spill
-; AVX2-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
-; AVX2-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
-; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    vpermilpd $1, {{[0-9]+}}(%rsp), %xmm0 # 16-byte Folded Reload
+; AVX2-NEXT:    # xmm0 = mem[1,0]
 ; AVX2-NEXT:    callq __truncdfhf2
 ; AVX2-NEXT:    movl %eax, %r12d
 ; AVX2-NEXT:    vmovupd {{[0-9]+}}(%rsp), %ymm0 # 32-byte Reload
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vmovapd %xmm0, {{[0-9]+}}(%rsp) # 16-byte Spill
 ; AVX2-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    callq __truncdfhf2