[X86][AVX512DQ] Use packed instructions for scalar FP<->i64 conversions on 32-bit targets (PR31630)

[craig.topper]

Summary

As i64 types are not legal on 32-bit targets, insert these into a suitable zero vector and use the packed vXi64<->FP conversion instructions instead.

[craig.topper]

Why a DAG combine and not Custom type legalization?

lib/Target/X86/X86ISelLowering.cpp

52

Why bother with the AVX512 check here if we're just going to check DQI inside?

70

Do we need the zeros or could this be scalar_to_vector?

[craig.topper]

Commandeering from Simon to finish this up.

[craig.topper]

Move everything into type legalization lowering.

[delena]

lib/Target/X86/X86ISelLowering.cpp

16065

This case is the same for SINT and UINT. May be put them in a function?

25275

VecVT = MVT::getVectorVT(VT, NumElts);

25279

Why do you need to insert into zero vector? Can you insert to undef?

test/CodeGen/X86/scalar-fp-to-i64.ll

541

Can the memory operand be folded here?
VCVTTPD2UQQ ymm1 {k1}{z},ymm2/m256/m64bcst

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

25279

I think so. I asked the same question before I commandeered it. It's probably no worse than the widening with undef we do for v2f32 legalization.

test/CodeGen/X86/scalar-fp-to-i64.ll

541

We'd have to detect the load and the possibilty of folding it during this lowering code. Or we'd have to use undef for the upper elts and add a DAG combine to turn insert into undef into a broadcast if its foldable.

[craig.topper]

Create helper function to share between SINT_TO_FP and UINT_TO_FP. Use scalar_to_vector instead of inserting into a zero vector.

[delena]

In D43441#1098588, @craig.topper wrote:

Or we'd have to use undef for the upper elts and add a DAG combine to turn insert into undef into a broadcast if its foldable.

I think it is simpler than load folding in td. You can open a bugzilla ticket for possible optimization.

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

25279

In the original patch I was just trying to be very sure there wasn't anything in the other source elements that could cause fp exceptions/overflow flags etc.

[DavidKreitzer]

Could you please add full context to the patch, Craig?

lib/Target/X86/X86ISelLowering.cpp

16048

I suspect we also want to use this vector sequence for unsigned i64 conventions on 64-bit.

25279

I think the only possible side effect from the other source element being undef is raising "inexact". Do we care?

[craig.topper]

Full context

[DavidKreitzer]

Thanks, Craig.

lib/Target/X86/X86ISelLowering.cpp

25279

Forget what I wrote. I was thinking of the INT-->FP case. For the FP-->INT case, why wouldn't we need to worry about raising spurious exceptions?

Also, I'm probably missing something, but it looks like this code is expected to kick in for both 32-bit and 64-bit and both signed and unsigned FP-->i64. Is that intentional? For the 64-bit signed case, I would think we would prefer CVTTSx2SI.

[craig.topper]

Use zeros for other elements for fp->int to avoid spurious invalid exceptions. Since these sorts of scalar conversions are common there's a chance someone could notice the spurious exceptions and complain. We seem to have generated the same code in the test cases either way. Since we ultimatley had to load two i32s into an xmm register. We need use a movd/movss that 0 the upper bits anyway.

Also addeded an assert for !Subtarget.is64Bit() to the fp->int conversion. That code is in ReplaceNodeResults is a hook for the type legalizer to legalize the result type so we should only get there when i64 isn't legal.

[DavidKreitzer]

Looks good, Craig. And thanks for the explanation on the 64-bit cases. I had forgotten about the VCVTTSx2USI & VCVTUSI2Sx instructions, which we appear to be generating already.

[RKSimon]

LGTM - thanks Craig!

lib/Target/X86/X86ISelLowering.cpp

 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCExpr.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/KnownBits.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetOptions.h"
 #include <algorithm>
 #include <bitset>
 #include <cctype>
 #include <numeric>
 using namespace llvm;
 
 #define DEBUG_TYPE "x86-isel"
 
 STATISTIC(NumTailCalls, "Number of tail calls");
 
 static cl::opt<bool> ExperimentalVectorWideningLegalization(
     "x86-experimental-vector-widening-legalization", cl::init(false),
     cl::desc("Enable an experimental vector type legalization through widening "
              "rather than promotion."),
     cl::Hidden);
 
 static cl::opt<int> ExperimentalPrefLoopAlignment(
     "x86-experimental-pref-loop-alignment", cl::init(4),
     cl::desc("Sets the preferable loop alignment for experiments "
SDValue X86TargetLowering::LowerSINT_TO_FP(SDValue Op,
   SDValue Src = Op.getOperand(0);
   MVT SrcVT = Src.getSimpleValueType();
   MVT VT = Op.getSimpleValueType();
   SDLoc dl(Op);
 
   if (SrcVT.isVector()) {
     if (SrcVT == MVT::v2i32 && VT == MVT::v2f64) {
       return DAG.getNode(X86ISD::CVTSI2P, dl, VT,
                          DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v4i32, Src,
                                      DAG.getUNDEF(SrcVT)));
     }
     return SDValue();
   }
 
   assert(SrcVT <= MVT::i64 && SrcVT >= MVT::i16 &&
          "Unknown SINT_TO_FP to lower!");
 
   // These are really Legal; return the operand so the caller accepts it as
   // Legal.
-  if (SrcVT == MVT::i32 && isScalarFPTypeInSSEReg(Op.getValueType()))
+  if (SrcVT == MVT::i32 && isScalarFPTypeInSSEReg(VT))
     return Op;
-  if (SrcVT == MVT::i64 && isScalarFPTypeInSSEReg(Op.getValueType()) &&
-      Subtarget.is64Bit()) {
+  if (SrcVT == MVT::i64 && isScalarFPTypeInSSEReg(VT) && Subtarget.is64Bit()) {
     return Op;
   }
 
+  // Pack the i64 into a vector, do the operation and extract.
+  if (Subtarget.hasDQI() && isScalarFPTypeInSSEReg(VT) &&
+      (SrcVT == MVT::i64 && !Subtarget.is64Bit())) {
+    // Using 256-bit to ensure result is 128-bits for f32 case.
+    MVT VecInVT = Subtarget.hasVLX() ? MVT::v4i64 : MVT::v8i64;
+    unsigned NumElts = Subtarget.hasVLX() ? 4 : 8;
+    MVT VecVT = MVT::getVectorVT(VT, NumElts);
+
+    SDValue ZeroIdx = DAG.getIntPtrConstant(0, dl);
+    SDValue InVec = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, VecInVT,
+                                DAG.getConstant(0, dl, VecInVT), Src, ZeroIdx);
+    SDValue CvtVec = DAG.getNode(ISD::SINT_TO_FP, dl, VecVT, InVec);
+    return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, VT, CvtVec, ZeroIdx);
+  }
+
   SDValue ValueToStore = Op.getOperand(0);
-  if (SrcVT == MVT::i64 && isScalarFPTypeInSSEReg(Op.getValueType()) &&
+  if (SrcVT == MVT::i64 && isScalarFPTypeInSSEReg(VT) &&
       !Subtarget.is64Bit())
     // Bitcasting to f64 here allows us to do a single 64-bit store from
     // an SSE register, avoiding the store forwarding penalty that would come
     // with two 32-bit stores.
     ValueToStore = DAG.getBitcast(MVT::f64, ValueToStore);
 
   unsigned Size = SrcVT.getSizeInBits()/8;
   MachineFunction &MF = DAG.getMachineFunction();
SDValue X86TargetLowering::LowerUINT_TO_FP(SDValue Op,
 
   if (Subtarget.hasAVX512() && isScalarFPTypeInSSEReg(DstVT) &&
       (SrcVT == MVT::i32 || (SrcVT == MVT::i64 && Subtarget.is64Bit()))) {
     // Conversions from unsigned i32 to f32/f64 are legal,
     // using VCVTUSI2SS/SD.  Same for i64 in 64-bit mode.
     return Op;
   }
 
+  // Pack the i64 into a vector, do the operation and extract.
+  if (Subtarget.hasDQI() && isScalarFPTypeInSSEReg(DstVT) &&
+      (SrcVT == MVT::i64 && !Subtarget.is64Bit())) {
+    // Using 256-bit to ensure result is 128-bits for f32 case.
+    MVT VecInVT = Subtarget.hasVLX() ? MVT::v4i64 : MVT::v8i64;
+    unsigned NumElts = Subtarget.hasVLX() ? 4 : 8;
+    MVT VecVT = MVT::getVectorVT(DstVT, NumElts);
+
+    SDValue ZeroIdx = DAG.getIntPtrConstant(0, dl);
+    SDValue InVec = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, VecInVT,
+                                DAG.getConstant(0, dl, VecInVT), N0, ZeroIdx);
+    SDValue CvtVec = DAG.getNode(ISD::UINT_TO_FP, dl, VecVT, InVec);
+    return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, DstVT, CvtVec, ZeroIdx);
+  }
+
   if (SrcVT == MVT::i64 && DstVT == MVT::f64 && X86ScalarSSEf64)
     return LowerUINT_TO_FP_i64(Op, DAG, Subtarget);
   if (SrcVT == MVT::i32 && X86ScalarSSEf64)
     return LowerUINT_TO_FP_i32(Op, DAG, Subtarget);
   if (Subtarget.is64Bit() && SrcVT == MVT::i64 && DstVT == MVT::f32)
     return SDValue();
 
   // Make a 64-bit buffer, and use it to build an FILD.
void X86TargetLowering::ReplaceNodeResults(SDNode *N,
   case ISD::UDIVREM: {
     SDValue V = LowerWin64_i128OP(SDValue(N,0), DAG);
     Results.push_back(V);
     return;
   }
   case ISD::FP_TO_SINT:
   case ISD::FP_TO_UINT: {
     bool IsSigned = N->getOpcode() == ISD::FP_TO_SINT;
+    EVT VT = N->getValueType(0);
+    SDValue Src = N->getOperand(0);
+    EVT SrcVT = Src.getValueType();
 
-    if (N->getValueType(0) == MVT::v2i32) {
+    if (VT == MVT::v2i32) {
       assert((IsSigned || Subtarget.hasAVX512()) &&
              "Can only handle signed conversion without AVX512");
       assert(Subtarget.hasSSE2() && "Requires at least SSE2!");
-      SDValue Src = N->getOperand(0);
       if (Src.getValueType() == MVT::v2f64) {
         MVT ResVT = MVT::v4i32;
         unsigned Opc = IsSigned ? X86ISD::CVTTP2SI : X86ISD::CVTTP2UI;
         if (!IsSigned && !Subtarget.hasVLX()) {
           // Widen to 512-bits.
           ResVT = MVT::v8i32;
           Opc = ISD::FP_TO_UINT;
           Src = DAG.getNode(ISD::INSERT_SUBVECTOR, dl, MVT::v8f64,
                             DAG.getUNDEF(MVT::v8f64),
                             Src, DAG.getIntPtrConstant(0, dl));
         }
         SDValue Res = DAG.getNode(Opc, dl, ResVT, Src);
         bool WidenType = getTypeAction(*DAG.getContext(),
                                        MVT::v2i32) == TypeWidenVector;
         ResVT = WidenType ? MVT::v4i32 : MVT::v2i32;
         Res = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, ResVT, Res,
                           DAG.getIntPtrConstant(0, dl));
         Results.push_back(Res);
         return;
       }
-      if (Src.getValueType() == MVT::v2f32) {
+      if (SrcVT == MVT::v2f32) {
         SDValue Idx = DAG.getIntPtrConstant(0, dl);
         SDValue Res = DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v4f32, Src,
                                   DAG.getUNDEF(MVT::v2f32));
         Res = DAG.getNode(IsSigned ? ISD::FP_TO_SINT
                                    : ISD::FP_TO_UINT, dl, MVT::v4i32, Res);
         if (getTypeAction(*DAG.getContext(), MVT::v2i32) != TypeWidenVector)
           Res = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, MVT::v2i32, Res, Idx);
         Results.push_back(Res);
         return;
       }
 
       // The FP_TO_INTHelper below only handles f32/f64/f80 scalar inputs,
       // so early out here.
       return;
     }
 
+    if (Subtarget.hasDQI() && VT == MVT::i64 &&
+        (SrcVT == MVT::f32 || SrcVT == MVT::f64)) {
+      unsigned NumElts = Subtarget.hasVLX() ? 4 : 8;
+      // Using a 256-bit input here to guarantee 128-bit input for f32 case.
+      // TODO: Use 128-bit vectors for f64 case?
+      // TODO: Use 128-bit vectors for f32 by using CVTTP2SI/CVTTP2UI.
+      MVT VecVT = Subtarget.hasVLX() ? MVT::v4i64 : MVT::v8i64;
+      MVT VecInVT = MVT::getVectorVT(SrcVT.getSimpleVT(), NumElts);
+
+      SDValue ZeroIdx = DAG.getIntPtrConstant(0, dl);
+      SDValue Res = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, VecInVT,
+                                DAG.getConstantFP(0.0, dl, VecInVT),
+                                Src, ZeroIdx);
+      Res = DAG.getNode(N->getOpcode(), SDLoc(N), VecVT, Res);
+      Res = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, VT, Res, ZeroIdx);
+      Results.push_back(Res);
+      return;
+    }
+
     std::pair<SDValue,SDValue> Vals =
         FP_TO_INTHelper(SDValue(N, 0), DAG, IsSigned, /*IsReplace=*/ true);
     SDValue FIST = Vals.first, StackSlot = Vals.second;
     if (FIST.getNode()) {
-      EVT VT = N->getValueType(0);
       // Return a load from the stack slot.
       if (StackSlot.getNode())
         Results.push_back(
             DAG.getLoad(VT, dl, FIST, StackSlot, MachinePointerInfo()));
       else
         Results.push_back(FIST);
     }
     return;

test/CodeGen/X86/avx512-regcall-NoMask.ll

; LINUXOSX64-NEXT:    retq
   %x8 = fadd <32 x float> %c2, %x7
   ret <32 x float> %x8
 }
 
 ; Test regcall when passing/retrieving mixed types
 define x86_regcallcc i32 @test_argRetMixTypes(double, float, i8 signext, i32, i64, i16 signext, i32*) #0 {
 ; X32-LABEL: test_argRetMixTypes:
 ; X32:       # %bb.0:
-; X32-NEXT:    pushl %ebp
-; X32-NEXT:    movl %esp, %ebp
-; X32-NEXT:    andl $-8, %esp
-; X32-NEXT:    subl $16, %esp
-; X32-NEXT:    vmovd %edx, %xmm2
-; X32-NEXT:    vpinsrd $1, %edi, %xmm2, %xmm2
-; X32-NEXT:    movl 8(%ebp), %edx
+; X32-NEXT:    pushl %ebx
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X32-NEXT:    vcvtss2sd %xmm1, %xmm1, %xmm1
 ; X32-NEXT:    vaddsd %xmm0, %xmm1, %xmm0
-; X32-NEXT:    vcvtsi2sdl %eax, %xmm3, %xmm1
+; X32-NEXT:    vcvtsi2sdl %eax, %xmm2, %xmm1
 ; X32-NEXT:    vaddsd %xmm1, %xmm0, %xmm0
-; X32-NEXT:    vcvtsi2sdl %ecx, %xmm3, %xmm1
+; X32-NEXT:    vcvtsi2sdl %ecx, %xmm2, %xmm1
+; X32-NEXT:    vaddsd %xmm1, %xmm0, %xmm0
+; X32-NEXT:    vmovd %edi, %xmm1
+; X32-NEXT:    vmovd %edx, %xmm2
+; X32-NEXT:    vpunpckldq {{.*#+}} xmm1 = xmm2[0],xmm1[0],xmm2[1],xmm1[1]
+; X32-NEXT:    vmovq {{.*#+}} xmm1 = xmm1[0],zero
+; X32-NEXT:    vcvtqq2pd %ymm1, %ymm1
 ; X32-NEXT:    vaddsd %xmm1, %xmm0, %xmm0
-; X32-NEXT:    vmovq %xmm2, {{[0-9]+}}(%esp)
-; X32-NEXT:    fildll {{[0-9]+}}(%esp)
-; X32-NEXT:    fstpl (%esp)
-; X32-NEXT:    vaddsd (%esp), %xmm0, %xmm0
 ; X32-NEXT:    vcvtsi2sdl %esi, %xmm3, %xmm1
 ; X32-NEXT:    vaddsd %xmm1, %xmm0, %xmm0
-; X32-NEXT:    vcvtsi2sdl (%edx), %xmm3, %xmm1
+; X32-NEXT:    vcvtsi2sdl (%ebx), %xmm3, %xmm1
 ; X32-NEXT:    vaddsd %xmm1, %xmm0, %xmm0
 ; X32-NEXT:    vcvttsd2si %xmm0, %eax
-; X32-NEXT:    movl %ebp, %esp
-; X32-NEXT:    popl %ebp
+; X32-NEXT:    popl %ebx
+; X32-NEXT:    vzeroupper
 ; X32-NEXT:    retl
 ;
 ; WIN64-LABEL: test_argRetMixTypes:
 ; WIN64:       # %bb.0:
 ; WIN64-NEXT:    vcvtss2sd %xmm1, %xmm1, %xmm1
 ; WIN64-NEXT:    vaddsd %xmm0, %xmm1, %xmm0
 ; WIN64-NEXT:    vcvtsi2sdl %eax, %xmm2, %xmm1
 ; WIN64-NEXT:    vaddsd %xmm1, %xmm0, %xmm0

test/CodeGen/X86/scalar-fp-to-i64.ll

 ; Otherwise the sequence will involve an FP subtract (fsub, subss or subsd),
 ; and a truncating conversion (cvtts[ds]2si, fisttp, or fnstcw+fist).  When
 ; both a subtract and fnstcw are needed, they can occur in either order.
 ;
 ; The interesting subtargets are AVX512F (vcvtts[ds]2usi), SSE3 (fisttp),
 ; SSE2 (cvtts[ds]2si) and vanilla X87 (fnstcw+fist, 32-bit only).
 
 define i64 @f_to_u64(float %a) nounwind {
-; AVX512_32_WIN-LABEL: f_to_u64:
-; AVX512_32_WIN:       # %bb.0:
-; AVX512_32_WIN-NEXT:    pushl %ebp
-; AVX512_32_WIN-NEXT:    movl %esp, %ebp
-; AVX512_32_WIN-NEXT:    andl $-8, %esp
-; AVX512_32_WIN-NEXT:    subl $16, %esp
-; AVX512_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512_32_WIN-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; AVX512_32_WIN-NEXT:    vcmpltss %xmm1, %xmm0, %k1
-; AVX512_32_WIN-NEXT:    vsubss %xmm1, %xmm0, %xmm2
-; AVX512_32_WIN-NEXT:    vmovss %xmm0, %xmm0, %xmm2 {%k1}
-; AVX512_32_WIN-NEXT:    vmovss %xmm2, {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    flds {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fisttpll (%esp)
-; AVX512_32_WIN-NEXT:    xorl %edx, %edx
-; AVX512_32_WIN-NEXT:    vucomiss %xmm0, %xmm1
-; AVX512_32_WIN-NEXT:    setbe %dl
-; AVX512_32_WIN-NEXT:    shll $31, %edx
-; AVX512_32_WIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
-; AVX512_32_WIN-NEXT:    movl (%esp), %eax
-; AVX512_32_WIN-NEXT:    movl %ebp, %esp
-; AVX512_32_WIN-NEXT:    popl %ebp
-; AVX512_32_WIN-NEXT:    retl
-;
-; AVX512_32_LIN-LABEL: f_to_u64:
-; AVX512_32_LIN:       # %bb.0:
-; AVX512_32_LIN-NEXT:    subl $20, %esp
-; AVX512_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512_32_LIN-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; AVX512_32_LIN-NEXT:    vcmpltss %xmm1, %xmm0, %k1
-; AVX512_32_LIN-NEXT:    vsubss %xmm1, %xmm0, %xmm2
-; AVX512_32_LIN-NEXT:    vmovss %xmm0, %xmm0, %xmm2 {%k1}
-; AVX512_32_LIN-NEXT:    vmovss %xmm2, {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    flds {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fisttpll (%esp)
-; AVX512_32_LIN-NEXT:    xorl %edx, %edx
-; AVX512_32_LIN-NEXT:    vucomiss %xmm0, %xmm1
-; AVX512_32_LIN-NEXT:    setbe %dl
-; AVX512_32_LIN-NEXT:    shll $31, %edx
-; AVX512_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
-; AVX512_32_LIN-NEXT:    movl (%esp), %eax
-; AVX512_32_LIN-NEXT:    addl $20, %esp
-; AVX512_32_LIN-NEXT:    retl
+; AVX512DQVL_32_WIN-LABEL: f_to_u64:
+; AVX512DQVL_32_WIN:       # %bb.0:
+; AVX512DQVL_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQVL_32_WIN-NEXT:    vcvttps2uqq %xmm0, %ymm0
+; AVX512DQVL_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_WIN-NEXT:    vzeroupper
+; AVX512DQVL_32_WIN-NEXT:    retl
+;
+; AVX512DQVL_32_LIN-LABEL: f_to_u64:
+; AVX512DQVL_32_LIN:       # %bb.0:
+; AVX512DQVL_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQVL_32_LIN-NEXT:    vcvttps2uqq %xmm0, %ymm0
+; AVX512DQVL_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_LIN-NEXT:    vzeroupper
+; AVX512DQVL_32_LIN-NEXT:    retl
 ;
 ; AVX512_64-LABEL: f_to_u64:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvttss2usi %xmm0, %rax
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32_WIN-LABEL: f_to_u64:
+; AVX512DQ_32_WIN:       # %bb.0:
+; AVX512DQ_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQ_32_WIN-NEXT:    vcvttps2uqq %ymm0, %zmm0
+; AVX512DQ_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_WIN-NEXT:    vzeroupper
+; AVX512DQ_32_WIN-NEXT:    retl
+;
+; AVX512DQ_32_LIN-LABEL: f_to_u64:
+; AVX512DQ_32_LIN:       # %bb.0:
+; AVX512DQ_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQ_32_LIN-NEXT:    vcvttps2uqq %ymm0, %zmm0
+; AVX512DQ_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_LIN-NEXT:    vzeroupper
+; AVX512DQ_32_LIN-NEXT:    retl
+;
+; AVX512F_32_WIN-LABEL: f_to_u64:
+; AVX512F_32_WIN:       # %bb.0:
+; AVX512F_32_WIN-NEXT:    pushl %ebp
+; AVX512F_32_WIN-NEXT:    movl %esp, %ebp
+; AVX512F_32_WIN-NEXT:    andl $-8, %esp
+; AVX512F_32_WIN-NEXT:    subl $16, %esp
+; AVX512F_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512F_32_WIN-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512F_32_WIN-NEXT:    vcmpltss %xmm1, %xmm0, %k1
+; AVX512F_32_WIN-NEXT:    vsubss %xmm1, %xmm0, %xmm2
+; AVX512F_32_WIN-NEXT:    vmovss %xmm0, %xmm0, %xmm2 {%k1}
+; AVX512F_32_WIN-NEXT:    vmovss %xmm2, {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    flds {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_WIN-NEXT:    xorl %edx, %edx
+; AVX512F_32_WIN-NEXT:    vucomiss %xmm0, %xmm1
+; AVX512F_32_WIN-NEXT:    setbe %dl
+; AVX512F_32_WIN-NEXT:    shll $31, %edx
+; AVX512F_32_WIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_WIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_WIN-NEXT:    movl %ebp, %esp
+; AVX512F_32_WIN-NEXT:    popl %ebp
+; AVX512F_32_WIN-NEXT:    retl
+;
+; AVX512F_32_LIN-LABEL: f_to_u64:
+; AVX512F_32_LIN:       # %bb.0:
+; AVX512F_32_LIN-NEXT:    subl $20, %esp
+; AVX512F_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512F_32_LIN-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512F_32_LIN-NEXT:    vcmpltss %xmm1, %xmm0, %k1
+; AVX512F_32_LIN-NEXT:    vsubss %xmm1, %xmm0, %xmm2
+; AVX512F_32_LIN-NEXT:    vmovss %xmm0, %xmm0, %xmm2 {%k1}
+; AVX512F_32_LIN-NEXT:    vmovss %xmm2, {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    flds {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_LIN-NEXT:    xorl %edx, %edx
+; AVX512F_32_LIN-NEXT:    vucomiss %xmm0, %xmm1
+; AVX512F_32_LIN-NEXT:    setbe %dl
+; AVX512F_32_LIN-NEXT:    shll $31, %edx
+; AVX512F_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_LIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_LIN-NEXT:    addl $20, %esp
+; AVX512F_32_LIN-NEXT:    retl
+;
 ; SSE3_32_WIN-LABEL: f_to_u64:
 ; SSE3_32_WIN:       # %bb.0:
 ; SSE3_32_WIN-NEXT:    pushl %ebp
 ; SSE3_32_WIN-NEXT:    movl %esp, %ebp
 ; SSE3_32_WIN-NEXT:    andl $-8, %esp
 ; SSE3_32_WIN-NEXT:    subl $16, %esp
 ; SSE3_32_WIN-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; SSE3_32_WIN-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero

 ; X87_LIN-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X87_LIN-NEXT:    addl $20, %esp
 ; X87_LIN-NEXT:    retl
   %r = fptoui float %a to i64
   ret i64 %r
 }
 
 define i64 @f_to_s64(float %a) nounwind {
-; AVX512_32_WIN-LABEL: f_to_s64:
-; AVX512_32_WIN:       # %bb.0:
-; AVX512_32_WIN-NEXT:    pushl %ebp
-; AVX512_32_WIN-NEXT:    movl %esp, %ebp
-; AVX512_32_WIN-NEXT:    andl $-8, %esp
-; AVX512_32_WIN-NEXT:    subl $16, %esp
-; AVX512_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512_32_WIN-NEXT:    vmovss %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    flds {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fisttpll (%esp)
-; AVX512_32_WIN-NEXT:    movl (%esp), %eax
-; AVX512_32_WIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; AVX512_32_WIN-NEXT:    movl %ebp, %esp
-; AVX512_32_WIN-NEXT:    popl %ebp
-; AVX512_32_WIN-NEXT:    retl
-;
-; AVX512_32_LIN-LABEL: f_to_s64:
-; AVX512_32_LIN:       # %bb.0:
-; AVX512_32_LIN-NEXT:    subl $20, %esp
-; AVX512_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512_32_LIN-NEXT:    vmovss %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    flds {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fisttpll (%esp)
-; AVX512_32_LIN-NEXT:    movl (%esp), %eax
-; AVX512_32_LIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; AVX512_32_LIN-NEXT:    addl $20, %esp
-; AVX512_32_LIN-NEXT:    retl
+; AVX512DQVL_32_WIN-LABEL: f_to_s64:
+; AVX512DQVL_32_WIN:       # %bb.0:
+; AVX512DQVL_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQVL_32_WIN-NEXT:    vcvttps2qq %xmm0, %ymm0
+; AVX512DQVL_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_WIN-NEXT:    vzeroupper
+; AVX512DQVL_32_WIN-NEXT:    retl
+;
+; AVX512DQVL_32_LIN-LABEL: f_to_s64:
+; AVX512DQVL_32_LIN:       # %bb.0:
+; AVX512DQVL_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQVL_32_LIN-NEXT:    vcvttps2qq %xmm0, %ymm0
+; AVX512DQVL_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_LIN-NEXT:    vzeroupper
+; AVX512DQVL_32_LIN-NEXT:    retl
 ;
 ; AVX512_64-LABEL: f_to_s64:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvttss2si %xmm0, %rax
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32_WIN-LABEL: f_to_s64:
+; AVX512DQ_32_WIN:       # %bb.0:
+; AVX512DQ_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQ_32_WIN-NEXT:    vcvttps2qq %ymm0, %zmm0
+; AVX512DQ_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_WIN-NEXT:    vzeroupper
+; AVX512DQ_32_WIN-NEXT:    retl
+;
+; AVX512DQ_32_LIN-LABEL: f_to_s64:
+; AVX512DQ_32_LIN:       # %bb.0:
+; AVX512DQ_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512DQ_32_LIN-NEXT:    vcvttps2qq %ymm0, %zmm0
+; AVX512DQ_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_LIN-NEXT:    vzeroupper
+; AVX512DQ_32_LIN-NEXT:    retl
+;
+; AVX512F_32_WIN-LABEL: f_to_s64:
+; AVX512F_32_WIN:       # %bb.0:
+; AVX512F_32_WIN-NEXT:    pushl %ebp
+; AVX512F_32_WIN-NEXT:    movl %esp, %ebp
+; AVX512F_32_WIN-NEXT:    andl $-8, %esp
+; AVX512F_32_WIN-NEXT:    subl $16, %esp
+; AVX512F_32_WIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512F_32_WIN-NEXT:    vmovss %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    flds {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_WIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_WIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_WIN-NEXT:    movl %ebp, %esp
+; AVX512F_32_WIN-NEXT:    popl %ebp
+; AVX512F_32_WIN-NEXT:    retl
+;
+; AVX512F_32_LIN-LABEL: f_to_s64:
+; AVX512F_32_LIN:       # %bb.0:
+; AVX512F_32_LIN-NEXT:    subl $20, %esp
+; AVX512F_32_LIN-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512F_32_LIN-NEXT:    vmovss %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    flds {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_LIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_LIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_LIN-NEXT:    addl $20, %esp
+; AVX512F_32_LIN-NEXT:    retl
+;
 ; SSE3_32_WIN-LABEL: f_to_s64:
 ; SSE3_32_WIN:       # %bb.0:
 ; SSE3_32_WIN-NEXT:    pushl %ebp
 ; SSE3_32_WIN-NEXT:    movl %esp, %ebp
 ; SSE3_32_WIN-NEXT:    andl $-8, %esp
 ; SSE3_32_WIN-NEXT:    subl $16, %esp
 ; SSE3_32_WIN-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; SSE3_32_WIN-NEXT:    movss %xmm0, {{[0-9]+}}(%esp)

 ; X87_LIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X87_LIN-NEXT:    addl $20, %esp
 ; X87_LIN-NEXT:    retl
   %r = fptosi float %a to i64
   ret i64 %r
 }
 
 define i64 @d_to_u64(double %a) nounwind {
-; AVX512_32_WIN-LABEL: d_to_u64:
-; AVX512_32_WIN:       # %bb.0:
-; AVX512_32_WIN-NEXT:    pushl %ebp
-; AVX512_32_WIN-NEXT:    movl %esp, %ebp
-; AVX512_32_WIN-NEXT:    andl $-8, %esp
-; AVX512_32_WIN-NEXT:    subl $16, %esp
-; AVX512_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32_WIN-NEXT:    vmovsd {{.*#+}} xmm1 = mem[0],zero
-; AVX512_32_WIN-NEXT:    vcmpltsd %xmm1, %xmm0, %k1
-; AVX512_32_WIN-NEXT:    vsubsd %xmm1, %xmm0, %xmm2
-; AVX512_32_WIN-NEXT:    vmovsd %xmm0, %xmm0, %xmm2 {%k1}
-; AVX512_32_WIN-NEXT:    vmovsd %xmm2, {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fldl {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fisttpll (%esp)
-; AVX512_32_WIN-NEXT:    xorl %edx, %edx
-; AVX512_32_WIN-NEXT:    vucomisd %xmm0, %xmm1
-; AVX512_32_WIN-NEXT:    setbe %dl
-; AVX512_32_WIN-NEXT:    shll $31, %edx
-; AVX512_32_WIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
-; AVX512_32_WIN-NEXT:    movl (%esp), %eax
-; AVX512_32_WIN-NEXT:    movl %ebp, %esp
-; AVX512_32_WIN-NEXT:    popl %ebp
-; AVX512_32_WIN-NEXT:    retl
-;
-; AVX512_32_LIN-LABEL: d_to_u64:
-; AVX512_32_LIN:       # %bb.0:
-; AVX512_32_LIN-NEXT:    subl $20, %esp
-; AVX512_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32_LIN-NEXT:    vmovsd {{.*#+}} xmm1 = mem[0],zero
-; AVX512_32_LIN-NEXT:    vcmpltsd %xmm1, %xmm0, %k1
-; AVX512_32_LIN-NEXT:    vsubsd %xmm1, %xmm0, %xmm2
-; AVX512_32_LIN-NEXT:    vmovsd %xmm0, %xmm0, %xmm2 {%k1}
-; AVX512_32_LIN-NEXT:    vmovsd %xmm2, {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fldl {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fisttpll (%esp)
-; AVX512_32_LIN-NEXT:    xorl %edx, %edx
-; AVX512_32_LIN-NEXT:    vucomisd %xmm0, %xmm1
-; AVX512_32_LIN-NEXT:    setbe %dl
-; AVX512_32_LIN-NEXT:    shll $31, %edx
-; AVX512_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
-; AVX512_32_LIN-NEXT:    movl (%esp), %eax
-; AVX512_32_LIN-NEXT:    addl $20, %esp
-; AVX512_32_LIN-NEXT:    retl
+; AVX512DQVL_32_WIN-LABEL: d_to_u64:
+; AVX512DQVL_32_WIN:       # %bb.0:
+; AVX512DQVL_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32_WIN-NEXT:    vcvttpd2uqq %ymm0, %ymm0
+; AVX512DQVL_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_WIN-NEXT:    vzeroupper
+; AVX512DQVL_32_WIN-NEXT:    retl
+;
+; AVX512DQVL_32_LIN-LABEL: d_to_u64:
+; AVX512DQVL_32_LIN:       # %bb.0:
+; AVX512DQVL_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32_LIN-NEXT:    vcvttpd2uqq %ymm0, %ymm0
+; AVX512DQVL_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_LIN-NEXT:    vzeroupper
+; AVX512DQVL_32_LIN-NEXT:    retl
 ;
 ; AVX512_64-LABEL: d_to_u64:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvttsd2usi %xmm0, %rax
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32_WIN-LABEL: d_to_u64:
+; AVX512DQ_32_WIN:       # %bb.0:
+; AVX512DQ_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32_WIN-NEXT:    vcvttpd2uqq %zmm0, %zmm0
+; AVX512DQ_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_WIN-NEXT:    vzeroupper
+; AVX512DQ_32_WIN-NEXT:    retl
+;
+; AVX512DQ_32_LIN-LABEL: d_to_u64:
+; AVX512DQ_32_LIN:       # %bb.0:
+; AVX512DQ_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32_LIN-NEXT:    vcvttpd2uqq %zmm0, %zmm0
+; AVX512DQ_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_LIN-NEXT:    vzeroupper
+; AVX512DQ_32_LIN-NEXT:    retl
+;
+; AVX512F_32_WIN-LABEL: d_to_u64:
+; AVX512F_32_WIN:       # %bb.0:
+; AVX512F_32_WIN-NEXT:    pushl %ebp
+; AVX512F_32_WIN-NEXT:    movl %esp, %ebp
+; AVX512F_32_WIN-NEXT:    andl $-8, %esp
+; AVX512F_32_WIN-NEXT:    subl $16, %esp
+; AVX512F_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32_WIN-NEXT:    vmovsd {{.*#+}} xmm1 = mem[0],zero
+; AVX512F_32_WIN-NEXT:    vcmpltsd %xmm1, %xmm0, %k1
+; AVX512F_32_WIN-NEXT:    vsubsd %xmm1, %xmm0, %xmm2
+; AVX512F_32_WIN-NEXT:    vmovsd %xmm0, %xmm0, %xmm2 {%k1}
+; AVX512F_32_WIN-NEXT:    vmovsd %xmm2, {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fldl {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_WIN-NEXT:    xorl %edx, %edx
+; AVX512F_32_WIN-NEXT:    vucomisd %xmm0, %xmm1
+; AVX512F_32_WIN-NEXT:    setbe %dl
+; AVX512F_32_WIN-NEXT:    shll $31, %edx
+; AVX512F_32_WIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_WIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_WIN-NEXT:    movl %ebp, %esp
+; AVX512F_32_WIN-NEXT:    popl %ebp
+; AVX512F_32_WIN-NEXT:    retl
+;
+; AVX512F_32_LIN-LABEL: d_to_u64:
+; AVX512F_32_LIN:       # %bb.0:
+; AVX512F_32_LIN-NEXT:    subl $20, %esp
+; AVX512F_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32_LIN-NEXT:    vmovsd {{.*#+}} xmm1 = mem[0],zero
+; AVX512F_32_LIN-NEXT:    vcmpltsd %xmm1, %xmm0, %k1
+; AVX512F_32_LIN-NEXT:    vsubsd %xmm1, %xmm0, %xmm2
+; AVX512F_32_LIN-NEXT:    vmovsd %xmm0, %xmm0, %xmm2 {%k1}
+; AVX512F_32_LIN-NEXT:    vmovsd %xmm2, {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fldl {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_LIN-NEXT:    xorl %edx, %edx
+; AVX512F_32_LIN-NEXT:    vucomisd %xmm0, %xmm1
+; AVX512F_32_LIN-NEXT:    setbe %dl
+; AVX512F_32_LIN-NEXT:    shll $31, %edx
+; AVX512F_32_LIN-NEXT:    xorl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_LIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_LIN-NEXT:    addl $20, %esp
+; AVX512F_32_LIN-NEXT:    retl
+;
 ; SSE3_32_WIN-LABEL: d_to_u64:
 ; SSE3_32_WIN:       # %bb.0:
 ; SSE3_32_WIN-NEXT:    pushl %ebp
 ; SSE3_32_WIN-NEXT:    movl %esp, %ebp
 ; SSE3_32_WIN-NEXT:    andl $-8, %esp
 ; SSE3_32_WIN-NEXT:    subl $16, %esp
 ; SSE3_32_WIN-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; SSE3_32_WIN-NEXT:    movsd {{.*#+}} xmm1 = mem[0],zero

 ; X87_LIN-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X87_LIN-NEXT:    addl $20, %esp
 ; X87_LIN-NEXT:    retl
   %r = fptoui double %a to i64
   ret i64 %r
 }
 
 define i64 @d_to_s64(double %a) nounwind {
-; AVX512_32_WIN-LABEL: d_to_s64:
-; AVX512_32_WIN:       # %bb.0:
-; AVX512_32_WIN-NEXT:    pushl %ebp
-; AVX512_32_WIN-NEXT:    movl %esp, %ebp
-; AVX512_32_WIN-NEXT:    andl $-8, %esp
-; AVX512_32_WIN-NEXT:    subl $16, %esp
-; AVX512_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32_WIN-NEXT:    vmovsd %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fldl {{[0-9]+}}(%esp)
-; AVX512_32_WIN-NEXT:    fisttpll (%esp)
-; AVX512_32_WIN-NEXT:    movl (%esp), %eax
-; AVX512_32_WIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; AVX512_32_WIN-NEXT:    movl %ebp, %esp
-; AVX512_32_WIN-NEXT:    popl %ebp
-; AVX512_32_WIN-NEXT:    retl
-;
-; AVX512_32_LIN-LABEL: d_to_s64:
-; AVX512_32_LIN:       # %bb.0:
-; AVX512_32_LIN-NEXT:    subl $20, %esp
-; AVX512_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32_LIN-NEXT:    vmovsd %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fldl {{[0-9]+}}(%esp)
-; AVX512_32_LIN-NEXT:    fisttpll (%esp)
-; AVX512_32_LIN-NEXT:    movl (%esp), %eax
-; AVX512_32_LIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; AVX512_32_LIN-NEXT:    addl $20, %esp
-; AVX512_32_LIN-NEXT:    retl
+; AVX512DQVL_32_WIN-LABEL: d_to_s64:
+; AVX512DQVL_32_WIN:       # %bb.0:
+; AVX512DQVL_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32_WIN-NEXT:    vcvttpd2qq %ymm0, %ymm0
+; AVX512DQVL_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_WIN-NEXT:    vzeroupper
+; AVX512DQVL_32_WIN-NEXT:    retl
+;
+; AVX512DQVL_32_LIN-LABEL: d_to_s64:
+; AVX512DQVL_32_LIN:       # %bb.0:
+; AVX512DQVL_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32_LIN-NEXT:    vcvttpd2qq %ymm0, %ymm0
+; AVX512DQVL_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQVL_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQVL_32_LIN-NEXT:    vzeroupper
+; AVX512DQVL_32_LIN-NEXT:    retl
 ;
 ; AVX512_64-LABEL: d_to_s64:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvttsd2si %xmm0, %rax
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32_WIN-LABEL: d_to_s64:
+; AVX512DQ_32_WIN:       # %bb.0:
+; AVX512DQ_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32_WIN-NEXT:    vcvttpd2qq %zmm0, %zmm0
+; AVX512DQ_32_WIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_WIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_WIN-NEXT:    vzeroupper
+; AVX512DQ_32_WIN-NEXT:    retl
+;
+; AVX512DQ_32_LIN-LABEL: d_to_s64:
+; AVX512DQ_32_LIN:       # %bb.0:
+; AVX512DQ_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32_LIN-NEXT:    vcvttpd2qq %zmm0, %zmm0
+; AVX512DQ_32_LIN-NEXT:    vmovd %xmm0, %eax
+; AVX512DQ_32_LIN-NEXT:    vpextrd $1, %xmm0, %edx
+; AVX512DQ_32_LIN-NEXT:    vzeroupper
+; AVX512DQ_32_LIN-NEXT:    retl
+;
+; AVX512F_32_WIN-LABEL: d_to_s64:
+; AVX512F_32_WIN:       # %bb.0:
+; AVX512F_32_WIN-NEXT:    pushl %ebp
+; AVX512F_32_WIN-NEXT:    movl %esp, %ebp
+; AVX512F_32_WIN-NEXT:    andl $-8, %esp
+; AVX512F_32_WIN-NEXT:    subl $16, %esp
+; AVX512F_32_WIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32_WIN-NEXT:    vmovsd %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fldl {{[0-9]+}}(%esp)
+; AVX512F_32_WIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_WIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_WIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_WIN-NEXT:    movl %ebp, %esp
+; AVX512F_32_WIN-NEXT:    popl %ebp
+; AVX512F_32_WIN-NEXT:    retl
+;
+; AVX512F_32_LIN-LABEL: d_to_s64:
+; AVX512F_32_LIN:       # %bb.0:
+; AVX512F_32_LIN-NEXT:    subl $20, %esp
+; AVX512F_32_LIN-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32_LIN-NEXT:    vmovsd %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fldl {{[0-9]+}}(%esp)
+; AVX512F_32_LIN-NEXT:    fisttpll (%esp)
+; AVX512F_32_LIN-NEXT:    movl (%esp), %eax
+; AVX512F_32_LIN-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; AVX512F_32_LIN-NEXT:    addl $20, %esp
+; AVX512F_32_LIN-NEXT:    retl
+;
 ; SSE3_32_WIN-LABEL: d_to_s64:
 ; SSE3_32_WIN:       # %bb.0:
 ; SSE3_32_WIN-NEXT:    pushl %ebp
 ; SSE3_32_WIN-NEXT:    movl %esp, %ebp
 ; SSE3_32_WIN-NEXT:    andl $-8, %esp
 ; SSE3_32_WIN-NEXT:    subl $16, %esp
 ; SSE3_32_WIN-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; SSE3_32_WIN-NEXT:    movsd %xmm0, {{[0-9]+}}(%esp)

test/CodeGen/X86/scalar-int-to-fp.ll

 ; CHECK64-NEXT:    movl %edi, -{{[0-9]+}}(%rsp)
 ; CHECK64-NEXT:    fildl -{{[0-9]+}}(%rsp)
 ; CHECK64-NEXT:    retq
   %r = sitofp i32 %a to x86_fp80
   ret x86_fp80 %r
 }
 
 define float @u64_to_f(i64 %a) nounwind {
-; AVX512_32-LABEL: u64_to_f:
-; AVX512_32:       # %bb.0:
-; AVX512_32-NEXT:    pushl %ebp
-; AVX512_32-NEXT:    movl %esp, %ebp
-; AVX512_32-NEXT:    andl $-8, %esp
-; AVX512_32-NEXT:    subl $16, %esp
-; AVX512_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32-NEXT:    vmovlps %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    xorl %eax, %eax
-; AVX512_32-NEXT:    cmpl $0, 12(%ebp)
-; AVX512_32-NEXT:    setns %al
-; AVX512_32-NEXT:    fildll {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    fadds {{\.LCPI.*}}(,%eax,4)
-; AVX512_32-NEXT:    fstps {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX512_32-NEXT:    vmovss %xmm0, (%esp)
-; AVX512_32-NEXT:    flds (%esp)
-; AVX512_32-NEXT:    movl %ebp, %esp
-; AVX512_32-NEXT:    popl %ebp
-; AVX512_32-NEXT:    retl
+; AVX512DQVL_32-LABEL: u64_to_f:
+; AVX512DQVL_32:       # %bb.0:
+; AVX512DQVL_32-NEXT:    pushl %eax
+; AVX512DQVL_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32-NEXT:    vcvtuqq2ps %ymm0, %xmm0
+; AVX512DQVL_32-NEXT:    vmovss %xmm0, (%esp)
+; AVX512DQVL_32-NEXT:    flds (%esp)
+; AVX512DQVL_32-NEXT:    popl %eax
+; AVX512DQVL_32-NEXT:    vzeroupper
+; AVX512DQVL_32-NEXT:    retl
 ;
 ; AVX512_64-LABEL: u64_to_f:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvtusi2ssq %rdi, %xmm0, %xmm0
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32-LABEL: u64_to_f:
+; AVX512DQ_32:       # %bb.0:
+; AVX512DQ_32-NEXT:    pushl %eax
+; AVX512DQ_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32-NEXT:    vcvtuqq2ps %zmm0, %ymm0
+; AVX512DQ_32-NEXT:    vmovss %xmm0, (%esp)
+; AVX512DQ_32-NEXT:    flds (%esp)
+; AVX512DQ_32-NEXT:    popl %eax
+; AVX512DQ_32-NEXT:    vzeroupper
+; AVX512DQ_32-NEXT:    retl
+;
+; AVX512F_32-LABEL: u64_to_f:
+; AVX512F_32:       # %bb.0:
+; AVX512F_32-NEXT:    pushl %ebp
+; AVX512F_32-NEXT:    movl %esp, %ebp
+; AVX512F_32-NEXT:    andl $-8, %esp
+; AVX512F_32-NEXT:    subl $16, %esp
+; AVX512F_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32-NEXT:    vmovlps %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    xorl %eax, %eax
+; AVX512F_32-NEXT:    cmpl $0, 12(%ebp)
+; AVX512F_32-NEXT:    setns %al
+; AVX512F_32-NEXT:    fildll {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    fadds {{\.LCPI.*}}(,%eax,4)
+; AVX512F_32-NEXT:    fstps {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512F_32-NEXT:    vmovss %xmm0, (%esp)
+; AVX512F_32-NEXT:    flds (%esp)
+; AVX512F_32-NEXT:    movl %ebp, %esp
+; AVX512F_32-NEXT:    popl %ebp
+; AVX512F_32-NEXT:    retl
+;
 ; SSE2_32-LABEL: u64_to_f:
 ; SSE2_32:       # %bb.0:
 ; SSE2_32-NEXT:    pushl %ebp
 ; SSE2_32-NEXT:    movl %esp, %ebp
 ; SSE2_32-NEXT:    andl $-8, %esp
 ; SSE2_32-NEXT:    subl $16, %esp
 ; SSE2_32-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; SSE2_32-NEXT:    movlps %xmm0, {{[0-9]+}}(%esp)

 ; X87:       # %bb.0:
 ; X87-NEXT:    fildll {{[0-9]+}}(%esp)
 ; X87-NEXT:    retl
   %r = sitofp i64 %a to float
   ret float %r
 }
 
 define float @s64_to_f_2(i64 %a) nounwind {
-; AVX512_32-LABEL: s64_to_f_2:
-; AVX512_32:       # %bb.0:
-; AVX512_32-NEXT:    pushl %ebp
-; AVX512_32-NEXT:    movl %esp, %ebp
-; AVX512_32-NEXT:    andl $-8, %esp
-; AVX512_32-NEXT:    subl $16, %esp
-; AVX512_32-NEXT:    movl 8(%ebp), %eax
-; AVX512_32-NEXT:    movl 12(%ebp), %ecx
-; AVX512_32-NEXT:    addl $5, %eax
-; AVX512_32-NEXT:    adcl $0, %ecx
-; AVX512_32-NEXT:    vmovd %eax, %xmm0
-; AVX512_32-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
-; AVX512_32-NEXT:    vmovq %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    fildll {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    fstps {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    flds {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    movl %ebp, %esp
-; AVX512_32-NEXT:    popl %ebp
-; AVX512_32-NEXT:    retl
+; AVX512DQVL_32-LABEL: s64_to_f_2:
+; AVX512DQVL_32:       # %bb.0:
+; AVX512DQVL_32-NEXT:    pushl %eax
+; AVX512DQVL_32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; AVX512DQVL_32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; AVX512DQVL_32-NEXT:    addl $5, %eax
+; AVX512DQVL_32-NEXT:    adcl $0, %ecx
+; AVX512DQVL_32-NEXT:    vmovd %ecx, %xmm0
+; AVX512DQVL_32-NEXT:    vmovd %eax, %xmm1
+; AVX512DQVL_32-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
+; AVX512DQVL_32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; AVX512DQVL_32-NEXT:    vcvtqq2ps %ymm0, %xmm0
+; AVX512DQVL_32-NEXT:    vmovss %xmm0, (%esp)
+; AVX512DQVL_32-NEXT:    flds (%esp)
+; AVX512DQVL_32-NEXT:    popl %eax
+; AVX512DQVL_32-NEXT:    vzeroupper
+; AVX512DQVL_32-NEXT:    retl
 ;
 ; AVX512_64-LABEL: s64_to_f_2:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    addq $5, %rdi
 ; AVX512_64-NEXT:    vcvtsi2ssq %rdi, %xmm0, %xmm0
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32-LABEL: s64_to_f_2:
+; AVX512DQ_32:       # %bb.0:
+; AVX512DQ_32-NEXT:    pushl %eax
+; AVX512DQ_32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; AVX512DQ_32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; AVX512DQ_32-NEXT:    addl $5, %eax
+; AVX512DQ_32-NEXT:    adcl $0, %ecx
+; AVX512DQ_32-NEXT:    vmovd %ecx, %xmm0
+; AVX512DQ_32-NEXT:    vmovd %eax, %xmm1
+; AVX512DQ_32-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
+; AVX512DQ_32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; AVX512DQ_32-NEXT:    vcvtqq2ps %zmm0, %ymm0
+; AVX512DQ_32-NEXT:    vmovss %xmm0, (%esp)
+; AVX512DQ_32-NEXT:    flds (%esp)
+; AVX512DQ_32-NEXT:    popl %eax
+; AVX512DQ_32-NEXT:    vzeroupper
+; AVX512DQ_32-NEXT:    retl
+;
+; AVX512F_32-LABEL: s64_to_f_2:
+; AVX512F_32:       # %bb.0:
+; AVX512F_32-NEXT:    pushl %ebp
+; AVX512F_32-NEXT:    movl %esp, %ebp
+; AVX512F_32-NEXT:    andl $-8, %esp
+; AVX512F_32-NEXT:    subl $16, %esp
+; AVX512F_32-NEXT:    movl 8(%ebp), %eax
+; AVX512F_32-NEXT:    movl 12(%ebp), %ecx
+; AVX512F_32-NEXT:    addl $5, %eax
+; AVX512F_32-NEXT:    adcl $0, %ecx
+; AVX512F_32-NEXT:    vmovd %eax, %xmm0
+; AVX512F_32-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
+; AVX512F_32-NEXT:    vmovq %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    fildll {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    fstps {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    flds {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    movl %ebp, %esp
+; AVX512F_32-NEXT:    popl %ebp
+; AVX512F_32-NEXT:    retl
+;
 ; SSE2_32-LABEL: s64_to_f_2:
 ; SSE2_32:       # %bb.0:
 ; SSE2_32-NEXT:    pushl %ebp
 ; SSE2_32-NEXT:    movl %esp, %ebp
 ; SSE2_32-NEXT:    andl $-8, %esp
 ; SSE2_32-NEXT:    subl $16, %esp
 ; SSE2_32-NEXT:    movl 8(%ebp), %eax
 ; SSE2_32-NEXT:    movl 12(%ebp), %ecx

 ; X87-NEXT:    popl %ebp
 ; X87-NEXT:    retl
   %a1 = add i64 %a, 5
   %r = sitofp i64 %a1 to float
   ret float %r
 }
 
 define double @u64_to_d(i64 %a) nounwind {
-; AVX512_32-LABEL: u64_to_d:
-; AVX512_32:       # %bb.0:
-; AVX512_32-NEXT:    pushl %ebp
-; AVX512_32-NEXT:    movl %esp, %ebp
-; AVX512_32-NEXT:    andl $-8, %esp
-; AVX512_32-NEXT:    subl $8, %esp
-; AVX512_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX512_32-NEXT:    vunpcklps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[1],mem[1]
-; AVX512_32-NEXT:    vsubpd {{\.LCPI.*}}, %xmm0, %xmm0
-; AVX512_32-NEXT:    vhaddpd %xmm0, %xmm0, %xmm0
-; AVX512_32-NEXT:    vmovlpd %xmm0, (%esp)
-; AVX512_32-NEXT:    fldl (%esp)
-; AVX512_32-NEXT:    movl %ebp, %esp
-; AVX512_32-NEXT:    popl %ebp
-; AVX512_32-NEXT:    retl
+; AVX512DQVL_32-LABEL: u64_to_d:
+; AVX512DQVL_32:       # %bb.0:
+; AVX512DQVL_32-NEXT:    pushl %ebp
+; AVX512DQVL_32-NEXT:    movl %esp, %ebp
+; AVX512DQVL_32-NEXT:    andl $-8, %esp
+; AVX512DQVL_32-NEXT:    subl $8, %esp
+; AVX512DQVL_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQVL_32-NEXT:    vcvtuqq2pd %ymm0, %ymm0
+; AVX512DQVL_32-NEXT:    vmovlps %xmm0, (%esp)
+; AVX512DQVL_32-NEXT:    fldl (%esp)
+; AVX512DQVL_32-NEXT:    movl %ebp, %esp
+; AVX512DQVL_32-NEXT:    popl %ebp
+; AVX512DQVL_32-NEXT:    vzeroupper
+; AVX512DQVL_32-NEXT:    retl
 ;
 ; AVX512_64-LABEL: u64_to_d:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    vcvtusi2sdq %rdi, %xmm0, %xmm0
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32-LABEL: u64_to_d:
+; AVX512DQ_32:       # %bb.0:
+; AVX512DQ_32-NEXT:    pushl %ebp
+; AVX512DQ_32-NEXT:    movl %esp, %ebp
+; AVX512DQ_32-NEXT:    andl $-8, %esp
+; AVX512DQ_32-NEXT:    subl $8, %esp
+; AVX512DQ_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512DQ_32-NEXT:    vcvtuqq2pd %zmm0, %zmm0
+; AVX512DQ_32-NEXT:    vmovlps %xmm0, (%esp)
+; AVX512DQ_32-NEXT:    fldl (%esp)
+; AVX512DQ_32-NEXT:    movl %ebp, %esp
+; AVX512DQ_32-NEXT:    popl %ebp
+; AVX512DQ_32-NEXT:    vzeroupper
+; AVX512DQ_32-NEXT:    retl
+;
+; AVX512F_32-LABEL: u64_to_d:
+; AVX512F_32:       # %bb.0:
+; AVX512F_32-NEXT:    pushl %ebp
+; AVX512F_32-NEXT:    movl %esp, %ebp
+; AVX512F_32-NEXT:    andl $-8, %esp
+; AVX512F_32-NEXT:    subl $8, %esp
+; AVX512F_32-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX512F_32-NEXT:    vunpcklps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[1],mem[1]
+; AVX512F_32-NEXT:    vsubpd {{\.LCPI.*}}, %xmm0, %xmm0
+; AVX512F_32-NEXT:    vhaddpd %xmm0, %xmm0, %xmm0
+; AVX512F_32-NEXT:    vmovlpd %xmm0, (%esp)
+; AVX512F_32-NEXT:    fldl (%esp)
+; AVX512F_32-NEXT:    movl %ebp, %esp
+; AVX512F_32-NEXT:    popl %ebp
+; AVX512F_32-NEXT:    retl
+;
 ; SSE2_32-LABEL: u64_to_d:
 ; SSE2_32:       # %bb.0:
 ; SSE2_32-NEXT:    pushl %ebp
 ; SSE2_32-NEXT:    movl %esp, %ebp
 ; SSE2_32-NEXT:    andl $-8, %esp
 ; SSE2_32-NEXT:    subl $8, %esp
 ; SSE2_32-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; SSE2_32-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],mem[0],xmm0[1],mem[1]

 ; X87:       # %bb.0:
 ; X87-NEXT:    fildll {{[0-9]+}}(%esp)
 ; X87-NEXT:    retl
   %r = sitofp i64 %a to double
   ret double %r
 }
 
 define double @s64_to_d_2(i64 %a) nounwind {
-; AVX512_32-LABEL: s64_to_d_2:
-; AVX512_32:       # %bb.0:
-; AVX512_32-NEXT:    pushl %ebp
-; AVX512_32-NEXT:    movl %esp, %ebp
-; AVX512_32-NEXT:    andl $-8, %esp
-; AVX512_32-NEXT:    subl $16, %esp
-; AVX512_32-NEXT:    movl 8(%ebp), %eax
-; AVX512_32-NEXT:    movl 12(%ebp), %ecx
-; AVX512_32-NEXT:    addl $5, %eax
-; AVX512_32-NEXT:    adcl $0, %ecx
-; AVX512_32-NEXT:    vmovd %eax, %xmm0
-; AVX512_32-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
-; AVX512_32-NEXT:    vmovq %xmm0, {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    fildll {{[0-9]+}}(%esp)
-; AVX512_32-NEXT:    fstpl (%esp)
-; AVX512_32-NEXT:    fldl (%esp)
-; AVX512_32-NEXT:    movl %ebp, %esp
-; AVX512_32-NEXT:    popl %ebp
-; AVX512_32-NEXT:    retl
+; AVX512DQVL_32-LABEL: s64_to_d_2:
+; AVX512DQVL_32:       # %bb.0:
+; AVX512DQVL_32-NEXT:    pushl %ebp
+; AVX512DQVL_32-NEXT:    movl %esp, %ebp
+; AVX512DQVL_32-NEXT:    andl $-8, %esp
+; AVX512DQVL_32-NEXT:    subl $8, %esp
+; AVX512DQVL_32-NEXT:    movl 8(%ebp), %eax
+; AVX512DQVL_32-NEXT:    movl 12(%ebp), %ecx
+; AVX512DQVL_32-NEXT:    addl $5, %eax
+; AVX512DQVL_32-NEXT:    adcl $0, %ecx
+; AVX512DQVL_32-NEXT:    vmovd %ecx, %xmm0
+; AVX512DQVL_32-NEXT:    vmovd %eax, %xmm1
+; AVX512DQVL_32-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
+; AVX512DQVL_32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; AVX512DQVL_32-NEXT:    vcvtqq2pd %ymm0, %ymm0
+; AVX512DQVL_32-NEXT:    vmovlps %xmm0, (%esp)
+; AVX512DQVL_32-NEXT:    fldl (%esp)
+; AVX512DQVL_32-NEXT:    movl %ebp, %esp
+; AVX512DQVL_32-NEXT:    popl %ebp
+; AVX512DQVL_32-NEXT:    vzeroupper
+; AVX512DQVL_32-NEXT:    retl
 ;
 ; AVX512_64-LABEL: s64_to_d_2:
 ; AVX512_64:       # %bb.0:
 ; AVX512_64-NEXT:    addq $5, %rdi
 ; AVX512_64-NEXT:    vcvtsi2sdq %rdi, %xmm0, %xmm0
 ; AVX512_64-NEXT:    retq
 ;
+; AVX512DQ_32-LABEL: s64_to_d_2:
+; AVX512DQ_32:       # %bb.0:
+; AVX512DQ_32-NEXT:    pushl %ebp
+; AVX512DQ_32-NEXT:    movl %esp, %ebp
+; AVX512DQ_32-NEXT:    andl $-8, %esp
+; AVX512DQ_32-NEXT:    subl $8, %esp
+; AVX512DQ_32-NEXT:    movl 8(%ebp), %eax
+; AVX512DQ_32-NEXT:    movl 12(%ebp), %ecx
+; AVX512DQ_32-NEXT:    addl $5, %eax
+; AVX512DQ_32-NEXT:    adcl $0, %ecx
+; AVX512DQ_32-NEXT:    vmovd %ecx, %xmm0
+; AVX512DQ_32-NEXT:    vmovd %eax, %xmm1
+; AVX512DQ_32-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm1[0],xmm0[0],xmm1[1],xmm0[1]
+; AVX512DQ_32-NEXT:    vmovq {{.*#+}} xmm0 = xmm0[0],zero
+; AVX512DQ_32-NEXT:    vcvtqq2pd %zmm0, %zmm0
+; AVX512DQ_32-NEXT:    vmovlps %xmm0, (%esp)
+; AVX512DQ_32-NEXT:    fldl (%esp)
+; AVX512DQ_32-NEXT:    movl %ebp, %esp
+; AVX512DQ_32-NEXT:    popl %ebp
+; AVX512DQ_32-NEXT:    vzeroupper
+; AVX512DQ_32-NEXT:    retl
+;
+; AVX512F_32-LABEL: s64_to_d_2:
+; AVX512F_32:       # %bb.0:
+; AVX512F_32-NEXT:    pushl %ebp
+; AVX512F_32-NEXT:    movl %esp, %ebp
+; AVX512F_32-NEXT:    andl $-8, %esp
+; AVX512F_32-NEXT:    subl $16, %esp
+; AVX512F_32-NEXT:    movl 8(%ebp), %eax
+; AVX512F_32-NEXT:    movl 12(%ebp), %ecx
+; AVX512F_32-NEXT:    addl $5, %eax
+; AVX512F_32-NEXT:    adcl $0, %ecx
+; AVX512F_32-NEXT:    vmovd %eax, %xmm0
+; AVX512F_32-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
+; AVX512F_32-NEXT:    vmovq %xmm0, {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    fildll {{[0-9]+}}(%esp)
+; AVX512F_32-NEXT:    fstpl (%esp)
+; AVX512F_32-NEXT:    fldl (%esp)
+; AVX512F_32-NEXT:    movl %ebp, %esp
+; AVX512F_32-NEXT:    popl %ebp
+; AVX512F_32-NEXT:    retl
+;
 ; SSE2_32-LABEL: s64_to_d_2:
 ; SSE2_32:       # %bb.0:
 ; SSE2_32-NEXT:    pushl %ebp
 ; SSE2_32-NEXT:    movl %esp, %ebp
 ; SSE2_32-NEXT:    andl $-8, %esp
 ; SSE2_32-NEXT:    subl $16, %esp
 ; SSE2_32-NEXT:    movl 8(%ebp), %eax
 ; SSE2_32-NEXT:    movl 12(%ebp), %ecx