[X86] Combining CMOVs with [ANY,SIGN,ZERO]_EXTEND for cases where CMOV has constant arguments

[aivchenk]

Summary

Combine CMOV[i16]<-[SIGN,ZERO,ANY]_EXTEND to [i32,i64] into CMOV[i32,i64].
One example of where it is useful is:

before (20 bytes)
<foo>:
test $0x1,%dil
mov $0x307e,%ax
mov $0xffff,%cx
cmovne %ax,%cx
movzwl %cx,%eax
retq

after (18 bytes)
<foo>:
test $0x1,%dil
mov $0x307e,%ecx
mov $0xffff,%eax
cmovne %ecx,%eax
retq

[RKSimon]

Please don't forget to include llvm-commits as a subscriber

[spatel]

test/CodeGen/X86/cmov-promotion.ll

1–2

1. Please add this file to trunk with the utils script auto-generated baseline checks, so we just see the diffs in this patch.
2. It's not clear to me what the 2nd run is showing. That target has cmov, but it gets converted to test+branch later?
3. Assuming the 2nd run is necessary, why are there no MCU checks for the 2nd test?
4. Should there be an spromotion_16_to_64 for completeness?

[spatel]

It's worth noting that we could convert any select of constants to math ops. I was planning a follow-up to https://reviews.llvm.org/rL310717 that would do that, but there might be cases where we would prefer to select cmov? If not, that would make this patch obsolete.

[spatel]

In D36711#842026, @spatel wrote:

It's worth noting that we could convert any select of constants to math ops. I was planning a follow-up to https://reviews.llvm.org/rL310717 that would do that, but there might be cases where we would prefer to select cmov? If not, that would make this patch obsolete.

GCC has already implemented this:
https://godbolt.org/g/i5DVLe

[aivchenk]

Thank you for your comments, I updated the patch to address them:

1. Done
2. MCU target does not have CMOV instruction so it is always converted into test+branch. I firstly cared about it - that's the reason why it is here
3. Just a miss. It is fixed
4. Ditto

About converting into math ops.. I think that in most cases my patch would be obsolete then. The only reason I see why it would be not obsolete is: assume that CMOV with constants is part of a CMOV-group (as defined in X86CmovConversion.cpp: consecutive/having the same CC) and the decision taken by x86-cmov-converter is to transform this CMOV-group into branch. Since branch is generated anyways now, basically this "constant" CMOV is transformed into two addition MOVs, which should be more effective than converting it to three ops math logic

[spatel]

In D36711#842468, @aivchenk wrote:

Thank you for your comments, I updated the patch to address them:

1. Done
2. MCU target does not have CMOV instruction so it is always converted into test+branch. I firstly cared about it - that's the reason why it is here

I don't have a good understanding of this path then. Why are we generating x86-specific cmov nodes for a target that doesn't support cmov? I would think these should remain as select nodes to avoid creating a flags result that won't exist, but I must be missing something?

About converting into math ops.. I think that in most cases my patch would be obsolete then. The only reason I see why it would be not obsolete is: assume that CMOV with constants is part of a CMOV-group (as defined in X86CmovConversion.cpp: consecutive/having the same CC) and the decision taken by x86-cmov-converter is to transform this CMOV-group into branch. Since branch is generated anyways now, basically this "constant" CMOV is transformed into two addition MOVs, which should be more effective than converting it to three ops math logic

That's a good point - if the compare has multiple uses, we probably shouldn't convert to math. This may already be wrong in the existing transforms. It's also possible that a cmov sequence is faster (the dependency chain is shorter), but I think a CPU would have to be able to execute cmov at the same speed/throughput as the simple ops for that to be true.

[craig.topper]

I believe X86 lowers all selects of any type to X86ISD::CMOV. If the target supports CMOV for a type, isel will emit a real CMOV instruction. If the target doesn't support CMOV or the type isn't supported by CMOV we emit a CMOV peusdo instruction which gets custom inserted to control flow via EmitLoweredSelect.

[spatel]

I checked in the tests from cmov-promotion.ll at rL311052, so we'd see the baseline codegen. Can you rebase after that?

I put a couple of style nit comments inline. I think this patch makes sense, but would appreciate if another reviewer can take a look too to make sure the non-cmov code is better too.

lib/Target/X86/X86ISelLowering.cpp

34359–34361

It's a matter of taste, but seems more typical to shrink the function signature to just (SDNode *N, SelectionDAG &DAG) and then extract the VT, DL, and opcode in local variables.

Another style note: it's completely inconsistent, but I think we prefer "DL" now that variables are supposed to be capitalized.

34379–34382

Could just use the 4 operand override of getNode here to avoid the explicit SmallVector?

[aivchenk]

Thank you very much for your comments and for submitting the baseline.

I did the rebase and addressed your comments in the new diff

[spatel]

Thanks for the updates. I have no other suggestions.

[aivchenk]

Kind ping :)

[RKSimon]

Minor style comments.

lib/Target/X86/X86ISelLowering.cpp

34362

(style)

if (CMovN.getOpcode() != X86ISD::CMOV)
  return SDValue();

34379

(style) early-out:

  if (!DoPromoteCMOV)
    return SDValue();

  CMovOp0 = DAG.getNode(ExtendOpcode, DL, TargetVT, CMovOp0);
  CMovOp1 = DAG.getNode(ExtendOpcode, DL, TargetVT, CMovOp1);
  return DAG.getNode(X86ISD::CMOV, DL, TargetVT, CMovOp0, CMovOp1,
                                        CMovN.getOperand(2), CMovN.getOperand(3));
}

test/CodeGen/X86/cmov-promotion.ll

98

Please can you add 32->64 tests to check that they aren't happening? Maybe 8->X test as well?

[aivchenk]

Thanks for your comments, I attached the updated patch

[RKSimon]

No more comments from me - anybody else (@spatel)?

[spatel]

LGTM.

Side note about my earlier comment about transforming select-of-constants to bitwise logic: according to Agner's tables, AMD Zen can do cmov in a single uop and at full throughput, so that transform won't make sense in general.

[RKSimon]

@aivchenk Are you intending to commit this?

[aivchenk]

yep, sorry for the delay

lib/Target/X86/X86ISelLowering.cpp

static SDValue getDivRem8(SDNode *N, SelectionDAG &DAG) {
   auto DivRemOpcode = OpcodeN0 == ISD::SDIVREM ? X86ISD::SDIVREM8_SEXT_HREG
                                                : X86ISD::UDIVREM8_ZEXT_HREG;
   SDValue R = DAG.getNode(DivRemOpcode, SDLoc(N), NodeTys, N0.getOperand(0),
                           N0.getOperand(1));
   DAG.ReplaceAllUsesOfValueWith(N0.getValue(0), R.getValue(0));
   return R.getValue(1);
 }
 
+// If we face {ANY,SIGN,ZERO}_EXTEND that is applied to a CMOV with constant
+// operands and the result of CMOV is not used anywhere else - promote CMOV
+// itself instead of promoting its result. This could be beneficial, because:
+//     1) X86TargetLowering::EmitLoweredSelect later can do merging of two
+//        (or more) pseudo-CMOVs only when they go one-after-another and
+//        getting rid of result extension code after CMOV will help that.
+//     2) Promotion of constant CMOV arguments is free, hence the
+//        {ANY,SIGN,ZERO}_EXTEND will just be deleted.
+//     3) 16-bit CMOV encoding is 4 bytes, 32-bit CMOV is 3-byte, so this
+//        promotion is also good in terms of code-size.
+//        (64-bit CMOV is 4-bytes, that's why we don't do 32-bit => 64-bit
+//         promotion).
+static SDValue combineToExtendCMOV(SDValue N, EVT TargetVT,
+                                   unsigned ExtendOpcode, const SDLoc &dl,
+                                   SelectionDAG &DAG) {
+  unsigned Opcode = N.getOpcode();
+  if (Opcode != X86ISD::CMOV)
+    return SDValue();
+
+  EVT VT = N.getValueType();
+  SDValue CMovOp0 = N.getOperand(0);
+  SDValue CMovOp1 = N.getOperand(1);
+
+  bool DoPromoteCMOV =
+      (VT == MVT::i16 && (TargetVT == MVT::i32 || TargetVT == MVT::i64)) &&
+      N.hasOneUse() && (isa<ConstantSDNode>(CMovOp0.getNode()) &&
+                        isa<ConstantSDNode>(CMovOp1.getNode()));
+
+  if (DoPromoteCMOV) {
+    CMovOp0 = DAG.getNode(ExtendOpcode, dl, TargetVT, CMovOp0);
+    CMovOp1 = DAG.getNode(ExtendOpcode, dl, TargetVT, CMovOp1);
+
+    SmallVector<SDValue, 4> Ops = {CMovOp0, CMovOp1, N.getOperand(2),
+                                   N.getOperand(3)};
+
+    return DAG.getNode(X86ISD::CMOV, dl, TargetVT, Ops);
+  }
+  return SDValue();
+}
+
 /// Convert a SEXT or ZEXT of a vector to a SIGN_EXTEND_VECTOR_INREG or
 /// ZERO_EXTEND_VECTOR_INREG, this requires the splitting (or concatenating
 /// with UNDEFs) of the input to vectors of the same size as the target type
 /// which then extends the lowest elements.
 static SDValue combineToExtendVectorInReg(SDNode *N, SelectionDAG &DAG,
                                           TargetLowering::DAGCombinerInfo &DCI,
                                           const X86Subtarget &Subtarget) {
   unsigned Opcode = N->getOpcode();
static SDValue combineSext(SDNode *N, SelectionDAG &DAG,
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
   EVT InVT = N0.getValueType();
   SDLoc DL(N);
 
   if (SDValue DivRem8 = getDivRem8(N, DAG))
     return DivRem8;
 
+  if (SDValue NewCMov = combineToExtendCMOV(N0, VT, N->getOpcode(), DL, DAG))
+    return NewCMov;
+
   if (!DCI.isBeforeLegalizeOps()) {
     if (InVT == MVT::i1) {
       SDValue Zero = DAG.getConstant(0, DL, VT);
       SDValue AllOnes = DAG.getAllOnesConstant(DL, VT);
       return DAG.getSelect(DL, VT, N0, AllOnes, Zero);
     }
     return SDValue();
   }
if (N0.getOpcode() == ISD::TRUNCATE &&
     if (N00.getOpcode() == X86ISD::SETCC_CARRY) {
       return DAG.getNode(ISD::AND, dl, VT,
                          DAG.getNode(X86ISD::SETCC_CARRY, dl, VT,
                                      N00.getOperand(0), N00.getOperand(1)),
                          DAG.getConstant(1, dl, VT));
     }
   }
 
+  if (SDValue NewCMov = combineToExtendCMOV(N0, VT, N->getOpcode(), dl, DAG))
+    return NewCMov;
+
   if (SDValue V = combineToExtendVectorInReg(N, DAG, DCI, Subtarget))
     return V;
 
   if (VT.is256BitVector())
     if (SDValue R = WidenMaskArithmetic(N, DAG, DCI, Subtarget))
       return R;
 
   if (SDValue DivRem8 = getDivRem8(N, DAG))

test/CodeGen/X86/cmov-promotion.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-linux | FileCheck %s --check-prefix=CHECK
+; RUN: llc < %s -mtriple=i386-intel-elfiamcu  | FileCheck %s --check-prefix=MCU
+
+define i32 @cmov_zpromotion_16_to_32(i1 %c) {
+
+; CHECK-LABEL: cmov_zpromotion_16_to_32:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    testb   $1, %dil
+; CHECK-NEXT:    movl    $12414, %ecx
+; CHECK-NEXT:    movl    $65535, %eax
+; CHECK-NEXT:    cmovnel %ecx, %eax
+; CHECK-NEXT:    retq
+
+; MCU-LABEL: cmov_zpromotion_16_to_32:
+; MCU:       # BB#0:
+; MCU-NEXT:    testb   $1, %al
+; MCU-NEXT:    movl    $12414, %eax
+; MCU-NEXT:    jne     .LBB0_2
+; MCU-NEXT:  # BB#1:
+; MCU-NEXT:    movl    $65535, %eax
+; MCU-NEXT:  .LBB0_2:
+; MCU-NEXT:    retl
+entry:
+  %0 = select i1 %c, i16 12414, i16 -1
+  %ret = zext i16 %0 to i32
+  ret i32 %ret
+}
+
+define i64 @cmov_zpromotion_16_to_64(i1 %c) {
+
+; CHECK-LABEL: cmov_zpromotion_16_to_64:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    testb   $1, %dil
+; CHECK-NEXT:    movl    $12414, %ecx
+; CHECK-NEXT:    movl    $65535, %eax
+; CHECK-NEXT:    cmovneq %rcx, %rax
+; CHECK-NEXT:    retq
+
+entry:
+  %0 = select i1 %c, i16 12414, i16 -1
+  %ret = zext i16 %0 to i64
+  ret i64 %ret
+}
+
+define i32 @cmov_spromotion_16_to_32(i1 %c) {
+
+; CHECK-LABEL: cmov_spromotion_16_to_32:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    testb   $1, %dil
+; CHECK-NEXT:    movl    $12414, %ecx
+; CHECK-NEXT:    movl    $-1, %eax
+; CHECK-NEXT:    cmovnel %ecx, %eax
+; CHECK-NEXT:    retq
+
+; MCU-LABEL: cmov_spromotion_16_to_32:
+; MCU:       # BB#0:
+; MCU-NEXT:    testb   $1, %al
+; MCU-NEXT:    movl    $12414, %eax
+; MCU-NEXT:    jne     .LBB2_2
+; MCU-NEXT:  # BB#1:
+; MCU-NEXT:    movl    $-1, %eax
+; MCU-NEXT:  .LBB2_2:
+; MCU-NEXT:    retl
+entry:
+  %0 = select i1 %c, i16 12414, i16 -1
+  %ret = sext i16 %0 to i32
+  ret i32 %ret
+}

test/CodeGen/X86/select.ll

; MCU-NEXT:    retl
   %t1 = load %0, %0* %q
   %t4 = select i1 %r, %0 %t0, %0 %t1
   %t5 = extractvalue %0 %t4, 1
   ret i32 %t5
 }
 
 ; PR2139
 define i32 @test2() nounwind {
-; CHECK-LABEL: test2:
-; CHECK:       ## BB#0: ## %entry
-; CHECK-NEXT:    pushq %rax
-; CHECK-NEXT:    callq _return_false
-; CHECK-NEXT:    xorl %ecx, %ecx
-; CHECK-NEXT:    testb $1, %al
-; CHECK-NEXT:    movw $-480, %ax ## imm = 0xFE20
-; CHECK-NEXT:    cmovnew %cx, %ax
-; CHECK-NEXT:    cwtl
-; CHECK-NEXT:    shll $3, %eax
-; CHECK-NEXT:    cmpl $32768, %eax ## imm = 0x8000
-; CHECK-NEXT:    jge LBB1_1
-; CHECK-NEXT:  ## BB#2: ## %bb91
-; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    popq %rcx
-; CHECK-NEXT:    retq
-; CHECK-NEXT:  LBB1_1: ## %bb90
-; CHECK-NEXT:    ## -- End function
+; GENERIC-LABEL: test2:
+; GENERIC:       ## BB#0: ## %entry
+; GENERIC-NEXT:    pushq %rax
+; GENERIC-NEXT:    callq _return_false
+; GENERIC-NEXT:    xorl %ecx, %ecx
+; GENERIC-NEXT:    testb $1, %al
+; GENERIC-NEXT:    movl $-480, %eax
+; GENERIC-NEXT:    cmovnel %ecx, %eax
+; GENERIC-NEXT:    shll $3, %eax
+; GENERIC-NEXT:    cmpl $32768, %eax ## imm = 0x8000
+; GENERIC-NEXT:    jge LBB1_1
+; GENERIC-NEXT:  ## BB#2: ## %bb91
+; GENERIC-NEXT:    xorl %eax, %eax
+; GENERIC-NEXT:    popq %rcx
+; GENERIC-NEXT:    retq
+; GENERIC-NEXT:  LBB1_1: ## %bb90
+; GENERIC-NEXT:    ## -- End function
+;
+; ATOM-LABEL: test2:
+; ATOM:       ## BB#0: ## %entry
+; ATOM-NEXT:    pushq %rax
+; ATOM-NEXT:    callq _return_false
+; ATOM-NEXT:    xorl %ecx, %ecx
+; ATOM-NEXT:    movl $-480, %edx
+; ATOM-NEXT:    testb $1, %al
+; ATOM-NEXT:    cmovnel %ecx, %edx
+; ATOM-NEXT:    shll $3, %edx
+; ATOM-NEXT:    cmpl $32768, %edx ## imm = 0x8000
+; ATOM-NEXT:    jge LBB1_1
+; ATOM-NEXT:  ## BB#2: ## %bb91
+; ATOM-NEXT:    xorl %eax, %eax
+; ATOM-NEXT:    popq %rcx
+; ATOM-NEXT:    retq
+; ATOM-NEXT:  LBB1_1: ## %bb90
+; ATOM-NEXT:    ## -- End function
 ;
 ; MCU-LABEL: test2:
 ; MCU:       # BB#0: # %entry
 ; MCU-NEXT:    calll return_false
+; MCU-NEXT:    xorl    %ecx, %ecx
 ; MCU-NEXT:    testb $1, %al
-; MCU-NEXT:    jne .LBB1_1
-; MCU-NEXT:  # BB#2: # %entry
-; MCU-NEXT:    movw $-480, %ax # imm = 0xFE20
-; MCU-NEXT:    jmp .LBB1_3
-; MCU-NEXT:  .LBB1_1:
-; MCU-NEXT:    xorl %eax, %eax
-; MCU-NEXT:  .LBB1_3: # %entry
-; MCU-NEXT:    cwtl
-; MCU-NEXT:    shll $3, %eax
-; MCU-NEXT:    cmpl $32768, %eax # imm = 0x8000
-; MCU-NEXT:    jge .LBB1_4
-; MCU-NEXT:  # BB#5: # %bb91
+; MCU-NEXT:    jne .LBB1_2
+; MCU-NEXT:  # BB#1: # %entry
+; MCU-NEXT:    movl $-480, %ecx # imm = 0xFE20
+; MCU-NEXT:  .LBB1_2:
+; MCU-NEXT:    shll $3, %ecx
+; MCU-NEXT:    cmpl $32768, %ecx # imm = 0x8000
+; MCU-NEXT:    jge .LBB1_3
+; MCU-NEXT:  # BB#4: # %bb91
 ; MCU-NEXT:    xorl %eax, %eax
 ; MCU-NEXT:    retl
-; MCU-NEXT:  .LBB1_4: # %bb90
+; MCU-NEXT:  .LBB1_3: # %bb90
 entry:
   %tmp73 = tail call i1 @return_false()
   %g.0 = select i1 %tmp73, i16 0, i16 -480
   %tmp7778 = sext i16 %g.0 to i32
   %tmp80 = shl i32 %tmp7778, 3
   %tmp87 = icmp sgt i32 %tmp80, 32767
   br i1 %tmp87, label %bb90, label %bb91
 bb90:

test/CodeGen/X86/vector-compare-results.ll

 ; AVX512BW-LABEL: test_cmp_v64i16:
 ; AVX512BW:       # BB#0:
 ; AVX512BW-NEXT:    vextracti32x4 $3, %zmm2, %xmm4
 ; AVX512BW-NEXT:    vpextrw $1, %xmm4, %ecx
 ; AVX512BW-NEXT:    vextracti32x4 $3, %zmm0, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm5, %edx
 ; AVX512BW-NEXT:    xorl %eax, %eax
 ; AVX512BW-NEXT:    cmpw %cx, %dx
-; AVX512BW-NEXT:    movw $-1, %cx
+; AVX512BW-NEXT:    movl $65535, %ecx # imm = 0xFFFF
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm4, %esi
 ; AVX512BW-NEXT:    vmovd %xmm5, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm6
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $2, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $3, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $4, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $5, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $6, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $7, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm6, %xmm4
 ; AVX512BW-NEXT:    vextracti32x4 $2, %zmm2, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm5, %edx
 ; AVX512BW-NEXT:    vextracti32x4 $2, %zmm0, %xmm6
 ; AVX512BW-NEXT:    vpextrw $1, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm5, %esi
 ; AVX512BW-NEXT:    vmovd %xmm6, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm7
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $2, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $3, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $4, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $5, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $6, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $7, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm7, %xmm5
 ; AVX512BW-NEXT:    vinserti128 $1, %xmm4, %ymm5, %ymm4
 ; AVX512BW-NEXT:    vextracti32x4 $1, %zmm2, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm5, %edx
 ; AVX512BW-NEXT:    vextracti32x4 $1, %zmm0, %xmm6
 ; AVX512BW-NEXT:    vpextrw $1, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm5, %esi
 ; AVX512BW-NEXT:    vmovd %xmm6, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm7
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $2, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $3, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $4, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $5, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $6, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm7, %xmm7
 ; AVX512BW-NEXT:    vpextrw $7, %xmm5, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm6, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm7, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $1, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm2, %esi
 ; AVX512BW-NEXT:    vmovd %xmm0, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm6
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $2, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $3, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $4, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $5, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $6, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $7, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm0, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm6, %xmm0
 ; AVX512BW-NEXT:    vinserti128 $1, %xmm5, %ymm0, %ymm0
 ; AVX512BW-NEXT:    vinserti64x4 $1, %ymm4, %zmm0, %zmm0
 ; AVX512BW-NEXT:    vpmovwb %zmm0, %ymm0
 ; AVX512BW-NEXT:    vextracti32x4 $3, %zmm3, %xmm2
 ; AVX512BW-NEXT:    vpextrw $1, %xmm2, %edx
 ; AVX512BW-NEXT:    vextracti32x4 $3, %zmm1, %xmm4
 ; AVX512BW-NEXT:    vpextrw $1, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm2, %esi
 ; AVX512BW-NEXT:    vmovd %xmm4, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm5
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $2, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $3, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $4, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $5, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $6, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $7, %xmm2, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm4, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm5, %xmm2
 ; AVX512BW-NEXT:    vextracti32x4 $2, %zmm3, %xmm4
 ; AVX512BW-NEXT:    vpextrw $1, %xmm4, %edx
 ; AVX512BW-NEXT:    vextracti32x4 $2, %zmm1, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm4, %esi
 ; AVX512BW-NEXT:    vmovd %xmm5, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm6
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $2, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $3, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $4, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $5, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $6, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $7, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm6, %xmm4
 ; AVX512BW-NEXT:    vinserti128 $1, %xmm2, %ymm4, %ymm2
 ; AVX512BW-NEXT:    vextracti32x4 $1, %zmm3, %xmm4
 ; AVX512BW-NEXT:    vpextrw $1, %xmm4, %edx
 ; AVX512BW-NEXT:    vextracti32x4 $1, %zmm1, %xmm5
 ; AVX512BW-NEXT:    vpextrw $1, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm4, %esi
 ; AVX512BW-NEXT:    vmovd %xmm5, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm6
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $2, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $3, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $4, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $5, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $6, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm6, %xmm6
 ; AVX512BW-NEXT:    vpextrw $7, %xmm4, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm5, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $7, %edx, %xmm6, %xmm4
 ; AVX512BW-NEXT:    vpextrw $1, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $1, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vmovd %xmm3, %esi
 ; AVX512BW-NEXT:    vmovd %xmm1, %edi
 ; AVX512BW-NEXT:    cmpw %si, %di
 ; AVX512BW-NEXT:    movl $0, %esi
-; AVX512BW-NEXT:    cmovgw %cx, %si
+; AVX512BW-NEXT:    cmovgl %ecx, %esi
 ; AVX512BW-NEXT:    vmovd %esi, %xmm5
 ; AVX512BW-NEXT:    vpinsrw $1, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $2, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $2, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $2, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $3, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $3, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $3, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $4, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $4, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $4, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $5, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $5, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $5, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $6, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $6, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
 ; AVX512BW-NEXT:    movl $0, %edx
-; AVX512BW-NEXT:    cmovgw %cx, %dx
+; AVX512BW-NEXT:    cmovgl %ecx, %edx
 ; AVX512BW-NEXT:    vpinsrw $6, %edx, %xmm5, %xmm5
 ; AVX512BW-NEXT:    vpextrw $7, %xmm3, %edx
 ; AVX512BW-NEXT:    vpextrw $7, %xmm1, %esi
 ; AVX512BW-NEXT:    cmpw %dx, %si
-; AVX512BW-NEXT:    cmovgw %cx, %ax
+; AVX512BW-NEXT:    cmovgl %ecx, %eax
 ; AVX512BW-NEXT:    vpinsrw $7, %eax, %xmm5, %xmm1
 ; AVX512BW-NEXT:    vinserti128 $1, %xmm4, %ymm1, %ymm1
 ; AVX512BW-NEXT:    vinserti64x4 $1, %ymm2, %zmm1, %zmm1
 ; AVX512BW-NEXT:    vpmovwb %zmm1, %ymm1
 ; AVX512BW-NEXT:    vinserti64x4 $1, %ymm1, %zmm0, %zmm0
 ; AVX512BW-NEXT:    retq
   %1 = icmp sgt <64 x i16> %a0, %a1
   ret <64 x i1> %1