[SelectionDAG] Add icmp UNDEF handling to SelectionDAG::FoldSetCC

[RKSimon]

Summary

First half of PR40800, this patch adds DAG undef handling to icmp instructions to match the behaviour in SimplifyICmpInst, this permits constant folding of vector comparisons where some elements had been reduced to UNDEF (by SimplifyDemandedVectorElts etc.).

This involves a lot of tweaking to reduced tests as bugpoint loves to reduce icmp arguments to undef........

I've gone through these tests best I can, and AFAICT I haven't changed their purposes in any critical way - many are fragile and difficult to reduce from scratch as they are often 5+ years old.... I'd appreciate it if the respective target specialists could check my changes. I'd like to pre-commit some of these changes where its just a case of 'unreducing' lost arguments etc.

[efriedma]

test/CodeGen/ARM/ifcvt-branch-weight-bug.ll

1 ↗

(On Diff #190615)

LGTM to commit separately.

test/CodeGen/ARM/struct-byval-frame-index.ll

1 ↗

(On Diff #190615)

Also LGTM to commit separately. (It's sort of hard to tell what this is aiming for, but I think you're fine as long as you preserve the CFG.)

[tlively]

test/CodeGen/WebAssembly/cfg-stackify.ll

370 ↗

(On Diff #190615)

WebAssembly changes LGTM. Thanks for checking!

[RKSimon]

rebase - thanks @efriedma and @tlively

[RKSimon]

rebase after regenerating sparc test in rL356253

[kparzysz]

The Hexagon changes look good.

[RKSimon]

rebase - thanks @kparzysz

[jonpa]

I think the SystemZ test changes look ok (by replacing the undef operands with an argument the tests are more or less unaffected by this patch), but as usual I will let Uli do the formal approval.

[RKSimon]

@craig.topper Are you happy with the x86 changes?

[uweigand]

In D59363#1431933, @jonpa wrote:

I think the SystemZ test changes look ok (by replacing the undef operands with an argument the tests are more or less unaffected by this patch), but as usual I will let Uli do the formal approval.

The SystemZ test changes LGTM. It's probably better anyway to have real arguments instead of undefs ...

[RKSimon]

In D59363#1432683, @uweigand wrote:

In D59363#1431933, @jonpa wrote:

I think the SystemZ test changes look ok (by replacing the undef operands with an argument the tests are more or less unaffected by this patch), but as usual I will let Uli do the formal approval.

The SystemZ test changes LGTM. It's probably better anyway to have real arguments instead of undefs ...

Thanks @uweigand - once this patch goes in bugpoint should be less likely to reduce icmp arguments.

[RKSimon]

rebase - thanks @uweigand

[lebedev.ri]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

Hmm, but -instsimplify says it's 0 https://godbolt.org/z/1dpyQG
Either answer is correct?

[RKSimon]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

Interesting (please can you raise a bug?) - the patch isn't actually changing the result in DAG, its just managing to constant fold.

[spatel]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

Probably not necessary? The urem invokes UB:
https://llvm.org/docs/LangRef.html#urem-instruction

So it's not that either answer is correct...*any* answer is correct. :)

[lebedev.ri]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

https://bugs.llvm.org/show_bug.cgi?id=41125

[spatel]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

So there are 2 questions of UB/undef/poison:

1. The urem causes immediate UB (same as div-by-0), so anything goes in this example.
2. Independent of that, what is the output of an icmp with undef operand? PR41125 shows that we have potentially 2 different answers in IR, so we need to decide which behavior is correct/optimal to mimic here in the DAG.

[RKSimon]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

686

What I think I'm going to do is add the llvm::ConstantFoldCompareInstruction logic to both SimplifyICmpInst (in a new patch) and to FoldSetCC (in this patch):

If the predicate is eq/ne and any operand is undef then return undef - else return a bool constant using isTrueWhenEqual. That way we are consistent for all cases where the other operand is constant/non-constant,

Does that make sense?

[RKSimon]

Updated undef behaviour to match llvm::ConstantFoldCompareInstruction (see D59541 as well).

[RKSimon]

test/CodeGen/SPARC/missinglabel.ll

7

@dcederman Are you OK with this change please?

[RKSimon]

@spatel @craig.topper @efriedma any more comments?

[spatel]

The x86 regression test changes to preserve behavior look good. Pre-commit those, so we're just left actual diffs here?

[RKSimon]

rebase

[spatel]

LGTM

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

SDValue SelectionDAG::FoldSetCC(EVT VT, SDValue N1, SDValue N2,
   case ISD::SETO:
   case ISD::SETUO:
   case ISD::SETUEQ:
   case ISD::SETUNE:
     assert(!OpVT.isInteger() && "Illegal setcc for integer!");
     break;
   }
 
-  // We can always fold X == X for integer setcc's.
-  if (N1 == N2 && OpVT.isInteger())
+  // icmp X, X -> true/false
+  // icmp X, undef -> true/false because undef could be X.
+  if (OpVT.isInteger())
+    if (N1 == N2 || N1.isUndef() || N2.isUndef())
       return getBoolConstant(ISD::isTrueWhenEqual(Cond), dl, VT, OpVT);
 
   if (ConstantSDNode *N2C = dyn_cast<ConstantSDNode>(N2)) {
     const APInt &C2 = N2C->getAPIntValue();
     if (ConstantSDNode *N1C = dyn_cast<ConstantSDNode>(N1)) {
       const APInt &C1 = N1C->getAPIntValue();
 
       switch (Cond) {
       default: llvm_unreachable("Unknown integer setcc!");

test/CodeGen/AArch64/urem-seteq-vec-nonsplat.ll

; CHECK-NEXT:    ret
   %cmp = icmp eq <4 x i32> %urem, <i32 1, i32 0, i32 1, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_div_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_div_undef:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
-; CHECK-NEXT:    movi v1.4s, #1
-; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    movi v0.4s, #1
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 undef, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_comp_undef(<4 x i32> %X) nounwind readnone {
; CHECK-NEXT:    ret
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_both_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_both_undef:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
-; CHECK-NEXT:    movi v1.4s, #1
-; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    movi v0.4s, #1
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 undef, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_div_even_odd(<4 x i32> %X) nounwind readnone {

test/CodeGen/SPARC/missinglabel.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -verify-machineinstrs | FileCheck %s
 
 target datalayout = "E-m:e-i64:64-n32:64-S128"
 target triple = "sparc64-unknown-linux-gnu"
 
-define void @f() align 2 {
+define void @f(i64 %a0) align 2 {
 ; CHECK-LABEL: f:
 ; CHECK:         .cfi_startproc
 ; CHECK-NEXT:  ! %bb.0: ! %entry
 ; CHECK-NEXT:    cmp %o0, 0
 ; CHECK-NEXT:    be %xcc, .LBB0_2
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    ba .LBB0_1
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:  .LBB0_2: ! %targetblock
 ; CHECK-NEXT:    mov %g0, %o0
 ; CHECK-NEXT:    cmp %o0, 0
 ; CHECK-NEXT:    bne .LBB0_4
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:  ! %bb.3: ! %cond.false.i83
 ; CHECK-NEXT:  .LBB0_1: ! %cond.false
 ; CHECK-NEXT:  .LBB0_4: ! %exit.i85
 entry:
-  %cmp = icmp eq i64 undef, 0
+  %cmp = icmp eq i64 %a0, 0
   br i1 %cmp, label %targetblock, label %cond.false
 
 cond.false:
   unreachable
 
 targetblock:
   br i1 undef, label %cond.false.i83, label %exit.i85
 
 cond.false.i83:
   unreachable
 
 exit.i85:
   unreachable
 }

test/CodeGen/SystemZ/buildvector-00.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
 
 ; Test that the dag combiner can understand that some vector operands are
 ; all-zeros and then optimize the logical operations.
 
-define void @f1() {
+define void @f1(<2 x i64> %a0) {
 ; CHECK-LABEL: f1:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    vlrepg %v0, 0(%r1)
 ; CHECK-NEXT:    vgbm %v1, 0
-; CHECK-NEXT:    vceqg %v2, %v0, %v1
+; CHECK-NEXT:    vceqg %v2, %v24, %v1
 ; CHECK-NEXT:    vn %v0, %v0, %v0
 ; CHECK-NEXT:    vno %v2, %v2, %v2
 ; CHECK-NEXT:    vceqg %v0, %v0, %v1
 ; CHECK-NEXT:    vx %v0, %v2, %v0
 ; CHECK-NEXT:    vnc %v0, %v2, %v0
 ; CHECK-NEXT:    vlgvf %r0, %v0, 1
 ; CHECK-NEXT:    tmll %r0, 1
 ; CHECK-NEXT:  # %bb.1: # %bb15
 
 bb:
   %tmp = shufflevector <2 x i64> undef, <2 x i64> undef, <2 x i32> zeroinitializer
   br label %bb1
 
 bb1:                                              ; preds = %bb
   %tmp2 = load i64, i64* undef, align 8
   %tmp3 = insertelement <2 x i64> undef, i64 %tmp2, i32 1
-  %tmp4 = icmp ne <2 x i64> undef, zeroinitializer
+  %tmp4 = icmp ne <2 x i64> %a0, zeroinitializer
   %tmp5 = xor <2 x i1> %tmp4, zeroinitializer
   %tmp6 = xor <2 x i1> zeroinitializer, %tmp5
   %tmp7 = and <2 x i64> %tmp3, %tmp
   %tmp8 = icmp ne <2 x i64> %tmp7, zeroinitializer
   %tmp9 = xor <2 x i1> zeroinitializer, %tmp8
-  %tmp10 = icmp ne <2 x i64> undef, zeroinitializer
+  %tmp10 = icmp ne <2 x i64> %a0, zeroinitializer
   %tmp11 = xor <2 x i1> %tmp10, %tmp9
   %tmp12 = and <2 x i1> %tmp6, %tmp11
   %tmp13 = extractelement <2 x i1> %tmp12, i32 0
   br i1 %tmp13, label %bb14, label %bb15
 
 bb14:                                             ; preds = %bb1
   store i64 undef, i64* undef, align 8
   br label %bb15
 
 bb15:                                             ; preds = %bb14, %bb1
   unreachable
 }

test/CodeGen/SystemZ/dag-combine-03.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=s390x-linux-gnu -mcpu=z13 < %s  | FileCheck %s
 
 ; Test that DAGCombiner gets helped by getKnownBitsForTargetNode() when
 ; BITCAST nodes are involved on a big-endian target.
 ;
 ; The EXTRACT_VECTOR_ELT is done first into an i32, and then AND:ed with
 ; 1. The AND is not actually necessary since the element contains a CC (i1)
 ; value. Test that the BITCAST nodes in the DAG when computing KnownBits is
 ; handled so that the AND is removed. If this succeeds, this results in a CHI
 ; instead of TMLL.
 
-define void @fun() {
+define void @fun(i64 %a0) {
 ; CHECK-LABEL: fun:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    lghi %r1, 0
 ; CHECK-NEXT:  .LBB0_1: # %lab0
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    la %r0, 2(%r1)
 ; CHECK-NEXT:    la %r1, 1(%r1)
-; CHECK-NEXT:    cgr %r1, %r0
-; CHECK-NEXT:    lhi %r2, 0
-; CHECK-NEXT:    lochie %r2, 1
-; CHECK-NEXT:    cgr %r0, %r0
+; CHECK-NEXT:    cgr %r1, %r2
+; CHECK-NEXT:    lhi %r3, 0
+; CHECK-NEXT:    lochie %r3, 1
+; CHECK-NEXT:    cgr %r0, %r2
 ; CHECK-NEXT:    lhi %r0, 0
 ; CHECK-NEXT:    lochie %r0, 1
-; CHECK-NEXT:    vlvgp %v0, %r2, %r2
+; CHECK-NEXT:    vlvgp %v0, %r3, %r3
 ; CHECK-NEXT:    vlvgp %v1, %r0, %r0
 ; CHECK-NEXT:    vx %v0, %v0, %v1
 ; CHECK-NEXT:    vlgvf %r0, %v0, 1
 ; CHECK-NEXT:    chi %r0, 0
 ; CHECK-NEXT:    locghie %r1, 0
 ; CHECK-NEXT:    j .LBB0_1
 entry:
   br label %lab0
 
 lab0:
   %phi = phi i64 [ %sel, %lab0 ], [ 0, %entry ]
   %add = add nuw nsw i64 %phi, 1
   %add2 = add nuw nsw i64 %phi, 2
-  %cmp = icmp eq i64 %add, undef
-  %cmp2 = icmp eq i64 %add2, undef
+  %cmp = icmp eq i64 %add, %a0
+  %cmp2 = icmp eq i64 %add2, %a0
   %ins = insertelement <2 x i1> undef, i1 %cmp, i32 0
   %ins2 = insertelement <2 x i1> undef, i1 %cmp2, i32 0
   %xor = xor <2 x i1> %ins, %ins2
   %extr = extractelement <2 x i1> %xor, i32 0
 
   %sel = select i1 %extr, i64 %add, i64 0
   br label %lab0
 }

test/CodeGen/X86/2006-11-17-IllegalMove.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 
 define void @handle_vector_size_attribute() nounwind {
 ; CHECK-LABEL: handle_vector_size_attribute:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movl 0, %eax
 ; CHECK-NEXT:    decl %eax
 ; CHECK-NEXT:    cmpl $1, %eax
 ; CHECK-NEXT:    ja .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %bb77
 ; CHECK-NEXT:    movb 0, %al
-; CHECK-NEXT:    movzbl %al, %eax
-; CHECK-NEXT:    # kill: def $eax killed $eax def $ax
-; CHECK-NEXT:    divb 0
-; CHECK-NEXT:    movzbl %al, %eax
-; CHECK-NEXT:    cmpq %rax, %rax
+; CHECK-NEXT:    movb 0, %al
+; CHECK-NEXT:    movb $1, %al
+; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:  .LBB0_2: # %bb84
 ; CHECK-NEXT:    retq
 entry:
   %tmp69 = load i32, i32* null    ; <i32> [#uses=1]
   switch i32 %tmp69, label %bb84 [
      i32 2, label %bb77
      i32 1, label %bb77
   ]

test/CodeGen/X86/hoist-spill.ll

 ; RUN: llc < %s | FileCheck %s
 
 ; Check no spills to the same stack slot after hoisting.
 ; CHECK: mov{{.}} %{{.*}}, [[SPOFFSET1:-?[0-9]*]](%rsp)
 ; CHECK: mov{{.}} %{{.*}}, [[SPOFFSET2:-?[0-9]*]](%rsp)
 ; CHECK-NOT: mov{{.}} %{{.*}}, [[SPOFFSET1]](%rsp)
 ; CHECK-NOT: mov{{.}} %{{.*}}, [[SPOFFSET2]](%rsp)
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @a = external global i32*, align 8
 @b = external global i32, align 4
 @d = external global i32*, align 8
 
 ; Function Attrs: norecurse noreturn nounwind uwtable
-define void @fn1(i32 %p1) {
+define void @fn1(i32 %p1, i32 %p2, i64 %p3) {
 entry:
   %tmp = load i32*, i32** @d, align 8
   %tmp1 = load i32*, i32** @a, align 8
   %tmp2 = sext i32 %p1 to i64
   br label %for.cond
 
 for.cond:                                         ; preds = %for.inc14, %entry
   %indvar = phi i32 [ %indvar.next, %for.inc14 ], [ 0, %entry ]
   %indvars.iv30.in = phi i32 [ %indvars.iv30, %for.inc14 ], [ %p1, %entry ]
   %c.0 = phi i32 [ %inc15, %for.inc14 ], [ 1, %entry ]
   %k.0 = phi i32 [ %k.1.lcssa, %for.inc14 ], [ undef, %entry ]
-  %tmp3 = icmp sgt i32 undef, 0
+  %tmp3 = icmp sgt i32 %p2, 0
   %smax52 = select i1 %tmp3, i32 %c.0, i32 0
   %tmp4 = zext i32 %smax52 to i64
-  %tmp5 = icmp sgt i64 undef, %tmp4
+  %tmp5 = icmp sgt i64 %p3, %tmp4
   %smax53 = select i1 %tmp5, i64 %tmp2, i64 %tmp4
   %tmp6 = add nsw i64 %smax53, 1
   %tmp7 = sub nsw i64 %tmp6, %tmp4
   %tmp8 = add nsw i64 %tmp7, -8
   %tmp9 = sub i32 undef, %indvar
   %tmp10 = icmp sgt i64 %tmp2, 0
   %smax40 = select i1 %tmp10, i64 %tmp2, i64 0
   %scevgep41 = getelementptr i32, i32* %tmp1, i64 %smax40

test/CodeGen/X86/select.ll

; MCU-NEXT:    retl
   %sel1 = select i1 %cmp, i32 32767, i32 %src
   %cmp1 = icmp slt i32 %sel1, -32768
   %sel2 = select i1 %cmp1, i32 -32768, i32 %sel1
   %conv = trunc i32 %sel2 to i16
   store i16 %conv, i16* %dst, align 2
   ret void
 }
 
-define void @test19() {
-; This is a massive reduction of an llvm-stress test case that generates
-; interesting chains feeding setcc and eventually a f32 select operation. This
-; is intended to exercise the SELECT formation in the DAG combine simplifying
-; a simplified select_cc node. If it it regresses and is no longer triggering
-; that code path, it can be deleted.
-;
-; CHECK-LABEL: test19:
-; CHECK:       ## %bb.0: ## %BB
-; CHECK-NEXT:    movl $-1, %eax
-; CHECK-NEXT:    movb $1, %cl
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  LBB23_1: ## %CF
-; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    testb %cl, %cl
-; CHECK-NEXT:    jne LBB23_1
-; CHECK-NEXT:  ## %bb.2: ## %CF250
-; CHECK-NEXT:    ## in Loop: Header=BB23_1 Depth=1
-; CHECK-NEXT:    jne LBB23_1
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  LBB23_3: ## %CF242
-; CHECK-NEXT:    ## =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    cmpl %eax, %eax
-; CHECK-NEXT:    ucomiss %xmm0, %xmm0
-; CHECK-NEXT:    jp LBB23_3
-; CHECK-NEXT:  ## %bb.4: ## %CF244
-; CHECK-NEXT:    retq
-;
-; ATHLON-LABEL: test19:
-; ATHLON:       ## %bb.0: ## %BB
-; ATHLON-NEXT:    movb $1, %al
-; ATHLON-NEXT:    .p2align 4, 0x90
-; ATHLON-NEXT:  LBB23_1: ## %CF
-; ATHLON-NEXT:    ## =>This Inner Loop Header: Depth=1
-; ATHLON-NEXT:    testb %al, %al
-; ATHLON-NEXT:    jne LBB23_1
-; ATHLON-NEXT:  ## %bb.2: ## %CF250
-; ATHLON-NEXT:    ## in Loop: Header=BB23_1 Depth=1
-; ATHLON-NEXT:    jne LBB23_1
-; ATHLON-NEXT:  ## %bb.3: ## %CF242.preheader
-; ATHLON-NEXT:    fldz
-; ATHLON-NEXT:    .p2align 4, 0x90
-; ATHLON-NEXT:  LBB23_4: ## %CF242
-; ATHLON-NEXT:    ## =>This Inner Loop Header: Depth=1
-; ATHLON-NEXT:    fucomi %st(0), %st
-; ATHLON-NEXT:    jp LBB23_4
-; ATHLON-NEXT:  ## %bb.5: ## %CF244
-; ATHLON-NEXT:    fstp %st(0)
-; ATHLON-NEXT:    retl
-;
-; MCU-LABEL: test19:
-; MCU:       # %bb.0: # %BB
-; MCU-NEXT:    movl $-1, %ecx
-; MCU-NEXT:    movb $1, %al
-; MCU-NEXT:    .p2align 4, 0x90
-; MCU-NEXT:  .LBB23_1: # %CF
-; MCU-NEXT:    # =>This Inner Loop Header: Depth=1
-; MCU-NEXT:    testb %al, %al
-; MCU-NEXT:    jne .LBB23_1
-; MCU-NEXT:  # %bb.2: # %CF250
-; MCU-NEXT:    # in Loop: Header=BB23_1 Depth=1
-; MCU-NEXT:    jne .LBB23_1
-; MCU-NEXT:  # %bb.3: # %CF242.preheader
-; MCU-NEXT:    fldz
-; MCU-NEXT:    .p2align 4, 0x90
-; MCU-NEXT:  .LBB23_4: # %CF242
-; MCU-NEXT:    # =>This Inner Loop Header: Depth=1
-; MCU-NEXT:    cmpl %eax, %ecx
-; MCU-NEXT:    fucom %st(0)
-; MCU-NEXT:    fnstsw %ax
-; MCU-NEXT:    # kill: def $ah killed $ah killed $ax
-; MCU-NEXT:    sahf
-; MCU-NEXT:    jp .LBB23_4
-; MCU-NEXT:  # %bb.5: # %CF244
-; MCU-NEXT:    fstp %st(0)
-; MCU-NEXT:    retl
-BB:
-  br label %CF
-
-CF:
-  %Cmp10 = icmp ule i8 undef, undef
-  br i1 %Cmp10, label %CF, label %CF250
-
-CF250:
-  %E12 = extractelement <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, i32 2
-  %Cmp32 = icmp ugt i1 %Cmp10, false
-  br i1 %Cmp32, label %CF, label %CF242
-
-CF242:
-  %Cmp38 = icmp uge i32 %E12, undef
-  %FC = uitofp i1 %Cmp38 to float
-  %Sl59 = select i1 %Cmp32, float %FC, float undef
-  %Cmp60 = fcmp ugt float undef, undef
-  br i1 %Cmp60, label %CF242, label %CF244
-
-CF244:
-  %B122 = fadd float %Sl59, undef
-  ret void
-}
-
 define i16 @select_xor_1(i16 %A, i8 %cond) {
 ; CHECK-LABEL: select_xor_1:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    movl %edi, %eax
 ; CHECK-NEXT:    xorl $43, %eax
 ; CHECK-NEXT:    testb $1, %sil
 ; CHECK-NEXT:    cmovel %edi, %eax
 ; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax

 ; ATHLON-NEXT:    testb $1, {{[0-9]+}}(%esp)
 ; ATHLON-NEXT:    cmovel %ecx, %eax
 ; ATHLON-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: select_xor_1b:
 ; MCU:       # %bb.0: # %entry
 ; MCU-NEXT:    testb $1, %dl
-; MCU-NEXT:    je .LBB25_2
+; MCU-NEXT:    je .LBB24_2
 ; MCU-NEXT:  # %bb.1:
 ; MCU-NEXT:    xorl $43, %eax
-; MCU-NEXT:  .LBB25_2: # %entry
+; MCU-NEXT:  .LBB24_2: # %entry
 ; MCU-NEXT:    # kill: def $ax killed $ax killed $eax
 ; MCU-NEXT:    retl
 entry:
  %and = and i8 %cond, 1
  %cmp10 = icmp ne i8 %and, 1
  %0 = xor i16 %A, 43
  %1 = select i1 %cmp10, i16 %A, i16 %0
  ret i16 %1

 ; ATHLON-NEXT:    xorl %ecx, %eax
 ; ATHLON-NEXT:    testb $1, {{[0-9]+}}(%esp)
 ; ATHLON-NEXT:    cmovel %ecx, %eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: select_xor_2b:
 ; MCU:       # %bb.0: # %entry
 ; MCU-NEXT:    testb $1, %cl
-; MCU-NEXT:    je .LBB27_2
+; MCU-NEXT:    je .LBB26_2
 ; MCU-NEXT:  # %bb.1:
 ; MCU-NEXT:    xorl %edx, %eax
-; MCU-NEXT:  .LBB27_2: # %entry
+; MCU-NEXT:  .LBB26_2: # %entry
 ; MCU-NEXT:    retl
 entry:
  %and = and i8 %cond, 1
  %cmp10 = icmp ne i8 %and, 1
  %0 = xor i32 %B, %A
  %1 = select i1 %cmp10, i32 %A, i32 %0
  ret i32 %1
 }

 ; ATHLON-NEXT:    orl %ecx, %eax
 ; ATHLON-NEXT:    testb $1, {{[0-9]+}}(%esp)
 ; ATHLON-NEXT:    cmovel %ecx, %eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: select_or_b:
 ; MCU:       # %bb.0: # %entry
 ; MCU-NEXT:    testb $1, %cl
-; MCU-NEXT:    je .LBB29_2
+; MCU-NEXT:    je .LBB28_2
 ; MCU-NEXT:  # %bb.1:
 ; MCU-NEXT:    orl %edx, %eax
-; MCU-NEXT:  .LBB29_2: # %entry
+; MCU-NEXT:  .LBB28_2: # %entry
 ; MCU-NEXT:    retl
 entry:
  %and = and i8 %cond, 1
  %cmp10 = icmp ne i8 %and, 1
  %0 = or i32 %B, %A
  %1 = select i1 %cmp10, i32 %A, i32 %0
  ret i32 %1
 }

 ; ATHLON-NEXT:    orl %ecx, %eax
 ; ATHLON-NEXT:    testb $1, {{[0-9]+}}(%esp)
 ; ATHLON-NEXT:    cmovel %ecx, %eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: select_or_1b:
 ; MCU:       # %bb.0: # %entry
 ; MCU-NEXT:    testb $1, %cl
-; MCU-NEXT:    je .LBB31_2
+; MCU-NEXT:    je .LBB30_2
 ; MCU-NEXT:  # %bb.1:
 ; MCU-NEXT:    orl %edx, %eax
-; MCU-NEXT:  .LBB31_2: # %entry
+; MCU-NEXT:  .LBB30_2: # %entry
 ; MCU-NEXT:    retl
 entry:
  %and = and i32 %cond, 1
  %cmp10 = icmp ne i32 %and, 1
  %0 = or i32 %B, %A
  %1 = select i1 %cmp10, i32 %A, i32 %0
  ret i32 %1
 }

test/CodeGen/X86/tail-dup-merge-loop-headers.ll

 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    pushq %r15
 ; CHECK-NEXT:    pushq %r14
 ; CHECK-NEXT:    pushq %r13
 ; CHECK-NEXT:    pushq %r12
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    pushq %rax
-; CHECK-NEXT:    movl $1, %ebx
+; CHECK-NEXT:    movl $1, %r14d
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    jne .LBB1_26
+; CHECK-NEXT:    jne .LBB1_15
 ; CHECK-NEXT:  # %bb.1: # %if.end19
 ; CHECK-NEXT:    movl %esi, %ebp
 ; CHECK-NEXT:    movq %rdi, %r12
-; CHECK-NEXT:    movl (%rax), %r14d
-; CHECK-NEXT:    leal (,%r14,4), %r13d
+; CHECK-NEXT:    movl (%rax), %ebx
+; CHECK-NEXT:    leal (,%rbx,4), %r13d
 ; CHECK-NEXT:    movl %r13d, %r15d
 ; CHECK-NEXT:    movl $1, %esi
 ; CHECK-NEXT:    movq %r15, %rdi
 ; CHECK-NEXT:    callq cli_calloc
 ; CHECK-NEXT:    testl %ebp, %ebp
-; CHECK-NEXT:    je .LBB1_25
+; CHECK-NEXT:    je .LBB1_15
 ; CHECK-NEXT:  # %bb.2: # %if.end19
-; CHECK-NEXT:    testl %r14d, %r14d
-; CHECK-NEXT:    je .LBB1_25
+; CHECK-NEXT:    testl %ebx, %ebx
+; CHECK-NEXT:    je .LBB1_15
 ; CHECK-NEXT:  # %bb.3: # %if.end19
-; CHECK-NEXT:    movq %rax, %rbx
-; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    jne .LBB1_25
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    testb %cl, %cl
+; CHECK-NEXT:    jne .LBB1_15
 ; CHECK-NEXT:  # %bb.4: # %if.end19
 ; CHECK-NEXT:    cmpq %r12, %rax
-; CHECK-NEXT:    jb .LBB1_25
+; CHECK-NEXT:    jb .LBB1_15
 ; CHECK-NEXT:  # %bb.5: # %if.end50
-; CHECK-NEXT:    movq %rbx, %rdi
+; CHECK-NEXT:    movq %rax, %rdi
 ; CHECK-NEXT:    movq %r15, %rdx
 ; CHECK-NEXT:    callq memcpy
 ; CHECK-NEXT:    cmpl $4, %r13d
-; CHECK-NEXT:    jb .LBB1_28
+; CHECK-NEXT:    jb .LBB1_23
 ; CHECK-NEXT:  # %bb.6: # %shared_preheader
-; CHECK-NEXT:    movb $32, %dl
+; CHECK-NEXT:    movb $32, %sil
 ; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    # implicit-def: $rcx
-; CHECK-NEXT:    testl %eax, %eax
-; CHECK-NEXT:    je .LBB1_18
+; CHECK-NEXT:    movb $1, %cl
+; CHECK-NEXT:    # implicit-def: $rdx
+; CHECK-NEXT:    testl %ebx, %ebx
+; CHECK-NEXT:    jne .LBB1_11
+; CHECK-NEXT:    jmp .LBB1_8
+; CHECK-NEXT:  .LBB1_21: # %outer_loop_latch
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    movzwl %di, %edi
+; CHECK-NEXT:    decl %edi
+; CHECK-NEXT:    movzwl %di, %edi
+; CHECK-NEXT:    leaq 1(%rdx,%rdi), %rdx
+; CHECK-NEXT:    testl %ebx, %ebx
+; CHECK-NEXT:    je .LBB1_8
 ; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB1_8: # %shared_loop_header
+; CHECK-NEXT:  .LBB1_11: # %shared_loop_header
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    cmpq %rbx, %rax
-; CHECK-NEXT:    jb .LBB1_27
-; CHECK-NEXT:  # %bb.9: # %inner_loop_body
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    testl %eax, %eax
-; CHECK-NEXT:    jns .LBB1_8
-; CHECK-NEXT:  # %bb.10: # %if.end96.i
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    cmpl $3, %eax
-; CHECK-NEXT:    jae .LBB1_22
-; CHECK-NEXT:  # %bb.11: # %if.end287.i
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %esi, %esi
-; CHECK-NEXT:    cmpl $1, %eax
-; CHECK-NEXT:    setne %dl
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    jne .LBB1_15
-; CHECK-NEXT:  # %bb.12: # %if.end308.i
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
+; CHECK-NEXT:    jne .LBB1_22
+; CHECK-NEXT:  # %bb.12: # %inner_loop_body
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    je .LBB1_17
-; CHECK-NEXT:  # %bb.13: # %if.end335.i
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:    testb %dl, %dl
-; CHECK-NEXT:    movl $0, %esi
-; CHECK-NEXT:    jne .LBB1_7
-; CHECK-NEXT:  # %bb.14: # %merge_other
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    xorl %esi, %esi
-; CHECK-NEXT:    jmp .LBB1_16
-; CHECK-NEXT:  .LBB1_15: # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movb %dl, %sil
-; CHECK-NEXT:    addl $3, %esi
-; CHECK-NEXT:  .LBB1_16: # %outer_loop_latch
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    # implicit-def: $dl
-; CHECK-NEXT:    jmp .LBB1_7
-; CHECK-NEXT:  .LBB1_17: # %merge_predecessor_split
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movb $32, %dl
+; CHECK-NEXT:    je .LBB1_11
+; CHECK-NEXT:  # %bb.13: # %if.end96.i
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    testb %cl, %cl
+; CHECK-NEXT:    je .LBB1_14
+; CHECK-NEXT:  # %bb.16: # %if.end287.i
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    movw $3, %di
+; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    # implicit-def: $sil
+; CHECK-NEXT:    jne .LBB1_21
+; CHECK-NEXT:  # %bb.17: # %if.end308.i
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    je .LBB1_18
+; CHECK-NEXT:  # %bb.19: # %if.end335.i
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
 ; CHECK-NEXT:    xorl %esi, %esi
-; CHECK-NEXT:  .LBB1_7: # %outer_loop_latch
-; CHECK-NEXT:    # in Loop: Header=BB1_8 Depth=1
-; CHECK-NEXT:    movzwl %si, %esi
-; CHECK-NEXT:    decl %esi
-; CHECK-NEXT:    movzwl %si, %esi
-; CHECK-NEXT:    leaq 1(%rcx,%rsi), %rcx
-; CHECK-NEXT:    testl %eax, %eax
-; CHECK-NEXT:    jne .LBB1_8
-; CHECK-NEXT:  .LBB1_18: # %while.cond.us1412.i
+; CHECK-NEXT:    testb %sil, %sil
+; CHECK-NEXT:    movl $0, %edi
+; CHECK-NEXT:    jne .LBB1_21
+; CHECK-NEXT:  # %bb.20: # %merge_other
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    # implicit-def: $sil
+; CHECK-NEXT:    jmp .LBB1_21
+; CHECK-NEXT:  .LBB1_18: # %merge_predecessor_split
+; CHECK-NEXT:    # in Loop: Header=BB1_11 Depth=1
+; CHECK-NEXT:    movb $32, %sil
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    jmp .LBB1_21
+; CHECK-NEXT:  .LBB1_8: # %while.cond.us1412.i
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    movl $1, %ebx
-; CHECK-NEXT:    jne .LBB1_20
-; CHECK-NEXT:  # %bb.19: # %while.cond.us1412.i
-; CHECK-NEXT:    decb %dl
-; CHECK-NEXT:    jne .LBB1_26
-; CHECK-NEXT:  .LBB1_20: # %if.end41.us1436.i
-; CHECK-NEXT:  .LBB1_25:
-; CHECK-NEXT:    movl $1, %ebx
-; CHECK-NEXT:    jmp .LBB1_26
-; CHECK-NEXT:  .LBB1_22: # %if.then99.i
-; CHECK-NEXT:    xorl %ebx, %ebx
+; CHECK-NEXT:    jne .LBB1_10
+; CHECK-NEXT:  # %bb.9: # %while.cond.us1412.i
+; CHECK-NEXT:    decb %sil
+; CHECK-NEXT:    jne .LBB1_15
+; CHECK-NEXT:  .LBB1_10: # %if.end41.us1436.i
+; CHECK-NEXT:  .LBB1_14: # %if.then99.i
+; CHECK-NEXT:    xorl %r14d, %r14d
 ; CHECK-NEXT:    movl $.str.6, %edi
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    callq cli_dbgmsg
-; CHECK-NEXT:  .LBB1_26: # %cleanup
-; CHECK-NEXT:    movl %ebx, %eax
+; CHECK-NEXT:  .LBB1_15: # %cleanup
+; CHECK-NEXT:    movl %r14d, %eax
 ; CHECK-NEXT:    addq $8, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    retq
-; CHECK-NEXT:  .LBB1_27: # %wunpsect.exit.thread.loopexit389
-; CHECK-NEXT:  .LBB1_28: # %wunpsect.exit.thread.loopexit391
+; CHECK-NEXT:  .LBB1_22: # %wunpsect.exit.thread.loopexit389
+; CHECK-NEXT:  .LBB1_23: # %wunpsect.exit.thread.loopexit391
 entry:
   %0 = load i32, i32* undef, align 4
   %mul = shl nsw i32 %0, 2
   br i1 undef, label %if.end19, label %cleanup
 
 if.end19:                                         ; preds = %entry
   %conv = zext i32 %mul to i64
   %call = tail call i8* @cli_calloc(i64 %conv, i64 1)
   %1 = icmp eq i32 %exesz, 0
   %notrhs = icmp eq i32 %0, 0
   %or.cond117.not = or i1 %1, %notrhs
   %or.cond202 = or i1 %or.cond117.not, undef
-  %cmp35 = icmp ult i8* undef, %exe
+  %cmp35 = icmp ult i8* %call, %exe
   %or.cond203 = or i1 %or.cond202, %cmp35
   br i1 %or.cond203, label %cleanup, label %if.end50
 
 if.end50:                                         ; preds = %if.end19
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull %call, i8* undef, i64 %conv, i1 false)
   %cmp1.i.i = icmp ugt i32 %mul, 3
   br i1 %cmp1.i.i, label %shared_preheader, label %wunpsect.exit.thread.loopexit391
 
 shared_preheader:                                 ; preds = %if.end50
   br label %outer_loop_header
 
 outer_loop_header:                                ; preds = %outer_loop_latch, %shared_preheader
   %bits.1.i = phi i8 [ 32, %shared_preheader ], [ %bits.43.i, %outer_loop_latch ]
   %dst.0.ph.i = phi i8* [ undef, %shared_preheader ], [ %scevgep679.i, %outer_loop_latch ]
-  %2 = icmp eq i32 undef, 0
+  %2 = icmp eq i32 %0, 0
   br i1 %2, label %while.cond.us1412.i, label %shared_loop_header
 
 while.cond.us1412.i:                              ; preds = %outer_loop_header
   %.pre.i = add i8 %bits.1.i, -1
   %tobool2.us1420.i = icmp eq i8 %.pre.i, 0
   %or.cond.us1421.i = or i1 undef, %tobool2.us1420.i
   br i1 %or.cond.us1421.i, label %if.end41.us1436.i, label %cleanup
 

test/CodeGen/X86/tail-dup-repeat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 ; Function Attrs: uwtable
 ; When tail-duplicating during placement, we work backward from blocks with
 ; multiple successors. In this case, the block dup1 gets duplicated into dup2
 ; and if.then64, and then the block dup2 gets duplicated into land.lhs.true
 ; and if.end70
 
-define void @repeated_tail_dup(i1 %a1, i1 %a2, i32* %a4, i32* %a5, i8* %a6) #0 align 2 {
+define void @repeated_tail_dup(i1 %a1, i1 %a2, i32* %a4, i32* %a5, i8* %a6, i32 %a7) #0 align 2 {
 ; CHECK-LABEL: repeated_tail_dup:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %for.cond
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB0_3
 ; CHECK-NEXT:  # %bb.2: # %land.lhs.true
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    movl $10, (%rdx)
 ; CHECK-NEXT:    jmp .LBB0_6
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_3: # %if.end56
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    testb $1, %sil
 ; CHECK-NEXT:    je .LBB0_5
 ; CHECK-NEXT:  # %bb.4: # %if.then64
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    movb $1, (%r8)
-; CHECK-NEXT:    testl %eax, %eax
+; CHECK-NEXT:    testl %r9d, %r9d
 ; CHECK-NEXT:    je .LBB0_1
 ; CHECK-NEXT:    jmp .LBB0_8
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_5: # %if.end70
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    movl $12, (%rdx)
 ; CHECK-NEXT:  .LBB0_6: # %dup2
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
 ; CHECK-NEXT:    movl $2, (%rcx)
-; CHECK-NEXT:    testl %eax, %eax
+; CHECK-NEXT:    testl %r9d, %r9d
 ; CHECK-NEXT:    je .LBB0_1
 ; CHECK-NEXT:  .LBB0_8: # %for.end
 ; CHECK-NEXT:    retq
 entry:
   br label %for.cond
 
 for.cond:                                         ; preds = %dup1, %entry
   br i1 %a1, label %land.lhs.true, label %if.end56
if.end70:                                         ; preds = %if.end56
   br label %dup2
 
 dup2:                                             ; preds = %if.end70, %land.lhs.true
   store i32 2, i32* %a5, align 4
   br label %dup1
 
 dup1:                                             ; preds = %dup2, %if.then64
   %val = load i32, i32* %a4, align 8
-  %switch = icmp ult i32 undef, 1
+  %switch = icmp ult i32 %a7, 1
   br i1 %switch, label %for.cond, label %for.end
 
 for.end:                                          ; preds = %dup1
   ret void
 }
 
 attributes #0 = { uwtable }

test/CodeGen/X86/undef-ops.ll

 
 ; This would crash because the shift amount is an i8 operand,
 ; but the result of the shift is i32. We can't just propagate
 ; the existing undef as the result.
 
 define i1 @undef_operand_size_not_same_as_result() {
 ; CHECK-LABEL: undef_operand_size_not_same_as_result:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    testl %eax, %eax
-; CHECK-NEXT:    sete %al
+; CHECK-NEXT:    movb $1, %al
 ; CHECK-NEXT:    retq
   %sh = shl i32 7, undef
   %cmp = icmp eq i32 0, %sh
   ret i1 %cmp
 }
 

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

; CHECK-AVX512VL-NEXT:    retq
   %cmp = icmp eq <4 x i32> %urem, <i32 1, i32 0, i32 1, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_div_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-SSE-LABEL: test_urem_div_undef:
 ; CHECK-SSE:       # %bb.0:
-; CHECK-SSE-NEXT:    pxor %xmm0, %xmm0
-; CHECK-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
-; CHECK-SSE-NEXT:    psrld $31, %xmm0
+; CHECK-SSE-NEXT:    movaps {{.*#+}} xmm0 = [1,1,1,1]
 ; CHECK-SSE-NEXT:    retq
 ;
-; CHECK-AVX-LABEL: test_urem_div_undef:
-; CHECK-AVX:       # %bb.0:
-; CHECK-AVX-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    vpsrld $31, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    retq
+; CHECK-AVX1-LABEL: test_urem_div_undef:
+; CHECK-AVX1:       # %bb.0:
+; CHECK-AVX1-NEXT:    vmovaps {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX1-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_urem_div_undef:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vbroadcastss {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-AVX512VL-LABEL: test_urem_div_undef:
+; CHECK-AVX512VL:       # %bb.0:
+; CHECK-AVX512VL-NEXT:    vbroadcastss {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX512VL-NEXT:    retq
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 undef, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_comp_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-SSE2-LABEL: test_urem_comp_undef:
; CHECK-AVX512VL-NEXT:    retq
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_both_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-SSE-LABEL: test_urem_both_undef:
 ; CHECK-SSE:       # %bb.0:
-; CHECK-SSE-NEXT:    pxor %xmm0, %xmm0
-; CHECK-SSE-NEXT:    pcmpeqd %xmm0, %xmm0
-; CHECK-SSE-NEXT:    psrld $31, %xmm0
+; CHECK-SSE-NEXT:    movaps {{.*#+}} xmm0 = [1,1,1,1]
 ; CHECK-SSE-NEXT:    retq
 ;
-; CHECK-AVX-LABEL: test_urem_both_undef:
-; CHECK-AVX:       # %bb.0:
-; CHECK-AVX-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    vpcmpeqd %xmm0, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    vpsrld $31, %xmm0, %xmm0
-; CHECK-AVX-NEXT:    retq
+; CHECK-AVX1-LABEL: test_urem_both_undef:
+; CHECK-AVX1:       # %bb.0:
+; CHECK-AVX1-NEXT:    vmovaps {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX1-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_urem_both_undef:
+; CHECK-AVX2:       # %bb.0:
+; CHECK-AVX2-NEXT:    vbroadcastss {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX2-NEXT:    retq
+;
+; CHECK-AVX512VL-LABEL: test_urem_both_undef:
+; CHECK-AVX512VL:       # %bb.0:
+; CHECK-AVX512VL-NEXT:    vbroadcastss {{.*#+}} xmm0 = [1,1,1,1]
+; CHECK-AVX512VL-NEXT:    retq
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 undef, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_div_even_odd(<4 x i32> %X) nounwind readnone {
 ; CHECK-SSE2-LABEL: test_urem_div_even_odd:

test/CodeGen/X86/vec_int_to_fp-widen.ll

 ; VEX-NEXT:    movq %rax, %rcx
 ; VEX-NEXT:    shrq %rcx
 ; VEX-NEXT:    andl $1, %eax
 ; VEX-NEXT:    orq %rcx, %rax
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm0
 ; VEX-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; VEX-NEXT:  .LBB39_6:
 ; VEX-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; VEX-NEXT:    testq %rax, %rax
-; VEX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; VEX-NEXT:    js .LBB39_8
-; VEX-NEXT:  # %bb.7:
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm1
-; VEX-NEXT:  .LBB39_8:
 ; VEX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; VEX-NEXT:    retq
 ;
 ; AVX512F-LABEL: uitofp_2i64_to_4f32:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm1, %xmm1
 ; AVX512F-NEXT:    vmovq %xmm0, %rax

 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ssq %rax, %xmm1
 ; SSE2-NEXT:    addss %xmm1, %xmm1
 ; SSE2-NEXT:  .LBB41_6:
 ; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
-; SSE2-NEXT:    testq %rax, %rax
-; SSE2-NEXT:    xorps %xmm1, %xmm1
-; SSE2-NEXT:    js .LBB41_8
-; SSE2-NEXT:  # %bb.7:
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ssq %rax, %xmm1
-; SSE2-NEXT:  .LBB41_8:
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: uitofp_4i64_to_4f32_undef:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    pextrq $1, %xmm0, %rax
 ; SSE41-NEXT:    testq %rax, %rax
 ; SSE41-NEXT:    js .LBB41_1

 ; SSE41-NEXT:    shrq %rcx
 ; SSE41-NEXT:    andl $1, %eax
 ; SSE41-NEXT:    orq %rcx, %rax
 ; SSE41-NEXT:    xorps %xmm0, %xmm0
 ; SSE41-NEXT:    cvtsi2ssq %rax, %xmm0
 ; SSE41-NEXT:    addss %xmm0, %xmm0
 ; SSE41-NEXT:  .LBB41_6:
 ; SSE41-NEXT:    insertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; SSE41-NEXT:    testq %rax, %rax
-; SSE41-NEXT:    xorps %xmm1, %xmm1
-; SSE41-NEXT:    js .LBB41_8
-; SSE41-NEXT:  # %bb.7:
 ; SSE41-NEXT:    xorps %xmm1, %xmm1
 ; SSE41-NEXT:    cvtsi2ssq %rax, %xmm1
-; SSE41-NEXT:  .LBB41_8:
 ; SSE41-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; SSE41-NEXT:    retq
 ;
 ; VEX-LABEL: uitofp_4i64_to_4f32_undef:
 ; VEX:       # %bb.0:
 ; VEX-NEXT:    vpextrq $1, %xmm0, %rax
 ; VEX-NEXT:    testq %rax, %rax
 ; VEX-NEXT:    js .LBB41_1

 ; VEX-NEXT:    movq %rax, %rcx
 ; VEX-NEXT:    shrq %rcx
 ; VEX-NEXT:    andl $1, %eax
 ; VEX-NEXT:    orq %rcx, %rax
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm0
 ; VEX-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; VEX-NEXT:  .LBB41_6:
 ; VEX-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; VEX-NEXT:    testq %rax, %rax
-; VEX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; VEX-NEXT:    js .LBB41_8
-; VEX-NEXT:  # %bb.7:
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm1
-; VEX-NEXT:  .LBB41_8:
 ; VEX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; VEX-NEXT:    retq
 ;
 ; AVX512F-LABEL: uitofp_4i64_to_4f32_undef:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm1, %xmm1
 ; AVX512F-NEXT:    vmovq %xmm0, %rax

test/CodeGen/X86/vec_int_to_fp.ll

 ; VEX-NEXT:    movq %rax, %rcx
 ; VEX-NEXT:    shrq %rcx
 ; VEX-NEXT:    andl $1, %eax
 ; VEX-NEXT:    orq %rcx, %rax
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm0
 ; VEX-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; VEX-NEXT:  .LBB39_6:
 ; VEX-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; VEX-NEXT:    testq %rax, %rax
-; VEX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; VEX-NEXT:    js .LBB39_8
-; VEX-NEXT:  # %bb.7:
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm1
-; VEX-NEXT:  .LBB39_8:
 ; VEX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; VEX-NEXT:    retq
 ;
 ; AVX512F-LABEL: uitofp_2i64_to_4f32:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm1, %xmm1
 ; AVX512F-NEXT:    vmovq %xmm0, %rax

 ; SSE2-NEXT:    shrq %rcx
 ; SSE2-NEXT:    andl $1, %eax
 ; SSE2-NEXT:    orq %rcx, %rax
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ssq %rax, %xmm1
 ; SSE2-NEXT:    addss %xmm1, %xmm1
 ; SSE2-NEXT:  .LBB41_6:
 ; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
-; SSE2-NEXT:    testq %rax, %rax
-; SSE2-NEXT:    xorps %xmm1, %xmm1
-; SSE2-NEXT:    js .LBB41_8
-; SSE2-NEXT:  # %bb.7:
 ; SSE2-NEXT:    xorps %xmm1, %xmm1
 ; SSE2-NEXT:    cvtsi2ssq %rax, %xmm1
-; SSE2-NEXT:  .LBB41_8:
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: uitofp_4i64_to_4f32_undef:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    pextrq $1, %xmm0, %rax
 ; SSE41-NEXT:    testq %rax, %rax
 ; SSE41-NEXT:    js .LBB41_1

 ; SSE41-NEXT:    shrq %rcx
 ; SSE41-NEXT:    andl $1, %eax
 ; SSE41-NEXT:    orq %rcx, %rax
 ; SSE41-NEXT:    xorps %xmm0, %xmm0
 ; SSE41-NEXT:    cvtsi2ssq %rax, %xmm0
 ; SSE41-NEXT:    addss %xmm0, %xmm0
 ; SSE41-NEXT:  .LBB41_6:
 ; SSE41-NEXT:    insertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; SSE41-NEXT:    testq %rax, %rax
-; SSE41-NEXT:    xorps %xmm1, %xmm1
-; SSE41-NEXT:    js .LBB41_8
-; SSE41-NEXT:  # %bb.7:
 ; SSE41-NEXT:    xorps %xmm1, %xmm1
 ; SSE41-NEXT:    cvtsi2ssq %rax, %xmm1
-; SSE41-NEXT:  .LBB41_8:
 ; SSE41-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; SSE41-NEXT:    retq
 ;
 ; VEX-LABEL: uitofp_4i64_to_4f32_undef:
 ; VEX:       # %bb.0:
 ; VEX-NEXT:    vpextrq $1, %xmm0, %rax
 ; VEX-NEXT:    testq %rax, %rax
 ; VEX-NEXT:    js .LBB41_1

 ; VEX-NEXT:    movq %rax, %rcx
 ; VEX-NEXT:    shrq %rcx
 ; VEX-NEXT:    andl $1, %eax
 ; VEX-NEXT:    orq %rcx, %rax
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm0
 ; VEX-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; VEX-NEXT:  .LBB41_6:
 ; VEX-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[2,3]
-; VEX-NEXT:    testq %rax, %rax
-; VEX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
-; VEX-NEXT:    js .LBB41_8
-; VEX-NEXT:  # %bb.7:
 ; VEX-NEXT:    vcvtsi2ssq %rax, %xmm2, %xmm1
-; VEX-NEXT:  .LBB41_8:
 ; VEX-NEXT:    vshufps {{.*#+}} xmm0 = xmm0[0,1],xmm1[0,0]
 ; VEX-NEXT:    retq
 ;
 ; AVX512F-LABEL: uitofp_4i64_to_4f32_undef:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512F-NEXT:    vcvtusi2ssq %rax, %xmm1, %xmm1
 ; AVX512F-NEXT:    vmovq %xmm0, %rax

test/CodeGen/X86/vector-shift-ashr-sub128-widen.ll

 ; SSE2-NEXT:    andps %xmm2, %xmm0
 ; SSE2-NEXT:    psraw $1, %xmm1
 ; SSE2-NEXT:    andnps %xmm1, %xmm2
 ; SSE2-NEXT:    orps %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: constant_shift_v4i16:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    movdqa %xmm0, %xmm1
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    movdqa {{.*#+}} xmm2 = <0,1,2,3,u,u,u,u>
-; SSE41-NEXT:    pcmpeqw %xmm2, %xmm0
-; SSE41-NEXT:    movdqa {{.*#+}} xmm3 = <u,32768,16384,8192,u,u,u,u>
-; SSE41-NEXT:    pmulhw %xmm1, %xmm3
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm3
-; SSE41-NEXT:    pcmpeqw {{.*}}(%rip), %xmm2
-; SSE41-NEXT:    psraw $1, %xmm1
-; SSE41-NEXT:    movdqa %xmm2, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm3
-; SSE41-NEXT:    movdqa %xmm3, %xmm0
+; SSE41-NEXT:    movdqa {{.*#+}} xmm1 = <u,32768,16384,8192,u,u,u,u>
+; SSE41-NEXT:    pmulhw %xmm0, %xmm1
+; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
+; SSE41-NEXT:    psraw $1, %xmm0
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm1[0],xmm0[1],xmm1[2,3],xmm0[4,5,6,7]
 ; SSE41-NEXT:    retq
 ;
-; AVX1-LABEL: constant_shift_v4i16:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vmovddup {{.*#+}} xmm2 = [4.1720559249406128E-309,4.1720559249406128E-309]
-; AVX1-NEXT:    # xmm2 = mem[0,0]
-; AVX1-NEXT:    vpcmpeqw %xmm1, %xmm2, %xmm1
-; AVX1-NEXT:    vpmulhw {{.*}}(%rip), %xmm0, %xmm3
-; AVX1-NEXT:    vpblendvb %xmm1, %xmm0, %xmm3, %xmm1
-; AVX1-NEXT:    vpcmpeqw {{.*}}(%rip), %xmm2, %xmm2
-; AVX1-NEXT:    vpsraw $1, %xmm0, %xmm0
-; AVX1-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: constant_shift_v4i16:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX2-NEXT:    vpbroadcastq {{.*#+}} xmm2 = [844433520132096,844433520132096]
-; AVX2-NEXT:    vpcmpeqw %xmm1, %xmm2, %xmm1
-; AVX2-NEXT:    vpmulhw {{.*}}(%rip), %xmm0, %xmm3
-; AVX2-NEXT:    vpblendvb %xmm1, %xmm0, %xmm3, %xmm1
-; AVX2-NEXT:    vpcmpeqw {{.*}}(%rip), %xmm2, %xmm2
-; AVX2-NEXT:    vpsraw $1, %xmm0, %xmm0
-; AVX2-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
-; AVX2-NEXT:    retq
+; AVX-LABEL: constant_shift_v4i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpmulhw {{.*}}(%rip), %xmm0, %xmm1
+; AVX-NEXT:    vpblendw {{.*#+}} xmm1 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
+; AVX-NEXT:    vpsraw $1, %xmm0, %xmm0
+; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm1[0],xmm0[1],xmm1[2,3],xmm0[4,5,6,7]
+; AVX-NEXT:    retq
 ;
 ; XOP-LABEL: constant_shift_v4i16:
 ; XOP:       # %bb.0:
 ; XOP-NEXT:    vpshaw {{.*}}(%rip), %xmm0, %xmm0
 ; XOP-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: constant_shift_v4i16:
 ; AVX512DQ:       # %bb.0:

test/CodeGen/X86/vector-shift-lshr-sub128-widen.ll

 ; X32-SSE-NEXT:    retl
   %shift = lshr <2 x i32> %a, <i32 4, i32 5>
   ret <2 x i32> %shift
 }
 
 define <4 x i16> @constant_shift_v4i16(<4 x i16> %a) nounwind {
 ; SSE2-LABEL: constant_shift_v4i16:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movdqa {{.*#+}} xmm1 = <u,32768,16384,8192,u,u,u,u>
-; SSE2-NEXT:    pmulhuw %xmm0, %xmm1
-; SSE2-NEXT:    pxor %xmm2, %xmm2
-; SSE2-NEXT:    pcmpeqw {{.*}}(%rip), %xmm2
-; SSE2-NEXT:    pand %xmm2, %xmm0
-; SSE2-NEXT:    pandn %xmm1, %xmm2
-; SSE2-NEXT:    por %xmm2, %xmm0
+; SSE2-NEXT:    movdqa {{.*#+}} xmm2 = [65535,0,0,0,65535,65535,65535,65535]
+; SSE2-NEXT:    movdqa %xmm0, %xmm1
+; SSE2-NEXT:    pand %xmm2, %xmm1
+; SSE2-NEXT:    pmulhuw {{.*}}(%rip), %xmm0
+; SSE2-NEXT:    pandn %xmm0, %xmm2
+; SSE2-NEXT:    por %xmm2, %xmm1
+; SSE2-NEXT:    movdqa %xmm1, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: constant_shift_v4i16:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    movdqa %xmm0, %xmm1
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    pcmpeqw {{.*}}(%rip), %xmm0
-; SSE41-NEXT:    movdqa {{.*#+}} xmm2 = <u,32768,16384,8192,u,u,u,u>
-; SSE41-NEXT:    pmulhuw %xmm1, %xmm2
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    movdqa {{.*#+}} xmm1 = <u,32768,16384,8192,u,u,u,u>
+; SSE41-NEXT:    pmulhuw %xmm0, %xmm1
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: constant_shift_v4i16:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX-NEXT:    vpcmpeqw {{.*}}(%rip), %xmm1, %xmm1
-; AVX-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm2
-; AVX-NEXT:    vpblendvb %xmm1, %xmm0, %xmm2, %xmm0
+; AVX-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm1
+; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
 ; AVX-NEXT:    retq
 ;
 ; XOP-LABEL: constant_shift_v4i16:
 ; XOP:       # %bb.0:
 ; XOP-NEXT:    vpshlw {{.*}}(%rip), %xmm0, %xmm0
 ; XOP-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: constant_shift_v4i16:
 ; AVX512DQ:       # %bb.0:
-; AVX512DQ-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX512DQ-NEXT:    vpcmpeqw {{.*}}(%rip), %xmm1, %xmm1
-; AVX512DQ-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm2
-; AVX512DQ-NEXT:    vpblendvb %xmm1, %xmm0, %xmm2, %xmm0
+; AVX512DQ-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm1
+; AVX512DQ-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
 ; AVX512DQ-NEXT:    retq
 ;
 ; AVX512BW-LABEL: constant_shift_v4i16:
 ; AVX512BW:       # %bb.0:
 ; AVX512BW-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; AVX512BW-NEXT:    vmovdqa {{.*#+}} xmm1 = <0,1,2,3,u,u,u,u>
 ; AVX512BW-NEXT:    vpsrlvw %zmm1, %zmm0, %zmm0
 ; AVX512BW-NEXT:    # kill: def $xmm0 killed $xmm0 killed $zmm0
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQVL-LABEL: constant_shift_v4i16:
 ; AVX512DQVL:       # %bb.0:
-; AVX512DQVL-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX512DQVL-NEXT:    vpcmpeqw {{.*}}(%rip), %xmm1, %xmm1
-; AVX512DQVL-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm2
-; AVX512DQVL-NEXT:    vpblendvb %xmm1, %xmm0, %xmm2, %xmm0
+; AVX512DQVL-NEXT:    vpmulhuw {{.*}}(%rip), %xmm0, %xmm1
+; AVX512DQVL-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3],xmm0[4,5,6,7]
 ; AVX512DQVL-NEXT:    retq
 ;
 ; AVX512BWVL-LABEL: constant_shift_v4i16:
 ; AVX512BWVL:       # %bb.0:
 ; AVX512BWVL-NEXT:    vpsrlvw {{.*}}(%rip), %xmm0, %xmm0
 ; AVX512BWVL-NEXT:    retq
 ;
 ; X32-SSE-LABEL: constant_shift_v4i16:
 ; X32-SSE:       # %bb.0:
-; X32-SSE-NEXT:    movdqa {{.*#+}} xmm1 = <u,32768,16384,8192,u,u,u,u>
-; X32-SSE-NEXT:    pmulhuw %xmm0, %xmm1
-; X32-SSE-NEXT:    pxor %xmm2, %xmm2
-; X32-SSE-NEXT:    pcmpeqw {{\.LCPI.*}}, %xmm2
-; X32-SSE-NEXT:    pand %xmm2, %xmm0
-; X32-SSE-NEXT:    pandn %xmm1, %xmm2
-; X32-SSE-NEXT:    por %xmm2, %xmm0
+; X32-SSE-NEXT:    movdqa {{.*#+}} xmm2 = [65535,0,0,0,65535,65535,65535,65535]
+; X32-SSE-NEXT:    movdqa %xmm0, %xmm1
+; X32-SSE-NEXT:    pand %xmm2, %xmm1
+; X32-SSE-NEXT:    pmulhuw {{\.LCPI.*}}, %xmm0
+; X32-SSE-NEXT:    pandn %xmm0, %xmm2
+; X32-SSE-NEXT:    por %xmm2, %xmm1
+; X32-SSE-NEXT:    movdqa %xmm1, %xmm0
 ; X32-SSE-NEXT:    retl
   %shift = lshr <4 x i16> %a, <i16 0, i16 1, i16 2, i16 3>
   ret <4 x i16> %shift
 }
 
 define <2 x i16> @constant_shift_v2i16(<2 x i16> %a) nounwind {
 ; SSE2-LABEL: constant_shift_v2i16:
 ; SSE2:       # %bb.0: