[SelectionDAG][AArch64][X86] Move legalization of vector MULHS/MULHU from LegalizeDAG to LegalizeVectorOps

[craig.topper]

Summary

I believe we should be legalizing these with the rest of vector binary operations. If any custom lowering is required for these nodes, this will give the DAG combine between LegalizeVectorOps and LegalizeDAG to run on the custom code before constant build_vectors are lowered in LegalizeDAG.

Unfortunately, this regressed some AArch64 tests because that DAG combine is now combining (extract_subvector (build_vector)) into a narrower build_vector. This caused it to stop matching an isel pattern that was looking for a multipy and extract_subvector to generate u/smull2. Perhaps the MULHS/MULHU lowering should pick some new AArchISD node that can be directly matched u/smull2?

X86 is showing some weird code too due to aggressive constant broadcast formation using GPR even when it doesn't remove the BUILD_VECTOR completely.

[RKSimon]

If at all possible the X86 BROADCAST issue needs fixing first.

[RKSimon]

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

669

On AVX512 shouldn't these be using the broadcast load fold?

[craig.topper]

Handle AArch64 expansion with isel patterns instead of custom lowering. This prevents DAG combine from seeing the extract+build_vector opportunity.

Add a one use check to lowerShuffleVectorAsBroadcast in X86. This seems to reign in the aggressive broadcast formation. We now seem to be sharing constant pool entries a little better.

[RKSimon]

test/CodeGen/X86/combine-udiv.ll

726

It's a little odd that this hasn't been simplified to a VPSHUFB (or at least removed the xmm2 zero register - VPPERM can generates its own zero elements) - at that point this wouldn't be XOP specific at all......

[craig.topper]

test/CodeGen/X86/combine-udiv.ll

726

It looks like a packus was turned in vpperm before some zero_extend_vector_inreg and mul constant folding kicked in later in the same dag combine round. So at the time the VPPERM was created it wasn't known that it was zero. This all occurred on the DAG combine between vector op legalization and DAG legalization. This vpperm itself didn't get revisited after the input became 0 due to a bitcast. But the last DAG combine after legalize DAG should have revisited it. Should we have been able to recombine the VPPERM to a VPPERM with zero controls?

[craig.topper]

Rebase after r346697

[craig.topper]

test/CodeGen/X86/combine-udiv.ll

709

This is zero extending a truncate implemented with packuswb which means the upper bits before the truncate were already zero. So that's pretty silly. I'll file a bug.

[RKSimon]

test/CodeGen/X86/combine-udiv.ll

726

Yes either VPPERM with zero or just a VSHUFB (it'd be unary by then). The current codegen avoids the variable shuffles completely which is better still.

[craig.topper]

Ping

[craig.topper]

Ping

[RKSimon]

LGTM

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

case ISD::STRICT_FTRUNC:
     // is also legal, but if ISD::FSQRT requires expansion then so does
     // ISD::STRICT_FSQRT.
     Action = TLI.getStrictFPOperationAction(Node->getOpcode(),
                                             Node->getValueType(0));
     break;
   case ISD::ADD:
   case ISD::SUB:
   case ISD::MUL:
+  case ISD::MULHS:
+  case ISD::MULHU:
   case ISD::SDIV:
   case ISD::UDIV:
   case ISD::SREM:
   case ISD::UREM:
   case ISD::SDIVREM:
   case ISD::UDIVREM:
   case ISD::FADD:
   case ISD::FSUB:

test/CodeGen/AArch64/arm64-neon-mul-div-cte.ll

 ; RUN: llc < %s -verify-machineinstrs -mtriple=arm64-none-linux-gnu -mattr=+neon | FileCheck %s
 
 define <16 x i8> @div16xi8(<16 x i8> %x) {
 ; CHECK-LABEL: div16xi8:
-; CHECK:       movi   [[DIVISOR:(v[0-9]+)]].16b, #41
-; CHECK-NEXT:  smull2 [[SMULL2:(v[0-9]+)]].8h, v0.16b, [[DIVISOR]].16b
-; CHECK-NEXT:  smull  [[SMULL:(v[0-9]+)]].8h, v0.8b, [[DIVISOR]].8b
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).16b]], [[SMULL]].16b, [[SMULL2]].16b
+; CHECK:       movi   [[DIVISOR:(v[0-9]+)]].8b, #41
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  smull  [[SMULL1:(v[0-9]+)]].8h, v0.8b, [[DIVISOR]].8b
+; CHECK-NEXT:  smull  [[SMULL2:(v[0-9]+)]].8h, [[EXT]].8b, [[DIVISOR]].8b
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).16b]], [[SMULL1]].16b, [[SMULL2]].16b
 ; CHECK-NEXT:  sshr   [[SSHR:(v[0-9]+.16b)]], [[UZP2]], #2
 ; CHECK-NEXT:  usra   v0.16b, [[SSHR]], #7
   %div = sdiv <16 x i8> %x, <i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25, i8 25>
   ret <16 x i8> %div
 }
 
 define <8 x i16> @div8xi16(<8 x i16> %x) {
 ; CHECK-LABEL: div8xi16:
 ; CHECK:       mov    [[TMP:(w[0-9]+)]], #40815
-; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].8h, [[TMP]]
-; CHECK-NEXT:  smull2 [[SMULL2:(v[0-9]+)]].4s, v0.8h, [[DIVISOR]].8h
-; CHECK-NEXT:  smull  [[SMULL:(v[0-9]+)]].4s, v0.4h, [[DIVISOR]].4h
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).8h]], [[SMULL]].8h, [[SMULL2]].8h
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].4h, [[TMP]]
+; CHECK-NEXT:  smull  [[SMULL1:(v[0-9]+)]].4s, v0.4h, [[DIVISOR]].4h
+; CHECK-NEXT:  smull  [[SMULL2:(v[0-9]+)]].4s, [[EXT]].4h, [[DIVISOR]].4h
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).8h]], [[SMULL1]].8h, [[SMULL2]].8h
 ; CHECK-NEXT:  add    [[ADD:(v[0-9]+).8h]], [[UZP2]], v0.8h
 ; CHECK-NEXT:  sshr   [[SSHR:(v[0-9]+).8h]], [[ADD]], #12
 ; CHECK-NEXT:  usra   v0.8h, [[SSHR]], #15
   %div = sdiv <8 x i16> %x, <i16 6577, i16 6577, i16 6577, i16 6577, i16 6577, i16 6577, i16 6577, i16 6577>
   ret <8 x i16> %div
 }
 
 define <4 x i32> @div32xi4(<4 x i32> %x) {
 ; CHECK-LABEL: div32xi4:
 ; CHECK:       mov    [[TMP:(w[0-9]+)]], #7527
 ; CHECK-NEXT:  movk   [[TMP]], #28805, lsl #16
-; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].4s, [[TMP]]
-; CHECK-NEXT:  smull2 [[SMULL2:(v[0-9]+)]].2d, v0.4s, [[DIVISOR]].4s
-; CHECK-NEXT:  smull  [[SMULL:(v[0-9]+)]].2d, v0.2s, [[DIVISOR]].2s
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).4s]], [[SMULL]].4s, [[SMULL2]].4s
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].2s, [[TMP]]
+; CHECK-NEXT:  smull  [[SMULL1:(v[0-9]+)]].2d, v0.2s, [[DIVISOR]].2s
+; CHECK-NEXT:  smull  [[SMULL2:(v[0-9]+)]].2d, [[EXT]].2s, [[DIVISOR]].2s
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).4s]], [[SMULL1]].4s, [[SMULL2]].4s
 ; CHECK-NEXT:  sshr   [[SSHR:(v[0-9]+.4s)]], [[UZP2]], #22
 ; CHECK-NEXT:  usra   v0.4s, [[UZP2]], #31
   %div = sdiv <4 x i32> %x, <i32 9542677, i32 9542677, i32 9542677, i32 9542677>
   ret <4 x i32> %div
 }
 
 define <16 x i8> @udiv16xi8(<16 x i8> %x) {
 ; CHECK-LABEL: udiv16xi8:
-; CHECK:       movi   [[DIVISOR:(v[0-9]+)]].16b, #121
-; CHECK-NEXT:  umull2 [[UMULL2:(v[0-9]+)]].8h, v0.16b, [[DIVISOR]].16b
-; CHECK-NEXT:  umull  [[UMULL:(v[0-9]+)]].8h, v0.8b, [[DIVISOR]].8b
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).16b]], [[UMULL]].16b, [[UMULL2]].16b
+; CHECK:       movi   [[DIVISOR:(v[0-9]+)]].8b, #121
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  umull  [[UMULL1:(v[0-9]+)]].8h, v0.8b, [[DIVISOR]].8b
+; CHECK-NEXT:  umull  [[UMULL2:(v[0-9]+)]].8h, [[EXT]].8b, [[DIVISOR]].8b
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).16b]], [[UMULL1]].16b, [[UMULL2]].16b
 ; CHECK-NEXT:  ushr   v0.16b, [[UZP2]], #5
   %div = udiv <16 x i8> %x, <i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68, i8 68>
   ret <16 x i8> %div
 }
 
 define <8 x i16> @udiv8xi16(<8 x i16> %x) {
 ; CHECK-LABEL: udiv8xi16:
 ; CHECK:       mov    [[TMP:(w[0-9]+)]], #16593
-; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].8h, [[TMP]]
-; CHECK-NEXT:  umull2 [[UMULL2:(v[0-9]+)]].4s, v0.8h, [[DIVISOR]].8h
-; CHECK-NEXT:  umull  [[UMULL:(v[0-9]+)]].4s, v0.4h, [[DIVISOR]].4h
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).8h]], [[UMULL]].8h, [[SMULL2]].8h
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].4h, [[TMP]]
+; CHECK-NEXT:  umull  [[UMULL1:(v[0-9]+)]].4s, v0.4h, [[DIVISOR]].4h
+; CHECK-NEXT:  umull  [[UMULL2:(v[0-9]+)]].4s, [[EXT]].4h, [[DIVISOR]].4h
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).8h]], [[UMULL1]].8h, [[UMULL2]].8h
 ; CHECK-NEXT:  sub    [[SUB:(v[0-9]+).8h]], v0.8h, [[UZP2]]
 ; CHECK-NEXT:  usra   [[USRA:(v[0-9]+).8h]], [[SUB]], #1
 ; CHECK-NEXT:  ushr   v0.8h, [[USRA]], #12
   %div = udiv <8 x i16> %x, <i16 6537, i16 6537, i16 6537, i16 6537, i16 6537, i16 6537, i16 6537, i16 6537>
   ret <8 x i16> %div
 }
 
 define <4 x i32> @udiv32xi4(<4 x i32> %x) {
 ; CHECK-LABEL: udiv32xi4:
 ; CHECK:       mov    [[TMP:(w[0-9]+)]], #16747
 ; CHECK-NEXT:  movk   [[TMP]], #31439, lsl #16
-; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].4s, [[TMP]]
-; CHECK-NEXT:  umull2 [[UMULL2:(v[0-9]+)]].2d, v0.4s, [[DIVISOR]].4s
-; CHECK-NEXT:  umull  [[UMULL:(v[0-9]+)]].2d, v0.2s, [[DIVISOR]].2s
-; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).4s]], [[UMULL]].4s, [[SMULL2]].4s
+; CHECK-NEXT:  ext    [[EXT:(v[0-9]+)]].16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:  dup    [[DIVISOR:(v[0-9]+)]].2s, [[TMP]]
+; CHECK-NEXT:  umull  [[UMULL1:(v[0-9]+)]].2d, v0.2s, [[DIVISOR]].2s
+; CHECK-NEXT:  umull  [[UMULL2:(v[0-9]+)]].2d, [[EXT]].2s, [[DIVISOR]].2s
+; CHECK-NEXT:  uzp2   [[UZP2:(v[0-9]+).4s]], [[UMULL1]].4s, [[UMULL2]].4s
 ; CHECK-NEXT:  ushr   v0.4s, [[UZP2]], #22
   %div = udiv <4 x i32> %x, <i32 8743143, i32 8743143, i32 8743143, i32 8743143>
   ret <4 x i32> %div
 }

test/CodeGen/AArch64/neon-idiv.ll

 ; RUN: llc -mtriple=aarch64-none-linux-gnu < %s -mattr=+neon | FileCheck %s
 
 define <4 x i32> @test1(<4 x i32> %a) {
   %rem = srem <4 x i32> %a, <i32 7, i32 7, i32 7, i32 7>
   ret <4 x i32> %rem
 ; For C constant X/C is simplified to X-X/C*C. The X/C division is lowered
 ; to MULHS due the simplification by multiplying by a magic number
 ; (TargetLowering::BuildSDIV).
 ; CHECK-LABEL: test1:
-; CHECK: smull2 [[SMULL2:(v[0-9]+)]].2d, {{v[0-9]+}}.4s, {{v[0-9]+}}.4s
-; CHECK: smull  [[SMULL:(v[0-9]+)]].2d, {{v[0-9]+}}.2s, {{v[0-9]+}}.2s
-; CHECK: uzp2   [[UZP2:(v[0-9]+).4s]], [[SMULL]].4s, [[SMULL2]].4s
+; CHECK: ext    [[EXT:(v[0-9]+)]].16b, {{v[0-9]+}}.16b, {{v[0-9]+}}.16b, #8
+; CHECK: dup    [[DUP:(v[0-9]+)]].2s, w8
+; CHECK: smull  [[SMULL1:(v[0-9]+)]].2d, {{v[0-9]+}}.2s, [[DUP]].2s
+; CHECK: smull  [[SMULL2:(v[0-9]+)]].2d, [[EXT]].2s, [[DUP]].2s
+; CHECK: uzp2   [[UZP2:(v[0-9]+).4s]], [[SMULL1]].4s, [[SMULL2]].4s
 ; CHECK: add    [[ADD:(v[0-9]+.4s)]], [[UZP2]], v0.4s
 ; CHECK: sshr   [[SSHR:(v[0-9]+.4s)]], [[ADD]], #2
 }

test/CodeGen/AArch64/urem-seteq-vec-nonsplat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
 
 ; At the moment, BuildUREMEqFold does not handle nonsplat vectors.
 
 define <4 x i32> @test_urem_odd_div(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_odd_div:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    adrp x8, .LCPI0_0
-; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI0_0]
 ; CHECK-NEXT:    adrp x8, .LCPI0_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI0_1]
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI0_1]
+; CHECK-NEXT:    adrp x8, .LCPI0_0
+; CHECK-NEXT:    ldr d4, [x8, :lo12:.LCPI0_0]
+; CHECK-NEXT:    ext v2.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    adrp x8, .LCPI0_2
-; CHECK-NEXT:    umull2 v3.2d, v0.4s, v1.4s
-; CHECK-NEXT:    umull v1.2d, v0.2s, v1.2s
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v3.4s
-; CHECK-NEXT:    sub v3.4s, v0.4s, v1.4s
-; CHECK-NEXT:    umull2 v4.2d, v3.4s, v2.4s
-; CHECK-NEXT:    umull v2.2d, v3.2s, v2.2s
-; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI0_2]
+; CHECK-NEXT:    umull v1.2d, v2.2s, v1.2s
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI0_2]
+; CHECK-NEXT:    umull v4.2d, v0.2s, v4.2s
+; CHECK-NEXT:    uzp2 v1.4s, v4.4s, v1.4s
+; CHECK-NEXT:    movi v3.2d, #0000000000000000
+; CHECK-NEXT:    sub v4.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    adrp x8, .LCPI0_3
-; CHECK-NEXT:    uzp2 v2.4s, v2.4s, v4.4s
+; CHECK-NEXT:    umull v3.2d, v4.2s, v3.2s
+; CHECK-NEXT:    ext v4.16b, v4.16b, v4.16b, #8
+; CHECK-NEXT:    umull v2.2d, v4.2s, v2.2s
 ; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI0_3]
-; CHECK-NEXT:    neg v3.4s, v3.4s
+; CHECK-NEXT:    adrp x8, .LCPI0_4
+; CHECK-NEXT:    uzp2 v2.4s, v3.4s, v2.4s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI0_4]
+; CHECK-NEXT:    neg v4.4s, v4.4s
 ; CHECK-NEXT:    add v1.4s, v2.4s, v1.4s
-; CHECK-NEXT:    ushl v1.4s, v1.4s, v3.4s
-; CHECK-NEXT:    mls v0.4s, v1.4s, v4.4s
+; CHECK-NEXT:    ushl v1.4s, v1.4s, v4.4s
+; CHECK-NEXT:    mls v0.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 3, i32 5, i32 7, i32 9>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_even_div(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_even_div:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI1_0
 ; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI1_0]
 ; CHECK-NEXT:    adrp x8, .LCPI1_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI1_1]
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI1_1]
 ; CHECK-NEXT:    adrp x8, .LCPI1_2
-; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI1_2]
-; CHECK-NEXT:    neg v1.4s, v1.4s
+; CHECK-NEXT:    ldr d3, [x8, :lo12:.LCPI1_2]
 ; CHECK-NEXT:    adrp x8, .LCPI1_3
+; CHECK-NEXT:    neg v1.4s, v1.4s
+; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI1_3]
 ; CHECK-NEXT:    ushl v1.4s, v0.4s, v1.4s
-; CHECK-NEXT:    umull2 v4.2d, v1.4s, v2.4s
-; CHECK-NEXT:    umull v1.2d, v1.2s, v2.2s
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI1_3]
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v4.4s
-; CHECK-NEXT:    neg v3.4s, v3.4s
-; CHECK-NEXT:    ushl v1.4s, v1.4s, v3.4s
-; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
+; CHECK-NEXT:    adrp x8, .LCPI1_4
+; CHECK-NEXT:    umull v2.2d, v1.2s, v2.2s
+; CHECK-NEXT:    ext v1.16b, v1.16b, v1.16b, #8
+; CHECK-NEXT:    umull v1.2d, v1.2s, v3.2s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI1_4]
+; CHECK-NEXT:    uzp2 v1.4s, v2.4s, v1.4s
+; CHECK-NEXT:    neg v2.4s, v4.4s
+; CHECK-NEXT:    ushl v1.4s, v1.4s, v2.4s
+; CHECK-NEXT:    mls v0.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 6, i32 10, i32 12, i32 14>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_pow2(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_pow2:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI2_0
-; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI2_0]
+; CHECK-NEXT:    ldr d1, [x8, :lo12:.LCPI2_0]
 ; CHECK-NEXT:    adrp x8, .LCPI2_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI2_1]
+; CHECK-NEXT:    ldr d3, [x8, :lo12:.LCPI2_1]
 ; CHECK-NEXT:    adrp x8, .LCPI2_2
-; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI2_2]
-; CHECK-NEXT:    umull2 v4.2d, v0.4s, v1.4s
+; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI2_2]
+; CHECK-NEXT:    ext v2.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    adrp x8, .LCPI2_3
+; CHECK-NEXT:    umull v2.2d, v2.2s, v3.2s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI2_3]
 ; CHECK-NEXT:    umull v1.2d, v0.2s, v1.2s
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v4.4s
-; CHECK-NEXT:    neg v2.4s, v2.4s
+; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v2.4s
+; CHECK-NEXT:    neg v2.4s, v4.4s
 ; CHECK-NEXT:    ushl v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    mls v0.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 6, i32 10, i32 12, i32 16>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_one(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_one:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI3_0
 ; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI3_0]
 ; CHECK-NEXT:    adrp x8, .LCPI3_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI3_1]
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI3_1]
 ; CHECK-NEXT:    adrp x8, .LCPI3_2
-; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI3_2]
-; CHECK-NEXT:    neg v1.4s, v1.4s
+; CHECK-NEXT:    ldr d3, [x8, :lo12:.LCPI3_2]
 ; CHECK-NEXT:    adrp x8, .LCPI3_3
+; CHECK-NEXT:    neg v1.4s, v1.4s
+; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI3_3]
 ; CHECK-NEXT:    ushl v1.4s, v0.4s, v1.4s
-; CHECK-NEXT:    umull2 v4.2d, v1.4s, v2.4s
-; CHECK-NEXT:    umull v1.2d, v1.2s, v2.2s
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI3_3]
 ; CHECK-NEXT:    adrp x8, .LCPI3_4
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v4.4s
-; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI3_4]
-; CHECK-NEXT:    neg v3.4s, v3.4s
-; CHECK-NEXT:    ushl v1.4s, v1.4s, v3.4s
-; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
-; CHECK-NEXT:    mls v0.4s, v2.4s, v4.4s
+; CHECK-NEXT:    umull v2.2d, v1.2s, v2.2s
+; CHECK-NEXT:    ext v1.16b, v1.16b, v1.16b, #8
+; CHECK-NEXT:    umull v1.2d, v1.2s, v3.2s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI3_4]
+; CHECK-NEXT:    adrp x8, .LCPI3_5
+; CHECK-NEXT:    uzp2 v1.4s, v2.4s, v1.4s
+; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI3_5]
+; CHECK-NEXT:    neg v4.4s, v4.4s
+; CHECK-NEXT:    ushl v1.4s, v1.4s, v4.4s
+; CHECK-NEXT:    bsl v3.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mls v0.4s, v3.4s, v2.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 6, i32 1, i32 12, i32 14>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_comp(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_comp:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #52429
 ; CHECK-NEXT:    movk w8, #52428, lsl #16
-; CHECK-NEXT:    adrp x9, .LCPI4_0
-; CHECK-NEXT:    dup v2.4s, w8
-; CHECK-NEXT:    ldr q3, [x9, :lo12:.LCPI4_0]
-; CHECK-NEXT:    umull2 v4.2d, v0.4s, v2.4s
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    dup v2.2s, w8
+; CHECK-NEXT:    adrp x8, .LCPI4_0
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI4_0]
+; CHECK-NEXT:    umull v1.2d, v1.2s, v2.2s
 ; CHECK-NEXT:    umull v2.2d, v0.2s, v2.2s
-; CHECK-NEXT:    uzp2 v2.4s, v2.4s, v4.4s
-; CHECK-NEXT:    movi v1.4s, #5
-; CHECK-NEXT:    ushr v2.4s, v2.4s, #2
-; CHECK-NEXT:    mls v0.4s, v2.4s, v1.4s
+; CHECK-NEXT:    uzp2 v1.4s, v2.4s, v1.4s
+; CHECK-NEXT:    movi v2.4s, #5
+; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
+; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, v3.4s
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 5, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 1, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_both(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_both:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x8, .LCPI5_0
-; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI5_0]
+; CHECK-NEXT:    ldr d2, [x8, :lo12:.LCPI5_0]
 ; CHECK-NEXT:    adrp x8, .LCPI5_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI5_1]
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI5_1]
 ; CHECK-NEXT:    adrp x8, .LCPI5_2
-; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI5_2]
-; CHECK-NEXT:    umull2 v4.2d, v0.4s, v1.4s
-; CHECK-NEXT:    umull v1.2d, v0.2s, v1.2s
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v4.4s
+; CHECK-NEXT:    ldr q4, [x8, :lo12:.LCPI5_2]
+; CHECK-NEXT:    umull v1.2d, v1.2s, v2.2s
+; CHECK-NEXT:    umull v2.2d, v0.2s, v2.2s
+; CHECK-NEXT:    uzp2 v1.4s, v2.4s, v1.4s
 ; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
-; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
-; CHECK-NEXT:    cmeq v0.4s, v0.4s, v3.4s
+; CHECK-NEXT:    mls v0.4s, v1.4s, v3.4s
+; CHECK-NEXT:    cmeq v0.4s, v0.4s, v4.4s
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 6, i32 5, i32 6, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 1, i32 0, i32 1, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
; CHECK-NEXT:    ret
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_comp_undef(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_comp_undef:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #52429
 ; CHECK-NEXT:    movk w8, #52428, lsl #16
-; CHECK-NEXT:    dup v2.4s, w8
-; CHECK-NEXT:    umull2 v3.2d, v0.4s, v2.4s
-; CHECK-NEXT:    umull v2.2d, v0.2s, v2.2s
-; CHECK-NEXT:    uzp2 v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    movi v1.4s, #5
-; CHECK-NEXT:    ushr v2.4s, v2.4s, #2
-; CHECK-NEXT:    mls v0.4s, v2.4s, v1.4s
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    dup v3.2s, w8
+; CHECK-NEXT:    umull v4.2d, v0.2s, v3.2s
+; CHECK-NEXT:    umull v1.2d, v1.2s, v3.2s
+; CHECK-NEXT:    uzp2 v1.4s, v4.4s, v1.4s
+; CHECK-NEXT:    movi v2.4s, #5
+; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
+; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 5, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
; CHECK-NEXT:    ret
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 undef, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }
 
 define <4 x i32> @test_urem_div_even_odd(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_div_even_odd:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w8, #52429
+; CHECK-NEXT:    movk w8, #52428, lsl #16
+; CHECK-NEXT:    dup v1.2s, w8
+; CHECK-NEXT:    mov w8, #43691
+; CHECK-NEXT:    movk w8, #43690, lsl #16
+; CHECK-NEXT:    ext v2.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    dup v3.2s, w8
 ; CHECK-NEXT:    adrp x8, .LCPI9_0
-; CHECK-NEXT:    ldr q1, [x8, :lo12:.LCPI9_0]
-; CHECK-NEXT:    adrp x8, .LCPI9_1
-; CHECK-NEXT:    ldr q2, [x8, :lo12:.LCPI9_1]
-; CHECK-NEXT:    umull2 v3.2d, v0.4s, v1.4s
+; CHECK-NEXT:    umull v2.2d, v2.2s, v3.2s
+; CHECK-NEXT:    ldr q3, [x8, :lo12:.LCPI9_0]
 ; CHECK-NEXT:    umull v1.2d, v0.2s, v1.2s
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v3.4s
+; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
-; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
+; CHECK-NEXT:    mls v0.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 6, i32 6>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret
 }

test/CodeGen/AArch64/urem-seteq-vec-splat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
 
 ; Tests BuildUREMEqFold for 4 x i32 splat vectors with odd divisor.
 ; See urem-seteq.ll for justification behind constants emitted.
 define <4 x i32> @test_urem_odd_vec_i32(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_odd_vec_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #52429
 ; CHECK-NEXT:    movk w8, #52428, lsl #16
-; CHECK-NEXT:    dup v2.4s, w8
-; CHECK-NEXT:    umull2 v3.2d, v0.4s, v2.4s
-; CHECK-NEXT:    umull v2.2d, v0.2s, v2.2s
-; CHECK-NEXT:    uzp2 v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    movi v1.4s, #5
-; CHECK-NEXT:    ushr v2.4s, v2.4s, #2
-; CHECK-NEXT:    mls v0.4s, v2.4s, v1.4s
+; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; CHECK-NEXT:    dup v3.2s, w8
+; CHECK-NEXT:    umull v4.2d, v0.2s, v3.2s
+; CHECK-NEXT:    umull v1.2d, v1.2s, v3.2s
+; CHECK-NEXT:    uzp2 v1.4s, v4.4s, v1.4s
+; CHECK-NEXT:    movi v2.4s, #5
+; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
+; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 5, i32 5, i32 5, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 0, i32 0, i32 0, i32 0>
   %ret = zext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %ret

 ; The expected behavior is that the fold is _not_ applied
 ; because it requires a ROTR in the even case, which has to be expanded.
 define <4 x i32> @test_urem_even_vec_i32(<4 x i32> %X) nounwind readnone {
 ; CHECK-LABEL: test_urem_even_vec_i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #9363
 ; CHECK-NEXT:    movk w8, #37449, lsl #16
 ; CHECK-NEXT:    ushr v1.4s, v0.4s, #1
-; CHECK-NEXT:    dup v3.4s, w8
-; CHECK-NEXT:    umull2 v4.2d, v1.4s, v3.4s
+; CHECK-NEXT:    dup v3.2s, w8
+; CHECK-NEXT:    ext v4.16b, v1.16b, v1.16b, #8
 ; CHECK-NEXT:    umull v1.2d, v1.2s, v3.2s
-; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v4.4s
+; CHECK-NEXT:    umull v3.2d, v4.2s, v3.2s
+; CHECK-NEXT:    uzp2 v1.4s, v1.4s, v3.4s
 ; CHECK-NEXT:    movi v2.4s, #14
 ; CHECK-NEXT:    ushr v1.4s, v1.4s, #2
 ; CHECK-NEXT:    mls v0.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    movi v1.4s, #1
 ; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %urem = urem <4 x i32> %X, <i32 14, i32 14, i32 14, i32 14>

test/CodeGen/X86/combine-udiv.ll

 }
 
 define <16 x i8> @combine_vec_udiv_nonuniform4(<16 x i8> %x) {
 ; SSE2-LABEL: combine_vec_udiv_nonuniform4:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm1, %xmm1
 ; SSE2-NEXT:    movdqa %xmm0, %xmm2
 ; SSE2-NEXT:    punpcklbw {{.*#+}} xmm2 = xmm2[0],xmm1[0],xmm2[1],xmm1[1],xmm2[2],xmm1[2],xmm2[3],xmm1[3],xmm2[4],xmm1[4],xmm2[5],xmm1[5],xmm2[6],xmm1[6],xmm2[7],xmm1[7]
-; SSE2-NEXT:    movl $255, %eax
-; SSE2-NEXT:    movd %eax, %xmm1
 ; SSE2-NEXT:    movl $171, %eax
-; SSE2-NEXT:    movd %eax, %xmm3
-; SSE2-NEXT:    pand %xmm1, %xmm3
-; SSE2-NEXT:    pmullw %xmm2, %xmm3
-; SSE2-NEXT:    psrlw $8, %xmm3
-; SSE2-NEXT:    pmullw {{.*}}(%rip), %xmm3
-; SSE2-NEXT:    psrlw $8, %xmm3
-; SSE2-NEXT:    pand %xmm1, %xmm3
+; SSE2-NEXT:    movd %eax, %xmm1
+; SSE2-NEXT:    pmullw %xmm2, %xmm1
+; SSE2-NEXT:    psrlw $8, %xmm1
+; SSE2-NEXT:    pmullw {{.*}}(%rip), %xmm1
+; SSE2-NEXT:    psrlw $8, %xmm1
+; SSE2-NEXT:    movl $255, %eax
+; SSE2-NEXT:    movd %eax, %xmm2
+; SSE2-NEXT:    pand %xmm1, %xmm2
 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
-; SSE2-NEXT:    por %xmm3, %xmm0
+; SSE2-NEXT:    por %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: combine_vec_udiv_nonuniform4:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    movdqa %xmm0, %xmm1
 ; SSE41-NEXT:    movl $171, %eax
 ; SSE41-NEXT:    movd %eax, %xmm0
 ; SSE41-NEXT:    pmovzxbw {{.*#+}} xmm2 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
 ; SSE41-NEXT:    pmullw %xmm0, %xmm2
 ; SSE41-NEXT:    psrlw $8, %xmm2
 ; SSE41-NEXT:    movdqa %xmm2, %xmm0
-; SSE41-NEXT:    psllw $8, %xmm0
-; SSE41-NEXT:    pxor %xmm3, %xmm3
-; SSE41-NEXT:    packuswb %xmm3, %xmm2
-; SSE41-NEXT:    pmovzxbw {{.*#+}} xmm2 = xmm2[0],zero,xmm2[1],zero,xmm2[2],zero,xmm2[3],zero,xmm2[4],zero,xmm2[5],zero,xmm2[6],zero,xmm2[7],zero
-; SSE41-NEXT:    psllw $1, %xmm2
-; SSE41-NEXT:    pblendw {{.*#+}} xmm2 = xmm2[0],xmm0[1,2,3,4,5,6,7]
+; SSE41-NEXT:    psllw $1, %xmm0
+; SSE41-NEXT:    psllw $8, %xmm2
+; SSE41-NEXT:    pblendw {{.*#+}} xmm2 = xmm0[0],xmm2[1,2,3,4,5,6,7]
 ; SSE41-NEXT:    psrlw $8, %xmm2
-; SSE41-NEXT:    packuswb %xmm3, %xmm2
+; SSE41-NEXT:    pxor %xmm0, %xmm0
+; SSE41-NEXT:    packuswb %xmm0, %xmm2
 ; SSE41-NEXT:    movaps {{.*#+}} xmm0 = [0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
 ; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
 ; SSE41-NEXT:    movdqa %xmm2, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: combine_vec_udiv_nonuniform4:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    movl $171, %eax
 ; AVX1-NEXT:    vmovd %eax, %xmm1
 ; AVX1-NEXT:    vpmovzxbw {{.*#+}} xmm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; AVX1-NEXT:    vpmullw %xmm1, %xmm2, %xmm1
 ; AVX1-NEXT:    vpsrlw $8, %xmm1, %xmm1
-; AVX1-NEXT:    vpsllw $8, %xmm1, %xmm2
-; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
-; AVX1-NEXT:    vpackuswb %xmm3, %xmm1, %xmm1
-; AVX1-NEXT:    vpmovzxbw {{.*#+}} xmm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
-; AVX1-NEXT:    vpsllw $1, %xmm1, %xmm1
-; AVX1-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0],xmm2[1,2,3,4,5,6,7]
+; AVX1-NEXT:    vpsllw $1, %xmm1, %xmm2
+; AVX1-NEXT:    vpsllw $8, %xmm1, %xmm1
+; AVX1-NEXT:    vpblendw {{.*#+}} xmm1 = xmm2[0],xmm1[1,2,3,4,5,6,7]
 ; AVX1-NEXT:    vpsrlw $8, %xmm1, %xmm1
-; AVX1-NEXT:    vpackuswb %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; AVX1-NEXT:    vpackuswb %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
 ; AVX1-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: combine_vec_udiv_nonuniform4:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpmovzxbw {{.*#+}} ymm1 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero,xmm0[8],zero,xmm0[9],zero,xmm0[10],zero,xmm0[11],zero,xmm0[12],zero,xmm0[13],zero,xmm0[14],zero,xmm0[15],zero
 ; AVX2-NEXT:    movl $171, %eax
 ; AVX2-NEXT:    vmovd %eax, %xmm2
 ; AVX2-NEXT:    vpmullw %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vpsrlw $8, %ymm1, %ymm1
 ; AVX2-NEXT:    vextracti128 $1, %ymm1, %xmm2
 ; AVX2-NEXT:    vpackuswb %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vpmovzxbw {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero,xmm1[8],zero,xmm1[9],zero,xmm1[10],zero,xmm1[11],zero,xmm1[12],zero,xmm1[13],zero,xmm1[14],zero,xmm1[15],zero
 ; AVX2-NEXT:    vpmullw {{.*}}(%rip), %ymm1, %ymm1
 ; AVX2-NEXT:    vpsrlw $8, %ymm1, %ymm1
 ; AVX2-NEXT:    vextracti128 $1, %ymm1, %xmm2
 ; AVX2-NEXT:    vpackuswb %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
 ; AVX2-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; XOP-LABEL: combine_vec_udiv_nonuniform4:
 ; XOP:       # %bb.0:
 ; XOP-NEXT:    movl $171, %eax
 ; XOP-NEXT:    vmovd %eax, %xmm1
 ; XOP-NEXT:    vpmovzxbw {{.*#+}} xmm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; XOP-NEXT:    vpmullw %xmm1, %xmm2, %xmm2
-; XOP-NEXT:    vpshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
-; XOP-NEXT:    vpmovzxbw {{.*#+}} xmm3 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero,xmm3[4],zero,xmm3[5],zero,xmm3[6],zero,xmm3[7],zero
-; XOP-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
-; XOP-NEXT:    vpmovzxbw {{.*#+}} xmm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
-; XOP-NEXT:    vpmullw %xmm1, %xmm3, %xmm1
-; XOP-NEXT:    vpperm {{.*#+}} xmm1 = xmm2[1,3,5,7,9,11,13,15],xmm1[1,3,5,7,9,11,13,15]
+; XOP-NEXT:    vpmullw %xmm1, %xmm2, %xmm1
+; XOP-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; XOP-NEXT:    vpperm {{.*#+}} xmm1 = xmm1[1,3,5,7,9,11,13,15],xmm2[1,3,5,7,9,11,13,15]
 ; XOP-NEXT:    movl $249, %eax
 ; XOP-NEXT:    vmovd %eax, %xmm2
 ; XOP-NEXT:    vpshlb %xmm2, %xmm1, %xmm1
 ; XOP-NEXT:    vmovdqa {{.*#+}} xmm2 = [0,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
 ; XOP-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
 ; XOP-NEXT:    retq
   %div = udiv <16 x i8> %x, <i8 -64, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1, i8 1>
   ret <16 x i8> %div

test/CodeGen/X86/urem-seteq-vec-nonsplat.ll

 ; CHECK-SSE41-NEXT:    pmulld {{.*}}(%rip), %xmm1
 ; CHECK-SSE41-NEXT:    psubd %xmm1, %xmm0
 ; CHECK-SSE41-NEXT:    pcmpeqd {{.*}}(%rip), %xmm0
 ; CHECK-SSE41-NEXT:    psrld $31, %xmm0
 ; CHECK-SSE41-NEXT:    retq
 ;
 ; CHECK-AVX1-LABEL: test_urem_both:
 ; CHECK-AVX1:       # %bb.0:
-; CHECK-AVX1-NEXT:    vmovddup {{.*#+}} xmm1 = [-9.255967385052751E+61,-9.255967385052751E+61]
-; CHECK-AVX1-NEXT:    # xmm1 = mem[0,0]
+; CHECK-AVX1-NEXT:    vmovdqa {{.*#+}} xmm1 = [2863311531,3435973837,2863311531,3435973837]
+; CHECK-AVX1-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
+; CHECK-AVX1-NEXT:    vpshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
+; CHECK-AVX1-NEXT:    vpmuludq %xmm2, %xmm3, %xmm2
 ; CHECK-AVX1-NEXT:    vpmuludq %xmm1, %xmm0, %xmm1
 ; CHECK-AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
-; CHECK-AVX1-NEXT:    vpshufd {{.*#+}} xmm2 = xmm0[1,1,3,3]
-; CHECK-AVX1-NEXT:    vpmuludq {{.*}}(%rip), %xmm2, %xmm2
 ; CHECK-AVX1-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0,1],xmm2[2,3],xmm1[4,5],xmm2[6,7]
 ; CHECK-AVX1-NEXT:    vpsrld $2, %xmm1, %xmm1
 ; CHECK-AVX1-NEXT:    vpmulld {{.*}}(%rip), %xmm1, %xmm1
 ; CHECK-AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; CHECK-AVX1-NEXT:    vpcmpeqd {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-AVX1-NEXT:    vpsrld $31, %xmm0, %xmm0
 ; CHECK-AVX1-NEXT:    retq
 ;
 ; CHECK-AVX2-LABEL: test_urem_both:
 ; CHECK-AVX2:       # %bb.0:
-; CHECK-AVX2-NEXT:    vpbroadcastq {{.*#+}} xmm1 = [14757395262689946283,14757395262689946283]
-; CHECK-AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
-; CHECK-AVX2-NEXT:    vpshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
-; CHECK-AVX2-NEXT:    vpmuludq %xmm2, %xmm3, %xmm2
-; CHECK-AVX2-NEXT:    vpmuludq %xmm1, %xmm0, %xmm1
-; CHECK-AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
-; CHECK-AVX2-NEXT:    vpblendd {{.*#+}} xmm1 = xmm1[0],xmm2[1],xmm1[2],xmm2[3]
+; CHECK-AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[1,1,3,3]
+; CHECK-AVX2-NEXT:    movl $-858993459, %eax # imm = 0xCCCCCCCD
+; CHECK-AVX2-NEXT:    vmovd %eax, %xmm2
+; CHECK-AVX2-NEXT:    vpbroadcastd %xmm2, %xmm2
+; CHECK-AVX2-NEXT:    vpmuludq %xmm2, %xmm1, %xmm1
+; CHECK-AVX2-NEXT:    vpmuludq {{.*}}(%rip), %xmm0, %xmm2
+; CHECK-AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm2[1,1,3,3]
+; CHECK-AVX2-NEXT:    vpblendd {{.*#+}} xmm1 = xmm2[0],xmm1[1],xmm2[2],xmm1[3]
 ; CHECK-AVX2-NEXT:    vpsrld $2, %xmm1, %xmm1
 ; CHECK-AVX2-NEXT:    vpmulld {{.*}}(%rip), %xmm1, %xmm1
 ; CHECK-AVX2-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; CHECK-AVX2-NEXT:    vpcmpeqd {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-AVX2-NEXT:    vpsrld $31, %xmm0, %xmm0
 ; CHECK-AVX2-NEXT:    retq
 ;
 ; CHECK-AVX512VL-LABEL: test_urem_both:
 ; CHECK-AVX512VL:       # %bb.0:
-; CHECK-AVX512VL-NEXT:    vpbroadcastq {{.*#+}} xmm1 = [14757395262689946283,14757395262689946283]
-; CHECK-AVX512VL-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
-; CHECK-AVX512VL-NEXT:    vpshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
-; CHECK-AVX512VL-NEXT:    vpmuludq %xmm2, %xmm3, %xmm2
-; CHECK-AVX512VL-NEXT:    vpmuludq %xmm1, %xmm0, %xmm1
-; CHECK-AVX512VL-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
-; CHECK-AVX512VL-NEXT:    vpblendd {{.*#+}} xmm1 = xmm1[0],xmm2[1],xmm1[2],xmm2[3]
+; CHECK-AVX512VL-NEXT:    movl $-858993459, %eax # imm = 0xCCCCCCCD
+; CHECK-AVX512VL-NEXT:    vpbroadcastd %eax, %xmm1
+; CHECK-AVX512VL-NEXT:    vpshufd {{.*#+}} xmm2 = xmm0[1,1,3,3]
+; CHECK-AVX512VL-NEXT:    vpmuludq %xmm1, %xmm2, %xmm1
+; CHECK-AVX512VL-NEXT:    vpmuludq {{.*}}(%rip), %xmm0, %xmm2
+; CHECK-AVX512VL-NEXT:    vpshufd {{.*#+}} xmm2 = xmm2[1,1,3,3]
+; CHECK-AVX512VL-NEXT:    vpblendd {{.*#+}} xmm1 = xmm2[0],xmm1[1],xmm2[2],xmm1[3]
 ; CHECK-AVX512VL-NEXT:    vpsrld $2, %xmm1, %xmm1
 ; CHECK-AVX512VL-NEXT:    vpmulld {{.*}}(%rip), %xmm1, %xmm1
 ; CHECK-AVX512VL-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; CHECK-AVX512VL-NEXT:    vpcmpeqd {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-AVX512VL-NEXT:    vpsrld $31, %xmm0, %xmm0
 ; CHECK-AVX512VL-NEXT:    retq
   %urem = urem <4 x i32> %X, <i32 6, i32 5, i32 6, i32 5>
   %cmp = icmp eq <4 x i32> %urem, <i32 1, i32 0, i32 1, i32 0>