[DAG] Match USUBSAT patterns through zext/trunc

[RKSimon]

Summary

This patch handles usubsat patterns hidden through zext/trunc and uses the getTruncatedUSUBSAT helper to determine if the USUBSAT can be correctly performed in the truncated form:

zext(x) >= y ? x - trunc(y) : 0 --> usubsat(x,trunc(umin(y,SatLimit)))
zext(x) >  y ? x - trunc(y) : 0 --> usubsat(x,trunc(umin(y,SatLimit)))

Based on original examples:

void foo(unsigned short *p, int max, int n) {
    int i;
    unsigned m;
    for (i = 0; i < n; i++) {
        m = *--p;
        *p = (unsigned short)(m >= max ? m-max : 0);
    }
}

[yulia_koval]

Gentle ping

[RKSimon]

Please can you regenerate the psubus.ll tests with utils/update_llc_test_checks.py

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

27379 ↗

(On Diff #75873)

If you replace this with VT.getScalarSizeInBits() then clang-format might be able to do a better job

[yulia_koval]

1. Changed regex-based test to generated from update_llc_test_checks.py script
2. Replaced getScalarType().getSizeInBits() with VT.getScalarSizeInBits()

[RKSimon]

Please can you rebase to trunk latest? I've added the current codegen from your tests and your patch appears to introduce scalarization regressions for SSE targets.

[yulia_koval]

This optimization requires the umin instruction. It didn't exist before SSE4.1, so I added requirement for SSE4 for it and added new target to test. I haven't found profitable equivalent for umin in sse\sse2 for this situation.

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

28958 ↗

(On Diff #83614)

SSE2 has some min/max support - replace the hasSSE41 requirement with TLI.isOperationLegalOrCustom() ?

[yulia_koval]

I updated the patch to trunk, because test output changed due to rL292479 optimization. I failed to find a better check, based on TLI.isLegal.., because it doesn't requires exact type. For example, in case of SSE4.1, isOperationLegalOrCustom(MIN, v8i32) returns false and disables the optimization. However, the transformation works fine as the illegal operation is split into two v4i32 operations. The hasSSE41 check is sufficient, because it includes both v8i16 and v4i32 for each integer type. i8(SSE hass i8 based umin) can't be used in this optimisation, because the ExtType is always larger then psubus type and can be minimum i16.

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

29378 ↗

(On Diff #85883)

clang-format

[yulia_koval]

Sorry, fixed clang-format.

[yulia_koval]

Gentle ping

[spatel]

It's difficult for me to see what's going on in these tests because there are a bunch of irrelevant load/store/gep/bitcast instructions. Can you remove those?

If you do that, the root of the problem is very similar to:
https://reviews.llvm.org/rL286776

Ie, if we recognized the umin/umax patterns in IR as ISD::UMIN/ISD::UMAX when we created the DAG, the subsequent matching to the x86-specific PSUBUS might already happen with the existing lowering?

[yulia_koval]

In D25987#673846, @spatel wrote:

It's difficult for me to see what's going on in these tests because there are a bunch of irrelevant load/store/gep/bitcast instructions. Can you remove those?

If you do that, the root of the problem is very similar to:
https://reviews.llvm.org/rL286776

Ie, if we recognized the umin/umax patterns in IR as ISD::UMIN/ISD::UMAX when we created the DAG, the subsequent matching to the x86-specific PSUBUS might already happen with the existing lowering?

I fixed tests(https://reviews.llvm.org/D32643). New revision is based on this updated tests revision.

We have (i16) (a>=b ? a - b : 0), where both a and b are 32 bit. We can't apply max here, because unsigned max between something and zero doesn't make sense. And we can't use signed max, because we can't fit 32bit unsigned values in it.

On the other hand, the result we select is 16bit PSUBUS, which only exists for 16bit and 8bit vector elements, not for 32bit. This optimization truncates vectors to 16bit, using the information, that it was zero extended and this vector is on the left side of the cmp. We use that information to substitute b with min(b, max_16bit_type). and then we can truncate a(it can be truncated, because it is zero extended value) and b(because we truncated it properly with our min). In this case the result of the sub will be valid when a >=b and not significant if not(the actual result is zero). This pattern can be then recognized as PSUBUS instruction.

This transformation is profitable only if PSUBUS exists. And this function is the place. where PSUBUS is selected and we can be 100% sure, that it will be profitable. This transformation is not profitable in general.

So in pseudo code it will be:
a >= b ? a - b : 0
transformed to:
trunc(a) >= min(b, 0xFFFF) ? trunc(a) - min(b, 0xFFFF) : 0 ->the result of the patch
This result is then transformed to PSUBUS by existing code.

[spatel]

I think we need to take a step back here. The tests are assuming a specific form of IR, but I'm not sure it's canonical. If we change this upstream (in InstCombine), the transform may fail to activate when you hoped it would. It's also possible that generic DAG combines could interfere with the pattern you're expecting. See rL301781 as an example. In either case, this seems like a fragile solution because the pattern can change above this.

http://rise4fun.com/Alive/5e4
Unless I made a mistake in transcribing that, the 1st form is what you're expecting in these tests. The 2nd form might be considered better since it includes a canonical 'max'.

Can you test this patch with more examples like this to see if it behaves as expected:

define i16 @scalar(i16 %x, i32 %y) nounwind {
  %lhs = zext i16 %x to i32
  %cond = icmp ugt i32 %lhs, %y
  %sub = sub i32 %lhs, %y
  %truncsub = trunc i32 %sub to i16
  %res = select i1 %cond, i16 %truncsub, i16 zeroinitializer
  ret i16 %res
}

define i16 @scalar_max(i16 %x, i32 %y) nounwind {
  %lhs = zext i16 %x to i32
  %cond = icmp ugt i32 %lhs, %y
  %umax = select i1 %cond, i32 %lhs, i32 %y
  %sub = sub i32 %umax, %y
  %truncsub = trunc i32 %sub to i16
  ret i16 %truncsub
}

define <8 x i16> @test15(<8 x i16> %x, <8 x i32> %y) nounwind {
  %lhs = zext <8 x i16> %x to <8 x i32>
  %cond = icmp ugt <8 x i32> %lhs, %y
  %sub = sub <8 x i32> %lhs, %y
  %truncsub = trunc <8 x i32> %sub to <8 x i16>
  %res = select <8 x i1> %cond, <8 x i16> %truncsub, <8 x i16> zeroinitializer
  ret <8 x i16> %res
}

define <8 x i16> @test15_max(<8 x i16> %x, <8 x i32> %y) nounwind {
  %lhs = zext <8 x i16> %x to <8 x i32>
  %cond = icmp ugt <8 x i32> %lhs, %y
  %umax = select <8 x i1> %cond, <8 x i32> %lhs, <8 x i32> %y
  %sub = sub <8 x i32> %umax, %y
  %truncsub = trunc <8 x i32> %sub to <8 x i16>
  ret <8 x i16> %truncsub
}

define <8 x i8> @test15_narrow(<8 x i8> %x, <8 x i16> %y) nounwind {
  %lhs = zext <8 x i8> %x to <8 x i16>
  %cond = icmp ugt <8 x i16> %lhs, %y
  %sub = sub <8 x i16> %lhs, %y
  %truncsub = trunc <8 x i16> %sub to <8 x i8>
  %res = select <8 x i1> %cond, <8 x i8> %truncsub, <8 x i8> zeroinitializer
  ret <8 x i8> %res
}

define <8 x i8> @test15_narrow_max(<8 x i8> %x, <8 x i16> %y) nounwind {
  %lhs = zext <8 x i8> %x to <8 x i16>
  %cond = icmp ugt <8 x i16> %lhs, %y
  %umax = select <8 x i1> %cond, <8 x i16> %lhs, <8 x i16> %y
  %sub = sub <8 x i16> %umax, %y
  %truncsub = trunc <8 x i16> %sub to <8 x i8>
  ret <8 x i8> %truncsub
}

[yulia_koval]

In D25987#742378, @spatel wrote:

I think we need to take a step back here. The tests are assuming a specific form of IR, but I'm not sure it's canonical. If we change this upstream (in InstCombine), the transform may fail to activate when you hoped it would. It's also possible that generic DAG combines could interfere with the pattern you're expecting. See rL301781 as an example. In either case, this seems like a fragile solution because the pattern can change above this.

http://rise4fun.com/Alive/5e4
Unless I made a mistake in transcribing that, the 1st form is what you're expecting in these tests. The 2nd form might be considered better since it includes a canonical 'max'.

Can you test this patch with more examples like this to see if it behaves as expected:

define i16 @scalar(i16 %x, i32 %y) nounwind {
  %lhs = zext i16 %x to i32
  %cond = icmp ugt i32 %lhs, %y
  %sub = sub i32 %lhs, %y
  %truncsub = trunc i32 %sub to i16
  %res = select i1 %cond, i16 %truncsub, i16 zeroinitializer
  ret i16 %res
}

define i16 @scalar_max(i16 %x, i32 %y) nounwind {
  %lhs = zext i16 %x to i32
  %cond = icmp ugt i32 %lhs, %y
  %umax = select i1 %cond, i32 %lhs, i32 %y
  %sub = sub i32 %umax, %y
  %truncsub = trunc i32 %sub to i16
  ret i16 %truncsub
}

define <8 x i16> @test15(<8 x i16> %x, <8 x i32> %y) nounwind {
  %lhs = zext <8 x i16> %x to <8 x i32>
  %cond = icmp ugt <8 x i32> %lhs, %y
  %sub = sub <8 x i32> %lhs, %y
  %truncsub = trunc <8 x i32> %sub to <8 x i16>
  %res = select <8 x i1> %cond, <8 x i16> %truncsub, <8 x i16> zeroinitializer
  ret <8 x i16> %res
}

define <8 x i16> @test15_max(<8 x i16> %x, <8 x i32> %y) nounwind {
  %lhs = zext <8 x i16> %x to <8 x i32>
  %cond = icmp ugt <8 x i32> %lhs, %y
  %umax = select <8 x i1> %cond, <8 x i32> %lhs, <8 x i32> %y
  %sub = sub <8 x i32> %umax, %y
  %truncsub = trunc <8 x i32> %sub to <8 x i16>
  ret <8 x i16> %truncsub
}

define <8 x i8> @test15_narrow(<8 x i8> %x, <8 x i16> %y) nounwind {
  %lhs = zext <8 x i8> %x to <8 x i16>
  %cond = icmp ugt <8 x i16> %lhs, %y
  %sub = sub <8 x i16> %lhs, %y
  %truncsub = trunc <8 x i16> %sub to <8 x i8>
  %res = select <8 x i1> %cond, <8 x i8> %truncsub, <8 x i8> zeroinitializer
  ret <8 x i8> %res
}

define <8 x i8> @test15_narrow_max(<8 x i8> %x, <8 x i16> %y) nounwind {
  %lhs = zext <8 x i8> %x to <8 x i16>
  %cond = icmp ugt <8 x i16> %lhs, %y
  %umax = select <8 x i1> %cond, <8 x i16> %lhs, <8 x i16> %y
  %sub = sub <8 x i16> %umax, %y
  %truncsub = trunc <8 x i16> %sub to <8 x i8>
  ret <8 x i8> %truncsub
}

Well, first 2 tests don't work with my patch because PSUBUS is a vector instruction. Test15 is working perfectly. Test15_max doesn't work, because I didn't try to recognize this pattern(the one you showed in a link), after optimization, because currently(for at least a year) llvm generates the code like in my tests, for example for psubus.c:

void foo(unsigned short *p, int max, int n)
{
int i;
unsigned m;
for (i = 0; i < n; i++) {

m = *--p;
*p = (unsigned short)(m >= max ? m-max : 0);

}
}

IR will be:

%10 = zext <8 x i16> %reverse to <8 x i32>
%11 = icmp ugt <8 x i32> %broadcast.splat16, %10
%12 = sub <8 x i32> %10, %broadcast.splat16
%13 = trunc <8 x i32> %12 to <8 x i16>
%14 = select <8 x i1> %11, <8 x i16> zeroinitializer, <8 x i16> %13

I see, that optimization you provided in http://rise4fun.com/Alive/5e4 is valid, but it extends from i16 to i32 which may trigger some regressions, because it switches to wider operations. Maybe something like this will appear in the future, but currently the pattern above is generated. Also, whether the pattern generated from this particular C code will be changed or not, this patch will still be useful for this IR pattern, if it appears. I can add recognition of the test15_max pattern as well, should I do it?

Test15_narrow converted to psubus before my patch and test15_narrow_max converted to max before my patch. The only test, changed by my patch is @test15.

Looks like rL301781 doesn't affect my patch at any tests..
Also if some optimization replace select(sub) with sub(select), won't it ruin any other psubus pattern as well?

[spatel]

In D25987#743378, @yulia_koval wrote:

Well, first 2 tests don't work with my patch because PSUBUS is a vector instruction. Test15 is working perfectly. Test15_max doesn't work, because I didn't try to recognize this pattern(the one you showed in a link), after optimization, because currently(for at least a year) llvm generates the code like in my tests, for example for psubus.c:

It is irrelevant how long the current IR has been produced. It is also irrelevant what form the current IR takes until we agree that it is the canonical form. Let me try to explain.

void foo(unsigned short *p, int max, int n)
{
int i;
unsigned m;
for (i = 0; i < n; i++) {

m = *--p;
*p = (unsigned short)(m >= max ? m-max : 0);

}
}

IR will be:

%10 = zext <8 x i16> %reverse to <8 x i32>
%11 = icmp ugt <8 x i32> %broadcast.splat16, %10
%12 = sub <8 x i32> %10, %broadcast.splat16
%13 = trunc <8 x i32> %12 to <8 x i16>
%14 = select <8 x i1> %11, <8 x i16> zeroinitializer, <8 x i16> %13

This looks good. I'm glad we can vectorize this. :)

But do you agree that it is reasonable to write this function in C as:

void goo(unsigned short *p, int max, int n) {
  int i;
  unsigned m;
  for (i = 0; i < n; i++) {
    m = *--p;
    unsigned umax = m > max ? m : max;
    *p = (unsigned short)(umax - max);
  }
}

If yes, then I think you would also agree that LLVM should optimize that code in exactly the same way (to eventually produce a psubus when compiled for x86 -msse4.1). Now, if you agree with that, then the question is where should we recognize that these 2 functions are logically equivalent? I'm saying that it should occur in IR because that's the main job of IR: to translate source code into a canonical form that the backend can then optimize to the best machine-code for a given target.

I see, that optimization you provided in http://rise4fun.com/Alive/5e4 is valid, but it extends from i16 to i32 which may trigger some regressions, because it switches to wider operations. Maybe something like this will appear in the future, but currently the pattern above is generated. Also, whether the pattern generated from this particular C code will be changed or not, this patch will still be useful for this IR pattern, if it appears. I can add recognition of the test15_max pattern as well, should I do it?

No. Until we decide what the canonical IR for this pattern looks like, you may be adding a useless transform (dead code) to the backend. "The future" where we no longer produce the IR pattern that this patch is looking for could be today, so let's not prematurely add a backend optimization until we know that pattern will actually exist. You're suggesting that we match multiple patterns in the backend to produce psubus. I'm suggesting that if we can canonicalize this pattern in IR, then we will more efficiently optimize it in the backend regardless of how it was written in source. Does this make sense?

Also if some optimization replace select(sub) with sub(select), won't it ruin any other psubus pattern as well?

It is certainly possible. We should investigate the current patterns that match to psubus. As shown in your examples in this patch and my follow-up test15_max, the matching of psubus is not complete. Will we break the existing matching by canonicalizing the IR? If yes, we should add those transforms first to avoid regressions.

So the steps to fix this properly:

1. Decide what is the canonical IR.
2. Add codegen tests for that IR (this should likely include more than x86).
3. Optimize those patterns in the backend (Do other targets have instructions equivalent to psubus? If yes, a generic DAGCombiner transform should be added.)
4. Add the canonicalization to IR.

This should ensure that no matter how the source is written, we'll get optimal codegen for all targets, and we'll do that efficiently.

[yulia_koval]

In D25987#743482, @spatel wrote:

But do you agree that it is reasonable to write this function in C as:

void goo(unsigned short *p, int max, int n) {
  int i;
  unsigned m;
  for (i = 0; i < n; i++) {
    m = *--p;
    unsigned umax = m > max ? m : max;
    *p = (unsigned short)(umax - max);
  }
}

This function's IR is

%10 = zext <8 x i16> %reverse to <8 x i32>
%11 = icmp ult <8 x i32> %broadcast.splat18, %10
%12 = select <8 x i1> %11, <8 x i32> %10, <8 x i32> %broadcast.splat18
%13 = sub <8 x i32> %12, %broadcast.splat18
%14 = trunc <8 x i32> %13 to <8 x i16>

I agree, that it looks better as canonical, then my example, when trunc is in between the pattern's instructions. What do you think about defining trunc(sub(select())) canonical for this pattern? Then we can canonicalize my initial pattern using the InstCombine patch I attached(https://reviews.llvm.org/D33118) - it looks like this way the .ll code generated for both of them would be the same. After this transformation, the initial pattern can be transformed into 32bit sub(max). I haven't yet investigated how to transform it into psubus, but at least max is more profitable then currently generated instructions.

What do you think about this idea?

[spatel]

In D25987#752968, @yulia_koval wrote:

I agree, that it looks better as canonical, then my example, when trunc is in between the pattern's instructions. What do you think about defining trunc(sub(select())) canonical for this pattern? Then we can canonicalize my initial pattern using the InstCombine patch I attached(https://reviews.llvm.org/D33118) - it looks like this way the .ll code generated for both of them would be the same. After this transformation, the initial pattern can be transformed into 32bit sub(max). I haven't yet investigated how to transform it into psubus, but at least max is more profitable then currently generated instructions.

What do you think about this idea?

I think it's a good idea to form canonical min/max in IR when possible. There is precedent for a canonicalization that is similar to this one:
http://lists.llvm.org/pipermail/llvm-dev/2016-November/106868.html

But please send a note to llvm-dev requesting feedback for this canonicalization. We want to make sure there are no objections before proceeding.

I think the example can be minimized to this:
http://rise4fun.com/Alive/qBn

(ie, the leading zext is not needed, and this pattern is not limited to vectors)

[RKSimon]

@yulia_koval What is the state of this after D37510 and D37534?

[yulia_koval]

In D25987#908551, @RKSimon wrote:

@yulia_koval What is the state of this after D37510 and D37534?

The one additional patch is required to apply the changes - cannonicalization part. I will submit it soon.

[spatel]

Did D41480 affect anything with these tests? We still need to handle 'umin' patterns in IR and update the tests here with canonical IR forms to show psubus codegen?

[yulia_koval]

In D25987#1005390, @spatel wrote:

Did D41480 affect anything with these tests? We still need to handle 'umin' patterns in IR and update the tests here with canonical IR forms to show psubus codegen?

Looks like all possible umin patterns are already handled in existing patterns we added for max. The last tests in psubus.ll is also cannonical cmp-select-sub sequence. What should I add which is missing?

[RKSimon]

We've added custom lowering support for all (US)MIN/MAX vector types, plus we have improved saturation packing so you might be able to relax the SSE41 requirement - please can you check the codegen on SSE2/SSSE3 if you allow the combines?

[spatel]

In D25987#1028292, @yulia_koval wrote:

In D25987#1005390, @spatel wrote:

Did D41480 affect anything with these tests? We still need to handle 'umin' patterns in IR and update the tests here with canonical IR forms to show psubus codegen?

Looks like all possible umin patterns are already handled in existing patterns we added for max. The last tests in psubus.ll is also cannonical cmp-select-sub sequence. What should I add which is missing?

What does this comment in InstCombineSelect refer to?
/// TODO: Also support a - UMIN(a,b) patterns.

Also, there's a proposal to remove the intrinsics now - D44785. How does that relate to whatever is left to do here?

[tkrupa]

In D25987#1045827, @spatel wrote:

Also, there's a proposal to remove the intrinsics now - D44785. How does that relate to whatever is left to do here?

It turns out there is a canonical represention of subus in place. D44785 patch does not take it into account - I'm going to modify it so that there's no conflict.

[RKSimon]

@yulia_koval Are you still working on this?

[kristina]

In D25987#1228245, @RKSimon wrote:

@yulia_koval Are you still working on this?

It seems to continue in D33118. Mentioning it so Phabricator makes the link.

[yulia_koval]

What does this comment in InstCombineSelect refer to?

/// TODO: Also support a - UMIN(a,b) patterns.

This was a todo to investigate possibilities for generating min patterns, but seems like all of them are already handled, so we can just remove this TODO.

Also, there's a proposal to remove the intrinsics now - D44785. How does that relate to whatever is left to do here?

This is already commited, I think. This was converting intrinsics to cannonical form.

It seems to continue in D33118. Mentioning it so Phabricator makes the link.

This is old version of same revision, abandoned it.

@yulia_koval Are you still working on this?

No, I think this issue is finished..

[xbolva00]

Re-ping

[RKSimon]

I've been working on PR40111 to make most x86 ISD::USUBSAT code generic and moved into DAGCombine - I'll look at this as well.

[RKSimon]

Converted patch to a generic DAGCombine fold

[RKSimon]

llvm/test/CodeGen/X86/psubus.ll

563

Still investigating these cases (test14 + test16) - annoyingly alive2 is broken which is slowing me down :-( - but it should be just a similar pattern to this patch.

[RKSimon]

alive2: https://alive2.llvm.org/ce/z/u5DkxY

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

9888

x and y don't have the same bitwidth so the result of the fold isn't usubsat x, y. Is it usubsat x, trunc(umin(y, SatLimit))?

[RKSimon]

Fix comment

[craig.topper]

LGTM

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (hasOperation(ISD::USUBSAT, VT)) {
       } else if (ISD::isBuildVectorAllZeros(N2.getNode())) {
         Other = N1;
       }
 
       if (Other && Other.getNumOperands() == 2) {
         SDValue CondRHS = RHS;
         SDValue OpLHS = Other.getOperand(0), OpRHS = Other.getOperand(1);
 
+        if (Other.getOpcode() == ISD::SUB &&
+            LHS.getOpcode() == ISD::ZERO_EXTEND && LHS.getOperand(0) == OpLHS &&
+            OpRHS.getOpcode() == ISD::TRUNCATE && OpRHS.getOperand(0) == RHS) {
+          // Look for a general sub with unsigned saturation first.
+          // zext(x) >= y ? x - trunc(y) : 0
+          // --> usubsat(x,trunc(umin(y,SatLimit)))
+          // zext(x) >  y ? x - trunc(y) : 0
+          // --> usubsat(x,trunc(umin(y,SatLimit)))
+          if (SatCC == ISD::SETUGE || SatCC == ISD::SETUGT)
+            return getTruncatedUSUBSAT(VT, LHS.getValueType(), LHS, RHS, DAG,
+                                       DL);
+        }
+
         if (OpLHS == LHS) {
           // Look for a general sub with unsigned saturation first.
           // x >= y ? x-y : 0 --> usubsat x, y
           // x >  y ? x-y : 0 --> usubsat x, y
           if ((SatCC == ISD::SETUGE || SatCC == ISD::SETUGT) &&
               Other.getOpcode() == ISD::SUB && OpRHS == CondRHS)
             return DAG.getNode(ISD::USUBSAT, DL, VT, OpLHS, OpRHS);
 

llvm/test/CodeGen/X86/psubus.ll

vector.ph:
   %0 = insertelement <32 x i8> undef, i8 %w, i32 0
   %broadcast15 = shufflevector <32 x i8> %0, <32 x i8> undef, <32 x i32> zeroinitializer
   %1 = icmp ult <32 x i8> %x, %broadcast15
   %2 = sub <32 x i8> %x, %broadcast15
   %res = select <32 x i1> %1, <32 x i8> zeroinitializer, <32 x i8> %2
   ret <32 x i8> %res
 }
 
-; FIXME: match this to UMIN+TRUNC+PSUBUS
 define <8 x i16> @test13(<8 x i16> %x, <8 x i32> %y) nounwind {
 ; SSE2-LABEL: test13:
 ; SSE2:       # %bb.0: # %vector.ph
-; SSE2-NEXT:    pxor %xmm3, %xmm3
-; SSE2-NEXT:    movdqa %xmm0, %xmm4
-; SSE2-NEXT:    punpcklwd {{.*#+}} xmm4 = xmm4[0],xmm3[0],xmm4[1],xmm3[1],xmm4[2],xmm3[2],xmm4[3],xmm3[3]
-; SSE2-NEXT:    movdqa %xmm0, %xmm5
-; SSE2-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
-; SSE2-NEXT:    movdqa {{.*#+}} xmm6 = [2147483648,2147483648,2147483648,2147483648]
-; SSE2-NEXT:    movdqa %xmm2, %xmm7
-; SSE2-NEXT:    pxor %xmm6, %xmm7
-; SSE2-NEXT:    por %xmm6, %xmm5
-; SSE2-NEXT:    pcmpgtd %xmm5, %xmm7
-; SSE2-NEXT:    movdqa %xmm1, %xmm3
-; SSE2-NEXT:    pxor %xmm6, %xmm3
-; SSE2-NEXT:    por %xmm6, %xmm4
-; SSE2-NEXT:    pcmpgtd %xmm4, %xmm3
-; SSE2-NEXT:    packssdw %xmm7, %xmm3
-; SSE2-NEXT:    pslld $16, %xmm2
-; SSE2-NEXT:    psrad $16, %xmm2
-; SSE2-NEXT:    pslld $16, %xmm1
-; SSE2-NEXT:    psrad $16, %xmm1
-; SSE2-NEXT:    packssdw %xmm2, %xmm1
-; SSE2-NEXT:    psubw %xmm1, %xmm0
-; SSE2-NEXT:    pandn %xmm0, %xmm3
-; SSE2-NEXT:    movdqa %xmm3, %xmm0
+; SSE2-NEXT:    movdqa {{.*#+}} xmm3 = [2147483648,2147483648,2147483648,2147483648]
+; SSE2-NEXT:    movdqa %xmm2, %xmm4
+; SSE2-NEXT:    pxor %xmm3, %xmm4
+; SSE2-NEXT:    movdqa {{.*#+}} xmm5 = [2147549183,2147549183,2147549183,2147549183]
+; SSE2-NEXT:    movdqa %xmm5, %xmm6
+; SSE2-NEXT:    pcmpgtd %xmm4, %xmm6
+; SSE2-NEXT:    pcmpeqd %xmm4, %xmm4
+; SSE2-NEXT:    pand %xmm6, %xmm2
+; SSE2-NEXT:    pxor %xmm4, %xmm6
+; SSE2-NEXT:    por %xmm2, %xmm6
+; SSE2-NEXT:    pslld $16, %xmm6
+; SSE2-NEXT:    psrad $16, %xmm6
+; SSE2-NEXT:    pxor %xmm1, %xmm3
+; SSE2-NEXT:    pcmpgtd %xmm3, %xmm5
+; SSE2-NEXT:    pxor %xmm5, %xmm4
+; SSE2-NEXT:    pand %xmm1, %xmm5
+; SSE2-NEXT:    por %xmm4, %xmm5
+; SSE2-NEXT:    pslld $16, %xmm5
+; SSE2-NEXT:    psrad $16, %xmm5
+; SSE2-NEXT:    packssdw %xmm6, %xmm5
+; SSE2-NEXT:    psubusw %xmm5, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: test13:
 ; SSSE3:       # %bb.0: # %vector.ph
-; SSSE3-NEXT:    pxor %xmm3, %xmm3
-; SSSE3-NEXT:    movdqa %xmm0, %xmm4
-; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm4 = xmm4[0],xmm3[0],xmm4[1],xmm3[1],xmm4[2],xmm3[2],xmm4[3],xmm3[3]
-; SSSE3-NEXT:    movdqa %xmm0, %xmm5
-; SSSE3-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
-; SSSE3-NEXT:    movdqa {{.*#+}} xmm6 = [2147483648,2147483648,2147483648,2147483648]
-; SSSE3-NEXT:    movdqa %xmm2, %xmm7
-; SSSE3-NEXT:    pxor %xmm6, %xmm7
-; SSSE3-NEXT:    por %xmm6, %xmm5
-; SSSE3-NEXT:    pcmpgtd %xmm5, %xmm7
-; SSSE3-NEXT:    movdqa %xmm1, %xmm3
-; SSSE3-NEXT:    pxor %xmm6, %xmm3
-; SSSE3-NEXT:    por %xmm6, %xmm4
-; SSSE3-NEXT:    pcmpgtd %xmm4, %xmm3
-; SSSE3-NEXT:    packssdw %xmm7, %xmm3
-; SSSE3-NEXT:    movdqa {{.*#+}} xmm4 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
-; SSSE3-NEXT:    pshufb %xmm4, %xmm2
-; SSSE3-NEXT:    pshufb %xmm4, %xmm1
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
-; SSSE3-NEXT:    psubw %xmm1, %xmm0
-; SSSE3-NEXT:    pandn %xmm0, %xmm3
-; SSSE3-NEXT:    movdqa %xmm3, %xmm0
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm3 = [2147483648,2147483648,2147483648,2147483648]
+; SSSE3-NEXT:    movdqa %xmm2, %xmm4
+; SSSE3-NEXT:    pxor %xmm3, %xmm4
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm5 = [2147549183,2147549183,2147549183,2147549183]
+; SSSE3-NEXT:    movdqa %xmm5, %xmm6
+; SSSE3-NEXT:    pcmpgtd %xmm4, %xmm6
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm4 = [65535,65535,65535,65535]
+; SSSE3-NEXT:    pand %xmm6, %xmm2
+; SSSE3-NEXT:    pandn %xmm4, %xmm6
+; SSSE3-NEXT:    por %xmm2, %xmm6
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm2 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
+; SSSE3-NEXT:    pshufb %xmm2, %xmm6
+; SSSE3-NEXT:    pxor %xmm1, %xmm3
+; SSSE3-NEXT:    pcmpgtd %xmm3, %xmm5
+; SSSE3-NEXT:    pand %xmm5, %xmm1
+; SSSE3-NEXT:    pandn %xmm4, %xmm5
+; SSSE3-NEXT:    por %xmm1, %xmm5
+; SSSE3-NEXT:    pshufb %xmm2, %xmm5
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm5 = xmm5[0],xmm6[0]
+; SSSE3-NEXT:    psubusw %xmm5, %xmm0
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: test13:
 ; SSE41:       # %bb.0: # %vector.ph
-; SSE41-NEXT:    pxor %xmm4, %xmm4
-; SSE41-NEXT:    movdqa %xmm0, %xmm3
-; SSE41-NEXT:    punpckhwd {{.*#+}} xmm3 = xmm3[4],xmm4[4],xmm3[5],xmm4[5],xmm3[6],xmm4[6],xmm3[7],xmm4[7]
-; SSE41-NEXT:    pmovzxwd {{.*#+}} xmm5 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; SSE41-NEXT:    movdqa %xmm3, %xmm6
-; SSE41-NEXT:    pmaxud %xmm2, %xmm6
-; SSE41-NEXT:    pcmpeqd %xmm3, %xmm6
-; SSE41-NEXT:    pcmpeqd %xmm7, %xmm7
-; SSE41-NEXT:    pxor %xmm7, %xmm6
-; SSE41-NEXT:    movdqa %xmm5, %xmm3
-; SSE41-NEXT:    pmaxud %xmm1, %xmm3
-; SSE41-NEXT:    pcmpeqd %xmm5, %xmm3
-; SSE41-NEXT:    pxor %xmm7, %xmm3
-; SSE41-NEXT:    packssdw %xmm6, %xmm3
-; SSE41-NEXT:    pblendw {{.*#+}} xmm2 = xmm2[0],xmm4[1],xmm2[2],xmm4[3],xmm2[4],xmm4[5],xmm2[6],xmm4[7]
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0],xmm4[1],xmm1[2],xmm4[3],xmm1[4],xmm4[5],xmm1[6],xmm4[7]
+; SSE41-NEXT:    movdqa {{.*#+}} xmm3 = [65535,65535,65535,65535]
+; SSE41-NEXT:    pminud %xmm3, %xmm2
+; SSE41-NEXT:    pminud %xmm3, %xmm1
 ; SSE41-NEXT:    packusdw %xmm2, %xmm1
-; SSE41-NEXT:    psubw %xmm1, %xmm0
-; SSE41-NEXT:    pandn %xmm0, %xmm3
-; SSE41-NEXT:    movdqa %xmm3, %xmm0
+; SSE41-NEXT:    psubusw %xmm1, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test13:
 ; AVX1:       # %bb.0: # %vector.ph
-; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpunpckhwd {{.*#+}} xmm2 = xmm0[4],xmm2[4],xmm0[5],xmm2[5],xmm0[6],xmm2[6],xmm0[7],xmm2[7]
-; AVX1-NEXT:    vpmovzxwd {{.*#+}} xmm3 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
-; AVX1-NEXT:    vpmaxud %xmm4, %xmm2, %xmm5
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm2, %xmm2
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm5, %xmm5
-; AVX1-NEXT:    vpxor %xmm5, %xmm2, %xmm2
-; AVX1-NEXT:    vpmaxud %xmm1, %xmm3, %xmm6
-; AVX1-NEXT:    vpcmpeqd %xmm6, %xmm3, %xmm3
-; AVX1-NEXT:    vpxor %xmm5, %xmm3, %xmm3
-; AVX1-NEXT:    vpackssdw %xmm2, %xmm3, %xmm2
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = <0,1,4,5,8,9,12,13,u,u,u,u,u,u,u,u>
-; AVX1-NEXT:    vpshufb %xmm3, %xmm4, %xmm4
-; AVX1-NEXT:    vpshufb %xmm3, %xmm1, %xmm1
-; AVX1-NEXT:    vpunpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm4[0]
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vpandn %xmm0, %xmm2, %xmm0
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = [65535,65535,65535,65535]
+; AVX1-NEXT:    vpminud %xmm3, %xmm2, %xmm2
+; AVX1-NEXT:    vpminud %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vpackusdw %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test13:
 ; AVX2:       # %bb.0: # %vector.ph
-; AVX2-NEXT:    vpmovzxwd {{.*#+}} ymm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX2-NEXT:    vpmaxud %ymm1, %ymm2, %ymm3
-; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm2
-; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm3, %ymm3
-; AVX2-NEXT:    vpxor %ymm3, %ymm2, %ymm2
-; AVX2-NEXT:    vextracti128 $1, %ymm2, %xmm3
-; AVX2-NEXT:    vpackssdw %xmm3, %xmm2, %xmm2
-; AVX2-NEXT:    vpshufb {{.*#+}} ymm1 = ymm1[0,1,4,5,8,9,12,13,u,u,u,u,u,u,u,u,16,17,20,21,24,25,28,29,u,u,u,u,u,u,u,u]
-; AVX2-NEXT:    vpermq {{.*#+}} ymm1 = ymm1[0,2,2,3]
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpandn %xmm0, %xmm2, %xmm0
+; AVX2-NEXT:    vpbroadcastd {{.*#+}} ymm2 = [65535,65535,65535,65535,65535,65535,65535,65535]
+; AVX2-NEXT:    vpminud %ymm2, %ymm1, %ymm1
+; AVX2-NEXT:    vextracti128 $1, %ymm1, %xmm2
+; AVX2-NEXT:    vpackusdw %xmm2, %xmm1, %xmm1
+; AVX2-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test13:
 ; AVX512:       # %bb.0: # %vector.ph
-; AVX512-NEXT:    vpmovzxwd {{.*#+}} ymm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX512-NEXT:    vpcmpnltud %ymm1, %ymm2, %k1
-; AVX512-NEXT:    vpmovdw %ymm1, %xmm1
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm0 {%k1} {z}
+; AVX512-NEXT:    vpmovusdw %ymm1, %xmm1
+; AVX512-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
 vector.ph:
   %lhs = zext <8 x i16> %x to <8 x i32>
   %cond = icmp ult <8 x i32> %lhs, %y
   %sub = sub <8 x i32> %lhs, %y
   %trunc = trunc <8 x i32> %sub to <8 x i16>
   %res = select <8 x i1> %cond, <8 x i16> zeroinitializer, <8 x i16> %trunc
   ret <8 x i16> %res
 }
 
 ; FIXME: match this to UMIN+TRUNC+PSUBUS
 define <16 x i8> @test14(<16 x i8> %x, <16 x i32> %y) nounwind {
 ; SSE2OR3-LABEL: test14:
 ; SSE2OR3:       # %bb.0: # %vector.ph
 ; SSE2OR3-NEXT:    pxor %xmm8, %xmm8
 ; SSE2OR3-NEXT:    movdqa %xmm0, %xmm6
 ; SSE2OR3-NEXT:    movdqa %xmm4, %xmm9
 ; SSE2OR3-NEXT:    movdqa %xmm3, %xmm10
 ; SSE2OR3-NEXT:    movdqa %xmm2, %xmm7
vector.ph:
   %rhs = zext <16 x i8> %x to <16 x i32>
   %cond = icmp ult <16 x i32> %y, %rhs
   %sub = sub <16 x i32> %y, %rhs
   %truncsub = trunc <16 x i32> %sub to <16 x i8>
   %res = select <16 x i1> %cond, <16 x i8> zeroinitializer, <16 x i8> %truncsub
   ret <16 x i8> %res
 }
 
-; FIXME: match this to UMIN+TRUNC+PSUBUS
 define <8 x i16> @test15(<8 x i16> %x, <8 x i32> %y) nounwind {
 ; SSE2-LABEL: test15:
 ; SSE2:       # %bb.0: # %vector.ph
-; SSE2-NEXT:    pxor %xmm3, %xmm3
-; SSE2-NEXT:    movdqa %xmm0, %xmm4
-; SSE2-NEXT:    punpcklwd {{.*#+}} xmm4 = xmm4[0],xmm3[0],xmm4[1],xmm3[1],xmm4[2],xmm3[2],xmm4[3],xmm3[3]
-; SSE2-NEXT:    movdqa %xmm0, %xmm5
-; SSE2-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
 ; SSE2-NEXT:    movdqa {{.*#+}} xmm3 = [2147483648,2147483648,2147483648,2147483648]
-; SSE2-NEXT:    movdqa %xmm2, %xmm6
-; SSE2-NEXT:    pxor %xmm3, %xmm6
-; SSE2-NEXT:    por %xmm3, %xmm5
-; SSE2-NEXT:    pcmpgtd %xmm6, %xmm5
-; SSE2-NEXT:    movdqa %xmm1, %xmm6
-; SSE2-NEXT:    pxor %xmm3, %xmm6
-; SSE2-NEXT:    por %xmm3, %xmm4
-; SSE2-NEXT:    pcmpgtd %xmm6, %xmm4
-; SSE2-NEXT:    packssdw %xmm5, %xmm4
-; SSE2-NEXT:    pslld $16, %xmm2
-; SSE2-NEXT:    psrad $16, %xmm2
-; SSE2-NEXT:    pslld $16, %xmm1
-; SSE2-NEXT:    psrad $16, %xmm1
-; SSE2-NEXT:    packssdw %xmm2, %xmm1
-; SSE2-NEXT:    psubw %xmm1, %xmm0
-; SSE2-NEXT:    pand %xmm4, %xmm0
+; SSE2-NEXT:    movdqa %xmm2, %xmm4
+; SSE2-NEXT:    pxor %xmm3, %xmm4
+; SSE2-NEXT:    movdqa {{.*#+}} xmm5 = [2147549183,2147549183,2147549183,2147549183]
+; SSE2-NEXT:    movdqa %xmm5, %xmm6
+; SSE2-NEXT:    pcmpgtd %xmm4, %xmm6
+; SSE2-NEXT:    pcmpeqd %xmm4, %xmm4
+; SSE2-NEXT:    pand %xmm6, %xmm2
+; SSE2-NEXT:    pxor %xmm4, %xmm6
+; SSE2-NEXT:    por %xmm2, %xmm6
+; SSE2-NEXT:    pslld $16, %xmm6
+; SSE2-NEXT:    psrad $16, %xmm6
+; SSE2-NEXT:    pxor %xmm1, %xmm3
+; SSE2-NEXT:    pcmpgtd %xmm3, %xmm5
+; SSE2-NEXT:    pxor %xmm5, %xmm4
+; SSE2-NEXT:    pand %xmm1, %xmm5
+; SSE2-NEXT:    por %xmm4, %xmm5
+; SSE2-NEXT:    pslld $16, %xmm5
+; SSE2-NEXT:    psrad $16, %xmm5
+; SSE2-NEXT:    packssdw %xmm6, %xmm5
+; SSE2-NEXT:    psubusw %xmm5, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: test15:
 ; SSSE3:       # %bb.0: # %vector.ph
-; SSSE3-NEXT:    pxor %xmm3, %xmm3
-; SSSE3-NEXT:    movdqa %xmm0, %xmm4
-; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm4 = xmm4[0],xmm3[0],xmm4[1],xmm3[1],xmm4[2],xmm3[2],xmm4[3],xmm3[3]
-; SSSE3-NEXT:    movdqa %xmm0, %xmm5
-; SSSE3-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
 ; SSSE3-NEXT:    movdqa {{.*#+}} xmm3 = [2147483648,2147483648,2147483648,2147483648]
-; SSSE3-NEXT:    movdqa %xmm2, %xmm6
-; SSSE3-NEXT:    pxor %xmm3, %xmm6
-; SSSE3-NEXT:    por %xmm3, %xmm5
-; SSSE3-NEXT:    pcmpgtd %xmm6, %xmm5
-; SSSE3-NEXT:    movdqa %xmm1, %xmm6
-; SSSE3-NEXT:    pxor %xmm3, %xmm6
-; SSSE3-NEXT:    por %xmm3, %xmm4
-; SSSE3-NEXT:    pcmpgtd %xmm6, %xmm4
-; SSSE3-NEXT:    packssdw %xmm5, %xmm4
-; SSSE3-NEXT:    movdqa {{.*#+}} xmm3 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
-; SSSE3-NEXT:    pshufb %xmm3, %xmm2
-; SSSE3-NEXT:    pshufb %xmm3, %xmm1
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
-; SSSE3-NEXT:    psubw %xmm1, %xmm0
-; SSSE3-NEXT:    pand %xmm4, %xmm0
+; SSSE3-NEXT:    movdqa %xmm2, %xmm4
+; SSSE3-NEXT:    pxor %xmm3, %xmm4
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm5 = [2147549183,2147549183,2147549183,2147549183]
+; SSSE3-NEXT:    movdqa %xmm5, %xmm6
+; SSSE3-NEXT:    pcmpgtd %xmm4, %xmm6
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm4 = [65535,65535,65535,65535]
+; SSSE3-NEXT:    pand %xmm6, %xmm2
+; SSSE3-NEXT:    pandn %xmm4, %xmm6
+; SSSE3-NEXT:    por %xmm2, %xmm6
+; SSSE3-NEXT:    movdqa {{.*#+}} xmm2 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
+; SSSE3-NEXT:    pshufb %xmm2, %xmm6
+; SSSE3-NEXT:    pxor %xmm1, %xmm3
+; SSSE3-NEXT:    pcmpgtd %xmm3, %xmm5
+; SSSE3-NEXT:    pand %xmm5, %xmm1
+; SSSE3-NEXT:    pandn %xmm4, %xmm5
+; SSSE3-NEXT:    por %xmm1, %xmm5
+; SSSE3-NEXT:    pshufb %xmm2, %xmm5
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm5 = xmm5[0],xmm6[0]
+; SSSE3-NEXT:    psubusw %xmm5, %xmm0
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: test15:
 ; SSE41:       # %bb.0: # %vector.ph
-; SSE41-NEXT:    pxor %xmm3, %xmm3
-; SSE41-NEXT:    movdqa %xmm0, %xmm4
-; SSE41-NEXT:    punpckhwd {{.*#+}} xmm4 = xmm4[4],xmm3[4],xmm4[5],xmm3[5],xmm4[6],xmm3[6],xmm4[7],xmm3[7]
-; SSE41-NEXT:    pmovzxwd {{.*#+}} xmm5 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; SSE41-NEXT:    movdqa %xmm4, %xmm6
-; SSE41-NEXT:    pminud %xmm2, %xmm6
-; SSE41-NEXT:    pcmpeqd %xmm4, %xmm6
-; SSE41-NEXT:    pcmpeqd %xmm4, %xmm4
-; SSE41-NEXT:    pxor %xmm4, %xmm6
-; SSE41-NEXT:    movdqa %xmm5, %xmm7
-; SSE41-NEXT:    pminud %xmm1, %xmm7
-; SSE41-NEXT:    pcmpeqd %xmm5, %xmm7
-; SSE41-NEXT:    pxor %xmm4, %xmm7
-; SSE41-NEXT:    packssdw %xmm6, %xmm7
-; SSE41-NEXT:    pblendw {{.*#+}} xmm2 = xmm2[0],xmm3[1],xmm2[2],xmm3[3],xmm2[4],xmm3[5],xmm2[6],xmm3[7]
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0],xmm3[1],xmm1[2],xmm3[3],xmm1[4],xmm3[5],xmm1[6],xmm3[7]
+; SSE41-NEXT:    movdqa {{.*#+}} xmm3 = [65535,65535,65535,65535]
+; SSE41-NEXT:    pminud %xmm3, %xmm2
+; SSE41-NEXT:    pminud %xmm3, %xmm1
 ; SSE41-NEXT:    packusdw %xmm2, %xmm1
-; SSE41-NEXT:    psubw %xmm1, %xmm0
-; SSE41-NEXT:    pand %xmm7, %xmm0
+; SSE41-NEXT:    psubusw %xmm1, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test15:
 ; AVX1:       # %bb.0: # %vector.ph
-; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpunpckhwd {{.*#+}} xmm2 = xmm0[4],xmm2[4],xmm0[5],xmm2[5],xmm0[6],xmm2[6],xmm0[7],xmm2[7]
-; AVX1-NEXT:    vpmovzxwd {{.*#+}} xmm3 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
-; AVX1-NEXT:    vpminud %xmm4, %xmm2, %xmm5
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm2, %xmm2
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm5, %xmm5
-; AVX1-NEXT:    vpxor %xmm5, %xmm2, %xmm2
-; AVX1-NEXT:    vpminud %xmm1, %xmm3, %xmm6
-; AVX1-NEXT:    vpcmpeqd %xmm6, %xmm3, %xmm3
-; AVX1-NEXT:    vpxor %xmm5, %xmm3, %xmm3
-; AVX1-NEXT:    vpackssdw %xmm2, %xmm3, %xmm2
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = <0,1,4,5,8,9,12,13,u,u,u,u,u,u,u,u>
-; AVX1-NEXT:    vpshufb %xmm3, %xmm4, %xmm4
-; AVX1-NEXT:    vpshufb %xmm3, %xmm1, %xmm1
-; AVX1-NEXT:    vpunpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm4[0]
-; AVX1-NEXT:    vpsubw %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vpand %xmm0, %xmm2, %xmm0
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = [65535,65535,65535,65535]
+; AVX1-NEXT:    vpminud %xmm3, %xmm2, %xmm2
+; AVX1-NEXT:    vpminud %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vpackusdw %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test15:
 ; AVX2:       # %bb.0: # %vector.ph
-; AVX2-NEXT:    vpmovzxwd {{.*#+}} ymm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX2-NEXT:    vpminud %ymm1, %ymm2, %ymm3
-; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm2
-; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm3, %ymm3
-; AVX2-NEXT:    vpxor %ymm3, %ymm2, %ymm2
-; AVX2-NEXT:    vextracti128 $1, %ymm2, %xmm3
-; AVX2-NEXT:    vpackssdw %xmm3, %xmm2, %xmm2
-; AVX2-NEXT:    vpshufb {{.*#+}} ymm1 = ymm1[0,1,4,5,8,9,12,13,u,u,u,u,u,u,u,u,16,17,20,21,24,25,28,29,u,u,u,u,u,u,u,u]
-; AVX2-NEXT:    vpermq {{.*#+}} ymm1 = ymm1[0,2,2,3]
-; AVX2-NEXT:    vpsubw %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpand %xmm0, %xmm2, %xmm0
+; AVX2-NEXT:    vpbroadcastd {{.*#+}} ymm2 = [65535,65535,65535,65535,65535,65535,65535,65535]
+; AVX2-NEXT:    vpminud %ymm2, %ymm1, %ymm1
+; AVX2-NEXT:    vextracti128 $1, %ymm1, %xmm2
+; AVX2-NEXT:    vpackusdw %xmm2, %xmm1, %xmm1
+; AVX2-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test15:
 ; AVX512:       # %bb.0: # %vector.ph
-; AVX512-NEXT:    vpmovzxwd {{.*#+}} ymm2 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX512-NEXT:    vpcmpnleud %ymm1, %ymm2, %k1
-; AVX512-NEXT:    vpmovdw %ymm1, %xmm1
-; AVX512-NEXT:    vpsubw %xmm1, %xmm0, %xmm0 {%k1} {z}
+; AVX512-NEXT:    vpmovusdw %ymm1, %xmm1
+; AVX512-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
 vector.ph:
   %lhs = zext <8 x i16> %x to <8 x i32>
   %cond = icmp ugt <8 x i32> %lhs, %y
   %sub = sub <8 x i32> %lhs, %y
   %truncsub = trunc <8 x i32> %sub to <8 x i16>
   %res = select <8 x i1> %cond, <8 x i16> %truncsub, <8 x i16> zeroinitializer