Handle COPYs of physregs better (regalloc hints)

[RKSimon]

Summary

While enabling the mischeduler for SystemZ, it was discovered that fore some reason a test needed one extra seemingly needless COPY (test/CodeGen/SystemZ/call-03.ll). The handling for that is resulted in this patch, which improves the register coalescing by providing not just one copy hint, but a sorted list of copy hints. On SystemZ, this gives ~12500 less register moves on SPEC, as well as marginally less spilling.

Instead of improving just the SystemZ backend, the improvement has been implemented in common-code (calculateSpillWeightAndHint(). This gives a lot of test failures, but since this should be a general improvement I hope that the involved targets will help and review the test updates.

[uweigand]

In D38128#904781, @jonpa wrote:

In D38128#904581, @efriedma wrote:

Not sure if it is clear that coalescing with %R8 is generally better than %R0.

What do you mean, it isn't clear? Is the performance problem not clear? Or do you mean you're not sure how to detect this situation when you're sorting the hints?

I guess given your initial wording ("not great"), I was not sure if this is general and serious enough so that we really want to add an additional heuristic like "COPY to compare" on top of the sorting by weight. I suppose then there should be a flag like HasCompareUser which is then the tie-breaker when the weight is the same, or?

I don't believe there's anything special about compare here, this applies the same to any other use. In general, if you have something like "COPY dest, src" followed by "use dest", it is nearly always better to replace the second instruction by "use src" if this is possible (i.e. if src is still live and unchanged at that point). This is simply because on most processor architectures "COPY dest, src / use src" can execute in parallel, while "COPY dest, src / use dest" must execute sequentially.

[jonpa]

In D38128#904861, @uweigand wrote:

In D38128#904781, @jonpa wrote:

In D38128#904581, @efriedma wrote:

Not sure if it is clear that coalescing with %R8 is generally better than %R0.

What do you mean, it isn't clear? Is the performance problem not clear? Or do you mean you're not sure how to detect this situation when you're sorting the hints?

I guess given your initial wording ("not great"), I was not sure if this is general and serious enough so that we really want to add an additional heuristic like "COPY to compare" on top of the sorting by weight. I suppose then there should be a flag like HasCompareUser which is then the tie-breaker when the weight is the same, or?

I don't believe there's anything special about compare here, this applies the same to any other use. In general, if you have something like "COPY dest, src" followed by "use dest", it is nearly always better to replace the second instruction by "use src" if this is possible (i.e. if src is still live and unchanged at that point). This is simply because on most processor architectures "COPY dest, src / use src" can execute in parallel, while "COPY dest, src / use dest" must execute sequentially.

That makes sense.

Would any of these be an improvement worth trying, then?

(1) If the source phys-reg is contained in the regclass of an immediately following instruction using the vreg, then increase the priority of the hint of that phys-reg.
(2) A simpler alternative would be to simply prefer phys-reg sources more than phys-reg dest-regs (if weight is equal). That would catch all the cases of (1).

[uweigand]

In D38128#904899, @jonpa wrote:

Would any of these be an improvement worth trying, then?

(1) If the source phys-reg is contained in the regclass of an immediately following instruction using the vreg, then increase the priority of the hint of that phys-reg.
(2) A simpler alternative would be to simply prefer phys-reg sources more than phys-reg dest-regs (if weight is equal). That would catch all the cases of (1).

I believe (2) makes sense ... certainly worth a try.

[jonpa]

In D38128#905052, @uweigand wrote:

In D38128#904899, @jonpa wrote:

Would any of these be an improvement worth trying, then?

(1) If the source phys-reg is contained in the regclass of an immediately following instruction using the vreg, then increase the priority of the hint of that phys-reg.
(2) A simpler alternative would be to simply prefer phys-reg sources more than phys-reg dest-regs (if weight is equal). That would catch all the cases of (1).

I believe (2) makes sense ... certainly worth a try.

I see that on SPEC/z13, I now get a few more copys left. Compared to master:

lgr            :               341784               328956   -12828
lr             :                26208                25803     -405
...
Spill|Reload   :               165135               164761     -374

-> with prefer phys reg source COPYs:

lgr            :               341784               329060   -12724
lr             :                26208                25817     -391
...
Spill|Reload   :               165135               164934     -201

I also see looking at just the loops that while 120 innner loops have changed size, 103 of them has gotten
bigger, which seems to indicate that at least currently (Wei Mi has an upcoming regsplit patch that may
change this), this idea is not quite promising.

It did however handle the test case under discussion (ARM/swifterror.ll)

Reversing this heuristic so that phys-reg dest copys are prioritized:

lgr            :               341784               328907   -12877
lr             :                26208                25762     -446
...
Spill|Reload   :               165135               164807     -328

For the innermost loops, there are only 34 different (15 smaller).

I am not sure it is right to tamper with this in this way given the above...

[uweigand]

Right. Always coalescing with the copy source of course extends the live range of the source, and thus increases overall register pressure. So it's not a good idea to do this unconditionally.

On the other hand, in the example that triggered this discussion, there was no register pressure problem and source and destination register just happened to be both still live at the point of use, anyway. In this specific case, it would be preferable to use the source instead of the dest register at the point of use. On the other hand, if keeping the source alive would cause extra spilling, we don't want that, but want to use the dest after all.

Not sure how exactly to handle this in LLVM. It is probably not simply a matter of preference in the register coalescer.

[jonpa]

Eli, do you have any comment on this?

[efriedma]

What specifically do you want me to comment on? I don't really understand the register allocator well enough to comment on your code changes.

[jonpa]

In D38128#908337, @efriedma wrote:

What specifically do you want me to comment on? I don't really understand the register allocator well enough to comment on your code changes.

Given that this is something that Quentin has been meaning to fix before I did, and that this is *generally* good, the question is now if we can accept minor issues like this. On SystemZ, I think this is quite clear since we get 12500 less COPYs and also less spilling (on SPEC). Could you perhaps try it on your target and see if you likewise would agree this is good enough despite the swifterror test issue?

Also, it could be that other tests with a similar code actually now improve, since this was a random result. So just one regression alone should not have to stop the patch, unless it is a sign of a more general issue.

[efriedma]

It's a minor issue; I'm okay with committing this as-is and trying to address the issue later.

[jonpa]

It's a minor issue; I'm okay with committing this as-is and trying to address the issue later.

All right - I take this to mean that you approve of the ARM test changes, then. Thanks.

PING!

Patch updated with a few X86 tests updated. Again, someone from each target with the right experience should review the test changes I have made:

TODO:
CodeGen : AArch64 AMDGPU BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

DONE:
CodeGen : ARM
DebugInfo :

ping @tstellar: CodeGen/AMDGPU/multilevel-break.ll

[mstorsjo]

In D38128#910425, @jonpa wrote:

TODO:
CodeGen : AArch64 AMDGPU BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore

AArch64/win64_vararg.ll is still ok.
arm64-aapcs.ll and func-argpassing.ll also seem fine.

swifterror.ll seems to get a very minor regression, not sure who to get an authoritative approval from though.

[jonpa]

PING!

@Quentin, I implemented this in common code because you had already plans for this... It's been weeks now since I updated all the tests manually, which was not a small effort. Still, there is very little progress on the review of the tests :-/ Do you know who I should add as reviewers? (BTW patch slightly changed since you approved it - still ok, I hope?)

@Tom, I hope you can find the time to look into the AMDGPU tests as I thought you meant to do...

Review of test changes:

TODO:
CodeGen : AMDGPU BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

PARTIALLY DONE (still more tests in need of review):
CodeGen :

• AArch64: win64_vararg.ll, arm64-aapcs.ll, func-argpassing.ll (Martin Storsjo) swifterror.ll small regression / needs approval (Martin Storsjo)

DONE:
CodeGen :

• ARM (Eli Friedman)

[nemanjai]

Just looking at the number and placement of reg-reg moves for PowerPC seems fine. It doesn't seem like there are regressions but no huge reduction in the number of copies. However, since the diff is without context, it's hard to tell exactly what is going on in the test cases. I'll apply this patch to ToT and have a look at how things change before signing off on the PPC changes.

[tstellar]

The AMDGPU tests look OK, but the CHECK lines for test/CodeGen/AMDGPU/sgpr-control-flow.ll should be updated like this:
https://reviews.llvm.org/P8044

[jonpa]

@Nemanja:

Just looking at the number and placement of reg-reg moves for PowerPC seems fine. It doesn't seem like there are regressions but no huge reduction in the number of copies. However, since the diff is without context, it's hard to tell exactly what is going on in the test cases. I'll apply this patch to ToT and have a look at how things change before signing off on the PPC changes.

Sorry about forgetting the context - fixed.

@Tom:

The AMDGPU tests look OK, but the CHECK lines for test/CodeGen/AMDGPU/sgpr-control-flow.ll should be updated like this: https://reviews.llvm.org/P8044

Thanks! sgpr-control-flow.ll updated, but there is also still the multilevel-break.ll failure. This is the only test that is failing even with this patch, and the reason for this is that I was not sure how to update it. We discussed this on Oct 11 here on Phabricator. Could you please take a look?

Test updates in need of review:

TODO:
CodeGen : BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

PARTIALLY DONE (still more tests in need of review):
CodeGen :

• AArch64 : win64_vararg.ll, arm64-aapcs.ll, func-argpassing.ll (Martin Storsjo) swifterror.ll: small regression, needs approval (Martin Storsjo)
• AMDGPU: (Tom Stellard) All reviewed, but multilevel-break.ll needs updating (fails with patch)

DONE:
CodeGen :

• ARM (Eli Friedman)

[tstellar]

I think the changes to the multilevel-break.ll test are OK. The register usage is increased by 2, but this won't have any impact on performance for this test. Here is a patch to make this test pass with your changes: https://reviews.llvm.org/P8046

[nemanjai]

The PowerPC CodeGen test changes are fine. I've also confirmed that this patch reduces the total number of register move instructions (which implement the copies). So that LGTM.

[jonpa]

@Tom:

I think the changes to the multilevel-break.ll test are OK. The register usage is increased by 2, but this won't have any impact on performance for this test. Here is a patch to make this test pass with your changes: https://reviews.llv\

m.org/P8046
Applied. Thanks!

@Nemanja:

The PowerPC CodeGen test changes are fine. I've also confirmed that this patch reduces the total number of register move instructions (which implement the copies). So that LGTM.

Thanks, nice to hear :-)

Regression? : test/DebugInfo/X86/live-debug-variables.ll

Test updates in need of review:

TODO:
CodeGen   : BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

PARTIALLY DONE :
 CodeGen :
  - AArch64: All done but swifterror.ll (small regression and needs approval according to Martin Storsjo)

DONE:
 CodeGen :
  - ARM (Eli Friedman)
  - AMDGPU (Tom Stellard)
  - PPC (nemanjai)

[jonpa]

Ping!

Three backends now done (with confirmed reductions in number of COPYs after regalloc). ninja check passes with patch (all failing tests updated).

Please, could someone from each backend listed below take a look at the test changes (already updated).

TODO:
CodeGen   : BPF Hexagon Mips PowerPC SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

PARTIALLY DONE :
 CodeGen :
  - AArch64: All done but swifterror.ll (small regression and needs approval according to Martin Storsjo)

APPROVED:
 CodeGen :
  - ARM (Eli Friedman)
  - AMDGPU (Tom Stellard)
  - PPC (Nemanja Ivanovic)

(Regression? : test/DebugInfo/X86/live-debug-variables.ll)

[jonpa]

@Quentin: I think the patch is starting to be somewhat convincing, as it is now also confirmed and approved on PowerPC, as well as has gotten approved test changes for AMDGPU and CodeGen/ARM. Given that it's now been two months of review, and that I updated all the tests already a month ago, I wonder what you would say about activating this for now under a target preference hook? I think that SystemZ, PowerPC and AMDGPU could benefit from this now, while other targets could do so after reviewing test changes. Would this work?

Status table corrected:

TODO:
CodeGen   : BPF Hexagon Mips SPARC Thumb Thumb2 X86 XCore
DebugInfo : COFF X86

PARTIALLY DONE :
 CodeGen :
  - AArch64: All done but swifterror.ll (small regression and needs approval according to Martin Storsjo)

APPROVED:
 CodeGen :
  - CodeGen/ARM (Eli Friedman)
  - AMDGPU (Tom Stellard)
  - PowerPC (Nemanja Ivanovic)

(Regression? : test/DebugInfo/X86/live-debug-variables.ll)

[qcolombet]

Still LGTM :).
It makes sense to have it behind a hook for now, but could you follow-up with the targets owners so that it gets to be the default and we can get rid of the hook.
I believe this is general goodness and that we should use it for all targets going forward.

[jonpa]

Patch updated to include a TargetRegisterInfo hook 'enableMultipleCopyHints()' that returns false per default, which gives NFC to trunk. OnlySystemZ returns true for now. It is up to all target maintainers to enable this and review test changes, which hopefully will be done
soon so that the hook can be removed again.

Some minor changes compared to last version in order to get the NFC to trunk when returning false in enableMultipleCopyHints():

• reinstate the mri.isAllocatable() check for the hint, which was dropped for some reason (makes a small difference on the resulting weight in some rare cases).
• If target has created a simple hint, it must be cleared before the best hint from CopyHints is inserted, so a new clearSimpleHint() method in MachineRegisterInfo was needed. Not sure that all those different methods for manipulating hints are really needed in the end once all targets have been converted. In particular, I am curious why AMDGPU sets those hints (of generic type) when they will be recomputed always here...
• A new temporary variable CopyHint::HintOrder, in order to properly mimic the current behavior on trunk for NFC. This is because on trunk, the best hint is kept in the order of discovery, which is not what the CopyHints set was doing.

These hacks aren't very pretty, so PLEASE EVERYONE: HELP REMOVE THEM BY ENABLING THE MULTIPLE COPY HINTS!

@nemanjai: You approved this for Power PC, but one more test has changed now (licm-tocReg.ll), so please get the previous test changes from here on Phabricator (which apply fine) and review the new one, and then return true in enableMultipleCopyHints() once this is approved.

@tstellard, @efriedman: Thanks for helping reviewing the tests on your targets. I have not enabled those targets as I have not checked on those tests on latest trunk. I hope you can download those tests from here on Phabricator and make sure eveything's green as you enable this.

Please let me know if this is ok to commit.

[jonpa]

ping!

[uweigand]

The SystemZ changes LGTM. Given that Quentin approved the common part, this should be fine.

[jonpa]

Commited as r319754.

I will not close this review since the next step is to remove the temporary enableMultipleCopyHints() hook, after everyone is returning true.

Tom, Nemanja, Martin, Eli... I hope you will try this soon...

Targets that implement getRegAllocationHints() (ARM), might want to review that implementation in regards to the new common-code...

[jonpa]

This has been committed, but is still only enabled for SystemZ.

I thought we might try to make progress on enabling this on all targets, so I decided to now do one target at a time.

First out is PowerPC:

CodeGen/PowerPC/licm-tocReg.ll: Updated. Seems to use one instruction (mr) less.
load-two-flts.ll: Updated: Smaller immediate offsets should be better, I hope.
ppc64-byval-align.ll: Updated: One move less, it seems.
select-i1-vs-i1.ll: Updated: a few functions a bit different. If-conversion?

Nemanja?

[jonpa]

Ping!

Could anyone on PowerPC confirm these test changes and perhaps also that number of COPYs are decreasing overall, please.

[nemanjai]

In D38128#980403, @jonpa wrote:

Ping!

Could anyone on PowerPC confirm these test changes and perhaps also that number of COPYs are decreasing overall, please.

Sorry about the delay. The test case changes look neutral or like improvements (slect-i1-vs-i1.ll looks like an improvement). I'm running it now to see the overall impact on the number of register copies. I'll report back soon.

[nemanjai]

This also reduces the overall number of register moves slightly. LGTM

[jonpa]

Thanks Nemanja, this is now enabled (r323858) also for PowerPC :-)

A 'ping' to the other targets that I updated tests for a while back ago - has anyone made any progress?

[nemanjai]

Thank you Jonas. As this no longer needs PPC input, I'm resigning from the review.

[jonpa]

Let's try to enable this for AArch64, next.

I have reapplied the test changes from earlier. Martin, does this look OK?

[mstorsjo]

Yes, this looks like mostly no-op reorderings. I don't remember exactly what part I thought was a potential performance regression in swifterror.ll before - it moves one "mov" instruction (in two places) to before a branch, so potentially executing one instruction more than before. I would say it's most probably ok (and the gains in some of the other register shuffling tests would make it sound like a net gain in any case).

I'm in no way authoritative for this though, so perhaps @aemerson or @qcolombet would like to give such a stamp of approval?

[mstorsjo]

So with the absolutely minimal performance regression in swifterror.ll and lack of response otherwise, I think I'm ok with giving it the LGTM for aarch64.

[jonpa]

In D38128#1002953, @mstorsjo wrote:

So with the absolutely minimal performance regression in swifterror.ll and lack of response otherwise, I think I'm ok with giving it the LGTM for aarch64.

Thank you, Martin. Committed as r324720.

[aemerson]

Sorry I didn't see this, I need to fix my email filters.

[jonpa]

Enabled for ARM. Mostly same test updates as previously (still passing), plus a few more.

Somebody take a look, please. Eli?

[jonpa]

Ping!

ARM test updates are waiting for approval.

[efriedma]

ARM test changes LGTM

[jonpa]

In D38128#1009606, @efriedma wrote:

ARM test changes LGTM

Thanks, Eli.

r325327.

[jonpa]

Enabled for AMDGPU, with just a few minor test updates.

@tstellar : This looks to be a subset of the test changes you previously approved, so I am hoping they still look ok?

[jonpa]

[AMDGPU] Just a few test updates in need of approval.

[rampitec]

Changes look non-essential to me.

[jonpa]

In D38128#1010271, @rampitec wrote:

Changes look non-essential to me.

Thank you, Stanislav. I committed this even though a last-minute change slipped in (diff from approved changes):

--- a/test/CodeGen/AMDGPU/ret.ll
+++ b/test/CodeGen/AMDGPU/ret.ll
@@ -178,8 +178,8 @@ bb:
 }

 ; GCN-LABEL: {{^}}sgpr:
-; GCN-DAG: s_add_i32 s0, s3, 2
-; GCN-DAG: s_mov_b32 s2, s3
+; GCN: s_mov_b32 s2, s3
+; GCN: s_add_i32 s0, s2, 2
 ; GCN-NOT: s_endpgm
 define amdgpu_vs { i32, i32, i32 } @sgpr([9 x <16 x i8>] addrspace(4)* byval %arg, i32 inreg %arg1, i32 inreg %arg2, float %arg3) #0 {
 bb:

This looks harmless to me - the add is using s2 instead of s3, directly after the move of s3 to s2, right? Please take a look.

Committed as r325425.

[jonpa]

[BPF] multiple copy hints enabled. Tests updated and passing, please review.

[jonpa]

[BPF] Please review updated tests.

[rampitec]

In D38128#1011287, @jonpa wrote:

In D38128#1010271, @rampitec wrote:

Changes look non-essential to me.

Thank you, Stanislav. I committed this even though a last-minute change slipped in (diff from approved changes):

--- a/test/CodeGen/AMDGPU/ret.ll
+++ b/test/CodeGen/AMDGPU/ret.ll
@@ -178,8 +178,8 @@ bb:
 }

 ; GCN-LABEL: {{^}}sgpr:
-; GCN-DAG: s_add_i32 s0, s3, 2
-; GCN-DAG: s_mov_b32 s2, s3
+; GCN: s_mov_b32 s2, s3
+; GCN: s_add_i32 s0, s2, 2
 ; GCN-NOT: s_endpgm
 define amdgpu_vs { i32, i32, i32 } @sgpr([9 x <16 x i8>] addrspace(4)* byval %arg, i32 inreg %arg1, i32 inreg %arg2, float %arg3) #0 {
 bb:

This looks harmless to me - the add is using s2 instead of s3, directly after the move of s3 to s2, right? Please take a look.

Committed as r325425.

This change in mov/add does not affect anything, thanks.

[jonpa]

(Yonghong Song)
I checked BFP, yes. The patch looks good to me and you can add my ACK and push in.

BFP enabled by r325457.

[jonpa]

[Hexagon] Enabled with updated tests.

Please take a look at the test updates.

[kparzysz]

The Hexagon changes look ok.

[jonpa]

In D38128#1014574, @kparzysz wrote:

The Hexagon changes look ok.

Thank you.
r325697

[jonpa]

[Mips] Multiple regalloc hints enabled with updated tests.

[sdardis]

Hi Jonas, I've regenerated the tests in rL325770 and the changes seem ok. The main difference I'm seeing is some moves are being eliminated, but in a few cases in test/CodeGen/Mips/analyzebranch.ll the earlier move prevents a delay slot being filled but otherwise looks ok.

https://reviews.llvm.org/differential/diff/135383 is my local copy of this patch against the regenerated tests.

[jonpa]

sdardis patch applied.

[jonpa]

In D38128#1015672, @sdardis wrote:

Hi Jonas, I've regenerated the tests in rL325770 and the changes seem ok. The main difference I'm seeing is some moves are being eliminated, but in a few cases in test/CodeGen/Mips/analyzebranch.ll the earlier move prevents a delay slot being filled but otherwise looks ok.

https://reviews.llvm.org/differential/diff/135383 is my local copy of this patch against the regenerated tests.

Thank you, Simon. I applied your patch with regenerated tests and committed it as r325870.

[jonpa]

[SPARC] Patch enabled - please review the updated tests.

[jyknight]

Sparc changes look reasonable.

[jonpa]

In D38128#1017405, @jyknight wrote:

Sparc changes look reasonable.

Thank you, James. r326028.

[jonpa]

[XCore] patch enabled - please review updated test and approve patch.

[robertlytton]

please see in-line comments

test/CodeGen/XCore/byVal.ll

45–46 ↗

(On Diff #135790)

Hi jonpa,
The output in test/CodeGen/XCore/byVal.ll is incorrect.
viz the value in r1 is not copied into r0 before being overwritten by the value in sp[2] (r0's indirected value).
These two lines should be the other way around.
sorry
robert

[jonpa]

Test updated again.

[jonpa]

test/CodeGen/XCore/byVal.ll

45–46 ↗

(On Diff #135790)

Ah, I think 'r1' slipped through in the place of 'r11'... Does it look ok now?

[robertlytton]

LGTM
thank you
robert

[jonpa]

In D38128#1018974, @robertlytton wrote:

LGTM
thank you
robert

Thank you, Robert. r326069

[jonpa]

X86 is now the final backend to enable :-)

This patch enables X86 with regenerated tests in all (155) tests that previously had the "NOTE: : Assertions have been autogenerated..." comment. I have manually updated the remaining (28) ones.

Please review these test changes.

After this, we can finally get rid of the enableMultipleCopyHints() hook.

[niravd]

This looks like a nice improvement modulo a few issues:

• In a number of places (notably register arguments to shift instructions) we generate movq instead of movl that's shorter and equivalent. I believe there's no real performance difference. Certainly we should for when compiling for size.
• There's some unnecessary shuffling of register names in SSE4 tests.

Someone who's a more familiar with Debug should double check those tests though they seem fine to me.

test/CodeGen/X86/fast-isel-shift.ll

25

Why are we getting two kill comments about cx here?

test/CodeGen/X86/schedule-x86-64-shld.ll

12–1

This should be movl given optsize.

test/CodeGen/X86/sret-implicit.ll

0–1

This shouldn't be DAG matches anymore the 2nd line should always come first

0–1

Same as above

test/CodeGen/X86/vector-shift-ashr-128.ll

1

All of the SSE4 changes regarding shifts seem to generate unnecessary register shuffling.

test/CodeGen/X86/vectorcall.ll

1

This is probably fine, but confusing.

Can you decompose this up into the X86 and X64? In fact, this file should probably be autogenerated with utils/update_llc_test_checks.py

2

another case of movq vs movl

[jonpa]

ping! Please don't forget to do this for X86, we still want to get rid of that temporary hook... :-)

[RKSimon]

Updated all x86 tests to trunk latest

[craig.topper]

Has anyone looked at @niravd's comments?

[RKSimon]

In D38128#1180526, @craig.topper wrote:

Has anyone looked at @niravd's comments?

I haven't - just updated all the tests

[RKSimon]

test/CodeGen/X86/vector-shift-ashr-128.ll

1

If I had to guess - this is probably due to SSE41's PBLENDV instructions being hardwired to use xmm0

[xbolva00]

Is this ready or any blockers?

[RKSimon]

rebased to trunk

[RKSimon]

In D38128#1235889, @xbolva00 wrote:

Is this ready or any blockers?

Other than the poor codegen issues (but not regressions) commented on by @niravd I don't think there are any blockers.

[RKSimon]

OK to commit?

[niravd]

The codesize issues are minor and shouldn't hold this patch up. The only blocker I see is the unnecessary data shuffling for SSE41 codegen which someone else should decide on.

In D38128#1238605, @RKSimon wrote:

OK to commit?

[spatel]

In D38128#1238797, @niravd wrote:

The codesize issues are minor and shouldn't hold this patch up. The only blocker I see is the unnecessary data shuffling for SSE41 codegen which someone else should decide on.

Is this just about the extra 'movdqa' in vector-shift-ashr-128.ll, or are there other diffs to look at?

[niravd]

I've gone through and marked all the places.

Is this just about the extra 'movdqa' in vector-shift-ashr-128.ll, or are there other diffs to look at?

test/CodeGen/X86/vector-shift-lshr-128.ll

232

Extra instruction here

test/CodeGen/X86/vselect-minmax.ll

4539

Extra instruction here.

4660

Extra instruction here.

4781

Extra Instruction here.

4901

Extra instruction here.

5019–5020

Extra instructions here.

5170–5176

Extra instructions here.

5319–5320

Extra instructions here.

5468–5469

Extra instructions here.

6995

Extra instructions here.

7116–7117

Extra instruction here.

7235

Extra instruction here.

7356–7357

Extra instructions here.

7505–7506

Extra Instructions here.

7655–7656

Extra instructions here.

[RKSimon]

In D38128#1239480, @niravd wrote:

I've gone through and marked all the places.

Is this just about the extra 'movdqa' in vector-shift-ashr-128.ll, or are there other diffs to look at?

These are all due to pblendvb/blendvpd/blendvps being hardwired to use the xmm0 as the mask register (limit goes away for avx)

[spatel]

In D38128#1239512, @RKSimon wrote:

In D38128#1239480, @niravd wrote:

I've gone through and marked all the places.

Is this just about the extra 'movdqa' in vector-shift-ashr-128.ll, or are there other diffs to look at?

These are all due to pblendvb/blendvpd/blendvps being hardwired to use the xmm0 as the mask register (limit goes away for avx)

Yep - thanks for marking all of those. That corner case shouldn't hold up the general improvement, and the number of customers specifically targeting SSE4.1 should go to zero over time, so LGTM.

test/CodeGen/X86/fast-isel-shift.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -fast-isel -fast-isel-abort=1 -mtriple=x86_64-apple-darwin10 | FileCheck %s
 
 define i8 @shl_i8(i8 %a, i8 %b) {
 ; CHECK-LABEL: shl_i8:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    shlb %cl, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $cl killed $ecx
+; CHECK-NEXT:    shlb %cl, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = shl i8 %a, %b
   ret i8 %c
 }
 
 define i16 @shl_i16(i16 %a, i16 %b) {
 ; CHECK-LABEL: shl_i16:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $cx
-; CHECK-NEXT:    shlw %cl, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cx killed $cx killed $ecx
+; CHECK-NEXT:    ## kill: def $cl killed $cx
+; CHECK-NEXT:    shlw %cl, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = shl i16 %a, %b
   ret i16 %c
 }
 
 define i32 @shl_i32(i32 %a, i32 %b) {
 ; CHECK-LABEL: shl_i32:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $ecx
-; CHECK-NEXT:    shll %cl, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $ecx
+; CHECK-NEXT:    shll %cl, %eax
 ; CHECK-NEXT:    retq
   %c = shl i32 %a, %b
   ret i32 %c
 }
 
 define i64 @shl_i64(i64 %a, i64 %b) {
 ; CHECK-LABEL: shl_i64:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movq %rsi, %rcx
-; CHECK-NEXT:    ## kill: def $cl killed $rcx
-; CHECK-NEXT:    shlq %cl, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    ## kill: def $cl killed $rcx
+; CHECK-NEXT:    shlq %cl, %rax
 ; CHECK-NEXT:    retq
   %c = shl i64 %a, %b
   ret i64 %c
 }
 
 define i8 @lshr_i8(i8 %a, i8 %b) {
 ; CHECK-LABEL: lshr_i8:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    shrb %cl, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $cl killed $ecx
+; CHECK-NEXT:    shrb %cl, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i8 %a, %b
   ret i8 %c
 }
 
 define i16 @lshr_i16(i16 %a, i16 %b) {
 ; CHECK-LABEL: lshr_i16:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $cx
-; CHECK-NEXT:    shrw %cl, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cx killed $cx killed $ecx
+; CHECK-NEXT:    ## kill: def $cl killed $cx
+; CHECK-NEXT:    shrw %cl, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i16 %a, %b
   ret i16 %c
 }
 
 define i32 @lshr_i32(i32 %a, i32 %b) {
 ; CHECK-LABEL: lshr_i32:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $ecx
-; CHECK-NEXT:    shrl %cl, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $ecx
+; CHECK-NEXT:    shrl %cl, %eax
 ; CHECK-NEXT:    retq
   %c = lshr i32 %a, %b
   ret i32 %c
 }
 
 define i64 @lshr_i64(i64 %a, i64 %b) {
 ; CHECK-LABEL: lshr_i64:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movq %rsi, %rcx
-; CHECK-NEXT:    ## kill: def $cl killed $rcx
-; CHECK-NEXT:    shrq %cl, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    ## kill: def $cl killed $rcx
+; CHECK-NEXT:    shrq %cl, %rax
 ; CHECK-NEXT:    retq
   %c = lshr i64 %a, %b
   ret i64 %c
 }
 
 define i8 @ashr_i8(i8 %a, i8 %b) {
 ; CHECK-LABEL: ashr_i8:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    sarb %cl, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $cl killed $ecx
+; CHECK-NEXT:    sarb %cl, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i8 %a, %b
   ret i8 %c
 }
 
 define i16 @ashr_i16(i16 %a, i16 %b) {
 ; CHECK-LABEL: ashr_i16:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $cx
-; CHECK-NEXT:    sarw %cl, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cx killed $cx killed $ecx
+; CHECK-NEXT:    ## kill: def $cl killed $cx
+; CHECK-NEXT:    sarw %cl, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i16 %a, %b
   ret i16 %c
 }
 
 define i32 @ashr_i32(i32 %a, i32 %b) {
 ; CHECK-LABEL: ashr_i32:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, %ecx
-; CHECK-NEXT:    ## kill: def $cl killed $ecx
-; CHECK-NEXT:    sarl %cl, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    ## kill: def $cl killed $ecx
+; CHECK-NEXT:    sarl %cl, %eax
 ; CHECK-NEXT:    retq
   %c = ashr i32 %a, %b
   ret i32 %c
 }
 
 define i64 @ashr_i64(i64 %a, i64 %b) {
 ; CHECK-LABEL: ashr_i64:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movq %rsi, %rcx
-; CHECK-NEXT:    ## kill: def $cl killed $rcx
-; CHECK-NEXT:    sarq %cl, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    ## kill: def $cl killed $rcx
+; CHECK-NEXT:    sarq %cl, %rax
 ; CHECK-NEXT:    retq
   %c = ashr i64 %a, %b
   ret i64 %c
 }
 
 define i8 @shl_imm1_i8(i8 %a) {
 ; CHECK-LABEL: shl_imm1_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shlb $1, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shlb $1, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = shl i8 %a, 1
   ret i8 %c
 }
 
 define i16 @shl_imm1_i16(i16 %a) {
 ; CHECK-LABEL: shl_imm1_i16:
 ; CHECK:       ## %bb.0:

 ; CHECK-NEXT:    retq
   %c = shl i64 %a, 1
   ret i64 %c
 }
 
 define i8 @lshr_imm1_i8(i8 %a) {
 ; CHECK-LABEL: lshr_imm1_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrb $1, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrb $1, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i8 %a, 1
   ret i8 %c
 }
 
 define i16 @lshr_imm1_i16(i16 %a) {
 ; CHECK-LABEL: lshr_imm1_i16:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrw $1, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrw $1, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i16 %a, 1
   ret i16 %c
 }
 
 define i32 @lshr_imm1_i32(i32 %a) {
 ; CHECK-LABEL: lshr_imm1_i32:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrl $1, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrl $1, %eax
 ; CHECK-NEXT:    retq
   %c = lshr i32 %a, 1
   ret i32 %c
 }
 
 define i64 @lshr_imm1_i64(i64 %a) {
 ; CHECK-LABEL: lshr_imm1_i64:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrq $1, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    shrq $1, %rax
 ; CHECK-NEXT:    retq
   %c = lshr i64 %a, 1
   ret i64 %c
 }
 
 define i8 @ashr_imm1_i8(i8 %a) {
 ; CHECK-LABEL: ashr_imm1_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarb $1, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarb $1, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i8 %a, 1
   ret i8 %c
 }
 
 define i16 @ashr_imm1_i16(i16 %a) {
 ; CHECK-LABEL: ashr_imm1_i16:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarw $1, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarw $1, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i16 %a, 1
   ret i16 %c
 }
 
 define i32 @ashr_imm1_i32(i32 %a) {
 ; CHECK-LABEL: ashr_imm1_i32:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarl $1, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarl $1, %eax
 ; CHECK-NEXT:    retq
   %c = ashr i32 %a, 1
   ret i32 %c
 }
 
 define i64 @ashr_imm1_i64(i64 %a) {
 ; CHECK-LABEL: ashr_imm1_i64:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarq $1, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    sarq $1, %rax
 ; CHECK-NEXT:    retq
   %c = ashr i64 %a, 1
   ret i64 %c
 }
 
 define i8 @shl_imm4_i8(i8 %a) {
 ; CHECK-LABEL: shl_imm4_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shlb $4, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shlb $4, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = shl i8 %a, 4
   ret i8 %c
 }
 
 define i16 @shl_imm4_i16(i16 %a) {
 ; CHECK-LABEL: shl_imm4_i16:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shlw $4, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shlw $4, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = shl i16 %a, 4
   ret i16 %c
 }
 
 define i32 @shl_imm4_i32(i32 %a) {
 ; CHECK-LABEL: shl_imm4_i32:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shll $4, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shll $4, %eax
 ; CHECK-NEXT:    retq
   %c = shl i32 %a, 4
   ret i32 %c
 }
 
 define i64 @shl_imm4_i64(i64 %a) {
 ; CHECK-LABEL: shl_imm4_i64:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shlq $4, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    shlq $4, %rax
 ; CHECK-NEXT:    retq
   %c = shl i64 %a, 4
   ret i64 %c
 }
 
 define i8 @lshr_imm4_i8(i8 %a) {
 ; CHECK-LABEL: lshr_imm4_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrb $4, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrb $4, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i8 %a, 4
   ret i8 %c
 }
 
 define i16 @lshr_imm4_i16(i16 %a) {
 ; CHECK-LABEL: lshr_imm4_i16:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrw $4, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrw $4, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = lshr i16 %a, 4
   ret i16 %c
 }
 
 define i32 @lshr_imm4_i32(i32 %a) {
 ; CHECK-LABEL: lshr_imm4_i32:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrl $4, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shrl $4, %eax
 ; CHECK-NEXT:    retq
   %c = lshr i32 %a, 4
   ret i32 %c
 }
 
 define i64 @lshr_imm4_i64(i64 %a) {
 ; CHECK-LABEL: lshr_imm4_i64:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    shrq $4, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    shrq $4, %rax
 ; CHECK-NEXT:    retq
   %c = lshr i64 %a, 4
   ret i64 %c
 }
 
 define i8 @ashr_imm4_i8(i8 %a) {
 ; CHECK-LABEL: ashr_imm4_i8:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarb $4, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarb $4, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i8 %a, 4
   ret i8 %c
 }
 
 define i16 @ashr_imm4_i16(i16 %a) {
 ; CHECK-LABEL: ashr_imm4_i16:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarw $4, %di
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarw $4, %ax
+; CHECK-NEXT:    ## kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %c = ashr i16 %a, 4
   ret i16 %c
 }
 
 define i32 @ashr_imm4_i32(i32 %a) {
 ; CHECK-LABEL: ashr_imm4_i32:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarl $4, %edi
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    sarl $4, %eax
 ; CHECK-NEXT:    retq
   %c = ashr i32 %a, 4
   ret i32 %c
 }
 
 define i64 @ashr_imm4_i64(i64 %a) {
 ; CHECK-LABEL: ashr_imm4_i64:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    sarq $4, %rdi
 ; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    sarq $4, %rax
 ; CHECK-NEXT:    retq
   %c = ashr i64 %a, 4
   ret i64 %c
 }
 
 ; Make sure we don't crash on out of bounds i8 shifts.
 define i8 @PR36731(i8 %a) {
 ; CHECK-LABEL: PR36731:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    movb $255, %cl
-; CHECK-NEXT:    shlb %cl, %dil
 ; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    movb $255, %cl
+; CHECK-NEXT:    shlb %cl, %al
+; CHECK-NEXT:    ## kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %b = shl i8 %a, -1
   ret i8 %b
 }

test/CodeGen/X86/schedule-x86-64-shld.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -print-schedule -mcpu=x86-64 | FileCheck %s --check-prefix=CHECK --check-prefix=GENERIC
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -print-schedule -mcpu=btver2 | FileCheck %s --check-prefix=CHECK --check-prefix=BTVER2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -print-schedule -mcpu=bdver1 | FileCheck %s --check-prefix=CHECK --check-prefix=BDVER1
 
 
 ; uint64_t lshift10(uint64_t a, uint64_t b)
 ; {
 ;     return (a << 10) | (b >> 54);
 ; }
 
 define i64 @lshift10_optsize(i64 %a, i64 %b) nounwind readnone optsize {
 ; GENERIC-LABEL: lshift10_optsize:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    shldq $10, %rsi, %rdi # sched: [2:0.67]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    shldq $10, %rsi, %rax # sched: [2:0.67]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift10_optsize:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    shldq $10, %rsi, %rdi # sched: [3:3.00]
 ; BTVER2-NEXT:    movq %rdi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    shldq $10, %rsi, %rax # sched: [3:3.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: lshift10_optsize:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    shldq $10, %rsi, %rdi
 ; BDVER1-NEXT:    movq %rdi, %rax
+; BDVER1-NEXT:    shldq $10, %rsi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shl = shl i64 %a, 10
   %shr = lshr i64 %b, 54
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 define i64 @lshift10(i64 %a, i64 %b) nounwind readnone {
 ; GENERIC-LABEL: lshift10:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    shldq $10, %rsi, %rdi # sched: [2:0.67]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    shldq $10, %rsi, %rax # sched: [2:0.67]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift10:
 ; BTVER2:       # %bb.0: # %entry
 ; BTVER2-NEXT:    shlq $10, %rdi # sched: [1:0.50]
 ; BTVER2-NEXT:    shrq $54, %rsi # sched: [1:0.50]
 ; BTVER2-NEXT:    leaq (%rsi,%rdi), %rax # sched: [1:0.50]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]

 ; {
 ;     return (a >> 62) | (b << 2);
 ; }
 
 ; Should be done via shld
 define i64 @rshift10_optsize(i64 %a, i64 %b) nounwind readnone optsize {
 ; GENERIC-LABEL: rshift10_optsize:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    shrdq $62, %rsi, %rdi # sched: [2:0.67]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    shrdq $62, %rsi, %rax # sched: [2:0.67]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: rshift10_optsize:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    shrdq $62, %rsi, %rdi # sched: [3:3.00]
 ; BTVER2-NEXT:    movq %rdi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    shrdq $62, %rsi, %rax # sched: [3:3.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: rshift10_optsize:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    shrdq $62, %rsi, %rdi
 ; BDVER1-NEXT:    movq %rdi, %rax
+; BDVER1-NEXT:    shrdq $62, %rsi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shl = lshr i64 %a, 62
   %shr = shl i64 %b, 2
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 ; Should be done via lea (x,y,4),z
 define i64 @rshift10(i64 %a, i64 %b) nounwind readnone {
 ; GENERIC-LABEL: rshift10:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    shrdq $62, %rsi, %rdi # sched: [2:0.67]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    shrdq $62, %rsi, %rax # sched: [2:0.67]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: rshift10:
 ; BTVER2:       # %bb.0: # %entry
 ; BTVER2-NEXT:    shrq $62, %rdi # sched: [1:0.50]
 ; BTVER2-NEXT:    leaq (%rdi,%rsi,4), %rax # sched: [2:1.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;

 ;uint64_t lshift(uint64_t a, uint64_t b, uint64_t c)
 ;{
 ;    return (a << c) | (b >> (64-c));
 ;}
 
 define i64 @lshift_cl_optsize(i64 %a, i64 %b, i64 %c) nounwind readnone optsize {
 ; GENERIC-LABEL: lshift_cl_optsize:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %edx, %ecx # sched: [1:0.33]
-; GENERIC-NEXT:    shldq %cl, %rsi, %rdi # sched: [4:1.50]
+; GENERIC-NEXT:    movq %rdx, %rcx # sched: [1:0.33]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
+; GENERIC-NEXT:    shldq %cl, %rsi, %rax # sched: [4:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift_cl_optsize:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    shldq %cl, %rsi, %rdi # sched: [4:4.00]
+; BTVER2-NEXT:    movq %rdx, %rcx # sched: [1:0.50]
 ; BTVER2-NEXT:    movq %rdi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    # kill: def $cl killed $cl killed $rcx
+; BTVER2-NEXT:    shldq %cl, %rsi, %rax # sched: [4:4.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: lshift_cl_optsize:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movl %edx, %ecx
-; BDVER1-NEXT:    shldq %cl, %rsi, %rdi
+; BDVER1-NEXT:    movq %rdx, %rcx
 ; BDVER1-NEXT:    movq %rdi, %rax
+; BDVER1-NEXT:    # kill: def $cl killed $cl killed $rcx
+; BDVER1-NEXT:    shldq %cl, %rsi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shl = shl i64 %a, %c
   %sub = sub nsw i64 64, %c
   %shr = lshr i64 %b, %sub
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 define i64 @lshift_cl(i64 %a, i64 %b, i64 %c) nounwind readnone {
 ; GENERIC-LABEL: lshift_cl:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %edx, %ecx # sched: [1:0.33]
-; GENERIC-NEXT:    shldq %cl, %rsi, %rdi # sched: [4:1.50]
+; GENERIC-NEXT:    movq %rdx, %rcx # sched: [1:0.33]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
+; GENERIC-NEXT:    shldq %cl, %rsi, %rax # sched: [4:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift_cl:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    shlq %cl, %rdi # sched: [1:0.50]
-; BTVER2-NEXT:    movl $64, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    subl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    # kill: def $cl killed $cl killed $ecx
-; BTVER2-NEXT:    shrq %cl, %rsi # sched: [1:0.50]
-; BTVER2-NEXT:    orq %rdi, %rsi # sched: [1:0.50]
+; BTVER2-NEXT:    movq %rdx, %rcx # sched: [1:0.50]
+; BTVER2-NEXT:    movl $64, %edx # sched: [1:0.50]
 ; BTVER2-NEXT:    movq %rsi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    subl %ecx, %edx # sched: [1:0.50]
+; BTVER2-NEXT:    shlq %cl, %rdi # sched: [1:0.50]
+; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
+; BTVER2-NEXT:    shrq %cl, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    orq %rdi, %rax # sched: [1:0.50]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: lshift_cl:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movl %edx, %ecx
-; BDVER1-NEXT:    shlq %cl, %rdi
-; BDVER1-NEXT:    movl $64, %ecx
-; BDVER1-NEXT:    subl %edx, %ecx
-; BDVER1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; BDVER1-NEXT:    shrq %cl, %rsi
-; BDVER1-NEXT:    orq %rdi, %rsi
+; BDVER1-NEXT:    movq %rdx, %rcx
 ; BDVER1-NEXT:    movq %rsi, %rax
+; BDVER1-NEXT:    shlq %cl, %rdi
+; BDVER1-NEXT:    movl $64, %edx
+; BDVER1-NEXT:    subl %ecx, %edx
+; BDVER1-NEXT:    movl %edx, %ecx
+; BDVER1-NEXT:    shrq %cl, %rax
+; BDVER1-NEXT:    orq %rdi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shl = shl i64 %a, %c
   %sub = sub nsw i64 64, %c
   %shr = lshr i64 %b, %sub
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 
 ;uint64_t rshift(uint64_t a, uint64_t b, int c)
 ;{
 ;    return (a >> c) | (b << (64-c));
 ;}
 
 define i64 @rshift_cl_optsize(i64 %a, i64 %b, i64 %c) nounwind readnone optsize {
 ; GENERIC-LABEL: rshift_cl_optsize:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %edx, %ecx # sched: [1:0.33]
-; GENERIC-NEXT:    shrdq %cl, %rsi, %rdi # sched: [4:1.50]
+; GENERIC-NEXT:    movq %rdx, %rcx # sched: [1:0.33]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
+; GENERIC-NEXT:    shrdq %cl, %rsi, %rax # sched: [4:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: rshift_cl_optsize:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    shrdq %cl, %rsi, %rdi # sched: [4:4.00]
+; BTVER2-NEXT:    movq %rdx, %rcx # sched: [1:0.50]
 ; BTVER2-NEXT:    movq %rdi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    # kill: def $cl killed $cl killed $rcx
+; BTVER2-NEXT:    shrdq %cl, %rsi, %rax # sched: [4:4.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: rshift_cl_optsize:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movl %edx, %ecx
-; BDVER1-NEXT:    shrdq %cl, %rsi, %rdi
+; BDVER1-NEXT:    movq %rdx, %rcx
 ; BDVER1-NEXT:    movq %rdi, %rax
+; BDVER1-NEXT:    # kill: def $cl killed $cl killed $rcx
+; BDVER1-NEXT:    shrdq %cl, %rsi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shr = lshr i64 %a, %c
   %sub = sub nsw i64 64, %c
   %shl = shl i64 %b, %sub
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 define i64 @rshift_cl(i64 %a, i64 %b, i64 %c) nounwind readnone {
 ; GENERIC-LABEL: rshift_cl:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %edx, %ecx # sched: [1:0.33]
-; GENERIC-NEXT:    shrdq %cl, %rsi, %rdi # sched: [4:1.50]
+; GENERIC-NEXT:    movq %rdx, %rcx # sched: [1:0.33]
 ; GENERIC-NEXT:    movq %rdi, %rax # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
+; GENERIC-NEXT:    shrdq %cl, %rsi, %rax # sched: [4:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: rshift_cl:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    shrq %cl, %rdi # sched: [1:0.50]
-; BTVER2-NEXT:    movl $64, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    subl %edx, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    # kill: def $cl killed $cl killed $ecx
-; BTVER2-NEXT:    shlq %cl, %rsi # sched: [1:0.50]
-; BTVER2-NEXT:    orq %rdi, %rsi # sched: [1:0.50]
+; BTVER2-NEXT:    movq %rdx, %rcx # sched: [1:0.50]
+; BTVER2-NEXT:    movl $64, %edx # sched: [1:0.50]
 ; BTVER2-NEXT:    movq %rsi, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    subl %ecx, %edx # sched: [1:0.50]
+; BTVER2-NEXT:    shrq %cl, %rdi # sched: [1:0.50]
+; BTVER2-NEXT:    movl %edx, %ecx # sched: [1:0.50]
+; BTVER2-NEXT:    shlq %cl, %rax # sched: [1:0.50]
+; BTVER2-NEXT:    orq %rdi, %rax # sched: [1:0.50]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: rshift_cl:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movl %edx, %ecx
-; BDVER1-NEXT:    shrq %cl, %rdi
-; BDVER1-NEXT:    movl $64, %ecx
-; BDVER1-NEXT:    subl %edx, %ecx
-; BDVER1-NEXT:    # kill: def $cl killed $cl killed $ecx
-; BDVER1-NEXT:    shlq %cl, %rsi
-; BDVER1-NEXT:    orq %rdi, %rsi
+; BDVER1-NEXT:    movq %rdx, %rcx
 ; BDVER1-NEXT:    movq %rsi, %rax
+; BDVER1-NEXT:    shrq %cl, %rdi
+; BDVER1-NEXT:    movl $64, %edx
+; BDVER1-NEXT:    subl %ecx, %edx
+; BDVER1-NEXT:    movl %edx, %ecx
+; BDVER1-NEXT:    shlq %cl, %rax
+; BDVER1-NEXT:    orq %rdi, %rax
 ; BDVER1-NEXT:    retq
 entry:
   %shr = lshr i64 %a, %c
   %sub = sub nsw i64 64, %c
   %shl = shl i64 %b, %sub
   %or = or i64 %shr, %shl
   ret i64 %or
 }
 
 ; extern uint64_t x;
 ;void lshift(uint64_t a, uint64_t b, uint_64_t c)
 ;{
 ;    x = (x << c) | (a >> (64-c));
 ;}
 @x = global i64 0, align 4
 
 define void @lshift_mem_cl_optsize(i64 %a, i64 %c) nounwind readnone optsize {
 ; GENERIC-LABEL: lshift_mem_cl_optsize:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %esi, %ecx # sched: [1:0.33]
+; GENERIC-NEXT:    movq %rsi, %rcx # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
 ; GENERIC-NEXT:    shldq %cl, %rdi, {{.*}}(%rip) # sched: [10:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift_mem_cl_optsize:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movl %esi, %ecx # sched: [1:0.50]
+; BTVER2-NEXT:    movq %rsi, %rcx # sched: [1:0.50]
+; BTVER2-NEXT:    # kill: def $cl killed $cl killed $rcx
 ; BTVER2-NEXT:    shldq %cl, %rdi, {{.*}}(%rip) # sched: [9:11.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: lshift_mem_cl_optsize:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movl %esi, %ecx
+; BDVER1-NEXT:    movq %rsi, %rcx
+; BDVER1-NEXT:    # kill: def $cl killed $cl killed $rcx
 ; BDVER1-NEXT:    shldq %cl, %rdi, {{.*}}(%rip)
 ; BDVER1-NEXT:    retq
 entry:
   %b = load i64, i64* @x
   %shl = shl i64 %b, %c
   %sub = sub nsw i64 64, %c
   %shr = lshr i64 %a, %sub
   %or = or i64 %shl, %shr
   store i64 %or, i64* @x
   ret void
 }
 
 define void @lshift_mem_cl(i64 %a, i64 %c) nounwind readnone {
 ; GENERIC-LABEL: lshift_mem_cl:
 ; GENERIC:       # %bb.0: # %entry
-; GENERIC-NEXT:    movl %esi, %ecx # sched: [1:0.33]
+; GENERIC-NEXT:    movq %rsi, %rcx # sched: [1:0.33]
+; GENERIC-NEXT:    # kill: def $cl killed $cl killed $rcx
 ; GENERIC-NEXT:    shldq %cl, %rdi, {{.*}}(%rip) # sched: [10:1.50]
 ; GENERIC-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: lshift_mem_cl:
 ; BTVER2:       # %bb.0: # %entry
-; BTVER2-NEXT:    movq {{.*}}(%rip), %rax # sched: [5:1.00]
-; BTVER2-NEXT:    movl %esi, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    shlq %cl, %rax # sched: [1:0.50]
-; BTVER2-NEXT:    movl $64, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    subl %esi, %ecx # sched: [1:0.50]
-; BTVER2-NEXT:    # kill: def $cl killed $cl killed $ecx
+; BTVER2-NEXT:    movq {{.*}}(%rip), %rdx # sched: [5:1.00]
+; BTVER2-NEXT:    movq %rsi, %rcx # sched: [1:0.50]
+; BTVER2-NEXT:    movl $64, %eax # sched: [1:0.50]
+; BTVER2-NEXT:    subl %ecx, %eax # sched: [1:0.50]
+; BTVER2-NEXT:    shlq %cl, %rdx # sched: [1:0.50]
+; BTVER2-NEXT:    movl %eax, %ecx # sched: [1:0.50]
 ; BTVER2-NEXT:    shrq %cl, %rdi # sched: [1:0.50]
-; BTVER2-NEXT:    orq %rax, %rdi # sched: [1:0.50]
+; BTVER2-NEXT:    orq %rdx, %rdi # sched: [1:0.50]
 ; BTVER2-NEXT:    movq %rdi, {{.*}}(%rip) # sched: [1:1.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; BDVER1-LABEL: lshift_mem_cl:
 ; BDVER1:       # %bb.0: # %entry
-; BDVER1-NEXT:    movq {{.*}}(%rip), %rax
-; BDVER1-NEXT:    movl %esi, %ecx
-; BDVER1-NEXT:    shlq %cl, %rax
-; BDVER1-NEXT:    movl $64, %ecx
-; BDVER1-NEXT:    subl %esi, %ecx
-; BDVER1-NEXT:    # kill: def $cl killed $cl killed $ecx
+; BDVER1-NEXT:    movq %rsi, %rcx
+; BDVER1-NEXT:    movq {{.*}}(%rip), %rdx
+; BDVER1-NEXT:    shlq %cl, %rdx
+; BDVER1-NEXT:    movl $64, %eax
+; BDVER1-NEXT:    subl %ecx, %eax
+; BDVER1-NEXT:    movl %eax, %ecx
 ; BDVER1-NEXT:    shrq %cl, %rdi
-; BDVER1-NEXT:    orq %rax, %rdi
+; BDVER1-NEXT:    orq %rdx, %rdi
 ; BDVER1-NEXT:    movq %rdi, {{.*}}(%rip)
 ; BDVER1-NEXT:    retq
 entry:
   %b = load i64, i64* @x
   %shl = shl i64 %b, %c
   %sub = sub nsw i64 64, %c
   %shr = lshr i64 %a, %sub
   %or = or i64 %shl, %shr

test/CodeGen/X86/sret-implicit.ll

 ; RUN: llc -mtriple=x86_64-apple-darwin8 < %s | FileCheck %s --check-prefix=X64
 ; RUN: llc -mtriple=x86_64-pc-linux < %s | FileCheck %s --check-prefix=X64
 ; RUN: llc -mtriple=i686-pc-linux < %s | FileCheck %s --check-prefix=X86
 ; RUN: llc -mtriple=x86_64-apple-darwin8 -terminal-rule < %s | FileCheck %s --check-prefix=X64
 ; RUN: llc -mtriple=x86_64-pc-linux -terminal-rule < %s | FileCheck %s --check-prefix=X64
 
 define void @sret_void(i32* sret %p) {
   store i32 0, i32* %p
   ret void
 }
 
 ; X64-LABEL: sret_void
-; X64-DAG: movl $0, (%rdi)
 ; X64-DAG: movq %rdi, %rax
+; X64-DAG: movl $0, (%rdi)
 ; X64: retq
 
 ; X86-LABEL: sret_void
 ; X86: movl 4(%esp), %eax
 ; X86: movl $0, (%eax)
 ; X86: retl
 
 define i256 @sret_demoted() {
   ret i256 0
 }
 
 ; X64-LABEL: sret_demoted
-; X64-DAG: movq $0, (%rdi)
 ; X64-DAG: movq %rdi, %rax
+; X64-DAG: movq $0, (%rdi)
 ; X64: retq
 
 ; X86-LABEL: sret_demoted
 ; X86: movl 4(%esp), %eax
 ; X86: movl $0, (%eax)
 ; X86: retl

test/CodeGen/X86/vector-shift-ashr-128.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefix=ALL --check-prefix=SSE --check-prefix=SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.1 | FileCheck %s --check-prefix=ALL --check-prefix=SSE --check-prefix=SSE41
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefix=ALL --check-prefix=AVX --check-prefix=AVX1
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefix=ALL --check-prefix=AVX --check-prefix=AVX2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+xop,+avx | FileCheck %s --check-prefix=ALL --check-prefix=XOP --check-prefix=XOPAVX1
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+xop,+avx2 | FileCheck %s --check-prefix=ALL --check-prefix=XOP --check-prefix=XOPAVX2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512dq | FileCheck %s --check-prefix=ALL --check-prefix=AVX512 --check-prefix=AVX512DQ
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw | FileCheck %s --check-prefix=ALL --check-prefix=AVX512 --check-prefix=AVX512BW

 ; SSE2-NEXT:    pandn %xmm0, %xmm2
 ; SSE2-NEXT:    psraw $1, %xmm0
 ; SSE2-NEXT:    pand %xmm1, %xmm0
 ; SSE2-NEXT:    por %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: var_shift_v8i16:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    movdqa %xmm0, %xmm2
-; SSE41-NEXT:    movdqa %xmm1, %xmm0
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    movdqa %xmm0, %xmm1
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
 ; SSE41-NEXT:    psllw $12, %xmm0
-; SSE41-NEXT:    psllw $4, %xmm1
-; SSE41-NEXT:    por %xmm0, %xmm1
-; SSE41-NEXT:    movdqa %xmm1, %xmm3
-; SSE41-NEXT:    paddw %xmm1, %xmm3
-; SSE41-NEXT:    movdqa %xmm2, %xmm4
+; SSE41-NEXT:    psllw $4, %xmm2
+; SSE41-NEXT:    por %xmm0, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm3
+; SSE41-NEXT:    paddw %xmm2, %xmm3
+; SSE41-NEXT:    movdqa %xmm1, %xmm4
 ; SSE41-NEXT:    psraw $8, %xmm4
-; SSE41-NEXT:    movdqa %xmm1, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm4, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psraw $4, %xmm1
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    pblendvb %xmm0, %xmm4, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psraw $4, %xmm2
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psraw $2, %xmm1
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psraw $2, %xmm2
 ; SSE41-NEXT:    paddw %xmm3, %xmm3
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psraw $1, %xmm1
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psraw $1, %xmm2
 ; SSE41-NEXT:    paddw %xmm3, %xmm3
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: var_shift_v8i16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpsllw $12, %xmm1, %xmm2
 ; AVX1-NEXT:    vpsllw $4, %xmm1, %xmm1
 ; AVX1-NEXT:    vpor %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vpaddw %xmm1, %xmm1, %xmm2

test/CodeGen/X86/vector-shift-lshr-128.ll

 ; SSE2-NEXT:    pandn %xmm0, %xmm2
 ; SSE2-NEXT:    psrlw $1, %xmm0
 ; SSE2-NEXT:    pand %xmm1, %xmm0
 ; SSE2-NEXT:    por %xmm2, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: var_shift_v8i16:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    movdqa %xmm0, %xmm2
-; SSE41-NEXT:    movdqa %xmm1, %xmm0
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    movdqa %xmm0, %xmm1
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
 ; SSE41-NEXT:    psllw $12, %xmm0
-; SSE41-NEXT:    psllw $4, %xmm1
-; SSE41-NEXT:    por %xmm0, %xmm1
-; SSE41-NEXT:    movdqa %xmm1, %xmm3
-; SSE41-NEXT:    paddw %xmm1, %xmm3
-; SSE41-NEXT:    movdqa %xmm2, %xmm4
+; SSE41-NEXT:    psllw $4, %xmm2
+; SSE41-NEXT:    por %xmm0, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm3
+; SSE41-NEXT:    paddw %xmm2, %xmm3
+; SSE41-NEXT:    movdqa %xmm1, %xmm4
 ; SSE41-NEXT:    psrlw $8, %xmm4
-; SSE41-NEXT:    movdqa %xmm1, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm4, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psrlw $4, %xmm1
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    pblendvb %xmm0, %xmm4, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psrlw $4, %xmm2
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psrlw $2, %xmm1
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psrlw $2, %xmm2
 ; SSE41-NEXT:    paddw %xmm3, %xmm3
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm1
-; SSE41-NEXT:    psrlw $1, %xmm1
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm2
+; SSE41-NEXT:    psrlw $1, %xmm2
 ; SSE41-NEXT:    paddw %xmm3, %xmm3
 ; SSE41-NEXT:    movdqa %xmm3, %xmm0
-; SSE41-NEXT:    pblendvb %xmm0, %xmm1, %xmm2
-; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    pblendvb %xmm0, %xmm2, %xmm1
+; SSE41-NEXT:    movdqa %xmm1, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: var_shift_v8i16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpsllw $12, %xmm1, %xmm2
 ; AVX1-NEXT:    vpsllw $4, %xmm1, %xmm1
 ; AVX1-NEXT:    vpor %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vpaddw %xmm1, %xmm1, %xmm2

test/CodeGen/X86/vectorcall.ll

 ; RUN: llc -mtriple=i686-pc-win32 -mattr=+sse2 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=X86
 ; RUN: llc -mtriple=x86_64-pc-win32 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=X64
 
 ; Test integer arguments.
 
 define x86_vectorcallcc i32 @test_int_1() {
   ret i32 0
 }
 ; CHECK-LABEL: {{^}}test_int_1@@0:
 ; CHECK: xorl %eax, %eax
 
 define x86_vectorcallcc i32 @test_int_2(i32 inreg %a) {
   ret i32 %a
 }
 ; X86-LABEL: {{^}}test_int_2@@4:
 ; X64-LABEL: {{^}}test_int_2@@8:
 ; CHECK: movl %ecx, %eax
 
 define x86_vectorcallcc i32 @test_int_3(i64 inreg %a) {
   %at = trunc i64 %a to i32
   ret i32 %at
 }
 ; X86-LABEL: {{^}}test_int_3@@8:
 ; X64-LABEL: {{^}}test_int_3@@8:
-; CHECK: movl %ecx, %eax
+; X86: movl %ecx, %eax
+; X64: movq %rcx, %rax
 
 define x86_vectorcallcc i32 @test_int_4(i32 inreg %a, i32 inreg %b) {
   %s = add i32 %a, %b
   ret i32 %s
 }
 ; X86-LABEL: {{^}}test_int_4@@8:
 ; X86: leal (%ecx,%edx), %eax
 ; X64-LABEL: {{^}}test_int_4@@16:

 entry:
   %d = alloca %struct.HVA2, align 16
   store %struct.HVA2 %dd, %struct.HVA2* %d, align 16
   %y5 = getelementptr inbounds %struct.HVA2, %struct.HVA2* %d, i32 0, i32 1
   %0 = load <4 x float>, <4 x float>* %y5, align 16
   ret <4 x float> %0
 }
 ; CHECK-LABEL: test_mixed_5
-; CHECK:       movaps %xmm5, 16(%{{(e|r)}}sp)
-; CHECK:       movaps %xmm5, %xmm0
+; CHECK-DAG:   movaps %xmm{{[0,5]}}, 16(%{{(e|r)}}sp)
+; CHECK-DAG:   movaps %xmm5, %xmm0
 ; CHECK:       ret{{[ql]}}
 
 define x86_vectorcallcc %struct.HVA4 @test_mixed_6(%struct.HVA4 inreg %a, %struct.HVA4* %b) {
 entry:
   %retval = alloca %struct.HVA4, align 16
   %0 = bitcast %struct.HVA4* %retval to i8*
   %1 = bitcast %struct.HVA4* %b to i8*
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 16 %0, i8* align 16 %1, i32 64, i1 false)
entry:
   %0 = bitcast %struct.HVA5* %a to i8*
   call void @llvm.memset.p0i8.i64(i8* align 16 %0, i8 0, i64 80, i1 false)
   %1 = bitcast %struct.HVA5* %agg.result to i8*
   %2 = bitcast %struct.HVA5* %a to i8*
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 16 %1, i8* align 16 %2, i64 80, i1 false)
   ret void
 }
 ; CHECK-LABEL: test_mixed_7
+; X64:         mov{{[ql]}}  %rcx, %rax
 ; CHECK:       movaps %xmm{{[0-9]}}, 64(%{{rcx|eax}})
 ; CHECK:       movaps %xmm{{[0-9]}}, 48(%{{rcx|eax}})
 ; CHECK:       movaps %xmm{{[0-9]}}, 32(%{{rcx|eax}})
 ; CHECK:       movaps %xmm{{[0-9]}}, 16(%{{rcx|eax}})
 ; CHECK:       movaps %xmm{{[0-9]}}, (%{{rcx|eax}})
-; X64:         mov{{[ql]}}  %rcx, %rax
 ; CHECK:       ret{{[ql]}}
 
 define x86_vectorcallcc <4 x float> @test_mixed_8(<4 x float> %a, <4 x float> %b, <4 x float> %c, <4 x float> %d, i32 %e, <4 x float> %f) {
 entry:
   %f.addr = alloca <4 x float>, align 16
   store <4 x float> %f, <4 x float>* %f.addr, align 16
   %0 = load <4 x float>, <4 x float>* %f.addr, align 16
   ret <4 x float> %0

test/CodeGen/X86/vselect-minmax.ll

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test121:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm5, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm6, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    movdqa %xmm8, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm3, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test121:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm2, %xmm5

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test122:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm5, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm6, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    movdqa %xmm8, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm3, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test122:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm2, %xmm5

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test123:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test123:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm2, %xmm0, %xmm5

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test124:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test124:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm2, %xmm0, %xmm5

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test125:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm0, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm0, %xmm5
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    pxor %xmm7, %xmm6
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm1, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
-; SSE4-NEXT:    movdqa %xmm5, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm4
+; SSE4-NEXT:    movdqa %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
 ; SSE4-NEXT:    movdqa %xmm2, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
-; SSE4-NEXT:    movdqa %xmm6, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm1
+; SSE4-NEXT:    movdqa %xmm9, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm7, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
-; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm8, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test125:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm6

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test126:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm0, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm0, %xmm5
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    pxor %xmm7, %xmm6
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm1, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
-; SSE4-NEXT:    movdqa %xmm5, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm4
+; SSE4-NEXT:    movdqa %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
 ; SSE4-NEXT:    movdqa %xmm2, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
-; SSE4-NEXT:    movdqa %xmm6, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm1
+; SSE4-NEXT:    movdqa %xmm9, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm7, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
-; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm8, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test126:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm6

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test127:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm4, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm5, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm4, %xmm5
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    pxor %xmm7, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
-; SSE4-NEXT:    movdqa %xmm6, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
+; SSE4-NEXT:    movdqa %xmm9, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm1
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm3, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
-; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
+; SSE4-NEXT:    movapd %xmm5, %xmm0
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test127:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm6

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test128:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm4, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm5, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm4, %xmm5
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    pxor %xmm7, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
-; SSE4-NEXT:    movdqa %xmm6, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
+; SSE4-NEXT:    movdqa %xmm9, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm1
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm3, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
-; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
+; SSE4-NEXT:    movapd %xmm5, %xmm0
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test128:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm6

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test153:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test153:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm2, %xmm0, %xmm5

 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test154:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test154:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm2, %xmm0, %xmm5

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test155:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm8
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm0, %xmm7
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm4
+; SSE4-NEXT:    pcmpgtq %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm5, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
 ; SSE4-NEXT:    movdqa %xmm6, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    movdqa %xmm8, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm3, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    movapd %xmm5, %xmm1
 ; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test155:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm5
 ; AVX1-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX1-NEXT:    vpcmpgtq %xmm0, %xmm2, %xmm5

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test156:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm4, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm5, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm4, %xmm5
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    pxor %xmm7, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm4
+; SSE4-NEXT:    pxor %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm1, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
-; SSE4-NEXT:    movdqa %xmm6, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm4, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
+; SSE4-NEXT:    movdqa %xmm9, %xmm1
+; SSE4-NEXT:    pxor %xmm7, %xmm1
 ; SSE4-NEXT:    movdqa %xmm2, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
-; SSE4-NEXT:    movdqa %xmm7, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm3, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
-; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm3, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
+; SSE4-NEXT:    movapd %xmm5, %xmm0
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test156:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm6

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test159:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm0, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm0, %xmm5
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    pxor %xmm7, %xmm6
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm1, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
-; SSE4-NEXT:    movdqa %xmm5, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm4
+; SSE4-NEXT:    movdqa %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
 ; SSE4-NEXT:    movdqa %xmm2, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
-; SSE4-NEXT:    movdqa %xmm6, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm1
+; SSE4-NEXT:    movdqa %xmm9, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm7, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
-; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm8, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test159:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm6

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm5
 ; SSE2-NEXT:    pand %xmm5, %xmm3
 ; SSE2-NEXT:    pandn %xmm7, %xmm5
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: test160:
 ; SSE4:       # %bb.0: # %entry
-; SSE4-NEXT:    movdqa %xmm0, %xmm9
-; SSE4-NEXT:    movdqa {{.*#+}} xmm8 = [9223372036854775808,9223372036854775808]
-; SSE4-NEXT:    movdqa %xmm0, %xmm10
-; SSE4-NEXT:    pxor %xmm8, %xmm10
+; SSE4-NEXT:    movdqa %xmm7, %xmm8
+; SSE4-NEXT:    movdqa %xmm6, %xmm9
+; SSE4-NEXT:    movdqa %xmm5, %xmm10
+; SSE4-NEXT:    movdqa %xmm0, %xmm5
+; SSE4-NEXT:    movdqa {{.*#+}} xmm7 = [9223372036854775808,9223372036854775808]
+; SSE4-NEXT:    movdqa %xmm0, %xmm6
+; SSE4-NEXT:    pxor %xmm7, %xmm6
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm10, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm9, %xmm4
-; SSE4-NEXT:    movdqa %xmm1, %xmm9
-; SSE4-NEXT:    pxor %xmm8, %xmm9
-; SSE4-NEXT:    movdqa %xmm5, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pcmpgtq %xmm9, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm6, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm4
+; SSE4-NEXT:    movdqa %xmm1, %xmm5
+; SSE4-NEXT:    pxor %xmm7, %xmm5
+; SSE4-NEXT:    movdqa %xmm10, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm10
 ; SSE4-NEXT:    movdqa %xmm2, %xmm1
-; SSE4-NEXT:    pxor %xmm8, %xmm1
-; SSE4-NEXT:    movdqa %xmm6, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm1
+; SSE4-NEXT:    movdqa %xmm9, %xmm0
+; SSE4-NEXT:    pxor %xmm7, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm6
+; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm9
 ; SSE4-NEXT:    movdqa %xmm3, %xmm0
-; SSE4-NEXT:    pxor %xmm8, %xmm0
-; SSE4-NEXT:    pxor %xmm7, %xmm8
-; SSE4-NEXT:    pcmpgtq %xmm0, %xmm8
-; SSE4-NEXT:    movdqa %xmm8, %xmm0
-; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm7
+; SSE4-NEXT:    pxor %xmm7, %xmm0
+; SSE4-NEXT:    pxor %xmm8, %xmm7
+; SSE4-NEXT:    pcmpgtq %xmm0, %xmm7
+; SSE4-NEXT:    movdqa %xmm7, %xmm0
+; SSE4-NEXT:    blendvpd %xmm0, %xmm3, %xmm8
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
-; SSE4-NEXT:    movapd %xmm5, %xmm1
-; SSE4-NEXT:    movapd %xmm6, %xmm2
-; SSE4-NEXT:    movapd %xmm7, %xmm3
+; SSE4-NEXT:    movapd %xmm10, %xmm1
+; SSE4-NEXT:    movapd %xmm9, %xmm2
+; SSE4-NEXT:    movapd %xmm8, %xmm3
 ; SSE4-NEXT:    retq
 ;
 ; AVX1-LABEL: test160:
 ; AVX1:       # %bb.0: # %entry
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; AVX1-NEXT:    vpxor %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm6