[DAG] Enable ISD::SRL SimplifyMultipleUseDemandedBits handling inside SimplifyDemandedBits

[RKSimon]

Summary

This patch allows SimplifyDemandedBits to call SimplifyMultipleUseDemandedBits in cases where the ISD::SRL source operand has other uses, enabling us to peek through the shifted value if we don't demand all the bits/elts.

This is another step towards removing SelectionDAG::GetDemandedBits and just using TargetLowering::SimplifyMultipleUseDemandedBits.

There a few cases where we end up with extra register moves which I think we can accept in exchange for the increased ILP.

[RKSimon]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

@arsenm @foad Not sure if pulling out the immediate is a good idea or not - shouldn't a u16 immediate be cheap?

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

This is worse. Integer constants -16 to 64 and a handful of FP values are free, but 0xffff is not so it requires materialization.

[RKSimon]

rebase

[RKSimon]

still looking at the remaining regressions

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433 ↗

(On Diff #319268)

Looks like we've defeated the RISCVISD::GORCI matching code

[craig.topper]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433 ↗

(On Diff #319268)

Running the tests through instcombine also breaks GORCI matching.

[craig.topper]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433 ↗

(On Diff #319268)

It's also worth noting, the tests that are failing are repeating the same pattern gorc pattern twice, which is redundant. The test was trying to test that we could detect the redundancy. I guess this patch may have seen some of the redundancy?

[RKSimon]

rebase (still needs work)

[RKSimon]

I've raised https://bugs.llvm.org/show_bug.cgi?id=51209 about the poor quality of the gorc2 pattern matching and the gorc2, gorc2 -> gorc2 tests.

[craig.topper]

@RKSimon are the other problems with this patch than just the GORCI matching?

[RKSimon]

In D77804#3073304, @craig.topper wrote:

@RKSimon are the other problems with this patch than just the GORCI matching?

The GORCI matching is the main one.

There is also some minor issues with MatchRotate - we should be allowed to match rotate/funnel by constant pre-legalization (see ARM/ror.ll) as that can be re-expanded later without any harm done, before we see through the pattern and lose it, although now that we match this quite well in InstCombine I'm not sure is this is as likely to happen.

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase - squashed a few more regressions...

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

Waiting for D124839 to land

[RKSimon]

Rebased after D124839 to just handle ISD::SRL shifts

[RKSimon]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

@arsenm @foad At EuroLLVM Matt suggested that maybe we should increase the tolerance to 2 uses of the large immediates before pulling out the constant?

llvm/test/CodeGen/ARM/uxtb.ll

112

I'm going to take a look at this, but I'm really not familiar with the UXTB matching code, so any pointers would be appreciated.

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

s_mov_b32 K + 2 * v_and_b32_32 = 16 bytes, 12 cycles
2 * (v_and_b32_e32 K) = 16 bytes, 8 cycles which is clearly better.

3 * (v_and_b32_e32 K) = 24 bytes, 12 cycles

So 2 uses of a constant seems plainly better for VOP1/VOP2 ops. Abbe that it becomes a code size vs. latency tradeoff

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

This decision is also generally made by SIFoldOperands. Probably need to fix it there and not in the DAG

[foad]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

I'm strongly in favour of never pulling out the constant (or rather, always folding into the instruction) and I have patches to that effect starting with D114643, which I'm hoping to get back to pretty soon.

[RKSimon]

rebase

[foad]

AMDGPU changes LGTM.

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

instcombine optimises this as well:

define i32 @test10(i32 %p0) {
  %tmp1 = lshr i32 %p0, 7
  %tmp2 = and i32 %tmp1, 16253176
  %tmp4 = lshr i32 %p0, 12
  %tmp5 = and i32 %tmp4, 458759
  %tmp7 = or i32 %tmp5, %tmp2
  ret i32 %tmp7
}

which has the same problem:

_test10:
@ %bb.0:
        mov     r1, #248
        mov     r2, #7
        orr     r1, r1, #16252928
        orr     r2, r2, #458752
        and     r1, r1, r0, lsr #7
        and     r0, r2, r0, lsr #12
        orr     r0, r0, r1
        bx      lr

[RKSimon]

llvm/test/CodeGen/Thumb2/thumb2-uxtb.ll

185

same problem - instcombine will have already optimized this to:

define i32 @test10(i32 %p0) {
  %tmp1 = lshr i32 %p0, 7
  %tmp2 = and i32 %tmp1, 16253176
  %tmp4 = lshr i32 %p0, 12
  %tmp5 = and i32 %tmp4, 458759
  %tmp7 = or i32 %tmp5, %tmp2
  ret i32 %tmp7
}

It feels like I'm avoiding the issue - but should I update the arm/thumb2 UXTB16 tests to match what the middle-end will have generated?

[dmgreen]

llvm/test/CodeGen/ARM/uxtb.ll

112

I was taking a look. The test is super old now, so old that it had signed types when it was originally added.

I was surprised to see that and 0x70007 is being recognised via an and 0xff00ff tablegen pattern - it goes into SelectionDAGISel::CheckAndMask which checks that the other mask bits are already 0.

I think that is what this is trying to test - that a smaller and mask still matches the UXTB16. Is it possible to change it to something that still captures that, without relying on the multi-use fold of the %tmp2 not happening?

Maybe something like this?

%p = and i32 %p0, 3
%a = shl i32 65537, %p
%b = lshr i32 %a, 1
%tmp7 = and i32 %b, 458759

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

Thanks for the hint - I'll give it a try

[RKSimon]

rebase with alternative uxtb16 tests

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

Thanks @dmgreen - those still match fine. Should I pre-commit these new tests and possibly alter the existing test10 variants with the -instcombine optimized IR to show they already fail to match?

[dmgreen]

llvm/test/CodeGen/ARM/uxtb.ll

112

That sounds good to me.

[RKSimon]

rebase

[RKSimon]

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

128–139

@jonpa @uweigand These tests are proving very fragile depending on the order of and/shifts - should SystemZ be preferring masking leading/trailing bits with shift-pairs over shift+and / and+shift do you think? We have TLI::shouldFoldConstantShiftPairToMask to hand that.

[uweigand]

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

128–139

Well, this specific test only loads and then saves unmodified a 3xi31 vector, so ideally however the masking is done, it should be optimized away as unnecessary in either case. That's what currently happens, not sure why this is changing with this PR.

In general, I think using an and-mask would be preferable over a shift pair on SystemZ.

[RKSimon]

Thanks @uweigand I'll take another look at this soon

[RKSimon]

rebase after D125836

[RKSimon]

rebase

[RKSimon]

rebase and prefer SimplifyDemandedBits over GetDemandedBits for trunc stores

[RKSimon]

Added (or (and X, C1), (and (or X, Y), C2)) -> (or (and X, C1|C2), (and Y, C2)) fold to try to reduce the SystemZ regression

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

This could be a preliminary patch. I don't think we'd get that in IR either (even without extra uses):
https://alive2.llvm.org/ce/z/g61VRe

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

If I'm reading the SystemZ debug spew correctly, we should have gotten this transform to fire twice, so it would do this:
https://alive2.llvm.org/ce/z/tUsepa
...but we miss it because we don't revisit the last 'or' node? Is that what D127115 would solve?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

I've confirmed that D127115 solves the SystemZ fun3 regression but not fun2

[uweigand]

IMO the fun2 regression probably shouldn't block the patch from being merged. I've looked into the sequences, and actually neither of them is even close to optimal.

Looking at the semantics, we have 8 x i32 inputs, which need to be truncated to i31, concatenated, and then stored, occupying 31 bytes of memory. Memory is written via three 8-byte stores, followed by a 4-byte, a 2-byte, and a 1-byte store, which does look optimal to me. However, the computation of the 64-bit values to be stored is not.

The first of these should be the value

(A << 33) | ((B << 2) & 0x1fffffffc) | ((C >> 29) & 3)

where A, B, and C are the first three i32 inputs.

However, the computation being performed is more like

((A << 25) | ((B >> 6) & 0x01ffffff)) << 8
| ((B << 58) | ((C & 0x7fffffff) << 27)) >> 56

which gets the correct result, but in about double the number of instructions or cycles that should be required.

While the variant with this PR is even slightly worse than the variant before, that's probably not really relevant given the fact both sequences are rather inefficient. Ideally, we could fix this to get (close to) an optimal sequence, but that would be a different issue. (I'm not even sure yet whether the current inefficiency is due to the middle end or the back end.)

[RKSimon]

Thanks - I have a lot of individual DAG / SimplifyDemanded patches in progress atm, plus we're now getting closer to completing D127115.

A few patches still have minor regressions that I'm addressing, but this one in particular I've been wondering how much of a real world issue illegal type copies like this actually are? If we were further away from 15.x branch I'd ask to get this in and we ensure we address it once all the patches are in, but given how close we are I'm going to wait for now.

[RKSimon]

I think I've covered all the remaining regressions now - D129765 has cleaned up a number of annoying cases - including the SystemZ v3i31 copy test!

[uweigand]

In D77804#3681806, @RKSimon wrote:

I think I've covered all the remaining regressions now - D129765 has cleaned up a number of annoying cases - including the SystemZ v3i31 copy test!

Thanks! SystemZ changes LGTM now as discussed above.

[RKSimon]

I think this is patch is good to go now - any more comments?

[foad]

AMDGPU changes still LGTM.

[spatel]

x86 diffs LGTM

llvm/test/CodeGen/X86/ins_subreg_coalesce-1.ll

8–10

Not sure if this test still models some situation that we care about, but you could put a TODO note on it (don't need to copy to %ecx?).

[yonghong-song]

Hi, we found a regression with some bpf code with this patch. The following shows the problem:

[$ ~/tmp] cat run.sh
/home/yhs/work/llvm-project/llvm/build.cur/install/bin/clang -target bpf -O2 -g -c t.c
[$ ~/tmp] cat t.c
typedef unsigned char u8;
struct event {
  u8 tag;
  u8 hostname[84];
};

void *g;
void bar(void *);

int foo() {
  struct event event = {};

  event.tag = 1;
  __builtin_memcpy(&event.hostname, g, 84);
  bar(&event);
  return 0;
}
[$ ~/tmp] ./run.sh
t.c:14:3: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

  __builtin_memcpy(&event.hostname, g, 84);
  ^
t.c:14:3: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

2 errors generated.
[$ ~/tmp]

The BPF program enforces the stack size <= 512 bytes. For the above program, with this patch, the code after dag insn selection is worse and eventually in register allocation stage, the stack size is more than 512 and caused the above issue.

To illustrate the problem in more details, without this patch, the lowered machine code looks like

  STB killed %7:gpr, %stack.1.event.i, 0, debug-location !21355 :: (store (s8) into %ir.event.i, align 8, !tbaa !21356); tracecon/src/bpf/tracecon.bpf.c:78:12 @[ tracecon/src/b
pf/tracecon.bpf.c:68:5 ]
  %8:gpr = LDB %6:gpr, 7, debug-location !21358 :: (load (s8) from %ir.call1.i + 7); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %8:gpr, %stack.1.event.i, 12, debug-location !21358 :: (store (s8) into %ir.hostname.i + 7); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %9:gpr = LDB %6:gpr, 6, debug-location !21358 :: (load (s8) from %ir.call1.i + 6); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %9:gpr, %stack.1.event.i, 11, debug-location !21358 :: (store (s8) into %ir.hostname.i + 6); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %10:gpr = LDB %6:gpr, 5, debug-location !21358 :: (load (s8) from %ir.call1.i + 5); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %10:gpr, %stack.1.event.i, 10, debug-location !21358 :: (store (s8) into %ir.hostname.i + 5); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf
.c:68:5 ]
  %11:gpr = LDB %6:gpr, 4, debug-location !21358 :: (load (s8) from %ir.call1.i + 4); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %11:gpr, %stack.1.event.i, 9, debug-location !21358 :: (store (s8) into %ir.hostname.i + 4); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %12:gpr = LDB %6:gpr, 3, debug-location !21358 :: (load (s8) from %ir.call1.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %12:gpr, %stack.1.event.i, 8, debug-location !21358 :: (store (s8) into %ir.hostname.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %13:gpr = LDB %6:gpr, 2, debug-location !21358 :: (load (s8) from %ir.call1.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %13:gpr, %stack.1.event.i, 7, debug-location !21358 :: (store (s8) into %ir.hostname.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %14:gpr = LDB %6:gpr, 1, debug-location !21358 :: (load (s8) from %ir.call1.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %14:gpr, %stack.1.event.i, 6, debug-location !21358 :: (store (s8) into %ir.hostname.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %15:gpr = LDB %6:gpr, 0, debug-location !21358 :: (load (s8) from %ir.call1.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %15:gpr, %stack.1.event.i, 5, debug-location !21358 :: (store (s8) into %ir.hostname.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68
:5 ]
  %16:gpr = LDB %6:gpr, 15, debug-location !21358 :: (load (s8) from %ir.call1.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %16:gpr, %stack.1.event.i, 20, debug-location !21358 :: (store (s8) into %ir.hostname.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bp
f.c:68:5 ]
  %17:gpr = LDB %6:gpr, 14, debug-location !21358 :: (load (s8) from %ir.call1.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %17:gpr, %stack.1.event.i, 19, debug-location !21358 :: (store (s8) into %ir.hostname.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bp
f.c:68:5 ]
...
  %88:gpr = LDB %6:gpr, 83, debug-location !21358 :: (load (s8) from %ir.call1.i + 83); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %88:gpr, %stack.1.event.i, 88, debug-location !21358 :: (store (s8) into %ir.hostname.i + 83); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %89:gpr = LDB %6:gpr, 82, debug-location !21358 :: (load (s8) from %ir.call1.i + 82); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %89:gpr, %stack.1.event.i, 87, debug-location !21358 :: (store (s8) into %ir.hostname.i + 82); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %90:gpr = LDB %6:gpr, 81, debug-location !21358 :: (load (s8) from %ir.call1.i + 81); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %90:gpr, %stack.1.event.i, 86, debug-location !21358 :: (store (s8) into %ir.hostname.i + 81); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %91:gpr = LDB %6:gpr, 80, debug-location !21358 :: (load (s8) from %ir.call1.i + 80); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %91:gpr, %stack.1.event.i, 85, debug-location !21358 :: (store (s8) into %ir.hostname.i + 80); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]

Pretty straightforward byte load and stores and the corresponding stack after register allocation,

# *** IR Dump After Greedy Register Allocator (greedy) ***:
# Machine code for function tcp_v4_connect_exit: NoPHIs, TracksLiveness, TiedOpsRewritten, TracksDebugUserValues
Frame Objects:
  fi#0: size=4, align=4, at location [SP]
  fi#1: size=89, align=8, at location [SP]
  fi#2: size=4, align=4, at location [SP]
Function Live Ins: $r1 in %0

But this patch, the code becomes very complex,

  %8:gpr = LDB %6:gpr, 71, debug-location !21358 :: (load (s8) from %ir.call1.i + 71); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %9:gpr = SLL_ri %8:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %10:gpr = LDB %6:gpr, 70, debug-location !21358 :: (load (s8) from %ir.call1.i + 70); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %11:gpr = OR_rr %9:gpr(tied-def 0), killed %10:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %12:gpr = LDB %6:gpr, 15, debug-location !21358 :: (load (s8) from %ir.call1.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %13:gpr = SLL_ri %12:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %14:gpr = LDB %6:gpr, 14, debug-location !21358 :: (load (s8) from %ir.call1.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %15:gpr = OR_rr %13:gpr(tied-def 0), killed %14:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %71:gpr = OR_rr %69:gpr(tied-def 0), killed %70:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %72:gpr = LDB %6:gpr, 77, debug-location !21358 :: (load (s8) from %ir.call1.i + 77); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %73:gpr = SLL_ri %72:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %74:gpr = LDB %6:gpr, 76, debug-location !21358 :: (load (s8) from %ir.call1.i + 76); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %75:gpr = OR_rr %73:gpr(tied-def 0), killed %74:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %76:gpr = LDB %6:gpr, 79, debug-location !21358 :: (load (s8) from %ir.call1.i + 79); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %77:gpr = SLL_ri %76:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %78:gpr = LDB %6:gpr, 78, debug-location !21358 :: (load (s8) from %ir.call1.i + 78); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %79:gpr = OR_rr %77:gpr(tied-def 0), killed %78:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %80:gpr = SLL_ri %27:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %81:gpr = OR_rr %80:gpr(tied-def 0), killed %23:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %82:gpr = SLL_ri %19:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %83:gpr = OR_rr %82:gpr(tied-def 0), killed %71:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %84:gpr = SLL_ri %15:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %85:gpr = OR_rr %84:gpr(tied-def 0), killed %67:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %86:gpr = SLL_ri %11:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %87:gpr = OR_rr %86:gpr(tied-def 0), killed %63:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %88:gpr = SLL_ri %43:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %100:gpr = LDB %6:gpr, 74, debug-location !21358 :: (load (s8) from %ir.call1.i + 74); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %101:gpr = LDB %6:gpr, 75, debug-location !21358 :: (load (s8) from %ir.call1.i + 75); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %102:gpr = SLL_ri %101:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %103:gpr = OR_rr %102:gpr(tied-def 0), %100:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %104:gpr = SLL_ri %103:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %105:gpr = OR_rr %104:gpr(tied-def 0), killed %99:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %106:gpr = SLL_ri %79:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %107:gpr = OR_rr %106:gpr(tied-def 0), killed %75:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %108:gpr = LDB %6:gpr, 64, debug-location !21358 :: (load (s8) from %ir.call1.i + 64); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %109:gpr = LDB %6:gpr, 65, debug-location !21358 :: (load (s8) from %ir.call1.i + 65); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %110:gpr = SLL_ri %109:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %111:gpr = OR_rr %110:gpr(tied-def 0), %108:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %112:gpr = LDB %6:gpr, 66, debug-location !21358 :: (load (s8) from %ir.call1.i + 66); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %113:gpr = LDB %6:gpr, 67, debug-location !21358 :: (load (s8) from %ir.call1.i + 67); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %114:gpr = SLL_ri %113:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %115:gpr = OR_rr %114:gpr(tied-def 0), %112:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %116:gpr = SLL_ri %115:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %117:gpr = OR_rr %116:gpr(tied-def 0), killed %111:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %225:gpr = OR_rr %224:gpr(tied-def 0), killed %117:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %226:gpr = SLL_ri %107:gpr(tied-def 0), 32, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %227:gpr = OR_rr %226:gpr(tied-def 0), killed %105:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %193:gpr, %stack.1.event.i, 8, debug-location !21358 :: (store (s8) into %ir.hostname.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5
 ]
  STB %192:gpr, %stack.1.event.i, 7, debug-location !21358 :: (store (s8) into %ir.hostname.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %189:gpr, %stack.1.event.i, 6, debug-location !21358 :: (store (s8) into %ir.hostname.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %188:gpr, %stack.1.event.i, 5, debug-location !21358 :: (store (s8) into %ir.hostname.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %183:gpr, %stack.1.event.i, 16, debug-location !21358 :: (store (s8) into %ir.hostname.i + 11); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68
:5 ]
...

And the code becomes very complex and inefficient and this caused later larger stack size,

# *** IR Dump After Greedy Register Allocator (greedy) ***:
# Machine code for function tcp_v4_connect_exit: NoPHIs, TracksLiveness, TiedOpsRewritten, TracksDebugUserValues
Frame Objects:
  fi#0: size=4, align=4, at location [SP]
  fi#1: size=89, align=8, at location [SP]
  fi#2: size=4, align=4, at location [SP]
  fi#3: size=8, align=8, at location [SP]
...
  fi#57: size=8, align=8, at location [SP]
  fi#58: size=8, align=8, at location [SP]
  fi#59: size=8, align=8, at location [SP]
  fi#60: size=8, align=8, at location [SP]
Function Live Ins: $r1 in %0

Could you help take a look at this problem and suggest how to fix it?

[RKSimon]

@yonghong-song Please can you raise this as an issue and include the IR as well? AFAICT this is a perf regression, and not an actual bug

[yonghong-song]

In D77804#3804930, @RKSimon wrote:

@yonghong-song Please can you raise this as an issue and include the IR as well? AFAICT this is a perf regression, and not an actual bug

Thanks @RKSimon, just created an llvm-project issue https://github.com/llvm/llvm-project/issues/57872 thanks for taking care of this!

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

return TLI->SimplifyMultipleUseDemandedBits(V, DemandedBits, DemandedElts,
                                                 *this);
   case ISD::Constant: {
     const APInt &CVal = cast<ConstantSDNode>(V)->getAPIntValue();
     APInt NewVal = CVal & DemandedBits;
     if (NewVal != CVal)
       return getConstant(NewVal, SDLoc(V), V.getValueType());
     break;
   }
-  case ISD::SRL:
-    // Only look at single-use SRLs.
-    if (!V.getNode()->hasOneUse())
-      break;
-    if (auto *RHSC = dyn_cast<ConstantSDNode>(V.getOperand(1))) {
-      // See if we can recursively simplify the LHS.
-      unsigned Amt = RHSC->getZExtValue();
-
-      // Watch out for shift count overflow though.
-      if (Amt >= DemandedBits.getBitWidth())
-        break;
-      APInt SrcDemandedBits = DemandedBits << Amt;
-      if (SDValue SimplifyLHS =
-              GetDemandedBits(V.getOperand(0), SrcDemandedBits))
-        return getNode(ISD::SRL, SDLoc(V), V.getValueType(), SimplifyLHS,
-                       V.getOperand(1));
-    }
-    break;
   }
   return SDValue();
 }
 
 /// SignBitIsZero - Return true if the sign bit of Op is known to be zero.  We
 /// use this predicate to simplify operations downstream.
 bool SelectionDAG::SignBitIsZero(SDValue Op, unsigned Depth) const {
   unsigned BitWidth = Op.getScalarValueSizeInBits();

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (const APInt *SA =
                                Depth + 1))
         return true;
       assert(!Known.hasConflict() && "Bits known to be one AND zero?");
       Known.Zero <<= ShAmt;
       Known.One <<= ShAmt;
       // low bits known zero.
       Known.Zero.setLowBits(ShAmt);
 
+      // Attempt to avoid multi-use ops if we don't need anything from them.
+      if (!InDemandedMask.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+        SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+            Op0, InDemandedMask, DemandedElts, TLO.DAG, Depth + 1);
+        if (DemandedOp0) {
+          SDValue NewOp = TLO.DAG.getNode(ISD::SHL, dl, VT, DemandedOp0, Op1);
+          return TLO.CombineTo(Op, NewOp);
+        }
+      }
+
       // Try shrinking the operation as long as the shift amount will still be
       // in range.
       if ((ShAmt < DemandedBits.getActiveBits()) &&
           ShrinkDemandedOp(Op, BitWidth, DemandedBits, TLO))
         return true;
     }
 
     // If we are only demanding sign bits then we can use the shift source
if (const APInt *SA =
       if (SimplifyDemandedBits(Op0, InDemandedMask, DemandedElts, Known, TLO,
                                Depth + 1))
         return true;
       assert(!Known.hasConflict() && "Bits known to be one AND zero?");
       Known.Zero.lshrInPlace(ShAmt);
       Known.One.lshrInPlace(ShAmt);
       // High bits known zero.
       Known.Zero.setHighBits(ShAmt);
+
+      // Attempt to avoid multi-use ops if we don't need anything from them.
+      if (!InDemandedMask.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+        SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+            Op0, InDemandedMask, DemandedElts, TLO.DAG, Depth + 1);
+        if (DemandedOp0) {
+          SDValue NewOp = TLO.DAG.getNode(ISD::SRL, dl, VT, DemandedOp0, Op1);
+          return TLO.CombineTo(Op, NewOp);
+        }
+      }
     }
     break;
   }
   case ISD::SRA: {
     SDValue Op0 = Op.getOperand(0);
     SDValue Op1 = Op.getOperand(1);
     EVT ShiftVT = Op1.getValueType();
 

llvm/test/CodeGen/AArch64/parity.ll

; CHECK-NEXT:    ret
   %2 = and i16 %1, 1
   ret i16 %2
 }
 
 define i17 @parity_17(i17 %x) {
 ; CHECK-LABEL: parity_17:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    and w8, w0, #0x1ffff
-; CHECK-NEXT:    eor w8, w8, w8, lsr #16
-; CHECK-NEXT:    eor w8, w8, w8, lsr #8
+; CHECK-NEXT:    eor w9, w8, w8, lsr #16
+; CHECK-NEXT:    eor w8, w9, w8, lsr #8
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #4
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #2
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #1
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
   %1 = tail call i17 @llvm.ctpop.i17(i17 %x)
   %2 = and i17 %1, 1
   ret i17 %2

llvm/test/CodeGen/AArch64/shift-accumulate.ll

; CHECK-NEXT:    ret
   %4 = ashr <4 x i32> %2, <i32 30, i32 30, i32 30, i32 30>
   %5 = or <4 x i32> %3, %4
   ret <4 x i32> %5
 }
 
 define <1 x i64> @ssra_v1i64(<2 x i32> %0) {
 ; CHECK-LABEL: ssra_v1i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.2s, #64, lsl #24
 ; CHECK-NEXT:    ushr d1, d0, #63
+; CHECK-NEXT:    bic v0.2s, #64, lsl #24
 ; CHECK-NEXT:    ssra d1, d0, #62
 ; CHECK-NEXT:    fmov d0, d1
 ; CHECK-NEXT:    ret
   %2 = and <2 x i32> %0, <i32 3221225471, i32 3221225471>
   %3 = bitcast <2 x i32> %2 to <1 x i64>
   %4 = lshr <1 x i64> %3, <i64 63>
   %5 = ashr <1 x i64> %3, <i64 62>
   %6 = or <1 x i64> %4, %5
   ret <1 x i64> %6
 }
 
 define <2 x i64> @ssra_v2i64(<4 x i32> %0) {
 ; CHECK-LABEL: ssra_v2i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.4s, #64, lsl #24
 ; CHECK-NEXT:    ushr v1.2d, v0.2d, #63
+; CHECK-NEXT:    bic v0.4s, #64, lsl #24
 ; CHECK-NEXT:    ssra v1.2d, v0.2d, #62
 ; CHECK-NEXT:    mov v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %2 = and <4 x i32> %0, <i32 3221225471, i32 3221225471,i32 3221225471,i32 3221225471>
   %3 = bitcast <4 x i32> %2 to <2 x i64>
   %4 = lshr <2 x i64> %3, <i64 63, i64 63>
   %5 = ashr <2 x i64> %3, <i64 62, i64 62>
   %6 = or <2 x i64> %4, %5
   ret <2 x i64> %6
 }

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

 ; GFX6-LABEL: srem_v3i15:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0xb
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    s_bfe_i32 s10, s2, 0xf0000
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v5, s10
+; GFX6-NEXT:    v_mov_b32_e32 v2, s0
+; GFX6-NEXT:    v_alignbit_b32 v2, s1, v2, 30
+; GFX6-NEXT:    s_bfe_i32 s1, s0, 0xf0000
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v4, s1
+; GFX6-NEXT:    s_xor_b32 s1, s10, s1
+; GFX6-NEXT:    s_ashr_i32 s1, s1, 30
+; GFX6-NEXT:    s_or_b32 s1, s1, 1
+; GFX6-NEXT:    v_rcp_iflag_f32_e32 v6, v4
+; GFX6-NEXT:    v_mov_b32_e32 v7, s1
+; GFX6-NEXT:    s_lshr_b32 s9, s0, 15
+; GFX6-NEXT:    s_bfe_i32 s1, s2, 0xf000f
+; GFX6-NEXT:    v_mul_f32_e32 v6, v5, v6
+; GFX6-NEXT:    v_trunc_f32_e32 v6, v6
+; GFX6-NEXT:    v_mad_f32 v5, -v6, v4, v5
+; GFX6-NEXT:    v_cvt_i32_f32_e32 v6, v6
+; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v5|, |v4|
+; GFX6-NEXT:    v_cndmask_b32_e32 v4, 0, v7, vcc
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
+; GFX6-NEXT:    v_mul_lo_u32 v4, v4, s0
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0xf000f
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v5, s0
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v6, s1
 ; GFX6-NEXT:    v_alignbit_b32 v0, s3, v0, 30
 ; GFX6-NEXT:    s_movk_i32 s3, 0x7fff
-; GFX6-NEXT:    s_and_b32 s11, s0, s3
-; GFX6-NEXT:    s_bfe_i32 s11, s11, 0xf0000
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v2, s11
-; GFX6-NEXT:    s_and_b32 s9, s2, s3
-; GFX6-NEXT:    s_bfe_i32 s9, s9, 0xf0000
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v3, s9
-; GFX6-NEXT:    v_rcp_iflag_f32_e32 v4, v2
-; GFX6-NEXT:    s_xor_b32 s9, s9, s11
-; GFX6-NEXT:    s_ashr_i32 s9, s9, 30
-; GFX6-NEXT:    s_or_b32 s9, s9, 1
-; GFX6-NEXT:    v_mul_f32_e32 v4, v3, v4
-; GFX6-NEXT:    v_trunc_f32_e32 v4, v4
-; GFX6-NEXT:    v_mad_f32 v3, -v4, v2, v3
-; GFX6-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX6-NEXT:    v_mov_b32_e32 v5, s9
-; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
-; GFX6-NEXT:    v_cndmask_b32_e32 v2, 0, v5, vcc
-; GFX6-NEXT:    v_mov_b32_e32 v1, s0
-; GFX6-NEXT:    s_bfe_u32 s12, s0, 0xf000f
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
-; GFX6-NEXT:    v_alignbit_b32 v1, s1, v1, 30
-; GFX6-NEXT:    s_lshr_b32 s1, s0, 15
-; GFX6-NEXT:    v_mul_lo_u32 v2, v2, s0
-; GFX6-NEXT:    s_bfe_i32 s0, s12, 0xf0000
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v3, s0
-; GFX6-NEXT:    s_bfe_u32 s10, s2, 0xf000f
-; GFX6-NEXT:    s_lshr_b32 s8, s2, 15
-; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s2, v2
-; GFX6-NEXT:    s_bfe_i32 s2, s10, 0xf0000
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v4, s2
-; GFX6-NEXT:    v_rcp_iflag_f32_e32 v5, v3
-; GFX6-NEXT:    s_xor_b32 s0, s2, s0
+; GFX6-NEXT:    v_rcp_iflag_f32_e32 v7, v5
+; GFX6-NEXT:    v_and_b32_e32 v3, s3, v2
+; GFX6-NEXT:    v_sub_i32_e32 v4, vcc, s2, v4
+; GFX6-NEXT:    v_mul_f32_e32 v7, v6, v7
+; GFX6-NEXT:    v_trunc_f32_e32 v7, v7
+; GFX6-NEXT:    s_xor_b32 s0, s1, s0
+; GFX6-NEXT:    v_mad_f32 v6, -v7, v5, v6
+; GFX6-NEXT:    v_bfe_i32 v2, v2, 0, 15
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, 30
+; GFX6-NEXT:    v_cvt_i32_f32_e32 v7, v7
+; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v6|, |v5|
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v6, v2
 ; GFX6-NEXT:    s_or_b32 s0, s0, 1
-; GFX6-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX6-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX6-NEXT:    v_mad_f32 v4, -v5, v3, v4
-; GFX6-NEXT:    v_cvt_i32_f32_e32 v5, v5
-; GFX6-NEXT:    v_and_b32_e32 v1, s3, v1
-; GFX6-NEXT:    v_mov_b32_e32 v6, s0
-; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, |v3|
-; GFX6-NEXT:    v_cndmask_b32_e32 v3, 0, v6, vcc
-; GFX6-NEXT:    v_bfe_i32 v4, v1, 0, 15
-; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v5, v4
-; GFX6-NEXT:    v_and_b32_e32 v0, s3, v0
-; GFX6-NEXT:    v_bfe_i32 v6, v0, 0, 15
-; GFX6-NEXT:    v_cvt_f32_i32_e32 v7, v6
-; GFX6-NEXT:    v_rcp_iflag_f32_e32 v8, v5
-; GFX6-NEXT:    v_xor_b32_e32 v4, v6, v4
-; GFX6-NEXT:    v_ashrrev_i32_e32 v4, 30, v4
-; GFX6-NEXT:    v_or_b32_e32 v4, 1, v4
-; GFX6-NEXT:    v_mul_f32_e32 v6, v7, v8
-; GFX6-NEXT:    v_trunc_f32_e32 v6, v6
-; GFX6-NEXT:    v_mad_f32 v7, -v6, v5, v7
-; GFX6-NEXT:    v_cvt_i32_f32_e32 v6, v6
-; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v5|
-; GFX6-NEXT:    v_cndmask_b32_e32 v4, 0, v4, vcc
-; GFX6-NEXT:    v_mul_lo_u32 v3, v3, s1
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GFX6-NEXT:    v_mul_lo_u32 v1, v4, v1
-; GFX6-NEXT:    v_sub_i32_e32 v3, vcc, s8, v3
-; GFX6-NEXT:    v_and_b32_e32 v3, s3, v3
-; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, v1, v0
-; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
+; GFX6-NEXT:    v_mov_b32_e32 v8, s0
+; GFX6-NEXT:    v_and_b32_e32 v1, s3, v0
+; GFX6-NEXT:    v_cndmask_b32_e32 v5, 0, v8, vcc
+; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 15
+; GFX6-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GFX6-NEXT:    v_cvt_f32_i32_e32 v7, v0
+; GFX6-NEXT:    v_rcp_iflag_f32_e32 v8, v6
+; GFX6-NEXT:    v_xor_b32_e32 v0, v0, v2
+; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, 1, v0
+; GFX6-NEXT:    v_mul_f32_e32 v2, v7, v8
+; GFX6-NEXT:    v_trunc_f32_e32 v2, v2
+; GFX6-NEXT:    v_mad_f32 v7, -v2, v6, v7
+; GFX6-NEXT:    v_cvt_i32_f32_e32 v2, v2
+; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v6|
+; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
+; GFX6-NEXT:    v_mul_lo_u32 v5, v5, s9
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; GFX6-NEXT:    v_mul_lo_u32 v0, v0, v3
+; GFX6-NEXT:    s_lshr_b32 s8, s2, 15
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s8, v5
+; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, s3, v2
-; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
-; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
+; GFX6-NEXT:    v_and_b32_e32 v3, s3, v4
+; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 15, v2
+; GFX6-NEXT:    v_or_b32_e32 v2, v3, v2
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: srem_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_bfe_i32 s6, s2, 0xf0000
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s6
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s2
-; GFX9-NEXT:    v_alignbit_b32 v0, s3, v0, 30
-; GFX9-NEXT:    s_and_b32 s3, s2, s8
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s0
 ; GFX9-NEXT:    v_alignbit_b32 v1, s1, v1, 30
-; GFX9-NEXT:    s_and_b32 s1, s0, s8
-; GFX9-NEXT:    s_bfe_i32 s1, s1, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, s1
-; GFX9-NEXT:    s_bfe_i32 s3, s3, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s3
-; GFX9-NEXT:    s_xor_b32 s1, s3, s1
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v2
+; GFX9-NEXT:    s_bfe_i32 s1, s0, 0xf0000
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s1
+; GFX9-NEXT:    s_xor_b32 s1, s6, s1
 ; GFX9-NEXT:    s_ashr_i32 s1, s1, 30
-; GFX9-NEXT:    s_lshr_b32 s9, s2, 15
-; GFX9-NEXT:    s_bfe_u32 s10, s2, 0xf000f
-; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
-; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
-; GFX9-NEXT:    v_mad_f32 v3, -v4, v2, v3
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX9-NEXT:    s_lshr_b32 s11, s0, 15
-; GFX9-NEXT:    s_bfe_u32 s12, s0, 0xf000f
+; GFX9-NEXT:    v_alignbit_b32 v0, s3, v0, 30
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v6, v4
+; GFX9-NEXT:    s_lshr_b32 s3, s2, 15
+; GFX9-NEXT:    s_lshr_b32 s9, s0, 15
 ; GFX9-NEXT:    s_or_b32 s1, s1, 1
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[6:7], |v3|, |v2|
+; GFX9-NEXT:    v_mul_f32_e32 v6, v5, v6
+; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
+; GFX9-NEXT:    v_mad_f32 v5, -v6, v4, v5
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v6, v6
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[6:7], |v5|, |v4|
 ; GFX9-NEXT:    s_and_b64 s[6:7], s[6:7], exec
 ; GFX9-NEXT:    s_cselect_b32 s1, s1, 0
-; GFX9-NEXT:    v_add_u32_e32 v2, s1, v4
-; GFX9-NEXT:    s_bfe_i32 s1, s12, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s1
-; GFX9-NEXT:    v_mul_lo_u32 v2, v2, s0
-; GFX9-NEXT:    s_bfe_i32 s0, s10, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
+; GFX9-NEXT:    v_add_u32_e32 v4, s1, v6
+; GFX9-NEXT:    s_bfe_i32 s1, s0, 0xf000f
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s1
+; GFX9-NEXT:    v_mul_lo_u32 v4, v4, s0
+; GFX9-NEXT:    s_bfe_i32 s0, s2, 0xf000f
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v6, s0
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v5
 ; GFX9-NEXT:    s_xor_b32 s0, s0, s1
+; GFX9-NEXT:    v_and_b32_e32 v3, s8, v1
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
-; GFX9-NEXT:    s_or_b32 s3, s0, 1
-; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX9-NEXT:    v_and_b32_e32 v1, s8, v1
+; GFX9-NEXT:    v_mul_f32_e32 v7, v6, v7
+; GFX9-NEXT:    v_trunc_f32_e32 v7, v7
+; GFX9-NEXT:    v_mad_f32 v6, -v7, v5, v6
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v7, v7
+; GFX9-NEXT:    v_bfe_i32 v1, v1, 0, 15
+; GFX9-NEXT:    s_or_b32 s6, s0, 1
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v6|, |v5|
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v6, v1
 ; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
-; GFX9-NEXT:    s_cselect_b32 s0, s3, 0
-; GFX9-NEXT:    v_bfe_i32 v4, v1, 0, 15
-; GFX9-NEXT:    v_add_u32_e32 v3, s0, v5
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, v4
+; GFX9-NEXT:    s_cselect_b32 s0, s6, 0
+; GFX9-NEXT:    v_add_u32_e32 v5, s0, v7
+; GFX9-NEXT:    v_bfe_i32 v7, v0, 0, 15
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v8, v7
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v9, v6
+; GFX9-NEXT:    v_xor_b32_e32 v1, v7, v1
+; GFX9-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
+; GFX9-NEXT:    v_or_b32_e32 v1, 1, v1
+; GFX9-NEXT:    v_mul_f32_e32 v7, v8, v9
+; GFX9-NEXT:    v_trunc_f32_e32 v7, v7
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v9, v7
+; GFX9-NEXT:    v_mad_f32 v7, -v7, v6, v8
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v6|
+; GFX9-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
+; GFX9-NEXT:    v_mul_lo_u32 v5, v5, s9
+; GFX9-NEXT:    v_add_u32_e32 v1, v9, v1
+; GFX9-NEXT:    v_mul_lo_u32 v1, v1, v3
 ; GFX9-NEXT:    v_and_b32_e32 v0, s8, v0
-; GFX9-NEXT:    v_bfe_i32 v6, v0, 0, 15
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v7, v6
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v8, v5
-; GFX9-NEXT:    v_xor_b32_e32 v4, v6, v4
-; GFX9-NEXT:    v_ashrrev_i32_e32 v4, 30, v4
-; GFX9-NEXT:    v_or_b32_e32 v4, 1, v4
-; GFX9-NEXT:    v_mul_f32_e32 v6, v7, v8
-; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v8, v6
-; GFX9-NEXT:    v_mad_f32 v6, -v6, v5, v7
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v6|, |v5|
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, 0, v4, vcc
-; GFX9-NEXT:    v_mul_lo_u32 v3, v3, s11
-; GFX9-NEXT:    v_add_u32_e32 v4, v8, v4
-; GFX9-NEXT:    v_mul_lo_u32 v1, v4, v1
-; GFX9-NEXT:    v_sub_u32_e32 v2, s2, v2
-; GFX9-NEXT:    v_sub_u32_e32 v3, s9, v3
-; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
+; GFX9-NEXT:    v_sub_u32_e32 v3, s2, v4
+; GFX9-NEXT:    v_sub_u32_e32 v4, s3, v5
 ; GFX9-NEXT:    v_sub_u32_e32 v0, v0, v1
+; GFX9-NEXT:    v_and_b32_e32 v4, s8, v4
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 30, v[0:1]
-; GFX9-NEXT:    v_and_b32_e32 v2, s8, v2
-; GFX9-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
-; GFX9-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
-; GFX9-NEXT:    v_or_b32_e32 v0, v2, v0
-; GFX9-NEXT:    global_store_dword v4, v0, s[4:5]
+; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
+; GFX9-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX9-NEXT:    v_or_b32_e32 v0, v3, v0
+; GFX9-NEXT:    global_store_dword v2, v0, s[4:5]
 ; GFX9-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
-; GFX9-NEXT:    global_store_short v4, v0, s[4:5] offset:4
+; GFX9-NEXT:    global_store_short v2, v0, s[4:5] offset:4
 ; GFX9-NEXT:    s_endpgm
   %r = srem <3 x i15> %x, %y
   store <3 x i15> %r, <3 x i15> addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @udiv_i32_oddk_denom(i32 addrspace(1)* %out, i32 %x) {
 ; CHECK-LABEL: @udiv_i32_oddk_denom(

llvm/test/CodeGen/AMDGPU/bswap.ll

 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_alignbit_b32 v2, v1, v1, 8
 ; SI-NEXT:    v_alignbit_b32 v1, v1, v1, 24
 ; SI-NEXT:    s_mov_b32 s4, 0xff00ff
 ; SI-NEXT:    v_alignbit_b32 v3, v0, v0, 8
 ; SI-NEXT:    v_alignbit_b32 v0, v0, v0, 24
 ; SI-NEXT:    v_bfi_b32 v1, s4, v1, v2
 ; SI-NEXT:    v_bfi_b32 v0, s4, v0, v3
-; SI-NEXT:    v_and_b32_e32 v1, 0xffff0000, v1
+; SI-NEXT:    v_and_b32_e32 v2, 0xffff0000, v1
 ; SI-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_bswap_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s4, 0x2030001
 ; VI-NEXT:    v_perm_b32 v0, 0, v0, s4
 ; VI-NEXT:    s_setpc_b64 s[30:31]

 ; SI-NEXT:    v_alignbit_b32 v7, v2, v2, 8
 ; SI-NEXT:    v_alignbit_b32 v2, v2, v2, 24
 ; SI-NEXT:    v_bfi_b32 v1, s4, v1, v4
 ; SI-NEXT:    v_bfi_b32 v0, s4, v0, v5
 ; SI-NEXT:    v_bfi_b32 v3, s4, v3, v6
 ; SI-NEXT:    v_bfi_b32 v2, s4, v2, v7
 ; SI-NEXT:    v_and_b32_e32 v4, s5, v1
 ; SI-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_and_b32_e32 v3, s5, v3
+; SI-NEXT:    v_and_b32_e32 v5, s5, v3
 ; SI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v4
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
+; SI-NEXT:    v_or_b32_e32 v2, v2, v5
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_bswap_v4i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s4, 0x2030001
 ; VI-NEXT:    v_perm_b32 v0, 0, v0, s4
 ; VI-NEXT:    v_perm_b32 v1, 0, v1, s4

llvm/test/CodeGen/AMDGPU/ds-alignment.ll

 }
 
 define amdgpu_kernel void @ds8align1(<2 x i32> addrspace(3)* %in, <2 x i32> addrspace(3)* %out) {
 ; ALIGNED-SDAG-LABEL: ds8align1:
 ; ALIGNED-SDAG:       ; %bb.0:
 ; ALIGNED-SDAG-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; ALIGNED-SDAG-NEXT:    v_mov_b32_e32 v0, s0
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v2, v0
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v3, v0 offset:1
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v4, v0 offset:2
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v5, v0 offset:3
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v6, v0 offset:4
-; ALIGNED-SDAG-NEXT:    ds_read_u8 v7, v0 offset:5
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v1, v0
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v2, v0 offset:1
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v3, v0 offset:2
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v4, v0 offset:3
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v5, v0 offset:4
+; ALIGNED-SDAG-NEXT:    ds_read_u8 v6, v0 offset:5
 ; ALIGNED-SDAG-NEXT:    ds_read_u8 v8, v0 offset:6
 ; ALIGNED-SDAG-NEXT:    ds_read_u8 v0, v0 offset:7
-; ALIGNED-SDAG-NEXT:    v_mov_b32_e32 v1, s1
-; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(5)
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v4 offset:2
-; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(5)
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v5 offset:3
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v2
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v3 offset:1
-; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(5)
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v8 offset:6
-; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(5)
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v0 offset:7
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v6 offset:4
-; ALIGNED-SDAG-NEXT:    ds_write_b8 v1, v7 offset:5
+; ALIGNED-SDAG-NEXT:    v_mov_b32_e32 v7, s1
+; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v5 offset:4
+; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v6 offset:5
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v3 offset:2
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v4 offset:3
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v1
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v2 offset:1
+; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(7)
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v8 offset:6
+; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(7)
+; ALIGNED-SDAG-NEXT:    ds_write_b8 v7, v0 offset:7
 ; ALIGNED-SDAG-NEXT:    s_endpgm
 ;
 ; ALIGNED-GISEL-LABEL: ds8align1:
 ; ALIGNED-GISEL:       ; %bb.0:
 ; ALIGNED-GISEL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; ALIGNED-GISEL-NEXT:    s_mov_b32 s2, 8
 ; ALIGNED-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; ALIGNED-GISEL-NEXT:    v_mov_b32_e32 v0, s0

 }
 
 define amdgpu_kernel void @ds8align2(<2 x i32> addrspace(3)* %in, <2 x i32> addrspace(3)* %out) {
 ; ALIGNED-SDAG-LABEL: ds8align2:
 ; ALIGNED-SDAG:       ; %bb.0:
 ; ALIGNED-SDAG-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; ALIGNED-SDAG-NEXT:    v_mov_b32_e32 v0, s0
-; ALIGNED-SDAG-NEXT:    ds_read_u16 v1, v0 offset:2
-; ALIGNED-SDAG-NEXT:    ds_read_u16 v2, v0
-; ALIGNED-SDAG-NEXT:    ds_read_u16 v3, v0 offset:6
-; ALIGNED-SDAG-NEXT:    ds_read_u16 v0, v0 offset:4
+; ALIGNED-SDAG-NEXT:    ds_read_u16 v1, v0 offset:4
+; ALIGNED-SDAG-NEXT:    ds_read_u16 v2, v0 offset:2
+; ALIGNED-SDAG-NEXT:    ds_read_u16 v3, v0
+; ALIGNED-SDAG-NEXT:    ds_read_u16 v0, v0 offset:6
 ; ALIGNED-SDAG-NEXT:    v_mov_b32_e32 v4, s1
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
-; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v1 offset:2
+; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v1 offset:4
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
-; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v2
+; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v2 offset:2
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
-; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v3 offset:6
+; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v3
 ; ALIGNED-SDAG-NEXT:    s_waitcnt lgkmcnt(3)
-; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v0 offset:4
+; ALIGNED-SDAG-NEXT:    ds_write_b16 v4, v0 offset:6
 ; ALIGNED-SDAG-NEXT:    s_endpgm
 ;
 ; ALIGNED-GISEL-LABEL: ds8align2:
 ; ALIGNED-GISEL:       ; %bb.0:
 ; ALIGNED-GISEL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; ALIGNED-GISEL-NEXT:    s_waitcnt lgkmcnt(0)
 ; ALIGNED-GISEL-NEXT:    v_mov_b32_e32 v0, s0
 ; ALIGNED-GISEL-NEXT:    ds_read_u16 v1, v0

llvm/test/CodeGen/AMDGPU/extract-subvector-16bit.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mtriple=amdgcn-- -verify-machineinstrs -o - %s | FileCheck -check-prefix=SI %s
 ; RUN: llc -march=amdgcn -mtriple=amdgcn-- -mcpu=gfx900 -verify-machineinstrs -o - %s | FileCheck -check-prefix=GFX9 %s
 
 define <4 x i16> @extract_4xi16(<8 x i16> addrspace(1) * %p0, <8 x i16> addrspace(1) * %p1) {
 ; SI-LABEL: extract_4xi16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    s_cbranch_scc0 .LBB0_2
 ; SI-NEXT:  ; %bb.1: ; %F
 ; SI-NEXT:    s_mov_b32 s6, 0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
-; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 glc
+; SI-NEXT:    buffer_load_ushort v5, v[2:3], s[4:7], 0 addr64 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v5, v[2:3], s[4:7], 0 addr64 offset:2 glc
+; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:2 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v6, v[2:3], s[4:7], 0 addr64 offset:4 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v7, v[2:3], s[4:7], 0 addr64 offset:6 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v8, v[2:3], s[4:7], 0 addr64 offset:8 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v8, v[2:3], s[4:7], 0 addr64 offset:10 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v8, v[2:3], s[4:7], 0 addr64 offset:12 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[2:3], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v7
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v5
+; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v4
 ; SI-NEXT:    v_or_b32_e32 v2, v6, v2
-; SI-NEXT:    v_or_b32_e32 v3, v4, v3
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v3
+; SI-NEXT:    v_or_b32_e32 v3, v5, v3
 ; SI-NEXT:    s_mov_b64 vcc, exec
 ; SI-NEXT:    s_cbranch_execz .LBB0_3
 ; SI-NEXT:    s_branch .LBB0_4
 ; SI-NEXT:  .LBB0_2:
 ; SI-NEXT:    ; implicit-def: $vgpr3
 ; SI-NEXT:    ; implicit-def: $vgpr4
 ; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:    s_mov_b64 vcc, 0
 ; SI-NEXT:  .LBB0_3: ; %T
 ; SI-NEXT:    s_mov_b32 s6, 0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
 ; SI-NEXT:    buffer_load_ushort v3, v[0:1], s[4:7], 0 addr64 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:2 glc
+; SI-NEXT:    buffer_load_ushort v4, v[0:1], s[4:7], 0 addr64 offset:2 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v4, v[0:1], s[4:7], 0 addr64 offset:4 glc
+; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:4 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v5, v[0:1], s[4:7], 0 addr64 offset:6 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v6, v[0:1], s[4:7], 0 addr64 offset:8 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v6, v[0:1], s[4:7], 0 addr64 offset:10 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v6, v[0:1], s[4:7], 0 addr64 offset:12 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v0, v[0:1], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v5
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
-; SI-NEXT:    v_or_b32_e32 v2, v4, v0
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v4
+; SI-NEXT:    v_or_b32_e32 v2, v2, v0
 ; SI-NEXT:    v_or_b32_e32 v3, v3, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v3
 ; SI-NEXT:  .LBB0_4: ; %exit
 ; SI-NEXT:    v_bfe_i32 v0, v3, 0, 16
 ; SI-NEXT:    v_bfe_i32 v1, v4, 0, 16
 ; SI-NEXT:    v_bfe_i32 v2, v2, 0, 16
 ; SI-NEXT:    s_mov_b32 s4, 0xffff
 ; SI-NEXT:    v_mov_b32_e32 v3, 0x8000
 ; SI-NEXT:    v_mov_b32_e32 v4, 0xffff0000
 ; SI-NEXT:    v_bfrev_b32_e32 v5, 1
 ; SI-NEXT:    v_mov_b32_e32 v6, 0xffff8000
 ; SI-NEXT:    v_mov_b32_e32 v7, s4
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v7, v3, vcc
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v1
 ; SI-NEXT:    v_cndmask_b32_e32 v1, v4, v5, vcc
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v2
 ; SI-NEXT:    v_cndmask_b32_e32 v2, -1, v6, vcc
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v2
-; SI-NEXT:    v_and_b32_e32 v2, s4, v2
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
+; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v2
+; SI-NEXT:    v_and_b32_e32 v3, s4, v2
+; SI-NEXT:    v_or_b32_e32 v2, v3, v4
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: extract_4xi16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    s_cbranch_scc0 .LBB0_2
 ; GFX9-NEXT:  ; %bb.1: ; %F
 ; GFX9-NEXT:    global_load_dwordx4 v[2:5], v[2:3], off glc

 ; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:2 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:4 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:6 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:8 glc
+; SI-NEXT:    buffer_load_ushort v5, v[2:3], s[4:7], 0 addr64 offset:8 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v5, v[2:3], s[4:7], 0 addr64 offset:10 glc
+; SI-NEXT:    buffer_load_ushort v4, v[2:3], s[4:7], 0 addr64 offset:10 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v6, v[2:3], s[4:7], 0 addr64 offset:12 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[2:3], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v5
-; SI-NEXT:    v_or_b32_e32 v2, v6, v2
-; SI-NEXT:    v_or_b32_e32 v3, v4, v3
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
-; SI-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_lshlrev_b32_e32 v7, 16, v4
+; SI-NEXT:    v_or_b32_e32 v3, v6, v3
+; SI-NEXT:    v_or_b32_e32 v5, v5, v7
 ; SI-NEXT:    s_mov_b64 vcc, exec
 ; SI-NEXT:    s_cbranch_execz .LBB1_3
 ; SI-NEXT:    s_branch .LBB1_4
 ; SI-NEXT:  .LBB1_2:
-; SI-NEXT:    ; implicit-def: $vgpr3
 ; SI-NEXT:    ; implicit-def: $vgpr5
-; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:    ; implicit-def: $vgpr4
+; SI-NEXT:    ; implicit-def: $vgpr3
+; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:    s_mov_b64 vcc, 0
 ; SI-NEXT:  .LBB1_3: ; %T
 ; SI-NEXT:    s_mov_b32 s6, 0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
 ; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:2 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:4 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:6 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v3, v[0:1], s[4:7], 0 addr64 offset:8 glc
+; SI-NEXT:    buffer_load_ushort v5, v[0:1], s[4:7], 0 addr64 offset:8 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:10 glc
+; SI-NEXT:    buffer_load_ushort v4, v[0:1], s[4:7], 0 addr64 offset:10 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v4, v[0:1], s[4:7], 0 addr64 offset:12 glc
+; SI-NEXT:    buffer_load_ushort v3, v[0:1], s[4:7], 0 addr64 offset:12 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    buffer_load_ushort v0, v[0:1], s[4:7], 0 addr64 offset:14 glc
+; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
-; SI-NEXT:    v_or_b32_e32 v2, v4, v0
-; SI-NEXT:    v_or_b32_e32 v3, v3, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
-; SI-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v2
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v4
+; SI-NEXT:    v_or_b32_e32 v3, v3, v0
+; SI-NEXT:    v_or_b32_e32 v5, v5, v1
 ; SI-NEXT:  .LBB1_4: ; %exit
-; SI-NEXT:    v_bfe_i32 v0, v3, 0, 16
-; SI-NEXT:    v_bfe_i32 v1, v5, 0, 16
+; SI-NEXT:    v_bfe_i32 v0, v5, 0, 16
+; SI-NEXT:    v_bfe_i32 v1, v4, 0, 16
+; SI-NEXT:    v_bfe_i32 v3, v3, 0, 16
 ; SI-NEXT:    v_bfe_i32 v2, v2, 0, 16
-; SI-NEXT:    v_bfe_i32 v3, v4, 0, 16
 ; SI-NEXT:    v_mov_b32_e32 v4, 0xffff
 ; SI-NEXT:    v_mov_b32_e32 v5, 0x8000
 ; SI-NEXT:    v_mov_b32_e32 v6, 0xffff0000
 ; SI-NEXT:    v_bfrev_b32_e32 v7, 1
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v4, v5, vcc
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v1
 ; SI-NEXT:    v_cndmask_b32_e32 v1, v6, v7, vcc
-; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v2
-; SI-NEXT:    v_cndmask_b32_e32 v2, v4, v5, vcc
 ; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v3
-; SI-NEXT:    v_cndmask_b32_e32 v3, v6, v7, vcc
+; SI-NEXT:    v_cndmask_b32_e32 v3, v4, v5, vcc
+; SI-NEXT:    v_cmp_lt_i32_e32 vcc, -1, v2
+; SI-NEXT:    v_cndmask_b32_e32 v4, v6, v7, vcc
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
+; SI-NEXT:    v_or_b32_e32 v2, v3, v4
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v4
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: extract_4xi16_2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    s_cbranch_scc0 .LBB1_2
 ; GFX9-NEXT:  ; %bb.1: ; %F
 ; GFX9-NEXT:    global_load_dwordx4 v[2:5], v[2:3], off glc

 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v8, v[2:3], s[4:7], 0 addr64 offset:10 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v8, v[2:3], s[4:7], 0 addr64 offset:12 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v2, v[2:3], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v7
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v5
+; SI-NEXT:    v_lshlrev_b32_e32 v7, 16, v5
+; SI-NEXT:    v_cvt_f32_f16_e32 v3, v5
 ; SI-NEXT:    v_or_b32_e32 v2, v6, v2
-; SI-NEXT:    v_or_b32_e32 v4, v4, v3
+; SI-NEXT:    v_or_b32_e32 v4, v4, v7
 ; SI-NEXT:    v_cvt_f32_f16_e32 v2, v2
-; SI-NEXT:    v_cvt_f32_f16_e32 v3, v4
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_cvt_f32_f16_e32 v4, v4
 ; SI-NEXT:    s_mov_b64 vcc, exec
 ; SI-NEXT:    s_cbranch_execz .LBB2_3
 ; SI-NEXT:    s_branch .LBB2_4
 ; SI-NEXT:  .LBB2_2:
-; SI-NEXT:    ; implicit-def: $vgpr3
 ; SI-NEXT:    ; implicit-def: $vgpr4
+; SI-NEXT:    ; implicit-def: $vgpr3
 ; SI-NEXT:    ; implicit-def: $vgpr2
 ; SI-NEXT:    s_mov_b64 vcc, 0
 ; SI-NEXT:  .LBB2_3: ; %T
 ; SI-NEXT:    s_mov_b32 s6, 0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
 ; SI-NEXT:    buffer_load_ushort v2, v[0:1], s[4:7], 0 addr64 glc

 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v0, v[0:1], s[4:7], 0 addr64 offset:14 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v5
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v3
 ; SI-NEXT:    v_or_b32_e32 v0, v4, v0
 ; SI-NEXT:    v_or_b32_e32 v1, v2, v1
 ; SI-NEXT:    v_cvt_f32_f16_e32 v2, v0
-; SI-NEXT:    v_cvt_f32_f16_e32 v3, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v0, 16, v1
-; SI-NEXT:    v_cvt_f32_f16_e32 v4, v0
+; SI-NEXT:    v_cvt_f32_f16_e32 v4, v1
+; SI-NEXT:    v_cvt_f32_f16_e32 v3, v3
 ; SI-NEXT:  .LBB2_4: ; %exit
-; SI-NEXT:    v_cvt_f16_f32_e32 v0, v3
-; SI-NEXT:    v_cvt_f16_f32_e32 v1, v4
+; SI-NEXT:    v_cvt_f16_f32_e32 v0, v4
+; SI-NEXT:    v_cvt_f16_f32_e32 v1, v3
 ; SI-NEXT:    v_cvt_f16_f32_e32 v2, v2
 ; SI-NEXT:    v_mov_b32_e32 v3, 0x3fa00000
 ; SI-NEXT:    v_mov_b32_e32 v4, 0x3f200000
 ; SI-NEXT:    v_cvt_f32_f16_e32 v0, v0
 ; SI-NEXT:    v_cvt_f32_f16_e32 v1, v1
 ; SI-NEXT:    v_cvt_f32_f16_e32 v2, v2
 ; SI-NEXT:    v_cmp_nge_f32_e32 vcc, 0.5, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc

llvm/test/CodeGen/AMDGPU/fshr.ll

 }
 
 define <2 x i16> @v_fshr_v2i16(<2 x i16> %src0, <2 x i16> %src1, <2 x i16> %src2) {
 ; SI-LABEL: v_fshr_v2i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_or_b32_e32 v5, 16, v5
 ; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; SI-NEXT:    v_alignbit_b32 v1, v1, v3, v5
-; SI-NEXT:    v_or_b32_e32 v3, 16, v4
+; SI-NEXT:    v_or_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; SI-NEXT:    v_alignbit_b32 v0, v0, v2, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_alignbit_b32 v1, v1, v3, v5
+; SI-NEXT:    v_alignbit_b32 v0, v0, v2, v4
+; SI-NEXT:    s_mov_b32 s4, 0xffff
+; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; SI-NEXT:    v_and_b32_e32 v0, s4, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v3
+; SI-NEXT:    v_and_b32_e32 v1, s4, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_fshr_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
 ; VI-NEXT:    v_mov_b32_e32 v5, 1
 ; VI-NEXT:    v_lshrrev_b16_sdwa v4, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1

 ; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
 ; SI-NEXT:    v_alignbit_b32 v1, v1, v5, v9
 ; SI-NEXT:    v_or_b32_e32 v5, 16, v8
 ; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_alignbit_b32 v0, v0, v4, v5
 ; SI-NEXT:    v_or_b32_e32 v4, 16, v11
 ; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v7
 ; SI-NEXT:    v_alignbit_b32 v3, v3, v5, v4
-; SI-NEXT:    v_or_b32_e32 v4, 16, v10
-; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v6
+; SI-NEXT:    v_or_b32_e32 v5, 16, v10
+; SI-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
 ; SI-NEXT:    s_mov_b32 s4, 0xffff
-; SI-NEXT:    v_alignbit_b32 v2, v2, v5, v4
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; SI-NEXT:    v_alignbit_b32 v2, v2, v6, v5
+; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v3
 ; SI-NEXT:    v_and_b32_e32 v2, s4, v2
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    v_and_b32_e32 v0, s4, v0
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
+; SI-NEXT:    v_or_b32_e32 v2, v2, v4
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_and_b32_e32 v3, s4, v3
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_fshr_v4i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshrrev_b32_e32 v6, 16, v5
 ; VI-NEXT:    v_mov_b32_e32 v8, 1
 ; VI-NEXT:    v_lshrrev_b16_sdwa v7, v6, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1

llvm/test/CodeGen/AMDGPU/idot4s.ll

 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_i32 v3, v2, 8, 8
+; GFX7-NEXT:    v_bfe_i32 v3, v2, 16, 8
 ; GFX7-NEXT:    v_bfe_i32 v4, v2, 0, 8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v5, 24, v2
+; GFX7-NEXT:    v_bfe_i32 v2, v2, 8, 8
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v6, v0, 8, 8
+; GFX7-NEXT:    v_bfe_i32 v6, v0, 16, 8
 ; GFX7-NEXT:    v_bfe_i32 v7, v0, 0, 8
-; GFX7-NEXT:    v_or_b32_e32 v3, v4, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v6
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v7
-; GFX7-NEXT:    v_bfe_i32 v8, v0, 16, 8
-; GFX7-NEXT:    v_or_b32_e32 v4, v6, v4
-; GFX7-NEXT:    v_and_b32_e32 v7, s4, v8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v8, 24, v0
+; GFX7-NEXT:    v_bfe_i32 v0, v0, 8, 8
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_bfe_i32 v5, v2, 16, 8
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
+; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
+; GFX7-NEXT:    v_alignbit_b32 v2, 0, v2, 16
+; GFX7-NEXT:    v_alignbit_b32 v0, 0, v0, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v4, v1
-; GFX7-NEXT:    v_ashrrev_i32_e32 v2, 24, v2
-; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
-; GFX7-NEXT:    v_ashrrev_i32_e32 v0, 24, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v8, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
-; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v7, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v7, v1
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
+; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v6, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v8, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

llvm/test/CodeGen/AMDGPU/idot4u.ll

 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b32 s4, 0xff00
 ; GFX7-NEXT:    s_movk_i32 s5, 0xff
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
 ; GFX7-NEXT:    v_and_b32_e32 v3, s4, v2
-; GFX7-NEXT:    v_and_b32_e32 v4, s5, v2
+; GFX7-NEXT:    v_bfe_u32 v4, v2, 16, 8
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
 ; GFX7-NEXT:    v_and_b32_e32 v6, s4, v0
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
+; GFX7-NEXT:    v_and_b32_e32 v2, s5, v2
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
-; GFX7-NEXT:    v_and_b32_e32 v7, s5, v0
-; GFX7-NEXT:    v_or_b32_e32 v3, v4, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 8, v6
-; GFX7-NEXT:    v_or_b32_e32 v4, v7, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v3, s5, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v4
-; GFX7-NEXT:    v_and_b32_e32 v4, s5, v4
+; GFX7-NEXT:    v_bfe_u32 v7, v0, 16, 8
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 24, v0
+; GFX7-NEXT:    v_and_b32_e32 v0, s5, v0
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
+; GFX7-NEXT:    v_alignbit_b32 v3, s10, v3, 16
+; GFX7-NEXT:    v_alignbit_b32 v6, 0, v6, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v4, v1
-; GFX7-NEXT:    v_bfe_u32 v5, v2, 16, 8
-; GFX7-NEXT:    v_bfe_u32 v8, v0, 16, 8
-; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v7, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 24, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, 24, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v8, v1
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v6, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v7, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v8, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX9-NODL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NODL-NEXT:    global_load_ubyte v3, v0, s[2:3]
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(2)
 ; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v4, 16, v1
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v5, 16, v2
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v6, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
-; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v8, v4, v5
-; GFX9-NODL-NEXT:    v_or_b32_sdwa v6, v8, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v7, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-NODL-NEXT:    v_lshlrev_b32_e32 v8, 16, v6
-; GFX9-NODL-NEXT:    v_or_b32_e32 v7, v7, v8
-; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
+; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v7, v4, v5
+; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v8, 8, v6
+; GFX9-NODL-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v6, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PRESERVE src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_legacy_u16 v1, v1, v2, v3
-; GFX9-NODL-NEXT:    v_add_u16_e32 v1, v1, v7
-; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
-; GFX9-NODL-NEXT:    v_mad_legacy_u16 v1, v4, v5, v1
 ; GFX9-NODL-NEXT:    v_add_u16_e32 v1, v1, v6
+; GFX9-NODL-NEXT:    v_mad_legacy_u16 v1, v4, v5, v1
+; GFX9-NODL-NEXT:    v_add_u16_e32 v1, v1, v8
 ; GFX9-NODL-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-DL-NEXT:    global_load_ubyte v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v4, 16, v1
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 16, v2
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:BYTE_3
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v8, v4, v5
-; GFX9-DL-NEXT:    v_or_b32_sdwa v6, v8, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v7, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v8, 16, v6
-; GFX9-DL-NEXT:    v_or_b32_e32 v7, v7, v8
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v4, v5
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v6
+; GFX9-DL-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PRESERVE src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_legacy_u16 v1, v1, v2, v3
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v7
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v1, v4, v5, v1
 ; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v6
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v1, v4, v5, v1
+; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v8
 ; GFX9-DL-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot4_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    global_load_ubyte v3, v0, s[0:1]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 24, v1
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 16, v2
-; GFX10-DL-NEXT:    v_lshrrev_b16 v8, 8, v2
+; GFX10-DL-NEXT:    v_lshrrev_b16 v6, 8, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 16, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 16, v2
+; GFX10-DL-NEXT:    v_lshrrev_b16 v9, 8, v2
 ; GFX10-DL-NEXT:    v_mul_lo_u16 v4, v4, v5
-; GFX10-DL-NEXT:    v_lshrrev_b16 v5, 8, v1
-; GFX10-DL-NEXT:    v_mul_lo_u16 v9, v6, v7
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-NEXT:    v_mad_u16 v1, v1, v2, v3
+; GFX10-DL-NEXT:    v_mul_lo_u16 v5, v7, v8
+; GFX10-DL-NEXT:    v_mul_lo_u16 v6, v6, v9
 ; GFX10-DL-NEXT:    v_lshlrev_b16 v4, 8, v4
-; GFX10-DL-NEXT:    v_mul_lo_u16 v5, v5, v8
-; GFX10-DL-NEXT:    v_or_b32_sdwa v4, v9, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_lshlrev_b16 v5, 8, v5
-; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v8, 16, v4
+; GFX10-DL-NEXT:    v_lshlrev_b16 v6, 8, v6
+; GFX10-DL-NEXT:    v_or_b32_sdwa v5, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v4
-; GFX10-DL-NEXT:    v_or_b32_sdwa v5, v5, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_or_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v5
 ; GFX10-DL-NEXT:    v_add_nc_u16 v1, v1, v5
-; GFX10-DL-NEXT:    v_mad_u16 v1, v6, v7, v1
+; GFX10-DL-NEXT:    v_mad_u16 v1, v7, v8, v1
 ; GFX10-DL-NEXT:    v_add_nc_u16 v1, v1, v2
 ; GFX10-DL-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <4 x i8> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <4 x i8>, <4 x i8> addrspace(1)* %src1, i32 %idx

llvm/test/CodeGen/AMDGPU/idot8s.ll

 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_i32 v3, v2, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v4, v2, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v5, v2, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v6, v2, 0, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
+; GFX7-NEXT:    v_bfe_i32 v8, v2, 0, 4
+; GFX7-NEXT:    v_bfe_i32 v6, v2, 4, 4
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v10, v0, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v11, v0, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v12, v0, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v13, v0, 0, 4
-; GFX7-NEXT:    v_or_b32_e32 v3, v4, v3
-; GFX7-NEXT:    v_or_b32_e32 v4, v6, v5
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v10
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v11
-; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 16, v12
-; GFX7-NEXT:    v_and_b32_e32 v11, s4, v13
-; GFX7-NEXT:    v_bfe_i32 v14, v0, 24, 4
-; GFX7-NEXT:    v_ashrrev_i32_e32 v16, 28, v0
-; GFX7-NEXT:    v_or_b32_e32 v5, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v6, v11, v10
-; GFX7-NEXT:    v_and_b32_e32 v12, s4, v14
-; GFX7-NEXT:    v_and_b32_e32 v14, s4, v16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v16, 16, v4
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 16, v6
+; GFX7-NEXT:    v_bfe_i32 v15, v0, 0, 4
+; GFX7-NEXT:    v_bfe_i32 v13, v0, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
+; GFX7-NEXT:    v_and_b32_e32 v15, s4, v15
+; GFX7-NEXT:    v_bfe_i32 v5, v2, 8, 4
 ; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
-; GFX7-NEXT:    v_bfe_i32 v8, v2, 8, 4
-; GFX7-NEXT:    v_bfe_i32 v15, v0, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v12, v0, 8, 4
+; GFX7-NEXT:    v_and_b32_e32 v13, s4, v13
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v6, v1
-; GFX7-NEXT:    v_bfe_i32 v7, v2, 24, 4
+; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v15, v1
+; GFX7-NEXT:    v_bfe_i32 v3, v2, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v4, v2, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v7, v2, 16, 4
 ; GFX7-NEXT:    v_ashrrev_i32_e32 v9, 28, v2
 ; GFX7-NEXT:    v_bfe_i32 v2, v2, 12, 4
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_bfe_i32 v10, v0, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v11, v0, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v14, v0, 16, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v16, 28, v0
 ; GFX7-NEXT:    v_bfe_i32 v0, v0, 12, 4
-; GFX7-NEXT:    v_and_b32_e32 v13, s4, v15
-; GFX7-NEXT:    v_mad_u32_u24 v1, v16, v11, v1
+; GFX7-NEXT:    v_and_b32_e32 v12, s4, v12
+; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v13, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v13, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
-; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v5, v0
+; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v12, v1
 ; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
-; GFX7-NEXT:    v_mad_u32_u24 v0, v15, v10, v0
+; GFX7-NEXT:    v_and_b32_e32 v14, s4, v14
+; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
+; GFX7-NEXT:    v_and_b32_e32 v11, s4, v11
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v14, v0
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
+; GFX7-NEXT:    v_and_b32_e32 v10, s4, v10
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
 ; GFX7-NEXT:    v_and_b32_e32 v9, s4, v9
-; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v12, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v14, v0
+; GFX7-NEXT:    v_and_b32_e32 v16, s4, v16
+; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ubyte v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_movk_i32 s4, 0xff
-; GFX7-NEXT:    s_mov_b32 s5, 0xffff
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_ashrrev_i32_e32 v3, 28, v2
-; GFX7-NEXT:    v_bfe_i32 v4, v2, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v5, v2, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v6, v2, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v7, v2, 12, 4
-; GFX7-NEXT:    v_bfe_i32 v8, v2, 8, 4
-; GFX7-NEXT:    v_bfe_i32 v9, v2, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v2, v2, 0, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 8, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 8, v7
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 8, v9
-; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
+; GFX7-NEXT:    v_bfe_i32 v7, v2, 0, 4
+; GFX7-NEXT:    v_bfe_i32 v3, v2, 24, 4
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_ashrrev_i32_e32 v11, 28, v0
-; GFX7-NEXT:    v_bfe_i32 v12, v0, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v13, v0, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v14, v0, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v15, v0, 12, 4
-; GFX7-NEXT:    v_bfe_i32 v16, v0, 8, 4
-; GFX7-NEXT:    v_bfe_i32 v17, v0, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v0, v0, 0, 4
-; GFX7-NEXT:    v_or_b32_e32 v4, v4, v10
-; GFX7-NEXT:    v_or_b32_e32 v5, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v6, v8, v7
-; GFX7-NEXT:    v_or_b32_e32 v2, v2, v9
-; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 8, v11
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v12
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 8, v13
-; GFX7-NEXT:    v_and_b32_e32 v10, s4, v14
-; GFX7-NEXT:    v_lshlrev_b32_e32 v12, 8, v15
-; GFX7-NEXT:    v_and_b32_e32 v13, s4, v16
-; GFX7-NEXT:    v_lshlrev_b32_e32 v14, 8, v17
+; GFX7-NEXT:    v_bfe_i32 v14, v0, 0, 4
+; GFX7-NEXT:    v_bfe_i32 v4, v2, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v5, v2, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v6, v2, 8, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v8, 28, v2
+; GFX7-NEXT:    v_bfe_i32 v9, v2, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v2, v2, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
+; GFX7-NEXT:    v_bfe_i32 v10, v0, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v11, v0, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v12, v0, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v13, v0, 8, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v15, 28, v0
+; GFX7-NEXT:    v_bfe_i32 v16, v0, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v0, v0, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v14, s4, v14
+; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
-; GFX7-NEXT:    v_and_b32_e32 v5, s5, v5
-; GFX7-NEXT:    v_or_b32_e32 v7, v8, v7
-; GFX7-NEXT:    v_or_b32_e32 v8, v10, v9
-; GFX7-NEXT:    v_or_b32_e32 v9, v13, v12
-; GFX7-NEXT:    v_or_b32_e32 v0, v0, v14
-; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
-; GFX7-NEXT:    v_and_b32_e32 v2, s5, v2
-; GFX7-NEXT:    v_or_b32_e32 v4, v5, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v7
-; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 16, v9
-; GFX7-NEXT:    v_and_b32_e32 v0, s5, v0
-; GFX7-NEXT:    v_or_b32_e32 v2, v2, v6
-; GFX7-NEXT:    v_or_b32_e32 v0, v0, v7
-; GFX7-NEXT:    v_and_b32_e32 v7, s4, v2
-; GFX7-NEXT:    v_and_b32_e32 v13, s4, v0
-; GFX7-NEXT:    v_and_b32_e32 v6, s5, v8
-; GFX7-NEXT:    v_bfe_u32 v8, v2, 8, 8
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 8, 8
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v13, v1
-; GFX7-NEXT:    v_or_b32_e32 v5, v6, v5
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
-; GFX7-NEXT:    v_bfe_u32 v2, v2, 16, 8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v12, 24, v0
-; GFX7-NEXT:    v_bfe_u32 v0, v0, 16, 8
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v14, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v14, v1
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 24, v9
+; GFX7-NEXT:    v_and_b32_e32 v13, s4, v13
+; GFX7-NEXT:    v_lshlrev_b32_e32 v16, 24, v16
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_and_b32_e32 v9, s4, v4
-; GFX7-NEXT:    v_and_b32_e32 v15, s4, v5
-; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v12, v0
-; GFX7-NEXT:    v_bfe_u32 v10, v4, 8, 8
-; GFX7-NEXT:    v_bfe_u32 v16, v5, 8, 8
-; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v15, v0
-; GFX7-NEXT:    v_bfe_u32 v4, v4, 16, 8
-; GFX7-NEXT:    v_bfe_u32 v5, v5, 16, 8
-; GFX7-NEXT:    v_mad_u32_u24 v0, v10, v16, v0
-; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
+; GFX7-NEXT:    v_alignbit_b32 v9, 0, v9, 24
+; GFX7-NEXT:    v_alignbit_b32 v16, 0, v16, 24
+; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v13, v0
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_and_b32_e32 v12, s4, v12
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
+; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
 ; GFX7-NEXT:    v_and_b32_e32 v11, s4, v11
-; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v5, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v11, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
+; GFX7-NEXT:    v_and_b32_e32 v10, s4, v10
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
+; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
+; GFX7-NEXT:    v_and_b32_e32 v15, s4, v15
+; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 20, v1
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 12, v1
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 4, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 20, v2
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v15, 12, v1
 ; GFX9-NEXT:    v_lshlrev_b16_sdwa v16, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    v_lshlrev_b16_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v17, 12, v2
 ; GFX9-NEXT:    v_lshlrev_b16_sdwa v18, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    v_lshlrev_b16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 12, v2
-; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v2, 12, v9
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v15
+; GFX9-NEXT:    v_lshlrev_b16_e32 v8, 12, v8
+; GFX9-NEXT:    v_lshlrev_b16_e32 v7, 12, v7
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v15, 12, v16
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v6, 12, v6
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v16, 12, v1
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v1, 12, v5
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v5, 12, v14
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v14, 12, v17
+; GFX9-NEXT:    v_lshlrev_b16_e32 v13, 12, v13
+; GFX9-NEXT:    v_lshlrev_b16_e32 v12, 12, v12
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v17, 12, v18
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v11, 12, v11
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v18, 12, v0
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v0, 12, v10
-; GFX9-NEXT:    v_lshlrev_b16_e32 v8, 12, v8
-; GFX9-NEXT:    v_lshlrev_b16_e32 v7, 12, v7
-; GFX9-NEXT:    v_lshlrev_b16_e32 v13, 12, v13
-; GFX9-NEXT:    v_lshlrev_b16_e32 v12, 12, v12
-; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-NEXT:    v_ashrrev_i16_e32 v1, 12, v1
-; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-NEXT:    v_ashrrev_i16_e32 v0, 12, v0
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-NEXT:    v_ashrrev_i16_e32 v1, 12, v1
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v13
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-NEXT:    v_mul_lo_u16_e32 v19, v15, v17
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v0, v1, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v1, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
+; GFX9-NEXT:    v_ashrrev_i16_e32 v0, 12, v0
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v2, 12, v2
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v13, v16, v18
+; GFX9-NEXT:    v_mul_lo_u16_e32 v19, v15, v17
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v0, v1, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v1, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, v7, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v7, v8, v10
-; GFX9-NEXT:    v_or_b32_sdwa v1, v19, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v9, v9, v14
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v2, v2, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v5, v13, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v0, v13, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v5, v19, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshlrev_b32_e32 v8, 16, v1
-; GFX9-NEXT:    v_or_b32_sdwa v7, v9, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v9, v9, v14
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-NEXT:    v_or_b32_sdwa v1, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v6
-; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 8, v1
-; GFX9-NEXT:    v_or_b32_sdwa v1, v5, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v2, v2, v0
+; GFX9-NEXT:    v_or_b32_sdwa v7, v9, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 8, v1
+; GFX9-NEXT:    v_or_b32_e32 v2, v2, v0
 ; GFX9-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 8, v2
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u16_e32 v1, v7, v4
-; GFX9-NEXT:    v_add_u16_e32 v1, v1, v2
+; GFX9-NEXT:    v_add_u16_e32 v2, v7, v4
+; GFX9-NEXT:    v_add_u16_e32 v1, v2, v1
 ; GFX9-NEXT:    v_add_u16_e32 v1, v1, v6
 ; GFX9-NEXT:    v_add_u16_e32 v0, v1, v0
 ; GFX9-NEXT:    v_mad_legacy_u16 v0, v16, v18, v0
 ; GFX9-NEXT:    v_add_u16_e32 v0, v0, v5
 ; GFX9-NEXT:    v_mad_legacy_u16 v0, v15, v17, v0
-; GFX9-NEXT:    v_add_u16_e32 v0, v0, v9
+; GFX9-NEXT:    v_add_u16_e32 v0, v0, v8
 ; GFX9-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1

 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 20, v1
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v1
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 4, v1
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 20, v2
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v15, 12, v1
 ; GFX9-DL-NEXT:    v_lshlrev_b16_sdwa v16, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    v_lshlrev_b16_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v17, 12, v2
 ; GFX9-DL-NEXT:    v_lshlrev_b16_sdwa v18, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    v_lshlrev_b16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 12, v2
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v2, 12, v9
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v15
+; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v8, 12, v8
+; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v7, 12, v7
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v15, 12, v16
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v6, 12, v6
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v16, 12, v1
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v1, 12, v5
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v5, 12, v14
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v14, 12, v17
+; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v13, 12, v13
+; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v12, 12, v12
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v17, 12, v18
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v11, 12, v11
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v18, 12, v0
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v0, 12, v10
-; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v8, 12, v8
-; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v7, 12, v7
-; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v13, 12, v13
-; GFX9-DL-NEXT:    v_lshlrev_b16_e32 v12, 12, v12
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v1, 12, v1
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v0, 12, v0
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v1, 12, v1
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v13
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v19, v15, v17
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v0, v1, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v1, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v0, 12, v0
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v2, 12, v2
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v13, v16, v18
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v19, v15, v17
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v0, v1, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v1, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v7, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v8, v10
-; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v19, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v9, v9, v14
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v2, v2, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v13, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_or_b32_sdwa v0, v13, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v19, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v8, 16, v1
-; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v9, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v9, v9, v14
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v6
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v1
-; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v5, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v2, v2, v0
+; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v9, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v1
+; GFX9-DL-NEXT:    v_or_b32_e32 v2, v2, v0
 ; GFX9-DL-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v7, v4
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v2
+; GFX9-DL-NEXT:    v_add_u16_e32 v2, v7, v4
+; GFX9-DL-NEXT:    v_add_u16_e32 v1, v2, v1
 ; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v6
 ; GFX9-DL-NEXT:    v_add_u16_e32 v0, v1, v0
 ; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v16, v18, v0
 ; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v5
 ; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v15, v17, v0
-; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v9
+; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v8
 ; GFX9-DL-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-XNACK-LABEL: idot8_acc8_vecMul:
 ; GFX10-DL-XNACK:       ; %bb.0: ; %entry
 ; GFX10-DL-XNACK-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-XNACK-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34

 ; GFX10-DL-XNACK-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX10-DL-XNACK-NEXT:    global_load_ubyte v3, v4, s[0:1]
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v8, 12, v1
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(1)
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v15, 12, v2
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v9, 8, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v16, 8, v2
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v8, 12, v8
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v15, 12, v15
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v13, 28, v2
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v17, 4, v2
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v9, 12, v9
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v16, 12, v16
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v8, 12, v8
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v15, 12, v15
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v0, 20, v1
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v11, 20, v2
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v14, 24, v2
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v6, 12, v6
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v13, 12, v13
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v13, 28, v2
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v10, 12, v10
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v17, 12, v17
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v9, 12, v9
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v16, 12, v16
 ; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v8, v8, v15
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v12, 16, v2
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v7, 12, v7
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v0, 12, v0
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v14, 12, v14
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v11, 12, v11
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v6, 12, v6
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v13, 12, v13
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v14, 24, v2
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v1, 12, v1
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v2, 12, v2
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v6, 12, v6
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v0, 12, v0
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v13, 12, v13
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v11, 12, v11
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v10, 12, v10
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v15, 12, v17
 ; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v9, v9, v16
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v8, 8, v8
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v1, 12, v1
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v7, 12, v7
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v5, 12, v5
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v2, 12, v2
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v14, 12, v14
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v12, 12, v12
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v7, 12, v7
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v6, 12, v6
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v0, 12, v0
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v14, 12, v14
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v13, 12, v13
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v11, 12, v11
-; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v6, v6, v13
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v1, 12, v1
-; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v2, 12, v2
 ; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v10, v10, v15
 ; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v8, v9, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v7, 12, v7
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v5, 12, v5
+; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v14, 12, v14
 ; GFX10-DL-XNACK-NEXT:    v_ashrrev_i16 v12, 12, v12
-; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v9, v0, v11
+; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v1, v1, v2
+; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v2, v0, v11
+; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v6, v6, v13
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v9, 8, v10
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
+; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v10, v5, v12
 ; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v11, v7, v14
+; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v2, 8, v2
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v6, 8, v6
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v10, 8, v10
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
-; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v1, v1, v2
-; GFX10-DL-XNACK-NEXT:    v_mul_lo_u16 v2, v5, v12
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b16 v9, 8, v9
-; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v6, v11, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v11, v10, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v1, v1, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v2, v2, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-XNACK-NEXT:    v_lshlrev_b32_e32 v9, 16, v6
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 8, v11
+; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v13, v9, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v1, v1, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v2, v10, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v9, v11, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 8, v13
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-XNACK-NEXT:    v_add_nc_u16 v3, v1, v3
 ; GFX10-DL-XNACK-NEXT:    v_or_b32_sdwa v1, v2, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX10-DL-XNACK-NEXT:    v_add_nc_u16 v9, v3, v10
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
 ; GFX10-DL-XNACK-NEXT:    v_add_nc_u16 v0, v9, v8
 ; GFX10-DL-XNACK-NEXT:    v_add_nc_u16 v0, v0, v2

 ; GFX10-DL-NOXNACK-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX10-DL-NOXNACK-NEXT:    global_load_dword v0, v0, s[6:7]
 ; GFX10-DL-NOXNACK-NEXT:    global_load_ubyte v2, v4, s[0:1]
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v8, 12, v1
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(1)
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v15, 12, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v9, 8, v1
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v11, 20, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v12, 16, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v13, 28, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v14, 24, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v16, 8, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v17, 4, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v0, 12, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v8, 12, v8
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v15, 12, v15
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v17, 4, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v9, 12, v9
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v18, 12, v0
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v0, 12, v16
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v16, 12, v16
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v8, 12, v8
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v15, 12, v15
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v3, 20, v1
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v6, 12, v6
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v13, 12, v13
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v11, 20, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v13, 28, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v10, 12, v10
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v17, 12, v17
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v9, 12, v9
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v0, 12, v0
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v16, 12, v16
 ; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v8, v8, v15
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v7, 12, v7
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v12, 16, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v14, 24, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v6, 12, v6
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v3, 12, v3
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v14, 12, v14
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v13, 12, v13
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v11, 12, v11
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v6, 12, v6
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v13, 12, v13
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v1, 12, v1
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v0, 12, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v10, 12, v10
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v15, 12, v17
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v0, v9, v0
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v9, v9, v16
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v8, 8, v8
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v7, 12, v7
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v5, 12, v5
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v14, 12, v14
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v12, 12, v12
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v7, 12, v7
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v6, 12, v6
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v3, 12, v3
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v14, 12, v14
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v9, 12, v11
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v6, v6, v13
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v13, 12, v13
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v11, 12, v11
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v1, 12, v1
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v0, 12, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v10, v10, v15
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v8, v0, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v8, v9, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v7, 12, v7
 ; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v5, 12, v5
-; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v11, 12, v12
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v3, v3, v9
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v9, v7, v14
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v6, 8, v6
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v10, 8, v10
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v14, 12, v14
+; GFX10-DL-NOXNACK-NEXT:    v_ashrrev_i16 v12, 12, v12
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v3, v3, v11
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v6, v6, v13
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v1, v1, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v9, 8, v10
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v1, v1, v18
-; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v12, v5, v11
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v10, v5, v12
+; GFX10-DL-NOXNACK-NEXT:    v_mul_lo_u16 v11, v7, v14
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v3, 8, v3
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v6, v9, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v9, v10, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v1, v1, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v3, v12, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b32_e32 v10, 16, v6
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v9, 8, v9
+; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b16 v6, 8, v6
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v13, v9, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v1, v1, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v3, v10, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v9, v11, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v10, 8, v13
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v2, v1, v2
-; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v1, v3, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v9, v2, v9
+; GFX10-DL-NOXNACK-NEXT:    v_or_b32_sdwa v1, v3, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v9, v2, v10
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v0, v9, v8
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v0, v0, v2
-; GFX10-DL-NOXNACK-NEXT:    v_mad_u16 v0, v5, v11, v0
+; GFX10-DL-NOXNACK-NEXT:    v_mad_u16 v0, v5, v12, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v0, v0, v1
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v1, 8, v6
 ; GFX10-DL-NOXNACK-NEXT:    v_mad_u16 v0, v7, v14, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16 v0, v0, v1
 ; GFX10-DL-NOXNACK-NEXT:    global_store_byte v4, v0, s[0:1]
 ; GFX10-DL-NOXNACK-NEXT:    s_endpgm
 ; GFX10-DL-LABEL: idot8_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry

llvm/test/CodeGen/AMDGPU/idot8u.ll

 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
-; GFX7-NEXT:    s_mov_b32 s4, 0xf0000
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_u32 v8, v2, 20, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 12, v2
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 28, v2
 ; GFX7-NEXT:    v_bfe_u32 v4, v2, 24, 4
-; GFX7-NEXT:    v_bfe_u32 v5, v2, 12, 4
-; GFX7-NEXT:    v_bfe_u32 v6, v2, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v7, 15, v2
-; GFX7-NEXT:    v_alignbit_b32 v2, v8, v2, 16
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v9
+; GFX7-NEXT:    v_bfe_u32 v5, v2, 20, 4
+; GFX7-NEXT:    v_bfe_u32 v6, v2, 16, 4
+; GFX7-NEXT:    v_bfe_u32 v7, v2, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v8, v2, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v9, v2, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 12, v0
-; GFX7-NEXT:    v_and_b32_e32 v14, 15, v0
-; GFX7-NEXT:    v_or_b32_e32 v7, v7, v8
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v9
-; GFX7-NEXT:    v_or_b32_e32 v8, v14, v8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v9, 16, v7
-; GFX7-NEXT:    v_and_b32_e32 v7, 15, v7
-; GFX7-NEXT:    v_lshrrev_b32_e32 v14, 16, v8
-; GFX7-NEXT:    v_and_b32_e32 v8, 15, v8
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v8, v1
-; GFX7-NEXT:    v_bfe_u32 v13, v0, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v15, v0, 20, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v14, v1
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 28, v0
 ; GFX7-NEXT:    v_bfe_u32 v11, v0, 24, 4
-; GFX7-NEXT:    v_bfe_u32 v12, v0, 12, 4
-; GFX7-NEXT:    v_alignbit_b32 v0, v15, v0, 16
-; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v13, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v16, 16, v2
-; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 16, v0
+; GFX7-NEXT:    v_bfe_u32 v12, v0, 20, 4
+; GFX7-NEXT:    v_bfe_u32 v13, v0, 16, 4
+; GFX7-NEXT:    v_bfe_u32 v14, v0, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v15, v0, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v16, v0, 4, 4
 ; GFX7-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v12, v1
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v16, v15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v14, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v13, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24

 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ubyte v1, off, s[0:3], 0
-; GFX7-NEXT:    s_movk_i32 s4, 0xf00
-; GFX7-NEXT:    s_movk_i32 s5, 0xf0f
+; GFX7-NEXT:    s_movk_i32 s4, 0xf0f
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 4, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
-; GFX7-NEXT:    v_bfe_u32 v3, v2, 8, 4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 28, v2
-; GFX7-NEXT:    v_bfe_u32 v7, v2, 16, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 4, v2
+; GFX7-NEXT:    v_and_b32_e32 v6, 15, v2
+; GFX7-NEXT:    v_bfe_u32 v5, v2, 4, 4
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 4, v0
-; GFX7-NEXT:    v_lshrrev_b32_e32 v13, 28, v0
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_and_b32_e32 v5, 15, v2
-; GFX7-NEXT:    v_bfe_u32 v10, v0, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v12, 15, v0
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 12, v0
-; GFX7-NEXT:    v_alignbit_b32 v2, v6, v2, 24
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v9
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 4, v0
-; GFX7-NEXT:    v_or_b32_e32 v7, v7, v8
-; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
-; GFX7-NEXT:    v_alignbit_b32 v0, v13, v0, 24
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v11
-; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v15
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v9
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v0, s5, v0
-; GFX7-NEXT:    v_or_b32_e32 v8, v10, v8
-; GFX7-NEXT:    v_and_b32_e32 v2, s5, v2
-; GFX7-NEXT:    v_or_b32_e32 v4, v14, v4
-; GFX7-NEXT:    v_or_b32_e32 v6, v12, v6
-; GFX7-NEXT:    v_or_b32_e32 v3, v5, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; GFX7-NEXT:    v_or_b32_e32 v0, v4, v0
-; GFX7-NEXT:    v_or_b32_e32 v4, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v2, v7, v2
-; GFX7-NEXT:    v_and_b32_e32 v7, 15, v3
-; GFX7-NEXT:    v_and_b32_e32 v13, 15, v4
-; GFX7-NEXT:    v_bfe_u32 v8, v3, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v14, v4, 8, 4
+; GFX7-NEXT:    v_and_b32_e32 v13, 15, v0
+; GFX7-NEXT:    v_bfe_u32 v7, v2, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v12, v0, 4, 4
+; GFX7-NEXT:    v_bfe_u32 v14, v0, 12, 4
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v13, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v3
-; GFX7-NEXT:    v_bfe_u32 v3, v3, 16, 4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 24, v4
-; GFX7-NEXT:    v_bfe_u32 v4, v4, 16, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v14, v1
-; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v4, v1
-; GFX7-NEXT:    v_and_b32_e32 v9, 15, v2
-; GFX7-NEXT:    v_and_b32_e32 v15, 15, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v11, v1
-; GFX7-NEXT:    v_bfe_u32 v10, v2, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v16, v0, 8, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v15, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
-; GFX7-NEXT:    v_bfe_u32 v2, v2, 16, 4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v12, 24, v0
-; GFX7-NEXT:    v_bfe_u32 v0, v0, 16, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v10, v16, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v13, v1
+; GFX7-NEXT:    v_bfe_u32 v4, v2, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v11, v0, 8, 4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 24, v7
+; GFX7-NEXT:    v_lshlrev_b32_e32 v14, 24, v14
+; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v12, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 28, v2
+; GFX7-NEXT:    v_alignbit_b32 v7, 0, v7, 24
+; GFX7-NEXT:    v_alignbit_b32 v14, 0, v14, 24
+; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v11, v1
+; GFX7-NEXT:    v_bfe_u32 v3, v2, 20, 4
+; GFX7-NEXT:    v_bfe_u32 v9, v2, 16, 4
+; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 28, v0
+; GFX7-NEXT:    v_alignbit_b32 v2, v8, v2, 24
+; GFX7-NEXT:    v_bfe_u32 v8, v0, 16, 4
+; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v14, v1
+; GFX7-NEXT:    v_bfe_u32 v10, v0, 20, 4
+; GFX7-NEXT:    v_alignbit_b32 v0, v15, v0, 24
+; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v8, v1
+; GFX7-NEXT:    v_and_b32_e32 v16, s4, v2
+; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v0
+; GFX7-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v10, v1
+; GFX7-NEXT:    v_bfe_u32 v13, v16, 8, 8
+; GFX7-NEXT:    v_bfe_u32 v5, v6, 8, 8
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v12, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v13, v5, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-NEXT:    global_load_ubyte v4, v3, s[2:3]
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
-; GFX9-NEXT:    v_bfe_u32 v9, v1, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v10, v1, 20, 4
-; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 28, v2
-; GFX9-NEXT:    v_bfe_u32 v16, v2, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v17, v2, 20, 4
 ; GFX9-NEXT:    v_bfe_u32 v0, v1, 4, 4
 ; GFX9-NEXT:    v_and_b32_e32 v5, 15, v1
 ; GFX9-NEXT:    v_bfe_u32 v6, v1, 12, 4
 ; GFX9-NEXT:    v_bfe_u32 v7, v1, 8, 4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
+; GFX9-NEXT:    v_bfe_u32 v9, v1, 24, 4
+; GFX9-NEXT:    v_bfe_u32 v10, v1, 20, 4
 ; GFX9-NEXT:    v_bfe_u32 v11, v1, 16, 4
+; GFX9-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-NEXT:    v_bfe_u32 v1, v2, 4, 4
 ; GFX9-NEXT:    v_and_b32_e32 v12, 15, v2
 ; GFX9-NEXT:    v_bfe_u32 v13, v2, 12, 4
 ; GFX9-NEXT:    v_bfe_u32 v14, v2, 8, 4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 28, v2
+; GFX9-NEXT:    v_bfe_u32 v16, v2, 24, 4
+; GFX9-NEXT:    v_bfe_u32 v17, v2, 20, 4
 ; GFX9-NEXT:    v_bfe_u32 v2, v2, 16, 4
+; GFX9-NEXT:    v_mul_lo_u16_e32 v18, v11, v2
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v10, v10, v17 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v17, v9, v16
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v18, v11, v2
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, v6, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v8, v17, v8
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v5, v5, v12
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v12, v0, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v1, v18, v10
+; GFX9-NEXT:    v_or_b32_e32 v0, v18, v10
+; GFX9-NEXT:    v_or_b32_sdwa v1, v17, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_e32 v6, v7, v6
-; GFX9-NEXT:    v_lshlrev_b32_e32 v7, 16, v8
+; GFX9-NEXT:    v_or_b32_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v6
-; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_e32 v5, v5, v12
-; GFX9-NEXT:    v_or_b32_e32 v7, v12, v0
-; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 8, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v8
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-NEXT:    v_or_b32_e32 v10, v12, v0
 ; GFX9-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 8, v10
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u16_e32 v1, v5, v4
-; GFX9-NEXT:    v_add_u16_e32 v1, v1, v7
+; GFX9-NEXT:    v_add_u16_e32 v4, v5, v4
+; GFX9-NEXT:    v_add_u16_e32 v1, v4, v1
 ; GFX9-NEXT:    v_add_u16_e32 v1, v1, v6
 ; GFX9-NEXT:    v_add_u16_e32 v0, v1, v0
 ; GFX9-NEXT:    v_mad_legacy_u16 v0, v11, v2, v0
-; GFX9-NEXT:    v_add_u16_e32 v0, v0, v10
-; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v8
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v9, v16, v0
 ; GFX9-NEXT:    v_add_u16_e32 v0, v0, v8
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v9, v16, v0
+; GFX9-NEXT:    v_add_u16_e32 v0, v0, v7
 ; GFX9-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-DL-NEXT:    global_load_ubyte v4, v3, s[2:3]
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v1, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v10, v1, 20, 4
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 28, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v2, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v17, v2, 20, 4
 ; GFX9-DL-NEXT:    v_bfe_u32 v0, v1, 4, 4
 ; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v1
 ; GFX9-DL-NEXT:    v_bfe_u32 v6, v1, 12, 4
 ; GFX9-DL-NEXT:    v_bfe_u32 v7, v1, 8, 4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
+; GFX9-DL-NEXT:    v_bfe_u32 v9, v1, 24, 4
+; GFX9-DL-NEXT:    v_bfe_u32 v10, v1, 20, 4
 ; GFX9-DL-NEXT:    v_bfe_u32 v11, v1, 16, 4
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-DL-NEXT:    v_bfe_u32 v1, v2, 4, 4
 ; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v2
 ; GFX9-DL-NEXT:    v_bfe_u32 v13, v2, 12, 4
 ; GFX9-DL-NEXT:    v_bfe_u32 v14, v2, 8, 4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 28, v2
+; GFX9-DL-NEXT:    v_bfe_u32 v16, v2, 24, 4
+; GFX9-DL-NEXT:    v_bfe_u32 v17, v2, 20, 4
 ; GFX9-DL-NEXT:    v_bfe_u32 v2, v2, 16, 4
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v18, v11, v2
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v10, v10, v17 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v17, v9, v16
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v18, v11, v2
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v6, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v8, v17, v8
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v5, v5, v12
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v12, v0, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v1, v18, v10
+; GFX9-DL-NEXT:    v_or_b32_e32 v0, v18, v10
+; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v17, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_or_b32_e32 v6, v7, v6
-; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v7, 16, v8
+; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v6
-; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v1, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_or_b32_e32 v5, v5, v12
-; GFX9-DL-NEXT:    v_or_b32_e32 v7, v12, v0
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 8, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v8
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-DL-NEXT:    v_or_b32_e32 v10, v12, v0
 ; GFX9-DL-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v10
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v5, v4
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v7
+; GFX9-DL-NEXT:    v_add_u16_e32 v4, v5, v4
+; GFX9-DL-NEXT:    v_add_u16_e32 v1, v4, v1
 ; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v6
 ; GFX9-DL-NEXT:    v_add_u16_e32 v0, v1, v0
 ; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v11, v2, v0
-; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v10
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v8
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v9, v16, v0
 ; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v8
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v9, v16, v0
+; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v7
 ; GFX9-DL-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX10-DL-NEXT:    global_load_ubyte v3, v4, s[0:1]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
 ; GFX10-DL-NEXT:    v_bfe_u32 v6, v1, 12, 4
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_bfe_u32 v10, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v1, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v13, v2, 8, 4
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
-; GFX10-DL-NEXT:    v_mul_lo_u16 v6, v6, v10
+; GFX10-DL-NEXT:    v_bfe_u32 v9, v2, 12, 4
 ; GFX10-DL-NEXT:    v_bfe_u32 v0, v1, 4, 4
 ; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v1
-; GFX10-DL-NEXT:    v_bfe_u32 v9, v1, 24, 4
+; GFX10-DL-NEXT:    v_bfe_u32 v7, v1, 8, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 28, v1
+; GFX10-DL-NEXT:    v_bfe_u32 v10, v1, 24, 4
 ; GFX10-DL-NEXT:    v_bfe_u32 v11, v1, 20, 4
 ; GFX10-DL-NEXT:    v_bfe_u32 v12, v1, 16, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v1, v2, 4, 4
-; GFX10-DL-NEXT:    v_mul_lo_u16 v7, v7, v13
+; GFX10-DL-NEXT:    v_bfe_u32 v1, v2, 8, 4
+; GFX10-DL-NEXT:    v_mul_lo_u16 v6, v6, v9
+; GFX10-DL-NEXT:    v_bfe_u32 v9, v2, 4, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
+; GFX10-DL-NEXT:    v_bfe_u32 v15, v2, 20, 4
+; GFX10-DL-NEXT:    v_mul_lo_u16 v1, v7, v1
 ; GFX10-DL-NEXT:    v_lshlrev_b16 v6, 8, v6
-; GFX10-DL-NEXT:    v_and_b32_e32 v10, 15, v2
-; GFX10-DL-NEXT:    v_bfe_u32 v15, v2, 24, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v13, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v16, v2, 16, 4
-; GFX10-DL-NEXT:    v_mul_lo_u16 v2, v8, v14
-; GFX10-DL-NEXT:    v_mul_lo_u16 v0, v0, v1
-; GFX10-DL-NEXT:    v_or_b32_e32 v6, v7, v6
-; GFX10-DL-NEXT:    v_mul_lo_u16 v1, v11, v13
-; GFX10-DL-NEXT:    v_mul_lo_u16 v7, v9, v15
-; GFX10-DL-NEXT:    v_lshlrev_b16 v2, 8, v2
-; GFX10-DL-NEXT:    v_lshlrev_b16 v8, 8, v0
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_mul_lo_u16 v0, v0, v9
+; GFX10-DL-NEXT:    v_bfe_u32 v7, v2, 16, 4
+; GFX10-DL-NEXT:    v_bfe_u32 v16, v2, 24, 4
+; GFX10-DL-NEXT:    v_or_b32_e32 v6, v1, v6
+; GFX10-DL-NEXT:    v_mul_lo_u16 v2, v11, v15
+; GFX10-DL-NEXT:    v_mul_lo_u16 v8, v8, v14
+; GFX10-DL-NEXT:    v_lshlrev_b16 v9, 8, v0
+; GFX10-DL-NEXT:    v_mul_lo_u16 v5, v5, v13
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v6
-; GFX10-DL-NEXT:    v_mul_lo_u16 v5, v5, v10
-; GFX10-DL-NEXT:    v_mul_lo_u16 v10, v12, v16
-; GFX10-DL-NEXT:    v_lshlrev_b16 v1, 8, v1
-; GFX10-DL-NEXT:    v_or_b32_e32 v7, v7, v2
-; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v8, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_or_b32_e32 v5, v5, v8
-; GFX10-DL-NEXT:    v_or_b32_e32 v1, v10, v1
-; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v8, 16, v7
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX10-DL-NEXT:    v_mul_lo_u16 v1, v12, v7
+; GFX10-DL-NEXT:    v_mul_lo_u16 v11, v10, v16
+; GFX10-DL-NEXT:    v_lshlrev_b16 v2, 8, v2
+; GFX10-DL-NEXT:    v_lshlrev_b16 v8, 8, v8
+; GFX10-DL-NEXT:    v_or_b32_sdwa v13, v9, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_or_b32_e32 v5, v5, v9
+; GFX10-DL-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v11, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v13
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-NEXT:    v_add_nc_u16 v3, v5, v3
-; GFX10-DL-NEXT:    v_or_b32_sdwa v1, v1, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_add_nc_u16 v5, v3, v2
+; GFX10-DL-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_add_nc_u16 v5, v3, v9
 ; GFX10-DL-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
 ; GFX10-DL-NEXT:    v_add_nc_u16 v0, v5, v6
 ; GFX10-DL-NEXT:    v_add_nc_u16 v0, v0, v2
-; GFX10-DL-NEXT:    v_mad_u16 v0, v12, v16, v0
+; GFX10-DL-NEXT:    v_mad_u16 v0, v12, v7, v0
 ; GFX10-DL-NEXT:    v_add_nc_u16 v0, v0, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v7
-; GFX10-DL-NEXT:    v_mad_u16 v0, v9, v15, v0
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v8
+; GFX10-DL-NEXT:    v_mad_u16 v0, v10, v16, v0
 ; GFX10-DL-NEXT:    v_add_nc_u16 v0, v0, v1
 ; GFX10-DL-NEXT:    global_store_byte v4, v0, s[0:1]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx

llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll

 ; VI-NEXT:    s_load_dword s4, s[4:5], 0x10
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
 ; VI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    s_and_b32 s1, s4, s2
+; VI-NEXT:    s_lshl_b32 s1, s4, 16
+; VI-NEXT:    s_and_b32 s4, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
-; VI-NEXT:    s_lshl_b32 s0, s1, 16
-; VI-NEXT:    s_or_b32 s0, s1, s0
+; VI-NEXT:    s_or_b32 s0, s4, s1
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    v_lshlrev_b32_e32 v4, 4, v4
 ; VI-NEXT:    v_lshlrev_b64 v[4:5], v4, s[2:3]
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_bfi_b32 v1, v5, s0, v1
 ; VI-NEXT:    v_bfi_b32 v0, v4, s0, v0
 ; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; VI-NEXT:    s_endpgm

 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
 ; VI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
 ; VI-NEXT:    s_lshl_b32 s1, s5, 4
+; VI-NEXT:    s_lshl_b32 s5, s4, 16
 ; VI-NEXT:    s_and_b32 s4, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[2:3], s1
-; VI-NEXT:    s_lshl_b32 s2, s4, 16
-; VI-NEXT:    s_or_b32 s2, s4, s2
+; VI-NEXT:    s_or_b32 s2, s4, s5
 ; VI-NEXT:    v_mov_b32_e32 v4, s2
 ; VI-NEXT:    v_mov_b32_e32 v5, s2
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_bfi_b32 v1, s1, v4, v1
 ; VI-NEXT:    v_bfi_b32 v0, s0, v5, v0
 ; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; VI-NEXT:    s_endpgm

llvm/test/CodeGen/AMDGPU/saddsat.ll

 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    s_movk_i32 s4, 0x7fff
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_i32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_movk_i32 s5, 0x8000
 ; GFX6-NEXT:    v_min_i32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_max_i32_e32 v1, s5, v1
 ; GFX6-NEXT:    v_max_i32_e32 v0, s5, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    s_mov_b32 s4, 0xffff
+; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_saddsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v2, 16, v1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
 ; GFX8-NEXT:    v_add_u16_e32 v4, v3, v2

llvm/test/CodeGen/AMDGPU/scalar_to_vector.ll

 ; VI-NEXT:    s_mov_b32 s11, s7
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s8, s2
 ; VI-NEXT:    s_mov_b32 s9, s3
 ; VI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    s_mov_b32 s5, s1
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; VI-NEXT:    v_alignbit_b32 v0, v1, v0, 16
+; VI-NEXT:    v_alignbit_b32 v0, s0, v0, 16
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
   %tmp1 = load i32, i32 addrspace(1)* %in, align 4
   %bc = bitcast i32 %tmp1 to <2 x i16>
   %tmp2 = shufflevector <2 x i16> %bc, <2 x i16> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
   store <4 x i16> %tmp2, <4 x i16> addrspace(1)* %out, align 8
   ret void

 ; VI-NEXT:    s_mov_b32 s11, s7
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s8, s2
 ; VI-NEXT:    s_mov_b32 s9, s3
 ; VI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    s_mov_b32 s5, s1
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; VI-NEXT:    v_alignbit_b32 v0, v1, v0, 16
+; VI-NEXT:    v_alignbit_b32 v0, s0, v0, 16
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
   %tmp1 = load float, float addrspace(1)* %in, align 4
   %bc = bitcast float %tmp1 to <2 x i16>
   %tmp2 = shufflevector <2 x i16> %bc, <2 x i16> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
   store <4 x i16> %tmp2, <4 x i16> addrspace(1)* %out, align 8
   ret void
 }
 
 define amdgpu_kernel void @scalar_to_vector_v4i16() {
 ; SI-LABEL: scalar_to_vector_v4i16:
 ; SI:       ; %bb.0: ; %bb
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v1, v0
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 8, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v1, v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: scalar_to_vector_v4i16:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_lshlrev_b16_e32 v1, 8, v0
-; VI-NEXT:    v_or_b32_e32 v0, v1, v0
-; VI-NEXT:    v_lshrrev_b16_e32 v1, 8, v0
-; VI-NEXT:    v_lshlrev_b16_e32 v2, 8, v1
-; VI-NEXT:    v_or_b32_e32 v1, v1, v2
-; VI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; VI-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; VI-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 bb:
   %tmp = load <2 x i8>, <2 x i8> addrspace(1)* undef, align 1
   %tmp1 = shufflevector <2 x i8> %tmp, <2 x i8> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
   %tmp2 = shufflevector <8 x i8> %tmp1, <8 x i8> undef, <8 x i32> <i32 0, i32 9, i32 9, i32 9, i32 9, i32 9, i32 9, i32 9>
   store <8 x i8> %tmp2, <8 x i8> addrspace(1)* undef, align 8
   ret void
 }
 
 define amdgpu_kernel void @scalar_to_vector_v4f16() {
 ; SI-LABEL: scalar_to_vector_v4f16:
 ; SI:       ; %bb.0: ; %bb
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v1, v0
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 8, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v1, v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: scalar_to_vector_v4f16:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0

llvm/test/CodeGen/AMDGPU/shift-i128.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i128 %shl
 }
 
 
 define i128 @v_shl_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_shl_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_alignbit_b32 v4, v2, v1, 15
+; GCN-NEXT:    v_lshl_b64 v[2:3], v[2:3], 17
+; GCN-NEXT:    v_lshrrev_b32_e32 v4, 15, v1
+; GCN-NEXT:    v_or_b32_e32 v2, v2, v4
 ; GCN-NEXT:    v_alignbit_b32 v1, v1, v0, 15
-; GCN-NEXT:    v_alignbit_b32 v3, v3, v2, 15
 ; GCN-NEXT:    v_lshlrev_b32_e32 v0, 17, v0
-; GCN-NEXT:    v_mov_b32_e32 v2, v4
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = shl i128 %lhs, 17
   ret i128 %shl
 }
 
 define i128 @v_lshr_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_lshr_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    v_alignbit_b32 v0, v3, v2, 1
 ; GCN-NEXT:    v_lshrrev_b32_e32 v1, 1, v3
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    v_mov_b32_e32 v3, 0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = lshr i128 %lhs, 65
   ret i128 %shl
 }
 
 define i128 @v_ashr_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_ashr_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_ashr_i64 v[4:5], v[2:3], 33
-; GCN-NEXT:    v_alignbit_b32 v0, v2, v1, 1
-; GCN-NEXT:    v_alignbit_b32 v1, v3, v2, 1
-; GCN-NEXT:    v_mov_b32_e32 v2, v4
-; GCN-NEXT:    v_mov_b32_e32 v3, v5
+; GCN-NEXT:    v_mov_b32_e32 v4, v1
+; GCN-NEXT:    v_lshl_b64 v[0:1], v[2:3], 31
+; GCN-NEXT:    v_lshrrev_b32_e32 v4, 1, v4
+; GCN-NEXT:    v_ashr_i64 v[2:3], v[2:3], 33
+; GCN-NEXT:    v_or_b32_e32 v0, v4, v0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = ashr i128 %lhs, 33
   ret i128 %shl
 }
 
 define i128 @v_shl_i128_kv(i128 %rhs) {
 ; GCN-LABEL: v_shl_i128_kv:
 ; GCN:       ; %bb.0:

llvm/test/CodeGen/AMDGPU/ssubsat.ll

 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    s_movk_i32 s4, 0x7fff
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_i32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_movk_i32 s5, 0x8000
 ; GFX6-NEXT:    v_min_i32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_max_i32_e32 v1, s5, v1
 ; GFX6-NEXT:    v_max_i32_e32 v0, s5, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    s_mov_b32 s4, 0xffff
+; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_ssubsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v2, 16, v1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
 ; GFX8-NEXT:    v_sub_u16_e32 v4, v3, v2

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

bb:
   store <2 x i16> %tmp14, <2 x i16> addrspace(1)* %tmp15, align 4
   ret void
 }
 
 define <2 x i16> @trunc_v2i64_arg_to_v2i16(<2 x i64> %arg0) #0 {
 ; SI-LABEL: trunc_v2i64_arg_to_v2i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s4, 0xffff
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
-; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; SI-NEXT:    v_and_b32_e32 v0, s4, v0
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_and_b32_e32 v1, s4, v2
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: trunc_v2i64_arg_to_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
 ; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    s_setpc_b64 s[30:31]
   %trunc = trunc <2 x i64> %arg0 to <2 x i16>
   ret <2 x i16> %trunc
 }

llvm/test/CodeGen/AMDGPU/uaddsat.ll

 ; GFX6-NEXT:    v_and_b32_e32 v3, s4, v3
 ; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_u32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_min_u32_e32 v0, s4, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_uaddsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_add_u16_sdwa v2, v0, v1 clamp dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_add_u16_e64 v0, v0, v1 clamp
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2

llvm/test/CodeGen/AMDGPU/usubsat.ll

; GFX10-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %result
 }
 
 define <2 x i16> @v_usubsat_v2i16(<2 x i16> %lhs, <2 x i16> %rhs) {
 ; GFX6-LABEL: v_usubsat_v2i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX6-NEXT:    s_mov_b32 s4, 0xffff
-; GFX6-NEXT:    v_and_b32_e32 v4, s4, v3
+; GFX6-NEXT:    v_and_b32_e32 v3, s4, v3
 ; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX6-NEXT:    v_max_u32_e32 v1, v1, v4
+; GFX6-NEXT:    v_max_u32_e32 v1, v1, v3
 ; GFX6-NEXT:    v_max_u32_e32 v0, v0, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_usubsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_sub_u16_sdwa v2, v0, v1 clamp dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_sub_u16_e64 v0, v0, v1 clamp
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2

llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll

 ; LE-NEXT:    ldr r1, [r0]
 ; LE-NEXT:    orr r1, r1, #384
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_or:
 ; BE:       @ %bb.0:
 ; BE-NEXT:    mov r1, r0
-; BE-NEXT:    ldr r12, [r0]
+; BE-NEXT:    ldr r0, [r0]
 ; BE-NEXT:    ldrh r2, [r1, #4]!
 ; BE-NEXT:    ldrb r3, [r1, #2]
 ; BE-NEXT:    orr r2, r3, r2, lsl #8
-; BE-NEXT:    orr r2, r2, r12, lsl #24
-; BE-NEXT:    orr r2, r2, #384
-; BE-NEXT:    strb r2, [r1, #2]
-; BE-NEXT:    lsr r3, r2, #8
-; BE-NEXT:    strh r3, [r1]
-; BE-NEXT:    bic r1, r12, #255
-; BE-NEXT:    orr r1, r1, r2, lsr #24
-; BE-NEXT:    str r1, [r0]
+; BE-NEXT:    orr r0, r2, r0, lsl #24
+; BE-NEXT:    orr r0, r0, #384
+; BE-NEXT:    strb r0, [r1, #2]
+; BE-NEXT:    lsr r0, r0, #8
+; BE-NEXT:    strh r0, [r1]
 ; BE-NEXT:    mov pc, lr
   %aa = load i56, i56* %a
   %b = or i56 %aa, 384
   store i56 %b, i56* %a
   ret void
 }
 
 define void @i56_and_or(i56* %a) {
 ; LE-LABEL: i56_and_or:
 ; LE:       @ %bb.0:
 ; LE-NEXT:    ldr r1, [r0]
 ; LE-NEXT:    orr r1, r1, #384
 ; LE-NEXT:    bic r1, r1, #127
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_and_or:
 ; BE:       @ %bb.0:
-; BE-NEXT:    mov r1, r0
+; BE-NEXT:    ldrh r1, [r0, #4]!
 ; BE-NEXT:    mov r2, #128
-; BE-NEXT:    ldrh r12, [r1, #4]!
-; BE-NEXT:    ldrb r3, [r1, #2]
-; BE-NEXT:    strb r2, [r1, #2]
-; BE-NEXT:    orr r2, r3, r12, lsl #8
-; BE-NEXT:    ldr r12, [r0]
-; BE-NEXT:    orr r2, r2, r12, lsl #24
-; BE-NEXT:    orr r2, r2, #384
-; BE-NEXT:    lsr r3, r2, #8
-; BE-NEXT:    strh r3, [r1]
-; BE-NEXT:    bic r1, r12, #255
-; BE-NEXT:    orr r1, r1, r2, lsr #24
-; BE-NEXT:    str r1, [r0]
+; BE-NEXT:    orr r1, r1, #1
+; BE-NEXT:    strb r2, [r0, #2]
+; BE-NEXT:    strh r1, [r0]
 ; BE-NEXT:    mov pc, lr
 
   %b = load i56, i56* %a, align 1
   %c = and i56 %b, -128
   %d = or i56 %c, 384
   store i56 %d, i56* %a, align 1
   ret void
 }
 
 define void @i56_insert_bit(i56* %a, i1 zeroext %bit) {
 ; LE-LABEL: i56_insert_bit:
 ; LE:       @ %bb.0:
 ; LE-NEXT:    ldr r2, [r0]
 ; LE-NEXT:    bic r2, r2, #8192
 ; LE-NEXT:    orr r1, r2, r1, lsl #13
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_insert_bit:
 ; BE:       @ %bb.0:
-; BE-NEXT:    .save {r11, lr}
-; BE-NEXT:    push {r11, lr}
-; BE-NEXT:    mov r2, r0
-; BE-NEXT:    ldr lr, [r0]
-; BE-NEXT:    ldrh r12, [r2, #4]!
-; BE-NEXT:    ldrb r3, [r2, #2]
-; BE-NEXT:    orr r12, r3, r12, lsl #8
-; BE-NEXT:    orr r3, r12, lr, lsl #24
-; BE-NEXT:    bic r3, r3, #8192
-; BE-NEXT:    orr r1, r3, r1, lsl #13
-; BE-NEXT:    lsr r3, r1, #8
-; BE-NEXT:    strh r3, [r2]
-; BE-NEXT:    bic r2, lr, #255
-; BE-NEXT:    orr r1, r2, r1, lsr #24
-; BE-NEXT:    str r1, [r0]
-; BE-NEXT:    pop {r11, lr}
+; BE-NEXT:    ldrh r2, [r0, #4]!
+; BE-NEXT:    mov r3, #57088
+; BE-NEXT:    orr r3, r3, #16711680
+; BE-NEXT:    and r2, r3, r2, lsl #8
+; BE-NEXT:    orr r1, r2, r1, lsl #13
+; BE-NEXT:    lsr r1, r1, #8
+; BE-NEXT:    strh r1, [r0]
 ; BE-NEXT:    mov pc, lr
   %extbit = zext i1 %bit to i56
   %b = load i56, i56* %a, align 1
   %extbit.shl = shl nuw nsw i56 %extbit, 13
   %c = and i56 %b, -8193
   %d = or i56 %c, %extbit.shl
   store i56 %d, i56* %a, align 1
   ret void
 }
 

llvm/test/CodeGen/ARM/parity.ll

; CHECK-NEXT:    bx lr
   %2 = and i16 %1, 1
   ret i16 %2
 }
 
 define i17 @parity_17(i17 %x) {
 ; CHECK-LABEL: parity_17:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    bfc r0, #17, #15
-; CHECK-NEXT:    eor r0, r0, r0, lsr #16
-; CHECK-NEXT:    eor r0, r0, r0, lsr #8
+; CHECK-NEXT:    eor r1, r0, r0, lsr #16
+; CHECK-NEXT:    eor r0, r1, r0, lsr #8
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #4
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #2
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #1
 ; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    bx lr
   %1 = tail call i17 @llvm.ctpop.i17(i17 %x)
   %2 = and i17 %1, 1
   ret i17 %2

llvm/test/CodeGen/ARM/ror.ll

entry:
   %result.i.3 = or i32 %low_part.i.2, %high_part.i.1
   ret i32 %result.i.3
 }
 
 ; the same vector test
 define <2 x i32> @test2(<2 x i32> %x) nounwind readnone {
 ; CHECK-LABEL: test2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ror r0, r0, #10
-; CHECK-NEXT:    ror r1, r1, #10
+; CHECK-NEXT:    bic r2, r0, #15
+; CHECK-NEXT:    ror r0, r0, #4
+; CHECK-NEXT:    lsr r0, r0, #6
+; CHECK-NEXT:    orr r0, r0, r2, lsl #22
+; CHECK-NEXT:    bic r2, r1, #15
+; CHECK-NEXT:    ror r1, r1, #4
+; CHECK-NEXT:    lsr r1, r1, #6
+; CHECK-NEXT:    orr r1, r1, r2, lsl #22
 ; CHECK-NEXT:    bx lr
 entry:
   %high_part.i = shl <2 x i32> %x, <i32 28, i32 28>
   %low_part.i = lshr <2 x i32> %x, <i32 4, i32 4>
   %result.i = or <2 x i32> %high_part.i, %low_part.i
   %high_part.i.1 = shl <2 x i32> %result.i, <i32 26, i32 26>
   %low_part.i.2 = lshr <2 x i32> %result.i, <i32 6, i32 6>
   %result.i.3 = or <2 x i32> %low_part.i.2, %high_part.i.1
   ret <2 x i32> %result.i.3
 }
 

llvm/test/CodeGen/ARM/uxtb.ll

; CHECK-NEXT:    bx lr
   %tmp6 = or i32 %tmp5, %tmp1
   ret i32 %tmp6
 }
 
 define i32 @test10(i32 %p0) {
 ; CHECK-LABEL: test10:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    mov r1, #248
+; CHECK-NEXT:    mov r2, #7
 ; CHECK-NEXT:    orr r1, r1, #16252928
-; CHECK-NEXT:    and r0, r1, r0, lsr #7
-; CHECK-NEXT:    lsr r1, r0, #5
-; CHECK-NEXT:    uxtb16 r1, r1
-; CHECK-NEXT:    orr r0, r1, r0
+; CHECK-NEXT:    orr r2, r2, #458752
+; CHECK-NEXT:    and r1, r1, r0, lsr #7
+; CHECK-NEXT:    and r0, r2, r0, lsr #12
+; CHECK-NEXT:    orr r0, r0, r1
 ; CHECK-NEXT:    bx lr
   %tmp1 = lshr i32 %p0, 7
   %tmp2 = and i32 %tmp1, 16253176
   %tmp4 = lshr i32 %tmp2, 5
   %tmp5 = and i32 %tmp4, 458759
   %tmp7 = or i32 %tmp5, %tmp2
   ret i32 %tmp7
 }
 

llvm/test/CodeGen/Mips/urem-seteq-illegal-types.ll

 ;   %urem = urem <3 x i11> %X, <i11 6, i11 7, i11 -5>
 ;   %cmp = icmp ne <3 x i11> %urem, <i11 0, i11 1, i11 2>
 ;   ret <3 x i1> %cmp
 ; }
 
 define i1 @test_urem_oversized(i66 %X) nounwind {
 ; MIPSEL-LABEL: test_urem_oversized:
 ; MIPSEL:       # %bb.0:
-; MIPSEL-NEXT:    lui $1, 12057
-; MIPSEL-NEXT:    ori $1, $1, 37186
+; MIPSEL-NEXT:    lui $1, 52741
+; MIPSEL-NEXT:    ori $1, $1, 40665
 ; MIPSEL-NEXT:    multu $6, $1
-; MIPSEL-NEXT:    mflo $1
 ; MIPSEL-NEXT:    mfhi $2
-; MIPSEL-NEXT:    lui $3, 52741
-; MIPSEL-NEXT:    ori $3, $3, 40665
-; MIPSEL-NEXT:    multu $6, $3
-; MIPSEL-NEXT:    mflo $7
-; MIPSEL-NEXT:    mfhi $8
-; MIPSEL-NEXT:    multu $5, $3
-; MIPSEL-NEXT:    mfhi $9
+; MIPSEL-NEXT:    mflo $3
+; MIPSEL-NEXT:    multu $5, $1
+; MIPSEL-NEXT:    mfhi $7
+; MIPSEL-NEXT:    mflo $8
+; MIPSEL-NEXT:    lui $9, 12057
+; MIPSEL-NEXT:    ori $9, $9, 37186
+; MIPSEL-NEXT:    multu $6, $9
 ; MIPSEL-NEXT:    mflo $10
-; MIPSEL-NEXT:    addu $8, $10, $8
-; MIPSEL-NEXT:    addu $11, $1, $8
-; MIPSEL-NEXT:    sltu $8, $8, $10
-; MIPSEL-NEXT:    sll $10, $11, 31
-; MIPSEL-NEXT:    sltu $1, $11, $1
-; MIPSEL-NEXT:    srl $12, $7, 1
-; MIPSEL-NEXT:    sll $7, $7, 1
-; MIPSEL-NEXT:    addu $1, $2, $1
-; MIPSEL-NEXT:    or $10, $12, $10
-; MIPSEL-NEXT:    srl $2, $11, 1
-; MIPSEL-NEXT:    addu $8, $9, $8
-; MIPSEL-NEXT:    mul $3, $4, $3
-; MIPSEL-NEXT:    sll $4, $6, 1
-; MIPSEL-NEXT:    sll $5, $5, 1
+; MIPSEL-NEXT:    mfhi $11
+; MIPSEL-NEXT:    addu $2, $8, $2
+; MIPSEL-NEXT:    addu $12, $10, $2
+; MIPSEL-NEXT:    sltu $2, $2, $8
+; MIPSEL-NEXT:    addu $2, $7, $2
+; MIPSEL-NEXT:    sltu $7, $12, $10
+; MIPSEL-NEXT:    sll $8, $12, 31
+; MIPSEL-NEXT:    srl $10, $12, 1
+; MIPSEL-NEXT:    sll $12, $3, 1
+; MIPSEL-NEXT:    srl $3, $3, 1
+; MIPSEL-NEXT:    mul $1, $4, $1
+; MIPSEL-NEXT:    mul $4, $5, $9
+; MIPSEL-NEXT:    sll $5, $6, 1
 ; MIPSEL-NEXT:    lui $6, 60010
-; MIPSEL-NEXT:    ori $6, $6, 61135
-; MIPSEL-NEXT:    addu $1, $8, $1
-; MIPSEL-NEXT:    subu $1, $1, $5
-; MIPSEL-NEXT:    addu $3, $4, $3
-; MIPSEL-NEXT:    addu $1, $1, $3
-; MIPSEL-NEXT:    andi $1, $1, 3
-; MIPSEL-NEXT:    sll $3, $1, 31
-; MIPSEL-NEXT:    or $3, $2, $3
-; MIPSEL-NEXT:    sltiu $2, $3, 13
-; MIPSEL-NEXT:    xori $3, $3, 13
-; MIPSEL-NEXT:    sltu $4, $10, $6
-; MIPSEL-NEXT:    movz $2, $4, $3
+; MIPSEL-NEXT:    addu $7, $11, $7
+; MIPSEL-NEXT:    addu $2, $2, $7
+; MIPSEL-NEXT:    addu $2, $4, $2
+; MIPSEL-NEXT:    addu $1, $5, $1
+; MIPSEL-NEXT:    addu $1, $2, $1
+; MIPSEL-NEXT:    sll $2, $1, 31
+; MIPSEL-NEXT:    or $4, $10, $2
+; MIPSEL-NEXT:    sltiu $2, $4, 13
+; MIPSEL-NEXT:    xori $4, $4, 13
+; MIPSEL-NEXT:    or $3, $3, $8
+; MIPSEL-NEXT:    ori $5, $6, 61135
+; MIPSEL-NEXT:    sltu $3, $3, $5
+; MIPSEL-NEXT:    movz $2, $3, $4
+; MIPSEL-NEXT:    andi $1, $1, 2
 ; MIPSEL-NEXT:    srl $1, $1, 1
-; MIPSEL-NEXT:    or $1, $1, $7
+; MIPSEL-NEXT:    or $1, $1, $12
 ; MIPSEL-NEXT:    andi $1, $1, 3
 ; MIPSEL-NEXT:    jr $ra
 ; MIPSEL-NEXT:    movn $2, $zero, $1
 ;
 ; MIPS64EL-LABEL: test_urem_oversized:
 ; MIPS64EL:       # %bb.0:
 ; MIPS64EL-NEXT:    lui $1, 6029
 ; MIPS64EL-NEXT:    daddiu $1, $1, -14175
 ; MIPS64EL-NEXT:    dsll $1, $1, 16
 ; MIPS64EL-NEXT:    daddiu $1, $1, 26371
 ; MIPS64EL-NEXT:    dsll $1, $1, 17
 ; MIPS64EL-NEXT:    daddiu $1, $1, -24871
 ; MIPS64EL-NEXT:    dmult $5, $1
 ; MIPS64EL-NEXT:    mflo $2
 ; MIPS64EL-NEXT:    dmultu $4, $1
 ; MIPS64EL-NEXT:    mflo $1
 ; MIPS64EL-NEXT:    mfhi $3
 ; MIPS64EL-NEXT:    lui $5, 14
 ; MIPS64EL-NEXT:    daddiu $5, $5, -5525
 ; MIPS64EL-NEXT:    dsll $5, $5, 16
 ; MIPS64EL-NEXT:    daddiu $5, $5, -4401
 ; MIPS64EL-NEXT:    dsll $4, $4, 1
 ; MIPS64EL-NEXT:    daddu $3, $3, $4
-; MIPS64EL-NEXT:    daddu $2, $3, $2
-; MIPS64EL-NEXT:    andi $3, $2, 3
+; MIPS64EL-NEXT:    daddu $3, $3, $2
 ; MIPS64EL-NEXT:    dsll $2, $3, 63
 ; MIPS64EL-NEXT:    dsrl $4, $1, 1
 ; MIPS64EL-NEXT:    or $2, $4, $2
 ; MIPS64EL-NEXT:    sltu $2, $2, $5
+; MIPS64EL-NEXT:    andi $3, $3, 2
 ; MIPS64EL-NEXT:    dsrl $3, $3, 1
 ; MIPS64EL-NEXT:    dsll $1, $1, 1
 ; MIPS64EL-NEXT:    or $1, $3, $1
 ; MIPS64EL-NEXT:    andi $1, $1, 3
 ; MIPS64EL-NEXT:    jr $ra
 ; MIPS64EL-NEXT:    movn $2, $zero, $1
   %urem = urem i66 %X, 1234567890
   %cmp = icmp eq i66 %urem, 0
   ret i1 %cmp
 }

llvm/test/CodeGen/PowerPC/fp-to-int-to-fp.ll

 ; PPC64-NEXT:    ld 5, -16(1)
 ; PPC64-NEXT:    xor 3, 3, 4
 ; PPC64-NEXT:    bc 12, 0, .LBB2_1
 ; PPC64-NEXT:    b .LBB2_2
 ; PPC64-NEXT:  .LBB2_1: # %entry
 ; PPC64-NEXT:    addi 3, 5, 0
 ; PPC64-NEXT:  .LBB2_2: # %entry
 ; PPC64-NEXT:    sradi 4, 3, 53
-; PPC64-NEXT:    clrldi 5, 3, 63
+; PPC64-NEXT:    rldicl 5, 3, 63, 1
 ; PPC64-NEXT:    addi 4, 4, 1
+; PPC64-NEXT:    clrldi 6, 3, 63
 ; PPC64-NEXT:    cmpldi 4, 1
-; PPC64-NEXT:    rldicl 4, 3, 63, 1
-; PPC64-NEXT:    or 5, 5, 4
-; PPC64-NEXT:    rldicl 6, 5, 11, 53
-; PPC64-NEXT:    addi 6, 6, 1
-; PPC64-NEXT:    clrldi 7, 5, 53
-; PPC64-NEXT:    cmpldi 1, 6, 1
-; PPC64-NEXT:    clrldi 6, 3, 53
+; PPC64-NEXT:    clrldi 4, 3, 53
+; PPC64-NEXT:    or 6, 6, 5
+; PPC64-NEXT:    clrldi 7, 6, 53
+; PPC64-NEXT:    addi 4, 4, 2047
 ; PPC64-NEXT:    addi 7, 7, 2047
-; PPC64-NEXT:    addi 6, 6, 2047
-; PPC64-NEXT:    or 4, 7, 4
-; PPC64-NEXT:    or 6, 6, 3
-; PPC64-NEXT:    rldicl 4, 4, 53, 11
-; PPC64-NEXT:    rldicr 6, 6, 0, 52
+; PPC64-NEXT:    or 4, 4, 3
+; PPC64-NEXT:    or 5, 7, 5
+; PPC64-NEXT:    rldicl 7, 3, 10, 54
+; PPC64-NEXT:    rldicr 4, 4, 0, 52
+; PPC64-NEXT:    addi 7, 7, 1
 ; PPC64-NEXT:    bc 12, 1, .LBB2_4
 ; PPC64-NEXT:  # %bb.3: # %entry
-; PPC64-NEXT:    ori 6, 3, 0
+; PPC64-NEXT:    ori 4, 3, 0
 ; PPC64-NEXT:    b .LBB2_4
 ; PPC64-NEXT:  .LBB2_4: # %entry
-; PPC64-NEXT:    rldicl 4, 4, 11, 1
-; PPC64-NEXT:    cmpdi 3, 0
-; PPC64-NEXT:    std 6, -32(1)
-; PPC64-NEXT:    bc 12, 5, .LBB2_6
+; PPC64-NEXT:    rldicl 5, 5, 53, 11
+; PPC64-NEXT:    std 4, -32(1)
+; PPC64-NEXT:    rldicl 4, 5, 11, 1
+; PPC64-NEXT:    cmpldi 7, 1
+; PPC64-NEXT:    bc 12, 1, .LBB2_6
 ; PPC64-NEXT:  # %bb.5: # %entry
-; PPC64-NEXT:    ori 4, 5, 0
+; PPC64-NEXT:    ori 4, 6, 0
 ; PPC64-NEXT:    b .LBB2_6
 ; PPC64-NEXT:  .LBB2_6: # %entry
+; PPC64-NEXT:    cmpdi 3, 0
 ; PPC64-NEXT:    std 4, -24(1)
 ; PPC64-NEXT:    bc 12, 0, .LBB2_8
 ; PPC64-NEXT:  # %bb.7: # %entry
 ; PPC64-NEXT:    lfd 0, -32(1)
 ; PPC64-NEXT:    fcfid 0, 0
 ; PPC64-NEXT:    frsp 1, 0
 ; PPC64-NEXT:    blr
 ; PPC64-NEXT:  .LBB2_8:

llvm/test/CodeGen/RISCV/bswap-bitreverse.ll

 ; RV32I-NEXT:    and a2, a2, a3
 ; RV32I-NEXT:    slli a0, a0, 24
 ; RV32I-NEXT:    or a0, a0, a2
 ; RV32I-NEXT:    or a0, a0, a1
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: test_bswap_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a1, a0, 8
+; RV64I-NEXT:    srli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 16
 ; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srliw a2, a0, 24
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3

 ; RV32I-NEXT:    and a1, a1, a2
 ; RV32I-NEXT:    and a0, a0, a2
 ; RV32I-NEXT:    slli a0, a0, 1
 ; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: test_bitreverse_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a1, a0, 8
+; RV64I-NEXT:    srli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 16
 ; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srliw a2, a0, 24
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3

llvm/test/CodeGen/RISCV/ctlz-cttz-ctpop.ll

 ; RV64I-NEXT:  # %bb.1: # %cond.false
 ; RV64I-NEXT:    addiw a1, a0, -1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    srliw a1, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    addiw a1, a0, -1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-LABEL: test_ctpop_i32:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

llvm/test/CodeGen/RISCV/rv32zbp.ll

 ; RV32I-NEXT:    lui a2, 838861
 ; RV32I-NEXT:    addi a2, a2, -820
 ; RV32I-NEXT:    and a1, a1, a2
 ; RV32I-NEXT:    srli a3, a0, 2
 ; RV32I-NEXT:    lui a4, 209715
 ; RV32I-NEXT:    addi a4, a4, 819
 ; RV32I-NEXT:    and a3, a3, a4
 ; RV32I-NEXT:    or a0, a3, a0
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    slli a1, a0, 2
-; RV32I-NEXT:    and a1, a1, a2
-; RV32I-NEXT:    srli a2, a0, 2
+; RV32I-NEXT:    or a1, a0, a1
+; RV32I-NEXT:    slli a0, a0, 2
+; RV32I-NEXT:    and a0, a0, a2
+; RV32I-NEXT:    srli a2, a1, 2
 ; RV32I-NEXT:    and a2, a2, a4
-; RV32I-NEXT:    or a0, a2, a0
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    or a1, a2, a1
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32ZBP-LABEL: gorc2b_i32:
 ; RV32ZBP:       # %bb.0:
+; RV32ZBP-NEXT:    srli a1, a0, 2
+; RV32ZBP-NEXT:    or a1, a1, a0
 ; RV32ZBP-NEXT:    orc2.n a0, a0
+; RV32ZBP-NEXT:    slli a1, a1, 2
+; RV32ZBP-NEXT:    lui a2, 838861
+; RV32ZBP-NEXT:    addi a2, a2, -820
+; RV32ZBP-NEXT:    and a1, a1, a2
+; RV32ZBP-NEXT:    srli a2, a0, 2
+; RV32ZBP-NEXT:    lui a3, 209715
+; RV32ZBP-NEXT:    addi a3, a3, 819
+; RV32ZBP-NEXT:    and a2, a2, a3
+; RV32ZBP-NEXT:    or a0, a2, a0
+; RV32ZBP-NEXT:    or a0, a0, a1
 ; RV32ZBP-NEXT:    ret
   %and1 = shl i32 %a, 2
   %shl1 = and i32 %and1, -858993460
   %and1b = lshr i32 %a, 2
   %shr1 = and i32 %and1b, 858993459
   %or1 = or i32 %shr1, %a
   %or1b = or i32 %or1, %shl1
   %and2 = shl i32 %or1b, 2
   %shl2 = and i32 %and2, -858993460
   %and2b = lshr i32 %or1b, 2
   %shr2 = and i32 %and2b, 858993459
   %or2 = or i32 %shr2, %or1b
   %or2b = or i32 %or2, %shl2
   ret i32 %or2b
 }
 
 ; gorc2, gorc2 -> gorc2
 define i64 @gorc2b_i64(i64 %a) nounwind {
 ; RV32I-LABEL: gorc2b_i64:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    slli a2, a1, 2
-; RV32I-NEXT:    slli a3, a0, 2
+; RV32I-NEXT:    slli a2, a0, 2
+; RV32I-NEXT:    slli a3, a1, 2
 ; RV32I-NEXT:    lui a4, 838861
 ; RV32I-NEXT:    addi a4, a4, -820
 ; RV32I-NEXT:    and a3, a3, a4
 ; RV32I-NEXT:    and a2, a2, a4
-; RV32I-NEXT:    srli a5, a0, 2
-; RV32I-NEXT:    srli a6, a1, 2
+; RV32I-NEXT:    srli a5, a1, 2
+; RV32I-NEXT:    srli a6, a0, 2
 ; RV32I-NEXT:    lui a7, 209715
 ; RV32I-NEXT:    addi a7, a7, 819
 ; RV32I-NEXT:    and a6, a6, a7
 ; RV32I-NEXT:    and a5, a5, a7
-; RV32I-NEXT:    or a0, a5, a0
-; RV32I-NEXT:    or a1, a6, a1
-; RV32I-NEXT:    or a1, a1, a2
-; RV32I-NEXT:    or a0, a0, a3
-; RV32I-NEXT:    slli a2, a0, 2
-; RV32I-NEXT:    slli a3, a1, 2
-; RV32I-NEXT:    and a3, a3, a4
-; RV32I-NEXT:    and a2, a2, a4
-; RV32I-NEXT:    srli a4, a1, 2
-; RV32I-NEXT:    srli a5, a0, 2
+; RV32I-NEXT:    or a1, a5, a1
+; RV32I-NEXT:    or a0, a6, a0
+; RV32I-NEXT:    or a2, a0, a2
+; RV32I-NEXT:    or a3, a1, a3
+; RV32I-NEXT:    slli a0, a0, 2
+; RV32I-NEXT:    slli a1, a1, 2
+; RV32I-NEXT:    and a1, a1, a4
+; RV32I-NEXT:    and a0, a0, a4
+; RV32I-NEXT:    srli a4, a3, 2
+; RV32I-NEXT:    srli a5, a2, 2
 ; RV32I-NEXT:    and a5, a5, a7
 ; RV32I-NEXT:    and a4, a4, a7
-; RV32I-NEXT:    or a1, a4, a1
-; RV32I-NEXT:    or a0, a5, a0
-; RV32I-NEXT:    or a0, a0, a2
-; RV32I-NEXT:    or a1, a1, a3
+; RV32I-NEXT:    or a3, a4, a3
+; RV32I-NEXT:    or a2, a5, a2
+; RV32I-NEXT:    or a0, a2, a0
+; RV32I-NEXT:    or a1, a3, a1
 ; RV32I-NEXT:    ret
 ;
 ; RV32ZBP-LABEL: gorc2b_i64:
 ; RV32ZBP:       # %bb.0:
-; RV32ZBP-NEXT:    orc2.n a0, a0
+; RV32ZBP-NEXT:    srli a2, a1, 2
+; RV32ZBP-NEXT:    srli a3, a0, 2
+; RV32ZBP-NEXT:    lui a4, 209715
+; RV32ZBP-NEXT:    addi a4, a4, 819
+; RV32ZBP-NEXT:    and a3, a3, a4
+; RV32ZBP-NEXT:    or a3, a3, a0
+; RV32ZBP-NEXT:    or a2, a2, a1
 ; RV32ZBP-NEXT:    orc2.n a1, a1
+; RV32ZBP-NEXT:    orc2.n a0, a0
+; RV32ZBP-NEXT:    slli a2, a2, 2
+; RV32ZBP-NEXT:    slli a3, a3, 2
+; RV32ZBP-NEXT:    lui a5, 838861
+; RV32ZBP-NEXT:    addi a5, a5, -820
+; RV32ZBP-NEXT:    and a3, a3, a5
+; RV32ZBP-NEXT:    and a2, a2, a5
+; RV32ZBP-NEXT:    srli a5, a0, 2
+; RV32ZBP-NEXT:    srli a6, a1, 2
+; RV32ZBP-NEXT:    and a6, a6, a4
+; RV32ZBP-NEXT:    and a4, a5, a4
+; RV32ZBP-NEXT:    or a0, a4, a0
+; RV32ZBP-NEXT:    or a1, a6, a1
+; RV32ZBP-NEXT:    or a1, a1, a2
+; RV32ZBP-NEXT:    or a0, a0, a3
 ; RV32ZBP-NEXT:    ret
   %and1 = shl i64 %a, 2
   %shl1 = and i64 %and1, -3689348814741910324
   %and1b = lshr i64 %a, 2
   %shr1 = and i64 %and1b, 3689348814741910323
   %or1 = or i64 %shr1, %a
   %or1b = or i64 %or1, %shl1
   %and2 = shl i64 %or1b, 2

 ; RV32ZBP-NEXT:    ret
   %1 = call i64 @llvm.bitreverse.i64(i64 %a)
   ret i64 %1
 }
 
 define i32 @bswap_rotr_i32(i32 %a) {
 ; RV32I-LABEL: bswap_rotr_i32:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    srli a1, a0, 8
-; RV32I-NEXT:    lui a2, 16
-; RV32I-NEXT:    addi a2, a2, -256
+; RV32I-NEXT:    slli a1, a0, 8
+; RV32I-NEXT:    lui a2, 4080
 ; RV32I-NEXT:    and a1, a1, a2
+; RV32I-NEXT:    slli a2, a0, 24
+; RV32I-NEXT:    or a1, a2, a1
 ; RV32I-NEXT:    srli a2, a0, 24
-; RV32I-NEXT:    or a1, a1, a2
-; RV32I-NEXT:    slli a2, a0, 8
-; RV32I-NEXT:    lui a3, 4080
-; RV32I-NEXT:    and a2, a2, a3
-; RV32I-NEXT:    slli a0, a0, 24
-; RV32I-NEXT:    or a0, a0, a2
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    slli a1, a0, 16
 ; RV32I-NEXT:    srli a0, a0, 16
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    slli a0, a0, 8
+; RV32I-NEXT:    or a0, a0, a2
+; RV32I-NEXT:    slli a0, a0, 16
+; RV32I-NEXT:    srli a1, a1, 16
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32ZBP-LABEL: bswap_rotr_i32:
 ; RV32ZBP:       # %bb.0:
 ; RV32ZBP-NEXT:    rev8.h a0, a0
 ; RV32ZBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshr.i32(i32 %1, i32 %1, i32 16)
   ret i32 %2
 }
 
 define i32 @bswap_rotl_i32(i32 %a) {
 ; RV32I-LABEL: bswap_rotl_i32:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    srli a1, a0, 8
-; RV32I-NEXT:    lui a2, 16
-; RV32I-NEXT:    addi a2, a2, -256
-; RV32I-NEXT:    and a1, a1, a2
-; RV32I-NEXT:    srli a2, a0, 24
-; RV32I-NEXT:    or a1, a1, a2
+; RV32I-NEXT:    srli a1, a0, 24
+; RV32I-NEXT:    srli a2, a0, 16
+; RV32I-NEXT:    slli a2, a2, 8
+; RV32I-NEXT:    or a1, a2, a1
 ; RV32I-NEXT:    slli a2, a0, 8
 ; RV32I-NEXT:    lui a3, 4080
 ; RV32I-NEXT:    and a2, a2, a3
 ; RV32I-NEXT:    slli a0, a0, 24
 ; RV32I-NEXT:    or a0, a0, a2
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    srli a1, a0, 16
-; RV32I-NEXT:    slli a0, a0, 16
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    srli a0, a0, 16
+; RV32I-NEXT:    slli a1, a1, 16
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32ZBP-LABEL: bswap_rotl_i32:
 ; RV32ZBP:       # %bb.0:
 ; RV32ZBP-NEXT:    rev8.h a0, a0
 ; RV32ZBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshl.i32(i32 %1, i32 %1, i32 16)

llvm/test/CodeGen/RISCV/rv64zbb-zbp-zbkb.ll

 
 ; This is similar to the type legalized roriw pattern, but the and mask is more
 ; than 32 bits so the lshr doesn't shift zeroes into the lower 32 bits. Make
 ; sure we don't match it to roriw.
 define i64 @roriw_bug(i64 %x) nounwind {
 ; RV64I-LABEL: roriw_bug:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a1, a0, 31
-; RV64I-NEXT:    andi a0, a0, -2
-; RV64I-NEXT:    srli a2, a0, 1
-; RV64I-NEXT:    or a1, a1, a2
-; RV64I-NEXT:    sext.w a1, a1
-; RV64I-NEXT:    xor a0, a0, a1
+; RV64I-NEXT:    andi a2, a0, -2
+; RV64I-NEXT:    srli a0, a0, 1
+; RV64I-NEXT:    or a0, a1, a0
+; RV64I-NEXT:    sext.w a0, a0
+; RV64I-NEXT:    xor a0, a2, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBB-ZBP-ZBKB-LABEL: roriw_bug:
 ; RV64ZBB-ZBP-ZBKB:       # %bb.0:
 ; RV64ZBB-ZBP-ZBKB-NEXT:    slli a1, a0, 31
-; RV64ZBB-ZBP-ZBKB-NEXT:    andi a0, a0, -2
-; RV64ZBB-ZBP-ZBKB-NEXT:    srli a2, a0, 1
-; RV64ZBB-ZBP-ZBKB-NEXT:    or a1, a1, a2
-; RV64ZBB-ZBP-ZBKB-NEXT:    sext.w a1, a1
-; RV64ZBB-ZBP-ZBKB-NEXT:    xor a0, a0, a1
+; RV64ZBB-ZBP-ZBKB-NEXT:    andi a2, a0, -2
+; RV64ZBB-ZBP-ZBKB-NEXT:    srli a0, a0, 1
+; RV64ZBB-ZBP-ZBKB-NEXT:    or a0, a1, a0
+; RV64ZBB-ZBP-ZBKB-NEXT:    sext.w a0, a0
+; RV64ZBB-ZBP-ZBKB-NEXT:    xor a0, a2, a0
 ; RV64ZBB-ZBP-ZBKB-NEXT:    ret
   %a = shl i64 %x, 31
   %b = and i64 %x, 18446744073709551614
   %c = lshr i64 %b, 1
   %d = or i64 %a, %c
   %e = shl i64 %d, 32
   %f = ashr i64 %e, 32
   %g = xor i64 %b, %f ; to increase the use count on %b to disable SimplifyDemandedBits.

llvm/test/CodeGen/RISCV/rv64zbb.ll

 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    srliw a1, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    srliw a1, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    srliw a1, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    srliw a1, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    slli a1, a0, 33
 ; RV64I-NEXT:    srli a1, a1, 49
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:  # %bb.1: # %cond.false
 ; RV64I-NEXT:    addiw a1, a0, -1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    addiw a1, a0, -1
 ; RV64I-NEXT:    not a0, a0
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    mv s0, a0
 ; RV64I-NEXT:    addiw a0, a0, -1
 ; RV64I-NEXT:    not a1, s0
 ; RV64I-NEXT:    and a0, a1, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-NEXT:    mv s0, a0
 ; RV64I-NEXT:    addiw a0, a0, -1
 ; RV64I-NEXT:    not a1, s0
 ; RV64I-NEXT:    and a0, a1, a0
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I-LABEL: ctpop_i32:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1

 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    lw a0, 0(a0)
 ; RV64I-NEXT:    srli a1, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    subw a0, a0, a1
+; RV64I-NEXT:    sub a0, a0, a1
 ; RV64I-NEXT:    lui a1, 209715
 ; RV64I-NEXT:    addiw a1, a1, 819
 ; RV64I-NEXT:    and a2, a0, a1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, a1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a1, a0, 4
 ; RV64I-NEXT:    add a0, a0, a1
; RV64ZBB-NEXT:    ret
   ret i64 %and
 }
 
 declare i32 @llvm.bswap.i32(i32)
 
 define signext i32 @bswap_i32(i32 signext %a) nounwind {
 ; RV64I-LABEL: bswap_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a1, a0, 8
+; RV64I-NEXT:    srli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 16
 ; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srliw a2, a0, 24
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3
; RV64ZBB-NEXT:    ret
   %1 = tail call i32 @llvm.bswap.i32(i32 %a)
   ret i32 %1
 }
 
 ; Similar to bswap_i32 but the result is not sign extended.
 define void @bswap_i32_nosext(i32 signext %a, i32* %x) nounwind {
 ; RV64I-LABEL: bswap_i32_nosext:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a2, a0, 8
+; RV64I-NEXT:    srli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 16
 ; RV64I-NEXT:    addiw a3, a3, -256
 ; RV64I-NEXT:    and a2, a2, a3
 ; RV64I-NEXT:    srliw a3, a0, 24
 ; RV64I-NEXT:    or a2, a2, a3
 ; RV64I-NEXT:    slli a3, a0, 8
 ; RV64I-NEXT:    lui a4, 4080
 ; RV64I-NEXT:    and a3, a3, a4

llvm/test/CodeGen/RISCV/rv64zbp.ll

 ; RV64I-NEXT:    lui a2, 838861
 ; RV64I-NEXT:    addiw a2, a2, -820
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srli a3, a0, 2
 ; RV64I-NEXT:    lui a4, 209715
 ; RV64I-NEXT:    addiw a4, a4, 819
 ; RV64I-NEXT:    and a3, a3, a4
 ; RV64I-NEXT:    or a0, a3, a0
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    slliw a1, a0, 2
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 2
+; RV64I-NEXT:    or a1, a0, a1
+; RV64I-NEXT:    slliw a0, a0, 2
+; RV64I-NEXT:    and a0, a0, a2
+; RV64I-NEXT:    srli a2, a1, 2
 ; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a0, a2, a0
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    or a1, a2, a1
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: gorc2b_i32:
 ; RV64ZBP:       # %bb.0:
-; RV64ZBP-NEXT:    gorciw a0, a0, 2
+; RV64ZBP-NEXT:    srliw a1, a0, 2
+; RV64ZBP-NEXT:    or a1, a1, a0
+; RV64ZBP-NEXT:    orc2.n a0, a0
+; RV64ZBP-NEXT:    slli a1, a1, 2
+; RV64ZBP-NEXT:    lui a2, 838861
+; RV64ZBP-NEXT:    addiw a2, a2, -820
+; RV64ZBP-NEXT:    and a1, a1, a2
+; RV64ZBP-NEXT:    srli a2, a0, 2
+; RV64ZBP-NEXT:    lui a3, 209715
+; RV64ZBP-NEXT:    addiw a3, a3, 819
+; RV64ZBP-NEXT:    and a2, a2, a3
+; RV64ZBP-NEXT:    or a0, a2, a0
+; RV64ZBP-NEXT:    or a0, a0, a1
+; RV64ZBP-NEXT:    sext.w a0, a0
 ; RV64ZBP-NEXT:    ret
   %and1 = shl i32 %a, 2
   %shl1 = and i32 %and1, -858993460
   %and1b = lshr i32 %a, 2
   %shr1 = and i32 %and1b, 858993459
   %or1 = or i32 %shr1, %a
   %or1b = or i32 %or1, %shl1
   %and2 = shl i32 %or1b, 2

 ; RV64I-NEXT:    ld a1, %lo(.LCPI26_0)(a1)
 ; RV64I-NEXT:    lui a2, %hi(.LCPI26_1)
 ; RV64I-NEXT:    ld a2, %lo(.LCPI26_1)(a2)
 ; RV64I-NEXT:    slli a3, a0, 2
 ; RV64I-NEXT:    and a3, a3, a1
 ; RV64I-NEXT:    srli a4, a0, 2
 ; RV64I-NEXT:    and a4, a4, a2
 ; RV64I-NEXT:    or a0, a4, a0
-; RV64I-NEXT:    or a0, a0, a3
-; RV64I-NEXT:    slli a3, a0, 2
-; RV64I-NEXT:    and a1, a3, a1
-; RV64I-NEXT:    srli a3, a0, 2
-; RV64I-NEXT:    and a2, a3, a2
-; RV64I-NEXT:    or a0, a2, a0
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    or a3, a0, a3
+; RV64I-NEXT:    slli a0, a0, 2
+; RV64I-NEXT:    and a0, a0, a1
+; RV64I-NEXT:    srli a1, a3, 2
+; RV64I-NEXT:    and a1, a1, a2
+; RV64I-NEXT:    or a1, a1, a3
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: gorc2b_i64:
 ; RV64ZBP:       # %bb.0:
+; RV64ZBP-NEXT:    srli a1, a0, 2
+; RV64ZBP-NEXT:    or a1, a1, a0
 ; RV64ZBP-NEXT:    orc2.n a0, a0
+; RV64ZBP-NEXT:    lui a2, %hi(.LCPI26_0)
+; RV64ZBP-NEXT:    ld a2, %lo(.LCPI26_0)(a2)
+; RV64ZBP-NEXT:    lui a3, %hi(.LCPI26_1)
+; RV64ZBP-NEXT:    ld a3, %lo(.LCPI26_1)(a3)
+; RV64ZBP-NEXT:    slli a1, a1, 2
+; RV64ZBP-NEXT:    and a1, a1, a2
+; RV64ZBP-NEXT:    srli a2, a0, 2
+; RV64ZBP-NEXT:    and a2, a2, a3
+; RV64ZBP-NEXT:    or a0, a2, a0
+; RV64ZBP-NEXT:    or a0, a0, a1
 ; RV64ZBP-NEXT:    ret
   %and1 = shl i64 %a, 2
   %shl1 = and i64 %and1, -3689348814741910324
   %and1b = lshr i64 %a, 2
   %shr1 = and i64 %and1b, 3689348814741910323
   %or1 = or i64 %shr1, %a
   %or1b = or i64 %or1, %shl1
   %and2 = shl i64 %or1b, 2
; RV64ZBP-NEXT:    ret
   ret i16 %1
 }
 
 declare i32 @llvm.bswap.i32(i32)
 
 define signext i32 @bswap_i32(i32 signext %a) nounwind {
 ; RV64I-LABEL: bswap_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a1, a0, 8
+; RV64I-NEXT:    srli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 16
 ; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srliw a2, a0, 24
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3
; RV64ZBP-NEXT:    ret
   %1 = tail call i32 @llvm.bswap.i32(i32 %a)
   ret i32 %1
 }
 
 ; Similar to bswap_i32 but the result is not sign extended.
 define void @bswap_i32_nosext(i32 signext %a, i32* %x) nounwind {
 ; RV64I-LABEL: bswap_i32_nosext:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a2, a0, 8
+; RV64I-NEXT:    srli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 16
 ; RV64I-NEXT:    addiw a3, a3, -256
 ; RV64I-NEXT:    and a2, a2, a3
 ; RV64I-NEXT:    srliw a3, a0, 24
 ; RV64I-NEXT:    or a2, a2, a3
 ; RV64I-NEXT:    slli a3, a0, 8
 ; RV64I-NEXT:    lui a4, 4080
 ; RV64I-NEXT:    and a3, a3, a4
; RV64ZBP-NEXT:    ret
   ret i16 %1
 }
 
 declare i32 @llvm.bitreverse.i32(i32)
 
 define signext i32 @bitreverse_i32(i32 signext %a) nounwind {
 ; RV64I-LABEL: bitreverse_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a1, a0, 8
+; RV64I-NEXT:    srli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 16
 ; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srliw a2, a0, 24
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3
; RV64ZBP-NEXT:    ret
   %1 = tail call i32 @llvm.bitreverse.i32(i32 %a)
   ret i32 %1
 }
 
 ; Similar to bitreverse_i32 but the result is not sign extended.
 define void @bitreverse_i32_nosext(i32 signext %a, i32* %x) nounwind {
 ; RV64I-LABEL: bitreverse_i32_nosext:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srliw a2, a0, 8
+; RV64I-NEXT:    srli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 16
 ; RV64I-NEXT:    addiw a3, a3, -256
 ; RV64I-NEXT:    and a2, a2, a3
 ; RV64I-NEXT:    srliw a3, a0, 24
 ; RV64I-NEXT:    or a2, a2, a3
 ; RV64I-NEXT:    slli a3, a0, 8
 ; RV64I-NEXT:    lui a4, 4080
 ; RV64I-NEXT:    and a3, a3, a4

 ; RV64I-LABEL: bswap_rotr_i32:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a1, a0, 8
 ; RV64I-NEXT:    lui a2, 4080
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    slli a2, a0, 24
 ; RV64I-NEXT:    or a1, a2, a1
 ; RV64I-NEXT:    srliw a2, a0, 24
-; RV64I-NEXT:    srliw a0, a0, 16
+; RV64I-NEXT:    srli a0, a0, 16
 ; RV64I-NEXT:    slli a0, a0, 8
 ; RV64I-NEXT:    or a0, a0, a2
 ; RV64I-NEXT:    slliw a0, a0, 16
 ; RV64I-NEXT:    srliw a1, a1, 16
 ; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: bswap_rotr_i32:
 ; RV64ZBP:       # %bb.0:
 ; RV64ZBP-NEXT:    greviw a0, a0, 8
 ; RV64ZBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshr.i32(i32 %1, i32 %1, i32 16)
   ret i32 %2
 }
 
 define i32 @bswap_rotl_i32(i32 %a) {
 ; RV64I-LABEL: bswap_rotl_i32:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    srliw a1, a0, 24
-; RV64I-NEXT:    srliw a2, a0, 16
+; RV64I-NEXT:    srli a2, a0, 16
 ; RV64I-NEXT:    slli a2, a2, 8
 ; RV64I-NEXT:    or a1, a2, a1
 ; RV64I-NEXT:    slli a2, a0, 8
 ; RV64I-NEXT:    lui a3, 4080
 ; RV64I-NEXT:    and a2, a2, a3
 ; RV64I-NEXT:    slli a0, a0, 24
 ; RV64I-NEXT:    or a0, a0, a2
 ; RV64I-NEXT:    srliw a0, a0, 16
 ; RV64I-NEXT:    slliw a1, a1, 16
 ; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: bswap_rotl_i32:
 ; RV64ZBP:       # %bb.0:
 ; RV64ZBP-NEXT:    greviw a0, a0, 8
 ; RV64ZBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshl.i32(i32 %1, i32 %1, i32 16)
   ret i32 %2
 }
 
 define i64 @bswap_rotr_i64(i64 %a) {
 ; RV64I-LABEL: bswap_rotr_i64:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srli a1, a0, 24
-; RV64I-NEXT:    lui a2, 4080
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 8
-; RV64I-NEXT:    li a3, 255
-; RV64I-NEXT:    slli a4, a3, 24
-; RV64I-NEXT:    and a2, a2, a4
+; RV64I-NEXT:    slli a1, a0, 24
+; RV64I-NEXT:    li a2, 255
+; RV64I-NEXT:    slli a3, a2, 40
+; RV64I-NEXT:    and a1, a1, a3
+; RV64I-NEXT:    srliw a3, a0, 24
+; RV64I-NEXT:    slli a3, a3, 32
+; RV64I-NEXT:    or a1, a1, a3
+; RV64I-NEXT:    slli a3, a0, 40
+; RV64I-NEXT:    slli a2, a2, 48
+; RV64I-NEXT:    and a2, a3, a2
+; RV64I-NEXT:    slli a3, a0, 56
+; RV64I-NEXT:    or a2, a3, a2
 ; RV64I-NEXT:    or a1, a2, a1
 ; RV64I-NEXT:    srli a2, a0, 40
-; RV64I-NEXT:    lui a4, 16
-; RV64I-NEXT:    addiw a4, a4, -256
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    srli a4, a0, 56
-; RV64I-NEXT:    or a2, a2, a4
-; RV64I-NEXT:    or a1, a1, a2
-; RV64I-NEXT:    slli a2, a0, 24
-; RV64I-NEXT:    slli a4, a3, 40
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    srliw a4, a0, 24
-; RV64I-NEXT:    slli a4, a4, 32
-; RV64I-NEXT:    or a2, a2, a4
-; RV64I-NEXT:    slli a4, a0, 40
-; RV64I-NEXT:    slli a3, a3, 48
-; RV64I-NEXT:    and a3, a4, a3
-; RV64I-NEXT:    slli a0, a0, 56
+; RV64I-NEXT:    lui a3, 16
+; RV64I-NEXT:    addiw a3, a3, -256
+; RV64I-NEXT:    and a2, a2, a3
+; RV64I-NEXT:    srli a3, a0, 56
+; RV64I-NEXT:    or a2, a2, a3
+; RV64I-NEXT:    srli a3, a0, 24
+; RV64I-NEXT:    lui a4, 4080
+; RV64I-NEXT:    and a3, a3, a4
+; RV64I-NEXT:    srli a0, a0, 32
+; RV64I-NEXT:    slli a0, a0, 24
 ; RV64I-NEXT:    or a0, a0, a3
 ; RV64I-NEXT:    or a0, a0, a2
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    slli a1, a0, 32
-; RV64I-NEXT:    srli a0, a0, 32
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    slli a0, a0, 32
+; RV64I-NEXT:    srli a1, a1, 32
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: bswap_rotr_i64:
 ; RV64ZBP:       # %bb.0:
 ; RV64ZBP-NEXT:    rev8.w a0, a0
 ; RV64ZBP-NEXT:    ret
   %1 = call i64 @llvm.bswap.i64(i64 %a)
   %2 = call i64 @llvm.fshr.i64(i64 %1, i64 %1, i64 32)
   ret i64 %2
 }
 
 define i64 @bswap_rotl_i64(i64 %a) {
 ; RV64I-LABEL: bswap_rotl_i64:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srli a1, a0, 24
-; RV64I-NEXT:    lui a2, 4080
+; RV64I-NEXT:    srli a1, a0, 40
+; RV64I-NEXT:    lui a2, 16
+; RV64I-NEXT:    addiw a2, a2, -256
 ; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 8
-; RV64I-NEXT:    li a3, 255
-; RV64I-NEXT:    slli a4, a3, 24
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a1, a2, a1
-; RV64I-NEXT:    srli a2, a0, 40
-; RV64I-NEXT:    lui a4, 16
-; RV64I-NEXT:    addiw a4, a4, -256
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    srli a4, a0, 56
-; RV64I-NEXT:    or a2, a2, a4
+; RV64I-NEXT:    srli a2, a0, 56
 ; RV64I-NEXT:    or a1, a1, a2
+; RV64I-NEXT:    srli a2, a0, 24
+; RV64I-NEXT:    lui a3, 4080
+; RV64I-NEXT:    and a2, a2, a3
+; RV64I-NEXT:    srli a3, a0, 32
+; RV64I-NEXT:    slli a3, a3, 24
+; RV64I-NEXT:    or a2, a3, a2
+; RV64I-NEXT:    or a1, a2, a1
 ; RV64I-NEXT:    slli a2, a0, 24
+; RV64I-NEXT:    li a3, 255
 ; RV64I-NEXT:    slli a4, a3, 40
 ; RV64I-NEXT:    and a2, a2, a4
 ; RV64I-NEXT:    srliw a4, a0, 24
 ; RV64I-NEXT:    slli a4, a4, 32
 ; RV64I-NEXT:    or a2, a2, a4
 ; RV64I-NEXT:    slli a4, a0, 40
 ; RV64I-NEXT:    slli a3, a3, 48
 ; RV64I-NEXT:    and a3, a4, a3
 ; RV64I-NEXT:    slli a0, a0, 56
 ; RV64I-NEXT:    or a0, a0, a3
 ; RV64I-NEXT:    or a0, a0, a2
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    srli a1, a0, 32
-; RV64I-NEXT:    slli a0, a0, 32
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    srli a0, a0, 32
+; RV64I-NEXT:    slli a1, a1, 32
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64ZBP-LABEL: bswap_rotl_i64:
 ; RV64ZBP:       # %bb.0:
 ; RV64ZBP-NEXT:    rev8.w a0, a0
 ; RV64ZBP-NEXT:    ret
   %1 = call i64 @llvm.bswap.i64(i64 %a)
   %2 = call i64 @llvm.fshl.i64(i64 %1, i64 %1, i64 32)

llvm/test/CodeGen/RISCV/sextw-removal.ll

 ; RV64I-NEXT:    lui a1, 4112
 ; RV64I-NEXT:    addiw s3, a1, 257
 ; RV64I-NEXT:  .LBB4_1: # %bb2
 ; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
 ; RV64I-NEXT:    call bar@plt
 ; RV64I-NEXT:    mv a1, a0
 ; RV64I-NEXT:    srli a0, a0, 1
 ; RV64I-NEXT:    and a0, a0, s0
-; RV64I-NEXT:    subw a0, a1, a0
+; RV64I-NEXT:    sub a0, a1, a0
 ; RV64I-NEXT:    and a2, a0, s1
 ; RV64I-NEXT:    srli a0, a0, 2
 ; RV64I-NEXT:    and a0, a0, s1
 ; RV64I-NEXT:    add a0, a2, a0
 ; RV64I-NEXT:    srli a2, a0, 4
 ; RV64I-NEXT:    add a0, a0, a2
 ; RV64I-NEXT:    and a0, a0, s2
 ; RV64I-NEXT:    mulw a0, a0, s3

llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll

 ; RV32-NEXT:    sw s1, 20(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s2, 16(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s3, 12(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s4, 8(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s5, 4(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s6, 0(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    mv s0, a0
 ; RV32-NEXT:    lw a0, 4(a0)
-; RV32-NEXT:    lbu a1, 12(s0)
+; RV32-NEXT:    lb a1, 12(s0)
 ; RV32-NEXT:    lw a2, 8(s0)
 ; RV32-NEXT:    andi a3, a0, 1
 ; RV32-NEXT:    neg s1, a3
 ; RV32-NEXT:    slli a3, a1, 30
 ; RV32-NEXT:    srli a4, a2, 2
 ; RV32-NEXT:    or s2, a4, a3
 ; RV32-NEXT:    srli a1, a1, 2
 ; RV32-NEXT:    andi a1, a1, 1

 ; RV64-NEXT:    sd s1, 24(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    sd s2, 16(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    sd s3, 8(sp) # 8-byte Folded Spill
 ; RV64-NEXT:    mv s0, a0
 ; RV64-NEXT:    lb a0, 12(a0)
 ; RV64-NEXT:    lwu a1, 8(s0)
 ; RV64-NEXT:    slli a0, a0, 32
 ; RV64-NEXT:    or a0, a1, a0
-; RV64-NEXT:    slli a0, a0, 24
-; RV64-NEXT:    srli a0, a0, 24
-; RV64-NEXT:    ld a1, 0(s0)
-; RV64-NEXT:    slli a2, a0, 29
-; RV64-NEXT:    srai s1, a2, 31
-; RV64-NEXT:    slli a0, a0, 31
-; RV64-NEXT:    srli a2, a1, 33
-; RV64-NEXT:    or a0, a2, a0
+; RV64-NEXT:    ld a2, 0(s0)
+; RV64-NEXT:    slli a0, a0, 29
+; RV64-NEXT:    srai s1, a0, 31
+; RV64-NEXT:    slli a0, a1, 31
+; RV64-NEXT:    srli a1, a2, 33
+; RV64-NEXT:    or a0, a1, a0
 ; RV64-NEXT:    slli a0, a0, 31
 ; RV64-NEXT:    srai a0, a0, 31
-; RV64-NEXT:    slli a1, a1, 31
+; RV64-NEXT:    slli a1, a2, 31
 ; RV64-NEXT:    srai s2, a1, 31
 ; RV64-NEXT:    li a1, 7
 ; RV64-NEXT:    call __moddi3@plt
 ; RV64-NEXT:    mv s3, a0
 ; RV64-NEXT:    li a1, -5
 ; RV64-NEXT:    mv a0, s1
 ; RV64-NEXT:    call __moddi3@plt
 ; RV64-NEXT:    mv s1, a0
 ; RV64-NEXT:    lui a0, %hi(.LCPI3_0)
 ; RV64-NEXT:    ld a1, %lo(.LCPI3_0)(a0)
 ; RV64-NEXT:    mv a0, s2
 ; RV64-NEXT:    call __muldi3@plt
 ; RV64-NEXT:    lui a1, %hi(.LCPI3_1)
 ; RV64-NEXT:    ld a1, %lo(.LCPI3_1)(a1)
 ; RV64-NEXT:    add a0, a0, a1
 ; RV64-NEXT:    slli a2, a0, 63
 ; RV64-NEXT:    srli a0, a0, 1
 ; RV64-NEXT:    or a0, a0, a2
 ; RV64-NEXT:    sltu a0, a1, a0
 ; RV64-NEXT:    addi a1, s1, -2
 ; RV64-NEXT:    snez a1, a1
 ; RV64-NEXT:    addi a2, s3, -1
 ; RV64-NEXT:    snez a2, a2
 ; RV64-NEXT:    neg a0, a0
-; RV64-NEXT:    neg a2, a2
-; RV64-NEXT:    neg a3, a1
-; RV64-NEXT:    slli a3, a3, 29
-; RV64-NEXT:    srli a3, a3, 61
-; RV64-NEXT:    sb a3, 12(s0)
+; RV64-NEXT:    neg a3, a2
+; RV64-NEXT:    neg a4, a1
+; RV64-NEXT:    slli a4, a4, 29
+; RV64-NEXT:    srli a4, a4, 61
+; RV64-NEXT:    sb a4, 12(s0)
 ; RV64-NEXT:    slliw a1, a1, 2
-; RV64-NEXT:    li a3, -1
-; RV64-NEXT:    srli a3, a3, 31
-; RV64-NEXT:    and a2, a2, a3
-; RV64-NEXT:    srli a4, a2, 31
-; RV64-NEXT:    subw a1, a4, a1
+; RV64-NEXT:    slli a3, a3, 31
+; RV64-NEXT:    srli a3, a3, 62
+; RV64-NEXT:    subw a1, a3, a1
 ; RV64-NEXT:    sw a1, 8(s0)
-; RV64-NEXT:    and a0, a0, a3
+; RV64-NEXT:    slli a0, a0, 31
+; RV64-NEXT:    srli a0, a0, 31
 ; RV64-NEXT:    slli a1, a2, 33
-; RV64-NEXT:    or a0, a0, a1
+; RV64-NEXT:    sub a0, a0, a1
 ; RV64-NEXT:    sd a0, 0(s0)
 ; RV64-NEXT:    ld ra, 40(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    ld s0, 32(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    ld s1, 24(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    ld s2, 16(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    ld s3, 8(sp) # 8-byte Folded Reload
 ; RV64-NEXT:    addi sp, sp, 48
 ; RV64-NEXT:    ret
 ;
 ; RV32M-LABEL: test_srem_vec:
 ; RV32M:       # %bb.0:
 ; RV32M-NEXT:    addi sp, sp, -32
 ; RV32M-NEXT:    sw ra, 28(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s0, 24(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s1, 20(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s2, 16(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s3, 12(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s4, 8(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s5, 4(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s6, 0(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    mv s0, a0
 ; RV32M-NEXT:    lw a0, 4(a0)
-; RV32M-NEXT:    lbu a1, 12(s0)
+; RV32M-NEXT:    lb a1, 12(s0)
 ; RV32M-NEXT:    lw a2, 8(s0)
 ; RV32M-NEXT:    andi a3, a0, 1
 ; RV32M-NEXT:    neg s1, a3
 ; RV32M-NEXT:    slli a3, a1, 30
 ; RV32M-NEXT:    srli a4, a2, 2
 ; RV32M-NEXT:    or s2, a4, a3
 ; RV32M-NEXT:    srli a1, a1, 2
 ; RV32M-NEXT:    andi a1, a1, 1

 ; RV32M-NEXT:    ret
 ;
 ; RV64M-LABEL: test_srem_vec:
 ; RV64M:       # %bb.0:
 ; RV64M-NEXT:    lb a1, 12(a0)
 ; RV64M-NEXT:    lwu a2, 8(a0)
 ; RV64M-NEXT:    slli a1, a1, 32
 ; RV64M-NEXT:    or a1, a2, a1
-; RV64M-NEXT:    slli a1, a1, 24
-; RV64M-NEXT:    srli a1, a1, 24
-; RV64M-NEXT:    ld a2, 0(a0)
-; RV64M-NEXT:    slli a3, a1, 29
-; RV64M-NEXT:    srai a3, a3, 31
-; RV64M-NEXT:    slli a1, a1, 31
-; RV64M-NEXT:    srli a4, a2, 33
+; RV64M-NEXT:    ld a3, 0(a0)
+; RV64M-NEXT:    slli a1, a1, 29
+; RV64M-NEXT:    srai a1, a1, 31
+; RV64M-NEXT:    slli a2, a2, 31
+; RV64M-NEXT:    srli a4, a3, 33
 ; RV64M-NEXT:    lui a5, %hi(.LCPI3_0)
 ; RV64M-NEXT:    ld a5, %lo(.LCPI3_0)(a5)
-; RV64M-NEXT:    or a1, a4, a1
-; RV64M-NEXT:    slli a1, a1, 31
-; RV64M-NEXT:    srai a1, a1, 31
-; RV64M-NEXT:    mulh a4, a1, a5
+; RV64M-NEXT:    or a2, a4, a2
+; RV64M-NEXT:    slli a2, a2, 31
+; RV64M-NEXT:    srai a2, a2, 31
+; RV64M-NEXT:    mulh a4, a2, a5
 ; RV64M-NEXT:    srli a5, a4, 63
 ; RV64M-NEXT:    srai a4, a4, 1
 ; RV64M-NEXT:    add a4, a4, a5
 ; RV64M-NEXT:    slli a5, a4, 3
 ; RV64M-NEXT:    sub a4, a4, a5
 ; RV64M-NEXT:    lui a5, %hi(.LCPI3_1)
 ; RV64M-NEXT:    ld a5, %lo(.LCPI3_1)(a5)
-; RV64M-NEXT:    slli a2, a2, 31
-; RV64M-NEXT:    srai a2, a2, 31
-; RV64M-NEXT:    add a1, a1, a4
-; RV64M-NEXT:    mulh a4, a3, a5
+; RV64M-NEXT:    slli a3, a3, 31
+; RV64M-NEXT:    srai a3, a3, 31
+; RV64M-NEXT:    add a2, a2, a4
+; RV64M-NEXT:    mulh a4, a1, a5
 ; RV64M-NEXT:    srli a5, a4, 63
 ; RV64M-NEXT:    srai a4, a4, 1
 ; RV64M-NEXT:    add a4, a4, a5
 ; RV64M-NEXT:    slli a5, a4, 2
 ; RV64M-NEXT:    add a4, a5, a4
-; RV64M-NEXT:    add a3, a3, a4
-; RV64M-NEXT:    addi a3, a3, -2
-; RV64M-NEXT:    snez a3, a3
+; RV64M-NEXT:    add a1, a1, a4
+; RV64M-NEXT:    addi a1, a1, -2
+; RV64M-NEXT:    snez a1, a1
 ; RV64M-NEXT:    lui a4, %hi(.LCPI3_2)
 ; RV64M-NEXT:    ld a4, %lo(.LCPI3_2)(a4)
 ; RV64M-NEXT:    lui a5, %hi(.LCPI3_3)
 ; RV64M-NEXT:    ld a5, %lo(.LCPI3_3)(a5)
-; RV64M-NEXT:    addi a1, a1, -1
-; RV64M-NEXT:    snez a1, a1
-; RV64M-NEXT:    mul a2, a2, a4
-; RV64M-NEXT:    add a2, a2, a5
-; RV64M-NEXT:    slli a4, a2, 63
-; RV64M-NEXT:    srli a2, a2, 1
-; RV64M-NEXT:    or a2, a2, a4
-; RV64M-NEXT:    sltu a2, a5, a2
-; RV64M-NEXT:    neg a1, a1
-; RV64M-NEXT:    neg a4, a3
-; RV64M-NEXT:    neg a2, a2
-; RV64M-NEXT:    slli a4, a4, 29
-; RV64M-NEXT:    srli a4, a4, 61
-; RV64M-NEXT:    sb a4, 12(a0)
-; RV64M-NEXT:    slliw a3, a3, 2
-; RV64M-NEXT:    li a4, -1
-; RV64M-NEXT:    srli a4, a4, 31
-; RV64M-NEXT:    and a1, a1, a4
-; RV64M-NEXT:    srli a5, a1, 31
-; RV64M-NEXT:    subw a3, a5, a3
-; RV64M-NEXT:    sw a3, 8(a0)
-; RV64M-NEXT:    slli a1, a1, 33
-; RV64M-NEXT:    and a2, a2, a4
-; RV64M-NEXT:    or a1, a2, a1
-; RV64M-NEXT:    sd a1, 0(a0)
+; RV64M-NEXT:    addi a2, a2, -1
+; RV64M-NEXT:    snez a2, a2
+; RV64M-NEXT:    mul a3, a3, a4
+; RV64M-NEXT:    add a3, a3, a5
+; RV64M-NEXT:    slli a4, a3, 63
+; RV64M-NEXT:    srli a3, a3, 1
+; RV64M-NEXT:    or a3, a3, a4
+; RV64M-NEXT:    sltu a3, a5, a3
+; RV64M-NEXT:    neg a4, a2
+; RV64M-NEXT:    neg a5, a1
+; RV64M-NEXT:    neg a3, a3
+; RV64M-NEXT:    slli a2, a2, 33
+; RV64M-NEXT:    slli a3, a3, 31
+; RV64M-NEXT:    srli a3, a3, 31
+; RV64M-NEXT:    sub a2, a3, a2
+; RV64M-NEXT:    sd a2, 0(a0)
+; RV64M-NEXT:    slli a2, a5, 29
+; RV64M-NEXT:    srli a2, a2, 61
+; RV64M-NEXT:    sb a2, 12(a0)
+; RV64M-NEXT:    slliw a1, a1, 2
+; RV64M-NEXT:    slli a2, a4, 31
+; RV64M-NEXT:    srli a2, a2, 62
+; RV64M-NEXT:    subw a1, a2, a1
+; RV64M-NEXT:    sw a1, 8(a0)
 ; RV64M-NEXT:    ret
 ;
 ; RV32MV-LABEL: test_srem_vec:
 ; RV32MV:       # %bb.0:
 ; RV32MV-NEXT:    addi sp, sp, -96
 ; RV32MV-NEXT:    sw ra, 92(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s0, 88(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s2, 84(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s3, 80(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s4, 76(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s5, 72(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    sw s6, 68(sp) # 4-byte Folded Spill
 ; RV32MV-NEXT:    addi s0, sp, 96
 ; RV32MV-NEXT:    andi sp, sp, -32
 ; RV32MV-NEXT:    mv s2, a0
 ; RV32MV-NEXT:    lw a0, 8(a0)
 ; RV32MV-NEXT:    lw a1, 4(s2)
 ; RV32MV-NEXT:    slli a2, a0, 31
 ; RV32MV-NEXT:    srli a3, a1, 1
 ; RV32MV-NEXT:    or s3, a3, a2
-; RV32MV-NEXT:    lbu a2, 12(s2)
+; RV32MV-NEXT:    lb a2, 12(s2)
 ; RV32MV-NEXT:    srli a3, a0, 1
 ; RV32MV-NEXT:    andi a3, a3, 1
 ; RV32MV-NEXT:    neg s4, a3
 ; RV32MV-NEXT:    slli a3, a2, 30
 ; RV32MV-NEXT:    srli a0, a0, 2
 ; RV32MV-NEXT:    or s5, a0, a3
 ; RV32MV-NEXT:    srli a0, a2, 2
 ; RV32MV-NEXT:    andi a2, a0, 1

 ;
 ; RV64MV-LABEL: test_srem_vec:
 ; RV64MV:       # %bb.0:
 ; RV64MV-NEXT:    addi sp, sp, -96
 ; RV64MV-NEXT:    sd ra, 88(sp) # 8-byte Folded Spill
 ; RV64MV-NEXT:    sd s0, 80(sp) # 8-byte Folded Spill
 ; RV64MV-NEXT:    addi s0, sp, 96
 ; RV64MV-NEXT:    andi sp, sp, -32
-; RV64MV-NEXT:    lb a1, 12(a0)
-; RV64MV-NEXT:    lwu a2, 8(a0)
-; RV64MV-NEXT:    slli a1, a1, 32
-; RV64MV-NEXT:    or a1, a2, a1
+; RV64MV-NEXT:    lwu a1, 8(a0)
 ; RV64MV-NEXT:    ld a2, 0(a0)
-; RV64MV-NEXT:    slli a1, a1, 24
-; RV64MV-NEXT:    srli a1, a1, 24
 ; RV64MV-NEXT:    slli a3, a1, 31
 ; RV64MV-NEXT:    srli a4, a2, 33
+; RV64MV-NEXT:    lb a5, 12(a0)
 ; RV64MV-NEXT:    or a3, a4, a3
 ; RV64MV-NEXT:    slli a3, a3, 31
 ; RV64MV-NEXT:    srai a3, a3, 31
+; RV64MV-NEXT:    slli a4, a5, 32
+; RV64MV-NEXT:    or a1, a1, a4
 ; RV64MV-NEXT:    lui a4, %hi(.LCPI3_0)
 ; RV64MV-NEXT:    ld a4, %lo(.LCPI3_0)(a4)
 ; RV64MV-NEXT:    slli a1, a1, 29
 ; RV64MV-NEXT:    slli a2, a2, 31
 ; RV64MV-NEXT:    srai a2, a2, 31
 ; RV64MV-NEXT:    mulh a4, a2, a4
 ; RV64MV-NEXT:    srli a5, a4, 63
 ; RV64MV-NEXT:    add a4, a4, a5

 ; RV64MV-NEXT:    vmv.v.i v8, 0
 ; RV64MV-NEXT:    vmerge.vim v8, v8, -1, v0
 ; RV64MV-NEXT:    vsetivli zero, 1, e64, m2, ta, mu
 ; RV64MV-NEXT:    vslidedown.vi v10, v8, 2
 ; RV64MV-NEXT:    vmv.x.s a2, v10
 ; RV64MV-NEXT:    srli a3, a2, 30
 ; RV64MV-NEXT:    andi a3, a3, 7
 ; RV64MV-NEXT:    sb a3, 12(a0)
-; RV64MV-NEXT:    slli a2, a2, 2
-; RV64MV-NEXT:    vslidedown.vi v10, v8, 1
-; RV64MV-NEXT:    vmv.x.s a3, v10
-; RV64MV-NEXT:    and a3, a3, a1
-; RV64MV-NEXT:    srli a4, a3, 31
-; RV64MV-NEXT:    or a2, a4, a2
-; RV64MV-NEXT:    sw a2, 8(a0)
-; RV64MV-NEXT:    vmv.x.s a2, v8
-; RV64MV-NEXT:    and a1, a2, a1
-; RV64MV-NEXT:    slli a2, a3, 33
-; RV64MV-NEXT:    or a1, a1, a2
+; RV64MV-NEXT:    vmv.x.s a3, v8
+; RV64MV-NEXT:    and a1, a3, a1
+; RV64MV-NEXT:    vslidedown.vi v8, v8, 1
+; RV64MV-NEXT:    vmv.x.s a3, v8
+; RV64MV-NEXT:    slli a4, a3, 33
+; RV64MV-NEXT:    or a1, a1, a4
 ; RV64MV-NEXT:    sd a1, 0(a0)
+; RV64MV-NEXT:    slli a1, a2, 2
+; RV64MV-NEXT:    slli a2, a3, 31
+; RV64MV-NEXT:    srli a2, a2, 62
+; RV64MV-NEXT:    or a1, a2, a1
+; RV64MV-NEXT:    sw a1, 8(a0)
 ; RV64MV-NEXT:    addi sp, s0, -96
 ; RV64MV-NEXT:    ld ra, 88(sp) # 8-byte Folded Reload
 ; RV64MV-NEXT:    ld s0, 80(sp) # 8-byte Folded Reload
 ; RV64MV-NEXT:    addi sp, sp, 96
 ; RV64MV-NEXT:    ret
   %ld = load <3 x i33>, <3 x i33>* %X
   %srem = srem <3 x i33> %ld, <i33 6, i33 7, i33 -5>
   %cmp = icmp ne <3 x i33> %srem, <i33 0, i33 1, i33 2>
   %ext = sext <3 x i1> %cmp to <3 x i33>
   store <3 x i33> %ext, <3 x i33>* %X
   ret void
 }

llvm/test/CodeGen/RISCV/vec3-setcc-crash.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -mattr=+v -verify-machineinstrs < %s \
 ; RUN:     | FileCheck %s --check-prefix=RV32
 ; RUN: llc -mtriple=riscv64 -mattr=+v -verify-machineinstrs < %s \
 ; RUN:     | FileCheck %s --check-prefix=RV64
 
 ; This test would lead one of the DAGCombiner's visitVSELECT optimizations to
 ; call getSetCCResultType, from which we'd return an invalid MVT (<3 x i1>)
 ; upon seeing that the V extension is enabled. The invalid MVT has a null
 ; Type*, which then segfaulted when accessed (as an EVT).
 define void @vec3_setcc_crash(<3 x i8>* %in, <3 x i8>* %out) {
 ; RV32-LABEL: vec3_setcc_crash:
 ; RV32:       # %bb.0:
 ; RV32-NEXT:    lw a0, 0(a0)
-; RV32-NEXT:    lui a2, 16
-; RV32-NEXT:    addi a2, a2, -256
-; RV32-NEXT:    and a2, a0, a2
-; RV32-NEXT:    slli a3, a2, 16
+; RV32-NEXT:    slli a2, a0, 8
+; RV32-NEXT:    slli a3, a0, 24
+; RV32-NEXT:    slli a4, a0, 16
+; RV32-NEXT:    srai a5, a4, 24
 ; RV32-NEXT:    srai a3, a3, 24
-; RV32-NEXT:    slli a4, a0, 24
-; RV32-NEXT:    srai a6, a4, 24
-; RV32-NEXT:    slli a4, a0, 8
-; RV32-NEXT:    mv a5, a0
-; RV32-NEXT:    bgtz a6, .LBB0_2
+; RV32-NEXT:    bgtz a5, .LBB0_2
 ; RV32-NEXT:  # %bb.1:
 ; RV32-NEXT:    li a5, 0
+; RV32-NEXT:    j .LBB0_3
 ; RV32-NEXT:  .LBB0_2:
-; RV32-NEXT:    srai a4, a4, 24
-; RV32-NEXT:    andi a5, a5, 255
-; RV32-NEXT:    bgtz a3, .LBB0_4
-; RV32-NEXT:  # %bb.3:
-; RV32-NEXT:    li a2, 0
-; RV32-NEXT:    j .LBB0_5
-; RV32-NEXT:  .LBB0_4:
-; RV32-NEXT:    srli a2, a2, 8
+; RV32-NEXT:    srli a5, a4, 24
+; RV32-NEXT:  .LBB0_3:
+; RV32-NEXT:    srai a4, a2, 24
+; RV32-NEXT:    slli a2, a5, 8
+; RV32-NEXT:    mv a5, a0
+; RV32-NEXT:    bgtz a3, .LBB0_5
+; RV32-NEXT:  # %bb.4:
+; RV32-NEXT:    li a5, 0
 ; RV32-NEXT:  .LBB0_5:
-; RV32-NEXT:    slli a2, a2, 8
-; RV32-NEXT:    or a2, a5, a2
+; RV32-NEXT:    andi a3, a5, 255
+; RV32-NEXT:    or a2, a3, a2
 ; RV32-NEXT:    bgtz a4, .LBB0_7
 ; RV32-NEXT:  # %bb.6:
 ; RV32-NEXT:    li a0, 0
 ; RV32-NEXT:    j .LBB0_8
 ; RV32-NEXT:  .LBB0_7:
 ; RV32-NEXT:    srli a0, a0, 16
 ; RV32-NEXT:  .LBB0_8:
 ; RV32-NEXT:    sb a0, 2(a1)
 ; RV32-NEXT:    sh a2, 0(a1)
 ; RV32-NEXT:    ret
 ;
 ; RV64-LABEL: vec3_setcc_crash:
 ; RV64:       # %bb.0:
 ; RV64-NEXT:    lwu a0, 0(a0)
-; RV64-NEXT:    lui a2, 16
-; RV64-NEXT:    addiw a2, a2, -256
-; RV64-NEXT:    and a2, a0, a2
-; RV64-NEXT:    slli a3, a2, 48
+; RV64-NEXT:    slli a2, a0, 40
+; RV64-NEXT:    slli a3, a0, 56
+; RV64-NEXT:    slli a4, a0, 48
+; RV64-NEXT:    srai a5, a4, 56
 ; RV64-NEXT:    srai a3, a3, 56
-; RV64-NEXT:    slli a4, a0, 56
-; RV64-NEXT:    srai a6, a4, 56
-; RV64-NEXT:    slli a4, a0, 40
-; RV64-NEXT:    mv a5, a0
-; RV64-NEXT:    bgtz a6, .LBB0_2
+; RV64-NEXT:    bgtz a5, .LBB0_2
 ; RV64-NEXT:  # %bb.1:
 ; RV64-NEXT:    li a5, 0
+; RV64-NEXT:    j .LBB0_3
 ; RV64-NEXT:  .LBB0_2:
-; RV64-NEXT:    srai a4, a4, 56
-; RV64-NEXT:    andi a5, a5, 255
-; RV64-NEXT:    bgtz a3, .LBB0_4
-; RV64-NEXT:  # %bb.3:
-; RV64-NEXT:    li a2, 0
-; RV64-NEXT:    j .LBB0_5
-; RV64-NEXT:  .LBB0_4:
-; RV64-NEXT:    srli a2, a2, 8
+; RV64-NEXT:    srli a5, a4, 56
+; RV64-NEXT:  .LBB0_3:
+; RV64-NEXT:    srai a4, a2, 56
+; RV64-NEXT:    slli a2, a5, 8
+; RV64-NEXT:    mv a5, a0
+; RV64-NEXT:    bgtz a3, .LBB0_5
+; RV64-NEXT:  # %bb.4:
+; RV64-NEXT:    li a5, 0
 ; RV64-NEXT:  .LBB0_5:
-; RV64-NEXT:    slli a2, a2, 8
-; RV64-NEXT:    or a2, a5, a2
+; RV64-NEXT:    andi a3, a5, 255
+; RV64-NEXT:    or a2, a3, a2
 ; RV64-NEXT:    bgtz a4, .LBB0_7
 ; RV64-NEXT:  # %bb.6:
 ; RV64-NEXT:    li a0, 0
 ; RV64-NEXT:    j .LBB0_8
 ; RV64-NEXT:  .LBB0_7:
 ; RV64-NEXT:    srli a0, a0, 16
 ; RV64-NEXT:  .LBB0_8:
 ; RV64-NEXT:    sb a0, 2(a1)
 ; RV64-NEXT:    sh a2, 0(a1)
 ; RV64-NEXT:    ret
   %a = load <3 x i8>, <3 x i8>* %in
   %cmp = icmp sgt <3 x i8> %a, zeroinitializer
   %c = select <3 x i1> %cmp, <3 x i8> %a, <3 x i8> zeroinitializer
   store <3 x i8> %c, <3 x i8>* %out
   ret void
 }

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=s390x-linux-gnu -mcpu=z13 < %s  | FileCheck %s
 
 ; Store a <4 x i31> vector.
 define void @fun0(<4 x i31> %src, <4 x i31>* %p)
 ; CHECK-LABEL: fun0:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vlgvf %r1, %v24, 0
 ; CHECK-NEXT:    vlgvf %r0, %v24, 1
 ; CHECK-NEXT:    sllg %r1, %r1, 29
 ; CHECK-NEXT:    rosbg %r1, %r0, 35, 63, 62
 ; CHECK-NEXT:    nihh %r1, 4095
 ; CHECK-NEXT:    stg %r1, 0(%r2)
 ; CHECK-NEXT:    vlgvf %r1, %v24, 2
-; CHECK-NEXT:    risbgn %r0, %r0, 0, 129, 62
+; CHECK-NEXT:    sllg %r0, %r0, 62
 ; CHECK-NEXT:    rosbg %r0, %r1, 2, 32, 31
 ; CHECK-NEXT:    vlgvf %r1, %v24, 3
 ; CHECK-NEXT:    rosbg %r0, %r1, 33, 63, 0
 ; CHECK-NEXT:    stg %r0, 8(%r2)
 ; CHECK-NEXT:    br %r14
 {
   store <4 x i31> %src, <4 x i31>* %p
   ret void

 
 ; Truncate a <8 x i32> vector to <8 x i31> and store it (test splitting).
 define void @fun2(<8 x i32> %src, <8 x i31>* %p)
 ; CHECK-LABEL: fun2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    stmg %r14, %r15, 112(%r15)
 ; CHECK-NEXT:    .cfi_offset %r14, -48
 ; CHECK-NEXT:    .cfi_offset %r15, -40
-; CHECK-NEXT:    vlgvf %r0, %v26, 3
-; CHECK-NEXT:    vlgvf %r4, %v24, 1
-; CHECK-NEXT:    vlgvf %r3, %v24, 2
-; CHECK-NEXT:    srlk %r1, %r0, 8
+; CHECK-NEXT:    vlgvf %r1, %v26, 3
+; CHECK-NEXT:    vlgvf %r0, %v26, 2
+; CHECK-NEXT:    stc %r1, 30(%r2)
+; CHECK-NEXT:    srlk %r3, %r1, 8
+; CHECK-NEXT:    risbgn %r1, %r1, 33, 167, 0
+; CHECK-NEXT:    vlgvf %r5, %v24, 2
+; CHECK-NEXT:    rosbg %r1, %r0, 2, 32, 31
+; CHECK-NEXT:    sth %r3, 28(%r2)
+; CHECK-NEXT:    srlg %r1, %r1, 24
+; CHECK-NEXT:    vlgvf %r3, %v24, 3
+; CHECK-NEXT:    st %r1, 24(%r2)
+; CHECK-NEXT:    vlgvf %r1, %v26, 0
+; CHECK-NEXT:    risbgn %r14, %r5, 6, 164, 27
+; CHECK-NEXT:    sllg %r4, %r3, 60
+; CHECK-NEXT:    rosbg %r14, %r3, 37, 63, 60
+; CHECK-NEXT:    sllg %r3, %r14, 8
+; CHECK-NEXT:    rosbg %r4, %r1, 4, 34, 29
+; CHECK-NEXT:    rosbg %r3, %r4, 56, 63, 8
+; CHECK-NEXT:    stg %r3, 8(%r2)
+; CHECK-NEXT:    vlgvf %r3, %v24, 1
+; CHECK-NEXT:    sllg %r4, %r3, 58
+; CHECK-NEXT:    rosbg %r4, %r5, 6, 36, 27
 ; CHECK-NEXT:    vlgvf %r5, %v24, 0
-; CHECK-NEXT:    sth %r1, 28(%r2)
-; CHECK-NEXT:    risbgn %r1, %r4, 0, 133, 58
 ; CHECK-NEXT:    sllg %r5, %r5, 25
-; CHECK-NEXT:    stc %r0, 30(%r2)
-; CHECK-NEXT:    rosbg %r1, %r3, 6, 36, 27
-; CHECK-NEXT:    vlgvf %r3, %v24, 3
-; CHECK-NEXT:    rosbg %r5, %r4, 39, 63, 58
-; CHECK-NEXT:    sllg %r4, %r5, 8
-; CHECK-NEXT:    rosbg %r1, %r3, 37, 63, 60
-; CHECK-NEXT:    vlgvf %r5, %v26, 1
-; CHECK-NEXT:    rosbg %r4, %r1, 56, 63, 8
-; CHECK-NEXT:    stg %r4, 0(%r2)
-; CHECK-NEXT:    vlgvf %r4, %v26, 2
-; CHECK-NEXT:    risbgn %r14, %r5, 0, 129, 62
-; CHECK-NEXT:    risbgn %r3, %r3, 0, 131, 60
-; CHECK-NEXT:    rosbg %r14, %r4, 2, 32, 31
-; CHECK-NEXT:    rosbg %r14, %r0, 33, 63, 0
-; CHECK-NEXT:    srlg %r0, %r14, 24
-; CHECK-NEXT:    st %r0, 24(%r2)
-; CHECK-NEXT:    vlgvf %r0, %v26, 0
-; CHECK-NEXT:    rosbg %r3, %r0, 4, 34, 29
-; CHECK-NEXT:    sllg %r0, %r1, 8
-; CHECK-NEXT:    rosbg %r3, %r5, 35, 63, 62
-; CHECK-NEXT:    rosbg %r0, %r3, 56, 63, 8
-; CHECK-NEXT:    stg %r0, 8(%r2)
-; CHECK-NEXT:    sllg %r0, %r3, 8
-; CHECK-NEXT:    rosbg %r0, %r14, 56, 63, 8
+; CHECK-NEXT:    rosbg %r5, %r3, 39, 63, 58
+; CHECK-NEXT:    sllg %r3, %r5, 8
+; CHECK-NEXT:    rosbg %r3, %r4, 56, 63, 8
+; CHECK-NEXT:    stg %r3, 0(%r2)
+; CHECK-NEXT:    vlgvf %r3, %v26, 1
+; CHECK-NEXT:    sllg %r4, %r3, 62
+; CHECK-NEXT:    rosbg %r4, %r0, 2, 32, 31
+; CHECK-NEXT:    risbgn %r0, %r1, 4, 162, 29
+; CHECK-NEXT:    rosbg %r0, %r3, 35, 63, 62
+; CHECK-NEXT:    sllg %r0, %r0, 8
+; CHECK-NEXT:    rosbg %r0, %r4, 56, 63, 8
 ; CHECK-NEXT:    stg %r0, 16(%r2)
 ; CHECK-NEXT:    lmg %r14, %r15, 112(%r15)
 ; CHECK-NEXT:    br %r14
 {
   %tmp = trunc <8 x i32> %src to <8 x i31>
   store <8 x i31> %tmp, <8 x i31>* %p
   ret void
 }
 
 ; Load and store a <3 x i31> vector (test widening).
 define void @fun3(<3 x i31>* %src, <3 x i31>* %p)
 ; CHECK-LABEL: fun3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    l %r0, 8(%r2)
+; CHECK-NEXT:    llgf %r0, 8(%r2)
 ; CHECK-NEXT:    lg %r1, 0(%r2)
 ; CHECK-NEXT:    sllg %r2, %r1, 32
 ; CHECK-NEXT:    lr %r2, %r0
-; CHECK-NEXT:    srlg %r0, %r2, 62
-; CHECK-NEXT:    st %r2, 8(%r3)
-; CHECK-NEXT:    rosbg %r0, %r1, 33, 61, 34
-; CHECK-NEXT:    sllg %r1, %r0, 62
-; CHECK-NEXT:    rosbg %r1, %r2, 2, 32, 0
-; CHECK-NEXT:    srlg %r1, %r1, 32
-; CHECK-NEXT:    sllg %r0, %r0, 30
-; CHECK-NEXT:    lr %r0, %r1
-; CHECK-NEXT:    nihh %r0, 8191
-; CHECK-NEXT:    stg %r0, 0(%r3)
+; CHECK-NEXT:    risbgn %r2, %r2, 2, 160, 0
+; CHECK-NEXT:    lgr %r4, %r2
+; CHECK-NEXT:    rosbg %r2, %r1, 0, 1, 32
+; CHECK-NEXT:    rosbg %r4, %r0, 33, 63, 0
+; CHECK-NEXT:    srlg %r0, %r2, 32
+; CHECK-NEXT:    lr %r1, %r0
+; CHECK-NEXT:    nihh %r1, 8191
+; CHECK-NEXT:    st %r4, 8(%r3)
+; CHECK-NEXT:    stg %r1, 0(%r3)
 ; CHECK-NEXT:    br %r14
 {
   %tmp = load <3 x i31>, <3 x i31>* %src
   store <3 x i31> %tmp, <3 x i31>* %p
   ret void
 }

llvm/test/CodeGen/Thumb2/mve-fptosi-sat-vector.ll

 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    vmov q4, q0
 ; CHECK-NEXT:    mov r8, r0
-; CHECK-NEXT:    vmov r0, s18
+; CHECK-NEXT:    vmov r0, s17
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    mov r9, r0
 ; CHECK-NEXT:    vmov r0, s19
 ; CHECK-NEXT:    vldr s20, .LCPI28_0
-; CHECK-NEXT:    mov r5, r1
-; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    mov r7, r1
+; CHECK-NEXT:    vmov r4, s16
+; CHECK-NEXT:    vcmp.f32 s17, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r5, #0
-; CHECK-NEXT:    movtlt r5, #65534
+; CHECK-NEXT:    movlt r7, #0
+; CHECK-NEXT:    movtlt r7, #65534
 ; CHECK-NEXT:    bl __aeabi_f2lz
-; CHECK-NEXT:    mov r7, r0
-; CHECK-NEXT:    vmov r0, s16
 ; CHECK-NEXT:    vldr s22, .LCPI28_1
 ; CHECK-NEXT:    vcmp.f32 s19, s20
+; CHECK-NEXT:    mov r6, r0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    mov r6, r1
-; CHECK-NEXT:    vcmp.f32 s18, s22
+; CHECK-NEXT:    mov r5, r1
+; CHECK-NEXT:    mov r0, r4
+; CHECK-NEXT:    vcmp.f32 s17, s22
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r6, #0
-; CHECK-NEXT:    movtlt r6, #65534
+; CHECK-NEXT:    movlt r5, #0
+; CHECK-NEXT:    movtlt r5, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r5, #65535
-; CHECK-NEXT:    movtgt r5, #1
+; CHECK-NEXT:    movwgt r7, #65535
+; CHECK-NEXT:    movtgt r7, #1
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    mov r4, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s19, s22
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r6, #65535
-; CHECK-NEXT:    movtgt r6, #1
+; CHECK-NEXT:    movwgt r5, #65535
+; CHECK-NEXT:    movtgt r5, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s16
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s19, s20
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s19, s22
 ; CHECK-NEXT:    str.w r0, [r8]
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r7, #0
-; CHECK-NEXT:    vcmp.f32 s19, s22
+; CHECK-NEXT:    movlt r6, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r7, #-1
 ; CHECK-NEXT:    vcmp.f32 s19, s19
+; CHECK-NEXT:    it gt
+; CHECK-NEXT:    movgt.w r6, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s17, s17
 ; CHECK-NEXT:    itt vs
-; CHECK-NEXT:    movvs r7, #0
 ; CHECK-NEXT:    movvs r6, #0
-; CHECK-NEXT:    lsls r0, r6, #22
-; CHECK-NEXT:    orr.w r1, r0, r7, lsr #10
-; CHECK-NEXT:    vmov r0, s17
-; CHECK-NEXT:    str.w r1, [r8, #20]
-; CHECK-NEXT:    bl __aeabi_f2lz
+; CHECK-NEXT:    movvs r5, #0
+; CHECK-NEXT:    lsls r0, r5, #22
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s17, s20
-; CHECK-NEXT:    lsrs r2, r6, #10
+; CHECK-NEXT:    orr.w r0, r0, r6, lsr #10
+; CHECK-NEXT:    str.w r0, [r8, #20]
+; CHECK-NEXT:    it vs
+; CHECK-NEXT:    movvs r7, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    movlt.w r9, #0
 ; CHECK-NEXT:    vcmp.f32 s17, s22
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    it gt
+; CHECK-NEXT:    movgt.w r9, #-1
+; CHECK-NEXT:    vcmp.f32 s17, s17
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    it vs
+; CHECK-NEXT:    movvs.w r9, #0
+; CHECK-NEXT:    lsr.w r0, r9, #14
+; CHECK-NEXT:    orr.w r1, r0, r7, lsl #18
+; CHECK-NEXT:    vmov r0, s18
+; CHECK-NEXT:    str.w r1, [r8, #8]
+; CHECK-NEXT:    bl __aeabi_f2lz
+; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    lsrs r2, r5, #10
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s18, s22
 ; CHECK-NEXT:    itt lt
 ; CHECK-NEXT:    movlt r1, #0
 ; CHECK-NEXT:    movtlt r1, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r1, #65535
 ; CHECK-NEXT:    movtgt r1, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    itt lt
 ; CHECK-NEXT:    movlt r4, #0
 ; CHECK-NEXT:    movtlt r4, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, s20
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r4, #65535
 ; CHECK-NEXT:    movtgt r4, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, s22
+; CHECK-NEXT:    vcmp.f32 s18, s22
 ; CHECK-NEXT:    strb.w r2, [r8, #24]
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, s17
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    ubfx r2, r7, #14, #4
 ; CHECK-NEXT:    vcmp.f32 s16, s16
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r0, #0
+; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    str.w r2, [r8, #12]
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r4, #0
-; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    bfc r4, #18, #14
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s22
-; CHECK-NEXT:    orr.w r2, r4, r0, lsl #18
+; CHECK-NEXT:    orr.w r2, r4, r9, lsl #18
 ; CHECK-NEXT:    str.w r2, [r8, #4]
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r9, #0
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s18
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r9, #-1
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, s17
-; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs.w r9, #0
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r1, #0
-; CHECK-NEXT:    vcmp.f32 s18, s18
+; CHECK-NEXT:    lsrs r0, r0, #28
 ; CHECK-NEXT:    bfc r1, #18, #14
-; CHECK-NEXT:    lsrs r0, r0, #14
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    lsr.w r2, r1, #14
-; CHECK-NEXT:    orr.w r0, r0, r1, lsl #18
-; CHECK-NEXT:    orr.w r2, r2, r9, lsl #4
-; CHECK-NEXT:    str.w r2, [r8, #12]
-; CHECK-NEXT:    str.w r0, [r8, #8]
-; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r5, #0
-; CHECK-NEXT:    bfc r5, #18, #14
-; CHECK-NEXT:    lsr.w r0, r9, #28
-; CHECK-NEXT:    orr.w r0, r0, r5, lsl #4
-; CHECK-NEXT:    orr.w r0, r0, r7, lsl #22
+; CHECK-NEXT:    orr.w r0, r0, r1, lsl #4
+; CHECK-NEXT:    orr.w r0, r0, r6, lsl #22
 ; CHECK-NEXT:    str.w r0, [r8, #16]
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI28_0:
 ; CHECK-NEXT:    .long 0xd8000000 @ float -5.62949953E+14

 
 define arm_aapcs_vfpcc <8 x i19> @test_signed_v8f16_v8i19(<8 x half> %f) {
 ; CHECK-LABEL: test_signed_v8f16_v8i19:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    .vsave {d8}
 ; CHECK-NEXT:    vpush {d8}
-; CHECK-NEXT:    vldr s5, .LCPI46_0
+; CHECK-NEXT:    vldr s12, .LCPI46_0
 ; CHECK-NEXT:    vcvtt.f32.f16 s15, s3
-; CHECK-NEXT:    vldr s7, .LCPI46_1
-; CHECK-NEXT:    vcvtb.f32.f16 s8, s2
-; CHECK-NEXT:    vmaxnm.f32 s16, s15, s5
+; CHECK-NEXT:    vldr s14, .LCPI46_1
+; CHECK-NEXT:    vcvtb.f32.f16 s7, s0
+; CHECK-NEXT:    vmaxnm.f32 s16, s15, s12
 ; CHECK-NEXT:    vcvtb.f32.f16 s4, s1
-; CHECK-NEXT:    vcvtt.f32.f16 s12, s1
-; CHECK-NEXT:    vcvtb.f32.f16 s1, s0
-; CHECK-NEXT:    vminnm.f32 s16, s16, s7
+; CHECK-NEXT:    vcvtt.f32.f16 s8, s1
+; CHECK-NEXT:    vcvtb.f32.f16 s1, s2
 ; CHECK-NEXT:    vcvtt.f32.f16 s0, s0
 ; CHECK-NEXT:    vcvtt.f32.f16 s2, s2
 ; CHECK-NEXT:    vcvtb.f32.f16 s3, s3
-; CHECK-NEXT:    vmaxnm.f32 s6, s4, s5
-; CHECK-NEXT:    vmaxnm.f32 s10, s8, s5
-; CHECK-NEXT:    vmaxnm.f32 s14, s12, s5
-; CHECK-NEXT:    vmaxnm.f32 s9, s1, s5
-; CHECK-NEXT:    vmaxnm.f32 s11, s0, s5
-; CHECK-NEXT:    vmaxnm.f32 s13, s2, s5
-; CHECK-NEXT:    vmaxnm.f32 s5, s3, s5
+; CHECK-NEXT:    vmaxnm.f32 s6, s4, s12
+; CHECK-NEXT:    vmaxnm.f32 s10, s8, s12
+; CHECK-NEXT:    vmaxnm.f32 s5, s1, s12
+; CHECK-NEXT:    vmaxnm.f32 s9, s7, s12
+; CHECK-NEXT:    vmaxnm.f32 s11, s0, s12
+; CHECK-NEXT:    vmaxnm.f32 s13, s2, s12
+; CHECK-NEXT:    vminnm.f32 s16, s16, s14
+; CHECK-NEXT:    vmaxnm.f32 s12, s3, s12
 ; CHECK-NEXT:    vcvt.s32.f32 s16, s16
-; CHECK-NEXT:    vminnm.f32 s5, s5, s7
-; CHECK-NEXT:    vminnm.f32 s13, s13, s7
-; CHECK-NEXT:    vcvt.s32.f32 s5, s5
-; CHECK-NEXT:    vminnm.f32 s11, s11, s7
+; CHECK-NEXT:    vminnm.f32 s12, s12, s14
+; CHECK-NEXT:    vminnm.f32 s13, s13, s14
+; CHECK-NEXT:    vcvt.s32.f32 s12, s12
+; CHECK-NEXT:    vminnm.f32 s9, s9, s14
 ; CHECK-NEXT:    vcvt.s32.f32 s13, s13
-; CHECK-NEXT:    vminnm.f32 s9, s9, s7
-; CHECK-NEXT:    vcmp.f32 s15, s15
-; CHECK-NEXT:    vminnm.f32 s10, s10, s7
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vminnm.f32 s11, s11, s14
 ; CHECK-NEXT:    vcvt.s32.f32 s11, s11
-; CHECK-NEXT:    vcmp.f32 s3, s3
-; CHECK-NEXT:    vminnm.f32 s14, s14, s7
+; CHECK-NEXT:    vminnm.f32 s5, s5, s14
+; CHECK-NEXT:    vcvt.s32.f32 s9, s9
+; CHECK-NEXT:    vminnm.f32 s10, s10, s14
+; CHECK-NEXT:    vcmp.f32 s15, s15
+; CHECK-NEXT:    vminnm.f32 s6, s6, s14
 ; CHECK-NEXT:    vmov r1, s16
-; CHECK-NEXT:    vminnm.f32 s6, s6, s7
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r1, #0
 ; CHECK-NEXT:    lsrs r2, r1, #11
+; CHECK-NEXT:    vcmp.f32 s3, s3
 ; CHECK-NEXT:    strb r2, [r0, #18]
-; CHECK-NEXT:    vmov r2, s5
+; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r2, #0
-; CHECK-NEXT:    vcvt.s32.f32 s9, s9
-; CHECK-NEXT:    bfc r2, #19, #13
+; CHECK-NEXT:    movvs r3, #0
+; CHECK-NEXT:    ubfx r2, r3, #14, #5
+; CHECK-NEXT:    vcvt.s32.f32 s5, s5
+; CHECK-NEXT:    orr.w r1, r2, r1, lsl #5
 ; CHECK-NEXT:    vcmp.f32 s2, s2
-; CHECK-NEXT:    vmov r12, s13
-; CHECK-NEXT:    lsrs r3, r2, #14
-; CHECK-NEXT:    orr.w r1, r3, r1, lsl #5
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    strh r1, [r0, #16]
-; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs.w r12, #0
-; CHECK-NEXT:    vcvt.s32.f32 s10, s10
-; CHECK-NEXT:    bfc r12, #19, #13
-; CHECK-NEXT:    vcvt.s32.f32 s14, s14
-; CHECK-NEXT:    lsr.w r3, r12, #1
-; CHECK-NEXT:    vcmp.f32 s0, s0
-; CHECK-NEXT:    vmov lr, s11
-; CHECK-NEXT:    orr.w r2, r3, r2, lsl #18
+; CHECK-NEXT:    vmov lr, s13
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    str r2, [r0, #12]
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs.w lr, #0
-; CHECK-NEXT:    vcmp.f32 s1, s1
+; CHECK-NEXT:    ubfx r1, lr, #1, #18
+; CHECK-NEXT:    vcmp.f32 s0, s0
+; CHECK-NEXT:    orr.w r1, r1, r3, lsl #18
+; CHECK-NEXT:    vcvt.s32.f32 s10, s10
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vmov r12, s11
+; CHECK-NEXT:    str r1, [r0, #12]
 ; CHECK-NEXT:    vmov r3, s9
+; CHECK-NEXT:    it vs
+; CHECK-NEXT:    movvs.w r12, #0
+; CHECK-NEXT:    vcmp.f32 s7, s7
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r3, #0
-; CHECK-NEXT:    bfc lr, #19, #13
 ; CHECK-NEXT:    bfc r3, #19, #13
-; CHECK-NEXT:    vcmp.f32 s12, s12
-; CHECK-NEXT:    orr.w r3, r3, lr, lsl #19
 ; CHECK-NEXT:    vcvt.s32.f32 s6, s6
+; CHECK-NEXT:    orr.w r3, r3, r12, lsl #19
 ; CHECK-NEXT:    str r3, [r0]
+; CHECK-NEXT:    vcmp.f32 s1, s1
+; CHECK-NEXT:    vmov r3, s5
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vmov r3, s14
-; CHECK-NEXT:    vmov r1, s10
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r3, #0
 ; CHECK-NEXT:    vcmp.f32 s8, s8
 ; CHECK-NEXT:    bfc r3, #19, #13
+; CHECK-NEXT:    vmov r1, s10
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r1, #0
-; CHECK-NEXT:    bfc r1, #19, #13
-; CHECK-NEXT:    lsrs r2, r3, #7
+; CHECK-NEXT:    ubfx r2, r1, #7, #12
 ; CHECK-NEXT:    vcmp.f32 s4, s4
-; CHECK-NEXT:    orr.w r1, r2, r1, lsl #12
+; CHECK-NEXT:    orr.w r2, r2, r3, lsl #12
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    orr.w r1, r1, r12, lsl #31
-; CHECK-NEXT:    str r1, [r0, #8]
-; CHECK-NEXT:    vmov r1, s6
-; CHECK-NEXT:    lsr.w r2, lr, #13
+; CHECK-NEXT:    orr.w r2, r2, lr, lsl #31
+; CHECK-NEXT:    str r2, [r0, #8]
+; CHECK-NEXT:    vmov r2, s6
+; CHECK-NEXT:    ubfx r3, r12, #13, #6
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r1, #0
-; CHECK-NEXT:    bfc r1, #19, #13
-; CHECK-NEXT:    orr.w r1, r2, r1, lsl #6
-; CHECK-NEXT:    orr.w r1, r1, r3, lsl #25
+; CHECK-NEXT:    movvs r2, #0
+; CHECK-NEXT:    bfc r2, #19, #13
+; CHECK-NEXT:    orr.w r2, r3, r2, lsl #6
+; CHECK-NEXT:    orr.w r1, r2, r1, lsl #25
 ; CHECK-NEXT:    str r1, [r0, #4]
 ; CHECK-NEXT:    vpop {d8}
 ; CHECK-NEXT:    pop {r7, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI46_0:
 ; CHECK-NEXT:    .long 0xc8800000 @ float -262144
 ; CHECK-NEXT:  .LCPI46_1:

 
 define arm_aapcs_vfpcc <8 x i50> @test_signed_v8f16_v8i50(<8 x half> %f) {
 ; CHECK-LABEL: test_signed_v8f16_v8i50:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14}
+; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14}
+; CHECK-NEXT:    .pad #16
+; CHECK-NEXT:    sub sp, #16
 ; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    mov r10, r0
-; CHECK-NEXT:    vcvtt.f32.f16 s30, s19
-; CHECK-NEXT:    vmov r0, s30
+; CHECK-NEXT:    mov r11, r0
+; CHECK-NEXT:    vcvtt.f32.f16 s28, s19
+; CHECK-NEXT:    vmov r0, s28
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcvtb.f32.f16 s26, s18
-; CHECK-NEXT:    mov r4, r0
+; CHECK-NEXT:    mov r7, r0
 ; CHECK-NEXT:    vmov r0, s26
 ; CHECK-NEXT:    vldr s22, .LCPI48_1
 ; CHECK-NEXT:    vcvtb.f32.f16 s24, s16
-; CHECK-NEXT:    vcvtt.f32.f16 s28, s17
-; CHECK-NEXT:    vcmp.f32 s30, s22
-; CHECK-NEXT:    mov r6, r1
+; CHECK-NEXT:    vcvtt.f32.f16 s18, s18
+; CHECK-NEXT:    vcmp.f32 s28, s22
+; CHECK-NEXT:    mov r4, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vmov r7, s24
+; CHECK-NEXT:    vmov r6, s24
 ; CHECK-NEXT:    vldr s20, .LCPI48_0
-; CHECK-NEXT:    vmov r8, s28
+; CHECK-NEXT:    vmov r5, s18
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r6, #0
-; CHECK-NEXT:    movtlt r6, #65534
+; CHECK-NEXT:    movlt r4, #0
+; CHECK-NEXT:    movtlt r4, #65534
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s26, s22
-; CHECK-NEXT:    mov r5, r1
+; CHECK-NEXT:    str r1, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s30, s20
+; CHECK-NEXT:    vcmp.f32 s28, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s26, s20
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r6, #65535
-; CHECK-NEXT:    movtgt r6, #1
+; CHECK-NEXT:    movwgt r4, #65535
+; CHECK-NEXT:    movtgt r4, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vcmp.f32 s26, s26
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r0, #0
-; CHECK-NEXT:    str.w r0, [r10, #25]
-; CHECK-NEXT:    mov r0, r7
+; CHECK-NEXT:    str.w r0, [r11, #25]
+; CHECK-NEXT:    mov r0, r6
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s24, s22
-; CHECK-NEXT:    mov r11, r1
+; CHECK-NEXT:    str r1, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s24, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s24, s24
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s30, s22
+; CHECK-NEXT:    vcmp.f32 s28, s22
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    str.w r0, [r10]
+; CHECK-NEXT:    str.w r0, [r11]
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r4, #0
-; CHECK-NEXT:    vcmp.f32 s30, s20
+; CHECK-NEXT:    movlt r7, #0
+; CHECK-NEXT:    vcmp.f32 s28, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r4, #-1
-; CHECK-NEXT:    vcmp.f32 s30, s30
+; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    vcmp.f32 s28, s28
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r4, #0
-; CHECK-NEXT:    str r4, [sp, #4] @ 4-byte Spill
+; CHECK-NEXT:    movvs r7, #0
+; CHECK-NEXT:    str r7, [sp, #12] @ 4-byte Spill
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r6, #0
-; CHECK-NEXT:    lsls r0, r6, #22
-; CHECK-NEXT:    orr.w r7, r0, r4, lsr #10
-; CHECK-NEXT:    mov r0, r8
+; CHECK-NEXT:    movvs r4, #0
+; CHECK-NEXT:    lsls r0, r4, #22
+; CHECK-NEXT:    orr.w r7, r0, r7, lsr #10
+; CHECK-NEXT:    mov r0, r5
 ; CHECK-NEXT:    bl __aeabi_f2lz
-; CHECK-NEXT:    vcmp.f32 s28, s22
-; CHECK-NEXT:    mov r4, r1
+; CHECK-NEXT:    vcmp.f32 s18, s22
+; CHECK-NEXT:    mov r6, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s28, s20
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r4, #0
-; CHECK-NEXT:    movtlt r4, #65534
+; CHECK-NEXT:    movlt r6, #0
+; CHECK-NEXT:    movtlt r6, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s28, s22
-; CHECK-NEXT:    mov r1, r0
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r4, #65535
-; CHECK-NEXT:    movtgt r4, #1
+; CHECK-NEXT:    movwgt r6, #65535
+; CHECK-NEXT:    movtgt r6, #1
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    mov r5, r0
+; CHECK-NEXT:    vcmp.f32 s18, s22
+; CHECK-NEXT:    str.w r7, [r11, #45]
+; CHECK-NEXT:    it vs
+; CHECK-NEXT:    movvs r6, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    str.w r7, [r10, #45]
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r1, #0
-; CHECK-NEXT:    vcmp.f32 s28, s20
-; CHECK-NEXT:    vcvtt.f32.f16 s18, s18
+; CHECK-NEXT:    movlt r5, #0
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r1, #-1
-; CHECK-NEXT:    vcmp.f32 s28, s28
+; CHECK-NEXT:    movgt.w r5, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r1, #0
-; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
-; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r4, #0
-; CHECK-NEXT:    lsls r0, r4, #22
-; CHECK-NEXT:    orr.w r0, r0, r1, lsr #10
-; CHECK-NEXT:    str.w r0, [r10, #20]
+; CHECK-NEXT:    movvs r5, #0
+; CHECK-NEXT:    lsrs r0, r5, #14
+; CHECK-NEXT:    orr.w r0, r0, r6, lsl #18
+; CHECK-NEXT:    vcvtt.f32.f16 s18, s17
+; CHECK-NEXT:    str.w r0, [r11, #33]
 ; CHECK-NEXT:    vmov r0, s18
-; CHECK-NEXT:    lsrs r1, r6, #10
-; CHECK-NEXT:    strb.w r1, [r10, #49]
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s18, s22
-; CHECK-NEXT:    mov r9, r0
+; CHECK-NEXT:    mov r9, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r9, #0
+; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r9, #-1
+; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s26, s22
+; CHECK-NEXT:    vcmp.f32 s18, s22
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs.w r9, #0
+; CHECK-NEXT:    movvs r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    str r0, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r5, #0
-; CHECK-NEXT:    movtlt r5, #65534
-; CHECK-NEXT:    vcmp.f32 s26, s20
+; CHECK-NEXT:    movwlt r9, #0
+; CHECK-NEXT:    movtlt r9, #65534
+; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    mov r1, r0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r5, #65535
-; CHECK-NEXT:    movtgt r5, #1
-; CHECK-NEXT:    vcmp.f32 s26, s26
+; CHECK-NEXT:    movwgt r9, #65535
+; CHECK-NEXT:    movtgt r9, #1
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    vcvtt.f32.f16 s16, s16
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r5, #0
-; CHECK-NEXT:    bfc r5, #18, #14
-; CHECK-NEXT:    mov r7, r1
-; CHECK-NEXT:    orr.w r0, r5, r9, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #29]
+; CHECK-NEXT:    movvs.w r9, #0
+; CHECK-NEXT:    lsl.w r0, r9, #22
+; CHECK-NEXT:    orr.w r0, r0, r1, lsr #10
+; CHECK-NEXT:    str.w r0, [r11, #20]
 ; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    lsrs r1, r4, #10
-; CHECK-NEXT:    strb.w r1, [r10, #24]
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    mov r8, r0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt.w r8, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s16
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r8, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, s22
+; CHECK-NEXT:    mov r10, r1
+; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs.w r8, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movwlt r11, #0
-; CHECK-NEXT:    movtlt r11, #65534
-; CHECK-NEXT:    vcmp.f32 s24, s20
+; CHECK-NEXT:    movwlt r10, #0
+; CHECK-NEXT:    movtlt r10, #65534
+; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    lsr.w r0, r8, #14
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, s24
+; CHECK-NEXT:    vcmp.f32 s16, s16
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r11, #65535
-; CHECK-NEXT:    movtgt r11, #1
+; CHECK-NEXT:    movwgt r10, #65535
+; CHECK-NEXT:    movtgt r10, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs.w r11, #0
-; CHECK-NEXT:    vcvtb.f32.f16 s24, s19
-; CHECK-NEXT:    bfc r11, #18, #14
-; CHECK-NEXT:    mov r6, r1
-; CHECK-NEXT:    orr.w r0, r11, r8, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #4]
-; CHECK-NEXT:    vmov r0, s24
+; CHECK-NEXT:    movvs.w r10, #0
+; CHECK-NEXT:    orr.w r0, r0, r10, lsl #18
+; CHECK-NEXT:    str.w r0, [r11, #8]
+; CHECK-NEXT:    lsrs r0, r4, #10
+; CHECK-NEXT:    vcvtb.f32.f16 s16, s19
+; CHECK-NEXT:    strb.w r0, [r11, #49]
+; CHECK-NEXT:    vmov r0, s16
 ; CHECK-NEXT:    bl __aeabi_f2lz
-; CHECK-NEXT:    vcmp.f32 s24, s22
-; CHECK-NEXT:    mov r5, r0
+; CHECK-NEXT:    mov r7, r0
+; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r5, #0
+; CHECK-NEXT:    movlt r7, #0
+; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    ubfx r0, r6, #14, #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, s24
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r5, #-1
+; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    vcmp.f32 s16, s16
+; CHECK-NEXT:    vcvtb.f32.f16 s18, s17
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s22
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r5, #0
+; CHECK-NEXT:    movvs r7, #0
+; CHECK-NEXT:    orr.w r0, r0, r7, lsl #4
+; CHECK-NEXT:    str.w r0, [r11, #37]
+; CHECK-NEXT:    vcmp.f32 s26, s22
+; CHECK-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r7, #0
-; CHECK-NEXT:    movtlt r7, #65534
-; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    movtlt r0, #65534
+; CHECK-NEXT:    vcmp.f32 s26, s20
+; CHECK-NEXT:    mov r4, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r7, #65535
-; CHECK-NEXT:    movtgt r7, #1
+; CHECK-NEXT:    movwgt r0, #65535
+; CHECK-NEXT:    movtgt r0, #1
+; CHECK-NEXT:    vcmp.f32 s26, s26
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r7, #0
-; CHECK-NEXT:    vcvtb.f32.f16 s18, s17
-; CHECK-NEXT:    bfc r7, #18, #14
-; CHECK-NEXT:    mov r4, r1
-; CHECK-NEXT:    lsrs r0, r7, #14
-; CHECK-NEXT:    orr.w r0, r0, r5, lsl #4
-; CHECK-NEXT:    str.w r0, [r10, #37]
-; CHECK-NEXT:    lsr.w r0, r9, #14
-; CHECK-NEXT:    orr.w r0, r0, r7, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #33]
+; CHECK-NEXT:    movvs r0, #0
+; CHECK-NEXT:    bfc r0, #18, #14
+; CHECK-NEXT:    orr.w r0, r0, r5, lsl #18
+; CHECK-NEXT:    str.w r0, [r11, #29]
+; CHECK-NEXT:    lsr.w r0, r9, #10
+; CHECK-NEXT:    strb.w r0, [r11, #24]
 ; CHECK-NEXT:    vmov r0, s18
 ; CHECK-NEXT:    bl __aeabi_f2lz
 ; CHECK-NEXT:    vcmp.f32 s18, s22
+; CHECK-NEXT:    ubfx r2, r10, #14, #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
+; CHECK-NEXT:    vcmp.f32 s18, s18
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r0, #0
+; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
+; CHECK-NEXT:    str.w r2, [r11, #12]
+; CHECK-NEXT:    vcmp.f32 s24, s22
+; CHECK-NEXT:    ldr r2, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s24, s20
 ; CHECK-NEXT:    itt lt
-; CHECK-NEXT:    movlt r6, #0
-; CHECK-NEXT:    movtlt r6, #65534
-; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    movlt r2, #0
+; CHECK-NEXT:    movtlt r2, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s24, s24
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r6, #65535
-; CHECK-NEXT:    movtgt r6, #1
-; CHECK-NEXT:    vcmp.f32 s16, s16
+; CHECK-NEXT:    movwgt r2, #65535
+; CHECK-NEXT:    movtgt r2, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    it vs
-; CHECK-NEXT:    movvs r6, #0
-; CHECK-NEXT:    bfc r6, #18, #14
 ; CHECK-NEXT:    vcmp.f32 s18, s22
+; CHECK-NEXT:    it vs
+; CHECK-NEXT:    movvs r2, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s18, s20
-; CHECK-NEXT:    lsr.w r2, r6, #14
-; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
-; CHECK-NEXT:    str.w r2, [r10, #12]
 ; CHECK-NEXT:    itt lt
 ; CHECK-NEXT:    movlt r1, #0
 ; CHECK-NEXT:    movtlt r1, #65534
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, s22
+; CHECK-NEXT:    vcmp.f32 s16, s22
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r1, #65535
 ; CHECK-NEXT:    movtgt r1, #1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt lt
 ; CHECK-NEXT:    movlt r4, #0
 ; CHECK-NEXT:    movtlt r4, #65534
-; CHECK-NEXT:    vcmp.f32 s24, s20
-; CHECK-NEXT:    lsr.w r2, r8, #14
+; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    bfc r2, #18, #14
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r4, #65535
 ; CHECK-NEXT:    movtgt r4, #1
-; CHECK-NEXT:    vcmp.f32 s24, s24
-; CHECK-NEXT:    orr.w r2, r2, r6, lsl #18
-; CHECK-NEXT:    str.w r2, [r10, #8]
+; CHECK-NEXT:    vcmp.f32 s16, s16
+; CHECK-NEXT:    orr.w r2, r2, r8, lsl #18
+; CHECK-NEXT:    str.w r2, [r11, #4]
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r4, #0
-; CHECK-NEXT:    ldr r3, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    bfc r4, #18, #14
-; CHECK-NEXT:    lsrs r2, r5, #28
+; CHECK-NEXT:    ldr r3, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    lsrs r2, r7, #28
 ; CHECK-NEXT:    vcmp.f32 s18, s18
-; CHECK-NEXT:    lsrs r0, r0, #28
 ; CHECK-NEXT:    orr.w r2, r2, r4, lsl #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    orr.w r2, r2, r3, lsl #22
-; CHECK-NEXT:    str.w r2, [r10, #41]
+; CHECK-NEXT:    str.w r2, [r11, #41]
 ; CHECK-NEXT:    it vs
 ; CHECK-NEXT:    movvs r1, #0
+; CHECK-NEXT:    lsrs r0, r0, #28
 ; CHECK-NEXT:    bfc r1, #18, #14
 ; CHECK-NEXT:    orr.w r0, r0, r1, lsl #4
 ; CHECK-NEXT:    ldr r1, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    orr.w r0, r0, r1, lsl #22
-; CHECK-NEXT:    str.w r0, [r10, #16]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
+; CHECK-NEXT:    str.w r0, [r11, #16]
+; CHECK-NEXT:    add sp, #16
+; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI48_0:
 ; CHECK-NEXT:    .long 0x57ffffff @ float 5.6294992E+14
 ; CHECK-NEXT:  .LCPI48_1:
 ; CHECK-NEXT:    .long 0xd8000000 @ float -5.62949953E+14

llvm/test/CodeGen/Thumb2/mve-fptoui-sat-vector.ll

 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10}
 ; CHECK-NEXT:    vpush {d8, d9, d10}
 ; CHECK-NEXT:    vmov q4, q0
 ; CHECK-NEXT:    mov r8, r0
 ; CHECK-NEXT:    vmov r0, s16
 ; CHECK-NEXT:    vldr s20, .LCPI28_0
-; CHECK-NEXT:    vmov r5, s18
+; CHECK-NEXT:    vmov r4, s17
 ; CHECK-NEXT:    vmov r6, s19
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    mov r4, r0
+; CHECK-NEXT:    mov r7, r0
 ; CHECK-NEXT:    vcmp.f32 s16, #0
-; CHECK-NEXT:    mov r0, r5
+; CHECK-NEXT:    mov r0, r4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    mov r7, r1
+; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r4, #0
+; CHECK-NEXT:    movlt r7, #0
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcmp.f32 s18, #0
-; CHECK-NEXT:    mov r5, r1
+; CHECK-NEXT:    vcmp.f32 s17, #0
+; CHECK-NEXT:    mov r10, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    mov r9, r0
 ; CHECK-NEXT:    mov r0, r6
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r5, #0
-; CHECK-NEXT:    vcmp.f32 s18, s20
+; CHECK-NEXT:    movlt.w r10, #0
+; CHECK-NEXT:    vcmp.f32 s17, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r5, #65535
-; CHECK-NEXT:    movtgt r5, #3
+; CHECK-NEXT:    movwgt r10, #65535
+; CHECK-NEXT:    movtgt r10, #3
 ; CHECK-NEXT:    bl __aeabi_f2ulz
 ; CHECK-NEXT:    vcmp.f32 s19, #0
 ; CHECK-NEXT:    mov r6, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s19, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r6, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r6, #65535
 ; CHECK-NEXT:    movtgt r6, #3
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    mov r10, r0
 ; CHECK-NEXT:    vcmp.f32 s19, #0
 ; CHECK-NEXT:    it gt
+; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    mov r4, r0
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s19, s20
+; CHECK-NEXT:    str.w r7, [r8]
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    movlt r4, #0
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    lsl.w r0, r6, #22
+; CHECK-NEXT:    vcmp.f32 s17, #0
+; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r4, #-1
+; CHECK-NEXT:    orr.w r0, r0, r4, lsr #10
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    str.w r4, [r8]
+; CHECK-NEXT:    str.w r0, [r8, #20]
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r10, #0
-; CHECK-NEXT:    vcmp.f32 s19, s20
-; CHECK-NEXT:    lsls r0, r6, #22
+; CHECK-NEXT:    movlt.w r9, #0
+; CHECK-NEXT:    vcmp.f32 s17, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r10, #-1
-; CHECK-NEXT:    orr.w r1, r0, r10, lsr #10
-; CHECK-NEXT:    vmov r0, s17
-; CHECK-NEXT:    str.w r1, [r8, #20]
+; CHECK-NEXT:    movgt.w r9, #-1
+; CHECK-NEXT:    lsr.w r0, r9, #14
+; CHECK-NEXT:    orr.w r1, r0, r10, lsl #18
+; CHECK-NEXT:    vmov r0, s18
+; CHECK-NEXT:    str.w r1, [r8, #8]
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcmp.f32 s17, #0
+; CHECK-NEXT:    vcmp.f32 s18, #0
 ; CHECK-NEXT:    lsrs r2, r6, #10
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, s20
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, #0
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r1, #65535
 ; CHECK-NEXT:    movtgt r1, #3
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r7, #0
+; CHECK-NEXT:    movlt r5, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s17, #0
+; CHECK-NEXT:    vcmp.f32 s18, #0
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r7, #65535
-; CHECK-NEXT:    movtgt r7, #3
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    movwgt r5, #65535
+; CHECK-NEXT:    movtgt r5, #3
 ; CHECK-NEXT:    strb.w r2, [r8, #24]
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
-; CHECK-NEXT:    vcmp.f32 s17, s20
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
-; CHECK-NEXT:    bfc r7, #18, #14
+; CHECK-NEXT:    ubfx r2, r10, #14, #4
 ; CHECK-NEXT:    bfc r1, #18, #14
-; CHECK-NEXT:    orr.w r2, r7, r0, lsl #18
-; CHECK-NEXT:    lsrs r0, r0, #14
-; CHECK-NEXT:    vcmp.f32 s18, #0
-; CHECK-NEXT:    str.w r2, [r8, #4]
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r9, #0
-; CHECK-NEXT:    vcmp.f32 s18, s20
-; CHECK-NEXT:    lsrs r2, r1, #14
-; CHECK-NEXT:    orr.w r0, r0, r1, lsl #18
+; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
+; CHECK-NEXT:    lsrs r0, r0, #28
+; CHECK-NEXT:    orr.w r0, r0, r1, lsl #4
 ; CHECK-NEXT:    bfc r5, #18, #14
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r9, #-1
-; CHECK-NEXT:    orr.w r2, r2, r9, lsl #4
 ; CHECK-NEXT:    str.w r2, [r8, #12]
-; CHECK-NEXT:    str.w r0, [r8, #8]
-; CHECK-NEXT:    lsr.w r0, r9, #28
-; CHECK-NEXT:    orr.w r0, r0, r5, lsl #4
-; CHECK-NEXT:    orr.w r0, r0, r10, lsl #22
+; CHECK-NEXT:    orr.w r2, r5, r9, lsl #18
+; CHECK-NEXT:    str.w r2, [r8, #4]
+; CHECK-NEXT:    orr.w r0, r0, r4, lsl #22
 ; CHECK-NEXT:    str.w r0, [r8, #16]
 ; CHECK-NEXT:    vpop {d8, d9, d10}
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI28_0:
 ; CHECK-NEXT:    .long 0x587fffff @ float 1.12589984E+15
     %x = call <4 x i50> @llvm.fptoui.sat.v4f32.v4i50(<4 x float> %f)

 
 define arm_aapcs_vfpcc <8 x i50> @test_unsigned_v8f16_v8i50(<8 x half> %f) {
 ; CHECK-LABEL: test_unsigned_v8f16_v8i50:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
+; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13}
+; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13}
+; CHECK-NEXT:    .pad #16
+; CHECK-NEXT:    sub sp, #16
 ; CHECK-NEXT:    vmov q4, q0
-; CHECK-NEXT:    mov r10, r0
+; CHECK-NEXT:    mov r4, r0
 ; CHECK-NEXT:    vcvtb.f32.f16 s24, s18
 ; CHECK-NEXT:    vmov r0, s24
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcvtt.f32.f16 s28, s19
-; CHECK-NEXT:    mov r6, r0
-; CHECK-NEXT:    vmov r0, s28
+; CHECK-NEXT:    vcvtt.f32.f16 s26, s19
+; CHECK-NEXT:    mov r7, r0
+; CHECK-NEXT:    vmov r0, s26
 ; CHECK-NEXT:    vcvtb.f32.f16 s22, s16
-; CHECK-NEXT:    vcvtt.f32.f16 s26, s17
+; CHECK-NEXT:    vcvtt.f32.f16 s18, s18
 ; CHECK-NEXT:    vcmp.f32 s24, #0
+; CHECK-NEXT:    str r1, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    mov r7, r1
 ; CHECK-NEXT:    vmov r5, s22
 ; CHECK-NEXT:    vldr s20, .LCPI48_0
-; CHECK-NEXT:    vmov r8, s26
+; CHECK-NEXT:    vmov r8, s18
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r6, #0
+; CHECK-NEXT:    movlt r7, #0
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcmp.f32 s28, #0
-; CHECK-NEXT:    mov r4, r1
+; CHECK-NEXT:    vcmp.f32 s26, #0
+; CHECK-NEXT:    mov r10, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s28, s20
-; CHECK-NEXT:    mov r9, r0
+; CHECK-NEXT:    vcmp.f32 s26, s20
+; CHECK-NEXT:    mov r6, r0
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r4, #0
+; CHECK-NEXT:    movlt.w r10, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    mov r0, r5
 ; CHECK-NEXT:    vcmp.f32 s24, s20
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r4, #65535
-; CHECK-NEXT:    movtgt r4, #3
+; CHECK-NEXT:    movwgt r10, #65535
+; CHECK-NEXT:    movtgt r10, #3
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r6, #-1
-; CHECK-NEXT:    str.w r6, [r10, #25]
+; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    str.w r7, [r4, #25]
 ; CHECK-NEXT:    bl __aeabi_f2ulz
 ; CHECK-NEXT:    vcmp.f32 s22, #0
-; CHECK-NEXT:    mov r11, r1
+; CHECK-NEXT:    str r1, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s22, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
-; CHECK-NEXT:    vcmp.f32 s22, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s26, #0
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
-; CHECK-NEXT:    vcmp.f32 s28, #0
-; CHECK-NEXT:    str.w r0, [r10]
+; CHECK-NEXT:    str r0, [r4]
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r9, #0
-; CHECK-NEXT:    vcmp.f32 s28, s20
-; CHECK-NEXT:    lsls r0, r4, #22
+; CHECK-NEXT:    movlt r6, #0
+; CHECK-NEXT:    vcmp.f32 s26, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r9, #-1
-; CHECK-NEXT:    orr.w r5, r0, r9, lsr #10
+; CHECK-NEXT:    movgt.w r6, #-1
+; CHECK-NEXT:    lsl.w r0, r10, #22
+; CHECK-NEXT:    str r6, [sp, #12] @ 4-byte Spill
+; CHECK-NEXT:    orr.w r6, r0, r6, lsr #10
 ; CHECK-NEXT:    mov r0, r8
-; CHECK-NEXT:    str.w r9, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcmp.f32 s26, #0
-; CHECK-NEXT:    mov r6, r1
+; CHECK-NEXT:    vcmp.f32 s18, #0
+; CHECK-NEXT:    mov r5, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s26, s20
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r6, #0
+; CHECK-NEXT:    movlt r5, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s26, #0
-; CHECK-NEXT:    mov r1, r0
+; CHECK-NEXT:    mov r7, r0
+; CHECK-NEXT:    vcmp.f32 s18, #0
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r6, #65535
-; CHECK-NEXT:    movtgt r6, #3
+; CHECK-NEXT:    movwgt r5, #65535
+; CHECK-NEXT:    movtgt r5, #3
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    str.w r5, [r10, #45]
+; CHECK-NEXT:    str.w r6, [r4, #45]
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r1, #0
-; CHECK-NEXT:    vcmp.f32 s26, s20
-; CHECK-NEXT:    lsls r0, r6, #22
+; CHECK-NEXT:    movlt r7, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r1, #-1
-; CHECK-NEXT:    orr.w r0, r0, r1, lsr #10
-; CHECK-NEXT:    vcvtt.f32.f16 s18, s18
-; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
-; CHECK-NEXT:    lsrs r1, r4, #10
-; CHECK-NEXT:    str.w r0, [r10, #20]
+; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    lsrs r0, r7, #14
+; CHECK-NEXT:    orr.w r0, r0, r5, lsl #18
+; CHECK-NEXT:    vcvtt.f32.f16 s18, s17
+; CHECK-NEXT:    str.w r0, [r4, #33]
 ; CHECK-NEXT:    vmov r0, s18
-; CHECK-NEXT:    strb.w r1, [r10, #49]
 ; CHECK-NEXT:    bl __aeabi_f2ulz
 ; CHECK-NEXT:    vcmp.f32 s18, #0
-; CHECK-NEXT:    mov r9, r0
+; CHECK-NEXT:    mov r9, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r9, #0
+; CHECK-NEXT:    movlt r0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s24, #0
+; CHECK-NEXT:    vcmp.f32 s18, #0
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r9, #-1
+; CHECK-NEXT:    movgt.w r0, #-1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    str r0, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r7, #0
-; CHECK-NEXT:    vcmp.f32 s24, s20
-; CHECK-NEXT:    vcvtt.f32.f16 s16, s16
+; CHECK-NEXT:    movlt.w r9, #0
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    mov r1, r0
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r7, #65535
-; CHECK-NEXT:    movtgt r7, #3
-; CHECK-NEXT:    bfc r7, #18, #14
-; CHECK-NEXT:    mov r5, r1
-; CHECK-NEXT:    orr.w r0, r7, r9, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #29]
+; CHECK-NEXT:    movwgt r9, #65535
+; CHECK-NEXT:    movtgt r9, #3
+; CHECK-NEXT:    lsl.w r0, r9, #22
+; CHECK-NEXT:    orr.w r0, r0, r1, lsr #10
+; CHECK-NEXT:    vcvtt.f32.f16 s16, s16
+; CHECK-NEXT:    str r0, [r4, #20]
 ; CHECK-NEXT:    vmov r0, s16
-; CHECK-NEXT:    lsrs r1, r6, #10
-; CHECK-NEXT:    strb.w r1, [r10, #24]
 ; CHECK-NEXT:    bl __aeabi_f2ulz
 ; CHECK-NEXT:    vcmp.f32 s16, #0
-; CHECK-NEXT:    mov r8, r0
+; CHECK-NEXT:    mov r11, r1
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt.w r8, #0
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s22, #0
-; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r8, #-1
-; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s22, s20
-; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt.w r11, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    mov r8, r0
+; CHECK-NEXT:    vcmp.f32 s16, #0
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r11, #65535
 ; CHECK-NEXT:    movtgt r11, #3
-; CHECK-NEXT:    vcvtb.f32.f16 s22, s19
-; CHECK-NEXT:    bfc r11, #18, #14
-; CHECK-NEXT:    mov r6, r1
-; CHECK-NEXT:    orr.w r0, r11, r8, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #4]
-; CHECK-NEXT:    vmov r0, s22
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    movlt.w r8, #0
+; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    it gt
+; CHECK-NEXT:    movgt.w r8, #-1
+; CHECK-NEXT:    lsr.w r0, r8, #14
+; CHECK-NEXT:    vcvtb.f32.f16 s16, s19
+; CHECK-NEXT:    orr.w r0, r0, r11, lsl #18
+; CHECK-NEXT:    str r0, [r4, #8]
+; CHECK-NEXT:    lsr.w r0, r10, #10
+; CHECK-NEXT:    strb.w r0, [r4, #49]
+; CHECK-NEXT:    vmov r0, s16
 ; CHECK-NEXT:    bl __aeabi_f2ulz
-; CHECK-NEXT:    vcmp.f32 s22, #0
-; CHECK-NEXT:    mov r7, r0
+; CHECK-NEXT:    mov r6, r0
+; CHECK-NEXT:    vcmp.f32 s16, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s22, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r7, #0
+; CHECK-NEXT:    movlt r6, #0
+; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    ubfx r0, r5, #14, #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, #0
 ; CHECK-NEXT:    it gt
-; CHECK-NEXT:    movgt.w r7, #-1
+; CHECK-NEXT:    movgt.w r6, #-1
+; CHECK-NEXT:    orr.w r0, r0, r6, lsl #4
+; CHECK-NEXT:    str.w r0, [r4, #37]
+; CHECK-NEXT:    vcmp.f32 s24, #0
+; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r5, #0
+; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    vcmp.f32 s24, s20
+; CHECK-NEXT:    vcvtb.f32.f16 s18, s17
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r5, #65535
-; CHECK-NEXT:    movtgt r5, #3
-; CHECK-NEXT:    vcvtb.f32.f16 s18, s17
-; CHECK-NEXT:    bfc r5, #18, #14
-; CHECK-NEXT:    mov r4, r1
-; CHECK-NEXT:    lsrs r0, r5, #14
-; CHECK-NEXT:    orr.w r0, r0, r7, lsl #4
-; CHECK-NEXT:    str.w r0, [r10, #37]
-; CHECK-NEXT:    lsr.w r0, r9, #14
-; CHECK-NEXT:    orr.w r0, r0, r5, lsl #18
-; CHECK-NEXT:    str.w r0, [r10, #33]
+; CHECK-NEXT:    movwgt r0, #65535
+; CHECK-NEXT:    movtgt r0, #3
+; CHECK-NEXT:    bfc r0, #18, #14
+; CHECK-NEXT:    mov r10, r1
+; CHECK-NEXT:    orr.w r0, r0, r7, lsl #18
+; CHECK-NEXT:    str.w r0, [r4, #29]
+; CHECK-NEXT:    lsr.w r0, r9, #10
+; CHECK-NEXT:    strb r0, [r4, #24]
 ; CHECK-NEXT:    vmov r0, s18
 ; CHECK-NEXT:    bl __aeabi_f2ulz
 ; CHECK-NEXT:    vcmp.f32 s18, #0
+; CHECK-NEXT:    ubfx r2, r11, #14, #4
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r0, #0
+; CHECK-NEXT:    vcmp.f32 s18, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s16, #0
 ; CHECK-NEXT:    it gt
 ; CHECK-NEXT:    movgt.w r0, #-1
+; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
+; CHECK-NEXT:    str r2, [r4, #12]
+; CHECK-NEXT:    vcmp.f32 s22, #0
+; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
+; CHECK-NEXT:    vcmp.f32 s22, s20
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r6, #0
-; CHECK-NEXT:    vcmp.f32 s16, s20
+; CHECK-NEXT:    movlt r2, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r6, #65535
-; CHECK-NEXT:    movtgt r6, #3
-; CHECK-NEXT:    bfc r6, #18, #14
 ; CHECK-NEXT:    vcmp.f32 s18, #0
+; CHECK-NEXT:    itt gt
+; CHECK-NEXT:    movwgt r2, #65535
+; CHECK-NEXT:    movtgt r2, #3
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vcmp.f32 s18, s20
-; CHECK-NEXT:    lsr.w r2, r6, #14
-; CHECK-NEXT:    orr.w r2, r2, r0, lsl #4
-; CHECK-NEXT:    str.w r2, [r10, #12]
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    movlt r1, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
-; CHECK-NEXT:    vcmp.f32 s22, #0
+; CHECK-NEXT:    vcmp.f32 s16, #0
 ; CHECK-NEXT:    itt gt
 ; CHECK-NEXT:    movwgt r1, #65535
 ; CHECK-NEXT:    movtgt r1, #3
-; CHECK-NEXT:    lsr.w r2, r8, #14
+; CHECK-NEXT:    bfc r2, #18, #14
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    it lt
-; CHECK-NEXT:    movlt r4, #0
-; CHECK-NEXT:    vcmp.f32 s22, s20
+; CHECK-NEXT:    movlt.w r10, #0
+; CHECK-NEXT:    vcmp.f32 s16, s20
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    itt gt
-; CHECK-NEXT:    movwgt r4, #65535
-; CHECK-NEXT:    movtgt r4, #3
-; CHECK-NEXT:    orr.w r2, r2, r6, lsl #18
-; CHECK-NEXT:    str.w r2, [r10, #8]
-; CHECK-NEXT:    bfc r4, #18, #14
-; CHECK-NEXT:    ldr r3, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:    lsrs r2, r7, #28
+; CHECK-NEXT:    movwgt r10, #65535
+; CHECK-NEXT:    movtgt r10, #3
+; CHECK-NEXT:    orr.w r2, r2, r8, lsl #18
+; CHECK-NEXT:    str r2, [r4, #4]
+; CHECK-NEXT:    bfc r10, #18, #14
+; CHECK-NEXT:    ldr r3, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    lsrs r2, r6, #28
 ; CHECK-NEXT:    bfc r1, #18, #14
-; CHECK-NEXT:    orr.w r2, r2, r4, lsl #4
+; CHECK-NEXT:    orr.w r2, r2, r10, lsl #4
 ; CHECK-NEXT:    lsrs r0, r0, #28
 ; CHECK-NEXT:    orr.w r2, r2, r3, lsl #22
-; CHECK-NEXT:    str.w r2, [r10, #41]
+; CHECK-NEXT:    str.w r2, [r4, #41]
 ; CHECK-NEXT:    orr.w r0, r0, r1, lsl #4
 ; CHECK-NEXT:    ldr r1, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    orr.w r0, r0, r1, lsl #22
-; CHECK-NEXT:    str.w r0, [r10, #16]
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14}
+; CHECK-NEXT:    str r0, [r4, #16]
+; CHECK-NEXT:    add sp, #16
+; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI48_0:
 ; CHECK-NEXT:    .long 0x587fffff @ float 1.12589984E+15
     %x = call <8 x i50> @llvm.fptoui.sat.v8f16.v8i50(<8 x half> %f)
     ret <8 x i50> %x

llvm/test/CodeGen/Thumb2/thumb2-uxtb.ll

; CHECK-NO-DSP-NEXT:    bx lr
   %tmp6 = or i32 %tmp5, %tmp1   ; <i32> [#uses=1]
   ret i32 %tmp6
 }
 
 define i32 @test10(i32 %p0) {
 ; CHECK-DSP-LABEL: test10:
 ; CHECK-DSP:       @ %bb.0:
 ; CHECK-DSP-NEXT:    mov.w r1, #16253176
-; CHECK-DSP-NEXT:    and.w r0, r1, r0, lsr #7
-; CHECK-DSP-NEXT:    lsrs r1, r0, #5
-; CHECK-DSP-NEXT:    uxtb16 r1, r1
+; CHECK-DSP-NEXT:    mov.w r2, #458759
+; CHECK-DSP-NEXT:    and.w r1, r1, r0, lsr #7
+; CHECK-DSP-NEXT:    and.w r0, r2, r0, lsr #12
 ; CHECK-DSP-NEXT:    add r0, r1
 ; CHECK-DSP-NEXT:    bx lr
 ;
 ; CHECK-NO-DSP-LABEL: test10:
 ; CHECK-NO-DSP:       @ %bb.0:
 ; CHECK-NO-DSP-NEXT:    mov.w r1, #16253176
-; CHECK-NO-DSP-NEXT:    and.w r0, r1, r0, lsr #7
-; CHECK-NO-DSP-NEXT:    mov.w r1, #458759
-; CHECK-NO-DSP-NEXT:    and.w r1, r1, r0, lsr #5
+; CHECK-NO-DSP-NEXT:    mov.w r2, #458759
+; CHECK-NO-DSP-NEXT:    and.w r1, r1, r0, lsr #7
+; CHECK-NO-DSP-NEXT:    and.w r0, r2, r0, lsr #12
 ; CHECK-NO-DSP-NEXT:    add r0, r1
 ; CHECK-NO-DSP-NEXT:    bx lr
 
   %tmp1 = lshr i32 %p0, 7   ; <i32> [#uses=1]
   %tmp2 = and i32 %tmp1, 16253176   ; <i32> [#uses=2]
   %tmp4 = lshr i32 %tmp2, 5   ; <i32> [#uses=1]
   %tmp5 = and i32 %tmp4, 458759   ; <i32> [#uses=1]
   %tmp7 = or i32 %tmp5, %tmp2   ; <i32> [#uses=1]
   ret i32 %tmp7
 }

llvm/test/CodeGen/X86/ctpop-combine.ll

 ; POPCOUNT:       # %bb.0:
 ; POPCOUNT-NEXT:    andl $127, %edi
 ; POPCOUNT-NEXT:    popcntl %edi, %eax
 ; POPCOUNT-NEXT:    # kill: def $al killed $al killed $eax
 ; POPCOUNT-NEXT:    retq
 ;
 ; NO-POPCOUNT-LABEL: test4:
 ; NO-POPCOUNT:       # %bb.0:
-; NO-POPCOUNT-NEXT:    andb $127, %dil
-; NO-POPCOUNT-NEXT:    movl %edi, %eax
-; NO-POPCOUNT-NEXT:    shrb %al
-; NO-POPCOUNT-NEXT:    andb $21, %al
-; NO-POPCOUNT-NEXT:    subb %al, %dil
 ; NO-POPCOUNT-NEXT:    movl %edi, %ecx
+; NO-POPCOUNT-NEXT:    andb $127, %cl
+; NO-POPCOUNT-NEXT:    shrb %dil
+; NO-POPCOUNT-NEXT:    andb $21, %dil
+; NO-POPCOUNT-NEXT:    subb %dil, %cl
+; NO-POPCOUNT-NEXT:    movl %ecx, %eax
+; NO-POPCOUNT-NEXT:    andb $51, %al
+; NO-POPCOUNT-NEXT:    shrb $2, %cl
 ; NO-POPCOUNT-NEXT:    andb $51, %cl
-; NO-POPCOUNT-NEXT:    shrb $2, %dil
-; NO-POPCOUNT-NEXT:    andb $51, %dil
-; NO-POPCOUNT-NEXT:    addb %dil, %cl
+; NO-POPCOUNT-NEXT:    addb %al, %cl
 ; NO-POPCOUNT-NEXT:    movl %ecx, %eax
 ; NO-POPCOUNT-NEXT:    shrb $4, %al
 ; NO-POPCOUNT-NEXT:    addb %cl, %al
 ; NO-POPCOUNT-NEXT:    andb $15, %al
 ; NO-POPCOUNT-NEXT:    retq
   %x2 = and i8 %x, 127
   %count = tail call i8 @llvm.ctpop.i8(i8 %x2)
   %and = and i8 %count, 7

llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll

 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzwl (%eax), %edx
 ; X86-NEXT:    movzbl 2(%eax), %ecx
 ; X86-NEXT:    movb %cl, 2(%eax)
 ; X86-NEXT:    shll $16, %ecx
 ; X86-NEXT:    orl %edx, %ecx
 ; X86-NEXT:    orl $384, %ecx # imm = 0x180
-; X86-NEXT:    andl $16777088, %ecx # imm = 0xFFFF80
+; X86-NEXT:    andl $-128, %ecx
 ; X86-NEXT:    movw %cx, (%eax)
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i24_and_or:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
 ; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    orl $384, %ecx # imm = 0x180
-; X64-NEXT:    andl $16777088, %ecx # imm = 0xFFFF80
+; X64-NEXT:    andl $-128, %ecx
 ; X64-NEXT:    movw %cx, (%rdi)
 ; X64-NEXT:    retq
   %b = load i24, i24* %a, align 1
   %c = and i24 %b, -128
   %d = or i24 %c, 384
   store i24 %d, i24* %a, align 1
   ret void
 }

 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 6(%rdi), %ecx
 ; X64-NEXT:    movb %cl, 6(%rdi)
 ; X64-NEXT:    # kill: def $ecx killed $ecx killed $rcx def $rcx
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    shlq $32, %rcx
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    orq %rcx, %rax
-; X64-NEXT:    orq $384, %rax # imm = 0x180
-; X64-NEXT:    movl %eax, (%rdi)
-; X64-NEXT:    shrq $32, %rax
+; X64-NEXT:    movl (%rdi), %edx
+; X64-NEXT:    orq %rcx, %rdx
+; X64-NEXT:    orq $384, %rdx # imm = 0x180
 ; X64-NEXT:    movw %ax, 4(%rdi)
+; X64-NEXT:    movl %edx, (%rdi)
 ; X64-NEXT:    retq
   %aa = load i56, i56* %a, align 1
   %b = or i56 %aa, 384
   store i56 %b, i56* %a, align 1
   ret void
 }
 
 define void @i56_and_or(i56* %a) {

 ; X64-NEXT:    movl %esi, %eax
 ; X64-NEXT:    movzwl 4(%rdi), %ecx
 ; X64-NEXT:    movzbl 6(%rdi), %edx
 ; X64-NEXT:    movb %dl, 6(%rdi)
 ; X64-NEXT:    # kill: def $edx killed $edx killed $rdx def $rdx
 ; X64-NEXT:    shll $16, %edx
 ; X64-NEXT:    orl %ecx, %edx
 ; X64-NEXT:    shlq $32, %rdx
-; X64-NEXT:    movl (%rdi), %ecx
-; X64-NEXT:    orq %rdx, %rcx
+; X64-NEXT:    movl (%rdi), %esi
+; X64-NEXT:    orq %rdx, %rsi
 ; X64-NEXT:    shlq $13, %rax
 ; X64-NEXT:    movabsq $72057594037919743, %rdx # imm = 0xFFFFFFFFFFDFFF
-; X64-NEXT:    andq %rcx, %rdx
+; X64-NEXT:    andq %rsi, %rdx
 ; X64-NEXT:    orq %rax, %rdx
+; X64-NEXT:    movw %cx, 4(%rdi)
 ; X64-NEXT:    movl %edx, (%rdi)
-; X64-NEXT:    shrq $32, %rdx
-; X64-NEXT:    movw %dx, 4(%rdi)
 ; X64-NEXT:    retq
   %extbit = zext i1 %bit to i56
   %b = load i56, i56* %a, align 1
   %extbit.shl = shl nuw nsw i56 %extbit, 13
   %c = and i56 %b, -8193
   %d = or i56 %c, %extbit.shl
   store i56 %d, i56* %a, align 1
   ret void
 }
 

llvm/test/CodeGen/X86/ins_subreg_coalesce-1.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -mattr=-bmi | FileCheck %s
 
 define fastcc i32 @t() nounwind  {
 ; CHECK-LABEL: t:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movzwl 0, %eax
-; CHECK-NEXT:    orl $2, %eax
-; CHECK-NEXT:    movw %ax, 0
+; CHECK-NEXT:    movl %eax, %ecx
+; CHECK-NEXT:    orl $2, %ecx
+; CHECK-NEXT:    movw %cx, 0
 ; CHECK-NEXT:    shrl $3, %eax
 ; CHECK-NEXT:    andl $1, %eax
 ; CHECK-NEXT:    retl
 entry:
   br i1 false, label %UnifiedReturnBlock, label %bb4
 bb4:    ; preds = %entry
   br i1 false, label %bb17, label %bb22
 bb17:   ; preds = %bb4

llvm/test/CodeGen/X86/load-local-v3i129.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 
 define void @_start() {
 ; CHECK-LABEL: _start:
 ; CHECK:       # %bb.0: # %Entry
 ; CHECK-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-NEXT:    shrdq $2, %rcx, %rax
+; CHECK-NEXT:    movq %rcx, %rdx
+; CHECK-NEXT:    shlq $62, %rdx
 ; CHECK-NEXT:    shrq $2, %rcx
-; CHECK-NEXT:    leaq 1(,%rax,4), %rdx
-; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrdq $62, %rcx, %rax
+; CHECK-NEXT:    shldq $2, %rdx, %rcx
+; CHECK-NEXT:    andq $-4, %rax
+; CHECK-NEXT:    orq $1, %rax
 ; CHECK-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    orq $-2, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movq $-1, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    retq
 Entry:
   %y = alloca <3 x i129>, align 4
   %L = load <3 x i129>, <3 x i129>* %y
   %I1 = insertelement <3 x i129> %L, i129 340282366920938463463374607431768211455, i32 1
   store <3 x i129> %I1, <3 x i129>* %y
   ret void
 }

llvm/test/CodeGen/X86/load-local-v4i5.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 @0 = internal unnamed_addr constant [4 x i5] [i5 2, i5 0, i5 2, i5 -1], align 1
 
 ; Function Attrs: nobuiltin nounwind
 define void @_start() {
 ; CHECK-LABEL: _start:
 ; CHECK:       # %bb.0: # %Entry
 ; CHECK-NEXT:    movl __unnamed_1(%rip), %eax
 ; CHECK-NEXT:    movl %eax, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb -{{[0-9]+}}(%rsp), %cl
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %edx
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %esi
+; CHECK-NEXT:    movzbl %cl, %edi
+; CHECK-NEXT:    shrb %cl
+; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    andl $31, %eax
 ; CHECK-NEXT:    andl $31, %esi
 ; CHECK-NEXT:    shll $5, %esi
 ; CHECK-NEXT:    orl %eax, %esi
 ; CHECK-NEXT:    andl $31, %edx
 ; CHECK-NEXT:    shll $10, %edx
 ; CHECK-NEXT:    orl %esi, %edx
-; CHECK-NEXT:    movzbl %cl, %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    shll $15, %ecx
-; CHECK-NEXT:    orl %edx, %ecx
-; CHECK-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrl $16, %ecx
-; CHECK-NEXT:    andl $15, %ecx
-; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movb %al, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    cmpb $31, %al
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shll $15, %eax
+; CHECK-NEXT:    orl %edx, %eax
+; CHECK-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    cmpb $31, %dil
 ; CHECK-NEXT:    je .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %Then
 ; CHECK-NEXT:    int3
 ; CHECK-NEXT:  .LBB0_2: # %EndIf
 ; CHECK-NEXT:    retq
 Entry:
   %x = alloca [4 x i5], align 1
   %y = alloca <4 x i5>, align 4

llvm/test/CodeGen/X86/masked_compressstore.ll

 ; AVX1OR2-NEXT:    retq
 ;
 ; AVX512F-LABEL: compressstore_v16f64_v16i1:
 ; AVX512F:       ## %bb.0:
 ; AVX512F-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; AVX512F-NEXT:    vpslld $31, %zmm2, %zmm2
 ; AVX512F-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; AVX512F-NEXT:    kmovw %k1, %eax
-; AVX512F-NEXT:    movzbl %al, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    shrl %ecx
-; AVX512F-NEXT:    andl $-43, %ecx
-; AVX512F-NEXT:    subl %ecx, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
-; AVX512F-NEXT:    shrl $2, %eax
+; AVX512F-NEXT:    movzbl %al, %ecx
+; AVX512F-NEXT:    shrl %eax
+; AVX512F-NEXT:    andl $85, %eax
+; AVX512F-NEXT:    subl %eax, %ecx
+; AVX512F-NEXT:    movl %ecx, %eax
 ; AVX512F-NEXT:    andl $858993459, %eax ## imm = 0x33333333
-; AVX512F-NEXT:    addl %ecx, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    shrl $4, %ecx
+; AVX512F-NEXT:    shrl $2, %ecx
+; AVX512F-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512F-NEXT:    addl %eax, %ecx
-; AVX512F-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
-; AVX512F-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
+; AVX512F-NEXT:    movl %ecx, %eax
+; AVX512F-NEXT:    shrl $4, %eax
+; AVX512F-NEXT:    addl %ecx, %eax
+; AVX512F-NEXT:    andl $252645135, %eax ## imm = 0xF0F0F0F
+; AVX512F-NEXT:    imull $16843009, %eax, %eax ## imm = 0x1010101
 ; AVX512F-NEXT:    shrl $24, %eax
 ; AVX512F-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512F-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
 ; AVX512F-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512VLDQ-LABEL: compressstore_v16f64_v16i1:

 ; AVX512VLDQ-NEXT:    vzeroupper
 ; AVX512VLDQ-NEXT:    retq
 ;
 ; AVX512VLBW-LABEL: compressstore_v16f64_v16i1:
 ; AVX512VLBW:       ## %bb.0:
 ; AVX512VLBW-NEXT:    vpsllw $7, %xmm2, %xmm2
 ; AVX512VLBW-NEXT:    vpmovb2m %xmm2, %k1
 ; AVX512VLBW-NEXT:    kmovd %k1, %eax
-; AVX512VLBW-NEXT:    movzbl %al, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    shrl %ecx
-; AVX512VLBW-NEXT:    andl $-43, %ecx
-; AVX512VLBW-NEXT:    subl %ecx, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
-; AVX512VLBW-NEXT:    shrl $2, %eax
+; AVX512VLBW-NEXT:    movzbl %al, %ecx
+; AVX512VLBW-NEXT:    shrl %eax
+; AVX512VLBW-NEXT:    andl $85, %eax
+; AVX512VLBW-NEXT:    subl %eax, %ecx
+; AVX512VLBW-NEXT:    movl %ecx, %eax
 ; AVX512VLBW-NEXT:    andl $858993459, %eax ## imm = 0x33333333
-; AVX512VLBW-NEXT:    addl %ecx, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    shrl $4, %ecx
+; AVX512VLBW-NEXT:    shrl $2, %ecx
+; AVX512VLBW-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512VLBW-NEXT:    addl %eax, %ecx
-; AVX512VLBW-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
-; AVX512VLBW-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
+; AVX512VLBW-NEXT:    movl %ecx, %eax
+; AVX512VLBW-NEXT:    shrl $4, %eax
+; AVX512VLBW-NEXT:    addl %ecx, %eax
+; AVX512VLBW-NEXT:    andl $252645135, %eax ## imm = 0xF0F0F0F
+; AVX512VLBW-NEXT:    imull $16843009, %eax, %eax ## imm = 0x1010101
 ; AVX512VLBW-NEXT:    shrl $24, %eax
 ; AVX512VLBW-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512VLBW-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
 ; AVX512VLBW-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512VLBW-NEXT:    vzeroupper
 ; AVX512VLBW-NEXT:    retq
   call void @llvm.masked.compressstore.v16f64(<16 x double> %V, double* %base, <16 x i1> %mask)
   ret void

llvm/test/CodeGen/X86/mul128.ll

 }
 
 @aaa = external dso_local global i128
 @bbb = external dso_local global i128
 
 define void @PR13897() nounwind {
 ; X64-LABEL: PR13897:
 ; X64:       # %bb.0: # %"0x0"
-; X64-NEXT:    movl bbb(%rip), %eax
-; X64-NEXT:    movq %rax, %rcx
-; X64-NEXT:    shlq $32, %rcx
-; X64-NEXT:    orq %rax, %rcx
-; X64-NEXT:    movq %rcx, aaa+8(%rip)
-; X64-NEXT:    movq %rcx, aaa(%rip)
+; X64-NEXT:    movq bbb(%rip), %rax
+; X64-NEXT:    movl %eax, %ecx
+; X64-NEXT:    shlq $32, %rax
+; X64-NEXT:    orq %rcx, %rax
+; X64-NEXT:    movq %rax, aaa+8(%rip)
+; X64-NEXT:    movq %rax, aaa(%rip)
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: PR13897:
 ; X86:       # %bb.0: # %"0x0"
 ; X86-NEXT:    movl bbb, %eax
 ; X86-NEXT:    movl %eax, aaa+12
 ; X86-NEXT:    movl %eax, aaa+8
 ; X86-NEXT:    movl %eax, aaa+4

llvm/test/CodeGen/X86/shift-mask.ll

; X64-BMI2-NEXT:    retq
   %2 = lshr i64 %1, 3
   ret i64 %2
 }
 
 define i64 @test_i64_lshr_lshr_1(i64 %a0) {
 ; X86-LABEL: test_i64_lshr_lshr_1:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    shldl $3, %eax, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    leal (,%ecx,8), %edx
+; X86-NEXT:    shldl $3, %eax, %ecx
 ; X86-NEXT:    shll $3, %eax
-; X86-NEXT:    shrdl $5, %edx, %eax
+; X86-NEXT:    shrdl $5, %ecx, %eax
 ; X86-NEXT:    shrl $5, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_i64_lshr_lshr_1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    leaq (,%rdi,8), %rax
 ; X64-NEXT:    shrq $5, %rax
 ; X64-NEXT:    retq

llvm/test/CodeGen/X86/udiv_fix_sat.ll

 ; X64-NEXT:    shrl %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $rax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: func7:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movzwl %cx, %ecx
-; X86-NEXT:    addl %ecx, %ecx
 ; X86-NEXT:    movl %ecx, %edx
-; X86-NEXT:    shrl $16, %edx
-; X86-NEXT:    shll $16, %ecx
+; X86-NEXT:    shll $17, %edx
+; X86-NEXT:    shrl $15, %ecx
+; X86-NEXT:    andl $1, %ecx
 ; X86-NEXT:    pushl $0
 ; X86-NEXT:    pushl %eax
-; X86-NEXT:    pushl %edx
 ; X86-NEXT:    pushl %ecx
+; X86-NEXT:    pushl %edx
 ; X86-NEXT:    calll __udivdi3
 ; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:    cmpl $131071, %eax # imm = 0x1FFFF
 ; X86-NEXT:    movl $131071, %ecx # imm = 0x1FFFF
 ; X86-NEXT:    cmovael %ecx, %eax
 ; X86-NEXT:    testl %edx, %edx
 ; X86-NEXT:    cmovnel %ecx, %eax
 ; X86-NEXT:    shrl %eax

llvm/test/CodeGen/X86/vector-mulfix-legalize.ll

 ; CHECK-NEXT:    sarl $16, %ecx
 ; CHECK-NEXT:    cmpl $16384, %ecx # imm = 0x4000
 ; CHECK-NEXT:    movl $32767, %r8d # imm = 0x7FFF
 ; CHECK-NEXT:    cmovgel %r8d, %edx
 ; CHECK-NEXT:    cmpl $-16384, %ecx # imm = 0xC000
 ; CHECK-NEXT:    movl $32768, %ecx # imm = 0x8000
 ; CHECK-NEXT:    cmovll %ecx, %edx
 ; CHECK-NEXT:    pextrw $1, %xmm0, %esi
-; CHECK-NEXT:    movswl %si, %edi
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    shrl $16, %eax
-; CHECK-NEXT:    leal (%rdi,%rdi), %esi
-; CHECK-NEXT:    shrdw $15, %ax, %si
-; CHECK-NEXT:    sarl $15, %edi
-; CHECK-NEXT:    cmpl $16384, %edi # imm = 0x4000
+; CHECK-NEXT:    leal (%rsi,%rsi), %edi
+; CHECK-NEXT:    movswl %si, %eax
+; CHECK-NEXT:    movl %eax, %esi
+; CHECK-NEXT:    shrl $16, %esi
+; CHECK-NEXT:    shldw $1, %di, %si
+; CHECK-NEXT:    sarl $15, %eax
+; CHECK-NEXT:    cmpl $16384, %eax # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %esi
-; CHECK-NEXT:    cmpl $-16384, %edi # imm = 0xC000
+; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    movd %xmm0, %eax
 ; CHECK-NEXT:    cwtl
 ; CHECK-NEXT:    movl %eax, %edi
 ; CHECK-NEXT:    shrl $16, %edi
 ; CHECK-NEXT:    shldw $1, %ax, %di
 ; CHECK-NEXT:    sarl $16, %eax
 ; CHECK-NEXT:    cmpl $16384, %eax # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %edi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %edi
 ; CHECK-NEXT:    movzwl %di, %eax
 ; CHECK-NEXT:    movd %eax, %xmm1
 ; CHECK-NEXT:    pinsrw $1, %esi, %xmm1
 ; CHECK-NEXT:    pinsrw $2, %edx, %xmm1
 ; CHECK-NEXT:    pextrw $3, %xmm0, %eax
+; CHECK-NEXT:    leal (,%rax,4), %edx
 ; CHECK-NEXT:    cwtl
-; CHECK-NEXT:    movl %eax, %edx
-; CHECK-NEXT:    shrl $14, %edx
-; CHECK-NEXT:    leal (,%rax,4), %esi
-; CHECK-NEXT:    shrdw $15, %dx, %si
+; CHECK-NEXT:    movl %eax, %esi
+; CHECK-NEXT:    shrl $14, %esi
+; CHECK-NEXT:    shldw $1, %dx, %si
 ; CHECK-NEXT:    sarl $14, %eax
 ; CHECK-NEXT:    cmpl $16384, %eax # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %esi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    pinsrw $3, %esi, %xmm1
 ; CHECK-NEXT:    movdqa %xmm1, %xmm0
 ; CHECK-NEXT:    retq