[DAG] Enable ISD::SRL SimplifyMultipleUseDemandedBits handling inside SimplifyDemandedBits

[RKSimon]

Summary

This patch allows SimplifyDemandedBits to call SimplifyMultipleUseDemandedBits in cases where the ISD::SRL source operand has other uses, enabling us to peek through the shifted value if we don't demand all the bits/elts.

This is another step towards removing SelectionDAG::GetDemandedBits and just using TargetLowering::SimplifyMultipleUseDemandedBits.

There a few cases where we end up with extra register moves which I think we can accept in exchange for the increased ILP.

[RKSimon]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

@arsenm @foad Not sure if pulling out the immediate is a good idea or not - shouldn't a u16 immediate be cheap?

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

This is worse. Integer constants -16 to 64 and a handful of FP values are free, but 0xffff is not so it requires materialization.

[RKSimon]

rebase

[RKSimon]

still looking at the remaining regressions

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433

Looks like we've defeated the RISCVISD::GORCI matching code

[craig.topper]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433

Running the tests through instcombine also breaks GORCI matching.

[craig.topper]

llvm/test/CodeGen/RISCV/rv64Zbp.ll

1433

It's also worth noting, the tests that are failing are repeating the same pattern gorc pattern twice, which is redundant. The test was trying to test that we could detect the redundancy. I guess this patch may have seen some of the redundancy?

[RKSimon]

rebase (still needs work)

[RKSimon]

I've raised https://bugs.llvm.org/show_bug.cgi?id=51209 about the poor quality of the gorc2 pattern matching and the gorc2, gorc2 -> gorc2 tests.

[craig.topper]

@RKSimon are the other problems with this patch than just the GORCI matching?

[RKSimon]

In D77804#3073304, @craig.topper wrote:

@RKSimon are the other problems with this patch than just the GORCI matching?

The GORCI matching is the main one.

There is also some minor issues with MatchRotate - we should be allowed to match rotate/funnel by constant pre-legalization (see ARM/ror.ll) as that can be re-expanded later without any harm done, before we see through the pattern and lose it, although now that we match this quite well in InstCombine I'm not sure is this is as likely to happen.

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase - squashed a few more regressions...

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

rebase

[RKSimon]

Waiting for D124839 to land

[RKSimon]

Rebased after D124839 to just handle ISD::SRL shifts

[RKSimon]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

@arsenm @foad At EuroLLVM Matt suggested that maybe we should increase the tolerance to 2 uses of the large immediates before pulling out the constant?

llvm/test/CodeGen/ARM/uxtb.ll

112

I'm going to take a look at this, but I'm really not familiar with the UXTB matching code, so any pointers would be appreciated.

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

s_mov_b32 K + 2 * v_and_b32_32 = 16 bytes, 12 cycles
2 * (v_and_b32_e32 K) = 16 bytes, 8 cycles which is clearly better.

3 * (v_and_b32_e32 K) = 24 bytes, 12 cycles

So 2 uses of a constant seems plainly better for VOP1/VOP2 ops. Abbe that it becomes a code size vs. latency tradeoff

[arsenm]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

This decision is also generally made by SIFoldOperands. Probably need to fix it there and not in the DAG

[foad]

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

148

I'm strongly in favour of never pulling out the constant (or rather, always folding into the instruction) and I have patches to that effect starting with D114643, which I'm hoping to get back to pretty soon.

[RKSimon]

rebase

[foad]

AMDGPU changes LGTM.

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

instcombine optimises this as well:

define i32 @test10(i32 %p0) {
  %tmp1 = lshr i32 %p0, 7
  %tmp2 = and i32 %tmp1, 16253176
  %tmp4 = lshr i32 %p0, 12
  %tmp5 = and i32 %tmp4, 458759
  %tmp7 = or i32 %tmp5, %tmp2
  ret i32 %tmp7
}

which has the same problem:

_test10:
@ %bb.0:
        mov     r1, #248
        mov     r2, #7
        orr     r1, r1, #16252928
        orr     r2, r2, #458752
        and     r1, r1, r0, lsr #7
        and     r0, r2, r0, lsr #12
        orr     r0, r0, r1
        bx      lr

[RKSimon]

llvm/test/CodeGen/Thumb2/thumb2-uxtb.ll

175

same problem - instcombine will have already optimized this to:

define i32 @test10(i32 %p0) {
  %tmp1 = lshr i32 %p0, 7
  %tmp2 = and i32 %tmp1, 16253176
  %tmp4 = lshr i32 %p0, 12
  %tmp5 = and i32 %tmp4, 458759
  %tmp7 = or i32 %tmp5, %tmp2
  ret i32 %tmp7
}

It feels like I'm avoiding the issue - but should I update the arm/thumb2 UXTB16 tests to match what the middle-end will have generated?

[dmgreen]

llvm/test/CodeGen/ARM/uxtb.ll

112

I was taking a look. The test is super old now, so old that it had signed types when it was originally added.

I was surprised to see that and 0x70007 is being recognised via an and 0xff00ff tablegen pattern - it goes into SelectionDAGISel::CheckAndMask which checks that the other mask bits are already 0.

I think that is what this is trying to test - that a smaller and mask still matches the UXTB16. Is it possible to change it to something that still captures that, without relying on the multi-use fold of the %tmp2 not happening?

Maybe something like this?

%p = and i32 %p0, 3
%a = shl i32 65537, %p
%b = lshr i32 %a, 1
%tmp7 = and i32 %b, 458759

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

Thanks for the hint - I'll give it a try

[RKSimon]

rebase with alternative uxtb16 tests

[RKSimon]

llvm/test/CodeGen/ARM/uxtb.ll

112

Thanks @dmgreen - those still match fine. Should I pre-commit these new tests and possibly alter the existing test10 variants with the -instcombine optimized IR to show they already fail to match?

[dmgreen]

llvm/test/CodeGen/ARM/uxtb.ll

112

That sounds good to me.

[RKSimon]

rebase

[RKSimon]

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

128–139

@jonpa @uweigand These tests are proving very fragile depending on the order of and/shifts - should SystemZ be preferring masking leading/trailing bits with shift-pairs over shift+and / and+shift do you think? We have TLI::shouldFoldConstantShiftPairToMask to hand that.

[uweigand]

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

128–139

Well, this specific test only loads and then saves unmodified a 3xi31 vector, so ideally however the masking is done, it should be optimized away as unnecessary in either case. That's what currently happens, not sure why this is changing with this PR.

In general, I think using an and-mask would be preferable over a shift pair on SystemZ.

[RKSimon]

Thanks @uweigand I'll take another look at this soon

[RKSimon]

rebase after D125836

[RKSimon]

rebase

[RKSimon]

rebase and prefer SimplifyDemandedBits over GetDemandedBits for trunc stores

[RKSimon]

Added (or (and X, C1), (and (or X, Y), C2)) -> (or (and X, C1|C2), (and Y, C2)) fold to try to reduce the SystemZ regression

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

This could be a preliminary patch. I don't think we'd get that in IR either (even without extra uses):
https://alive2.llvm.org/ce/z/g61VRe

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

If I'm reading the SystemZ debug spew correctly, we should have gotten this transform to fire twice, so it would do this:
https://alive2.llvm.org/ce/z/tUsepa
...but we miss it because we don't revisit the last 'or' node? Is that what D127115 would solve?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6871 ↗

(On Diff #444009)

I've confirmed that D127115 solves the SystemZ fun3 regression but not fun2

[uweigand]

IMO the fun2 regression probably shouldn't block the patch from being merged. I've looked into the sequences, and actually neither of them is even close to optimal.

Looking at the semantics, we have 8 x i32 inputs, which need to be truncated to i31, concatenated, and then stored, occupying 31 bytes of memory. Memory is written via three 8-byte stores, followed by a 4-byte, a 2-byte, and a 1-byte store, which does look optimal to me. However, the computation of the 64-bit values to be stored is not.

The first of these should be the value

(A << 33) | ((B << 2) & 0x1fffffffc) | ((C >> 29) & 3)

where A, B, and C are the first three i32 inputs.

However, the computation being performed is more like

((A << 25) | ((B >> 6) & 0x01ffffff)) << 8
| ((B << 58) | ((C & 0x7fffffff) << 27)) >> 56

which gets the correct result, but in about double the number of instructions or cycles that should be required.

While the variant with this PR is even slightly worse than the variant before, that's probably not really relevant given the fact both sequences are rather inefficient. Ideally, we could fix this to get (close to) an optimal sequence, but that would be a different issue. (I'm not even sure yet whether the current inefficiency is due to the middle end or the back end.)

[RKSimon]

Thanks - I have a lot of individual DAG / SimplifyDemanded patches in progress atm, plus we're now getting closer to completing D127115.

A few patches still have minor regressions that I'm addressing, but this one in particular I've been wondering how much of a real world issue illegal type copies like this actually are? If we were further away from 15.x branch I'd ask to get this in and we ensure we address it once all the patches are in, but given how close we are I'm going to wait for now.

[RKSimon]

I think I've covered all the remaining regressions now - D129765 has cleaned up a number of annoying cases - including the SystemZ v3i31 copy test!

[uweigand]

In D77804#3681806, @RKSimon wrote:

I think I've covered all the remaining regressions now - D129765 has cleaned up a number of annoying cases - including the SystemZ v3i31 copy test!

Thanks! SystemZ changes LGTM now as discussed above.

[RKSimon]

I think this is patch is good to go now - any more comments?

[foad]

AMDGPU changes still LGTM.

[spatel]

x86 diffs LGTM

llvm/test/CodeGen/X86/ins_subreg_coalesce-1.ll

8–10

Not sure if this test still models some situation that we care about, but you could put a TODO note on it (don't need to copy to %ecx?).

[yonghong-song]

Hi, we found a regression with some bpf code with this patch. The following shows the problem:

[$ ~/tmp] cat run.sh
/home/yhs/work/llvm-project/llvm/build.cur/install/bin/clang -target bpf -O2 -g -c t.c
[$ ~/tmp] cat t.c
typedef unsigned char u8;
struct event {
  u8 tag;
  u8 hostname[84];
};

void *g;
void bar(void *);

int foo() {
  struct event event = {};

  event.tag = 1;
  __builtin_memcpy(&event.hostname, g, 84);
  bar(&event);
  return 0;
}
[$ ~/tmp] ./run.sh
t.c:14:3: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

  __builtin_memcpy(&event.hostname, g, 84);
  ^
t.c:14:3: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

2 errors generated.
[$ ~/tmp]

The BPF program enforces the stack size <= 512 bytes. For the above program, with this patch, the code after dag insn selection is worse and eventually in register allocation stage, the stack size is more than 512 and caused the above issue.

To illustrate the problem in more details, without this patch, the lowered machine code looks like

  STB killed %7:gpr, %stack.1.event.i, 0, debug-location !21355 :: (store (s8) into %ir.event.i, align 8, !tbaa !21356); tracecon/src/bpf/tracecon.bpf.c:78:12 @[ tracecon/src/b
pf/tracecon.bpf.c:68:5 ]
  %8:gpr = LDB %6:gpr, 7, debug-location !21358 :: (load (s8) from %ir.call1.i + 7); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %8:gpr, %stack.1.event.i, 12, debug-location !21358 :: (store (s8) into %ir.hostname.i + 7); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %9:gpr = LDB %6:gpr, 6, debug-location !21358 :: (load (s8) from %ir.call1.i + 6); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %9:gpr, %stack.1.event.i, 11, debug-location !21358 :: (store (s8) into %ir.hostname.i + 6); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %10:gpr = LDB %6:gpr, 5, debug-location !21358 :: (load (s8) from %ir.call1.i + 5); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %10:gpr, %stack.1.event.i, 10, debug-location !21358 :: (store (s8) into %ir.hostname.i + 5); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf
.c:68:5 ]
  %11:gpr = LDB %6:gpr, 4, debug-location !21358 :: (load (s8) from %ir.call1.i + 4); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %11:gpr, %stack.1.event.i, 9, debug-location !21358 :: (store (s8) into %ir.hostname.i + 4); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %12:gpr = LDB %6:gpr, 3, debug-location !21358 :: (load (s8) from %ir.call1.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %12:gpr, %stack.1.event.i, 8, debug-location !21358 :: (store (s8) into %ir.hostname.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %13:gpr = LDB %6:gpr, 2, debug-location !21358 :: (load (s8) from %ir.call1.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %13:gpr, %stack.1.event.i, 7, debug-location !21358 :: (store (s8) into %ir.hostname.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %14:gpr = LDB %6:gpr, 1, debug-location !21358 :: (load (s8) from %ir.call1.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %14:gpr, %stack.1.event.i, 6, debug-location !21358 :: (store (s8) into %ir.hostname.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.
c:68:5 ]
  %15:gpr = LDB %6:gpr, 0, debug-location !21358 :: (load (s8) from %ir.call1.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %15:gpr, %stack.1.event.i, 5, debug-location !21358 :: (store (s8) into %ir.hostname.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68
:5 ]
  %16:gpr = LDB %6:gpr, 15, debug-location !21358 :: (load (s8) from %ir.call1.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %16:gpr, %stack.1.event.i, 20, debug-location !21358 :: (store (s8) into %ir.hostname.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bp
f.c:68:5 ]
  %17:gpr = LDB %6:gpr, 14, debug-location !21358 :: (load (s8) from %ir.call1.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %17:gpr, %stack.1.event.i, 19, debug-location !21358 :: (store (s8) into %ir.hostname.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bp
f.c:68:5 ]
...
  %88:gpr = LDB %6:gpr, 83, debug-location !21358 :: (load (s8) from %ir.call1.i + 83); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %88:gpr, %stack.1.event.i, 88, debug-location !21358 :: (store (s8) into %ir.hostname.i + 83); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %89:gpr = LDB %6:gpr, 82, debug-location !21358 :: (load (s8) from %ir.call1.i + 82); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %89:gpr, %stack.1.event.i, 87, debug-location !21358 :: (store (s8) into %ir.hostname.i + 82); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %90:gpr = LDB %6:gpr, 81, debug-location !21358 :: (load (s8) from %ir.call1.i + 81); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %90:gpr, %stack.1.event.i, 86, debug-location !21358 :: (store (s8) into %ir.hostname.i + 81); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %91:gpr = LDB %6:gpr, 80, debug-location !21358 :: (load (s8) from %ir.call1.i + 80); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB killed %91:gpr, %stack.1.event.i, 85, debug-location !21358 :: (store (s8) into %ir.hostname.i + 80); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]

Pretty straightforward byte load and stores and the corresponding stack after register allocation,

# *** IR Dump After Greedy Register Allocator (greedy) ***:
# Machine code for function tcp_v4_connect_exit: NoPHIs, TracksLiveness, TiedOpsRewritten, TracksDebugUserValues
Frame Objects:
  fi#0: size=4, align=4, at location [SP]
  fi#1: size=89, align=8, at location [SP]
  fi#2: size=4, align=4, at location [SP]
Function Live Ins: $r1 in %0

But this patch, the code becomes very complex,

  %8:gpr = LDB %6:gpr, 71, debug-location !21358 :: (load (s8) from %ir.call1.i + 71); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %9:gpr = SLL_ri %8:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %10:gpr = LDB %6:gpr, 70, debug-location !21358 :: (load (s8) from %ir.call1.i + 70); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %11:gpr = OR_rr %9:gpr(tied-def 0), killed %10:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %12:gpr = LDB %6:gpr, 15, debug-location !21358 :: (load (s8) from %ir.call1.i + 15); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %13:gpr = SLL_ri %12:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %14:gpr = LDB %6:gpr, 14, debug-location !21358 :: (load (s8) from %ir.call1.i + 14); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %15:gpr = OR_rr %13:gpr(tied-def 0), killed %14:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %71:gpr = OR_rr %69:gpr(tied-def 0), killed %70:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %72:gpr = LDB %6:gpr, 77, debug-location !21358 :: (load (s8) from %ir.call1.i + 77); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %73:gpr = SLL_ri %72:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %74:gpr = LDB %6:gpr, 76, debug-location !21358 :: (load (s8) from %ir.call1.i + 76); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %75:gpr = OR_rr %73:gpr(tied-def 0), killed %74:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %76:gpr = LDB %6:gpr, 79, debug-location !21358 :: (load (s8) from %ir.call1.i + 79); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %77:gpr = SLL_ri %76:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %78:gpr = LDB %6:gpr, 78, debug-location !21358 :: (load (s8) from %ir.call1.i + 78); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %79:gpr = OR_rr %77:gpr(tied-def 0), killed %78:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %80:gpr = SLL_ri %27:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %81:gpr = OR_rr %80:gpr(tied-def 0), killed %23:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %82:gpr = SLL_ri %19:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %83:gpr = OR_rr %82:gpr(tied-def 0), killed %71:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %84:gpr = SLL_ri %15:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %85:gpr = OR_rr %84:gpr(tied-def 0), killed %67:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %86:gpr = SLL_ri %11:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %87:gpr = OR_rr %86:gpr(tied-def 0), killed %63:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %88:gpr = SLL_ri %43:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %100:gpr = LDB %6:gpr, 74, debug-location !21358 :: (load (s8) from %ir.call1.i + 74); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %101:gpr = LDB %6:gpr, 75, debug-location !21358 :: (load (s8) from %ir.call1.i + 75); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %102:gpr = SLL_ri %101:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %103:gpr = OR_rr %102:gpr(tied-def 0), %100:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %104:gpr = SLL_ri %103:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %105:gpr = OR_rr %104:gpr(tied-def 0), killed %99:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %106:gpr = SLL_ri %79:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %107:gpr = OR_rr %106:gpr(tied-def 0), killed %75:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %108:gpr = LDB %6:gpr, 64, debug-location !21358 :: (load (s8) from %ir.call1.i + 64); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %109:gpr = LDB %6:gpr, 65, debug-location !21358 :: (load (s8) from %ir.call1.i + 65); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %110:gpr = SLL_ri %109:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %111:gpr = OR_rr %110:gpr(tied-def 0), %108:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %112:gpr = LDB %6:gpr, 66, debug-location !21358 :: (load (s8) from %ir.call1.i + 66); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %113:gpr = LDB %6:gpr, 67, debug-location !21358 :: (load (s8) from %ir.call1.i + 67); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %114:gpr = SLL_ri %113:gpr(tied-def 0), 8, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %115:gpr = OR_rr %114:gpr(tied-def 0), %112:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %116:gpr = SLL_ri %115:gpr(tied-def 0), 16, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %117:gpr = OR_rr %116:gpr(tied-def 0), killed %111:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
...
  %225:gpr = OR_rr %224:gpr(tied-def 0), killed %117:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %226:gpr = SLL_ri %107:gpr(tied-def 0), 32, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  %227:gpr = OR_rr %226:gpr(tied-def 0), killed %105:gpr, debug-location !21358; tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %193:gpr, %stack.1.event.i, 8, debug-location !21358 :: (store (s8) into %ir.hostname.i + 3); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5
 ]
  STB %192:gpr, %stack.1.event.i, 7, debug-location !21358 :: (store (s8) into %ir.hostname.i + 2); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %189:gpr, %stack.1.event.i, 6, debug-location !21358 :: (store (s8) into %ir.hostname.i + 1); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %188:gpr, %stack.1.event.i, 5, debug-location !21358 :: (store (s8) into %ir.hostname.i); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68:5 ]
  STB %183:gpr, %stack.1.event.i, 16, debug-location !21358 :: (store (s8) into %ir.hostname.i + 11); tracecon/src/bpf/tracecon.bpf.c:79:2 @[ tracecon/src/bpf/tracecon.bpf.c:68
:5 ]
...

And the code becomes very complex and inefficient and this caused later larger stack size,

# *** IR Dump After Greedy Register Allocator (greedy) ***:
# Machine code for function tcp_v4_connect_exit: NoPHIs, TracksLiveness, TiedOpsRewritten, TracksDebugUserValues
Frame Objects:
  fi#0: size=4, align=4, at location [SP]
  fi#1: size=89, align=8, at location [SP]
  fi#2: size=4, align=4, at location [SP]
  fi#3: size=8, align=8, at location [SP]
...
  fi#57: size=8, align=8, at location [SP]
  fi#58: size=8, align=8, at location [SP]
  fi#59: size=8, align=8, at location [SP]
  fi#60: size=8, align=8, at location [SP]
Function Live Ins: $r1 in %0

Could you help take a look at this problem and suggest how to fix it?

[RKSimon]

@yonghong-song Please can you raise this as an issue and include the IR as well? AFAICT this is a perf regression, and not an actual bug

[yonghong-song]

In D77804#3804930, @RKSimon wrote:

@yonghong-song Please can you raise this as an issue and include the IR as well? AFAICT this is a perf regression, and not an actual bug

Thanks @RKSimon, just created an llvm-project issue https://github.com/llvm/llvm-project/issues/57872 thanks for taking care of this!

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

return TLI->SimplifyMultipleUseDemandedBits(V, DemandedBits, DemandedElts,
                                                 *this, 0);
   case ISD::Constant: {
     const APInt &CVal = cast<ConstantSDNode>(V)->getAPIntValue();
     APInt NewVal = CVal & DemandedBits;
     if (NewVal != CVal)
       return getConstant(NewVal, SDLoc(V), V.getValueType());
     break;
   }
-  case ISD::SRL:
-    // Only look at single-use SRLs.
-    if (!V.getNode()->hasOneUse())
-      break;
-    if (auto *RHSC = dyn_cast<ConstantSDNode>(V.getOperand(1))) {
-      // See if we can recursively simplify the LHS.
-      unsigned Amt = RHSC->getZExtValue();
-
-      // Watch out for shift count overflow though.
-      if (Amt >= DemandedBits.getBitWidth())
-        break;
-      APInt SrcDemandedBits = DemandedBits << Amt;
-      if (SDValue SimplifyLHS =
-              GetDemandedBits(V.getOperand(0), SrcDemandedBits))
-        return getNode(ISD::SRL, SDLoc(V), V.getValueType(), SimplifyLHS,
-                       V.getOperand(1));
-    }
-    break;
   }
   return SDValue();
 }
 
 /// SignBitIsZero - Return true if the sign bit of Op is known to be zero.  We
 /// use this predicate to simplify operations downstream.
 bool SelectionDAG::SignBitIsZero(SDValue Op, unsigned Depth) const {
   unsigned BitWidth = Op.getScalarValueSizeInBits();

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (const APInt *SA =
                                Depth + 1))
         return true;
       assert(!Known.hasConflict() && "Bits known to be one AND zero?");
       Known.Zero <<= ShAmt;
       Known.One <<= ShAmt;
       // low bits known zero.
       Known.Zero.setLowBits(ShAmt);
 
+      // Attempt to avoid multi-use ops if we don't need anything from them.
+      if (!InDemandedMask.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+        SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+            Op0, InDemandedMask, DemandedElts, TLO.DAG, Depth + 1);
+        if (DemandedOp0) {
+          SDValue NewOp = TLO.DAG.getNode(ISD::SHL, dl, VT, DemandedOp0, Op1);
+          return TLO.CombineTo(Op, NewOp);
+        }
+      }
+
       // Try shrinking the operation as long as the shift amount will still be
       // in range.
       if ((ShAmt < DemandedBits.getActiveBits()) &&
           ShrinkDemandedOp(Op, BitWidth, DemandedBits, TLO))
         return true;
     }
 
     // If we are only demanding sign bits then we can use the shift source
if (const APInt *SA =
       if (SimplifyDemandedBits(Op0, InDemandedMask, DemandedElts, Known, TLO,
                                Depth + 1))
         return true;
       assert(!Known.hasConflict() && "Bits known to be one AND zero?");
       Known.Zero.lshrInPlace(ShAmt);
       Known.One.lshrInPlace(ShAmt);
       // High bits known zero.
       Known.Zero.setHighBits(ShAmt);
+
+      // Attempt to avoid multi-use ops if we don't need anything from them.
+      if (!InDemandedMask.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+        SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+            Op0, InDemandedMask, DemandedElts, TLO.DAG, Depth + 1);
+        if (DemandedOp0) {
+          SDValue NewOp = TLO.DAG.getNode(ISD::SRL, dl, VT, DemandedOp0, Op1);
+          return TLO.CombineTo(Op, NewOp);
+        }
+      }
     }
     break;
   }
   case ISD::SRA: {
     SDValue Op0 = Op.getOperand(0);
     SDValue Op1 = Op.getOperand(1);
     EVT ShiftVT = Op1.getValueType();
 

llvm/test/CodeGen/AArch64/parity.ll

; CHECK-NEXT:    ret
   %2 = and i16 %1, 1
   ret i16 %2
 }
 
 define i17 @parity_17(i17 %x) {
 ; CHECK-LABEL: parity_17:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    and w8, w0, #0x1ffff
-; CHECK-NEXT:    eor w8, w8, w8, lsr #16
-; CHECK-NEXT:    eor w8, w8, w8, lsr #8
+; CHECK-NEXT:    eor w9, w8, w8, lsr #16
+; CHECK-NEXT:    eor w8, w9, w8, lsr #8
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #4
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #2
 ; CHECK-NEXT:    eor w8, w8, w8, lsr #1
 ; CHECK-NEXT:    and w0, w8, #0x1
 ; CHECK-NEXT:    ret
   %1 = tail call i17 @llvm.ctpop.i17(i17 %x)
   %2 = and i17 %1, 1
   ret i17 %2

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

 ; GCN-LABEL: srem_v3i15:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GCN-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0xb
 ; GCN-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
 ; GCN-NEXT:    s_mov_b32 s7, 0xf000
 ; GCN-NEXT:    s_mov_b32 s6, -1
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    s_bfe_i32 s10, s2, 0xf0000
+; GCN-NEXT:    v_mov_b32_e32 v2, s0
+; GCN-NEXT:    v_alignbit_b32 v2, s1, v2, 30
+; GCN-NEXT:    s_bfe_i32 s1, s0, 0xf0000
+; GCN-NEXT:    v_cvt_f32_i32_e32 v4, s1
+; GCN-NEXT:    v_cvt_f32_i32_e32 v5, s10
+; GCN-NEXT:    s_xor_b32 s1, s10, s1
+; GCN-NEXT:    s_ashr_i32 s1, s1, 30
+; GCN-NEXT:    v_rcp_iflag_f32_e32 v6, v4
+; GCN-NEXT:    s_or_b32 s1, s1, 1
+; GCN-NEXT:    v_mov_b32_e32 v7, s1
+; GCN-NEXT:    s_lshr_b32 s9, s0, 15
+; GCN-NEXT:    v_mul_f32_e32 v6, v5, v6
+; GCN-NEXT:    v_trunc_f32_e32 v6, v6
+; GCN-NEXT:    v_mad_f32 v5, -v6, v4, v5
+; GCN-NEXT:    v_cvt_i32_f32_e32 v6, v6
+; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v5|, |v4|
+; GCN-NEXT:    v_cndmask_b32_e32 v4, 0, v7, vcc
+; GCN-NEXT:    s_bfe_i32 s1, s2, 0xf000f
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
+; GCN-NEXT:    v_mul_lo_u32 v4, v4, s0
+; GCN-NEXT:    s_bfe_i32 s0, s0, 0xf000f
+; GCN-NEXT:    v_cvt_f32_i32_e32 v5, s0
+; GCN-NEXT:    v_cvt_f32_i32_e32 v6, s1
 ; GCN-NEXT:    v_mov_b32_e32 v0, s2
 ; GCN-NEXT:    v_alignbit_b32 v0, s3, v0, 30
+; GCN-NEXT:    v_rcp_iflag_f32_e32 v7, v5
 ; GCN-NEXT:    s_movk_i32 s3, 0x7fff
-; GCN-NEXT:    s_and_b32 s11, s0, s3
-; GCN-NEXT:    s_bfe_i32 s11, s11, 0xf0000
-; GCN-NEXT:    v_cvt_f32_i32_e32 v2, s11
-; GCN-NEXT:    s_and_b32 s9, s2, s3
-; GCN-NEXT:    s_bfe_i32 s9, s9, 0xf0000
-; GCN-NEXT:    v_cvt_f32_i32_e32 v3, s9
-; GCN-NEXT:    v_rcp_iflag_f32_e32 v4, v2
-; GCN-NEXT:    s_xor_b32 s9, s9, s11
-; GCN-NEXT:    s_ashr_i32 s9, s9, 30
-; GCN-NEXT:    s_or_b32 s9, s9, 1
-; GCN-NEXT:    v_mul_f32_e32 v4, v3, v4
-; GCN-NEXT:    v_trunc_f32_e32 v4, v4
-; GCN-NEXT:    v_mad_f32 v3, -v4, v2, v3
-; GCN-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GCN-NEXT:    v_mov_b32_e32 v5, s9
-; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
-; GCN-NEXT:    v_cndmask_b32_e32 v2, 0, v5, vcc
-; GCN-NEXT:    v_mov_b32_e32 v1, s0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
-; GCN-NEXT:    s_bfe_u32 s12, s0, 0xf000f
-; GCN-NEXT:    v_alignbit_b32 v1, s1, v1, 30
-; GCN-NEXT:    v_mul_lo_u32 v2, v2, s0
-; GCN-NEXT:    s_lshr_b32 s1, s0, 15
-; GCN-NEXT:    s_bfe_i32 s0, s12, 0xf0000
-; GCN-NEXT:    v_cvt_f32_i32_e32 v3, s0
-; GCN-NEXT:    s_bfe_u32 s10, s2, 0xf000f
-; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s2, v2
-; GCN-NEXT:    s_lshr_b32 s8, s2, 15
-; GCN-NEXT:    s_bfe_i32 s2, s10, 0xf0000
-; GCN-NEXT:    v_cvt_f32_i32_e32 v4, s2
-; GCN-NEXT:    v_rcp_iflag_f32_e32 v5, v3
-; GCN-NEXT:    s_xor_b32 s0, s2, s0
+; GCN-NEXT:    s_xor_b32 s0, s1, s0
+; GCN-NEXT:    v_and_b32_e32 v3, s3, v2
+; GCN-NEXT:    v_mul_f32_e32 v7, v6, v7
+; GCN-NEXT:    v_trunc_f32_e32 v7, v7
+; GCN-NEXT:    v_mad_f32 v6, -v7, v5, v6
+; GCN-NEXT:    v_sub_i32_e32 v4, vcc, s2, v4
+; GCN-NEXT:    v_bfe_i32 v2, v2, 0, 15
 ; GCN-NEXT:    s_ashr_i32 s0, s0, 30
+; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v6|, |v5|
+; GCN-NEXT:    v_cvt_i32_f32_e32 v7, v7
+; GCN-NEXT:    v_cvt_f32_i32_e32 v6, v2
 ; GCN-NEXT:    s_or_b32 s0, s0, 1
-; GCN-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GCN-NEXT:    v_trunc_f32_e32 v5, v5
-; GCN-NEXT:    v_mad_f32 v4, -v5, v3, v4
-; GCN-NEXT:    v_cvt_i32_f32_e32 v5, v5
-; GCN-NEXT:    v_and_b32_e32 v1, s3, v1
-; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, |v3|
-; GCN-NEXT:    v_mov_b32_e32 v6, s0
-; GCN-NEXT:    v_cndmask_b32_e32 v3, 0, v6, vcc
-; GCN-NEXT:    v_bfe_i32 v4, v1, 0, 15
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
-; GCN-NEXT:    v_cvt_f32_i32_e32 v5, v4
-; GCN-NEXT:    v_and_b32_e32 v0, s3, v0
-; GCN-NEXT:    v_bfe_i32 v6, v0, 0, 15
-; GCN-NEXT:    v_cvt_f32_i32_e32 v7, v6
-; GCN-NEXT:    v_rcp_iflag_f32_e32 v8, v5
-; GCN-NEXT:    v_xor_b32_e32 v4, v6, v4
-; GCN-NEXT:    v_ashrrev_i32_e32 v4, 30, v4
-; GCN-NEXT:    v_or_b32_e32 v4, 1, v4
-; GCN-NEXT:    v_mul_f32_e32 v6, v7, v8
-; GCN-NEXT:    v_trunc_f32_e32 v6, v6
-; GCN-NEXT:    v_mad_f32 v7, -v6, v5, v7
-; GCN-NEXT:    v_cvt_i32_f32_e32 v6, v6
-; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v5|
-; GCN-NEXT:    v_cndmask_b32_e32 v4, 0, v4, vcc
-; GCN-NEXT:    v_mul_lo_u32 v3, v3, s1
-; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GCN-NEXT:    v_mul_lo_u32 v1, v4, v1
+; GCN-NEXT:    v_mov_b32_e32 v8, s0
+; GCN-NEXT:    v_cndmask_b32_e32 v5, 0, v8, vcc
+; GCN-NEXT:    v_and_b32_e32 v1, s3, v0
+; GCN-NEXT:    v_bfe_i32 v0, v0, 0, 15
+; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GCN-NEXT:    v_cvt_f32_i32_e32 v7, v0
+; GCN-NEXT:    v_rcp_iflag_f32_e32 v8, v6
+; GCN-NEXT:    v_xor_b32_e32 v0, v0, v2
+; GCN-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
+; GCN-NEXT:    v_or_b32_e32 v0, 1, v0
+; GCN-NEXT:    v_mul_f32_e32 v2, v7, v8
+; GCN-NEXT:    v_trunc_f32_e32 v2, v2
+; GCN-NEXT:    v_mad_f32 v7, -v2, v6, v7
+; GCN-NEXT:    v_cvt_i32_f32_e32 v2, v2
+; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v6|
+; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
+; GCN-NEXT:    v_mul_lo_u32 v5, v5, s9
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; GCN-NEXT:    v_mul_lo_u32 v0, v0, v3
+; GCN-NEXT:    s_lshr_b32 s8, s2, 15
+; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s8, v5
 ; GCN-NEXT:    v_and_b32_e32 v2, s3, v2
-; GCN-NEXT:    v_sub_i32_e32 v3, vcc, s8, v3
-; GCN-NEXT:    v_and_b32_e32 v3, s3, v3
-; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v1, v0
-; GCN-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
+; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
-; GCN-NEXT:    v_or_b32_e32 v2, v2, v3
+; GCN-NEXT:    v_and_b32_e32 v3, s3, v4
+; GCN-NEXT:    v_lshlrev_b32_e32 v2, 15, v2
+; GCN-NEXT:    v_or_b32_e32 v2, v3, v2
 ; GCN-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GCN-NEXT:    s_waitcnt expcnt(0)
 ; GCN-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GCN-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GCN-NEXT:    s_endpgm
   %r = srem <3 x i15> %x, %y
   store <3 x i15> %r, <3 x i15> addrspace(1)* %out

llvm/test/CodeGen/AMDGPU/bswap.ll

 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_alignbit_b32 v2, v1, v1, 8
 ; SI-NEXT:    v_alignbit_b32 v1, v1, v1, 24
 ; SI-NEXT:    s_mov_b32 s4, 0xff00ff
 ; SI-NEXT:    v_alignbit_b32 v3, v0, v0, 8
 ; SI-NEXT:    v_alignbit_b32 v0, v0, v0, 24
 ; SI-NEXT:    v_bfi_b32 v1, s4, v1, v2
 ; SI-NEXT:    v_bfi_b32 v0, s4, v0, v3
-; SI-NEXT:    v_and_b32_e32 v1, 0xffff0000, v1
+; SI-NEXT:    v_and_b32_e32 v2, 0xffff0000, v1
 ; SI-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_bswap_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s4, 0x2030001
 ; VI-NEXT:    v_perm_b32 v0, 0, v0, s4
 ; VI-NEXT:    s_setpc_b64 s[30:31]

 ; SI-NEXT:    v_alignbit_b32 v7, v2, v2, 8
 ; SI-NEXT:    v_alignbit_b32 v2, v2, v2, 24
 ; SI-NEXT:    v_bfi_b32 v1, s4, v1, v4
 ; SI-NEXT:    v_bfi_b32 v0, s4, v0, v5
 ; SI-NEXT:    v_bfi_b32 v3, s4, v3, v6
 ; SI-NEXT:    v_bfi_b32 v2, s4, v2, v7
 ; SI-NEXT:    v_and_b32_e32 v4, s5, v1
 ; SI-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_and_b32_e32 v3, s5, v3
+; SI-NEXT:    v_and_b32_e32 v5, s5, v3
 ; SI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v4
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
+; SI-NEXT:    v_or_b32_e32 v2, v2, v5
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_bswap_v4i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s4, 0x2030001
 ; VI-NEXT:    v_perm_b32 v0, 0, v0, s4
 ; VI-NEXT:    v_perm_b32 v1, 0, v1, s4

llvm/test/CodeGen/AMDGPU/cvt_f32_ubyte.ll

 ; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, 0
 ; SI-NEXT:    s_mov_b32 s3, s7
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    buffer_load_ubyte v4, v[0:1], s[0:3], 0 addr64
-; SI-NEXT:    buffer_load_ubyte v2, v[0:1], s[0:3], 0 addr64 offset:1
-; SI-NEXT:    buffer_load_ubyte v3, v[0:1], s[0:3], 0 addr64 offset:2
+; SI-NEXT:    buffer_load_ubyte v5, v[0:1], s[0:3], 0 addr64 offset:1
+; SI-NEXT:    buffer_load_ubyte v2, v[0:1], s[0:3], 0 addr64 offset:2
 ; SI-NEXT:    buffer_load_ubyte v0, v[0:1], s[0:3], 0 addr64 offset:3
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    s_waitcnt vmcnt(2)
-; SI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v2
+; SI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v5
+; SI-NEXT:    s_waitcnt vmcnt(1)
+; SI-NEXT:    v_cvt_f32_ubyte0_e32 v2, v2
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_cvt_f32_ubyte3_e32 v3, v0
-; SI-NEXT:    v_cvt_f32_ubyte2_e32 v2, v0
+; SI-NEXT:    v_cvt_f32_ubyte0_e32 v3, v0
 ; SI-NEXT:    v_cvt_f32_ubyte0_e32 v0, v4
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: load_v4i8_to_v4f32_unaligned:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x2c

 ; VI-NEXT:    v_cvt_f32_ubyte2_e32 v5, v8
 ; VI-NEXT:    s_waitcnt vmcnt(4)
 ; VI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v9
 ; VI-NEXT:    s_waitcnt vmcnt(3)
 ; VI-NEXT:    v_cvt_f32_ubyte0_e32 v4, v6
 ; VI-NEXT:    s_waitcnt vmcnt(2)
 ; VI-NEXT:    v_cvt_f32_ubyte0_e32 v6, v7
 ; VI-NEXT:    s_waitcnt vmcnt(1)
-; VI-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
-; VI-NEXT:    v_or_b32_sdwa v2, v2, v12 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; VI-NEXT:    v_cvt_f32_ubyte3_e32 v3, v2
+; VI-NEXT:    v_lshlrev_b32_e32 v3, 8, v2
+; VI-NEXT:    v_or_b32_sdwa v3, v3, v12 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_cvt_f32_ubyte0_e32 v0, v0
-; VI-NEXT:    v_cvt_f32_ubyte2_e32 v2, v2
+; VI-NEXT:    v_cvt_f32_ubyte0_e32 v2, v12
+; VI-NEXT:    v_cvt_f32_ubyte3_e32 v3, v3
 ; VI-NEXT:    buffer_store_dwordx3 v[4:6], off, s[4:7], 0 offset:16
 ; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr <7 x i8>, <7 x i8> addrspace(1)* %in, i32 %tid
   %load = load <7 x i8>, <7 x i8> addrspace(1)* %gep, align 1
   %cvt = uitofp <7 x i8> %load to <7 x float>
   store <7 x float> %cvt, <7 x float> addrspace(1)* %out, align 16

 ; SI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, 0
 ; SI-NEXT:    s_mov_b32 s3, s7
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    buffer_load_ubyte v4, v[0:1], s[0:3], 0 addr64
-; SI-NEXT:    buffer_load_ubyte v2, v[0:1], s[0:3], 0 addr64 offset:1
-; SI-NEXT:    buffer_load_ubyte v3, v[0:1], s[0:3], 0 addr64 offset:2
+; SI-NEXT:    buffer_load_ubyte v5, v[0:1], s[0:3], 0 addr64 offset:1
+; SI-NEXT:    buffer_load_ubyte v2, v[0:1], s[0:3], 0 addr64 offset:2
 ; SI-NEXT:    buffer_load_ubyte v0, v[0:1], s[0:3], 0 addr64 offset:3
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    s_waitcnt vmcnt(2)
-; SI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v2
+; SI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v5
+; SI-NEXT:    s_waitcnt vmcnt(1)
+; SI-NEXT:    v_cvt_f32_ubyte0_e32 v2, v2
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_cvt_f32_ubyte3_e32 v3, v0
-; SI-NEXT:    v_cvt_f32_ubyte2_e32 v2, v0
+; SI-NEXT:    v_cvt_f32_ubyte0_e32 v3, v0
 ; SI-NEXT:    v_cvt_f32_ubyte0_e32 v0, v4
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v4i8_zext_v4i32_to_v4f32:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x2c
 ; VI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s0, v0
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; VI-NEXT:    v_add_u32_e32 v2, vcc, 3, v0
+; VI-NEXT:    v_add_u32_e32 v2, vcc, 1, v0
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
 ; VI-NEXT:    v_add_u32_e32 v4, vcc, 2, v0
 ; VI-NEXT:    v_addc_u32_e32 v5, vcc, 0, v1, vcc
-; VI-NEXT:    v_add_u32_e32 v6, vcc, 1, v0
+; VI-NEXT:    v_add_u32_e32 v6, vcc, 3, v0
 ; VI-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_ubyte v4, v[4:5]
 ; VI-NEXT:    flat_load_ubyte v5, v[6:7]
-; VI-NEXT:    flat_load_ubyte v2, v[2:3]
+; VI-NEXT:    flat_load_ubyte v6, v[2:3]
 ; VI-NEXT:    flat_load_ubyte v0, v[0:1]
+; VI-NEXT:    s_waitcnt vmcnt(3)
+; VI-NEXT:    v_cvt_f32_ubyte0_e32 v2, v4
+; VI-NEXT:    s_waitcnt vmcnt(2)
+; VI-NEXT:    v_cvt_f32_ubyte0_e32 v3, v5
 ; VI-NEXT:    s_waitcnt vmcnt(1)
-; VI-NEXT:    v_lshlrev_b32_e32 v1, 8, v2
-; VI-NEXT:    v_or_b32_sdwa v1, v1, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; VI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v6
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_cvt_f32_ubyte0_e32 v0, v0
-; VI-NEXT:    v_cvt_f32_ubyte3_e32 v3, v1
-; VI-NEXT:    v_cvt_f32_ubyte2_e32 v2, v1
-; VI-NEXT:    v_cvt_f32_ubyte2_e32 v1, v5
 ; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr <4 x i8>, <4 x i8> addrspace(1)* %in, i32 %tid
   %load = load <4 x i8>, <4 x i8> addrspace(1)* %gep, align 1
   %ext = zext <4 x i8> %load to <4 x i32>
   %cvt = uitofp <4 x i32> %ext to <4 x float>
   store <4 x float> %cvt, <4 x float> addrspace(1)* %out, align 16

llvm/test/CodeGen/AMDGPU/fshr.ll

 }
 
 define <2 x i16> @v_fshr_v2i16(<2 x i16> %src0, <2 x i16> %src1, <2 x i16> %src2) {
 ; SI-LABEL: v_fshr_v2i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_or_b32_e32 v5, 16, v5
 ; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; SI-NEXT:    v_alignbit_b32 v1, v1, v3, v5
-; SI-NEXT:    v_or_b32_e32 v3, 16, v4
+; SI-NEXT:    v_or_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; SI-NEXT:    v_alignbit_b32 v0, v0, v2, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_alignbit_b32 v1, v1, v3, v5
+; SI-NEXT:    v_alignbit_b32 v0, v0, v2, v4
+; SI-NEXT:    s_mov_b32 s4, 0xffff
+; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; SI-NEXT:    v_and_b32_e32 v0, s4, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v3
+; SI-NEXT:    v_and_b32_e32 v1, s4, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_fshr_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
 ; VI-NEXT:    v_and_b32_e32 v4, 15, v3
 ; VI-NEXT:    v_mov_b32_e32 v5, 1

 ; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
 ; SI-NEXT:    v_alignbit_b32 v1, v1, v5, v9
 ; SI-NEXT:    v_or_b32_e32 v5, 16, v8
 ; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_alignbit_b32 v0, v0, v4, v5
 ; SI-NEXT:    v_or_b32_e32 v4, 16, v11
 ; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v7
 ; SI-NEXT:    v_alignbit_b32 v3, v3, v5, v4
-; SI-NEXT:    v_or_b32_e32 v4, 16, v10
-; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v6
+; SI-NEXT:    v_or_b32_e32 v5, 16, v10
+; SI-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
 ; SI-NEXT:    s_mov_b32 s4, 0xffff
-; SI-NEXT:    v_alignbit_b32 v2, v2, v5, v4
-; SI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; SI-NEXT:    v_alignbit_b32 v2, v2, v6, v5
+; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v3
 ; SI-NEXT:    v_and_b32_e32 v2, s4, v2
-; SI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    v_and_b32_e32 v0, s4, v0
+; SI-NEXT:    v_or_b32_e32 v2, v2, v4
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; SI-NEXT:    v_alignbit_b32 v1, v2, v1, 16
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; SI-NEXT:    v_and_b32_e32 v3, s4, v3
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: v_fshr_v4i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshrrev_b32_e32 v6, 16, v5
 ; VI-NEXT:    v_and_b32_e32 v7, 15, v6
 ; VI-NEXT:    v_xor_b32_e32 v6, -1, v6

llvm/test/CodeGen/AMDGPU/idot4u.ll

 ; TODO: Support this pattern.
 define amdgpu_kernel void @udot4_acc8_vecMul(<4 x i8> addrspace(1)* %src1,
 ; GFX7-LABEL: udot4_acc8_vecMul:
 ; GFX7:       ; %bb.0: ; %entry
 ; GFX7-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
 ; GFX7-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s2, -1
-; GFX7-NEXT:    s_movk_i32 s8, 0xff
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    s_load_dword s5, s[6:7], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_bfe_u32 s6, s4, 0x80008
-; GFX7-NEXT:    s_bfe_u32 s10, s5, 0x80008
-; GFX7-NEXT:    s_lshr_b32 s11, s5, 16
-; GFX7-NEXT:    s_lshr_b32 s12, s5, 24
-; GFX7-NEXT:    v_mov_b32_e32 v3, s10
+; GFX7-NEXT:    s_lshr_b32 s6, s4, 24
+; GFX7-NEXT:    s_lshr_b32 s11, s5, 8
 ; GFX7-NEXT:    s_lshr_b32 s7, s4, 16
-; GFX7-NEXT:    v_mov_b32_e32 v2, s11
-; GFX7-NEXT:    s_lshr_b32 s9, s4, 24
-; GFX7-NEXT:    v_mov_b32_e32 v1, s12
+; GFX7-NEXT:    s_lshr_b32 s8, s4, 8
 ; GFX7-NEXT:    s_mul_i32 s4, s4, s5
-; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s9, v1
-; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s7, v2
-; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s6, v3
-; GFX7-NEXT:    s_and_b32 s5, s4, s8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s8, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, s5, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 8, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 24, v1
+; GFX7-NEXT:    s_lshr_b32 s10, s5, 16
+; GFX7-NEXT:    v_mov_b32_e32 v1, s11
+; GFX7-NEXT:    s_lshr_b32 s9, s5, 24
+; GFX7-NEXT:    v_mov_b32_e32 v2, s10
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX7-NEXT:    v_mad_u32_u24 v0, s8, v1, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s7, v2, v0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s9
+; GFX7-NEXT:    v_mad_u32_u24 v0, s6, v1, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_movk_i32 s0, 0xff
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s0
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 24
-; GFX8-NEXT:    s_lshr_b32 s4, s1, 24
 ; GFX8-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s1
 ; GFX8-NEXT:    s_mul_i32 s0, s0, s1
+; GFX8-NEXT:    s_lshr_b32 s4, s1, 24
 ; GFX8-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v4, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
 ; GFX8-NEXT:    v_and_b32_e32 v3, s0, v3
+; GFX8-NEXT:    v_mov_b32_e32 v5, s5
+; GFX8-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s4
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s2
-; GFX8-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX8-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s3, v5
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v7, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX8-NEXT:    v_or_b32_sdwa v4, v5, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_e32 v4, v3, v4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v4
+; GFX8-NEXT:    v_or_b32_sdwa v4, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v6, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_acc8_vecMul:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s3, s[6:7], 0x0
-; GFX9-NODL-NEXT:    global_load_ubyte v4, v0, s[0:1]
+; GFX9-NODL-NEXT:    global_load_ubyte v5, v0, s[0:1]
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_lshr_b32 s4, s2, 16
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s3
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v2, s3
 ; GFX9-NODL-NEXT:    s_lshr_b32 s6, s3, 16
 ; GFX9-NODL-NEXT:    s_lshr_b32 s7, s3, 24
-; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v1, s2, v1
-; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v2, s2, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-NODL-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s6
 ; GFX9-NODL-NEXT:    s_lshr_b32 s5, s2, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v2, s7
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v2, s5, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v3, s4, v3
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s3
+; GFX9-NODL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v1, s2, v1
+; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v3, s2, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-NODL-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NODL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
-; GFX9-NODL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NODL-NEXT:    v_or_b32_e32 v2, v1, v2
-; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v3, 8, v2
+; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v3, 8, v1
+; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v4, 8, v2
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_add_u32_e32 v1, v1, v4
+; GFX9-NODL-NEXT:    v_add_u32_e32 v1, v1, v5
 ; GFX9-NODL-NEXT:    v_add_u32_e32 v1, v1, v3
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-NODL-NEXT:    v_add_u32_e32 v1, v1, v2
+; GFX9-NODL-NEXT:    v_add_u32_e32 v1, v1, v4
 ; GFX9-NODL-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s3, s[6:7], 0x0
-; GFX9-DL-NEXT:    global_load_ubyte v4, v0, s[0:1]
+; GFX9-DL-NEXT:    global_load_ubyte v5, v0, s[0:1]
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s2, 16
-; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s3
-; GFX9-DL-NEXT:    v_mov_b32_e32 v2, s3
 ; GFX9-DL-NEXT:    s_lshr_b32 s6, s3, 16
 ; GFX9-DL-NEXT:    s_lshr_b32 s7, s3, 24
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v1, s2, v1
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v2, s2, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s6
 ; GFX9-DL-NEXT:    s_lshr_b32 s5, s2, 24
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v2, s7
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v2, s5, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v3, s4, v3
+; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s3
+; GFX9-DL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v1, s2, v1
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v3, s2, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_and_b32_e32 v1, 0xffff, v1
-; GFX9-DL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v2, v1, v2
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v4, 8, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v4
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v5
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v3
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v2
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v4
 ; GFX9-DL-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot4_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    global_load_ubyte v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX10-DL-NEXT:    s_load_dword s1, s[2:3], 0x0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    v_lshrrev_b16_e64 v2, 8, s0
 ; GFX10-DL-NEXT:    v_lshrrev_b16_e64 v3, 8, s1
 ; GFX10-DL-NEXT:    s_lshr_b32 s2, s0, 24
 ; GFX10-DL-NEXT:    s_lshr_b32 s3, s1, 24
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s2, s3
 ; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, v2, v3
 ; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, s0, s1
 ; GFX10-DL-NEXT:    s_lshr_b32 s0, s0, 16
 ; GFX10-DL-NEXT:    s_lshr_b32 s1, s1, 16
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 8, v2
-; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v4
 ; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s0, s1
+; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, s2, s3
 ; GFX10-DL-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX10-DL-NEXT:    v_or_b32_sdwa v3, v4, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_or_b32_e32 v3, v2, v3
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 8, v3
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v2
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v2, v1
-; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v4
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v7, v2, v1
+; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v7, v5
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v2
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v3
 ; GFX10-DL-NEXT:    global_store_byte v0, v1, s[4:5]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <4 x i8> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <4 x i8>, <4 x i8> addrspace(1)* %src1
   %vec2 = load <4 x i8>, <4 x i8> addrspace(1)* %src2
 

llvm/test/CodeGen/AMDGPU/idot8s.ll

 ; GFX7-NEXT:    s_bfe_i32 s12, s4, 0x4000c
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s19
 ; GFX7-NEXT:    s_bfe_i32 s13, s4, 0x40000
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s20
 ; GFX7-NEXT:    s_bfe_i32 s4, s4, 0x40004
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s4, v1
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v2, s13, v2
-; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s12, v3
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v4, s11, v4
+; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s12, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 16, v3
+; GFX7-NEXT:    v_and_b32_e32 v4, s8, v4
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, s8, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s8, v4
-; GFX7-NEXT:    v_or_b32_e32 v3, v4, v3
 ; GFX7-NEXT:    v_or_b32_e32 v2, v2, v1
-; GFX7-NEXT:    v_alignbit_b32 v1, v3, v1, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 16, v3
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v8
+; GFX7-NEXT:    v_alignbit_b32 v1, v4, v1, 16
+; GFX7-NEXT:    v_and_b32_e32 v3, s8, v3
 ; GFX7-NEXT:    v_mov_b32_e32 v5, s17
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s16
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s15
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
 ; GFX7-NEXT:    v_mad_i32_i24 v0, s10, v5, v0
 ; GFX7-NEXT:    v_mad_i32_i24 v0, s9, v6, v0
 ; GFX7-NEXT:    v_mad_i32_i24 v0, s7, v7, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s14
 ; GFX7-NEXT:    v_mad_i32_i24 v0, s6, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;

 ; GFX7-NEXT:    v_mov_b32_e32 v7, s16
 ; GFX7-NEXT:    s_bfe_i32 s10, s4, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s17
 ; GFX7-NEXT:    s_bfe_i32 s11, s4, 0x4000c
 ; GFX7-NEXT:    v_mov_b32_e32 v5, s18
 ; GFX7-NEXT:    s_bfe_i32 s12, s4, 0x40010
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s19
 ; GFX7-NEXT:    s_bfe_i32 s13, s4, 0x40014
-; GFX7-NEXT:    v_mov_b32_e32 v3, s20
 ; GFX7-NEXT:    s_bfe_i32 s14, s4, 0x40018
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s21
+; GFX7-NEXT:    v_mov_b32_e32 v3, s20
 ; GFX7-NEXT:    s_ashr_i32 s4, s4, 28
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
-; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s4, v1
-; GFX7-NEXT:    v_mul_i32_i24_e32 v2, s14, v2
-; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s13, v3
-; GFX7-NEXT:    v_mul_i32_i24_e32 v9, s12, v4
-; GFX7-NEXT:    v_mul_i32_i24_e32 v5, s11, v5
+; GFX7-NEXT:    v_mul_i32_i24_e32 v9, s4, v1
+; GFX7-NEXT:    v_mul_i32_i24_e32 v10, s14, v2
+; GFX7-NEXT:    v_mul_i32_i24_e32 v11, s13, v3
+; GFX7-NEXT:    v_mul_i32_i24_e32 v12, s12, v4
+; GFX7-NEXT:    v_mul_i32_i24_e32 v13, s11, v5
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v6, s10, v6
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v7, s7, v7
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v8, s6, v8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s8, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
-; GFX7-NEXT:    v_and_b32_e32 v9, s8, v9
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
+; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 8, v9
+; GFX7-NEXT:    v_and_b32_e32 v10, s8, v10
+; GFX7-NEXT:    v_lshlrev_b32_e32 v11, 8, v11
+; GFX7-NEXT:    v_and_b32_e32 v12, s8, v12
+; GFX7-NEXT:    v_lshlrev_b32_e32 v13, 8, v13
 ; GFX7-NEXT:    v_and_b32_e32 v6, s8, v6
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 8, v7
 ; GFX7-NEXT:    v_and_b32_e32 v8, s8, v8
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v9, v3
-; GFX7-NEXT:    v_or_b32_e32 v3, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v5, v8, v7
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s9, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v5, s9, v5
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v5, v3
-; GFX7-NEXT:    v_alignbit_b32 v3, v1, v2, 8
-; GFX7-NEXT:    v_alignbit_b32 v5, v1, v2, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 8, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 24, v1
+; GFX7-NEXT:    v_or_b32_e32 v6, v6, v13
+; GFX7-NEXT:    v_or_b32_e32 v7, v8, v7
+; GFX7-NEXT:    v_or_b32_e32 v9, v10, v9
+; GFX7-NEXT:    v_or_b32_e32 v10, v12, v11
+; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 16, v9
+; GFX7-NEXT:    v_and_b32_e32 v9, s9, v10
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX7-NEXT:    v_and_b32_e32 v7, s9, v7
+; GFX7-NEXT:    v_or_b32_e32 v6, v7, v6
+; GFX7-NEXT:    v_or_b32_e32 v8, v9, v8
+; GFX7-NEXT:    v_alignbit_b32 v7, v8, v6, 8
+; GFX7-NEXT:    v_alignbit_b32 v8, v8, v6, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v6
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s11, v5, v0
 ; GFX7-NEXT:    v_mad_i32_i24 v0, s12, v4, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX7-NEXT:    v_mad_i32_i24 v0, s13, v3, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s14, v2, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s4, v1, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX8-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s22, -1
 ; GFX8-NEXT:    s_mov_b32 s23, 0xe80000
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX8-NEXT:    s_add_u32 s20, s20, s3
 ; GFX8-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX8-NEXT:    s_mov_b32 s0, 0xffff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_bfe_i32 s7, s1, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s9, s1, 0x4000c
-; GFX8-NEXT:    s_bfe_i32 s14, s2, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s15, s2, 0x40000
-; GFX8-NEXT:    s_bfe_i32 s16, s2, 0x4000c
-; GFX8-NEXT:    s_bfe_i32 s3, s1, 0x40014
-; GFX8-NEXT:    s_ashr_i32 s5, s1, 28
-; GFX8-NEXT:    s_bfe_i32 s10, s2, 0x40014
-; GFX8-NEXT:    s_bfe_i32 s11, s2, 0x40010
-; GFX8-NEXT:    s_ashr_i32 s12, s2, 28
-; GFX8-NEXT:    s_bfe_i32 s13, s2, 0x40018
-; GFX8-NEXT:    s_bfe_i32 s2, s2, 0x40008
-; GFX8-NEXT:    s_bfe_i32 s8, s1, 0x40000
+; GFX8-NEXT:    s_bfe_i32 s9, s1, 0x40004
+; GFX8-NEXT:    s_ashr_i32 s10, s2, 28
+; GFX8-NEXT:    s_bfe_i32 s11, s2, 0x40018
+; GFX8-NEXT:    s_bfe_i32 s12, s2, 0x40014
+; GFX8-NEXT:    s_bfe_i32 s13, s2, 0x40010
+; GFX8-NEXT:    s_bfe_i32 s14, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_i32 s15, s2, 0x40008
+; GFX8-NEXT:    s_bfe_i32 s16, s2, 0x40004
+; GFX8-NEXT:    s_bfe_i32 s2, s2, 0x40000
+; GFX8-NEXT:    s_ashr_i32 s3, s1, 28
+; GFX8-NEXT:    s_bfe_i32 s5, s1, 0x40014
+; GFX8-NEXT:    s_bfe_i32 s7, s1, 0x4000c
+; GFX8-NEXT:    s_bfe_i32 s4, s1, 0x40018
+; GFX8-NEXT:    s_bfe_i32 s6, s1, 0x40010
+; GFX8-NEXT:    s_bfe_i32 s8, s1, 0x40008
+; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x40000
+; GFX8-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s16
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s9
+; GFX8-NEXT:    v_mul_i32_i24_e32 v3, s1, v3
+; GFX8-NEXT:    v_mul_i32_i24_sdwa v4, v5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s15
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s14
 ; GFX8-NEXT:    v_mov_b32_e32 v8, s7
-; GFX8-NEXT:    v_mul_i32_i24_sdwa v4, v5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_mul_i32_i24_e32 v5, s8, v6
-; GFX8-NEXT:    v_mul_i32_i24_sdwa v6, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    s_bfe_i32 s4, s1, 0x40010
-; GFX8-NEXT:    s_bfe_i32 s6, s1, 0x40018
 ; GFX8-NEXT:    v_mov_b32_e32 v9, s13
-; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v10, s12
 ; GFX8-NEXT:    v_mov_b32_e32 v11, s5
+; GFX8-NEXT:    v_mul_i32_i24_e32 v5, s8, v6
+; GFX8-NEXT:    v_mul_i32_i24_sdwa v6, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v3, s0, v3
+; GFX8-NEXT:    v_mul_i32_i24_e32 v7, s6, v9
+; GFX8-NEXT:    v_mul_i32_i24_sdwa v8, v11, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
+; GFX8-NEXT:    v_or_b32_sdwa v5, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_mov_b32_e32 v12, s11
 ; GFX8-NEXT:    v_mov_b32_e32 v13, s10
 ; GFX8-NEXT:    v_mov_b32_e32 v14, s3
-; GFX8-NEXT:    v_mul_i32_i24_e32 v3, s1, v3
-; GFX8-NEXT:    v_or_b32_sdwa v5, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_mul_i32_i24_e32 v7, s6, v9
-; GFX8-NEXT:    v_mul_i32_i24_sdwa v8, v11, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
 ; GFX8-NEXT:    v_mul_i32_i24_e32 v9, s4, v12
 ; GFX8-NEXT:    v_mul_i32_i24_sdwa v10, v14, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_and_b32_e32 v5, s0, v5
-; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v4, s0, v7
 ; GFX8-NEXT:    v_or_b32_sdwa v9, v9, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_and_b32_e32 v4, s0, v9
-; GFX8-NEXT:    v_or_b32_e32 v3, v5, v3
-; GFX8-NEXT:    v_or_b32_e32 v6, v4, v7
-; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v3
-; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v6
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 8, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v9
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v7, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_2 src1_sel:BYTE_0
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v8, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v6
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v10
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v9
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v7
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc8_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s22, -1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_ubyte v1, v0, s[0:1]
 ; GFX9-NEXT:    s_mov_b32 s23, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s20, s20, s3
 ; GFX9-NEXT:    s_load_dword s3, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX9-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_lshr_b32 s9, s3, 4
-; GFX9-NEXT:    s_lshr_b32 s16, s4, 4
+; GFX9-NEXT:    s_lshr_b32 s11, s3, 4
+; GFX9-NEXT:    s_lshr_b32 s18, s4, 4
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v2, 12, s3
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v3, 12, s4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
-; GFX9-NEXT:    v_lshlrev_b16_e64 v13, 12, s16
-; GFX9-NEXT:    s_lshr_b32 s10, s3, 12
-; GFX9-NEXT:    s_lshr_b32 s11, s3, 8
-; GFX9-NEXT:    s_lshr_b32 s17, s4, 12
-; GFX9-NEXT:    s_lshr_b32 s18, s4, 8
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v11, 12, s18
-; GFX9-NEXT:    v_lshlrev_b16_e64 v12, 12, s17
+; GFX9-NEXT:    s_lshr_b32 s7, s3, 20
+; GFX9-NEXT:    s_lshr_b32 s8, s3, 16
+; GFX9-NEXT:    s_lshr_b32 s14, s4, 20
+; GFX9-NEXT:    s_lshr_b32 s15, s4, 16
+; GFX9-NEXT:    v_lshlrev_b16_e64 v7, 12, s8
+; GFX9-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
+; GFX9-NEXT:    v_lshlrev_b16_e64 v14, 12, s15
+; GFX9-NEXT:    v_lshlrev_b16_e64 v15, 12, s14
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v2, 12, v2
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
-; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
+; GFX9-NEXT:    s_lshr_b32 s9, s3, 12
+; GFX9-NEXT:    s_lshr_b32 s10, s3, 8
+; GFX9-NEXT:    s_lshr_b32 s16, s4, 12
+; GFX9-NEXT:    s_lshr_b32 s17, s4, 8
+; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX9-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
+; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
+; GFX9-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
+; GFX9-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v4, v4, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
+; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
+; GFX9-NEXT:    v_lshlrev_b16_e64 v12, 12, s17
+; GFX9-NEXT:    v_lshlrev_b16_e64 v13, 12, s16
+; GFX9-NEXT:    v_or_b32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
+; GFX9-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v2, s2, v2
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
+; GFX9-NEXT:    v_and_b32_e32 v4, s2, v7
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, v6, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    s_lshr_b32 s5, s3, 20
-; GFX9-NEXT:    s_lshr_b32 s6, s3, 16
-; GFX9-NEXT:    s_lshr_b32 s12, s4, 20
-; GFX9-NEXT:    s_lshr_b32 s13, s4, 16
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, v5, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v4, v4, v11
+; GFX9-NEXT:    v_mul_lo_u16_e32 v5, v5, v12
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v2
+; GFX9-NEXT:    v_or_b32_sdwa v5, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    s_lshr_b32 s5, s3, 28
+; GFX9-NEXT:    s_lshr_b32 s6, s3, 24
+; GFX9-NEXT:    s_lshr_b32 s12, s4, 28
+; GFX9-NEXT:    s_lshr_b32 s13, s4, 24
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v9, 12, s6
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v10, 12, s5
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v16, 12, s13
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v17, 12, s12
-; GFX9-NEXT:    s_lshr_b32 s7, s3, 28
-; GFX9-NEXT:    s_lshr_b32 s8, s3, 24
-; GFX9-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-NEXT:    s_lshr_b32 s15, s4, 24
-; GFX9-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX9-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshlrev_b16_e64 v7, 12, s8
-; GFX9-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
-; GFX9-NEXT:    v_lshlrev_b16_e64 v14, 12, s15
-; GFX9-NEXT:    v_lshlrev_b16_e64 v15, 12, s14
-; GFX9-NEXT:    v_or_b32_e32 v4, v2, v4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v16, 12, v16
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v17, 12, v17
-; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
-; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v3, v10, v17 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v9, v9, v16
-; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v4
 ; GFX9-NEXT:    v_or_b32_sdwa v3, v9, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
-; GFX9-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_and_b32_e32 v3, s2, v3
-; GFX9-NEXT:    v_or_b32_e32 v5, v3, v7
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v1, v2, v1
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v7
+; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v6
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v4
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v8
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v3
-; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v5
+; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v3
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-DL-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s22, -1
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_ubyte v1, v0, s[0:1]
 ; GFX9-DL-NEXT:    s_mov_b32 s23, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s20, s20, s3
 ; GFX9-DL-NEXT:    s_load_dword s3, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX9-DL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_lshr_b32 s9, s3, 4
-; GFX9-DL-NEXT:    s_lshr_b32 s16, s4, 4
+; GFX9-DL-NEXT:    s_lshr_b32 s11, s3, 4
+; GFX9-DL-NEXT:    s_lshr_b32 s18, s4, 4
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v2, 12, s3
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v3, 12, s4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s16
-; GFX9-DL-NEXT:    s_lshr_b32 s10, s3, 12
-; GFX9-DL-NEXT:    s_lshr_b32 s11, s3, 8
-; GFX9-DL-NEXT:    s_lshr_b32 s17, s4, 12
-; GFX9-DL-NEXT:    s_lshr_b32 s18, s4, 8
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s18
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s17
+; GFX9-DL-NEXT:    s_lshr_b32 s7, s3, 20
+; GFX9-DL-NEXT:    s_lshr_b32 s8, s3, 16
+; GFX9-DL-NEXT:    s_lshr_b32 s14, s4, 20
+; GFX9-DL-NEXT:    s_lshr_b32 s15, s4, 16
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v7, 12, s8
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s15
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s14
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v2, 12, v2
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
+; GFX9-DL-NEXT:    s_lshr_b32 s9, s3, 12
+; GFX9-DL-NEXT:    s_lshr_b32 s10, s3, 8
+; GFX9-DL-NEXT:    s_lshr_b32 s16, s4, 12
+; GFX9-DL-NEXT:    s_lshr_b32 s17, s4, 8
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v4, v4, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s17
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s16
+; GFX9-DL-NEXT:    v_or_b32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
+; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v2, s2, v2
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v2, v2, v3
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
+; GFX9-DL-NEXT:    v_and_b32_e32 v4, s2, v7
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v6, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s3, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s6, s3, 16
-; GFX9-DL-NEXT:    s_lshr_b32 s12, s4, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s13, s4, 16
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v5, v5, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v4, v4, v11
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v5, v5, v12
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v2
+; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v5, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    s_lshr_b32 s5, s3, 28
+; GFX9-DL-NEXT:    s_lshr_b32 s6, s3, 24
+; GFX9-DL-NEXT:    s_lshr_b32 s12, s4, 28
+; GFX9-DL-NEXT:    s_lshr_b32 s13, s4, 24
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s6
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s5
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v16, 12, s13
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v17, 12, s12
-; GFX9-DL-NEXT:    s_lshr_b32 s7, s3, 28
-; GFX9-DL-NEXT:    s_lshr_b32 s8, s3, 24
-; GFX9-DL-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-DL-NEXT:    s_lshr_b32 s15, s4, 24
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX9-DL-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v7, 12, s8
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s15
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s14
-; GFX9-DL-NEXT:    v_or_b32_e32 v4, v2, v4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v16, 12, v16
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v17, 12, v17
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v3, v10, v17 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v9, v9, v16
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v4
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v9, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v7, v14
-; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, s2, v3
-; GFX9-DL-NEXT:    v_or_b32_e32 v5, v3, v7
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v2, v1
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v7
+; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v6
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v4
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v8
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v3
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v5
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v3
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: idot8_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s22, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s23, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s20, s20, s3
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    global_load_ubyte v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX10-DL-NEXT:    s_load_dword s1, s[2:3], 0x0
 ; GFX10-DL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-DL-NEXT:    s_lshr_b32 s9, s0, 4
-; GFX10-DL-NEXT:    s_lshr_b32 s16, s1, 4
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s16
-; GFX10-DL-NEXT:    s_lshr_b32 s10, s0, 12
-; GFX10-DL-NEXT:    s_lshr_b32 s17, s1, 12
+; GFX10-DL-NEXT:    s_lshr_b32 s11, s0, 4
+; GFX10-DL-NEXT:    s_lshr_b32 s18, s1, 4
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s18
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 12, s0
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 12, s1
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s17
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v6, 12, v6
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v12, 12, v12
-; GFX10-DL-NEXT:    s_lshr_b32 s11, s0, 8
-; GFX10-DL-NEXT:    s_lshr_b32 s18, s1, 8
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s18
+; GFX10-DL-NEXT:    s_lshr_b32 s8, s0, 16
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v15, 12, v4
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v8, 12, v8
 ; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v2, 12, v2
 ; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v3, 12, v3
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v19, 12, v5
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v6, v6, v12
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v13, 12, v13
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v4, 12, v4
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v11, 12, v11
+; GFX10-DL-NEXT:    s_lshr_b32 s9, s0, 12
+; GFX10-DL-NEXT:    s_lshr_b32 s10, s0, 8
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v15, v8
+; GFX10-DL-NEXT:    s_lshr_b32 s16, s1, 12
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s10
 ; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, v2, v3
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 8, v6
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, v19, v13
-; GFX10-DL-NEXT:    s_lshr_b32 s3, s0, 20
-; GFX10-DL-NEXT:    s_lshr_b32 s6, s0, 16
-; GFX10-DL-NEXT:    s_lshr_b32 s7, s0, 28
-; GFX10-DL-NEXT:    s_lshr_b32 s8, s0, 24
-; GFX10-DL-NEXT:    s_lshr_b32 s12, s1, 20
-; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s9
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v4
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v7, 12, s8
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s6
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s3
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s12
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v4, v11
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v3
-; GFX10-DL-NEXT:    s_lshr_b32 s13, s1, 16
-; GFX10-DL-NEXT:    s_lshr_b32 s14, s1, 28
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s13
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v5, 12, v7
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v7, 12, v8
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v8, 12, v9
-; GFX10-DL-NEXT:    v_or_b32_sdwa v3, v4, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s16
+; GFX10-DL-NEXT:    s_lshr_b32 s7, s0, 20
+; GFX10-DL-NEXT:    s_lshr_b32 s14, s1, 20
+; GFX10-DL-NEXT:    s_lshr_b32 s17, s1, 8
+; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s17
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s7
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v19, 12, v5
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v5, 12, v6
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v6, 12, v7
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v7, 12, v13
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s14
+; GFX10-DL-NEXT:    s_lshr_b32 s15, s1, 16
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v3, 12, v9
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v12, 12, v12
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v5, v5, v7
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s15
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v9, 12, v14
 ; GFX10-DL-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s14
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v4, 12, v10
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v9, 12, v12
-; GFX10-DL-NEXT:    s_lshr_b32 s15, s1, 24
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v6, 12, v6
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v10, 12, v15
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s15
-; GFX10-DL-NEXT:    v_or_b32_e32 v3, v2, v3
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v4, v9
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v15, v8, v6
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v7, v7, v10
-; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v11, 12, v14
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v19, v12
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 8, v5
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v7, 12, v8
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, v3, v9
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX10-DL-NEXT:    s_lshr_b32 s3, s0, 28
+; GFX10-DL-NEXT:    s_lshr_b32 s12, s1, 28
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s3
+; GFX10-DL-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v6, v6, v7
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v3
+; GFX10-DL-NEXT:    s_lshr_b32 s6, s0, 24
+; GFX10-DL-NEXT:    s_lshr_b32 s13, s1, 24
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s6
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s13
+; GFX10-DL-NEXT:    v_or_b32_sdwa v3, v6, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v5, 12, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v7, 12, v15
+; GFX10-DL-NEXT:    v_and_b32_e32 v3, s2, v3
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v2, v1
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 8, v4
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v5, v11
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 8, v7
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 12, s12
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v8
-; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v15, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX10-DL-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
-; GFX10-DL-NEXT:    v_or_b32_e32 v3, v2, v4
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v2, 12, v2
+; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, v5, v2
+; GFX10-DL-NEXT:    v_ashrrev_i16_e64 v4, 12, v10
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v6
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 8, v2
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, v4, v7
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v3, v1, v3
+; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v3, v5
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v2
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX10-DL-NEXT:    global_store_byte v0, v1, s[4:5]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

llvm/test/CodeGen/AMDGPU/idot8u.ll

 ; GFX7-NEXT:    buffer_load_ushort v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    s_load_dword s5, s[6:7], 0x0
 ; GFX7-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_bfe_u32 s10, s4, 0x4000c
 ; GFX7-NEXT:    s_bfe_u32 s17, s5, 0x4000c
 ; GFX7-NEXT:    s_bfe_u32 s19, s5, 0x40004
-; GFX7-NEXT:    v_mov_b32_e32 v4, s17
 ; GFX7-NEXT:    s_bfe_u32 s14, s5, 0x40018
 ; GFX7-NEXT:    s_bfe_u32 s15, s5, 0x40014
 ; GFX7-NEXT:    s_bfe_u32 s16, s5, 0x40010
 ; GFX7-NEXT:    s_bfe_u32 s18, s5, 0x40008
 ; GFX7-NEXT:    s_lshr_b32 s13, s5, 28
 ; GFX7-NEXT:    s_and_b32 s5, s5, 15
+; GFX7-NEXT:    v_mov_b32_e32 v4, s17
 ; GFX7-NEXT:    s_bfe_u32 s12, s4, 0x40004
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s19
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s12, v2
-; GFX7-NEXT:    v_mul_u32_u24_e32 v4, s10, v4
+; GFX7-NEXT:    v_mul_u32_u24_e32 v8, s10, v4
 ; GFX7-NEXT:    s_lshr_b32 s6, s4, 28
 ; GFX7-NEXT:    s_bfe_u32 s7, s4, 0x40018
 ; GFX7-NEXT:    s_bfe_u32 s8, s4, 0x40014
 ; GFX7-NEXT:    s_bfe_u32 s9, s4, 0x40010
 ; GFX7-NEXT:    s_bfe_u32 s11, s4, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s18
 ; GFX7-NEXT:    s_and_b32 s4, s4, 15
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s4, v1
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s11, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
-; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 16, v8
 ; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX7-NEXT:    v_or_b32_e32 v3, v3, v8
 ; GFX7-NEXT:    v_alignbit_b32 v2, v3, v2, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 16, v3
 ; GFX7-NEXT:    v_mov_b32_e32 v5, s16
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s15
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s14
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v2, v0
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s10, v4, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, s9, v5, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, s8, v6, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, s7, v7, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s13
 ; GFX7-NEXT:    v_mad_u32_u24 v0, s6, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;

 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    s_load_dword s5, s[6:7], 0x0
 ; GFX7-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_bfe_u32 s6, s4, 0x4000c
 ; GFX7-NEXT:    s_bfe_u32 s13, s5, 0x4000c
 ; GFX7-NEXT:    s_bfe_u32 s15, s5, 0x40004
 ; GFX7-NEXT:    s_lshr_b32 s17, s5, 28
-; GFX7-NEXT:    v_mov_b32_e32 v8, s13
+; GFX7-NEXT:    s_bfe_u32 s19, s5, 0x40014
 ; GFX7-NEXT:    s_bfe_u32 s14, s5, 0x40008
 ; GFX7-NEXT:    s_and_b32 s16, s5, 15
 ; GFX7-NEXT:    s_bfe_u32 s18, s5, 0x40018
-; GFX7-NEXT:    s_bfe_u32 s19, s5, 0x40014
+; GFX7-NEXT:    s_bfe_u32 s5, s5, 0x40010
+; GFX7-NEXT:    v_mov_b32_e32 v8, s13
 ; GFX7-NEXT:    s_bfe_u32 s8, s4, 0x40004
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s15
 ; GFX7-NEXT:    s_lshr_b32 s10, s4, 28
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s17
-; GFX7-NEXT:    v_mul_u32_u24_e32 v4, s10, v4
+; GFX7-NEXT:    s_bfe_u32 s12, s4, 0x40014
+; GFX7-NEXT:    v_mov_b32_e32 v2, s19
+; GFX7-NEXT:    v_mul_u32_u24_e32 v10, s12, v2
+; GFX7-NEXT:    v_mul_u32_u24_e32 v12, s10, v4
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v6, s8, v6
-; GFX7-NEXT:    v_mul_u32_u24_e32 v8, s6, v8
-; GFX7-NEXT:    s_bfe_u32 s5, s5, 0x40010
+; GFX7-NEXT:    v_mul_u32_u24_e32 v13, s6, v8
 ; GFX7-NEXT:    s_bfe_u32 s7, s4, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s14
 ; GFX7-NEXT:    s_and_b32 s9, s4, 15
 ; GFX7-NEXT:    v_mov_b32_e32 v5, s16
 ; GFX7-NEXT:    s_bfe_u32 s11, s4, 0x40018
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s18
-; GFX7-NEXT:    s_bfe_u32 s12, s4, 0x40014
-; GFX7-NEXT:    v_mov_b32_e32 v2, s19
-; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s12, v2
 ; GFX7-NEXT:    s_bfe_u32 s4, s4, 0x40010
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
-; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s11, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 8, v4
+; GFX7-NEXT:    v_mul_u32_u24_e32 v9, s4, v1
+; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 8, v10
+; GFX7-NEXT:    v_mul_u32_u24_e32 v11, s11, v3
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v5, s9, v5
-; GFX7-NEXT:    v_mul_u32_u24_e32 v7, s7, v7
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 8, v8
-; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
-; GFX7-NEXT:    v_or_b32_e32 v4, v5, v6
-; GFX7-NEXT:    v_or_b32_e32 v5, v7, v8
-; GFX7-NEXT:    v_mul_u32_u24_e32 v9, s4, v1
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
-; GFX7-NEXT:    v_or_b32_e32 v2, v9, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
-; GFX7-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX7-NEXT:    v_or_b32_e32 v3, v4, v5
-; GFX7-NEXT:    v_alignbit_b32 v4, v2, v3, 8
-; GFX7-NEXT:    v_alignbit_b32 v5, v2, v3, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 8, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 24, v2
+; GFX7-NEXT:    v_mul_u32_u24_e32 v7, s7, v7
+; GFX7-NEXT:    v_lshlrev_b32_e32 v12, 8, v12
+; GFX7-NEXT:    v_lshlrev_b32_e32 v13, 8, v13
+; GFX7-NEXT:    v_or_b32_e32 v9, v9, v10
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    v_or_b32_e32 v6, v7, v13
+; GFX7-NEXT:    v_or_b32_e32 v10, v11, v12
+; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 16, v10
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
+; GFX7-NEXT:    v_or_b32_e32 v7, v9, v7
+; GFX7-NEXT:    v_alignbit_b32 v6, v7, v5, 8
+; GFX7-NEXT:    v_alignbit_b32 v7, v7, v5, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v5
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s6, v8, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, s4, v1, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; GFX7-NEXT:    v_mad_u32_u24 v0, s12, v2, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s11, v3, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s10, v4, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX8-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s22, -1
 ; GFX8-NEXT:    s_mov_b32 s23, 0xe80000
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX8-NEXT:    s_add_u32 s20, s20, s3
 ; GFX8-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX8-NEXT:    s_mov_b32 s0, 0xffff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_bfe_u32 s7, s1, 0x40004
-; GFX8-NEXT:    s_bfe_u32 s9, s1, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40004
-; GFX8-NEXT:    s_and_b32 s15, s2, 15
-; GFX8-NEXT:    s_bfe_u32 s16, s2, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s3, s1, 0x40014
-; GFX8-NEXT:    s_lshr_b32 s5, s1, 28
-; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s11, s2, 0x40010
-; GFX8-NEXT:    s_lshr_b32 s12, s2, 28
-; GFX8-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX8-NEXT:    s_bfe_u32 s2, s2, 0x40008
-; GFX8-NEXT:    s_and_b32 s8, s1, 15
+; GFX8-NEXT:    s_bfe_u32 s9, s1, 0x40004
+; GFX8-NEXT:    s_lshr_b32 s10, s2, 28
+; GFX8-NEXT:    s_bfe_u32 s11, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s13, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s15, s2, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s16, s2, 0x40004
+; GFX8-NEXT:    s_and_b32 s2, s2, 15
+; GFX8-NEXT:    s_lshr_b32 s3, s1, 28
+; GFX8-NEXT:    s_bfe_u32 s5, s1, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s7, s1, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s8, s1, 0x40008
+; GFX8-NEXT:    s_and_b32 s1, s1, 15
+; GFX8-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s16
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s9
+; GFX8-NEXT:    v_mul_u32_u24_e32 v3, s1, v3
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s15
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s14
 ; GFX8-NEXT:    v_mov_b32_e32 v8, s7
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v6
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x40018
 ; GFX8-NEXT:    v_mov_b32_e32 v9, s13
-; GFX8-NEXT:    s_bfe_u32 s1, s1, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v10, s12
 ; GFX8-NEXT:    v_mov_b32_e32 v11, s5
+; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v6
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v3, s0, v3
+; GFX8-NEXT:    v_mul_u32_u24_e32 v7, s6, v9
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v8, v11, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
+; GFX8-NEXT:    v_or_b32_e32 v5, v5, v6
 ; GFX8-NEXT:    v_mov_b32_e32 v12, s11
 ; GFX8-NEXT:    v_mov_b32_e32 v13, s10
 ; GFX8-NEXT:    v_mov_b32_e32 v14, s3
-; GFX8-NEXT:    v_mul_u32_u24_e32 v3, s1, v3
-; GFX8-NEXT:    v_or_b32_e32 v5, v5, v6
-; GFX8-NEXT:    v_mul_u32_u24_e32 v7, s6, v9
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v8, v11, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
 ; GFX8-NEXT:    v_mul_u32_u24_e32 v9, s4, v12
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v10, v14, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_and_b32_e32 v5, s0, v5
-; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v4, s0, v7
 ; GFX8-NEXT:    v_or_b32_e32 v9, v9, v10
-; GFX8-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_and_b32_e32 v4, s0, v9
-; GFX8-NEXT:    v_or_b32_e32 v3, v5, v3
-; GFX8-NEXT:    v_or_b32_e32 v6, v4, v7
-; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v3
-; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v6
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 8, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v9
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v7, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_2 src1_sel:BYTE_0
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v8, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v6
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v10
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v9
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v7
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc8_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s22, -1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_ubyte v1, v0, s[0:1]
 ; GFX9-NEXT:    s_mov_b32 s23, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s20, s20, s3
 ; GFX9-NEXT:    s_load_dword s3, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX9-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_bfe_u32 s5, s3, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s12, s4, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s14, s4, 0x40018
-; GFX9-NEXT:    s_lshr_b32 s15, s4, 28
-; GFX9-NEXT:    s_and_b32 s16, s4, 15
-; GFX9-NEXT:    s_bfe_u32 s17, s4, 0x40004
-; GFX9-NEXT:    s_bfe_u32 s18, s4, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s5, s3, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s12, s4, 0x40018
+; GFX9-NEXT:    s_lshr_b32 s13, s4, 28
+; GFX9-NEXT:    s_bfe_u32 s14, s4, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s15, s4, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s16, s4, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s17, s4, 0x4000c
+; GFX9-NEXT:    s_and_b32 s18, s4, 15
+; GFX9-NEXT:    s_bfe_u32 s4, s4, 0x40004
 ; GFX9-NEXT:    v_mov_b32_e32 v2, s12
-; GFX9-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s6, s3, 0x40014
+; GFX9-NEXT:    s_lshr_b32 s6, s3, 28
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s13
-; GFX9-NEXT:    s_bfe_u32 s7, s3, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s7, s3, 0x40010
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s14
-; GFX9-NEXT:    s_lshr_b32 s8, s3, 28
+; GFX9-NEXT:    s_bfe_u32 s8, s3, 0x40014
 ; GFX9-NEXT:    v_mov_b32_e32 v5, s15
-; GFX9-NEXT:    s_and_b32 s9, s3, 15
-; GFX9-NEXT:    v_mov_b32_e32 v6, s16
-; GFX9-NEXT:    s_bfe_u32 s10, s3, 0x40004
-; GFX9-NEXT:    v_mov_b32_e32 v7, s17
-; GFX9-NEXT:    s_bfe_u32 s11, s3, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s9, s3, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s10, s3, 0x4000c
+; GFX9-NEXT:    s_and_b32 s11, s3, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v8, s18
-; GFX9-NEXT:    s_bfe_u32 s3, s3, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s3, s3, 0x40004
 ; GFX9-NEXT:    v_mov_b32_e32 v9, s4
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v2, s5, v2
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v3, s6, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v4, s7, v4
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, s8, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v8, s11, v8
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, s3, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mov_b32_e32 v6, s16
+; GFX9-NEXT:    v_mov_b32_e32 v7, s17
+; GFX9-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX9-NEXT:    v_or_b32_e32 v3, v4, v5
+; GFX9-NEXT:    v_or_b32_e32 v5, v8, v9
+; GFX9-NEXT:    v_and_b32_e32 v5, s2, v5
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v6, s9, v6
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v7, s10, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX9-NEXT:    v_or_b32_sdwa v3, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_e32 v4, v6, v7
-; GFX9-NEXT:    v_mul_lo_u16_e32 v8, s11, v8
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, s3, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-NEXT:    v_or_b32_sdwa v5, v8, v9 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v5, v4, v5
-; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
-; GFX9-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX9-NEXT:    v_or_b32_e32 v3, v2, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v5
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX9-NEXT:    v_and_b32_e32 v3, s2, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u32_e32 v1, v4, v1
+; GFX9-NEXT:    v_add_u32_e32 v1, v5, v1
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v7
+; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v6
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v3
+; GFX9-NEXT:    v_add_u32_e32 v1, v1, v8
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
 ; GFX9-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-DL-NEXT:    s_mov_b32 s20, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s21, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s22, -1
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_ubyte v1, v0, s[0:1]
 ; GFX9-DL-NEXT:    s_mov_b32 s23, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s20, s20, s3
 ; GFX9-DL-NEXT:    s_load_dword s3, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_addc_u32 s21, s21, 0
 ; GFX9-DL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s3, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s4, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s14, s4, 0x40018
-; GFX9-DL-NEXT:    s_lshr_b32 s15, s4, 28
-; GFX9-DL-NEXT:    s_and_b32 s16, s4, 15
-; GFX9-DL-NEXT:    s_bfe_u32 s17, s4, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s18, s4, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s5, s3, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s12, s4, 0x40018
+; GFX9-DL-NEXT:    s_lshr_b32 s13, s4, 28
+; GFX9-DL-NEXT:    s_bfe_u32 s14, s4, 0x40010
+; GFX9-DL-NEXT:    s_bfe_u32 s15, s4, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s16, s4, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s17, s4, 0x4000c
+; GFX9-DL-NEXT:    s_and_b32 s18, s4, 15
+; GFX9-DL-NEXT:    s_bfe_u32 s4, s4, 0x40004
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v2, s12
-; GFX9-DL-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s3, 0x40014
+; GFX9-DL-NEXT:    s_lshr_b32 s6, s3, 28
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s13
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s3, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s3, 0x40010
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s14
-; GFX9-DL-NEXT:    s_lshr_b32 s8, s3, 28
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s3, 0x40014
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s15
-; GFX9-DL-NEXT:    s_and_b32 s9, s3, 15
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s16
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s3, 0x40004
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s17
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s3, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s3, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s10, s3, 0x4000c
+; GFX9-DL-NEXT:    s_and_b32 s11, s3, 15
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s18
-; GFX9-DL-NEXT:    s_bfe_u32 s3, s3, 0x4000c
+; GFX9-DL-NEXT:    s_bfe_u32 s3, s3, 0x40004
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s4
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v2, s5, v2
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v3, s6, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v4, s7, v4
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v5, s8, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v8, s11, v8
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v9, s3, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s16
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s17
+; GFX9-DL-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX9-DL-NEXT:    v_or_b32_e32 v3, v4, v5
+; GFX9-DL-NEXT:    v_or_b32_e32 v5, v8, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, s2, v5
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v6, s9, v6
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v7, s10, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_or_b32_e32 v4, v6, v7
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v8, s11, v8
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v9, s3, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v8, v9 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v5, v4, v5
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v5
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, s2, v2
-; GFX9-DL-NEXT:    v_or_b32_e32 v3, v2, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v5
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, s2, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u32_e32 v1, v4, v1
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v5, v1
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v7
+; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v6
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v3
+; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v8
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
 ; GFX9-DL-NEXT:    v_add_u32_e32 v1, v1, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    global_store_byte v0, v1, s[0:1]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    s_mov_b32 s12, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s13, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s14, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s15, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s12, s12, s3
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_addc_u32 s13, s13, 0
+; GFX10-DL-NEXT:    s_mov_b32 s6, 0xffff
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    global_load_ubyte v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX10-DL-NEXT:    s_load_dword s1, s[2:3], 0x0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-DL-NEXT:    s_bfe_u32 s3, s0, 0x40004
-; GFX10-DL-NEXT:    s_bfe_u32 s7, s1, 0x40004
+; GFX10-DL-NEXT:    s_bfe_u32 s2, s0, 0x40004
+; GFX10-DL-NEXT:    s_bfe_u32 s3, s1, 0x40004
+; GFX10-DL-NEXT:    s_bfe_u32 s8, s0, 0x4000c
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, s2, s3
 ; GFX10-DL-NEXT:    s_and_b32 s2, s0, 15
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, s3, s7
 ; GFX10-DL-NEXT:    s_and_b32 s3, s1, 15
-; GFX10-DL-NEXT:    s_bfe_u32 s8, s0, 0x4000c
-; GFX10-DL-NEXT:    s_bfe_u32 s7, s1, 0x4000c
+; GFX10-DL-NEXT:    s_bfe_u32 s10, s1, 0x4000c
 ; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, s2, s3
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s8, s7
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 8, v2
-; GFX10-DL-NEXT:    s_bfe_u32 s6, s0, 0x40008
-; GFX10-DL-NEXT:    s_bfe_u32 s2, s1, 0x40008
-; GFX10-DL-NEXT:    s_mov_b32 s3, 0xffff
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v5, s6, s2
+; GFX10-DL-NEXT:    s_bfe_u32 s3, s0, 0x40014
+; GFX10-DL-NEXT:    s_bfe_u32 s11, s1, 0x40014
+; GFX10-DL-NEXT:    s_bfe_u32 s7, s0, 0x40008
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s3, s11
 ; GFX10-DL-NEXT:    v_or_b32_e32 v2, v3, v2
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v4, 8, v4
-; GFX10-DL-NEXT:    s_bfe_u32 s6, s0, 0x40014
-; GFX10-DL-NEXT:    s_bfe_u32 s8, s1, 0x40014
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, s8, s10
+; GFX10-DL-NEXT:    s_bfe_u32 s8, s1, 0x40008
 ; GFX10-DL-NEXT:    s_bfe_u32 s2, s0, 0x40010
-; GFX10-DL-NEXT:    v_and_b32_e32 v2, s3, v2
-; GFX10-DL-NEXT:    v_or_b32_sdwa v3, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s6, s8
-; GFX10-DL-NEXT:    s_bfe_u32 s7, s0, 0x40018
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v5, s7, s8
+; GFX10-DL-NEXT:    v_and_b32_e32 v2, s6, v2
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v3
 ; GFX10-DL-NEXT:    s_bfe_u32 s9, s1, 0x40010
-; GFX10-DL-NEXT:    s_lshr_b32 s0, s0, 28
-; GFX10-DL-NEXT:    v_or_b32_e32 v3, v2, v3
-; GFX10-DL-NEXT:    s_lshr_b32 s6, s1, 28
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v5, s2, s9
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v6, s0, s6
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v4, 8, v4
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v3
-; GFX10-DL-NEXT:    s_bfe_u32 s0, s1, 0x40018
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v11, s7, s0
-; GFX10-DL-NEXT:    v_or_b32_e32 v4, v5, v4
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v6, 8, v6
-; GFX10-DL-NEXT:    v_and_b32_e32 v4, s3, v4
+; GFX10-DL-NEXT:    s_lshr_b32 s3, s1, 28
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v2
+; GFX10-DL-NEXT:    v_or_b32_e32 v3, v5, v3
+; GFX10-DL-NEXT:    s_bfe_u32 s1, s1, 0x40018
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v2, v1
-; GFX10-DL-NEXT:    v_or_b32_sdwa v2, v11, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v7
-; GFX10-DL-NEXT:    v_or_b32_e32 v2, v4, v2
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v2
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, s2, s9
+; GFX10-DL-NEXT:    s_bfe_u32 s2, s0, 0x40018
+; GFX10-DL-NEXT:    s_lshr_b32 s0, s0, 28
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v6
+; GFX10-DL-NEXT:    v_or_b32_e32 v2, v2, v4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 8, v3
+; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v3, s0, s3
+; GFX10-DL-NEXT:    v_and_b32_e32 v2, s6, v2
 ; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v4
-; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v3
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX10-DL-NEXT:    v_add_nc_u32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v4, s2, s1
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v3, 8, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v2
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v7, v1, v2
+; GFX10-DL-NEXT:    v_or_b32_e32 v2, v4, v3
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v7, v5
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX10-DL-NEXT:    v_add_nc_u32_e32 v1, v1, v2
 ; GFX10-DL-NEXT:    global_store_byte v0, v1, s[4:5]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll

 ; VI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dword v4, v[0:1] glc
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; VI-NEXT:    s_mov_b32 s3, 0
 ; VI-NEXT:    s_mov_b32 s2, 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    s_mov_b32 s3, 0
-; VI-NEXT:    s_and_b32 s1, s4, s2
+; VI-NEXT:    s_lshl_b32 s1, s4, 16
+; VI-NEXT:    s_and_b32 s4, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
-; VI-NEXT:    s_lshl_b32 s0, s1, 16
-; VI-NEXT:    s_or_b32 s0, s1, s0
+; VI-NEXT:    s_or_b32 s0, s4, s1
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    v_lshlrev_b32_e32 v4, 4, v4
 ; VI-NEXT:    v_lshlrev_b64 v[4:5], v4, s[2:3]
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_bfi_b32 v1, v5, s0, v1
 ; VI-NEXT:    v_bfi_b32 v0, v4, s0, v0
 ; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; VI-NEXT:    s_endpgm

 ; VI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
 ; VI-NEXT:    s_mov_b32 s2, 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    s_mov_b32 s3, 0
 ; VI-NEXT:    s_lshl_b32 s1, s5, 4
+; VI-NEXT:    s_lshl_b32 s5, s4, 16
+; VI-NEXT:    s_mov_b32 s3, 0
 ; VI-NEXT:    s_and_b32 s4, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[2:3], s1
-; VI-NEXT:    s_lshl_b32 s2, s4, 16
-; VI-NEXT:    s_or_b32 s2, s4, s2
+; VI-NEXT:    s_or_b32 s2, s4, s5
 ; VI-NEXT:    v_mov_b32_e32 v4, s2
 ; VI-NEXT:    v_mov_b32_e32 v5, s2
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_bfi_b32 v1, s1, v4, v1
 ; VI-NEXT:    v_bfi_b32 v0, s0, v5, v0
 ; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; VI-NEXT:    s_endpgm

llvm/test/CodeGen/AMDGPU/saddsat.ll

 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    s_movk_i32 s4, 0x7fff
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_i32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_movk_i32 s5, 0x8000
 ; GFX6-NEXT:    v_min_i32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_max_i32_e32 v1, s5, v1
 ; GFX6-NEXT:    v_max_i32_e32 v0, s5, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    s_mov_b32 s4, 0xffff
+; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_saddsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v2, 16, v1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
 ; GFX8-NEXT:    v_add_u16_e32 v4, v3, v2

llvm/test/CodeGen/AMDGPU/scalar_to_vector.ll

 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_mov_b32 s4, s6
 ; VI-NEXT:    s_mov_b32 s5, s7
 ; VI-NEXT:    s_mov_b32 s6, s2
 ; VI-NEXT:    s_mov_b32 s7, s3
 ; VI-NEXT:    buffer_load_dword v0, off, s[4:7], 0
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; VI-NEXT:    v_alignbit_b32 v0, v1, v0, 16
+; VI-NEXT:    v_alignbit_b32 v0, s0, v0, 16
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %tmp1 = load i32, i32 addrspace(1)* %in, align 4
   %bc = bitcast i32 %tmp1 to <2 x i16>
   %tmp2 = shufflevector <2 x i16> %bc, <2 x i16> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
   store <4 x i16> %tmp2, <4 x i16> addrspace(1)* %out, align 8
   ret void

 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_mov_b32 s4, s6
 ; VI-NEXT:    s_mov_b32 s5, s7
 ; VI-NEXT:    s_mov_b32 s6, s2
 ; VI-NEXT:    s_mov_b32 s7, s3
 ; VI-NEXT:    buffer_load_dword v0, off, s[4:7], 0
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; VI-NEXT:    v_alignbit_b32 v0, v1, v0, 16
+; VI-NEXT:    v_alignbit_b32 v0, s0, v0, 16
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %tmp1 = load float, float addrspace(1)* %in, align 4
   %bc = bitcast float %tmp1 to <2 x i16>
   %tmp2 = shufflevector <2 x i16> %bc, <2 x i16> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
   store <4 x i16> %tmp2, <4 x i16> addrspace(1)* %out, align 8
   ret void
 }
 
 define amdgpu_kernel void @scalar_to_vector_v4i16() {
 ; SI-LABEL: scalar_to_vector_v4i16:
 ; SI:       ; %bb.0: ; %bb
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v1, v0
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 8, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v1, v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: scalar_to_vector_v4i16:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_lshlrev_b16_e32 v1, 8, v0
-; VI-NEXT:    v_or_b32_e32 v0, v1, v0
-; VI-NEXT:    v_lshrrev_b16_e32 v1, 8, v0
-; VI-NEXT:    v_lshlrev_b16_e32 v2, 8, v1
-; VI-NEXT:    v_or_b32_e32 v1, v1, v2
-; VI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; VI-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; VI-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_or_b32_e32 v0, v0, v1
+; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; VI-NEXT:    v_mov_b32_e32 v1, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 bb:
   %tmp = load <2 x i8>, <2 x i8> addrspace(1)* undef, align 1
   %tmp1 = shufflevector <2 x i8> %tmp, <2 x i8> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
   %tmp2 = shufflevector <8 x i8> %tmp1, <8 x i8> undef, <8 x i32> <i32 0, i32 9, i32 9, i32 9, i32 9, i32 9, i32 9, i32 9>
   store <8 x i8> %tmp2, <8 x i8> addrspace(1)* undef, align 8
   ret void
 }
 
 define amdgpu_kernel void @scalar_to_vector_v4f16() {
 ; SI-LABEL: scalar_to_vector_v4f16:
 ; SI:       ; %bb.0: ; %bb
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_or_b32_e32 v0, v1, v0
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v0
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 8, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
-; SI-NEXT:    v_or_b32_e32 v1, v1, v2
-; SI-NEXT:    v_or_b32_e32 v0, v0, v2
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_or_b32_e32 v0, v0, v1
+; SI-NEXT:    v_mov_b32_e32 v1, v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: scalar_to_vector_v4f16:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    buffer_load_ubyte v0, off, s[0:3], 0

llvm/test/CodeGen/AMDGPU/shift-i128.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i128 %shl
 }
 
 
 define i128 @v_shl_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_shl_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_alignbit_b32 v4, v2, v1, 15
+; GCN-NEXT:    v_lshl_b64 v[2:3], v[2:3], 17
+; GCN-NEXT:    v_lshrrev_b32_e32 v4, 15, v1
 ; GCN-NEXT:    v_alignbit_b32 v1, v1, v0, 15
-; GCN-NEXT:    v_alignbit_b32 v3, v3, v2, 15
+; GCN-NEXT:    v_or_b32_e32 v2, v2, v4
 ; GCN-NEXT:    v_lshlrev_b32_e32 v0, 17, v0
-; GCN-NEXT:    v_mov_b32_e32 v2, v4
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = shl i128 %lhs, 17
   ret i128 %shl
 }
 
 define i128 @v_lshr_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_lshr_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    v_alignbit_b32 v0, v3, v2, 1
 ; GCN-NEXT:    v_lshrrev_b32_e32 v1, 1, v3
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    v_mov_b32_e32 v3, 0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = lshr i128 %lhs, 65
   ret i128 %shl
 }
 
 define i128 @v_ashr_i128_vk(i128 %lhs) {
 ; GCN-LABEL: v_ashr_i128_vk:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_ashr_i64 v[4:5], v[2:3], 33
-; GCN-NEXT:    v_alignbit_b32 v0, v2, v1, 1
-; GCN-NEXT:    v_alignbit_b32 v1, v3, v2, 1
-; GCN-NEXT:    v_mov_b32_e32 v2, v4
-; GCN-NEXT:    v_mov_b32_e32 v3, v5
+; GCN-NEXT:    v_mov_b32_e32 v4, v1
+; GCN-NEXT:    v_lshl_b64 v[0:1], v[2:3], 31
+; GCN-NEXT:    v_lshrrev_b32_e32 v4, 1, v4
+; GCN-NEXT:    v_ashr_i64 v[2:3], v[2:3], 33
+; GCN-NEXT:    v_or_b32_e32 v0, v4, v0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = ashr i128 %lhs, 33
   ret i128 %shl
 }
 
 define i128 @v_shl_i128_kv(i128 %rhs) {
 ; GCN-LABEL: v_shl_i128_kv:
 ; GCN:       ; %bb.0:

llvm/test/CodeGen/AMDGPU/ssubsat.ll

 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    s_movk_i32 s4, 0x7fff
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_i32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_movk_i32 s5, 0x8000
 ; GFX6-NEXT:    v_min_i32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_max_i32_e32 v1, s5, v1
 ; GFX6-NEXT:    v_max_i32_e32 v0, s5, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    s_mov_b32 s4, 0xffff
+; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
+; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_ssubsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v2, 16, v1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
 ; GFX8-NEXT:    v_sub_u16_e32 v4, v3, v2

llvm/test/CodeGen/AMDGPU/trunc-combine.ll

bb:
   store <2 x i16> %tmp14, <2 x i16> addrspace(1)* %tmp15, align 4
   ret void
 }
 
 define <2 x i16> @trunc_v2i64_arg_to_v2i16(<2 x i64> %arg0) #0 {
 ; SI-LABEL: trunc_v2i64_arg_to_v2i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SI-NEXT:    s_mov_b32 s4, 0xffff
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
-; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; SI-NEXT:    v_and_b32_e32 v0, s4, v0
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; SI-NEXT:    v_and_b32_e32 v1, s4, v2
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: trunc_v2i64_arg_to_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_lshlrev_b32_e32 v1, 16, v2
 ; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; VI-NEXT:    s_setpc_b64 s[30:31]
   %trunc = trunc <2 x i64> %arg0 to <2 x i16>
   ret <2 x i16> %trunc
 }

llvm/test/CodeGen/AMDGPU/uaddsat.ll

 ; GFX6-NEXT:    v_and_b32_e32 v3, s4, v3
 ; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_min_u32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_min_u32_e32 v0, s4, v0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v1
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_uaddsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_add_u16_sdwa v2, v0, v1 clamp dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_add_u16_e64 v0, v0, v1 clamp
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD

llvm/test/CodeGen/AMDGPU/usubsat.ll

 ; GFX6-NEXT:    v_and_b32_e32 v4, s4, v2
 ; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_and_b32_e32 v5, s4, v3
 ; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    v_max_u32_e32 v1, v1, v5
 ; GFX6-NEXT:    v_max_u32_e32 v0, v0, v4
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
-; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
-; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX6-NEXT:    v_or_b32_e32 v0, v0, v3
+; GFX6-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_usubsat_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_sub_u16_sdwa v2, v0, v1 clamp dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; GFX8-NEXT:    v_sub_u16_e64 v0, v0, v1 clamp
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD

llvm/test/CodeGen/ARM/illegal-bitfield-loadstore.ll

 ; LE-NEXT:    ldr r1, [r0]
 ; LE-NEXT:    orr r1, r1, #384
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_or:
 ; BE:       @ %bb.0:
 ; BE-NEXT:    mov r1, r0
-; BE-NEXT:    ldr r12, [r0]
+; BE-NEXT:    ldr r0, [r0]
 ; BE-NEXT:    ldrh r2, [r1, #4]!
 ; BE-NEXT:    ldrb r3, [r1, #2]
 ; BE-NEXT:    orr r2, r3, r2, lsl #8
-; BE-NEXT:    orr r2, r2, r12, lsl #24
-; BE-NEXT:    orr r2, r2, #384
-; BE-NEXT:    strb r2, [r1, #2]
-; BE-NEXT:    lsr r3, r2, #8
-; BE-NEXT:    strh r3, [r1]
-; BE-NEXT:    bic r1, r12, #255
-; BE-NEXT:    orr r1, r1, r2, lsr #24
-; BE-NEXT:    str r1, [r0]
+; BE-NEXT:    orr r0, r2, r0, lsl #24
+; BE-NEXT:    orr r0, r0, #384
+; BE-NEXT:    strb r0, [r1, #2]
+; BE-NEXT:    lsr r0, r0, #8
+; BE-NEXT:    strh r0, [r1]
 ; BE-NEXT:    mov pc, lr
   %aa = load i56, i56* %a
   %b = or i56 %aa, 384
   store i56 %b, i56* %a
   ret void
 }
 
 define void @i56_and_or(i56* %a) {
 ; LE-LABEL: i56_and_or:
 ; LE:       @ %bb.0:
 ; LE-NEXT:    ldr r1, [r0]
 ; LE-NEXT:    orr r1, r1, #384
 ; LE-NEXT:    bic r1, r1, #127
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_and_or:
 ; BE:       @ %bb.0:
-; BE-NEXT:    mov r1, r0
+; BE-NEXT:    ldrh r1, [r0, #4]!
 ; BE-NEXT:    mov r2, #128
-; BE-NEXT:    ldrh r12, [r1, #4]!
-; BE-NEXT:    ldrb r3, [r1, #2]
-; BE-NEXT:    strb r2, [r1, #2]
-; BE-NEXT:    orr r2, r3, r12, lsl #8
-; BE-NEXT:    ldr r12, [r0]
-; BE-NEXT:    orr r2, r2, r12, lsl #24
-; BE-NEXT:    orr r2, r2, #384
-; BE-NEXT:    lsr r3, r2, #8
-; BE-NEXT:    strh r3, [r1]
-; BE-NEXT:    bic r1, r12, #255
-; BE-NEXT:    orr r1, r1, r2, lsr #24
-; BE-NEXT:    str r1, [r0]
+; BE-NEXT:    orr r1, r1, #1
+; BE-NEXT:    strb r2, [r0, #2]
+; BE-NEXT:    strh r1, [r0]
 ; BE-NEXT:    mov pc, lr
 
   %b = load i56, i56* %a, align 1
   %c = and i56 %b, -128
   %d = or i56 %c, 384
   store i56 %d, i56* %a, align 1
   ret void
 }
 
 define void @i56_insert_bit(i56* %a, i1 zeroext %bit) {
 ; LE-LABEL: i56_insert_bit:
 ; LE:       @ %bb.0:
 ; LE-NEXT:    ldr r2, [r0]
 ; LE-NEXT:    bic r2, r2, #8192
 ; LE-NEXT:    orr r1, r2, r1, lsl #13
 ; LE-NEXT:    str r1, [r0]
 ; LE-NEXT:    mov pc, lr
 ;
 ; BE-LABEL: i56_insert_bit:
 ; BE:       @ %bb.0:
-; BE-NEXT:    .save {r11, lr}
-; BE-NEXT:    push {r11, lr}
-; BE-NEXT:    mov r2, r0
-; BE-NEXT:    ldr lr, [r0]
-; BE-NEXT:    ldrh r12, [r2, #4]!
-; BE-NEXT:    ldrb r3, [r2, #2]
-; BE-NEXT:    orr r12, r3, r12, lsl #8
-; BE-NEXT:    orr r3, r12, lr, lsl #24
-; BE-NEXT:    bic r3, r3, #8192
-; BE-NEXT:    orr r1, r3, r1, lsl #13
-; BE-NEXT:    lsr r3, r1, #8
-; BE-NEXT:    strh r3, [r2]
-; BE-NEXT:    bic r2, lr, #255
-; BE-NEXT:    orr r1, r2, r1, lsr #24
-; BE-NEXT:    str r1, [r0]
-; BE-NEXT:    pop {r11, lr}
+; BE-NEXT:    ldrh r2, [r0, #4]!
+; BE-NEXT:    mov r3, #57088
+; BE-NEXT:    orr r3, r3, #16711680
+; BE-NEXT:    and r2, r3, r2, lsl #8
+; BE-NEXT:    orr r1, r2, r1, lsl #13
+; BE-NEXT:    lsr r1, r1, #8
+; BE-NEXT:    strh r1, [r0]
 ; BE-NEXT:    mov pc, lr
   %extbit = zext i1 %bit to i56
   %b = load i56, i56* %a, align 1
   %extbit.shl = shl nuw nsw i56 %extbit, 13
   %c = and i56 %b, -8193
   %d = or i56 %c, %extbit.shl
   store i56 %d, i56* %a, align 1
   ret void
 }
 

llvm/test/CodeGen/ARM/parity.ll

; CHECK-NEXT:    bx lr
   %2 = and i16 %1, 1
   ret i16 %2
 }
 
 define i17 @parity_17(i17 %x) {
 ; CHECK-LABEL: parity_17:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    bfc r0, #17, #15
-; CHECK-NEXT:    eor r0, r0, r0, lsr #16
-; CHECK-NEXT:    eor r0, r0, r0, lsr #8
+; CHECK-NEXT:    eor r1, r0, r0, lsr #16
+; CHECK-NEXT:    eor r0, r1, r0, lsr #8
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #4
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #2
 ; CHECK-NEXT:    eor r0, r0, r0, lsr #1
 ; CHECK-NEXT:    and r0, r0, #1
 ; CHECK-NEXT:    bx lr
   %1 = tail call i17 @llvm.ctpop.i17(i17 %x)
   %2 = and i17 %1, 1
   ret i17 %2

llvm/test/CodeGen/ARM/ror.ll

entry:
   %result.i.3 = or i32 %low_part.i.2, %high_part.i.1
   ret i32 %result.i.3
 }
 
 ; the same vector test
 define <2 x i32> @test2(<2 x i32> %x) nounwind readnone {
 ; CHECK-LABEL: test2:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    ror r0, r0, #10
-; CHECK-NEXT:    ror r1, r1, #10
+; CHECK-NEXT:    bic r2, r0, #15
+; CHECK-NEXT:    ror r0, r0, #4
+; CHECK-NEXT:    lsr r0, r0, #6
+; CHECK-NEXT:    orr r0, r0, r2, lsl #22
+; CHECK-NEXT:    bic r2, r1, #15
+; CHECK-NEXT:    ror r1, r1, #4
+; CHECK-NEXT:    lsr r1, r1, #6
+; CHECK-NEXT:    orr r1, r1, r2, lsl #22
 ; CHECK-NEXT:    bx lr
 entry:
   %high_part.i = shl <2 x i32> %x, <i32 28, i32 28>
   %low_part.i = lshr <2 x i32> %x, <i32 4, i32 4>
   %result.i = or <2 x i32> %high_part.i, %low_part.i
   %high_part.i.1 = shl <2 x i32> %result.i, <i32 26, i32 26>
   %low_part.i.2 = lshr <2 x i32> %result.i, <i32 6, i32 6>
   %result.i.3 = or <2 x i32> %low_part.i.2, %high_part.i.1
   ret <2 x i32> %result.i.3
 }
 

llvm/test/CodeGen/ARM/uxtb.ll

; CHECK-NEXT:    bx lr
   %tmp6 = or i32 %tmp5, %tmp1
   ret i32 %tmp6
 }
 
 define i32 @test10(i32 %p0) {
 ; CHECK-LABEL: test10:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    mov r1, #248
+; CHECK-NEXT:    mov r2, #7
 ; CHECK-NEXT:    orr r1, r1, #16252928
-; CHECK-NEXT:    and r0, r1, r0, lsr #7
-; CHECK-NEXT:    lsr r1, r0, #5
-; CHECK-NEXT:    uxtb16 r1, r1
-; CHECK-NEXT:    orr r0, r1, r0
+; CHECK-NEXT:    orr r2, r2, #458752
+; CHECK-NEXT:    and r1, r1, r0, lsr #7
+; CHECK-NEXT:    and r0, r2, r0, lsr #12
+; CHECK-NEXT:    orr r0, r0, r1
 ; CHECK-NEXT:    bx lr
   %tmp1 = lshr i32 %p0, 7
   %tmp2 = and i32 %tmp1, 16253176
   %tmp4 = lshr i32 %tmp2, 5
   %tmp5 = and i32 %tmp4, 458759
   %tmp7 = or i32 %tmp5, %tmp2
   ret i32 %tmp7
 }
 

llvm/test/CodeGen/Mips/funnel-shift.ll

 ; CHECK-BE-NEXT:    andi $2, $3, 63
 ; CHECK-BE-NEXT:    not $4, $2
 ; CHECK-BE-NEXT:    srl $5, $18, 1
 ; CHECK-BE-NEXT:    sllv $6, $19, $2
 ; CHECK-BE-NEXT:    srlv $4, $5, $4
 ; CHECK-BE-NEXT:    andi $5, $1, 63
 ; CHECK-BE-NEXT:    srl $7, $16, 5
 ; CHECK-BE-NEXT:    sll $8, $17, 27
-; CHECK-BE-NEXT:    or $7, $8, $7
-; CHECK-BE-NEXT:    srl $8, $7, 1
+; CHECK-BE-NEXT:    or $8, $8, $7
+; CHECK-BE-NEXT:    srl $8, $8, 1
 ; CHECK-BE-NEXT:    srlv $9, $8, $5
 ; CHECK-BE-NEXT:    andi $1, $1, 32
 ; CHECK-BE-NEXT:    move $10, $9
 ; CHECK-BE-NEXT:    movn $10, $zero, $1
 ; CHECK-BE-NEXT:    or $4, $6, $4
 ; CHECK-BE-NEXT:    sllv $6, $18, $2
 ; CHECK-BE-NEXT:    andi $3, $3, 32
 ; CHECK-BE-NEXT:    movn $4, $6, $3

 ; CHECK-LE-NEXT:    andi $3, $2, 63
 ; CHECK-LE-NEXT:    not $4, $3
 ; CHECK-LE-NEXT:    srl $5, $19, 1
 ; CHECK-LE-NEXT:    sllv $6, $18, $3
 ; CHECK-LE-NEXT:    srlv $4, $5, $4
 ; CHECK-LE-NEXT:    andi $5, $1, 63
 ; CHECK-LE-NEXT:    srl $7, $17, 5
 ; CHECK-LE-NEXT:    sll $8, $16, 27
-; CHECK-LE-NEXT:    or $7, $8, $7
-; CHECK-LE-NEXT:    srl $8, $7, 1
+; CHECK-LE-NEXT:    or $8, $8, $7
+; CHECK-LE-NEXT:    srl $8, $8, 1
 ; CHECK-LE-NEXT:    srlv $9, $8, $5
 ; CHECK-LE-NEXT:    andi $1, $1, 32
 ; CHECK-LE-NEXT:    move $10, $9
 ; CHECK-LE-NEXT:    movn $10, $zero, $1
 ; CHECK-LE-NEXT:    or $4, $6, $4
 ; CHECK-LE-NEXT:    sllv $6, $19, $3
 ; CHECK-LE-NEXT:    andi $2, $2, 32
 ; CHECK-LE-NEXT:    movn $4, $6, $2

llvm/test/CodeGen/PowerPC/fp-to-int-to-fp.ll

 ; PPC64-NEXT:    ld 5, -16(1)
 ; PPC64-NEXT:    xor 3, 3, 4
 ; PPC64-NEXT:    bc 12, 0, .LBB2_1
 ; PPC64-NEXT:    b .LBB2_2
 ; PPC64-NEXT:  .LBB2_1: # %entry
 ; PPC64-NEXT:    addi 3, 5, 0
 ; PPC64-NEXT:  .LBB2_2: # %entry
 ; PPC64-NEXT:    sradi 4, 3, 53
-; PPC64-NEXT:    clrldi 5, 3, 63
+; PPC64-NEXT:    rldicl 5, 3, 63, 1
 ; PPC64-NEXT:    addi 4, 4, 1
+; PPC64-NEXT:    clrldi 6, 3, 63
 ; PPC64-NEXT:    cmpldi 4, 1
-; PPC64-NEXT:    rldicl 4, 3, 63, 1
-; PPC64-NEXT:    or 5, 5, 4
-; PPC64-NEXT:    rldicl 6, 5, 11, 53
-; PPC64-NEXT:    addi 6, 6, 1
-; PPC64-NEXT:    clrldi 7, 5, 53
-; PPC64-NEXT:    cmpldi 1, 6, 1
-; PPC64-NEXT:    clrldi 6, 3, 53
+; PPC64-NEXT:    clrldi 4, 3, 53
+; PPC64-NEXT:    or 6, 6, 5
+; PPC64-NEXT:    clrldi 7, 6, 53
+; PPC64-NEXT:    addi 4, 4, 2047
 ; PPC64-NEXT:    addi 7, 7, 2047
-; PPC64-NEXT:    addi 6, 6, 2047
-; PPC64-NEXT:    or 4, 7, 4
-; PPC64-NEXT:    or 6, 6, 3
-; PPC64-NEXT:    rldicl 4, 4, 53, 11
-; PPC64-NEXT:    rldicr 6, 6, 0, 52
+; PPC64-NEXT:    or 4, 4, 3
+; PPC64-NEXT:    or 5, 7, 5
+; PPC64-NEXT:    rldicl 7, 3, 10, 54
+; PPC64-NEXT:    rldicr 4, 4, 0, 52
+; PPC64-NEXT:    addi 7, 7, 1
 ; PPC64-NEXT:    bc 12, 1, .LBB2_4
 ; PPC64-NEXT:  # %bb.3: # %entry
-; PPC64-NEXT:    ori 6, 3, 0
+; PPC64-NEXT:    ori 4, 3, 0
 ; PPC64-NEXT:    b .LBB2_4
 ; PPC64-NEXT:  .LBB2_4: # %entry
-; PPC64-NEXT:    rldicl 4, 4, 11, 1
-; PPC64-NEXT:    cmpdi 3, 0
-; PPC64-NEXT:    std 6, -32(1)
-; PPC64-NEXT:    bc 12, 5, .LBB2_6
+; PPC64-NEXT:    rldicl 5, 5, 53, 11
+; PPC64-NEXT:    std 4, -32(1)
+; PPC64-NEXT:    rldicl 4, 5, 11, 1
+; PPC64-NEXT:    cmpldi 7, 1
+; PPC64-NEXT:    bc 12, 1, .LBB2_6
 ; PPC64-NEXT:  # %bb.5: # %entry
-; PPC64-NEXT:    ori 4, 5, 0
+; PPC64-NEXT:    ori 4, 6, 0
 ; PPC64-NEXT:    b .LBB2_6
 ; PPC64-NEXT:  .LBB2_6: # %entry
+; PPC64-NEXT:    cmpdi 3, 0
 ; PPC64-NEXT:    std 4, -24(1)
 ; PPC64-NEXT:    bc 12, 0, .LBB2_8
 ; PPC64-NEXT:  # %bb.7: # %entry
 ; PPC64-NEXT:    lfd 0, -32(1)
 ; PPC64-NEXT:    fcfid 0, 0
 ; PPC64-NEXT:    frsp 1, 0
 ; PPC64-NEXT:    blr
 ; PPC64-NEXT:  .LBB2_8:

llvm/test/CodeGen/RISCV/rv32Zbp.ll

 ; RV32I-NEXT:    lui a2, 838861
 ; RV32I-NEXT:    addi a2, a2, -820
 ; RV32I-NEXT:    and a1, a1, a2
 ; RV32I-NEXT:    srli a3, a0, 2
 ; RV32I-NEXT:    lui a4, 209715
 ; RV32I-NEXT:    addi a4, a4, 819
 ; RV32I-NEXT:    and a3, a3, a4
 ; RV32I-NEXT:    or a0, a3, a0
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    slli a1, a0, 2
-; RV32I-NEXT:    and a1, a1, a2
-; RV32I-NEXT:    srli a2, a0, 2
+; RV32I-NEXT:    or a1, a0, a1
+; RV32I-NEXT:    slli a0, a0, 2
+; RV32I-NEXT:    and a0, a0, a2
+; RV32I-NEXT:    srli a2, a1, 2
 ; RV32I-NEXT:    and a2, a2, a4
-; RV32I-NEXT:    or a0, a2, a0
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    or a1, a2, a1
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32IB-LABEL: gorc2b_i32:
 ; RV32IB:       # %bb.0:
+; RV32IB-NEXT:    srli a1, a0, 2
+; RV32IB-NEXT:    or a1, a1, a0
 ; RV32IB-NEXT:    orc2.n a0, a0
+; RV32IB-NEXT:    slli a1, a1, 2
+; RV32IB-NEXT:    lui a2, 838861
+; RV32IB-NEXT:    addi a2, a2, -820
+; RV32IB-NEXT:    and a1, a1, a2
+; RV32IB-NEXT:    srli a2, a0, 2
+; RV32IB-NEXT:    lui a3, 209715
+; RV32IB-NEXT:    addi a3, a3, 819
+; RV32IB-NEXT:    and a2, a2, a3
+; RV32IB-NEXT:    or a0, a2, a0
+; RV32IB-NEXT:    or a0, a0, a1
 ; RV32IB-NEXT:    ret
 ;
 ; RV32IBP-LABEL: gorc2b_i32:
 ; RV32IBP:       # %bb.0:
+; RV32IBP-NEXT:    srli a1, a0, 2
+; RV32IBP-NEXT:    or a1, a1, a0
 ; RV32IBP-NEXT:    orc2.n a0, a0
+; RV32IBP-NEXT:    slli a1, a1, 2
+; RV32IBP-NEXT:    lui a2, 838861
+; RV32IBP-NEXT:    addi a2, a2, -820
+; RV32IBP-NEXT:    and a1, a1, a2
+; RV32IBP-NEXT:    srli a2, a0, 2
+; RV32IBP-NEXT:    lui a3, 209715
+; RV32IBP-NEXT:    addi a3, a3, 819
+; RV32IBP-NEXT:    and a2, a2, a3
+; RV32IBP-NEXT:    or a0, a2, a0
+; RV32IBP-NEXT:    or a0, a0, a1
 ; RV32IBP-NEXT:    ret
   %and1 = shl i32 %a, 2
   %shl1 = and i32 %and1, -858993460
   %and1b = lshr i32 %a, 2
   %shr1 = and i32 %and1b, 858993459
   %or1 = or i32 %shr1, %a
   %or1b = or i32 %or1, %shl1
   %and2 = shl i32 %or1b, 2
   %shl2 = and i32 %and2, -858993460
   %and2b = lshr i32 %or1b, 2
   %shr2 = and i32 %and2b, 858993459
   %or2 = or i32 %shr2, %or1b
   %or2b = or i32 %or2, %shl2
   ret i32 %or2b
 }
 
 ; gorc2, gorc2 -> gorc2
 define i64 @gorc2b_i64(i64 %a) nounwind {
 ; RV32I-LABEL: gorc2b_i64:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    slli a2, a1, 2
-; RV32I-NEXT:    slli a3, a0, 2
+; RV32I-NEXT:    slli a2, a0, 2
+; RV32I-NEXT:    slli a3, a1, 2
 ; RV32I-NEXT:    lui a4, 838861
 ; RV32I-NEXT:    addi a4, a4, -820
 ; RV32I-NEXT:    and a6, a3, a4
 ; RV32I-NEXT:    and a7, a2, a4
-; RV32I-NEXT:    srli a5, a0, 2
-; RV32I-NEXT:    srli a3, a1, 2
+; RV32I-NEXT:    srli a5, a1, 2
+; RV32I-NEXT:    srli a3, a0, 2
 ; RV32I-NEXT:    lui a2, 209715
 ; RV32I-NEXT:    addi a2, a2, 819
 ; RV32I-NEXT:    and a3, a3, a2
 ; RV32I-NEXT:    and a5, a5, a2
-; RV32I-NEXT:    or a0, a5, a0
-; RV32I-NEXT:    or a1, a3, a1
-; RV32I-NEXT:    or a1, a1, a7
-; RV32I-NEXT:    or a0, a0, a6
-; RV32I-NEXT:    slli a3, a0, 2
-; RV32I-NEXT:    slli a5, a1, 2
-; RV32I-NEXT:    and a6, a5, a4
-; RV32I-NEXT:    and a3, a3, a4
-; RV32I-NEXT:    srli a4, a1, 2
-; RV32I-NEXT:    srli a5, a0, 2
-; RV32I-NEXT:    and a5, a5, a2
+; RV32I-NEXT:    or a1, a5, a1
+; RV32I-NEXT:    or a0, a3, a0
+; RV32I-NEXT:    or a3, a0, a7
+; RV32I-NEXT:    or a5, a1, a6
+; RV32I-NEXT:    slli a0, a0, 2
+; RV32I-NEXT:    slli a1, a1, 2
+; RV32I-NEXT:    and a6, a1, a4
+; RV32I-NEXT:    and a0, a0, a4
+; RV32I-NEXT:    srli a4, a5, 2
+; RV32I-NEXT:    srli a1, a3, 2
+; RV32I-NEXT:    and a1, a1, a2
 ; RV32I-NEXT:    and a2, a4, a2
-; RV32I-NEXT:    or a1, a2, a1
-; RV32I-NEXT:    or a0, a5, a0
-; RV32I-NEXT:    or a0, a0, a3
-; RV32I-NEXT:    or a1, a1, a6
+; RV32I-NEXT:    or a2, a2, a5
+; RV32I-NEXT:    or a1, a1, a3
+; RV32I-NEXT:    or a0, a1, a0
+; RV32I-NEXT:    or a1, a2, a6
 ; RV32I-NEXT:    ret
 ;
 ; RV32IB-LABEL: gorc2b_i64:
 ; RV32IB:       # %bb.0:
-; RV32IB-NEXT:    orc2.n a0, a0
+; RV32IB-NEXT:    srli a2, a1, 2
+; RV32IB-NEXT:    srli a3, a0, 2
+; RV32IB-NEXT:    lui a4, 209715
+; RV32IB-NEXT:    addi a4, a4, 819
+; RV32IB-NEXT:    and a3, a3, a4
+; RV32IB-NEXT:    or a3, a3, a0
+; RV32IB-NEXT:    or a2, a2, a1
 ; RV32IB-NEXT:    orc2.n a1, a1
+; RV32IB-NEXT:    orc2.n a0, a0
+; RV32IB-NEXT:    slli a2, a2, 2
+; RV32IB-NEXT:    slli a3, a3, 2
+; RV32IB-NEXT:    lui a5, 838861
+; RV32IB-NEXT:    addi a5, a5, -820
+; RV32IB-NEXT:    and a6, a3, a5
+; RV32IB-NEXT:    and a2, a2, a5
+; RV32IB-NEXT:    srli a5, a0, 2
+; RV32IB-NEXT:    srli a3, a1, 2
+; RV32IB-NEXT:    and a3, a3, a4
+; RV32IB-NEXT:    and a4, a5, a4
+; RV32IB-NEXT:    or a0, a4, a0
+; RV32IB-NEXT:    or a1, a3, a1
+; RV32IB-NEXT:    or a1, a1, a2
+; RV32IB-NEXT:    or a0, a0, a6
 ; RV32IB-NEXT:    ret
 ;
 ; RV32IBP-LABEL: gorc2b_i64:
 ; RV32IBP:       # %bb.0:
-; RV32IBP-NEXT:    orc2.n a0, a0
+; RV32IBP-NEXT:    srli a2, a1, 2
+; RV32IBP-NEXT:    srli a3, a0, 2
+; RV32IBP-NEXT:    lui a4, 209715
+; RV32IBP-NEXT:    addi a4, a4, 819
+; RV32IBP-NEXT:    and a3, a3, a4
+; RV32IBP-NEXT:    or a3, a3, a0
+; RV32IBP-NEXT:    or a2, a2, a1
 ; RV32IBP-NEXT:    orc2.n a1, a1
+; RV32IBP-NEXT:    orc2.n a0, a0
+; RV32IBP-NEXT:    slli a2, a2, 2
+; RV32IBP-NEXT:    slli a3, a3, 2
+; RV32IBP-NEXT:    lui a5, 838861
+; RV32IBP-NEXT:    addi a5, a5, -820
+; RV32IBP-NEXT:    and a6, a3, a5
+; RV32IBP-NEXT:    and a2, a2, a5
+; RV32IBP-NEXT:    srli a5, a0, 2
+; RV32IBP-NEXT:    srli a3, a1, 2
+; RV32IBP-NEXT:    and a3, a3, a4
+; RV32IBP-NEXT:    and a4, a5, a4
+; RV32IBP-NEXT:    or a0, a4, a0
+; RV32IBP-NEXT:    or a1, a3, a1
+; RV32IBP-NEXT:    or a1, a1, a2
+; RV32IBP-NEXT:    or a0, a0, a6
 ; RV32IBP-NEXT:    ret
   %and1 = shl i64 %a, 2
   %shl1 = and i64 %and1, -3689348814741910324
   %and1b = lshr i64 %a, 2
   %shr1 = and i64 %and1b, 3689348814741910323
   %or1 = or i64 %shr1, %a
   %or1b = or i64 %or1, %shl1
   %and2 = shl i64 %or1b, 2

 ; RV32IBP-NEXT:    ret
   %1 = call i64 @llvm.bitreverse.i64(i64 %a)
   ret i64 %1
 }
 
 define i32 @bswap_rotr_i32(i32 %a) {
 ; RV32I-LABEL: bswap_rotr_i32:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    srli a1, a0, 8
-; RV32I-NEXT:    lui a2, 16
-; RV32I-NEXT:    addi a2, a2, -256
+; RV32I-NEXT:    slli a1, a0, 8
+; RV32I-NEXT:    lui a2, 4080
 ; RV32I-NEXT:    and a1, a1, a2
+; RV32I-NEXT:    slli a2, a0, 24
+; RV32I-NEXT:    or a1, a2, a1
 ; RV32I-NEXT:    srli a2, a0, 24
-; RV32I-NEXT:    or a1, a1, a2
-; RV32I-NEXT:    slli a2, a0, 8
-; RV32I-NEXT:    lui a3, 4080
-; RV32I-NEXT:    and a2, a2, a3
-; RV32I-NEXT:    slli a0, a0, 24
+; RV32I-NEXT:    srli a0, a0, 8
+; RV32I-NEXT:    andi a0, a0, -256
 ; RV32I-NEXT:    or a0, a0, a2
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    slli a1, a0, 16
-; RV32I-NEXT:    srli a0, a0, 16
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    slli a0, a0, 16
+; RV32I-NEXT:    srli a1, a1, 16
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32IB-LABEL: bswap_rotr_i32:
 ; RV32IB:       # %bb.0:
 ; RV32IB-NEXT:    rev8.h a0, a0
 ; RV32IB-NEXT:    ret
 ;
 ; RV32IBP-LABEL: bswap_rotr_i32:
 ; RV32IBP:       # %bb.0:
 ; RV32IBP-NEXT:    rev8.h a0, a0
 ; RV32IBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshr.i32(i32 %1, i32 %1, i32 16)
   ret i32 %2
 }
 
 define i32 @bswap_rotl_i32(i32 %a) {
 ; RV32I-LABEL: bswap_rotl_i32:
 ; RV32I:       # %bb.0:
-; RV32I-NEXT:    srli a1, a0, 8
-; RV32I-NEXT:    lui a2, 16
-; RV32I-NEXT:    addi a2, a2, -256
-; RV32I-NEXT:    and a1, a1, a2
-; RV32I-NEXT:    srli a2, a0, 24
-; RV32I-NEXT:    or a1, a1, a2
+; RV32I-NEXT:    srli a1, a0, 24
+; RV32I-NEXT:    srli a2, a0, 8
+; RV32I-NEXT:    andi a2, a2, -256
+; RV32I-NEXT:    or a1, a2, a1
 ; RV32I-NEXT:    slli a2, a0, 8
 ; RV32I-NEXT:    lui a3, 4080
 ; RV32I-NEXT:    and a2, a2, a3
 ; RV32I-NEXT:    slli a0, a0, 24
 ; RV32I-NEXT:    or a0, a0, a2
-; RV32I-NEXT:    or a0, a0, a1
-; RV32I-NEXT:    srli a1, a0, 16
-; RV32I-NEXT:    slli a0, a0, 16
-; RV32I-NEXT:    or a0, a0, a1
+; RV32I-NEXT:    srli a0, a0, 16
+; RV32I-NEXT:    slli a1, a1, 16
+; RV32I-NEXT:    or a0, a1, a0
 ; RV32I-NEXT:    ret
 ;
 ; RV32IB-LABEL: bswap_rotl_i32:
 ; RV32IB:       # %bb.0:
 ; RV32IB-NEXT:    rev8.h a0, a0
 ; RV32IB-NEXT:    ret
 ;
 ; RV32IBP-LABEL: bswap_rotl_i32:

llvm/test/CodeGen/RISCV/rv64Zbb.ll

 
 define signext i32 @ctpop_i32(i32 signext %a) nounwind {
 ; RV64I-LABEL: ctpop_i32:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    addi sp, sp, -16
 ; RV64I-NEXT:    sd ra, 8(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    slli a1, a0, 32
 ; RV64I-NEXT:    srli a1, a1, 32
-; RV64I-NEXT:    srliw a0, a0, 1
+; RV64I-NEXT:    srli a0, a0, 1
 ; RV64I-NEXT:    lui a2, 349525
 ; RV64I-NEXT:    addiw a2, a2, 1365
 ; RV64I-NEXT:    and a0, a0, a2
 ; RV64I-NEXT:    sub a0, a1, a0
 ; RV64I-NEXT:    srli a1, a0, 2
 ; RV64I-NEXT:    lui a2, 13107
 ; RV64I-NEXT:    addiw a2, a2, 819
 ; RV64I-NEXT:    slli a2, a2, 12

llvm/test/CodeGen/RISCV/rv64Zbbp.ll

 
 ; This is similar to the type legalized roriw pattern, but the and mask is more
 ; than 32 bits so the lshr doesn't shift zeroes into the lower 32 bits. Make
 ; sure we don't match it to roriw.
 define i64 @roriw_bug(i64 %x) nounwind {
 ; RV64I-LABEL: roriw_bug:
 ; RV64I:       # %bb.0:
 ; RV64I-NEXT:    slli a1, a0, 31
-; RV64I-NEXT:    andi a0, a0, -2
-; RV64I-NEXT:    srli a2, a0, 1
-; RV64I-NEXT:    or a1, a1, a2
-; RV64I-NEXT:    sext.w a1, a1
-; RV64I-NEXT:    xor a0, a0, a1
+; RV64I-NEXT:    andi a2, a0, -2
+; RV64I-NEXT:    srli a0, a0, 1
+; RV64I-NEXT:    or a0, a1, a0
+; RV64I-NEXT:    sext.w a0, a0
+; RV64I-NEXT:    xor a0, a2, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: roriw_bug:
 ; RV64IB:       # %bb.0:
 ; RV64IB-NEXT:    slli a1, a0, 31
-; RV64IB-NEXT:    andi a0, a0, -2
-; RV64IB-NEXT:    srli a2, a0, 1
-; RV64IB-NEXT:    or a1, a1, a2
-; RV64IB-NEXT:    sext.w a1, a1
-; RV64IB-NEXT:    xor a0, a0, a1
+; RV64IB-NEXT:    andi a2, a0, -2
+; RV64IB-NEXT:    srli a0, a0, 1
+; RV64IB-NEXT:    or a0, a1, a0
+; RV64IB-NEXT:    sext.w a0, a0
+; RV64IB-NEXT:    xor a0, a2, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBB-LABEL: roriw_bug:
 ; RV64IBB:       # %bb.0:
 ; RV64IBB-NEXT:    slli a1, a0, 31
-; RV64IBB-NEXT:    andi a0, a0, -2
-; RV64IBB-NEXT:    srli a2, a0, 1
-; RV64IBB-NEXT:    or a1, a1, a2
-; RV64IBB-NEXT:    sext.w a1, a1
-; RV64IBB-NEXT:    xor a0, a0, a1
+; RV64IBB-NEXT:    andi a2, a0, -2
+; RV64IBB-NEXT:    srli a0, a0, 1
+; RV64IBB-NEXT:    or a0, a1, a0
+; RV64IBB-NEXT:    sext.w a0, a0
+; RV64IBB-NEXT:    xor a0, a2, a0
 ; RV64IBB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: roriw_bug:
 ; RV64IBP:       # %bb.0:
 ; RV64IBP-NEXT:    slli a1, a0, 31
-; RV64IBP-NEXT:    andi a0, a0, -2
-; RV64IBP-NEXT:    srli a2, a0, 1
-; RV64IBP-NEXT:    or a1, a1, a2
-; RV64IBP-NEXT:    sext.w a1, a1
-; RV64IBP-NEXT:    xor a0, a0, a1
+; RV64IBP-NEXT:    andi a2, a0, -2
+; RV64IBP-NEXT:    srli a0, a0, 1
+; RV64IBP-NEXT:    or a0, a1, a0
+; RV64IBP-NEXT:    sext.w a0, a0
+; RV64IBP-NEXT:    xor a0, a2, a0
 ; RV64IBP-NEXT:    ret
   %a = shl i64 %x, 31
   %b = and i64 %x, 18446744073709551614
   %c = lshr i64 %b, 1
   %d = or i64 %a, %c
   %e = shl i64 %d, 32
   %f = ashr i64 %e, 32
   %g = xor i64 %b, %f ; to increase the use count on %b to disable SimplifyDemandedBits.

llvm/test/CodeGen/RISCV/rv64Zbp.ll

 ; RV64I-NEXT:    lui a2, 838861
 ; RV64I-NEXT:    addiw a2, a2, -820
 ; RV64I-NEXT:    and a1, a1, a2
 ; RV64I-NEXT:    srli a3, a0, 2
 ; RV64I-NEXT:    lui a4, 209715
 ; RV64I-NEXT:    addiw a4, a4, 819
 ; RV64I-NEXT:    and a3, a3, a4
 ; RV64I-NEXT:    or a0, a3, a0
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    slli a1, a0, 2
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 2
+; RV64I-NEXT:    or a1, a0, a1
+; RV64I-NEXT:    slli a0, a0, 2
+; RV64I-NEXT:    and a0, a0, a2
+; RV64I-NEXT:    srli a2, a1, 2
 ; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a0, a2, a0
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    or a1, a2, a1
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    sext.w a0, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: gorc2b_i32:
 ; RV64IB:       # %bb.0:
+; RV64IB-NEXT:    srli a1, a0, 2
+; RV64IB-NEXT:    or a1, a1, a0
 ; RV64IB-NEXT:    gorciw a0, a0, 2
+; RV64IB-NEXT:    slli a1, a1, 2
+; RV64IB-NEXT:    lui a2, 838861
+; RV64IB-NEXT:    addiw a2, a2, -820
+; RV64IB-NEXT:    and a1, a1, a2
+; RV64IB-NEXT:    srli a2, a0, 2
+; RV64IB-NEXT:    lui a3, 209715
+; RV64IB-NEXT:    addiw a3, a3, 819
+; RV64IB-NEXT:    and a2, a2, a3
+; RV64IB-NEXT:    or a0, a2, a0
+; RV64IB-NEXT:    or a0, a0, a1
+; RV64IB-NEXT:    sext.w a0, a0
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: gorc2b_i32:
 ; RV64IBP:       # %bb.0:
+; RV64IBP-NEXT:    srli a1, a0, 2
+; RV64IBP-NEXT:    or a1, a1, a0
 ; RV64IBP-NEXT:    gorciw a0, a0, 2
+; RV64IBP-NEXT:    slli a1, a1, 2
+; RV64IBP-NEXT:    lui a2, 838861
+; RV64IBP-NEXT:    addiw a2, a2, -820
+; RV64IBP-NEXT:    and a1, a1, a2
+; RV64IBP-NEXT:    srli a2, a0, 2
+; RV64IBP-NEXT:    lui a3, 209715
+; RV64IBP-NEXT:    addiw a3, a3, 819
+; RV64IBP-NEXT:    and a2, a2, a3
+; RV64IBP-NEXT:    or a0, a2, a0
+; RV64IBP-NEXT:    or a0, a0, a1
+; RV64IBP-NEXT:    sext.w a0, a0
 ; RV64IBP-NEXT:    ret
   %and1 = shl i32 %a, 2
   %shl1 = and i32 %and1, -858993460
   %and1b = lshr i32 %a, 2
   %shr1 = and i32 %and1b, 858993459
   %or1 = or i32 %shr1, %a
   %or1b = or i32 %or1, %shl1
   %and2 = shl i32 %or1b, 2

 ; RV64I-NEXT:    slli a4, a4, 12
 ; RV64I-NEXT:    addi a4, a4, 819
 ; RV64I-NEXT:    slli a4, a4, 12
 ; RV64I-NEXT:    addi a4, a4, 819
 ; RV64I-NEXT:    slli a4, a4, 12
 ; RV64I-NEXT:    addi a4, a4, 819
 ; RV64I-NEXT:    and a3, a3, a4
 ; RV64I-NEXT:    or a0, a3, a0
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    slli a1, a0, 2
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 2
+; RV64I-NEXT:    or a1, a0, a1
+; RV64I-NEXT:    slli a0, a0, 2
+; RV64I-NEXT:    and a0, a0, a2
+; RV64I-NEXT:    srli a2, a1, 2
 ; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a0, a2, a0
-; RV64I-NEXT:    or a0, a0, a1
+; RV64I-NEXT:    or a1, a2, a1
+; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: gorc2b_i64:
 ; RV64IB:       # %bb.0:
+; RV64IB-NEXT:    srli a1, a0, 2
+; RV64IB-NEXT:    or a1, a1, a0
 ; RV64IB-NEXT:    orc2.n a0, a0
+; RV64IB-NEXT:    slli a1, a1, 2
+; RV64IB-NEXT:    lui a2, 1035469
+; RV64IB-NEXT:    addiw a2, a2, -819
+; RV64IB-NEXT:    slli a2, a2, 12
+; RV64IB-NEXT:    addi a2, a2, -819
+; RV64IB-NEXT:    slli a2, a2, 12
+; RV64IB-NEXT:    addi a2, a2, -819
+; RV64IB-NEXT:    slli a2, a2, 12
+; RV64IB-NEXT:    addi a2, a2, -820
+; RV64IB-NEXT:    and a1, a1, a2
+; RV64IB-NEXT:    srli a2, a0, 2
+; RV64IB-NEXT:    lui a3, 13107
+; RV64IB-NEXT:    addiw a3, a3, 819
+; RV64IB-NEXT:    slli a3, a3, 12
+; RV64IB-NEXT:    addi a3, a3, 819
+; RV64IB-NEXT:    slli a3, a3, 12
+; RV64IB-NEXT:    addi a3, a3, 819
+; RV64IB-NEXT:    slli a3, a3, 12
+; RV64IB-NEXT:    addi a3, a3, 819
+; RV64IB-NEXT:    and a2, a2, a3
+; RV64IB-NEXT:    or a0, a2, a0
+; RV64IB-NEXT:    or a0, a0, a1
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: gorc2b_i64:
 ; RV64IBP:       # %bb.0:
+; RV64IBP-NEXT:    srli a1, a0, 2
+; RV64IBP-NEXT:    or a1, a1, a0
 ; RV64IBP-NEXT:    orc2.n a0, a0
+; RV64IBP-NEXT:    slli a1, a1, 2
+; RV64IBP-NEXT:    lui a2, 1035469
+; RV64IBP-NEXT:    addiw a2, a2, -819
+; RV64IBP-NEXT:    slli a2, a2, 12
+; RV64IBP-NEXT:    addi a2, a2, -819
+; RV64IBP-NEXT:    slli a2, a2, 12
+; RV64IBP-NEXT:    addi a2, a2, -819
+; RV64IBP-NEXT:    slli a2, a2, 12
+; RV64IBP-NEXT:    addi a2, a2, -820
+; RV64IBP-NEXT:    and a1, a1, a2
+; RV64IBP-NEXT:    srli a2, a0, 2
+; RV64IBP-NEXT:    lui a3, 13107
+; RV64IBP-NEXT:    addiw a3, a3, 819
+; RV64IBP-NEXT:    slli a3, a3, 12
+; RV64IBP-NEXT:    addi a3, a3, 819
+; RV64IBP-NEXT:    slli a3, a3, 12
+; RV64IBP-NEXT:    addi a3, a3, 819
+; RV64IBP-NEXT:    slli a3, a3, 12
+; RV64IBP-NEXT:    addi a3, a3, 819
+; RV64IBP-NEXT:    and a2, a2, a3
+; RV64IBP-NEXT:    or a0, a2, a0
+; RV64IBP-NEXT:    or a0, a0, a1
 ; RV64IBP-NEXT:    ret
   %and1 = shl i64 %a, 2
   %shl1 = and i64 %and1, -3689348814741910324
   %and1b = lshr i64 %a, 2
   %shr1 = and i64 %and1b, 3689348814741910323
   %or1 = or i64 %shr1, %a
   %or1b = or i64 %or1, %shl1
   %and2 = shl i64 %or1b, 2

 ; RV64IBP-NEXT:    ret
   %1 = call i64 @llvm.bitreverse.i64(i64 %a)
   ret i64 %1
 }
 
 define i32 @bswap_rotr_i32(i32 %a) {
 ; RV64I-LABEL: bswap_rotr_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srli a1, a0, 24
-; RV64I-NEXT:    lui a2, 4080
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 8
-; RV64I-NEXT:    addi a3, zero, 255
-; RV64I-NEXT:    slli a4, a3, 24
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a1, a2, a1
-; RV64I-NEXT:    srli a2, a0, 40
-; RV64I-NEXT:    lui a4, 16
-; RV64I-NEXT:    addiw a4, a4, -256
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    srli a4, a0, 56
-; RV64I-NEXT:    or a2, a2, a4
-; RV64I-NEXT:    or a1, a1, a2
-; RV64I-NEXT:    slli a2, a0, 8
-; RV64I-NEXT:    slli a4, a3, 32
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    slli a4, a0, 24
-; RV64I-NEXT:    slli a5, a3, 40
-; RV64I-NEXT:    and a4, a4, a5
-; RV64I-NEXT:    or a2, a4, a2
-; RV64I-NEXT:    slli a4, a0, 40
-; RV64I-NEXT:    slli a3, a3, 48
-; RV64I-NEXT:    and a3, a4, a3
+; RV64I-NEXT:    slli a1, a0, 8
+; RV64I-NEXT:    addi a2, zero, 255
+; RV64I-NEXT:    slli a3, a2, 32
+; RV64I-NEXT:    and a1, a1, a3
+; RV64I-NEXT:    slli a3, a0, 24
+; RV64I-NEXT:    slli a4, a2, 40
+; RV64I-NEXT:    and a3, a3, a4
+; RV64I-NEXT:    or a1, a3, a1
+; RV64I-NEXT:    slli a3, a0, 40
+; RV64I-NEXT:    slli a2, a2, 48
+; RV64I-NEXT:    and a2, a3, a2
 ; RV64I-NEXT:    slli a0, a0, 56
-; RV64I-NEXT:    or a0, a0, a3
 ; RV64I-NEXT:    or a0, a0, a2
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    srli a1, a0, 16
-; RV64I-NEXT:    lui a2, 1048560
-; RV64I-NEXT:    and a1, a1, a2
+; RV64I-NEXT:    or a1, a0, a1
+; RV64I-NEXT:    srli a1, a1, 16
 ; RV64I-NEXT:    srli a0, a0, 48
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: bswap_rotr_i32:
 ; RV64IB:       # %bb.0:
 ; RV64IB-NEXT:    greviw a0, a0, 8
 ; RV64IB-NEXT:    ret
 ;
 ; RV64IBP-LABEL: bswap_rotr_i32:
 ; RV64IBP:       # %bb.0:
 ; RV64IBP-NEXT:    greviw a0, a0, 8
 ; RV64IBP-NEXT:    ret
   %1 = call i32 @llvm.bswap.i32(i32 %a)
   %2 = call i32 @llvm.fshr.i32(i32 %1, i32 %1, i32 16)
   ret i32 %2
 }
 
 define i32 @bswap_rotl_i32(i32 %a) {
 ; RV64I-LABEL: bswap_rotl_i32:
 ; RV64I:       # %bb.0:
-; RV64I-NEXT:    srli a1, a0, 24
-; RV64I-NEXT:    lui a2, 4080
-; RV64I-NEXT:    and a1, a1, a2
-; RV64I-NEXT:    srli a2, a0, 8
-; RV64I-NEXT:    addi a3, zero, 255
-; RV64I-NEXT:    slli a4, a3, 24
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    or a1, a2, a1
-; RV64I-NEXT:    srli a2, a0, 40
-; RV64I-NEXT:    lui a4, 16
-; RV64I-NEXT:    addiw a4, a4, -256
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    srli a4, a0, 56
-; RV64I-NEXT:    or a2, a2, a4
-; RV64I-NEXT:    or a1, a1, a2
-; RV64I-NEXT:    slli a2, a0, 8
-; RV64I-NEXT:    slli a4, a3, 32
-; RV64I-NEXT:    and a2, a2, a4
-; RV64I-NEXT:    slli a4, a0, 24
-; RV64I-NEXT:    slli a5, a3, 40
-; RV64I-NEXT:    and a4, a4, a5
-; RV64I-NEXT:    or a2, a4, a2
-; RV64I-NEXT:    slli a4, a0, 40
-; RV64I-NEXT:    slli a3, a3, 48
-; RV64I-NEXT:    and a3, a4, a3
+; RV64I-NEXT:    slli a1, a0, 8
+; RV64I-NEXT:    addi a2, zero, 255
+; RV64I-NEXT:    slli a3, a2, 32
+; RV64I-NEXT:    and a1, a1, a3
+; RV64I-NEXT:    slli a3, a0, 24
+; RV64I-NEXT:    slli a4, a2, 40
+; RV64I-NEXT:    and a3, a3, a4
+; RV64I-NEXT:    or a1, a3, a1
+; RV64I-NEXT:    slli a3, a0, 40
+; RV64I-NEXT:    slli a2, a2, 48
+; RV64I-NEXT:    and a2, a3, a2
 ; RV64I-NEXT:    slli a0, a0, 56
-; RV64I-NEXT:    or a0, a0, a3
 ; RV64I-NEXT:    or a0, a0, a2
-; RV64I-NEXT:    or a0, a0, a1
-; RV64I-NEXT:    srli a1, a0, 16
-; RV64I-NEXT:    lui a2, 1048560
-; RV64I-NEXT:    and a1, a1, a2
+; RV64I-NEXT:    or a1, a0, a1
+; RV64I-NEXT:    srli a1, a1, 16
 ; RV64I-NEXT:    srli a0, a0, 48
 ; RV64I-NEXT:    or a0, a1, a0
 ; RV64I-NEXT:    ret
 ;
 ; RV64IB-LABEL: bswap_rotl_i32:
 ; RV64IB:       # %bb.0:
 ; RV64IB-NEXT:    greviw a0, a0, 8
 ; RV64IB-NEXT:    ret

llvm/test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=s390x-linux-gnu -mcpu=z13 < %s  | FileCheck %s
 
 ; Store a <4 x i31> vector.
 define void @fun0(<4 x i31> %src, <4 x i31>* %p)
 ; CHECK-LABEL: fun0:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vlgvf %r1, %v24, 0
 ; CHECK-NEXT:    vlgvf %r0, %v24, 1
 ; CHECK-NEXT:    sllg %r1, %r1, 29
 ; CHECK-NEXT:    rosbg %r1, %r0, 35, 63, 62
 ; CHECK-NEXT:    nihh %r1, 4095
 ; CHECK-NEXT:    stg %r1, 0(%r2)
 ; CHECK-NEXT:    vlgvf %r1, %v24, 2
-; CHECK-NEXT:    risbgn %r0, %r0, 0, 129, 62
+; CHECK-NEXT:    sllg %r0, %r0, 62
 ; CHECK-NEXT:    rosbg %r0, %r1, 2, 32, 31
 ; CHECK-NEXT:    vlgvf %r1, %v24, 3
 ; CHECK-NEXT:    rosbg %r0, %r1, 33, 63, 0
 ; CHECK-NEXT:    stg %r0, 8(%r2)
 ; CHECK-NEXT:    br %r14
 {
   store <4 x i31> %src, <4 x i31>* %p
   ret void

 
 ; Truncate a <8 x i32> vector to <8 x i31> and store it (test splitting).
 define void @fun2(<8 x i32> %src, <8 x i31>* %p)
 ; CHECK-LABEL: fun2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    stmg %r14, %r15, 112(%r15)
 ; CHECK-NEXT:    .cfi_offset %r14, -48
 ; CHECK-NEXT:    .cfi_offset %r15, -40
-; CHECK-NEXT:    vlgvf %r0, %v26, 3
-; CHECK-NEXT:    vlgvf %r4, %v24, 1
-; CHECK-NEXT:    vlgvf %r3, %v24, 2
-; CHECK-NEXT:    srlk %r1, %r0, 8
+; CHECK-NEXT:    vlgvf %r1, %v26, 3
+; CHECK-NEXT:    vlgvf %r0, %v26, 2
+; CHECK-NEXT:    stc %r1, 30(%r2)
+; CHECK-NEXT:    srlk %r3, %r1, 8
+; CHECK-NEXT:    risbgn %r1, %r1, 33, 167, 0
+; CHECK-NEXT:    vlgvf %r5, %v24, 2
+; CHECK-NEXT:    rosbg %r1, %r0, 2, 32, 31
+; CHECK-NEXT:    sth %r3, 28(%r2)
+; CHECK-NEXT:    srlg %r1, %r1, 24
+; CHECK-NEXT:    vlgvf %r3, %v24, 3
+; CHECK-NEXT:    st %r1, 24(%r2)
+; CHECK-NEXT:    vlgvf %r1, %v26, 0
+; CHECK-NEXT:    risbgn %r14, %r5, 6, 164, 27
+; CHECK-NEXT:    sllg %r4, %r3, 60
+; CHECK-NEXT:    rosbg %r14, %r3, 37, 63, 60
+; CHECK-NEXT:    sllg %r3, %r14, 8
+; CHECK-NEXT:    rosbg %r4, %r1, 4, 34, 29
+; CHECK-NEXT:    rosbg %r3, %r4, 56, 63, 8
+; CHECK-NEXT:    stg %r3, 8(%r2)
+; CHECK-NEXT:    vlgvf %r3, %v24, 1
+; CHECK-NEXT:    sllg %r4, %r3, 58
+; CHECK-NEXT:    rosbg %r4, %r5, 6, 36, 27
 ; CHECK-NEXT:    vlgvf %r5, %v24, 0
-; CHECK-NEXT:    sth %r1, 28(%r2)
-; CHECK-NEXT:    risbgn %r1, %r4, 0, 133, 58
 ; CHECK-NEXT:    sllg %r5, %r5, 25
-; CHECK-NEXT:    stc %r0, 30(%r2)
-; CHECK-NEXT:    rosbg %r1, %r3, 6, 36, 27
-; CHECK-NEXT:    vlgvf %r3, %v24, 3
-; CHECK-NEXT:    rosbg %r5, %r4, 39, 63, 58
-; CHECK-NEXT:    sllg %r4, %r5, 8
-; CHECK-NEXT:    rosbg %r1, %r3, 37, 63, 60
-; CHECK-NEXT:    vlgvf %r5, %v26, 1
-; CHECK-NEXT:    rosbg %r4, %r1, 56, 63, 8
-; CHECK-NEXT:    stg %r4, 0(%r2)
-; CHECK-NEXT:    vlgvf %r4, %v26, 2
-; CHECK-NEXT:    risbgn %r14, %r5, 0, 129, 62
-; CHECK-NEXT:    risbgn %r3, %r3, 0, 131, 60
-; CHECK-NEXT:    rosbg %r14, %r4, 2, 32, 31
-; CHECK-NEXT:    rosbg %r14, %r0, 33, 63, 0
-; CHECK-NEXT:    srlg %r0, %r14, 24
-; CHECK-NEXT:    st %r0, 24(%r2)
-; CHECK-NEXT:    vlgvf %r0, %v26, 0
-; CHECK-NEXT:    rosbg %r3, %r0, 4, 34, 29
-; CHECK-NEXT:    sllg %r0, %r1, 8
-; CHECK-NEXT:    rosbg %r3, %r5, 35, 63, 62
-; CHECK-NEXT:    rosbg %r0, %r3, 56, 63, 8
-; CHECK-NEXT:    stg %r0, 8(%r2)
-; CHECK-NEXT:    sllg %r0, %r3, 8
-; CHECK-NEXT:    rosbg %r0, %r14, 56, 63, 8
+; CHECK-NEXT:    rosbg %r5, %r3, 39, 63, 58
+; CHECK-NEXT:    sllg %r3, %r5, 8
+; CHECK-NEXT:    rosbg %r3, %r4, 56, 63, 8
+; CHECK-NEXT:    stg %r3, 0(%r2)
+; CHECK-NEXT:    vlgvf %r3, %v26, 1
+; CHECK-NEXT:    sllg %r4, %r3, 62
+; CHECK-NEXT:    rosbg %r4, %r0, 2, 32, 31
+; CHECK-NEXT:    risbgn %r0, %r1, 4, 162, 29
+; CHECK-NEXT:    rosbg %r0, %r3, 35, 63, 62
+; CHECK-NEXT:    sllg %r0, %r0, 8
+; CHECK-NEXT:    rosbg %r0, %r4, 56, 63, 8
 ; CHECK-NEXT:    stg %r0, 16(%r2)
 ; CHECK-NEXT:    lmg %r14, %r15, 112(%r15)
 ; CHECK-NEXT:    br %r14
 {
   %tmp = trunc <8 x i32> %src to <8 x i31>
   store <8 x i31> %tmp, <8 x i31>* %p
   ret void
 }
 
 ; Load and store a <3 x i31> vector (test widening).
 define void @fun3(<3 x i31>* %src, <3 x i31>* %p)
 ; CHECK-LABEL: fun3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    l %r0, 8(%r2)
+; CHECK-NEXT:    llgf %r0, 8(%r2)
 ; CHECK-NEXT:    lg %r1, 0(%r2)
 ; CHECK-NEXT:    sllg %r2, %r1, 32
 ; CHECK-NEXT:    lr %r2, %r0
-; CHECK-NEXT:    srlg %r0, %r2, 62
-; CHECK-NEXT:    st %r2, 8(%r3)
-; CHECK-NEXT:    rosbg %r0, %r1, 33, 61, 34
-; CHECK-NEXT:    sllg %r1, %r0, 62
-; CHECK-NEXT:    rosbg %r1, %r2, 2, 32, 0
-; CHECK-NEXT:    srlg %r1, %r1, 32
-; CHECK-NEXT:    sllg %r0, %r0, 30
-; CHECK-NEXT:    lr %r0, %r1
-; CHECK-NEXT:    nihh %r0, 8191
-; CHECK-NEXT:    stg %r0, 0(%r3)
+; CHECK-NEXT:    risbgn %r2, %r2, 2, 160, 0
+; CHECK-NEXT:    lgr %r4, %r2
+; CHECK-NEXT:    rosbg %r2, %r1, 0, 1, 32
+; CHECK-NEXT:    rosbg %r4, %r0, 33, 63, 0
+; CHECK-NEXT:    srlg %r0, %r2, 32
+; CHECK-NEXT:    lr %r1, %r0
+; CHECK-NEXT:    nihh %r1, 8191
+; CHECK-NEXT:    st %r4, 8(%r3)
+; CHECK-NEXT:    stg %r1, 0(%r3)
 ; CHECK-NEXT:    br %r14
 {
   %tmp = load <3 x i31>, <3 x i31>* %src
   store <3 x i31> %tmp, <3 x i31>* %p
   ret void
 }

llvm/test/CodeGen/Thumb2/thumb2-uxtb.ll

; CHECK-NO-DSP-NEXT:    bx lr
   %tmp1 = lshr i32 %x, 24   ; <i32> [#uses=1]
   %tmp4 = shl i32 %x, 8   ; <i32> [#uses=1]
   %tmp5 = and i32 %tmp4, 16711680   ; <i32> [#uses=1]
   %tmp6 = or i32 %tmp5, %tmp1   ; <i32> [#uses=1]
   ret i32 %tmp6
 }
 
 define i32 @test10(i32 %p0) {
-; CHECK-DSP-LABEL: test10:
-; CHECK-DSP:       @ %bb.0:
-; CHECK-DSP-NEXT:    mov.w r1, #16253176
-; CHECK-DSP-NEXT:    and.w r0, r1, r0, lsr #7
-; CHECK-DSP-NEXT:    lsrs r1, r0, #5
-; CHECK-DSP-NEXT:    uxtb16 r1, r1
-; CHECK-DSP-NEXT:    add r0, r1
-; CHECK-DSP-NEXT:    bx lr
-;
-; CHECK-NO-DSP-LABEL: test10:
-; CHECK-NO-DSP:       @ %bb.0:
-; CHECK-NO-DSP-NEXT:    mov.w r1, #16253176
-; CHECK-NO-DSP-NEXT:    and.w r0, r1, r0, lsr #7
-; CHECK-NO-DSP-NEXT:    mov.w r1, #458759
-; CHECK-NO-DSP-NEXT:    and.w r1, r1, r0, lsr #5
-; CHECK-NO-DSP-NEXT:    add r0, r1
-; CHECK-NO-DSP-NEXT:    bx lr
-
+; CHECK-LABEL: test10:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    mov.w r1, #16253176
+; CHECK-NEXT:    mov.w r2, #458759
+; CHECK-NEXT:    and.w r1, r1, r0, lsr #7
+; CHECK-NEXT:    and.w r0, r2, r0, lsr #12
+; CHECK-NEXT:    add r0, r1
+; CHECK-NEXT:    bx lr
   %tmp1 = lshr i32 %p0, 7   ; <i32> [#uses=1]
   %tmp2 = and i32 %tmp1, 16253176   ; <i32> [#uses=2]
   %tmp4 = lshr i32 %tmp2, 5   ; <i32> [#uses=1]
   %tmp5 = and i32 %tmp4, 458759   ; <i32> [#uses=1]
   %tmp7 = or i32 %tmp5, %tmp2   ; <i32> [#uses=1]
   ret i32 %tmp7
 }

llvm/test/CodeGen/X86/ctpop-combine.ll

 ; POPCOUNT:       # %bb.0:
 ; POPCOUNT-NEXT:    andl $127, %edi
 ; POPCOUNT-NEXT:    popcntl %edi, %eax
 ; POPCOUNT-NEXT:    # kill: def $al killed $al killed $eax
 ; POPCOUNT-NEXT:    retq
 ;
 ; NO-POPCOUNT-LABEL: test4:
 ; NO-POPCOUNT:       # %bb.0:
-; NO-POPCOUNT-NEXT:    # kill: def $edi killed $edi def $rdi
-; NO-POPCOUNT-NEXT:    andb $127, %dil
-; NO-POPCOUNT-NEXT:    movl %edi, %eax
-; NO-POPCOUNT-NEXT:    shrb %al
-; NO-POPCOUNT-NEXT:    andb $21, %al
-; NO-POPCOUNT-NEXT:    subb %al, %dil
 ; NO-POPCOUNT-NEXT:    movl %edi, %eax
+; NO-POPCOUNT-NEXT:    andb $127, %al
+; NO-POPCOUNT-NEXT:    shrb %dil
+; NO-POPCOUNT-NEXT:    andb $21, %dil
+; NO-POPCOUNT-NEXT:    subb %dil, %al
+; NO-POPCOUNT-NEXT:    movl %eax, %ecx
+; NO-POPCOUNT-NEXT:    andb $51, %cl
+; NO-POPCOUNT-NEXT:    shrb $2, %al
 ; NO-POPCOUNT-NEXT:    andb $51, %al
-; NO-POPCOUNT-NEXT:    shrb $2, %dil
-; NO-POPCOUNT-NEXT:    andb $51, %dil
-; NO-POPCOUNT-NEXT:    addb %al, %dil
-; NO-POPCOUNT-NEXT:    movl %edi, %eax
-; NO-POPCOUNT-NEXT:    shrb $4, %al
-; NO-POPCOUNT-NEXT:    addl %edi, %eax
+; NO-POPCOUNT-NEXT:    addb %cl, %al
+; NO-POPCOUNT-NEXT:    movl %eax, %ecx
+; NO-POPCOUNT-NEXT:    shrb $4, %cl
+; NO-POPCOUNT-NEXT:    addl %ecx, %eax
 ; NO-POPCOUNT-NEXT:    andb $15, %al
 ; NO-POPCOUNT-NEXT:    # kill: def $al killed $al killed $eax
 ; NO-POPCOUNT-NEXT:    retq
   %x2 = and i8 %x, 127
   %count = tail call i8 @llvm.ctpop.i8(i8 %x2)
   %and = and i8 %count, 7
   ret i8 %and
 }

llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll

 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzwl (%ecx), %edx
 ; X86-NEXT:    movzbl 2(%ecx), %eax
 ; X86-NEXT:    movb %al, 2(%ecx)
 ; X86-NEXT:    shll $16, %eax
 ; X86-NEXT:    orl %edx, %eax
 ; X86-NEXT:    orl $384, %eax # imm = 0x180
-; X86-NEXT:    andl $16777088, %eax # imm = 0xFFFF80
+; X86-NEXT:    andl $-128, %eax
 ; X86-NEXT:    movw %ax, (%ecx)
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i24_and_or:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
 ; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    orl $384, %ecx # imm = 0x180
-; X64-NEXT:    andl $16777088, %ecx # imm = 0xFFFF80
+; X64-NEXT:    andl $-128, %ecx
 ; X64-NEXT:    movw %cx, (%rdi)
 ; X64-NEXT:    retq
   %b = load i24, i24* %a, align 1
   %c = and i24 %b, -128
   %d = or i24 %c, 384
   store i24 %d, i24* %a, align 1
   ret void
 }

 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 6(%rdi), %ecx
 ; X64-NEXT:    movb %cl, 6(%rdi)
 ; X64-NEXT:    # kill: def $ecx killed $ecx killed $rcx def $rcx
 ; X64-NEXT:    shll $16, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    shlq $32, %rcx
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    orq %rcx, %rax
-; X64-NEXT:    orq $384, %rax # imm = 0x180
-; X64-NEXT:    movl %eax, (%rdi)
-; X64-NEXT:    shrq $32, %rax
+; X64-NEXT:    movl (%rdi), %edx
+; X64-NEXT:    orq %rcx, %rdx
+; X64-NEXT:    orq $384, %rdx # imm = 0x180
 ; X64-NEXT:    movw %ax, 4(%rdi)
+; X64-NEXT:    movl %edx, (%rdi)
 ; X64-NEXT:    retq
   %aa = load i56, i56* %a, align 1
   %b = or i56 %aa, 384
   store i56 %b, i56* %a, align 1
   ret void
 }
 
 define void @i56_and_or(i56* %a) {

 ; X64-NEXT:    movl %esi, %eax
 ; X64-NEXT:    movzwl 4(%rdi), %ecx
 ; X64-NEXT:    movzbl 6(%rdi), %edx
 ; X64-NEXT:    movb %dl, 6(%rdi)
 ; X64-NEXT:    # kill: def $edx killed $edx killed $rdx def $rdx
 ; X64-NEXT:    shll $16, %edx
 ; X64-NEXT:    orl %ecx, %edx
 ; X64-NEXT:    shlq $32, %rdx
-; X64-NEXT:    movl (%rdi), %ecx
-; X64-NEXT:    orq %rdx, %rcx
+; X64-NEXT:    movl (%rdi), %esi
+; X64-NEXT:    orq %rdx, %rsi
 ; X64-NEXT:    shlq $13, %rax
 ; X64-NEXT:    movabsq $72057594037919743, %rdx # imm = 0xFFFFFFFFFFDFFF
-; X64-NEXT:    andq %rcx, %rdx
+; X64-NEXT:    andq %rsi, %rdx
 ; X64-NEXT:    orq %rax, %rdx
+; X64-NEXT:    movw %cx, 4(%rdi)
 ; X64-NEXT:    movl %edx, (%rdi)
-; X64-NEXT:    shrq $32, %rdx
-; X64-NEXT:    movw %dx, 4(%rdi)
 ; X64-NEXT:    retq
   %extbit = zext i1 %bit to i56
   %b = load i56, i56* %a, align 1
   %extbit.shl = shl nuw nsw i56 %extbit, 13
   %c = and i56 %b, -8193
   %d = or i56 %c, %extbit.shl
   store i56 %d, i56* %a, align 1
   ret void
 }
 

llvm/test/CodeGen/X86/ins_subreg_coalesce-1.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -mattr=-bmi | FileCheck %s
 
 define fastcc i32 @t() nounwind  {
 ; CHECK-LABEL: t:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movzwl 0, %eax
-; CHECK-NEXT:    orl $2, %eax
-; CHECK-NEXT:    movw %ax, 0
+; CHECK-NEXT:    movl %eax, %ecx
+; CHECK-NEXT:    orl $2, %ecx
+; CHECK-NEXT:    movw %cx, 0
 ; CHECK-NEXT:    shrl $3, %eax
 ; CHECK-NEXT:    andl $1, %eax
 ; CHECK-NEXT:    retl
 entry:
   br i1 false, label %UnifiedReturnBlock, label %bb4
 bb4:    ; preds = %entry
   br i1 false, label %bb17, label %bb22
 bb17:   ; preds = %bb4

llvm/test/CodeGen/X86/load-local-v3i129.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 
 define void @_start() {
 ; CHECK-LABEL: _start:
 ; CHECK:       # %bb.0: # %Entry
 ; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    movq -{{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq -{{[0-9]+}}(%rsp), %rcx
-; CHECK-NEXT:    shrdq $2, %rcx, %rax
+; CHECK-NEXT:    movq %rcx, %rdx
+; CHECK-NEXT:    shlq $62, %rdx
 ; CHECK-NEXT:    shrq $2, %rcx
-; CHECK-NEXT:    leaq 1(,%rax,4), %rdx
-; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrdq $62, %rcx, %rax
+; CHECK-NEXT:    shldq $2, %rdx, %rcx
+; CHECK-NEXT:    andq $-4, %rax
+; CHECK-NEXT:    orq $1, %rax
 ; CHECK-NEXT:    movq %rax, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    orq $-2, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movq $-1, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    popq %rax
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
 Entry:
   %y = alloca <3 x i129>, align 4
   %L = load <3 x i129>, <3 x i129>* %y
   %I1 = insertelement <3 x i129> %L, i129 340282366920938463463374607431768211455, i32 1
   store <3 x i129> %I1, <3 x i129>* %y
   ret void
 }

llvm/test/CodeGen/X86/load-local-v4i5.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 @0 = internal unnamed_addr constant [4 x i5] [i5 2, i5 0, i5 2, i5 -1], align 1
 
 ; Function Attrs: nobuiltin nounwind
 define void @_start() {
 ; CHECK-LABEL: _start:
 ; CHECK:       # %bb.0: # %Entry
 ; CHECK-NEXT:    movl {{.*}}(%rip), %eax
 ; CHECK-NEXT:    movl %eax, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb -{{[0-9]+}}(%rsp), %cl
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %edx
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %esi
+; CHECK-NEXT:    movzbl %cl, %edi
+; CHECK-NEXT:    shrb %cl
+; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    andl $31, %eax
 ; CHECK-NEXT:    andl $31, %esi
 ; CHECK-NEXT:    shll $5, %esi
 ; CHECK-NEXT:    orl %eax, %esi
 ; CHECK-NEXT:    andl $31, %edx
 ; CHECK-NEXT:    shll $10, %edx
 ; CHECK-NEXT:    orl %esi, %edx
-; CHECK-NEXT:    movzbl %cl, %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    shll $15, %ecx
-; CHECK-NEXT:    orl %edx, %ecx
-; CHECK-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrl $16, %ecx
-; CHECK-NEXT:    andl $15, %ecx
-; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movb %al, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    cmpb $31, %al
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shll $15, %eax
+; CHECK-NEXT:    orl %edx, %eax
+; CHECK-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    cmpb $31, %dil
 ; CHECK-NEXT:    je .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %Then
 ; CHECK-NEXT:    int3
 ; CHECK-NEXT:  .LBB0_2: # %EndIf
 ; CHECK-NEXT:    retq
 Entry:
   %x = alloca [4 x i5], align 1
   %y = alloca <4 x i5>, align 4

llvm/test/CodeGen/X86/masked_compressstore.ll

 ; AVX512F-LABEL: compressstore_v16f64_v16i1:
 ; AVX512F:       ## %bb.0:
 ; AVX512F-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; AVX512F-NEXT:    vpslld $31, %zmm2, %zmm2
 ; AVX512F-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; AVX512F-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512F-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512F-NEXT:    kmovw %k1, %eax
-; AVX512F-NEXT:    movzbl %al, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    shrl %ecx
-; AVX512F-NEXT:    andl $-43, %ecx
-; AVX512F-NEXT:    subl %ecx, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
-; AVX512F-NEXT:    shrl $2, %eax
+; AVX512F-NEXT:    movzbl %al, %ecx
+; AVX512F-NEXT:    shrl %eax
+; AVX512F-NEXT:    andl $85, %eax
+; AVX512F-NEXT:    subl %eax, %ecx
+; AVX512F-NEXT:    movl %ecx, %eax
 ; AVX512F-NEXT:    andl $858993459, %eax ## imm = 0x33333333
-; AVX512F-NEXT:    addl %ecx, %eax
-; AVX512F-NEXT:    movl %eax, %ecx
-; AVX512F-NEXT:    shrl $4, %ecx
+; AVX512F-NEXT:    shrl $2, %ecx
+; AVX512F-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512F-NEXT:    addl %eax, %ecx
-; AVX512F-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
-; AVX512F-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
+; AVX512F-NEXT:    movl %ecx, %eax
+; AVX512F-NEXT:    shrl $4, %eax
+; AVX512F-NEXT:    addl %ecx, %eax
+; AVX512F-NEXT:    andl $252645135, %eax ## imm = 0xF0F0F0F
+; AVX512F-NEXT:    imull $16843009, %eax, %eax ## imm = 0x1010101
 ; AVX512F-NEXT:    shrl $24, %eax
 ; AVX512F-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512VLDQ-LABEL: compressstore_v16f64_v16i1:
 ; AVX512VLDQ:       ## %bb.0:
 ; AVX512VLDQ-NEXT:    vpmovsxbd %xmm2, %zmm2

 ;
 ; AVX512VLBW-LABEL: compressstore_v16f64_v16i1:
 ; AVX512VLBW:       ## %bb.0:
 ; AVX512VLBW-NEXT:    vpsllw $7, %xmm2, %xmm2
 ; AVX512VLBW-NEXT:    vpmovb2m %xmm2, %k1
 ; AVX512VLBW-NEXT:    kshiftrw $8, %k1, %k2
 ; AVX512VLBW-NEXT:    vcompresspd %zmm0, (%rdi) {%k1}
 ; AVX512VLBW-NEXT:    kmovd %k1, %eax
-; AVX512VLBW-NEXT:    movzbl %al, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    shrl %ecx
-; AVX512VLBW-NEXT:    andl $-43, %ecx
-; AVX512VLBW-NEXT:    subl %ecx, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
-; AVX512VLBW-NEXT:    shrl $2, %eax
+; AVX512VLBW-NEXT:    movzbl %al, %ecx
+; AVX512VLBW-NEXT:    shrl %eax
+; AVX512VLBW-NEXT:    andl $85, %eax
+; AVX512VLBW-NEXT:    subl %eax, %ecx
+; AVX512VLBW-NEXT:    movl %ecx, %eax
 ; AVX512VLBW-NEXT:    andl $858993459, %eax ## imm = 0x33333333
-; AVX512VLBW-NEXT:    addl %ecx, %eax
-; AVX512VLBW-NEXT:    movl %eax, %ecx
-; AVX512VLBW-NEXT:    shrl $4, %ecx
+; AVX512VLBW-NEXT:    shrl $2, %ecx
+; AVX512VLBW-NEXT:    andl $858993459, %ecx ## imm = 0x33333333
 ; AVX512VLBW-NEXT:    addl %eax, %ecx
-; AVX512VLBW-NEXT:    andl $252645135, %ecx ## imm = 0xF0F0F0F
-; AVX512VLBW-NEXT:    imull $16843009, %ecx, %eax ## imm = 0x1010101
+; AVX512VLBW-NEXT:    movl %ecx, %eax
+; AVX512VLBW-NEXT:    shrl $4, %eax
+; AVX512VLBW-NEXT:    addl %ecx, %eax
+; AVX512VLBW-NEXT:    andl $252645135, %eax ## imm = 0xF0F0F0F
+; AVX512VLBW-NEXT:    imull $16843009, %eax, %eax ## imm = 0x1010101
 ; AVX512VLBW-NEXT:    shrl $24, %eax
 ; AVX512VLBW-NEXT:    vcompresspd %zmm1, (%rdi,%rax,8) {%k2}
 ; AVX512VLBW-NEXT:    vzeroupper
 ; AVX512VLBW-NEXT:    retq
   call void @llvm.masked.compressstore.v16f64(<16 x double> %V, double* %base, <16 x i1> %mask)
   ret void
 }
 

llvm/test/CodeGen/X86/mul128.ll

 }
 
 @aaa = external dso_local global i128
 @bbb = external dso_local global i128
 
 define void @PR13897() nounwind {
 ; X64-LABEL: PR13897:
 ; X64:       # %bb.0: # %"0x0"
-; X64-NEXT:    movl {{.*}}(%rip), %ecx
+; X64-NEXT:    movq {{.*}}(%rip), %rsi
+; X64-NEXT:    movl %esi, %ecx
+; X64-NEXT:    shlq $32, %rsi
 ; X64-NEXT:    movabsq $4294967297, %rdx # imm = 0x100000001
 ; X64-NEXT:    movq %rcx, %rax
 ; X64-NEXT:    mulq %rdx
 ; X64-NEXT:    addq %rcx, %rdx
-; X64-NEXT:    shlq $32, %rcx
-; X64-NEXT:    addq %rcx, %rdx
+; X64-NEXT:    addq %rsi, %rdx
 ; X64-NEXT:    movq %rax, {{.*}}(%rip)
 ; X64-NEXT:    movq %rdx, aaa+{{.*}}(%rip)
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: PR13897:
 ; X86:       # %bb.0: # %"0x0"
 ; X86-NEXT:    movl bbb, %eax
 ; X86-NEXT:    movl %eax, aaa+12

llvm/test/CodeGen/X86/shift-mask.ll

; X64-BMI2-NEXT:    retq
   %2 = lshr i64 %1, 3
   ret i64 %2
 }
 
 define i64 @test_i64_lshr_lshr_1(i64 %a0) {
 ; X86-LABEL: test_i64_lshr_lshr_1:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    shldl $3, %eax, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    leal (,%ecx,8), %edx
+; X86-NEXT:    shldl $3, %eax, %ecx
 ; X86-NEXT:    shll $3, %eax
-; X86-NEXT:    shrdl $5, %edx, %eax
+; X86-NEXT:    shrdl $5, %ecx, %eax
 ; X86-NEXT:    shrl $5, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_i64_lshr_lshr_1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    leaq (,%rdi,8), %rax
 ; X64-NEXT:    shrq $5, %rax
 ; X64-NEXT:    retq

llvm/test/CodeGen/X86/udiv_fix_sat.ll

 ; X64-NEXT:    shrl %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $rax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: func7:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movzwl %cx, %ecx
-; X86-NEXT:    addl %ecx, %ecx
 ; X86-NEXT:    movl %ecx, %edx
-; X86-NEXT:    shrl $16, %edx
-; X86-NEXT:    shll $16, %ecx
+; X86-NEXT:    shll $17, %edx
+; X86-NEXT:    shrl $15, %ecx
+; X86-NEXT:    andl $1, %ecx
 ; X86-NEXT:    pushl $0
 ; X86-NEXT:    pushl %eax
-; X86-NEXT:    pushl %edx
 ; X86-NEXT:    pushl %ecx
+; X86-NEXT:    pushl %edx
 ; X86-NEXT:    calll __udivdi3
 ; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:    cmpl $131071, %eax # imm = 0x1FFFF
 ; X86-NEXT:    movl $131071, %ecx # imm = 0x1FFFF
 ; X86-NEXT:    cmovael %ecx, %eax
 ; X86-NEXT:    testl %edx, %edx
 ; X86-NEXT:    cmovnel %ecx, %eax
 ; X86-NEXT:    shrl %eax

llvm/test/CodeGen/X86/vector-mulfix-legalize.ll

 ; CHECK-NEXT:    sarl $16, %ecx
 ; CHECK-NEXT:    cmpl $16383, %ecx # imm = 0x3FFF
 ; CHECK-NEXT:    movl $32767, %r8d # imm = 0x7FFF
 ; CHECK-NEXT:    cmovgl %r8d, %edx
 ; CHECK-NEXT:    cmpl $-16384, %ecx # imm = 0xC000
 ; CHECK-NEXT:    movl $32768, %ecx # imm = 0x8000
 ; CHECK-NEXT:    cmovll %ecx, %edx
 ; CHECK-NEXT:    pextrw $1, %xmm0, %esi
-; CHECK-NEXT:    movswl %si, %edi
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    shrl $16, %eax
-; CHECK-NEXT:    leal (%rdi,%rdi), %esi
-; CHECK-NEXT:    shrdw $15, %ax, %si
-; CHECK-NEXT:    sarl $15, %edi
-; CHECK-NEXT:    cmpl $16383, %edi # imm = 0x3FFF
+; CHECK-NEXT:    leal (%rsi,%rsi), %edi
+; CHECK-NEXT:    movswl %si, %eax
+; CHECK-NEXT:    movl %eax, %esi
+; CHECK-NEXT:    shrl $16, %esi
+; CHECK-NEXT:    shldw $1, %di, %si
+; CHECK-NEXT:    sarl $15, %eax
+; CHECK-NEXT:    cmpl $16383, %eax # imm = 0x3FFF
 ; CHECK-NEXT:    cmovgl %r8d, %esi
-; CHECK-NEXT:    cmpl $-16384, %edi # imm = 0xC000
+; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    movd %xmm0, %eax
 ; CHECK-NEXT:    cwtl
 ; CHECK-NEXT:    movl %eax, %edi
 ; CHECK-NEXT:    shrl $16, %edi
 ; CHECK-NEXT:    shldw $1, %ax, %di
 ; CHECK-NEXT:    sarl $16, %eax
 ; CHECK-NEXT:    cmpl $16383, %eax # imm = 0x3FFF
 ; CHECK-NEXT:    cmovgl %r8d, %edi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %edi
 ; CHECK-NEXT:    movzwl %di, %eax
 ; CHECK-NEXT:    movd %eax, %xmm1
 ; CHECK-NEXT:    pinsrw $1, %esi, %xmm1
 ; CHECK-NEXT:    pinsrw $2, %edx, %xmm1
 ; CHECK-NEXT:    pextrw $3, %xmm0, %eax
+; CHECK-NEXT:    leal (,%rax,4), %edx
 ; CHECK-NEXT:    cwtl
-; CHECK-NEXT:    movl %eax, %edx
-; CHECK-NEXT:    shrl $14, %edx
-; CHECK-NEXT:    leal (,%rax,4), %esi
-; CHECK-NEXT:    shrdw $15, %dx, %si
+; CHECK-NEXT:    movl %eax, %esi
+; CHECK-NEXT:    shrl $14, %esi
+; CHECK-NEXT:    shldw $1, %dx, %si
 ; CHECK-NEXT:    sarl $14, %eax
 ; CHECK-NEXT:    cmpl $16383, %eax # imm = 0x3FFF
 ; CHECK-NEXT:    cmovgl %r8d, %esi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    pinsrw $3, %esi, %xmm1
 ; CHECK-NEXT:    movdqa %xmm1, %xmm0
 ; CHECK-NEXT:    retq