[DAGCombine] Prune unnused nodes.

[niravd]

Summary

Nodes that have no uses are eventually pruned when they are selected
from the worklist. Record nodes newly added to the worklist or DAG and
perform pruning after every combine attempt.

[niravd]

Rebase to tip and add FIXME for regression.

[RKSimon]

llvm/test/CodeGen/PowerPC/pr39478.ll

10

This looks like an over-reduced test?

[niravd]

llvm/test/CodeGen/PowerPC/pr39478.ll

10

With the improved pruning, we are able to narrow the load to 1 byte which as both the load and store are to undef cancel out the both ops.

[RKSimon]

llvm/test/CodeGen/PowerPC/pr39478.ll

10

I'm hitting issues with this test in D58017 as well, but I'd prefer to see it still do what its supposed to (even if its driving me nuts at the moment....)

IMO we need to replace the undef pointers so it still tests https://bugs.llvm.org/show_bug.cgi?id=39478

[nemanjai]

llvm/test/CodeGen/PowerPC/pr39478.ll

10

I am sorry, I produced this test case by reducing the original input with bugpoint. I can commit an update for it so it doesn't use an undefined pointer as NFC if you'd like so that it stops causing you problems.

[jyknight]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

140

Couldn't this just be represented by an index of the last position in the worklist which has been examined for new-node-prunning, instead of a separate list?

Set it to Worklist.size() after popping entries in getNextWorklistEntry(), and iterate from it to the end of worklist in clearNewUnusedWorklistEntries.

160

Unused?

205

Unused?

llvm/test/CodeGen/X86/constant-combines.ll

12–14

The comment here says this test is now useless.

llvm/test/CodeGen/X86/pr28504.ll

9

Given this comment, I wonder if this test case is still valid?

llvm/test/CodeGen/X86/pr33844.ll

13

This looks like it might not be testing what it intended to test anymore, too?

[niravd]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

140

Unfortunately, that won't catch all nodes. Nodes added to the worklist may already be in the list and therefore won't be put at the end of the list, but we still need to prune those.

160

Used at the start of getNextWorklistEntry. I've have it as a helper, because in principle we need to clear dangling nodes after any failed combine as we may have a created a dangling node that will needlessly impede subsequent combines on the same node and we should call it in those cases.

I've added some explanatory comments.

205

Yes. Pruned.

llvm/test/CodeGen/PowerPC/pr39478.ll

10

That would be great.

[niravd]

Address some of James' comments.

[jyknight]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

140

For some reason I had gotten it into my head that this was only important for newly-created nodes, not for all newly-modified nodes, but I see now that's not what this is doing. OK.

160

I meant the line this comment was attached to -- the local variable "Nodes" looks unused here.

[niravd]

Address lingering test comments.

[niravd]

Consider inserted nodes for pruning.

[jyknight]

It'd be nice if we knew why adding new nodes to the worklist seemed to cause an issue, but the reduced form as here seems okay as far as it goes.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

688–689

Might be infinite loop, not exponential time?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

class DAGCombiner {
     SmallVector<SDNode *, 64> Worklist;
 
     /// Mapping from an SDNode to its position on the worklist.
     ///
     /// This is used to find and remove nodes from the worklist (by nulling
     /// them) when they are deleted from the underlying DAG. It relies on
     /// stable indices of nodes within the worklist.
     DenseMap<SDNode *, unsigned> WorklistMap;
+    SmallSetVector<SDNode *, 32> PruningList;
 
     /// Set of nodes which have been combined (at least once).
     ///
     /// This is used to allow us to reliably add any operands of a DAG node
     /// which have not yet been combined to the worklist.
     SmallPtrSet<SDNode *, 32> CombinedNodes;
 
     // AA - Used for DAG load/store alias analysis.
     AliasAnalysis *AA;
 
     /// When an instruction is simplified, add all users of the instruction to
     /// the work lists because they might get more simplified now.
     void AddUsersToWorklist(SDNode *N) {
       for (SDNode *Node : N->uses())
         AddToWorklist(Node);
     }
 
+    void clearNewUnusedWorklistEntries() {
+      // Check any nodes added to the worklist to see if they are prunable.
+      SmallSetVector<SDNode *, 16> Nodes;
+
+      while (!PruningList.empty()) {
+        auto *N = PruningList.pop_back_val();
+        if (N->use_empty())
+          recursivelyDeleteUnusedNodes(N);
+      }
+    }
+
+    SDNode *getNextWorklistEntry() {
+      // Before we do any work, remove nodes that are not in use.
+      clearNewUnusedWorklistEntries();
+      SDNode *N = nullptr;
+      // The Worklist holds the SDNodes in order, but it may contain null
+      // entries.
+      while (!N && !Worklist.empty()) {
+        N = Worklist.pop_back_val();
+      }
+
+      if (N) {
+        bool GoodWorklistEntry = WorklistMap.erase(N);
+        (void)GoodWorklistEntry;
+        assert(GoodWorklistEntry &&
+               "Found a worklist entry without a corresponding map entry!");
+      }
+      return N;
+    }
+
     /// Call the node-specific routine that folds each particular type of node.
     SDValue visit(SDNode *N);
 
   public:
     DAGCombiner(SelectionDAG &D, AliasAnalysis *AA, CodeGenOpt::Level OL)
         : DAG(D), TLI(D.getTargetLoweringInfo()), Level(BeforeLegalizeTypes),
-          OptLevel(OL), AA(AA) {
+          OptLevel(OL), PruningList(), AA(AA) {
       ForCodeSize = DAG.getMachineFunction().getFunction().optForSize();
 
       MaximumLegalStoreInBits = 0;
       for (MVT VT : MVT::all_valuetypes())
         if (EVT(VT).isSimple() && VT != MVT::Other &&
             TLI.isTypeLegal(EVT(VT)) &&
             VT.getSizeInBits() >= MaximumLegalStoreInBits)
           MaximumLegalStoreInBits = VT.getSizeInBits();
     }
 
+    void AddNewNode(SDNode *N) { PruningList.insert(N); }
+
     /// Add to the worklist making sure its instance is at the back (next to be
     /// processed.)
     void AddToWorklist(SDNode *N) {
       assert(N->getOpcode() != ISD::DELETED_NODE &&
              "Deleted Node added to Worklist");
 
       // Skip handle nodes as they can't usefully be combined and confuse the
       // zero-use deletion strategy.
       if (N->getOpcode() == ISD::HANDLENODE)
         return;
 
+      // Mark this for potential pruning later.
+      PruningList.insert(N);
+
       if (WorklistMap.insert(std::make_pair(N, Worklist.size())).second)
         Worklist.push_back(N);
     }
 
     /// Remove all instances of N from the worklist.
     void removeFromWorklist(SDNode *N) {
       CombinedNodes.erase(N);
+      PruningList.remove(N);
 
       auto It = WorklistMap.find(N);
       if (It == WorklistMap.end())
         return; // Not in the worklist.
 
       // Null out the entry rather than erasing it to avoid a linear operation.
       Worklist[It->second] = nullptr;
       WorklistMap.erase(It);

 
 class WorklistInserter : public SelectionDAG::DAGUpdateListener {
   DAGCombiner &DC;
 
 public:
   explicit WorklistInserter(DAGCombiner &dc)
       : SelectionDAG::DAGUpdateListener(dc.getDAG()), DC(dc) {}
 
   void NodeInserted(SDNode *N) override { DC.AddToWorklist(N); }
 };
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 //  TargetLowering::DAGCombinerInfo implementation
 //===----------------------------------------------------------------------===//
 
 void TargetLowering::DAGCombinerInfo::AddToWorklist(SDNode *N) {
void DAGCombiner::Run(CombineLevel AtLevel) {
   for (SDNode &Node : DAG.allnodes())
     AddToWorklist(&Node);
 
   // Create a dummy node (which is not added to allnodes), that adds a reference
   // to the root node, preventing it from being deleted, and tracking any
   // changes of the root.
   HandleSDNode Dummy(DAG.getRoot());
 
-  // While the worklist isn't empty, find a node and try to combine it.
-  while (!WorklistMap.empty()) {
-    SDNode *N;
-    // The Worklist holds the SDNodes in order, but it may contain null entries.
-    do {
-      N = Worklist.pop_back_val();
-    } while (!N);
-
-    bool GoodWorklistEntry = WorklistMap.erase(N);
-    (void)GoodWorklistEntry;
-    assert(GoodWorklistEntry &&
-           "Found a worklist entry without a corresponding map entry!");
-
+  // While we have a valid worklist entry node, try to combine it.
+  while (SDNode *N = getNextWorklistEntry()) {
     // If N has no uses, it is dead.  Make sure to revisit all N's operands once
     // N is deleted from the DAG, since they too may now be dead or may have a
     // reduced number of uses, allowing other xforms.
     if (recursivelyDeleteUnusedNodes(N))
       continue;
 
     WorklistRemover DeadNodes(*this);
 

llvm/test/CodeGen/AMDGPU/and.ll

define amdgpu_kernel void @s_and_inline_imm_1_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_1.0_i64
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 1.0
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0x3ff00000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_1.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 4607182418800017408
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_neg_1.0_i64
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, -1.0
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0xbff00000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_neg_1.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 13830554455654793216
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_0.5_i64
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0.5
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0x3fe00000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_0.5_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 4602678819172646912
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_neg_0.5_i64:
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, -0.5
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0xbfe00000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_neg_0.5_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 13826050856027422720
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_2.0_i64:
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 2.0
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_2.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 4611686018427387904
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_neg_2.0_i64:
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, -2.0
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_neg_2.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 13835058055282163712
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_4.0_i64:
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 4.0
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0x40100000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_4.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 4616189618054758400
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_imm_neg_4.0_i64:
 ; XSI: s_and_b64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, -4.0
 
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 {{s[0-9]+}}, {{s[0-9]+}}, 0xc0100000
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_imm_neg_4.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 13839561654909534208
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void

 define amdgpu_kernel void @s_and_inline_imm_f32_neg_4.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, -1065353216
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; Shift into upper 32-bits
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 s[[K_HI:[0-9]+]], s{{[0-9]+}}, 4.0
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_high_imm_f32_4.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 4647714815446351872
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 
 ; FUNC-LABEL: {{^}}s_and_inline_high_imm_f32_neg_4.0_i64:
 ; SI: s_load_dwordx2
-; SI: s_load_dwordx2
+; SI: s_load_dword
 ; SI-NOT: and
 ; SI: s_and_b32 s[[K_HI:[0-9]+]], s{{[0-9]+}}, -4.0
 ; SI-NOT: and
 ; SI: buffer_store_dwordx2
 define amdgpu_kernel void @s_and_inline_high_imm_f32_neg_4.0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
   %and = and i64 %a, 13871086852301127680
   store i64 %and, i64 addrspace(1)* %out, align 8
   ret void
 }
 attributes #0 = { nounwind readnone }

llvm/test/CodeGen/AMDGPU/calling-conventions.ll

 ; SI: v_or_b32
 define amdgpu_ps void @ps_mesa_v2i16(<2 x i16> %arg0) {
   %add = add <2 x i16> %arg0, <i16 1, i16 1>
   store <2 x i16> %add, <2 x i16> addrspace(1)* undef
   ret void
 }
 
 ; GCN-LABEL: {{^}}ps_mesa_inreg_v2i16:
-; VI: s_lshr_b32 s1, s0, 16
-; VI: s_add_i32 s1, s1, 1
+; VI: s_and_b32 s1, s0, 0xffff0000
 ; VI: s_add_i32 s0, s0, 1
+; VI: s_add_i32 s1, s1, 0x10000
 ; VI: s_and_b32 s0, s0, 0xffff
-; VI: s_lshl_b32 s1, s1, 16
 ; VI: s_or_b32 s0, s0, s1
 ; VI: v_mov_b32_e32 v0, s0
 
 ; SI: s_lshl_b32 s1, s1, 16
 ; SI: s_add_i32 s0, s0, 1
 ; SI: s_add_i32 s1, s1, 0x10000
 ; SI: s_and_b32 s0, s0, 0xffff
 ; SI: s_or_b32 s0, s0, s1
 define amdgpu_ps void @ps_mesa_inreg_v2i16(<2 x i16> inreg %arg0) {
   %add = add <2 x i16> %arg0, <i16 1, i16 1>
   store <2 x i16> %add, <2 x i16> addrspace(1)* undef
   ret void
 }
 
 attributes #0 = { nounwind noinline }

llvm/test/CodeGen/AMDGPU/fneg-combines.ll

 ; GCN: {{buffer|flat}}_load_dword [[A:v[0-9]+]]
 ; GCN: {{buffer|flat}}_load_dword [[B:v[0-9]+]]
 ; GCN: {{buffer|flat}}_load_dword [[C:v[0-9]+]]
 
 ; GCN-SAFE: v_mac_f32_e32 [[C]], [[A]], [[B]]
 ; GCN-SAFE: v_xor_b32_e32 [[NEG_MAD:v[0-9]+]], 0x80000000, [[C]]
 ; GCN-SAFE-NEXT: v_mul_f32_e32 [[MUL:v[0-9]+]], 4.0, [[C]]
 
-; GCN-NSZ: v_mad_f32 [[NEG_MAD:v[0-9]+]], -[[A]], [[B]], -[[C]]
+; GCN-NSZ: v_mad_f32 [[NEG_MAD:v[0-9]+]], [[A]], -[[B]], -[[C]]
 ; GCN-NSZ-NEXT: v_mul_f32_e32 [[MUL:v[0-9]+]], -4.0, [[NEG_MAD]]
 
 ; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[NEG_MAD]]
 ; GCN-NEXT: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[MUL]]
 define amdgpu_kernel void @v_fneg_fmad_multi_use_fmad_f32(float addrspace(1)* %out, float addrspace(1)* %a.ptr, float addrspace(1)* %b.ptr, float addrspace(1)* %c.ptr) #0 {
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %a.gep = getelementptr inbounds float, float addrspace(1)* %a.ptr, i64 %tid.ext

llvm/test/CodeGen/AMDGPU/idot2.ll

 ; GFX7-NEXT:    v_mad_u32_u24 v0, s4, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot2_acc16:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX8-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
-; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
-; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
-; GFX8-NEXT:    s_mov_b32 s0, 0xffff
+; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
+; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_and_b32 s3, s1, s0
-; GFX8-NEXT:    s_and_b32 s0, s2, s0
-; GFX8-NEXT:    s_lshr_b32 s2, s2, 16
+; GFX8-NEXT:    s_and_b32 s3, s1, s2
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, 16
-; GFX8-NEXT:    v_mov_b32_e32 v3, s2
+; GFX8-NEXT:    s_and_b32 s2, s0, s2
+; GFX8-NEXT:    s_lshr_b32 s0, s0, 16
+; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    v_mad_u32_u24 v2, s3, v3, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot2_acc16:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GFX9-NODL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ushort v2, v[0:1], off
-; GFX9-NODL-NEXT:    s_load_dword s1, s[4:5], 0x0
-; GFX9-NODL-NEXT:    s_load_dword s2, s[6:7], 0x0
-; GFX9-NODL-NEXT:    s_mov_b32 s0, 0xffff
+; GFX9-NODL-NEXT:    s_load_dword s0, s[4:5], 0x0
+; GFX9-NODL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s0
-; GFX9-NODL-NEXT:    s_and_b32 s0, s2, s0
-; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 16
+; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s2
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 16
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s2
+; GFX9-NODL-NEXT:    s_and_b32 s2, s0, s2
+; GFX9-NODL-NEXT:    s_lshr_b32 s0, s0, 16
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s3, v3, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NODL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot2_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/idot4.ll

 ; GFX8-NEXT:    s_movk_i32 s2, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_bfe_u32 s4, s0, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s5, s0, 0x80008
 ; GFX8-NEXT:    s_and_b32 s3, s1, s2
-; GFX8-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX8-NEXT:    s_and_b32 s2, s0, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX8-NEXT:    v_mov_b32_e32 v4, s5
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX8-NEXT:    s_bfe_u32 s7, s0, 0x80010
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, 24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_acc8:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NODL-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_bfe_u32 s4, s0, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s5, s0, 0x80008
 ; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s2
-; GFX9-NODL-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX9-NODL-NEXT:    s_and_b32 s2, s0, s2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NODL-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s5
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX9-NODL-NEXT:    s_bfe_u32 s7, s0, 0x80010
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NODL-NEXT:    s_lshr_b32 s0, s0, 24
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry

 ; GFX7-NEXT:    v_mad_u32_u24 v0, s4, v1, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot2_8:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX8-NEXT:    s_movk_i32 s2, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
-; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
+; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
+; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
+; GFX8-NEXT:    s_movk_i32 s0, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_and_b32 s3, s1, s2
-; GFX8-NEXT:    s_and_b32 s2, s0, s2
+; GFX8-NEXT:    s_and_b32 s3, s2, s0
+; GFX8-NEXT:    s_and_b32 s0, s1, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8-NEXT:    s_bfe_u32 s2, s2, 0x80008
 ; GFX8-NEXT:    s_bfe_u32 s1, s1, 0x80008
-; GFX8-NEXT:    s_bfe_u32 s0, s0, 0x80008
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot2_8:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-NODL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
-; GFX9-NODL-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX9-NODL-NEXT:    s_load_dword s1, s[6:7], 0x0
+; GFX9-NODL-NEXT:    s_load_dword s1, s[4:5], 0x0
+; GFX9-NODL-NEXT:    s_load_dword s2, s[6:7], 0x0
+; GFX9-NODL-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s2
-; GFX9-NODL-NEXT:    s_and_b32 s2, s0, s2
+; GFX9-NODL-NEXT:    s_and_b32 s3, s2, s0
+; GFX9-NODL-NEXT:    s_and_b32 s0, s1, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NODL-NEXT:    s_bfe_u32 s2, s2, 0x80008
 ; GFX9-NODL-NEXT:    s_bfe_u32 s1, s1, 0x80008
-; GFX9-NODL-NEXT:    s_bfe_u32 s0, s0, 0x80008
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot2_8:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-DL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
-; GFX9-DL-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX9-DL-NEXT:    s_load_dword s1, s[6:7], 0x0
+; GFX9-DL-NEXT:    s_load_dword s1, s[4:5], 0x0
+; GFX9-DL-NEXT:    s_load_dword s2, s[6:7], 0x0
+; GFX9-DL-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_and_b32 s3, s1, s2
-; GFX9-DL-NEXT:    s_and_b32 s2, s0, s2
+; GFX9-DL-NEXT:    s_and_b32 s3, s2, s0
+; GFX9-DL-NEXT:    s_and_b32 s0, s1, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-DL-NEXT:    s_bfe_u32 s2, s2, 0x80008
 ; GFX9-DL-NEXT:    s_bfe_u32 s1, s1, 0x80008
-; GFX9-DL-NEXT:    s_bfe_u32 s0, s0, 0x80008
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                    <4 x i8> addrspace(1)* %src2,
                                    i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <4 x i8>, <4 x i8> addrspace(1)* %src1
   %vec2 = load <4 x i8>, <4 x i8> addrspace(1)* %src2
 

 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX8-NEXT:    s_movk_i32 s0, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s3, s1, s0
-; GFX8-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX8-NEXT:    s_and_b32 s0, s2, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX8-NEXT:    s_bfe_u32 s4, s2, 0x80008
-; GFX8-NEXT:    v_mov_b32_e32 v4, s5
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x80008
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x80010
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_CommutationInsideMAD:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NODL-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s0
-; GFX9-NODL-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX9-NODL-NEXT:    s_and_b32 s0, s2, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NODL-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX9-NODL-NEXT:    s_bfe_u32 s4, s2, 0x80008
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s5
+; GFX9-NODL-NEXT:    s_bfe_u32 s5, s2, 0x80008
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX9-NODL-NEXT:    s_bfe_u32 s7, s2, 0x80010
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_CommutationInsideMAD:
 ; GFX9-DL:       ; %bb.0: ; %entry

 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX8-NEXT:    s_movk_i32 s0, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x80008
-; GFX8-NEXT:    s_and_b32 s3, s2, s0
-; GFX8-NEXT:    s_and_b32 s0, s1, s0
+; GFX8-NEXT:    s_and_b32 s3, s1, s0
 ; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x80008
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX8-NEXT:    v_mov_b32_e32 v4, s0
+; GFX8-NEXT:    s_and_b32 s0, s2, s0
+; GFX8-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x80010
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s3, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v4, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_CommutationAccrossMADs:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NODL-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_bfe_u32 s4, s1, 0x80008
-; GFX9-NODL-NEXT:    s_and_b32 s3, s2, s0
-; GFX9-NODL-NEXT:    s_and_b32 s0, s1, s0
+; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s0
 ; GFX9-NODL-NEXT:    s_bfe_u32 s5, s2, 0x80008
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NODL-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s0
+; GFX9-NODL-NEXT:    s_and_b32 s0, s2, s0
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX9-NODL-NEXT:    s_bfe_u32 s7, s2, 0x80010
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s3, v4, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v4, v2
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_CommutationAccrossMADs:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s2, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_bfe_u32 s4, s1, 0x80008
-; GFX9-DL-NEXT:    s_and_b32 s3, s2, s0
-; GFX9-DL-NEXT:    s_and_b32 s0, s1, s0
+; GFX9-DL-NEXT:    s_and_b32 s3, s1, s0
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x80008
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s0
+; GFX9-DL-NEXT:    s_and_b32 s0, s2, s0
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x80010
 ; GFX9-DL-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s3, v4, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v4, v2
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                                         <4 x i8> addrspace(1)* %src2,
                                                         i8 addrspace(1)* nocapture %dst) {
 entry:

 ; GFX7-NEXT:    v_mad_u32_u24 v0, s4, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: notdot4_mixedtypes:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX8-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX8-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
-; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_bfe_i32 s3, s0, 0x80000
-; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80008
-; GFX8-NEXT:    s_bfe_i32 s4, s1, 0x80000
-; GFX8-NEXT:    s_and_b32 s3, s2, s3
-; GFX8-NEXT:    s_and_b32 s2, s2, s4
-; GFX8-NEXT:    s_bfe_u32 s5, s0, 0x80008
-; GFX8-NEXT:    v_mov_b32_e32 v3, s6
-; GFX8-NEXT:    s_bfe_u32 s8, s1, 0x80010
-; GFX8-NEXT:    v_mov_b32_e32 v5, s2
-; GFX8-NEXT:    s_bfe_u32 s7, s0, 0x80010
-; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX8-NEXT:    v_mov_b32_e32 v4, s8
-; GFX8-NEXT:    s_lshr_b32 s0, s0, 24
-; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s3, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v4, v2
+; GFX8-NEXT:    s_bfe_u32 s0, s2, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s1, s3, 0x80008
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
+; GFX8-NEXT:    s_sext_i32_i8 s4, s3
+; GFX8-NEXT:    s_bfe_u32 s5, s3, 0x80010
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
+; GFX8-NEXT:    s_sext_i32_i8 s1, s2
+; GFX8-NEXT:    s_bfe_u32 s4, s2, 0x80010
+; GFX8-NEXT:    s_lshr_b32 s3, s3, 24
+; GFX8-NEXT:    v_mov_b32_e32 v5, s5
+; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s1, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: notdot4_mixedtypes:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-NODL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NODL-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX9-NODL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ushort v2, v[0:1], off
-; GFX9-NODL-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX9-NODL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_bfe_i32 s3, s0, 0x80000
-; GFX9-NODL-NEXT:    s_bfe_u32 s6, s1, 0x80008
-; GFX9-NODL-NEXT:    s_bfe_i32 s4, s1, 0x80000
-; GFX9-NODL-NEXT:    s_and_b32 s3, s2, s3
-; GFX9-NODL-NEXT:    s_and_b32 s2, s2, s4
-; GFX9-NODL-NEXT:    s_bfe_u32 s5, s0, 0x80008
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s6
-; GFX9-NODL-NEXT:    s_bfe_u32 s8, s1, 0x80010
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s2
-; GFX9-NODL-NEXT:    s_bfe_u32 s7, s0, 0x80010
-; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s8
-; GFX9-NODL-NEXT:    s_lshr_b32 s0, s0, 24
-; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s3, v5, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s7, v4, v2
+; GFX9-NODL-NEXT:    s_bfe_u32 s0, s2, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s1, s3, 0x80008
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NODL-NEXT:    s_sext_i32_i8 s4, s3
+; GFX9-NODL-NEXT:    s_bfe_u32 s5, s3, 0x80010
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NODL-NEXT:    s_sext_i32_i8 s1, s2
+; GFX9-NODL-NEXT:    s_bfe_u32 s4, s2, 0x80010
+; GFX9-NODL-NEXT:    s_lshr_b32 s3, s3, 24
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 24
+; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s1, v4, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NODL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: notdot4_mixedtypes:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-DL-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX9-DL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ushort v2, v[0:1], off
-; GFX9-DL-NEXT:    s_load_dword s0, s[4:5], 0x0
-; GFX9-DL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_bfe_i32 s3, s0, 0x80000
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s1, 0x80008
-; GFX9-DL-NEXT:    s_bfe_i32 s4, s1, 0x80000
-; GFX9-DL-NEXT:    s_and_b32 s3, s2, s3
-; GFX9-DL-NEXT:    s_and_b32 s2, s2, s4
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s0, 0x80008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s6
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s1, 0x80010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s2
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s0, 0x80010
-; GFX9-DL-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s8
-; GFX9-DL-NEXT:    s_lshr_b32 s0, s0, 24
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s3, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v4, v2
+; GFX9-DL-NEXT:    s_bfe_u32 s0, s2, 0x80008
+; GFX9-DL-NEXT:    s_bfe_u32 s1, s3, 0x80008
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-DL-NEXT:    s_sext_i32_i8 s4, s3
+; GFX9-DL-NEXT:    s_bfe_u32 s5, s3, 0x80010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-DL-NEXT:    s_sext_i32_i8 s1, s2
+; GFX9-DL-NEXT:    s_bfe_u32 s4, s2, 0x80010
+; GFX9-DL-NEXT:    s_lshr_b32 s3, s3, 24
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 24
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s1, v4, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                               <4 x i8> addrspace(1)* %src2,
                                               i16 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <4 x i8>, <4 x i8> addrspace(1)* %src1
   %vec2 = load <4 x i8>, <4 x i8> addrspace(1)* %src2
 

 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot4_acc16:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX8-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
-; GFX8-NEXT:    s_load_dword s1, s[4:5], 0x0
-; GFX8-NEXT:    s_load_dword s2, s[6:7], 0x0
-; GFX8-NEXT:    s_mov_b32 s0, 0xffff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_lshr_b32 s5, s1, 8
-; GFX8-NEXT:    s_lshr_b32 s6, s2, 8
-; GFX8-NEXT:    s_sext_i32_i8 s4, s2
-; GFX8-NEXT:    s_bfe_i32 s5, s5, 0x80000
-; GFX8-NEXT:    s_bfe_i32 s6, s6, 0x80000
-; GFX8-NEXT:    s_bfe_i32 s8, s2, 0x80010
-; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
-; GFX8-NEXT:    s_sext_i32_i8 s3, s1
-; GFX8-NEXT:    s_bfe_i32 s7, s1, 0x80010
-; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX8-NEXT:    s_and_b32 s4, s0, s5
-; GFX8-NEXT:    s_and_b32 s5, s0, s6
-; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x80000
-; GFX8-NEXT:    s_bfe_i32 s2, s2, 0x80000
+; GFX8-NEXT:    s_sext_i32_i8 s0, s2
+; GFX8-NEXT:    s_sext_i32_i8 s1, s3
+; GFX8-NEXT:    v_mov_b32_e32 v3, s1
+; GFX8-NEXT:    s_bfe_i32 s4, s3, 0x80008
+; GFX8-NEXT:    s_bfe_i32 s5, s3, 0x80010
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
+; GFX8-NEXT:    s_bfe_i32 s1, s2, 0x80008
+; GFX8-NEXT:    s_bfe_i32 s4, s2, 0x80010
+; GFX8-NEXT:    s_ashr_i32 s3, s3, 24
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    s_and_b32 s1, s0, s1
-; GFX8-NEXT:    v_mov_b32_e32 v4, s8
-; GFX8-NEXT:    s_and_b32 s0, s0, s2
+; GFX8-NEXT:    s_ashr_i32 s2, s2, 24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_i32_i24 v2, s3, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s1, v4, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s4, v5, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s3
+; GFX8-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: idot4_acc16:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NODL-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX9-NODL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ushort v2, v[0:1], off
-; GFX9-NODL-NEXT:    s_load_dword s1, s[4:5], 0x0
-; GFX9-NODL-NEXT:    s_load_dword s2, s[6:7], 0x0
-; GFX9-NODL-NEXT:    s_mov_b32 s0, 0xffff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_lshr_b32 s5, s1, 8
-; GFX9-NODL-NEXT:    s_lshr_b32 s6, s2, 8
-; GFX9-NODL-NEXT:    s_sext_i32_i8 s4, s2
-; GFX9-NODL-NEXT:    s_bfe_i32 s5, s5, 0x80000
-; GFX9-NODL-NEXT:    s_bfe_i32 s6, s6, 0x80000
-; GFX9-NODL-NEXT:    s_bfe_i32 s8, s2, 0x80010
-; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-NODL-NEXT:    s_sext_i32_i8 s3, s1
-; GFX9-NODL-NEXT:    s_bfe_i32 s7, s1, 0x80010
-; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX9-NODL-NEXT:    s_and_b32 s4, s0, s5
-; GFX9-NODL-NEXT:    s_and_b32 s5, s0, s6
-; GFX9-NODL-NEXT:    s_bfe_i32 s1, s1, 0x80000
-; GFX9-NODL-NEXT:    s_bfe_i32 s2, s2, 0x80000
+; GFX9-NODL-NEXT:    s_sext_i32_i8 s0, s2
+; GFX9-NODL-NEXT:    s_sext_i32_i8 s1, s3
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NODL-NEXT:    s_bfe_i32 s4, s3, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_i32 s5, s3, 0x80010
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
+; GFX9-NODL-NEXT:    s_bfe_i32 s1, s2, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_i32 s4, s2, 0x80010
+; GFX9-NODL-NEXT:    s_ashr_i32 s3, s3, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-NODL-NEXT:    s_and_b32 s1, s0, s1
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s8
-; GFX9-NODL-NEXT:    s_and_b32 s0, s0, s2
+; GFX9-NODL-NEXT:    s_ashr_i32 s2, s2, 24
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s3, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
-; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
+; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
+; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s1, v4, v2
+; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s4, v5, v2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
+; GFX9-NODL-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
 ; GFX9-NODL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot4_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
+; GFX9-DL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ushort v2, v[0:1], off
-; GFX9-DL-NEXT:    s_load_dword s1, s[4:5], 0x0
-; GFX9-DL-NEXT:    s_load_dword s2, s[6:7], 0x0
-; GFX9-DL-NEXT:    s_mov_b32 s0, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s1, 8
-; GFX9-DL-NEXT:    s_lshr_b32 s6, s2, 8
-; GFX9-DL-NEXT:    s_sext_i32_i8 s4, s2
-; GFX9-DL-NEXT:    s_bfe_i32 s5, s5, 0x80000
-; GFX9-DL-NEXT:    s_bfe_i32 s6, s6, 0x80000
-; GFX9-DL-NEXT:    s_bfe_i32 s8, s2, 0x80010
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-DL-NEXT:    s_sext_i32_i8 s3, s1
-; GFX9-DL-NEXT:    s_bfe_i32 s7, s1, 0x80010
-; GFX9-DL-NEXT:    s_lshr_b32 s1, s1, 24
-; GFX9-DL-NEXT:    s_and_b32 s4, s0, s5
-; GFX9-DL-NEXT:    s_and_b32 s5, s0, s6
-; GFX9-DL-NEXT:    s_bfe_i32 s1, s1, 0x80000
-; GFX9-DL-NEXT:    s_bfe_i32 s2, s2, 0x80000
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-DL-NEXT:    s_and_b32 s1, s0, s1
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s8
-; GFX9-DL-NEXT:    s_and_b32 s0, s0, s2
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s3, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s4, v5, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v3, v2
+; GFX9-DL-NEXT:    v_dot4_i32_i8 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <4 x i8> addrspace(1)* %src2,
                                        i16 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <4 x i8>, <4 x i8> addrspace(1)* %src1
   %vec2 = load <4 x i8>, <4 x i8> addrspace(1)* %src2
 

 ; GFX8-NEXT:    s_movk_i32 s2, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_bfe_u32 s4, s0, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s5, s0, 0x80008
 ; GFX8-NEXT:    s_and_b32 s3, s1, s2
-; GFX8-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX8-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX8-NEXT:    s_and_b32 s2, s0, s2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX8-NEXT:    v_mov_b32_e32 v4, s5
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX8-NEXT:    s_bfe_u32 s7, s0, 0x80010
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, 24
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: idot4_acc8:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NODL-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_bfe_u32 s4, s0, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s5, s0, 0x80008
 ; GFX9-NODL-NEXT:    s_and_b32 s3, s1, s2
-; GFX9-NODL-NEXT:    s_bfe_u32 s5, s1, 0x80008
+; GFX9-NODL-NEXT:    s_bfe_u32 s4, s1, 0x80008
 ; GFX9-NODL-NEXT:    s_and_b32 s2, s0, s2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NODL-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s5
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX9-NODL-NEXT:    s_bfe_u32 s7, s0, 0x80010
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s1, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NODL-NEXT:    s_lshr_b32 s0, s0, 24
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
-; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s4, v4, v2
+; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s5, v4, v2
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s7, v5, v2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NODL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot4_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry

llvm/test/CodeGen/AMDGPU/idot8.ll

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v4, s6
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v6, s5
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v7, s7
-; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v8, s8
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s7
+; GFX8-NEXT:    v_mov_b32_e32 v6, s6
+; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v7, s9
+; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v8, s11
+; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s9
+; GFX8-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc4:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX9-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v7, s7
-; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v8, s8
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s9
+; GFX9-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc4:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX9-DL-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s7
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s8
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s9
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                       <8 x i4> addrspace(1)* %src2,
                                       i4 addrspace(1)* nocapture %dst) {
 entry:

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v6, s7
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v7, s8
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v8, s9
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s7
+; GFX8-NEXT:    v_mov_b32_e32 v6, s6
+; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v7, s9
+; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v8, s11
+; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s10
+; GFX8-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
+; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_CommutationInsideMAD:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v6, s7
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v8, s9
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s10
+; GFX9-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
+; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-NEXT:    v_add_u32_e32 v2, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_CommutationInsideMAD:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s7
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s4, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s9
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s10
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v5, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                                       <8 x i4> addrspace(1)* %src2,
                                                       i4 addrspace(1)* nocapture %dst) {
 entry:

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v4, s6
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v6, s5
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v7, s7
-; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v8, s8
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s7
+; GFX8-NEXT:    v_mov_b32_e32 v6, s6
+; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v7, s9
+; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v8, s11
+; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s9
+; GFX8-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc4_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX9-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v7, s7
-; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v8, s8
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s9
+; GFX9-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc4_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v4, s7, v4
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x4000c
-; GFX9-DL-NEXT:    v_and_b32_e32 v4, 15, v4
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s7
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s8
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
+; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
+; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s9
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
 ; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v5, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s8, v9, v2
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i4 addrspace(1)* nocapture %dst) {
 entry:

 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_lshr_b32 s0, s2, 4
-; GFX8-NEXT:    s_lshr_b32 s1, s4, 4
-; GFX8-NEXT:    s_bfe_i32 s5, s4, 0x40000
-; GFX8-NEXT:    v_lshlrev_b16_e64 v3, 12, s0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s1
-; GFX8-NEXT:    s_bfe_i32 s0, s4, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    s_bfe_i32 s6, s2, 0x40000
+; GFX8-NEXT:    s_bfe_i32 s0, s2, 0x40000
+; GFX8-NEXT:    s_bfe_i32 s1, s4, 0x40000
+; GFX8-NEXT:    v_mov_b32_e32 v3, s1
+; GFX8-NEXT:    s_bfe_i32 s5, s4, 0x40004
+; GFX8-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX8-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX8-NEXT:    s_lshr_b32 s5, s4, 12
-; GFX8-NEXT:    v_mov_b32_e32 v6, s0
-; GFX8-NEXT:    s_bfe_i32 s7, s2, 0x40008
-; GFX8-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
-; GFX8-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX8-NEXT:    v_lshlrev_b16_e64 v7, 12, s1
-; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX8-NEXT:    v_mul_i32_i24_e32 v6, s7, v6
-; GFX8-NEXT:    s_lshr_b32 s0, s2, 20
-; GFX8-NEXT:    s_lshr_b32 s1, s4, 20
-; GFX8-NEXT:    s_bfe_i32 s5, s4, 0x40010
-; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX8-NEXT:    v_lshlrev_b16_e64 v9, 12, s0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
-; GFX8-NEXT:    s_bfe_i32 s8, s2, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v13, s5
-; GFX8-NEXT:    s_lshr_b32 s0, s2, 28
-; GFX8-NEXT:    s_lshr_b32 s9, s4, 28
-; GFX8-NEXT:    s_bfe_i32 s4, s4, 0x40018
-; GFX8-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX8-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX8-NEXT:    v_lshlrev_b16_e64 v11, 12, s0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v12, 12, s9
-; GFX8-NEXT:    s_bfe_i32 s2, s2, 0x40018
-; GFX8-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX8-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
+; GFX8-NEXT:    s_lshr_b32 s7, s4, 12
+; GFX8-NEXT:    s_bfe_i32 s8, s2, 0x40004
+; GFX8-NEXT:    s_bfe_i32 s9, s2, 0x40008
+; GFX8-NEXT:    v_mov_b32_e32 v4, s6
+; GFX8-NEXT:    v_mov_b32_e32 v7, s5
+; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
+; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
+; GFX8-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
+; GFX8-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX8-NEXT:    s_bfe_i32 s12, s4, 0x40014
+; GFX8-NEXT:    s_bfe_i32 s11, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v8, s10
+; GFX8-NEXT:    s_bfe_i32 s14, s4, 0x40018
+; GFX8-NEXT:    s_bfe_i32 s13, s2, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v9, s12
+; GFX8-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX8-NEXT:    s_ashr_i32 s4, s4, 28
+; GFX8-NEXT:    v_mov_b32_e32 v10, s14
+; GFX8-NEXT:    s_ashr_i32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
-; GFX8-NEXT:    v_mad_u32_u24 v2, v7, v8, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v13, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v9, v10, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX8-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v11, v12, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc16:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 4
-; GFX9-NEXT:    s_lshr_b32 s1, s4, 4
-; GFX9-NEXT:    s_bfe_i32 s5, s4, 0x40000
-; GFX9-NEXT:    v_lshlrev_b16_e64 v3, 12, s0
-; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s1
-; GFX9-NEXT:    s_bfe_i32 s0, s4, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-NEXT:    s_bfe_i32 s6, s2, 0x40000
+; GFX9-NEXT:    s_bfe_i32 s0, s2, 0x40000
+; GFX9-NEXT:    s_bfe_i32 s1, s4, 0x40000
+; GFX9-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-NEXT:    s_bfe_i32 s5, s4, 0x40004
+; GFX9-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX9-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX9-NEXT:    s_lshr_b32 s5, s4, 12
-; GFX9-NEXT:    v_mov_b32_e32 v6, s0
-; GFX9-NEXT:    s_bfe_i32 s7, s2, 0x40008
-; GFX9-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
-; GFX9-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v7, 12, s1
-; GFX9-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX9-NEXT:    v_mul_i32_i24_e32 v6, s7, v6
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 20
-; GFX9-NEXT:    s_lshr_b32 s1, s4, 20
-; GFX9-NEXT:    s_bfe_i32 s5, s4, 0x40010
-; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-NEXT:    v_lshlrev_b16_e64 v9, 12, s0
-; GFX9-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
-; GFX9-NEXT:    s_bfe_i32 s8, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v13, s5
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 28
-; GFX9-NEXT:    s_lshr_b32 s9, s4, 28
-; GFX9-NEXT:    s_bfe_i32 s4, s4, 0x40018
-; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX9-NEXT:    v_lshlrev_b16_e64 v11, 12, s0
-; GFX9-NEXT:    v_lshlrev_b16_e64 v12, 12, s9
-; GFX9-NEXT:    s_bfe_i32 s2, s2, 0x40018
-; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
+; GFX9-NEXT:    s_lshr_b32 s7, s4, 12
+; GFX9-NEXT:    s_bfe_i32 s8, s2, 0x40004
+; GFX9-NEXT:    s_bfe_i32 s9, s2, 0x40008
+; GFX9-NEXT:    v_mov_b32_e32 v4, s6
+; GFX9-NEXT:    v_mov_b32_e32 v7, s5
+; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
+; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
+; GFX9-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
+; GFX9-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-NEXT:    s_bfe_i32 s12, s4, 0x40014
+; GFX9-NEXT:    s_bfe_i32 s11, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v8, s10
+; GFX9-NEXT:    s_bfe_i32 s14, s4, 0x40018
+; GFX9-NEXT:    s_bfe_i32 s13, s2, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v9, s12
+; GFX9-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX9-NEXT:    s_ashr_i32 s4, s4, 28
+; GFX9-NEXT:    v_mov_b32_e32 v10, s14
+; GFX9-NEXT:    s_ashr_i32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
-; GFX9-NEXT:    v_mad_u32_u24 v2, v7, v8, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v13, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v9, v10, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v11, v12, v2
 ; GFX9-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 4
-; GFX9-DL-NEXT:    s_lshr_b32 s1, s4, 4
-; GFX9-DL-NEXT:    s_bfe_i32 s5, s4, 0x40000
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v3, 12, s0
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s1
-; GFX9-DL-NEXT:    s_bfe_i32 s0, s4, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-DL-NEXT:    s_bfe_i32 s6, s2, 0x40000
+; GFX9-DL-NEXT:    s_bfe_i32 s0, s2, 0x40000
+; GFX9-DL-NEXT:    s_bfe_i32 s1, s4, 0x40000
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
+; GFX9-DL-NEXT:    s_bfe_i32 s5, s4, 0x40004
+; GFX9-DL-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX9-DL-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s4, 12
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s0
-; GFX9-DL-NEXT:    s_bfe_i32 s7, s2, 0x40008
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v7, 12, s1
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v6, s7, v6
-; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s1, s4, 20
-; GFX9-DL-NEXT:    s_bfe_i32 s5, s4, 0x40010
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s0
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
-; GFX9-DL-NEXT:    s_bfe_i32 s8, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v13, s5
-; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 28
-; GFX9-DL-NEXT:    s_lshr_b32 s9, s4, 28
-; GFX9-DL-NEXT:    s_bfe_i32 s4, s4, 0x40018
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s0
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s9
-; GFX9-DL-NEXT:    s_bfe_i32 s2, s2, 0x40018
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
+; GFX9-DL-NEXT:    s_lshr_b32 s7, s4, 12
+; GFX9-DL-NEXT:    s_bfe_i32 s8, s2, 0x40004
+; GFX9-DL-NEXT:    s_bfe_i32 s9, s2, 0x40008
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s6
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s5
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
+; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
+; GFX9-DL-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
+; GFX9-DL-NEXT:    s_bfe_i32 s12, s4, 0x40014
+; GFX9-DL-NEXT:    s_bfe_i32 s11, s2, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s10
+; GFX9-DL-NEXT:    s_bfe_i32 s14, s4, 0x40018
+; GFX9-DL-NEXT:    s_bfe_i32 s13, s2, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s12
+; GFX9-DL-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX9-DL-NEXT:    s_ashr_i32 s4, s4, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s14
+; GFX9-DL-NEXT:    s_ashr_i32 s2, s2, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v7, v8, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v13, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v9, v10, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v11, v12, v2
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,
                                        i16 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

 ; GFX8-NEXT:    s_movk_i32 s2, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_lshr_b32 s7, s0, 4
-; GFX8-NEXT:    s_lshr_b32 s11, s1, 4
-; GFX8-NEXT:    v_lshlrev_b16_e64 v3, 12, s7
-; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX8-NEXT:    s_bfe_i32 s13, s1, 0x40000
-; GFX8-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
+; GFX8-NEXT:    s_lshr_b32 s4, s0, 12
+; GFX8-NEXT:    s_bfe_i32 s7, s1, 0x40000
+; GFX8-NEXT:    s_lshr_b32 s5, s1, 12
+; GFX8-NEXT:    s_bfe_i32 s9, s1, 0x40004
+; GFX8-NEXT:    s_bfe_i32 s11, s1, 0x40008
+; GFX8-NEXT:    s_bfe_i32 s6, s0, 0x40000
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
+; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX8-NEXT:    s_bfe_i32 s8, s0, 0x40004
+; GFX8-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX8-NEXT:    v_mov_b32_e32 v3, s11
+; GFX8-NEXT:    v_mov_b32_e32 v7, s9
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX8-NEXT:    s_lshr_b32 s6, s0, 12
-; GFX8-NEXT:    s_lshr_b32 s10, s1, 12
-; GFX8-NEXT:    s_bfe_i32 s15, s1, 0x40008
-; GFX8-NEXT:    s_bfe_i32 s12, s0, 0x40000
-; GFX8-NEXT:    v_mov_b32_e32 v12, s13
-; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
-; GFX8-NEXT:    v_lshlrev_b16_e64 v7, 12, s10
-; GFX8-NEXT:    s_bfe_i32 s14, s0, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s15
-; GFX8-NEXT:    v_and_b32_e32 v3, s2, v3
+; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX8-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX8-NEXT:    s_bfe_i32 s13, s1, 0x40010
 ; GFX8-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX8-NEXT:    s_lshr_b32 s5, s0, 20
-; GFX8-NEXT:    s_lshr_b32 s9, s1, 20
-; GFX8-NEXT:    v_mul_i32_i24_e32 v5, s14, v5
-; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX8-NEXT:    v_lshlrev_b16_e64 v9, 12, s9
-; GFX8-NEXT:    s_bfe_i32 s17, s1, 0x40010
-; GFX8-NEXT:    v_and_b32_e32 v6, s2, v6
-; GFX8-NEXT:    v_and_b32_e32 v7, s2, v7
-; GFX8-NEXT:    s_lshr_b32 s8, s1, 28
-; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX8-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX8-NEXT:    s_lshr_b32 s4, s0, 28
-; GFX8-NEXT:    s_bfe_i32 s16, s0, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v13, s17
-; GFX8-NEXT:    v_lshlrev_b16_e64 v10, 12, s4
-; GFX8-NEXT:    v_lshlrev_b16_e64 v11, 12, s8
-; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x40018
-; GFX8-NEXT:    v_and_b32_e32 v8, s2, v8
-; GFX8-NEXT:    v_and_b32_e32 v9, s2, v9
-; GFX8-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX8-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX8-NEXT:    s_bfe_i32 s0, s0, 0x40018
-; GFX8-NEXT:    v_and_b32_e32 v10, s2, v10
-; GFX8-NEXT:    v_and_b32_e32 v11, s2, v11
+; GFX8-NEXT:    v_and_b32_e32 v5, s2, v5
+; GFX8-NEXT:    s_bfe_i32 s15, s1, 0x40014
+; GFX8-NEXT:    s_bfe_i32 s12, s0, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v8, s13
+; GFX8-NEXT:    s_bfe_i32 s17, s1, 0x40018
+; GFX8-NEXT:    s_bfe_i32 s14, s0, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v9, s15
+; GFX8-NEXT:    s_bfe_i32 s16, s0, 0x40018
+; GFX8-NEXT:    s_ashr_i32 s1, s1, 28
+; GFX8-NEXT:    v_mov_b32_e32 v10, s17
+; GFX8-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_i32_i24 v2, s12, v12, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mad_u32_u24 v2, v6, v7, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s16, v13, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v8, v9, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v10, v11, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc8:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_lshr_b32 s7, s0, 4
-; GFX9-NEXT:    s_lshr_b32 s11, s1, 4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v3, 12, s7
-; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX9-NEXT:    s_bfe_i32 s13, s1, 0x40000
-; GFX9-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
+; GFX9-NEXT:    s_lshr_b32 s4, s0, 12
+; GFX9-NEXT:    s_bfe_i32 s7, s1, 0x40000
+; GFX9-NEXT:    s_lshr_b32 s5, s1, 12
+; GFX9-NEXT:    s_bfe_i32 s9, s1, 0x40004
+; GFX9-NEXT:    s_bfe_i32 s11, s1, 0x40008
+; GFX9-NEXT:    s_bfe_i32 s6, s0, 0x40000
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
+; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX9-NEXT:    s_bfe_i32 s8, s0, 0x40004
+; GFX9-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX9-NEXT:    v_mov_b32_e32 v3, s11
+; GFX9-NEXT:    v_mov_b32_e32 v7, s9
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-NEXT:    s_lshr_b32 s6, s0, 12
-; GFX9-NEXT:    s_lshr_b32 s10, s1, 12
-; GFX9-NEXT:    s_bfe_i32 s15, s1, 0x40008
-; GFX9-NEXT:    s_bfe_i32 s12, s0, 0x40000
-; GFX9-NEXT:    v_mov_b32_e32 v12, s13
-; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
-; GFX9-NEXT:    v_lshlrev_b16_e64 v7, 12, s10
-; GFX9-NEXT:    s_bfe_i32 s14, s0, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s15
-; GFX9-NEXT:    v_and_b32_e32 v3, s2, v3
+; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX9-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX9-NEXT:    s_bfe_i32 s13, s1, 0x40010
 ; GFX9-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-NEXT:    s_lshr_b32 s5, s0, 20
-; GFX9-NEXT:    s_lshr_b32 s9, s1, 20
-; GFX9-NEXT:    v_mul_i32_i24_e32 v5, s14, v5
-; GFX9-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX9-NEXT:    v_lshlrev_b16_e64 v9, 12, s9
-; GFX9-NEXT:    s_bfe_i32 s17, s1, 0x40010
-; GFX9-NEXT:    v_and_b32_e32 v6, s2, v6
-; GFX9-NEXT:    v_and_b32_e32 v7, s2, v7
-; GFX9-NEXT:    s_lshr_b32 s8, s1, 28
-; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-NEXT:    s_lshr_b32 s4, s0, 28
-; GFX9-NEXT:    s_bfe_i32 s16, s0, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v13, s17
-; GFX9-NEXT:    v_lshlrev_b16_e64 v10, 12, s4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v11, 12, s8
-; GFX9-NEXT:    s_bfe_i32 s1, s1, 0x40018
-; GFX9-NEXT:    v_and_b32_e32 v8, s2, v8
-; GFX9-NEXT:    v_and_b32_e32 v9, s2, v9
-; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-NEXT:    s_bfe_i32 s0, s0, 0x40018
-; GFX9-NEXT:    v_and_b32_e32 v10, s2, v10
-; GFX9-NEXT:    v_and_b32_e32 v11, s2, v11
+; GFX9-NEXT:    v_and_b32_e32 v5, s2, v5
+; GFX9-NEXT:    s_bfe_i32 s15, s1, 0x40014
+; GFX9-NEXT:    s_bfe_i32 s12, s0, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v8, s13
+; GFX9-NEXT:    s_bfe_i32 s17, s1, 0x40018
+; GFX9-NEXT:    s_bfe_i32 s14, s0, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v9, s15
+; GFX9-NEXT:    s_bfe_i32 s16, s0, 0x40018
+; GFX9-NEXT:    s_ashr_i32 s1, s1, 28
+; GFX9-NEXT:    v_mov_b32_e32 v10, s17
+; GFX9-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_i32_i24 v2, s12, v12, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-NEXT:    v_mad_u32_u24 v2, v6, v7, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s16, v13, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v8, v9, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX9-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, v10, v11, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_lshr_b32 s7, s0, 4
-; GFX9-DL-NEXT:    s_lshr_b32 s11, s1, 4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v3, 12, s7
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s11
-; GFX9-DL-NEXT:    s_bfe_i32 s13, s1, 0x40000
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
+; GFX9-DL-NEXT:    s_lshr_b32 s4, s0, 12
+; GFX9-DL-NEXT:    s_bfe_i32 s7, s1, 0x40000
+; GFX9-DL-NEXT:    s_lshr_b32 s5, s1, 12
+; GFX9-DL-NEXT:    s_bfe_i32 s9, s1, 0x40004
+; GFX9-DL-NEXT:    s_bfe_i32 s11, s1, 0x40008
+; GFX9-DL-NEXT:    s_bfe_i32 s6, s0, 0x40000
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX9-DL-NEXT:    s_bfe_i32 s8, s0, 0x40004
+; GFX9-DL-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s11
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-DL-NEXT:    s_lshr_b32 s6, s0, 12
-; GFX9-DL-NEXT:    s_lshr_b32 s10, s1, 12
-; GFX9-DL-NEXT:    s_bfe_i32 s15, s1, 0x40008
-; GFX9-DL-NEXT:    s_bfe_i32 s12, s0, 0x40000
-; GFX9-DL-NEXT:    v_mov_b32_e32 v12, s13
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v7, 12, s10
-; GFX9-DL-NEXT:    s_bfe_i32 s14, s0, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s15
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, s2, v3
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
+; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX9-DL-NEXT:    s_bfe_i32 s13, s1, 0x40010
 ; GFX9-DL-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s0, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s9, s1, 20
-; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v5, s14, v5
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s9
-; GFX9-DL-NEXT:    s_bfe_i32 s17, s1, 0x40010
-; GFX9-DL-NEXT:    v_and_b32_e32 v6, s2, v6
-; GFX9-DL-NEXT:    v_and_b32_e32 v7, s2, v7
-; GFX9-DL-NEXT:    s_lshr_b32 s8, s1, 28
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-DL-NEXT:    s_lshr_b32 s4, s0, 28
-; GFX9-DL-NEXT:    s_bfe_i32 s16, s0, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v13, s17
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s8
-; GFX9-DL-NEXT:    s_bfe_i32 s1, s1, 0x40018
-; GFX9-DL-NEXT:    v_and_b32_e32 v8, s2, v8
-; GFX9-DL-NEXT:    v_and_b32_e32 v9, s2, v9
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX9-DL-NEXT:    s_bfe_i32 s0, s0, 0x40018
-; GFX9-DL-NEXT:    v_and_b32_e32 v10, s2, v10
-; GFX9-DL-NEXT:    v_and_b32_e32 v11, s2, v11
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, s2, v5
+; GFX9-DL-NEXT:    s_bfe_i32 s15, s1, 0x40014
+; GFX9-DL-NEXT:    s_bfe_i32 s12, s0, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s13
+; GFX9-DL-NEXT:    s_bfe_i32 s17, s1, 0x40018
+; GFX9-DL-NEXT:    s_bfe_i32 s14, s0, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s15
+; GFX9-DL-NEXT:    s_bfe_i32 s16, s0, 0x40018
+; GFX9-DL-NEXT:    s_ashr_i32 s1, s1, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s17
+; GFX9-DL-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s12, v12, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v6, v7, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s16, v13, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v8, v9, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v10, v11, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,
                                        i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

llvm/test/CodeGen/AMDGPU/mad_uint24.ll

 ; RUN: llc < %s -march=r600 -mcpu=redwood | FileCheck %s --check-prefix=EG --check-prefix=FUNC
 ; RUN: llc < %s -march=r600 -mcpu=cayman | FileCheck %s --check-prefix=EG --check-prefix=FUNC
-; RUN: llc < %s -march=amdgcn -verify-machineinstrs | FileCheck %s --check-prefix=SI --check-prefix=FUNC --check-prefix=GCN
-; RUN: llc < %s -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs | FileCheck %s --check-prefix=VI --check-prefix=FUNC --check-prefix=GCN
-; RUN: llc < %s -march=amdgcn -mcpu=fiji -mattr=-flat-for-global -verify-machineinstrs | FileCheck %s --check-prefix=VI --check-prefix=FUNC --check-prefix=GCN
+; RUN: llc < %s -march=amdgcn -verify-machineinstrs | FileCheck %s --check-prefix=SI --check-prefix=FUNC --check-prefix=GCN --check-prefix=GCN1
+; RUN: llc < %s -march=amdgcn -mcpu=tonga -mattr=-flat-for-global -verify-machineinstrs | FileCheck %s --check-prefix=VI --check-prefix=FUNC --check-prefix=GCN --check-prefix=GCN2
+; RUN: llc < %s -march=amdgcn -mcpu=fiji -mattr=-flat-for-global -verify-machineinstrs | FileCheck %s --check-prefix=VI --check-prefix=FUNC --check-prefix=GCN --check-prefix=GCN2
 
 declare i32 @llvm.r600.read.tidig.x() nounwind readnone
 
 ; FUNC-LABEL: {{^}}u32_mad24:
 ; EG: MULADD_UINT24
 ; SI: v_mad_u32_u24
 ; VI: v_mad_u32_u24
 

 
 ; FUNC-LABEL: {{^}}i16_mad24:
 ; The order of A and B does not matter.
 ; EG: MULADD_UINT24 {{[* ]*}}T{{[0-9]}}.[[MAD_CHAN:[XYZW]]]
 ; The result must be sign-extended
 ; EG: BFE_INT {{[* ]*}}T{{[0-9]\.[XYZW]}}, PV.[[MAD_CHAN]], 0.0, literal.x
 ; EG: 16
 ; FIXME: Should be using scalar instructions here.
-; GCN: v_mad_u32_u24 [[MAD:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
-; GCN: v_bfe_i32 v{{[0-9]}}, [[MAD]], 0, 16
+
+; GCN1: v_mad_u32_u24 [[MAD:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
+; GCN1: v_bfe_i32 v{{[0-9]}}, [[MAD]], 0, 16
+
+; GCN2: s_mul_i32 [[MUL:s[0-9]]], {{[s][0-9], [s][0-9]}}
+; GCN2: s_add_i32 [[MAD:s[0-9]]], [[MUL]], s{{[0-9]}}
+; GCN2: s_sext_i32_i16 s0, [[MAD]]
+; GCN2: v_mov_b32_e32 v0, s0
+
 define amdgpu_kernel void @i16_mad24(i32 addrspace(1)* %out, i16 %a, i16 %b, i16 %c) {
 entry:
   %0 = mul i16 %a, %b
   %1 = add i16 %0, %c
   %2 = sext i16 %1 to i32
   store i32 %2, i32 addrspace(1)* %out
   ret void
 }
 
 ; FIXME: Need to handle non-uniform case for function below (load without gep).
 ; FUNC-LABEL: {{^}}i8_mad24:
 ; EG: MULADD_UINT24 {{[* ]*}}T{{[0-9]}}.[[MAD_CHAN:[XYZW]]]
 ; The result must be sign-extended
 ; EG: BFE_INT {{[* ]*}}T{{[0-9]\.[XYZW]}}, PV.[[MAD_CHAN]], 0.0, literal.x
 ; EG: 8
-; GCN: v_mad_u32_u24 [[MUL:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
-; GCN: v_bfe_i32 v{{[0-9]}}, [[MUL]], 0, 8
+
+; GCN1: v_mad_u32_u24 [[MUL:v[0-9]]], {{[sv][0-9], [sv][0-9]}}
+; GCN1: v_bfe_i32 v{{[0-9]}}, [[MUL]], 0, 8
+
+; GCN2: s_mul_i32 [[MUL:s[0-9]]], {{[s][0-9], [s][0-9]}}
+; GCN2: s_add_i32 [[MAD:s[0-9]]], [[MUL]], s{{[0-9]}}
+; GCN2: s_sext_i32_i8 s0, [[MAD]]
+; GCN2: v_mov_b32_e32 v0, s0
 define amdgpu_kernel void @i8_mad24(i32 addrspace(1)* %out, i8 %a, i8 %b, i8 %c) {
 entry:
   %0 = mul i8 %a, %b
   %1 = add i8 %0, %c
   %2 = sext i8 %1 to i32
   store i32 %2, i32 addrspace(1)* %out
   ret void
 }

llvm/test/CodeGen/AMDGPU/shift-and-i64-ubfe.ll

define amdgpu_kernel void @v_uextract_bit_31_32_i64_trunc_i32(i32 addrspace(1)* %out, i64 addrspace(1)* %in) #1 {
   %srl = lshr i64 %ld.64, 31
   %trunc = trunc i64 %srl to i32
   %bit = and i32 %trunc, 3
   store i32 %bit, i32 addrspace(1)* %out.gep
   ret void
 }
 
 ; GCN-LABEL: {{^}}and_not_mask_i64:
-; GCN-DAG: buffer_load_dwordx2 v{{\[}}[[VALLO:[0-9]+]]:[[VALHI:[0-9]+]]{{\]}}
+; GCN-DAG: buffer_load_dword v[[VAL:[0-9]+]]
 ; GCN-DAG: v_mov_b32_e32 v[[ZERO:[0-9]+]], 0{{$}}
 ; GCN-DAG: v_mov_b32_e32 v[[SHRHI:[0-9]+]], v[[ZERO]]{{$}}
-; GCN: v_lshrrev_b32_e32 [[SHR:v[0-9]+]], 20, v[[VALLO]]
+; GCN: v_lshrrev_b32_e32 [[SHR:v[0-9]+]], 20, v[[VAL]]
 ; GCN-DAG: v_and_b32_e32 v[[SHRLO:[0-9]+]], 4, [[SHR]]
 ; GCN-NOT: v[[SHRLO]]
 ; GCN-NOT: v[[SHRHI]]
 ; GCN: buffer_store_dwordx2 v{{\[}}[[SHRLO]]:[[SHRHI]]{{\]}}
 define amdgpu_kernel void @and_not_mask_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %in) #1 {
   %id.x = tail call i32 @llvm.amdgcn.workitem.id.x()
   %in.gep = getelementptr i64, i64 addrspace(1)* %in, i32 %id.x
   %out.gep = getelementptr i64, i64 addrspace(1)* %out, i32 %id.x

llvm/test/CodeGen/AMDGPU/shift-i64-opts.ll

define amdgpu_kernel void @lshr_i64_32(i64 addrspace(1)* %out, i64 addrspace(1)* %in) {
   store i64 %shl, i64 addrspace(1)* %out
   ret void
 }
 
 ; Make sure the and of the constant doesn't prevent bfe from forming
 ; after 64-bit shift is split.
 
 ; GCN-LABEL: {{^}}lshr_and_i64_35:
-; GCN: buffer_load_dwordx2 v{{\[}}[[LO:[0-9]+]]:[[HI:[0-9]+]]{{\]}}
-; GCN: v_bfe_u32 v[[BFE:[0-9]+]], v[[HI]], 8, 23
 ; GCN: v_mov_b32_e32 v[[ZERO:[0-9]+]], 0{{$}}
+; GCN: buffer_load_dword v[[LO:[0-9]+]]
+; GCN: v_bfe_u32 v[[BFE:[0-9]+]], v[[LO]], 8, 23
 ; GCN: buffer_store_dwordx2 v{{\[}}[[BFE]]:[[ZERO]]{{\]}}
 define amdgpu_kernel void @lshr_and_i64_35(i64 addrspace(1)* %out, i64 addrspace(1)* %in) {
   %val = load i64, i64 addrspace(1)* %in
   %and = and i64 %val, 9223372036854775807 ; 0x7fffffffffffffff
   %shl = lshr i64 %and, 40
   store i64 %shl, i64 addrspace(1)* %out
   ret void
 }

llvm/test/CodeGen/AMDGPU/shl_add_ptr.ll

define void @shl_add_ptr_combine_2use_both_max_private_offset(i16 zeroext %idx.arg) #0 {
   %shl1 = shl i32 %idx.add, 5
   %ptr0 = inttoptr i32 %shl0 to i32 addrspace(5)*
   %ptr1 = inttoptr i32 %shl1 to i32 addrspace(5)*
   store volatile i32 9, i32 addrspace(5)* %ptr0
   store volatile i32 10, i32 addrspace(5)* %ptr1
   ret void
 }
 
+; FIXME: This or should fold into an offset on the write
 ; GCN-LABEL: {{^}}shl_or_ptr_combine_2use_lds:
 ; GCN: v_lshlrev_b32_e32 [[SCALE0:v[0-9]+]], 3, v0
-; GCN: ds_write_b32 [[SCALE0]], v{{[0-9]+}} offset:32
+; GCN: v_or_b32_e32 [[SCALE1:v[0-9]+]], 32, [[SCALE0]]
+; GCN: ds_write_b32 [[SCALE1]], v{{[0-9]+}}
 
 ; GCN: v_lshlrev_b32_e32 [[SCALE1:v[0-9]+]], 4, v0
 ; GCN: ds_write_b32 [[SCALE1]], v{{[0-9]+}} offset:64
 define void @shl_or_ptr_combine_2use_lds(i32 %idx) #0 {
   %idx.add = or i32 %idx, 4
   %shl0 = shl i32 %idx.add, 3
   %shl1 = shl i32 %idx.add, 4
   %ptr0 = inttoptr i32 %shl0 to i32 addrspace(3)*

llvm/test/CodeGen/AMDGPU/widen-smrd-loads.ll

 ; VI-LABEL: widen_i16_constant_load:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; VI-NEXT:    v_mov_b32_e32 v0, 0
 ; VI-NEXT:    v_mov_b32_e32 v1, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_and_b32 s0, s0, 0xffff
 ; VI-NEXT:    s_addk_i32 s0, 0x3e7
 ; VI-NEXT:    s_or_b32 s0, s0, 4
 ; VI-NEXT:    v_mov_b32_e32 v2, s0
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
   %load = load i16, i16 addrspace(4)* %arg, align 4
   %add = add i16 %load, 999
   %or = or i16 %add, 4

 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s1, 0
 ; VI-NEXT:    v_mov_b32_e32 v0, 0
 ; VI-NEXT:    v_mov_b32_e32 v1, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_and_b32 s0, s0, 0xffff
 ; VI-NEXT:    s_addk_i32 s0, 0x3e7
 ; VI-NEXT:    s_or_b32 s0, s0, 4
 ; VI-NEXT:    v_mov_b32_e32 v2, s0
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
   %load = load i16, i16 addrspace(6)* %arg, align 4
   %add = add i16 %load, 999
   %or = or i16 %add, 4

 ; VI-LABEL: widen_i16_global_invariant_load:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; VI-NEXT:    v_mov_b32_e32 v0, 0
 ; VI-NEXT:    v_mov_b32_e32 v1, 0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_and_b32 s0, s0, 0xffff
 ; VI-NEXT:    s_addk_i32 s0, 0x3e7
 ; VI-NEXT:    s_or_b32 s0, s0, 1
 ; VI-NEXT:    v_mov_b32_e32 v2, s0
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
   %load = load i16, i16 addrspace(1)* %arg, align 4, !invariant.load !0
   %add = add i16 %load, 999
   %or = or i16 %add, 1
   store i16 %or, i16 addrspace(1)* null
   ret void
 }
 
 declare i32 @llvm.amdgcn.workitem.id.x()
 
 !0 = !{}

llvm/test/CodeGen/ARM/CGP/arm-cgp-overflow.ll

 define i32 @overflow_add_positive_const_limit(i8 zeroext %a) {
   %add = add i8 %a, 1
   %cmp = icmp ugt i8 %add, 128
   %res = select i1 %cmp, i32 8, i32 16
   ret i32 %res
 }
 
 ; CHECK-LABEL: unsafe_add_underflow:
-; CHECK:  subs r0, #2
-; CHECK:  uxtb [[EXT:r[0-9]+]], r0
-; CHECK:  cmp [[EXT]], #255
-; CHECK:  moveq r0, #8
+; CHECK: movs r1, #16
+; CHECK: cmp  r0, #1
+; CHECK: it eq
+; CHECK: moveq  r1, #8
+; CHECK: mov  r0, r1
 define i32 @unsafe_add_underflow(i8 zeroext %a) {
   %add = add i8 %a, -2
   %cmp = icmp ugt i8 %add, 254
   %res = select i1 %cmp, i32 8, i32 16
   ret i32 %res
 }
 
 ; CHECK-LABEL: safe_add_underflow:

llvm/test/CodeGen/ARM/vdup.ll

; CHECK-NEXT:    mov pc, lr
   %1 = insertelement  <2 x i32> undef, i32 %x, i32 0
   %2 = insertelement  <2 x i32> %1, i32 %x, i32 1
   ret <2 x i32> %2
 }
 
 define <4 x i16> @check_i16(<8 x i16> %v) nounwind {
 ; CHECK-LABEL: check_i16:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmov d17, r2, r3
 ; CHECK-NEXT:    vmov d16, r0, r1
 ; CHECK-NEXT:    vdup.16 d16, d16[3]
 ; CHECK-NEXT:    vmov r0, r1, d16
 ; CHECK-NEXT:    mov pc, lr
   %x = extractelement <8 x i16> %v, i32 3
   %1 = insertelement  <4 x i16> undef, i16 %x, i32 0
   %2 = insertelement  <4 x i16> %1, i16 %x, i32 1
   ret <4 x i16> %2
 }
 
 define <8 x i8> @check_i8(<16 x i8> %v) nounwind {
 ; CHECK-LABEL: check_i8:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmov d17, r2, r3
 ; CHECK-NEXT:    vmov d16, r0, r1
 ; CHECK-NEXT:    vdup.8 d16, d16[3]
 ; CHECK-NEXT:    vmov r0, r1, d16
 ; CHECK-NEXT:    mov pc, lr
   %x = extractelement <16 x i8> %v, i32 3
   %1 = insertelement  <8  x i8> undef, i8 %x, i32 0
   %2 = insertelement  <8  x i8> %1, i8 %x, i32 1
   ret <8 x i8> %2

llvm/test/CodeGen/PowerPC/pr39478.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=powerpc64le-unknown-unknown -verify-machineinstrs < %s | \
 ; RUN:   FileCheck %s
 ; RUN: llc -mtriple=powerpc64-unknown-unknown -verify-machineinstrs < %s | \
 ; RUN:   FileCheck %s
 
 define void @pr39478() {
 ; CHECK-LABEL: pr39478:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    lwz 3, 0(3)
-; CHECK-NEXT:    stb 3, 0(3)
 ; CHECK-NEXT:    blr
 entry:
   %tmp32 = load i64, i64* undef, align 8
   %tmp33 = load i32, i32* undef, align 4
   %tmp34 = and i32 %tmp33, -256
   %tmp35 = lshr i64 %tmp32, 32
   %tmp36 = shl nuw nsw i64 %tmp35, 24
   %tmp37 = trunc i64 %tmp36 to i32
   %tmp38 = call i32 @llvm.bswap.i32(i32 %tmp37)
   %tmp39 = or i32 %tmp38, %tmp34
   store i32 %tmp39, i32* undef, align 4
   ret void
 }
 
 declare i32 @llvm.bswap.i32(i32)

llvm/test/CodeGen/PowerPC/testComparesigesll.ll

 define signext i32 @test_igesll_sext_z(i64 %a) {
 ; CHECK-LABEL: test_igesll_sext_z:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    sradi r3, r3, 63
 ; CHECK-NEXT:    not r3, r3
 ; CHECK-NEXT:    blr
 ; CHECK-BE-LABEL: test_igesll_sext_z:
 ; CHECK-BE:       # %bb.0: # %entry
-; CHECK-BE-NEXT:    sradi r3, r3, 63
 ; CHECK-BE-NEXT:    not r3, r3
+; CHECK-BE-NEXT:    sradi r3, r3, 63
 ; CHECK-BE-NEXT:    blr
 ;
 ; CHECK-LE-LABEL: test_igesll_sext_z:
 ; CHECK-LE:       # %bb.0: # %entry
-; CHECK-LE-NEXT:    sradi r3, r3, 63
 ; CHECK-LE-NEXT:    not r3, r3
+; CHECK-LE-NEXT:    sradi r3, r3, 63
 ; CHECK-LE-NEXT:    blr
 entry:
   %cmp = icmp sgt i64 %a, -1
   %sub = sext i1 %cmp to i32
   ret i32 %sub
 }
 
 define void @test_igesll_store(i64 %a, i64 %b) {

llvm/test/CodeGen/X86/constant-combines.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-unknown"
 
 define void @PR22524({ float, float }* %arg) {
 ; Check that we can materialize the zero constants we store in two places here,
 ; and at least form a legal store of the floating point value at the end.
 ; The DAG combiner at one point contained bugs that given enough permutations
 ; would incorrectly form an illegal operation for the last of these stores when
 ; it folded it to a zero too late to legalize the zero store operation. If this
 ; ever starts forming a zero store instead of movss, the test case has stopped
 ; being useful.
 ;
 ; CHECK-LABEL: PR22524:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    movd %eax, %xmm0
-; CHECK-NEXT:    xorps %xmm1, %xmm1
-; CHECK-NEXT:    mulss %xmm0, %xmm1
 ; CHECK-NEXT:    movq $0, (%rdi)
-; CHECK-NEXT:    movss %xmm1, 4(%rdi)
+; CHECK-NEXT:    movl $0, 4(%rdi)
 ; CHECK-NEXT:    retq
 entry:
   %0 = getelementptr inbounds { float, float }, { float, float }* %arg,  i32 0, i32 1
   store float 0.000000e+00, float* %0, align 4
   %1 = getelementptr inbounds { float, float }, { float, float }* %arg, i64 0,  i32 0
   %2 = bitcast float* %1 to i64*
   %3 = load i64, i64* %2, align 8
   %4 = trunc i64 %3 to i32

llvm/test/CodeGen/X86/jump_sign.ll

 ; CHECK-NEXT:  # %bb.3: # %sw.bb
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne .LBB12_8
 ; CHECK-NEXT:  # %bb.4: # %if.end29
 ; CHECK-NEXT:    movzwl (%eax), %eax
 ; CHECK-NEXT:    movzwl %ax, %eax
 ; CHECK-NEXT:    imull $52429, %eax, %ecx # imm = 0xCCCD
-; CHECK-NEXT:    shrl $19, %ecx
-; CHECK-NEXT:    addl %ecx, %ecx
+; CHECK-NEXT:    shrl $18, %ecx
+; CHECK-NEXT:    andl $-2, %ecx
 ; CHECK-NEXT:    leal (%ecx,%ecx,4), %ecx
 ; CHECK-NEXT:    cmpw %cx, %ax
 ; CHECK-NEXT:    jne .LBB12_5
 ; CHECK-NEXT:  .LBB12_8: # %if.then44
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je .LBB12_9
 ; CHECK-NEXT:  # %bb.10: # %if.else.i104

llvm/test/CodeGen/X86/known-signbits-vector.ll

 ;
 ; X64-LABEL: signbits_ashr_sext_sextinreg_and_extract_sitofp:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vpsrlq $61, %xmm0, %xmm0
 ; X64-NEXT:    vmovdqa {{.*#+}} xmm1 = [4,8]
 ; X64-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; X64-NEXT:    vpsubq %xmm1, %xmm0, %xmm0
 ; X64-NEXT:    movslq %edi, %rax
-; X64-NEXT:    vpinsrq $0, %rax, %xmm0, %xmm1
+; X64-NEXT:    vmovq %rax, %xmm1
 ; X64-NEXT:    vpand %xmm1, %xmm0, %xmm0
 ; X64-NEXT:    vmovq %xmm0, %rax
 ; X64-NEXT:    vcvtsi2ssl %eax, %xmm2, %xmm0
 ; X64-NEXT:    retq
   %1 = ashr <2 x i64> %a0, <i64 61, i64 60>
   %2 = sext i32 %a2 to i64
   %3 = insertelement <2 x i64> %a1, i64 %2, i32 0
   %4 = shl <2 x i64> %3, <i64 20, i64 20>

llvm/test/CodeGen/X86/legalize-shift-64.ll

 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    .cfi_def_cfa_register %ebp
 ; CHECK-NEXT:    andl $-8, %esp
 ; CHECK-NEXT:    subl $16, %esp
 ; CHECK-NEXT:    movl $1, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    orl $0, %eax
-; CHECK-NEXT:    je .LBB5_3
+; CHECK-NEXT:    movb $1, %al
+; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    jne .LBB5_3
 ; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    jmp .LBB5_2
 ; CHECK-NEXT:  .LBB5_3: # %if.end
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:  .LBB5_2: # %if.then
 ; CHECK-NEXT:    movl %ebp, %esp
 ; CHECK-NEXT:    popl %ebp

llvm/test/CodeGen/X86/masked_gather_scatter.ll

 ; SKX-NEXT:    vpsrad $24, %ymm0, %ymm1
 ; SKX-NEXT:    vgatherdps (%rdi,%ymm1,4), %ymm0 {%k1}
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: sext_v8i8_index:
 ; SKX_32:       # %bb.0:
 ; SKX_32-NEXT:    vpmovzxwd {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; SKX_32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; SKX_32-NEXT:    kxnorw %k0, %k0, %k1
 ; SKX_32-NEXT:    vpslld $24, %ymm0, %ymm0
 ; SKX_32-NEXT:    vpsrad $24, %ymm0, %ymm1
-; SKX_32-NEXT:    kxnorw %k0, %k0, %k1
 ; SKX_32-NEXT:    vgatherdps (%eax,%ymm1,4), %ymm0 {%k1}
 ; SKX_32-NEXT:    retl
 
   %sext_ind = sext <8 x i8> %ind to <8 x i64>
   %gep.random = getelementptr float, float *%base, <8 x i64> %sext_ind
 
   %res = call <8 x float> @llvm.masked.gather.v8f32.v8p0f32(<8 x float*> %gep.random, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x float> undef)
   ret <8 x float>%res

llvm/test/CodeGen/X86/movmsk.ll

entry:
   %__f.i = getelementptr inbounds %union.anon, %union.anon* %__u.i, i64 0, i32 0
   store float %add, float* %__f.i, align 4
   %2 = bitcast float %add to i32
   %shr.i = lshr i32 %2, 31
   ret i32 %shr.i
 }
 
 ; PR11570
-; FIXME: This should also use movmskps; we don't form the FGETSIGN node
-; in this case, though.
 define void @float_call_signbit(double %n) {
 ; CHECK-LABEL: float_call_signbit:
 ; CHECK:       ## %bb.0: ## %entry
-; CHECK-NEXT:    movq %xmm0, %rdi
-; CHECK-NEXT:    shrq $63, %rdi
-; CHECK-NEXT:    ## kill: def $edi killed $edi killed $rdi
+; CHECK-NEXT:    movmskpd %xmm0, %edi
+; CHECK-NEXT:    andl $1, %edi
 ; CHECK-NEXT:    jmp _float_call_signbit_callee ## TAILCALL
 entry:
   %t0 = bitcast double %n to i64
   %tobool.i.i.i.i = icmp slt i64 %t0, 0
   tail call void @float_call_signbit_callee(i1 zeroext %tobool.i.i.i.i)
   ret void
 }
 declare void @float_call_signbit_callee(i1 zeroext)

llvm/test/CodeGen/X86/not-and-simplify.ll

; ALL-NEXT:    retq
   %not = xor i32 %sh, -1
   %and = and i32 %not, 1
   ret i32 %and
 }
 
 define <4 x i32> @shrink_xor_constant1_splat(<4 x i32> %x) {
 ; ALL-LABEL: shrink_xor_constant1_splat:
 ; ALL:       # %bb.0:
+; ALL-NEXT:    pcmpeqd %xmm1, %xmm1
+; ALL-NEXT:    pxor %xmm1, %xmm0
 ; ALL-NEXT:    psrld $31, %xmm0
-; ALL-NEXT:    pxor {{.*}}(%rip), %xmm0
 ; ALL-NEXT:    retq
   %sh = lshr <4 x i32> %x, <i32 31, i32 31, i32 31, i32 31>
   %not = xor <4 x i32> %sh, <i32 -1, i32 -1, i32 -1, i32 -1>
   %and = and <4 x i32> %not, <i32 1, i32 1, i32 1, i32 1>
   ret <4 x i32> %and
 }
 
 ; Clear low bits via shift, set them with xor (not), then mask them off.

llvm/test/CodeGen/X86/pr28504.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck %s
 
 ; The test case is rather involved, because we need to get to a state where
 ; We have a sext(setcc x, y, cc) -> (select (setcc x, y, cc), T, 0) combine,
 ; BUT this combine is only triggered post-legalization, so the setcc's return
 ; type is i8. So we can't have the combine opportunity be exposed too early.
 ; Basically, what we want to see is that the compare result zero-extended, and 
 ; then stored. Only one zext, and no sexts.
 
-; CHECK-LABEL: main:
-; CHECK: movzbl (%rdi), %[[EAX:.*]]
-; CHECK-NEXT: xorl  %e[[C:.]]x, %e[[C]]x
-; CHECK-NEXT: cmpl  $1, %[[EAX]]
-; CHECK-NEXT: sete  %[[C]]l
-; CHECK-NEXT: movl  %e[[C]]x, (%rsi)
 define void @main(i8* %p, i32* %q) {
+; CHECK-LABEL: main:
+; CHECK:       # %bb.0: # %bb
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    cmpb $1, (%rdi)
+; CHECK-NEXT:    sete %al
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    je .LBB0_1
+; CHECK-NEXT:  # %bb.2: # %bb22
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  .LBB0_1: # %bb21
 bb:
   %tmp4 = load i8, i8* %p, align 1
   %tmp5 = sext i8 %tmp4 to i32
   %tmp6 = load i8, i8* %p, align 1
   %tmp7 = zext i8 %tmp6 to i32
   %tmp8 = sub nsw i32 %tmp5, %tmp7
   %tmp11 = icmp eq i32 %tmp7, 1
   %tmp12 = zext i1 %tmp11 to i32

llvm/test/CodeGen/X86/pr33844.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -o - %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @global = external global i32
 @global.1 = external global i64
 
 define void @patatino() {
 ; CHECK-LABEL: patatino:
 ; CHECK:       # %bb.0: # %bb
-; CHECK-NEXT:    movl {{.*}}(%rip), %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    shrl $31, %ecx
-; CHECK-NEXT:    addl $2147483647, %ecx # imm = 0x7FFFFFFF
-; CHECK-NEXT:    shrl $31, %ecx
-; CHECK-NEXT:    andl $-2, %ecx
-; CHECK-NEXT:    andl $-536870912, %eax # imm = 0xE0000000
-; CHECK-NEXT:    orl %ecx, %eax
-; CHECK-NEXT:    movl %eax, {{.*}}(%rip)
+; CHECK-NEXT:    andl $-536870912, {{.*}}(%rip) # imm = 0xE0000000
 ; CHECK-NEXT:    retq
 bb:
   %tmp = load i32, i32* @global
   %tmp1 = lshr i32 %tmp, 31
   %tmp2 = add nuw nsw i32 %tmp1, 2147483647
   %tmp3 = load i64, i64* @global.1
   %tmp4 = shl i64 %tmp3, 23
   %tmp5 = add nsw i64 %tmp4, 8388639

llvm/test/CodeGen/X86/sse3.ll

entry:
   %tmp9 = shufflevector <16 x i8> %tmp8, <16 x i8> %T0,  <16 x i32> < i32 0, i32 1, i32 2, i32 17,  i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef , i32 undef >
   ret <16 x i8> %tmp9
 }
 
 ; rdar://8520311
 define <4 x i32> @t17() nounwind {
 ; X86-LABEL: t17:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movddup {{.*#+}} xmm0 = mem[0,0]
-; X86-NEXT:    andpd {{\.LCPI.*}}, %xmm0
+; X86-NEXT:    pshufd {{.*#+}} xmm0 = mem[0,1,0,1]
+; X86-NEXT:    pand {{\.LCPI.*}}, %xmm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: t17:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movddup {{.*#+}} xmm0 = mem[0,0]
-; X64-NEXT:    andpd {{.*}}(%rip), %xmm0
+; X64-NEXT:    pshufd {{.*#+}} xmm0 = mem[0,1,0,1]
+; X64-NEXT:    pand {{.*}}(%rip), %xmm0
 ; X64-NEXT:    retq
 entry:
   %tmp1 = load <4 x float>, <4 x float>* undef, align 16
   %tmp2 = shufflevector <4 x float> %tmp1, <4 x float> undef, <4 x i32> <i32 4, i32 1, i32 2, i32 3>
   %tmp3 = load <4 x float>, <4 x float>* undef, align 16
   %tmp4 = shufflevector <4 x float> %tmp2, <4 x float> undef, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
   %tmp5 = bitcast <4 x float> %tmp3 to <4 x i32>
   %tmp6 = shufflevector <4 x i32> %tmp5, <4 x i32> undef, <4 x i32> <i32 undef, i32 undef, i32 0, i32 1>
   %tmp7 = and <4 x i32> %tmp6, <i32 undef, i32 undef, i32 -1, i32 0>
   ret <4 x i32> %tmp7
 }

llvm/test/CodeGen/X86/vec_extract-mmx.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown -mattr=+mmx,+sse2 | FileCheck %s --check-prefix=X32
 ; RUN: llc < %s -mtriple=x86_64-unknown -mattr=+mmx,+sse2 | FileCheck %s --check-prefix=X64
 
 define i32 @test0(<1 x i64>* %v4) nounwind {
 ; X32-LABEL: test0:
 ; X32:       # %bb.0: # %entry
-; X32-NEXT:    pushl %ebp
-; X32-NEXT:    movl %esp, %ebp
-; X32-NEXT:    andl $-8, %esp
-; X32-NEXT:    subl $8, %esp
-; X32-NEXT:    movl 8(%ebp), %eax
-; X32-NEXT:    movl (%eax), %ecx
-; X32-NEXT:    movl 4(%eax), %eax
-; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp)
-; X32-NEXT:    movl %ecx, (%esp)
-; X32-NEXT:    pshufw $238, (%esp), %mm0 # mm0 = mem[2,3,2,3]
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-NEXT:    pshufw $238, (%eax), %mm0 # mm0 = mem[2,3,2,3]
 ; X32-NEXT:    movd %mm0, %eax
 ; X32-NEXT:    addl $32, %eax
-; X32-NEXT:    movl %ebp, %esp
-; X32-NEXT:    popl %ebp
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test0:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    pshufw $238, (%rdi), %mm0 # mm0 = mem[2,3,2,3]
 ; X64-NEXT:    movd %mm0, %eax
 ; X64-NEXT:    addl $32, %eax
 ; X64-NEXT:    retq