[AMDGPU] Improve codegen of extractelement/insertelement in some cases

[jpages]

Summary

This patch improves the codegen of extractelement and insertelement for vector containing 8 elements.
Before, a dag combine transformation was generating a sequence of 8 select/cmp.
This patch changes the upper limit for this transformation and the movrel
instruction will eventually be used instead.
Extractlement/insertelement for vectors containing less than 8 elements are unchanged.

This patch is saving some instructions in these cases.
This case was identified in several game shaders.

[jpages]

Updated a testcase

[rampitec]

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

[foad]

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

[jpages]

In D126389#3539881, @foad wrote:

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

I will try to get some performance numbers on specific games. Do you know if this performance problem was specific to an architecture?
Like Jay said, I could tweak the heuristic for this and only generate it for GFX10+.

[rampitec]

In D126389#3540098, @jpages wrote:

In D126389#3539881, @foad wrote:

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

I will try to get some performance numbers on specific games. Do you know if this performance problem was specific to an architecture?
Like Jay said, I could tweak the heuristic for this and only generate it for GFX10+.

Real cases that was tested with were divergent, so it was v_movrel_b32. A performance of s_movrel_b32 could be different. Unfortunately we do not know here if that is divergent or not.

[rampitec]

In D126389#3540098, @jpages wrote:

In D126389#3539881, @foad wrote:

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

I will try to get some performance numbers on specific games. Do you know if this performance problem was specific to an architecture?
Like Jay said, I could tweak the heuristic for this and only generate it for GFX10+.

All the tests were done on gfx9 flavors. It is also worth noting gfx9 does not have movrel, but uses s_set_gpr_idx_on instead, which may be the difference.

I.e. it may be linked to FeatureMovrel.

[jpages]

In D126389#3540486, @rampitec wrote:

In D126389#3540098, @jpages wrote:

In D126389#3539881, @foad wrote:

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

I will try to get some performance numbers on specific games. Do you know if this performance problem was specific to an architecture?
Like Jay said, I could tweak the heuristic for this and only generate it for GFX10+.

All the tests were done on gfx9 flavors. It is also worth noting gfx9 does not have movrel, but uses s_set_gpr_idx_on instead, which may be the difference.

I.e. it may be linked to FeatureMovrel.

I measured the performance with/without the patch on the original game in which we spotted this issue.
It's a small improvement of about 0.5% in the current version compared to without the patch. This was measured on GFX1030.

It seems to be a little bit faster with the patch, I can't really test on gfx9 with this game.

If it's better to keep the initial version for gfx9, I could:

• Only do this patch for GFX10+
• Some flag?

[piotr]

In D126389#3546536, @jpages wrote:

In D126389#3540486, @rampitec wrote:

In D126389#3540098, @jpages wrote:

In D126389#3539881, @foad wrote:

In D126389#3538346, @rampitec wrote:

Any performance numbers? The 8 element case was driven by a specific customer program and the performance of the cmp/select was better than movrel.

I don't know why that would be. Maybe the performance characteristics are different on GFX10+ compared to GFX9.

Also on GFX10+ sgpr usage does not affect occupancy, so perhaps the heuristic could be tweaked to make it more likely to use s_movrel (not v_movrel) on GFX10+.

I will try to get some performance numbers on specific games. Do you know if this performance problem was specific to an architecture?
Like Jay said, I could tweak the heuristic for this and only generate it for GFX10+.

All the tests were done on gfx9 flavors. It is also worth noting gfx9 does not have movrel, but uses s_set_gpr_idx_on instead, which may be the difference.

I.e. it may be linked to FeatureMovrel.

I measured the performance with/without the patch on the original game in which we spotted this issue.
It's a small improvement of about 0.5% in the current version compared to without the patch. This was measured on GFX1030.

It seems to be a little bit faster with the patch, I can't really test on gfx9 with this game.

If it's better to keep the initial version for gfx9, I could:

• Only do this patch for GFX10+
• Some flag?

Since the current heuristic was driven by gfx9 that doesn't have movrel, Stas's suggestion to make it conditional on FeatureMovrel makes perfect sense to me.

[jpages]

Rebased, keep the current version for GFX9 and only apply this patch on architectures with movrel.

Thanks for the comments!

[arsenm]

I wouldn’t really expect there to be a performance difference between gpr indexing and movrel requences. Gfx8 has both so you could directly compare there

[jpages]

In D126389#3549129, @arsenm wrote:

I wouldn’t really expect there to be a performance difference between gpr indexing and movrel requences. Gfx8 has both so you could directly compare there

I'm using a game for my benchmarks and unfortunately this is a ray-tracing title, I can't really test it on GFX8 :(
These tests were performed on GFX1030 for this reason.

If you still have access to the benchmarks done on GFX8, I could try to test again with this patch.

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

bool AMDGPURegisterBankInfo::foldExtractEltToCmpSelect(
 
   bool IsDivergentIdx = IdxBank != AMDGPU::SGPRRegBank;
 
   LLT VecTy = MRI.getType(VecReg);
   unsigned EltSize = VecTy.getScalarSizeInBits();
   unsigned NumElem = VecTy.getNumElements();
 
   if (!SITargetLowering::shouldExpandVectorDynExt(EltSize, NumElem,
-                                                  IsDivergentIdx))
+                                                  IsDivergentIdx, &Subtarget))
     return false;
 
   MachineIRBuilder B(MI);
   LLT S32 = LLT::scalar(32);
 
   const RegisterBank &DstBank =
     *OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
   const RegisterBank &SrcBank =
bool AMDGPURegisterBankInfo::foldInsertEltToCmpSelect(
 
   bool IsDivergentIdx = IdxBank != AMDGPU::SGPRRegBank;
 
   LLT VecTy = MRI.getType(VecReg);
   unsigned EltSize = VecTy.getScalarSizeInBits();
   unsigned NumElem = VecTy.getNumElements();
 
   if (!SITargetLowering::shouldExpandVectorDynExt(EltSize, NumElem,
-                                                  IsDivergentIdx))
+                                                  IsDivergentIdx, &Subtarget))
     return false;
 
   MachineIRBuilder B(MI);
   LLT S32 = LLT::scalar(32);
 
   const RegisterBank &DstBank =
     *OpdMapper.getInstrMapping().getOperandMapping(0).BreakDown[0].RegBank;
   const RegisterBank &SrcBank =

llvm/lib/Target/AMDGPU/SIISelLowering.h

public:
 
   /// \returns true if this should use a literal constant for an LDS address,
   /// and not emit a relocation for an LDS global.
   bool shouldUseLDSConstAddress(const GlobalValue *GV) const;
 
   /// Check if EXTRACT_VECTOR_ELT/INSERT_VECTOR_ELT (<n x e>, var-idx) should be
   /// expanded into a set of cmp/select instructions.
   static bool shouldExpandVectorDynExt(unsigned EltSize, unsigned NumElem,
-                                       bool IsDivergentIdx);
+                                       bool IsDivergentIdx,
+                                       const GCNSubtarget *Subtarget);
+
+  bool shouldExpandVectorDynExt(SDNode *N) const;
 
 private:
   // Analyze a combined offset from an amdgcn_buffer_ intrinsic and store the
   // three offsets (voffset, soffset and instoffset) into the SDValue[3] array
   // pointed to by Offsets.
   void setBufferOffsets(SDValue CombinedOffset, SelectionDAG &DAG,
                         SDValue *Offsets, Align Alignment = Align(4)) const;
 

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

if (Src0.isUndef() && Src1.isUndef())
     return DCI.DAG.getUNDEF(N->getValueType(0));
   return SDValue();
 }
 
 // Check if EXTRACT_VECTOR_ELT/INSERT_VECTOR_ELT (<n x e>, var-idx) should be
 // expanded into a set of cmp/select instructions.
 bool SITargetLowering::shouldExpandVectorDynExt(unsigned EltSize,
                                                 unsigned NumElem,
-                                                bool IsDivergentIdx) {
+                                                bool IsDivergentIdx,
+                                                const GCNSubtarget *Subtarget) {
   if (UseDivergentRegisterIndexing)
     return false;
 
   unsigned VecSize = EltSize * NumElem;
 
   // Sub-dword vectors of size 2 dword or less have better implementation.
   if (VecSize <= 64 && EltSize < 32)
     return false;
 
   // Always expand the rest of sub-dword instructions, otherwise it will be
   // lowered via memory.
   if (EltSize < 32)
     return true;
 
   // Always do this if var-idx is divergent, otherwise it will become a loop.
   if (IsDivergentIdx)
     return true;
 
   // Large vectors would yield too many compares and v_cndmask_b32 instructions.
   unsigned NumInsts = NumElem /* Number of compares */ +
                       ((EltSize + 31) / 32) * NumElem /* Number of cndmasks */;
+
+  // On some architectures (GFX9) movrel is not available and it's better
+  // to expand.
+  if (!Subtarget->hasMovrel())
     return NumInsts <= 16;
+
+  // If movrel is available, use it instead of expanding for vector of 8
+  // elements.
+  return NumInsts <= 15;
 }
 
-static bool shouldExpandVectorDynExt(SDNode *N) {
+bool SITargetLowering::shouldExpandVectorDynExt(SDNode *N) const {
   SDValue Idx = N->getOperand(N->getNumOperands() - 1);
   if (isa<ConstantSDNode>(Idx))
     return false;
 
   SDValue Vec = N->getOperand(0);
   EVT VecVT = Vec.getValueType();
   EVT EltVT = VecVT.getVectorElementType();
   unsigned EltSize = EltVT.getSizeInBits();
   unsigned NumElem = VecVT.getVectorNumElements();
 
-  return SITargetLowering::shouldExpandVectorDynExt(EltSize, NumElem,
-                                                    Idx->isDivergent());
+  return SITargetLowering::shouldExpandVectorDynExt(
+      EltSize, NumElem, Idx->isDivergent(), getSubtarget());
 }
 
 SDValue SITargetLowering::performExtractVectorEltCombine(
   SDNode *N, DAGCombinerInfo &DCI) const {
   SDValue Vec = N->getOperand(0);
   SelectionDAG &DAG = DCI.DAG;
 
   EVT VecVT = Vec.getValueType();
if (Vec.hasOneUse() && DCI.isBeforeLegalize()) {
     }
     }
   }
 
   unsigned VecSize = VecVT.getSizeInBits();
   unsigned EltSize = EltVT.getSizeInBits();
 
   // EXTRACT_VECTOR_ELT (<n x e>, var-idx) => n x select (e, const-idx)
-  if (::shouldExpandVectorDynExt(N)) {
+  if (shouldExpandVectorDynExt(N)) {
     SDLoc SL(N);
     SDValue Idx = N->getOperand(1);
     SDValue V;
     for (unsigned I = 0, E = VecVT.getVectorNumElements(); I < E; ++I) {
       SDValue IC = DAG.getVectorIdxConstant(I, SL);
       SDValue Elt = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, SL, EltVT, Vec, IC);
       if (I == 0)
         V = Elt;
SITargetLowering::performInsertVectorEltCombine(SDNode *N,
                                                 DAGCombinerInfo &DCI) const {
   SDValue Vec = N->getOperand(0);
   SDValue Idx = N->getOperand(2);
   EVT VecVT = Vec.getValueType();
   EVT EltVT = VecVT.getVectorElementType();
 
   // INSERT_VECTOR_ELT (<n x e>, var-idx)
   // => BUILD_VECTOR n x select (e, const-idx)
-  if (!::shouldExpandVectorDynExt(N))
+  if (!shouldExpandVectorDynExt(N))
     return SDValue();
 
   SelectionDAG &DAG = DCI.DAG;
   SDLoc SL(N);
   SDValue Ins = N->getOperand(1);
   EVT IdxVT = Idx.getValueType();
 
   SmallVector<SDValue, 16> Ops;

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll

 ; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, 0x41000000, vcc_lo
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 entry:
   %ext = extractelement <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, i32 %sel
   ret float %ext
 }
 
 define amdgpu_ps float @dyn_extract_v8f32_const_s_s(i32 inreg %sel) {
-; GCN-LABEL: dyn_extract_v8f32_const_s_s:
-; GCN:       ; %bb.0: ; %entry
-; GCN-NEXT:    s_cmp_eq_u32 s2, 1
-; GCN-NEXT:    s_cselect_b32 s0, 2.0, 1.0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 2
-; GCN-NEXT:    s_cselect_b32 s0, 0x40400000, s0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 3
-; GCN-NEXT:    s_cselect_b32 s0, 4.0, s0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 4
-; GCN-NEXT:    s_cselect_b32 s0, 0x40a00000, s0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 5
-; GCN-NEXT:    s_cselect_b32 s0, 0x40c00000, s0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 6
-; GCN-NEXT:    s_cselect_b32 s0, 0x40e00000, s0
-; GCN-NEXT:    s_cmp_eq_u32 s2, 7
-; GCN-NEXT:    s_cselect_b32 s0, 0x41000000, s0
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
-; GCN-NEXT:    ; return to shader part epilog
+; GPRIDX-LABEL: dyn_extract_v8f32_const_s_s:
+; GPRIDX:       ; %bb.0: ; %entry
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 1
+; GPRIDX-NEXT:    s_cselect_b32 s0, 2.0, 1.0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 2
+; GPRIDX-NEXT:    s_cselect_b32 s0, 0x40400000, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 3
+; GPRIDX-NEXT:    s_cselect_b32 s0, 4.0, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 4
+; GPRIDX-NEXT:    s_cselect_b32 s0, 0x40a00000, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 5
+; GPRIDX-NEXT:    s_cselect_b32 s0, 0x40c00000, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 6
+; GPRIDX-NEXT:    s_cselect_b32 s0, 0x40e00000, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s2, 7
+; GPRIDX-NEXT:    s_cselect_b32 s0, 0x41000000, s0
+; GPRIDX-NEXT:    v_mov_b32_e32 v0, s0
+; GPRIDX-NEXT:    ; return to shader part epilog
+;
+; MOVREL-LABEL: dyn_extract_v8f32_const_s_s:
+; MOVREL:       ; %bb.0: ; %entry
+; MOVREL-NEXT:    s_mov_b32 s4, 1.0
+; MOVREL-NEXT:    s_mov_b32 m0, s2
+; MOVREL-NEXT:    s_mov_b32 s11, 0x41000000
+; MOVREL-NEXT:    s_mov_b32 s10, 0x40e00000
+; MOVREL-NEXT:    s_mov_b32 s9, 0x40c00000
+; MOVREL-NEXT:    s_mov_b32 s8, 0x40a00000
+; MOVREL-NEXT:    s_mov_b32 s7, 4.0
+; MOVREL-NEXT:    s_mov_b32 s6, 0x40400000
+; MOVREL-NEXT:    s_mov_b32 s5, 2.0
+; MOVREL-NEXT:    s_movrels_b32 s0, s4
+; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
+; MOVREL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: dyn_extract_v8f32_const_s_s:
 ; GFX10:       ; %bb.0: ; %entry
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 1
-; GFX10-NEXT:    s_cselect_b32 s0, 2.0, 1.0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 2
-; GFX10-NEXT:    s_cselect_b32 s0, 0x40400000, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 3
-; GFX10-NEXT:    s_cselect_b32 s0, 4.0, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 4
-; GFX10-NEXT:    s_cselect_b32 s0, 0x40a00000, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 5
-; GFX10-NEXT:    s_cselect_b32 s0, 0x40c00000, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 6
-; GFX10-NEXT:    s_cselect_b32 s0, 0x40e00000, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 7
-; GFX10-NEXT:    s_cselect_b32 s0, 0x41000000, s0
+; GFX10-NEXT:    s_mov_b32 s4, 1.0
+; GFX10-NEXT:    s_mov_b32 m0, s2
+; GFX10-NEXT:    s_mov_b32 s11, 0x41000000
+; GFX10-NEXT:    s_mov_b32 s10, 0x40e00000
+; GFX10-NEXT:    s_mov_b32 s9, 0x40c00000
+; GFX10-NEXT:    s_mov_b32 s8, 0x40a00000
+; GFX10-NEXT:    s_mov_b32 s7, 4.0
+; GFX10-NEXT:    s_mov_b32 s6, 0x40400000
+; GFX10-NEXT:    s_mov_b32 s5, 2.0
+; GFX10-NEXT:    s_movrels_b32 s0, s4
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX10-NEXT:    ; return to shader part epilog
 entry:
   %ext = extractelement <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, i32 %sel
   ret float %ext
 }
 
 define amdgpu_ps float @dyn_extract_v8f32_s_v(<8 x float> inreg %vec, i32 %sel) {

 ; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc_lo
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 entry:
   %ext = extractelement <8 x float> %vec, i32 %sel
   ret float %ext
 }
 
 define amdgpu_ps float @dyn_extract_v8f32_v_s(<8 x float> %vec, i32 inreg %sel) {
-; GCN-LABEL: dyn_extract_v8f32_v_s:
-; GCN:       ; %bb.0: ; %entry
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 1
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 2
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 3
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 4
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 5
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
-; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc
-; GCN-NEXT:    ; return to shader part epilog
+; GPRIDX-LABEL: dyn_extract_v8f32_v_s:
+; GPRIDX:       ; %bb.0: ; %entry
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 1
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 2
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 3
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 4
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 5
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
+; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
+; GPRIDX-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc
+; GPRIDX-NEXT:    ; return to shader part epilog
+;
+; MOVREL-LABEL: dyn_extract_v8f32_v_s:
+; MOVREL:       ; %bb.0: ; %entry
+; MOVREL-NEXT:    s_mov_b32 m0, s2
+; MOVREL-NEXT:    v_movrels_b32_e32 v0, v0
+; MOVREL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: dyn_extract_v8f32_v_s:
 ; GFX10:       ; %bb.0: ; %entry
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 2
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 3
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 4
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 5
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 6
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 7
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc_lo
+; GFX10-NEXT:    s_mov_b32 m0, s2
+; GFX10-NEXT:    v_movrels_b32_e32 v0, v0
 ; GFX10-NEXT:    ; return to shader part epilog
 entry:
   %ext = extractelement <8 x float> %vec, i32 %sel
   ret float %ext
 }
 
 define amdgpu_ps float @dyn_extract_v8f32_s_s(<8 x float> inreg %vec, i32 inreg %sel) {
-; GCN-LABEL: dyn_extract_v8f32_s_s:
-; GCN:       ; %bb.0: ; %entry
-; GCN-NEXT:    s_cmp_eq_u32 s10, 1
-; GCN-NEXT:    s_cselect_b32 s0, s3, s2
-; GCN-NEXT:    s_cmp_eq_u32 s10, 2
-; GCN-NEXT:    s_cselect_b32 s0, s4, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 3
-; GCN-NEXT:    s_cselect_b32 s0, s5, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 4
-; GCN-NEXT:    s_cselect_b32 s0, s6, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 5
-; GCN-NEXT:    s_cselect_b32 s0, s7, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 6
-; GCN-NEXT:    s_cselect_b32 s0, s8, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 7
-; GCN-NEXT:    s_cselect_b32 s0, s9, s0
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
-; GCN-NEXT:    ; return to shader part epilog
+; GPRIDX-LABEL: dyn_extract_v8f32_s_s:
+; GPRIDX:       ; %bb.0: ; %entry
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 1
+; GPRIDX-NEXT:    s_cselect_b32 s0, s3, s2
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 2
+; GPRIDX-NEXT:    s_cselect_b32 s0, s4, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 3
+; GPRIDX-NEXT:    s_cselect_b32 s0, s5, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 4
+; GPRIDX-NEXT:    s_cselect_b32 s0, s6, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 5
+; GPRIDX-NEXT:    s_cselect_b32 s0, s7, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 6
+; GPRIDX-NEXT:    s_cselect_b32 s0, s8, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 7
+; GPRIDX-NEXT:    s_cselect_b32 s0, s9, s0
+; GPRIDX-NEXT:    v_mov_b32_e32 v0, s0
+; GPRIDX-NEXT:    ; return to shader part epilog
+;
+; MOVREL-LABEL: dyn_extract_v8f32_s_s:
+; MOVREL:       ; %bb.0: ; %entry
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 m0, s10
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movrels_b32 s0, s0
+; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
+; MOVREL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: dyn_extract_v8f32_s_s:
 ; GFX10:       ; %bb.0: ; %entry
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 1
-; GFX10-NEXT:    s_cselect_b32 s0, s3, s2
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 2
-; GFX10-NEXT:    s_cselect_b32 s0, s4, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 3
-; GFX10-NEXT:    s_cselect_b32 s0, s5, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 4
-; GFX10-NEXT:    s_cselect_b32 s0, s6, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 5
-; GFX10-NEXT:    s_cselect_b32 s0, s7, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 6
-; GFX10-NEXT:    s_cselect_b32 s0, s8, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 7
-; GFX10-NEXT:    s_cselect_b32 s0, s9, s0
+; GFX10-NEXT:    s_mov_b32 s0, s2
+; GFX10-NEXT:    s_mov_b32 m0, s10
+; GFX10-NEXT:    s_mov_b32 s1, s3
+; GFX10-NEXT:    s_mov_b32 s2, s4
+; GFX10-NEXT:    s_mov_b32 s3, s5
+; GFX10-NEXT:    s_mov_b32 s4, s6
+; GFX10-NEXT:    s_mov_b32 s5, s7
+; GFX10-NEXT:    s_mov_b32 s6, s8
+; GFX10-NEXT:    s_mov_b32 s7, s9
+; GFX10-NEXT:    s_movrels_b32 s0, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX10-NEXT:    ; return to shader part epilog
 entry:
   %ext = extractelement <8 x float> %vec, i32 %sel
   ret float %ext
 }
 
 define i64 @dyn_extract_v8i64_const_s_v(i32 %sel) {

 ; GFX10-NEXT:    s_endpgm
 entry:
   %ext = extractelement <8 x i64> %vec, i32 %sel
   store i64 %ext, i64 addrspace(1)* undef
   ret void
 }
 
 define amdgpu_ps float @dyn_extract_v8f32_s_s_offset3(<8 x float> inreg %vec, i32 inreg %sel) {
-; GCN-LABEL: dyn_extract_v8f32_s_s_offset3:
-; GCN:       ; %bb.0: ; %entry
-; GCN-NEXT:    s_add_i32 s10, s10, 3
-; GCN-NEXT:    s_cmp_eq_u32 s10, 1
-; GCN-NEXT:    s_cselect_b32 s0, s3, s2
-; GCN-NEXT:    s_cmp_eq_u32 s10, 2
-; GCN-NEXT:    s_cselect_b32 s0, s4, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 3
-; GCN-NEXT:    s_cselect_b32 s0, s5, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 4
-; GCN-NEXT:    s_cselect_b32 s0, s6, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 5
-; GCN-NEXT:    s_cselect_b32 s0, s7, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 6
-; GCN-NEXT:    s_cselect_b32 s0, s8, s0
-; GCN-NEXT:    s_cmp_eq_u32 s10, 7
-; GCN-NEXT:    s_cselect_b32 s0, s9, s0
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
-; GCN-NEXT:    ; return to shader part epilog
+; GPRIDX-LABEL: dyn_extract_v8f32_s_s_offset3:
+; GPRIDX:       ; %bb.0: ; %entry
+; GPRIDX-NEXT:    s_add_i32 s10, s10, 3
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 1
+; GPRIDX-NEXT:    s_cselect_b32 s0, s3, s2
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 2
+; GPRIDX-NEXT:    s_cselect_b32 s0, s4, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 3
+; GPRIDX-NEXT:    s_cselect_b32 s0, s5, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 4
+; GPRIDX-NEXT:    s_cselect_b32 s0, s6, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 5
+; GPRIDX-NEXT:    s_cselect_b32 s0, s7, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 6
+; GPRIDX-NEXT:    s_cselect_b32 s0, s8, s0
+; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 7
+; GPRIDX-NEXT:    s_cselect_b32 s0, s9, s0
+; GPRIDX-NEXT:    v_mov_b32_e32 v0, s0
+; GPRIDX-NEXT:    ; return to shader part epilog
+;
+; MOVREL-LABEL: dyn_extract_v8f32_s_s_offset3:
+; MOVREL:       ; %bb.0: ; %entry
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 m0, s10
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movrels_b32 s0, s3
+; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
+; MOVREL-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: dyn_extract_v8f32_s_s_offset3:
 ; GFX10:       ; %bb.0: ; %entry
-; GFX10-NEXT:    s_add_i32 s10, s10, 3
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 1
-; GFX10-NEXT:    s_cselect_b32 s0, s3, s2
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 2
-; GFX10-NEXT:    s_cselect_b32 s0, s4, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 3
-; GFX10-NEXT:    s_cselect_b32 s0, s5, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 4
-; GFX10-NEXT:    s_cselect_b32 s0, s6, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 5
-; GFX10-NEXT:    s_cselect_b32 s0, s7, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 6
-; GFX10-NEXT:    s_cselect_b32 s0, s8, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 7
-; GFX10-NEXT:    s_cselect_b32 s0, s9, s0
+; GFX10-NEXT:    s_mov_b32 s1, s3
+; GFX10-NEXT:    s_mov_b32 s3, s5
+; GFX10-NEXT:    s_mov_b32 m0, s10
+; GFX10-NEXT:    s_mov_b32 s0, s2
+; GFX10-NEXT:    s_mov_b32 s2, s4
+; GFX10-NEXT:    s_mov_b32 s4, s6
+; GFX10-NEXT:    s_mov_b32 s5, s7
+; GFX10-NEXT:    s_mov_b32 s6, s8
+; GFX10-NEXT:    s_mov_b32 s7, s9
+; GFX10-NEXT:    s_movrels_b32 s0, s3
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX10-NEXT:    ; return to shader part epilog
 entry:
   %add = add i32 %sel, 3
   %ext = extractelement <8 x float> %vec, i32 %add
   ret float %ext
 }
 

 ; GPRIDX-NEXT:    s_cmp_eq_u32 s10, 7
 ; GPRIDX-NEXT:    s_cselect_b32 s0, s9, s0
 ; GPRIDX-NEXT:    v_mov_b32_e32 v0, s0
 ; GPRIDX-NEXT:    ds_write_b32 v0, v0
 ; GPRIDX-NEXT:    s_endpgm
 ;
 ; MOVREL-LABEL: dyn_extract_v8p3_s_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 1
-; MOVREL-NEXT:    s_cselect_b32 s0, s3, s2
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 2
-; MOVREL-NEXT:    s_cselect_b32 s0, s4, s0
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 3
-; MOVREL-NEXT:    s_cselect_b32 s0, s5, s0
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 4
-; MOVREL-NEXT:    s_cselect_b32 s0, s6, s0
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 5
-; MOVREL-NEXT:    s_cselect_b32 s0, s7, s0
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 6
-; MOVREL-NEXT:    s_cselect_b32 s0, s8, s0
-; MOVREL-NEXT:    s_cmp_eq_u32 s10, 7
-; MOVREL-NEXT:    s_cselect_b32 s0, s9, s0
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 m0, s10
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movrels_b32 s0, s0
 ; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
 ; MOVREL-NEXT:    s_mov_b32 m0, -1
 ; MOVREL-NEXT:    ds_write_b32 v0, v0
 ; MOVREL-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: dyn_extract_v8p3_s_s:
 ; GFX10:       ; %bb.0: ; %entry
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 1
-; GFX10-NEXT:    s_cselect_b32 s0, s3, s2
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 2
-; GFX10-NEXT:    s_cselect_b32 s0, s4, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 3
-; GFX10-NEXT:    s_cselect_b32 s0, s5, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 4
-; GFX10-NEXT:    s_cselect_b32 s0, s6, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 5
-; GFX10-NEXT:    s_cselect_b32 s0, s7, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 6
-; GFX10-NEXT:    s_cselect_b32 s0, s8, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s10, 7
-; GFX10-NEXT:    s_cselect_b32 s0, s9, s0
+; GFX10-NEXT:    s_mov_b32 s0, s2
+; GFX10-NEXT:    s_mov_b32 m0, s10
+; GFX10-NEXT:    s_mov_b32 s1, s3
+; GFX10-NEXT:    s_mov_b32 s2, s4
+; GFX10-NEXT:    s_mov_b32 s3, s5
+; GFX10-NEXT:    s_mov_b32 s4, s6
+; GFX10-NEXT:    s_mov_b32 s5, s7
+; GFX10-NEXT:    s_mov_b32 s6, s8
+; GFX10-NEXT:    s_mov_b32 s7, s9
+; GFX10-NEXT:    s_movrels_b32 s0, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX10-NEXT:    ds_write_b32 v0, v0
 ; GFX10-NEXT:    s_endpgm
 entry:
   %ext = extractelement <8 x i8 addrspace(3)*> %vec, i32 %idx
   store i8 addrspace(3)* %ext, i8 addrspace(3)* addrspace(3)* undef
   ret void
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement.i16.ll

 ; GFX9-NEXT:    v_mov_b32_e32 v2, s6
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s7
 ; GFX9-NEXT:    global_store_dwordx4 v[4:5], v[0:3], off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: insertelement_s_v16i16_s_s:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX8-NEXT:    s_lshr_b32 s7, s5, 1
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 1
-; GFX8-NEXT:    v_mov_b32_e32 v4, 0
-; GFX8-NEXT:    v_mov_b32_e32 v5, 0
-; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_cselect_b32 s0, s9, s8
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX8-NEXT:    s_cselect_b32 s0, s10, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX8-NEXT:    s_cselect_b32 s0, s11, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX8-NEXT:    s_cselect_b32 s0, s12, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX8-NEXT:    s_cselect_b32 s0, s13, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX8-NEXT:    s_cselect_b32 s0, s14, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX8-NEXT:    s_cselect_b32 s0, s15, s0
 ; GFX8-NEXT:    s_and_b32 s1, s5, 1
+; GFX8-NEXT:    s_lshr_b32 m0, s5, 1
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, 4
 ; GFX8-NEXT:    s_and_b32 s2, s4, 0xffff
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_movrels_b32 s0, s8
 ; GFX8-NEXT:    s_lshl_b32 s2, s2, s1
 ; GFX8-NEXT:    s_lshl_b32 s1, 0xffff, s1
 ; GFX8-NEXT:    s_andn2_b32 s0, s0, s1
-; GFX8-NEXT:    s_or_b32 s16, s0, s2
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 0
-; GFX8-NEXT:    s_cselect_b32 s0, s16, s8
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 1
-; GFX8-NEXT:    s_cselect_b32 s1, s16, s9
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX8-NEXT:    s_cselect_b32 s2, s16, s10
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX8-NEXT:    s_cselect_b32 s3, s16, s11
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX8-NEXT:    s_cselect_b32 s4, s16, s12
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX8-NEXT:    s_cselect_b32 s5, s16, s13
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX8-NEXT:    v_mov_b32_e32 v0, s0
-; GFX8-NEXT:    s_cselect_b32 s6, s16, s14
-; GFX8-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX8-NEXT:    v_mov_b32_e32 v1, s1
-; GFX8-NEXT:    v_mov_b32_e32 v2, s2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s3
-; GFX8-NEXT:    s_cselect_b32 s7, s16, s15
+; GFX8-NEXT:    s_or_b32 s0, s0, s2
+; GFX8-NEXT:    s_movreld_b32 s8, s0
+; GFX8-NEXT:    v_mov_b32_e32 v4, 0
+; GFX8-NEXT:    v_mov_b32_e32 v0, s8
+; GFX8-NEXT:    v_mov_b32_e32 v5, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s9
+; GFX8-NEXT:    v_mov_b32_e32 v2, s10
+; GFX8-NEXT:    v_mov_b32_e32 v3, s11
 ; GFX8-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; GFX8-NEXT:    v_mov_b32_e32 v4, 16
-; GFX8-NEXT:    v_mov_b32_e32 v0, s4
+; GFX8-NEXT:    v_mov_b32_e32 v0, s12
 ; GFX8-NEXT:    v_mov_b32_e32 v5, 0
-; GFX8-NEXT:    v_mov_b32_e32 v1, s5
-; GFX8-NEXT:    v_mov_b32_e32 v2, s6
-; GFX8-NEXT:    v_mov_b32_e32 v3, s7
+; GFX8-NEXT:    v_mov_b32_e32 v1, s13
+; GFX8-NEXT:    v_mov_b32_e32 v2, s14
+; GFX8-NEXT:    v_mov_b32_e32 v3, s15
 ; GFX8-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX7-LABEL: insertelement_s_v16i16_s_s:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX7-NEXT:    s_lshr_b32 s7, s5, 1
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 1
-; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_cselect_b32 s0, s9, s8
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX7-NEXT:    s_cselect_b32 s0, s10, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX7-NEXT:    s_cselect_b32 s0, s11, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX7-NEXT:    s_cselect_b32 s0, s12, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX7-NEXT:    s_cselect_b32 s0, s13, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX7-NEXT:    s_cselect_b32 s0, s14, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX7-NEXT:    s_cselect_b32 s0, s15, s0
 ; GFX7-NEXT:    s_and_b32 s1, s5, 1
+; GFX7-NEXT:    s_lshr_b32 m0, s5, 1
 ; GFX7-NEXT:    s_lshl_b32 s1, s1, 4
 ; GFX7-NEXT:    s_and_b32 s2, s4, 0xffff
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_movrels_b32 s0, s8
 ; GFX7-NEXT:    s_lshl_b32 s2, s2, s1
 ; GFX7-NEXT:    s_lshl_b32 s1, 0xffff, s1
 ; GFX7-NEXT:    s_andn2_b32 s0, s0, s1
-; GFX7-NEXT:    s_or_b32 s16, s0, s2
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 0
-; GFX7-NEXT:    s_cselect_b32 s0, s16, s8
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 1
-; GFX7-NEXT:    s_cselect_b32 s1, s16, s9
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX7-NEXT:    s_cselect_b32 s2, s16, s10
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX7-NEXT:    s_cselect_b32 s3, s16, s11
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX7-NEXT:    s_cselect_b32 s4, s16, s12
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX7-NEXT:    s_cselect_b32 s5, s16, s13
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX7-NEXT:    v_mov_b32_e32 v0, s0
-; GFX7-NEXT:    s_cselect_b32 s6, s16, s14
-; GFX7-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX7-NEXT:    s_mov_b64 s[8:9], 0
-; GFX7-NEXT:    v_mov_b32_e32 v1, s1
-; GFX7-NEXT:    v_mov_b32_e32 v2, s2
-; GFX7-NEXT:    v_mov_b32_e32 v3, s3
-; GFX7-NEXT:    s_mov_b32 s10, -1
-; GFX7-NEXT:    s_mov_b32 s11, 0xf000
-; GFX7-NEXT:    s_cselect_b32 s7, s16, s15
-; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
-; GFX7-NEXT:    s_mov_b64 s[8:9], 16
-; GFX7-NEXT:    v_mov_b32_e32 v0, s4
-; GFX7-NEXT:    v_mov_b32_e32 v1, s5
-; GFX7-NEXT:    v_mov_b32_e32 v2, s6
-; GFX7-NEXT:    v_mov_b32_e32 v3, s7
-; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
+; GFX7-NEXT:    s_or_b32 s0, s0, s2
+; GFX7-NEXT:    s_movreld_b32 s8, s0
+; GFX7-NEXT:    v_mov_b32_e32 v0, s8
+; GFX7-NEXT:    s_mov_b64 s[0:1], 0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s9
+; GFX7-NEXT:    v_mov_b32_e32 v2, s10
+; GFX7-NEXT:    v_mov_b32_e32 v3, s11
+; GFX7-NEXT:    s_mov_b32 s2, -1
+; GFX7-NEXT:    s_mov_b32 s3, 0xf000
+; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
+; GFX7-NEXT:    s_mov_b64 s[0:1], 16
+; GFX7-NEXT:    v_mov_b32_e32 v0, s12
+; GFX7-NEXT:    v_mov_b32_e32 v1, s13
+; GFX7-NEXT:    v_mov_b32_e32 v2, s14
+; GFX7-NEXT:    v_mov_b32_e32 v3, s15
+; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: insertelement_s_v16i16_s_s:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX10-NEXT:    s_lshr_b32 s7, s5, 1
+; GFX10-NEXT:    s_and_b32 s0, s5, 1
+; GFX10-NEXT:    s_lshr_b32 m0, s5, 1
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
+; GFX10-NEXT:    s_and_b32 s1, s4, 0xffff
+; GFX10-NEXT:    s_lshl_b32 s2, 0xffff, s0
+; GFX10-NEXT:    s_lshl_b32 s0, s1, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v8, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 1
 ; GFX10-NEXT:    v_mov_b32_e32 v9, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX10-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    s_cselect_b32 s0, s9, s8
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX10-NEXT:    s_cselect_b32 s0, s10, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX10-NEXT:    s_cselect_b32 s0, s11, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX10-NEXT:    s_cselect_b32 s0, s12, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX10-NEXT:    s_cselect_b32 s0, s13, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX10-NEXT:    s_cselect_b32 s0, s14, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX10-NEXT:    s_cselect_b32 s0, s15, s0
-; GFX10-NEXT:    s_and_b32 s1, s5, 1
-; GFX10-NEXT:    s_and_b32 s2, s4, 0xffff
-; GFX10-NEXT:    s_lshl_b32 s1, s1, 4
-; GFX10-NEXT:    s_lshl_b32 s3, 0xffff, s1
-; GFX10-NEXT:    s_lshl_b32 s1, s2, s1
-; GFX10-NEXT:    s_andn2_b32 s0, s0, s3
-; GFX10-NEXT:    s_or_b32 s16, s0, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 0
-; GFX10-NEXT:    s_cselect_b32 s0, s16, s8
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 1
-; GFX10-NEXT:    s_cselect_b32 s1, s16, s9
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 2
-; GFX10-NEXT:    s_cselect_b32 s2, s16, s10
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 3
-; GFX10-NEXT:    s_cselect_b32 s3, s16, s11
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 4
-; GFX10-NEXT:    v_mov_b32_e32 v0, s0
-; GFX10-NEXT:    s_cselect_b32 s4, s16, s12
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 5
-; GFX10-NEXT:    v_mov_b32_e32 v1, s1
-; GFX10-NEXT:    s_cselect_b32 s5, s16, s13
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 6
-; GFX10-NEXT:    v_mov_b32_e32 v2, s2
-; GFX10-NEXT:    s_cselect_b32 s6, s16, s14
-; GFX10-NEXT:    s_cmp_eq_u32 s7, 7
-; GFX10-NEXT:    v_mov_b32_e32 v3, s3
-; GFX10-NEXT:    s_cselect_b32 s7, s16, s15
-; GFX10-NEXT:    v_mov_b32_e32 v4, s4
-; GFX10-NEXT:    v_mov_b32_e32 v5, s5
-; GFX10-NEXT:    v_mov_b32_e32 v6, s6
-; GFX10-NEXT:    v_mov_b32_e32 v7, s7
+; GFX10-NEXT:    s_movrels_b32 s3, s8
+; GFX10-NEXT:    s_andn2_b32 s1, s3, s2
+; GFX10-NEXT:    s_or_b32 s0, s1, s0
+; GFX10-NEXT:    s_movreld_b32 s8, s0
+; GFX10-NEXT:    v_mov_b32_e32 v0, s8
+; GFX10-NEXT:    v_mov_b32_e32 v1, s9
+; GFX10-NEXT:    v_mov_b32_e32 v2, s10
+; GFX10-NEXT:    v_mov_b32_e32 v3, s11
+; GFX10-NEXT:    v_mov_b32_e32 v4, s12
+; GFX10-NEXT:    v_mov_b32_e32 v5, s13
+; GFX10-NEXT:    v_mov_b32_e32 v6, s14
+; GFX10-NEXT:    v_mov_b32_e32 v7, s15
 ; GFX10-NEXT:    global_store_dwordx4 v[8:9], v[0:3], off
 ; GFX10-NEXT:    global_store_dwordx4 v[10:11], v[4:7], off
 ; GFX10-NEXT:    s_endpgm
   %vec = load <16 x i16>, <16 x i16> addrspace(4)* %ptr
   %insert = insertelement <16 x i16> %vec, i16 %val, i32 %idx
   store <16 x i16> %insert, <16 x i16> addrspace(1)* null
   ret void
 }

 ; GFX9-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX9-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX9-NEXT:    global_store_dwordx4 v[8:9], v[0:3], off
 ; GFX9-NEXT:    global_store_dwordx4 v[10:11], v[4:7], off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: insertelement_v_v16i16_s_s:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    flat_load_dwordx4 v[2:5], v[0:1]
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 16, v0
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT:    flat_load_dwordx4 v[6:9], v[0:1]
+; GFX8-NEXT:    v_add_u32_e32 v4, vcc, 16, v0
+; GFX8-NEXT:    v_addc_u32_e32 v5, vcc, 0, v1, vcc
+; GFX8-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GFX8-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
 ; GFX8-NEXT:    s_and_b32 s0, s3, 1
-; GFX8-NEXT:    s_lshr_b32 s12, s3, 1
 ; GFX8-NEXT:    s_and_b32 s1, s2, 0xffff
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX8-NEXT:    s_lshl_b32 s13, s1, s0
+; GFX8-NEXT:    s_lshr_b32 m0, s3, 1
+; GFX8-NEXT:    s_lshl_b32 s1, s1, s0
 ; GFX8-NEXT:    s_lshl_b32 s0, 0xffff, s0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s12, 1
-; GFX8-NEXT:    s_not_b32 s14, s0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[0:1], s12, 2
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[2:3], s12, 3
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[4:5], s12, 4
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[6:7], s12, 5
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[8:9], s12, 6
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[10:11], s12, 7
-; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v2, v3, vcc
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v4, s[0:1]
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v5, s[2:3]
-; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[4:5]
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v7, s[6:7]
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v8, s[8:9]
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v0, v9, s[10:11]
-; GFX8-NEXT:    v_and_b32_e32 v0, s14, v0
-; GFX8-NEXT:    v_or_b32_e32 v10, s13, v0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[12:13], s12, 0
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v2, v10, s[12:13]
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v3, v10, vcc
-; GFX8-NEXT:    v_cndmask_b32_e64 v2, v4, v10, s[0:1]
-; GFX8-NEXT:    v_cndmask_b32_e64 v3, v5, v10, s[2:3]
-; GFX8-NEXT:    v_cndmask_b32_e64 v4, v6, v10, s[4:5]
-; GFX8-NEXT:    v_cndmask_b32_e64 v5, v7, v10, s[6:7]
-; GFX8-NEXT:    v_cndmask_b32_e64 v6, v8, v10, s[8:9]
-; GFX8-NEXT:    v_cndmask_b32_e64 v7, v9, v10, s[10:11]
+; GFX8-NEXT:    s_not_b32 s0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v8, 0
 ; GFX8-NEXT:    v_mov_b32_e32 v9, 0
 ; GFX8-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX8-NEXT:    v_mov_b32_e32 v11, 0
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    v_movrels_b32_e32 v12, v0
+; GFX8-NEXT:    v_and_b32_e32 v12, s0, v12
+; GFX8-NEXT:    v_or_b32_e32 v12, s1, v12
+; GFX8-NEXT:    v_movreld_b32_e32 v0, v12
 ; GFX8-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
 ; GFX8-NEXT:    flat_store_dwordx4 v[10:11], v[4:7]
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX7-LABEL: insertelement_v_v16i16_s_s:
 ; GFX7:       ; %bb.0:
-; GFX7-NEXT:    s_mov_b32 s18, 0
-; GFX7-NEXT:    s_mov_b32 s19, 0xf000
-; GFX7-NEXT:    s_mov_b64 s[16:17], 0
-; GFX7-NEXT:    buffer_load_dwordx4 v[2:5], v[0:1], s[16:19], 0 addr64
-; GFX7-NEXT:    buffer_load_dwordx4 v[6:9], v[0:1], s[16:19], 0 addr64 offset:16
+; GFX7-NEXT:    s_mov_b32 s6, 0
+; GFX7-NEXT:    s_mov_b32 s7, 0xf000
+; GFX7-NEXT:    s_mov_b64 s[4:5], 0
+; GFX7-NEXT:    buffer_load_dwordx4 v[2:5], v[0:1], s[4:7], 0 addr64
+; GFX7-NEXT:    buffer_load_dwordx4 v[6:9], v[0:1], s[4:7], 0 addr64 offset:16
 ; GFX7-NEXT:    s_and_b32 s0, s3, 1
-; GFX7-NEXT:    s_lshr_b32 s12, s3, 1
 ; GFX7-NEXT:    s_and_b32 s1, s2, 0xffff
 ; GFX7-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX7-NEXT:    s_lshl_b32 s13, s1, s0
+; GFX7-NEXT:    s_lshr_b32 m0, s3, 1
+; GFX7-NEXT:    s_lshl_b32 s1, s1, s0
 ; GFX7-NEXT:    s_lshl_b32 s0, 0xffff, s0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s12, 1
-; GFX7-NEXT:    s_not_b32 s14, s0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[0:1], s12, 2
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[2:3], s12, 3
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[4:5], s12, 4
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[6:7], s12, 5
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[8:9], s12, 6
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[10:11], s12, 7
-; GFX7-NEXT:    s_mov_b32 s18, -1
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v2, v3, vcc
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v4, s[0:1]
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v5, s[2:3]
+; GFX7-NEXT:    s_not_b32 s0, s0
+; GFX7-NEXT:    s_mov_b32 s6, -1
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[4:5]
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v7, s[6:7]
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v8, s[8:9]
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v0, v9, s[10:11]
-; GFX7-NEXT:    v_and_b32_e32 v0, s14, v0
-; GFX7-NEXT:    v_or_b32_e32 v10, s13, v0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[12:13], s12, 0
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v2, v10, s[12:13]
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v3, v10, vcc
-; GFX7-NEXT:    v_cndmask_b32_e64 v2, v4, v10, s[0:1]
-; GFX7-NEXT:    v_cndmask_b32_e64 v3, v5, v10, s[2:3]
-; GFX7-NEXT:    v_cndmask_b32_e64 v4, v6, v10, s[4:5]
-; GFX7-NEXT:    v_cndmask_b32_e64 v5, v7, v10, s[6:7]
-; GFX7-NEXT:    v_cndmask_b32_e64 v6, v8, v10, s[8:9]
-; GFX7-NEXT:    v_cndmask_b32_e64 v7, v9, v10, s[10:11]
-; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[16:19], 0
-; GFX7-NEXT:    s_mov_b64 s[16:17], 16
-; GFX7-NEXT:    buffer_store_dwordx4 v[4:7], off, s[16:19], 0
+; GFX7-NEXT:    v_movrels_b32_e32 v0, v2
+; GFX7-NEXT:    v_and_b32_e32 v0, s0, v0
+; GFX7-NEXT:    v_or_b32_e32 v0, s1, v0
+; GFX7-NEXT:    v_movreld_b32_e32 v2, v0
+; GFX7-NEXT:    buffer_store_dwordx4 v[2:5], off, s[4:7], 0
+; GFX7-NEXT:    s_mov_b64 s[4:5], 16
+; GFX7-NEXT:    buffer_store_dwordx4 v[6:9], off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: insertelement_v_v16i16_s_s:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    global_load_dwordx4 v[2:5], v[0:1], off
 ; GFX10-NEXT:    global_load_dwordx4 v[6:9], v[0:1], off offset:16
-; GFX10-NEXT:    s_lshr_b32 s7, s3, 1
-; GFX10-NEXT:    s_and_b32 s8, s2, 0xffff
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s7, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s0, s7, 2
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s1, s7, 3
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s4, s7, 4
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s5, s7, 5
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s6, s7, 6
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s2, s7, 7
-; GFX10-NEXT:    s_and_b32 s3, s3, 1
-; GFX10-NEXT:    v_mov_b32_e32 v10, 0
-; GFX10-NEXT:    s_lshl_b32 s3, s3, 4
+; GFX10-NEXT:    s_and_b32 s0, s3, 1
+; GFX10-NEXT:    s_lshr_b32 m0, s3, 1
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
+; GFX10-NEXT:    s_and_b32 s1, s2, 0xffff
+; GFX10-NEXT:    s_lshl_b32 s2, 0xffff, s0
+; GFX10-NEXT:    s_lshl_b32 s0, s1, s0
+; GFX10-NEXT:    s_not_b32 s1, s2
+; GFX10-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX10-NEXT:    v_mov_b32_e32 v11, 0
-; GFX10-NEXT:    s_lshl_b32 s9, 0xffff, s3
-; GFX10-NEXT:    s_lshl_b32 s3, s8, s3
-; GFX10-NEXT:    s_not_b32 s8, s9
-; GFX10-NEXT:    v_mov_b32_e32 v12, 16
-; GFX10-NEXT:    v_mov_b32_e32 v13, 0
-; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v2, v3, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v4, s0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v5, s1
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s4
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v7, s5
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v8, s6
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v9, s2
-; GFX10-NEXT:    v_and_or_b32 v14, v0, s8, s3
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s3, s7, 0
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v3, v14, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v2, v14, s3
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v4, v14, s0
-; GFX10-NEXT:    v_cndmask_b32_e64 v3, v5, v14, s1
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, v6, v14, s4
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, v7, v14, s5
-; GFX10-NEXT:    v_cndmask_b32_e64 v6, v8, v14, s6
-; GFX10-NEXT:    v_cndmask_b32_e64 v7, v9, v14, s2
-; GFX10-NEXT:    global_store_dwordx4 v[10:11], v[0:3], off
-; GFX10-NEXT:    global_store_dwordx4 v[12:13], v[4:7], off
+; GFX10-NEXT:    v_movrels_b32_e32 v0, v2
+; GFX10-NEXT:    v_and_or_b32 v12, v0, s1, s0
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GFX10-NEXT:    v_movreld_b32_e32 v2, v12
+; GFX10-NEXT:    global_store_dwordx4 v[0:1], v[2:5], off
+; GFX10-NEXT:    global_store_dwordx4 v[10:11], v[6:9], off
 ; GFX10-NEXT:    s_endpgm
   %vec = load <16 x i16>, <16 x i16> addrspace(1 )* %ptr
   %insert = insertelement <16 x i16> %vec, i16 %val, i32 %idx
   store <16 x i16> %insert, <16 x i16> addrspace(1)* null
   ret void
 }
 
 define amdgpu_ps void @insertelement_s_v16i16_v_s(<16 x i16> addrspace(4)* inreg %ptr, i16 %val, i32 inreg %idx) {

 ; GFX9-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX9-NEXT:    global_store_dwordx4 v[8:9], v[0:3], off
 ; GFX9-NEXT:    global_store_dwordx4 v[10:11], v[4:7], off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: insertelement_s_v16i16_v_s:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX8-NEXT:    s_lshr_b32 s2, s4, 1
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 1
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 0
-; GFX8-NEXT:    v_mov_b32_e32 v10, 16
-; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    s_cselect_b32 s0, s9, s8
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 2
-; GFX8-NEXT:    s_cselect_b32 s0, s10, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 3
-; GFX8-NEXT:    s_cselect_b32 s0, s11, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 4
-; GFX8-NEXT:    s_cselect_b32 s0, s12, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 5
-; GFX8-NEXT:    s_cselect_b32 s0, s13, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 6
-; GFX8-NEXT:    s_cselect_b32 s0, s14, s0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 7
-; GFX8-NEXT:    s_cselect_b32 s0, s15, s0
 ; GFX8-NEXT:    s_and_b32 s1, s4, 1
+; GFX8-NEXT:    s_lshr_b32 m0, s4, 1
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, 4
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_movrels_b32 s0, s8
 ; GFX8-NEXT:    s_lshl_b32 s1, 0xffff, s1
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    s_andn2_b32 s0, s0, s1
 ; GFX8-NEXT:    v_or_b32_e32 v8, s0, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s8
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s9
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 1
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s10
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s11
-; GFX8-NEXT:    v_cndmask_b32_e32 v2, v2, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 3
+; GFX8-NEXT:    v_mov_b32_e32 v4, s12
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s13
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 5
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s14
-; GFX8-NEXT:    v_cndmask_b32_e32 v5, v5, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
-; GFX8-NEXT:    v_mov_b32_e32 v4, s12
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s15
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[0:1], s2, 4
-; GFX8-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
-; GFX8-NEXT:    v_cndmask_b32_e64 v4, v4, v8, s[0:1]
-; GFX8-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc
+; GFX8-NEXT:    v_movreld_b32_e32 v0, v8
 ; GFX8-NEXT:    v_mov_b32_e32 v8, 0
 ; GFX8-NEXT:    v_mov_b32_e32 v9, 0
+; GFX8-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX8-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX8-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
 ; GFX8-NEXT:    flat_store_dwordx4 v[10:11], v[4:7]
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX7-LABEL: insertelement_s_v16i16_v_s:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX7-NEXT:    s_lshr_b32 s2, s4, 1
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 1
-; GFX7-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 0
-; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_cselect_b32 s0, s9, s8
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 2
-; GFX7-NEXT:    s_cselect_b32 s0, s10, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 3
-; GFX7-NEXT:    s_cselect_b32 s0, s11, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 4
-; GFX7-NEXT:    s_cselect_b32 s0, s12, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 5
-; GFX7-NEXT:    s_cselect_b32 s0, s13, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 6
-; GFX7-NEXT:    s_cselect_b32 s0, s14, s0
-; GFX7-NEXT:    s_cmp_eq_u32 s2, 7
-; GFX7-NEXT:    s_cselect_b32 s0, s15, s0
 ; GFX7-NEXT:    s_and_b32 s1, s4, 1
+; GFX7-NEXT:    s_lshr_b32 m0, s4, 1
 ; GFX7-NEXT:    s_lshl_b32 s1, s1, 4
+; GFX7-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_movrels_b32 s0, s8
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, s1, v0
 ; GFX7-NEXT:    s_lshl_b32 s1, 0xffff, s1
 ; GFX7-NEXT:    s_andn2_b32 s0, s0, s1
 ; GFX7-NEXT:    v_or_b32_e32 v8, s0, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v0, s8
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s9
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 1
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s10
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 2
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s11
-; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 3
-; GFX7-NEXT:    v_mov_b32_e32 v5, s13
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 5
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s12
+; GFX7-NEXT:    v_mov_b32_e32 v5, s13
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s14
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[0:1], s2, 4
-; GFX7-NEXT:    v_cndmask_b32_e32 v5, v5, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s15
-; GFX7-NEXT:    v_cndmask_b32_e64 v4, v4, v8, s[0:1]
-; GFX7-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
 ; GFX7-NEXT:    s_mov_b64 s[0:1], 0
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
-; GFX7-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc
+; GFX7-NEXT:    v_movreld_b32_e32 v0, v8
 ; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b64 s[0:1], 16
 ; GFX7-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: insertelement_s_v16i16_v_s:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx8 s[8:15], s[2:3], 0x0
-; GFX10-NEXT:    s_lshr_b32 s0, s4, 1
+; GFX10-NEXT:    s_and_b32 s0, s4, 1
+; GFX10-NEXT:    s_lshr_b32 m0, s4, 1
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
 ; GFX10-NEXT:    v_and_b32_e32 v8, 0xffff, v0
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 0
+; GFX10-NEXT:    s_lshl_b32 s1, 0xffff, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX10-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    s_cselect_b32 s1, s9, s8
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 2
+; GFX10-NEXT:    s_movrels_b32 s2, s8
 ; GFX10-NEXT:    v_mov_b32_e32 v0, s8
-; GFX10-NEXT:    s_cselect_b32 s1, s10, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 3
+; GFX10-NEXT:    s_andn2_b32 s1, s2, s1
 ; GFX10-NEXT:    v_mov_b32_e32 v1, s9
-; GFX10-NEXT:    s_cselect_b32 s1, s11, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 4
+; GFX10-NEXT:    v_lshl_or_b32 v12, v8, s0, s1
 ; GFX10-NEXT:    v_mov_b32_e32 v2, s10
-; GFX10-NEXT:    s_cselect_b32 s1, s12, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 5
 ; GFX10-NEXT:    v_mov_b32_e32 v3, s11
-; GFX10-NEXT:    s_cselect_b32 s1, s13, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 6
 ; GFX10-NEXT:    v_mov_b32_e32 v4, s12
-; GFX10-NEXT:    s_cselect_b32 s1, s14, s1
-; GFX10-NEXT:    s_cmp_eq_u32 s0, 7
 ; GFX10-NEXT:    v_mov_b32_e32 v5, s13
-; GFX10-NEXT:    s_cselect_b32 s1, s15, s1
-; GFX10-NEXT:    s_and_b32 s2, s4, 1
 ; GFX10-NEXT:    v_mov_b32_e32 v6, s14
-; GFX10-NEXT:    s_lshl_b32 s2, s2, 4
 ; GFX10-NEXT:    v_mov_b32_e32 v7, s15
-; GFX10-NEXT:    s_lshl_b32 s3, 0xffff, s2
-; GFX10-NEXT:    s_andn2_b32 s1, s1, s3
-; GFX10-NEXT:    v_lshl_or_b32 v12, v8, s2, s1
 ; GFX10-NEXT:    v_mov_b32_e32 v8, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v9, 0
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v1, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 2
-; GFX10-NEXT:    v_cndmask_b32_e32 v2, v2, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 3
-; GFX10-NEXT:    v_cndmask_b32_e32 v3, v3, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 4
-; GFX10-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 5
-; GFX10-NEXT:    v_cndmask_b32_e32 v5, v5, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 6
-; GFX10-NEXT:    v_cndmask_b32_e32 v6, v6, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 7
-; GFX10-NEXT:    v_cndmask_b32_e32 v7, v7, v12, vcc_lo
+; GFX10-NEXT:    v_movreld_b32_e32 v0, v12
 ; GFX10-NEXT:    global_store_dwordx4 v[8:9], v[0:3], off
 ; GFX10-NEXT:    global_store_dwordx4 v[10:11], v[4:7], off
 ; GFX10-NEXT:    s_endpgm
   %vec = load <16 x i16>, <16 x i16> addrspace(4)* %ptr
   %insert = insertelement <16 x i16> %vec, i16 %val, i32 %idx
   store <16 x i16> %insert, <16 x i16> addrspace(1)* null
   ret void
 }

 ; GFX9-NEXT:    v_mov_b32_e32 v10, 16
 ; GFX9-NEXT:    v_mov_b32_e32 v11, 0
 ; GFX9-NEXT:    global_store_dwordx4 v[8:9], v[0:3], off
 ; GFX9-NEXT:    global_store_dwordx4 v[10:11], v[4:7], off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: insertelement_v_v16i16_v_s:
 ; GFX8:       ; %bb.0:
+; GFX8-NEXT:    v_add_u32_e32 v7, vcc, 16, v0
+; GFX8-NEXT:    v_addc_u32_e32 v8, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dwordx4 v[3:6], v[0:1]
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 16, v0
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT:    flat_load_dwordx4 v[7:10], v[0:1]
+; GFX8-NEXT:    flat_load_dwordx4 v[7:10], v[7:8]
 ; GFX8-NEXT:    s_and_b32 s0, s2, 1
-; GFX8-NEXT:    s_lshr_b32 s12, s2, 1
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX8-NEXT:    v_mov_b32_e32 v0, s0
+; GFX8-NEXT:    s_lshr_b32 m0, s2, 1
+; GFX8-NEXT:    v_mov_b32_e32 v13, s0
 ; GFX8-NEXT:    s_lshl_b32 s0, 0xffff, s0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s12, 1
-; GFX8-NEXT:    s_not_b32 s13, s0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[0:1], s12, 2
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[2:3], s12, 3
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[4:5], s12, 4
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[6:7], s12, 5
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[8:9], s12, 6
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[10:11], s12, 7
-; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
-; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v5, s[0:1]
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v6, s[2:3]
+; GFX8-NEXT:    v_lshlrev_b32_sdwa v2, v13, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; GFX8-NEXT:    s_not_b32 s0, s0
+; GFX8-NEXT:    v_mov_b32_e32 v0, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, 0
+; GFX8-NEXT:    v_mov_b32_e32 v11, 16
+; GFX8-NEXT:    v_mov_b32_e32 v12, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[4:5]
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v8, s[6:7]
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v9, s[8:9]
-; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v10, s[10:11]
-; GFX8-NEXT:    v_and_b32_e32 v1, s13, v1
-; GFX8-NEXT:    v_or_b32_e32 v11, v1, v0
-; GFX8-NEXT:    v_cmp_eq_u32_e64 s[12:13], s12, 0
-; GFX8-NEXT:    v_cndmask_b32_e64 v0, v3, v11, s[12:13]
-; GFX8-NEXT:    v_cndmask_b32_e64 v2, v5, v11, s[0:1]
-; GFX8-NEXT:    v_cndmask_b32_e64 v3, v6, v11, s[2:3]
-; GFX8-NEXT:    v_cndmask_b32_e64 v5, v8, v11, s[6:7]
-; GFX8-NEXT:    v_cndmask_b32_e64 v6, v9, v11, s[8:9]
-; GFX8-NEXT:    v_mov_b32_e32 v8, 0
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v4, v11, vcc
-; GFX8-NEXT:    v_cndmask_b32_e64 v4, v7, v11, s[4:5]
-; GFX8-NEXT:    v_cndmask_b32_e64 v7, v10, v11, s[10:11]
-; GFX8-NEXT:    v_mov_b32_e32 v9, 0
-; GFX8-NEXT:    v_mov_b32_e32 v10, 16
-; GFX8-NEXT:    v_mov_b32_e32 v11, 0
-; GFX8-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
-; GFX8-NEXT:    flat_store_dwordx4 v[10:11], v[4:7]
+; GFX8-NEXT:    v_movrels_b32_e32 v13, v3
+; GFX8-NEXT:    v_and_b32_e32 v13, s0, v13
+; GFX8-NEXT:    v_or_b32_e32 v2, v13, v2
+; GFX8-NEXT:    v_movreld_b32_e32 v3, v2
+; GFX8-NEXT:    flat_store_dwordx4 v[0:1], v[3:6]
+; GFX8-NEXT:    flat_store_dwordx4 v[11:12], v[7:10]
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX7-LABEL: insertelement_v_v16i16_v_s:
 ; GFX7:       ; %bb.0:
-; GFX7-NEXT:    s_mov_b32 s18, 0
-; GFX7-NEXT:    s_mov_b32 s19, 0xf000
-; GFX7-NEXT:    s_mov_b64 s[16:17], 0
-; GFX7-NEXT:    buffer_load_dwordx4 v[3:6], v[0:1], s[16:19], 0 addr64
-; GFX7-NEXT:    buffer_load_dwordx4 v[7:10], v[0:1], s[16:19], 0 addr64 offset:16
+; GFX7-NEXT:    s_mov_b32 s6, 0
+; GFX7-NEXT:    s_mov_b32 s7, 0xf000
+; GFX7-NEXT:    s_mov_b64 s[4:5], 0
+; GFX7-NEXT:    buffer_load_dwordx4 v[3:6], v[0:1], s[4:7], 0 addr64
+; GFX7-NEXT:    buffer_load_dwordx4 v[7:10], v[0:1], s[4:7], 0 addr64 offset:16
 ; GFX7-NEXT:    s_and_b32 s0, s2, 1
-; GFX7-NEXT:    s_lshr_b32 s12, s2, 1
 ; GFX7-NEXT:    v_and_b32_e32 v0, 0xffff, v2
 ; GFX7-NEXT:    s_lshl_b32 s0, s0, 4
+; GFX7-NEXT:    s_lshr_b32 m0, s2, 1
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, s0, v0
 ; GFX7-NEXT:    s_lshl_b32 s0, 0xffff, s0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s12, 1
-; GFX7-NEXT:    s_not_b32 s13, s0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[0:1], s12, 2
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[2:3], s12, 3
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[4:5], s12, 4
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[6:7], s12, 5
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[8:9], s12, 6
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[10:11], s12, 7
-; GFX7-NEXT:    s_mov_b32 s18, -1
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v5, s[0:1]
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v6, s[2:3]
+; GFX7-NEXT:    s_not_b32 s0, s0
+; GFX7-NEXT:    s_mov_b32 s6, -1
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[4:5]
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v8, s[6:7]
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v9, s[8:9]
-; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v10, s[10:11]
-; GFX7-NEXT:    v_and_b32_e32 v1, s13, v1
-; GFX7-NEXT:    v_or_b32_e32 v11, v1, v0
-; GFX7-NEXT:    v_cmp_eq_u32_e64 s[12:13], s12, 0
-; GFX7-NEXT:    v_cndmask_b32_e64 v0, v3, v11, s[12:13]
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v4, v11, vcc
-; GFX7-NEXT:    v_cndmask_b32_e64 v2, v5, v11, s[0:1]
-; GFX7-NEXT:    v_cndmask_b32_e64 v3, v6, v11, s[2:3]
-; GFX7-NEXT:    v_cndmask_b32_e64 v4, v7, v11, s[4:5]
-; GFX7-NEXT:    v_cndmask_b32_e64 v5, v8, v11, s[6:7]
-; GFX7-NEXT:    v_cndmask_b32_e64 v6, v9, v11, s[8:9]
-; GFX7-NEXT:    v_cndmask_b32_e64 v7, v10, v11, s[10:11]
-; GFX7-NEXT:    buffer_store_dwordx4 v[0:3], off, s[16:19], 0
-; GFX7-NEXT:    s_mov_b64 s[16:17], 16
-; GFX7-NEXT:    buffer_store_dwordx4 v[4:7], off, s[16:19], 0
+; GFX7-NEXT:    v_movrels_b32_e32 v1, v3
+; GFX7-NEXT:    v_and_b32_e32 v1, s0, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v1, v0
+; GFX7-NEXT:    v_movreld_b32_e32 v3, v0
+; GFX7-NEXT:    buffer_store_dwordx4 v[3:6], off, s[4:7], 0
+; GFX7-NEXT:    s_mov_b64 s[4:5], 16
+; GFX7-NEXT:    buffer_store_dwordx4 v[7:10], off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: insertelement_v_v16i16_v_s:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    global_load_dwordx4 v[3:6], v[0:1], off
 ; GFX10-NEXT:    global_load_dwordx4 v[7:10], v[0:1], off offset:16
-; GFX10-NEXT:    s_lshr_b32 s6, s2, 1
-; GFX10-NEXT:    s_and_b32 s5, s2, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s6, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s0, s6, 2
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s1, s6, 3
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s3, s6, 4
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s4, s6, 5
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s2, s6, 6
-; GFX10-NEXT:    s_lshl_b32 s7, s5, 4
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s5, s6, 7
-; GFX10-NEXT:    v_lshlrev_b32_sdwa v1, s7, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
-; GFX10-NEXT:    s_lshl_b32 s7, 0xffff, s7
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s6, s6, 0
-; GFX10-NEXT:    s_not_b32 s7, s7
-; GFX10-NEXT:    v_mov_b32_e32 v11, 0
+; GFX10-NEXT:    s_and_b32 s0, s2, 1
+; GFX10-NEXT:    s_lshr_b32 m0, s2, 1
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
+; GFX10-NEXT:    v_mov_b32_e32 v11, 16
+; GFX10-NEXT:    v_lshlrev_b32_sdwa v0, s0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; GFX10-NEXT:    s_lshl_b32 s0, 0xffff, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v12, 0
-; GFX10-NEXT:    v_mov_b32_e32 v13, 16
-; GFX10-NEXT:    v_mov_b32_e32 v14, 0
-; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v5, s0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s1
+; GFX10-NEXT:    s_not_b32 s0, s0
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v7, s3
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v8, s4
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v9, s2
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v0, v10, s5
-; GFX10-NEXT:    v_and_or_b32 v15, v0, s7, v1
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v3, v15, s6
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v4, v15, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v5, v15, s0
-; GFX10-NEXT:    v_cndmask_b32_e64 v3, v6, v15, s1
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, v7, v15, s3
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, v8, v15, s4
-; GFX10-NEXT:    v_cndmask_b32_e64 v6, v9, v15, s2
-; GFX10-NEXT:    v_cndmask_b32_e64 v7, v10, v15, s5
-; GFX10-NEXT:    global_store_dwordx4 v[11:12], v[0:3], off
-; GFX10-NEXT:    global_store_dwordx4 v[13:14], v[4:7], off
+; GFX10-NEXT:    v_movrels_b32_e32 v1, v3
+; GFX10-NEXT:    v_and_or_b32 v2, v1, s0, v0
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GFX10-NEXT:    v_movreld_b32_e32 v3, v2
+; GFX10-NEXT:    global_store_dwordx4 v[0:1], v[3:6], off
+; GFX10-NEXT:    global_store_dwordx4 v[11:12], v[7:10], off
 ; GFX10-NEXT:    s_endpgm
   %vec = load <16 x i16>, <16 x i16> addrspace(1)* %ptr
   %insert = insertelement <16 x i16> %vec, i16 %val, i32 %idx
   store <16 x i16> %insert, <16 x i16> addrspace(1)* null
   ret void
 }
 
 define amdgpu_ps void @insertelement_v_v16i16_v_v(<16 x i16> addrspace(1)* %ptr, i16 %val, i32 %idx) {

llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement.ll

 ; GPRIDX-NEXT:    s_cmp_eq_u32 s11, 6
 ; GPRIDX-NEXT:    s_cselect_b32 s6, s10, s8
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s11, 7
 ; GPRIDX-NEXT:    s_cselect_b32 s7, s10, s9
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8i32_s_s_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 0
-; MOVREL-NEXT:    s_cselect_b32 s0, s10, s2
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 1
-; MOVREL-NEXT:    s_cselect_b32 s1, s10, s3
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 2
-; MOVREL-NEXT:    s_cselect_b32 s2, s10, s4
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 3
-; MOVREL-NEXT:    s_cselect_b32 s3, s10, s5
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 4
-; MOVREL-NEXT:    s_cselect_b32 s4, s10, s6
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 5
-; MOVREL-NEXT:    s_cselect_b32 s5, s10, s7
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 6
-; MOVREL-NEXT:    s_cselect_b32 s6, s10, s8
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 7
-; MOVREL-NEXT:    s_cselect_b32 s7, s10, s9
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 m0, s11
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movreld_b32 s0, s10
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x i32> %vec, i32 %val, i32 %idx
   ret <8 x i32> %insert
 }
 
 define amdgpu_ps <8 x i8 addrspace(3)*> @dyn_insertelement_v8p3i8_s_s_s(<8 x i8 addrspace(3)*> inreg %vec, i8 addrspace(3)* inreg %val, i32 inreg %idx) {
 ; GPRIDX-LABEL: dyn_insertelement_v8p3i8_s_s_s:

 ; GPRIDX-NEXT:    s_cmp_eq_u32 s11, 6
 ; GPRIDX-NEXT:    s_cselect_b32 s6, s10, s8
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s11, 7
 ; GPRIDX-NEXT:    s_cselect_b32 s7, s10, s9
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8p3i8_s_s_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 0
-; MOVREL-NEXT:    s_cselect_b32 s0, s10, s2
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 1
-; MOVREL-NEXT:    s_cselect_b32 s1, s10, s3
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 2
-; MOVREL-NEXT:    s_cselect_b32 s2, s10, s4
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 3
-; MOVREL-NEXT:    s_cselect_b32 s3, s10, s5
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 4
-; MOVREL-NEXT:    s_cselect_b32 s4, s10, s6
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 5
-; MOVREL-NEXT:    s_cselect_b32 s5, s10, s7
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 6
-; MOVREL-NEXT:    s_cselect_b32 s6, s10, s8
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 7
-; MOVREL-NEXT:    s_cselect_b32 s7, s10, s9
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 m0, s11
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movreld_b32 s0, s10
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x i8 addrspace(3)*> %vec, i8 addrspace(3)* %val, i32 %idx
   ret <8 x i8 addrspace(3)*> %insert
 }
 
 define <8 x float> @dyn_insertelement_v8f32_const_s_v_v(float %val, i32 %idx) {
 ; GPRIDX-LABEL: dyn_insertelement_v8f32_const_s_v_v:

 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v6, v14, v0, vcc
 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s10, 7
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v7, v15, v0, vcc
 ; GPRIDX-NEXT:    v_mov_b32_e32 v0, v8
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8f32_s_v_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_mov_b32 s1, s3
-; MOVREL-NEXT:    s_mov_b32 s3, s5
-; MOVREL-NEXT:    s_mov_b32 s5, s7
-; MOVREL-NEXT:    s_mov_b32 s7, s9
 ; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 s1, s3
 ; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
 ; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
 ; MOVREL-NEXT:    s_mov_b32 s6, s8
-; MOVREL-NEXT:    v_mov_b32_e32 v15, s7
-; MOVREL-NEXT:    v_mov_b32_e32 v8, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 0
-; MOVREL-NEXT:    v_mov_b32_e32 v9, s1
-; MOVREL-NEXT:    v_mov_b32_e32 v10, s2
-; MOVREL-NEXT:    v_mov_b32_e32 v11, s3
-; MOVREL-NEXT:    v_mov_b32_e32 v12, s4
-; MOVREL-NEXT:    v_cndmask_b32_e32 v8, v8, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 1
-; MOVREL-NEXT:    v_mov_b32_e32 v13, s5
-; MOVREL-NEXT:    v_mov_b32_e32 v14, s6
-; MOVREL-NEXT:    v_cndmask_b32_e32 v1, v9, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 2
-; MOVREL-NEXT:    v_cndmask_b32_e32 v2, v10, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 3
-; MOVREL-NEXT:    v_cndmask_b32_e32 v3, v11, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 4
-; MOVREL-NEXT:    v_cndmask_b32_e32 v4, v12, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 5
-; MOVREL-NEXT:    v_cndmask_b32_e32 v5, v13, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 6
-; MOVREL-NEXT:    v_cndmask_b32_e32 v6, v14, v0, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s10, 7
-; MOVREL-NEXT:    v_cndmask_b32_e32 v7, v15, v0, vcc_lo
-; MOVREL-NEXT:    v_mov_b32_e32 v0, v8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    v_mov_b32_e32 v8, v0
+; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
+; MOVREL-NEXT:    s_mov_b32 m0, s10
+; MOVREL-NEXT:    v_mov_b32_e32 v1, s1
+; MOVREL-NEXT:    v_mov_b32_e32 v2, s2
+; MOVREL-NEXT:    v_mov_b32_e32 v3, s3
+; MOVREL-NEXT:    v_mov_b32_e32 v4, s4
+; MOVREL-NEXT:    v_mov_b32_e32 v5, s5
+; MOVREL-NEXT:    v_mov_b32_e32 v6, s6
+; MOVREL-NEXT:    v_mov_b32_e32 v7, s7
+; MOVREL-NEXT:    v_movreld_b32_e32 v0, v8
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x float> %vec, float %val, i32 %idx
   ret <8 x float> %insert
 }
 
 define amdgpu_ps <8 x float> @dyn_insertelement_v8f32_v_s_s(<8 x float> %vec, float inreg %val, i32 inreg %idx) {
 ; GPRIDX-LABEL: dyn_insertelement_v8f32_v_s_s:

 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s3, 6
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc
 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s3, 7
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8f32_v_s_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 0
-; MOVREL-NEXT:    v_cndmask_b32_e64 v0, v0, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 1
-; MOVREL-NEXT:    v_cndmask_b32_e64 v1, v1, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 2
-; MOVREL-NEXT:    v_cndmask_b32_e64 v2, v2, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 3
-; MOVREL-NEXT:    v_cndmask_b32_e64 v3, v3, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 4
-; MOVREL-NEXT:    v_cndmask_b32_e64 v4, v4, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 5
-; MOVREL-NEXT:    v_cndmask_b32_e64 v5, v5, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 6
-; MOVREL-NEXT:    v_cndmask_b32_e64 v6, v6, s2, s0
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 s0, s3, 7
-; MOVREL-NEXT:    v_cndmask_b32_e64 v7, v7, s2, s0
+; MOVREL-NEXT:    s_mov_b32 m0, s3
+; MOVREL-NEXT:    v_movreld_b32_e32 v0, s2
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x float> %vec, float %val, i32 %idx
   ret <8 x float> %insert
 }
 
 define amdgpu_ps <8 x float> @dyn_insertelement_v8f32_s_v_v(<8 x float> inreg %vec, float %val, i32 %idx) {
 ; GPRIDX-LABEL: dyn_insertelement_v8f32_s_v_v:

 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc
 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8f32_v_v_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 0
-; MOVREL-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 1
-; MOVREL-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 2
-; MOVREL-NEXT:    v_cndmask_b32_e32 v2, v2, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 3
-; MOVREL-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 4
-; MOVREL-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 5
-; MOVREL-NEXT:    v_cndmask_b32_e32 v5, v5, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 6
-; MOVREL-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 7
-; MOVREL-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc_lo
+; MOVREL-NEXT:    s_mov_b32 m0, s2
+; MOVREL-NEXT:    v_movreld_b32_e32 v0, v8
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x float> %vec, float %val, i32 %idx
   ret <8 x float> %insert
 }
 
 define amdgpu_ps <8 x float> @dyn_insertelement_v8p3i8_v_v_s(<8 x i8 addrspace(3)*> %vec, i8 addrspace(3)* %val, i32 inreg %idx) {
 ; GPRIDX-LABEL: dyn_insertelement_v8p3i8_v_v_s:

 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 6
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc
 ; GPRIDX-NEXT:    v_cmp_eq_u32_e64 vcc, s2, 7
 ; GPRIDX-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8p3i8_v_v_s:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 0
-; MOVREL-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 1
-; MOVREL-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 2
-; MOVREL-NEXT:    v_cndmask_b32_e32 v2, v2, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 3
-; MOVREL-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 4
-; MOVREL-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 5
-; MOVREL-NEXT:    v_cndmask_b32_e32 v5, v5, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 6
-; MOVREL-NEXT:    v_cndmask_b32_e32 v6, v6, v8, vcc_lo
-; MOVREL-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s2, 7
-; MOVREL-NEXT:    v_cndmask_b32_e32 v7, v7, v8, vcc_lo
+; MOVREL-NEXT:    s_mov_b32 m0, s2
+; MOVREL-NEXT:    v_movreld_b32_e32 v0, v8
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %insert = insertelement <8 x i8 addrspace(3)*> %vec, i8 addrspace(3)* %val, i32 %idx
   %cast.0 = ptrtoint <8 x i8 addrspace(3)*> %insert to <8 x i32>
   %cast.1 = bitcast <8 x i32> %cast.0 to <8 x float>
   ret <8 x float> %cast.1
 }
 

 ; GPRIDX-NEXT:    v_mov_b32_e32 v4, s4
 ; GPRIDX-NEXT:    v_mov_b32_e32 v5, s5
 ; GPRIDX-NEXT:    v_mov_b32_e32 v6, s6
 ; GPRIDX-NEXT:    v_mov_b32_e32 v7, s7
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8f32_s_s_s_add_1:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_add_i32 s11, s11, 1
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 0
-; MOVREL-NEXT:    s_cselect_b32 s0, s10, s2
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 1
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 m0, s11
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_movreld_b32 s1, s10
 ; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
-; MOVREL-NEXT:    s_cselect_b32 s1, s10, s3
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 2
 ; MOVREL-NEXT:    v_mov_b32_e32 v1, s1
-; MOVREL-NEXT:    s_cselect_b32 s2, s10, s4
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 3
 ; MOVREL-NEXT:    v_mov_b32_e32 v2, s2
-; MOVREL-NEXT:    s_cselect_b32 s3, s10, s5
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 4
 ; MOVREL-NEXT:    v_mov_b32_e32 v3, s3
-; MOVREL-NEXT:    s_cselect_b32 s4, s10, s6
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 5
 ; MOVREL-NEXT:    v_mov_b32_e32 v4, s4
-; MOVREL-NEXT:    s_cselect_b32 s5, s10, s7
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 6
 ; MOVREL-NEXT:    v_mov_b32_e32 v5, s5
-; MOVREL-NEXT:    s_cselect_b32 s6, s10, s8
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 7
 ; MOVREL-NEXT:    v_mov_b32_e32 v6, s6
-; MOVREL-NEXT:    s_cselect_b32 s7, s10, s9
 ; MOVREL-NEXT:    v_mov_b32_e32 v7, s7
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %idx.add = add i32 %idx, 1
   %insert = insertelement <8 x float> %vec, float %val, i32 %idx.add
   ret <8 x float> %insert
 }
 

 ; GPRIDX-NEXT:    v_mov_b32_e32 v4, s4
 ; GPRIDX-NEXT:    v_mov_b32_e32 v5, s5
 ; GPRIDX-NEXT:    v_mov_b32_e32 v6, s6
 ; GPRIDX-NEXT:    v_mov_b32_e32 v7, s7
 ; GPRIDX-NEXT:    ; return to shader part epilog
 ;
 ; MOVREL-LABEL: dyn_insertelement_v8f32_s_s_s_add_7:
 ; MOVREL:       ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_add_i32 s11, s11, 7
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 0
-; MOVREL-NEXT:    s_cselect_b32 s0, s10, s2
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 1
+; MOVREL-NEXT:    s_mov_b32 s1, s3
+; MOVREL-NEXT:    s_mov_b32 s3, s5
+; MOVREL-NEXT:    s_mov_b32 s5, s7
+; MOVREL-NEXT:    s_mov_b32 s7, s9
+; MOVREL-NEXT:    s_mov_b32 m0, s11
+; MOVREL-NEXT:    s_mov_b32 s0, s2
+; MOVREL-NEXT:    s_mov_b32 s2, s4
+; MOVREL-NEXT:    s_mov_b32 s4, s6
+; MOVREL-NEXT:    s_mov_b32 s6, s8
+; MOVREL-NEXT:    s_movreld_b32 s7, s10
 ; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
-; MOVREL-NEXT:    s_cselect_b32 s1, s10, s3
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 2
 ; MOVREL-NEXT:    v_mov_b32_e32 v1, s1
-; MOVREL-NEXT:    s_cselect_b32 s2, s10, s4
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 3
 ; MOVREL-NEXT:    v_mov_b32_e32 v2, s2
-; MOVREL-NEXT:    s_cselect_b32 s3, s10, s5
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 4
 ; MOVREL-NEXT:    v_mov_b32_e32 v3, s3
-; MOVREL-NEXT:    s_cselect_b32 s4, s10, s6
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 5
 ; MOVREL-NEXT:    v_mov_b32_e32 v4, s4
-; MOVREL-NEXT:    s_cselect_b32 s5, s10, s7
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 6
 ; MOVREL-NEXT:    v_mov_b32_e32 v5, s5
-; MOVREL-NEXT:    s_cselect_b32 s6, s10, s8
-; MOVREL-NEXT:    s_cmp_eq_u32 s11, 7
 ; MOVREL-NEXT:    v_mov_b32_e32 v6, s6
-; MOVREL-NEXT:    s_cselect_b32 s7, s10, s9
 ; MOVREL-NEXT:    v_mov_b32_e32 v7, s7
 ; MOVREL-NEXT:    ; return to shader part epilog
 entry:
   %idx.add = add i32 %idx, 7
   %insert = insertelement <8 x float> %vec, float %val, i32 %idx.add
   ret <8 x float> %insert
 }
 

llvm/test/CodeGen/AMDGPU/extract_vector_dynelt.ll

 define amdgpu_kernel void @short8_extelt(i16 addrspace(1)* %out, i32 %sel) {
 entry:
   %ext = extractelement <8 x i16> <i16 1, i16 2, i16 3, i16 4, i16 5, i16 6, i16 7, i16 8>, i32 %sel
   store i16 %ext, i16 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}float8_extelt:
-; GCN-NOT: buffer_
-; GCN-DAG: s_cmp_eq_u32 [[IDX:s[0-9]+]], 1
-; GCN-DAG: s_cselect_b64 [[C1:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 2
-; GCN-DAG: s_cselect_b64 [[C2:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 3
-; GCN-DAG: s_cselect_b64 [[C3:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 4
-; GCN-DAG: s_cselect_b64 [[C4:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 5
-; GCN-DAG: s_cselect_b64 [[C5:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 6
-; GCN-DAG: s_cselect_b64 [[C6:[^,]+]], -1, 0
-; GCN-DAG: s_cmp_lg_u32 [[IDX]], 7
-; GCN-DAG: s_cselect_b64 [[C7:[^,]+]], -1, 0
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V1:v[0-9]+]], {{[^,]+}}, {{[^,]+}}, [[C1]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V2:v[0-9]+]], {{[^,]+}}, [[V1]], [[C2]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V3:v[0-9]+]], {{[^,]+}}, [[V2]], [[C3]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V4:v[0-9]+]], {{[^,]+}}, [[V3]], [[C4]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V5:v[0-9]+]], {{[^,]+}}, [[V4]], [[C5]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V6:v[0-9]+]], {{[^,]+}}, [[V5]], [[C6]]
-; GCN-DAG: v_cndmask_b32_e{{32|64}} [[V7:v[0-9]+]], {{[^,]+}}, [[V6]], [[C7]]
-; GCN:     store_dword v[{{[0-9:]+}}], [[V7]]
+; GCN-DAG: s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GCN-DAG: s_load_dword [[S0:s[0-9]+]], s[0:1], 0x2c
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 1.0
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 2.0
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 0x40400000
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 4.0
+; GCN-DAG: s_waitcnt lgkmcnt(0)
+; GCN-DAG: s_mov_b32 m0, [[S0]]
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 0x40a00000
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 0x40c00000
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 0x40e00000
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, 0x41000000
+; GCN-DAG: v_movrels_b32_e32 [[RES:v[0-9]+]], v{{[0-9]+}}
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, s{{[0-9]+}}
+; GCN-DAG: v_mov_b32_e32 v{{[0-9]+}}, s{{[0-9]+}}
+; GCN:     flat_store_dword v[{{[0-9:]+}}], [[RES]]
 define amdgpu_kernel void @float8_extelt(float addrspace(1)* %out, i32 %sel) {
 entry:
   %ext = extractelement <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, i32 %sel
   store float %ext, float addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}double8_extelt:

llvm/test/CodeGen/AMDGPU/insert_vector_dynelt.ll

entry:
   %v = insertelement <2 x float> %vec, float 1.000000e+00, i32 %sel
   store <2 x float> %v, <2 x float> addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @float8_inselt(<8 x float> addrspace(1)* %out, <8 x float> %vec, i32 %sel) {
 ; GCN-LABEL: float8_inselt:
 ; GCN:       ; %bb.0: ; %entry
-; GCN-NEXT:    s_load_dword s2, s[0:1], 0x64
 ; GCN-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x44
-; GCN-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GCN-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GCN-NEXT:    s_load_dword s1, s[0:1], 0x64
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_cmp_lg_u32 s2, 3
-; GCN-NEXT:    v_mov_b32_e32 v0, s7
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 2
-; GCN-NEXT:    v_cndmask_b32_e32 v3, 1.0, v0, vcc
-; GCN-NEXT:    v_mov_b32_e32 v0, s6
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 1
-; GCN-NEXT:    v_cndmask_b32_e32 v2, 1.0, v0, vcc
-; GCN-NEXT:    v_mov_b32_e32 v0, s5
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 0
-; GCN-NEXT:    v_cndmask_b32_e32 v1, 1.0, v0, vcc
 ; GCN-NEXT:    v_mov_b32_e32 v0, s4
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 7
-; GCN-NEXT:    v_cndmask_b32_e32 v0, 1.0, v0, vcc
-; GCN-NEXT:    v_mov_b32_e32 v4, s11
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 6
-; GCN-NEXT:    v_cndmask_b32_e32 v7, 1.0, v4, vcc
-; GCN-NEXT:    v_mov_b32_e32 v4, s10
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 5
-; GCN-NEXT:    v_cndmask_b32_e32 v6, 1.0, v4, vcc
-; GCN-NEXT:    v_mov_b32_e32 v4, s9
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_cmp_lg_u32 s2, 4
-; GCN-NEXT:    v_cndmask_b32_e32 v5, 1.0, v4, vcc
-; GCN-NEXT:    s_cselect_b64 vcc, -1, 0
-; GCN-NEXT:    s_add_u32 s2, s0, 16
-; GCN-NEXT:    s_addc_u32 s3, s1, 0
+; GCN-NEXT:    s_add_u32 s0, s2, 16
+; GCN-NEXT:    s_mov_b32 m0, s1
+; GCN-NEXT:    s_addc_u32 s1, s3, 0
+; GCN-NEXT:    v_mov_b32_e32 v1, s5
+; GCN-NEXT:    v_mov_b32_e32 v2, s6
+; GCN-NEXT:    v_mov_b32_e32 v3, s7
 ; GCN-NEXT:    v_mov_b32_e32 v4, s8
-; GCN-NEXT:    v_mov_b32_e32 v9, s3
-; GCN-NEXT:    v_cndmask_b32_e32 v4, 1.0, v4, vcc
-; GCN-NEXT:    v_mov_b32_e32 v8, s2
+; GCN-NEXT:    v_mov_b32_e32 v5, s9
+; GCN-NEXT:    v_mov_b32_e32 v6, s10
+; GCN-NEXT:    v_mov_b32_e32 v7, s11
+; GCN-NEXT:    v_mov_b32_e32 v9, s1
+; GCN-NEXT:    v_movreld_b32_e32 v0, 1.0
+; GCN-NEXT:    v_mov_b32_e32 v8, s0
 ; GCN-NEXT:    flat_store_dwordx4 v[8:9], v[4:7]
 ; GCN-NEXT:    s_nop 0
-; GCN-NEXT:    v_mov_b32_e32 v5, s1
-; GCN-NEXT:    v_mov_b32_e32 v4, s0
+; GCN-NEXT:    v_mov_b32_e32 v5, s3
+; GCN-NEXT:    v_mov_b32_e32 v4, s2
 ; GCN-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
 ; GCN-NEXT:    s_endpgm
 entry:
   %v = insertelement <8 x float> %vec, float 1.000000e+00, i32 %sel
   store <8 x float> %v, <8 x float> addrspace(1)* %out
   ret void
 }
 

llvm/test/CodeGen/AMDGPU/insert_vector_elt.ll

; VI-NEXT:    s_endpgm
   %vecins = insertelement <4 x float> %a, float 5.000000e+00, i32 %b
   store <4 x float> %vecins, <4 x float> addrspace(1)* %out, align 16
   ret void
 }
 
 define amdgpu_kernel void @dynamic_insertelement_v8f32(<8 x float> addrspace(1)* %out, <8 x float> %a, i32 %b) nounwind {
 ; SI-LABEL: dynamic_insertelement_v8f32:
 ; SI:       ; %bb.0:
-; SI-NEXT:    s_load_dword s6, s[4:5], 0x10
-; SI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x8
-; SI-NEXT:    v_mov_b32_e32 v4, 0x40a00000
 ; SI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
+; SI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x8
+; SI-NEXT:    s_load_dword s4, s[4:5], 0x10
+; SI-NEXT:    v_mov_b32_e32 v8, 0x40a00000
 ; SI-NEXT:    s_mov_b32 s3, 0x100f000
+; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_cmp_lg_u32 s6, 3
-; SI-NEXT:    v_mov_b32_e32 v0, s11
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 2
-; SI-NEXT:    v_cndmask_b32_e32 v3, v4, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, s10
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 1
-; SI-NEXT:    v_cndmask_b32_e32 v2, v4, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, s9
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 0
-; SI-NEXT:    v_cndmask_b32_e32 v1, v4, v0, vcc
 ; SI-NEXT:    v_mov_b32_e32 v0, s8
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 7
-; SI-NEXT:    v_cndmask_b32_e32 v0, v4, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v5, s15
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 6
-; SI-NEXT:    v_cndmask_b32_e32 v7, v4, v5, vcc
-; SI-NEXT:    v_mov_b32_e32 v5, s14
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 5
-; SI-NEXT:    v_cndmask_b32_e32 v6, v4, v5, vcc
+; SI-NEXT:    v_mov_b32_e32 v1, s9
+; SI-NEXT:    v_mov_b32_e32 v2, s10
+; SI-NEXT:    v_mov_b32_e32 v3, s11
+; SI-NEXT:    v_mov_b32_e32 v4, s12
 ; SI-NEXT:    v_mov_b32_e32 v5, s13
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 4
-; SI-NEXT:    v_cndmask_b32_e32 v5, v4, v5, vcc
-; SI-NEXT:    v_mov_b32_e32 v8, s12
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_mov_b32 s2, -1
-; SI-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
+; SI-NEXT:    v_mov_b32_e32 v6, s14
+; SI-NEXT:    v_mov_b32_e32 v7, s15
+; SI-NEXT:    s_mov_b32 m0, s4
+; SI-NEXT:    v_movreld_b32_e32 v0, v8
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: dynamic_insertelement_v8f32:
 ; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s6, s[4:5], 0x40
-; VI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x20
-; VI-NEXT:    v_mov_b32_e32 v4, 0x40a00000
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
+; VI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x20
+; VI-NEXT:    s_load_dword s4, s[4:5], 0x40
+; VI-NEXT:    v_mov_b32_e32 v8, 0x40a00000
 ; VI-NEXT:    s_mov_b32 s3, 0x1100f000
+; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_cmp_lg_u32 s6, 3
-; VI-NEXT:    v_mov_b32_e32 v0, s11
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 2
-; VI-NEXT:    v_cndmask_b32_e32 v3, v4, v0, vcc
-; VI-NEXT:    v_mov_b32_e32 v0, s10
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 1
-; VI-NEXT:    v_cndmask_b32_e32 v2, v4, v0, vcc
-; VI-NEXT:    v_mov_b32_e32 v0, s9
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 0
-; VI-NEXT:    v_cndmask_b32_e32 v1, v4, v0, vcc
 ; VI-NEXT:    v_mov_b32_e32 v0, s8
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 7
-; VI-NEXT:    v_cndmask_b32_e32 v0, v4, v0, vcc
-; VI-NEXT:    v_mov_b32_e32 v5, s15
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 6
-; VI-NEXT:    v_cndmask_b32_e32 v7, v4, v5, vcc
-; VI-NEXT:    v_mov_b32_e32 v5, s14
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 5
-; VI-NEXT:    v_cndmask_b32_e32 v6, v4, v5, vcc
+; VI-NEXT:    v_mov_b32_e32 v1, s9
+; VI-NEXT:    v_mov_b32_e32 v2, s10
+; VI-NEXT:    v_mov_b32_e32 v3, s11
+; VI-NEXT:    v_mov_b32_e32 v4, s12
 ; VI-NEXT:    v_mov_b32_e32 v5, s13
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_cmp_lg_u32 s6, 4
-; VI-NEXT:    v_cndmask_b32_e32 v5, v4, v5, vcc
-; VI-NEXT:    v_mov_b32_e32 v8, s12
-; VI-NEXT:    s_cselect_b64 vcc, -1, 0
-; VI-NEXT:    s_mov_b32 s2, -1
-; VI-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
+; VI-NEXT:    v_mov_b32_e32 v6, s14
+; VI-NEXT:    v_mov_b32_e32 v7, s15
+; VI-NEXT:    s_mov_b32 m0, s4
+; VI-NEXT:    v_movreld_b32_e32 v0, v8
 ; VI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %vecins = insertelement <8 x float> %a, float 5.000000e+00, i32 %b
   store <8 x float> %vecins, <8 x float> addrspace(1)* %out, align 32
   ret void
 }
 
; VI-NEXT:    s_endpgm
   %vecins = insertelement <4 x i32> %a, i32 %val, i32 %b
   store <4 x i32> %vecins, <4 x i32> addrspace(1)* %out, align 16
   ret void
 }
 
 define amdgpu_kernel void @dynamic_insertelement_v8i32(<8 x i32> addrspace(1)* %out, <8 x i32> %a, i32 %b) nounwind {
 ; SI-LABEL: dynamic_insertelement_v8i32:
 ; SI:       ; %bb.0:
-; SI-NEXT:    s_load_dword s6, s[4:5], 0x10
 ; SI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x8
+; SI-NEXT:    s_load_dword s6, s[4:5], 0x10
 ; SI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; SI-NEXT:    s_mov_b32 s3, 0x100f000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
-; SI-NEXT:    s_cmp_lg_u32 s6, 3
-; SI-NEXT:    v_mov_b32_e32 v0, s11
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 2
-; SI-NEXT:    v_cndmask_b32_e32 v3, 5, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, s10
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 1
-; SI-NEXT:    v_cndmask_b32_e32 v2, 5, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v0, s9
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 0
-; SI-NEXT:    v_cndmask_b32_e32 v1, 5, v0, vcc
 ; SI-NEXT:    v_mov_b32_e32 v0, s8
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 7
-; SI-NEXT:    v_cndmask_b32_e32 v0, 5, v0, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, s15
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 6
-; SI-NEXT:    v_cndmask_b32_e32 v7, 5, v4, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, s14
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 5
-; SI-NEXT:    v_cndmask_b32_e32 v6, 5, v4, vcc
-; SI-NEXT:    v_mov_b32_e32 v4, s13
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    s_cmp_lg_u32 s6, 4
-; SI-NEXT:    v_cndmask_b32_e32 v5, 5, v4, vcc
+; SI-NEXT:    v_mov_b32_e32 v1, s9
+; SI-NEXT:    v_mov_b32_e32 v2, s10
+; SI-NEXT:    v_mov_b32_e32 v3, s11
 ; SI-NEXT:    v_mov_b32_e32 v4, s12
-; SI-NEXT:    s_cselect_b64 vcc, -1, 0
-; SI-NEXT:    v_cndmask_b32_e32 v4, 5, v4, vcc
+; SI-NEXT:    v_mov_b32_e32 v5, s13
+; SI-NEXT:    v_mov_b32_e32 v6, s14
+; SI-NEXT:    v_mov_b32_e32 v7, s15
+; SI-NEXT:    s_mov_b32 m0, s6
+; SI-NEXT:    v_movreld_b32_e32 v0, 5
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: dynamic_insertelement_v8i32:
 ; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s6, s[4:5], 0x40
 ; VI-NEXT:    s_load_dwordx8 s[8:15], s[4:5], 0x20
+; VI-NEXT:    s_load_dword s6, s[4:5], 0x40
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; VI-NEXT:    s_mov_b32 s3, 0x1100f000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_cmp_lg_u32 s6, 3
-; VI-NEXT:    s_cselect_b32 s4, s11, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 2
-; VI-NEXT:    s_cselect_b32 s5, s10, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 1
-; VI-NEXT:    s_cselect_b32 s7, s9, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 0
-; VI-NEXT:    s_cselect_b32 s8, s8, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 7
-; VI-NEXT:    s_cselect_b32 s9, s15, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 6
-; VI-NEXT:    s_cselect_b32 s10, s14, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 5
-; VI-NEXT:    s_cselect_b32 s11, s13, 5
-; VI-NEXT:    s_cmp_lg_u32 s6, 4
-; VI-NEXT:    s_cselect_b32 s6, s12, 5
-; VI-NEXT:    v_mov_b32_e32 v0, s6
-; VI-NEXT:    v_mov_b32_e32 v1, s11
-; VI-NEXT:    v_mov_b32_e32 v2, s10
-; VI-NEXT:    v_mov_b32_e32 v3, s9
-; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0 offset:16
-; VI-NEXT:    s_nop 0
 ; VI-NEXT:    v_mov_b32_e32 v0, s8
-; VI-NEXT:    v_mov_b32_e32 v1, s7
-; VI-NEXT:    v_mov_b32_e32 v2, s5
-; VI-NEXT:    v_mov_b32_e32 v3, s4
+; VI-NEXT:    v_mov_b32_e32 v1, s9
+; VI-NEXT:    v_mov_b32_e32 v2, s10
+; VI-NEXT:    v_mov_b32_e32 v3, s11
+; VI-NEXT:    v_mov_b32_e32 v4, s12
+; VI-NEXT:    v_mov_b32_e32 v5, s13
+; VI-NEXT:    v_mov_b32_e32 v6, s14
+; VI-NEXT:    v_mov_b32_e32 v7, s15
+; VI-NEXT:    s_mov_b32 m0, s6
+; VI-NEXT:    v_movreld_b32_e32 v0, 5
+; VI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
 ; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %vecins = insertelement <8 x i32> %a, i32 5, i32 %b
   store <8 x i32> %vecins, <8 x i32> addrspace(1)* %out, align 32
   ret void
 }
 
 define amdgpu_kernel void @dynamic_insertelement_v16i32(<16 x i32> addrspace(1)* %out, <16 x i32> %a, i32 %b) nounwind {