[AMDGPU][GISel] Legalize V2S16 G_BUILD_VECTOR

[Pierre-vh]

Summary

Preparation patch for D134354 to make V2S16 G_BUILD_VECTOR legal.
Also removes RegBankInfo's scalarization of small BUILD_VECTORs,
replacing it with InstructionSelector logic instead.

This allows for V2S16 BUILD_VECTOR instructions to survive
all the way to ISel so we can select FMA/MAD_MIX instructions
in D134354.

[foad]

Looks like there are some code quality regressions to address?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

1533–1534

Isn't this redundant with line 1529?

llvm/test/CodeGen/AMDGPU/GlobalISel/andn2.ll

521–522 ↗

(On Diff #462152)

Regression here.

llvm/test/CodeGen/AMDGPU/GlobalISel/clamp-minmax-const-combine.ll

49–51 ↗

(On Diff #462152)

Regression here.

[Pierre-vh]

Thank you for the very quick review, indeed the patch isn't perfect yet (hence why I didn't add reviewers yet :) )
I fixed the regressions you pointed out, there's probably a lot more since the patch is
pretty large so I'll do a few more passes over the diff before I add reviewers.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

91–95 ↗

(On Diff #462152)

This combine can be a separate patch

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

729

This shouldn't change

1475–1492

This mostly looks like unwelcome clang-format changes. I think the only thing that really changed here was the S32->S16?

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

341–342 ↗

(On Diff #462152)

In the future we may want to handle <2 x i32> and <4 x i16>, at least for gfx940

368–369 ↗

(On Diff #462152)

Probably should have an asserting getRegBank to go along with getRegClass instead of littering asserts like this

373 ↗

(On Diff #462152)

You can do auto Const = B.buildConstant(S32,...); MRI.setRegBank(Const.getReg(0))

[Pierre-vh]

Update, address comments

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

635–638

We don't want to die on AGPR vectors. Should return false if not handling them for now

642–644

We could technically do this in tablegen. Not sure why this was manual in the DAG path

687

D134463 switches to using v_perm_b32 here. Most everything in this function should be handled by tablegen though

[Pierre-vh]

For this to land, are there things I must migrate to TableGen or is the diff in good shape and just needs a bit of polishing ?

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

91–95 ↗

(On Diff #462152)

Actually it was a leftover from a previous attempt. I totally removed the combine now and refactored the InstructionSelection logic to fix the testcases the combine intended to fix. Seems more stable/cleaner.

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

642–644

I think there are some annoying cases like a copy in-between the constant and the build_vector that make it annoying to handle in TableGen

Is it fine to leave this in Cpp or do I need to migrate this to TableGen for this to land? The diff is already quite large and I wanted to avoid moving things between Cpp/TableGen to avoid adding more complexity to it

687

Do you mean that D134463 will make this code path obsolete/dead? Do I need to rebase on top of it?
My understanding is that this won't interfere with D134463 as that diff adds tablegen patterns (which are matched above) and this code path only triggers when tablegen doesn't match

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

729

This shouldn't change

Without this, we get the following IR post-legalization:

`
  %86:_(<2 x s16>) = G_FCANONICALIZE %126:_
  %87:_(<2 x s16>) = G_FCANONICALIZE %124:_
  %82:_(<2 x s16>) = G_FMAXNUM_IEEE %86:_, %87:_
  %84:_(<2 x s16>) = G_FCANONICALIZE %127:_
  %85:_(<2 x s16>) = G_FCANONICALIZE %125:_
  %83:_(<2 x s16>) = G_FMAXNUM_IEEE %84:_, %85:_
  %151:_(s32) = G_BITCAST %82:_(<2 x s16>)
  %74:_(s16) = G_TRUNC %151:_(s32)
  %152:_(s32) = G_LSHR %151:_, %135:_(s32)
  %75:_(s16) = G_TRUNC %152:_(s32)
  %153:_(s32) = G_BITCAST %83:_(<2 x s16>)
  %76:_(s16) = G_TRUNC %153:_(s32)
  %130:_(<2 x s16>) = G_BUILD_VECTOR %74:_(s16), %75:_(s16)
  %131:_(<2 x s16>) = G_BUILD_VECTOR %76:_(s16), %40:_(s16)
  %128:_(<2 x s16>) = G_BUILD_VECTOR %33:_(s16), %33:_(s16)
  %129:_(<2 x s16>) = G_BUILD_VECTOR %33:_(s16), %40:_(s16)
  %63:_(<2 x s16>) = G_FCANONICALIZE %130:_
  %64:_(<2 x s16>) = G_FCANONICALIZE %128:_
  %59:_(<2 x s16>) = G_FMINNUM_IEEE %63:_, %64:_
  %61:_(<2 x s16>) = G_FCANONICALIZE %131:_
  %62:_(<2 x s16>) = G_FCANONICALIZE %129:_
  %60:_(<2 x s16>) = G_FMINNUM_IEEE %61:_, %62:_

in the following test:

; GCN-LABEL: {{^}}v_mad_mix_v3f32_clamp_postcvt:
; GCN: s_waitcnt
; GFX900-DAG: v_mad_mixlo_f16 v{{[0-9]+}}, v0, v2, v4 op_sel_hi:[1,1,1] clamp
; GFX900-DAG: v_mad_mixhi_f16 v{{[0-9]+}}, v0, v2, v4 op_sel:[1,1,1] op_sel_hi:[1,1,1] clamp
; GFX900-DAG: v_mad_mixlo_f16 v{{[0-9]+}}, v1, v3, v5 op_sel_hi:[1,1,1] clamp

; GFX906-DAG: v_fma_mixlo_f16 v{{[0-9]+}}, v0, v2, v4 op_sel_hi:[1,1,1] clamp
; GFX906-DAG: v_fma_mixhi_f16 v{{[0-9]+}}, v0, v2, v4 op_sel:[1,1,1] op_sel_hi:[1,1,1] clamp
; GFX906-DAG: v_fma_mixlo_f16 v{{[0-9]+}}, v1, v3, v5 op_sel_hi:[1,1,1] clamp

; GFX9: v_mov_b32_e32 v0, v{{[0-9]+}}
; GFX9-NEXT: s_setpc_b64
define <3 x half> @v_mad_mix_v3f32_clamp_postcvt(<3 x half> %src0, <3 x half> %src1, <3 x half> %src2) #0 {
  %src0.ext = fpext <3 x half> %src0 to <3 x float>
  %src1.ext = fpext <3 x half> %src1 to <3 x float>
  %src2.ext = fpext <3 x half> %src2 to <3 x float>
  %result = tail call <3 x float> @llvm.fmuladd.v3f32(<3 x float> %src0.ext, <3 x float> %src1.ext, <3 x float> %src2.ext)
  %cvt.result = fptrunc <3 x float> %result to <3 x half>
  %max = call <3 x half> @llvm.maxnum.v3f16(<3 x half> %cvt.result, <3 x half> zeroinitializer)
  %clamp = call <3 x half> @llvm.minnum.v3f16(<3 x half> %max, <3 x half> <half 1.0, half 1.0, half 1.0>)
  ret <3 x half> %clamp
}

I can see three possibilities why:

• The legalizer rule should go (easiest, it's why I did that for now)
• The FPMinMadToClamp combine should be pre-legalizer instead of post-regbankcombiner
• This test is wrong/shouldn't fold

What do you prefer?

[Pierre-vh]

(Some) comments

[Pierre-vh]

Rebase

[foad]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

Using buffer_store/load instead of shifts for non-constant indices is a big regression.

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

This seems to be coming from the change in the legalizer of G_EXTRACT_VECTOR_ELT at line 1445. Reverting it fixes it.
It's because with that change, it gets lowered to the stack, so a G_STORE is emitted and it causes what we're seeing here.

However if I revert the change, it becomes impossible to match mad_mix in v_mad_mix_v2f32_clamp_postcvt_lo, its G_INSERT_VECTOR_ELT/G_EXTRACT_VECTOR_ELT get lowered to the point that we can't even match the CLAMP.

Perhaps I can add a condition to not bitcast if it's a 2x16 vector (vectorWiderThan(VecTypeIdx, 32)) ? It works, but seems like a bandaid fix though, no?
@arsenm thoughts?

[foad]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

I would hope that it only gets lowered to memory accesses if the index is non-constant, and for matching mad_mix you are only interested in constant indices. Does that give you a way to fix both cases?

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

LegalityQuery does not seem to contain any information about whether the indice is constant unfortunately, so it doesn't look like I can make that distinction in the legalizer unless I rewrite some of the legalization logic in a custom handler. I'd like to get @arsenm 's input first before implementing that though

I would hope that it only gets lowered to memory accesses if the index is non-constant

That seems a bit orthogonal to the current issue though, the legalizer currently doesn't make a distinction based on the const-ness of the indice (I think) so maybe it's best to not start taking it into account here, as it may have other implications. I feel like the vectorWiderThan(VecTypeIdx, 32) solution would be fine for now even if it's a bit hacky
I'll take another look at the situation tomorrow and see if I can come up with another fix

[Pierre-vh]

I reverted the extract_element legalizer changes because it's too sensitive to changes. I tried to do custom legalization or just applying the new rule to v2s16, but it still leaves huge regressions.
It messes up one test case in D134354 (v_mad_mix_v2f32_clamp_postcvt_lo). The changes are pretty bad and I'm not sure it's doable to pattern-match the MIR to get the same result as the DAG (I'll keep looking into it if I have time though)

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

Legality cannot be context dependent. You can't have some cases be legal or not based on regular value operands.

I think treating the legalization different isn't the right strategy. In the DAG path we have combines to turn 16-bit vector operations into 32-bit vector operations with some shifts to extract of the elements. With optimizations, you wouldn't really ever use the 16->32 vector path

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

So the latest changes I made to the legalizer are good, and the remaining changes need to be in a pre-legalizer combine for V2S16 INSERT/EXTRACT_VECTOR_ELT ?

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

So, the combine I was thinking of seems to be already done as the bitcast lowering for packed vectors (as is done in the old code). You should continue to use the bitcast lowering for vectors. You only want G_BUILD_VECTOR to be legal specifically for <2 x s16>

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

G_BUILD_VECTOR legalization isn't a pain point for now, the current rules seem to be working but I can surely restrict it to 2x16 only.
The current problem is with G_EXTRACT_VECTOR_ELT/G_INSERT_VECTOR_ELT, for matching the mad_mix stuff they need to be lowered (.lowerFor({V2S16, S16}) does the trick), but it introduces huge regressions in insertelement.i16 where some bitwise ops become stack operations with load/stores/etc. What can be done there?

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

Yes, lower for these uses the stack (We could also add a compare and select path). The 16-bit vectors should continue to use the bitcast lowering (I'm not actually sure why you are touching these)

[Pierre-vh]

Rebase

[Petar.Avramovic]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

9

There are too many changes in this patch. I am stuck on legalizer changes.
Maybe something from this stack of patches D109242 would be helpful.
Can you add test with G_EXTRACT_VECTOR_ELT/G_INSERT_VECTOR_ELT you mentioned and how you would like that to look like?

[Pierre-vh]

@Petar.Avramovic In the latest diffs, I don't change legalization rules from G_EXTRACT/INSERT_VECTOR_ELEMENT anymore, so it doesn't affect this patch
Instead I started D134870 which adds combines to achieve the fma/mad_mix selection in D134354

[Petar.Avramovic]

You don't want (<2 x s16>) but only s16 G_FMAXNUM_IEEEwhen lowering larger vectors, also you remove G_BUILD_VECTOR_TRUNC but use G_BUILD_VECTOR instead.
Can you point me to the test case that requires those changes? (You didn't change legalization rules for other <2 x s16> instructions).
To be more precise what D134354 requires from Legalizer (and pre/post legalizer combiner). Maybe there is simpler way to achieve it.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

626–630

Theoretically the type checks should be unnecessary. If the legality rules are correct the verification should reject the illegally typed operations

674–675

Should look into dropping this next

687

All of this code should really go through the same tablegen patterns. A future change should try to get rid of this custom code. The only custom selected case in the DAG path is for constants (and even that could be moved to tablegen)

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

729

The goal is to produce the packed minnum/maxnum. The mess around it is for other combines

[Pierre-vh]

In D134433#3824171, @Petar.Avramovic wrote:

You don't want (<2 x s16>) but only s16 G_FMAXNUM_IEEEwhen lowering larger vectors, also you remove G_BUILD_VECTOR_TRUNC but use G_BUILD_VECTOR instead.
Can you point me to the test case that requires those changes? (You didn't change legalization rules for other <2 x s16> instructions).
To be more precise what D134354 requires from Legalizer (and pre/post legalizer combiner). Maybe there is simpler way to achieve it.

D134354 requires G_BUILD_VECTOR V2S16 to be legal, we tried other options first that didn't involve legalizer changes but we discussed with @arsenm and concluded that legalizer changes were needed to support G_BUILD_VECTOR V2S16, which is what this diff is doing.
The G_FMINNUM change seems to be needed else the following:

; GISEL-GFX900-LABEL: v_mad_mix_v3f32_clamp_postcvt:
; GISEL-GFX900:       ; %bb.0:
; GISEL-GFX900-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GISEL-GFX900-NEXT:    v_mad_mixlo_f16 v6, v0, v2, v4 op_sel_hi:[1,1,1] clamp
; GISEL-GFX900-NEXT:    v_mad_mixhi_f16 v6, v0, v2, v4 op_sel:[1,1,1] op_sel_hi:[1,1,1] clamp
; GISEL-GFX900-NEXT:    v_mad_mixlo_f16 v1, v1, v3, v5 op_sel_hi:[1,1,1] clamp
; GISEL-GFX900-NEXT:    v_mov_b32_e32 v0, v6
; GISEL-GFX900-NEXT:    s_setpc_b64 s[30:31]

becomes

; GISEL-GFX900-LABEL: v_mad_mix_v3f32_clamp_postcvt:
; GISEL-GFX900:       ; %bb.0:
; GISEL-GFX900-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GISEL-GFX900-NEXT:    v_mad_mixlo_f16 v6, v0, v2, v4 op_sel_hi:[1,1,1]
; GISEL-GFX900-NEXT:    v_mad_mixhi_f16 v6, v0, v2, v4 op_sel:[1,1,1] op_sel_hi:[1,1,1]
; GISEL-GFX900-NEXT:    v_pk_max_f16 v0, v6, 0
; GISEL-GFX900-NEXT:    v_lshrrev_b32_e32 v2, 16, v0
; GISEL-GFX900-NEXT:    v_and_b32_e32 v0, 0xffff, v0
; GISEL-GFX900-NEXT:    v_mad_mixlo_f16 v1, v1, v3, v5 op_sel_hi:[1,1,1]
; GISEL-GFX900-NEXT:    v_lshl_or_b32 v0, v2, 16, v0
; GISEL-GFX900-NEXT:    v_pk_max_f16 v1, v1, v1
; GISEL-GFX900-NEXT:    v_pk_max_f16 v0, v0, v0
; GISEL-GFX900-NEXT:    v_pk_max_f16 v1, v1, 0
; GISEL-GFX900-NEXT:    v_pk_min_f16 v0, v0, 1.0 op_sel_hi:[1,0]
; GISEL-GFX900-NEXT:    v_pk_max_f16 v1, v1, v1
; GISEL-GFX900-NEXT:    v_lshrrev_b32_e32 v2, 16, v0
; GISEL-GFX900-NEXT:    v_and_b32_e32 v0, 0xffff, v0
; GISEL-GFX900-NEXT:    v_pk_min_f16 v1, v1, 1.0
; GISEL-GFX900-NEXT:    v_lshl_or_b32 v0, v2, 16, v0
; GISEL-GFX900-NEXT:    s_setpc_b64 s[30:31]
`

If it's really a blocker ( @arsenm ?) I can take another look at it but it seems like leaving it in causes a lot of trouble

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

674–675

Do you mean moving it into a combine? In this patch or a future patch?

687

Is it fine to keep as-is for now, or do I need to move it all to tablegen for this to land?
I don't mind going back to it later but currently there's a lot of diffs open for mad_mix and it's becoming more and more difficult to differentiate essential changes from "nice improvements". If the current version is acceptable I'd rather keep it for now, is that ok?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

729

I'm not sure I understand, is the legalizer change wrong and I need to look into adding more combines to remove the extra instructions?
What kind of combine can be done there? I don't see anything obvious

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

687

Yes, moving to tablegen is a separate change for later

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

729

<2 x s16> minnum/maxnum are legal with VOP3P and the legalizer rules should express this and try to legalize wider vectors to <2 x s16> pieces. The MIR here does correctly produce the <2 x s16> operations. The additional context in this particular test isn't relevant to what the legalizer rules here should be (I'm not seeing what the problem is here, other than the legalized MIR has some vector conversion mess)

G_BUILD_VECTOR (TRUNC (BITCAST x)), (TRUNC (LSHR (BITCAST x), 16)) is an identity that can fold out

[Petar.Avramovic]

Does D134354 work for non vector cases?
My best guess is that it does not work for vectors of two elements where it would need to create two instructions (missing some build vector combine/legalizer rule). You would expect that all those build vectors in between are gone after legalizer?

[Petar.Avramovic]

Just to point out most notable regression:

define amdgpu_ps <3 x half> @min3(<3 x half> %src0, <3 x half> %src1) {
  %min3 = call <3 x half> @llvm.minnum.v3f16(<3 x half> %src0, <3 x half> %src1)
  ret <3 x half> %min3
}

declare <3 x half> @llvm.minnum.v3f16(<3 x half>, <3 x half>)

goes from

	v_pk_min_f16 v0, v0, v2
	v_pk_min_f16 v1, v1, v3

to

	v_min_f16_e32 v4, v0, v2
	v_min_f16_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
	v_min_f16_e32 v1, v1, v3
	v_lshl_or_b32 v0, v0, 16, v4

D134354 does mad_mix combine for straightforward test cases same as sdag. The test you mentioned here v_mad_mix_v3f32_clamp_postcvt
does the mad_mix selection in the same way in both cases, the difference you pointed out is clamp that failed to combine, i.e you don't need this patch in order to select mad_mix.

to get same result as sdag you will need:
combine for clamp for v2f16 where one element is undef
maybe to not lower build_vector_trunc in regbankselect so that it is easier to look through, it is easy to select.

Overall it is probably best to go with D134354 first and then optimize code.

[Pierre-vh]

Remove G_FMIN/MAXNUM legalizer changes.
This now only touches build_vector. I renamed the diff to reflect that.

[arsenm]

LGTM

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

for (int I = 0, E = NumDst; I != E; ++I) {
     if (DstRC && !RBI.constrainGenericRegister(Dst.getReg(), *DstRC, *MRI))
       return false;
   }
 
   MI.eraseFromParent();
   return true;
 }
 
-bool AMDGPUInstructionSelector::selectG_BUILD_VECTOR_TRUNC(
-  MachineInstr &MI) const {
-  if (selectImpl(MI, *CoverageInfo))
-    return true;
+bool AMDGPUInstructionSelector::selectG_BUILD_VECTOR(MachineInstr &MI) const {
+  assert(MI.getOpcode() == AMDGPU::G_BUILD_VECTOR_TRUNC ||
+         MI.getOpcode() == AMDGPU::G_BUILD_VECTOR);
 
-  const LLT S32 = LLT::scalar(32);
-  const LLT V2S16 = LLT::fixed_vector(2, 16);
+  Register Src0 = MI.getOperand(1).getReg();
+  Register Src1 = MI.getOperand(2).getReg();
+  LLT SrcTy = MRI->getType(Src0);
+  const unsigned SrcSize = SrcTy.getSizeInBits();
 
+  // BUILD_VECTOR with >=32 bits source is handled by MERGE_VALUE.
+  if (MI.getOpcode() == AMDGPU::G_BUILD_VECTOR && SrcSize >= 32) {
+    return selectG_MERGE_VALUES(MI);
+  }
+
+  // Selection logic below is for V2S16 only.
+  // For G_BUILD_VECTOR_TRUNC, additionally check that the operands are s32.
   Register Dst = MI.getOperand(0).getReg();
-  if (MRI->getType(Dst) != V2S16)
-    return false;
+  if (MRI->getType(Dst) != LLT::fixed_vector(2, 16) ||
+      (MI.getOpcode() == AMDGPU::G_BUILD_VECTOR_TRUNC &&
+       SrcTy != LLT::scalar(32)))
+    return selectImpl(MI, *CoverageInfo);
 
   const RegisterBank *DstBank = RBI.getRegBank(Dst, *MRI, TRI);
-  if (DstBank->getID() != AMDGPU::SGPRRegBankID)
+  if (DstBank->getID() == AMDGPU::AGPRRegBankID)
     return false;
 
-  Register Src0 = MI.getOperand(1).getReg();
-  Register Src1 = MI.getOperand(2).getReg();
-  if (MRI->getType(Src0) != S32)
-    return false;
+  assert(DstBank->getID() == AMDGPU::SGPRRegBankID ||
+         DstBank->getID() == AMDGPU::VGPRRegBankID);
+  const bool IsVector = DstBank->getID() == AMDGPU::VGPRRegBankID;
 
   const DebugLoc &DL = MI.getDebugLoc();
   MachineBasicBlock *BB = MI.getParent();
 
+  // First, before trying TableGen patterns, check if both sources are
+  // constants. In those cases, we can trivially compute the final constant
+  // and emit a simple move.
   auto ConstSrc1 = getAnyConstantVRegValWithLookThrough(Src1, *MRI, true, true);
   if (ConstSrc1) {
     auto ConstSrc0 =
         getAnyConstantVRegValWithLookThrough(Src0, *MRI, true, true);
     if (ConstSrc0) {
       const int64_t K0 = ConstSrc0->Value.getSExtValue();
       const int64_t K1 = ConstSrc1->Value.getSExtValue();
       uint32_t Lo16 = static_cast<uint32_t>(K0) & 0xffff;
       uint32_t Hi16 = static_cast<uint32_t>(K1) & 0xffff;
+      uint32_t Imm = Lo16 | (Hi16 << 16);
+
+      // VALU
+      if (IsVector) {
+        BuildMI(*BB, &MI, DL, TII.get(AMDGPU::V_MOV_B32_e32), Dst).addImm(Imm);
+        MI.eraseFromParent();
+        return RBI.constrainGenericRegister(Dst, AMDGPU::VGPR_32RegClass, *MRI);
+      }
 
-      BuildMI(*BB, &MI, DL, TII.get(AMDGPU::S_MOV_B32), Dst)
-        .addImm(Lo16 | (Hi16 << 16));
+      // SALU
+      BuildMI(*BB, &MI, DL, TII.get(AMDGPU::S_MOV_B32), Dst).addImm(Imm);
       MI.eraseFromParent();
       return RBI.constrainGenericRegister(Dst, AMDGPU::SReg_32RegClass, *MRI);
     }
   }
 
+  // Now try TableGen patterns.
+  if (selectImpl(MI, *CoverageInfo))
+    return true;
+
   // TODO: This should probably be a combine somewhere
-  // (build_vector_trunc $src0, undef -> copy $src0
+  // (build_vector $src0, undef)  -> copy $src0
   MachineInstr *Src1Def = getDefIgnoringCopies(Src1, *MRI);
   if (Src1Def && Src1Def->getOpcode() == AMDGPU::G_IMPLICIT_DEF) {
     MI.setDesc(TII.get(AMDGPU::COPY));
     MI.removeOperand(2);
-    return RBI.constrainGenericRegister(Dst, AMDGPU::SReg_32RegClass, *MRI) &&
-           RBI.constrainGenericRegister(Src0, AMDGPU::SReg_32RegClass, *MRI);
+    const auto &RC =
+        IsVector ? AMDGPU::VGPR_32RegClass : AMDGPU::SReg_32RegClass;
+    return RBI.constrainGenericRegister(Dst, RC, *MRI) &&
+           RBI.constrainGenericRegister(Src0, RC, *MRI);
+  }
+
+  // TODO: Can be improved?
+  if (IsVector) {
+    Register TmpReg = MRI->createVirtualRegister(&AMDGPU::VGPR_32RegClass);
+    BuildMI(*BB, MI, DL, TII.get(AMDGPU::V_AND_B32_e32), TmpReg)
+        .addImm(0xFFFF)
+        .addReg(Src0);
+    BuildMI(*BB, MI, DL, TII.get(AMDGPU::V_LSHL_OR_B32_e64), Dst)
+        .addReg(Src1)
+        .addImm(16)
+        .addReg(TmpReg);
+    MI.eraseFromParent();
+    return true;
   }
 
   Register ShiftSrc0;
   Register ShiftSrc1;
 
   // With multiple uses of the shift, this will duplicate the shift and
   // increase register pressure.
   //
-  // (build_vector_trunc (lshr_oneuse $src0, 16), (lshr_oneuse $src1, 16)
+  // (build_vector (lshr_oneuse $src0, 16), (lshr_oneuse $src1, 16)
   //  => (S_PACK_HH_B32_B16 $src0, $src1)
-  // (build_vector_trunc (lshr_oneuse SReg_32:$src0, 16), $src1)
+  // (build_vector (lshr_oneuse SReg_32:$src0, 16), $src1)
   //  => (S_PACK_HL_B32_B16 $src0, $src1)
-  // (build_vector_trunc $src0, (lshr_oneuse SReg_32:$src1, 16))
+  // (build_vector $src0, (lshr_oneuse SReg_32:$src1, 16))
   //  => (S_PACK_LH_B32_B16 $src0, $src1)
-  // (build_vector_trunc $src0, $src1)
+  // (build_vector $src0, $src1)
   //  => (S_PACK_LL_B32_B16 $src0, $src1)
 
   bool Shift0 = mi_match(
       Src0, *MRI, m_OneUse(m_GLShr(m_Reg(ShiftSrc0), m_SpecificICst(16))));
 
   bool Shift1 = mi_match(
       Src1, *MRI, m_OneUse(m_GLShr(m_Reg(ShiftSrc1), m_SpecificICst(16))));
 
   unsigned Opc = AMDGPU::S_PACK_LL_B32_B16;
   if (Shift0 && Shift1) {
     Opc = AMDGPU::S_PACK_HH_B32_B16;
     MI.getOperand(1).setReg(ShiftSrc0);
     MI.getOperand(2).setReg(ShiftSrc1);
   } else if (Shift1) {
     Opc = AMDGPU::S_PACK_LH_B32_B16;
     MI.getOperand(2).setReg(ShiftSrc1);
   } else if (Shift0) {
+    auto ConstSrc1 =
+        getAnyConstantVRegValWithLookThrough(Src1, *MRI, true, true);
     if (ConstSrc1 && ConstSrc1->Value == 0) {
       // build_vector_trunc (lshr $src0, 16), 0 -> s_lshr_b32 $src0, 16
       auto MIB = BuildMI(*BB, &MI, DL, TII.get(AMDGPU::S_LSHR_B32), Dst)
                      .addReg(ShiftSrc0)
                      .addImm(16);
 
       MI.eraseFromParent();
       return constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
case TargetOpcode::G_FNEG:
     return selectG_FNEG(I);
   case TargetOpcode::G_FABS:
     if (selectImpl(I, *CoverageInfo))
       return true;
     return selectG_FABS(I);
   case TargetOpcode::G_EXTRACT:
     return selectG_EXTRACT(I);
   case TargetOpcode::G_MERGE_VALUES:
-  case TargetOpcode::G_BUILD_VECTOR:
   case TargetOpcode::G_CONCAT_VECTORS:
     return selectG_MERGE_VALUES(I);
   case TargetOpcode::G_UNMERGE_VALUES:
     return selectG_UNMERGE_VALUES(I);
+  case TargetOpcode::G_BUILD_VECTOR:
   case TargetOpcode::G_BUILD_VECTOR_TRUNC:
-    return selectG_BUILD_VECTOR_TRUNC(I);
+    return selectG_BUILD_VECTOR(I);
   case TargetOpcode::G_PTR_ADD:
     if (selectImpl(I, *CoverageInfo))
       return true;
     return selectG_PTR_ADD(I);
   case TargetOpcode::G_IMPLICIT_DEF:
     return selectG_IMPLICIT_DEF(I);
   case TargetOpcode::G_FREEZE:
     return selectCOPY(I);

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

 //===- AMDGPULegalizerInfo.cpp -----------------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file implements the targeting of the Machinelegalizer class for
AMDGPULegalizerInfo::AMDGPULegalizerInfo(const GCNSubtarget &ST_,
   }
 
   auto &MinNumMaxNum = getActionDefinitionsBuilder({
       G_FMINNUM, G_FMAXNUM, G_FMINNUM_IEEE, G_FMAXNUM_IEEE});
 
   if (ST.hasVOP3PInsts()) {
     MinNumMaxNum.customFor(FPTypesPK16)
       .moreElementsIf(isSmallOddVector(0), oneMoreElement(0))
-      .clampMaxNumElements(0, S16, 2)
       .clampScalar(0, S16, S64)
       .scalarize(0);
   } else if (ST.has16BitInsts()) {
     MinNumMaxNum.customFor(FPTypes16)
       .clampScalar(0, S16, S64)
       .scalarize(0);
   } else {
     MinNumMaxNum.customFor(FPTypesBase)
getActionDefinitionsBuilder(Op)
           const LLT VecTy = Query.Types[VecTypeIdx];
           const LLT IdxTy = Query.Types[IdxTypeIdx];
           const unsigned EltSize = EltTy.getSizeInBits();
           return (EltSize == 32 || EltSize == 64) &&
                   VecTy.getSizeInBits() % 32 == 0 &&
                   VecTy.getSizeInBits() <= MaxRegisterSize &&
                   IdxTy.getSizeInBits() == 32;
         })
-      .bitcastIf(all(sizeIsMultipleOf32(VecTypeIdx), scalarOrEltNarrowerThan(VecTypeIdx, 32)),
+      .bitcastIf(all(sizeIsMultipleOf32(VecTypeIdx), scalarOrEltNarrowerThan(VecTypeIdx, 16)),
                  bitcastToVectorElement32(VecTypeIdx))
       //.bitcastIf(vectorSmallerThan(1, 32), bitcastToScalar(1))
       .bitcastIf(
         all(sizeIsMultipleOf32(VecTypeIdx), scalarOrEltWiderThan(VecTypeIdx, 64)),
         [=](const LegalityQuery &Query) {
           // For > 64-bit element types, try to turn this into a 64-bit
           // element vector since we may be able to do better indexing
           // if this is scalar. If not, fall back to 32.
           const LLT EltTy = Query.Types[EltTypeIdx];
           const LLT VecTy = Query.Types[VecTypeIdx];
           const unsigned DstEltSize = EltTy.getSizeInBits();
           const unsigned VecSize = VecTy.getSizeInBits();
 
           const unsigned TargetEltSize = DstEltSize % 64 == 0 ? 64 : 32;
           return std::make_pair(
               VecTypeIdx,
               LLT::fixed_vector(VecSize / TargetEltSize, TargetEltSize));
         })
-      .clampScalar(EltTypeIdx, S32, S64)
+      .clampScalar(EltTypeIdx, S16, S64)
       .clampScalar(VecTypeIdx, S32, S64)
       .clampScalar(IdxTypeIdx, S32, S32)
       .clampMaxNumElements(VecTypeIdx, S32, 32)
       // TODO: Clamp elements for 64-bit vectors?
       // It should only be necessary with variable indexes.
       // As a last resort, lower to the stack
       .lower();
   }
 
   getActionDefinitionsBuilder(G_EXTRACT_VECTOR_ELT)
     .unsupportedIf([=](const LegalityQuery &Query) {
         const LLT &EltTy = Query.Types[1].getElementType();
         return Query.Types[0] != EltTy;
       });
 
   for (unsigned Op : {G_EXTRACT, G_INSERT}) {
     unsigned BigTyIdx = Op == G_EXTRACT ? 1 : 0;
     unsigned LitTyIdx = Op == G_EXTRACT ? 0 : 1;
 
     // FIXME: Doesn't handle extract of illegal sizes.
     getActionDefinitionsBuilder(Op)
       .lowerIf(all(typeIs(LitTyIdx, S16), sizeIs(BigTyIdx, 32)))
       .lowerIf([=](const LegalityQuery &Query) {
           // Sub-vector(or single element) insert and extract.
           // TODO: verify immediate offset here since lower only works with
           // whole elements.
           const LLT BigTy = Query.Types[BigTyIdx];
           return BigTy.isVector();
         })
       // FIXME: Multiples of 16 should not be legal.
       .legalIf([=](const LegalityQuery &Query) {
           const LLT BigTy = Query.Types[BigTyIdx];
           const LLT LitTy = Query.Types[LitTyIdx];
           return (BigTy.getSizeInBits() % 32 == 0) &&
                  (LitTy.getSizeInBits() % 16 == 0);
         })
auto &BuildVector = getActionDefinitionsBuilder(G_BUILD_VECTOR)
     .clampNumElements(0, V16S32, V32S32)
     .clampNumElements(0, V2S64, V16S64)
     .fewerElementsIf(isWideVec16(0), changeTo(0, V2S16));
 
   if (ST.hasScalarPackInsts()) {
     BuildVector
       // FIXME: Should probably widen s1 vectors straight to s32
       .minScalarOrElt(0, S16)
-      // Widen source elements and produce a G_BUILD_VECTOR_TRUNC
-      .minScalar(1, S32);
+      .minScalar(1, S16);
 
     getActionDefinitionsBuilder(G_BUILD_VECTOR_TRUNC)
       .legalFor({V2S16, S32})
       .lower();
-    BuildVector.minScalarOrElt(0, S32);
   } else {
     BuildVector.customFor({V2S16, S16});
     BuildVector.minScalarOrElt(0, S32);
 
     getActionDefinitionsBuilder(G_BUILD_VECTOR_TRUNC)
       .customFor({V2S16, S32})
       .lower();
   }
 

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i16.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX8 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx1100 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX11 %s
 
 define amdgpu_ps i16 @extractelement_sgpr_v4i16_sgpr_idx(<4 x i16> addrspace(4)* inreg %ptr, i32 inreg %idx) {
-; GCN-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
-; GCN-NEXT:    s_lshr_b32 s2, s4, 1
-; GCN-NEXT:    s_cmp_eq_u32 s2, 1
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_cselect_b32 s0, s1, s0
-; GCN-NEXT:    s_and_b32 s1, s4, 1
-; GCN-NEXT:    s_lshl_b32 s1, s1, 4
-; GCN-NEXT:    s_lshr_b32 s0, s0, s1
-; GCN-NEXT:    ; return to shader part epilog
+; GFX9-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s10, -1
+; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
+; GFX9-NEXT:    s_add_u32 s8, s8, s5
+; GFX9-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-NEXT:    v_mov_b32_e32 v0, 8
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    s_and_b32 s0, s4, 3
+; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
+; GFX9-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:8
+; GFX9-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:12
+; GFX9-NEXT:    v_add_u32_e32 v0, s0, v0
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX9-NEXT:    ; return to shader part epilog
+;
+; GFX8-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
+; GFX8:       ; %bb.0:
+; GFX8-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s10, -1
+; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
+; GFX8-NEXT:    s_add_u32 s8, s8, s5
+; GFX8-NEXT:    s_addc_u32 s9, s9, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-NEXT:    s_and_b32 s0, s4, 3
+; GFX8-NEXT:    v_mov_b32_e32 v0, 8
+; GFX8-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX8-NEXT:    v_mov_b32_e32 v2, s1
+; GFX8-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:8
+; GFX8-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:12
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX8-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX7-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s10, -1
+; GFX7-NEXT:    s_mov_b32 s11, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s8, s8, s5
+; GFX7-NEXT:    s_addc_u32 s9, s9, 0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s0
+; GFX7-NEXT:    s_and_b32 s0, s4, 3
+; GFX7-NEXT:    v_mov_b32_e32 v0, 8
+; GFX7-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
+; GFX7-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:8
+; GFX7-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:12
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s0, v0
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
 ; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s10, -1
 ; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
-; GFX10-NEXT:    s_lshr_b32 s2, s4, 1
-; GFX10-NEXT:    s_cmp_eq_u32 s2, 1
+; GFX10-NEXT:    s_mov_b32 s11, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s8, s8, s5
+; GFX10-NEXT:    s_addc_u32 s9, s9, 0
+; GFX10-NEXT:    s_and_b32 s2, s4, 3
+; GFX10-NEXT:    s_lshl_b32 s2, s2, 1
+; GFX10-NEXT:    v_add_nc_u32_e64 v2, s2, 8
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    s_cselect_b32 s0, s1, s0
-; GFX10-NEXT:    s_and_b32 s1, s4, 1
-; GFX10-NEXT:    s_lshl_b32 s1, s1, 4
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s1
+; GFX10-NEXT:    buffer_store_dword v0, off, s[8:11], 0 offset:8
+; GFX10-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:12
+; GFX10-NEXT:    buffer_load_ushort v0, v2, s[8:11], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_sgpr_v4i16_sgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[2:3], 0x0
-; GFX11-NEXT:    s_lshr_b32 s2, s4, 1
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
-; GFX11-NEXT:    s_cmp_eq_u32 s2, 1
+; GFX11-NEXT:    s_and_b32 s2, s4, 3
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_lshl_b32 s2, s2, 1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    s_cselect_b32 s0, s1, s0
-; GFX11-NEXT:    s_and_b32 s1, s4, 1
-; GFX11-NEXT:    s_lshl_b32 s1, s1, 4
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    scratch_store_b64 off, v[0:1], off offset:8
+; GFX11-NEXT:    scratch_load_u16 v0, v2, off offset:8
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <4 x i16>, <4 x i16> addrspace(4)* %ptr
   %element = extractelement <4 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_vgpr_v4i16_sgpr_idx(<4 x i16> addrspace(1)* %ptr, i32 inreg %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v4i16_sgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    global_load_dwordx2 v[0:1], v[0:1], off
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX9-NEXT:    s_and_b32 s1, s2, 1
-; GFX9-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX9-NEXT:    s_lshl_b32 s0, s1, 4
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s7, 0xe00000
+; GFX9-NEXT:    s_add_u32 s4, s4, s3
+; GFX9-NEXT:    s_addc_u32 s5, s5, 0
+; GFX9-NEXT:    s_and_b32 s0, s2, 3
+; GFX9-NEXT:    v_mov_b32_e32 v2, 8
+; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:8
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:12
+; GFX9-NEXT:    v_add_u32_e32 v0, s0, v2
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
 ; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: extractelement_vgpr_v4i16_sgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; GFX8-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX8-NEXT:    s_and_b32 s1, s2, 1
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX8-NEXT:    s_lshl_b32 s0, s1, 4
+; GFX8-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s6, -1
+; GFX8-NEXT:    s_mov_b32 s7, 0xe80000
+; GFX8-NEXT:    s_add_u32 s4, s4, s3
+; GFX8-NEXT:    s_addc_u32 s5, s5, 0
+; GFX8-NEXT:    s_and_b32 s0, s2, 3
+; GFX8-NEXT:    v_mov_b32_e32 v2, 8
+; GFX8-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:8
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:12
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s0, v2
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
 ; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: extractelement_vgpr_v4i16_sgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; GFX7-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX7-NEXT:    s_and_b32 s1, s2, 1
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX7-NEXT:    s_lshl_b32 s0, s1, 4
+; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s6, -1
+; GFX7-NEXT:    s_mov_b32 s7, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s4, s4, s3
+; GFX7-NEXT:    s_addc_u32 s5, s5, 0
+; GFX7-NEXT:    s_and_b32 s0, s2, 3
+; GFX7-NEXT:    v_mov_b32_e32 v2, 8
+; GFX7-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:8
+; GFX7-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:12
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s0, v2
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
 ; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_vgpr_v4i16_sgpr_idx:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v[0:1], off
-; GFX10-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 1
-; GFX10-NEXT:    s_and_b32 s0, s2, 1
-; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s4, s4, s3
+; GFX10-NEXT:    s_addc_u32 s5, s5, 0
+; GFX10-NEXT:    s_and_b32 s0, s2, 3
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:8
+; GFX10-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:12
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX10-NEXT:    v_add_nc_u32_e64 v2, s0, 8
+; GFX10-NEXT:    buffer_load_ushort v0, v2, s[4:7], 0 offen
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
 ; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_vgpr_v4i16_sgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off
-; GFX11-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
-; GFX11-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 1
-; GFX11-NEXT:    s_and_b32 s0, s2, 1
-; GFX11-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
+; GFX11-NEXT:    s_and_b32 s0, s2, 3
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX11-NEXT:    v_mov_b32_e32 v2, s0
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    scratch_store_b64 off, v[0:1], off offset:8
+; GFX11-NEXT:    scratch_load_u16 v0, v2, off offset:8
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <4 x i16>, <4 x i16> addrspace(1)* %ptr
   %element = extractelement <4 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define i16 @extractelement_vgpr_v4i16_vgpr_idx(<4 x i16> addrspace(1)* %ptr, i32 %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v4i16_vgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dwordx2 v[0:1], v[0:1], off
-; GFX9-NEXT:    v_lshrrev_b32_e32 v3, 1, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v3
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX9-NEXT:    v_and_b32_e32 v2, 3, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], s32
+; GFX9-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4
+; GFX9-NEXT:    v_lshrrev_b32_e64 v1, 6, s32
+; GFX9-NEXT:    v_add_u32_e32 v0, v1, v2
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: extractelement_vgpr_v4i16_vgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 1, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v3
-; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX8-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX8-NEXT:    v_and_b32_e32 v2, 3, v2
+; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    buffer_store_dword v0, off, s[0:3], s32
+; GFX8-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4
+; GFX8-NEXT:    v_lshrrev_b32_e64 v1, 6, s32
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v2
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX7-LABEL: extractelement_vgpr_v4i16_vgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 1, v2
-; GFX7-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v3
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX7-NEXT:    v_and_b32_e32 v2, 3, v2
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    buffer_store_dword v0, off, s[0:3], s32
+; GFX7-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4
+; GFX7-NEXT:    v_lshr_b32_e64 v1, s32, 6
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v1, v2
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: extractelement_vgpr_v4i16_vgpr_idx:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    global_load_dwordx2 v[0:1], v[0:1], off
-; GFX10-NEXT:    v_lshrrev_b32_e32 v3, 1, v2
-; GFX10-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v3
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX10-NEXT:    v_and_b32_e32 v2, 3, v2
+; GFX10-NEXT:    v_lshrrev_b32_e64 v3, 5, s32
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dword v0, off, s[0:3], s32
+; GFX10-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4
+; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX10-NEXT:    v_add_nc_u32_e32 v2, v3, v2
+; GFX10-NEXT:    buffer_load_ushort v0, v2, s[0:3], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-LABEL: extractelement_vgpr_v4i16_vgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    global_load_b64 v[0:1], v[0:1], off
-; GFX11-NEXT:    v_lshrrev_b32_e32 v3, 1, v2
-; GFX11-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v3
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_dual_cndmask_b32 v0, v0, v1 :: v_dual_lshlrev_b32 v1, 4, v2
+; GFX11-NEXT:    v_and_b32_e32 v2, 3, v2
 ; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    scratch_store_b64 off, v[0:1], s32
+; GFX11-NEXT:    scratch_load_u16 v0, v2, s32
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
   %vector = load <4 x i16>, <4 x i16> addrspace(1)* %ptr
   %element = extractelement <4 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_sgpr_v4i16_vgpr_idx(<4 x i16> addrspace(4)* inreg %ptr, i32 %idx) {
-; GCN-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
-; GCN-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v1
-; GCN-NEXT:    v_and_b32_e32 v0, 1, v0
-; GCN-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v2, s0
-; GCN-NEXT:    v_mov_b32_e32 v3, s1
-; GCN-NEXT:    v_cndmask_b32_e32 v1, v2, v3, vcc
-; GCN-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
-; GCN-NEXT:    v_readfirstlane_b32 s0, v0
-; GCN-NEXT:    ; return to shader part epilog
+; GFX9-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_mov_b32 s8, s4
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX9-NEXT:    s_mov_b32 s7, 0xe00000
+; GFX9-NEXT:    s_add_u32 s4, s4, s8
+; GFX9-NEXT:    v_and_b32_e32 v0, 3, v0
+; GFX9-NEXT:    s_addc_u32 s5, s5, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:8
+; GFX9-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:12
+; GFX9-NEXT:    v_add_u32_e32 v0, 8, v0
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX9-NEXT:    ; return to shader part epilog
+;
+; GFX8-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
+; GFX8:       ; %bb.0:
+; GFX8-NEXT:    s_mov_b32 s8, s4
+; GFX8-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s6, -1
+; GFX8-NEXT:    s_mov_b32 s7, 0xe80000
+; GFX8-NEXT:    s_add_u32 s4, s4, s8
+; GFX8-NEXT:    v_and_b32_e32 v0, 3, v0
+; GFX8-NEXT:    s_addc_u32 s5, s5, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s1
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:8
+; GFX8-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:12
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 8, v0
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX8-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_mov_b32 s8, s4
+; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX7-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s6, -1
+; GFX7-NEXT:    s_mov_b32 s7, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s4, s4, s8
+; GFX7-NEXT:    v_and_b32_e32 v0, 3, v0
+; GFX7-NEXT:    s_addc_u32 s5, s5, 0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s0
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
+; GFX7-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:8
+; GFX7-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:12
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, 8, v0
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
 ; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_mov_b32 s8, s4
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX10-NEXT:    v_and_b32_e32 v0, 3, v0
 ; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[2:3], 0x0
-; GFX10-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v1
-; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s4, s4, s8
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX10-NEXT:    s_addc_u32 s5, s5, 0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 8, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, s1
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, s0, v2, vcc_lo
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
+; GFX10-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:8
+; GFX10-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:12
+; GFX10-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_sgpr_v4i16_vgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b64 s[0:1], s[2:3], 0x0
-; GFX11-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v1
+; GFX11-NEXT:    v_and_b32_e32 v2, 3, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    v_mov_b32_e32 v2, s1
-; GFX11-NEXT:    v_dual_cndmask_b32 v1, s0, v2 :: v_dual_and_b32 v0, 1, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; GFX11-NEXT:    v_lshlrev_b32_e32 v2, 1, v2
+; GFX11-NEXT:    scratch_store_b64 off, v[0:1], off offset:8
+; GFX11-NEXT:    scratch_load_u16 v0, v2, off offset:8
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <4 x i16>, <4 x i16> addrspace(4)* %ptr
   %element = extractelement <4 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_sgpr_v4i16_idx0(<4 x i16> addrspace(4)* inreg %ptr) {
 ; GCN-LABEL: extractelement_sgpr_v4i16_idx0:

 ; GFX11-NEXT:    v_lshrrev_b32_e32 v0, 16, v1
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
   %vector = load <4 x i16>, <4 x i16> addrspace(1)* %ptr
   %element = extractelement <4 x i16> %vector, i32 3
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_sgpr_v8i16_sgpr_idx(<8 x i16> addrspace(4)* inreg %ptr, i32 inreg %idx) {
-; GCN-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
-; GCN-NEXT:    s_lshr_b32 s5, s4, 1
-; GCN-NEXT:    s_cmp_eq_u32 s5, 1
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_cselect_b32 s0, s1, s0
-; GCN-NEXT:    s_cmp_eq_u32 s5, 2
-; GCN-NEXT:    s_cselect_b32 s0, s2, s0
-; GCN-NEXT:    s_cmp_eq_u32 s5, 3
-; GCN-NEXT:    s_cselect_b32 s0, s3, s0
-; GCN-NEXT:    s_and_b32 s1, s4, 1
-; GCN-NEXT:    s_lshl_b32 s1, s1, 4
-; GCN-NEXT:    s_lshr_b32 s0, s0, s1
-; GCN-NEXT:    ; return to shader part epilog
+; GFX9-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s10, -1
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
+; GFX9-NEXT:    s_add_u32 s8, s8, s5
+; GFX9-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-NEXT:    v_mov_b32_e32 v0, 16
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
+; GFX9-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:16
+; GFX9-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:20
+; GFX9-NEXT:    v_mov_b32_e32 v1, s2
+; GFX9-NEXT:    s_and_b32 s0, s4, 7
+; GFX9-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:24
+; GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX9-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:28
+; GFX9-NEXT:    v_add_u32_e32 v0, s0, v0
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX9-NEXT:    ; return to shader part epilog
+;
+; GFX8-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
+; GFX8:       ; %bb.0:
+; GFX8-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX8-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s10, -1
+; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
+; GFX8-NEXT:    s_add_u32 s8, s8, s5
+; GFX8-NEXT:    s_addc_u32 s9, s9, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s1
+; GFX8-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:16
+; GFX8-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:20
+; GFX8-NEXT:    v_mov_b32_e32 v1, s2
+; GFX8-NEXT:    s_and_b32 s0, s4, 7
+; GFX8-NEXT:    v_mov_b32_e32 v0, 16
+; GFX8-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:24
+; GFX8-NEXT:    v_mov_b32_e32 v1, s3
+; GFX8-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX8-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:28
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX8-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s10, -1
+; GFX7-NEXT:    s_mov_b32 s11, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s8, s8, s5
+; GFX7-NEXT:    s_addc_u32 s9, s9, 0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s0
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
+; GFX7-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:16
+; GFX7-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:20
+; GFX7-NEXT:    v_mov_b32_e32 v1, s2
+; GFX7-NEXT:    s_and_b32 s0, s4, 7
+; GFX7-NEXT:    v_mov_b32_e32 v0, 16
+; GFX7-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:24
+; GFX7-NEXT:    v_mov_b32_e32 v1, s3
+; GFX7-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX7-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:28
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s0, v0
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
 ; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s10, -1
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
-; GFX10-NEXT:    s_lshr_b32 s5, s4, 1
-; GFX10-NEXT:    s_cmp_eq_u32 s5, 1
+; GFX10-NEXT:    s_mov_b32 s11, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s8, s8, s5
+; GFX10-NEXT:    s_addc_u32 s9, s9, 0
+; GFX10-NEXT:    s_and_b32 s4, s4, 7
+; GFX10-NEXT:    s_lshl_b32 s4, s4, 1
+; GFX10-NEXT:    v_add_nc_u32_e64 v4, s4, 16
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    s_cselect_b32 s0, s1, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s5, 2
-; GFX10-NEXT:    s_cselect_b32 s0, s2, s0
-; GFX10-NEXT:    s_cmp_eq_u32 s5, 3
-; GFX10-NEXT:    s_cselect_b32 s0, s3, s0
-; GFX10-NEXT:    s_and_b32 s1, s4, 1
-; GFX10-NEXT:    s_lshl_b32 s1, s1, 4
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s1
+; GFX10-NEXT:    v_mov_b32_e32 v2, s2
+; GFX10-NEXT:    v_mov_b32_e32 v3, s3
+; GFX10-NEXT:    buffer_store_dword v0, off, s[8:11], 0 offset:16
+; GFX10-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:20
+; GFX10-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:24
+; GFX10-NEXT:    buffer_store_dword v3, off, s[8:11], 0 offset:28
+; GFX10-NEXT:    buffer_load_ushort v0, v4, s[8:11], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_sgpr_v8i16_sgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b128 s[0:3], s[2:3], 0x0
-; GFX11-NEXT:    s_lshr_b32 s5, s4, 1
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; GFX11-NEXT:    s_cmp_eq_u32 s5, 1
-; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    s_cselect_b32 s0, s1, s0
-; GFX11-NEXT:    s_cmp_eq_u32 s5, 2
-; GFX11-NEXT:    s_cselect_b32 s0, s2, s0
-; GFX11-NEXT:    s_cmp_eq_u32 s5, 3
-; GFX11-NEXT:    s_cselect_b32 s0, s3, s0
-; GFX11-NEXT:    s_and_b32 s1, s4, 1
+; GFX11-NEXT:    s_and_b32 s4, s4, 7
 ; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
-; GFX11-NEXT:    s_lshl_b32 s1, s1, 4
-; GFX11-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX11-NEXT:    s_lshl_b32 s4, s4, 1
+; GFX11-NEXT:    v_mov_b32_e32 v4, s4
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v1, s1
+; GFX11-NEXT:    v_dual_mov_b32 v2, s2 :: v_dual_mov_b32 v3, s3
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], off offset:16
+; GFX11-NEXT:    scratch_load_u16 v0, v4, off offset:16
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <8 x i16>, <8 x i16> addrspace(4)* %ptr
   %element = extractelement <8 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_vgpr_v8i16_sgpr_idx(<8 x i16> addrspace(1)* %ptr, i32 inreg %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v8i16_sgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    global_load_dwordx4 v[0:3], v[0:1], off
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX9-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX9-NEXT:    s_and_b32 s1, s2, 1
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 2
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 3
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
-; GFX9-NEXT:    s_lshl_b32 s0, s1, 4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_mov_b32 s7, 0xe00000
+; GFX9-NEXT:    s_add_u32 s4, s4, s3
+; GFX9-NEXT:    s_addc_u32 s5, s5, 0
+; GFX9-NEXT:    s_and_b32 s0, s2, 7
+; GFX9-NEXT:    v_mov_b32_e32 v4, 16
+; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:16
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:20
+; GFX9-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:24
+; GFX9-NEXT:    buffer_store_dword v3, off, s[4:7], 0 offset:28
+; GFX9-NEXT:    v_add_u32_e32 v0, s0, v4
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: extractelement_vgpr_v8i16_sgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
-; GFX8-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX8-NEXT:    s_and_b32 s1, s2, 1
-; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 2
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 3
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
-; GFX8-NEXT:    s_lshl_b32 s0, s1, 4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
+; GFX8-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s6, -1
+; GFX8-NEXT:    s_mov_b32 s7, 0xe80000
+; GFX8-NEXT:    s_add_u32 s4, s4, s3
+; GFX8-NEXT:    s_addc_u32 s5, s5, 0
+; GFX8-NEXT:    s_and_b32 s0, s2, 7
+; GFX8-NEXT:    v_mov_b32_e32 v4, 16
+; GFX8-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:16
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:20
+; GFX8-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:24
+; GFX8-NEXT:    buffer_store_dword v3, off, s[4:7], 0 offset:28
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s0, v4
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: extractelement_vgpr_v8i16_sgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_mov_b32 s6, 0
 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_mov_b64 s[4:5], 0
 ; GFX7-NEXT:    buffer_load_dwordx4 v[0:3], v[0:1], s[4:7], 0 addr64
-; GFX7-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 1
-; GFX7-NEXT:    s_and_b32 s1, s2, 1
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 2
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e64 vcc, s0, 3
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
-; GFX7-NEXT:    s_lshl_b32 s0, s1, 4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
+; GFX7-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s10, -1
+; GFX7-NEXT:    s_mov_b32 s11, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s8, s8, s3
+; GFX7-NEXT:    s_addc_u32 s9, s9, 0
+; GFX7-NEXT:    s_and_b32 s0, s2, 7
+; GFX7-NEXT:    v_mov_b32_e32 v4, 16
+; GFX7-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    buffer_store_dword v0, off, s[8:11], 0 offset:16
+; GFX7-NEXT:    buffer_store_dword v1, off, s[8:11], 0 offset:20
+; GFX7-NEXT:    buffer_store_dword v2, off, s[8:11], 0 offset:24
+; GFX7-NEXT:    buffer_store_dword v3, off, s[8:11], 0 offset:28
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s0, v4
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[8:11], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_vgpr_v8i16_sgpr_idx:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    global_load_dwordx4 v[0:3], v[0:1], off
-; GFX10-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 1
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s4, s4, s3
+; GFX10-NEXT:    s_addc_u32 s5, s5, 0
+; GFX10-NEXT:    s_and_b32 s0, s2, 7
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dword v0, off, s[4:7], 0 offset:16
+; GFX10-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:20
+; GFX10-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:24
+; GFX10-NEXT:    buffer_store_dword v3, off, s[4:7], 0 offset:28
+; GFX10-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX10-NEXT:    v_add_nc_u32_e64 v4, s0, 16
+; GFX10-NEXT:    buffer_load_ushort v0, v4, s[4:7], 0 offen
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 2
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 3
-; GFX10-NEXT:    s_and_b32 s0, s2, 1
-; GFX10-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc_lo
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
 ; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_vgpr_v8i16_sgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    global_load_b128 v[0:3], v[0:1], off
-; GFX11-NEXT:    s_lshr_b32 s0, s2, 1
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 1
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 2
-; GFX11-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e64 vcc_lo, s0, 3
-; GFX11-NEXT:    s_and_b32 s0, s2, 1
-; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    s_lshl_b32 s0, s0, 4
-; GFX11-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc_lo
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, s0, v0
+; GFX11-NEXT:    s_and_b32 s0, s2, 7
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX11-NEXT:    v_mov_b32_e32 v4, s0
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], off offset:16
+; GFX11-NEXT:    scratch_load_u16 v0, v4, off offset:16
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <8 x i16>, <8 x i16> addrspace(1)* %ptr
   %element = extractelement <8 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define i16 @extractelement_vgpr_v8i16_vgpr_idx(<8 x i16> addrspace(1)* %ptr, i32 %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v8i16_vgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dwordx4 v[3:6], v[0:1], off
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 1, v2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
-; GFX9-NEXT:    v_and_b32_e32 v1, 1, v2
-; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 4, v1
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_cndmask_b32_e32 v2, v3, v4, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
-; GFX9-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v2, v6, vcc
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX9-NEXT:    v_and_b32_e32 v0, 7, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    v_lshrrev_b32_e64 v1, 6, s32
+; GFX9-NEXT:    v_add_u32_e32 v0, v1, v0
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    buffer_store_dword v3, off, s[0:3], s32
+; GFX9-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:4
+; GFX9-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:8
+; GFX9-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:12
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: extractelement_vgpr_v8i16_vgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    flat_load_dwordx4 v[3:6], v[0:1]
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, 1, v2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
-; GFX8-NEXT:    v_and_b32_e32 v1, 1, v2
-; GFX8-NEXT:    v_lshlrev_b32_e32 v1, 4, v1
-; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_cndmask_b32_e32 v2, v3, v4, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
-; GFX8-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v2, v6, vcc
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX8-NEXT:    v_and_b32_e32 v0, 7, v2
+; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX8-NEXT:    v_lshrrev_b32_e64 v1, 6, s32
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    buffer_store_dword v3, off, s[0:3], s32
+; GFX8-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:4
+; GFX8-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:8
+; GFX8-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:12
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX7-LABEL: extractelement_vgpr_v8i16_vgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7-NEXT:    s_mov_b32 s6, 0
 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_mov_b64 s[4:5], 0
 ; GFX7-NEXT:    buffer_load_dwordx4 v[3:6], v[0:1], s[4:7], 0 addr64
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, 1, v2
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
-; GFX7-NEXT:    v_and_b32_e32 v1, 1, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 4, v1
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_cndmask_b32_e32 v2, v3, v4, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
-; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v2, v6, vcc
-; GFX7-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX7-NEXT:    v_and_b32_e32 v0, 7, v2
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX7-NEXT:    v_lshr_b32_e64 v1, s32, 6
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    buffer_store_dword v3, off, s[0:3], s32
+; GFX7-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:4
+; GFX7-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:8
+; GFX7-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:12
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: extractelement_vgpr_v8i16_vgpr_idx:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    global_load_dwordx4 v[3:6], v[0:1], off
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, 1, v2
-; GFX10-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v0
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 2, v0
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 3, v0
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v6, vcc_lo
-; GFX10-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 7, v2
+; GFX10-NEXT:    v_lshrrev_b32_e64 v1, 5, s32
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    buffer_store_dword v3, off, s[0:3], s32
+; GFX10-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:4
+; GFX10-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:8
+; GFX10-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:12
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, v1, v0
+; GFX10-NEXT:    buffer_load_ushort v0, v0, s[0:3], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX11-LABEL: extractelement_vgpr_v8i16_vgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    global_load_b128 v[3:6], v[0:1], off
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, 1, v2
-; GFX11-NEXT:    v_and_b32_e32 v2, 1, v2
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_3) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v0
-; GFX11-NEXT:    s_waitcnt vmcnt(0)
-; GFX11-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 2, v0
-; GFX11-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 3, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_cndmask_b32_e32 v0, v1, v6, vcc_lo
-; GFX11-NEXT:    v_lshlrev_b32_e32 v1, 4, v2
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; GFX11-NEXT:    v_and_b32_e32 v0, 7, v2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    scratch_store_b128 off, v[3:6], s32
+; GFX11-NEXT:    scratch_load_u16 v0, v0, s32
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    s_setpc_b64 s[30:31]
   %vector = load <8 x i16>, <8 x i16> addrspace(1)* %ptr
   %element = extractelement <8 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_sgpr_v8i16_vgpr_idx(<8 x i16> addrspace(4)* inreg %ptr, i32 %idx) {
-; GCN-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
-; GCN-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v1
-; GCN-NEXT:    v_and_b32_e32 v0, 1, v0
-; GCN-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v2, s0
-; GCN-NEXT:    v_mov_b32_e32 v3, s1
-; GCN-NEXT:    v_mov_b32_e32 v4, s2
-; GCN-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v1
-; GCN-NEXT:    v_mov_b32_e32 v5, s3
-; GCN-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
-; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v1
-; GCN-NEXT:    v_cndmask_b32_e32 v1, v2, v5, vcc
-; GCN-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
-; GCN-NEXT:    v_readfirstlane_b32 s0, v0
-; GCN-NEXT:    ; return to shader part epilog
+; GFX9-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_mov_b32 s8, s4
+; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_mov_b32 s6, -1
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX9-NEXT:    s_mov_b32 s7, 0xe00000
+; GFX9-NEXT:    s_add_u32 s4, s4, s8
+; GFX9-NEXT:    s_addc_u32 s5, s5, 0
+; GFX9-NEXT:    v_and_b32_e32 v0, 7, v0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:16
+; GFX9-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:20
+; GFX9-NEXT:    v_mov_b32_e32 v1, s2
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:24
+; GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:28
+; GFX9-NEXT:    v_add_u32_e32 v0, 16, v0
+; GFX9-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX9-NEXT:    ; return to shader part epilog
+;
+; GFX8-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
+; GFX8:       ; %bb.0:
+; GFX8-NEXT:    s_mov_b32 s8, s4
+; GFX8-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX8-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    s_mov_b32 s6, -1
+; GFX8-NEXT:    s_mov_b32 s7, 0xe80000
+; GFX8-NEXT:    s_add_u32 s4, s4, s8
+; GFX8-NEXT:    s_addc_u32 s5, s5, 0
+; GFX8-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-NEXT:    v_mov_b32_e32 v2, s1
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:16
+; GFX8-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:20
+; GFX8-NEXT:    v_mov_b32_e32 v1, s2
+; GFX8-NEXT:    v_and_b32_e32 v0, 7, v0
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:24
+; GFX8-NEXT:    v_mov_b32_e32 v1, s3
+; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX8-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:28
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 16, v0
+; GFX8-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX8-NEXT:    s_waitcnt vmcnt(0)
+; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX8-NEXT:    ; return to shader part epilog
+;
+; GFX7-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
+; GFX7:       ; %bb.0:
+; GFX7-NEXT:    s_mov_b32 s8, s4
+; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX7-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b32 s6, -1
+; GFX7-NEXT:    s_mov_b32 s7, 0xe8f000
+; GFX7-NEXT:    s_add_u32 s4, s4, s8
+; GFX7-NEXT:    s_addc_u32 s5, s5, 0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s0
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
+; GFX7-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:16
+; GFX7-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:20
+; GFX7-NEXT:    v_mov_b32_e32 v1, s2
+; GFX7-NEXT:    v_and_b32_e32 v0, 7, v0
+; GFX7-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:24
+; GFX7-NEXT:    v_mov_b32_e32 v1, s3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX7-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:28
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, 16, v0
+; GFX7-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX7-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
 ; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_mov_b32 s8, s4
+; GFX10-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x0
+; GFX10-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[2:3], 0x0
-; GFX10-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v1
-; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_mov_b32 s6, -1
+; GFX10-NEXT:    s_mov_b32 s7, 0x31c16000
+; GFX10-NEXT:    s_add_u32 s4, s4, s8
+; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX10-NEXT:    s_addc_u32 s5, s5, 0
+; GFX10-NEXT:    v_add_nc_u32_e32 v0, 16, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s0
 ; GFX10-NEXT:    v_mov_b32_e32 v2, s1
-; GFX10-NEXT:    v_cndmask_b32_e32 v2, s0, v2, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 2, v1
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
-; GFX10-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 3, v1
-; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s3, vcc_lo
-; GFX10-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
+; GFX10-NEXT:    v_mov_b32_e32 v3, s2
+; GFX10-NEXT:    v_mov_b32_e32 v4, s3
+; GFX10-NEXT:    buffer_store_dword v1, off, s[4:7], 0 offset:16
+; GFX10-NEXT:    buffer_store_dword v2, off, s[4:7], 0 offset:20
+; GFX10-NEXT:    buffer_store_dword v3, off, s[4:7], 0 offset:24
+; GFX10-NEXT:    buffer_store_dword v4, off, s[4:7], 0 offset:28
+; GFX10-NEXT:    buffer_load_ushort v0, v0, s[4:7], 0 offen
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    ; return to shader part epilog
 ;
 ; GFX11-LABEL: extractelement_sgpr_v8i16_vgpr_idx:
 ; GFX11:       ; %bb.0:
 ; GFX11-NEXT:    s_load_b128 s[0:3], s[2:3], 0x0
-; GFX11-NEXT:    v_lshrrev_b32_e32 v1, 1, v0
-; GFX11-NEXT:    v_and_b32_e32 v0, 1, v0
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 1, v1
-; GFX11-NEXT:    v_lshlrev_b32_e32 v0, 4, v0
+; GFX11-NEXT:    v_and_b32_e32 v4, 7, v0
 ; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX11-NEXT:    v_mov_b32_e32 v2, s1
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT:    v_cndmask_b32_e32 v2, s0, v2, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 2, v1
-; GFX11-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
-; GFX11-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 3, v1
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT:    v_cndmask_b32_e64 v1, v2, s3, vcc_lo
-; GFX11-NEXT:    v_lshrrev_b32_e32 v0, v0, v1
-; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT:    v_dual_mov_b32 v0, s0 :: v_dual_mov_b32 v3, s3
+; GFX11-NEXT:    v_dual_mov_b32 v1, s1 :: v_dual_mov_b32 v2, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_3)
+; GFX11-NEXT:    v_lshlrev_b32_e32 v4, 1, v4
+; GFX11-NEXT:    scratch_store_b128 off, v[0:3], off offset:16
+; GFX11-NEXT:    scratch_load_u16 v0, v4, off offset:16
+; GFX11-NEXT:    s_waitcnt vmcnt(0)
 ; GFX11-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX11-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX11-NEXT:    ; return to shader part epilog
   %vector = load <8 x i16>, <8 x i16> addrspace(4)* %ptr
   %element = extractelement <8 x i16> %vector, i32 %idx
   ret i16 %element
 }
 
 define amdgpu_ps i16 @extractelement_sgpr_v8i16_idx0(<8 x i16> addrspace(4)* inreg %ptr) {
 ; GCN-LABEL: extractelement_sgpr_v8i16_idx0: