[AMDGPU] Check for unneeded shift mask in shift PatFrags.

[abinavpp]

Summary

The existing constrained shift PatFrags only dealt with masked shift
from OpenCL front-ends. This change copies the
X86DAGToDAGISel::isUnneededShiftMask() function to AMDGPU and uses it in
the shift PatFrag predicates.

[abinavpp]

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

Do we see anything obvious in this change that's not allowing us to eliminate the and in global-isel for the divergent cases?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

3887

Could use getIConstantVRegVal here to get the APInt value directly?

3887

Opnd1 is a bit confusing because it's MI.getOperand(2). Maybe call it something vague like MaskVal?

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

261

Maybe change this to foreach logwidth = [4, 5, 6] so you can put the definition of csh_mask_#logwidth inside the loop? Or maybe that's impossible because you need to refer to logwidth inside a C++ code fragment?

[foad]

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

I think a cross-regbank copy is getting in the way of matching the constant value 15. Maybe use getIConstantVRegValWithLookThrough to look through the copy?

[arsenm]

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

This is another case where regbankselect or a the post regbank combiner should have materialized the constant in VGPR to begin with

[abinavpp]

Partially addressed review comments.

[abinavpp]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

3887

For some reason, naming the operands as Val and MaskVal is confusing me, how about LHS and RHS?

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

261

Right, I'm not able to refer them in the code fragments.

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

I think a cross-regbank copy is getting in the way of matching the constant value 15. Maybe use getIConstantVRegValWithLookThrough to look through the copy?

getIConstantVRegValWithLookThrough() in the predicate alone won't help here
since we're not able to match the pattern in the first place.

19

This is another case where regbankselect or a the post regbank combiner should have materialized the constant in VGPR to begin with

Doing something like:

--- a/llvm/lib/CodeGen/GlobalISel/RegBankSelect.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/RegBankSelect.cpp
@@ -472,6 +472,10 @@ RegBankSelect::MappingCost RegBankSelect::computeMapping(
     Register Reg = MO.getReg();
     if (!Reg)
       continue;
+
+    if (MO.isUse() && isConstantOrConstantVector(*(MRI->getVRegDef(Reg)), *MRI))
+      continue;
+
     LLVM_DEBUG(dbgs() << "Opd" << OpIdx << '\n');

or forcing SGPRRegBank for constant operands in
AMDGPURegisterBankInfo::getDefaultMappingVOP() fixes this problem buts ends up
violating the constant bus restriction for a lot of AMDGPU tests.

I'm not sure how the original PatFrags (i.e. the ones with the masks as literal
constants without predicates) are working correctly in global-isel for *some*
(vector cases and scalar 64-bit cases are not working) of the divergent cases.

Is there a way to write a constant operand in a tblgen DAG that peeks through
trivial cross regbank copies? Or, is there a better way to fix this?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

3887

Sure.

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

Does D113784 help? Anyway see the discussion in that review about how to pick better banks for constants.

[abinavpp]

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

Yes, D113784 will fix this. We can wait till that gets merged.

[arsenm]

LGTM. Solving constant regbankselect is not really related and shouldn't hold this up

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

The solution I decided on for the constant bus problem is we should just not handle it during globalisel at all. VALU mapped instructions should get all VGPR operands. We should have a new and improved SIFoldOperands which would fold SGPRs into instruction operands. The current scheme was built around the assumption that there were attempts to fold before

[abinavpp]

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

19

Sounds good to me.

[foad]

llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll

2927

@abinavpp Something has gone wrong here. These ANDs were removed by D110231, but now they have come back.

[abinavpp]

llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll

2927

I was aware of this, but I don't think I made it clear in my previous comment:

I'm not sure how the original PatFrags (i.e. the ones with the masks as literal
constants without predicates) are working correctly in global-isel for *some*
(vector cases and scalar 64-bit cases are not working) of the divergent cases.

Sorry about that. The right way to fix this is to fix the cross regbank constant match problem in global-isel. A temporary workaround for this is to keep both the predicated and the literal match versions in PatFrags, i.e., to do:

--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructions.td
@@ -263,21 +263,24 @@ defvar mask = !sub(width, 1);
 defvar csh_mask = !cast<SDPatternOperator>("csh_mask_"#width);
 
 def cshl_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(shl node:$src0, node:$src1), (shl node:$src0, (csh_mask node:$src1))]>;
+  [(shl node:$src0, node:$src1), (shl node:$src0, (csh_mask node:$src1)),
+    (shl node:$src0, (and node:$src1, mask))]>;
 defvar cshl = !cast<SDPatternOperator>("cshl_"#width);
 def cshl_#width#_oneuse : HasOneUseBinOp<cshl>;
 def clshl_rev_#width : PatFrag <(ops node:$src0, node:$src1),
   (cshl $src1, $src0)>;
 
 def csrl_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(srl node:$src0, node:$src1), (srl node:$src0, (csh_mask node:$src1))]>;
+  [(srl node:$src0, node:$src1), (srl node:$src0, (csh_mask node:$src1)),
+    (srl node:$src0, (and node:$src1, mask))]>;
 defvar csrl = !cast<SDPatternOperator>("csrl_"#width);
 def csrl_#width#_oneuse : HasOneUseBinOp<csrl>;
 def clshr_rev_#width : PatFrag <(ops node:$src0, node:$src1),
   (csrl $src1, $src0)>;
 
 def csra_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(sra node:$src0, node:$src1), (sra node:$src0, (csh_mask node:$src1))]>;
+  [(sra node:$src0, node:$src1), (sra node:$src0, (csh_mask node:$src1)),
+    (sra node:$src0, (and node:$src1, mask))]>;

Should I create a revision for the above change (for now) and then revert it after we fix the constant match problem in global-isel?

[foad]

llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll

2927

Oh, my bad, I had forgotten about the known problem with cross regbank copies. No need to do anything now, let's just wait for a proper fix for that problem.

llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h

private:
   bool isInlineImmediate(const APFloat &Imm) const {
     return Subtarget->getInstrInfo()->isInlineConstant(Imm);
   }
 
   bool isVGPRImm(const SDNode *N) const;
   bool isUniformLoad(const SDNode *N) const;
   bool isUniformBr(const SDNode *N) const;
 
+  // Returns true if ISD::AND SDNode `N`'s masking of the shift amount operand's
+  // `ShAmtBits` bits is unneeded.
+  bool isUnneededShiftMask(const SDNode *N, unsigned ShAmtBits) const;
+
   bool isBaseWithConstantOffset64(SDValue Addr, SDValue &LHS,
                                   SDValue &RHS) const;
 
   MachineSDNode *buildSMovImm64(SDLoc &DL, uint64_t Val, EVT VT) const;
 
   SDNode *glueCopyToOp(SDNode *N, SDValue NewChain, SDValue Glue) const;
   SDNode *glueCopyToM0(SDNode *N, SDValue Val) const;
   SDNode *glueCopyToM0LDSInit(SDNode *N) const;

llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

 
 bool AMDGPUDAGToDAGISel::isUniformBr(const SDNode *N) const {
   const BasicBlock *BB = FuncInfo->MBB->getBasicBlock();
   const Instruction *Term = BB->getTerminator();
   return Term->getMetadata("amdgpu.uniform") ||
          Term->getMetadata("structurizecfg.uniform");
 }
 
+bool AMDGPUDAGToDAGISel::isUnneededShiftMask(const SDNode *N,
+                                             unsigned ShAmtBits) const {
+  assert(N->getOpcode() == ISD::AND);
+
+  const APInt &RHS = cast<ConstantSDNode>(N->getOperand(1))->getAPIntValue();
+  if (RHS.countTrailingOnes() >= ShAmtBits)
+    return true;
+
+  const APInt &LHSKnownZeros = CurDAG->computeKnownBits(N->getOperand(0)).Zero;
+  return (LHSKnownZeros | RHS).countTrailingOnes() >= ShAmtBits;
+}
+
 static bool getBaseWithOffsetUsingSplitOR(SelectionDAG &DAG, SDValue Addr,
                                           SDValue &N0, SDValue &N1) {
   if (Addr.getValueType() == MVT::i64 && Addr.getOpcode() == ISD::BITCAST &&
       Addr.getOperand(0).getOpcode() == ISD::BUILD_VECTOR) {
     // As we split 64-bit `or` earlier, it's complicated pattern to match, i.e.
     // (i64 (bitcast (v2i32 (build_vector
     //                        (or (extract_vector_elt V, 0), OFFSET),
     //                        (extract_vector_elt V, 1)))))

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h

private:
   void renderFrameIndex(MachineInstrBuilder &MIB, const MachineInstr &MI,
                         int OpIdx) const;
 
   bool isInlineImmediate16(int64_t Imm) const;
   bool isInlineImmediate32(int64_t Imm) const;
   bool isInlineImmediate64(int64_t Imm) const;
   bool isInlineImmediate(const APFloat &Imm) const;
 
+  // Returns true if TargetOpcode::G_AND MachineInstr `MI`'s masking of the
+  // shift amount operand's `ShAmtBits` bits is unneeded.
+  bool isUnneededShiftMask(const MachineInstr &MI, unsigned ShAmtBits) const;
+
   const SIInstrInfo &TII;
   const SIRegisterInfo &TRI;
   const AMDGPURegisterBankInfo &RBI;
   const AMDGPUTargetMachine &TM;
   const GCNSubtarget &STI;
   bool EnableLateStructurizeCFG;
 #define GET_GLOBALISEL_PREDICATES_DECL
 #define AMDGPUSubtarget GCNSubtarget

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

bool AMDGPUInstructionSelector::isDSOffset2Legal(Register Base, int64_t Offset0,
   if (STI.hasUsableDSOffset() || STI.unsafeDSOffsetFoldingEnabled())
     return true;
 
   // On Southern Islands instruction with a negative base value and an offset
   // don't seem to work.
   return KnownBits->signBitIsZero(Base);
 }
 
+bool AMDGPUInstructionSelector::isUnneededShiftMask(const MachineInstr &MI,
+                                                    unsigned ShAmtBits) const {
+  assert(MI.getOpcode() == TargetOpcode::G_AND);
+
+  Optional<APInt> RHS = getIConstantVRegVal(MI.getOperand(2).getReg(), *MRI);
+  if (!RHS)
+    return false;
+
+  if (RHS->countTrailingOnes() >= ShAmtBits)
+    return true;
+
+  const APInt &LHSKnownZeros =
+      KnownBits->getKnownZeroes(MI.getOperand(1).getReg());
+  return (LHSKnownZeros | *RHS).countTrailingOnes() >= ShAmtBits;
+}
+
 InstructionSelector::ComplexRendererFns
 AMDGPUInstructionSelector::selectMUBUFScratchOffset(
     MachineOperand &Root) const {
   MachineInstr *MI = Root.getParent();
   MachineBasicBlock *MBB = MI->getParent();
 
   int64_t Offset = 0;
   if (!mi_match(Root.getReg(), *MRI, m_ICst(Offset)) ||

llvm/lib/Target/AMDGPU/AMDGPUInstructions.td

 def AMDGPUmul_u24_oneuse : HasOneUseBinOp<AMDGPUmul_u24>;
 def AMDGPUmul_i24_oneuse : HasOneUseBinOp<AMDGPUmul_i24>;
 
 //===----------------------------------------------------------------------===//
 // PatFrags for shifts
 //===----------------------------------------------------------------------===//
 
 // Constrained shift PatFrags.
+
+def csh_mask_16 : PatFrag<(ops node:$src0), (and node:$src0, imm),
+  [{ return isUnneededShiftMask(N, 4); }]> {
+    let GISelPredicateCode = [{ return isUnneededShiftMask(MI, 4); }];
+  }
+
+def csh_mask_32 : PatFrag<(ops node:$src0), (and node:$src0, imm),
+  [{ return isUnneededShiftMask(N, 5); }]> {
+    let GISelPredicateCode = [{ return isUnneededShiftMask(MI, 5); }];
+  }
+
+def csh_mask_64 : PatFrag<(ops node:$src0), (and node:$src0, imm),
+  [{ return isUnneededShiftMask(N, 6); }]> {
+    let GISelPredicateCode = [{ return isUnneededShiftMask(MI, 6); }];
+  }
+
 foreach width = [16, 32, 64] in {
 defvar mask = !sub(width, 1);
+defvar csh_mask = !cast<SDPatternOperator>("csh_mask_"#width);
 
 def cshl_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(shl node:$src0, node:$src1), (shl node:$src0, (and node:$src1, mask))]>;
+  [(shl node:$src0, node:$src1), (shl node:$src0, (csh_mask node:$src1))]>;
 defvar cshl = !cast<SDPatternOperator>("cshl_"#width);
 def cshl_#width#_oneuse : HasOneUseBinOp<cshl>;
 def clshl_rev_#width : PatFrag <(ops node:$src0, node:$src1),
   (cshl $src1, $src0)>;
 
 def csrl_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(srl node:$src0, node:$src1), (srl node:$src0, (and node:$src1, mask))]>;
+  [(srl node:$src0, node:$src1), (srl node:$src0, (csh_mask node:$src1))]>;
 defvar csrl = !cast<SDPatternOperator>("csrl_"#width);
 def csrl_#width#_oneuse : HasOneUseBinOp<csrl>;
 def clshr_rev_#width : PatFrag <(ops node:$src0, node:$src1),
   (csrl $src1, $src0)>;
 
 def csra_#width : PatFrags<(ops node:$src0, node:$src1),
-  [(sra node:$src0, node:$src1), (sra node:$src0, (and node:$src1, mask))]>;
+  [(sra node:$src0, node:$src1), (sra node:$src0, (csh_mask node:$src1))]>;
 defvar csra = !cast<SDPatternOperator>("csra_"#width);
 def csra_#width#_oneuse : HasOneUseBinOp<csra>;
 def cashr_rev_#width : PatFrag <(ops node:$src0, node:$src1),
   (csra $src1, $src0)>;
 } // end foreach width
 
 def srl_16 : PatFrag<
   (ops node:$src0), (srl_oneuse node:$src0, (i32 16))

llvm/test/CodeGen/AMDGPU/GlobalISel/ashr.ll

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr i8 %value, 7
   ret i8 %result
 }
 
 define amdgpu_ps i8 @s_ashr_i8(i8 inreg %value, i8 inreg %amount) {
 ; GFX6-LABEL: s_ashr_i8:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_and_b32 s1, s1, 0xff
 ; GFX6-NEXT:    s_sext_i32_i8 s0, s0
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_ashr_i8:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_sext_i32_i8 s0, s0
 ; GFX8-NEXT:    s_sext_i32_i8 s1, s1

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr i24 %value, 7
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_ashr_i24(i24 inreg %value, i24 inreg %amount) {
 ; GCN-LABEL: s_ashr_i24:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_and_b32 s1, s1, 0xffffff
 ; GCN-NEXT:    s_bfe_i32 s0, s0, 0x180000
 ; GCN-NEXT:    s_ashr_i32 s0, s0, s1
 ; GCN-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_ashr_i24:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_and_b32 s1, s1, 0xffffff
 ; GFX10-NEXT:    s_bfe_i32 s0, s0, 0x180000
 ; GFX10-NEXT:    s_ashr_i32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = ashr i24 %value, %amount
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_ashr_i24_7(i24 inreg %value) {

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr i16 %value, 31
   ret i16 %result
 }
 
 define amdgpu_ps i16 @s_ashr_i16(i16 inreg %value, i16 inreg %amount) {
 ; GFX6-LABEL: s_ashr_i16:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_and_b32 s1, s1, 0xffff
 ; GFX6-NEXT:    s_sext_i32_i16 s0, s0
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_ashr_i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_sext_i32_i16 s0, s0
 ; GFX8-NEXT:    s_sext_i32_i16 s1, s1

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr <2 x i16> %value, <i16 15, i16 15>
   ret <2 x i16> %result
 }
 
 define amdgpu_ps i32 @s_ashr_v2i16(<2 x i16> inreg %value, <2 x i16> inreg %amount) {
 ; GFX6-LABEL: s_ashr_v2i16:
 ; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_sext_i32_i16 s1, s1
 ; GFX6-NEXT:    s_mov_b32 s4, 0xffff
-; GFX6-NEXT:    s_and_b32 s2, s2, s4
 ; GFX6-NEXT:    s_sext_i32_i16 s0, s0
+; GFX6-NEXT:    s_ashr_i32 s1, s1, s3
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, s2
-; GFX6-NEXT:    s_and_b32 s2, s3, s4
-; GFX6-NEXT:    s_sext_i32_i16 s1, s1
-; GFX6-NEXT:    s_ashr_i32 s1, s1, s2
 ; GFX6-NEXT:    s_and_b32 s1, s1, s4
 ; GFX6-NEXT:    s_and_b32 s0, s0, s4
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_ashr_v2i16:
 ; GFX8:       ; %bb.0:
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr <4 x i16> %value, %amount
   %cast = bitcast <4 x i16> %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x i32> @s_ashr_v4i16(<4 x i16> inreg %value, <4 x i16> inreg %amount) {
 ; GFX6-LABEL: s_ashr_v4i16:
 ; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_sext_i32_i16 s1, s1
 ; GFX6-NEXT:    s_mov_b32 s8, 0xffff
-; GFX6-NEXT:    s_and_b32 s4, s4, s8
 ; GFX6-NEXT:    s_sext_i32_i16 s0, s0
+; GFX6-NEXT:    s_ashr_i32 s1, s1, s5
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, s4
-; GFX6-NEXT:    s_and_b32 s4, s5, s8
-; GFX6-NEXT:    s_sext_i32_i16 s1, s1
-; GFX6-NEXT:    s_ashr_i32 s1, s1, s4
-; GFX6-NEXT:    s_and_b32 s4, s6, s8
 ; GFX6-NEXT:    s_sext_i32_i16 s2, s2
-; GFX6-NEXT:    s_ashr_i32 s2, s2, s4
-; GFX6-NEXT:    s_and_b32 s4, s7, s8
 ; GFX6-NEXT:    s_sext_i32_i16 s3, s3
 ; GFX6-NEXT:    s_and_b32 s1, s1, s8
-; GFX6-NEXT:    s_ashr_i32 s3, s3, s4
+; GFX6-NEXT:    s_ashr_i32 s2, s2, s6
+; GFX6-NEXT:    s_ashr_i32 s3, s3, s7
 ; GFX6-NEXT:    s_and_b32 s0, s0, s8
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_and_b32 s1, s2, s8
 ; GFX6-NEXT:    s_and_b32 s2, s3, s8
 ; GFX6-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX6-NEXT:    s_or_b32 s1, s1, s2
 ; GFX6-NEXT:    ; return to shader part epilog
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = ashr <8 x i16> %value, %amount
   %cast = bitcast <8 x i16> %result to <4 x float>
   ret <4 x float> %cast
 }
 
 define amdgpu_ps <4 x i32> @s_ashr_v8i16(<8 x i16> inreg %value, <8 x i16> inreg %amount) {
 ; GFX6-LABEL: s_ashr_v8i16:
 ; GFX6:       ; %bb.0:
+; GFX6-NEXT:    s_sext_i32_i16 s1, s1
 ; GFX6-NEXT:    s_mov_b32 s16, 0xffff
-; GFX6-NEXT:    s_and_b32 s8, s8, s16
 ; GFX6-NEXT:    s_sext_i32_i16 s0, s0
+; GFX6-NEXT:    s_ashr_i32 s1, s1, s9
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, s8
-; GFX6-NEXT:    s_and_b32 s8, s9, s16
-; GFX6-NEXT:    s_sext_i32_i16 s1, s1
-; GFX6-NEXT:    s_ashr_i32 s1, s1, s8
-; GFX6-NEXT:    s_and_b32 s8, s10, s16
 ; GFX6-NEXT:    s_sext_i32_i16 s2, s2
-; GFX6-NEXT:    s_ashr_i32 s2, s2, s8
-; GFX6-NEXT:    s_and_b32 s8, s11, s16
 ; GFX6-NEXT:    s_sext_i32_i16 s3, s3
-; GFX6-NEXT:    s_ashr_i32 s3, s3, s8
-; GFX6-NEXT:    s_and_b32 s8, s12, s16
-; GFX6-NEXT:    s_sext_i32_i16 s4, s4
-; GFX6-NEXT:    s_ashr_i32 s4, s4, s8
-; GFX6-NEXT:    s_and_b32 s8, s13, s16
-; GFX6-NEXT:    s_sext_i32_i16 s5, s5
 ; GFX6-NEXT:    s_and_b32 s1, s1, s16
-; GFX6-NEXT:    s_ashr_i32 s5, s5, s8
-; GFX6-NEXT:    s_and_b32 s8, s14, s16
-; GFX6-NEXT:    s_sext_i32_i16 s6, s6
+; GFX6-NEXT:    s_ashr_i32 s2, s2, s10
+; GFX6-NEXT:    s_ashr_i32 s3, s3, s11
+; GFX6-NEXT:    s_sext_i32_i16 s5, s5
 ; GFX6-NEXT:    s_and_b32 s0, s0, s16
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
-; GFX6-NEXT:    s_ashr_i32 s6, s6, s8
-; GFX6-NEXT:    s_and_b32 s8, s15, s16
+; GFX6-NEXT:    s_sext_i32_i16 s4, s4
+; GFX6-NEXT:    s_ashr_i32 s5, s5, s13
 ; GFX6-NEXT:    s_sext_i32_i16 s7, s7
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_and_b32 s1, s2, s16
 ; GFX6-NEXT:    s_and_b32 s2, s3, s16
-; GFX6-NEXT:    s_ashr_i32 s7, s7, s8
+; GFX6-NEXT:    s_ashr_i32 s4, s4, s12
+; GFX6-NEXT:    s_sext_i32_i16 s6, s6
+; GFX6-NEXT:    s_ashr_i32 s7, s7, s15
 ; GFX6-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX6-NEXT:    s_and_b32 s3, s5, s16
+; GFX6-NEXT:    s_ashr_i32 s6, s6, s14
 ; GFX6-NEXT:    s_or_b32 s1, s1, s2
 ; GFX6-NEXT:    s_and_b32 s2, s4, s16
 ; GFX6-NEXT:    s_lshl_b32 s3, s3, 16
 ; GFX6-NEXT:    s_and_b32 s4, s7, s16
 ; GFX6-NEXT:    s_or_b32 s2, s2, s3
 ; GFX6-NEXT:    s_and_b32 s3, s6, s16
 ; GFX6-NEXT:    s_lshl_b32 s4, s4, 16
 ; GFX6-NEXT:    s_or_b32 s3, s3, s4

llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll

 ; GFX6-NEXT:    v_lshlrev_b32_e32 v0, v3, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, v2, v1
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fshl_i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    v_xor_b32_e32 v3, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v3, 15, v2
+; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v1, 1, v1
-; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v2, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
+; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v3, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v2, v1
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_fshl_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_xor_b32_e32 v3, -1, v2
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v2
+; GFX9-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    v_lshrrev_b16_e32 v1, 1, v1
-; GFX9-NEXT:    v_lshlrev_b16_e32 v0, v2, v0
-; GFX9-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
+; GFX9-NEXT:    v_lshlrev_b16_e32 v0, v3, v0
+; GFX9-NEXT:    v_lshrrev_b16_e32 v1, v2, v1
 ; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: v_fshl_i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    v_xor_b32_e32 v3, -1, v2
+; GFX10-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX10-NEXT:    v_lshrrev_b16 v1, 1, v1
+; GFX10-NEXT:    v_and_b32_e32 v3, 15, v3
 ; GFX10-NEXT:    v_lshlrev_b16 v0, v2, v0
 ; GFX10-NEXT:    v_lshrrev_b16 v1, v3, v1
 ; GFX10-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call i16 @llvm.fshl.i16(i16 %lhs, i16 %rhs, i16 %amt)
   ret i16 %result
 }
 

 ; GFX6-NEXT:    s_bfe_u32 s0, s1, 0xf0001
 ; GFX6-NEXT:    v_bfe_u32 v0, v0, 0, 16
 ; GFX6-NEXT:    v_lshr_b32_e32 v0, s0, v0
 ; GFX6-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshl_i16_ssv:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    v_xor_b32_e32 v1, -1, v0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_and_b32_e32 v1, 15, v0
+; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v1, v1, s0
 ; GFX8-NEXT:    s_bfe_u32 s0, s1, 0x100000
 ; GFX8-NEXT:    s_bfe_u32 s1, 1, 0x100000
+; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX8-NEXT:    v_lshrrev_b16_e64 v1, v1, s0
-; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX8-NEXT:    v_lshrrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshl_i16_ssv:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    v_xor_b32_e32 v1, -1, v0
-; GFX9-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX9-NEXT:    v_and_b32_e32 v1, 15, v0
+; GFX9-NEXT:    v_xor_b32_e32 v0, -1, v0
+; GFX9-NEXT:    v_lshlrev_b16_e64 v1, v1, s0
 ; GFX9-NEXT:    s_bfe_u32 s0, s1, 0x100000
 ; GFX9-NEXT:    s_bfe_u32 s1, 1, 0x100000
+; GFX9-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX9-NEXT:    v_lshrrev_b16_e64 v1, v1, s0
-; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX9-NEXT:    v_lshrrev_b16_e64 v0, v0, s0
+; GFX9-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshl_i16_ssv:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    v_xor_b32_e32 v1, -1, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX10-NEXT:    s_bfe_u32 s1, s1, 0x100000
 ; GFX10-NEXT:    s_bfe_u32 s2, 1, 0x100000
-; GFX10-NEXT:    v_lshlrev_b16 v0, v0, s0
 ; GFX10-NEXT:    s_lshr_b32 s1, s1, s2
+; GFX10-NEXT:    v_and_b32_e32 v1, 15, v1
+; GFX10-NEXT:    v_lshlrev_b16 v0, v0, s0
 ; GFX10-NEXT:    v_lshrrev_b16 v1, v1, s1
 ; GFX10-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call i16 @llvm.fshl.i16(i16 %lhs, i16 %rhs, i16 %amt)
   %cast.result = bitcast i16 %result to half
   ret half %cast.result
 }
 

 ; GFX9-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX9-NEXT:    s_lshr_b32 s5, s4, 16
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s4
 ; GFX9-NEXT:    s_lshl_b32 s3, s3, s5
 ; GFX9-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s3
 ; GFX9-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX9-NEXT:    s_and_b32 s1, s1, s4
-; GFX9-NEXT:    s_lshr_b32 s1, s1, 1
+; GFX9-NEXT:    s_lshr_b32 s1, s1, 0x10001
 ; GFX9-NEXT:    s_lshr_b32 s3, s3, 1
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s3
 ; GFX9-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX9-NEXT:    s_and_b32 s1, s1, s4
-; GFX9-NEXT:    s_lshr_b32 s5, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s4
+; GFX9-NEXT:    s_lshr_b32 s4, s2, 16
 ; GFX9-NEXT:    s_lshr_b32 s1, s1, s2
-; GFX9-NEXT:    s_lshr_b32 s2, s3, s5
+; GFX9-NEXT:    s_lshr_b32 s2, s3, s4
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX9-NEXT:    s_or_b32 s0, s0, s1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshl_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_mov_b32 s5, 0xffff
 ; GFX10-NEXT:    s_mov_b32 s3, 0xf000f
 ; GFX10-NEXT:    s_and_b32 s7, s1, s5
 ; GFX10-NEXT:    s_lshr_b32 s1, s1, 16
-; GFX10-NEXT:    s_lshr_b32 s7, s7, 1
+; GFX10-NEXT:    s_lshr_b32 s7, s7, 0x10001
 ; GFX10-NEXT:    s_lshr_b32 s1, s1, 1
 ; GFX10-NEXT:    s_and_b32 s4, s2, s3
 ; GFX10-NEXT:    s_andn2_b32 s2, s3, s2
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s7, s1
 ; GFX10-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX10-NEXT:    s_lshr_b32 s6, s4, 16
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s4
 ; GFX10-NEXT:    s_lshr_b32 s4, s1, 16
 ; GFX10-NEXT:    s_and_b32 s1, s1, s5
-; GFX10-NEXT:    s_and_b32 s5, s2, s5
-; GFX10-NEXT:    s_lshr_b32 s2, s2, 16
+; GFX10-NEXT:    s_lshr_b32 s5, s2, 16
 ; GFX10-NEXT:    s_lshl_b32 s3, s3, s6
-; GFX10-NEXT:    s_lshr_b32 s1, s1, s5
-; GFX10-NEXT:    s_lshr_b32 s2, s4, s2
+; GFX10-NEXT:    s_lshr_b32 s1, s1, s2
+; GFX10-NEXT:    s_lshr_b32 s2, s4, s5
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s3
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX10-NEXT:    s_or_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i16> @llvm.fshl.v2i16(<2 x i16> %lhs, <2 x i16> %rhs, <2 x i16> %amt)
   %cast = bitcast <2 x i16> %result to i32
   ret i32 %cast
 }

 ; GFX6-NEXT:    v_bfe_u32 v3, v4, 0, 16
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v2, v3, v2
 ; GFX6-NEXT:    v_or_b32_e32 v1, v1, v2
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fshl_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v2
-; GFX8-NEXT:    v_lshrrev_b16_e32 v5, 1, v1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; GFX8-NEXT:    v_lshlrev_b16_e32 v2, v2, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v4, v4, v5
-; GFX8-NEXT:    v_or_b32_e32 v2, v2, v4
-; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v3
-; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_mov_b32_e32 v3, 1
-; GFX8-NEXT:    v_lshrrev_b16_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v4, v1
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v2
+; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX8-NEXT:    v_lshrrev_b16_e32 v5, 1, v1
+; GFX8-NEXT:    v_lshlrev_b16_e32 v4, v4, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v2, v5
+; GFX8-NEXT:    v_or_b32_e32 v2, v4, v2
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v3
+; GFX8-NEXT:    v_xor_b32_e32 v3, -1, v3
+; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v4, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_mov_b32_e32 v4, 1
+; GFX8-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX8-NEXT:    v_lshrrev_b16_sdwa v1, v4, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v1, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_fshl_v2i16:
 ; GFX9:       ; %bb.0:

 ; GFX6-NEXT:    v_bfe_u32 v1, v1, 0, 16
 ; GFX6-NEXT:    v_bfe_u32 v0, v0, 0, 16
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshl_v2i16_ssv:
 ; GFX8:       ; %bb.0:
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v0
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX8-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v2, v2, s0
 ; GFX8-NEXT:    s_bfe_u32 s0, s1, 0x100000
 ; GFX8-NEXT:    s_bfe_u32 s1, 1, 0x100000
+; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX8-NEXT:    v_lshrrev_b16_e64 v2, v2, s0
-; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v1
+; GFX8-NEXT:    v_lshrrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_or_b32_e32 v0, v2, v0
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v1
+; GFX8-NEXT:    v_xor_b32_e32 v1, -1, v1
+; GFX8-NEXT:    v_and_b32_e32 v1, 15, v1
 ; GFX8-NEXT:    s_lshr_b32 s0, s3, s1
-; GFX8-NEXT:    v_lshlrev_b16_e64 v1, v1, s2
-; GFX8-NEXT:    v_lshrrev_b16_e64 v2, v2, s0
-; GFX8-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX8-NEXT:    v_lshlrev_b16_e64 v2, v2, s2
+; GFX8-NEXT:    v_lshrrev_b16_e64 v1, v1, s0
+; GFX8-NEXT:    v_or_b32_e32 v1, v2, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v2, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshl_v2i16_ssv:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s2, 0xf000f
 ; GFX9-NEXT:    v_and_b32_e32 v1, s2, v0
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v1, v1, s0
 ; GFX9-NEXT:    s_lshr_b32 s0, s1, 16
 ; GFX9-NEXT:    s_and_b32 s1, s1, 0xffff
 ; GFX9-NEXT:    v_xor_b32_e32 v0, -1, v0
-; GFX9-NEXT:    s_lshr_b32 s1, s1, 1
+; GFX9-NEXT:    s_lshr_b32 s1, s1, 0x10001
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, 1
 ; GFX9-NEXT:    v_and_b32_e32 v0, s2, v0
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s1, s0
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v0, v0, s0
 ; GFX9-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshl_v2i16_ssv:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    v_xor_b32_e32 v1, -1, v0
 ; GFX10-NEXT:    s_mov_b32 s2, 0xf000f
 ; GFX10-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX10-NEXT:    s_and_b32 s1, s1, 0xffff
 ; GFX10-NEXT:    v_and_b32_e32 v0, s2, v0
 ; GFX10-NEXT:    v_and_b32_e32 v1, s2, v1
-; GFX10-NEXT:    s_lshr_b32 s1, s1, 1
+; GFX10-NEXT:    s_lshr_b32 s1, s1, 0x10001
 ; GFX10-NEXT:    s_lshr_b32 s2, s3, 1
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX10-NEXT:    v_pk_lshlrev_b16 v0, v0, s0
 ; GFX10-NEXT:    v_pk_lshrrev_b16 v1, v1, s1
 ; GFX10-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i16> @llvm.fshl.v2i16(<2 x i16> %lhs, <2 x i16> %rhs, <2 x i16> %amt)
   %cast = bitcast <2 x i16> %result to float

 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s2, 0xf000f
 ; GFX9-NEXT:    s_and_b32 s3, s1, s2
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v0, s3, v0
 ; GFX9-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX9-NEXT:    s_andn2_b32 s1, s2, s1
 ; GFX9-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX9-NEXT:    s_and_b32 s0, s0, s3
-; GFX9-NEXT:    s_lshr_b32 s0, s0, 1
+; GFX9-NEXT:    s_lshr_b32 s0, s0, 0x10001
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 1
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX9-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX9-NEXT:    s_and_b32 s0, s0, s3
-; GFX9-NEXT:    s_lshr_b32 s4, s1, 16
-; GFX9-NEXT:    s_and_b32 s1, s1, s3
+; GFX9-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX9-NEXT:    s_lshr_b32 s1, s2, s4
+; GFX9-NEXT:    s_lshr_b32 s1, s2, s3
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX9-NEXT:    v_or_b32_e32 v0, s0, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshl_v2i16_vss:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX10-NEXT:    s_mov_b32 s2, 0xf000f
 ; GFX10-NEXT:    s_and_b32 s5, s0, s3
 ; GFX10-NEXT:    s_lshr_b32 s0, s0, 16
-; GFX10-NEXT:    s_lshr_b32 s5, s5, 1
+; GFX10-NEXT:    s_lshr_b32 s5, s5, 0x10001
 ; GFX10-NEXT:    s_lshr_b32 s0, s0, 1
 ; GFX10-NEXT:    s_and_b32 s4, s1, s2
 ; GFX10-NEXT:    s_andn2_b32 s1, s2, s1
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s5, s0
 ; GFX10-NEXT:    v_pk_lshlrev_b16 v0, s4, v0
 ; GFX10-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX10-NEXT:    s_and_b32 s0, s0, s3
-; GFX10-NEXT:    s_and_b32 s3, s1, s3
-; GFX10-NEXT:    s_lshr_b32 s1, s1, 16
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s3
-; GFX10-NEXT:    s_lshr_b32 s1, s2, s1
+; GFX10-NEXT:    s_lshr_b32 s3, s1, 16
+; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX10-NEXT:    s_lshr_b32 s1, s2, s3
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX10-NEXT:    v_or_b32_e32 v0, s0, v0
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i16> @llvm.fshl.v2i16(<2 x i16> %lhs, <2 x i16> %rhs, <2 x i16> %amt)
   %cast = bitcast <2 x i16> %result to float
   ret float %cast
 }
 

 ; GFX8-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX8-NEXT:    s_or_b32 s1, s1, s2
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_fshl_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s6, 0xf000f
 ; GFX9-NEXT:    s_and_b32 s7, s4, s6
-; GFX9-NEXT:    s_lshr_b32 s8, s0, 16
-; GFX9-NEXT:    s_lshr_b32 s9, s7, 16
+; GFX9-NEXT:    s_lshr_b32 s9, s0, 16
+; GFX9-NEXT:    s_lshr_b32 s10, s7, 16
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s7
-; GFX9-NEXT:    s_lshl_b32 s7, s8, s9
-; GFX9-NEXT:    s_mov_b32 s8, 0xffff
+; GFX9-NEXT:    s_lshl_b32 s7, s9, s10
+; GFX9-NEXT:    s_mov_b32 s9, 0xffff
+; GFX9-NEXT:    s_mov_b32 s8, 0x10001
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s7
 ; GFX9-NEXT:    s_lshr_b32 s7, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s8
-; GFX9-NEXT:    s_lshr_b32 s2, s2, 1
+; GFX9-NEXT:    s_and_b32 s2, s2, s9
+; GFX9-NEXT:    s_lshr_b32 s2, s2, s8
 ; GFX9-NEXT:    s_lshr_b32 s7, s7, 1
 ; GFX9-NEXT:    s_andn2_b32 s4, s6, s4
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s7
 ; GFX9-NEXT:    s_lshr_b32 s7, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s8
-; GFX9-NEXT:    s_lshr_b32 s9, s4, 16
-; GFX9-NEXT:    s_and_b32 s4, s4, s8
+; GFX9-NEXT:    s_and_b32 s2, s2, s9
+; GFX9-NEXT:    s_lshr_b32 s10, s4, 16
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, s4
-; GFX9-NEXT:    s_lshr_b32 s4, s7, s9
+; GFX9-NEXT:    s_lshr_b32 s4, s7, s10
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s4
 ; GFX9-NEXT:    s_or_b32 s0, s0, s2
 ; GFX9-NEXT:    s_and_b32 s2, s5, s6
 ; GFX9-NEXT:    s_andn2_b32 s4, s6, s5
 ; GFX9-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX9-NEXT:    s_lshr_b32 s6, s2, 16
 ; GFX9-NEXT:    s_lshl_b32 s1, s1, s2
 ; GFX9-NEXT:    s_lshl_b32 s2, s5, s6
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX9-NEXT:    s_lshr_b32 s2, s3, 16
-; GFX9-NEXT:    s_and_b32 s3, s3, s8
-; GFX9-NEXT:    s_lshr_b32 s3, s3, 1
+; GFX9-NEXT:    s_and_b32 s3, s3, s9
+; GFX9-NEXT:    s_lshr_b32 s3, s3, s8
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 1
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s3, s2
 ; GFX9-NEXT:    s_lshr_b32 s3, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s8
+; GFX9-NEXT:    s_and_b32 s2, s2, s9
 ; GFX9-NEXT:    s_lshr_b32 s5, s4, 16
-; GFX9-NEXT:    s_and_b32 s4, s4, s8
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, s4
 ; GFX9-NEXT:    s_lshr_b32 s3, s3, s5
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s3
 ; GFX9-NEXT:    s_or_b32 s1, s1, s2
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshl_v4i16:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_mov_b32 s9, 0xffff
+; GFX10-NEXT:    s_mov_b32 s10, 0xffff
 ; GFX10-NEXT:    s_mov_b32 s6, 0xf000f
-; GFX10-NEXT:    s_and_b32 s11, s2, s9
+; GFX10-NEXT:    s_mov_b32 s8, 0x10001
+; GFX10-NEXT:    s_and_b32 s12, s2, s10
 ; GFX10-NEXT:    s_lshr_b32 s2, s2, 16
 ; GFX10-NEXT:    s_and_b32 s7, s4, s6
-; GFX10-NEXT:    s_lshr_b32 s11, s11, 1
+; GFX10-NEXT:    s_lshr_b32 s12, s12, s8
 ; GFX10-NEXT:    s_lshr_b32 s2, s2, 1
 ; GFX10-NEXT:    s_andn2_b32 s4, s6, s4
-; GFX10-NEXT:    s_lshr_b32 s8, s0, 16
-; GFX10-NEXT:    s_lshr_b32 s10, s7, 16
-; GFX10-NEXT:    s_pack_ll_b32_b16 s2, s11, s2
+; GFX10-NEXT:    s_lshr_b32 s9, s0, 16
+; GFX10-NEXT:    s_lshr_b32 s11, s7, 16
+; GFX10-NEXT:    s_pack_ll_b32_b16 s2, s12, s2
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s7
-; GFX10-NEXT:    s_lshl_b32 s7, s8, s10
-; GFX10-NEXT:    s_lshr_b32 s8, s2, 16
-; GFX10-NEXT:    s_and_b32 s2, s2, s9
-; GFX10-NEXT:    s_and_b32 s10, s4, s9
-; GFX10-NEXT:    s_lshr_b32 s4, s4, 16
-; GFX10-NEXT:    s_lshr_b32 s2, s2, s10
-; GFX10-NEXT:    s_lshr_b32 s4, s8, s4
-; GFX10-NEXT:    s_and_b32 s8, s3, s9
+; GFX10-NEXT:    s_lshl_b32 s7, s9, s11
+; GFX10-NEXT:    s_lshr_b32 s9, s2, 16
+; GFX10-NEXT:    s_and_b32 s2, s2, s10
+; GFX10-NEXT:    s_lshr_b32 s11, s4, 16
+; GFX10-NEXT:    s_lshr_b32 s2, s2, s4
+; GFX10-NEXT:    s_lshr_b32 s4, s9, s11
+; GFX10-NEXT:    s_and_b32 s9, s3, s10
 ; GFX10-NEXT:    s_lshr_b32 s3, s3, 16
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s2, s2, s4
 ; GFX10-NEXT:    s_and_b32 s4, s5, s6
-; GFX10-NEXT:    s_lshr_b32 s8, s8, 1
+; GFX10-NEXT:    s_lshr_b32 s8, s9, s8
 ; GFX10-NEXT:    s_lshr_b32 s3, s3, 1
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s7
 ; GFX10-NEXT:    s_andn2_b32 s5, s6, s5
 ; GFX10-NEXT:    s_lshr_b32 s6, s1, 16
 ; GFX10-NEXT:    s_lshr_b32 s7, s4, 16
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s3, s8, s3
 ; GFX10-NEXT:    s_lshl_b32 s1, s1, s4
 ; GFX10-NEXT:    s_lshl_b32 s4, s6, s7
 ; GFX10-NEXT:    s_lshr_b32 s6, s3, 16
-; GFX10-NEXT:    s_and_b32 s3, s3, s9
-; GFX10-NEXT:    s_and_b32 s7, s5, s9
-; GFX10-NEXT:    s_lshr_b32 s5, s5, 16
-; GFX10-NEXT:    s_lshr_b32 s3, s3, s7
-; GFX10-NEXT:    s_lshr_b32 s5, s6, s5
+; GFX10-NEXT:    s_and_b32 s3, s3, s10
+; GFX10-NEXT:    s_lshr_b32 s7, s5, 16
+; GFX10-NEXT:    s_lshr_b32 s3, s3, s5
+; GFX10-NEXT:    s_lshr_b32 s5, s6, s7
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s3, s3, s5
 ; GFX10-NEXT:    s_or_b32 s0, s0, s2
 ; GFX10-NEXT:    s_or_b32 s1, s1, s3
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <4 x i16> @llvm.fshl.v4i16(<4 x i16> %lhs, <4 x i16> %rhs, <4 x i16> %amt)
   %cast.result = bitcast <4 x i16> %result to <2 x i32>
   ret <2 x i32> %cast.result

 ; GFX6-NEXT:    v_bfe_u32 v5, v5, 0, 16
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v4, v5, v4
 ; GFX6-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fshl_v4i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    v_xor_b32_e32 v8, -1, v4
-; GFX8-NEXT:    v_lshrrev_b16_e32 v9, 1, v2
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 16, v4
-; GFX8-NEXT:    v_lshlrev_b16_e32 v4, v4, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v8, v8, v9
-; GFX8-NEXT:    v_or_b32_e32 v4, v4, v8
-; GFX8-NEXT:    v_xor_b32_e32 v8, -1, v6
-; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v6, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_mov_b32_e32 v6, 1
-; GFX8-NEXT:    v_lshrrev_b16_sdwa v2, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v8, v2
-; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v5
-; GFX8-NEXT:    v_lshrrev_b16_e32 v8, 1, v3
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v4
+; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v4
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b16_e32 v9, 1, v2
+; GFX8-NEXT:    v_lshlrev_b16_e32 v8, v8, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v4, v4, v9
+; GFX8-NEXT:    v_or_b32_e32 v4, v8, v4
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v6
+; GFX8-NEXT:    v_xor_b32_e32 v6, -1, v6
+; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v8, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_mov_b32_e32 v8, 1
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX8-NEXT:    v_lshrrev_b16_sdwa v2, v8, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v6, v2
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 16, v5
-; GFX8-NEXT:    v_lshlrev_b16_e32 v5, v5, v1
-; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v2, v8
-; GFX8-NEXT:    v_or_b32_e32 v2, v5, v2
-; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v7
-; GFX8-NEXT:    v_lshrrev_b16_sdwa v3, v6, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_lshlrev_b16_sdwa v1, v7, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_lshrrev_b16_e32 v3, v5, v3
+; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v5
+; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v5
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX8-NEXT:    v_lshrrev_b16_e32 v6, 1, v3
+; GFX8-NEXT:    v_lshlrev_b16_e32 v2, v2, v1
+; GFX8-NEXT:    v_lshrrev_b16_e32 v5, v5, v6
+; GFX8-NEXT:    v_or_b32_e32 v2, v2, v5
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v7
+; GFX8-NEXT:    v_xor_b32_e32 v6, -1, v7
+; GFX8-NEXT:    v_lshlrev_b16_sdwa v1, v5, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_mov_b32_e32 v5, 1
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX8-NEXT:    v_lshrrev_b16_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_lshrrev_b16_e32 v3, v6, v3
 ; GFX8-NEXT:    v_or_b32_e32 v1, v1, v3
 ; GFX8-NEXT:    v_mov_b32_e32 v3, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v4, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;

llvm/test/CodeGen/AMDGPU/GlobalISel/fshr.ll

 ; GFX6-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, v2, v1
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_fshr_i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    v_xor_b32_e32 v3, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v3, 15, v2
+; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v0, 1, v0
-; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v3, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v2, v1
+; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v2, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_fshr_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-NEXT:    v_xor_b32_e32 v3, -1, v2
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v2
+; GFX9-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v0, 1, v0
-; GFX9-NEXT:    v_lshlrev_b16_e32 v0, v3, v0
-; GFX9-NEXT:    v_lshrrev_b16_e32 v1, v2, v1
+; GFX9-NEXT:    v_lshlrev_b16_e32 v0, v2, v0
+; GFX9-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
 ; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: v_fshr_i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    v_xor_b32_e32 v3, -1, v2
 ; GFX10-NEXT:    v_lshlrev_b16 v0, 1, v0
+; GFX10-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX10-NEXT:    v_and_b32_e32 v3, 15, v3
 ; GFX10-NEXT:    v_lshrrev_b16 v1, v2, v1
 ; GFX10-NEXT:    v_lshlrev_b16 v0, v3, v0
 ; GFX10-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call i16 @llvm.fshr.i16(i16 %lhs, i16 %rhs, i16 %amt)
   ret i16 %result
 }
 

 ; GFX6-NEXT:    v_bfe_u32 v1, v1, 0, 16
 ; GFX6-NEXT:    s_and_b32 s0, s1, 0xffff
 ; GFX6-NEXT:    v_lshr_b32_e32 v1, s0, v1
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshr_i16_ssv:
 ; GFX8:       ; %bb.0:
+; GFX8-NEXT:    v_and_b32_e32 v1, 15, v0
+; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
 ; GFX8-NEXT:    s_bfe_u32 s2, 1, 0x100000
-; GFX8-NEXT:    v_xor_b32_e32 v1, -1, v0
+; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s2
-; GFX8-NEXT:    v_lshlrev_b16_e64 v1, v1, s0
-; GFX8-NEXT:    v_lshrrev_b16_e64 v0, v0, s1
-; GFX8-NEXT:    v_or_b32_e32 v0, v1, v0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_lshrrev_b16_e64 v1, v1, s1
+; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshr_i16_ssv:
 ; GFX9:       ; %bb.0:
+; GFX9-NEXT:    v_and_b32_e32 v1, 15, v0
+; GFX9-NEXT:    v_xor_b32_e32 v0, -1, v0
 ; GFX9-NEXT:    s_bfe_u32 s2, 1, 0x100000
-; GFX9-NEXT:    v_xor_b32_e32 v1, -1, v0
+; GFX9-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s2
-; GFX9-NEXT:    v_lshlrev_b16_e64 v1, v1, s0
-; GFX9-NEXT:    v_lshrrev_b16_e64 v0, v0, s1
-; GFX9-NEXT:    v_or_b32_e32 v0, v1, v0
+; GFX9-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX9-NEXT:    v_lshrrev_b16_e64 v1, v1, s1
+; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshr_i16_ssv:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    v_xor_b32_e32 v1, -1, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX10-NEXT:    s_bfe_u32 s2, 1, 0x100000
-; GFX10-NEXT:    v_lshrrev_b16 v0, v0, s1
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s2
+; GFX10-NEXT:    v_and_b32_e32 v1, 15, v1
+; GFX10-NEXT:    v_lshrrev_b16 v0, v0, s1
 ; GFX10-NEXT:    v_lshlrev_b16 v1, v1, s0
 ; GFX10-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call i16 @llvm.fshr.i16(i16 %lhs, i16 %rhs, i16 %amt)
   %cast.result = bitcast i16 %result to half
   ret half %cast.result
 }
 

 ; GFX9-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, 0x10001
 ; GFX9-NEXT:    s_lshl_b32 s3, s3, 1
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s3
 ; GFX9-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX9-NEXT:    s_lshr_b32 s5, s2, 16
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s2
 ; GFX9-NEXT:    s_lshl_b32 s2, s3, s5
-; GFX9-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX9-NEXT:    s_lshr_b32 s2, s1, 16
-; GFX9-NEXT:    s_and_b32 s1, s1, s3
-; GFX9-NEXT:    s_lshr_b32 s5, s4, 16
-; GFX9-NEXT:    s_and_b32 s3, s4, s3
-; GFX9-NEXT:    s_lshr_b32 s1, s1, s3
-; GFX9-NEXT:    s_lshr_b32 s2, s2, s5
+; GFX9-NEXT:    s_and_b32 s1, s1, 0xffff
+; GFX9-NEXT:    s_lshr_b32 s3, s4, 16
+; GFX9-NEXT:    s_lshr_b32 s1, s1, s4
+; GFX9-NEXT:    s_lshr_b32 s2, s2, s3
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX9-NEXT:    s_or_b32 s0, s0, s1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshr_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_lshr_b32 s4, s0, 16
 ; GFX10-NEXT:    s_mov_b32 s3, 0xf000f
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, 0x10001
 ; GFX10-NEXT:    s_lshl_b32 s4, s4, 1
 ; GFX10-NEXT:    s_and_b32 s5, s2, s3
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s4
 ; GFX10-NEXT:    s_andn2_b32 s2, s3, s2
 ; GFX10-NEXT:    s_lshr_b32 s3, s0, 16
 ; GFX10-NEXT:    s_lshr_b32 s4, s2, 16
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s2
 ; GFX10-NEXT:    s_lshl_b32 s2, s3, s4
-; GFX10-NEXT:    s_mov_b32 s3, 0xffff
-; GFX10-NEXT:    s_lshr_b32 s4, s1, 16
-; GFX10-NEXT:    s_and_b32 s1, s1, s3
-; GFX10-NEXT:    s_and_b32 s3, s5, s3
-; GFX10-NEXT:    s_lshr_b32 s5, s5, 16
-; GFX10-NEXT:    s_lshr_b32 s1, s1, s3
-; GFX10-NEXT:    s_lshr_b32 s3, s4, s5
+; GFX10-NEXT:    s_lshr_b32 s3, s1, 16
+; GFX10-NEXT:    s_and_b32 s1, s1, 0xffff
+; GFX10-NEXT:    s_lshr_b32 s4, s5, 16
+; GFX10-NEXT:    s_lshr_b32 s1, s1, s5
+; GFX10-NEXT:    s_lshr_b32 s3, s3, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s3
 ; GFX10-NEXT:    s_or_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i16> @llvm.fshr.v2i16(<2 x i16> %lhs, <2 x i16> %rhs, <2 x i16> %amt)
   %cast = bitcast <2 x i16> %result to i32
   ret i32 %cast
 }

 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v3, 1, v0
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v4, 15, v1
 ; GFX8-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX8-NEXT:    v_mov_b32_e32 v4, 1
 ; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v4, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b16_sdwa v5, v5, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v5
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v5, 1, v1
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v1, v4, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
-; GFX8-NEXT:    v_xor_b32_e32 v6, -1, v2
-; GFX8-NEXT:    v_lshlrev_b16_e32 v2, v2, v3
-; GFX8-NEXT:    v_lshrrev_b16_e32 v3, 1, v5
-; GFX8-NEXT:    v_lshrrev_b16_e32 v3, v6, v3
-; GFX8-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX8-NEXT:    v_xor_b32_e32 v3, -1, v4
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v2
+; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX8-NEXT:    v_lshrrev_b16_e32 v5, 1, v5
+; GFX8-NEXT:    v_lshlrev_b16_e32 v3, v6, v3
+; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v2, v5
+; GFX8-NEXT:    v_or_b32_e32 v2, v3, v2
+; GFX8-NEXT:    v_and_b32_e32 v3, 15, v4
+; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v4
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v1, 1, v1
-; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v4, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v3, v1
+; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v3, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v1, v4, v1
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v1, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_fshr_v2i16:
 ; GFX9:       ; %bb.0:

 ; GFX8-LABEL: v_fshr_v2i16_ssv:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_bfe_u32 s4, 1, 0x100000
 ; GFX8-NEXT:    s_bfe_u32 s5, s1, 0x100000
 ; GFX8-NEXT:    s_bfe_u32 s6, 15, 0x100000
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s4
 ; GFX8-NEXT:    s_lshr_b32 s5, s5, s6
+; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
 ; GFX8-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX8-NEXT:    s_or_b32 s0, s0, s5
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, s4
-; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v0
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v0, v0, s0
+; GFX8-NEXT:    v_xor_b32_e32 v0, -1, v0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v2, v2, s0
 ; GFX8-NEXT:    s_bfe_u32 s0, s1, 0x100000
+; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    s_lshr_b32 s0, s0, s4
 ; GFX8-NEXT:    s_lshr_b32 s5, s3, s6
 ; GFX8-NEXT:    s_lshl_b32 s3, s3, s4
-; GFX8-NEXT:    s_lshr_b32 s0, s0, s4
+; GFX8-NEXT:    v_lshrrev_b16_e64 v0, v0, s0
 ; GFX8-NEXT:    s_lshl_b32 s2, s2, s4
-; GFX8-NEXT:    v_lshrrev_b16_e64 v2, v2, s0
+; GFX8-NEXT:    v_or_b32_e32 v0, v2, v0
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v1
+; GFX8-NEXT:    v_xor_b32_e32 v1, -1, v1
 ; GFX8-NEXT:    s_bfe_u32 s0, s3, 0x100000
 ; GFX8-NEXT:    s_or_b32 s2, s2, s5
-; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2
-; GFX8-NEXT:    v_xor_b32_e32 v2, -1, v1
+; GFX8-NEXT:    v_and_b32_e32 v1, 15, v1
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s4
-; GFX8-NEXT:    v_lshlrev_b16_e64 v1, v1, s2
-; GFX8-NEXT:    v_lshrrev_b16_e64 v2, v2, s0
-; GFX8-NEXT:    v_or_b32_e32 v1, v1, v2
+; GFX8-NEXT:    v_lshlrev_b16_e64 v2, v2, s2
+; GFX8-NEXT:    v_lshrrev_b16_e64 v1, v1, s0
+; GFX8-NEXT:    v_or_b32_e32 v1, v2, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v2, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshr_v2i16_ssv:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s2, 0xf000f

 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshr_v2i16_vss:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s2, 0xf000f
 ; GFX9-NEXT:    s_and_b32 s3, s1, s2
 ; GFX9-NEXT:    s_andn2_b32 s1, s2, s1
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v0, 1, v0 op_sel_hi:[0,1]
-; GFX9-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v0, s1, v0
 ; GFX9-NEXT:    s_lshr_b32 s1, s0, 16
-; GFX9-NEXT:    s_and_b32 s0, s0, s2
-; GFX9-NEXT:    s_lshr_b32 s4, s3, 16
-; GFX9-NEXT:    s_and_b32 s2, s3, s2
-; GFX9-NEXT:    s_lshr_b32 s0, s0, s2
-; GFX9-NEXT:    s_lshr_b32 s1, s1, s4
+; GFX9-NEXT:    s_and_b32 s0, s0, 0xffff
+; GFX9-NEXT:    s_lshr_b32 s2, s3, 16
+; GFX9-NEXT:    s_lshr_b32 s0, s0, s3
+; GFX9-NEXT:    s_lshr_b32 s1, s1, s2
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX9-NEXT:    v_or_b32_e32 v0, s0, v0
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshr_v2i16_vss:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_pk_lshlrev_b16 v0, 1, v0 op_sel_hi:[0,1]
 ; GFX10-NEXT:    s_mov_b32 s2, 0xf000f
-; GFX10-NEXT:    s_mov_b32 s3, 0xffff
-; GFX10-NEXT:    s_and_b32 s4, s1, s2
+; GFX10-NEXT:    v_pk_lshlrev_b16 v0, 1, v0 op_sel_hi:[0,1]
+; GFX10-NEXT:    s_and_b32 s3, s1, s2
 ; GFX10-NEXT:    s_andn2_b32 s1, s2, s1
 ; GFX10-NEXT:    s_lshr_b32 s2, s0, 16
-; GFX10-NEXT:    s_and_b32 s0, s0, s3
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xffff
+; GFX10-NEXT:    s_lshr_b32 s4, s3, 16
 ; GFX10-NEXT:    v_pk_lshlrev_b16 v0, s1, v0
-; GFX10-NEXT:    s_and_b32 s1, s4, s3
-; GFX10-NEXT:    s_lshr_b32 s3, s4, 16
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX10-NEXT:    s_lshr_b32 s1, s2, s3
+; GFX10-NEXT:    s_lshr_b32 s0, s0, s3
+; GFX10-NEXT:    s_lshr_b32 s1, s2, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX10-NEXT:    v_or_b32_e32 v0, s0, v0
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i16> @llvm.fshr.v2i16(<2 x i16> %lhs, <2 x i16> %rhs, <2 x i16> %amt)
   %cast = bitcast <2 x i16> %result to float
   ret float %cast
 }
 

 ; GFX9-NEXT:    s_lshr_b32 s10, s4, 16
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s4
 ; GFX9-NEXT:    s_lshl_b32 s4, s9, s10
 ; GFX9-NEXT:    s_mov_b32 s9, 0xffff
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s2, 16
 ; GFX9-NEXT:    s_and_b32 s2, s2, s9
 ; GFX9-NEXT:    s_lshr_b32 s10, s7, 16
-; GFX9-NEXT:    s_and_b32 s7, s7, s9
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, s7
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, s10
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s4
 ; GFX9-NEXT:    s_or_b32 s0, s0, s2
 ; GFX9-NEXT:    s_and_b32 s2, s5, s6
 ; GFX9-NEXT:    s_andn2_b32 s4, s6, s5
 ; GFX9-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX9-NEXT:    s_lshl_b32 s1, s1, s8
 ; GFX9-NEXT:    s_lshl_b32 s5, s5, 1
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s5
 ; GFX9-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX9-NEXT:    s_lshr_b32 s6, s4, 16
 ; GFX9-NEXT:    s_lshl_b32 s1, s1, s4
 ; GFX9-NEXT:    s_lshl_b32 s4, s5, s6
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s3, 16
 ; GFX9-NEXT:    s_and_b32 s3, s3, s9
 ; GFX9-NEXT:    s_lshr_b32 s5, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s9
 ; GFX9-NEXT:    s_lshr_b32 s2, s3, s2
 ; GFX9-NEXT:    s_lshr_b32 s3, s4, s5
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s3
 ; GFX9-NEXT:    s_or_b32 s1, s1, s2
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshr_v4i16:
 ; GFX10:       ; %bb.0:

 ; GFX10-NEXT:    s_lshl_b32 s4, s4, 1
 ; GFX10-NEXT:    s_and_b32 s7, s5, s6
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX10-NEXT:    s_andn2_b32 s4, s6, s5
 ; GFX10-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX10-NEXT:    s_lshr_b32 s6, s4, 16
 ; GFX10-NEXT:    s_lshr_b32 s10, s2, 16
 ; GFX10-NEXT:    s_and_b32 s2, s2, s8
-; GFX10-NEXT:    s_and_b32 s11, s9, s8
-; GFX10-NEXT:    s_lshr_b32 s9, s9, 16
+; GFX10-NEXT:    s_lshr_b32 s11, s9, 16
 ; GFX10-NEXT:    s_lshl_b32 s1, s1, s4
 ; GFX10-NEXT:    s_lshl_b32 s4, s5, s6
 ; GFX10-NEXT:    s_lshr_b32 s5, s3, 16
 ; GFX10-NEXT:    s_and_b32 s3, s3, s8
-; GFX10-NEXT:    s_and_b32 s6, s7, s8
-; GFX10-NEXT:    s_lshr_b32 s7, s7, 16
-; GFX10-NEXT:    s_lshr_b32 s2, s2, s11
-; GFX10-NEXT:    s_lshr_b32 s9, s10, s9
-; GFX10-NEXT:    s_lshr_b32 s3, s3, s6
-; GFX10-NEXT:    s_lshr_b32 s5, s5, s7
+; GFX10-NEXT:    s_lshr_b32 s6, s7, 16
+; GFX10-NEXT:    s_lshr_b32 s2, s2, s9
+; GFX10-NEXT:    s_lshr_b32 s9, s10, s11
+; GFX10-NEXT:    s_lshr_b32 s3, s3, s7
+; GFX10-NEXT:    s_lshr_b32 s5, s5, s6
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s2, s2, s9
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s3, s3, s5
 ; GFX10-NEXT:    s_or_b32 s0, s0, s2
 ; GFX10-NEXT:    s_or_b32 s1, s1, s3
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <4 x i16> @llvm.fshr.v4i16(<4 x i16> %lhs, <4 x i16> %rhs, <4 x i16> %amt)
   %cast.result = bitcast <4 x i16> %result to <2 x i32>

 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v6, 1, v0
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v7, 15, v2
 ; GFX8-NEXT:    v_or_b32_e32 v6, v6, v7
 ; GFX8-NEXT:    v_mov_b32_e32 v7, 1
 ; GFX8-NEXT:    v_mov_b32_e32 v8, 15
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v0, v7, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b16_sdwa v9, v8, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v4
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v9
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v9, 1, v2
-; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v4
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v2, v7, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 16, v4
-; GFX8-NEXT:    v_xor_b32_e32 v10, -1, v4
-; GFX8-NEXT:    v_lshlrev_b16_e32 v4, v4, v6
-; GFX8-NEXT:    v_lshrrev_b16_e32 v6, 1, v9
-; GFX8-NEXT:    v_lshrrev_b16_e32 v6, v10, v6
-; GFX8-NEXT:    v_or_b32_e32 v4, v4, v6
-; GFX8-NEXT:    v_xor_b32_e32 v6, -1, v7
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v4
+; GFX8-NEXT:    v_xor_b32_e32 v4, -1, v4
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b16_e32 v9, 1, v9
+; GFX8-NEXT:    v_lshlrev_b16_e32 v6, v10, v6
+; GFX8-NEXT:    v_lshrrev_b16_e32 v4, v4, v9
+; GFX8-NEXT:    v_or_b32_e32 v4, v6, v4
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v7
+; GFX8-NEXT:    v_xor_b32_e32 v7, -1, v7
+; GFX8-NEXT:    v_and_b32_e32 v7, 15, v7
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v2, 1, v2
-; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v7, v0
-; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v6, v2
+; GFX8-NEXT:    v_lshlrev_b16_e32 v0, v6, v0
+; GFX8-NEXT:    v_lshrrev_b16_e32 v2, v7, v2
 ; GFX8-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v2, 16
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v0, v4, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v4, 1, v1
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v6, 15, v3
 ; GFX8-NEXT:    v_or_b32_e32 v4, v4, v6
 ; GFX8-NEXT:    v_mov_b32_e32 v6, 1
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v1, v6, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b16_sdwa v7, v8, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v5
 ; GFX8-NEXT:    v_or_b32_e32 v1, v1, v7
 ; GFX8-NEXT:    v_lshlrev_b16_e32 v7, 1, v3
-; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v5
 ; GFX8-NEXT:    v_lshlrev_b16_sdwa v3, v6, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 16, v5
-; GFX8-NEXT:    v_xor_b32_e32 v8, -1, v5
-; GFX8-NEXT:    v_lshlrev_b16_e32 v4, v5, v4
-; GFX8-NEXT:    v_lshrrev_b16_e32 v5, 1, v7
-; GFX8-NEXT:    v_lshrrev_b16_e32 v5, v8, v5
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v5
+; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v5
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX8-NEXT:    v_lshrrev_b16_e32 v7, 1, v7
+; GFX8-NEXT:    v_lshlrev_b16_e32 v4, v8, v4
+; GFX8-NEXT:    v_lshrrev_b16_e32 v5, v5, v7
 ; GFX8-NEXT:    v_or_b32_e32 v4, v4, v5
-; GFX8-NEXT:    v_xor_b32_e32 v5, -1, v6
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v6
+; GFX8-NEXT:    v_xor_b32_e32 v6, -1, v6
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v6
 ; GFX8-NEXT:    v_lshrrev_b16_e32 v3, 1, v3
-; GFX8-NEXT:    v_lshlrev_b16_e32 v1, v6, v1
-; GFX8-NEXT:    v_lshrrev_b16_e32 v3, v5, v3
+; GFX8-NEXT:    v_lshlrev_b16_e32 v1, v5, v1
+; GFX8-NEXT:    v_lshrrev_b16_e32 v3, v6, v3
 ; GFX8-NEXT:    v_or_b32_e32 v1, v1, v3
 ; GFX8-NEXT:    v_lshlrev_b32_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX8-NEXT:    v_or_b32_sdwa v1, v4, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_fshr_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/GlobalISel/lshr.ll

 ; GFX10-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX10-NEXT:    v_lshrrev_b16 v0, 7, v0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = lshr i8 %value, 7
   ret i8 %result
 }
 
 define amdgpu_ps i8 @s_lshr_i8(i8 inreg %value, i8 inreg %amount) {
-; GFX6-LABEL: s_lshr_i8:
-; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s2, 0xff
-; GFX6-NEXT:    s_and_b32 s1, s1, s2
-; GFX6-NEXT:    s_and_b32 s0, s0, s2
-; GFX6-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX6-NEXT:    ; return to shader part epilog
-;
-; GFX8-LABEL: s_lshr_i8:
-; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s2, 0xff
-; GFX8-NEXT:    s_and_b32 s0, s0, s2
-; GFX8-NEXT:    s_and_b32 s1, s1, s2
-; GFX8-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX8-NEXT:    ; return to shader part epilog
-;
-; GFX9-LABEL: s_lshr_i8:
-; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s2, 0xff
-; GFX9-NEXT:    s_and_b32 s0, s0, s2
-; GFX9-NEXT:    s_and_b32 s1, s1, s2
-; GFX9-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX9-NEXT:    ; return to shader part epilog
+; GCN-LABEL: s_lshr_i8:
+; GCN:       ; %bb.0:
+; GCN-NEXT:    s_and_b32 s0, s0, 0xff
+; GCN-NEXT:    s_lshr_b32 s0, s0, s1
+; GCN-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_i8:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s2, 0xff
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
-; GFX10-NEXT:    s_and_b32 s1, s1, s2
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xff
 ; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr i8 %value, %amount
   ret i8 %result
 }
 
 define amdgpu_ps i8 @s_lshr_i8_7(i8 inreg %value) {
 ; GCN-LABEL: s_lshr_i8_7:

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = lshr i24 %value, 7
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_lshr_i24(i24 inreg %value, i24 inreg %amount) {
 ; GCN-LABEL: s_lshr_i24:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_mov_b32 s2, 0xffffff
-; GCN-NEXT:    s_and_b32 s1, s1, s2
-; GCN-NEXT:    s_and_b32 s0, s0, s2
+; GCN-NEXT:    s_and_b32 s0, s0, 0xffffff
 ; GCN-NEXT:    s_lshr_b32 s0, s0, s1
 ; GCN-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_i24:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_mov_b32 s2, 0xffffff
-; GFX10-NEXT:    s_and_b32 s1, s1, s2
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xffffff
 ; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr i24 %value, %amount
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_lshr_i24_7(i24 inreg %value) {
 ; GCN-LABEL: s_lshr_i24_7:

 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = lshr i16 %value, 31
   ret i16 %result
 }
 
 define amdgpu_ps i16 @s_lshr_i16(i16 inreg %value, i16 inreg %amount) {
-; GFX6-LABEL: s_lshr_i16:
-; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_mov_b32 s2, 0xffff
-; GFX6-NEXT:    s_and_b32 s1, s1, s2
-; GFX6-NEXT:    s_and_b32 s0, s0, s2
-; GFX6-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX6-NEXT:    ; return to shader part epilog
-;
-; GFX8-LABEL: s_lshr_i16:
-; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_mov_b32 s2, 0xffff
-; GFX8-NEXT:    s_and_b32 s0, s0, s2
-; GFX8-NEXT:    s_and_b32 s1, s1, s2
-; GFX8-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX8-NEXT:    ; return to shader part epilog
-;
-; GFX9-LABEL: s_lshr_i16:
-; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_mov_b32 s2, 0xffff
-; GFX9-NEXT:    s_and_b32 s0, s0, s2
-; GFX9-NEXT:    s_and_b32 s1, s1, s2
-; GFX9-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX9-NEXT:    ; return to shader part epilog
+; GCN-LABEL: s_lshr_i16:
+; GCN:       ; %bb.0:
+; GCN-NEXT:    s_and_b32 s0, s0, 0xffff
+; GCN-NEXT:    s_lshr_b32 s0, s0, s1
+; GCN-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_i16:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_mov_b32 s2, 0xffff
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
-; GFX10-NEXT:    s_and_b32 s1, s1, s2
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xffff
 ; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr i16 %value, %amount
   ret i16 %result
 }
 
 define amdgpu_ps i16 @s_lshr_i16_15(i16 inreg %value) {
 ; GCN-LABEL: s_lshr_i16_15:
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = lshr <2 x i16> %value, <i16 15, i16 15>
   ret <2 x i16> %result
 }
 
 define amdgpu_ps i32 @s_lshr_v2i16(<2 x i16> inreg %value, <2 x i16> inreg %amount) {
 ; GFX6-LABEL: s_lshr_v2i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s4, 0xffff
-; GFX6-NEXT:    s_and_b32 s2, s2, s4
+; GFX6-NEXT:    s_and_b32 s1, s1, s4
 ; GFX6-NEXT:    s_and_b32 s0, s0, s4
+; GFX6-NEXT:    s_lshr_b32 s1, s1, s3
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, s2
-; GFX6-NEXT:    s_and_b32 s2, s3, s4
-; GFX6-NEXT:    s_and_b32 s1, s1, s4
-; GFX6-NEXT:    s_lshr_b32 s1, s1, s2
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_lshr_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s3
 ; GFX8-NEXT:    s_lshr_b32 s4, s1, 16
-; GFX8-NEXT:    s_and_b32 s1, s1, s3
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s1
 ; GFX8-NEXT:    s_lshr_b32 s1, s2, s4
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s3
 ; GFX8-NEXT:    s_or_b32 s0, s1, s0
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_lshr_v2i16:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX9-NEXT:    s_lshr_b32 s2, s0, 16
-; GFX9-NEXT:    s_and_b32 s0, s0, s3
-; GFX9-NEXT:    s_lshr_b32 s4, s1, 16
-; GFX9-NEXT:    s_and_b32 s1, s1, s3
+; GFX9-NEXT:    s_and_b32 s0, s0, 0xffff
+; GFX9-NEXT:    s_lshr_b32 s3, s1, 16
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, s1
-; GFX9-NEXT:    s_lshr_b32 s1, s2, s4
+; GFX9-NEXT:    s_lshr_b32 s1, s2, s3
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_v2i16:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_mov_b32 s2, 0xffff
-; GFX10-NEXT:    s_lshr_b32 s3, s0, 16
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
-; GFX10-NEXT:    s_and_b32 s2, s1, s2
-; GFX10-NEXT:    s_lshr_b32 s1, s1, 16
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s2
-; GFX10-NEXT:    s_lshr_b32 s1, s3, s1
+; GFX10-NEXT:    s_lshr_b32 s2, s0, 16
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xffff
+; GFX10-NEXT:    s_lshr_b32 s3, s1, 16
+; GFX10-NEXT:    s_lshr_b32 s0, s0, s1
+; GFX10-NEXT:    s_lshr_b32 s1, s2, s3
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr <2 x i16> %value, %amount
   %cast = bitcast <2 x i16> %result to i32
   ret i32 %cast
 }
 
 define amdgpu_ps float @lshr_v2i16_sv(<2 x i16> inreg %value, <2 x i16> %amount) {
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %cast = bitcast <4 x i16> %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x i32> @s_lshr_v4i16(<4 x i16> inreg %value, <4 x i16> inreg %amount) {
 ; GFX6-LABEL: s_lshr_v4i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s8, 0xffff
-; GFX6-NEXT:    s_and_b32 s4, s4, s8
+; GFX6-NEXT:    s_and_b32 s1, s1, s8
 ; GFX6-NEXT:    s_and_b32 s0, s0, s8
+; GFX6-NEXT:    s_lshr_b32 s1, s1, s5
+; GFX6-NEXT:    s_and_b32 s3, s3, s8
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, s4
-; GFX6-NEXT:    s_and_b32 s4, s5, s8
-; GFX6-NEXT:    s_and_b32 s1, s1, s8
-; GFX6-NEXT:    s_lshr_b32 s1, s1, s4
-; GFX6-NEXT:    s_and_b32 s4, s6, s8
 ; GFX6-NEXT:    s_and_b32 s2, s2, s8
-; GFX6-NEXT:    s_lshr_b32 s2, s2, s4
-; GFX6-NEXT:    s_and_b32 s4, s7, s8
-; GFX6-NEXT:    s_and_b32 s3, s3, s8
-; GFX6-NEXT:    s_lshr_b32 s3, s3, s4
+; GFX6-NEXT:    s_lshr_b32 s3, s3, s7
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
+; GFX6-NEXT:    s_lshr_b32 s2, s2, s6
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_lshl_b32 s1, s3, 16
 ; GFX6-NEXT:    s_or_b32 s1, s2, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_lshr_v4i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s6, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s4, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s6
 ; GFX8-NEXT:    s_lshr_b32 s7, s2, 16
-; GFX8-NEXT:    s_and_b32 s2, s2, s6
 ; GFX8-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s6
 ; GFX8-NEXT:    s_lshr_b32 s8, s3, 16
-; GFX8-NEXT:    s_and_b32 s3, s3, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s2
 ; GFX8-NEXT:    s_lshr_b32 s2, s4, s7
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, s3
 ; GFX8-NEXT:    s_lshr_b32 s3, s5, s8
 ; GFX8-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s6
 ; GFX8-NEXT:    s_or_b32 s0, s2, s0
 ; GFX8-NEXT:    s_lshl_b32 s2, s3, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s6
 ; GFX8-NEXT:    s_or_b32 s1, s2, s1
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_lshr_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s5, 0xffff
 ; GFX9-NEXT:    s_lshr_b32 s4, s0, 16
 ; GFX9-NEXT:    s_and_b32 s0, s0, s5
 ; GFX9-NEXT:    s_lshr_b32 s6, s2, 16
-; GFX9-NEXT:    s_and_b32 s2, s2, s5
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, s2
 ; GFX9-NEXT:    s_lshr_b32 s2, s4, s6
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX9-NEXT:    s_lshr_b32 s2, s1, 16
 ; GFX9-NEXT:    s_and_b32 s1, s1, s5
 ; GFX9-NEXT:    s_lshr_b32 s4, s3, 16
-; GFX9-NEXT:    s_and_b32 s3, s3, s5
 ; GFX9-NEXT:    s_lshr_b32 s1, s1, s3
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, s4
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s2
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_v4i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX10-NEXT:    s_lshr_b32 s5, s0, 16
-; GFX10-NEXT:    s_and_b32 s6, s2, s4
-; GFX10-NEXT:    s_lshr_b32 s2, s2, 16
 ; GFX10-NEXT:    s_and_b32 s0, s0, s4
-; GFX10-NEXT:    s_lshr_b32 s2, s5, s2
+; GFX10-NEXT:    s_lshr_b32 s6, s2, 16
+; GFX10-NEXT:    s_lshr_b32 s0, s0, s2
+; GFX10-NEXT:    s_lshr_b32 s2, s5, s6
 ; GFX10-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX10-NEXT:    s_and_b32 s1, s1, s4
-; GFX10-NEXT:    s_and_b32 s4, s3, s4
-; GFX10-NEXT:    s_lshr_b32 s3, s3, 16
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s6
-; GFX10-NEXT:    s_lshr_b32 s1, s1, s4
-; GFX10-NEXT:    s_lshr_b32 s3, s5, s3
+; GFX10-NEXT:    s_lshr_b32 s4, s3, 16
+; GFX10-NEXT:    s_lshr_b32 s1, s1, s3
+; GFX10-NEXT:    s_lshr_b32 s3, s5, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s3
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr <4 x i16> %value, %amount
   %cast = bitcast <4 x i16> %result to <2 x i32>
   ret <2 x i32> %cast
 }
 
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %cast = bitcast <8 x i16> %result to <4 x float>
   ret <4 x float> %cast
 }
 
 define amdgpu_ps <4 x i32> @s_lshr_v8i16(<8 x i16> inreg %value, <8 x i16> inreg %amount) {
 ; GFX6-LABEL: s_lshr_v8i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s16, 0xffff
-; GFX6-NEXT:    s_and_b32 s8, s8, s16
+; GFX6-NEXT:    s_and_b32 s1, s1, s16
 ; GFX6-NEXT:    s_and_b32 s0, s0, s16
+; GFX6-NEXT:    s_lshr_b32 s1, s1, s9
+; GFX6-NEXT:    s_and_b32 s3, s3, s16
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, s8
-; GFX6-NEXT:    s_and_b32 s8, s9, s16
-; GFX6-NEXT:    s_and_b32 s1, s1, s16
-; GFX6-NEXT:    s_lshr_b32 s1, s1, s8
-; GFX6-NEXT:    s_and_b32 s8, s10, s16
 ; GFX6-NEXT:    s_and_b32 s2, s2, s16
-; GFX6-NEXT:    s_lshr_b32 s2, s2, s8
-; GFX6-NEXT:    s_and_b32 s8, s11, s16
-; GFX6-NEXT:    s_and_b32 s3, s3, s16
-; GFX6-NEXT:    s_lshr_b32 s3, s3, s8
-; GFX6-NEXT:    s_and_b32 s8, s12, s16
-; GFX6-NEXT:    s_and_b32 s4, s4, s16
-; GFX6-NEXT:    s_lshr_b32 s4, s4, s8
-; GFX6-NEXT:    s_and_b32 s8, s13, s16
+; GFX6-NEXT:    s_lshr_b32 s3, s3, s11
 ; GFX6-NEXT:    s_and_b32 s5, s5, s16
-; GFX6-NEXT:    s_lshr_b32 s5, s5, s8
-; GFX6-NEXT:    s_and_b32 s8, s14, s16
-; GFX6-NEXT:    s_and_b32 s6, s6, s16
-; GFX6-NEXT:    s_lshr_b32 s6, s6, s8
-; GFX6-NEXT:    s_and_b32 s8, s15, s16
 ; GFX6-NEXT:    s_and_b32 s7, s7, s16
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
-; GFX6-NEXT:    s_lshr_b32 s7, s7, s8
+; GFX6-NEXT:    s_lshr_b32 s2, s2, s10
+; GFX6-NEXT:    s_and_b32 s4, s4, s16
+; GFX6-NEXT:    s_lshr_b32 s5, s5, s13
+; GFX6-NEXT:    s_and_b32 s6, s6, s16
+; GFX6-NEXT:    s_lshr_b32 s7, s7, s15
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_lshl_b32 s1, s3, 16
+; GFX6-NEXT:    s_lshr_b32 s4, s4, s12
+; GFX6-NEXT:    s_lshr_b32 s6, s6, s14
 ; GFX6-NEXT:    s_or_b32 s1, s2, s1
 ; GFX6-NEXT:    s_lshl_b32 s2, s5, 16
 ; GFX6-NEXT:    s_lshl_b32 s3, s7, 16
 ; GFX6-NEXT:    s_or_b32 s2, s4, s2
 ; GFX6-NEXT:    s_or_b32 s3, s6, s3
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_lshr_v8i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s12, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s8, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s12
 ; GFX8-NEXT:    s_lshr_b32 s13, s4, 16
-; GFX8-NEXT:    s_and_b32 s4, s4, s12
 ; GFX8-NEXT:    s_lshr_b32 s9, s1, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s12
 ; GFX8-NEXT:    s_lshr_b32 s14, s5, 16
-; GFX8-NEXT:    s_and_b32 s5, s5, s12
 ; GFX8-NEXT:    s_lshr_b32 s0, s0, s4
 ; GFX8-NEXT:    s_lshr_b32 s4, s8, s13
 ; GFX8-NEXT:    s_lshr_b32 s10, s2, 16
 ; GFX8-NEXT:    s_and_b32 s2, s2, s12
 ; GFX8-NEXT:    s_lshr_b32 s15, s6, 16
-; GFX8-NEXT:    s_and_b32 s6, s6, s12
 ; GFX8-NEXT:    s_lshr_b32 s1, s1, s5
 ; GFX8-NEXT:    s_lshr_b32 s5, s9, s14
 ; GFX8-NEXT:    s_lshl_b32 s4, s4, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s12
 ; GFX8-NEXT:    s_lshr_b32 s11, s3, 16
 ; GFX8-NEXT:    s_and_b32 s3, s3, s12
 ; GFX8-NEXT:    s_lshr_b32 s16, s7, 16
-; GFX8-NEXT:    s_and_b32 s7, s7, s12
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, s6
 ; GFX8-NEXT:    s_lshr_b32 s6, s10, s15
 ; GFX8-NEXT:    s_or_b32 s0, s4, s0
 ; GFX8-NEXT:    s_lshl_b32 s4, s5, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s12
 ; GFX8-NEXT:    s_lshr_b32 s3, s3, s7
 ; GFX8-NEXT:    s_lshr_b32 s7, s11, s16
 ; GFX8-NEXT:    s_or_b32 s1, s4, s1
 ; GFX8-NEXT:    s_lshl_b32 s4, s6, 16
 ; GFX8-NEXT:    s_and_b32 s2, s2, s12
 ; GFX8-NEXT:    s_or_b32 s2, s4, s2
 ; GFX8-NEXT:    s_lshl_b32 s4, s7, 16
 ; GFX8-NEXT:    s_and_b32 s3, s3, s12
 ; GFX8-NEXT:    s_or_b32 s3, s4, s3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_lshr_v8i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_mov_b32 s9, 0xffff
 ; GFX9-NEXT:    s_lshr_b32 s8, s0, 16
 ; GFX9-NEXT:    s_and_b32 s0, s0, s9
 ; GFX9-NEXT:    s_lshr_b32 s10, s4, 16
-; GFX9-NEXT:    s_and_b32 s4, s4, s9
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s8, s10
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s1, 16
 ; GFX9-NEXT:    s_and_b32 s1, s1, s9
 ; GFX9-NEXT:    s_lshr_b32 s8, s5, 16
-; GFX9-NEXT:    s_and_b32 s5, s5, s9
 ; GFX9-NEXT:    s_lshr_b32 s1, s1, s5
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, s8
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s2, 16
 ; GFX9-NEXT:    s_and_b32 s2, s2, s9
 ; GFX9-NEXT:    s_lshr_b32 s5, s6, 16
-; GFX9-NEXT:    s_and_b32 s6, s6, s9
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, s6
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, s5
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s2, s4
 ; GFX9-NEXT:    s_lshr_b32 s4, s3, 16
 ; GFX9-NEXT:    s_and_b32 s3, s3, s9
 ; GFX9-NEXT:    s_lshr_b32 s5, s7, 16
-; GFX9-NEXT:    s_and_b32 s6, s7, s9
-; GFX9-NEXT:    s_lshr_b32 s3, s3, s6
+; GFX9-NEXT:    s_lshr_b32 s3, s3, s7
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, s5
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s3, s3, s4
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_lshr_v8i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX10-NEXT:    s_lshr_b32 s9, s0, 16
 ; GFX10-NEXT:    s_and_b32 s0, s0, s8
-; GFX10-NEXT:    s_and_b32 s10, s4, s8
-; GFX10-NEXT:    s_lshr_b32 s4, s4, 16
-; GFX10-NEXT:    s_lshr_b32 s0, s0, s10
-; GFX10-NEXT:    s_lshr_b32 s4, s9, s4
+; GFX10-NEXT:    s_lshr_b32 s10, s4, 16
+; GFX10-NEXT:    s_lshr_b32 s0, s0, s4
+; GFX10-NEXT:    s_lshr_b32 s4, s9, s10
 ; GFX10-NEXT:    s_lshr_b32 s9, s1, 16
 ; GFX10-NEXT:    s_and_b32 s1, s1, s8
-; GFX10-NEXT:    s_and_b32 s10, s5, s8
-; GFX10-NEXT:    s_lshr_b32 s5, s5, 16
-; GFX10-NEXT:    s_lshr_b32 s1, s1, s10
-; GFX10-NEXT:    s_lshr_b32 s5, s9, s5
+; GFX10-NEXT:    s_lshr_b32 s10, s5, 16
+; GFX10-NEXT:    s_lshr_b32 s1, s1, s5
+; GFX10-NEXT:    s_lshr_b32 s5, s9, s10
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s0, s0, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s1, s1, s5
 ; GFX10-NEXT:    s_lshr_b32 s4, s2, 16
 ; GFX10-NEXT:    s_and_b32 s2, s2, s8
-; GFX10-NEXT:    s_and_b32 s5, s6, s8
-; GFX10-NEXT:    s_lshr_b32 s6, s6, 16
-; GFX10-NEXT:    s_lshr_b32 s2, s2, s5
-; GFX10-NEXT:    s_lshr_b32 s4, s4, s6
+; GFX10-NEXT:    s_lshr_b32 s5, s6, 16
+; GFX10-NEXT:    s_lshr_b32 s2, s2, s6
+; GFX10-NEXT:    s_lshr_b32 s4, s4, s5
 ; GFX10-NEXT:    s_lshr_b32 s5, s3, 16
 ; GFX10-NEXT:    s_and_b32 s3, s3, s8
-; GFX10-NEXT:    s_and_b32 s6, s7, s8
-; GFX10-NEXT:    s_lshr_b32 s7, s7, 16
-; GFX10-NEXT:    s_lshr_b32 s3, s3, s6
-; GFX10-NEXT:    s_lshr_b32 s5, s5, s7
+; GFX10-NEXT:    s_lshr_b32 s6, s7, 16
+; GFX10-NEXT:    s_lshr_b32 s3, s3, s7
+; GFX10-NEXT:    s_lshr_b32 s5, s5, s6
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s2, s2, s4
 ; GFX10-NEXT:    s_pack_ll_b32_b16 s3, s3, s5
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = lshr <8 x i16> %value, %amount
   %cast = bitcast <8 x i16> %result to <4 x i32>
   ret <4 x i32> %cast
 }
 

llvm/test/CodeGen/AMDGPU/GlobalISel/shl.ll

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = shl i8 %value, 7
   ret i8 %result
 }
 
 define amdgpu_ps i8 @s_shl_i8(i8 inreg %value, i8 inreg %amount) {
 ; GFX6-LABEL: s_shl_i8:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_and_b32 s1, s1, 0xff
 ; GFX6-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_shl_i8:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s2, 0xff
-; GFX8-NEXT:    s_and_b32 s0, s0, s2
-; GFX8-NEXT:    s_and_b32 s1, s1, s2
+; GFX8-NEXT:    s_and_b32 s0, s0, 0xff
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_shl_i8:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s2, 0xff
-; GFX9-NEXT:    s_and_b32 s0, s0, s2
-; GFX9-NEXT:    s_and_b32 s1, s1, s2
+; GFX9-NEXT:    s_and_b32 s0, s0, 0xff
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_shl_i8:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s2, 0xff
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
-; GFX10-NEXT:    s_and_b32 s1, s1, s2
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xff
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = shl i8 %value, %amount
   ret i8 %result
 }
 
 define amdgpu_ps i8 @s_shl_i8_7(i8 inreg %value) {
 ; GFX6-LABEL: s_shl_i8_7:

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = shl i24 %value, 7
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_shl_i24(i24 inreg %value, i24 inreg %amount) {
 ; GCN-LABEL: s_shl_i24:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_and_b32 s1, s1, 0xffffff
 ; GCN-NEXT:    s_lshl_b32 s0, s0, s1
 ; GCN-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_shl_i24:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_and_b32 s1, s1, 0xffffff
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = shl i24 %value, %amount
   ret i24 %result
 }
 
 define amdgpu_ps i24 @s_shl_i24_7(i24 inreg %value) {
 ; GCN-LABEL: s_shl_i24_7:

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = shl i16 %value, 31
   ret i16 %result
 }
 
 define amdgpu_ps i16 @s_shl_i16(i16 inreg %value, i16 inreg %amount) {
 ; GFX6-LABEL: s_shl_i16:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_and_b32 s1, s1, 0xffff
 ; GFX6-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_shl_i16:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_mov_b32 s2, 0xffff
-; GFX8-NEXT:    s_and_b32 s0, s0, s2
-; GFX8-NEXT:    s_and_b32 s1, s1, s2
+; GFX8-NEXT:    s_and_b32 s0, s0, 0xffff
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_shl_i16:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_mov_b32 s2, 0xffff
-; GFX9-NEXT:    s_and_b32 s0, s0, s2
-; GFX9-NEXT:    s_and_b32 s1, s1, s2
+; GFX9-NEXT:    s_and_b32 s0, s0, 0xffff
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_shl_i16:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_mov_b32 s2, 0xffff
-; GFX10-NEXT:    s_and_b32 s0, s0, s2
-; GFX10-NEXT:    s_and_b32 s1, s1, s2
+; GFX10-NEXT:    s_and_b32 s0, s0, 0xffff
 ; GFX10-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = shl i16 %value, %amount
   ret i16 %result
 }
 
 define amdgpu_ps i16 @s_shl_i16_15(i16 inreg %value) {
 ; GFX6-LABEL: s_shl_i16_15:
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = shl <2 x i16> %value, <i16 15, i16 15>
   ret <2 x i16> %result
 }
 
 define amdgpu_ps i32 @s_shl_v2i16(<2 x i16> inreg %value, <2 x i16> inreg %amount) {
 ; GFX6-LABEL: s_shl_v2i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s4, 0xffff
-; GFX6-NEXT:    s_and_b32 s2, s2, s4
+; GFX6-NEXT:    s_lshl_b32 s1, s1, s3
 ; GFX6-NEXT:    s_lshl_b32 s0, s0, s2
-; GFX6-NEXT:    s_and_b32 s2, s3, s4
-; GFX6-NEXT:    s_lshl_b32 s1, s1, s2
 ; GFX6-NEXT:    s_and_b32 s1, s1, s4
 ; GFX6-NEXT:    s_and_b32 s0, s0, s4
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_shl_v2i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s3, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s3
 ; GFX8-NEXT:    s_lshr_b32 s4, s1, 16
-; GFX8-NEXT:    s_and_b32 s1, s1, s3
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s1
 ; GFX8-NEXT:    s_lshl_b32 s1, s2, s4
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s3
 ; GFX8-NEXT:    s_or_b32 s0, s1, s0
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_shl_v2i16:
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %cast = bitcast <4 x i16> %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x i32> @s_shl_v4i16(<4 x i16> inreg %value, <4 x i16> inreg %amount) {
 ; GFX6-LABEL: s_shl_v4i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s8, 0xffff
-; GFX6-NEXT:    s_and_b32 s4, s4, s8
+; GFX6-NEXT:    s_lshl_b32 s1, s1, s5
 ; GFX6-NEXT:    s_lshl_b32 s0, s0, s4
-; GFX6-NEXT:    s_and_b32 s4, s5, s8
-; GFX6-NEXT:    s_lshl_b32 s1, s1, s4
-; GFX6-NEXT:    s_and_b32 s4, s6, s8
-; GFX6-NEXT:    s_lshl_b32 s2, s2, s4
-; GFX6-NEXT:    s_and_b32 s4, s7, s8
 ; GFX6-NEXT:    s_and_b32 s1, s1, s8
-; GFX6-NEXT:    s_lshl_b32 s3, s3, s4
+; GFX6-NEXT:    s_lshl_b32 s2, s2, s6
+; GFX6-NEXT:    s_lshl_b32 s3, s3, s7
 ; GFX6-NEXT:    s_and_b32 s0, s0, s8
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_and_b32 s1, s2, s8
 ; GFX6-NEXT:    s_and_b32 s2, s3, s8
 ; GFX6-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX6-NEXT:    s_or_b32 s1, s1, s2
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_shl_v4i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s6, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s4, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s6
 ; GFX8-NEXT:    s_lshr_b32 s7, s2, 16
-; GFX8-NEXT:    s_and_b32 s2, s2, s6
 ; GFX8-NEXT:    s_lshr_b32 s5, s1, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s6
 ; GFX8-NEXT:    s_lshr_b32 s8, s3, 16
-; GFX8-NEXT:    s_and_b32 s3, s3, s6
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s2
 ; GFX8-NEXT:    s_lshl_b32 s2, s4, s7
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, s3
 ; GFX8-NEXT:    s_lshl_b32 s3, s5, s8
 ; GFX8-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s6
 ; GFX8-NEXT:    s_or_b32 s0, s2, s0
 ; GFX8-NEXT:    s_lshl_b32 s2, s3, 16
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %cast = bitcast <8 x i16> %result to <4 x float>
   ret <4 x float> %cast
 }
 
 define amdgpu_ps <4 x i32> @s_shl_v8i16(<8 x i16> inreg %value, <8 x i16> inreg %amount) {
 ; GFX6-LABEL: s_shl_v8i16:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_mov_b32 s16, 0xffff
-; GFX6-NEXT:    s_and_b32 s8, s8, s16
+; GFX6-NEXT:    s_lshl_b32 s1, s1, s9
 ; GFX6-NEXT:    s_lshl_b32 s0, s0, s8
-; GFX6-NEXT:    s_and_b32 s8, s9, s16
-; GFX6-NEXT:    s_lshl_b32 s1, s1, s8
-; GFX6-NEXT:    s_and_b32 s8, s10, s16
-; GFX6-NEXT:    s_lshl_b32 s2, s2, s8
-; GFX6-NEXT:    s_and_b32 s8, s11, s16
-; GFX6-NEXT:    s_lshl_b32 s3, s3, s8
-; GFX6-NEXT:    s_and_b32 s8, s12, s16
-; GFX6-NEXT:    s_lshl_b32 s4, s4, s8
-; GFX6-NEXT:    s_and_b32 s8, s13, s16
 ; GFX6-NEXT:    s_and_b32 s1, s1, s16
-; GFX6-NEXT:    s_lshl_b32 s5, s5, s8
-; GFX6-NEXT:    s_and_b32 s8, s14, s16
+; GFX6-NEXT:    s_lshl_b32 s2, s2, s10
+; GFX6-NEXT:    s_lshl_b32 s3, s3, s11
 ; GFX6-NEXT:    s_and_b32 s0, s0, s16
 ; GFX6-NEXT:    s_lshl_b32 s1, s1, 16
-; GFX6-NEXT:    s_lshl_b32 s6, s6, s8
-; GFX6-NEXT:    s_and_b32 s8, s15, s16
+; GFX6-NEXT:    s_lshl_b32 s5, s5, s13
 ; GFX6-NEXT:    s_or_b32 s0, s0, s1
 ; GFX6-NEXT:    s_and_b32 s1, s2, s16
 ; GFX6-NEXT:    s_and_b32 s2, s3, s16
-; GFX6-NEXT:    s_lshl_b32 s7, s7, s8
+; GFX6-NEXT:    s_lshl_b32 s4, s4, s12
+; GFX6-NEXT:    s_lshl_b32 s7, s7, s15
 ; GFX6-NEXT:    s_lshl_b32 s2, s2, 16
 ; GFX6-NEXT:    s_and_b32 s3, s5, s16
+; GFX6-NEXT:    s_lshl_b32 s6, s6, s14
 ; GFX6-NEXT:    s_or_b32 s1, s1, s2
 ; GFX6-NEXT:    s_and_b32 s2, s4, s16
 ; GFX6-NEXT:    s_lshl_b32 s3, s3, 16
 ; GFX6-NEXT:    s_and_b32 s4, s7, s16
 ; GFX6-NEXT:    s_or_b32 s2, s2, s3
 ; GFX6-NEXT:    s_and_b32 s3, s6, s16
 ; GFX6-NEXT:    s_lshl_b32 s4, s4, 16
 ; GFX6-NEXT:    s_or_b32 s3, s3, s4
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_shl_v8i16:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_mov_b32 s12, 0xffff
 ; GFX8-NEXT:    s_lshr_b32 s8, s0, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s12
 ; GFX8-NEXT:    s_lshr_b32 s13, s4, 16
-; GFX8-NEXT:    s_and_b32 s4, s4, s12
 ; GFX8-NEXT:    s_lshr_b32 s9, s1, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s12
 ; GFX8-NEXT:    s_lshr_b32 s14, s5, 16
-; GFX8-NEXT:    s_and_b32 s5, s5, s12
 ; GFX8-NEXT:    s_lshl_b32 s0, s0, s4
 ; GFX8-NEXT:    s_lshl_b32 s4, s8, s13
 ; GFX8-NEXT:    s_lshr_b32 s10, s2, 16
 ; GFX8-NEXT:    s_and_b32 s2, s2, s12
 ; GFX8-NEXT:    s_lshr_b32 s15, s6, 16
-; GFX8-NEXT:    s_and_b32 s6, s6, s12
 ; GFX8-NEXT:    s_lshl_b32 s1, s1, s5
 ; GFX8-NEXT:    s_lshl_b32 s5, s9, s14
 ; GFX8-NEXT:    s_lshl_b32 s4, s4, 16
 ; GFX8-NEXT:    s_and_b32 s0, s0, s12
 ; GFX8-NEXT:    s_lshr_b32 s11, s3, 16
 ; GFX8-NEXT:    s_and_b32 s3, s3, s12
 ; GFX8-NEXT:    s_lshr_b32 s16, s7, 16
-; GFX8-NEXT:    s_and_b32 s7, s7, s12
 ; GFX8-NEXT:    s_lshl_b32 s2, s2, s6
 ; GFX8-NEXT:    s_lshl_b32 s6, s10, s15
 ; GFX8-NEXT:    s_or_b32 s0, s4, s0
 ; GFX8-NEXT:    s_lshl_b32 s4, s5, 16
 ; GFX8-NEXT:    s_and_b32 s1, s1, s12
 ; GFX8-NEXT:    s_lshl_b32 s3, s3, s7
 ; GFX8-NEXT:    s_lshl_b32 s7, s11, s16
 ; GFX8-NEXT:    s_or_b32 s1, s4, s1

llvm/test/CodeGen/AMDGPU/ashr.v2i16.ll

 ; GCN: {{buffer|flat|global}}_load_dword [[LHS:v[0-9]+]]
 ; GCN: {{buffer|flat|global}}_load_dword [[RHS:v[0-9]+]]
 ; GFX9: v_pk_ashrrev_i16 [[RESULT:v[0-9]+]], [[RHS]], [[LHS]]
 
 ; VI: v_ashrrev_i16_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 ; VI: v_ashrrev_i16_sdwa v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}} dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
 ; VI: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 
-; CI: s_mov_b32 [[MASK:s[0-9]+]], 0xffff{{$}}
 ; CI-DAG: v_bfe_i32 v{{[0-9]+}}, v{{[0-9]+}}, 0, 16
 ; CI: v_ashrrev_i32_e32 v{{[0-9]+}}, 16, [[LHS]]
-; CI-DAG: v_and_b32_e32 v{{[0-9]+}}, [[MASK]], [[RHS]]
 ; CI: v_lshrrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
 ; CI: v_ashr_i32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 ; CI: v_ashr_i32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 ; CI: v_lshlrev_b32_e32 v{{[0-9]+}}, 16, v{{[0-9]+}}
-; CI: v_and_b32_e32 v{{[0-9]+}}, [[MASK]], v{{[0-9]+}}
+; CI: v_and_b32_e32 v{{[0-9]+}}, 0xffff, v{{[0-9]+}}
 ; CI: v_or_b32_e32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
 define amdgpu_kernel void @v_ashr_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> addrspace(1)* %in) #0 {
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %in.gep = getelementptr inbounds <2 x i16>, <2 x i16> addrspace(1)* %in, i64 %tid.ext
   %out.gep = getelementptr inbounds <2 x i16>, <2 x i16> addrspace(1)* %out, i64 %tid.ext
   %b_ptr = getelementptr <2 x i16>, <2 x i16> addrspace(1)* %in.gep, i32 1
   %a = load <2 x i16>, <2 x i16> addrspace(1)* %in.gep

llvm/test/CodeGen/AMDGPU/constrained-shift.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx900 < %s | FileCheck %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -global-isel < %s | FileCheck -check-prefix=GISEL %s
 
 define i16 @csh_16(i16 %a, i16 %b) {
 ; CHECK-LABEL: csh_16:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshlrev_b16_e32 v2, v1, v0
 ; CHECK-NEXT:    v_lshrrev_b16_e32 v3, v1, v0
 ; CHECK-NEXT:    v_ashrrev_i16_e32 v0, v1, v0
 ; CHECK-NEXT:    v_add_u16_e32 v1, v2, v3
 ; CHECK-NEXT:    v_add_u16_e32 v0, v1, v0
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: csh_16:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v1, 15, v1
+; GISEL-NEXT:    v_lshlrev_b16_e32 v2, v1, v0
+; GISEL-NEXT:    v_lshrrev_b16_e32 v3, v1, v0
+; GISEL-NEXT:    v_ashrrev_i16_e32 v0, v1, v0
+; GISEL-NEXT:    v_add_u16_e32 v1, v2, v3
+; GISEL-NEXT:    v_add_u16_e32 v0, v1, v0
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and i16 %b, 15
   %shl = shl i16 %a, %and
   %lshr = lshr i16 %a, %and
   %ashr = ashr i16 %a, %and
   %ret.0 = add i16 %shl, %lshr
   %ret = add i16 %ret.0, %ashr
   ret i16 %ret
 }
 
 define i32 @csh_32(i32 %a, i32 %b) {
 ; CHECK-LABEL: csh_32:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshlrev_b32_e32 v2, v1, v0
 ; CHECK-NEXT:    v_lshrrev_b32_e32 v3, v1, v0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, v1, v0
 ; CHECK-NEXT:    v_add3_u32 v0, v2, v3, v0
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: csh_32:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v1, 31, v1
+; GISEL-NEXT:    v_lshlrev_b32_e32 v2, v1, v0
+; GISEL-NEXT:    v_lshrrev_b32_e32 v3, v1, v0
+; GISEL-NEXT:    v_ashrrev_i32_e32 v0, v1, v0
+; GISEL-NEXT:    v_add3_u32 v0, v2, v3, v0
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and i32 %b, 31
   %shl = shl i32 %a, %and
   %lshr = lshr i32 %a, %and
   %ashr = ashr i32 %a, %and
   %ret.0 = add i32 %shl, %lshr
   %ret = add i32 %ret.0, %ashr
   ret i32 %ret
 }
 
-define amdgpu_ps i32 @s_csh_32(i32 inreg %a, i32 inreg %b) {
-; CHECK-LABEL: s_csh_32:
+define amdgpu_ps i32 @s_csh_32_0(i32 inreg %a, i32 inreg %b) {
+; CHECK-LABEL: s_csh_32_0:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_lshl_b32 s2, s0, s1
 ; CHECK-NEXT:    s_lshr_b32 s3, s0, s1
 ; CHECK-NEXT:    s_ashr_i32 s0, s0, s1
 ; CHECK-NEXT:    s_add_i32 s1, s2, s3
 ; CHECK-NEXT:    s_add_i32 s0, s1, s0
 ; CHECK-NEXT:    ; return to shader part epilog
+;
+; GISEL-LABEL: s_csh_32_0:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_lshl_b32 s2, s0, s1
+; GISEL-NEXT:    s_lshr_b32 s3, s0, s1
+; GISEL-NEXT:    s_ashr_i32 s0, s0, s1
+; GISEL-NEXT:    s_add_i32 s1, s2, s3
+; GISEL-NEXT:    s_add_i32 s0, s1, s0
+; GISEL-NEXT:    ; return to shader part epilog
   %and = and i32 %b, 31
   %shl = shl i32 %a, %and
   %lshr = lshr i32 %a, %and
   %ashr = ashr i32 %a, %and
   %ret.0 = add i32 %shl, %lshr
   %ret = add i32 %ret.0, %ashr
   ret i32 %ret
 }
 
+define amdgpu_ps i32 @s_csh_32_1(i32 inreg %a, i32 inreg %b) {
+; CHECK-LABEL: s_csh_32_1:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_lshl_b32 s2, s0, s1
+; CHECK-NEXT:    s_lshr_b32 s3, s0, s1
+; CHECK-NEXT:    s_ashr_i32 s0, s0, s1
+; CHECK-NEXT:    s_add_i32 s1, s2, s3
+; CHECK-NEXT:    s_add_i32 s0, s1, s0
+; CHECK-NEXT:    ; return to shader part epilog
+;
+; GISEL-LABEL: s_csh_32_1:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_lshl_b32 s2, s0, s1
+; GISEL-NEXT:    s_lshr_b32 s3, s0, s1
+; GISEL-NEXT:    s_ashr_i32 s0, s0, s1
+; GISEL-NEXT:    s_add_i32 s1, s2, s3
+; GISEL-NEXT:    s_add_i32 s0, s1, s0
+; GISEL-NEXT:    ; return to shader part epilog
+  %and = and i32 %b, 127
+  %shl = shl i32 %a, %and
+  %lshr = lshr i32 %a, %and
+  %ashr = ashr i32 %a, %and
+  %ret.0 = add i32 %shl, %lshr
+  %ret = add i32 %ret.0, %ashr
+  ret i32 %ret
+}
+
 define <4 x i32> @csh_v4i32(<4 x i32> %a, <4 x i32> %b) {
 ; CHECK-LABEL: csh_v4i32:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshlrev_b32_e32 v8, v7, v3
 ; CHECK-NEXT:    v_lshlrev_b32_e32 v9, v6, v2
 ; CHECK-NEXT:    v_lshlrev_b32_e32 v10, v5, v1
 ; CHECK-NEXT:    v_lshlrev_b32_e32 v11, v4, v0
 ; CHECK-NEXT:    v_lshrrev_b32_e32 v12, v7, v3
 ; CHECK-NEXT:    v_lshrrev_b32_e32 v13, v6, v2
 ; CHECK-NEXT:    v_lshrrev_b32_e32 v14, v5, v1
 ; CHECK-NEXT:    v_lshrrev_b32_e32 v15, v4, v0
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v3, v7, v3
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v2, v6, v2
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v1, v5, v1
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v0, v4, v0
 ; CHECK-NEXT:    v_add3_u32 v0, v11, v15, v0
 ; CHECK-NEXT:    v_add3_u32 v1, v10, v14, v1
 ; CHECK-NEXT:    v_add3_u32 v2, v9, v13, v2
 ; CHECK-NEXT:    v_add3_u32 v3, v8, v12, v3
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: csh_v4i32:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v4, 31, v4
+; GISEL-NEXT:    v_and_b32_e32 v5, 31, v5
+; GISEL-NEXT:    v_and_b32_e32 v6, 31, v6
+; GISEL-NEXT:    v_and_b32_e32 v7, 31, v7
+; GISEL-NEXT:    v_lshlrev_b32_e32 v8, v4, v0
+; GISEL-NEXT:    v_lshlrev_b32_e32 v9, v5, v1
+; GISEL-NEXT:    v_lshlrev_b32_e32 v10, v6, v2
+; GISEL-NEXT:    v_lshlrev_b32_e32 v11, v7, v3
+; GISEL-NEXT:    v_lshrrev_b32_e32 v12, v4, v0
+; GISEL-NEXT:    v_lshrrev_b32_e32 v13, v5, v1
+; GISEL-NEXT:    v_lshrrev_b32_e32 v14, v6, v2
+; GISEL-NEXT:    v_lshrrev_b32_e32 v15, v7, v3
+; GISEL-NEXT:    v_ashrrev_i32_e32 v0, v4, v0
+; GISEL-NEXT:    v_ashrrev_i32_e32 v1, v5, v1
+; GISEL-NEXT:    v_ashrrev_i32_e32 v2, v6, v2
+; GISEL-NEXT:    v_ashrrev_i32_e32 v3, v7, v3
+; GISEL-NEXT:    v_add3_u32 v0, v8, v12, v0
+; GISEL-NEXT:    v_add3_u32 v1, v9, v13, v1
+; GISEL-NEXT:    v_add3_u32 v2, v10, v14, v2
+; GISEL-NEXT:    v_add3_u32 v3, v11, v15, v3
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and <4 x i32> %b, <i32 31, i32 31, i32 31, i32 31>
   %shl = shl <4 x i32> %a, %and
   %lshr = lshr <4 x i32> %a, %and
   %ashr = ashr <4 x i32> %a, %and
   %ret.0 = add <4 x i32> %shl, %lshr
   %ret = add <4 x i32> %ret.0, %ashr
   ret <4 x i32> %ret
 }

 ; CHECK-NEXT:    s_add_i32 s5, s10, s14
 ; CHECK-NEXT:    s_add_i32 s6, s9, s13
 ; CHECK-NEXT:    s_add_i32 s7, s8, s12
 ; CHECK-NEXT:    s_add_i32 s0, s7, s0
 ; CHECK-NEXT:    s_add_i32 s1, s6, s1
 ; CHECK-NEXT:    s_add_i32 s2, s5, s2
 ; CHECK-NEXT:    s_add_i32 s3, s4, s3
 ; CHECK-NEXT:    ; return to shader part epilog
+;
+; GISEL-LABEL: s_csh_v4i32:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_mov_b32 s8, 31
+; GISEL-NEXT:    s_mov_b32 s9, s8
+; GISEL-NEXT:    s_and_b64 s[4:5], s[4:5], s[8:9]
+; GISEL-NEXT:    s_and_b64 s[6:7], s[6:7], s[8:9]
+; GISEL-NEXT:    s_lshl_b32 s8, s0, s4
+; GISEL-NEXT:    s_lshl_b32 s9, s1, s5
+; GISEL-NEXT:    s_lshl_b32 s10, s2, s6
+; GISEL-NEXT:    s_lshl_b32 s11, s3, s7
+; GISEL-NEXT:    s_lshr_b32 s12, s0, s4
+; GISEL-NEXT:    s_lshr_b32 s13, s1, s5
+; GISEL-NEXT:    s_lshr_b32 s14, s2, s6
+; GISEL-NEXT:    s_lshr_b32 s15, s3, s7
+; GISEL-NEXT:    s_ashr_i32 s0, s0, s4
+; GISEL-NEXT:    s_ashr_i32 s1, s1, s5
+; GISEL-NEXT:    s_ashr_i32 s2, s2, s6
+; GISEL-NEXT:    s_ashr_i32 s3, s3, s7
+; GISEL-NEXT:    s_add_i32 s4, s8, s12
+; GISEL-NEXT:    s_add_i32 s5, s9, s13
+; GISEL-NEXT:    s_add_i32 s6, s10, s14
+; GISEL-NEXT:    s_add_i32 s7, s11, s15
+; GISEL-NEXT:    s_add_i32 s0, s4, s0
+; GISEL-NEXT:    s_add_i32 s1, s5, s1
+; GISEL-NEXT:    s_add_i32 s2, s6, s2
+; GISEL-NEXT:    s_add_i32 s3, s7, s3
+; GISEL-NEXT:    ; return to shader part epilog
   %and = and <4 x i32> %b, <i32 31, i32 31, i32 31, i32 31>
   %shl = shl <4 x i32> %a, %and
   %lshr = lshr <4 x i32> %a, %and
   %ashr = ashr <4 x i32> %a, %and
   %ret.0 = add <4 x i32> %shl, %lshr
   %ret = add <4 x i32> %ret.0, %ashr
   ret <4 x i32> %ret
 }
 
 define i64 @csh_64(i64 %a, i64 %b) {
 ; CHECK-LABEL: csh_64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshlrev_b64 v[3:4], v2, v[0:1]
 ; CHECK-NEXT:    v_lshrrev_b64 v[5:6], v2, v[0:1]
 ; CHECK-NEXT:    v_ashrrev_i64 v[0:1], v2, v[0:1]
 ; CHECK-NEXT:    v_add_co_u32_e32 v2, vcc, v3, v5
 ; CHECK-NEXT:    v_addc_co_u32_e32 v3, vcc, v4, v6, vcc
 ; CHECK-NEXT:    v_add_co_u32_e32 v0, vcc, v2, v0
 ; CHECK-NEXT:    v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: csh_64:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v6, 63, v2
+; GISEL-NEXT:    v_lshlrev_b64 v[2:3], v6, v[0:1]
+; GISEL-NEXT:    v_lshrrev_b64 v[4:5], v6, v[0:1]
+; GISEL-NEXT:    v_ashrrev_i64 v[0:1], v6, v[0:1]
+; GISEL-NEXT:    v_add_co_u32_e32 v2, vcc, v2, v4
+; GISEL-NEXT:    v_addc_co_u32_e32 v3, vcc, v3, v5, vcc
+; GISEL-NEXT:    v_add_co_u32_e32 v0, vcc, v2, v0
+; GISEL-NEXT:    v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and i64 %b, 63
   %shl = shl i64 %a, %and
   %lshr = lshr i64 %a, %and
   %ashr = ashr i64 %a, %and
   %ret.0 = add i64 %shl, %lshr
   %ret = add i64 %ret.0, %ashr
   ret i64 %ret
 }
 
-define amdgpu_ps i64 @s_csh_64(i64 inreg %a, i64 inreg %b) {
-; CHECK-LABEL: s_csh_64:
+define amdgpu_ps i64 @s_csh_64_0(i64 inreg %a, i64 inreg %b) {
+; CHECK-LABEL: s_csh_64_0:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_lshl_b64 s[4:5], s[0:1], s2
 ; CHECK-NEXT:    s_lshr_b64 s[6:7], s[0:1], s2
 ; CHECK-NEXT:    s_ashr_i64 s[0:1], s[0:1], s2
 ; CHECK-NEXT:    s_add_u32 s2, s4, s6
 ; CHECK-NEXT:    s_addc_u32 s3, s5, s7
 ; CHECK-NEXT:    s_add_u32 s0, s2, s0
 ; CHECK-NEXT:    s_addc_u32 s1, s3, s1
 ; CHECK-NEXT:    ; return to shader part epilog
+;
+; GISEL-LABEL: s_csh_64_0:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_and_b64 s[2:3], s[2:3], 63
+; GISEL-NEXT:    s_lshl_b64 s[4:5], s[0:1], s2
+; GISEL-NEXT:    s_lshr_b64 s[6:7], s[0:1], s2
+; GISEL-NEXT:    s_ashr_i64 s[0:1], s[0:1], s2
+; GISEL-NEXT:    s_add_u32 s2, s4, s6
+; GISEL-NEXT:    s_cselect_b32 s3, 1, 0
+; GISEL-NEXT:    s_and_b32 s3, s3, 1
+; GISEL-NEXT:    s_cmp_lg_u32 s3, 0
+; GISEL-NEXT:    s_addc_u32 s3, s5, s7
+; GISEL-NEXT:    s_add_u32 s0, s2, s0
+; GISEL-NEXT:    s_cselect_b32 s2, 1, 0
+; GISEL-NEXT:    s_and_b32 s2, s2, 1
+; GISEL-NEXT:    s_cmp_lg_u32 s2, 0
+; GISEL-NEXT:    s_addc_u32 s1, s3, s1
+; GISEL-NEXT:    ; return to shader part epilog
   %and = and i64 %b, 63
   %shl = shl i64 %a, %and
   %lshr = lshr i64 %a, %and
   %ashr = ashr i64 %a, %and
   %ret.0 = add i64 %shl, %lshr
   %ret = add i64 %ret.0, %ashr
   ret i64 %ret
 }
 
+define amdgpu_ps i64 @s_csh_64_1(i64 inreg %a, i64 inreg %b) {
+; CHECK-LABEL: s_csh_64_1:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    s_lshl_b64 s[4:5], s[0:1], s2
+; CHECK-NEXT:    s_lshr_b64 s[6:7], s[0:1], s2
+; CHECK-NEXT:    s_ashr_i64 s[0:1], s[0:1], s2
+; CHECK-NEXT:    s_add_u32 s2, s4, s6
+; CHECK-NEXT:    s_addc_u32 s3, s5, s7
+; CHECK-NEXT:    s_add_u32 s0, s2, s0
+; CHECK-NEXT:    s_addc_u32 s1, s3, s1
+; CHECK-NEXT:    ; return to shader part epilog
+;
+; GISEL-LABEL: s_csh_64_1:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_mov_b64 s[4:5], 0xff
+; GISEL-NEXT:    s_and_b64 s[2:3], s[2:3], s[4:5]
+; GISEL-NEXT:    s_lshl_b64 s[4:5], s[0:1], s2
+; GISEL-NEXT:    s_lshr_b64 s[6:7], s[0:1], s2
+; GISEL-NEXT:    s_ashr_i64 s[0:1], s[0:1], s2
+; GISEL-NEXT:    s_add_u32 s2, s4, s6
+; GISEL-NEXT:    s_cselect_b32 s3, 1, 0
+; GISEL-NEXT:    s_and_b32 s3, s3, 1
+; GISEL-NEXT:    s_cmp_lg_u32 s3, 0
+; GISEL-NEXT:    s_addc_u32 s3, s5, s7
+; GISEL-NEXT:    s_add_u32 s0, s2, s0
+; GISEL-NEXT:    s_cselect_b32 s2, 1, 0
+; GISEL-NEXT:    s_and_b32 s2, s2, 1
+; GISEL-NEXT:    s_cmp_lg_u32 s2, 0
+; GISEL-NEXT:    s_addc_u32 s1, s3, s1
+; GISEL-NEXT:    ; return to shader part epilog
+  %and = and i64 %b, 255
+  %shl = shl i64 %a, %and
+  %lshr = lshr i64 %a, %and
+  %ashr = ashr i64 %a, %and
+  %ret.0 = add i64 %shl, %lshr
+  %ret = add i64 %ret.0, %ashr
+  ret i64 %ret
+}
+
 define i32 @cshl_or(i32 %a, i32 %b) {
 ; CHECK-LABEL: cshl_or:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshl_or_b32 v0, v0, v1, v0
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: cshl_or:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v1, 31, v1
+; GISEL-NEXT:    v_lshl_or_b32 v0, v0, v1, v0
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and i32 %b, 31
   %shl = shl i32 %a, %and
   %or = or i32 %shl, %a
   ret i32 %or
 }
 
 define i32 @cshl_add(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: cshl_add:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_lshl_add_u32 v0, v0, v1, v2
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: cshl_add:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v1, 31, v1
+; GISEL-NEXT:    v_lshl_add_u32 v0, v0, v1, v2
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %and = and i32 %b, 31
   %shl = shl i32 %a, %and
   %add = add i32 %shl, %c
   ret i32 %add
 }
 
 define i32 @add_cshl(i32 %a, i32 %b) {
 ; CHECK-LABEL: add_cshl:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_add_lshl_u32 v0, v0, v1, v1
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: add_cshl:
+; GISEL:       ; %bb.0:
+; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT:    v_and_b32_e32 v2, 31, v1
+; GISEL-NEXT:    v_add_lshl_u32 v0, v0, v1, v2
+; GISEL-NEXT:    s_setpc_b64 s[30:31]
   %add = add i32 %a, %b
   %and = and i32 %b, 31
   %shl = shl i32 %add, %and
   ret i32 %shl
 }

llvm/test/CodeGen/AMDGPU/extract-lowbits.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %masked
 }
 
 define i32 @bzhi32_c1_indexzext(i32 %val, i8 %numlowbits) nounwind {
 ; SI-LABEL: bzhi32_c1_indexzext:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_sub_i32_e32 v1, vcc, 32, v1
-; SI-NEXT:    v_and_b32_e32 v1, 0xff, v1
 ; SI-NEXT:    v_lshr_b32_e32 v1, -1, v1
 ; SI-NEXT:    v_and_b32_e32 v0, v1, v0
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: bzhi32_c1_indexzext:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_sub_u16_e32 v1, 32, v1
-; VI-NEXT:    v_mov_b32_e32 v2, -1
-; VI-NEXT:    v_lshrrev_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; VI-NEXT:    v_lshrrev_b32_e64 v1, v1, -1
 ; VI-NEXT:    v_and_b32_e32 v0, v1, v0
 ; VI-NEXT:    s_setpc_b64 s[30:31]
   %numhighbits = sub i8 32, %numlowbits
   %sh_prom = zext i8 %numhighbits to i32
   %mask = lshr i32 -1, %sh_prom
   %masked = and i32 %mask, %val
   ret i32 %masked
 }
; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %masked
 }
 
 define i32 @bzhi32_d1_indexzext(i32 %val, i8 %numlowbits) nounwind {
 ; SI-LABEL: bzhi32_d1_indexzext:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    v_sub_i32_e32 v1, vcc, 32, v1
-; SI-NEXT:    v_and_b32_e32 v1, 0xff, v1
 ; SI-NEXT:    v_lshl_b32_e32 v0, v0, v1
 ; SI-NEXT:    v_lshr_b32_e32 v0, v0, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; VI-LABEL: bzhi32_d1_indexzext:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; VI-NEXT:    v_sub_u16_e32 v1, 32, v1
-; VI-NEXT:    v_and_b32_e32 v1, 0xff, v1
 ; VI-NEXT:    v_lshlrev_b32_e32 v0, v1, v0
 ; VI-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
 ; VI-NEXT:    s_setpc_b64 s[30:31]
   %numhighbits = sub i8 32, %numlowbits
   %sh_prom = zext i8 %numhighbits to i32
   %highbitscleared = shl i32 %val, %sh_prom
   %masked = lshr i32 %highbitscleared, %sh_prom
   ret i32 %masked
 }

llvm/test/CodeGen/AMDGPU/imm16.ll

 ; VI-NEXT:    flat_store_short v[0:1], v2 ; encoding: [0x00,0x00,0x68,0xdc,0x00,0x02,0x00,0x00]
 ; VI-NEXT:    s_waitcnt vmcnt(0) ; encoding: [0x70,0x0f,0x8c,0xbf]
 ; VI-NEXT:    s_setpc_b64 s[30:31] ; encoding: [0x1e,0x1d,0x80,0xbe]
 ;
 ; SI-LABEL: shl_inline_imm_2.0_i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s6, 0
-; SI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
 ; SI-NEXT:    v_lshl_b32_e32 v2, 0x4000, v2
 ; SI-NEXT:    buffer_store_short v2, v[0:1], s[4:7], 0 addr64
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0)
 ; SI-NEXT:    s_setpc_b64 s[30:31]
   %y = shl i16 bitcast (half 2.0 to i16), %x

 ; VI-NEXT:    flat_store_short v[0:1], v2 ; encoding: [0x00,0x00,0x68,0xdc,0x00,0x02,0x00,0x00]
 ; VI-NEXT:    s_waitcnt vmcnt(0) ; encoding: [0x70,0x0f,0x8c,0xbf]
 ; VI-NEXT:    s_setpc_b64 s[30:31] ; encoding: [0x1e,0x1d,0x80,0xbe]
 ;
 ; SI-LABEL: shl_inline_imm_neg_2.0_i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s6, 0
-; SI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s6
 ; SI-NEXT:    v_lshl_b32_e32 v2, 0xffffc000, v2
 ; SI-NEXT:    buffer_store_short v2, v[0:1], s[4:7], 0 addr64
 ; SI-NEXT:    s_waitcnt vmcnt(0) expcnt(0)
 ; SI-NEXT:    s_setpc_b64 s[30:31]
   %y = shl i16 bitcast (half -2.0 to i16), %x

llvm/test/CodeGen/AMDGPU/lshr.v2i16.ll

 ; GFX9-NEXT:    v_mov_b32_e32 v1, s4
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v1, s5, v1
 ; GFX9-NEXT:    global_store_dword v0, v1, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_lshr_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
-; VI-NEXT:    s_load_dword s5, s[0:1], 0x2c
+; VI-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x30
-; VI-NEXT:    s_mov_b32 s4, 0xffff
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    s_lshr_b32 s1, s5, 16
-; VI-NEXT:    s_lshr_b32 s6, s0, 16
-; VI-NEXT:    s_and_b32 s5, s5, s4
-; VI-NEXT:    s_and_b32 s0, s0, s4
-; VI-NEXT:    s_lshr_b32 s1, s1, s6
-; VI-NEXT:    s_lshr_b32 s0, s5, s0
-; VI-NEXT:    s_lshl_b32 s1, s1, 16
+; VI-NEXT:    s_and_b32 s1, s4, 0xffff
+; VI-NEXT:    s_lshr_b32 s4, s4, 16
+; VI-NEXT:    s_lshr_b32 s5, s0, 16
+; VI-NEXT:    s_lshr_b32 s4, s4, s5
+; VI-NEXT:    s_lshr_b32 s0, s1, s0
+; VI-NEXT:    s_lshl_b32 s1, s4, 16
 ; VI-NEXT:    s_or_b32 s0, s0, s1
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_mov_b32_e32 v2, s0
 ; VI-NEXT:    flat_store_dword v[0:1], v2
 ; VI-NEXT:    s_endpgm
 ;
 ; CI-LABEL: s_lshr_v2i16:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; CI-NEXT:    s_load_dword s2, s[0:1], 0xb
 ; CI-NEXT:    s_load_dword s0, s[0:1], 0xc
-; CI-NEXT:    s_mov_b32 s3, 0xffff
 ; CI-NEXT:    s_mov_b32 s7, 0xf000
 ; CI-NEXT:    s_mov_b32 s6, -1
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    s_lshr_b32 s1, s2, 16
-; CI-NEXT:    s_lshr_b32 s8, s0, 16
-; CI-NEXT:    s_and_b32 s2, s2, s3
-; CI-NEXT:    s_and_b32 s0, s0, s3
-; CI-NEXT:    s_lshr_b32 s1, s1, s8
-; CI-NEXT:    s_lshr_b32 s0, s2, s0
-; CI-NEXT:    s_lshl_b32 s1, s1, 16
-; CI-NEXT:    s_or_b32 s0, s0, s1
+; CI-NEXT:    s_and_b32 s1, s2, 0xffff
+; CI-NEXT:    s_lshr_b32 s2, s2, 16
+; CI-NEXT:    s_lshr_b32 s3, s0, 16
+; CI-NEXT:    s_lshr_b32 s2, s2, s3
+; CI-NEXT:    s_lshl_b32 s2, s2, 16
+; CI-NEXT:    s_lshr_b32 s0, s1, s0
+; CI-NEXT:    s_or_b32 s0, s0, s2
 ; CI-NEXT:    v_mov_b32_e32 v0, s0
 ; CI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: s_lshr_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x2
 ; GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c

 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    buffer_load_dword v3, v[0:1], s[0:3], 0 addr64 offset:4
-; CI-NEXT:    s_mov_b32 s0, 0xffff
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(1)
 ; CI-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
-; CI-NEXT:    v_and_b32_e32 v2, s0, v2
+; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
-; CI-NEXT:    v_and_b32_e32 v3, s0, v3
 ; CI-NEXT:    v_lshr_b32_e32 v2, v2, v3
 ; CI-NEXT:    v_lshr_b32_e32 v3, v4, v5
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: v_lshr_v2i16:

 ; CI-NEXT:    s_load_dword s8, s[0:1], 0xd
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
-; CI-NEXT:    s_mov_b32 s0, 0xffff
-; CI-NEXT:    s_lshr_b32 s1, s8, 16
-; CI-NEXT:    s_and_b32 s8, s8, s0
+; CI-NEXT:    s_lshr_b32 s0, s8, 16
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; CI-NEXT:    v_and_b32_e32 v2, s0, v2
-; CI-NEXT:    v_lshrrev_b32_e32 v3, s1, v3
+; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; CI-NEXT:    v_lshrrev_b32_e32 v3, s0, v3
 ; CI-NEXT:    v_lshrrev_b32_e32 v2, s8, v2
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: lshr_v_s_v2i16:
 ; GFX10:       ; %bb.0:

 ; CI-NEXT:    s_load_dword s8, s[0:1], 0xd
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
-; CI-NEXT:    s_mov_b32 s0, 0xffff
-; CI-NEXT:    s_lshr_b32 s1, s8, 16
-; CI-NEXT:    s_and_b32 s8, s8, s0
+; CI-NEXT:    s_lshr_b32 s0, s8, 16
+; CI-NEXT:    s_and_b32 s1, s8, 0xffff
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; CI-NEXT:    v_and_b32_e32 v2, s0, v2
-; CI-NEXT:    v_lshr_b32_e32 v3, s1, v3
-; CI-NEXT:    v_lshr_b32_e32 v2, s8, v2
+; CI-NEXT:    v_lshr_b32_e32 v3, s0, v3
+; CI-NEXT:    v_lshr_b32_e32 v2, s1, v2
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: lshr_s_v_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24

 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; CI-NEXT:    v_lshr_b32_e32 v3, 8, v3
 ; CI-NEXT:    v_lshr_b32_e32 v2, 8, v2
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: lshr_imm_v_v2i16:

 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(1)
 ; CI-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; CI-NEXT:    v_and_b32_e32 v2, s0, v2
 ; CI-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
 ; CI-NEXT:    v_and_b32_e32 v3, s0, v3
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v8, 16, v4
-; CI-NEXT:    v_and_b32_e32 v4, s0, v4
 ; CI-NEXT:    v_lshrrev_b32_e32 v9, 16, v5
-; CI-NEXT:    v_and_b32_e32 v5, s0, v5
 ; CI-NEXT:    v_lshr_b32_e32 v3, v3, v5
 ; CI-NEXT:    v_lshr_b32_e32 v5, v7, v9
 ; CI-NEXT:    v_lshr_b32_e32 v2, v2, v4
 ; CI-NEXT:    v_lshr_b32_e32 v4, v6, v8
 ; CI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
 ; CI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
 ; CI-NEXT:    v_or_b32_e32 v3, v3, v5
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v4

llvm/test/CodeGen/AMDGPU/shl.ll

 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s7
 ; SI-NEXT:    s_mov_b32 s6, s2
 ; SI-NEXT:    s_mov_b32 s7, s3
 ; SI-NEXT:    buffer_load_ushort v0, off, s[4:7], 0
-; SI-NEXT:    s_and_b32 s8, s8, 0xffff
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, s8, v0
 ; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_i16_v_s:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s8, s[0:1], 0x34
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_mov_b32 s4, s6
 ; VI-NEXT:    s_mov_b32 s5, s7
 ; VI-NEXT:    s_mov_b32 s6, s2
 ; VI-NEXT:    s_mov_b32 s7, s3
 ; VI-NEXT:    buffer_load_ushort v0, off, s[4:7], 0
-; VI-NEXT:    s_and_b32 s4, s8, 0xffff
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshlrev_b32_e32 v0, s4, v0
+; VI-NEXT:    v_lshlrev_b32_e32 v0, s8, v0
 ; VI-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; EG-LABEL: shl_i16_v_s:
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 1, @10, KC0[CB0:0-32], KC1[]
 ; EG-NEXT:    TEX 1 @6
 ; EG-NEXT:    ALU 12, @12, KC0[CB0:0-32], KC1[]

 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_mov_b32 s4, s6
 ; SI-NEXT:    s_mov_b32 s5, s7
 ; SI-NEXT:    s_mov_b32 s6, s2
 ; SI-NEXT:    s_mov_b32 s7, s3
 ; SI-NEXT:    buffer_load_ushort v0, off, s[4:7], 0
 ; SI-NEXT:    s_add_i32 s8, s8, 3
-; SI-NEXT:    s_and_b32 s4, s8, 0xffff
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v0, s4, v0
+; SI-NEXT:    v_lshlrev_b32_e32 v0, s8, v0
 ; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_i16_v_compute_s:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s8, s[0:1], 0x34
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_mov_b32 s4, s6
 ; VI-NEXT:    s_mov_b32 s5, s7
 ; VI-NEXT:    s_mov_b32 s6, s2
 ; VI-NEXT:    s_mov_b32 s7, s3
 ; VI-NEXT:    buffer_load_ushort v0, off, s[4:7], 0
 ; VI-NEXT:    s_add_i32 s8, s8, 3
-; VI-NEXT:    s_and_b32 s4, s8, 0xffff
 ; VI-NEXT:    s_waitcnt vmcnt(0)
-; VI-NEXT:    v_lshlrev_b32_e32 v0, s4, v0
+; VI-NEXT:    v_lshlrev_b32_e32 v0, s8, v0
 ; VI-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; EG-LABEL: shl_i16_v_compute_s:
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @12, KC0[], KC1[]
 ; EG-NEXT:    TEX 0 @8
 ; EG-NEXT:    ALU 0, @13, KC0[CB0:0-32], KC1[]

 ; SI-NEXT:    s_mov_b64 s[12:13], s[6:7]
 ; SI-NEXT:    buffer_load_ushort v2, off, s[8:11], 0 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    buffer_load_ushort v0, v[0:1], s[12:15], 0 addr64 offset:2 glc
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    v_add_i32_e32 v0, vcc, 3, v0
-; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; SI-NEXT:    v_lshl_b32_e32 v0, v2, v0
 ; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_i16_computed_amount:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    v_lshlrev_b32_e32 v0, 1, v0

 ; SI-NEXT:    s_mov_b32 s9, s7
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    s_mov_b32 s14, 0
 ; SI-NEXT:    s_mov_b32 s15, s3
 ; SI-NEXT:    s_mov_b64 s[12:13], s[6:7]
 ; SI-NEXT:    buffer_load_dword v2, off, s[8:11], 0
 ; SI-NEXT:    buffer_load_dword v0, v[0:1], s[12:15], 0 addr64 offset:4
-; SI-NEXT:    s_mov_b32 s6, 0xffff
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(1)
 ; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v2
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
-; SI-NEXT:    v_and_b32_e32 v0, s6, v0
 ; SI-NEXT:    v_lshl_b32_e32 v0, v2, v0
 ; SI-NEXT:    v_lshl_b32_e32 v1, v1, v3
-; SI-NEXT:    v_and_b32_e32 v0, s6, v0
+; SI-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24

 ; SI-NEXT:    buffer_load_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
 ; SI-NEXT:    buffer_load_dwordx2 v[4:5], v[0:1], s[0:3], 0 addr64 offset:8
 ; SI-NEXT:    s_mov_b32 s0, 0xffff
 ; SI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; SI-NEXT:    s_waitcnt vmcnt(1)
 ; SI-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; SI-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_and_b32_e32 v8, s0, v4
-; SI-NEXT:    v_lshrrev_b32_e32 v4, 16, v4
-; SI-NEXT:    v_and_b32_e32 v9, s0, v5
-; SI-NEXT:    v_lshrrev_b32_e32 v5, 16, v5
-; SI-NEXT:    v_lshl_b32_e32 v5, v7, v5
-; SI-NEXT:    v_lshl_b32_e32 v3, v3, v9
-; SI-NEXT:    v_lshl_b32_e32 v4, v6, v4
-; SI-NEXT:    v_lshl_b32_e32 v2, v2, v8
-; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; SI-NEXT:    v_lshrrev_b32_e32 v8, 16, v4
+; SI-NEXT:    v_lshrrev_b32_e32 v9, 16, v5
+; SI-NEXT:    v_lshl_b32_e32 v3, v3, v5
+; SI-NEXT:    v_lshl_b32_e32 v2, v2, v4
+; SI-NEXT:    v_lshl_b32_e32 v4, v7, v9
+; SI-NEXT:    v_lshl_b32_e32 v5, v6, v8
 ; SI-NEXT:    v_and_b32_e32 v3, s0, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
 ; SI-NEXT:    v_and_b32_e32 v2, s0, v2
-; SI-NEXT:    v_or_b32_e32 v3, v3, v5
-; SI-NEXT:    v_or_b32_e32 v2, v2, v4
+; SI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; SI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; SI-NEXT:    v_or_b32_e32 v3, v3, v4
+; SI-NEXT:    v_or_b32_e32 v2, v2, v5
 ; SI-NEXT:    buffer_store_dwordx2 v[2:3], v[0:1], s[4:7], 0 addr64
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_v4i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/shl.v2i16.ll

 ; GFX9-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_shl_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x30
-; VI-NEXT:    s_mov_b32 s3, 0xffff
+; VI-NEXT:    s_mov_b32 s1, 0xffff
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_lshr_b32 s1, s2, 16
-; VI-NEXT:    s_and_b32 s2, s2, s3
+; VI-NEXT:    s_and_b32 s3, s2, s1
+; VI-NEXT:    s_lshr_b32 s2, s2, 16
 ; VI-NEXT:    s_lshr_b32 s8, s0, 16
-; VI-NEXT:    s_and_b32 s0, s0, s3
-; VI-NEXT:    s_lshl_b32 s0, s2, s0
-; VI-NEXT:    s_lshl_b32 s1, s1, s8
-; VI-NEXT:    s_lshl_b32 s1, s1, 16
-; VI-NEXT:    s_and_b32 s0, s0, s3
-; VI-NEXT:    s_or_b32 s0, s0, s1
+; VI-NEXT:    s_lshl_b32 s2, s2, s8
+; VI-NEXT:    s_lshl_b32 s0, s3, s0
+; VI-NEXT:    s_lshl_b32 s2, s2, 16
+; VI-NEXT:    s_and_b32 s0, s0, s1
+; VI-NEXT:    s_or_b32 s0, s0, s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; CI-LABEL: s_shl_v2i16:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; CI-NEXT:    s_load_dword s2, s[0:1], 0xb
 ; CI-NEXT:    s_load_dword s0, s[0:1], 0xc
-; CI-NEXT:    s_mov_b32 s3, 0xffff
 ; CI-NEXT:    s_mov_b32 s7, 0xf000
 ; CI-NEXT:    s_mov_b32 s6, -1
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_lshr_b32 s1, s2, 16
-; CI-NEXT:    s_and_b32 s8, s0, s3
-; CI-NEXT:    s_lshr_b32 s0, s0, 16
-; CI-NEXT:    s_lshl_b32 s0, s1, s0
-; CI-NEXT:    s_lshl_b32 s1, s2, s8
-; CI-NEXT:    s_lshl_b32 s0, s0, 16
-; CI-NEXT:    s_and_b32 s1, s1, s3
-; CI-NEXT:    s_or_b32 s0, s1, s0
+; CI-NEXT:    s_lshr_b32 s3, s0, 16
+; CI-NEXT:    s_lshl_b32 s1, s1, s3
+; CI-NEXT:    s_lshl_b32 s0, s2, s0
+; CI-NEXT:    s_lshl_b32 s1, s1, 16
+; CI-NEXT:    s_and_b32 s0, s0, 0xffff
+; CI-NEXT:    s_or_b32 s0, s0, s1
 ; CI-NEXT:    v_mov_b32_e32 v0, s0
 ; CI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: s_shl_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_clause 0x2
 ; GFX10-NEXT:    s_load_dword s2, s[0:1], 0x2c

 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    buffer_load_dword v3, v[0:1], s[0:3], 0 addr64 offset:4
-; CI-NEXT:    s_mov_b32 s0, 0xffff
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(1)
 ; CI-NEXT:    v_lshrrev_b32_e32 v4, 16, v2
 ; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_and_b32_e32 v5, s0, v3
-; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
-; CI-NEXT:    v_lshl_b32_e32 v3, v4, v3
-; CI-NEXT:    v_lshl_b32_e32 v2, v2, v5
+; CI-NEXT:    v_lshrrev_b32_e32 v5, 16, v3
+; CI-NEXT:    v_lshl_b32_e32 v2, v2, v3
+; CI-NEXT:    v_lshl_b32_e32 v3, v4, v5
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; CI-NEXT:    v_and_b32_e32 v2, s0, v2
+; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: v_shl_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0

 ; CI-NEXT:    s_load_dword s8, s[0:1], 0xd
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
-; CI-NEXT:    s_mov_b32 s0, 0xffff
-; CI-NEXT:    s_lshr_b32 s1, s8, 16
-; CI-NEXT:    s_and_b32 s8, s8, s0
+; CI-NEXT:    s_lshr_b32 s0, s8, 16
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
 ; CI-NEXT:    v_lshlrev_b32_e32 v2, s8, v2
-; CI-NEXT:    v_lshlrev_b32_e32 v3, s1, v3
-; CI-NEXT:    v_and_b32_e32 v2, s0, v2
+; CI-NEXT:    v_lshlrev_b32_e32 v3, s0, v3
+; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: shl_v_s_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24

 ; CI-NEXT:    s_load_dword s8, s[0:1], 0xd
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
-; CI-NEXT:    s_mov_b32 s0, 0xffff
-; CI-NEXT:    s_lshr_b32 s1, s8, 16
+; CI-NEXT:    s_lshr_b32 s0, s8, 16
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_and_b32_e32 v3, s0, v2
-; CI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
-; CI-NEXT:    v_lshl_b32_e32 v2, s1, v2
-; CI-NEXT:    v_lshl_b32_e32 v3, s8, v3
-; CI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; CI-NEXT:    v_and_b32_e32 v3, s0, v3
-; CI-NEXT:    v_or_b32_e32 v2, v3, v2
+; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
+; CI-NEXT:    v_lshl_b32_e32 v2, s8, v2
+; CI-NEXT:    v_lshl_b32_e32 v3, s0, v3
+; CI-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: shl_s_v_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    s_load_dword s0, s[0:1], 0x34

 ; CI-NEXT:    s_mov_b32 s2, 0
 ; CI-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; CI-NEXT:    v_mov_b32_e32 v1, 0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; CI-NEXT:    buffer_load_dword v2, v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_and_b32_e32 v3, 0xffff, v2
-; CI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; CI-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
 ; CI-NEXT:    v_lshl_b32_e32 v2, 8, v2
 ; CI-NEXT:    v_lshl_b32_e32 v3, 8, v3
-; CI-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; CI-NEXT:    v_and_b32_e32 v3, 0xfff8, v3
-; CI-NEXT:    v_or_b32_e32 v2, v3, v2
+; CI-NEXT:    v_and_b32_e32 v2, 0xfff8, v2
+; CI-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; CI-NEXT:    v_or_b32_e32 v2, v2, v3
 ; CI-NEXT:    buffer_store_dword v2, v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: shl_imm_v_v2i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)

 ; CI-NEXT:    buffer_load_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
 ; CI-NEXT:    buffer_load_dwordx2 v[4:5], v[0:1], s[0:3], 0 addr64 offset:8
 ; CI-NEXT:    s_mov_b32 s0, 0xffff
 ; CI-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; CI-NEXT:    s_waitcnt vmcnt(1)
 ; CI-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; CI-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
 ; CI-NEXT:    s_waitcnt vmcnt(0)
-; CI-NEXT:    v_and_b32_e32 v8, s0, v4
-; CI-NEXT:    v_lshrrev_b32_e32 v4, 16, v4
-; CI-NEXT:    v_and_b32_e32 v9, s0, v5
-; CI-NEXT:    v_lshrrev_b32_e32 v5, 16, v5
-; CI-NEXT:    v_lshl_b32_e32 v5, v7, v5
-; CI-NEXT:    v_lshl_b32_e32 v3, v3, v9
-; CI-NEXT:    v_lshl_b32_e32 v4, v6, v4
-; CI-NEXT:    v_lshl_b32_e32 v2, v2, v8
-; CI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; CI-NEXT:    v_lshrrev_b32_e32 v8, 16, v4
+; CI-NEXT:    v_lshrrev_b32_e32 v9, 16, v5
+; CI-NEXT:    v_lshl_b32_e32 v3, v3, v5
+; CI-NEXT:    v_lshl_b32_e32 v2, v2, v4
+; CI-NEXT:    v_lshl_b32_e32 v4, v7, v9
+; CI-NEXT:    v_lshl_b32_e32 v5, v6, v8
 ; CI-NEXT:    v_and_b32_e32 v3, s0, v3
-; CI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
 ; CI-NEXT:    v_and_b32_e32 v2, s0, v2
-; CI-NEXT:    v_or_b32_e32 v3, v3, v5
-; CI-NEXT:    v_or_b32_e32 v2, v2, v4
+; CI-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; CI-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; CI-NEXT:    v_or_b32_e32 v3, v3, v4
+; CI-NEXT:    v_or_b32_e32 v2, v2, v5
 ; CI-NEXT:    buffer_store_dwordx2 v[2:3], v[0:1], s[4:7], 0 addr64
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: v_shl_v4i16:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/sra.ll

 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s10, s2
 ; SI-NEXT:    s_mov_b32 s11, s3
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s6
 ; SI-NEXT:    s_mov_b32 s9, s7
 ; SI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
-; SI-NEXT:    s_mov_b32 s6, 0xffff
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_bfe_i32 v2, v0, 0, 16
 ; SI-NEXT:    v_ashrrev_i32_e32 v0, 16, v0
-; SI-NEXT:    v_and_b32_e32 v3, s6, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
-; SI-NEXT:    v_ashrrev_i32_e32 v0, v1, v0
-; SI-NEXT:    v_ashrrev_i32_e32 v1, v3, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
+; SI-NEXT:    v_ashrrev_i32_e32 v0, v3, v0
+; SI-NEXT:    v_ashrrev_i32_e32 v1, v1, v2
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; SI-NEXT:    v_and_b32_e32 v1, s6, v1
+; SI-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; SI-NEXT:    v_or_b32_e32 v0, v1, v0
 ; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: ashr_v2i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s3, 0xf000

 ; SI-NEXT:    s_mov_b32 s6, 0xffff
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_bfe_i32 v4, v0, 0, 16
 ; SI-NEXT:    v_ashrrev_i32_e32 v0, 16, v0
 ; SI-NEXT:    v_bfe_i32 v5, v1, 0, 16
 ; SI-NEXT:    v_ashrrev_i32_e32 v1, 16, v1
-; SI-NEXT:    v_and_b32_e32 v6, s6, v2
-; SI-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
-; SI-NEXT:    v_and_b32_e32 v7, s6, v3
-; SI-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
-; SI-NEXT:    v_ashrrev_i32_e32 v1, v3, v1
-; SI-NEXT:    v_ashrrev_i32_e32 v3, v7, v5
-; SI-NEXT:    v_ashrrev_i32_e32 v0, v2, v0
-; SI-NEXT:    v_ashrrev_i32_e32 v2, v6, v4
+; SI-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
+; SI-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
+; SI-NEXT:    v_ashrrev_i32_e32 v1, v7, v1
+; SI-NEXT:    v_ashrrev_i32_e32 v3, v3, v5
+; SI-NEXT:    v_ashrrev_i32_e32 v0, v6, v0
+; SI-NEXT:    v_ashrrev_i32_e32 v2, v2, v4
 ; SI-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; SI-NEXT:    v_and_b32_e32 v3, s6, v3
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
 ; SI-NEXT:    v_and_b32_e32 v2, s6, v2
 ; SI-NEXT:    v_or_b32_e32 v1, v3, v1
 ; SI-NEXT:    v_or_b32_e32 v0, v2, v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm