[AMDGPU][GlobalISel] Add Shift/Shufflevector Combine

[Pierre-vh]

Summary

Folds G_LSHR + G_SHUFFLEVECTOR away when the shift amount
is 1/2 of the destination scalar size, and the mask starts with 1.

Depends on D134870.

[arsenm]

Another thing worth thinking about is whether we should really be using legal vector_shuffle at all. The DAG path doesn't use it. Given that we have such a restricted set of shuffles (only 0, 1), it may still make sense to keep lowering to simple BUILD_VECTORs with 2 elements

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

357–364

Should use mi_match

384

eraseFromParent

[Pierre-vh]

Comments

[Pierre-vh]

I haven't touched the legalization of G_SHUFFLE_VECTOR I think

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

357–364

It doesn't look like there is a m_GShuffleVector, do I need to add it myself? I'm not too familiar with MIPatternMatch

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

357–364

Yes. This one is also a bit strange with the shuffle mask operand

[Pierre-vh]

Use mi_match

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–340

This doesn't make sense to me:

• The original has 0s in its high bits but the replacement does not.
• AMDGPU is little-endian so (G_BITCAST (G_SHUFFLE_VECTOR %a, %b, shufflemask(1, ?))) puts the high bits of %a in the low bits of the result - and then the G_LSHR shifts them out completely.

369

"... of the SHUFFLE op"?

377

LHS and RHS could have more than two elements, in which case I would expect this bitcast to fail machine verification. Can you add a test for that case?

[Pierre-vh]

Comments

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–340

This is really a headache to understand for me, but IIRC endianness doesn't affect values themselves, but how they're represented in memory? (https://stackoverflow.com/questions/7184789/does-bit-shift-depend-on-endianness)

So I think that G_SHUFFLE_VECTOR (1,?) puts the low bits in the high bits of the results, and the LSHR puts them back in place.

%a:(<2 x s16>) = [0xDEAD, 0xBEEF]
%b:(<2 x s16>) = undef
%c = G_SHUFFLE_VECTOR %a, %b, shufflemask(1,0)
; %c = [0xBEEF, 0xDEAD] 
%d = G_LSHR %c, 16
; %d = [0x0000, 0xBEEF]

So the lower 16 words stay the same, but indeed a difference before/after the combine is that the high words have 0000 if we leave the combine, but now they have "junk".

Maybe an additional trunc is needed in between or something?

What do you think @arsenm ?

377

Good catch, I'll enforce that the operands must be the same size as the dest.

[Pierre-vh]

Fix type check

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–340

I think you need to check for a mask of (?,0) instead of (1,?).

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–340

I think the mask is fine, for instance, in the test this is trying to fix, there's those two shufflevectors that need to be folded away to match the DAG codegen

%3:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_(<2 x s16>), %4:_, shufflemask(1, 0)
%37:_(s32) = G_BITCAST %3:_(<2 x s16>)
%33:_(s32) = G_CONSTANT i32 16
%38:_(s32) = G_LSHR %37:_, %33:_(s32)
%29:_(s16) = G_TRUNC %38:_(s32)

%5:_(<2 x s16>) = G_SHUFFLE_VECTOR %2:_(<2 x s16>), %4:_, shufflemask(1, 1)
%32:_(s32) = G_BITCAST %5:_(<2 x s16>)
%34:_(s32) = G_LSHR %32:_, %33:_(s32)
%21:_(s16) = G_TRUNC %34:_(s32)

Though to make this fully correct I think I need to take the trunc into account, so I will do that.

[Pierre-vh]

Take trunc into account

[arsenm]

Also note using legal shuffle_vector is a difference from the DAG we could go back on. Ultimately it's probably fewer total combines to only have to think about 16-bit shifts

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–339

Should break up this comment to avoid wrapping the pattern bit

368

LHS and RHS have to have the same type

394

Need to call the observer if modifying the argument list

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-trunc-shift-shufflevector.mir

273

Could use a test where the shuffle has multiple users

[Pierre-vh]

Comments

[Pierre-vh]

Rebase

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

338–340

This is really a headache to understand for me, but IIRC endianness doesn't affect values themselves, but how they're represented in memory?

It affects bitcasts. See https://llvm.org/docs/LangRef.html#vector-type:

A bitcast from a vector type to a scalar integer type will see the elements being packed together (without padding). The order in which elements are inserted in the integer depends on endianess. For little endian element zero is put in the least significant bits of the integer, and for big endian element zero is put in the most significant bits.

[Pierre-vh]

Rebase

[Pierre-vh]

Not the right kind of fix + too controversial of a change for a relatively small regression in mad_mix GISel.

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 
 template <typename Src0Ty, typename Src1Ty, typename Src2Ty>
 inline TernaryOp_match<Src0Ty, Src1Ty, Src2Ty, TargetOpcode::G_SELECT>
 m_GISelect(const Src0Ty &Src0, const Src1Ty &Src1, const Src2Ty &Src2) {
   return TernaryOp_match<Src0Ty, Src1Ty, Src2Ty, TargetOpcode::G_SELECT>(
       Src0, Src1, Src2);
 }
 
+// TODO: We could just use TernaryOp_match if we allow match() functions to
+//   optionally take a MachineOperand instead of always taking registers.
+//   We could then just have a m_ShuffleMask matcher like m_Reg for instance.
+template <typename Src0Ty, typename Src1Ty> struct GShuffleVector_match {
+  Src0Ty Src0;
+  Src1Ty Src1;
+  ArrayRef<int> &ShuffleMask;
+
+  GShuffleVector_match(const Src0Ty &Src0, const Src1Ty &Src1,
+                       ArrayRef<int> &ShuffleMask)
+      : Src0(Src0), Src1(Src1), ShuffleMask(ShuffleMask) {}
+
+  template <typename OpTy>
+  bool match(const MachineRegisterInfo &MRI, OpTy &&Op) {
+    MachineInstr *TmpMI;
+    if (mi_match(Op, MRI, m_MInstr(TmpMI))) {
+      if (TmpMI->getOpcode() == TargetOpcode::G_SHUFFLE_VECTOR &&
+          TmpMI->getNumOperands() == 4) {
+        ShuffleMask = TmpMI->getOperand(3).getShuffleMask();
+        return Src0.match(MRI, TmpMI->getOperand(1).getReg()) &&
+               Src1.match(MRI, TmpMI->getOperand(2).getReg());
+      }
+    }
+
+    return false;
+  }
+};
+
+template <typename Src0Ty, typename Src1Ty>
+inline GShuffleVector_match<Src0Ty, Src1Ty>
+m_GShuffleVector(const Src0Ty &Src0, const Src1Ty &Src1,
+                 ArrayRef<int> &ShuffleMask) {
+  return GShuffleVector_match<Src0Ty, Src1Ty>(Src0, Src1, ShuffleMask);
+}
+
 /// Matches a register negated by a G_SUB.
 /// G_SUB 0, %negated_reg
 template <typename SrcTy>
 inline BinaryOp_match<SpecificConstantMatch, SrcTy, TargetOpcode::G_SUB>
 m_Neg(const SrcTy &&Src) {
   return m_GSub(m_ZeroInt(), Src);
 }
 

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

def fp_minmax_to_clamp : GICombineRule<
   (apply [{ RegBankHelper.applyClamp(*${min_or_max}, ${matchinfo}); }])>;
 
 def fmed3_intrinsic_to_clamp : GICombineRule<
   (defs root:$fmed3, register_matchinfo:$matchinfo),
   (match (wip_match_opcode G_INTRINSIC):$fmed3,
          [{ return RegBankHelper.matchFPMed3ToClamp(*${fmed3}, ${matchinfo}); }]),
   (apply [{ RegBankHelper.applyClamp(*${fmed3}, ${matchinfo}); }])>;
 
+def trunc_shift_shufflevector_fold : GICombineRule<
+  (defs root:$trunc, register_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_TRUNC):$trunc,
+         [{ return RegBankHelper.matchTruncShiftShuffleVectorFold(*${trunc}, ${matchinfo}); }]),
+  (apply [{ RegBankHelper.applyTruncShiftShuffleVectorFold(*${trunc}, ${matchinfo}); }])>;
+
 def remove_fcanonicalize_matchinfo : GIDefMatchData<"Register">;
 
 def remove_fcanonicalize : GICombineRule<
   (defs root:$fcanonicalize, remove_fcanonicalize_matchinfo:$matchinfo),
   (match (wip_match_opcode G_FCANONICALIZE):$fcanonicalize,
          [{ return PostLegalizerHelper.matchRemoveFcanonicalize(*${fcanonicalize}, ${matchinfo}); }]),
   (apply [{ Helper.replaceSingleDefInstWithReg(*${fcanonicalize}, ${matchinfo}); }])>;
 
def AMDGPUPostLegalizerCombinerHelper: GICombinerHelper<
   let DisableRuleOption = "amdgpupostlegalizercombiner-disable-rule";
   let StateClass = "AMDGPUPostLegalizerCombinerHelperState";
   let AdditionalArguments = [];
 }
 
 def AMDGPURegBankCombinerHelper : GICombinerHelper<
   "AMDGPUGenRegBankCombinerHelper",
   [zext_trunc_fold, int_minmax_to_med3, ptr_add_immed_chain,
-   fp_minmax_to_clamp, fp_minmax_to_med3, fmed3_intrinsic_to_clamp]> {
+   fp_minmax_to_clamp, fp_minmax_to_med3, fmed3_intrinsic_to_clamp,
+   trunc_shift_shufflevector_fold]> {
   let DisableRuleOption = "amdgpuregbankcombiner-disable-rule";
   let StateClass = "AMDGPURegBankCombinerHelperState";
   let AdditionalArguments = [];
 }

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

protected:
   MachineIRBuilder &B;
   MachineFunction &MF;
   MachineRegisterInfo &MRI;
   const GCNSubtarget &Subtarget;
   const RegisterBankInfo &RBI;
   const TargetRegisterInfo &TRI;
   const SIInstrInfo &TII;
   CombinerHelper &Helper;
+  GISelChangeObserver &Observer;
 
 public:
-  AMDGPURegBankCombinerHelper(MachineIRBuilder &B, CombinerHelper &Helper)
+  AMDGPURegBankCombinerHelper(MachineIRBuilder &B, CombinerHelper &Helper,
+                              GISelChangeObserver &Observer)
       : B(B), MF(B.getMF()), MRI(*B.getMRI()),
         Subtarget(MF.getSubtarget<GCNSubtarget>()),
         RBI(*Subtarget.getRegBankInfo()), TRI(*Subtarget.getRegisterInfo()),
-        TII(*Subtarget.getInstrInfo()), Helper(Helper){};
+        TII(*Subtarget.getInstrInfo()), Helper(Helper), Observer(Observer){};
 
   bool isVgprRegBank(Register Reg);
   Register getAsVgpr(Register Reg);
 
   struct MinMaxMedOpc {
     unsigned Min, Max, Med;
   };
 
public:
 
   bool matchIntMinMaxToMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   bool matchFPMinMaxToMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   bool matchFPMinMaxToClamp(MachineInstr &MI, Register &Reg);
   bool matchFPMed3ToClamp(MachineInstr &MI, Register &Reg);
   void applyMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   void applyClamp(MachineInstr &MI, Register &Reg);
 
+  bool matchTruncShiftShuffleVectorFold(MachineInstr &MI, Register &Reg);
+  void applyTruncShiftShuffleVectorFold(MachineInstr &MI, Register &Reg);
+
 private:
   AMDGPU::SIModeRegisterDefaults getMode();
   bool getIEEE();
   bool getDX10Clamp();
   bool isFminnumIeee(const MachineInstr &MI);
   bool isFCst(MachineInstr *MI);
   bool isClampZeroToOne(MachineInstr *K0, MachineInstr *K1);
 };
void AMDGPURegBankCombinerHelper::applyMed3(MachineInstr &MI,
   B.setInstrAndDebugLoc(MI);
   B.buildInstr(MatchInfo.Opc, {MI.getOperand(0)},
                {getAsVgpr(MatchInfo.Val0), getAsVgpr(MatchInfo.Val1),
                 getAsVgpr(MatchInfo.Val2)},
                MI.getFlags());
   MI.eraseFromParent();
 }
 
+bool AMDGPURegBankCombinerHelper::matchTruncShiftShuffleVectorFold(
+    MachineInstr &MI, Register &Reg) {
+  // Fold
+  //
+  // (G_TRUNC (G_LSHR
+  //   (G_BITCAST (G_SHUFFLE_VECTOR %a, %b, shufflemask(1, ?)))
+  // , K))
+  //
+  // into a simple
+  //
+  //    (G_TRUNC (G_BITCAST(%a))
+  //
+  // if the shift amount (K) is  1/2 of the destination type an
+  // the vector types have 2 elements.
+
+  assert(MI.getOpcode() == AMDGPU::G_TRUNC);
+
+  Register TruncSrc = MI.getOperand(1).getReg();
+  const LLT TruncSrcTy = MRI.getType(TruncSrc);
+  const unsigned TruncSrcSize = TruncSrcTy.getSizeInBits();
+
+  Register LHS, RHS;
+  ArrayRef<int> ShuffleMask;
+  Optional<ValueAndVReg> ShiftAmount;
+  if (!mi_match(TruncSrc, MRI,
+                m_GLShr(m_GBitcast(m_GShuffleVector(m_Reg(LHS), m_Reg(RHS),
+                                                    ShuffleMask)),
+                        m_GCst(ShiftAmount)))) {
+    return false;
+  }
+
+  // The shift amount is 1/2 of the scalar type.
+  if (ShiftAmount->Value != (TruncSrcSize / 2))
+    return false;
+
+  // The operands of the SHUFFLE_VECTOR must be the same size as its
+  // destination. This limitation could be lifted if needed, but
+  // applyTruncShiftShuffleVectorFold will need to take it into account and
+  // generate additional operations to trunc the input instead of just
+  // generating a bitcast.
+  if (MRI.getType(LHS).getSizeInBits() != TruncSrcSize)
+    return false;
+
+  // The SHUFFLE_VECTOR's dest is a 2-element vector
+  // and the first element is 1 (2nd element from first vector).
+  if (ShuffleMask.size() != 2 || ShuffleMask[0] != 1)
+    return false;
+
+  // We can just replace the trunc src with a bitcast of the LHS of the
+  // G_SHUFFLE_VECTOR.
+  Reg = LHS;
+  return true;
+}
+
+void AMDGPURegBankCombinerHelper::applyTruncShiftShuffleVectorFold(
+    MachineInstr &MI, Register &Reg) {
+  B.setInstrAndDebugLoc(MI);
+
+  Register TruncSrc = MI.getOperand(1).getReg();
+  const RegisterBank *TruncSrcRB = MRI.getRegBankOrNull(TruncSrc);
+  assert(TruncSrcRB && "TruncSrc has no RegBank assigned");
+
+  Register NewSrc = MRI.createGenericVirtualRegister(MRI.getType(TruncSrc));
+  MRI.setRegBank(NewSrc, *TruncSrcRB);
+
+  B.buildBitcast(NewSrc, Reg);
+
+  Observer.changingInstr(MI);
+  MI.getOperand(1).setReg(NewSrc);
+  Observer.changedInstr(MI);
+}
+
 AMDGPU::SIModeRegisterDefaults AMDGPURegBankCombinerHelper::getMode() {
   return MF.getInfo<SIMachineFunctionInfo>()->getMode();
 }
 
 bool AMDGPURegBankCombinerHelper::getIEEE() { return getMode().IEEE; }
 
 bool AMDGPURegBankCombinerHelper::getDX10Clamp() { return getMode().DX10Clamp; }
 
public:
   bool combine(GISelChangeObserver &Observer, MachineInstr &MI,
                MachineIRBuilder &B) const override;
 };
 
 bool AMDGPURegBankCombinerInfo::combine(GISelChangeObserver &Observer,
                                               MachineInstr &MI,
                                               MachineIRBuilder &B) const {
   CombinerHelper Helper(Observer, B, KB, MDT);
-  AMDGPURegBankCombinerHelper RegBankHelper(B, Helper);
+  AMDGPURegBankCombinerHelper RegBankHelper(B, Helper, Observer);
   AMDGPUGenRegBankCombinerHelper Generated(GeneratedRuleCfg, Helper,
                                            RegBankHelper);
 
   if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   return false;
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-trunc-shift-shufflevector.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -run-pass=amdgpu-regbank-combiner -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name:            lshr16_v2s16_mask10
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask10
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[COPY]](<2 x s16>)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[BITCAST]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: $vgpr0 = COPY [[ANYEXT]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    $vgpr0 = COPY %7
+...
+
+---
+name:            lshr16_v2s16_mask10_multiple_shuffle_uses
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask10_multiple_shuffle_uses
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[COPY]](<2 x s16>)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[BITCAST]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: [[BITCAST1:%[0-9]+]]:vgpr(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[AND:%[0-9]+]]:vgpr(s32) = G_AND [[BITCAST1]], [[ANYEXT]]
+    ; CHECK-NEXT: $vgpr0 = COPY [[AND]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    %8:vgpr(s32) = G_BITCAST %2
+    %9:vgpr(s32) = G_AND %8, %7
+    $vgpr0 = COPY %9
+...
+
+---
+name:            lshr16_v2s16_mask10_notrunc_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask10_notrunc_nofold
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: $vgpr0 = COPY [[LSHR]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    $vgpr0 = COPY %5
+...
+
+---
+name:            lshr16_v2s16_v4s16_mask10_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr16_v2s16_v4s16_mask10_nofold
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<4 x s16>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<4 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<4 x s16>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[LSHR]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: $vgpr0 = COPY [[ANYEXT]](s32)
+    %0:vgpr(<4 x s16>) = COPY $vgpr0_vgpr1
+    %1:vgpr(<4 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    $vgpr0 = COPY %7
+...
+
+---
+name:            lshr8_v2s16_mask10_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr8_v2s16_mask10_nofold
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 8
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[LSHR]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: $vgpr0 = COPY [[ANYEXT]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 8
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    $vgpr0 = COPY %7
+...
+
+---
+name:            lshr16_v2s16_mask11
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask11
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[COPY]](<2 x s16>)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[BITCAST]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: $vgpr0 = COPY [[ANYEXT]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 1)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    $vgpr0 = COPY %7
+...
+
+---
+name:            lshr16_v2s16_mask01_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask01_nofold
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(0, 1)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s16) = G_TRUNC [[LSHR]](s32)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s32) = G_ANYEXT [[TRUNC]](s16)
+    ; CHECK-NEXT: $vgpr0 = COPY [[ANYEXT]](s32)
+    %0:vgpr(<2 x s16>) = COPY $vgpr0
+    %1:vgpr(<2 x s16>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s16>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(0, 1)
+    %3:vgpr(s32) = G_BITCAST %2
+    %4:vgpr(s32) = G_CONSTANT i32 16
+    %5:vgpr(s32) = G_LSHR %3, %4
+    %6:vgpr(s16) = G_TRUNC %5
+    %7:vgpr(s32) = G_ANYEXT %6
+    $vgpr0 = COPY %7
+...
+
+---
+name:            lshr32_v2s32_mask10
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask10
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s64) = G_BITCAST [[COPY]](<2 x s32>)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s32) = G_TRUNC [[BITCAST]](s64)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s64) = G_ANYEXT [[TRUNC]](s32)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[ANYEXT]](s64)
+    %0:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s64) = G_BITCAST %2
+    %4:vgpr(s64) = G_CONSTANT i64 32
+    %5:vgpr(s64) = G_LSHR %3, %4
+    %6:vgpr(s32) = G_TRUNC %5
+    %7:vgpr(s64) = G_ANYEXT %6
+    $vgpr0_vgpr1 = COPY %7
+...
+
+---
+name:            lshr16_v2s32_mask10_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr16_v2s32_mask10_nofold
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s32>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s64) = G_BITCAST [[SHUF]](<2 x s32>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s64) = G_CONSTANT i64 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s64) = G_LSHR [[BITCAST]], [[C]](s64)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s32) = G_TRUNC [[LSHR]](s64)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s64) = G_ANYEXT [[TRUNC]](s32)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[ANYEXT]](s64)
+    %0:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 0)
+    %3:vgpr(s64) = G_BITCAST %2
+    %4:vgpr(s64) = G_CONSTANT i64 16
+    %5:vgpr(s64) = G_LSHR %3, %4
+    %6:vgpr(s32) = G_TRUNC %5
+    %7:vgpr(s64) = G_ANYEXT %6
+    $vgpr0_vgpr1 = COPY %7
+...
+
+---
+name:            lshr32_v2s32_mask11
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask11
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s64) = G_BITCAST [[COPY]](<2 x s32>)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s32) = G_TRUNC [[BITCAST]](s64)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s64) = G_ANYEXT [[TRUNC]](s32)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[ANYEXT]](s64)
+    %0:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(1, 1)
+    %3:vgpr(s64) = G_BITCAST %2
+    %4:vgpr(s64) = G_CONSTANT i64 32
+    %5:vgpr(s64) = G_LSHR %3, %4
+    %6:vgpr(s32) = G_TRUNC %5
+    %7:vgpr(s64) = G_ANYEXT %6
+    $vgpr0_vgpr1 = COPY %7
+...
+
+---
+name:            lshr32_v2s32_mask01_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask01_nofold
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s32>), [[DEF]], shufflemask(0, 1)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:vgpr(s64) = G_BITCAST [[SHUF]](<2 x s32>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:vgpr(s64) = G_CONSTANT i64 32
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:vgpr(s64) = G_LSHR [[BITCAST]], [[C]](s64)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:vgpr(s32) = G_TRUNC [[LSHR]](s64)
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:vgpr(s64) = G_ANYEXT [[TRUNC]](s32)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[ANYEXT]](s64)
+    %0:vgpr(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:vgpr(<2 x s32>) = G_IMPLICIT_DEF
+    %2:vgpr(<2 x s32>) = G_SHUFFLE_VECTOR %0:vgpr, %1:vgpr, shufflemask(0, 1)
+    %3:vgpr(s64) = G_BITCAST %2
+    %4:vgpr(s64) = G_CONSTANT i64 32
+    %5:vgpr(s64) = G_LSHR %3, %4
+    %6:vgpr(s32) = G_TRUNC %5
+    %7:vgpr(s64) = G_ANYEXT %6
+    $vgpr0_vgpr1 = COPY %7
+...