[AMDGPU][GlobalISel] Add Shift/Shufflevector Combine

[Pierre-vh]

Summary

Folds G_LSHR + G_SHUFFLEVECTOR away when the shift amount
is 1/2 of the destination scalar size, and the mask starts with 1.

Depends on D134870.

[arsenm]

Another thing worth thinking about is whether we should really be using legal vector_shuffle at all. The DAG path doesn't use it. Given that we have such a restricted set of shuffles (only 0, 1), it may still make sense to keep lowering to simple BUILD_VECTORs with 2 elements

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

355–362

Should use mi_match

382

eraseFromParent

[Pierre-vh]

Comments

[Pierre-vh]

I haven't touched the legalization of G_SHUFFLE_VECTOR I think

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

355–362

It doesn't look like there is a m_GShuffleVector, do I need to add it myself? I'm not too familiar with MIPatternMatch

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

355–362

Yes. This one is also a bit strange with the shuffle mask operand

[Pierre-vh]

Use mi_match

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–338

This doesn't make sense to me:

• The original has 0s in its high bits but the replacement does not.
• AMDGPU is little-endian so (G_BITCAST (G_SHUFFLE_VECTOR %a, %b, shufflemask(1, ?))) puts the high bits of %a in the low bits of the result - and then the G_LSHR shifts them out completely.

367

"... of the SHUFFLE op"?

375

LHS and RHS could have more than two elements, in which case I would expect this bitcast to fail machine verification. Can you add a test for that case?

[Pierre-vh]

Comments

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–338

This is really a headache to understand for me, but IIRC endianness doesn't affect values themselves, but how they're represented in memory? (https://stackoverflow.com/questions/7184789/does-bit-shift-depend-on-endianness)

So I think that G_SHUFFLE_VECTOR (1,?) puts the low bits in the high bits of the results, and the LSHR puts them back in place.

%a:(<2 x s16>) = [0xDEAD, 0xBEEF]
%b:(<2 x s16>) = undef
%c = G_SHUFFLE_VECTOR %a, %b, shufflemask(1,0)
; %c = [0xBEEF, 0xDEAD] 
%d = G_LSHR %c, 16
; %d = [0x0000, 0xBEEF]

So the lower 16 words stay the same, but indeed a difference before/after the combine is that the high words have 0000 if we leave the combine, but now they have "junk".

Maybe an additional trunc is needed in between or something?

What do you think @arsenm ?

375

Good catch, I'll enforce that the operands must be the same size as the dest.

[Pierre-vh]

Fix type check

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–338

I think you need to check for a mask of (?,0) instead of (1,?).

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–338

I think the mask is fine, for instance, in the test this is trying to fix, there's those two shufflevectors that need to be folded away to match the DAG codegen

%3:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_(<2 x s16>), %4:_, shufflemask(1, 0)
%37:_(s32) = G_BITCAST %3:_(<2 x s16>)
%33:_(s32) = G_CONSTANT i32 16
%38:_(s32) = G_LSHR %37:_, %33:_(s32)
%29:_(s16) = G_TRUNC %38:_(s32)

%5:_(<2 x s16>) = G_SHUFFLE_VECTOR %2:_(<2 x s16>), %4:_, shufflemask(1, 1)
%32:_(s32) = G_BITCAST %5:_(<2 x s16>)
%34:_(s32) = G_LSHR %32:_, %33:_(s32)
%21:_(s16) = G_TRUNC %34:_(s32)

Though to make this fully correct I think I need to take the trunc into account, so I will do that.

[Pierre-vh]

Take trunc into account

[arsenm]

Also note using legal shuffle_vector is a difference from the DAG we could go back on. Ultimately it's probably fewer total combines to only have to think about 16-bit shifts

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–337

Should break up this comment to avoid wrapping the pattern bit

366

LHS and RHS have to have the same type

392

Need to call the observer if modifying the argument list

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-trunc-shift-shufflevector.mir

272 ↗

(On Diff #463180)

Could use a test where the shuffle has multiple users

[Pierre-vh]

Comments

[Pierre-vh]

Rebase

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

336–338

This is really a headache to understand for me, but IIRC endianness doesn't affect values themselves, but how they're represented in memory?

It affects bitcasts. See https://llvm.org/docs/LangRef.html#vector-type:

A bitcast from a vector type to a scalar integer type will see the elements being packed together (without padding). The order in which elements are inserted in the integer depends on endianess. For little endian element zero is put in the least significant bits of the integer, and for big endian element zero is put in the most significant bits.

[Pierre-vh]

Rebase

[Pierre-vh]

Not the right kind of fix + too controversial of a change for a relatively small regression in mad_mix GISel.

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 
 template <typename Src0Ty, typename Src1Ty, typename Src2Ty>
 inline TernaryOp_match<Src0Ty, Src1Ty, Src2Ty, TargetOpcode::G_SELECT>
 m_GISelect(const Src0Ty &Src0, const Src1Ty &Src1, const Src2Ty &Src2) {
   return TernaryOp_match<Src0Ty, Src1Ty, Src2Ty, TargetOpcode::G_SELECT>(
       Src0, Src1, Src2);
 }
 
+// TODO: We could just use TernaryOp_match if we allow match() functions to
+//   optionally take a MachineOperand instead of always taking registers.
+//   We could then just have a m_ShuffleMask matcher like m_Reg for instance.
+template <typename Src0Ty, typename Src1Ty> struct GShuffleVector_match {
+  Src0Ty Src0;
+  Src1Ty Src1;
+  ArrayRef<int> &ShuffleMask;
+
+  GShuffleVector_match(const Src0Ty &Src0, const Src1Ty &Src1,
+                       ArrayRef<int> &ShuffleMask)
+      : Src0(Src0), Src1(Src1), ShuffleMask(ShuffleMask) {}
+
+  template <typename OpTy>
+  bool match(const MachineRegisterInfo &MRI, OpTy &&Op) {
+    MachineInstr *TmpMI;
+    if (mi_match(Op, MRI, m_MInstr(TmpMI))) {
+      if (TmpMI->getOpcode() == TargetOpcode::G_SHUFFLE_VECTOR &&
+          TmpMI->getNumOperands() == 4) {
+        ShuffleMask = TmpMI->getOperand(3).getShuffleMask();
+        return Src0.match(MRI, TmpMI->getOperand(1).getReg()) &&
+               Src1.match(MRI, TmpMI->getOperand(2).getReg());
+      }
+    }
+
+    return false;
+  }
+};
+
+template <typename Src0Ty, typename Src1Ty>
+inline GShuffleVector_match<Src0Ty, Src1Ty>
+m_GShuffleVector(const Src0Ty &Src0, const Src1Ty &Src1,
+                 ArrayRef<int> &ShuffleMask) {
+  return GShuffleVector_match<Src0Ty, Src1Ty>(Src0, Src1, ShuffleMask);
+}
+
 /// Matches a register negated by a G_SUB.
 /// G_SUB 0, %negated_reg
 template <typename SrcTy>
 inline BinaryOp_match<SpecificConstantMatch, SrcTy, TargetOpcode::G_SUB>
 m_Neg(const SrcTy &&Src) {
   return m_GSub(m_ZeroInt(), Src);
 }
 

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

def fp_minmax_to_clamp : GICombineRule<
   (apply [{ RegBankHelper.applyClamp(*${min_or_max}, ${matchinfo}); }])>;
 
 def fmed3_intrinsic_to_clamp : GICombineRule<
   (defs root:$fmed3, register_matchinfo:$matchinfo),
   (match (wip_match_opcode G_INTRINSIC):$fmed3,
          [{ return RegBankHelper.matchFPMed3ToClamp(*${fmed3}, ${matchinfo}); }]),
   (apply [{ RegBankHelper.applyClamp(*${fmed3}, ${matchinfo}); }])>;
 
+def shift_shufflevector_fold : GICombineRule<
+  (defs root:$shift, register_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_LSHR):$shift,
+         [{ return RegBankHelper.matchShiftShuffleVectorFold(*${shift}, ${matchinfo}); }]),
+  (apply [{ RegBankHelper.applyShiftShuffleVectorFold(*${shift}, ${matchinfo}); }])>;
+
 def remove_fcanonicalize_matchinfo : GIDefMatchData<"Register">;
 
 def remove_fcanonicalize : GICombineRule<
   (defs root:$fcanonicalize, remove_fcanonicalize_matchinfo:$matchinfo),
   (match (wip_match_opcode G_FCANONICALIZE):$fcanonicalize,
          [{ return PostLegalizerHelper.matchRemoveFcanonicalize(*${fcanonicalize}, ${matchinfo}); }]),
   (apply [{ Helper.replaceSingleDefInstWithReg(*${fcanonicalize}, ${matchinfo}); }])>;
 
def AMDGPUPostLegalizerCombinerHelper: GICombinerHelper<
   let DisableRuleOption = "amdgpupostlegalizercombiner-disable-rule";
   let StateClass = "AMDGPUPostLegalizerCombinerHelperState";
   let AdditionalArguments = [];
 }
 
 def AMDGPURegBankCombinerHelper : GICombinerHelper<
   "AMDGPUGenRegBankCombinerHelper",
   [zext_trunc_fold, int_minmax_to_med3, ptr_add_immed_chain,
-   fp_minmax_to_clamp, fp_minmax_to_med3, fmed3_intrinsic_to_clamp]> {
+   fp_minmax_to_clamp, fp_minmax_to_med3, fmed3_intrinsic_to_clamp,
+   shift_shufflevector_fold]> {
   let DisableRuleOption = "amdgpuregbankcombiner-disable-rule";
   let StateClass = "AMDGPURegBankCombinerHelperState";
   let AdditionalArguments = [];
 }

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

public:
 
   bool matchIntMinMaxToMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   bool matchFPMinMaxToMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   bool matchFPMinMaxToClamp(MachineInstr &MI, Register &Reg);
   bool matchFPMed3ToClamp(MachineInstr &MI, Register &Reg);
   void applyMed3(MachineInstr &MI, Med3MatchInfo &MatchInfo);
   void applyClamp(MachineInstr &MI, Register &Reg);
 
+  bool matchShiftShuffleVectorFold(MachineInstr &MI, Register &Reg);
+  void applyShiftShuffleVectorFold(MachineInstr &MI, Register &Reg);
+
 private:
   AMDGPU::SIModeRegisterDefaults getMode();
   bool getIEEE();
   bool getDX10Clamp();
   bool isFminnumIeee(const MachineInstr &MI);
   bool isFCst(MachineInstr *MI);
   bool isClampZeroToOne(MachineInstr *K0, MachineInstr *K1);
 };
void AMDGPURegBankCombinerHelper::applyMed3(MachineInstr &MI,
   B.setInstrAndDebugLoc(MI);
   B.buildInstr(MatchInfo.Opc, {MI.getOperand(0)},
                {getAsVgpr(MatchInfo.Val0), getAsVgpr(MatchInfo.Val1),
                 getAsVgpr(MatchInfo.Val2)},
                MI.getFlags());
   MI.eraseFromParent();
 }
 
+bool AMDGPURegBankCombinerHelper::matchShiftShuffleVectorFold(MachineInstr &MI,
+                                                              Register &Reg) {
+  // Fold (G_LSHR (G_BITCAST (G_SHUFFLE_VECTOR %a, %b, shufflemask(1, ?))), K)
+  // into a simple G_BITCAST(%a) if the shift amount (K) is 1/2 of the
+  // destination type and the vector types have 2 elements.
+  //
+  // TODO: This could be made more generic (e.g. to support more vector sizes,
+  // or even LSHL)
+  //  if need be.
+
+  assert(MI.getOpcode() == AMDGPU::G_LSHR);
+
+  LLT DstTy = MRI.getType(MI.getOperand(0).getReg());
+  const auto ShiftAmount =
+      getIConstantVRegValWithLookThrough(MI.getOperand(2).getReg(), MRI);
+
+  // The shift amount is 1/2 of the scalar type.
+  if (!ShiftAmount || ShiftAmount->Value != (DstTy.getSizeInBits() / 2))
+    return false;
+
+  Register LHS, RHS;
+  ArrayRef<int> ShuffleMask;
+  if (!mi_match(
+          MI.getOperand(1).getReg(), MRI,
+          m_GBitcast(m_GShuffleVector(m_Reg(LHS), m_Reg(RHS), ShuffleMask)))) {
+    return false;
+  }
+
+  // The SHUFFLE_VECTOR's dest is a 2-element vector
+  // and the first element is 1 (2nd element from first vector).
+  if (ShuffleMask.size() != 2 || ShuffleMask[0] != 1)
+    return false;
+
+  // We can just replace this with a bitcast of the LHS of the ShiftOp.
+  Reg = LHS;
+  return true;
+}
+
+void AMDGPURegBankCombinerHelper::applyShiftShuffleVectorFold(MachineInstr &MI,
+                                                              Register &Reg) {
+  B.setInstrAndDebugLoc(MI);
+  B.buildBitcast(MI.getOperand(0).getReg(), Reg);
+  MI.eraseFromParent();
+}
+
 AMDGPU::SIModeRegisterDefaults AMDGPURegBankCombinerHelper::getMode() {
   return MF.getInfo<SIMachineFunctionInfo>()->getMode();
 }
 
 bool AMDGPURegBankCombinerHelper::getIEEE() { return getMode().IEEE; }
 
 bool AMDGPURegBankCombinerHelper::getDX10Clamp() { return getMode().DX10Clamp; }
 
 bool AMDGPURegBankCombinerHelper::isFminnumIeee(const MachineInstr &MI) {
   return MI.getOpcode() == AMDGPU::G_FMINNUM_IEEE;
 }
 
 bool AMDGPURegBankCombinerHelper::isFCst(MachineInstr *MI) {
   return MI->getOpcode() == AMDGPU::G_FCONSTANT;
 }
 
 bool AMDGPURegBankCombinerHelper::isClampZeroToOne(MachineInstr *K0,
                                                    MachineInstr *K1) {
   if (isFCst(K0) && isFCst(K1)) {
     const ConstantFP *KO_FPImm = K0->getOperand(1).getFPImm();
     const ConstantFP *K1_FPImm = K1->getOperand(1).getFPImm();
     return (KO_FPImm->isExactlyValue(0.0) && K1_FPImm->isExactlyValue(1.0)) ||

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-shift-shufflevector.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -run-pass=amdgpu-regbank-combiner -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name:            lshr16_v2s16_mask10
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask10
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[COPY]](<2 x s16>)
+    ; CHECK-NEXT: $vgpr0 = COPY [[BITCAST]](s32)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %1:_(<2 x s16>) = G_IMPLICIT_DEF
+    %2:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 0)
+    %3:_(s32) = G_BITCAST %2
+    %4:_(s32) = G_CONSTANT i32 16
+    %5:_(s32) = G_LSHR %3, %4
+    $vgpr0 = COPY %5
+...
+
+---
+name:            lshr8_v2s16_mask10_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr8_v2s16_mask10_nofold
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:_(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:_(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 8
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: $vgpr0 = COPY [[LSHR]](s32)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %1:_(<2 x s16>) = G_IMPLICIT_DEF
+    %2:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 0)
+    %3:_(s32) = G_BITCAST %2
+    %4:_(s32) = G_CONSTANT i32 8
+    %5:_(s32) = G_LSHR %3, %4
+    $vgpr0 = COPY %5
+...
+
+---
+name:            lshr16_v2s16_mask11
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask11
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[COPY]](<2 x s16>)
+    ; CHECK-NEXT: $vgpr0 = COPY [[BITCAST]](s32)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %1:_(<2 x s16>) = G_IMPLICIT_DEF
+    %2:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 1)
+    %3:_(s32) = G_BITCAST %2
+    %4:_(s32) = G_CONSTANT i32 16
+    %5:_(s32) = G_LSHR %3, %4
+    $vgpr0 = COPY %5
+...
+
+---
+name:            lshr16_v2s16_mask01_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    ; CHECK-LABEL: name: lshr16_v2s16_mask01_nofold
+    ; CHECK: liveins: $vgpr0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr0
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:_(<2 x s16>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:_(<2 x s16>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s16>), [[DEF]], shufflemask(0, 1)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[SHUF]](<2 x s16>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+    ; CHECK-NEXT: $vgpr0 = COPY [[LSHR]](s32)
+    %0:_(<2 x s16>) = COPY $vgpr0
+    %1:_(<2 x s16>) = G_IMPLICIT_DEF
+    %2:_(<2 x s16>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(0, 1)
+    %3:_(s32) = G_BITCAST %2
+    %4:_(s32) = G_CONSTANT i32 16
+    %5:_(s32) = G_LSHR %3, %4
+    $vgpr0 = COPY %5
+...
+
+---
+name:            lshr32_v2s32_mask10
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask10
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s64) = G_BITCAST [[COPY]](<2 x s32>)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[BITCAST]](s64)
+    %0:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:_(<2 x s32>) = G_IMPLICIT_DEF
+    %2:_(<2 x s32>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 0)
+    %3:_(s64) = G_BITCAST %2
+    %4:_(s64) = G_CONSTANT i64 32
+    %5:_(s64) = G_LSHR %3, %4
+    $vgpr0_vgpr1 = COPY %5
+...
+
+---
+name:            lshr16_v2s32_mask10_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr16_v2s32_mask10_nofold
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:_(<2 x s32>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:_(<2 x s32>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s32>), [[DEF]], shufflemask(1, 0)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s64) = G_BITCAST [[SHUF]](<2 x s32>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s64) = G_LSHR [[BITCAST]], [[C]](s64)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[LSHR]](s64)
+    %0:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:_(<2 x s32>) = G_IMPLICIT_DEF
+    %2:_(<2 x s32>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 0)
+    %3:_(s64) = G_BITCAST %2
+    %4:_(s64) = G_CONSTANT i64 16
+    %5:_(s64) = G_LSHR %3, %4
+    $vgpr0_vgpr1 = COPY %5
+...
+
+---
+name:            lshr32_v2s32_mask11
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask11
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s64) = G_BITCAST [[COPY]](<2 x s32>)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[BITCAST]](s64)
+    %0:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:_(<2 x s32>) = G_IMPLICIT_DEF
+    %2:_(<2 x s32>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(1, 1)
+    %3:_(s64) = G_BITCAST %2
+    %4:_(s64) = G_CONSTANT i64 32
+    %5:_(s64) = G_LSHR %3, %4
+    $vgpr0_vgpr1 = COPY %5
+...
+
+---
+name:            lshr32_v2s32_mask01_nofold
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1
+    ; CHECK-LABEL: name: lshr32_v2s32_mask01_nofold
+    ; CHECK: liveins: $vgpr0_vgpr1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    ; CHECK-NEXT: [[DEF:%[0-9]+]]:_(<2 x s32>) = G_IMPLICIT_DEF
+    ; CHECK-NEXT: [[SHUF:%[0-9]+]]:_(<2 x s32>) = G_SHUFFLE_VECTOR [[COPY]](<2 x s32>), [[DEF]], shufflemask(0, 1)
+    ; CHECK-NEXT: [[BITCAST:%[0-9]+]]:_(s64) = G_BITCAST [[SHUF]](<2 x s32>)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 32
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s64) = G_LSHR [[BITCAST]], [[C]](s64)
+    ; CHECK-NEXT: $vgpr0_vgpr1 = COPY [[LSHR]](s64)
+    %0:_(<2 x s32>) = COPY $vgpr0_vgpr1
+    %1:_(<2 x s32>) = G_IMPLICIT_DEF
+    %2:_(<2 x s32>) = G_SHUFFLE_VECTOR %0:_, %1:_, shufflemask(0, 1)
+    %3:_(s64) = G_BITCAST %2
+    %4:_(s64) = G_CONSTANT i64 32
+    %5:_(s64) = G_LSHR %3, %4
+    $vgpr0_vgpr1 = COPY %5
+...