[GlobalISel] Add a combine for ashr(shl x, c), c --> sext_inreg x, c'

[aemerson]

Summary

By detecting this sign extend pattern early, we can uncover opportunities for more optimizations.

[arsenm]

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

1895

Add a fixme for splat vectors?

llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-ashr-shl-to-sext-inreg.mir

63

Could use some vector tests. I don't think m_ICst is smart enough to match splat vectors though. Adding the tests won't hurt for when that's solved.

[aemerson]

In D85965#2218104, @arsenm wrote:

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

This is intended to run prelegalizer so I don't think we need it.

[arsenm]

In D85965#2218352, @aemerson wrote:

In D85965#2218104, @arsenm wrote:

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

This is intended to run prelegalizer so I don't think we need it.

Combines should probably try to support both. There's fragments of half implemented support for this

[aemerson]

In D85965#2218363, @arsenm wrote:

In D85965#2218352, @aemerson wrote:

In D85965#2218104, @arsenm wrote:

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

This is intended to run prelegalizer so I don't think we need it.

Combines should probably try to support both. There's fragments of half implemented support for this

Every combine should check legality for each instruction it wants to generate? That seems like way overkill to me. The majority of the combines we do will be pre-legalize with post being used to clean up legalizer related things or do some lowering for isel. I think it should be done on as needed basis by whichever target needs a specific combine to be done post-legalize. Unless we have a better mechanism to do this that doesn't involve manually calling into LI every time,

[arsenm]

In D85965#2218403, @aemerson wrote:

In D85965#2218363, @arsenm wrote:

In D85965#2218352, @aemerson wrote:

In D85965#2218104, @arsenm wrote:

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

This is intended to run prelegalizer so I don't think we need it.

Combines should probably try to support both. There's fragments of half implemented support for this

Every combine should check legality for each instruction it wants to generate? That seems like way overkill to me. The majority of the combines we do will be pre-legalize with post being used to clean up legalizer related things or do some lowering for isel. I think it should be done on as needed basis by whichever target needs a specific combine to be done post-legalize. Unless we have a better mechanism to do this that doesn't involve manually calling into LI every time,

Currently we have CombinerHelper::isLegalOrBeforeLegalizer, which some combines use. CombinerInfo has IllegalOpsAllowed and LegalizeIllegalOps (which has the comment // TODO: Make use of this.) I guess the intent was these could re-legalize as it goes, and some reasonable subset of legality checks could be use for a profitability heuristic?

I think the majority of interesting combines are post-legalize, particularly when it comes to bit operations. Before legalization the MIR isn't particularly different than the source optimized IR, and the legalizer introduces new patterns.

[aemerson]

In D85965#2222602, @arsenm wrote:

In D85965#2218403, @aemerson wrote:

In D85965#2218363, @arsenm wrote:

In D85965#2218352, @aemerson wrote:

In D85965#2218104, @arsenm wrote:

Seems like it's missing a legality check, although I'm unclear on what the overall strategy for those is supposed to be

This is intended to run prelegalizer so I don't think we need it.

Combines should probably try to support both. There's fragments of half implemented support for this

Every combine should check legality for each instruction it wants to generate? That seems like way overkill to me. The majority of the combines we do will be pre-legalize with post being used to clean up legalizer related things or do some lowering for isel. I think it should be done on as needed basis by whichever target needs a specific combine to be done post-legalize. Unless we have a better mechanism to do this that doesn't involve manually calling into LI every time,

Currently we have CombinerHelper::isLegalOrBeforeLegalizer, which some combines use. CombinerInfo has IllegalOpsAllowed and LegalizeIllegalOps (which has the comment // TODO: Make use of this.) I guess the intent was these could re-legalize as it goes, and some reasonable subset of legality checks could be use for a profitability heuristic?

I think the majority of interesting combines are post-legalize, particularly when it comes to bit operations. Before legalization the MIR isn't particularly different than the source optimized IR, and the legalizer introduces new patterns.

I think those flags were intended for the full-fledged tablegen pattern combiner to use, but your point still stands. I'll add a legality check here but it seems like we need some more infrastructure to make use of IllegalOpsAllowed on the patterns in a more concise way.

[aemerson]

Use isLegalOrBeforeLegalizer() and make the combine run unconditionally as part of the "all" set. This causes codegen changes in AMDGPU.

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
   bool
   matchHoistLogicOpWithSameOpcodeHands(MachineInstr &MI,
                                        InstructionStepsMatchInfo &MatchInfo);
 
   /// Replace \p MI with a series of instructions described in \p MatchInfo.
   bool applyBuildInstructionSteps(MachineInstr &MI,
                                   InstructionStepsMatchInfo &MatchInfo);
 
+  /// Match ashr (shl x, C), C -> sext_inreg (C)
+  bool matchAshrShlToSextInreg(MachineInstr &MI,
+                               std::tuple<Register, int64_t> &MatchInfo);
+  bool applyAshShlToSextInreg(MachineInstr &MI,
+                              std::tuple<Register, int64_t> &MatchInfo);
+
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
 private:
   // Memcpy family optimization helpers.
   bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
                       unsigned KnownLen, Align DstAlign, Align SrcAlign,

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 }
 
 template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_LSHR, false>
 m_GLShr(const LHS &L, const RHS &R) {
   return BinaryOp_match<LHS, RHS, TargetOpcode::G_LSHR, false>(L, R);
 }
 
+template <typename LHS, typename RHS>
+inline BinaryOp_match<LHS, RHS, TargetOpcode::G_ASHR, false>
+m_GAShr(const LHS &L, const RHS &R) {
+  return BinaryOp_match<LHS, RHS, TargetOpcode::G_ASHR, false>(L, R);
+}
+
 // Helper for unary instructions (G_[ZSA]EXT/G_TRUNC) etc
 template <typename SrcTy, unsigned Opcode> struct UnaryOp_match {
   SrcTy L;
 
   UnaryOp_match(const SrcTy &LHS) : L(LHS) {}
   template <typename OpTy>
   bool match(const MachineRegisterInfo &MRI, OpTy &&Op) {
     MachineInstr *TmpMI;

llvm/include/llvm/Target/GlobalISel/Combine.td

 // Simplify: (logic_op (op x...), (op y...)) -> (op (logic_op x, y))
 def hoist_logic_op_with_same_opcode_hands: GICombineRule <
   (defs root:$root, instruction_steps_matchdata:$info),
   (match (wip_match_opcode G_AND, G_OR, G_XOR):$root,
     [{ return Helper.matchHoistLogicOpWithSameOpcodeHands(*${root}, ${info}); }]),
   (apply [{ return Helper.applyBuildInstructionSteps(*${root}, ${info});}])
 >;
 
+// Fold ashr (shl x, C), C -> sext_inreg (C)
+def shl_ashr_to_sext_inreg_matchinfo : GIDefMatchData<"std::tuple<Register, int64_t>">;
+def shl_ashr_to_sext_inreg : GICombineRule<
+  (defs root:$root, shl_ashr_to_sext_inreg_matchinfo:$info),
+  (match (wip_match_opcode G_ASHR): $root,
+    [{ return Helper.matchAshrShlToSextInreg(*${root}, ${info}); }]),
+  (apply [{ return Helper.applyAshShlToSextInreg(*${root}, ${info});}])
+>;
+
 // FIXME: These should use the custom predicate feature once it lands.
 def undef_combines : GICombineGroup<[undef_to_fp_zero, undef_to_int_zero,
                                      undef_to_negative_one,
                                      propagate_undef_any_op,
                                      propagate_undef_all_ops,
                                      propagate_undef_shuffle_mask,
                                      erase_undef_store]>;
 
 def identity_combines : GICombineGroup<[select_same_val, right_identity_zero,
                                         binop_same_val, binop_left_to_zero,
                                         binop_right_to_zero, p2i_to_i2p,
                                         i2p_to_p2i]>;
 
 def trivial_combines : GICombineGroup<[copy_prop, mul_to_shl]>;
 def all_combines : GICombineGroup<[trivial_combines, ptr_add_immed_chain,
     combines_for_extload, combine_indexed_load_store, undef_combines,
     identity_combines, simplify_add_to_sub,
-    hoist_logic_op_with_same_opcode_hands]>;
+    hoist_logic_op_with_same_opcode_hands,
+    shl_ashr_to_sext_inreg]>;

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

for (auto &InstrToBuild : MatchInfo.InstrsToBuild) {
     MachineInstrBuilder Instr = Builder.buildInstr(InstrToBuild.Opcode);
     for (auto &OperandFn : InstrToBuild.OperandFns)
       OperandFn(Instr);
   }
   MI.eraseFromParent();
   return true;
 }
 
+bool CombinerHelper::matchAshrShlToSextInreg(
+    MachineInstr &MI, std::tuple<Register, int64_t> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_ASHR);
+  int64_t ShlCst, AshrCst;
+  Register Src;
+  // FIXME: detect splat constant vectors.
+  if (!mi_match(MI.getOperand(0).getReg(), MRI,
+                m_GAShr(m_GShl(m_Reg(Src), m_ICst(ShlCst)), m_ICst(AshrCst))))
+    return false;
+  if (ShlCst != AshrCst)
+    return false;
+  if (!isLegalOrBeforeLegalizer(
+          {TargetOpcode::G_SEXT_INREG, {MRI.getType(Src)}}))
+    return false;
+  MatchInfo = {Src, ShlCst};
+  return true;
+}
+bool CombinerHelper::applyAshShlToSextInreg(
+    MachineInstr &MI, std::tuple<Register, int64_t> &MatchInfo) {
+  assert(MI.getOpcode() == TargetOpcode::G_ASHR);
+  Register Src;
+  int64_t ShiftAmt;
+  std::tie(Src, ShiftAmt) = MatchInfo;
+  unsigned Size = MRI.getType(Src).getScalarSizeInBits();
+  Builder.setInstrAndDebugLoc(MI);
+  Builder.buildSExtInReg(MI.getOperand(0).getReg(), Src, Size - ShiftAmt);
+  MI.eraseFromParent();
+  return true;
+}
+
 bool CombinerHelper::tryCombine(MachineInstr &MI) {
   if (tryCombineCopy(MI))
     return true;
   if (tryCombineExtendingLoads(MI))
     return true;
   if (tryCombineIndexedLoadStore(MI))
     return true;
   return false;
 }

llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-ashr-shl-to-sext-inreg.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64 -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs %s -o - | FileCheck %s
+---
+name:            ashr_shl_to_sext_inreg
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$w0' }
+body:             |
+  bb.1:
+    liveins: $w0
+
+    ; CHECK-LABEL: name: ashr_shl_to_sext_inreg
+    ; CHECK: liveins: $w0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[COPY]](s32)
+    ; CHECK: [[SEXT_INREG:%[0-9]+]]:_(s16) = G_SEXT_INREG [[TRUNC]], 8
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[SEXT_INREG]](s16)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %1:_(s32) = COPY $w0
+    %0:_(s16) = G_TRUNC %1(s32)
+    %2:_(s16) = G_CONSTANT i16 8
+    %3:_(s16) = G_SHL %0, %2(s16)
+    %4:_(s16) = exact G_ASHR %3, %2(s16)
+    %5:_(s32) = G_ANYEXT %4(s16)
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            different_shift_amts
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$w0' }
+body:             |
+  bb.1:
+    liveins: $w0
+
+    ; CHECK-LABEL: name: different_shift_amts
+    ; CHECK: liveins: $w0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[COPY]](s32)
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 12
+    ; CHECK: [[C1:%[0-9]+]]:_(s16) = G_CONSTANT i16 8
+    ; CHECK: [[SHL:%[0-9]+]]:_(s16) = G_SHL [[TRUNC]], [[C]](s16)
+    ; CHECK: [[ASHR:%[0-9]+]]:_(s16) = exact G_ASHR [[SHL]], [[C1]](s16)
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[ASHR]](s16)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %1:_(s32) = COPY $w0
+    %0:_(s16) = G_TRUNC %1(s32)
+    %2:_(s16) = G_CONSTANT i16 12
+    %4:_(s16) = G_CONSTANT i16 8
+    %3:_(s16) = G_SHL %0, %2(s16)
+    %5:_(s16) = exact G_ASHR %3, %4(s16)
+    %6:_(s32) = G_ANYEXT %5(s16)
+    $w0 = COPY %6(s32)
+    RET_ReallyLR implicit $w0
+
+...
+---
+name:            ashr_shl_to_sext_inreg_vector
+alignment:       4
+tracksRegLiveness: true
+liveins:
+  - { reg: '$d0' }
+body:             |
+  bb.1:
+    liveins: $d0
+    ; Currently don't support this for vectors just yet, this will need updating
+    ; when we do.
+    ; CHECK-LABEL: name: ashr_shl_to_sext_inreg_vector
+    ; CHECK: liveins: $d0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $d0
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 8
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s16>) = G_BUILD_VECTOR [[C]](s16), [[C]](s16), [[C]](s16), [[C]](s16)
+    ; CHECK: [[SHL:%[0-9]+]]:_(<4 x s16>) = G_SHL [[COPY]], [[BUILD_VECTOR]](<4 x s16>)
+    ; CHECK: [[ASHR:%[0-9]+]]:_(<4 x s16>) = exact G_ASHR [[SHL]], [[BUILD_VECTOR]](<4 x s16>)
+    ; CHECK: $d0 = COPY [[ASHR]](<4 x s16>)
+    ; CHECK: RET_ReallyLR implicit $d0
+    %0:_(<4 x s16>) = COPY $d0
+    %2:_(s16) = G_CONSTANT i16 8
+    %1:_(<4 x s16>) = G_BUILD_VECTOR %2(s16), %2(s16), %2(s16), %2(s16)
+    %3:_(<4 x s16>) = G_SHL %0, %1(<4 x s16>)
+    %4:_(<4 x s16>) = exact G_ASHR %3, %1(<4 x s16>)
+    $d0 = COPY %4(<4 x s16>)
+    RET_ReallyLR implicit $d0
+...

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.sbfe.ll

 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x180000
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 8
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 8
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x180000
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %x = load i32, i32 addrspace(1)* %in, align 4
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %x, i32 0, i32 24)
   %shl = shl i32 %bfe, 8
   %ashr = ashr i32 %shl, 8
   store i32 %ashr, i32 addrspace(1)* %out, align 4

 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dword s2, s[0:1], 0xb
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0xc
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_add_i32 s2, s2, s0
 ; GFX6-NEXT:    s_bfe_i32 s0, s2, 0x80000
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 24
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 24
+; GFX6-NEXT:    s_sext_i32_i8 s0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %c = add i32 %a, %b ; add to prevent folding into extload
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %c, i32 0, i32 8)
   %shl = shl i32 %bfe, 24
   %ashr = ashr i32 %shl, 24
   store i32 %ashr, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sext_in_reg_i8_to_i32_bfe_wrong(i32 addrspace(1)* %out, i32 %a, i32 %b) #0 {
 ; GFX6-LABEL: sext_in_reg_i8_to_i32_bfe_wrong:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dword s2, s[0:1], 0xb
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0xc
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_add_i32 s2, s2, s0
 ; GFX6-NEXT:    s_bfe_i32 s0, s2, 8
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 24
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 24
+; GFX6-NEXT:    s_sext_i32_i8 s0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %c = add i32 %a, %b ; add to prevent folding into extload
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %c, i32 8, i32 0)
   %shl = shl i32 %bfe, 24
   %ashr = ashr i32 %shl, 24
   store i32 %ashr, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sextload_i8_to_i32_bfe(i32 addrspace(1)* %out, i8 addrspace(1)* %ptr) #0 {
 ; GFX6-LABEL: sextload_i8_to_i32_bfe:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    buffer_load_sbyte v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 8
-; GFX6-NEXT:    v_lshlrev_b32_e32 v0, 24, v0
-; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 24, v0
+; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 8
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %load = load i8, i8 addrspace(1)* %ptr, align 1
   %sext = sext i8 %load to i32
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %sext, i32 0, i32 8)
   %shl = shl i32 %bfe, 24
   %ashr = ashr i32 %shl, 24
   store i32 %ashr, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sextload_i8_to_i32_bfe_0(i32 addrspace(1)* %out, i8 addrspace(1)* %ptr) #0 {
 ; GFX6-LABEL: sextload_i8_to_i32_bfe_0:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    buffer_load_sbyte v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    v_bfe_i32 v0, v0, 8, 0
-; GFX6-NEXT:    v_lshlrev_b32_e32 v0, 24, v0
-; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 24, v0
+; GFX6-NEXT:    v_bfe_i32 v0, v0, 0, 8
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %load = load i8, i8 addrspace(1)* %ptr, align 1
   %sext = sext i8 %load to i32
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %sext, i32 8, i32 0)
   %shl = shl i32 %bfe, 24
   %ashr = ashr i32 %shl, 24
   store i32 %ashr, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sext_in_reg_i1_bfe_offset_0(i32 addrspace(1)* %out, i32 addrspace(1)* %in) #0 {
 ; GFX6-LABEL: sext_in_reg_i1_bfe_offset_0:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 31
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 31
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x10000
 ; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x10000
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %x = load i32, i32 addrspace(1)* %in, align 4
   %shl = shl i32 %x, 31
   %shr = ashr i32 %shl, 31
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %shr, i32 0, i32 1)
   store i32 %bfe, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sext_in_reg_i1_bfe_offset_1(i32 addrspace(1)* %out, i32 addrspace(1)* %in) #0 {
 ; GFX6-LABEL: sext_in_reg_i1_bfe_offset_1:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 30
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 30
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x20000
 ; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x10001
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %x = load i32, i32 addrspace(1)* %in, align 4
   %shl = shl i32 %x, 30
   %shr = ashr i32 %shl, 30
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %shr, i32 1, i32 1)
   store i32 %bfe, i32 addrspace(1)* %out, align 4
   ret void
 }
 
 define amdgpu_kernel void @sext_in_reg_i2_bfe_offset_1(i32 addrspace(1)* %out, i32 addrspace(1)* %in) #0 {
 ; GFX6-LABEL: sext_in_reg_i2_bfe_offset_1:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 30
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 30
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x20000
 ; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x20001
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %x = load i32, i32 addrspace(1)* %in, align 4
   %shl = shl i32 %x, 30
   %shr = ashr i32 %shl, 30
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %shr, i32 1, i32 2)

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.ubfe.ll

 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:    s_lshl_b32 s0, s0, 31
-; GFX6-NEXT:    s_ashr_i32 s0, s0, 31
+; GFX6-NEXT:    s_bfe_i32 s0, s0, 0x10000
 ; GFX6-NEXT:    s_bfe_u32 s0, s0, 0x10000
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
   %x = load i32, i32 addrspace(1)* %in, align 4
   %shl = shl i32 %x, 31
   %shr = ashr i32 %shl, 31
   %bfe = call i32 @llvm.amdgcn.ubfe.i32(i32 %shr, i32 0, i32 1)

 
 ; Make sure that SimplifyDemandedBits doesn't cause the and to be
 ; reduced to the bits demanded by the bfe.
 
 ; XXX: The operand to v_bfe_u32 could also just directly be the load register.
 define amdgpu_kernel void @simplify_bfe_u32_multi_use_arg(i32 addrspace(1)* %out0,
 ; GFX6-LABEL: simplify_bfe_u32_multi_use_arg:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:        s_load_dwordx2 s[4:5], s[0:1], 0x9
-; GFX6-NEXT:        s_load_dwordx2 s[8:9], s[0:1], 0xb
-; GFX6-NEXT:        s_load_dwordx2 s[0:1], s[0:1], 0xd
-; GFX6-NEXT:        s_mov_b32 s6, -1
-; GFX6-NEXT:        s_mov_b32 s7, 0xf000
-; GFX6-NEXT:        s_mov_b64 s[10:11], s[6:7]
-; GFX6-NEXT:        s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:        s_load_dword s0, s[0:1], 0x0
-; GFX6-NEXT:        s_waitcnt lgkmcnt(0)
-; GFX6-NEXT:        s_and_b32 s0, s0, 63
-; GFX6-NEXT:        s_bfe_u32 s1, s0, 0x20002
-; GFX6-NEXT:        v_mov_b32_e32 v1, s1
-; GFX6-NEXT:        v_mov_b32_e32 v0, s0
-; GFX6-NEXT:        buffer_store_dword v1, off, s[4:7], 0
-; GFX6-NEXT:        buffer_store_dword v0, off, s[8:11], 0
-; GFX6-NEXT:        s_endpgm
+; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
+; GFX6-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0xb
+; GFX6-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; GFX6-NEXT:    s_mov_b32 s6, -1
+; GFX6-NEXT:    s_mov_b32 s7, 0xf000
+; GFX6-NEXT:    s_mov_b64 s[10:11], s[6:7]
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    s_load_dword s0, s[0:1], 0x0
+; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX6-NEXT:    s_and_b32 s0, s0, 63
+; GFX6-NEXT:    s_bfe_u32 s1, s0, 0x20002
+; GFX6-NEXT:    v_mov_b32_e32 v1, s1
+; GFX6-NEXT:    v_mov_b32_e32 v0, s0
+; GFX6-NEXT:    buffer_store_dword v1, off, s[4:7], 0
+; GFX6-NEXT:    buffer_store_dword v0, off, s[8:11], 0
+; GFX6-NEXT:    s_endpgm
                                             i32 addrspace(1)* %out1,
                                             i32 addrspace(1)* %in) #0 {
   %src = load i32, i32 addrspace(1)* %in, align 4
   %and = and i32 %src, 63
   %bfe_u32 = call i32 @llvm.amdgcn.ubfe.i32(i32 %and, i32 2, i32 2)
   store i32 %bfe_u32, i32 addrspace(1)* %out0, align 4
   store i32 %and, i32 addrspace(1)* %out1, align 4
   ret void

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; CGP-NEXT:    v_or_b32_e32 v0, 1, v0
 ; CGP-NEXT:    v_mul_f32_e32 v1, v3, v4
 ; CGP-NEXT:    v_trunc_f32_e32 v1, v1
 ; CGP-NEXT:    v_mad_f32 v3, -v1, v2, v3
 ; CGP-NEXT:    v_cvt_i32_f32_e32 v1, v1
 ; CGP-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
-; CGP-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; CGP-NEXT:    v_ashrrev_i32_e32 v0, 7, v0
+; CGP-NEXT:    v_bfe_i32 v0, v0, 0, 25
 ; CGP-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and i64 %num, 16777215
   %den.mask = and i64 %den, 16777215
   %result = sdiv i64 %num.mask, %den.mask
   ret i64 %result
 }
 

 ; CGP-NEXT:    v_trunc_f32_e32 v4, v4
 ; CGP-NEXT:    v_mad_f32 v5, -v4, v3, v5
 ; CGP-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; CGP-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
 ; CGP-NEXT:    v_or_b32_e32 v2, 1, v2
 ; CGP-NEXT:    v_cmp_ge_f32_e64 vcc, |v5|, |v3|
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, 0, v2, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
-; CGP-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; CGP-NEXT:    v_lshlrev_b32_e32 v2, 7, v2
-; CGP-NEXT:    v_ashrrev_i32_e32 v0, 7, v0
-; CGP-NEXT:    v_ashrrev_i32_e32 v2, 7, v2
+; CGP-NEXT:    v_bfe_i32 v0, v0, 0, 25
+; CGP-NEXT:    v_bfe_i32 v2, v2, 0, 25
 ; CGP-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; CGP-NEXT:    v_ashrrev_i32_e32 v3, 31, v2
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and <2 x i64> %num, <i64 16777215, i64 16777215>
   %den.mask = and <2 x i64> %den, <i64 16777215, i64 16777215>
   %result = sdiv <2 x i64> %num.mask, %den.mask
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; CGP-NEXT:    v_trunc_f32_e32 v4, v4
 ; CGP-NEXT:    v_mad_f32 v3, -v4, v2, v3
 ; CGP-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, 0, v5, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v1, v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
-; CGP-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
-; CGP-NEXT:    v_ashrrev_i32_e32 v0, 7, v0
+; CGP-NEXT:    v_bfe_i32 v0, v0, 0, 25
 ; CGP-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and i64 %num, 16777215
   %den.mask = and i64 %den, 16777215
   %result = srem i64 %num.mask, %den.mask
   ret i64 %result
 }
 

 ; CGP-NEXT:    v_and_b32_e32 v3, s4, v6
 ; CGP-NEXT:    v_cvt_f32_i32_e32 v4, v3
 ; CGP-NEXT:    v_xor_b32_e32 v6, v2, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_cvt_f32_i32_e32 v1, v2
 ; CGP-NEXT:    v_rcp_f32_e32 v5, v4
 ; CGP-NEXT:    v_ashrrev_i32_e32 v6, 30, v6
 ; CGP-NEXT:    v_or_b32_e32 v6, 1, v6
-; CGP-NEXT:    v_lshlrev_b32_e32 v0, 7, v0
+; CGP-NEXT:    v_bfe_i32 v0, v0, 0, 25
 ; CGP-NEXT:    v_mul_f32_e32 v5, v1, v5
 ; CGP-NEXT:    v_trunc_f32_e32 v5, v5
 ; CGP-NEXT:    v_mad_f32 v1, -v5, v4, v1
 ; CGP-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; CGP-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v4|
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, 0, v6, vcc
-; CGP-NEXT:    v_ashrrev_i32_e32 v0, 7, v0
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v5, v1
 ; CGP-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; CGP-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v3
-; CGP-NEXT:    v_lshlrev_b32_e32 v2, 7, v2
-; CGP-NEXT:    v_ashrrev_i32_e32 v2, 7, v2
+; CGP-NEXT:    v_bfe_i32 v2, v2, 0, 25
 ; CGP-NEXT:    v_ashrrev_i32_e32 v3, 31, v2
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %num.mask = and <2 x i64> %num, <i64 16777215, i64 16777215>
   %den.mask = and <2 x i64> %den, <i64 16777215, i64 16777215>
   %result = srem <2 x i64> %num.mask, %den.mask
   ret <2 x i64> %result
 }