[X86] Disable BMI BEXTR in X86DAGToDAGISel::matchBEXTRFromAnd unless we're on compiling for a CPU with single uop BEXTR

[craig.topper]

Summary

This function turns (X >> C1) & C2 into a BMI BEXTR or TBM BEXTRI instruction. For BMI BEXTR we have to materialize an immediate into a register to feed to the BEXTR instruction.

The BMI BEXTR instruction is 2 uops on Intel CPUs. It looks like on SKL its one port 0/6 uop and one port 1/5 uop. Despite what Agner's tables say. I know one of the uops is a regular shift uop so it would have to go through the port 0/6 shifter unit. So that's the same or worse execution wise than the shift+and which is one 0/6 uop and one 0/1/5/6 uop. The move immediate into register is an additional 0/1/5/6 uop.

For now I've limited this transform to AMD CPUs which have a single uop BEXTR. If may also might make sense if we can fold a load or if the and immediate is larger than 32-bits and can't be encoded as a sign extended 32-bit value or if LICM or CSE can hoist the move immediate and share it. But we'd need to look more carefully at that. In the regression I looked at it doesn't look load folding or large immediates were occurring so the regression isn't caused by the loss of those. So we could try to be smarter here if we find a compelling case.

[RKSimon]

The code that led to PR38938 saw a perf improvement by moving to BEXTR on btver2 - both from the load-folding and hoisting the control constant out of the loop.

I'd really like to avoid yet another fast/slow feature flag but there might not be a better way right now if you want to get this in - @andreadb any thoughts?

https://godbolt.org/z/zdktz_

[craig.topper]

Forgot to mention that BEXTR breaks the two addressness of the shift+and pattern to avoid a copy which can also be beneficial. Unfortunately isel can't generally determine that a copy will be needed.

I agree, I don't really want to add a new fast/slow flag either.

[lebedev.ri]

Can we come up with a slightly better logic rather than just haveTBM()?
Else, i'm not sure how to fix D52293 / PR38938.
Can we perhaps also allow cases when we are shifting a load?

lib/Target/X86/X86ISelDAGToDAG.cpp

2598

I think we are also missing a check that NVT is not i64 if we are in 32-bit mode.

[andreadb]

In D52570#1247140, @craig.topper wrote:

Forgot to mention that BEXTR breaks the two addressness of the shift+and pattern to avoid a copy which can also be beneficial. Unfortunately isel can't generally determine that a copy will be needed.

I agree, I don't really want to add a new fast/slow flag either.

As Simon already wrote, BEXTR is very fast on AMD.
The throughput reported by llvm-mca matches what I see with perf.

What if at ISel stage we just select BEXTR, and then we have a later (peephole) pass that expands it based on the subtarget and properties of the machine instruction?

Basically, for BEXTR we could do something similar to what is done for LEA. We could have a "fixup" pass (or custom patterns matched by the MachineCombiner) to expand BEXTR when the shift-and sequence is more convenient. This would still require a subtarget target hook.

If the decision only depends on the subtarget, and (maybe) properties of the instructions, then we could have tablegen automatically expand a TargetSubtarget hook for us.

Something like this (just an example...):

Index: lib/Target/X86/X86SchedPredicates.td
===================================================================
--- lib/Target/X86/X86SchedPredicates.td        (revision 343198)
+++ lib/Target/X86/X86SchedPredicates.td        (working copy)
@@ -54,3 +54,5 @@
// X86GenInstrInfo.
def IsThreeOperandsLEAFn :
    TIIPredicate<"isThreeOperandsLEA", IsThreeOperandsLEABody>;
+
+def ShouldExpandBEXTRDecl : STIPredicateDecl<"shouldExpandBEXTR", TruePred, /* overrides */ 0, /* expandForMC */ 0>;
Index: lib/Target/X86/X86ScheduleBtVer2.td
===================================================================
--- lib/Target/X86/X86ScheduleBtVer2.td (revision 343198)
+++ lib/Target/X86/X86ScheduleBtVer2.td (working copy)
@@ -772,4 +772,9 @@
  ], ZeroIdiomPredicate>
]>;

+def : STIPredicate<
+  ShouldExpandBEXTRDecl,
+  [ InstructionEquivalenceClass<[BEXTR32rr, BEXTR64rr], FalsePred> ]
+>;
+
} // SchedModel

..that generates this:

bool X86GenSubtargetInfo::shouldExpandBEXTR(const MachineInstr *MI) const {
  unsigned ProcessorID = getSchedModel().getProcessorID();
  switch(MI->getOpcode()) {
  default:
    break;
  case X86::BEXTR32rr:
  case X86::BEXTR64rr:
    if (ProcessorID == 4) {
      return false;
    }
    break;
  }

  return true;
} // X86GenSubtargetInfo::shouldExpandBEXTR

As I wrote. This is just an idea...

Not sure if that helps.
-Andrea

[RKSimon]

The time for X86InstrInfo::getMachineCombinerPatterns might be upon us.....

[craig.topper]

In D52570#1247523, @lebedev.ri wrote:

Can we come up with a slightly better logic rather than just haveTBM()?
Else, i'm not sure how to fix D52293 / PR38938.
Can we perhaps also allow cases when we are shifting a load?

Folding a load isn't clearly a perf win here either. Doesn't look like load-op fusion occurs on Intel CPUs so BEXTR with load sends 3 uops out of the frontend. So that's the same number of uops at load+shift+and. If if I've added up the sizes correctly, I think the move immediate + bextr with load is one byte longer than load+shift+and at least for 32-bit. 64-bit would incur a rex prefix on the shift. But we'd probably still have a 32-bit and. So that's probably the same size.

lib/Target/X86/X86ISelDAGToDAG.cpp

2598

We're so far past type legalization here that i64 isn't possible on a 32-bit target.

[RKSimon]

In the short term I think a fast feature flag (for bdver/btver/znver amd targets) is the way to go - this will solve your immediate regression and give us time to get a proper scheduler model driven solution in place (I have my eye on BEXTR, SHLD/SHRD and LEA feature flags to be replaced by this).

[lebedev.ri]

In D52570#1248811, @RKSimon wrote:

In the short term I think a fast feature flag (for bdver/btver/znver amd targets) is the way to go

Hm, but those *are* all the AMD targets that support BMI.
Perhaps this can be simplified down to isAMD() ?

[spatel]

In D52570#1248819, @lebedev.ri wrote:

In D52570#1248811, @RKSimon wrote:

In the short term I think a fast feature flag (for bdver/btver/znver amd targets) is the way to go

Hm, but those *are* all the AMD targets that support BMI.
Perhaps this can be simplified down to isAMD() ?

No - it is never a good idea to map a uarch feature/bug onto a model/vendor predicate. It usually becomes an inaccurate description at some point because companies and chips change. We've experienced this problem in x86 already with "isAtom()".

[RKSimon]

In D52570#1248819, @lebedev.ri wrote:

In D52570#1248811, @RKSimon wrote:

In the short term I think a fast feature flag (for bdver/btver/znver amd targets) is the way to go

Hm, but those *are* all the AMD targets that support BMI.
Perhaps this can be simplified down to isAMD() ?

Please don't - we need to be getting rid of X86ProcFamilyEnum not adding yet more complications to it - handling all those enums is even worse than feature flags.

[craig.topper]

Add feature flag and put it on AMD CPUs.

I removed some of the unused check prefixes from bmi.ll so I didn't have to add even more variations of them.

[lebedev.ri]

So the target feature, was the way forward, great!

test/CodeGen/X86/extract-bits.ll

1–11

Can you please add +fast-bextr to all these?

[RKSimon]

<pedantic>This phab's title and summary don't match the contents of the patch any more

[craig.topper]

Add +fast-bextr to extract-bits.ll

[lebedev.ri]

Looks good.

[RKSimon]

LGTM with one minor cleanup of the CHECKS in bmi.ll

test/CodeGen/X86/bmi.ll

447

cleanup?

lib/Target/X86/X86.td

: SubtargetFeature<
           "feature.", [FeatureRetpolineIndirectCalls]>;
 
 // Direct Move instructions.
 def FeatureMOVDIRI  : SubtargetFeature<"movdiri", "HasMOVDIRI", "true",
                                        "Support movdiri instruction">;
 def FeatureMOVDIR64B : SubtargetFeature<"movdir64b", "HasMOVDIR64B", "true",
                                         "Support movdir64b instruction">;
 
+def FeatureFastBEXTR : SubtargetFeature<"fast-bextr", "HasFastBEXTR", "true",
+          "Indicates that the BEXTR instruction is implemented as a single uop "
+          "with good throughput.">;
+
 //===----------------------------------------------------------------------===//
 // Register File Description
 //===----------------------------------------------------------------------===//
 
 include "X86RegisterInfo.td"
 include "X86RegisterBanks.td"
 
 //===----------------------------------------------------------------------===//
def : ProcessorModel<"btver2", BtVer2Model, [
   FeatureLZCNT,
   FeatureFastLZCNT,
   FeaturePOPCNT,
   FeatureXSAVE,
   FeatureXSAVEOPT,
   FeatureSlowSHLD,
   FeatureLAHFSAHF,
   FeatureFast15ByteNOP,
+  FeatureFastBEXTR,
   FeatureFastPartialYMMorZMMWrite
 ]>;
 
 // Bulldozer
 def : Proc<"bdver1", [
   FeatureX87,
   FeatureCMOV,
   FeatureXOP,
def : Proc<"bdver2", [
   FeatureXSAVE,
   FeatureBMI,
   FeatureTBM,
   FeatureLWP,
   FeatureFMA,
   FeatureSlowSHLD,
   FeatureLAHFSAHF,
   FeatureFast11ByteNOP,
+  FeatureFastBEXTR,
   FeatureMacroFusion
 ]>;
 
 // Steamroller
 def : Proc<"bdver3", [
   FeatureX87,
   FeatureCMOV,
   FeatureXOP,
def : Proc<"bdver3", [
   FeatureTBM,
   FeatureLWP,
   FeatureFMA,
   FeatureXSAVEOPT,
   FeatureSlowSHLD,
   FeatureFSGSBase,
   FeatureLAHFSAHF,
   FeatureFast11ByteNOP,
+  FeatureFastBEXTR,
   FeatureMacroFusion
 ]>;
 
 // Excavator
 def : Proc<"bdver4", [
   FeatureX87,
   FeatureCMOV,
   FeatureMMX,
def : Proc<"bdver4", [
   FeatureBMI2,
   FeatureTBM,
   FeatureLWP,
   FeatureFMA,
   FeatureXSAVEOPT,
   FeatureSlowSHLD,
   FeatureFSGSBase,
   FeatureLAHFSAHF,
+  FeatureFastBEXTR,
   FeatureFast11ByteNOP,
   FeatureMWAITX,
   FeatureMacroFusion
 ]>;
 
 // Znver1
 def: ProcessorModel<"znver1", Znver1Model, [
   FeatureADX,
def: ProcessorModel<"znver1", Znver1Model, [
   FeatureF16C,
   FeatureFMA,
   FeatureFSGSBase,
   FeatureFXSR,
   FeatureNOPL,
   FeatureFastLZCNT,
   FeatureLAHFSAHF,
   FeatureLZCNT,
+  FeatureFastBEXTR,
   FeatureFast15ByteNOP,
   FeatureMacroFusion,
   FeatureMMX,
   FeatureMOVBE,
   FeatureMWAITX,
   FeaturePCLMUL,
   FeaturePOPCNT,
   FeaturePRFCHW,

lib/Target/X86/X86ISelDAGToDAG.cpp

 // See if this is an (X >> C1) & C2 that we can match to BEXTR/BEXTRI.
 bool X86DAGToDAGISel::matchBEXTRFromAnd(SDNode *Node) {
   MVT NVT = Node->getSimpleValueType(0);
   SDLoc dl(Node);
 
   SDValue N0 = Node->getOperand(0);
   SDValue N1 = Node->getOperand(1);
 
-  if (!Subtarget->hasBMI() && !Subtarget->hasTBM())
+  // If we have TBM we can use an immediate for the control. If we have BMI
+  // we should only do this if the BEXTR instruction is implemented well.
+  // Otherwise moving the control into a register makes this more costly.
+  // TODO: Maybe load folding, greater than 32-bit masks, or a guarantee of LICM
+  // hoisting the move immediate would make it worthwhile with a less optimal
+  // BEXTR?
+  if (!Subtarget->hasTBM() &&
+      !(Subtarget->hasBMI() && Subtarget->hasFastBEXTR()))
     return false;
 
   // Must have a shift right.
   if (N0->getOpcode() != ISD::SRL && N0->getOpcode() != ISD::SRA)
     return false;
 
   // Shift can't have additional users.
   if (!N0->hasOneUse())

lib/Target/X86/X86Subtarget.h

protected:
   bool HasRDPID = false;
 
   /// Processor supports WaitPKG instructions
   bool HasWAITPKG = false;
 
   /// Processor supports PCONFIG instruction
   bool HasPCONFIG = false;
 
+  /// Processor has a single uop BEXTR implementation.
+  bool HasFastBEXTR = false;
+
   /// Use a retpoline thunk rather than indirect calls to block speculative
   /// execution.
   bool UseRetpolineIndirectCalls = false;
 
   /// Use a retpoline thunk or remove any indirect branch to block speculative
   /// execution.
   bool UseRetpolineIndirectBranches = false;
 
public:
   bool hasFastPartialYMMorZMMWrite() const {
     return HasFastPartialYMMorZMMWrite;
   }
   bool hasFastGather() const { return HasFastGather; }
   bool hasFastScalarFSQRT() const { return HasFastScalarFSQRT; }
   bool hasFastVectorFSQRT() const { return HasFastVectorFSQRT; }
   bool hasFastLZCNT() const { return HasFastLZCNT; }
   bool hasFastSHLDRotate() const { return HasFastSHLDRotate; }
+  bool hasFastBEXTR() const { return HasFastBEXTR; }
   bool hasMacroFusion() const { return HasMacroFusion; }
   bool hasERMSB() const { return HasERMSB; }
   bool hasSlowDivide32() const { return HasSlowDivide32; }
   bool hasSlowDivide64() const { return HasSlowDivide64; }
   bool padShortFunctions() const { return PadShortFunctions; }
   bool slowTwoMemOps() const { return SlowTwoMemOps; }
   bool LEAusesAG() const { return LEAUsesAG; }
   bool slowLEA() const { return SlowLEA; }

test/CodeGen/X86/bmi-x86_64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,BMI1
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,BMI2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,BEXTR-SLOW,BMI1,BMI1-SLOW
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,BEXTR-SLOW,BMI2,BMI2-SLOW
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+fast-bextr | FileCheck %s --check-prefixes=CHECK,BEXTR-FAST,BMI1,BMI1-FAST
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+bmi2,+fast-bextr | FileCheck %s --check-prefixes=CHECK,BEXTR-FAST,BMI2,BMI2-FAST
 
 declare i64 @llvm.x86.bmi.bextr.64(i64, i64)
 
 define i64 @bextr64(i64 %x, i64 %y)   {
 ; CHECK-LABEL: bextr64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    bextrq %rsi, %rdi, %rax
 ; CHECK-NEXT:    retq
   %tmp = tail call i64 @llvm.x86.bmi.bextr.64(i64 %x, i64 %y)
   ret i64 %tmp
 }
 
 define i64 @bextr64b(i64 %x)  uwtable  ssp {
-; CHECK-LABEL: bextr64b:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl $3076, %eax # imm = 0xC04
-; CHECK-NEXT:    bextrl %eax, %edi, %eax
-; CHECK-NEXT:    retq
+; BEXTR-SLOW-LABEL: bextr64b:
+; BEXTR-SLOW:       # %bb.0:
+; BEXTR-SLOW-NEXT:    movq %rdi, %rax
+; BEXTR-SLOW-NEXT:    shrl $4, %eax
+; BEXTR-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; BEXTR-SLOW-NEXT:    retq
+;
+; BEXTR-FAST-LABEL: bextr64b:
+; BEXTR-FAST:       # %bb.0:
+; BEXTR-FAST-NEXT:    movl $3076, %eax # imm = 0xC04
+; BEXTR-FAST-NEXT:    bextrl %eax, %edi, %eax
+; BEXTR-FAST-NEXT:    retq
   %1 = lshr i64 %x, 4
   %2 = and i64 %1, 4095
   ret i64 %2
 }
 
 ; Make sure we still use the AH subreg trick to extract 15:8
 define i64 @bextr64_subreg(i64 %x)  uwtable  ssp {
 ; CHECK-LABEL: bextr64_subreg:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movq %rdi, %rax
 ; CHECK-NEXT:    movzbl %ah, %eax
 ; CHECK-NEXT:    retq
   %1 = lshr i64 %x, 8
   %2 = and i64 %1, 255
   ret i64 %2
 }
 
 define i64 @bextr64b_load(i64* %x) {
-; CHECK-LABEL: bextr64b_load:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl $3076, %eax # imm = 0xC04
-; CHECK-NEXT:    bextrl %eax, (%rdi), %eax
-; CHECK-NEXT:    retq
+; BEXTR-SLOW-LABEL: bextr64b_load:
+; BEXTR-SLOW:       # %bb.0:
+; BEXTR-SLOW-NEXT:    movl (%rdi), %eax
+; BEXTR-SLOW-NEXT:    shrl $4, %eax
+; BEXTR-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; BEXTR-SLOW-NEXT:    retq
+;
+; BEXTR-FAST-LABEL: bextr64b_load:
+; BEXTR-FAST:       # %bb.0:
+; BEXTR-FAST-NEXT:    movl $3076, %eax # imm = 0xC04
+; BEXTR-FAST-NEXT:    bextrl %eax, (%rdi), %eax
+; BEXTR-FAST-NEXT:    retq
   %1 = load i64, i64* %x, align 8
   %2 = lshr i64 %1, 4
   %3 = and i64 %2, 4095
   ret i64 %3
 }
 
 ; PR34042
 define i64 @bextr64c(i64 %x, i32 %y) {
 ; CHECK-LABEL: bextr64c:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
 ; CHECK-NEXT:    bextrq %rsi, %rdi, %rax
 ; CHECK-NEXT:    retq
   %tmp0 = sext i32 %y to i64
   %tmp1 = tail call i64 @llvm.x86.bmi.bextr.64(i64 %x, i64 %tmp0)
   ret i64 %tmp1
 }
 
 define i64 @bextr64d(i64 %a) {
-; CHECK-LABEL: bextr64d:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl $8450, %eax # imm = 0x2102
-; CHECK-NEXT:    bextrq %rax, %rdi, %rax
-; CHECK-NEXT:    retq
+; BMI1-SLOW-LABEL: bextr64d:
+; BMI1-SLOW:       # %bb.0: # %entry
+; BMI1-SLOW-NEXT:    shrq $2, %rdi
+; BMI1-SLOW-NEXT:    movl $8448, %eax # imm = 0x2100
+; BMI1-SLOW-NEXT:    bextrq %rax, %rdi, %rax
+; BMI1-SLOW-NEXT:    retq
+;
+; BMI2-SLOW-LABEL: bextr64d:
+; BMI2-SLOW:       # %bb.0: # %entry
+; BMI2-SLOW-NEXT:    shrq $2, %rdi
+; BMI2-SLOW-NEXT:    movb $33, %al
+; BMI2-SLOW-NEXT:    bzhiq %rax, %rdi, %rax
+; BMI2-SLOW-NEXT:    retq
+;
+; BEXTR-FAST-LABEL: bextr64d:
+; BEXTR-FAST:       # %bb.0: # %entry
+; BEXTR-FAST-NEXT:    movl $8450, %eax # imm = 0x2102
+; BEXTR-FAST-NEXT:    bextrq %rax, %rdi, %rax
+; BEXTR-FAST-NEXT:    retq
 entry:
   %shr = lshr i64 %a, 2
   %and = and i64 %shr, 8589934591
   ret i64 %and
 }
 
 define i64 @non_bextr64(i64 %x) {
 ; CHECK-LABEL: non_bextr64:

test/CodeGen/X86/bmi.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1
-; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,X86,BMI2,X86-BMI2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,X64,BMI2,X64-BMI2
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,X86,X86-SLOW-BEXTR
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,X86,X86-SLOW-BEXTR
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi | FileCheck %s --check-prefixes=CHECK,X64,X64-SLOW-BEXTR
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+bmi2 | FileCheck %s --check-prefixes=CHECK,X64,X64-SLOW-BEXTR
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+bmi,+fast-bextr | FileCheck %s --check-prefixes=CHECK,X86,X86-FAST-BEXTR
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi,+fast-bextr | FileCheck %s --check-prefixes=CHECK,X64,X64-FAST-BEXTR
 
 define i32 @andn32(i32 %x, i32 %y)   {
 ; X86-LABEL: andn32:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    andnl {{[0-9]+}}(%esp), %eax, %eax
 ; X86-NEXT:    retl
 ;

 ; X64-NEXT:    bextrl %esi, (%rdi), %eax
 ; X64-NEXT:    retq
   %x1 = load i32, i32* %x
   %tmp = tail call i32 @llvm.x86.bmi.bextr.32(i32 %x1, i32 %y)
   ret i32 %tmp
 }
 
 define i32 @bextr32b(i32 %x)  uwtable  ssp {
-; X86-LABEL: bextr32b:
-; X86:       # %bb.0:
-; X86-NEXT:    movl $3076, %eax # imm = 0xC04
-; X86-NEXT:    bextrl %eax, {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    retl
-;
-; X64-LABEL: bextr32b:
-; X64:       # %bb.0:
-; X64-NEXT:    movl $3076, %eax # imm = 0xC04
-; X64-NEXT:    bextrl %eax, %edi, %eax
-; X64-NEXT:    retq
+; X86-BMI1-SLOW-LABEL: bextr32b:
+; X86-BMI1-SLOW:       # %bb.0:
+; X86-BMI1-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-SLOW-NEXT:    shrl $4, %eax
+; X86-BMI1-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X86-BMI1-SLOW-NEXT:    retl
+;
+; X64-BMI1-SLOW-LABEL: bextr32b:
+; X64-BMI1-SLOW:       # %bb.0:
+; X64-BMI1-SLOW-NEXT:    movl %edi, %eax
+; X64-BMI1-SLOW-NEXT:    shrl $4, %eax
+; X64-BMI1-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X64-BMI1-SLOW-NEXT:    retq
+;
+; X86-BMI1-FAST-LABEL: bextr32b:
+; X86-BMI1-FAST:       # %bb.0:
+; X86-BMI1-FAST-NEXT:    movl $3076, %eax # imm = 0xC04
+; X86-BMI1-FAST-NEXT:    bextrl %eax, {{[0-9]+}}(%esp), %eax
+; X86-BMI1-FAST-NEXT:    retl
+;
+; X64-BMI1-FAST-LABEL: bextr32b:
+; X64-BMI1-FAST:       # %bb.0:
+; X64-BMI1-FAST-NEXT:    movl $3076, %eax # imm = 0xC04
+; X64-BMI1-FAST-NEXT:    bextrl %eax, %edi, %eax
+; X64-BMI1-FAST-NEXT:    retq
+; X86-SLOW-BEXTR-LABEL: bextr32b:
+; X86-SLOW-BEXTR:       # %bb.0:
+; X86-SLOW-BEXTR-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-SLOW-BEXTR-NEXT:    shrl $4, %eax
+; X86-SLOW-BEXTR-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X86-SLOW-BEXTR-NEXT:    retl
+;
+; X64-SLOW-BEXTR-LABEL: bextr32b:
+; X64-SLOW-BEXTR:       # %bb.0:
+; X64-SLOW-BEXTR-NEXT:    movl %edi, %eax
+; X64-SLOW-BEXTR-NEXT:    shrl $4, %eax
+; X64-SLOW-BEXTR-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X64-SLOW-BEXTR-NEXT:    retq
+;
+; X86-FAST-BEXTR-LABEL: bextr32b:
+; X86-FAST-BEXTR:       # %bb.0:
+; X86-FAST-BEXTR-NEXT:    movl $3076, %eax # imm = 0xC04
+; X86-FAST-BEXTR-NEXT:    bextrl %eax, {{[0-9]+}}(%esp), %eax
+; X86-FAST-BEXTR-NEXT:    retl
+;
+; X64-FAST-BEXTR-LABEL: bextr32b:
+; X64-FAST-BEXTR:       # %bb.0:
+; X64-FAST-BEXTR-NEXT:    movl $3076, %eax # imm = 0xC04
+; X64-FAST-BEXTR-NEXT:    bextrl %eax, %edi, %eax
+; X64-FAST-BEXTR-NEXT:    retq
   %1 = lshr i32 %x, 4
   %2 = and i32 %1, 4095
   ret i32 %2
 }
 
 ; Make sure we still use AH subreg trick to extract 15:8
 define i32 @bextr32_subreg(i32 %x)  uwtable  ssp {
 ; X86-LABEL: bextr32_subreg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bextr32_subreg:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %edi, %eax
 ; X64-NEXT:    movzbl %ah, %eax
 ; X64-NEXT:    retq
   %1 = lshr i32 %x, 8
   %2 = and i32 %1, 255
   ret i32 %2
 }
 
 define i32 @bextr32b_load(i32* %x)  uwtable  ssp {
-; X86-LABEL: bextr32b_load:
-; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movl $3076, %ecx # imm = 0xC04
-; X86-NEXT:    bextrl %ecx, (%eax), %eax
-; X86-NEXT:    retl
-;
-; X64-LABEL: bextr32b_load:
-; X64:       # %bb.0:
-; X64-NEXT:    movl $3076, %eax # imm = 0xC04
-; X64-NEXT:    bextrl %eax, (%rdi), %eax
-; X64-NEXT:    retq
+; X86-BMI1-SLOW-LABEL: bextr32b_load:
+; X86-BMI1-SLOW:       # %bb.0:
+; X86-BMI1-SLOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-SLOW-NEXT:    movl (%eax), %eax
+; X86-BMI1-SLOW-NEXT:    shrl $4, %eax
+; X86-BMI1-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X86-BMI1-SLOW-NEXT:    retl
+;
+; X64-BMI1-SLOW-LABEL: bextr32b_load:
+; X64-BMI1-SLOW:       # %bb.0:
+; X64-BMI1-SLOW-NEXT:    movl (%rdi), %eax
+; X64-BMI1-SLOW-NEXT:    shrl $4, %eax
+; X64-BMI1-SLOW-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X64-BMI1-SLOW-NEXT:    retq
+;
+; X86-BMI1-FAST-LABEL: bextr32b_load:
+; X86-BMI1-FAST:       # %bb.0:
+; X86-BMI1-FAST-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-BMI1-FAST-NEXT:    movl $3076, %ecx # imm = 0xC04
+; X86-BMI1-FAST-NEXT:    bextrl %ecx, (%eax), %eax
+; X86-BMI1-FAST-NEXT:    retl
+;
+; X64-BMI1-FAST-LABEL: bextr32b_load:
+; X64-BMI1-FAST:       # %bb.0:
+; X64-BMI1-FAST-NEXT:    movl $3076, %eax # imm = 0xC04
+; X64-BMI1-FAST-NEXT:    bextrl %eax, (%rdi), %eax
+; X64-BMI1-FAST-NEXT:    retq
+; X86-SLOW-BEXTR-LABEL: bextr32b_load:
+; X86-SLOW-BEXTR:       # %bb.0:
+; X86-SLOW-BEXTR-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-SLOW-BEXTR-NEXT:    movl (%eax), %eax
+; X86-SLOW-BEXTR-NEXT:    shrl $4, %eax
+; X86-SLOW-BEXTR-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X86-SLOW-BEXTR-NEXT:    retl
+;
+; X64-SLOW-BEXTR-LABEL: bextr32b_load:
+; X64-SLOW-BEXTR:       # %bb.0:
+; X64-SLOW-BEXTR-NEXT:    movl (%rdi), %eax
+; X64-SLOW-BEXTR-NEXT:    shrl $4, %eax
+; X64-SLOW-BEXTR-NEXT:    andl $4095, %eax # imm = 0xFFF
+; X64-SLOW-BEXTR-NEXT:    retq
+;
+; X86-FAST-BEXTR-LABEL: bextr32b_load:
+; X86-FAST-BEXTR:       # %bb.0:
+; X86-FAST-BEXTR-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-FAST-BEXTR-NEXT:    movl $3076, %ecx # imm = 0xC04
+; X86-FAST-BEXTR-NEXT:    bextrl %ecx, (%eax), %eax
+; X86-FAST-BEXTR-NEXT:    retl
+;
+; X64-FAST-BEXTR-LABEL: bextr32b_load:
+; X64-FAST-BEXTR:       # %bb.0:
+; X64-FAST-BEXTR-NEXT:    movl $3076, %eax # imm = 0xC04
+; X64-FAST-BEXTR-NEXT:    bextrl %eax, (%rdi), %eax
+; X64-FAST-BEXTR-NEXT:    retq
   %1 = load i32, i32* %x
   %2 = lshr i32 %1, 4
   %3 = and i32 %2, 4095
   ret i32 %3
 }
 
 ; PR34042
 define i32 @bextr32c(i32 %x, i16 zeroext %y) {

test/CodeGen/X86/extract-bits.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=-bmi,-tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X86,NOBMI,X86-NOBMI
-; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,-tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1NOTBM,X86-BMI1NOTBM
-; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,+tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1TBM,X86-BMI1TBM
-; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,+tbm,+bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1BMI2,X86-BMI1BMI2,BMI1TBM,X86-BMI1TBM,BMI1TBMBMI2,X86-BMI1TBMBMI2
-; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,-tbm,+bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1BMI2,X86-BMI1BMI2,BMI1NOTBMBMI2,X86-BMI1NOTBMBMI2
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=-bmi,-tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X64,NOBMI,X64-NOBMI
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,-tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1NOTBM,X64-BMI1NOTBM
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,+tbm,-bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1TBM,X64-BMI1TBM
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,+tbm,+bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1BMI2,X64-BMI1BMI2,BMI1TBM,X64-BMI1TBM,BMI1TBMBMI2,X64-BMI1TBMBMI2
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,-tbm,+bmi2 < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1BMI2,X64-BMI1BMI2,BMI1NOTBMBMI2,X64-BMI1NOTBMBMI2
+; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=-bmi,-tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X86,NOBMI,X86-NOBMI
+; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,-tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1NOTBM,X86-BMI1NOTBM
+; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,+tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1TBM,X86-BMI1TBM
+; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,+tbm,+bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1BMI2,X86-BMI1BMI2,BMI1TBM,X86-BMI1TBM,BMI1TBMBMI2,X86-BMI1TBMBMI2
+; RUN: llc -mtriple=i686-unknown-linux-gnu -mattr=+bmi,-tbm,+bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X86,BMI1,X86-BMI1,BMI1BMI2,X86-BMI1BMI2,BMI1NOTBMBMI2,X86-BMI1NOTBMBMI2
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=-bmi,-tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X64,NOBMI,X64-NOBMI
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,-tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1NOTBM,X64-BMI1NOTBM
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,+tbm,-bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1TBM,X64-BMI1TBM
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,+tbm,+bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1BMI2,X64-BMI1BMI2,BMI1TBM,X64-BMI1TBM,BMI1TBMBMI2,X64-BMI1TBMBMI2
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+bmi,-tbm,+bmi2,+fast-bextr < %s | FileCheck %s --check-prefixes=CHECK,X64,BMI1,X64-BMI1,BMI1BMI2,X64-BMI1BMI2,BMI1NOTBMBMI2,X64-BMI1NOTBMBMI2
 
 ; *Please* keep in sync with test/CodeGen/AArch64/extract-bits.ll
 
 ; https://bugs.llvm.org/show_bug.cgi?id=36419
 ; https://bugs.llvm.org/show_bug.cgi?id=37603
 ; https://bugs.llvm.org/show_bug.cgi?id=37610
 
 ; Patterns: