This is an archive of the discontinued LLVM Phabricator instance.

Differential D53080

[X86] X86ISelLowering: combineAndIntoBEXTR(): support shifted mask.
AbandonedPublic

Authored by lebedev.ri on Oct 10 2018, 7:28 AM.

Details

Reviewers
RKSimon
craig.topper
spatel
Summary

Continuation of D52426.

As discussed in PR38938,
we fail to emit BEXTR if the mask is shifted.
We can't deal with that in X86DAGToDAGISel before the address mode for the inc is selected,
and we can't really do it in the normal DAGCombine, because we don't have generic ISD::BitFieldExtract node,
and if we simply turn the shifted mask into a normal mask + shift-left, it will be folded back.
So it would seem X86ISelLowering is the place to handle this.

Here, we extend the combine to actually support the pattern with shifted mask.

Diff Detail

Repository
rL LLVM

Event Timeline

lebedev.ri created this revision.Oct 10 2018, 7:28 AM
lebedev.ri added a parent revision: D52426: [X86] Move X86DAGToDAGISel::matchBEXTRFromAnd() into X86ISelLowering.
lebedev.ri added a comment.Oct 10 2018, 11:53 AM

@RKSimon note that only this differential does finally solve the PR38938.

craig.topper added inline comments.Oct 10 2018, 12:05 PM
test/CodeGen/X86/extract-bits.ll
5843

In a placement that isn't register constrained where the "mov %edi, %eax" copy in the original code doesn't need to exist, isn't this code worse?

For example I would expect this to from 3 instructions to 4 with this patch.

define void @foo(i32 %arg, i32* %ptr) {
  %tmp0 = lshr i32 %arg, 19
  %tmp1 = and i32 %tmp0, 1023
  %tmp2 = shl i32 %tmp1, 2
  store i32 %tmp2, i32* %ptr
  ret void
}
lebedev.ri updated this revision to Diff 169087.Oct 10 2018, 1:56 PM

Rebased, more tests.

test/CodeGen/X86/extract-bits.ll
5843

Tests added, but did not analyze them yet.

lebedev.ri added inline comments.Oct 10 2018, 2:08 PM
test/CodeGen/X86/extract-bits.ll
6232

Hm, indeed, these tests degrade.
I don't have any good ideas on how to avoid this.

craig.topper mentioned this in D53126: [X86] Restore X86ISelDAGToDAG::matchBEXTRFromAnd. Teach address matching to create a BEXTR pattern from a (shl (and X, mask >> C1) if C1 can be folded into addressing mode..Oct 10 2018, 11:20 PM
lebedev.ri abandoned this revision.Oct 11 2018, 11:08 AM

Superseeded by D53126.

Diffusion mentioned this in rL344270: [X86] Restore X86ISelDAGToDAG::matchBEXTRFromAnd. Teach address matching to….Oct 11 2018, 11:09 AM

Revision Contents

PathSize
lib/
Target/
X86/
32 lines
test/
CodeGen/
X86/
19 lines
477 lines

Diff 169008

lib/Target/X86/X86ISelLowering.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 35,284 Lines • ▼ Show 20 Linesstatic bool hasBEXTR(const X86Subtarget &Subtarget, EVT VT) {
// TODO: Maybe load folding, greater than 32-bit masks, or a guarantee of LICM // TODO: Maybe load folding, greater than 32-bit masks, or a guarantee of LICM
// hoisting the move immediate would make it worthwhile with a less optimal // hoisting the move immediate would make it worthwhile with a less optimal
// BEXTR? // BEXTR?
if (!Subtarget.hasTBM() && !(Subtarget.hasBMI() && Subtarget.hasFastBEXTR())) if (!Subtarget.hasTBM() && !(Subtarget.hasBMI() && Subtarget.hasFastBEXTR()))
return false; return false;
return (VT == MVT::i32 || (VT == MVT::i64 && Subtarget.is64Bit())); return (VT == MVT::i32 || (VT == MVT::i64 && Subtarget.is64Bit()));
} }
// See if this is an (X >> C1) & C2 that we can match to BEXTR/BEXTRI. // Look for the pattern:
static SDValue combineAndIntoBEXTR(SDNode *Node, SelectionDAG &DAG, // (X l>> C1) & C2 or (X l>> C1) & (C2 << C3)
// Where C2 is all-ones in lowbits, and C3 might be zero,
// and transform it into:
// ((X l>> (C1 + C3)) & C2) << C3
// Which is combined into:
// (bextr X, (((C1 + C3) << 8) | popcnt(C2))) << C3
// The last `<< C3` shift only exists if C3 is not zero.
static SDValue combineShiftAndIntoBEXTR(SDNode *Node, SelectionDAG &DAG,
const X86Subtarget &Subtarget) { const X86Subtarget &Subtarget) {
EVT NVT = Node->getValueType(0); EVT NVT = Node->getValueType(0);
SDLoc dl(Node); SDLoc dl(Node);
SDValue N0 = Node->getOperand(0); SDValue N0 = Node->getOperand(0);
SDValue N1 = Node->getOperand(1); SDValue N1 = Node->getOperand(1);
// Check if subtarget has BEXTR instruction for the node's type // Check if subtarget has BEXTR instruction for the node's type
if (!hasBEXTR(Subtarget, NVT)) if (!hasBEXTR(Subtarget, NVT))
return SDValue(); return SDValue();
// Must have a shift right. // Must have a shift right.
if (N0->getOpcode() != ISD::SRL && N0->getOpcode() != ISD::SRA) if (N0->getOpcode() != ISD::SRL && N0->getOpcode() != ISD::SRA)
return SDValue(); return SDValue();
// Shift can't have additional users. // Shift can't have additional users.
if (!N0->hasOneUse()) if (!N0->hasOneUse())
return SDValue(); return SDValue();
// Shift amount and RHS of and must be constant. // Shift amount and RHS of and must be constant.
ConstantSDNode *MaskCst = dyn_cast<ConstantSDNode>(N1); ConstantSDNode *MaskCst = dyn_cast<ConstantSDNode>(N1);
ConstantSDNode *ShiftCst = dyn_cast<ConstantSDNode>(N0->getOperand(1)); ConstantSDNode *ShiftCst = dyn_cast<ConstantSDNode>(N0->getOperand(1));
if (!MaskCst || !ShiftCst) if (!MaskCst || !ShiftCst)
return SDValue(); return SDValue();
// And RHS must be a mask. // And RHS must be a (potentially shifted) mask.
uint64_t Mask = MaskCst->getZExtValue(); uint64_t Mask = MaskCst->getZExtValue();
if (!isMask_64(Mask)) if (!isShiftedMask_64(Mask))
return SDValue(); return SDValue();
uint64_t Shift = ShiftCst->getZExtValue();
uint64_t MaskSize = countPopulation(Mask); uint64_t MaskSize = countPopulation(Mask);
uint64_t ZeroPaddingSize = countTrailingZeros(Mask);
uint64_t Shift = ZeroPaddingSize + ShiftCst->getZExtValue();
// Don't interfere with something that can be handled by extracting AH. // Don't interfere with something that can be handled by extracting AH.
// TODO: If we are able to fold a load, BEXTR might still be better than AH. // TODO: If we are able to fold a load, BEXTR might still be better than AH.
if (Shift == 8 && MaskSize == 8) if (Shift == 8 && MaskSize == 8 && ZeroPaddingSize == 0)
return SDValue(); return SDValue();
// Make sure we are only using bits that were in the original value, not // Make sure we are only using bits that were in the original value, not
// shifted in. // shifted in.
if (Shift + MaskSize > NVT.getSizeInBits()) if (Shift + MaskSize > NVT.getSizeInBits())
return SDValue(); return SDValue();
// Create a BEXTR node. // Create a BEXTR node.
SDValue C = DAG.getConstant(Shift | (MaskSize << 8), dl, NVT); SDValue C = DAG.getConstant(Shift | (MaskSize << 8), dl, NVT);
SDValue New = DAG.getNode(X86ISD::BEXTR, dl, NVT, N0->getOperand(0), C); SDValue New = DAG.getNode(X86ISD::BEXTR, dl, NVT, N0->getOperand(0), C);
// If the mask had some zero low bits, we need to re-introduce them.
if (ZeroPaddingSize > 0) {
SDValue Cp = DAG.getConstant(ZeroPaddingSize, dl, MVT::i8);
New = DAG.getNode(ISD::SHL, dl, NVT, New, Cp);
}
return New; return New;
} }
// Look for (and (ctpop X), 1) which is the IR form of __builtin_parity. // Look for (and (ctpop X), 1) which is the IR form of __builtin_parity.
// Turn it into series of XORs and a setnp. // Turn it into series of XORs and a setnp.
static SDValue combineParity(SDNode *N, SelectionDAG &DAG, static SDValue combineParity(SDNode *N, SelectionDAG &DAG,
const X86Subtarget &Subtarget) { const X86Subtarget &Subtarget) {
EVT VT = N->getValueType(0); EVT VT = N->getValueType(0);
▲ Show 20 Lines • Show All 88 Lines • ▼ Show 20 Linesstatic SDValue combineAnd(SDNode *N, SelectionDAG &DAG,
// This must be done before legalization has expanded the ctpop. // This must be done before legalization has expanded the ctpop.
if (SDValue V = combineParity(N, DAG, Subtarget)) if (SDValue V = combineParity(N, DAG, Subtarget))
return V; return V;
if (DCI.isBeforeLegalizeOps()) if (DCI.isBeforeLegalizeOps())
return SDValue(); return SDValue();
if (SDValue R = combineAndIntoBEXTR(N, DAG, Subtarget)) if (SDValue R = combineShiftAndIntoBEXTR(N, DAG, Subtarget))
return R; return R;
if (SDValue R = combineCompareEqual(N, DAG, DCI, Subtarget)) if (SDValue R = combineCompareEqual(N, DAG, DCI, Subtarget))
return R; return R;
if (SDValue FPLogic = convertIntLogicToFPLogic(N, DAG, Subtarget)) if (SDValue FPLogic = convertIntLogicToFPLogic(N, DAG, Subtarget))
return FPLogic; return FPLogic;
▲ Show 20 Lines • Show All 6,173 Lines • Show Last 20 Lines

test/CodeGen/X86/bmi-x86_64.ll

Show First 20 Lines • Show All 97 Lines • ▼ Show 20 Lines
; BEXTR-FAST-NEXT: retq ; BEXTR-FAST-NEXT: retq
entry: entry:
%shr = lshr i64 %a, 2 %shr = lshr i64 %a, 2
%and = and i64 %shr, 8589934591 %and = and i64 %shr, 8589934591
ret i64 %and ret i64 %and
} }
define i64 @non_bextr64(i64 %x) { define i64 @non_bextr64(i64 %x) {
; CHECK-LABEL: non_bextr64: ; BEXTR-SLOW-LABEL: non_bextr64:
; CHECK: # %bb.0: # %entry ; BEXTR-SLOW: # %bb.0: # %entry
; CHECK-NEXT: shrq $2, %rdi ; BEXTR-SLOW-NEXT: shrq $2, %rdi
; CHECK-NEXT: movabsq $8589934590, %rax # imm = 0x1FFFFFFFE ; BEXTR-SLOW-NEXT: movabsq $8589934590, %rax # imm = 0x1FFFFFFFE
; CHECK-NEXT: andq %rdi, %rax ; BEXTR-SLOW-NEXT: andq %rdi, %rax
; CHECK-NEXT: retq ; BEXTR-SLOW-NEXT: retq
;
; BEXTR-FAST-LABEL: non_bextr64:
; BEXTR-FAST: # %bb.0: # %entry
; BEXTR-FAST-NEXT: movl $8195, %eax # imm = 0x2003
; BEXTR-FAST-NEXT: bextrq %rax, %rdi, %rax
; BEXTR-FAST-NEXT: addq %rax, %rax
; BEXTR-FAST-NEXT: retq
entry: entry:
%shr = lshr i64 %x, 2 %shr = lshr i64 %x, 2
%and = and i64 %shr, 8589934590 %and = and i64 %shr, 8589934590
ret i64 %and ret i64 %and
} }

test/CodeGen/X86/extract-bits.ll

Show First 20 Lines • Show All 5,609 Lines • ▼ Show 20 Lines
} }
; ---------------------------------------------------------------------------- ; ; ---------------------------------------------------------------------------- ;
; Constant ; Constant
; ---------------------------------------------------------------------------- ; ; ---------------------------------------------------------------------------- ;
; https://bugs.llvm.org/show_bug.cgi?id=38938 ; https://bugs.llvm.org/show_bug.cgi?id=38938
define void @pr38938(i32* %a0, i64* %a1) { define void @pr38938(i32* %a0, i64* %a1) {
; X86-LABEL: pr38938: ; X86-NOBMI-LABEL: pr38938:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: movl (%ecx), %ecx ; X86-NOBMI-NEXT: movl (%ecx), %ecx
; X86-NEXT: shrl $19, %ecx ; X86-NOBMI-NEXT: shrl $19, %ecx
; X86-NEXT: andl $4092, %ecx # imm = 0xFFC ; X86-NOBMI-NEXT: andl $4092, %ecx # imm = 0xFFC
; X86-NEXT: incl (%eax,%ecx) ; X86-NOBMI-NEXT: incl (%eax,%ecx)
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: pr38938: ; X86-BMI1NOTBM-LABEL: pr38938:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movq (%rsi), %rax ; X86-BMI1NOTBM-NEXT: movl {{[0-9]+}}(%esp), %eax
; X64-NEXT: shrq $19, %rax ; X86-BMI1NOTBM-NEXT: movl {{[0-9]+}}(%esp), %ecx
; X64-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-BMI1NOTBM-NEXT: movl $2581, %edx # imm = 0xA15
; X64-NEXT: incl (%rdi,%rax) ; X86-BMI1NOTBM-NEXT: bextrl %edx, (%ecx), %ecx
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: incl (%eax,%ecx,4)
; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: pr38938:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-BMI1TBM-NEXT: movl {{[0-9]+}}(%esp), %ecx
; X86-BMI1TBM-NEXT: bextrl $2581, (%ecx), %ecx # imm = 0xA15
; X86-BMI1TBM-NEXT: incl (%eax,%ecx,4)
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: pr38938:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: movl {{[0-9]+}}(%esp), %ecx
; X86-BMI1NOTBMBMI2-NEXT: movl $2581, %edx # imm = 0xA15
; X86-BMI1NOTBMBMI2-NEXT: bextrl %edx, (%ecx), %ecx
; X86-BMI1NOTBMBMI2-NEXT: incl (%eax,%ecx,4)
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: pr38938:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movq (%rsi), %rax
; X64-NOBMI-NEXT: shrq $19, %rax
; X64-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X64-NOBMI-NEXT: incl (%rdi,%rax)
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: pr38938:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $2581, %eax # imm = 0xA15
; X64-BMI1NOTBM-NEXT: bextrq %rax, (%rsi), %rax
; X64-BMI1NOTBM-NEXT: incl (%rdi,%rax,4)
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: pr38938:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrq $2581, (%rsi), %rax # imm = 0xA15
; X64-BMI1TBM-NEXT: incl (%rdi,%rax,4)
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: pr38938:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $2581, %eax # imm = 0xA15
; X64-BMI1NOTBMBMI2-NEXT: bextrq %rax, (%rsi), %rax
; X64-BMI1NOTBMBMI2-NEXT: incl (%rdi,%rax,4)
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp = load i64, i64* %a1, align 8 %tmp = load i64, i64* %a1, align 8
%tmp1 = lshr i64 %tmp, 21 %tmp1 = lshr i64 %tmp, 21
%tmp2 = and i64 %tmp1, 1023 %tmp2 = and i64 %tmp1, 1023
%tmp3 = getelementptr inbounds i32, i32* %a0, i64 %tmp2 %tmp3 = getelementptr inbounds i32, i32* %a0, i64 %tmp2
%tmp4 = load i32, i32* %tmp3, align 4 %tmp4 = load i32, i32* %tmp3, align 4
%tmp5 = add nsw i32 %tmp4, 1 %tmp5 = add nsw i32 %tmp4, 1
store i32 %tmp5, i32* %tmp3, align 4 store i32 %tmp5, i32* %tmp3, align 4
ret void ret void
▲ Show 20 Lines • Show All 50 Lines • ▼ Show 20 Lines
; X64-BMI1NOTBMBMI2-NEXT: retq ; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i32 %arg, 19 %tmp0 = lshr i32 %arg, 19
%tmp1 = and i32 %tmp0, 1023 %tmp1 = and i32 %tmp0, 1023
ret i32 %tmp1 ret i32 %tmp1
} }
; Should be still fine, but the mask is shifted ; Should be still fine, but the mask is shifted
define i32 @c1_i32(i32 %arg) { define i32 @c1_i32(i32 %arg) {
; X86-LABEL: c1_i32: ; X86-NOBMI-LABEL: c1_i32:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $19, %eax ; X86-NOBMI-NEXT: shrl $19, %eax
; X86-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c1_i32: ; X86-BMI1NOTBM-LABEL: c1_i32:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movl %edi, %eax ; X86-BMI1NOTBM-NEXT: movl $2581, %eax # imm = 0xA15
; X64-NEXT: shrl $19, %eax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-BMI1NOTBM-NEXT: shll $2, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c1_i32:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $2581, {{[0-9]+}}(%esp), %eax # imm = 0xA15
; X86-BMI1TBM-NEXT: shll $2, %eax
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c1_i32:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $2581, %eax # imm = 0xA15
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c1_i32:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movl %edi, %eax
; X64-NOBMI-NEXT: shrl $19, %eax
; X64-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c1_i32:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $2581, %eax # imm = 0xA15
; X64-BMI1NOTBM-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBM-NEXT: shll $2, %eax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c1_i32:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrl $2581, %edi, %eax # imm = 0xA15
; X64-BMI1TBM-NEXT: shll $2, %eax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c1_i32:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $2581, %eax # imm = 0xA15
; X64-BMI1NOTBMBMI2-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i32 %arg, 19 %tmp0 = lshr i32 %arg, 19
%tmp1 = and i32 %tmp0, 4092 %tmp1 = and i32 %tmp0, 4092
ret i32 %tmp1 ret i32 %tmp1
} }
; Should be still fine, but the result is shifted left afterwards ; Should be still fine, but the result is shifted left afterwards
define i32 @c2_i32(i32 %arg) { define i32 @c2_i32(i32 %arg) {
; X86-LABEL: c2_i32: ; X86-NOBMI-LABEL: c2_i32:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $17, %eax ; X86-NOBMI-NEXT: shrl $17, %eax
; X86-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c2_i32: ; X86-BMI1NOTBM-LABEL: c2_i32:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movl %edi, %eax ; X86-BMI1NOTBM-NEXT: movl $2579, %eax # imm = 0xA13
; X64-NEXT: shrl $17, %eax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-BMI1NOTBM-NEXT: shll $2, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c2_i32:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $2579, {{[0-9]+}}(%esp), %eax # imm = 0xA13
; X86-BMI1TBM-NEXT: shll $2, %eax
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c2_i32:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $2579, %eax # imm = 0xA13
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c2_i32:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movl %edi, %eax
; X64-NOBMI-NEXT: shrl $17, %eax
; X64-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c2_i32:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $2579, %eax # imm = 0xA13
craig.topperUnsubmitted
Not Done
ReplyInline Actions

In a placement that isn't register constrained where the "mov %edi, %eax" copy in the original code doesn't need to exist, isn't this code worse?

For example I would expect this to from 3 instructions to 4 with this patch.

define void @foo(i32 %arg, i32* %ptr) {
  %tmp0 = lshr i32 %arg, 19
  %tmp1 = and i32 %tmp0, 1023
  %tmp2 = shl i32 %tmp1, 2
  store i32 %tmp2, i32* %ptr
  ret void
}
craig.topper: In a placement that isn't register constrained where the "mov %edi, %eax" copy in the original…
lebedev.riAuthorUnsubmitted
Not Done
ReplyInline Actions

Tests added, but did not analyze them yet.

lebedev.ri: Tests added, but did not analyze them yet.
; X64-BMI1NOTBM-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBM-NEXT: shll $2, %eax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c2_i32:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrl $2579, %edi, %eax # imm = 0xA13
; X64-BMI1TBM-NEXT: shll $2, %eax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c2_i32:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $2579, %eax # imm = 0xA13
; X64-BMI1NOTBMBMI2-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i32 %arg, 19 %tmp0 = lshr i32 %arg, 19
%tmp1 = and i32 %tmp0, 1023 %tmp1 = and i32 %tmp0, 1023
%tmp2 = shl i32 %tmp1, 2 %tmp2 = shl i32 %tmp1, 2
ret i32 %tmp2 ret i32 %tmp2
} }
; The mask covers newly shifted-in bit ; The mask covers newly shifted-in bit
define i32 @c4_i32_bad(i32 %arg) { define i32 @c4_i32_bad(i32 %arg) {
; X86-LABEL: c4_i32_bad: ; X86-NOBMI-LABEL: c4_i32_bad:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $19, %eax ; X86-NOBMI-NEXT: shrl $19, %eax
; X86-NEXT: andl $-2, %eax ; X86-NOBMI-NEXT: andl $-2, %eax
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c4_i32_bad: ; X86-BMI1NOTBM-LABEL: c4_i32_bad:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movl %edi, %eax ; X86-BMI1NOTBM-NEXT: movl $3092, %eax # imm = 0xC14
; X64-NEXT: shrl $19, %eax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $-2, %eax ; X86-BMI1NOTBM-NEXT: addl %eax, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c4_i32_bad:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $3092, {{[0-9]+}}(%esp), %eax # imm = 0xC14
; X86-BMI1TBM-NEXT: addl %eax, %eax
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c4_i32_bad:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $3092, %eax # imm = 0xC14
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: addl %eax, %eax
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c4_i32_bad:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movl %edi, %eax
; X64-NOBMI-NEXT: shrl $19, %eax
; X64-NOBMI-NEXT: andl $-2, %eax
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c4_i32_bad:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $3092, %eax # imm = 0xC14
; X64-BMI1NOTBM-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBM-NEXT: addl %eax, %eax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c4_i32_bad:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrl $3092, %edi, %eax # imm = 0xC14
; X64-BMI1TBM-NEXT: addl %eax, %eax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c4_i32_bad:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $3092, %eax # imm = 0xC14
; X64-BMI1NOTBMBMI2-NEXT: bextrl %eax, %edi, %eax
; X64-BMI1NOTBMBMI2-NEXT: addl %eax, %eax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i32 %arg, 19 %tmp0 = lshr i32 %arg, 19
%tmp1 = and i32 %tmp0, 16382 %tmp1 = and i32 %tmp0, 16382
ret i32 %tmp1 ret i32 %tmp1
} }
; i64 ; i64
; The most canonical variant ; The most canonical variant
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines
; X64-BMI1NOTBMBMI2-NEXT: retq ; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i64 %arg, 51 %tmp0 = lshr i64 %arg, 51
%tmp1 = and i64 %tmp0, 1023 %tmp1 = and i64 %tmp0, 1023
ret i64 %tmp1 ret i64 %tmp1
} }
; Should be still fine, but the mask is shifted ; Should be still fine, but the mask is shifted
define i64 @c1_i64(i64 %arg) { define i64 @c1_i64(i64 %arg) {
; X86-LABEL: c1_i64: ; X86-NOBMI-LABEL: c1_i64:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $19, %eax ; X86-NOBMI-NEXT: shrl $19, %eax
; X86-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X86-NEXT: xorl %edx, %edx ; X86-NOBMI-NEXT: xorl %edx, %edx
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c1_i64: ; X86-BMI1NOTBM-LABEL: c1_i64:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movq %rdi, %rax ; X86-BMI1NOTBM-NEXT: movl $2581, %eax # imm = 0xA15
; X64-NEXT: shrq $51, %rax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-BMI1NOTBM-NEXT: shll $2, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: xorl %edx, %edx
; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c1_i64:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $2581, {{[0-9]+}}(%esp), %eax # imm = 0xA15
; X86-BMI1TBM-NEXT: shll $2, %eax
; X86-BMI1TBM-NEXT: xorl %edx, %edx
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c1_i64:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $2581, %eax # imm = 0xA15
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X86-BMI1NOTBMBMI2-NEXT: xorl %edx, %edx
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c1_i64:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movq %rdi, %rax
; X64-NOBMI-NEXT: shrq $51, %rax
; X64-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c1_i64:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $2613, %eax # imm = 0xA35
; X64-BMI1NOTBM-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBM-NEXT: shlq $2, %rax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c1_i64:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrq $2613, %rdi, %rax # imm = 0xA35
; X64-BMI1TBM-NEXT: shlq $2, %rax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c1_i64:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $2613, %eax # imm = 0xA35
; X64-BMI1NOTBMBMI2-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBMBMI2-NEXT: shlq $2, %rax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i64 %arg, 51 %tmp0 = lshr i64 %arg, 51
%tmp1 = and i64 %tmp0, 4092 %tmp1 = and i64 %tmp0, 4092
ret i64 %tmp1 ret i64 %tmp1
} }
; Should be still fine, but the result is shifted left afterwards ; Should be still fine, but the result is shifted left afterwards
define i64 @c2_i64(i64 %arg) { define i64 @c2_i64(i64 %arg) {
; X86-LABEL: c2_i64: ; X86-NOBMI-LABEL: c2_i64:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $17, %eax ; X86-NOBMI-NEXT: shrl $17, %eax
; X86-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X86-NEXT: xorl %edx, %edx ; X86-NOBMI-NEXT: xorl %edx, %edx
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c2_i64: ; X86-BMI1NOTBM-LABEL: c2_i64:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movq %rdi, %rax ; X86-BMI1NOTBM-NEXT: movl $2579, %eax # imm = 0xA13
; X64-NEXT: shrq $49, %rax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $4092, %eax # imm = 0xFFC ; X86-BMI1NOTBM-NEXT: shll $2, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: xorl %edx, %edx
; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c2_i64:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $2579, {{[0-9]+}}(%esp), %eax # imm = 0xA13
; X86-BMI1TBM-NEXT: shll $2, %eax
; X86-BMI1TBM-NEXT: xorl %edx, %edx
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c2_i64:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $2579, %eax # imm = 0xA13
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: shll $2, %eax
; X86-BMI1NOTBMBMI2-NEXT: xorl %edx, %edx
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c2_i64:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movq %rdi, %rax
; X64-NOBMI-NEXT: shrq $49, %rax
; X64-NOBMI-NEXT: andl $4092, %eax # imm = 0xFFC
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c2_i64:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $2611, %eax # imm = 0xA33
; X64-BMI1NOTBM-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBM-NEXT: shlq $2, %rax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c2_i64:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrq $2611, %rdi, %rax # imm = 0xA33
; X64-BMI1TBM-NEXT: shlq $2, %rax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c2_i64:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $2611, %eax # imm = 0xA33
; X64-BMI1NOTBMBMI2-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBMBMI2-NEXT: shlq $2, %rax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i64 %arg, 51 %tmp0 = lshr i64 %arg, 51
%tmp1 = and i64 %tmp0, 1023 %tmp1 = and i64 %tmp0, 1023
%tmp2 = shl i64 %tmp1, 2 %tmp2 = shl i64 %tmp1, 2
ret i64 %tmp2 ret i64 %tmp2
} }
; The mask covers newly shifted-in bit ; The mask covers newly shifted-in bit
define i64 @c4_i64_bad(i64 %arg) { define i64 @c4_i64_bad(i64 %arg) {
; X86-LABEL: c4_i64_bad: ; X86-NOBMI-LABEL: c4_i64_bad:
; X86: # %bb.0: ; X86-NOBMI: # %bb.0:
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax ; X86-NOBMI-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: shrl $19, %eax ; X86-NOBMI-NEXT: shrl $19, %eax
; X86-NEXT: andl $-2, %eax ; X86-NOBMI-NEXT: andl $-2, %eax
; X86-NEXT: xorl %edx, %edx ; X86-NOBMI-NEXT: xorl %edx, %edx
; X86-NEXT: retl ; X86-NOBMI-NEXT: retl
; ;
; X64-LABEL: c4_i64_bad: ; X86-BMI1NOTBM-LABEL: c4_i64_bad:
; X64: # %bb.0: ; X86-BMI1NOTBM: # %bb.0:
; X64-NEXT: movq %rdi, %rax ; X86-BMI1NOTBM-NEXT: movl $3092, %eax # imm = 0xC14
; X64-NEXT: shrq $51, %rax ; X86-BMI1NOTBM-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X64-NEXT: andl $-2, %eax ; X86-BMI1NOTBM-NEXT: addl %eax, %eax
; X64-NEXT: retq ; X86-BMI1NOTBM-NEXT: xorl %edx, %edx
; X86-BMI1NOTBM-NEXT: retl
;
; X86-BMI1TBM-LABEL: c4_i64_bad:
; X86-BMI1TBM: # %bb.0:
; X86-BMI1TBM-NEXT: bextrl $3092, {{[0-9]+}}(%esp), %eax # imm = 0xC14
; X86-BMI1TBM-NEXT: addl %eax, %eax
; X86-BMI1TBM-NEXT: xorl %edx, %edx
; X86-BMI1TBM-NEXT: retl
;
; X86-BMI1NOTBMBMI2-LABEL: c4_i64_bad:
; X86-BMI1NOTBMBMI2: # %bb.0:
; X86-BMI1NOTBMBMI2-NEXT: movl $3092, %eax # imm = 0xC14
; X86-BMI1NOTBMBMI2-NEXT: bextrl %eax, {{[0-9]+}}(%esp), %eax
; X86-BMI1NOTBMBMI2-NEXT: addl %eax, %eax
; X86-BMI1NOTBMBMI2-NEXT: xorl %edx, %edx
; X86-BMI1NOTBMBMI2-NEXT: retl
;
; X64-NOBMI-LABEL: c4_i64_bad:
; X64-NOBMI: # %bb.0:
; X64-NOBMI-NEXT: movq %rdi, %rax
; X64-NOBMI-NEXT: shrq $51, %rax
; X64-NOBMI-NEXT: andl $-2, %eax
; X64-NOBMI-NEXT: retq
;
; X64-BMI1NOTBM-LABEL: c4_i64_bad:
; X64-BMI1NOTBM: # %bb.0:
; X64-BMI1NOTBM-NEXT: movl $3124, %eax # imm = 0xC34
; X64-BMI1NOTBM-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBM-NEXT: addq %rax, %rax
; X64-BMI1NOTBM-NEXT: retq
;
; X64-BMI1TBM-LABEL: c4_i64_bad:
; X64-BMI1TBM: # %bb.0:
; X64-BMI1TBM-NEXT: bextrq $3124, %rdi, %rax # imm = 0xC34
; X64-BMI1TBM-NEXT: addq %rax, %rax
; X64-BMI1TBM-NEXT: retq
;
; X64-BMI1NOTBMBMI2-LABEL: c4_i64_bad:
; X64-BMI1NOTBMBMI2: # %bb.0:
; X64-BMI1NOTBMBMI2-NEXT: movl $3124, %eax # imm = 0xC34
; X64-BMI1NOTBMBMI2-NEXT: bextrq %rax, %rdi, %rax
; X64-BMI1NOTBMBMI2-NEXT: addq %rax, %rax
; X64-BMI1NOTBMBMI2-NEXT: retq
%tmp0 = lshr i64 %arg, 51 %tmp0 = lshr i64 %arg, 51
%tmp1 = and i64 %tmp0, 16382 %tmp1 = and i64 %tmp0, 16382
ret i64 %tmp1 ret i64 %tmp1
} }
lebedev.riAuthorUnsubmitted
Not Done
ReplyInline Actions

Hm, indeed, these tests degrade.
I don't have any good ideas on how to avoid this.

lebedev.ri: Hm, indeed, these tests degrade. I don't have any good ideas on how to avoid this.