This is an archive of the discontinued LLVM Phabricator instance.

Differential D106388

[AArch64][GlobalISel] Legalize ctpop for v2s64, v2s32, v4s32, v4s16, v8s16
ClosedPublic

Authored by jroelofs on Jul 20 2021, 12:30 PM.

Details

Reviewers
aemerson
paquette
Commits
rG75187aa35255: [AArch64][GlobalISel] Legalize ctpop for v2s64, v2s32, v4s32, v4s16, v8s16
Summary

https://llvm.godbolt.org/z/nTTK6M5qe

Diff Detail

Event Timeline

jroelofs created this revision.Jul 20 2021, 12:30 PM
Herald added subscribers: danielkiss, hiraditya, kristof.beyls, rovka. · View Herald TranscriptJul 20 2021, 12:30 PM
jroelofs requested review of this revision.Jul 20 2021, 12:30 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 20 2021, 12:30 PM
Herald added a subscriber: llvm-commits. · View Herald Transcript
paquette added inline comments.Jul 20 2021, 1:22 PM
llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp
1217

doesn't SmallVector need an initial size?

1237

I think CTPOP is only used here? Maybe it's a little cleaner to move it and related stuff down near it.

1244

I don't think you should have to do this here, since we're before register bank selection?

1250

Logic here is getting a little complex. Maybe this?

if (Ty.isScalar() && Size == 64)
  MIRBuilder.buildZExt(Dst, UADD);
else
  UADD->getOperand(0).setReg(Dst);
1258

This shouldn't be necessary since we haven't selected regbanks yet.

jroelofs added inline comments.Jul 20 2021, 1:55 PM
llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp
1217

Sean Silva recently added a default to it: https://reviews.llvm.org/D92522

1237

I wanted to keep HAdds near where it was used.

1244

MachineVerifier seems to expect them just after legalization:

*** Bad machine code: Virtual register does not match instruction constraint ***
- function:    s32_lower
- basic block: %bb.0  (0x7fba998570c8)
- instruction: %ctpop:_(s32) = UADDLVv8i8v %4:_(<8 x s8>)
- operand 0:   %ctpop:_
Expect register class FPR16 but got nothing
1250

👍

jroelofs updated this revision to Diff 360260.Jul 20 2021, 1:55 PM
paquette added inline comments.Jul 20 2021, 2:16 PM
llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp
1244

ah, yeah I guess it's cause it's not a generic instr. :/

Maybe it would make sense to add a G_UADDLV generic instruction to avoid emitting target instrs.

IIRC we import tablegen patterns for the uaddlv intrinsic allowing us to select it easily. Maybe if we had a generic instruction, we could do something similar.

jroelofs added inline comments.Jul 20 2021, 2:33 PM
llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp
1244

maybe I ought to just use the intrinsics then

jroelofs updated this revision to Diff 360279.Jul 20 2021, 2:34 PM
paquette accepted this revision.Jul 20 2021, 3:04 PM

LGTM

This revision is now accepted and ready to land.Jul 20 2021, 3:04 PM
This revision was landed with ongoing or failed builds.Jul 20 2021, 3:39 PM
Closed by commit rG75187aa35255: [AArch64][GlobalISel] Legalize ctpop for v2s64, v2s32, v4s32, v4s16, v8s16 (authored by jroelofs). · Explain Why
This revision was automatically updated to reflect the committed changes.
jroelofs added a commit: rG75187aa35255: [AArch64][GlobalISel] Legalize ctpop for v2s64, v2s32, v4s32, v4s16, v8s16.
Harbormaster completed remote builds in B115194: Diff 360279.Jul 20 2021, 4:28 PM

Revision Contents

PathSize
llvm/
lib/
Target/
AArch64/
GISel/
94 lines
test/
CodeGen/
AArch64/
GlobalISel/
109 lines

Diff 360301

llvm/lib/Target/AArch64/GISel/AArch64LegalizerInfo.cpp

Show First 20 Lines • Show All 757 Lines • ▼ Show 20 Lines getActionDefinitionsBuilder(G_ROTR)
}) })
.lower(); .lower();
getActionDefinitionsBuilder(G_ROTL).lower(); getActionDefinitionsBuilder(G_ROTL).lower();
getActionDefinitionsBuilder({G_SBFX, G_UBFX}) getActionDefinitionsBuilder({G_SBFX, G_UBFX})
.customFor({{s32, s32}, {s64, s64}}); .customFor({{s32, s32}, {s64, s64}});
// TODO: Custom legalization for s128 // TODO: Custom legalization for s128
// TODO: v2s64, v2s32, v4s32, v4s16, v8s16
// TODO: Use generic lowering when custom lowering is not possible. // TODO: Use generic lowering when custom lowering is not possible.
auto always = [=](const LegalityQuery &Q) { return true; }; auto always = [=](const LegalityQuery &Q) { return true; };
getActionDefinitionsBuilder(G_CTPOP) getActionDefinitionsBuilder(G_CTPOP)
.legalFor({{v8s8, v8s8}, {v16s8, v16s8}}) .legalFor({{v8s8, v8s8}, {v16s8, v16s8}})
.clampScalar(0, s32, s128) .clampScalar(0, s32, s128)
.widenScalarToNextPow2(0) .widenScalarToNextPow2(0)
.minScalarEltSameAsIf(always, 1, 0) .minScalarEltSameAsIf(always, 1, 0)
.maxScalarEltSameAsIf(always, 1, 0) .maxScalarEltSameAsIf(always, 1, 0)
.customFor({{s32, s32}, {s64, s64}}); .customFor({{s32, s32},
{s64, s64},
{v2s64, v2s64},
{v2s32, v2s32},
{v4s32, v4s32},
{v4s16, v4s16},
{v8s16, v8s16}});
getLegacyLegalizerInfo().computeTables(); getLegacyLegalizerInfo().computeTables();
verify(*ST.getInstrInfo()); verify(*ST.getInstrInfo());
} }
bool AArch64LegalizerInfo::legalizeCustom(LegalizerHelper &Helper, bool AArch64LegalizerInfo::legalizeCustom(LegalizerHelper &Helper,
MachineInstr &MI) const { MachineInstr &MI) const {
MachineIRBuilder &MIRBuilder = Helper.MIRBuilder; MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
▲ Show 20 Lines • Show All 326 Lines • ▼ Show 20 Linesbool AArch64LegalizerInfo::legalizeCTPOP(MachineInstr &MI,
// While there is no integer popcount instruction, it can // While there is no integer popcount instruction, it can
// be more efficiently lowered to the following sequence that uses // be more efficiently lowered to the following sequence that uses
// AdvSIMD registers/instructions as long as the copies to/from // AdvSIMD registers/instructions as long as the copies to/from
// the AdvSIMD registers are cheap. // the AdvSIMD registers are cheap.
// FMOV D0, X0 // copy 64-bit int to vector, high bits zero'd // FMOV D0, X0 // copy 64-bit int to vector, high bits zero'd
// CNT V0.8B, V0.8B // 8xbyte pop-counts // CNT V0.8B, V0.8B // 8xbyte pop-counts
// ADDV B0, V0.8B // sum 8xbyte pop-counts // ADDV B0, V0.8B // sum 8xbyte pop-counts
// UMOV X0, V0.B[0] // copy byte result back to integer reg // UMOV X0, V0.B[0] // copy byte result back to integer reg
//
// For 128 bit vector popcounts, we lower to the following sequence:
// cnt.16b v0, v0 // v8s16, v4s32, v2s64
// uaddlp.8h v0, v0 // v8s16, v4s32, v2s64
// uaddlp.4s v0, v0 // v4s32, v2s64
// uaddlp.2d v0, v0 // v2s64
//
// For 64 bit vector popcounts, we lower to the following sequence:
// cnt.8b v0, v0 // v4s16, v2s32
// uaddlp.4h v0, v0 // v4s16, v2s32
// uaddlp.2s v0, v0 // v2s32
if (!ST->hasNEON() || if (!ST->hasNEON() ||
MI.getMF()->getFunction().hasFnAttribute(Attribute::NoImplicitFloat)) MI.getMF()->getFunction().hasFnAttribute(Attribute::NoImplicitFloat))
return false; return false;
MachineIRBuilder &MIRBuilder = Helper.MIRBuilder; MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
Register Dst = MI.getOperand(0).getReg(); Register Dst = MI.getOperand(0).getReg();
Register Val = MI.getOperand(1).getReg(); Register Val = MI.getOperand(1).getReg();
LLT Ty = MRI.getType(Val); LLT Ty = MRI.getType(Val);
// TODO: Handle vector types.
assert(!Ty.isVector() && "Vector types not handled yet!");
assert(Ty == MRI.getType(Dst) && assert(Ty == MRI.getType(Dst) &&
"Expected src and dst to have the same type!"); "Expected src and dst to have the same type!");
// TODO: Handle s128.
unsigned Size = Ty.getSizeInBits(); unsigned Size = Ty.getSizeInBits();
// Pre-conditioning: widen Val up to the nearest vector type.
// s32,s64,v4s16,v2s32 -> v8i8
// v8s16,v4s32,v2s64 -> v16i8
LLT VTy = Size == 128 ? LLT::fixed_vector(16, 8) : LLT::fixed_vector(8, 8);
if (Ty.isScalar()) {
// TODO: Handle s128.
assert((Size == 32 || Size == 64) && "Expected only 32 or 64 bit scalars!"); assert((Size == 32 || Size == 64) && "Expected only 32 or 64 bit scalars!");
if (Size == 32) if (Size == 32) {
Val = MIRBuilder.buildZExt(LLT::scalar(64), Val).getReg(0); Val = MIRBuilder.buildZExt(LLT::scalar(64), Val).getReg(0);
const LLT V8S8 = LLT::fixed_vector(8, LLT::scalar(8)); }
Val = MIRBuilder.buildBitcast(V8S8, Val).getReg(0); }
auto CTPOP = MIRBuilder.buildCTPOP(V8S8, Val); Val = MIRBuilder.buildBitcast(VTy, Val).getReg(0);
auto UADDLV =
MIRBuilder // Count bits in each byte-sized lane.
.buildIntrinsic(Intrinsic::aarch64_neon_uaddlv, {LLT::scalar(32)}, auto CTPOP = MIRBuilder.buildCTPOP(VTy, Val);
/*HasSideEffects = */ false)
.addUse(CTPOP.getReg(0)); // Sum across lanes.
if (Size == 64) Register HSum = CTPOP.getReg(0);
MIRBuilder.buildZExt(Dst, UADDLV); unsigned Opc;
SmallVector<LLT> HAddTys;
if (Ty.isScalar()) {
Opc = Intrinsic::aarch64_neon_uaddlv;
HAddTys.push_back(LLT::scalar(32));
} else if (Ty == LLT::fixed_vector(8, 16)) {
Opc = Intrinsic::aarch64_neon_uaddlp;
HAddTys.push_back(LLT::fixed_vector(8, 16));
} else if (Ty == LLT::fixed_vector(4, 32)) {
Opc = Intrinsic::aarch64_neon_uaddlp;
HAddTys.push_back(LLT::fixed_vector(8, 16));
HAddTys.push_back(LLT::fixed_vector(4, 32));
} else if (Ty == LLT::fixed_vector(2, 64)) {
Opc = Intrinsic::aarch64_neon_uaddlp;
HAddTys.push_back(LLT::fixed_vector(8, 16));
HAddTys.push_back(LLT::fixed_vector(4, 32));
HAddTys.push_back(LLT::fixed_vector(2, 64));
} else if (Ty == LLT::fixed_vector(4, 16)) {
Opc = Intrinsic::aarch64_neon_uaddlp;
HAddTys.push_back(LLT::fixed_vector(4, 16));
} else if (Ty == LLT::fixed_vector(2, 32)) {
Opc = Intrinsic::aarch64_neon_uaddlp;
HAddTys.push_back(LLT::fixed_vector(4, 16));
HAddTys.push_back(LLT::fixed_vector(2, 32));
} else
llvm_unreachable("unexpected vector shape");
MachineInstrBuilder UADD;
for (LLT HTy : HAddTys) {
UADD = MIRBuilder.buildIntrinsic(Opc, {HTy}, /*HasSideEffects =*/false)
.addUse(HSum);
HSum = UADD.getReg(0);
}
// Post-conditioning.
if (Ty.isScalar() && Size == 64)
MIRBuilder.buildZExt(Dst, UADD);
else else
UADDLV->getOperand(0).setReg(Dst); UADD->getOperand(0).setReg(Dst);
MI.eraseFromParent(); MI.eraseFromParent();
return true; return true;
} }
bool AArch64LegalizerInfo::legalizeAtomicCmpxchg128( bool AArch64LegalizerInfo::legalizeAtomicCmpxchg128(
MachineInstr &MI, MachineRegisterInfo &MRI, LegalizerHelper &Helper) const { MachineInstr &MI, MachineRegisterInfo &MRI, LegalizerHelper &Helper) const {
MachineIRBuilder &MIRBuilder = Helper.MIRBuilder; MachineIRBuilder &MIRBuilder = Helper.MIRBuilder;
LLT s64 = LLT::scalar(64); LLT s64 = LLT::scalar(64);
auto Addr = MI.getOperand(1).getReg(); auto Addr = MI.getOperand(1).getReg();
auto DesiredI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(2)); auto DesiredI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(2));
auto NewI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(3)); auto NewI = MIRBuilder.buildUnmerge({s64, s64}, MI.getOperand(3));
auto DstLo = MRI.createGenericVirtualRegister(s64); auto DstLo = MRI.createGenericVirtualRegister(s64);
auto DstHi = MRI.createGenericVirtualRegister(s64); auto DstHi = MRI.createGenericVirtualRegister(s64);
MachineInstrBuilder CAS; MachineInstrBuilder CAS;
paquetteUnsubmitted
Not Done
ReplyInline Actions

doesn't SmallVector need an initial size?

paquette: doesn't SmallVector need an initial size?
jroelofsAuthorUnsubmitted
Done
ReplyInline Actions

Sean Silva recently added a default to it: https://reviews.llvm.org/D92522

jroelofs: Sean Silva recently added a default to it: https://reviews.llvm.org/D92522
if (ST->hasLSE()) { if (ST->hasLSE()) {
// We have 128-bit CASP instructions taking XSeqPair registers, which are // We have 128-bit CASP instructions taking XSeqPair registers, which are
// s128. We need the merge/unmerge to bracket the expansion and pair up with // s128. We need the merge/unmerge to bracket the expansion and pair up with
// the rest of the MIR so we must reassemble the extracted registers into a // the rest of the MIR so we must reassemble the extracted registers into a
// 128-bit known-regclass one with code like this: // 128-bit known-regclass one with code like this:
// //
// %in1 = REG_SEQUENCE Lo, Hi ; One for each input // %in1 = REG_SEQUENCE Lo, Hi ; One for each input
// %out = CASP %in1, ... // %out = CASP %in1, ...
// %OldLo = G_EXTRACT %out, 0 // %OldLo = G_EXTRACT %out, 0
// %OldHi = G_EXTRACT %out, 64 // %OldHi = G_EXTRACT %out, 64
auto Ordering = (*MI.memoperands_begin())->getMergedOrdering(); auto Ordering = (*MI.memoperands_begin())->getMergedOrdering();
unsigned Opcode; unsigned Opcode;
switch (Ordering) { switch (Ordering) {
case AtomicOrdering::Acquire: case AtomicOrdering::Acquire:
Opcode = AArch64::CASPAX; Opcode = AArch64::CASPAX;
break; break;
case AtomicOrdering::Release: case AtomicOrdering::Release:
Opcode = AArch64::CASPLX; Opcode = AArch64::CASPLX;
break; break;
case AtomicOrdering::AcquireRelease: case AtomicOrdering::AcquireRelease:
paquetteUnsubmitted
Not Done
ReplyInline Actions
llvm_unreachable("unexpected vector shape");
- Register HSum = CTPOP.getReg(0);
+ LLT CTPOPTy = Size == 128 ? LLT::fixed_vector(16, 8) : LLT::fixed_vector(8, 8);
+ Val = MIRBuilder.buildBitcast(CTPOPTy, Val).getReg(0);
+ Register HSum = MIRBuilder.buildCTPOP(CTPOPTy, Val).getReg(0);
MachineInstrBuilder UADD;

I think CTPOP is only used here? Maybe it's a little cleaner to move it and related stuff down near it.

paquette: I think `CTPOP` is only used here? Maybe it's a little cleaner to move it and related stuff…
jroelofsAuthorUnsubmitted
Done
ReplyInline Actions

I wanted to keep HAdds near where it was used.

jroelofs: I wanted to keep HAdds near where it was used.
case AtomicOrdering::SequentiallyConsistent: case AtomicOrdering::SequentiallyConsistent:
Opcode = AArch64::CASPALX; Opcode = AArch64::CASPALX;
break; break;
default: default:
Opcode = AArch64::CASPX; Opcode = AArch64::CASPX;
break; break;
} }
paquetteUnsubmitted
Not Done
ReplyInline Actions

I don't think you should have to do this here, since we're before register bank selection?

paquette: I don't think you should have to do this here, since we're before register bank selection?
jroelofsAuthorUnsubmitted
Done
ReplyInline Actions

MachineVerifier seems to expect them just after legalization:

*** Bad machine code: Virtual register does not match instruction constraint ***
- function:    s32_lower
- basic block: %bb.0  (0x7fba998570c8)
- instruction: %ctpop:_(s32) = UADDLVv8i8v %4:_(<8 x s8>)
- operand 0:   %ctpop:_
Expect register class FPR16 but got nothing
jroelofs: MachineVerifier seems to expect them just after legalization: ``` *** Bad machine code…
paquetteUnsubmitted
Not Done
ReplyInline Actions

ah, yeah I guess it's cause it's not a generic instr. :/

Maybe it would make sense to add a G_UADDLV generic instruction to avoid emitting target instrs.

IIRC we import tablegen patterns for the uaddlv intrinsic allowing us to select it easily. Maybe if we had a generic instruction, we could do something similar.

paquette: ah, yeah I guess it's cause it's not a generic instr. :/ Maybe it would make sense to add a…
jroelofsAuthorUnsubmitted
Done
ReplyInline Actions

maybe I ought to just use the intrinsics then

jroelofs: maybe I ought to just use the intrinsics then
LLT s128 = LLT::scalar(128); LLT s128 = LLT::scalar(128);
auto CASDst = MRI.createGenericVirtualRegister(s128); auto CASDst = MRI.createGenericVirtualRegister(s128);
auto CASDesired = MRI.createGenericVirtualRegister(s128); auto CASDesired = MRI.createGenericVirtualRegister(s128);
auto CASNew = MRI.createGenericVirtualRegister(s128); auto CASNew = MRI.createGenericVirtualRegister(s128);
MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASDesired}, {}) MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASDesired}, {})
paquetteUnsubmitted
Not Done
ReplyInline Actions

Logic here is getting a little complex. Maybe this?

if (Ty.isScalar() && Size == 64)
  MIRBuilder.buildZExt(Dst, UADD);
else
  UADD->getOperand(0).setReg(Dst);
paquette: Logic here is getting a little complex. Maybe this? ``` if (Ty.isScalar() && Size == 64)…
jroelofsAuthorUnsubmitted
Done
ReplyInline Actions

👍

jroelofs: 👍
.addUse(DesiredI->getOperand(0).getReg()) .addUse(DesiredI->getOperand(0).getReg())
.addImm(AArch64::sube64) .addImm(AArch64::sube64)
.addUse(DesiredI->getOperand(1).getReg()) .addUse(DesiredI->getOperand(1).getReg())
.addImm(AArch64::subo64); .addImm(AArch64::subo64);
MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASNew}, {}) MIRBuilder.buildInstr(TargetOpcode::REG_SEQUENCE, {CASNew}, {})
.addUse(NewI->getOperand(0).getReg()) .addUse(NewI->getOperand(0).getReg())
.addImm(AArch64::sube64) .addImm(AArch64::sube64)
.addUse(NewI->getOperand(1).getReg()) .addUse(NewI->getOperand(1).getReg())
paquetteUnsubmitted
Not Done
ReplyInline Actions

This shouldn't be necessary since we haven't selected regbanks yet.

paquette: This shouldn't be necessary since we haven't selected regbanks yet.
.addImm(AArch64::subo64); .addImm(AArch64::subo64);
CAS = MIRBuilder.buildInstr(Opcode, {CASDst}, {CASDesired, CASNew, Addr}); CAS = MIRBuilder.buildInstr(Opcode, {CASDst}, {CASDesired, CASNew, Addr});
MIRBuilder.buildExtract({DstLo}, {CASDst}, 0); MIRBuilder.buildExtract({DstLo}, {CASDst}, 0);
MIRBuilder.buildExtract({DstHi}, {CASDst}, 64); MIRBuilder.buildExtract({DstHi}, {CASDst}, 64);
} else { } else {
// The -O0 CMP_SWAP_128 is friendlier to generate code for because LDXP/STXP // The -O0 CMP_SWAP_128 is friendlier to generate code for because LDXP/STXP
▲ Show 20 Lines • Show All 47 Lines • Show Last 20 Lines

llvm/test/CodeGen/AArch64/GlobalISel/legalize-ctpop.mir

Show First 20 Lines • Show All 182 Lines • ▼ Show 20 Lines bb.0:
; CHECK: $w0 = COPY %ext(s32) ; CHECK: $w0 = COPY %ext(s32)
; CHECK: RET_ReallyLR implicit $w0 ; CHECK: RET_ReallyLR implicit $w0
%copy:_(s32) = COPY $w0 %copy:_(s32) = COPY $w0
%trunc:_(s8) = G_TRUNC %copy(s32) %trunc:_(s8) = G_TRUNC %copy(s32)
%ctpop:_(s16) = G_CTPOP %trunc(s8) %ctpop:_(s16) = G_CTPOP %trunc(s8)
%ext:_(s32) = G_ANYEXT %ctpop(s16) %ext:_(s32) = G_ANYEXT %ctpop(s16)
$w0 = COPY %ext(s32) $w0 = COPY %ext(s32)
RET_ReallyLR implicit $w0 RET_ReallyLR implicit $w0
...
---
name: custom_8x16
tracksRegLiveness: true
body: |
bb.0:
liveins: $q0
; CHECK-LABEL: name: custom_8x16
; CHECK: liveins: $q0
; CHECK: [[COPY:%[0-9]+]]:_(<8 x s16>) = COPY $q0
; CHECK: [[BITCAST:%[0-9]+]]:_(<16 x s8>) = G_BITCAST [[COPY]](<8 x s16>)
; CHECK: [[CTPOP:%[0-9]+]]:_(<16 x s8>) = G_CTPOP [[BITCAST]](<16 x s8>)
; CHECK: [[INT:%[0-9]+]]:_(<8 x s16>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[CTPOP]](<16 x s8>)
; CHECK: $q0 = COPY [[INT]](<8 x s16>)
; CHECK: RET_ReallyLR implicit $q0
%0:_(<8 x s16>) = COPY $q0
%1:_(<8 x s16>) = G_CTPOP %0(<8 x s16>)
$q0 = COPY %1(<8 x s16>)
RET_ReallyLR implicit $q0
...
---
name: custom_4x32
tracksRegLiveness: true
body: |
bb.0:
liveins: $q0
; CHECK-LABEL: name: custom_4x32
; CHECK: liveins: $q0
; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
; CHECK: [[BITCAST:%[0-9]+]]:_(<16 x s8>) = G_BITCAST [[COPY]](<4 x s32>)
; CHECK: [[CTPOP:%[0-9]+]]:_(<16 x s8>) = G_CTPOP [[BITCAST]](<16 x s8>)
; CHECK: [[INT:%[0-9]+]]:_(<8 x s16>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[CTPOP]](<16 x s8>)
; CHECK: [[INT1:%[0-9]+]]:_(<4 x s32>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[INT]](<8 x s16>)
; CHECK: $q0 = COPY [[INT1]](<4 x s32>)
; CHECK: RET_ReallyLR implicit $q0
%0:_(<4 x s32>) = COPY $q0
%1:_(<4 x s32>) = G_CTPOP %0(<4 x s32>)
$q0 = COPY %1(<4 x s32>)
RET_ReallyLR implicit $q0
...
---
name: custom_2x64
tracksRegLiveness: true
body: |
bb.0:
liveins: $q0
; CHECK-LABEL: name: custom_2x64
; CHECK: liveins: $q0
; CHECK: [[COPY:%[0-9]+]]:_(<2 x s64>) = COPY $q0
; CHECK: [[BITCAST:%[0-9]+]]:_(<16 x s8>) = G_BITCAST [[COPY]](<2 x s64>)
; CHECK: [[CTPOP:%[0-9]+]]:_(<16 x s8>) = G_CTPOP [[BITCAST]](<16 x s8>)
; CHECK: [[INT:%[0-9]+]]:_(<8 x s16>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[CTPOP]](<16 x s8>)
; CHECK: [[INT1:%[0-9]+]]:_(<4 x s32>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[INT]](<8 x s16>)
; CHECK: [[INT2:%[0-9]+]]:_(<2 x s64>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[INT1]](<4 x s32>)
; CHECK: $q0 = COPY [[INT2]](<2 x s64>)
; CHECK: RET_ReallyLR implicit $q0
%0:_(<2 x s64>) = COPY $q0
%1:_(<2 x s64>) = G_CTPOP %0(<2 x s64>)
$q0 = COPY %1(<2 x s64>)
RET_ReallyLR implicit $q0
...
---
name: custom_4x16
tracksRegLiveness: true
body: |
bb.0:
liveins: $d0
; CHECK-LABEL: name: custom_4x16
; CHECK: liveins: $d0
; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $d0
; CHECK: [[BITCAST:%[0-9]+]]:_(<8 x s8>) = G_BITCAST [[COPY]](<4 x s16>)
; CHECK: [[CTPOP:%[0-9]+]]:_(<8 x s8>) = G_CTPOP [[BITCAST]](<8 x s8>)
; CHECK: [[INT:%[0-9]+]]:_(<4 x s16>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[CTPOP]](<8 x s8>)
; CHECK: $d0 = COPY [[INT]](<4 x s16>)
; CHECK: RET_ReallyLR implicit $d0
%0:_(<4 x s16>) = COPY $d0
%1:_(<4 x s16>) = G_CTPOP %0(<4 x s16>)
$d0 = COPY %1(<4 x s16>)
RET_ReallyLR implicit $d0
...
---
name: custom_2x32
tracksRegLiveness: true
body: |
bb.0:
liveins: $d0
; CHECK-LABEL: name: custom_2x32
; CHECK: liveins: $d0
; CHECK: [[COPY:%[0-9]+]]:_(<2 x s32>) = COPY $d0
; CHECK: [[BITCAST:%[0-9]+]]:_(<8 x s8>) = G_BITCAST [[COPY]](<2 x s32>)
; CHECK: [[CTPOP:%[0-9]+]]:_(<8 x s8>) = G_CTPOP [[BITCAST]](<8 x s8>)
; CHECK: [[INT:%[0-9]+]]:_(<4 x s16>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[CTPOP]](<8 x s8>)
; CHECK: [[INT1:%[0-9]+]]:_(<2 x s32>) = G_INTRINSIC intrinsic(@llvm.aarch64.neon.uaddlp), [[INT]](<4 x s16>)
; CHECK: $d0 = COPY [[INT1]](<2 x s32>)
; CHECK: RET_ReallyLR implicit $d0
%0:_(<2 x s32>) = COPY $d0
%1:_(<2 x s32>) = G_CTPOP %0(<2 x s32>)
$d0 = COPY %1(<2 x s32>)
RET_ReallyLR implicit $d0

llvm/test/CodeGen/AArch64/GlobalISel/legalize-vector-ctpop.mir

  • This file was deleted.
# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
# RUN: llc -mtriple=aarch64-unknown-unknown -verify-machineinstrs -run-pass=legalizer %s -o - -global-isel-abort=2 | FileCheck %s
...
# This test doesn't currently legalize but should at least not crash.
---
name: v2s64_dont_crash
tracksRegLiveness: true
body: |
bb.0:
liveins: $q0
; CHECK-LABEL: name: v2s64_dont_crash
; CHECK: liveins: $q0
; CHECK: %copy:_(<2 x s64>) = COPY $q0
; CHECK: %ctpop:_(<2 x s64>) = G_CTPOP %copy(<2 x s64>)
; CHECK: $q0 = COPY %ctpop(<2 x s64>)
; CHECK: RET_ReallyLR implicit $q0
%copy:_(<2 x s64>) = COPY $q0
%ctpop:_(<2 x s64>) = G_CTPOP %copy(<2 x s64>)
$q0 = COPY %ctpop(<2 x s64>)
RET_ReallyLR implicit $q0
...
llvm/test/CodeGen/AArch64/GlobalISel/legalize-ctpop.mir