This is an archive of the discontinued LLVM Phabricator instance.

Differential D130740

[TTI][AArch64][SLP] Sets the cost of an ADD reduction 2xi64 to 2.
ClosedPublic

Authored by vporpo on Jul 28 2022, 5:59 PM.

Details

Reviewers
dmgreen
ABataev
RKSimon
Commits
rGf6690303732e: [TTI][AArch64][SLP] Sets the cost of an ADD reduction 2xi64 to 2.
Summary

2xi64 is the legalized type for wide reductions (like 16xi64) and setting the
cost to 2 makes load-reduce and load-zext-reduce patterns profitable.

The few performance measurments that I did on an aarch64 machine confirm that
these patterns are actually faster when vectorized.

Diff Detail

Event Timeline

vporpo created this revision.Jul 28 2022, 5:59 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 28 2022, 5:59 PM
Herald added subscribers: hiraditya, kristof.beyls. · View Herald Transcript
vporpo requested review of this revision.Jul 28 2022, 5:59 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 28 2022, 5:59 PM
Herald added a subscriber: llvm-commits. · View Herald Transcript
vporpo added a parent revision: D130739: [TTI][SLP][AArch64] Precommit test for a TTI cost fix of i64 add reductions..Jul 28 2022, 5:59 PM
Harbormaster completed remote builds in B178186: Diff 448496.Jul 28 2022, 5:59 PM
vporpo added reviewers: dmgreen, ABataev, RKSimon.Jul 28 2022, 6:00 PM
dmgreen accepted this revision.Jul 30 2022, 9:37 AM

I agree, 2 sounds OK to me.
LGTM

This revision is now accepted and ready to land.Jul 30 2022, 9:37 AM
This revision was landed with ongoing or failed builds.Aug 1 2022, 1:04 PM
Closed by commit rGf6690303732e: [TTI][AArch64][SLP] Sets the cost of an ADD reduction 2xi64 to 2. (authored by vporpo). · Explain Why
This revision was automatically updated to reflect the committed changes.
vporpo added a commit: rGf6690303732e: [TTI][AArch64][SLP] Sets the cost of an ADD reduction 2xi64 to 2..

Revision Contents

PathSize
llvm/
lib/
Target/
AArch64/
1 line
test/
Analysis/
CostModel/
AArch64/
2 lines
Transforms/
SLPVectorizer/
AArch64/
192 lines

Diff 449106

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

Show First 20 Lines • Show All 2,703 Lines • ▼ Show 20 LinesAArch64TTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *ValTy,
// XOR: llvm/test/CodeGen/AArch64/reduce-xor.ll // XOR: llvm/test/CodeGen/AArch64/reduce-xor.ll
// AND: llvm/test/CodeGen/AArch64/reduce-and.ll // AND: llvm/test/CodeGen/AArch64/reduce-and.ll
static const CostTblEntry CostTblNoPairwise[]{ static const CostTblEntry CostTblNoPairwise[]{
{ISD::ADD, MVT::v8i8, 2}, {ISD::ADD, MVT::v8i8, 2},
{ISD::ADD, MVT::v16i8, 2}, {ISD::ADD, MVT::v16i8, 2},
{ISD::ADD, MVT::v4i16, 2}, {ISD::ADD, MVT::v4i16, 2},
{ISD::ADD, MVT::v8i16, 2}, {ISD::ADD, MVT::v8i16, 2},
{ISD::ADD, MVT::v4i32, 2}, {ISD::ADD, MVT::v4i32, 2},
{ISD::ADD, MVT::v2i64, 2},
{ISD::OR, MVT::v8i8, 15}, {ISD::OR, MVT::v8i8, 15},
{ISD::OR, MVT::v16i8, 17}, {ISD::OR, MVT::v16i8, 17},
{ISD::OR, MVT::v4i16, 7}, {ISD::OR, MVT::v4i16, 7},
{ISD::OR, MVT::v8i16, 9}, {ISD::OR, MVT::v8i16, 9},
{ISD::OR, MVT::v2i32, 3}, {ISD::OR, MVT::v2i32, 3},
{ISD::OR, MVT::v4i32, 5}, {ISD::OR, MVT::v4i32, 5},
{ISD::OR, MVT::v2i64, 3}, {ISD::OR, MVT::v2i64, 3},
{ISD::XOR, MVT::v8i8, 15}, {ISD::XOR, MVT::v8i8, 15},
▲ Show 20 Lines • Show All 327 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/AArch64/reduce-add.ll

Show All 12 Lines
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i16 = call i16 @llvm.vector.reduce.add.v2i16(<2 x i16> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i16 = call i16 @llvm.vector.reduce.add.v2i16(<2 x i16> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V4i16 = call i16 @llvm.vector.reduce.add.v4i16(<4 x i16> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V4i16 = call i16 @llvm.vector.reduce.add.v4i16(<4 x i16> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V8i16 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V8i16 = call i16 @llvm.vector.reduce.add.v8i16(<8 x i16> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V16i16 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V16i16 = call i16 @llvm.vector.reduce.add.v16i16(<16 x i16> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i32 = call i32 @llvm.vector.reduce.add.v2i32(<2 x i32> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i32 = call i32 @llvm.vector.reduce.add.v2i32(<2 x i32> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V4i32 = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V4i32 = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V8i32 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V8i32 = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i64 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %V2i64 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 9 for instruction: %V4i64 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> undef) ; CHECK-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V4i64 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void ; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
; ;
%V1i8 = call i8 @llvm.vector.reduce.add.v1i8(<1 x i8> undef) %V1i8 = call i8 @llvm.vector.reduce.add.v1i8(<1 x i8> undef)
%V3i8 = call i8 @llvm.vector.reduce.add.v3i8(<3 x i8> undef) %V3i8 = call i8 @llvm.vector.reduce.add.v3i8(<3 x i8> undef)
%V4i8 = call i8 @llvm.vector.reduce.add.v4i8(<4 x i8> undef) %V4i8 = call i8 @llvm.vector.reduce.add.v4i8(<4 x i8> undef)
%V8i8 = call i8 @llvm.vector.reduce.add.v8i8(<8 x i8> undef) %V8i8 = call i8 @llvm.vector.reduce.add.v8i8(<8 x i8> undef)
%V16i8 = call i8 @llvm.vector.reduce.add.v16i8(<16 x i8> undef) %V16i8 = call i8 @llvm.vector.reduce.add.v16i8(<16 x i8> undef)
%V32i8 = call i8 @llvm.vector.reduce.add.v32i8(<32 x i8> undef) %V32i8 = call i8 @llvm.vector.reduce.add.v32i8(<32 x i8> undef)
Show All 29 Lines

llvm/test/Transforms/SLPVectorizer/AArch64/reduce-add-i64.ll

Show First 20 Lines • Show All 59 Lines • ▼ Show 20 Linesentry:
%zext.3 = zext i8 %ld3 to i64 %zext.3 = zext i8 %ld3 to i64
%add.3 = add nuw nsw i64 %add.2, %zext.3 %add.3 = add nuw nsw i64 %add.2, %zext.3
ret i64 %add.3 ret i64 %add.3
} }
define i64 @red_zext_ld_8xi64(ptr %ptr) { define i64 @red_zext_ld_8xi64(ptr %ptr) {
; CHECK-LABEL: @red_zext_ld_8xi64( ; CHECK-LABEL: @red_zext_ld_8xi64(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[LD0:%.*]] = load i8, ptr [[PTR:%.*]], align 1 ; CHECK-NEXT: [[TMP0:%.*]] = load <8 x i8>, ptr [[PTR:%.*]], align 1
; CHECK-NEXT: [[ZEXT:%.*]] = zext i8 [[LD0]] to i64 ; CHECK-NEXT: [[TMP1:%.*]] = zext <8 x i8> [[TMP0]] to <8 x i64>
; CHECK-NEXT: [[GEP:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 1 ; CHECK-NEXT: [[TMP2:%.*]] = call i64 @llvm.vector.reduce.add.v8i64(<8 x i64> [[TMP1]])
; CHECK-NEXT: [[LD1:%.*]] = load i8, ptr [[GEP]], align 1 ; CHECK-NEXT: ret i64 [[TMP2]]
; CHECK-NEXT: [[ZEXT_1:%.*]] = zext i8 [[LD1]] to i64
; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i64 [[ZEXT]], [[ZEXT_1]]
; CHECK-NEXT: [[GEP_1:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 2
; CHECK-NEXT: [[LD2:%.*]] = load i8, ptr [[GEP_1]], align 1
; CHECK-NEXT: [[ZEXT_2:%.*]] = zext i8 [[LD2]] to i64
; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i64 [[ADD_1]], [[ZEXT_2]]
; CHECK-NEXT: [[GEP_2:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 3
; CHECK-NEXT: [[LD3:%.*]] = load i8, ptr [[GEP_2]], align 1
; CHECK-NEXT: [[ZEXT_3:%.*]] = zext i8 [[LD3]] to i64
; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i64 [[ADD_2]], [[ZEXT_3]]
; CHECK-NEXT: [[GEP_3:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 4
; CHECK-NEXT: [[LD4:%.*]] = load i8, ptr [[GEP_3]], align 1
; CHECK-NEXT: [[ZEXT_4:%.*]] = zext i8 [[LD4]] to i64
; CHECK-NEXT: [[ADD_4:%.*]] = add nuw nsw i64 [[ADD_3]], [[ZEXT_4]]
; CHECK-NEXT: [[GEP_4:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 5
; CHECK-NEXT: [[LD5:%.*]] = load i8, ptr [[GEP_4]], align 1
; CHECK-NEXT: [[ZEXT_5:%.*]] = zext i8 [[LD5]] to i64
; CHECK-NEXT: [[ADD_5:%.*]] = add nuw nsw i64 [[ADD_4]], [[ZEXT_5]]
; CHECK-NEXT: [[GEP_5:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 6
; CHECK-NEXT: [[LD6:%.*]] = load i8, ptr [[GEP_5]], align 1
; CHECK-NEXT: [[ZEXT_6:%.*]] = zext i8 [[LD6]] to i64
; CHECK-NEXT: [[ADD_6:%.*]] = add nuw nsw i64 [[ADD_5]], [[ZEXT_6]]
; CHECK-NEXT: [[GEP_6:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 7
; CHECK-NEXT: [[LD7:%.*]] = load i8, ptr [[GEP_6]], align 1
; CHECK-NEXT: [[ZEXT_7:%.*]] = zext i8 [[LD7]] to i64
; CHECK-NEXT: [[ADD_7:%.*]] = add nuw nsw i64 [[ADD_6]], [[ZEXT_7]]
; CHECK-NEXT: ret i64 [[ADD_7]]
; ;
entry: entry:
%ld0 = load i8, ptr %ptr %ld0 = load i8, ptr %ptr
%zext = zext i8 %ld0 to i64 %zext = zext i8 %ld0 to i64
%gep = getelementptr inbounds i8, ptr %ptr, i64 1 %gep = getelementptr inbounds i8, ptr %ptr, i64 1
%ld1 = load i8, ptr %gep %ld1 = load i8, ptr %gep
%zext.1 = zext i8 %ld1 to i64 %zext.1 = zext i8 %ld1 to i64
%add.1 = add nuw nsw i64 %zext, %zext.1 %add.1 = add nuw nsw i64 %zext, %zext.1
Show All 22 Linesentry:
%zext.7 = zext i8 %ld7 to i64 %zext.7 = zext i8 %ld7 to i64
%add.7 = add nuw nsw i64 %add.6, %zext.7 %add.7 = add nuw nsw i64 %add.6, %zext.7
ret i64 %add.7 ret i64 %add.7
} }
define i64 @red_zext_ld_16xi64(ptr %ptr) { define i64 @red_zext_ld_16xi64(ptr %ptr) {
; CHECK-LABEL: @red_zext_ld_16xi64( ; CHECK-LABEL: @red_zext_ld_16xi64(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[LD0:%.*]] = load i8, ptr [[PTR:%.*]], align 1 ; CHECK-NEXT: [[TMP0:%.*]] = load <16 x i8>, ptr [[PTR:%.*]], align 1
; CHECK-NEXT: [[ZEXT:%.*]] = zext i8 [[LD0]] to i64 ; CHECK-NEXT: [[TMP1:%.*]] = zext <16 x i8> [[TMP0]] to <16 x i64>
; CHECK-NEXT: [[GEP:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 1 ; CHECK-NEXT: [[TMP2:%.*]] = call i64 @llvm.vector.reduce.add.v16i64(<16 x i64> [[TMP1]])
; CHECK-NEXT: [[LD1:%.*]] = load i8, ptr [[GEP]], align 1 ; CHECK-NEXT: ret i64 [[TMP2]]
; CHECK-NEXT: [[ZEXT_1:%.*]] = zext i8 [[LD1]] to i64
; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i64 [[ZEXT]], [[ZEXT_1]]
; CHECK-NEXT: [[GEP_1:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 2
; CHECK-NEXT: [[LD2:%.*]] = load i8, ptr [[GEP_1]], align 1
; CHECK-NEXT: [[ZEXT_2:%.*]] = zext i8 [[LD2]] to i64
; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i64 [[ADD_1]], [[ZEXT_2]]
; CHECK-NEXT: [[GEP_2:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 3
; CHECK-NEXT: [[LD3:%.*]] = load i8, ptr [[GEP_2]], align 1
; CHECK-NEXT: [[ZEXT_3:%.*]] = zext i8 [[LD3]] to i64
; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i64 [[ADD_2]], [[ZEXT_3]]
; CHECK-NEXT: [[GEP_3:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 4
; CHECK-NEXT: [[LD4:%.*]] = load i8, ptr [[GEP_3]], align 1
; CHECK-NEXT: [[ZEXT_4:%.*]] = zext i8 [[LD4]] to i64
; CHECK-NEXT: [[ADD_4:%.*]] = add nuw nsw i64 [[ADD_3]], [[ZEXT_4]]
; CHECK-NEXT: [[GEP_4:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 5
; CHECK-NEXT: [[LD5:%.*]] = load i8, ptr [[GEP_4]], align 1
; CHECK-NEXT: [[ZEXT_5:%.*]] = zext i8 [[LD5]] to i64
; CHECK-NEXT: [[ADD_5:%.*]] = add nuw nsw i64 [[ADD_4]], [[ZEXT_5]]
; CHECK-NEXT: [[GEP_5:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 6
; CHECK-NEXT: [[LD6:%.*]] = load i8, ptr [[GEP_5]], align 1
; CHECK-NEXT: [[ZEXT_6:%.*]] = zext i8 [[LD6]] to i64
; CHECK-NEXT: [[ADD_6:%.*]] = add nuw nsw i64 [[ADD_5]], [[ZEXT_6]]
; CHECK-NEXT: [[GEP_6:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 7
; CHECK-NEXT: [[LD7:%.*]] = load i8, ptr [[GEP_6]], align 1
; CHECK-NEXT: [[ZEXT_7:%.*]] = zext i8 [[LD7]] to i64
; CHECK-NEXT: [[ADD_7:%.*]] = add nuw nsw i64 [[ADD_6]], [[ZEXT_7]]
; CHECK-NEXT: [[GEP_7:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 8
; CHECK-NEXT: [[LD8:%.*]] = load i8, ptr [[GEP_7]], align 1
; CHECK-NEXT: [[ZEXT_8:%.*]] = zext i8 [[LD8]] to i64
; CHECK-NEXT: [[ADD_8:%.*]] = add nuw nsw i64 [[ADD_7]], [[ZEXT_8]]
; CHECK-NEXT: [[GEP_8:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 9
; CHECK-NEXT: [[LD9:%.*]] = load i8, ptr [[GEP_8]], align 1
; CHECK-NEXT: [[ZEXT_9:%.*]] = zext i8 [[LD9]] to i64
; CHECK-NEXT: [[ADD_9:%.*]] = add nuw nsw i64 [[ADD_8]], [[ZEXT_9]]
; CHECK-NEXT: [[GEP_9:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 10
; CHECK-NEXT: [[LD10:%.*]] = load i8, ptr [[GEP_9]], align 1
; CHECK-NEXT: [[ZEXT_10:%.*]] = zext i8 [[LD10]] to i64
; CHECK-NEXT: [[ADD_10:%.*]] = add nuw nsw i64 [[ADD_9]], [[ZEXT_10]]
; CHECK-NEXT: [[GEP_10:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 11
; CHECK-NEXT: [[LD11:%.*]] = load i8, ptr [[GEP_10]], align 1
; CHECK-NEXT: [[ZEXT_11:%.*]] = zext i8 [[LD11]] to i64
; CHECK-NEXT: [[ADD_11:%.*]] = add nuw nsw i64 [[ADD_10]], [[ZEXT_11]]
; CHECK-NEXT: [[GEP_11:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 12
; CHECK-NEXT: [[LD12:%.*]] = load i8, ptr [[GEP_11]], align 1
; CHECK-NEXT: [[ZEXT_12:%.*]] = zext i8 [[LD12]] to i64
; CHECK-NEXT: [[ADD_12:%.*]] = add nuw nsw i64 [[ADD_11]], [[ZEXT_12]]
; CHECK-NEXT: [[GEP_12:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 13
; CHECK-NEXT: [[LD13:%.*]] = load i8, ptr [[GEP_12]], align 1
; CHECK-NEXT: [[ZEXT_13:%.*]] = zext i8 [[LD13]] to i64
; CHECK-NEXT: [[ADD_13:%.*]] = add nuw nsw i64 [[ADD_12]], [[ZEXT_13]]
; CHECK-NEXT: [[GEP_13:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 14
; CHECK-NEXT: [[LD14:%.*]] = load i8, ptr [[GEP_13]], align 1
; CHECK-NEXT: [[ZEXT_14:%.*]] = zext i8 [[LD14]] to i64
; CHECK-NEXT: [[ADD_14:%.*]] = add nuw nsw i64 [[ADD_13]], [[ZEXT_14]]
; CHECK-NEXT: [[GEP_14:%.*]] = getelementptr inbounds i8, ptr [[PTR]], i64 15
; CHECK-NEXT: [[LD15:%.*]] = load i8, ptr [[GEP_14]], align 1
; CHECK-NEXT: [[ZEXT_15:%.*]] = zext i8 [[LD15]] to i64
; CHECK-NEXT: [[ADD_15:%.*]] = add nuw nsw i64 [[ADD_14]], [[ZEXT_15]]
; CHECK-NEXT: ret i64 [[ADD_15]]
; ;
entry: entry:
%ld0 = load i8, ptr %ptr %ld0 = load i8, ptr %ptr
%zext = zext i8 %ld0 to i64 %zext = zext i8 %ld0 to i64
%gep = getelementptr inbounds i8, ptr %ptr, i64 1 %gep = getelementptr inbounds i8, ptr %ptr, i64 1
%ld1 = load i8, ptr %gep %ld1 = load i8, ptr %gep
%zext.1 = zext i8 %ld1 to i64 %zext.1 = zext i8 %ld1 to i64
%add.1 = add nuw nsw i64 %zext, %zext.1 %add.1 = add nuw nsw i64 %zext, %zext.1
▲ Show 20 Lines • Show All 73 Lines • ▼ Show 20 Linesentry:
%ld1 = load i64, ptr %gep %ld1 = load i64, ptr %gep
%add.1 = add nuw nsw i64 %ld0, %ld1 %add.1 = add nuw nsw i64 %ld0, %ld1
ret i64 %add.1 ret i64 %add.1
} }
define i64 @red_ld_4xi64(ptr %ptr) { define i64 @red_ld_4xi64(ptr %ptr) {
; CHECK-LABEL: @red_ld_4xi64( ; CHECK-LABEL: @red_ld_4xi64(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[LD0:%.*]] = load i64, ptr [[PTR:%.*]], align 4 ; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i64>, ptr [[PTR:%.*]], align 4
; CHECK-NEXT: [[GEP:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 1 ; CHECK-NEXT: [[TMP1:%.*]] = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> [[TMP0]])
; CHECK-NEXT: [[LD1:%.*]] = load i64, ptr [[GEP]], align 4 ; CHECK-NEXT: ret i64 [[TMP1]]
; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i64 [[LD0]], [[LD1]]
; CHECK-NEXT: [[GEP_1:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 2
; CHECK-NEXT: [[LD2:%.*]] = load i64, ptr [[GEP_1]], align 4
; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i64 [[ADD_1]], [[LD2]]
; CHECK-NEXT: [[GEP_2:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 3
; CHECK-NEXT: [[LD3:%.*]] = load i64, ptr [[GEP_2]], align 4
; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i64 [[ADD_2]], [[LD3]]
; CHECK-NEXT: ret i64 [[ADD_3]]
; ;
entry: entry:
%ld0 = load i64, ptr %ptr %ld0 = load i64, ptr %ptr
%gep = getelementptr inbounds i64, ptr %ptr, i64 1 %gep = getelementptr inbounds i64, ptr %ptr, i64 1
%ld1 = load i64, ptr %gep %ld1 = load i64, ptr %gep
%add.1 = add nuw nsw i64 %ld0, %ld1 %add.1 = add nuw nsw i64 %ld0, %ld1
%gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2 %gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2
%ld2 = load i64, ptr %gep.1 %ld2 = load i64, ptr %gep.1
%add.2 = add nuw nsw i64 %add.1, %ld2 %add.2 = add nuw nsw i64 %add.1, %ld2
%gep.2 = getelementptr inbounds i64, ptr %ptr, i64 3 %gep.2 = getelementptr inbounds i64, ptr %ptr, i64 3
%ld3 = load i64, ptr %gep.2 %ld3 = load i64, ptr %gep.2
%add.3 = add nuw nsw i64 %add.2, %ld3 %add.3 = add nuw nsw i64 %add.2, %ld3
ret i64 %add.3 ret i64 %add.3
} }
define i64 @red_ld_8xi64(ptr %ptr) { define i64 @red_ld_8xi64(ptr %ptr) {
; CHECK-LABEL: @red_ld_8xi64( ; CHECK-LABEL: @red_ld_8xi64(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[LD0:%.*]] = load i64, ptr [[PTR:%.*]], align 4 ; CHECK-NEXT: [[TMP0:%.*]] = load <8 x i64>, ptr [[PTR:%.*]], align 4
; CHECK-NEXT: [[GEP:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 1 ; CHECK-NEXT: [[TMP1:%.*]] = call i64 @llvm.vector.reduce.add.v8i64(<8 x i64> [[TMP0]])
; CHECK-NEXT: [[LD1:%.*]] = load i64, ptr [[GEP]], align 4 ; CHECK-NEXT: ret i64 [[TMP1]]
; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i64 [[LD0]], [[LD1]]
; CHECK-NEXT: [[GEP_1:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 2
; CHECK-NEXT: [[LD2:%.*]] = load i64, ptr [[GEP_1]], align 4
; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i64 [[ADD_1]], [[LD2]]
; CHECK-NEXT: [[GEP_2:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 3
; CHECK-NEXT: [[LD3:%.*]] = load i64, ptr [[GEP_2]], align 4
; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i64 [[ADD_2]], [[LD3]]
; CHECK-NEXT: [[GEP_3:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 4
; CHECK-NEXT: [[LD4:%.*]] = load i64, ptr [[GEP_3]], align 4
; CHECK-NEXT: [[ADD_4:%.*]] = add nuw nsw i64 [[ADD_3]], [[LD4]]
; CHECK-NEXT: [[GEP_4:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 5
; CHECK-NEXT: [[LD5:%.*]] = load i64, ptr [[GEP_4]], align 4
; CHECK-NEXT: [[ADD_5:%.*]] = add nuw nsw i64 [[ADD_4]], [[LD5]]
; CHECK-NEXT: [[GEP_5:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 6
; CHECK-NEXT: [[LD6:%.*]] = load i64, ptr [[GEP_5]], align 4
; CHECK-NEXT: [[ADD_6:%.*]] = add nuw nsw i64 [[ADD_5]], [[LD6]]
; CHECK-NEXT: [[GEP_6:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 7
; CHECK-NEXT: [[LD7:%.*]] = load i64, ptr [[GEP_6]], align 4
; CHECK-NEXT: [[ADD_7:%.*]] = add nuw nsw i64 [[ADD_6]], [[LD7]]
; CHECK-NEXT: ret i64 [[ADD_7]]
; ;
entry: entry:
%ld0 = load i64, ptr %ptr %ld0 = load i64, ptr %ptr
%gep = getelementptr inbounds i64, ptr %ptr, i64 1 %gep = getelementptr inbounds i64, ptr %ptr, i64 1
%ld1 = load i64, ptr %gep %ld1 = load i64, ptr %gep
%add.1 = add nuw nsw i64 %ld0, %ld1 %add.1 = add nuw nsw i64 %ld0, %ld1
%gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2 %gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2
%ld2 = load i64, ptr %gep.1 %ld2 = load i64, ptr %gep.1
Show All 14 Linesentry:
%ld7 = load i64, ptr %gep.6 %ld7 = load i64, ptr %gep.6
%add.7 = add nuw nsw i64 %add.6, %ld7 %add.7 = add nuw nsw i64 %add.6, %ld7
ret i64 %add.7 ret i64 %add.7
} }
define i64 @red_ld_16xi64(ptr %ptr) { define i64 @red_ld_16xi64(ptr %ptr) {
; CHECK-LABEL: @red_ld_16xi64( ; CHECK-LABEL: @red_ld_16xi64(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[LD0:%.*]] = load i64, ptr [[PTR:%.*]], align 4 ; CHECK-NEXT: [[TMP0:%.*]] = load <16 x i64>, ptr [[PTR:%.*]], align 4
; CHECK-NEXT: [[GEP:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 1 ; CHECK-NEXT: [[TMP1:%.*]] = call i64 @llvm.vector.reduce.add.v16i64(<16 x i64> [[TMP0]])
; CHECK-NEXT: [[LD1:%.*]] = load i64, ptr [[GEP]], align 4 ; CHECK-NEXT: ret i64 [[TMP1]]
; CHECK-NEXT: [[ADD_1:%.*]] = add nuw nsw i64 [[LD0]], [[LD1]]
; CHECK-NEXT: [[GEP_1:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 2
; CHECK-NEXT: [[LD2:%.*]] = load i64, ptr [[GEP_1]], align 4
; CHECK-NEXT: [[ADD_2:%.*]] = add nuw nsw i64 [[ADD_1]], [[LD2]]
; CHECK-NEXT: [[GEP_2:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 3
; CHECK-NEXT: [[LD3:%.*]] = load i64, ptr [[GEP_2]], align 4
; CHECK-NEXT: [[ADD_3:%.*]] = add nuw nsw i64 [[ADD_2]], [[LD3]]
; CHECK-NEXT: [[GEP_3:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 4
; CHECK-NEXT: [[LD4:%.*]] = load i64, ptr [[GEP_3]], align 4
; CHECK-NEXT: [[ADD_4:%.*]] = add nuw nsw i64 [[ADD_3]], [[LD4]]
; CHECK-NEXT: [[GEP_4:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 5
; CHECK-NEXT: [[LD5:%.*]] = load i64, ptr [[GEP_4]], align 4
; CHECK-NEXT: [[ADD_5:%.*]] = add nuw nsw i64 [[ADD_4]], [[LD5]]
; CHECK-NEXT: [[GEP_5:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 6
; CHECK-NEXT: [[LD6:%.*]] = load i64, ptr [[GEP_5]], align 4
; CHECK-NEXT: [[ADD_6:%.*]] = add nuw nsw i64 [[ADD_5]], [[LD6]]
; CHECK-NEXT: [[GEP_6:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 7
; CHECK-NEXT: [[LD7:%.*]] = load i64, ptr [[GEP_6]], align 4
; CHECK-NEXT: [[ADD_7:%.*]] = add nuw nsw i64 [[ADD_6]], [[LD7]]
; CHECK-NEXT: [[GEP_7:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 8
; CHECK-NEXT: [[LD8:%.*]] = load i64, ptr [[GEP_7]], align 4
; CHECK-NEXT: [[ADD_8:%.*]] = add nuw nsw i64 [[ADD_7]], [[LD8]]
; CHECK-NEXT: [[GEP_8:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 9
; CHECK-NEXT: [[LD9:%.*]] = load i64, ptr [[GEP_8]], align 4
; CHECK-NEXT: [[ADD_9:%.*]] = add nuw nsw i64 [[ADD_8]], [[LD9]]
; CHECK-NEXT: [[GEP_9:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 10
; CHECK-NEXT: [[LD10:%.*]] = load i64, ptr [[GEP_9]], align 4
; CHECK-NEXT: [[ADD_10:%.*]] = add nuw nsw i64 [[ADD_9]], [[LD10]]
; CHECK-NEXT: [[GEP_10:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 11
; CHECK-NEXT: [[LD11:%.*]] = load i64, ptr [[GEP_10]], align 4
; CHECK-NEXT: [[ADD_11:%.*]] = add nuw nsw i64 [[ADD_10]], [[LD11]]
; CHECK-NEXT: [[GEP_11:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 12
; CHECK-NEXT: [[LD12:%.*]] = load i64, ptr [[GEP_11]], align 4
; CHECK-NEXT: [[ADD_12:%.*]] = add nuw nsw i64 [[ADD_11]], [[LD12]]
; CHECK-NEXT: [[GEP_12:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 13
; CHECK-NEXT: [[LD13:%.*]] = load i64, ptr [[GEP_12]], align 4
; CHECK-NEXT: [[ADD_13:%.*]] = add nuw nsw i64 [[ADD_12]], [[LD13]]
; CHECK-NEXT: [[GEP_13:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 14
; CHECK-NEXT: [[LD14:%.*]] = load i64, ptr [[GEP_13]], align 4
; CHECK-NEXT: [[ADD_14:%.*]] = add nuw nsw i64 [[ADD_13]], [[LD14]]
; CHECK-NEXT: [[GEP_14:%.*]] = getelementptr inbounds i64, ptr [[PTR]], i64 15
; CHECK-NEXT: [[LD15:%.*]] = load i64, ptr [[GEP_14]], align 4
; CHECK-NEXT: [[ADD_15:%.*]] = add nuw nsw i64 [[ADD_14]], [[LD15]]
; CHECK-NEXT: ret i64 [[ADD_15]]
; ;
entry: entry:
%ld0 = load i64, ptr %ptr %ld0 = load i64, ptr %ptr
%gep = getelementptr inbounds i64, ptr %ptr, i64 1 %gep = getelementptr inbounds i64, ptr %ptr, i64 1
%ld1 = load i64, ptr %gep %ld1 = load i64, ptr %gep
%add.1 = add nuw nsw i64 %ld0, %ld1 %add.1 = add nuw nsw i64 %ld0, %ld1
%gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2 %gep.1 = getelementptr inbounds i64, ptr %ptr, i64 2
%ld2 = load i64, ptr %gep.1 %ld2 = load i64, ptr %gep.1
▲ Show 20 Lines • Show All 42 Lines • Show Last 20 Lines