This is an archive of the discontinued LLVM Phabricator instance.

Differential D88787

[llvm][mlir] Promote the experimental reduction intrinsics to be first class intrinsics.
ClosedPublic

Authored by aemerson on Oct 3 2020, 12:27 PM.

Details

Reviewers
lattner
nikic
efriedma
craig.topper
spatel
simoll
dmgreen
RKSimon
sdesmalen
reames
aartbik
rriddle
nicolasvasilache
ftynse
jdoerfert
Commits
rG322d0afd875d: [llvm][mlir] Promote the experimental reduction intrinsics to be first class…
Summary

This change renames the intrinsics to not have "experimental" in the name.

The autoupgrader will handle legacy intrinsics.

Relevant ML thread: http://lists.llvm.org/pipermail/llvm-dev/2020-April/140729.html

Diff Detail

Event Timeline

aemerson created this revision.Oct 3 2020, 12:27 PM
Herald added a reviewer: ftynse. · View Herald TranscriptOct 3 2020, 12:27 PM
Herald added a reviewer: jdoerfert. · View Herald Transcript
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: tatianashp, msifontes, jurahul and 13 others. · View Herald Transcript
aemerson requested review of this revision.Oct 3 2020, 12:27 PM
Herald added a subscriber: stephenneuendorffer. · View Herald TranscriptOct 3 2020, 12:27 PM
Harbormaster completed remote builds in B73887: Diff 295998.Oct 3 2020, 12:27 PM
aemerson added a comment.Oct 3 2020, 12:28 PM

@aartbik @rriddle @nicolasvasilache MLIR also needed updating to use these new intrinsics, please take a look.

nikic added inline comments.Oct 3 2020, 1:05 PM
llvm/docs/LangRef.rst
15547

Underlines for titles need adjustment.

llvm/test/Bitcode/upgrade-vecreduce-intrinsics.ll
87

Please also add fast variants for these two tests, to make sure the operand is not replaced with 0/1 if the v2 intrinsics were used.

RKSimon added inline comments.Oct 3 2020, 1:12 PM
llvm/lib/IR/AutoUpgrade.cpp
3647–3648

Can we not just drop support for the v1 experimental fmul/fadd now? As @nikic said there is a danger of this matching against the v2 variants upgraded above as well.

aemerson added inline comments.Oct 3 2020, 4:25 PM
llvm/lib/IR/AutoUpgrade.cpp
3647–3648

Yeah I think we can probably drop support. I'll remove this special handling so we're only effectively renaming the existing set.

aemerson updated this revision to Diff 296010.Oct 3 2020, 4:44 PM
aemerson marked an inline comment as done.

Don't upgrade v1 intrinsics.

Harbormaster completed remote builds in B73895: Diff 296010.Oct 3 2020, 4:44 PM
ftynse added a comment.Oct 5 2020, 1:31 AM

@aartbik @rriddle @nicolasvasilache MLIR also needed updating to use these new intrinsics, please take a look.

MLIR part LGTM

RKSimon added a comment.Oct 5 2020, 2:20 AM

Please can you add an entry to the llvm release docs

aemerson updated this revision to Diff 296236.Oct 5 2020, 11:10 AM

Add a note to the release notes.

Harbormaster completed remote builds in B74033: Diff 296236.Oct 5 2020, 11:10 AM
nikic added inline comments.Oct 5 2020, 11:58 AM
llvm/docs/LangRef.rst
15556

There's more underlines that need adjustment here.

llvm/lib/IR/AutoUpgrade.cpp
745

Seeing this, I'm wondering whether we shouldn't change the type signature for fadd/fmul to omit the separate scalar type. It should be sufficient to have vector.reduce.fadd.v4f32 rather than vector.reduce.f32.v4f32. The scalar argument has to match the vector type, just the like return type already has to match.

spatel added inline comments.Oct 5 2020, 12:56 PM
llvm/docs/LangRef.rst
15597

Given that we're using the term "sequential" in D88791 (and I think that's a good change), should we do the same here for consistency? That could be considered an improvement independently of removing "experimental".

aemerson added inline comments.Oct 5 2020, 5:35 PM
llvm/docs/LangRef.rst
15597

Sounds like a good idea. I'll do that as a separate change after D88791 lands.

aemerson updated this revision to Diff 296603.Oct 6 2020, 10:27 PM

Remove the scalar input overload for fadd/fmul. Now the signature is simple llvm.reduce.fadd.v4f32 rather than llvm.reduce.fadd.f32.v4f32

Also fix some langref formatting, and fold in the change to use "sequential" instead of "ordered" while I'm there.

Harbormaster completed remote builds in B74228: Diff 296603.Oct 6 2020, 10:28 PM
RKSimon mentioned this in D88026: [X86][SSE2] Follow up patch for D87236 to avoid the regression regarding horizontal unsigned 16 bit minimums and maximums..Oct 7 2020, 2:27 AM
RKSimon added a comment.Oct 7 2020, 2:33 AM

One last minor comment - anyone else?

llvm/test/Instrumentation/MemorySanitizer/experimental-reduce.ll
2

Should we rename this file to reduce.ll?

spatel accepted this revision.Oct 7 2020, 10:00 AM

LGTM

llvm/docs/LangRef.rst
15595

It's redundant/unnecessary to mention 'fast' here (and in the fmul overview) because 'fast' is just an alias for all FMF including 'reassoc'.

This revision is now accepted and ready to land.Oct 7 2020, 10:00 AM
aemerson added a comment.Oct 7 2020, 10:29 AM

I'll make those last changes in the commit.

Thanks to everyone who's pushed this forward over the last few years with fixes and reviews!

This revision was landed with ongoing or failed builds.Oct 7 2020, 10:37 AM
Closed by commit rG322d0afd875d: [llvm][mlir] Promote the experimental reduction intrinsics to be first class… (authored by aemerson). · Explain Why
This revision was automatically updated to reflect the committed changes.
aemerson added a commit: rG322d0afd875d: [llvm][mlir] Promote the experimental reduction intrinsics to be first class….
NathanielMcVicar added a subscriber: NathanielMcVicar.Oct 7 2020, 11:18 AM

I think there may be a small remaining issue on the llvmir-intrinsics.mlir test on Windows, as seen here: http://lab.llvm.org:8011/#/builders/82/builds/40/steps/7/logs/stdio

$ ":" "RUN: at line 1"
$ "e:\build_slave\mlir-x64-windows-ninja\build\bin\mlir-translate.exe" "-mlir-to-llvmir" "E:\build_slave\mlir-x64-windows-ninja\llvm-project\mlir\test\Target\llvmir-intrinsics.mlir"
# command stderr:
Expected an argument of Vector Type
UNREACHABLE executed at E:\build_slave\mlir-x64-windows-ninja\llvm-project\llvm\lib\IR\Function.cpp:1167!
aemerson added a comment.Oct 7 2020, 11:23 AM
In D88787#2317464, @NathanielMcVicar wrote:

I think there may be a small remaining issue on the llvmir-intrinsics.mlir test on Windows, as seen here: http://lab.llvm.org:8011/#/builders/82/builds/40/steps/7/logs/stdio

$ ":" "RUN: at line 1"
$ "e:\build_slave\mlir-x64-windows-ninja\build\bin\mlir-translate.exe" "-mlir-to-llvmir" "E:\build_slave\mlir-x64-windows-ninja\llvm-project\mlir\test\Target\llvmir-intrinsics.mlir"
# command stderr:
Expected an argument of Vector Type
UNREACHABLE executed at E:\build_slave\mlir-x64-windows-ninja\llvm-project\llvm\lib\IR\Function.cpp:1167!

Yep, I've just pushed c1247f0e74bff00ab9a896a8132318916f3e84a7 to hopefully fix that.

dmgreen mentioned this in rG81b4f3380b01: [ARM] Replace llvm.experimental.vector.reduce.smax with llvm.vector.reduce.smax..Oct 8 2020, 12:06 AM

Large Diff

This large diff affects 183 files. Files without inline comments have been collapsed. Expand All Files

Revision Contents

PathSize
llvm/
docs/
156 lines
4 lines
include/
llvm/
CodeGen/
52 lines
IR/
57 lines
lib/
Analysis/
72 lines
CodeGen/
92 lines
SelectionDAG/
52 lines
IR/
57 lines
39 lines
Target/
AArch64/
4 lines
ARM/
4 lines
2 lines
Transforms/
InstCombine/
17 lines
Instrumentation/
14 lines
test/
Analysis/
CostModel/
AArch64/
174 lines
ARM/
214 lines
230 lines
230 lines
230 lines
230 lines
X86/
374 lines
358 lines
110 lines
110 lines
368 lines
358 lines
430 lines
430 lines
430 lines
430 lines
390 lines
Assembler/
28 lines
Bitcode/
142 lines
CodeGen/
AArch64/
24 lines
92 lines
12 lines
16 lines
154 lines
518 lines
58 lines
58 lines
48 lines
28 lines
28 lines
32 lines
28 lines
30 lines
30 lines
28 lines
4 lines
62 lines
ARM/
16 lines
28 lines
16 lines
16 lines
16 lines
28 lines
Generic/
74 lines
Thumb2/
LowOverheadLoops/
4 lines
10 lines
4 lines
4 lines
6 lines
6 lines
4 lines
12 lines
10 lines
4 lines
4 lines
4 lines
14 lines
28 lines
4 lines
4 lines
8 lines
4 lines
50 lines
4 lines
4 lines
4 lines
8 lines
6 lines
4 lines
4 lines
4 lines
4 lines
8 lines
8 lines
2 lines
42 lines
132 lines
132 lines
216 lines
58 lines
188 lines
60 lines
106 lines
156 lines
156 lines
72 lines
88 lines
X86/
8 lines
4 lines
94 lines
60 lines
88 lines
88 lines
64 lines
64 lines
32 lines
40 lines
40 lines
44 lines
36 lines
64 lines
64 lines
88 lines
60 lines
98 lines
88 lines
88 lines
88 lines
88 lines
88 lines
60 lines
88 lines
Instrumentation/
MemorySanitizer/
reduce.ll
experimental-reduce.ll
28 lines
Transforms/
InstCombine/
76 lines
InstSimplify/
ConstProp/
180 lines
LoopVectorize/
AArch64/
2 lines
6 lines
ARM/
2 lines
20 lines
26 lines
24 lines
4 lines
8 lines
X86/
2 lines
2 lines
20 lines
2 lines
2 lines
6 lines
2 lines
2 lines
8 lines
60 lines
20 lines
PhaseOrdering/
X86/
10 lines
SLPVectorizer/
AArch64/
2 lines
10 lines
6 lines
2 lines
X86/
2 lines
2 lines
4 lines
2 lines
8 lines
52 lines
14 lines
40 lines
2 lines
2 lines
6 lines
12 lines
2 lines
4 lines
8 lines
2 lines
2 lines
2 lines
6 lines
mlir/
include/
mlir/
Dialect/
LLVMIR/
8 lines
28 lines
Vector/
2 lines
integration_test/
Dialect/
LLVMIR/
CPU/
20 lines
18 lines
lib/
Conversion/
VectorToLLVM/
26 lines
test/
Conversion/
VectorToLLVM/
8 lines
12 lines
Target/
60 lines

Diff 296735

llvm/docs/LangRef.rst

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 15,537 Lines • ▼ Show 20 Lines
"""""""""" """"""""""
The '``llvm.loop.decrement.*``' intrinsics do a ``SUB`` of the loop iteration The '``llvm.loop.decrement.*``' intrinsics do a ``SUB`` of the loop iteration
counter with the given loop decrement value, and return false if the loop counter with the given loop decrement value, and return false if the loop
should exit, this ``SUB`` is not allowed to wrap. The result is a condition should exit, this ``SUB`` is not allowed to wrap. The result is a condition
that is used by the conditional branch controlling the loop. that is used by the conditional branch controlling the loop.
Experimental Vector Reduction Intrinsics Vector Reduction Intrinsics
---------------------------------------- ---------------------------
nikicUnsubmitted
Done
ReplyInline Actions

Underlines for titles need adjustment.

nikic: Underlines for titles need adjustment.
Horizontal reductions of vectors can be expressed using the following Horizontal reductions of vectors can be expressed using the following
intrinsics. Each one takes a vector operand as an input and applies its intrinsics. Each one takes a vector operand as an input and applies its
respective operation across all elements of the vector, returning a single respective operation across all elements of the vector, returning a single
scalar result of the same element type. scalar result of the same element type.
'``llvm.experimental.vector.reduce.add.*``' Intrinsic '``llvm.vector.reduce.add.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nikicUnsubmitted
Not Done
ReplyInline Actions

There's more underlines that need adjustment here.

nikic: There's more underlines that need adjustment here.
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.add.v4i32(<4 x i32> %a)
declare i64 @llvm.experimental.vector.reduce.add.v2i64(<2 x i64> %a) declare i64 @llvm.vector.reduce.add.v2i64(<2 x i64> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.add.*``' intrinsics do an integer ``ADD`` The '``llvm.vector.reduce.add.*``' intrinsics do an integer ``ADD``
reduction of a vector, returning the result as a scalar. The return type matches reduction of a vector, returning the result as a scalar. The return type matches
the element-type of the vector input. the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.v2.fadd.*``' Intrinsic '``llvm.vector.reduce.fadd.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float %start_value, <4 x float> %a) declare float @llvm.vector.reduce.fadd.v4f32(float %start_value, <4 x float> %a)
declare double @llvm.experimental.vector.reduce.v2.fadd.f64.v2f64(double %start_value, <2 x double> %a) declare double @llvm.vector.reduce.fadd.v2f64(double %start_value, <2 x double> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.v2.fadd.*``' intrinsics do a floating-point The '``llvm.vector.reduce.fadd.*``' intrinsics do a floating-point
``ADD`` reduction of a vector, returning the result as a scalar. The return type ``ADD`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
If the intrinsic call has the 'reassoc' or 'fast' flags set, then the If the intrinsic call has the 'reassoc' flag set, then the reduction will not
spatelUnsubmitted
Not Done
ReplyInline Actions

It's redundant/unnecessary to mention 'fast' here (and in the fmul overview) because 'fast' is just an alias for all FMF including 'reassoc'.

spatel: It's redundant/unnecessary to mention 'fast' here (and in the fmul overview) because 'fast' is…
reduction will not preserve the associativity of an equivalent scalarized preserve the associativity of an equivalent scalarized counterpart. Otherwise
counterpart. Otherwise the reduction will be *ordered*, thus implying that the reduction will be *sequential*, thus implying that the operation respects
spatelUnsubmitted
Not Done
ReplyInline Actions

Given that we're using the term "sequential" in D88791 (and I think that's a good change), should we do the same here for consistency? That could be considered an improvement independently of removing "experimental".

spatel: Given that we're using the term "sequential" in D88791 (and I think that's a good change)…
aemersonAuthorUnsubmitted
Done
ReplyInline Actions

Sounds like a good idea. I'll do that as a separate change after D88791 lands.

aemerson: Sounds like a good idea. I'll do that as a separate change after D88791 lands.
the operation respects the associativity of a scalarized reduction. That is, the the associativity of a scalarized reduction. That is, the reduction begins with
reduction begins with the start value and performs an fadd operation with consecutively the start value and performs an fadd operation with consecutively increasing
increasing vector element indices. See the following pseudocode: vector element indices. See the following pseudocode:
:: ::
float ordered_fadd(start_value, input_vector) float sequential_fadd(start_value, input_vector)
result = start_value result = start_value
for i = 0 to length(input_vector) for i = 0 to length(input_vector)
result = result + input_vector[i] result = result + input_vector[i]
return result return result
Arguments: Arguments:
"""""""""" """"""""""
The first argument to this intrinsic is a scalar start value for the reduction. The first argument to this intrinsic is a scalar start value for the reduction.
The type of the start value matches the element-type of the vector input. The type of the start value matches the element-type of the vector input.
The second argument must be a vector of floating-point values. The second argument must be a vector of floating-point values.
Examples: Examples:
""""""""" """""""""
:: ::
%unord = call reassoc float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.0, <4 x float> %input) ; unordered reduction %unord = call reassoc float @llvm.vector.reduce.fadd.v4f32(float 0.0, <4 x float> %input) ; relaxed reduction
%ord = call float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float %start_value, <4 x float> %input) ; ordered reduction %ord = call float @llvm.vector.reduce.fadd.v4f32(float %start_value, <4 x float> %input) ; sequential reduction
'``llvm.experimental.vector.reduce.mul.*``' Intrinsic '``llvm.vector.reduce.mul.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> %a)
declare i64 @llvm.experimental.vector.reduce.mul.v2i64(<2 x i64> %a) declare i64 @llvm.vector.reduce.mul.v2i64(<2 x i64> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.mul.*``' intrinsics do an integer ``MUL`` The '``llvm.vector.reduce.mul.*``' intrinsics do an integer ``MUL``
reduction of a vector, returning the result as a scalar. The return type matches reduction of a vector, returning the result as a scalar. The return type matches
the element-type of the vector input. the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.v2.fmul.*``' Intrinsic '``llvm.vector.reduce.fmul.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float %start_value, <4 x float> %a) declare float @llvm.vector.reduce.fmul.v4f32(float %start_value, <4 x float> %a)
declare double @llvm.experimental.vector.reduce.v2.fmul.f64.v2f64(double %start_value, <2 x double> %a) declare double @llvm.vector.reduce.fmul.v2f64(double %start_value, <2 x double> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.v2.fmul.*``' intrinsics do a floating-point The '``llvm.vector.reduce.fmul.*``' intrinsics do a floating-point
``MUL`` reduction of a vector, returning the result as a scalar. The return type ``MUL`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
If the intrinsic call has the 'reassoc' or 'fast' flags set, then the If the intrinsic call has the 'reassoc' flag set, then the reduction will not
reduction will not preserve the associativity of an equivalent scalarized preserve the associativity of an equivalent scalarized counterpart. Otherwise
counterpart. Otherwise the reduction will be *ordered*, thus implying that the reduction will be *sequential*, thus implying that the operation respects
the operation respects the associativity of a scalarized reduction. That is, the the associativity of a scalarized reduction. That is, the reduction begins with
reduction begins with the start value and performs an fmul operation with consecutively the start value and performs an fmul operation with consecutively increasing
increasing vector element indices. See the following pseudocode: vector element indices. See the following pseudocode:
:: ::
float ordered_fmul(start_value, input_vector) float sequential_fmul(start_value, input_vector)
result = start_value result = start_value
for i = 0 to length(input_vector) for i = 0 to length(input_vector)
result = result * input_vector[i] result = result * input_vector[i]
return result return result
Arguments: Arguments:
"""""""""" """"""""""
The first argument to this intrinsic is a scalar start value for the reduction. The first argument to this intrinsic is a scalar start value for the reduction.
The type of the start value matches the element-type of the vector input. The type of the start value matches the element-type of the vector input.
The second argument must be a vector of floating-point values. The second argument must be a vector of floating-point values.
Examples: Examples:
""""""""" """""""""
:: ::
%unord = call reassoc float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.0, <4 x float> %input) ; unordered reduction %unord = call reassoc float @llvm.vector.reduce.fmul.v4f32(float 1.0, <4 x float> %input) ; relaxed reduction
%ord = call float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float %start_value, <4 x float> %input) ; ordered reduction %ord = call float @llvm.vector.reduce.fmul.v4f32(float %start_value, <4 x float> %input) ; sequential reduction
'``llvm.experimental.vector.reduce.and.*``' Intrinsic '``llvm.vector.reduce.and.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.and.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.and.*``' intrinsics do a bitwise ``AND`` The '``llvm.vector.reduce.and.*``' intrinsics do a bitwise ``AND``
reduction of a vector, returning the result as a scalar. The return type matches reduction of a vector, returning the result as a scalar. The return type matches
the element-type of the vector input. the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.or.*``' Intrinsic '``llvm.vector.reduce.or.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.or.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.or.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.or.*``' intrinsics do a bitwise ``OR`` reduction The '``llvm.vector.reduce.or.*``' intrinsics do a bitwise ``OR`` reduction
of a vector, returning the result as a scalar. The return type matches the of a vector, returning the result as a scalar. The return type matches the
element-type of the vector input. element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.xor.*``' Intrinsic '``llvm.vector.reduce.xor.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.xor.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.xor.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.xor.*``' intrinsics do a bitwise ``XOR`` The '``llvm.vector.reduce.xor.*``' intrinsics do a bitwise ``XOR``
reduction of a vector, returning the result as a scalar. The return type matches reduction of a vector, returning the result as a scalar. The return type matches
the element-type of the vector input. the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.smax.*``' Intrinsic '``llvm.vector.reduce.smax.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.smax.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.smax.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.smax.*``' intrinsics do a signed integer The '``llvm.vector.reduce.smax.*``' intrinsics do a signed integer
``MAX`` reduction of a vector, returning the result as a scalar. The return type ``MAX`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.smin.*``' Intrinsic '``llvm.vector.reduce.smin.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.smin.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.smin.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.smin.*``' intrinsics do a signed integer The '``llvm.vector.reduce.smin.*``' intrinsics do a signed integer
``MIN`` reduction of a vector, returning the result as a scalar. The return type ``MIN`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.umax.*``' Intrinsic '``llvm.vector.reduce.umax.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.umax.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.umax.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.umax.*``' intrinsics do an unsigned The '``llvm.vector.reduce.umax.*``' intrinsics do an unsigned
integer ``MAX`` reduction of a vector, returning the result as a scalar. The integer ``MAX`` reduction of a vector, returning the result as a scalar. The
return type matches the element-type of the vector input. return type matches the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.umin.*``' Intrinsic '``llvm.vector.reduce.umin.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare i32 @llvm.experimental.vector.reduce.umin.v4i32(<4 x i32> %a) declare i32 @llvm.vector.reduce.umin.v4i32(<4 x i32> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.umin.*``' intrinsics do an unsigned The '``llvm.vector.reduce.umin.*``' intrinsics do an unsigned
integer ``MIN`` reduction of a vector, returning the result as a scalar. The integer ``MIN`` reduction of a vector, returning the result as a scalar. The
return type matches the element-type of the vector input. return type matches the element-type of the vector input.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of integer values. The argument to this intrinsic must be a vector of integer values.
'``llvm.experimental.vector.reduce.fmax.*``' Intrinsic '``llvm.vector.reduce.fmax.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
:: ::
declare float @llvm.experimental.vector.reduce.fmax.v4f32(<4 x float> %a) declare float @llvm.vector.reduce.fmax.v4f32(<4 x float> %a)
declare double @llvm.experimental.vector.reduce.fmax.v2f64(<2 x double> %a) declare double @llvm.vector.reduce.fmax.v2f64(<2 x double> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.fmax.*``' intrinsics do a floating-point The '``llvm.vector.reduce.fmax.*``' intrinsics do a floating-point
``MAX`` reduction of a vector, returning the result as a scalar. The return type ``MAX`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
This instruction has the same comparison semantics as the '``llvm.maxnum.*``' This instruction has the same comparison semantics as the '``llvm.maxnum.*``'
intrinsic. That is, the result will always be a number unless all elements of intrinsic. That is, the result will always be a number unless all elements of
the vector are NaN. For a vector with maximum element magnitude 0.0 and the vector are NaN. For a vector with maximum element magnitude 0.0 and
containing both +0.0 and -0.0 elements, the sign of the result is unspecified. containing both +0.0 and -0.0 elements, the sign of the result is unspecified.
If the intrinsic call has the ``nnan`` fast-math flag, then the operation can If the intrinsic call has the ``nnan`` fast-math flag, then the operation can
assume that NaNs are not present in the input vector. assume that NaNs are not present in the input vector.
Arguments: Arguments:
"""""""""" """"""""""
The argument to this intrinsic must be a vector of floating-point values. The argument to this intrinsic must be a vector of floating-point values.
'``llvm.experimental.vector.reduce.fmin.*``' Intrinsic '``llvm.vector.reduce.fmin.*``' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Syntax: Syntax:
""""""" """""""
This is an overloaded intrinsic. This is an overloaded intrinsic.
:: ::
declare float @llvm.experimental.vector.reduce.fmin.v4f32(<4 x float> %a) declare float @llvm.vector.reduce.fmin.v4f32(<4 x float> %a)
declare double @llvm.experimental.vector.reduce.fmin.v2f64(<2 x double> %a) declare double @llvm.vector.reduce.fmin.v2f64(<2 x double> %a)
Overview: Overview:
""""""""" """""""""
The '``llvm.experimental.vector.reduce.fmin.*``' intrinsics do a floating-point The '``llvm.vector.reduce.fmin.*``' intrinsics do a floating-point
``MIN`` reduction of a vector, returning the result as a scalar. The return type ``MIN`` reduction of a vector, returning the result as a scalar. The return type
matches the element-type of the vector input. matches the element-type of the vector input.
This instruction has the same comparison semantics as the '``llvm.minnum.*``' This instruction has the same comparison semantics as the '``llvm.minnum.*``'
intrinsic. That is, the result will always be a number unless all elements of intrinsic. That is, the result will always be a number unless all elements of
the vector are NaN. For a vector with minimum element magnitude 0.0 and the vector are NaN. For a vector with minimum element magnitude 0.0 and
containing both +0.0 and -0.0 elements, the sign of the result is unspecified. containing both +0.0 and -0.0 elements, the sign of the result is unspecified.
▲ Show 20 Lines • Show All 4,921 Lines • Show Last 20 Lines

llvm/docs/ReleaseNotes.rst

llvm/include/llvm/CodeGen/BasicTTIImpl.h

llvm/include/llvm/IR/Intrinsics.td

llvm/lib/Analysis/ConstantFolding.cpp

llvm/lib/CodeGen/ExpandReductions.cpp

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

llvm/lib/IR/AutoUpgrade.cpp

Show All 17 Lines
#include "llvm/IR/DIBuilder.h" #include "llvm/IR/DIBuilder.h"
#include "llvm/IR/DebugInfo.h" #include "llvm/IR/DebugInfo.h"
#include "llvm/IR/DiagnosticInfo.h" #include "llvm/IR/DiagnosticInfo.h"
#include "llvm/IR/Function.h" #include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h" #include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instruction.h" #include "llvm/IR/Instruction.h"
#include "llvm/IR/InstVisitor.h" #include "llvm/IR/InstVisitor.h"
#include "llvm/IR/IntrinsicInst.h" #include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/IntrinsicsAArch64.h" #include "llvm/IR/IntrinsicsAArch64.h"
#include "llvm/IR/IntrinsicsARM.h" #include "llvm/IR/IntrinsicsARM.h"
#include "llvm/IR/IntrinsicsX86.h" #include "llvm/IR/IntrinsicsX86.h"
#include "llvm/IR/LLVMContext.h" #include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Module.h" #include "llvm/IR/Module.h"
#include "llvm/IR/Verifier.h" #include "llvm/IR/Verifier.h"
#include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ErrorHandling.h"
#include "llvm/Support/Regex.h" #include "llvm/Support/Regex.h"
▲ Show 20 Lines • Show All 678 Lines • ▼ Show 20 Lines if (Name == "dbg.value" && F->arg_size() == 4) {
rename(F); rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::dbg_value); NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::dbg_value);
return true; return true;
} }
break; break;
} }
case 'e': { case 'e': {
SmallVector<StringRef, 2> Groups; SmallVector<StringRef, 2> Groups;
static const Regex R("^experimental.vector.reduce.([a-z]+)\\.[fi][0-9]+"); static const Regex R("^experimental.vector.reduce.([a-z]+)\\.[a-z][0-9]+");
if (R.match(Name, &Groups)) { if (R.match(Name, &Groups)) {
Intrinsic::ID ID;
ID = StringSwitch<Intrinsic::ID>(Groups[1])
.Case("add", Intrinsic::vector_reduce_add)
.Case("mul", Intrinsic::vector_reduce_mul)
.Case("and", Intrinsic::vector_reduce_and)
.Case("or", Intrinsic::vector_reduce_or)
.Case("xor", Intrinsic::vector_reduce_xor)
.Case("smax", Intrinsic::vector_reduce_smax)
.Case("smin", Intrinsic::vector_reduce_smin)
.Case("umax", Intrinsic::vector_reduce_umax)
.Case("umin", Intrinsic::vector_reduce_umin)
.Case("fmax", Intrinsic::vector_reduce_fmax)
.Case("fmin", Intrinsic::vector_reduce_fmin)
.Default(Intrinsic::not_intrinsic);
if (ID != Intrinsic::not_intrinsic) {
rename(F);
auto Args = F->getFunctionType()->params();
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, {Args[0]});
return true;
}
}
static const Regex R2(
"^experimental.vector.reduce.v2.([a-z]+)\\.[fi][0-9]+");
nikicUnsubmitted
Not Done
ReplyInline Actions

Seeing this, I'm wondering whether we shouldn't change the type signature for fadd/fmul to omit the separate scalar type. It should be sufficient to have vector.reduce.fadd.v4f32 rather than vector.reduce.f32.v4f32. The scalar argument has to match the vector type, just the like return type already has to match.

nikic: Seeing this, I'm wondering whether we shouldn't change the type signature for fadd/fmul to omit…
Groups.clear();
if (R2.match(Name, &Groups)) {
Intrinsic::ID ID = Intrinsic::not_intrinsic; Intrinsic::ID ID = Intrinsic::not_intrinsic;
if (Groups[1] == "fadd") if (Groups[1] == "fadd")
ID = Intrinsic::experimental_vector_reduce_v2_fadd; ID = Intrinsic::vector_reduce_fadd;
if (Groups[1] == "fmul") if (Groups[1] == "fmul")
ID = Intrinsic::experimental_vector_reduce_v2_fmul; ID = Intrinsic::vector_reduce_fmul;
if (ID != Intrinsic::not_intrinsic) { if (ID != Intrinsic::not_intrinsic) {
rename(F); rename(F);
auto Args = F->getFunctionType()->params(); auto Args = F->getFunctionType()->params();
Type *Tys[] = {F->getFunctionType()->getReturnType(), Args[1]}; Type *Tys[] = {Args[1]};
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, Tys); NewFn = Intrinsic::getDeclaration(F->getParent(), ID, Tys);
return true; return true;
} }
} }
break; break;
} }
case 'i': case 'i':
case 'l': { case 'l': {
▲ Show 20 Lines • Show All 2,874 Lines • ▼ Show 20 Lines assert(
"Unknown function for CallInst upgrade and isn't just a name change"); "Unknown function for CallInst upgrade and isn't just a name change");
CI->setCalledFunction(NewFn); CI->setCalledFunction(NewFn);
}; };
CallInst *NewCall = nullptr; CallInst *NewCall = nullptr;
switch (NewFn->getIntrinsicID()) { switch (NewFn->getIntrinsicID()) {
default: { default: {
DefaultCase(); DefaultCase();
return; return;
} }
case Intrinsic::experimental_vector_reduce_v2_fmul: {
SmallVector<Value *, 2> Args;
if (CI->isFast())
Args.push_back(ConstantFP::get(CI->getOperand(0)->getType(), 1.0));
else
Args.push_back(CI->getOperand(0));
Args.push_back(CI->getOperand(1));
NewCall = Builder.CreateCall(NewFn, Args);
cast<Instruction>(NewCall)->copyFastMathFlags(CI);
break;
}
case Intrinsic::experimental_vector_reduce_v2_fadd: {
SmallVector<Value *, 2> Args;
if (CI->isFast())
Args.push_back(Constant::getNullValue(CI->getOperand(0)->getType()));
else
Args.push_back(CI->getOperand(0));
Args.push_back(CI->getOperand(1));
NewCall = Builder.CreateCall(NewFn, Args);
cast<Instruction>(NewCall)->copyFastMathFlags(CI);
break;
}
case Intrinsic::arm_neon_vld1: case Intrinsic::arm_neon_vld1:
RKSimonUnsubmitted
Not Done
ReplyInline Actions

Can we not just drop support for the v1 experimental fmul/fadd now? As @nikic said there is a danger of this matching against the v2 variants upgraded above as well.

RKSimon: Can we not just drop support for the v1 experimental fmul/fadd now? As @nikic said there is a…
aemersonAuthorUnsubmitted
Done
ReplyInline Actions

Yeah I think we can probably drop support. I'll remove this special handling so we're only effectively renaming the existing set.

aemerson: Yeah I think we can probably drop support. I'll remove this special handling so we're only…
case Intrinsic::arm_neon_vld2: case Intrinsic::arm_neon_vld2:
case Intrinsic::arm_neon_vld3: case Intrinsic::arm_neon_vld3:
case Intrinsic::arm_neon_vld4: case Intrinsic::arm_neon_vld4:
case Intrinsic::arm_neon_vld2lane: case Intrinsic::arm_neon_vld2lane:
case Intrinsic::arm_neon_vld3lane: case Intrinsic::arm_neon_vld3lane:
case Intrinsic::arm_neon_vld4lane: case Intrinsic::arm_neon_vld4lane:
case Intrinsic::arm_neon_vst1: case Intrinsic::arm_neon_vst1:
case Intrinsic::arm_neon_vst2: case Intrinsic::arm_neon_vst2:
▲ Show 20 Lines • Show All 770 Lines • Show Last 20 Lines

llvm/lib/IR/IRBuilder.cpp

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

llvm/lib/Target/ARM/MVETailPredication.cpp

llvm/lib/Transforms/InstCombine/InstCombineAddSub.cpp

llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp

llvm/test/Analysis/CostModel/AArch64/vector-reduce.ll

llvm/test/Analysis/CostModel/ARM/reduce-add.ll

llvm/test/Analysis/CostModel/ARM/reduce-smax.ll

llvm/test/Analysis/CostModel/ARM/reduce-smin.ll

llvm/test/Analysis/CostModel/ARM/reduce-umax.ll

llvm/test/Analysis/CostModel/ARM/reduce-umin.ll

llvm/test/Analysis/CostModel/X86/reduce-add.ll

llvm/test/Analysis/CostModel/X86/reduce-and.ll

llvm/test/Analysis/CostModel/X86/reduce-fmax.ll

llvm/test/Analysis/CostModel/X86/reduce-fmin.ll

llvm/test/Analysis/CostModel/X86/reduce-mul.ll

llvm/test/Analysis/CostModel/X86/reduce-or.ll

llvm/test/Analysis/CostModel/X86/reduce-smax.ll

llvm/test/Analysis/CostModel/X86/reduce-smin.ll

llvm/test/Analysis/CostModel/X86/reduce-umax.ll

llvm/test/Analysis/CostModel/X86/reduce-umin.ll

llvm/test/Analysis/CostModel/X86/reduce-xor.ll

llvm/test/Assembler/invalid-vecreduce.ll

llvm/test/Bitcode/upgrade-vecreduce-intrinsics.ll

; RUN: opt -S < %s | FileCheck %s ; RUN: opt -S < %s | FileCheck %s
; RUN: llvm-dis < %s.bc | FileCheck %s ; RUN: llvm-dis < %s.bc | FileCheck %s
define float @fadd_acc(<4 x float> %in, float %acc) {
; CHECK-LABEL: @fadd_acc define float @fadd_v2(<4 x float> %in, float %acc) {
; CHECK: %res = call float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float %acc, <4 x float> %in) ; CHECK-LABEL: @fadd_v2
%res = call float @llvm.experimental.vector.reduce.fadd.f32.v4f32(float %acc, <4 x float> %in) ; CHECK: %res = call float @llvm.vector.reduce.fadd.v4f32(float %acc, <4 x float> %in)
%res = call float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float %acc, <4 x float> %in)
ret float %res ret float %res
} }
define float @fadd_undef(<4 x float> %in) { define float @fadd_v2_fast(<4 x float> %in, float %acc) {
; CHECK-LABEL: @fadd_undef ; CHECK-LABEL: @fadd_v2_fast
; CHECK: %res = call float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float undef, <4 x float> %in) ; CHECK: %res = call fast float @llvm.vector.reduce.fadd.v4f32(float %acc, <4 x float> %in)
%res = call float @llvm.experimental.vector.reduce.fadd.f32.v4f32(float undef, <4 x float> %in) %res = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float %acc, <4 x float> %in)
ret float %res ret float %res
} }
define float @fadd_fast_acc(<4 x float> %in, float %acc) { define float @fmul_v2(<4 x float> %in, float %acc) {
; CHECK-LABEL: @fadd_fast_acc ; CHECK-LABEL: @fmul_v2
; CHECK: %res = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.000000e+00, <4 x float> %in) ; CHECK: %res = call float @llvm.vector.reduce.fmul.v4f32(float %acc, <4 x float> %in)
%res = call fast float @llvm.experimental.vector.reduce.fadd.f32.v4f32(float %acc, <4 x float> %in) %res = call float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float %acc, <4 x float> %in)
ret float %res ret float %res
} }
define float @fadd_fast_undef(<4 x float> %in) { define float @fmul_v2_fast(<4 x float> %in, float %acc) {
; CHECK-LABEL: @fadd_fast_undef ; CHECK-LABEL: @fmul_v2_fast
; CHECK: %res = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.000000e+00, <4 x float> %in) ; CHECK: %res = call fast float @llvm.vector.reduce.fmul.v4f32(float %acc, <4 x float> %in)
%res = call fast float @llvm.experimental.vector.reduce.fadd.f32.v4f32(float undef, <4 x float> %in) %res = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float %acc, <4 x float> %in)
ret float %res ret float %res
} }
define float @fmul_acc(<4 x float> %in, float %acc) { define float @fmin(<4 x float> %in) {
; CHECK-LABEL: @fmul_acc ; CHECK-LABEL: @fmin
; CHECK: %res = call float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float %acc, <4 x float> %in) ; CHECK: %res = call float @llvm.vector.reduce.fmin.v4f32(<4 x float> %in)
%res = call float @llvm.experimental.vector.reduce.fmul.f32.v4f32(float %acc, <4 x float> %in) %res = call float @llvm.experimental.vector.reduce.fmin.v4f32(<4 x float> %in)
ret float %res ret float %res
} }
define float @fmul_undef(<4 x float> %in) { define float @fmax(<4 x float> %in) {
; CHECK-LABEL: @fmul_undef ; CHECK-LABEL: @fmax
; CHECK: %res = call float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float undef, <4 x float> %in) ; CHECK: %res = call float @llvm.vector.reduce.fmax.v4f32(<4 x float> %in)
%res = call float @llvm.experimental.vector.reduce.fmul.f32.v4f32(float undef, <4 x float> %in) %res = call float @llvm.experimental.vector.reduce.fmax.v4f32(<4 x float> %in)
ret float %res ret float %res
} }
define float @fmul_fast_acc(<4 x float> %in, float %acc) { define i32 @and(<4 x i32> %in) {
; CHECK-LABEL: @fmul_fast_acc ; CHECK-LABEL: @and
; CHECK: %res = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.000000e+00, <4 x float> %in) ; CHECK: %res = call i32 @llvm.vector.reduce.and.v4i32(<4 x i32> %in)
%res = call fast float @llvm.experimental.vector.reduce.fmul.f32.v4f32(float %acc, <4 x float> %in) %res = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> %in)
ret float %res ret i32 %res
} }
define float @fmul_fast_undef(<4 x float> %in) { define i32 @or(<4 x i32> %in) {
; CHECK-LABEL: @fmul_fast_undef ; CHECK-LABEL: @or
; CHECK: %res = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.000000e+00, <4 x float> %in) ; CHECK: %res = call i32 @llvm.vector.reduce.or.v4i32(<4 x i32> %in)
%res = call fast float @llvm.experimental.vector.reduce.fmul.f32.v4f32(float undef, <4 x float> %in) %res = call i32 @llvm.experimental.vector.reduce.or.v4i32(<4 x i32> %in)
ret float %res ret i32 %res
}
define i32 @xor(<4 x i32> %in) {
; CHECK-LABEL: @xor
; CHECK: %res = call i32 @llvm.vector.reduce.xor.v4i32(<4 x i32> %in)
%res = call i32 @llvm.experimental.vector.reduce.xor.v4i32(<4 x i32> %in)
ret i32 %res
}
define i32 @smin(<4 x i32> %in) {
; CHECK-LABEL: @smin
; CHECK: %res = call i32 @llvm.vector.reduce.smin.v4i32(<4 x i32> %in)
%res = call i32 @llvm.experimental.vector.reduce.smin.v4i32(<4 x i32> %in)
ret i32 %res
} }
declare float @llvm.experimental.vector.reduce.fadd.f32.v4f32(float, <4 x float>) define i32 @smax(<4 x i32> %in) {
; CHECK: declare float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float, <4 x float>) ; CHECK-LABEL: @smax
; CHECK: %res = call i32 @llvm.vector.reduce.smax.v4i32(<4 x i32> %in)
%res = call i32 @llvm.experimental.vector.reduce.smax.v4i32(<4 x i32> %in)
ret i32 %res
}
define i32 @umin(<4 x i32> %in) {
; CHECK-LABEL: @umin
; CHECK: %res = call i32 @llvm.vector.reduce.umin.v4i32(<4 x i32> %in)
%res = call i32 @llvm.experimental.vector.reduce.umin.v4i32(<4 x i32> %in)
ret i32 %res
}
nikicUnsubmitted
Not Done
ReplyInline Actions

Please also add fast variants for these two tests, to make sure the operand is not replaced with 0/1 if the v2 intrinsics were used.

nikic: Please also add `fast` variants for these two tests, to make sure the operand is not replaced…
define i32 @umax(<4 x i32> %in) {
; CHECK-LABEL: @umax
; CHECK: %res = call i32 @llvm.vector.reduce.umax.v4i32(<4 x i32> %in)
%res = call i32 @llvm.experimental.vector.reduce.umax.v4i32(<4 x i32> %in)
ret i32 %res
}
declare float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float, <4 x float>)
declare float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float, <4 x float>)
declare float @llvm.experimental.vector.reduce.fmin.v4f32(<4 x float>)
; CHECK: declare float @llvm.vector.reduce.fmin.v4f32(<4 x float>)
declare float @llvm.experimental.vector.reduce.fmax.v4f32(<4 x float>)
; CHECK: declare float @llvm.vector.reduce.fmax.v4f32(<4 x float>)
declare i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.and.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.or.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.or.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.xor.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.xor.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.smin.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.smin.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.smax.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.smax.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.umin.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.umin.v4i32(<4 x i32>)
declare i32 @llvm.experimental.vector.reduce.umax.v4i32(<4 x i32>)
; CHECK: declare i32 @llvm.vector.reduce.umax.v4i32(<4 x i32>)
declare float @llvm.experimental.vector.reduce.fmul.f32.v4f32(float, <4 x float>)
; CHECK: declare float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float, <4 x float>)

llvm/test/Bitcode/upgrade-vecreduce-intrinsics.ll.bc

llvm/test/CodeGen/AArch64/aarch64-addv.ll

llvm/test/CodeGen/AArch64/aarch64-minmaxv.ll

llvm/test/CodeGen/AArch64/arm64-vabs.ll

llvm/test/CodeGen/AArch64/neon-dot-product.ll

llvm/test/CodeGen/AArch64/sve-fixed-length-fp-reduce.ll

llvm/test/CodeGen/AArch64/sve-fixed-length-int-reduce.ll

llvm/test/CodeGen/AArch64/vecreduce-add-legalization.ll

llvm/test/CodeGen/AArch64/vecreduce-and-legalization.ll

llvm/test/CodeGen/AArch64/vecreduce-bool.ll

llvm/test/CodeGen/AArch64/vecreduce-fadd-legalization-strict.ll

llvm/test/CodeGen/AArch64/vecreduce-fadd-legalization.ll

llvm/test/CodeGen/AArch64/vecreduce-fadd.ll

llvm/test/CodeGen/AArch64/vecreduce-fmax-legalization-nan.ll

llvm/test/CodeGen/AArch64/vecreduce-fmax-legalization.ll

llvm/test/CodeGen/AArch64/vecreduce-fmin-legalization.ll

llvm/test/CodeGen/AArch64/vecreduce-fmul-legalization-strict.ll

llvm/test/CodeGen/AArch64/vecreduce-propagate-sd-flags.ll

llvm/test/CodeGen/AArch64/vecreduce-umax-legalization.ll

llvm/test/CodeGen/ARM/vecreduce-fadd-legalization-soft-float.ll

llvm/test/CodeGen/ARM/vecreduce-fadd-legalization-strict.ll

llvm/test/CodeGen/ARM/vecreduce-fmax-legalization-soft-float.ll

llvm/test/CodeGen/ARM/vecreduce-fmin-legalization-soft-float.ll

llvm/test/CodeGen/ARM/vecreduce-fmul-legalization-soft-float.ll

llvm/test/CodeGen/ARM/vecreduce-fmul-legalization-strict.ll

llvm/test/CodeGen/Generic/expand-experimental-reductions.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/add_reduce.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/cond-vector-reduce-mve-codegen.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/inloop-vpsel-1.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/inloop-vpsel-2.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/invariant-qreg.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/lstp-insertion-position.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/matrix.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/mve-tail-data-types.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/nested.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/no-vpsel-liveout.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/non-masked-load.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/predicated-liveout.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions-vpt-liveout.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/skip-debug.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/tail-pred-reduce.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/tp-multiple-vpst.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/unpredicated-max.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vaddv.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/varying-outer-2d-reduction.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vctp-add-operand-liveout.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vctp16-reduce.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vector-arith-codegen.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vector-reduce-mve-tail.ll

llvm/test/CodeGen/Thumb2/LowOverheadLoops/wlstp.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/wrong-liveout-lsr-shift.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/wrong-vctp-opcode-liveout.mir

llvm/test/CodeGen/Thumb2/LowOverheadLoops/wrong-vctp-operand-liveout.mir

llvm/test/CodeGen/Thumb2/mve-gather-scatter-optimisation.ll

llvm/test/CodeGen/Thumb2/mve-gather-scatter-tailpred.ll

llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll

llvm/test/CodeGen/Thumb2/mve-vaddv.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-add.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-addpred.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-bit.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-fadd.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-fminmax.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-fmul.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-loops.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-mla.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-mlapred.ll

llvm/test/CodeGen/Thumb2/mve-vecreduce-mul.ll

llvm/test/CodeGen/Thumb2/mve-vmaxv.ll

llvm/test/CodeGen/X86/haddsub.ll

llvm/test/CodeGen/X86/pr45378.ll

llvm/test/CodeGen/X86/vector-reduce-add.ll

llvm/test/CodeGen/X86/vector-reduce-and-bool.ll

llvm/test/CodeGen/X86/vector-reduce-and-cmp.ll

llvm/test/CodeGen/X86/vector-reduce-and.ll

llvm/test/CodeGen/X86/vector-reduce-fadd-fast.ll

llvm/test/CodeGen/X86/vector-reduce-fadd.ll

llvm/test/CodeGen/X86/vector-reduce-fmax-fmin-fast.ll

llvm/test/CodeGen/X86/vector-reduce-fmax-nnan.ll

llvm/test/CodeGen/X86/vector-reduce-fmax.ll

llvm/test/CodeGen/X86/vector-reduce-fmin-nnan.ll

llvm/test/CodeGen/X86/vector-reduce-fmin.ll

llvm/test/CodeGen/X86/vector-reduce-fmul-fast.ll

llvm/test/CodeGen/X86/vector-reduce-fmul.ll

llvm/test/CodeGen/X86/vector-reduce-mul.ll

llvm/test/CodeGen/X86/vector-reduce-or-bool.ll

llvm/test/CodeGen/X86/vector-reduce-or-cmp.ll

llvm/test/CodeGen/X86/vector-reduce-or.ll

llvm/test/CodeGen/X86/vector-reduce-smax.ll

llvm/test/CodeGen/X86/vector-reduce-smin.ll

llvm/test/CodeGen/X86/vector-reduce-umax.ll

llvm/test/CodeGen/X86/vector-reduce-umin.ll

llvm/test/CodeGen/X86/vector-reduce-xor-bool.ll

llvm/test/CodeGen/X86/vector-reduce-xor.ll

llvm/test/Instrumentation/MemorySanitizer/experimental-reduce.ll

  • This file was moved to llvm/test/Instrumentation/MemorySanitizer/reduce.ll.

llvm/test/Instrumentation/MemorySanitizer/reduce.ll

llvm/test/Transforms/InstCombine/vector-reductions.ll

llvm/test/Transforms/InstSimplify/ConstProp/vecreduce.ll

llvm/test/Transforms/LoopVectorize/AArch64/pr33053.ll

llvm/test/Transforms/LoopVectorize/AArch64/reduction-small-size.ll

llvm/test/Transforms/LoopVectorize/ARM/mve-gather-scatter-tailpred.ll

llvm/test/Transforms/LoopVectorize/ARM/mve-reduction-predselect.ll

llvm/test/Transforms/LoopVectorize/ARM/mve-reduction-types.ll

llvm/test/Transforms/LoopVectorize/ARM/mve-reductions.ll

llvm/test/Transforms/LoopVectorize/ARM/tail-fold-multiple-icmps.ll

llvm/test/Transforms/LoopVectorize/ARM/tail-folding-not-allowed.ll

llvm/test/Transforms/LoopVectorize/X86/imprecise-through-phis.ll

llvm/test/Transforms/LoopVectorize/X86/invariant-store-vectorization.ll

llvm/test/Transforms/LoopVectorize/X86/load-deref-pred.ll

llvm/test/Transforms/LoopVectorize/X86/pr35432.ll

llvm/test/Transforms/LoopVectorize/X86/pr42674.ll

llvm/test/Transforms/LoopVectorize/X86/reduction-fastmath.ll

llvm/test/Transforms/LoopVectorize/X86/strided_load_cost.ll

llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll

llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll

llvm/test/Transforms/LoopVectorize/reduction-inloop.ll

llvm/test/Transforms/LoopVectorize/reduction-predselect.ll

llvm/test/Transforms/PhaseOrdering/X86/vector-reductions.ll

llvm/test/Transforms/SLPVectorizer/AArch64/gather-cost.ll

llvm/test/Transforms/SLPVectorizer/AArch64/gather-root.ll

llvm/test/Transforms/SLPVectorizer/AArch64/horizontal.ll

llvm/test/Transforms/SLPVectorizer/AArch64/transpose.ll

llvm/test/Transforms/SLPVectorizer/X86/PR35628_1.ll

llvm/test/Transforms/SLPVectorizer/X86/PR35628_2.ll

llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll

llvm/test/Transforms/SLPVectorizer/X86/PR40310.ll

llvm/test/Transforms/SLPVectorizer/X86/compare-reduce.ll

llvm/test/Transforms/SLPVectorizer/X86/horizontal-list.ll

llvm/test/Transforms/SLPVectorizer/X86/horizontal-minmax.ll

llvm/test/Transforms/SLPVectorizer/X86/horizontal.ll

llvm/test/Transforms/SLPVectorizer/X86/reassociated-loads.ll

llvm/test/Transforms/SLPVectorizer/X86/reduction.ll

llvm/test/Transforms/SLPVectorizer/X86/reduction_loads.ll

llvm/test/Transforms/SLPVectorizer/X86/reduction_unrolled.ll

llvm/test/Transforms/SLPVectorizer/X86/remark_horcost.ll

llvm/test/Transforms/SLPVectorizer/X86/reorder_repeated_ops.ll

llvm/test/Transforms/SLPVectorizer/X86/reverse_extract_elements.ll

llvm/test/Transforms/SLPVectorizer/X86/scheduling.ll

llvm/test/Transforms/SLPVectorizer/X86/undef_vect.ll

llvm/test/Transforms/SLPVectorizer/X86/used-reduced-op.ll

llvm/test/Transforms/SLPVectorizer/X86/vectorize-reorder-reuse.ll

mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td

mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td

mlir/include/mlir/Dialect/Vector/VectorOps.td

mlir/integration_test/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir

mlir/integration_test/Dialect/LLVMIR/CPU/test-vector-reductions-int.mlir

mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir

mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir

mlir/test/Target/llvmir-intrinsics.mlir