This is an archive of the discontinued LLVM Phabricator instance.

Differential D103874

[IR] Rename the shufflevector's undef mask to poison
Needs ReviewPublic

Authored by aqjune on Jun 8 2021, 12:36 AM.

Details

Reviewers
nikic
efriedma
spatel
fhahn
lebedev.ri
RKSimon
deadalnix
Summary

This is a patch that renames shufflevector's undef mask to poison.

By D93818, shufflevector's undef mask isn't undef anymore; it returns poison instead.

%v = shufflevector <2 x i8> %x, <2 x i8> %y, <2 x i8> <i8 0, i8 poison>
; %v[0] = %x[0]
; %v[1] = poison

Since poison is more undefined than undef, this validates many existing transformations that we wanted to support.
Also, this allows more aggressive optimizations because poison is more propagative (e.g. poison & 0 = poison whereas undef & 0 != undef).

This patch updates shufflevector mask's printed string to be poison to match its new semantics.

This has changes in clang tests as well.
They are mainly about vector intrinsics being lowered into shufflevector.
The unused elements were filled with undef previously, but with this patch they are filled with poison.
Since they are unused elements anyway, I believe this isn't a functional change in fact.
But, I'm happy with this being double-checked by someone who works on these intrinsics as well.

Diff Detail

Event Timeline

aqjune created this revision.Jun 8 2021, 12:36 AM
Herald added a reviewer: deadalnix. · View Herald TranscriptJun 8 2021, 12:36 AM
Herald added subscribers: dexonsmith, kerbowa, pengfei and 7 others. · View Herald Transcript
aqjune requested review of this revision.Jun 8 2021, 12:36 AM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptJun 8 2021, 12:36 AM
Herald added subscribers: llvm-commits, cfe-commits. · View Herald Transcript
Harbormaster completed remote builds in B108148: Diff 350513.Jun 8 2021, 1:27 AM
efriedma added a comment.Jun 8 2021, 2:22 PM

I noted the cases where it looks like the undef->poison change might actually impact code using compiler intrinsic functions that have external specifications. The relevant specifications say the elements in question are "undefined", without really specifying what that means.

Currently, for the Intel intrinsics, we treat "undefined" as something more conservative than LLVM undef; see https://github.com/llvm/llvm-project/blob/d2012d965d60c3258b3a69d024491698f8aec386/clang/lib/CodeGen/CGBuiltin.cpp#L12483 . Maybe we should make the cast intrinsics more conservative to match. And maybe we should do the same for OpenCL. Would need to do some backend work to make sure we don't regress the generated code, though.

For __builtin_shufflevector, I think I'm fine with this changing the "-1" to mean poison; we don't have any external spec to conform to, and anyone explicitly passing -1 should know what they're doing. But maybe worth noting in the clang documentation.

clang/test/CodeGen/X86/avx-builtins.c
182

This change might be visible to user code.

clang/test/CodeGen/builtinshufflevector2.c
41

This might be visible to user code.

clang/test/CodeGenOpenCL/preserve_vec3.cl
27

This change might be visible to user code.

aqjune added a comment.EditedJun 8 2021, 7:50 PM
In D103874#2806519, @efriedma wrote:

I noted the cases where it looks like the undef->poison change might actually impact code using compiler intrinsic functions that have external specifications. The relevant specifications say the elements in question are "undefined", without really specifying what that means.

Currently, for the Intel intrinsics, we treat "undefined" as something more conservative than LLVM undef; see https://github.com/llvm/llvm-project/blob/d2012d965d60c3258b3a69d024491698f8aec386/clang/lib/CodeGen/CGBuiltin.cpp#L12483 . Maybe we should make the cast intrinsics more conservative to match. And maybe we should do the same for OpenCL. Would need to do some backend work to make sure we don't regress the generated code, though.

Makes sense, the PR (https://llvm.org/PR32176) that is left at the comment says it should be something like freeze poison as well. (BTW, this means the current shufflevector lowering is already incorrect as well..)

Then, _mm256_castsi128_si256 should be lowered into something like this:

%fr = freeze <2 x i64> poison
shufflevector <2 x i64> %x, <2 x i64> %fr, <4 x i32> <i32 0, i32 1, i32 2, i32 3>

BTW, the Intel intrinsic guide for _mm256_castsi128_si256 ( https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm256_castsi128_si256&expand=628 ) says:

This intrinsic is only used for compilation and does not generate any instructions, thus it has zero latency.

We should teach the backend to understand the shufflevector+freeze form and lower it into an efficient assembly.

I *guess* teaching this form to the backend will be enough in terms of performance.
In practice, the frozen element won't be used in most of the cases; the middle-end's demanded elements analysis will trigger instcombine to almost always remove the freeze.

What do you think? If people agree with the shufflevector+freeze lowering, I'll create a separate patch that lowers this to the new freeze+shufflevector format (since it is already incorrect).

efriedma added a comment.Jun 8 2021, 9:15 PM

In practice, the frozen element won't be used in most of the cases; the middle-end's demanded elements analysis will trigger instcombine to almost always remove the freeze.

Well, in the cases it gets removed, it doesn't really matter what we use. It's likely if someone is reaching for these more obscure constructs, they're seeing that the compiler isn't doing what they want with more normal code, though.

What do you think? If people agree with the shufflevector+freeze lowering, I'll create a separate patch that lowers this to the new freeze+shufflevector format (since it is already incorrect).

Using "freeze poison" seems reasonable.

aqjune mentioned this in D104790: [x86] fix mm*_undefined* intrinsics to use arbitrary frozen bit pattern.Jun 23 2021, 9:03 AM
aqjune mentioned this in D93818: [LangRef] Update shufflevector's semantics to return poison if the mask is undef.Sep 20 2021, 1:57 AM
RKSimon added inline comments.Sep 20 2021, 3:36 AM
clang/test/CodeGen/X86/avx-builtins.c
182

Yes the length changing casts are worrying me as well - we could update the header to insert zero into the upper elements I suppose, in many cases these would be folded away by AVX ops implicitly zeroing the 128-bits. But we'd definitely have the potential for regressions.

1237

These look out of date - D109497 changes the loadu2 codegen to be a single 'concat' shuffle.

aqjune updated this revision to Diff 375094.Sep 26 2021, 6:20 AM

Rebase

Herald added a subscriber: arphaman. · View Herald TranscriptSep 26 2021, 6:20 AM
aqjune updated this revision to Diff 375095.Sep 26 2021, 6:36 AM

Resurrect mistakenly removed test statements

Harbormaster completed remote builds in B125751: Diff 375095.Sep 26 2021, 6:52 AM
aqjune marked an inline comment as done.Sep 26 2021, 7:04 AM
aqjune added inline comments.
clang/test/CodeGen/X86/avx-builtins.c
182

I quickly skimmed through the headers in clang/lib/Headers and listed the functions calling __builtin_shufflevector with at least one -1 mask operand.
It seems there aren't very many, which is good news; I found 17 functions only (

list.txt563 BDownload
).

But, correctly fixing these headers seems to require a lot of work.
Since using the zero vector can cause performance regressions, we need to use a frozen poison (undef) vector to encode a vector having unspecified bits.
A few months ago, I created D104790 to start using freeze(vector poison) for mm*_undefined* intrinsics. However, teaching the existing codebase to successfully deal with the frozen poison vector was a pretty tough job.
When it comes to fixing the headers, there is even no C intrinsic function that represents a frozen poison vector AFAIK.

I'll appreciate any idea or help in addressing this issue. :/

nikic mentioned this in D115526: [InstCombine] don't automatically drop poison-generating flags in SimplifyVectorDemandedElts.Dec 10 2021, 8:16 AM
spatel mentioned this in D115460: Add FMF to hasPoisonGeneratingFlags/dropPoisonGeneratingFlags.Dec 13 2021, 11:40 AM
RKSimon mentioned this in rG1b07bd9034bd: [X86] Add tests for vector widening with freeze(undef).May 13 2022, 4:12 AM
aqjune added a comment.Jun 19 2022, 4:55 AM

It seems llvm/lib/Target/X86/X86ISelLowering.cpp's LowerAVXCONCAT_VECTORS is relevant to efficient lowering of shufflevector %x, freeze(poison), mask.

Herald added a project: Restricted Project. · View Herald TranscriptJun 19 2022, 4:55 AM
Herald added subscribers: jsji, kosarev. · View Herald Transcript
dexonsmith removed a subscriber: dexonsmith.Jun 19 2022, 8:12 AM
aqjune added a comment.Jun 27 2022, 1:58 AM
In D103874#3594617, @aqjune wrote:

It seems llvm/lib/Target/X86/X86ISelLowering.cpp's LowerAVXCONCAT_VECTORS is relevant to efficient lowering of shufflevector %x, freeze(poison), mask.

After patching LowerAVXCONCAT_VECTORS, lowering https://github.com/llvm/llvm-project/blob/main/llvm/test/CodeGen/X86/avx-intrinsics-fast-isel.ll#L257 generates:

vblendps  $15, %ymm0, %ymm0, %ymm0

To make it fully no-op, tablegen files must be edited. I gave it a try, .td compiled successfully, but weirdly - perhaps due to either incorrect use of tablegen's pattern matcher or some hidden rule that I didn't address - vblendps is still there. The written patch is as follows:
https://github.com/aqjune/llvm-project/commit/b4393e36b33ca08ce77ae662479ceaf9a76eab8b

One of relevant, edited parts:

// llvm/lib/Target/X86/X86InstrVecCompiler.td
  def : Pat<(VT (insert_subvector undef, subRC:$src, (iPTR 0))),
            (VT (INSERT_SUBREG (IMPLICIT_DEF), subRC:$src, subIdx))>;
+
+  def : Pat<(VT (insert_subvector (freeze undef), subRC:$src, (iPTR 0))),
+            (VT (INSERT_SUBREG (IMPLICIT_DEF), subRC:$src, subIdx))>;

I spent some time but couldn't figure out why it does not work.
Can someone tell me whether the pattern matching is being correctly used? Any help is appreciated.

nikic added a comment.Jun 27 2022, 2:04 AM

Which intrinsic are you working on here? If this is about the mm_undefined intrinsics, why do we need to change those from the current status quo of using a zero value instead of undef?

aqjune added a comment.Jun 27 2022, 2:13 AM
In D103874#3611483, @nikic wrote:

Which intrinsic are you working on here? If this is about the mm_undefined intrinsics, why do we need to change those from the current status quo of using a zero value instead of undef?

It is about the mm256_castpd128_pd256 intrinsic and its friends (clang/test/CodeGen/X86/avx-builtins.c, line 146).
It was previously using shufflevector with undef masks - since the results are poison, an alternative pattern as below is necessary to represent the intrinsic:

%a1 = freeze <2 x double> poison
%res = shufflevector <2 x double> %a0, <2 x double> %a1, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
nikic added a comment.Jun 27 2022, 2:36 AM
In D103874#3611507, @aqjune wrote:
In D103874#3611483, @nikic wrote:

Which intrinsic are you working on here? If this is about the mm_undefined intrinsics, why do we need to change those from the current status quo of using a zero value instead of undef?

It is about the mm256_castpd128_pd256 intrinsic and its friends (clang/test/CodeGen/X86/avx-builtins.c, line 146).
It was previously using shufflevector with undef masks - since the results are poison, an alternative pattern as below is necessary to represent the intrinsic:

%a1 = freeze <2 x double> poison
%res = shufflevector <2 x double> %a0, <2 x double> %a1, <4 x i32> <i32 0, i32 1, i32 2, i32 3>

How sure are we that we cannot simply use poison elements here? I checked what the Intel compiler guide has to say on the topic, and it uses the following wording. For "undefined" style intrinsics:

This intrinsic returns a vector of eight single precision floating point elements. The content of the vector is not specified.

For "cast" style intrinsics:

The lower 128-bits of the 256-bit resulting vector contains the source vector values; the upper 128-bits of the resulting vector are undefined. This intrinsic does not introduce extra moves to the generated code

It's not really clear what "undefined" is supposed to mean here (and how it differs from "not specified").

Unless we're aware of a specific problems in this area, I think it's okay to start out with just doing the undef -> poison replacement, and possibly backtrack if there are real-world assumptions about the specific meaning of "undefined" in this context.

aqjune added a comment.Jun 27 2022, 6:39 PM

PR31524 (https://github.com/llvm/llvm-project/issues/31524) discusses about the lowering of such intrinsics.
According to the PR, it seems users consider undefined intrinsics as returning unknown but consistent bits.

If it works, my updates in the draft (https://github.com/aqjune/llvm-project/commit/b4393e36b33ca08ce77ae662479ceaf9a76eab8b) will improve backends dealing with freeze(poison), which will help lowering _mm256_cast* with zero copy.

aqjune mentioned this in D130339: [CodeGen] Generate efficient assembly for freeze(poison) version of `mm*_cast*` intel intrinsics.Jul 22 2022, 2:48 AM
aqjune mentioned this in rG02e56e253302: [CodeGen] Generate efficient assembly for freeze(poison) version of `mm*_cast*`….Aug 10 2022, 9:38 PM
aqjune added inline comments.Aug 10 2022, 9:48 PM
clang/test/CodeGen/X86/avx-builtins.c
182

Okay, D130339 has finally been merged.

I will make a patch that updates the mm256_castsi128_si256 and its family functions to emit shufflevector with freeze(poison) operand.

ManuelJBrito mentioned this in D143287: [Clang][X86] Change X86 cast intrinsics to use __builtin_nondeterministic_value.Feb 3 2023, 10:25 AM

Large Diff

This large diff affects 195 files. Files without inline comments have been collapsed. Expand All Files

Revision Contents

PathSize
clang/
test/
CodeGen/
X86/
6 lines
10 lines
8 lines
8 lines
2 lines
4 lines
CodeGenOpenCL/
6 lines
4 lines
2 lines
22 lines
2 lines
llvm/
include/
llvm-c/
6 lines
llvm/
IR/
14 lines
lib/
Analysis/
2 lines
2 lines
IR/
8 lines
5 lines
2 lines
14 lines
Transforms/
InstCombine/
4 lines
4 lines
2 lines
18 lines
4 lines
Scalar/
2 lines
Vectorize/
38 lines
10 lines
test/
Analysis/
CostModel/
AMDGPU/
28 lines
X86/
670 lines
6 lines
550 lines
6 lines
CodeGen/
AMDGPU/
14 lines
Generic/
34 lines
PowerPC/
6 lines
Transforms/
CodeGenPrepare/
X86/
8 lines
8 lines
DeadStoreElimination/
16 lines
16 lines
InstCombine/
AMDGPU/
14 lines
14 lines
X86/
4 lines
4 lines
6 lines
6 lines
56 lines
56 lines
4 lines
4 lines
12 lines
12 lines
10 lines
10 lines
34 lines
34 lines
22 lines
22 lines
6 lines
6 lines
2 lines
2 lines
16 lines
4 lines
4 lines
12 lines
58 lines
58 lines
2 lines
2 lines
2 lines
2 lines
8 lines
4 lines
44 lines
2 lines
12 lines
12 lines
46 lines
46 lines
2 lines
2 lines
10 lines
10 lines
4 lines
4 lines
2 lines
2 lines
2 lines
2 lines
4 lines
10 lines
70 lines
70 lines
98 lines
98 lines
16 lines
24 lines
24 lines
InstSimplify/
8 lines
8 lines
InterleavedAccess/
AArch64/
4 lines
4 lines
ARM/
146 lines
146 lines
X86/
2 lines
2 lines
14 lines
14 lines
LoopUnroll/
X86/
2 lines
2 lines
LoopVectorize/
AArch64/
2 lines
X86/
6 lines
4 lines
LowerMatrixIntrinsics/
36 lines
8 lines
2 lines
8 lines
8 lines
8 lines
36 lines
36 lines
8 lines
8 lines
36 lines
36 lines
36 lines
8 lines
16 lines
24 lines
46 lines
2 lines
2 lines
2 lines
2 lines
2 lines
2 lines
80 lines
PhaseOrdering/
ARM/
2 lines
X86/
44 lines
12 lines
10 lines
SLPVectorizer/
AArch64/
12 lines
12 lines
10 lines
10 lines
18 lines
AMDGPU/
8 lines
8 lines
SystemZ/
2 lines
X86/
2 lines
2 lines
30 lines
30 lines
8 lines
8 lines
4 lines
4 lines
60 lines
60 lines
8 lines
8 lines
2 lines
2 lines
4 lines
4 lines
2 lines
4 lines
6 lines
4 lines
2 lines
2 lines
4 lines
2 lines
142 lines
142 lines
2 lines
4 lines
2 lines
6 lines
4 lines
8 lines
2 lines
2 lines
SROA/
2 lines
8 lines
VectorCombine/
AArch64/
2 lines
AMDGPU/
2 lines
2 lines
X86/
40 lines
40 lines
8 lines
10 lines
38 lines
38 lines
tools/
llvm-c-test/
2 lines

Diff 375095

clang/test/CodeGen/X86/avx-builtins.c

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Lines
__m256i test_mm256_castpd_si256(__m256d A) { __m256i test_mm256_castpd_si256(__m256d A) {
// CHECK-LABEL: test_mm256_castpd_si256 // CHECK-LABEL: test_mm256_castpd_si256
// CHECK: bitcast <4 x double> %{{.*}} to <4 x i64> // CHECK: bitcast <4 x double> %{{.*}} to <4 x i64>
return _mm256_castpd_si256(A); return _mm256_castpd_si256(A);
} }
__m256d test_mm256_castpd128_pd256(__m128d A) { __m256d test_mm256_castpd128_pd256(__m128d A) {
// CHECK-LABEL: test_mm256_castpd128_pd256 // CHECK-LABEL: test_mm256_castpd128_pd256
// CHECK: shufflevector <2 x double> %{{.*}}, <2 x double> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef> // CHECK: shufflevector <2 x double> %{{.*}}, <2 x double> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
return _mm256_castpd128_pd256(A); return _mm256_castpd128_pd256(A);
} }
__m128d test_mm256_castpd256_pd128(__m256d A) { __m128d test_mm256_castpd256_pd128(__m256d A) {
// CHECK-LABEL: test_mm256_castpd256_pd128 // CHECK-LABEL: test_mm256_castpd256_pd128
// CHECK: shufflevector <4 x double> %{{.*}}, <4 x double> %{{.*}}, <2 x i32> <i32 0, i32 1> // CHECK: shufflevector <4 x double> %{{.*}}, <4 x double> %{{.*}}, <2 x i32> <i32 0, i32 1>
return _mm256_castpd256_pd128(A); return _mm256_castpd256_pd128(A);
} }
__m256d test_mm256_castps_pd(__m256 A) { __m256d test_mm256_castps_pd(__m256 A) {
// CHECK-LABEL: test_mm256_castps_pd // CHECK-LABEL: test_mm256_castps_pd
// CHECK: bitcast <8 x float> %{{.*}} to <4 x double> // CHECK: bitcast <8 x float> %{{.*}} to <4 x double>
return _mm256_castps_pd(A); return _mm256_castps_pd(A);
} }
__m256i test_mm256_castps_si256(__m256 A) { __m256i test_mm256_castps_si256(__m256 A) {
// CHECK-LABEL: test_mm256_castps_si256 // CHECK-LABEL: test_mm256_castps_si256
// CHECK: bitcast <8 x float> %{{.*}} to <4 x i64> // CHECK: bitcast <8 x float> %{{.*}} to <4 x i64>
return _mm256_castps_si256(A); return _mm256_castps_si256(A);
} }
__m256 test_mm256_castps128_ps256(__m128 A) { __m256 test_mm256_castps128_ps256(__m128 A) {
// CHECK-LABEL: test_mm256_castps128_ps256 // CHECK-LABEL: test_mm256_castps128_ps256
// CHECK: shufflevector <4 x float> %{{.*}}, <4 x float> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef> // CHECK: shufflevector <4 x float> %{{.*}}, <4 x float> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 poison, i32 poison, i32 poison, i32 poison>
return _mm256_castps128_ps256(A); return _mm256_castps128_ps256(A);
} }
__m128 test_mm256_castps256_ps128(__m256 A) { __m128 test_mm256_castps256_ps128(__m256 A) {
// CHECK-LABEL: test_mm256_castps256_ps128 // CHECK-LABEL: test_mm256_castps256_ps128
// CHECK: shufflevector <8 x float> %{{.*}}, <8 x float> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 2, i32 3> // CHECK: shufflevector <8 x float> %{{.*}}, <8 x float> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
return _mm256_castps256_ps128(A); return _mm256_castps256_ps128(A);
} }
__m256i test_mm256_castsi128_si256(__m128i A) { __m256i test_mm256_castsi128_si256(__m128i A) {
// CHECK-LABEL: test_mm256_castsi128_si256 // CHECK-LABEL: test_mm256_castsi128_si256
// CHECK: shufflevector <2 x i64> %{{.*}}, <2 x i64> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef> // CHECK: shufflevector <2 x i64> %{{.*}}, <2 x i64> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
efriedmaUnsubmitted
Not Done
ReplyInline Actions

This change might be visible to user code.

efriedma: This change might be visible to user code.
RKSimonUnsubmitted
Not Done
ReplyInline Actions

Yes the length changing casts are worrying me as well - we could update the header to insert zero into the upper elements I suppose, in many cases these would be folded away by AVX ops implicitly zeroing the 128-bits. But we'd definitely have the potential for regressions.

RKSimon: Yes the length changing casts are worrying me as well - we could update the header to insert…
aqjuneAuthorUnsubmitted
Done
ReplyInline Actions

I quickly skimmed through the headers in clang/lib/Headers and listed the functions calling __builtin_shufflevector with at least one -1 mask operand.
It seems there aren't very many, which is good news; I found 17 functions only (

list.txt563 BDownload
).

But, correctly fixing these headers seems to require a lot of work.
Since using the zero vector can cause performance regressions, we need to use a frozen poison (undef) vector to encode a vector having unspecified bits.
A few months ago, I created D104790 to start using freeze(vector poison) for mm*_undefined* intrinsics. However, teaching the existing codebase to successfully deal with the frozen poison vector was a pretty tough job.
When it comes to fixing the headers, there is even no C intrinsic function that represents a frozen poison vector AFAIK.

I'll appreciate any idea or help in addressing this issue. :/

aqjune: I quickly skimmed through the headers in clang/lib/Headers and listed the functions calling…
aqjuneAuthorUnsubmitted
Done
ReplyInline Actions

Okay, D130339 has finally been merged.

I will make a patch that updates the mm256_castsi128_si256 and its family functions to emit shufflevector with freeze(poison) operand.

aqjune: Okay, D130339 has finally been merged. I will make a patch that updates the…
return _mm256_castsi128_si256(A); return _mm256_castsi128_si256(A);
} }
__m256d test_mm256_castsi256_pd(__m256i A) { __m256d test_mm256_castsi256_pd(__m256i A) {
// CHECK-LABEL: test_mm256_castsi256_pd // CHECK-LABEL: test_mm256_castsi256_pd
// CHECK: bitcast <4 x i64> %{{.*}} to <4 x double> // CHECK: bitcast <4 x i64> %{{.*}} to <4 x double>
return _mm256_castsi256_pd(A); return _mm256_castsi256_pd(A);
} }
▲ Show 20 Lines • Show All 1,038 Lines • ▼ Show 20 Lines__m256i test_mm256_loadu_si256(__m256i* A) {
// CHECK: load <4 x i64>, <4 x i64>* %{{.+}}, align 1{{$}} // CHECK: load <4 x i64>, <4 x i64>* %{{.+}}, align 1{{$}}
return _mm256_loadu_si256(A); return _mm256_loadu_si256(A);
} }
__m256 test_mm256_loadu2_m128(float* A, float* B) { __m256 test_mm256_loadu2_m128(float* A, float* B) {
// CHECK-LABEL: test_mm256_loadu2_m128 // CHECK-LABEL: test_mm256_loadu2_m128
// CHECK: load <4 x float>, <4 x float>* %{{.*}}, align 1{{$}} // CHECK: load <4 x float>, <4 x float>* %{{.*}}, align 1{{$}}
// CHECK: load <4 x float>, <4 x float>* %{{.*}}, align 1{{$}} // CHECK: load <4 x float>, <4 x float>* %{{.*}}, align 1{{$}}
// CHECK: shufflevector <4 x float> %{{.*}}, <4 x float> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7> // CHECK: shufflevector <4 x float> %{{.*}}, <4 x float> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
RKSimonUnsubmitted
Done
ReplyInline Actions

These look out of date - D109497 changes the loadu2 codegen to be a single 'concat' shuffle.

RKSimon: These look out of date - D109497 changes the loadu2 codegen to be a single 'concat' shuffle.
return _mm256_loadu2_m128(A, B); return _mm256_loadu2_m128(A, B);
} }
__m256d test_mm256_loadu2_m128d(double* A, double* B) { __m256d test_mm256_loadu2_m128d(double* A, double* B) {
// CHECK-LABEL: test_mm256_loadu2_m128d // CHECK-LABEL: test_mm256_loadu2_m128d
// CHECK: load <2 x double>, <2 x double>* %{{.*}}, align 1{{$}} // CHECK: load <2 x double>, <2 x double>* %{{.*}}, align 1{{$}}
// CHECK: load <2 x double>, <2 x double>* %{{.*}}, align 1{{$}} // CHECK: load <2 x double>, <2 x double>* %{{.*}}, align 1{{$}}
// CHECK: shufflevector <2 x double> %{{.*}}, <2 x double> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 2, i32 3> // CHECK: shufflevector <2 x double> %{{.*}}, <2 x double> %{{.*}}, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
▲ Show 20 Lines • Show All 919 Lines • Show Last 20 Lines

clang/test/CodeGen/X86/avx512f-builtins.c

clang/test/CodeGen/builtins-ppc-p10vector.c

clang/test/CodeGen/builtins-ppc-p9vector.c

clang/test/CodeGen/builtinshufflevector2.c

Show All 30 Lines
void clang_shufflevector_v_v_c( float4* A, float4 x, float4 y) { void clang_shufflevector_v_v_c( float4* A, float4 x, float4 y) {
// CHECK: [[V:%.*]] = shufflevector <4 x float> {{%.*}}, <4 x float> {{%.*}}, <4 x i32> <i32 0, i32 4, i32 1, i32 5> // CHECK: [[V:%.*]] = shufflevector <4 x float> {{%.*}}, <4 x float> {{%.*}}, <4 x i32> <i32 0, i32 4, i32 1, i32 5>
// CHECK: store <4 x float> [[V]], <4 x float>* {{%.*}} // CHECK: store <4 x float> [[V]], <4 x float>* {{%.*}}
*A = __builtin_shufflevector( x, y, 0, 4, 1, 5 ); *A = __builtin_shufflevector( x, y, 0, 4, 1, 5 );
} }
// CHECK-LABEL: define {{.*}}void @clang_shufflevector_v_v_undef( // CHECK-LABEL: define {{.*}}void @clang_shufflevector_v_v_undef(
void clang_shufflevector_v_v_undef( float4* A, float4 x, float4 y) { void clang_shufflevector_v_v_undef( float4* A, float4 x, float4 y) {
// CHECK: [[V:%.*]] = shufflevector <4 x float> {{%.*}}, <4 x float> {{%.*}}, <4 x i32> <i32 0, i32 4, i32 undef, i32 5> // CHECK: [[V:%.*]] = shufflevector <4 x float> {{%.*}}, <4 x float> {{%.*}}, <4 x i32> <i32 0, i32 4, i32 poison, i32 5>
// CHECK: store <4 x float> [[V]], <4 x float>* {{%.*}} // CHECK: store <4 x float> [[V]], <4 x float>* {{%.*}}
*A = __builtin_shufflevector( x, y, 0, 4, -1, 5 ); *A = __builtin_shufflevector( x, y, 0, 4, -1, 5 );
efriedmaUnsubmitted
Not Done
ReplyInline Actions

This might be visible to user code.

efriedma: This might be visible to user code.
} }

clang/test/CodeGen/ext-vector.c

clang/test/CodeGenOpenCL/as_type.cl

clang/test/CodeGenOpenCL/partial_initializer.cl

clang/test/CodeGenOpenCL/preserve_vec3.cl

Show All 18 Linesvoid kernel float4_to_float3(global float3 *a, global float4 *b) {
// CHECK: %[[ASTYPE:.*]] = shufflevector <4 x float> %[[LOAD_A]], <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2> // CHECK: %[[ASTYPE:.*]] = shufflevector <4 x float> %[[LOAD_A]], <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2>
// CHECK: store <3 x float> %[[ASTYPE]], <3 x float> addrspace(1)* %a, align 16 // CHECK: store <3 x float> %[[ASTYPE]], <3 x float> addrspace(1)* %a, align 16
*a = __builtin_astype(*b, float3); *a = __builtin_astype(*b, float3);
} }
void kernel float3_to_float4(global float3 *a, global float4 *b) { void kernel float3_to_float4(global float3 *a, global float4 *b) {
// CHECK-LABEL: spir_kernel void @float3_to_float4 // CHECK-LABEL: spir_kernel void @float3_to_float4
// CHECK: %[[LOAD_A:.*]] = load <3 x float>, <3 x float> addrspace(1)* %a, align 16 // CHECK: %[[LOAD_A:.*]] = load <3 x float>, <3 x float> addrspace(1)* %a, align 16
// CHECK: %[[ASTYPE:.*]] = shufflevector <3 x float> %[[LOAD_A]], <3 x float> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 undef> // CHECK: %[[ASTYPE:.*]] = shufflevector <3 x float> %[[LOAD_A]], <3 x float> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 poison>
efriedmaUnsubmitted
Not Done
ReplyInline Actions

This change might be visible to user code.

efriedma: This change might be visible to user code.
// CHECK: store <4 x float> %[[ASTYPE]], <4 x float> addrspace(1)* %b, align 16 // CHECK: store <4 x float> %[[ASTYPE]], <4 x float> addrspace(1)* %b, align 16
*b = __builtin_astype(*a, float4); *b = __builtin_astype(*a, float4);
} }
void kernel float3_to_double2(global float3 *a, global double2 *b) { void kernel float3_to_double2(global float3 *a, global double2 *b) {
// CHECK-LABEL: spir_kernel void @float3_to_double2 // CHECK-LABEL: spir_kernel void @float3_to_double2
// CHECK: %[[LOAD_A:.*]] = load <3 x float>, <3 x float> addrspace(1)* %a, align 16 // CHECK: %[[LOAD_A:.*]] = load <3 x float>, <3 x float> addrspace(1)* %a, align 16
// CHECK: %[[ASTYPE:.*]] = shufflevector <3 x float> %[[LOAD_A]], <3 x float> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 undef> // CHECK: %[[ASTYPE:.*]] = shufflevector <3 x float> %[[LOAD_A]], <3 x float> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 undef>
Show All 20 Lines

clang/test/CodeGenOpenCL/vector_literals.cl

clang/test/CodeGenOpenCL/vector_shufflevector.cl

llvm/include/llvm-c/Core.h

llvm/include/llvm/IR/Instructions.h

llvm/lib/Analysis/InstructionSimplify.cpp

llvm/lib/Analysis/ValueTracking.cpp

llvm/lib/IR/AsmWriter.cpp

llvm/lib/IR/ConstantFold.cpp

llvm/lib/IR/Core.cpp

llvm/lib/IR/Instructions.cpp

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

llvm/lib/Transforms/InstCombine/InstCombineSimplifyDemanded.cpp

llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

llvm/lib/Transforms/Scalar/ScalarizeMaskedMemIntrin.cpp

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

llvm/test/Analysis/CostModel/AMDGPU/shufflevector.ll

llvm/test/Analysis/CostModel/X86/reduction.ll

llvm/test/Analysis/CostModel/X86/shuffle-extract_subvector.ll

llvm/test/Analysis/CostModel/X86/shuffle-insert_subvector.ll

llvm/test/Analysis/CostModel/X86/shuffle-single-src.ll

llvm/test/CodeGen/AMDGPU/rewrite-out-arguments.ll

llvm/test/CodeGen/Generic/expand-experimental-reductions.ll

llvm/test/CodeGen/PowerPC/arg_promotion.ll

llvm/test/Transforms/CodeGenPrepare/X86/x86-shuffle-sink-inseltpoison.ll

llvm/test/Transforms/CodeGenPrepare/X86/x86-shuffle-sink.ll

llvm/test/Transforms/DeadStoreElimination/masked-dead-store-inseltpoison.ll

llvm/test/Transforms/DeadStoreElimination/masked-dead-store.ll

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts-inseltpoison.ll

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts.ll

llvm/test/Transforms/InstCombine/X86/x86-addsub-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-addsub.ll

llvm/test/Transforms/InstCombine/X86/x86-avx2-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-avx2.ll

llvm/test/Transforms/InstCombine/X86/x86-avx512-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-avx512.ll

llvm/test/Transforms/InstCombine/X86/x86-muldq-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-muldq.ll

llvm/test/Transforms/InstCombine/X86/x86-pack-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-pack.ll

llvm/test/Transforms/InstCombine/X86/x86-pshufb-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-pshufb.ll

llvm/test/Transforms/InstCombine/X86/x86-sse4a-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-sse4a.ll

llvm/test/Transforms/InstCombine/X86/x86-vpermil-inseltpoison.ll

llvm/test/Transforms/InstCombine/X86/x86-vpermil.ll

llvm/test/Transforms/InstCombine/broadcast-inseltpoison.ll

llvm/test/Transforms/InstCombine/broadcast.ll

llvm/test/Transforms/InstCombine/bswap-inseltpoison.ll

llvm/test/Transforms/InstCombine/bswap.ll

llvm/test/Transforms/InstCombine/canonicalize-vector-insert.ll

llvm/test/Transforms/InstCombine/extractelement-inseltpoison.ll

llvm/test/Transforms/InstCombine/extractelement.ll

llvm/test/Transforms/InstCombine/insert-const-shuf.ll

llvm/test/Transforms/InstCombine/insert-extract-shuffle-inseltpoison.ll

llvm/test/Transforms/InstCombine/insert-extract-shuffle.ll

llvm/test/Transforms/InstCombine/masked_intrinsics-inseltpoison.ll

llvm/test/Transforms/InstCombine/masked_intrinsics.ll

llvm/test/Transforms/InstCombine/nsw-inseltpoison.ll

llvm/test/Transforms/InstCombine/nsw.ll

llvm/test/Transforms/InstCombine/reduction-shufflevector.ll

llvm/test/Transforms/InstCombine/select-extractelement-inseltpoison.ll

llvm/test/Transforms/InstCombine/select-extractelement.ll

llvm/test/Transforms/InstCombine/select-select.ll

llvm/test/Transforms/InstCombine/shuffle-select-narrow-inseltpoison.ll

llvm/test/Transforms/InstCombine/shuffle-select-narrow.ll

llvm/test/Transforms/InstCombine/shuffle_select-inseltpoison.ll

llvm/test/Transforms/InstCombine/shuffle_select.ll

llvm/test/Transforms/InstCombine/shufflevec-bitcast-inseltpoison.ll

llvm/test/Transforms/InstCombine/shufflevec-bitcast.ll

llvm/test/Transforms/InstCombine/shufflevector-div-rem-inseltpoison.ll

llvm/test/Transforms/InstCombine/shufflevector-div-rem.ll

llvm/test/Transforms/InstCombine/sub-of-negatible-inseltpoison.ll

llvm/test/Transforms/InstCombine/sub-of-negatible.ll

llvm/test/Transforms/InstCombine/trunc-inseltpoison.ll

llvm/test/Transforms/InstCombine/trunc.ll

llvm/test/Transforms/InstCombine/type_pun-inseltpoison.ll

llvm/test/Transforms/InstCombine/type_pun.ll

llvm/test/Transforms/InstCombine/vec-binop-select-inseltpoison.ll

llvm/test/Transforms/InstCombine/vec-binop-select.ll

llvm/test/Transforms/InstCombine/vec_demanded_elts-inseltpoison.ll

llvm/test/Transforms/InstCombine/vec_demanded_elts.ll

llvm/test/Transforms/InstCombine/vec_shuffle-inseltpoison.ll

llvm/test/Transforms/InstCombine/vec_shuffle.ll

llvm/test/Transforms/InstCombine/vector-casts.ll

llvm/test/Transforms/InstCombine/vector-concat-binop-inseltpoison.ll

llvm/test/Transforms/InstCombine/vector-concat-binop.ll

llvm/test/Transforms/InstSimplify/shufflevector-inseltpoison.ll

llvm/test/Transforms/InstSimplify/shufflevector.ll

llvm/test/Transforms/InterleavedAccess/AArch64/interleaved-accesses-inseltpoison.ll

llvm/test/Transforms/InterleavedAccess/AArch64/interleaved-accesses.ll

llvm/test/Transforms/InterleavedAccess/ARM/interleaved-accesses-inseltpoison.ll

llvm/test/Transforms/InterleavedAccess/ARM/interleaved-accesses.ll

llvm/test/Transforms/InterleavedAccess/X86/interleaved-accesses-64bits-avx-inseltpoison.ll

llvm/test/Transforms/InterleavedAccess/X86/interleaved-accesses-64bits-avx.ll

llvm/test/Transforms/InterleavedAccess/X86/interleavedStore-inseltpoison.ll

llvm/test/Transforms/InterleavedAccess/X86/interleavedStore.ll

llvm/test/Transforms/LoopUnroll/X86/pr46430-inseltpoison.ll

llvm/test/Transforms/LoopUnroll/X86/pr46430.ll

llvm/test/Transforms/LoopVectorize/AArch64/interleaved-store-of-first-order-recurrence.ll

llvm/test/Transforms/LoopVectorize/X86/x86-interleaved-store-accesses-with-gaps.ll

llvm/test/Transforms/LoopVectorize/interleaved-accesses.ll

llvm/test/Transforms/LowerMatrixIntrinsics/bigger-expressions-double.ll

llvm/test/Transforms/LowerMatrixIntrinsics/const-gep.ll

llvm/test/Transforms/LowerMatrixIntrinsics/load-align-volatile.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-add-sub-double-row-major.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-double-contraction-fmf.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-double-contraction.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-double-row-major.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-double.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-float-contraction-fmf.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-float-contraction.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-float.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-i32-row-major.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-i32.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-left-transpose-row-major.ll

llvm/test/Transforms/LowerMatrixIntrinsics/multiply-right-transpose.ll

llvm/test/Transforms/LowerMatrixIntrinsics/preserve-existing-fast-math-flags.ll

llvm/test/Transforms/LowerMatrixIntrinsics/propagate-backwards-unsupported.ll

llvm/test/Transforms/LowerMatrixIntrinsics/strided-load-double.ll

llvm/test/Transforms/LowerMatrixIntrinsics/strided-load-float.ll

llvm/test/Transforms/LowerMatrixIntrinsics/strided-load-i32.ll

llvm/test/Transforms/LowerMatrixIntrinsics/transpose-double.ll

llvm/test/Transforms/LowerMatrixIntrinsics/transpose-float.ll

llvm/test/Transforms/LowerMatrixIntrinsics/transpose-i32.ll

llvm/test/Transforms/LowerMatrixIntrinsics/transpose-opts.ll

llvm/test/Transforms/PhaseOrdering/ARM/mve-floatreduce.ll

llvm/test/Transforms/PhaseOrdering/X86/vector-reductions-expanded.ll

llvm/test/Transforms/PhaseOrdering/X86/vector-reductions-logical.ll

llvm/test/Transforms/PhaseOrdering/X86/vector-reductions.ll

llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/AArch64/accelerate-vector-functions.ll

llvm/test/Transforms/SLPVectorizer/AArch64/transpose-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/AArch64/transpose.ll

llvm/test/Transforms/SLPVectorizer/AArch64/vectorize-free-extracts-inserts.ll

llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/AMDGPU/add_sub_sat.ll

llvm/test/Transforms/SLPVectorizer/SystemZ/pr34619.ll

llvm/test/Transforms/SLPVectorizer/X86/PR35865-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/PR35865.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-calls-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-calls.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-cast-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-cast.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-fp-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-fp.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll

llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll

llvm/test/Transforms/SLPVectorizer/X86/blending-shuffle-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/blending-shuffle.ll

llvm/test/Transforms/SLPVectorizer/X86/cmp_commute-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/cmp_commute.ll

llvm/test/Transforms/SLPVectorizer/X86/extract_with_non_const_index.ll

llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll

llvm/test/Transforms/SLPVectorizer/X86/insert-shuffle.ll

llvm/test/Transforms/SLPVectorizer/X86/load-merge-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/load-merge.ll

llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll

llvm/test/Transforms/SLPVectorizer/X86/phi.ll

llvm/test/Transforms/SLPVectorizer/X86/pr47629-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/pr47629.ll

llvm/test/Transforms/SLPVectorizer/X86/pr47642.ll

llvm/test/Transforms/SLPVectorizer/X86/pr49081.ll

llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll

llvm/test/Transforms/SLPVectorizer/X86/resched.ll

llvm/test/Transforms/SLPVectorizer/X86/sitofp-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/sitofp.ll

llvm/test/Transforms/SLPVectorizer/X86/vec_list_bias-inseltpoison.ll

llvm/test/Transforms/SLPVectorizer/X86/vec_list_bias.ll

llvm/test/Transforms/SROA/slice-width.ll

llvm/test/Transforms/SROA/vector-promotion.ll

llvm/test/Transforms/VectorCombine/AArch64/load-extractelement-scalarization.ll

llvm/test/Transforms/VectorCombine/AMDGPU/as-transition-inseltpoison.ll

llvm/test/Transforms/VectorCombine/AMDGPU/as-transition.ll

llvm/test/Transforms/VectorCombine/X86/extract-binop-inseltpoison.ll

llvm/test/Transforms/VectorCombine/X86/extract-binop.ll

llvm/test/Transforms/VectorCombine/X86/extract-cmp-binop.ll

llvm/test/Transforms/VectorCombine/X86/extract-cmp.ll

llvm/test/Transforms/VectorCombine/X86/load-inseltpoison.ll

llvm/test/Transforms/VectorCombine/X86/load.ll

llvm/tools/llvm-c-test/echo.cpp