[test-suite] Add unit test for libmvec sincos() auto-vectorisation

[tim.schmielau]

Summary

Prepare for merging https://reviews.llvm.org/D116879 into LLVM,
which enables vectorization of sincos() via libmvec.

Compared to the other functions in libmvec, sincos() vectorisation
offers additional pitfalls as it returns two results via pointers.
Thus a test seems justified that checks the transformations performed
by llvm match the interface of the underlying vector library.

The test is run for any instruction set in {SSE, SSE2, AVX, AVX2, AVX512F}
supported by the CPU as well as with -march=native to improve coverage.
The supplied CMakeLists.txt only enables the check for -fveclib=libmvec on x86.
I leave enabling other vector libraries to people who can test those setups.
There is no check for successfull vectorization here, however a
regression test in the llvm repo already checks that.

[tim.schmielau]

fix mixed up comments

[tim.schmielau]

my previous update seems to have eaten the original changes.
Let's see if I can get this right.

[fhahn]

Thanks for putting up the test. I am not sure if it should be Libmvec or X86 specific though. There are other vector libraries on different platforms that provide vector versions of sincos. I don't think we should exclude those from this testing here.

SingleSource/UnitTests/Vectorizer/CMakeLists.txt

4 ↗

(On Diff #412952)

Is this enough to ensure that the library is actually available on the system or could the compiler have the flag but the linker can't find libmvec?

[tim.schmielau]

I am not sure if it should be Libmvec or X86 specific though. There are other vector libraries on different platforms that provide vector versions of sincos. I don't think we should exclude those from this testing here.

I've been unsure about this as well. My preference would be to always run the test, but (somehow) issue a warning if the code does not get vectorized.

However a warning would alter output and thus make the test fail (unless the warning goes to stderr, but I'm not sure that is desirable either). I'm open to suggestions!

SingleSource/UnitTests/Vectorizer/CMakeLists.txt

4 ↗

(On Diff #412952)

Indeed, thanks for pointing that out. With consideration of your other comment, shall I drop the entire SSE2 and libmvec detection logic?

[tim.schmielau]

Even if we run the test unconditionally, some logic will still be required to select the right flags to enable auto-vectorization.

I am wondering if the aim should be to make auto-vectorization of math library calls a commodity that does not require special compiler flags but is automatically enabled with -O3. In that case, the detection logic shouldn't go into the test suite but the compiler itself.

[fhahn]

In D120977#3359480, @tim.schmielau wrote:

Even if we run the test unconditionally, some logic will still be required to select the right flags to enable auto-vectorization.

I'm not sure what you mean here. Are you talking about automatically choosing a vector library (-fveclib)? I am not sure that will be feasible, as there are 2 potential complications:

1. Unless the platform guarantees that the vector library will always be available the compiler has no way of knowing what libraries are available.
2. Most vector libraries are not drop-in replacements for existing math functions, i.e. they will not produce the same result in all cases

Note that we could check in CMake if the library is available by trying to link a simple program that uses a function from libmvec.

[tim.schmielau]

Are you talking about automatically choosing a vector library (-fveclib)?

Yes, where available I want this test to be run with a vector library. As you point out, this is not entirely trivial though. I'll follow your suggestion and make the directory generic, with a specific test for libmvec presence to enable the test with required flags. Others can then add suitable setups for other vector libraries where they are able to check the setup is working.

[tim.schmielau]

Add check that libmvec is available for linking and at runtime
and rename Libmvec/ directory to Veclib/ to indicate the test is
not specific to libmvec and might be enabled for other vector
libraries as well.

[RKSimon]

cmake/modules/DetectLibmvecX86.c

6 ↗

(On Diff #413045)

do we need to test for 32 and 64 bit x86? with 32-bit pointers will the args take __m128i ?

[tim.schmielau]

• check for AVX2, not SSE2

What a blunder...

[tim.schmielau]

cmake/modules/DetectLibmvecX86.c

6 ↗

(On Diff #413045)

I don't know. https://sourceware.org/glibc/wiki/libmvec is silent about anything else bit x86-64 so I'll write a 32 bit test to find out...

[tim.schmielau]

• handle 32 bit pointer size

[tim.schmielau]

cmake/modules/DetectLibmvecX86.c

6 ↗

(On Diff #413045)

You are of course right. I've updated the code to handle 32 bit wordsize.
Now anyone tell me about a case where pointer size isn't the same as word size?

[tim.schmielau]

• run libmvec tests for each supported ISA
• don't depend on immintrin.h

It turns out my mixup of AVX2 and SSE2 had its value, although in an unexpected way:
I've modified the test so that it runs for all instruction sets in {SSE, SSE2, AVX, AVX2, AVX512F} as well as with -march=native to improve coverage.
And indeed SSE and SSE2 tests are currently failing when run with the changes proposed in https://reviews.llvm.org/D116879.
The failures look like they could be related to outer loop vectorisation, but will require further investigation.

[tim.schmielau]

• run tests in both single and double precision

Interestingly, the SSE and SSE2 tests succeed in single precision.

I haven't tried to debug the failures yet but that is what I need to do next.
I am happy with the coverage this test now provides, but maybe I'll still learn more from debugging the failures.

[tim.schmielau]

• compute reference result in long double

to ensure it cannot get vectorized as well.

[tim.schmielau]

• increase nest count
• add nounroll sincos() tests
• make sincos test fail reliably even in single precision
• protect against wrongly computed reference results by printing a few values

So it turns out I was quite mistaken when I started the LLVM sincos() autovectorization
patch assuming I am only exposing functionality already available via OpenMP.
Careful reading of the OpenMP documentation for the current development branch at
https://clang.llvm.org/docs/OpenMPSupport.html, or a look at the LLVM sources reveals
that while #pragma omp declare simd is implemented, #pragma omp simd is not.
I.e., the OpenMP equivalent for declaring vectorized functions via -fveclib is present,
but the necessary analysis for deciding whether a function call can safely be vectorized
is not.
Instead, the decision relies on the simple heuristc that any currently exposed function is safe.
This assumption is of course violated if sincos() is added to the list of vectorized functions.

Given this status, all nested tests in this commit should have failed. What made some of them pass is
that LLVM chooses to aggressively interleave the inner loop of the nested test with interleave
factors up to 4. With vector lengths up to 8 for single precision AVX2 this means that at least
32 inner loop iterations are required before the incorrect dependency analysis reliably leads to
incorrect concurrent evaluation of dependent expressions and reliable test failure.

An alternative way of making the test fail is to avoid interleaving the loop by adding #pragma unroll 1.
I've done both for better coverage - increased the nested loop iteration beyond 32, and added a separate
test with unrolling disabled.

The updated test now always fails with my current incorrect patch at https://reviews.llvm.org/D120977,
and succeeds with the fixed version that I'll post there shortly.

[tim.schmielau]

• at newlines at end of file where missing

[tim.schmielau]

Add tests using #pragma clang loop vectorize(...).

SingleSource/UnitTests/Vectorizer/Veclib/sincos.h

 #define _GNU_SOURCE
 
 #include <stdio.h>
 #include <math.h>
 
 // inner loop vectorization with linear access
 void sincos_arr(REAL* sines, REAL* cosines, REAL* phases, int size) {
     for (int i=0; i<size; i++) {
         SINCOS(phases[i], sines+i, cosines+i);
     }
 }
 
+// inner loop vectorization with linear access, assume safety
+void sincos_arr_assume_safety(REAL* sines, REAL* cosines, REAL* phases, int size) {
+#pragma clang loop vectorize(assume_safety)
+    for (int i=0; i<size; i++) {
+        SINCOS(phases[i], sines+i, cosines+i);
+    }
+}
+
+// dependent inner loop, not vectorizable
+void sincos_arr_dependent(REAL* sines, REAL* cosines, int size) {
+    SINCOS((REAL)0, sines, cosines);
+#pragma clang loop vectorize(enable)
+    for (int i=1; i<size; i++) {
+        SINCOS(cosines[i-1]-sines[i-1], sines+i, cosines+i);
+    }
+}
+
 // outer loop vectorization with linear access
 void sincos_arr_nested(REAL* sines, REAL* cosines, REAL* phases, int size, int nests) {
     for (int i=0; i<size; i++) {
         SINCOS(phases[i], sines+i, cosines+i);
         for (int k=0; k<nests; k++) {
             SINCOS(cosines[i]-sines[i], sines+i, cosines+i);
         }
     }
if (fabs(val - ref) > max_ulp_err * ulp) {
         return 1;
     } else if (verbose) {
         printf("%s.%s at %2d: argument %2g, value %10.7f, reference %10.7f.\n",
                msg1, msg2, index, arg, val, ref);
     }
     return 0;
 }
 
-int check_arr(REAL* sines, REAL* coses, REAL* phases, int* indices, int size, int nests, REAL max_ulp_err, const char* msg)
+int check_arr(REAL* sines, REAL* coses, REAL* phases, int* indices, int size, int dependent, int nests, REAL max_ulp_err, const char* msg)
 {
     int fail = 0;
 
     for (int i = 0; i < size; i++) {
         int j = indices ? indices[i] : i;
-        long double ref_sin = sinl(phases[j]);
-        long double ref_cos = cosl(phases[j]);
+        REAL phase = dependent ? (i > 0 ? coses[i-1] - sines[i-1] : 0) : phases[j];
+        long double ref_sin = sinl(phase);
+        long double ref_cos = cosl(phase);
         for (int k=0; k<nests; k++) {
                 REAL arg = ref_cos - ref_sin;
                 ref_sin = sinl(arg);
                 ref_cos = cosl(arg);
             }
         fail |= check_ULP(phases[j], sines[j], (REAL)ref_sin, max_ulp_err, i, msg, "sin", i%8 == 0);
         fail |= check_ULP(phases[j], coses[j], (REAL)ref_cos, max_ulp_err, i, msg, "cos", i%8 == 0);
     }
for (int l=0; l<loops; l++) {
         // test with linear access:
         for (int i = 0; i < N; i++) {
             phases[i] = i;
             indices[i] = i;
             sines[i] = 0.0 / 0.0;
             coses[i] = 0.0 / 0.0;
         }
         sincos_arr(sines, coses, phases, N);
-        fail |= check_arr(sines, coses, phases, NULL, N, 0, MAX_ULP_ERR, "sincos_arr");
+        fail |= check_arr(sines, coses, phases, NULL, N, 0, 0, MAX_ULP_ERR, "sincos_arr");
+        sincos_arr_assume_safety(sines, coses, phases, N);
+        fail |= check_arr(sines, coses, phases, NULL, N, 0, 0, MAX_ULP_ERR, "sincos_arr_assume_safety");
+        sincos_arr_dependent(sines, coses, N);
+        fail |= check_arr(sines, coses, phases, NULL, N, 1, 0, MAX_ULP_ERR, "sincos_arr_dependent");
         sincos_arr_nested(sines, coses, phases, N, nests);
-        fail |= check_arr(sines, coses, phases, NULL, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_nested");
+        fail |= check_arr(sines, coses, phases, NULL, N, 0, nests, MAX_ULP_ERR_NESTED, "sincos_arr_nested");
         sincos_arr_nested_nounroll(sines, coses, phases, N, nests);
-        fail |= check_arr(sines, coses, phases, NULL, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_nested_nounroll");
+        fail |= check_arr(sines, coses, phases, NULL, N, 0, nests, MAX_ULP_ERR_NESTED, "sincos_arr_nested_nounroll");
 
         // test with nonlinear access:
         for (int i = 0; i < N; i++) {
             indices[i] = (i < 2) ? 1 : (indices[i - 2] + indices[i - 1]) % N;
             sines[i] = 0.0 / 0.0;
             coses[i] = 0.0 / 0.0;
         }
         sincos_arr_scrambled(sines, coses, phases, indices, N);
-        fail |= check_arr(sines, coses, phases, indices, N, 0, MAX_ULP_ERR, "sincos_arr_scrambled");
+        fail |= check_arr(sines, coses, phases, indices, N, 0, 0, MAX_ULP_ERR, "sincos_arr_scrambled");
         sincos_arr_scrambled_nested(sines, coses, phases, indices, N, nests);
-        fail |= check_arr(sines, coses, phases, indices, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_scrambled_nested");
+        fail |= check_arr(sines, coses, phases, indices, N, 0, nests, MAX_ULP_ERR_NESTED, "sincos_arr_scrambled_nested");
         sincos_arr_scrambled_nested_nounroll(sines, coses, phases, indices, N, nests);
-        fail |= check_arr(sines, coses, phases, indices, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_scrambled_nested_nounroll");
+        fail |= check_arr(sines, coses, phases, indices, N, 0, nests, MAX_ULP_ERR_NESTED, "sincos_arr_scrambled_nested_nounroll");
     }
     if (fail) {
         printf("FAILED!\n");
     } else {
         printf("Passed.\n");
     }
     return fail;
 }

SingleSource/UnitTests/Vectorizer/Veclib/sincos.reference_output

 sincos_arr.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
 sincos_arr.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
 sincos_arr.sin at  8: argument  8, value  0.9893582, reference  0.9893582.
 sincos_arr.cos at  8: argument  8, value -0.1455000, reference -0.1455000.
 sincos_arr.sin at 16: argument 16, value -0.2879033, reference -0.2879033.
 sincos_arr.cos at 16: argument 16, value -0.9576595, reference -0.9576595.
 sincos_arr.sin at 24: argument 24, value -0.9055784, reference -0.9055784.
 sincos_arr.cos at 24: argument 24, value  0.4241790, reference  0.4241790.
+sincos_arr_assume_safety.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
+sincos_arr_assume_safety.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
+sincos_arr_assume_safety.sin at  8: argument  8, value  0.9893582, reference  0.9893582.
+sincos_arr_assume_safety.cos at  8: argument  8, value -0.1455000, reference -0.1455000.
+sincos_arr_assume_safety.sin at 16: argument 16, value -0.2879033, reference -0.2879033.
+sincos_arr_assume_safety.cos at 16: argument 16, value -0.9576595, reference -0.9576595.
+sincos_arr_assume_safety.sin at 24: argument 24, value -0.9055784, reference -0.9055784.
+sincos_arr_assume_safety.cos at 24: argument 24, value  0.4241790, reference  0.4241790.
+sincos_arr_dependent.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
+sincos_arr_dependent.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
+sincos_arr_dependent.sin at  8: argument  8, value -0.7387255, reference -0.7387255.
+sincos_arr_dependent.cos at  8: argument  8, value  0.6740065, reference  0.6740065.
+sincos_arr_dependent.sin at 16: argument 16, value -0.7380653, reference -0.7380653.
+sincos_arr_dependent.cos at 16: argument 16, value  0.6747292, reference  0.6747292.
+sincos_arr_dependent.sin at 24: argument 24, value -0.7380653, reference -0.7380653.
+sincos_arr_dependent.cos at 24: argument 24, value  0.6747293, reference  0.6747293.
 sincos_arr_nested.sin at  0: argument  0, value  0.9875437, reference  0.9875437.
 sincos_arr_nested.cos at  0: argument  0, value  0.1573452, reference  0.1573452.
 sincos_arr_nested.sin at  8: argument  8, value -0.7380653, reference -0.7380653.
 sincos_arr_nested.cos at  8: argument  8, value  0.6747293, reference  0.6747293.
 sincos_arr_nested.sin at 16: argument 16, value -0.7380653, reference -0.7380653.
 sincos_arr_nested.cos at 16: argument 16, value  0.6747293, reference  0.6747293.
 sincos_arr_nested.sin at 24: argument 24, value  0.9875437, reference  0.9875437.
 sincos_arr_nested.cos at 24: argument 24, value  0.1573452, reference  0.1573452.

SingleSource/UnitTests/Vectorizer/Veclib/sincosf.reference_output

 sincos_arr.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
 sincos_arr.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
 sincos_arr.sin at  8: argument  8, value  0.9893582, reference  0.9893582.
 sincos_arr.cos at  8: argument  8, value -0.1455000, reference -0.1455000.
 sincos_arr.sin at 16: argument 16, value -0.2879033, reference -0.2879033.
 sincos_arr.cos at 16: argument 16, value -0.9576595, reference -0.9576595.
 sincos_arr.sin at 24: argument 24, value -0.9055784, reference -0.9055784.
 sincos_arr.cos at 24: argument 24, value  0.4241790, reference  0.4241790.
+sincos_arr_assume_safety.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
+sincos_arr_assume_safety.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
+sincos_arr_assume_safety.sin at  8: argument  8, value  0.9893582, reference  0.9893582.
+sincos_arr_assume_safety.cos at  8: argument  8, value -0.1455000, reference -0.1455000.
+sincos_arr_assume_safety.sin at 16: argument 16, value -0.2879033, reference -0.2879033.
+sincos_arr_assume_safety.cos at 16: argument 16, value -0.9576595, reference -0.9576595.
+sincos_arr_assume_safety.sin at 24: argument 24, value -0.9055784, reference -0.9055784.
+sincos_arr_assume_safety.cos at 24: argument 24, value  0.4241790, reference  0.4241790.
+sincos_arr_dependent.sin at  0: argument  0, value  0.0000000, reference  0.0000000.
+sincos_arr_dependent.cos at  0: argument  0, value  1.0000000, reference  1.0000000.
+sincos_arr_dependent.sin at  8: argument  8, value -0.7387254, reference -0.7387254.
+sincos_arr_dependent.cos at  8: argument  8, value  0.6740065, reference  0.6740065.
+sincos_arr_dependent.sin at 16: argument 16, value -0.7380654, reference -0.7380654.
+sincos_arr_dependent.cos at 16: argument 16, value  0.6747292, reference  0.6747292.
+sincos_arr_dependent.sin at 24: argument 24, value -0.7380654, reference -0.7380654.
+sincos_arr_dependent.cos at 24: argument 24, value  0.6747292, reference  0.6747292.
 sincos_arr_nested.sin at  0: argument  0, value  0.9875436, reference  0.9875436.
 sincos_arr_nested.cos at  0: argument  0, value  0.1573451, reference  0.1573451.
 sincos_arr_nested.sin at  8: argument  8, value -0.7380654, reference -0.7380654.
 sincos_arr_nested.cos at  8: argument  8, value  0.6747292, reference  0.6747292.
 sincos_arr_nested.sin at 16: argument 16, value -0.7380654, reference -0.7380654.
 sincos_arr_nested.cos at 16: argument 16, value  0.6747292, reference  0.6747292.
 sincos_arr_nested.sin at 24: argument 24, value  0.9875436, reference  0.9875436.
 sincos_arr_nested.cos at 24: argument 24, value  0.1573451, reference  0.1573451.