[test-suite] Add unit test for libmvec sincos() auto-vectorisation

[tim.schmielau]

Summary

Prepare for merging https://reviews.llvm.org/D116879 into LLVM,
which enables vectorization of sincos() via libmvec.

Compared to the other functions in libmvec, sincos() vectorisation
offers additional pitfalls as it returns two results via pointers.
Thus a test seems justified that checks the transformations performed
by llvm match the interface of the underlying vector library.

The test is run for any instruction set in {SSE, SSE2, AVX, AVX2, AVX512F}
supported by the CPU as well as with -march=native to improve coverage.
The supplied CMakeLists.txt only enables the check for -fveclib=libmvec on x86.
I leave enabling other vector libraries to people who can test those setups.
There is no check for successfull vectorization here, however a
regression test in the llvm repo already checks that.

[tim.schmielau]

fix mixed up comments

[tim.schmielau]

my previous update seems to have eaten the original changes.
Let's see if I can get this right.

[fhahn]

Thanks for putting up the test. I am not sure if it should be Libmvec or X86 specific though. There are other vector libraries on different platforms that provide vector versions of sincos. I don't think we should exclude those from this testing here.

SingleSource/UnitTests/Vectorizer/CMakeLists.txt

5

Is this enough to ensure that the library is actually available on the system or could the compiler have the flag but the linker can't find libmvec?

[tim.schmielau]

I am not sure if it should be Libmvec or X86 specific though. There are other vector libraries on different platforms that provide vector versions of sincos. I don't think we should exclude those from this testing here.

I've been unsure about this as well. My preference would be to always run the test, but (somehow) issue a warning if the code does not get vectorized.

However a warning would alter output and thus make the test fail (unless the warning goes to stderr, but I'm not sure that is desirable either). I'm open to suggestions!

SingleSource/UnitTests/Vectorizer/CMakeLists.txt

5

Indeed, thanks for pointing that out. With consideration of your other comment, shall I drop the entire SSE2 and libmvec detection logic?

[tim.schmielau]

Even if we run the test unconditionally, some logic will still be required to select the right flags to enable auto-vectorization.

I am wondering if the aim should be to make auto-vectorization of math library calls a commodity that does not require special compiler flags but is automatically enabled with -O3. In that case, the detection logic shouldn't go into the test suite but the compiler itself.

[fhahn]

In D120977#3359480, @tim.schmielau wrote:

Even if we run the test unconditionally, some logic will still be required to select the right flags to enable auto-vectorization.

I'm not sure what you mean here. Are you talking about automatically choosing a vector library (-fveclib)? I am not sure that will be feasible, as there are 2 potential complications:

1. Unless the platform guarantees that the vector library will always be available the compiler has no way of knowing what libraries are available.
2. Most vector libraries are not drop-in replacements for existing math functions, i.e. they will not produce the same result in all cases

Note that we could check in CMake if the library is available by trying to link a simple program that uses a function from libmvec.

[tim.schmielau]

Are you talking about automatically choosing a vector library (-fveclib)?

Yes, where available I want this test to be run with a vector library. As you point out, this is not entirely trivial though. I'll follow your suggestion and make the directory generic, with a specific test for libmvec presence to enable the test with required flags. Others can then add suitable setups for other vector libraries where they are able to check the setup is working.

[tim.schmielau]

Add check that libmvec is available for linking and at runtime
and rename Libmvec/ directory to Veclib/ to indicate the test is
not specific to libmvec and might be enabled for other vector
libraries as well.

[RKSimon]

cmake/modules/DetectLibmvecX86.c

7

do we need to test for 32 and 64 bit x86? with 32-bit pointers will the args take __m128i ?

[tim.schmielau]

• check for AVX2, not SSE2

What a blunder...

[tim.schmielau]

cmake/modules/DetectLibmvecX86.c

7

I don't know. https://sourceware.org/glibc/wiki/libmvec is silent about anything else bit x86-64 so I'll write a 32 bit test to find out...

[tim.schmielau]

• handle 32 bit pointer size

[tim.schmielau]

cmake/modules/DetectLibmvecX86.c

7

You are of course right. I've updated the code to handle 32 bit wordsize.
Now anyone tell me about a case where pointer size isn't the same as word size?

[tim.schmielau]

• run libmvec tests for each supported ISA
• don't depend on immintrin.h

It turns out my mixup of AVX2 and SSE2 had its value, although in an unexpected way:
I've modified the test so that it runs for all instruction sets in {SSE, SSE2, AVX, AVX2, AVX512F} as well as with -march=native to improve coverage.
And indeed SSE and SSE2 tests are currently failing when run with the changes proposed in https://reviews.llvm.org/D116879.
The failures look like they could be related to outer loop vectorisation, but will require further investigation.

[tim.schmielau]

• run tests in both single and double precision

Interestingly, the SSE and SSE2 tests succeed in single precision.

I haven't tried to debug the failures yet but that is what I need to do next.
I am happy with the coverage this test now provides, but maybe I'll still learn more from debugging the failures.

[tim.schmielau]

• compute reference result in long double

to ensure it cannot get vectorized as well.

[tim.schmielau]

• increase nest count
• add nounroll sincos() tests
• make sincos test fail reliably even in single precision
• protect against wrongly computed reference results by printing a few values

So it turns out I was quite mistaken when I started the LLVM sincos() autovectorization
patch assuming I am only exposing functionality already available via OpenMP.
Careful reading of the OpenMP documentation for the current development branch at
https://clang.llvm.org/docs/OpenMPSupport.html, or a look at the LLVM sources reveals
that while #pragma omp declare simd is implemented, #pragma omp simd is not.
I.e., the OpenMP equivalent for declaring vectorized functions via -fveclib is present,
but the necessary analysis for deciding whether a function call can safely be vectorized
is not.
Instead, the decision relies on the simple heuristc that any currently exposed function is safe.
This assumption is of course violated if sincos() is added to the list of vectorized functions.

Given this status, all nested tests in this commit should have failed. What made some of them pass is
that LLVM chooses to aggressively interleave the inner loop of the nested test with interleave
factors up to 4. With vector lengths up to 8 for single precision AVX2 this means that at least
32 inner loop iterations are required before the incorrect dependency analysis reliably leads to
incorrect concurrent evaluation of dependent expressions and reliable test failure.

An alternative way of making the test fail is to avoid interleaving the loop by adding #pragma unroll 1.
I've done both for better coverage - increased the nested loop iteration beyond 32, and added a separate
test with unrolling disabled.

The updated test now always fails with my current incorrect patch at https://reviews.llvm.org/D120977,
and succeeds with the fixed version that I'll post there shortly.

[tim.schmielau]

• at newlines at end of file where missing

[tim.schmielau]

Add tests using #pragma clang loop vectorize(...).

CMakeLists.txt

 endif()
 if(NOT DEFINED ARCH)
   include(DetectArchitecture)
   detect_architecture(ARCH)
 endif()
 if(NOT DEFINED X86CPU_ARCH AND ARCH STREQUAL "x86")
   include(DetectArchitecture)
   detect_x86_cpu_architecture(X86CPU_ARCH)
+  detect_x86_feature(sse HAVE_X86_SSE_INSTRUCTIONS)
+  detect_x86_feature(sse2 HAVE_X86_SSE2_INSTRUCTIONS)
+  detect_x86_feature(avx HAVE_X86_AVX_INSTRUCTIONS)
+  detect_x86_feature(avx2 HAVE_X86_AVX2_INSTRUCTIONS)
+  detect_x86_feature(avx512f HAVE_X86_AVX512F_INSTRUCTIONS)
+endif()
+if(NOT HAVE_LIBMVEC_X86 AND ARCH STREQUAL "x86")
+  include(DetectVectorLibrary)
+  detect_libmvec_x86(HAVE_LIBMVEC_X86)
 endif()
 if(NOT DEFINED ENDIAN)
   include(TestBigEndian)
   test_big_endian(IS_BIGENDIAN)
   if(IS_BIGENDIAN)
     set(ENDIAN "big")
   else()
     set(ENDIAN "little")

SingleSource/UnitTests/Vectorizer/CMakeLists.txt

+add_subdirectory(Veclib)
 llvm_singlesource()
 set_property(TARGET runtime-checks PROPERTY CXX_STANDARD 17)

SingleSource/UnitTests/Vectorizer/Veclib/CMakeLists.txt

+function(llvm_singlesource_with_cflags_ldflags prefix additional_cflags additional_ldflags)
+    list(APPEND CFLAGS ${additional_cflags})
+    list(APPEND LDFLAGS ${additional_ldflags})
+    llvm_singlesource(PREFIX ${prefix})
+endfunction(llvm_singlesource_with_cflags_ldflags)
+
+if(HAVE_LIBMVEC_X86)
+    check_c_compiler_flag(-fveclib=libmvec COMPILER_HAS_FVECLIB_LIBMVEC_FLAG)
+    if(COMPILER_HAS_FVECLIB_LIBMVEC_FLAG)
+        llvm_singlesource_with_cflags_ldflags("libmvec-native-" "-fveclib=libmvec;-O2;-march=native" "-lm;-fveclib=libmvec")
+        if(HAVE_X86_SSE_INSTRUCTIONS AND HAVE_LIBMVEC_X86)
+            llvm_singlesource_with_cflags_ldflags("libmvec-sse-" "-fveclib=libmvec;-O2;-msse" "-lm;-fveclib=libmvec")
+        endif()
+        if(HAVE_X86_SSE2_INSTRUCTIONS AND HAVE_LIBMVEC_X86)
+            llvm_singlesource_with_cflags_ldflags("libmvec-sse2-" "-fveclib=libmvec;-O2;-msse2" "-lm;-fveclib=libmvec")
+        endif()
+        if(HAVE_X86_AVX_INSTRUCTIONS AND HAVE_LIBMVEC_X86)
+            llvm_singlesource_with_cflags_ldflags("libmvec-avx-" "-fveclib=libmvec;-O2;-mavx" "-lm;-fveclib=libmvec")
+        endif()
+        if(HAVE_X86_AVX2_INSTRUCTIONS AND HAVE_LIBMVEC_X86)
+            llvm_singlesource_with_cflags_ldflags("libmvec-avx2-" "-fveclib=libmvec;-O2;-mavx2" "-lm;-fveclib=libmvec")
+        endif()
+        if(HAVE_X86_AVX512F_INSTRUCTIONS AND HAVE_LIBMVEC_X86)
+            llvm_singlesource_with_cflags_ldflags("libmvec-avx512f-" "-fveclib=libmvec;-O2;-mavx512f" "-lm;-fveclib=libmvec")
+        endif()
+    endif()
+endif()
+# Other vector math libraries may be added here...

SingleSource/UnitTests/Vectorizer/Veclib/sincos.c

+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <math.h>
+
+// inner loop vectorization with linear access
+void sincos_arr(double* sines, double* cosines, double* phases, int size) {
+    for (int i=0; i<size; i++) {
+        sincos(phases[i], sines+i, cosines+i);
+    }
+}
+
+// outer loop vectorization with linear access
+void sincos_arr_nested(double* sines, double* cosines, double* phases, int size, int nests) {
+    for (int i=0; i<size; i++) {
+        sincos(phases[i], sines+i, cosines+i);
+        for (int k=0; k<nests; k++) {
+            sincos(sines[i]+cosines[i], sines+i, cosines+i);
+        }
+    }
+}
+
+// inner loop vectorization with nonlinear access
+void sincos_arr_scrambled(double* sines, double* cosines, double* phases, int* indices, int size) {
+    for (int i=0; i<size; i++) {
+        int j = indices[i];
+        sincos(phases[j], sines+j, cosines+j);
+    }
+}
+
+// outer loop vectorization with nonlinear access
+void sincos_arr_scrambled_nested(double* sines, double* cosines, double* phases, int* indices, int size, int nests) {
+    for (int i=0; i<size; i++) {
+        int j = indices[i];
+        sincos(phases[j], sines+j, cosines+j);
+        for (int k=0; k<nests; k++) {
+            sincos(sines[j]+cosines[j], sines+j, cosines+j);
+        }
+    }
+}
+
+// returns true (=failure) iff ULP error is larger than max_ulp_err, with a simplified definition of ULP
+int check_ULP(double arg, double val, double ref, double max_ulp_err, int index, const char* msg1, const char* msg2)
+{
+    double ulp = 0.5 * (nextafter(ref, INFINITY) - nextafter(ref, -INFINITY));
+    if (fabs(val - ref) > max_ulp_err * ulp) {
+        printf("%s,%s FAILED at %d: argument %g, value %g, reference %g, difference %g, (%g ULP).\n",
+               msg1, msg2, index, arg, val, ref, (val-ref), fabs(val-ref)/ulp);
+        return 1;
+    }
+    return 0;
+}
+
+int check_arr(double* sines, double* coses, double* phases, int* indices, int size, int nests, double max_ulp_err, const char* msg)
+{
+    int fail = 0;
+
+    for (int i = 0; i < size; i++) {
+        int j = indices ? indices[i] : i;
+        double ref_sin = sin(phases[j]);
+        double ref_cos = cos(phases[j]);
+        for (int k=0; k<nests; k++) {
+                double arg = ref_sin+ref_cos;
+                ref_sin = sin(arg);
+                ref_cos = cos(arg);
+            }
+        fail |= check_ULP(phases[j], sines[j], ref_sin, max_ulp_err, i, msg, "sin");
+        fail |= check_ULP(phases[j], coses[j], ref_cos, max_ulp_err, i, msg, "cos");
+    }
+    return fail;
+}
+
+int main(int argc, char *argv[])
+{
+    argc |= 1; // ensure argc is odd for better test coverage
+    const int loops = argc;
+    const int N = 26 + argc;
+    const int nests = 4 + argc;
+    int indices[N];
+    double phases[N], sines[N], coses[N];
+    int fail = 0;
+    const double MAX_ULP_ERR_NESTED = 64;
+    const double MAX_ULP_ERR = 16;
+
+    for (int l=0; l<loops; l++) {
+        // test with linear access:
+        for (int i = 0; i < N; i++) {
+            phases[i] = i;
+            indices[i] = i;
+            sines[i] = 0.0 / 0.0;
+            coses[i] = 0.0 / 0.0;
+        }
+        sincos_arr(sines, coses, phases, N);
+        fail |= check_arr(sines, coses, phases, NULL, N, 0, MAX_ULP_ERR, "sincos_arr");
+        sincos_arr_nested(sines, coses, phases, N, nests);
+        fail |= check_arr(sines, coses, phases, NULL, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_nested");
+
+        // test with nonlinear access:
+        for (int i = 0; i < N; i++) {
+            indices[i] = (i < 2) ? 1 : (indices[i - 2] + indices[i - 1]) % N;
+            sines[i] = 0.0 / 0.0;
+            coses[i] = 0.0 / 0.0;
+        }
+        sincos_arr_scrambled(sines, coses, phases, indices, N);
+        fail |= check_arr(sines, coses, phases, indices, N, 0, MAX_ULP_ERR, "sincos_arr_scrambled");
+        sincos_arr_scrambled_nested(sines, coses, phases, indices, N, nests);
+        fail |= check_arr(sines, coses, phases, indices, N, nests, MAX_ULP_ERR_NESTED, "sincos_arr_scrambled_nested");
+    }
+    if (fail) {
+        printf("FAILED!\n");
+    } else {
+        printf("Passed.\n");
+    }
+    return fail;
+}

SingleSource/UnitTests/Vectorizer/Veclib/sincos.reference_output

+Passed.
+exit 0

cmake/modules/DetectArchitecture.cmake

function(detect_x86_cpu_architecture variable)
   else()
     message(STATUS "Determine the system cpu architecture - failed")
     file(APPEND ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeError.log
       "Determining the system cpu architecture failed with the following output:\n${COMP_OUTPUT}\n${RUN_OUTPUT}")
     set(${variable} PARENT_SCOPE)
   endif()
 
 endfunction(detect_x86_cpu_architecture)
+
+#
+# Performs a try_run to check cpu feature flag of target
+#
+
+function(detect_x86_feature feature variable)
+  try_run(HAVE_RUN_CPU_FEATURE_${variable} HAVE_COMPILE_CPU_FEATURE_${variable}
+          ${CMAKE_BINARY_DIR}
+          ${CMAKE_SOURCE_DIR}/cmake/modules/DetectX86FeatureFlag.c
+          COMPILE_DEFINITIONS -DFEATURE_FLAG=${feature}
+          COMPILE_OUTPUT_VARIABLE COMP_OUTPUT
+          RUN_OUTPUT_VARIABLE RUN_OUTPUT)
+
+  if(HAVE_COMPILE_CPU_FEATURE_${variable} AND NOT (HAVE_RUN_CPU_FEATURE_${variable} STREQUAL  FAILED_TO_RUN))
+    if(RUN_OUTPUT MATCHES "^(YES|NO)$")
+      message(STATUS "Check target cpu feature ${feature}: ${RUN_OUTPUT}")
+      set(${variable} ${RUN_OUTPUT} PARENT_SCOPE)
+    else()
+      message(SEND_ERROR "Could not detect target cpu feature ${feature}!")
+      file(APPEND ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeError.log
+              "Determining the system cpu feature failed with the following output:\n${COMP_OUTPUT}\n${RUN_OUTPUT}")
+    endif()
+  else()
+    message(STATUS "Check target system for feature ${feature} - failed")
+    file(APPEND ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeError.log
+            "Determining the system cpu feature failed with the following output:\n${COMP_OUTPUT}\n${RUN_OUTPUT}")
+    set(${variable} PARENT_SCOPE)
+  endif()
+
+endfunction(detect_x86_feature)

cmake/modules/DetectLibmvecX86.c

+#define _GNU_SOURCE
+
+#include <bits/wordsize.h>
+
+typedef float __m128 __attribute__((__vector_size__(16), __aligned__(16)));
+
+#if __WORDSIZE==32
+void _ZGVbN4vvv_sincosf(__m128 x, __m128 sinptr, __m128 cosptr);
+#else
+void _ZGVbN4vvv_sincosf(__m128 x, __m128 sinptrlo, __m128 sinptrhi, __m128 cosptrlo, __m128 cosptrhi);
+#endif
+
+int main(void) {
+    union {__m128 vec; float arr[4];} x, sines, coses;
+    union {__m128 vec[2]; float* arr[4];} sinptr, cosptr;
+    for (int i=0; i<4; i++) {
+        x.arr[i] = 0.1f * i;
+        sinptr.arr[i] = &sines.arr[i];
+        cosptr.arr[i] = &coses.arr[i];
+    }
+    // use function from libmvec (the one we have a test for):
+#if __WORDSIZE==32
+    _ZGVbN4vvv_sincosf(x.vec, sinptr.vec[0], cosptr.vec[0]);
+#else
+    _ZGVbN4vvv_sincosf(x.vec, sinptr.vec[0], sinptr.vec[1], cosptr.vec[0], cosptr.vec[1]);
+#endif
+    // ensure the call cannot get optimized out:
+    int sin_is_less_than_cos = 1;
+    for (int i=0; i<4; i++) {
+        sin_is_less_than_cos &= (sines.arr[i] < coses.arr[i]);
+    }
+    return !sin_is_less_than_cos;
+}

cmake/modules/DetectVectorLibrary.cmake

+##===- DetectVectorLibrary.cmake ------------------------------------------===##
+#
+# Performs a try_run to determine if a vector library is available.
+#
+##===----------------------------------------------------------------------===##
+
+#
+# Performs a try_run to determine if libmvec-x86 is available
+#
+
+function(detect_libmvec_x86 variable)
+  set(CMAKE_C_FLAGS -msse)
+  try_run(HAVE_RUN_${variable} HAVE_COMPILE_${variable}
+    ${CMAKE_BINARY_DIR}
+    ${CMAKE_SOURCE_DIR}/cmake/modules/DetectLibmvecX86.c
+    LINK_LIBRARIES -lmvec)
+
+  if(HAVE_COMPILE_${variable} AND NOT (HAVE_RUN_${variable} STREQUAL  FAILED_TO_RUN))
+    set(HAVE_LIBMVEC_X86 YES)
+  else()
+    set(HAVE_LIBMVEC_X86 NO)
+  endif()
+  set(${variable} ${HAVE_LIBMVEC_X86} PARENT_SCOPE)
+  message(STATUS "Check libmvec X86 vector math library available: ${HAVE_LIBMVEC_X86}")
+
+endfunction(detect_libmvec_x86)
+
+# Add other vector libraries here...
 No newline at end of file

cmake/modules/DetectX86FeatureFlag.c

+#include <stdio.h>
+
+#define xstr(s) str(s)
+#define str(s) #s
+
+int main(void) {
+  __builtin_cpu_init();
+  if (__builtin_cpu_supports(xstr(FEATURE_FLAG))) {
+      printf("YES");
+  } else {
+      printf("NO");
+  }
+  return 0;
+}