[MicroBenchmarks,AArch64] Added correctness test & other performance tests for truncate or zero-extend vector operations

[nilanjana_basu]

Summary

This patch adds a correctness test to check the outcome of vectorized truncate or zero-extend operations in a loop for different vector types.
This patch also adds performance tests for combined operations of truncate/zero-extend and addition.

The goal of this benchmark is to check the impact of AArch64 specific changes in
D133495, D120571, D135229 and D136722.

[paquette]

Why randomized?

[nilanjana_basu]

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

[nilanjana_basu]

In D138059#3928627, @paquette wrote:

Why randomized?

You're right, it was unnecessary. Removed it in the latest patch & combined the correctness tests with the performance ones.

[fhahn]

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

[nilanjana_basu]

In D138059#3929083, @fhahn wrote:

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

In D138059#3929083, @fhahn wrote:

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

I checked that at the IR level the generated code had the relevant trunc & zext instructions, 4 times per function for interleave_count 4. I could also see a performance difference based on the related patches it is meant to test. To be on the safe side, what do you think about adding "benchmark::DoNotOptimize(A)" instead? Or should we prefer reverting to the old form?

[fhahn]

In D138059#3929188, @nilanjana_basu wrote:

I checked that at the IR level the generated code had the relevant trunc & zext instructions, 4 times per function for interleave_count 4. I could also see a performance difference based on the related patches it is meant to test. To be on the safe side, what do you think about adding "benchmark::DoNotOptimize(A)" instead? Or should we prefer reverting to the old form?

Yeah, it is probably fine now, but testing with a single value also seems to make the test less interesting. You could keep the random initialization and add a version of truncOrZextVecInLoopWithVW8 that disables vectorization to generate comparison data for testing.

[nilanjana_basu]

Reverted to using random inputs & changed correctness test to compare against same operations with no vectorization

[nilanjana_basu]

In D138059#3935023, @fhahn wrote:

Yeah, it is probably fine now, but testing with a single value also seems to make the test less interesting. You could keep the random initialization and add a version of truncOrZextVecInLoopWithVW8 that disables vectorization to generate comparison data for testing.

Yes, that sounds like a better correctness test. Have updated it.

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

22

Added noinline for ease of verifying the IR to check if this part is not optimized out. Can be removed if deemed unnecessary.

[fhahn]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

22

it might confuse readers, so its better to remove it I think

23

LLM coding style uses uppercase for variables

116

it would also be good to benchmark a version of the loop without any pragmas.

Also, why fix the VF?

136

It might be better to move this before the main loop so we fail early.

176–177

we also need versions with different dst types than i8. Same for the src types for zexts

[nilanjana_basu]

Addressed reviewer comments

[nilanjana_basu]

Variable name changes

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

116

Added a version of the benchmark with only vectorization enabled but no other pragmas. Changed the VF to test vectors of length 16 specifically, similar to the VF 8 case, assuming these two are commonly used VFs.

136

Done. Also added the correctness for each version of the benchmarks i.e. for different vectorization configurations. Each configuration should trigger different paths of the trunc/zext lowering & therefore, will be good to be tested for correctness.

[fhahn]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

128

Is this effectively the same code as benchForTruncOrZextVecInLoopWithVW8 and just calling a different benchmark function? If so, better to make the function an argument to avoid duplication?

[nilanjana_basu]

Removed duplicate code by adding function pointers as parameter as advised in the reviews. Added more performance tests using ZExt/Trunc operations in combination with addition operation.

[fhahn]

LGTM, thanks!

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

144

it might help with readability if there was a newline after each BENCHMARK...

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

+// This program tests vectorized truncates & zero-extends for performance and
+// correctness
 #include <iostream>
 #include <memory>
 #include <random>
 
 #include "benchmark/benchmark.h"
 
 #define ITERATIONS 10000
 
 static std::mt19937 rng;
 
 // Initialize array A with random numbers.
 template <typename Ty>
 static void init_data(const std::unique_ptr<Ty[]> &A, unsigned N) {
-  std::uniform_int_distribution<uint64_t> distrib(
-      std::numeric_limits<Ty>::min(), std::numeric_limits<Ty>::max());
-  for (unsigned i = 0; i < N; i++)
-    A[i] = static_cast<Ty>(distrib(rng));
+  std::uniform_int_distribution<Ty> distrib(std::numeric_limits<Ty>::min(),
+                                            std::numeric_limits<Ty>::max());
+  for (unsigned I = 0; I < N; I++)
+    A[I] = distrib(rng);
+}
+
+// Truncate/Zero-extend elements to create expected results with no
+// vectorization
+template <typename Ty1, typename Ty2>
+static void truncOrZextWithNoVec(const Ty1 *A, Ty2 *B, int Iterations) {
+#pragma clang loop vectorize(disable)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] = A[I];
+  }
 }
 
 // Truncate/Zero-extend each vector element in a vectorized loop with vectorization width 8
-template <typename Ty1, typename Ty2> static void truncOrZextVecInLoopWithVW8(const Ty1 *A, Ty2 *B, int iterations) {
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecInLoopWithVW8(const Ty1 *A, Ty2 *B, int Iterations) {
+#pragma clang loop vectorize_width(8) interleave_count(4)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] = A[I];
+  }
+}
+
+// Truncate/Zero-extend each vector element in a vectorized loop with
+// vectorization width 16
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecInLoopWithVW16(const Ty1 *A, Ty2 *B, int Iterations) {
+#pragma clang loop vectorize_width(16) interleave_count(4)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] = A[I];
+  }
+}
+
+// Truncate/Zero-extend each vector element in a vectorized loop
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecInLoop(const Ty1 *A, Ty2 *B, int Iterations) {
+#pragma clang loop vectorize(enable)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] = A[I];
+  }
+}
+
+// Truncate/Zero-extend each vector element while adding in a vectorized loop
+// with vectorization width 8
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecWithAddInLoopWithVW8(const Ty1 *A, Ty2 *B,
+                                               int Iterations) {
 #pragma clang loop vectorize_width(8) interleave_count(4)
-  for (unsigned i = 0; i < iterations; i++) {
-    B[i] = A[i];
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] += A[I];
+  }
+}
+
+// Truncate/Zero-extend each vector element while adding in a vectorized loop
+// vectorization width 16
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecWithAddInLoopWithVW16(const Ty1 *A, Ty2 *B,
+                                                int Iterations) {
+#pragma clang loop vectorize_width(16) interleave_count(4)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] += A[I];
+  }
+}
+
+// Truncate/Zero-extend each vector element while adding in a vectorized loop
+template <typename Ty1, typename Ty2>
+static void truncOrZextVecWithAddInLoop(const Ty1 *A, Ty2 *B, int Iterations) {
+#pragma clang loop vectorize(enable)
+  for (unsigned I = 0; I < Iterations; I++) {
+    B[I] += A[I];
   }
 }
 
-template <typename Ty1, typename Ty2> static void __attribute__((always_inline))
-benchForTruncOrZextVecInLoopWithVW8(benchmark::State &state) {
+template <typename Ty1, typename Ty2>
+static void __attribute__((always_inline))
+benchForTruncOrZextVecInLoop(benchmark::State &state,
+                             void (*Fn)(const Ty1 *, Ty2 *, int)) {
   std::unique_ptr<Ty1[]> A(new Ty1[ITERATIONS]);
   std::unique_ptr<Ty2[]> B(new Ty2[ITERATIONS]);
+  std::unique_ptr<Ty2[]> C(new Ty2[ITERATIONS]);
+
   init_data(A, ITERATIONS);
-  init_data(B, ITERATIONS);
-  for (auto _ : state) {
-    benchmark::DoNotOptimize(B);
-    benchmark::ClobberMemory();
-    truncOrZextVecInLoopWithVW8(&A[0], &B[0], ITERATIONS);
+
+  // Check for correctness
+  truncOrZextWithNoVec(&A[0], &C[0], ITERATIONS);
+  Fn(&A[0], &B[0], ITERATIONS);
+  for (int I = 0; I < ITERATIONS; I++) {
+    if (B[I] != C[I]) {
+      std::cerr << "ERROR: Trunc or ZExt operation on " << A[I]
+                << " is showing result " << B[I] << " instead of " << C[I]
+                << "\n";
+      exit(1);
     }
   }
 
-// Truncate/Zero-extend each vector element in a vectorized loop
-template <typename Ty1, typename Ty2> static void truncOrZextVecInLoop(const Ty1 *A, Ty2 *B, int iterations) {
-#pragma clang loop interleave_count(4)
-  for (unsigned i = 0; i < iterations; i++) {
-    B[i] = A[i];
+  for (auto _ : state) {
+    benchmark::DoNotOptimize(B);
+    benchmark::ClobberMemory();
+    Fn(&A[0], &B[0], ITERATIONS);
   }
 }
 
-template <typename Ty1, typename Ty2> static void __attribute__((always_inline))
-benchForTruncOrZextVecInLoop(benchmark::State &state) {
+template <typename Ty1, typename Ty2>
+static void __attribute__((always_inline))
+benchForTruncOrZextVecWithAddInLoop(benchmark::State &state,
+                                    void (*Fn)(const Ty1 *, Ty2 *, int)) {
   std::unique_ptr<Ty1[]> A(new Ty1[ITERATIONS]);
   std::unique_ptr<Ty2[]> B(new Ty2[ITERATIONS]);
+  std::unique_ptr<Ty2[]> C(new Ty2[ITERATIONS]);
   init_data(A, ITERATIONS);
   init_data(B, ITERATIONS);
   for (auto _ : state) {
     benchmark::DoNotOptimize(B);
     benchmark::ClobberMemory();
-    truncOrZextVecInLoop(&A[0], &B[0], ITERATIONS);
+    Fn(&A[0], &B[0], ITERATIONS);
   }
 }
 
 // Add vectorized truncate or zero-extend operation benchmarks for different element types
-#define ADD_BENCHMARK(ty1, ty2)                                                  \
-  void benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_(benchmark::State &state) {             \
-    benchForTruncOrZextVecInLoopWithVW8<ty1, ty2>(state);                   \
+#define ADD_BENCHMARK(ty1, ty2)                                                \
+  void benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_(            \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecInLoop<ty1, ty2>(state,                              \
+                                           &truncOrZextVecInLoopWithVW8);      \
   }                                                                            \
-  BENCHMARK(benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_); \
-  void benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_(benchmark::State &state) {             \
-    benchForTruncOrZextVecInLoop<ty1, ty2>(state);                   \
+  BENCHMARK(benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_);      \
+  void benchForTruncOrZextVecInLoopWithVW16From_##ty1##_To_##ty2##_(           \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecInLoop<ty1, ty2>(state,                              \
+                                           &truncOrZextVecInLoopWithVW16);     \
+  }                                                                            \
+  BENCHMARK(benchForTruncOrZextVecInLoopWithVW16From_##ty1##_To_##ty2##_);     \
+  void benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_(                   \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecInLoop<ty1, ty2>(state, &truncOrZextVecInLoop);      \
   }                                                                            \
-  BENCHMARK(benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_); \
+  BENCHMARK(benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_);             \
+  void benchForTruncOrZextVecWithAddInLoopWithVW8From_##ty1##_To_##ty2##_(     \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecWithAddInLoop<ty1, ty2>(                             \
+        state, &truncOrZextVecWithAddInLoopWithVW8);                           \
+  }                                                                            \
+  BENCHMARK(                                                                   \
+      benchForTruncOrZextVecWithAddInLoopWithVW8From_##ty1##_To_##ty2##_);     \
+  void benchForTruncOrZextVecWithAddInLoopWithVW16From_##ty1##_To_##ty2##_(    \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecWithAddInLoop<ty1, ty2>(                             \
+        state, &truncOrZextVecWithAddInLoopWithVW16);                          \
+  }                                                                            \
+  BENCHMARK(                                                                   \
+      benchForTruncOrZextVecWithAddInLoopWithVW16From_##ty1##_To_##ty2##_);    \
+  void benchForTruncOrZextVecWithAddInLoopFrom_##ty1##_To_##ty2##_(            \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecWithAddInLoop<ty1, ty2>(                             \
+        state, &truncOrZextVecWithAddInLoop);                                  \
+  }                                                                            \
+  BENCHMARK(benchForTruncOrZextVecWithAddInLoopFrom_##ty1##_To_##ty2##_);
 
 /* Vectorized truncate operations */
-ADD_BENCHMARK(uint64_t, uint8_t)
-ADD_BENCHMARK(uint32_t, uint8_t)
 ADD_BENCHMARK(uint16_t, uint8_t)
-
+ADD_BENCHMARK(uint32_t, uint8_t)
+ADD_BENCHMARK(uint64_t, uint8_t)
+ADD_BENCHMARK(uint32_t, uint16_t)
+ADD_BENCHMARK(uint64_t, uint16_t)
+ADD_BENCHMARK(uint64_t, uint32_t)
 
 /* Vectorized zero extend operations */
+ADD_BENCHMARK(uint8_t, uint16_t)
 ADD_BENCHMARK(uint8_t, uint32_t)
+ADD_BENCHMARK(uint8_t, uint64_t)
+ADD_BENCHMARK(uint16_t, uint32_t)
+ADD_BENCHMARK(uint16_t, uint64_t)
+ADD_BENCHMARK(uint32_t, uint64_t)