[MicroBenchmarks,AArch64] Added correctness test & other performance tests for truncate or zero-extend vector operations

[nilanjana_basu]

Summary

This patch adds a correctness test to check the outcome of vectorized truncate or zero-extend operations in a loop for different vector types.
This patch also adds performance tests for combined operations of truncate/zero-extend and addition.

The goal of this benchmark is to check the impact of AArch64 specific changes in
D133495, D120571, D135229 and D136722.

[paquette]

Why randomized?

[nilanjana_basu]

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

[nilanjana_basu]

In D138059#3928627, @paquette wrote:

Why randomized?

You're right, it was unnecessary. Removed it in the latest patch & combined the correctness tests with the performance ones.

[fhahn]

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

[nilanjana_basu]

In D138059#3929083, @fhahn wrote:

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

In D138059#3929083, @fhahn wrote:

In D138059#3928627, @paquette wrote:

Why randomized?

! In D138059#3929074, @nilanjana_basu wrote:

Removed randomization in input & combined correctness tests with performance ones. Explicitly added vectorization width for 16 elements since the related patches target this width.

I think the main reason for initializing with random data is to make the benchmarks more robust so the optimizer won't be able to (partly) optimize out our benchmark code?

I checked that at the IR level the generated code had the relevant trunc & zext instructions, 4 times per function for interleave_count 4. I could also see a performance difference based on the related patches it is meant to test. To be on the safe side, what do you think about adding "benchmark::DoNotOptimize(A)" instead? Or should we prefer reverting to the old form?

[fhahn]

In D138059#3929188, @nilanjana_basu wrote:

I checked that at the IR level the generated code had the relevant trunc & zext instructions, 4 times per function for interleave_count 4. I could also see a performance difference based on the related patches it is meant to test. To be on the safe side, what do you think about adding "benchmark::DoNotOptimize(A)" instead? Or should we prefer reverting to the old form?

Yeah, it is probably fine now, but testing with a single value also seems to make the test less interesting. You could keep the random initialization and add a version of truncOrZextVecInLoopWithVW8 that disables vectorization to generate comparison data for testing.

[nilanjana_basu]

Reverted to using random inputs & changed correctness test to compare against same operations with no vectorization

[nilanjana_basu]

In D138059#3935023, @fhahn wrote:

Yeah, it is probably fine now, but testing with a single value also seems to make the test less interesting. You could keep the random initialization and add a version of truncOrZextVecInLoopWithVW8 that disables vectorization to generate comparison data for testing.

Yes, that sounds like a better correctness test. Have updated it.

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

19

Added noinline for ease of verifying the IR to check if this part is not optimized out. Can be removed if deemed unnecessary.

[fhahn]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

19

it might confuse readers, so its better to remove it I think

20

LLM coding style uses uppercase for variables

49

it would also be good to benchmark a version of the loop without any pragmas.

Also, why fix the VF?

66

It might be better to move this before the main loop so we fail early.

117

we also need versions with different dst types than i8. Same for the src types for zexts

[nilanjana_basu]

Addressed reviewer comments

[nilanjana_basu]

Variable name changes

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

49

Added a version of the benchmark with only vectorization enabled but no other pragmas. Changed the VF to test vectors of length 16 specifically, similar to the VF 8 case, assuming these two are commonly used VFs.

66

Done. Also added the correctness for each version of the benchmarks i.e. for different vectorization configurations. Each configuration should trigger different paths of the trunc/zext lowering & therefore, will be good to be tested for correctness.

[fhahn]

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

58

Is this effectively the same code as benchForTruncOrZextVecInLoopWithVW8 and just calling a different benchmark function? If so, better to make the function an argument to avoid duplication?

[nilanjana_basu]

Removed duplicate code by adding function pointers as parameter as advised in the reviews. Added more performance tests using ZExt/Trunc operations in combination with addition operation.

[fhahn]

LGTM, thanks!

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

110

it might help with readability if there was a newline after each BENCHMARK...

MicroBenchmarks/LoopVectorization/VectorOperations.cpp

 #include <iostream>
 #include <memory>
 #include <random>
 
 #include "benchmark/benchmark.h"
 
 #define ITERATIONS 10000
 
 static std::mt19937 rng;
 
-// Initialize array A with random numbers.
+// Initialize array A with random numbers or the maximum for that type
 template <typename Ty>
-static void init_data(const std::unique_ptr<Ty[]> &A, unsigned N) {
-  std::uniform_int_distribution<uint64_t> distrib(
-      std::numeric_limits<Ty>::min(), std::numeric_limits<Ty>::max());
+static void init_data(const std::unique_ptr<Ty[]> &A, unsigned N,
+                      bool Randomize = true) {
+  if (Randomize) {
+    std::uniform_int_distribution<Ty> distrib(std::numeric_limits<Ty>::min(),
+                                              std::numeric_limits<Ty>::max());
     for (unsigned i = 0; i < N; i++)
-    A[i] = static_cast<Ty>(distrib(rng));
+      A[i] = distrib(rng);
+  } else {
+    for (unsigned i = 0; i < N; i++)
+      A[i] = std::numeric_limits<Ty>::max();
+  }
 }
 
 // Truncate/Zero-extend each vector element in a vectorized loop with vectorization width 8
 template <typename Ty1, typename Ty2> static void truncOrZextVecInLoopWithVW8(const Ty1 *A, Ty2 *B, int iterations) {
 #pragma clang loop vectorize_width(8) interleave_count(4)
   for (unsigned i = 0; i < iterations; i++) {
     B[i] = A[i];
   }
 }
 
 template <typename Ty1, typename Ty2> static void __attribute__((always_inline))
 benchForTruncOrZextVecInLoopWithVW8(benchmark::State &state) {
   std::unique_ptr<Ty1[]> A(new Ty1[ITERATIONS]);
   std::unique_ptr<Ty2[]> B(new Ty2[ITERATIONS]);
   init_data(A, ITERATIONS);
-  init_data(B, ITERATIONS);
   for (auto _ : state) {
     benchmark::DoNotOptimize(B);
     benchmark::ClobberMemory();
     truncOrZextVecInLoopWithVW8(&A[0], &B[0], ITERATIONS);
   }
 }
 
 // Truncate/Zero-extend each vector element in a vectorized loop
 template <typename Ty1, typename Ty2> static void truncOrZextVecInLoop(const Ty1 *A, Ty2 *B, int iterations) {
 #pragma clang loop interleave_count(4)
   for (unsigned i = 0; i < iterations; i++) {
     B[i] = A[i];
   }
 }
 
 template <typename Ty1, typename Ty2> static void __attribute__((always_inline))
 benchForTruncOrZextVecInLoop(benchmark::State &state) {
   std::unique_ptr<Ty1[]> A(new Ty1[ITERATIONS]);
   std::unique_ptr<Ty2[]> B(new Ty2[ITERATIONS]);
   init_data(A, ITERATIONS);
-  init_data(B, ITERATIONS);
   for (auto _ : state) {
     benchmark::DoNotOptimize(B);
     benchmark::ClobberMemory();
     truncOrZextVecInLoop(&A[0], &B[0], ITERATIONS);
   }
 }
 
+// Check correctness of Truncate/Zero-extend operation of each vector element in
+// a vectorized loop
+template <typename Ty1, typename Ty2>
+static void __attribute__((always_inline))
+correctnessTestForTruncOrZextVecInLoop(benchmark::State &state) {
+  std::unique_ptr<Ty1[]> A(new Ty1[ITERATIONS]);
+  std::unique_ptr<Ty2[]> B(new Ty2[ITERATIONS]);
+  // Create source vector elements with maximum value for the type
+  init_data(A, ITERATIONS, false);
+  for (auto _ : state) {
+    benchmark::DoNotOptimize(B);
+    benchmark::ClobberMemory();
+    truncOrZextVecInLoop(&A[0], &B[0], ITERATIONS);
+  }
+
+  // Create expected result of the Trunc or ZExt operation
+  Ty2 C;
+  if (sizeof(Ty1) > sizeof(Ty2)) {
+    C = std::numeric_limits<Ty2>::max();
+  } else {
+    C = std::numeric_limits<Ty1>::max();
+  }
+
+  for (int I = 0; I < ITERATIONS; I++) {
+    if (B[I] != C) {
+      std::cerr << "ERROR: Trunc or ZExt operation on " << A[I]
+                << " is showing result " << B[I] << " instead of " << C << "\n";
+      exit(1);
+    }
+  }
+}
+
 // Add vectorized truncate or zero-extend operation benchmarks for different element types
-#define ADD_BENCHMARK(ty1, ty2)                                                  \
-  void benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_(benchmark::State &state) {             \
-    benchForTruncOrZextVecInLoopWithVW8<ty1, ty2>(state);                   \
+#define ADD_BENCHMARK(ty1, ty2)                                                \
+  void benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_(            \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecInLoopWithVW8<ty1, ty2>(state);                      \
   }                                                                            \
-  BENCHMARK(benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_); \
-  void benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_(benchmark::State &state) {             \
-    benchForTruncOrZextVecInLoop<ty1, ty2>(state);                   \
+  BENCHMARK(benchForTruncOrZextVecInLoopWithVW8From_##ty1##_To_##ty2##_);      \
+  void benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_(                   \
+      benchmark::State &state) {                                               \
+    benchForTruncOrZextVecInLoop<ty1, ty2>(state);                             \
   }                                                                            \
-  BENCHMARK(benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_); \
+  BENCHMARK(benchForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_);             \
+  void correctnessTestForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_(         \
+      benchmark::State &state) {                                               \
+    correctnessTestForTruncOrZextVecInLoop<ty1, ty2>(state);                   \
+  }                                                                            \
+  BENCHMARK(correctnessTestForTruncOrZextVecInLoopFrom_##ty1##_To_##ty2##_);
 
 /* Vectorized truncate operations */
 ADD_BENCHMARK(uint64_t, uint8_t)
 ADD_BENCHMARK(uint32_t, uint8_t)
 ADD_BENCHMARK(uint16_t, uint8_t)
 
 
 /* Vectorized zero extend operations */
+ADD_BENCHMARK(uint8_t, uint16_t)
 ADD_BENCHMARK(uint8_t, uint32_t)
+ADD_BENCHMARK(uint8_t, uint64_t)