[OpenMP] [test] Fix intermittent omp_get_wtime.c test failures

[jlpeyton]

Summary

The omp_get_wtime.c test fails intermittently if the recorded times are off by too much which can happen when many tests are run in parallel.
Instead of failing if one timing is a little off, take average of 100 timings minus the 10 worst. Put the threshold at a generous 33%.

Brought to attention by Ron Lieberman (Thanks!)

[ronlieb]

i applied the patch , compiled and ran it 200+ invocations.
no fails, thanks for fixing this

[JonChesterfield]

Complicated, but robust wall clock timing was always going to be complicated. This looks reasonable and more robust that the current one.

Timing usually involves a fixed size integer that occasionally overflows so a little care is needed around end - start. I don't know how that works out if the return type is a double, maybe it abruptly goes back to zero, but it's hard to guess at what point. However wtime() is specified to return a floating point value so here we are.

[protze.joachim]

If this test still has issues after the latest patch changing to 100%, basically avg() < 2* expected, we could change to median() < 2*expected. This would effectively ignore the larger half of values.

[JonChesterfield]

We might also consider dropping this test, it's starting to feel quite a lot more complicated than the implementation

[Meinersbur]

Flang does a more basic test for the time function without risk of failing because of OS scheduling jitter: rG4498137bd7857c07921b4cd6313baac62ead24e2

[protze.joachim]

In my local test execution I just got:

error: average of 100 runs (0.029358) is of by 193.581486%

I think, changing the test as suggested by @Meinersbur sounds reasonable.

[ronlieb]

simpler approach like flang per MKruse does seem more attractive for this test. agree.

openmp/runtime/test/api/omp_get_wtime.c

 // RUN: %libomp-compile-and-run
 #include <stdio.h>
 #include <stdlib.h>
 #include "omp_testsuite.h"
 #include "omp_my_sleep.h"
 
-int test_omp_get_wtime()
-{
+#define NTIMES 100
+
+// This is the error % threshold. Be generous with the error threshold since
+// this test may be run in parallel with many other tests it may throw off the
+// sleep timing.
+#define THRESHOLD 33.0
+
+double test_omp_get_wtime(double desired_wait_time) {
   double start;
   double end;
-  double measured_time;
-  double wait_time = 0.2;
   start = 0;
   end = 0;
   start = omp_get_wtime();
-  my_sleep (wait_time);
+  my_sleep(desired_wait_time);
   end = omp_get_wtime();
-  measured_time = end-start;
-  return ((measured_time > 0.97 * wait_time) && (measured_time < 1.03 * wait_time)) ;
+  return end - start;
 }
 
-int main()
-{
-  int i;
-  int num_failed=0;
+int compare_times(const void *lhs, const void *rhs) {
+  const double *a = (const double *)lhs;
+  const double *b = (const double *)rhs;
+  return *a - *b;
+}
 
-  for(i = 0; i < REPETITIONS; i++) {
-    if(!test_omp_get_wtime()) {
-      num_failed++;
-    }
+int main() {
+  int i, final_count;
+  double percent_off;
+  double *begin, *end, *ptr;
+  double wait_time = 0.01;
+  double average = 0.0;
+  double n = 0.0;
+  double *times = (double *)malloc(sizeof(double) * NTIMES);
+
+  // Get each timing
+  for (i = 0; i < NTIMES; i++) {
+    times[i] = test_omp_get_wtime(wait_time);
+
+  // Remove approx the "worst" tenth of the timings
+  qsort(times, NTIMES, sizeof(double), compare_times);
+  begin = times;
+  end = times + NTIMES;
+  for (i = 0; i < NTIMES / 10; ++i) {
+    if (i % 2 == 0)
+      begin++;
+    else
+      end--;
+  }
+
+  // Get the average of the remaining timings
+  for (ptr = begin, final_count = 0; ptr != end; ++ptr, ++final_count)
+    average += times[i];
+  average /= (double)final_count;
+  free(times);
+
+  // Calculate the percent off of desired wait time
+  percent_off = (average - wait_time) / wait_time * 100.0;
+  // Should always be positive, but just in case
+  if (percent_off < 0)
+    percent_off = -percent_off;
+
+  if (percent_off > (double)THRESHOLD) {
+    fprintf(stderr, "error: average of %d runs (%lf) is of by %lf%%\n", NTIMES,
+            average, percent_off);
+    return EXIT_FAILURE;
   }
-  return num_failed;
+  printf("pass: average of %d runs (%lf) is only off by %lf%%\n", NTIMES,
+         average, percent_off);
+  return EXIT_SUCCESS;
 }