[test-suite] Backprop kernel from Rodinia Benchmark

[proton]

Summary

Added Backprop kernel from Rodinia benchmark.
The kernel is modified such that Polly recognizes SCoPs in it.

Runtime:
With Polly: 0.40 sec
Without Polly: 0.81 sec

[Meinersbur]

How long does the benchmark run with/without Polly?

[Meinersbur]

Polly only optimizes part of the kernel. The loops containing exp calls are not. The swtich -ffast-math might help.

[proton]

Added -ffast-math flag... Polly recognizes entire kernel but still no performance improvement with Polly.

[MatzeB]

• Patches should be relative to the toplevel test-suite directory.
• 94k of reference output seems a lot and smells like you may end up in the anti-pattern of spending most of the benchmark time in printing instead of doing calculations.
• We currently tend to not add -ffast-math as part of the makefiles, but instead people choose it together with their other optimization flags.
• How long does this benchmark run? (We aim for 0.5-1s on typical PC hardware); Bonus points if the duration can be changed from the commandline (no expections for matching reference outputs for different input size though).
• srand()/rand() produce different output with different libc implementations!

[MatzeB]

backprop/backprop.c

13 ↗

(On Diff #152379)

This ABS implementation doesn't work correctly for negative zeros, use fabs() which is likely to be faster anyway.

17–28 ↗

(On Diff #152379)

Why not convert this to proper C99 so you can use the standardized restrict instead of the __restrict__ compiler extension. That would also allow you to declare variables on assignment instead of the noise early declarations.

backprop/main.c

153–158 ↗

(On Diff #152379)

Do you need to cast the malloc results to restrict? I would expect the compiler to figure that out on it's own.

[proton]

It runs for 1.1 sec for normal problem size and 0.6 sec for smaller
The main kernel function takes around 0.7-0.8 sec of 1.1 sec.
Polly detects scops here but runtime is still same

Changes:

• Removed -ffast-math flag by default
• Clearer array declaration
• Code Cleanup
• differential wrt latest repository
• Checksum of output compare instead of entire output compare
• removed use of restrict keyword
• added rand implemetation (glibc_compat_rand.c)
• Added Rodinia License

[proton]

The kernel runs for 0.79 sec and the complete program runs for 0.81 sec.

[MatzeB]

test-suite integration LGTM

MultiSource/Benchmarks/Rodinia/backprop/glibc_compat_rand.c

2–21

this looks odd, copy&paste with smart indentation enabled?

MultiSource/Benchmarks/Rodinia/backprop/glibc_compat_rand.h

3–9

this looks odd, copy&paste with smart indentation enabled?

[proton]

Changes:

• Use glibc_compat_rand.c from Common folder of Rodinia
• Use float * instead of the pointer to VLA in main.c
• Changed CMakeLists.txt

[Meinersbur]

LGTM

MultiSource/Benchmarks/Rodinia/backprop/backpropKernel.c

17–19

You could add static const restrict to those as well.

66

[Nit] parenthesis not necessary around array name

[proton]

made changes requested by @Meinersbur

LICENSE.TXT

                     llvm-test/SingleSource/Gizmos
                     llvm-test/SingleSource/Benchmarks/SmallPT
 zlib:               llvm-test/MultiSource/Applications/ClamAV/zlib_*
+Rodinia:            llvm-test/MultiSource/Benchmarks/Rodinia/LICENSE
+

MultiSource/Benchmarks/CMakeLists.txt

 add_subdirectory(nbench)
 add_subdirectory(sim)
 add_subdirectory(DOE-ProxyApps-C)
+add_subdirectory(Rodinia)
 
 if((NOT "${TARGET_OS}" STREQUAL "Darwin") OR (NOT "${ARCH}" STREQUAL "ARM"))
   add_subdirectory(TSVC)

MultiSource/Benchmarks/Rodinia/CMakeLists.txt

+add_subdirectory(backprop)

MultiSource/Benchmarks/Rodinia/LICENSE

+LICENSE TERMS
+
+Copyright (c)2008-2011 University of Virginia
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted without royalty fees or other restrictions, provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+    * Neither the name of the University of Virginia, the Dept. of Computer Science, nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY OF VIRGINIA OR THE SOFTWARE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+If you use this software or a modified version of it, please cite the most relevant among the following papers:
+
+- M. A. Goodrum, M. J. Trotter, A. Aksel, S. T. Acton, and K. Skadron. Parallelization of Particle Filter Algorithms. In Proceedings 
+of the 3rd Workshop on Emerging Applications and Many-core Architecture (EAMA), in conjunction with the IEEE/ACM International 
+Symposium on Computer Architecture (ISCA), June 2010.
+
+- S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, Sang-Ha Lee and K. Skadron.
+"Rodinia: A Benchmark Suite for Heterogeneous Computing". IEEE International Symposium
+on Workload Characterization, Oct 2009.
+
+- J. Meng and K. Skadron. "Performance Modeling and Automatic Ghost Zone Optimization
+for Iterative Stencil Loops on GPUs." In Proceedings of the 23rd Annual ACM International
+Conference on Supercomputing (ICS), June 2009.
+
+- L.G. Szafaryn, K. Skadron and J. Saucerman. "Experiences Accelerating MATLAB Systems
+Biology Applications." in Workshop on Biomedicine in Computing (BiC) at the International
+Symposium on Computer Architecture (ISCA), June 2009.
+
+- M. Boyer, D. Tarjan, S. T. Acton, and K. Skadron. "Accelerating Leukocyte Tracking using CUDA:
+A Case Study in Leveraging Manycore Coprocessors." In Proceedings of the International Parallel
+and Distributed Processing Symposium (IPDPS), May 2009.
+
+- S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, and K. Skadron. "A Performance
+Study of General Purpose Applications on Graphics Processors using CUDA" Journal of
+Parallel and Distributed Computing, Elsevier, June 2008.

MultiSource/Benchmarks/Rodinia/Makefile

+LEVEL = ../../..
+PARALLEL_DIRS  := backprop
+include $(LEVEL)/Makefile.programs

MultiSource/Benchmarks/Rodinia/backprop/CMakeLists.txt

+set(PROG backprop)
+list(APPEND LDFLAGS -lm)
+
+if(SMALL_PROBLEM_SIZE)
+  list(APPEND CFLAGS -DSMALL_DATASET=1)
+endif()
+
+set(HASH_PROGRAM_OUTPUT 1)
+
+set(WORKDIR ${CMAKE_CURRENT_BINARY_DIR})
+llvm_multisource()

MultiSource/Benchmarks/Rodinia/backprop/Makefile

+LEVEL = ../../../..
+LIBS += -lm
+LDFLAGS += -lm
+PROG     = backprop
+
+ifdef SMALL_PROBLEM_SIZE
+CPPFLAGS += -DSMALL_DATASET=1
+endif
+
+HASH_PROGRAM_OUTPUT = 1
+
+include ../../Makefile.multisrc

MultiSource/Benchmarks/Rodinia/backprop/backprop.h

+#ifndef _BACKPROP_H_
+#define _BACKPROP_H_
+
+#define ETA 0.3
+#define MOMENTUM 0.3
+
+#define LAYERSIZE 555365
+#define HIDDEN_SIZE 16
+#define OUTPUT_SIZE 1
+
+// GAP Between array index, which we want to compare
+// Otherwise very big reference output
+#define GAP 100
+
+#ifdef SMALL_DATASET
+#define ITERATIONS 1
+#else
+#define ITERATIONS 3
+#endif
+
+#define SEED 7
+
+#endif

MultiSource/Benchmarks/Rodinia/backprop/backprop.reference_output

+365b336d3ef6764dd9f769d0a0819453

MultiSource/Benchmarks/Rodinia/backprop/backpropKernel.c

+/*
+ ******************************************************************
+ * HISTORY
+ * 15-Oct-94  Jeff Shufelt (js), Carnegie Mellon University
+ * Prepared for 15-681, Fall 1994.
+ * Modified by Shuai Che
+ * 28-May-2018: Modified by Pankaj Kukreja,
+ * Indian Institute of Technology Hyderabad, India
+ ******************************************************************
+ */
+#include "backprop.h"
+#include <math.h>
+#define ABS(x) ((fabs(x) > 0.0) ? (x) : (-(x)))
+
+void bpnn_train_kernel(int in, int hid, int out, float *input_units,
+                       float *hidden_units, float *output_units,
+                       float *hidden_delta, float *output_delta, float *target,
+                       float *inputWeights, float *hiddenWeights,
+                       float *input_prevWeights, float *hidden_prevWeights,
+                       int iterations) {
+
+  typedef float layer1[1 + in][1 + hid];
+  typedef float layer2[1 + hid][1 + out];
+
+  layer1 *input_prev_weights = (layer1 *)input_prevWeights;
+  layer1 *input_weights = (layer1 *)inputWeights;
+  layer2 *hidden_prev_weights = (layer2 *)hidden_prevWeights;
+  layer2 *hidden_weights = (layer2 *)hiddenWeights;
+
+  for (int iteration = 0; iteration < iterations; iteration++) {
+    float sum;
+    int j, k;
+
+    input_units[0] = 1.0;
+    for (j = 1; j <= hid; j++) {
+      sum = 0.0;
+      for (k = 0; k <= in; k++) {
+        sum += (*input_weights)[k][j] * input_units[k];
+      }
+      hidden_units[j] = (1.0 / (1.0 + exp(-sum)));
+    }
+
+    hidden_units[0] = 1.0;
+    for (j = 1; j <= out; j++) {
+      sum = 0.0;
+      for (k = 0; k <= hid; k++) {
+        sum += (*hidden_weights)[k][j] * hidden_units[k];
+      }
+      output_units[j] = (1.0 / (1.0 + exp(-sum)));
+    }
+
+    float o, t, errsum;
+    errsum = 0.0;
+    for (j = 1; j <= out; j++) {
+      o = output_units[j];
+      t = target[j];
+      output_delta[j] = o * (1.0 - o) * (t - o);
+      errsum += ABS(output_delta[j]);
+    }
+
+    errsum = 0.0;
+    for (j = 1; j <= hid; j++) {
+      sum = 0.0;
+      for (k = 1; k <= out; k++) {
+        sum += output_delta[k] * (*hidden_weights)[j][k];
+      }
+      hidden_delta[j] = hidden_units[j] * (1.0 - hidden_units[j]) * sum;
+      errsum += ABS(hidden_delta[j]);
+    }
+
+    float new_dw;
+    hidden_units[0] = 1.0;
+    for (j = 1; j <= out; j++) {
+      for (k = 0; k <= hid; k++) {
+        new_dw = ((ETA * output_delta[j] * hidden_units[k]) +
+                  (MOMENTUM * (*hidden_prev_weights)[k][j]));
+        (*hidden_weights)[k][j] += new_dw;
+        (*hidden_prev_weights)[k][j] = new_dw;
+      }
+    }
+
+    input_units[0] = 1.0;
+    for (j = 1; j <= hid; j++) {
+      for (k = 0; k <= in; k++) {
+        new_dw = ((ETA * hidden_delta[j] * input_units[k]) +
+                  (MOMENTUM * (*input_prev_weights)[k][j]));
+        (*input_weights)[k][j] += new_dw;
+        (*input_prev_weights)[k][j] = new_dw;
+      }
+    }
+  }
+}

MultiSource/Benchmarks/Rodinia/backprop/glibc_compat_rand.h

+/*===------------- glibc_compat_rand.h- glibc rand emulation --------------===*\
+ * |*
+ * |*                     The LLVM Compiler Infrastructure
+ * |*
+ * |* This file is distributed under the University of Illinois Open Source
+ * |* License. See LICENSE.TXT for details.
+ * |*
+ * \*===----------------------------------------------------------------------===*/
+
+#ifndef GLIBC_COMPAT_RAND_H
+#define GLIBC_COMPAT_RAND_H
+
+int glibc_compat_rand(void);
+void glibc_compat_srand(unsigned int seed);
+
+#endif /* GLIBC_COMPAT_RAND_H */

MultiSource/Benchmarks/Rodinia/backprop/glibc_compat_rand.c

+/*===------------ glibc_compat_rand.c - glibc rand emulation --------------===*\
+ *  *
+ *   *                     The LLVM Compiler Infrastructure
+ *    *
+ *     * This file is distributed under the University of Illinois Open Source
+ *      * License. See LICENSE.TXT for details.
+ *       *
+ *       \*===----------------------------------------------------------------------===*/
+
+#include "glibc_compat_rand.h"
+
+/**
+ *  * This rand implementation is designed to emulate the implementation of
+ *   * rand/srand in recent versions of glibc. This is used for programs which
+ *    * require this specific rand implementation in order to pass verification
+ *     * tests.
+ *      *
+ *       * For more information, see: http://www.mathstat.dal.ca/~selinger/random/
+ *        **/
+
+#define TABLE_SIZE 344
+static unsigned int table[TABLE_SIZE];
+static int next;
+
+int glibc_compat_rand(void) {
+  /* Calculate the indices i-3 and i-31 in the circular vector. */
+  int i3 = (next < 3) ? (TABLE_SIZE + next - 3) : (next - 3);
+  int i31 = (next < 31) ? (TABLE_SIZE + next - 31) : (next - 31);
+
+  table[next] = table[i3] + table[i31];
+  unsigned int r = table[next] >> 1;
+
+  ++next;
+  if (next >= TABLE_SIZE)
+    next = 0;
+
+  return r;
+}
+
+void glibc_compat_srand(unsigned int seed) {
+  if (seed == 0)
+    seed = 1;
+
+  table[0] = seed;
+
+  for (int i = 1; i < 31; i++) {
+    int r = (16807ll * table[i - 1]) % 2147483647;
+    if (r < 0)
+      r += 2147483647;
+
+    table[i] = r;
+  }
+
+  for (int i = 31; i < 34; i++)
+    table[i] = table[i - 31];
+  for (int i = 34; i < TABLE_SIZE; i++)
+    table[i] = table[i - 31] + table[i - 3];
+
+  next = 0;
+}

MultiSource/Benchmarks/Rodinia/backprop/main.c

+#include "backprop.h"
+#include <glibc_compat_rand.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+void bpnn_train_kernel(int in, int hid, int out, float *input_units,
+                       float *hidden_units, float *output_units,
+                       float *hidden_delta, float *output_delta, float *target,
+                       float *input_weights, float *hidden_weights,
+                       float *input_prev_weights, float *hidden_prev_weights,
+                       int iterations);
+void bpnn_dump(float *input_weights, float *hidden_weights);
+
+////////////////////////////////////////////////////////////////////////////////
+// Program main
+////////////////////////////////////////////////////////////////////////////////
+int main() {
+  glibc_compat_srand(SEED);
+
+  float *input_weights =
+      (float *)malloc(sizeof(float) * (1 + LAYERSIZE) * (1 + HIDDEN_SIZE));
+  float *input_prev_weights =
+      (float *)malloc(sizeof(float) * (1 + LAYERSIZE) * (1 + HIDDEN_SIZE));
+  float *hidden_weights =
+      (float *)malloc(sizeof(float) * (1 + HIDDEN_SIZE) * (1 + OUTPUT_SIZE));
+  float *hidden_prev_weights =
+      (float *)malloc(sizeof(float) * (1 + HIDDEN_SIZE) * (1 + OUTPUT_SIZE));
+
+  float *input_units = (float *)malloc(((LAYERSIZE + 1) * sizeof(float)));
+  float *hidden_units = (float *)malloc(((HIDDEN_SIZE + 1) * sizeof(float)));
+  float *output_units = (float *)malloc(((OUTPUT_SIZE + 1) * sizeof(float)));
+  float *hidden_delta = (float *)malloc(((HIDDEN_SIZE + 1) * sizeof(float)));
+  float *output_delta = (float *)malloc(((OUTPUT_SIZE + 1) * sizeof(float)));
+  float *target = (float *)malloc(((OUTPUT_SIZE + 1) * sizeof(float)));
+
+  // Initializing input and hidden weights to random
+  int i, j;
+  for (i = 0; i <= LAYERSIZE; i++) {
+    for (j = 0; j <= HIDDEN_SIZE; j++) {
+      input_weights[i * (HIDDEN_SIZE + 1) + j] =
+          (float)glibc_compat_rand() / RAND_MAX;
+    }
+  }
+
+  for (i = 0; i <= HIDDEN_SIZE; i++) {
+    for (j = 0; j <= OUTPUT_SIZE; j++) {
+      hidden_weights[i * (1 + OUTPUT_SIZE) + j] =
+          (float)glibc_compat_rand() / RAND_MAX;
+    }
+  }
+
+  for (i = 0; i <= LAYERSIZE; i++) {
+    for (j = 0; j <= HIDDEN_SIZE; j++) {
+      input_prev_weights[i * (1 + HIDDEN_SIZE) + j] = 0.0;
+    }
+  }
+
+  for (i = 0; i <= HIDDEN_SIZE; i++) {
+    for (j = 0; j <= OUTPUT_SIZE; j++) {
+      hidden_prev_weights[i * (1 + OUTPUT_SIZE) + j] = 0.0;
+    }
+  }
+
+  for (i = 0; i <= OUTPUT_SIZE; i++) {
+    target[i] = 0.1;
+  }
+
+  // Random input
+  input_units[0] = 0;
+  for (int i = 0; i <= LAYERSIZE; i++) {
+    input_units[i] = (float)glibc_compat_rand() / RAND_MAX;
+  }
+
+  // entering the training kernel
+  bpnn_train_kernel(LAYERSIZE, HIDDEN_SIZE, OUTPUT_SIZE, input_units,
+                    hidden_units, output_units, hidden_delta, output_delta,
+                    target, input_weights, hidden_weights, input_prev_weights,
+                    hidden_prev_weights, ITERATIONS);
+
+  bpnn_dump(input_weights, hidden_weights);
+
+  free(input_weights);
+  free(hidden_weights);
+  free(input_prev_weights);
+  free(hidden_prev_weights);
+  free(input_units);
+  free(hidden_units);
+  free(output_units);
+  free(hidden_delta);
+  free(output_delta);
+  free(target);
+}
+
+void bpnn_dump(float *input_weights, float *hidden_weights) {
+  int i, j;
+
+  fflush(stdout);
+  for (i = 0; i <= LAYERSIZE; i++) {
+    for (j = 0; j <= HIDDEN_SIZE; j++) {
+      if ((i * (HIDDEN_SIZE + 1) + j) % GAP == 0) {
+        printf("%.6f\n", input_weights[i * (HIDDEN_SIZE + 1) + j]);
+      }
+    }
+  }
+
+  for (i = 0; i <= HIDDEN_SIZE; i++) {
+    for (j = 0; j <= OUTPUT_SIZE; j++) {
+      if ((i * (OUTPUT_SIZE + 1) + j) % GAP == 0) {
+        printf("%.6f\n", hidden_weights[i * (1 + OUTPUT_SIZE) + j]);
+      }
+    }
+  }
+  return;
+}