[test-suite] Added CMake files for external CUDA tests.

[tra]

Summary

Creates test executables and RUN-only test scripts for all combinations of:

• CUDA version: all versions found in externals/cuda
• C++ dialect: C++98,C++11
• C++ library: libc++, libstdc++ (for all gcc versions found in externals/cuda)

Supported tests:

• simple CUDA executables (included)
• thrust tests (enabled if if thrust sources are found in externals/cuda)

[MatzeB]

This should not affect users without users without cuda stuff installed so this LGTM.

I don't have any experience from the cuda side and do not plan to use this. So consider the rest as a suggestion I got while glancing over the code (but feel free to do whatever you want in the cuda parts).

External/CUDA/CMakeLists.txt

84–89

It's unusual to do stuff like setting CPPFLAGS in a function rather than toplevel. At least in the sense that it is inconsistent with the style in the rest of the test-suite.

115

if(LARGE_PROBLEM_SIZE) is probably enough.

133–160

This looks like magic to discover the host system. Maybe this would better fit in some global configuration variables combines with cmake/cache files for typical configurations? Esp. the hardcoding of the libstdc++ paths looks out of place here...

176–190

ditto.

203–223

ditto.

External/CUDA/README

5–7

Note that as a convenience you can even do without this flag: If you checkout your externals in test-suite/test-suite-externals it will get picked up automatically at cmake time.

[tra]

External/CUDA/CMakeLists.txt

84–89

Part of the problem is that cmake wants to have one configuration for *everything* while I want to build multiple variants of the same test without creating separate build directory for each, configure things there, etc. because I'd need *a lot* of them. My current test set involves 12 build variants. Once CUDA-8 is out, it 's likely to become 24.

Also, within each build variant I need to build more than one application, each of them potentially requiring its own set of compiler flags. Hence I create thrust app inside a function which gives each application its own set of flags within build variant.

Granted, I may be forcing CMake to do something it was not intended for. I'm new to CMake, so if you have suggestions how to do it better, I'd appreciate your input.

I guess I could move thrust into its own variant iteration loop and set compiler flags that are identical for all variants outside of the loop. It would not do much to change the fact that I'll have large number of targets that each needs its own set of compiler options and libraries to link with.

115

Fixed.

133–160

Not quite. I need to test with different libc++ variants provided in the externals directory. In order to do that I need to provide explicit path to libstdc++ shared library so executable is linked with appropriate -Wl,rpath and picks correct shared library when it's executed. Otherwise I end up loading whatever libstdc++ is installed on host or failing to run because it's not found.

Again, it's probably something I can move out of per-variant loop, but at the moment I can't think of a way to do just let CMake magically pick it only for specific subset of targets I'm creating.

176–190

Similar story here. Per CUDA version installed I need to link with specific cuda runtime library and provide specific include paths. Can't set it globally for everyone. I can move it out of the per-variant loop and do it once, though.

203–223

I need to find *specific* libc++/libstdc++ library to link with. CMake's standard "find me something" facilities are not flexible enough for what I need here.

External/CUDA/README

5–7

Good to know. I'll add this info.

[tra]

I've moved as much as I could from inner loop to outer ones.
Switched most of helper functions to macros as it makes it easier to update state variables in current scope.
Instead of creating variant build targets in separate function's scope,
pass custom options as parameters to llvm_multisource() (see D20221)

[tra]

Updated cmake scripts to use D20842.

Use llvm_test_executable(), retrieve target name it creates, and customize the way we compile/link it using standard cmake functions.

External/CMakeLists.txt

if(NOT ${PATHVAR} AND
     set(${PATHVAR} "${CMAKE_SOURCE_DIR}/test-suite-externals/${NAME}")
   endif()
   if(${PATHVAR})
     message(STATUS "Found ${DESCRIPTION}: ${${PATHVAR}}")
   endif()
 endmacro()
 
 llvm_add_subdirectories(
+  CUDA
   HMMER
   Nurbs
   Povray
   SPEC
   skidmarks10
 )

External/CUDA/CMakeLists.txt

+llvm_externals_find(TEST_SUITE_CUDA_ROOT "cuda" "CUDA prerequisites")
+
+# Helper macro to extract version number at the end of the string
+# Input: get_version(Var String)
+#    Where String = /some/string/with/version-x.y.z
+# Output:
+#    Sets Var=x.y.z
+macro(get_version Var Path)
+    string(REGEX REPLACE ".*-" "" Version ${Path})
+    set(${Var} ${Version})
+endmacro (get_version)
+
+# Helper function to glob CUDA source files and set LANGUAGE property
+# to CXX on each of them. Sets Var in parent scope to the list of
+# files found.
+macro(cuda_glob Var)
+  file(GLOB FileList ${ARGN})
+  foreach(File IN LISTS FileList)
+    if (${File} MATCHES ".*\.cu$")
+      set_source_files_properties(${File} PROPERTIES LANGUAGE CXX)
+    endif()
+  endforeach()
+  set(${Var} ${FileList})
+endmacro(cuda_glob)
+
+macro(create_one_local_test Name FileGlob)
+  cuda_glob(Source ${FileGlob})
+  llvm_test_run()
+  set(_Target ${Name}-${VariantSuffix})
+  llvm_multisource(PROG ${_Target} LIBS ${VariantLibs})
+  list(APPEND VARIANT_CUDA_TESTS ${_Target})
+endmacro()
+
+# Create targets for CUDA tests that are part of the test suite.
+macro(create_local_cuda_tests VariantSuffix)
+  create_one_local_test(axpy axpy.cu)
+  create_one_local_test(empty empty.cu)
+endmacro()
+
+macro(thrust_make_test_name TestName TestSourcePath)
+  string(REPLACE "${THRUST_PATH}/testing/" "" _tmp ${TestSourcePath})
+  string(REPLACE "/" "-" _tmp ${_tmp})
+  string(REPLACE "<" "_" _tmp ${_tmp})
+  string(REPLACE ">" "_" _tmp ${_tmp})
+  string(REGEX REPLACE "\.(cpp|cu)$" "" _tmp ${_tmp})
+  set(${TestName} ${_tmp})
+endmacro()
+
+macro(create_one_thrust_test TestSourcePath)
+  thrust_make_test_name(_TestName ${TestSourcePath})
+  set(Source ${TestSourcePath})
+  set(_Target thrust-${_TestName}-${VariantSuffix})
+  #set(executable_path ${CMAKE_CURRENT_BINARY_DIR}/${_Target})
+  llvm_test_run(--verbose ${_ExtraThrustTestArgs})
+  #llvm_add_test("${executable_path}-${TestName}.test" "${executable_path}")
+  llvm_multisource(PROG ${_Target} CFLAGS ${THRUST_CPPFLAGS} LIBS ${VariantLibs})
+  list(APPEND THRUST_VARIANT_TESTS ${_Target})
+endmacro()
+
+function(create_thrust_tests VariantSuffix)
+  set(_ThrustMainTarget thrust-${VariantSuffix})
+  if(LARGE_PROBLEM_SIZE)
+    set(_ExtraThrustTestArgs "--sizes=large")
+  endif()
+  if(DEFINED THRUST_SPLIT_TESTS AND "${THRUST_SPLIT_TESTS}")
+    # test framework is common for all tests, so we build it once as a
+    # library. Test suite's cmake files don't handle libraries well, so
+    # we need to set compile flags manually.
+    add_library(ThrustTestFrameworkLib-${VariantSuffix}
+      STATIC ${ThrustTestFramework})
+    append_compile_flags(ThrustTestFrameworkLib-${VariantSuffix} ${CPPFLAGS} ${THRUST_CPPFLAGS})
+    add_dependencies(ThrustTestFrameworkLib-${VariantSuffix} timeit-host fpcmp-host)
+    list(APPEND VariantLibs ThrustTestFrameworkLib-${VariantSuffix})
+
+    # Create individual test executable per test source file. This
+    # stresses cmake -- it consumes tons of memory and takes forever
+    # to finish.
+    foreach(TestSourcePath IN LISTS ThrustAllTestSources)
+      create_one_thrust_test(${TestSourcePath})
+    endforeach()
+    # Create target to build all thrust tests for this variant
+    add_custom_target(${_ThrustMainTarget} DEPENDS ${THRUST_VARIANT_TESTS}
+      COMMENT "Build CUDA test variant ${VariantSuffix}")
+  else()
+    # Single monolitic test executable. Alas this stresses linker
+    # during debug build. Final executable may end up being too large
+    # to link.
+    # We can create one test script per thrust test, but running
+    # thrust tests in parallel is bottlenecked by GPU and startup
+    # overhead, so it's actually way slower than running all tests
+    # sequentially.
+    set(Source ${ThrustTestFramework} ${ThrustAllTestSources})
+    llvm_test_run(--verbose ${_ExtraThrustTestArgs})
+    llvm_multisource(PROG ${_ThrustMainTarget} CPPFLAGS ${THRUST_CPPFLAGS} LIBS ${VariantLibs})
+  endif()
+  list(APPEND VARIANT_CUDA_TESTS ${_ThrustMainTarget})
+  set(VARIANT_CUDA_TESTS ${VARIANT_CUDA_TESTS} PARENT_SCOPE)
+endfunction()
+
+# Create set of tests for a given {CUDA,C++ standard,C++ library} tuple.
+# Sets VARIANT_CUDA_TESTS targets in parent's scope.
+function(create_cuda_test_variant VariantSuffix)
+  message(STATUS "Creating CUDA test variant ${VariantSuffix}")
+
+  set(VariantLibs ${_Cuda_Libs} ${_Stdlib_Libs})
+  list(APPEND CPPFLAGS ${_Cuda_CPPFLAGS} ${_Std_CPPFLAGS} ${_Stdlib_CPPFLAGS})
+  list(APPEND LDFLAGS ${_Cuda_LDFLAGS} ${_Std_LDFLAGS} ${_Stdlib_LDFLAGS})
+
+  create_local_cuda_tests(${VariantSuffix})
+
+  if(EXISTS ${THRUST_PATH})
+    create_thrust_tests(${VariantSuffix})
+  endif()
+
+  # Create target to build all tests for this variant
+  add_custom_target(cuda-tests-${VariantSuffix} DEPENDS ${VARIANT_CUDA_TESTS}
+    COMMENT "Build CUDA test variant ${VariantSuffix}")
+  # And pass it up to the caller
+  set(VARIANT_CUDA_TESTS cuda-tests-${VariantSuffix} PARENT_SCOPE)
+endfunction(create_cuda_test_variant)
+
+macro(create_cuda_tests)
+  message(STATUS "Checking CUDA prerequisites in ${TEST_SUITE_CUDA_ROOT}")
+  file(GLOB CudaVersions ${TEST_SUITE_CUDA_ROOT}/cuda-*)
+  foreach(CudaDir IN LISTS CudaVersions)
+    get_version(CudaVersion ${CudaDir})
+    message(STATUS "Found CUDA ${CudaVersion} ${CudaDir}")
+    list(APPEND CUDA_PATHS ${CudaDir})
+    add_library(cudart-${CudaVersion} SHARED IMPORTED)
+    set_property(TARGET cudart-${CudaVersion} PROPERTY IMPORTED_LOCATION
+      ${CudaDir}/lib64/libcudart.so)
+  endforeach(CudaDir)
+
+  if(NOT CUDA_PATHS)
+    message(SEND_ERROR
+      "There are no CUDA installations in ${TEST_SUITE_CUDA_ROOT}")
+    return()
+  endif()
+
+  # set default GPU arch
+  if(NOT CUDA_GPU_ARCH)
+    list(APPEND CUDA_GPU_ARCH sm_35)
+  endif()
+
+  foreach(GpuArch IN LISTS CUDA_GPU_ARCH)
+    list(APPEND CPPFLAGS --cuda-gpu-arch=${GpuArch})
+  endforeach()
+
+  file(GLOB GccVersions ${TEST_SUITE_CUDA_ROOT}/gcc-*)
+  foreach(GccDir IN LISTS GccVersions)
+    get_version(GccVersion ${GccDir})
+    message(STATUS "Found GCC ${GccVersion}")
+    list(APPEND GCC_PATHS ${GccDir})
+    add_library(libstdcxx-${GccVersion} SHARED IMPORTED)
+    set_property(TARGET libstdcxx-${GccVersion} PROPERTY IMPORTED_LOCATION
+      ${GccDir}/usr/local/lib64/libstdc++.so)
+  endforeach(GccDir)
+
+  # Find location of libc++
+  execute_process(
+    COMMAND ${CMAKE_CXX_COMPILER} -print-file-name=libc++.so
+    OUTPUT_VARIABLE _path_to_libcxx
+    OUTPUT_STRIP_TRAILING_WHITESPACE)
+
+  if (EXISTS ${_path_to_libcxx})
+    add_library(libcxx SHARED IMPORTED)
+    set_property(TARGET libcxx PROPERTY IMPORTED_LOCATION ${_path_to_libcxx})
+  else()
+    message(ERROR "Can't find libcxx location.")
+    return()
+  endif()
+
+  if(EXISTS "${TEST_SUITE_CUDA_ROOT}/thrust")
+    message(STATUS "Found Thrust ${THRUST_PATH}")
+    set(THRUST_PATH "${TEST_SUITE_CUDA_ROOT}/thrust" CACHE
+      PATH "Thrust library path")
+    set(THRUST_CPPFLAGS
+      -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP
+      -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA
+      -I${THRUST_PATH} -I${THRUST_PATH}/testing)
+
+    cuda_glob(ThrustTestCppSources ${THRUST_PATH}/testing/*.cpp)
+    cuda_glob(ThrustTestCudaSources ${THRUST_PATH}/testing/*.cu)
+    cuda_glob(ThrustTestCudaBackendSources
+      ${THRUST_PATH}/testing/backend/decompose.cu
+      ${THRUST_PATH}/testing/backend/cuda/*.cu)
+
+    list(APPEND ThrustAllTestSources ${ThrustTestCppSources}
+      ${ThrustTestCudaSources} ${ThrustTestCudaBackendSources})
+    list(APPEND ThrustTestFramework
+      ${THRUST_PATH}/testing/testframework.cpp
+      ${THRUST_PATH}/testing/reduce_large.cu
+      ${THRUST_PATH}/testing/unittest_tester.cu
+      ${THRUST_PATH}/testing/backend/cuda/testframework.cu)
+
+    # Remove test framework files from the list of test files.
+    foreach(File IN LISTS ThrustTestFramework)
+      list(REMOVE_ITEM ThrustAllTestSources ${File})
+    endforeach()
+  endif()
+
+  foreach(_CudaPath ${CUDA_PATHS})
+    get_version(_CudaVersion ${_CudaPath})
+    set(_Cuda_Suffix "cuda-${_CudaVersion}")
+    set(_Cuda_CPPFLAGS --cuda-path=${_CudaPath} -I${_CudaPath}/include)
+    set(_Cuda_Libs cudart-${_CudaVersion})
+    foreach(_Std IN ITEMS "c++98" "c++11")
+      set(_Std_Suffix "${_Std}")
+      set(_Std_CPPFLAGS -std=${_Std})
+      set(_Std_LDFLAGS -std=${_Std})
+      foreach(_GccPath IN LISTS GCC_PATHS)
+        get_version(_GccVersion ${_GccPath})
+        set(_Gcc_Suffix "libstdc++-${_GccVersion}")
+        # Tell clang to use libstdc++ and where to find it.
+        set(_Stdlib_CPPFLAGS -stdlib=libstdc++ -gcc-toolchain ${_GccPath}/usr/local)
+        set(_Stdlib_LDFLAGS  -stdlib=libstdc++)
+        # Add libstdc++ as link dependency.
+        set(_Stdlib_Libs libstdcxx-${_GccVersion})
+
+        set(VARIANT_CUDA_TESTS)
+        create_cuda_test_variant("${_Cuda_Suffix}-${_Std_Suffix}-${_Gcc_Suffix}")
+        list(APPEND ALL_CUDA_TESTS ${VARIANT_CUDA_TESTS})
+      endforeach()
+
+      # Same as above, but for libc++
+      # Tell clang to use libc++
+      # We also need to add compiler's include path for cxxabi.h
+      get_filename_component(_compiler_path ${CMAKE_CXX_COMPILER} DIRECTORY)
+      set(_Stdlib_CPPFLAGS -stdlib=libc++ -I${_compiler_path}/../include)
+      set(_Stdlib_LDFLAGS  -stdlib=libc++)
+      set(_Stdlib_Libs libcxx)
+      set(VARIANT_CUDA_TESTS)
+      create_cuda_test_variant("${_Cuda_Suffix}-${_Std_Suffix}-libc++")
+      list(APPEND ALL_CUDA_TESTS ${VARIANT_CUDA_TESTS})
+    endforeach()
+  endforeach()
+
+  # convenience target to build all CUDA tests.
+  add_custom_target(cuda-tests-all DEPENDS ${ALL_CUDA_TESTS}
+    COMMENT "Build all CUDA tests.")
+endmacro(create_cuda_tests)
+
+if(TEST_SUITE_CUDA_ROOT)
+  create_cuda_tests()
+endif()
+

External/CUDA/README

+CUDA Tests
+==========
+
+Cuda tests are enabled if cmake is invoked with
+-DTEST_SUITE_EXTERNALS_DIR=<externals path> and specified externals
+directory contains at least one CUDA installation.
+
+Expected externals directory structure:
+Externals/
+  cuda/
+    cuda-X.Y/  -- One or more CUDA installation.
+    gcc-X.Y.Z/ -- One or more GCC installation for libstdc++.
+
+export EXTERNALS=/your/externals/path
+export CUDA_EXTERNALS=$EXTERNALS/cuda
+export CLANG_DIR=/your/clang/build/dir
+export TEST_SUITE_DIR=/path/to/test-suite-sources
+
+* Externals installation
+  For each cuda version:
+    # bash cuda-linux64-rel-XXXXXX.run -prefix=$CUDA_EXTERNALS/cuda-7.5 -noprompt -nosymlink
+
+  For each GCC version:
+    * extract GCC sources and cd into gcc-X.Y.Z
+    # ../configure --enable-languages=c,c++ --disable-libsanitizer
+    # make bootstrap
+    # make DESTDIR=$CUDA_EXTERNALS/gcc-X.Y.Z install
+
+  Fetch thrust:
+    # cd $CUDA_EXTERNALS
+    # git clone https://github.com/thrust/thrust.git
+
+* Configure, build and run tests
+  Note that if externals are checked out into <test-suite>/test-suite-externals
+  there's no need to specify location explicitly with -DTEST_SUITE_EXTERNALS_DIR=
+
+  # mkdir test-suite-build-dir
+  # cd test-suite-build-dir
+  # PATH=$CLANG_DIR/bin:$PATH CXX=clang++ CC=clang \
+    cmake -G Ninja -DTEST_SUITE_EXTERNALS_DIR=$EXTERNALS $TEST_SUITE_DIR 
+  # ninja cuda-tests-all
+  # lit -j1 Externals/CUDA
+
+  For convenience there are top-level targets that would build only
+  one particular variant of the tests:
+
+    cuda-tests-<variant_suffix>
+
+  where <variant_suffix> is <cuda-version>-<c++ std>-<libc++|libstdc++-version>
+
+  Similarly individual test variants can be built using the same convention:
+
+    <test_name>-<variant suffix>
+
+  E.g: thrust-cuda-7.5-c++98-libstdc++-4.9.3 OR axpy-cuda-7.0-c++11-libc++

External/CUDA/axpy.cu

+#include <iostream>
+
+__global__ void axpy(float a, float* x, float* y) {
+  y[threadIdx.x] = a * x[threadIdx.x];
+}
+
+int main(int argc, char* argv[]) {
+  const int kDataLen = 4;
+
+  float a = 2.0f;
+  float host_x[kDataLen] = {1.0f, 2.0f, 3.0f, 4.0f};
+  float host_y[kDataLen];
+
+  // Copy input data to device.
+  float* device_x;
+  float* device_y;
+  cudaMalloc(&device_x, kDataLen * sizeof(float));
+  cudaMalloc(&device_y, kDataLen * sizeof(float));
+  cudaMemcpy(device_x, host_x, kDataLen * sizeof(float),
+             cudaMemcpyHostToDevice);
+
+  // Launch the kernel.
+  axpy<<<1, kDataLen>>>(a, device_x, device_y);
+
+  // Copy output data to host.
+  cudaDeviceSynchronize();
+  cudaMemcpy(host_y, device_y, kDataLen * sizeof(float),
+             cudaMemcpyDeviceToHost);
+
+  // Print the results.
+  for (int i = 0; i < kDataLen; ++i) {
+    std::cout << "y[" << i << "] = " << host_y[i] << "\n";
+  }
+
+  cudaDeviceReset();
+  return 0;
+}

External/CUDA/empty.cu

+int main(int argc, char **argv) { return 0; }

cmake/modules/TestFile.cmake

 # Create a .test driver file suitable for LIT.
 # The file contents come from previous llvm_test_XXX() calls.
 function(llvm_add_test testfile executable)
   # Replace $EXECUTABLE$ placeholder.
   string(REPLACE "$EXECUTABLE$" "${executable}" TESTSCRIPT "${TESTSCRIPT}")
 
   # Produce .test file
   file(GENERATE OUTPUT ${testfile} CONTENT "${TESTSCRIPT}")
+  # flush the test script
+  set(TESTSCRIPT "" PARENT_SCOPE)
 endfunction()