[SE] Let users specify CUDA path

[jhen]

Summary

Add logic to allow users to specify the CUDA path at configuration time.

[jlebar]

Seems simple enough when someone else writes it. :)

[jhen]

• Use CMake's standard FindCUDA

[jhen]

In D24580#543009, @jlebar wrote:

Seems simple enough when someone else writes it. :)

Speaking of which, it turns out we can use CMake's standard FindCUDA module to do everything except find the driver library. :) So I switched to doing things that way. It also allows you to control where it looks for CUDA by using the CUDA_TOOLKIT_ROOT_DIR variable.

[jlebar]

streamexecutor/lib/platforms/cuda/cmake/modules/FindLibcuda.cmake

4

Do you still want this?

[jprice]

Does this also need the CUDA library to be added to streamexecutor-config output for --libs?

At the moment (with this patch), if I try and link against SE using streamexecutor-config, I get undefined references to all of the CUDA driver functions. If I manually add -framework cuda (OS X), it all works. The CUDA SAXPY example works for me, but I can see from the verbose ninja output that the example is building with -framework cuda already.

[jhen]

• streamexecutor-config report CUDA lib

[jhen]

In D24580#543120, @jprice wrote:

Does this also need the CUDA library to be added to streamexecutor-config output for --libs?

At the moment (with this patch), if I try and link against SE using streamexecutor-config, I get undefined references to all of the CUDA driver functions. If I manually add -framework cuda (OS X), it all works. The CUDA SAXPY example works for me, but I can see from the verbose ninja output that the example is building with -framework cuda already.

Thanks for bringing this up! I added logic to streamexecutor-config to add in the CUDA library for the --libs output if CUDA is enabled. It seems to work on Ubuntu. I'd definitely be glad to know if it works on OS X as well.

streamexecutor/lib/platforms/cuda/cmake/modules/FindLibcuda.cmake

1

I think this was a phabricator bug. This whole file is gone from my standpoint (and I'm glad it's gone :)).

[jprice]

In D24580#543239, @jhen wrote:

Thanks for bringing this up! I added logic to streamexecutor-config to add in the CUDA library for the --libs output if CUDA is enabled. It seems to work on Ubuntu. I'd definitely be glad to know if it works on OS X as well.

Alas, this still doesn't work for OS X. It seems that CUDA_DRIVER_LIBRARY is set to the full path to the framework (e.g. /Library/Frameworks/cuda.framework) whereas the flag that needs to be passed is -framework cuda. CMake automagically sorts this out when passing the framework path to target_link_libraries, but I'm not sure there's a simple way of getting it to spit out the correct flag for streamexecutor-config in a platform independent manner.

This doesn't have to hold up this patch BTW, I just thought I'd mention it here since I noticed it.

[jhen]

In D24580#543280, @jprice wrote:

Alas, this still doesn't work for OS X. It seems that CUDA_DRIVER_LIBRARY is set to the full path to the framework (e.g. /Library/Frameworks/cuda.framework) whereas the flag that needs to be passed is -framework cuda. CMake automagically sorts this out when passing the framework path to target_link_libraries, but I'm not sure there's a simple way of getting it to spit out the correct flag for streamexecutor-config in a platform independent manner.

What do you think of the idea of doing a check in Python to see if the library base name is *.framework and then emitting the flag -lframework * instead of the raw path, if so? It's hacky, but it seems like it might work pretty much all the time.

This doesn't have to hold up this patch BTW, I just thought I'd mention it here since I noticed it.

I can't tell you how much I appreciate the work you are doing checking StreamExecutor on different platforms.

[jprice]

In D24580#543354, @jhen wrote:

What do you think of the idea of doing a check in Python to see if the library base name is *.framework and then emitting the flag -lframework * instead of the raw path, if so? It's hacky, but it seems like it might work pretty much all the time.

Sounds good to me. It sounds hacky, but I've just looked at the CMake source code for this and that's almost exactly what they do :-)

[jhen]

• Convert framework library names

[jhen]

In D24580#543554, @jprice wrote:

I've just looked at the CMake source code for this and that's almost exactly what they do :-)

I'll take a positive view of that and say that great minds hack alike. :)

The hack is in now. Please let me know if it works for you. Thanks!

[jprice]

In D24580#543785, @jhen wrote:

The hack is in now. Please let me know if it works for you. Thanks!

Great, it now works perfectly :-)

streamexecutor/CMakeLists.txt

 cmake_minimum_required(VERSION 3.1)
 
 option(STREAM_EXECUTOR_UNIT_TESTS "enable unit tests" ON)
 option(STREAM_EXECUTOR_ENABLE_DOXYGEN "enable StreamExecutor doxygen" ON)
-option(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL "enable building streamexecutor-config tool" ON)
-option(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM "enable building the CUDA StreamExecutor platform" OFF)
-
-configure_file("include/streamexecutor/PlatformOptions.h.in" "include/streamexecutor/PlatformOptions.h")
+option(
+    STREAM_EXECUTOR_ENABLE_CONFIG_TOOL
+    "enable building streamexecutor-config tool"
+    ON)
+option(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM
+    "enable building the CUDA StreamExecutor platform \
+(see CMake's 'FindCUDA' documentation for info on specifying the CUDA path)"
+    OFF)
+
+configure_file(
+    "include/streamexecutor/PlatformOptions.h.in"
+    "include/streamexecutor/PlatformOptions.h")
 
 # First find includes relative to the streamexecutor top-level source path.
 include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/include)
 # Also look for configured headers in the top-level binary directory.
 include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/include)
 
 # If we are not building as part of LLVM, build StreamExecutor as a standalone
 # project using LLVM as an external library:

 
 # Insist on C++ 11 features.
 set(CMAKE_CXX_STANDARD 11)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
 # Add warning flags.
 set(CMAKE_CXX_FLAGS  "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wno-unused-parameter")
 
+# Check for CUDA if it is enabled.
+if(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM)
+    find_package(CUDA REQUIRED)
+    include_directories(${CUDA_INCLUDE_DIRS})
+    find_library(CUDA_DRIVER_LIBRARY cuda)
+    if(NOT CUDA_DRIVER_LIBRARY)
+        message(FATAL_ERROR
+            "could not find libcuda, \
+is the CUDA driver is installed on your system?")
+    endif()
+    set(
+        STREAM_EXECUTOR_CUDA_PLATFORM_TARGET_OBJECT
+        $<TARGET_OBJECTS:streamexecutor_cuda_platform>)
+    set(
+        STREAM_EXECUTOR_LIBCUDA_LIBRARIES
+        ${CUDA_DRIVER_LIBRARY})
+endif(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM)
+
 add_subdirectory(lib)
 add_subdirectory(examples)
 
 if(STREAM_EXECUTOR_UNIT_TESTS)
     add_subdirectory(unittests)
 endif()
 
 if(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL )

streamexecutor/lib/CMakeLists.txt

 macro(add_se_library name)
   add_llvm_library(${name} ${ARGN})
   set_target_properties(${name} PROPERTIES FOLDER "streamexecutor libraries")
 endmacro(add_se_library)
 
-if(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM)
-    set(
-        CMAKE_MODULE_PATH
-        ${CMAKE_MODULE_PATH}
-        "${CMAKE_CURRENT_SOURCE_DIR}/platforms/cuda/cmake/modules/")
-
-    find_package(Libcuda REQUIRED)
-    include_directories(${LIBCUDA_INCLUDE_DIRS})
-
-    set(
-        STREAM_EXECUTOR_CUDA_PLATFORM_TARGET_OBJECT
-        $<TARGET_OBJECTS:streamexecutor_cuda_platform>)
-
-    set(
-        STREAM_EXECUTOR_LIBCUDA_LIBRARIES
-        ${LIBCUDA_LIBRARIES})
-endif(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM)
-
 add_subdirectory(platforms)
 
 add_se_library(
     streamexecutor
     Device.cpp
     DeviceMemory.cpp
     Error.cpp
     HostMemory.cpp

streamexecutor/lib/platforms/cuda/cmake/modules/FindLibcuda.cmake

-# - Try to find the libcuda library
-# Once done this will define
-#  LIBCUDA_FOUND - System has libcuda
-#  LIBCUDA_INCLUDE_DIRS - The libcuda include directories
-#  LIBCUDA_LIBRARIES - The libraries needed to use libcuda
-
-# TODO(jhen): Allow users to specify a search path.
-find_path(LIBCUDA_INCLUDE_DIR cuda.h /usr/local/cuda/include)
-# TODO(jhen): Use the library that goes with the headers.
-find_library(LIBCUDA_LIBRARY cuda)
-
-include(FindPackageHandleStandardArgs)
-# handle the QUIETLY and REQUIRED arguments and set LIBCUDA_FOUND to TRUE if
-# all listed variables are TRUE
-find_package_handle_standard_args(
-    LIBCUDA DEFAULT_MSG LIBCUDA_INCLUDE_DIR LIBCUDA_LIBRARY)
-
-mark_as_advanced(LIBCUDA_INCLUDE_DIR LIBCUDA_LIBRARY)
-
-set(LIBCUDA_LIBRARIES ${LIBCUDA_LIBRARY})
-set(LIBCUDA_INCLUDE_DIRS ${LIBCUDA_INCLUDE_DIR})

streamexecutor/tools/streamexecutor-config/streamexecutor-config.in

 
 import argparse
 import errno
 import os
 import shlex
 import subprocess
 import sys
 
-def get_llvm_config_dir():
-  """Gets the path to the llvm-config executable.
+# The following functions are configured by cmake. They use raw triple-quoted
+# strings to surround values that are substituted by cmake at configure time.
+# This kind of quoting should allow for paths that contain spaces.
 
-  Surrounding the cmake replacement with triple quotes should allow for paths
-  that contain quotes."""
+def get_llvm_config_dir():
+  """Gets the path to the llvm-config executable."""
   return r"""@LLVM_BINARY_DIR@/bin"""
 
 def get_cmake_install_prefix():
-  """Gets the value of the cmake variable CMAKE_INSTALL_PREFIX.
-
-  Surrounding the cmake replacement with triple quotes should allow for paths
-  that contain quotes."""
+  """Gets the value of the cmake variable CMAKE_INSTALL_PREFIX."""
   return r"""@CMAKE_INSTALL_PREFIX@"""
 
+def convert_library_name(library_name):
+  """Converts a library name ending in '.framework' into a '-framework' flag.
+
+  This is used to support OS X.
+
+  >>> convert_library_name('')
+  ''
+
+  >>> convert_library_name('/usr/local/lib64/libcuda.so')
+  '/usr/local/lib64/libcuda.so'
+
+  >>> convert_library_name('/Library/Frameworks/cuda.framework')
+  '-framework cuda'
+  """
+  framework_suffix = '.framework'
+  if library_name.endswith(framework_suffix):
+    framework_name = os.path.basename(library_name)[:-len(framework_suffix)]
+    library_name = '-framework ' + framework_name
+  return library_name
+
+def get_cuda_driver_library():
+  """Gets the value of the cmake variable CUDA_DRIVER_LIBRARY."""
+  return convert_library_name(r"""@CUDA_DRIVER_LIBRARY@""")
+
 def cuddle_flag(flag, tokens):
   """If flag appears by itself in tokens, combines it with the next token.
 
   >>> tokens = ['-I', '/usr/include']
   >>> cuddle_flag('-I', tokens)
   >>> tokens
   ['-I/usr/include']
 
if args.ldflags:
     if not has_token(token=se_flag, string=llvm_flags):
       all_flags.append(se_flag)
 
   if args.libs:
     llvm_flags = get_llvm_config_output('--libs support symbolize')
     se_flag = '-lstreamexecutor'
     if not has_token(token=se_flag, string=llvm_flags):
       all_flags.append(se_flag)
+    cuda_driver_library = get_cuda_driver_library()
+    if cuda_driver_library:
+      all_flags.append(cuda_driver_library)
     all_flags.append(llvm_flags)
 
   if args.system_libs:
     all_flags.append(get_llvm_config_output('--system-libs'))
 
   print(' '.join(all_flags))
 
 if __name__ == '__main__':
   main()