[SE] Let users specify CUDA path

[jhen]

Summary

Add logic to allow users to specify the CUDA path at configuration time.

[jlebar]

Seems simple enough when someone else writes it. :)

[jhen]

• Use CMake's standard FindCUDA

[jhen]

In D24580#543009, @jlebar wrote:

Seems simple enough when someone else writes it. :)

Speaking of which, it turns out we can use CMake's standard FindCUDA module to do everything except find the driver library. :) So I switched to doing things that way. It also allows you to control where it looks for CUDA by using the CUDA_TOOLKIT_ROOT_DIR variable.

[jlebar]

streamexecutor/lib/platforms/cuda/cmake/modules/FindLIBCUDA.cmake

9

Do you still want this?

[jprice]

Does this also need the CUDA library to be added to streamexecutor-config output for --libs?

At the moment (with this patch), if I try and link against SE using streamexecutor-config, I get undefined references to all of the CUDA driver functions. If I manually add -framework cuda (OS X), it all works. The CUDA SAXPY example works for me, but I can see from the verbose ninja output that the example is building with -framework cuda already.

[jhen]

• streamexecutor-config report CUDA lib

[jhen]

In D24580#543120, @jprice wrote:

Does this also need the CUDA library to be added to streamexecutor-config output for --libs?

At the moment (with this patch), if I try and link against SE using streamexecutor-config, I get undefined references to all of the CUDA driver functions. If I manually add -framework cuda (OS X), it all works. The CUDA SAXPY example works for me, but I can see from the verbose ninja output that the example is building with -framework cuda already.

Thanks for bringing this up! I added logic to streamexecutor-config to add in the CUDA library for the --libs output if CUDA is enabled. It seems to work on Ubuntu. I'd definitely be glad to know if it works on OS X as well.

streamexecutor/lib/platforms/cuda/cmake/modules/FindLibcuda.cmake

1

I think this was a phabricator bug. This whole file is gone from my standpoint (and I'm glad it's gone :)).

[jprice]

In D24580#543239, @jhen wrote:

Thanks for bringing this up! I added logic to streamexecutor-config to add in the CUDA library for the --libs output if CUDA is enabled. It seems to work on Ubuntu. I'd definitely be glad to know if it works on OS X as well.

Alas, this still doesn't work for OS X. It seems that CUDA_DRIVER_LIBRARY is set to the full path to the framework (e.g. /Library/Frameworks/cuda.framework) whereas the flag that needs to be passed is -framework cuda. CMake automagically sorts this out when passing the framework path to target_link_libraries, but I'm not sure there's a simple way of getting it to spit out the correct flag for streamexecutor-config in a platform independent manner.

This doesn't have to hold up this patch BTW, I just thought I'd mention it here since I noticed it.

[jhen]

In D24580#543280, @jprice wrote:

Alas, this still doesn't work for OS X. It seems that CUDA_DRIVER_LIBRARY is set to the full path to the framework (e.g. /Library/Frameworks/cuda.framework) whereas the flag that needs to be passed is -framework cuda. CMake automagically sorts this out when passing the framework path to target_link_libraries, but I'm not sure there's a simple way of getting it to spit out the correct flag for streamexecutor-config in a platform independent manner.

What do you think of the idea of doing a check in Python to see if the library base name is *.framework and then emitting the flag -lframework * instead of the raw path, if so? It's hacky, but it seems like it might work pretty much all the time.

This doesn't have to hold up this patch BTW, I just thought I'd mention it here since I noticed it.

I can't tell you how much I appreciate the work you are doing checking StreamExecutor on different platforms.

[jprice]

In D24580#543354, @jhen wrote:

What do you think of the idea of doing a check in Python to see if the library base name is *.framework and then emitting the flag -lframework * instead of the raw path, if so? It's hacky, but it seems like it might work pretty much all the time.

Sounds good to me. It sounds hacky, but I've just looked at the CMake source code for this and that's almost exactly what they do :-)

[jhen]

• Convert framework library names

[jhen]

In D24580#543554, @jprice wrote:

I've just looked at the CMake source code for this and that's almost exactly what they do :-)

I'll take a positive view of that and say that great minds hack alike. :)

The hack is in now. Please let me know if it works for you. Thanks!

[jprice]

In D24580#543785, @jhen wrote:

The hack is in now. Please let me know if it works for you. Thanks!

Great, it now works perfectly :-)

streamexecutor/CMakeLists.txt

 cmake_minimum_required(VERSION 3.1)
 
 option(STREAM_EXECUTOR_UNIT_TESTS "enable unit tests" ON)
 option(STREAM_EXECUTOR_ENABLE_DOXYGEN "enable StreamExecutor doxygen" ON)
-option(STREAM_EXECUTOR_ENABLE_CONFIG_TOOL "enable building streamexecutor-config tool" ON)
-option(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM "enable building the CUDA StreamExecutor platform" OFF)
-
-configure_file("include/streamexecutor/PlatformOptions.h.in" "include/streamexecutor/PlatformOptions.h")
+option(
+    STREAM_EXECUTOR_ENABLE_CONFIG_TOOL
+    "enable building streamexecutor-config tool"
+    ON)
+option(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM
+    "enable building the CUDA StreamExecutor platform \
+(use CUDA_ROOT to specify the CUDA path such as /usr/local/cuda)"
+    OFF)
+
+configure_file(
+    "include/streamexecutor/PlatformOptions.h.in"
+    "include/streamexecutor/PlatformOptions.h")
 
 # First find includes relative to the streamexecutor top-level source path.
 include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/include)
 # Also look for configured headers in the top-level binary directory.
 include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}/include)
 
 # If we are not building as part of LLVM, build StreamExecutor as a standalone
 # project using LLVM as an external library:

streamexecutor/lib/CMakeLists.txt

 macro(add_se_library name)
   add_llvm_library(${name} ${ARGN})
   set_target_properties(${name} PROPERTIES FOLDER "streamexecutor libraries")
 endmacro(add_se_library)
 
 if(STREAM_EXECUTOR_ENABLE_CUDA_PLATFORM)
     set(
         CMAKE_MODULE_PATH
         ${CMAKE_MODULE_PATH}
         "${CMAKE_CURRENT_SOURCE_DIR}/platforms/cuda/cmake/modules/")
 
-    find_package(Libcuda REQUIRED)
+    find_package(LIBCUDA REQUIRED)
     include_directories(${LIBCUDA_INCLUDE_DIRS})
 
     set(
         STREAM_EXECUTOR_CUDA_PLATFORM_TARGET_OBJECT
         $<TARGET_OBJECTS:streamexecutor_cuda_platform>)
 
     set(
         STREAM_EXECUTOR_LIBCUDA_LIBRARIES

streamexecutor/lib/platforms/cuda/cmake/modules/FindLIBCUDA.cmake

 # - Try to find the libcuda library
 # Once done this will define
 #  LIBCUDA_FOUND - System has libcuda
 #  LIBCUDA_INCLUDE_DIRS - The libcuda include directories
 #  LIBCUDA_LIBRARIES - The libraries needed to use libcuda
+#
+# Use the CUDA_ROOT environment variable or the CUDA_ROOT CMake variable to
+# specify the first directory to search for cuda.h.
+
+# First search based on the CUDA_ROOT cmake variable.
+find_path(
+    LIBCUDA_INCLUDE_DIR
+    cuda.h
+    PATHS
+    ${CUDA_ROOT}/include
+    NO_DEFAULT_PATH)
+
+# Then search based on the CUDA_ROOT environment variable.
+find_path(
+    LIBCUDA_INCLUDE_DIR
+    cuda.h
+    PATHS
+    ENV CUDA_ROOT
+    PATH_SUFFIXES include
+    NO_DEFAULT_PATH)
+
+# Finally, search based on a well-known location.
+find_path(
+    LIBCUDA_INCLUDE_DIR
+    cuda.h
+    PATHS
+    /usr/local/cuda/include
+    NO_DEFAULT_PATH)
 
-# TODO(jhen): Allow users to specify a search path.
-find_path(LIBCUDA_INCLUDE_DIR cuda.h /usr/local/cuda/include)
-# TODO(jhen): Use the library that goes with the headers.
 find_library(LIBCUDA_LIBRARY cuda)
 
 include(FindPackageHandleStandardArgs)
 # handle the QUIETLY and REQUIRED arguments and set LIBCUDA_FOUND to TRUE if
 # all listed variables are TRUE
 find_package_handle_standard_args(
     LIBCUDA DEFAULT_MSG LIBCUDA_INCLUDE_DIR LIBCUDA_LIBRARY)
 
 mark_as_advanced(LIBCUDA_INCLUDE_DIR LIBCUDA_LIBRARY)
 
 set(LIBCUDA_LIBRARIES ${LIBCUDA_LIBRARY})
 set(LIBCUDA_INCLUDE_DIRS ${LIBCUDA_INCLUDE_DIR})