[OpenMP][CMake] Pass --cuda-path to regression tests.

[Meinersbur]

Summary

The OpenMP runtime can be compiled using a CUDA installed at non-default location with the -DCUDA_TOOLKIT_ROOT_DIR setting. However, check-openmp will fail afterwards because Clang needs to know where to find the CUDA headers.

Fix by passing -cuda-path to Clang using the value of CUDA_TOOLKIT_ROOT_DIR which has been determined by CMake. Also set LD_LIBRARY_PATH such that it can find the cuda runtime when executing. This will ensure that the regression test do not depend on the current environment, but use the environment it was configured for.

See the current builder and the build with this patch applied.

[tianshilei1992]

openmp/libomptarget/deviceRTLs/nvptx/test/lit.site.cfg.in

8

This variable will not be set by CMake based on CMake document.

[Meinersbur]

• Define CUDA_LIBDIR for lit.site.cfg

[Meinersbur]

openmp/libomptarget/deviceRTLs/nvptx/test/lit.site.cfg.in

8

Thanks.

[tianshilei1992]

LGTM. Thanks!

[JonChesterfield]

The ld_library_path requirement is nasty. We should burn rpath/runpath into the cuda plugin (and probably also corresponding rpath/runpath into executables so they can find the openmp runtime libraries)

[protze.joachim]

On our site, the dependencies are not necessarily found in the same directory when moving from build system to compute system. So, for production use I actually prefer LD_LIBRARY_PATH & CO, which are managed by the environment-modules system.

[Meinersbur]

In D101266#2717667, @JonChesterfield wrote:

The ld_library_path requirement is nasty. We should burn rpath/runpath into the cuda plugin (and probably also corresponding rpath/runpath into executables so they can find the openmp runtime libraries)

What is so nasty about it? It is already sets a couple of other library paths. rpath unfortunately is less portable.

Does someone know why in one place it is prepend_library_path, but append_dynamic_library_path in the other? I think all libraries should be pre-pended, in order to take precedence over other system-installed libraries, such as in paths set by module load cuda/module load llvm.

[JonChesterfield]

In D101266#2718171, @Meinersbur wrote:

In D101266#2717667, @JonChesterfield wrote:

The ld_library_path requirement is nasty. We should burn rpath/runpath into the cuda plugin (and probably also corresponding rpath/runpath into executables so they can find the openmp runtime libraries)

What is so nasty about it? It is already sets a couple of other library paths. rpath unfortunately is less portable.

It leaks an internal detail to the user. Someone builds their application with openmp enabled, goes to run it, and it fails because it can't find the libraries that clang implicitly created dependencies on, even though clang knows where said dependencies are. Defaulting to the openmp that ships with clang removes that hurdle and still lets people override it with LD_LIBRARY_PATH if they wish (iirc it's RUNPATH that allows that override and rpath that doesn't).

As far as I know rpath works everywhere, but where it doesn't, those users can continue doing whatever they do today to find the libraries.

[Meinersbur]

In D101266#2718200, @JonChesterfield wrote:

It leaks an internal detail to the user.

It's only used during testing by llvm-lit whether the built was successful.

Someone builds their application with openmp enabled, goes to run it, and it fails because it can't find the libraries that clang implicitly created dependencies on, even though clang knows where said dependencies are.

Again, no consequence to the end-user of clang. To compile a program, the user will have to pass --cuda-path and/or the environment variables manually (or use module load).

Or is your concern that when check-openmp succeeds, the user might think that they do not have to point to a CUDA installation anymore?

Defaulting to the openmp that ships with clang

This is for the search path of libdevice.so and cudart.so, not libomp.so nor libomptarget.so.

[JonChesterfield]

Ah, I have confused this with needing to set LD_LIBRARY_PATH to find libomp.so when running an application, which may have been fixed somewhat recently. The tests using environment variables set by cmake to find cuda seems fine.

Needing to pass --cuda-path in the probably common case of a single cuda install in one of a few likely destinations is inconvenient but probably OK in practice. Iirc there's a rocm-path with similar properties, though I think that has some defaults it looks in when not specified.

[tianshilei1992]

In D101266#2718386, @JonChesterfield wrote:

Ah, I have confused this with needing to set LD_LIBRARY_PATH to find libomp.so when running an application, which may have been fixed somewhat recently. The tests using environment variables set by cmake to find cuda seems fine.

Actually LD_LIBRARY_PATH is indeed also set to find libomp.so and libomptarget.so in test. You can find it just a few lines above the code change in openmp/libomptarget/test/lit.cfg. However I don't think it's a problem, as long as they're prepended.

[Meinersbur]

• Rebase after D101315

openmp/libomptarget/deviceRTLs/nvptx/test/lit.cfg

config.test_flags = config.test_flags + \
         " -L " + config.omp_host_rtl_directory
 
 config.test_flags = config.test_flags + " " + config.test_extra_flags
 
 # Setup environment to find dynamic library at runtime.
 prepend_library_path('LIBRARY_PATH', config.library_dir, ":")
 prepend_library_path('LD_LIBRARY_PATH', config.library_dir, ":")
 prepend_library_path('LD_LIBRARY_PATH', config.omp_host_rtl_directory, ":")
+if config.cuda_libdir:
+    prepend_library_path('LD_LIBRARY_PATH', config.cuda_libdir, ":")
 
 # Forbid fallback to host.
 config.environment["OMP_TARGET_OFFLOAD"] = "MANDATORY"
 
 # substitutions
 config.substitutions.append(("%compilexx-run-and-check",
     "%compilexx-and-run | " + config.libomptarget_filecheck + " %s"))
 config.substitutions.append(("%compile-run-and-check",
     "%compile-and-run | " + config.libomptarget_filecheck + " %s"))
 config.substitutions.append(("%compilexx-and-run", "%compilexx && %run"))
 config.substitutions.append(("%compile-and-run", "%compile && %run"))
 
 config.substitutions.append(("%compilexx",
-    "%clangxx %openmp_flags %flags %s -o %t"))
+    "%clangxx %openmp_flags %cuda_flags %flags %s -o %t"))
 config.substitutions.append(("%compile",
-    "%clang %openmp_flags %flags %s -o %t"))
+    "%clang %openmp_flags %cuda_flags %flags %s -o %t"))
 
 config.substitutions.append(("%clangxx", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+if config.cuda_path:
+  config.substitutions.append(("%cuda_flags", "--cuda-path=" + config.cuda_path))
+else:
+  config.substitutions.append(("%cuda_flags", ""))
 config.substitutions.append(("%flags", config.test_flags))
 
 config.substitutions.append(("%run", "%t"))
 config.substitutions.append(("%not", config.libomptarget_not))

openmp/libomptarget/deviceRTLs/nvptx/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_openmp_flags = "@LIBOMPTARGET_NVPTX_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@LIBOMPTARGET_NVPTX_TEST_FLAGS@"
+config.cuda_path = "@CUDA_TOOLKIT_ROOT_DIR@"
+config.cuda_libdir = "@CUDA_LIBDIR@"
 config.binary_dir = "@CMAKE_CURRENT_BINARY_DIR@"
 config.library_dir = "@LIBOMPTARGET_LIBRARY_DIR@"
 config.omp_header_directory = "@LIBOMPTARGET_OPENMP_HEADER_FOLDER@"
 config.omp_host_rtl_directory = "@LIBOMPTARGET_OPENMP_HOST_RTL_FOLDER@"
 config.libomptarget_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
 config.libomptarget_not = "@OPENMP_NOT_EXECUTABLE@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

openmp/libomptarget/test/CMakeLists.txt

 string(REGEX MATCHALL "([^\ ]+\ |[^\ ]+$)" SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS}")
 foreach(CURRENT_TARGET IN LISTS SYSTEM_TARGETS)
   string(STRIP "${CURRENT_TARGET}" CURRENT_TARGET)
   add_openmp_testsuite(check-libomptarget-${CURRENT_TARGET} "Running libomptarget tests" ${CMAKE_CURRENT_BINARY_DIR}/${CURRENT_TARGET} DEPENDS omptarget omp ${LIBOMPTARGET_TESTED_PLUGINS})
   list(APPEND LIBOMPTARGET_LIT_TESTSUITES ${CMAKE_CURRENT_BINARY_DIR}/${CURRENT_TARGET})
 
   # Configure the lit.site.cfg.in file
   set(AUTO_GEN_COMMENT "## Autogenerated by libomptarget configuration.\n# Do not edit!")
+  get_filename_component(CUDA_LIBDIR "${CUDA_cudart_static_LIBRARY}" DIRECTORY)
   configure_file(lit.site.cfg.in ${CURRENT_TARGET}/lit.site.cfg @ONLY)
 endforeach()
 
 add_openmp_testsuite(check-libomptarget "Running libomptarget tests" ${LIBOMPTARGET_LIT_TESTSUITES} EXCLUDE_FROM_CHECK_ALL DEPENDS omptarget omp ${LIBOMPTARGET_TESTED_PLUGINS})
 No newline at end of file

openmp/libomptarget/test/lit.cfg

elif config.operating_system == 'Darwin':
     append_dynamic_library_path('DYLD_LIBRARY_PATH', \
         config.omp_host_rtl_directory, ";")
     config.test_flags += " -Wl,-rpath," + config.library_dir
     config.test_flags += " -Wl,-rpath," + config.omp_host_rtl_directory
 else: # Unices
     append_dynamic_library_path('LD_LIBRARY_PATH', config.library_dir, ":")
     append_dynamic_library_path('LD_LIBRARY_PATH', \
         config.omp_host_rtl_directory, ":")
+    if config.cuda_libdir:
+        append_dynamic_library_path('LD_LIBRARY_PATH', config.cuda_libdir, ":")
     append_dynamic_library_path('LIBRARY_PATH', config.library_dir, ":")
     append_dynamic_library_path('LIBRARY_PATH', \
         config.omp_host_rtl_directory, ":")
 
 # substitutions
 # - for targets that exist in the system create the actual command.
 # - for valid targets that do not exist in the system, return false, so that the
 #   same test can be used for different targets.
if libomptarget_target == config.libomptarget_current_target:
             "%clang-" + libomptarget_target + " %s -o %t"))
         config.substitutions.append(("%libomptarget-run-" + \
             libomptarget_target, \
             "%t"))
         config.substitutions.append(("%libomptarget-run-fail-" + \
             libomptarget_target, \
             "%not --crash %t"))
         config.substitutions.append(("%clangxx-" + libomptarget_target, \
-            "%clangxx %openmp_flags %flags -fopenmp-targets=" + libomptarget_target))
+            "%clangxx %openmp_flags %cuda_flags %flags -fopenmp-targets=" + libomptarget_target))
         config.substitutions.append(("%clang-" + libomptarget_target, \
-            "%clang %openmp_flags %flags -fopenmp-targets=" + libomptarget_target))
+            "%clang %openmp_flags %cuda_flags %flags -fopenmp-targets=" + libomptarget_target))
         config.substitutions.append(("%fcheck-" + libomptarget_target, \
             config.libomptarget_filecheck + " %s"))
     else:
         config.substitutions.append(("%libomptarget-compile-run-and-check-" + \
             libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%libomptarget-compilexx-run-and-check-" + \
             libomptarget_target, \
else:
         config.substitutions.append(("%clangxx-" + libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%fcheck-" + libomptarget_target, \
             "echo ignored-command"))
 
 config.substitutions.append(("%clangxx", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
+if config.cuda_path:
+  config.substitutions.append(("%cuda_flags", "--cuda-path=" + config.cuda_path))
+else:
+  config.substitutions.append(("%cuda_flags", ""))
 config.substitutions.append(("%flags", config.test_flags))
 config.substitutions.append(("%not", config.libomptarget_not))

openmp/libomptarget/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
 config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
+config.cuda_path = "@CUDA_TOOLKIT_ROOT_DIR@"
+config.cuda_libdir = "@CUDA_LIBDIR@"
 config.libomptarget_obj_root = "@CMAKE_CURRENT_BINARY_DIR@/@CURRENT_TARGET@"
 config.library_dir = "@LIBOMPTARGET_LIBRARY_DIR@"
 config.omp_header_directory = "@LIBOMPTARGET_OPENMP_HEADER_FOLDER@"
 config.omp_host_rtl_directory = "@LIBOMPTARGET_OPENMP_HOST_RTL_FOLDER@"
 config.operating_system = "@CMAKE_SYSTEM_NAME@"
 config.libomptarget_all_targets = "@LIBOMPTARGET_ALL_TARGETS@".split()
 config.libomptarget_current_target = "@CURRENT_TARGET@"
 config.libomptarget_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
 config.libomptarget_not = "@OPENMP_NOT_EXECUTABLE@"
 config.libomptarget_debug = @LIBOMPTARGET_DEBUG@
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")