[OpenMP] Add scaffolding for negative runtime tests

[jdenny]

Summary

Without this patch, the openmp project's test suites do not appear to
have support for negative tests. However, D78170 needs to add a test
that an expected runtime failure occurs.

This patch makes not visible in all of the openmp project's test
suites. In all but libomptarget/test, it should be possible for a
test author to insert not before a use of the lit substitution for
running a test program. In libomptarget/test, that substitution is
target-specific, and its value is echo when the target is not
available. In that case, inserting not before a lit substitution
would expect an echo fail, so this patch instead defines a separate
lit substitution for expected runtime fails.

[jdoerfert]

LGTM.

[Hahnfeld]

This patch makes not visible in all of the openmp project's test
suites. In all but libomptarget/test, it should be possible for a
test author to insert not before a use of the lit substitution for
running a test program.

I see no config.substitutions for the other test suites, so I guess it will try to use not from PATH? IIRC LLVM and Clang add a substitution from not to the full path (via use_default_substitutions, but that's probably not available for standalone builds). Doing so is a bit tricky because "not" may be part of the checked strings. LLVM does some funny \| \bnot\b regex (which I have yet to understand), so I leave it up to you if you want to dig into that. Alternatively, we can either provide %not (which makes it unambiguous) or just not wire it up for the other suites and adapt the commit message.

[jdenny]

In D78566#1994835, @Hahnfeld wrote:

I see no config.substitutions for the other test suites, so I guess it will try to use not from PATH?

Ah, thanks for pointing that out. I wasn't being careful with those test suites because I'm not actually adding tests to them yet. No, not is not guaranteed to be in PATH.

IIRC LLVM and Clang add a substitution from not to the full path (via use_default_substitutions, but that's probably not available for standalone builds). Doing so is a bit tricky because "not" may be part of the checked strings.

What do you mean by "checked strings" here? Lit substitutions only happen on RUN lines, right?

LLVM does some funny \| \bnot\b regex (which I have yet to understand), so I leave it up to you if you want to dig into that.

I don't understand that either. That makes sense only when not stdin is a pipe. Grepping for RUN: not shows that's not always true. That's probably why not is also in PATH, but that makes the substitution seem redundant. When I remove the substitution (from lit/llvm/config.py), check-llvm behaves. I haven't yet run check-clang, etc.

Alternatively, we can either provide %not (which makes it unambiguous)

My only argument against that solution is that no other LLVM test suite does this. I grepped for RUN:.*%not to verify.

\bnot\b works in a quick example I just tried.

or just not wire it up for the other suites and adapt the commit message.

Maybe.

I just noticed another issue: ompt tests are skipped if FileCheck is not available. This is configured in openmp/runtime/test/lit.cfg. However, other openmp test suites that use FileCheck don't bother. Why is ompt different?

[Hahnfeld]

In D78566#1995119, @jdenny wrote:

I just noticed another issue: ompt tests are skipped if FileCheck is not available. This is configured in openmp/runtime/test/lit.cfg. However, other openmp test suites that use FileCheck don't bother. Why is ompt different?

Probably historic reasons, the OMPT tests were the first to use FileCheck. If other tests use it, we should probably make it required for all suites.

[jdenny]

Added the substitutions to actually make not available.

[jdenny]

In D78566#1995119, @jdenny wrote:

In D78566#1994835, @Hahnfeld wrote:

LLVM does some funny \| \bnot\b regex (which I have yet to understand), so I leave it up to you if you want to dig into that.

I don't understand that either. That makes sense only when not stdin is a pipe. Grepping for RUN: not shows that's not always true. That's probably why not is also in PATH, but that makes the substitution seem redundant. When I remove the substitution (from lit/llvm/config.py), check-llvm behaves. I haven't yet run check-clang, etc.

check-clang also behaves.

Alternatively, we can either provide %not (which makes it unambiguous)

My only argument against that solution is that no other LLVM test suite does this. I grepped for RUN:.*%not to verify.

\bnot\b works in a quick example I just tried.

or just not wire it up for the other suites and adapt the commit message.

Maybe.

Hopefully by providing the familiar not now, we'll encourage more negative tests. To that end, I went with the \bnot\b substitution.

I checked that it actually works this time by adding a RUN: not false in each openmp test suite. Of course, I didn't include that in the commit.

I'll wait a bit longer to push in case people have comments for this update.

[jdenny]

In D78566#1995220, @Hahnfeld wrote:

In D78566#1995119, @jdenny wrote:

I just noticed another issue: ompt tests are skipped if FileCheck is not available. This is configured in openmp/runtime/test/lit.cfg. However, other openmp test suites that use FileCheck don't bother. Why is ompt different?

Probably historic reasons, the OMPT tests were the first to use FileCheck.

Thanks.

If other tests use it, we should probably make it required for all suites.

Maybe in another patch.

[Hahnfeld]

LGTM as well

[jdenny]

@Hahnfeld @jdoerfert thanks!

openmp/README.rst

 **OPENMP_LLVM_LIT_EXECUTABLE** = ``/path/to/llvm-lit``
   Specify full path to ``llvm-lit`` executable for running tests.  The default
   is to search the ``PATH`` and the directory in **OPENMP_LLVM_TOOLS_DIR**.
 
 **OPENMP_FILECHECK_EXECUTABLE** = ``/path/to/FileCheck``
   Specify full path to ``FileCheck`` executable for running tests.  The default
   is to search the ``PATH`` and the directory in **OPENMP_LLVM_TOOLS_DIR**.
 
+**OPENMP_NOT_EXECUTABLE** = ``/path/to/not``
+  Specify full path to ``not`` executable for running tests.  The default
+  is to search the ``PATH`` and the directory in **OPENMP_LLVM_TOOLS_DIR**.
+
 Options for ``libomp``
 ----------------------
 
 **LIBOMP_ARCH** = ``aarch64|arm|i386|mic|mips|mips64|ppc64|ppc64le|x86_64|riscv64``
   The default value for this option is chosen based on probing the compiler for
   architecture macros (e.g., is ``__x86_64__`` predefined by compiler?).
 
 **LIBOMP_MIC_ARCH** = ``knc|knf``

openmp/cmake/OpenMPTesting.cmake

find_program(OPENMP_FILECHECK_EXECUTABLE
     PATHS ${OPENMP_LLVM_TOOLS_DIR})
   if (NOT OPENMP_FILECHECK_EXECUTABLE)
     message(STATUS "Cannot find FileCheck.")
     message(STATUS "Please put FileCheck in your PATH, set OPENMP_FILECHECK_EXECUTABLE to its full path, or point OPENMP_LLVM_TOOLS_DIR to its directory.")
     message(WARNING "The check targets will not be available!")
     set(ENABLE_CHECK_TARGETS FALSE PARENT_SCOPE)
     return()
   endif()
+
+  find_program(OPENMP_NOT_EXECUTABLE
+    NAMES not
+    PATHS ${OPENMP_LLVM_TOOLS_DIR})
+  if (NOT OPENMP_NOT_EXECUTABLE)
+    message(STATUS "Cannot find 'not'.")
+    message(STATUS "Please put 'not' in your PATH, set OPENMP_NOT_EXECUTABLE to its full path, or point OPENMP_LLVM_TOOLS_DIR to its directory.")
+    message(WARNING "The check targets will not be available!")
+    set(ENABLE_CHECK_TARGETS FALSE PARENT_SCOPE)
+    return()
+  endif()
 endfunction()
 
 if (${OPENMP_STANDALONE_BUILD})
   find_standalone_test_dependencies()
 
   # Make sure we can use the console pool for recent CMake and Ninja > 1.5.
   if (CMAKE_VERSION VERSION_LESS 3.1.20141117)
     set(cmake_3_2_USES_TERMINAL)
   else()
     set(cmake_3_2_USES_TERMINAL USES_TERMINAL)
   endif()
 
   # Set lit arguments.
   set(DEFAULT_LIT_ARGS "-sv --show-unsupported --show-xfail")
   if (MSVC OR XCODE)
     set(DEFAULT_LIT_ARGS "${DEFAULT_LIT_ARGS} --no-progress-bar")
   endif()
   set(OPENMP_LIT_ARGS "${DEFAULT_LIT_ARGS}" CACHE STRING "Options for lit.")
   separate_arguments(OPENMP_LIT_ARGS)
 else()
   set(OPENMP_FILECHECK_EXECUTABLE ${LLVM_RUNTIME_OUTPUT_INTDIR}/FileCheck)
+  set(OPENMP_NOT_EXECUTABLE ${LLVM_RUNTIME_OUTPUT_INTDIR}/not)
 endif()
 
 # Macro to extract information about compiler from file. (no own scope)
 macro(extract_test_compiler_information lang file)
   file(READ ${file} information)
   list(GET information 0 path)
   list(GET information 1 id)
   list(GET information 2 version)

openmp/libomptarget/deviceRTLs/nvptx/test/lit.cfg

config.substitutions.append(("%compile",
     "%clang %openmp_flags %flags %s -o %t"))
 
 config.substitutions.append(("%clangxx", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
 config.substitutions.append(("%flags", config.test_flags))
 
 config.substitutions.append(("%run", "%t"))
+config.substitutions.append((r"\bnot\b", config.libomptarget_not))

openmp/libomptarget/deviceRTLs/nvptx/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_openmp_flags = "@LIBOMPTARGET_NVPTX_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@LIBOMPTARGET_NVPTX_TEST_FLAGS@"
 config.binary_dir = "@CMAKE_CURRENT_BINARY_DIR@"
 config.library_dir = "@LIBOMPTARGET_LIBRARY_DIR@"
 config.omp_header_directory = "@LIBOMPTARGET_OPENMP_HEADER_FOLDER@"
 config.omp_host_rtl_directory = "@LIBOMPTARGET_OPENMP_HOST_RTL_FOLDER@"
 config.libomptarget_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.libomptarget_not = "@OPENMP_NOT_EXECUTABLE@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

openmp/libomptarget/test/lit.cfg

if libomptarget_target in config.libomptarget_system_targets:
             libomptarget_target))
         config.substitutions.append(("%libomptarget-compile-" + \
             libomptarget_target, \
             "%clang-" + libomptarget_target + " %s -o %t-" + \
             libomptarget_target))
         config.substitutions.append(("%libomptarget-run-" + \
             libomptarget_target, \
             "%t-" + libomptarget_target))
+        config.substitutions.append(("%libomptarget-run-fail-" + \
+            libomptarget_target, \
+            "not %t-" + libomptarget_target))
         config.substitutions.append(("%clangxx-" + libomptarget_target, \
             "%clangxx %openmp_flags %flags -fopenmp-targets=" + libomptarget_target))
         config.substitutions.append(("%clang-" + libomptarget_target, \
             "%clang %openmp_flags %flags -fopenmp-targets=" + libomptarget_target))
         config.substitutions.append(("%fcheck-" + libomptarget_target, \
             config.libomptarget_filecheck + " %s"))
     else:
         config.substitutions.append(("%libomptarget-compile-run-and-check-" + \
else:
             libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%libomptarget-compile-" + \
             libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%libomptarget-run-" + \
             libomptarget_target, \
             "echo ignored-command"))
+        config.substitutions.append(("%libomptarget-run-fail-" + \
+            libomptarget_target, \
+            "echo ignored-command"))
         config.substitutions.append(("%clang-" + libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%clangxx-" + libomptarget_target, \
             "echo ignored-command"))
         config.substitutions.append(("%fcheck-" + libomptarget_target, \
             "echo ignored-command"))
 
 config.substitutions.append(("%clangxx", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
 config.substitutions.append(("%flags", config.test_flags))
+config.substitutions.append((r"\bnot\b", config.libomptarget_not))

openmp/libomptarget/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
 config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
 config.libomptarget_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
 config.library_dir = "@LIBOMPTARGET_LIBRARY_DIR@"
 config.omp_header_directory = "@LIBOMPTARGET_OPENMP_HEADER_FOLDER@"
 config.omp_host_rtl_directory = "@LIBOMPTARGET_OPENMP_HOST_RTL_FOLDER@"
 config.operating_system = "@CMAKE_SYSTEM_NAME@"
 config.libomptarget_all_targets = "@LIBOMPTARGET_ALL_TARGETS@".split()
 config.libomptarget_system_targets = "@LIBOMPTARGET_SYSTEM_TARGETS@".split()
 config.libomptarget_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.libomptarget_not = "@OPENMP_NOT_EXECUTABLE@"
 config.libomptarget_debug = @LIBOMPTARGET_DEBUG@
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")

openmp/runtime/test/lit.cfg

 config.substitutions.append(("%libomp-compile", \
     "%clang %openmp_flags %flags %s -o %t" + libs))
 config.substitutions.append(("%libomp-run", "%t"))
 config.substitutions.append(("%clangXX", config.test_cxx_compiler))
 config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
 config.substitutions.append(("%flags", config.test_flags))
 config.substitutions.append(("%python", '"%s"' % (sys.executable)))
+config.substitutions.append((r"\bnot\b", config.test_not))
 
 if config.has_ompt:
     config.substitutions.append(("FileCheck", "tee %%t.out | %s" % config.test_filecheck))
     config.substitutions.append(("%sort-threads", "sort -n -s"))
     if config.operating_system == 'Windows':
         # No such environment variable on Windows.
         config.substitutions.append(("%preload-tool", "true ||"))
         config.substitutions.append(("%no-as-needed-flag", "-Wl,--no-as-needed"))
     elif config.operating_system == 'Darwin':
         config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
         # No such linker flag on Darwin.
         config.substitutions.append(("%no-as-needed-flag", ""))
     else:
         config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))
         config.substitutions.append(("%no-as-needed-flag", "-Wl,--no-as-needed"))

openmp/runtime/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
 config.test_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.test_not = "@OPENMP_NOT_EXECUTABLE@"
 config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
 config.libomp_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
 config.library_dir = "@LIBOMP_LIBRARY_DIR@"
 config.omp_header_directory = "@LIBOMP_BINARY_DIR@/src"
 config.operating_system = "@CMAKE_SYSTEM_NAME@"
 config.hwloc_library_dir = "@LIBOMP_HWLOC_LIBRARY_DIR@"
 config.using_hwloc = @LIBOMP_USE_HWLOC@
 config.has_ompt = @LIBOMP_OMPT_SUPPORT@ and @LIBOMP_OMPT_OPTIONAL@
 config.has_libm = @LIBOMP_HAVE_LIBM@
 config.has_libatomic = @LIBOMP_HAVE_LIBATOMIC@
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@LIBOMP_BASE_DIR@/test/lit.cfg")

openmp/tools/archer/tests/lit.cfg

 config.substitutions.append(("%clang-archer", config.test_c_compiler))
 config.substitutions.append(("%openmp_flags", config.test_openmp_flags))
 config.substitutions.append(("%archer_flags", config.archer_flags))
 config.substitutions.append(("%flags", config.test_flags))
 config.substitutions.append(("%suppression", "env TSAN_OPTIONS='ignore_noninstrumented_modules=1'"))
 config.substitutions.append(("%deflake", os.path.join(os.path.dirname(__file__), "deflake.bash")))
 
 config.substitutions.append(("FileCheck", config.test_filecheck))
+config.substitutions.append((r"\bnot\b", config.test_not))
 config.substitutions.append(("%sort-threads", "sort --numeric-sort --stable"))
 if config.operating_system == 'Windows':
     # No such environment variable on Windows.
     config.substitutions.append(("%preload-tool", "true ||"))
 elif config.operating_system == 'Darwin':
     config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
 else:
     config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))

openmp/tools/archer/tests/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
 
 config.test_c_compiler = "@OPENMP_TEST_C_COMPILER@"
 config.test_cxx_compiler = "@OPENMP_TEST_CXX_COMPILER@"
 config.test_compiler_features = @OPENMP_TEST_COMPILER_FEATURES@
 config.test_filecheck = "@OPENMP_FILECHECK_EXECUTABLE@"
+config.test_not = "@OPENMP_NOT_EXECUTABLE@"
 config.test_openmp_flags = "@OPENMP_TEST_OPENMP_FLAGS@"
 config.test_extra_flags = "@OPENMP_TEST_FLAGS@"
 config.libomp_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
 config.omp_library_dir = "@LIBOMP_LIBRARY_DIR@"
 config.omp_header_dir = "@LIBOMP_INCLUDE_DIR@"
 config.operating_system = "@CMAKE_SYSTEM_NAME@"
 config.has_libatomic = @LIBARCHER_HAVE_LIBATOMIC@
 config.has_tsan = @OPENMP_TEST_COMPILER_HAS_TSAN_FLAGS@
 
 config.test_archer_flags = "@LIBARCHER_TEST_FLAGS@"
 config.libarcher_obj_root = "@CMAKE_CURRENT_BINARY_DIR@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@CMAKE_CURRENT_SOURCE_DIR@/lit.cfg")