[libc++] Add from-scratch testing configs for Windows

[mstorsjo]

Summary

The paths to the compiler and to the python executable may need to
be quoted (if they're installed into e.g. "C:\Program Files").

All testing commands that are executed expect a gcc compatible command
line interface, while clang-cl uses different command line options.
In the original testing config, if the chosen compiler was clang-cl, it
was replaced with clang++ by looking for such an executable in the path.

For the new from-scratch test configs, I instead chose to add
"--driver-mode=g++" to flags - invoking "clang-cl --driver-mode=g++"
has the same effect as invoking "clang++", without needing to run any
heuristics for picking a different compiler executable.

This requires including %{{flags}} in the _supportsVerify function,
as running plain %{{cxx} on its own won't work that way. Alternatively,
the --driver-mode option could be appended on the %{cxx} substitution
(but it's not straightforward to change substitutions that already are
defined).

[ldionne]

Rebase onto main

[ldionne]

Split the GCC part off.

[ldionne]

@mstorsjo Could you please commandeer this per our discussion on Discord?

[mstorsjo]

Sure. I'll pick this up and try to complete it a bit later.

[mstorsjo]

Updated to work for both the clang-cl and mingw configurations. This goes on top of D11539.

[ldionne]

libcxx/test/configs/llvm-libc++-mingw.cfg.in

15

There's a weird character here.

libcxx/utils/libcxx/test/format.py

24 ↗

(On Diff #392934)

If we do this, I think it should then be %{{flags}} %{{compile_flags}}.

libcxx/utils/libcxx/test/newconfig.py

14–17 ↗

(On Diff #392934)

I suggest we simply move that to lit.TestRunner. It's too general purpose to be included in newconfig.py, even though I know that is convenient.

50 ↗

(On Diff #392934)

If this is necessary, IMO this should be part of lit_config.note() itself.

[mstorsjo]

libcxx/test/configs/llvm-libc++-mingw.cfg.in

15

Oops, will fix.

libcxx/utils/libcxx/test/format.py

24 ↗

(On Diff #392934)

Yes, that's probably most consistent, and that substitution shouldn't have anything that we can't use here.

libcxx/utils/libcxx/test/newconfig.py

14–17 ↗

(On Diff #392934)

Ok, I'll see about that.

50 ↗

(On Diff #392934)

Ok, I'll try that too.

[mstorsjo]

Updated for changes to %{executor} in the previous patch, removing the need for using quote() in the individual configs. Moved the quote function to lit.TestRunner and flushing into lit.LitConfig._write_message(). Added %{{compile_flags}} in _supportsVerify for consistency. Got rid of the unexpected unicode NBSP.

The lit changes probably should be split out and reviewed separately, but keeping them in this review for one round at least, to simplify running the CI.

[mstorsjo]

Fix the final CI config

[ldionne]

Adding @yln for the Lit part.

[mstorsjo]

In D111202#3183868, @ldionne wrote:

Adding @yln for the Lit part.

I can of course split out the Lit parts into separate patches, but the quote() function on its own is a bit tricky to submit without the use case attached.

[mstorsjo]

Rebased on top of the split out changes. Now the remaining bits here should be pretty much straightforward.

I opted to rename the config files from "llvm-libc++-static-clangcl.cfg"
to "llvm-libc++-clangcl-static.cfg", where the latter feels like a more
logical hierarchical name to me.

[ldionne]

Regarding naming, we already have llvm-libc++-shared-gcc.cfg.in, so please either update that one in a separate patch too, or keep the compiler last for consistency. Either way works for me.

Thanks a lot for splitting off the other changes, I think we've managed to improve the infrastructure while adding support for this platform, which is the way it should be done. Thanks for your work on this.

LGTM if CI is green -- I don't think I need to see this even if further changes are required, so don't be blocked on me being absent during the holidays in case you need to make more changes to make this pass.

[mstorsjo]

Restored the naming to llvm-libc++-{static,shared}-clangcl.cfg.in for now (not worth breaking the consistency with the -gcc one for now), rerunning CI.

libcxx/test/configs/cmake-bridge.cfg.in

 @AUTO_GEN_COMMENT@
 
 @SERIALIZED_LIT_PARAMS@
 
+import shlex
+
 #
 # This file performs the bridge between the CMake configuration and the Lit
 # configuration files by setting up the LitConfig object and various Lit
 # substitutions from CMake variables.
 #
 # Individual configuration files can take advantage of this bridge by
 # loading the file and then setting up the remaining Lit substitutions.
 #
 
 import os, site
 site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
 import libcxx.test.format
 
 # Basic configuration of the test suite
 config.name = os.path.basename('@LIBCXX_TEST_CONFIG@')
 config.test_source_root = os.path.join('@LIBCXX_SOURCE_DIR@', 'test')
 config.test_format = libcxx.test.format.CxxStandardLibraryTest()
 config.recursiveExpansionLimit = 10
 config.test_exec_root = '@CMAKE_BINARY_DIR@'
 
 # Add substitutions for bootstrapping the test suite configuration
-config.substitutions.append(('%{cxx}', '@CMAKE_CXX_COMPILER@'))
+config.substitutions.append(('%{cxx}', shlex.quote('@CMAKE_CXX_COMPILER@')))
 config.substitutions.append(('%{libcxx}', '@LIBCXX_SOURCE_DIR@'))
 config.substitutions.append(('%{install}', '@CMAKE_BINARY_DIR@'))
 config.substitutions.append(('%{include}', '%{install}/@LIBCXX_INSTALL_INCLUDE_DIR@'))
 config.substitutions.append(('%{target-include}', '%{install}/@LIBCXX_INSTALL_INCLUDE_TARGET_DIR@'))
 config.substitutions.append(('%{lib}', '%{install}/@LIBCXX_INSTALL_LIBRARY_DIR@'))
 config.substitutions.append(('%{executor}', '@LIBCXX_EXECUTOR@'))

libcxx/test/configs/llvm-libc++-mingw.cfg.in

+# This testing configuration handles running the test suite against LLVM's libc++
+# using either a DLL or a static library, with MinGW/Clang on Windows.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+config.substitutions.append(('%{flags}', ''))
+config.substitutions.append(('%{compile_flags}',
+    '-nostdinc++ -isystem %{include} -isystem %{target-include} -I %{libcxx}/test/support'
+))
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib++ -L %{lib} -lc++'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor} --execdir %T --env PATH=%{lib} -- '
+))
+
+# LIBCXX-WINDOWS-FIXME is the feature name used to XFAIL the
+# initial Windows failures until they can be properly diagnosed
+# and fixed. This allows easier detection of new test failures
+# and regressions. Note: New failures should not be suppressed
+# using this feature. (Also see llvm.org/PR32730)
+config.available_features.add('LIBCXX-WINDOWS-FIXME')
+
+import os, site
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+import libcxx.test.params, libcxx.test.newconfig
+libcxx.test.newconfig.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/test/configs/llvm-libc++-shared-clangcl.cfg.in

+# This testing configuration handles running the test suite against LLVM's libc++
+# using a DLL, with Clang-cl on Windows.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+config.substitutions.append(('%{flags}', '--driver-mode=g++'))
+config.substitutions.append(('%{compile_flags}',
+    '-nostdinc++ -isystem %{include} -isystem %{target-include} -I %{libcxx}/test/support -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_STDIO_ISO_WIDE_SPECIFIERS -DNOMINMAX'
+))
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib -L %{lib} -lc++ -lmsvcrt -lmsvcprt -loldnames'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor} --execdir %T --env PATH=%{lib} -- '
+))
+
+# LIBCXX-WINDOWS-FIXME is the feature name used to XFAIL the
+# initial Windows failures until they can be properly diagnosed
+# and fixed. This allows easier detection of new test failures
+# and regressions. Note: New failures should not be suppressed
+# using this feature. (Also see llvm.org/PR32730)
+config.available_features.add('LIBCXX-WINDOWS-FIXME')
+
+import os, site
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+import libcxx.test.params, libcxx.test.newconfig
+libcxx.test.newconfig.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/test/configs/llvm-libc++-static-clangcl.cfg.in

+# This testing configuration handles running the test suite against LLVM's libc++
+# using a static library, with Clang-cl on Windows.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+config.substitutions.append(('%{flags}', '--driver-mode=g++'))
+config.substitutions.append(('%{compile_flags}',
+    '-nostdinc++ -isystem %{include} -isystem %{target-include} -I %{libcxx}/test/support -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_STDIO_ISO_WIDE_SPECIFIERS -DNOMINMAX'
+))
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib -L %{lib} -llibc++ -lmsvcrt -lmsvcprt -loldnames'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor} --execdir %T --env PATH=%{lib} -- '
+))
+
+# LIBCXX-WINDOWS-FIXME is the feature name used to XFAIL the
+# initial Windows failures until they can be properly diagnosed
+# and fixed. This allows easier detection of new test failures
+# and regressions. Note: New failures should not be suppressed
+# using this feature. (Also see llvm.org/PR32730)
+config.available_features.add('LIBCXX-WINDOWS-FIXME')
+
+import os, site
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+import libcxx.test.params, libcxx.test.newconfig
+libcxx.test.newconfig.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/utils/ci/run-buildbot

 clang-cl-dll)
     clean
     # TODO: Currently, building with the experimental library breaks running
     # tests (the test linking look for the c++experimental library with the
     # wrong name, and the statically linked c++experimental can't be linked
     # correctly when libc++ visibility attributes indicate dllimport linkage
     # anyway), thus just disable the experimental library. Remove this
     # setting when cmake and the test driver does the right thing automatically.
-    generate-cmake-libcxx-win -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF
+    generate-cmake-libcxx-win -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF \
+                              -DLIBCXX_TEST_CONFIG="llvm-libc++-shared-clangcl.cfg.in"
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
 ;;
 clang-cl-static)
     clean
-    generate-cmake-libcxx-win -DLIBCXX_ENABLE_SHARED=OFF
+    generate-cmake-libcxx-win -DLIBCXX_ENABLE_SHARED=OFF \
+                              -DLIBCXX_TEST_CONFIG="llvm-libc++-static-clangcl.cfg.in"
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
 ;;
 mingw-dll)
     clean
     # Explicitly specify the compiler with a triple prefix. The CI
     # environment has got two installations of Clang; the default one
     # defaults to MSVC mode, while there's an installation of llvm-mingw
     # further back in PATH. By calling the compiler with an explicit
     # triple prefix, we use the one that is bundled with a mingw sysroot.
     generate-cmake \
           -DCMAKE_C_COMPILER=x86_64-w64-mingw32-clang \
           -DCMAKE_CXX_COMPILER=x86_64-w64-mingw32-clang++ \
+          -DLIBCXX_TEST_CONFIG="llvm-libc++-mingw.cfg.in" \
           -C "${MONOREPO_ROOT}/libcxx/cmake/caches/MinGW.cmake"
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
 ;;
 mingw-static)
     clean
     generate-cmake \
           -DCMAKE_C_COMPILER=x86_64-w64-mingw32-clang \
           -DCMAKE_CXX_COMPILER=x86_64-w64-mingw32-clang++ \
+          -DLIBCXX_TEST_CONFIG="llvm-libc++-mingw.cfg.in" \
           -C "${MONOREPO_ROOT}/libcxx/cmake/caches/MinGW.cmake" \
           -DLIBCXX_ENABLE_SHARED=OFF \
           -DLIBUNWIND_ENABLE_SHARED=OFF
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}" check-cxx
 ;;
 aix)
     export CC=ibm-clang