[libcxx] [test] Use proper UTF-8 locales on Windows

[mstorsjo]

Summary

Since Windows 10 version 1803 (10.0.17134.0) (or Windows Server 2019),
the Windows Universal C Runtime (UCRT) actually does support UTF-8
locales - they're available e.g. via the same names as commonly on Unices,
e.g. "en_US.UTF-8".

The UTF-8 locale support unfortunately has a bug which breaks a couple
tests that were passing previously. That bug is fixed in the very
latest version of the UCRT (in UCRT 10.0.20348.0, available in Windows
11 or Windows Server 2022), so it will get resolved at some point
eventually, provided that the CI environment does get upgraded to a
newer version of Windows Server.

While the net number of xfailed/passing tests in this patch is a loss,
this does allow fixing a lot more locale tests properly for Windows
in later patches.

Intentionally not touching the ISO-8859-1/2 locales used for testing;
they're not detected and tested/used right now, and fixing that up
is another project.

[Mordante]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

23

Would it be easy to detect the UCRT version in CMake and add an appropriate flag libcpp-no-vcruntime? Then we can use UNSUPPORTED: libcpp-ucrt-has-utf8-bug. That way others who want to test on Windows don't need to use the exact same version as the CI has.

When that's not possible I would prefer UNSUPPORTED over XFAIL. Again to avoid failing tests depending on which Windows platform they're executed on.

[mstorsjo]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

23

Hmm, unsure if it's possible to check the version of UCRT. But I guess we could do a proper runtime check when starting testing - like we already do to see if the system supports e.g. the locales we test. I'll give that a shot. (That can be used for other temporary platform bugs too.)

Yeah UNSUPPORTED is good if there's going to be variability - but with the drawback that it's easy to forget that in place for too long, when the issue is resolved in the CI environment. (While it's nice if the tests pass on any version of Windows, once the CI environment no longer require workarounds, I'd move towards removing them.)

[ldionne]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

23

I think it's definitely better to use XFAIL than UNSUPPORTED, especially in these circumstances. XFAIL allows forgetting about this, since it'll start XPASSing when the issue gets fixed eventually. If we use UNSUPPORTED, we are basically dropping this test coverage forever, unless someone comes back and notices that.

I agree even better would be to have a version number -- then we can use either UNSUPPORTED or XFAIL, it doesn't matter, since it won't apply once we update to something newer.

[mstorsjo]

Add a feature flag broken-utf8-wchar-ctype, check for this bug by building and running a code snippet (just like the locale detection); only test for that bug if building for Windows.

Refactored hasAnyLocale to use an intermediate helper programSucceeds instead of calling programOutput directly. (Also added a testcase for this new DSL function.)

[mstorsjo]

Rebased, reordered local patches to make the diff apply cleanly on upstream.

[Mordante]

Great that we can test it in CMake, thanks!
LGTM modulo one small issue.

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

23

My reason for UNSUPPORTED over XFAIL in this case, is that it depends on the version of the Windows system the user runs. So XFAIL would be nice for our CI, but will be annoying when a user tests it on a newer Windows than our CI.

In general I agree that an XFAIL is better or a UNSUPPORTED on a specific version.

But it seems that @mstorsjo found a way to detect it in CMake :-)

libcxx/utils/libcxx/test/features.py

81

This header provides towlower.

[ldionne]

LGTM with test added. Also @Mordante had some comments, let's address those.

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

221

Can you add a test checking that we raise ConfigurationCompilationError when we fail to compile the program? I.e. it's good to test that we don't swallow the error and return False.

[mstorsjo]

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

221

Sure, will do!

libcxx/utils/libcxx/test/features.py

81

Thanks, will fix before pushing.

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

def test_program_stderr_is_not_conflated_with_stdout(self):
             std::fprintf(stdout, "STDOUT-OUTPUT");
             std::fprintf(stderr, "STDERR-OUTPUT");
             return 0;
         }
         """
         self.assertEqual(dsl.programOutput(self.config, source), "STDOUT-OUTPUT")
 
 
+class TestProgramSucceeds(SetupConfigs):
+    """
+    Tests for libcxx.test.dsl.programSucceeds
+    """
+    def test_success(self):
+        source = """
+        int main(int, char**) { return 0; }
+        """
+        self.assertTrue(dsl.programSucceeds(self.config, source))
+
+    def test_failure(self):
+        source = """
+        int main(int, char**) { return 1; }
+        """
+        self.assertFalse(dsl.programSucceeds(self.config, source))
+
 class TestHasLocale(SetupConfigs):
     """
     Tests for libcxx.test.dsl.hasLocale
     """
     def test_doesnt_explode(self):
         # It's really hard to test that a system has a given locale, so at least
         # make sure we don't explode when we try to check it.
         try:

libcxx/test/std/input.output/file.streams/fstreams/filebuf.virtuals/overflow.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <fstream>
 
 // int_type overflow(int_type c = traits::eof());
 
 // This test is not entirely portable
 
 #include <fstream>
 #include <cassert>

libcxx/test/std/input.output/file.streams/fstreams/filebuf.virtuals/underflow.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // FILE_DEPENDENCIES: underflow.dat, underflow_utf8.dat
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <fstream>
 
 // int_type underflow();
 
 // This test is not entirely portable
 
 #include <fstream>
 #include <cstddef>

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
+// XFAIL: broken-utf8-wchar-ctype
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT tolower(charT) const;
 
 #include <locale>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
 
 int main(int, char**)
 {
     {
         std::locale l;
         {
             typedef std::ctype_byname<char> F;
             std::locale ll(l, new F(LOCALE_en_US_UTF_8));
             const F& f = std::use_facet<F>(ll);

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_many.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
+// XFAIL: broken-utf8-wchar-ctype
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* tolower(charT* low, const charT* high) const;
 
 #include <locale>

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_1.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
+// XFAIL: broken-utf8-wchar-ctype
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT toupper(charT) const;
 
 

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_many.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
+// XFAIL: broken-utf8-wchar-ctype
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* toupper(charT* low, const charT* high) const;
 
 #include <locale>

libcxx/test/std/localization/locale.categories/category.time/locale.time.get.byname/get_monthname.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // REQUIRES: locale.fr_FR.UTF-8
 // REQUIRES: locale.zh_CN.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <locale>
 
 // class time_get_byname<charT, InputIterator>
 
 // iter_type
 // get_monthname(iter_type s, iter_type end, ios_base& str,
 //               ios_base::iostate& err, tm* t) const;
 

libcxx/test/std/localization/locale.categories/category.time/locale.time.get.byname/get_weekday.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // REQUIRES: locale.fr_FR.UTF-8
 // REQUIRES: locale.ru_RU.UTF-8
 // REQUIRES: locale.zh_CN.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <locale>
 
 // class time_get_byname<charT, InputIterator>
 
 // iter_type
 // get_weekday(iter_type s, iter_type end, ios_base& str,
 //             ios_base::iostate& err, tm* t) const;
 

libcxx/test/std/re/re.traits/translate_nocase.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <regex>
 
 // template <class charT> struct regex_traits;
 
 // charT translate_nocase(charT c) const;
 
 // REQUIRES: locale.en_US.UTF-8
+// XFAIL: broken-utf8-wchar-ctype
 
 #include <regex>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h"
 
 int main(int, char**)

libcxx/test/support/platform_support.h

 //  implementation defined assumptions:
 //   - locale names
 //   - floating point number string output
 
 #ifndef PLATFORM_SUPPORT_H
 #define PLATFORM_SUPPORT_H
 
 // locale names
-#ifdef _WIN32
-    // WARNING: Windows does not support UTF-8 codepages.
-    // Locales are "converted" using https://docs.moodle.org/dev/Table_of_locales
-#   define LOCALE_en_US           "en-US"
-#   define LOCALE_en_US_UTF_8     "en-US"
-#   define LOCALE_cs_CZ_ISO8859_2 "cs-CZ"
-#   define LOCALE_fr_FR_UTF_8     "fr-FR"
-#   define LOCALE_fr_CA_ISO8859_1 "fr-CA"
-#   define LOCALE_ru_RU_UTF_8     "ru-RU"
-#   define LOCALE_zh_CN_UTF_8     "zh-CN"
-#else
-#   define LOCALE_en_US           "en_US"
-#   define LOCALE_en_US_UTF_8     "en_US.UTF-8"
-#   define LOCALE_fr_FR_UTF_8     "fr_FR.UTF-8"
-#   ifdef __linux__
-#       define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO-8859-1"
-#       define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO-8859-2"
-#   else
-#       define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO8859-1"
-#       define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO8859-2"
-#   endif
-#   define LOCALE_ru_RU_UTF_8     "ru_RU.UTF-8"
-#   define LOCALE_zh_CN_UTF_8     "zh_CN.UTF-8"
-#endif
+#define LOCALE_en_US           "en_US"
+#define LOCALE_en_US_UTF_8     "en_US.UTF-8"
+#define LOCALE_fr_FR_UTF_8     "fr_FR.UTF-8"
+#ifdef __linux__
+#    define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO-8859-1"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO-8859-2"
+#elif defined(_WIN32)
+#    define LOCALE_fr_CA_ISO8859_1 "fr-CA"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs-CZ"
+#else
+#    define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO8859-1"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO8859-2"
+#endif
+#define LOCALE_ru_RU_UTF_8     "ru_RU.UTF-8"
+#define LOCALE_zh_CN_UTF_8     "zh_CN.UTF-8"
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <codecvt>
 #include <locale>
 #include <string>
 #if defined(_WIN32)
 #   include <io.h> // _mktemp_s

libcxx/utils/libcxx/test/dsl.py

if exitCode != 0:
       raise ConfigurationRuntimeError("Failed to run program, stderr is:\n{}".format(err))
 
     actualOut = re.search("# command output:\n(.+)\n$", out, flags=re.DOTALL)
     actualOut = actualOut.group(1) if actualOut else ""
     if platform.system() == 'Windows':
       actualOut = actualOut.replace('\r\n', '\n')
     return actualOut
 
+@_memoizeExpensiveOperation(lambda c, p, args=None: (c.substitutions, c.environment, p, args))
+def programSucceeds(config, program, args=None):
+  """
+  Compiles a program for the test target, run it on the test target and return
+  whether it completed successfully.
+
+  Note that execution of the program is done through the %{exec} substitution,
+  which means that the program may be run on a remote host depending on what
+  %{exec} does.
+  """
+  try:
+    programOutput(config, program, args)
+  except ConfigurationRuntimeError:
+    return False
+  return True
+
 @_memoizeExpensiveOperation(lambda c, f: (c.substitutions, c.environment, f))
 def hasCompileFlag(config, flag):
   """
   Return whether the compiler in the configuration supports a given compiler flag.
 
   This is done by executing the %{cxx} substitution with the given flag and
   checking whether that succeeds.
   """
#else
           if (::setlocale(LC_ALL, argv[i]) != NULL) {
             return 0;
           }
         }
         return 1;
       }
     #endif
   """
-  try:
-    programOutput(config, program, args=[pipes.quote(l) for l in locales])
-  except ConfigurationRuntimeError:
-    return False
-  return True
+  return programSucceeds(config, program, args=[pipes.quote(l) for l in locales])
 
 @_memoizeExpensiveOperation(lambda c, flags='': (c.substitutions, c.environment, flags))
 def compilerMacros(config, flags=''):
   """
   Return a dictionary of predefined compiler macros.
 
   The keys are strings representing macros, and the values are strings
   representing what each macro is defined to.

libcxx/utils/libcxx/test/features.py

DEFAULT_FEATURES = [
   # Note: To implement this check properly, we need to make sure that we use something inside the
   # compiled library, not only in the headers. It should be safe to assume that all implementations
   # define `operator new` in the compiled library.
   Feature(name='cant-build-shared-library',
           when=lambda cfg: not sourceBuilds(cfg, """
             void f() { new int(3); }
           """, ['-shared'])),
 
+  # Check for a Windows UCRT bug (fixed in UCRT/Windows 10.0.20348.0):
+  # https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+  Feature(name='broken-utf8-wchar-ctype',
+          when=lambda cfg: '_WIN32' in compilerMacros(cfg) and not programSucceeds(cfg, """
+          #include <locale.h>
+          #include <ctype.h>
+          int main(int, char**) {
+            setlocale(LC_ALL, "en_US.UTF-8");
+            return towlower(L'\\xDA') != L'\\xFA';
+          }
+          """)),
+
   # Whether Bash can run on the executor.
   # This is not always the case, for example when running on embedded systems.
   #
   # For the corner case of bash existing, but it being missing in the path
   # set in %{exec} as "--env PATH=one-single-dir", the executor does find
   # and executes bash, but bash then can't find any other common shell
   # utilities. Test executing "bash -c 'bash --version'" to see if bash
   # manages to find binaries to execute.