[libcxx] [test] Use proper UTF-8 locales on Windows

[mstorsjo]

Summary

Since Windows 10 version 1803 (10.0.17134.0) (or Windows Server 2019),
the Windows Universal C Runtime (UCRT) actually does support UTF-8
locales - they're available e.g. via the same names as commonly on Unices,
e.g. "en_US.UTF-8".

The UTF-8 locale support unfortunately has a bug which breaks a couple
tests that were passing previously. That bug is fixed in the very
latest version of the UCRT (in UCRT 10.0.20348.0, available in Windows
11 or Windows Server 2022), so it will get resolved at some point
eventually, provided that the CI environment does get upgraded to a
newer version of Windows Server.

While the net number of xfailed/passing tests in this patch is a loss,
this does allow fixing a lot more locale tests properly for Windows
in later patches.

Intentionally not touching the ISO-8859-1/2 locales used for testing;
they're not detected and tested/used right now, and fixing that up
is another project.

[Mordante]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

22

Would it be easy to detect the UCRT version in CMake and add an appropriate flag libcpp-no-vcruntime? Then we can use UNSUPPORTED: libcpp-ucrt-has-utf8-bug. That way others who want to test on Windows don't need to use the exact same version as the CI has.

When that's not possible I would prefer UNSUPPORTED over XFAIL. Again to avoid failing tests depending on which Windows platform they're executed on.

[mstorsjo]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

22

Hmm, unsure if it's possible to check the version of UCRT. But I guess we could do a proper runtime check when starting testing - like we already do to see if the system supports e.g. the locales we test. I'll give that a shot. (That can be used for other temporary platform bugs too.)

Yeah UNSUPPORTED is good if there's going to be variability - but with the drawback that it's easy to forget that in place for too long, when the issue is resolved in the CI environment. (While it's nice if the tests pass on any version of Windows, once the CI environment no longer require workarounds, I'd move towards removing them.)

[ldionne]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

22

I think it's definitely better to use XFAIL than UNSUPPORTED, especially in these circumstances. XFAIL allows forgetting about this, since it'll start XPASSing when the issue gets fixed eventually. If we use UNSUPPORTED, we are basically dropping this test coverage forever, unless someone comes back and notices that.

I agree even better would be to have a version number -- then we can use either UNSUPPORTED or XFAIL, it doesn't matter, since it won't apply once we update to something newer.

[mstorsjo]

Add a feature flag broken-utf8-wchar-ctype, check for this bug by building and running a code snippet (just like the locale detection); only test for that bug if building for Windows.

Refactored hasAnyLocale to use an intermediate helper programSucceeds instead of calling programOutput directly. (Also added a testcase for this new DSL function.)

[mstorsjo]

Rebased, reordered local patches to make the diff apply cleanly on upstream.

[Mordante]

Great that we can test it in CMake, thanks!
LGTM modulo one small issue.

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

22

My reason for UNSUPPORTED over XFAIL in this case, is that it depends on the version of the Windows system the user runs. So XFAIL would be nice for our CI, but will be annoying when a user tests it on a newer Windows than our CI.

In general I agree that an XFAIL is better or a UNSUPPORTED on a specific version.

But it seems that @mstorsjo found a way to detect it in CMake :-)

libcxx/utils/libcxx/test/features.py

81 ↗

(On Diff #409519)

This header provides towlower.

[ldionne]

LGTM with test added. Also @Mordante had some comments, let's address those.

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

223 ↗

(On Diff #409519)

Can you add a test checking that we raise ConfigurationCompilationError when we fail to compile the program? I.e. it's good to test that we don't swallow the error and return False.

[mstorsjo]

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

223 ↗

(On Diff #409519)

Sure, will do!

libcxx/utils/libcxx/test/features.py

81 ↗

(On Diff #409519)

Thanks, will fix before pushing.

libcxx/test/std/input.output/file.streams/fstreams/filebuf.virtuals/overflow.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <fstream>
 
 // int_type overflow(int_type c = traits::eof());
 
 // This test is not entirely portable
 
 #include <fstream>
 #include <cassert>

libcxx/test/std/input.output/file.streams/fstreams/filebuf.virtuals/underflow.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // FILE_DEPENDENCIES: underflow.dat, underflow_utf8.dat
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <fstream>
 
 // int_type underflow();
 
 // This test is not entirely portable
 
 #include <fstream>
 #include <cstddef>

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT tolower(charT) const;
 
+// Windows' UCRT has a bug regarding tolower/touppwer when using UTF-8
+// locales, breaking this test:
+// https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+// The bug is fixed in UCRT 10.0.20348.0 and later (Windows Server 2022
+// or Windows 11).
+// XFAIL: windows
+
 #include <locale>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
 
 int main(int, char**)
 {

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_many.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* tolower(charT* low, const charT* high) const;
 
+// Windows' UCRT has a bug regarding tolower/touppwer when using UTF-8
+// locales, breaking this test:
+// https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+// The bug is fixed in UCRT 10.0.20348.0 and later (Windows Server 2022
+// or Windows 11).
+// XFAIL: windows
+
 #include <locale>
 #include <string>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
 
 int main(int, char**)

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_1.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT toupper(charT) const;
 
+// Windows' UCRT has a bug regarding tolower/touppwer when using UTF-8
+// locales, breaking this test:
+// https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+// The bug is fixed in UCRT 10.0.20348.0 and later (Windows Server 2022
+// or Windows 11).
+// XFAIL: windows
 
 #include <locale>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
 
 int main(int, char**)

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_many.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* toupper(charT* low, const charT* high) const;
 
+// Windows' UCRT has a bug regarding tolower/touppwer when using UTF-8
+// locales, breaking this test:
+// https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+// The bug is fixed in UCRT 10.0.20348.0 and later (Windows Server 2022
+// or Windows 11).
+// XFAIL: windows
+
 #include <locale>
 #include <string>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
 
 int main(int, char**)

libcxx/test/std/localization/locale.categories/category.time/locale.time.get.byname/get_monthname.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // REQUIRES: locale.fr_FR.UTF-8
 // REQUIRES: locale.zh_CN.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <locale>
 
 // class time_get_byname<charT, InputIterator>
 
 // iter_type
 // get_monthname(iter_type s, iter_type end, ios_base& str,
 //               ios_base::iostate& err, tm* t) const;
 

libcxx/test/std/localization/locale.categories/category.time/locale.time.get.byname/get_weekday.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: locale.en_US.UTF-8
 // REQUIRES: locale.fr_FR.UTF-8
 // REQUIRES: locale.ru_RU.UTF-8
 // REQUIRES: locale.zh_CN.UTF-8
 
-// XFAIL: LIBCXX-WINDOWS-FIXME
-
 // <locale>
 
 // class time_get_byname<charT, InputIterator>
 
 // iter_type
 // get_weekday(iter_type s, iter_type end, ios_base& str,
 //             ios_base::iostate& err, tm* t) const;
 

libcxx/test/std/re/re.traits/translate_nocase.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <regex>
 
 // template <class charT> struct regex_traits;
 
 // charT translate_nocase(charT c) const;
 
 // REQUIRES: locale.en_US.UTF-8
 
+// Windows' UCRT has a bug regarding tolower/touppwer when using UTF-8
+// locales, breaking this test:
+// https://developercommunity.visualstudio.com/t/utf-8-locales-break-ctype-functions-for-wchar-type/1653678
+// The bug is fixed in UCRT 10.0.20348.0 and later (Windows Server 2022
+// or Windows 11).
+// XFAIL: windows
+
 #include <regex>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h"
 
 int main(int, char**)
 {

libcxx/test/support/platform_support.h

 //  implementation defined assumptions:
 //   - locale names
 //   - floating point number string output
 
 #ifndef PLATFORM_SUPPORT_H
 #define PLATFORM_SUPPORT_H
 
 // locale names
-#ifdef _WIN32
-    // WARNING: Windows does not support UTF-8 codepages.
-    // Locales are "converted" using https://docs.moodle.org/dev/Table_of_locales
-#   define LOCALE_en_US           "en-US"
-#   define LOCALE_en_US_UTF_8     "en-US"
-#   define LOCALE_cs_CZ_ISO8859_2 "cs-CZ"
-#   define LOCALE_fr_FR_UTF_8     "fr-FR"
-#   define LOCALE_fr_CA_ISO8859_1 "fr-CA"
-#   define LOCALE_ru_RU_UTF_8     "ru-RU"
-#   define LOCALE_zh_CN_UTF_8     "zh-CN"
-#else
-#   define LOCALE_en_US           "en_US"
-#   define LOCALE_en_US_UTF_8     "en_US.UTF-8"
-#   define LOCALE_fr_FR_UTF_8     "fr_FR.UTF-8"
-#   ifdef __linux__
-#       define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO-8859-1"
-#       define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO-8859-2"
-#   else
-#       define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO8859-1"
-#       define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO8859-2"
-#   endif
-#   define LOCALE_ru_RU_UTF_8     "ru_RU.UTF-8"
-#   define LOCALE_zh_CN_UTF_8     "zh_CN.UTF-8"
-#endif
+#define LOCALE_en_US           "en_US"
+#define LOCALE_en_US_UTF_8     "en_US.UTF-8"
+#define LOCALE_fr_FR_UTF_8     "fr_FR.UTF-8"
+#ifdef __linux__
+#    define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO-8859-1"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO-8859-2"
+#elif defined(_WIN32)
+#    define LOCALE_fr_CA_ISO8859_1 "fr-CA"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs-CZ"
+#else
+#    define LOCALE_fr_CA_ISO8859_1 "fr_CA.ISO8859-1"
+#    define LOCALE_cs_CZ_ISO8859_2 "cs_CZ.ISO8859-2"
+#endif
+#define LOCALE_ru_RU_UTF_8     "ru_RU.UTF-8"
+#define LOCALE_zh_CN_UTF_8     "zh_CN.UTF-8"
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <codecvt>
 #include <locale>
 #include <string>
 #if defined(_WIN32)
 #   include <io.h> // _mktemp_s