[libcxx] Fix the ctype `is` (pointer version) function for Windows

[mstorsjo]

Summary

Previously, this test snippet would report incorrect information:

F::mask m;
std::wstring in(L"\u00DA"); // LATIN CAPITAL LETTER U WITH ACUTE
f.is(in.data(), in.data() + 1, &m);
// m & F::lower would be set

The single-character version of the is function wasn't
affected by this issue though.

Define _LIBCPP_CTYPE_MASK_IS_COMPOSITE_ALPHA for Windows,
as the alpha / _ALPHA constant is a mask consisting of
multiple bits set, which avoids setting alpha whenver any
of the bits is set, in the do_is implementation.

On Windows, with the "C" locale, characters outside of ASCII are
interpreted according to the current system code page, which
can consider chars like e.g. 0xDA as an uppercase alphabetical
character.

Due to the differing classification of some characters, the
scan_is and scan_not tests are quite annoying to fix, thus just
ifdef out some of the tests for the "C" locale there - the code gets
tested with the more standard en_US.UTF-8 locale anyway.

[• Quuxplusone]

libcxx/include/__locale

457

This looks like a bugfix, so "Fix the ... tests" seems like the wrong commit summary.
Before this patch, did f.is(F::upper, L'a') return true instead of false? That'd be a good thing to put in the commit message.

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

110–112

This comment is too vague for my liking. I infer that "can be" means sometimes it is and sometimes it isn't. I'd rather see something very specific, like

// On Windows, this depends on the current codepage.
// If the codepage is CP-437, \x00DA is U+250C BOX DRAWINGS LIGHT DOWN AND RIGHT.
// If the codepage is CP-1252, \x00DA is U+00DA CAPITAL LETTER U WITH ACUTE.

And then I'm not actually sure why the non-Windows behavior claims that U+00DA is not alpha+upper.

[mstorsjo]

Updated the comments as requested (will update subject+description too).

[mstorsjo]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

110–112

I think in the non-Windows case, the "C" locale might correspond to plain ASCII or something like that, where \x00DA isn't considered an alphabetical character.

[• Quuxplusone]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

110–112

Nit: for readability, I'd prefer linebreaks before each "If", and ideally no other linebreaks.

I'm still confused as to why the comment says "it depends" but the code on lines 114-115 is very clearly assuming that the character is U+00DA CAPITAL LETTER U WITH ACUTE. Is this because the comment is wrong? or because the test runner forces a specific codepage? or because we just luck into a specific codepage and this is super fragile? (Ditto throughout.)

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/scan_not.pass.cpp

75

Pre-existing, no action required: This test seems silly. All of the expected results are 0.

[mstorsjo]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

110–112

I think this (and a bunch of other similar tests for the "C" locale) might behave differently, if the systemwide default codepage was a more exotic one, yes. Hence, it strictly depends on what it is set to. I dno't think the tests have much say over what the "C" locale is.

Sure, I can unwrap the comment to make it more readable as you wrote originally.

[mstorsjo]

Clarified the reliance on the system's default codepage in the comments, unwrapped the unicode code point explanations as requested.

[mstorsjo]

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

110–112

As an alternative, we could also just ifdef out all tests for the non-ASCII chars in the "C" locales on Windows, as the interpretation strictly does depend on what the system's default codepage is.

[mstorsjo]

I investigated the behaviour of these tests on odd system codepages, and it actually doesn't affect it. On Windows, wchars are correctly classified according to what unicode the wchar represents even when the "C" locale is set (contrary to other OSes) - regardless of the system codepage.

Thus, I changed the comments to something clearer - now this should be safe and good to go as is.

[• Quuxplusone]

LGTM % nit: remove the word "proper", mainly for unnecessary-editorializing but also because I think the editorializing is wrong? Seems to me like the C locale "properly" shouldn't be using Unicode interpretations! :)

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

111

:)

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_many.pass.cpp

159

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/scan_is.pass.cpp

67

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/scan_not.pass.cpp

67

libcxx/include/__locale

#elif defined(_LIBCPP_MSVCRT_LIKE)
     static const mask lower  = _LOWER;
     static const mask alpha  = _ALPHA;
     static const mask digit  = _DIGIT;
     static const mask punct  = _PUNCT;
     static const mask xdigit = _HEX;
     static const mask blank  = _BLANK;
     static const mask __regex_word = 0x4000; // 0x8000 and 0x0100 and 0x00ff are used
 # define _LIBCPP_CTYPE_MASK_IS_COMPOSITE_PRINT
+# define _LIBCPP_CTYPE_MASK_IS_COMPOSITE_ALPHA
 #elif defined(__APPLE__) || defined(__FreeBSD__) || defined(__EMSCRIPTEN__) || defined(__NetBSD__)
 # ifdef __APPLE__
     typedef __uint32_t mask;
 # elif defined(__FreeBSD__)
     typedef unsigned long mask;
 # elif defined(__EMSCRIPTEN__) || defined(__NetBSD__)
     typedef unsigned short mask;
 # endif

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_1.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // bool is(mask m, charT c) const;
 
 // REQUIRES: locale.en_US.UTF-8
-// XFAIL: LIBCXX-WINDOWS-FIXME
 // XFAIL: libcpp-has-no-wide-characters
 
 #include <locale>
 #include <type_traits>
 #include <cassert>
 
 #include "test_macros.h"
 #include "platform_support.h" // locale name macros
int main(int, char**)
             assert(!f.is(F::xdigit, L'g'));
 
             assert(f.is(F::alnum, L'a'));
             assert(!f.is(F::alnum, L'.'));
 
             assert(f.is(F::graph, L'.'));
             assert(!f.is(F::graph,  L'\x07'));
 
+#if defined(_WIN32)
+            // On Windows, these wchars are classified according to their
+            // Unicode interpretation even in the "C" locale.
+            assert(f.is(F::alpha, L'\x00DA'));
+            assert(f.is(F::upper, L'\x00DA'));
+#else
             assert(!f.is(F::alpha, L'\x00DA'));
             assert(!f.is(F::upper, L'\x00DA'));
+#endif
         }
     }
 
   return 0;
 }

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/is_many.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* do_is(const charT* low, const charT* high, mask* vec) const;
 
 // REQUIRES: locale.en_US.UTF-8
-// XFAIL: LIBCXX-WINDOWS-FIXME
 // XFAIL: libcpp-has-no-wide-characters
 
 #include <locale>
 #include <string>
 #include <vector>
 #include <cassert>
 
 #include <stdio.h>
int main(int, char**)
             const F& f = std::use_facet<F>(l);
             const std::wstring in(L"\x00DA A\x07.a1");
             std::vector<F::mask> m(in.size());
             const wchar_t* h = f.is(in.data(), in.data() + in.size(), m.data());
             assert(h == in.data() + in.size());
 
             // L'\x00DA'
             assert(!(m[0] & F::space));
-            assert(!(m[0] & F::print));
             assert(!(m[0] & F::cntrl));
-            assert(!(m[0] & F::upper));
             assert(!(m[0] & F::lower));
-            assert(!(m[0] & F::alpha));
             assert(!(m[0] & F::digit));
             assert(!(m[0] & F::punct));
             assert(!(m[0] & F::xdigit));
             assert(!(m[0] & F::blank));
+#if defined(_WIN32)
+            // On Windows, these wchars are classified according to their
+            // Unicode interpretation even in the "C" locale.
+            assert( (m[0] & F::alpha));
+            assert( (m[0] & F::upper));
+            assert( (m[0] & F::print));
+            assert( (m[0] & F::alnum));
+            assert( (m[0] & F::graph));
+#else
+            assert(!(m[0] & F::alpha));
+            assert(!(m[0] & F::upper));
+            assert(!(m[0] & F::print));
             assert(!(m[0] & F::alnum));
             assert(!(m[0] & F::graph));
+#endif
 
             // L' '
             assert( (m[1] & F::space));
             assert( (m[1] & F::print));
             assert(!(m[1] & F::cntrl));
             assert(!(m[1] & F::upper));
             assert(!(m[1] & F::lower));
             assert(!(m[1] & F::alpha));

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/scan_is.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* scan_is(mask m, const charT* low, const charT* high) const;
 
 // REQUIRES: locale.en_US.UTF-8
-// XFAIL: LIBCXX-WINDOWS-FIXME
 // XFAIL: libcpp-has-no-wide-characters
 
 #include <locale>
 #include <string>
 #include <vector>
 #include <cassert>
 
 #include <stdio.h>
int main(int, char**)
     {
         std::locale l("C");
         {
             typedef std::ctype<wchar_t> F;
             const F& f = std::use_facet<F>(l);
             const std::wstring in(L"\x00DA A\x07.a1");
             std::vector<F::mask> m(in.size());
             assert(f.scan_is(F::space, in.data(), in.data() + in.size()) - in.data() == 1);
-            assert(f.scan_is(F::print, in.data(), in.data() + in.size()) - in.data() == 1);
             assert(f.scan_is(F::cntrl, in.data(), in.data() + in.size()) - in.data() == 3);
-            assert(f.scan_is(F::upper, in.data(), in.data() + in.size()) - in.data() == 2);
             assert(f.scan_is(F::lower, in.data(), in.data() + in.size()) - in.data() == 5);
-            assert(f.scan_is(F::alpha, in.data(), in.data() + in.size()) - in.data() == 2);
             assert(f.scan_is(F::digit, in.data(), in.data() + in.size()) - in.data() == 6);
             assert(f.scan_is(F::punct, in.data(), in.data() + in.size()) - in.data() == 4);
             assert(f.scan_is(F::xdigit, in.data(), in.data() + in.size()) - in.data() == 2);
             assert(f.scan_is(F::blank, in.data(), in.data() + in.size()) - in.data() == 1);
+#if !defined(_WIN32)
+            // On Windows, these wchars are classified according to their
+            // Unicode interpretation even in the "C" locale, where
+            // the scan_is function returns the same as above for the
+            // en_US.UTF-8 locale.
+            assert(f.scan_is(F::print, in.data(), in.data() + in.size()) - in.data() == 1);
+            assert(f.scan_is(F::upper, in.data(), in.data() + in.size()) - in.data() == 2);
+            assert(f.scan_is(F::alpha, in.data(), in.data() + in.size()) - in.data() == 2);
             assert(f.scan_is(F::alnum, in.data(), in.data() + in.size()) - in.data() == 2);
             assert(f.scan_is(F::graph, in.data(), in.data() + in.size()) - in.data() == 2);
+#endif
         }
     }
 
   return 0;
 }

libcxx/test/std/localization/locale.categories/category.ctype/locale.ctype.byname/scan_not.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* scan_not(mask m, const charT* low, const charT* high) const;
 
 // REQUIRES: locale.en_US.UTF-8
-// XFAIL: LIBCXX-WINDOWS-FIXME
 // XFAIL: libcpp-has-no-wide-characters
 
 #include <locale>
 #include <string>
 #include <vector>
 #include <cassert>
 
 #include <stdio.h>
int main(int, char**)
     {
         std::locale l("C");
         {
             typedef std::ctype<wchar_t> F;
             const F& f = std::use_facet<F>(l);
             const std::wstring in(L"\x00DA A\x07.a1");
             std::vector<F::mask> m(in.size());
             assert(f.scan_not(F::space, in.data(), in.data() + in.size()) - in.data() == 0);
-            assert(f.scan_not(F::print, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::cntrl, in.data(), in.data() + in.size()) - in.data() == 0);
-            assert(f.scan_not(F::upper, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::lower, in.data(), in.data() + in.size()) - in.data() == 0);
-            assert(f.scan_not(F::alpha, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::digit, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::punct, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::xdigit, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::blank, in.data(), in.data() + in.size()) - in.data() == 0);
+#if !defined(_WIN32)
+            // On Windows, these wchars are classified according to their
+            // Unicode interpretation even in the "C" locale, where
+            // the scan_is function returns the same as above for the
+            // en_US.UTF-8 locale.
+            assert(f.scan_not(F::print, in.data(), in.data() + in.size()) - in.data() == 0);
+            assert(f.scan_not(F::upper, in.data(), in.data() + in.size()) - in.data() == 0);
+            assert(f.scan_not(F::alpha, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::alnum, in.data(), in.data() + in.size()) - in.data() == 0);
             assert(f.scan_not(F::graph, in.data(), in.data() + in.size()) - in.data() == 0);
+#endif
         }
     }
 
   return 0;
 }