[libc++] Add missing short wchar handling for codecvt_utf8, codecvt_utf16 and codecvt_utf8_utf16

[jasonliu]

Summary

AIX have 2 byte wchar in 32 bit mode and 4 byte wchar in 64 bit mode.
This patch add more missing short wchar handling under the existing _LIBCPP_SHORT_WCHAR macro.

[SeanP]

Why did you have to add a specialization based on size? Could you provide a test case or two to help explain. My first thought is the _LIBCPP_SHORT_WCHAR macro should be good enough since aix & z/OS only support one wchar_t size per 32/64 bit mode.

[jasonliu]

In D100777#2707189, @SeanP wrote:

Why did you have to add a specialization based on size? Could you provide a test case or two to help explain. My first thought is the _LIBCPP_SHORT_WCHAR macro should be good enough since aix & z/OS only support one wchar_t size per 32/64 bit mode.

With -fshort-wchar, you could have a short wchar in 64 bit mode. So this implementation still gives you the correct result if that's used.

[zibi]

It looks like there are few test cases failing in Windows. Was that investigated to see if it is related to this patch?

[jasonliu]

In D100777#2709968, @jasonliu wrote:

In D100777#2707189, @SeanP wrote:

Why did you have to add a specialization based on size? Could you provide a test case or two to help explain. My first thought is the _LIBCPP_SHORT_WCHAR macro should be good enough since aix & z/OS only support one wchar_t size per 32/64 bit mode.

With -fshort-wchar, you could have a short wchar in 64 bit mode. So this implementation still gives you the correct result if that's used.

Relay some offline discussion: t -fshort-wchar is something that most of libc libraries doesn’t support. i.e. wchar functions won’t return a sensible result when -fshort-wchar is enabled. So there is no point to try to make it work with libc++. So choosing to use existing macro _LIBCPP_SHORT_WCHAR to handle the wchar size differences here.

[jasonliu]

As expected, this changes the behavior of windows platform for the do_length and do_max_length of __codecvt_utf8<wchar_t>.
But I think the changes are actually desired. Could anyone working on Windows platform confirm that?
If we could agree on that, then I could work on updating the test case.

[jasonliu]

Updated test case to account for short wchar_t.

[ldionne]

Can you rebase onto main to trigger CI? This LGTM but I'd like @mstorsjo to peek at this since that setting is used on Windows too.

[jasonliu]

Rebase to trigger CI.

[mstorsjo]

In D100777#2726757, @jasonliu wrote:

As expected, this changes the behavior of windows platform for the do_length and do_max_length of __codecvt_utf8<wchar_t>.
But I think the changes are actually desired. Could anyone working on Windows platform confirm that?

Yes, Windows has got 2 byte wchar_t - so if I read this correctly, libc++ has got an existing define for _LIBCPP_SHORT_WCHAR, but it was incomplete/inconsistent, and this fixes up things to work more correctly for that case? That's certainly good.

In practice, which cases didn't work as they should in that configuration before? I haven't noticed that aspect anywhere in the libc++ tests on Windows (although it could be hidden by some XFAIL: LIBCXX-WINDOWS-FIXME, but that would show up as XPASS in that case too).

If we could agree on that, then I could work on updating the test case.

I don't on a quick glance see what the practical effect of the test updates is, but if it still passes in the Windows CI configurations I guess it should be fine.

[jasonliu]

In D100777#2911594, @mstorsjo wrote:

In practice, which cases didn't work as they should in that configuration before? I haven't noticed that aspect anywhere in the libc++ tests on Windows (although it could be hidden by some XFAIL: LIBCXX-WINDOWS-FIXME, but that would show up as XPASS in that case too).

@mstorsjo I modified the test case so that it would produce the correct result for 2 byte wchar. Without those test case modification, windows CI would fail.
Those test case weren't testing the correct thing for 2 byte wchar and thus previous behavior that produced a result that's intended for 4 byte wchar on Windows verified "clean".

[mstorsjo]

FWIW, no objection from me to proceed with this patch, thanks for the explanation!

[jasonliu]

@daltenty @xingxue
Since I don't have access to AIX machine any more, would you help to test and land this patch if this still aligns to what you want on AIX?
If not, feel free to commandeering this patch if needed.

libcxx/include/__config

 #  include <sys/isa_defs.h>
 #  ifdef _LITTLE_ENDIAN
 #    define _LIBCPP_LITTLE_ENDIAN
 #  else
 #    define _LIBCPP_BIG_ENDIAN
 #  endif
 #endif // __sun__
 
+#if defined(_AIX) && !defined(__64BIT__)
+   // The size of wchar is 2 byte on 32-bit mode on AIX.
+#  define _LIBCPP_SHORT_WCHAR   1
+#endif
+
 #if defined(__OpenBSD__) || defined(__CloudABI__)
    // Certain architectures provide arc4random(). Prefer using
    // arc4random() over /dev/{u,}random to make it possible to obtain
    // random data even when using sandboxing mechanisms such as chroots,
    // Capsicum, etc.
 #  define _LIBCPP_USING_ARC4_RANDOM
 #elif defined(__Fuchsia__) || defined(__wasi__)
 #  define _LIBCPP_USING_GETENTROPY

libcxx/src/locale.cpp

 }
 
 int
 __codecvt_utf8<wchar_t>::do_length(state_type&,
     const extern_type* frm, const extern_type* frm_end, size_t mx) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
+#if defined(_LIBCPP_SHORT_WCHAR)
+    return utf8_to_ucs2_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#else
     return utf8_to_ucs4_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#endif
 }
 
 int
 __codecvt_utf8<wchar_t>::do_max_length() const noexcept
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    if (_Mode_ & consume_header)
+        return 6;
+    return 3;
+#else
     if (_Mode_ & consume_header)
         return 7;
     return 4;
+#endif
 }
 
 // __codecvt_utf8<char16_t>
 
 __codecvt_utf8<char16_t>::result
 __codecvt_utf8<char16_t>::do_out(state_type&,
     const intern_type* frm, const intern_type* frm_end, const intern_type*& frm_nxt,
     extern_type* to, extern_type* to_end, extern_type*& to_nxt) const

 
 // __codecvt_utf16<wchar_t, false>
 
 __codecvt_utf16<wchar_t, false>::result
 __codecvt_utf16<wchar_t, false>::do_out(state_type&,
     const intern_type* frm, const intern_type* frm_end, const intern_type*& frm_nxt,
     extern_type* to, extern_type* to_end, extern_type*& to_nxt) const
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    const uint16_t* _frm = reinterpret_cast<const uint16_t*>(frm);
+    const uint16_t* _frm_end = reinterpret_cast<const uint16_t*>(frm_end);
+    const uint16_t* _frm_nxt = _frm;
+#else
     const uint32_t* _frm = reinterpret_cast<const uint32_t*>(frm);
     const uint32_t* _frm_end = reinterpret_cast<const uint32_t*>(frm_end);
     const uint32_t* _frm_nxt = _frm;
+#endif
     uint8_t* _to = reinterpret_cast<uint8_t*>(to);
     uint8_t* _to_end = reinterpret_cast<uint8_t*>(to_end);
     uint8_t* _to_nxt = _to;
+#if defined(_LIBCPP_SHORT_WCHAR)
+    result r = ucs2_to_utf16be(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
+                               _Maxcode_, _Mode_);
+#else
     result r = ucs4_to_utf16be(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                                _Maxcode_, _Mode_);
+#endif
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf16<wchar_t, false>::result
 __codecvt_utf16<wchar_t, false>::do_in(state_type&,
     const extern_type* frm, const extern_type* frm_end, const extern_type*& frm_nxt,
     intern_type* to, intern_type* to_end, intern_type*& to_nxt) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
     const uint8_t* _frm_nxt = _frm;
+#if defined(_LIBCPP_SHORT_WCHAR)
+    uint16_t* _to = reinterpret_cast<uint16_t*>(to);
+    uint16_t* _to_end = reinterpret_cast<uint16_t*>(to_end);
+    uint16_t* _to_nxt = _to;
+    result r = utf16be_to_ucs2(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
+                               _Maxcode_, _Mode_);
+#else
     uint32_t* _to = reinterpret_cast<uint32_t*>(to);
     uint32_t* _to_end = reinterpret_cast<uint32_t*>(to_end);
     uint32_t* _to_nxt = _to;
     result r = utf16be_to_ucs4(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                                _Maxcode_, _Mode_);
+#endif
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf16<wchar_t, false>::result
 __codecvt_utf16<wchar_t, false>::do_unshift(state_type&,
     extern_type* to, extern_type*, extern_type*& to_nxt) const

 }
 
 int
 __codecvt_utf16<wchar_t, false>::do_length(state_type&,
     const extern_type* frm, const extern_type* frm_end, size_t mx) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
+#if defined(_LIBCPP_SHORT_WCHAR)
+    return utf16be_to_ucs2_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#else
     return utf16be_to_ucs4_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#endif
 }
 
 int
 __codecvt_utf16<wchar_t, false>::do_max_length() const noexcept
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    if (_Mode_ & consume_header)
+        return 4;
+    return 2;
+#else
     if (_Mode_ & consume_header)
         return 6;
     return 4;
+#endif
 }
 
 // __codecvt_utf16<wchar_t, true>
 
 __codecvt_utf16<wchar_t, true>::result
 __codecvt_utf16<wchar_t, true>::do_out(state_type&,
     const intern_type* frm, const intern_type* frm_end, const intern_type*& frm_nxt,
     extern_type* to, extern_type* to_end, extern_type*& to_nxt) const
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    const uint16_t* _frm = reinterpret_cast<const uint16_t*>(frm);
+    const uint16_t* _frm_end = reinterpret_cast<const uint16_t*>(frm_end);
+    const uint16_t* _frm_nxt = _frm;
+#else
     const uint32_t* _frm = reinterpret_cast<const uint32_t*>(frm);
     const uint32_t* _frm_end = reinterpret_cast<const uint32_t*>(frm_end);
     const uint32_t* _frm_nxt = _frm;
+#endif
     uint8_t* _to = reinterpret_cast<uint8_t*>(to);
     uint8_t* _to_end = reinterpret_cast<uint8_t*>(to_end);
     uint8_t* _to_nxt = _to;
+#if defined(_LIBCPP_SHORT_WCHAR)
+    result r = ucs2_to_utf16le(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
+                               _Maxcode_, _Mode_);
+#else
     result r = ucs4_to_utf16le(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                                _Maxcode_, _Mode_);
+#endif
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf16<wchar_t, true>::result
 __codecvt_utf16<wchar_t, true>::do_in(state_type&,
     const extern_type* frm, const extern_type* frm_end, const extern_type*& frm_nxt,
     intern_type* to, intern_type* to_end, intern_type*& to_nxt) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
     const uint8_t* _frm_nxt = _frm;
+#if defined(_LIBCPP_SHORT_WCHAR)
+    uint16_t* _to = reinterpret_cast<uint16_t*>(to);
+    uint16_t* _to_end = reinterpret_cast<uint16_t*>(to_end);
+    uint16_t* _to_nxt = _to;
+    result r = utf16le_to_ucs2(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
+                               _Maxcode_, _Mode_);
+#else
     uint32_t* _to = reinterpret_cast<uint32_t*>(to);
     uint32_t* _to_end = reinterpret_cast<uint32_t*>(to_end);
     uint32_t* _to_nxt = _to;
     result r = utf16le_to_ucs4(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                                _Maxcode_, _Mode_);
+#endif
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf16<wchar_t, true>::result
 __codecvt_utf16<wchar_t, true>::do_unshift(state_type&,
     extern_type* to, extern_type*, extern_type*& to_nxt) const

 }
 
 int
 __codecvt_utf16<wchar_t, true>::do_length(state_type&,
     const extern_type* frm, const extern_type* frm_end, size_t mx) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
+#if defined(_LIBCPP_SHORT_WCHAR)
+    return utf16le_to_ucs2_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#else
     return utf16le_to_ucs4_length(_frm, _frm_end, mx, _Maxcode_, _Mode_);
+#endif
 }
 
 int
 __codecvt_utf16<wchar_t, true>::do_max_length() const noexcept
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    if (_Mode_ & consume_header)
+        return 4;
+    return 2;
+#else
     if (_Mode_ & consume_header)
         return 6;
     return 4;
+#endif
 }
 
 // __codecvt_utf16<char16_t, false>
 
 __codecvt_utf16<char16_t, false>::result
 __codecvt_utf16<char16_t, false>::do_out(state_type&,
     const intern_type* frm, const intern_type* frm_end, const intern_type*& frm_nxt,
     extern_type* to, extern_type* to_end, extern_type*& to_nxt) const

 
 // __codecvt_utf8_utf16<wchar_t>
 
 __codecvt_utf8_utf16<wchar_t>::result
 __codecvt_utf8_utf16<wchar_t>::do_out(state_type&,
     const intern_type* frm, const intern_type* frm_end, const intern_type*& frm_nxt,
     extern_type* to, extern_type* to_end, extern_type*& to_nxt) const
 {
+#if defined(_LIBCPP_SHORT_WCHAR)
+    const uint16_t* _frm = reinterpret_cast<const uint16_t*>(frm);
+    const uint16_t* _frm_end = reinterpret_cast<const uint16_t*>(frm_end);
+    const uint16_t* _frm_nxt = _frm;
+#else
     const uint32_t* _frm = reinterpret_cast<const uint32_t*>(frm);
     const uint32_t* _frm_end = reinterpret_cast<const uint32_t*>(frm_end);
     const uint32_t* _frm_nxt = _frm;
+#endif
     uint8_t* _to = reinterpret_cast<uint8_t*>(to);
     uint8_t* _to_end = reinterpret_cast<uint8_t*>(to_end);
     uint8_t* _to_nxt = _to;
     result r = utf16_to_utf8(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                              _Maxcode_, _Mode_);
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf8_utf16<wchar_t>::result
 __codecvt_utf8_utf16<wchar_t>::do_in(state_type&,
     const extern_type* frm, const extern_type* frm_end, const extern_type*& frm_nxt,
     intern_type* to, intern_type* to_end, intern_type*& to_nxt) const
 {
     const uint8_t* _frm = reinterpret_cast<const uint8_t*>(frm);
     const uint8_t* _frm_end = reinterpret_cast<const uint8_t*>(frm_end);
     const uint8_t* _frm_nxt = _frm;
+#if defined(_LIBCPP_SHORT_WCHAR)
+    uint16_t* _to = reinterpret_cast<uint16_t*>(to);
+    uint16_t* _to_end = reinterpret_cast<uint16_t*>(to_end);
+    uint16_t* _to_nxt = _to;
+#else
     uint32_t* _to = reinterpret_cast<uint32_t*>(to);
     uint32_t* _to_end = reinterpret_cast<uint32_t*>(to_end);
     uint32_t* _to_nxt = _to;
+#endif
     result r = utf8_to_utf16(_frm, _frm_end, _frm_nxt, _to, _to_end, _to_nxt,
                              _Maxcode_, _Mode_);
     frm_nxt = frm + (_frm_nxt - _frm);
     to_nxt = to + (_to_nxt - _to);
     return r;
 }
 
 __codecvt_utf8_utf16<wchar_t>::result