[libc++][format] Adds a UTF transcoder.

[Mordante]

Summary

This is a preparation for

P2093R14 Formatted output

When the output of print is to the terminal it needs to use the native
API. This means transcoding UTF-8 to UTF-16 on Windows. The encoder's
interface is modeled after

P2728 Unicode in the Library, Part 1: UTF Transcoding

But only the required part for P2093R14 is implemented.

On Windows wchar_t is 16 bits, in order to test on platforms where
wchar_t is 32 bits the transcoder has support for char16_t. It also adds
and UTF-8 to UTF-32 encoder which is useful for other tests.

Note it is possible to use <codecvt> for transcoding, but that header is
deprecated. So rather write new code that is not deprecated; the hard
part, decoding, has already been done. The <codecvt> header also
requires locale support while the new code works without including
<locale>.

Note the current transcoder implementation can be optimized since it
basically does UTF-8 -> UTF-32 -> UTF-16. The first goal is to have a
working implementation. Since it's not part of the ABI it's possible to
do the optimization later.

Depends on D149672

[Mordante]

CI fixes.

[Mordante]

CI fixes.

[Mordante]

CI fixes.

[Mordante]

Rebased and adjusted to upstream changes.

[tahonermann]

The approach looks pretty good though I don't love the encoding inference based on sizeof(). I see the motivation though. I'll think about this some more, but I think we should just infer the encoding based on the character type; char16_t => UTF-16, char32_t => UTF-32, wchar_t => platform dependent.

libcxx/include/print

45–49

The underlying type of char16_t is uint_least16_t, so the sizeof constraint is technically incorrect here. Similarly, a size of 2 for wchar_t doesn't imply UTF-16 if CHAR_BIT is 16 (but that is only the case for a few embedded platforms).

We'll need to see how P2728R0 eventually shakes out, but I think any inference based on character type size should be avoided in favor of an approach that states the desired encoding; e.g., __encode_utf16(...). Constraints could then be added to ensure that the element size is at least 16-bit (sizeof(T) * CHAR_BIT).

52

72

79

basic_string_view or span? I would expect more like the latter.

82–84

[Mordante]

Thanks for the review!

libcxx/include/print

45–49

Ah I totally forgot about char16_t not being required to be 16 bits. I see P2728 uses the same assumption, I'll drop Zach a note.

For now I just hard-code it for char, charX_t and use the _LIBCPP_SHORT_WCHAR define in libc++ for wchar_t. This macro is based on the platform, for Windows and AIX 32-bit and it's defined and for others not. Once P2728 lands I will switch to its definitions.

79

The __code_point_view was written for grapheme clustering in <format>. For string formatters I use a basic_string_view as format argument, so using a basic_string_view sounds natural. Since P2728R0 aims at non-character types as code units it would indeed make sense to use a span. Especially since char_traits<int> will be removed in LLVM 18. For now I keep it this way, once P2728R0 accepted I can change it. Then I probably want to model things closer to that paper.

[Mordante]

Addresses review comments.

[Mordante]

Rebased so D150044 can be rebased.
Fixed merge conflicts.
Added header to the modules.

[Mordante]

Rebased to fix CI.

[Mordante]

Rebased to test D150044

[Mordante]

@tahonermann, it would be great if you can have another look.

[tahonermann]

Sorry for the delay, again. I think this is looking good. I noticed one issue that I think needs to be addressed and otherwise just want to confirm that lack of support for encoding UTF-8 is intentional for now.

libcxx/include/print

40–44

As far as I can tell, the __utf8_code_unit concept is not used, nor is there a UTF-8 overload of __encode(). That is ok if intentional, I'm just mentioning it in case it isn't.

The use of char8_t should be predicated on _LIBCPP_HAS_NO_CHAR8_T not being defined since char8_t support can be disabled with -fno-char8_t.

[Mordante]

In D150031#4445220, @tahonermann wrote:

Sorry for the delay, again. I think this is looking good. I noticed one issue that I think needs to be addressed and otherwise just want to confirm that lack of support for encoding UTF-8 is intentional for now.

No problem, thanks for the review!

libcxx/include/print

40–44

Nice catch. It's indeed not used at the moment. I'll remove it for now. When Zach's paper is approved I will add the concepts based on the wording of the paper.

[Mordante]

Addresses review comment.

[tahonermann]

Looks good to me.

[ldionne]

libcxx/include/print

2

Are we going to have a lot of stuff inside <print>? If so, it would probably make sense to introduce this utility under __print/encode.h instead.

45

Can we use #ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS instead? I'd like to reduce usage of _LIBCPP_IF_WIDE_CHARACTERS and eventually remove it since it's a really unusual macro.

libcxx/test/libcxx/input.output/iostream.format/print.fun/transcoding.pass.cpp

10–11

Does this really time out on GCC 12? What's special about this test that makes it timeout?

19–23

Those don't seem sorted :)

[Mordante]

Thanks for the reviews!

libcxx/include/print

2

No print is small, 2 more overloads and it's complete. I want to move the encoding part to a new header once P2728 is voted in. Then the code becomes part of that header.

45

To be honest I feel the macro is easier to read. I don't feel too strongly about it so I change it.

libcxx/test/libcxx/input.output/iostream.format/print.fun/transcoding.pass.cpp

10–11

Actually it's the same GCC12 issue as the other format parts. This patch was written before we started to use // UNSUPPORTED: GCC-ALWAYS_INLINE-FIXME so I copied it from some format header. I've changed it to the GCC-ALWAYS_INLINE-FIXME

19–23

I trust on clang-format ;-) Fixed it.

[Mordante]

Rebased and addresses review comments.

[ldionne]

LGTM, thanks @tahonermann for reviewing the hard parts :)

libcxx/include/CMakeLists.txt

set(files
   memory
   memory_resource
   mutex
   new
   numbers
   numeric
   optional
   ostream
+  print
   queue
   random
   ranges
   ratio
   regex
   scoped_allocator
   semaphore
   set

libcxx/include/module.modulemap.in

module optional {
     export *
   }
   module ostream {
     @requires_LIBCXX_ENABLE_LOCALIZATION@
     header "ostream"
     // FIXME: should re-export ios, streambuf?
     export *
   }
+  module print {
+    header "print"
+    export *
+  }
   module queue {
     header "queue"
     export *
   }
   module random {
     header "random"
     export *
 

libcxx/include/print

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_PRINT
+#define _LIBCPP_PRINT
+
+#include <__assert> // all public C++ headers provide the assertion handler
+#include <__concepts/same_as.h>
+#include <__config>
+#include <__format/unicode.h>
+#include <string_view>
+#include <version>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if _LIBCPP_STD_VER >= 23
+
+#  ifndef _LIBCPP_HAS_NO_UNICODE
+// This is the code to transcode UTF-8 to UTF-16. This is used on
+// Windows for the native Unicode API. The code is modeled to make it
+// easier to extend to
+//
+//  P2728R0 Unicode in the Library, Part 1: UTF Transcoding
+//
+// This paper is still under heavy development so it makes no sense yet
+// to strictly follow the paper.
+namespace __unicode {
+
+// The names of these concepts are modelled after P2728R0, but the
+// implementation is not. char16_t may contain 32-bits so depending on the
+// number of bits is an issue.
+#    ifdef _LIBCPP_SHORT_WCHAR
+template <class _Tp>
+concept __utf16_code_unit =
+    same_as<_Tp, char16_t>
+#      ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
+    || same_as<_Tp, wchar_t>
+#      endif
+    ;
+template <class _Tp>
+concept __utf32_code_unit = same_as<_Tp, char32_t>;
+#    else // _LIBCPP_SHORT_WCHAR
+template <class _Tp>
+concept __utf16_code_unit = same_as<_Tp, char16_t>;
+template <class _Tp>
+concept __utf32_code_unit =
+    same_as<_Tp, char32_t>
+#      ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
+    || same_as<_Tp, wchar_t>
+#      endif
+    ;
+#    endif // _LIBCPP_SHORT_WCHAR
+
+// Pass by reference since an output_iterator may not be copyable.
+template <class _OutIt>
+_LIBCPP_HIDE_FROM_ABI constexpr void __encode(_OutIt&, char32_t) = delete;
+
+template <class _OutIt>
+  requires __utf16_code_unit<iter_value_t<_OutIt>>
+_LIBCPP_HIDE_FROM_ABI constexpr void __encode(_OutIt& __out_it, char32_t __value) {
+  _LIBCPP_ASSERT(__is_scalar_value(__value), "an invalid unicode scalar value results in invalid UTF-16");
+
+  if (__value < 0x10000) {
+    *__out_it++ = __value;
+    return;
+  }
+
+  __value -= 0x10000;
+  *__out_it++ = 0xd800 + (__value >> 10);
+  *__out_it++ = 0xdc00 + (__value & 0x3FF);
+}
+
+template <class _OutIt>
+  requires __utf32_code_unit<iter_value_t<_OutIt>>
+_LIBCPP_HIDE_FROM_ABI constexpr void __encode(_OutIt& __out_it, char32_t __value) {
+  _LIBCPP_ASSERT(__is_scalar_value(__value), "an invalid unicode scalar value results in invalid UTF-32");
+  *__out_it++ = __value;
+}
+
+template <class _OutIt, input_iterator _InIt>
+  requires output_iterator<_OutIt, const iter_value_t<_OutIt>&> && (!same_as<iter_value_t<_OutIt>, iter_value_t<_InIt>>)
+_LIBCPP_HIDE_FROM_ABI constexpr _OutIt __transcode(_InIt __first, _InIt __last, _OutIt __out_it) {
+  // The __code_point_view has a basic_string_view interface.
+  // When transcoding becomes part of the standard we probably want to
+  // look at smarter algorithms.
+  // For example, when processing a code point that is encoded in
+  // 1 to 3 code units in UTF-8, the result will always be encoded
+  // in 1 code unit in UTF-16 (code points that require 4 code
+  // units in UTF-8 will require 2 code units in UTF-16).
+  //
+  // Note if P2728 is accepted types like int may become valid. In that case
+  // the __code_point_view should use a span. Libc++ will remove support for
+  // char_traits<int>.
+  basic_string_view<iter_value_t<_InIt>> __data{__first, __last};
+  __code_point_view<iter_value_t<_InIt>> __view{__data.begin(), __data.end()};
+  while (!__view.__at_end())
+    __unicode::__encode(__out_it, __view.__consume().__code_point);
+  return __out_it;
+}
+
+} // namespace __unicode
+
+#  endif //  _LIBCPP_HAS_NO_UNICODE
+
+#endif // _LIBCPP_STD_VER >= 23
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP_PRINT

libcxx/modules/std/print.cppm

 // -*- C++ -*-
 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 module;
-#if __has_include(<print>)
-#  error "include this header unconditionally and uncomment the exported symbols"
-#  include <print>
-#endif
+#include <print>
 
 export module std:print;
 export namespace std {
 #if 0
   // [print.fun], print functions
   using std::print;
   using std::println;
 
   using std::vprint_nonunicode;
   using std::vprint_unicode;
 #endif
 } // namespace std

libcxx/test/libcxx/input.output/iostream.format/print.fun/transcoding.pass.cpp

+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03, c++11, c++14, c++17, c++20
+// UNSUPPORTED: GCC-ALWAYS_INLINE-FIXME
+
+// <print>
+
+// Tests the UTF-8 to UTF-16/32 encoding.
+// UTF-16 is used on Windows to write to the Unicode API.
+// UTF-32 is used to test the Windows behaviour on Linux using 32-bit wchar_t.
+
+#include <algorithm>
+#include <array>
+#include <cassert>
+#include <print>
+#include <string_view>
+
+#include "test_macros.h"
+#include "make_string.h"
+
+#define SV(S) MAKE_STRING_VIEW(CharT, S)
+
+template <class CharT>
+constexpr void test(std::basic_string_view<CharT> expected, std::string_view input) {
+  assert(expected.size() < 1024);
+  std::array<CharT, 1024> buffer;
+  std::ranges::fill(buffer, CharT('*'));
+
+  CharT* out = std::__unicode::__transcode(input.begin(), input.end(), buffer.data());
+
+  assert(std::basic_string_view<CharT>(buffer.data(), out) == expected);
+
+  out = std::find_if(out, buffer.end(), [](CharT c) { return c != CharT('*'); });
+  assert(out == buffer.end());
+}
+
+template <class CharT>
+constexpr void test() {
+  // *** Test valid UTF-8 ***
+#define TEST(S) test(SV(S), S)
+  TEST("hello world");
+  // copied from benchmarks/std_format_spec_string_unicode.bench.cpp
+  TEST("Lorem ipsum dolor sit amet, ne sensibus evertitur aliquando his. Iuvaret fabulas qui ex.");
+  TEST("Lōrem ipsūm dolor sīt æmeÞ, ea vel nostrud feuġǣit, muciūs tēmporiȝusrefērrēnÞur no mel.");
+  TEST("Лорем ипсум долор сит амет, еу диам тамяуам принципес вис, еяуидем цонцептам диспутандо");
+  TEST("入ト年媛ろ舗学ラロ準募ケカ社金スノ屋検れう策他セヲシ引口ぎ集7独ぱクふ出車ぽでぱ円輪ルノ受打わ。");
+  TEST("\U0001f636\u200d\U0001f32b\ufe0f");
+#undef TEST
+
+  // *** Test invalid UTF-8 ***
+  test(SV("\ufffd"), "\xc3");
+  test(SV("\ufffd("), "\xc3\x28");
+
+  // Surrogate range
+  test(SV("\ufffd"), "\xed\xa0\x80"); // U+D800
+  test(SV("\ufffd"), "\xed\xaf\xbf"); // U+DBFF
+  test(SV("\ufffd"), "\xed\xbf\x80"); // U+DC00
+  test(SV("\ufffd"), "\xed\xbf\xbf"); // U+DFFF
+
+  // Beyond valid values
+  test(SV("\ufffd"), "\xf4\x90\x80\x80"); // U+110000
+  test(SV("\ufffd"), "\xf4\xbf\xbf\xbf"); // U+11FFFF
+
+  // Validates http://unicode.org/review/pr-121.html option 3.
+  test(SV("\u0061\uFFFD\uFFFD\uFFFD\uFFFD\uFFFD\uFFFD\u0062"), "\x61\xF1\x80\x80\xE1\x80\xC2\x62");
+}
+
+constexpr bool test() {
+  test<char16_t>();
+  test<char32_t>();
+#if !defined(TEST_HAS_NO_WIDE_CHARACTERS)
+  test<wchar_t>();
+#endif
+  return true;
+}
+
+int main(int, char**) {
+  test();
+  static_assert(test());
+
+  return 0;
+}

libcxx/test/libcxx/transitive_includes/cxx03.csv

 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream type_traits
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue concepts
 queue cstddef
 queue cstdlib
 queue deque
 queue functional
 queue initializer_list
 queue limits

libcxx/test/libcxx/transitive_includes/cxx11.csv

 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream type_traits
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue concepts
 queue cstddef
 queue cstdlib
 queue deque
 queue functional
 queue initializer_list
 queue limits

libcxx/test/libcxx/transitive_includes/cxx14.csv

 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream type_traits
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue concepts
 queue cstddef
 queue cstdlib
 queue deque
 queue functional
 queue initializer_list
 queue limits

libcxx/test/libcxx/transitive_includes/cxx17.csv

 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream type_traits
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue concepts
 queue cstddef
 queue cstdlib
 queue deque
 queue functional
 queue initializer_list
 queue limits

libcxx/test/libcxx/transitive_includes/cxx20.csv

 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream type_traits
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue concepts
 queue cstddef
 queue cstdlib
 queue deque
 queue functional
 queue initializer_list
 queue limits

libcxx/test/libcxx/transitive_includes/cxx23.csv

 ostream limits
 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue cstddef
 queue cstdint
 queue deque
 queue initializer_list
 queue limits
 queue new
 queue vector

libcxx/test/libcxx/transitive_includes/cxx26.csv

 ostream limits
 ostream locale
 ostream new
 ostream stdexcept
 ostream streambuf
 ostream string
 ostream typeinfo
 ostream version
+print cstddef
+print cstdint
+print initializer_list
+print limits
+print string_view
+print version
 queue compare
 queue cstddef
 queue cstdint
 queue deque
 queue initializer_list
 queue limits
 queue new
 queue vector

libcxx/utils/ci/run-buildbot

! grep -rn '[^ -~]' libcxx/include libcxx/src libcxx/test libcxx/benchmarks \
            --exclude 'fill.unicode.pass.cpp' \
            --exclude 'format_tests.h' \
            --exclude 'format.functions.tests.h' \
            --exclude 'formatter.*.pass.cpp' \
            --exclude 'grep.pass.cpp' \
            --exclude 'locale-specific_form.pass.cpp' \
            --exclude 'ostream.pass.cpp' \
            --exclude 'std_format_spec_string_unicode.bench.cpp' \
+           --exclude 'transcoding.pass.cpp' \
            --exclude 'underflow.pass.cpp' \
            || false
 
     # Reject code with trailing whitespace
     ! grep -rn '[[:blank:]]$' libcxx/include libcxx/src libcxx/test libcxx/benchmarks || false
 ;;
 #
 # Various Standard modes