[libc++] Add missing header <cuchar>

[ldionne]

Summary

https://llvm.org/PR44871

[ldionne]

@curdeius I don't have a clear picture of what's the status of our implementation of P0482, do you?

[curdeius]

In D97870#2600707, @ldionne wrote:

@curdeius I don't have a clear picture of what's the status of our implementation of P0482, do you?

Just a few days ago I started looking at it. I'll try to review the whole status of it and create a patch if needed. It's good that you are adding <cuchar>. That was one of missing parts.

[curdeius]

I see from the build failures that char16_t typedefs should be guarded not to conflict with C++ built-in types. At least for glibc the solution will be to define __USE_ISOCXX11, cf. https://sourceware.org/git/?p=glibc.git;a=blob;f=wcsmbs/uchar.h;h=6020f66cf6d27d86d78213c3efdf17a9e52f45e7;hb=HEAD.

Other thing, with this patch you add cuchar header, but there is no intermediate uchar.h in libc++ as it's done for other headers (which then #include_next their C library equivalent). I guess that it should be added here and it will probably be the place to define the aforementioned macro guard.

Correct me if I'm wrong.

[curdeius]

libcxx/test/std/strings/c.strings/cuchar.pass.cpp

22

Shouldn't this be guarded on support of char8_t as well?

#if TEST_STD_VER > 17 && defined(__cpp_char8_t) ...
// or
#if defined(__cpp_lib_char8_t) && __cpp_lib_char8_t >= 201811L ...

[curdeius]

@ldionne, regarding P0482, I'll send a small patch updating the synopses and adding missing tests very soon.
Apart from that, the missing things are only: <cuchar> (mbrtoc8, c8rtomb) and <memory_resource> (pmr::u8string, tests for hash<pmr::*string>).

[ldionne]

In D97870#2607828, @curdeius wrote:

I see from the build failures that char16_t typedefs should be guarded not to conflict with C++ built-in types. At least for glibc the solution will be to define __USE_ISOCXX11, cf. https://sourceware.org/git/?p=glibc.git;a=blob;f=wcsmbs/uchar.h;h=6020f66cf6d27d86d78213c3efdf17a9e52f45e7;hb=HEAD.

So the root cause of the issue here is that libc++ tries to define its own char16_t type inside __config in C++03 mode (those types were added in C++11). If we didn't try to provide those as extensions in C++03 mode, we wouldn't hit this issue.

Since I don't think it's going to be easy to stop providing those in C++03 mode, instead what I'll do is mark <cuchar> and <uchar.h> as unsupported in C++03 mode. Those headers were added in C++11, so it seems reasonable if they don't work as intended in C++03.

Other thing, with this patch you add cuchar header, but there is no intermediate uchar.h in libc++ as it's done for other headers (which then #include_next their C library equivalent). I guess that it should be added here and it will probably be the place to define the aforementioned macro guard.

I added uchar.h, thanks for the catch. TBH, I'm not a big fan of defining headers that should be provided by the underlying C library, however we do it for all other libc++ headers, so let's do it for consistency if nothing else.

[ldionne]

Rebase onto main, implement review feedback, add uchar.h and fix tests.

[ldionne]

Mark test as UNSUPPORTED in C++03 and fail clearly when <cuchar> is included in C++03 mode

[ldionne]

I did something that we usually don't do here - I made the <cuchar> and <uchar.h> headers fail with an explicit error when included in C++03 mode. The rationale for doing that is that the headers won't work in C++03 mode (they require the char16_t type, which is in C++11), and I thought getting a clear error was better than getting a cryptic error about missing char16_t (or even worse, clashing with our broken attempt to provide char16_t in C++03 mode in <__config>).

Another approach would have been to render the header empty when used in C++03 mode. This is what we do elsewhere. However, I quite dislike this as a general approach because it hides the fact that the header shouldn't be used pre C++11. Users could rightfully file bugs against us for providing an empty <cuchar> header in C++03 mode. Instead, I thought providing an error message was more helpful and more in line with the "fail fast and in obvious ways" line of thinking.

[• Quuxplusone]

I did something that we usually don't do here - I made the <cuchar> and <uchar.h> headers fail with an explicit error when included in C++03 mode. The rationale for doing that is that the headers won't work in C++03 mode (they require the char16_t type, which is in C++11)

In the spirit of my "remove _LIBCPP_CXX03_LANG" patch series: We care about compiling in C++03 mode only with Clang, right? So how about doing some macro magic in <__config> that makes our char16_t an alias for Clang's uglified built-in __char16_t on-Clang-in-C++03-mode? And then on every other compiler we just use char16_t directly.

[ldionne]

In D97870#2859714, @Quuxplusone wrote:

In the spirit of my "remove _LIBCPP_CXX03_LANG" patch series: We care about compiling in C++03 mode only with Clang, right? So how about doing some macro magic in <__config> that makes our char16_t an alias for Clang's uglified built-in __char16_t on-Clang-in-C++03-mode? And then on every other compiler we just use char16_t directly.

We already do, and it's broken! https://github.com/llvm/llvm-project/blob/main/libcxx/include/__config#L441

It's broken because in some cases, even in C++03 mode, the underlying C library tries to do the exact same thing and provide a typedef for char16_t. We end up trying to redefine that typedef in <__config>, and that doesn't work. I think that what we want to do here is instead not try to backport <uchar.h> to C++03 (as I do in the patch right now).

[• Quuxplusone]

We already do, and it's broken! https://github.com/llvm/llvm-project/blob/main/libcxx/include/__config#L441

Well, it's definitely broken, because we do it again later on:
https://github.com/llvm/llvm-project/blob/main/libcxx/include/__config#L825
(er, I guess redundant typedef aliases actually became legal in C++17 or thereabouts? okay...)

I think the original sin there is that <__config> is trying to define typedefs in the global namespace. What if we defined char16_t etc. inside namespace _VSTD instead, so that they didn't conflict with the C library's aliases in the global namespace?

I mean, you've looked at this more than I have, and I'm not trying to block this if you just want to get it in; but it sure would be nice not to have <cuchar> be the one special header you can't include in C++03 mode, so I'm trying to suggest ways to get around that.

[ldionne]

Rebase onto main and define the header like we do for other headers (i.e. don't hard-fail).

[• Quuxplusone]

Continues to look reasonable to me, mod comment about consistency with cwctype.

libcxx/include/cuchar

47–57

Looking at how libc++ does its cwctype and wctype.h, I suspect you ought to do something isomorphic over here and wrap all this in #if defined(_LIBCPP_INCLUDED_C_LIBRARY_UCHAR_H). (Or else explain in the commit summary why uchar.h's situation is different.)

libcxx/test/std/depr/depr.c.headers/uchar_h.compile.pass.cpp

28

Weird mix of 0-to-mean-nullptr and {}-to-mean-zero, but okay. ;)

[ldionne]

Address comments.

libcxx/include/cuchar

47–57

Excellent point. I think we have the same problem, however the only declaration that can cause problems is size_t. So instead, if the platform doesn't provide <uchar.h>, I'll include <stddef.h> so we get the definition of size_t` and we don't cause problems when we do using ::size_t _LIBCPP_USING_IF_EXISTS;.

[ldionne]

Fix pragma indentation

[ldionne]

Disable <uchar.h> in C++03 mode.

[ldionne]

Rebase and mark a few tests as XFAIL on AIX. CC @daltenty

I'll ship this if CI is green.

libcxx/include/CMakeLists.txt

set(files
   cstddef
   cstdint
   cstdio
   cstdlib
   cstring
   ctgmath
   ctime
   ctype.h
+  cuchar
   cwchar
   cwctype
   deque
   errno.h
   exception
   execution
   experimental/__config
   experimental/__memory
set(files
   strstream
   system_error
   tgmath.h
   thread
   tuple
   type_traits
   typeindex
   typeinfo
+  uchar.h
   unordered_map
   unordered_set
   utility
   valarray
   variant
   vector
   version
   wchar.h

libcxx/include/cuchar

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_CUCHAR
+#define _LIBCPP_CUCHAR
+
+/*
+    cuchar synopsis // since C++11
+
+Macros:
+
+    __STDC_UTF_16__
+    __STDC_UTF_32__
+
+namespace std {
+
+Types:
+
+  mbstate_t
+  size_t
+
+size_t mbrtoc8(char8_t* pc8, const char* s, size_t n, mbstate_t* ps); // since C++20
+size_t c8rtomb(char* s, char8_t c8, mbstate_t* ps);                   // since C++20
+size_t mbrtoc16(char16_t* pc16, const char* s, size_t n, mbstate_t* ps);
+size_t c16rtomb(char* s, char16_t c16, mbstate_t* ps);
+size_t mbrtoc32(char32_t* pc32, const char* s, size_t n, mbstate_t* ps);
+size_t c32rtomb(char* s, char32_t c32, mbstate_t* ps);
+
+} // std
+
+*/
+
+#include <__config>
+#include <uchar.h>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#pragma GCC system_header
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+using ::mbstate_t _LIBCPP_USING_IF_EXISTS;
+using ::size_t _LIBCPP_USING_IF_EXISTS;
+
+using ::mbrtoc16 _LIBCPP_USING_IF_EXISTS;
+using ::c16rtomb _LIBCPP_USING_IF_EXISTS;
+using ::mbrtoc32 _LIBCPP_USING_IF_EXISTS;
+using ::c32rtomb _LIBCPP_USING_IF_EXISTS;
+_LIBCPP_END_NAMESPACE_STD
+
+#endif  // _LIBCPP_CUCHAR

libcxx/include/module.modulemap

module stdlib_h {
       // <stdlib.h>'s __need_* macros require textual inclusion.
       textual header "stdlib.h"
       export *
     }
     module string_h {
       header "string.h"
       export *
     }
-    // FIXME: <uchar.h> is missing.
+    module uchar_h {
+      header "uchar.h"
+      export *
+    }
     // <time.h> provided by C library.
     module wchar_h {
       // <wchar.h>'s __need_* macros require textual inclusion.
       textual header "wchar.h"
       export *
     }
     module wctype_h {
       header "wctype.h"
module ctgmath {
       export ccomplex
       export cmath
       export *
     }
     module ctime {
       header "ctime"
       export *
     }
-    // FIXME: <cuchar> is missing.
+    module cuchar {
+      header "cuchar"
+      export *
+    }
     module cwchar {
       header "cwchar"
       export depr.stdio_h
       export *
     }
     module cwctype {
       header "cwctype"
       export *

libcxx/include/uchar.h

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_UCHAR_H
+#define _LIBCPP_UCHAR_H
+
+/*
+    uchar.h synopsis // since C++11
+
+Macros:
+
+    __STDC_UTF_16__
+    __STDC_UTF_32__
+
+Types:
+
+  mbstate_t
+  size_t
+
+size_t mbrtoc16(char16_t* pc16, const char* s, size_t n, mbstate_t* ps);
+size_t c16rtomb(char* s, char16_t c16, mbstate_t* ps);
+size_t mbrtoc32(char32_t* pc32, const char* s, size_t n, mbstate_t* ps);
+size_t c32rtomb(char* s, char32_t c32, mbstate_t* ps);
+
+*/
+
+#include <__config>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#pragma GCC system_header
+#endif
+
+#include_next <uchar.h>
+
+#endif // _LIBCPP_UCHAR_H

libcxx/test/libcxx/double_include.sh.cpp

 #include <cstddef>
 #include <cstdint>
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
 #include <ctgmath>
 #include <ctime>
 #include <ctype.h>
+#if __cplusplus >= 201103L
+#    include <cuchar>
+#endif
 #include <cwchar>
 #include <cwctype>
 #include <deque>
 #include <errno.h>
 #include <exception>
 #include <execution>
 #include <fenv.h>
 #ifndef _LIBCPP_HAS_NO_FILESYSTEM_LIBRARY

 #include <tgmath.h>
 #ifndef _LIBCPP_HAS_NO_THREADS
 #    include <thread>
 #endif
 #include <tuple>
 #include <type_traits>
 #include <typeindex>
 #include <typeinfo>
+#if __cplusplus >= 201103L
+#    include <uchar.h>
+#endif
 #include <unordered_map>
 #include <unordered_set>
 #include <utility>
 #include <valarray>
 #include <variant>
 #include <vector>
 #include <version>
 #include <wchar.h>

libcxx/test/libcxx/min_max_macros.compile.pass.cpp

 #include <cstring>
 TEST_MACROS();
 #include <ctgmath>
 TEST_MACROS();
 #include <ctime>
 TEST_MACROS();
 #include <ctype.h>
 TEST_MACROS();
+#if __cplusplus >= 201103L
+#    include <cuchar>
+TEST_MACROS();
+#endif
 #include <cwchar>
 TEST_MACROS();
 #include <cwctype>
 TEST_MACROS();
 #include <deque>
 TEST_MACROS();
 #include <errno.h>
 TEST_MACROS();

 #include <tuple>
 TEST_MACROS();
 #include <type_traits>
 TEST_MACROS();
 #include <typeindex>
 TEST_MACROS();
 #include <typeinfo>
 TEST_MACROS();
+#if __cplusplus >= 201103L
+#    include <uchar.h>
+TEST_MACROS();
+#endif
 #include <unordered_map>
 TEST_MACROS();
 #include <unordered_set>
 TEST_MACROS();
 #include <utility>
 TEST_MACROS();
 #include <valarray>
 TEST_MACROS();

libcxx/test/libcxx/no_assert_include.compile.pass.cpp

 #include <cstddef>
 #include <cstdint>
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
 #include <ctgmath>
 #include <ctime>
 #include <ctype.h>
+#if __cplusplus >= 201103L
+#    include <cuchar>
+#endif
 #include <cwchar>
 #include <cwctype>
 #include <deque>
 #include <errno.h>
 #include <exception>
 #include <execution>
 #include <fenv.h>
 #ifndef _LIBCPP_HAS_NO_FILESYSTEM_LIBRARY

 #include <tgmath.h>
 #ifndef _LIBCPP_HAS_NO_THREADS
 #    include <thread>
 #endif
 #include <tuple>
 #include <type_traits>
 #include <typeindex>
 #include <typeinfo>
+#if __cplusplus >= 201103L
+#    include <uchar.h>
+#endif
 #include <unordered_map>
 #include <unordered_set>
 #include <utility>
 #include <valarray>
 #include <variant>
 #include <vector>
 #include <version>
 #include <wchar.h>

libcxx/test/libcxx/strings/c.strings/version_cuchar.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
-//
-// XFAIL: *
-
-// Skip this test on windows. If built on top of the MSVC runtime, the
-// <cuchar> header actually does exist (although not provided by us).
-// This should be removed once D97870 has landed.
-// UNSUPPORTED: windows
 
 // <cuchar>
 
 #include <cuchar>
 
 #include "test_macros.h"
 
 #ifndef _LIBCPP_VERSION
 #error _LIBCPP_VERSION not defined
 #endif
 
 int main(int, char**)
 {
 
   return 0;
 }

libcxx/test/std/depr/depr.c.headers/uchar_h.compile.pass.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03
+
+// XFAIL: suse-linux-enterprise-server-11
+// XFAIL: netbsd
+
+// The macOS SDK doesn't contain <uchar.h> yet, so these tests fail.
+// XFAIL: darwin
+
+// <uchar.h>
+
+#include <uchar.h>
+
+#include "test_macros.h"
+
+// __STDC_UTF_16__ may or may not be defined by the C standard library
+// __STDC_UTF_32__ may or may not be defined by the C standard library
+
+ASSERT_SAME_TYPE(size_t, decltype(mbrtoc16((char16_t*)0, (const char*)0, size_t{}, (mbstate_t*)0)));
+ASSERT_SAME_TYPE(size_t, decltype(c16rtomb((char*)0, char16_t{}, (mbstate_t*)0)));
+
+ASSERT_SAME_TYPE(size_t, decltype(mbrtoc32((char32_t*)0, (const char*)0, size_t{}, (mbstate_t*)0)));
+ASSERT_SAME_TYPE(size_t, decltype(c16rtomb((char*)0, char32_t{}, (mbstate_t*)0)));

libcxx/test/std/depr/depr.c.headers/uchar_h.pass.cpp

-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// XFAIL: suse-linux-enterprise-server-11
-// XFAIL: darwin
-// XFAIL: netbsd
-
-// <uchar.h>
-
-#include <uchar.h>
-
-int main(int, char**)
-{
-
-  return 0;
-}

libcxx/test/std/strings/c.strings/cuchar.compile.pass.cpp

+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// UNSUPPORTED: c++03
+
+// The macOS SDK doesn't contain <uchar.h> yet, so these tests fail.
+// XFAIL: darwin
+
+// <cuchar>
+
+#include <cuchar>
+
+#include "test_macros.h"
+
+// __STDC_UTF_16__ may or may not be defined by the C standard library
+// __STDC_UTF_32__ may or may not be defined by the C standard library
+
+#if defined(__cpp_lib_char8_t) && __cpp_lib_char8_t >= 201811L
+#if !defined(_LIBCPP_VERSION) // TODO: implement this in libc++
+ASSERT_SAME_TYPE(size_t, decltype(std::mbrtoc8((char8_t*)0, (char const*)0, size_t(), (mbstate_t*)0)));
+ASSERT_SAME_TYPE(size_t, decltype(std::c8rtomb((char*)0, char8_t{}, (mbstate_t*)0)));
+#endif
+#endif
+
+ASSERT_SAME_TYPE(size_t, decltype(std::mbrtoc16((char16_t*)0, (const char*)0, size_t{}, (mbstate_t*)0)));
+ASSERT_SAME_TYPE(size_t, decltype(std::c16rtomb((char*)0, char16_t{}, (mbstate_t*)0)));
+
+ASSERT_SAME_TYPE(size_t, decltype(std::mbrtoc32((char32_t*)0, (const char*)0, size_t{}, (mbstate_t*)0)));
+ASSERT_SAME_TYPE(size_t, decltype(std::c16rtomb((char*)0, char32_t{}, (mbstate_t*)0)));

libcxx/test/std/strings/c.strings/cuchar.pass.cpp

-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// XFAIL: libc++
-
-// Skip this test on windows. If built on top of the MSVC runtime, the
-// <cuchar> header actually does exist (although not provided by us).
-// This should be removed once D97870 has landed.
-// UNSUPPORTED: windows
-
-// <cuchar>
-
-#include <cuchar>
-
-#include "test_macros.h"
-
-int main(int, char**)
-{
-
-  return 0;
-}

libcxx/utils/generate_header_inclusion_tests.py

mandatory_inclusions = {
     "vector": ["compare", "initializer_list"],
 }
 
 new_in_version = {
     "chrono": "11",
     "compare": "20",
     "concepts": "20",
     "coroutine": "20",
+    "cuchar": "11",
     "filesystem": "17",
     "initializer_list": "11",
     "optional": "17",
     "ranges": "20",
     "string_view": "17",
     "syncstream": "20",
     "system_error": "11",
     "thread": "11",
     "tuple": "11",
+    "uchar.h": "11",
     "unordered_map": "11",
     "unordered_set": "11",
     "variant": "17",
 }
 
 assert all(v == sorted(v) for k, v in mandatory_inclusions.items())
 
 # Map from each header to the Lit annotations that should be used for

libcxx/utils/generate_header_tests.py

header_markup = {
     "ostream": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
     "regex": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
     "sstream": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
     "streambuf": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
     "strstream": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
 
     "experimental/coroutine": ["if defined(__cpp_coroutines)"],
     "experimental/regex": ["ifndef _LIBCPP_HAS_NO_LOCALIZATION"],
+
+    "cuchar": ["if __cplusplus >= 201103L"],
+    "uchar.h": ["if __cplusplus >= 201103L"],
 }
 
 allowed_extensions = ['', '.h']
 indent_width = 4
 
 
 begin_pattern = """\
 ////////////////////////////////////////////////////////////////////////////////