[1/N] [libcxx] Implement c++2a char8_t input/output of std::filesystem::path

[mstorsjo]

Summary

This implements the std::filesystem parts of P0482 (which is already marked as in progress), and applies the actions that are suggested in P1423.

[ldionne]

Some questions:

• Which part of which paper is this implementing?
• Do we need to update the cxx2a_status page?
• Is there a feature test macro associated to this?

[mstorsjo]

In D90222#2360091, @ldionne wrote:

Some questions:

• Which part of which paper is this implementing?
• Do we need to update the cxx2a_status page?
• Is there a feature test macro associated to this?

This implements bits of P0482; some parts of it seem to be implemented already, and this completes the filesystem related bits, but afaik there's other bits of it left unimplemented as well. It also takes some suggestions from P1423 into account. I guess the cxx2a_status page already could be updated to mark at least P0482 as in progress as things are already.

As far as I can see, that paper adds the feature macro __cpp_lib_char8_t which already is defined, since D55308.

[mstorsjo]

I made this patch the start of the patch series implementing filesystem for windows; as character sets are handled a bit differently on windows (the narrow charset isn't necessarily utf8), and this patch adds new ways of interacting with utf8 in filesystem, I wanted to have this in place first, to get those aspects right from the start - but if needed, the windows patches can be rebased to not rely on this one.

[mstorsjo]

Rebased past conflicts in include/__config.

[mstorsjo]

Rebased on top of D92250.

[curdeius]

It looks pretty good for me, just minor comments.
And please put the relevant papers in the patch description, so that they appear in the commit later.

libcxx/include/filesystem

1236

Unnecessary comment? You should maybe mark P1483 as In Progress.

1240–1244

You should update the message too.

1254–1258

Ditto. Comment about P1423 and message.

libcxx/test/support/filesystem_test_helper.h

433–439

Shouldn't you guard it on __cpp_char8_t instead of using dummy below?

[mstorsjo]

libcxx/include/filesystem

1236

Sure

1240–1244

Oh, good catch, thanks!

libcxx/test/support/filesystem_test_helper.h

433–439

Yeah, it's possible, with something like this:

#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
#define CHAR8_ONLY(x) x,
#else
#define CHAR8_ONLY
#endif
#define MKSTR(Str) {... CHAR8_ONLY(TEST_CONCAT(u8, Str)) TEST_CONCAT(u, Str), ...

The trailing comma is a bit tricky; it has to be included as part of the expansion of CHAR8_ONLY(), but this way it seems like it would work.

If you prefer that form, I can change it that way.

[mstorsjo]

Updated the status of P1423, removed unnecessary comments, added optional references to char8_t in static assert messages, removed a dummy variable in the pre-C++20 version of the test helper, updated the commit message to include references to both papers.

[curdeius]

LGTM. But wait for an approval from a higher instance :).

libcxx/test/support/filesystem_test_helper.h

433–439

I thought of just doing:

#if TEST_STD_VER... 
#define MKSTR(...u8...)
#else 
#define MKSTR(...)
#endif

But that's OK as it is for me. Macros are tricky (or evil).

[ldionne]

libcxx/include/filesystem

1006

That's an ABI break in C++20. Do we think it's reasonable?

I think we at least need to add it to the release notes.

[mstorsjo]

libcxx/include/filesystem

1006

This method is marked with the _LIBCPP_INLINE_VISIBILITY attribute, which afaik is supposed to exclude it from the ABI altogether - exactly for this reason?

The fact that it's a source compat break, and whether to worry about it or not, is acknowledged and discussed in P1423.

[ldionne]

libcxx/include/filesystem

1006

I was trying to come up with examples where a library was already compiled with an inlined version of path::u8string that returns a std::string, and then link against another binary where path::u8string() returns a std::u8string instead. But I couldn't find something that wasn't very tortuous, so I think this is fine from the ABI perspective.

Can we please add a release note because this is a source break?

[mstorsjo]

Added a release note about source incompatibility.

libcxx/docs/Cxx2aStatusPaperStatus.csv

 "`P1004 <https://wg21.link/P1004>`__","LWG","Making std::vector constexpr","Cologne","",""
 "`P1035 <https://wg21.link/P1035>`__","LWG","Input Range Adaptors","Cologne","",""
 "`P1065 <https://wg21.link/P1065>`__","LWG","Constexpr INVOKE","Cologne","",""
 "`P1135 <https://wg21.link/P1135>`__","LWG","The C++20 Synchronization Library","Cologne","|Complete|",""
 "`P1207 <https://wg21.link/P1207>`__","LWG","Movability of Single-pass Iterators","Cologne","",""
 "`P1208 <https://wg21.link/P1208>`__","LWG","Adopt source_location for C++20","Cologne","",""
 "`P1355 <https://wg21.link/P1355>`__","LWG","Exposing a narrow contract for ceil2","Cologne","|Complete|","9.0"
 "`P1361 <https://wg21.link/P1361>`__","LWG","Integration of chrono with text formatting","Cologne","",""
-"`P1423 <https://wg21.link/P1423>`__","LWG","char8_t backward compatibility remediation","Cologne","",""
+"`P1423 <https://wg21.link/P1423>`__","LWG","char8_t backward compatibility remediation","Cologne","|In Progress|",""
 "`P1424 <https://wg21.link/P1424>`__","LWG","'constexpr' feature macro concerns","Cologne","Superseded by `P1902 <https://wg21.link/P1902>`__",""
 "`P1466 <https://wg21.link/P1466>`__","LWG","Miscellaneous minor fixes for chrono","Cologne","",""
 "`P1474 <https://wg21.link/P1474>`__","LWG","Helpful pointers for ContiguousIterator","Cologne","",""
 "`P1502 <https://wg21.link/P1502>`__","LWG","Standard library header units for C++20","Cologne","",""
 "`P1522 <https://wg21.link/P1522>`__","LWG","Iterator Difference Type and Integer Overflow","Cologne","",""
 "`P1523 <https://wg21.link/P1523>`__","LWG","Views and Size Types","Cologne","",""
 "`P1612 <https://wg21.link/P1612>`__","LWG","Relocate Endian's Specification","Cologne","|Complete|","10.0"
 "`P1614 <https://wg21.link/P1614>`__","LWG","The Mothership has Landed","Cologne","|In Progress|",""

libcxx/docs/ReleaseNotes.rst

- By default, libc++ will _not_ include the definition for new and delete,
   they are needed from libc++abi (which would create a circular dependency).
 - During the C++20 standardization process some new low-level bit functions
   have been renamed. Libc++ has renamed these functions to match the C++20
   Standard.
   - ``ispow2`` has been renamed to ``has_single_bit``
   - ``ceil2`` has been renamed to ``bit_ceil``
   - ``floor2`` has been renamed to ``bit_floor``
   - ``log2p1`` has been renamed to ``bit_width``
+
+- In C++20 mode, ``std::filesystem::path::u8string()`` and
+  ``generic_u8string()`` now return ``std::u8string`` according to P0428,
+  while they return ``std::string`` in C++17. This can cause source
+  incompatibility, which is discussed and acknowledged in P1423, but that
+  paper doesn't suggest any remediation for this incompatibility.

libcxx/include/__config

 #endif
 
 #if _LIBCPP_STD_VER > 17
 #  define _LIBCPP_DEPRECATED_IN_CXX20 _LIBCPP_DEPRECATED
 #else
 #  define _LIBCPP_DEPRECATED_IN_CXX20
 #endif
 
+#if !defined(_LIBCPP_NO_HAS_CHAR8_T)
+#  define _LIBCPP_DEPRECATED_WITH_CHAR8_T _LIBCPP_DEPRECATED
+#else
+#  define _LIBCPP_DEPRECATED_WITH_CHAR8_T
+#endif
+
 // Macros to enter and leave a state where deprecation warnings are suppressed.
 #if !defined(_LIBCPP_SUPPRESS_DEPRECATED_PUSH) && \
     (defined(_LIBCPP_COMPILER_CLANG) || defined(_LIBCPP_COMPILER_GCC))
 #  define _LIBCPP_SUPPRESS_DEPRECATED_PUSH \
     _Pragma("GCC diagnostic push") \
     _Pragma("GCC diagnostic ignored \"-Wdeprecated\"")
 #  define _LIBCPP_SUPPRESS_DEPRECATED_POP \
     _Pragma("GCC diagnostic pop")

libcxx/include/filesystem

struct __can_convert_char<char> {
   static const bool value = true;
   using __char_type = char;
 };
 template <>
 struct __can_convert_char<wchar_t> {
   static const bool value = true;
   using __char_type = wchar_t;
 };
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+template <>
+struct __can_convert_char<char8_t> {
+  static const bool value = true;
+  using __char_type = char8_t;
+};
+#endif
 template <>
 struct __can_convert_char<char16_t> {
   static const bool value = true;
   using __char_type = char16_t;
 };
 template <>
 struct __can_convert_char<char32_t> {
   static const bool value = true;
public:
   const string_type& native() const noexcept { return __pn_; }
 
   _LIBCPP_INLINE_VISIBILITY
   const value_type* c_str() const noexcept { return __pn_.c_str(); }
 
   _LIBCPP_INLINE_VISIBILITY operator string_type() const { return __pn_; }
 
   _LIBCPP_INLINE_VISIBILITY _VSTD::string string() const { return __pn_; }
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+  _LIBCPP_INLINE_VISIBILITY _VSTD::u8string u8string() const { return _VSTD::u8string(__pn_.begin(), __pn_.end()); }
+#else
   _LIBCPP_INLINE_VISIBILITY _VSTD::string u8string() const { return __pn_; }
+#endif
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
   template <class _ECharT, class _Traits = char_traits<_ECharT>,
             class _Allocator = allocator<_ECharT> >
   basic_string<_ECharT, _Traits, _Allocator>
   string(const _Allocator& __a = _Allocator()) const {
     using _CVT = __widen_from_utf8<sizeof(_ECharT) * __CHAR_BIT__>;
     using _Str = basic_string<_ECharT, _Traits, _Allocator>;
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
   }
   _LIBCPP_INLINE_VISIBILITY _VSTD::u32string u32string() const {
     return string<char32_t>();
   }
 #endif
 
   // generic format observers
   _VSTD::string generic_string() const { return __pn_; }
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+  _VSTD::u8string generic_u8string() const { return _VSTD::u8string(__pn_.begin(), __pn_.end()); }
+#else
   _VSTD::string generic_u8string() const { return __pn_; }
+#endif
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
   template <class _ECharT, class _Traits = char_traits<_ECharT>,
             class _Allocator = allocator<_ECharT> >
   basic_string<_ECharT, _Traits, _Allocator>
   generic_string(const _Allocator& __a = _Allocator()) const {
     return string<_ECharT, _Traits, _Allocator>(__a);
   }

 inline _LIBCPP_INLINE_VISIBILITY void swap(path& __lhs, path& __rhs) noexcept {
   __lhs.swap(__rhs);
 }
 
 _LIBCPP_FUNC_VIS
 size_t hash_value(const path& __p) noexcept;
 
 template <class _Source>
-_LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     typename enable_if<__is_pathable<_Source>::value, path>::type
     u8path(const _Source& __s) {
   static_assert(
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      is_same<typename __is_pathable<_Source>::__char_type, char8_t>::value ||
+#endif
       is_same<typename __is_pathable<_Source>::__char_type, char>::value,
       "u8path(Source const&) requires Source have a character type of type "
-      "'char'");
+      "'char'"
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      " or 'char8_t'"
+#endif
+      );
   return path(__s);
 }
 
 template <class _InputIt>
-_LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     typename enable_if<__is_pathable<_InputIt>::value, path>::type
     u8path(_InputIt __f, _InputIt __l) {
   static_assert(
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      is_same<typename __is_pathable<_InputIt>::__char_type, char8_t>::value ||
+#endif
       is_same<typename __is_pathable<_InputIt>::__char_type, char>::value,
-      "u8path(Iter, Iter) requires Iter have a value_type of type 'char'");
+      "u8path(Iter, Iter) requires Iter have a value_type of type 'char'"
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      " or 'char8_t'"
+#endif
+      );
   return path(__f, __l);
 }
 
 class _LIBCPP_TYPE_VIS path::iterator {
 public:
   enum _ParserState : unsigned char {
     _Singular,
     _BeforeBegin,

libcxx/test/std/input.output/filesystems/class.path/path.member/path.construct/source.pass.cpp

void test_sfinae() {
   {
     static_assert(!std::is_constructible<path, int*>::value, "");
   }
 }
 
 int main(int, char**) {
   for (auto const& MS : PathList) {
     RunTestCase<char>(MS);
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    RunTestCase<char8_t>(MS);
+#endif
     RunTestCase<wchar_t>(MS);
     RunTestCase<char16_t>(MS);
     RunTestCase<char32_t>(MS);
   }
   test_sfinae();
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.member/path.generic.obs/named_overloads.pass.cpp

int main(int, char**)
   auto const& MS = longString;
   const char* value = longString;
   const path p(value);
   {
     std::string s = p.generic_string();
     assert(s == value);
   }
   {
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    ASSERT_SAME_TYPE(decltype(p.generic_u8string()), std::u8string);
+    std::u8string s = p.generic_u8string();
+    assert(s == (const char8_t*)MS);
+#else
+    ASSERT_SAME_TYPE(decltype(p.generic_u8string()), std::string);
     std::string s = p.generic_u8string();
     assert(s == (const char*)MS);
+#endif
   }
   {
     std::wstring s = p.generic_wstring();
     assert(s == (const wchar_t*)MS);
   }
   {
     std::u16string s = p.generic_u16string();
     assert(s == (const char16_t*)MS);
   }
   {
     std::u32string s = p.generic_u32string();
     assert(s == (const char32_t*)MS);
   }
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.member/path.native.obs/named_overloads.pass.cpp

int main(int, char**)
   auto const& MS = longString;
   const char* value = longString;
   const path p(value);
   {
     std::string s = p.string();
     assert(s == value);
   }
   {
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    ASSERT_SAME_TYPE(decltype(p.u8string()), std::u8string);
+    std::u8string s = p.u8string();
+    assert(s == (const char8_t*)MS);
+#else
+    ASSERT_SAME_TYPE(decltype(p.u8string()), std::string);
     std::string s = p.u8string();
     assert(s == (const char*)MS);
+#endif
   }
   {
     std::wstring s = p.wstring();
     assert(s == (const wchar_t*)MS);
   }
   {
     std::u16string s = p.u16string();
     assert(s == (const char16_t*)MS);
   }
   {
     std::u32string s = p.u32string();
     assert(s == (const char32_t*)MS);
   }
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.nonmember/path.factory.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03
+// ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_DISABLE_DEPRECATION_WARNINGS
 
 // <filesystem>
 
 // template <class Source>
 //    path u8path(Source const&);
 // template <class InputIter>
 //   path u8path(InputIter, InputIter);
 
int main(int, char**)
   {
     path p = fs::u8path(In3);
     assert(p == In1);
   }
   {
     path p = fs::u8path(In3, In3End);
     assert(p == In1);
   }
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t) && defined(_LIBCPP_VERSION)
+  const char8_t* u8In1 = u8"abcd/efg";
+  const std::u8string u8In2(u8In1);
+  const auto u8In3 = u8In2.begin();
+  const auto u8In3End = u8In2.end();
+  // Proposed in P1423, marked tested only for libc++
+  {
+    path p = fs::u8path(u8In1);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In2);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In3);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In3, u8In3End);
+    assert(p == In1);
+  }
+#endif
 
   return 0;
 }

libcxx/test/support/filesystem_test_helper.h

struct CWDGuard {
   }
 
   CWDGuard(CWDGuard const&) = delete;
   CWDGuard& operator=(CWDGuard const&) = delete;
 };
 
 // Misc test types
 
-#define MKSTR(Str) {Str, TEST_CONCAT(L, Str), TEST_CONCAT(u, Str), TEST_CONCAT(U, Str)}
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+#define CHAR8_ONLY(x) x,
+#else
+#define CHAR8_ONLY(x)
+#endif
+
+#define MKSTR(Str) {Str, TEST_CONCAT(L, Str), CHAR8_ONLY(TEST_CONCAT(u8, Str)) TEST_CONCAT(u, Str), TEST_CONCAT(U, Str)}
 
 struct MultiStringType {
   const char* s;
   const wchar_t* w;
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+  const char8_t* u8;
+#endif
   const char16_t* u16;
   const char32_t* u32;
 
   operator const char* () const { return s; }
   operator const wchar_t* () const { return w; }
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+  operator const char8_t* () const { return u8; }
+#endif
   operator const char16_t* () const { return u16; }
   operator const char32_t* () const { return u32; }
 };
 
 const MultiStringType PathList[] = {
         MKSTR(""),
         MKSTR(" "),
         MKSTR("//"),