[1/N] [libcxx] Implement c++2a char8_t input/output of std::filesystem::path

[mstorsjo]

Summary

This implements the std::filesystem parts of P0482 (which is already marked as in progress), and applies the actions that are suggested in P1423.

[ldionne]

Some questions:

• Which part of which paper is this implementing?
• Do we need to update the cxx2a_status page?
• Is there a feature test macro associated to this?

[mstorsjo]

In D90222#2360091, @ldionne wrote:

Some questions:

• Which part of which paper is this implementing?
• Do we need to update the cxx2a_status page?
• Is there a feature test macro associated to this?

This implements bits of P0482; some parts of it seem to be implemented already, and this completes the filesystem related bits, but afaik there's other bits of it left unimplemented as well. It also takes some suggestions from P1423 into account. I guess the cxx2a_status page already could be updated to mark at least P0482 as in progress as things are already.

As far as I can see, that paper adds the feature macro __cpp_lib_char8_t which already is defined, since D55308.

[mstorsjo]

I made this patch the start of the patch series implementing filesystem for windows; as character sets are handled a bit differently on windows (the narrow charset isn't necessarily utf8), and this patch adds new ways of interacting with utf8 in filesystem, I wanted to have this in place first, to get those aspects right from the start - but if needed, the windows patches can be rebased to not rely on this one.

[mstorsjo]

Rebased past conflicts in include/__config.

[mstorsjo]

Rebased on top of D92250.

[curdeius]

It looks pretty good for me, just minor comments.
And please put the relevant papers in the patch description, so that they appear in the commit later.

libcxx/include/filesystem

1217

Unnecessary comment? You should maybe mark P1483 as In Progress.

1222

You should update the message too.

1232–1236

Ditto. Comment about P1423 and message.

libcxx/test/support/filesystem_test_helper.h

421

Shouldn't you guard it on __cpp_char8_t instead of using dummy below?

[mstorsjo]

libcxx/include/filesystem

1217

Sure

1222

Oh, good catch, thanks!

libcxx/test/support/filesystem_test_helper.h

421

Yeah, it's possible, with something like this:

#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
#define CHAR8_ONLY(x) x,
#else
#define CHAR8_ONLY
#endif
#define MKSTR(Str) {... CHAR8_ONLY(TEST_CONCAT(u8, Str)) TEST_CONCAT(u, Str), ...

The trailing comma is a bit tricky; it has to be included as part of the expansion of CHAR8_ONLY(), but this way it seems like it would work.

If you prefer that form, I can change it that way.

[mstorsjo]

Updated the status of P1423, removed unnecessary comments, added optional references to char8_t in static assert messages, removed a dummy variable in the pre-C++20 version of the test helper, updated the commit message to include references to both papers.

[curdeius]

LGTM. But wait for an approval from a higher instance :).

libcxx/test/support/filesystem_test_helper.h

421

I thought of just doing:

#if TEST_STD_VER... 
#define MKSTR(...u8...)
#else 
#define MKSTR(...)
#endif

But that's OK as it is for me. Macros are tricky (or evil).

[ldionne]

libcxx/include/filesystem

994

That's an ABI break in C++20. Do we think it's reasonable?

I think we at least need to add it to the release notes.

[mstorsjo]

libcxx/include/filesystem

994

This method is marked with the _LIBCPP_INLINE_VISIBILITY attribute, which afaik is supposed to exclude it from the ABI altogether - exactly for this reason?

The fact that it's a source compat break, and whether to worry about it or not, is acknowledged and discussed in P1423.

[ldionne]

libcxx/include/filesystem

994

I was trying to come up with examples where a library was already compiled with an inlined version of path::u8string that returns a std::string, and then link against another binary where path::u8string() returns a std::u8string instead. But I couldn't find something that wasn't very tortuous, so I think this is fine from the ABI perspective.

Can we please add a release note because this is a source break?

[mstorsjo]

Added a release note about source incompatibility.

libcxx/include/__config

 #endif
 
 #if _LIBCPP_STD_VER >= 17
 #  define _LIBCPP_DEPRECATED_IN_CXX17 _LIBCPP_DEPRECATED
 #else
 #  define _LIBCPP_DEPRECATED_IN_CXX17
 #endif
 
+#if !defined(_LIBCPP_NO_HAS_CHAR8_T)
+#  define _LIBCPP_DEPRECATED_WITH_CHAR8_T _LIBCPP_DEPRECATED
+#else
+#  define _LIBCPP_DEPRECATED_WITH_CHAR8_T
+#endif
+
 // Macros to enter and leave a state where deprecation warnings are suppressed.
 #if !defined(_LIBCPP_SUPPRESS_DEPRECATED_PUSH) && \
     (defined(_LIBCPP_COMPILER_CLANG) || defined(_LIBCPP_COMPILER_GCC))
 #  define _LIBCPP_SUPPRESS_DEPRECATED_PUSH \
     _Pragma("GCC diagnostic push") \
     _Pragma("GCC diagnostic ignored \"-Wdeprecated\"")
 #  define _LIBCPP_SUPPRESS_DEPRECATED_POP \
     _Pragma("GCC diagnostic pop")

libcxx/include/filesystem

struct __can_convert_char<char> {
   static const bool value = true;
   using __char_type = char;
 };
 template <>
 struct __can_convert_char<wchar_t> {
   static const bool value = true;
   using __char_type = wchar_t;
 };
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+template <>
+struct __can_convert_char<char8_t> {
+  static const bool value = true;
+  using __char_type = char8_t;
+};
+#endif
 template <>
 struct __can_convert_char<char16_t> {
   static const bool value = true;
   using __char_type = char16_t;
 };
 template <>
 struct __can_convert_char<char32_t> {
   static const bool value = true;
public:
   const string_type& native() const noexcept { return __pn_; }
 
   _LIBCPP_INLINE_VISIBILITY
   const value_type* c_str() const noexcept { return __pn_.c_str(); }
 
   _LIBCPP_INLINE_VISIBILITY operator string_type() const { return __pn_; }
 
   template <class _ECharT, class _Traits = char_traits<_ECharT>,
             class _Allocator = allocator<_ECharT> >
   basic_string<_ECharT, _Traits, _Allocator>
   string(const _Allocator& __a = _Allocator()) const {
     using _CVT = __widen_from_utf8<sizeof(_ECharT) * __CHAR_BIT__>;
     using _Str = basic_string<_ECharT, _Traits, _Allocator>;
     _Str __s(__a);
     __s.reserve(__pn_.size());
     _CVT()(back_inserter(__s), __pn_.data(), __pn_.data() + __pn_.size());
     return __s;
   }
 
   _LIBCPP_INLINE_VISIBILITY std::string string() const { return __pn_; }
   _LIBCPP_INLINE_VISIBILITY std::wstring wstring() const {
     return string<wchar_t>();
   }
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+  _LIBCPP_INLINE_VISIBILITY std::u8string u8string() const { return std::u8string(__pn_.begin(), __pn_.end()); }
+#else
   _LIBCPP_INLINE_VISIBILITY std::string u8string() const { return __pn_; }
+#endif
   _LIBCPP_INLINE_VISIBILITY std::u16string u16string() const {
     return string<char16_t>();
   }
   _LIBCPP_INLINE_VISIBILITY std::u32string u32string() const {
     return string<char32_t>();
   }
 
   // generic format observers
   template <class _ECharT, class _Traits = char_traits<_ECharT>,
             class _Allocator = allocator<_ECharT> >
   basic_string<_ECharT, _Traits, _Allocator>
   generic_string(const _Allocator& __a = _Allocator()) const {
     return string<_ECharT, _Traits, _Allocator>(__a);
   }
 
   std::string generic_string() const { return __pn_; }
   std::wstring generic_wstring() const { return string<wchar_t>(); }
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+  std::u8string generic_u8string() const { return std::u8string(__pn_.begin(), __pn_.end()); }
+#else
   std::string generic_u8string() const { return __pn_; }
+#endif
   std::u16string generic_u16string() const { return string<char16_t>(); }
   std::u32string generic_u32string() const { return string<char32_t>(); }
 
 private:
   int __compare(__string_view) const;
   __string_view __root_name() const;
   __string_view __root_directory() const;
   __string_view __root_path_raw() const;

 inline _LIBCPP_INLINE_VISIBILITY void swap(path& __lhs, path& __rhs) noexcept {
   __lhs.swap(__rhs);
 }
 
 _LIBCPP_FUNC_VIS
 size_t hash_value(const path& __p) noexcept;
 
 template <class _Source>
-_LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     typename enable_if<__is_pathable<_Source>::value, path>::type
     u8path(const _Source& __s) {
   static_assert(
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      // Proposed in P1423
+      is_same<typename __is_pathable<_Source>::__char_type, char8_t>::value ||
+#endif
       is_same<typename __is_pathable<_Source>::__char_type, char>::value,
       "u8path(Source const&) requires Source have a character type of type "
       "'char'");
   return path(__s);
 }
 
 template <class _InputIt>
-_LIBCPP_INLINE_VISIBILITY
+_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     typename enable_if<__is_pathable<_InputIt>::value, path>::type
     u8path(_InputIt __f, _InputIt __l) {
   static_assert(
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      // Proposed in P1423
+      is_same<typename __is_pathable<_InputIt>::__char_type, char8_t>::value ||
+#endif
       is_same<typename __is_pathable<_InputIt>::__char_type, char>::value,
       "u8path(Iter, Iter) requires Iter have a value_type of type 'char'");
   return path(__f, __l);
 }
 
 class _LIBCPP_TYPE_VIS path::iterator {
 public:
   enum _ParserState : unsigned char {
     _Singular,
     _BeforeBegin,

libcxx/test/std/input.output/filesystems/class.path/path.member/path.construct/source.pass.cpp

#endif
   {
     static_assert(!std::is_constructible<path, int*>::value, "");
   }
 }
 
 int main(int, char**) {
   for (auto const& MS : PathList) {
     RunTestCase<char>(MS);
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    RunTestCase<char8_t>(MS);
+#endif
     RunTestCase<wchar_t>(MS);
     RunTestCase<char16_t>(MS);
     RunTestCase<char32_t>(MS);
   }
   test_sfinae();
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.member/path.generic.obs/named_overloads.pass.cpp

int main(int, char**)
   auto const& MS = longString;
   const char* value = longString;
   const path p(value);
   {
     std::string s = p.generic_string();
     assert(s == value);
   }
   {
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    ASSERT_SAME_TYPE(decltype(p.generic_u8string()), std::u8string);
+    std::u8string s = p.generic_u8string();
+    assert(s == (const char8_t*)MS);
+#else
+    ASSERT_SAME_TYPE(decltype(p.generic_u8string()), std::string);
     std::string s = p.generic_u8string();
     assert(s == (const char*)MS);
+#endif
   }
   {
     std::wstring s = p.generic_wstring();
     assert(s == (const wchar_t*)MS);
   }
   {
     std::u16string s = p.generic_u16string();
     assert(s == (const char16_t*)MS);
   }
   {
     std::u32string s = p.generic_u32string();
     assert(s == (const char32_t*)MS);
   }
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.member/path.native.obs/named_overloads.pass.cpp

int main(int, char**)
   auto const& MS = longString;
   const char* value = longString;
   const path p(value);
   {
     std::string s = p.string();
     assert(s == value);
   }
   {
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+    ASSERT_SAME_TYPE(decltype(p.u8string()), std::u8string);
+    std::u8string s = p.u8string();
+    assert(s == (const char8_t*)MS);
+#else
+    ASSERT_SAME_TYPE(decltype(p.u8string()), std::string);
     std::string s = p.u8string();
     assert(s == (const char*)MS);
+#endif
   }
   {
     std::wstring s = p.wstring();
     assert(s == (const wchar_t*)MS);
   }
   {
     std::u16string s = p.u16string();
     assert(s == (const char16_t*)MS);
   }
   {
     std::u32string s = p.u32string();
     assert(s == (const char32_t*)MS);
   }
 
   return 0;
 }

libcxx/test/std/input.output/filesystems/class.path/path.nonmember/path.factory.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03
+// ADDITIONAL_COMPILE_FLAGS: -D_LIBCPP_DISABLE_DEPRECATION_WARNINGS
 
 // <filesystem>
 
 // template <class Source>
 //    path u8path(Source const&);
 // template <class InputIter>
 //   path u8path(InputIter, InputIter);
 
int main(int, char**)
   {
     path p = fs::u8path(In3);
     assert(p == In1);
   }
   {
     path p = fs::u8path(In3, In3End);
     assert(p == In1);
   }
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t) && defined(_LIBCPP_VERSION)
+  const char8_t* u8In1 = u8"abcd/efg";
+  const std::u8string u8In2(u8In1);
+  const auto u8In3 = u8In2.begin();
+  const auto u8In3End = u8In2.end();
+  // Proposed in P1423, marked tested only for libc++
+  {
+    path p = fs::u8path(u8In1);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In2);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In3);
+    assert(p == In1);
+  }
+  {
+    path p = fs::u8path(u8In3, u8In3End);
+    assert(p == In1);
+  }
+#endif
 
   return 0;
 }

libcxx/test/support/filesystem_test_helper.h

struct CWDGuard {
   }
 
   CWDGuard(CWDGuard const&) = delete;
   CWDGuard& operator=(CWDGuard const&) = delete;
 };
 
 // Misc test types
 
-#define MKSTR(Str) {Str, TEST_CONCAT(L, Str), TEST_CONCAT(u, Str), TEST_CONCAT(U, Str)}
+#define MKSTR(Str) {Str, TEST_CONCAT(L, Str), TEST_CONCAT(u8, Str), TEST_CONCAT(u, Str), TEST_CONCAT(U, Str)}
 
 struct MultiStringType {
   const char* s;
   const wchar_t* w;
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+  const char8_t* u8;
+#else
+  const char* dummy;
+#endif
   const char16_t* u16;
   const char32_t* u32;
 
   operator const char* () const { return s; }
   operator const wchar_t* () const { return w; }
+#if TEST_STD_VER > 17 && defined(__cpp_char8_t)
+  operator const char8_t* () const { return u8; }
+#endif
   operator const char16_t* () const { return u16; }
   operator const char32_t* () const { return u32; }
 };
 
 const MultiStringType PathList[] = {
         MKSTR(""),
         MKSTR(" "),
         MKSTR("//"),