[5/N] [libcxx] Convert paths to/from the right narrow code page for narrow strings on windows

[mstorsjo]

Summary

On windows, the narrow, char based paths normally don't use utf8, but can use many different native code pages, and this is what system functions that operate on files, taking such paths/file names, interpret them as.

[mstorsjo]

Updated to define WIN32_LEAN_AND_MEAN and NOMINMAX when including windows.h in operations.cpp.

[amccarth]

The Windows details look correct, though I have a couple questions on the code page conversions.

libcxx/src/filesystem/operations.cpp

1696

I'm not clear on what AreFileApisANSI does, nor have I found it using code search. In what case would any APIs use the OEM code page when the user's code page is something else?

I'm also wondering whether CP_ACP should be CP_THREAD_ACP, which, by default, will be the user's current code page which could be different than the system code page.

[mstorsjo]

libcxx/src/filesystem/operations.cpp

1696

See D91133 for a testcase (which passes with MSVC STL) for this and some more - a process can switch between whether the narrow file apis take OEM CP or ACP with SetFileApisToANSI() and SetFileApisToOEM().

The MSVC STL sources don't seem to use CP_THREAD_ACP at least... however they also check for ___lc_codepage_func() == CP_UTF8, which I guess we also should...

[mstorsjo]

libcxx/src/filesystem/operations.cpp

1696

To follow up on the last bit regarding ___lc_codepage_func(), contrary to SetFileApisToOEM() which operates on the kernel32 level, affecting how all narrow file names are interpreted, one can also do setlocale(LC_ALL, ".utf8");, which makes e.g. fopen() work with utf8 file names, but it has no effect on actual kernel32 -A suffixed file APIs.

But in any case, I guess adding the extra check for ___lc_codepage_func(), like ___lc_codepage_func() == CP_UTF8 ? CP_UTF8 : AreFileApisANSI() ? CP_ACP : CP_OEMCP.

After reading up on CP_THREAD_ACP, I don't think that's relevant. The main point is that if I export a narrow form of a filename from a std::filesystem::path object (or create a path object from a narrow string), I expect the string to be in the same codepage that fopen() or CreateFileA() would accept and interpreted in the same way, and as far as I'm reading about CP_THREAD_ACP, it only affects other bits, but not how kernel32 file APIs map narrow chars to wchar.

[amccarth]

LGTM.

Excellent. Somehow I missed that AreFileApisANSI was a Win32 function. I must have mistyped it in my searches the other day. This looks good.

[curdeius]

I take your word on what you explained about codepages as I'm not much familiar with it. Otherwise LGTM with some nits.

libcxx/include/filesystem

1442

Or just use it instead of _Str?

1471

1476

Please add comments to longer #endifs.

[mstorsjo]

Applied the suggested changes.

[mstorsjo]

Set the repository, to allow the CI to run.

[curdeius]

libcxx/include/filesystem

1444

It's there a patch where you add _w.reserve(...)? If not, this one seems like a good candidate.
If I'm not mistaken, widening can decrease the number of characters, so that might get tricky (like going through the input twice, once to count the output size, then doing the real conversion), but at least a FIXME note would be great.
BTW, you probably know that, MultiByteToWideChar will return the required output buffer size without doing the conversion if you pass 0 as cchWideChar.

[mstorsjo]

libcxx/include/filesystem

1444

I guess I could call reserve() here using the length of the utf8 string as size. As converting from utf8 to wchar in most cases will make the input shorter, so the size allocated by reserve() should be enough in most concievable cases. For strings consisting mostly of ascii chars, the actual difference in length shouldn't be much, so it shouldn't hurt much with such a rough estimate.

Yeah I know that MultiByteToWideChar can count the needed output size, this is used in e.g. size_t __size = __char_to_wide(__str, nullptr, 0); further up in the same patch when operating on the native narrow charset. I'd rather not mix MultiByteToWideChar with libc++'s __widen_from_utf8, especially as the exact length isn't needed beforehand here as that conversion allocates more as needed.

[curdeius]

libcxx/include/filesystem

1444

I agree that mixing MultiByteToWideChar with __widen_from_utf8 doesn't seem like a good idea.
But I'll be for what you suggested, so just reserve possibly more.

[mstorsjo]

Added calls to reserve() where suitable.

[ldionne]

LGTM except nits!

libcxx/include/filesystem

1464

I don't think #ifdefing the static_assert message adds anything. It just makes things harder to read IMO. I know this was done the same way elsewhere, but I think we should actually remove the ifdefs in those other places instead.

libcxx/src/filesystem/operations.cpp

21

Please indent includes like:

#if defined(_LIBCPP_WIN32API`
#   define WIN32_LEAN_AND_MEAN
#   include <stuff>
#else
#   include <more stuff>
#endif

[mstorsjo]

libcxx/include/filesystem

1464

Fair enough, I can remove that bit from this patch, and I'll send a separate patch for removing the other ifdefs, and rebase these patches on top of that (where relevant) once that's merged.

libcxx/src/filesystem/operations.cpp

21

Ok, will change.

[mstorsjo]

Indented preprocessor directives within ifdefs (indented with one space, to match other existing similar cases), removed ifdef in assert message.

[mstorsjo]

Reupload to re-set the repository, to trigger CI

[mstorsjo]

Rebased

[ldionne]

Thanks!

libcxx/include/filesystem

 #if defined(_LIBCPP_WIN32API)
 typedef wstring __path_string;
 typedef wchar_t __path_value;
 #else
 typedef string __path_string;
 typedef char __path_value;
 #endif
 
+#if defined(_LIBCPP_WIN32API)
+_LIBCPP_FUNC_VIS
+size_t __wide_to_char(const wstring&, char*, size_t);
+_LIBCPP_FUNC_VIS
+size_t __char_to_wide(const string&, wchar_t*, size_t);
+#endif
+
 template <class _ECharT>
 struct _PathCVT;
 
 #if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
 template <class _ECharT>
 struct _PathCVT {
   static_assert(__can_convert_char<_ECharT>::value,
                 "Char type not convertible");
struct _PathCVT<__path_value> {
   static void __append_source(__path_string& __dest, _Source const& __s) {
     using _Traits = __is_pathable<_Source>;
     __append_range(__dest, _Traits::__range_begin(__s),
                    _Traits::__range_end(__s));
   }
 };
 
 #if defined(_LIBCPP_WIN32API)
+template <>
+struct _PathCVT<char> {
+
+  static void
+  __append_string(__path_string& __dest, const basic_string<char> &__str) {
+      size_t __size = __char_to_wide(__str, nullptr, 0);
+      size_t __pos = __dest.size();
+      __dest.resize(__pos + __size);
+      __char_to_wide(__str, const_cast<__path_value*>(__dest.data()) + __pos, __size);
+  }
+
+  template <class _Iter>
+  static typename enable_if<__is_exactly_cpp17_input_iterator<_Iter>::value>::type
+  __append_range(__path_string& __dest, _Iter __b, _Iter __e) {
+    basic_string<char> __tmp(__b, __e);
+    __append_string(__dest, __tmp);
+  }
+
+  template <class _Iter>
+  static typename enable_if<__is_cpp17_forward_iterator<_Iter>::value>::type
+  __append_range(__path_string& __dest, _Iter __b, _Iter __e) {
+    basic_string<char> __tmp(__b, __e);
+    __append_string(__dest, __tmp);
+  }
+
+  template <class _Iter>
+  static void __append_range(__path_string& __dest, _Iter __b, _NullSentinel) {
+    const char __sentinel = char{};
+    basic_string<char> __tmp;
+    for (; *__b != __sentinel; ++__b)
+      __tmp.push_back(*__b);
+    __append_string(__dest, __tmp);
+  }
+
+  template <class _Source>
+  static void __append_source(__path_string& __dest, _Source const& __s) {
+    using _Traits = __is_pathable<_Source>;
+    __append_range(__dest, _Traits::__range_begin(__s),
+                   _Traits::__range_end(__s));
+  }
+};
+
 template <class _ECharT>
 struct _PathExport {
   typedef __narrow_to_utf8<sizeof(wchar_t) * __CHAR_BIT__> _Narrower;
   typedef __widen_from_utf8<sizeof(_ECharT) * __CHAR_BIT__> _Widener;
 
   template <class _Str>
   static void __append(_Str& __dest, const __path_string& __src) {
     string __utf8;
     _Narrower()(back_inserter(__utf8), __src.data(), __src.data() + __src.size());
     _Widener()(back_inserter(__dest), __utf8.data(), __utf8.data() + __utf8.size());
   }
 };
 
 template <>
+struct _PathExport<char> {
+  template <class _Str>
+  static void __append(_Str& __dest, const __path_string& __src) {
+    size_t __size = __wide_to_char(__src, nullptr, 0);
+    size_t __pos = __dest.size();
+    __dest.resize(__size);
+    __wide_to_char(__src, const_cast<char*>(__dest.data()) + __pos, __size);
+  }
+};
+
+template <>
 struct _PathExport<wchar_t> {
   template <class _Str>
   static void __append(_Str& __dest, const __path_string& __src) {
     __dest.append(__src.begin(), __src.end());
   }
 };
 
 template <>
string(const _Allocator& __a = _Allocator()) const {
     _PathExport<_ECharT>::__append(__s, __pn_);
     return __s;
   }
 
   _LIBCPP_INLINE_VISIBILITY _VSTD::string string() const {
     return string<char>();
   }
   _LIBCPP_INLINE_VISIBILITY __u8_string u8string() const {
-    return string<__u8_string::value_type>();
+    using _CVT = __narrow_to_utf8<sizeof(wchar_t) * __CHAR_BIT__>;
+    __u8_string __s;
+    __s.reserve(__pn_.size());
+    _CVT()(back_inserter(__s), __pn_.data(), __pn_.data() + __pn_.size());
+    return __s;
   }
 
   _LIBCPP_INLINE_VISIBILITY _VSTD::u16string u16string() const {
     return string<char16_t>();
   }
   _LIBCPP_INLINE_VISIBILITY _VSTD::u32string u32string() const {
     return string<char32_t>();
   }
_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     u8path(_InputIt __f, _InputIt __l) {
   static_assert(
 #ifndef _LIBCPP_NO_HAS_CHAR8_T
       is_same<typename __is_pathable<_InputIt>::__char_type, char8_t>::value ||
 #endif
       is_same<typename __is_pathable<_InputIt>::__char_type, char>::value,
       "u8path(Iter, Iter) requires Iter have a value_type of type 'char'"
       " or 'char8_t'");
+#if defined(_LIBCPP_WIN32API)
+  string __tmp(__f, __l);
+  using _CVT = __widen_from_utf8<sizeof(wchar_t) * __CHAR_BIT__>;
+  _VSTD::wstring __w;
+  __w.reserve(__tmp.size());
+  _CVT()(back_inserter(__w), __tmp.data(), __tmp.data() + __tmp.size());
+  return path(__w);
+#else
   return path(__f, __l);
+#endif /* !_LIBCPP_WIN32API */
+}
+
+#if defined(_LIBCPP_WIN32API)
+template <class _InputIt>
+_LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
+    typename enable_if<__is_pathable<_InputIt>::value, path>::type
+    u8path(_InputIt __f, _NullSentinel) {
+  static_assert(
+#ifndef _LIBCPP_NO_HAS_CHAR8_T
+      is_same<typename __is_pathable<_InputIt>::__char_type, char8_t>::value ||
+#endif
+      is_same<typename __is_pathable<_InputIt>::__char_type, char>::value,
+      "u8path(Iter, Iter) requires Iter have a value_type of type 'char'"
+      " or 'char8_t'");
+  string __tmp;
+  const char __sentinel = char{};
+  for (; *__f != __sentinel; ++__f)
+    __tmp.push_back(*__f);
+  using _CVT = __widen_from_utf8<sizeof(wchar_t) * __CHAR_BIT__>;
+  _VSTD::wstring __w;
+  __w.reserve(__tmp.size());
+  _CVT()(back_inserter(__w), __tmp.data(), __tmp.data() + __tmp.size());
+  return path(__w);
 }
+#endif /* _LIBCPP_WIN32API */
 
 template <class _Source>
 _LIBCPP_INLINE_VISIBILITY _LIBCPP_DEPRECATED_WITH_CHAR8_T
     typename enable_if<__is_pathable<_Source>::value, path>::type
     u8path(const _Source& __s) {
   static_assert(
 #ifndef _LIBCPP_NO_HAS_CHAR8_T
       is_same<typename __is_pathable<_Source>::__char_type, char8_t>::value ||
 #endif
       is_same<typename __is_pathable<_Source>::__char_type, char>::value,
       "u8path(Source const&) requires Source have a character type of type "
       "'char' or 'char8_t'");
+#if defined(_LIBCPP_WIN32API)
+  using _Traits = __is_pathable<_Source>;
+  return u8path(__unwrap_iter(_Traits::__range_begin(__s)), __unwrap_iter(_Traits::__range_end(__s)));
+#else
   return path(__s);
+#endif
 }
 
 class _LIBCPP_TYPE_VIS path::iterator {
 public:
   enum _ParserState : unsigned char {
     _Singular,
     _BeforeBegin,
     _InRootName,

libcxx/src/filesystem/filesystem_common.h

 template <class T>
 T error_value();
 template <>
 _LIBCPP_CONSTEXPR_AFTER_CXX11 void error_value<void>() {}
 template <>
 bool error_value<bool>() {
   return false;
 }
+#if __SIZEOF_SIZE_T__ != __SIZEOF_LONG_LONG__
+template <>
+size_t error_value<size_t>() {
+  return size_t(-1);
+}
+#endif
 template <>
 uintmax_t error_value<uintmax_t>() {
   return uintmax_t(-1);
 }
 template <>
 _LIBCPP_CONSTEXPR_AFTER_CXX11 file_time_type error_value<file_time_type>() {
   return file_time_type::min();
 }

libcxx/src/filesystem/operations.cpp

 #include "string_view"
 #include "type_traits"
 #include "vector"
 #include "cstdlib"
 #include "climits"
 
 #include "filesystem_common.h"
 
-#include <unistd.h>
-#include <sys/stat.h>
-#include <sys/statvfs.h>
+#if defined(_LIBCPP_WIN32API)
+# define WIN32_LEAN_AND_MEAN
+# define NOMINMAX
+# include <windows.h>
+#else
+# include <unistd.h>
+# include <sys/stat.h>
+# include <sys/statvfs.h>
+#endif
 #include <time.h>
 #include <fcntl.h> /* values for fchmodat */
 
 #if __has_include(<sys/sendfile.h>)
 # include <sys/sendfile.h>
 # define _LIBCPP_FILESYSTEM_USE_SENDFILE
 #elif defined(__APPLE__) || __has_include(<copyfile.h>)
 # include <copyfile.h>
path::iterator& path::iterator::__decrement() {
   PathParser PP(__path_ptr_->native(), __entry_, __state_);
   --PP;
   __state_ = static_cast<_ParserState>(PP.State);
   __entry_ = PP.RawEntry;
   __stashed_elem_.__assign_view(*PP);
   return *this;
 }
 
+#if defined(_LIBCPP_WIN32API)
+////////////////////////////////////////////////////////////////////////////
+// Windows path conversions
+size_t __wide_to_char(const wstring &str, char *out, size_t outlen) {
+  if (str.empty())
+    return 0;
+  ErrorHandler<size_t> err("__wide_to_char", nullptr);
+  UINT codepage = AreFileApisANSI() ? CP_ACP : CP_OEMCP;
+  BOOL used_default = FALSE;
+  int ret = WideCharToMultiByte(codepage, 0, str.data(), str.size(), out,
+                                outlen, nullptr, &used_default);
+  if (ret <= 0 || used_default)
+    return err.report(errc::illegal_byte_sequence);
+  return ret;
+}
+
+size_t __char_to_wide(const string &str, wchar_t *out, size_t outlen) {
+  if (str.empty())
+    return 0;
+  ErrorHandler<size_t> err("__char_to_wide", nullptr);
+  UINT codepage = AreFileApisANSI() ? CP_ACP : CP_OEMCP;
+  int ret = MultiByteToWideChar(codepage, MB_ERR_INVALID_CHARS, str.data(),
+                                str.size(), out, outlen);
+  if (ret <= 0)
+    return err.report(errc::illegal_byte_sequence);
+  return ret;
+}
+#endif
+
+
 ///////////////////////////////////////////////////////////////////////////////
 //                           directory entry definitions
 ///////////////////////////////////////////////////////////////////////////////
 
 #ifndef _LIBCPP_WIN32API
 error_code directory_entry::__do_refresh() noexcept {
   __data_.__reset();
   error_code failure_ec;