[libcxx] Allow to set locale on Windows.

[halyavin]

Summary

Fix the problem PR31516 with setting locale on Windows by wrapping _locale_t with a pointer-like class.

Reduces 74 test failures in std/localization test suite to 47 test failures (on llvm clang, Visual Studio 2015). Number of test failures doesn't depend on the platform (x86 or x64).

[EricWF]

This LGTM. I would love if another party interested in Windows could review it though.

[mstorsjo]

In D40181#930602, @EricWF wrote:

This LGTM. I would love if another party interested in Windows could review it though.

I can test this in a MinGW context and see if what it uses happens to be available there or not.

[mstorsjo]

In order to make this work with MinGW (more or less), I had to change the _LIBCPP_MSVCRT into _LIBCPP_MSVCRT_LIKE both in include/__locale and in include/__config (where _LIBCPP_LOCALE__L_EXTENSIONS is defined). Normal mingw that uses msvcrt.dll doesn't have per-thread locales so it won't really work in any case (but I think it does define some sort of dummy functions that at least will allow it to build). Nowadays, mingw can be built to target ucrtbase.dll as well though, and there it should be possible to make it work just like for MSVC although it might need some patches.

include/__locale

72 ↗

(On Diff #123360)

This syntax only seems to work if the code using this is built in C++11 mode; this breaks building C++98 code.

include/support/win32/locale_win32.h

57 ↗

(On Diff #123360)

In my testing, I had to change all these nullptr_t into std::nullptr_t in order to build it successfully.

[halyavin]

Fixed problem with C++98, added std for nullptr_t and switched on _l functions for MinGW.

[halyavin]

In D40181#930779, @mstorsjo wrote:

(but I think it does define some sort of dummy functions that at least will allow it to build)

Looked into MinGW-w64 sources and this is indeed the case. _configthreadlocale will return -1 and will not do anything.

[mstorsjo]

LGTM, thanks for updating it!

[halyavin]

@EricWF, could you please commit this change?

[Alexpux]

I'm not be able to build libcxx with 64-bit mingw-w64 due to this commit.
Have error like:

[ 10%] Building CXX object projects/libcxx/lib/CMakeFiles/cxx_objects.dir/__/src/algorithm.cpp.obj
In file included from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ctype.h:39:0,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cctype:39,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cwctype:54,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cwchar:107,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/string:481,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/random:1645,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/src/algorithm.cpp:11:
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale: In static member function 'static void std::__1::__num_put<_CharT>::__widen_and_group_float(char*, char*, char*, _CharT*, _CharT*&, _CharT*&, const std::__1::locale&)':
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:150:20: error: ambiguous overload for 'operator!=' (operand types are 'locale_t' and 'long long int')
 #define isxdigit_l _isxdigit_l
                    ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1198:18: note: in expansion of macro 'isxdigit_l'
             if (!isxdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:150:20: note: candidate: operator!=(int, long long int) <built-in>
 #define isxdigit_l _isxdigit_l
                    ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1198:18: note: in expansion of macro 'isxdigit_l'
             if (!isxdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:150:20: note: candidate: operator!=(_locale_t {aka localeinfo_struct*}, _locale_t {aka localeinfo_struct*}) <built-in>
 #define isxdigit_l _isxdigit_l
                    ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1198:18: note: in expansion of macro 'isxdigit_l'
             if (!isxdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~~
In file included from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/__locale:23:0,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ios:216,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ostream:138,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/istream:163,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/random:1646,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/src/algorithm.cpp:11:
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:73:17: note: candidate: bool operator!=(int, const locale_t&)
     friend bool operator!=(int __left, const locale_t& __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:69:17: note: candidate: bool operator!=(const locale_t&, std::nullptr_t)
     friend bool operator!=(const locale_t& __left, std::nullptr_t __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:65:17: note: candidate: bool operator!=(const locale_t&, int)
     friend bool operator!=(const locale_t& __left, int __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:61:17: note: candidate: bool operator!=(const locale_t&, const locale_t&)
     friend bool operator!=(const locale_t& __left, const locale_t& __right) {
                 ^~~~~~~~
In file included from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ctype.h:39:0,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cctype:39,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cwctype:54,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/cwchar:107,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/string:481,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/random:1645,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/src/algorithm.cpp:11:
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:149:19: error: ambiguous overload for 'operator!=' (operand types are 'locale_t' and 'long long int')
 #define isdigit_l _isdigit_l
                   ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1204:18: note: in expansion of macro 'isdigit_l'
             if (!isdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:149:19: note: candidate: operator!=(int, long long int) <built-in>
 #define isdigit_l _isdigit_l
                   ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1204:18: note: in expansion of macro 'isdigit_l'
             if (!isdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:149:19: note: candidate: operator!=(_locale_t {aka localeinfo_struct*}, _locale_t {aka localeinfo_struct*}) <built-in>
 #define isdigit_l _isdigit_l
                   ^
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/locale:1204:18: note: in expansion of macro 'isdigit_l'
             if (!isdigit_l(*__ns, _LIBCPP_GET_C_LOCALE))
                  ^~~~~~~~~
In file included from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/__locale:23:0,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ios:216,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/ostream:138,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/istream:163,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/random:1646,
                 from C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/src/algorithm.cpp:11:
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:73:17: note: candidate: bool operator!=(int, const locale_t&)
     friend bool operator!=(int __left, const locale_t& __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:69:17: note: candidate: bool operator!=(const locale_t&, std::nullptr_t)
     friend bool operator!=(const locale_t& __left, std::nullptr_t __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:65:17: note: candidate: bool operator!=(const locale_t&, int)
     friend bool operator!=(const locale_t& __left, int __right) {
                 ^~~~~~~~
C:/repo/mingw-w64-clang/src/llvm-6.0.0.src/projects/libcxx/include/support/win32/locale_win32.h:61:17: note: candidate: bool operator!=(const locale_t&, const locale_t&)
     friend bool operator!=(const locale_t& __left, const locale_t& __right) {
                 ^~~~~~~~
make[2]: *** [projects/libcxx/lib/CMakeFiles/cxx_objects.dir/build.make:63: projects/libcxx/lib/CMakeFiles/cxx_objects.dir/__/src/algorithm.cpp.obj] Ошибка 1
make[1]: *** [CMakeFiles/Makefile2:11083: projects/libcxx/lib/CMakeFiles/cxx_objects.dir/all] Ошибка 2

[mclow.lists]

I have a bug report - https://bugs.llvm.org/show_bug.cgi?id=41131 - that says that we can reduce the number of calls in __libcpp_locale_guard from 10 to 2 by calling:
setlocale(LC_ALL, ...) . Was this considered when this patch was created?

[halyavin]

Yes, I considered it. But Microsoft documentation is not clear whether it will work or not for restoring inconsistent locales. The documentation shows how to save inconsistent locales with LC_ALL but doesn't provide an example for restoring them.

libcxx/trunk/include/__config

 #ifndef _LIBCPP_EXTERN_TEMPLATE2
 #define _LIBCPP_EXTERN_TEMPLATE2(...) extern template __VA_ARGS__;
 #endif
 
 #if defined(__APPLE__) && defined(__LP64__) && !defined(__x86_64__)
 #define _LIBCPP_NONUNIQUE_RTTI_BIT (1ULL << 63)
 #endif
 
-#if defined(__APPLE__) || defined(__FreeBSD__) || defined(_LIBCPP_MSVCRT) ||   \
+#if defined(__APPLE__) || defined(__FreeBSD__) || defined(_LIBCPP_MSVCRT_LIKE) || \
     defined(__sun__) || defined(__NetBSD__) || defined(__CloudABI__)
 #define _LIBCPP_LOCALE__L_EXTENSIONS 1
 #endif
 
 #if defined(__unix__) || (defined(__APPLE__) && defined(__MACH__))
 // Most unix variants have catopen.  These are the specific ones that don't.
 #if !defined(__BIONIC__) && !defined(_NEWLIB_VERSION)
 #define _LIBCPP_HAS_CATOPEN 1

libcxx/trunk/include/__locale

 #endif
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
-#if !defined(_LIBCPP_LOCALE__L_EXTENSIONS) || defined(_LIBCPP_MSVCRT)
+#if !defined(_LIBCPP_LOCALE__L_EXTENSIONS)
 struct __libcpp_locale_guard {
   _LIBCPP_INLINE_VISIBILITY
   __libcpp_locale_guard(locale_t& __loc) : __old_loc_(uselocale(__loc)) {}
 
   _LIBCPP_INLINE_VISIBILITY
   ~__libcpp_locale_guard() {
     if (__old_loc_)
       uselocale(__old_loc_);
   }
 
   locale_t __old_loc_;
 private:
   __libcpp_locale_guard(__libcpp_locale_guard const&);
   __libcpp_locale_guard& operator=(__libcpp_locale_guard const&);
 };
+#elif defined(_LIBCPP_MSVCRT_LIKE)
+struct __libcpp_locale_guard {
+    __libcpp_locale_guard(locale_t __l) :
+        __status(_configthreadlocale(_ENABLE_PER_THREAD_LOCALE)),
+        __locale_collate(setlocale(LC_COLLATE, __l.__get_locale())),
+        __locale_ctype(setlocale(LC_CTYPE, __l.__get_locale())),
+        __locale_monetary(setlocale(LC_MONETARY, __l.__get_locale())),
+        __locale_numeric(setlocale(LC_NUMERIC, __l.__get_locale())),
+        __locale_time(setlocale(LC_TIME, __l.__get_locale()))
+        // LC_MESSAGES is not supported on Windows.
+    {}
+    ~__libcpp_locale_guard() {
+        setlocale(LC_COLLATE, __locale_collate);
+        setlocale(LC_CTYPE, __locale_ctype);
+        setlocale(LC_MONETARY, __locale_monetary);
+        setlocale(LC_NUMERIC, __locale_numeric);
+        setlocale(LC_TIME, __locale_time);
+        _configthreadlocale(__status);
+    }
+    int __status;
+    char* __locale_collate;
+    char* __locale_ctype;
+    char* __locale_monetary;
+    char* __locale_numeric;
+    char* __locale_time;
+};
 #endif
 
 
 class _LIBCPP_TYPE_VIS locale;
 
 template <class _Facet>
 _LIBCPP_INLINE_VISIBILITY
 bool

libcxx/trunk/include/support/win32/locale_win32.h

 // -*- C++ -*-
 //===--------------------- support/win32/locale_win32.h -------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _LIBCPP_SUPPORT_WIN32_LOCALE_WIN32_H
 #define _LIBCPP_SUPPORT_WIN32_LOCALE_WIN32_H
 
 #include <__config>
 #include <stdio.h>
 #include <xlocinfo.h> // _locale_t
+#include <__nullptr>
 
 #define LC_COLLATE_MASK _M_COLLATE
 #define LC_CTYPE_MASK _M_CTYPE
 #define LC_MONETARY_MASK _M_MONETARY
 #define LC_NUMERIC_MASK _M_NUMERIC
 #define LC_TIME_MASK _M_TIME
 #define LC_MESSAGES_MASK _M_MESSAGES
 #define LC_ALL_MASK (  LC_COLLATE_MASK \
                      | LC_CTYPE_MASK \
                      | LC_MESSAGES_MASK \
                      | LC_MONETARY_MASK \
                      | LC_NUMERIC_MASK \
                      | LC_TIME_MASK )
 
-#define locale_t _locale_t
+class locale_t {
+public:
+    locale_t()
+        : __locale(nullptr), __locale_str(nullptr) {}
+    locale_t(std::nullptr_t)
+        : __locale(nullptr), __locale_str(nullptr) {}
+    locale_t(_locale_t __locale, const char* __locale_str)
+        : __locale(__locale), __locale_str(__locale_str) {}
+
+    friend bool operator==(const locale_t& __left, const locale_t& __right) {
+        return __left.__locale == __right.__locale;
+    }
+
+    friend bool operator==(const locale_t& __left, int __right) {
+        return __left.__locale == nullptr && __right == 0;
+    }
+
+    friend bool operator==(const locale_t& __left, std::nullptr_t) {
+        return __left.__locale == nullptr;
+    }
+
+    friend bool operator==(int __left, const locale_t& __right) {
+        return __left == 0 && nullptr == __right.__locale;
+    }
+
+    friend bool operator==(std::nullptr_t, const locale_t& __right) {
+        return nullptr == __right.__locale;
+    }
+
+    friend bool operator!=(const locale_t& __left, const locale_t& __right) {
+        return !(__left == __right);
+    }
+
+    friend bool operator!=(const locale_t& __left, int __right) {
+        return !(__left == __right);
+    }
+
+    friend bool operator!=(const locale_t& __left, std::nullptr_t __right) {
+        return !(__left == __right);
+    }
+
+    friend bool operator!=(int __left, const locale_t& __right) {
+        return !(__left == __right);
+    }
+
+    friend bool operator!=(std::nullptr_t __left, const locale_t& __right) {
+        return !(__left == __right);
+    }
+
+    operator bool() const {
+        return __locale != nullptr;
+    }
+
+    const char* __get_locale() const { return __locale_str; }
+
+    operator _locale_t() const {
+        return __locale;
+    }
+private:
+    _locale_t __locale;
+    const char* __locale_str;
+};
 
 // Locale management functions
 #define freelocale _free_locale
 // FIXME: base currently unused. Needs manual work to construct the new locale
 locale_t newlocale( int mask, const char * locale, locale_t base );
-locale_t uselocale( locale_t newloc );
+// uselocale can't be implemented on Windows because Windows allows partial modification
+// of thread-local locale and so _get_current_locale() returns a copy while uselocale does
+// not create any copies.
+// We can still implement raii even without uselocale though.
 
 
 lconv *localeconv_l( locale_t loc );
 size_t mbrlen_l( const char *__restrict s, size_t n,
                  mbstate_t *__restrict ps, locale_t loc);
 size_t mbsrtowcs_l( wchar_t *__restrict dst, const char **__restrict src,
                     size_t len, mbstate_t *__restrict ps, locale_t loc );
 size_t wcrtomb_l( char *__restrict s, wchar_t wc, mbstate_t *__restrict ps,

libcxx/trunk/src/support/win32/locale_win32.cpp

 #include <memory>
 #include <type_traits>
 
 using std::__libcpp_locale_guard;
 
 // FIXME: base currently unused. Needs manual work to construct the new locale
 locale_t newlocale( int mask, const char * locale, locale_t /*base*/ )
 {
-    return _create_locale( mask, locale );
-}
-
-locale_t uselocale( locale_t newloc )
-{
-    locale_t old_locale = _get_current_locale();
-    if ( newloc == NULL )
-        return old_locale;
-    // uselocale sets the thread's locale by definition, so unconditionally use thread-local locale
-    _configthreadlocale( _ENABLE_PER_THREAD_LOCALE );
-    // uselocale sets all categories
-    // disable setting locale on Windows temporarily because the structure is opaque (PR31516)
-    //setlocale( LC_ALL, newloc->locinfo->lc_category[LC_ALL].locale );
-    // uselocale returns the old locale_t
-    return old_locale;
+    return {_create_locale( LC_ALL, locale ), locale};
 }
 
 decltype(MB_CUR_MAX) MB_CUR_MAX_L( locale_t __l )
 {
 #if defined(_LIBCPP_MSVCRT)
   return ___mb_cur_max_l_func(__l);
 #else
   __libcpp_locale_guard __current(__l);