[libc++] Do not hard-code locale data in unit tests: get it from the OS instead

[vangyzen]

Summary

Locale data can vary across platforms. Rather than hard-code the expected
data in the units tests, get it from the OS via the C API.

This fixes these unit tests on FreeBSD. It might fix them on Linux, too.
I have not tested other platforms, but I will if I get agreement
on the approach.

[EricWF]

LGTM minus inline comments.

Thanks for working on this.

test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

50 ↗

(On Diff #78886)

I think we should reset the global locale so it's different than the one we pass to std::locale l(...). That way we're testing that we don't accidental use the global locale.

[vangyzen]

test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

50 ↗

(On Diff #78886)

Good idea.

[vangyzen]

Reset the global locale so it's different than the one we pass
to std::locale l(...). That way we're testing that we don't accidentally
use the global locale.

[vangyzen]

Fix comment

vim + Caps Lock == :(

[vangyzen]

Remove XFAIL on linux-gnu

These tests now pass on Fedora 24.

[vangyzen]

@EricWF, would you like to see anything more from me? I could set up an LLVM development environment on my Mac if you want me to run the unit tests on Mac.

[EricWF]

In D26979#603110, @vangyzen wrote:

@EricWF, would you like to see anything more from me? I could set up an LLVM development environment on my Mac if you want me to run the unit tests on Mac.

Nope. Feel free to commit after adding the requested assertions.

test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

65 ↗

(On Diff #78921)

We should probably assert that thousands_sep isn't a multibyte character.

[vangyzen]

In D26979#604013, @EricWF wrote:

Nope. Feel free to commit after adding the requested assertions.

I would gladly commit it, but I don't have commit access. I'm quite new here.

test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

65 ↗

(On Diff #78921)

I don't quite follow. C defines it as char*. Are you concerned about an implementation defining it as wchar_t*?

[vangyzen]

test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

65 ↗

(On Diff #78921)

Oh, I understand now. Sorry. I've never really done much with locales, so I'm not so fluent with them...especially early on a holiday morning. :)

[vangyzen]

Handle multibyte thousands_sep; do not reference possibly stale locale data

[vangyzen]

I'm glad you mentioned a multibyte thousands_sep, because it is multibyte in fr_FR.UTF-8 on FreeBSD 11. Specifically, it's a no-break space (U+00A0).

[vangyzen]

projects/libcxx/test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/grouping.pass.cpp

42

localeconv()->grouping can become invalid after we reset the locale to "C", so duplicate the string into a local.

projects/libcxx/test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

79

This assertion now fails for me. wexpected is 0xa0 (NBSP), which is correct. However, np.thousands_sep() is -62, which is 0xc2, which is the first byte of a UTF-8-encoded NBSP. It looks like the library isn't correctly handling multibyte thousands separators.

[vangyzen]

projects/libcxx/test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

79

D27167 adds support for multibyte strings in decimal_point and thousands_sep. With that change, this unit test passes.

[vangyzen]

@EricWF Do you have time and interest to review this again?

[vangyzen]

@EricWF Do you have time and interest to review this again?

I should ask, do you have time and interest to commit this?

projects/libcxx/test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/grouping.pass.cpp

 // REQUIRES: locale.fr_FR.UTF-8
 
 // <locale>
 
 // template <class charT> class numpunct_byname;
 
 // string grouping() const;
 
-// TODO: investigation needed
-// XFAIL: linux-gnu
-
 #include <locale>
+#include <clocale>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l("C");
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
             assert(np.grouping() == "");
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
             assert(np.grouping() == "");
         }
     }
     {
+        assert(std::setlocale(LC_NUMERIC, LOCALE_en_US_UTF_8) != NULL);
+        char *expected = strdup(std::localeconv()->grouping);
+        assert(std::setlocale(LC_NUMERIC, "C") != NULL);
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.grouping() == "\3\3");
+            assert(np.grouping() == expected);
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.grouping() == "\3\3");
+            assert(np.grouping() == expected);
         }
+        free(expected);
     }
     {
+        assert(std::setlocale(LC_NUMERIC, LOCALE_fr_FR_UTF_8) != NULL);
+        char *expected = strdup(std::localeconv()->grouping);
+        assert(std::setlocale(LC_NUMERIC, "C") != NULL);
         std::locale l(LOCALE_fr_FR_UTF_8);
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.grouping() == "\x7F");
+            assert(np.grouping() == expected);
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.grouping() == "\x7F");
+            assert(np.grouping() == expected);
         }
+        free(expected);
     }
 }

projects/libcxx/test/std/localization/locale.categories/facet.numpunct/locale.numpunct.byname/thousands_sep.pass.cpp

 // REQUIRES: locale.fr_FR.UTF-8
 
 // <locale>
 
 // template <class charT> class numpunct_byname;
 
 // char_type thousands_sep() const;
 
-// TODO: investigation needed
-// XFAIL: linux-gnu
-
 #include <locale>
+#include <clocale>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l("C");
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
             assert(np.thousands_sep() == ',');
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
             assert(np.thousands_sep() == L',');
         }
     }
     {
+        char expected;
+        wchar_t wexpected;
+        assert(std::setlocale(LC_ALL, LOCALE_en_US_UTF_8) != NULL);
+        const char *ts = std::localeconv()->thousands_sep;
+        expected = *ts;
+        int nb = mbtowc(&wexpected, ts, strlen(ts)+1);
+        assert(nb >= 0);
+        assert(std::setlocale(LC_ALL, "C") != NULL);
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.thousands_sep() == ',');
+            assert(np.thousands_sep() == expected);
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.thousands_sep() == L',');
+            assert(np.thousands_sep() == wexpected);
         }
     }
     {
+        char expected;
+        wchar_t wexpected;
+        assert(std::setlocale(LC_ALL, LOCALE_fr_FR_UTF_8) != NULL);
+        const char *ts = std::localeconv()->thousands_sep;
+        expected = *ts;
+        int nb = mbtowc(&wexpected, ts, strlen(ts)+1);
+        assert(nb >= 0);
+        assert(std::setlocale(LC_ALL, "C") != NULL);
         std::locale l(LOCALE_fr_FR_UTF_8);
         {
             typedef char C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.thousands_sep() == ',');
+            assert(np.thousands_sep() == expected);
         }
         {
             typedef wchar_t C;
             const std::numpunct<C>& np = std::use_facet<std::numpunct<C> >(l);
-            assert(np.thousands_sep() == L',');
+            assert(np.thousands_sep() == wexpected);
         }
     }
 }