[libcxx] Fix toupper/tolower tests for UTF-8 locale

[kparzysz]

Summary

The characters like '\xDA' or '\xFA' are not valid representations of anything in UTF-8, so toupper and tolower return the character itself in that locale.

[EricWF]

Seems like we should figure out why these pass on ToT OS X.

[kparzysz]

Unxfail these tests on Linux.

[kparzysz]

In D25811#575105, @EricWF wrote:

Seems like we should figure out why these pass on ToT OS X.

All of them have

XFAIL: with_system_cxx_lib=x86_64-apple-darwin11
XFAIL: with_system_cxx_lib=x86_64-apple-darwin12

Maybe that's it. I don't have access to OS X, so I can't test it there.

[arphaman]

I've tested this patch on OS X and these 4 tests that you've changed now fail.

The XFAIL: with_system_cxx_lib lines don't have any effect on my system - these tests get invoked as expected by lit. Let me know if there's anything else I can do to help you.

[kparzysz]

In D25811#586000, @arphaman wrote:

I've tested this patch on OS X and these 4 tests that you've changed now fail.

The XFAIL: with_system_cxx_lib lines don't have any effect on my system - these tests get invoked as expected by lit. Let me know if there's anything else I can do to help you.

That's surprising. Seems like something is different on OS X.
Could you try compiling/running this small program?

#include <locale>
#include <iostream>

int main() {
  std::locale x("en_US.UTF-8");
  std::cout << x.name() << std::endl;
}

If it prints "en_US.UTF-8", then I'm out of ideas... :)

[kparzysz]

And thanks for help. :)

[arphaman]

In D25811#586053, @kparzysz wrote:

In D25811#586000, @arphaman wrote:

I've tested this patch on OS X and these 4 tests that you've changed now fail.

The XFAIL: with_system_cxx_lib lines don't have any effect on my system - these tests get invoked as expected by lit. Let me know if there's anything else I can do to help you.

That's surprising. Seems like something is different on OS X.
Could you try compiling/running this small program?

#include <locale>
#include <iostream>

int main() {
  std::locale x("en_US.UTF-8");
  std::cout << x.name() << std::endl;
}

If it prints "en_US.UTF-8", then I'm out of ideas... :)

Just tried it, I get "en_US.UTF-8"

[EricWF]

In D25811#586053, @kparzysz wrote:

In D25811#586000, @arphaman wrote:

I've tested this patch on OS X and these 4 tests that you've changed now fail.

The XFAIL: with_system_cxx_lib lines don't have any effect on my system - these tests get invoked as expected by lit. Let me know if there's anything else I can do to help you.

That's surprising.

"XFAIL: with_system_lib=..." are used for testing against OS X's installed libc++, which is very rarely done. @arphaman is probably testing against the in-tree libc++ on OS X, which is why the XFAIL's have no effect.

Seems like something is different on OS X.

The reason for the different output on Linux and OS X is that they each provide their own different locale data. In particular they provide different toupper/tolower conversion tables. For some reason the ones on OS X seem to support UTF-16 conversions as well. Currently GLIBC doesn't even support UTF-8 multi-byte characters.

While I was attempting to review this patch I ended up coming up with fixed tests myself. I committed them in r290746. I'm very sorry to step on your toes.
However there are plenty more //XFAIL: linux localization tests to fix!

[kparzysz]

In D25811#632670, @EricWF wrote:

While I was attempting to review this patch I ended up coming up with fixed tests myself. I committed them in r290746. I'm very sorry to step on your toes.

No problem at all. Thanks for the fixes!

test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_1.pass.cpp

 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT tolower(charT) const;
 
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin11
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin12
-// XFAIL: linux
 
 #include <locale>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);
 
             assert(f.tolower(' ') == ' ');
             assert(f.tolower('A') == 'a');
             assert(f.tolower('\x07') == '\x07');
             assert(f.tolower('.') == '.');
             assert(f.tolower('a') == 'a');
             assert(f.tolower('1') == '1');
-            assert(f.tolower('\xDA') == '\xFA');
+            assert(f.tolower('\xDA') == '\xDA');
             assert(f.tolower('\xFA') == '\xFA');
         }
     }
     {
         std::locale l("C");
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);

test/std/localization/locale.categories/category.ctype/locale.ctype.byname/tolower_many.pass.cpp

 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* tolower(charT* low, const charT* high) const;
 
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin11
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin12
-// XFAIL: linux
 
 #include <locale>
 #include <string>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);
             std::string in("\xDA A\x07.a1");
 
             assert(f.tolower(&in[0], in.data() + in.size()) == in.data() + in.size());
-            assert(in[0] == '\xFA');
+            assert(in[0] == '\xDA');
             assert(in[1] == ' ');
             assert(in[2] == 'a');
             assert(in[3] == '\x07');
             assert(in[4] == '.');
             assert(in[5] == 'a');
             assert(in[6] == '1');
         }
     }

test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_1.pass.cpp

 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // charT toupper(charT) const;
 
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin11
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin12
-// XFAIL: linux
 
 #include <locale>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);
 
             assert(f.toupper(' ') == ' ');
             assert(f.toupper('A') == 'A');
             assert(f.toupper('\x07') == '\x07');
             assert(f.toupper('.') == '.');
             assert(f.toupper('a') == 'A');
             assert(f.toupper('1') == '1');
             assert(f.toupper('\xDA') == '\xDA');
-            assert(f.toupper('\xFA') == '\xDA');
+            assert(f.toupper('\xFA') == '\xFA');
         }
     }
     {
         std::locale l("C");
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);
 

test/std/localization/locale.categories/category.ctype/locale.ctype.byname/toupper_many.pass.cpp

 // <locale>
 
 // template <class charT> class ctype_byname;
 
 // const charT* toupper(charT* low, const charT* high) const;
 
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin11
 // XFAIL: with_system_cxx_lib=x86_64-apple-darwin12
-// XFAIL: linux
 
 #include <locale>
 #include <string>
 #include <cassert>
 
 #include "platform_support.h" // locale name macros
 
 int main()
 {
     {
         std::locale l(LOCALE_en_US_UTF_8);
         {
             typedef std::ctype<char> F;
             const F& f = std::use_facet<F>(l);
             std::string in("\xFA A\x07.a1");
 
             assert(f.toupper(&in[0], in.data() + in.size()) == in.data() + in.size());
-            assert(in[0] == '\xDA');
+            assert(in[0] == '\xFA');
             assert(in[1] == ' ');
             assert(in[2] == 'A');
             assert(in[3] == '\x07');
             assert(in[4] == '.');
             assert(in[5] == 'A');
             assert(in[6] == '1');
         }
     }