[libcxx][pretty printers] Correct locale for u16/u32 string tests

[DavidSpickett]

Summary

When the locale is not some UTF-8 these tests fail.
(different results for python2 linked gdbs vs. python3
but same issue)

Setting the locale just for the test works around this.
By default Ubuntu comes with just C.UTF-8. I've chosen
to use en_US.UTF-8 instead given that my Mac doesn't have
the former and there's a slim chance this test might run there.

This also enables the u16string tests which are now passing.

[DavidSpickett]

You can reproduce by building the CI docker image with Focal instead of Bionic.

The default gdb 9.2 with python 3.8

FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:196
GDB printed:
   'u"\\xd800\\xdd96\\x20ac\\243$"'
Value should match:
   'u"𐆖€£$"'
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:198
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:200
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:206
FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:209
GDB printed:
   'U"\\x1d552\\x1d553\\x1d554\\x1d555\\x1d556\\x1d557"'
Value should match:
   'U"𝕒𝕓𝕔𝕕𝕖𝕗"'
FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:213
GDB printed:
   'U"\\x4f60\\x597d"'
Value should match:
   'U"你好"'

A gdb-10.1 using python 2.7.18 that I built myself:

FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:196
GDB printed:
   u'u"\\xd800\\xdd96\\x20ac\\243$"'
Value should match:
   u'u"\U00010196\u20ac\xa3$"'
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:198
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:200
PASS: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:206
FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:209
GDB printed:
   u'U"\\x1d552\\x1d553\\x1d554\\x1d555\\x1d556\\x1d557"'
Value should match:
   u'U"\U0001d552\U0001d553\U0001d554\U0001d555\U0001d556\U0001d557"'
FAIL: /llvm/libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp:213
GDB printed:
   u'U"\\x4f60\\x597d"'
Value should match:
   u'U"\u4f60\u597d"'

You can see that Python2 is closer, putting \x instead of \u but I don't know where that points the blame. After some digging I still don't have a good idea what layer would be at fault there and if you could even fix it nicely.

@saugustine If you or others have an idea of where the problem might lie I'm happy to look into it more, rather than disabling them.

I should note that the u16string tests were never enabled but that was just a mistake I think. And that this test cannot run on a gdb <8.2 due to some of the API it uses. So that added to my confusion and maybe the reason I can't find any examples of it passing.

At Linaro we're looking to move all of our bots to Focal and when you move CI you'll hit the same issue.

[saugustine]

This looks very much like a locale issue to me. Do you happen to know which one is live in your test bed?

[DavidSpickett]

Not sure which parts are relevant but hopefully one of these.

$ locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
libcxx-builder@b200e5cf42aa:/tmp$ locale -c charmap
LC_CTYPE
ANSI_X3.4-1968

There is also the charset that gdb sees:

(gdb) show target-charset
The target character set is "auto; currently ANSI_X3.4-1968".
(gdb) show target-wide-charset
The target wide character set is "auto; currently UTF-32".

[saugustine]

Changing my current locale to POSIX from en_US.utf-8 reproduces this problem. I just use the default ubuntu locale normally. I don't have ANSI_X3.4-1968 installed, but I suspect the issue is the same. I expect the tests will pass if you switch LANG=en_US.utf-8--but would be good to check.

Not entirely sure the best thing to do here that will be robust to unexpected locales. We can either update the tests to set the locale, or not run if the locale isn't what we expect. Or try a couple of different matching strings.

[DavidSpickett]

The locale tip was spot on.

I've gone with setting the locale just for the test. Luckily we
already have a feature check for that.

[DavidSpickett]

@ldionne Since the test won't run on bionic for other reasons, there's no immediate reason to update the CI container. If you'd like to get localization tests running on more of them, you could update it now.

Either way, Linaro's bots will move to Focal soon and be running this test.

[DavidSpickett]

I overstated the localization tests bit, with this we'll run some more but not all. For that you need fr/ru and zh as well.

Which I can work on adding but for now just en is relevant to the gdb tests.

[DavidSpickett]

Moving the dockerfile change into a separate diff that installs all the needed locales for the test suite.

https://reviews.llvm.org/D111235

[ldionne]

FWIW this seems reasonable to me.

I'll let @saugustine give the final go-ahead since its their code!

[saugustine]

Great change. Thanks--especially for fixing the u16_string test.

[ldionne]

Ship it!

[leonardchan]

Hi again! I'm actually hitting the same failures locally again now that u16string_test was added:

FAIL: Something is wrong in the test framework.
Converting character sets: Invalid argument.
FAIL: Something is wrong in the test framework.
Converting character sets: Invalid argument.
FAIL: Something is wrong in the test framework.
Converting character sets: Invalid argument.
FAIL: Something is wrong in the test framework.
Converting character sets: Invalid argument.

Is there anything else I can help do/provide to debug this?

[saugustine]

I spent some time today trying to reproduce this in as many ways as I could think of, to no avail.

Let's just leave the 16-bit ones disabled for now. I'll work some more on it this weekend and I don't think it should block the very valid other changes here.

libcxx/test/libcxx/gdb/gdb_pretty_printer_test.sh.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: host-has-gdb-with-python
+// REQUIRES: locale.en_US.UTF-8
 // UNSUPPORTED: libcpp-has-no-localization
 // UNSUPPORTED: c++03
 
 // RUN: %{cxx} %{flags} %s -o %t.exe %{compile_flags} -g %{link_flags}
 // Ensure locale-independence for unicode tests.
-// RUN: %{gdb} -nx -batch -iex "set autoload off" -ex "source %S/../../../utils/gdb/libcxx/printers.py" -ex "python register_libcxx_printer_loader()" -ex "source %S/gdb_pretty_printer_test.py" %t.exe
+// RUN: env LANG=en_US.UTF-8 %{gdb} -nx -batch -iex "set autoload off" -ex "source %S/../../../utils/gdb/libcxx/printers.py" -ex "python register_libcxx_printer_loader()" -ex "source %S/gdb_pretty_printer_test.py" %t.exe
 
 #include <bitset>
 #include <deque>
 #include <list>
 #include <map>
 #include <memory>
 #include <queue>
 #include <set>

 }
 
 int main(int, char**) {
   framework_self_test();
 
   string_test();
   a_namespace::string_view_test();
 
+  u16string_test();
   u32string_test();
   tuple_test();
   unique_ptr_test();
   shared_ptr_test();
   bitset_test();
   list_test();
   deque_test();
   map_test();

libcxx/utils/ci/Dockerfile

 # Docker image forever.
 RUN echo 1
 
 RUN apt-get update && apt-get install -y bash curl
 
 # Install various tools used by the build or the test suite
 RUN apt-get update && apt-get install -y ninja-build python3 python3-sphinx python3-distutils git gdb
 RUN apt-get update && apt-get install -y libc6-dev-i386 # Required to cross-compile to 32 bits
+RUN apt-get update && apt-get install -y language-pack-en # To get en_US.UTF-8 local for gdb tests
 
 # Install Clang <latest>, <latest-1> and ToT, which are the ones we support.
 ENV LLVM_LATEST_VERSION=12
 RUN apt-get update && apt-get install -y lsb-release wget software-properties-common
 RUN wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
 RUN bash /tmp/llvm.sh $(($LLVM_LATEST_VERSION - 1)) # previous release
 RUN bash /tmp/llvm.sh $LLVM_LATEST_VERSION          # latest release
 RUN bash /tmp/llvm.sh $(($LLVM_LATEST_VERSION + 1)) # current ToT