[lldb] Add support for UTF-8 unicode formatting

[ljmf00]

Summary

This patch adds missing formatting for UTF-8 unicode.

Cross-referencing https://reviews.llvm.org/D66447

[ljmf00]

I can reproduce this with LDC (D LLVM-based compiler), but I don't know a way to do it with clang. Probably I need to manually write a .ll file for this. Can someone guide me on adding tests for this?

[ljmf00]

I will also create an issue and cross-reference this.

[ljmf00]

https://bugs.llvm.org/show_bug.cgi?id=52324

[labath]

What exactly do you need the test to do? Is printing a global variable (char8_t_ish foo[] = "my string") sufficient?
If so, you can take a look at the tests in test/Shell/SymbolFile/DWARF for inspiration. There are .s and .ll tests there.
The pattern would roughly be:

# RUN: llvm-mc (or llc) -o %t %s
# RUN: %lldb %t -o "target variable foo" | FileCheck %s

# CHECK: (char8_t_ish[]) foo = "my string"

your code here

[ljmf00]

In D112564#3089399, @labath wrote:

What exactly do you need the test to do? Is printing a global variable (char8_t_ish foo[] = "my string") sufficient?
If so, you can take a look at the tests in test/Shell/SymbolFile/DWARF for inspiration. There are .s and .ll tests there.
The pattern would roughly be:

# RUN: llvm-mc (or llc) -o %t %s
# RUN: %lldb %t -o "target variable foo" | FileCheck %s

# CHECK: (char8_t_ish[]) foo = "my string"

your code here

Yeah, it is a bit more complicated than that although, due to https://bugs.llvm.org/show_bug.cgi?id=45856 . I'm going to take a look at those tests and write some.

[ljmf00]

In D112564#3089399, @labath wrote:

What exactly do you need the test to do? Is printing a global variable (char8_t_ish foo[] = "my string") sufficient?
If so, you can take a look at the tests in test/Shell/SymbolFile/DWARF for inspiration. There are .s and .ll tests there.
The pattern would roughly be:

# RUN: llvm-mc (or llc) -o %t %s
# RUN: %lldb %t -o "target variable foo" | FileCheck %s

# CHECK: (char8_t_ish[]) foo = "my string"

your code here

I was writing the tests locally, although I can't reproduce the failure with a simple type. I can get the error when using a custom LLDB formatter, however. See https://github.com/ljmf00/lldb-d/blob/main/lldb/source/Plugins/Language/D/DTypeUtils.cpp#L82 . I should also rename this patch to something less misleading, since it is not the type name the issue, I just got this right due to fallback on invalid encoding on my side.

That said, do you know an easier way to test this?

[ljmf00]

@labath I found the missing part, DumpTypeValue depends on lldb::Format and eFormatUnicode8 case should be added there too. I added tests and also changed the patch message. Can you re-review?

[labath]

This looks fine. Thanks for tracking this down.

lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp

case clang::BuiltinType::Bool:
       return lldb::eFormatBoolean;
     case clang::BuiltinType::Char_S:
     case clang::BuiltinType::SChar:
     case clang::BuiltinType::WChar_S:
     case clang::BuiltinType::Char_U:
     case clang::BuiltinType::UChar:
     case clang::BuiltinType::WChar_U:
       return lldb::eFormatChar;
+    case clang::BuiltinType::Char8:
+      return lldb::eFormatUnicode8;
     case clang::BuiltinType::Char16:
       return lldb::eFormatUnicode16;
     case clang::BuiltinType::Char32:
       return lldb::eFormatUnicode32;
     case clang::BuiltinType::UShort:
       return lldb::eFormatUnsigned;
     case clang::BuiltinType::Short:
       return lldb::eFormatDecimal;
default:
         case eFormatVectorOfFloat64:
         case eFormatVectorOfUInt128:
           break;
 
         case eFormatChar:
         case eFormatCharPrintable:
         case eFormatCharArray:
         case eFormatBytes:
+        case eFormatUnicode8:
         case eFormatBytesWithASCII:
           item_count = byte_size;
           byte_size = 1;
           break;
 
         case eFormatUnicode16:
           item_count = byte_size / 2;
           byte_size = 2;

lldb/test/API/functionalities/data-formatter/builtin-formats/TestBuiltinFormats.py

def test(self):
         for c in cstring_chars2:
              string_expr += format(ord(c), "x").zfill(2)
         string_expr += "ull"
 
         # Try to print that uint128_t with the different char formatters.
         self.assertIn('= \\0\\e90zaZA\\v\\t\\r\\n\\f\\b\\a \n', self.getFormatted("character array", string_expr))
         self.assertIn('= \\0\\e90zaZA\\v\\t\\r\\n\\f\\b\\a \n', self.getFormatted("character", string_expr))
         self.assertIn('= ..90zaZA....... \n', self.getFormatted("printable character", string_expr))
-        # FIXME: This should probably print the characters in the uint128_t.
-        self.assertIn('= 0x2007080c0a0d090b415a617a30391b00\n', self.getFormatted("unicode8", string_expr))
+        self.assertIn('= 0x00 0x1b 0x39 0x30 0x7a 0x61 0x5a 0x41 0x0b 0x09 0x0d 0x0a 0x0c 0x08 0x07 0x20\n', self.getFormatted("unicode8", string_expr))
 
         # OSType
         ostype_expr = "(__UINT64_TYPE__)0x"
         for c in cstring_chars1:
              ostype_expr += format(ord(c), "x").zfill(2)
         self.assertIn("= ' \\a\\b\\f\\n\\r\\t\\v'\n", self.getFormatted("OSType", ostype_expr))
 
         ostype_expr = "(__UINT64_TYPE__)0x"
         for c in cstring_chars2:
              ostype_expr += format(ord(c), "x").zfill(2)
         self.assertIn("= 'AZaz09\\e\\0'\n", self.getFormatted("OSType", ostype_expr))
 
         self.assertIn('= 0x2007080c0a0d090b415a617a30391b00\n', self.getFormatted("OSType", string_expr))
 
         # bytes
         self.assertIn(r'= " \U0000001b\a\b\f\n\r\t\vaA09\0"', self.getFormatted("bytes", "cstring"))
 
         # bytes with ASCII
         self.assertIn(r'= " \U0000001b\a\b\f\n\r\t\vaA09\0"', self.getFormatted("bytes with ASCII", "cstring"))
 
+        # unicode8
+        self.assertIn('= 0x78 0x56 0x34 0x12\n', self.getFormatted("unicode8", "0x12345678"))
+
         # unicode16
         self.assertIn('= U+5678 U+1234\n', self.getFormatted("unicode16", "0x12345678"))
 
         # unicode32
         self.assertIn('= U+0x89abcdef U+0x01234567\n', self.getFormatted("unicode32", "(__UINT64_TYPE__)0x123456789ABCDEFll"))
 
         # address
         self.assertIn("= 0x00000012\n", self.getFormatted("address", "0x12"))