[lldb] Also recognize DWARF UTF base types using their size

[Geod24]

Summary

The D programming language has 'char', 'wchar', and 'dchar' as base types,
which are defined as UTF-8, UTF-16, and UTF-32, respectively.

It also has type constructors (e.g. 'const' and 'immutable'),
that leads to D compilers emitting DW_TAG_base_type with DW_ATE_UTF
and name 'char', 'immutable(wchar)', 'const(char)', etc...

Before this patch, DW_ATE_UTF would only recognize types that
followed the C/C++ naming, and emit an error message for the rest, e.g.:

error: need to add support for DW_TAG_base_type 'immutable(char)'
encoded with DW_ATE = 0x10, bit_size = 8

The code was changed to check the byte size first,
then fall back to the old name-based check.

[Geod24]

I wasn't sure what would be the best place / how to add an unit test for this. Could a reviewer provide some pointers ?

[labath]

Thanks for the patch.

In D79559#2024573, @Geod24 wrote:

I wasn't sure what would be the best place / how to add an unit test for this. Could a reviewer provide some pointers ?

I think the simplest approach would be to create some assembly representing a compile unit which declares a couple of global variables of this type and then run "lldb -o "target variable var1 var2..." over the compiled object. You can look at some of the files in test/Shell/SymbolFile/DWARF (e.g. DW_OP_piece-struct.s) for inspiration. Using llvm IR instead of assembly would work too, assuming you can represent this thing in IR..

[Geod24]

Added Shell test

A shell test was added using LLVM IR.
The test has to run the generated binary, as there is currently a (likely) bug
that leads to types being printed differently before and after 'run' is issued.

[labath]

The running aspect is somewhat unfortunate, as it forces the input to be compatible with the host system. I wouldn't be surprised if the test broke somewhere (in fact, I would be surprised if it didn't), because ir already hardcodes a lot of assumptions about the host system.

Given that the fix really is about how types are parsed, it should be sufficient to run something like "type lookup" on these types and verifying that. Right now I get

struct string {
    unsigned long length;
    void *ptr;
}

for type lookup string. I'm guessing that after your patch the void * will change into something else. Alternatively, you could just check for the existing formatting of "target variable utf8", but leave a note that this does not check the precise formatting of the variable, just its utf8-ness. Then we can just update the expectation when the pretty printer bug is fixed.

[Geod24]

Changed the test from running to 'type lookup string'

Due to bug https://bugs.llvm.org/show_bug.cgi?id=45856 the test was originally runnable,
but a non-runnable alternative was suggested during review.

[labath]

Looks great now, just a small tweak to the test case.

lldb/test/Shell/SymbolFile/DWARF/DW_TAG_basic_type_DW_ATE_UTF_nonC.ll

20

Now that we're not running the binary, you can remove the main function, and replace %clang_host with %clang --target original-ldc2-target -c. Without that, we might as well have run the binary, because if it compiles for the host, chances are it would have run correctly too...

[Geod24]

Remove un-needed main & use x86_64-pc-gnu as target instead of host

Since we're not running the program anymore, we no longer need a main.
The comment, build step, and IR of the test were updated accordingly.

[Geod24]

Remove a trailing whitespace in TypeSystemClang.cpp

The pre-merge checks are apparently failing, and from what I can gather from the logs,
it seems to be because of a whitespace issue:

[2020-05-12T16:19:00.309Z] + clang-tidy-diff -p1 -quiet
[2020-05-12T16:19:00.309Z] + sed '/^[[:space:]]*$/d'
[2020-05-12T16:19:03.616Z] 972 warnings and 1 error generated.
[2020-05-12T16:19:03.616Z] Error while processing [...]/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp.

Since there was a trailing whitespace in that file, I added an extra commit to remove it.

[Geod24]

Add missing commit

See previous diff.

[Geod24]

Merge whitespace-removing commit into the main one

Since I can't seem to get arc to comply.

[labath]

Yeah, harbormaster is a bit temperamental these days. I wouldn't worry too much about it reporting problems clearly unrelated to your patch.

Btw, do you have commit access, or need someone to commit this for you?

[Geod24]

Good to know, thanks! It's also my first time using arc, so I had to poke around a bit. In any case I didn't seem to include that whitespace fix even after squashing the commits together. If this is fine as is, then great.

No I don't have commit access.

[labath]

Committed as e16111ce2f. I've also added checks for the dstring/wstring types.

[Geod24]

Thanks!
I suppose this won't be in the next patch release (10.0.1) but instead in the next major ?

[labath]

Yep.

lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp

case DW_ATE_unsigned_char:
     if (QualTypeMatchesBitSize(bit_size, ast, ast.UnsignedShortTy))
       return GetType(ast.UnsignedShortTy);
     break;
 
   case DW_ATE_imaginary_float:
     break;
 
   case DW_ATE_UTF:
+    switch (bit_size) {
+    case 8:
+      return GetType(ast.Char8Ty);
+    case 16:
+      return GetType(ast.Char16Ty);
+    case 32:
+      return GetType(ast.Char32Ty);
+    default:
       if (!type_name.empty()) {
         if (type_name == "char16_t")
           return GetType(ast.Char16Ty);
         if (type_name == "char32_t")
           return GetType(ast.Char32Ty);
         if (type_name == "char8_t")
           return GetType(ast.Char8Ty);
       }
+    }
     break;
   }
   // This assert should fire for anything that we don't catch above so we know
   // to fix any issues we run into.
   if (!type_name.empty()) {
     std::string type_name_str = type_name.str();
     Host::SystemLog(Host::eSystemLogError,
                     "error: need to add support for DW_TAG_base_type '%s' "
if (type) {
     } break;
 
     default: {
       if (auto *tag_type =
               llvm::dyn_cast<clang::TagType>(qual_type.getTypePtr())) {
         if (clang::TagDecl *tag_decl = tag_type->getDecl()) {
           if (level == eDescriptionLevelVerbose)
             tag_decl->dump(llvm_ostrm);
-          else 
+          else
             tag_decl->print(llvm_ostrm, 0);
         }
       } else {
         if (level == eDescriptionLevelVerbose)
           qual_type->dump(llvm_ostrm);
         else {
           std::string clang_type_name(qual_type.getAsString());
           if (!clang_type_name.empty())

lldb/test/Shell/SymbolFile/DWARF/DW_TAG_basic_type_DW_ATE_UTF_nonC.ll

+;
+; This test verifies that DWARF DIE of type DW_TAG_basic_type with DW_ATE_UTF
+; are matched based on their bit size (8, 16, 32) in addition to their name.
+;
+; This is used by languages which don't use the C(++) naming of
+; `char{8,16,32}_t`, e.g. the D programming language uses `char`, `wchar`, `dchar`.
+;
+; The D code used to generate this IR is:
+; ```
+; // Compiled with `ldc2 --mtriple=x86_64-pc-linux -betterC -g -c --output-ll utftypes.d`
+; __gshared string utf8 = "Hello";
+; __gshared wstring utf16 = "Dlang"w;
+; __gshared dstring utf32 = "World"d;
+; ```
+;
+; Note: lldb will print types differently before and after 'run'.
+;
+; RUN: %clang --target=x86_64-pc-linu -c -g -o %t %s
+; RUN: %lldb %t -o 'type lookup string' -o exit | FileCheck %s
+;
+; CHECK: struct string {
+; CHECK:     unsigned long length;
+; CHECK:     char8_t *ptr;
+; CHECK: }
+
+$_D8utftypes4utf8Aya = comdat any
+$_D8utftypes5utf16Ayu = comdat any
+$_D8utftypes5utf32Ayw = comdat any
+
+@_D8utftypes4utf8Aya = global { i64, i8* } { i64 5, i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str, i32 0, i32 0) }, comdat, align 8, !dbg !0 ; [#uses = 0]
+@.str = private unnamed_addr constant [6 x i8] c"Hello\00" ; [#uses = 1]
+@_D8utftypes5utf16Ayu = global { i64, i16* } { i64 5, i16* getelementptr inbounds ([6 x i16], [6 x i16]* @.str.1, i32 0, i32 0) }, comdat, align 8, !dbg !11 ; [#uses = 0]
+@.str.1 = private unnamed_addr constant [6 x i16] [i16 68, i16 108, i16 97, i16 110, i16 103, i16 0] ; [#uses = 1]
+@_D8utftypes5utf32Ayw = global { i64, i32* } { i64 5, i32* getelementptr inbounds ([6 x i32], [6 x i32]* @.str.2, i32 0, i32 0) }, comdat, align 8, !dbg !18 ; [#uses = 0]
+@.str.2 = private unnamed_addr constant [6 x i32] [i32 87, i32 111, i32 114, i32 108, i32 100, i32 0] ; [#uses = 1]
+
+!llvm.module.flags = !{!25}
+!llvm.dbg.cu = !{!26}
+!llvm.ident = !{!32}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "utf8", linkageName: "_D8utftypes4utf8Aya", scope: !2, file: !3, line: 1, type: !4, isLocal: false, isDefinition: true)
+!2 = !DIModule(scope: null, name: "utftypes")
+!3 = !DIFile(filename: "utftypes.d", directory: "/tmp")
+!4 = !DICompositeType(tag: DW_TAG_structure_type, name: "string", file: !3, size: 128, align: 64, elements: !5, identifier: "Aya")
+!5 = !{!6, !8}
+!6 = !DIDerivedType(tag: DW_TAG_member, name: "length", file: !3, baseType: !7, size: 64, align: 64, flags: DIFlagPublic)
+!7 = !DIBasicType(name: "ulong", size: 64, encoding: DW_ATE_unsigned)
+!8 = !DIDerivedType(tag: DW_TAG_member, name: "ptr", file: !3, baseType: !9, size: 64, align: 64, offset: 64, flags: DIFlagPublic)
+!9 = !DIDerivedType(tag: DW_TAG_pointer_type, name: "immutable(char)*", baseType: !10, size: 64, align: 64)
+!10 = !DIBasicType(name: "immutable(char)", size: 8, encoding: DW_ATE_UTF)
+!11 = !DIGlobalVariableExpression(var: !12, expr: !DIExpression())
+!12 = distinct !DIGlobalVariable(name: "utf16", linkageName: "_D8utftypes5utf16Ayu", scope: !2, file: !3, line: 2, type: !13, isLocal: false, isDefinition: true)
+!13 = !DICompositeType(tag: DW_TAG_structure_type, name: "wstring", file: !3, size: 128, align: 64, elements: !14, identifier: "Ayu")
+!14 = !{!6, !15}
+!15 = !DIDerivedType(tag: DW_TAG_member, name: "ptr", file: !3, baseType: !16, size: 64, align: 64, offset: 64, flags: DIFlagPublic)
+!16 = !DIDerivedType(tag: DW_TAG_pointer_type, name: "immutable(wchar)*", baseType: !17, size: 64, align: 64)
+!17 = !DIBasicType(name: "immutable(wchar)", size: 16, encoding: DW_ATE_UTF)
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(name: "utf32", linkageName: "_D8utftypes5utf32Ayw", scope: !2, file: !3, line: 3, type: !20, isLocal: false, isDefinition: true)
+!20 = !DICompositeType(tag: DW_TAG_structure_type, name: "dstring", file: !3, size: 128, align: 64, elements: !21, identifier: "Ayw")
+!21 = !{!6, !22}
+!22 = !DIDerivedType(tag: DW_TAG_member, name: "ptr", file: !3, baseType: !23, size: 64, align: 64, offset: 64, flags: DIFlagPublic)
+!23 = !DIDerivedType(tag: DW_TAG_pointer_type, name: "immutable(dchar)*", baseType: !24, size: 64, align: 64)
+!24 = !DIBasicType(name: "immutable(dchar)", size: 32, encoding: DW_ATE_UTF)
+!25 = !{i32 2, !"Debug Info Version", i32 3}
+!26 = distinct !DICompileUnit(language: DW_LANG_D, file: !3, producer: "LDC 1.20.1 (LLVM 9.0.1)", isOptimized: false, runtimeVersion: 1, emissionKind: FullDebug, enums: !27, globals: !28, imports: !29)
+!27 = !{}
+!28 = !{!0, !11, !18}
+!29 = !{!30}
+!30 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !31, file: !3)
+!31 = !DIModule(scope: null, name: "object")
+!32 = !{!"ldc version 1.20.1"}