This is an archive of the discontinued LLVM Phabricator instance.

[LLDB][NativePDB] Fix struct layout when it has anonymous unions.
ClosedPublic

Authored by zequanwu on Sep 28 2022, 7:57 PM.

Details

Summary

Previously, lldb mistook fields in anonymous union in a struct as the direct
field of the struct, which causes lldb crashes due to multiple fields sharing
the same offset in a struct. This patch fixes it.

MSVC generated pdb doesn't have the debug info entity representing a anonymous
union in a struct. It looks like the following:

struct S {
union {
  char c;
  int i;
};
};

0x1004 | LF_FIELDLIST [size = 40]
         - LF_MEMBER [name = `c`, Type = 0x0070 (char), offset = 0, attrs = public]
         - LF_MEMBER [name = `i`, Type = 0x0074 (int), offset = 0, attrs = public]
0x1005 | LF_STRUCTURE [size = 32] `S`
         unique name: `.?AUS@@`
         vtable: <no type>, base list: <no type>, field list: 0x1004

Clang generated pdb is similar, though due to the bug,
it's not more useful than the debug info above. But that's not very relavent,
lldb should still be able to understand MSVC geneerated pdb.

0x1003 | LF_UNION [size = 60] `S::<unnamed-tag>`
         unique name: `.?AT<unnamed-type-$S1>@S@@`
         field list: <no type>
         options: forward ref (= 0x1003) | has unique name | is nested, sizeof 0
0x1004 | LF_FIELDLIST [size = 40]
         - LF_MEMBER [name = `c`, Type = 0x0070 (char), offset = 0, attrs = public]
         - LF_MEMBER [name = `i`, Type = 0x0074 (int), offset = 0, attrs = public]
         - LF_NESTTYPE [name = ``, parent = 0x1003]
0x1005 | LF_STRUCTURE [size = 32] `S`
         unique name: `.?AUS@@`
         vtable: <no type>, base list: <no type>, field list: 0x1004
         options: contains nested class | has unique name, sizeof 4
0x1006 | LF_FIELDLIST [size = 28]
         - LF_MEMBER [name = `c`, Type = 0x0070 (char), offset = 0, attrs = public]
         - LF_MEMBER [name = `i`, Type = 0x0074 (int), offset = 0, attrs = public]
0x1007 | LF_UNION [size = 60] `S::<unnamed-tag>`
         unique name: `.?AT<unnamed-type-$S1>@S@@`
         field list: 0x1006
         options: has unique name | is nested | sealed, sizeof

This patch delays the FieldDecl creation when travesing LF_FIELDLIST so we know
if there are multiple fields are in the same offsets and are able to group them
into different anonymous unions based on offsets. Nested anonymous union will
be flatten into one anonymous union, because we simply don't have that info, but
they are equivalent in terms of union layout.

Diff Detail

Event Timeline

zequanwu created this revision.Sep 28 2022, 7:57 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 28 2022, 7:57 PM
zequanwu requested review of this revision.Sep 28 2022, 7:57 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 28 2022, 7:57 PM

The fact that MSVC does not store all of the anonymous union data is unfortunate, though not entirely surprising, given that the goal of debug info never was to offer an exact reconstruction of the source AST.

That, I'm not sure if checking for matching initial offsets is sufficient to create a semblance of a consistent structure definition. What will this code do with structures like this:

struct S3 {
  char x[3];
};
struct A {
  union {
    struct {
      char c1;
      S3 s1;
    };
    struct {
      S3 s2;
      char c2;
    };
  };
} a;

In this case, a.s1 and a.c2 overlap, but they don't share the same starting offset. If the compiler does not provide data for anonymous structs as well, then I think this algorithm will need to be more complicated. I'm not even sure they can be reconstructed correctly, as the anonymous structs and unions can nest indefinitely. Maybe it would be sufficient to represent this as a "union of structs", where each struct gets (greedily) packed with as many members as it can hold, and we create as many structs as we need (we can replace the struct with a simple member if it would hold just one member)?

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
334

How are these opaque ids computed? Will they produce unique IDs if you have two structs like this close together? Might it be better to just make a global (local to a symbol file or something) counter and always increment/decrement that when creating another type ?

zequanwu updated this revision to Diff 465565.Oct 5 2022, 2:59 PM
zequanwu marked an inline comment as done.

Updated to handle the nested anonymous union and struct situation.
It's done by:

  1. Create the record layout with Field which could be a field member or an anonymous struct or an anonymous union. If it's struct/union, it has fields to refer to its members.
  2. Create the AST from the record layout above.
labath added inline comments.Oct 6 2022, 7:30 AM
lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
330

Now multiple symbol files can race when accessing this variable (and this also introduces a strange interaction between two supposedly-independent symbol files). Better make this member variable of the parent symbol file, or something like that.

391–442

This could use a high-level explanation of the algorithm. Like, I know it tries to stuff the fields into structs and unions, but I wasn't able to figure out how it does that, and what are the operating invariants.

The thing I like about this algorithm, is that the most complicated part (the thing I highlighted) is basically just playing with numbers and it is independent of any complex objects and state. If this part is separated out into a separate function, then it would be perfectly suitable for unit testing, and the unit tests could also serve as documentation/examples of the kinds of transformations that the algorithm does.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.h
54
63

m_start_offset ?

64

Can we drop the SP part? I think that the owning references (I guess that's this field) could be just plain Field instances (unique_ptr<Field> at worst), and the rest could just be plain pointers and references.

lldb/test/Shell/SymbolFile/NativePDB/class_layout.cpp
54–60

I think we should exclude all of these DefinitionData fields from the test expectation, as they are largely irrelevant to the test (and they also obfuscate it very well). Otherwise, people will have to update these whenever a new field gets added.

I don't know whether it contains the thing you wanted to test (as I don't know what that is), but the type lookup C output will contain the general structure of the type, and it will be a lot more readable.

zequanwu updated this revision to Diff 465933.Oct 6 2022, 4:57 PM
zequanwu marked 5 inline comments as done.

Add unittests and address comments.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
391–442

Moved into a separate function and added unit tests.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.h
64

The Field object is shared between fields and m_fields. And we can't have Field member instance inside Field class.

lldb/test/Shell/SymbolFile/NativePDB/class_layout.cpp
54–60

Thanks, I didn't know that before.

The test looks great, and the comments have helped. I still have a couple of questions about the algorithm though.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
431

This shadowing the fields member confused me for quite some time. Could you choose a different name for one of them?

459

Can parent be a union here? Would the algorithm still be correct?

471

IIUC, this code is reached when the parent object is a union. However, the parent is chosen such that it ends before the start of these new fields? Therefore its start address will be before the start of these fields as well. Is it correct to add the fields to the union under these circumstances, or am I misunderstanding something?

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.h
64

You can't have a Field member, but you can have a Field*, unique_ptr<Field> and std::vector<Field> members. SmallVector<Field> is also not possible, for reasons that are mostly obvious, but then again, storing a pointer inside a SmallVector negates most of the benefits that one could hope to gain by using it.

My point is that that using a shared pointer makes it harder to understand the relationships between the objects because it obfuscates the ownership aspect. Sometimes that is unavoidable, like when there just isn't a single object that can own some other object (although llvm mostly avoids that by putting the ownership inside some "context" object), but it's not clear to me why that would be the case here.

zequanwu updated this revision to Diff 466575.Oct 10 2022, 11:44 AM
zequanwu marked 2 inline comments as done.

Address comments.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
459

parent could only be union when the top level record is a union (at this line Member *parent = &record;). That's the only case when we add an union into end_offset_map. In that case, all the fields would start at the same offset and we only iterate the loop for (auto &pair : fields) { once and then we are done.
Otherwise, we only insert {end offset, struct/field} into end_offset_map where field must be within an union.

471

This is a special case to handle the case when top level record is an union. That's the only case we would reach here. This is to avoid adding unnecessary nested union. For example, the unit test TestAnonymousUnionInUnion would become the following if we always add an anonymous union:

union {
  union {
    m1;
    m2;
    m3;
    m4;
  }
}
labath added inline comments.Oct 12 2022, 5:36 AM
lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
459

Are you sure about that? I've created what I think is a counterexample to these statements. Here a top-level union contains a bunch of sequential fields, which is perfectly possible if the only member of that union is an anonymous struct which contains those fields.

I don't think that's what supposed to happen in this case, but I'm open to being convinced otherwise.

(I've also rewritten the test logic so it produces better error messages than "false is not true".)

zequanwu updated this revision to Diff 467266.Oct 12 2022, 2:43 PM
  • Handle the case when anonymous struct inside an union.
  • Append labath's changes.
lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
459

Thanks for writing the counter example and making better error message. I appended your changes in this file.
I updated to handle the case when parent is an union here by creating an anonymous struct to hold it.

labath accepted this revision.Oct 13 2022, 3:36 AM

Looks good. I suggested some changes, which I hope will reduce duplication and better empasize the differences between the various branches in the code. I think I understand the algorithm now, and I believe it is ok, though I can't escape the feeling that this could have been done in a simpler way.

lldb/source/Plugins/SymbolFile/NativePDB/UdtRecordCompleter.cpp
459–469

The current version will create a needless nested struct in case of a union with a single member. I think it should be possible to just insert a single field first, and let it be converted to a struct later -- if it is necessary.

471–491

This should be equivalent but shorter, and -- I hope -- more obvious.

This revision is now accepted and ready to land.Oct 13 2022, 3:36 AM
zequanwu updated this revision to Diff 467573.Oct 13 2022, 12:43 PM
zequanwu marked 2 inline comments as done.

Address comments

This revision was landed with ongoing or failed builds.Oct 13 2022, 12:44 PM
This revision was automatically updated to reflect the committed changes.
lldb/test/Shell/SymbolFile/NativePDB/global-classes.cpp