[formatters] Add a libstdcpp formatter for for unordered_map, unordered_set, unordered_multimap, unordered_multiset

[danilashtefan]

Summary

This diff adds a data formatter and tests for libstdcpp's unordered_map, unordered_set, unordered_multimap, unordered_multiset

[danilashtefan]

Small cosmetic change

[danilashtefan]

Hi Walter :) This is the Draft PR for the unordered_map/set data formatter. Currently it works well in case both key and values are of the same primitive data types (except string).

Unfortunately, the following problems are encountered:

1. When key and value are of different types value is detected wrong (some garbage). I will leave the separate comment with the details

2. Currently I print the unordered_map in the following form [key] = [value]. However, the right from would be: [index] = (first = key, second = value). I am struggling through finding the way to do so

3. Whether key and/or value are of the string type, formatter cannot read the data. Is the string formatter issue.

Many thanks for your help in advance

[danilashtefan]

lldb/examples/synthetic/gnu_libstdcpp.py

77

I assume that the cause of the first issue (different key value type) is located here. self.next.GetType().GetByteSize() + self.data_size does not work in this case. Key is read, however value - not. It means that we cannot always add self.data_size.

I tried to manually figure out on the unordered_map<char, int> case what can I add instead of self.data_size (which is 1 in this case), but got no success

[wallace]

When you do your tests, don't care too much about std::string yet, as it has its own issues, but instead create a non-standard size type like

struct Foo {
  int a;
  int b;
  int c;
};

...
 std::unordered_map<int, Foo> u;

lldb/examples/synthetic/gnu_libstdcpp.py

16–24

you can simplify this

18

you are already passing stl_deref_flags when you declare this formatter in C++, so you shouldn't get a Reference type here

21

GetTemplateArgumentType(0) will give you the key, and GetTemplateArgumentType(1) should give you the value type. The idea is that in this place you get both key_type and value_type, so that you don't need to figure them out later

44

remove if not used

55

this looks weird. you are already doing a while loop in get_child_at_index that is taking you to the node you want, but you do again another loop. You have to use current instead. My suggestion is to remove this function and make everything happen in get_child_at_index with one single loop

71

why do you do this? that means that you are traversing backwards. Just move an 'index' number of times. The unordered map is unordered by definition

77

do what i mentioned above regarding fetching the key and value types. I imagine that the internal allocator is using a pair<key, value>, which means that the memory layout will be:

key
value [with an offset of key size]

77

here you are returning a child of type self.data_type, which is the key type. And we don't want that. We want to do the same thing that the map formatter does for getting the pair<key, value> and returning it: https://github.com/llvm-mirror/lldb/blob/master/examples/synthetic/gnu_libstdcpp.py#L366
Notice how data type in that case is gotten here https://github.com/llvm-mirror/lldb/blob/master/examples/synthetic/gnu_libstdcpp.py#L367 and is actually the pair. And then they pair is returned here https://github.com/llvm-mirror/lldb/blob/master/examples/synthetic/gnu_libstdcpp.py#L425

that means that you don't need to individually find the key and the value, you only need to find the pair and return the pair

Btw, if raw print an unordered_map, I get this:

(std::unordered_map<int, double, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<const int, double> > >)

that means that the fifth template argument contains the pair. I've confirmed that also libcxx works the same way

[danilashtefan]

All the changes are done and tests pass

[danilashtefan]

Hi Walter,

Thank you for the hints. Everything works now, including the string case. There was a small difficulty with set case, but then I figured out that for set case I need to take not the 5th, but 4th template argument. I will be waiting for your comments. After this I will unify the tests across platforms. Thank you!

[danilashtefan]

Cosmetic change

[danilashtefan]

Some naming corrections

[wallace]

just some minor changes.
Could you unify these tests with libcxx?

lldb/examples/synthetic/gnu_libstdcpp.py

10

add a few empty lines before this to avoid confusing that comment with this specific formatter. Besides that, rename this to StdUnorderedMapLikeSynthProvider and add a comment similar to

"""
 This formatter can be applied to all
 unordered map-like structures (unordered_map, unordered_multimap, unordered_set, unordered_multiset)
"""

25

an exception is too much. Normally we just fail silently, as it obvious to the user when the formatter fails and investigation is needed. Just return "map" if "set" is not present in the type name.

29

this comment

1. type of std::pair<key, value> is the first template
2. argument type of the 4th template argument to std::map and
3. 3rd template argument for std::set. That's why
4. we need to know kind of the object

should be here

32

if we used something that is not a map or set, this data type will be a dummy object that won't fail but won't do anything useful

46

you see? we fail silently by default

83

remove this comment, as this information is easily available to the user

86

this one is fine

lldb/test/API/functionalities/data-formatter/data-formatter-stl/generic/unordered/main.cpp

5 ↗

(On Diff #387084)

also remove this, you barely save a few characters by not typing std::string

7–13 ↗

(On Diff #387084)

avoid using these defines, they just obscure the code when reading

[danilashtefan]

All the above-mentioned is changed. Thank you!

[wallace]

lldb/examples/synthetic/gnu_libstdcpp.py

110–115

[danilashtefan]

Done!

[wallace]

good job!

lldb/examples/synthetic/gnu_libstdcpp.py

 from __future__ import division
 import re
 import lldb.formatters.Logger
 
 # C++ STL formatters for LLDB
 # These formatters are based upon the version of the GNU libstdc++
 # as it ships with Mac OS X 10.6.8 thru 10.8.0
 # You are encouraged to look at the STL implementation for your platform
 # before relying on these formatters to do the right thing for your setup
+class StdUnorderedMapSynthProvider:
+    def __init__(self, valobj, dict):
+        self.valobj = valobj
+        self.count = None
+
+    def extract_type(self):
+        logger = lldb.formatters.Logger.Logger()
+        list_type = self.valobj.GetType().GetUnqualifiedType()
+        if list_type.IsReferenceType():
+            list_type = list_type.GetDereferencedType()
+        if list_type.GetNumberOfTemplateArguments() > 0:
+            data_type = list_type.GetTemplateArgumentType(0)
+        else:
+            data_type = None
+        return data_type
+
+    def update(self): 
+        logger = lldb.formatters.Logger.Logger()
+        # preemptively setting this to None - we might end up changing our mind
+        # later
+        self.count = None
+        try:
+            self.head = self.valobj.GetChildMemberWithName('_M_h')
+            self.before_begin = self.head.GetChildMemberWithName('_M_before_begin')
+            self.next = self.before_begin.GetChildMemberWithName('_M_nxt')
+            self.data_type = self.extract_type()
+            self.skip_size = self.head.GetType().GetByteSize()
+            logger >> "Data type is this: " + str(self.data_type)
+            self.data_size = self.data_type.GetByteSize()
+        except:
+            pass
+        return False
+
+    def get_child_index(self, name):
+        logger = lldb.formatters.Logger.Logger()
+        try:
+            return int(name.lstrip('[').rstrip(']'))
+        except:
+            return -1
+
+    def get_child_key(self, index):
+        try:
+            logger = lldb.formatters.Logger.Logger()
+            offset = self.count - index - 1
+            current = self.next     
+            while offset > 0:
+                current = current.GetChildMemberWithName('_M_nxt')
+                offset = offset - 1
+            return current.CreateChildAtOffset('[' + str(index) + ']', self.next.GetType().GetByteSize(), self.data_type)
+        except:
+            logger >> "Error in get_child_key"
+            return None
+
+    def get_child_at_index(self, index):
+        logger = lldb.formatters.Logger.Logger()
+        logger >> "Being asked to fetch child[" + str(index) + "]"
+        if index < 0:
+            return None
+        if index >= self.num_children():
+            return None
+        try:
+            offset = self.count - index - 1
+            current = self.next     
+            while offset > 0:
+                current = current.GetChildMemberWithName('_M_nxt')
+                offset = offset - 1
+            key = self.get_child_key(index)
+            return current.CreateChildAtOffset('[' + str(key.GetValue()) + ']', self.next.GetType().GetByteSize() + 9, self.data_type)
+        
+        except:
+            logger >> "Error in get_child_at_index"
+            return None
+
+    def num_children(self):
+        logger = lldb.formatters.Logger.Logger()
+        if self.count is None:
+            self.count = self.num_children_impl()
+        return self.count
+
+    def num_children_impl(self):
+        logger = lldb.formatters.Logger.Logger()
+        try:
+            count = self.head.GetChildMemberWithName('_M_element_count').GetValueAsUnsigned(0)
+            logger >> "I have " + str(count) + " children available"
+            return count
+        except:
+            logger >> "Could not determine the size"
+            return 0
 
 
 class StdListSynthProvider:
 
     def __init__(self, valobj, dict):
         logger = lldb.formatters.Logger.Logger()
         self.valobj = valobj
         self.count = None
         logger >> "Providing synthetic children for a list named " + \
             str(valobj.GetName())
 
     def next_node(self, node):
         logger = lldb.formatters.Logger.Logger()
         return node.GetChildMemberWithName('_M_next')
 
     def is_valid(self, node):
         logger = lldb.formatters.Logger.Logger()
         valid = self.value(self.next_node(node)) != self.node_address
         if valid:
             logger >> "%s is valid" % str(self.valobj.GetName())
         else:
             logger >> "synthetic value is not valid"
         return valid
 
     def value(self, node):
         logger = lldb.formatters.Logger.Logger()

lldb/source/Plugins/Language/CPlusPlus/CPlusPlusLanguage.cpp

cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(
           stl_deref_flags,
           "lldb.formatters.cpp.gnu_libstdcpp.StdMapLikeSynthProvider")));
   cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(
       RegularExpression("^std::multiset<.+> >(( )?&)?$"),
       SyntheticChildrenSP(new ScriptedSyntheticChildren(
           stl_deref_flags,
           "lldb.formatters.cpp.gnu_libstdcpp.StdMapLikeSynthProvider")));
   cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(
+      RegularExpression("^std::unordered_(multi)?(map|set)<.+> >$"),
+      SyntheticChildrenSP(new ScriptedSyntheticChildren(
+          stl_deref_flags,
+          "lldb.formatters.cpp.gnu_libstdcpp.StdUnorderedMapSynthProvider")));
+  cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(
       RegularExpression("^std::(__cxx11::)?list<.+>(( )?&)?$"),
       SyntheticChildrenSP(new ScriptedSyntheticChildren(
           stl_synth_flags,
           "lldb.formatters.cpp.gnu_libstdcpp.StdListSynthProvider")));
   stl_summary_flags.SetDontShowChildren(false);
   stl_summary_flags.SetSkipPointers(false);
   cpp_category_sp->GetRegexTypeSummariesContainer()->Add(
       RegularExpression("^std::bitset<.+>(( )?&)?$"),
cpp_category_sp->GetRegexTypeSummariesContainer()->Add(
       RegularExpression("^std::multimap<.+> >(( )?&)?$"),
       TypeSummaryImplSP(
           new StringSummaryFormat(stl_summary_flags, "size=${svar%#}")));
   cpp_category_sp->GetRegexTypeSummariesContainer()->Add(
       RegularExpression("^std::multiset<.+> >(( )?&)?$"),
       TypeSummaryImplSP(
           new StringSummaryFormat(stl_summary_flags, "size=${svar%#}")));
   cpp_category_sp->GetRegexTypeSummariesContainer()->Add(
+      RegularExpression("^std::unordered_(multi)?(map|set)<.+> >$"),
+      TypeSummaryImplSP(
+          new StringSummaryFormat(stl_summary_flags, "size=${svar%#}")));
+  cpp_category_sp->GetRegexTypeSummariesContainer()->Add(
       RegularExpression("^std::(__cxx11::)?list<.+>(( )?&)?$"),
       TypeSummaryImplSP(
           new StringSummaryFormat(stl_summary_flags, "size=${svar%#}")));
 
   AddCXXSynthetic(
       cpp_category_sp,
       lldb_private::formatters::LibStdcppVectorIteratorSyntheticFrontEndCreator,
       "std::vector iterator synthetic children",