Implement formatter for std::vector<bool, ...> of libstdc++ in Python.

[sivachandra]

Summary

The existing formatter in C++ has been removed as it was not being used.
The associated test TestDataFormatterStdVBool.py has been enabled for
both Clang and GCC on Linux.

[sivachandra]

Fix an indent in the Python changes.

[sivachandra]

Setting the Python changes in perspective: Though it appears in the diff as if I have added a formatter for std::vector<non-bool> as well, I have only moved it into an implementation class StdVectorSynthProvider.StdVectorImplementation. What I have really added is the class StdVectorSynthProvider.StdVBoolImplementation. The main class StdVectorSynthProvider has been refactored to reflect the common parts.

[clayborg]

Why did you move code from C++ into Python? We want this to be in C++ so it works on systems without python.

[sivachandra]

The existing formatter for std::vector<bool, ...> from libstdc++ is not used. If I run "type synthetic list", I see this:

...
-----------------------
Category: gnu-libstdc++ (enabled)
-----------------------
Regex-based synthetic providers (slower): 
^std::_Rb_tree_iterator<.+>$:  Generator at 0x7f13f9f0c310 - std::map iterator synthetic children
^__gnu_cxx::__normal_iterator<.+>$:  Generator at 0x7f13f9f0c390 - std::vector iterator synthetic children
^std::vector<.+>(( )?&)?$:  Python class lldb.formatters.cpp.gnu_libstdcpp.StdVectorSynthProvider
^std::map<.+> >(( )?&)?$:  Python class lldb.formatters.cpp.gnu_libstdcpp.StdMapSynthProvider
^std::list<.+>(( )?&)?$:  Python class lldb.formatters.cpp.gnu_libstdcpp.StdListSynthProvider
...

Formatters for map iterator and vector iterator are implemented in C++, but the formatters for std::vector, std::list and std::map are all implemented in Python. The approach I have taken is to extend the existing std::vector formatter (Python) to handle std::vector<bool, ...> as a special case. Since the c++ implementation of std::vector<bool, ...> is anyway not used, I removed it.

Do you suggest that we should actually be moving the Python formatters into C++?

[clayborg]

As long as the C++ code you removed was not being used that is ok.

Just spoke with Enrico Granata, our data formatters guys, and he said everything looks good.

[sivachandra]

Mention in the commit message that the removed c++ implementation
of the formatter was not being used.

examples/synthetic/gnu_libstdcpp.py

def update(self):
     except:
       pass
 
   def has_children(self):
     return True
 
 class StdVectorSynthProvider:
 
-  def __init__(self, valobj, dict):
-    logger = lldb.formatters.Logger.Logger()
-    self.count = None
+  class StdVectorImplementation(object):
+    def __init__(self, valobj):
       self.valobj = valobj
-    logger >> "Providing synthetic children for a vector named " + str(valobj.GetName())
+      self.count = None
 
     def num_children(self):
-    if self.count == None:
-      self.count = self.num_children_impl()
-    return self.count
+                        if self.count == None:
+                                self.count = self.num_children_impl()
+                        return self.count
 
-  def num_children_impl(self):
+                def num_children_impl(self):
       try:
         start_val = self.start.GetValueAsUnsigned(0)
         finish_val = self.finish.GetValueAsUnsigned(0)
         end_val  = self.end.GetValueAsUnsigned(0)
         # Before a vector has been constructed, it will contain bad values
         # so we really need to be careful about the length we return since
         # unitialized data can cause us to return a huge number. We need
         # to also check for any of the start, finish or end of storage values
-      # being zero (NULL). If any are, then this vector has not been 
+        # being zero (NULL). If any are, then this vector has not been
         # initialized yet and we should return zero
 
         # Make sure nothing is NULL
         if start_val == 0 or finish_val == 0 or end_val == 0:
           return 0
         # Make sure start is less than finish
         if start_val >= finish_val:
           return 0
         # Make sure finish is less than or equal to end of storage
         if finish_val > end_val:
           return 0
 
         # if we have a struct (or other data type that the compiler pads to native word size)
         # this check might fail, unless the sizeof() we get is itself incremented to take the
         # padding bytes into account - on current clang it looks like this is the case
         num_children = (finish_val-start_val)
         if (num_children % self.data_size) != 0:
           return 0
         else:
           num_children = num_children/self.data_size
         return num_children
       except:
         return 0;
 
-  def get_child_index(self,name):
-    try:
-      return int(name.lstrip('[').rstrip(']'))
-    except:
-      return -1
-
-  def get_child_at_index(self,index):
+    def get_child_at_index(self, index):
       logger = lldb.formatters.Logger.Logger()
       logger >> "Retrieving child " + str(index)
       if index < 0:
         return None;
       if index >= self.num_children():
         return None;
       try:
         offset = index * self.data_size
         return self.start.CreateChildAtOffset('['+str(index)+']',offset,self.data_type)
       except:
         return None
 
     def update(self):
       # preemptively setting this to None - we might end up changing our mind later
       self.count = None
       try:
         impl = self.valobj.GetChildMemberWithName('_M_impl')
         self.start = impl.GetChildMemberWithName('_M_start')
         self.finish = impl.GetChildMemberWithName('_M_finish')
         self.end = impl.GetChildMemberWithName('_M_end_of_storage')
         self.data_type = self.start.GetType().GetPointeeType()
         self.data_size = self.data_type.GetByteSize()
         # if any of these objects is invalid, it means there is no point in trying to fetch anything
         if self.start.IsValid() and self.finish.IsValid() and self.end.IsValid() and self.data_type.IsValid():
           self.count = None
         else:
           self.count = 0
       except:
         pass
-      
+      return True
+
+  class StdVBoolImplementation(object):
+    def __init__(self, valobj, bool_type):
+      self.valobj = valobj
+      self.bool_type = bool_type
+      self.valid = False
+
+    def num_children(self):
+      if self.valid:
+        start = self.start_p.GetValueAsUnsigned(0)
+        finish = self.finish_p.GetValueAsUnsigned(0)
+        offset = self.offset.GetValueAsUnsigned(0)
+        if finish >= start:
+          return (finish - start) * 8 + offset
+      return 0
+
+    def get_child_at_index(self, index):
+      if index >= self.num_children():
+        return None
+      byte_offset = index / 8
+      bit_offset = index % 8
+      data = self.start_p.GetPointeeData()
+      bit = data.GetUnsignedInt8(lldb.SBError(), byte_offset) & (1 << bit_offset)
+      if bit != 0:
+        value_expr = "(bool)true"
+      else:
+        value_expr = "(bool)false"
+      return self.valobj.CreateValueFromExpression("[%d]" % index, value_expr)
+    def update(self):
+      try:
+        m_impl = self.valobj.GetChildMemberWithName('_M_impl')
+        self.m_start = m_impl.GetChildMemberWithName('_M_start')
+        self.m_finish = m_impl.GetChildMemberWithName('_M_finish')
+        self.start_p = self.m_start.GetChildMemberWithName('_M_p')
+        self.finish_p = self.m_finish.GetChildMemberWithName('_M_p')
+        self.offset = self.m_finish.GetChildMemberWithName('_M_offset')
+        self.valid = True
+      except:
+        self.valid = False
+      return True
+
+  def __init__(self, valobj, dict):
+    logger = lldb.formatters.Logger.Logger()
+    first_template_arg_type = valobj.GetType().GetTemplateArgumentType(0)
+    if str(first_template_arg_type.GetName()) == "bool":
+      self.impl = self.StdVBoolImplementation(valobj, first_template_arg_type)
+    else:
+      self.impl = self.StdVectorImplementation(valobj)
+    logger >> "Providing synthetic children for a vector named " + str(valobj.GetName())
+
+  def num_children(self):
+    return self.impl.num_children()
+
+  def get_child_index(self,name):
+    try:
+      return int(name.lstrip('[').rstrip(']'))
+    except:
+      return -1
+
+  def get_child_at_index(self, index):
+    return self.impl.get_child_at_index(index)
+
+  def update(self):
+    return self.impl.update()
 
   def has_children(self):
     return True
 
 
 class StdMapSynthProvider:
 
   def __init__(self, valobj, dict):

include/lldb/DataFormatters/CXXFormatterFunctions.h

namespace formatters
             std::map<size_t,lldb::ValueObjectSP> m_children;
         };
         
         SyntheticChildrenFrontEnd* LibcxxVectorBoolSyntheticFrontEndCreator (CXXSyntheticChildren*, lldb::ValueObjectSP);
         
         bool
         LibcxxContainerSummaryProvider (ValueObject& valobj, Stream& stream, const TypeSummaryOptions& options);
         
-        class LibstdcppVectorBoolSyntheticFrontEnd : public SyntheticChildrenFrontEnd
-        {
-        public:
-            LibstdcppVectorBoolSyntheticFrontEnd (lldb::ValueObjectSP valobj_sp);
-            
-            virtual size_t
-            CalculateNumChildren ();
-            
-            virtual lldb::ValueObjectSP
-            GetChildAtIndex (size_t idx);
-            
-            virtual bool
-            Update();
-            
-            virtual bool
-            MightHaveChildren ();
-            
-            virtual size_t
-            GetIndexOfChildWithName (const ConstString &name);
-            
-            virtual
-            ~LibstdcppVectorBoolSyntheticFrontEnd ();
-        private:
-            ExecutionContextRef m_exe_ctx_ref;
-            uint64_t m_count;
-            lldb::addr_t m_base_data_address;
-            EvaluateExpressionOptions m_options;
-        };
-        
-        SyntheticChildrenFrontEnd* LibstdcppVectorBoolSyntheticFrontEndCreator (CXXSyntheticChildren*, lldb::ValueObjectSP);
-        
         class LibstdcppMapIteratorSyntheticFrontEnd : public SyntheticChildrenFrontEnd
         {
         public:
             LibstdcppMapIteratorSyntheticFrontEnd (lldb::ValueObjectSP valobj_sp);
             
             virtual size_t
             CalculateNumChildren ();
             

source/DataFormatters/LibStdcpp.cpp

 #include "lldb/Symbol/ClangASTContext.h"
 #include "lldb/Target/ObjCLanguageRuntime.h"
 #include "lldb/Target/Target.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::formatters;
 
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::LibstdcppVectorBoolSyntheticFrontEnd (lldb::ValueObjectSP valobj_sp) :
-SyntheticChildrenFrontEnd(*valobj_sp.get()),
-m_exe_ctx_ref(),
-m_count(0),
-m_base_data_address(0),
-m_options()
-{
-    if (valobj_sp)
-        Update();
-    m_options.SetCoerceToId(false);
-    m_options.SetUnwindOnError(true);
-    m_options.SetKeepInMemory(true);
-    m_options.SetUseDynamic(lldb::eDynamicCanRunTarget);
-}
-
-size_t
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::CalculateNumChildren ()
-{
-    return m_count;
-}
-
-lldb::ValueObjectSP
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::GetChildAtIndex (size_t idx)
-{
-    if (idx >= m_count)
-        return ValueObjectSP();
-    if (m_base_data_address == 0 || m_count == 0)
-        return ValueObjectSP();
-    size_t byte_idx = (idx >> 3); // divide by 8 to get byte index
-    size_t bit_index = (idx & 7); // efficient idx % 8 for bit index
-    lldb::addr_t byte_location = m_base_data_address + byte_idx;
-    ProcessSP process_sp(m_exe_ctx_ref.GetProcessSP());
-    if (!process_sp)
-        return ValueObjectSP();
-    uint8_t byte = 0;
-    uint8_t mask = 0;
-    Error err;
-    size_t bytes_read = process_sp->ReadMemory(byte_location, &byte, 1, err);
-    if (err.Fail() || bytes_read == 0)
-        return ValueObjectSP();
-    switch (bit_index)
-    {
-        case 0:
-            mask = 1; break;
-        case 1:
-            mask = 2; break;
-        case 2:
-            mask = 4; break;
-        case 3:
-            mask = 8; break;
-        case 4:
-            mask = 16; break;
-        case 5:
-            mask = 32; break;
-        case 6:
-            mask = 64; break;
-        case 7:
-            mask = 128; break;
-        default:
-            return ValueObjectSP();
-    }
-    bool bit_set = ((byte & mask) != 0);
-    Target& target(process_sp->GetTarget());
-    ValueObjectSP retval_sp;
-    if (bit_set)
-        target.EvaluateExpression("(bool)true", NULL, retval_sp);
-    else
-        target.EvaluateExpression("(bool)false", NULL, retval_sp);
-    StreamString name; name.Printf("[%" PRIu64 "]", (uint64_t)idx);
-    if (retval_sp)
-        retval_sp->SetName(ConstString(name.GetData()));
-    return retval_sp;
-}
-
-/*((std::vector<std::allocator<bool> >) vBool = {
- (std::_Bvector_base<std::allocator<bool> >) std::_Bvector_base<std::allocator<bool> > = {
- (std::_Bvector_base<std::allocator<bool> >::_Bvector_impl) _M_impl = {
- (std::_Bit_iterator) _M_start = {
- (std::_Bit_iterator_base) std::_Bit_iterator_base = {
- (_Bit_type *) _M_p = 0x0016b160
- (unsigned int) _M_offset = 0
- }
- }
- (std::_Bit_iterator) _M_finish = {
- (std::_Bit_iterator_base) std::_Bit_iterator_base = {
- (_Bit_type *) _M_p = 0x0016b16c
- (unsigned int) _M_offset = 16
- }
- }
- (_Bit_type *) _M_end_of_storage = 0x0016b170
- }
- }
- }
- */
-
-bool
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::Update()
-{
-    ValueObjectSP valobj_sp = m_backend.GetSP();
-    if (!valobj_sp)
-        return false;
-    if (!valobj_sp)
-        return false;
-    m_exe_ctx_ref = valobj_sp->GetExecutionContextRef();
-    
-    ValueObjectSP m_impl_sp(valobj_sp->GetChildMemberWithName(ConstString("_M_impl"), true));
-    if (!m_impl_sp)
-        return false;
-    
-    ValueObjectSP m_start_sp(m_impl_sp->GetChildMemberWithName(ConstString("_M_start"), true));
-    ValueObjectSP m_finish_sp(m_impl_sp->GetChildMemberWithName(ConstString("_M_finish"), true));
-    
-    ValueObjectSP start_p_sp, finish_p_sp, finish_offset_sp;
-    
-    if (!m_start_sp || !m_finish_sp)
-        return false;
-    
-    start_p_sp = m_start_sp->GetChildMemberWithName(ConstString("_M_p"), true);
-    finish_p_sp = m_finish_sp->GetChildMemberWithName(ConstString("_M_p"), true);
-    finish_offset_sp = m_finish_sp->GetChildMemberWithName(ConstString("_M_offset"), true);
-    
-    if (!start_p_sp || !finish_offset_sp || !finish_p_sp)
-        return false;
-    
-    m_base_data_address = start_p_sp->GetValueAsUnsigned(0);
-    if (!m_base_data_address)
-        return false;
-    
-    lldb::addr_t end_data_address(finish_p_sp->GetValueAsUnsigned(0));
-    if (!end_data_address)
-        return false;
-    
-    if (end_data_address < m_base_data_address)
-        return false;
-    else
-        m_count = finish_offset_sp->GetValueAsUnsigned(0) + (end_data_address-m_base_data_address)*8;
-    
-    return true;
-}
-
-bool
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::MightHaveChildren ()
-{
-    return true;
-}
-
-size_t
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::GetIndexOfChildWithName (const ConstString &name)
-{
-    if (!m_count || !m_base_data_address)
-        return UINT32_MAX;
-    const char* item_name = name.GetCString();
-    uint32_t idx = ExtractIndexFromString(item_name);
-    if (idx < UINT32_MAX && idx >= CalculateNumChildren())
-        return UINT32_MAX;
-    return idx;
-}
-
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEnd::~LibstdcppVectorBoolSyntheticFrontEnd ()
-{}
-
-SyntheticChildrenFrontEnd*
-lldb_private::formatters::LibstdcppVectorBoolSyntheticFrontEndCreator (CXXSyntheticChildren*, lldb::ValueObjectSP valobj_sp)
-{
-    if (!valobj_sp)
-        return NULL;
-    return (new LibstdcppVectorBoolSyntheticFrontEnd(valobj_sp));
-}
-
 /*
  (std::_Rb_tree_iterator<std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) ibeg = {
  (_Base_ptr) _M_node = 0x0000000100103910 {
  (std::_Rb_tree_color) _M_color = _S_black
  (std::_Rb_tree_node_base::_Base_ptr) _M_parent = 0x00000001001038c0
  (std::_Rb_tree_node_base::_Base_ptr) _M_left = 0x0000000000000000
  (std::_Rb_tree_node_base::_Base_ptr) _M_right = 0x0000000000000000
  }

test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

class StdVBoolDataFormatterTestCase(TestBase):
     @dsym_test
     @skipIfDarwin
     def test_with_dsym_and_run_command(self):
         """Test data formatter commands."""
         self.buildDsym()
         self.data_formatter_commands()
 
     @expectedFailureFreeBSD("llvm.org/pr20548") # fails to build on lab.llvm.org buildbot
-    @expectedFailureLinux # non-core functionality, need to reenable and fix
-                          # later (DES 2014.11.07). Most likely failing because
-                          # of mis-match is version of libstdc++ supported by
-                          # the data-formatters.
     @dwarf_test
     @skipIfWindows # http://llvm.org/pr21800
     @skipIfDarwin
     def test_with_dwarf_and_run_command(self):
         """Test data formatter commands."""
         self.buildDwarf()
         self.data_formatter_commands()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @expectedFailureGcc # llvm.org/pr15301: lldb does not print the correct sizes of STL containers when building with GCC
     @expectedFailureIcc # llvm.org/pr15301: lldb does not print the correct sizes of STL containers when building with ICC
     def data_formatter_commands(self):
         """Test that that file and class static variables display correctly."""
         self.runCmd("file a.out", CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line (self, "main.cpp", self.line, num_expected_locations=-1)
 
         self.runCmd("run", RUN_SUCCEEDED)