Fix libstdc++ data formatters on Ubuntu 15.10 x86_64

[tfiala]

Summary

This change fixes data formatters for libstdc++ 6.0.21:

• Adds std::string data formatter support in C++ (rather than Python) for 6.0.21+.
• Uses std::list count field support when present.
• Fixes std container and std::string failing tests on Ubuntu 15.10 x86_64.
• Fixes what looks like a broken compilation on TOT which assumes libc++ is used but doesn't guarantee it, then uses libc++ details, which breaks when compiling with libstdc++. (The test would later figure it out, but only after the failing build step). Modified the using of libc++ to define a LLDB_USING_LIBCPP flag when we're really compiling with it, and conditionally compile the libc++-specific code so it doesn't break when built with libstdc++.

Anybody on the Linux/BSD side, feel free to review.

This has been tested on Ubuntu 14.04 x86_64 (with libstdc++ 6.0.20) and Ubuntu 15.10 x86_64 (with libstdc++ 6.0.21). Both built using clang-3.6 and using clang-3.6 inferiors.

[granata.enrico]

Looks good to me

[labath]

test/make/Makefile.rules

304

Thanks for fixing the build.

When I get libc++ detection and skipping logic centralized, I'd like to remove this magic and make self.build() fail with a hard error in the case when the Makefile requests libc++ use, but the library is not present. The skipping logic should fire before self.build(), but it we still end up attempting to build, then that's something we should know about.

How does that sound?

[tfiala]

Yeah, if you can make that happen, that sounds good to me. This is a "if you have libc++, please build with it, otherwise skip this test"-kinda test... :-)

[tfiala]

Forgot to mark this as closed:

Closed by commit:

------------------------------------------------------------------------
r250965 | tfiala | 2015-10-21 17:23:38 -0700 (Wed, 21 Oct 2015) | 4 lines

Fix libstdc++ data formatters on Ubuntu 15.10 x86_64

Also, I forgot to mention that I only added the string formatter support for remote host (i.e. lldb-server/debugserver) connections. For those systems using in-host attaching (I think *BSD), the string formatter will not display with host memory content. It would be a smallish fix, but since I don't have a system to test that on, I didn't want to implement that. The string formatter should fall back to printing the raw content in that case.

examples/synthetic/gnu_libstdcpp.py

 import re
 import lldb.formatters.Logger
 
 # C++ STL formatters for LLDB
 # These formatters are based upon the version of the GNU libstdc++
 # as it ships with Mac OS X 10.6.8 thru 10.8.0
 # You are encouraged to look at the STL implementation for your platform
 # before relying on these formatters to do the right thing for your setup
 
 class StdListSynthProvider:
 
   def __init__(self, valobj, dict):
     logger = lldb.formatters.Logger.Logger()
     self.valobj = valobj
     self.count = None
-    logger >> "Providing synthetic children for a map named " + str(valobj.GetName())
+    logger >> "Providing synthetic children for a list named " + str(valobj.GetName())
 
   def next_node(self,node):
     logger = lldb.formatters.Logger.Logger()
     return node.GetChildMemberWithName('_M_next')
 
   def is_valid(self,node):
     logger = lldb.formatters.Logger.Logger()
-    return self.value(self.next_node(node)) != self.node_address
+    valid = self.value(self.next_node(node)) != self.node_address
+                if valid:
+                        logger >> "%s is valid" % str(self.valobj.GetName())
+                else:
+                        logger >> "synthetic value is not valid"
+                return valid
 
   def value(self,node):
     logger = lldb.formatters.Logger.Logger()
-    return node.GetValueAsUnsigned()
+                value = node.GetValueAsUnsigned()
+                logger >> "synthetic value for {}: {}".format(
+                        str(self.valobj.GetName()),
+                        value)
+                return value
 
   # Floyd's cycle-finding algorithm
   # try to detect if this list has a loop
   def has_loop(self):
     global _list_uses_loop_detector
     logger = lldb.formatters.Logger.Logger()
     if _list_uses_loop_detector == False:
       logger >> "Asked not to use loop detection"
       return False
     slow = self.next
     fast1 = self.next
     fast2 = self.next
     while self.is_valid(slow):
       slow_value = self.value(slow)
       fast1 = self.next_node(fast2)
       fast2 = self.next_node(fast1)
       if self.value(fast1) == slow_value or self.value(fast2) == slow_value:
         return True
       slow = self.next_node(slow)
     return False
 
   def num_children(self):
     logger = lldb.formatters.Logger.Logger()
-    if self.count == None:
+    if self.count is None:
+                        # libstdc++ 6.0.21 added dedicated count field.
+                        count_child = self.node.GetChildMemberWithName('_M_data')
+                        if count_child and count_child.IsValid():
+                                self.count = count_child.GetValueAsUnsigned(0)
+                if self.count is None:
       self.count = self.num_children_impl()
     return self.count
 
   def num_children_impl(self):
     logger = lldb.formatters.Logger.Logger()
     try:
       next_val = self.next.GetValueAsUnsigned(0)
       prev_val = self.prev.GetValueAsUnsigned(0)
def extract_type(self):
     return data_type
 
   def update(self):
     logger = lldb.formatters.Logger.Logger()
     # preemptively setting this to None - we might end up changing our mind later
     self.count = None
     try:
       impl = self.valobj.GetChildMemberWithName('_M_impl')
-      node = impl.GetChildMemberWithName('_M_node')
+      self.node = impl.GetChildMemberWithName('_M_node')
       self.node_address = self.valobj.AddressOf().GetValueAsUnsigned(0)
-      self.next = node.GetChildMemberWithName('_M_next')
-      self.prev = node.GetChildMemberWithName('_M_prev')
+      self.next = self.node.GetChildMemberWithName('_M_next')
+      self.prev = self.node.GetChildMemberWithName('_M_prev')
       self.data_type = self.extract_type()
       self.data_size = self.data_type.GetByteSize()
     except:
       pass
 
   def has_children(self):
     return True
 

source/Plugins/Language/CPlusPlus/CPlusPlusLanguage.cpp

LoadLibStdcppFormatters(lldb::TypeCategoryImplSP cpp_category_sp)
     .SetSkipReferences(false)
     .SetDontShowChildren(true)
     .SetDontShowValue(true)
     .SetShowMembersOneLiner(false)
     .SetHideItemNames(false);
     
     lldb::TypeSummaryImplSP std_string_summary_sp(new StringSummaryFormat(stl_summary_flags,
                                                                           "${var._M_dataplus._M_p}"));
-    
+    lldb::TypeSummaryImplSP cxx11_string_summary_sp(new CXXFunctionSummaryFormat(stl_summary_flags,
+                                                                                 LibStdcppStringSummaryProvider,
+                                                                                 "libstdc++ c++11 std::string summary provider"));
+    lldb::TypeSummaryImplSP cxx11_wstring_summary_sp(new CXXFunctionSummaryFormat(stl_summary_flags,
+                                                                                 LibStdcppWStringSummaryProvider,
+                                                                                 "libstdc++ c++11 std::wstring summary provider"));
+
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::string"),
                                                       std_string_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<char>"),
                                                       std_string_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<char,std::char_traits<char>,std::allocator<char> >"),
                                                       std_string_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<char, std::char_traits<char>, std::allocator<char> >"),
                                                       std_string_summary_sp);
-    
+    cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::__cxx11::string"),
+                                                      cxx11_string_summary_sp);
+    cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >"),
+                                                      cxx11_string_summary_sp);
+
     // making sure we force-pick the summary for printing wstring (_M_p is a wchar_t*)
     lldb::TypeSummaryImplSP std_wstring_summary_sp(new StringSummaryFormat(stl_summary_flags,
                                                                            "${var._M_dataplus._M_p%S}"));
     
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::wstring"),
                                                       std_wstring_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<wchar_t>"),
                                                       std_wstring_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >"),
                                                       std_wstring_summary_sp);
     cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >"),
                                                       std_wstring_summary_sp);
-    
-    
+    cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::__cxx11::wstring"),
+                                                      cxx11_wstring_summary_sp);
+    cpp_category_sp->GetTypeSummariesContainer()->Add(ConstString("std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >"),
+                                                      cxx11_wstring_summary_sp);
 #ifndef LLDB_DISABLE_PYTHON
     
     SyntheticChildren::Flags stl_synth_flags;
     stl_synth_flags.SetCascades(true).SetSkipPointers(false).SetSkipReferences(false);
     
     cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(RegularExpressionSP(new RegularExpression("^std::vector<.+>(( )?&)?$")),
                                                             SyntheticChildrenSP(new ScriptedSyntheticChildren(stl_synth_flags,
                                                                                                               "lldb.formatters.cpp.gnu_libstdcpp.StdVectorSynthProvider")));
     cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(RegularExpressionSP(new RegularExpression("^std::map<.+> >(( )?&)?$")),
                                                             SyntheticChildrenSP(new ScriptedSyntheticChildren(stl_synth_flags,
                                                                                                               "lldb.formatters.cpp.gnu_libstdcpp.StdMapSynthProvider")));
-    cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(RegularExpressionSP(new RegularExpression("^std::list<.+>(( )?&)?$")),
+    cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(RegularExpressionSP(new RegularExpression("^std::(__cxx11::)?list<.+>(( )?&)?$")),
+                                                            SyntheticChildrenSP(new ScriptedSyntheticChildren(stl_synth_flags,
+                                                                                                              "lldb.formatters.cpp.gnu_libstdcpp.StdListSynthProvider")));
+#if 0
+    // With only this, I get std::list showing the content, all children on the same line.
+    // With this and the section below, I see one child element per line.
+    cpp_category_sp->GetRegexTypeSyntheticsContainer()->Add(RegularExpressionSP(new RegularExpression("^std::__cxx11::_List_base<.+>(( )?&)?$")),
                                                             SyntheticChildrenSP(new ScriptedSyntheticChildren(stl_synth_flags,
                                                                                                               "lldb.formatters.cpp.gnu_libstdcpp.StdListSynthProvider")));
+#endif
     
     stl_summary_flags.SetDontShowChildren(false);stl_summary_flags.SetSkipPointers(true);
     cpp_category_sp->GetRegexTypeSummariesContainer()->Add(RegularExpressionSP(new RegularExpression("^std::vector<.+>(( )?&)?$")),
                                                            TypeSummaryImplSP(new StringSummaryFormat(stl_summary_flags,
                                                                                                      "size=${svar%#}")));
     cpp_category_sp->GetRegexTypeSummariesContainer()->Add(RegularExpressionSP(new RegularExpression("^std::map<.+> >(( )?&)?$")),
                                                            TypeSummaryImplSP(new StringSummaryFormat(stl_summary_flags,
                                                                                                      "size=${svar%#}")));
-    cpp_category_sp->GetRegexTypeSummariesContainer()->Add(RegularExpressionSP(new RegularExpression("^std::list<.+>(( )?&)?$")),
+    cpp_category_sp->GetRegexTypeSummariesContainer()->Add(RegularExpressionSP(new RegularExpression("^std::(__cxx11::)?list<.+>(( )?&)?$")),
+                                                           TypeSummaryImplSP(new StringSummaryFormat(stl_summary_flags,
+                                                                                                     "size=${svar%#}")));
+#if 0
+    // With this, I get std::list showing one child per line.  Requires the change above to get anything, though.
+    cpp_category_sp->GetRegexTypeSummariesContainer()->Add(RegularExpressionSP(new RegularExpression("^std::__cxx11::_List_base<.+>(( )?&)?$")),
                                                            TypeSummaryImplSP(new StringSummaryFormat(stl_summary_flags,
                                                                                                      "size=${svar%#}")));
-    
+#endif
     AddCXXSynthetic(cpp_category_sp, lldb_private::formatters::LibStdcppVectorIteratorSyntheticFrontEndCreator, "std::vector iterator synthetic children", ConstString("^__gnu_cxx::__normal_iterator<.+>$"), stl_synth_flags, true);
     
     AddCXXSynthetic(cpp_category_sp, lldb_private::formatters::LibstdcppMapIteratorSyntheticFrontEndCreator, "std::map iterator synthetic children", ConstString("^std::_Rb_tree_iterator<.+>$"), stl_synth_flags, true);
 #endif
 }
 
 static void
 LoadSystemFormatters(lldb::TypeCategoryImplSP cpp_category_sp)

source/Plugins/Language/CPlusPlus/LibStdcpp.h

 #include "lldb/Core/Stream.h"
 #include "lldb/Core/ValueObject.h"
 #include "lldb/DataFormatters/TypeSummary.h"
 #include "lldb/DataFormatters/TypeSynthetic.h"
 
 namespace lldb_private {
     namespace formatters
     {
+        bool
+        LibStdcppStringSummaryProvider (ValueObject& valobj, Stream& stream, const TypeSummaryOptions& options); // libcstdc++ c++11 std::string
+
+        bool
+        LibStdcppWStringSummaryProvider (ValueObject& valobj, Stream& stream, const TypeSummaryOptions& options); // libcstdc++ c++11 std::wstring
+
         SyntheticChildrenFrontEnd* LibstdcppMapIteratorSyntheticFrontEndCreator (CXXSyntheticChildren*, lldb::ValueObjectSP);
         
         SyntheticChildrenFrontEnd* LibStdcppVectorIteratorSyntheticFrontEndCreator (CXXSyntheticChildren*, lldb::ValueObjectSP);
     } // namespace formatters
 } // namespace lldb_private
 
 #endif // liblldb_LibStdCpp_h_

source/Plugins/Language/CPlusPlus/LibStdcpp.cpp

 //===-- LibStdcpp.cpp ---------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "LibStdcpp.h"
 
 #include "lldb/Core/DataBufferHeap.h"
 #include "lldb/Core/Error.h"
 #include "lldb/Core/Stream.h"
 #include "lldb/Core/ValueObject.h"
 #include "lldb/Core/ValueObjectConstResult.h"
+#include "lldb/DataFormatters/StringPrinter.h"
 #include "lldb/DataFormatters/VectorIterator.h"
 #include "lldb/Host/Endian.h"
 #include "lldb/Symbol/ClangASTContext.h"
 #include "lldb/Target/Target.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::formatters;
lldb_private::formatters::VectorIteratorSyntheticFrontEnd::GetIndexOfChildWithName (const ConstString &name)
     if (name == ConstString("item"))
         return 0;
     return UINT32_MAX;
 }
 
 lldb_private::formatters::VectorIteratorSyntheticFrontEnd::~VectorIteratorSyntheticFrontEnd ()
 {
 }
+
+bool
+lldb_private::formatters::LibStdcppStringSummaryProvider (ValueObject& valobj, Stream& stream, const TypeSummaryOptions& options)
+{
+    const bool scalar_is_load_addr = true;
+    AddressType addr_type;
+    lldb::addr_t addr_of_string = valobj.GetAddressOf(scalar_is_load_addr, &addr_type);
+    if (addr_of_string != LLDB_INVALID_ADDRESS)
+    {
+        switch (addr_type)
+        {
+            case eAddressTypeLoad:
+            {
+                ProcessSP process_sp(valobj.GetProcessSP());
+                if (!process_sp)
+                    return false;
+
+                StringPrinter::ReadStringAndDumpToStreamOptions options(valobj);
+                Error error;
+                lldb::addr_t addr_of_data = process_sp->ReadPointerFromMemory(addr_of_string, error);
+                if (error.Fail() || addr_of_data == 0 || addr_of_data == LLDB_INVALID_ADDRESS)
+                    return false;
+                options.SetLocation(addr_of_data);
+                options.SetProcessSP(process_sp);
+                options.SetStream(&stream);
+                options.SetNeedsZeroTermination(false);
+                options.SetBinaryZeroIsTerminator(true);
+                lldb::addr_t size_of_data = process_sp->ReadPointerFromMemory(addr_of_string + process_sp->GetAddressByteSize(), error);
+                if (error.Fail())
+                    return false;
+                options.SetSourceSize(size_of_data);
+
+                if (!StringPrinter::ReadStringAndDumpToStream<StringPrinter::StringElementType::UTF8>(options))
+                {
+                    stream.Printf("Summary Unavailable");
+                    return true;
+                }
+                else
+                    return true;
+            }
+                break;
+            case eAddressTypeHost:
+                break;
+            case eAddressTypeInvalid:
+            case eAddressTypeFile:
+                break;
+        }
+    }
+    return false;
+}
+
+bool
+lldb_private::formatters::LibStdcppWStringSummaryProvider (ValueObject& valobj, Stream& stream, const TypeSummaryOptions& options)
+{
+    const bool scalar_is_load_addr = true;
+    AddressType addr_type;
+    lldb::addr_t addr_of_string = valobj.GetAddressOf(scalar_is_load_addr, &addr_type);
+    if (addr_of_string != LLDB_INVALID_ADDRESS)
+    {
+        switch (addr_type)
+        {
+            case eAddressTypeLoad:
+            {
+                ProcessSP process_sp(valobj.GetProcessSP());
+                if (!process_sp)
+                    return false;
+
+                CompilerType wchar_compiler_type = valobj.GetCompilerType().GetBasicTypeFromAST(lldb::eBasicTypeWChar);
+
+                if (!wchar_compiler_type)
+                    return false;
+
+                const uint32_t wchar_size = wchar_compiler_type.GetBitSize(nullptr); // Safe to pass NULL for exe_scope here
+
+                StringPrinter::ReadStringAndDumpToStreamOptions options(valobj);
+                Error error;
+                lldb::addr_t addr_of_data = process_sp->ReadPointerFromMemory(addr_of_string, error);
+                if (error.Fail() || addr_of_data == 0 || addr_of_data == LLDB_INVALID_ADDRESS)
+                    return false;
+                options.SetLocation(addr_of_data);
+                options.SetProcessSP(process_sp);
+                options.SetStream(&stream);
+                options.SetNeedsZeroTermination(false);
+                options.SetBinaryZeroIsTerminator(false);
+                lldb::addr_t size_of_data = process_sp->ReadPointerFromMemory(addr_of_string + process_sp->GetAddressByteSize(), error);
+                if (error.Fail())
+                    return false;
+                options.SetSourceSize(size_of_data);
+                options.SetPrefixToken("L");
+
+                switch (wchar_size)
+                {
+                    case 8:
+                        return StringPrinter::ReadStringAndDumpToStream<StringPrinter::StringElementType::UTF8>(options);
+                    case 16:
+                        return StringPrinter::ReadStringAndDumpToStream<StringPrinter::StringElementType::UTF16>(options);
+                    case 32:
+                        return StringPrinter::ReadStringAndDumpToStream<StringPrinter::StringElementType::UTF32>(options);
+                    default:
+                        stream.Printf("size for wchar_t is not valid");
+                        return true;
+                }
+                return true;
+            }
+                break;
+            case eAddressTypeHost:
+                break;
+            case eAddressTypeInvalid:
+            case eAddressTypeFile:
+                break;
+        }
+    }
+    return false;
+}

test/functionalities/data-formatter/data-formatter-stl/libcxx/list/loop/main.cpp

 #include <list>
 
 #include <assert.h>
 
 typedef std::list<int> int_list;
 
 int main()
 {
+#ifdef LLDB_USING_LIBCPP
     int_list *numbers_list = new int_list{1,2,3,4,5,6,7,8,9,10};
 
     auto *third_elem = numbers_list->__end_.__next_->__next_->__next_; // Set break point at this line.
     assert(third_elem->__value_ == 3);
     auto *fifth_elem = third_elem->__next_->__next_;
     assert(fifth_elem->__value_ == 5);
     fifth_elem->__next_ = third_elem;
+#endif
 
     // Any attempt to free the list will probably crash the program. Let's just leak it.
     return 0; // Set second break point at this line.
 }

test/functionalities/data-formatter/data-formatter-stl/libstdcpp/string/TestDataFormatterStdString.py

def test_with_run_command(self):
             self.runCmd('type summary clear', check=False)
             self.runCmd('type filter clear', check=False)
             self.runCmd('type synth clear', check=False)
             self.runCmd("settings set target.max-children-count 256", check=False)
 
         # Execute the cleanup function during test case tear down.
         self.addTearDownHook(cleanup)
 
-        self.expect("frame variable",
-                    substrs = ['(std::wstring) s = L"hello world! מזל טוב!"',
-                    '(std::wstring) S = L"!!!!"',
-                    '(const wchar_t *) mazeltov = 0x','L"מזל טוב"',
-                    '(std::string) q = "hello world"',
-                    '(std::string) Q = "quite a long std::strin with lots of info inside it"'])
-
-        self.runCmd("n")
-
-        self.expect("frame variable",
-                    substrs = ['(std::wstring) s = L"hello world! מזל טוב!"',
-                    '(std::wstring) S = L"!!!!!"',
-                    '(const wchar_t *) mazeltov = 0x','L"מזל טוב"',
-                    '(std::string) q = "hello world"',
-                    '(std::string) Q = "quite a long std::strin with lots of info inside it"'])
+        var_s = self.frame().FindVariable('s')
+        var_S = self.frame().FindVariable('S')
+        var_mazeltov = self.frame().FindVariable('mazeltov')
+        var_q = self.frame().FindVariable('q')
+        var_Q = self.frame().FindVariable('Q')
+
+        self.assertTrue(var_s.GetSummary() == 'L"hello world! מזל טוב!"', "s summary wrong")
+        self.assertTrue(var_S.GetSummary() == 'L"!!!!"', "S summary wrong")
+        self.assertTrue(var_mazeltov.GetSummary() == 'L"מזל טוב"', "mazeltov summary wrong")
+        self.assertTrue(var_q.GetSummary() == '"hello world"', "q summary wrong")
+        self.assertTrue(var_Q.GetSummary() == '"quite a long std::strin with lots of info inside it"', "Q summary wrong")
+
+        self.runCmd("next")
+
+        self.assertTrue(var_S.GetSummary() == 'L"!!!!!"', "new S summary wrong")
 
 if __name__ == '__main__':
     import atexit
     lldb.SBDebugger.Initialize()
     atexit.register(lambda: lldb.SBDebugger.Terminate())
     unittest2.main()

test/make/Makefile.rules

 endif
 
 ifeq (1,$(USE_LIBCPP))
   # Clang requires an extra flag: -stdlib=libstdc++
   ifneq (,$(findstring clang,$(CC)))
     ifeq "$(OS)" "Linux"
       # This is the default install location on Ubuntu 14.04
       ifneq ($(wildcard /usr/include/c++/v1/.),)
-        CXXFLAGS += -stdlib=libc++
+        CXXFLAGS += -stdlib=libc++ -DLLDB_USING_LIBCPP
         LDFLAGS += -stdlib=libc++
         CXXFLAGS += -I/usr/include/c++/v1
       endif
     else
-      CXXFLAGS += -stdlib=libc++
+      CXXFLAGS += -stdlib=libc++ -DLLDB_USING_LIBCPP
       LDFLAGS += -stdlib=libc++
     endif
   endif
 endif
 
 #----------------------------------------------------------------------
 # dylib settings
 #----------------------------------------------------------------------