[testsuite] Split Objective-C data formatter

[JDevlieghere]

Summary

The testcase for objective-c data formatters is very big as it checks a bunch of stuff. This is annoying when using the lit test driver, because it prevents us from running the different cases in parallel. As a result, it's always one of the last few tests that complete. This patch splits the test into multiple files that share a common base class. This way lit can run the different tests in parallel.

[JDevlieghere]

Fix CF test

[davide]

I think this is good regardless for readability, but I would really appreciate if we can collect some numbers to see how effective this change actually is.

[labath]

Looks fine to me, with some inline ideas you can implement if you think they're worthwhile.

Also, since you're looking at reducing test time, the question you can ask yourself is: is it really worth it running separate dwarf+dsym flavours of these tests? I'm of the view that this should not be necessary, as the data formatters operate on high level abstractions, which should hide any differences in the debug info format (and the fact that the debug info parsers generate correct and consistent abstractions should be tested elsewhere). However, I don't have a horse in this race, so the decision is entirely up to you...

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCCF.py

22–24

I'm pretty sure these are unneeded, as all you do is call the overridden method.

30–33

I think this pattern is a relict from the days when we didn't have the magic test multiplicator. In those days, tests were written as:

def test_foo_dsym(self):
  self.buildDsym()
  self.foo()

def test_foo_dwarf(self):
  self.buildDwarf()
  self.foo()

def foo(self):
  # do the real stuff here

However, now that we can generate test flavours automatically, I don't think it makes much sense. So, unless you find it useful for some reason, I think we can just remove the helper function (by deleting these four lines).

46–54

It doesn't look like this test is doing anything to the type summaries, so I don't think it's necessary to clean up anything here.

[aprantl]

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/ObjCDataFormatterTestCase.py

22

All uses of this variable should also be replaced by lldbutil.run_to_source_breakpoint()

34

i.e.: this

[aprantl]

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjNSException.py

31

It would be cleaner to replace
self.expect('frame var'

with lldbutil.check_variable()

but that's less important.

[JDevlieghere]

Code review feedback

[JDevlieghere]

In D60300#1455790, @davide wrote:

I think this is good regardless for readability, but I would really appreciate if we can collect some numbers to see how effective this change actually is.

I see 120 -> 12 seconds on my iMac Pro.

[davide]

great!

[clayborg]

In D60300#1456468, @JDevlieghere wrote:

In D60300#1455790, @davide wrote:

I think this is good regardless for readability, but I would really appreciate if we can collect some numbers to see how effective this change actually is.

I see 120 -> 12 seconds on my iMac Pro.

Nice! Might be nice to have some automated info that gets dumped when one file with multiple tests takes a long time and suggest breaking them up in the python test infrastructure modules. Might be other nice gains to be had?

[aprantl]

What's the motivation behind all the @no_debug_info_test decorators? Are those codepaths guaranteed to be tested elsewhere?

[aprantl]

What's the motivation behind all the @no_debug_info_test decorators? Are those codepaths guaranteed to be tested elsewhere?

As a rule of thumb I'd say, let's only stick @no_debug_info_test on tests where a (future) coverage bot proves that it is safe. How does that sound?

[JDevlieghere]

In D60300#1456485, @aprantl wrote:

What's the motivation behind all the @no_debug_info_test decorators? Are those codepaths guaranteed to be tested elsewhere?

As a rule of thumb I'd say, let's only stick @no_debug_info_test on tests where a (future) coverage bot proves that it is safe. How does that sound?

Sounds good, I'll remove the decorators and we can do this incrementally.

[davide]

In D60300#1456472, @clayborg wrote:

In D60300#1456468, @JDevlieghere wrote:

In D60300#1455790, @davide wrote:

I think this is good regardless for readability, but I would really appreciate if we can collect some numbers to see how effective this change actually is.

I see 120 -> 12 seconds on my iMac Pro.

Nice! Might be nice to have some automated info that gets dumped when one file with multiple tests takes a long time and suggest breaking them up in the python test infrastructure modules. Might be other nice gains to be had?

lit has this feature. It's called --time-tests. That's how we discovered the problem. There are other tests that we can fix.

--Davide

[davide]

In D60300#1456491, @davide wrote:

In D60300#1456472, @clayborg wrote:

In D60300#1456468, @JDevlieghere wrote:

In D60300#1455790, @davide wrote:

I think this is good regardless for readability, but I would really appreciate if we can collect some numbers to see how effective this change actually is.

I see 120 -> 12 seconds on my iMac Pro.

Nice! Might be nice to have some automated info that gets dumped when one file with multiple tests takes a long time and suggest breaking them up in the python test infrastructure modules. Might be other nice gains to be had?

lit has this feature. It's called --time-tests. That's how we discovered the problem. There are other tests that we can fix.

--Davide

And, as you ask, these are the slowest tests for now.

68.25s: lldb-Suite :: types/TestIntegerTypes.py
86.52s: lldb-Suite :: types/TestIntegerTypesExpr.py
95.66s: lldb-Suite :: lang/objc/foundation/TestObjCMethods2.py
131.24s: lldb-Suite :: lang/objc/objc-new-syntax/TestObjCNewSyntax.py

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/ObjCDataFormatterTestCase.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class ObjCDataFormatterTestCase(TestBase):
+
+   mydir = TestBase.compute_mydir(__file__)
+
+   def appkit_tester_impl(self, commands):
+      self.build()
+      self.appkit_common_data_formatters_command()
+      commands()
+
+   def appkit_common_data_formatters_command(self):
+      """Test formatters for AppKit classes."""
+      self.target, process, thread, bkpt = lldbutil.run_to_source_breakpoint(
+          self, '// Set break point at this line.',
+          lldb.SBFileSpec('main.m', False))
+
+      # The stop reason of the thread should be breakpoint.
+      self.expect(
+          "thread list",
+          STOPPED_DUE_TO_BREAKPOINT,
+          substrs=['stopped', 'stop reason = breakpoint'])
+
+      # This is the function to remove the custom formats in order to have a
+      # clean slate for the next test case.
+      def cleanup():
+         self.runCmd('type format clear', check=False)
+         self.runCmd('type summary clear', check=False)
+         self.runCmd('type synth clear', check=False)
+
+      # Execute the cleanup function during test case tear down.
+      self.addTearDownHook(cleanup)

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCCF.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterCF(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_coreframeworks_and_run_command(self):
+        """Test formatters for Core OSX frameworks."""
+        self.build()
+        self.target, process, thread, bkpt = lldbutil.run_to_source_breakpoint(
+            self, '// Set break point at this line.',
+            lldb.SBFileSpec('main.m', False))
+
+        # The stop reason of the thread should be breakpoint.
+        self.expect(
+            "thread list",
+            STOPPED_DUE_TO_BREAKPOINT,
+            substrs=['stopped', 'stop reason = breakpoint'])
+
+        # check formatters for common Objective-C types
+        expect_strings = [
+            '(CFGregorianUnits) cf_greg_units = 1 years, 3 months, 5 days, 12 hours, 5 minutes 7 seconds',
+            '(CFRange) cf_range = location=4 length=4',
+            '(NSPoint) ns_point = (x = 4, y = 4)',
+            '(NSRange) ns_range = location=4, length=4',
+            '(NSRect) ns_rect = (origin = (x = 1, y = 1), size = (width = 5, height = 5))',
+            '(NSRectArray) ns_rect_arr = ((x = 1, y = 1), (width = 5, height = 5)), ...',
+            '(NSSize) ns_size = (width = 5, height = 7)',
+            '(CGSize) cg_size = (width = 1, height = 6)',
+            '(CGPoint) cg_point = (x = 2, y = 7)',
+            '(CGRect) cg_rect = (origin = (x = 1, y = 2), size = (width = 7, height = 7))',
+            '(Rect) rect = (t=4, l=8, b=4, r=7)',
+            '(Rect *) rect_ptr = (t=4, l=8, b=4, r=7)',
+            '(Point) point = (v=7, h=12)', '(Point *) point_ptr = (v=7, h=12)',
+            '1985', 'foo_selector_impl'
+        ]
+
+        if self.getArchitecture() in ['i386', 'x86_64']:
+            expect_strings.append('(HIPoint) hi_point = (x=7, y=12)')
+            expect_strings.append(
+                '(HIRect) hi_rect = origin=(x = 3, y = 5) size=(width = 4, height = 6)'
+            )
+            expect_strings.append(
+                '(RGBColor) rgb_color = red=3 green=56 blue=35')
+            expect_strings.append(
+                '(RGBColor *) rgb_color_ptr = red=3 green=56 blue=35')
+
+        self.expect("frame variable", substrs=expect_strings)

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCExpr.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterExpr(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_expr_with_run_command(self):
+        """Test common cases of expression parser <--> formatters interaction."""
+        self.build()
+        self.target, process, thread, bkpt = lldbutil.run_to_source_breakpoint(
+            self, '// Set break point at this line.',
+            lldb.SBFileSpec('main.m', False))
+
+        # The stop reason of the thread should be breakpoint.
+        self.expect(
+            "thread list",
+            STOPPED_DUE_TO_BREAKPOINT,
+            substrs=['stopped', 'stop reason = breakpoint'])
+
+        # This is the function to remove the custom formats in order to have a
+        # clean slate for the next test case.
+        def cleanup():
+            self.runCmd('type format clear', check=False)
+            self.runCmd('type summary clear', check=False)
+            self.runCmd('type synth clear', check=False)
+
+        # Execute the cleanup function during test case tear down.
+        self.addTearDownHook(cleanup)
+
+        # check that the formatters are able to deal safely and correctly
+        # with ValueObjects that the expression parser returns
+        self.expect(
+            'expression ((id)@"Hello for long enough to avoid short string types")',
+            matching=False,
+            substrs=['Hello for long enough to avoid short string types'])
+
+        self.expect(
+            'expression -d run -- ((id)@"Hello for long enough to avoid short string types")',
+            substrs=['Hello for long enough to avoid short string types'])
+
+        self.expect('expr -d run -- label1', substrs=['Process Name'])
+
+        self.expect(
+            'expr -d run -- @"Hello for long enough to avoid short string types"',
+            substrs=['Hello for long enough to avoid short string types'])
+
+        self.expect(
+            'expr -d run --object-description -- @"Hello for long enough to avoid short string types"',
+            substrs=['Hello for long enough to avoid short string types'])
+        self.expect(
+            'expr -d run --object-description -- @"Hello"',
+            matching=False,
+            substrs=['@"Hello" Hello'])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCKVO.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterKVO(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_kvo_with_run_command(self):
+        """Test the behavior of formatters when KVO is in use."""
+        self.build()
+        self.target, process, thread, bkpt = lldbutil.run_to_source_breakpoint(
+            self, '// Set break point at this line.',
+            lldb.SBFileSpec('main.m', False))
+
+        # The stop reason of the thread should be breakpoint.
+        self.expect(
+            "thread list",
+            STOPPED_DUE_TO_BREAKPOINT,
+            substrs=['stopped', 'stop reason = breakpoint'])
+
+        # This is the function to remove the custom formats in order to have a
+        # clean slate for the next test case.
+        def cleanup():
+            self.runCmd('type format clear', check=False)
+            self.runCmd('type summary clear', check=False)
+            self.runCmd('type synth clear', check=False)
+
+        # Execute the cleanup function during test case tear down.
+        self.addTearDownHook(cleanup)
+
+        # as long as KVO is implemented by subclassing, this test should succeed
+        # we should be able to dynamically figure out that the KVO implementor class
+        # is a subclass of Molecule, and use the appropriate summary for it
+        self.runCmd("type summary add -s JustAMoleculeHere Molecule")
+        self.expect('frame variable molecule', substrs=['JustAMoleculeHere'])
+        self.runCmd("next")
+        self.expect("thread list", substrs=['stopped', 'step over'])
+        self.expect('frame variable molecule', substrs=['JustAMoleculeHere'])
+
+        self.runCmd("next")
+        # check that NSMutableDictionary's formatter is not confused when
+        # dealing with a KVO'd dictionary
+        self.expect(
+            'frame variable newMutableDictionary',
+            substrs=[
+                '(NSDictionary *) newMutableDictionary = ',
+                ' 21 key/value pairs'
+            ])
+
+        lldbutil.run_break_set_by_regexp(self, 'setAtoms')
+
+        self.runCmd("continue")
+        self.expect("frame variable _cmd", substrs=['setAtoms:'])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSBundle.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSBundle(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nsbundle_with_run_command(self):
+        """Test formatters for NSBundle."""
+        self.appkit_tester_impl(self.nsbundle_data_formatter_commands)
+
+    def nsbundle_data_formatter_commands(self):
+        self.expect(
+            'frame variable bundle_string bundle_url main_bundle',
+            substrs=[
+                '(NSBundle *) bundle_string = ',
+                ' @"/System/Library/Frameworks/Accelerate.framework"',
+                '(NSBundle *) bundle_url = ',
+                ' @"/System/Library/Frameworks/Foundation.framework"',
+                '(NSBundle *) main_bundle = ', 'data-formatter-objc'
+            ])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSContainer.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSContainer(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nscontainers_with_run_command(self):
+        """Test formatters for  NS container classes."""
+        self.appkit_tester_impl(self.nscontainers_data_formatter_commands)
+
+    def nscontainers_data_formatter_commands(self):
+        self.expect(
+            'frame variable newArray nsDictionary newDictionary nscfDictionary cfDictionaryRef newMutableDictionary cfarray_ref mutable_array_ref',
+            substrs=[
+                '(NSArray *) newArray = ', '@"50 elements"',
+                '(NSDictionary *) newDictionary = ', ' 12 key/value pairs',
+                '(NSDictionary *) newMutableDictionary = ',
+                ' 21 key/value pairs', '(NSDictionary *) nsDictionary = ',
+                ' 2 key/value pairs', '(CFDictionaryRef) cfDictionaryRef = ',
+                ' 3 key/value pairs', '(CFArrayRef) cfarray_ref = ',
+                '@"3 elements"', '(CFMutableArrayRef) mutable_array_ref = ',
+                '@"11 elements"'
+            ])
+
+        self.expect(
+            'frame variable iset1 iset2 imset',
+            substrs=['4 indexes', '512 indexes', '10 indexes'])
+
+        self.expect(
+            'frame variable binheap_ref',
+            substrs=['(CFBinaryHeapRef) binheap_ref = ', '@"21 items"'])
+
+        self.expect(
+            'expression -d run -- (NSArray*)[NSArray new]',
+            substrs=['@"0 elements"'])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSData.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSData(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nsdata_with_run_command(self):
+        """Test formatters for  NSData."""
+        self.appkit_tester_impl(self.nsdata_data_formatter_commands)
+
+    def nsdata_data_formatter_commands(self):
+        self.expect(
+            'frame variable immutableData mutableData data_ref mutable_data_ref mutable_string_ref concreteData concreteMutableData',
+            substrs=[
+                '(NSData *) immutableData = ', ' 4 bytes',
+                '(NSData *) mutableData = ', ' 14 bytes',
+                '(CFDataRef) data_ref = ', '@"5 bytes"',
+                '(CFMutableDataRef) mutable_data_ref = ', '@"5 bytes"',
+                '(CFMutableStringRef) mutable_string_ref = ',
+                ' @"Wish ya knew"', '(NSData *) concreteData = ',
+                ' 100000 bytes', '(NSMutableData *) concreteMutableData = ',
+                ' 100000 bytes'
+            ])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSDate.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSDate(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nsdata_with_run_command(self):
+        """Test formatters for  NSData."""
+        self.appkit_tester_impl(self.nsdata_data_formatter_commands)
+
+    def nsdata_data_formatter_commands(self):
+        self.expect(
+            'frame variable immutableData mutableData data_ref mutable_data_ref mutable_string_ref concreteData concreteMutableData',
+            substrs=[
+                '(NSData *) immutableData = ', ' 4 bytes',
+                '(NSData *) mutableData = ', ' 14 bytes',
+                '(CFDataRef) data_ref = ', '@"5 bytes"',
+                '(CFMutableDataRef) mutable_data_ref = ', '@"5 bytes"',
+                '(CFMutableStringRef) mutable_string_ref = ',
+                ' @"Wish ya knew"', '(NSData *) concreteData = ',
+                ' 100000 bytes', '(NSMutableData *) concreteMutableData = ',
+                ' 100000 bytes'
+            ])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSError.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSError(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nserror_with_run_command(self):
+        """Test formatters for NSError."""
+        self.appkit_tester_impl(self.nserror_data_formatter_commands)
+
+    def nserror_data_formatter_commands(self):
+        self.expect(
+            'frame variable nserror', substrs=['domain: @"Foobar" - code: 12'])
+
+        self.expect(
+            'frame variable nserrorptr',
+            substrs=['domain: @"Foobar" - code: 12'])
+
+        self.expect(
+            'frame variable nserror->_userInfo', substrs=['2 key/value pairs'])
+
+        self.expect(
+            'frame variable nserror->_userInfo --ptr-depth 1 -d run-target',
+            substrs=['@"a"', '@"b"', "1", "2"])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCNSURL.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSURL(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nsurl_with_run_command(self):
+        """Test formatters for NSURL."""
+        self.appkit_tester_impl(self.nsurl_data_formatter_commands)
+
+    def nsurl_data_formatter_commands(self):
+        self.expect(
+            'frame variable cfurl_ref cfchildurl_ref cfgchildurl_ref',
+            substrs=[
+                '(CFURLRef) cfurl_ref = ', '@"http://www.foo.bar',
+                'cfchildurl_ref = ', '@"page.html -- http://www.foo.bar',
+                '(CFURLRef) cfgchildurl_ref = ',
+                '@"?whatever -- http://www.foo.bar/page.html"'
+            ])
+
+        self.expect(
+            'frame variable nsurl nsurl2 nsurl3',
+            substrs=[
+                '(NSURL *) nsurl = ', '@"http://www.foo.bar',
+                '(NSURL *) nsurl2 =', '@"page.html -- http://www.foo.bar',
+                '(NSURL *) nsurl3 = ',
+                '@"?whatever -- http://www.foo.bar/page.html"'
+            ])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjCPlain.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSPlain(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_plain_objc_with_run_command(self):
+        """Test basic ObjC formatting behavior."""
+        self.build()
+        self.target, process, thread, bkpt = lldbutil.run_to_source_breakpoint(
+            self, '// Set break point at this line.',
+            lldb.SBFileSpec('main.m', False))
+
+        # The stop reason of the thread should be breakpoint.
+        self.expect(
+            "thread list",
+            STOPPED_DUE_TO_BREAKPOINT,
+            substrs=['stopped', 'stop reason = breakpoint'])
+
+        # This is the function to remove the custom formats in order to have a
+        # clean slate for the next test case.
+        def cleanup():
+            self.runCmd('type format clear', check=False)
+            self.runCmd('type summary clear', check=False)
+            self.runCmd('type synth clear', check=False)
+
+        # Execute the cleanup function during test case tear down.
+        self.addTearDownHook(cleanup)
+
+        self.runCmd("type summary add --summary-string \"${var%@}\" MyClass")
+
+        self.expect("frame variable object2", substrs=['MyOtherClass'])
+
+        self.expect("frame variable *object2", substrs=['MyOtherClass'])
+
+        # Now let's delete the 'MyClass' custom summary.
+        self.runCmd("type summary delete MyClass")
+
+        # The type format list should not show 'MyClass' at this point.
+        self.expect("type summary list", matching=False, substrs=['MyClass'])
+
+        self.runCmd("type summary add --summary-string \"a test\" MyClass")
+
+        self.expect(
+            "frame variable *object2",
+            substrs=['*object2 =', 'MyClass = a test', 'backup = '])
+
+        self.expect(
+            "frame variable object2", matching=False, substrs=['a test'])
+
+        self.expect("frame variable object", substrs=['a test'])
+
+        self.expect("frame variable *object", substrs=['a test'])
+
+        self.expect(
+            'frame variable myclass', substrs=['(Class) myclass = NSValue'])
+        self.expect(
+            'frame variable myclass2',
+            substrs=['(Class) myclass2 = ', 'NS', 'String'])
+        self.expect(
+            'frame variable myclass3', substrs=['(Class) myclass3 = Molecule'])
+        self.expect(
+            'frame variable myclass4',
+            substrs=['(Class) myclass4 = NSMutableArray'])
+        self.expect(
+            'frame variable myclass5', substrs=['(Class) myclass5 = nil'])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjNSException.py

+# encoding: utf-8
+"""
+Test lldb data formatter subsystem.
+"""
+
+from __future__ import print_function
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+from ObjCDataFormatterTestCase import ObjCDataFormatterTestCase
+
+
+class ObjCDataFormatterNSException(ObjCDataFormatterTestCase):
+
+    @skipUnlessDarwin
+    @no_debug_info_test
+    def test_nsexception_with_run_command(self):
+        """Test formatters for NSException."""
+        self.appkit_tester_impl(self.nsexception_data_formatter_commands)
+
+    def nsexception_data_formatter_commands(self):
+        self.expect(
+            'frame variable except0 except1 except2 except3',
+            substrs=[
+                '(NSException *) except0 = ',
+                'name: @"TheGuyWhoHasNoName" - reason: @"cuz it\'s funny"',
+                '(NSException *) except1 = ',
+                'name: @"TheGuyWhoHasNoName~1" - reason: @"cuz it\'s funny"',
+                '(NSException *) except2 = ',
+                'name: @"TheGuyWhoHasNoName`2" - reason: @"cuz it\'s funny"',
+                '(NSException *) except3 = ',
+                'name: @"TheGuyWhoHasNoName/3" - reason: @"cuz it\'s funny"'
+            ])

lldb/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-objc/nsstring/TestDataFormatterNSString.py

def appkit_tester_impl(self, commands):
             self.runCmd('type summary clear', check=False)
             self.runCmd('type synth clear', check=False)
 
         # Execute the cleanup function during test case tear down.
         self.addTearDownHook(cleanup)
         commands()
 
     @skipUnlessDarwin
+    @no_debug_info_test
     def test_nsstring_with_run_command(self):
         """Test formatters for NSString."""
         self.appkit_tester_impl(self.nsstring_data_formatter_commands)
 
     @skipUnlessDarwin
+    @no_debug_info_test
     def test_rdar11106605_with_run_command(self):
         """Check that Unicode characters come out of CFString summary correctly."""
         self.appkit_tester_impl(self.rdar11106605_commands)
 
     @skipUnlessDarwin
+    @no_debug_info_test
     def test_nsstring_withNULS_with_run_command(self):
         """Test formatters for NSString."""
         self.appkit_tester_impl(self.nsstring_withNULs_commands)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.