[testsuite] Implement a category to skip libstdcxx tests

[davide]

Summary

On systems where it's not supported.
As far as I understand Linux is the only systems which now ships with libstdcxx (maybe NetBSD?, but I'm not entirely sure of the state of lldb on the platform).
We could make this more fine grained looking for the header as we do for libcxx. This is a little tricky as there's no such thing as /usr/include/c++/v1, but libstdcxx encodes the version number in the path (i.e. /usr/include/c++/5.4). I guess we might match a regex, but it seems fragile to me.

@labath I hope this is more in line with what you had in mind, thanks for your feedback!

[davide]

@labath this is breaking several bots for us internally so I would really love to get this in.
As this seems to be reasonably close to what you ask, I might consider checking this in before end of day (PST) today and we can iterate from there.

[JDevlieghere]

LGTM

[labath]

Sorry for the slow response, I am OOO officially :P.

I think this is fine in general. As for the list of platforms, android should be on that list but not for long (which is why one of the things I was planning to do next week was to implement something like this). I am not sure about NetBSD, but @krytarowski should be able to add that easily if that is the case.

For the category markup, I was hoping you would just place a .categories file in the appropriate subfolder, but after checking I see that this is actually consistent with how libc++ tests are marked up, so that is not a big deal either. I'll probably convert both of them to the filesystem-based markup next week.

[davide]

Thank you Pavel!

[krytarowski]

NetBSD ships with GCC style and Clang style userland. The former one ships with GNU libstdc++, the latter with libc++.

[krytarowski]

Probably the former with ship GDB at least for long time, the latter will switch to LLDB.

[krytarowski]

Through pkgsrc a user can install LLDB with either style of distribution, including GCC with libstdc++ one (I'm using this myself also for the development of LLVM related code).

BTW. NetBSD uses GPLv3 GNU toolchain.

lldb/trunk/packages/Python/lldbsuite/test/dotest.py

def checkLibcxxSupport():
     result, reason = canRunLibcxxTests()
     if result:
         return # libc++ supported
     if "libc++" in configuration.categoriesList:
         return # libc++ category explicitly requested, let it run.
     print("Libc++ tests will not be run because: " + reason)
     configuration.skipCategories.append("libc++")
 
+def canRunLibstdcxxTests():
+    from lldbsuite.test import lldbplatformutil
+
+    platform = lldbplatformutil.getPlatform()
+    if platform == "linux":
+      return True, "libstdcxx always present"
+    return False, "Don't know how to build with libstdcxx on %s" % platform
+
+def checkLibstdcxxSupport():
+    result, reason = canRunLibstdcxxTests()
+    if result:
+        return # libstdcxx supported
+    if "libstdcxx" in configuration.categoriesList:
+        return # libstdcxx category explicitly requested, let it run.
+    print("libstdcxx tests will not be run because: " + reason)
+    configuration.skipCategories.append("libstdcxx")
+
 def checkDebugInfoSupport():
     import lldb
 
     platform = lldb.DBG.GetSelectedPlatform().GetTriple().split('-')[2]
     compiler = configuration.compiler
     skipped = []
     for cat in test_categories.debug_info_categories:
         if cat in configuration.categoriesList:
def run_suite():
     # Note that it's not dotest's job to clean this directory.
     import lldbsuite.test.lldbutil as lldbutil
     build_dir = configuration.test_build_dir
     lldbutil.mkdir_p(build_dir)
 
     target_platform = lldb.DBG.GetSelectedPlatform().GetTriple().split('-')[2]
 
     checkLibcxxSupport()
+    checkLibstdcxxSupport()
     checkDebugInfoSupport()
 
     # Don't do debugserver tests on anything except OS X.
     configuration.dont_do_debugserver_test = "linux" in target_platform or "freebsd" in target_platform or "windows" in target_platform
 
     # Don't do lldb-server (llgs) tests on anything except Linux.
     configuration.dont_do_llgs_test = not ("linux" in target_platform)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/iterator/TestDataFormatterStdIterator.py

class StdIteratorDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that libstdcpp iterators format properly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line(
             self, "main.cpp", self.line, num_expected_locations=-1)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/list/TestDataFormatterStdList.py

def setUp(self):
         TestBase.setUp(self)
         # Find the line numbers to break at for the different tests.
         self.line = line_number('main.cpp', '// Set break point at this line.')
         self.optional_line = line_number(
             'main.cpp', '// Optional break point at this line.')
         self.final_line = line_number(
             'main.cpp', '// Set final break point at this line.')
 
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that that file and class static variables display correctly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line(
             self, "main.cpp", self.line, num_expected_locations=-1)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/map/TestDataFormatterStdMap.py

class StdMapDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfFreeBSD
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that that file and class static variables display correctly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_source_regexp(
             self, "Set break point at this line.")
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/smart_ptr/TestDataFormatterStdSmartPtr.py

 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 class StdSmartPtrDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
-    @skipIfFreeBSD
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfDarwin  # doesn't compile on Darwin
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_source_regexp(
             self, "Set break point at this line.")
         self.runCmd("run", RUN_SUCCEEDED)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/string/TestDataFormatterStdString.py

class StdStringDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that that file and class static variables display correctly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line(
             self, "main.cpp", self.line, num_expected_locations=-1)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py

 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 class StdTupleDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
-    @skipIfFreeBSD
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfDarwin  # doesn't compile on Darwin
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_source_regexp(
             self, "Set break point at this line.")
         self.runCmd("run", RUN_SUCCEEDED)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/unique_ptr/TestDataFormatterStdUniquePtr.py

 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 class StdUniquePtrDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
-    @skipIfFreeBSD
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfDarwin  # doesn't compile on Darwin
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_source_regexp(
             self, "Set break point at this line.")
         self.runCmd("run", RUN_SUCCEEDED)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

class StdVBoolDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @expectedFailureAll(
-        oslist=['freebsd'],
-        bugnumber='llvm.org/pr20548 fails to build on lab.llvm.org buildbot')
-    @skipIfWindows  # libstdcpp not ported to Windows.
-    @skipIfDarwin
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that that file and class static variables display correctly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_file_and_line(
             self, "main.cpp", self.line, num_expected_locations=-1)
 

lldb/trunk/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vector/TestDataFormatterStdVector.py

class StdVectorDataFormatterTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break at.
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @skipIfFreeBSD
-    @skipIfWindows  # libstdcpp not ported to Windows
-    @skipIfwatchOS  # libstdcpp not ported to watchos
+    @add_test_categories(["libstdcxx"])
     def test_with_run_command(self):
         """Test that that file and class static variables display correctly."""
         self.build()
         self.runCmd("file " + self.getBuildArtifact("a.out"), CURRENT_EXECUTABLE_SET)
 
         lldbutil.run_break_set_by_source_regexp(
             self, "Set break point at this line.")
 

lldb/trunk/packages/Python/lldbsuite/test/test_categories.py

 all_categories = {
     'dataformatters': 'Tests related to the type command and the data formatters subsystem',
     'dwarf': 'Tests that can be run with DWARF debug information',
     'dwo': 'Tests that can be run with DWO debug information',
     'dsym': 'Tests that can be run with DSYM debug information',
     'gmodules': 'Tests that can be run with -gmodules debug information',
     'expression': 'Tests related to the expression parser',
     'libc++': 'Test for libc++ data formatters',
+    'libstdcxx': 'Test for libstdcxx data formatters',
     'objc': 'Tests related to the Objective-C programming language support',
     'pyapi': 'Tests related to the Python API',
     'basic_process': 'Basic process execution sniff tests.',
     'cmdline': 'Tests related to the LLDB command-line interface',
     'dyntype': 'Tests related to dynamic type support',
     'stresstest': 'Tests related to stressing lldb limits',
     'flakey': 'Flakey test cases, i.e. tests that do not reliably pass at each execution',
     'lldb-mi': 'lldb-mi tests',