xfail tests that failed with clang-3.7, gcc4.8.2 and on i386 to get buildbot green

[chying]

Summary

-Refer to bug https://buganizer.corp.google.com/issues/19893563
-Test log http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/1145

[chying]

List of ignored tests,

For clang>=3.6,
FAIL: LLDB (suite):: TestCPPThis.py

For gcc,
FAIL: LLDB (suite):: TestMiBreak.py
FAIL: LLDB (suite):: TestPluginCommands.py
FAIL: LLDB (suite):: TestRegisterVariables.py
FAIL: LLDB (suite):: TestTargetWatchAddress.py
FAIL: LLDB (suite):: TestTypedefArray.py
FAIL: LLDB (suite):: TestWatchLocation.py
FAIL: LLDB (suite):: TestWatchLocationWithWatchSet.py
FAIL: LLDB (suite):: TestWatchpointSetErrorCases.py

For i386,
FAIL: LLDB (suite):: TestDataFormatterStdVBool.py
FAIL: LLDB (suite):: TestMiExec.py
FAIL: LLDB (suite):: TestNoreturnUnwind.py

[sivachandra]

My comments are not just for the author, but also for other reviewers on this change.

test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

27 ↗

(On Diff #22536)

This and similar comments attached to expectedFailurei386 below are confusing. Perhaps s/x86_64/i386 ? Even then, seems redundant.

Another note: Tamas has a fix for this approved. He might land it any time soon. So, may be observe the bot after his fix lands and take out the xfail decorator if the test starts passing?

test/lang/cpp/this/TestCPPThis.py

25 ↗

(On Diff #22536)

Why do we have a bug number only for this? Also, seems like the right tracking bug number is 19893491.

Unrelated: If we should be putting internal bug numbers at all, could we just put http://b/<bugnumber> similar to how apple puts links to rdar bugs?

[sivachandra]

Oh, another point: The right subscribing list for this change is lldb-commits and not llvm-commits.

[vharron]

test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

27 ↗

(On Diff #22536)

Suggested replacement comment

# failing config: i386 binary running on lldb built for ubuntu 14.04 x86_64

test/lang/cpp/this/TestCPPThis.py

25 ↗

(On Diff #22536)

Yeah, let's put llvm bugzilla urls here only, try to keep it open.

[chying]

test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

27 ↗

(On Diff #22536)

The test was compiled for i386, but failed when running on x86_64 ubuntu system. It may or may not fail on actual i386 system, i didn't run this test on i386 system.

test/lang/cpp/this/TestCPPThis.py

25 ↗

(On Diff #22536)

I agree that we do not include internal bug numbers. I put bug number for this one, only because we need the second parameter. And currently expectedFailureClang doesn't work if the first argument is skipped.
I have another patch to fix this problem.
http://reviews.llvm.org/D8595
Once that one is landed, I will remove bugnumber from here.

[chying]

Update comments to be more descriptive
Change bug number to llvm bugzilla url

lldb/trunk/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/vbool/TestDataFormatterStdVBool.py

def test_with_dsym_and_run_command(self):
         """Test data formatter commands."""
         self.buildDsym()
         self.data_formatter_commands()
 
     @expectedFailureFreeBSD("llvm.org/pr20548") # fails to build on lab.llvm.org buildbot
     @dwarf_test
     @skipIfWindows # http://llvm.org/pr21800
     @skipIfDarwin
+    @expectedFailurei386 #xfail to get buildbot green, failing config: i386 binary running on ubuntu 14.04 x86_64
     def test_with_dwarf_and_run_command(self):
         """Test data formatter commands."""
         self.buildDwarf()
         self.data_formatter_commands()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)

lldb/trunk/test/functionalities/plugins/commands/TestPluginCommands.py

 """
 Test that plugins that load commands work correctly.
 """
 
 import os, time
 import re
 import unittest2
 import lldb
 from lldbtest import *
 import lldbutil
 
+@expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
 class PluginCommandTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         self.lib_dir = os.environ["LLDB_LIB_DIR"]

lldb/trunk/test/functionalities/unwind/noreturn/TestNoreturnUnwind.py

class NoreturnUnwind(TestBase):
     @dsym_test
     def test_with_dsym (self):
         """Test that we can backtrace correctly with 'noreturn' functions on the stack"""
         self.buildDsym()
         self.setTearDownCleanup()
         self.noreturn_unwind_tests()
 
     @dwarf_test
+    @expectedFailurei386 #xfail to get buildbot green, failing config: i386 binary running on ubuntu 14.04 x86_64
     def test_with_dwarf (self):
         """Test that we can backtrace correctly with 'noreturn' functions on the stack"""
         self.buildDwarf()
         self.setTearDownCleanup()
         self.noreturn_unwind_tests()
 
     def noreturn_unwind_tests (self):
         exe = os.path.join(os.getcwd(), "a.out")

lldb/trunk/test/functionalities/watchpoint/hello_watchlocation/TestWatchLocation.py

class HelloWatchLocationTestCase(TestBase):
     def test_hello_watchlocation_with_dsym(self):
         """Test watching a location with '-x size' option."""
         self.buildDsym(dictionary=self.d)
         self.setTearDownCleanup(dictionary=self.d)
         self.hello_watchlocation()
 
     @expectedFailureFreeBSD("llvm.org/pr18832")
     @dwarf_test
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_hello_watchlocation_with_dwarf(self):
         """Test watching a location with '-x size' option."""
         self.buildDwarf(dictionary=self.d)
         self.setTearDownCleanup(dictionary=self.d)
         self.hello_watchlocation()
 
     def setUp(self):
         # Call super's setUp().

lldb/trunk/test/functionalities/watchpoint/watchpoint_set_command/TestWatchLocationWithWatchSet.py

class WatchLocationUsingWatchpointSetTestCase(TestBase):
     def test_watchlocation_with_dsym_using_watchpoint_set(self):
         """Test watching a location with 'watchpoint set expression -w write -x size' option."""
         self.buildDsym(dictionary=self.d)
         self.setTearDownCleanup(dictionary=self.d)
         self.watchlocation_using_watchpoint_set()
 
     @expectedFailureFreeBSD('llvm.org/pr18832')
     @dwarf_test
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_watchlocation_with_dwarf_using_watchpoint_set(self):
         """Test watching a location with 'watchpoint set expression -w write -x size' option."""
         self.buildDwarf(dictionary=self.d)
         self.setTearDownCleanup(dictionary=self.d)
         self.watchlocation_using_watchpoint_set()
 
     def setUp(self):
         # Call super's setUp().

lldb/trunk/test/functionalities/watchpoint/watchpoint_set_command/TestWatchpointSetErrorCases.py

 """
 Test error cases for the 'watchpoint set' command to make sure it errors out when necessary.
 """
 
 import os, time
 import unittest2
 import lldb
 from lldbtest import *
 import lldbutil
 
 class WatchpointSetErrorTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @expectedFailureFreeBSD('llvm.org/pr18832')
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_error_cases_with_watchpoint_set(self):
         """Test error cases with the 'watchpoint set' command."""
         self.buildDwarf(dictionary=self.d)
         self.setTearDownCleanup(dictionary=self.d)
         self.error_cases_with_watchpoint_set()
 
     def setUp(self):
         # Call super's setUp().

lldb/trunk/test/lang/c/register_variables/TestRegisterVariables.py

class RegisterVariableTestCase(TestBase):
     @expectedFailureDarwin("llvm.org/pr20266")
     def test_with_dsym_and_run_command(self):
         """Test expressions on register values."""
         self.buildDsym()
         self.const_variable()
 
     @expectedFailureClang
     @dwarf_test
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_with_dwarf_and_run_command(self):
         """Test expressions on register values."""
         self.buildDwarf()
         self.const_variable()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)

lldb/trunk/test/lang/cpp/this/TestCPPThis.py

def test_with_dsym_and_run_command(self):
         """Test that the appropriate member variables are available when stopped in C++ static, inline, and const methods"""
         self.buildDsym()
         self.static_method_commands()
 
     #rdar://problem/9962849
     @expectedFailureGcc # llvm.org/pr15439 The 'this' pointer isn't available during expression evaluation when stopped in an inlined member function.
     @expectedFailureIcc # ICC doesn't emit correct DWARF inline debug info for inlined member functions
     @dwarf_test
+    @expectedFailureClang(bugnumber='llvm.org/pr23012', compiler_version=['>=','3.6'])#xfail to get buildbot green, test failed with totclang - clang3.7
     def test_with_dwarf_and_run_command(self):
         """Test that the appropriate member variables are available when stopped in C++ static, inline, and const methods"""
         self.buildDwarf()
         self.static_method_commands()
 
     def setUp(self):
         TestBase.setUp(self)
     

lldb/trunk/test/lldbinline.py

class InlineTest(TestBase):
 
     @unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
     def __test_with_dsym(self):
         self.using_dsym = True
         self.BuildMakefile()
         self.buildDsym()
         self.do_test()
 
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def __test_with_dwarf(self):
         self.using_dsym = False
         self.BuildMakefile()
         self.buildDwarf()
         self.do_test()
 
     def execute_user_command(self, __command):
         exec __command in globals(), locals()

lldb/trunk/test/python_api/watchpoint/watchlocation/TestTargetWatchAddress.py

class TargetWatchAddressAPITestCase(TestBase):
     @dsym_test
     def test_watch_address_with_invalid_watch_size_with_dsym(self):
         """Exercise SBTarget.WatchAddress() API but pass an invalid watch_size."""
         self.buildDsym()
         self.do_set_watchaddress_with_invalid_watch_size()
 
     @python_api_test
     @dwarf_test
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_watch_address_with_invalid_watch_size_with_dwarf(self):
         """Exercise SBTarget.WatchAddress() API but pass an invalid watch_size."""
         self.buildDwarf()
         self.do_set_watchaddress_with_invalid_watch_size()
 
     def do_set_watchaddress(self):
         """Use SBTarget.WatchAddress() to set a watchpoint and verify that the program stops later due to the watchpoint."""
         exe = os.path.join(os.getcwd(), "a.out")

lldb/trunk/test/tools/lldb-mi/breakpoint/TestMiBreak.py

 """
 Test lldb-mi -break-xxx commands.
 """
 
 import lldbmi_testcase
 from lldbtest import *
 import unittest2
 
 class MiBreakTestCase(lldbmi_testcase.MiTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @lldbmi_test
     @expectedFailureWindows("llvm.org/pr22274: need a pexpect replacement for windows")
     @skipIfFreeBSD # llvm.org/pr22411: Failure presumably due to known thread races
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_lldbmi_break_insert_function_pending(self):
         """Test that 'lldb-mi --interpreter' works for pending function breakpoints."""
 
         self.spawnLldbMi(args = None)
 
         self.runCmd("-file-exec-and-symbols %s" % self.myexe)
         self.expect("\^done")
 
         self.runCmd("-break-insert -f printf")
         self.expect("\^done,bkpt={number=\"1\"")
 
         self.runCmd("-exec-run")
         self.expect("\^running")
         self.expect("\*stopped,reason=\"breakpoint-hit\"")
 
     @lldbmi_test
     @expectedFailureWindows("llvm.org/pr22274: need a pexpect replacement for windows")
     @skipIfFreeBSD # llvm.org/pr22411: Failure presumably due to known thread races
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_lldbmi_break_insert_function(self):
         """Test that 'lldb-mi --interpreter' works for function breakpoints."""
 
         self.spawnLldbMi(args = None)
 
         self.runCmd("-file-exec-and-symbols %s" % self.myexe)
         self.expect("\^done")
 
def test_lldbmi_break_insert_file_line_pending(self):
 
         self.runCmd("-exec-run")
         self.expect("\^running")
         self.expect("\*stopped,reason=\"breakpoint-hit\"")
 
     @lldbmi_test
     @expectedFailureWindows("llvm.org/pr22274: need a pexpect replacement for windows")
     @skipIfFreeBSD # llvm.org/pr22411: Failure presumably due to known thread races
+    @expectedFailureGcc #xfail to get buildbot green, test failed with gcc4.8.2
     def test_lldbmi_break_insert_file_line(self):
         """Test that 'lldb-mi --interpreter' works for file:line breakpoints."""
 
         self.spawnLldbMi(args = None)
 
         self.runCmd("-file-exec-and-symbols %s" % self.myexe)
         self.expect("\^done")
 

lldb/trunk/test/tools/lldb-mi/control/TestMiExec.py

def test_lldbmi_exec_next(self):
         # Test that an invalid --frame is handled
         # FIXME: no error is returned
         self.runCmd("-exec-next --frame 10")
         #self.expect("\^error: Frame index 10 is out of range")
 
     @lldbmi_test
     @expectedFailureWindows("llvm.org/pr22274: need a pexpect replacement for windows")
     @skipIfFreeBSD # llvm.org/pr22411: Failure presumably due to known thread races
+    @expectedFailurei386 #xfail to get buildbot green, failing config: i386 binary running on ubuntu 14.04 x86_64
     def test_lldbmi_exec_next_instruction(self):
         """Test that 'lldb-mi --interpreter' works for instruction stepping."""
 
         self.spawnLldbMi(args = None)
 
         # Load executable
         self.runCmd("-file-exec-and-symbols %s" % self.myexe)
         self.expect("\^done")