Page MenuHomePhabricator

[lldb] Add @skipIfAddressSanitizerUnsupported test decorator
AbandonedPublic

Authored by kubamracek on Nov 14 2014, 10:37 AM.

Details

Reviewers
None
Summary

Adding the @skipIfAddressSanitizerUnsupported decorator, which detects whether the compiler supports "-fsanitize=address", and skips the marked tests if not. Getting rid of the stupid "findBuiltClang" logic, in favor of just using the same compiler that you use for the rest of the tests.

Related to http://reviews.llvm.org/D6205. I also updated the ASan test cases to react more properly to the "stop reason = exec", which is probably a OS X only thing, on Linux where ASan is a static library, the relaunch doesn't happen. Doug, could you try this patch on Ubuntu, if it resolves the test failure you were seeing?

Diff Detail

Event Timeline

kubamracek retitled this revision from to [lldb] Add @skipIfAddressSanitizerUnsupported test decorator.
kubamracek updated this object.
kubamracek edited the test plan for this revision. (Show Details)
kubamracek added subscribers: Unknown Object (MLST), DougSnyder, granata.enrico and 2 others.
emaste added a subscriber: emaste.Nov 14 2014, 11:28 AM
emaste added inline comments.
functionalities/asan/TestMemoryHistory.py
23 ↗(On Diff #16228)

remove this

functionalities/asan/TestReportData.py
24 ↗(On Diff #16228)

remove this

lldbtest.py
672–687 ↗(On Diff #16228)

Can we change this to the way expectedFail is now done, i.e. add a skipIf(skip_fn), and then implement asan unsupported as something like

def skipIfAddressSanitizerUnsupported():
    def fn(self):
        return self.compilerSupportsAddressSanitizer()
    return skipIf(fn)
`

with the intent of moving the other skipIf conditions over later too

kubamracek updated this revision to Diff 16250.Nov 14 2014, 2:28 PM

Updating the patch.

with the updated patch, i still see fails for TestMemoryHistory.py and
TestReportData.py on ubuntu14.04 w/ clang

i believe i figured out how to apply the svn patch correctly. here is a
subset of TestMemoryHistory.py post-patch:

@skipIfRemote
@skipIfAddressSanitizerUnsupported
@dwarf_test
def test_with_dwarf (self):
    self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
    self.asan_tests ()

TestMemoryHistory has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

FAIL: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 28, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 72, in asan_tests

  'Memory deallocated at', 'a.out`f2', 'main.c:%d' % self.line_free])
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Command 'memory history 'pointer''
returns successfully

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.450s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

TestReportData.py has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestReportData.AsanTestReportDataCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestReportData.AsanTestReportDataCase)

FAIL: test_with_dwarf (TestReportData.AsanTestReportDataCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
29, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
69, in asan_tests

substrs = ['stopped', 'stop reason = Use of deallocated memory

detected'])

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Process should be stopped due to ASan
report

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 2.435s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestReportData.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

it doesn't appear that these tests are being skipped

kubamracek updated this revision to Diff 16271.Nov 15 2014, 2:18 PM

Updating the patch to only perform the ASan tests for clang version >= 3.6.0. The older versions do not have the new ASan debugging API. Doug, could you try the patch now?

i'll try to test it monday

i think i can test it for clang 3.5 and clang 3.6.1

i was able to try your new patch, using ubuntu14.04

the patch behaves well for both clang3.5 and clang3.6, but only if i use a
custom test rig called runtest.py.
the patch behaves poorly for both clang3.5 and clang3.6 if i use the
standard test method "ninja check-lldb"

generally, runtest.py and "ninja check-lld" produce the same results, so
the problem is probably something subtle

with runtest.py, i get the following message for both compilers:
Test: TestMemoryHistory.py found in functionalities/asan
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dsym
(TestMemoryHistory.AsanTestCase) (requires Darwin)
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dwarf
(TestMemoryHistory.AsanTestCase) (skipping because compiler doesn't support

AddressSanitizer)

Ran 2 tests in 0.071s

OK (skipped=2)

with "ninja check-lldb", i get the following messages for both compilers:
[1/1] Testing LLDB (with a separate subprocess per test)
FAILED: cd /media/psf/Raid0/projects/tot/build/tools/lldb/test &&
/usr/bin/python /media/psf/Raid0/projects/tot/llvm/tools/lldb/test/dosep.py
-o "-q --arch=x86_64 --executable
/media/psf/Raid0/projects/tot/build/bin/lldb -s
/media/psf/Raid0/projects/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS
-C clang\ -Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

ERROR: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 524, in

wrapper

  func(*args, **kwargs)
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 506, in

wrapper

  if skip_fn(self):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 689, in

fn

  return not self.compilerSupportsAddressSanitizer()
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1462, in

compilerSupportsAddressSanitizer

  if self.expectedCompilerVersion(["<", "3.6.0"]):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1257, in

expectedCompilerVersion

  return self.getCompilerVersion() < version
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1228, in

getCompilerVersion

  version_output = system([[which(compiler), "-v"]])[1]
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 265, in

system

commandList = [' '.join(x) for x in commands]

TypeError: sequence item 0: expected string, NoneType found

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.061s

FAILED (errors=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

in case someone wants to try running runtest.py on ubuntu, i have attached
runtest.py

to run the test with runtest.py:

copy runtest.py to the folder that contains lldb
type: ./runtest.py -c -p functionalities/asan/TestMemoryHistory.py

doug

kubamracek abandoned this revision.Apr 11 2016, 2:51 AM

This is now done properly by 1) checking out libcxx, and compiler-rt for an LLDB build, and 2) running tests against the just-built clang.