[lldb] Add @skipIfAddressSanitizerUnsupported test decorator

[kubamracek]

Summary

Adding the @skipIfAddressSanitizerUnsupported decorator, which detects whether the compiler supports "-fsanitize=address", and skips the marked tests if not. Getting rid of the stupid "findBuiltClang" logic, in favor of just using the same compiler that you use for the rest of the tests.

Related to http://reviews.llvm.org/D6205. I also updated the ASan test cases to react more properly to the "stop reason = exec", which is probably a OS X only thing, on Linux where ASan is a static library, the relaunch doesn't happen. Doug, could you try this patch on Ubuntu, if it resolves the test failure you were seeing?

[emaste]

functionalities/asan/TestMemoryHistory.py

23

remove this

functionalities/asan/TestReportData.py

24

remove this

lldbtest.py

672–687

Can we change this to the way expectedFail is now done, i.e. add a skipIf(skip_fn), and then implement asan unsupported as something like

def skipIfAddressSanitizerUnsupported():
    def fn(self):
        return self.compilerSupportsAddressSanitizer()
    return skipIf(fn)
`

with the intent of moving the other skipIf conditions over later too

[kubamracek]

Updating the patch.

[DougSnyder]

with the updated patch, i still see fails for TestMemoryHistory.py and
TestReportData.py on ubuntu14.04 w/ clang

i believe i figured out how to apply the svn patch correctly. here is a
subset of TestMemoryHistory.py post-patch:

@skipIfRemote
@skipIfAddressSanitizerUnsupported
@dwarf_test
def test_with_dwarf (self):
    self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
    self.asan_tests ()

TestMemoryHistory has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

FAIL: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 28, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 72, in asan_tests

  'Memory deallocated at', 'a.out`f2', 'main.c:%d' % self.line_free])
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Command 'memory history 'pointer''
returns successfully

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.450s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

TestReportData.py has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestReportData.AsanTestReportDataCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestReportData.AsanTestReportDataCase)

FAIL: test_with_dwarf (TestReportData.AsanTestReportDataCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
29, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
69, in asan_tests

substrs = ['stopped', 'stop reason = Use of deallocated memory

detected'])

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Process should be stopped due to ASan
report

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 2.435s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestReportData.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

it doesn't appear that these tests are being skipped

[kubamracek]

Updating the patch to only perform the ASan tests for clang version >= 3.6.0. The older versions do not have the new ASan debugging API. Doug, could you try the patch now?

[DougSnyder]

i'll try to test it monday

i think i can test it for clang 3.5 and clang 3.6.1

[DougSnyder]

i was able to try your new patch, using ubuntu14.04

the patch behaves well for both clang3.5 and clang3.6, but only if i use a
custom test rig called runtest.py.
the patch behaves poorly for both clang3.5 and clang3.6 if i use the
standard test method "ninja check-lldb"

generally, runtest.py and "ninja check-lld" produce the same results, so
the problem is probably something subtle

with runtest.py, i get the following message for both compilers:
Test: TestMemoryHistory.py found in functionalities/asan
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dsym
(TestMemoryHistory.AsanTestCase) (requires Darwin)
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dwarf
(TestMemoryHistory.AsanTestCase) (skipping because compiler doesn't support

AddressSanitizer)

Ran 2 tests in 0.071s

OK (skipped=2)

with "ninja check-lldb", i get the following messages for both compilers:
[1/1] Testing LLDB (with a separate subprocess per test)
FAILED: cd /media/psf/Raid0/projects/tot/build/tools/lldb/test &&
/usr/bin/python /media/psf/Raid0/projects/tot/llvm/tools/lldb/test/dosep.py
-o "-q --arch=x86_64 --executable
/media/psf/Raid0/projects/tot/build/bin/lldb -s
/media/psf/Raid0/projects/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS
-C clang\ -Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

ERROR: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 524, in

wrapper

  func(*args, **kwargs)
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 506, in

wrapper

  if skip_fn(self):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 689, in

fn

  return not self.compilerSupportsAddressSanitizer()
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1462, in

compilerSupportsAddressSanitizer

  if self.expectedCompilerVersion(["<", "3.6.0"]):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1257, in

expectedCompilerVersion

  return self.getCompilerVersion() < version
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1228, in

getCompilerVersion

  version_output = system([[which(compiler), "-v"]])[1]
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 265, in

system

commandList = [' '.join(x) for x in commands]

TypeError: sequence item 0: expected string, NoneType found

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.061s

FAILED (errors=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

in case someone wants to try running runtest.py on ubuntu, i have attached
runtest.py

to run the test with runtest.py:

copy runtest.py to the folder that contains lldb
type: ./runtest.py -c -p functionalities/asan/TestMemoryHistory.py

doug

• runtest.py5 KBDownload

[kubamracek]

This is now done properly by 1) checking out libcxx, and compiler-rt for an LLDB build, and 2) running tests against the just-built clang.

functionalities/asan/Makefile

 LEVEL = ../../make
 
 C_SOURCES := main.c
-CFLAGS := $(CFLAGS) -fsanitize=address -g
 
 include $(LEVEL)/Makefile.rules

functionalities/asan/TestMemoryHistory.py

 """
 Test that ASan memory history provider returns correct stack traces
 """
 
 import os, time
 import unittest2
 import lldb
 from lldbtest import *
 import lldbutil
 
 class AsanTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # The default compiler ("clang") may not support Address Sanitizer or it
-    # may not have the debugging API which was recently added, so we're calling
-    # self.useBuiltClang() to use clang from the llvm-build directory instead
-
     @unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dsym_test
     def test_with_dsym (self):
-        compiler = self.findBuiltClang ()
-        self.buildDsym (None, compiler)
+        self.buildDsym (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     @skipIfFreeBSD # llvm.org/pr21136 runtimes not yet available by default
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dwarf_test
     def test_with_dwarf (self):
-        compiler = self.findBuiltClang ()
-        self.buildDwarf (None, compiler)
+        self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         self.line_malloc = line_number('main.c', '// malloc line')
         self.line_malloc2 = line_number('main.c', '// malloc2 line')
         self.line_free = line_number('main.c', '// free line')
         self.line_breakpoint = line_number('main.c', '// break line')
 
     def asan_tests (self):
         exe = os.path.join (os.getcwd(), "a.out")
         self.expect("file " + exe, patterns = [ "Current executable set to .*a.out" ])
 
         self.runCmd("breakpoint set -f main.c -l %d" % self.line_breakpoint)
 
         # "memory history" command should not work without a process
         self.expect("memory history 0",
             error = True,
             substrs = ["invalid process"])
 
         self.runCmd("run")
 
-        # ASan will relaunch the process to insert its library.
+        process = self.dbg.GetSelectedTarget().process
+        thread = process.GetSelectedThread()
+        stop_reason = thread.GetStopReason()
+        if stop_reason == lldb.eStopReasonExec:
+            # On OS X, ASan will relaunch the process to insert its library.
             self.expect("thread list", "Process should be stopped due to exec.",
                 substrs = ['stopped', 'stop reason = exec'])
-
             self.runCmd("continue")
 
         # the stop reason of the thread should be breakpoint.
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
             substrs = ['stopped', 'stop reason = breakpoint'])
 
         # test that the ASan dylib is present
         self.expect("image lookup -n __asan_describe_address", "__asan_describe_address should be present",
             substrs = ['1 match found'])
 
         # test the 'memory history' command
         self.expect("memory history 'pointer'",
             substrs = [
                 'Memory allocated at', 'a.out`f1', 'main.c:%d' % self.line_malloc,
                 'Memory deallocated at', 'a.out`f2', 'main.c:%d' % self.line_free])
 
         # do the same using SB API
-        process = self.dbg.GetSelectedTarget().process
         val = process.GetSelectedThread().GetSelectedFrame().EvaluateExpression("pointer")
         addr = val.GetValueAsUnsigned()
         threads = process.GetHistoryThreads(addr);
         self.assertEqual(threads.GetSize(), 2)
         
         history_thread = threads.GetThreadAtIndex(0)
         self.assertTrue(history_thread.num_frames >= 2)
         self.assertEqual(history_thread.frames[1].GetLineEntry().GetFileSpec().GetFilename(), "main.c")

functionalities/asan/TestReportData.py

 """
 Test the AddressSanitizer runtime support for report breakpoint and data extraction.
 """
 
 import os, time
 import unittest2
 import lldb
 from lldbtest import *
 import lldbutil
 import json
 
 class AsanTestReportDataCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # The default compiler ("clang") may not support Address Sanitizer or it
-    # may not have the debugging API which was recently added, so we're calling
-    # self.useBuiltClang() to use clang from the llvm-build directory instead
-
     @unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dsym_test
     def test_with_dsym (self):
-        compiler = self.findBuiltClang ()
-        self.buildDsym (None, compiler)
+        self.buildDsym (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     @skipIfFreeBSD # llvm.org/pr21136 runtimes not yet available by default
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dwarf_test
     def test_with_dwarf (self):
-        compiler = self.findBuiltClang ()
-        self.buildDwarf (None, compiler)
+        self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         self.line_malloc = line_number('main.c', '// malloc line')
         self.line_malloc2 = line_number('main.c', '// malloc2 line')
         self.line_free = line_number('main.c', '// free line')
         self.line_breakpoint = line_number('main.c', '// break line')
         self.line_crash = line_number('main.c', '// BOOM line')
 
     def asan_tests (self):
         exe = os.path.join (os.getcwd(), "a.out")
         self.expect("file " + exe, patterns = [ "Current executable set to .*a.out" ])
+
+        self.runCmd("breakpoint set -f main.c -l %d" % self.line_breakpoint)
+
         self.runCmd("run")
 
-        # ASan will relaunch the process to insert its library.
+        process = self.dbg.GetSelectedTarget().process
+        thread = process.GetSelectedThread()
+        stop_reason = thread.GetStopReason()
+        if stop_reason == lldb.eStopReasonExec:
+            # On OS X, ASan will relaunch the process to insert its library.
             self.expect("thread list", "Process should be stopped due to exec.",
                 substrs = ['stopped', 'stop reason = exec'])
+            self.runCmd("continue")
+
+        # the stop reason of the thread should be breakpoint.
+        self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
+            substrs = ['stopped', 'stop reason = breakpoint'])
 
         # no extended info when we have no ASan report
         thread = self.dbg.GetSelectedTarget().process.GetSelectedThread()
         s = lldb.SBStream()
         self.assertFalse(thread.GetStopReasonExtendedInfoAsJSON(s))
 
         self.runCmd("continue")
 

lldbtest.py

         from unittest2 import case
         self = args[0]
         if "i386" == self.getArchitecture():
             self.skipTest("skipping because i386 is not a supported architecture")
         else:
             func(*args, **kwargs)
     return wrapper
 
+def skipIfAddressSanitizerUnsupported(func):
+    """
+    Decorate the item to skip tests that should skipped when the compiler we're using doesn't
+    support compiling with AddressSanitizer.
+    """
+    if isinstance(func, type) and issubclass(func, unittest2.TestCase):
+        raise Exception("@skipIfAddressSanitizerUnsupported can only be used to decorate a test method")
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        from unittest2 import case
+        self = args[0]
+        if not self.compilerSupportsAddressSanitizer():
+            self.skipTest("skipping because compiler doesn't support AddressSanitizer")
+        else:
+            func(*args, **kwargs)
+    return wrapper
 
 class Base(unittest2.TestCase):
     """
     Abstract base for performing lldb (see TestBase) or other generic tests (see
     BenchBase for one example).  lldbtest.Base works with the test driver to
     accomplish things.
     
     """

 
         # See addTearDownHook(self, hook) which allows the client to add a hook
         # function to be run during tearDown() time.
         self.hooks = []
 
         # See HideStdout(self).
         self.sys_stdout_hidden = False
 
+        self.cachedCompilerSupportsAddressSanitizer = None
+
         # set environment variable names for finding shared libraries
         if sys.platform.startswith("darwin"):
             self.dylibPath = 'DYLD_LIBRARY_PATH'
         elif sys.platform.startswith("linux") or sys.platform.startswith("freebsd"):
             self.dylibPath = 'LD_LIBRARY_PATH'
 
     def runHooks(self, child=None, child_prompt=None, use_cmd_api=False):
         """Perform the run hooks to bring lldb debugger to the desired state.

     def buildDwarf(self, architecture=None, compiler=None, dictionary=None, clean=True):
         """Platform specific way to build binaries with dwarf maps."""
         if lldb.skip_build_and_cleanup:
             return
         module = builder_module()
         if not module.buildDwarf(self, architecture, compiler, dictionary, clean):
             raise Exception("Don't know how to build binary with dwarf")
 
-    def findBuiltClang(self):
-        """Tries to find and use Clang from the build directory as the compiler (instead of the system compiler)."""
-        paths_to_try = [
-          "llvm-build/Release+Asserts/x86_64/Release+Asserts/bin/clang",
-          "llvm-build/Debug+Asserts/x86_64/Debug+Asserts/bin/clang",
-          "llvm-build/Release/x86_64/Release/bin/clang",
-          "llvm-build/Debug/x86_64/Debug/bin/clang",
-        ]
-        lldb_root_path = os.path.join(os.path.dirname(__file__), "..")
-        for p in paths_to_try:
-            path = os.path.join(lldb_root_path, p)
-            if os.path.exists(path):
-                return path
-        
-        return os.environ["CC"]
-
     def getBuildFlags(self, use_cpp11=True, use_libcxx=False, use_libstdcxx=False, use_pthreads=True):
         """ Returns a dictionary (which can be provided to build* functions above) which
             contains OS-specific build flags.
         """
         cflags = ""
 
         # On Mac OS X, unless specifically requested to use libstdc++, use libc++
         if not use_libstdcxx and sys.platform.startswith('darwin'):

             return self.lib_dir
 
     def getLibcPlusPlusLibs(self):
         if sys.platform.startswith('freebsd'):
             return ['libc++.so.1']
         else:
             return ['libc++.1.dylib','libc++abi.dylib']
 
+    def compilerSupportsAddressSanitizer(self):
+        if self.cachedCompilerSupportsAddressSanitizer != None:
+            return self.cachedCompilerSupportsAddressSanitizer
+
+        self.cachedCompilerSupportsAddressSanitizer = False
+
+        # Save old working directory.
+        oldcwd = os.getcwd()
+
+        os.chdir(os.path.dirname(__file__) + "/functionalities/asan")
+
+        # This should always work
+        self.buildDefault();
+
+        # This will fail if the compiler doesn't have AddressSanitizer
+        try:
+            self.buildDefault (None, None, {"CFLAGS": "-fsanitize=address"})
+            self.cachedCompilerSupportsAddressSanitizer = True
+        except CalledProcessError:
+            pass
+
+        # Clean up
+        self.cleanup();
+        os.chdir(oldcwd)
+
+        return self.cachedCompilerSupportsAddressSanitizer
+
 class TestBase(Base):
     """
     This abstract base class is meant to be subclassed.  It provides default
     implementations for setUpClass(), tearDownClass(), setUp(), and tearDown(),
     among other things.
 
     Important things for test class writers: