[lldb] Add @skipIfAddressSanitizerUnsupported test decorator

[kubamracek]

Summary

Adding the @skipIfAddressSanitizerUnsupported decorator, which detects whether the compiler supports "-fsanitize=address", and skips the marked tests if not. Getting rid of the stupid "findBuiltClang" logic, in favor of just using the same compiler that you use for the rest of the tests.

Related to http://reviews.llvm.org/D6205. I also updated the ASan test cases to react more properly to the "stop reason = exec", which is probably a OS X only thing, on Linux where ASan is a static library, the relaunch doesn't happen. Doug, could you try this patch on Ubuntu, if it resolves the test failure you were seeing?

[emaste]

functionalities/asan/TestMemoryHistory.py

23 ↗

(On Diff #16228)

remove this

functionalities/asan/TestReportData.py

24 ↗

(On Diff #16228)

remove this

lldbtest.py

672–687 ↗

(On Diff #16228)

Can we change this to the way expectedFail is now done, i.e. add a skipIf(skip_fn), and then implement asan unsupported as something like

def skipIfAddressSanitizerUnsupported():
    def fn(self):
        return self.compilerSupportsAddressSanitizer()
    return skipIf(fn)
`

with the intent of moving the other skipIf conditions over later too

[kubamracek]

Updating the patch.

[DougSnyder]

with the updated patch, i still see fails for TestMemoryHistory.py and
TestReportData.py on ubuntu14.04 w/ clang

i believe i figured out how to apply the svn patch correctly. here is a
subset of TestMemoryHistory.py post-patch:

@skipIfRemote
@skipIfAddressSanitizerUnsupported
@dwarf_test
def test_with_dwarf (self):
    self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
    self.asan_tests ()

TestMemoryHistory has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

FAIL: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 28, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestMemoryHistory.py",
line 72, in asan_tests

  'Memory deallocated at', 'a.out`f2', 'main.c:%d' % self.line_free])
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Command 'memory history 'pointer''
returns successfully

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.450s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

TestReportData.py has the following output:

FAILED: cd /home/dsnyder/tot/build/tools/lldb/test && /usr/bin/python
/home/dsnyder/tot/llvm/tools/lldb/test/dosep.py -o "-q --arch=x86_64
--executable /home/dsnyder/tot/build/bin/lldb -s
/home/dsnyder/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS -C clang\
-Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestReportData.AsanTestReportDataCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestReportData.AsanTestReportDataCase)

FAIL: test_with_dwarf (TestReportData.AsanTestReportDataCase)

Traceback (most recent call last):

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 524, in wrapper
  func(*args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 509, in wrapper
  return func(self, *args, **kwargs)
File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 382, in wrapper
  return func(self, *args, **kwargs)
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
29, in test_with_dwarf

  self.asan_tests ()
File

"/home/dsnyder/tot/lldb/test/functionalities/asan/TestReportData.py", line
69, in asan_tests

substrs = ['stopped', 'stop reason = Use of deallocated memory

detected'])

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1873, in expect
  self.runCmd(str, msg=msg, trace = (True if trace else False), check =

not error, inHistory=inHistory)

File "/home/dsnyder/tot/lldb/test/lldbtest.py", line 1799, in runCmd
  msg if msg else CMD_MSG(cmd))

AssertionError: False is not True : Process should be stopped due to ASan
report

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 2.435s

FAILED (failures=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestReportData.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

it doesn't appear that these tests are being skipped

[kubamracek]

Updating the patch to only perform the ASan tests for clang version >= 3.6.0. The older versions do not have the new ASan debugging API. Doug, could you try the patch now?

[DougSnyder]

i'll try to test it monday

i think i can test it for clang 3.5 and clang 3.6.1

[DougSnyder]

i was able to try your new patch, using ubuntu14.04

the patch behaves well for both clang3.5 and clang3.6, but only if i use a
custom test rig called runtest.py.
the patch behaves poorly for both clang3.5 and clang3.6 if i use the
standard test method "ninja check-lldb"

generally, runtest.py and "ninja check-lld" produce the same results, so
the problem is probably something subtle

with runtest.py, i get the following message for both compilers:
Test: TestMemoryHistory.py found in functionalities/asan
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dsym
(TestMemoryHistory.AsanTestCase) (requires Darwin)
UNSUPPORTED: LLDB (/usr/bin/clang-x86_64) :: test_with_dwarf
(TestMemoryHistory.AsanTestCase) (skipping because compiler doesn't support

AddressSanitizer)

Ran 2 tests in 0.071s

OK (skipped=2)

with "ninja check-lldb", i get the following messages for both compilers:
[1/1] Testing LLDB (with a separate subprocess per test)
FAILED: cd /media/psf/Raid0/projects/tot/build/tools/lldb/test &&
/usr/bin/python /media/psf/Raid0/projects/tot/llvm/tools/lldb/test/dosep.py
-o "-q --arch=x86_64 --executable
/media/psf/Raid0/projects/tot/build/bin/lldb -s
/media/psf/Raid0/projects/tot/build/lldb-test-traces -u CXXFLAGS -u CFLAGS
-C clang\ -Qunused-arguments\ -fcolor-diagnostics"
UNSUPPORTED: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::
test_with_dsym (TestMemoryHistory.AsanTestCase) (requires Darwin)
FAIL: LLDB (clang -Qunused-arguments -fcolor-diagnostics-x86_64) ::

test_with_dwarf (TestMemoryHistory.AsanTestCase)

ERROR: test_with_dwarf (TestMemoryHistory.AsanTestCase)

Traceback (most recent call last):

File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 524, in

wrapper

  func(*args, **kwargs)
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 506, in

wrapper

  if skip_fn(self):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 689, in

fn

  return not self.compilerSupportsAddressSanitizer()
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1462, in

compilerSupportsAddressSanitizer

  if self.expectedCompilerVersion(["<", "3.6.0"]):
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1257, in

expectedCompilerVersion

  return self.getCompilerVersion() < version
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 1228, in

getCompilerVersion

  version_output = system([[which(compiler), "-v"]])[1]
File "/media/psf/Raid0/projects/tot/lldb/test/lldbtest.py", line 265, in

system

commandList = [' '.join(x) for x in commands]

TypeError: sequence item 0: expected string, NoneType found

Config=x86_64-clang -Qunused-arguments -fcolor-diagnostics

Ran 2 tests in 0.061s

FAILED (errors=1, skipped=1)
Ran 1 tests.
Failing Tests (1)
FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux ubuntu 3.13.0-29-generic
#53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64)
ninja: build stopped: subcommand failed.

in case someone wants to try running runtest.py on ubuntu, i have attached
runtest.py

to run the test with runtest.py:

copy runtest.py to the folder that contains lldb
type: ./runtest.py -c -p functionalities/asan/TestMemoryHistory.py

doug

• runtest.py5 KBDownload

[kubamracek]

This is now done properly by 1) checking out libcxx, and compiler-rt for an LLDB build, and 2) running tests against the just-built clang.

test/functionalities/asan/Makefile

 LEVEL = ../../make
 
 C_SOURCES := main.c
-CFLAGS := $(CFLAGS) -fsanitize=address -g
 
 include $(LEVEL)/Makefile.rules

test/functionalities/asan/TestMemoryHistory.py

 
     mydir = TestBase.compute_mydir(__file__)
 
-    # The default compiler ("clang") may not support Address Sanitizer or it
-    # may not have the debugging API which was recently added, so we're calling
-    # self.useBuiltClang() to use clang from the llvm-build directory instead
-
     @unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dsym_test
     def test_with_dsym (self):
-        compiler = self.findBuiltClang ()
-        self.buildDsym (None, compiler)
+        self.buildDsym (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
-    @skipIfFreeBSD # llvm.org/pr21136 runtimes not yet available by default
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dwarf_test
     def test_with_dwarf (self):
-        compiler = self.findBuiltClang ()
-        self.buildDwarf (None, compiler)
+        self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     def setUp(self):

 
         self.runCmd("run")
 
-        # ASan will relaunch the process to insert its library.
-        self.expect("thread list", "Process should be stopped due to exec.",
-            substrs = ['stopped', 'stop reason = exec'])
+        process = self.dbg.GetSelectedTarget().process
+        thread = process.GetSelectedThread()
+        stop_reason = thread.GetStopReason()
+        if stop_reason == lldb.eStopReasonExec:
+            # On OS X, ASan will relaunch the process to insert its library.
+            self.expect("thread list", "Process should be stopped due to exec.",
+                substrs = ['stopped', 'stop reason = exec'])
+            self.runCmd("continue")
 
-        self.runCmd("continue")
-
         # the stop reason of the thread should be breakpoint.
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
             substrs = ['stopped', 'stop reason = breakpoint'])

                 'Memory deallocated at', 'a.out`f2', 'main.c:%d' % self.line_free])
 
         # do the same using SB API
-        process = self.dbg.GetSelectedTarget().process
         val = process.GetSelectedThread().GetSelectedFrame().EvaluateExpression("pointer")
         addr = val.GetValueAsUnsigned()
         threads = process.GetHistoryThreads(addr);

test/functionalities/asan/TestReportData.py

 
     mydir = TestBase.compute_mydir(__file__)
 
-    # The default compiler ("clang") may not support Address Sanitizer or it
-    # may not have the debugging API which was recently added, so we're calling
-    # self.useBuiltClang() to use clang from the llvm-build directory instead
-
     @unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dsym_test
     def test_with_dsym (self):
-        compiler = self.findBuiltClang ()
-        self.buildDsym (None, compiler)
+        self.buildDsym (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
-    @skipIfFreeBSD # llvm.org/pr21136 runtimes not yet available by default
     @skipIfRemote
+    @skipIfAddressSanitizerUnsupported
     @dwarf_test
     def test_with_dwarf (self):
-        compiler = self.findBuiltClang ()
-        self.buildDwarf (None, compiler)
+        self.buildDwarf (None, None, {"CFLAGS": "-fsanitize=address -g"})
         self.asan_tests ()
 
     def setUp(self):

     def asan_tests (self):
         exe = os.path.join (os.getcwd(), "a.out")
         self.expect("file " + exe, patterns = [ "Current executable set to .*a.out" ])
+
+        self.runCmd("breakpoint set -f main.c -l %d" % self.line_breakpoint)
+
         self.runCmd("run")
 
-        # ASan will relaunch the process to insert its library.
-        self.expect("thread list", "Process should be stopped due to exec.",
-            substrs = ['stopped', 'stop reason = exec'])
+        process = self.dbg.GetSelectedTarget().process
+        thread = process.GetSelectedThread()
+        stop_reason = thread.GetStopReason()
+        if stop_reason == lldb.eStopReasonExec:
+            # On OS X, ASan will relaunch the process to insert its library.
+            self.expect("thread list", "Process should be stopped due to exec.",
+                substrs = ['stopped', 'stop reason = exec'])
+            self.runCmd("continue")
 
+        # the stop reason of the thread should be breakpoint.
+        self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
+            substrs = ['stopped', 'stop reason = breakpoint'])
+
         # no extended info when we have no ASan report
         thread = self.dbg.GetSelectedTarget().process.GetSelectedThread()
         s = lldb.SBStream()

test/lldbtest.py

 def expectedFailureLinux(bugnumber=None, compilers=None):
     if bugnumber: return expectedFailureOS('linux', bugnumber, compilers)
 
+def skipIf(skip_fn, method, reason):
+    def skipIf_impl(func):
+        @wraps(func)
+        def wrapper(self, *args, **kwargs):
+            if skip_fn(self):
+                self.skipTest(reason)
+            else:
+                return func(self, *args, **kwargs)
+        return wrapper
+    return skipIf_impl(method)
+
 def skipIfRemote(func):
     """Decorate the item to skip tests if testing remotely."""
     if isinstance(func, type) and issubclass(func, unittest2.TestCase):

             func(*args, **kwargs)
     return wrapper
 
+def skipIfAddressSanitizerUnsupported(method):
+    """
+    Decorate the item to skip tests that should skipped when the compiler we're using doesn't
+    support compiling with AddressSanitizer.
+    """
+    def fn(self):
+        return not self.compilerSupportsAddressSanitizer()
+    return skipIf(fn, method, "skipping because compiler doesn't support AddressSanitizer")
 
 class Base(unittest2.TestCase):
     """

         # See HideStdout(self).
         self.sys_stdout_hidden = False
 
+        self.cachedCompilerSupportsAddressSanitizer = None
+
         # set environment variable names for finding shared libraries
         if sys.platform.startswith("darwin"):
             self.dylibPath = 'DYLD_LIBRARY_PATH'

         if not module.buildDwarf(self, architecture, compiler, dictionary, clean):
             raise Exception("Don't know how to build binary with dwarf")
 
-    def findBuiltClang(self):
-        """Tries to find and use Clang from the build directory as the compiler (instead of the system compiler)."""
-        paths_to_try = [
-          "llvm-build/Release+Asserts/x86_64/Release+Asserts/bin/clang",
-          "llvm-build/Debug+Asserts/x86_64/Debug+Asserts/bin/clang",
-          "llvm-build/Release/x86_64/Release/bin/clang",
-          "llvm-build/Debug/x86_64/Debug/bin/clang",
-        ]
-        lldb_root_path = os.path.join(os.path.dirname(__file__), "..")
-        for p in paths_to_try:
-            path = os.path.join(lldb_root_path, p)
-            if os.path.exists(path):
-                return path
-        
-        return os.environ["CC"]
-
     def getBuildFlags(self, use_cpp11=True, use_libcxx=False, use_libstdcxx=False, use_pthreads=True):
         """ Returns a dictionary (which can be provided to build* functions above) which
             contains OS-specific build flags.

         else:
             return ['libc++.1.dylib','libc++abi.dylib']
 
+    def compilerSupportsAddressSanitizer(self):
+        if self.cachedCompilerSupportsAddressSanitizer != None:
+            return self.cachedCompilerSupportsAddressSanitizer
+
+        self.cachedCompilerSupportsAddressSanitizer = False
+
+        # We only support clang's AddressSanitizer.
+        if not "clang" in self.getCompiler():
+            return False
+
+        # Older versions of clang don't have the debugging functions in ASan.
+        if self.expectedCompilerVersion(["<", "3.6.0"]):
+            return False
+
+        # Save old working directory.
+        oldcwd = os.getcwd()
+
+        os.chdir(os.path.dirname(__file__) + "/functionalities/asan")
+
+        # This should always work
+        self.buildDefault();
+
+        # This will fail if the compiler doesn't have AddressSanitizer
+        try:
+            self.buildDefault (None, None, {"CFLAGS": "-fsanitize=address"})
+            self.cachedCompilerSupportsAddressSanitizer = True
+        except CalledProcessError:
+            pass
+
+        # Clean up
+        self.cleanup();
+        os.chdir(oldcwd)
+
+        return self.cachedCompilerSupportsAddressSanitizer
+
 class TestBase(Base):
     """
     This abstract base class is meant to be subclassed.  It provides default