test infra: catch bad decorators and import-time errors

[tfiala]

Summary

This change enhances the LLDB test infrastructure to convert
load-time exceptions in a given Python test module into errors.
Before this change, specifying a non-existent test decorator,
or otherwise having some load-time error in a python test module,
would not get flagged as an error.

With this change, typos and other load-time errors in a python
test file get converted to errors and reported by the
test runner.

This change also includes test infrastructure tests that include
covering the new work here. I'm going to wait until we have
these infrastructure tests runnable on the main platforms before
I try to work that into all the normal testing workflows.

The test infrastructure tests can be run by using the standard python module testing practice of doing the following:

cd packages/Python/lldbsuite/test_event
python -m unittest discover -s test/src -p 'Test*.py'

Those tests run the dotest inferior with a known broken test and verify that the errors are caught. These tests did not pass until I modified dotest.py to capture them properly.

@zturner, if you have the chance, if you could try those steps above (the python -m unittest ... line) on Windows, that would be great if we can address any python2/3/Windows bits there. I don't think there's anything fancy, but I didn't want to hook it into test flow until I know it works there.

I'll be slowly adding more tests that cover some of the other breakage I've occasionally seen that didn't get collected as part of the summarization. This is the biggest one I'm aware of.

[labath]

I'm glad to see this getting fixed. I have a couple comments though. :)

packages/Python/lldbsuite/test/dotest.py

695 ↗

(On Diff #56990)

This function (visit) is getting quite big, and due to the way it's structured, you were forced to duplicate some code. We could solve both problems with a bit of refactoring.

How about this:

• put everything inside the for loop below this line into a separate function (visit_file ?)
• put the try-catch block around the call to visit_file in this function

This way we will have less duplication and will reduce the nesting depth of this function.

What do you think?

725 ↗

(On Diff #56990)

Will this ever be executed? I mean, if we get a pyc file, we shouldn't even get here due to the check on line 689.

packages/Python/lldbsuite/test_event/formatter/pickled.py

57 ↗

(On Diff #56990)

Why do we need the format to be different based on the kind of object we are writing to? The added magic (introspection) seems like a bad tradeoff, if all it does is avoid a couple of lines in event_collector.py

packages/Python/lldbsuite/test_event/test/src/TestCatchInvalidDecorator.py

38 ↗

(On Diff #56990)

I don't fully understand the whole event collector logic yet, but I would expect to see a check that the returned result is in fact an error result, no?

(Also, I prefer using assertGreater(foo, bar), as it provides a more descriptive error message.)

packages/Python/lldbsuite/test_event/test/src/event_collector.py

25 ↗

(On Diff #56990)

I'm pretty sure this won't work on windows.. :)
tempfile.NamedTemporaryFile looks like a portable alternative.

[tfiala]

I'll make some adjustments per your review, Pavel. Thanks! I'll put up another patch set at that point.

packages/Python/lldbsuite/test/dotest.py

695 ↗

(On Diff #56990)

Sounds like a good change.

725 ↗

(On Diff #56990)

Ah yes, that is true. Since we assert downstream from here if it was a .pyc file, I was being safe.

This will get refactored per previous comment, and at that point I will need it.

packages/Python/lldbsuite/test_event/formatter/pickled.py

57 ↗

(On Diff #56990)

When we're using sockets, we have to be able to know the size of the full info when reconstructing on the receiving side. This is the normal mode in which this is used. However, that also complicates the parsing of the data for the simple test driver.

The code later on in the test_event unit tests:

if os.path.exists(pickled_events_filename()):
    with open(pickled_events_filename(), "rb") as events_file:
        while True:
            try:
                # print("reading event")
                event = cPickle.load(events_file)
                # print("read event: {}".format(event))
                if event:
                    events.append(event)
            except EOFError:
                # This is okay.
                Break

Would get considerably more complicated to deal with the same format that is only required for going over a network-style protocol. I prefer this tradeoff. In the unit tests, I just send the event output to a file, and then read it with the simple loop I included above.

However, to verify that I really prefer it, I will try writing it the other way.

packages/Python/lldbsuite/test_event/test/src/TestCatchInvalidDecorator.py

38 ↗

(On Diff #56990)

.. but I would expect to see a check that the returned result is in fact an error result, no?

Ah yes. That is missing. (Before, we weren't getting any job_result or test_result events).

I'll add that.

(Also, I prefer using assertGreater(foo, bar), as it provides a more descriptive error message.)

Haha okay so I looked for that method when I originally worked this. The Python 2.7 page has a link to take to the assertion methods in a table. The table it points to doesn't have that call. However, they helpfully have two or three more tables below that. Not sure why all those calls don't go into the same table, particularly on a link for assertion methods from earlier in the page.

I totally agree, I'll change that.

packages/Python/lldbsuite/test_event/test/src/event_collector.py

25 ↗

(On Diff #56990)

Oh yes. That's right. I was thinking I'd get around to trying this on Windows myself, but that didn't happen. At that point I was going to crack out the docs on the temp file handling.

Thanks for catching.

[tfiala]

packages/Python/lldbsuite/test_event/formatter/pickled.py

57 ↗

(On Diff #56990)

However, to verify that I really prefer it, I will try writing it the other way.

The flip side is I can try to factor out the listener side logic that reconstructs these. However, that is currently rather tightly coupled to the network listening transport. And most of the work it does has purely to do with needing to receive the whole message before it can try to un-pickle it. So I'm not really seeing that as a big win. (Except for testability. So maybe it's okay to break that out.) I'll see what that looks like since that is probably the better high-level way to handle this if we didn't want the change I made to the sender side.

[zturner]

I will look at this in more detail later, but hold off on committing until
i have a chance to run it on Windows

[tfiala]

In D20193#428364, @zturner wrote:

I will look at this in more detail later, but hold off on committing until
i have a chance to run it on Windows

Hi Zachary.

No worries - I need to update the patch. Pavel caught one thing in the test for the new code that won't work on Windows that I forgot to fix up in the initial implementation. You might just as well wait for the next patch set to review it.

[labath]

packages/Python/lldbsuite/test_event/formatter/pickled.py

57 ↗

(On Diff #56990)

The other alternative I see is to make pass in a "serializer" object (or a lambda or something), which knows how to write to the right output. Then, you can construct the correct serializer object depending on whether you got passed --results-file or --results-port.

[tfiala]

Address most of Pavel's review comments.

New:

• Also catches multiple matching base names and converts that to a caught test error.

Comments from Pavel not covered:

• Modifying the dual behavior of the pickling results formatter.

I'm going to look at that one next.

[tfiala]

Adjust last patch to include full context.

[tfiala]

packages/Python/lldbsuite/test/dotest.py

725 ↗

(On Diff #56990)

You were right, once I refactored I saw I didn't need this check. I pulled it out.

packages/Python/lldbsuite/test_event/formatter/pickled.py

57 ↗

(On Diff #56990)

Still outstanding.

[tfiala]

Now addresses the remaining review comment from Pavel. ResultsFormatter instances are now told if their file is a stream rather than a block/char file.

The pickling formatter uses two different serializers, one for streaming and one for not.

[tfiala]

@zturner I think this is ready to go now. Can you give this a try on Windows?

Also, can you try the following part in cmd.exe?

cd packages\Python\lldbsuite\test_event
python -m unittest discover -s test\src -p 'Test*.py'

That should run 2 tests and they should pass, assuming no extra bits need to be added to the dotest.py that is called internally by them. The event_collector.py class, used to test the test infrastructure, builds a 'python dotest.py' command line that does what a dotest inferior will do during a normal test run. The output of that gets verified in the TestCatchInvalidDecorator.py test.

[zturner]

I had to leave early today, can I check it tomorrow?

[tfiala]

In D20193#428971, @zturner wrote:

I had to leave early today, can I check it tomorrow?

Sure, sounds good.

[labath]

Looks great from my side. Thank you for fixing this.

[tfiala]

Thanks, Pavel!

[zturner]

Going to look at this today, but I'm getting some crashes when I run the
lldb test suite, so I need to figure these out first so I have a clean
baseline for comparison.

[tfiala]

In D20193#429750, @zturner wrote:

Going to look at this today, but I'm getting some crashes when I run the
lldb test suite, so I need to figure these out first so I have a clean
baseline for comparison.

Okay.

[zturner]

No issues on Windows, looks good over here.

[zturner]

Do I still need to try the python -m unittest line? I didn't do that yet.

[zturner]

D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event>c:\Python35\python.exe
-m unittest discover -s test/src -p "Test*.py"
Traceback (most recent call last):

File "c:\Python35\lib\runpy.py", line 170, in _run_module_as_main
  "__main__", mod_spec)
File "c:\Python35\lib\runpy.py", line 85, in _run_code
  exec(code, run_globals)
File "c:\Python35\lib\unittest\__main__.py", line 18, in <module>
  main(module=None)
File "c:\Python35\lib\unittest\main.py", line 93, in __init__
  self.parseArgs(argv)
File "c:\Python35\lib\unittest\main.py", line 117, in parseArgs
  self._do_discovery(argv[2:])
File "c:\Python35\lib\unittest\main.py", line 228, in _do_discovery
  self.test = loader.discover(self.start, self.pattern, self.top)
File "c:\Python35\lib\unittest\loader.py", line 338, in discover
  raise ImportError('Start directory is not importable: %r' % start_dir)

ImportError: Start directory is not importable: 'test/src'

This directory named "src" doesn't exist. Am I missing something?

[tfiala]

This directory named "src" doesn't exist. Am I missing something?

Hmm let me try with a clean sync of the patch. Back in a moment...

[tfiala]

In D20193#429889, @tfiala wrote:

This directory named "src" doesn't exist. Am I missing something?

Hmm let me try with a clean sync of the patch. Back in a moment...

Oh you need to be one directory down - in test_event\test.

[tfiala]

(The test directory in the test_event package has a src dir for the package testing tests, and a resources dir for data/files used by the test.)

[tfiala]

Hey Zachary, forget about the test discovery part.

If you can just run this directly and let me know if that works, that'll be fine:

C:\Python35\python.exe D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event\test\src\TestCatchInvalidDecorator.py

That will be good enough to know the test can run on Windows. I may need to do a bit more homework to figure out how to get the discovery to work, but that's a completely orthogonal issue.

[zturner]

It can't find six, I guess because it's trying to run a script directly
rather than loading it as part of a package, so it never runs `import
use_lldb_suite`.

[tfiala]

In D20193#429903, @zturner wrote:

It can't find six, I guess because it's trying to run a script directly
rather than loading it as part of a package, so it never runs `import
use_lldb_suite`.

Ah of course. And this runs on OS X because six is part of the default Python site installation.

So an 'import use_lldb_suite' at the top of the event_collector file fixes it for you?

[zturner]

Well, I have to copy use_lldb_suite.py into lldbsuite/test_event first.
But then I get another error.

D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event>c:\Python35\python.exe
-m unittest discover -s test/src -p "Test*.py"

FF

FAIL: test_with_function_filter

(TestCatchInvalidDecorator.TestCatchInvalidDecorator)

Traceback (most recent call last):

File

"D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event\test\src\TestCatchInvalidDecorator.py",
line 56, in test_with_function_filter

"At least one job or test error result should have been returned")

AssertionError: 0 not greater than 0 : At least one job or test error
result should have been returned

FAIL: test_with_whole_file

(TestCatchInvalidDecorator.TestCatchInvalidDecorator)

Traceback (most recent call last):

File

"D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event\test\src\TestCatchInvalidDecorator.py",
line 37, in test_with_whole_file

"At least one job or test error result should have been returned")

AssertionError: 0 not greater than 0 : At least one job or test error
result should have been returned

[zturner]

I'm looking into a little bit. Currently I'm seeing that the temporary
events file is completely empty. I can run the same command line that
we're using in the subprocess.call() line, where in the code should I be
looking to see events getting written to the file?

[zturner]

Traceback (most recent call last):

File "D:\src\llvm\tools\lldb\test\dotest.py", line 7, in <module>
  lldbsuite.test.run_suite()
File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",

line 897, in run_suite

  setupTestResults()
File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",

line 481, in setupTestResults

  formatter_spec.formatter.handle_event(initialize_event)
File

"D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event\formatter\pickled.py",
line 72, in handle_event

  self.serializer.serialize(test_event, self.out_file)
File

"D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test_event\formatter\pickled.py",
line 45, in serialize

cPickle.dump(test_event, out_file)

TypeError: write() argument must be str, not bytes
Press any key to continue . . .

[tfiala]

This chunk of code:

with tempfile.NamedTemporaryFile(
        prefix="lldb_test_event_pickled_event_output",
        delete=False) as temp_file:
    return temp_file.name

inside event_collector.py comes up with a temp file name. (It opens and then immediately closes the file). That temp file name is then used as the file for which a "dotest.py" inferior will write output to, similar to how the real test suite has the dotest.py inferiors send messages over sockets. So the part that is really doing the writing is the dotest.py inferior.

Most likely, the dotest.py inferior isn't running. That means this subprocess is probably failing to run inside event_collector.py:

def _collect_events_with_command(command, events_filename):
    # Run the single test with dotest.py, outputting
    # the raw pickled events to a temp file.
    with open(os.devnull, 'w') as dev_null_file:
        subprocess.call(
            command,
            stdout=dev_null_file,
            stderr=dev_null_file)

    # Unpickle the events
    events = []
    if os.path.exists(events_filename):
        with open(events_filename, "rb") as events_file:
            while True:
                try:
                    # print("reading event")
                    event = cPickle.load(events_file)
                    # print("read event: {}".format(event))
                    if event:
                        events.append(event)
                except EOFError:
                    # This is okay.
                    break
        os.remove(events_filename)
    return events

There's a part that pipes that output to devnull. If you were to remove that piping to dev null, and just do this:

subprocess.call(command)

We'd probably see the actual error in trying to invoke dotest.py.

But at this point, I'd say you've helped a ton and put in more than enough time by verifying this isn't working quite yet. I will get a setup to repro and get it running there and address it directly.

Thanks, Zachary!

[zturner]

In lldbsuite\test_event\formatter\__init__.py, if I open the file with "wb"
instead of "w" I get further, but I still get other errors. This might
create a problem because you conditionally set the output file to stderr or
stdout, which are opened for text already, so those will never work with
cPickle.dump(). Do you have any good idea how to fix that?

In any case, for now I've set it to "wb" and I'm getting further. It's
returning results back to the unit test, but the results only contain a
begin and an end, but no error results.

[zturner]

Here's the patch I'm using so far to get to the point where dotest is
actually running and successfully returning results. It just doesn't
indicate any error results, only a job_begin and a job_end.

All this patch does is add use_lldb_suite.py in 2 places, import it, and
change a "w" to a "wb".

The question of why no errors are getting returned is still open.

• diff.txt3 KBDownload

[zturner]

Ok the problem is that it cannot find lldb.exe anywhere. Normally we use a
--executable argument when launching dotest.py, but there's no such
argument here., but

[tfiala]

In D20193#429982, @zturner wrote:

Ok the problem is that it cannot find lldb.exe anywhere. Normally we use a
--executable argument when launching dotest.py, but there's no such
argument here., but

Great, thanks for the patch!

I'll have a look at how we figure out where LLDB is for the real runner. I'm getting a repro setup together now.

Thanks again,
Todd

[tfiala]

Zachary and I have been talking. I'm going to get this in, but I'll revisit the logic of the "test the test runner" test, which needs some work to get running on Windows.

[zturner]

Ahh wait. Does the "wb" issue still need to be resolved first, or is that
specific to testing the test?

[tfiala]

In D20193#430073, @zturner wrote:

Ahh wait. Does the "wb" issue still need to be resolved first, or is that
specific to testing the test?

That was specific to testing the test. It normally goes over a socket, which doesn't have that open issue.

lldb/trunk/packages/Python/lldbsuite/test/dotest.py

if lldbPythonDir:
         lldbPythonDir = os.path.abspath(lldbPythonDir)
 
         # If tests need to find LLDB_FRAMEWORK, now they can do it
         os.environ["LLDB_FRAMEWORK"] = os.path.dirname(os.path.dirname(lldbPythonDir))
 
         # This is to locate the lldb.py module.  Insert it right after sys.path[0].
         sys.path[1:1] = [lldbPythonDir]
 
-def visit(prefix, dir, names):
-    """Visitor function for os.path.walk(path, visit, arg)."""
-
-    dir_components = set(dir.split(os.sep))
-    excluded_components = set(['.svn', '.git'])
-    if dir_components.intersection(excluded_components):
-        #print("Detected an excluded dir component: %s" % dir)
-        return
-
-    for name in names:
-        if '.py' == os.path.splitext(name)[1] and name.startswith(prefix):
-
-            if name in configuration.all_tests:
-                raise Exception("Found multiple tests with the name %s" % name)
-            configuration.all_tests.add(name)
 
-            # Try to match the regexp pattern, if specified.
-            if configuration.regexp:
-                import re
-                if re.search(configuration.regexp, name):
-                    #print("Filename: '%s' matches pattern: '%s'" % (name, regexp))
-                    pass
-                else:
-                    #print("Filename: '%s' does not match pattern: '%s'" % (name, regexp))
-                    continue
+def visit_file(dir, name):
+        if not re.search(configuration.regexp, name):
+            # We didn't match the regex, we're done.
+            return
 
-            # We found a match for our test.  Add it to the suite.
 
-            # Update the sys.path first.
-            if not sys.path.count(dir):
-                sys.path.insert(0, dir)
-            base = os.path.splitext(name)[0]
 
-            # Thoroughly check the filterspec against the base module and admit
-            # the (base, filterspec) combination only when it makes sense.
-            filterspec = None
-            for filterspec in configuration.filters:
-                # Optimistically set the flag to True.
-                filtered = True
-                module = __import__(base)
-                parts = filterspec.split('.')
-                obj = module
-                for part in parts:
-                    try:
-                        parent, obj = obj, getattr(obj, part)
-                    except AttributeError:
-                        # The filterspec has failed.
-                        filtered = False
-                        break
 
-                # If filtered, we have a good filterspec.  Add it.
-                if filtered:
-                    #print("adding filter spec %s to module %s" % (filterspec, module))
-                    configuration.suite.addTests(
-                        unittest2.defaultTestLoader.loadTestsFromName(filterspec, module))
-                    continue
+            # print("adding filter spec %s to module %s" % (filterspec, module))
 
-            # Forgo this module if the (base, filterspec) combo is invalid
-            if configuration.filters and not filtered:
-                continue
+        return
 
-            # Add either the filtered test case(s) (which is done before) or the entire test class.
-            if not filterspec or not filtered:
-                # A simple case of just the module name.  Also the failover case
-                # from the filterspec branch when the (base, filterspec) combo
-                # doesn't make sense.
-                configuration.suite.addTests(unittest2.defaultTestLoader.loadTestsFromName(base))
+        # Add the entire file's worth of tests since we're not filtered.
+        # Also the fail-over case when the filterspec branch
+        # (base, filterspec) combo doesn't make sense.
 
 
+def visit(prefix, dir, names):
+    """Visitor function for os.path.walk(path, visit, arg)."""
+
+    dir_components = set(dir.split(os.sep))
+    excluded_components = set(['.svn', '.git'])
+    if dir_components.intersection(excluded_components):
+        return
+
+    # Gather all the Python test file names that follow the Test*.py pattern.
+    python_test_files = [
+        name
+        for name in names
+        if name.endswith('.py') and name.startswith(prefix)]
+
+    # Visit all the python test files.
+    for name in python_test_files:
+        try:
+            # Ensure we error out if we have multiple tests with the same
+            # base name.
+            # Future improvement: find all the places where we work with base
+            # names and convert to full paths.  We have directory structure
+            # to disambiguate these, so we shouldn't need this constraint.
+            if name in configuration.all_tests:
+                raise Exception("Found multiple tests with the name %s" % name)
+            configuration.all_tests.add(name)
+
+            # Run the relevant tests in the python file.
+            visit_file(dir, name)
+        except Exception as ex:
+            # Convert this exception to a test event error for the file.
+            test_filename = os.path.abspath(os.path.join(dir, name))
+            if configuration.results_formatter_object is not None:
+                # Grab the backtrace for the exception.
+                import traceback
+                backtrace = traceback.format_exc()
+
+                # Generate the test event.
+                configuration.results_formatter_object.handle_event(
+                    EventBuilder.event_for_job_test_add_error(
+                        test_filename, ex, backtrace))
+            raise
+
+
 def disabledynamics():
     import lldb
     ci = lldb.DBG.GetCommandInterpreter()
     res = lldb.SBCommandReturnObject()
     ci.HandleCommand("setting set target.prefer-dynamic-value no-dynamic-values", res, False)    
     if not res.Succeeded():
         raise Exception('disabling dynamic type support failed')
 

lldb/trunk/packages/Python/lldbsuite/test/issue_verification/TestInvalidDecorator.py.park

+from __future__ import print_function
+from lldbsuite.test import lldbtest
+from lldbsuite.test import decorators
+
+
+class NonExistentDecoratorTestCase(lldbtest.TestBase):
+
+    mydir = lldbtest.TestBase.compute_mydir(__file__)
+
+    @decorators.nonExistentDecorator(bugnumber="yt/1300")
+    def test(self):
+        """Verify non-existent decorators are picked up by test runner."""
+        pass

lldb/trunk/packages/Python/lldbsuite/test_event/event_builder.py

def event_for_cleanup_error(test, error_tuple):
         @return the event dictionary
         """
         event = EventBuilder._event_dictionary_issue(
             test, EventBuilder.STATUS_ERROR, error_tuple)
         event["issue_phase"] = "cleanup"
         return event
 
     @staticmethod
+    def event_for_job_test_add_error(test_filename, exception, backtrace):
+        event = EventBuilder.bare_event(EventBuilder.TYPE_JOB_RESULT)
+        event["status"] = EventBuilder.STATUS_ERROR
+        if test_filename is not None:
+            event["test_filename"] = EventBuilder._assert_is_python_sourcefile(test_filename)
+        if exception is not None and "__class__" in dir(exception):
+            event["issue_class"] = exception.__class__
+        event["issue_message"] = exception
+        if backtrace is not None:
+            event["issue_backtrace"] = backtrace
+        return event
+
+    @staticmethod
     def event_for_job_exceptional_exit(
             pid, worker_index, exception_code, exception_description,
             test_filename, command_line):
         """Creates an event for a job (i.e. process) exit due to signal.
 
         @param pid the process id for the job that failed
         @param worker_index optional id for the job queue running the process
         @param exception_code optional code

lldb/trunk/packages/Python/lldbsuite/test_event/formatter/__init__.py

def create_socket(port):
             raise Exception("listening socket did not respond with ack byte: response={}".format(read_bytes))
 
         return sock, lambda: socket_closer(sock)
 
     default_formatter_name = None
     results_file_object = None
     cleanup_func = None
 
+    file_is_stream = False
     if config.filename:
         # Open the results file for writing.
         if config.filename == 'stdout':
             results_file_object = sys.stdout
             cleanup_func = None
         elif config.filename == 'stderr':
             results_file_object = sys.stderr
             cleanup_func = None
         else:
             results_file_object = open(config.filename, "w")
             cleanup_func = results_file_object.close
         default_formatter_name = (
             "lldbsuite.test_event.formatter.xunit.XunitFormatter")
     elif config.port:
         # Connect to the specified localhost port.
         results_file_object, cleanup_func = create_socket(config.port)
         default_formatter_name = (
             "lldbsuite.test_event.formatter.pickled.RawPickledFormatter")
+        file_is_stream = True
 
     # If we have a results formatter name specified and we didn't specify
     # a results file, we should use stdout.
     if config.formatter_name is not None and results_file_object is None:
         # Use stdout.
         results_file_object = sys.stdout
         cleanup_func = None
 
if results_file_object:
             command_line_options = config.formatter_options
         else:
             command_line_options = []
 
         formatter_options = formatter_arg_parser.parse_args(
             command_line_options)
 
         # Create the TestResultsFormatter given the processed options.
-        results_formatter_object = cls(results_file_object, formatter_options)
+        results_formatter_object = cls(
+            results_file_object,
+            formatter_options,
+            file_is_stream)
 
         def shutdown_formatter():
             """Shuts down the formatter when it is no longer needed."""
             # Tell the formatter to write out anything it may have
             # been saving until the very end (e.g. xUnit results
             # can't complete its output until this point).
             results_formatter_object.send_terminate_as_needed()
 

lldb/trunk/packages/Python/lldbsuite/test_event/formatter/curses.py

 
 from . import results_formatter
 from ..event_builder import EventBuilder
 
 
 class Curses(results_formatter.ResultsFormatter):
     """Receives live results from tests that are running and reports them to the terminal in a curses GUI"""
 
-    def __init__(self, out_file, options):
+    def __init__(self, out_file, options, file_is_stream):
         # Initialize the parent
-        super(Curses, self).__init__(out_file, options)
+        super(Curses, self).__init__(out_file, options, file_is_stream)
         self.using_terminal = True
         self.have_curses = True
         self.initialize_event = None
         self.jobs = [None] * 64
         self.job_tests = [None] * 64
         self.results = list()
         try:
             self.main_window = lldbcurses.intialize_curses()

lldb/trunk/packages/Python/lldbsuite/test_event/formatter/pickled.py

class RawPickledFormatter(ResultsFormatter):
     """
 
     @classmethod
     def arg_parser(cls):
         """@return arg parser used to parse formatter-specific options."""
         parser = super(RawPickledFormatter, cls).arg_parser()
         return parser
 
-    def __init__(self, out_file, options):
-        super(RawPickledFormatter, self).__init__(out_file, options)
+    class StreamSerializer(object):
+        @staticmethod
+        def serialize(test_event, out_file):
+            # Send it as {serialized_length_of_serialized_bytes}{serialized_bytes}
+            import struct
+            msg = cPickle.dumps(test_event)
+            packet = struct.pack("!I%ds" % len(msg), len(msg), msg)
+            out_file.send(packet)
+
+    class BlockSerializer(object):
+        @staticmethod
+        def serialize(test_event, out_file):
+            cPickle.dump(test_event, out_file)
+
+    def __init__(self, out_file, options, file_is_stream):
+        super(RawPickledFormatter, self).__init__(out_file, options, file_is_stream)
         self.pid = os.getpid()
+        if file_is_stream:
+            self.serializer = self.StreamSerializer()
+        else:
+            self.serializer = self.BlockSerializer()
 
     def handle_event(self, test_event):
         super(RawPickledFormatter, self).handle_event(test_event)
 
         # Convert initialize/terminate events into job_begin/job_end events.
         event_type = test_event["event"]
         if event_type is None:
             return
 
         if event_type == "initialize":
             test_event["event"] = "job_begin"
         elif event_type == "terminate":
             test_event["event"] = "job_end"
 
         # Tack on the pid.
         test_event["pid"] = self.pid
 
-        # Send it as {serialized_length_of_serialized_bytes}{serialized_bytes}
-        import struct
-        msg = cPickle.dumps(test_event)
-        packet = struct.pack("!I%ds" % len(msg), len(msg), msg)
-        self.out_file.send(packet)
+        # Serialize the test event.
+        self.serializer.serialize(test_event, self.out_file)

lldb/trunk/packages/Python/lldbsuite/test_event/formatter/results_formatter.py

def arg_parser(cls):
                    '"--option1 value1 [--option2 value2 [...]]"'))
         parser.add_argument(
             "--dump-results",
             action="store_true",
             help=('dump the raw results data after printing '
                   'the summary output.'))
         return parser
 
-    def __init__(self, out_file, options):
+    def __init__(self, out_file, options, file_is_stream):
         super(ResultsFormatter, self).__init__()
         self.out_file = out_file
         self.options = options
         self.using_terminal = False
         if not self.out_file:
             raise Exception("ResultsFormatter created with no file object")
         self.start_time_by_test = {}
         self.terminate_called = False
+        self.file_is_stream = file_is_stream
 
         # Store counts of test_result events by status.
         self.result_status_counts = {
             EventBuilder.STATUS_SUCCESS: 0,
             EventBuilder.STATUS_EXPECTED_FAILURE: 0,
             EventBuilder.STATUS_EXPECTED_TIMEOUT: 0,
             EventBuilder.STATUS_SKIP: 0,
             EventBuilder.STATUS_UNEXPECTED_SUCCESS: 0,

lldb/trunk/packages/Python/lldbsuite/test_event/formatter/xunit.py

def _build_regex_list_from_patterns(patterns):
         patterns provided.
         """
         regex_list = []
         if patterns is not None:
             for pattern in patterns:
                 regex_list.append(re.compile(pattern))
         return regex_list
 
-    def __init__(self, out_file, options):
+    def __init__(self, out_file, options, file_is_stream):
         """Initializes the XunitFormatter instance.
         @param out_file file-like object where formatted output is written.
         @param options specifies a dictionary of options for the
         formatter.
         """
         # Initialize the parent
-        super(XunitFormatter, self).__init__(out_file, options)
+        super(XunitFormatter, self).__init__(out_file, options, file_is_stream)
         self.text_encoding = "UTF-8"
         self.invalid_xml_re = XunitFormatter._build_illegal_xml_regex()
         self.total_test_count = 0
         self.ignore_skip_name_regexes = (
             XunitFormatter._build_regex_list_from_patterns(
                 options.ignore_skip_name_patterns))
         self.ignore_skip_reason_regexes = (
             XunitFormatter._build_regex_list_from_patterns(

lldb/trunk/packages/Python/lldbsuite/test_event/test/resources/invalid_decorator/TestInvalidDecorator.py

+from __future__ import print_function
+from lldbsuite.test import lldbtest
+from lldbsuite.test import decorators
+
+
+class NonExistentDecoratorTestCase(lldbtest.TestBase):
+
+    mydir = lldbtest.TestBase.compute_mydir(__file__)
+
+    @decorators.nonExistentDecorator(bugnumber="yt/1300")
+    def test(self):
+        """Verify non-existent decorators are picked up by test runner."""
+        pass

lldb/trunk/packages/Python/lldbsuite/test_event/test/src/TestCatchInvalidDecorator.py

+#!/usr/bin/env python
+"""
+Tests that the event system reports issues during decorator
+handling as errors.
+"""
+# System-provided imports
+import os
+import unittest
+
+# Local-provided imports
+import event_collector
+
+
+class TestCatchInvalidDecorator(unittest.TestCase):
+
+    TEST_DIR = os.path.join(
+        os.path.dirname(__file__),
+        os.path.pardir,
+        "resources",
+        "invalid_decorator")
+
+    def test_with_whole_file(self):
+        """
+        Test that a non-existent decorator generates a test-event error
+        when running all tests in the file.
+        """
+        # Determine the test case file we're using.
+        test_file = os.path.join(self.TEST_DIR, "TestInvalidDecorator.py")
+
+        # Collect all test events generated for this file.
+        error_results = _filter_error_results(
+            event_collector.collect_events_whole_file(test_file))
+
+        self.assertGreater(
+            len(error_results),
+            0,
+            "At least one job or test error result should have been returned")
+
+    def test_with_function_filter(self):
+        """
+        Test that a non-existent decorator generates a test-event error
+        when running a filtered test.
+        """
+        # Collect all test events generated during running of tests
+        # in a given directory using a test name filter.  Internally,
+        # this runs through a different code path that needs to be
+        # set up to catch exceptions.
+        error_results = _filter_error_results(
+            event_collector.collect_events_for_directory_with_filter(
+                self.TEST_DIR,
+                "NonExistentDecoratorTestCase.test"))
+
+        self.assertGreater(
+            len(error_results),
+            0,
+            "At least one job or test error result should have been returned")
+
+
+def _filter_error_results(events):
+    # Filter out job result events.
+    return [
+        event
+        for event in events
+        if event.get("event", None) in ["job_result", "test_result"] and
+        event.get("status", None) == "error"
+        ]
+
+
+if __name__ == "__main__":
+    unittest.main()

lldb/trunk/packages/Python/lldbsuite/test_event/test/src/event_collector.py

+from __future__ import absolute_import
+from __future__ import print_function
+
+import os
+import subprocess
+import sys
+import tempfile
+
+# noinspection PyUnresolvedReferences
+from six.moves import cPickle
+
+
+def path_to_dotest_py():
+    return os.path.join(
+        os.path.dirname(__file__),
+        os.path.pardir,
+        os.path.pardir,
+        os.path.pardir,
+        os.path.pardir,
+        os.path.pardir,
+        os.path.pardir,
+        "test",
+        "dotest.py")
+
+
+def _make_pickled_events_filename():
+    with tempfile.NamedTemporaryFile(
+            prefix="lldb_test_event_pickled_event_output",
+            delete=False) as temp_file:
+        return temp_file.name
+
+
+def _collect_events_with_command(command, events_filename):
+    # Run the single test with dotest.py, outputting
+    # the raw pickled events to a temp file.
+    with open(os.devnull, 'w') as dev_null_file:
+        subprocess.call(
+            command,
+            stdout=dev_null_file,
+            stderr=dev_null_file)
+
+    # Unpickle the events
+    events = []
+    if os.path.exists(events_filename):
+        with open(events_filename, "rb") as events_file:
+            while True:
+                try:
+                    # print("reading event")
+                    event = cPickle.load(events_file)
+                    # print("read event: {}".format(event))
+                    if event:
+                        events.append(event)
+                except EOFError:
+                    # This is okay.
+                    break
+        os.remove(events_filename)
+    return events
+
+
+def collect_events_whole_file(test_filename):
+    events_filename = _make_pickled_events_filename()
+    command = [
+        sys.executable,
+        path_to_dotest_py(),
+        "--inferior",
+        "--results-formatter=lldbsuite.test_event.formatter.pickled.RawPickledFormatter",
+        "--results-file={}".format(events_filename),
+        "-p", os.path.basename(test_filename),
+        os.path.dirname(test_filename)
+        ]
+    return _collect_events_with_command(command, events_filename)
+
+
+def collect_events_for_directory_with_filter(test_filename, filter_desc):
+    events_filename = _make_pickled_events_filename()
+    command = [
+        sys.executable,
+        path_to_dotest_py(),
+        "--inferior",
+        "--results-formatter=lldbsuite.test_event.formatter.pickled.RawPickledFormatter",
+        "--results-file={}".format(events_filename),
+        "-f", filter_desc,
+        os.path.dirname(test_filename)
+        ]
+    return _collect_events_with_command(command, events_filename)