[Dexter] Add DexFinishTest command to conditionally early-exit a test program

[StephenTozer]

Summary

Some programs to be tested by Dexter have inconvenient termination conditions, either terminating long after all test commands have been exhausted or never self-terminating at all. This can be fixed by editing the test source to add a new exit condition(s), but doing so may affect debug info, and the "correct" place to exit may be different across different tests for the same program.

This patch adds a command, DexFinishTest, that allows a Dexter test to be conditionally finished at a given breakpoint. This command has the same set of arguments as DexLimitSteps, except that it does not allow a line range (from_line, to_line), only a single line (on_line). When it is triggered the debugger is closed and the test is finished - this does not affect the scoring in Dexter, except in that no further step information will be gathered.

[Orlando]

LGTM, nice to see so many tests

cross-project-tests/debuginfo-tests/dexter/Commands.md

 # Dexter commands
 
 * [DexExpectProgramState](Commands.md#DexExpectProgramState)
 * [DexExpectStepKind](Commands.md#DexExpectStepKind)
 * [DexExpectStepOrder](Commands.md#DexExpectStepOrder)
 * [DexExpectWatchType](Commands.md#DexExpectWatchType)
 * [DexExpectWatchValue](Commands.md#DexExpectWatchValue)
 * [DexUnreachable](Commands.md#DexUnreachable)
 * [DexLimitSteps](Commands.md#DexLimitSteps)
 * [DexLabel](Commands.md#DexLabel)
 * [DexWatch](Commands.md#DexWatch)
 * [DexDeclareFile](Commands.md#DexDeclareFile)
+* [DexFinishTest](Commands.md#DexFinishTest)
 
 ---
 ## DexExpectProgramState
     DexExpectProgramState(state [,**times])
 
     Args:
         state (dict): { 'frames': [
                         {

 The new path holds until the end of the test file or until a new DexDeclareFile
 command is encountered. Used in conjunction with .dex files, DexDeclareFile can
 be used to write your dexter commands in a separate test file avoiding inlined
 Dexter commands mixed with test source.
 
 ### Heuristic
 This command does not contribute to the heuristic score.
 
+----
+## DexFinishTest
+    DexFinishTest([expr, *values], **on_line[, **hit_count=0])
+
+    Args:
+        expr (str): variable or value to compare.
+
+    Arg list:
+        values (str): At least one potential value the expr may evaluate to.
+
+    Keyword args:
+        on_line (int): Define the line on which this command will be triggered.
+        hit_count (int): If provided, triggers this command only after the line
+                         and condition have been hit the given number of times.
+
+### Description
+Defines a point at which Dexter will exit out of the debugger without waiting
+for the program to finish. This is primarily useful for testing a program that
+either does not automatically terminate or would otherwise continue for a long
+time after all test commands have finished.
+
+The command will trigger when the line 'on_line' is stepped on and either the
+condition '(expr) == (values[n])' is true or there are no conditions. If the
+optional argument 'hit_count' is provided, then the command will not trigger
+for the first 'hit_count' times the line and condition are hit.
+
+### Heuristic
+This command does not contribute to the heuristic score.
+
 ---
 ## DexWatch
     DexWatch(*expressions)
 
     Arg list:
         expressions (str): `expression` to evaluate on this line.
 
 ### Description
 [Deprecated] Evaluate each given `expression` when the debugger steps onto the
 line this command is found on.
 
 ### Heuristic
 [Deprecated]

cross-project-tests/debuginfo-tests/dexter/dex/command/ParseCommand.py

 from dex.command.commands.DexDeclareFile import DexDeclareFile
 from dex.command.commands.DexExpectProgramState import DexExpectProgramState
 from dex.command.commands.DexExpectStepKind import DexExpectStepKind
 from dex.command.commands.DexExpectStepOrder import DexExpectStepOrder
 from dex.command.commands.DexExpectWatchType import DexExpectWatchType
 from dex.command.commands.DexExpectWatchValue import DexExpectWatchValue
 from dex.command.commands.DexLabel import DexLabel
 from dex.command.commands.DexLimitSteps import DexLimitSteps
+from dex.command.commands.DexFinishTest import DexFinishTest
 from dex.command.commands.DexUnreachable import DexUnreachable
 from dex.command.commands.DexWatch import DexWatch
 from dex.utils import Timer
 from dex.utils.Exceptions import CommandParseError, DebuggerException
 
 def _get_valid_commands():
     """Return all top level DExTer test commands.
 
     Returns:
         { name (str): command (class) }
     """
     return {
       DexDeclareFile.get_name() : DexDeclareFile,
       DexExpectProgramState.get_name() : DexExpectProgramState,
       DexExpectStepKind.get_name() : DexExpectStepKind,
       DexExpectStepOrder.get_name() : DexExpectStepOrder,
       DexExpectWatchType.get_name() : DexExpectWatchType,
       DexExpectWatchValue.get_name() : DexExpectWatchValue,
       DexLabel.get_name() : DexLabel,
       DexLimitSteps.get_name() : DexLimitSteps,
+      DexFinishTest.get_name() : DexFinishTest,
       DexUnreachable.get_name() : DexUnreachable,
       DexWatch.get_name() : DexWatch
     }
 
 
 def _get_command_name(command_raw: str) -> str:
     """Return command name by splitting up DExTer command contained in
     command_raw on the first opening paranthesis and further stripping

cross-project-tests/debuginfo-tests/dexter/dex/command/commands/DexFinishTest.py

+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""A Command that enables test writers to terminate a test after a specified
+breakpoint has been hit a number of times.
+"""
+
+from dex.command.CommandBase import CommandBase
+
+class DexFinishTest(CommandBase):
+    def __init__(self, *args, **kwargs):
+        if len(args) == 0:
+            self.expression = None
+            self.values = []
+        elif len(args) == 1:
+            raise TypeError("expected 0 or at least 2 positional arguments")
+        else:
+            self.expression = args[0]
+            self.values = [str(arg) for arg in args[1:]]
+        self.on_line = kwargs.pop('on_line')
+        self.hit_count = kwargs.pop('hit_count', 0)
+        if kwargs:
+            raise TypeError('unexpected named args: {}'.format(
+                ', '.join(kwargs)))
+        super(DexFinishTest, self).__init__()
+
+    def eval(self):
+        raise NotImplementedError('DexFinishTest commands cannot be evaled.')
+
+    @staticmethod
+    def get_name():
+        return __class__.__name__
+
+    @staticmethod
+    def get_subcommands() -> dict:
+        return None

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/ConditionalController.py

Args:
        expression: None for no conditions, or a str expression to compare
        against `values`.
 
        hit_count: None for no limit, or int to set the number of times the
                   leading breakpoint is triggered before it is removed.
     """
 
     def __init__(self, expression: str, path: str, range_from: int, range_to: int,
-                 values: list, hit_count: int):
+                 values: list, hit_count: int, finish_on_remove: bool):
         self.expression = expression
         self.path = path
         self.range_from = range_from
         self.range_to = range_to
         self.conditional_values = values
         self.max_hit_count = hit_count
         self.current_hit_count = 0
+        self.finish_on_remove = finish_on_remove
 
     def has_conditions(self):
         return self.expression != None
 
     def get_conditional_expression_list(self):
         conditional_list = []
         for value in self.conditional_values:
             # (<expression>) == (<value>)
def _build_bp_ranges(self):
             limit_commands = commands['DexLimitSteps']
             for lc in limit_commands:
                 bpr = BreakpointRange(
                   lc.expression,
                   lc.path,
                   lc.from_line,
                   lc.to_line,
                   lc.values,
-                  lc.hit_count)
+                  lc.hit_count,
+                  False)
                 self._bp_ranges.append(bpr)
         except KeyError:
             raise DebuggerException('Missing DexLimitSteps commands, cannot conditionally step.')
+        if 'DexFinishTest' in commands:
+            finish_commands = commands['DexFinishTest']
+            for ic in finish_commands:
+                bpr = BreakpointRange(
+                  ic.expression,
+                  ic.path,
+                  ic.on_line,
+                  ic.on_line,
+                  ic.values,
+                  ic.hit_count + 1,
+                  True)
+                self._bp_ranges.append(bpr)
 
     def _set_leading_bps(self):
         # Set a leading breakpoint for each BreakpointRange, building a
         # map of {leading bp id: BreakpointRange}.
         for bpr in self._bp_ranges:
             if bpr.has_conditions():
                 # Add a conditional breakpoint for each condition.
                 for cond_expr in bpr.get_conditional_expression_list():
def _run_debugger_custom(self):
         self.step_collection.clear_steps()
         self._set_leading_bps()
 
         for command_obj in chain.from_iterable(self.step_collection.commands.values()):
             self._watches.update(command_obj.get_watches())
 
         self.debugger.launch()
         time.sleep(self._pause_between_steps)
+
+        exit_desired = False
+
         while not self.debugger.is_finished:
             while self.debugger.is_running:
                 pass
 
             step_info = self.debugger.get_step_info(self._watches, self._step_index)
             if step_info.current_frame:
                 self._step_index += 1
                 update_step_watches(step_info, self._watches, self.step_collection.commands)
                 self.step_collection.new_step(self.context, step_info)
 
             bp_to_delete = []
             for bp_id in self.debugger.get_triggered_breakpoint_ids():
                 try:
                     # See if this is one of our leading breakpoints.
                     bpr = self._leading_bp_handles[bp_id]
                 except KeyError:
                     # This is a trailing bp. Mark it for removal.
                     bp_to_delete.append(bp_id)
                     continue
 
                 bpr.add_hit()
                 if bpr.should_be_removed():
+                    if bpr.finish_on_remove:
+                        exit_desired = True
                     bp_to_delete.append(bp_id)
                     del self._leading_bp_handles[bp_id]
                 # Add a range of trailing breakpoints covering the lines
                 # requested in the DexLimitSteps command. Ignore first line as
                 # that's covered by the leading bp we just hit and include the
                 # final line.
                 for line in range(bpr.range_from + 1, bpr.range_to + 1):
                     self.debugger.add_breakpoint(bpr.path, line)
 
             # Remove any trailing or expired leading breakpoints we just hit.
             for bp_id in bp_to_delete:
                 self.debugger.delete_breakpoint(bp_id)
 
+            if exit_desired:
+                break
             self.debugger.go()
             time.sleep(self._pause_between_steps)

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DefaultController.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Default class for controlling debuggers."""
 
 from itertools import chain
 import os
 import time
 
 from dex.debugger.DebuggerControllers.DebuggerControllerBase import DebuggerControllerBase
 from dex.debugger.DebuggerControllers.ControllerHelpers import in_source_file, update_step_watches
 from dex.utils.Exceptions import DebuggerException, LoadDebuggerException
 
+class EarlyExitCondition(object):
+    def __init__(self, on_line, hit_count, expression, values):
+        self.on_line = on_line
+        self.hit_count = hit_count
+        self.expression = expression
+        self.values = values
+
 class DefaultController(DebuggerControllerBase):
     def __init__(self, context, step_collection):
         self.context = context
         self.step_collection = step_collection
         self.source_files = self.context.options.source_files
         self.watches = set()
         self.step_index = 0
 
     def _break_point_all_lines(self):
         for s in self.context.options.source_files:
             with open(s, 'r') as fp:
                 num_lines = len(fp.readlines())
             for line in range(1, num_lines + 1):
                 try:
                    self.debugger.add_breakpoint(s, line)
                 except DebuggerException:
                    raise LoadDebuggerException(DebuggerException.msg)
 
+    def _get_early_exit_conditions(self):
+        commands = self.step_collection.commands
+        early_exit_conditions = []
+        if 'DexFinishTest' in commands:
+            finish_commands = commands['DexFinishTest']
+            for fc in finish_commands:
+                condition = EarlyExitCondition(on_line=fc.on_line,
+                                               hit_count=fc.hit_count,
+                                               expression=fc.expression,
+                                               values=fc.values)
+                early_exit_conditions.append(condition)
+        return early_exit_conditions
+
+    def _should_exit(self, early_exit_conditions, line_no):
+        for condition in early_exit_conditions:
+            if condition.on_line == line_no:
+                exit_condition_hit = condition.expression is None
+                if condition.expression is not None:
+                    # For the purposes of consistent behaviour with the
+                    # Conditional Controller, check equality in the debugger
+                    # rather than in python (as the two can differ).
+                    for value in condition.values:
+                        expr_val = self.debugger.evaluate_expression(f'({condition.expression}) == ({value})')
+                        if expr_val.value == 'true':
+                            exit_condition_hit = True
+                            break
+                if exit_condition_hit:
+                    if condition.hit_count <= 0:
+                        return True
+                    else:
+                        condition.hit_count -= 1
+        return False
+
+
     def _run_debugger_custom(self):
         self.step_collection.debugger = self.debugger.debugger_info
         self._break_point_all_lines()
         self.debugger.launch()
 
         for command_obj in chain.from_iterable(self.step_collection.commands.values()):
             self.watches.update(command_obj.get_watches())
+        early_exit_conditions = self._get_early_exit_conditions()
 
         max_steps = self.context.options.max_steps
         for _ in range(max_steps):
             while self.debugger.is_running:
                 pass
 
             if self.debugger.is_finished:
                 break
 
             self.step_index += 1
             step_info = self.debugger.get_step_info(self.watches, self.step_index)
 
             if step_info.current_frame:
                 update_step_watches(step_info, self.watches, self.step_collection.commands)
                 self.step_collection.new_step(self.context, step_info)
+                if self._should_exit(early_exit_conditions, step_info.current_frame.loc.lineno):
+                    break
 
             if in_source_file(self.source_files, step_info):
                 self.debugger.step()
             else:
                 self.debugger.go()
 
             time.sleep(self.context.options.pause_between_steps)
         else:
             raise DebuggerException(
                 'maximum number of steps reached ({})'.format(max_steps))

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/default_conditional.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a condition, so the test exits
+//      when the line referenced by \DexFinishTest is stepped on and the given
+//      condition (x == 5) is satisfied.
+//      Tests using the default controller (no \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: default_conditional.cpp
+
+int main() {
+    for (int x = 0; x < 10; ++x)
+        (void)0; // DexLabel('finish_line')
+}
+
+// DexFinishTest('x', 5, on_line=ref('finish_line'))
+// DexExpectWatchValue('x', 0, 1, 2, 3, 4, 5)

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/default_conditional_hit_count.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a combination of a hit_count
+//      and a condition, so that the test exits after the line referenced
+//      by \DexFinishTest is stepped on while the condition (x == 2) is true a
+//      given number of times.
+//      Tests using the default controller (no \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: default_conditional_hit_count.cpp
+
+int main() {
+    for (int y = 0; y < 4; ++y)
+        for (int x = 0; x < 4; ++x)
+            (void)0; // DexLabel('finish_line')
+}
+
+// DexFinishTest('x', 2, on_line=ref('finish_line'), hit_count=2)
+// DexExpectWatchValue('x', 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, on_line=ref('finish_line'))
+// DexExpectWatchValue('y', 0, 1, 2, on_line=ref('finish_line'))

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/default_hit_count.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a hit_count, so the test exits
+//      after the line referenced by \DexFinishTest has been stepped on a
+//      specific number of times.
+//      Tests using the default controller (no \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: default_hit_count.cpp
+
+int main() {
+    for (int x = 0; x < 10; ++x)
+        (void)0; // DexLabel('finish_line')
+}
+
+// DexFinishTest(on_line=ref('finish_line'), hit_count=5)
+// DexExpectWatchValue('x', 0, 1, 2, 3, 4, 5)

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/default_simple.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used without a condition or hit_count,
+//      so the test simply exits as soon as the line referenced by \DexFinishTest
+//      is stepped on.
+//      Tests using the default controller (no \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: default_simple.cpp
+
+int main() {
+    int x = 0; // DexLabel('start_line')
+    x = 1;
+    x = 2; // DexLabel('finish_line')
+}
+
+// DexFinishTest(on_line=ref('finish_line'))
+// DexExpectWatchValue('x', 0, 1)

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/limit_steps_conditional.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a condition, so the test exits
+//      when the line referenced by \DexFinishTest is stepped on and the given
+//      condition (x == 5) is satisfied.
+//      Test using the conditional controller (using \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: limit_steps_conditional.cpp
+
+int main() {
+    for (int x = 0; x < 10; ++x)
+        (void)0; // DexLabel('finish_line')
+}
+
+// DexLimitSteps(on_line=ref('finish_line'))
+// DexFinishTest('x', 5, on_line=ref('finish_line'))
+// DexExpectWatchValue('x', 0, 1, 2, 3, 4, 5)

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/limit_steps_conditional_hit_count.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a combination of a hit_count
+//      and a condition, so that the test exits after the line referenced
+//      by \DexFinishTest is stepped on while the condition (x == 2) is true a
+//      given number of times.
+//      Test using the conditional controller (using \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: limit_steps_conditional_hit_count.cpp
+
+int main() {
+    for (int y = 0; y < 4; ++y)
+        for (int x = 0; x < 4; ++x)
+            (void)0; // DexLabel('finish_line')
+}
+
+// DexLimitSteps(on_line=ref('finish_line'))
+// DexFinishTest('x', 2, on_line=ref('finish_line'), hit_count=2)
+// DexExpectWatchValue('x', 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, on_line=ref('finish_line'))
+// DexExpectWatchValue('y', 0, 1, 2, on_line=ref('finish_line'))

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/limit_steps_hit_count.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used with a hit_count, so the test exits
+//      after the line referenced by \DexFinishTest has been stepped on a
+//      specific number of times.
+//      Test using the conditional controller (using \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: limit_steps_hit_count.cpp
+
+int main() {
+    for (int x = 0; x < 10; ++x)
+        (void)0; // DexLabel('finish_line')
+}
+
+// DexLimitSteps(on_line=ref('finish_line'))
+// DexFinishTest(on_line=ref('finish_line'), hit_count=5)
+// DexExpectWatchValue('x', 0, 1, 2, 3, 4, 5)

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/dex_finish_test/limit_steps_simple.cpp

+// Purpose:
+//      Test that \DexFinishTest can be used without a condition or hit_count,
+//      so the test simply exits as soon as the line referenced by \DexFinishTest
+//      is stepped on.
+//      Test using the conditional controller (using \DexLimitSteps).
+//
+// REQUIRES: system-linux
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: limit_steps_simple.cpp
+
+int main() {
+    int x = 0; // DexLabel('start')
+    x = 1;
+    x = 2; // DexLabel('finish_line')
+} // DexLabel('finish')
+
+// DexLimitSteps(from_line=ref('start'), to_line=ref('finish'))
+// DexFinishTest(on_line=ref('finish_line'))
+// DexExpectWatchValue('x', 0, 1)