[Dexter] Allow tests to specify their command line options

[jmorse]

Summary

This patch adds a "DexCommandLine" command, allowing dexter tests to specify what command line options the test should be started with. I've also plumbed it through into the debuggers.

[Orlando]

Looks good, LGTM with two questions inline.

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DebuggerControllerBase.py

33

Probably a better user experience if this is caught in the parsing stage and prettily reported as an error. wdyt?

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/command_line.c

9–12

Is this part of the test (and the #include) necessary?

[jmorse]

Produce a parsing error if DexCommandLine is specified more than once; refresh a test.

[jmorse]

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DebuggerControllerBase.py

33

I've added a filter to ParseCommand.py, which I think is the correct place for validating these things?

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/command_line.c

9–12

Yup, that was due to some testing, now removed.

[Orlando]

(Still LGTM)

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DebuggerControllerBase.py

33

Thanks, and yeah that's the right place.

cross-project-tests/debuginfo-tests/dexter/Commands.md

 # Dexter commands
 
 * [DexExpectProgramState](Commands.md#DexExpectProgramState)
 * [DexExpectStepKind](Commands.md#DexExpectStepKind)
 * [DexExpectStepOrder](Commands.md#DexExpectStepOrder)
 * [DexExpectWatchType](Commands.md#DexExpectWatchType)
 * [DexExpectWatchValue](Commands.md#DexExpectWatchValue)
 * [DexUnreachable](Commands.md#DexUnreachable)
 * [DexLimitSteps](Commands.md#DexLimitSteps)
 * [DexLabel](Commands.md#DexLabel)
 * [DexWatch](Commands.md#DexWatch)
 * [DexDeclareFile](Commands.md#DexDeclareFile)
 * [DexFinishTest](Commands.md#DexFinishTest)
+* [DexCommandLine](Commands.md#DexCommandLine)
 
 ---
 ## DexExpectProgramState
     DexExpectProgramState(state [,**times])
 
     Args:
         state (dict): { 'frames': [
                         {

 The command will trigger when the line 'on_line' is stepped on and either the
 condition '(expr) == (values[n])' is true or there are no conditions. If the
 optional argument 'hit_count' is provided, then the command will not trigger
 for the first 'hit_count' times the line and condition are hit.
 
 ### Heuristic
 This command does not contribute to the heuristic score.
 
+----
+## DexCommandLine
+    DexCommandLine(command_line)
+
+    Args:
+        command_line (list): List of strings that form the command line.
+
+### Description
+Specifies the command line with which to launch the test. The arguments will
+be appended to the default command line, i.e. the path to the compiled binary,
+and will be passed to the program under test.
+
+This command does not contribute to any part of the debug experience testing or
+runtime instrumentation -- it's only for communicating arguments to the program
+under test.
+
+### Heuristic
+This command does not contribute to the heuristic score.
+
 ---
 ## DexWatch
     DexWatch(*expressions)
 
     Arg list:
         expressions (str): `expression` to evaluate on this line.
 
 ### Description
 [Deprecated] Evaluate each given `expression` when the debugger steps onto the
 line this command is found on.
 
 ### Heuristic
 [Deprecated]

cross-project-tests/debuginfo-tests/dexter/dex/command/ParseCommand.py

 import unittest
 from copy import copy
 from pathlib import PurePath
 from collections import defaultdict, OrderedDict
 
 from dex.utils.Exceptions import CommandParseError
 
 from dex.command.CommandBase import CommandBase
+from dex.command.commands.DexCommandLine import DexCommandLine
 from dex.command.commands.DexDeclareFile import DexDeclareFile
 from dex.command.commands.DexExpectProgramState import DexExpectProgramState
 from dex.command.commands.DexExpectStepKind import DexExpectStepKind
 from dex.command.commands.DexExpectStepOrder import DexExpectStepOrder
 from dex.command.commands.DexExpectWatchType import DexExpectWatchType
 from dex.command.commands.DexExpectWatchValue import DexExpectWatchValue
 from dex.command.commands.DexLabel import DexLabel
 from dex.command.commands.DexLimitSteps import DexLimitSteps
 from dex.command.commands.DexFinishTest import DexFinishTest
 from dex.command.commands.DexUnreachable import DexUnreachable
 from dex.command.commands.DexWatch import DexWatch
 from dex.utils import Timer
 from dex.utils.Exceptions import CommandParseError, DebuggerException
 
 def _get_valid_commands():
     """Return all top level DExTer test commands.
 
     Returns:
         { name (str): command (class) }
     """
     return {
+      DexCommandLine.get_name() : DexCommandLine,
       DexDeclareFile.get_name() : DexDeclareFile,
       DexExpectProgramState.get_name() : DexExpectProgramState,
       DexExpectStepKind.get_name() : DexExpectStepKind,
       DexExpectStepOrder.get_name() : DexExpectStepOrder,
       DexExpectWatchType.get_name() : DexExpectWatchType,
       DexExpectWatchValue.get_name() : DexExpectWatchValue,
       DexLabel.get_name() : DexLabel,
       DexLimitSteps.get_name() : DexLimitSteps,

cross-project-tests/debuginfo-tests/dexter/dex/command/commands/DexCommandLine.py

+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""A Command that specifies the command line with which to run the test.
+"""
+
+from dex.command.CommandBase import CommandBase
+
+class DexCommandLine(CommandBase):
+    def __init__(self, the_cmdline):
+        if type(the_cmdline) is not list:
+            raise TypeError('Expected list, got {}'.format(type(the_cmdline)))
+        for x in the_cmdline:
+          if type(x) is not str:
+              raise TypeError('Command line element "{}" has type {}'.format(x, type(x)))
+        self.the_cmdline = the_cmdline
+        super(DexCommandLine, self).__init__()
+
+    def eval(self):
+        raise NotImplementedError('DexCommandLine commands cannot be evaled.')
+
+    @staticmethod
+    def get_name():
+        return __class__.__name__
+
+    @staticmethod
+    def get_subcommands() -> dict:
+        return None

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/ConditionalController.py

class BreakpointRange:
     def should_be_removed(self):
         if self.max_hit_count == None:
             return False
         return self.current_hit_count >= self.max_hit_count
 
 
 class ConditionalController(DebuggerControllerBase):
     def __init__(self, context, step_collection):
-      self.context = context
-      self.step_collection = step_collection
       self._bp_ranges = None
-      self._build_bp_ranges()
       self._watches = set()
       self._step_index = 0
       self._pause_between_steps = context.options.pause_between_steps
       self._max_steps = context.options.max_steps
       # Map {id: BreakpointRange}
       self._leading_bp_handles = {}
+      super(ConditionalController, self).__init__(context, step_collection)
+      self._build_bp_ranges()
 
     def _build_bp_ranges(self):
         commands = self.step_collection.commands
         self._bp_ranges = []
         try:
             limit_commands = commands['DexLimitSteps']
             for lc in limit_commands:
                 bpr = BreakpointRange(
def _set_leading_bps(self):
                                                                   bpr.range_from,
                                                                   cond_expr)
                     self._leading_bp_handles[id] = bpr
             else:
                 # Add an unconditional breakpoint.
                 id = self.debugger.add_breakpoint(bpr.path, bpr.range_from)
                 self._leading_bp_handles[id] = bpr
 
-    def _run_debugger_custom(self):
+    def _run_debugger_custom(self, cmdline):
         # TODO: Add conditional and unconditional breakpoint support to dbgeng.
         if self.debugger.get_name() == 'dbgeng':
             raise DebuggerException('DexLimitSteps commands are not supported by dbgeng')
 
         self.step_collection.clear_steps()
         self._set_leading_bps()
 
         for command_obj in chain.from_iterable(self.step_collection.commands.values()):
             self._watches.update(command_obj.get_watches())
 
-        self.debugger.launch()
+        self.debugger.launch(cmdline)
         time.sleep(self._pause_between_steps)
 
         exit_desired = False
 
         while not self.debugger.is_finished:
             while self.debugger.is_running:
                 pass
 

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DebuggerControllerBase.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Abstract Base class for controlling debuggers."""
 
 import abc
 
 class DebuggerControllerBase(object, metaclass=abc.ABCMeta):
+    def __init__(self, context, step_collection):
+        self.context = context
+        self.step_collection = step_collection
+
     @abc.abstractclassmethod
     def _run_debugger_custom(self):
         """Specify your own implementation of run_debugger_custom in your own
         controller.
         """
         pass
 
     def run_debugger(self, debugger):
         """Responsible for correctly launching and tearing down the debugger.
         """
         self.debugger = debugger
+
+        # Fetch command line options, if any.
+        the_cmdline = []
+        commands = self.step_collection.commands
+        if 'DexCommandLine' in commands:
+            cmd_line_objs = commands['DexCommandLine']
+            assert len(cmd_line_objs) == 1
+            cmd_line_obj = cmd_line_objs[0]
+            the_cmdline = cmd_line_obj.the_cmdline
+
         with self.debugger:
             if not self.debugger.loading_error:
-                self._run_debugger_custom()
+                self._run_debugger_custom(the_cmdline)
 
         # We may need to pickle this debugger controller after running the
         # debugger. Debuggers are not picklable objects, so set to None.
         self.debugger = None

cross-project-tests/debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DefaultController.py

class EarlyExitCondition(object):
     def __init__(self, on_line, hit_count, expression, values):
         self.on_line = on_line
         self.hit_count = hit_count
         self.expression = expression
         self.values = values
 
 class DefaultController(DebuggerControllerBase):
     def __init__(self, context, step_collection):
-        self.context = context
-        self.step_collection = step_collection
-        self.source_files = self.context.options.source_files
+        self.source_files = context.options.source_files
         self.watches = set()
         self.step_index = 0
+        super(DefaultController, self).__init__(context, step_collection)
 
     def _break_point_all_lines(self):
         for s in self.context.options.source_files:
             with open(s, 'r') as fp:
                 num_lines = len(fp.readlines())
             for line in range(1, num_lines + 1):
                 try:
                    self.debugger.add_breakpoint(s, line)
def _should_exit(self, early_exit_conditions, line_no):
                 if exit_condition_hit:
                     if condition.hit_count <= 0:
                         return True
                     else:
                         condition.hit_count -= 1
         return False
 
 
-    def _run_debugger_custom(self):
+    def _run_debugger_custom(self, cmdline):
         self.step_collection.debugger = self.debugger.debugger_info
         self._break_point_all_lines()
-        self.debugger.launch()
+        self.debugger.launch(cmdline)
 
         for command_obj in chain.from_iterable(self.step_collection.commands.values()):
             self.watches.update(command_obj.get_watches())
         early_exit_conditions = self._get_early_exit_conditions()
 
         max_steps = self.context.options.max_steps
         for _ in range(max_steps):
             while self.debugger.is_running:

cross-project-tests/debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py

class DbgEng(DebuggerBase):
     def get_triggered_breakpoint_ids(self):
       raise NotImplementedError('get_triggered_breakpoint_ids is not yet implemented by dbgeng')
 
     def delete_breakpoint(self, id):
         # breakpoint setting/deleting is not supported by dbgeng at this moment
         # but is something that should be considered in the future.
         raise NotImplementedError('delete_conditional_breakpoint is not yet implemented by dbgeng')
 
-    def launch(self):
+    def launch(self, cmdline):
+        assert len(cmdline) == 0, "Command lines unimplemented for dbgeng right now"
         # We are, by this point, already launched.
         self.step_info = probe_process.probe_state(self.client)
 
     def step(self):
         res = setup.step_once(self.client)
         if not res:
           self.finished = True
         self.step_info = res

cross-project-tests/debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py

def delete_breakpoint(self, id):
             raise KeyError
         try:
             del self._breakpoint_conditions[id]
         except KeyError:
             # This must be an unconditional breakpoint.
             pass
         self._target.BreakpointDelete(id)
 
-    def launch(self):
-        self._process = self._target.LaunchSimple(None, None, os.getcwd())
+    def launch(self, cmdline):
+        self._process = self._target.LaunchSimple(cmdline, None, os.getcwd())
         if not self._process or self._process.GetNumThreads() == 0:
             raise DebuggerException('could not launch process')
         if self._process.GetNumThreads() != 1:
             raise DebuggerException('multiple threads not supported')
         self._thread = self._process.GetThreadAtIndex(0)
         assert self._thread, (self._process, self._thread)
 
     def step(self):

cross-project-tests/debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py

def delete_breakpoint(self, id):
             # Parent.
             assert bp.Parent == None
             this_vsbp = VSBreakpoint(PurePath(bp.File), bp.FileLine,
                                      bp.FileColumn, bp.Condition)
             if vsbp == this_vsbp:
                 bp.Delete()
                 break
 
-    def launch(self):
+    def _fetch_property(self, props, name):
+        num_props = props.Count
+        result = None
+        for x in range(1, num_props+1):
+            item = props.Item(x)
+            if item.Name == name:
+                return item
+        assert False, "Couldn't find property {}".format(name)
+
+    def launch(self, cmdline):
+        cmdline_str = ' '.join(cmdline)
+
+        # In a slightly baroque manner, lookup the VS project that runs when
+        # you click "run", and set its command line options to the desired
+        # command line options.
+        startup_proj_name = str(self._fetch_property(self._interface.Solution.Properties, 'StartupProject'))
+        project = self._fetch_property(self._interface.Solution, startup_proj_name)
+        ActiveConfiguration = self._fetch_property(project.Properties, 'ActiveConfiguration').Object
+        ActiveConfiguration.DebugSettings.CommandArguments = cmdline_str
+
         self._fn_go()
 
     def step(self):
         self._fn_step()
 
     def go(self) -> ReturnCode:
         self._fn_go()
         return ReturnCode.OK

cross-project-tests/debuginfo-tests/dexter/feature_tests/commands/perfect/command_line.c

+// UNSUPPORTED: dbgeng
+//
+// RUN: %dexter_regression_test -- %s | FileCheck %s
+// CHECK: command_line.c:
+
+#include <stdio.h>
+
+int main(int argc, const char **argv) {
+  FILE *foo = fopen("/tmp/bees", "w");
+  fprintf(foo, "%d\n", argc);
+  fprintf(foo, "%s\n", argv[1]);
+  fclose(foo);
+  if (argc == 4)
+    return 0; // DexLabel('retline')
+
+  return 1; // DexUnreachable()
+}
+
+// DexExpectWatchValue('argc', '4', on_line=ref('retline'))
+
+// Three args will be appended to the 'default' argument.
+// DexCommandLine(['a', 'b', 'c'])