[Dexter] Add Debugger Controller To Dexter

[TWeaver]

Summary

Hello LLVM and Dexter Community,

I present to you the 'Debugger Controller' patch.

Recently there's been interest in running the debugger frame work of Dexter in new and interesting ways. My own work coupled with the work of some other Dexter contributors has highlighted an issue:

The current architecture of Dexter doesn't support new and novel ways of running the debugger. If one wishes to write their own debugger running behaviour, that behaviour has to sit awkwardly with the existing behaviour. But fear not, this patch goes some of the way to fixing this issue.

The original implementation of DebuggerBase was a monolith of 3 things - It's responsible for exposing the features of the debuggers (good), operating the debuggers behaviour to get step information (bad) and responsible for archiving and generating said step behaviour (also bad).

This patch addresses behaviour 2. The operating behaviour of the debugger has been promoted to a first class object, the DebuggerController. If you wish to implement your own debugger operating behaviour simply inherit from DebuggerControllerBase and extend the _run_debugger_custom method.

Also rolled in with this patch is how the debugger controller, context, options and step collections are passed around Dexter. No longer is the get_debugger_steps method in Debuggers.py responsible be passing context, options or step collections to the internal_run_debugger_ tool. It's job now is to pass a fully formed debugger controller object that contains all the information required to run the debugger. Context, Options and step information are all stored in the DefaultController now (which represents the default dexter behaviour before this patch). As a result, get_debugger_steps is now known as run_debugger_subprocess as that is all it's responsible for.

We now pass the controller as a pickled object, unpickle and run it in the internal tool and then repickle to be returned back to run_debugger_subprocess, this is then returned to the caller of run_debugger_subprocess. This could be the test tool, the clang-opt-bisect tool or any other tool that exists within dexter. Which debugger controller is passed is up to us as developers, so we can pick and choose for whatever case suits our needs.

A follow up patch for this change is also incoming that will have a new type of debugger controller, the ConditionalSteppingController. In order to bring this new behaviour to dexter it's paramount that we have an extensible debugger operating architecture, and it's my hope that this patch delivers on this premise.

[Orlando]

Makes sense to separate out these parts. LGTM, but it's a sizeable patch so maybe wait a little and see if anyone else has anything to say before committing.

[tbosch]

Hello, I believe this broke the test "clang-opt-bisect" (ran at ee12edcb76423c78b55cdddae2edfe45cbb2ccd6):

******************** TEST 'debuginfo-tests :: dexter/feature_tests/subtools/clang-opt-bisect/clang-opt-bisect.cpp' FAILED ********************
Script:
--
: 'RUN: at line 6';   true
: 'RUN: at line 7';   "/usr/bin/python" "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dexter.py" clang-opt-bisect --debugger 'lldb' --builder 'clang'      --cflags "-O0 -g" -- /usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/feature_tests/subtools/clang-opt-bisect/clang-opt-bisect.cpp      | /usr/local/google/home/tbosch/llvm-project/build/bin/FileCheck /usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/feature_tests/subtools/clang-opt-bisect/clang-opt-bisect.cpp
--
Exit Code: 2

Command Output (stderr):
--


****************************************
****************************************
****************************************
**                                    **
** This is a bug in DExTer.           **
**                                    **
**                  Please report it. **
**                                    **
****************************************
****************************************
****************************************

system:
linux

version:
DExTer 1.0.0 [master ee12edcb76423c78b55cdddae2edfe45cbb2ccd6] ([e-mail]:tbosch/llvm-project.git)
  using Python 3.7.6 (default, Dec 19 2019, 09:25:23)
[GCC 9.2.1 20191109]

args:
['/usr/bin/python', '/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dexter.py', 'clang-opt-bisect', '--debugger', 'lldb', '--builder', 'clang', '--cflags', '-O0 -g', '--', '/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/feature_tests/subtools/clang-opt-bisect/clang-opt-bisect.cpp']

Traceback (most recent call last):
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dexter.py", line 15, in <module>
    return_code = main()
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dex/tools/Main.py", line 193, in main
    return tool_main(context, module.Tool(context), args)
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dex/tools/Main.py", line 162, in tool_main
    return_code = tool.go()
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dex/tools/TestToolBase.py", line 122, in go
    self._run_test(self._get_test_name(options.test_path))
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dex/tools/clang_opt_bisect/Tool.py", line 141, in _run_test
    self.context, debugger_controller)
  File "/usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/dex/debugger/Debuggers.py", line 140, in run_debugger_subprocess
    dir=working_dir_path, delete=False, mode='wb') as fp:
  File "/usr/lib/python3.7/tempfile.py", line 686, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.7/tempfile.py", line 395, in _mkstemp_inner
    file = _os.path.join(dir, pre + name + suf)
  File "/usr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not DefaultController
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /usr/local/google/home/tbosch/llvm-project/build/bin/FileCheck /usr/local/google/home/tbosch/llvm-project/debuginfo-tests/dexter/feature_tests/subtools/clang-opt-bisect/clang-opt-bisect.cpp

P.S.: You might have received this comment 2x, I am new to phrabricator...

[TWeaver]

Hello,

thanks for reporting. this issue should now be fixed.

thanks again,
Tom W

[TWeaver]

fix for reported test failure at:

rGb1097e29d3d06be1d5767dffddaa8f9c5dbdcc83

[tbosch]

Thanks, just verified that this is fixed for me too.

debuginfo-tests/dexter/dex/command/ParseCommand.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Parse a DExTer command. In particular, ensure that only a very limited
 subset of Python is allowed, in order to prevent the possibility of unsafe
 Python code being embedded within DExTer commands.
 """
 
 import os
 import unittest
 from copy import copy
 
-from collections import defaultdict
+from collections import defaultdict, OrderedDict
 
 from dex.utils.Exceptions import CommandParseError
 
 from dex.command.CommandBase import CommandBase
 from dex.command.commands.DexExpectProgramState import DexExpectProgramState
 from dex.command.commands.DexExpectStepKind import DexExpectStepKind
 from dex.command.commands.DexExpectStepOrder import DexExpectStepOrder
 from dex.command.commands.DexExpectWatchType import DexExpectWatchType
 from dex.command.commands.DexExpectWatchValue import DexExpectWatchValue
 from dex.command.commands.DexLabel import DexLabel
 from dex.command.commands.DexUnreachable import DexUnreachable
 from dex.command.commands.DexWatch import DexWatch
-
+from dex.utils import Timer
+from dex.utils.Exceptions import CommandParseError, DebuggerException
 
 def _get_valid_commands():
     """Return all top level DExTer test commands.
 
     Returns:
         { name (str): command (class) }
     """
     return {
def _find_all_commands_in_file(path, file_lines, valid_commands):
     if paren_balance != 0:
         # This err should always point to the end of the command name.
         err_point = copy(cmd_point)
         err_point.char += len(command_name)
         msg = "Unbalanced parenthesis starting here"
         raise format_parse_err(msg, path, file_lines, err_point)
     return dict(commands)
 
-
-
-def find_all_commands(source_files):
+def _find_all_commands(source_files):
     commands = defaultdict(dict)
     valid_commands = _get_valid_commands()
     for source_file in source_files:
         with open(source_file) as fp:
             lines = fp.readlines()
         file_commands = _find_all_commands_in_file(source_file, lines,
                                                    valid_commands)
         for command_name in file_commands:
             commands[command_name].update(file_commands[command_name])
 
     return dict(commands)
 
+def get_command_infos(source_files):
+  with Timer('parsing commands'):
+      try:
+          commands = _find_all_commands(source_files)
+          command_infos = OrderedDict()
+          for command_type in commands:
+              for command in commands[command_type].values():
+                  if command_type not in command_infos:
+                      command_infos[command_type] = []
+                  command_infos[command_type].append(command)
+          return OrderedDict(command_infos)
+      except CommandParseError as e:
+          msg = 'parser error: <d>{}({}):</> {}\n{}\n{}\n'.format(
+                e.filename, e.lineno, e.info, e.src, e.caret)
+          raise DebuggerException(msg)
 
 class TestParseCommand(unittest.TestCase):
     class MockCmd(CommandBase):
         """A mock DExTer command for testing parsing.
 
         Args:
             value (str): Unique name for this instance.
         """

debuginfo-tests/dexter/dex/command/__init__.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-from dex.command.ParseCommand import find_all_commands
+from dex.command.ParseCommand import get_command_infos
 from dex.command.StepValueInfo import StepValueInfo

debuginfo-tests/dexter/dex/command/commands/DexExpectStepOrder.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 from dex.command.CommandBase import CommandBase
+from dex.dextIR import LocIR
 from dex.dextIR import ValueIR
 
 class DexExpectStepOrder(CommandBase):
     """Expect the line every `DexExpectStepOrder` is found on to be stepped on
     in `order`. Each instance must have a set of unique ascending indicies.
 
     DexExpectStepOrder(*order)
 
     See Commands.md for more info.
     """
 
     def __init__(self, *args):
         if not args:
             raise TypeError('Need at least one order number')
 
         self.sequence = [int(x) for x in args]
         super(DexExpectStepOrder, self).__init__()
 
     @staticmethod
     def get_name():
         return __class__.__name__
 
-    def eval(self, debugger):
-        step_info = debugger.get_step_info()
-        loc = step_info.current_location
-        return {'DexExpectStepOrder': ValueIR(expression=str(loc.lineno),
-                      value=str(debugger.step_index), type_name=None,
+    def eval(self, step_info):
+        return {'DexExpectStepOrder': ValueIR(expression=str(step_info.current_location.lineno),
+                      value=str(step_info.step_index), type_name=None,
                       error_string=None,
                       could_evaluate=True,
                       is_optimized_away=True,
                       is_irretrievable=False)}

debuginfo-tests/dexter/dex/command/commands/DexUnreachable.py

class DexUnreachable(CommandBase):
     def __init(self):
         super(DexUnreachable, self).__init__()
         pass
 
     @staticmethod
     def get_name():
         return __class__.__name__
 
-    def eval(self, debugger):
+    def eval(self, step_info):
         # If we're ever called, at all, then we're evaluating a line that has
         # been marked as unreachable. Which means a failure.
         vir = ValueIR(expression="Unreachable",
                       value="True", type_name=None,
                       error_string=None,
                       could_evaluate=True,
                       is_optimized_away=True,
                       is_irretrievable=False)
         return {'DexUnreachable' : vir}

debuginfo-tests/dexter/dex/debugger/DebuggerBase.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Base class for all debugger interface implementations."""
 
 import abc
-from itertools import chain
-import os
 import sys
-import time
 import traceback
 
 from dex.dextIR import DebuggerIR, ValueIR
 from dex.utils.Exceptions import DebuggerException
 from dex.utils.Exceptions import NotYetLoadedDebuggerException
 from dex.utils.ReturnCode import ReturnCode
 
 
 class DebuggerBase(object, metaclass=abc.ABCMeta):
-    def __init__(self, context, step_collection):
+    def __init__(self, context):
         self.context = context
-        self.steps = step_collection
         self._interface = None
         self.has_loaded = False
         self._loading_error = NotYetLoadedDebuggerException()
-        self.watches = set()
-
         try:
             self._interface = self._load_interface()
             self.has_loaded = True
             self._loading_error = None
         except DebuggerException:
             self._loading_error = sys.exc_info()
 
-        self.step_index = 0
-
     def __enter__(self):
         try:
             self._custom_init()
             self.clear_breakpoints()
-            self.add_breakpoints()
         except DebuggerException:
             self._loading_error = sys.exc_info()
         return self
 
     def __exit__(self, *args):
         self._custom_exit()
 
     def _custom_init(self):
def loading_error_trace(self):
 
             if ''.join(orig_exception) not in ''.join(tb):
                 tb.extend(['\n'])
                 tb.extend(orig_exception)
 
         tb = ''.join(tb).splitlines(True)
         return tb
 
-    def add_breakpoints(self):
-        for s in self.context.options.source_files:
-            with open(s, 'r') as fp:
-                num_lines = len(fp.readlines())
-            for line in range(1, num_lines + 1):
-                self.add_breakpoint(s, line)
-
-    def _update_step_watches(self, step_info):
-        loc = step_info.current_location
-        watch_cmds = ['DexUnreachable', 'DexExpectStepOrder']
-        towatch = chain.from_iterable(self.steps.commands[x]
-                                      for x in watch_cmds
-                                      if x in self.steps.commands)
-        try:
-            # Iterate over all watches of the types named in watch_cmds
-            for watch in towatch:
-                if (os.path.exists(loc.path)
-                        and os.path.samefile(watch.path, loc.path)
-                        and watch.lineno == loc.lineno):
-                    result = watch.eval(self)
-                    step_info.watches.update(result)
-                    break
-        except KeyError:
-            pass
-
     def _sanitize_function_name(self, name):  # pylint: disable=no-self-use
         """If the function name returned by the debugger needs any post-
         processing to make it fit (for example, if it includes a byte offset),
         do that here.
         """
         return name
 
-    def start(self):
-        self.steps.clear_steps()
-        self.launch()
-
-        for command_obj in chain.from_iterable(self.steps.commands.values()):
-            self.watches.update(command_obj.get_watches())
-
-        max_steps = self.context.options.max_steps
-        for _ in range(max_steps):
-            while self.is_running:
-                pass
-
-            if self.is_finished:
-                break
-
-            self.step_index += 1
-            step_info = self.get_step_info()
-
-            if step_info.current_frame:
-                self._update_step_watches(step_info)
-                self.steps.new_step(self.context, step_info)
-
-            if self.in_source_file(step_info):
-                self.step()
-            else:
-                self.go()
-
-            time.sleep(self.context.options.pause_between_steps)
-        else:
-            raise DebuggerException(
-                'maximum number of steps reached ({})'.format(max_steps))
-
-    def in_source_file(self, step_info):
-        if not step_info.current_frame:
-            return False
-        if not step_info.current_location.path:
-            return False
-        if not os.path.exists(step_info.current_location.path):
-            return False
-        return any(os.path.samefile(step_info.current_location.path, f) \
-                   for f in self.context.options.source_files)
-
     @abc.abstractmethod
     def _load_interface(self):
         pass
 
     @classmethod
     def get_option_name(cls):
         """Short name that will be used on the command line to specify this
         debugger.
class DebuggerBase(object, metaclass=abc.ABCMeta):
     def step(self):
         pass
 
     @abc.abstractmethod
     def go(self) -> ReturnCode:
         pass
 
     @abc.abstractmethod
-    def get_step_info(self):
+    def get_step_info(self, watches, step_index):
         pass
 
     @abc.abstractproperty
     def is_running(self):
         pass
 
     @abc.abstractproperty
     def is_finished(self):

debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DebuggerControllerBase.py

+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Default class for controlling debuggers."""
+
+import abc
+
+class DebuggerControllerBase(object, metaclass=abc.ABCMeta):
+    @abc.abstractclassmethod
+    def _run_debugger_custom(self):
+        """Specify your own implementation of run_debugger_custom in your own
+        controller.
+        """
+        pass
+
+    def run_debugger(self, debugger):
+        """Responsible for correctly launching and tearing down the debugger.
+        """
+        self.debugger = debugger
+        with self.debugger:
+            self._run_debugger_custom()
+        # We may need to pickle this debugger controller after running the
+        # debugger. Debuggers are not picklable objects, so set to None.
+        self.debugger = None

debuginfo-tests/dexter/dex/debugger/DebuggerControllers/DefaultController.py

+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Base class for controlling debuggers."""
+
+from itertools import chain
+import os
+import time
+
+from dex.debugger.DebuggerControllers.DebuggerControllerBase import DebuggerControllerBase
+from dex.utils.Exceptions import DebuggerException
+
+class DefaultController(DebuggerControllerBase):
+    def __init__(self, context, step_collection):
+        self.context = context
+        self.step_collection = step_collection
+        self.watches = set()
+        self.step_index = 0
+
+    def _update_step_watches(self, step_info):
+        watch_cmds = ['DexUnreachable', 'DexExpectStepOrder']
+        towatch = chain.from_iterable(self.step_collection.commands[x]
+                                      for x in watch_cmds
+                                      if x in self.step_collection.commands)
+        try:
+            # Iterate over all watches of the types named in watch_cmds
+            for watch in towatch:
+                loc = step_info.current_location
+                if (os.path.exists(loc.path)
+                        and os.path.samefile(watch.path, loc.path)
+                        and watch.lineno == loc.lineno):
+                    result = watch.eval(step_info)
+                    step_info.watches.update(result)
+                    break
+        except KeyError:
+            pass
+
+    def _break_point_all_lines(self):
+        for s in self.context.options.source_files:
+            with open(s, 'r') as fp:
+                num_lines = len(fp.readlines())
+            for line in range(1, num_lines + 1):
+                self.debugger.add_breakpoint(s, line)
+
+    def _in_source_file(self, step_info):
+        if not step_info.current_frame:
+            return False
+        if not step_info.current_location.path:
+            return False
+        if not os.path.exists(step_info.current_location.path):
+            return False
+        return any(os.path.samefile(step_info.current_location.path, f) \
+                   for f in self.context.options.source_files)
+
+    def _run_debugger_custom(self):
+        self.step_collection.debugger = self.debugger.debugger_info
+        self._break_point_all_lines()
+
+        self.debugger.launch()
+
+        for command_obj in chain.from_iterable(self.step_collection.commands.values()):
+            self.watches.update(command_obj.get_watches())
+
+        max_steps = self.context.options.max_steps
+        for _ in range(max_steps):
+            while self.debugger.is_running:
+                pass
+
+            if self.debugger.is_finished:
+                break
+
+            self.step_index += 1
+            step_info = self.debugger.get_step_info(self.watches, self.step_index)
+
+            if step_info.current_frame:
+                self._update_step_watches(step_info)
+                self.step_collection.new_step(self.context, step_info)
+
+            if self._in_source_file(step_info):
+                self.debugger.step()
+            else:
+                self.debugger.go()
+
+            time.sleep(self.context.options.pause_between_steps)
+        else:
+            raise DebuggerException(
+                'maximum number of steps reached ({})'.format(max_steps))

debuginfo-tests/dexter/dex/debugger/Debuggers.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Discover potential/available debugger interfaces."""
 
 from collections import OrderedDict
 import os
 import pickle
 import subprocess
 import sys
 from tempfile import NamedTemporaryFile
 
-from dex.command import find_all_commands
+from dex.command import get_command_infos
 from dex.dextIR import DextIR
 from dex.utils import get_root_directory, Timer
 from dex.utils.Environment import is_native_windows
-from dex.utils.Exceptions import CommandParseError, DebuggerException
 from dex.utils.Exceptions import ToolArgumentError
 from dex.utils.Warning import warn
+from dex.utils.Exceptions import DebuggerException
+
+from dex.debugger.DebuggerControllers.DefaultController import DefaultController
 
 from dex.debugger.dbgeng.dbgeng import DbgEng
 from dex.debugger.lldb.LLDB import LLDB
 from dex.debugger.visualstudio.VisualStudio2015 import VisualStudio2015
 from dex.debugger.visualstudio.VisualStudio2017 import VisualStudio2017
 
 
 def _get_potential_debuggers():  # noqa
def handle_debugger_tool_options(context, defaults):  # noqa
 
     if options.show_debugger is None:
         options.show_debugger = defaults.show_debugger
     else:
         if options.debugger == 'lldb':
             _warn_meaningless_option(context, '--show-debugger')
 
 
-def _get_command_infos(context):
-    commands = find_all_commands(context.options.source_files)
-    command_infos = OrderedDict()
-    for command_type in commands:
-        for command in commands[command_type].values():
-            if command_type not in command_infos:
-                command_infos[command_type] = []
-            command_infos[command_type].append(command)
-    return OrderedDict(command_infos)
-
-
-def empty_debugger_steps(context):
-    return DextIR(
-        executable_path=context.options.executable,
-        source_paths=context.options.source_files,
-        dexter_version=context.version)
-
-
-def get_debugger_steps(context):
-    step_collection = empty_debugger_steps(context)
-
-    with Timer('parsing commands'):
-        try:
-            step_collection.commands = _get_command_infos(context)
-        except CommandParseError as e:
-            msg = 'parser error: <d>{}({}):</> {}\n{}\n{}\n'.format(
-                e.filename, e.lineno, e.info, e.src, e.caret)
-            raise DebuggerException(msg)
-
-    with NamedTemporaryFile(
-            dir=context.working_directory.path, delete=False) as fp:
-        pickle.dump(step_collection, fp, protocol=pickle.HIGHEST_PROTOCOL)
-        steps_path = fp.name
-
+def run_debugger_subprocess(debugger_controller, working_dir_path):
     with NamedTemporaryFile(
-            dir=context.working_directory.path, delete=False, mode='wb') as fp:
-        pickle.dump(context.options, fp, protocol=pickle.HIGHEST_PROTOCOL)
-        options_path = fp.name
+            dir=working_dir_path, delete=False, mode='wb') as fp:
+        pickle.dump(debugger_controller, fp, protocol=pickle.HIGHEST_PROTOCOL)
+        controller_path = fp.name
 
     dexter_py = os.path.basename(sys.argv[0])
     if not os.path.isfile(dexter_py):
         dexter_py = os.path.join(get_root_directory(), '..', dexter_py)
     assert os.path.isfile(dexter_py)
 
-    with NamedTemporaryFile(dir=context.working_directory.path) as fp:
+    with NamedTemporaryFile(dir=working_dir_path) as fp:
         args = [
-            sys.executable, dexter_py, 'run-debugger-internal-', steps_path,
-            options_path, '--working-directory', context.working_directory.path,
-            '--unittest=off', '--indent-timer-level={}'.format(Timer.indent + 2)
+            sys.executable,
+            dexter_py,
+            'run-debugger-internal-',
+            controller_path,
+            '--working-directory={}'.format(working_dir_path),
+            '--unittest=off',
+            '--indent-timer-level={}'.format(Timer.indent + 2)
         ]
         try:
             with Timer('running external debugger process'):
                 subprocess.check_call(args)
         except subprocess.CalledProcessError as e:
             raise DebuggerException(e)
 
-    with open(steps_path, 'rb') as fp:
-        step_collection = pickle.load(fp)
+    with open(controller_path, 'rb') as fp:
+        debugger_controller = pickle.load(fp)
 
-    return step_collection
+    return debugger_controller
 
 
 class Debuggers(object):
     @classmethod
     def potential_debuggers(cls):
         try:
             return cls._potential_debuggers
         except AttributeError:
             cls._potential_debuggers = _get_potential_debuggers()
             return cls._potential_debuggers
 
     def __init__(self, context):
         self.context = context
 
-    def load(self, key, step_collection=None):
+    def load(self, key):
         with Timer('load {}'.format(key)):
-            return Debuggers.potential_debuggers()[key](self.context,
-                                                        step_collection)
+            return Debuggers.potential_debuggers()[key](self.context)
 
     def _populate_debugger_cache(self):
         debuggers = []
         for key in sorted(Debuggers.potential_debuggers()):
             debugger = self.load(key)
 
             class LoadedDebugger(object):
                 pass

debuginfo-tests/dexter/dex/debugger/__init__.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 from dex.debugger.Debuggers import Debuggers
+from dex.debugger.DebuggerControllers.DebuggerControllerBase import DebuggerControllerBase
+from dex.debugger.DebuggerControllers.DefaultController import DefaultController

debuginfo-tests/dexter/dex/debugger/dbgeng/dbgeng.py

def step(self):
         if not res:
           self.finished = True
         self.step_info = res
 
     def go(self):
         # We never go -- we always single step.
         pass
 
-    def get_step_info(self):
+    def get_step_info(self, watches, step_index):
         frames = self.step_info
         state_frames = []
 
         # For now assume the base function is the... function, ignoring
         # inlining.
         dex_frames = []
         for i, x in enumerate(frames):
           # XXX Might be able to get columns out through
           # GetSourceEntriesByOffset, not a priority now
           loc = LocIR(path=x.source_file, lineno=x.line_no, column=0)
           new_frame = FrameIR(function=x.function_name, is_inlined=False, loc=loc)
           dex_frames.append(new_frame)
 
           state_frame = StackFrame(function=new_frame.function,
                                    is_inlined=new_frame.is_inlined,
                                    location=SourceLocation(path=x.source_file,
                                                            lineno=x.line_no,
                                                            column=0),
                                    watches={})
           for expr in map(
               lambda watch, idx=i: self.evaluate_expression(watch, idx),
-              self.watches):
+              watches):
               state_frame.watches[expr.expression] = expr
           state_frames.append(state_frame)
 
         return StepIR(
-            step_index=self.step_index, frames=dex_frames,
+            step_index=step_index, frames=dex_frames,
             stop_reason=StopReason.STEP,
             program_state=ProgramState(state_frames))
 
     @property
     def is_running(self):
         return False # We're never free-running
 
     @property

debuginfo-tests/dexter/dex/debugger/lldb/LLDB.py

class LLDB(DebuggerBase):
 
     def step(self):
         self._thread.StepInto()
 
     def go(self) -> ReturnCode:
         self._process.Continue()
         return ReturnCode.OK
 
-    def get_step_info(self):
+    def get_step_info(self, watches, step_index):
         frames = []
         state_frames = []
 
         for i in range(0, self._thread.GetNumFrames()):
             sb_frame = self._thread.GetFrameAtIndex(i)
             sb_line = sb_frame.GetLineEntry()
             sb_filespec = sb_line.GetFileSpec()
 
def get_step_info(self, watches, step_index):
             frames.append(frame)
 
             state_frame = StackFrame(function=frame.function,
                                      is_inlined=frame.is_inlined,
                                      location=SourceLocation(**loc_dict),
                                      watches={})
             for expr in map(
                 lambda watch, idx=i: self.evaluate_expression(watch, idx),
-                self.watches):
+                watches):
                 state_frame.watches[expr.expression] = expr
             state_frames.append(state_frame)
 
         if len(frames) == 1 and frames[0].function is None:
             frames = []
             state_frames = []
 
         reason = self._translate_stop_reason(self._thread.GetStopReason())
 
         return StepIR(
-            step_index=self.step_index, frames=frames, stop_reason=reason,
+            step_index=step_index, frames=frames, stop_reason=reason,
             program_state=ProgramState(state_frames))
 
     @property
     def is_running(self):
         # We're not running in async mode so this is always False.
         return False
 
     @property

debuginfo-tests/dexter/dex/debugger/visualstudio/VisualStudio.py

def set_current_stack_frame(self, idx: int = 0):
         stack_frames = thread.StackFrames
         try:
             stack_frame = stack_frames[idx]
             self._debugger.CurrentStackFrame = stack_frame.raw
         except IndexError:
             raise Error('attempted to access stack frame {} out of {}'
                 .format(idx, len(stack_frames)))
 
-    def get_step_info(self):
+    def get_step_info(self, watches, step_index):
         thread = self._debugger.CurrentThread
         stackframes = thread.StackFrames
 
         frames = []
         state_frames = []
 
 
         for idx, sf in enumerate(stackframes):
             frame = FrameIR(
                 function=self._sanitize_function_name(sf.FunctionName),
                 is_inlined=sf.FunctionName.startswith('[Inline Frame]'),
                 loc=LocIR(path=None, lineno=None, column=None))
 
             fname = frame.function or ''  # pylint: disable=no-member
             if any(name in fname for name in self.frames_below_main):
                 break
 
 
             state_frame = StackFrame(function=frame.function,
                                      is_inlined=frame.is_inlined,
                                      watches={})
 
-            for watch in self.watches:
+            for watch in watches:
                 state_frame.watches[watch] = self.evaluate_expression(
                     watch, idx)
 
 
             state_frames.append(state_frame)
             frames.append(frame)
 
         loc = LocIR(**self._location)
         if frames:
             frames[0].loc = loc
             state_frames[0].location = SourceLocation(**self._location)
 
         reason = StopReason.BREAKPOINT
         if loc.path is None:  # pylint: disable=no-member
             reason = StopReason.STEP
 
         program_state = ProgramState(frames=state_frames)
 
         return StepIR(
-            step_index=self.step_index, frames=frames, stop_reason=reason,
+            step_index=step_index, frames=frames, stop_reason=reason,
             program_state=program_state)
 
     @property
     def is_running(self):
         return self._mode == VisualStudio.dbgRunMode
 
     @property
     def is_finished(self):

debuginfo-tests/dexter/dex/tools/clang_opt_bisect/Tool.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Clang opt-bisect tool."""
 
 from collections import defaultdict
 import os
 import csv
 import re
 import pickle
 
 from dex.builder import run_external_build_script
-from dex.debugger.Debuggers import empty_debugger_steps, get_debugger_steps
+from dex.command.ParseCommand import get_command_infos
+from dex.debugger.Debuggers import run_debugger_subprocess
+from dex.debugger.DebuggerControllers.DefaultController import DefaultController
+from dex.dextIR.DextIR import DextIR
 from dex.heuristic import Heuristic
 from dex.tools import TestToolBase
 from dex.utils.Exceptions import DebuggerException, Error
 from dex.utils.Exceptions import BuildScriptException, HeuristicException
 from dex.utils.PrettyOutputBase import Stream
 from dex.utils.ReturnCode import ReturnCode
 
 
class Tool(TestToolBase):
 
     def handle_options(self, defaults):
         options = self.context.options
         if "clang" not in options.builder.lower():
             raise Error("--builder %s is not supported by the clang-opt-bisect tool - only 'clang' is "
                         "supported " % options.builder)
         super(Tool, self).handle_options(defaults)
 
+    def _init_debugger_controller(self):
+        step_collection = DextIR(
+            executable_path=self.context.options.executable,
+            source_paths=self.context.options.source_files,
+            dexter_version=self.context.version)
+        step_collection.commands = get_command_infos(
+            self.context.options.source_files)
+        debugger_controller = DefaultController(self.context, step_collection)
+        return debugger_controller
+
     def _run_test(self, test_name):  # noqa
         options = self.context.options
 
         per_pass_score = []
         current_bisect_pass_summary = defaultdict(list)
 
         max_limits = self._get_bisect_limits()
         overall_limit = sum(max_limits)
def _run_test(self, test_name):  # noqa
                 match = Tool._re_running_pass.match(l)
                 if match:
                     pass_info = match.groups()
                     break
             else:
                 pass_info = (0, None, None)
 
             try:
-                steps = get_debugger_steps(self.context)
+                debugger_controller =self._init_debugger_controller()
+                debugger_controller = run_debugger_subprocess(
+                    self.context, debugger_controller)
+                steps = debugger_controller.step_collection
             except DebuggerException:
-                steps = empty_debugger_steps(self.context)
+                steps =  DextIR(
+                    executable_path=self.context.options.executable,
+                    source_paths=self.context.options.source_files,
+                    dexter_version=self.context.version)
 
             steps.builder = builderIR
 
             try:
                 heuristic = Heuristic(self.context, steps)
             except HeuristicException as e:
                 raise Error(e)
 

debuginfo-tests/dexter/dex/tools/run_debugger_internal_/Tool.py

 import pickle
 
 from dex.debugger import Debuggers
 from dex.tools import ToolBase
 from dex.utils import Timer
 from dex.utils.Exceptions import DebuggerException, Error
 from dex.utils.ReturnCode import ReturnCode
 
-
 class Tool(ToolBase):
     def __init__(self, *args, **kwargs):
+        self.controller_path = None
+        self.debugger_controller = None
+        self.options = None
         super(Tool, self).__init__(*args, **kwargs)
-        self.dextIR = None
 
     @property
     def name(self):
         return 'DExTer run debugger internal'
 
     def add_tool_arguments(self, parser, defaults):
-        parser.add_argument('dextIR_path', type=str, help='dextIR file')
         parser.add_argument(
-            'pickled_options', type=str, help='pickled options file')
+            'controller_path',
+            type=str,
+            help='pickled debugger controller file')
 
     def handle_options(self, defaults):
-        with open(self.context.options.dextIR_path, 'rb') as fp:
-            self.dextIR = pickle.load(fp)
-
-        with open(self.context.options.pickled_options, 'rb') as fp:
-            poptions = pickle.load(fp)
-            poptions.working_directory = (
-                self.context.options.working_directory[:])
-            poptions.unittest = self.context.options.unittest
-            poptions.dextIR_path = self.context.options.dextIR_path
-            self.context.options = poptions
-
-        Timer.display = self.context.options.time_report
+        with open(self.context.options.controller_path, 'rb') as fp:
+            self.debugger_controller = pickle.load(fp)
+        self.controller_path = self.context.options.controller_path   
+        self.context = self.debugger_controller.context
+        self.options = self.context.options
+        Timer.display = self.options.time_report
 
     def go(self) -> ReturnCode:
-        options = self.context.options
-
         with Timer('loading debugger'):
-            debugger = Debuggers(self.context).load(options.debugger,
-                                                    self.dextIR)
-            self.dextIR.debugger = debugger.debugger_info
+            debugger = Debuggers(self.context).load(self.options.debugger)
 
         with Timer('running debugger'):
             if not debugger.is_available:
                 msg = '<d>could not load {}</> ({})\n'.format(
                     debugger.name, debugger.loading_error)
-                if options.verbose:
+                if self.options.verbose:
                     msg = '{}\n    {}'.format(
                         msg, '    '.join(debugger.loading_error_trace))
                 raise Error(msg)
 
-            with debugger:
-                try:
-                    debugger.start()
-                except DebuggerException as e:
-                    raise Error(e)
+        self.debugger_controller.run_debugger(debugger)
 
-        with open(self.context.options.dextIR_path, 'wb') as fp:
-            pickle.dump(self.dextIR, fp)
+        with open(self.controller_path, 'wb') as fp:
+            pickle.dump(self.debugger_controller, fp)
         return ReturnCode.OK

debuginfo-tests/dexter/dex/tools/test/Tool.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Test tool."""
 
 import math
 import os
 import csv
 import pickle
 import shutil
 
 from dex.builder import run_external_build_script
-from dex.debugger.Debuggers import get_debugger_steps
+from dex.command.ParseCommand import get_command_infos
+from dex.debugger.Debuggers import run_debugger_subprocess
+from dex.debugger.DebuggerControllers.DefaultController import DefaultController
+from dex.dextIR.DextIR import DextIR
 from dex.heuristic import Heuristic
 from dex.tools import TestToolBase
 from dex.utils.Exceptions import DebuggerException
 from dex.utils.Exceptions import BuildScriptException, HeuristicException
 from dex.utils.PrettyOutputBase import Stream
 from dex.utils.ReturnCode import ReturnCode
 from dex.dextIR import BuilderIR
 
def _build_test_case(self):
                 self.context,
                 script_path=self.build_script,
                 source_files=options.source_files,
                 compiler_options=compiler_options,
                 linker_options=linker_options,
                 executable_file=options.executable)
         return builderIR
 
+    def _init_debugger_controller(self):
+        step_collection = DextIR(
+            executable_path=self.context.options.executable,
+            source_paths=self.context.options.source_files,
+            dexter_version=self.context.version)
+        step_collection.commands = get_command_infos(
+            self.context.options.source_files)
+        debugger_controller = DefaultController(self.context, step_collection)
+        return debugger_controller
+
     def _get_steps(self, builderIR):
         """Generate a list of debugger steps from a test case.
         """
-        steps = get_debugger_steps(self.context)
+        debugger_controller = self._init_debugger_controller()
+        debugger_controller = run_debugger_subprocess(
+            debugger_controller, self.context.working_directory.path)
+        steps = debugger_controller.step_collection
         steps.builder = builderIR
         return steps
 
     def _get_results_basename(self, test_name):
         def splitall(x):
             while len(x) > 0:
               x, y = os.path.split(x)
               yield y