[Dexter] Implement DexDeclareFile, a new Dexter command

[TWeaver]

Summary

DexDeclareFile allows test producers to write test files with .dex extensions that
contain pure dexter commands.

.dex file commands do not need to be commented out like they do when written
inline within test source files.

DexDeclareFile commands are declarative in behaviour, they state that any Dexter
command seen from this point on will have its path attribute set to the path
declared in the DexDeclareFile command.

Provided in this patch is in the implementation of DexDeclareFile and 4 tests.

[Orlando]

LGTM with a few tiny nits, but please wait for a +1 from someone else too.

debuginfo-tests/dexter/Commands.md

229

Changes -> Change (or maybe Set) fits the existing style of this document better imo, but this isn't very important.

debuginfo-tests/dexter/dex/command/ParseCommand.py

264

Should cmd_path get normalized with os.path.normcase here since the user can spell the path name in a few different ways on windows? Unhelpfully I can't remember exactly why it was, but when tinkering with this patch, at one point I did have to make that change.

Does everything work if the DexDeclareFile commands use\\annOying/PathSpelling (i.e. a mix of case and slashes that doesn't match the "canonical" spelling)? Maybe this deserves a test.

debuginfo-tests/dexter/dex/tools/TestToolBase.py

103

Tiny nit: comments should probably follow the llvm style (comments added here don't start with capitals but should).

[jmorse]

LGTM, with a couple of nits, plus Orlandos comments should be addressed.

The precompiled binary test is great, that should get us going in the right direction.

debuginfo-tests/dexter/dex/tools/test/Tool.py

144–145

Mega nit: self.context.options.source_files.extend(list(new_source_files)) is slightly more concise.

debuginfo-tests/dexter/feature_tests/commands/penalty/dex_declare_file.cpp

17

Major nit, but I think this should be a filename that absolutely screams that it doesn't exist; thus something like "this_file_should_not_exist.cpp". that avoids the reader spending a second wondering whether the "not" prefix is significant.

(This might be because my default name for anything, when moved, is not_$originalname).

[jmorse]

I ended up taking this for a test drive, and found some difficulty when using the LLDB bindings -- _add_conditional_breakpoint ends up being passed a PurePath object for the file_, which is then passed to the LLDB bindings, which croak. This is easily fixed by calling str(file_) to convert the PurePath object to a string, but this might affect other debugger drivers & need some assertions adding.

[Orlando]

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/test.cpp

10

This lit test fails for me locally - dexter doesn't run the test. Adding a test.cfg file to this directory fixes it.

[TWeaver]

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/test.cpp

10

slightly perplexing, this patch contains the test.cfg for the directory in question...

[Orlando]

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/test.cpp

10

Ah yes I see it, I must've messed up applying the patch and not noticed. Sorry!

[TWeaver]

Addressed feedback and fixed a test failure in the clang-opt-bisect tool test.

[TWeaver]

re-uploaded new diff with more context

[Orlando]

It looks like you've addressed both of our comments (except my suggestion for a non-canonical path spelling test for windows exclusively, which would still be welcomed but isn't critical) so LGTM. I'll leave it to @jmorse to Accept this if he's happy.

[jmorse]

LGTM

[Orlando]

Hi, before this lands you will need to update your dexter tests to use the new label reference syntax added in D101147. You can do so manually, or alternatively use this python script by running python make_label_refs.py debuginfo-tests/feature_tests (please note it will modify files in place). I'm happy for you to amend your commit with that update without bringing it back up here for review.

[Orlando]

In D99651#2773497, @Orlando wrote:

Hi, before this lands you will need to update your dexter tests to use the new label reference syntax added in D101147. You can do so manually, or alternatively use this python script by running python make_label_refs.py debuginfo-tests/feature_tests (please note it will modify files in place). I'm happy for you to amend your commit with that update without bringing it back up here for review.

Of course, you don't need to do this if you don't use DexLabels in your tests, which you haven't. Please ignore that, sorry!

[TWeaver]

Apologies for the missing test, here it is! :)

Will be pushing this shortly, thanks a bunch.

# Purpose:
#    Check that non canonical paths resolve correctly on windows.
#
# REQUIRES: system-windows
#
# RUN: %clang "%S/source/test file.cpp" -O0 -g -o %t
# RUN: %dexter_regression_test --binary %t %s | FileCheck %s
# CHECK: test.dex
#
# ./source/test file.cpp
# 1 int main(const int argc, const char * argv[]) {
# 2 int result = argc;
# 3 return result;
# 4 }

DexDeclareFile('./sOuRce\\test filE.cpp')
DexExpectWatchValue('result', 1, on_line=3)

[Orlando]

In D99651#2777244, @TWeaver wrote:

Apologies for the missing test, here it is! :)

Will be pushing this shortly, thanks a bunch.

# Purpose:
#    Check that non canonical paths resolve correctly on windows.
#
# REQUIRES: system-windows
#
# RUN: %clang "%S/source/test file.cpp" -O0 -g -o %t
# RUN: %dexter_regression_test --binary %t %s | FileCheck %s
# CHECK: test.dex
#
# ./source/test file.cpp
# 1 int main(const int argc, const char * argv[]) {
# 2 int result = argc;
# 3 return result;
# 4 }

DexDeclareFile('./sOuRce\\test filE.cpp')
DexExpectWatchValue('result', 1, on_line=3)

Thanks! Looks good.

debuginfo-tests/dexter/Commands.md

 # Dexter commands
 
 * [DexExpectProgramState](Commands.md#DexExpectProgramState)
 * [DexExpectStepKind](Commands.md#DexExpectStepKind)
 * [DexExpectStepOrder](Commands.md#DexExpectStepOrder)
 * [DexExpectWatchType](Commands.md#DexExpectWatchType)
 * [DexExpectWatchValue](Commands.md#DexExpectWatchValue)
 * [DexUnreachable](Commands.md#DexUnreachable)
 * [DexWatch](Commands.md#DexWatch)
+* [DexDeclareFile](Commands.md#DexDeclareFile)
 
 ---
 ## DexExpectProgramState
     DexExpectProgramState(state [,**times])
 
     Args:
         state (dict): { 'frames': [
                         {

 ### Description
 Name the line this command is found on. Line names can be referenced by other
 commands expecting line number arguments.
 For example, `DexExpectWatchValues(..., on_line='my_line_name')`.
 
 ### Heuristic
 This command does not contribute to the heuristic score.
 
+----
+## DexDeclareFile
+    DexDeclareFile(declared_file)
+
+    Args:
+        name (str): A declared file path for which all subsequent commands
+          will have their path attribute set too.
+
+### Description
+Set the path attribute of all commands from this point in the test onwards.
+The new path holds until the end of the test file or until a new DexDeclareFile
+command is encountered. Used in conjunction with .dex files, DexDeclareFile can
+be used to write your dexter commands in a separate test file avoiding inlined
+Dexter commands mixed with test source.
+
+### Heuristic
+This command does not contribute to the heuristic score.
 
 ---
 ## DexWatch
     DexWatch(*expressions)
 
     Arg list:
         expressions (str): `expression` to evaluate on this line.
 
 ### Description
 [Deprecated] Evaluate each given `expression` when the debugger steps onto the
 line this command is found on.
 
 ### Heuristic
 [Deprecated]

debuginfo-tests/dexter/dex/command/ParseCommand.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Parse a DExTer command. In particular, ensure that only a very limited
 subset of Python is allowed, in order to prevent the possibility of unsafe
 Python code being embedded within DExTer commands.
 """
 
 import os
 import unittest
 from copy import copy
-
+from pathlib import PurePath
 from collections import defaultdict, OrderedDict
 
 from dex.utils.Exceptions import CommandParseError
 
 from dex.command.CommandBase import CommandBase
+from dex.command.commands.DexDeclareFile import DexDeclareFile
 from dex.command.commands.DexExpectProgramState import DexExpectProgramState
 from dex.command.commands.DexExpectStepKind import DexExpectStepKind
 from dex.command.commands.DexExpectStepOrder import DexExpectStepOrder
 from dex.command.commands.DexExpectWatchType import DexExpectWatchType
 from dex.command.commands.DexExpectWatchValue import DexExpectWatchValue
 from dex.command.commands.DexLabel import DexLabel
 from dex.command.commands.DexLimitSteps import DexLimitSteps
 from dex.command.commands.DexUnreachable import DexUnreachable
 from dex.command.commands.DexWatch import DexWatch
 from dex.utils import Timer
 from dex.utils.Exceptions import CommandParseError, DebuggerException
 
 def _get_valid_commands():
     """Return all top level DExTer test commands.
 
     Returns:
         { name (str): command (class) }
     """
     return {
+      DexDeclareFile.get_name() : DexDeclareFile,
       DexExpectProgramState.get_name() : DexExpectProgramState,
       DexExpectStepKind.get_name() : DexExpectStepKind,
       DexExpectStepOrder.get_name() : DexExpectStepOrder,
       DexExpectWatchType.get_name() : DexExpectWatchType,
       DexExpectWatchValue.get_name() : DexExpectWatchValue,
       DexLabel.get_name() : DexLabel,
       DexLimitSteps.get_name() : DexLimitSteps,
       DexUnreachable.get_name() : DexUnreachable,

 def skip_horizontal_whitespace(line, point):
     for idx, char in enumerate(line[point.char:]):
         if char not in ' \t':
             point.char += idx
             return
 
 
 def _find_all_commands_in_file(path, file_lines, valid_commands):
+    cmd_path = path
+    declared_files = set()
     commands = defaultdict(dict)
     paren_balance = 0
     region_start = TextPoint(0, 0)
     for region_start.line in range(len(file_lines)):
         line = file_lines[region_start.line]
         region_start.char = 0
 
         # Search this line till we find no more commands.
for region_start.line in range(len(file_lines)):
                 break  # Read next line.
 
             # Parens are balanced, we have a full command to evaluate.
             raw_text = "".join(cmd_text_list)
             try:
                 command = _build_command(
                     valid_commands[command_name],
                     raw_text,
-                    path,
+                    cmd_path,
                     cmd_point.get_lineno(),
                 )
             except SyntaxError as e:
                 # This err should point to the problem line.
                 err_point = copy(cmd_point)
                 # To e the command start is the absolute start, so use as offset.
                 err_point.line += e.lineno - 1 # e.lineno is a position, not index.
                 err_point.char += e.offset - 1 # e.offset is a position, not index.
                 raise format_parse_err(e.msg, path, file_lines, err_point)
             except TypeError as e:
                 # This err should always point to the end of the command name.
                 err_point = copy(cmd_point)
                 err_point.char += len(command_name)
                 raise format_parse_err(str(e), path, file_lines, err_point)
             else:
+                if type(command) is DexDeclareFile:
+                    cmd_path = command.declared_file
+                    if not os.path.isabs(cmd_path):
+                        source_dir = os.path.dirname(path)
+                        cmd_path = os.path.join(source_dir, cmd_path)
+                    # TODO: keep stored paths as PurePaths for 'longer'.
+                    cmd_path = str(PurePath(cmd_path))
+                    declared_files.add(cmd_path)
                 resolve_labels(command, commands)
                 assert (path, cmd_point) not in commands[command_name], (
                     command_name, commands[command_name])
                 commands[command_name][path, cmd_point] = command
 
     if paren_balance != 0:
         # This err should always point to the end of the command name.
         err_point = copy(cmd_point)
         err_point.char += len(command_name)
         msg = "Unbalanced parenthesis starting here"
         raise format_parse_err(msg, path, file_lines, err_point)
-    return dict(commands)
+    return dict(commands), declared_files
 
-def _find_all_commands(source_files):
+def _find_all_commands(test_files):
     commands = defaultdict(dict)
     valid_commands = _get_valid_commands()
-    for source_file in source_files:
-        with open(source_file) as fp:
+    new_source_files = set()
+    for test_file in test_files:
+        with open(test_file) as fp:
             lines = fp.readlines()
-        file_commands = _find_all_commands_in_file(source_file, lines,
-                                                   valid_commands)
+        file_commands, declared_files = _find_all_commands_in_file(test_file,
+                                                  lines, valid_commands)
         for command_name in file_commands:
             commands[command_name].update(file_commands[command_name])
+        new_source_files |= declared_files
 
-    return dict(commands)
+    return dict(commands), new_source_files
 
-def get_command_infos(source_files):
+def get_command_infos(test_files):
   with Timer('parsing commands'):
       try:
-          commands = _find_all_commands(source_files)
+          commands, new_source_files = _find_all_commands(test_files)
           command_infos = OrderedDict()
           for command_type in commands:
               for command in commands[command_type].values():
                   if command_type not in command_infos:
                       command_infos[command_type] = []
                   command_infos[command_type].append(command)
-          return OrderedDict(command_infos)
+          return OrderedDict(command_infos), new_source_files
       except CommandParseError as e:
           msg = 'parser error: <d>{}({}):</> {}\n{}\n{}\n'.format(
                 e.filename, e.lineno, e.info, e.src, e.caret)
           raise DebuggerException(msg)
 
 class TestParseCommand(unittest.TestCase):
     class MockCmd(CommandBase):
         """A mock DExTer command for testing parsing.
class TestParseCommand(unittest.TestCase):
 
 
     def _find_all_commands_in_lines(self, lines):
         """Use DExTer parsing methods to find all the mock commands in lines.
 
         Returns:
             { cmd_name: { (path, line): command_obj } }
         """
-        return _find_all_commands_in_file(__file__, lines, self.valid_commands)
+        cmds, declared_files = _find_all_commands_in_file(__file__, lines, self.valid_commands)
+        return cmds
 
 
     def _find_all_mock_values_in_lines(self, lines):
         """Use DExTer parsing methods to find all mock command values in lines.
 
         Returns:
             values (list(str)): MockCmd values found in lines.
         """

debuginfo-tests/dexter/dex/command/commands/DexDeclareFile.py

+# DExTer : Debugging Experience Tester
+# ~~~~~~   ~         ~~         ~   ~~
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Commmand sets the path for all following commands to 'declared_file'.
+"""
+
+from pathlib import PurePath
+
+from dex.command.CommandBase import CommandBase
+
+
+class DexDeclareFile(CommandBase):
+    def __init__(self, declared_file):
+
+        if not isinstance(declared_file, str):
+            raise TypeError('invalid argument type')
+
+        # Use PurePath to create a cannonical platform path.
+        # TODO: keep paths as PurePath objects for 'longer'
+        self.declared_file = str(PurePath(declared_file))
+        super(DexDeclareFile, self).__init__()
+
+    @staticmethod
+    def get_name():
+        return __class__.__name__
+
+    def eval(self):
+        return self.declared_file

debuginfo-tests/dexter/dex/tools/TestToolBase.py

def handle_options(self, defaults):
                     format(options.results_directory, e.strerror))
 
     def go(self) -> ReturnCode:  # noqa
         options = self.context.options
 
         options.executable = os.path.join(
             self.context.working_directory.path, 'tmp.exe')
 
+        # Test files contain dexter commands.
+        options.test_files = []
+        # Source files are to be compiled by the builder script and may also
+        # contains dexter commands.
+        options.source_files = []
         if os.path.isdir(options.test_path):
-
             subdirs = sorted([
                 r for r, _, f in os.walk(options.test_path)
                 if 'test.cfg' in f
             ])
 
             for subdir in subdirs:
-
+                for f in os.listdir(subdir):
                     # TODO: read file extensions from the test.cfg file instead so
                     # that this isn't just limited to C and C++.
-                options.source_files = [
-                    os.path.normcase(os.path.join(subdir, f))
-                    for f in os.listdir(subdir) if any(
-                        f.endswith(ext) for ext in ['.c', '.cpp'])
-                ]
+                    file_path = os.path.normcase(os.path.join(subdir, f))
+                    if f.endswith('.cpp'):
+                        options.source_files.append(file_path)
+                    elif f.endswith('.c'):
+                        options.source_files.append(file_path)
+                    elif f.endswith('.dex'):
+                        options.test_files.append(file_path)
+                # Source files can contain dexter commands too.
+                options.test_files = options.test_files + options.source_files
 
                 self._run_test(self._get_test_name(subdir))
         else:
+            # We're dealing with a direct file path to a test file. If the file is non
+            # .dex, then it must be a source file.
+            if not options.test_path.endswith('.dex'):
                 options.source_files = [options.test_path]
+            options.test_files = [options.test_path]
             self._run_test(self._get_test_name(options.test_path))
 
         return self._handle_results()
 
     @staticmethod
     def _is_current_directory(test_directory):
         return test_directory == '.'
 

debuginfo-tests/dexter/dex/tools/clang_opt_bisect/Tool.py

def handle_options(self, defaults):
                         "supported " % options.builder)
         super(Tool, self).handle_options(defaults)
 
     def _init_debugger_controller(self):
         step_collection = DextIR(
             executable_path=self.context.options.executable,
             source_paths=self.context.options.source_files,
             dexter_version=self.context.version)
-        step_collection.commands = get_command_infos(
-            self.context.options.source_files)
+        step_collection.commands, new_source_files = get_command_infos(
+            self.context.options.test_files)
+        self.context.options.source_files.extend(list(new_source_files))
         debugger_controller = DefaultController(self.context, step_collection)
         return debugger_controller
 
     def _run_test(self, test_name):  # noqa
         options = self.context.options
 
         per_pass_score = []
         current_bisect_pass_summary = defaultdict(list)

debuginfo-tests/dexter/dex/tools/test/Tool.py

def _build_test_case(self):
         return builderIR
 
     def _init_debugger_controller(self):
         step_collection = DextIR(
             executable_path=self.context.options.executable,
             source_paths=self.context.options.source_files,
             dexter_version=self.context.version)
 
-        step_collection.commands = get_command_infos(
-            self.context.options.source_files)
+        step_collection.commands, new_source_files = get_command_infos(
+            self.context.options.test_files)
+
+        self.context.options.source_files.extend(list(new_source_files))
 
         if 'DexLimitSteps' in step_collection.commands:
             debugger_controller = ConditionalController(self.context, step_collection)
         else:
             debugger_controller = DefaultController(self.context, step_collection)
 
         return debugger_controller
 
     def _get_steps(self, builderIR):

debuginfo-tests/dexter/feature_tests/commands/penalty/dex_declare_file.cpp

+// Purpose:
+//    Check that \DexDeclareFile causes a DexExpectWatchValue's to generate a
+//    missing value penalty when the declared path is incorrect.
+//
+// UNSUPPORTED: system-darwin
+//
+//
+// RUN: not %dexter_regression_test -- %s | FileCheck %s
+// CHECK: dex_declare_file.cpp
+
+int main() {
+  int result = 0;
+  return result; //DexLabel('return')
+}
+
+// DexDeclareFile('this_file_does_not_exist.cpp')
+// DexExpectWatchValue('result', 0, on_line='return')

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/commands.dex

+DexDeclareFile('test.cpp')
+DexExpectWatchValue('result', 0, on_line=14)

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/lit.local.cfg.py

+config.suffixes = ['.cpp']

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/dex_and_source/test.cpp

+// Purpose:
+//    Check that \DexDeclareFile changes the path of all succeeding commands
+//    to the file path it declares. Also check that dexter correctly accepts
+//    files with .dex extensions.
+//
+// UNSUPPORTED: system-darwin
+//
+//
+// RUN: %dexter_regression_test -- %S | FileCheck %s
+// CHECK: dex_and_source
+
+int main() {
+  int result = 0;
+  return result;
+}

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary/commands.dex

+# Purpose:
+#    Check that \DexDeclareFile's file declaration can reference source files
+#    in a precompiled binary.
+#
+# UNSUPPORTED: system-darwin
+#
+# RUN: %clang %S/test.cpp -O0 -g -o %t
+# RUN: %dexter_regression_test --binary %t %s | FileCheck %s
+# CHECK: commands.dex
+#
+# test.cpp
+# 1. int main() {
+# 2.   int result = 0;
+# 3.   return result;
+# 4. }
+
+DexDeclareFile('test.cpp')
+DexExpectWatchValue('result', 0, on_line=3)

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary/lit.local.cfg.py

+config.suffixes = ['.dex']

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary/test.cpp

+int main() {
+  int result = 0;
+  return result;
+}

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary_different_dir/dex_commands/commands.dex

+# Purpose:
+#    Check that \DexDeclareFile's file declaration can reference source files
+#    not included in the test directory 
+#
+# UNSUPPORTED: system-darwin
+#
+# RUN: %clang %S/../source/test.cpp -O0 -g -o %t
+# RUN: %dexter_regression_test --binary %t %s | FileCheck %s
+# RUN: rm %t
+# CHECK: commands.dex
+#
+# test.cpp
+# 1. int main() {
+# 2.   int result = 0;
+# 3.   return result;
+# 4. }
+
+DexDeclareFile('../source/test.cpp')
+DexExpectWatchValue('result', 0, on_line=3)

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary_different_dir/lit.local.cfg.py

+config.suffixes = ['.dex']

debuginfo-tests/dexter/feature_tests/commands/perfect/dex_declare_file/precompiled_binary_different_dir/source/test.cpp

+int main() {
+  int result = 0;
+  return result;
+}