WIP: [lit] Enable debugging lit tests with python's pdb

[jdenny]

Summary

This patch enables using python's default debugger, pdb, to interactively step through and debug directives in lit tests, even existing tests that do *not* use PYTHON directives (proposed in D154987). For example, let's say we have a test llvm/test/example.txt:

$ LIT_OPTS=--pdb LIT_FILTER=example.txt ninja check-llvm
[0/1] Running the LLVM regression tests
-- Testing: 1 of 49248 tests, 0 workers --
******************** STARTED DEBUGGING 'LLVM :: example.txt' ********************
# started executing 'PYTHON:' directive from line 1 to 8

# At the pdb prompt, you can type:
#
# - 'q', press enter, and then immediately hit Ctrl-C to terminate lit.
# - '!help(lit)' for help with the 'lit' object.
# - 'help' for help with other pdb commands.
#
# All RUN, DEFINE, and REDEFINE directives are automatically converted to
# equivalent PYTHON directives for execution by the debugger.

> /home/jdenny/llvm-project/llvm/test/example.txt(1)<module>()
-> ; DEFINE: %{msg} =
(Pdb) list
  1  -> ; DEFINE: %{msg} =
  2     ; DEFINE: %{greet} = echo %{msg}
  3
  4     ; REDEFINE: %{msg} = hello world
  5     ; RUN: %{greet}
  6
  7     ; REDEFINE: %{msg} = goodby world
  8     ; RUN: %{greet}
[EOF]
(Pdb) next
> /home/jdenny/llvm-project/llvm/test/example.txt(2)<module>()
-> ; DEFINE: %{greet} = echo %{msg}
(Pdb) next
> /home/jdenny/llvm-project/llvm/test/example.txt(4)<module>()
-> ; REDEFINE: %{msg} = hello world
(Pdb) next
> /home/jdenny/llvm-project/llvm/test/example.txt(5)<module>()
-> ; RUN: %{greet}
(Pdb) p lit.expand("%{greet}")
'echo hello world'
(Pdb) next
# lit.run called from /home/jdenny/llvm-project/llvm/test/example.txt:5
#         called from /usr/lib/python3.8/bdb.py:587
#         called from /usr/lib/python3.8/pdb.py:1595
echo hello world
# executed command: echo hello world
# .---command stdout------------
# | hello world
# `-----------------------------
> /home/jdenny/llvm-project/llvm/test/example.txt(7)<module>()
-> ; REDEFINE: %{msg} = goodby world
(Pdb) next
> /home/jdenny/llvm-project/llvm/test/example.txt(8)<module>()
-> ; RUN: %{greet}
(Pdb) p lit.expand("%{greet}")
'echo goodby world'
(Pdb) p lit.expand("%{msg}")
'goodby world'
(Pdb) !lit.redefine("%{msg}", "goodbye world")
(Pdb) p lit.expand("%{greet}")
'echo goodbye world'
(Pdb) next
# lit.run called from /home/jdenny/llvm-project/llvm/test/example.txt:8
#         called from /usr/lib/python3.8/bdb.py:587
#         called from /usr/lib/python3.8/pdb.py:1595
echo goodbye world
# executed command: echo goodbye world
# .---command stdout------------
# | goodbye world
# `-----------------------------
--Return--
> /home/jdenny/llvm-project/llvm/test/example.txt(8)<module>()->None
-> ; RUN: %{greet}
(Pdb) next
# finished executing 'PYTHON:' directive from line 1 to 8
******************** FINISHED DEBUGGING 'LLVM :: example.txt' ********************

Testing Time: 87.06s
  Excluded: 48678
  Passed  :     1

A fundamental component of this patch is that, when lit is passed the --pdb command-line option, lit treats RUN, DEFINE, and REDEFINE directives as if they were PYTHON directives with corresponding calls to the lit object. That's why lit claims it's executing a PYTHON directive. Of course, --pdb also works fine for tests that actually use PYTHON directives or config.prologue.

(For the above example, I configured the test suite to use lit's internal shell in order to improve the execution trace, but I have found bash works fine here too.)

This patch also introduces lit.define(name, value) and lit.redefine(name, value) functions in order to support DEFINE and REDEFINE directives. Those are likely useful on their own, and I'll probably extract them into a parent patch later and improve them.

This patch is a WIP and is not ready for a detailed review. First, I still need to extend the documentation and lit's test suite. Second, I wonder if the patch should be generalized to work with other python debuggers besides pdb, but I haven't given that idea enough thought yet. I am sharing this patch early to demonstrate another benefit of extending lit with PYTHON directives, as proposed in D154987.

[jdenny]

Rebased onto the current D154987. See its summary for how to apply it successfully first.

Added a brief help message before the pdb prompt, and updated example in commit log.

Applied python formatting suggestions from darker.

[jdenny]

Rebased onto latest D154987. Improved help message a little.

llvm/utils/lit/lit/LitConfig.py

def __init__(
         debug,
         isWindows,
         order,
         params,
         config_prefix=None,
         maxIndividualTestTime=0,
         parallelism_groups={},
         per_test_coverage=False,
+        pdb=False,
     ):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
         self.noExecute = noExecute
         self.debug = debug
+        self.pdb = pdb
         self.isWindows = bool(isWindows)
         self.order = order
         self.params = dict(params)
         self.bashPath = None
 
         # Configuration files to look for when discovering test suites.
         self.config_prefix = config_prefix or "lit"
         self.suffixes = ["cfg.py", "cfg"]

llvm/utils/lit/lit/TestRunner.py

for f in named_temp_files:
         except OSError:
             pass
 
     if cmd.negate:
         exitCode = not exitCode
 
     return exitCode
 
-
 def formatOutput(title, data, limit=None):
     if not data.strip():
         return ""
     if not limit is None and len(data) > limit:
         data = data[:limit] + "\n...\n"
         msg = "data was truncated"
     else:
         msg = ""
def get_location(self):
         """
         Get a phrase describing the line or range of lines so far included by
         this directive and any line continuations.
         """
         if self.start_line_number == self.end_line_number:
             return f"at line {self.start_line_number}"
         return f"from line {self.start_line_number} to {self.end_line_number}"
 
+    def asPythonDirective(self):
+        """
+        Return a PythonDirective equivalent to this directive, or just return
+        this directive if it's already a PythonDirective.
+        """
+        assert False, "expected method to be called on derived class"
+
+    def continuePythonDirective(self, pythonDir):
+        """
+        Append lines to pythonDir that are equivalent to this directive.  Each
+        implementation of this function should probably call
+        pythonDir.doNotRequireIndentation() as its first step.
+        """
+        assert False, "expected method to be called on derived class"
+
 
 class CommandDirective(ExpandableScriptDirective):
     """
     A lit directive taking a shell command line.  For example,
     'RUN: echo hello world'.
 
     command: The content accumulated so far from the directive and its
         continuation lines.
def add_continuation(self, line_number, keyword, line):
         self.end_line_number = line_number
         return True
 
     def needs_continuation(self):
         # Trailing whitespace is stripped immediately when each line is added,
         # so '\' is never hidden here.
         return self.command[-1] == "\\"
 
+    def _makePythonLitRun(self):
+        match = re.match(kPdbgRegex, self.command)
+        cmd = match.group(2) if match else self.command
+        return f"lit.run({repr(cmd)})"
+
+    def _addBlankLinesToPythonDirective(self, pythonDir):
+        for i in range(self.start_line_number + 1, self.end_line_number + 1):
+            res = pythonDir.add_continuation(i, pythonDir.keyword, "")
+            assert res, "expected success adding blank line as PYTHON continuation"
+
+    def asPythonDirective(self):
+        pythonDir = PythonDirective(
+            self.start_line_number,
+            self.start_line_number,
+            PythonDirective.getDefaultKeyword(),
+            self._makePythonLitRun(),
+        )
+        self._addBlankLinesToPythonDirective(pythonDir)
+        return pythonDir
+
+    def continuePythonDirective(self, pythonDir):
+        pythonDir.doNotRequireIndentation()
+        res = pythonDir.add_continuation(
+            self.start_line_number, pythonDir.keyword, self._makePythonLitRun()
+        )
+        assert res, "expected success adding RUN line as PYTHON continuation"
+        self._addBlankLinesToPythonDirective(pythonDir)
+
 
 class SubstDirective(ExpandableScriptDirective):
     """
     A lit directive taking a substitution definition or redefinition.  For
     example, 'DEFINE: %{name} = value'.
 
     new_subst: True if this directive defines a new substitution.  False if it
         redefines an existing substitution.
def _parse_body(self):
         if not re.fullmatch(r"%{[_a-zA-Z][-_:0-9a-zA-Z]*}", self.name):
             raise ValueError(
                 f"Substitution name '{self.name}' is malformed as it must "
                 f"start with '%{{', it must end with '}}', and the rest must "
                 f"start with a letter or underscore and contain only "
                 f"alphanumeric characters, hyphens, underscores, and colons"
             )
 
-    def adjust_substitutions(self, substitutions):
-        """
-        Modify the specified substitution list as specified by this directive.
-        """
+    def _makePythonLitCall(self):
+        name = repr(self.name)
+        value = repr(self.value)
+        func = "define" if self.new_subst else "redefine"
+        return f"lit.{func}({name}, {value})"
+
+    def _addBlankLinesToPythonDirective(self, pythonDir):
+        for i in range(self.start_line_number + 1, self.end_line_number + 1):
+            res = pythonDir.add_continuation(i, pythonDir.keyword, "")
+            assert res, "expected success adding blank line as PYTHON continuation"
+
+    def asPythonDirective(self):
+        pythonDir = PythonDirective(
+            self.start_line_number,
+            self.start_line_number,
+            PythonDirective.getDefaultKeyword(),
+            self._makePythonLitCall(),
+        )
+        self._addBlankLinesToPythonDirective(pythonDir)
+        return pythonDir
+
+    def continuePythonDirective(self, pythonDir):
+        pythonDir.doNotRequireIndentation()
+        res = pythonDir.add_continuation(
+            self.start_line_number, pythonDir.keyword, self._makePythonLitCall()
+        )
         assert (
-            not self.needs_continuation()
-        ), "expected directive continuations to be parsed before applying"
-        value_repl = self.value.replace("\\", "\\\\")
-        existing = [i for i, subst in enumerate(substitutions) if self.name in subst[0]]
+            res
+        ), "expected success adding DEFINE/REDEFINE line as PYTHON continuation"
+        self._addBlankLinesToPythonDirective(pythonDir)
+
+    @staticmethod
+    def applySubstitution(substitutions, what, newSubst, name, value):
+        value_repl = value.replace("\\", "\\\\")
+        existing = [i for i, subst in enumerate(substitutions) if name in subst[0]]
         existing_res = "".join(
             "\nExisting pattern: " + substitutions[i][0] for i in existing
         )
-        if self.new_subst:
+        if newSubst:
             if existing:
                 raise ValueError(
-                    f"Substitution whose pattern contains '{self.name}' is "
-                    f"already defined before '{self.keyword}' directive "
-                    f"{self.get_location()}"
+                    f"Substitution whose pattern contains '{name}' is already "
+                    f"defined before {what}"
                     f"{existing_res}"
                 )
-            substitutions.insert(0, (self.name, value_repl))
+            substitutions.insert(0, (name, value_repl))
             return
         if len(existing) > 1:
             raise ValueError(
-                f"Multiple substitutions whose patterns contain '{self.name}' "
-                f"are defined before '{self.keyword}' directive "
-                f"{self.get_location()}"
+                f"Multiple substitutions whose patterns contain '{name}' are "
+                f"defined before {what}"
                 f"{existing_res}"
             )
         if not existing:
+            raise ValueError(f"No substitution for '{name}' is defined before {what}")
+        if substitutions[existing[0]][0] != name:
             raise ValueError(
-                f"No substitution for '{self.name}' is defined before "
-                f"'{self.keyword}' directive {self.get_location()}"
-            )
-        if substitutions[existing[0]][0] != self.name:
-            raise ValueError(
-                f"Existing substitution whose pattern contains '{self.name}' "
-                f"does not have the pattern specified by '{self.keyword}' "
-                f"directive {self.get_location()}\n"
-                f"Expected pattern: {self.name}"
+                f"Existing substitution whose pattern contains '{name}' does "
+                f"not have the pattern specified by {what}\n"
+                f"Expected pattern: {name}"
                 f"{existing_res}"
             )
-        substitutions[existing[0]] = (self.name, value_repl)
+        substitutions[existing[0]] = (name, value_repl)
+
+    def apply(self, substitutions):
+        """
+        Modify the specified substitution list as specified by this directive.
+        """
+        assert (
+            not self.needs_continuation()
+        ), "expected directive continuations to be parsed before applying"
+        what = f"'{self.keyword}' directive {self.get_location()}"
+        self.applySubstitution(
+            substitutions, what, self.new_subst, self.name, self.value
+        )
 
 
 class PythonDirective(ExpandableScriptDirective):
     """
     A lit directive taking a python statement.  For example,
     'PYTHON: lit.run(cmd)'.
 
     indent: The indentation found on the first line and expected on continuation
         lines.  It must be stripped for python to compile the code.
     body: The uncompiled code accumulated so far from the directive and its
         continuation lines.
     """
 
+    @staticmethod
+    def getDefaultKeyword():
+        return "PYTHON:"
+
     def __init__(self, start_line_number, end_line_number, keyword, line):
         super().__init__(start_line_number, end_line_number, keyword)
         # Add blank lines so python diagnostics produce correct line numbers.
         self.body = (start_line_number - 1) * "\n"
         # Determine the indentation and remove it so that it's valid python.
         line_stripped = line.lstrip()
         self.indent = line[0 : len(line) - len(line_stripped)]
         self.body += line_stripped + "\n"
def needs_continuation(self):
         # trailing '\\', sometimes you cannot tell whether a python directive is
         # continued until you see the indentation of the next python directive.
         # In general, lit shouldn't reimplement python's line continuation and
         # block syntax handling.  Thus, lit just concatenates all consecutive
         # python directives and compiles them together even if they could be
         # compiled separately.
         return False
 
+    def doNotRequireIndentation(self):
+        """
+        Do not require indentation on future continuation lines.  This is useful
+        when merging PYTHON blocks such that, in each, the indentation from the
+        first line was already verified as present and was already stripped from
+        subsequent lines.  Normal python indentation is still required by the
+        python interprerter.
+        """
+        self.indent = ""
+
+    def asPythonDirective(self):
+        return self
+
+    def continuePythonDirective(self, pythonDir):
+        pythonDir.doNotRequireIndentation()
+        for i, line in enumerate(self.body.splitlines()):
+            lineNumber = i + 1
+            if lineNumber < self.start_line_number:
+                assert line == "", f"expected blank line before start line"
+                continue
+            res = pythonDir.add_continuation(lineNumber, self.keyword, line)
+            assert res, "expected success merging PYTHON directives"
+
     @staticmethod
-    def executePython(what, filename, pythonCode, python_dict, log):
+    def executePython(what, filename, pythonCode, python_dict, log, usePdb):
         """
         Execute python code passed as an argument.
         """
         python_dict["__file__"] = filename = os.path.abspath(filename)
         try:
             c = compile(pythonCode, filename, "exec")
         except Exception as e:
             # Report the exception without this stack frame.  Bothering the lit
             # user with lit internals is just a distraction.
             tb = e.__traceback__.tb_next
             traceText = "".join(traceback.format_exception(None, e, tb))
             raise ScriptFatal(f"error compiling {what}:\n{traceText}")
+        if usePdb:
+            print(f"# started executing {what}")
+        else:
             log(f"# executed {what}\n")
             outIO = StringIO()
             errIO = StringIO()
         traceText = ""
         try:
-            with contextlib.redirect_stdout(outIO), contextlib.redirect_stderr(errIO):
+            if usePdb:
+                print(
+                    """
+# At the pdb prompt, you can type:
+# - 'q', press enter, and then immediately hit Ctrl-C to terminate lit.
+# - '!help(lit)' for help with the 'lit' object.
+# - 'help' for help with other pdb commands.
+"""
+                )
+                import pdb, time
+
+                pdb.run(c, python_dict)
+                # Give the user a little time to hit Ctrl-C before pdb.run is
+                # reached again and starts intercepting Ctrl-C again.
+                time.sleep(1)
+            else:
+                with contextlib.redirect_stdout(outIO), contextlib.redirect_stderr(
+                    errIO
+                ):
                     exec(c, python_dict)
         except ScriptFail:
             raise
         except Exception as e:
             # Report the exception without this stack frame.  Bothering the lit
             # user with lit internals is just a distraction.
             tb = e.__traceback__.tb_next
             traceText = "".join(traceback.format_exception(None, e, tb))
+            import bdb
+
+            if isinstance(e, bdb.BdbQuit):
+                traceText += """
+Note: It looks like you are trying to use a python debugger, like pdb.
+Note: Try passing --pdb to lit, perhaps via the LIT_OPTS environment variable.
+"""
             raise ScriptFail(1, None)
         finally:
+            if usePdb:
+                print(traceText, end="")
+                print(f"# finished executing {what}")
+            else:
                 outText = outIO.getvalue()
                 errText = errIO.getvalue() + traceText
                 log(formatOutput(f"stdout from {what}", outText))
                 log(formatOutput(f"stderr from {what}", errText))
 
-    def execute(self, filename, python_dict, log):
+    def execute(self, filename, python_dict, log, usePdb):
         """
         Execute the python code in this directive.
         """
         self.executePython(
             f"'{self.keyword}' directive {self.get_location()}",
             filename,
             self.body,
             python_dict,
             log,
+            usePdb,
         )
 
 
 def applySubstitutions(script, substitutions, conditions={}, recursion_limit=None):
     """
     Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln.
def processLineToFixedPoint(ln):
             )
 
         return processed
 
     process = processLine if recursion_limit is None else processLineToFixedPoint
     output = []
     for directive in script:
         if isinstance(directive, SubstDirective):
-            directive.adjust_substitutions(substitutions)
+            directive.apply(substitutions)
         else:
             if isinstance(directive, CommandDirective):
                 line = directive.command
             else:
                 # Can come from preamble_commands or from lit.expand.
                 assert isinstance(directive, str)
                 line = directive
             output.append(unescapePercents(process(line)))

 # that accepts a list of strings and returns the same list with lit
 # substitutions expanded.
 def _runShTest(
     test,
     litConfig,
     useExternalSh,
     script,
     tmpBase,
+    substitutions=[],
     substitutionApplier=lambda x: x,
 ):
     # Make one attempt to run the test.
     def runOnce(execdir):
         # Counter for litConfig.per_test_coverage file names.
         perTestCoverageIndex = 0
 
         # scriptPart has the same constraints as script for _runShTest except
def runOnce(execdir):
             out, err, exitCode, timeoutInfo = res
             addStdout(out)
             addStderr(err)
             if exitCode != 0:
                 raise ScriptFail(exitCode, timeoutInfo)
 
         class PythonDirectiveLitAPI(object):
             """
-            Class for the 'lit' object accessible in 'PYTHON:' directives and in
-            config.prologue.
+            Class for the 'lit' object accessible in 'PYTHON:' directives, in
+            config.prologue, and at the pdb prompt produced by lit's '--pdb' option.
             """
 
             # All members of PythonDirectiveLitAPI are exposed within 'PYTHON:'
             # directives.  Symbols only required for lit iternals should be declared
             # outside it.
 
-            # TODO: In the future, extend the 'lit' API to be able to *write*
-            # substitutions?
-
             @staticmethod
             def has(feature):
                 """
                 Check if config.available_features indicates a feature is enabled.
                 """
                 return feature in test.config.available_features
 
             @staticmethod
             def expand(text):
                 """
                 Expand substitutions in 'text' with their most recent values (from,
                 for example, the most recent 'DEFINE:' and 'REDEFINE:' directives),
                 and return the result.
                 """
                 return substitutionApplier([text])[0]
 
             @staticmethod
+            def define(name, value):
+                """
+                Define new substitution as if it appeared in a 'DEFINE:' directive.
+                """
+                try:
+                    what = "lit.define call"
+                    SubstDirective.applySubstitution(
+                        substitutions, what, True, name, value
+                    )
+                except ValueError as e:
+                    raise ScriptFatal(str(e)) from None
+
+            @staticmethod
+            def redefine(name, value):
+                """
+                Redefine existing substitution as if it appeared in a 'REDEFINE:'
+                directive.
+                """
+                try:
+                    what = "lit.redefine call"
+                    SubstDirective.applySubstitution(
+                        substitutions, what, False, name, value
+                    )
+                except ValueError as e:
+                    raise ScriptFatal(str(e)) from None
+
+            @staticmethod
             def run(cmd):
                 """
                 Execute 'cmd' as if it appeared in a 'RUN:' directive.
                 """
                 msg = ""
                 # In the execution trace, report the call stack for this lit.run
                 # call.  However, skip this stack frame, and then climb the call
                 # stack, adding each frame, until we've reached this file again,
def runOnce(execdir):
         python_dict["lit"] = PythonDirectiveLitAPI
         try:
             # Execute the python prologue, if any.
             prologue = test.config.prologue
             if prologue:
                 with open(prologue) as f:
                     what = f"config.prologue='{prologue}'"
                     PythonDirective.executePython(
-                        what, prologue, f.read(), python_dict, addStdout
+                        what, prologue, f.read(), python_dict, addStdout, litConfig.pdb
                     )
             # Execute the script.
             scriptRemaining = script
             while scriptRemaining:
                 # Execute all leading python directives.
                 for pythonDirIndex, pythonDir in enumerate(scriptRemaining):
                     if not isinstance(pythonDir, PythonDirective):
                         break
-                    pythonDir.execute(test.getSourcePath(), python_dict, addStdout)
+                    pythonDir.execute(
+                        test.getSourcePath(), python_dict, addStdout, litConfig.pdb
+                    )
                 else:
                     scriptRemaining = []
                     break
                 scriptRemaining = scriptRemaining[pythonDirIndex:]
                 # Extract all directives before the next python directive into
                 # a script fragment.
                 pythonDirIndex = next(
                     (
):
 
     # Create the output directory if it does not already exist.
     lit.util.mkdir_p(os.path.dirname(tmpBase))
 
     # Re-run failed tests up to test.allowed_retries times.
     execdir = os.path.dirname(test.getExecPath())
     attempts = test.allowed_retries + 1
     for i in range(attempts):
+        if litConfig.pdb:
+            print(f"{'*' * 20} STARTED DEBUGGING '{test.getFullName()}' {'*' * 20}")
         res = runOnce(execdir)
+        if litConfig.pdb:
+            print(f"{'*' * 20} FINISHED DEBUGGING '{test.getFullName()}' {'*' * 20}")
         out, err, exitCode, timeoutInfo, status = res
         if status != Test.FAIL:
             break
 
     # If we had to run the test more than once, count it as a flaky pass. These
     # will be printed separately in the test summary.
     if i > 0 and status == Test.PASS:
         status = Test.FLAKYPASS
):
     def substitutionApplier(lines):
         return applySubstitutions(
             lines,
             substitutions,
             conditions,
             recursion_limit=test.config.recursiveExpansionLimit,
         )
 
+    # If --pdb, convert RUN, DEFINE, and REDEFINE directives to lit object
+    # calls in PYTHON directives so they can be tracked by pdb.  We try to put
+    # them all in the same PYTHON block so they're within the same pdb.run
+    # call.
+    if litConfig.pdb:
+        oldScript = script
+        script = []
+        for cmd in oldScript:
+            if isinstance(cmd, str):
+                # We don't attempt to support debugging of preamble commands.
+                # They have no source file location.
+                script.append(cmd)
+            else:
+                assert isinstance(cmd, ExpandableScriptDirective)
+                if not script or not isinstance(script[-1], PythonDirective):
+                    script.append(cmd.asPythonDirective())
+                else:
+                    cmd.continuePythonDirective(script[-1])
+
     return _runShTest(
         test,
         litConfig,
         useExternalSh,
         script,
         tmpBase,
+        substitutions=substitutions,
         substitutionApplier=substitutionApplier,
     )

llvm/utils/lit/lit/cl_arguments.py

def parse_args():
     debug_group.add_argument(
         "--show-tests", help="Show all discovered tests and exit", action="store_true"
     )
     debug_group.add_argument(
         "--show-used-features",
         help="Show all features used in the test suite (in XFAIL, UNSUPPORTED and REQUIRES) and exit",
         action="store_true",
     )
+    debug_group.add_argument(
+        "--pdb",
+        help="Enable debugging lit tests with python's standard debugger, pdb",
+        action="store_true",
+    )
 
     # LIT is special: environment variables override command line arguments.
     env_args = shlex.split(os.environ.get("LIT_OPTS", ""))
     args = sys.argv[1:] + env_args
     opts = parser.parse_args(args)
 
     # Validate command line options
     if opts.incremental:

llvm/utils/lit/lit/display.py

 import sys
 
 
 def create_display(opts, tests, total_tests, workers):
     if opts.quiet:
         return NopDisplay()
 
     num_tests = len(tests)
     of_total = (" of %d" % total_tests) if (num_tests != total_tests) else ""
     header = "-- Testing: %d%s tests, %d workers --" % (num_tests, of_total, workers)
 
     progress_bar = None
-    if opts.succinct and opts.useProgressBar:
+    if opts.succinct and opts.useProgressBar and not opts.pdb:
         import lit.ProgressBar
 
         try:
             tc = lit.ProgressBar.TerminalController()
             progress_bar = lit.ProgressBar.ProgressBar(tc, header)
             header = None
         except ValueError:
             progress_bar = lit.ProgressBar.SimpleProgressBar("Testing: ")

llvm/utils/lit/lit/main.py

lit_config = lit.LitConfig.LitConfig(
         progname=os.path.basename(sys.argv[0]),
         path=opts.path,
         quiet=opts.quiet,
         useValgrind=opts.useValgrind,
         valgrindLeakCheck=opts.valgrindLeakCheck,
         valgrindArgs=opts.valgrindArgs,
         noExecute=opts.noExecute,
         debug=opts.debug,
+        pdb=opts.pdb,
         isWindows=is_windows,
         order=opts.order,
         params=params,
         config_prefix=opts.configPrefix,
         per_test_coverage=opts.per_test_coverage,
     )
 
     discovered_tests = lit.discovery.find_tests_for_inputs(
def mark_excluded(discovered_tests, selected_tests):
     excluded_tests = set(discovered_tests) - set(selected_tests)
     result = lit.Test.Result(lit.Test.EXCLUDED)
     for t in excluded_tests:
         t.setResult(result)
 
 
 def run_tests(tests, lit_config, opts, discovered_tests):
     workers = min(len(tests), opts.workers)
+    timeout = opts.timeout
+    if lit_config.pdb:
+        if workers != 1:
+            lit_config.warning(
+                "Ignoring requested parallelism because debugging with pdb was "
+                "requested"
+            )
+        # workers = 1 still creates a subprocess, which prevents pdb from
+        # successfully debugging tests.
+        workers = 0
+        if timeout:
+            lit_config.warning(
+                "Ignoring timeout because debugging with pdb was requested"
+            )
+            timeout = None
+
     display = lit.display.create_display(opts, tests, discovered_tests, workers)
 
     run = lit.run.Run(
-        tests, lit_config, workers, display.update, opts.max_failures, opts.timeout
+        tests, lit_config, workers, display.update, opts.max_failures, timeout
     )
 
     display.print_header()
 
     interrupted = False
     error = None
     try:
         execute_in_tmp_dir(run, lit_config)

llvm/utils/lit/lit/run.py

def __init__(
         self, tests, lit_config, workers, progress_callback, max_failures, timeout
     ):
         self.tests = tests
         self.lit_config = lit_config
         self.workers = workers
         self.progress_callback = progress_callback
         self.max_failures = max_failures
         self.timeout = timeout
-        assert workers > 0
+        assert workers >= 0
 
     def execute(self):
         """
         Execute the tests in the run using up to the specified number of
         parallel tasks, and inform the caller of each individual result. The
         provided tests should be a subset of the tests available in this run
         object.
 
         The progress_callback will be invoked for each completed test.
 
         If timeout is non-None, it should be a time in seconds after which to
         stop executing tests.
 
+        If workers is 0, tests will be executed sequentially in the same
+        process, and timeout must be None.
+
         Returns the elapsed testing time.
 
         Upon completion, each test in the run will have its result
         computed. Tests which were not actually executed (for any reason) will
         be marked SKIPPED.
         """
         self.failures = 0
 
+        assert (
+            self.workers > 0 or not self.timeout
+        ), "expected at least one worker to be able to apply timeout"
+
         # Larger timeouts (one year, positive infinity) don't work on Windows.
         one_week = 7 * 24 * 60 * 60  # days * hours * minutes * seconds
         timeout = self.timeout or one_week
         deadline = time.time() + timeout
 
         try:
             self._execute(deadline)
         finally:
             skipped = lit.Test.Result(lit.Test.SKIPPED)
             for test in self.tests:
                 if test.result is None:
                     test.setResult(skipped)
 
     def _execute(self, deadline):
+        if self.workers == 0:
+            for test in self.tests:
+                result = lit.worker._execute(test, self.lit_config)
+                test.setResult(result)
+                self.progress_callback(test)
+            return
+
         self._increase_process_limit()
 
         semaphores = {
             k: multiprocessing.BoundedSemaphore(v)
             for k, v in self.lit_config.parallelism_groups.items()
             if v is not None
         }
 

llvm/utils/lit/lit/worker.py

 
 def _get_parallelism_semaphore(test):
     pg = test.config.parallelism_group
     if callable(pg):
         pg = pg(test)
     return _parallelism_semaphores.get(pg, NopSemaphore())
 
 
-# Do not inline! Directly used by LitTestCase.py
+# Do not inline! Directly used by LitTestCase.py and run.py
 def _execute(test, lit_config):
     start = time.time()
     result = _execute_test_handle_errors(test, lit_config)
     result.elapsed = time.time() - start
     result.start = start
     result.pid = os.getpid()
     return result
 
 
 def _execute_test_handle_errors(test, lit_config):
     try:
         result = test.config.test_format.execute(test, lit_config)
         return _adapt_result(result)
-    except:
+    except BaseException as e:
+        # lit_config.pdb disables multiprocessing and thus graceful handling of
+        # Ctrl-C.  Let KeyboardInterrupt exceptions propagate from here so the
+        # user can easily terminate lit.
+        if lit_config.pdb and isinstance(e, KeyboardInterrupt):
+            raise
         if lit_config.debug:
             raise
         output = "Exception during script execution:\n"
         output += traceback.format_exc()
         output += "\n"
         return lit.Test.Result(lit.Test.UNRESOLVED, output)
 
 
 # Support deprecated result from execute() which returned the result
 # code and additional output as a tuple.
 def _adapt_result(result):
     if isinstance(result, lit.Test.Result):
         return result
     assert isinstance(result, tuple)
     code, output = result
     return lit.Test.Result(code, output)