[Reproducers] Add a tool to transparently capture and replay lldb sessions

[JDevlieghere]

Summary

This patch introduces a small new utility (lldb-repro) to transparently capture and replay debugger sessions from the driver. This makes it possible to test the reproducers by running the test suite twice, once to capture a reproducer for every lldb invocation, and then running the test suite again that replays the previously recorded session.

$ ./bin/llvm-lit ../llvm-project/lldb/test/Shell/ -sv --param=lldb-run-with-repro=capture
$ ./bin/llvm-lit ../llvm-project/lldb/test/Shell/ -sv --param=lldb-run-with-repro=replay

[labath]

This seems like it could be useful. Some random questions inline..

lldb/test/Shell/lit.cfg.py

42–57

The difference between these two is super-unclear. Any chance to unify them?

lldb/tools/lldb-repro/lldb-repro.cpp

23 ↗

(On Diff #238413)

Is the r in the name intentional?

63 ↗

(On Diff #238413)

I am wondering if this shouldn't even be the default behavior. If I already passed --capture to lldb then it does not seem unreasonable to always write it out when lldb exits (we already do it after a crash, right?)

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

Are you sure this will work fine with multi-config generators? You might be better off just relying on the fact that the lldb executable will sit right next to this binary...

[labath]

Considered making this a python script?

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

Actually how, is this thing going to be invoked exactly? Couldn't the path to lldb be passed simply as argv[1]?

[JDevlieghere]

In D72823#1824152, @labath wrote:

Considered making this a python script?

I think that makes sense

lldb/test/Shell/lit.cfg.py

42–57

Not really, one is for the tool, the other is to enable reproducer capture by default. They're totally orthogonal, but I agree it's confusing.

lldb/tools/lldb-repro/lldb-repro.cpp

23 ↗

(On Diff #238413)

Nope, this is a typo

63 ↗

(On Diff #238413)

My ultimate goal is to have reproducers enabled by default. The capture flag is just away to make that behavior opt-in for now. You might use it to get a reproducer on a crash, but you don't necessary want to keep all reproducers around otherwise.

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

It just needs patching up like lldb-dotest and lit. Assuming you mean argv[0], it think we could make that work if I replace "%lldb" with a path to lldb-repro.

[aprantl]

lldb/test/Shell/lit.cfg.py

42–57

How about adding this as a comment here?

lldb/tools/lldb-repro/lldb-repro.cpp

25 ↗

(On Diff #238413)

hash_combine()?

lldb/tools/lldb-repro/lldb-repro.h.cmake

5 ↗

(On Diff #238413)

Out of curiosity, is the Copyright 2020 line part of the recommended default file header?

[aprantl]

lldb/tools/lldb-repro/lldb-repro.h.cmake

5 ↗

(On Diff #238413)

To answer my own question: https://www.llvm.org/docs/CodingStandards.html#file-headers

No, that's a Swift-lldb-ism. That's great because those lines are annoying to keep up to date anyway.

[JDevlieghere]

• Convert to Python
• Integrate with lit

[aprantl]

lldb/utils/lldb-repro/lldb-repro.py

2

Could we add a top-level comment (and/or a Usage: message) that explains what this tool is used for? That's not really obvious for someone who hasn't read the commit message.

[JDevlieghere]

Add a comment explaining the purpose of the utility.

[aprantl]

lldb/utils/lldb-repro/lldb-repro.py

49

Thanks! Sorry for being such a nuisance, but I think this should be a doc-comment (""") at the top of the file?

[JDevlieghere]

lldb/utils/lldb-repro/lldb-repro.py

49

No worries, totally fair!

[labath]

lldb/test/Shell/helper/toolchain.py

13

Maybe _get_lldb_init_path? This way it looks like this function is doing some fancy initialization..

lldb/test/Shell/lit.cfg.py

42–57

Yes, that's better now, though, per the other comment, I think the env vars can be removed completely.

lldb/tools/lldb-repro/lldb-repro.cpp

63 ↗

(On Diff #238413)

Ok, that makes sense. That said, I think we will always want a way to disable this -- this just looks too much like one of those "this shit is tracking everything I do" features a lot of people will want to stay away from.

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

No, I really meant argv[1]. :)

The idea was that %lldb would expand to /src/path/to/lldb-repro.py /build/path/to/lldb.exe --whatever. That way, you wouldn't need to rely on the "same directory" trick and could get rid of all the cmake code. In fact, we could even throw in a --capture/--replay argument to the command line, and ditch the environment variables too...

[JDevlieghere]

• Remove environment variables

[JDevlieghere]

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

I like the idea but FindTool is a class that's resolved by lit, and the arguments are strings. So I kept the current approach that expects to find lldb next to lldb-repro.

[labath]

lldb/tools/lldb-repro/lldb-repro.h.cmake

12 ↗

(On Diff #238413)

Ah, yes, that does make things tricky.. Yeah, having both things in the same directory seems fine...

lldb/utils/lldb-repro/lldb-repro.py

25

I guess you don't need the /path/to/lldb part then?

[JDevlieghere]

lldb/utils/lldb-repro/lldb-repro.py

25

Right, forgot to undo that change again. Thx!

lldb/CMakeLists.txt

 add_subdirectory(docs)
 
 option(LLDB_INCLUDE_TESTS "Generate build targets for the LLDB unit tests." ${LLVM_INCLUDE_TESTS})
 if(LLDB_INCLUDE_TESTS)
   add_subdirectory(test)
   add_subdirectory(unittests)
   add_subdirectory(utils/lit-cpuid)
   add_subdirectory(utils/lldb-dotest)
+  add_subdirectory(utils/lldb-repro)
 endif()
 
 if (LLDB_ENABLE_PYTHON)
   get_target_property(lldb_bindings_dir swig_wrapper BINARY_DIR)
 
   if(LLDB_BUILD_FRAMEWORK)
     set(lldb_python_build_path "${LLDB_FRAMEWORK_ABSOLUTE_BUILD_DIR}/LLDB.framework/Resources/Python/lldb")
   else()

lldb/test/Shell/Reproducer/lit.local.cfg

 # Enable crash reports for the reproducer tests.
 if 'LLVM_DISABLE_CRASH_REPORT' in config.environment:
   del config.environment['LLVM_DISABLE_CRASH_REPORT']
 
+# Unset the always capture environment override.
 if 'LLDB_CAPTURE_REPRODUCER' in config.environment:
   del config.environment['LLDB_CAPTURE_REPRODUCER']
+
+# Running the reproducer tests under the lldb-repro utility doesn't make sense.
+if 'LLDB_REPRO_UTIL_CAPTURE' in config.environment or \
+   'LLDB_REPRO_UTIL_REPLAY' in config.environment:
+  config.unsupported = True

lldb/test/Shell/helper/toolchain.py

 import os
 import itertools
 import platform
 import subprocess
 import sys
 
 import lit.util
 from lit.llvm import llvm_config
 from lit.llvm.subst import FindTool
 from lit.llvm.subst import ToolSubst
 
+
+def _lldb_init(config):
+    return os.path.join(config.test_exec_root, 'Shell', 'lit-lldb-init')
+
+
 def use_lldb_substitutions(config):
     # Set up substitutions for primary tools.  These tools must come from config.lldb_tools_dir
     # which is basically the build output directory.  We do not want to find these in path or
     # anywhere else, since they are specifically the programs which are actually being tested.
 
     dsname = 'debugserver' if platform.system() in ['Darwin'] else 'lldb-server'
     dsargs = [] if platform.system() in ['Darwin'] else ['gdbserver']
 
     build_script = os.path.dirname(__file__)
     build_script = os.path.join(build_script, 'build.py')
     build_script_args = [build_script,
                         '--compiler=any', # Default to best compiler
                         '--arch=' + str(config.lldb_bitness)]
     if config.lldb_lit_tools_dir:
         build_script_args.append('--tools-dir={0}'.format(config.lldb_lit_tools_dir))
     if config.lldb_tools_dir:
         build_script_args.append('--tools-dir={0}'.format(config.lldb_tools_dir))
     if config.llvm_libs_dir:
         build_script_args.append('--libs-dir={0}'.format(config.llvm_libs_dir))
 
-    lldb_init = os.path.join(config.test_exec_root, 'Shell', 'lit-lldb-init')
+    lldb_init = _lldb_init(config)
 
     primary_tools = [
         ToolSubst('%lldb',
                   command=FindTool('lldb'),
                   extra_args=['--no-lldbinit', '-S', lldb_init]),
         ToolSubst('%lldb-init',
                   command=FindTool('lldb'),
                   extra_args=['-S', lldb_init]),
if have_lld:
         config.available_features.add('lld')
 
 
     support_tools = ['yaml2obj', 'obj2yaml', 'llvm-pdbutil',
                      'llvm-mc', 'llvm-readobj', 'llvm-objdump',
                      'llvm-objcopy', 'lli']
     additional_tool_dirs += [config.lldb_tools_dir, config.llvm_tools_dir]
     llvm_config.add_tool_substitutions(support_tools, additional_tool_dirs)
+
+def use_lldb_repro_substitutions(config):
+    lldb_init = _lldb_init(config)
+    substitutions = [
+        ToolSubst('%lldb',
+                  command=FindTool('lldb-repro'),
+                  extra_args=['--no-lldbinit', '-S', lldb_init]),
+        ToolSubst('%lldb-init',
+                  command=FindTool('lldb-repro'),
+                  extra_args=['-S', lldb_init]),
+    ]
+    llvm_config.add_tool_substitutions(substitutions)

lldb/test/Shell/lit.cfg.py

 config.excludes = ['Inputs', 'CMakeLists.txt', 'README.txt', 'LICENSE.txt']
 
 # test_source_root: The root path where tests are located.
 config.test_source_root = os.path.dirname(__file__)
 
 # test_exec_root: The root path where tests should be run.
 config.test_exec_root = os.path.join(config.lldb_obj_root, 'test')
 
-# Propagate LLDB_CAPTURE_REPRODUCER
+# Propagate reproducer environment vars.
 if 'LLDB_CAPTURE_REPRODUCER' in os.environ:
   config.environment['LLDB_CAPTURE_REPRODUCER'] = os.environ[
       'LLDB_CAPTURE_REPRODUCER']
 
+# Support running the test suite under the lldb-repro wrapper. This makes it
+# possible to capture a test suite run and then rerun all the test from the
+# just captured reproducer.
+run_under_lldb_repro = lit_config.params.get('lldb-run-with-repro', '').lower()
+if run_under_lldb_repro == "capture":
+  config.environment['LLDB_REPRO_UTIL_CAPTURE'] = '1'
+  toolchain.use_lldb_repro_substitutions(config)
+elif run_under_lldb_repro == "replay":
+  lit_config.note("Running Shell test from reproducer replay")
+  config.environment['LLDB_REPRO_UTIL_REPLAY'] = '1'
+  toolchain.use_lldb_repro_substitutions(config)
+
 llvm_config.use_default_substitutions()
 toolchain.use_lldb_substitutions(config)
 toolchain.use_support_substitutions(config)
 
-
 if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
     config.available_features.add("armhf-linux")
 
 def calculate_arch_features(arch_string):
     # This will add a feature such as x86, arm, mips, etc for each built
     # target
     features = []
     for arch in arch_string.split():

lldb/utils/lldb-repro/CMakeLists.txt

+add_custom_target(lldb-repro)
+add_dependencies(lldb-repro lldb-test-deps)
+set_target_properties(lldb-repro PROPERTIES FOLDER "lldb utils")
+configure_file(lldb-repro.py ${LLVM_RUNTIME_OUTPUT_INTDIR}/lldb-repro COPYONLY)

lldb/utils/lldb-repro/lldb-repro.py

+#!/usr/bin/env python
+
+import sys
+import os
+import tempfile
+import subprocess
+
+
+def main():
+    # Compute a hash based on the input arguments and the current working
+    # directory.
+    args = ' '.join(sys.argv[1:])
+    cwd = os.getcwd()
+    input_hash = str(hash((cwd, args)))
+
+    # Use the hash to "uniquely" identify a reproducer path.
+    reproducer_path = os.path.join(tempfile.gettempdir(), input_hash)
+
+    # Create a new lldb invocation with capture or replay enabled.
+    lldb = os.path.join(os.path.dirname(sys.argv[0]), 'lldb')
+    new_args = [lldb]
+    if 'LLDB_REPRO_UTIL_REPLAY' in os.environ:
+        new_args.extend(['--replay', reproducer_path])
+    elif 'LLDB_REPRO_UTIL_CAPTURE' in os.environ:
+        new_args.extend([
+            '--capture', '--capture-path', reproducer_path,
+            '--reproducer-auto-generate'
+        ])
+        new_args.extend(sys.argv[1:])
+    else:
+        print("error: enable capture or replay by setting the environment "
+              "variables LLDB_REPRO_UTIL_CAPTURE and LLDB_REPRO_UTIL_REPLAY "
+              "respectively")
+        return 1
+
+    return subprocess.call(new_args)
+
+
+# lldb-repro is a utility to transparently capture and replay debugger sessions
+# through the command line driver. Its used to test the reproducers by running
+# the test suite twice.
+#
+# During the first run, with LLDB_REPRO_UTIL_CAPTURE set, it captures a
+# reproducer for every lldb invocation and saves it to a well-know location
+# derived from the arguments and current working directory.
+#
+# During the second run, with LLDB_REPRO_UTIL_CAPTURE set, the test suite is
+# run again but this time every invocation of lldb replays the previously
+# recorded session.
+if __name__ == '__main__':
+    exit(main())