[lldb/Reproducers] Make it possible to capture reproducers for the Python test suite.

[JDevlieghere]

Summary

This adds the necessary boiler plate to capture and replay API tests.

http://lists.llvm.org/pipermail/lldb-dev/2020-April/016100.html

[labath]

lldb/bindings/python.swig

132–136 ↗

(On Diff #255469)

I am doubly unhappy about this. Not only does it leak testing logic into production code, it actually does that using environment variables... :/

I take it the problem here is that reproducers need to be set up before the Initialize call, but the lldb module does not let us do that as it calls Initialize automatically? I am not sure that was such a wise idea, but that ship has sailed a long time ago.

So, can we at least do something like this instead? I.e. have a separate configuration module which would be imported here and would drive this logic.

I believe it should be possible to arrange it so that the configuration module does not even have to get shipped -- this code could just attempt the import and ignore any errors.

And the configuration module itself would not need to do anything reproducer-related either. It could just contain a single flag to disable the automatic initialization. That sounds like a thing someone might conceivably want anyway, and it could be later extended to handle other kinds of global lldb configuration (ConstString pool size maybe ?).

Once the Initialize call is disabled, the normal test harness can handle the rest...

[JDevlieghere]

Use lldbconfig to initialize reproducers.

[JDevlieghere]

Update lldbtest.py too.

[labath]

lldb/packages/Python/lldbsuite/test/decorators.py

861

Should this use variables from configuration.py now?

lldb/test/API/lit.cfg.py

68

lldb-repro

lldb/test/API/lldbtest.py

90

I remember some objections to shoving everything in $TMP. Are we not able to put this in the build dir somewhere?

[JDevlieghere]

Address code review feedback

[labath]

I think the code looks ok now, though I have to say I am still somewhat fuzzy on what exactly will this patch enable. I think I understand the capture part, but what happens during replay? Will that already be smart enough to match the SB calls made by the test harness during replay to the captured calls that were made during recording, or is there some more work needed there? Could you add a brief documentation on this somewhere?

[JDevlieghere]

In D77588#1968700, @labath wrote:

I think the code looks ok now, though I have to say I am still somewhat fuzzy on what exactly will this patch enable. I think I understand the capture part, but what happens during replay? Will that already be smart enough to match the SB calls made by the test harness during replay to the captured calls that were made during recording, or is there some more work needed there?

That's implemented in https://reviews.llvm.org/D77602

Could you add a brief documentation on this somewhere?

Sure thing: https://reviews.llvm.org/D77771

[labath]

Ok, I'm starting to understand what's going on here. This patch looks mostly fine, but IIUC, it won't really do anything until the subsequent patch lands as that is what implements the inline/api/passive replay. I'm going to start looking at that other patch now.

[JDevlieghere]

In D77588#1971345, @labath wrote:

Ok, I'm starting to understand what's going on here. This patch looks mostly fine, but IIUC, it won't really do anything until the subsequent patch lands as that is what implements the inline/api/passive replay. I'm going to start looking at that other patch now.

It's sufficient for stage one, i.e. capture the test suite and use the driver (active replay), but it also contains the necessary changes for stage 2 (passive replay). I could've split it up in two patches but given that the changes are so similar that didn't seem worth it.

[JDevlieghere]

Update naming to passive replay.

[labath]

In D77588#1972418, @JDevlieghere wrote:

In D77588#1971345, @labath wrote:

Ok, I'm starting to understand what's going on here. This patch looks mostly fine, but IIUC, it won't really do anything until the subsequent patch lands as that is what implements the inline/api/passive replay. I'm going to start looking at that other patch now.

It's sufficient for stage one, i.e. capture the test suite and use the driver (active replay), but it also contains the necessary changes for stage 2 (passive replay). I could've split it up in two patches but given that the changes are so similar that didn't seem worth it.

Ah, ok. I can live with that.

lldb/bindings/headers.swig

 #include "lldb/API/SBMemoryRegionInfoList.h"
 #include "lldb/API/SBModule.h"
 #include "lldb/API/SBModuleSpec.h"
 #include "lldb/API/SBPlatform.h"
 #include "lldb/API/SBProcess.h"
 #include "lldb/API/SBProcessInfo.h"
 #include "lldb/API/SBQueue.h"
 #include "lldb/API/SBQueueItem.h"
+#include "lldb/API/SBReproducer.h"
 #include "lldb/API/SBSection.h"
 #include "lldb/API/SBSourceManager.h"
 #include "lldb/API/SBStream.h"
 #include "lldb/API/SBStringList.h"
 #include "lldb/API/SBStructuredData.h"
 #include "lldb/API/SBSymbol.h"
 #include "lldb/API/SBSymbolContext.h"
 #include "lldb/API/SBSymbolContextList.h"

lldb/bindings/interface/SBReproducer.i

+//===-- SWIG Interface for SBReproducer--------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+namespace lldb {
+class SBReproducer
+{
+    public:
+        static const char *Capture(const char *path);
+        static const char *PassiveReplay(const char *path);
+        static bool SetAutoGenerate(bool b);
+};
+}

lldb/bindings/interfaces.swig

 %include "./interface/SBMemoryRegionInfoList.i"
 %include "./interface/SBModule.i"
 %include "./interface/SBModuleSpec.i"
 %include "./interface/SBPlatform.i"
 %include "./interface/SBProcess.i"
 %include "./interface/SBProcessInfo.i"
 %include "./interface/SBQueue.i"
 %include "./interface/SBQueueItem.i"
+%include "./interface/SBReproducer.i"
 %include "./interface/SBSection.i"
 %include "./interface/SBSourceManager.i"
 %include "./interface/SBStream.i"
 %include "./interface/SBStringList.i"
 %include "./interface/SBStructuredData.i"
 %include "./interface/SBSymbol.i"
 %include "./interface/SBSymbolContext.i"
 %include "./interface/SBSymbolContextList.i"

lldb/include/lldb/API/SBReproducer.h

 /// replay of SB API calls. As a result we cannot rely on any other SB objects
 /// in the interface or implementation of this class.
 class LLDB_API SBReproducer {
 public:
   static const char *Capture();
   static const char *Capture(const char *path);
   static const char *Replay(const char *path);
   static const char *Replay(const char *path, bool skip_version_check);
+  static const char *PassiveReplay(const char *path);
   static const char *GetPath();
   static bool SetAutoGenerate(bool b);
   static bool Generate();
 };
 
 } // namespace lldb
 
 #endif

lldb/packages/Python/lldbsuite/test/configuration.py

 # m - test method name
 # a - architecture
 # c - compiler path
 # The default is to write all fields.
 session_file_format = 'fnmac'
 
 # Set this flag if there is any session info dumped during the test run.
 sdir_has_content = False
-
 # svn_info stores the output from 'svn info lldb.base.dir'.
 svn_info = ''
 
 # Default verbosity is 0.
 verbose = 0
 
 # By default, search from the script directory.
 # We can't use sys.path[0] to determine the script directory

 
 # Test results handling globals
 results_filename = None
 results_formatter_name = None
 results_formatter_object = None
 results_formatter_options = None
 test_result = None
 
+# Reproducers
+capture_path = None
+replay_path = None
+
 # Test rerun configuration vars
 rerun_all_issues = False
 
 # The names of all tests. Used to assert we don't have two tests with the
 # same base name.
 all_tests = set()
 
 # LLDB library directory.

lldb/packages/Python/lldbsuite/test/decorators.py

 
 def skipIfAsan(func):
     """Skip this test if the environment is set up to run LLDB itself under ASAN."""
     def is_asan():
         if ('ASAN_OPTIONS' in os.environ):
             return "ASAN unsupported"
         return None
     return skipTestIfFn(is_asan)(func)
+
+def skipIfReproducer(func):
+    """Skip this test if the environment is set up to run LLDB with reproducers."""
+    def is_reproducer():
+        if configuration.capture_path or configuration.replay_path:
+            return "reproducers unsupported"
+        return None
+    return skipTestIfFn(is_reproducer)(func)

lldb/packages/Python/lldbsuite/test/dotest.py

def parseOptionsAndInitTestdirs():
     if args.results_formatter_options:
         configuration.results_formatter_options = args.results_formatter_options
 
     # Default to using the BasicResultsFormatter if no formatter is specified.
     if configuration.results_formatter_name is None:
         configuration.results_formatter_name = (
             "lldbsuite.test_event.formatter.results_formatter.ResultsFormatter")
 
+    # Reproducer arguments
+    if args.capture_path and args.replay_path:
+        logging.error('Cannot specify both a capture and a replay path.')
+        sys.exit(-1)
+
+    if args.capture_path:
+        configuration.capture_path = args.capture_path
+
+    if args.replay_path:
+        configuration.replay_path = args.replay_path
+
     # rerun-related arguments
     configuration.rerun_all_issues = args.rerun_all_issues
 
     if args.lldb_platform_name:
         configuration.lldb_platform_name = args.lldb_platform_name
     if args.lldb_platform_url:
         configuration.lldb_platform_url = args.lldb_platform_url
     if args.lldb_platform_working_dir:
def run_suite():
     parseOptionsAndInitTestdirs()
 
     # Setup test results (test results formatter and output handling).
     setupTestResults()
 
     setupSysPath()
 
     import lldbconfig
+    if configuration.capture_path or configuration.replay_path:
+        lldbconfig.INITIALIZE = False
     import lldb
 
+    if configuration.capture_path:
+        lldb.SBReproducer.Capture(configuration.capture_path)
+        lldb.SBReproducer.SetAutoGenerate(True)
+    elif configuration.replay_path:
+        lldb.SBReproducer.PassiveReplay(configuration.replay_path)
+
+    if not lldbconfig.INITIALIZE:
+        lldb.SBDebugger.Initialize()
+
     # Use host platform by default.
     lldb.selected_platform = lldb.SBPlatform.GetHostPlatform()
 
     # Now we can also import lldbutil
     from lldbsuite.test import lldbutil
 
     if configuration.lldb_platform_name:
         print("Setting up remote platform '%s'" %

lldb/packages/Python/lldbsuite/test/dotest_args.py

group.add_argument(
         metavar='platform-url',
         help='A LLDB platform URL to use when connecting to a remote platform to run the test suite')
     group.add_argument(
         '--platform-working-dir',
         dest='lldb_platform_working_dir',
         metavar='platform-working-dir',
         help='The directory to use on the remote platform.')
 
+    # Reproducer options
+    group = parser.add_argument_group('Reproducer options')
+    group.add_argument(
+        '--capture-path',
+        metavar='reproducer path',
+        help='The reproducer capture path')
+    group.add_argument(
+        '--replay-path',
+        metavar='reproducer path',
+        help='The reproducer replay path')
+
     # Test-suite behaviour
     group = parser.add_argument_group('Runtime behaviour options')
     X('-d', 'Suspend the process after launch to wait indefinitely for a debugger to attach')
     X('-t', 'Turn on tracing of lldb command and other detailed test executions')
     group.add_argument(
         '-u',
         dest='unset_env_varnames',
         metavar='variable',

lldb/source/API/SBReproducer.cpp

const char *SBReproducer::Capture(const char *path) {
   if (auto e =
           Reproducer::Initialize(ReproducerMode::Capture, FileSpec(path))) {
     error = llvm::toString(std::move(e));
     return error.c_str();
   }
   return nullptr;
 }
 
+const char *SBReproducer::PassiveReplay(const char *path) {
+  static std::string error;
+  if (auto e = Reproducer::Initialize(ReproducerMode::Replay, FileSpec(path))) {
+    error = llvm::toString(std::move(e));
+    return error.c_str();
+  }
+  return nullptr;
+}
+
 const char *SBReproducer::Replay(const char *path) {
   return SBReproducer::Replay(path, false);
 }
 
 const char *SBReproducer::Replay(const char *path, bool skip_version_check) {
   static std::string error;
   if (auto e = Reproducer::Initialize(ReproducerMode::Replay, FileSpec(path))) {
     error = llvm::toString(std::move(e));

lldb/test/API/functionalities/reproducers/attach/TestReproducerAttach.py

class ReproducerAttachTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
     NO_DEBUG_INFO_TESTCASE = True
 
     @skipIfFreeBSD
     @skipIfNetBSD
     @skipIfWindows
     @skipIfRemote
     @skipIfiOSSimulator
+    @skipIfReproducer
     def test_reproducer_attach(self):
         """Test thread creation after process attach."""
         exe = '%s_%d' % (self.testMethodName, os.getpid())
 
         token = self.getBuildArtifact(exe + '.token')
         if os.path.exists(token):
             os.remove(token)
 

lldb/test/API/lit.cfg.py

else:
     lit_config.warning("unable to inject shared library path on '{}'".format(
         platform.system()))
 
 # Propagate LLDB_CAPTURE_REPRODUCER
 if 'LLDB_CAPTURE_REPRODUCER' in os.environ:
   config.environment['LLDB_CAPTURE_REPRODUCER'] = os.environ[
       'LLDB_CAPTURE_REPRODUCER']
 
+# Support running the test suite under the lldb-repro wrapper. This makes it
+# possible to capture a test suite run and then rerun all the test from the
+# just captured reproducer.
+lldb_repro_mode = lit_config.params.get('lldb-run-with-repro', None)
+if lldb_repro_mode:
+  lit_config.note("Running API tests in {} mode.".format(lldb_repro_mode))
+  if lldb_repro_mode == 'capture':
+    config.available_features.add('lldb-repro-capture')
+  elif lldb_repro_mode == 'replay':
+    config.available_features.add('lldb-repro-replay')
+
 # Clean the module caches in the test build directory. This is necessary in an
 # incremental build whenever clang changes underneath, so doing it once per
 # lit.py invocation is close enough.
 for cachedir in [config.clang_module_cache, config.lldb_module_cache]:
   if os.path.isdir(cachedir):
     print("Deleting module cache at %s." % cachedir)
     shutil.rmtree(cachedir)
 

lldb/test/API/lldbtest.py

 from __future__ import absolute_import
 import os
-
+import tempfile
 import subprocess
 import sys
 
 import lit.Test
 import lit.TestRunner
 import lit.util
 from lit.formats.base import TestFormat
 
def execute(self, test, litConfig):
             return (lit.Test.UNSUPPORTED, 'Test is unsupported')
 
         testPath, testFile = os.path.split(test.getSourcePath())
         # On Windows, the system does not always correctly interpret
         # shebang lines.  To make sure we can execute the tests, add
         # python exe as the first parameter of the command.
         cmd = [sys.executable] + self.dotest_cmd + [testPath, '-p', testFile]
 
+        builddir = getBuildDir(cmd)
+        mkdir_p(builddir)
+
         # The macOS system integrity protection (SIP) doesn't allow injecting
         # libraries into system binaries, but this can be worked around by
         # copying the binary into a different location.
         if 'DYLD_INSERT_LIBRARIES' in test.config.environment and \
                 (sys.executable.startswith('/System/') or \
                 sys.executable.startswith('/usr/bin/')):
-            builddir = getBuildDir(cmd)
-            mkdir_p(builddir)
             copied_python = os.path.join(builddir, 'copied-system-python')
             if not os.path.isfile(copied_python):
                 import shutil, subprocess
                 python = subprocess.check_output([
                     sys.executable,
                     '-c',
                     'import sys; print(sys.executable)'
                 ]).decode('utf-8').strip()
                 shutil.copy(python, copied_python)
             cmd[0] = copied_python
 
+        if 'lldb-repro-capture' in test.config.available_features or \
+           'lldb-repro-replay' in test.config.available_features:
+            reproducer_root = os.path.join(builddir, 'reproducers')
+            mkdir_p(reproducer_root)
+            reproducer_path = os.path.join(reproducer_root, testFile)
+            if 'lldb-repro-capture' in test.config.available_features:
+                cmd.extend(['--capture-path', reproducer_path])
+            else:
+                cmd.extend(['--replay-path', reproducer_path])
+
         timeoutInfo = None
         try:
             out, err, exitCode = lit.util.executeCommand(
                 cmd,
                 env=test.config.environment,
                 timeout=litConfig.maxIndividualTestTime)
         except lit.util.ExecuteCommandTimeoutException as e:
             out = e.out