[lldb] Add ScriptedPlatform python implementation

[mib]

Summary

This patch introduces both the Scripted Platform python base
implementation and an example for it.

The base implementation is embedded in lldb python module under
lldb.plugins.scripted_platform.

This patch also refactor the various SWIG methods to create scripted
objects into a single method, that is now shared between the Scripted
Platform, Process and Thread. It also replaces the target argument by a
execution context object.

Signed-off-by: Med Ismail Bennani <[e-mail]>

[mib]

Fix typo

[bulbazord]

lldb/bindings/python/python-wrapper.swig

353–354

I think the error string needs to be adjusted here. It should say should be 1 " based on line 350.

lldb/examples/python/scripted_process/scripted_platform.py

37–40

Each line here needs to have a comma at the end of it.

44–47

I think you could add a little more information here. Based on the example below with MyScriptedPlatform it looks like the Dictionary maps PID (int) to Process information (dict). However, what you've written here doesn't indicate that. You could change the example above to be something like:

processes = {
    420: {
            name: a.out,
            arch: aarch64,
            pid: 420,
            parent_pid: 42 (optional),
            uid: 0 (optional),
            gid: 0 (optional),
    },
}

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

44–47

Good point!

[mib]

Address @bulbazord comments

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

I am surprised that you want to go down the "run" path for this functionality. I think most of the launch functionality does not make sense for this use case (e.g., you can't provide arguments to these processes, when you "run" them, can you?), and it is not consistent with what the "process listing" functionality does for regular platforms.

OTOH, the "attach" flow makes perfect sense here -- you take the pid of an existing process, attach to it, and stop it at a random point in its execution. You can't customize anything about how that process is run (because it's already running) -- all you can do is choose how you want to select the target process.

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

For now, there is no support for attaching to a scripted process, because we didn't have any use for it quite yet: cripted processes were mostly used for doing post-mortem debugging, so we "ran" them artificially in lldb by providing some launch options (the name of the class managing the process and an optional user-provided dictionary) through the command line or using an SBLaunchInfo object.

I guess I'll need to extend the platform process launch/attach commands and SBAttachInfo object to also support these options since they're required for the scripted process instantiation.

Note that we aren't really attaching to the real running process, we're creating a scripted process that knows how to read memory to mock the real process.

[mib]

ping @JDevlieghere

lldb/examples/python/scripted_process/scripted_platform.py

32

@labath, I'll do that work on a follow-up patch

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

@labath here D139945 :)

[JDevlieghere]

lldb/bindings/python/python-wrapper.swig

317

This looks pretty similar to LLDBSwigPythonCreateScriptedThread and LLDBSwigPythonCreateScriptedProcess. Can we factor out the common parts?

lldb/examples/python/scripted_process/scripted_platform.py

34

Why is this method implemented and not a pass like the others?

[bulbazord]

lldb/examples/python/scripted_process/scripted_platform.py

34

This method is not implemented, it's just a docstring explaining how this method should function. The pass is on line 50.

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

34

+1

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

Thanks. However, are you still planning to use the launch path for your feature? Because if you're not, then I think this comment should say "Get a list of processes that are running" (or that can be attached to).

And if you are, then I'd like to hear your thoughts on the discrepancy between what "launching" means for scripted and non-scripted platforms.

lldb/test/API/functionalities/scripted_platform/my_scripted_platform.py

34

??

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

The way I see it is that the scripted platform will create a process with the right process plugin. In the case of scripted processes, the ProcessLaunchInfo argument should have the script class name set (which automatically sets the process plugin name to "ScriptedProcess" in the launch info). Once the process is instantiated (before the launch), the scripted platform will need to redirect to process stop events through its event multiplexer. So the way I see it essentially, running a regular process with the scripted platform should be totally transparent.

Something that is also worth discussing IMO, is the discrepancy between launching and attaching from the scripted platform:

One possibility could be that platform process launch would launch all the scripted processes listed by the scripted platform and set them up with the multiplexer, whereas platform process attach would just create a scripted process individually. I know this doesn't match the current behavior of the platform commands so if you guys think we should preserve the expected behavior, I guess.

May be @jingham has some opinion about this ?

lldb/test/API/functionalities/scripted_platform/my_scripted_platform.py

34

This is so the user can decide if the platform select command should be ran when the script is imported in lldb or not. If the user sets this env variable, the command will only be printed to the user, so they can copy it and run it whenever they want.

[mib]

lldb/bindings/python/python-wrapper.swig

317

True, but this need some refactor unrelated to this patch, so I'll do it in a follow-up

[mib]

Update comments for scripted_platform.list_processes

[mib]

ping @JDevlieghere

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

Before we do that, maybe we could take a step back. Could you explain why you chose to use the "launch" flow for this use case?

To me, it just seems confusing to be using "launching" for any of this, particularly given that "attaching" looks like a much better match for what is happening here:

• launch allows you to specify process cmdline arguments, attach does not - I don't think you will be able to specify cmdline arguments for these scripted processes
• launch allows you to specify env vars, attach does not -- ditto
• launch allows you to stop-at-entry, attach does not -- you cannot stop at entry for these processes, as they have been started already
• attach allows you to specify a pid, launch does not -- you (I think) want to be able to choose the process (pid) that you want to create a scripted process for

For me, the choice is obvious, particularly considering that there *is* an obvious equivalent for "launching" for the kernel co-debugging use case. One could actually have the kernel create a new process --and then it would make sense to specify cmdline arguments, environment, and all of the other launch flags. I don't expect anyone to actually support this, as creating a brand new process like this is going to be very tricky, but one could still conceivably do that.

Now, I don't want to be designing the feature for you, but I do have a feeling that building the scripted platform feature around this "launch-is-attach" model is going to limit its usefulness to other, more conventional use cases. However, if the feature you're looking for is "launching all processes", then I don't see a problem with adding something like attach --all, which would attach to all (attachable) processes. It's probably not something one would want to use for normal platforms very often (so we may want to implement some kind of a "are you sure?" dialog), but functionally that makes sense to me regardless of the platform.

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

I don't have any strong opinion for one over the other. The reason I'm going with launch is because this is what Scripted Processes already support. Originally, Scripted Processes were made for post-mortem debugging, so "re-launching" the process made sense to me, instead of attaching to a non-running process.

Regarding passing command line arguments, env variables, etc. this could be done using the -k/-v options or a structured data dictionary in the Process{Launch,Attach}Info, so both cases should be covered.

For the attach --all suggestion, I was thinking of something similar and I actually like it :) That would iterate over every process on the platform and call the attach method on it. For scripted processes, the process attach behavior could be customized by the implementor.

[JDevlieghere]

I'm happy with the launching/attaching discussion resolved. Similar to the deduplication I think it makes more sense to tackle that first instead of landing this "as is" and then fixing it after the fact.

lldb/bindings/python/python-wrapper.swig

317

Instead of adding a lot of duplicate code and factoring it out after, maybe it makes more sense to clean it up first and rebase this patch on top of that.

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

Well, I do have a medium-strong opinion on that. :)
I believe you that you can make it work through the launch code path. The -k/-v thing is a stringly typed api, and you can pass anything through that. But I have to ask: why would you be doing that, if you already have a bespoke api to do that? My concern is two fold:

• Even if "process launch" does an "attach" to userspace process and the process-to-attach is specified using -k/-v pairs, the user can still pass "launchy" arguments (cmdline, env) to that command. So you now have to either ignore them, or do extra work to make sure they are rejected
• Doing it this way would mean duplicating some existing lldb functionality. The attach command already supports attach-by-pid and attach-by-name modes, and I would expect that the users would want to use that in a scripted scenario as well. If they are meant to go through the launch path, then the launch code would have to support that as well.

I think that, for the post-mortem use case, the "load-core" flow (which actually uses parts of the attach code under the hood) would make more sense. I'm not sure what it would take to make that usable from a script.

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

Hi @labath, I just getting back to this, and I'm a bit confused by what changes you're asking for exactly. Would you mind clarifying ? Thanks!

[mib]

Address @JDevlieghere comment:

• refactor SWIG scripted object create methods into a single one

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

Ideally, I would like you to say "I'm not going to implement the 'user/kernel co-debugging feature' using the 'launch' flow" :D. I don't know if you already decided that, but if you have, I was not aware of that.
If you really do want to go with the launch approach, I'd appreciate some response my concerns above (i.e., what are you going to do if the user specifies arguments when "launching" these processes? Will you be duplicating the attach-by-name functionality?) Or just some general description of how will the launch look like (e.g. what arguments will the user be providing to launch a given process).

In terms of this patch, I think the only part that's rubbing me the wrong way is the function description highlighted above -- specifically the "ran or" part. If a platform reports running processes (like normal platforms do), then you cannot "run" them again. I think we should drop that part and say "can be attached" (or simply "are running"). help platform process list says "List processes on a remote platform by name, pid, or many other matching attributes." and I think this should say something effectively equivalent to that (modulo the filtering part, as that does not happen here).

65

Does this really have to be a scripted process? Ideally, one could also launch a "normal" ProcessGdbRemote this way. This is pretty much what happens in PlatformQemuUser, and if that worked, then we could theoretically replace that built-in platform with a simple python script.

77

ditto

[mib]

lldb/examples/python/scripted_process/scripted_platform.py

32

Done! I had misunderstood what you were asking for originally, but I actually think it makes more sense to use attach instead of launch in the context of user/kernel co-debugging :-) So just to recap, Scripted Platform will have both affordances but we will use attach for the co-debugging part.

[mib]

Address @labath comments:

• Rephrase documentation to remove any Scripted Process occurrence
• Add attach_to_process affordance to python Scripted Platform class

[labath]

lldb/examples/python/scripted_process/scripted_platform.py

32

Awesome. I'm glad we're in agreement. Thanks.

lldb/bindings/python/CMakeLists.txt

function(finish_swig_python swig_target lldb_python_bindings_dir lldb_python_target_dir)
 
   create_python_package(
     ${swig_target}
     ${lldb_python_target_dir}
     "plugins"
     FILES
     "${LLDB_SOURCE_DIR}/examples/python/scripted_process/scripted_process.py")
 
+  create_python_package(
+    ${swig_target}
+    ${lldb_python_target_dir}
+    "plugins"
+    FILES
+    "${LLDB_SOURCE_DIR}/examples/python/scripted_process/scripted_platform.py")
+
   if(APPLE)
     create_python_package(
       ${swig_target}
       ${lldb_python_target_dir} "macosx"
       FILES "${LLDB_SOURCE_DIR}/examples/python/crashlog.py"
             "${LLDB_SOURCE_DIR}/examples/python/scripted_process/crashlog_scripted_process.py"
             "${LLDB_SOURCE_DIR}/examples/darwin/heap_find/heap.py")
 

lldb/bindings/python/python-wrapper.swig

PythonObject lldb_private::LLDBSwigPythonCreateScriptedThread(
   if (arg_info.get().max_positional_args == 2)
     return pfunc(ToSWIGWrapper(process_sp), ToSWIGWrapper(args_impl));
 
   error_string.assign("wrong number of arguments in __init__, should be 2 "
                       "(not including self)");
   return PythonObject();
 }
 
+PythonObject lldb_private::LLDBSwigPythonCreateScriptedPlatform(
+    const char *python_class_name, const char *session_dictionary_name,
+    const lldb_private::StructuredDataImpl &args_impl,
+    std::string &error_string) {
+  if (python_class_name == NULL || python_class_name[0] == '\0' ||
+      !session_dictionary_name)
+    return PythonObject();
+
+  PyErr_Cleaner py_err_cleaner(true);
+
+  auto dict = PythonModule::MainModule().ResolveName<PythonDictionary>(
+      session_dictionary_name);
+  auto pfunc = PythonObject::ResolveNameWithDictionary<PythonCallable>(
+      python_class_name, dict);
+
+  if (!pfunc.IsAllocated()) {
+    error_string.append("could not find script class: ");
+    error_string.append(python_class_name);
+    return PythonObject();
+  }
+
+  llvm::Expected<PythonCallable::ArgInfo> arg_info = pfunc.GetArgInfo();
+  if (!arg_info) {
+    llvm::handleAllErrors(
+        arg_info.takeError(),
+        [&](PythonException &E) { error_string.append(E.ReadBacktrace()); },
+        [&](const llvm::ErrorInfoBase &E) {
+          error_string.append(E.message());
+        });
+    return PythonObject();
+  }
+
+  PythonObject result = {};
+  if (arg_info.get().max_positional_args == 1) {
+    result = pfunc(ToSWIGWrapper(args_impl));
+  } else {
+    error_string.assign("wrong number of arguments in __init__, should be 2 "
+                        "(not including self)");
+  }
+  return result;
+}
+
 PythonObject lldb_private::LLDBSwigPythonCreateScriptedThreadPlan(
     const char *python_class_name, const char *session_dictionary_name,
     const lldb_private::StructuredDataImpl &args_impl,
     std::string &error_string, const lldb::ThreadPlanSP &thread_plan_sp) {
   if (python_class_name == NULL || python_class_name[0] == '\0' ||
       !session_dictionary_name)
     return PythonObject();
 

lldb/examples/python/scripted_process/scripted_platform.py

+from abc import ABCMeta, abstractmethod
+
+import lldb
+from lldb.plugins.scripted_process import ScriptedProcess
+from lldb.plugins.scripted_process import ScriptedThread
+
+class ScriptedPlatform(metaclass=ABCMeta):
+
+    """
+    The base class for a scripted platform.
+
+    Most of the base class methods are `@abstractmethod` that need to be
+    overwritten by the inheriting class.
+
+    DISCLAIMER: THIS INTERFACE IS STILL UNDER DEVELOPMENT AND NOT STABLE.
+                THE METHODS EXPOSED MIGHT CHANGE IN THE FUTURE.
+    """
+
+    processes = None
+
+    @abstractmethod
+    def __init__(self, args):
+        """ Construct a scripted process.
+
+        Args:
+            args (lldb.SBStructuredData): A Dictionary holding arbitrary
+                key/value pairs used by the scripted process.
+        """
+        processes = []
+
+    @abstractmethod
+    def list_processes(self):
+        """ Get a list of processes that can be ran on the platform.
+
+        process_info = {
+            name = a.out,
+            arch = aarch64,
+            pid = 420
+            parent_pid = 42 (optional)
+            uid = 0 (optional)
+            gid = 0 (optional)
+        }
+
+        Returns:
+            Dict: The processes represented as a dictionary, with at least the
+                process ID, name, architecture. Optionally, the user can also
+                provide the parent process ID and the user and group IDs.
+                The dictionary can be empty.
+        """
+        pass
+
+    def get_process_info(self, pid):
+        """ Get the dictionary describing the process.
+
+        Returns:
+            Dict: The dictionary of process info that matched process ID.
+            None if the process doesn't exists
+        """
+        pass
+
+    @abstractmethod
+    def launch_process(self, launch_info):
+        """ Launch a scripted process.
+
+        Args:
+            launch_info (lldb.SBLaunchInfo): The information related to the process launch.
+
+        Returns:
+            lldb.SBError: A status object notifying if the launch succeeded.
+        """
+        pass
+
+    @abstractmethod
+    def kill_process(self, pid):
+        """ Kill a scripted process.
+
+        Args:
+            pid (int): Process ID for the process to be killed.
+
+        Returns:
+            lldb.SBError: A status object notifying if the shutdown succeeded.
+        """
+        pass

lldb/source/Plugins/ScriptInterpreter/Python/SWIGPythonBridge.h

python::PythonObject LLDBSwigPythonCreateScriptedProcess(
     const lldb::TargetSP &target_sp, const StructuredDataImpl &args_impl,
     std::string &error_string);
 
 python::PythonObject LLDBSwigPythonCreateScriptedThread(
     const char *python_class_name, const char *session_dictionary_name,
     const lldb::ProcessSP &process_sp, const StructuredDataImpl &args_impl,
     std::string &error_string);
 
+python::PythonObject LLDBSwigPythonCreateScriptedPlatform(
+    const char *python_class_name, const char *session_dictionary_name,
+    const StructuredDataImpl &args_impl, std::string &error_string);
+
 llvm::Expected<bool> LLDBSwigPythonBreakpointCallbackFunction(
     const char *python_function_name, const char *session_dictionary_name,
     const lldb::StackFrameSP &sb_frame,
     const lldb::BreakpointLocationSP &sb_bp_loc,
     const lldb_private::StructuredDataImpl &args_impl);
 
 bool LLDBSwigPythonWatchpointCallbackFunction(
     const char *python_function_name, const char *session_dictionary_name,

lldb/test/API/functionalities/scripted_platform/my_scripted_platform.py

+import os
+
+import lldb
+from lldb.plugins.scripted_platform import ScriptedPlatform
+
+class MyScriptedPlatform(ScriptedPlatform):
+
+    def __init__(self, args):
+        self.processes = {}
+
+        proc = {}
+        proc['name'] = 'a.out'
+        proc['arch'] = 'arm64-apple-macosx'
+        proc['pid'] = 420
+        proc['parent'] = 42
+        proc['uid'] = 501
+        proc['gid'] = 20
+        self.processes[420] = proc
+
+    def list_processes(self):
+        return self.processes
+
+    def get_process_info(self, pid):
+        return self.processes[pid]
+
+    def launch_process(self, launch_info):
+        return lldb.SBError()
+
+    def kill_process(self, pid):
+        return lldb.SBError()
+
+def __lldb_init_module(debugger, dict):
+    if not 'SKIP_SCRIPTED_PLATFORM_SELECT' in os.environ:
+        debugger.HandleCommand(
+            "platform select scripted-platform -C %s.%s" % (__name__, MyScriptedPlatform.__name__))
+    else:
+        print("Name of the class that will manage the scripted platform: '%s.%s'"
+                % (__name__, MyScriptedPlatform.__name__))

lldb/unittests/ScriptInterpreter/Python/PythonTestSuite.cpp

 
 python::PythonObject lldb_private::LLDBSwigPythonCreateScriptedThread(
     const char *python_class_name, const char *session_dictionary_name,
     const lldb::ProcessSP &process_sp, const StructuredDataImpl &args_impl,
     std::string &error_string) {
   return python::PythonObject();
 }
 
+python::PythonObject lldb_private::LLDBSwigPythonCreateScriptedPlatform(
+    const char *python_class_name, const char *session_dictionary_name,
+    const StructuredDataImpl &args_impl, std::string &error_string) {
+  return python::PythonObject();
+}
+
 python::PythonObject lldb_private::LLDBSWIGPython_CreateFrameRecognizer(
     const char *python_class_name, const char *session_dictionary_name) {
   return python::PythonObject();
 }
 
 PyObject *lldb_private::LLDBSwigPython_GetRecognizedArguments(
     PyObject *implementor, const lldb::StackFrameSP &frame_sp) {
   return nullptr;