This is an archive of the discontinued LLVM Phabricator instance.

[lldb] Add ScriptedPlatform python implementation
ClosedPublic

Authored by mib on Dec 2 2022, 11:52 PM.

Details

Summary

This patch introduces both the Scripted Platform python base
implementation and an example for it.

The base implementation is embedded in lldb python module under
lldb.plugins.scripted_platform.

This patch also refactor the various SWIG methods to create scripted
objects into a single method, that is now shared between the Scripted
Platform, Process and Thread. It also replaces the target argument by a
execution context object.

Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>

Diff Detail

Event Timeline

mib created this revision.Dec 2 2022, 11:52 PM
Herald added a project: Restricted Project. · View Herald TranscriptDec 2 2022, 11:52 PM
mib requested review of this revision.Dec 2 2022, 11:52 PM
mib updated this revision to Diff 479809.Dec 2 2022, 11:56 PM

Fix typo

bulbazord added inline comments.Dec 6 2022, 2:15 PM
lldb/bindings/python/python-wrapper.swig
311–312

I think the error string needs to be adjusted here. It should say should be 1 " based on line 350.

lldb/examples/python/scripted_process/scripted_platform.py
37–40

Each line here needs to have a comma at the end of it.

44–47

I think you could add a little more information here. Based on the example below with MyScriptedPlatform it looks like the Dictionary maps PID (int) to Process information (dict). However, what you've written here doesn't indicate that. You could change the example above to be something like:

processes = {
    420: {
            name: a.out,
            arch: aarch64,
            pid: 420,
            parent_pid: 42 (optional),
            uid: 0 (optional),
            gid: 0 (optional),
    },
}
mib added inline comments.Dec 6 2022, 2:17 PM
lldb/examples/python/scripted_process/scripted_platform.py
44–47

Good point!

mib updated this revision to Diff 481844.Dec 10 2022, 4:35 AM
mib marked 3 inline comments as done.

Address @bulbazord comments

labath added a subscriber: labath.Dec 12 2022, 4:29 AM
labath added inline comments.
lldb/examples/python/scripted_process/scripted_platform.py
32

I am surprised that you want to go down the "run" path for this functionality. I think most of the launch functionality does not make sense for this use case (e.g., you can't provide arguments to these processes, when you "run" them, can you?), and it is not consistent with what the "process listing" functionality does for regular platforms.

OTOH, the "attach" flow makes perfect sense here -- you take the pid of an existing process, attach to it, and stop it at a random point in its execution. You can't customize anything about how that process is run (because it's already running) -- all you can do is choose how you want to select the target process.

mib added inline comments.Dec 12 2022, 4:54 AM
lldb/examples/python/scripted_process/scripted_platform.py
32

For now, there is no support for attaching to a scripted process, because we didn't have any use for it quite yet: cripted processes were mostly used for doing post-mortem debugging, so we "ran" them artificially in lldb by providing some launch options (the name of the class managing the process and an optional user-provided dictionary) through the command line or using an SBLaunchInfo object.

I guess I'll need to extend the platform process launch/attach commands and SBAttachInfo object to also support these options since they're required for the scripted process instantiation.

Note that we aren't really attaching to the real running process, we're creating a scripted process that knows how to read memory to mock the real process.

mib added a comment.Dec 12 2022, 4:01 PM
lldb/examples/python/scripted_process/scripted_platform.py
32

@labath, I'll do that work on a follow-up patch

mib added inline comments.Dec 13 2022, 8:46 AM
lldb/examples/python/scripted_process/scripted_platform.py
32
JDevlieghere added inline comments.Dec 13 2022, 1:39 PM
lldb/bindings/python/python-wrapper.swig
275

This looks pretty similar to LLDBSwigPythonCreateScriptedThread and LLDBSwigPythonCreateScriptedProcess. Can we factor out the common parts?

lldb/examples/python/scripted_process/scripted_platform.py
34

Why is this method implemented and not a pass like the others?

bulbazord added inline comments.Dec 13 2022, 2:13 PM
lldb/examples/python/scripted_process/scripted_platform.py
34

This method is not implemented, it's just a docstring explaining how this method should function. The pass is on line 50.

mib marked 2 inline comments as done.Dec 14 2022, 2:05 AM
mib added inline comments.
lldb/examples/python/scripted_process/scripted_platform.py
34

+1

labath added inline comments.Dec 15 2022, 1:40 PM
lldb/examples/python/scripted_process/scripted_platform.py
32

Thanks. However, are you still planning to use the launch path for your feature? Because if you're not, then I think this comment should say "Get a list of processes that are running" (or that can be attached to).

And if you are, then I'd like to hear your thoughts on the discrepancy between what "launching" means for scripted and non-scripted platforms.

lldb/test/API/functionalities/scripted_platform/my_scripted_platform.py
34

??

mib marked 3 inline comments as done.Dec 16 2022, 4:58 AM
mib added a subscriber: jingham.
mib added inline comments.
lldb/examples/python/scripted_process/scripted_platform.py
32

The way I see it is that the scripted platform will create a process with the right process plugin. In the case of scripted processes, the ProcessLaunchInfo argument should have the script class name set (which automatically sets the process plugin name to "ScriptedProcess" in the launch info). Once the process is instantiated (before the launch), the scripted platform will need to redirect to process stop events through its event multiplexer. So the way I see it essentially, running a regular process with the scripted platform should be totally transparent.

Something that is also worth discussing IMO, is the discrepancy between launching and attaching from the scripted platform:

One possibility could be that platform process launch would launch all the scripted processes listed by the scripted platform and set them up with the multiplexer, whereas platform process attach would just create a scripted process individually. I know this doesn't match the current behavior of the platform commands so if you guys think we should preserve the expected behavior, I guess.

May be @jingham has some opinion about this ?

lldb/test/API/functionalities/scripted_platform/my_scripted_platform.py
34

This is so the user can decide if the platform select command should be ran when the script is imported in lldb or not. If the user sets this env variable, the command will only be printed to the user, so they can copy it and run it whenever they want.

mib marked 3 inline comments as done.Dec 16 2022, 5:05 AM
mib added inline comments.Dec 16 2022, 5:21 AM
lldb/bindings/python/python-wrapper.swig
275

True, but this need some refactor unrelated to this patch, so I'll do it in a follow-up

mib updated this revision to Diff 483509.Dec 16 2022, 6:12 AM

Update comments for scripted_platform.list_processes

labath added inline comments.Dec 19 2022, 6:53 AM
lldb/examples/python/scripted_process/scripted_platform.py
32

Before we do that, maybe we could take a step back. Could you explain why you chose to use the "launch" flow for this use case?

To me, it just seems confusing to be using "launching" for any of this, particularly given that "attaching" looks like a much better match for what is happening here:

  • launch allows you to specify process cmdline arguments, attach does not - I don't think you will be able to specify cmdline arguments for these scripted processes
  • launch allows you to specify env vars, attach does not -- ditto
  • launch allows you to stop-at-entry, attach does not -- you cannot stop at entry for these processes, as they have been started already
  • attach allows you to specify a pid, launch does not -- you (I think) want to be able to choose the process (pid) that you want to create a scripted process for

For me, the choice is obvious, particularly considering that there *is* an obvious equivalent for "launching" for the kernel co-debugging use case. One could actually have the kernel create a new process --and then it would make sense to specify cmdline arguments, environment, and all of the other launch flags. I don't expect anyone to actually support this, as creating a brand new process like this is going to be very tricky, but one could still conceivably do that.

Now, I don't want to be designing the feature for you, but I do have a feeling that building the scripted platform feature around this "launch-is-attach" model is going to limit its usefulness to other, more conventional use cases. However, if the feature you're looking for is "launching all processes", then I don't see a problem with adding something like attach --all, which would attach to all (attachable) processes. It's probably not something one would want to use for normal platforms very often (so we may want to implement some kind of a "are you sure?" dialog), but functionally that makes sense to me regardless of the platform.

mib added inline comments.Dec 20 2022, 9:55 AM
lldb/examples/python/scripted_process/scripted_platform.py
32

I don't have any strong opinion for one over the other. The reason I'm going with launch is because this is what Scripted Processes already support. Originally, Scripted Processes were made for post-mortem debugging, so "re-launching" the process made sense to me, instead of attaching to a non-running process.

Regarding passing command line arguments, env variables, etc. this could be done using the -k/-v options or a structured data dictionary in the Process{Launch,Attach}Info, so both cases should be covered.

For the attach --all suggestion, I was thinking of something similar and I actually like it :) That would iterate over every process on the platform and call the attach method on it. For scripted processes, the process attach behavior could be customized by the implementor.

I'm happy with the launching/attaching discussion resolved. Similar to the deduplication I think it makes more sense to tackle that first instead of landing this "as is" and then fixing it after the fact.

lldb/bindings/python/python-wrapper.swig
275

Instead of adding a lot of duplicate code and factoring it out after, maybe it makes more sense to clean it up first and rebase this patch on top of that.

mib marked 2 inline comments as done.Dec 21 2022, 9:09 AM
labath added inline comments.Dec 21 2022, 12:58 PM
lldb/examples/python/scripted_process/scripted_platform.py
32

Well, I do have a medium-strong opinion on that. :)
I believe you that you can make it work through the launch code path. The -k/-v thing is a stringly typed api, and you can pass anything through that. But I have to ask: why would you be doing that, if you already have a bespoke api to do that? My concern is two fold:

  • Even if "process launch" does an "attach" to userspace process and the process-to-attach is specified using -k/-v pairs, the user can still pass "launchy" arguments (cmdline, env) to that command. So you now have to either ignore them, or do extra work to make sure they are rejected
  • Doing it this way would mean duplicating some existing lldb functionality. The attach command already supports attach-by-pid and attach-by-name modes, and I would expect that the users would want to use that in a scripted scenario as well. If they are meant to go through the launch path, then the launch code would have to support that as well.

I think that, for the post-mortem use case, the "load-core" flow (which actually uses parts of the attach code under the hood) would make more sense. I'm not sure what it would take to make that usable from a script.

mib added inline comments.Jan 10 2023, 12:30 AM
lldb/examples/python/scripted_process/scripted_platform.py
32

Hi @labath, I just getting back to this, and I'm a bit confused by what changes you're asking for exactly. Would you mind clarifying ? Thanks!

mib updated this revision to Diff 488004.Jan 10 2023, 3:00 PM
mib edited the summary of this revision. (Show Details)

Address @JDevlieghere comment:

  • refactor SWIG scripted object create methods into a single one
labath added inline comments.Jan 11 2023, 6:35 AM
lldb/examples/python/scripted_process/scripted_platform.py
32

Ideally, I would like you to say "I'm not going to implement the 'user/kernel co-debugging feature' using the 'launch' flow" :D. I don't know if you already decided that, but if you have, I was not aware of that.
If you really do want to go with the launch approach, I'd appreciate some response my concerns above (i.e., what are you going to do if the user specifies arguments when "launching" these processes? Will you be duplicating the attach-by-name functionality?) Or just some general description of how will the launch look like (e.g. what arguments will the user be providing to launch a given process).

In terms of this patch, I think the only part that's rubbing me the wrong way is the function description highlighted above -- specifically the "ran or" part. If a platform reports running processes (like normal platforms do), then you cannot "run" them again. I think we should drop that part and say "can be attached" (or simply "are running"). help platform process list says "List processes on a remote platform by name, pid, or many other matching attributes." and I think this should say something effectively equivalent to that (modulo the filtering part, as that does not happen here).

65

Does this really have to be a scripted process? Ideally, one could also launch a "normal" ProcessGdbRemote this way. This is pretty much what happens in PlatformQemuUser, and if that worked, then we could theoretically replace that built-in platform with a simple python script.

77

ditto

mib marked 5 inline comments as done.Jan 11 2023, 11:03 PM
mib added inline comments.
lldb/examples/python/scripted_process/scripted_platform.py
32

Done! I had misunderstood what you were asking for originally, but I actually think it makes more sense to use attach instead of launch in the context of user/kernel co-debugging :-) So just to recap, Scripted Platform will have both affordances but we will use attach for the co-debugging part.

mib updated this revision to Diff 488485.Jan 11 2023, 11:07 PM
mib marked an inline comment as done.

Address @labath comments:

  • Rephrase documentation to remove any Scripted Process occurrence
  • Add attach_to_process affordance to python Scripted Platform class
mib added a reviewer: labath.Jan 11 2023, 11:07 PM
labath accepted this revision.Jan 12 2023, 4:52 AM
labath added inline comments.
lldb/examples/python/scripted_process/scripted_platform.py
32

Awesome. I'm glad we're in agreement. Thanks.

This revision is now accepted and ready to land.Jan 12 2023, 4:52 AM
This revision was automatically updated to reflect the committed changes.