This is an archive of the discontinued LLVM Phabricator instance.

[lldb/Plugins] Add support of multiple ScriptedThreads in a ScriptedProcess
ClosedPublic

Authored by mib on Jan 11 2022, 5:33 PM.

Details

Summary

This patch adds support of multiple Scripted Threads in a ScriptedProcess.

This is done by fetching the Scripted Threads info dictionary at every
ScriptedProcess::DoUpdateThreadList and iterate over each element to
create a new ScriptedThread using the object instance, if it was not
already available.

This patch also adds the ability to pass a pointer of a script interpreter
object instance to initialize a ScriptedInterface instead of having to call
the script object initializer in the ScriptedInterface constructor.

This is used to instantiate the ScriptedThreadInterface from the
ScriptedThread constructor, to be able to perform call on that script
interpreter object instance.

Finally, the patch also updates the scripted process test to check for
multiple threads.

rdar://84507704

Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>

Diff Detail

Event Timeline

mib requested review of this revision.Jan 11 2022, 5:33 PM
mib created this revision.
mib edited the summary of this revision. (Show Details)Jan 11 2022, 5:33 PM
mib updated this revision to Diff 399146.Jan 11 2022, 5:45 PM
mib updated this revision to Diff 399149.Jan 11 2022, 5:50 PM
mib updated this revision to Diff 399152.Jan 11 2022, 6:04 PM

Update invalid_scripted_process.py

mib added a subscriber: labath.Jan 12 2022, 7:57 AM
mib added inline comments.
lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp
332

@labath This is why I switched to a raw pointer in D117065. Basically, I fetch a python dictionary for the thread_info_sp. It gets converted to a StructuredData::Dictionary and every element gets re-allocated in pythonObject::CreateStructuredDictionary. Then I iterate over each of element of the StructuredData::Dictionary and passes the StructuredData::Object to the ScriptedThread constructor.

The ScriptedThread needs a valid reference to the python object instance to perform calls on it. However, when the structured dictionary goes out of scope (at the end of the function), all the objects it holds get destroyed so when the ScriptedThread tries to make a call on the python object is causes a heap-use-after-free.

mib updated this revision to Diff 399413.Jan 12 2022, 12:31 PM
mib added a reviewer: labath.

Update after abandoning D117065 for D117139.

To create the ScriptedThread, we pass a StructuredData::Object* that contains a borrowed reference of the python ScriptedThread instance, to the constructor.

Previously, we just re-assigned the ScriptedThread StructuredData script object shared_ptr with the StructuredData::Object*.
Now, we "extract" the python borrowed reference from the StructuredData::Object* and create a new StructuredData object with it, to make sure we can access it even after the StructuredData::Object* gets freed when exiting ScriptedProcess::DoUpdateThreadList

mib updated this revision to Diff 399426.Jan 12 2022, 12:48 PM
mib added inline comments.Jan 12 2022, 12:49 PM
lldb/source/Plugins/Process/scripted/ScriptedThread.cpp
51

Instead of re-assigning m_script_object_sp with script_object directly, we get the python object ptr and pass it to CreatePluginObject that will create a new StructuredData::GenericSP holding that ref.

Never thought I'd ask someone to merge two patches, but I think it might make reviewing easier if you merge D117139 into this patch. :-)

If I understand the patch correctly, you're getting the underlying PyObject out of the PythonDateObject when passing it to the scripted thread interface. I assume that works because the objects are retained by being stored in a dict on the Python side (referring to threads in ScriptedProcess, from D117068). Why can't we pass the PythonObject around instead? That seems simpler but more importantly would guarantee the underlying object remains alive (even if it weren't stored on the Python side) by keeping the ref-count incremented.

mib updated this revision to Diff 399662.Jan 13 2022, 6:29 AM
mib edited the summary of this revision. (Show Details)

Address @JDevlieghere comment by merging D117139 into D117071

mib added a comment.Jan 13 2022, 7:45 AM

Never thought I'd ask someone to merge two patches, but I think it might make reviewing easier if you merge D117139 into this patch. :-)

Done ^^

If I understand the patch correctly, you're getting the underlying PyObject out of the PythonDateObject when passing it to the scripted thread interface. I assume that works because the objects are retained by being stored in a dict on the Python side (referring to threads in ScriptedProcess, from D117068). Why can't we pass the PythonObject around instead? That seems simpler but more importantly would guarantee the underlying object remains alive (even if it weren't stored on the Python side) by keeping the ref-count incremented.

Right. Sure, it would be easier to pass the PythonObject around but the Scripted(Thread)Interface tries to be as language-agnostic as possible, that's why we use StructuredData::Generic instead. I think it should remain this way.

Continuing the discussion from the other patch, I've made some comments about who I think owns various objects going around in this patch. They should be read in the control-flow order (which happens to coincide with how they appear in the source code), not in the order the phabricator puts them at the bottom of the page. I'd appreciate it if you could go through them and point out any errors in my reasoning/assumptions.

lldb/source/Plugins/Process/scripted/ScriptedProcess.cpp
304

This looks like the "start" of this patch. IIUC, this shared_ptr is the only shared_ptr pointing to this object (use_count() == 1).

313

this object appears to come out of thread_info_sp. I am going to assume it is owned by that object.

332

This just does a cast so it is still owned by that object.

lldb/source/Plugins/Process/scripted/ScriptedThread.cpp
32

Control continues here. script_object is not owned by this function/class.

49

This is when things start to get fuzzy, as this function seems to support both a nullptr and a non-nullptr argument. Not necessarily bad, but it makes reasoning about anything harder. It'd be better if this function could always be called with a valid object.

51

I'm assuming this is a PyObject disguised as a void* and that it is still being owned(increffed) by the initial shared_ptr.

lldb/source/Plugins/ScriptInterpreter/Python/ScriptedThreadPythonInterface.cpp
34 ↗(On Diff #399662)

continuing here

47 ↗(On Diff #399662)

Now it gets really confusing. *If* instance_obj was non-null, then it will point to an object which is not owned by this function. *If* it was null, then it set to point to an object which *is* owned (or at least *should be* owned -- as noone else owns it) by this function.

This doesn't seem correct to me, but even if it is, it sure is confusing.

56 ↗(On Diff #399662)

now this maybe-owned reference gets passed into a StructuredPythonObject which assumes it is getting a borrowed reference (it does an incref).

That seems correct in the instance_obj@entry != nullptr case, but not in the other one (=> leak).

mib marked 9 inline comments as done.Jan 13 2022, 2:01 PM
mib added inline comments.
lldb/source/Plugins/Process/scripted/ScriptedThread.cpp
49

I understand but even if it might be possible to do it for ScriptedThreads, we want to give the user multiple way of creating them (in lldb or directly from python).

lldb/source/Plugins/ScriptInterpreter/Python/ScriptedThreadPythonInterface.cpp
47 ↗(On Diff #399662)

Correct, when instance_obj is null at entry, it gets allocated by LLDBSwigPythonCreateScriptedThread which creates an owned reference as you mentioned in https://reviews.llvm.org/D117065#inline-1120555, and that reference gets passed to m_object_instance_sp.

So in my understanding, at this stage, instance_obj use_count() == 1.

56 ↗(On Diff #399662)

I might be missing something but I don't understand why does the StructuredPythonObject expects a borrowed reference ?
Why can't it wrap an owned reference ?

I think checking the PyObject type (Borrowed/Owned) on the StructuredPythonObject constructor would allow us to incref it only in the case of a borrowed ref, and fix the leak in the case of an owned ref, right ?

mib marked 3 inline comments as done.Jan 13 2022, 2:07 PM

Continuing the discussion from the other patch, I've made some comments about who I think owns various objects going around in this patch. They should be read in the control-flow order (which happens to coincide with how they appear in the source code), not in the order the phabricator puts them at the bottom of the page. I'd appreciate it if you could go through them and point out any errors in my reasoning/assumptions.

I've marked as done all of the comments that sounded good to me, and left some comments and questions myself.

I think this patch can stay as is, as long as add support for owned references in StructuredPythonObject. I don't see any reason why this should not be allowed, but I might be missing something. If this solution sounds good to you, I can write a separate patch for that.

labath added inline comments.Jan 14 2022, 10:39 AM
lldb/source/Plugins/Process/scripted/ScriptedThread.cpp
49

While not ideal, I don't think this is a big deal. I might consider passing the script_object to the CreatePluginObject though -- it would make a typed interface, and avoid a branch here.

lldb/source/Plugins/ScriptInterpreter/Python/ScriptedThreadPythonInterface.cpp
56 ↗(On Diff #399662)

Why can't it wrap an owned reference ?

There's no reason it *can't* do that. It's just that it doesn't do that now.

checking the PyObject type (Borrowed/Owned)

borrowedness/ownedness is not a real property of the PyObject. It's just an abstract notion describing the relationship between a reference (PyObject instance) and code handling that reference. If some code "owns" a reference it is responsible for (eventually) freeing (decreffing) it (or passing the ownership onto someone else, etc.) If some code "borrows" a reference, then it must not free it, and it must be careful to not use it after the actual owner frees it.

The reference received through the instance_obj is borrowed since the thread_info_sp destructor will eventually free it. The reference created through LLDBSwigPythonCreateScriptedThread is owned. The fact that you have owning and non-owning code paths converging means one of them is incorrect.

The fix is actually pretty straightforward -- you can change a borrowed reference into an owned one by increffing it yourself. Ideally the PythonObject helper class, as that way the ownership will be explicitly managed.

mib updated this revision to Diff 400148.Jan 14 2022, 2:30 PM
mib marked an inline comment as done.

Address @labath feedbacks:

  • Pass StructuredData::Generic *script_object to ScriptedInterface::CreatePluginObject.
  • IncRef the borrowed reference to make it an owned reference.
mib marked an inline comment as done.Jan 14 2022, 2:31 PM

Unifying the two paths by making turning the other reference into an owned one is a step in the right direction, but it is not enough, as StructuredPythonObject expects a *borrowed* reference. So, instead of fixing things, you've made both code paths leak. :)

I've created D117462, which should make reasoning about ownership much easier. StructuredPythonObject and all of the LLDBSwigPython functions now tracks ownership automatically, and one only needs to be careful when converting to/from a void *. Ideally those would be removed too, but that would require a bigger refactor. If you rebase this patch on top of that, you should be able to transfer ownership correctly.

While working on the patch I also realized that pretty much all of our StructuredPythonObject interfaces were leaking. So, in a way, you were very unfortunate to hit a use after free here (like, it it must have taken multiple decrefs on borrowed references (one for each thread, I guess) to undo all of those leaks). OTOH, we were also fortunate that you've found this, as we can now improve the interfaces and fix those leaks.

mib updated this revision to Diff 400801.Jan 18 2022, 3:50 AM

Rebase

labath accepted this revision.Jan 18 2022, 6:39 AM

I think this is as good as we can make it. Thanks for your patience.

This revision is now accepted and ready to land.Jan 18 2022, 6:39 AM
mib added a comment.Jan 18 2022, 7:48 AM

I think this is as good as we can make it. Thanks for your patience.

Thanks for your help :) !

This revision was landed with ongoing or failed builds.Jan 24 2022, 11:26 AM
This revision was automatically updated to reflect the committed changes.
lldb/test/API/functionalities/scripted_process/invalid_scripted_process.py