This is an archive of the discontinued LLVM Phabricator instance.

Thread safety analysis: Allow scoped releasing of capabilities
ClosedPublic

Authored by aaronpuchert on Sep 26 2018, 2:55 PM.

Details

Summary

The pattern is problematic with C++ exceptions, and not as widespread as
scoped locks, but it's still used by some, for example Chromium.

We are a bit stricter here at join points, patterns that are allowed for
scoped locks aren't allowed here. That could still be changed in the
future, but I'd argue we should only relax this if people ask for it.

Fixes PR36162.

Diff Detail

Repository
rL LLVM

Event Timeline

aaronpuchert created this revision.Sep 26 2018, 2:55 PM
aaronpuchert added a comment.EditedSep 26 2018, 3:09 PM

See the bug for further discussion. I'm not sure if we want to have this, but if the pattern is used more widely it might make sense. It blows up the code a bit, although I hope that D51187 might reduce it again.

delesley added inline comments.Sep 27 2018, 3:53 PM
lib/Analysis/ThreadSafety.cpp
893 ↗(On Diff #167205)

IMHO, it would make more sense to break this into two properties (or bits): acquired/released and shared/exclusive.

916 ↗(On Diff #167205)

UCK_ReleasedShared? (Shouldn't this have been caught by a test case?)

926 ↗(On Diff #167205)

This if statement makes my head hurt. Can you find a different way of expressing it?

951 ↗(On Diff #167205)

I find this very confusing. A lock here unlocks the underlying mutex, and vice versa. At the very least, some methods need to be renamed, or maybe we can have separate classes for ScopedLockable and ScopedUnlockable.

992 ↗(On Diff #167205)

Shouldn't this be outside of the else?

Maybe you should have a look at the tests first. I thought about the semantics that I think you are suggesting, but then we could have code like:

class SCOPED_LOCKABLE MutexLockUnlock {
public:
  MutexLockUnlock(Mutex *mu1, Mutex *mu2) EXCLUSIVE_UNLOCK_FUNCTION(mu1) EXCLUSIVE_LOCK_FUNCTION(mu2);
  ~MutexLockUnlock() /* what annotation do we use here? */;

  void a() EXCLUSIVE_UNLOCK_FUNCTION();
  void b() EXCLUSIVE_LOCK_FUNCTION();
};

Then what do a and b do? Or do we not allow this pattern? It's not straightforward either way, which is why I wanted to talk my way out of this in the bug report. ;)

lib/Analysis/ThreadSafety.cpp
893 ↗(On Diff #167205)

We don't use the information (shared/exclusive) for acquired locks, but we need two bits anyway, so why not.

916 ↗(On Diff #167205)

There is no test case for the shared variant yet. I'll add one.

951 ↗(On Diff #167205)

I agree that it's confusing, but it follows what I think was the idea behind scoped capabilities: that they are also capabilities that can be acquired/released. Now since the scoped capability releases a mutex on construction (i.e. when it is acquired), it has to acquire the mutex when released. So the way it handles the released mutexes mirrors what happens on the scoped capability itself.

It's definitely not very intuitive, but I feel it's the most consistent variant with what we have already.

The nice thing about this is that it works pretty well with the existing semantics and allows constructs such as MutexLockUnlock (see the tests) that unlocks one mutex and locks another. Not sure if anyone needs this, but why not?

992 ↗(On Diff #167205)

If we don't find UnderCp anymore, the negation should be there already. But I can also keep it outside if you like.

delesley added inline comments.Oct 4 2018, 3:17 PM
lib/Analysis/ThreadSafety.cpp
951 ↗(On Diff #167205)

A scoped_lockable object is not a capability, it is an object that manages capabilities. IMHO, conflating those two concepts is a bad idea, and recipe for confusion. You would not, for example, use a scoped_lockable object in a guarded_by attribute.

Unfortunately, the existing code tends to conflate "capabilities" and "facts", which is my fault. A scoped_lockable object is a valid fact -- the fact records whether the object is in scope -- it's just not a valid capability.

Simply renaming the methods from handleLock/Unlock to addFact/removeFact would be a nice first step in making the code clearer, and would distinguish between facts that refer to capabilities, and facts that refer to scoped objects.

992 ↗(On Diff #167205)

That's not necessarily true. Not knowing whether a mutex is locked is different from knowing that it is unlocked.

You're probably right in this case, because capabilities that are managed by scoped objects obey a more restricted set of rules. But then, you're also extending the ways in which scoped objects can be used. :-)

delesley added inline comments.Oct 4 2018, 3:21 PM
lib/Analysis/ThreadSafety.cpp
893 ↗(On Diff #167205)

The normal unlock doesn't distinguish between shared/exclusive, but there is a release_shared_capability which does...

aaronpuchert added inline comments.Oct 4 2018, 4:00 PM
lib/Analysis/ThreadSafety.cpp
893 ↗(On Diff #167205)

Right, but see bug 33504: currently release_shared_capability does not work with scoped capabilities. If we allow it, we would have to discuss when it is allowed. A scoped capability can lock multiple locks, some of them shared, some of them exclusive.

My assumption, as I wrote in the bug: scoped capabilities are always exclusive, hence can only be unlocked exclusively, but automatically release underlying mutexes in the right mode.

951 ↗(On Diff #167205)

Makes sense to me. I'll see if I can refactor some of the code to clarify this.

I think that we should also separate between unlocking the underlying mutexes and destructing the scoped_lockable, which is currently handled via the extra parameter FullyRemove of handleUnlock.

992 ↗(On Diff #167205)

Emphasis is on "anymore". We know we held UnderCp once (on construction) and it has been released since then. Since we have to add the negative capability whenever a lock is removed, it must already be there.

If we introduce support for deferred locking (like std::defer_lock_t) we would have to make sure we add the negative capability right at the beginning.

pwnall accepted this revision.Oct 5 2018, 11:08 AM
pwnall added a subscriber: pwnall.

test/SemaCXX/warn-thread-safety-analysis.cpp LGTM -- this is the functionality that we need in Chrome to use thread safety annotations for AutoUnlock.

very non-authoritative opinion - I think the tests would be a bit easier to follow if Lock() would be the EXCLUSIVE_LOCK_FUNCTION and Unlock() would be the EXCLUSIVE_UNLOCK_FUNCTION.

aaronpuchert: Thank you for the patch! I'd be happy and grateful to have this functionality in Chrome.
delesley: Thank you for your review and guidance!

This revision is now accepted and ready to land.Oct 5 2018, 11:08 AM
aaronpuchert planned changes to this revision.Oct 5 2018, 12:35 PM

I think I'll try to simplify this and address @delesley's comments before we commit this. I'll admit that the semantics are somewhat counter-intuitive, but as I explained I think it's more consistent this way. Because the scoped unlock releases a lock on construction, the operation on the underlying lock mirrors the operation on the scoped capability: releasing the scoped capability (on destruction, for example) acquires the lock again.

pwnall added a comment.Oct 5 2018, 1:02 PM

Thank you for clarifying, Aaron!

I probably used Phabricator incorrectly. My intent was to state that the tests LGTM and express support for the functionality in this patch. Please definitely address all review comments before landing.

aaronpuchert marked 3 inline comments as done.

Addressed some review comments and simplified the code.

There is a lot less duplication and maybe it's even easier to understand.

This revision is now accepted and ready to land.Oct 22 2018, 5:56 PM

I hope the cleanup makes the code more easily digestible, and to some extent might also transport why I think this is the most elegant approach.

I think we should document the semantics of scoped capabilities in more detail, and I will do so once this is either merged, or we decided that we don't want to merge it.

lib/Analysis/ThreadSafety.cpp
893 ↗(On Diff #167205)

I've thought about splitting up UCK_Acquired into a shared and exclusive variant, but not done it. Here's why: we don't need to know how the mutex was initially acquired for releasing it, and for re-acquisition we use the attribute on the "relock" method of the scoped capability. With released capabilities however we need to know in which mode they should be acquired again. Another reason is that we might want a fourth enumeration value to implement something in the direction of [std::defer_lock_t](https://en.cppreference.com/w/cpp/thread/unique_lock/unique_lock), and we might only have two bits.

951 ↗(On Diff #167205)

Scoped capabilities are not the same as capabilities, but they are also more than just facts. (Except if you follow Wittgenstein, perhaps.) They can be released, and reacquired, which makes them more than mere immutable facts. I think we can consider them as proxy capabilities. They cannot directly protect any resource, but we can acquire and release (actual) capabilities through them. That's why it doesn't sound crazy to me to say that releasing a scoped capability might acquire an underlying mutex, if that was released on construction of the scoped capability.

Introduced helper functions to clarify lock handling.

The previous version was too tightly coupled, and the introduction of AddCp and RemoveCp didn't help readability.

Negative capabilities don't need a LockKind.

delesley added inline comments.Nov 15 2018, 11:41 AM
lib/Analysis/ThreadSafety.cpp
951 ↗(On Diff #167205)

The notion of "proxy" is how scoped objects have always been treated in the past, but that's mostly a historical accident, rather than a well-reasoned design choice. As a bit of history, the first version of thread safety analysis only tracked mutexes. Thus, the original implementation pretended that a scoped object was a mutex, because that's the only thing that the analysis could deal with.

As the analysis grew more complex, I switched to the current system based on "facts". There are a number of facts that are potentially useful in static analysis, such as whether one expression aliases another, and most of them don't look at all like capabilities. IMHO, knowing whether an object is within scope falls into this class. The basic feature of facts is that they are control flow dependent -- they can be added to sets, removed from sets, and removed from intersections at joint points in the CFG. Renaming the methods to reflect this design will make future extensions easier to understand.

The only real argument for treating scoped lockable objects as proxies, which can be "locked" and "unlocked", is that you can (in a future patch) reuse the existing acquire_capability and release_capability attributes to support releasing and then re-acquiring the proxy. It's a bit counter-intuitive, but the alternative is adding new attributes, which is also bad.

However, even if you reuse the existing attributes, when it comes time to define a ScopedUnlockable class, you *still* shouldn't provide methods named "lock" and "unlock", where "lock" releases the underlying mutex, and "unlock" reacquires it. That's bad API design which will be endlessly confusing to users.

Thus, I would still like to rename the methods in this file to addFact and removeFact instead of handleLock and handleUnlock. That will support future extensions to the analysis.

As the analysis grew more complex, I switched to the current system based on "facts". There are a number of facts that are potentially useful in static analysis, such as whether one expression aliases another, and most of them don't look at all like capabilities. IMHO, knowing whether an object is within scope falls into this class.

Agreed. Though currently scoped capabilities aren't (necessarily) added/removed from the fact set on construction/destruction, only when the constructor/destructor is annotated as ACQUIRE()/RELEASE(). I was thinking about changing this, but haven't looked into it yet. This would clearly separate the fact of holding a scoped “capability” from the actual capabilities it represents.

The only real argument for treating scoped lockable objects as proxies, which can be "locked" and "unlocked", is that you can (in a future patch) reuse the existing acquire_capability and release_capability attributes to support releasing and then re-acquiring the proxy. It's a bit counter-intuitive, but the alternative is adding new attributes, which is also bad.

Scoped capabilities can already be released (before destruction) since your change rC159387, and I added the possibility to reacquire them in rC340459. I agree that it's technically an abuse of notation, but one can wrap it's head around it after a while. It is not possible though to annotate functions outside of the scoped capability class as acquiring/releasing a scoped capability. Right now I don't see a reason to change that.

To be clear, I'm not a big fan of this change myself, I just wanted to see if it was feasible. My personal opinion, as I wrote in the bug report, is that scoped releasing of mutexes is taking RAII a step too far. I'm putting this on ice for now until we're reached a state where it looks a bit less crazy. I hope @pwnall can live with that, since Clang 8 will not come out soon anyway.

To be clear, I'm not a big fan of this change myself, I just wanted to see if it was feasible. My personal opinion, as I wrote in the bug report, is that scoped releasing of mutexes is taking RAII a step too far. I'm putting this on ice for now until we're reached a state where it looks a bit less crazy. I hope @pwnall can live with that, since Clang 8 will not come out soon anyway.

If it makes a difference, Chrome (loosely) tracks Clang trunk. We wouldn't wait for a stable release to adopt the new code.

delesley accepted this revision.Nov 29 2018, 2:04 PM

Just to be clear, I'm approving the change, but I'd still like the methods to be renamed. :-)

This revision was automatically updated to reflect the committed changes.