This is an archive of the discontinued LLVM Phabricator instance.

func.wrap.func.con: Unset function before destroying anything
ClosedPublic

Authored by vsapsai on Jun 18 2017, 11:52 AM.

Details

Summary

Be defensive against a reentrant std::function::operator=(), in case the held function object has a non-trivial destructor. Destroying the function object in-place can lead to the destructor being called twice.

rdar://problem/32836603

Event Timeline

dexonsmith created this revision.Jun 18 2017, 11:52 AM
EricWF edited edge metadata.Jul 7 2017, 4:14 PM

@dexonsmith Mind if I hijack this and check in your changes to <functional> with my tests?

libcxx/test/libcxx/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/move_reentrant.pass.cpp
2

It seems like this test is testing behavior that should be required by the standard, right?

If so it should live under test/std.

26

Is asm("") just to prevent optimizations? If so please use DoNotOptimize from test_macros.h.

EricWF requested changes to this revision.Jul 7 2017, 7:26 PM

@dexonsmith I'm not sure it's sane to allow reentrant behavior. Could you explain why you think it is? Should the copy assignment operator allow reentrancy as well?

However there is another bug here. operator=(function&&) doesn't correctly call the destructor of the functor. I'll fix that as a separate commit.

This revision now requires changes to proceed.Jul 7 2017, 7:26 PM
EricWF added a comment.Jul 7 2017, 7:34 PM

However there is another bug here. operator=(function&&) doesn't correctly call the destructor of the functor. I'll fix that as a separate commit.

Woops, I misread the diff. There is no existing bug W.R.T. missing destructor calls.

@dexonsmith Mind if I hijack this and check in your changes to <functional> with my tests?

Not at all.

@dexonsmith I'm not sure it's sane to allow reentrant behavior. Could you explain why you think it is?

It didn't seem sane to me at first either, despite this being supported by std::unique_ptr and std::shared_ptr. I found our user fairly convincing, though:

  • They had an underlying reference counted object, so only the destruction of the last copy of A nulled out the pointer (mimicked that with the bool cancel).
  • They had a callback function intended to be called once, and dropping that function on cancellation (mimicked that with a global variable). The cancel operation nulled out a std::function, but destroying the objects captured in the lambda in that std::function also separately decided to perform a cancel, which should have been OK. The cancel function just contained callback = nullptr.

Should the copy assignment operator allow reentrancy as well?

I'm not sure; what do you think?

libcxx/test/libcxx/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/move_reentrant.pass.cpp
2

Yes, that likely is a better place.

26

Sounds fine to me.

@dexonsmith Mind if I hijack this and check in your changes to <functional> with my tests?

Not at all.

@dexonsmith I'm not sure it's sane to allow reentrant behavior. Could you explain why you think it is?

It didn't seem sane to me at first either, despite this being supported by std::unique_ptr and std::shared_ptr. I found our user fairly convincing, though:

  • They had an underlying reference counted object, so only the destruction of the last copy of A nulled out the pointer (mimicked that with the bool cancel).
  • They had a callback function intended to be called once, and dropping that function on cancellation (mimicked that with a global variable). The cancel operation nulled out a std::function, but destroying the objects captured in the lambda in that std::function also separately decided to perform a cancel, which should have been OK. The cancel function just contained callback = nullptr.

According to [reentrancy] it is implementation defined what STL functions are allowed to be recursively reentered. I'm not opposed to providing reentrancy where it's useful, but we would have to provide it well.
This is where I was running into problems while I was writing tests. There are so many different ways you can reenter std::function's special members that it makes it impossible for an implementation
to truly support reentrancy as the standard would require.

If you consider that every possible copy construction or destruction of a user type is a potential reentrancy point, the complexity of having well-defined reentrant behavior starts to become clear.
Any time a copy constructor or destructor is invoked you have a potential reentrancy point which, in order to provide well defined behavior, must also act as a sort of _sequence point_ where the side effects of the current call are visible to the reentrant callers (For example your patches use of operator=(nullptr_t)).

The methods fixed in this patch are seemingly improvements; It's clear that operator=(nullptr) can safely be made reentrant. On the other hand consider operator=(function const& rhs), which is specified as equivalent to function(rhs).swap(*this).
I posit swap is not the kind of thing that can easily be made reentrant, but that making std::function assignment reentrant in general clearly requires it.

If fixing this bug is sufficiently important I'm willing to accept that, but I don't think it improves the status quo; We may have fixed a specific users bug, but we haven't made these functions safely reentrant (in fact most of the special members are likely still full of reentrancy bugs).

@dexonsmith Mind if I hijack this and check in your changes to <functional> with my tests?

Not at all.

@dexonsmith I'm not sure it's sane to allow reentrant behavior. Could you explain why you think it is?

It didn't seem sane to me at first either, despite this being supported by std::unique_ptr and std::shared_ptr. I found our user fairly convincing, though:

  • They had an underlying reference counted object, so only the destruction of the last copy of A nulled out the pointer (mimicked that with the bool cancel).
  • They had a callback function intended to be called once, and dropping that function on cancellation (mimicked that with a global variable). The cancel operation nulled out a std::function, but destroying the objects captured in the lambda in that std::function also separately decided to perform a cancel, which should have been OK. The cancel function just contained callback = nullptr.

According to [reentrancy] it is implementation defined what STL functions are allowed to be recursively reentered. I'm not opposed to providing reentrancy where it's useful, but we would have to provide it well.
This is where I was running into problems while I was writing tests. There are so many different ways you can reenter std::function's special members that it makes it impossible for an implementation
to truly support reentrancy as the standard would require.

If you consider that every possible copy construction or destruction of a user type is a potential reentrancy point, the complexity of having well-defined reentrant behavior starts to become clear.
Any time a copy constructor or destructor is invoked you have a potential reentrancy point which, in order to provide well defined behavior, must also act as a sort of _sequence point_ where the side effects of the current call are visible to the reentrant callers (For example your patches use of operator=(nullptr_t)).

The methods fixed in this patch are seemingly improvements; It's clear that operator=(nullptr) can safely be made reentrant. On the other hand consider operator=(function const& rhs), which is specified as equivalent to function(rhs).swap(*this).
I posit swap is not the kind of thing that can easily be made reentrant, but that making std::function assignment reentrant in general clearly requires it.

If fixing this bug is sufficiently important I'm willing to accept that, but I don't think it improves the status quo; We may have fixed a specific users bug, but we haven't made these functions safely reentrant (in fact most of the special members are likely still full of reentrancy bugs).

IMO, function::operator=(nullptr_t) is a clear, restricted subset of reentrancy: it's the cutely-named std::function equivalent of unique_ptr::reset(). I think it's worth supporting reentrancy here.

After my own experimentation creating a testcase (and iterating with our internal users on motivation), I agree that supporting reentrancy for anything else would be untenable.

Don't you need // UNSUPPORTED: c++98, c++03 since std::function is supported in C++11 only?

Don't you need // UNSUPPORTED: c++98, c++03 since std::function is supported in C++11 only?

The tests in libcxx/test/std/utilities/function.objects/ don't have UNSUPPORTED lines, and I don't see a lit.local.cfg that sets config.unsupported. What would be different here?

The tests actually do compile in C++03, but they still fail because of infinite recursion:

frame #196562: 0x000000010000155c nullptr_t_assign_reentrant.pass.cpp.exe`A::~A() + 92
frame #196563: 0x0000000100001405 nullptr_t_assign_reentrant.pass.cpp.exe`std::__1::__compressed_pair_elem<A, 0, true>::~__compressed_pair_elem() + 21
frame #196564: 0x0000000100002685 nullptr_t_assign_reentrant.pass.cpp.exe`std::__1::__compressed_pair<A, std::__1::allocator<A> >::~__compressed_pair() + 21
frame #196565: 0x0000000100002665 nullptr_t_assign_reentrant.pass.cpp.exe`std::__1::__compressed_pair<A, std::__1::allocator<A> >::~__compressed_pair() + 21
frame #196566: 0x0000000100002389 nullptr_t_assign_reentrant.pass.cpp.exe`std::__1::__function::__func<A, std::__1::allocator<A>, void ()>::destroy() + 25
frame #196567: 0x00000001000014b9 nullptr_t_assign_reentrant.pass.cpp.exe`std::__1::function<void ()>::operator=(std::__1::nullptr_t) + 57
frame #196568: 0x000000010000155c nullptr_t_assign_reentrant.pass.cpp.exe`A::~A() + 92

Looks like prior to C++11 some different destruction behaviour is triggered which isn't fixed by this patch. So the tests should be either guarded by UNSUPPORTED/XFAIL or the patch should support C++03.

If you look at the AST diff between C++11 and C++03 this error still happens because of how nullptr is processed:

C++11:

| |-CXXDestructorDecl 0x7fdab27a2498 <line:24:3, line:28:3> line:24:3 used ~A 'void (void) noexcept'
| | `-CompoundStmt 0x7fdab27bcfb8 <col:8, line:28:3>
| |   |-GCCAsmStmt 0x7fdab27a2670 <line:25:5, col:11>
| |   `-IfStmt 0x7fdab27bcf88 <line:26:5, line:27:16>
| |     |-<<<NULL>>>
| |     |-ImplicitCastExpr 0x7fdab27a26e0 <line:26:9> '_Bool' <LValueToRValue>
| |     | `-DeclRefExpr 0x7fdab27a26b8 <col:9> '_Bool' lvalue Var 0x7fdab27a23b8 'cancel' '_Bool'
| |     |-CXXOperatorCallExpr 0x7fdab27bcf40 <line:27:7, col:16> 'class std::__1::function<void (void)>' lvalue
| |     | |-ImplicitCastExpr 0x7fdab27bcf28 <col:14> 'class std::__1::function<void (void)> &(*)(nullptr_t) noexcept' <FunctionToPointerDecay>
| |     | | `-DeclRefExpr 0x7fdab27bcea0 <col:14> 'class std::__1::function<void (void)> &(nullptr_t) noexcept' lvalue CXXMethod 0x7fdab27ab010 'operator=' 'class std::__1::function<void (void)> &(nullptr_t) noexcept'
| |     | |-DeclRefExpr 0x7fdab27a26f8 <col:7> 'std::function<void (void)>':'class std::__1::function<void (void)>' lvalue Var 0x7fdab27a2348 'global' 'std::function<void (void)>':'class std::__1::function<void (void)>'
| |     | `-CXXNullPtrLiteralExpr 0x7fdab27a2720 <col:16> 'nullptr_t'
| |     `-<<<NULL>>>

vs C++03:

| |-CXXDestructorDecl 0x7fc72b6d80d8 <line:24:3, line:28:3> line:24:3 used ~A 'void (void)'
| | `-CompoundStmt 0x7fc72b6dfd88 <col:8, line:28:3>
| |   |-GCCAsmStmt 0x7fc72b6d82a0 <line:25:5, col:11>
| |   `-IfStmt 0x7fc72b6dfd58 <line:26:5, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40>
| |     |-<<<NULL>>>
| |     |-ImplicitCastExpr 0x7fc72b6d8310 </Users/buildslave/jenkins/libcxx-build/libcxx.src/test/libcxx/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/nullptr_t_assign_reentrant.pass.cpp:26:9> '_Bool' <LValueToRValue>
| |     | `-DeclRefExpr 0x7fc72b6d82e8 <col:9> '_Bool' lvalue Var 0x7fc72b6d8028 'cancel' '_Bool'
| |     |-CXXOperatorCallExpr 0x7fc72b6dfd10 <line:27:7, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40> 'class std::__1::function<void (void)>' lvalue
| |     | |-ImplicitCastExpr 0x7fc72b6dfcf8 </Users/buildslave/jenkins/libcxx-build/libcxx.src/test/libcxx/utilities/function.objects/func.wrap/func.wrap.func/func.wrap.func.con/nullptr_t_assign_reentrant.pass.cpp:27:14> 'class std::__1::function<void (void)> &(*)(struct std::__1::nullptr_t)' <FunctionToPointerDecay>
| |     | | `-DeclRefExpr 0x7fc72b6dfca0 <col:14> 'class std::__1::function<void (void)> &(struct std::__1::nullptr_t)' lvalue CXXMethod 0x7fc72b6daea0 'operator=' 'class std::__1::function<void (void)> &(struct std::__1::nullptr_t)'
| |     | |-DeclRefExpr 0x7fc72b6d8328 <col:7> 'std::function<void (void)>':'class std::__1::function<void (void)>' lvalue Var 0x7fc72b6d7fb8 'global' 'std::function<void (void)>':'class std::__1::function<void (void)>'
| |     | `-CXXConstructExpr 0x7fc72b6dfc68 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40> 'struct std::__1::nullptr_t' 'void (const struct std::__1::nullptr_t &) throw()' elidable
| |     |   `-MaterializeTemporaryExpr 0x7fc72b6dfc50 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40> 'const struct std::__1::nullptr_t' lvalue
| |     |     `-ImplicitCastExpr 0x7fc72b6dfc38 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40> 'const struct std::__1::nullptr_t' <NoOp>
| |     |       `-CallExpr 0x7fc72b6d83d0 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:40> 'struct std::__1::nullptr_t'
| |     |         `-ImplicitCastExpr 0x7fc72b6d83b8 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:24> 'struct std::__1::nullptr_t (*)(void)' <FunctionToPointerDecay>
| |     |           `-DeclRefExpr 0x7fc72b6d8380 </Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__config:441:15, /Users/buildslave/jenkins/libcxx-build/libcxx.src/include/__nullptr:49:24> 'struct std::__1::nullptr_t (void)' lvalue Function 0x7fc72b08d340 '__get_nullptr_t' 'struct std::__1::nullptr_t (void)'
| |     `-<<<NULL>>>
arphaman added a comment.EditedAug 9 2017, 4:19 AM

Ah, I got it. There's a __functional_03 header that seems to implement function for C++03 because of manual variadic template expansions.
You have to update the operators in that header as well.

vsapsai commandeered this revision.Mar 21 2018, 2:03 PM
vsapsai added a reviewer: dexonsmith.
vsapsai updated this revision to Diff 139369.Mar 21 2018, 2:07 PM
vsapsai edited the summary of this revision. (Show Details)
  • Implement the same functionality for C++98 and C++03.
  • Use DoNotOptimize instead of asm.

Didn't move the tests as the standard doesn't require assignment operator to be
reentrant and we implement that not in general case but only for assigning
nullptr. Tests might be still worth moving, just previous arguments don't
apply anymore, as for me.

vsapsai marked an inline comment as done.Mar 21 2018, 2:08 PM

A few small comments...

libcxx/include/functional
1821

Couldn't this be more clearly written as *this = nullptr; ?

1822

At this point __f_ == 0 (because we just set it to 0).
We probably don't need to do that again.

1843

I see this dance 4x in <functional_03> and once here. Did you give any thought to making it a function of __base? Maybe call it __clear?

Then line #1821 can be written as __clear();

vsapsai added inline comments.Mar 21 2018, 3:36 PM
libcxx/include/functional
1722–1733

Here is the similar code I've mentioned.

1821

Personally I prefer to call operator explicitly but checking the code shows it isn't very common. Will change.

1822

This code is the same as in function(function&& __f). Do you think there is enough value to deviate from that implementation?

1843

Haven't really thought about that. Not sure it can be easily done. All __base classes are separate templates and don't have a common ancestor. Don't want to introduce macros as it doesn't look like libc++ style. You achieve the most consistency with current code by copy-pasting and repeating the same dance a few times.

vsapsai updated this revision to Diff 139506.Mar 22 2018, 2:03 PM
  • Replace function::operator=(nullptr); with *this = nullptr;
vsapsai marked an inline comment as done.Mar 22 2018, 2:03 PM

Are there more thoughts on this change?

mclow.lists accepted this revision.Apr 24 2018, 9:13 AM

Please move the tests into test/std and commit.

vsapsai updated this revision to Diff 143797.Apr 24 2018, 12:31 PM
  • Move tests to test/std.
This revision was not accepted when it landed; it landed in state Needs Review.Apr 25 2018, 4:44 PM
This revision was automatically updated to reflect the committed changes.

@mclow.lists
@STL_MSFT

Why did tests for this this go into std? [reentrancy]/1 says this isn't required to work. Moreover, assignments in the dtor like this *can't* work in the general case because they would try to overwrite the SSO space. e.g. what do you expect this to do?

std::function<void()> global;

struct B {
    int data = 1729;
    void operator() {}
};

struct A {
    int data = 42;
    ~A() {
        global = std::function<void()>(B{}); // whoops, constructs a B on top of A if Small Functor Optimization engages
        assert(data == 42);
    }

    void operator() {}
};

int main() {
    global = std::function<void ()>(A{});
    global = nullptr;
}

It is agreed that it is untenable to support reentrancy for std::function assignment operators and I'm not trying to achieve that. Here we restrict reentrancy only to std::function::operator=(nullptr_t).

@mclow.lists where should the tests go if the standard specifies the functionality as implementation-defined?

Except where explicitly specified in this standard, it is implementation-defined which functions in the Standard C++ library may be recursively reentered.

[reentrancy] p17.6.5.8