Page MenuHomePhabricator

[TSAN] Handle musttail call properly in EscapeEnumerator (and TSAN)
ClosedPublic

Authored by lxfind on Sep 14 2020, 10:12 AM.

Details

Summary

Call instructions with musttail tag must be optimized as a tailcall, otherwise could lead to incorrect program behavior.
When TSAN is instrumenting functions, it broke the contract by adding a call to the tsan exit function inbetween the musttail call and return instruction, and also inserted exception handling code.
This happend throguh EscapeEnumerator, which adds exception handling code and returns ret instructions as the place to insert instrumentation calls.
This becomes especially problematic for coroutines, because coroutines rely on tail calls to do symmetric transfers properly.
To fix this, this patch moves the location to insert instrumentation calls prior to the musttail call for ret instructions that are following musttail calls, and also does not handle exception for musttail calls.

Diff Detail

Event Timeline

lxfind created this revision.Sep 14 2020, 10:12 AM
lxfind requested review of this revision.Sep 14 2020, 10:12 AM
wenlei added inline comments.Sep 14 2020, 12:08 PM
llvm/lib/Transforms/Utils/EscapeEnumerator.cpp
111

Does TSAN rely on exit cleanup instrumentation for correctness? How does it work for existing Invoke?

This patch would be fine if missing some exit instrumentation is ok in general..

lxfind updated this revision to Diff 291653.Sep 14 2020, 12:40 PM

fix test failures

lxfind added inline comments.Sep 14 2020, 12:43 PM
llvm/lib/Transforms/Utils/EscapeEnumerator.cpp
111

The change would just make the call to tsan exit function happen earlier, before the tail call. So we are not missing any exit instrumentation after this change.

wenlei added inline comments.Sep 14 2020, 12:50 PM
llvm/lib/Transforms/Utils/EscapeEnumerator.cpp
111

The change would just make the call to tsan exit function happen earlier, before the tail call. So we are not missing any exit instrumentation after this change.

That's the part from AdjustMustTailCall.

IIUC, the other part on line 82 would skip changing throwing tail call to invoke, which would also skip the clean up in landing pad. That's what I was referring to,

lxfind added inline comments.Sep 14 2020, 12:54 PM
llvm/lib/Transforms/Utils/EscapeEnumerator.cpp
111

Ah I see. So the reason EscapeEnumerator tries to turn calls into invoke with an exception return path is so that when exceptions are thrown, it still has a chance to call the tsan exit function before propagating the exception back.
Now that this changes make the call to tsan exit function before the musttail call, we no longer need to try to catch the exception after the musttail call and call exit function again if it ever throws.

wenlei added inline comments.Sep 14 2020, 3:43 PM
llvm/lib/Transforms/Utils/EscapeEnumerator.cpp
111

Ok, yeah. Tailcall is different in that it's always going to be the block terminator, so guaranteed to be covered by the loop over blocks, unlike regular calls. So we're not missing out exit cleanup here. (I'm still curious how pre-existing invoke is handled, because we don't add cleanup to existing landing pad. But that is not directly related to this patch)

I think there's a still slight difference though for the order of cleanup. Now between cleanup for the caller of tail call and the actual return from the caller, there's the entire tail callee. I would think that is ok, but it really depends on the contract of TSAN, so would be nice if someone familiar with TSAN can confirm.

Otherwise LGTM, accept to unblock.

wenlei accepted this revision.Sep 14 2020, 3:43 PM
This revision is now accepted and ready to land.Sep 14 2020, 3:43 PM