This is an archive of the discontinued LLVM Phabricator instance.

[tsan] Add support for C++ exceptions into TSan (call __tsan_func_exit during unwinding)
ClosedPublic

Authored by kubamracek on Oct 31 2016, 5:49 PM.

Details

Summary

This patch is a re-incarnation of https://reviews.llvm.org/D10740 (which seems to be abandoned now). This adds support for TSan C++ exception handling, where we need to add extra calls to __tsan_func_exit when a function is exitted via exception mechanisms. Otherwise the shadow stack gets corrupted (leaked). There is an existing implementation that finds all possible function exit points, and adds extra EH cleanup blocks where needed.

I moved the implementation of EscapeEnumerator to a header file, but note that I made one change, which is the addition of the II->setDebugLoc(CI->getDebugLoc()); line. This makes sure we don't lose debug information.

Diff Detail

Event Timeline

kubamracek updated this revision to Diff 76509.Oct 31 2016, 5:49 PM
kubamracek retitled this revision from to [tsan] Add support for C++ exceptions into TSan (call __tsan_func_exit during unwinding).
kubamracek updated this object.
kubamracek added reviewers: dvyukov, kcc, rnk.
kubamracek set the repository for this revision to rL LLVM.
kubamracek added a project: Restricted Project.
kubamracek added subscribers: llvm-commits, zaks.anna.

Also note that this change increases the code size of TSanified binaries, roughly by 10-20%.

rnk edited edge metadata.Nov 2 2016, 10:06 AM
In D26177#585299, @kubabrecka wrote:

Also note that this change increases the code size of TSanified binaries, roughly by 10-20%.

I don't expect there will be any increase in code size if -fno-exceptions is set, because all functions will be marked nounwind, so we won't do the call-to-invoke transformation.

include/llvm/Transforms/Utils/EscapeEnumerator.h
31–32

The state machine is superfluous, it only has two real states: 1 (running) and 2 (done). The state 0 logic could be handled in the constructor by setting StateBB and StateE. Mind killing the switch and this comment, assuming we don't implement the code size optimization I described below?

46

The implementation of Next should not live in a header.

55

LLVM_FALLTHROUGH would help me understand what's supposed to happen here.

75–77

Range-based for?

79–80

The right way to ask this is !CI->doesNotThrow(). I'm surprised we don't have a non-virtual override of Instruction::mayThrow in CallInst, but whatever.

93

This isn't going to be right everywhere. You probably want to add some function to lib/Analysis/EHPersonalities.cpp to pick a good default based on the target triple.

98

A landingpad instruction would be wrong on Windows, you would want to insert a cleanuppad that unwinds to the caller. For now I think it would be OK to report_fatal_error if isFuncletEHPersonality(classifyEHPersonality(PersFn)) is true.

123

This fails to handle operand bundles on calls. We should factor this logic out of the inliner, which gets it 100% correct, into lib/Transforms/Util/Local.cpp next to changeToCall, and use it from here.

lib/Transforms/Instrumentation/ThreadSanitizer.cpp
456–457

This is probably outside the scope of this patch, but is this really what you want to do? Instead of inserting a call before every return, you could create a new single return block with only one call. That might be better for code size. The same is true for resumes: you can make all resumes a jump to the same resume block, and then you only need one exit call, for a total of two calls per function, max.

Actually, this gives me a maybe too clever idea for code size reduction... what if we had __tsan_func_exit check if we are unwinding and call _Unwind_Resume from there? We'd still have binary size increase from the LSDA and we probably couldn't eliminate the landingpad basic block, but it's an interesting idea.

Note that I'm just moving the code for EscapeEnumerator, which is some pretty old code (r45670), and I'd CC the author, but I don't think he works on LLVM anymore. Anyway, thanks for the review, I'll update the patch!

kubamracek updated this revision to Diff 76796.Nov 2 2016, 3:12 PM
kubamracek edited edge metadata.
kubamracek removed rL LLVM as the repository for this revision.

Updating patch. Added some helper function to EHPersonalitites.h/.cpp. I still kept getDefaultEHPersonality always return EHPersonality::GNU_C -- please advise what the behavior should be. Annotated all TSan callbacks as "nounwind", which means we don't need to transform leaf functions. Extracted the BB splitting and CallInst-to-InvokeInst logic from InlineFunction.cpp into Utils/Local.cpp. Removed the state machine from EscapeEnumerator.

I'll look into the size optimizations later, but shouldn't the optimizer already do what you describe for higher -O levels?

rnk added a comment.Nov 2 2016, 3:34 PM

Looks good to me, but we should probably wait to see if Dmitry has any concerns about turning this on by default.

In D26177#586077, @kubabrecka wrote:

Note that I'm just moving the code for EscapeEnumerator, which is some pretty old code (r45670), and I'd CC the author, but I don't think he works on LLVM anymore. Anyway, thanks for the review, I'll update the patch!

Yeah, I noticed that after I got halfway through it, and then I felt like it was still worth a good rewrite. :)

dvyukov edited edge metadata.Nov 8 2016, 4:42 PM
  1. Please add tsan pass flag that turns exception interception off. We may need to disable it during deployment if it causes instability, compilation errors or too large binary increase.
  1. Are we sure that longjmp won't cause double call to __tsan_func_exit? At least MSVC longjmp will unwind stack and call destructors, which will most likely lead to double exit calls. Linux docs are vague in this regard. Not sure about mac.
projects/compiler-rt/test/tsan/exceptions.cc
112

More than 80 columns. Please fix.

  1. Please add tsan pass flag that turns exception interception off. We may need to disable it during deployment if it causes instability, compilation errors or too large binary increase.

Will do. What's "too large" for you? This does cause increases, which are larger in -O0 builds.

  1. Are we sure that longjmp won't cause double call to __tsan_func_exit? At least MSVC longjmp will unwind stack and call destructors, which will most likely lead to double exit calls. Linux docs are vague in this regard. Not sure about mac.

I'll try to check. Besides longjmp, do you have some particular scenario that's worth adding a testcase for?

In D26177#590243, @kubabrecka wrote:
  1. Please add tsan pass flag that turns exception interception off. We may need to disable it during deployment if it causes instability, compilation errors or too large binary increase.

Will do. What's "too large" for you? This does cause increases, which are larger in -O0 builds.

Any increase will cause some failures. We should have -fno-exceptions for most binaries, but maybe not for all.

  1. Are we sure that longjmp won't cause double call to __tsan_func_exit? At least MSVC longjmp will unwind stack and call destructors, which will most likely lead to double exit calls. Linux docs are vague in this regard. Not sure about mac.

I'll try to check. Besides longjmp, do you have some particular scenario that's worth adding a testcase for?

Can't think of anything else.

rnk added a comment.Nov 9 2016, 11:32 AM

Maybe there's a better way to do this. What if we wrote a custom personality function in the TSan RTL that wraps the normal personality function variants, and calls __tsan_func_exit internally? We can definitely do this for now, but it might be nice to experiment with the custom personality to win back the 10% code size bloat here.

kubamracek updated this revision to Diff 77571.Nov 10 2016, 4:38 PM
kubamracek edited edge metadata.

Updating patch:

  • added a switch to turn this off
  • added a testcase for the IR transformations and for the switch
  • added two testcase for setjmp/longjmp
  • changed EscapeEnumerator to have an "HandleExceptions" argument (default true)
  • changed EscapeEnumerator to not insert EH landingpads when the function itself is nounwind

I tested the setjmp/longjmp and it's okay on Darwin and Linux. Since this C++ exceptions support doesn't support exceptions on Windows, I didn't even try that. Reid, could you test this patch for me if it breaks compilation on Windows?

I'm also not sure what does it mean when a nounwind function calls something that's not nounwind. Should we instrument in this case?

Lastly, do you have an example that would trigger this instrumentation, but should not with -fno-exceptions?

rnk added a comment.Nov 10 2016, 4:54 PM
In D26177#592464, @kubabrecka wrote:

I tested the setjmp/longjmp and it's okay on Darwin and Linux. Since this C++ exceptions support doesn't support exceptions on Windows, I didn't even try that. Reid, could you test this patch for me if it breaks compilation on Windows?

I'm not set up to build tsan on Windows, it's only supported for gotsan. Dmitry knows how to build that, maybe he has time to test it.

I'm also not sure what does it mean when a nounwind function calls something that's not nounwind. Should we instrument in this case?

That's UB, so we don't need a cleanup, and we shouldn't insert one.

Lastly, do you have an example that would trigger this instrumentation, but should not with -fno-exceptions?

I think the only way this can happen is if there's a bug in clang or a subsequent IR instrumentation pass that inserts new functions.

I'm not set up to build tsan on Windows, it's only supported for gotsan. Dmitry knows how to build that, maybe he has time to test it.

gotsan uses Go compiler for instrumentation.

I am happy with tsan pass and runtime changes, leaving to Reid for approval.

rnk accepted this revision.Nov 11 2016, 7:24 AM
rnk edited edge metadata.

lgtm

This revision is now accepted and ready to land.Nov 11 2016, 7:24 AM
This revision was automatically updated to reflect the committed changes.
This revision was automatically updated to reflect the committed changes.

This landed in r286893 (LLVM) and r286894 (compiler-rt). Thanks for reviewing!