[ASan] Enable optional ASan recovery

[ygribov]

Summary

This patch adds flags to clang and libasan to continue execution after detection of first error (in hope that more errors can be detected during single run). Such mode allows for a faster run-report-fix loop on embedded system where fixing a bug and preparing new firmware may take a significant time. To enable, user should build his code with -fsanitize-recover=address and then run it under ASAN_OPTIONS=keep_going=1.

I know kcc's generally negative attitude towards keep_going mode but I want the patch to at least be available publicly so that poor souls dealing with complex systems may access it easily.

[eugenis]

include/clang/Frontend/CodeGenOptions.def

118 ↗

(On Diff #33064)

Could you make it not specific to ASan? Smth like SanitizeRecover so that it could affect MSan as well in the future.

lib/CodeGen/BackendUtil.cpp

343 ↗

(On Diff #33064)

Why do you need separate "noabort" functions for this? See addMemorySanitizerPass which inspects CodeGenOptions in the function body.

lib/Transforms/Instrumentation/AddressSanitizer.cpp

125 ↗

(On Diff #33064)

Lets stick to existing terminology (we already have "recover" and "keep-going" names for the same concept).

lib/asan/asan_flags.inc

147 ↗

(On Diff #33064)

call it max_errors, similarly to the compiler flag?

lib/asan/asan_poisoning.cc

221 ↗

(On Diff #33064)

why "false"? Do you need to check the keep_going flag?

[kcc]

lib/asan/asan_report.cc

652 ↗

(On Diff #33064)

OMG

[ygribov]

include/clang/Frontend/CodeGenOptions.def

118 ↗

(On Diff #33064)

Right.

lib/CodeGen/BackendUtil.cpp

343 ↗

(On Diff #33064)

Ok.

lib/Transforms/Instrumentation/AddressSanitizer.cpp

125 ↗

(On Diff #33064)

Ok, I'll use "recover" then. Most of this was done in pre-recover times.

lib/asan/asan_flags.inc

147 ↗

(On Diff #33064)

Sure.

lib/asan/asan_poisoning.cc

221 ↗

(On Diff #33064)

Hm, probably yes.

lib/asan/asan_report.cc

652 ↗

(On Diff #33064)

This section could be removed if we are ok with ignoring bugs that run in parallel.

[ygribov]

lib/asan/asan_report.cc

652 ↗

(On Diff #33064)

Could you suggest a cleaner way to do this?

[ygribov]

Updated based on initial feedback. It seems we have way too many different names for recovery feature (UBSan's trap-on-error, MSan's keep-going, ASan's recover, noabort, etc.).

[samsonov]

include/clang/Basic/Sanitizers.h

82 ↗

(On Diff #33203)

Do you actually use it anywhere?

lib/CodeGen/BackendUtil.cpp

214 ↗

(On Diff #33203)

Is recovery supposed to work for -fsanitize=kernel-address? Should you mark it as Unrecoverable?

lib/asan/asan_flags.inc

144 ↗

(On Diff #33203)

This is really sad, but we tend to use halt_on_error runtime flag for that. This name is already used in TSan, MSan and UBSan.

lib/asan/asan_interface_internal.h

134 ↗

(On Diff #33203)

This function is also declared in public header: include/sanitizer/asan_interface.h (which is kind of stupid), so you have to fix it in both places. Looks like it's not an ABI break, though.

lib/asan/asan_internal.h

129 ↗

(On Diff #33203)

We have atomic_fetch_add

[samsonov]

In D12318#233103, @ygribov wrote:

Updated based on initial feedback. It seems we have way too many different names for recovery feature (UBSan's trap-on-error, MSan's keep-going, ASan's recover, noabort, etc.).

Right, naming is hard :( Speaking of user-visible flags, we should use -fsanitize-recover=address flag (UBSan's trap-on-error corresponds to -fsanitize-trap= and is slightly different).

[ygribov]

include/clang/Basic/Sanitizers.h

82 ↗

(On Diff #33203)

No, that's remains of some removed code.

lib/CodeGen/BackendUtil.cpp

214 ↗

(On Diff #33203)

KAsan is actually recoverable by default. Perhaps I should do explicit Recover for CompileKernel as well.

lib/asan/asan_internal.h

129 ↗

(On Diff #33203)

Yeah but it may be slower.

[filcab]

What about the file test/asan/TestCases/Darwin/interface_symbols_darwin.c? Shouldn't it be changed too?

lib/asan/asan_report.cc

695 ↗

(On Diff #33203)

Comment on line 632 would apply here too.

697 ↗

(On Diff #33203)

The usual "I see no benefit in using LIKELY here".

lib/asan/scripts/asan_symbolize.py

429 ↗

(On Diff #33203)

Why? I might have missed the reason for this change.

test/Driver/fsanitize.c

168 ↗

(On Diff #33203)

Shouldn't address be added here too?

[filcab]

lib/asan/asan_internal.h

125 ↗

(On Diff #33203)

Technically it might be unlikely, but this is noise in the code.
If we're not recovering, we will blow up anyway. Removing UNLIKELY would make the code easier to read with 0 practical difference in performance for any case.
In fact, it we're not silencing, *then* we want to be as fast as possible. (I'd go with the simple if, though)

129 ↗

(On Diff #33203)

Slower is better than undefined behavior, no? ;-)

lib/asan/asan_report.cc

632 ↗

(On Diff #33203)

cmpxchg should work, no?

646 ↗

(On Diff #33203)

Again, no benefit in tagging these as UNLIKELY.

652 ↗

(On Diff #33203)

At the very least, you can create a helper function with the code executed when you get the lock, no?

659 ↗

(On Diff #33203)

This comment should be changed, too. It's not true after this diff gets applied.

661 ↗

(On Diff #33203)

Why did you remove the first part of the comment?
It's still the reason why we have the sleep call here.

[ygribov]

Thanks a lot for detailed review!

What about the file test/asan/TestCases/Darwin/interface_symbols_darwin.c? Shouldn't it be changed too?

Yup.

lib/asan/asan_internal.h

125 ↗

(On Diff #33203)

Sorry, could you give more details? I indeed favor the default (non-recovery) case here. As for the UNLIKELY, we've seen cases where it significantly improved performance.

129 ↗

(On Diff #33203)

Again, I was probably too scary to slow down the default path. I'll switch to atomics in next version then.

lib/asan/asan_poisoning.cc

221 ↗

(On Diff #33203)

BTW what are these sanitizer_unaligned functions for? I don't think compiler emits then but they are also missing in public headers.

lib/asan/asan_report.cc

632 ↗

(On Diff #33203)

Yup.

652 ↗

(On Diff #33203)

Ah, so it's all about this tiny goto (not about hacky trial-and-error mutex acquisition). Ok, I'll put the code to a separate function.

659 ↗

(On Diff #33203)

Right.

661 ↗

(On Diff #33203)

I moved the first part of the comment above because it's applicable to both branches.

lib/asan/scripts/asan_symbolize.py

429 ↗

(On Diff #33203)

Hm, I don't remember doing this hunk at all. I'll double check when at work.

test/Driver/fsanitize.c

168 ↗

(On Diff #33203)

Ok.

[ygribov]

Updated based on reviews.

[filcab]

Still missing the Darwin interface symbols file.

Thanks for working on this.

Filipe

lib/asan/asan_interceptors.cc

73 ↗

(On Diff #33321)

Why false? Shouldn't it be the flag for halt_on_error?

lib/asan/asan_internal.h

125 ↗

(On Diff #33321)

So, I see two things that can happen:

halt_on_error is true:
We go through the if once (maybe more times, if we find several errors in several threads).
QuickCheckSilence returns false
LIKELY Isn't really doing much. Tweaking a jump that is executed once doesn't really matter.

halt_on_error is false:
We go through the if roughly max_errors times (let's say around 100, which seems to me is already a too big number for max_errors (if you actually manage to skip 100 errors and still execute code, that's a huge amount of luck!)

From these two, I would actually say that, at most, you should have an UNLIKELY, and not a LIKELY. *If* we execute that if more than once, then I should always expect it to be false.

Still, even in the case where we execute it "a bunch of times" we're still "almost never" executing it. QuickCheckSilence is called only when reporting errors, which means that:
If halt_on_error, then the program is terminating right after this report. We're not winning anything.
If halt_on_error, then the LIKELY is reversed. And we're still not winning much, we're not expecting this code to be executed more than max_errors, which, I'm assuming, will always be a small number.

My opinion is that it's not making the code faster, and it's making it less clear.

You mentioned performance increases. What tests were you running, and how?

131 ↗

(On Diff #33321)

You can change the previous atomic_load to be an atomic_fetch_add of 0 or 1, no?

test/Driver/fsanitize.c

168 ↗

(On Diff #33321)

Why didn't you update the number too? Does the test pass?

test/asan/TestCases/Linux/halt_on_error-torture.cc

1 ↗

(On Diff #33321)

Why is this test linux only?

[kcc]

lib/asan/asan_flags.inc

147 ↗

(On Diff #33321)

Shouldn't this go to sanitizer_common?

lib/asan/asan_interceptors.cc

52 ↗

(On Diff #33321)

why do we need it here?

lib/asan/asan_internal.h

124 ↗

(On Diff #33321)

Does it have to be static inline?
Also, I don't like the name. But it depends on another question later.

lib/asan/asan_report.cc

29 ↗

(On Diff #33321)

Don't define this macro, just use the C++ line as is

697 ↗

(On Diff #33321)

Agree.

1014 ↗

(On Diff #33321)

I think we should revert the name and the meaning of the flag:
make it "int recover" instead of "int fatal" to match the flag.
No?

Also, we need to make the parameter more clear.
One way is to declare a enum {ABORT_ON_ERROR, RECOVER_ON_ERROR} and pass it as the parameter.
Another is to
define two more internal functions
asan_report_error_and_abort() &
asan_report_error_and_maybe_recover()
and instead of calling __asan_report_error with a constant in the last parameter call one of these.

At the very least, write /*recover=*/false

test/asan/TestCases/Linux/halt_on_error-torture.cc

29 ↗

(On Diff #33321)

don't we want to check that multiple errors were reported?

check how the tests set ASAN_OPTIONS nowadays.
There were some massive changes...

44 ↗

(On Diff #33321)

use C++ new/delete

[ygribov]

Still missing the Darwin interface symbols file.

Right, I iterated Phab's comments and completely forgot about this item. I can add a test but I don't have Mac at hand so I'll have to XFAIL it if problem arises after commit.

lib/asan/asan_flags.inc

147 ↗

(On Diff #33321)

Hm, perhaps we should implement it for UBSan/MSan and then move to sanitizer_common?

lib/asan/asan_interceptors.cc

52 ↗

(On Diff #33321)

This enables fast path once error reporting limit is reached. Otherwise an error in a busy loop may cause significant (100x) performance degradation. One might argue that we should not be interested in continuing execution at all after we can't report more errors but in fully sanitized distribution it may be beneficial to continue execution to provoke errors in dependent processes.

73 ↗

(On Diff #33321)

This is covered by quick check above.

lib/asan/asan_internal.h

124 ↗

(On Diff #33321)

It doesn't but I thought this would be more efficient.

125 ↗

(On Diff #33321)

halt_on_error is true:
We go through the if once (maybe more times, if we find several errors in several threads).

I don't quite see it like this. Quick check may be called in non-fatal context (e.g. to enable fast path in ACCESS_MEMORY_RANGE). Information about likeliness may inform compiler to e.g. allocate registers in favor of preferred code or align it in a more appropriate way (e.g. to the same cache line).

You mentioned performance increases. What tests were you running, and how?

Check https://www.mail-archive.com/[e-mail]/msg00403.html . Adding unlikely hint to noreturning if improved code by some 40%.

131 ↗

(On Diff #33321)

Well yeah but this would add a risk of overflow.

lib/asan/asan_report.cc

29 ↗

(On Diff #33321)

Ok.

1014 ↗

(On Diff #33321)

Frankly I don't like the idea of adding fatal/recover parameter to public interface at all. Fatality should be controlled by ASAN_OPTIONS and be transparent for user code. Thoughts?

test/Driver/fsanitize.c

168 ↗

(On Diff #33321)

Yup, it did. I'll reexamine it.

test/asan/TestCases/Linux/halt_on_error-torture.cc

1 ↗

(On Diff #33321)

Mainly because I'm afraid that Mac/BSD maintainers will shout at me when I break their bots)

29 ↗

(On Diff #33321)

Ok.

44 ↗

(On Diff #33321)

Ok.

[ygribov]

Updated based on review. I haven't yet made torture test generic because I'm yet to test this patch on Mac (I plan to do this for next iteration).

[ygribov]

test/Driver/fsanitize.c

168 ↗

(On Diff #33570)

This should be ok - recovery is not enabled by default for ASan so resulting number didn't change.

[ygribov]

Folks, any feedback on this?

[kcc]

The patch is already too long and has too many comments.
Some parts are less controversial, some are more.
I suggest you to split it at this point and start reviewing/submitting it in parts.
Probably start from the driver/flags changes.

lib/Transforms/Instrumentation/AddressSanitizer.cpp

1099 ↗

(On Diff #33570)

I think we use {} in such cases (if followed by an else with {})

lib/asan/asan_flags.inc

141 ↗

(On Diff #33570)

No, just add it to sanitizer_common and write something like ("currently supported by asan") in the description.

If we add things that are potentially useful to other sanitizers to one of them (not to sanitizer_common)
we then end up having duplicates in all sanitizers.

lib/asan/asan_internal.h

122 ↗

(On Diff #33570)

I don't like the name,
I don't like the fact that it has a parameter,
I don't like that it's static inline,
I don't like its behavior.

We have two situations:

1. We are reporting an error and don't want to do it (is_error=true).

There is no harm in making this function non-inline in this case.
bool WantToReportMoreError();

2. We don't know whether there will be an error, but we are shutting down the checking functionality

just in case (or to win a bit of performance).
I don't think we need to do this at all.
The win is not clear, but the added complexity is.

lib/asan/asan_report.cc

626 ↗

(On Diff #33570)

This is all exceptionally horrible.
I've spent quite a bit of time and failed to understand how it works,
also I don't understand how to test it.
Why not just grab a lock for the entire duration of error reporting?

The current logic was there because we never reported more than one bug,
now that we do -- simply grab a lock!

698 ↗

(On Diff #33570)

class members have the _ suffix.

lib/asan/asan_rtl.cc

126 ↗

(On Diff #33570)

please leave __asan_report_error here. It is used as a target for break points in debugger.

You may then check halt_on_error inside __asan_report_error

[kcc]

One more interesting question that came to my mind: is this going to be async-signal-safe?
If you think it will, please add tests.
If not, the whole idea is kind of questionable.

[ygribov]

Some parts are less controversial, some are more.

What's your general impression on the feature though?

I suggest you to split it at this point and start reviewing/submitting it in parts.
Probably start from the driver/flags changes.

Hm but I don't see any functionality which can be moved to independent commit (except the flags part which is really minor). We could split to separate patches for rt, clang and llvm but I don't see how this is going to simplify review.

lib/asan/asan_flags.inc

141 ↗

(On Diff #33570)

If we add things that are potentially useful to other sanitizers to one of them (not to sanitizer_common)
we then end up having duplicates in all sanitizers.

That makes sense, we already have too many semi-duplicate flags.

lib/asan/asan_internal.h

122 ↗

(On Diff #33570)

Problem is that lack of this check would cause program to crawl in some cases. But ok, this may live in a private branch.

lib/asan/asan_report.cc

626 ↗

(On Diff #33570)

also I don't understand how to test it.

Did you ever understand how to test original code :) ? The behavior for fatal errors didn't change by the way.

Why not just grab a lock for the entire duration of error reporting?

Because we may run into second error while reporting the first one in the same thread. Grabbing a lock would cause a deadlock in this case.

698 ↗

(On Diff #33570)

I'm not sure that's true for statics, I see numerous places where they aren't undered.

[ygribov]

One more interesting question that came to my mind: is this going to be async-signal-safe?
If not, the whole idea is kind of questionable.

I don't see why it shouldn't - the only problematic place is acquiring the mutex which will either wait (in case reporting_tid is different from current_tid) or fail with informative message after timeout.

If you think it will, please add tests.

I could generate errors in endless loop in a thread and bombard it with signals but that's probably not very reliable.

[kcc]

In D12318#240258, @ygribov wrote:

One more interesting question that came to my mind: is this going to be async-signal-safe?
If not, the whole idea is kind of questionable.

I don't see why it shouldn't - the only problematic place is acquiring the mutex which will either wait (in case reporting_tid is different from current_tid) or fail with informative message after timeout.

If you think it will, please add tests.

I could generate errors in endless loop in a thread and bombard it with signals but that's probably not very reliable.

The more interesting would be to test the case where we have asan-ish reports in signal handlers (and at the same time in regular code).

[kcc]

What's your general impression on the feature though?

Same as before.
It is an anti-feature for the majority of users and it will bring much additional complexity to the code
and extra maintenance cost.

I am willing to go forward with it because for some (still obscure) reasons you need this feature
and because your contribution is valuable.

[kcc]

Hm but I don't see any functionality which can be moved to independent commit (except the flags part which is really minor). We could split to separate patches for rt, clang and llvm but I don't see how this is going to simplify review.

Review is an O(N^2) process. The smaller are the patches the quicker you get through.

[kcc]

lib/asan/asan_report.cc

626 ↗

(On Diff #33570)

Did you ever understand how to test original code :) ?

That's not an argument. We don't want to make things worse just because they are not good at the moment.

Because we may run into second error while reporting the first one in the same thread. Grabbing a lock would cause a deadlock in this case.

Yea. Async signal safety...

[ygribov]

Folks,

In case anyone wants this feature it's probably good time to speak up. If there's no interest, I'll probably abandon this revision in a few days.

-Y

[eugenis]

I think this is a useful feature but I'm a bit worried about the implementation complexity.

We've been running entire Android devices with ASan, and having them crash at boot due to a single bug is moderately annoying.

For the reference, https://source.android.com/devices/tech/debug/asan.html#sanitize_target.

[ygribov]

I think this is a useful feature but I'm a bit worried about the implementation complexity.

You mean the report lock part? I agree, although I'm not sure how to simplify it. Perhaps always abort in case of conflict?

We've been running entire Android devices with ASan, and having them crash at boot due to a single bug is moderately annoying.

Yup, that's my use-case as well.

[kcc]

Let's try to finish it.
I won't be able to participate in the review before 28-th, but feel free to
continue/finish w/o me.

[eugenis]

lib/asan/asan_report.cc

649 ↗

(On Diff #33570)

I think this timeout complicates things without any benefit. Why not wait indefinitely?

I think the logic should be like this:

1. if the current thread is already reporting, die immediately
2. if (recover) {lock()} else { if (!try_lock()) sleep(inf) }
3. report
4. unlock()

1006–1007 ↗

(On Diff #33570)

I agree, if the outside caller thinks the error is fatal, it can just abort() or something.

[samsonov]

lib/asan/asan_internal.h

122 ↗

(On Diff #33570)

The name is really confusing. It should be smth. like IgnoreError. Also, instead of passing is_error as a parameter, and doing conditional atomic_fetch_add, can you just split this function into two: one would check if need to ignore the error (halt_on_error is true, or number of reported errors is too large already), and another one would increment the number of detected errors. (could you then call the latter one from ScopedInErrorReport constructor)?

Can you make it ALWAYS_INLINE?

lib/asan/asan_report.cc

698 ↗

(On Diff #33570)

There are no trailing _ in simple structs, but there should be in classes.

[ygribov]

Sorry, I got distracted by other projects. I'm mostly done with all remarks with just one issue (can't commonize halt_on_error).

lib/asan/asan_flags.inc

141 ↗

(On Diff #33570)

Here's the problem - we need two halt_on_error's for ASan and integrated UBSan. Another problem is that msan does some local magic to alias current halt_on_error to keep_going flag.

lib/asan/asan_report.cc

626 ↗

(On Diff #33570)

Funny enough, even current ASan isn't asynch signal stable. If you have repeated error in nodeferred signals which are being send to a thread, it may deadlock during error reporting in ScopedInErrorLock. The sequence is roughly:

• kernel delivers signal
• trigger error
• lock report file
• call internal_write
• enter kernel
• kernel delivers next nodeferred signal
• trigger the same error
• etc.

[ygribov]

I've addressed the remarks from previous review except the one for halt_on_error (we want separate halt_on_error for ASan and UBSan). The llvm and clang parts of the patch will be submitted for review separately per Kostya's request.

[ygribov]

BTW I haven't yet ported tests to Darwin, this is something tbd.

[ygribov]

lib/asan/asan_report.cc

644 ↗

(On Diff #38931)

This could be much simpler if we had recursive locks.

[eugenis]

LGTM
Thanks for working on this!

[ygribov]

BTW I haven't yet ported tests to Darwin, this is something tbd.

Filipe, Kuba, the feature has been merged to upstream and seems to work robustly. But I don't think I'll have time to test it on OS X in near future (I don't have easy access to hardware and my customer does not need it). So I was wondering whether you could help? I don't expect any errors, just maybe some really simple portability fixes.

[kubamracek]

In D12318#290215, @ygribov wrote:

BTW I haven't yet ported tests to Darwin, this is something tbd.

Filipe, Kuba, the feature has been merged to upstream and seems to work robustly. But I don't think I'll have time to test it on OS X in near future (I don't have easy access to hardware and my customer does not need it). So I was wondering whether you could help? I don't expect any errors, just maybe some really simple portability fixes.

Hi, are you talking about the halt_on_error-signals and halt_on_error-torture testcases? On my OS X machine, they both pass, so you might just want to move them out of the Linux/ directory. Let me know if the OS X buildbot fails and I'll take a look at it.

[ygribov]

On my OS X machine, they both pass, so you might just want to move them out of the Linux/ directory

Great, thanks!

compiler-rt/trunk/lib/asan/asan_flags.inc

ASAN_FLAG(
     "See https://code.google.com/p/address-sanitizer/wiki/ContainerOverflow")
 ASAN_FLAG(int, detect_odr_violation, 2,
           "If >=2, detect violation of One-Definition-Rule (ODR); "
           "If ==1, detect ODR-violation only if the two variables "
           "have different sizes")
 ASAN_FLAG(bool, dump_instruction_bytes, false,
           "If true, dump 16 bytes starting at the instruction that caused SEGV")
 ASAN_FLAG(const char *, suppressions, "", "Suppressions file name.")
+ASAN_FLAG(bool, halt_on_error, true,
+          "Crash the program after printing the first error report "
+          "(WARNING: USE AT YOUR OWN RISK!)")

compiler-rt/trunk/lib/asan/asan_interceptors.cc

if (!QuickCheckForUnpoisonedRegion(__offset, __size) &&             \
         suppressed = IsInterceptorSuppressed(_ctx->interceptor_name);   \
         if (!suppressed && HaveStackTraceBasedSuppressions()) {         \
           GET_STACK_TRACE_FATAL_HERE;                                   \
           suppressed = IsStackTraceSuppressed(&stack);                  \
         }                                                               \
       }                                                                 \
       if (!suppressed) {                                                \
         GET_CURRENT_PC_BP_SP;                                           \
-        __asan_report_error(pc, bp, sp, __bad, isWrite, __size, 0);     \
+        ReportGenericError(pc, bp, sp, __bad, isWrite, __size, 0, false);\
       }                                                                 \
     }                                                                   \
   } while (0)
 
 #define ASAN_READ_RANGE(ctx, offset, size) \
   ACCESS_MEMORY_RANGE(ctx, offset, size, false)
 #define ASAN_WRITE_RANGE(ctx, offset, size) \
   ACCESS_MEMORY_RANGE(ctx, offset, size, true)

compiler-rt/trunk/lib/asan/asan_interface_internal.h

extern "C" {
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_store1(uptr p);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_store2(uptr p);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_store4(uptr p);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_store8(uptr p);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_store16(uptr p);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_loadN(uptr p, uptr size);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_storeN(uptr p, uptr size);
 
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_load1_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_load2_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_load4_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_load8_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_load16_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_store1_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_store2_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_store4_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_store8_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_store16_noabort(uptr p);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_loadN_noabort(uptr p, uptr size);
+  SANITIZER_INTERFACE_ATTRIBUTE void __asan_storeN_noabort(uptr p, uptr size);
+
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_load1(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_load2(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_load4(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_load8(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_load16(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_store1(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_store2(uptr p, u32 exp);
   SANITIZER_INTERFACE_ATTRIBUTE void __asan_exp_store4(uptr p, u32 exp);

compiler-rt/trunk/lib/asan/asan_report.h

 bool DescribeAddressIfShadow(uptr addr, AddressDescription *descr = nullptr,
                              bool print = true);
 bool ParseFrameDescription(const char *frame_descr,
                            InternalMmapVector<StackVarDescr> *vars);
 bool DescribeAddressIfStack(uptr addr, uptr access_size);
 void DescribeThread(AsanThreadContext *context);
 
 // Different kinds of error reports.
-void NORETURN ReportStackOverflow(const SignalContext &sig);
-void NORETURN ReportDeadlySignal(const char* description,
-                                 const SignalContext &sig);
-void NORETURN ReportNewDeleteSizeMismatch(uptr addr, uptr delete_size,
-                                          BufferedStackTrace *free_stack);
-void NORETURN ReportDoubleFree(uptr addr, BufferedStackTrace *free_stack);
-void NORETURN ReportFreeNotMalloced(uptr addr, BufferedStackTrace *free_stack);
-void NORETURN ReportAllocTypeMismatch(uptr addr, BufferedStackTrace *free_stack,
-                                      AllocType alloc_type,
-                                      AllocType dealloc_type);
-void NORETURN
-    ReportMallocUsableSizeNotOwned(uptr addr, BufferedStackTrace *stack);
-void NORETURN
-    ReportSanitizerGetAllocatedSizeNotOwned(uptr addr,
+void ReportGenericError(uptr pc, uptr bp, uptr sp, uptr addr, bool is_write,
+                        uptr access_size, u32 exp, bool fatal);
+void ReportStackOverflow(const SignalContext &sig);
+void ReportDeadlySignal(const char *description, const SignalContext &sig);
+void ReportNewDeleteSizeMismatch(uptr addr, uptr delete_size,
+void ReportDoubleFree(uptr addr, BufferedStackTrace *free_stack);
+void ReportFreeNotMalloced(uptr addr, BufferedStackTrace *free_stack);
+void ReportAllocTypeMismatch(uptr addr, BufferedStackTrace *free_stack,
+void ReportMallocUsableSizeNotOwned(uptr addr, BufferedStackTrace *stack);
+void ReportSanitizerGetAllocatedSizeNotOwned(uptr addr,
                                              BufferedStackTrace *stack);
-void NORETURN
-    ReportStringFunctionMemoryRangesOverlap(const char *function,
+void ReportStringFunctionMemoryRangesOverlap(const char *function,
                                              const char *offset1, uptr length1,
                                              const char *offset2, uptr length2,
                                              BufferedStackTrace *stack);
-void NORETURN ReportStringFunctionSizeOverflow(uptr offset, uptr size,
-                                               BufferedStackTrace *stack);
-void NORETURN
-    ReportBadParamsToAnnotateContiguousContainer(uptr beg, uptr end,
+void ReportStringFunctionSizeOverflow(uptr offset, uptr size,
+void ReportBadParamsToAnnotateContiguousContainer(uptr beg, uptr end,
                                                   uptr old_mid, uptr new_mid,
                                                   BufferedStackTrace *stack);
 
-void NORETURN
-ReportODRViolation(const __asan_global *g1, u32 stack_id1,
+void ReportODRViolation(const __asan_global *g1, u32 stack_id1,
                         const __asan_global *g2, u32 stack_id2);
 
 // Mac-specific errors and warnings.
 void WarnMacFreeUnallocated(uptr addr, uptr zone_ptr, const char *zone_name,
                             BufferedStackTrace *stack);
-void NORETURN ReportMacMzReallocUnknown(uptr addr, uptr zone_ptr,
-                                        const char *zone_name,
-                                        BufferedStackTrace *stack);
-void NORETURN ReportMacCfReallocUnknown(uptr addr, uptr zone_ptr,
-                                        const char *zone_name,
-                                        BufferedStackTrace *stack);
+void ReportMacMzReallocUnknown(uptr addr, uptr zone_ptr,
+void ReportMacCfReallocUnknown(uptr addr, uptr zone_ptr,
 
 }  // namespace __asan

compiler-rt/trunk/lib/asan/asan_report.cc

 }
 
 // -------------------- Different kinds of reports ----------------- {{{1
 
 // Use ScopedInErrorReport to run common actions just before and
 // immediately after printing error report.
 class ScopedInErrorReport {
  public:
-  explicit ScopedInErrorReport(ReportData *report = nullptr) {
-    static atomic_uint32_t num_calls;
-    static u32 reporting_thread_tid;
-    if (atomic_fetch_add(&num_calls, 1, memory_order_relaxed) != 0) {
+  explicit ScopedInErrorReport(ReportData *report = nullptr,
+                               bool fatal = false) {
+    halt_on_error_ = fatal || flags()->halt_on_error;
+
+    if (lock_.TryLock()) {
+      StartReporting(report);
+      return;
+    }
+
+    // ASan found two bugs in different threads simultaneously.
+
+    u32 current_tid = GetCurrentTidOrInvalid();
+    if (current_tid == reporting_thread_tid_ || current_tid == kInvalidTid) {
+      // This is either asynch signal or nested error during error reporting.
+      // Fail fast to avoid deadlocks.
+
+      // Can't use Report() here because of potential deadlocks
+      // in nested signal handlers.
+      const char msg[] = "AddressSanitizer: nested bug in the same thread, "
+                         "aborting.\n";
+      WriteToFile(kStderrFd, msg, sizeof(msg));
+
+      internal__exit(common_flags()->exitcode);
+    }
+
+    if (halt_on_error_) {
       // Do not print more than one report, otherwise they will mix up.
       // Error reporting functions shouldn't return at this situation, as
-      // they are defined as no-return.
+      // they are effectively no-returns.
+
       Report("AddressSanitizer: while reporting a bug found another one. "
-                 "Ignoring.\n");
-      u32 current_tid = GetCurrentTidOrInvalid();
-      if (current_tid != reporting_thread_tid) {
-        // ASan found two bugs in different threads simultaneously. Sleep
-        // long enough to make sure that the thread which started to print
-        // an error report will finish doing it.
-        SleepForSeconds(Max(100, flags()->sleep_before_dying + 1));
-      }
+
+      // Sleep long enough to make sure that the thread which started
+      // to print an error report will finish doing it.
+
       // If we're still not dead for some reason, use raw _exit() instead of
       // Die() to bypass any additional checks.
       internal__exit(common_flags()->exitcode);
+    } else {
+      // The other thread will eventually finish reporting
+      // so it's safe to wait
+      lock_.Lock();
     }
-    if (report) report_data = *report;
-    report_happened = true;
-    ASAN_ON_ERROR();
-    // Make sure the registry and sanitizer report mutexes are locked while
-    // we're printing an error report.
-    // We can lock them only here to avoid self-deadlock in case of
-    // recursive reports.
-    asanThreadRegistry().Lock();
-    CommonSanitizerReportMutex.Lock();
-    reporting_thread_tid = GetCurrentTidOrInvalid();
-    Printf("===================================================="
-           "=============\n");
+
+    StartReporting(report);
   }
-  // Destructor is NORETURN, as functions that report errors are.
-  NORETURN ~ScopedInErrorReport() {
+
+  ~ScopedInErrorReport() {
     // Make sure the current thread is announced.
     DescribeThread(GetCurrentThread());
     // We may want to grab this lock again when printing stats.
     asanThreadRegistry().Unlock();
     // Print memory stats.
     if (flags()->print_stats)
       __asan_print_accumulated_stats();
     if (error_report_callback) {
       error_report_callback(error_message_buffer);
     }
+    CommonSanitizerReportMutex.Unlock();
+    reporting_thread_tid_ = kInvalidTid;
+    lock_.Unlock();
+    if (halt_on_error_) {
       Report("ABORTING\n");
       Die();
     }
+  }
+
+ private:
+  void StartReporting(ReportData *report) {
+    if (report) report_data = *report;
+    report_happened = true;
+    ASAN_ON_ERROR();
+    // Make sure the registry and sanitizer report mutexes are locked while
+    // we're printing an error report.
+    // We can lock them only here to avoid self-deadlock in case of
+    // recursive reports.
+    asanThreadRegistry().Lock();
+    CommonSanitizerReportMutex.Lock();
+    reporting_thread_tid_ = GetCurrentTidOrInvalid();
+    Printf("===================================================="
+           "=============\n");
+  }
+
+  static StaticSpinMutex lock_;
+  static u32 reporting_thread_tid_;
+  bool halt_on_error_;
 };
 
+StaticSpinMutex ScopedInErrorReport::lock_;
+u32 ScopedInErrorReport::reporting_thread_tid_;
+
 void ReportStackOverflow(const SignalContext &sig) {
   ScopedInErrorReport in_report;
   Decorator d;
   Printf("%s", d.Warning());
   Report(
       "ERROR: AddressSanitizer: stack-overflow on address %p"
       " (pc %p bp %p sp %p T%d)\n",
       (void *)sig.addr, (void *)sig.pc, (void *)sig.bp, (void *)sig.sp,
       GetCurrentTidOrInvalid());
   Printf("%s", d.EndWarning());
   GET_STACK_TRACE_SIGNAL(sig);
   stack.Print();
   ReportErrorSummary("stack-overflow", &stack);
 }
 
 void ReportDeadlySignal(const char *description, const SignalContext &sig) {
-  ScopedInErrorReport in_report;
+  ScopedInErrorReport in_report(/*report*/nullptr, /*fatal*/true);
   Decorator d;
   Printf("%s", d.Warning());
   Report(
       "ERROR: AddressSanitizer: %s on unknown address %p"
       " (pc %p bp %p sp %p T%d)\n",
       description, (void *)sig.addr, (void *)sig.pc, (void *)sig.bp,
       (void *)sig.sp, GetCurrentTidOrInvalid());
   if (sig.pc < GetPageSizeCached()) {
switch (*shadow_addr) {
       case kAsanAllocaRightMagic:
         bug_descr = "dynamic-stack-buffer-overflow";
         break;
     }
   }
 
   ReportData report = { pc, sp, bp, addr, (bool)is_write, access_size,
                         bug_descr };
-  ScopedInErrorReport in_report(&report);
+  ScopedInErrorReport in_report(&report, fatal);
 
   Decorator d;
   Printf("%s", d.Warning());
   Report("ERROR: AddressSanitizer: %s on address "
              "%p at pc %p bp %p sp %p\n",
              bug_descr, (void*)addr, pc, bp, sp);
   Printf("%s", d.EndWarning());
 
void __asan_report_error(uptr pc, uptr bp, uptr sp, uptr addr, int is_write,
   GET_STACK_TRACE_FATAL(pc, bp);
   stack.Print();
 
   DescribeAddress(addr, access_size, bug_descr);
   ReportErrorSummary(bug_descr, &stack);
   PrintShadowMemoryForAddress(addr);
 }
 
+}  // namespace __asan
+
+// --------------------------- Interface --------------------- {{{1
+using namespace __asan;  // NOLINT
+
+void __asan_report_error(uptr pc, uptr bp, uptr sp, uptr addr, int is_write,
+                         uptr access_size, u32 exp) {
+  ENABLE_FRAME_POINTER;
+  bool fatal = flags()->halt_on_error;
+  ReportGenericError(pc, bp, sp, addr, is_write, access_size, exp, fatal);
+}
+
 void NOINLINE __asan_set_error_report_callback(void (*callback)(const char*)) {
   error_report_callback = callback;
   if (callback) {
     error_message_buffer_size = 1 << 16;
     error_message_buffer =
         (char*)MmapOrDie(error_message_buffer_size, __func__);
     error_message_buffer_pos = 0;
   }

compiler-rt/trunk/lib/asan/asan_rtl.cc

 }
 
 // -------------------------- Run-time entry ------------------- {{{1
 // exported functions
 #define ASAN_REPORT_ERROR(type, is_write, size)                     \
 extern "C" NOINLINE INTERFACE_ATTRIBUTE                             \
 void __asan_report_ ## type ## size(uptr addr) {                    \
   GET_CALLER_PC_BP_SP;                                              \
-  __asan_report_error(pc, bp, sp, addr, is_write, size, 0);         \
+  ReportGenericError(pc, bp, sp, addr, is_write, size, 0, true);    \
 }                                                                   \
 extern "C" NOINLINE INTERFACE_ATTRIBUTE                             \
 void __asan_report_exp_ ## type ## size(uptr addr, u32 exp) {       \
   GET_CALLER_PC_BP_SP;                                              \
-  __asan_report_error(pc, bp, sp, addr, is_write, size, exp);       \
-}
+  ReportGenericError(pc, bp, sp, addr, is_write, size, exp, true);  \
+}                                                                   \
+extern "C" NOINLINE INTERFACE_ATTRIBUTE                             \
+void __asan_report_ ## type ## size ## _noabort(uptr addr) {        \
+  GET_CALLER_PC_BP_SP;                                              \
+  ReportGenericError(pc, bp, sp, addr, is_write, size, 0, false);   \
+}                                                                   \
 
 ASAN_REPORT_ERROR(load, false, 1)
 ASAN_REPORT_ERROR(load, false, 2)
 ASAN_REPORT_ERROR(load, false, 4)
 ASAN_REPORT_ERROR(load, false, 8)
 ASAN_REPORT_ERROR(load, false, 16)
 ASAN_REPORT_ERROR(store, true, 1)
 ASAN_REPORT_ERROR(store, true, 2)
 ASAN_REPORT_ERROR(store, true, 4)
 ASAN_REPORT_ERROR(store, true, 8)
 ASAN_REPORT_ERROR(store, true, 16)
 
-#define ASAN_REPORT_ERROR_N(type, is_write)                    \
-extern "C" NOINLINE INTERFACE_ATTRIBUTE                        \
-void __asan_report_ ## type ## _n(uptr addr, uptr size) {      \
-  GET_CALLER_PC_BP_SP;                                         \
-  __asan_report_error(pc, bp, sp, addr, is_write, size, 0);    \
-}                                                              \
-extern "C" NOINLINE INTERFACE_ATTRIBUTE                        \
+#define ASAN_REPORT_ERROR_N(type, is_write)                                 \
+extern "C" NOINLINE INTERFACE_ATTRIBUTE                                     \
+void __asan_report_ ## type ## _n(uptr addr, uptr size) {                   \
+  GET_CALLER_PC_BP_SP;                                                      \
+  ReportGenericError(pc, bp, sp, addr, is_write, size, 0, true);            \
+}                                                                           \
+extern "C" NOINLINE INTERFACE_ATTRIBUTE                                     \
 void __asan_report_exp_ ## type ## _n(uptr addr, uptr size, u32 exp) {      \
   GET_CALLER_PC_BP_SP;                                                      \
-  __asan_report_error(pc, bp, sp, addr, is_write, size, exp);               \
-}
+  ReportGenericError(pc, bp, sp, addr, is_write, size, exp, true);          \
+}                                                                           \
+extern "C" NOINLINE INTERFACE_ATTRIBUTE                                     \
+void __asan_report_ ## type ## _n_noabort(uptr addr, uptr size) {           \
+  GET_CALLER_PC_BP_SP;                                                      \
+  ReportGenericError(pc, bp, sp, addr, is_write, size, 0, false);           \
+}                                                                           \
 
 ASAN_REPORT_ERROR_N(load, false)
 ASAN_REPORT_ERROR_N(store, true)
 
-#define ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp_arg)        \
+#define ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp_arg, fatal) \
     uptr sp = MEM_TO_SHADOW(addr);                                             \
     uptr s = size <= SHADOW_GRANULARITY ? *reinterpret_cast<u8 *>(sp)          \
                                         : *reinterpret_cast<u16 *>(sp);        \
     if (UNLIKELY(s)) {                                                         \
       if (UNLIKELY(size >= SHADOW_GRANULARITY ||                               \
                    ((s8)((addr & (SHADOW_GRANULARITY - 1)) + size - 1)) >=     \
                        (s8)s)) {                                               \
         if (__asan_test_only_reported_buggy_pointer) {                         \
           *__asan_test_only_reported_buggy_pointer = addr;                     \
         } else {                                                               \
           GET_CALLER_PC_BP_SP;                                                 \
-          __asan_report_error(pc, bp, sp, addr, is_write, size, exp_arg);      \
+          ReportGenericError(pc, bp, sp, addr, is_write, size, exp_arg,        \
+                              fatal);                                          \
         }                                                                      \
       }                                                                        \
     }
 
 #define ASAN_MEMORY_ACCESS_CALLBACK(type, is_write, size)                      \
   extern "C" NOINLINE INTERFACE_ATTRIBUTE                                      \
   void __asan_##type##size(uptr addr) {                                        \
-    ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, 0)                  \
+    ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, 0, true)            \
   }                                                                            \
   extern "C" NOINLINE INTERFACE_ATTRIBUTE                                      \
   void __asan_exp_##type##size(uptr addr, u32 exp) {                           \
-    ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp)                \
-  }
+    ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp, true)          \
+  }                                                                            \
+  extern "C" NOINLINE INTERFACE_ATTRIBUTE                                      \
+  void __asan_##type##size ## _noabort(uptr addr) {                            \
+    ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, 0, false)           \
+  }                                                                            \
 
 ASAN_MEMORY_ACCESS_CALLBACK(load, false, 1)
 ASAN_MEMORY_ACCESS_CALLBACK(load, false, 2)
 ASAN_MEMORY_ACCESS_CALLBACK(load, false, 4)
 ASAN_MEMORY_ACCESS_CALLBACK(load, false, 8)
 ASAN_MEMORY_ACCESS_CALLBACK(load, false, 16)
 ASAN_MEMORY_ACCESS_CALLBACK(store, true, 1)
 ASAN_MEMORY_ACCESS_CALLBACK(store, true, 2)
 ASAN_MEMORY_ACCESS_CALLBACK(store, true, 4)
 ASAN_MEMORY_ACCESS_CALLBACK(store, true, 8)
 ASAN_MEMORY_ACCESS_CALLBACK(store, true, 16)
 
 extern "C"
 NOINLINE INTERFACE_ATTRIBUTE
 void __asan_loadN(uptr addr, uptr size) {
   if (__asan_region_is_poisoned(addr, size)) {
     GET_CALLER_PC_BP_SP;
-    __asan_report_error(pc, bp, sp, addr, false, size, 0);
+    ReportGenericError(pc, bp, sp, addr, false, size, 0, true);
   }
 }
 
 extern "C"
 NOINLINE INTERFACE_ATTRIBUTE
 void __asan_exp_loadN(uptr addr, uptr size, u32 exp) {
   if (__asan_region_is_poisoned(addr, size)) {
     GET_CALLER_PC_BP_SP;
-    __asan_report_error(pc, bp, sp, addr, false, size, exp);
+    ReportGenericError(pc, bp, sp, addr, false, size, exp, true);
+  }
+}
+
+extern "C"
+NOINLINE INTERFACE_ATTRIBUTE
+void __asan_loadN_noabort(uptr addr, uptr size) {
+  if (__asan_region_is_poisoned(addr, size)) {
+    GET_CALLER_PC_BP_SP;
+    ReportGenericError(pc, bp, sp, addr, false, size, 0, false);
   }
 }
 
 extern "C"
 NOINLINE INTERFACE_ATTRIBUTE
 void __asan_storeN(uptr addr, uptr size) {
   if (__asan_region_is_poisoned(addr, size)) {
     GET_CALLER_PC_BP_SP;
-    __asan_report_error(pc, bp, sp, addr, true, size, 0);
+    ReportGenericError(pc, bp, sp, addr, true, size, 0, true);
   }
 }
 
 extern "C"
 NOINLINE INTERFACE_ATTRIBUTE
 void __asan_exp_storeN(uptr addr, uptr size, u32 exp) {
   if (__asan_region_is_poisoned(addr, size)) {
     GET_CALLER_PC_BP_SP;
-    __asan_report_error(pc, bp, sp, addr, true, size, exp);
+    ReportGenericError(pc, bp, sp, addr, true, size, exp, true);
+  }
+}
+
+extern "C"
+NOINLINE INTERFACE_ATTRIBUTE
+void __asan_storeN_noabort(uptr addr, uptr size) {
+  if (__asan_region_is_poisoned(addr, size)) {
+    GET_CALLER_PC_BP_SP;
+    ReportGenericError(pc, bp, sp, addr, true, size, 0, false);
   }
 }
 
 // Force the linker to keep the symbols for various ASan interface functions.
 // We want to keep those in the executable in order to let the instrumented
 // dynamic libraries access the symbol even if it is not used by the executable
 // itself. This should help if the build system is removing dead code at link
 // time.

compiler-rt/trunk/test/asan/TestCases/Darwin/interface_symbols_darwin.c

 // RUN: echo __asan_report_load16 >> %t.interface
 // RUN: echo __asan_report_store1 >> %t.interface
 // RUN: echo __asan_report_store2 >> %t.interface
 // RUN: echo __asan_report_store4 >> %t.interface
 // RUN: echo __asan_report_store8 >> %t.interface
 // RUN: echo __asan_report_store16 >> %t.interface
 // RUN: echo __asan_report_load_n >> %t.interface
 // RUN: echo __asan_report_store_n >> %t.interface
+// RUN: echo __asan_report_load1_noabort >> %t.interface
+// RUN: echo __asan_report_load2_noabort >> %t.interface
+// RUN: echo __asan_report_load4_noabort >> %t.interface
+// RUN: echo __asan_report_load8_noabort >> %t.interface
+// RUN: echo __asan_report_load16_noabort >> %t.interface
+// RUN: echo __asan_report_store1_noabort >> %t.interface
+// RUN: echo __asan_report_store2_noabort >> %t.interface
+// RUN: echo __asan_report_store4_noabort >> %t.interface
+// RUN: echo __asan_report_store8_noabort >> %t.interface
+// RUN: echo __asan_report_store16_noabort >> %t.interface
+// RUN: echo __asan_report_load_n_noabort >> %t.interface
+// RUN: echo __asan_report_store_n_noabort >> %t.interface
 // RUN: echo __asan_report_exp_load1 >> %t.interface
 // RUN: echo __asan_report_exp_load2 >> %t.interface
 // RUN: echo __asan_report_exp_load4 >> %t.interface
 // RUN: echo __asan_report_exp_load8 >> %t.interface
 // RUN: echo __asan_report_exp_load16 >> %t.interface
 // RUN: echo __asan_report_exp_store1 >> %t.interface
 // RUN: echo __asan_report_exp_store2 >> %t.interface
 // RUN: echo __asan_report_exp_store4 >> %t.interface

compiler-rt/trunk/test/asan/TestCases/Linux/halt_on_error-torture.cc

+// Stress test recovery mode with many threads.
+//
+// RUN: %clangxx_asan -fsanitize-recover=address %s -o %t
+//
+// RUN: env ASAN_OPTIONS=halt_on_error=false:max_errors=1000 %run %t 1 10 >1.txt 2>&1
+// RUN: FileCheck %s < 1.txt
+// RUN: [ $(wc -l < 1.txt) -gt 1 ]
+//
+// RUN: env ASAN_OPTIONS=halt_on_error=false:max_errors=1000 %run %t 10 20 >10.txt 2>&1
+// RUN: FileCheck %s < 10.txt
+// This one is racy although very unlikely to fail:
+// RUN: [ $(wc -l < 10.txt) -gt 1 ]
+// Collisions are highly unlikely but still possible so we need the alternative:
+// RUN: FileCheck --check-prefix=CHECK-COLLISION %s < 1.txt || FileCheck --check-prefix=CHECK-NO-COLLISION %s < 1.txt
+//
+// REQUIRES: stable-runtime
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <pthread.h>
+#include <time.h>
+
+#include <sanitizer/asan_interface.h>
+
+size_t nthreads = 10;
+size_t niter = 10;
+
+void *run(void *arg) {
+  unsigned seed = (unsigned)(size_t)arg;
+
+  volatile char tmp[2];
+  __asan_poison_memory_region(&tmp, sizeof(tmp)); 
+
+  for (size_t i = 0; i < niter; ++i) {
+    struct timespec delay = { 0, rand_r(&seed) * 1000000 };
+    nanosleep(&delay, 0);
+
+    // Expect error collisions here
+    // CHECK: AddressSanitizer: use-after-poison
+    volatile int idx = 0;
+    tmp[idx] = 0;
+  }
+
+  return 0;
+}
+
+int main(int argc, char **argv) {
+  if (argc != 3) {
+    fprintf(stderr, "Syntax: %s nthreads niter\n", argv[0]);
+    exit(1);
+  }
+
+  nthreads = (size_t)strtoul(argv[1], 0, 0);
+  niter = (size_t)strtoul(argv[2], 0, 0);
+
+  pthread_t *tids = new pthread_t[nthreads];
+
+  for (size_t i = 0; i < nthreads; ++i) {
+    if (0 != pthread_create(&tids[i], 0, run, (void *)i)) {
+      fprintf(stderr, "Failed to create thread\n");
+      exit(1);
+    }
+  }
+
+  for (size_t i = 0; i < nthreads; ++i) {
+    if (0 != pthread_join(tids[i], 0)) {
+      fprintf(stderr, "Failed to join thread\n");
+      exit(1);
+    }
+  }
+
+  // CHECK-COLLISION: AddressSanitizer: nested bug in the same thread, aborting
+  // CHECK-NO-COLLISION: All threads terminated
+  printf("All threads terminated\n");
+
+  delete [] tids;
+
+  return 0;
+}

compiler-rt/trunk/test/asan/TestCases/Linux/interface_symbols_linux.c

 // RUN: echo __asan_report_load16 >> %t.interface
 // RUN: echo __asan_report_store1 >> %t.interface
 // RUN: echo __asan_report_store2 >> %t.interface
 // RUN: echo __asan_report_store4 >> %t.interface
 // RUN: echo __asan_report_store8 >> %t.interface
 // RUN: echo __asan_report_store16 >> %t.interface
 // RUN: echo __asan_report_load_n >> %t.interface
 // RUN: echo __asan_report_store_n >> %t.interface
+// RUN: echo __asan_report_load1_noabort >> %t.interface
+// RUN: echo __asan_report_load2_noabort >> %t.interface
+// RUN: echo __asan_report_load4_noabort >> %t.interface
+// RUN: echo __asan_report_load8_noabort >> %t.interface
+// RUN: echo __asan_report_load16_noabort >> %t.interface
+// RUN: echo __asan_report_store1_noabort >> %t.interface
+// RUN: echo __asan_report_store2_noabort >> %t.interface
+// RUN: echo __asan_report_store4_noabort >> %t.interface
+// RUN: echo __asan_report_store8_noabort >> %t.interface
+// RUN: echo __asan_report_store16_noabort >> %t.interface
+// RUN: echo __asan_report_load_n_noabort >> %t.interface
+// RUN: echo __asan_report_store_n_noabort >> %t.interface
 // RUN: echo __asan_report_exp_load1 >> %t.interface
 // RUN: echo __asan_report_exp_load2 >> %t.interface
 // RUN: echo __asan_report_exp_load4 >> %t.interface
 // RUN: echo __asan_report_exp_load8 >> %t.interface
 // RUN: echo __asan_report_exp_load16 >> %t.interface
 // RUN: echo __asan_report_exp_store1 >> %t.interface
 // RUN: echo __asan_report_exp_store2 >> %t.interface
 // RUN: echo __asan_report_exp_store4 >> %t.interface

compiler-rt/trunk/test/asan/TestCases/halt_on_error-1.c

+// Test recovery mode.
+//
+// RUN: %clang_asan -fsanitize-recover=address %s -o %t
+//
+// RUN: env not %run %t 2>&1 | FileCheck %s
+// RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:halt_on_error=true not %run %t 2>&1 | FileCheck %s
+// RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:halt_on_error=false %run %t 2>&1 | FileCheck %s --check-prefix CHECK-RECOVER
+
+#include <string.h>
+
+volatile int ten = 10;
+
+int main() {
+  char x[10];
+  // CHECK: WRITE of size 11
+  // CHECK-RECOVER: WRITE of size 11
+  memset(x, 0, 11);
+  // CHECK-NOT: READ of size 1
+  // CHECK-RECOVER: READ of size 1
+  volatile int res = x[ten];
+  // CHECK-NOT: WRITE of size 1
+  // CHECK-RECOVER: WRITE of size 1
+  x[ten] = res + 3;
+  // CHECK-NOT: READ of size 1
+  // CHECK-RECOVER: READ of size 1
+  res = x[ten];
+  return  0;
+}
+