[GwpAsan] Introduce GWP-ASan.

[hctim]

Summary

This patch introduces GWP-ASan, a sampled allocator framework that assists in finding use-after-free and heap-buffer-overflows bugs in production environments.

GWP-ASan supplements a traditional allocator (e.g. Scudo), and chooses random allocations to 'sample'. These sampled allocations are placed into a special guarded pool, which is based upon the traditional 'Electric Fence Malloc Debugger'. We surround the allocation with inaccessible pages, such that buffer under/overflows trap on the page fault. We also mark the allocation's page as inaccessible on free, meaning that any use-after-free bugs also cause a page fault trap, which we capture. For more implementation details, please see docs/GWPASan.rst.

Please note that this patch is quite large. The patchset contains the basic functionality of GWP-ASan (stack trace dumping and other debug information will be added shortly), the unit tests, and allocator shims into Scudo. This allows any code compiled with -fsanitize=scudo to have GWP-ASan enabled-by-default (including the unit tests). I've tried to slice out as much as possible for follow up patches but there isn't much left here to carve off.

[hctim]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

63

Shouldn't be allowed - the default flag parser has the check:

if (o->Enabled && o->SampleRate < 1) {
    __sanitizer::Printf("GWP-ASan ERROR: SampleRate must be >= 1 when "
                        "GWP-ASan is enabled.\n");
    exit(EXIT_FAILURE);
  }

... but allocators can implement their own flag parser. I've added an explicit check in init() to consider SampleRate == 0 as being the same as Enabled == false.

77

Fixed by thread-hostile requirements described above.

91

That's an awesome catch! Thanks!

156

Yep, this is intended, but one of the deallocate() was using this incorrectly (where they wanted slot address instead of page address). I've updated the callee.

169

Furthermore, this also had an issue where (if it worked correctly in the first place), it lead to more deterministic left/right alignment, as the alignment choice was made based on the slot number.

I've fixed this and will add some more tests before submitting. Leaving this comment open to remind me to do this at a later date, but focus right now is getting this out for another round of comments.

183

FYI - I have adjusted this to be purely PRNG based as if NumGuardedSlots is odd, it will give us a higher probability of left-alignment. If NumGuardedSlots == 1, we want to still do random left/right alignment.

207

At this point, we don't have access to a printf() library.

Also, I'm not sure whether this is actually reachable. SingletonPtr is always initialised in this TU before the signal handlers are installed. Have changed this to an assert.

263

I didn't know this, thanks!

299

Could this be useful for UaF where there is a nonzero offset? Have elided for zero offsets only, as I figure it's better to keep the information when it may be of use.

339

Have changed to simply echo "Segmentation fault" and exit.

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

68

It wont on x64 as AFAIK the static predictors can't be primed (and doesn't appear to change the optimiser), but have as it may affect other other platforms.

75

So upon further reading, looks like the only way to ensure atomic-correctness here is to establish a dependency between GPP and GPPEnd, which require sequentially consistent loads and stores (which would hurt performance a trememdous amount).

We currently do what Matt suggests, so I've updated the documentation to reflect that the entire class is thread-hostile until init() is called.

114

They may access GPA::Printf on failure. Would you prefer them to be static w/ passing Printf as an argument?

163

For future expansion, we will want to populate a slot index and a separate metadata index. I've reworded up the comment but left the implementation as-is for now. WDYT?

172

I had a good think about the best way to do this adjustment. Does the rework LGTY?

237

AFAICT it is required:

<snip>/llvm/compiler-rt/lib/scudo/../gwp_asan/guarded_pool_allocator.h:237:3: error: 'thread_local' is only allowed on variable declarations
  TLS_INITIAL_EXEC uint64_t NextSampleCounter;

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

72

Can you PTAL and see if it LGTY?

compiler-rt/lib/gwp_asan/options.inc

25

Added a test for this.

compiler-rt/lib/scudo/scudo_allocator.cpp

305

See attached image in description.

compiler-rt/test/gwp_asan/alignment_power_of_two.cpp

19

Updated w/ comment in the test. We have no direct influence on left/right alignment any more, so we just run it multiple times.

llvm/docs/GWPASan.rst

115

I think "transient" actually suits this better.

[hctim]

• Updated from Matt's comments.

[hctim]

• Added a thread contention test. Creates allocations using multiple threads and checks to see that a guarded slot is never allocated twice. Also tests against the random slot selection.

compiler-rt/lib/gwp_asan/optional/runtime_env_flag_parser.cpp

27

fn is removed.

49

Removed them without an unnamed namespace. This okay?

59

Nope, removed.

100

I could have sworn that this failed to build without it, but it appears to be a non-issue now. Removed.

compiler-rt/lib/gwp_asan/options.h

28

Yep, fixed the init-order-fiasco with moving to on-demand instantiation inside of getOptions().

[hctim]

• Removed the changed sanitizer_common from this changelist. No longer required, and the dependent change is abandoned.

[vlad.tsyrklevich]

In D60593#1483183, @hctim wrote:

• Added a thread contention test. Creates allocations using multiple threads and checks to see that a guarded slot is never allocated twice. Also tests against the random slot selection.

There's no way to access the allocator directly in the tests? I ask because you could test stronger conditions if you had things like PointerIsMine in the tests.

compiler-rt/lib/gwp_asan/options.inc

29

Feels like this should still be uint64/ULL (2^63 comment is otherwise not right on 32-bit platforms.)

compiler-rt/test/gwp_asan/alignment_power_of_two.cpp

39

Lets make this 2**-40 so it can never flake.

51

Seems redundant to the check above (if you assume that page size is always a power of two.)

compiler-rt/test/gwp_asan/num_usable_slots.cpp

12 ↗

(On Diff #197200)

Add a negative test or two (e.g. do we actually fail when invoked with 8 slots but we iterate 9 times? Do we fail when we have 8 slots but iterate 7 times?)

63 ↗

(On Diff #197200)

There's UB in the 129 case (SampleAllocs OOB access), replace with i<NumSlots and maybe just make SampledAllocs a stack-allocated VLA?

compiler-rt/test/gwp_asan/page_size.h

11 ↗

(On Diff #197200)

Delete, currently unused.

llvm/docs/GwpAsan.rst

177 ↗

(On Diff #197200)

s/NumUsableGuardedSlots/MaxSimultaneousAllocations/ ? It would also make the description a little clearer to explain it that way.

[vlad.tsyrklevich]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

115

getPageAddr()

compiler-rt/test/gwp_asan/thread_contention.cpp

1 ↗

(On Diff #197200)

This tests races in the allocation code but not in the deallocation (and reallocation) code.

9 ↗

(On Diff #197200)

I don't think the runtime configuration adds much here anyway. I think you could get rid of both options and just have it allocate a fixed amount (lets say 1024 times) twice (first w/o random eviction and second run with.) It simplifies the test.

If we had access to the underlying allocator we could be cuter and also verify that every available slot was used, but I wouldn't bother otherwise.

30 ↗

(On Diff #197200)

Nit: Why not left to right?

[hctim]

• Updated sample rate to guarantee 64-bit values.

compiler-rt/lib/gwp_asan/options.inc

29

Was trying to work out a solution around the fact that the sanitizer flag parser doesn't support uint64_t. I've created a patch to support long long in sanitizer_common which we can then truncate if > 64bits.

Signed-ness of this doesn't really matter as we multiply by two in init() anyway. We can't directly convert to an unsigned 64-bit value as internal_strtoll doesn't support unsigned 64-bit values (they internally use s64 and typedef it to long long).

[vitalybuka]

you should try to extract obvious parts into smaller patches

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

52

sizeof(*Metadata)

57

sizeof(*FreeSlots)
this is not Google style, but advice is still relevant https://google.github.io/styleguide/cppguide.html#sizeof

87

...== false) -> if (!...

100

Meta->Init(Size, Ptr, .....)

125

can you move all metadata manipulation logic into AllocationMetadata:: methods?

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

40

can you hide this struct into cpp file and just keep forward declaration here?

51

can you please add kInvalidThreadID ?

79

why do you need this this line at all "~GuardedPoolAllocator() = default;"?

179

can you avoid output arg by:
std::size_t reserveSlot();

237

"static private:" are not very useful
you can hide them into unnamed namespace of cpp file

compiler-rt/lib/gwp_asan/mutex.h

43

explicit?

64

do you need yieldProcessor and lockSlow inline?
maybe move into cpp file to move this ifdefs from header?

compiler-rt/lib/gwp_asan/random.h

25

it's called on slow path when NextSampleCounter == 0
I don't thins ALWAYS_INLINE will help here.
just move it into cpp file?
Also http://quick-bench.com/Jbi9nGqIlAfRkFGHjpJzBVR58Bw

Can you explain why you need own random?

50

they don't need to be in the header

[hctim]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

115

Not equal if sizeof(Slot) != sizeof(Page). Happy to introduce a helper fn for this if you want, but it's used in a single place.

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

40

See https://reviews.llvm.org/D60593?id=195942#inline-541269. Happy to change.

79

It's simply an explicit note to the reader that we have a trivial destructor. If they want to add a d'tor, they'll come to this line and read the note above.

179

See https://reviews.llvm.org/D60593#inline-542081

237

NextSampleCounter are referenced in shouldSample(), which is implemented in the header for inlining reasons.
SingletonPtr has been moved.

compiler-rt/lib/gwp_asan/mutex.h

64

http://quick-bench.com/5oJNxyiBe-7KwZ0nMCRDGBaNa5s

Looks like with clang gets a slight perf improvement with inlining, which is extended (to 10% perf improvement) w/ libc++.

Also I'm hesitent to create small cpp files, as per https://reviews.llvm.org/D60593?id=195942#inline-542189.

Okay to leave as is?

compiler-rt/lib/gwp_asan/random.h

25

Can't use PRNG from libc++, as we have to be c-compatible.

Can't use PRNG from sanitizer_common, as we don't depend on sanitizer_common in the core code.

libc rand() is too slow.

read from /dev/ is too slow.

compiler-rt/test/gwp_asan/thread_contention.cpp

1 ↗

(On Diff #197200)

I've also added a test that's pretty much ThreadedHighContention.

9 ↗

(On Diff #197200)

The runtime configuration is mostly to avoid the overhead of recompilation. The compilation of this file for a debug build of clang takes 3.5-4s on my machine, and I desperately wanted to avoid that overhead multiple times.

The additional runs have been deleted.

[hctim]

• From Vlad and Vitaly's comments.
• Merge branch 'master' into gwp-address with parent change completed.

[morehouse]

compiler-rt/CMakeLists.txt

281

This option has to do specifically with Scudo. Therefore it should go in the Scudo cmake.

And Kostya K. needs to be OK with hooks being installed by default.

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

183

I expect it doesn't matter in practice. GWP-ASan is already probabilistic, so who cares if we're x% more likely to detect underflow than overflow. You could even invert things so we get more right-aligned allocations since overflow is more common than underflow.

207

Ok, but the assert isn't enough. In release builds we could still null-deref.

264

Nit: else is unnecessary

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

26

Technically doesn't this make it "thread-compatible" until init(), then thread-safe?

40

GPA is the only user of AllocationMetadata, and the struct is relatively small. Do we really want an entire file for ~20 lines of code?

68

Regardless of branch prediction, the compiler may choose to make "other optimizations" to improve the hot path.

79

Not necessary, but I think it does make the intention clear.

163

I'd prefer simpler for now, with refactoring later. It could be a while until "future expansion", so let's keep the code simple in the meantime.

172

Looks better.

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

69

These two checks are probably unnecessary.

72

I think we should avoid dying here so the previous signal handler can examine the SEGV too.

compiler-rt/lib/gwp_asan/options.inc

29

What happens if I set a negative value?

39

What happens if I set a negative sample rate?

[morehouse]

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

77

There are also SIG_DFL and SIG_IGN cases. See https://critique.corp.google.com/#review/197822848/depot/google3/tcmalloc/guarded_page_allocator.cc for an example.

[vlad.tsyrklevich]

compiler-rt/test/gwp_asan/num_usable_slots.cpp

12 ↗

(On Diff #197200)

We test +1 case but not the -1 case.

60 ↗

(On Diff #197637)

I'm not sure what a better test is but this isn't always true (e.g. the allocator could actually take back that page if all the allocations on it are gone.) These tests could test stronger assertions and be simpler if there was a __gwp_asan_pointer_is_mine() or something like that.

compiler-rt/test/gwp_asan/thread_contention_with_release.cpp

32 ↗

(On Diff #197637)

*Ptr may not be 0 if we fall through to regular malloc() (either here or on the realloc call below), perhaps we should risk running this test on normal malloc()/realloc() sometimes and have NumAllocsTotal fully deplete the free list instead?

72 ↗

(On Diff #197637)

I'm not sure I understand this logic: 1) what std::set? (the std::vector<std::thread>?) 2) we should be able to consume more than the total number of guarded allocations if we wanted right since it would just fall back to normal malloc?

llvm/docs/GwpAsan.rst

152 ↗

(On Diff #197637)

Replace NumUsableGuardedPages

159 ↗

(On Diff #197637)

Ditto

[hctim]

• Updated with Vlad's and Matt's comments.

compiler-rt/CMakeLists.txt

281

I've changed this flag to not exist any more, as it's synonymous with COMPILER_RT_GWP_ASAN_ENABLED anyway.

@cryptoad and I have confirmed offline that always-built on supported platforms is okay :)

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

183

I think the major reason for me is that it may be a use case to have a guarded pool with only a single slot for short-lived processes on memory-constrained devices. We're planning on process sampling as well for Android, and it may be useful to service short-lived processes with a single allocation instead of disabling it completely.

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

26

Yep, updated documentation to reflect this.

163

Gotcha. ssize_t is POSIX-only (see Windows docs) so I've made it return kInvalidSlotID instead.

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

77

Done with some slight modifications. If the previous handler was SIG_IGN, we die as we'll constantly restart on the memory error and repeatedly spam GWP-ASan output. This also aligns with the exit behaviour of DOUBLE_FREE/INVALID_FREE and that of SIGSEGV-delivered errors.

We also don't explicitly handle SIG_DFL, as the fallthrough should call the default signal handler just fine.

compiler-rt/lib/gwp_asan/options.inc

29

The error checking is done in the flag parser, but I've also added explicit checking in init() as well.

compiler-rt/test/gwp_asan/num_usable_slots.cpp

12 ↗

(On Diff #197200)

I'm confused why we would need to test the -1 case (assuming that -1 means NumIterations == MaxSimultaneousAllocations - 1), given that NumIterations == MSA provide this coverage. Is my assumption correct?

60 ↗

(On Diff #197637)

Changed it to use the same mechanism from above to detect if it's in the pool.

I think the current assumption is strong enough, but may be broken if atoi() or alloca() changes to call malloc (unlikely), or if a shared library calls malloc() during dynamic loading. WDYT?

Edit: On further thought, this also will spuriously pass if the implementing allocator uses the page immediately after the allocation pool to back the allocation. I've added interface definitions and started a cleanup so that we can exercise the allocator internals directly.

[vlad.tsyrklevich]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

427

nullptr

432

if (!gwp_asan::SingletonPtr) return; to match the other functions?

compiler-rt/test/gwp_asan/no_reuse_ahead_of_time.cpp

35 ↗

(On Diff #198048)

Why does anything need to care about initializing the GPA ahead of time for anything other than allocate? The underlying methods should return the correct values. And even for allocate() we can get around this edge case by doing allocate(PageSize+1).

compiler-rt/test/gwp_asan/num_usable_slots.cpp

12 ↗

(On Diff #197200)

This looks fine as is.

60 ↗

(On Diff #197637)

If we want to be safe, because I can definitely see runtimes hitting malloc() before main()in normal code paths, we could make SampleRate high and use the GPA allocate directly for this test.

compiler-rt/test/gwp_asan/pattern_malloc_free.cpp

2

I would collapse the pattern_* tests into two tests:

1. Test that malloc, calloc, realloc, new, new[] all satisfy pointerismine
2. And then this test.

Less code and should have equal coverage.

compiler-rt/test/gwp_asan/reuse_quarantine.cpp

9

Maybe delete the 1/2 cases (will fail on runtimes that call malloc before main() and just call it with some other higher value that should always work like 32.

[vlad.tsyrklevich]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

432

I suppose it could also be assert(gwp_asan::SingletonPtr) since it's never correct to reach this function with an uninitialized GPA.

[hctim]

compiler-rt/test/gwp_asan/no_reuse_ahead_of_time.cpp

35 ↗

(On Diff #198048)

These helper funcs call through the singleton ptr, which is nullptr until init(). Calling pointerIsMine() (for example) would access (nullptr)->GuardedPagePool.

I like the maximumAllocSize() + 1. At the moment, GPA::maximumAllocSize()doesn't rely on any data members, but is still UB to call with a nullptr instance. I think I'll do this but with an arbritrarily large allocation and leave a note for me to update it when we support multipage slots.

compiler-rt/test/gwp_asan/reuse_quarantine.cpp

9

I've instead rewrote it so that it exercises the allocator directly with no sampling, so that we avoid this issue. LGTY?

[hctim]

• Updated from Vlad's comments.

[vlad.tsyrklevich]

compiler-rt/test/gwp_asan/no_reuse_ahead_of_time.cpp

35 ↗

(On Diff #198048)

But that's not true, all of the __gwp_asan_* functions fail gracefully if initialization hasn't occurred (except for deallocate which is never valid to call when uninitialized), e.g. pointer_is_mine would just return false. We do need initialization for the allocate() case so it works as expected, and I'm fine with wrapping all of the gwp_asan_* functions for consistency with allocate(), but I think we should only call maybeInit in the allocate case to make it explicit that that is the only one that needs it (and if that changes we can get rid of it and use the underlying methods directly.)

compiler-rt/test/gwp_asan/reuse_quarantine.cpp

9

Looks good.

[morehouse]

compiler-rt/include/sanitizer/gwp_asan_interface.h

1 ↗

(On Diff #198313)

I think I missed something. What is the purpose of these functions?

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

75

I'm not opposed to ignoring SIG_IGN, but this should be documented somewhere.

77

I am skeptical that calling SIG_DFL() will work.

SIG_DFL is not a function address.

https://en.cppreference.com/w/cpp/utility/program/SIG_strategies

compiler-rt/lib/gwp_asan/gwp_asan_interface_internal.h

1 ↗

(On Diff #198313)

Why do we want this?

compiler-rt/lib/gwp_asan/random.cpp

18

Looks like initialization of these globals could happen after the first call to getRandom.

compiler-rt/lib/scudo/scudo_allocator.cpp

305

Could you just post the assembly? Preferably a before/after of this fastpath change.

The CFG is confusing for me to read.

[hctim]

compiler-rt/include/sanitizer/gwp_asan_interface.h

1 ↗

(On Diff #198313)

See comment in compiler-rt/lib/gwp_asan/gwp_asan_interface_internal.h.

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

75

Documented in GwpAsan.rst and options.inc.

77

Surprisingly, it worked on my machine. Have changed to ensure this works everywhere.

compiler-rt/lib/gwp_asan/gwp_asan_interface_internal.h

1 ↗

(On Diff #198313)

To allow us to exercise the allocator directly for testing. The comments in compiler-rt/test/gwp_asan/num_usable_slots.cpp has some more context, but the summary is:

1. Allows us to be precise with exactly which allocations are sampled (important, for example, in tests/num_usable_slots.cpp where if a library calls malloc() during its init, we will break. Also, in the thread_contention* tests, we want to be certain that we are exercising the sampled allocator, rather than potentially falling back to the system allocator.
2. Allows us to test directly against the interface whether something is sampled through __gwp_asan_pointer_is_mine(), avoiding relying on implementation specific assumptions. In tests/num_usable_slots.cpp, we were previously relying on left-to-right sequential usage of the sampled slots to determine whether an allocation was sampled. We then had no method to determine whether an allocation was not sampled, as the system allocator may place the unsampled allocation in the page directly after the guarded pool, leading us to spurious failure.

compiler-rt/lib/scudo/scudo_allocator.cpp

305

before, after the changes

compiler-rt/test/gwp_asan/no_reuse_ahead_of_time.cpp

35 ↗

(On Diff #198048)

My mistake. They don't cause error, but they don't return correct values until init has taken place.

We also need init support for maximumAllocationSize() from interface_test.cpp and interface_test_failures.cpp with their current implementation. We could add explicit initialisation by caling a dummy __internal::allocate() / deallocate() in these specific tests. (And, I guess thread_contention_with_release.cpp as well if we can't make the strong guarantee that std::vector<std::thread>::emplace_back() always calls the system allocator.)

It seems to make the interface a lot less prone to test implementation errors. Happy to change if if you feel strongly about this, but apart from performance of the tests, I don't see what we gain (especially when scudo::allocate always does a once-init check as well through initThreadMaybe()).

[hctim]

• Matt's comments.

[hctim]

compiler-rt/lib/scudo/scudo_allocator.cpp

305

@vlad.tsyrklevich

[morehouse]

compiler-rt/lib/gwp_asan/gwp_asan_interface_internal.h

1 ↗

(On Diff #198313)

Can we just write unit tests, like we have in Google3 and Chrome GWP-ASans, Scudo, and every sanitizer? Then we don't need these interfaces.

[morehouse]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

98

The assembly you posted for this function looks way too slow. Is it actually optimized with -O2?

• The fast path (counter > 1) has *two* branches that check if NextSampleCounter's TLS init function has run and one actual call to the TLS init function. Ouch! We shouldn't need these.
  • Maybe we need to make the counter function-local or use __thread instead of thread_local.
• The counter decrement of NextSampleCounter requires 4 instructions? Load TLS offset from global, read TLS, increment, write TLS. Yikes. For reference, TCMalloc does this operation in 1 instruction.
• The + 1 operation on the NextSampleCounter == 0 branch is done with a mov-add, rather than an lea.

Again, please triple-check that the binary you compiled is actually optimized. Make sure it is compiled with clang++ -O2 ....

[vlad.tsyrklevich]

One thing I also just realized is that by explicitly setting TLS model to initial-exec we prevent dlopen() dynamic loading, I don't have a problem with that but just wanted to make sure @cryptoad was aware.

compiler-rt/lib/gwp_asan/gwp_asan_interface_internal.h

1 ↗

(On Diff #198313)

That is the way to go, I suggested this approach at first forgetting that there are non-lit tests.

[gribozavr]

What does GWP stand for?

[hctim]

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

98

As per offline discussion, declaring NextSampleCounter with __thread instead of thread_local fixed this. Also added a note in definitions.h.

[hctim]

• Updated tests to use unittests rather than expose a public interface and exercise that. Minor perf improvements to shouldSample().

[hctim]

In D60593#1495428, @gribozavr wrote:

What does GWP stand for?

Google Wide Profiling.

We use the name GWP-ASan, even though it's "technically neither GWP nor ASan", but because it describes well what the outcome is (sampled allocations with memory detection capabilities). It also mirrors the name used for the identical mechanism implemented in Chromium.

[morehouse]

This diff is helpful to get an overall idea of how things fit together, but it is very difficult to review thoroughly. Let's start splicing off pieces for individual review.

I suggest:

• Individual reviews for each prereq (mutex, random, etc.)
• Review for base GPA + unit tests
• Review for "optional" options parsing, etc.
• Review for documentation
• Review for Scudo integration + e2e tests

Submitting this in pieces (especially the Scudo integration) will also make things simpler if we need to revert something.

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

98

What is the assembly now?

[jfb]

In D60593#1498622, @hctim wrote:

In D60593#1495428, @gribozavr wrote:

What does GWP stand for?

Google Wide Profiling.

We use the name GWP-ASan, even though it's "technically neither GWP nor ASan", but because it describes well what the outcome is (sampled allocations with memory detection capabilities). It also mirrors the name used for the identical mechanism implemented in Chromium.

Hello I'm here to bikeshed, and this is a terrible name for an LLVM project. Or super appropriate because LLVM itself is a terrible name? In any case, backronym it to something else, or rename it. Thanks!

[eugenis]

Ha, I think it matches LLVM perfectly :)
G, of course, stands for "Galaxy".

[hctim]

In D60593#1502266, @jfb wrote:

Hello I'm here to bikeshed, and this is a terrible name for an LLVM project. Or super appropriate because LLVM itself is a terrible name? In any case, backronym it to something else, or rename it. Thanks!

Also see GwpAsan.rst:

It informally is a recursive acronym, "GWP-ASan Will Provide Allocation SANity".

[hctim]

Abandoned as this was merged in small chunks.

compiler-rt/CMakeLists.txt

 set(DEFAULT_COMPILER_RT_USE_BUILTINS_LIBRARY OFF)
 if (FUCHSIA)
   set(DEFAULT_COMPILER_RT_USE_BUILTINS_LIBRARY ON)
 endif()
 
 option(COMPILER_RT_USE_BUILTINS_LIBRARY
   "Use compiler-rt builtins instead of libgcc" ${DEFAULT_COMPILER_RT_USE_BUILTINS_LIBRARY})
 
+option(GWP_ASAN_SCUDO_HOOKS
+  "When building scudo, should we install GWP-ASan hooks (default=False)?" OFF)
+pythonize_bool(GWP_ASAN_SCUDO_HOOKS)
+
 include(config-ix)
 
 #================================
 # Setup Compiler Flags
 #================================
 
 if(MSVC)
   # Override any existing /W flags with /W4. This is what LLVM does.  Failing to

compiler-rt/cmake/config-ix.cmake

 
 set(ALL_SANITIZER_COMMON_SUPPORTED_ARCH ${X86} ${X86_64} ${PPC64}
     ${ARM32} ${ARM64} ${MIPS32} ${MIPS64} ${S390X})
 set(ALL_ASAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64}
     ${MIPS32} ${MIPS64} ${PPC64} ${S390X})
 set(ALL_DFSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64})
 set(ALL_FUZZER_SUPPORTED_ARCH ${X86_64} ${ARM64})
 
+set(ALL_GWP_ASAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM64} ${ARM32})
 if(APPLE)
   set(ALL_LSAN_SUPPORTED_ARCH ${X86} ${X86_64} ${MIPS64} ${ARM64})
 else()
   set(ALL_LSAN_SUPPORTED_ARCH ${X86} ${X86_64} ${MIPS64} ${ARM64} ${ARM32} ${PPC64})
 endif()
 set(ALL_MSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64} ${PPC64})
 set(ALL_HWASAN_SUPPORTED_ARCH ${X86_64} ${ARM64})
 set(ALL_PROFILE_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${PPC64}
if(APPLE)
   set(LSAN_COMMON_SUPPORTED_ARCH ${SANITIZER_COMMON_SUPPORTED_ARCH})
   set(UBSAN_COMMON_SUPPORTED_ARCH ${SANITIZER_COMMON_SUPPORTED_ARCH})
   list_intersect(ASAN_SUPPORTED_ARCH
     ALL_ASAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(DFSAN_SUPPORTED_ARCH
     ALL_DFSAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
+  list_intersect(GWP_ASAN_SUPPORTED_ARCH
+    ALL_GWP_ASAN_SUPPORTED_ARCH
+    SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(LSAN_SUPPORTED_ARCH
     ALL_LSAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(MSAN_SUPPORTED_ARCH
     ALL_MSAN_SUPPORTED_ARCH
     SANITIZER_COMMON_SUPPORTED_ARCH)
   list_intersect(HWASAN_SUPPORTED_ARCH
     ALL_HWASAN_SUPPORTED_ARCH
else()
   filter_available_targets(SAFESTACK_SUPPORTED_ARCH
     ${ALL_SAFESTACK_SUPPORTED_ARCH})
   filter_available_targets(CFI_SUPPORTED_ARCH ${ALL_CFI_SUPPORTED_ARCH})
   filter_available_targets(SCUDO_SUPPORTED_ARCH ${ALL_SCUDO_SUPPORTED_ARCH})
   filter_available_targets(SCUDO_STANDALONE_SUPPORTED_ARCH ${ALL_SCUDO_STANDALONE_SUPPORTED_ARCH})
   filter_available_targets(XRAY_SUPPORTED_ARCH ${ALL_XRAY_SUPPORTED_ARCH})
   filter_available_targets(SHADOWCALLSTACK_SUPPORTED_ARCH
     ${ALL_SHADOWCALLSTACK_SUPPORTED_ARCH})
+  filter_available_targets(GWP_ASAN_SUPPORTED_ARCH ${ALL_GWP_ASAN_SUPPORTED_ARCH})
 endif()
 
 if (MSVC)
   # See if the DIA SDK is available and usable.
   set(MSVC_DIA_SDK_DIR "$ENV{VSINSTALLDIR}DIA SDK")
   if (IS_DIRECTORY ${MSVC_DIA_SDK_DIR})
     set(CAN_SYMBOLIZE 1)
   else()

 message(STATUS "Compiler-RT supported architectures: ${COMPILER_RT_SUPPORTED_ARCH}")
 
 if(ANDROID)
   set(OS_NAME "Android")
 else()
   set(OS_NAME "${CMAKE_SYSTEM_NAME}")
 endif()
 
-set(ALL_SANITIZERS asan;dfsan;msan;hwasan;tsan;safestack;cfi;scudo;ubsan_minimal)
+set(ALL_SANITIZERS asan;dfsan;msan;hwasan;tsan;safestack;cfi;scudo;ubsan_minimal;gwp_asan)
 set(COMPILER_RT_SANITIZERS_TO_BUILD all CACHE STRING
     "sanitizers to build if supported on the target (all;${ALL_SANITIZERS})")
 list_replace(COMPILER_RT_SANITIZERS_TO_BUILD all "${ALL_SANITIZERS}")
 
 if (SANITIZER_COMMON_SUPPORTED_ARCH AND NOT LLVM_USE_SANITIZER AND
     (OS_NAME MATCHES "Android|Darwin|Linux|FreeBSD|NetBSD|OpenBSD|Fuchsia|SunOS" OR
     (OS_NAME MATCHES "Windows" AND NOT CYGWIN AND
         (NOT MINGW OR CMAKE_CXX_COMPILER_ID MATCHES "Clang"))))

 endif()
 
 if (COMPILER_RT_HAS_SANITIZER_COMMON AND SHADOWCALLSTACK_SUPPORTED_ARCH AND
     OS_NAME MATCHES "Linux|Android")
   set(COMPILER_RT_HAS_SHADOWCALLSTACK TRUE)
 else()
   set(COMPILER_RT_HAS_SHADOWCALLSTACK FALSE)
 endif()
+
+# Note: Fuchsia and Windows are not currently supported by GWP-ASan. Support
+# is planned for these platforms. Darwin is also not supported due to potential
+# TLS allocations on first-use.
+if (COMPILER_RT_HAS_SANITIZER_COMMON AND GWP_ASAN_SUPPORTED_ARCH AND
+    OS_NAME MATCHES "Android|Linux")
+  set(COMPILER_RT_HAS_GWP_ASAN TRUE)
+else()
+  set(COMPILER_RT_HAS_GWP_ASAN FALSE)
+endif()

compiler-rt/lib/gwp_asan/CMakeLists.txt

+add_compiler_rt_component(gwp_asan)
+
+include_directories(..)
+
+set(GWP_ASAN_SOURCES
+  guarded_pool_allocator.cpp
+  guarded_pool_allocator_posix.cpp
+  mutex_posix.cpp
+  random.cpp
+)
+
+set(GWP_ASAN_HEADERS
+  allocation_metadata.h
+  guarded_pool_allocator.h
+  mutex.h
+  options.h
+  options.inc
+  random.h
+)
+
+# Runtime environment support is optional. GwpAsan is totally independent of
+# sanitizer_common, the runtime env parser is not. This is an optional library
+# that can be used by an allocator to automatically parse GwpAsan flags from the
+# environment variable GWP_ASAN_FLAGS, but the allocator can choose to implement
+# its own flag parsing and populate the Options struct themselves.
+set(GWP_ASAN_FLAG_PARSER_SOURCES
+  optional/runtime_env_flag_parser.cpp
+)
+set(GWP_ASAN_FLAG_PARSER_HEADERS
+  optional/runtime_env_flag_parser.h
+)
+
+# Disable RTTI and exception support, as we want these libraries to be
+# C-compatible. Regular C source files can be linked against the generated
+# GwpAsan libraries using the Clang C compiler.
+set(GWP_ASAN_CFLAGS -fno-rtti -fno-exceptions)
+set(GWP_ASAN_FLAG_PARSER_CFLAGS ${GWP_ASAN_CFLAGS} ${SANITIZER_COMMON_CFLAGS})
+
+if (COMPILER_RT_HAS_GWP_ASAN)
+  foreach(arch ${GWP_ASAN_SUPPORTED_ARCH})
+    add_compiler_rt_runtime(
+      clang_rt.gwp_asan
+      STATIC
+      ARCHS ${arch}
+      SOURCES ${GWP_ASAN_SOURCES}
+      ADDITIONAL_HEADERS ${GWP_ASAN_HEADERS}
+      CFLAGS ${GWP_ASAN_CFLAGS}
+      PARENT_TARGET gwp_asan
+    )
+  endforeach()
+
+  add_compiler_rt_object_libraries(RTGwpAsan
+      ARCHS ${GWP_ASAN_SUPPORTED_ARCH}
+      SOURCES ${GWP_ASAN_SOURCES}
+      ADDITIONAL_HEADERS ${GWP_ASAN_HEADERS}
+      CFLAGS ${GWP_ASAN_CFLAGS})
+
+  # Note: If you choose to add this as an object library, ensure you also
+  # include the sanitizer_common flag parsing object lib
+  # 'RTSanitizerCommonNoTermination'.
+  add_compiler_rt_object_libraries(RTGwpAsanFlagParser
+      ARCHS ${GWP_ASAN_SUPPORTED_ARCH}
+      SOURCES ${GWP_ASAN_FLAG_PARSER_SOURCES}
+      ADDITIONAL_HEADERS ${GWP_ASAN_FLAG_PARSER_HEADERS}
+      CFLAGS ${GWP_ASAN_FLAG_PARSER_CFLAGS})
+endif()

compiler-rt/lib/gwp_asan/allocation_metadata.h

+//===-- allocation_metadata.h ---------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_ALLOCATION_METADATA_H_
+#define GWP_ASAN_ALLOCATION_METADATA_H_
+
+#include <cstdint>
+#include <cstddef>
+
+namespace gwp_asan {
+struct AllocationMetadata {
+  // Maximum number of stack trace frames to collect for allocations + frees.
+  // TODO(hctim): Implement stack frame compression, a-la Chromium.
+  // Currently the maximum stack frames is one, as we don't collect traces.
+  static constexpr std::size_t MaximumStackFrames = 1;
+
+  struct CallSiteInfo {
+    // The backtrace to the allocation/deallocation. If the first value is zero,
+    // we did not collect a trace.
+    uintptr_t Trace[MaximumStackFrames] = { 0lu };
+    // The thread ID for this trace, or UINT64_MAX if not available.
+    uint64_t ThreadID = UINT64_MAX;
+  };
+
+  // The address of this allocation.
+  uintptr_t Addr = 0;
+  // Represents the size requested by the caller.
+  std::size_t Size = 0;
+
+  CallSiteInfo AllocationTrace;
+  CallSiteInfo DeallocationTrace;
+
+  // Whether this allocation has been deallocated yet.
+  bool IsDeallocated = false;
+};
+}; // namespace gwp_asan
+
+#endif // GWP_ASAN_ALLOCATION_METADATA_H_

compiler-rt/lib/gwp_asan/definitions.h

+//===-- gwp_asan_definitions.h ----------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_DEFINITIONS_H_
+#define GWP_ASAN_DEFINITIONS_H_
+
+#define TLS_INITIAL_EXEC thread_local __attribute__((tls_model("initial-exec")))
+
+#ifdef LIKELY
+# undef LIKELY
+#endif // defined(LIKELY)
+#define LIKELY(X) __builtin_expect(!!(X), 1)
+
+#ifdef UNLIKELY
+# undef UNLIKELY
+#endif // defined(UNLIKELY)
+#define UNLIKELY(X) __builtin_expect(!!(X), 0)
+
+#ifdef NORETURN
+# undef NORETURN
+#endif // defined(NORETURN)
+#define NORETURN __attribute__((noreturn))
+
+#ifdef ALWAYS_INLINE
+# undef ALWAYS_INLINE
+#endif // defined(ALWAYS_INLINE)
+#define ALWAYS_INLINE inline __attribute__((always_inline))
+
+#ifdef ALIGNED
+# undef ALIGNED
+#endif // defined(ALIGNED)
+#define ALIGNED(X) __attribute__((aligned(X)))
+
+#endif // GWP_ASAN_DEFINITIONS_H_

compiler-rt/lib/gwp_asan/guarded_pool_allocator.h

+//===-- guarded_pool_allocator.h --------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_GUARDED_POOL_ALLOCATOR_H_
+#define GWP_ASAN_GUARDED_POOL_ALLOCATOR_H_
+
+#include "gwp_asan/allocation_metadata.h"
+#include "gwp_asan/definitions.h"
+#include "gwp_asan/mutex.h"
+#include "gwp_asan/options.h"
+#include "gwp_asan/random.h"
+
+#include <type_traits>
+
+namespace gwp_asan {
+// This class is the primary implementation of the allocator portion of GWP-
+// ASan. It is the sole owner of the pool of sequentially allocated guarded
+// slots. It should always be treated as a singleton.
+class GuardedPoolAllocator {
+public:
+  enum class GwpAsanError {
+    UNKNOWN,
+    USE_AFTER_FREE,
+    DOUBLE_FREE,
+    INVALID_FREE,
+    BUFFER_OVERFLOW,
+    BUFFER_UNDERFLOW
+  };
+
+  // During program startup, we must ensure that memory allocations do not land
+  // in this allocation pool until we're completely ready. The constructor
+  // ensures that shouldSample() and pointerIsMine() will always return false
+  // before this class has been fully initialised. Every call to non-static
+  // methods of this class should be guarded by a call to either of these
+  // functions before init() has been called.
+  constexpr GuardedPoolAllocator(){};
+  GuardedPoolAllocator(const GuardedPoolAllocator &) = delete;
+
+  // Note: This class is expected to be a singleton for the lifetime of the
+  // program. If this object is initialised, it will leak the guarded page pool
+  // and metadata allocations during destruction. We can't clean up these areas
+  // as this may cause a racy use-after-free on shutdown.
+  ~GuardedPoolAllocator() = default;
+
+  // Initialise the rest of the members of this class. Create the allocation
+  // pool using the provided options. See options.inc for runtime configuration
+  // options.
+  void init(const options::Options &Opts);
+
+  // Return whether the allocation should be randomly chosen for sampling.
+  ALWAYS_INLINE bool shouldSample() {
+    // NextSampleCounter == 0 means we "should regenerate the counter".
+    //                   == 1 means we "should sample this allocation".
+    if (UNLIKELY(NextSampleCounter == 0)) {
+      NextSampleCounter =
+          (Random::getRandomUnsigned64() %
+           __atomic_load_n(&AdjustedSampleRate, __ATOMIC_RELAXED)) +
+          1;
+    }
+
+    return NextSampleCounter-- == 1 &&
+           __atomic_load_n(&IsInitialised, __ATOMIC_RELAXED);
+  }
+
+  // Returns whether the provided pointer is a current sampled allocation that
+  // is owned by this pool.
+  ALWAYS_INLINE bool pointerIsMine(const void *Ptr) const {
+    uintptr_t P = reinterpret_cast<uintptr_t>(Ptr);
+    return GetGuardedPagePool() <= P && P < GetGuardedPagePoolEnd();
+  }
+
+  // Allocate memory in a guarded slot, and return a pointer to the new
+  // allocation. Returns nullptr if the pool is empty, the requested size is too
+  // large for this pool to handle, or the requested size is zero.
+  void *allocate(std::size_t Size);
+
+  // Deallocate memory in a guarded slot. The provided pointer must have been
+  // allocated using this pool. This will set the guarded slot as inaccessible.
+  void deallocate(void *Ptr);
+
+  // Returns the size of the allocation at Ptr.
+  std::size_t getSize(const void *Ptr) const;
+
+  // Returns the largest allocation that is supported by this pool. Any
+  // allocations larger than this should go to the regular system allocator.
+  std::size_t maximumAllocationSize() const;
+
+  // Dumps an error report (including allocation and deallocation stack traces)
+  // and exits the program. An optional error may be provided if the caller
+  // knows what the error is ahead of time. This is primarily a helper function
+  // to locate the static singleton pointer and call the internal version of
+  // this function. This helps solve the static initialisation order problem.
+  NORETURN static void
+  reportErrorAndDie(uintptr_t AccessPtr,
+                    GwpAsanError Error = GwpAsanError::UNKNOWN);
+
+private:
+  // These functions anonymously map memory or change the permissions of mapped
+  // memory into this process in a platform- specific way. Pointer and size
+  // arguments are expected to be page-aligned. These functions will never
+  // return on error, instead electing to kill the calling process on failure.
+  // Note that memory is initially mapped inaccessible. In order for RW
+  // mappings, call mapMemory() followed by markReadWrite() on the returned
+  // pointer.
+  void *mapMemory(std::size_t Size) const;
+  void unmapMemory(void *Ptr, std::size_t Size) const;
+  void markReadWrite(void *Ptr, std::size_t Size) const;
+  void markInaccessible(void *Ptr, std::size_t Size) const;
+
+  // Get the current thread ID, or UINT64_MAX if failure. Note: This
+  // implementation is platform-specific.
+  static uint64_t getThreadID();
+
+  // Get the page size from the platform-specific implementation. Only needs to
+  // be called once, and the result should be cached in PageSize in this class.
+  std::size_t getPlatformPageSize();
+
+  // Install the SIGSEGV crash handler for printing use-after-free and heap-
+  // buffer-{under|over}flow exceptions. This is platform specific as even
+  // though POSIX and Windows both support registering handlers through
+  // signal(), we have to use platform-specific signal handlers to obtain the
+  // address that caused the SIGSEGV exception.
+  static void installSignalHandlers();
+
+  ALWAYS_INLINE uintptr_t GetGuardedPagePool() const {
+    return __atomic_load_n(&GuardedPagePool, __ATOMIC_RELAXED);
+  }
+
+  ALWAYS_INLINE uintptr_t GetGuardedPagePoolEnd() const {
+    return __atomic_load_n(&GuardedPagePoolEnd, __ATOMIC_RELAXED);
+  }
+
+  // Returns the index of the slot that this pointer resides in. If the pointer
+  // is not owned by this pool, the result is undefined.
+  std::size_t addrToSlot(uintptr_t Ptr) const;
+
+  // Returns a pointer to the N-th guarded slot.
+  uintptr_t slotToAddr(std::size_t N) const;
+
+  // Returns a pointer to the metadata for the owned pointer. If the pointer is
+  // not owned by this pool, the result is undefined.
+  AllocationMetadata *addrToMetadata(uintptr_t Ptr) const;
+
+  // Returns the page that this pointer resides in.
+  uintptr_t getPageAddr(uintptr_t Ptr) const;
+
+  // Gets the nearest slot the the provided address.
+  std::size_t getNearestSlot(uintptr_t Ptr) const;
+
+  // Returns whether the provided pointer is a guard page or not. The pointer
+  // must be within memory owned by this pool, else the result is undefined.
+  bool isGuardPage(uintptr_t Ptr) const;
+
+  // Reserve a guarded slot Returns true if there was a slot to be reserved, and
+  // writes the available index to the provided parameter. Returns false if no
+  // slots are available.
+  bool reserveSlot(std::size_t *PageIndex);
+
+  // Unreserve the guarded slot.
+  void freeSlot(std::size_t PageIndex);
+
+  // Uses the allocation strategy from the runtime flags and the allocation
+  // size provided to update where the allocation should be made.
+  // AllocationPtr should be passed to this function as a pointer to the
+  // slot in which the allocation should live. This function updates *Size
+  // to be the real size of the allocation.
+  void applyAllocationStrategy(std::size_t *Size, uintptr_t *AllocationPtr);
+
+  // Try-lock all the mutexes in this class. This is used to attempt to stop any
+  // race wherein a use-after-free has been triggered (and we're reporting the
+  // error), and another thread attempts to make a new allocation in the same
+  // slot.
+  void tryLockAllMutexes();
+
+  // Diagnose an unknown error. Place the type of the error in *Error, and
+  // return a pointer to the metadata for the slot where the fault manifests.
+  // May return nullptr if the error was unable to be diagnosed, or if it
+  // resulted in GwpAsanError::INVALID_FREE or GwpAsanError::UNKNOWN.
+  AllocationMetadata *diagnoseUnknownError(uintptr_t AccessPtr,
+                                           GwpAsanError *Error);
+
+  NORETURN void reportErrorAndDieInternal(uintptr_t AccessPtr,
+                                          GwpAsanError Error);
+
+private:
+  // Whether this class has been initialised. This means the class is ready to
+  // accept calls to allocate() and deallocate(). Lazy initialisation may not
+  // have taken place even when this is true.
+  bool IsInitialised = false;
+
+  // Cached page size for this system in bytes.
+  std::size_t PageSize = 0;
+
+  // A mutex to protect the guarded slot and metadata pool for this class.
+  Mutex PoolMutex;
+  // The number of guarded slots that this pool holds.
+  std::size_t NumGuardedSlots = 0;
+  // Record the number of currently used slots. We store this amount so that we
+  // don't potentially randomly choose to recycle a slot that previously had an
+  // allocation before all the slots have been utilised.
+  std::size_t NumUsedSlots = 0;
+  // Pointer to the pool of guarded slots. Note that this points to the start of
+  // the pool (which is a guard page), not a pointer to the first guarded page.
+  uintptr_t GuardedPagePool = 0;
+  uintptr_t GuardedPagePoolEnd = 0;
+  // Pointer to the allocation metadata (allocation/deallocation stack traces),
+  // if any.
+  AllocationMetadata *Metadata = nullptr;
+
+  // Pointer to an array of free slot indexes.
+  std::size_t *FreeSlots = nullptr;
+  // The current length of the list of free slots.
+  std::size_t FreeSlotsLength = 0;
+
+  // See options.{h, inc} for more information.
+  bool PerfectlyRightAlign = false;
+
+  // Printf function supplied by the implementing allocator. We can't (in
+  // general) use printf() from the cstdlib as it may malloc(), causing infinite
+  // recursion.
+  options::Printf_t Printf = nullptr;
+
+  // The adjusted sample rate for allocation sampling. Default *must* be
+  // nonzero, as dynamic initialisation may call malloc (e.g. from libstdc++)
+  // before GPA::init() is called. This would cause an error in shouldSample(),
+  // where we would calculate modulo zero. This value is set UINT64_MAX - 1, as
+  // when GWP-ASan is disabled, we wish to never spend wasted cycles
+  // recalcualting the sample rate.
+  ALIGNED(8) uint64_t AdjustedSampleRate = UINT64_MAX - 1;
+  // Thread-local decrementing counter that indicates that a given allocation
+  // should be sampled when it reaches zero.
+  static TLS_INITIAL_EXEC uint64_t NextSampleCounter;
+
+  // Pointer to the singleton version of this class. Automatically assigned
+  // during initialisation, this allows the signal handler to find this class
+  // in order to deduce the root cause of failures.
+  static GuardedPoolAllocator *SingletonPtr;
+};
+
+static_assert(std::is_trivially_destructible<GuardedPoolAllocator>::value,
+              "GuardedPoolAllocator must be trivially destructible.");
+}; // namespace gwp_asan
+
+#endif // GWP_ASAN_GUARDED_POOL_ALLOCATOR_H_

compiler-rt/lib/gwp_asan/guarded_pool_allocator.cpp

+//===-- guarded_pool_allocator.cpp ------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gwp_asan/guarded_pool_allocator.h"
+
+#include "gwp_asan/allocation_metadata.h"
+#include "gwp_asan/options.h"
+
+#include <cassert>
+#include <cstdlib>
+#include <cstring>
+#include <ctime>
+#include <new>
+
+namespace gwp_asan {
+void GuardedPoolAllocator::init(const options::Options &Opts) {
+  SingletonPtr = this;
+
+  // Note: We return from the constructor here if GWP-ASan is not available.
+  // This will stop heap-allocation of class members, as well as mmap() of the
+  // guarded slots.
+  if (!Opts.Enabled)
+    return;
+
+  NumGuardedSlots = Opts.NumUsableGuardedSlots;
+
+  PageSize = getPlatformPageSize();
+
+  if (Opts.InstallSignalHandlers)
+    installSignalHandlers();
+
+  PerfectlyRightAlign = Opts.PerfectlyRightAlign;
+  Printf = Opts.Printf;
+
+  std::size_t PoolBytesRequired = PageSize * (1 + NumGuardedSlots) +
+                                  NumGuardedSlots * maximumAllocationSize();
+  void *GuardedPoolMemory = mapMemory(PoolBytesRequired);
+
+  // Round up memory mappings to the nearest page size.
+  std::size_t BytesRequired = NumGuardedSlots * sizeof(AllocationMetadata);
+  BytesRequired += PageSize - (BytesRequired % PageSize);
+  Metadata = reinterpret_cast<AllocationMetadata *>(mapMemory(BytesRequired));
+  markReadWrite(Metadata, BytesRequired);
+
+  // Allocate memory and set up the free pages queue.
+  BytesRequired = NumGuardedSlots * sizeof(std::size_t);
+  BytesRequired += PageSize - (BytesRequired % PageSize);
+  FreeSlots = reinterpret_cast<std::size_t *>(mapMemory(BytesRequired));
+  markReadWrite(FreeSlots, BytesRequired);
+  FreeSlotsLength = NumGuardedSlots;
+  for (std::size_t i = 0; i < NumGuardedSlots; ++i)
+    FreeSlots[i] = i;
+
+  // Multiply the sample rate by 2 to give a good, fast approximation for (1 /
+  // SampleRate) chance of sampling.
+  if (Opts.SampleRate != 1)
+    __atomic_store_n(&AdjustedSampleRate, Opts.SampleRate * 2,
+                     __ATOMIC_RELAXED);
+  else
+    __atomic_store_n(&AdjustedSampleRate, 1, __ATOMIC_RELAXED);
+
+  // Store these as the last values. This ensures that all calls to
+  // isPointerMine() return false until the allocator is completely ready.
+  __atomic_store_n(&GuardedPagePool,
+                   reinterpret_cast<uintptr_t>(GuardedPoolMemory),
+                   __ATOMIC_RELAXED);
+  __atomic_store_n(&GuardedPagePoolEnd,
+                   reinterpret_cast<uintptr_t>(GuardedPoolMemory) +
+                       PoolBytesRequired,
+                   __ATOMIC_RELAXED);
+  __atomic_store_n(&IsInitialised, true, __ATOMIC_RELAXED);
+}
+
+void *GuardedPoolAllocator::allocate(std::size_t Size) {
+  if (Size == 0 || Size > maximumAllocationSize())
+    return nullptr;
+
+  ScopedLock L(&PoolMutex);
+
+  size_t Index;
+  if (reserveSlot(&Index) == false)
+    return nullptr;
+
+  uintptr_t Ptr = slotToAddr(Index);
+  markReadWrite(reinterpret_cast<void *>(Ptr), maximumAllocationSize());
+
+  AllocationMetadata *Meta = addrToMetadata(Ptr);
+
+  Meta->Size = Size;
+  applyAllocationStrategy(&Size, &Ptr);
+
+  Meta->Addr = Ptr;
+  Meta->IsDeallocated = false;
+  Meta->AllocationTrace.ThreadID = getThreadID();
+  Meta->DeallocationTrace.ThreadID = UINT64_MAX;
+
+  // TODO(hctim): Implement stack trace collection.
+  Meta->AllocationTrace.Trace[0] = 0lu;
+  Meta->DeallocationTrace.Trace[0] = 0lu;
+
+  return reinterpret_cast<void *>(Ptr);
+}
+
+void GuardedPoolAllocator::deallocate(void *Ptr) {
+  assert(pointerIsMine(Ptr) && "Pointer is not mine!");
+  uintptr_t UPtr = reinterpret_cast<uintptr_t>(Ptr);
+  uintptr_t Page = getPageAddr(UPtr);
+  AllocationMetadata *Meta = addrToMetadata(UPtr);
+  if (Meta->Addr != UPtr)
+    reportErrorAndDie(UPtr, GwpAsanError::INVALID_FREE);
+
+  ScopedLock L(&PoolMutex);
+
+  if (Meta->IsDeallocated)
+    reportErrorAndDie(UPtr, GwpAsanError::DOUBLE_FREE);
+
+  Meta->IsDeallocated = true;
+  Meta->DeallocationTrace.ThreadID = getThreadID();
+
+  freeSlot(addrToSlot(UPtr));
+  markInaccessible(reinterpret_cast<void *>(Page), maximumAllocationSize());
+}
+
+std::size_t GuardedPoolAllocator::getSize(const void *Ptr) const {
+  assert(pointerIsMine(Ptr));
+  AllocationMetadata *Meta = addrToMetadata(reinterpret_cast<uintptr_t>(Ptr));
+  assert(Meta->Addr == reinterpret_cast<uintptr_t>(Ptr));
+  return Meta->Size;
+}
+
+std::size_t GuardedPoolAllocator::maximumAllocationSize() const {
+  return PageSize;
+}
+
+AllocationMetadata *GuardedPoolAllocator::addrToMetadata(uintptr_t Ptr) const {
+  assert(pointerIsMine(reinterpret_cast<void *>(Ptr)));
+  return Metadata + addrToSlot(Ptr);
+}
+
+std::size_t GuardedPoolAllocator::addrToSlot(uintptr_t Ptr) const {
+  std::size_t ByteOffsetFromPoolStart = Ptr - GetGuardedPagePool();
+  return ByteOffsetFromPoolStart / (maximumAllocationSize() + PageSize);
+}
+
+uintptr_t GuardedPoolAllocator::slotToAddr(std::size_t N) const {
+  return GetGuardedPagePool() + (PageSize * (1 + N)) + (maximumAllocationSize() * N);
+}
+
+uintptr_t GuardedPoolAllocator::getPageAddr(uintptr_t Ptr) const {
+  return Ptr & ~(static_cast<uintptr_t>(PageSize) - 1);
+}
+
+bool GuardedPoolAllocator::isGuardPage(uintptr_t Ptr) const {
+  return ((Ptr - GetGuardedPagePool()) / PageSize) % 2 == 0;
+}
+
+bool GuardedPoolAllocator::reserveSlot(std::size_t *PageIndex) {
+  if (FreeSlotsLength == 0)
+    return false;
+
+  size_t ReservedIndex;
+  if (NumUsedSlots < NumGuardedSlots) {
+    ReservedIndex = NumUsedSlots++;
+  } else {
+    ReservedIndex = Random::getRandomUnsigned64() % FreeSlotsLength;
+  }
+
+  *PageIndex = FreeSlots[ReservedIndex];
+
+  FreeSlots[ReservedIndex] = FreeSlots[--FreeSlotsLength];
+  return true;
+}
+
+void GuardedPoolAllocator::freeSlot(std::size_t PageIndex) {
+  assert(FreeSlotsLength < NumGuardedSlots);
+  FreeSlots[FreeSlotsLength++] = PageIndex;
+}
+
+void GuardedPoolAllocator::applyAllocationStrategy(std::size_t *Size,
+                                                   uintptr_t *AllocationPtr) {
+  assert(*Size > 0);
+
+  bool ShouldRightAlign = (*Size % 2 == 1);
+
+  if (!PerfectlyRightAlign) {
+    if (*Size == 3)
+      *Size = 4;
+    else if (*Size > 4 && *Size <= 8)
+      *Size = 8;
+    else if (*Size > 8 && (*Size & 15) != 0)
+      *Size += 16 - (*Size & 15);
+  }
+
+  if (ShouldRightAlign)
+    *AllocationPtr += maximumAllocationSize() - *Size;
+}
+
+NORETURN void GuardedPoolAllocator::reportErrorAndDie(uintptr_t AccessPtr,
+                                                      GwpAsanError Error) {
+  if (SingletonPtr == nullptr)
+    exit(EXIT_FAILURE);
+  SingletonPtr->reportErrorAndDieInternal(AccessPtr, Error);
+}
+
+std::size_t GuardedPoolAllocator::getNearestSlot(uintptr_t Ptr) const {
+  if (Ptr <= GetGuardedPagePool() + PageSize)
+    return 0;
+  else if (Ptr > GetGuardedPagePoolEnd() + PageSize)
+    return NumGuardedSlots - 1;
+
+  if (!isGuardPage(Ptr))
+    return addrToSlot(Ptr);
+
+  if (Ptr % PageSize <= PageSize / 2)
+    return addrToSlot(Ptr - PageSize); // Round down.
+  return addrToSlot(Ptr + PageSize);   // Round up.
+}
+
+AllocationMetadata *
+GuardedPoolAllocator::diagnoseUnknownError(uintptr_t AccessPtr,
+                                           GwpAsanError *Error) {
+  // Let's try and figure out what the source of this error is.
+  if (isGuardPage(AccessPtr)) {
+    std::size_t Slot = getNearestSlot(AccessPtr);
+    AllocationMetadata *Meta = addrToMetadata(slotToAddr(Slot));
+
+    // Ensure that this slot was allocated once upon a time.
+    if (!Meta->Addr)
+      return nullptr;
+
+    bool SlotRoundedDown = (Meta->Addr < AccessPtr);
+    if (SlotRoundedDown)
+      *Error = GwpAsanError::BUFFER_OVERFLOW;
+    else
+      *Error = GwpAsanError::BUFFER_UNDERFLOW;
+
+    return Meta;
+  } else {
+    // Access wasn't a guard page, check for use-after-free.
+    AllocationMetadata *Meta = addrToMetadata(AccessPtr);
+    if (Meta->IsDeallocated) {
+      *Error = GwpAsanError::USE_AFTER_FREE;
+      return Meta;
+    }
+  }
+
+  // If we have reached here, the error is still unknown. There is no metadata
+  // available.
+  return nullptr;
+}
+
+void printErrorTypeAndMaybeDie(GuardedPoolAllocator::GwpAsanError Error,
+                               uintptr_t AccessPtr, AllocationMetadata *Meta,
+                               options::Printf_t Printf) {
+  switch (Error) {
+  case GuardedPoolAllocator::GwpAsanError::UNKNOWN:
+    (*Printf)(
+        "The cause of the error is indeterminate. GWP-ASan couldn't "
+        "automatically determine the source of this memory error. "
+        "Note that this is almost definitely *not* a bug in GWP-ASan "
+        "itself. It was likely either caused by a memory bug for a non-sampled "
+        "allocation, or a wild memory access into the GWP-ASan pool.\n");
+    (*Printf)("When trying to access memory at: 0x%zx\n", AccessPtr);
+    exit(EXIT_FAILURE);
+  case GuardedPoolAllocator::GwpAsanError::USE_AFTER_FREE:
+    (*Printf)("Use after free occured when accessing memory at: 0x%zx\n",
+              AccessPtr);
+    break;
+  case GuardedPoolAllocator::GwpAsanError::DOUBLE_FREE:
+    (*Printf)("Double free occured when trying to free memory at: 0x%zx\n",
+              AccessPtr);
+    break;
+  case GuardedPoolAllocator::GwpAsanError::INVALID_FREE:
+    (*Printf)(
+        "Invalid (wild) free occured when trying to free memory at: 0x%zx\n",
+        AccessPtr);
+    // It's possible for an invalid free to fall onto a slot that has never been
+    // allocated. If this is the case, there is no valid metadata.
+    if (Meta == nullptr)
+      exit(EXIT_FAILURE);
+    break;
+  case GuardedPoolAllocator::GwpAsanError::BUFFER_OVERFLOW:
+    (*Printf)("Buffer overflow occured when accessing memory at: 0x%zx\n",
+              AccessPtr);
+    break;
+  case GuardedPoolAllocator::GwpAsanError::BUFFER_UNDERFLOW:
+    (*Printf)("Buffer underflow occured when accessing memory at: 0x%zx\n",
+              AccessPtr);
+    break;
+  }
+
+  (*Printf)("0x%zx is located ", AccessPtr);
+  if (AccessPtr < Meta->Addr)
+    (*Printf)("%zu bytes to the left ", Meta->Addr - AccessPtr);
+  else
+    (*Printf)("%zu bytes to the right ", AccessPtr - Meta->Addr);
+  (*Printf)("of a %zu-byte allocation located at 0x%zx\n", Meta->Size,
+            Meta->Addr);
+}
+
+void printThreadInformation(GuardedPoolAllocator::GwpAsanError Error,
+                            uintptr_t AccessPtr, AllocationMetadata *Meta,
+                            options::Printf_t Printf) {
+  (*Printf)("0x%zx was allocated by thread ", AccessPtr);
+  if (Meta->AllocationTrace.ThreadID == UINT64_MAX)
+    (*Printf)("UNKNOWN.\n");
+  else
+    (*Printf)("%zu.\n", Meta->AllocationTrace.ThreadID);
+
+  if (Error == GuardedPoolAllocator::GwpAsanError::USE_AFTER_FREE ||
+      Error == GuardedPoolAllocator::GwpAsanError::DOUBLE_FREE) {
+    (*Printf)("0x%zx was freed by thread ", AccessPtr);
+    if (Meta->AllocationTrace.ThreadID == UINT64_MAX)
+      (*Printf)("UNKNOWN.\n");
+    else
+      (*Printf)("%zu.\n", Meta->AllocationTrace.ThreadID);
+  }
+}
+
+NORETURN void
+GuardedPoolAllocator::reportErrorAndDieInternal(uintptr_t AccessPtr,
+                                                GwpAsanError Error) {
+  // Attempt to prevent races to re-use the same slot that triggered this error.
+  // This does not guarantee that there are no races, because another thread can
+  // take the locks during the time that the signal handler is being called.
+  tryLockAllMutexes();
+
+  (*Printf)("*** GWP-ASan detected a memory error ***\n");
+
+  AllocationMetadata *Meta = nullptr;
+
+  if (!pointerIsMine(reinterpret_cast<void *>(AccessPtr))) {
+    Error = GwpAsanError::UNKNOWN;
+  } else if (Error == GwpAsanError::UNKNOWN) {
+    Meta = diagnoseUnknownError(AccessPtr, &Error);
+  } else {
+    Meta = addrToMetadata(AccessPtr);
+    // Ensure that this slot has been previously allocated.
+    if (!Meta->Addr)
+      Meta = nullptr;
+  }
+
+  // Print the error information, and if there is no valid metadata, die here.
+  printErrorTypeAndMaybeDie(Error, AccessPtr, Meta, Printf);
+
+  // Ensure that we have a valid metadata pointer from this point forward.
+  if (Meta == nullptr) {
+    (*Printf)("GWP-ASan internal unreachable error. Metadata is not null.\n");
+    exit(EXIT_FAILURE);
+  }
+
+  printThreadInformation(Error, AccessPtr, Meta, Printf);
+  // TODO(hctim): Implement stack unwinding here. Ask the caller to provide us
+  // with the base pointer, and we unwind the stack to give a stack trace for
+  // the access.
+  // TODO(hctim): Implement dumping here of allocation/deallocation traces.
+
+  exit(EXIT_FAILURE);
+}
+
+void GuardedPoolAllocator::tryLockAllMutexes() { PoolMutex.tryLock(); }
+
+TLS_INITIAL_EXEC uint64_t GuardedPoolAllocator::NextSampleCounter = 0;
+GuardedPoolAllocator *GuardedPoolAllocator::SingletonPtr = nullptr;
+
+} // namespace gwp_asan

compiler-rt/lib/gwp_asan/guarded_pool_allocator_posix.cpp

+//===-- guarded_pool_allocator_posix.cpp ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gwp_asan/guarded_pool_allocator.h"
+
+#include <cstdlib>
+#include <errno.h>
+#include <signal.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+namespace gwp_asan {
+
+void *GuardedPoolAllocator::mapMemory(size_t Size) const {
+  void *Ptr =
+      mmap(nullptr, Size, PROT_NONE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+
+  if (Ptr == MAP_FAILED) {
+    (*Printf)("Failed to map guarded pool allocator memory, errno: %d\n",
+              errno);
+    (*Printf)("  mmap(nullptr, %zu, ...) failed.\n", Size);
+    exit(EXIT_FAILURE);
+  }
+  return Ptr;
+}
+
+void GuardedPoolAllocator::unmapMemory(void *Ptr, size_t Size) const {
+  if (munmap(Ptr, Size) != 0) {
+    (*Printf)("Failed to unmap guarded pool allocator memory, errno: %d\n",
+              errno);
+    (*Printf)("  munmap(%p, %zu) failed.\n", Ptr, Size);
+    exit(EXIT_FAILURE);
+  }
+}
+
+void GuardedPoolAllocator::markReadWrite(void *Ptr, size_t Size) const {
+  if (mprotect(Ptr, Size, PROT_READ | PROT_WRITE) != 0) {
+    (*Printf)(
+        "Failed to set guarded pool allocator memory at as RW, errno: %d\n",
+        errno);
+    (*Printf)("  mprotect(%p, %zu, RW) failed.\n", Ptr, Size);
+    exit(EXIT_FAILURE);
+  }
+}
+
+void GuardedPoolAllocator::markInaccessible(void *Ptr, size_t Size) const {
+  // mmap() a PROT_NONE page over the address to release it to the system, if
+  // we used mprotect() here the system would count pages in the quarantine
+  // against the RSS.
+  if (mmap(Ptr, Size, PROT_NONE, MAP_FIXED | MAP_ANONYMOUS | MAP_PRIVATE, 0,
+           0) == MAP_FAILED) {
+    (*Printf)("Failed to set guarded pool allocator memory as inaccessible, "
+              "errno: %d\n",
+              errno);
+    (*Printf)("  mmap(%p, %zu, NONE, ...) failed.\n", Ptr, Size);
+    exit(EXIT_FAILURE);
+  }
+}
+
+std::size_t GuardedPoolAllocator::getPlatformPageSize() {
+  return sysconf(_SC_PAGESIZE);
+}
+
+static void sigSegvHandler(int sig, siginfo_t *info, void *ucontext) {
+  if (sig != SIGSEGV)
+    return;
+
+  if (info == nullptr)
+    exit(EXIT_FAILURE);
+
+  gwp_asan::GuardedPoolAllocator::reportErrorAndDie(
+      reinterpret_cast<uintptr_t>(info->si_addr));
+}
+
+void GuardedPoolAllocator::installSignalHandlers() {
+  struct sigaction Action;
+  Action.sa_sigaction = sigSegvHandler;
+  Action.sa_flags = SA_SIGINFO;
+  sigaction(SIGSEGV, &Action, NULL);
+}
+
+uint64_t GuardedPoolAllocator::getThreadID() {
+#ifdef SYS_gettid
+  return syscall(SYS_gettid);
+#else
+  return UINT64_MAX;
+#endif
+}
+
+} // namespace gwp_asan

compiler-rt/lib/gwp_asan/mutex.h

+//===-- mutex.h -------------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// TODO(hctim): Move this implementation and Scudo's implementation into a
+// unified base implementaiton. We shouldn't need to have separate
+// implementations for this.
+
+#ifndef GWP_ASAN_MUTEX_H_
+#define GWP_ASAN_MUTEX_H_
+
+#include "gwp_asan/definitions.h"
+
+#include <cstdint>
+
+namespace gwp_asan {
+class Mutex {
+public:
+  void lock();
+  bool tryLock();
+  void unlock();
+
+private:
+  void yieldProcessor(uint8_t Count);
+  void yieldPlatform();
+
+  void lockSlow();
+
+  bool Semaphore = false;
+};
+
+class ScopedLock {
+public:
+  ScopedLock(Mutex *Mu) : Mutex(Mu) { Mutex->lock(); }
+  ~ScopedLock() { Mutex->unlock(); }
+
+private:
+  Mutex *Mutex;
+};
+
+ALWAYS_INLINE void Mutex::lock() {
+  if (tryLock())
+    return;
+  lockSlow();
+}
+
+ALWAYS_INLINE bool Mutex::tryLock() {
+  return __atomic_exchange_n(&Semaphore, true, __ATOMIC_ACQUIRE) == false;
+}
+
+ALWAYS_INLINE void Mutex::unlock() {
+  __atomic_store_n(&Semaphore, false, __ATOMIC_RELEASE);
+}
+
+ALWAYS_INLINE void Mutex::yieldProcessor(uint8_t Count) {
+#if defined(__i386__) || defined(__x86_64__)
+  __asm__ __volatile__("" ::: "memory");
+  for (uint8_t i = 0; i < Count; ++i)
+    __asm__ __volatile__("pause");
+#elif defined(__aarch64__) || defined(__arm__)
+  __asm__ __volatile__("" ::: "memory");
+  for (uint8_t i = 0; I < Count; ++i)
+    __asm__ __volatile__("yield");
+#endif
+  __asm__ __volatile__("" ::: "memory");
+}
+
+ALWAYS_INLINE void Mutex::lockSlow() {
+  for (uint32_t i = 0;; ++i) {
+    if (i < 10)
+      yieldProcessor(10);
+    else
+      yieldPlatform();
+
+    if (__atomic_load_n(&Semaphore, __ATOMIC_RELAXED) == false && tryLock())
+      return;
+  }
+}
+
+} // namespace gwp_asan
+
+#endif // GWP_ASAN_MUTEX_H_

compiler-rt/lib/gwp_asan/mutex_posix.cpp

+//===-- mutex_posix.cpp -----------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gwp_asan/mutex.h"
+
+#include <sched.h>
+
+namespace gwp_asan {
+void Mutex::yieldPlatform() { sched_yield(); }
+} // namespace gwp_asan

compiler-rt/lib/gwp_asan/optional/runtime_env_flag_parser.h

+//===-- runtime_env_flag_parser.h -------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_OPTIONAL_RUNTIME_ENV_FLAG_PARSER_H_
+#define GWP_ASAN_OPTIONAL_RUNTIME_ENV_FLAG_PARSER_H_
+
+#include "gwp_asan/options.h"
+#include "sanitizer_common/sanitizer_common.h"
+
+namespace gwp_asan {
+namespace options {
+
+// Parse the options from the GWP_ASAN_FLAGS environment variable.
+void initOptions();
+// Returns a pointer to the initialised options. Call initFlags() prior to
+// calling this function.
+Options *getOptions();
+
+} // namespace options
+} // namespace gwp_asan
+
+extern "C" {
+SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE const char *
+__gwp_asan_default_options();
+}
+
+#endif // GWP_ASAN_OPTIONAL_RUNTIME_ENV_FLAG_PARSER_H_

compiler-rt/lib/gwp_asan/optional/runtime_env_flag_parser.cpp

+//===-- runtime_env_flag_parser.cpp -----------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <cstdarg>
+#include <cstdlib>
+#include <cstring>
+
+#include "gwp_asan/optional/runtime_env_flag_parser.h"
+#include "gwp_asan/options.h"
+#include "sanitizer_common/sanitizer_common.h"
+#include "sanitizer_common/sanitizer_flag_parser.h"
+#include "sanitizer_common/sanitizer_flags.h"
+
+namespace gwp_asan {
+namespace options {
+int sanitizerPrintfWrapper(const char *Format, ...) {
+  va_list args;
+  va_start(args, Format);
+  __sanitizer::VariadicPrintf(Format, args);
+  va_end(args);
+  return 0;
+}
+
+static Options GWPAsanFlags; // Use via getOptions().
+
+static void registerGWPAsanFlags(__sanitizer::FlagParser *parser, Options *o) {
+#define GWP_ASAN_OPTION(Type, Name, DefaultValue, Description)                 \
+  RegisterFlag(parser, #Name, Description, &o->Name);
+#include "gwp_asan/options.inc"
+#undef GWP_ASAN_OPTION
+}
+
+static const char *getCompileDefinitionGWPAsanDefaultOptions() {
+#ifdef GWP_ASAN_DEFAULT_OPTIONS
+  return SANITIZER_STRINGIFY(GWP_ASAN_DEFAULT_OPTIONS);
+#else
+  return "";
+#endif
+}
+
+static const char *getGWPAsanDefaultOptions() {
+  return (&__gwp_asan_default_options) ? __gwp_asan_default_options() : "";
+}
+
+
+void initOptions() {
+  __sanitizer::SetCommonFlagsDefaults();
+  {
+    __sanitizer::CommonFlags cf;
+    cf.CopyFrom(*__sanitizer::common_flags());
+    cf.exitcode = 1;
+    OverrideCommonFlags(cf);
+  }
+  Options *o = getOptions();
+  o->setDefaults();
+
+  __sanitizer::FlagParser Parser;
+  registerGWPAsanFlags(&Parser, o);
+  RegisterCommonFlags(&Parser);
+
+  // Override from compile definition.
+  Parser.ParseString(getCompileDefinitionGWPAsanDefaultOptions());
+
+  // Override from user-specified string.
+  Parser.ParseString(getGWPAsanDefaultOptions());
+
+  // Override from environment.
+  Parser.ParseString(__sanitizer::GetEnv("GWP_ASAN_OPTIONS"));
+
+  __sanitizer::InitializeCommonFlags();
+
+  // Sanity checks and default settings for the parameters.
+  if (o->Enabled && o->NumUsableGuardedSlots <= 0) {
+    __sanitizer::Printf(
+        "GWP-ASan ERROR: NumUsableGuardedSlots must be >= 0 when "
+        "GWP-ASan is enabled.\n");
+    exit(EXIT_FAILURE);
+  }
+
+  if (o->Enabled && o->SampleRate < 1) {
+    __sanitizer::Printf("GWP-ASan ERROR: SampleRate must be >= 1 when "
+                        "GWP-ASan is enabled.\n");
+    exit(EXIT_FAILURE);
+  }
+
+  o->Printf = sanitizerPrintfWrapper;
+}
+
+Options *getOptions() { return &GWPAsanFlags; }
+
+} // namespace options
+} // namespace gwp_asan
+
+#if !SANITIZER_SUPPORTS_WEAK_HOOKS
+SANITIZER_INTERFACE_WEAK_DEF(const char *, __gwp_asan_default_options, void) {
+  return "";
+}
+#endif

compiler-rt/lib/gwp_asan/options.h

+//===-- options.h -----------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_OPTIONS_H_
+#define GWP_ASAN_OPTIONS_H_
+
+namespace gwp_asan {
+namespace options {
+// The function pointer type for printf(). Follows the standard format for the
+// C library. If the supported allocator exposes printing via a different
+// function signature, please provide a wrapper which has the normal printf()
+// signature, and pass the wrapper instead.
+typedef int (*Printf_t)(const char *Format, ...);
+
+struct Options {
+  Printf_t Printf = nullptr;
+
+  // Read the options from the included definitions file.
+#define GWP_ASAN_OPTION(Type, Name, DefaultValue, Description)                 \
+  Type Name = DefaultValue;
+#include "gwp_asan/options.inc"
+#undef GWP_ASAN_OPTION
+
+  void setDefaults() {
+#define GWP_ASAN_OPTION(Type, Name, DefaultValue, Description)                 \
+  Name = DefaultValue;
+#include "gwp_asan/options.inc"
+#undef GWP_ASAN_OPTION
+
+    Printf = nullptr;
+  }
+};
+} // namespace options
+} // namespace gwp_asan
+
+#endif // GWP_ASAN_OPTIONS_H_

compiler-rt/lib/gwp_asan/options.inc

+//===-- options.inc ---------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_OPTION
+#error "Define GWP_ASAN_OPTION prior to including this file!"
+#endif
+
+GWP_ASAN_OPTION(bool, Enabled, true, "Is GWP-ASan enabled? Defaults to true.")
+
+GWP_ASAN_OPTION(
+    bool, PerfectlyRightAlign, false,
+    "When allocations are right-aligned, should we perfectly align them up to "
+    "the page boundary? By default (false), we round up allocation size to the "
+    "nearest power of two (2, 4, 8, 16) up to a maximum of 16-byte alignment "
+    "for performance reasons. Setting this to true can find single byte "
+    "buffer-overflows at the cost of performance.")
+
+GWP_ASAN_OPTION(
+    int, NumUsableGuardedSlots, 16,
+    "Number of usable guarded slots in the allocation pool. Defaults to 16.")
+
+GWP_ASAN_OPTION(
+    unsigned long, SampleRate, 5000,
+    "The probability (1 / SampleRate) that an allocation is selected for "
+    "GWP-ASan sampling. Default is 5000. Sample rates up to 1 / (2^63) are "
+    "supported.")
+
+GWP_ASAN_OPTION(bool, InstallSignalHandlers, true,
+                "Install GWP-ASan signal handlers for SIGSEGV. This allows "
+                "better error reports by providing stack traces for "
+                "allocation/deallocation when reporting a use-after-free.")

compiler-rt/lib/gwp_asan/random.h

+//===-- random.h ------------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef GWP_ASAN_RANDOM_H_
+#define GWP_ASAN_RANDOM_H_
+
+#include <cstdint>
+
+#include "gwp_asan/definitions.h"
+
+namespace gwp_asan {
+
+class Random {
+public:
+  // TODO(hctim): This may have significant overhead for platforms where
+  // 64-bit arithmetic is emulated. Do we need less than a 2^32 chance of
+  // sampling?
+  // xorshift128+ (64-bit output). Avoids multiplication.
+  static ALWAYS_INLINE uint64_t getRandomUnsigned64() {
+    uint64_t A = RandomStateA;
+    const uint64_t B = RandomStateB;
+    RandomStateA = B;
+
+    A ^= A << 23;
+    A ^= A >> 17;
+    A ^= B ^ (B >> 26);
+
+    RandomStateB = A;
+    return A + B;
+  }
+
+  // xorshift* (64-bit output). This is primarily used to seed the xorshift128+
+  // generator.
+  static ALWAYS_INLINE uint64_t xorShiftStar64() {
+    uint64_t A = RandomStateA;
+    A ^= A >> 12;
+    A ^= A << 25;
+    A ^= A >> 27;
+    RandomStateA = A;
+    return A * 0x2545F4914F6CDD1D;
+  }
+
+private:
+  static TLS_INITIAL_EXEC uint64_t RandomStateA;
+  static TLS_INITIAL_EXEC uint64_t RandomStateB;
+};
+} // namespace gwp_asan
+
+#endif // GWP_ASAN_RANDOM_H_

compiler-rt/lib/gwp_asan/random.cpp

+//===-- random.cpp ----------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "gwp_asan/random.h"
+
+#include <ctime>
+
+namespace gwp_asan {
+TLS_INITIAL_EXEC uint64_t Random::RandomStateA =
+    static_cast<uint64_t>(time(nullptr));
+TLS_INITIAL_EXEC uint64_t Random::RandomStateB = xorShiftStar64();
+} // namespace gwp_asan

compiler-rt/lib/sanitizer_common/sanitizer_common.h

 
 #include "sanitizer_flags.h"
 #include "sanitizer_interface_internal.h"
 #include "sanitizer_internal_defs.h"
 #include "sanitizer_libc.h"
 #include "sanitizer_list.h"
 #include "sanitizer_mutex.h"
 
+#include <stdarg.h>
+
 #if defined(_MSC_VER) && !defined(__clang__)
 extern "C" void _ReadWriteBarrier();
 #pragma intrinsic(_ReadWriteBarrier)
 #endif
 
 namespace __sanitizer {
 
 struct AddressInfo;

 void SetLowLevelAllocateCallback(LowLevelAllocateCallback callback);
 
 // IO
 void CatastrophicErrorWrite(const char *buffer, uptr length);
 void RawWrite(const char *buffer);
 bool ColorizeReports();
 void RemoveANSIEscapeSequencesFromString(char *buffer);
 void Printf(const char *format, ...);
+void VariadicPrintf(const char *format, va_list ap);
 void Report(const char *format, ...);
 void SetPrintfAndReportCallback(void (*callback)(const char *));
 #define VReport(level, ...)                                              \
   do {                                                                   \
     if ((uptr)Verbosity() >= (level)) Report(__VA_ARGS__); \
   } while (0)
 #define VPrintf(level, ...)                                              \
   do {                                                                   \

compiler-rt/lib/sanitizer_common/sanitizer_printf.cc

 FORMAT(1, 2)
 void Printf(const char *format, ...) {
   va_list args;
   va_start(args, format);
   SharedPrintfCode(false, format, args);
   va_end(args);
 }
 
+void VariadicPrintf(const char *format, va_list ap) {
+  SharedPrintfCode(false, format, ap);
+}
+
 // Like Printf, but prints the current PID before the output string.
 FORMAT(1, 2)
 void Report(const char *format, ...) {
   va_list args;
   va_start(args, format);
   SharedPrintfCode(true, format, args);
   va_end(args);
 }

compiler-rt/lib/scudo/CMakeLists.txt

# android-changes-for-ndk-developers.md#changes-to-library-search-order
   endif()
 endif()
 
 # The minimal Scudo runtime does not inlude the UBSan runtime.
 set(SCUDO_MINIMAL_OBJECT_LIBS
   RTSanitizerCommonNoTermination
   RTSanitizerCommonLibc
   RTInterception)
+
+if (GWP_ASAN_SCUDO_HOOKS)
+  # Currently, Scudo uses the GwpAsan flag parser. This backs onto the flag
+  # parsing mechanism of sanitizer_common. Once Scudo has its own flag parsing,
+  # and parses GwpAsan options, you can remove this dependency.
+  list(APPEND SCUDO_MINIMAL_OBJECT_LIBS RTGwpAsan RTGwpAsanFlagParser)
+  list(APPEND SCUDO_CFLAGS -DGWP_ASAN_HOOKS)
+
+  # TODO(hctim): Why do we require the atomics library here? When compiling i386
+  # versions of Scudo, the linker fails to find __atomic_load_8 (used by
+  # GPA::shouldSample() to atomically load GPA::AdjustedSampleRate).
+  list(APPEND SCUDO_DYNAMIC_LINK_FLAGS -latomic)
+endif()
+
 set(SCUDO_OBJECT_LIBS ${SCUDO_MINIMAL_OBJECT_LIBS})
 set(SCUDO_DYNAMIC_LIBS ${SCUDO_MINIMAL_DYNAMIC_LIBS})
 
 if (FUCHSIA)
   list(APPEND SCUDO_CFLAGS -nostdinc++)
   list(APPEND SCUDO_DYNAMIC_LINK_FLAGS -nostdlib++)
 else()
   list(APPEND SCUDO_DYNAMIC_LIBS ${SANITIZER_CXX_ABI_LIBRARIES})

compiler-rt/lib/scudo/scudo_allocator.cpp

 #include "scudo_interface_internal.h"
 #include "scudo_tsd.h"
 #include "scudo_utils.h"
 
 #include "sanitizer_common/sanitizer_allocator_checks.h"
 #include "sanitizer_common/sanitizer_allocator_interface.h"
 #include "sanitizer_common/sanitizer_quarantine.h"
 
+#ifdef GWP_ASAN_HOOKS
+# include "gwp_asan/guarded_pool_allocator.h"
+# include "gwp_asan/optional/runtime_env_flag_parser.h"
+#endif // GWP_ASAN_HOOKS
+
 #include <errno.h>
 #include <string.h>
 
 namespace __scudo {
 
 // Global static cookie, initialized at start-up.
 static u32 Cookie;
 

 typedef QuarantineT::Cache QuarantineCacheT;
 COMPILER_CHECK(sizeof(QuarantineCacheT) <=
                sizeof(ScudoTSD::QuarantineCachePlaceHolder));
 
 QuarantineCacheT *getQuarantineCache(ScudoTSD *TSD) {
   return reinterpret_cast<QuarantineCacheT *>(TSD->QuarantineCachePlaceHolder);
 }
 
+#ifdef GWP_ASAN_HOOKS
+  static gwp_asan::GuardedPoolAllocator GuardedAlloc;
+#endif // GWP_ASAN_HOOKS
+
 struct Allocator {
   static const uptr MaxAllowedMallocSize =
       FIRST_32_SECOND_64(2UL << 30, 1ULL << 40);
 
   BackendT Backend;
   QuarantineT Quarantine;
 
   u32 QuarantineChunksUpToSize;
 
   bool DeallocationTypeMismatch;
   bool ZeroContents;
   bool DeleteSizeMismatch;
 
   bool CheckRssLimit;
   uptr HardRssLimitMb;
   uptr SoftRssLimitMb;
   atomic_uint8_t RssLimitExceeded;
   atomic_uint64_t RssLastCheckedAtNS;
 
   explicit Allocator(LinkerInitialized)
     : Quarantine(LINKER_INITIALIZED) {}
 
   NOINLINE void performSanityChecks();
 
   void init() {
     SanitizerToolName = "Scudo";
     PrimaryAllocatorName = "ScudoPrimary";
     SecondaryAllocatorName = "ScudoSecondary";
 
     initFlags();
struct Allocator {
   }
 
   NOINLINE bool isRssLimitExceeded();
 
   // Allocates a chunk.
   void *allocate(uptr Size, uptr Alignment, AllocType Type,
                  bool ForceZeroContents = false) {
     initThreadMaybe();
+
+#ifdef GWP_ASAN_HOOKS
+    if (UNLIKELY(GuardedAlloc.shouldSample())) {
+      if (void *Ptr = GuardedAlloc.allocate(Size))
+        return Ptr;
+    }
+#endif // GWP_ASAN_HOOKS
+
     if (UNLIKELY(Alignment > MaxAlignment)) {
       if (AllocatorMayReturnNull())
         return nullptr;
       reportAllocationAlignmentTooBig(Alignment, MaxAlignment);
     }
     if (UNLIKELY(Alignment < MinAlignment))
       Alignment = MinAlignment;
 
void deallocate(void *Ptr, uptr DeleteSize, uptr DeleteAlignment,
     // where the only heap operation performed in a thread would be a free past
     // the TLS destructors, ending up in initialized thread specific data never
     // being destroyed properly. Any other heap operation will do a full init.
     initThreadMaybe(/*MinimalInit=*/true);
     if (SCUDO_CAN_USE_HOOKS && &__sanitizer_free_hook)
       __sanitizer_free_hook(Ptr);
     if (UNLIKELY(!Ptr))
       return;
+
+#ifdef GWP_ASAN_HOOKS
+    if (UNLIKELY(GuardedAlloc.pointerIsMine(Ptr))) {
+      GuardedAlloc.deallocate(Ptr);
+      return;
+    }
+#endif // GWP_ASAN_HOOKS
+
     if (UNLIKELY(!Chunk::isAligned(Ptr)))
       dieWithMessage("misaligned pointer when deallocating address %p\n", Ptr);
     UnpackedHeader Header;
     Chunk::loadHeader(Ptr, &Header);
     if (UNLIKELY(Header.State != ChunkAllocated))
       dieWithMessage("invalid chunk state when deallocating address %p\n", Ptr);
     if (DeallocationTypeMismatch) {
       // The deallocation type has to match the allocation one.
#endif // GWP_ASAN_HOOKS
     (void)DeleteAlignment;  // TODO(kostyak): verify that the alignment matches.
     quarantineOrDeallocateChunk(Ptr, &Header, Size);
   }
 
   // Reallocates a chunk. We can save on a new allocation if the new requested
   // size still fits in the chunk.
   void *reallocate(void *OldPtr, uptr NewSize) {
     initThreadMaybe();
+
+#ifdef GWP_ASAN_HOOKS
+    if (UNLIKELY(GuardedAlloc.pointerIsMine(OldPtr))) {
+      size_t OldSize = GuardedAlloc.getSize(OldPtr);
+      void *NewPtr = allocate(NewSize, MinAlignment, FromMalloc);
+      if (NewPtr)
+        memcpy(NewPtr, OldPtr, (NewSize < OldSize) ? NewSize : OldSize);
+      GuardedAlloc.deallocate(OldPtr);
+      return NewPtr;
+    }
+#endif // GWP_ASAN_HOOKS
+
     if (UNLIKELY(!Chunk::isAligned(OldPtr)))
       dieWithMessage("misaligned address when reallocating address %p\n",
                      OldPtr);
     UnpackedHeader OldHeader;
     Chunk::loadHeader(OldPtr, &OldHeader);
     if (UNLIKELY(OldHeader.State != ChunkAllocated))
       dieWithMessage("invalid chunk state when reallocating address %p\n",
                      OldPtr);
#endif // GWP_ASAN_HOOKS
     return NewPtr;
   }
 
   // Helper function that returns the actual usable size of a chunk.
   uptr getUsableSize(const void *Ptr) {
     initThreadMaybe();
     if (UNLIKELY(!Ptr))
       return 0;
+
+#ifdef GWP_ASAN_HOOKS
+    if (UNLIKELY(GuardedAlloc.pointerIsMine(Ptr)))
+      return GuardedAlloc.getSize(Ptr);
+#endif // GWP_ASAN_HOOKS
+
     UnpackedHeader Header;
     Chunk::loadHeader(Ptr, &Header);
     // Getting the usable size of a chunk only makes sense if it's allocated.
     if (UNLIKELY(Header.State != ChunkAllocated))
       dieWithMessage("invalid chunk state when sizing address %p\n", Ptr);
     return Chunk::getUsableSize(Ptr, &Header);
   }
 

 static Allocator Instance(LINKER_INITIALIZED);
 
 static BackendT &getBackend() {
   return Instance.Backend;
 }
 
 void initScudo() {
   Instance.init();
+#ifdef GWP_ASAN_HOOKS
+  gwp_asan::options::initOptions();
+  GuardedAlloc.init(*gwp_asan::options::getOptions());
+#endif // GWP_ASAN_HOOKS
 }
 
 void ScudoTSD::init() {
   getBackend().initCache(&Cache);
   memset(QuarantineCachePlaceHolder, 0, sizeof(QuarantineCachePlaceHolder));
 }
 
 void ScudoTSD::commitBack() {

compiler-rt/test/gwp_asan/CMakeLists.txt

+set(GWP_ASAN_LIT_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
+set(GWP_ASAN_LIT_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
+
+set(GWP_ASAN_TESTSUITES)
+
+set(GWP_ASAN_TEST_DEPS ${SANITIZER_COMMON_LIT_TEST_DEPS} gwp_asan scudo)
+
+configure_lit_site_cfg(
+  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
+  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
+  )
+
+set(GWP_ASAN_TEST_ARCH ${GWP_ASAN_SUPPORTED_ARCH})
+foreach(arch ${GWP_ASAN_TEST_ARCH})
+  set(GWP_ASAN_TEST_TARGET_ARCH ${arch})
+  string(TOLOWER "-${arch}" GWP_ASAN_TEST_CONFIG_SUFFIX)
+  get_test_cc_for_arch(${arch} GWP_ASAN_TEST_TARGET_CC GWP_ASAN_TEST_TARGET_CFLAGS)
+  string(TOUPPER ${arch} ARCH_UPPER_CASE)
+  set(CONFIG_NAME ${ARCH_UPPER_CASE}${OS_NAME}Config)
+
+  configure_lit_site_cfg(
+    ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
+    ${CMAKE_CURRENT_BINARY_DIR}/${CONFIG_NAME}/lit.site.cfg)
+  list(APPEND GWP_ASAN_TESTSUITES ${CMAKE_CURRENT_BINARY_DIR}/${CONFIG_NAME})
+endforeach()
+
+add_lit_testsuite(check-gwp_asan "Running the GWP-ASan tests"
+  ${GWP_ASAN_TESTSUITES}
+  DEPENDS ${GWP_ASAN_TEST_DEPS})
+set_target_properties(check-gwp_asan PROPERTIES FOLDER "Compiler-RT Misc")

compiler-rt/test/gwp_asan/alignment_power_of_two.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=1: %run %t
+
+#include <cstdint>
+#include <cstdlib>
+#include <utility>
+#include <vector>
+
+int main() {
+  std::vector<std::pair<int, int>> AllocSizeToAlignment = {
+      {1, 1},   {2, 2},   {3, 4},       {4, 4},      {5, 8},   {7, 8},
+      {8, 8},   {9, 16},  {15, 16},     {16, 16},    {17, 16}, {31, 16},
+      {32, 16}, {33, 16}, {4095, 4096}, {4096, 4096}};
+
+
+  for (const auto& P : AllocSizeToAlignment) {
+    char *Ptr = reinterpret_cast<char *>(malloc(P.first));
+    if (reinterpret_cast<uintptr_t>(Ptr) % P.second != 0) {
+      exit(EXIT_FAILURE);
+    }
+
+    // Normally causes buffer overflow, but alignment should prevent this.
+    volatile char x = *(Ptr + P.second - 1);
+    free(Ptr);
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/alignment_static.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %env_gwp_asan_options= %run %t 4
+// RUN: %env_gwp_asan_options= %run %t 8
+// RUN: %env_gwp_asan_options= %run %t 16
+
+#include <cstdint>
+#include <cstdlib>
+
+int main(int argc, char** argv) {
+  if (argc < 2)
+    exit(EXIT_FAILURE);
+
+  int alignment = atoi(argv[1]);
+
+  char *Ptr = reinterpret_cast<char *>(malloc(alignment - 1));
+
+  if (reinterpret_cast<uintptr_t>(Ptr) % alignment != 0)
+    exit(EXIT_FAILURE);
+
+  // Normally causes buffer overflow, but alignment should prevent this.
+  volatile char x = *(Ptr + alignment - 1);
+  free(Ptr);
+  return 0;
+}

compiler-rt/test/gwp_asan/allocator_fallback.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=1 %run %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=2 %run %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=15 %run %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=16 %run %t
+// RUN: %env_gwp_asan_options=NumUsableGuardedPages=17 %run %t
+
+#include <cstdlib>
+
+int main() {
+  void* Pointers[16];
+  for (unsigned i = 0; i < 16; ++i) {
+    char *Ptr = reinterpret_cast<char*>(malloc(1 << i));
+    Pointers[i] = Ptr;
+    for (unsigned j = 0; j < i; ++j) {
+      *(Ptr + j) = 0x0;
+    }
+  }
+
+  for (unsigned i = 0; i < 16; ++i) {
+    free(Pointers[i]);
+  }
+
+  return 0;
+}

compiler-rt/test/gwp_asan/double_delete.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Double free occured when trying to free memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = new char;
+  delete Ptr;
+  delete Ptr;
+  return 0;
+}

compiler-rt/test/gwp_asan/double_deletea.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Double free occured when trying to free memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = new char[50];
+  delete[] Ptr;
+  delete[] Ptr;
+  return 0;
+}

compiler-rt/test/gwp_asan/double_free.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Double free occured when trying to free memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char*>(malloc(10));
+  free(Ptr);
+  free(Ptr);
+  return 0;
+}

compiler-rt/test/gwp_asan/heap_buffer_overflow.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Buffer overflow occured when accessing memory at:
+// CHECK: is located 49 bytes to the right of a 49-byte allocation
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char *>(malloc(49));
+  volatile char x = *(Ptr + 49);
+  return 0;
+}

compiler-rt/test/gwp_asan/heap_buffer_underflow.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Buffer underflow occured when accessing memory at:
+// CHECK: is located 1 bytes to the left of a 50-byte allocation
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char*>(malloc(50));
+  volatile char x = *(Ptr - 1);
+  return 0;
+}

compiler-rt/test/gwp_asan/invalid_free_left.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Invalid (wild) free occured when trying to free memory at:
+// CHECK: is located 1 bytes to the left of a 10-byte allocation located at
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char*>(malloc(10));
+  free(Ptr - 1);
+  return 0;
+}

compiler-rt/test/gwp_asan/invalid_free_right.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Invalid (wild) free occured when trying to free memory at:
+// CHECK: is located 1 bytes to the right of a 9-byte allocation located at
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char*>(malloc(9));
+  free(Ptr + 1);
+  return 0;
+}

compiler-rt/test/gwp_asan/lit.cfg

+# -*- Python -*-
+
+import os
+
+# Setup config name.
+config.name = 'GWP-ASan' + config.name_suffix
+
+# Setup source root.
+config.test_source_root = os.path.dirname(__file__)
+
+# Test suffixes.
+config.suffixes = ['.c', '.cc', '.cpp', '.test']
+
+# C & CXX flags.
+c_flags = ([config.target_cflags])
+
+# Android doesn't want -lrt.
+#if not config.android:
+  # c_flags += ["-lrt"]
+
+cxx_flags = (c_flags + config.cxx_mode_flags + ["-std=c++11"])
+
+gwp_asan_flags = ["-fsanitize=scudo"]
+
+def build_invocation(compile_flags):
+  return " " + " ".join([config.clang] + compile_flags) + " "
+
+# Add substitutions.
+config.substitutions.append(("%clang ", build_invocation(c_flags)))
+config.substitutions.append(("%clang_gwp_asan ", build_invocation(c_flags + gwp_asan_flags)))
+config.substitutions.append(("%clangxx_gwp_asan ", build_invocation(cxx_flags + gwp_asan_flags)))
+
+# Platform-specific default GWP_ASAN for lit tests. Ensure that GWP-ASan is
+# enabled and that it samples every allocation.
+default_gwp_asan_options = 'Enabled=1:SampleRate=1'
+
+config.environment['GWP_ASAN_OPTIONS'] = default_gwp_asan_options
+default_gwp_asan_options += ':'
+config.substitutions.append(('%env_gwp_asan_options=',
+                             'env GWP_ASAN_OPTIONS=' + default_gwp_asan_options))
+
+# GWP-ASan tests are currently supported on Linux only.
+if config.host_os not in ['Linux']:
+   config.unsupported = True

compiler-rt/test/gwp_asan/lit.site.cfg.in

+@LIT_SITE_CFG_IN_HEADER@
+
+config.name_suffix = "@GWP_ASAN_TEST_CONFIG_SUFFIX@"
+config.target_arch = "@GWP_ASAN_TEST_TARGET_ARCH@"
+config.target_cflags = "@GWP_ASAN_TEST_TARGET_CFLAGS@"
+
+# Load common config for all compiler-rt lit tests.
+lit_config.load_config(config, "@COMPILER_RT_BINARY_DIR@/test/lit.common.configured")
+
+# Load tool-specific config that would do the real work.
+lit_config.load_config(config, "@GWP_ASAN_LIT_SOURCE_DIR@/lit.cfg")

compiler-rt/test/gwp_asan/pattern_calloc_free.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %run %t
+// RUN: %env_gwp_asan_options= %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+
+#include <cstdlib>
+
+int main() {
+  for (unsigned i = 1; i < 0x100000; i <<= 1) {
+    char *Ptr = reinterpret_cast<char *>(calloc(i, sizeof(char)));
+
+    for (unsigned j = 0; j < i; ++j) {
+      *(Ptr + j) = 0x0;
+    }
+
+    free(Ptr);
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/pattern_malloc_free.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %run %t
+// RUN: %env_gwp_asan_options= %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+
+#include <cstdlib>
+
+int main() {
+  for (unsigned i = 1; i < 0x100000; i <<= 1) {
+    char *Ptr = reinterpret_cast<char *>(malloc(i));
+
+    for (unsigned j = 0; j < i; ++j) {
+      *(Ptr + j) = 0x0;
+    }
+
+    free(Ptr);
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/pattern_new_delete.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %run %t
+// RUN: %env_gwp_asan_options= %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+
+int main() {
+  for (unsigned i = 1; i < 0x100000; i <<= 1) {
+    char *Ptr = new char;
+    *Ptr = 0x0;
+    delete Ptr;
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/pattern_newa_deletea.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %run %t
+// RUN: %env_gwp_asan_options= %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+
+int main() {
+  for (unsigned i = 1; i < 0x100000; i <<= 1) {
+    char *Ptr = new char[i];
+
+    for (unsigned j = 0; j < i; ++j) {
+      *(Ptr + j) = 0x0;
+    }
+
+    delete[] Ptr;
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/pattern_realloc_free.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %run %t
+// RUN: %env_gwp_asan_options= %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+// RUN: %env_gwp_asan_options=PerfectRightAlign=true %run %t
+
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char *>(realloc(nullptr, 1));
+  for (unsigned i = 1; i < 0x100000; i <<= 1) {
+    Ptr = reinterpret_cast<char *>(realloc(Ptr, i));
+
+    for (unsigned j = 0; j < i; ++j) {
+      *(Ptr + j) = 0x0;
+    }
+  }
+  return 0;
+}

compiler-rt/test/gwp_asan/reuse_quarantine.cpp

+// REQUIRES: gwp_asan
+// This test ensures that normal allocation/memory access/deallocation works
+// as expected.
+
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=1:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=2:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=8:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=16:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=32:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=64:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=128:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+// RUN: %env_gwp_asan_options=PerfectlyRightAlign=true:NumUsableGuardedPages=129:RightAlignAllocations=true not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Buffer overflow occured when accessing memory at:
+// CHECK: is located 1 bytes to the right of a 1-byte allocation
+
+#include <cstdlib>
+
+int main() {
+  for (unsigned i = 0; i < 128; ++i) {
+    char *Ptr = reinterpret_cast<char*>(malloc(1));
+    *Ptr = 0x0;
+    free(Ptr);
+  }
+
+  char *Ptr = reinterpret_cast<char*>(malloc(1));
+  volatile char x = *(Ptr + 1);
+
+  return 0;
+}

compiler-rt/test/gwp_asan/use_after_delete.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Use after free occured when accessing memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = new char;
+
+  *Ptr = 0x0;
+
+  delete Ptr;
+  volatile char x = *Ptr;
+  return 0;
+}

compiler-rt/test/gwp_asan/use_after_deletea.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Use after free occured when accessing memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = new char[10];
+
+  for (unsigned i = 0; i < 10; ++i) {
+    *(Ptr + i) = 0x0;
+  }
+
+  delete[] Ptr;
+  volatile char x = *Ptr;
+  return 0;
+}

compiler-rt/test/gwp_asan/use_after_free.cpp

+// REQUIRES: gwp_asan
+// RUN: %clangxx_gwp_asan %s -o %t
+// RUN: not %run %t 2>&1 | FileCheck %s
+
+// CHECK: GWP-ASan detected a memory error
+// CHECK: Use after free occured when accessing memory at:
+
+#include <cstdlib>
+
+int main() {
+  char *Ptr = reinterpret_cast<char*>(malloc(10));
+
+  for (unsigned i = 0; i < 10; ++i) {
+    *(Ptr + i) = 0x0;
+  }
+
+  free(Ptr);
+  volatile char x = *Ptr;
+  return 0;
+}

compiler-rt/test/lit.common.cfg

if config.has_lld:
   config.available_features.add('lld-available')
 
 if config.use_lld:
   config.available_features.add('lld')
 
 if config.can_symbolize:
   config.available_features.add('can-symbolize')
 
+if config.gwp_asan_scudo_hooks:
+  config.available_features.add('gwp_asan')
+
 lit.util.usePlatformSdkOnDarwin(config, lit_config)
 
 if config.host_os == 'Darwin':
   osx_version = (10, 0, 0)
   try:
     osx_version = subprocess.check_output(["sw_vers", "-productVersion"])
     osx_version = tuple(int(x) for x in osx_version.split('.'))
     if len(osx_version) == 2: osx_version = (osx_version[0], osx_version[1], 0)

compiler-rt/test/lit.common.configured.in

 set_default("use_thinlto", False)
 set_default("use_lto", config.use_thinlto)
 set_default("use_newpm", False)
 set_default("android", @ANDROID_PYBOOL@)
 set_default("android_ndk_version", @ANDROID_NDK_VERSION@)
 set_default("android_serial", "@ANDROID_SERIAL_FOR_TESTING@")
 set_default("android_files_to_push", [])
 set_default("have_rpc_xdr_h", @HAVE_RPC_XDR_H@)
+set_default("gwp_asan_scudo_hooks", @GWP_ASAN_SCUDO_HOOKS_PYBOOL@)
 config.available_features.add('target-is-%s' % config.target_arch)
 
 if config.enable_per_target_runtime_dir:
   set_default("target_suffix", "")
 else:
   set_default("target_suffix", "-%s" % config.target_arch)
 
 set_default("have_zlib", "@HAVE_LIBZ@")

compiler-rt/test/scudo/lit.cfg

 config.substitutions.append(("%shared_minlibscudo", shared_minlibscudo))
 
 # Platform-specific default SCUDO_OPTIONS for lit tests.
 default_scudo_opts = ''
 if config.android:
   # Android defaults to abort_on_error=1, which doesn't work for us.
   default_scudo_opts = 'abort_on_error=0'
 
+# Disable GWP-ASan for scudo internal tests.
+gwp_asan_options = 'Enabled=0'
+config.environment['GWP_ASAN_OPTIONS'] = gwp_asan_options
+config.substitutions.append(('%env_gwp_asan_options=',
+                             'env GWP_ASAN_OPTIONS=' + gwp_asan_options))
+
 if default_scudo_opts:
   config.environment['SCUDO_OPTIONS'] = default_scudo_opts
   default_scudo_opts += ':'
 config.substitutions.append(('%env_scudo_opts=',
                              'env SCUDO_OPTIONS=' + default_scudo_opts))
 
 # Hardened Allocator tests are currently supported on Linux only.
 if config.host_os not in ['Linux']:
    config.unsupported = True

llvm/docs/GWPASan.rst

+========
+GWP-ASan
+========
+
+.. contents::
+   :local:
+   :depth: 2
+
+Introduction
+============
+
+GWP-ASan is a sampled allocator framework that assists in finding use-after-free
+and heap-buffer-overflow bugs in production environments. It informally is a
+recursive acronym, "**G**\WP-ASan **W**\ill **P**\rovide **A**\llocation
+**SAN**\ity".
+
+GWP-ASan is based on the classic
+`Electric Fence Malloc Debugger <https://linux.die.net/man/3/efence>`_, with a
+key adaptation. Notably, we only choose a very small percentage of allocations
+to sample, and apply guard pages to these sampled allocations only. The sampling
+is small enough to allow us to have very low performance overhead.
+
+There is a small, tunable memory overhead that is fixed for the lifetime of the
+process. This is approximately ~60KiB per process using the default settings,
+depending on the average size of your allocations. Future improvements should
+drastically reduce this amount.
+
+GWP-ASan vs. ASan
+=================
+
+Unlike `AddressSanitizer <https://clang.llvm.org/docs/AddressSanitizer.html>`_,
+GWP-ASan does not induce a significant performance overhead. ASan often requires
+the use of dedicated canaries to be viable in production environments, and as
+such is often impractical.
+
+GWP-ASan's is only capable of finding a subset of the memory issues detected by
+ASan. Furthermore, GWP-ASan's bug detection capabilities are only probabilistic.
+As such, we recommend using ASan over GWP-ASan in testing, as well as anywhere
+else that guaranteed error detection is more valuable than the 2x execution
+slowdown/binary size bloat. For the majority of production environments, this
+impact is too high, and GWP-ASan proves extremely useful.
+
+Design
+======
+
+**Please note:** The implementation of GWP-ASan is largely in-flux, and these
+details are subject to change. There are currently other implementations of
+GWP-ASan, such as the implementation featured in
+`Chromium <https://cs.chromium.org/chromium/src/components/gwp_asan/>`_. The
+long-term support goal is to ensure feature-parity where reasonble, and to
+support compiler-rt as the reference implementation.
+
+Allocator Support
+-----------------
+
+GWP-ASan is not a replacement for a traditional allocator. Instead, it works by
+inserting stubs into an existing allocator to redirect allocations when they're
+chosen to be sampled. These stubs are generally implemented in the implementaion
+of ``malloc()``, ``free()`` and ``realloc()``. The stubs are extremely small,
+which makes using GWP-ASan in most allocators fairly trivial. The stubs follow
+the same general pattern (example ``malloc()`` pseudocode below):
+
+.. code:: cpp
+  #ifdef INSTALL_GWP_ASAN_STUBS
+    gwp_asan::GuardedPoolAllocator GWPASanAllocator;
+  #endif
+  
+  void* YourAllocator::malloc(..) {
+  #ifdef INSTALL_GWP_ASAN_STUBS
+    if (GWPASanAllocator.shouldSample(..))
+      return GWPASanAllocator.allocate(..);
+  #endif
+  
+    // ... the rest of your allocator code here.
+  }
+
+Then, all the supported allocator needs to do is compile with
+``-DINSTALL_GWP_ASAN_STUBS`` and link against the GWP-ASan library!
+
+Guarded Allocation Pool
+-----------------------
+
+The core of GWP-ASan is the guarded allocation pool. Each individual sampled
+allocation is backed using its own *guarded* page. Each guarded page is
+surrounded by two *guard* pages, which are mapped as inaccessible. We create a
+contiguous buffer of this ``guard_page | guarded_page | guard_page`` pattern,
+which we call the *guarded allocation pool*.
+
+Buffer Underflow/Overflow Detection
+-----------------------------------
+
+We gain buffer-overflow and buffer-underflow through these guard pages. When a
+memory access overruns the allocated buffer, it will touch the inaccessible
+guard page, causing memory exception. This exception is caught and handled by
+the internal crash handler. Because each allocation is recorded with metadata
+about where (and by what thread) it was allocated and deallocated, we can
+provide helpful information that will help identify the root cause of the bug.
+
+In order to increase our detection of overflows, we generally align each sampled
+allocation to the right hand side of the guard page. This means that a 1-byte
+overflow will touch the guard page and trigger the crash handler. We can also
+optionally choose to left-align sampled allocations, which increases the chance
+of finding buffer underflows at the cost of reduced buffer overflow detection.
+
+Use after Free Detection
+------------------------
+
+The guarded allocation pool also provide use-after-free detection. Whenever a
+sampled allocation is deallocated, we map the guard page as inaccessible. Any
+memory accesses after deallocation will thus trigger the crash handler, and we
+can provide useful information about the source of the error.
+
+Please note that the use-after-free detection for a guarded allocation is only
+emphemeral. We reuse inaccessible slots on-demand as we have a fixed number of
+them, and wish to avoid starving the allocation pool to solely catch
+use-after-frees. We choose the inaccessible slot to reuse at random to provide
+a chance of detecting long-lived use-after-frees.
+
+Usage
+=====
+
+Currently, the only allocator that supports GWP-ASan is the
+`Scudo Hardened Allocator <https://llvm.org/docs/ScudoHardenedAllocator.html>`_.
+Building compiler-rt will install GWP-ASan into your local version of Scudo, and
+any binary built with Scudo will also have GWP-ASan enabled by default.
+Instructions on using Scudo for your applications can be found on the Scudo wiki
+page.
+
+Options
+-------
+
+GWP-ASan is configured on a per-allocator basis. We provide a default
+implementation of configuration that is used by Scudo. Several aspects of
+GWP-ASan can be configured on a per process basis through the following ways:
+
+- at compile time, by defining ``GWP_ASAN_DEFAULT_OPTIONS`` to the options
+  string you want set by default;
+
+- by defining a ``__gwp_asan_default_options`` function in one's program that
+  returns the options string to be parsed. Said function must have the following
+  prototype: ``extern "C" const char* __gwp_asan_default_options(void)``, with a
+  default visibility. This will override the compile time define;
+
+- through the environment variable ``GWP_ASAN_OPTIONS``, containing the options string
+  to be parsed. Options defined this way will override any definition made
+  through ``__gwp_asandefault_options``.
+
+The options string follows a syntax similar to ASan, where distinct options
+can be assigned in the same string, separated by colons.
+
+For example, using the environment variable:
+
+.. code:: console
+
+  GWP_ASAN_OPTIONS="NumUsableGuardedPages=16:SampleRate=5000" ./a.out
+
+Or using the function:
+
+.. code:: cpp
+
+  extern "C" const char *__gwp_asan_default_options() {
+    return "NumUsableGuardedPages=16:SampleRate=5000";
+  }
+
+The following options are available:
+
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| Option                 |  Default           |  Description                                                                    |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| Enabled                |  true              |  Is GWP-ASan enabled?                                                           |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| RightAlignAll          |  false             |  Should we right-align all allocations? By default (false), we left-align       |
+|                        |                    |  even-sized allocations, and right-align odd-sized allocations.                 |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| AlignmentStrategy      |  PowerOfTwo        |  When allocations are right-aligned, should we perfectly align them up to the   |
+|                        |                    |  page boundary? By default (false), we round up allocation size to the nearest  |
+|                        |                    |  power of two (2, 4, 8, 16) up to a maximum of 16-byte alignment for            |
+|                        |                    |  performance reasons. Setting this to true can find single byte                 |
+|                        |                    |  buffer-overflows at the cost of performance.                                   |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| NumUsableGuardedPages  |  16                |  Number of usable guarded pages in the allocation pool.                         |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| SampleRate             |  5000              |  The probability (1 / SampleRate) that a page is selected for GWP-ASan          |
+|                        |                    |  sampling. Sample rates up to 1 / (2^64) are supported, values higher than      |
+|                        |                    |  this are truncated.                                                            |
++------------------------+--------------------+---------------------------------------------------------------------------------+
+| InstallSignalHandlers  |  true              |  Install GWP-ASan signal handlers for SIGSEGV. This allows better error reports |
+|                        |                    |  by providing stack traces for allocation/deallocation when reporting a         |
+|                        |                    |  use-after-free.                                                                |
++------------------------+--------------------+---------------------------------------------------------------------------------+