This is an archive of the discontinued LLVM Phabricator instance.

Summary: [asan][win][msvc] override new and delete and seperate TUs
Needs RevisionPublic

Authored by barcharcraz on Jul 20 2023, 12:59 PM.

Details

Summary

Currently asan simply exports each overridden new/delete function from the DLL, this works fine normally, but fails if the user is overriding some, but not all, of these functions. In this case the non-overridden functions still come from the asan DLL, but they can't correctly call the user provided override (for example sized op delete should fall back to scalar op delete, if a scalar op delete is provided). Things were also broken in the static build because all the asan overrides were exported from the same TU, and so if you overrode one but not all of them then you'd get ODR violations. This PR should fix both of these cases, but the static case isn't really tested (and indeed one such test does fail) because linking asan statically basically doesn't work on windows right now with LLVM's version of asan. In fact, while we did fix this in our fork, it was a huge mess and we've now made the dynamic version work in all situations (/MD, /MT, /MDd, /MTd, etc) instead.

The following is the description from the internal PR that implemented most of this feature.

Previously, operator new/delete were provided as DLL exports when linking dynamically and wholearchived when linked statically. Both scenarios were broken. When linking statically, the user could not define their own op new/delete, because they were already brought into the link by ASAN. When dynamically linking, if the user provided some but not all of the overloads, new and delete would be partially hooked. For example, if the user defined scalar op delete, but the program then called sized op delete, the sized op delete would still be the version provided by ASAN instead of falling back to the user-defined scalar op delete, like the standard requires.

The change <internal PR number>: ASAN operator new/delete fallbacks in the ASAN libraries fixes this moving all operator new/delete definitions to be statically linked. However, this still won't work if /InferAsanLibs still whole-archives everything since then all the op new/deletes would always be provided by ASAN, which is why these changes are necessary.

With these changes, we will no longer wholearchive all of ASAN and will leave the c++ parts (the op new/delete definitions) to be included as a default library. However, it is also necessary to ensure that the asan library with op new/delete will be searched before the corresponding CRT library with the same op new/delete definitions. To accomplish this, we make sure to add the asan library to the beginning of the default lib list, or move it explicitly to the front if it's already in the list. If the C runtime library is explicitly provided, we make sure to warn the user if the current linker line will result in operator new/delete not being provided by ASAN.

Note that the rearrangement of defaultlibs is not in this diff.

Diff Detail

Event Timeline

barcharcraz created this revision.Jul 20 2023, 12:59 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 20 2023, 12:59 PM
Herald added a subscriber: Enna1. · View Herald Transcript
  • only do the implib injections on windows, fixes build on linux
  • Use conditionals at configure time to add implib object library
  • add new and delete asan implementations to interface.inc
  • Add dependency between overall asan target and the modified implib
barcharcraz published this revision for review.Aug 3 2023, 11:43 AM
barcharcraz edited the summary of this revision. (Show Details)
Herald added a project: Restricted Project. · View Herald TranscriptAug 3 2023, 11:44 AM
Herald added a subscriber: Restricted Project. · View Herald Transcript
barcharcraz edited reviewers, added: strega-nil; removed: ubsan.
strega-nil requested changes to this revision.Aug 10 2023, 2:22 PM

Some minor nits, mostly.

compiler-rt/lib/asan/asan_win_new_delete.cpp
27

Why is this defined differently between asan_win_new_delete.cpp and asan_win_new_delete_thunk_common.h?

compiler-rt/lib/asan/asan_win_new_delete_thunk_common.h
42

I know this is a nit, but 147 columns is super excessive and means I can't read this block comment on my laptop screen without it wrapping. Can these be split somehow?

72

Style nit: I don't like that this type is snek_case, and this should also likely be an enum class. Additionally, the enumerators are snek_case as well, as opposed to the two standard SCREAMING_SNAKES or kCamelCase from the rest of compiler-rt.

95

I believe this could be const.

Additionally, I absolutely _hate_ this name; I would much prefer something like static const bool is_asan; or static const bool has_user_overload; (with reversed meaning).

This revision now requires changes to proceed.Aug 10 2023, 2:22 PM