[compiler-rt] Avoid memintrinsic calls inserted by the compiler

[melver]

Summary

D135716 introduced -ftrivial-auto-var-init=pattern where supported.
Unfortunately this introduces unwanted memset() for large stack arrays,
as shown by the new tests added for asan and msan (tsan already had this
test).

In general, the problem of compiler-inserted memintrinsic calls
(memset/memcpy/memmove) is not new to compiler-rt, and has been a
problem before.

To avoid introducing unwanted memintrinsic calls, we redefine
memintrinsics as __sanitizer_internal_mem* at the assembly level for
most source files automatically (where sanitizer_common_internal_defs.h
is included).

In few cases, redefining a symbol in this way causes issues for
interceptors, namely the memintrinsic interceptor themselves. For such
source files we have to selectively disable the redefinition.

Other alternatives have been considered, but simply do not work well in
the context of compiler-rt:

1. Linker --wrap: this does not work because --wrap only

	   applies to the final link, and would not apply when building
	   sanitizer static libraries.

2. Changing references to memset() via objcopy: this may work,

	   but due to the complexities of the build system, introducing
	   such a post-processing step for the right object files (in
	   particular object files defining memset cannot be touched)
	   seems infeasible.

The chosen solution works well (as shown by the tests). Other libraries
have chosen the same solution where nothing else works (see e.g. glibc's
"symbol-hacks.h").

v4:

• Add interface attribute to __sanitizer_internal_mem* declarations as well, as otherwise some compilers (MSVC) will complain.
• Add SANITIZER_COMMON_NO_REDEFINE_BUILTINS to source files using C++STL, since this could lead to ODR violations (see added comment).

v3:

• Don't use ALIAS() to alias internal_mem*() functions to __sanitizer_internal_mem*() functions, but just define them as ALWAYS_INLINE functions instead. This will work on darwin and windows.

v2:

• Fix ubsan_minimal build where compiler decides to insert memset/memcpy: ubsan_minimal has work without RTSanitizerCommonLibc, therefore do not redefine the builtins.
• Fix definition of internal_mem* functions with compilers that want the aliased function to already be defined before.
• Fix definition of __sanitizer_internal_mem* functions with compilers more pedantic about attribute placement around extern "C".

[melver]

• Add tests for asan and msan.
• Fix new issues for asan and msan.

[dvyukov]

Marco mentioned offline that there is nolibc build, which is broken for asan/msan now. So we need to fix these as well.

Some more complex alternatives:

• add compiler flag to emit only inline initialization
• add compiler flag to emit calls to special __sanitizer_memset functions
• add objcopy pass to sanitizer runtime builds to rename memcpy/memset calls

But these are more complex and elaborate, and this solution looks reasonable as well.

compiler-rt/test/asan/TestCases/Linux/check_memcpy.c

13

Since we use regexp, I would 1 line (callq|jmpq). Or have you copied this form from somewhere else?
This is also x86-specific, so can add REQUIRES directive.

[dvyukov]

Vitaly, do you want to take a look as well?

[melver]

• Simplify CHECK-NOT of tests.
• Add REQUIRES for x86.

[leonardchan]

Hmm if possible I'd prefer not to add these holes into the runtime since they kinda defeat the purpose of adding the auto-var-init flag, and I think adding UNINITIALIZED until a test passes seems kinda fragile.

I think what we'd want is just a way to ensure compiler-emitted memset/cpy calls go through the internal/unsanitized versions of those functions. I think the cheapest way to do this might be through --wrap=memset as a link flag and compiler-rt defines something like:

int memset(...) {
  return internal_memset(...);
}

[melver]

Rework solution and find compromise:

• avoid attribute((uninitialized)) where possible by redefining

	  memset at the asm level.

• use the attribute only where necessary (where redefining memset isn't

	  possible due to interceptors).

[melver]

In D151152#4370264, @leonardchan wrote:

Hmm if possible I'd prefer not to add these holes into the runtime since they kinda defeat the purpose of adding the auto-var-init flag, and I think adding UNINITIALIZED until a test passes seems kinda fragile.

I think what we'd want is just a way to ensure compiler-emitted memset/cpy calls go through the internal/unsanitized versions of those functions. I think the cheapest way to do this might be through --wrap=memset as a link flag and compiler-rt defines something like:

int memset(...) {
  return internal_memset(...);
}

--wrap doesn't work because it only applies to the final link (doesn't work for static lib sanitizers), which means that it would also wrap the to-be-sanitized binary's memset calls, which is wrong. Furthermore, claiming the symbol __wrap_memset globally is precluding others from using --wrap (we already have interceptors, we don't need more interception pain).

I reworked the solution to a compromise. Unfortunately if a memset interceptor exists it's a little difficult to redefine memset though, so in those TUs we still need the attribute.

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h

356 ↗

(On Diff #525767)

if possible, can we rely on sanitizer_redefine_builtins.h and avoid UNINITIALIZED?

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

22

no, it's a problem is even without auto-var-init
if this trick works please make it always

also memcpy

[vitalybuka]

also, isn't D135716 was NOOP as it has a typo?

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

22

no, it's a problem is even without auto-var-init
if this trick works please make it always

also memcpy

maybe even move this into _defs.h to make sure it's everywhere

[vitalybuka]

compiler-rt/CMakeLists.txt

520 ↗

(On Diff #525767)

as I wrote, don't do this
if the trick works, please use it always

time to time w have to debug crashes caused by accidental interceptor from " = {}".

compiler-rt/lib/asan/asan_stack.h

36 ↗

(On Diff #525767)

This is use of UNINITIALIZED as intended.
We don't want in initialize performance critical buffers.

compiler-rt/lib/ubsan/ubsan_diag.cpp

22 ↗

(On Diff #525767)

please reorder in a separate patch

compiler-rt/test/asan/TestCases/Linux/check_memcpy.c

11

Please use stronger --implicit-check-not instead of -NOT

[vitalybuka]

Some more complex alternatives:

• add compiler flag to emit only inline initialization

compiler can emit, but there are passes that replace some common patterns with memset. Probably suppressible with some attributes.
For sanitizer instrumentation we insert tsan_memset, asan_memset... etc

• add compiler flag to emit calls to special __sanitizer_memset functions

late pass to replace memset/memcpy with __internal_ like would be nice. But we never had a good motivation to fix this.

• add objcopy pass to sanitizer runtime builds to rename memcpy/memset calls

But these are more complex and elaborate, and this solution looks reasonable as well.

It would be grate if D151152 works.

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

22

no, it's a problem is even without auto-var-init
if this trick works please make it always

also memcpy

maybe even move this into _defs.h to make sure it's everywhere

Or I guess it's going to conflict with interceptors?

[melver]

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

22

Interceptors make difficult to work everywhere. In general anything that includes sanitizer_common_interceptors.inc and wants the memset/cpy interceptor.

But let me try a little harder. glibc uses this solution, but they say clearly it's a bad hack: https://codebrowser.dev/glibc/glibc/sysdeps/generic/symbol-hacks.h.html

Maybe all it takes is moving interceptors for memset/cpy to the end of the source file (given that asm("f1 = f2") just turns into ".set f1, f2" in asm).

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

22

I guess we can move them into a separate cpp?
Like we do with sanitizer_signal_interceptors.inc

[melver]

• Rework solution to redefine builtins almost everywhere.

[melver]

• Also check memmove in tests.

[melver]

compiler-rt/lib/asan/asan_stack.h

36 ↗

(On Diff #525767)

Moved to separate patch.

compiler-rt/lib/ubsan/ubsan_diag.cpp

22 ↗

(On Diff #525767)

Removed entirely.

[vitalybuka]

Nice!

Not a strong request, but I assume D151552 has high chance of being reverted. To avoid conflicts, would be nice to land them with a couple of days of delay.

compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp

14

For consistently lets prefix sanitizer_common macros with SANITIZER_COMMON_
we don't follow that strictly, but it could be nice:
__NO_SANITIZER_REDEFINE_BUILTINS -> SANITIZER_COMMON_NO_REDEFINE_BUILTINS

[melver]

• Use SANITIZER_COMMON_NO_REDEFINE_BUILTINS for opt-out macro name.
• Do not use asm() hack with old MSVC compilers (not supported there).

[vitalybuka]

FYI @sivachandra

[thakis]

This doesn't build on mac: http://45.33.8.238/macm1/61839/step_4.txt

FAILED: stage2_unix/obj/compiler-rt/lib/sanitizer_common/sources.sanitizer_libc.o 
./bin/clang++ -MMD -MF stage2_unix/obj/compiler-rt/lib/sanitizer_common/sources.sanitizer_libc.o.d -o stage2_unix/obj/compiler-rt/lib/sanitizer_common/sources.sanitizer_libc.o -c ../../compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp  -I../../compiler-rt/lib -mmacos-version-min=10.10 -O3 -fdiagnostics-color -Wall -Wextra -Wno-unused-parameter -Wdelete-non-virtual-dtor -Wstring-conversion -no-canonical-prefixes -Werror=date-time -isysroot ../../sysroot/MacOSX.sdk -Wpoison-system-directories -Wcovered-switch-default -fno-builtin -gline-tables-only -fPIC -funwind-tables -fvisibility=hidden -Werror=thread-safety -Werror=thread-safety-reference -Werror=thread-safety-beta -std=c++17 -fvisibility-inlines-hidden -fno-exceptions -fno-rtti
../../compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:109:5: error: aliases are not supported on darwin
  109 |     ALIAS(__sanitizer_internal_memcpy);
      |     ^
../../compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h:222:34: note: expanded from macro 'ALIAS'
  222 | # define ALIAS(x) __attribute__((alias(SANITIZER_STRINGIFY(x))))
      |                                  ^
../../compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:111:5: error: aliases are not supported on darwin
  111 |     ALIAS(__sanitizer_internal_memmove);
      |     ^
../../compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h:222:34: note: expanded from macro 'ALIAS'
  222 | # define ALIAS(x) __attribute__((alias(SANITIZER_STRINGIFY(x))))
      |                                  ^
../../compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:113:5: error: aliases are not supported on darwin
  113 |     ALIAS(__sanitizer_internal_memset);
      |     ^
../../compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h:222:34: note: expanded from macro 'ALIAS'
  222 | # define ALIAS(x) __attribute__((alias(SANITIZER_STRINGIFY(x))))
      |                                  ^
3 errors generated.

Please take a look, and revert for now if it takes a while to fix.

[vitalybuka]

Windows https://lab.llvm.org/buildbot/#/builders/127/builds/49136/steps/4/logs/stdio

[thakis]

Thanks for the revert!

If it's not too much trouble, if you could fold https://github.com/llvm/llvm-project/commit/3825910c7316cf62549bd31c503c48e7526adcc2 into this when relanding, that'd be great.

[thakis]

In D151152#4384558, @vitalybuka wrote:

Windows https://lab.llvm.org/buildbot/#/builders/127/builds/49136/steps/4/logs/stdio

I'm seeing a different error (using lld-link) on win: http://45.33.8.238/win/79321/step_4.txt

lld-link: error: stage2_win_x64/obj/compiler-rt/lib/asan/asan_shared_library.asan_activation.obj: memcpy should not refer to special section 0
[2124/2134] CXX obj/clang/unittests/Tooling/ToolingTests.SourceCodeTest.obj

[melver]

• Fix darwin and windows by defining internal_X() aliases as ALWAYS_INLINE functions instead.

[melver]

• Try to fix Windows builds.
• Fix ODR violations.

[fmayer]

Seems like the check_memcpy.c is broken: https://lab.llvm.org/buildbot/#/builders/37/builds/22633/steps/2/logs/stdio

[melver]

In D151152#4400149, @fmayer wrote:

Seems like the check_memcpy.c is broken: https://lab.llvm.org/buildbot/#/builders/37/builds/22633/steps/2/logs/stdio

Apparently this is expected because a similar exception exists for the tsan check_memcpy.c test: https://github.com/llvm/llvm-zorg/pull/38/files

[mstorsjo]

This broke linking the sanitizers for mingw targets. Building now fails with errors like these:

[330/350] Linking CXX shared library lib/windows/libclang_rt.asan_dynamic-x86_64.dll
FAILED: lib/windows/libclang_rt.asan_dynamic-x86_64.dll lib/windows/libclang_rt.asan_dynamic-x86_64.dll.a
[...]
ld.lld: error: lib/interception/CMakeFiles/RTInterception.x86_64.dir/interception_linux.cpp.obj: memcpy should not refer to special section 0
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I see that this also was happening with clang-cl configurations, which was then fixed in caa2c1bacbd76c017ebbb4fd13861f0f66770299. I'll go ahead and push a change that changes _MSC_VER into _WIN32 there, to keep the same behaviour in mingw configs as in clang-cl configs.

[melver]

In D151152#4402854, @mstorsjo wrote:

This broke linking the sanitizers for mingw targets. Building now fails with errors like these:

[330/350] Linking CXX shared library lib/windows/libclang_rt.asan_dynamic-x86_64.dll
FAILED: lib/windows/libclang_rt.asan_dynamic-x86_64.dll lib/windows/libclang_rt.asan_dynamic-x86_64.dll.a
[...]
ld.lld: error: lib/interception/CMakeFiles/RTInterception.x86_64.dir/interception_linux.cpp.obj: memcpy should not refer to special section 0
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I see that this also was happening with clang-cl configurations, which was then fixed in caa2c1bacbd76c017ebbb4fd13861f0f66770299. I'll go ahead and push a change that changes _MSC_VER into _WIN32 there, to keep the same behaviour in mingw configs as in clang-cl configs.

That sounds reasonable, thanks.

Getting it to work on Windows (when built with Clang) would be nice, because ftrivial-auto-var-init is also available on Windows (AFAIK). But I've deferred for now, because it seems rather tricky (likely needs some changes to the inline asm?).

[mstorsjo]

In D151152#4402874, @melver wrote:

Getting it to work on Windows (when built with Clang) would be nice, because ftrivial-auto-var-init is also available on Windows (AFAIK). But I've deferred for now, because it seems rather tricky (likely needs some changes to the inline asm?).

Yes, it seems like those kinds of symbol aliases don't quite work as expected/desired on COFF.

[jakubjelinek]

Note, this doesn't really work on SPARC Solaris when using the Solaris assembler, see https://gcc.gnu.org/PR112563 for details.

c
extern "C" void *memcpy (void *, const void *, decltype (sizeof 0)) __asm ("__sanitizer_internal_memcpy");
extern "C" void *memmove (void *, const void *, decltype (sizeof 0)) __asm ("__sanitizer_internal_memmove");
extern "C" void *memset (void *, int, decltype (sizeof 0)) __asm ("__sanitizer_internal_memset");

would work if -fno-builtin flag wouldn't be used, but for some reason it is.

c
extern "C" {
  static void *memcpy (void *, const void *, decltype (sizeof 0)) __attribute__ ((weakref ("__sanitizer_internal_memcpy")));
  static void *memmove (void *, const void *, decltype (sizeof 0)) __attribute__ ((weakref ("__sanitizer_internal_memmove")));
  static void *memset (void *, int, decltype (sizeof 0)) __attribute__ ((weakref ("__sanitizer_internal_memset")));
}

seems to work somehow, but explicit calls to the mem* or __builtin_mem* functions seem to be at least on SPARC less optimized (indirect calls rather than direct). But I bet compiler-rt doesn't really call the functions directly but calls __sanitizer_internal_mem* instead.

compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp

 //===-- asan_interceptors_memintrinsics.cpp -------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===---------------------------------------------------------------------===//
 //
 // This file is a part of AddressSanitizer, an address sanity checker.
 //
 // ASan versions of memcpy, memmove, and memset.
 //===---------------------------------------------------------------------===//
 
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "asan_interceptors_memintrinsics.h"
 
 #include "asan_interceptors.h"
 #include "asan_report.h"
 #include "asan_stack.h"
 #include "asan_suppressions.h"
 
 using namespace __asan;

compiler-rt/lib/hwasan/hwasan_interceptors.cpp

 //===-- hwasan_interceptors.cpp -------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of HWAddressSanitizer.
 //
 // Interceptors for standard library functions.
 //
 // FIXME: move as many interceptors as possible into
 // sanitizer_common/sanitizer_common_interceptors.h
 //===----------------------------------------------------------------------===//
 
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "hwasan.h"
 #include "hwasan_allocator.h"
 #include "hwasan_checks.h"
 #include "hwasan_platform_interceptors.h"
 #include "hwasan_thread.h"
 #include "hwasan_thread_list.h"
 #include "interception/interception.h"
 #include "sanitizer_common/sanitizer_errno.h"

compiler-rt/lib/interception/tests/CMakeLists.txt

 
 set(INTERCEPTION_TEST_CFLAGS_COMMON
   ${COMPILER_RT_UNITTEST_CFLAGS}
   ${COMPILER_RT_GTEST_CFLAGS}
   ${SANITIZER_TEST_CXX_CFLAGS}
   -I${COMPILER_RT_SOURCE_DIR}/include
   -I${COMPILER_RT_SOURCE_DIR}/lib
   -I${COMPILER_RT_SOURCE_DIR}/lib/interception
+  -DSANITIZER_COMMON_NO_REDEFINE_BUILTINS
   -fno-rtti
   -O2
   -Werror=sign-compare)
 
 set(INTERCEPTION_TEST_LINK_FLAGS_COMMON
   ${COMPILER_RT_UNITTEST_LINK_FLAGS}
   ${COMPILER_RT_UNWINDER_LINK_LIBS}
   ${SANITIZER_TEST_CXX_LIBRARIES})

compiler-rt/lib/memprof/memprof_interceptors_memintrinsics.cpp

 //===-- memprof_interceptors_memintrinsics.cpp ---------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===---------------------------------------------------------------------===//
 //
 // This file is a part of MemProfiler, a memory profiler.
 //
 // MemProf versions of memcpy, memmove, and memset.
 //===---------------------------------------------------------------------===//
 
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "memprof_interceptors_memintrinsics.h"
 
 #include "memprof_interceptors.h"
 #include "memprof_stack.h"
 
 using namespace __memprof;
 
 // memcpy is called during __memprof_init() from the internals of printf(...).

compiler-rt/lib/msan/msan_interceptors.cpp

 //===-- msan_interceptors.cpp ---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of MemorySanitizer.
 //
 // Interceptors for standard library functions.
 //
 // FIXME: move as many interceptors as possible into
 // sanitizer_common/sanitizer_common_interceptors.h
 //===----------------------------------------------------------------------===//
 
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "interception/interception.h"
 #include "msan.h"
 #include "msan_chained_origin_depot.h"
 #include "msan_origin.h"
 #include "msan_poisoning.h"
 #include "msan_report.h"
 #include "msan_thread.h"
 #include "sanitizer_common/sanitizer_allocator.h"

compiler-rt/lib/sanitizer_common/CMakeLists.txt

set(SANITIZER_IMPL_HEADERS
   sanitizer_platform_interceptors.h
   sanitizer_platform_limits_netbsd.h
   sanitizer_platform_limits_posix.h
   sanitizer_platform_limits_solaris.h
   sanitizer_posix.h
   sanitizer_procmaps.h
   sanitizer_ptrauth.h
   sanitizer_quarantine.h
+  sanitizer_redefine_builtins.h
   sanitizer_report_decorator.h
   sanitizer_ring_buffer.h
   sanitizer_signal_interceptors.inc
   sanitizer_stack_store.h
   sanitizer_stackdepot.h
   sanitizer_stackdepotbase.h
   sanitizer_stacktrace.h
   sanitizer_stacktrace_printer.h

compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc

 //===-- sanitizer_common_interceptors_memintrinsics.inc ---------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Memintrinsic function interceptors for tools like AddressSanitizer,
 // ThreadSanitizer, MemorySanitizer, etc.
 //
+// These interceptors are part of the common interceptors, but separated out so
+// that implementations may add them, if necessary, to a separate source file
+// that should define SANITIZER_COMMON_NO_REDEFINE_BUILTINS at the top.
+//
 // This file should be included into the tool's memintrinsic interceptor file,
 // which has to define its own macros:
 //   COMMON_INTERCEPTOR_ENTER
 //   COMMON_INTERCEPTOR_READ_RANGE
 //   COMMON_INTERCEPTOR_WRITE_RANGE
 //   COMMON_INTERCEPTOR_MEMSET_IMPL
 //   COMMON_INTERCEPTOR_MEMMOVE_IMPL
 //   COMMON_INTERCEPTOR_MEMCPY_IMPL
 //   COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED
 //===----------------------------------------------------------------------===//
 
+#ifdef SANITIZER_REDEFINE_BUILTINS_H
+#error "Define SANITIZER_COMMON_NO_REDEFINE_BUILTINS in .cpp file"
+#endif
+
 #include "interception/interception.h"
 #include "sanitizer_platform_interceptors.h"
 
 // Platform-specific options.
 #if SANITIZER_APPLE
 #define PLATFORM_HAS_DIFFERENT_MEMCPY_AND_MEMMOVE 0
 #elif SANITIZER_WINDOWS64
 #define PLATFORM_HAS_DIFFERENT_MEMCPY_AND_MEMMOVE 0

compiler-rt/lib/sanitizer_common/sanitizer_common_interface.inc

 INTERFACE_FUNCTION(__sanitizer_get_heap_size)
 INTERFACE_FUNCTION(__sanitizer_get_ownership)
 INTERFACE_FUNCTION(__sanitizer_get_unmapped_bytes)
 INTERFACE_FUNCTION(__sanitizer_install_malloc_and_free_hooks)
 INTERFACE_FUNCTION(__sanitizer_purge_allocator)
 INTERFACE_FUNCTION(__sanitizer_print_memory_profile)
 INTERFACE_WEAK_FUNCTION(__sanitizer_free_hook)
 INTERFACE_WEAK_FUNCTION(__sanitizer_malloc_hook)
+// Memintrinsic functions.
+INTERFACE_FUNCTION(__sanitizer_internal_memcpy)
+INTERFACE_FUNCTION(__sanitizer_internal_memmove)
+INTERFACE_FUNCTION(__sanitizer_internal_memset)

compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h

 //===-- sanitizer_internal_defs.h -------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is shared between AddressSanitizer and ThreadSanitizer.
 // It contains macro used in run-time libraries code.
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_DEFS_H
 #define SANITIZER_DEFS_H
 
 #include "sanitizer_platform.h"
+#include "sanitizer_redefine_builtins.h"
 
 #ifndef SANITIZER_DEBUG
 # define SANITIZER_DEBUG 0
 #endif
 
 #define SANITIZER_STRINGIFY_(S) #S
 #define SANITIZER_STRINGIFY(S) SANITIZER_STRINGIFY_(S)
 

compiler-rt/lib/sanitizer_common/sanitizer_libc.h

 // ----------- ATTENTION -------------
 // This header should NOT include any other headers from sanitizer runtime.
 #include "sanitizer_internal_defs.h"
 
 namespace __sanitizer {
 
 // internal_X() is a custom implementation of X() for use in RTL.
 
+extern "C" {
+// These are used as builtin replacements; see sanitizer_redefine_builtins.h.
+// In normal runtime code, use the __sanitizer::internal_X() aliases instead.
+void *__sanitizer_internal_memcpy(void *dest, const void *src, uptr n);
+void *__sanitizer_internal_memmove(void *dest, const void *src, uptr n);
+void *__sanitizer_internal_memset(void *s, int c, uptr n);
+}  // extern "C"
+
 // String functions
 s64 internal_atoll(const char *nptr);
 void *internal_memchr(const void *s, int c, uptr n);
 void *internal_memrchr(const void *s, int c, uptr n);
 int internal_memcmp(const void* s1, const void* s2, uptr n);
-void *internal_memcpy(void *dest, const void *src, uptr n);
-void *internal_memmove(void *dest, const void *src, uptr n);
+ALWAYS_INLINE void *internal_memcpy(void *dest, const void *src, uptr n) {
+  return __sanitizer_internal_memcpy(dest, src, n);
+}
+ALWAYS_INLINE void *internal_memmove(void *dest, const void *src, uptr n) {
+  return __sanitizer_internal_memmove(dest, src, n);
+}
 // Should not be used in performance-critical places.
-void *internal_memset(void *s, int c, uptr n);
+ALWAYS_INLINE void *internal_memset(void *s, int c, uptr n) {
+  return __sanitizer_internal_memset(s, c, n);
+}
 char* internal_strchr(const char *s, int c);
 char *internal_strchrnul(const char *s, int c);
 int internal_strcmp(const char *s1, const char *s2);
 uptr internal_strcspn(const char *s, const char *reject);
 char *internal_strdup(const char *s);
 uptr internal_strlen(const char *s);
 uptr internal_strlcat(char *dst, const char *src, uptr maxlen);
 char *internal_strncat(char *dst, const char *src, uptr n);

compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp

 //===-- sanitizer_libc.cpp ------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is shared between AddressSanitizer and ThreadSanitizer
 // run-time libraries. See sanitizer_libc.h for details.
 //===----------------------------------------------------------------------===//
 
+// Do not redefine builtins; this file is defining the builtin replacements.
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "sanitizer_allocator_internal.h"
 #include "sanitizer_common.h"
 #include "sanitizer_libc.h"
 
 namespace __sanitizer {
 
 s64 internal_atoll(const char *nptr) {
   return internal_simple_strtoll(nptr, nullptr, 10);
int internal_memcmp(const void* s1, const void* s2, uptr n) {
   const char *t1 = (const char *)s1;
   const char *t2 = (const char *)s2;
   for (uptr i = 0; i < n; ++i, ++t1, ++t2)
     if (*t1 != *t2)
       return *t1 < *t2 ? -1 : 1;
   return 0;
 }
 
-void *internal_memcpy(void *dest, const void *src, uptr n) {
+extern "C" {
+SANITIZER_INTERFACE_ATTRIBUTE void *__sanitizer_internal_memcpy(void *dest,
+                                                                const void *src,
+                                                                uptr n) {
   char *d = (char*)dest;
   const char *s = (const char *)src;
   for (uptr i = 0; i < n; ++i)
     d[i] = s[i];
   return dest;
 }
 
-void *internal_memmove(void *dest, const void *src, uptr n) {
+SANITIZER_INTERFACE_ATTRIBUTE void *__sanitizer_internal_memmove(
+    void *dest, const void *src, uptr n) {
   char *d = (char*)dest;
   const char *s = (const char *)src;
   sptr i, signed_n = (sptr)n;
   CHECK_GE(signed_n, 0);
   if (d < s) {
     for (i = 0; i < signed_n; ++i)
       d[i] = s[i];
   } else {
     if (d > s && signed_n > 0) {
       for (i = signed_n - 1; i >= 0; --i) {
         d[i] = s[i];
       }
     }
   }
   return dest;
 }
 
-void *internal_memset(void* s, int c, uptr n) {
+SANITIZER_INTERFACE_ATTRIBUTE void *__sanitizer_internal_memset(void *s, int c,
+                                                                uptr n) {
   // Optimize for the most performance-critical case:
   if ((reinterpret_cast<uptr>(s) % 16) == 0 && (n % 16) == 0) {
     u64 *p = reinterpret_cast<u64*>(s);
     u64 *e = p + n / 8;
     u64 v = c;
     v |= v << 8;
     v |= v << 16;
     v |= v << 32;
     for (; p < e; p += 2)
       p[0] = p[1] = v;
     return s;
   }
   // The next line prevents Clang from making a call to memset() instead of the
   // loop below.
   // FIXME: building the runtime with -ffreestanding is a better idea. However
   // there currently are linktime problems due to PR12396.
   char volatile *t = (char*)s;
   for (uptr i = 0; i < n; ++i, ++t) {
     *t = c;
   }
   return s;
 }
+}  // extern "C"
 
 uptr internal_strcspn(const char *s, const char *reject) {
   uptr i;
   for (i = 0; s[i]; i++) {
     if (internal_strchr(reject, s[i]))
       return i;
   }
   return i;

compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h

+//===-- sanitizer_redefine_builtins.h ---------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Redefine builtin functions to use internal versions. This is needed where
+// compiler optimizations end up producing unwanted libcalls!
+//
+//===----------------------------------------------------------------------===//
+#ifndef SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+#ifndef SANITIZER_REDEFINE_BUILTINS_H
+#define SANITIZER_REDEFINE_BUILTINS_H
+
+// The asm hack only works with GCC and Clang.
+#if !defined(_MSC_VER) || defined(__clang__)
+
+asm("memcpy = __sanitizer_internal_memcpy");
+asm("memmove = __sanitizer_internal_memmove");
+asm("memset = __sanitizer_internal_memset");
+
+#endif  // !_MSC_VER || __clang__
+
+#endif  // SANITIZER_REDEFINE_BUILTINS_H
+#endif  // SANITIZER_COMMON_NO_REDEFINE_BUILTINS

compiler-rt/lib/tsan/rtl/tsan_interceptors_memintrinsics.cpp

 //===-- tsan_interceptors_posix.cpp ---------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of ThreadSanitizer (TSan), a race detector.
 //
 //===----------------------------------------------------------------------===//
 
+#define SANITIZER_COMMON_NO_REDEFINE_BUILTINS
+
 #include "tsan_interceptors.h"
 #include "tsan_interface.h"
 
 using namespace __tsan;
 
 #include "sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc"
 
 extern "C" {

compiler-rt/lib/ubsan_minimal/CMakeLists.txt

 # Build for the minimal undefined behavior sanitizer runtime support library.
 
 set(UBSAN_MINIMAL_SOURCES
   ubsan_minimal_handlers.cpp
   )
 
 include_directories(..)
 
-set(UBSAN_CFLAGS ${SANITIZER_COMMON_CFLAGS})
+set(UBSAN_CFLAGS
+  ${SANITIZER_COMMON_CFLAGS}
+  -DSANITIZER_COMMON_NO_REDEFINE_BUILTINS)
 append_rtti_flag(OFF UBSAN_CFLAGS)
 
 set(UBSAN_LINK_FLAGS ${SANITIZER_COMMON_LINK_FLAGS})
 
 set(UBSAN_DYNAMIC_LIBS ${SANITIZER_COMMON_LINK_LIBS})
 
 add_compiler_rt_component(ubsan-minimal)
 

compiler-rt/test/asan/TestCases/Linux/check_memcpy.c

+// Verify runtime doesn't contain compiler-emitted memcpy/memmove calls.
+//
+// REQUIRES: shared_unwind, x86_64-target-arch
+
+// RUN: %clang_asan -O1 %s -o %t
+// RUN: llvm-objdump -d -l %t | FileCheck --implicit-check-not="{{(callq|jmpq) .*<(__interceptor_.*)?mem(cpy|set|move)>}}" %s
+
+int main() { return 0; }

compiler-rt/test/msan/Linux/check_memcpy.c

+// Verify runtime doesn't contain compiler-emitted memcpy/memmove calls.
+//
+// REQUIRES: shared_unwind, x86_64-target-arch
+
+// RUN: %clang_msan -O1 %s -o %t
+// RUN: llvm-objdump -d -l %t | FileCheck --implicit-check-not="{{(callq|jmpq) .*<(__interceptor_.*)?mem(cpy|set|move)>}}" %s
+
+int main() { return 0; }

compiler-rt/test/tsan/Linux/check_memcpy.c

 // Test that verifies TSan runtime doesn't contain compiler-emitted
 // memcpy/memmove calls. It builds the binary with TSan and check's
 // its objdump.
 
 // This could fail if using a static libunwind because that static libunwind
 // could be uninstrumented and contain memcpy/memmove calls not intercepted by
 // tsan.
-// REQUIRES: shared_unwind
+// REQUIRES: shared_unwind, x86_64-target-arch
 
 // RUN: %clang_tsan -O1 %s -o %t
-// RUN: llvm-objdump -d -l %t | FileCheck %s
-
-int main() {
-  return 0;
-}
-
-// CHECK-NOT: callq {{.*<(__interceptor_)?mem(cpy|set)>}}
-// tail calls:
-// CHECK-NOT: jmpq {{.*<(__interceptor_)?mem(cpy|set)>}}
+// RUN: llvm-objdump -d -l %t | FileCheck --implicit-check-not="{{(callq|jmpq) .*<(__interceptor_.*)?mem(cpy|set|move)>}}" %s
 
+int main() { return 0; }

llvm/utils/gn/secondary/compiler-rt/lib/sanitizer_common/BUILD.gn

sources = [
     "sanitizer_procmaps_bsd.cpp",
     "sanitizer_procmaps_common.cpp",
     "sanitizer_procmaps_fuchsia.cpp",
     "sanitizer_procmaps_linux.cpp",
     "sanitizer_procmaps_mac.cpp",
     "sanitizer_procmaps_solaris.cpp",
     "sanitizer_ptrauth.h",
     "sanitizer_quarantine.h",
+    "sanitizer_redefine_builtins.h",
     "sanitizer_report_decorator.h",
     "sanitizer_ring_buffer.h",
     "sanitizer_solaris.cpp",
     "sanitizer_stack_store.cpp",
     "sanitizer_stack_store.h",
     "sanitizer_stackdepot.cpp",
     "sanitizer_stackdepot.h",
     "sanitizer_stackdepotbase.h",