[sanitizer] Don't run malloc hooks for stacktraces

[vitalybuka]

Summary

Usually when we generated stacktraces the process is in error state, so
running hooks may crash the process and prevent meaningfull error report.

Symbolizer, unwinder and pthread are potential source of mallocs.

https://b.corp.google.com/issues/228110771

[kda]

compiler-rt/test/msan/Linux/malloc_hooks.cpp

25

While this shows that the hooks are not called, I'm not sure that it demonstrates that they would usually be called. Perhaps call a non-symbolizer method and show them increasing?

[kstoimenov]

compiler-rt/test/msan/Linux/malloc_hooks.cpp

8

Make this static?

21

Should we allocate memory to make sure that 'hooks' is non-zero? Also maybe add assert to check that is it is > 0.?

[vitalybuka]

extend on other sanitizers

[vitalybuka]

compiler-rt/test/msan/Linux/malloc_hooks.cpp

21

No, we have multiple tests which check that callbacks works..

25

No, we have multiple tests which check that callbacks works..
E.g. llvm-project/compiler-rt/test/sanitizer_common/TestCases/malloc_hook.cpp

[kstoimenov]

compiler-rt/lib/sanitizer_common/sanitizer_common.cpp

313 ↗

(On Diff #422420)

Maybe this can be a static class member of ScopedDisableMallocHooks? You could add a static ScopedDisableMallocHooks::DisableMallocHooks accessor. Your call.

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_common.cpp

313 ↗

(On Diff #422420)

It makes it visible for linker

[ahatanak]

It looks like this is causing a test to fail:
https://green.lab.llvm.org/green/job/clang-san-iossim/5229/consoleFull#711521816a1ca8a51-895e-46c6-af87-ce24fa4cd561

[vitalybuka]

Reverting

[vitalybuka]

rebase

[vitalybuka]

I probably going to abandon this, as I don't see good solution for all platforms.
Seems having sanitizer-unclean hooks is not good idea anyway.

compiler-rt/lib/msan/msan.h

 
 void MsanTSDInit(void (*destructor)(void *tsd));
 void *MsanTSDGet();
 void MsanTSDSet(void *tsd);
 void MsanTSDDtor(void *tsd);
 
 }  // namespace __msan
 
-#define MSAN_MALLOC_HOOK(ptr, size)       \
-  do {                                    \
-    if (&__sanitizer_malloc_hook) {       \
-      UnpoisonParam(2);                   \
-      __sanitizer_malloc_hook(ptr, size); \
-    }                                     \
-    RunMallocHooks(ptr, size);            \
+#define MSAN_MALLOC_HOOK(ptr, size)                                            \
+  do {                                                                         \
+    if (!IsInSymbolizer()) {                                                   \
+      if (&__sanitizer_malloc_hook) {                                          \
+        UnpoisonParam(2);                                                      \
+        __sanitizer_malloc_hook(ptr, size);                                    \
+      }                                                                        \
+      RunMallocHooks(ptr, size);                                               \
+    }                                                                          \
   } while (false)
-#define MSAN_FREE_HOOK(ptr)       \
-  do {                            \
-    if (&__sanitizer_free_hook) { \
-      UnpoisonParam(1);           \
-      __sanitizer_free_hook(ptr); \
-    }                             \
-    RunFreeHooks(ptr);            \
+#define MSAN_FREE_HOOK(ptr)                                                    \
+  do {                                                                         \
+    if (!IsInSymbolizer()) {                                                   \
+      if (&__sanitizer_free_hook) {                                            \
+        UnpoisonParam(1);                                                      \
+        __sanitizer_free_hook(ptr);                                            \
+      }                                                                        \
+      RunFreeHooks(ptr);                                                       \
+    }                                                                          \
   } while (false)
 
 #endif  // MSAN_H

compiler-rt/test/msan/Linux/malloc_hooks.cpp

+// RUN: %clangxx_msan -O0 -g %s -o %t && %run %t
+
+#include <assert.h>
+#include <sanitizer/allocator_interface.h>
+#include <sanitizer/coverage_interface.h>
+#include <stdlib.h>
+
+int hooks;
+extern "C" {
+
+void __sanitizer_malloc_hook(const volatile void *ptr, size_t sz) { ++hooks; }
+
+void __sanitizer_free_hook(const volatile void *ptr) { ++hooks; }
+
+} // extern "C"
+
+void MallocHook(const volatile void *ptr, size_t sz) { ++hooks; }
+void FreeHook(const volatile void *ptr) { ++hooks; }
+
+int main() {
+  int before;
+
+  before = hooks;
+  __sanitizer_print_stack_trace();
+  assert(before == hooks);
+
+  __sanitizer_install_malloc_and_free_hooks(MallocHook, FreeHook);
+  before = hooks;
+  __sanitizer_print_stack_trace();
+  assert(before == hooks);
+
+  return 0;
+}