[memprof] dump memprof profile when receive deadly signals

[Enna1]

Summary

Currently memprof profile is dumped when program exits (call FinishAndWrite() in ~Allocator) or __memprof_profile_dump is manually called.
For programs that never exit (e.g. server-side application), it will be useful to dump memprof profile when specific signal is received.
This patch installs a signal handler for deadly signals(SIGSEGV, SIGBUS, SIGABRT, SIGILL, SIGTRAP, SIGFPE) like we do in other sanitizers. In the signal handler __memprof_profile_dump is called to dump memprof profile.

[Enna1]

update

[tejohnson]

Do you know how the PGO profile runtime handles deadly signals?

compiler-rt/lib/memprof/memprof_rtl.cpp

55

I see that the other *san deadly signal handlers invoke HandleDeadlySignal - should that be done here too?

compiler-rt/test/memprof/TestCases/memprof_profile_dump_on_abort.cpp

5

Not sure it is important to test both the text and raw dumper, probably ensuring that we get the text dump is sufficient? If you think it is important to test the raw dumper too, add a REQUIRES to ensure it only executes on a LE architecture like x86.

[Enna1]

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

compiler-rt/lib/memprof/memprof_rtl.cpp

55

Call HandleDeadlySignal will write something like following text to memprof profile, so I didn't call HandleDeadlySignal in MemprofOnDeadlySignal.

ERROR: MemProfiler: ABRT on unknown address 0x0bdc003dfd74 (pc 0x00000048487c bp 0x7ffc9e37cc40 sp 0x7ffc9e37cc00 T4102921)
...
MemProfiler can not provide additional info.
SUMMARY: MemProfiler: ABRT /data00/home/xumingjie.enna1/test/memprof-test/instr-and-rt/test_malloc_load_store_raise.cpp:23:3 in main
ABORTING

[tejohnson]

In D134795#3822950, @Enna1 wrote:

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

I don't think we can support continuous mode like PGO, since the information for memprof profiles is located in multiple places and needs to be aggregated on deallocations, and at profile dump time (to get the stack contexts).

I guess why I was asking was to understand how you address this problem when collecting PGO, or why it isn't an issue there. For our long running servers, we use llvm_profile_write_file for PGO and memprof_profile_dump for PGHO. Not opposed to making this change though.

compiler-rt/lib/memprof/memprof_rtl.cpp

55

I was wondering if this is useful to provide info to the user about which signal was received. Not sure if that is important though? If we are dumping the binary format profile, where does this message get printed? Does it try to print it into the binary profile file? If so, I'd agree we don't want this.

[Enna1]

In D134795#3825709, @tejohnson wrote:

In D134795#3822950, @Enna1 wrote:

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

I don't think we can support continuous mode like PGO, since the information for memprof profiles is located in multiple places and needs to be aggregated on deallocations, and at profile dump time (to get the stack contexts).

I guess why I was asking was to understand how you address this problem when collecting PGO, or why it isn't an issue there. For our long running servers, we use llvm_profile_write_file for PGO and memprof_profile_dump for PGHO. Not opposed to making this change though.

Do you mean, for your long running servers, you add function calls to __memprof_profile_dump in the code?
Could please provide more info about how do you deploy PGHO for your long running servers? Thanks!

compiler-rt/lib/memprof/memprof_rtl.cpp

55

If we are dumping the binary format profile, where does this message get printed? Does it try to print it into the binary profile file?

Yes, when dumping the binary format profile, the detailed deadly signal message is still printed to the binary profile file.

I was wondering if this is useful to provide info to the user about which signal was received.

Even if we do not call HandleDeadlySignal in MemprofOnDeadlySignal, we still get "MemProfiler:DEADLYSIGNAL" in stderr, though it does tell which signal it is.

[tejohnson]

In D134795#3825929, @Enna1 wrote:

In D134795#3825709, @tejohnson wrote:

In D134795#3822950, @Enna1 wrote:

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

I don't think we can support continuous mode like PGO, since the information for memprof profiles is located in multiple places and needs to be aggregated on deallocations, and at profile dump time (to get the stack contexts).

I guess why I was asking was to understand how you address this problem when collecting PGO, or why it isn't an issue there. For our long running servers, we use llvm_profile_write_file for PGO and memprof_profile_dump for PGHO. Not opposed to making this change though.

Do you mean, for your long running servers, you add function calls to __memprof_profile_dump in the code?
Could please provide more info about how do you deploy PGHO for your long running servers? Thanks!

Yes, we have the profile dump routines in a handler linked in to the binaries, which can be invoked by the load testing code or manually e.g. via rpc. Do you do something similar to collect regular PGO for your long running servers, or are you using the PGO continuous mode?

compiler-rt/lib/memprof/memprof_rtl.cpp

55

Ok, I agree HandleDeadlySignal can't be called then in this handler.

[Enna1]

In D134795#3826921, @tejohnson wrote:

In D134795#3825929, @Enna1 wrote:

In D134795#3825709, @tejohnson wrote:

In D134795#3822950, @Enna1 wrote:

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

I don't think we can support continuous mode like PGO, since the information for memprof profiles is located in multiple places and needs to be aggregated on deallocations, and at profile dump time (to get the stack contexts).

I guess why I was asking was to understand how you address this problem when collecting PGO, or why it isn't an issue there. For our long running servers, we use llvm_profile_write_file for PGO and memprof_profile_dump for PGHO. Not opposed to making this change though.

Do you mean, for your long running servers, you add function calls to __memprof_profile_dump in the code?
Could please provide more info about how do you deploy PGHO for your long running servers? Thanks!

Yes, we have the profile dump routines in a handler linked in to the binaries, which can be invoked by the load testing code or manually e.g. via rpc. Do you do something similar to collect regular PGO for your long running servers, or are you using the PGO continuous mode?

AFAIK, we do not have PGO for our long running servers. But we deployed AutoFDO for our long running servers. lifengxiang is more familiar with AutoFDO in our data center than me.
We are also very interested in PGHO/memprof. Because memprof now supports collecting memprof profile and annotating memprof metadata on LLVM IR, I want to try it on our long running servers, and collect the information about the heap allocations and accesses. And this is why I want memprof profile is dumped when receive deadly signals :)

[tejohnson]

In D134795#3827084, @Enna1 wrote:

In D134795#3826921, @tejohnson wrote:

In D134795#3825929, @Enna1 wrote:

In D134795#3825709, @tejohnson wrote:

In D134795#3822950, @Enna1 wrote:

Do you know how the PGO profile runtime handles deadly signals?

Sorry, I'm not familiar with PGO profile runtime.
I'm not sure if I found the right place, according to:

• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingFile.c#L995
• https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfiling.h#L42

It seems PGO profile has two modes: continuous mode and default mode:

• Continuous mode continuously synced profile via mmap()
• Default mode write the raw profile at program exit time. use atexit() to register __llvm_profile_write_file

Maybe we can add a continuous mode like PGO?

I don't think we can support continuous mode like PGO, since the information for memprof profiles is located in multiple places and needs to be aggregated on deallocations, and at profile dump time (to get the stack contexts).

I guess why I was asking was to understand how you address this problem when collecting PGO, or why it isn't an issue there. For our long running servers, we use llvm_profile_write_file for PGO and memprof_profile_dump for PGHO. Not opposed to making this change though.

Do you mean, for your long running servers, you add function calls to __memprof_profile_dump in the code?
Could please provide more info about how do you deploy PGHO for your long running servers? Thanks!

Yes, we have the profile dump routines in a handler linked in to the binaries, which can be invoked by the load testing code or manually e.g. via rpc. Do you do something similar to collect regular PGO for your long running servers, or are you using the PGO continuous mode?

AFAIK, we do not have PGO for our long running servers. But we deployed AutoFDO for our long running servers. lifengxiang is more familiar with AutoFDO in our data center than me.
We are also very interested in PGHO/memprof. Because memprof now supports collecting memprof profile and annotating memprof metadata on LLVM IR, I want to try it on our long running servers, and collect the information about the heap allocations and accesses. And this is why I want memprof profile is dumped when receive deadly signals :)

Ok got it. Another question below and please see my comments in the test.

compiler-rt/lib/memprof/memprof_rtl.cpp

55

Actually, do you know how the DEADLYSIGNAL message gets sent to stderr even without HandleDeadlySignal? It looks like this is printed by StartReportDeadlySignal, which is invoked from HandleDeadlySignal.

[Enna1]

address comments

[Enna1]

compiler-rt/lib/memprof/memprof_rtl.cpp

55

Sorry, you can right. Without HandleDeadlySignal, DEADLYSIGNAL message will not send to stderr.

[tejohnson]

compiler-rt/lib/memprof/memprof_rtl.cpp

55

Can you add a call to StartReportDeadlySignal here, and check that we get the message in the test stderr? That's what AsanOnDeadlySignal does.

[Enna1]

update

[tejohnson]

lgtm with one comment request below.

compiler-rt/lib/memprof/memprof_rtl.cpp

54

Sorry, one last thing: please add a comment here about why this is called and not HandleDeadlySignal (so we get the message to stderr but no writing to the binary profile output file).

[Enna1]

update

compiler-rt/lib/memprof/memprof_rtl.cpp

static void MemprofDie() {
   if (common_flags()->print_module_map >= 1)
     DumpProcessMap();
   if (flags()->unmap_shadow_on_exit) {
     if (kHighShadowEnd)
       UnmapOrDie((void *)kLowShadowBeg, kHighShadowEnd - kLowShadowBeg);
   }
 }
 
+static void MemprofOnDeadlySignal(int signo, void *siginfo, void *context) {
+  // We call StartReportDeadlySignal not HandleDeadlySignal so we get the
+  // deadly signal message to stderr but no writing to the profile output file
+  StartReportDeadlySignal();
+  __memprof_profile_dump();
+  Die();
+}
+
 static void CheckUnwind() {
   GET_STACK_TRACE(kStackTraceMax, common_flags()->fast_unwind_on_check);
   stack.Print();
 }
 
 // -------------------------- Globals --------------------- {{{1
 int memprof_inited;
 int memprof_init_done;
static void MemprofInitInternal() {
 
   ReplaceSystemMalloc();
 
   DisableCoreDumperIfNecessary();
 
   InitializeShadowMemory();
 
   TSDInit(PlatformTSDDtor);
+  InstallDeadlySignalHandlers(MemprofOnDeadlySignal);
 
   InitializeAllocator();
 
   // On Linux MemprofThread::ThreadStart() calls malloc() that's why
   // memprof_inited should be set to 1 prior to initializing the threads.
   memprof_inited = 1;
   memprof_init_is_running = false;
 

compiler-rt/test/memprof/TestCases/memprof_profile_dump_on_abort.cpp

+// RUN: %clangxx_memprof %s -o %t
+
+// RUN: %env_memprof_opts=print_text=true:log_path=stdout:handle_abort=1 not %run %t 2>&1 | FileCheck --check-prefix=CHECK-TEXT %s
+
+#include <stdlib.h>
+#include <string.h>
+int main(int argc, char **argv) {
+  char *x = (char *)malloc(10);
+  memset(x, 0, 10);
+  free(x);
+  abort();
+  return 0;
+}
+
+// CHECK-TEXT: MemProfiler:DEADLYSIGNAL
+// CHECK-TEXT: Recorded MIBs (incl. live on exit):
+// CHECK-TEXT: Memory allocation stack id
+// CHECK-TEXT: Stack for id