Dump current PrettyStackTrace on SIGINFO (Ctrl-T)

[jordan_rose]

Summary

Support SIGINFO (and SIGUSR1 for POSIX purposes) to tell what long-running jobs are doing, as inspired by BSD tools (including on macOS), by dumping the current PrettyStackTrace.

This adds a new kind of signal handler for non-fatal "info" signals, similar to the "interrupt" handler that already exists for SIGINT (Ctrl-C). It then uses that handler to update a "generation count" managed by the PrettyStackTrace infrastructure, which is then checked whenever a PrettyStackTraceEntry is pushed or popped on each thread. If the generation has changed—i.e. if the user has pressed Ctrl-T—the stack trace is dumped, though unfortunately it can't include the deepest entry because that one is currently being constructed/destructed.

Out of scope:

• Multiple SIGINFO callbacks. I don't know how interesting this is anyway, given that they're still bound by what a signal handler can do.

• Windows support. Windows console programs don't seem to have a SIGINFO equivalent anyway, so when someone figures out the right thing, they can add equivalent support.

• Choosing whether to include SIGUSR1. I did this just because it was mentioned when I was looking up things about SIGINFO, but SIGINFO is the one I care about, so if people tell me they're not sure about SIGUSR1 I'll just remove it.

• Figuring out how to print the actual current PrettyStackTrace, instead of the parent one, despite the construction/destruction problem. The simplest answer I can think of requires changing all of the PrettyStackTraceEntry instances in existence, which probably isn't going to fly for something that's ultimately a convenience feature. Someone else can figure it out.

[jfb]

lib/Support/PrettyStackTrace.cpp

51

Why not use seq_cst everywhere? It's not in any critical path, the the small extra cost isn't relevant. I'm also not sure that you really need volatile here: AddSignalHandler also synchronizes so a sufficiently smart optimizer won't be able to prove that anything can be aggregated.

52

Technically, in C++11, you need to initialize std::atomic with ATOMIC_VAR_INIT.

lib/Support/Unix/Signals.inc

231

Period at end of sentence

296–297

I'd rather have enum class SignalKind { IsKill, IsInfo }; instead of a bool parameter.

lib/Support/Windows/Signals.inc

560

wai?

[jordan_rose]

lib/Support/PrettyStackTrace.cpp

51

The store could be seq_cst, but I wanted the load to be unsychronized because I never want people thinking that PrettyStackTraces are expensive. It's also not AddSignalHandler we're doing here, but a bare setting of the signal function, which is the classic use for volatile.

lib/Support/Windows/Signals.inc

560

Copying my justification about "there doesn't seem to be a SIGINFO equivalent on Windows" seemed…not great, since it might dissuade people from correcting Past Me. I can add it if you think it's an improvement, though.

[jordan_rose]

Made it a per-thread opt-in, leaving the potential for it to be useful in multi-threaded programs. (It's off by default.) Also added a SaveAndRestore of errno for SIGINFO handlers, though this one doesn't need it.

[jfb]

include/llvm/Support/PrettyStackTrace.h

38

I don't understand what ShouldEnable means.

lib/Support/PrettyStackTrace.cpp

88

Can you remove the function name from the comment while you're here?

[jordan_rose]

include/llvm/Support/PrettyStackTrace.h

38

This allows you to turn it back off if you want. Got a better name for the parameter?

[jordan_rose]

include/llvm/Support/PrettyStackTrace.h

38

I changed the summary comment to "Enables (or disables)". That's probably good enough.

[jordan_rose]

Merged in rL365911.

[youling257]

I have build problem with clang.
can you help me fix it?

In file included from external/llvm/lib/Support/Signals.cpp:220:
external/llvm/lib/Support/Unix/Signals.inc:388:23: error: no matching constructor for initialization of 'SaveAndRestore<int>'

SaveAndRestore<int> SaveErrnoDuringASignalHandler(errno);
                    ^                             ~~~~~

external/llvm/lib/Support/../../include/llvm/Support/SaveAndRestore.h:21:30: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'volatile int' to 'const llvm::SaveAndRestore<int>' for 1st argument
template <typename T> struct SaveAndRestore {

^

external/llvm/lib/Support/../../include/llvm/Support/SaveAndRestore.h:22:3: note: candidate constructor not viable: 1st argument ('volatile int') would lose volatile qualifier

SaveAndRestore(T &X) : X(X), OldValue(X) {}
^

external/llvm/lib/Support/../../include/llvm/Support/SaveAndRestore.h:23:3: note: candidate constructor not viable: requires 2 arguments, but 1 was provided

SaveAndRestore(T &X, const T &NewValue) : X(X), OldValue(X) {
^

include/llvm/Support/PrettyStackTrace.h

 #define LLVM_SUPPORT_PRETTYSTACKTRACE_H
 
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/Compiler.h"
 
 namespace llvm {
   class raw_ostream;
 
+  /// Enables dumping a "pretty" stack trace when the program crashes.
+  ///
+  /// \see PrettyStackTraceEntry
   void EnablePrettyStackTrace();
 
+  /// Enables dumping a "pretty" stack trace when the user sends SIGINFO or
+  /// SIGUSR1 to the current process.
+  ///
+  /// This is a per-thread decision so that a program can choose to print
+  /// stack traces only on a primary thread, or on all threads that use
+  /// PrettyStackTraceEntry.
+  ///
+  /// \see EnablePrettyStackTrace
+  /// \see PrettyStackTraceEntry
+  void EnablePrettyStackTraceOnSigInfoForThisThread(bool ShouldEnable = true);
+
   /// PrettyStackTraceEntry - This class is used to represent a frame of the
   /// "pretty" stack trace that is dumped when a program crashes. You can define
   /// subclasses of this and declare them on the program stack: when they are
   /// constructed and destructed, they will add their symbolic frames to a
   /// virtual stack trace.  This gets dumped out if the program crashes.
   class PrettyStackTraceEntry {
     friend PrettyStackTraceEntry *ReverseStackTrace(PrettyStackTraceEntry *);
 

include/llvm/Support/Signals.h

namespace sys {
   /// Add a function to be called when an abort/kill signal is delivered to the
   /// process. The handler can have a cookie passed to it to identify what
   /// instance of the handler it is.
   void AddSignalHandler(SignalHandlerCallback FnPtr, void *Cookie);
 
   /// This function registers a function to be called when the user "interrupts"
   /// the program (typically by pressing ctrl-c).  When the user interrupts the
   /// program, the specified interrupt function is called instead of the program
-  /// being killed, and the interrupt function automatically disabled.  Note
-  /// that interrupt functions are not allowed to call any non-reentrant
+  /// being killed, and the interrupt function automatically disabled.
+  ///
+  /// Note that interrupt functions are not allowed to call any non-reentrant
   /// functions.  An null interrupt function pointer disables the current
   /// installed function.  Note also that the handler may be executed on a
   /// different thread on some platforms.
-  /// Register a function to be called when ctrl-c is pressed.
   void SetInterruptFunction(void (*IF)());
+
+  /// Registers a function to be called when an "info" signal is delivered to
+  /// the process.
+  ///
+  /// On POSIX systems, this will be SIGUSR1; on systems that have it, SIGINFO
+  /// will also be used (typically ctrl-t).
+  ///
+  /// Note that signal handlers are not allowed to call any non-reentrant
+  /// functions.  An null function pointer disables the current installed
+  /// function.  Note also that the handler may be executed on a different
+  /// thread on some platforms.
+  void SetInfoSignalFunction(void (*Handler)());
 } // End sys namespace
 } // End llvm namespace
 
 #endif

lib/Support/PrettyStackTrace.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm-c/ErrorHandling.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Config/config.h" // Get autoconf configuration settings
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/SaveAndRestore.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/Watchdog.h"
 #include "llvm/Support/raw_ostream.h"
 
 #include <cstdarg>
 #include <cstdio>
 #include <tuple>
 

 #if ENABLE_BACKTRACES
 
 // We need a thread local pointer to manage the stack of our stack trace
 // objects, but we *really* cannot tolerate destructors running and do not want
 // to pay any overhead of synchronizing. As a consequence, we use a raw
 // thread-local variable.
 static LLVM_THREAD_LOCAL PrettyStackTraceEntry *PrettyStackTraceHead = nullptr;
 
+// The use of 'volatile' here is to ensure that any particular thread always
+// reloads the value of the counter. The 'std::atomic' allows us to specify that
+// this variable is accessed in an unsychronized way (it's not actually
+// synchronizing). This does technically mean that the value may not appear to
+// be the same across threads running simultaneously on different CPUs, but in
+// practice the worst that will happen is that we won't print a stack trace when
+// we could have.
+//
+// This is initialized to 1 because 0 is used as a sentinel for "not enabled on
+// the current thread". If the user happens to overflow an 'unsigned' with
+// SIGINFO requests, it's possible that some threads will stop responding to it,
+// but the program won't crash.
+static volatile std::atomic<unsigned> GlobalSigInfoGenerationCounter =
+    ATOMIC_VAR_INIT(1);
+static LLVM_THREAD_LOCAL unsigned ThreadLocalSigInfoGenerationCounter = 0;
+
 namespace llvm {
 PrettyStackTraceEntry *ReverseStackTrace(PrettyStackTraceEntry *Head) {
   PrettyStackTraceEntry *Prev = nullptr;
   while (Head)
     std::tie(Prev, Head, Head->NextEntry) =
         std::make_tuple(Head, Head->NextEntry, Prev);
   return Prev;
 }
 }
 
 static void PrintStack(raw_ostream &OS) {
   // Print out the stack in reverse order. To avoid recursion (which is likely
   // to fail if we crashed due to stack overflow), we do an up-front pass to
   // reverse the stack, then print it, then reverse it again.
   unsigned ID = 0;
-  PrettyStackTraceEntry *ReversedStack =
-      llvm::ReverseStackTrace(PrettyStackTraceHead);
+  SaveAndRestore<PrettyStackTraceEntry *> SavedStack{PrettyStackTraceHead,
+                                                     nullptr};
+  PrettyStackTraceEntry *ReversedStack = ReverseStackTrace(SavedStack.get());
   for (const PrettyStackTraceEntry *Entry = ReversedStack; Entry;
        Entry = Entry->getNextEntry()) {
     OS << ID++ << ".\t";
     sys::Watchdog W(5);
     Entry->print(OS);
   }
   llvm::ReverseStackTrace(ReversedStack);
 }
 
 /// PrintCurStackTrace - Print the current stack trace to the specified stream.
+///
+/// Marked NOINLINE so it can be called from debuggers.
+LLVM_ATTRIBUTE_NOINLINE
 static void PrintCurStackTrace(raw_ostream &OS) {
   // Don't print an empty trace.
   if (!PrettyStackTraceHead) return;
 
   // If there are pretty stack frames registered, walk and emit them.
   OS << "Stack dump:\n";
 
   PrintStack(OS);
#elif HAVE_CRASHREPORTER_INFO
     __crashreporter_info__ = strdup(TmpStr.c_str());
 #endif
     errs() << TmpStr.str();
   }
 
 #endif
 }
 
+static void printForSigInfoIfNeeded() {
+  unsigned CurrentSigInfoGeneration =
+      GlobalSigInfoGenerationCounter.load(std::memory_order_relaxed);
+  if (ThreadLocalSigInfoGenerationCounter == 0 ||
+      ThreadLocalSigInfoGenerationCounter == CurrentSigInfoGeneration) {
+    return;
+  }
+
+  PrintCurStackTrace(errs());
+  ThreadLocalSigInfoGenerationCounter = CurrentSigInfoGeneration;
+}
+
 #endif // ENABLE_BACKTRACES
 
 PrettyStackTraceEntry::PrettyStackTraceEntry() {
 #if ENABLE_BACKTRACES
+  // Handle SIGINFO first, because we haven't finished constructing yet.
+  printForSigInfoIfNeeded();
   // Link ourselves.
   NextEntry = PrettyStackTraceHead;
   PrettyStackTraceHead = this;
 #endif
 }
 
 PrettyStackTraceEntry::~PrettyStackTraceEntry() {
 #if ENABLE_BACKTRACES
   assert(PrettyStackTraceHead == this &&
          "Pretty stack trace entry destruction is out of order");
   PrettyStackTraceHead = NextEntry;
+  // Handle SIGINFO first, because we already started destructing.
+  printForSigInfoIfNeeded();
 #endif
 }
 
 void PrettyStackTraceString::print(raw_ostream &OS) const { OS << Str << "\n"; }
 
 PrettyStackTraceFormat::PrettyStackTraceFormat(const char *Format, ...) {
   va_list AP;
   va_start(AP, Format);

 void llvm::EnablePrettyStackTrace() {
 #if ENABLE_BACKTRACES
   // The first time this is called, we register the crash printer.
   static bool HandlerRegistered = RegisterCrashPrinter();
   (void)HandlerRegistered;
 #endif
 }
 
+void llvm::EnablePrettyStackTraceOnSigInfoForThisThread(bool ShouldEnable) {
+#if ENABLE_BACKTRACES
+  if (!ShouldEnable) {
+    ThreadLocalSigInfoGenerationCounter = 0;
+    return;
+  }
+
+  // The first time this is called, we register the SIGINFO handler.
+  static bool HandlerRegistered = []{
+    sys::SetInfoSignalFunction([]{
+      GlobalSigInfoGenerationCounter.fetch_add(1, std::memory_order_relaxed);
+    });
+    return false;
+  }();
+  (void)HandlerRegistered;
+
+  // Next, enable it for the current thread.
+  ThreadLocalSigInfoGenerationCounter =
+      GlobalSigInfoGenerationCounter.load(std::memory_order_relaxed);
+#endif
+}
+
 const void *llvm::SavePrettyStackState() {
 #if ENABLE_BACKTRACES
   return PrettyStackTraceHead;
 #else
   return nullptr;
 #endif
 }
 

lib/Support/Unix/Signals.inc

 #include "llvm/Config/config.h"
 #include "llvm/Demangle/Demangle.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Mutex.h"
 #include "llvm/Support/Program.h"
+#include "llvm/Support/SaveAndRestore.h"
 #include "llvm/Support/UniqueLock.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <string>
 #include <sysexits.h>
 #ifdef HAVE_BACKTRACE
 # include BACKTRACE_HEADER         // For backtrace().
 #endif

 #else
 #undef HAVE__UNWIND_BACKTRACE
 #endif
 #endif
 
 using namespace llvm;
 
 static RETSIGTYPE SignalHandler(int Sig);  // defined below.
+static RETSIGTYPE InfoSignalHandler(int Sig);  // defined below.
 
+using SignalHandlerFunctionType = void (*)();
 /// The function to call if ctrl-c is pressed.
-using InterruptFunctionType = void (*)();
-static std::atomic<InterruptFunctionType> InterruptFunction =
+static std::atomic<SignalHandlerFunctionType> InterruptFunction =
+    ATOMIC_VAR_INIT(nullptr);
+static std::atomic<SignalHandlerFunctionType> InfoSignalFunction =
     ATOMIC_VAR_INIT(nullptr);
 
 namespace {
 /// Signal-safe removal of files.
 /// Inserting and erasing from the list isn't signal-safe, but removal of files
 /// themselves is signal-safe. Memory is freed when the head is freed, deletion
 /// is therefore not signal-safe either.
 class FileToRemoveList {
~FilesToRemoveCleanup() {
     if (Head)
       delete Head;
   }
 };
 } // namespace
 
 static StringRef Argv0;
 
-// Signals that represent requested termination. There's no bug or failure, or
-// if there is, it's not our direct responsibility. For whatever reason, our
-// continued execution is no longer desirable.
+/// Signals that represent requested termination. There's no bug or failure, or
+/// if there is, it's not our direct responsibility. For whatever reason, our
+/// continued execution is no longer desirable.
 static const int IntSigs[] = {
-  SIGHUP, SIGINT, SIGPIPE, SIGTERM, SIGUSR1, SIGUSR2
+  SIGHUP, SIGINT, SIGPIPE, SIGTERM, SIGUSR2
 };
 
-// Signals that represent that we have a bug, and our prompt termination has
-// been ordered.
+/// Signals that represent that we have a bug, and our prompt termination has
+/// been ordered.
 static const int KillSigs[] = {
   SIGILL, SIGTRAP, SIGABRT, SIGFPE, SIGBUS, SIGSEGV, SIGQUIT
 #ifdef SIGSYS
   , SIGSYS
 #endif
 #ifdef SIGXCPU
   , SIGXCPU
 #endif
 #ifdef SIGXFSZ
   , SIGXFSZ
 #endif
 #ifdef SIGEMT
   , SIGEMT
 #endif
 };
 
+/// Signals that represent requests for status.
+static const int InfoSigs[] = {
+  SIGUSR1
+#ifdef SIGINFO
+  , SIGINFO
+#endif
+};
+
+static const size_t NumSigs =
+    array_lengthof(IntSigs) + array_lengthof(KillSigs) +
+    array_lengthof(InfoSigs);
+
+
 static std::atomic<unsigned> NumRegisteredSignals = ATOMIC_VAR_INIT(0);
 static struct {
   struct sigaction SA;
   int SigNo;
-} RegisteredSignalInfo[array_lengthof(IntSigs) + array_lengthof(KillSigs)];
+} RegisteredSignalInfo[NumSigs];
 
 #if defined(HAVE_SIGALTSTACK)
 // Hold onto both the old and new alternate signal stack so that it's not
 // reported as a leak. We don't make any attempt to remove our alt signal
 // stack if we remove our signal handlers; that can't be done reliably if
 // someone else is also trying to do the same thing.
 static stack_t OldAltStack;
 static void* NewAltStackPointer;
static void RegisterHandlers() { // Not signal-safe.
   // If the handlers are already registered, we're done.
   if (NumRegisteredSignals.load() != 0)
     return;
 
   // Create an alternate stack for signal handling. This is necessary for us to
   // be able to reliably handle signals due to stack overflow.
   CreateSigAltStack();
 
-  auto registerHandler = [&](int Signal) {
+  enum class SignalKind { IsKill, IsInfo };
+  auto registerHandler = [&](int Signal, SignalKind Kind) {
     unsigned Index = NumRegisteredSignals.load();
     assert(Index < array_lengthof(RegisteredSignalInfo) &&
            "Out of space for signal handlers!");
 
     struct sigaction NewHandler;
 
+    switch (Kind) {
+    case SignalKind::IsKill:
       NewHandler.sa_handler = SignalHandler;
       NewHandler.sa_flags = SA_NODEFER | SA_RESETHAND | SA_ONSTACK;
+      break;
+    case SignalKind::IsInfo:
+      NewHandler.sa_handler = InfoSignalHandler;
+      NewHandler.sa_flags = SA_ONSTACK;
+      break;
+    }
     sigemptyset(&NewHandler.sa_mask);
 
     // Install the new handler, save the old one in RegisteredSignalInfo.
     sigaction(Signal, &NewHandler, &RegisteredSignalInfo[Index].SA);
     RegisteredSignalInfo[Index].SigNo = Signal;
     ++NumRegisteredSignals;
   };
 
   for (auto S : IntSigs)
-    registerHandler(S);
+    registerHandler(S, SignalKind::IsKill);
   for (auto S : KillSigs)
-    registerHandler(S);
+    registerHandler(S, SignalKind::IsKill);
+  for (auto S : InfoSigs)
+    registerHandler(S, SignalKind::IsInfo);
 }
 
 static void UnregisterHandlers() {
   // Restore all of the signal handlers to how they were before we showed up.
   for (unsigned i = 0, e = NumRegisteredSignals.load(); i != e; ++i) {
     sigaction(RegisteredSignalInfo[i].SigNo,
               &RegisteredSignalInfo[i].SA, nullptr);
     --NumRegisteredSignals;
#ifdef __s390__
   // *after* the faulting instruction.  Simply returning from the signal
   // handler would continue execution after that point, instead of
   // re-raising the signal.  Raise the signal manually in those cases.
   if (Sig == SIGILL || Sig == SIGFPE || Sig == SIGTRAP)
     raise(Sig);
 #endif
 }
 
+static RETSIGTYPE InfoSignalHandler(int Sig) {
+  SaveAndRestore<int> SaveErrnoDuringASignalHandler(errno);
+  if (SignalHandlerFunctionType CurrentInfoFunction = InfoSignalFunction)
+    CurrentInfoFunction();
+}
+
 void llvm::sys::RunInterruptHandlers() {
   RemoveFilesToRemove();
 }
 
 void llvm::sys::SetInterruptFunction(void (*IF)()) {
   InterruptFunction.exchange(IF);
   RegisterHandlers();
 }
 
+void llvm::sys::SetInfoSignalFunction(void (*Handler)()) {
+  InfoSignalFunction.exchange(Handler);
+  RegisterHandlers();
+}
+
 // The public API
 bool llvm::sys::RemoveFileOnSignal(StringRef Filename,
                                    std::string* ErrMsg) {
   // Ensure that cleanup will occur as soon as one file is added.
   static ManagedStatic<FilesToRemoveCleanup> FilesToRemoveCleanup;
   *FilesToRemoveCleanup;
   FileToRemoveList::insert(FilesToRemove, Filename.str());
   RegisterHandlers();

lib/Support/Windows/Signals.inc

 
 
 void llvm::sys::SetInterruptFunction(void (*IF)()) {
   RegisterHandler();
   InterruptFunction = IF;
   LeaveCriticalSection(&CriticalSection);
 }
 
+void llvm::sys::SetInfoSignalFunction(void (*Handler)()) {
+  // Unimplemented.
+}
+
 
 /// Add a function to be called when a signal is delivered to the process. The
 /// handler can have a cookie passed to it to identify what instance of the
 /// handler it is.
 void llvm::sys::AddSignalHandler(sys::SignalHandlerCallback FnPtr,
                                  void *Cookie) {
   insertSignalHandler(FnPtr, Cookie);
   RegisterHandler();