[LibFuzzer] Fix sending SIGALRM to main thread under Mac OSX

[delcypher]

Summary

[LibFuzzer] Fix sending SIGALRM signal to main thread on Mac OSX.

This replaces the Linux only implementation with an implementation that
relies on pthread_kill() which is more portable. For this to work
the Fuzzer now records the thread it was created in (presumably the main
thread).

This isn't an ideal fix because now we have a pthreads dependency
but arguably if we wanted portability we shouldn't be using signals
either.

[kcc]

dependency on -lpthread would be unfortunate...
And I don't like the current code either.
Let me rethink this one.
This may be solved by printing and Exiting_ in the getrusage thread --
the only question is how to avoid the race on CurrentUnitData/CurrentUnitSize w/o having to use a Mutex.

Want to try this?

[kcc]

Or maybe just grab a mutex around CurrentUnitData/CurrentUnitSize
and see how much it affects performance on e.g. test/FullCoverageSetTest.cpp
If not much, just use a mutex.

[delcypher]

@kcc : Just to note I think you already have an implicit pthreads dependency. The CMakeLists.txt file seems to link
against it also (side note: why are the flags only being set for the CMake build type being RELEASE?).

I didn't need to pass -lpthread when building (I'm building independently of LLVM's CMake build system) on my system. I suspect that std::thread might already be using pthreads internally.

[kcc]

In D20461#435255, @delcypher wrote:

@kcc : Just to note I think you already have an implicit pthreads dependency.

Very likely. But it's more like I want to stay away from including pthread.h, if possible.
And also, the current way of dying on OOMs is less portable than just dying inside the getrusage thread.

The CMakeLists.txt file seems to link
against it also (side note: why are the flags only being set for the CMake build type being RELEASE?).

For no good reason, feel free to fix that.

I didn't need to pass -lpthread when building (I'm building independently of LLVM's CMake build system) on my system. I suspect that std::thread might already be using pthreads internally.

Note that the majority of users use libFuzzer outside of the llvm build system,
will they need to add -lphtread to their build rules?

[delcypher]

@kcc

This may be solved by printing and Exiting_ in the getrusage thread --
the only question is how to avoid the race on CurrentUnitData/CurrentUnitSize w/o having to use a Mutex.

Want to try this?

I'll take a look tomorrow. I need to first take a look at how CurrentUnitData and CurrentUnitSize are being used and consequently where they need to locked.

Just a thought. Why not just be single threaded and call getrusage() after every iteration of LLVMFuzzerTestOneInput(...)? It will mean for long running and non-terminating runs of LLVMFuzzerTestOneInput(...) we might not (or ever) detect OOM but it would avoid the complexity and the races. Personally I feel like detecting OOM should really be done externally using something like cgroups, but maybe that's a bad idea.

[delcypher]

Note that the majority of users use libFuzzer outside of the llvm build system,
will they need to add -lphtread to their build rules?

I found that I didn't need to under OSX or Linux. Note I didn't really test under Linux properly as something about my distro's clang is broken (__sanitizer_get_coverage_pc_buffer seems to be NULL at runtime). I need to rebuild on trunk and test more thoroughly.

[kcc]

In D20461#435258, @delcypher wrote:

@kcc

This may be solved by printing and Exiting_ in the getrusage thread --

the only question is how to avoid the race on CurrentUnitData/CurrentUnitSize w/o having to use a Mutex.

Want to try this?

I'll take a look tomorrow. I need to first take a look at how CurrentUnitData and CurrentUnitSize are being used and consequently where they need to locked.

CurrentUnitData is used to dump the reproducer on disk.

Just a thought. Why not just be single threaded and call getrusage() after every iteration of LLVMFuzzerTestOneInput(...)? It will mean for long running and non-terminating runs of LLVMFuzzerTestOneInput(...) we might not (or ever) detect OOM but it would avoid the complexity and the races. Personally I feel like detecting OOM should really be done externally using something like cgroups, but maybe that's a bad idea.

One of the two reasons for detecting OOMs in libFuzzer is to not let it kill your machine.
You can't do it by checking getrusage after your machine has died.

cgroups are hard to set up, you should not expect every libFuzzer user to do it.
AFAICT, they also require sudo (no?)
Ideally we would have kernel support for RLIMIT_RSS, but there is no such thing today. :(

[kcc]

FYI: am working on refactoring this part, hold on your changes please...

[delcypher]

@kcc:

FYI: am working on refactoring this part, hold on your changes please...

Thanks for the info. I took a break from doing this as I wanted to make it possible to run the tests on OSX first. Glad to hear you are taking a look at it.

[kcc]

I think I've fixed this in r270945.
This part is tricky because it involves both threads and signals,
so I tried to make it simpler.

[delcypher]

This change isn't needed any more.

lib/Fuzzer/FuzzerInternal.h

 #define LLVM_FUZZER_INTERNAL_H
 
 #include <algorithm>
 #include <cassert>
 #include <chrono>
 #include <climits>
 #include <cstddef>
 #include <cstdlib>
+#include <pthread.h>
 #include <random>
 #include <string.h>
 #include <string>
 #include <unordered_set>
 #include <vector>
 
 #include "FuzzerInterface.h"
 #include "FuzzerTracePC.h"

 
 // Changes U to contain only ASCII (isprint+isspace) characters.
 // Returns true iff U has been changed.
 bool ToASCII(uint8_t *Data, size_t Size);
 bool IsASCII(const Unit &U);
 
 int NumberOfCpuCores();
 int GetPid();
-int SignalToMainThread();
+int SignalToMainThread(pthread_t MainThread);
 void SleepSeconds(int Seconds);
 
 class Random {
  public:
   Random(unsigned int seed) : R(seed) {}
   size_t Rand() { return R(); }
   size_t RandBool() { return Rand() % 2; }
   size_t operator()(size_t n) { return n ? Rand() % n : 0; }
private:
   FuzzingOptions Options;
   system_clock::time_point ProcessStartTime = system_clock::now();
   system_clock::time_point UnitStartTime;
   long TimeOfLongestUnitInSeconds = 0;
   long EpochOfLastReadOfOutputCorpus = 0;
 
   // Maximum recorded coverage.
   Coverage MaxCoverage;
+
+  // Used to allow other threads to communicate with
+  // the thread the fuzzer was created in.
+  pthread_t MainThread;
 };
 
 }; // namespace fuzzer
 
 #endif // LLVM_FUZZER_INTERNAL_H

lib/Fuzzer/FuzzerLoop.cpp

if (NewPcBufferLen > C->PcBufferLen) {
       C->PcBufferLen = NewPcBufferLen;
     }
 
     return Res;
   }
 };
 
 Fuzzer::Fuzzer(UserCallback CB, MutationDispatcher &MD, FuzzingOptions Options)
-    : CB(CB), MD(MD), Options(Options) {
+    : CB(CB), MD(MD), Options(Options), MainThread(pthread_self()) {
   SetDeathCallback();
   InitializeTraceState();
   assert(!F);
   F = this;
   ResetCoverage();
 }
 
 void Fuzzer::SetDeathCallback() {
if (Seconds >= (size_t)Options.UnitTimeoutSec) {
     Printf("SUMMARY: libFuzzer: timeout\n");
     PrintFinalStats();
     _Exit(Options.TimeoutExitCode); // Stop right now.
   }
 }
 
 void Fuzzer::RssLimitCallback() {
   InOOMState = true;
-  SignalToMainThread();
+  if (SignalToMainThread(MainThread)) {
+    Printf("WARNING: Failed to signal main thread.\n");
+  }
   SleepSeconds(5);
-  Printf("Signal to main thread failed (non-linux?). Exiting.\n");
+  Printf("ERROR: Should have exited already. Forcing exit.\n");
   _Exit(Options.ErrorExitCode);
   return;
 }
 
 void Fuzzer::PrintStats(const char *Where, const char *End) {
   size_t ExecPerSec = execPerSec();
   if (Options.OutputCSV) {
     static bool csvHeaderPrinted = false;

lib/Fuzzer/FuzzerUtil.cpp

bool ParseDictionaryFile(const std::string &Text, std::vector<Unit> *Units) {
   return true;
 }
 
 void SleepSeconds(int Seconds) {
   std::this_thread::sleep_for(std::chrono::seconds(Seconds));
 }
 
 int GetPid() { return getpid(); }
-int SignalToMainThread() {
-#ifdef __linux__
-  return syscall(SYS_tgkill, GetPid(), GetPid(), SIGALRM);
-#else
-  return 0;
-#endif
+int SignalToMainThread(pthread_t MainThread) {
+  return pthread_kill(MainThread, SIGALRM);
 }
 
 std::string Base64(const Unit &U) {
   static const char Table[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
                               "abcdefghijklmnopqrstuvwxyz"
                               "0123456789+/";
   std::string Res;
   size_t i;