[lib/Fuzzer] Add sanitizer runtime errors unit save option

[skomski]

Summary

Make it possible to save runtime errors units from for example fsanitize=undefined to make it easier to reproduce them.

[kcc]

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

lib/Fuzzer/FuzzerFlags.def

62

write more detailed description (e.g. "save sanitizer error report to a file with 'error-' prefix" or some such)

lib/Fuzzer/FuzzerLoop.cpp

16

__sanitizer_report_error_summary may be used by other parts of code linked with libSanitizer.
Also, summary is just a single line, not sure how it may be useful.

[skomski]

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

Can you not say the same thing for the current written timeout- and crash files? I like having the files right away it makes it more convenient.

lib/Fuzzer/FuzzerLoop.cpp

16

Is there an alternative to sanitizer_report_error_summary to receive the runtime error reports? I wanted to use asan_set_error_report_callback but that doesn't work

Also, summary is just a single line, not sure how it may be useful.

I wanted to use replicate the original behaviour of __sanitizer_report_error_summary that does fuzzer::Printf("%s\n", ErrorSummary);

[kcc]

In D11277#206781, @skomski wrote:

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

Can you not say the same thing for the current written timeout- and crash files? I like having the files right away it makes it more convenient.

That's a bit different. The timeout- and crash- files can be directly given back to the fuzzer (the target function).
The file with the log is for a human being to analyze -- the same as for the error report.

[kcc]

lib/Fuzzer/FuzzerLoop.cpp

16

asan_set_error_report_callback is closer to what you need but

1. it's only available in asan (well, this *may* need to be fixed separately)
2. it has the same problem: what if someone else want to call it to. I'e. it will need to be chained.
3. You say it does not work. Strange. Will need more info separately.

Does __sanitizer_set_report_path do (almost) what you need?

[skomski]

In D11277#207372, @kcc wrote:

In D11277#206781, @skomski wrote:

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

In D11277#206713, @kcc wrote:

First question: why is a log file not enough?
It will contain the full sanitizer report (not just summary) and the reproducer.
Frankly, to me this looks like a duplicated functionality.

Can you not say the same thing for the current written timeout- and crash files? I like having the files right away it makes it more convenient.

That's a bit different. The timeout- and crash- files can be directly given back to the fuzzer (the target function).
The file with the log is for a human being to analyze -- the same as for the error report.

I don't save the error log but the corresponding unit that is active during time the sanitizer reports the runtime error which results in the same file as timeout and crash files.

[kcc]

I am sorry, I may be missing something, can we start again?
What problem exactly are you trying to solve?

[skomski]

In D11277#207577, @kcc wrote:

I am sorry, I may be missing something, can we start again?
What problem exactly are you trying to solve?

My problem is that I want to have an easy way to reproduce undefined behaviour errors for further debugging purposes or to put it into my testcases. I want to achieve that saving the active unit during the time ubsan reported the error.

lib/Fuzzer/FuzzerLoop.cpp

16

3. Well it works but only the hard crashes from asan are reported that are already handled.

https://github.com/llvm-mirror/compiler-rt/blob/6e6e601da1cd63c2e797a4b185f5ebc77496e1dc/lib/asan/asan_report.cc#L666

[kcc]

In D11277#207792, @skomski wrote:

In D11277#207577, @kcc wrote:

I am sorry, I may be missing something, can we start again?
What problem exactly are you trying to solve?

My problem is that I want to have an easy way to reproduce undefined behaviour errors for further debugging purposes or to put it into my testcases. I want to achieve that saving the active unit during the time ubsan reported the error.

That's exactly what I always need too. But the way current way of doing this work fine to me:

• Run the fuzzer with stderr pointed to a file (if running with -jobs=N, stderr is directed to fuzz-NN.log)
• When the crash happens, the log file contains *all* the information needed to reproduce the error: the message from the sanitizers, the input file encoded in hex and base64, the name of a separate file where the reproducer has been dumped to.

So, before considering this patch I need to understand what's missing in the existing functionality.

BTW, I've noticed fixes in PCRE2 based on your reports. Great work!

[skomski]

My problem is that I want to have an easy way to reproduce undefined

behaviour errors for further debugging purposes or to put it into my
testcases. I want to achieve that saving the active unit during the time
ubsan reported the error.

It is about UBSAN (!) errors. Ubsan reports only and doesn't crash the
process.

For example:
src/pcre2_compile.c:6453:42: runtime error: unsigned integer overflow:
4294967295 + 1 cannot be represented in type 'unsigned int'
SUMMARY: AddressSanitizer: undefined-behavior src/pcre2_compile.c:6453:42
in

I want to save the unit that was running during the error.

Kind regards,

Karl Skomski

[kcc]

It is about UBSAN (!) errors. Ubsan reports only and doesn't crash the
process.

Aha, so that's what you need to fix, not the fuzzer.
Will -fno-sanitize-recover=undefined work for you?

[skomski]

Yes that would be a way to handle it albeit I would still use my patch
because I don't really want the fuzzer to crash from undefined behaviour. I
only want a reproducible unit and some log event with the unit information.
Also I don't want it to save under crash-*. But I am not desperate to merge
it I am happy with my patch :D

Kind regards,

Karl Skomski

[kcc]

Yes that would be a way to handle it albeit I would still use my patch
because I don't really want the fuzzer to crash from undefined behaviour. I

I think this is wrong.
The fuzzer is designed in assumption that any interesting bug leads to a process crash
and I'd like to preserve this simplicity.

lib/Fuzzer/FuzzerDriver.cpp

   if (Flags.sync_command)
     Options.SyncCommand = Flags.sync_command;
   Options.SyncTimeout = Flags.sync_timeout;
+  Options.SaveRuntimeSanitizerErrors = Flags.save_runtime_sanitizer_errors;
   Fuzzer F(USF, Options);
 
   if (Flags.apply_tokens)

lib/Fuzzer/FuzzerFlags.def

                                  "\"<sync_command> <test_corpus>\" "
                                  "to synchronize the test corpus.")
 FUZZER_FLAG_INT(sync_timeout, 600, "Minimum timeout between syncs.")
+FUZZER_FLAG_INT(save_runtime_sanitizer_errors, 0,
+                "Save runtime sanitizer errors"
+                " For example from fsanitize=undefined")

lib/Fuzzer/FuzzerInternal.h

     std::string OutputCorpus;
     std::string SyncCommand;
     std::vector<std::string> Tokens;
+    bool SaveRuntimeSanitizerErrors = false;
   };
   Fuzzer(UserSuppliedFuzzer &USF, FuzzingOptions Options);
   void AddToCorpus(const Unit &U) { Corpus.push_back(U); }

 
   Unit SubstituteTokens(const Unit &U) const;
 
+  static void StaticErrorReportCallback(const char *ErrorSummary);
+
  private:
   void AlarmCallback();
   void ExecuteCallback(const Unit &U);

   size_t RunOneMaximizeFullCoverageSet(const Unit &U);
   size_t RunOneMaximizeCoveragePairs(const Unit &U);
   void WriteToOutputCorpus(const Unit &U);
-  void WriteToCrash(const Unit &U, const char *Prefix);
+  void WriteToCrash(const Unit &U, const char *Prefix, const char* EventName);
   void PrintStats(const char *Where, size_t Cov, const char *End = "\n");
   void PrintUnitInASCIIOrTokens(const Unit &U, const char *PrintAfter = "");
 

   // Apply Idx-th trace-based mutation to U.
   void ApplyTraceBasedMutation(size_t Idx, Unit *U);
 
+
+  void ErrorReportCallback(const char *ErrorSummary);
+
   void SetDeathCallback();
   static void StaticDeathCallback();
   void DeathCallback();

   UserCallback Callback;
 };
 
-};  // namespace fuzzer
+}  // namespace fuzzer

lib/Fuzzer/FuzzerLoop.cpp

 #include <sanitizer/coverage_interface.h>
 #include <algorithm>
 
+void __sanitizer_report_error_summary(const char *ErrorSummary) {
+  fuzzer::Fuzzer::StaticErrorReportCallback(ErrorSummary);
+}
+
 namespace fuzzer {
 
 // Only one Fuzzer per process.

   }
 }
 
+void Fuzzer::StaticErrorReportCallback(const char* ErrorSummary) {
+  assert(F);
+  F->ErrorReportCallback(ErrorSummary);
+}
+
+void Fuzzer::ErrorReportCallback(const char* ErrorSummary) {
+  fuzzer::Printf("%s\n", ErrorSummary);
+
+  if (Options.SaveRuntimeSanitizerErrors) {
+    Printf("ERROR:\n");
+    Print(CurrentUnit, "\n");
+    PrintUnitInASCIIOrTokens(CurrentUnit, "\n");
+    WriteToCrash(CurrentUnit, "error-", "ERROR");
+  }
+}
+
 void Fuzzer::StaticDeathCallback() {
   assert(F);
   F->DeathCallback();

   Printf("DEATH:\n");
   Print(CurrentUnit, "\n");
   PrintUnitInASCIIOrTokens(CurrentUnit, "\n");
-  WriteToCrash(CurrentUnit, "crash-");
+  WriteToCrash(CurrentUnit, "crash-", "CRASHED");
 }
 
 void Fuzzer::StaticAlarmCallback() {

            Options.UnitTimeoutSec);
     Print(CurrentUnit, "\n");
     PrintUnitInASCIIOrTokens(CurrentUnit, "\n");
-    WriteToCrash(CurrentUnit, "timeout-");
+    WriteToCrash(CurrentUnit, "timeout-", "TIMEOUT");
     exit(1);
   }
 }

     Printf("Written to %s\n", Path.c_str());
 }
 
-void Fuzzer::WriteToCrash(const Unit &U, const char *Prefix) {
+void Fuzzer::WriteToCrash(const Unit &U, const char *Prefix, const char* name) {
   std::string Path = Prefix + Hash(U);
   WriteToFile(U, Path);
-  Printf("CRASHED; file written to %s\nBase64: ", Path.c_str());
+  Printf("%s; file written to %s\nBase64: ", name, Path.c_str());
   PrintFileAsBase64(Path);
+  Printf("\n");
 }
 
 void Fuzzer::SaveCorpus() {

lib/Fuzzer/test/CMakeLists.txt

 # basic blocks and we'll fail to discover the targets.
 # Also enable the coverage instrumentation back (it is disabled
 # for the Fuzzer lib)
-set(CMAKE_CXX_FLAGS_RELEASE "${LIBFUZZER_FLAGS_BASE} -O0 -fsanitize-coverage=edge,indirect-calls")
+set(CMAKE_CXX_FLAGS_RELEASE "${LIBFUZZER_FLAGS_BASE} -O0 -fsanitize=undefined -fsanitize-coverage=edge,indirect-calls")
 
 set(DFSanTests
   DFSanMemcmpTest

   NullDerefTest
   SimpleTest
   TimeoutTest
+  RuntimeErrorTest
   ${DFSanTests}
   )
 

lib/Fuzzer/test/RuntimeErrorTest.cpp

+#include <cstdint>
+#include <cstdlib>
+#include <cstddef>
+#include <iostream>
+
+static volatile unsigned int Sink;
+
+extern "C" void LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
+  if (Size > 0 && Data[0] == 'H') {
+    Sink = 1;
+    if (Size > 1 && Data[1] == 'i') {
+      Sink = 2;
+      if (Size > 2 && Data[2] == '!') {
+        int64_t i = 1; i <<= 72;
+        exit(0);
+      }
+    }
+  }
+}
+

lib/Fuzzer/test/fuzzer.test

 
 RUN: not ./LLVMFuzzer-InfiniteTest -timeout=2 2>&1 | FileCheck %s --check-prefix=InfiniteTest
 InfiniteTest: ALARM: working on the last Unit for
-InfiniteTest: CRASHED; file written to timeout
+InfiniteTest: TIMEOUT; file written to timeout
 
 RUN: not ./LLVMFuzzer-TimeoutTest -timeout=5 2>&1 | FileCheck %s --check-prefix=TimeoutTest
 TimeoutTest: ALARM: working on the last Unit for
-TimeoutTest: CRASHED; file written to timeout
+TimeoutTest: TIMEOUT; file written to timeout
 
 RUN: not ./LLVMFuzzer-NullDerefTest 2>&1 | FileCheck %s --check-prefix=NullDerefTest
 NullDerefTest: CRASHED; file written to crash-

 
 RUN: not ./LLVMFuzzer-UserSuppliedFuzzerTest -seed=1 -timeout=15 2>&1 | FileCheck %s
 
+RUN: ./LLVMFuzzer-RuntimeErrorTest -save_runtime_sanitizer_errors=1  2>&1 | FileCheck %s --check-prefix=RuntimeErrorTest
+RuntimeErrorTest: ERROR; file written to