[compiler-rt] Expose fuzzer coverage bitmaps to the target

[a1xndr]

Summary

This change enables a fuzz-target to locate the coverage bitmaps that are
checked at the end of each fuzz-run. The main use-case for this is
a fuzz-target which, similarly to AFL, forks to reset state. Unlike AFL,
libfuzzer doesn't place the coverage data in shared memory. To work around
this, the target can map some shared memory and copy the coverage data from the
child to the parent, at the end of each fuzz run. There is an example of such a
fuzz target in test/fuzzer/DeferredForkTest.cpp

Another use-case is state-resetting through memory-snapshotting. In this case,
the target would use the information to roll-back changes to memory outside of
the fuzzer-related pages.

I am fuzzing virtual QEMU devices as part of Google Summer of Code 2019.
Previously, I used virtual machine snapshots and reboots to reset state, but the
fork-based approach often performs better.

The patch exposes a LLVMFuzzerIterateFeatureRegions() function to the guest,
which leaves the actual implementation of the forking/other type of resetting
logic up to the fuzz-target developer, offering some flexibility over an afl-style
forkserver.

[a1xndr]

Ping. This has been a useful feature for fuzzing QEMU virtual devices. It has some overhead due to the memcpy, but there are ways to avoid this overhead, by adding a user-modifiable counters pointer, or possibly re-mapping existing counter regions as shared memory.

[a1xndr]

Ping.

[lebedev.ri]

Not a libfuzzer person, but i wonder if this lifts the veil from internal implementation detail too much?

[a1xndr]

In D65672#1644661, @lebedev.ri wrote:

Not a libfuzzer person, but i wonder if this lifts the veil from internal implementation detail too much?

I think this is true, but at the same time exposing this info to the fuzzer developer provides them with flexibility to build e.g. custom fork-servers. You can get most of the way there by wrapping libfuzzer's __sanitizer_cov_8bit_counters_init etc using ld --wrap, but this doesn't cover internal counters such as TracePC::ValueProfileMap.
To avoid copying for each fuzzing-run, the counters could even be stored in a dynamically-allocated buffers, provided by the fuzz-target. This way, there is no need to copy the counters, as they could be stored directly in SHM.

[morehouse]

My opinion is that this exposes too much implementation detail and seems too hacky/niche to be merged into libFuzzer. But I defer the ultimate decision to @kcc (who is unlikely to respond until next week when he is back in the office).

For your use-case would it work to implement your initialization in LLVMFuzzerInitialize and then use fork mode?

lib/fuzzer/FuzzerInterface.h

 FUZZER_INTERFACE_VISIBILITY size_t
 LLVMFuzzerMutate(uint8_t *Data, size_t Size, size_t MaxSize);
 
+// Call a user-provided callback for each bitmap region containing coverage
+// info. This can be useful for fuzzers that reset state by forking, or using 
+// memory snapshots, since they shouldn't interfere with the coverage data
+FUZZER_INTERFACE_VISIBILITY void
+LLVMFuzzerIterateFeatureRegions(void (*CB)(void *, size_t));
 #undef FUZZER_INTERFACE_VISIBILITY
 
 #ifdef __cplusplus

lib/fuzzer/FuzzerLoop.cpp

   assert(fuzzer::F);
   return fuzzer::F->GetMD().DefaultMutate(Data, Size, MaxSize);
 }
-
+ATTRIBUTE_INTERFACE void LLVMFuzzerIterateFeatureRegions(void (*CB)(void *,
+                                                                    size_t)) {
+  fuzzer::TPC.IterateFeatureRegions(CB);
+}
 } // extern "C"

lib/fuzzer/FuzzerTracePC.h

   const PCTableEntry *PCTableEntryByIdx(uintptr_t Idx);
   static uintptr_t GetNextInstructionPc(uintptr_t PC);
   bool PcIsFuncEntry(const PCTableEntry *TE) { return TE->PCFlags & 1; }
+  void IterateFeatureRegions(void (*CB)(void *, size_t));
 
 private:
   bool UseCounters = false;

lib/fuzzer/FuzzerTracePC.cpp

   exit(1);
 }
 
+void TracePC::IterateFeatureRegions(void (*CB)(void *, size_t)) {
+  CB(&fuzzer::TPC, sizeof(fuzzer::TPC));
+  IterateCounterRegions(
+      [&](Module::Region &R) { CB(R.Start, R.Stop - R.Start); });
+  CB(ExtraCountersBegin(), ExtraCountersEnd() - ExtraCountersBegin());
+}
 } // namespace fuzzer
 
 extern "C" {

test/fuzzer/DeferredForkTest.cpp

+#include <cstring>
+#include <stdint.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+extern "C" void LLVMFuzzerIterateFeatureRegions(void (*CB)(void *, size_t));
+
+size_t feature_shm_len = 0;
+uintptr_t feature_shm;
+size_t offset = 0;
+
+void count_shm_size(void *start, size_t len) {
+  feature_shm_len += len;
+}
+
+void feature_load(void *start, size_t len) {
+  std::memcpy(start, (void *)(feature_shm + offset), len);
+  offset += len;
+}
+
+void feature_store(void *start, size_t len) {
+  std::memcpy((void *)(feature_shm + offset), start, len);
+  offset += len;
+}
+
+int isPrime(long n, int i) {
+  if (n <= 2)
+    return (n == 2) ? 1 : 0;
+  if (n % i == 0)
+    return 0;
+  if (i * i > n)
+    return 1;
+  return isPrime(n, i + 1);
+}
+
+int fuzz(const uint8_t *Data, size_t Size) {
+  long *nums = (long *)Data;
+  while (Size > sizeof(long)) {
+    Size -= sizeof(long);
+    isPrime(*nums, 2);
+    nums++;
+  }
+  return 0;
+}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
+  if (fork() == 0) {
+    fuzz(Data, Size);
+    offset = 0;
+    LLVMFuzzerIterateFeatureRegions(&feature_store);
+    _exit(0); // Prevent libfuzzer from catching the exit
+  } else {
+    wait(NULL);
+    offset = 0;
+    LLVMFuzzerIterateFeatureRegions(&feature_load);
+  }
+  return 0;
+}
+
+extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv) {
+  LLVMFuzzerIterateFeatureRegions(&count_shm_size);
+
+  feature_shm = (uintptr_t)mmap(NULL, feature_shm_len,
+                                PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+  return 0;
+}

test/fuzzer/deferred-fork-test.test

+RUN: %cpp_compiler %S/DeferredForkTest.cpp -o %t-DeferredForkTest
+
+RUN: not %run %t-DeferredForkTest 2>&1 | FileCheck %s --check-prefix=DEFERRED_FORK