Generate error reports when a fuzz target exits.

[morehouse]

Summary

Implements https://github.com/google/sanitizers/issues/835.

Flush stdout before exiting in test cases.

Since the atexit hook is used for exit reports, pending prints to
stdout can be lost if they aren't flushed before calling exit().

Expect tests to have non-zero exit code if exit() is called.

[kcc]

The diff is done from $(LLVM)/..
Please make it from ($LLVM) instead
LGTM otherwise.
Also, please apply for the commit access.

llvm/lib/Fuzzer/test/SimpleThreadedTest.cpp

16

redundant change there? (abort, not exit)

[morehouse]

In D35602#814861, @kcc wrote:

The diff is done from $(LLVM)/..
Please make it from ($LLVM) instead

I don't quite understand what this means. Could you explain further?

[kcc]

When I do "arc patch" locally it works for me only if the paths are like "lib/Fuzzer/test/fuzzer.test", not "llvm/lib/Fuzzer/test/fuzzer.test".
(Maybe I am missing something in how arc might help me... )

[morehouse]

Pushed in r308669.

[Dor1s]

Sorry for necroposting, but there are some compatibility issues with Clang coverage (see my comment on FuzzerDriver.cpp#646). What could be a good workaround?

llvm/lib/Fuzzer/FuzzerDriver.cpp

646

This change breaks Clang Source-based Code Coverage. For example, if you run ./fuzzer -runs=0 ./corpus_dir with 123 files in corpus_dir, you'll see LLVMFuzzerTestOneInput executed only once, not 123 times.

We've temporarily rolled libFuzzer back Chromium because of that.

[morehouse]

One option may be to add a flag to disable the exit hook and use that when doing coverage builds. Actually I thought I included a flag in this revision, but apparently not.

[morehouse]

Also, are you sure this change is what broke coverage? I can replicate this issue at ToT libFuzzer, but not at r308669.

And if I comment out the atexit hook at ToT, the issue persists for me.

[Dor1s]

Matt, you are right, this CL is not a culprit. Sorry for the false signal. Now I found the real issue, it is the following revision: https://chromium.googlesource.com/chromium/llvm-project/compiler-rt/lib/fuzzer.git/+/1c0379f93124d95b87f3b62bf61d90882a669387

In particular, the new call added below is causing that:

diff --git a/FuzzerTracePC.h b/FuzzerTracePC.h
index ea6794c..56f1820 100644
--- a/FuzzerTracePC.h
+++ b/FuzzerTracePC.h
@@ -91,6 +91,7 @@
       memset(Counters(), 0, GetNumPCs());
     ClearExtraCounters();
     ClearInlineCounters();
+    ClearClangCounters();
   }

Disabling that with an extra cmd flag doesn't sound as a good solution to me. It would complicate usage for the end user (a regular developer) a little bit more. Can we think of anything else? //cc @kcc

[Dor1s]

Discussed with Kostya and fixed: https://reviews.llvm.org/rL315029

llvm/lib/Fuzzer/FuzzerDriver.cpp

 #include "FuzzerInternal.h"
 #include "FuzzerMutate.h"
 #include "FuzzerRandom.h"
 #include "FuzzerShmem.h"
 #include "FuzzerTracePC.h"
 #include <algorithm>
 #include <atomic>
 #include <chrono>
+#include <cstdlib>
 #include <cstring>
 #include <mutex>
 #include <string>
 #include <thread>
 
 // This function should be present in the libFuzzer so that the client
 // binary can test for its existence.
 extern "C" __attribute__((used)) void __libfuzzer_is_present() {}
int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   Options.HandleFpe = Flags.handle_fpe;
   Options.HandleIll = Flags.handle_ill;
   Options.HandleInt = Flags.handle_int;
   Options.HandleSegv = Flags.handle_segv;
   Options.HandleTerm = Flags.handle_term;
   Options.HandleXfsz = Flags.handle_xfsz;
   SetSignalHandler(Options);
 
+  std::atexit(Fuzzer::StaticExitCallback);
+
   if (Flags.minimize_crash)
     return MinimizeCrashInput(Args, Options);
 
   if (Flags.minimize_crash_internal_step)
     return MinimizeCrashInputInternalStep(F, Corpus);
 
   if (Flags.cleanse_crash)
     return CleanseCrashInput(Args, Options);

llvm/lib/Fuzzer/FuzzerInternal.h

size_t execPerSec() {
     size_t Seconds = secondsSinceProcessStartUp();
     return Seconds ? TotalNumberOfRuns / Seconds : 0;
   }
 
   size_t getTotalNumberOfRuns() { return TotalNumberOfRuns; }
 
   static void StaticAlarmCallback();
   static void StaticCrashSignalCallback();
+  static void StaticExitCallback();
   static void StaticInterruptCallback();
   static void StaticFileSizeExceedCallback();
 
   void ExecuteCallback(const uint8_t *Data, size_t Size);
   bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile = false,
               InputInfo *II = nullptr);
 
   // Merge Corpora[1:] into Corpora[0].
void TryDetectingAMemoryLeak(const uint8_t *Data, size_t Size,
                                bool DuringInitialCorpusExecution);
 
   void HandleMalloc(size_t Size);
   void AnnounceOutput(const uint8_t *Data, size_t Size);
 
 private:
   void AlarmCallback();
   void CrashCallback();
+  void ExitCallback();
   void CrashOnOverwrittenData();
   void InterruptCallback();
   void MutateAndTestOne();
   void ReportNewCoverage(InputInfo *II, const Unit &U);
   void PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size);
   void WriteToOutputCorpus(const Unit &U);
   void WriteUnitToFileWithPrefix(const Unit &U, const char *Prefix);
   void PrintStats(const char *Where, const char *End = "\n", size_t Units = 0);

llvm/lib/Fuzzer/FuzzerLoop.cpp

void Fuzzer::StaticAlarmCallback() {
   F->AlarmCallback();
 }
 
 void Fuzzer::StaticCrashSignalCallback() {
   assert(F);
   F->CrashCallback();
 }
 
+void Fuzzer::StaticExitCallback() {
+  assert(F);
+  F->ExitCallback();
+}
+
 void Fuzzer::StaticInterruptCallback() {
   assert(F);
   F->InterruptCallback();
 }
 
 void Fuzzer::StaticFileSizeExceedCallback() {
   Printf("==%lu== ERROR: libFuzzer: file size exceeded\n", GetPid());
   exit(1);
 }
 
 void Fuzzer::CrashCallback() {
   Printf("==%lu== ERROR: libFuzzer: deadly signal\n", GetPid());
   if (EF->__sanitizer_print_stack_trace)
     EF->__sanitizer_print_stack_trace();
   Printf("NOTE: libFuzzer has rudimentary signal handlers.\n"
          "      Combine libFuzzer with AddressSanitizer or similar for better "
          "crash reports.\n");
   Printf("SUMMARY: libFuzzer: deadly signal\n");
   DumpCurrentUnit("crash-");
   PrintFinalStats();
   _Exit(Options.ErrorExitCode);  // Stop right now.
 }
 
+void Fuzzer::ExitCallback() {
+  if (!RunningCB)
+    return; // This exit did not come from the user callback
+  Printf("==%lu== ERROR: libFuzzer: fuzz target exited\n", GetPid());
+  if (EF->__sanitizer_print_stack_trace)
+    EF->__sanitizer_print_stack_trace();
+  Printf("SUMMARY: libFuzzer: fuzz target exited\n");
+  DumpCurrentUnit("crash-");
+  PrintFinalStats();
+  _Exit(Options.ErrorExitCode);
+}
+
+
 void Fuzzer::InterruptCallback() {
   Printf("==%lu== libFuzzer: run interrupted; exiting\n", GetPid());
   PrintFinalStats();
   _Exit(0);  // Stop right now, don't perform any at-exit actions.
 }
 
 NO_SANITIZE_MEMORY
 void Fuzzer::AlarmCallback() {

llvm/lib/Fuzzer/test/AbsNegAndConstant64Test.cpp

 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   if (Size < 16 || Size > 64) return 0;
   int64_t x;
   uint64_t y;
   memcpy(&x, Data, sizeof(x));
   memcpy(&y, Data + sizeof(x), sizeof(y));
   if (llabs(x) < 0 && y == 0xbaddcafedeadbeefULL) {
     printf("BINGO; Found the target, exiting; x = 0x%lx y 0x%lx\n", x, y);
+    fflush(stdout);
     exit(1);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/AbsNegAndConstantTest.cpp

 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   if (Size < 8) return 0;
   int x;
   unsigned y;
   memcpy(&x, Data, sizeof(x));
   memcpy(&y, Data + sizeof(x), sizeof(y));
   if (abs(x) < 0 && y == 0xbaddcafe) {
     printf("BINGO; Found the target, exiting; x = 0x%x y 0x%x\n", x, y);
+    fflush(stdout);
     exit(1);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/BufferOverflowOnInput.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a fuzzer. The fuzzer must find the string "Hi!".
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <iostream>
+#include <ostream>
 
 static volatile bool SeedLargeBuffer;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   assert(Data);
   if (Size >= 4)
     SeedLargeBuffer = true;
   if (Size == 3 && SeedLargeBuffer && Data[3]) {
-    std::cout << "Woops, reading Data[3] w/o crashing\n";
+    std::cout << "Woops, reading Data[3] w/o crashing\n" << std::flush;
     exit(1);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/CustomCrossOverTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a cutom mutator.
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <iostream>
+#include <ostream>
 #include <random>
 #include <string.h>
 
 #include "FuzzerInterface.h"
 
 static const char *Separator = "-_^_-";
 static const char *Target = "012-_^_-abc";
 
 static volatile int sink;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   assert(Data);
   std::string Str(reinterpret_cast<const char *>(Data), Size);
 
   // Ensure that two different elements exist in the corpus.
   if (Size && Data[0] == '0') sink++;
   if (Size && Data[0] == 'a') sink--;
 
   if (Str.find(Target) != std::string::npos) {
-    std::cout << "BINGO; Found the target, exiting\n";
+    std::cout << "BINGO; Found the target, exiting\n" << std::flush;
     exit(1);
   }
   return 0;
 }
 
 extern "C" size_t LLVMFuzzerCustomCrossOver(const uint8_t *Data1, size_t Size1,
                                             const uint8_t *Data2, size_t Size2,
                                             uint8_t *Out, size_t MaxOutSize,

llvm/lib/Fuzzer/test/CustomMutatorTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a cutom mutator.
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <iostream>
+#include <ostream>
 
 #include "FuzzerInterface.h"
 
 static volatile int Sink;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   assert(Data);
   if (Size > 0 && Data[0] == 'H') {
     Sink = 1;
     if (Size > 1 && Data[1] == 'i') {
       Sink = 2;
       if (Size > 2 && Data[2] == '!') {
-        std::cout << "BINGO; Found the target, exiting\n";
+        std::cout << "BINGO; Found the target, exiting\n" << std::flush;
         exit(1);
       }
     }
   }
   return 0;
 }
 
 extern "C" size_t LLVMFuzzerCustomMutator(uint8_t *Data, size_t Size,
                                           size_t MaxSize, unsigned int Seed) {
   static bool Printed;
   if (!Printed) {
     std::cerr << "In LLVMFuzzerCustomMutator\n";
     Printed = true;
   }
   return LLVMFuzzerMutate(Data, Size, MaxSize);
 }

llvm/lib/Fuzzer/test/NthRunCrashTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Crash on the N-th execution.
 #include <cstddef>
 #include <cstdint>
 #include <iostream>
+#include <ostream>
 
 static int Counter;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   if (Counter++ == 1000) {
-    std::cout << "BINGO; Found the target, exiting\n";
+    std::cout << "BINGO; Found the target, exiting\n" << std::flush;
     exit(1);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/RepeatedBytesTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a fuzzer. The fuzzer must find repeated bytes.
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <iostream>
+#include <ostream>
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   assert(Data);
   // Looking for AAAAAAAAAAAAAAAAAAAAAA or some such.
   size_t CurA = 0, MaxA = 0;
   for (size_t i = 0; i < Size; i++) {
     // Make sure there are no conditionals in the loop so that
     // coverage can't help the fuzzer.
     int EQ = Data[i] == 'A';
     CurA = EQ * (CurA + 1);
     int GT = CurA > MaxA;
     MaxA = GT * CurA + (!GT) * MaxA;
   }
   if (MaxA >= 20) {
-    std::cout << "BINGO; Found the target (Max: " << MaxA << "), exiting\n";
+    std::cout << "BINGO; Found the target (Max: " << MaxA << "), exiting\n"
+              << std::flush;
     exit(0);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/SimpleDictionaryTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a fuzzer.
 // The fuzzer must find a string based on dictionary words:
 //   "Elvis"
 //   "Presley"
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <cstring>
 #include <iostream>
+#include <ostream>
 
 static volatile int Zero = 0;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   const char *Expected = "ElvisPresley";
   if (Size < strlen(Expected)) return 0;
   size_t Match = 0;
   for (size_t i = 0; Expected[i]; i++)
     if (Expected[i] + Zero == Data[i])
       Match++;
   if (Match == strlen(Expected)) {
-    std::cout << "BINGO; Found the target, exiting\n";
+    std::cout << "BINGO; Found the target, exiting\n" << std::flush;
     exit(1);
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/SimpleTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Simple test for a fuzzer. The fuzzer must find the string "Hi!".
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstdlib>
 #include <iostream>
+#include <ostream>
 
 static volatile int Sink;
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   assert(Data);
   if (Size > 0 && Data[0] == 'H') {
     Sink = 1;
     if (Size > 1 && Data[1] == 'i') {
       Sink = 2;
       if (Size > 2 && Data[2] == '!') {
-        std::cout << "BINGO; Found the target, exiting\n";
+        std::cout << "BINGO; Found the target, exiting\n" << std::flush;
         exit(0);
       }
     }
   }
   return 0;
 }
 

llvm/lib/Fuzzer/test/SimpleThreadedTest.cpp

 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 
 // Threaded test for a fuzzer. The fuzzer should find "H"
 #include <assert.h>
 #include <cstddef>
 #include <cstdint>
 #include <cstring>
 #include <iostream>
+#include <ostream>
 #include <thread>
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
   auto C = [&] {
     if (Size >= 2 && Data[0] == 'H') {
-        std::cout << "BINGO; Found the target, exiting\n";
+        std::cout << "BINGO; Found the target, exiting\n" << std::flush;
         abort();
     }
   };
   std::thread T[] = {std::thread(C), std::thread(C), std::thread(C),
                      std::thread(C), std::thread(C), std::thread(C)};
   for (auto &X : T)
     X.join();
   return 0;
 }
 

llvm/lib/Fuzzer/test/exit-report.test

+RUN: not LLVMFuzzer-SimpleTest 2>&1 | FileCheck %s
+
+CHECK: ERROR: libFuzzer: fuzz target exited
+CHECK: SUMMARY: libFuzzer: fuzz target exited
+CHECK: Test unit written to

llvm/lib/Fuzzer/test/fuzzer-flags.test

 # Does not work on windows for unknown reason.
 UNSUPPORTED: windows
 
-RUN: LLVMFuzzer-FlagsTest -foo_bar=1 2>&1 | FileCheck %s --check-prefix=FOO_BAR
+RUN: not LLVMFuzzer-FlagsTest -foo_bar=1 2>&1 | FileCheck %s --check-prefix=FOO_BAR
 FOO_BAR: WARNING: unrecognized flag '-foo_bar=1'; use -help=1 to list all flags
 FOO_BAR: BINGO
 
-RUN: LLVMFuzzer-FlagsTest -runs=10 --max_len=100 2>&1 | FileCheck %s --check-prefix=DASH_DASH
+RUN: not LLVMFuzzer-FlagsTest -runs=10 --max_len=100 2>&1 | FileCheck %s --check-prefix=DASH_DASH
 DASH_DASH: WARNING: did you mean '-max_len=100' (single dash)?
 DASH_DASH: INFO: A corpus is not provided, starting from an empty corpus
 
 RUN: LLVMFuzzer-FlagsTest -help=1 2>&1 | FileCheck %s --check-prefix=NO_INTERNAL
 NO_INTERNAL-NOT: internal flag
 
-RUN: LLVMFuzzer-FlagsTest --foo-bar -runs=10 -ignore_remaining_args=1 --baz -help=1 test 2>&1 | FileCheck %s --check-prefix=PASSTHRU
+RUN: not LLVMFuzzer-FlagsTest --foo-bar -runs=10 -ignore_remaining_args=1 --baz -help=1 test 2>&1 | FileCheck %s --check-prefix=PASSTHRU
 PASSTHRU: BINGO --foo-bar --baz -help=1 test
 
 RUN: mkdir -p %t/T0 %t/T1
 RUN: touch %t/T1/empty
-RUN: LLVMFuzzer-FlagsTest --foo-bar -merge=1 %t/T0 %t/T1 -ignore_remaining_args=1 --baz -help=1 test 2>&1 | FileCheck %s --check-prefix=PASSTHRU-MERGE
+RUN: not LLVMFuzzer-FlagsTest --foo-bar -merge=1 %t/T0 %t/T1 -ignore_remaining_args=1 --baz -help=1 test 2>&1 | FileCheck %s --check-prefix=PASSTHRU-MERGE
 PASSTHRU-MERGE: BINGO --foo-bar --baz -help=1 test

llvm/lib/Fuzzer/test/fuzzer-printcovpcs.test

-RUN: LLVMFuzzer-SimpleTest -print_pcs=1 -seed=1 2>&1 | FileCheck %s --check-prefix=PCS
+RUN: not LLVMFuzzer-SimpleTest -print_pcs=1 -seed=1 2>&1 | FileCheck %s --check-prefix=PCS
 PCS-NOT: NEW_PC
 PCS:INITED
 PCS:NEW_PC: {{0x[a-f0-9]+}}
 PCS:NEW_PC: {{0x[a-f0-9]+}}
 PCS:NEW
 PCS:BINGO
 

llvm/lib/Fuzzer/test/fuzzer.test

 CHECK: BINGO
 Done1000000: Done 1000000 runs in
 
-RUN: LLVMFuzzer-SimpleTest 2>&1 | FileCheck %s
+RUN: not LLVMFuzzer-SimpleTest 2>&1 | FileCheck %s
 
 # only_ascii mode. Will perform some minimal self-validation.
-RUN: LLVMFuzzer-SimpleTest -only_ascii=1 2>&1
+RUN: not LLVMFuzzer-SimpleTest -only_ascii=1 2>&1
 
 RUN: LLVMFuzzer-SimpleCmpTest -max_total_time=1 -use_cmp=0 2>&1 | FileCheck %s --check-prefix=MaxTotalTime
 MaxTotalTime: Done {{.*}} runs in {{.}} second(s)
 
 RUN: not LLVMFuzzer-NullDerefTest                  2>&1 | FileCheck %s --check-prefix=NullDerefTest
 RUN: not LLVMFuzzer-NullDerefTest -close_fd_mask=3 2>&1 | FileCheck %s --check-prefix=NullDerefTest
 NullDerefTest: ERROR: AddressSanitizer: {{SEGV|access-violation}} on unknown address
 NullDerefTest: Test unit written to ./crash-

 OOB: is located 0 bytes to the right of 3-byte region
 
 RUN: not LLVMFuzzer-InitializeTest -use_value_profile=1 2>&1 | FileCheck %s
 
 RUN: not LLVMFuzzer-DSOTest 2>&1 | FileCheck %s --check-prefix=DSO
 DSO: INFO: Loaded 3 modules
 DSO: BINGO
 
-RUN: LLVMFuzzer-SimpleTest  -exit_on_src_pos=SimpleTest.cpp:17                 2>&1 | FileCheck %s --check-prefix=EXIT_ON_SRC_POS
+RUN: LLVMFuzzer-SimpleTest  -exit_on_src_pos=SimpleTest.cpp:18                 2>&1 | FileCheck %s --check-prefix=EXIT_ON_SRC_POS
 RUN: LLVMFuzzer-ShrinkControlFlowTest  -exit_on_src_pos=ShrinkControlFlowTest.cpp:23 2>&1 | FileCheck %s --check-prefix=EXIT_ON_SRC_POS
 EXIT_ON_SRC_POS: INFO: found line matching '{{.*}}', exiting.
 
 RUN: env ASAN_OPTIONS=strict_string_checks=1 not LLVMFuzzer-StrncmpOOBTest -seed=1 -runs=1000000 2>&1 | FileCheck %s --check-prefix=STRNCMP
 STRNCMP: AddressSanitizer: heap-buffer-overflow
 STRNCMP-NOT: __sanitizer_weak_hook_strncmp
 STRNCMP: in LLVMFuzzerTestOneInput
 
 RUN: not LLVMFuzzer-BogusInitializeTest 2>&1 | FileCheck %s --check-prefix=BOGUS_INITIALIZE
 BOGUS_INITIALIZE: argv[0] has been modified in LLVMFuzzerInitialize

llvm/lib/Fuzzer/test/inline-8bit-counters.test

 REQUIRES: linux
 CHECK: INFO: Loaded 1 modules with {{.*}} inline 8-bit counters
 CHECK: BINGO
-RUN: LLVMFuzzer-SimpleTest-Inline8bitCounters -runs=1000000 -seed=1 2>&1 | FileCheck %s
+RUN: not LLVMFuzzer-SimpleTest-Inline8bitCounters -runs=1000000 -seed=1 2>&1 | FileCheck %s

llvm/lib/Fuzzer/test/repeated-bytes.test

 CHECK: BINGO
-RUN: LLVMFuzzer-RepeatedBytesTest -seed=1 -runs=1000000 2>&1 | FileCheck %s
+RUN: not LLVMFuzzer-RepeatedBytesTest -seed=1 -runs=1000000 2>&1 | FileCheck %s

llvm/lib/Fuzzer/test/trace-pc.test

 CHECK: BINGO
-RUN: LLVMFuzzer-SimpleTest-TracePC -runs=100000 -seed=1 2>&1 | FileCheck %s
+RUN: not LLVMFuzzer-SimpleTest-TracePC -runs=100000 -seed=1 2>&1 | FileCheck %s

llvm/lib/Fuzzer/test/ulimit.test

 REQUIRES: posix
 
 RUN: ulimit -s 1000
-RUN: LLVMFuzzer-SimpleTest
+RUN: not LLVMFuzzer-SimpleTest