[libFuzzer] Added -print_full_coverage flag.

[ryancao]

Summary

-print_full_coverage=1 produces a detailed branch coverage dump when run on a single file.
Uses same infrastructure as -print_coverage flag, but prints all branches (regardless of coverage status) in an easy-to-parse format.
Usage: For internal use with machine learning fuzzing models which require detailed coverage information on seed files to generate mutations.

Patch by Ryan Cao (@ryancao).

[Dor1s]

Good job on uploading this for review, Ryan!

You also need to add a test for the new functionality. Tests for libFuzzer are located in https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/fuzzer

You can run them by executing ninja check-fuzzer command in your build directory

You can use https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/fuzzer/coverage.test as an example

The tests are written using LIT, which is somewhat intuitive, but not always.

You can see some quick clarifications about it in go/llvm-dev-starting-kit, or from various sources on the internet e.g.

https://llvm.org/docs/TestingGuide.html

or
https://llvm.org/devmtg/2019-10/slides/Homerding-Kruse-LLVMTestingInfrastructureTutorial.pdf scroll to slide 16

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

297

we should be able to re-use the existing RunOneTest function. You can explicitly set detect_leaks option to zero when print_full_coverage is used

compiler-rt/lib/fuzzer/FuzzerFlags.def

114

print_full_coverage sounds a bit better to me

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

470

I don't think you need this call, as we just want to execute the input and get its coverage.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

300

this function duplicates the most of the existing PrintCoverage function. Let's try to re-use the existing one, add CoveredPCs collection inside it, and branch the behavior at the end of the callback (where print calls are located) based on the value of the command line flag

[ryancao]

Addressed @Dor1s comments. Extra files in the diff under llvm/ have not been touched.

[Dor1s]

@morehouse could you please take a look as well, when you get a chance?

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

297

Why can't you use the existing RunOneTest as suggested in my previous comment here?

802

since we print the coverage information from within PrintFinalStats, it seems like we can just go through the if body starting on the line 817? This literally is a copy of it without some additional printing / parameters check.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

326

let's rename full to something like PrintAllCounters

369

this comment is incorrect, just delete it

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

842 ↗

(On Diff #287507)

this and some other files look unrelated, can you exclude them from your commit please?

[morehouse]

I'll take a look once the diff is fixed and Max's current comments are addressed.

[ryancao]

Updated to address @Dor1s comments.

[morehouse]

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

313

Why does the flag disable leak detection? We should either leave it enabled, or explain why it is disabled in the flag description.

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

357–360

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

360

Nit: please remove curly braces

364

Do we need to exclude the U and C when the vectors are empty? It's slightly cleaner to exclude these if-statements.

compiler-rt/lib/fuzzer/FuzzerTracePC.h

98

Remove const, it is pointless for non-pointers.

compiler-rt/test/fuzzer/full-coverage.test

4 ↗

(On Diff #291121)

What does -mllvm -use-unknown-locations=Disable do, and why do we need it?

[Dor1s]

Rebase

[Dor1s]

addressing review feedback from Matt

[Dor1s]

Matt, I'm trying to finish this on behalf of Ryan. Marty has been enabling the use of that feature on ClusterFuzz already, so it makes sense to move forward with this. Please take another look when you can :)

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

313

added a comment

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

357–360

done!

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

326

here as well

360

done

364

I think it depends on the consuming side. Marty, what do you think?

compiler-rt/lib/fuzzer/FuzzerTracePC.h

98

done

compiler-rt/test/fuzzer/full-coverage.test

4 ↗

(On Diff #291121)

Does seem necessary!

[mbarbella-chromium]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

364

This should be easy enough to handle, though it may need a small update to the training script. I agree that it's cleaner to output them unconditionally, but I don't have strong feelings either way.

[Dor1s]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

364

I think leaving this is-is would be slightly better, as at least from the user perspective it'd be explicit that this code was executed and the counters were empty.

[morehouse]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

364

Leaving as-is actually leaves it a mystery whether this code was executed or not. I was suggesting printing the U and C unconditionally (removing the if statements).

[Dor1s]

print "U" and "C" unconditionally

[Dor1s]

Not that I like to merge CLs on Friday afternoon, but I'm OOO next week, so taking a shot now while I'm still online.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

364

My bad, let me remove the conditions then. Thanks for clarifying this!

compiler-rt/test/fuzzer/full-coverage.test

4 ↗

(On Diff #291121)

something's wrong with me. I meant "does NOT seem necessary". Removed. Thanks Matt!

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

 
 static void StartRssThread(Fuzzer *F, size_t RssLimitMb) {
   if (!RssLimitMb)
     return;
   std::thread T(RssThread, F, RssLimitMb);
   T.detach();
 }
 
+int RunOneTestCov(Fuzzer *F, const char *InputFilePath, size_t MaxLen) {
+  Unit U = FileToVector(InputFilePath);
+  if (MaxLen && MaxLen < U.size())
+    U.resize(MaxLen);
+  F->RunOneTestHelper(U.data(), U.size());
+  return 0;
+}
+
 int RunOneTest(Fuzzer *F, const char *InputFilePath, size_t MaxLen) {
   Unit U = FileToVector(InputFilePath);
   if (MaxLen && MaxLen < U.size())
     U.resize(MaxLen);
   F->ExecuteCallback(U.data(), U.size());
   F->TryDetectingAMemoryLeak(U.data(), U.size(), true);
   return 0;
 }
 
 static bool AllInputsAreFiles() {
   if (Inputs->empty()) return false;
   for (auto &Path : *Inputs)
     if (!IsFile(Path))
       return false;
   return true;
 }
 
if (Flags.verbosity > 0 && !Dictionary.empty())
     Printf("Dictionary: %zd entries\n", Dictionary.size());
   bool RunIndividualFiles = AllInputsAreFiles();
   Options.SaveArtifacts =
       !RunIndividualFiles || Flags.minimize_crash_internal_step;
   Options.PrintNewCovPcs = Flags.print_pcs;
   Options.PrintNewCovFuncs = Flags.print_funcs;
   Options.PrintFinalStats = Flags.print_final_stats;
   Options.PrintCorpusStats = Flags.print_corpus_stats;
+  Options.PrintRawCoverage = Flags.print_raw_coverage;
   Options.PrintCoverage = Flags.print_coverage;
   if (Flags.exit_on_src_pos)
     Options.ExitOnSrcPos = Flags.exit_on_src_pos;
   if (Flags.exit_on_item)
     Options.ExitOnItem = Flags.exit_on_item;
   if (Flags.focus_function)
     Options.FocusFunction = Flags.focus_function;
   if (Flags.data_flow_trace)
if (Flags.minimize_crash)
     return MinimizeCrashInput(Args, Options);
 
   if (Flags.minimize_crash_internal_step)
     return MinimizeCrashInputInternalStep(F, Corpus);
 
   if (Flags.cleanse_crash)
     return CleanseCrashInput(Args, Options);
 
+  if (Flags.print_raw_coverage) {
+    if (!RunIndividualFiles) {
+      Printf("ERROR: -print-raw-coverage=1 can only be used with "
+             "individual files.\n");
+      exit(1);
+    }
+    Options.SaveArtifacts = false;
+    for (auto &Path : *Inputs) {
+      RunOneTestCov(F, Path.c_str(), Options.MaxLen);
+    }
+    F->PrintFinalStats();
+    exit(0);
+  }
+
   if (RunIndividualFiles) {
     Options.SaveArtifacts = false;
     int Runs = std::max(1, Flags.runs);
     Printf("%s: Running %zd inputs %d time(s) each.\n", ProgName->c_str(),
            Inputs->size(), Runs);
     for (auto &Path : *Inputs) {
       auto StartTime = system_clock::now();
       Printf("Running: %s\n", Path.c_str());

compiler-rt/lib/fuzzer/FuzzerFlags.def

FUZZER_FLAG_STRING(exact_artifact_path,
                    "and will not use checksum in the file name. Do not "
                    "use the same path for several parallel processes.")
 FUZZER_FLAG_INT(print_pcs, 0, "If 1, print out newly covered PCs.")
 FUZZER_FLAG_INT(print_funcs, 2, "If >=1, print out at most this number of "
                                 "newly covered functions.")
 FUZZER_FLAG_INT(print_final_stats, 0, "If 1, print statistics at exit.")
 FUZZER_FLAG_INT(print_corpus_stats, 0,
   "If 1, print statistics on corpus elements at exit.")
+FUZZER_FLAG_INT(print_raw_coverage, 0, "If 1, print raw coverage information "
+                                       "(all branches) as text at exit.")
 FUZZER_FLAG_INT(print_coverage, 0, "If 1, print coverage information as text"
                                    " at exit.")
 FUZZER_FLAG_INT(dump_coverage, 0, "Deprecated.")
 FUZZER_FLAG_INT(handle_segv, 1, "If 1, try to intercept SIGSEGV.")
 FUZZER_FLAG_INT(handle_bus, 1, "If 1, try to intercept SIGBUS.")
 FUZZER_FLAG_INT(handle_abrt, 1, "If 1, try to intercept SIGABRT.")
 FUZZER_FLAG_INT(handle_ill, 1, "If 1, try to intercept SIGILL.")
 FUZZER_FLAG_INT(handle_fpe, 1, "If 1, try to intercept SIGFPE.")

compiler-rt/lib/fuzzer/FuzzerInternal.h

public:
   static void StaticAlarmCallback();
   static void StaticCrashSignalCallback();
   static void StaticExitCallback();
   static void StaticInterruptCallback();
   static void StaticFileSizeExceedCallback();
   static void StaticGracefulExitCallback();
 
   void ExecuteCallback(const uint8_t *Data, size_t Size);
+  void RunOneTestHelper(const uint8_t *Data, size_t Size);
   bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile = false,
               InputInfo *II = nullptr, bool *FoundUniqFeatures = nullptr);
 
   // Merge Corpora[1:] into Corpora[0].
   void Merge(const Vector<std::string> &Corpora);
   void CrashResistantMergeInternalStep(const std::string &ControlFilePath);
   MutationDispatcher &GetMD() { return MD; }
   void PrintFinalStats();

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

if (Units)
     Printf(" units: %zd", Units);
 
   Printf(" exec/s: %zd", ExecPerSec);
   Printf(" rss: %zdMb", GetPeakRSSMb());
   Printf("%s", End);
 }
 
 void Fuzzer::PrintFinalStats() {
+  if (Options.PrintRawCoverage)
+    TPC.PrintRawCoverage();
   if (Options.PrintCoverage)
     TPC.PrintCoverage();
   if (Options.PrintCorpusStats)
     Corpus.PrintStats();
   if (!Options.PrintFinalStats)
     return;
   size_t ExecPerSec = execPerSec();
   Printf("stat::number_of_executed_units: %zd\n", TotalNumberOfRuns);
   Printf("stat::average_exec_per_sec:     %zd\n", ExecPerSec);
   Printf("stat::new_units_added:          %zd\n", NumberOfNewUnitsAdded);

 static void RenameFeatureSetFile(const std::string &FeaturesDir,
                                  const std::string &OldFile,
                                  const std::string &NewFile) {
   if (FeaturesDir.empty()) return;
   RenameFile(DirPlusFile(FeaturesDir, OldFile),
              DirPlusFile(FeaturesDir, NewFile));
 }
 
+void Fuzzer::RunOneTestHelper(const uint8_t *Data, size_t Size) {
+  ExecuteCallback(Data, Size);
+  TryDetectingAMemoryLeak(Data, Size, true);
+  TPC.UpdateObservedPCs();
+}
+
 bool Fuzzer::RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile,
                     InputInfo *II, bool *FoundUniqFeatures) {
   if (!Size)
     return false;
 
   ExecuteCallback(Data, Size);
 
   UniqFeatureSetTmp.clear();

compiler-rt/lib/fuzzer/FuzzerOptions.h

struct FuzzingOptions {
   std::string FeaturesDir;
   std::string StopFile;
   bool SaveArtifacts = true;
   bool PrintNEW = true; // Print a status line when new units are found;
   bool PrintNewCovPcs = false;
   int PrintNewCovFuncs = 0;
   bool PrintFinalStats = false;
   bool PrintCorpusStats = false;
+  bool PrintRawCoverage = false;
   bool PrintCoverage = false;
   bool DumpCoverage = false;
   bool DetectLeaks = true;
   int PurgeAllocatorIntervalSec = 1;
   int  TraceMalloc = 0;
   bool HandleAbrt = false;
   bool HandleAlrm = false;
   bool HandleBus = false;

compiler-rt/lib/fuzzer/FuzzerTracePC.h

public:
 
   void ClearInlineCounters();
 
   void UpdateFeatureSet(size_t CurrentElementIdx, size_t CurrentElementSize);
   void PrintFeatureSet();
 
   void PrintModuleInfo();
 
+  void PrintRawCoverage();
   void PrintCoverage();
 
   template<class CallBack>
   void IterateCoveredFunctions(CallBack CB);
 
   void AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                          size_t n, bool StopAtZero);
 
   TableOfRecentCompares<uint32_t, 32> TORC4;

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

Printf("ERROR: Failed to set focus function. Make sure the function name is "
          "valid (%s) and symbolization is enabled.\n", FuncName.c_str());
   exit(1);
 }
 
 bool TracePC::ObservedFocusFunction() {
   return FocusFunctionCounterPtr && *FocusFunctionCounterPtr;
 }
 
+void TracePC::PrintRawCoverage() {
+  if (!EF->__sanitizer_symbolize_pc ||
+      !EF->__sanitizer_get_module_and_offset_for_pc) {
+    Printf("INFO: __sanitizer_symbolize_pc or "
+           "__sanitizer_get_module_and_offset_for_pc is not available,"
+           " not printing raw coverage\n");
+    return;
+  }
+  Printf("RAW COVERAGE:\n");
+
+  auto RawCoveredFunctionCallback = [&](const PCTableEntry *First,
+                                        const PCTableEntry *Last,
+                                        uintptr_t Counter) {
+    assert(First < Last);
+    auto VisualizePC = GetNextInstructionPc(First->PC);
+
+    // Grabs filename as a string.
+    std::string FileStr = DescribePC("%s", VisualizePC);
+    if (!IsInterestingCoverageFile(FileStr))
+      return;
+
+    // Gets all uncovered (and covered) program counters.
+    Vector<uintptr_t> UncoveredPCs;
+    Vector<uintptr_t> CoveredPCs;
+    for (auto TE = First; TE < Last; TE++) {
+      if (!ObservedPCs.count(TE)) {
+        UncoveredPCs.push_back(TE->PC);
+      } else {
+        CoveredPCs.push_back(TE->PC);
+      }
+    }
+
+    // Print all the program counters (whether covered or not) in consistent
+    // fashion.
+    if (UncoveredPCs.size() > 0) {
+      Printf("U");
+      for (auto PC : UncoveredPCs) {
+        Printf(DescribePC(" %l", GetNextInstructionPc(PC)).c_str());
+      }
+      Printf("\n");
+    }
+
+    if (CoveredPCs.size() > 0) {
+      Printf("C");
+      for (auto PC : CoveredPCs) {
+        Printf(DescribePC(" %l", GetNextInstructionPc(PC)).c_str());
+      }
+      Printf("\n");
+    }
+  };
+
+  IterateCoveredFunctions(RawCoveredFunctionCallback);
+}
+
 void TracePC::PrintCoverage() {
   if (!EF->__sanitizer_symbolize_pc ||
       !EF->__sanitizer_get_module_and_offset_for_pc) {
     Printf("INFO: __sanitizer_symbolize_pc or "
            "__sanitizer_get_module_and_offset_for_pc is not available,"
            " not printing coverage\n");
     return;
   }
   Printf("COVERAGE:\n");
auto CoveredFunctionCallback = [&](const PCTableEntry *First,
     Printf("%sCOVERED_FUNC: hits: %zd", Counter ? "" : "UN", Counter);
     Printf(" edges: %zd/%zd", NumEdges - UncoveredPCs.size(), NumEdges);
     Printf(" %s %s:%s\n", FunctionStr.c_str(), FileStr.c_str(),
            LineStr.c_str());
     if (Counter)
       for (auto PC : UncoveredPCs)
         Printf("  UNCOVERED_PC: %s\n",
                DescribePC("%s:%l", GetNextInstructionPc(PC)).c_str());
   };
-
   IterateCoveredFunctions(CoveredFunctionCallback);
 }
 
 // Value profile.
 // We keep track of various values that affect control flow.
 // These values are inserted into a bit-set-based hash map.
 // Every new bit in the map is treated as a new coverage.
 //
 // For memcmp/strcmp/etc the interesting value is the length of the common
 // prefix of the parameters.
 // For cmp instructions the interesting value is a XOR of the parameters.
 // The interesting value is mixed up with the PC and is then added to the map.
 
 ATTRIBUTE_NO_SANITIZE_ALL
 void TracePC::AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                                 size_t n, bool StopAtZero) {
   if (!n) return;