[libFuzzer] Added -print_full_coverage flag.

[ryancao]

Summary

-print_full_coverage=1 produces a detailed branch coverage dump when run on a single file.
Uses same infrastructure as -print_coverage flag, but prints all branches (regardless of coverage status) in an easy-to-parse format.
Usage: For internal use with machine learning fuzzing models which require detailed coverage information on seed files to generate mutations.

Patch by Ryan Cao (@ryancao).

[Dor1s]

Good job on uploading this for review, Ryan!

You also need to add a test for the new functionality. Tests for libFuzzer are located in https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/fuzzer

You can run them by executing ninja check-fuzzer command in your build directory

You can use https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/fuzzer/coverage.test as an example

The tests are written using LIT, which is somewhat intuitive, but not always.

You can see some quick clarifications about it in go/llvm-dev-starting-kit, or from various sources on the internet e.g.

https://llvm.org/docs/TestingGuide.html

or
https://llvm.org/devmtg/2019-10/slides/Homerding-Kruse-LLVMTestingInfrastructureTutorial.pdf scroll to slide 16

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

319

we should be able to re-use the existing RunOneTest function. You can explicitly set detect_leaks option to zero when print_full_coverage is used

compiler-rt/lib/fuzzer/FuzzerFlags.def

133

print_full_coverage sounds a bit better to me

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

470

I don't think you need this call, as we just want to execute the input and get its coverage.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

300

this function duplicates the most of the existing PrintCoverage function. Let's try to re-use the existing one, add CoveredPCs collection inside it, and branch the behavior at the end of the callback (where print calls are located) based on the value of the command line flag

[ryancao]

Addressed @Dor1s comments. Extra files in the diff under llvm/ have not been touched.

[Dor1s]

@morehouse could you please take a look as well, when you get a chance?

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

319

Why can't you use the existing RunOneTest as suggested in my previous comment here?

845

since we print the coverage information from within PrintFinalStats, it seems like we can just go through the if body starting on the line 817? This literally is a copy of it without some additional printing / parameters check.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

272

let's rename full to something like PrintAllCounters

332

this comment is incorrect, just delete it

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

842 ↗

(On Diff #287507)

this and some other files look unrelated, can you exclude them from your commit please?

[morehouse]

I'll take a look once the diff is fixed and Max's current comments are addressed.

[ryancao]

Updated to address @Dor1s comments.

[morehouse]

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

327

Why does the flag disable leak detection? We should either leave it enabled, or explain why it is disabled in the flag description.

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

357–360

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

306

Nit: please remove curly braces

310

Do we need to exclude the U and C when the vectors are empty? It's slightly cleaner to exclude these if-statements.

compiler-rt/lib/fuzzer/FuzzerTracePC.h

97

Remove const, it is pointless for non-pointers.

compiler-rt/test/fuzzer/full-coverage.test

5

What does -mllvm -use-unknown-locations=Disable do, and why do we need it?

[Dor1s]

Rebase

[Dor1s]

addressing review feedback from Matt

[Dor1s]

Matt, I'm trying to finish this on behalf of Ryan. Marty has been enabling the use of that feature on ClusterFuzz already, so it makes sense to move forward with this. Please take another look when you can :)

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

327

added a comment

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

357–360

done!

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

272

here as well

306

done

310

I think it depends on the consuming side. Marty, what do you think?

compiler-rt/lib/fuzzer/FuzzerTracePC.h

97

done

compiler-rt/test/fuzzer/full-coverage.test

5

Does seem necessary!

[mbarbella-chromium]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

310

This should be easy enough to handle, though it may need a small update to the training script. I agree that it's cleaner to output them unconditionally, but I don't have strong feelings either way.

[Dor1s]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

310

I think leaving this is-is would be slightly better, as at least from the user perspective it'd be explicit that this code was executed and the counters were empty.

[morehouse]

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

310

Leaving as-is actually leaves it a mystery whether this code was executed or not. I was suggesting printing the U and C unconditionally (removing the if statements).

[Dor1s]

print "U" and "C" unconditionally

[Dor1s]

Not that I like to merge CLs on Friday afternoon, but I'm OOO next week, so taking a shot now while I'm still online.

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

310

My bad, let me remove the conditions then. Thanks for clarifying this!

compiler-rt/test/fuzzer/full-coverage.test

5

something's wrong with me. I meant "does NOT seem necessary". Removed. Thanks Matt!

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

 
 static void StartRssThread(Fuzzer *F, size_t RssLimitMb) {
   if (!RssLimitMb)
     return;
   std::thread T(RssThread, F, RssLimitMb);
   T.detach();
 }
 
 int RunOneTest(Fuzzer *F, const char *InputFilePath, size_t MaxLen) {
   Unit U = FileToVector(InputFilePath);
   if (MaxLen && MaxLen < U.size())
     U.resize(MaxLen);
   F->ExecuteCallback(U.data(), U.size());
+  if (Flags.print_full_coverage) {
+    // Leak detection is not needed when collecting full coverage data.
+    F->TPCUpdateObservedPCs();
+  } else {
     F->TryDetectingAMemoryLeak(U.data(), U.size(), true);
+  }
   return 0;
 }
 
 static bool AllInputsAreFiles() {
   if (Inputs->empty()) return false;
   for (auto &Path : *Inputs)
     if (!IsFile(Path))
       return false;
int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   bool RunIndividualFiles = AllInputsAreFiles();
   Options.SaveArtifacts =
       !RunIndividualFiles || Flags.minimize_crash_internal_step;
   Options.PrintNewCovPcs = Flags.print_pcs;
   Options.PrintNewCovFuncs = Flags.print_funcs;
   Options.PrintFinalStats = Flags.print_final_stats;
   Options.PrintCorpusStats = Flags.print_corpus_stats;
   Options.PrintCoverage = Flags.print_coverage;
+  Options.PrintFullCoverage = Flags.print_full_coverage;
   if (Flags.exit_on_src_pos)
     Options.ExitOnSrcPos = Flags.exit_on_src_pos;
   if (Flags.exit_on_item)
     Options.ExitOnItem = Flags.exit_on_item;
   if (Flags.focus_function)
     Options.FocusFunction = Flags.focus_function;
   if (Flags.data_flow_trace)
     Options.DataFlowTrace = Flags.data_flow_trace;
if (Flags.minimize_crash)
     return MinimizeCrashInput(Args, Options);
 
   if (Flags.minimize_crash_internal_step)
     return MinimizeCrashInputInternalStep(F, Corpus);
 
   if (Flags.cleanse_crash)
     return CleanseCrashInput(Args, Options);
 
   if (RunIndividualFiles) {
     Options.SaveArtifacts = false;
     int Runs = std::max(1, Flags.runs);
     Printf("%s: Running %zd inputs %d time(s) each.\n", ProgName->c_str(),
            Inputs->size(), Runs);
     for (auto &Path : *Inputs) {
       auto StartTime = system_clock::now();
       Printf("Running: %s\n", Path.c_str());
       for (int Iter = 0; Iter < Runs; Iter++)

compiler-rt/lib/fuzzer/FuzzerFlags.def

FUZZER_FLAG_STRING(exact_artifact_path,
                    "and will not use checksum in the file name. Do not "
                    "use the same path for several parallel processes.")
 FUZZER_FLAG_INT(print_pcs, 0, "If 1, print out newly covered PCs.")
 FUZZER_FLAG_INT(print_funcs, 2, "If >=1, print out at most this number of "
                                 "newly covered functions.")
 FUZZER_FLAG_INT(print_final_stats, 0, "If 1, print statistics at exit.")
 FUZZER_FLAG_INT(print_corpus_stats, 0,
   "If 1, print statistics on corpus elements at exit.")
 FUZZER_FLAG_INT(print_coverage, 0, "If 1, print coverage information as text"
                                    " at exit.")
+FUZZER_FLAG_INT(print_full_coverage, 0, "If 1, print full coverage information "
+                                        "(all branches) as text at exit.")
 FUZZER_FLAG_INT(dump_coverage, 0, "Deprecated.")
 FUZZER_FLAG_INT(handle_segv, 1, "If 1, try to intercept SIGSEGV.")
 FUZZER_FLAG_INT(handle_bus, 1, "If 1, try to intercept SIGBUS.")
 FUZZER_FLAG_INT(handle_abrt, 1, "If 1, try to intercept SIGABRT.")
 FUZZER_FLAG_INT(handle_ill, 1, "If 1, try to intercept SIGILL.")
 FUZZER_FLAG_INT(handle_fpe, 1, "If 1, try to intercept SIGFPE.")
 FUZZER_FLAG_INT(handle_int, 1, "If 1, try to intercept SIGINT.")
 FUZZER_FLAG_INT(handle_term, 1, "If 1, try to intercept SIGTERM.")

compiler-rt/lib/fuzzer/FuzzerInternal.h

public:
   static void StaticInterruptCallback();
   static void StaticFileSizeExceedCallback();
   static void StaticGracefulExitCallback();
 
   void ExecuteCallback(const uint8_t *Data, size_t Size);
   bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile = false,
               InputInfo *II = nullptr, bool ForceAddToCorpus = false,
               bool *FoundUniqFeatures = nullptr);
+  void TPCUpdateObservedPCs();
 
   // Merge Corpora[1:] into Corpora[0].
   void Merge(const Vector<std::string> &Corpora);
   void CrashResistantMergeInternalStep(const std::string &ControlFilePath);
   MutationDispatcher &GetMD() { return MD; }
   void PrintFinalStats();
   void SetMaxInputLen(size_t MaxInputLen);
   void SetMaxMutationLen(size_t MaxMutationLen);

compiler-rt/lib/fuzzer/FuzzerLoop.cpp

if (Units)
     Printf(" units: %zd", Units);
 
   Printf(" exec/s: %zd", ExecPerSec);
   Printf(" rss: %zdMb", GetPeakRSSMb());
   Printf("%s", End);
 }
 
 void Fuzzer::PrintFinalStats() {
+  if (Options.PrintFullCoverage)
+    TPC.PrintCoverage(/*PrintAllCounters=*/true);
   if (Options.PrintCoverage)
-    TPC.PrintCoverage();
+    TPC.PrintCoverage(/*PrintAllCounters=*/false);
   if (Options.PrintCorpusStats)
     Corpus.PrintStats();
   if (!Options.PrintFinalStats)
     return;
   size_t ExecPerSec = execPerSec();
   Printf("stat::number_of_executed_units: %zd\n", TotalNumberOfRuns);
   Printf("stat::average_exec_per_sec:     %zd\n", ExecPerSec);
   Printf("stat::new_units_added:          %zd\n", NumberOfNewUnitsAdded);
static void RenameFeatureSetFile(const std::string &FeaturesDir,
                                  const std::string &NewFile) {
   if (FeaturesDir.empty()) return;
   RenameFile(DirPlusFile(FeaturesDir, OldFile),
              DirPlusFile(FeaturesDir, NewFile));
 }
 
 static void WriteEdgeToMutationGraphFile(const std::string &MutationGraphFile,
                                          const InputInfo *II,
                                          const InputInfo *BaseII,
                                          const std::string &MS) {
   if (MutationGraphFile.empty())
     return;
 
   std::string Sha1 = Sha1ToString(II->Sha1);
 
   std::string OutputString;
 
if (II && FoundUniqFeaturesOfII &&
     Corpus.Replace(II, {Data, Data + Size});
     RenameFeatureSetFile(Options.FeaturesDir, OldFeaturesFile,
                          Sha1ToString(II->Sha1));
     return true;
   }
   return false;
 }
 
+void Fuzzer::TPCUpdateObservedPCs() { TPC.UpdateObservedPCs(); }
+
 size_t Fuzzer::GetCurrentUnitInFuzzingThead(const uint8_t **Data) const {
   assert(InFuzzingThread());
   *Data = CurrentUnitData;
   return CurrentUnitSize;
 }
 
 void Fuzzer::CrashOnOverwrittenData() {
   Printf("==%d== ERROR: libFuzzer: fuzz target overwrites its const input\n",

compiler-rt/lib/fuzzer/FuzzerOptions.h

struct FuzzingOptions {
   std::string StopFile;
   bool SaveArtifacts = true;
   bool PrintNEW = true; // Print a status line when new units are found;
   bool PrintNewCovPcs = false;
   int PrintNewCovFuncs = 0;
   bool PrintFinalStats = false;
   bool PrintCorpusStats = false;
   bool PrintCoverage = false;
+  bool PrintFullCoverage = false;
   bool DumpCoverage = false;
   bool DetectLeaks = true;
   int PurgeAllocatorIntervalSec = 1;
   int  TraceMalloc = 0;
   bool HandleAbrt = false;
   bool HandleAlrm = false;
   bool HandleBus = false;
   bool HandleFpe = false;

compiler-rt/lib/fuzzer/FuzzerTracePC.h

public:
 
   void ClearInlineCounters();
 
   void UpdateFeatureSet(size_t CurrentElementIdx, size_t CurrentElementSize);
   void PrintFeatureSet();
 
   void PrintModuleInfo();
 
-  void PrintCoverage();
+  void PrintCoverage(bool PrintAllCounters);
 
   template<class CallBack>
   void IterateCoveredFunctions(CallBack CB);
 
   void AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                          size_t n, bool StopAtZero);
 
   TableOfRecentCompares<uint32_t, 32> TORC4;

compiler-rt/lib/fuzzer/FuzzerTracePC.cpp

Printf("ERROR: Failed to set focus function. Make sure the function name is "
          "valid (%s) and symbolization is enabled.\n", FuncName.c_str());
   exit(1);
 }
 
 bool TracePC::ObservedFocusFunction() {
   return FocusFunctionCounterPtr && *FocusFunctionCounterPtr;
 }
 
-void TracePC::PrintCoverage() {
+void TracePC::PrintCoverage(bool PrintAllCounters) {
   if (!EF->__sanitizer_symbolize_pc ||
       !EF->__sanitizer_get_module_and_offset_for_pc) {
     Printf("INFO: __sanitizer_symbolize_pc or "
            "__sanitizer_get_module_and_offset_for_pc is not available,"
            " not printing coverage\n");
     return;
   }
-  Printf("COVERAGE:\n");
+  Printf(PrintAllCounters ? "FULL COVERAGE:\n" : "COVERAGE:\n");
   auto CoveredFunctionCallback = [&](const PCTableEntry *First,
                                      const PCTableEntry *Last,
                                      uintptr_t Counter) {
     assert(First < Last);
     auto VisualizePC = GetNextInstructionPc(First->PC);
     std::string FileStr = DescribePC("%s", VisualizePC);
     if (!IsInterestingCoverageFile(FileStr))
       return;
     std::string FunctionStr = DescribePC("%F", VisualizePC);
     if (FunctionStr.find("in ") == 0)
       FunctionStr = FunctionStr.substr(3);
     std::string LineStr = DescribePC("%l", VisualizePC);
     size_t NumEdges = Last - First;
     Vector<uintptr_t> UncoveredPCs;
+    Vector<uintptr_t> CoveredPCs;
     for (auto TE = First; TE < Last; TE++)
       if (!ObservedPCs.count(TE))
         UncoveredPCs.push_back(TE->PC);
+      else
+        CoveredPCs.push_back(TE->PC);
+
+    if (PrintAllCounters) {
+      Printf("U");
+      for (auto PC : UncoveredPCs)
+        Printf(DescribePC(" %l", GetNextInstructionPc(PC)).c_str());
+      Printf("\n");
+
+      Printf("C");
+      for (auto PC : CoveredPCs)
+        Printf(DescribePC(" %l", GetNextInstructionPc(PC)).c_str());
+      Printf("\n");
+    } else {
       Printf("%sCOVERED_FUNC: hits: %zd", Counter ? "" : "UN", Counter);
       Printf(" edges: %zd/%zd", NumEdges - UncoveredPCs.size(), NumEdges);
       Printf(" %s %s:%s\n", FunctionStr.c_str(), FileStr.c_str(),
              LineStr.c_str());
       if (Counter)
         for (auto PC : UncoveredPCs)
           Printf("  UNCOVERED_PC: %s\n",
                  DescribePC("%s:%l", GetNextInstructionPc(PC)).c_str());
+    }
   };
 
   IterateCoveredFunctions(CoveredFunctionCallback);
 }
 
 // Value profile.
 // We keep track of various values that affect control flow.
 // These values are inserted into a bit-set-based hash map.
 // Every new bit in the map is treated as a new coverage.
 //
 // For memcmp/strcmp/etc the interesting value is the length of the common
 // prefix of the parameters.
 // For cmp instructions the interesting value is a XOR of the parameters.
 // The interesting value is mixed up with the PC and is then added to the map.
 
 ATTRIBUTE_NO_SANITIZE_ALL
 void TracePC::AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                                 size_t n, bool StopAtZero) {
   if (!n) return;

compiler-rt/test/fuzzer/dso-cov-input.txt

+123457
 No newline at end of file

compiler-rt/test/fuzzer/full-coverage.test

+# FIXME: Disabled on Windows because -fPIC cannot be used to compile for Windows.
+UNSUPPORTED: windows
+XFAIL: s390x
+RUN: %cpp_compiler %S/DSO1.cpp -fPIC %ld_flags_rpath_so1 -O0 -shared -o %dynamiclib1
+RUN: %cpp_compiler %S/DSO2.cpp -fPIC %ld_flags_rpath_so2 -O0 -shared -o %dynamiclib2
+RUN: %cpp_compiler %S/DSOTestMain.cpp %S/DSOTestExtra.cpp %ld_flags_rpath_exe1 %ld_flags_rpath_exe2 -o %t-DSOTest
+
+RUN: %run %t-DSOTest -print_full_coverage=1 %S/dso-cov-input.txt 2>&1 | FileCheck %s
+CHECK: FULL COVERAGE:
+CHECK-DAG: U{{( [0-9]+)*}}
+CHECK-DAG: C{{( [0-9]+)*}}
+CHECK-DAG: U{{( [0-9]+)*}}
+CHECK-DAG: U{{( [0-9]+)*}}
+CHECK-DAG: C{{( [0-9]+)*}}
+CHECK-DAG: U{{( [0-9]+)*}}