This is an archive of the discontinued LLVM Phabricator instance.

[NewPM][Sancov] Make Sancov a Module Pass instead of 2 Passes
ClosedPublic

Authored by leonardchan on Aug 29 2019, 6:19 PM.

Details

Summary

This patch merges the sancov module and funciton passes into one module pass.

The reason for this is because we ran into an out of memory error when attempting to run asan fuzzer on some protobufs (pc.cc files). I traced the OOM error to the destructor of SanitizerCoverage where we only call appendTo[Compiler]Used which calls appendToUsedList. I'm not sure where precisely in appendToUsedList causes the OOM, but I am able to confirm that it's calling this function *repeatedly* that causes the OOM. (I hacked sancov a bit such that I can still create and destroy a new sancov on every function run, but only call appendToUsedList after all functions in the module have finished. This passes, but when I make it such that appendToUsedList is called on every sancov destruction, we hit OOM.)

I don't think the OOM is from just adding to the SmallSet and SmallVector inside appendToUsedList since in either case for a given module, they'll have the same max size. I suspect that when the existing llvm.compiler.used global is erased, the memory behind it isn't freed. I could be wrong on this though.

This patch works around the OOM issue by just calling appendToUsedList at the end of every module run instead of function run. The same amount of constants still get added to llvm.compiler.used, abd we make the pass usage and logic simpler by not having any inter-pass dependencies.

Diff Detail

Repository
rL LLVM

Event Timeline

leonardchan created this revision.Aug 29 2019, 6:19 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptAug 29 2019, 6:19 PM

*ping* This patch also seems to fix an issue with mismatched PC table and inline counters:

./out/Fuzz2/region_deserialize                                                                                                                                                                                           
INFO: Seed: 264880470
INFO: Loaded 1 modules   (170004 inline 8-bit counters): 170004 [0x2259c90, 0x22834a4), 
INFO: Loaded 1 PC tables (144267 PCs): 144267 [0x22834a8,0x24b6d58)
ERROR: The size of coverage PC tables does not match the
number of instrumented PCs. This might be a compiler bug,
please contact the libFuzzer developers.
Also check https://bugs.llvm.org/show_bug.cgi?id=34636
for possible workarounds (tl;dr: don't use the old GNU ld)
chandlerc accepted this revision.Sep 3 2019, 11:03 PM

LGTM

This also makes sense to me why it would OOM. I suspect we're building a massive list, and at best we get lucky and they get de-duped later. At worst, this is actually fixing a serious functionality problem. We had the same thing w/ normal ASan before too. Since this pass doesn't need to be a function pass anyways, this seems totally fine. Thanks for tracking it down.

This revision is now accepted and ready to land.Sep 3 2019, 11:03 PM
This revision was automatically updated to reflect the committed changes.