Page MenuHomePhabricator

[llvm-profdata] Add intersect/exclude subcommands to aid differential code coverage analysis.
Needs ReviewPublic

Authored by dokyungs on Jul 28 2020, 10:28 AM.

Details

Summary

[llvm-profdata] Add intersect/exclude subcommands to aid differential code coverage analysis.

This patch adds llvm-profdata [intersect|exclude] subcommands to help with differential code coverage analysis.

  1. llvm-profdata intersect <base profdata> <test profdata> -o <output filename>

    takes two profile data as input, and outputs their intersection as a file on a given path.
  1. llvm-profdata exclude <base profdata> <test profdata> -o <output filename>

    takes two profile data as input (base and test profdata, respectively), reduces the the base profdata such that it excludes any data found in the test profdata, and outputs the resulting profile data as a file on a given path.

These commands can be useful when comparing two distinct code coverage data obtained from different sets of test cases. With these commands we can more easily identify code regions that are (i) commonly covered by different sets of test cases via llvm-profdata intersect, and (ii) only covered by certain set of test cases via llvm-profdata exclude.

Diff Detail

Unit TestsFailed

TimeTest
230 mslinux > AddressSanitizer-x86_64-linux-dynamic.TestCases::alloca_big_alignment.cpp
Script: -- : 'RUN: at line 1'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m64 -shared-libasan -O0 -mllvm -asan-instrument-dynamic-allocas /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/asan/TestCases/alloca_big_alignment.cpp -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/asan/X86_64LinuxDynamicConfig/TestCases/Output/alloca_big_alignment.cpp.tmp
210 mslinux > AddressSanitizer-x86_64-linux-dynamic.TestCases::alloca_detect_custom_size_.cpp
Script: -- : 'RUN: at line 1'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m64 -shared-libasan -O0 -mllvm -asan-instrument-dynamic-allocas /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/asan/TestCases/alloca_detect_custom_size_.cpp -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/asan/X86_64LinuxDynamicConfig/TestCases/Output/alloca_detect_custom_size_.cpp.tmp
250 mslinux > AddressSanitizer-x86_64-linux-dynamic.TestCases::alloca_overflow_partial.cpp
Script: -- : 'RUN: at line 1'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m64 -shared-libasan -O0 -mllvm -asan-instrument-dynamic-allocas /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/asan/TestCases/alloca_overflow_partial.cpp -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/asan/X86_64LinuxDynamicConfig/TestCases/Output/alloca_overflow_partial.cpp.tmp
240 mslinux > AddressSanitizer-x86_64-linux-dynamic.TestCases::alloca_overflow_right.cpp
Script: -- : 'RUN: at line 1'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m64 -shared-libasan -O0 -mllvm -asan-instrument-dynamic-allocas /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/asan/TestCases/alloca_overflow_right.cpp -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/asan/X86_64LinuxDynamicConfig/TestCases/Output/alloca_overflow_right.cpp.tmp
250 mslinux > AddressSanitizer-x86_64-linux-dynamic.TestCases::alloca_underflow_left.cpp
Script: -- : 'RUN: at line 1'; /mnt/disks/ssd0/agent/llvm-project/build/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m64 -shared-libasan -O0 -mllvm -asan-instrument-dynamic-allocas /mnt/disks/ssd0/agent/llvm-project/compiler-rt/test/asan/TestCases/alloca_underflow_left.cpp -o /mnt/disks/ssd0/agent/llvm-project/build/projects/compiler-rt/test/asan/X86_64LinuxDynamicConfig/TestCases/Output/alloca_underflow_left.cpp.tmp
View Full Test Results (959 Failed)

Event Timeline

dokyungs created this revision.Jul 28 2020, 10:28 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 28 2020, 10:28 AM
dokyungs requested review of this revision.Jul 28 2020, 10:28 AM
dokyungs updated this revision to Diff 281368.Jul 28 2020, 2:42 PM

Exit on errors, rename function names, and write comments.

dokyungs retitled this revision from [llvm-profdata] Add intersect/unique set operators to aid differential code coverage analysis. to [llvm-profdata] Add intersect/exclude subcommands to aid differential code coverage analysis..Jul 28 2020, 2:42 PM
dokyungs edited the summary of this revision. (Show Details)
dokyungs edited the summary of this revision. (Show Details)
morehouse added subscribers: vsk, davidxl, wmi.

Thanks, DK, this looks like approximately what we want.

@vsk, @wmi, @davidxl: Would you be willing to take this patch once we clean it up? The exclude operation is particularly useful for comparing coverage reports between fuzzing engines, or for checking new code coverage reached by a test case.

llvm/lib/ProfileData/InstrProf.cpp
653

It seems that we're adding all J's to ValueData, even if they don't intersect with existing values in ValueData. Is this what we really want?

751

This is nearly identical to mergeValueProfData. Can we reuse some code?

And same for uniqueValueProfData.

llvm/tools/llvm-profdata/llvm-profdata.cpp
1155

This is nearly identical to intersect_main. Can we reuse some code?

vsk added a comment.Jul 30 2020, 10:44 AM

I haven't had a chance to look at this patch yet, but have no objection to adding these subcommands.

dokyungs updated this revision to Diff 283038.Aug 4 2020, 3:19 PM
dokyungs edited the summary of this revision. (Show Details)

Fixed intersect/exclude logic; skip unmatching test profile data.

Added tests for both intersect/exclude subcommands to check their basic functionality.

Code deduplication/cleanup is work in progress; will be addressed in the next upload.

dokyungs updated this revision to Diff 284427.Aug 10 2020, 10:22 AM

Deduplicated almost identical code; merged intersect_main and
exclude_main into reduce_main, intersectValueProfdate and
excludeValueProfData into reduceValueProfData.

please also document the new commands in docs/CommandGuide/llvm-profdata.rst

dokyungs updated this revision to Diff 284473.Aug 10 2020, 12:25 PM

Document the new subcommands.

Support -text and -binary output only for the new subcommands.

dmajor added a subscriber: dmajor.Aug 11 2020, 6:16 AM