[utils] Test the code coverage regression-checking script

[vsk]

Summary

Make sure that it can parse llvm-cov report output, so that the script doesn't break when we update llvm-cov.

[MaggieYi]

Thanks for adding the test.

I have built the test in Windows Visual Studio and it failed due to that the environment variable (LLVM_SRC_ROOT) isn’t set in Visual Studio.

[vsk]

In D22692#494096, @MaggieYi wrote:

Thanks for adding the test.

I have built the test in Windows Visual Studio and it failed due to that the environment variable (LLVM_SRC_ROOT) isn’t set in Visual Studio.

Could you share more details about how you ran the test? I'm having a hard time understanding the failure because LLVM_SRC_ROOT is set in test/lit.cfg. That environment variable should be set on Windows builders if the test is run with llvm-lit test/tools/llvm-cov/report.cpp.

[MaggieYi]

In D22692#495195, @vsk wrote:

Could you share more details about how you ran the test? I'm having a hard time understanding the failure because LLVM_SRC_ROOT is set in test/lit.cfg. That environment variable should be set on Windows builders if the test is run with llvm-lit test/tools/llvm-cov/report.cpp.

I built all of LLVM projects and ran the regression tests on Windows using Visual Studio, which are mentioned in http://clang.llvm.org/get_started.html.

1. Check out LLVM/Clang;
2. Run CMake to generate the Visual Studio solution and project files: % mkdir build % cd build % cmake -G "Visual Studio 12" ..\llvm An LLVM.sln file is created in the build directory.
3. Open LLVM.sln in Visual Studio and build the "ALL_BUILD" project to build everything.
4. Build “check-all” project to run the regression tests. The report.cpp test failed during this step.

test/tools/llvm-cov/report.cpp

-// RUN: llvm-cov report %S/Inputs/report.covmapping -instr-profile %S/Inputs/report.profdata -filename-equivalence 2>&1 | FileCheck %s
+// RUN: llvm-cov report %S/Inputs/report.covmapping -instr-profile %S/Inputs/report.profdata -filename-equivalence > %t.report.txt
+// RUN: FileCheck %s -input-file=%t.report.txt
 // RUN: llvm-cov report %S/Inputs/report.covmapping -instr-profile %S/Inputs/report.profdata -filename-equivalence report.cpp 2>&1 | FileCheck -check-prefix=FILT-NEXT %s
 
 // CHECK:      Filename   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
 // CHECK-NEXT: ---
 // CHECK-NEXT: report.cpp       5                2     60.00%           4                 1    75.00%          13                 4    69.23%
 // CHECK-NEXT: ---
 // CHECK-NEXT: TOTAL            5                2     60.00%           4                 1    75.00%          13                 4    69.23%
 

 }
 
 // Test that listing multiple functions in a function view works.
 // RUN: llvm-cov show -o %t.dir %S/Inputs/report.covmapping -instr-profile=%S/Inputs/report.profdata -filename-equivalence -name-regex=".*" %s
 // RUN: FileCheck -check-prefix=FUNCTIONS -input-file %t.dir/functions.txt %s
 // FUNCTIONS: _Z3foob
 // FUNCTIONS: _Z3barv
 // FUNCTIONS: _Z4func
+
+// Test whether the coverage regression checking script can parse `report` output.
+// RUN: %python $LLVM_SRC_ROOT/utils/check-coverage-regressions.py %t.report.txt %t.report.txt | count 0