This is an archive of the discontinued LLVM Phabricator instance.

Differential D16115

[test-suite] Add ClangAnalyzerBenchmarks directory to test-suite repository
ClosedPublic

Authored by dcoughlin on Jan 12 2016, 8:59 AM.

Details

Reviewers
MatzeB
• ddunbar
cmatthews
Summary

Hi All,

This patch proposes adding a new directory in the llvm test-suite repository to hold projects and build commands to detect regressions in the clang static analyzer. This directory will be used by a Green Dragon build bot (discussed in http://lists.llvm.org/pipermail/cfe-dev/2015-September/045199.html that will periodically re-analyze the projects specified in this directory to detect when analysis results change unexpectedly.

Rather than check in the source code for these projects, we propose to instead check in (1) scripts to download, cache, and build the project and (2) expected reference results in plist form from running the analyzer over the project. There are pre-existing scripts in utils/analyzer in the clang project that understand the layout of this directory (see http://reviews.llvm.org/D14345). Internally we have found this approach to be effective at discovering unintended regressions on real-world projects.

Diff Detail

Event Timeline

dcoughlin updated this revision to Diff 44594.Jan 12 2016, 8:59 AM
dcoughlin retitled this revision from to [test-suite] Add ClangAnalyzerBenchmarks directory to test-suite repository.
dcoughlin updated this object.
dcoughlin added reviewers: MatzeB, cmatthews, • ddunbar.
dcoughlin added subscribers: cfe-commits, llvm-commits.
MatzeB edited edge metadata.Jan 12 2016, 11:16 AM

Putting static-analyzer tests into the test-suite sounds good to me, the static analyzer is part of llvm so this should be fine. Though I'd like to wait a bit to hear other opinions on this matter.

As for the patch itself:

  • I'd call the directory ClangAnalyzer
  • For the external file you should provide a sha256 (or md5) sum and ideally check it in the download script, so other people can be sure that the reference results do indeed match the artifact being analyzed.
  • Why is there a subdirectory "2016-01-11-134007-41609-1" in the RefScanBuildResults directory, that name appears arbitrary for reference results.
dcoughlin updated this revision to Diff 44700.Jan 12 2016, 5:39 PM
dcoughlin edited edge metadata.

I've changed the directory name to 'ClangAnalyzer' and updated the download script to check to md5 hash of the downloaded tarball.

The inhumane directory names in the RefScanBuildResults directory are the direct output of the scan-build tool. A new and improved python reimplementation of scan-build just landed in trunk today with slightly better directory names. The directory names here will improve when the python reimplementation is ready to be used in build bot scripts.

MatzeB accepted this revision.Jan 12 2016, 5:48 PM
MatzeB edited edge metadata.

LGTM, though you should wait a week or two for other opinions before committing.

This revision is now accepted and ready to land.Jan 12 2016, 5:48 PM
dcoughlin added a comment.Jan 19 2016, 1:30 PM

Ping.

Any objections to adding a new 'ClangAnalyzer' directory to test-suite?

cmatthews edited edge metadata.Jan 19 2016, 2:17 PM
cmatthews added a subscriber: cmatthews.

LGTM too.

dcoughlin closed this revision.Jan 20 2016, 1:02 PM

Committed in r258336.

Large Diff

This large diff affects 104 files. Files without inline comments have been collapsed. Expand All Files

Revision Contents

PathSize
ClangAnalyzerBenchmarks/
16 lines
openssl-1.0.0t/
RefScanBuildResults/
2016-01-11-134007-41609-1/
2371 lines
89 lines
89 lines
89 lines
163 lines
89 lines
89 lines
89 lines
89 lines
1437 lines
89 lines
89 lines
89 lines
89 lines
1914 lines
163 lines
89 lines
1596 lines
163 lines
599 lines
1469 lines
8516 lines
89 lines
89 lines
89 lines
89 lines
89 lines
903 lines
89 lines
10131 lines
89 lines
733 lines
163 lines
89 lines
89 lines
3615 lines
1645 lines
1303 lines
311 lines
89 lines
1113 lines
311 lines
89 lines
89 lines
89 lines
89 lines
163 lines
89 lines
1552 lines
163 lines
107 lines
89 lines
860 lines
89 lines
89 lines
89 lines
107 lines
89 lines
311 lines
89 lines
89 lines
89 lines
12364 lines
89 lines
903 lines
89 lines
89 lines
1518 lines
3258 lines
3260 lines
163 lines
893 lines
89 lines
89 lines
163 lines
533 lines
89 lines
1712 lines
163 lines
89 lines
693 lines
89 lines
163 lines
311 lines
89 lines
89 lines
163 lines
89 lines
89 lines
9669 lines
510 lines
2552 lines
89 lines
237 lines
163 lines
89 lines
237 lines
163 lines
89 lines
2 lines
5 lines
3 lines
1 line

Diff 44594

ClangAnalyzerBenchmarks/README.txt

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-0jQzwH.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-0leXPD.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1SCq_n.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1q8xEn.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1rHTRW.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-25RkrY.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3LOwRg.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3eHerS.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3wD6sI.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-4Itn6Q.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-5eYzQk.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-5xKe72.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-70n0rJ.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-AJaHrD.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-A_g1PK.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-BSiP9f.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-BSjjch.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-D86sqa.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-DNg7FR.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ETVLuZ.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-FSqthW.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-FfhYET.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Fk0wPl.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-GGuLER.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Gd9Qk8.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-GzYIE8.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Hxm3jG.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-IEZlZo.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Jaelfi.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Jao0Kv.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KSlvnA.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KZzQTG.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KlLu18.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LCQ9Nf.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LFTkeC.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LnaNYn.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LtYUpC.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-NTClRG.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-O1rOZT.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-OQlhF8.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Ok4dLX.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-PQUmRQ.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-QDn98D.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ReQ_GL.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-RjfTIP.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-SnWTTb.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-StC4tP.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-SvYOOI.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TVmrcU.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TZ1H1p.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TfLDsd.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TigCu1.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-USeWg5.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-UTVx9V.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-VDaNX6.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-WJPjNs.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-WtKK5Z.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-XebXE5.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-YL5xpz.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Ylm0Go.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Zn3w9M.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ZrBRO3.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-_6fciA.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-aJqoCc.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ak55z2.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-b7Xvct.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-cI5v8I.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-cir2X2.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-d0igml.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-eqoyVD.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-fKa4Rk.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-gkuOso.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-hY88qG.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-i7I6_j.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-iBFRMw.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-j8J8u3.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-jP0jEK.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-jwkvXk.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-kX__b0.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-mSwyc0.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-mfm0do.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oYIpvl.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-o_yGd7.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ofrSfi.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oijYTP.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oxvtbW.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ryurLe.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-sbBgmn.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-tCJJyr.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-u9OAPv.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ufHmc_.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ustv4K.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-v3HsbB.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-voOhUi.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-wa7yxJ.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-wjqQWg.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-xDsepu.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-z5FdWe.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-zwVix9.plist

ClangAnalyzerBenchmarks/openssl-1.0.0t/cleanup_run_static_analyzer.sh

ClangAnalyzerBenchmarks/openssl-1.0.0t/download_project.sh

ClangAnalyzerBenchmarks/openssl-1.0.0t/run_static_analyzer.cmd

ClangAnalyzerBenchmarks/projectMap.csv