[Analyzer] Static analyzer script for updating reference results

[george.karpenkov]

Summary

The script updates reference results from the previous run, and stages them.
Requires https://reviews.llvm.org/D38156 before application.

[dcoughlin]

Other than the quoting issues mentioned inline, this looks good.

utils/analyzer/SATestUpdateDiffs.py

7 ↗

(On Diff #116278)

I'm not a super fan of using this kind of abbreviation because it makes it harder to figure out where the stuff being called is defined. Are you adamant about it?

49 ↗

(On Diff #116278)

I'm pretty terrified about rm -f without quotation of the arguments. Is there a safer API? At a minimum, can we at at least quote the arguments here? (And for the 'cp' above?)

[george.karpenkov]

Applying review comments.

[george.karpenkov]

utils/analyzer/SATestUpdateDiffs.py

7 ↗

(On Diff #116278)

No. We can change it.

49 ↗

(On Diff #116278)

Right, depends how far we want to go.
For cp above let's just quote it.
For rm the safest thing would be to walk all directories inside, and remove diffs.txt in each, but that's quite a bit more code.

[george.karpenkov]

@dcoughlin note that this can not land without https://reviews.llvm.org/D38156

cfe/trunk/utils/analyzer/SATestUpdateDiffs.py

+#!/usr/bin/env python
+
+"""
+Update reference results for static analyzer.
+"""
+
+from subprocess import check_call, check_output, CalledProcessError
+import csv
+import os
+import sys
+
+Verbose = 1
+def runCmd(Command):
+    if Verbose:
+        print "Executing %s" % Command
+    check_call(Command, shell=True)
+
+def updateReferenceResults(ProjName, ProjBuildMode):
+    ProjDir = SATestBuild.getProjectDir(ProjName)
+
+    RefResultsPath = os.path.join(ProjDir,
+            SATestBuild.getSBOutputDirName(IsReferenceBuild=True))
+    CreatedResultsPath = os.path.join(ProjDir,
+            SATestBuild.getSBOutputDirName(IsReferenceBuild=False))
+
+    if not os.path.exists(CreatedResultsPath):
+        print >> sys.stderr, "New results not found, was SATestBuild.py "\
+                             "previously run?"
+        sys.exit(-1)
+
+    # Remove reference results.
+    runCmd('git rm -r "%s"' % (RefResultsPath,))
+
+    # Replace reference results with a freshly computed once.
+    runCmd('cp -r "%s" "%s"' % (CreatedResultsPath, RefResultsPath,))
+
+    # Run cleanup script.
+    with open(SATestBuild.getBuildLogPath(RefResultsPath), "wb+")
+            as PBuildLogFile:
+        SATestBuild.runCleanupScript(ProjDir, PBuildLogFile)
+
+    SATestBuild.normalizeReferenceResults(ProjDir, RefResultsPath, ProjBuildMode)
+
+    # Clean up the generated difference results.
+    SATestBuild.cleanupReferenceResults(RefResultsPath)
+
+    # Remove the created .diffs file before adding.
+    runCmd('rm -f "%s/*/%s"' % (RefResultsPath, SATestBuild.DiffsSummaryFileName))
+
+    runCmd('git add "%s"' % (RefResultsPath,))
+
+def main(argv):
+    if len(argv) == 2 and argv[1] in ('-h', '--help'):
+        print >> sys.stderr, "Update static analyzer reference results based "\
+                             "\non the previous run of SATestBuild.py.\n"
+                             "\nN.B.: Assumes that SATestBuild.py was just run."
+        sys.exit(-1)
+
+    with SATestBuild.projectFileHandler() as f:
+        for (ProjName, ProjBuildMode) in SATestBuild.iterateOverProjects(f):
+            updateReferenceResults(ProjName, int(ProjBuildMode))
+
+if __name__ == '__main__':
+    main(sys.argv)