[analyzer] Update SATestBuild.py to enable a 'download and patch' model for projects.

[dcoughlin]

Summary

Currently the SATestBuild.py and SATestAdd.py buildbot scripts expect project
sources to be checked into the project repository. This patch changes these
scripts to additionally support a model where project sources are downloaded
rather than checked into the repository. Sometimes projects may need to be
modified (for example, to support a newer versions of clang), so the updated scripts
also allow for an optional patch file that will be applied to the downloaded
project source before analysis.

To support this workflow, this patch changes the expected layout of
a project in the repository. The project-specific helper scripts will stay
in the root of each project directory, but the benchmark source itself (if
checked into the repo) should now be stored in a subdirectory named
'CachedSource':

project_name/
  cleanup_run_static_analyzer.sh [optional]
  run_static_analyzer.cmd [required]
  download_project.sh [optional]
  CachedSource/ [optional]
  changes_for_analyzer.patch [optional]

If the 'CachedSource' source directory is not present, the download script will
be executed. This script should download the project source into 'CachedSource'.
Then, if 'changes_for_analyzer.patch' is present its changes will
be applied to a copy of 'CachedSource' before analysis.

[xazax.hun]

Hi!

I think it is great to support this model in these scripts. Do you plan to check the list of the project urls in to the repository as well? I think it would be great to do so, so one can easily reproduce a buildbot error locally, or even run the whole testsuit. It also makes it easier to setup alternative build bots and to contribute projects to the official one.

What do you think?

I have one minor high level note about the handling of patches. I think it in case a project would need more modifications, it is cleaner to have a separate patch file for each kind of them. Maybe it would be worth to support patching the project with multiple patches? E.g. applying all patches with a name prefix.

[dcoughlin]

In D14345#281475, @xazax.hun wrote:

Hi!

I think it is great to support this model in these scripts. Do you plan to check the list of the project urls in to the repository as well?

The project URLs are implicitly included in the download script (which is checked into the repository). So, for example, here is a sample script to download the OmniGroup project from github:

curl -L -O https://github.com/omnigroup/OmniGroup/archive/c9074bc24c46516687acf44824ad8b00c2146ad6.zip
unzip c9074bc24c46516687acf44824ad8b00c2146ad6.zip
mv OmniGroup-c9074bc24c46516687acf44824ad8b00c2146ad6 CachedSource

I have one minor high level note about the handling of patches. I think it in case a project would need more modifications, it is cleaner to have a separate patch file for each kind of them. Maybe it would be worth to support patching the project with multiple patches? E.g. applying all patches with a name prefix.

I originally did it this way but I found that became cumbersome. In order to fix an issue I had to copy the original pristine source, apply all previous patches to that copy, then copy the patched tree, then fix the issue in the second copy and the diff first copy with the second to generate the incremental patch.

With the single-patch approach the workflow is: copy the pristine source, apply the patch to the copy, fix the issue in the copy, and diff the pristine and the (now fixed) copy.

Since these patches are checked into the repo, they can always be differed with older versions of the patch to see what has changed.

cfe/trunk/utils/analyzer/SATestAdd.py

 #!/usr/bin/env python
 
 """
 Static Analyzer qualification infrastructure: adding a new project to
 the Repository Directory.
 
  Add a new project for testing: build it and add to the Project Map file.
    Assumes it's being run from the Repository Directory.
    The project directory should be added inside the Repository Directory and
    have the same name as the project ID
 
  The project should use the following files for set up:
-      - pre_run_static_analyzer.sh - prepare the build environment.
+      - cleanup_run_static_analyzer.sh - prepare the build environment.
                                      Ex: make clean can be a part of it.
       - run_static_analyzer.cmd - a list of commands to run through scan-build.
                                      Each command should be on a separate line.
                                      Choose from: configure, make, xcodebuild
+      - download_project.sh - download the project into the CachedSource/
+                                     directory. For example, download a zip of
+                                     the project source from GitHub, unzip it,
+                                     and rename the unzipped directory to
+                                     'CachedSource'. This script is not called
+                                     when 'CachedSource' is already present,
+                                     so an alternative is to check the
+                                     'CachedSource' directory into the
+                                     repository directly.
+      - CachedSource/ - An optional directory containing the source of the
+                                     project being analyzed. If present,
+                                     download_project.sh will not be called.
+      - changes_for_analyzer.patch - An optional patch file for any local changes
+                                     (e.g., to adapt to newer version of clang)
+                                     that should be applied to CachedSource
+                                     before analysis. To construct this patch,
+                                     run the the download script to download
+                                     the project to CachedSource, copy the
+                                     CachedSource to another directory (for
+                                     example, PatchedSource) and make any needed
+                                     modifications to the the copied source.
+                                     Then run:
+                                          diff -ur CachedSource PatchedSource \
+                                              > changes_for_analyzer.patch
 """
 import SATestBuild
 
 import os
 import csv
 import sys
 
 def isExistingProject(PMapFile, projectID) :

cfe/trunk/utils/analyzer/SATestBuild.py

 
 # Number of jobs.
 Jobs = int(math.ceil(detectCPUs() * 0.75))
 
 # Project map stores info about all the "registered" projects.
 ProjectMapFile = "projectMap.csv"
 
 # Names of the project specific scripts.
+# The script that downloads the project.
+DownloadScript = "download_project.sh"
 # The script that needs to be executed before the build can start.
 CleanupScript = "cleanup_run_static_analyzer.sh"
 # This is a file containing commands for scan-build.
 BuildScript = "run_static_analyzer.cmd"
 
 # The log file name.
 LogFolderName = "Logs"
 BuildLogName = "run_static_analyzer.log"
 # Summary file - contains the summary of the failures. Ex: This info can be be
 # displayed when buildbot detects a build failure.
 NumOfFailuresInSummary = 10
 FailuresSummaryFileName = "failures.txt"
 # Summary of the result diffs.
 DiffsSummaryFileName = "diffs.txt"
 
 # The scan-build result directory.
 SBOutputDirName = "ScanBuildResults"
 SBOutputDirReferencePrefix = "Ref"
 
+# The name of the directory storing the cached project source. If this directory
+# does not exist, the download script will be executed. That script should
+# create the "CachedSource" directory and download the project source into it.
+CachedSourceDirName = "CachedSource"
+
+# The name of the directory containing the source code that will be analyzed.
+# Each time a project is analyzed, a fresh copy of its CachedSource directory
+# will be copied to the PatchedSource directory and then the local patches
+# in PatchfileName will be applied (if PatchfileName exists).
+PatchedSourceDirName = "PatchedSource"
+
+# The name of the patchfile specifying any changes that should be applied
+# to the CachedSource before analyzing.
+PatchfileName = "changes_for_analyzer.patch"
+
 # The list of checkers used during analyzes.
 # Currently, consists of all the non-experimental checkers, plus a few alpha
 # checkers we don't want to regress on.
 Checkers="alpha.unix.SimpleStream,alpha.security.taint,cplusplus.NewDeleteLeaks,core,cplusplus,deadcode,security,unix,osx"
 
 Verbose = 1
 
 #------------------------------------------------------------------------------
 # Test harness logic.
 #------------------------------------------------------------------------------
 
 # Run pre-processing script if any.
 def runCleanupScript(Dir, PBuildLogFile):
+    Cwd = os.path.join(Dir, PatchedSourceDirName)
     ScriptPath = os.path.join(Dir, CleanupScript)
+    runScript(ScriptPath, PBuildLogFile, Cwd)
+
+# Run the script to download the project, if it exists.
+def runDownloadScript(Dir, PBuildLogFile):
+    ScriptPath = os.path.join(Dir, DownloadScript)
+    runScript(ScriptPath, PBuildLogFile, Dir)
+
+# Run the provided script if it exists.
+def runScript(ScriptPath, PBuildLogFile, Cwd):
     if os.path.exists(ScriptPath):
         try:
             if Verbose == 1:
                 print "  Executing: %s" % (ScriptPath,)
-            check_call("chmod +x %s" % ScriptPath, cwd = Dir,
+            check_call("chmod +x %s" % ScriptPath, cwd = Cwd,
                                               stderr=PBuildLogFile,
                                               stdout=PBuildLogFile,
                                               shell=True)
-            check_call(ScriptPath, cwd = Dir, stderr=PBuildLogFile,
-                                              stdout=PBuildLogFile,
-                                              shell=True)
-        except:
-            print "Error: The pre-processing step failed. See ", \
-                  PBuildLogFile.name, " for details."
-            sys.exit(-1)
+            check_call(ScriptPath, cwd = Cwd, stderr=PBuildLogFile,
+                                              stdout=PBuildLogFile,
+                                              shell=True)
+        except:
+            print "Error: Running %s failed. See %s for details." % (ScriptPath,
+                PBuildLogFile.name)
+            sys.exit(-1)
+
+# Download the project and apply the local patchfile if it exists.
+def downloadAndPatch(Dir, PBuildLogFile):
+    CachedSourceDirPath = os.path.join(Dir, CachedSourceDirName)
+
+    # If the we don't already have the cached source, run the project's
+    # download script to download it.
+    if not os.path.exists(CachedSourceDirPath):
+      runDownloadScript(Dir, PBuildLogFile)
+      if not os.path.exists(CachedSourceDirPath):
+        print "Error: '%s' not found after download." % (CachedSourceDirPath)
+        exit(-1)
+
+    PatchedSourceDirPath = os.path.join(Dir, PatchedSourceDirName)
+
+    # Remove potentially stale patched source.
+    if os.path.exists(PatchedSourceDirPath):
+        shutil.rmtree(PatchedSourceDirPath)
+
+    # Copy the cached source and apply any patches to the copy.
+    shutil.copytree(CachedSourceDirPath, PatchedSourceDirPath, symlinks=True)
+    applyPatch(Dir, PBuildLogFile)
+
+def applyPatch(Dir, PBuildLogFile):
+    PatchfilePath = os.path.join(Dir, PatchfileName)
+    PatchedSourceDirPath = os.path.join(Dir, PatchedSourceDirName)
+    if not os.path.exists(PatchfilePath):
+        print "  No local patches."
+        return
+
+    print "  Applying patch."
+    try:
+        check_call("patch -p1 < %s" % (PatchfilePath),
+                    cwd = PatchedSourceDirPath,
+                    stderr=PBuildLogFile,
+        print "Error: Patch failed. See %s for details." % (PBuildLogFile.name)
 
 # Build the project with scan-build by reading in the commands and
 # prefixing them with the scan-build options.
 def runScanBuild(Dir, SBOutputDir, PBuildLogFile):
     BuildScriptPath = os.path.join(Dir, BuildScript)
     if not os.path.exists(BuildScriptPath):
         print "Error: build script is not defined: %s" % BuildScriptPath
         sys.exit(-1)
 
     AllCheckers = Checkers
     if os.environ.has_key('SA_ADDITIONAL_CHECKERS'):
         AllCheckers = AllCheckers + ',' + os.environ['SA_ADDITIONAL_CHECKERS']
 
+    # Run scan-build from within the patched source directory.
+    SBCwd = os.path.join(Dir, PatchedSourceDirName)
+
     SBOptions = "--use-analyzer " + Clang + " "
     SBOptions += "-plist-html -o " + SBOutputDir + " "
     SBOptions += "-enable-checker " + AllCheckers + " "
     SBOptions += "--keep-empty "
     # Always use ccc-analyze to ensure that we can locate the failures
     # directory.
     SBOptions += "--override-compiler "
     try:
         SBCommandFile = open(BuildScriptPath, "r")
         SBPrefix = "scan-build " + SBOptions + " "
         for Command in SBCommandFile:
             Command = Command.strip()
             if len(Command) == 0:
                 continue;
             # If using 'make', auto imply a -jX argument
             # to speed up analysis.  xcodebuild will
             # automatically use the maximum number of cores.
             if (Command.startswith("make ") or Command == "make") and \
                 "-j" not in Command:
                 Command += " -j%d" % Jobs
             SBCommand = SBPrefix + Command
             if Verbose == 1:
                 print "  Executing: %s" % (SBCommand,)
-            check_call(SBCommand, cwd = Dir, stderr=PBuildLogFile,
+            check_call(SBCommand, cwd = SBCwd, stderr=PBuildLogFile,
                                                stdout=PBuildLogFile,
                                                shell=True)
     except:
         print "Error: scan-build failed. See ",PBuildLogFile.name,\
               " for details."
         raise
 
 def hasNoExtension(FileName):
     (Root, Ext) = os.path.splitext(FileName)
     if ((Ext == "")) :
def buildProject(Dir, SBOutputDir, ProjectBuildMode, IsReferenceBuild):
     assert(not os.path.exists(SBOutputDir))
     os.makedirs(os.path.join(SBOutputDir, LogFolderName))
 
     # Open the log file.
     PBuildLogFile = open(BuildLogPath, "wb+")
 
     # Build and analyze the project.
     try:
-        runCleanupScript(Dir, PBuildLogFile)
-
         if (ProjectBuildMode == 1):
+            downloadAndPatch(Dir, PBuildLogFile)
+            runCleanupScript(Dir, PBuildLogFile)
             runScanBuild(Dir, SBOutputDir, PBuildLogFile)
         else:
             runAnalyzePreprocessed(Dir, SBOutputDir, ProjectBuildMode)
 
         if IsReferenceBuild :
             runCleanupScript(Dir, PBuildLogFile)
 
             # Make the absolute paths relative in the reference results.
             for (DirPath, Dirnames, Filenames) in os.walk(SBOutputDir):
                 for F in Filenames:
                     if (not F.endswith('plist')):
                         continue
                     Plist = os.path.join(DirPath, F)
                     Data = plistlib.readPlist(Plist)
-                    Paths = [SourceFile[len(Dir)+1:] if SourceFile.startswith(Dir)\
+                    PathPrefix = Dir
+                    if (ProjectBuildMode == 1):
+                        PathPrefix = os.path.join(Dir, PatchedSourceDirName)
+                    Paths = [SourceFile[len(PathPrefix)+1:]\
+                              if SourceFile.startswith(PathPrefix)\
                               else SourceFile for SourceFile in Data['files']]
                     Data['files'] = Paths
                     plistlib.writePlist(Data, Plist)
 
     finally:
         PBuildLogFile.close()
 
     print "Build complete (time: %.2f). See the log for more details: %s" % \
            ((time.time()-TBegin), BuildLogPath)
for P in PairList:
         RefDir = P[0]
         NewDir = P[1]
 
         assert(RefDir != NewDir)
         if Verbose == 1:
             print "  Comparing Results: %s %s" % (RefDir, NewDir)
 
         DiffsPath = os.path.join(NewDir, DiffsSummaryFileName)
-        Opts = CmpRuns.CmpOptions(DiffsPath, "", Dir)
+        PatchedSourceDirPath = os.path.join(Dir, PatchedSourceDirName)
+        Opts = CmpRuns.CmpOptions(DiffsPath, "", PatchedSourceDirPath)
         # Discard everything coming out of stdout (CmpRun produces a lot of them).
         OLD_STDOUT = sys.stdout
         sys.stdout = Discarder()
         # Scan the results, delete empty plist files.
         NumDiffs, ReportsInRef, ReportsInNew = \
             CmpRuns.dumpScanBuildResultsDiff(RefDir, NewDir, Opts, False)
         sys.stdout = OLD_STDOUT
         if (NumDiffs > 0) :