[analyzer] Add test directory for scan-build

[dcoughlin]

Summary

The static analyzer's scan-build script is critical infrastructure but
is not well tested. To start to address this, add a new test directory under
tests/Analysis for scan-build lit tests and seed it with several tests. The
goal is that future scan-build changes will be accompanied by corresponding
tests.

[NoQ]

Woohoo, that's much more tests than i made in D55683!

Let's hope it works on Windows :/

clang/test/Analysis/scan-build/plist_html_output.test

18–19

Clever!

[thakis]

If CLANG_INSTALL_SCANBUILD=NO, then this tries to run scan-build from PATH, where it might not exist. Does this need some lit feature to check if scan-build is being copied to the bin dir by the build system?

[thakis]

Also, you probably need to make clang/test/CMakeLists.txt depend on the install target, to make sure scan-build is in bin/.

[NoQ]

I'll try to re-land this. I heard there were a lot more issues than just honoring CLANG_INSTALL_SCANBUILD=NO (which i'll do my best to test before pushing), so i'll monitor the buildbots carefully and try to unbreak everything quickly.

clang/test/Analysis/scan-build/Inputs/multidirectory_project/directory1/file1.c

+int main() {
+  return 0;
+}
+
+void function1(int *p) {
+  if (!p) {
+    *p = 7; // This will emit a null pointer diagnostic.
+  }
+}

clang/test/Analysis/scan-build/Inputs/multidirectory_project/directory2/file2.c

+void function2(int *o) {
+  if (!o) {
+    *o = 7; // This will emit a null pointer diagnostic.
+  }
+}

clang/test/Analysis/scan-build/Inputs/single_null_dereference.c

+int main() {
+  int *p = 0;
+  *p = 7; // We expect a diagnostic about this.
+  return 0;
+}

clang/test/Analysis/scan-build/exclude_directories.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+// FIXME: Should ideally work on Windows.
+UNSUPPORTED: system-windows
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir %clang \
+RUN:     %S/Inputs/multidirectory_project/directory1/file1.c \
+RUN:     %S/Inputs/multidirectory_project/directory2/file2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-NO-EXCLUDE
+
+// The purpose of this test is to ensure that the --exclude command line option
+// actually excludes reports from inside the specified directories.
+
+
+// First, let's make sure that without --exclude issues in both
+// directory1 and directory2 are found.
+CHECK-NO-EXCLUDE: scan-build: 2 bugs found.
+
+
+// Only one issue should be found when directory1 is excluded.
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir --exclude directory1 %clang \
+RUN:     %S/Inputs/multidirectory_project/directory1/file1.c \
+RUN:     %S/Inputs/multidirectory_project/directory2/file2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-EXCLUDE1
+
+CHECK-EXCLUDE1: scan-build: 1 bug found.
+
+
+// When both directories are excluded, no issues should be reported.
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir --exclude directory1 --exclude directory2 %clang \
+RUN:     %S/Inputs/multidirectory_project/directory1/file1.c \
+RUN:     %S/Inputs/multidirectory_project/directory2/file2.c \
+RUN:     | FileCheck %s -check-prefix CHECK-EXCLUDE-BOTH
+
+CHECK-EXCLUDE-BOTH: scan-build: 0 bugs found.
+

clang/test/Analysis/scan-build/help.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+// FIXME: Should ideally work on Windows.
+UNSUPPORTED: system-windows
+
+RUN: %scan-build -h | FileCheck %s
+RUN: %scan-build --help | FileCheck %s
+
+Test for help output from scan-build.
+
+
+CHECK: USAGE: scan-build [options] <build command> [build options]
+
+...
+
+CHECK: AVAILABLE CHECKERS:
+...
+CHECK:  + core.NullDereference
+CHECK:    optin.performance.GCDAntipattern
+...
+
+
+

clang/test/Analysis/scan-build/html_output.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+// FIXME: Should ideally work on Windows.
+UNSUPPORTED: system-windows
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -o %t.output_dir %clang %S/Inputs/single_null_dereference.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+// Test html output
+
+CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
+CHECK-STDOUT: scan-build: 1 bug found.
+CHECK-STDOUT: scan-build: Run 'scan-view {{.*}}' to examine bug reports.
+
+// We expect an index file, a file for the report, and sibling support files.
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+CHECK-FILENAMES: index.html
+CHECK-FILENAMES: report-{{.*}}.html
+CHECK-FILENAMES: scanview.css
+CHECK-FILENAMES: sorttable.js
+
+
+// The index should have a link to the report for the single issue.
+RUN: cat %t.output_dir/*/index.html \
+RUN:     | FileCheck %s -check-prefix CHECK-INDEX-HTML
+
+CHECK-INDEX-HTML: <!-- REPORTBUG id="report-{{.*}}.html" -->
+
+// The report should describe the issue.
+RUN: cat %t.output_dir/*/report-*.html \
+RUN:     | FileCheck %s -check-prefix CHECK-REPORT-HTML
+
+CHECK-REPORT-HTML: <!-- BUGTYPE Dereference of null pointer -->

clang/test/Analysis/scan-build/lit.local.cfg

+# -*- Python -*-
+
+import lit.util
+import lit.formats
+import os
+
+use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
+config.test_format = lit.formats.ShTest(use_lit_shell == "0")
+
+config.substitutions.append(('%scan-build',
+                             '\'%s\' --use-analyzer=%s ' % (
+                                 lit.util.which('scan-build',
+                                                os.path.join(
+                                                    config.clang_src_dir,
+                                                    'tools',
+                                                    'scan-build',
+                                                    'bin')),
+                                 config.clang)))

clang/test/Analysis/scan-build/plist_html_output.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+// FIXME: Should ideally work on Windows.
+UNSUPPORTED: system-windows
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -plist-html -o %t.output_dir %clang %S/Inputs/single_null_dereference.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+// Test combined plist and html output with -plist-html
+
+CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
+CHECK-STDOUT: scan-build: Analysis run complete.
+CHECK-STDOUT: scan-build: Analysis results (plist files) deposited in '{{.*}}'
+CHECK-STDOUT: scan-build: 1 bug found.
+CHECK-STDOUT: scan-build: Run 'scan-view {{.*}}' to examine bug reports.
+
+// We expect both html files and the plist files.
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+CHECK-FILENAMES: index.html
+CHECK-FILENAMES-DAG: report-{{.*}}.html
+CHECK-FILENAMES-DAG: report-{{.*}}.plist
+CHECK-FILENAMES: scanview.css
+CHECK-FILENAMES: sorttable.js

clang/test/Analysis/scan-build/plist_output.test

+// FIXME: Actually, "perl".
+REQUIRES: shell
+
+// FIXME: Should ideally work on Windows.
+UNSUPPORTED: system-windows
+
+RUN: rm -rf %t.output_dir && mkdir %t.output_dir
+RUN: %scan-build -plist -o %t.output_dir %clang %S/Inputs/single_null_dereference.c \
+RUN:     | FileCheck %s -check-prefix CHECK-STDOUT
+
+// Test plist output
+
+CHECK-STDOUT: scan-build: Using '{{.*}}' for static analysis
+CHECK-STDOUT: scan-build: Analysis run complete.
+CHECK-STDOUT: scan-build: Analysis results (plist files) deposited in '{{.*}}'
+
+// We expect a single plist file
+RUN: ls %t.output_dir/*/ | FileCheck %s -check-prefix CHECK-FILENAMES
+
+CHECK-FILENAMES: report-{{.*}}.plist
+
+// The report should describe the issue.
+RUN: cat %t.output_dir/*/report-*.plist \
+RUN:     | FileCheck %s -check-prefix CHECK-REPORT-PLIST-CONTENTS
+
+CHECK-REPORT-PLIST-CONTENTS: <key>type</key><string>Dereference of null pointer</string>